Jan Eitzinger 09501df3c2 fix: reduce memory usage in parquet checkpoint archiver ...

Stream CheckpointFile trees directly to parquet rows instead of
materializing all rows in a giant intermediate slice. This eliminates
~1.9GB per host of redundant allocations (repeated string headers)
and removes the expensive sort on millions of 104-byte structs.

Key changes:
- Replace flattenCheckpointFile + sortParquetRows + WriteHostRows with
  streaming WriteCheckpointFile that walks the tree with sorted keys
- Reduce results channel buffer from len(hostEntries) to 2 for
  back-pressure (at most NumWorkers+2 results in flight)
- Workers send CheckpointFile trees instead of []ParquetMetricRow
- Write rows in small 1024-element batches via reusable buffer

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: f31dc1847539

2026-03-18 17:32:16 +01:00

..

archive

Reduce insert pressure in db. Increase sqlite timeout value

2026-03-16 11:17:47 +01:00

metricstore

fix: reduce memory usage in parquet checkpoint archiver

2026-03-18 17:32:16 +01:00