cc-backend

mirror of https://github.com/ClusterCockpit/cc-backend synced 2026-03-21 15:27:30 +01:00

Files

Jan Eitzinger 09501df3c2 fix: reduce memory usage in parquet checkpoint archiver

Stream CheckpointFile trees directly to parquet rows instead of
materializing all rows in a giant intermediate slice. This eliminates
~1.9GB per host of redundant allocations (repeated string headers)
and removes the expensive sort on millions of 104-byte structs.

Key changes:
- Replace flattenCheckpointFile + sortParquetRows + WriteHostRows with
  streaming WriteCheckpointFile that walks the tree with sorted keys
- Reduce results channel buffer from len(hostEntries) to 2 for
  back-pressure (at most NumWorkers+2 results in flight)
- Workers send CheckpointFile trees instead of []ParquetMetricRow
- Write rows in small 1024-element batches via reusable buffer

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: f31dc1847539

2026-03-18 17:32:16 +01:00

api.go

Convert Warn message on missing metrics to Debug level

2026-03-16 15:35:24 +01:00

archive.go

fix: reduce memory usage in parquet checkpoint archiver

2026-03-18 17:32:16 +01:00

buffer.go

fix: Fix metricstore memory explosion from broken emergency free and batch aborts

2026-03-13 07:57:35 +01:00

checkpoint.go

Make checkpointInterval an option config option again.