=== RUN   TestParseScopeFromName
=== RUN   TestParseScopeFromName/socket0
=== RUN   TestParseScopeFromName/socket12
=== RUN   TestParseScopeFromName/core0
=== RUN   TestParseScopeFromName/core127
=== RUN   TestParseScopeFromName/cpu0
=== RUN   TestParseScopeFromName/hwthread5
=== RUN   TestParseScopeFromName/memoryDomain0
=== RUN   TestParseScopeFromName/accelerator0
=== RUN   TestParseScopeFromName/unknown
=== RUN   TestParseScopeFromName/socketX
--- PASS: TestParseScopeFromName (0.00s)
    --- PASS: TestParseScopeFromName/socket0 (0.00s)
    --- PASS: TestParseScopeFromName/socket12 (0.00s)
    --- PASS: TestParseScopeFromName/core0 (0.00s)
    --- PASS: TestParseScopeFromName/core127 (0.00s)
    --- PASS: TestParseScopeFromName/cpu0 (0.00s)
    --- PASS: TestParseScopeFromName/hwthread5 (0.00s)
    --- PASS: TestParseScopeFromName/memoryDomain0 (0.00s)
    --- PASS: TestParseScopeFromName/accelerator0 (0.00s)
    --- PASS: TestParseScopeFromName/unknown (0.00s)
    --- PASS: TestParseScopeFromName/socketX (0.00s)
=== RUN   TestWriteCheckpointFile
--- PASS: TestWriteCheckpointFile (0.00s)
=== RUN   TestParquetArchiveRoundtrip
    parquetArchive_test.go:258: Parquet file size: 2064 bytes for 9 rows
--- PASS: TestParquetArchiveRoundtrip (0.00s)
=== RUN   TestLoadCheckpointFileFromDisk_JSON
--- PASS: TestLoadCheckpointFileFromDisk_JSON (0.00s)
PASS
ok  	github.com/ClusterCockpit/cc-backend/pkg/metricstore	0.492s


All 4 tests pass. Let me also run the full test suite to make sure nothing else broke:


Bash

go test ./pkg/metricstore/... -v 2>&1 | tail -30

Run all metricstore tests


=== RUN   TestBuildScopeQueries/MemoryDomain->Socket
=== RUN   TestBuildScopeQueries/Accelerator->Node
=== RUN   TestBuildScopeQueries/Accelerator->Core_(exception)
--- PASS: TestBuildScopeQueries (0.00s)
    --- PASS: TestBuildScopeQueries/HWThread->HWThread (0.00s)
    --- PASS: TestBuildScopeQueries/Core->Core (0.00s)
    --- PASS: TestBuildScopeQueries/Socket->Socket (0.00s)
    --- PASS: TestBuildScopeQueries/MemoryDomain->MemoryDomain (0.00s)
    --- PASS: TestBuildScopeQueries/Node->Node (0.00s)
    --- PASS: TestBuildScopeQueries/Accelerator->Accelerator (0.00s)
    --- PASS: TestBuildScopeQueries/HWThread->Core (0.00s)
    --- PASS: TestBuildScopeQueries/HWThread->Socket (0.00s)
    --- PASS: TestBuildScopeQueries/HWThread->Node (0.00s)
    --- PASS: TestBuildScopeQueries/Core->Socket (0.00s)
    --- PASS: TestBuildScopeQueries/Core->Node (0.00s)
    --- PASS: TestBuildScopeQueries/Socket->Node (0.00s)
    --- PASS: TestBuildScopeQueries/MemoryDomain->Node (0.00s)
    --- PASS: TestBuildScopeQueries/MemoryDomain->Socket (0.00s)
    --- PASS: TestBuildScopeQueries/Accelerator->Node (0.00s)
    --- PASS: TestBuildScopeQueries/Accelerator->Core_(exception) (0.00s)
=== RUN   TestBuildScopeQueries_UnhandledCase
--- PASS: TestBuildScopeQueries_UnhandledCase (0.00s)
=== RUN   TestIntToStringSlice
--- PASS: TestIntToStringSlice (0.00s)
=== RUN   TestSanitizeStats
--- PASS: TestSanitizeStats (0.00s)
=== RUN   TestNodeToNodeQuery
--- PASS: TestNodeToNodeQuery (0.00s)
PASS
ok  	github.com/ClusterCockpit/cc-backend/pkg/metricstore	0.329s


All tests pass. Here's a summary of the changes:

### Changes made

**`pkg/metricstore/parquetArchive.go`**:
- **Removed** `flattenCheckpointFile` — no longer materializes all rows into a giant slice
- **Removed** `sortParquetRows` — sorting is now implicit (metrics iterated in sorted key order)
- **Removed** `estimateRowCount` — no pre-allocation needed
- **Removed** `archiveCheckpointsToParquet` — replaced by `loadCheckpointFiles` (returns `[]*CheckpointFile` instead of `[]ParquetMetricRow`)
- **Removed** `WriteHostRows` — replaced by streaming `WriteCheckpointFile`
- **Added** `WriteCheckpointFile` — walks the `CheckpointFile` tree with sorted metric/child keys, writing rows in small batches (1024) directly to the parquet writer. No intermediate `[]ParquetMetricRow` slice for the whole host.
- **Added** `writeLevel` — recursive helper that iterates metrics in sorted key order (producing already-sorted output) and writes in batches
- **Added** reusable `batch []ParquetMetricRow` field on the writer (1024 capacity)

**`pkg/metricstore/archive.go`**:
- **Results channel buffer**: `len(hostEntries)` → `2` — provides back-pressure so at most `NumWorkers+2` results are in flight
- **Workers now send `[]*CheckpointFile`** instead of `[]ParquetMetricRow` — the `CheckpointFile` tree is much smaller than the flattened rows (stores `[]schema.Float` arrays, not per-row structs with 5 repeated strings)
- **Main thread streams** each `CheckpointFile` directly to parquet via `WriteCheckpointFile`, then the `CheckpointFile` becomes eligible for GC

**`pkg/metricstore/parquetArchive_test.go`**:
- Updated `TestFlattenCheckpointFile` → `TestWriteCheckpointFile` to use the new streaming API
- Updated `TestParquetArchiveRoundtrip` to use `loadCheckpointFiles` + `WriteCheckpointFile`


Commit it


Show working tree status


On branch hotfix
Your branch is up to date with 'origin/hotfix'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	modified:   pkg/metricstore/archive.go
	modified:   pkg/metricstore/parquetArchive.go
	modified:   pkg/metricstore/parquetArchive_test.go


Show staged and unstaged changes


(Bash completed with no output)


Show recent commits


bb691577 fix: clarify title
8b0881fb Exclude down nodes from HealthCheck
33beb3c8 fix: simplify stats query condition
c1d51959 Change dtermineState to enforce priority order
3328d2ca Update go version in CLAUDE.md


Changes are already staged. Let me see the diff of staged changes:


Show staged changes summary


 pkg/metricstore/archive.go             |  40 +++++---
 pkg/metricstore/parquetArchive.go      | 176 +++++++++++++++------------------
 pkg/metricstore/parquetArchive_test.go |  59 +++++++++--
 3 files changed, 155 insertions(+), 120 deletions(-)


git commit -m "$(cat <<'EOF'
fix: reduce memory usage in parquet checkpoint archiver

Stream CheckpointFile trees directly to parquet rows instead of
materializing all rows in a giant intermediate slice. This eliminates
~1.9GB per host of redundant allocations (repeated string headers)
and removes the expensive sort on millions of 104-byte structs.

Key changes:
- Replace flattenCheckpointFile + sortParquetRows + WriteHostRows with
  streaming WriteCheckpointFile that walks the tree with sorted keys
- Reduce results channel buffer from len(hostEntries) to 2 for
  back-pressure (at most NumWorkers+2 results in flight)
- Workers send CheckpointFile trees instead of []ParquetMetricRow
- Write rows in small 1024-element batches via reusable buffer

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
EOF
)"

Commit the parquet archiver optimization


[hotfix 09501df3] fix: reduce memory usage in parquet checkpoint archiver
 3 files changed, 155 insertions(+), 120 deletions(-)


Verify clean working tree


On branch hotfix
Your branch is ahead of 'origin/hotfix' by 1 commit.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean


Committed as `09501df3` on `hotfix`.