mirror of
https://github.com/ClusterCockpit/cc-backend
synced 2026-03-24 08:37:30 +01:00
Jan Eitzinger
688d41d11a
Entire-Session: 0943a044-b17e-4215-8591-c1a0c816ddf0 Entire-Strategy: manual-commit Entire-Agent: Claude Code Ephemeral-branch: entire/3314b8e-e3b0c4
105 lines
3.9 KiB
Plaintext
105 lines
3.9 KiB
Plaintext
Implement the following plan:
|
|
|
|
# Fix: Memory Escalation in flattenCheckpointFile (68GB+)
|
|
|
|
## Context
|
|
|
|
Production gops shows `flattenCheckpointFile` allocating 68GB+ (74.89% of memory). The archiving pipeline accumulates ALL metric data from ALL hosts into a single `[]ParquetMetricRow` slice before writing to Parquet. For large HPC clusters this is catastrophic. Additionally, the `SortingWriterConfig` in the parquet writer buffers everything again internally.
|
|
|
|
## Root Cause
|
|
|
|
Two-layer unbounded accumulation:
|
|
|
|
1. **`archive.go:239-242`**: `allRows = append(allRows, r.rows...)` merges every host's rows into one giant slice
|
|
2. **`parquetArchive.go:108-116`**: `SortingWriterConfig` creates a sorting writer that buffers ALL rows until `Close()`
|
|
3. **`parquetArchive.go:199`**: `var rows []ParquetMetricRow` starts at zero capacity, grows via append doubling
|
|
|
|
Peak memory = (all hosts' rows) + (sorting writer copy) + (append overhead) = ~3x raw data size.
|
|
|
|
## Fix: Stream per-host to parquet writer
|
|
|
|
Instead of accumulating all rows, write each host's data as a separate row group.
|
|
|
|
### Step 1: Add streaming parquet writer (`parquetArchive.go`)
|
|
|
|
Replace `writeParquetArchive(filename, rows)` with a struct that supports incremental writes:
|
|
|
|
```go
|
|
type parquetArchiveWriter struct {
|
|
writer *pq.GenericWriter[ParquetMetricRow]
|
|
bw *bufio.Writer
|
|
f *os.File
|
|
count int
|
|
}
|
|
|
|
func newParquetArchiveWriter(filename string) (*parquetArchiveWriter, error)
|
|
func (w *parquetArchiveWriter) WriteHostRows(rows []ParquetMetricRow) error // Write + Flush (creates row group)
|
|
func (w *parquetArchiveWriter) Close() error
|
|
```
|
|
|
|
- **Remove `SortingWriterConfig`** - no global sort buffer
|
|
- Sort each host's rows in-place with `sort.Slice` before writing (cheap: single host data)
|
|
- Each `Flush()` creates a separate row group per host
|
|
|
|
### Step 2: Add row count estimation (`parquetArchive.go`)
|
|
|
|
```go
|
|
func estimateRowCount(cf *CheckpointFile) int
|
|
```
|
|
|
|
Pre-allocate `rows` slice in `archiveCheckpointsToParquet` to avoid append doubling per host.
|
|
|
|
### Step 3: Restructure `archiveCheckpoints` (`archive.go`)
|
|
|
|
Change from:
|
|
```
|
|
workers → channel → accumulate allRows → writeParquetArchive(allRows)
|
|
```
|
|
|
|
To:
|
|
```
|
|
open writer → workers → channel → for each host: sort rows, writer.WriteHostRows(rows) → close writer
|
|
```
|
|
|
|
- Only one host's rows in memory at a time
|
|
- Track `files`/`dir` for deletion separately (don't retain rows)
|
|
- Check `writer.count > 0` instead of `len(allRows) == 0`
|
|
|
|
### Step 4: Update test (`parquetArchive_test.go`)
|
|
|
|
- `TestParquetArchiveRoundtrip`: use new streaming writer API
|
|
- Keep `archiveCheckpointsToParquet` returning rows (it's per-host, manageable size)
|
|
|
|
## Files to Modify
|
|
|
|
- **`pkg/metricstore/parquetArchive.go`**: Add `parquetArchiveWriter`, `estimateRowCount`; remove `writeParquetArchive`; add `"sort"` import
|
|
- **`pkg/metricstore/archive.go`**: Restructure `archiveCheckpoints` to stream
|
|
- **`pkg/metricstore/parquetArchive_test.go`**: Update roundtrip test
|
|
|
|
## Memory Impact
|
|
|
|
- **Before**: All hosts in memory (~40GB for 256 nodes) + sorting buffer (~40GB) = 68GB+
|
|
- **After**: One host at a time (~16MB) + parquet page buffer (~1MB) = ~17MB peak
|
|
|
|
## Sorting Tradeoff
|
|
|
|
The output changes from one globally-sorted row group to N row groups (one per host), each internally sorted by (metric, timestamp). This is actually better for ClusterCockpit's per-host query patterns (enables row group skipping).
|
|
|
|
## Verification
|
|
|
|
```bash
|
|
go test -v ./pkg/metricstore/...
|
|
```
|
|
|
|
Also verify with `go vet ./pkg/metricstore/...` for correctness.
|
|
|
|
|
|
If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-CC-cc-backend/71340843-de3d-4e83-9dcb-2fc130c50e0d.jsonl
|
|
|
|
---
|
|
|
|
Are the any other cases with memory spikes using the Parquet Writer, e.g. in the nodestate retention?
|
|
|
|
---
|
|
|
|
[Request interrupted by user for tool use] |