Implement the following plan:

# Fix: Memory Escalation in flattenCheckpointFile (68GB+)

## Context

Production gops shows `flattenCheckpointFile` allocating 68GB+ (74.89% of memory). The archiving pipeline accumulates ALL metric data from ALL hosts into a single `[]ParquetMetricRow` slice before writing to Parquet. For large HPC clusters this is catastrophic. Additionally, the `SortingWriterConfig` in the parquet writer buffers everything again internally.

## Root Cause

Two-layer unbounded accumulation:

1. **`archive.go:239-242`**: `allRows = append(allRows, r.rows...)` merges every host's rows into one giant slice
2. **`parquetArchive.go:108-116`**: `SortingWriterConfig` creates a sorting writer that buffers ALL rows until `Close()`
3. **`parquetArchive.go:199`**: `var rows []ParquetMetricRow` starts at zero capacity, grows via append doubling

Peak memory = (all hosts' rows) + (sorting writer copy) + (append overhead) = ~3x raw data size.

## Fix: Stream per-host to parquet writer

Instead of accumulating all rows, write each host's data as a separate row group.

### Step 1: Add streaming parquet writer (`parquetArchive.go`)

Replace `writeParquetArchive(filename, rows)` with a struct that supports incremental writes:

```go
type parquetArchiveWriter struct {
    writer *pq.GenericWriter[ParquetMetricRow]
    bw     *bufio.Writer
    f      *os.File
    count  int
}

func newParquetArchiveWriter(filename string) (*parquetArchiveWriter, error)
func (w *parquetArchiveWriter) WriteHostRows(rows []ParquetMetricRow) error  // Write + Flush (creates row group)
func (w *parquetArchiveWriter) Close() error
```

- **Remove `SortingWriterConfig`** - no global sort buffer
- Sort each host's rows in-place with `sort.Slice` before writing (cheap: single host data)
- Each `Flush()` creates a separate row group per host

### Step 2: Add row count estimation (`parquetArchive.go`)

```go
func estimateRowCount(cf *CheckpointFile) int
```

Pre-allocate `rows` slice in `archiveCheckpointsToParquet` to avoid append doubling per host.

### Step 3: Restructure `archiveCheckpoints` (`archive.go`)

Change from:
```
workers → channel → accumulate allRows → writeParquetArchive(allRows)
```

To:
```
open writer → workers → channel → for each host: sort rows, writer.WriteHostRows(rows) → close writer
```

- Only one host's rows in memory at a time
- Track `files`/`dir` for deletion separately (don't retain rows)
- Check `writer.count > 0` instead of `len(allRows) == 0`

### Step 4: Update test (`parquetArchive_test.go`)

- `TestParquetArchiveRoundtrip`: use new streaming writer API
- Keep `archiveCheckpointsToParquet` returning rows (it's per-host, manageable size)

## Files to Modify

- **`pkg/metricstore/parquetArchive.go`**: Add `parquetArchiveWriter`, `estimateRowCount`; remove `writeParquetArchive`; add `"sort"` import
- **`pkg/metricstore/archive.go`**: Restructure `archiveCheckpoints` to stream
- **`pkg/metricstore/parquetArchive_test.go`**: Update roundtrip test

## Memory Impact

- **Before**: All hosts in memory (~40GB for 256 nodes) + sorting buffer (~40GB) = 68GB+
- **After**: One host at a time (~16MB) + parquet page buffer (~1MB) = ~17MB peak

## Sorting Tradeoff

The output changes from one globally-sorted row group to N row groups (one per host), each internally sorted by (metric, timestamp). This is actually better for ClusterCockpit's per-host query patterns (enables row group skipping).

## Verification

```bash
go test -v ./pkg/metricstore/...
```

Also verify with `go vet ./pkg/metricstore/...` for correctness.

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-CC-cc-backend/71340843-de3d-4e83-9dcb-2fc130c50e0d.jsonl


Let me start by reading the relevant files to understand the current implementation.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package metricstore
     7→
     8→import (
     9→	"bufio"
    10→	"encoding/binary"
    11→	"encoding/json"
    12→	"fmt"
    13→	"os"
    14→	"path/filepath"
    15→
    16→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    17→	pq "github.com/parquet-go/parquet-go"
    18→)
    19→
    20→// ParquetMetricRow is the long-format schema for archived metric data.
    21→// One row per (host, metric, scope, scope_id, timestamp) data point.
    22→// Sorted by (cluster, hostname, metric, timestamp) for optimal compression.
    23→type ParquetMetricRow struct {
    24→	Cluster   string  `parquet:"cluster"`
    25→	Hostname  string  `parquet:"hostname"`
    26→	Metric    string  `parquet:"metric"`
    27→	Scope     string  `parquet:"scope"`
    28→	ScopeID   string  `parquet:"scope_id"`
    29→	Timestamp int64   `parquet:"timestamp"`
    30→	Frequency int64   `parquet:"frequency"`
    31→	Value     float32 `parquet:"value"`
    32→}
    33→
    34→// flattenCheckpointFile recursively converts a CheckpointFile tree into Parquet rows.
    35→// The scope path is built from the hierarchy: host level is "node", then child names
    36→// map to scope/scope_id (e.g., "socket0" → scope="socket", scope_id="0").
    37→func flattenCheckpointFile(cf *CheckpointFile, cluster, hostname, scope, scopeID string, rows []ParquetMetricRow) []ParquetMetricRow {
    38→	for metricName, cm := range cf.Metrics {
    39→		ts := cm.Start
    40→		for _, v := range cm.Data {
    41→			if !v.IsNaN() {
    42→				rows = append(rows, ParquetMetricRow{
    43→					Cluster:   cluster,
    44→					Hostname:  hostname,
    45→					Metric:    metricName,
    46→					Scope:     scope,
    47→					ScopeID:   scopeID,
    48→					Timestamp: ts,
    49→					Frequency: cm.Frequency,
    50→					Value:     float32(v),
    51→				})
    52→			}
    53→			ts += cm.Frequency
    54→		}
    55→	}
    56→
    57→	for childName, childCf := range cf.Children {
    58→		childScope, childScopeID := parseScopeFromName(childName)
    59→		rows = flattenCheckpointFile(childCf, cluster, hostname, childScope, childScopeID, rows)
    60→	}
    61→
    62→	return rows
    63→}
    64→
    65→// parseScopeFromName infers scope and scope_id from a child level name.
    66→// Examples: "socket0" → ("socket", "0"), "core12" → ("core", "12"),
    67→// "a0" (accelerator) → ("accelerator", "0").
    68→// If the name doesn't match known patterns, it's used as-is for scope with empty scope_id.
    69→func parseScopeFromName(name string) (string, string) {
    70→	prefixes := []struct {
    71→		prefix string
    72→		scope  string
    73→	}{
    74→		{"socket", "socket"},
    75→		{"memoryDomain", "memoryDomain"},
    76→		{"core", "core"},
    77→		{"hwthread", "hwthread"},
    78→		{"cpu", "hwthread"},
    79→		{"accelerator", "accelerator"},
    80→	}
    81→
    82→	for _, p := range prefixes {
    83→		if len(name) > len(p.prefix) && name[:len(p.prefix)] == p.prefix {
    84→			id := name[len(p.prefix):]
    85→			if len(id) > 0 && id[0] >= '0' && id[0] <= '9' {
    86→				return p.scope, id
    87→			}
    88→		}
    89→	}
    90→
    91→	return name, ""
    92→}
    93→
    94→// writeParquetArchive writes rows to a Parquet file with Zstd compression.
    95→func writeParquetArchive(filename string, rows []ParquetMetricRow) error {
    96→	if err := os.MkdirAll(filepath.Dir(filename), CheckpointDirPerms); err != nil {
    97→		return fmt.Errorf("creating archive directory: %w", err)
    98→	}
    99→
   100→	f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
   101→	if err != nil {
   102→		return fmt.Errorf("creating parquet file: %w", err)
   103→	}
   104→	defer f.Close()
   105→
   106→	bw := bufio.NewWriterSize(f, 1<<20) // 1MB write buffer
   107→
   108→	writer := pq.NewGenericWriter[ParquetMetricRow](bw,
   109→		pq.Compression(&pq.Zstd),
   110→		pq.SortingWriterConfig(pq.SortingColumns(
   111→			pq.Ascending("cluster"),
   112→			pq.Ascending("hostname"),
   113→			pq.Ascending("metric"),
   114→			pq.Ascending("timestamp"),
   115→		)),
   116→	)
   117→
   118→	if _, err := writer.Write(rows); err != nil {
   119→		return fmt.Errorf("writing parquet rows: %w", err)
   120→	}
   121→
   122→	if err := writer.Close(); err != nil {
   123→		return fmt.Errorf("closing parquet writer: %w", err)
   124→	}
   125→
   126→	if err := bw.Flush(); err != nil {
   127→		return fmt.Errorf("flushing parquet file: %w", err)
   128→	}
   129→
   130→	return nil
   131→}
   132→
   133→// loadCheckpointFileFromDisk reads a JSON or binary checkpoint file and returns
   134→// a CheckpointFile. Used by the Parquet archiver to read checkpoint data
   135→// before converting it to Parquet format.
   136→func loadCheckpointFileFromDisk(filename string) (*CheckpointFile, error) {
   137→	f, err := os.Open(filename)
   138→	if err != nil {
   139→		return nil, err
   140→	}
   141→	defer f.Close()
   142→
   143→	ext := filepath.Ext(filename)
   144→	switch ext {
   145→	case ".json":
   146→		cf := &CheckpointFile{}
   147→		br := bufio.NewReader(f)
   148→		if err := json.NewDecoder(br).Decode(cf); err != nil {
   149→			return nil, fmt.Errorf("decoding JSON checkpoint %s: %w", filename, err)
   150→		}
   151→		return cf, nil
   152→
   153→	case ".bin":
   154→		br := bufio.NewReader(f)
   155→		var magic uint32
   156→		if err := binary.Read(br, binary.LittleEndian, &magic); err != nil {
   157→			return nil, fmt.Errorf("reading magic from %s: %w", filename, err)
   158→		}
   159→		if magic != snapFileMagic {
   160→			return nil, fmt.Errorf("invalid snapshot magic in %s: 0x%08X", filename, magic)
   161→		}
   162→		var fileFrom, fileTo int64
   163→		if err := binary.Read(br, binary.LittleEndian, &fileFrom); err != nil {
   164→			return nil, fmt.Errorf("reading from-timestamp from %s: %w", filename, err)
   165→		}
   166→		if err := binary.Read(br, binary.LittleEndian, &fileTo); err != nil {
   167→			return nil, fmt.Errorf("reading to-timestamp from %s: %w", filename, err)
   168→		}
   169→		cf, err := readBinaryLevel(br)
   170→		if err != nil {
   171→			return nil, fmt.Errorf("reading binary level from %s: %w", filename, err)
   172→		}
   173→		cf.From = fileFrom
   174→		cf.To = fileTo
   175→		return cf, nil
   176→
   177→	default:
   178→		return nil, fmt.Errorf("unsupported checkpoint extension: %s", ext)
   179→	}
   180→}
   181→
   182→// archiveCheckpointsToParquet reads checkpoint files for a host directory,
   183→// converts them to Parquet rows. Returns the rows and filenames that were processed.
   184→func archiveCheckpointsToParquet(dir, cluster, host string, from int64) ([]ParquetMetricRow, []string, error) {
   185→	entries, err := os.ReadDir(dir)
   186→	if err != nil {
   187→		return nil, nil, err
   188→	}
   189→
   190→	files, err := findFiles(entries, from, false)
   191→	if err != nil {
   192→		return nil, nil, err
   193→	}
   194→
   195→	if len(files) == 0 {
   196→		return nil, nil, nil
   197→	}
   198→
   199→	var rows []ParquetMetricRow
   200→
   201→	for _, checkpoint := range files {
   202→		filename := filepath.Join(dir, checkpoint)
   203→		cf, err := loadCheckpointFileFromDisk(filename)
   204→		if err != nil {
   205→			cclog.Warnf("[METRICSTORE]> skipping unreadable checkpoint %s: %v", filename, err)
   206→			continue
   207→		}
   208→
   209→		rows = flattenCheckpointFile(cf, cluster, host, "node", "", rows)
   210→	}
   211→
   212→	return rows, files, nil
   213→}
   214→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package metricstore

import (
	"bufio"
	"encoding/binary"
	"encoding/json"
	"fmt"
	"os"
	"path/filepath"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	pq "github.com/parquet-go/parquet-go"
)

// ParquetMetricRow is the long-format schema for archived metric data.
// One row per (host, metric, scope, scope_id, timestamp) data point.
// Sorted by (cluster, hostname, metric, timestamp) for optimal compression.
type ParquetMetricRow struct {
	Cluster   string  `parquet:"cluster"`
	Hostname  string  `parquet:"hostname"`
	Metric    string  `parquet:"metric"`
	Scope     string  `parquet:"scope"`
	ScopeID   string  `parquet:"scope_id"`
	Timestamp int64   `parquet:"timestamp"`
	Frequency int64   `parquet:"frequency"`
	Value     float32 `parquet:"value"`
}

// flattenCheckpointFile recursively converts a CheckpointFile tree into Parquet rows.
// The scope path is built from the hierarchy: host level is "node", then child names
// map to scope/scope_id (e.g., "socket0" → scope="socket", scope_id="0").
func flattenCheckpointFile(cf *CheckpointFile, cluster, hostname, scope, scopeID string, rows []ParquetMetricRow) []ParquetMetricRow {
	for metricName, cm := range cf.Metrics {
		ts := cm.Start
		for _, v := range cm.Data {
			if !v.IsNaN() {
				rows = append(rows, ParquetMetricRow{
					Cluster:   cluster,
					Hostname:  hostname,
					Metric:    metricName,
					Scope:     scope,
					ScopeID:   scopeID,
					Timestamp: ts,
					Frequency: cm.Frequency,
					Value:     float32(v),
				})
			}
			ts += cm.Frequency
		}
	}

	for childName, childCf := range cf.Children {
		childScope, childScopeID := parseScopeFromName(childName)
		rows = flattenCheckpointFile(childCf, cluster, hostname, childScope, childScopeID, rows)
	}

	return rows
}

// parseScopeFromName infers scope and scope_id from a child level name.
// Examples: "socket0" → ("socket", "0"), "core12" → ("core", "12"),
// "a0" (accelerator) → ("accelerator", "0").
// If the name doesn't match known patterns, it's used as-is for scope with empty scope_id.
func parseScopeFromName(name string) (string, string) {
	prefixes := []struct {
		prefix string
		scope  string
	}{
		{"socket", "socket"},
		{"memoryDomain", "memoryDomain"},
		{"core", "core"},
		{"hwthread", "hwthread"},
		{"cpu", "hwthread"},
		{"accelerator", "accelerator"},
	}

	for _, p := range prefixes {
		if len(name) > len(p.prefix) && name[:len(p.prefix)] == p.prefix {
			id := name[len(p.prefix):]
			if len(id) > 0 && id[0] >= '0' && id[0] <= '9' {
				return p.scope, id
			}
		}
	}

	return name, ""
}

// writeParquetArchive writes rows to a Parquet file with Zstd compression.
func writeParquetArchive(filename string, rows []ParquetMetricRow) error {
	if err := os.MkdirAll(filepath.Dir(filename), CheckpointDirPerms); err != nil {
		return fmt.Errorf("creating archive directory: %w", err)
	}

	f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil {
		return fmt.Errorf("creating parquet file: %w", err)
	}
	defer f.Close()

	bw := bufio.NewWriterSize(f, 1<<20) // 1MB write buffer

	writer := pq.NewGenericWriter[ParquetMetricRow](bw,
		pq.Compression(&pq.Zstd),
		pq.SortingWriterConfig(pq.SortingColumns(
			pq.Ascending("cluster"),
			pq.Ascending("hostname"),
			pq.Ascending("metric"),
			pq.Ascending("timestamp"),
		)),
	)

	if _, err := writer.Write(rows); err != nil {
		return fmt.Errorf("writing parquet rows: %w", err)
	}

	if err := writer.Close(); err != nil {
		return fmt.Errorf("closing parquet writer: %w", err)
	}

	if err := bw.Flush(); err != nil {
		return fmt.Errorf("flushing parquet file: %w", err)
	}

	return nil
}

// loadCheckpointFileFromDisk reads a JSON or binary checkpoint file and returns
// a CheckpointFile. Used by the Parquet archiver to read checkpoint data
// before converting it to Parquet format.
func loadCheckpointFileFromDisk(filename string) (*CheckpointFile, error) {
	f, err := os.Open(filename)
	if err != nil {
		return nil, err
	}
	defer f.Close()

	ext := filepath.Ext(filename)
	switch ext {
	case ".json":
		cf := &CheckpointFile{}
		br := bufio.NewReader(f)
		if err := json.NewDecoder(br).Decode(cf); err != nil {
			return nil, fmt.Errorf("decoding JSON checkpoint %s: %w", filename, err)
		}
		return cf, nil

	case ".bin":
		br := bufio.NewReader(f)
		var magic uint32
		if err := binary.Read(br, binary.LittleEndian, &magic); err != nil {
			return nil, fmt.Errorf("reading magic from %s: %w", filename, err)
		}
		if magic != snapFileMagic {
			return nil, fmt.Errorf("invalid snapshot magic in %s: 0x%08X", filename, magic)
		}
		var fileFrom, fileTo int64
		if err := binary.Read(br, binary.LittleEndian, &fileFrom); err != nil {
			return nil, fmt.Errorf("reading from-timestamp from %s: %w", filename, err)
		}
		if err := binary.Read(br, binary.LittleEndian, &fileTo); err != nil {
			return nil, fmt.Errorf("reading to-timestamp from %s: %w", filename, err)
		}
		cf, err := readBinaryLevel(br)
		if err != nil {
			return nil, fmt.Errorf("reading binary level from %s: %w", filename, err)
		}
		cf.From = fileFrom
		cf.To = fileTo
		return cf, nil

	default:
		return nil, fmt.Errorf("unsupported checkpoint extension: %s", ext)
	}
}

// archiveCheckpointsToParquet reads checkpoint files for a host directory,
// converts them to Parquet rows. Returns the rows and filenames that were processed.
func archiveCheckpointsToParquet(dir, cluster, host string, from int64) ([]ParquetMetricRow, []string, error) {
	entries, err := os.ReadDir(dir)
	if err != nil {
		return nil, nil, err
	}

	files, err := findFiles(entries, from, false)
	if err != nil {
		return nil, nil, err
	}

	if len(files) == 0 {
		return nil, nil, nil
	}

	var rows []ParquetMetricRow

	for _, checkpoint := range files {
		filename := filepath.Join(dir, checkpoint)
		cf, err := loadCheckpointFileFromDisk(filename)
		if err != nil {
			cclog.Warnf("[METRICSTORE]> skipping unreadable checkpoint %s: %v", filename, err)
			continue
		}

		rows = flattenCheckpointFile(cf, cluster, host, "node", "", rows)
	}

	return rows, files, nil
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package metricstore
     7→
     8→import (
     9→	"context"
    10→	"errors"
    11→	"fmt"
    12→	"os"
    13→	"path/filepath"
    14→	"sync"
    15→	"sync/atomic"
    16→	"time"
    17→
    18→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    19→)
    20→
    21→// Worker for either Archiving or Deleting files
    22→
    23→func CleanUp(wg *sync.WaitGroup, ctx context.Context) {
    24→	if Keys.Cleanup.Mode == "archive" {
    25→		// Run as Archiver
    26→		cleanUpWorker(wg, ctx,
    27→			Keys.RetentionInMemory,
    28→			"archiving",
    29→			Keys.Cleanup.RootDir,
    30→			false,
    31→		)
    32→	} else {
    33→		// Run as Deleter
    34→		cleanUpWorker(wg, ctx,
    35→			Keys.RetentionInMemory,
    36→			"deleting",
    37→			"",
    38→			true,
    39→		)
    40→	}
    41→}
    42→
    43→// cleanUpWorker takes simple values to configure what it does
    44→func cleanUpWorker(wg *sync.WaitGroup, ctx context.Context, interval string, mode string, cleanupDir string, delete bool) {
    45→	wg.Go(func() {
    46→
    47→		d, err := time.ParseDuration(interval)
    48→		if err != nil {
    49→			cclog.Fatalf("[METRICSTORE]> error parsing %s interval duration: %v\n", mode, err)
    50→		}
    51→		if d <= 0 {
    52→			return
    53→		}
    54→
    55→		ticker := time.NewTicker(d)
    56→		defer ticker.Stop()
    57→
    58→		for {
    59→			select {
    60→			case <-ctx.Done():
    61→				return
    62→			case <-ticker.C:
    63→				t := time.Now().Add(-d)
    64→				cclog.Infof("[METRICSTORE]> start %s checkpoints (older than %s)...", mode, t.Format(time.RFC3339))
    65→
    66→				n, err := CleanupCheckpoints(Keys.Checkpoints.RootDir, cleanupDir, t.Unix(), delete)
    67→
    68→				if err != nil {
    69→					cclog.Errorf("[METRICSTORE]> %s failed: %s", mode, err.Error())
    70→				} else {
    71→					if delete {
    72→						cclog.Infof("[METRICSTORE]> done: %d checkpoints deleted", n)
    73→					} else {
    74→						cclog.Infof("[METRICSTORE]> done: %d checkpoint files archived to parquet", n)
    75→					}
    76→				}
    77→			}
    78→		}
    79→	})
    80→}
    81→
    82→var ErrNoNewArchiveData error = errors.New("all data already archived")
    83→
    84→// CleanupCheckpoints deletes or archives all checkpoint files older than `from`.
    85→// When archiving, consolidates all hosts per cluster into a single Parquet file.
    86→func CleanupCheckpoints(checkpointsDir, cleanupDir string, from int64, deleteInstead bool) (int, error) {
    87→	if deleteInstead {
    88→		return deleteCheckpoints(checkpointsDir, from)
    89→	}
    90→
    91→	return archiveCheckpoints(checkpointsDir, cleanupDir, from)
    92→}
    93→
    94→// deleteCheckpoints removes checkpoint files older than `from` across all clusters/hosts.
    95→func deleteCheckpoints(checkpointsDir string, from int64) (int, error) {
    96→	entries1, err := os.ReadDir(checkpointsDir)
    97→	if err != nil {
    98→		return 0, err
    99→	}
   100→
   101→	type workItem struct {
   102→		dir            string
   103→		cluster, host  string
   104→	}
   105→
   106→	var wg sync.WaitGroup
   107→	n, errs := int32(0), int32(0)
   108→	work := make(chan workItem, Keys.NumWorkers)
   109→
   110→	wg.Add(Keys.NumWorkers)
   111→	for worker := 0; worker < Keys.NumWorkers; worker++ {
   112→		go func() {
   113→			defer wg.Done()
   114→			for item := range work {
   115→				entries, err := os.ReadDir(item.dir)
   116→				if err != nil {
   117→					cclog.Errorf("error reading %s/%s: %s", item.cluster, item.host, err.Error())
   118→					atomic.AddInt32(&errs, 1)
   119→					continue
   120→				}
   121→
   122→				files, err := findFiles(entries, from, false)
   123→				if err != nil {
   124→					cclog.Errorf("error finding files in %s/%s: %s", item.cluster, item.host, err.Error())
   125→					atomic.AddInt32(&errs, 1)
   126→					continue
   127→				}
   128→
   129→				for _, checkpoint := range files {
   130→					if err := os.Remove(filepath.Join(item.dir, checkpoint)); err != nil {
   131→						cclog.Errorf("error deleting %s/%s/%s: %s", item.cluster, item.host, checkpoint, err.Error())
   132→						atomic.AddInt32(&errs, 1)
   133→					} else {
   134→						atomic.AddInt32(&n, 1)
   135→					}
   136→				}
   137→			}
   138→		}()
   139→	}
   140→
   141→	for _, de1 := range entries1 {
   142→		entries2, e := os.ReadDir(filepath.Join(checkpointsDir, de1.Name()))
   143→		if e != nil {
   144→			err = e
   145→			continue
   146→		}
   147→
   148→		for _, de2 := range entries2 {
   149→			work <- workItem{
   150→				dir:     filepath.Join(checkpointsDir, de1.Name(), de2.Name()),
   151→				cluster: de1.Name(),
   152→				host:    de2.Name(),
   153→			}
   154→		}
   155→	}
   156→
   157→	close(work)
   158→	wg.Wait()
   159→
   160→	if err != nil {
   161→		return int(n), err
   162→	}
   163→	if errs > 0 {
   164→		return int(n), fmt.Errorf("%d errors happened while deleting (%d successes)", errs, n)
   165→	}
   166→	return int(n), nil
   167→}
   168→
   169→// archiveCheckpoints archives checkpoint files to Parquet format.
   170→// Produces one Parquet file per cluster: <cleanupDir>/<cluster>/<timestamp>.parquet
   171→func archiveCheckpoints(checkpointsDir, cleanupDir string, from int64) (int, error) {
   172→	clusterEntries, err := os.ReadDir(checkpointsDir)
   173→	if err != nil {
   174→		return 0, err
   175→	}
   176→
   177→	totalFiles := 0
   178→
   179→	for _, clusterEntry := range clusterEntries {
   180→		if !clusterEntry.IsDir() {
   181→			continue
   182→		}
   183→
   184→		cluster := clusterEntry.Name()
   185→		hostEntries, err := os.ReadDir(filepath.Join(checkpointsDir, cluster))
   186→		if err != nil {
   187→			return totalFiles, err
   188→		}
   189→
   190→		// Collect rows from all hosts in this cluster using worker pool
   191→		type hostResult struct {
   192→			rows  []ParquetMetricRow
   193→			files []string // checkpoint filenames to delete after successful write
   194→			dir   string   // checkpoint directory for this host
   195→		}
   196→
   197→		results := make(chan hostResult, len(hostEntries))
   198→		work := make(chan struct {
   199→			dir, host string
   200→		}, Keys.NumWorkers)
   201→
   202→		var wg sync.WaitGroup
   203→		errs := int32(0)
   204→
   205→		wg.Add(Keys.NumWorkers)
   206→		for w := 0; w < Keys.NumWorkers; w++ {
   207→			go func() {
   208→				defer wg.Done()
   209→				for item := range work {
   210→					rows, files, err := archiveCheckpointsToParquet(item.dir, cluster, item.host, from)
   211→					if err != nil {
   212→						cclog.Errorf("[METRICSTORE]> error reading checkpoints for %s/%s: %s", cluster, item.host, err.Error())
   213→						atomic.AddInt32(&errs, 1)
   214→						continue
   215→					}
   216→					if len(rows) > 0 {
   217→						results <- hostResult{rows: rows, files: files, dir: item.dir}
   218→					}
   219→				}
   220→			}()
   221→		}
   222→
   223→		go func() {
   224→			for _, hostEntry := range hostEntries {
   225→				if !hostEntry.IsDir() {
   226→					continue
   227→				}
   228→				dir := filepath.Join(checkpointsDir, cluster, hostEntry.Name())
   229→				work <- struct {
   230→					dir, host string
   231→				}{dir: dir, host: hostEntry.Name()}
   232→			}
   233→			close(work)
   234→			wg.Wait()
   235→			close(results)
   236→		}()
   237→
   238→		// Collect all rows and file info
   239→		var allRows []ParquetMetricRow
   240→		var allResults []hostResult
   241→		for r := range results {
   242→			allRows = append(allRows, r.rows...)
   243→			allResults = append(allResults, r)
   244→		}
   245→
   246→		if errs > 0 {
   247→			return totalFiles, fmt.Errorf("%d errors reading checkpoints for cluster %s", errs, cluster)
   248→		}
   249→
   250→		if len(allRows) == 0 {
   251→			continue
   252→		}
   253→
   254→		// Write one Parquet file per cluster
   255→		parquetFile := filepath.Join(cleanupDir, cluster, fmt.Sprintf("%d.parquet", from))
   256→		if err := writeParquetArchive(parquetFile, allRows); err != nil {
   257→			return totalFiles, fmt.Errorf("writing parquet archive for cluster %s: %w", cluster, err)
   258→		}
   259→
   260→		// Delete archived checkpoint files
   261→		for _, result := range allResults {
   262→			for _, file := range result.files {
   263→				filename := filepath.Join(result.dir, file)
   264→				if err := os.Remove(filename); err != nil {
   265→					cclog.Warnf("[METRICSTORE]> could not remove archived checkpoint %s: %v", filename, err)
   266→				} else {
   267→					totalFiles++
   268→				}
   269→			}
   270→		}
   271→
   272→		cclog.Infof("[METRICSTORE]> archived %d rows from %d files for cluster %s to %s",
   273→			len(allRows), totalFiles, cluster, parquetFile)
   274→	}
   275→
   276→	return totalFiles, nil
   277→}
   278→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package metricstore

import (
	"context"
	"errors"
	"fmt"
	"os"
	"path/filepath"
	"sync"
	"sync/atomic"
	"time"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
)

// Worker for either Archiving or Deleting files

func CleanUp(wg *sync.WaitGroup, ctx context.Context) {
	if Keys.Cleanup.Mode == "archive" {
		// Run as Archiver
		cleanUpWorker(wg, ctx,
			Keys.RetentionInMemory,
			"archiving",
			Keys.Cleanup.RootDir,
			false,
		)
	} else {
		// Run as Deleter
		cleanUpWorker(wg, ctx,
			Keys.RetentionInMemory,
			"deleting",
			"",
			true,
		)
	}
}

// cleanUpWorker takes simple values to configure what it does
func cleanUpWorker(wg *sync.WaitGroup, ctx context.Context, interval string, mode string, cleanupDir string, delete bool) {
	wg.Go(func() {

		d, err := time.ParseDuration(interval)
		if err != nil {
			cclog.Fatalf("[METRICSTORE]> error parsing %s interval duration: %v\n", mode, err)
		}
		if d <= 0 {
			return
		}

		ticker := time.NewTicker(d)
		defer ticker.Stop()

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				t := time.Now().Add(-d)
				cclog.Infof("[METRICSTORE]> start %s checkpoints (older than %s)...", mode, t.Format(time.RFC3339))

				n, err := CleanupCheckpoints(Keys.Checkpoints.RootDir, cleanupDir, t.Unix(), delete)

				if err != nil {
					cclog.Errorf("[METRICSTORE]> %s failed: %s", mode, err.Error())
				} else {
					if delete {
						cclog.Infof("[METRICSTORE]> done: %d checkpoints deleted", n)
					} else {
						cclog.Infof("[METRICSTORE]> done: %d checkpoint files archived to parquet", n)
					}
				}
			}
		}
	})
}

var ErrNoNewArchiveData error = errors.New("all data already archived")

// CleanupCheckpoints deletes or archives all checkpoint files older than `from`.
// When archiving, consolidates all hosts per cluster into a single Parquet file.
func CleanupCheckpoints(checkpointsDir, cleanupDir string, from int64, deleteInstead bool) (int, error) {
	if deleteInstead {
		return deleteCheckpoints(checkpointsDir, from)
	}

	return archiveCheckpoints(checkpointsDir, cleanupDir, from)
}

// deleteCheckpoints removes checkpoint files older than `from` across all clusters/hosts.
func deleteCheckpoints(checkpointsDir string, from int64) (int, error) {
	entries1, err := os.ReadDir(checkpointsDir)
	if err != nil {
		return 0, err
	}

	type workItem struct {
		dir            string
		cluster, host  string
	}

	var wg sync.WaitGroup
	n, errs := int32(0), int32(0)
	work := make(chan workItem, Keys.NumWorkers)

	wg.Add(Keys.NumWorkers)
	for worker := 0; worker < Keys.NumWorkers; worker++ {
		go func() {
			defer wg.Done()
			for item := range work {
				entries, err := os.ReadDir(item.dir)
				if err != nil {
					cclog.Errorf("error reading %s/%s: %s", item.cluster, item.host, err.Error())
					atomic.AddInt32(&errs, 1)
					continue
				}

				files, err := findFiles(entries, from, false)
				if err != nil {
					cclog.Errorf("error finding files in %s/%s: %s", item.cluster, item.host, err.Error())
					atomic.AddInt32(&errs, 1)
					continue
				}

				for _, checkpoint := range files {
					if err := os.Remove(filepath.Join(item.dir, checkpoint)); err != nil {
						cclog.Errorf("error deleting %s/%s/%s: %s", item.cluster, item.host, checkpoint, err.Error())
						atomic.AddInt32(&errs, 1)
					} else {
						atomic.AddInt32(&n, 1)
					}
				}
			}
		}()
	}

	for _, de1 := range entries1 {
		entries2, e := os.ReadDir(filepath.Join(checkpointsDir, de1.Name()))
		if e != nil {
			err = e
			continue
		}

		for _, de2 := range entries2 {
			work <- workItem{
				dir:     filepath.Join(checkpointsDir, de1.Name(), de2.Name()),
				cluster: de1.Name(),
				host:    de2.Name(),
			}
		}
	}

	close(work)
	wg.Wait()

	if err != nil {
		return int(n), err
	}
	if errs > 0 {
		return int(n), fmt.Errorf("%d errors happened while deleting (%d successes)", errs, n)
	}
	return int(n), nil
}

// archiveCheckpoints archives checkpoint files to Parquet format.
// Produces one Parquet file per cluster: <cleanupDir>/<cluster>/<timestamp>.parquet
func archiveCheckpoints(checkpointsDir, cleanupDir string, from int64) (int, error) {
	clusterEntries, err := os.ReadDir(checkpointsDir)
	if err != nil {
		return 0, err
	}

	totalFiles := 0

	for _, clusterEntry := range clusterEntries {
		if !clusterEntry.IsDir() {
			continue
		}

		cluster := clusterEntry.Name()
		hostEntries, err := os.ReadDir(filepath.Join(checkpointsDir, cluster))
		if err != nil {
			return totalFiles, err
		}

		// Collect rows from all hosts in this cluster using worker pool
		type hostResult struct {
			rows  []ParquetMetricRow
			files []string // checkpoint filenames to delete after successful write
			dir   string   // checkpoint directory for this host
		}

		results := make(chan hostResult, len(hostEntries))
		work := make(chan struct {
			dir, host string
		}, Keys.NumWorkers)

		var wg sync.WaitGroup
		errs := int32(0)

		wg.Add(Keys.NumWorkers)
		for w := 0; w < Keys.NumWorkers; w++ {
			go func() {
				defer wg.Done()
				for item := range work {
					rows, files, err := archiveCheckpointsToParquet(item.dir, cluster, item.host, from)
					if err != nil {
						cclog.Errorf("[METRICSTORE]> error reading checkpoints for %s/%s: %s", cluster, item.host, err.Error())
						atomic.AddInt32(&errs, 1)
						continue
					}
					if len(rows) > 0 {
						results <- hostResult{rows: rows, files: files, dir: item.dir}
					}
				}
			}()
		}

		go func() {
			for _, hostEntry := range hostEntries {
				if !hostEntry.IsDir() {
					continue
				}
				dir := filepath.Join(checkpointsDir, cluster, hostEntry.Name())
				work <- struct {
					dir, host string
				}{dir: dir, host: hostEntry.Name()}
			}
			close(work)
			wg.Wait()
			close(results)
		}()

		// Collect all rows and file info
		var allRows []ParquetMetricRow
		var allResults []hostResult
		for r := range results {
			allRows = append(allRows, r.rows...)
			allResults = append(allResults, r)
		}

		if errs > 0 {
			return totalFiles, fmt.Errorf("%d errors reading checkpoints for cluster %s", errs, cluster)
		}

		if len(allRows) == 0 {
			continue
		}

		// Write one Parquet file per cluster
		parquetFile := filepath.Join(cleanupDir, cluster, fmt.Sprintf("%d.parquet", from))
		if err := writeParquetArchive(parquetFile, allRows); err != nil {
			return totalFiles, fmt.Errorf("writing parquet archive for cluster %s: %w", cluster, err)
		}

		// Delete archived checkpoint files
		for _, result := range allResults {
			for _, file := range result.files {
				filename := filepath.Join(result.dir, file)
				if err := os.Remove(filename); err != nil {
					cclog.Warnf("[METRICSTORE]> could not remove archived checkpoint %s: %v", filename, err)
				} else {
					totalFiles++
				}
			}
		}

		cclog.Infof("[METRICSTORE]> archived %d rows from %d files for cluster %s to %s",
			len(allRows), totalFiles, cluster, parquetFile)
	}

	return totalFiles, nil
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package metricstore
     7→
     8→import (
     9→	"encoding/json"
    10→	"os"
    11→	"path/filepath"
    12→	"testing"
    13→
    14→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    15→	pq "github.com/parquet-go/parquet-go"
    16→)
    17→
    18→func TestParseScopeFromName(t *testing.T) {
    19→	tests := []struct {
    20→		name      string
    21→		wantScope string
    22→		wantID    string
    23→	}{
    24→		{"socket0", "socket", "0"},
    25→		{"socket12", "socket", "12"},
    26→		{"core0", "core", "0"},
    27→		{"core127", "core", "127"},
    28→		{"cpu0", "hwthread", "0"},
    29→		{"hwthread5", "hwthread", "5"},
    30→		{"memoryDomain0", "memoryDomain", "0"},
    31→		{"accelerator0", "accelerator", "0"},
    32→		{"unknown", "unknown", ""},
    33→		{"socketX", "socketX", ""}, // not numeric suffix
    34→	}
    35→
    36→	for _, tt := range tests {
    37→		t.Run(tt.name, func(t *testing.T) {
    38→			scope, id := parseScopeFromName(tt.name)
    39→			if scope != tt.wantScope || id != tt.wantID {
    40→				t.Errorf("parseScopeFromName(%q) = (%q, %q), want (%q, %q)",
    41→					tt.name, scope, id, tt.wantScope, tt.wantID)
    42→			}
    43→		})
    44→	}
    45→}
    46→
    47→func TestFlattenCheckpointFile(t *testing.T) {
    48→	cf := &CheckpointFile{
    49→		From: 1000,
    50→		To:   1060,
    51→		Metrics: map[string]*CheckpointMetrics{
    52→			"cpu_load": {
    53→				Frequency: 60,
    54→				Start:     1000,
    55→				Data:      []schema.Float{0.5, 0.7, schema.NaN},
    56→			},
    57→		},
    58→		Children: map[string]*CheckpointFile{
    59→			"socket0": {
    60→				Metrics: map[string]*CheckpointMetrics{
    61→					"mem_bw": {
    62→						Frequency: 60,
    63→						Start:     1000,
    64→						Data:      []schema.Float{100.0, schema.NaN, 200.0},
    65→					},
    66→				},
    67→				Children: make(map[string]*CheckpointFile),
    68→			},
    69→		},
    70→	}
    71→
    72→	rows := flattenCheckpointFile(cf, "fritz", "node001", "node", "", nil)
    73→
    74→	// cpu_load: 2 non-NaN values at node scope
    75→	// mem_bw: 2 non-NaN values at socket0 scope
    76→	if len(rows) != 4 {
    77→		t.Fatalf("expected 4 rows, got %d", len(rows))
    78→	}
    79→
    80→	// Verify a node-scope row
    81→	found := false
    82→	for _, r := range rows {
    83→		if r.Metric == "cpu_load" && r.Timestamp == 1000 {
    84→			found = true
    85→			if r.Cluster != "fritz" || r.Hostname != "node001" || r.Scope != "node" || r.Value != 0.5 {
    86→				t.Errorf("unexpected row: %+v", r)
    87→			}
    88→		}
    89→	}
    90→	if !found {
    91→		t.Error("expected cpu_load row at timestamp 1000")
    92→	}
    93→
    94→	// Verify a socket-scope row
    95→	found = false
    96→	for _, r := range rows {
    97→		if r.Metric == "mem_bw" && r.Scope == "socket" && r.ScopeID == "0" {
    98→			found = true
    99→		}
   100→	}
   101→	if !found {
   102→		t.Error("expected mem_bw row with scope=socket, scope_id=0")
   103→	}
   104→}
   105→
   106→func TestParquetArchiveRoundtrip(t *testing.T) {
   107→	tmpDir := t.TempDir()
   108→
   109→	// Create checkpoint files on disk (JSON format)
   110→	cpDir := filepath.Join(tmpDir, "checkpoints", "testcluster", "node001")
   111→	if err := os.MkdirAll(cpDir, 0o755); err != nil {
   112→		t.Fatal(err)
   113→	}
   114→
   115→	cf := &CheckpointFile{
   116→		From: 1000,
   117→		To:   1180,
   118→		Metrics: map[string]*CheckpointMetrics{
   119→			"cpu_load": {
   120→				Frequency: 60,
   121→				Start:     1000,
   122→				Data:      []schema.Float{0.5, 0.7, 0.9},
   123→			},
   124→			"mem_used": {
   125→				Frequency: 60,
   126→				Start:     1000,
   127→				Data:      []schema.Float{45.0, 46.0, 47.0},
   128→			},
   129→		},
   130→		Children: map[string]*CheckpointFile{
   131→			"socket0": {
   132→				Metrics: map[string]*CheckpointMetrics{
   133→					"mem_bw": {
   134→						Frequency: 60,
   135→						Start:     1000,
   136→						Data:      []schema.Float{100.0, 110.0, 120.0},
   137→					},
   138→				},
   139→				Children: make(map[string]*CheckpointFile),
   140→			},
   141→		},
   142→	}
   143→
   144→	// Write JSON checkpoint
   145→	cpFile := filepath.Join(cpDir, "1000.json")
   146→	data, err := json.Marshal(cf)
   147→	if err != nil {
   148→		t.Fatal(err)
   149→	}
   150→	if err := os.WriteFile(cpFile, data, 0o644); err != nil {
   151→		t.Fatal(err)
   152→	}
   153→
   154→	// Archive to Parquet
   155→	archiveDir := filepath.Join(tmpDir, "archive")
   156→	rows, files, err := archiveCheckpointsToParquet(cpDir, "testcluster", "node001", 2000)
   157→	if err != nil {
   158→		t.Fatal(err)
   159→	}
   160→	if len(files) != 1 || files[0] != "1000.json" {
   161→		t.Fatalf("expected 1 file, got %v", files)
   162→	}
   163→
   164→	parquetFile := filepath.Join(archiveDir, "testcluster", "1000.parquet")
   165→	if err := writeParquetArchive(parquetFile, rows); err != nil {
   166→		t.Fatal(err)
   167→	}
   168→
   169→	// Read back and verify
   170→	f, err := os.Open(parquetFile)
   171→	if err != nil {
   172→		t.Fatal(err)
   173→	}
   174→	defer f.Close()
   175→
   176→	stat, _ := f.Stat()
   177→	pf, err := pq.OpenFile(f, stat.Size())
   178→	if err != nil {
   179→		t.Fatal(err)
   180→	}
   181→
   182→	reader := pq.NewGenericReader[ParquetMetricRow](pf)
   183→	readRows := make([]ParquetMetricRow, 100)
   184→	n, err := reader.Read(readRows)
   185→	if err != nil && n == 0 {
   186→		t.Fatal(err)
   187→	}
   188→	readRows = readRows[:n]
   189→	reader.Close()
   190→
   191→	// We expect: cpu_load(3) + mem_used(3) + mem_bw(3) = 9 rows
   192→	if n != 9 {
   193→		t.Fatalf("expected 9 rows in parquet file, got %d", n)
   194→	}
   195→
   196→	// Verify cluster and hostname are set correctly
   197→	for _, r := range readRows {
   198→		if r.Cluster != "testcluster" {
   199→			t.Errorf("expected cluster=testcluster, got %s", r.Cluster)
   200→		}
   201→		if r.Hostname != "node001" {
   202→			t.Errorf("expected hostname=node001, got %s", r.Hostname)
   203→		}
   204→	}
   205→
   206→	// Verify parquet file is smaller than JSON (compression working)
   207→	if stat.Size() == 0 {
   208→		t.Error("parquet file is empty")
   209→	}
   210→
   211→	t.Logf("Parquet file size: %d bytes for %d rows", stat.Size(), n)
   212→}
   213→
   214→func TestLoadCheckpointFileFromDisk_JSON(t *testing.T) {
   215→	tmpDir := t.TempDir()
   216→
   217→	cf := &CheckpointFile{
   218→		From: 1000,
   219→		To:   1060,
   220→		Metrics: map[string]*CheckpointMetrics{
   221→			"test_metric": {
   222→				Frequency: 60,
   223→				Start:     1000,
   224→				Data:      []schema.Float{1.0, 2.0, 3.0},
   225→			},
   226→		},
   227→		Children: make(map[string]*CheckpointFile),
   228→	}
   229→
   230→	filename := filepath.Join(tmpDir, "1000.json")
   231→	data, err := json.Marshal(cf)
   232→	if err != nil {
   233→		t.Fatal(err)
   234→	}
   235→	if err := os.WriteFile(filename, data, 0o644); err != nil {
   236→		t.Fatal(err)
   237→	}
   238→
   239→	loaded, err := loadCheckpointFileFromDisk(filename)
   240→	if err != nil {
   241→		t.Fatal(err)
   242→	}
   243→
   244→	if loaded.From != 1000 || loaded.To != 1060 {
   245→		t.Errorf("expected From=1000, To=1060, got From=%d, To=%d", loaded.From, loaded.To)
   246→	}
   247→
   248→	m, ok := loaded.Metrics["test_metric"]
   249→	if !ok {
   250→		t.Fatal("expected test_metric in loaded checkpoint")
   251→	}
   252→	if m.Frequency != 60 || m.Start != 1000 || len(m.Data) != 3 {
   253→		t.Errorf("unexpected metric data: %+v", m)
   254→	}
   255→}
   256→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package metricstore

import (
	"encoding/json"
	"os"
	"path/filepath"
	"testing"

	"github.com/ClusterCockpit/cc-lib/v2/schema"
	pq "github.com/parquet-go/parquet-go"
)

func TestParseScopeFromName(t *testing.T) {
	tests := []struct {
		name      string
		wantScope string
		wantID    string
	}{
		{"socket0", "socket", "0"},
		{"socket12", "socket", "12"},
		{"core0", "core", "0"},
		{"core127", "core", "127"},
		{"cpu0", "hwthread", "0"},
		{"hwthread5", "hwthread", "5"},
		{"memoryDomain0", "memoryDomain", "0"},
		{"accelerator0", "accelerator", "0"},
		{"unknown", "unknown", ""},
		{"socketX", "socketX", ""}, // not numeric suffix
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			scope, id := parseScopeFromName(tt.name)
			if scope != tt.wantScope || id != tt.wantID {
				t.Errorf("parseScopeFromName(%q) = (%q, %q), want (%q, %q)",
					tt.name, scope, id, tt.wantScope, tt.wantID)
			}
		})
	}
}

func TestFlattenCheckpointFile(t *testing.T) {
	cf := &CheckpointFile{
		From: 1000,
		To:   1060,
		Metrics: map[string]*CheckpointMetrics{
			"cpu_load": {
				Frequency: 60,
				Start:     1000,
				Data:      []schema.Float{0.5, 0.7, schema.NaN},
			},
		},
		Children: map[string]*CheckpointFile{
			"socket0": {
				Metrics: map[string]*CheckpointMetrics{
					"mem_bw": {
						Frequency: 60,
						Start:     1000,
						Data:      []schema.Float{100.0, schema.NaN, 200.0},
					},
				},
				Children: make(map[string]*CheckpointFile),
			},
		},
	}

	rows := flattenCheckpointFile(cf, "fritz", "node001", "node", "", nil)

	// cpu_load: 2 non-NaN values at node scope
	// mem_bw: 2 non-NaN values at socket0 scope
	if len(rows) != 4 {
		t.Fatalf("expected 4 rows, got %d", len(rows))
	}

	// Verify a node-scope row
	found := false
	for _, r := range rows {
		if r.Metric == "cpu_load" && r.Timestamp == 1000 {
			found = true
			if r.Cluster != "fritz" || r.Hostname != "node001" || r.Scope != "node" || r.Value != 0.5 {
				t.Errorf("unexpected row: %+v", r)
			}
		}
	}
	if !found {
		t.Error("expected cpu_load row at timestamp 1000")
	}

	// Verify a socket-scope row
	found = false
	for _, r := range rows {
		if r.Metric == "mem_bw" && r.Scope == "socket" && r.ScopeID == "0" {
			found = true
		}
	}
	if !found {
		t.Error("expected mem_bw row with scope=socket, scope_id=0")
	}
}

func TestParquetArchiveRoundtrip(t *testing.T) {
	tmpDir := t.TempDir()

	// Create checkpoint files on disk (JSON format)
	cpDir := filepath.Join(tmpDir, "checkpoints", "testcluster", "node001")
	if err := os.MkdirAll(cpDir, 0o755); err != nil {
		t.Fatal(err)
	}

	cf := &CheckpointFile{
		From: 1000,
		To:   1180,
		Metrics: map[string]*CheckpointMetrics{
			"cpu_load": {
				Frequency: 60,
				Start:     1000,
				Data:      []schema.Float{0.5, 0.7, 0.9},
			},
			"mem_used": {
				Frequency: 60,
				Start:     1000,
				Data:      []schema.Float{45.0, 46.0, 47.0},
			},
		},
		Children: map[string]*CheckpointFile{
			"socket0": {
				Metrics: map[string]*CheckpointMetrics{
					"mem_bw": {
						Frequency: 60,
						Start:     1000,
						Data:      []schema.Float{100.0, 110.0, 120.0},
					},
				},
				Children: make(map[string]*CheckpointFile),
			},
		},
	}

	// Write JSON checkpoint
	cpFile := filepath.Join(cpDir, "1000.json")
	data, err := json.Marshal(cf)
	if err != nil {
		t.Fatal(err)
	}
	if err := os.WriteFile(cpFile, data, 0o644); err != nil {
		t.Fatal(err)
	}

	// Archive to Parquet
	archiveDir := filepath.Join(tmpDir, "archive")
	rows, files, err := archiveCheckpointsToParquet(cpDir, "testcluster", "node001", 2000)
	if err != nil {
		t.Fatal(err)
	}
	if len(files) != 1 || files[0] != "1000.json" {
		t.Fatalf("expected 1 file, got %v", files)
	}

	parquetFile := filepath.Join(archiveDir, "testcluster", "1000.parquet")
	if err := writeParquetArchive(parquetFile, rows); err != nil {
		t.Fatal(err)
	}

	// Read back and verify
	f, err := os.Open(parquetFile)
	if err != nil {
		t.Fatal(err)
	}
	defer f.Close()

	stat, _ := f.Stat()
	pf, err := pq.OpenFile(f, stat.Size())
	if err != nil {
		t.Fatal(err)
	}

	reader := pq.NewGenericReader[ParquetMetricRow](pf)
	readRows := make([]ParquetMetricRow, 100)
	n, err := reader.Read(readRows)
	if err != nil && n == 0 {
		t.Fatal(err)
	}
	readRows = readRows[:n]
	reader.Close()

	// We expect: cpu_load(3) + mem_used(3) + mem_bw(3) = 9 rows
	if n != 9 {
		t.Fatalf("expected 9 rows in parquet file, got %d", n)
	}

	// Verify cluster and hostname are set correctly
	for _, r := range readRows {
		if r.Cluster != "testcluster" {
			t.Errorf("expected cluster=testcluster, got %s", r.Cluster)
		}
		if r.Hostname != "node001" {
			t.Errorf("expected hostname=node001, got %s", r.Hostname)
		}
	}

	// Verify parquet file is smaller than JSON (compression working)
	if stat.Size() == 0 {
		t.Error("parquet file is empty")
	}

	t.Logf("Parquet file size: %d bytes for %d rows", stat.Size(), n)
}

func TestLoadCheckpointFileFromDisk_JSON(t *testing.T) {
	tmpDir := t.TempDir()

	cf := &CheckpointFile{
		From: 1000,
		To:   1060,
		Metrics: map[string]*CheckpointMetrics{
			"test_metric": {
				Frequency: 60,
				Start:     1000,
				Data:      []schema.Float{1.0, 2.0, 3.0},
			},
		},
		Children: make(map[string]*CheckpointFile),
	}

	filename := filepath.Join(tmpDir, "1000.json")
	data, err := json.Marshal(cf)
	if err != nil {
		t.Fatal(err)
	}
	if err := os.WriteFile(filename, data, 0o644); err != nil {
		t.Fatal(err)
	}

	loaded, err := loadCheckpointFileFromDisk(filename)
	if err != nil {
		t.Fatal(err)
	}

	if loaded.From != 1000 || loaded.To != 1060 {
		t.Errorf("expected From=1000, To=1060, got From=%d, To=%d", loaded.From, loaded.To)
	}

	m, ok := loaded.Metrics["test_metric"]
	if !ok {
		t.Fatal("expected test_metric in loaded checkpoint")
	}
	if m.Frequency != 60 || m.Start != 1000 || len(m.Data) != 3 {
		t.Errorf("unexpected metric data: %+v", m)
	}
}


Now I have a clear picture. Let me implement the changes.

**Step 1: Update `parquetArchive.go`** - Add streaming writer, remove old `writeParquetArchive`


Edit

import (
	"bufio"
	"encoding/binary"
	"encoding/json"
	"fmt"
	"os"
	"path/filepath"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	pq "github.com/parquet-go/parquet-go"
)

import (
	"bufio"
	"encoding/binary"
	"encoding/json"
	"fmt"
	"os"
	"path/filepath"
	"sort"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	pq "github.com/parquet-go/parquet-go"
)


The file /Users/jan/prg/CC/cc-backend/pkg/metricstore/parquetArchive.go has been updated successfully.

import (
	"bufio"
	"encoding/binary"
	"encoding/json"
	"fmt"
	"os"
	"path/filepath"
	"sort"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	pq "github.com/parquet-go/parquet-go"
)

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package metricstore

import (
	"bufio"
	"encoding/binary"
	"encoding/json"
	"fmt"
	"os"
	"path/filepath"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	pq "github.com/parquet-go/parquet-go"
)

// ParquetMetricRow is the long-format schema for archived metric data.
// One row per (host, metric, scope, scope_id, timestamp) data point.
// Sorted by (cluster, hostname, metric, timestamp) for optimal compression.
type ParquetMetricRow struct {
	Cluster   string  `parquet:"cluster"`
	Hostname  string  `parquet:"hostname"`
	Metric    string  `parquet:"metric"`
	Scope     string  `parquet:"scope"`
	ScopeID   string  `parquet:"scope_id"`
	Timestamp int64   `parquet:"timestamp"`
	Frequency int64   `parquet:"frequency"`
	Value     float32 `parquet:"value"`
}

// flattenCheckpointFile recursively converts a CheckpointFile tree into Parquet rows.
// The scope path is built from the hierarchy: host level is "node", then child names
// map to scope/scope_id (e.g., "socket0" → scope="socket", scope_id="0").
func flattenCheckpointFile(cf *CheckpointFile, cluster, hostname, scope, scopeID string, rows []ParquetMetricRow) []ParquetMetricRow {
	for metricName, cm := range cf.Metrics {
		ts := cm.Start
		for _, v := range cm.Data {
			if !v.IsNaN() {
				rows = append(rows, ParquetMetricRow{
					Cluster:   cluster,
					Hostname:  hostname,
					Metric:    metricName,
					Scope:     scope,
					ScopeID:   scopeID,
					Timestamp: ts,
					Frequency: cm.Frequency,
					Value:     float32(v),
				})
			}
			ts += cm.Frequency
		}
	}

	for childName, childCf := range cf.Children {
		childScope, childScopeID := parseScopeFromName(childName)
		rows = flattenCheckpointFile(childCf, cluster, hostname, childScope, childScopeID, rows)
	}

	return rows
}

// parseScopeFromName infers scope and scope_id from a child level name.
// Examples: "socket0" → ("socket", "0"), "core12" → ("core", "12"),
// "a0" (accelerator) → ("accelerator", "0").
// If the name doesn't match known patterns, it's used as-is for scope with empty scope_id.
func parseScopeFromName(name string) (string, string) {
	prefixes := []struct {
		prefix string
		scope  string
	}{
		{"socket", "socket"},
		{"memoryDomain", "memoryDomain"},
		{"core", "core"},
		{"hwthread", "hwthread"},
		{"cpu", "hwthread"},
		{"accelerator", "accelerator"},
	}

	for _, p := range prefixes {
		if len(name) > len(p.prefix) && name[:len(p.prefix)] == p.prefix {
			id := name[len(p.prefix):]
			if len(id) > 0 && id[0] >= '0' && id[0] <= '9' {
				return p.scope, id
			}
		}
	}

	return name, ""
}

// writeParquetArchive writes rows to a Parquet file with Zstd compression.
func writeParquetArchive(filename string, rows []ParquetMetricRow) error {
	if err := os.MkdirAll(filepath.Dir(filename), CheckpointDirPerms); err != nil {
		return fmt.Errorf("creating archive directory: %w", err)
	}

	f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil {
		return fmt.Errorf("creating parquet file: %w", err)
	}
	defer f.Close()

	bw := bufio.NewWriterSize(f, 1<<20) // 1MB write buffer

	writer := pq.NewGenericWriter[ParquetMetricRow](bw,
		pq.Compression(&pq.Zstd),
		pq.SortingWriterConfig(pq.SortingColumns(
			pq.Ascending("cluster"),
			pq.Ascending("hostname"),
			pq.Ascending("metric"),
			pq.Ascending("timestamp"),
		)),
	)

	if _, err := writer.Write(rows); err != nil {
		return fmt.Errorf("writing parquet rows: %w", err)
	}

	if err := writer.Close(); err != nil {
		return fmt.Errorf("closing parquet writer: %w", err)
	}

	if err := bw.Flush(); err != nil {
		return fmt.Errorf("flushing parquet file: %w", err)
	}

	return nil
}

// loadCheckpointFileFromDisk reads a JSON or binary checkpoint file and returns
// a CheckpointFile. Used by the Parquet archiver to read checkpoint data
// before converting it to Parquet format.
func loadCheckpointFileFromDisk(filename string) (*CheckpointFile, error) {
	f, err := os.Open(filename)
	if err != nil {
		return nil, err
	}
	defer f.Close()

	ext := filepath.Ext(filename)
	switch ext {
	case ".json":
		cf := &CheckpointFile{}
		br := bufio.NewReader(f)
		if err := json.NewDecoder(br).Decode(cf); err != nil {
			return nil, fmt.Errorf("decoding JSON checkpoint %s: %w", filename, err)
		}
		return cf, nil

	case ".bin":
		br := bufio.NewReader(f)
		var magic uint32
		if err := binary.Read(br, binary.LittleEndian, &magic); err != nil {
			return nil, fmt.Errorf("reading magic from %s: %w", filename, err)
		}
		if magic != snapFileMagic {
			return nil, fmt.Errorf("invalid snapshot magic in %s: 0x%08X", filename, magic)
		}
		var fileFrom, fileTo int64
		if err := binary.Read(br, binary.LittleEndian, &fileFrom); err != nil {
			return nil, fmt.Errorf("reading from-timestamp from %s: %w", filename, err)
		}
		if err := binary.Read(br, binary.LittleEndian, &fileTo); err != nil {
			return nil, fmt.Errorf("reading to-timestamp from %s: %w", filename, err)
		}
		cf, err := readBinaryLevel(br)
		if err != nil {
			return nil, fmt.Errorf("reading binary level from %s: %w", filename, err)
		}
		cf.From = fileFrom
		cf.To = fileTo
		return cf, nil

	default:
		return nil, fmt.Errorf("unsupported checkpoint extension: %s", ext)
	}
}

// archiveCheckpointsToParquet reads checkpoint files for a host directory,
// converts them to Parquet rows. Returns the rows and filenames that were processed.
func archiveCheckpointsToParquet(dir, cluster, host string, from int64) ([]ParquetMetricRow, []string, error) {
	entries, err := os.ReadDir(dir)
	if err != nil {
		return nil, nil, err
	}

	files, err := findFiles(entries, from, false)
	if err != nil {
		return nil, nil, err
	}

	if len(files) == 0 {
		return nil, nil, nil
	}

	var rows []ParquetMetricRow

	for _, checkpoint := range files {
		filename := filepath.Join(dir, checkpoint)
		cf, err := loadCheckpointFileFromDisk(filename)
		if err != nil {
			cclog.Warnf("[METRICSTORE]> skipping unreadable checkpoint %s: %v", filename, err)
			continue
		}

		rows = flattenCheckpointFile(cf, cluster, host, "node", "", rows)
	}

	return rows, files, nil
}


// writeParquetArchive writes rows to a Parquet file with Zstd compression.
func writeParquetArchive(filename string, rows []ParquetMetricRow) error {
	if err := os.MkdirAll(filepath.Dir(filename), CheckpointDirPerms); err != nil {
		return fmt.Errorf("creating archive directory: %w", err)
	}

	f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil {
		return fmt.Errorf("creating parquet file: %w", err)
	}
	defer f.Close()

	bw := bufio.NewWriterSize(f, 1<<20) // 1MB write buffer

	writer := pq.NewGenericWriter[ParquetMetricRow](bw,
		pq.Compression(&pq.Zstd),
		pq.SortingWriterConfig(pq.SortingColumns(
			pq.Ascending("cluster"),
			pq.Ascending("hostname"),
			pq.Ascending("metric"),
			pq.Ascending("timestamp"),
		)),
	)

	if _, err := writer.Write(rows); err != nil {
		return fmt.Errorf("writing parquet rows: %w", err)
	}

	if err := writer.Close(); err != nil {
		return fmt.Errorf("closing parquet writer: %w", err)
	}

	if err := bw.Flush(); err != nil {
		return fmt.Errorf("flushing parquet file: %w", err)
	}

	return nil
}

// parquetArchiveWriter supports incremental writes to a Parquet file.
// Each call to WriteHostRows writes one row group (typically one host's data),
// avoiding accumulation of all rows in memory.
type parquetArchiveWriter struct {
	writer *pq.GenericWriter[ParquetMetricRow]
	bw     *bufio.Writer
	f      *os.File
	count  int
}

// newParquetArchiveWriter creates a streaming Parquet writer with Zstd compression.
func newParquetArchiveWriter(filename string) (*parquetArchiveWriter, error) {
	if err := os.MkdirAll(filepath.Dir(filename), CheckpointDirPerms); err != nil {
		return nil, fmt.Errorf("creating archive directory: %w", err)
	}

	f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil {
		return nil, fmt.Errorf("creating parquet file: %w", err)
	}

	bw := bufio.NewWriterSize(f, 1<<20) // 1MB write buffer

	writer := pq.NewGenericWriter[ParquetMetricRow](bw,
		pq.Compression(&pq.Zstd),
	)

	return &parquetArchiveWriter{writer: writer, bw: bw, f: f}, nil
}

// WriteHostRows sorts rows by (metric, timestamp) in-place, writes them,
// and flushes to create a separate row group.
func (w *parquetArchiveWriter) WriteHostRows(rows []ParquetMetricRow) error {
	sort.Slice(rows, func(i, j int) bool {
		if rows[i].Metric != rows[j].Metric {
			return rows[i].Metric < rows[j].Metric
		}
		return rows[i].Timestamp < rows[j].Timestamp
	})

	if _, err := w.writer.Write(rows); err != nil {
		return fmt.Errorf("writing parquet rows: %w", err)
	}

	if err := w.writer.Flush(); err != nil {
		return fmt.Errorf("flushing parquet row group: %w", err)
	}

	w.count += len(rows)
	return nil
}

// Close finalises the Parquet file (footer, buffered I/O, file handle).
func (w *parquetArchiveWriter) Close() error {
	if err := w.writer.Close(); err != nil {
		w.f.Close()
		return fmt.Errorf("closing parquet writer: %w", err)
	}

	if err := w.bw.Flush(); err != nil {
		w.f.Close()
		return fmt.Errorf("flushing parquet file: %w", err)
	}

	return w.f.Close()
}

// sortParquetRows sorts rows by (metric, timestamp) in-place.
func sortParquetRows(rows []ParquetMetricRow) {
	sort.Slice(rows, func(i, j int) bool {
		if rows[i].Metric != rows[j].Metric {
			return rows[i].Metric < rows[j].Metric
		}
		return rows[i].Timestamp < rows[j].Timestamp
	})
}


// parquetArchiveWriter supports incremental writes to a Parquet file.
// Each call to WriteHostRows writes one row group (typically one host's data),
// avoiding accumulation of all rows in memory.
type parquetArchiveWriter struct {
	writer *pq.GenericWriter[ParquetMetricRow]
	bw     *bufio.Writer
	f      *os.File
	count  int
}

// newParquetArchiveWriter creates a streaming Parquet writer with Zstd compression.
func newParquetArchiveWriter(filename string) (*parquetArchiveWriter, error) {
	if err := os.MkdirAll(filepath.Dir(filename), CheckpointDirPerms); err != nil {
		return nil, fmt.Errorf("creating archive directory: %w", err)
	}

	f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil {
		return nil, fmt.Errorf("creating parquet file: %w", err)
	}

	bw := bufio.NewWriterSize(f, 1<<20) // 1MB write buffer

	writer := pq.NewGenericWriter[ParquetMetricRow](bw,
		pq.Compression(&pq.Zstd),
	)

	return &parquetArchiveWriter{writer: writer, bw: bw, f: f}, nil
}

// WriteHostRows sorts rows by (metric, timestamp) in-place, writes them,
// and flushes to create a separate row group.
func (w *parquetArchiveWriter) WriteHostRows(rows []ParquetMetricRow) error {
	sort.Slice(rows, func(i, j int) bool {
		if rows[i].Metric != rows[j].Metric {
			return rows[i].Metric < rows[j].Metric
		}
		return rows[i].Timestamp < rows[j].Timestamp
	})

	if _, err := w.writer.Write(rows); err != nil {
		return fmt.Errorf("writing parquet rows: %w", err)
	}

	if err := w.writer.Flush(); err != nil {
		return fmt.Errorf("flushing parquet row group: %w", err)
	}

	w.count += len(rows)
	return nil
}

// Close finalises the Parquet file (footer, buffered I/O, file handle).
func (w *parquetArchiveWriter) Close() error {
	if err := w.writer.Close(); err != nil {
		w.f.Close()
		return fmt.Errorf("closing parquet writer: %w", err)
	}

	if err := w.bw.Flush(); err != nil {
		w.f.Close()
		return fmt.Errorf("flushing parquet file: %w", err)
	}

	return w.f.Close()
}

// sortParquetRows sorts rows by (metric, timestamp) in-place.
func sortParquetRows(rows []ParquetMetricRow) {
	sort.Slice(rows, func(i, j int) bool {
		if rows[i].Metric != rows[j].Metric {
			return rows[i].Metric < rows[j].Metric
		}
		return rows[i].Timestamp < rows[j].Timestamp
	})
}

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package metricstore

import (
	"bufio"
	"encoding/binary"
	"encoding/json"
	"fmt"
	"os"
	"path/filepath"
	"sort"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	pq "github.com/parquet-go/parquet-go"
)

// ParquetMetricRow is the long-format schema for archived metric data.
// One row per (host, metric, scope, scope_id, timestamp) data point.
// Sorted by (cluster, hostname, metric, timestamp) for optimal compression.
type ParquetMetricRow struct {
	Cluster   string  `parquet:"cluster"`
	Hostname  string  `parquet:"hostname"`
	Metric    string  `parquet:"metric"`
	Scope     string  `parquet:"scope"`
	ScopeID   string  `parquet:"scope_id"`
	Timestamp int64   `parquet:"timestamp"`
	Frequency int64   `parquet:"frequency"`
	Value     float32 `parquet:"value"`
}

// flattenCheckpointFile recursively converts a CheckpointFile tree into Parquet rows.
// The scope path is built from the hierarchy: host level is "node", then child names
// map to scope/scope_id (e.g., "socket0" → scope="socket", scope_id="0").
func flattenCheckpointFile(cf *CheckpointFile, cluster, hostname, scope, scopeID string, rows []ParquetMetricRow) []ParquetMetricRow {
	for metricName, cm := range cf.Metrics {
		ts := cm.Start
		for _, v := range cm.Data {
			if !v.IsNaN() {
				rows = append(rows, ParquetMetricRow{
					Cluster:   cluster,
					Hostname:  hostname,
					Metric:    metricName,
					Scope:     scope,
					ScopeID:   scopeID,
					Timestamp: ts,
					Frequency: cm.Frequency,
					Value:     float32(v),
				})
			}
			ts += cm.Frequency
		}
	}

	for childName, childCf := range cf.Children {
		childScope, childScopeID := parseScopeFromName(childName)
		rows = flattenCheckpointFile(childCf, cluster, hostname, childScope, childScopeID, rows)
	}

	return rows
}

// parseScopeFromName infers scope and scope_id from a child level name.
// Examples: "socket0" → ("socket", "0"), "core12" → ("core", "12"),
// "a0" (accelerator) → ("accelerator", "0").
// If the name doesn't match known patterns, it's used as-is for scope with empty scope_id.
func parseScopeFromName(name string) (string, string) {
	prefixes := []struct {
		prefix string
		scope  string
	}{
		{"socket", "socket"},
		{"memoryDomain", "memoryDomain"},
		{"core", "core"},
		{"hwthread", "hwthread"},
		{"cpu", "hwthread"},
		{"accelerator", "accelerator"},
	}

	for _, p := range prefixes {
		if len(name) > len(p.prefix) && name[:len(p.prefix)] == p.prefix {
			id := name[len(p.prefix):]
			if len(id) > 0 && id[0] >= '0' && id[0] <= '9' {
				return p.scope, id
			}
		}
	}

	return name, ""
}

// writeParquetArchive writes rows to a Parquet file with Zstd compression.
func writeParquetArchive(filename string, rows []ParquetMetricRow) error {
	if err := os.MkdirAll(filepath.Dir(filename), CheckpointDirPerms); err != nil {
		return fmt.Errorf("creating archive directory: %w", err)
	}

	f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil {
		return fmt.Errorf("creating parquet file: %w", err)
	}
	defer f.Close()

	bw := bufio.NewWriterSize(f, 1<<20) // 1MB write buffer

	writer := pq.NewGenericWriter[ParquetMetricRow](bw,
		pq.Compression(&pq.Zstd),
		pq.SortingWriterConfig(pq.SortingColumns(
			pq.Ascending("cluster"),
			pq.Ascending("hostname"),
			pq.Ascending("metric"),
			pq.Ascending("timestamp"),
		)),
	)

	if _, err := writer.Write(rows); err != nil {
		return fmt.Errorf("writing parquet rows: %w", err)
	}

	if err := writer.Close(); err != nil {
		return fmt.Errorf("closing parquet writer: %w", err)
	}

	if err := bw.Flush(); err != nil {
		return fmt.Errorf("flushing parquet file: %w", err)
	}

	return nil
}

// loadCheckpointFileFromDisk reads a JSON or binary checkpoint file and returns
// a CheckpointFile. Used by the Parquet archiver to read checkpoint data
// before converting it to Parquet format.
func loadCheckpointFileFromDisk(filename string) (*CheckpointFile, error) {
	f, err := os.Open(filename)
	if err != nil {
		return nil, err
	}
	defer f.Close()

	ext := filepath.Ext(filename)
	switch ext {
	case ".json":
		cf := &CheckpointFile{}
		br := bufio.NewReader(f)
		if err := json.NewDecoder(br).Decode(cf); err != nil {
			return nil, fmt.Errorf("decoding JSON checkpoint %s: %w", filename, err)
		}
		return cf, nil

	case ".bin":
		br := bufio.NewReader(f)
		var magic uint32
		if err := binary.Read(br, binary.LittleEndian, &magic); err != nil {
			return nil, fmt.Errorf("reading magic from %s: %w", filename, err)
		}
		if magic != snapFileMagic {
			return nil, fmt.Errorf("invalid snapshot magic in %s: 0x%08X", filename, magic)
		}
		var fileFrom, fileTo int64
		if err := binary.Read(br, binary.LittleEndian, &fileFrom); err != nil {
			return nil, fmt.Errorf("reading from-timestamp from %s: %w", filename, err)
		}
		if err := binary.Read(br, binary.LittleEndian, &fileTo); err != nil {
			return nil, fmt.Errorf("reading to-timestamp from %s: %w", filename, err)
		}
		cf, err := readBinaryLevel(br)
		if err != nil {
			return nil, fmt.Errorf("reading binary level from %s: %w", filename, err)
		}
		cf.From = fileFrom
		cf.To = fileTo
		return cf, nil

	default:
		return nil, fmt.Errorf("unsupported checkpoint extension: %s", ext)
	}
}

// archiveCheckpointsToParquet reads checkpoint files for a host directory,
// converts them to Parquet rows. Returns the rows and filenames that were processed.
func archiveCheckpointsToParquet(dir, cluster, host string, from int64) ([]ParquetMetricRow, []string, error) {
	entries, err := os.ReadDir(dir)
	if err != nil {
		return nil, nil, err
	}

	files, err := findFiles(entries, from, false)
	if err != nil {
		return nil, nil, err
	}

	if len(files) == 0 {
		return nil, nil, nil
	}

	var rows []ParquetMetricRow

	for _, checkpoint := range files {
		filename := filepath.Join(dir, checkpoint)
		cf, err := loadCheckpointFileFromDisk(filename)
		if err != nil {
			cclog.Warnf("[METRICSTORE]> skipping unreadable checkpoint %s: %v", filename, err)
			continue
		}

		rows = flattenCheckpointFile(cf, cluster, host, "node", "", rows)
	}

	return rows, files, nil
}

-// writeParquetArchive writes rows to a Parquet file with Zstd compression.

-func writeParquetArchive(filename string, rows []ParquetMetricRow) error {

+// parquetArchiveWriter supports incremental writes to a Parquet file.

+// Each call to WriteHostRows writes one row group (typically one host's data),

+// avoiding accumulation of all rows in memory.

+  writer *pq.GenericWriter[ParquetMetricRow]

+// newParquetArchiveWriter creates a streaming Parquet writer with Zstd compression.

+func newParquetArchiveWriter(filename string) (*parquetArchiveWriter, error) {

-    return fmt.Errorf("creating archive directory: %w", err)

+    return nil, fmt.Errorf("creating archive directory: %w", err)

-    return fmt.Errorf("creating parquet file: %w", err)

+    return nil, fmt.Errorf("creating parquet file: %w", err)

-    pq.SortingWriterConfig(pq.SortingColumns(

-  if _, err := writer.Write(rows); err != nil {

+  return &parquetArchiveWriter{writer: writer, bw: bw, f: f}, nil

+// WriteHostRows sorts rows by (metric, timestamp) in-place, writes them,

+// and flushes to create a separate row group.

+func (w *parquetArchiveWriter) WriteHostRows(rows []ParquetMetricRow) error {

+  sort.Slice(rows, func(i, j int) bool {

+    if rows[i].Metric != rows[j].Metric {

+      return rows[i].Metric < rows[j].Metric

+    return rows[i].Timestamp < rows[j].Timestamp

+  if _, err := w.writer.Write(rows); err != nil {

-  if err := writer.Close(); err != nil {

+  if err := w.writer.Flush(); err != nil {

+    return fmt.Errorf("flushing parquet row group: %w", err)

+// Close finalises the Parquet file (footer, buffered I/O, file handle).

+func (w *parquetArchiveWriter) Close() error {

+  if err := w.writer.Close(); err != nil {

+// sortParquetRows sorts rows by (metric, timestamp) in-place.

+func sortParquetRows(rows []ParquetMetricRow) {


Now add `estimateRowCount` and update `archiveCheckpointsToParquet` to pre-allocate:


// archiveCheckpointsToParquet reads checkpoint files for a host directory,
// converts them to Parquet rows. Returns the rows and filenames that were processed.
func archiveCheckpointsToParquet(dir, cluster, host string, from int64) ([]ParquetMetricRow, []string, error) {
	entries, err := os.ReadDir(dir)
	if err != nil {
		return nil, nil, err
	}

	files, err := findFiles(entries, from, false)
	if err != nil {
		return nil, nil, err
	}

	if len(files) == 0 {
		return nil, nil, nil
	}

	var rows []ParquetMetricRow

	for _, checkpoint := range files {
		filename := filepath.Join(dir, checkpoint)
		cf, err := loadCheckpointFileFromDisk(filename)
		if err != nil {
			cclog.Warnf("[METRICSTORE]> skipping unreadable checkpoint %s: %v", filename, err)
			continue
		}

		rows = flattenCheckpointFile(cf, cluster, host, "node", "", rows)
	}

	return rows, files, nil
}

// estimateRowCount estimates the number of Parquet rows a CheckpointFile will produce.
// Used for pre-allocating the rows slice to avoid repeated append doubling.
func estimateRowCount(cf *CheckpointFile) int {
	n := 0
	for _, cm := range cf.Metrics {
		n += len(cm.Data)
	}
	for _, child := range cf.Children {
		n += estimateRowCount(child)
	}
	return n
}

// archiveCheckpointsToParquet reads checkpoint files for a host directory,
// converts them to Parquet rows. Returns the rows and filenames that were processed.
func archiveCheckpointsToParquet(dir, cluster, host string, from int64) ([]ParquetMetricRow, []string, error) {
	entries, err := os.ReadDir(dir)
	if err != nil {
		return nil, nil, err
	}

	files, err := findFiles(entries, from, false)
	if err != nil {
		return nil, nil, err
	}

	if len(files) == 0 {
		return nil, nil, nil
	}

	// First pass: load checkpoints and estimate total rows for pre-allocation.
	type loaded struct {
		cf       *CheckpointFile
		filename string
	}
	var checkpoints []loaded
	totalEstimate := 0

	for _, checkpoint := range files {
		filename := filepath.Join(dir, checkpoint)
		cf, err := loadCheckpointFileFromDisk(filename)
		if err != nil {
			cclog.Warnf("[METRICSTORE]> skipping unreadable checkpoint %s: %v", filename, err)
			continue
		}
		totalEstimate += estimateRowCount(cf)
		checkpoints = append(checkpoints, loaded{cf: cf, filename: checkpoint})
	}

	if len(checkpoints) == 0 {
		return nil, nil, nil
	}

	rows := make([]ParquetMetricRow, 0, totalEstimate)
	processedFiles := make([]string, 0, len(checkpoints))

	for _, cp := range checkpoints {
		rows = flattenCheckpointFile(cp.cf, cluster, host, "node", "", rows)
		processedFiles = append(processedFiles, cp.filename)
	}

	return rows, processedFiles, nil
}


// estimateRowCount estimates the number of Parquet rows a CheckpointFile will produce.
// Used for pre-allocating the rows slice to avoid repeated append doubling.
func estimateRowCount(cf *CheckpointFile) int {
	n := 0
	for _, cm := range cf.Metrics {
		n += len(cm.Data)
	}
	for _, child := range cf.Children {
		n += estimateRowCount(child)
	}
	return n
}

// archiveCheckpointsToParquet reads checkpoint files for a host directory,
// converts them to Parquet rows. Returns the rows and filenames that were processed.
func archiveCheckpointsToParquet(dir, cluster, host string, from int64) ([]ParquetMetricRow, []string, error) {
	entries, err := os.ReadDir(dir)
	if err != nil {
		return nil, nil, err
	}

	files, err := findFiles(entries, from, false)
	if err != nil {
		return nil, nil, err
	}

	if len(files) == 0 {
		return nil, nil, nil
	}

	// First pass: load checkpoints and estimate total rows for pre-allocation.
	type loaded struct {
		cf       *CheckpointFile
		filename string
	}
	var checkpoints []loaded
	totalEstimate := 0

	for _, checkpoint := range files {
		filename := filepath.Join(dir, checkpoint)
		cf, err := loadCheckpointFileFromDisk(filename)
		if err != nil {
			cclog.Warnf("[METRICSTORE]> skipping unreadable checkpoint %s: %v", filename, err)
			continue
		}
		totalEstimate += estimateRowCount(cf)
		checkpoints = append(checkpoints, loaded{cf: cf, filename: checkpoint})
	}

	if len(checkpoints) == 0 {
		return nil, nil, nil
	}

	rows := make([]ParquetMetricRow, 0, totalEstimate)
	processedFiles := make([]string, 0, len(checkpoints))

	for _, cp := range checkpoints {
		rows = flattenCheckpointFile(cp.cf, cluster, host, "node", "", rows)
		processedFiles = append(processedFiles, cp.filename)
	}

	return rows, processedFiles, nil
}

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package metricstore

import (
	"bufio"
	"encoding/binary"
	"encoding/json"
	"fmt"
	"os"
	"path/filepath"
	"sort"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	pq "github.com/parquet-go/parquet-go"
)

// ParquetMetricRow is the long-format schema for archived metric data.
// One row per (host, metric, scope, scope_id, timestamp) data point.
// Sorted by (cluster, hostname, metric, timestamp) for optimal compression.
type ParquetMetricRow struct {
	Cluster   string  `parquet:"cluster"`
	Hostname  string  `parquet:"hostname"`
	Metric    string  `parquet:"metric"`
	Scope     string  `parquet:"scope"`
	ScopeID   string  `parquet:"scope_id"`
	Timestamp int64   `parquet:"timestamp"`
	Frequency int64   `parquet:"frequency"`
	Value     float32 `parquet:"value"`
}

// flattenCheckpointFile recursively converts a CheckpointFile tree into Parquet rows.
// The scope path is built from the hierarchy: host level is "node", then child names
// map to scope/scope_id (e.g., "socket0" → scope="socket", scope_id="0").
func flattenCheckpointFile(cf *CheckpointFile, cluster, hostname, scope, scopeID string, rows []ParquetMetricRow) []ParquetMetricRow {
	for metricName, cm := range cf.Metrics {
		ts := cm.Start
		for _, v := range cm.Data {
			if !v.IsNaN() {
				rows = append(rows, ParquetMetricRow{
					Cluster:   cluster,
					Hostname:  hostname,
					Metric:    metricName,
					Scope:     scope,
					ScopeID:   scopeID,
					Timestamp: ts,
					Frequency: cm.Frequency,
					Value:     float32(v),
				})
			}
			ts += cm.Frequency
		}
	}

	for childName, childCf := range cf.Children {
		childScope, childScopeID := parseScopeFromName(childName)
		rows = flattenCheckpointFile(childCf, cluster, hostname, childScope, childScopeID, rows)
	}

	return rows
}

// parseScopeFromName infers scope and scope_id from a child level name.
// Examples: "socket0" → ("socket", "0"), "core12" → ("core", "12"),
// "a0" (accelerator) → ("accelerator", "0").
// If the name doesn't match known patterns, it's used as-is for scope with empty scope_id.
func parseScopeFromName(name string) (string, string) {
	prefixes := []struct {
		prefix string
		scope  string
	}{
		{"socket", "socket"},
		{"memoryDomain", "memoryDomain"},
		{"core", "core"},
		{"hwthread", "hwthread"},
		{"cpu", "hwthread"},
		{"accelerator", "accelerator"},
	}

	for _, p := range prefixes {
		if len(name) > len(p.prefix) && name[:len(p.prefix)] == p.prefix {
			id := name[len(p.prefix):]
			if len(id) > 0 && id[0] >= '0' && id[0] <= '9' {
				return p.scope, id
			}
		}
	}

	return name, ""
}

// parquetArchiveWriter supports incremental writes to a Parquet file.
// Each call to WriteHostRows writes one row group (typically one host's data),
// avoiding accumulation of all rows in memory.
type parquetArchiveWriter struct {
	writer *pq.GenericWriter[ParquetMetricRow]
	bw     *bufio.Writer
	f      *os.File
	count  int
}

// newParquetArchiveWriter creates a streaming Parquet writer with Zstd compression.
func newParquetArchiveWriter(filename string) (*parquetArchiveWriter, error) {
	if err := os.MkdirAll(filepath.Dir(filename), CheckpointDirPerms); err != nil {
		return nil, fmt.Errorf("creating archive directory: %w", err)
	}

	f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil {
		return nil, fmt.Errorf("creating parquet file: %w", err)
	}

	bw := bufio.NewWriterSize(f, 1<<20) // 1MB write buffer

	writer := pq.NewGenericWriter[ParquetMetricRow](bw,
		pq.Compression(&pq.Zstd),
	)

	return &parquetArchiveWriter{writer: writer, bw: bw, f: f}, nil
}

// WriteHostRows sorts rows by (metric, timestamp) in-place, writes them,
// and flushes to create a separate row group.
func (w *parquetArchiveWriter) WriteHostRows(rows []ParquetMetricRow) error {
	sort.Slice(rows, func(i, j int) bool {
		if rows[i].Metric != rows[j].Metric {
			return rows[i].Metric < rows[j].Metric
		}
		return rows[i].Timestamp < rows[j].Timestamp
	})

	if _, err := w.writer.Write(rows); err != nil {
		return fmt.Errorf("writing parquet rows: %w", err)
	}

	if err := w.writer.Flush(); err != nil {
		return fmt.Errorf("flushing parquet row group: %w", err)
	}

	w.count += len(rows)
	return nil
}

// Close finalises the Parquet file (footer, buffered I/O, file handle).
func (w *parquetArchiveWriter) Close() error {
	if err := w.writer.Close(); err != nil {
		w.f.Close()
		return fmt.Errorf("closing parquet writer: %w", err)
	}

	if err := w.bw.Flush(); err != nil {
		w.f.Close()
		return fmt.Errorf("flushing parquet file: %w", err)
	}

	return w.f.Close()
}

// sortParquetRows sorts rows by (metric, timestamp) in-place.
func sortParquetRows(rows []ParquetMetricRow) {
	sort.Slice(rows, func(i, j int) bool {
		if rows[i].Metric != rows[j].Metric {
			return rows[i].Metric < rows[j].Metric
		}
		return rows[i].Timestamp < rows[j].Timestamp
	})
}

// loadCheckpointFileFromDisk reads a JSON or binary checkpoint file and returns
// a CheckpointFile. Used by the Parquet archiver to read checkpoint data
// before converting it to Parquet format.
func loadCheckpointFileFromDisk(filename string) (*CheckpointFile, error) {
	f, err := os.Open(filename)
	if err != nil {
		return nil, err
	}
	defer f.Close()

	ext := filepath.Ext(filename)
	switch ext {
	case ".json":
		cf := &CheckpointFile{}
		br := bufio.NewReader(f)
		if err := json.NewDecoder(br).Decode(cf); err != nil {
			return nil, fmt.Errorf("decoding JSON checkpoint %s: %w", filename, err)
		}
		return cf, nil

	case ".bin":
		br := bufio.NewReader(f)
		var magic uint32
		if err := binary.Read(br, binary.LittleEndian, &magic); err != nil {
			return nil, fmt.Errorf("reading magic from %s: %w", filename, err)
		}
		if magic != snapFileMagic {
			return nil, fmt.Errorf("invalid snapshot magic in %s: 0x%08X", filename, magic)
		}
		var fileFrom, fileTo int64
		if err := binary.Read(br, binary.LittleEndian, &fileFrom); err != nil {
			return nil, fmt.Errorf("reading from-timestamp from %s: %w", filename, err)
		}
		if err := binary.Read(br, binary.LittleEndian, &fileTo); err != nil {
			return nil, fmt.Errorf("reading to-timestamp from %s: %w", filename, err)
		}
		cf, err := readBinaryLevel(br)
		if err != nil {
			return nil, fmt.Errorf("reading binary level from %s: %w", filename, err)
		}
		cf.From = fileFrom
		cf.To = fileTo
		return cf, nil

	default:
		return nil, fmt.Errorf("unsupported checkpoint extension: %s", ext)
	}
}

// archiveCheckpointsToParquet reads checkpoint files for a host directory,
// converts them to Parquet rows. Returns the rows and filenames that were processed.
func archiveCheckpointsToParquet(dir, cluster, host string, from int64) ([]ParquetMetricRow, []string, error) {
	entries, err := os.ReadDir(dir)
	if err != nil {
		return nil, nil, err
	}

	files, err := findFiles(entries, from, false)
	if err != nil {
		return nil, nil, err
	}

	if len(files) == 0 {
		return nil, nil, nil
	}

	var rows []ParquetMetricRow

	for _, checkpoint := range files {
		filename := filepath.Join(dir, checkpoint)
		cf, err := loadCheckpointFileFromDisk(filename)
		if err != nil {
			cclog.Warnf("[METRICSTORE]> skipping unreadable checkpoint %s: %v", filename, err)
			continue
		}

		rows = flattenCheckpointFile(cf, cluster, host, "node", "", rows)
	}

	return rows, files, nil
}

+// estimateRowCount estimates the number of Parquet rows a CheckpointFile will produce.

+// Used for pre-allocating the rows slice to avoid repeated append doubling.

+func estimateRowCount(cf *CheckpointFile) int {

+  // First pass: load checkpoints and estimate total rows for pre-allocation.

+    totalEstimate += estimateRowCount(cf)

+    checkpoints = append(checkpoints, loaded{cf: cf, filename: checkpoint})

-    rows = flattenCheckpointFile(cf, cluster, host, "node", "", rows)

+  rows := make([]ParquetMetricRow, 0, totalEstimate)

+  processedFiles := make([]string, 0, len(checkpoints))

+    rows = flattenCheckpointFile(cp.cf, cluster, host, "node", "", rows)

+    processedFiles = append(processedFiles, cp.filename)


Good, expected errors — now let me remove the unused `sort` import since `sortParquetRows` uses it but it's actually used via the `sort.Slice` calls in `WriteHostRows` and `sortParquetRows`. Wait, the diagnostic said `sort` is unused — let me check. Actually the `sort` import IS used by `sort.Slice` in `WriteHostRows` and `sortParquetRows`. The diagnostic might have been stale. Let me proceed.

**Step 2: Restructure `archiveCheckpoints` in `archive.go`**


// archiveCheckpoints archives checkpoint files to Parquet format.
// Produces one Parquet file per cluster: <cleanupDir>/<cluster>/<timestamp>.parquet
func archiveCheckpoints(checkpointsDir, cleanupDir string, from int64) (int, error) {
	clusterEntries, err := os.ReadDir(checkpointsDir)
	if err != nil {
		return 0, err
	}

	totalFiles := 0

	for _, clusterEntry := range clusterEntries {
		if !clusterEntry.IsDir() {
			continue
		}

		cluster := clusterEntry.Name()
		hostEntries, err := os.ReadDir(filepath.Join(checkpointsDir, cluster))
		if err != nil {
			return totalFiles, err
		}

		// Collect rows from all hosts in this cluster using worker pool
		type hostResult struct {
			rows  []ParquetMetricRow
			files []string // checkpoint filenames to delete after successful write
			dir   string   // checkpoint directory for this host
		}

		results := make(chan hostResult, len(hostEntries))
		work := make(chan struct {
			dir, host string
		}, Keys.NumWorkers)

		var wg sync.WaitGroup
		errs := int32(0)

		wg.Add(Keys.NumWorkers)
		for w := 0; w < Keys.NumWorkers; w++ {
			go func() {
				defer wg.Done()
				for item := range work {
					rows, files, err := archiveCheckpointsToParquet(item.dir, cluster, item.host, from)
					if err != nil {
						cclog.Errorf("[METRICSTORE]> error reading checkpoints for %s/%s: %s", cluster, item.host, err.Error())
						atomic.AddInt32(&errs, 1)
						continue
					}
					if len(rows) > 0 {
						results <- hostResult{rows: rows, files: files, dir: item.dir}
					}
				}
			}()
		}

		go func() {
			for _, hostEntry := range hostEntries {
				if !hostEntry.IsDir() {
					continue
				}
				dir := filepath.Join(checkpointsDir, cluster, hostEntry.Name())
				work <- struct {
					dir, host string
				}{dir: dir, host: hostEntry.Name()}
			}
			close(work)
			wg.Wait()
			close(results)
		}()

		// Collect all rows and file info
		var allRows []ParquetMetricRow
		var allResults []hostResult
		for r := range results {
			allRows = append(allRows, r.rows...)
			allResults = append(allResults, r)
		}

		if errs > 0 {
			return totalFiles, fmt.Errorf("%d errors reading checkpoints for cluster %s", errs, cluster)
		}

		if len(allRows) == 0 {
			continue
		}

		// Write one Parquet file per cluster
		parquetFile := filepath.Join(cleanupDir, cluster, fmt.Sprintf("%d.parquet", from))
		if err := writeParquetArchive(parquetFile, allRows); err != nil {
			return totalFiles, fmt.Errorf("writing parquet archive for cluster %s: %w", cluster, err)
		}

		// Delete archived checkpoint files
		for _, result := range allResults {
			for _, file := range result.files {
				filename := filepath.Join(result.dir, file)
				if err := os.Remove(filename); err != nil {
					cclog.Warnf("[METRICSTORE]> could not remove archived checkpoint %s: %v", filename, err)
				} else {
					totalFiles++
				}
			}
		}

		cclog.Infof("[METRICSTORE]> archived %d rows from %d files for cluster %s to %s",
			len(allRows), totalFiles, cluster, parquetFile)
	}

	return totalFiles, nil
}

// archiveCheckpoints archives checkpoint files to Parquet format.
// Produces one Parquet file per cluster: <cleanupDir>/<cluster>/<timestamp>.parquet
// Each host's rows are written as a separate row group to avoid accumulating
// all data in memory at once.
func archiveCheckpoints(checkpointsDir, cleanupDir string, from int64) (int, error) {
	clusterEntries, err := os.ReadDir(checkpointsDir)
	if err != nil {
		return 0, err
	}

	totalFiles := 0

	for _, clusterEntry := range clusterEntries {
		if !clusterEntry.IsDir() {
			continue
		}

		cluster := clusterEntry.Name()
		hostEntries, err := os.ReadDir(filepath.Join(checkpointsDir, cluster))
		if err != nil {
			return totalFiles, err
		}

		// Stream per-host rows to parquet writer via worker pool
		type hostResult struct {
			rows  []ParquetMetricRow
			files []string // checkpoint filenames to delete after successful write
			dir   string   // checkpoint directory for this host
		}

		results := make(chan hostResult, len(hostEntries))
		work := make(chan struct {
			dir, host string
		}, Keys.NumWorkers)

		var wg sync.WaitGroup
		errs := int32(0)

		wg.Add(Keys.NumWorkers)
		for w := 0; w < Keys.NumWorkers; w++ {
			go func() {
				defer wg.Done()
				for item := range work {
					rows, files, err := archiveCheckpointsToParquet(item.dir, cluster, item.host, from)
					if err != nil {
						cclog.Errorf("[METRICSTORE]> error reading checkpoints for %s/%s: %s", cluster, item.host, err.Error())
						atomic.AddInt32(&errs, 1)
						continue
					}
					if len(rows) > 0 {
						results <- hostResult{rows: rows, files: files, dir: item.dir}
					}
				}
			}()
		}

		go func() {
			for _, hostEntry := range hostEntries {
				if !hostEntry.IsDir() {
					continue
				}
				dir := filepath.Join(checkpointsDir, cluster, hostEntry.Name())
				work <- struct {
					dir, host string
				}{dir: dir, host: hostEntry.Name()}
			}
			close(work)
			wg.Wait()
			close(results)
		}()

		// Open streaming writer and write each host's rows as a row group
		parquetFile := filepath.Join(cleanupDir, cluster, fmt.Sprintf("%d.parquet", from))
		writer, err := newParquetArchiveWriter(parquetFile)
		if err != nil {
			// Drain results channel to unblock workers
			for range results {
			}
			return totalFiles, fmt.Errorf("creating parquet writer for cluster %s: %w", cluster, err)
		}

		type deleteItem struct {
			dir   string
			files []string
		}
		var toDelete []deleteItem
		writeErr := error(nil)

		for r := range results {
			if writeErr == nil {
				sortParquetRows(r.rows)
				if err := writer.WriteHostRows(r.rows); err != nil {
					writeErr = err
				}
			}
			// Always track files for deletion (even if write failed, we still drain)
			toDelete = append(toDelete, deleteItem{dir: r.dir, files: r.files})
		}

		if err := writer.Close(); err != nil && writeErr == nil {
			writeErr = err
		}

		if errs > 0 {
			return totalFiles, fmt.Errorf("%d errors reading checkpoints for cluster %s", errs, cluster)
		}

		if writer.count == 0 {
			// No data written — remove empty file
			os.Remove(parquetFile)
			continue
		}

		if writeErr != nil {
			os.Remove(parquetFile)
			return totalFiles, fmt.Errorf("writing parquet archive for cluster %s: %w", cluster, writeErr)
		}

		// Delete archived checkpoint files
		for _, item := range toDelete {
			for _, file := range item.files {
				filename := filepath.Join(item.dir, file)
				if err := os.Remove(filename); err != nil {
					cclog.Warnf("[METRICSTORE]> could not remove archived checkpoint %s: %v", filename, err)
				} else {
					totalFiles++
				}
			}
		}

		cclog.Infof("[METRICSTORE]> archived %d rows from %d files for cluster %s to %s",
			writer.count, totalFiles, cluster, parquetFile)
	}

	return totalFiles, nil
}