Implement the following plan:

# Fix Memory and Performance Issues in Parquet Checkpoint Archiver

## Context

The Parquet archiver (`archiveCheckpoints` in `archive.go`) converts checkpoint files to Parquet format for long-term storage. When checkpoints are large (many metrics, many cores, high-frequency sampling), the current implementation has severe memory problems due to:

1. Materializing the entire checkpoint tree AND a massive flat row slice simultaneously
2. Each `ParquetMetricRow` is ~104 bytes (5 string headers + 2 int64 + float32), and a single host can produce millions of rows
3. The `results` channel is buffered to `len(hostEntries)` (i.e., ALL hosts), so all workers' results can pile up waiting for the single-threaded writer
4. `sort.Slice` on `[]ParquetMetricRow` swaps 104-byte value types — expensive for millions of rows

**Example**: A host with 50 metrics, 8 cores, 12h of data at 1s frequency produces ~19M rows = ~1.9 GB per host. With 10 workers, that's potentially ~19 GB in flight.

## Critical Files

- `pkg/metricstore/parquetArchive.go` — row schema, flatten, sort, writer
- `pkg/metricstore/archive.go` — orchestration (`archiveCheckpoints`)
- `pkg/metricstore/checkpoint.go` — `CheckpointFile` struct, `findFiles`

## Problems and Fixes

### Problem 1: Unbounded Results Channel (Critical — easiest fix)

**archive.go:202**: `results := make(chan hostResult, len(hostEntries))`

This buffers ALL host results. If workers are fast and the writer is slow (Zstd compression), hundreds of hosts' row slices accumulate in the channel simultaneously.

**Fix**: Reduce buffer to `Keys.NumWorkers` (or a small constant like 2). Workers will naturally back-pressure when the channel is full, keeping at most `NumWorkers + buffer` host results in memory.

```go
results := make(chan hostResult, 2)  // back-pressure: at most NumWorkers+2 in flight
```

### Problem 2: Redundant String Data in Flat Rows (Critical — biggest memory win)

Each `ParquetMetricRow` stores 5 strings (cluster, hostname, metric, scope, scope_id) = 80 bytes of string headers. For rows from the same host+metric, cluster/hostname/metric are identical. With 19M rows, that's ~1.5 GB of redundant string headers alone.

**Fix**: Write Parquet rows in a column-oriented streaming fashion directly from the `CheckpointFile` tree, avoiding the flat `[]ParquetMetricRow` intermediate entirely.

Replace `flattenCheckpointFile` + `sortParquetRows` + `WriteHostRows` with a method that walks the `CheckpointFile` tree metric-by-metric and writes rows in already-sorted order (metrics are iterated one at a time, timestamps are naturally ascending within each metric). This eliminates both the giant allocation and the sort.

```go
func (w *parquetArchiveWriter) WriteCheckpointFile(cf *CheckpointFile, cluster, hostname, scope, scopeID string) error {
    // Write metrics for this level — already in (metric, timestamp) order
    // since we iterate one metric at a time and timestamps are sequential
    for metricName, cm := range cf.Metrics {
        // Build rows for this one metric and write immediately
        // Small batch: only len(cm.Data) rows at a time
    }
    // Recurse into children
    for childName, childCf := range cf.Children {
        childScope, childScopeID := parseScopeFromName(childName)
        w.WriteCheckpointFile(childCf, cluster, hostname, childScope, childScopeID)
    }
}
```

**Caveat**: Map iteration order is random in Go, so metric names won't be in sorted order across metrics. Two options:
- (a) Collect and sort metric names first (tiny allocation — just the keys), then iterate in order. This gives perfect (metric, timestamp) sort with zero row accumulation.
- (b) Accumulate rows per-level only (much smaller than full tree) and sort that batch.

Option (a) is preferred — zero-copy, streaming, and produces optimally sorted output.

### Problem 3: sort.Slice Swaps 104-byte Structs (Medium)

If we keep any sorting, `sort.Slice` on value-type structs swaps the full 104 bytes per swap. For millions of rows this is slow.

**Fix**: If Problem 2's fix is adopted (streaming metric-by-metric), sorting is eliminated entirely. If sorting is still needed for any reason, sort an index slice and then permute:

```go
idx := make([]int, len(rows))
for i := range idx { idx[i] = i }
sort.Slice(idx, func(i, j int) bool { ... })
// Then write rows[idx[i]] in order
```

### Problem 4: Double Materialization — CheckpointFile + Rows (Medium)

`archiveCheckpointsToParquet` loads ALL checkpoint files for a host into `[]loaded` (line 256-268), then iterates them again to flatten. Both the `CheckpointFile` trees and the `[]ParquetMetricRow` slice coexist in memory.

**Fix**: Process one checkpoint file at a time — load, stream to writer, discard:

```go
for _, checkpoint := range files {
    cf, err := loadCheckpointFileFromDisk(filename)
    if err != nil { continue }
    writer.WriteCheckpointFile(cf, cluster, host, "node", "")
    // cf is now unreferenced and eligible for GC
}
```

### Problem 5: Duplicate Sort (Minor)

`sortParquetRows` is called at archive.go:262, and `WriteHostRows` calls `sort.Slice` again on the same data (parquetArchive.go:128). The data is sorted twice.

**Fix**: Remove one of them. If Problem 2's fix is adopted, both are eliminated.

## Implementation Plan

### Step 1: Reduce results channel buffer
- **File**: `pkg/metricstore/archive.go:202`
- Change `len(hostEntries)` to `2`

### Step 2: Add streaming writer method
- **File**: `pkg/metricstore/parquetArchive.go`
- Add `WriteCheckpointFileDirect(cf *CheckpointFile, cluster, hostname, scope, scopeID string) error`
- Iterates metrics with sorted keys, writes rows per-metric in timestamp order
- Recurses into children (sorted by name for determinism)
- Uses a small reusable `[]ParquetMetricRow` batch buffer (e.g., 1024 rows) to amortize writer calls

### Step 3: Simplify archiveCheckpoints to stream per-file
- **File**: `pkg/metricstore/archive.go`
- Replace `archiveCheckpointsToParquet` call pattern: instead of returning `[]ParquetMetricRow`, pass the writer directly to each worker
- Each worker loads one checkpoint file at a time, streams it through the writer, then discards
- **Concurrency note**: The parquet writer is NOT thread-safe, so the main loop must remain single-threaded for writes. Workers load+flatten, main thread writes. But with the streaming approach, the `hostResult` struct no longer carries `[]ParquetMetricRow` — it could carry the `*CheckpointFile` directly (much smaller than the flattened rows) or we serialize in the main thread.

**Revised architecture**:
```
Workers: load checkpoint files from disk → send CheckpointFile on channel
Main:    receive CheckpointFile → WriteCheckpointFileDirect → delete source files
```

This means workers only do I/O (disk reads), and the main thread does the streaming write. The `CheckpointFile` is much smaller than the flattened rows (it stores `[]schema.Float` arrays, not per-row structs with repeated strings).

### Step 4: Remove dead code
- Remove `flattenCheckpointFile`, `sortParquetRows`, `estimateRowCount` (all unused after refactor)
- Remove `WriteHostRows` if fully replaced
- Keep `ParquetMetricRow` schema (still needed by the writer)

### Step 5: Remove duplicate sort in WriteHostRows
- If `WriteHostRows` is kept for any reason, remove the redundant sort (it's also called from `archiveCheckpoints`)

## Verification

```bash
# Build
go build ./pkg/metricstore/...

# Run existing tests
go test ./pkg/metricstore/...

# Manual: run with archiving enabled and monitor memory
# Compare parquet output before/after to ensure same data
```

## Memory Impact Summary

| Scenario (50 metrics, 8 cores, 12h@1s) | Before | After |
|---|---|---|
| Per-host peak (rows) | ~1.9 GB | ~0 (streaming) |
| Per-host peak (CheckpointFile) | ~70 MB | ~70 MB (transient) |
| Results channel (1000 hosts) | 1000 slots | 2 slots |
| Sort overhead | O(n log n) on 19M×104B | eliminated |

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-ClusterCockpit-cc-backend/b8aa3f88-b5cb-4709-9bb6-99559b819b8b.jsonl


Let me read the critical files first.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package metricstore
     7→
     8→import (
     9→	"context"
    10→	"errors"
    11→	"fmt"
    12→	"os"
    13→	"path/filepath"
    14→	"sync"
    15→	"sync/atomic"
    16→	"time"
    17→
    18→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    19→)
    20→
    21→// Worker for either Archiving or Deleting files
    22→
    23→func CleanUp(wg *sync.WaitGroup, ctx context.Context) {
    24→	if Keys.Cleanup.Mode == "archive" {
    25→		// Run as Archiver
    26→		cleanUpWorker(wg, ctx,
    27→			Keys.RetentionInMemory,
    28→			"archiving",
    29→			Keys.Cleanup.RootDir,
    30→			false,
    31→		)
    32→	} else {
    33→		// Run as Deleter
    34→		cleanUpWorker(wg, ctx,
    35→			Keys.RetentionInMemory,
    36→			"deleting",
    37→			"",
    38→			true,
    39→		)
    40→	}
    41→}
    42→
    43→// cleanUpWorker takes simple values to configure what it does
    44→func cleanUpWorker(wg *sync.WaitGroup, ctx context.Context, interval string, mode string, cleanupDir string, delete bool) {
    45→	wg.Go(func() {
    46→
    47→		d, err := time.ParseDuration(interval)
    48→		if err != nil {
    49→			cclog.Fatalf("[METRICSTORE]> error parsing %s interval duration: %v\n", mode, err)
    50→		}
    51→		if d <= 0 {
    52→			return
    53→		}
    54→
    55→		ticker := time.NewTicker(d)
    56→		defer ticker.Stop()
    57→
    58→		for {
    59→			select {
    60→			case <-ctx.Done():
    61→				return
    62→			case <-ticker.C:
    63→				t := time.Now().Add(-d)
    64→				cclog.Infof("[METRICSTORE]> start %s checkpoints (older than %s)...", mode, t.Format(time.RFC3339))
    65→
    66→				n, err := CleanupCheckpoints(Keys.Checkpoints.RootDir, cleanupDir, t.Unix(), delete)
    67→
    68→				if err != nil {
    69→					cclog.Errorf("[METRICSTORE]> %s failed: %s", mode, err.Error())
    70→				} else {
    71→					if delete {
    72→						cclog.Infof("[METRICSTORE]> done: %d checkpoints deleted", n)
    73→					} else {
    74→						cclog.Infof("[METRICSTORE]> done: %d checkpoint files archived to parquet", n)
    75→					}
    76→				}
    77→			}
    78→		}
    79→	})
    80→}
    81→
    82→var ErrNoNewArchiveData error = errors.New("all data already archived")
    83→
    84→// CleanupCheckpoints deletes or archives all checkpoint files older than `from`.
    85→// When archiving, consolidates all hosts per cluster into a single Parquet file.
    86→func CleanupCheckpoints(checkpointsDir, cleanupDir string, from int64, deleteInstead bool) (int, error) {
    87→	if deleteInstead {
    88→		return deleteCheckpoints(checkpointsDir, from)
    89→	}
    90→
    91→	return archiveCheckpoints(checkpointsDir, cleanupDir, from)
    92→}
    93→
    94→// deleteCheckpoints removes checkpoint files older than `from` across all clusters/hosts.
    95→func deleteCheckpoints(checkpointsDir string, from int64) (int, error) {
    96→	entries1, err := os.ReadDir(checkpointsDir)
    97→	if err != nil {
    98→		return 0, err
    99→	}
   100→
   101→	type workItem struct {
   102→		dir            string
   103→		cluster, host  string
   104→	}
   105→
   106→	var wg sync.WaitGroup
   107→	n, errs := int32(0), int32(0)
   108→	work := make(chan workItem, Keys.NumWorkers)
   109→
   110→	wg.Add(Keys.NumWorkers)
   111→	for worker := 0; worker < Keys.NumWorkers; worker++ {
   112→		go func() {
   113→			defer wg.Done()
   114→			for item := range work {
   115→				entries, err := os.ReadDir(item.dir)
   116→				if err != nil {
   117→					cclog.Errorf("error reading %s/%s: %s", item.cluster, item.host, err.Error())
   118→					atomic.AddInt32(&errs, 1)
   119→					continue
   120→				}
   121→
   122→				files, err := findFiles(entries, from, false)
   123→				if err != nil {
   124→					cclog.Errorf("error finding files in %s/%s: %s", item.cluster, item.host, err.Error())
   125→					atomic.AddInt32(&errs, 1)
   126→					continue
   127→				}
   128→
   129→				for _, checkpoint := range files {
   130→					if err := os.Remove(filepath.Join(item.dir, checkpoint)); err != nil {
   131→						cclog.Errorf("error deleting %s/%s/%s: %s", item.cluster, item.host, checkpoint, err.Error())
   132→						atomic.AddInt32(&errs, 1)
   133→					} else {
   134→						atomic.AddInt32(&n, 1)
   135→					}
   136→				}
   137→			}
   138→		}()
   139→	}
   140→
   141→	for _, de1 := range entries1 {
   142→		entries2, e := os.ReadDir(filepath.Join(checkpointsDir, de1.Name()))
   143→		if e != nil {
   144→			err = e
   145→			continue
   146→		}
   147→
   148→		for _, de2 := range entries2 {
   149→			work <- workItem{
   150→				dir:     filepath.Join(checkpointsDir, de1.Name(), de2.Name()),
   151→				cluster: de1.Name(),
   152→				host:    de2.Name(),
   153→			}
   154→		}
   155→	}
   156→
   157→	close(work)
   158→	wg.Wait()
   159→
   160→	if err != nil {
   161→		return int(n), err
   162→	}
   163→	if errs > 0 {
   164→		return int(n), fmt.Errorf("%d errors happened while deleting (%d successes)", errs, n)
   165→	}
   166→	return int(n), nil
   167→}
   168→
   169→// archiveCheckpoints archives checkpoint files to Parquet format.
   170→// Produces one Parquet file per cluster: <cleanupDir>/<cluster>/<timestamp>.parquet
   171→// Each host's rows are written as a separate row group to avoid accumulating
   172→// all data in memory at once.
   173→func archiveCheckpoints(checkpointsDir, cleanupDir string, from int64) (int, error) {
   174→	cclog.Info("[METRICSTORE]> start archiving checkpoints to parquet")
   175→	startTime := time.Now()
   176→
   177→	clusterEntries, err := os.ReadDir(checkpointsDir)
   178→	if err != nil {
   179→		return 0, err
   180→	}
   181→
   182→	totalFiles := 0
   183→
   184→	for _, clusterEntry := range clusterEntries {
   185→		if !clusterEntry.IsDir() {
   186→			continue
   187→		}
   188→
   189→		cluster := clusterEntry.Name()
   190→		hostEntries, err := os.ReadDir(filepath.Join(checkpointsDir, cluster))
   191→		if err != nil {
   192→			return totalFiles, err
   193→		}
   194→
   195→		// Stream per-host rows to parquet writer via worker pool
   196→		type hostResult struct {
   197→			rows  []ParquetMetricRow
   198→			files []string // checkpoint filenames to delete after successful write
   199→			dir   string   // checkpoint directory for this host
   200→		}
   201→
   202→		results := make(chan hostResult, len(hostEntries))
   203→		work := make(chan struct {
   204→			dir, host string
   205→		}, Keys.NumWorkers)
   206→
   207→		var wg sync.WaitGroup
   208→		errs := int32(0)
   209→
   210→		wg.Add(Keys.NumWorkers)
   211→		for w := 0; w < Keys.NumWorkers; w++ {
   212→			go func() {
   213→				defer wg.Done()
   214→				for item := range work {
   215→					rows, files, err := archiveCheckpointsToParquet(item.dir, cluster, item.host, from)
   216→					if err != nil {
   217→						cclog.Errorf("[METRICSTORE]> error reading checkpoints for %s/%s: %s", cluster, item.host, err.Error())
   218→						atomic.AddInt32(&errs, 1)
   219→						continue
   220→					}
   221→					if len(rows) > 0 {
   222→						results <- hostResult{rows: rows, files: files, dir: item.dir}
   223→					}
   224→				}
   225→			}()
   226→		}
   227→
   228→		go func() {
   229→			for _, hostEntry := range hostEntries {
   230→				if !hostEntry.IsDir() {
   231→					continue
   232→				}
   233→				dir := filepath.Join(checkpointsDir, cluster, hostEntry.Name())
   234→				work <- struct {
   235→					dir, host string
   236→				}{dir: dir, host: hostEntry.Name()}
   237→			}
   238→			close(work)
   239→			wg.Wait()
   240→			close(results)
   241→		}()
   242→
   243→		// Open streaming writer and write each host's rows as a row group
   244→		parquetFile := filepath.Join(cleanupDir, cluster, fmt.Sprintf("%d.parquet", from))
   245→		writer, err := newParquetArchiveWriter(parquetFile)
   246→		if err != nil {
   247→			// Drain results channel to unblock workers
   248→			for range results {
   249→			}
   250→			return totalFiles, fmt.Errorf("creating parquet writer for cluster %s: %w", cluster, err)
   251→		}
   252→
   253→		type deleteItem struct {
   254→			dir   string
   255→			files []string
   256→		}
   257→		var toDelete []deleteItem
   258→		writeErr := error(nil)
   259→
   260→		for r := range results {
   261→			if writeErr == nil {
   262→				sortParquetRows(r.rows)
   263→				if err := writer.WriteHostRows(r.rows); err != nil {
   264→					writeErr = err
   265→				}
   266→			}
   267→			// Always track files for deletion (even if write failed, we still drain)
   268→			toDelete = append(toDelete, deleteItem{dir: r.dir, files: r.files})
   269→		}
   270→
   271→		if err := writer.Close(); err != nil && writeErr == nil {
   272→			writeErr = err
   273→		}
   274→
   275→		if errs > 0 {
   276→			return totalFiles, fmt.Errorf("%d errors reading checkpoints for cluster %s", errs, cluster)
   277→		}
   278→
   279→		if writer.count == 0 {
   280→			// No data written — remove empty file
   281→			os.Remove(parquetFile)
   282→			continue
   283→		}
   284→
   285→		if writeErr != nil {
   286→			os.Remove(parquetFile)
   287→			return totalFiles, fmt.Errorf("writing parquet archive for cluster %s: %w", cluster, writeErr)
   288→		}
   289→
   290→		// Delete archived checkpoint files
   291→		for _, item := range toDelete {
   292→			for _, file := range item.files {
   293→				filename := filepath.Join(item.dir, file)
   294→				if err := os.Remove(filename); err != nil {
   295→					cclog.Warnf("[METRICSTORE]> could not remove archived checkpoint %s: %v", filename, err)
   296→				} else {
   297→					totalFiles++
   298→				}
   299→			}
   300→		}
   301→
   302→		cclog.Infof("[METRICSTORE]> archived %d rows from %d files for cluster %s to %s",
   303→			writer.count, totalFiles, cluster, parquetFile)
   304→	}
   305→
   306→	cclog.Infof("[METRICSTORE]> archiving checkpoints completed in %s (%d files)", time.Since(startTime).Round(time.Millisecond), totalFiles)
   307→	return totalFiles, nil
   308→}
   309→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package metricstore

import (
	"context"
	"errors"
	"fmt"
	"os"
	"path/filepath"
	"sync"
	"sync/atomic"
	"time"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
)

// Worker for either Archiving or Deleting files

func CleanUp(wg *sync.WaitGroup, ctx context.Context) {
	if Keys.Cleanup.Mode == "archive" {
		// Run as Archiver
		cleanUpWorker(wg, ctx,
			Keys.RetentionInMemory,
			"archiving",
			Keys.Cleanup.RootDir,
			false,
		)
	} else {
		// Run as Deleter
		cleanUpWorker(wg, ctx,
			Keys.RetentionInMemory,
			"deleting",
			"",
			true,
		)
	}
}

// cleanUpWorker takes simple values to configure what it does
func cleanUpWorker(wg *sync.WaitGroup, ctx context.Context, interval string, mode string, cleanupDir string, delete bool) {
	wg.Go(func() {

		d, err := time.ParseDuration(interval)
		if err != nil {
			cclog.Fatalf("[METRICSTORE]> error parsing %s interval duration: %v\n", mode, err)
		}
		if d <= 0 {
			return
		}

		ticker := time.NewTicker(d)
		defer ticker.Stop()

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				t := time.Now().Add(-d)
				cclog.Infof("[METRICSTORE]> start %s checkpoints (older than %s)...", mode, t.Format(time.RFC3339))

				n, err := CleanupCheckpoints(Keys.Checkpoints.RootDir, cleanupDir, t.Unix(), delete)

				if err != nil {
					cclog.Errorf("[METRICSTORE]> %s failed: %s", mode, err.Error())
				} else {
					if delete {
						cclog.Infof("[METRICSTORE]> done: %d checkpoints deleted", n)
					} else {
						cclog.Infof("[METRICSTORE]> done: %d checkpoint files archived to parquet", n)
					}
				}
			}
		}
	})
}

var ErrNoNewArchiveData error = errors.New("all data already archived")

// CleanupCheckpoints deletes or archives all checkpoint files older than `from`.
// When archiving, consolidates all hosts per cluster into a single Parquet file.
func CleanupCheckpoints(checkpointsDir, cleanupDir string, from int64, deleteInstead bool) (int, error) {
	if deleteInstead {
		return deleteCheckpoints(checkpointsDir, from)
	}

	return archiveCheckpoints(checkpointsDir, cleanupDir, from)
}

// deleteCheckpoints removes checkpoint files older than `from` across all clusters/hosts.
func deleteCheckpoints(checkpointsDir string, from int64) (int, error) {
	entries1, err := os.ReadDir(checkpointsDir)
	if err != nil {
		return 0, err
	}

	type workItem struct {
		dir            string
		cluster, host  string
	}

	var wg sync.WaitGroup
	n, errs := int32(0), int32(0)
	work := make(chan workItem, Keys.NumWorkers)

	wg.Add(Keys.NumWorkers)
	for worker := 0; worker < Keys.NumWorkers; worker++ {
		go func() {
			defer wg.Done()
			for item := range work {
				entries, err := os.ReadDir(item.dir)
				if err != nil {
					cclog.Errorf("error reading %s/%s: %s", item.cluster, item.host, err.Error())
					atomic.AddInt32(&errs, 1)
					continue
				}

				files, err := findFiles(entries, from, false)
				if err != nil {
					cclog.Errorf("error finding files in %s/%s: %s", item.cluster, item.host, err.Error())
					atomic.AddInt32(&errs, 1)
					continue
				}

				for _, checkpoint := range files {
					if err := os.Remove(filepath.Join(item.dir, checkpoint)); err != nil {
						cclog.Errorf("error deleting %s/%s/%s: %s", item.cluster, item.host, checkpoint, err.Error())
						atomic.AddInt32(&errs, 1)
					} else {
						atomic.AddInt32(&n, 1)
					}
				}
			}
		}()
	}

	for _, de1 := range entries1 {
		entries2, e := os.ReadDir(filepath.Join(checkpointsDir, de1.Name()))
		if e != nil {
			err = e
			continue
		}

		for _, de2 := range entries2 {
			work <- workItem{
				dir:     filepath.Join(checkpointsDir, de1.Name(), de2.Name()),
				cluster: de1.Name(),
				host:    de2.Name(),
			}
		}
	}

	close(work)
	wg.Wait()

	if err != nil {
		return int(n), err
	}
	if errs > 0 {
		return int(n), fmt.Errorf("%d errors happened while deleting (%d successes)", errs, n)
	}
	return int(n), nil
}

// archiveCheckpoints archives checkpoint files to Parquet format.
// Produces one Parquet file per cluster: <cleanupDir>/<cluster>/<timestamp>.parquet
// Each host's rows are written as a separate row group to avoid accumulating
// all data in memory at once.
func archiveCheckpoints(checkpointsDir, cleanupDir string, from int64) (int, error) {
	cclog.Info("[METRICSTORE]> start archiving checkpoints to parquet")
	startTime := time.Now()

	clusterEntries, err := os.ReadDir(checkpointsDir)
	if err != nil {
		return 0, err
	}

	totalFiles := 0

	for _, clusterEntry := range clusterEntries {
		if !clusterEntry.IsDir() {
			continue
		}

		cluster := clusterEntry.Name()
		hostEntries, err := os.ReadDir(filepath.Join(checkpointsDir, cluster))
		if err != nil {
			return totalFiles, err
		}

		// Stream per-host rows to parquet writer via worker pool
		type hostResult struct {
			rows  []ParquetMetricRow
			files []string // checkpoint filenames to delete after successful write
			dir   string   // checkpoint directory for this host
		}

		results := make(chan hostResult, len(hostEntries))
		work := make(chan struct {
			dir, host string
		}, Keys.NumWorkers)

		var wg sync.WaitGroup
		errs := int32(0)

		wg.Add(Keys.NumWorkers)
		for w := 0; w < Keys.NumWorkers; w++ {
			go func() {
				defer wg.Done()
				for item := range work {
					rows, files, err := archiveCheckpointsToParquet(item.dir, cluster, item.host, from)
					if err != nil {
						cclog.Errorf("[METRICSTORE]> error reading checkpoints for %s/%s: %s", cluster, item.host, err.Error())
						atomic.AddInt32(&errs, 1)
						continue
					}
					if len(rows) > 0 {
						results <- hostResult{rows: rows, files: files, dir: item.dir}
					}
				}
			}()
		}

		go func() {
			for _, hostEntry := range hostEntries {
				if !hostEntry.IsDir() {
					continue
				}
				dir := filepath.Join(checkpointsDir, cluster, hostEntry.Name())
				work <- struct {
					dir, host string
				}{dir: dir, host: hostEntry.Name()}
			}
			close(work)
			wg.Wait()
			close(results)
		}()

		// Open streaming writer and write each host's rows as a row group
		parquetFile := filepath.Join(cleanupDir, cluster, fmt.Sprintf("%d.parquet", from))
		writer, err := newParquetArchiveWriter(parquetFile)
		if err != nil {
			// Drain results channel to unblock workers
			for range results {
			}
			return totalFiles, fmt.Errorf("creating parquet writer for cluster %s: %w", cluster, err)
		}

		type deleteItem struct {
			dir   string
			files []string
		}
		var toDelete []deleteItem
		writeErr := error(nil)

		for r := range results {
			if writeErr == nil {
				sortParquetRows(r.rows)
				if err := writer.WriteHostRows(r.rows); err != nil {
					writeErr = err
				}
			}
			// Always track files for deletion (even if write failed, we still drain)
			toDelete = append(toDelete, deleteItem{dir: r.dir, files: r.files})
		}

		if err := writer.Close(); err != nil && writeErr == nil {
			writeErr = err
		}

		if errs > 0 {
			return totalFiles, fmt.Errorf("%d errors reading checkpoints for cluster %s", errs, cluster)
		}

		if writer.count == 0 {
			// No data written — remove empty file
			os.Remove(parquetFile)
			continue
		}

		if writeErr != nil {
			os.Remove(parquetFile)
			return totalFiles, fmt.Errorf("writing parquet archive for cluster %s: %w", cluster, writeErr)
		}

		// Delete archived checkpoint files
		for _, item := range toDelete {
			for _, file := range item.files {
				filename := filepath.Join(item.dir, file)
				if err := os.Remove(filename); err != nil {
					cclog.Warnf("[METRICSTORE]> could not remove archived checkpoint %s: %v", filename, err)
				} else {
					totalFiles++
				}
			}
		}

		cclog.Infof("[METRICSTORE]> archived %d rows from %d files for cluster %s to %s",
			writer.count, totalFiles, cluster, parquetFile)
	}

	cclog.Infof("[METRICSTORE]> archiving checkpoints completed in %s (%d files)", time.Since(startTime).Round(time.Millisecond), totalFiles)
	return totalFiles, nil
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package metricstore
     7→
     8→import (
     9→	"bufio"
    10→	"encoding/binary"
    11→	"encoding/json"
    12→	"fmt"
    13→	"os"
    14→	"path/filepath"
    15→	"sort"
    16→
    17→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    18→	pq "github.com/parquet-go/parquet-go"
    19→)
    20→
    21→// ParquetMetricRow is the long-format schema for archived metric data.
    22→// One row per (host, metric, scope, scope_id, timestamp) data point.
    23→// Sorted by (cluster, hostname, metric, timestamp) for optimal compression.
    24→type ParquetMetricRow struct {
    25→	Cluster   string  `parquet:"cluster"`
    26→	Hostname  string  `parquet:"hostname"`
    27→	Metric    string  `parquet:"metric"`
    28→	Scope     string  `parquet:"scope"`
    29→	ScopeID   string  `parquet:"scope_id"`
    30→	Timestamp int64   `parquet:"timestamp"`
    31→	Frequency int64   `parquet:"frequency"`
    32→	Value     float32 `parquet:"value"`
    33→}
    34→
    35→// flattenCheckpointFile recursively converts a CheckpointFile tree into Parquet rows.
    36→// The scope path is built from the hierarchy: host level is "node", then child names
    37→// map to scope/scope_id (e.g., "socket0" → scope="socket", scope_id="0").
    38→func flattenCheckpointFile(cf *CheckpointFile, cluster, hostname, scope, scopeID string, rows []ParquetMetricRow) []ParquetMetricRow {
    39→	for metricName, cm := range cf.Metrics {
    40→		ts := cm.Start
    41→		for _, v := range cm.Data {
    42→			if !v.IsNaN() {
    43→				rows = append(rows, ParquetMetricRow{
    44→					Cluster:   cluster,
    45→					Hostname:  hostname,
    46→					Metric:    metricName,
    47→					Scope:     scope,
    48→					ScopeID:   scopeID,
    49→					Timestamp: ts,
    50→					Frequency: cm.Frequency,
    51→					Value:     float32(v),
    52→				})
    53→			}
    54→			ts += cm.Frequency
    55→		}
    56→	}
    57→
    58→	for childName, childCf := range cf.Children {
    59→		childScope, childScopeID := parseScopeFromName(childName)
    60→		rows = flattenCheckpointFile(childCf, cluster, hostname, childScope, childScopeID, rows)
    61→	}
    62→
    63→	return rows
    64→}
    65→
    66→// parseScopeFromName infers scope and scope_id from a child level name.
    67→// Examples: "socket0" → ("socket", "0"), "core12" → ("core", "12"),
    68→// "a0" (accelerator) → ("accelerator", "0").
    69→// If the name doesn't match known patterns, it's used as-is for scope with empty scope_id.
    70→func parseScopeFromName(name string) (string, string) {
    71→	prefixes := []struct {
    72→		prefix string
    73→		scope  string
    74→	}{
    75→		{"socket", "socket"},
    76→		{"memoryDomain", "memoryDomain"},
    77→		{"core", "core"},
    78→		{"hwthread", "hwthread"},
    79→		{"cpu", "hwthread"},
    80→		{"accelerator", "accelerator"},
    81→	}
    82→
    83→	for _, p := range prefixes {
    84→		if len(name) > len(p.prefix) && name[:len(p.prefix)] == p.prefix {
    85→			id := name[len(p.prefix):]
    86→			if len(id) > 0 && id[0] >= '0' && id[0] <= '9' {
    87→				return p.scope, id
    88→			}
    89→		}
    90→	}
    91→
    92→	return name, ""
    93→}
    94→
    95→// parquetArchiveWriter supports incremental writes to a Parquet file.
    96→// Each call to WriteHostRows writes one row group (typically one host's data),
    97→// avoiding accumulation of all rows in memory.
    98→type parquetArchiveWriter struct {
    99→	writer *pq.GenericWriter[ParquetMetricRow]
   100→	bw     *bufio.Writer
   101→	f      *os.File
   102→	count  int
   103→}
   104→
   105→// newParquetArchiveWriter creates a streaming Parquet writer with Zstd compression.
   106→func newParquetArchiveWriter(filename string) (*parquetArchiveWriter, error) {
   107→	if err := os.MkdirAll(filepath.Dir(filename), CheckpointDirPerms); err != nil {
   108→		return nil, fmt.Errorf("creating archive directory: %w", err)
   109→	}
   110→
   111→	f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
   112→	if err != nil {
   113→		return nil, fmt.Errorf("creating parquet file: %w", err)
   114→	}
   115→
   116→	bw := bufio.NewWriterSize(f, 1<<20) // 1MB write buffer
   117→
   118→	writer := pq.NewGenericWriter[ParquetMetricRow](bw,
   119→		pq.Compression(&pq.Zstd),
   120→	)
   121→
   122→	return &parquetArchiveWriter{writer: writer, bw: bw, f: f}, nil
   123→}
   124→
   125→// WriteHostRows sorts rows by (metric, timestamp) in-place, writes them,
   126→// and flushes to create a separate row group.
   127→func (w *parquetArchiveWriter) WriteHostRows(rows []ParquetMetricRow) error {
   128→	sort.Slice(rows, func(i, j int) bool {
   129→		if rows[i].Metric != rows[j].Metric {
   130→			return rows[i].Metric < rows[j].Metric
   131→		}
   132→		return rows[i].Timestamp < rows[j].Timestamp
   133→	})
   134→
   135→	if _, err := w.writer.Write(rows); err != nil {
   136→		return fmt.Errorf("writing parquet rows: %w", err)
   137→	}
   138→
   139→	if err := w.writer.Flush(); err != nil {
   140→		return fmt.Errorf("flushing parquet row group: %w", err)
   141→	}
   142→
   143→	w.count += len(rows)
   144→	return nil
   145→}
   146→
   147→// Close finalises the Parquet file (footer, buffered I/O, file handle).
   148→func (w *parquetArchiveWriter) Close() error {
   149→	if err := w.writer.Close(); err != nil {
   150→		w.f.Close()
   151→		return fmt.Errorf("closing parquet writer: %w", err)
   152→	}
   153→
   154→	if err := w.bw.Flush(); err != nil {
   155→		w.f.Close()
   156→		return fmt.Errorf("flushing parquet file: %w", err)
   157→	}
   158→
   159→	return w.f.Close()
   160→}
   161→
   162→// sortParquetRows sorts rows by (metric, timestamp) in-place.
   163→func sortParquetRows(rows []ParquetMetricRow) {
   164→	sort.Slice(rows, func(i, j int) bool {
   165→		if rows[i].Metric != rows[j].Metric {
   166→			return rows[i].Metric < rows[j].Metric
   167→		}
   168→		return rows[i].Timestamp < rows[j].Timestamp
   169→	})
   170→}
   171→
   172→// loadCheckpointFileFromDisk reads a JSON or binary checkpoint file and returns
   173→// a CheckpointFile. Used by the Parquet archiver to read checkpoint data
   174→// before converting it to Parquet format.
   175→func loadCheckpointFileFromDisk(filename string) (*CheckpointFile, error) {
   176→	f, err := os.Open(filename)
   177→	if err != nil {
   178→		return nil, err
   179→	}
   180→	defer f.Close()
   181→
   182→	ext := filepath.Ext(filename)
   183→	switch ext {
   184→	case ".json":
   185→		cf := &CheckpointFile{}
   186→		br := bufio.NewReader(f)
   187→		if err := json.NewDecoder(br).Decode(cf); err != nil {
   188→			return nil, fmt.Errorf("decoding JSON checkpoint %s: %w", filename, err)
   189→		}
   190→		return cf, nil
   191→
   192→	case ".bin":
   193→		br := bufio.NewReader(f)
   194→		var magic uint32
   195→		if err := binary.Read(br, binary.LittleEndian, &magic); err != nil {
   196→			return nil, fmt.Errorf("reading magic from %s: %w", filename, err)
   197→		}
   198→		if magic != snapFileMagic {
   199→			return nil, fmt.Errorf("invalid snapshot magic in %s: 0x%08X", filename, magic)
   200→		}
   201→		var fileFrom, fileTo int64
   202→		if err := binary.Read(br, binary.LittleEndian, &fileFrom); err != nil {
   203→			return nil, fmt.Errorf("reading from-timestamp from %s: %w", filename, err)
   204→		}
   205→		if err := binary.Read(br, binary.LittleEndian, &fileTo); err != nil {
   206→			return nil, fmt.Errorf("reading to-timestamp from %s: %w", filename, err)
   207→		}
   208→		cf, err := readBinaryLevel(br)
   209→		if err != nil {
   210→			return nil, fmt.Errorf("reading binary level from %s: %w", filename, err)
   211→		}
   212→		cf.From = fileFrom
   213→		cf.To = fileTo
   214→		return cf, nil
   215→
   216→	default:
   217→		return nil, fmt.Errorf("unsupported checkpoint extension: %s", ext)
   218→	}
   219→}
   220→
   221→// estimateRowCount estimates the number of Parquet rows a CheckpointFile will produce.
   222→// Used for pre-allocating the rows slice to avoid repeated append doubling.
   223→func estimateRowCount(cf *CheckpointFile) int {
   224→	n := 0
   225→	for _, cm := range cf.Metrics {
   226→		n += len(cm.Data)
   227→	}
   228→	for _, child := range cf.Children {
   229→		n += estimateRowCount(child)
   230→	}
   231→	return n
   232→}
   233→
   234→// archiveCheckpointsToParquet reads checkpoint files for a host directory,
   235→// converts them to Parquet rows. Returns the rows and filenames that were processed.
   236→func archiveCheckpointsToParquet(dir, cluster, host string, from int64) ([]ParquetMetricRow, []string, error) {
   237→	entries, err := os.ReadDir(dir)
   238→	if err != nil {
   239→		return nil, nil, err
   240→	}
   241→
   242→	files, err := findFiles(entries, from, false)
   243→	if err != nil {
   244→		return nil, nil, err
   245→	}
   246→
   247→	if len(files) == 0 {
   248→		return nil, nil, nil
   249→	}
   250→
   251→	// First pass: load checkpoints and estimate total rows for pre-allocation.
   252→	type loaded struct {
   253→		cf       *CheckpointFile
   254→		filename string
   255→	}
   256→	var checkpoints []loaded
   257→	totalEstimate := 0
   258→
   259→	for _, checkpoint := range files {
   260→		filename := filepath.Join(dir, checkpoint)
   261→		cf, err := loadCheckpointFileFromDisk(filename)
   262→		if err != nil {
   263→			cclog.Warnf("[METRICSTORE]> skipping unreadable checkpoint %s: %v", filename, err)
   264→			continue
   265→		}
   266→		totalEstimate += estimateRowCount(cf)
   267→		checkpoints = append(checkpoints, loaded{cf: cf, filename: checkpoint})
   268→	}
   269→
   270→	if len(checkpoints) == 0 {
   271→		return nil, nil, nil
   272→	}
   273→
   274→	rows := make([]ParquetMetricRow, 0, totalEstimate)
   275→	processedFiles := make([]string, 0, len(checkpoints))
   276→
   277→	for _, cp := range checkpoints {
   278→		rows = flattenCheckpointFile(cp.cf, cluster, host, "node", "", rows)
   279→		processedFiles = append(processedFiles, cp.filename)
   280→	}
   281→
   282→	return rows, processedFiles, nil
   283→}
   284→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package metricstore

import (
	"bufio"
	"encoding/binary"
	"encoding/json"
	"fmt"
	"os"
	"path/filepath"
	"sort"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	pq "github.com/parquet-go/parquet-go"
)

// ParquetMetricRow is the long-format schema for archived metric data.
// One row per (host, metric, scope, scope_id, timestamp) data point.
// Sorted by (cluster, hostname, metric, timestamp) for optimal compression.
type ParquetMetricRow struct {
	Cluster   string  `parquet:"cluster"`
	Hostname  string  `parquet:"hostname"`
	Metric    string  `parquet:"metric"`
	Scope     string  `parquet:"scope"`
	ScopeID   string  `parquet:"scope_id"`
	Timestamp int64   `parquet:"timestamp"`
	Frequency int64   `parquet:"frequency"`
	Value     float32 `parquet:"value"`
}

// flattenCheckpointFile recursively converts a CheckpointFile tree into Parquet rows.
// The scope path is built from the hierarchy: host level is "node", then child names
// map to scope/scope_id (e.g., "socket0" → scope="socket", scope_id="0").
func flattenCheckpointFile(cf *CheckpointFile, cluster, hostname, scope, scopeID string, rows []ParquetMetricRow) []ParquetMetricRow {
	for metricName, cm := range cf.Metrics {
		ts := cm.Start
		for _, v := range cm.Data {
			if !v.IsNaN() {
				rows = append(rows, ParquetMetricRow{
					Cluster:   cluster,
					Hostname:  hostname,
					Metric:    metricName,
					Scope:     scope,
					ScopeID:   scopeID,
					Timestamp: ts,
					Frequency: cm.Frequency,
					Value:     float32(v),
				})
			}
			ts += cm.Frequency
		}
	}

	for childName, childCf := range cf.Children {
		childScope, childScopeID := parseScopeFromName(childName)
		rows = flattenCheckpointFile(childCf, cluster, hostname, childScope, childScopeID, rows)
	}

	return rows
}

// parseScopeFromName infers scope and scope_id from a child level name.
// Examples: "socket0" → ("socket", "0"), "core12" → ("core", "12"),
// "a0" (accelerator) → ("accelerator", "0").
// If the name doesn't match known patterns, it's used as-is for scope with empty scope_id.
func parseScopeFromName(name string) (string, string) {
	prefixes := []struct {
		prefix string
		scope  string
	}{
		{"socket", "socket"},
		{"memoryDomain", "memoryDomain"},
		{"core", "core"},
		{"hwthread", "hwthread"},
		{"cpu", "hwthread"},
		{"accelerator", "accelerator"},
	}

	for _, p := range prefixes {
		if len(name) > len(p.prefix) && name[:len(p.prefix)] == p.prefix {
			id := name[len(p.prefix):]
			if len(id) > 0 && id[0] >= '0' && id[0] <= '9' {
				return p.scope, id
			}
		}
	}

	return name, ""
}

// parquetArchiveWriter supports incremental writes to a Parquet file.
// Each call to WriteHostRows writes one row group (typically one host's data),
// avoiding accumulation of all rows in memory.
type parquetArchiveWriter struct {
	writer *pq.GenericWriter[ParquetMetricRow]
	bw     *bufio.Writer
	f      *os.File
	count  int
}

// newParquetArchiveWriter creates a streaming Parquet writer with Zstd compression.
func newParquetArchiveWriter(filename string) (*parquetArchiveWriter, error) {
	if err := os.MkdirAll(filepath.Dir(filename), CheckpointDirPerms); err != nil {
		return nil, fmt.Errorf("creating archive directory: %w", err)
	}

	f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil {
		return nil, fmt.Errorf("creating parquet file: %w", err)
	}

	bw := bufio.NewWriterSize(f, 1<<20) // 1MB write buffer

	writer := pq.NewGenericWriter[ParquetMetricRow](bw,
		pq.Compression(&pq.Zstd),
	)

	return &parquetArchiveWriter{writer: writer, bw: bw, f: f}, nil
}

// WriteHostRows sorts rows by (metric, timestamp) in-place, writes them,
// and flushes to create a separate row group.
func (w *parquetArchiveWriter) WriteHostRows(rows []ParquetMetricRow) error {
	sort.Slice(rows, func(i, j int) bool {
		if rows[i].Metric != rows[j].Metric {
			return rows[i].Metric < rows[j].Metric
		}
		return rows[i].Timestamp < rows[j].Timestamp
	})

	if _, err := w.writer.Write(rows); err != nil {
		return fmt.Errorf("writing parquet rows: %w", err)
	}

	if err := w.writer.Flush(); err != nil {
		return fmt.Errorf("flushing parquet row group: %w", err)
	}

	w.count += len(rows)
	return nil
}

// Close finalises the Parquet file (footer, buffered I/O, file handle).
func (w *parquetArchiveWriter) Close() error {
	if err := w.writer.Close(); err != nil {
		w.f.Close()
		return fmt.Errorf("closing parquet writer: %w", err)
	}

	if err := w.bw.Flush(); err != nil {
		w.f.Close()
		return fmt.Errorf("flushing parquet file: %w", err)
	}

	return w.f.Close()
}

// sortParquetRows sorts rows by (metric, timestamp) in-place.
func sortParquetRows(rows []ParquetMetricRow) {
	sort.Slice(rows, func(i, j int) bool {
		if rows[i].Metric != rows[j].Metric {
			return rows[i].Metric < rows[j].Metric
		}
		return rows[i].Timestamp < rows[j].Timestamp
	})
}

// loadCheckpointFileFromDisk reads a JSON or binary checkpoint file and returns
// a CheckpointFile. Used by the Parquet archiver to read checkpoint data
// before converting it to Parquet format.
func loadCheckpointFileFromDisk(filename string) (*CheckpointFile, error) {
	f, err := os.Open(filename)
	if err != nil {
		return nil, err
	}
	defer f.Close()

	ext := filepath.Ext(filename)
	switch ext {
	case ".json":
		cf := &CheckpointFile{}
		br := bufio.NewReader(f)
		if err := json.NewDecoder(br).Decode(cf); err != nil {
			return nil, fmt.Errorf("decoding JSON checkpoint %s: %w", filename, err)
		}
		return cf, nil

	case ".bin":
		br := bufio.NewReader(f)
		var magic uint32
		if err := binary.Read(br, binary.LittleEndian, &magic); err != nil {
			return nil, fmt.Errorf("reading magic from %s: %w", filename, err)
		}
		if magic != snapFileMagic {
			return nil, fmt.Errorf("invalid snapshot magic in %s: 0x%08X", filename, magic)
		}
		var fileFrom, fileTo int64
		if err := binary.Read(br, binary.LittleEndian, &fileFrom); err != nil {
			return nil, fmt.Errorf("reading from-timestamp from %s: %w", filename, err)
		}
		if err := binary.Read(br, binary.LittleEndian, &fileTo); err != nil {
			return nil, fmt.Errorf("reading to-timestamp from %s: %w", filename, err)
		}
		cf, err := readBinaryLevel(br)
		if err != nil {
			return nil, fmt.Errorf("reading binary level from %s: %w", filename, err)
		}
		cf.From = fileFrom
		cf.To = fileTo
		return cf, nil

	default:
		return nil, fmt.Errorf("unsupported checkpoint extension: %s", ext)
	}
}

// estimateRowCount estimates the number of Parquet rows a CheckpointFile will produce.
// Used for pre-allocating the rows slice to avoid repeated append doubling.
func estimateRowCount(cf *CheckpointFile) int {
	n := 0
	for _, cm := range cf.Metrics {
		n += len(cm.Data)
	}
	for _, child := range cf.Children {
		n += estimateRowCount(child)
	}
	return n
}

// archiveCheckpointsToParquet reads checkpoint files for a host directory,
// converts them to Parquet rows. Returns the rows and filenames that were processed.
func archiveCheckpointsToParquet(dir, cluster, host string, from int64) ([]ParquetMetricRow, []string, error) {
	entries, err := os.ReadDir(dir)
	if err != nil {
		return nil, nil, err
	}

	files, err := findFiles(entries, from, false)
	if err != nil {
		return nil, nil, err
	}

	if len(files) == 0 {
		return nil, nil, nil
	}

	// First pass: load checkpoints and estimate total rows for pre-allocation.
	type loaded struct {
		cf       *CheckpointFile
		filename string
	}
	var checkpoints []loaded
	totalEstimate := 0

	for _, checkpoint := range files {
		filename := filepath.Join(dir, checkpoint)
		cf, err := loadCheckpointFileFromDisk(filename)
		if err != nil {
			cclog.Warnf("[METRICSTORE]> skipping unreadable checkpoint %s: %v", filename, err)
			continue
		}
		totalEstimate += estimateRowCount(cf)
		checkpoints = append(checkpoints, loaded{cf: cf, filename: checkpoint})
	}

	if len(checkpoints) == 0 {
		return nil, nil, nil
	}

	rows := make([]ParquetMetricRow, 0, totalEstimate)
	processedFiles := make([]string, 0, len(checkpoints))

	for _, cp := range checkpoints {
		rows = flattenCheckpointFile(cp.cf, cluster, host, "node", "", rows)
		processedFiles = append(processedFiles, cp.filename)
	}

	return rows, processedFiles, nil
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// This file implements checkpoint persistence for the in-memory metric store.
     7→//
     8→// Checkpoints enable graceful restarts by periodically saving in-memory metric
     9→// data to disk in JSON or binary format. The checkpoint system:
    10→//
    11→// Key Features:
    12→//   - Periodic background checkpointing via the Checkpointing() worker
    13→//   - Two format families: JSON (human-readable) and WAL+binary (compact, crash-safe)
    14→//   - Parallel checkpoint creation and loading using worker pools
    15→//   - Hierarchical file organization: checkpoint_dir/cluster/host/timestamp.{json|bin}
    16→//   - WAL file: checkpoint_dir/cluster/host/current.wal (append-only, per-entry)
    17→//   - Only saves unarchived data (archived data is already persisted elsewhere)
    18→//   - GC optimization during loading to prevent excessive heap growth
    19→//
    20→// Checkpoint Workflow:
    21→//  1. Init() loads checkpoints within retention window at startup
    22→//  2. Checkpointing() worker periodically saves new data
    23→//  3. Shutdown() writes final checkpoint before exit
    24→//
    25→// File Organization:
    26→//
    27→//	checkpoints/
    28→//	  cluster1/
    29→//	    host001/
    30→//	      1234567890.json  (JSON format: full subtree snapshot)
    31→//	      1234567890.bin   (binary format: full subtree snapshot)
    32→//	      current.wal      (WAL format: append-only per-entry log)
    33→//	    host002/
    34→//	      ...
    35→package metricstore
    36→
    37→import (
    38→	"bufio"
    39→	"context"
    40→	"encoding/json"
    41→	"errors"
    42→	"fmt"
    43→	"io/fs"
    44→	"os"
    45→	"path"
    46→	"path/filepath"
    47→	"sort"
    48→	"strconv"
    49→	"strings"
    50→	"sync"
    51→	"sync/atomic"
    52→	"time"
    53→
    54→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    55→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    56→)
    57→
    58→const (
    59→	CheckpointFilePerms = 0o644                    // File permissions for checkpoint files
    60→	CheckpointDirPerms  = 0o755                    // Directory permissions for checkpoint directories
    61→	GCTriggerInterval   = DefaultGCTriggerInterval // Interval for triggering GC during checkpoint loading
    62→)
    63→
    64→// CheckpointMetrics represents metric data in a checkpoint file.
    65→// Whenever the structure changes, update MarshalJSON as well!
    66→type CheckpointMetrics struct {
    67→	Data      []schema.Float `json:"data"`
    68→	Frequency int64          `json:"frequency"`
    69→	Start     int64          `json:"start"`
    70→}
    71→
    72→// CheckpointFile represents the hierarchical structure of a checkpoint file.
    73→// It mirrors the Level tree structure from the MemoryStore.
    74→type CheckpointFile struct {
    75→	Metrics  map[string]*CheckpointMetrics `json:"metrics"`
    76→	Children map[string]*CheckpointFile    `json:"children"`
    77→	From     int64                         `json:"from"`
    78→	To       int64                         `json:"to"`
    79→}
    80→
    81→// lastCheckpoint tracks the timestamp of the last checkpoint creation.
    82→var (
    83→	lastCheckpoint   time.Time
    84→	lastCheckpointMu sync.Mutex
    85→)
    86→
    87→// Checkpointing starts a background worker that periodically saves metric data to disk.
    88→//
    89→// Checkpoints are written every 12 hours (hardcoded).
    90→//
    91→// Format behaviour:
    92→//   - "json": Periodic checkpointing every checkpointInterval
    93→//   - "wal":  Periodic binary snapshots + WAL rotation every checkpointInterval
    94→func Checkpointing(wg *sync.WaitGroup, ctx context.Context) {
    95→	lastCheckpointMu.Lock()
    96→	lastCheckpoint = time.Now()
    97→	lastCheckpointMu.Unlock()
    98→
    99→	ms := GetMemoryStore()
   100→
   101→	wg.Go(func() {
   102→
   103→		d := 12 * time.Hour // default checkpoint interval
   104→		if Keys.CheckpointInterval != "" {
   105→			parsed, err := time.ParseDuration(Keys.CheckpointInterval)
   106→			if err != nil {
   107→				cclog.Errorf("[METRICSTORE]> invalid checkpoint-interval %q: %s, using default 12h", Keys.CheckpointInterval, err)
   108→			} else {
   109→				d = parsed
   110→			}
   111→		}
   112→
   113→		ticker := time.NewTicker(d)
   114→		defer ticker.Stop()
   115→
   116→		for {
   117→			select {
   118→			case <-ctx.Done():
   119→				return
   120→			case <-ticker.C:
   121→				lastCheckpointMu.Lock()
   122→				from := lastCheckpoint
   123→				lastCheckpointMu.Unlock()
   124→
   125→				now := time.Now()
   126→				cclog.Infof("[METRICSTORE]> start checkpointing (starting at %s)...", from.Format(time.RFC3339))
   127→
   128→				if Keys.Checkpoints.FileFormat == "wal" {
   129→					n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
   130→					if err != nil {
   131→						cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
   132→					} else {
   133→						cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
   134→						lastCheckpointMu.Lock()
   135→						lastCheckpoint = now
   136→						lastCheckpointMu.Unlock()
   137→						// Rotate WAL files for successfully checkpointed hosts.
   138→						RotateWALFiles(hostDirs)
   139→					}
   140→				} else {
   141→					n, err := ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
   142→					if err != nil {
   143→						cclog.Errorf("[METRICSTORE]> checkpointing failed: %s", err.Error())
   144→					} else {
   145→						cclog.Infof("[METRICSTORE]> done: %d checkpoint files created", n)
   146→						lastCheckpointMu.Lock()
   147→						lastCheckpoint = now
   148→						lastCheckpointMu.Unlock()
   149→					}
   150→				}
   151→			}
   152→		}
   153→	})
   154→}
   155→
   156→// MarshalJSON provides optimized JSON encoding for CheckpointMetrics.
   157→//
   158→// Since schema.Float has custom MarshalJSON, serializing []Float has significant overhead.
   159→// This method manually constructs JSON to avoid allocations and interface conversions.
   160→func (cm *CheckpointMetrics) MarshalJSON() ([]byte, error) {
   161→	buf := make([]byte, 0, 128+len(cm.Data)*8)
   162→	buf = append(buf, `{"frequency":`...)
   163→	buf = strconv.AppendInt(buf, cm.Frequency, 10)
   164→	buf = append(buf, `,"start":`...)
   165→	buf = strconv.AppendInt(buf, cm.Start, 10)
   166→	buf = append(buf, `,"data":[`...)
   167→	for i, x := range cm.Data {
   168→		if i != 0 {
   169→			buf = append(buf, ',')
   170→		}
   171→		if x.IsNaN() {
   172→			buf = append(buf, `null`...)
   173→		} else {
   174→			buf = strconv.AppendFloat(buf, float64(x), 'f', 1, 32)
   175→		}
   176→	}
   177→	buf = append(buf, `]}`...)
   178→	return buf, nil
   179→}
   180→
   181→// ToCheckpoint writes metric data to checkpoint files in parallel (JSON format).
   182→//
   183→// Metrics at root and cluster levels are skipped. One file per host is created.
   184→// Uses worker pool (Keys.NumWorkers) for parallel processing. Only locks one host
   185→// at a time, allowing concurrent writes/reads to other hosts.
   186→//
   187→// Returns the number of checkpoint files created and any errors encountered.
   188→func (m *MemoryStore) ToCheckpoint(dir string, from, to int64) (int, error) {
   189→	// Pre-calculate capacity by counting cluster/host pairs
   190→	m.root.lock.RLock()
   191→	totalHosts := 0
   192→	for _, l1 := range m.root.children {
   193→		l1.lock.RLock()
   194→		totalHosts += len(l1.children)
   195→		l1.lock.RUnlock()
   196→	}
   197→	m.root.lock.RUnlock()
   198→
   199→	levels := make([]*Level, 0, totalHosts)
   200→	selectors := make([][]string, 0, totalHosts)
   201→
   202→	m.root.lock.RLock()
   203→	for sel1, l1 := range m.root.children {
   204→		l1.lock.RLock()
   205→		for sel2, l2 := range l1.children {
   206→			levels = append(levels, l2)
   207→			selectors = append(selectors, []string{sel1, sel2})
   208→		}
   209→		l1.lock.RUnlock()
   210→	}
   211→	m.root.lock.RUnlock()
   212→
   213→	type workItem struct {
   214→		level    *Level
   215→		dir      string
   216→		selector []string
   217→	}
   218→
   219→	n, errs := int32(0), int32(0)
   220→
   221→	var wg sync.WaitGroup
   222→	wg.Add(Keys.NumWorkers)
   223→	work := make(chan workItem, Keys.NumWorkers*2)
   224→	for worker := 0; worker < Keys.NumWorkers; worker++ {
   225→		go func() {
   226→			defer wg.Done()
   227→
   228→			for workItem := range work {
   229→				if err := workItem.level.toCheckpoint(workItem.dir, from, to, m); err != nil {
   230→					if err == ErrNoNewArchiveData {
   231→						continue
   232→					}
   233→
   234→					cclog.Errorf("[METRICSTORE]> error while checkpointing %#v: %s", workItem.selector, err.Error())
   235→					atomic.AddInt32(&errs, 1)
   236→				} else {
   237→					atomic.AddInt32(&n, 1)
   238→				}
   239→			}
   240→		}()
   241→	}
   242→
   243→	for i := 0; i < len(levels); i++ {
   244→		dir := path.Join(dir, path.Join(selectors[i]...))
   245→		work <- workItem{
   246→			level:    levels[i],
   247→			dir:      dir,
   248→			selector: selectors[i],
   249→		}
   250→	}
   251→
   252→	close(work)
   253→	wg.Wait()
   254→
   255→	if errs > 0 {
   256→		return int(n), fmt.Errorf("[METRICSTORE]> %d errors happened while creating checkpoints (%d successes)", errs, n)
   257→	}
   258→	return int(n), nil
   259→}
   260→
   261→// toCheckpointFile recursively converts a Level tree to CheckpointFile structure.
   262→// Skips metrics that are already archived. Returns nil if no unarchived data exists.
   263→func (l *Level) toCheckpointFile(from, to int64, m *MemoryStore) (*CheckpointFile, error) {
   264→	l.lock.RLock()
   265→	defer l.lock.RUnlock()
   266→
   267→	retval := &CheckpointFile{
   268→		From:     from,
   269→		To:       to,
   270→		Metrics:  make(map[string]*CheckpointMetrics),
   271→		Children: make(map[string]*CheckpointFile),
   272→	}
   273→
   274→	for metric, minfo := range m.Metrics {
   275→		b := l.metrics[minfo.offset]
   276→		if b == nil {
   277→			continue
   278→		}
   279→
   280→		allArchived := true
   281→		b.iterFromTo(from, to, func(b *buffer) error {
   282→			if !b.archived {
   283→				allArchived = false
   284→				return fmt.Errorf("stop") // Early termination signal
   285→			}
   286→			return nil
   287→		})
   288→
   289→		if allArchived {
   290→			continue
   291→		}
   292→
   293→		data := make([]schema.Float, (to-from)/b.frequency+1)
   294→		data, start, end, err := b.read(from, to, data)
   295→		if err != nil {
   296→			return nil, err
   297→		}
   298→
   299→		for i := int((end - start) / b.frequency); i < len(data); i++ {
   300→			data[i] = schema.NaN
   301→		}
   302→
   303→		retval.Metrics[metric] = &CheckpointMetrics{
   304→			Frequency: b.frequency,
   305→			Start:     start,
   306→			Data:      data,
   307→		}
   308→	}
   309→
   310→	for name, child := range l.children {
   311→		val, err := child.toCheckpointFile(from, to, m)
   312→		if err != nil {
   313→			return nil, err
   314→		}
   315→
   316→		if val != nil {
   317→			retval.Children[name] = val
   318→		}
   319→	}
   320→
   321→	if len(retval.Children) == 0 && len(retval.Metrics) == 0 {
   322→		return nil, nil
   323→	}
   324→
   325→	return retval, nil
   326→}
   327→
   328→// toCheckpoint writes a Level's data to a JSON checkpoint file.
   329→// Creates directory if needed. Returns ErrNoNewArchiveData if nothing to save.
   330→func (l *Level) toCheckpoint(dir string, from, to int64, m *MemoryStore) error {
   331→	cf, err := l.toCheckpointFile(from, to, m)
   332→	if err != nil {
   333→		return err
   334→	}
   335→
   336→	if cf == nil {
   337→		return ErrNoNewArchiveData
   338→	}
   339→
   340→	filepath := path.Join(dir, fmt.Sprintf("%d.json", from))
   341→	f, err := os.OpenFile(filepath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
   342→	if err != nil && os.IsNotExist(err) {
   343→		err = os.MkdirAll(dir, CheckpointDirPerms)
   344→		if err == nil {
   345→			f, err = os.OpenFile(filepath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
   346→		}
   347→	}
   348→	if err != nil {
   349→		return err
   350→	}
   351→	defer f.Close()
   352→
   353→	bw := bufio.NewWriter(f)
   354→	if err = json.NewEncoder(bw).Encode(cf); err != nil {
   355→		return err
   356→	}
   357→
   358→	return bw.Flush()
   359→}
   360→
   361→// enqueueCheckpointHosts traverses checkpoint directory and enqueues cluster/host pairs.
   362→// Returns error if directory structure is invalid.
   363→func enqueueCheckpointHosts(dir string, work chan<- [2]string) error {
   364→	clustersDir, err := os.ReadDir(dir)
   365→	if err != nil {
   366→		return err
   367→	}
   368→
   369→	for _, clusterDir := range clustersDir {
   370→		if !clusterDir.IsDir() {
   371→			return errors.New("[METRICSTORE]> expected only directories at first level of checkpoints/ directory")
   372→		}
   373→
   374→		hostsDir, err := os.ReadDir(filepath.Join(dir, clusterDir.Name()))
   375→		if err != nil {
   376→			return err
   377→		}
   378→
   379→		for _, hostDir := range hostsDir {
   380→			if !hostDir.IsDir() {
   381→				return errors.New("[METRICSTORE]> expected only directories at second level of checkpoints/ directory")
   382→			}
   383→
   384→			work <- [2]string{clusterDir.Name(), hostDir.Name()}
   385→		}
   386→	}
   387→
   388→	return nil
   389→}
   390→
   391→// FromCheckpoint loads checkpoint files from disk into memory in parallel.
   392→//
   393→// Uses worker pool to load cluster/host combinations. Returns number of files
   394→// loaded and any errors.
   395→func (m *MemoryStore) FromCheckpoint(dir string, from int64) (int, error) {
   396→	var wg sync.WaitGroup
   397→	work := make(chan [2]string, Keys.NumWorkers*4)
   398→	n, errs := int32(0), int32(0)
   399→
   400→	wg.Add(Keys.NumWorkers)
   401→	for worker := 0; worker < Keys.NumWorkers; worker++ {
   402→		go func() {
   403→			defer wg.Done()
   404→			for host := range work {
   405→				lvl := m.root.findLevelOrCreate(host[:], len(m.Metrics))
   406→				nn, err := lvl.fromCheckpoint(m, filepath.Join(dir, host[0], host[1]), from)
   407→				if err != nil {
   408→					cclog.Errorf("[METRICSTORE]> error while loading checkpoints for %s/%s: %s", host[0], host[1], err.Error())
   409→					atomic.AddInt32(&errs, 1)
   410→				}
   411→				atomic.AddInt32(&n, int32(nn))
   412→			}
   413→		}()
   414→	}
   415→
   416→	err := enqueueCheckpointHosts(dir, work)
   417→	close(work)
   418→	wg.Wait()
   419→
   420→	if err != nil {
   421→		return int(n), err
   422→	}
   423→
   424→	if errs > 0 {
   425→		return int(n), fmt.Errorf("[METRICSTORE]> %d errors happened while creating checkpoints (%d successes)", errs, n)
   426→	}
   427→	return int(n), nil
   428→}
   429→
   430→// FromCheckpointFiles is the main entry point for loading checkpoints at startup.
   431→//
   432→// Creates checkpoint directory if it doesn't exist. This function must be called
   433→// before any writes or reads, and can only be called once.
   434→func (m *MemoryStore) FromCheckpointFiles(dir string, from int64) (int, error) {
   435→	if _, err := os.Stat(dir); os.IsNotExist(err) {
   436→		err := os.MkdirAll(dir, CheckpointDirPerms)
   437→		if err != nil {
   438→			cclog.Fatalf("[METRICSTORE]> Error creating directory: %#v\n", err)
   439→		}
   440→		cclog.Debugf("[METRICSTORE]> %#v Directory created successfully", dir)
   441→	}
   442→
   443→	return m.FromCheckpoint(dir, from)
   444→}
   445→
   446→func (l *Level) loadJSONFile(m *MemoryStore, f *os.File, from int64) error {
   447→	br := bufio.NewReader(f)
   448→	cf := &CheckpointFile{}
   449→	if err := json.NewDecoder(br).Decode(cf); err != nil {
   450→		return err
   451→	}
   452→
   453→	if cf.To != 0 && cf.To < from {
   454→		return nil
   455→	}
   456→
   457→	if err := l.loadFile(cf, m); err != nil {
   458→		return err
   459→	}
   460→
   461→	return nil
   462→}
   463→
   464→func (l *Level) loadFile(cf *CheckpointFile, m *MemoryStore) error {
   465→	for name, metric := range cf.Metrics {
   466→		n := len(metric.Data)
   467→		b := &buffer{
   468→			frequency: metric.Frequency,
   469→			start:     metric.Start,
   470→			data:      metric.Data[0:n:n],
   471→			prev:      nil,
   472→			next:      nil,
   473→			archived:  true,
   474→		}
   475→
   476→		minfo, ok := m.Metrics[name]
   477→		if !ok {
   478→			continue
   479→		}
   480→
   481→		prev := l.metrics[minfo.offset]
   482→		if prev == nil {
   483→			l.metrics[minfo.offset] = b
   484→		} else {
   485→			if prev.start > b.start {
   486→				return fmt.Errorf("[METRICSTORE]> buffer start time %d is before previous buffer start %d", b.start, prev.start)
   487→			}
   488→
   489→			b.prev = prev
   490→			prev.next = b
   491→		}
   492→		l.metrics[minfo.offset] = b
   493→	}
   494→
   495→	if len(cf.Children) > 0 && l.children == nil {
   496→		l.children = make(map[string]*Level)
   497→	}
   498→
   499→	for sel, childCf := range cf.Children {
   500→		child, ok := l.children[sel]
   501→		if !ok {
   502→			child = &Level{
   503→				metrics:  make([]*buffer, len(m.Metrics)),
   504→				children: nil,
   505→			}
   506→			l.children[sel] = child
   507→		}
   508→
   509→		if err := child.loadFile(childCf, m); err != nil {
   510→			return err
   511→		}
   512→	}
   513→
   514→	return nil
   515→}
   516→
   517→// fromCheckpoint loads all checkpoint files (JSON, binary snapshot, WAL) for a
   518→// single host directory. Snapshot files are loaded first (sorted by timestamp),
   519→// then current.wal is replayed on top.
   520→func (l *Level) fromCheckpoint(m *MemoryStore, dir string, from int64) (int, error) {
   521→	direntries, err := os.ReadDir(dir)
   522→	if err != nil {
   523→		if os.IsNotExist(err) {
   524→			return 0, nil
   525→		}
   526→		return 0, err
   527→	}
   528→
   529→	allFiles := make([]fs.DirEntry, 0)
   530→	var walEntry fs.DirEntry
   531→	filesLoaded := 0
   532→
   533→	for _, e := range direntries {
   534→		if e.IsDir() {
   535→			// Legacy: skip subdirectories (only used by old Avro format).
   536→			// These are ignored; their data is not loaded.
   537→			cclog.Debugf("[METRICSTORE]> skipping subdirectory %s in checkpoint dir %s", e.Name(), dir)
   538→			continue
   539→		}
   540→
   541→		name := e.Name()
   542→		if strings.HasSuffix(name, ".json") || strings.HasSuffix(name, ".bin") {
   543→			allFiles = append(allFiles, e)
   544→		} else if name == "current.wal" {
   545→			walEntry = e
   546→		}
   547→		// Silently ignore other files (e.g., .tmp, .bin.tmp from interrupted writes).
   548→	}
   549→
   550→	files, err := findFiles(allFiles, from, true)
   551→	if err != nil {
   552→		return filesLoaded, err
   553→	}
   554→
   555→	loaders := map[string]func(*MemoryStore, *os.File, int64) error{
   556→		".json": l.loadJSONFile,
   557→		".bin":  l.loadBinaryFile,
   558→	}
   559→
   560→	for _, filename := range files {
   561→		ext := filepath.Ext(filename)
   562→		loader := loaders[ext]
   563→		if loader == nil {
   564→			cclog.Warnf("[METRICSTORE]> unknown extension for checkpoint file %s", filename)
   565→			continue
   566→		}
   567→
   568→		err := func() error {
   569→			f, err := os.Open(path.Join(dir, filename))
   570→			if err != nil {
   571→				return err
   572→			}
   573→			defer f.Close()
   574→			return loader(m, f, from)
   575→		}()
   576→		if err != nil {
   577→			return filesLoaded, err
   578→		}
   579→		filesLoaded++
   580→	}
   581→
   582→	// Replay WAL after all snapshot files so it fills in data since the last snapshot.
   583→	if walEntry != nil {
   584→		err := func() error {
   585→			f, err := os.Open(path.Join(dir, walEntry.Name()))
   586→			if err != nil {
   587→				return err
   588→			}
   589→			defer f.Close()
   590→			return l.loadWALFile(m, f, from)
   591→		}()
   592→		if err != nil {
   593→			// WAL errors are non-fatal: the snapshot already loaded the bulk of data.
   594→			cclog.Warnf("[METRICSTORE]> WAL replay error for %s: %v (data since last snapshot may be missing)", dir, err)
   595→		} else {
   596→			filesLoaded++
   597→		}
   598→	}
   599→
   600→	return filesLoaded, nil
   601→}
   602→
   603→// parseTimestampFromFilename extracts a Unix timestamp from a checkpoint filename.
   604→// Supports ".json" (format: "<ts>.json") and ".bin" (format: "<ts>.bin").
   605→func parseTimestampFromFilename(name string) (int64, error) {
   606→	switch {
   607→	case strings.HasSuffix(name, ".json"):
   608→		return strconv.ParseInt(name[:len(name)-5], 10, 64)
   609→	case strings.HasSuffix(name, ".bin"):
   610→		return strconv.ParseInt(name[:len(name)-4], 10, 64)
   611→	default:
   612→		return 0, fmt.Errorf("unknown checkpoint extension for file %q", name)
   613→	}
   614→}
   615→
   616→// findFiles returns filenames from direntries whose timestamps satisfy the filter.
   617→// If findMoreRecentFiles is true, returns files with timestamps >= t (plus the
   618→// last file before t if t falls between two files).
   619→func findFiles(direntries []fs.DirEntry, t int64, findMoreRecentFiles bool) ([]string, error) {
   620→	nums := map[string]int64{}
   621→	for _, e := range direntries {
   622→		name := e.Name()
   623→		if !strings.HasSuffix(name, ".json") && !strings.HasSuffix(name, ".bin") {
   624→			continue
   625→		}
   626→
   627→		ts, err := parseTimestampFromFilename(name)
   628→		if err != nil {
   629→			return nil, err
   630→		}
   631→		nums[name] = ts
   632→	}
   633→
   634→	sort.Slice(direntries, func(i, j int) bool {
   635→		a, b := direntries[i], direntries[j]
   636→		return nums[a.Name()] < nums[b.Name()]
   637→	})
   638→
   639→	if len(nums) == 0 {
   640→		return nil, nil
   641→	}
   642→
   643→	filenames := make([]string, 0)
   644→
   645→	for i, e := range direntries {
   646→		ts1 := nums[e.Name()]
   647→
   648→		if findMoreRecentFiles && t <= ts1 {
   649→			filenames = append(filenames, e.Name())
   650→		} else if !findMoreRecentFiles && ts1 <= t && ts1 != 0 {
   651→			filenames = append(filenames, e.Name())
   652→		}
   653→
   654→		if i == len(direntries)-1 {
   655→			continue
   656→		}
   657→
   658→		enext := direntries[i+1]
   659→		ts2 := nums[enext.Name()]
   660→
   661→		if findMoreRecentFiles {
   662→			if ts1 < t && t < ts2 {
   663→				filenames = append(filenames, e.Name())
   664→			}
   665→		}
   666→	}
   667→
   668→	return filenames, nil
   669→}
   670→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// This file implements checkpoint persistence for the in-memory metric store.
//
// Checkpoints enable graceful restarts by periodically saving in-memory metric
// data to disk in JSON or binary format. The checkpoint system:
//
// Key Features:
//   - Periodic background checkpointing via the Checkpointing() worker
//   - Two format families: JSON (human-readable) and WAL+binary (compact, crash-safe)
//   - Parallel checkpoint creation and loading using worker pools
//   - Hierarchical file organization: checkpoint_dir/cluster/host/timestamp.{json|bin}
//   - WAL file: checkpoint_dir/cluster/host/current.wal (append-only, per-entry)
//   - Only saves unarchived data (archived data is already persisted elsewhere)
//   - GC optimization during loading to prevent excessive heap growth
//
// Checkpoint Workflow:
//  1. Init() loads checkpoints within retention window at startup
//  2. Checkpointing() worker periodically saves new data
//  3. Shutdown() writes final checkpoint before exit
//
// File Organization:
//
//	checkpoints/
//	  cluster1/
//	    host001/
//	      1234567890.json  (JSON format: full subtree snapshot)
//	      1234567890.bin   (binary format: full subtree snapshot)
//	      current.wal      (WAL format: append-only per-entry log)
//	    host002/
//	      ...
package metricstore

import (
	"bufio"
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"io/fs"
	"os"
	"path"
	"path/filepath"
	"sort"
	"strconv"
	"strings"
	"sync"
	"sync/atomic"
	"time"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

const (
	CheckpointFilePerms = 0o644                    // File permissions for checkpoint files
	CheckpointDirPerms  = 0o755                    // Directory permissions for checkpoint directories
	GCTriggerInterval   = DefaultGCTriggerInterval // Interval for triggering GC during checkpoint loading
)

// CheckpointMetrics represents metric data in a checkpoint file.
// Whenever the structure changes, update MarshalJSON as well!
type CheckpointMetrics struct {
	Data      []schema.Float `json:"data"`
	Frequency int64          `json:"frequency"`
	Start     int64          `json:"start"`
}

// CheckpointFile represents the hierarchical structure of a checkpoint file.
// It mirrors the Level tree structure from the MemoryStore.
type CheckpointFile struct {
	Metrics  map[string]*CheckpointMetrics `json:"metrics"`
	Children map[string]*CheckpointFile    `json:"children"`
	From     int64                         `json:"from"`
	To       int64                         `json:"to"`
}

// lastCheckpoint tracks the timestamp of the last checkpoint creation.
var (
	lastCheckpoint   time.Time
	lastCheckpointMu sync.Mutex
)

// Checkpointing starts a background worker that periodically saves metric data to disk.
//
// Checkpoints are written every 12 hours (hardcoded).
//
// Format behaviour:
//   - "json": Periodic checkpointing every checkpointInterval
//   - "wal":  Periodic binary snapshots + WAL rotation every checkpointInterval
func Checkpointing(wg *sync.WaitGroup, ctx context.Context) {
	lastCheckpointMu.Lock()
	lastCheckpoint = time.Now()
	lastCheckpointMu.Unlock()

	ms := GetMemoryStore()

	wg.Go(func() {

		d := 12 * time.Hour // default checkpoint interval
		if Keys.CheckpointInterval != "" {
			parsed, err := time.ParseDuration(Keys.CheckpointInterval)
			if err != nil {
				cclog.Errorf("[METRICSTORE]> invalid checkpoint-interval %q: %s, using default 12h", Keys.CheckpointInterval, err)
			} else {
				d = parsed
			}
		}

		ticker := time.NewTicker(d)
		defer ticker.Stop()

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				lastCheckpointMu.Lock()
				from := lastCheckpoint
				lastCheckpointMu.Unlock()

				now := time.Now()
				cclog.Infof("[METRICSTORE]> start checkpointing (starting at %s)...", from.Format(time.RFC3339))

				if Keys.Checkpoints.FileFormat == "wal" {
					n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
					if err != nil {
						cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
					} else {
						cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
						lastCheckpointMu.Lock()
						lastCheckpoint = now
						lastCheckpointMu.Unlock()
						// Rotate WAL files for successfully checkpointed hosts.
						RotateWALFiles(hostDirs)
					}
				} else {
					n, err := ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
					if err != nil {
						cclog.Errorf("[METRICSTORE]> checkpointing failed: %s", err.Error())
					} else {
						cclog.Infof("[METRICSTORE]> done: %d checkpoint files created", n)
						lastCheckpointMu.Lock()
						lastCheckpoint = now
						lastCheckpointMu.Unlock()
					}
				}
			}
		}
	})
}

// MarshalJSON provides optimized JSON encoding for CheckpointMetrics.
//
// Since schema.Float has custom MarshalJSON, serializing []Float has significant overhead.
// This method manually constructs JSON to avoid allocations and interface conversions.
func (cm *CheckpointMetrics) MarshalJSON() ([]byte, error) {
	buf := make([]byte, 0, 128+len(cm.Data)*8)
	buf = append(buf, `{"frequency":`...)
	buf = strconv.AppendInt(buf, cm.Frequency, 10)
	buf = append(buf, `,"start":`...)
	buf = strconv.AppendInt(buf, cm.Start, 10)
	buf = append(buf, `,"data":[`...)
	for i, x := range cm.Data {
		if i != 0 {
			buf = append(buf, ',')
		}
		if x.IsNaN() {
			buf = append(buf, `null`...)
		} else {
			buf = strconv.AppendFloat(buf, float64(x), 'f', 1, 32)
		}
	}
	buf = append(buf, `]}`...)
	return buf, nil
}

// ToCheckpoint writes metric data to checkpoint files in parallel (JSON format).
//
// Metrics at root and cluster levels are skipped. One file per host is created.
// Uses worker pool (Keys.NumWorkers) for parallel processing. Only locks one host
// at a time, allowing concurrent writes/reads to other hosts.
//
// Returns the number of checkpoint files created and any errors encountered.
func (m *MemoryStore) ToCheckpoint(dir string, from, to int64) (int, error) {
	// Pre-calculate capacity by counting cluster/host pairs
	m.root.lock.RLock()
	totalHosts := 0
	for _, l1 := range m.root.children {
		l1.lock.RLock()
		totalHosts += len(l1.children)
		l1.lock.RUnlock()
	}
	m.root.lock.RUnlock()

	levels := make([]*Level, 0, totalHosts)
	selectors := make([][]string, 0, totalHosts)

	m.root.lock.RLock()
	for sel1, l1 := range m.root.children {
		l1.lock.RLock()
		for sel2, l2 := range l1.children {
			levels = append(levels, l2)
			selectors = append(selectors, []string{sel1, sel2})
		}
		l1.lock.RUnlock()
	}
	m.root.lock.RUnlock()

	type workItem struct {
		level    *Level
		dir      string
		selector []string
	}

	n, errs := int32(0), int32(0)

	var wg sync.WaitGroup
	wg.Add(Keys.NumWorkers)
	work := make(chan workItem, Keys.NumWorkers*2)
	for worker := 0; worker < Keys.NumWorkers; worker++ {
		go func() {
			defer wg.Done()

			for workItem := range work {
				if err := workItem.level.toCheckpoint(workItem.dir, from, to, m); err != nil {
					if err == ErrNoNewArchiveData {
						continue
					}

					cclog.Errorf("[METRICSTORE]> error while checkpointing %#v: %s", workItem.selector, err.Error())
					atomic.AddInt32(&errs, 1)
				} else {
					atomic.AddInt32(&n, 1)
				}
			}
		}()
	}

	for i := 0; i < len(levels); i++ {
		dir := path.Join(dir, path.Join(selectors[i]...))
		work <- workItem{
			level:    levels[i],
			dir:      dir,
			selector: selectors[i],
		}
	}

	close(work)
	wg.Wait()

	if errs > 0 {
		return int(n), fmt.Errorf("[METRICSTORE]> %d errors happened while creating checkpoints (%d successes)", errs, n)
	}
	return int(n), nil
}

// toCheckpointFile recursively converts a Level tree to CheckpointFile structure.
// Skips metrics that are already archived. Returns nil if no unarchived data exists.
func (l *Level) toCheckpointFile(from, to int64, m *MemoryStore) (*CheckpointFile, error) {
	l.lock.RLock()
	defer l.lock.RUnlock()

	retval := &CheckpointFile{
		From:     from,
		To:       to,
		Metrics:  make(map[string]*CheckpointMetrics),
		Children: make(map[string]*CheckpointFile),
	}

	for metric, minfo := range m.Metrics {
		b := l.metrics[minfo.offset]
		if b == nil {
			continue
		}

		allArchived := true
		b.iterFromTo(from, to, func(b *buffer) error {
			if !b.archived {
				allArchived = false
				return fmt.Errorf("stop") // Early termination signal
			}
			return nil
		})

		if allArchived {
			continue
		}

		data := make([]schema.Float, (to-from)/b.frequency+1)
		data, start, end, err := b.read(from, to, data)
		if err != nil {
			return nil, err
		}

		for i := int((end - start) / b.frequency); i < len(data); i++ {
			data[i] = schema.NaN
		}

		retval.Metrics[metric] = &CheckpointMetrics{
			Frequency: b.frequency,
			Start:     start,
			Data:      data,
		}
	}

	for name, child := range l.children {
		val, err := child.toCheckpointFile(from, to, m)
		if err != nil {
			return nil, err
		}

		if val != nil {
			retval.Children[name] = val
		}
	}

	if len(retval.Children) == 0 && len(retval.Metrics) == 0 {
		return nil, nil
	}

	return retval, nil
}

// toCheckpoint writes a Level's data to a JSON checkpoint file.
// Creates directory if needed. Returns ErrNoNewArchiveData if nothing to save.
func (l *Level) toCheckpoint(dir string, from, to int64, m *MemoryStore) error {
	cf, err := l.toCheckpointFile(from, to, m)
	if err != nil {
		return err
	}

	if cf == nil {
		return ErrNoNewArchiveData
	}

	filepath := path.Join(dir, fmt.Sprintf("%d.json", from))
	f, err := os.OpenFile(filepath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil && os.IsNotExist(err) {
		err = os.MkdirAll(dir, CheckpointDirPerms)
		if err == nil {
			f, err = os.OpenFile(filepath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
		}
	}
	if err != nil {
		return err
	}
	defer f.Close()

	bw := bufio.NewWriter(f)
	if err = json.NewEncoder(bw).Encode(cf); err != nil {
		return err
	}

	return bw.Flush()
}

// enqueueCheckpointHosts traverses checkpoint directory and enqueues cluster/host pairs.
// Returns error if directory structure is invalid.
func enqueueCheckpointHosts(dir string, work chan<- [2]string) error {
	clustersDir, err := os.ReadDir(dir)
	if err != nil {
		return err
	}

	for _, clusterDir := range clustersDir {
		if !clusterDir.IsDir() {
			return errors.New("[METRICSTORE]> expected only directories at first level of checkpoints/ directory")
		}

		hostsDir, err := os.ReadDir(filepath.Join(dir, clusterDir.Name()))
		if err != nil {
			return err
		}

		for _, hostDir := range hostsDir {
			if !hostDir.IsDir() {
				return errors.New("[METRICSTORE]> expected only directories at second level of checkpoints/ directory")
			}

			work <- [2]string{clusterDir.Name(), hostDir.Name()}
		}
	}

	return nil
}

// FromCheckpoint loads checkpoint files from disk into memory in parallel.
//
// Uses worker pool to load cluster/host combinations. Returns number of files
// loaded and any errors.
func (m *MemoryStore) FromCheckpoint(dir string, from int64) (int, error) {
	var wg sync.WaitGroup
	work := make(chan [2]string, Keys.NumWorkers*4)
	n, errs := int32(0), int32(0)

	wg.Add(Keys.NumWorkers)
	for worker := 0; worker < Keys.NumWorkers; worker++ {
		go func() {
			defer wg.Done()
			for host := range work {
				lvl := m.root.findLevelOrCreate(host[:], len(m.Metrics))
				nn, err := lvl.fromCheckpoint(m, filepath.Join(dir, host[0], host[1]), from)
				if err != nil {
					cclog.Errorf("[METRICSTORE]> error while loading checkpoints for %s/%s: %s", host[0], host[1], err.Error())
					atomic.AddInt32(&errs, 1)
				}
				atomic.AddInt32(&n, int32(nn))
			}
		}()
	}

	err := enqueueCheckpointHosts(dir, work)
	close(work)
	wg.Wait()

	if err != nil {
		return int(n), err
	}

	if errs > 0 {
		return int(n), fmt.Errorf("[METRICSTORE]> %d errors happened while creating checkpoints (%d successes)", errs, n)
	}
	return int(n), nil
}

// FromCheckpointFiles is the main entry point for loading checkpoints at startup.
//
// Creates checkpoint directory if it doesn't exist. This function must be called
// before any writes or reads, and can only be called once.
func (m *MemoryStore) FromCheckpointFiles(dir string, from int64) (int, error) {
	if _, err := os.Stat(dir); os.IsNotExist(err) {
		err := os.MkdirAll(dir, CheckpointDirPerms)
		if err != nil {
			cclog.Fatalf("[METRICSTORE]> Error creating directory: %#v\n", err)
		}
		cclog.Debugf("[METRICSTORE]> %#v Directory created successfully", dir)
	}

	return m.FromCheckpoint(dir, from)
}

func (l *Level) loadJSONFile(m *MemoryStore, f *os.File, from int64) error {
	br := bufio.NewReader(f)
	cf := &CheckpointFile{}
	if err := json.NewDecoder(br).Decode(cf); err != nil {
		return err
	}

	if cf.To != 0 && cf.To < from {
		return nil
	}

	if err := l.loadFile(cf, m); err != nil {
		return err
	}

	return nil
}

func (l *Level) loadFile(cf *CheckpointFile, m *MemoryStore) error {
	for name, metric := range cf.Metrics {
		n := len(metric.Data)
		b := &buffer{
			frequency: metric.Frequency,
			start:     metric.Start,
			data:      metric.Data[0:n:n],
			prev:      nil,
			next:      nil,
			archived:  true,
		}

		minfo, ok := m.Metrics[name]
		if !ok {
			continue
		}

		prev := l.metrics[minfo.offset]
		if prev == nil {
			l.metrics[minfo.offset] = b
		} else {
			if prev.start > b.start {
				return fmt.Errorf("[METRICSTORE]> buffer start time %d is before previous buffer start %d", b.start, prev.start)
			}

			b.prev = prev
			prev.next = b
		}
		l.metrics[minfo.offset] = b
	}

	if len(cf.Children) > 0 && l.children == nil {
		l.children = make(map[string]*Level)
	}

	for sel, childCf := range cf.Children {
		child, ok := l.children[sel]
		if !ok {
			child = &Level{
				metrics:  make([]*buffer, len(m.Metrics)),
				children: nil,
			}
			l.children[sel] = child
		}

		if err := child.loadFile(childCf, m); err != nil {
			return err
		}
	}

	return nil
}

// fromCheckpoint loads all checkpoint files (JSON, binary snapshot, WAL) for a
// single host directory. Snapshot files are loaded first (sorted by timestamp),
// then current.wal is replayed on top.
func (l *Level) fromCheckpoint(m *MemoryStore, dir string, from int64) (int, error) {
	direntries, err := os.ReadDir(dir)
	if err != nil {
		if os.IsNotExist(err) {
			return 0, nil
		}
		return 0, err
	}

	allFiles := make([]fs.DirEntry, 0)
	var walEntry fs.DirEntry
	filesLoaded := 0

	for _, e := range direntries {
		if e.IsDir() {
			// Legacy: skip subdirectories (only used by old Avro format).
			// These are ignored; their data is not loaded.
			cclog.Debugf("[METRICSTORE]> skipping subdirectory %s in checkpoint dir %s", e.Name(), dir)
			continue
		}

		name := e.Name()
		if strings.HasSuffix(name, ".json") || strings.HasSuffix(name, ".bin") {
			allFiles = append(allFiles, e)
		} else if name == "current.wal" {
			walEntry = e
		}
		// Silently ignore other files (e.g., .tmp, .bin.tmp from interrupted writes).
	}

	files, err := findFiles(allFiles, from, true)
	if err != nil {
		return filesLoaded, err
	}

	loaders := map[string]func(*MemoryStore, *os.File, int64) error{
		".json": l.loadJSONFile,
		".bin":  l.loadBinaryFile,
	}

	for _, filename := range files {
		ext := filepath.Ext(filename)
		loader := loaders[ext]
		if loader == nil {
			cclog.Warnf("[METRICSTORE]> unknown extension for checkpoint file %s", filename)
			continue
		}

		err := func() error {
			f, err := os.Open(path.Join(dir, filename))
			if err != nil {
				return err
			}
			defer f.Close()
			return loader(m, f, from)
		}()
		if err != nil {
			return filesLoaded, err
		}
		filesLoaded++
	}

	// Replay WAL after all snapshot files so it fills in data since the last snapshot.
	if walEntry != nil {
		err := func() error {
			f, err := os.Open(path.Join(dir, walEntry.Name()))
			if err != nil {
				return err
			}
			defer f.Close()
			return l.loadWALFile(m, f, from)
		}()
		if err != nil {
			// WAL errors are non-fatal: the snapshot already loaded the bulk of data.
			cclog.Warnf("[METRICSTORE]> WAL replay error for %s: %v (data since last snapshot may be missing)", dir, err)
		} else {
			filesLoaded++
		}
	}

	return filesLoaded, nil
}

// parseTimestampFromFilename extracts a Unix timestamp from a checkpoint filename.
// Supports ".json" (format: "<ts>.json") and ".bin" (format: "<ts>.bin").
func parseTimestampFromFilename(name string) (int64, error) {
	switch {
	case strings.HasSuffix(name, ".json"):
		return strconv.ParseInt(name[:len(name)-5], 10, 64)
	case strings.HasSuffix(name, ".bin"):
		return strconv.ParseInt(name[:len(name)-4], 10, 64)
	default:
		return 0, fmt.Errorf("unknown checkpoint extension for file %q", name)
	}
}

// findFiles returns filenames from direntries whose timestamps satisfy the filter.
// If findMoreRecentFiles is true, returns files with timestamps >= t (plus the
// last file before t if t falls between two files).
func findFiles(direntries []fs.DirEntry, t int64, findMoreRecentFiles bool) ([]string, error) {
	nums := map[string]int64{}
	for _, e := range direntries {
		name := e.Name()
		if !strings.HasSuffix(name, ".json") && !strings.HasSuffix(name, ".bin") {
			continue
		}

		ts, err := parseTimestampFromFilename(name)
		if err != nil {
			return nil, err
		}
		nums[name] = ts
	}

	sort.Slice(direntries, func(i, j int) bool {
		a, b := direntries[i], direntries[j]
		return nums[a.Name()] < nums[b.Name()]
	})

	if len(nums) == 0 {
		return nil, nil
	}

	filenames := make([]string, 0)

	for i, e := range direntries {
		ts1 := nums[e.Name()]

		if findMoreRecentFiles && t <= ts1 {
			filenames = append(filenames, e.Name())
		} else if !findMoreRecentFiles && ts1 <= t && ts1 != 0 {
			filenames = append(filenames, e.Name())
		}

		if i == len(direntries)-1 {
			continue
		}

		enext := direntries[i+1]
		ts2 := nums[enext.Name()]

		if findMoreRecentFiles {
			if ts1 < t && t < ts2 {
				filenames = append(filenames, e.Name())
			}
		}
	}

	return filenames, nil
}


Now I have a clear picture. Let me implement all the changes.

**Step 1**: Reduce results channel buffer and restructure `archiveCheckpoints` to stream `CheckpointFile` instead of rows.

**Step 2**: Add `WriteCheckpointFileDirect` to the parquet writer.

**Step 3**: Remove dead code.