Implement the following plan:

# Checkpoint Correctness Analysis (Updated Code)

## Context

Analyzing data preservation when the application shuts down shortly after a periodic checkpoint. The shutdown checkpoint covers only a short duration (minutes), not the full retention period. On next startup, the system incorrectly assumes old checkpoint files will remain available indefinitely.

---

## Primary Bug: False assumption about checkpoint coverage after restart

### The Problem

The shutdown checkpoint covers `[lastCheckpoint, shutdown_time]` — potentially just a few minutes. The rest of the retention window depends on older checkpoint files. After restart, the system sets `lastCheckpoint = time.Now()` and marks all loaded data as `archived: true`, effectively "forgetting" that old checkpoint files are the **only** persistent copy of most of the retention window. New periodic checkpoints only cover data from restart onward — they never re-persist the loaded data. When `CleanUp` eventually deletes the old files, data within the retention window is permanently lost from disk.

### Root Cause

**`checkpoint.go:95-96`** — `lastCheckpoint` set to restart time, not to the start of loaded data:
```go
lastCheckpoint = time.Now()  // Should be restoreFrom instead
```

**`checkpoint.go:473`** — Loaded buffers marked as archived, preventing re-checkpointing:
```go
archived: true  // Should be false so data gets re-persisted
```

**`checkpoint.go:280-291`** — `toCheckpointFile` skips all-archived metrics:
```go
if allArchived {
    continue  // Loaded data never re-checkpointed
}
```

### Timeline Showing the Gap

```
Retention = 48h, CheckpointInterval = 12h

T-12h:  Periodic checkpoint → {T-12h}.bin [T-12h, T]
T:      Shutdown checkpoint → {T}.bin [T, T+5min]   ← SHORT
T+1h:   Restart. Loads both files. lastCheckpoint = T+1h.
         All loaded buffers: archived=true

T+13h:  Checkpoint → {T+1h}.bin [T+1h, T+13h]  ← only NEW data
         Old data [T-12h, T+5min]? SKIPPED (archived=true, from=T+1h)

T+49h:  CleanUp deletes {T-12h}.bin, {T}.bin, {T+1h}.bin
         Data [T-12h, T+13h] now ONLY in memory.
         CRASH → data permanently lost. 12h gap in retention window.
```

### Fix

**`pkg/metricstore/metricstore.go`** — Pass `restoreFrom` to `Checkpointing()` so `lastCheckpoint` is initialized to the start of loaded data, not `time.Now()`.

**`pkg/metricstore/checkpoint.go`**:
1. `Checkpointing()`: Accept `restoreFrom` parameter, set `lastCheckpoint = restoreFrom`
2. `loadFile()`: Set `archived: false` on loaded buffers (or remove the archived check from `toCheckpointFile`)

This ensures the first periodic checkpoint after restart re-persists the entire loaded retention window, creating new checkpoint files that replace the old ones before CleanUp deletes them.

**Trade-off**: The first checkpoint after restart will be larger (covers full retention window instead of just 12h). This is acceptable — it's a one-time cost for data safety.

---

## Secondary Bugs

### Bug 2: Partial checkpoint failure doesn't update `lastCheckpoint` (Medium)

**File**: `checkpoint.go:128-139`

When `ToCheckpointWAL` returns an error (some hosts failed), `lastCheckpoint` is not updated. Next checkpoint re-writes ALL hosts from the old `from` time. WAL deletion per-host inside workers (line 699-703) partially mitigates WAL growth, but redundant snapshot writes remain.

### Bug 3: JSON format non-atomic writes (Low)

**File**: `checkpoint.go:340-358`

JSON checkpoints write directly to final filename without temp+rename. Binary format correctly uses `.bin.tmp` + `os.Rename`. Crash during JSON write can corrupt the file.

### Bug 4: CleanUp uses `from` timestamp for age, not `to` (Low)

**File**: `archive.go:66` via `findFiles(entries, from, false)`

Files named `{from}.bin` are deleted when `from <= now - retention`. But a file covering [T-12h, T] contains data up to T. If T is still within retention but T-12h is not, the file is deleted prematurely. This widens the gap from Bug 1.

---

## Resolved Issues (fixed in current code vs. initial analysis)

| Issue | Fix |
|-------|-----|
| WAL goroutines not awaited during Shutdown | `WaitForWALStagingDrain()` at `metricstore.go:294` |
| WAL file deletion separated from checkpoint success | Per-host inside `ToCheckpointWAL` workers (`walCheckpoint.go:699-703`) |
| Send-on-closed-channel panic | `walShuttingDown` atomic bool (`walCheckpoint.go:100`) |
| `RotateWALFiles` could deadlock | Non-blocking sends + 30s timeout (`walCheckpoint.go:338-353`) |

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-tmp-cc-backend/c8e0adf0-46c9-4661-bee9-670810ce1e34.jsonl


Let me read the relevant files first.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// This file implements checkpoint persistence for the in-memory metric store.
     7→//
     8→// Checkpoints enable graceful restarts by periodically saving in-memory metric
     9→// data to disk in JSON or binary format. The checkpoint system:
    10→//
    11→// Key Features:
    12→//   - Periodic background checkpointing via the Checkpointing() worker
    13→//   - Two format families: JSON (human-readable) and WAL+binary (compact, crash-safe)
    14→//   - Parallel checkpoint creation and loading using worker pools
    15→//   - Hierarchical file organization: checkpoint_dir/cluster/host/timestamp.{json|bin}
    16→//   - WAL file: checkpoint_dir/cluster/host/current.wal (append-only, per-entry)
    17→//   - Only saves unarchived data (archived data is already persisted elsewhere)
    18→//   - GC optimization during loading to prevent excessive heap growth
    19→//
    20→// Checkpoint Workflow:
    21→//  1. Init() loads checkpoints within retention window at startup
    22→//  2. Checkpointing() worker periodically saves new data
    23→//  3. Shutdown() writes final checkpoint before exit
    24→//
    25→// File Organization:
    26→//
    27→//	checkpoints/
    28→//	  cluster1/
    29→//	    host001/
    30→//	      1234567890.json  (JSON format: full subtree snapshot)
    31→//	      1234567890.bin   (binary format: full subtree snapshot)
    32→//	      current.wal      (WAL format: append-only per-entry log)
    33→//	    host002/
    34→//	      ...
    35→package metricstore
    36→
    37→import (
    38→	"bufio"
    39→	"context"
    40→	"encoding/json"
    41→	"errors"
    42→	"fmt"
    43→	"io/fs"
    44→	"os"
    45→	"path"
    46→	"path/filepath"
    47→	"sort"
    48→	"strconv"
    49→	"strings"
    50→	"sync"
    51→	"sync/atomic"
    52→	"time"
    53→
    54→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    55→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    56→)
    57→
    58→const (
    59→	CheckpointFilePerms = 0o644                    // File permissions for checkpoint files
    60→	CheckpointDirPerms  = 0o755                    // Directory permissions for checkpoint directories
    61→	GCTriggerInterval   = DefaultGCTriggerInterval // Interval for triggering GC during checkpoint loading
    62→)
    63→
    64→// CheckpointMetrics represents metric data in a checkpoint file.
    65→// Whenever the structure changes, update MarshalJSON as well!
    66→type CheckpointMetrics struct {
    67→	Data      []schema.Float `json:"data"`
    68→	Frequency int64          `json:"frequency"`
    69→	Start     int64          `json:"start"`
    70→}
    71→
    72→// CheckpointFile represents the hierarchical structure of a checkpoint file.
    73→// It mirrors the Level tree structure from the MemoryStore.
    74→type CheckpointFile struct {
    75→	Metrics  map[string]*CheckpointMetrics `json:"metrics"`
    76→	Children map[string]*CheckpointFile    `json:"children"`
    77→	From     int64                         `json:"from"`
    78→	To       int64                         `json:"to"`
    79→}
    80→
    81→// lastCheckpoint tracks the timestamp of the last checkpoint creation.
    82→var (
    83→	lastCheckpoint   time.Time
    84→	lastCheckpointMu sync.Mutex
    85→)
    86→
    87→// Checkpointing starts a background worker that periodically saves metric data to disk.
    88→//
    89→// Checkpoints are written every 12 hours (hardcoded).
    90→//
    91→// Format behaviour:
    92→//   - "json": Periodic checkpointing every checkpointInterval
    93→//   - "wal":  Periodic binary snapshots + WAL rotation every checkpointInterval
    94→func Checkpointing(wg *sync.WaitGroup, ctx context.Context) {
    95→	lastCheckpointMu.Lock()
    96→	lastCheckpoint = time.Now()
    97→	lastCheckpointMu.Unlock()
    98→
    99→	ms := GetMemoryStore()
   100→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// This file implements checkpoint persistence for the in-memory metric store.
//
// Checkpoints enable graceful restarts by periodically saving in-memory metric
// data to disk in JSON or binary format. The checkpoint system:
//
// Key Features:
//   - Periodic background checkpointing via the Checkpointing() worker
//   - Two format families: JSON (human-readable) and WAL+binary (compact, crash-safe)
//   - Parallel checkpoint creation and loading using worker pools
//   - Hierarchical file organization: checkpoint_dir/cluster/host/timestamp.{json|bin}
//   - WAL file: checkpoint_dir/cluster/host/current.wal (append-only, per-entry)
//   - Only saves unarchived data (archived data is already persisted elsewhere)
//   - GC optimization during loading to prevent excessive heap growth
//
// Checkpoint Workflow:
//  1. Init() loads checkpoints within retention window at startup
//  2. Checkpointing() worker periodically saves new data
//  3. Shutdown() writes final checkpoint before exit
//
// File Organization:
//
//	checkpoints/
//	  cluster1/
//	    host001/
//	      1234567890.json  (JSON format: full subtree snapshot)
//	      1234567890.bin   (binary format: full subtree snapshot)
//	      current.wal      (WAL format: append-only per-entry log)
//	    host002/
//	      ...
package metricstore

import (
	"bufio"
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"io/fs"
	"os"
	"path"
	"path/filepath"
	"sort"
	"strconv"
	"strings"
	"sync"
	"sync/atomic"
	"time"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

const (
	CheckpointFilePerms = 0o644                    // File permissions for checkpoint files
	CheckpointDirPerms  = 0o755                    // Directory permissions for checkpoint directories
	GCTriggerInterval   = DefaultGCTriggerInterval // Interval for triggering GC during checkpoint loading
)

// CheckpointMetrics represents metric data in a checkpoint file.
// Whenever the structure changes, update MarshalJSON as well!
type CheckpointMetrics struct {
	Data      []schema.Float `json:"data"`
	Frequency int64          `json:"frequency"`
	Start     int64          `json:"start"`
}

// CheckpointFile represents the hierarchical structure of a checkpoint file.
// It mirrors the Level tree structure from the MemoryStore.
type CheckpointFile struct {
	Metrics  map[string]*CheckpointMetrics `json:"metrics"`
	Children map[string]*CheckpointFile    `json:"children"`
	From     int64                         `json:"from"`
	To       int64                         `json:"to"`
}

// lastCheckpoint tracks the timestamp of the last checkpoint creation.
var (
	lastCheckpoint   time.Time
	lastCheckpointMu sync.Mutex
)

// Checkpointing starts a background worker that periodically saves metric data to disk.
//
// Checkpoints are written every 12 hours (hardcoded).
//
// Format behaviour:
//   - "json": Periodic checkpointing every checkpointInterval
//   - "wal":  Periodic binary snapshots + WAL rotation every checkpointInterval
func Checkpointing(wg *sync.WaitGroup, ctx context.Context) {
	lastCheckpointMu.Lock()
	lastCheckpoint = time.Now()
	lastCheckpointMu.Unlock()

	ms := GetMemoryStore()


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstore provides an efficient in-memory time-series metric storage system
     7→// with support for hierarchical data organization, checkpointing, and archiving.
     8→//
     9→// The package organizes metrics in a tree structure (cluster → host → component) and
    10→// provides concurrent read/write access to metric data with configurable aggregation strategies.
    11→// Background goroutines handle periodic checkpointing (JSON or Avro format), archiving old data,
    12→// and enforcing retention policies.
    13→//
    14→// Key features:
    15→//   - In-memory metric storage with configurable retention
    16→//   - Hierarchical data organization (selectors)
    17→//   - Concurrent checkpoint/archive workers
    18→//   - Support for sum and average aggregation
    19→//   - NATS integration for metric ingestion
    20→package metricstore
    21→
    22→import (
    23→	"bytes"
    24→	"context"
    25→	"encoding/json"
    26→	"errors"
    27→	"fmt"
    28→	"runtime"
    29→	"runtime/debug"
    30→	"slices"
    31→	"sync"
    32→	"time"
    33→
    34→	"github.com/ClusterCockpit/cc-backend/internal/config"
    35→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    36→	"github.com/ClusterCockpit/cc-lib/v2/resampler"
    37→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    38→	"github.com/ClusterCockpit/cc-lib/v2/util"
    39→)
    40→
    41→// GlobalState holds the global state for the metric store with thread-safe access.
    42→type GlobalState struct {
    43→	mu                sync.RWMutex
    44→	lastRetentionTime int64
    45→	selectorsExcluded bool
    46→}
    47→
    48→var (
    49→	singleton  sync.Once
    50→	msInstance *MemoryStore
    51→	// shutdownFunc stores the context cancellation function created in Init
    52→	// and is called during Shutdown to cancel all background goroutines
    53→	shutdownFunc   context.CancelFunc
    54→	shutdownFuncMu sync.Mutex // Protects shutdownFunc from concurrent access
    55→	// Create a global instance
    56→	state = &GlobalState{}
    57→)
    58→
    59→// NodeProvider provides information about nodes currently in use by running jobs.
    60→//
    61→// This interface allows metricstore to query job information without directly
    62→// depending on the repository package, breaking the import cycle.
    63→//
    64→// Implementations should return nodes that are actively processing jobs started
    65→// before the given timestamp. These nodes will be excluded from retention-based
    66→// garbage collection to prevent data loss for jobs that are still running or
    67→// recently completed.
    68→type NodeProvider interface {
    69→	// GetUsedNodes returns a map of cluster names to sorted lists of unique hostnames
    70→	// that are currently in use by jobs that started before the given timestamp.
    71→	//
    72→	// Parameters:
    73→	//   - ts: Unix timestamp threshold - returns nodes with jobs started before this time
    74→	//
    75→	// Returns:
    76→	//   - Map of cluster names to lists of node hostnames that should be excluded from garbage collection
    77→	//   - Error if the query fails
    78→	GetUsedNodes(ts int64) (map[string][]string, error)
    79→}
    80→
    81→// Metric represents a single metric data point to be written to the store.
    82→type Metric struct {
    83→	Name  string
    84→	Value schema.Float
    85→	// MetricConfig contains frequency and aggregation settings for this metric.
    86→	// If Frequency is 0, configuration will be looked up from MemoryStore.Metrics during Write().
    87→	MetricConfig MetricConfig
    88→}
    89→
    90→// MemoryStore is the main in-memory time-series metric storage implementation.
    91→//
    92→// It organizes metrics in a hierarchical tree structure where each level represents
    93→// a component of the system hierarchy (e.g., cluster → host → CPU). Each level can
    94→// store multiple metrics as time-series buffers.
    95→//
    96→// The store is initialized as a singleton via InitMetrics() and accessed via GetMemoryStore().
    97→// All public methods are safe for concurrent use.
    98→type MemoryStore struct {
    99→	Metrics      map[string]MetricConfig
   100→	root         Level

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package metricstore provides an efficient in-memory time-series metric storage system
// with support for hierarchical data organization, checkpointing, and archiving.
//
// The package organizes metrics in a tree structure (cluster → host → component) and
// provides concurrent read/write access to metric data with configurable aggregation strategies.
// Background goroutines handle periodic checkpointing (JSON or Avro format), archiving old data,
// and enforcing retention policies.
//
// Key features:
//   - In-memory metric storage with configurable retention
//   - Hierarchical data organization (selectors)
//   - Concurrent checkpoint/archive workers
//   - Support for sum and average aggregation
//   - NATS integration for metric ingestion
package metricstore

import (
	"bytes"
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"runtime"
	"runtime/debug"
	"slices"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/config"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/resampler"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	"github.com/ClusterCockpit/cc-lib/v2/util"
)

// GlobalState holds the global state for the metric store with thread-safe access.
type GlobalState struct {
	mu                sync.RWMutex
	lastRetentionTime int64
	selectorsExcluded bool
}

var (
	singleton  sync.Once
	msInstance *MemoryStore
	// shutdownFunc stores the context cancellation function created in Init
	// and is called during Shutdown to cancel all background goroutines
	shutdownFunc   context.CancelFunc
	shutdownFuncMu sync.Mutex // Protects shutdownFunc from concurrent access
	// Create a global instance
	state = &GlobalState{}
)

// NodeProvider provides information about nodes currently in use by running jobs.
//
// This interface allows metricstore to query job information without directly
// depending on the repository package, breaking the import cycle.
//
// Implementations should return nodes that are actively processing jobs started
// before the given timestamp. These nodes will be excluded from retention-based
// garbage collection to prevent data loss for jobs that are still running or
// recently completed.
type NodeProvider interface {
	// GetUsedNodes returns a map of cluster names to sorted lists of unique hostnames
	// that are currently in use by jobs that started before the given timestamp.
	//
	// Parameters:
	//   - ts: Unix timestamp threshold - returns nodes with jobs started before this time
	//
	// Returns:
	//   - Map of cluster names to lists of node hostnames that should be excluded from garbage collection
	//   - Error if the query fails
	GetUsedNodes(ts int64) (map[string][]string, error)
}

// Metric represents a single metric data point to be written to the store.
type Metric struct {
	Name  string
	Value schema.Float
	// MetricConfig contains frequency and aggregation settings for this metric.
	// If Frequency is 0, configuration will be looked up from MemoryStore.Metrics during Write().
	MetricConfig MetricConfig
}

// MemoryStore is the main in-memory time-series metric storage implementation.
//
// It organizes metrics in a hierarchical tree structure where each level represents
// a component of the system hierarchy (e.g., cluster → host → CPU). Each level can
// store multiple metrics as time-series buffers.
//
// The store is initialized as a singleton via InitMetrics() and accessed via GetMemoryStore().
// All public methods are safe for concurrent use.
type MemoryStore struct {
	Metrics      map[string]MetricConfig
	root         Level


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package metricstore
     7→
     8→import (
     9→	"context"
    10→	"errors"
    11→	"fmt"
    12→	"os"
    13→	"path/filepath"
    14→	"strings"
    15→	"sync"
    16→	"sync/atomic"
    17→	"time"
    18→
    19→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    20→)
    21→
    22→// Worker for either Archiving or Deleting files
    23→
    24→func CleanUp(wg *sync.WaitGroup, ctx context.Context) {
    25→	if Keys.Cleanup.Mode == "archive" {
    26→		cclog.Info("[METRICSTORE]> enable archive cleanup to parquet")
    27→		// Run as Archiver
    28→		cleanUpWorker(wg, ctx,
    29→			Keys.RetentionInMemory,
    30→			"archiving",
    31→			Keys.Cleanup.RootDir,
    32→			false,
    33→		)
    34→	} else {
    35→		// Run as Deleter
    36→		cleanUpWorker(wg, ctx,
    37→			Keys.RetentionInMemory,
    38→			"deleting",
    39→			"",
    40→			true,
    41→		)
    42→	}
    43→}
    44→
    45→// cleanUpWorker takes simple values to configure what it does
    46→func cleanUpWorker(wg *sync.WaitGroup, ctx context.Context, interval string, mode string, cleanupDir string, delete bool) {
    47→	wg.Go(func() {
    48→		d, err := time.ParseDuration(interval)
    49→		if err != nil {
    50→			cclog.Fatalf("[METRICSTORE]> error parsing %s interval duration: %v\n", mode, err)
    51→		}
    52→		if d <= 0 {
    53→			return
    54→		}
    55→
    56→		ticker := time.NewTicker(d)
    57→		defer ticker.Stop()
    58→
    59→		for {
    60→			select {
    61→			case <-ctx.Done():
    62→				return
    63→			case <-ticker.C:
    64→				t := time.Now().Add(-d)
    65→				cclog.Infof("[METRICSTORE]> start %s checkpoints (older than %s)...", mode, t.Format(time.RFC3339))
    66→
    67→				n, err := CleanupCheckpoints(Keys.Checkpoints.RootDir, cleanupDir, t.Unix(), delete)
    68→
    69→				if err != nil {
    70→					cclog.Errorf("[METRICSTORE]> %s failed: %s", mode, err.Error())
    71→				} else {
    72→					if delete {
    73→						cclog.Infof("[METRICSTORE]> done: %d checkpoints deleted", n)
    74→					} else {
    75→						cclog.Infof("[METRICSTORE]> done: %d checkpoint files archived to parquet", n)
    76→					}
    77→				}
    78→			}
    79→		}
    80→	})
    81→}
    82→
    83→var ErrNoNewArchiveData error = errors.New("all data already archived")
    84→
    85→// CleanupCheckpoints deletes or archives all checkpoint files older than `from`.
    86→// When archiving, consolidates all hosts per cluster into a single Parquet file.
    87→func CleanupCheckpoints(checkpointsDir, cleanupDir string, from int64, deleteInstead bool) (int, error) {
    88→	if deleteInstead {
    89→		return deleteCheckpoints(checkpointsDir, from)
    90→	}
    91→
    92→	return archiveCheckpoints(checkpointsDir, cleanupDir, from)
    93→}
    94→
    95→// deleteCheckpoints removes checkpoint files older than `from` across all clusters/hosts.
    96→func deleteCheckpoints(checkpointsDir string, from int64) (int, error) {
    97→	entries1, err := os.ReadDir(checkpointsDir)
    98→	if err != nil {
    99→		return 0, err
   100→	}

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package metricstore

import (
	"context"
	"errors"
	"fmt"
	"os"
	"path/filepath"
	"strings"
	"sync"
	"sync/atomic"
	"time"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
)

// Worker for either Archiving or Deleting files

func CleanUp(wg *sync.WaitGroup, ctx context.Context) {
	if Keys.Cleanup.Mode == "archive" {
		cclog.Info("[METRICSTORE]> enable archive cleanup to parquet")
		// Run as Archiver
		cleanUpWorker(wg, ctx,
			Keys.RetentionInMemory,
			"archiving",
			Keys.Cleanup.RootDir,
			false,
		)
	} else {
		// Run as Deleter
		cleanUpWorker(wg, ctx,
			Keys.RetentionInMemory,
			"deleting",
			"",
			true,
		)
	}
}

// cleanUpWorker takes simple values to configure what it does
func cleanUpWorker(wg *sync.WaitGroup, ctx context.Context, interval string, mode string, cleanupDir string, delete bool) {
	wg.Go(func() {
		d, err := time.ParseDuration(interval)
		if err != nil {
			cclog.Fatalf("[METRICSTORE]> error parsing %s interval duration: %v\n", mode, err)
		}
		if d <= 0 {
			return
		}

		ticker := time.NewTicker(d)
		defer ticker.Stop()

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				t := time.Now().Add(-d)
				cclog.Infof("[METRICSTORE]> start %s checkpoints (older than %s)...", mode, t.Format(time.RFC3339))

				n, err := CleanupCheckpoints(Keys.Checkpoints.RootDir, cleanupDir, t.Unix(), delete)

				if err != nil {
					cclog.Errorf("[METRICSTORE]> %s failed: %s", mode, err.Error())
				} else {
					if delete {
						cclog.Infof("[METRICSTORE]> done: %d checkpoints deleted", n)
					} else {
						cclog.Infof("[METRICSTORE]> done: %d checkpoint files archived to parquet", n)
					}
				}
			}
		}
	})
}

var ErrNoNewArchiveData error = errors.New("all data already archived")

// CleanupCheckpoints deletes or archives all checkpoint files older than `from`.
// When archiving, consolidates all hosts per cluster into a single Parquet file.
func CleanupCheckpoints(checkpointsDir, cleanupDir string, from int64, deleteInstead bool) (int, error) {
	if deleteInstead {
		return deleteCheckpoints(checkpointsDir, from)
	}

	return archiveCheckpoints(checkpointsDir, cleanupDir, from)
}

// deleteCheckpoints removes checkpoint files older than `from` across all clusters/hosts.
func deleteCheckpoints(checkpointsDir string, from int64) (int, error) {
	entries1, err := os.ReadDir(checkpointsDir)
	if err != nil {
		return 0, err
	}


    87→// Checkpointing starts a background worker that periodically saves metric data to disk.
    88→//
    89→// Checkpoints are written every 12 hours (hardcoded).
    90→//
    91→// Format behaviour:
    92→//   - "json": Periodic checkpointing every checkpointInterval
    93→//   - "wal":  Periodic binary snapshots + WAL rotation every checkpointInterval
    94→func Checkpointing(wg *sync.WaitGroup, ctx context.Context) {
    95→	lastCheckpointMu.Lock()
    96→	lastCheckpoint = time.Now()
    97→	lastCheckpointMu.Unlock()
    98→
    99→	ms := GetMemoryStore()
   100→
   101→	wg.Go(func() {
   102→
   103→		d := 12 * time.Hour // default checkpoint interval
   104→		if Keys.CheckpointInterval != "" {
   105→			parsed, err := time.ParseDuration(Keys.CheckpointInterval)
   106→			if err != nil {
   107→				cclog.Errorf("[METRICSTORE]> invalid checkpoint-interval %q: %s, using default 12h", Keys.CheckpointInterval, err)
   108→			} else {
   109→				d = parsed
   110→			}
   111→		}
   112→
   113→		ticker := time.NewTicker(d)
   114→		defer ticker.Stop()
   115→
   116→		for {
   117→			select {
   118→			case <-ctx.Done():
   119→				return
   120→			case <-ticker.C:
   121→				lastCheckpointMu.Lock()
   122→				from := lastCheckpoint
   123→				lastCheckpointMu.Unlock()
   124→
   125→				now := time.Now()
   126→				cclog.Infof("[METRICSTORE]> start checkpointing (starting at %s)...", from.Format(time.RFC3339))
   127→
   128→				if Keys.Checkpoints.FileFormat == "wal" {
   129→					n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
   130→					if err != nil {
   131→						cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
   132→					} else {
   133→						cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
   134→						lastCheckpointMu.Lock()
   135→						lastCheckpoint = now
   136→						lastCheckpointMu.Unlock()
   137→						// Rotate WAL files for successfully checkpointed hosts.
   138→						RotateWALFiles(hostDirs)
   139→					}
   140→				} else {
   141→					n, err := ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
   142→					if err != nil {
   143→						cclog.Errorf("[METRICSTORE]> checkpointing failed: %s", err.Error())
   144→					} else {
   145→						cclog.Infof("[METRICSTORE]> done: %d checkpoint files created", n)
   146→						lastCheckpointMu.Lock()
   147→						lastCheckpoint = now
   148→						lastCheckpointMu.Unlock()
   149→					}
   150→				}
   151→			}
   152→		}
   153→	})
   154→}
   155→
   156→// MarshalJSON provides optimized JSON encoding for CheckpointMetrics.
   157→//
   158→// Since schema.Float has custom MarshalJSON, serializing []Float has significant overhead.
   159→// This method manually constructs JSON to avoid allocations and interface conversions.
   160→func (cm *CheckpointMetrics) MarshalJSON() ([]byte, error) {
   161→	buf := make([]byte, 0, 128+len(cm.Data)*8)
   162→	buf = append(buf, `{"frequency":`...)
   163→	buf = strconv.AppendInt(buf, cm.Frequency, 10)
   164→	buf = append(buf, `,"start":`...)
   165→	buf = strconv.AppendInt(buf, cm.Start, 10)
   166→	buf = append(buf, `,"data":[`...)

// Checkpointing starts a background worker that periodically saves metric data to disk.
//
// Checkpoints are written every 12 hours (hardcoded).
//
// Format behaviour:
//   - "json": Periodic checkpointing every checkpointInterval
//   - "wal":  Periodic binary snapshots + WAL rotation every checkpointInterval
func Checkpointing(wg *sync.WaitGroup, ctx context.Context) {
	lastCheckpointMu.Lock()
	lastCheckpoint = time.Now()
	lastCheckpointMu.Unlock()

	ms := GetMemoryStore()

	wg.Go(func() {

		d := 12 * time.Hour // default checkpoint interval
		if Keys.CheckpointInterval != "" {
			parsed, err := time.ParseDuration(Keys.CheckpointInterval)
			if err != nil {
				cclog.Errorf("[METRICSTORE]> invalid checkpoint-interval %q: %s, using default 12h", Keys.CheckpointInterval, err)
			} else {
				d = parsed
			}
		}

		ticker := time.NewTicker(d)
		defer ticker.Stop()

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				lastCheckpointMu.Lock()
				from := lastCheckpoint
				lastCheckpointMu.Unlock()

				now := time.Now()
				cclog.Infof("[METRICSTORE]> start checkpointing (starting at %s)...", from.Format(time.RFC3339))

				if Keys.Checkpoints.FileFormat == "wal" {
					n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
					if err != nil {
						cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
					} else {
						cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
						lastCheckpointMu.Lock()
						lastCheckpoint = now
						lastCheckpointMu.Unlock()
						// Rotate WAL files for successfully checkpointed hosts.
						RotateWALFiles(hostDirs)
					}
				} else {
					n, err := ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
					if err != nil {
						cclog.Errorf("[METRICSTORE]> checkpointing failed: %s", err.Error())
					} else {
						cclog.Infof("[METRICSTORE]> done: %d checkpoint files created", n)
						lastCheckpointMu.Lock()
						lastCheckpoint = now
						lastCheckpointMu.Unlock()
					}
				}
			}
		}
	})
}

// MarshalJSON provides optimized JSON encoding for CheckpointMetrics.
//
// Since schema.Float has custom MarshalJSON, serializing []Float has significant overhead.
// This method manually constructs JSON to avoid allocations and interface conversions.
func (cm *CheckpointMetrics) MarshalJSON() ([]byte, error) {
	buf := make([]byte, 0, 128+len(cm.Data)*8)
	buf = append(buf, `{"frequency":`...)
	buf = strconv.AppendInt(buf, cm.Frequency, 10)
	buf = append(buf, `,"start":`...)
	buf = strconv.AppendInt(buf, cm.Start, 10)
	buf = append(buf, `,"data":[`...)


   100→	root         Level
   101→	nodeProvider NodeProvider
   102→}
   103→
   104→// Init initializes the metric store from configuration and starts background workers.
   105→//
   106→// This function must be called exactly once before any other metricstore operations.
   107→// It performs the following initialization steps:
   108→//  1. Validates and decodes the metric store configuration
   109→//  2. Configures worker pool size (defaults to NumCPU/2+1, max 10)
   110→//  3. Loads metric configurations from all registered clusters
   111→//  4. Restores checkpoints within the retention window
   112→//  5. Starts background workers for retention, checkpointing, archiving, and monitoring
   113→//  6. Optionally subscribes to NATS for real-time metric ingestion
   114→//
   115→// Parameters:
   116→//   - rawConfig: JSON configuration for the metric store (see MetricStoreConfig)
   117→//   - wg: WaitGroup that will be incremented for each background goroutine started
   118→//
   119→// The function will call cclog.Fatal on critical errors during initialization.
   120→// Use Shutdown() to cleanly stop all background workers started by Init().
   121→//
   122→// Note: Signal handling must be implemented by the caller. Call Shutdown() when
   123→// receiving termination signals to ensure checkpoint data is persisted.
   124→func Init(rawConfig json.RawMessage, metrics map[string]MetricConfig, wg *sync.WaitGroup) {
   125→	startupTime := time.Now()
   126→
   127→	if rawConfig != nil {
   128→		config.Validate(configSchema, rawConfig)
   129→		dec := json.NewDecoder(bytes.NewReader(rawConfig))
   130→		dec.DisallowUnknownFields()
   131→		if err := dec.Decode(&Keys); err != nil {
   132→			cclog.Abortf("[METRICSTORE]> Metric Store Config Init: Could not decode config file '%s'.\nError: %s\n", rawConfig, err.Error())
   133→		}
   134→	}
   135→
   136→	// Set NumWorkers from config or use default
   137→	if Keys.NumWorkers <= 0 {
   138→		Keys.NumWorkers = min(runtime.NumCPU()/2+1, DefaultMaxWorkers)
   139→	}
   140→	cclog.Debugf("[METRICSTORE]> Using %d workers for checkpoint/archive operations\n", Keys.NumWorkers)
   141→
   142→	// Pass the config.MetricStoreKeys
   143→	InitMetrics(metrics)
   144→
   145→	ms := GetMemoryStore()
   146→
   147→	d, err := time.ParseDuration(Keys.RetentionInMemory)
   148→	if err != nil {
   149→		cclog.Fatal(err)
   150→	}
   151→
   152→	restoreFrom := startupTime.Add(-d)
   153→	cclog.Infof("[METRICSTORE]> Loading checkpoints newer than %s\n", restoreFrom.Format(time.RFC3339))
   154→	files, err := ms.FromCheckpointFiles(Keys.Checkpoints.RootDir, restoreFrom.Unix())
   155→	loadedData := ms.SizeInBytes() / 1024 / 1024 // In MB
   156→	if err != nil {
   157→		cclog.Fatalf("[METRICSTORE]> Loading checkpoints failed: %s\n", err.Error())
   158→	} else {
   159→		cclog.Infof("[METRICSTORE]> Checkpoints loaded (%d files, %d MB, that took %fs)\n", files, loadedData, time.Since(startupTime).Seconds())
   160→	}
   161→
   162→	// Try to use less memory by forcing a GC run here and then
   163→	// lowering the target percentage. The default of 100 means
   164→	// that only once the ratio of new allocations execeds the
   165→	// previously active heap, a GC is triggered.
   166→	// Forcing a GC here will set the "previously active heap"
   167→	// to a minumum.
   168→	// runtime.GC()
   169→
   170→	ctx, shutdown := context.WithCancel(context.Background())
   171→
   172→	Retention(wg, ctx)
   173→	Checkpointing(wg, ctx)
   174→	CleanUp(wg, ctx)
   175→	WALStaging(wg, ctx)
   176→	MemoryUsageTracker(wg, ctx)
   177→
   178→	// Note: Signal handling has been removed from this function.
   179→	// The caller is responsible for handling shutdown signals and calling
   180→	// the shutdown() function when appropriate.
   181→	// Store the shutdown function for later use by Shutdown()
   182→	shutdownFuncMu.Lock()
   183→	shutdownFunc = shutdown
   184→	shutdownFuncMu.Unlock()
   185→
   186→	if Keys.Subscriptions != nil {
   187→		wg.Go(func() {
   188→			if err := ReceiveNats(ms, Keys.NumWorkers, ctx); err != nil {
   189→				cclog.Fatal(err)
   190→			}
   191→		})
   192→	}
   193→}
   194→
   195→// InitMetrics initializes the singleton MemoryStore instance with the given metric configurations.
   196→//
   197→// This function must be called before GetMemoryStore() and can only be called once due to
   198→// the singleton pattern. It assigns each metric an internal offset for efficient buffer indexing.
   199→//
   200→// Parameters:
   201→//   - metrics: Map of metric names to their configurations (frequency and aggregation strategy)
   202→//
   203→// Panics if any metric has Frequency == 0, which indicates an invalid configuration.
   204→//
   205→// After this call, the global msInstance is ready for use via GetMemoryStore().
   206→func InitMetrics(metrics map[string]MetricConfig) {
   207→	singleton.Do(func() {
   208→		offset := 0
   209→		for key, cfg := range metrics {
   210→			if cfg.Frequency == 0 {
   211→				panic("[METRICSTORE]> invalid frequency")
   212→			}
   213→
   214→			metrics[key] = MetricConfig{
   215→				Frequency:   cfg.Frequency,
   216→				Aggregation: cfg.Aggregation,
   217→				offset:      offset,
   218→			}
   219→			offset += 1
   220→		}
   221→
   222→		msInstance = &MemoryStore{
   223→			root: Level{
   224→				metrics:  make([]*buffer, len(metrics)),
   225→				children: make(map[string]*Level),
   226→			},
   227→			Metrics: metrics,
   228→		}
   229→	})
   230→}
   231→
   232→// GetMemoryStore returns the singleton MemoryStore instance.
   233→//
   234→// Returns the initialized MemoryStore singleton. Calls cclog.Fatal if InitMetrics() was not called first.
   235→//
   236→// This function is safe for concurrent use after initialization.
   237→func GetMemoryStore() *MemoryStore {
   238→	if msInstance == nil {
   239→		cclog.Warnf("[METRICSTORE]> MemoryStore not initialized!")
   240→	}
   241→
   242→	return msInstance
   243→}
   244→
   245→func (ms *MemoryStore) GetMetricFrequency(metricName string) (int64, error) {
   246→	if metric, ok := ms.Metrics[metricName]; ok {
   247→		return metric.Frequency, nil
   248→	}
   249→	return 0, fmt.Errorf("[METRICSTORE]> metric %s not found", metricName)
   250→}
   251→
   252→// SetNodeProvider sets the NodeProvider implementation for the MemoryStore.
   253→// This must be called during initialization to provide job state information
   254→// for selective buffer retention during Free operations.
   255→// If not set, the Free function will fall back to freeing all buffers.
   256→func (ms *MemoryStore) SetNodeProvider(provider NodeProvider) {
   257→	ms.nodeProvider = provider
   258→}
   259→
   260→// Shutdown performs a graceful shutdown of the metric store.
   261→//
   262→// This function cancels all background goroutines started by Init() and writes
   263→// a final checkpoint to disk before returning. It should be called when the
   264→// application receives a termination signal.
   265→//
   266→// The function will:
   267→//  1. Cancel the context to stop all background workers
   268→//  2. Close the WAL messages channel if using WAL format
   269→//  3. Write a final checkpoint to preserve in-memory data
   270→//  4. Log any errors encountered during shutdown
   271→//
   272→// Note: This function blocks until the final checkpoint is written.
   273→func Shutdown() {
   274→	totalStart := time.Now()
   275→
   276→	shutdownFuncMu.Lock()
   277→	defer shutdownFuncMu.Unlock()
   278→	if shutdownFunc != nil {
   279→		shutdownFunc()
   280→	}
   281→	cclog.Infof("[METRICSTORE]> Background workers cancelled (%v)", time.Since(totalStart))
   282→
   283→	if Keys.Checkpoints.FileFormat == "wal" {
   284→		// Signal producers to stop sending before closing channels,
   285→		// preventing send-on-closed-channel panics from in-flight NATS workers.
   286→		walShuttingDown.Store(true)
   287→		// Brief grace period for in-flight DecodeLine calls to complete.
   288→		time.Sleep(100 * time.Millisecond)
   289→
   290→		for _, ch := range walShardChs {
   291→			close(ch)
   292→		}
   293→		drainStart := time.Now()
   294→		WaitForWALStagingDrain()
   295→		cclog.Infof("[METRICSTORE]> WAL staging goroutines exited (%v)", time.Since(drainStart))
   296→	}
   297→
   298→	cclog.Infof("[METRICSTORE]> Writing checkpoint to '%s'...", Keys.Checkpoints.RootDir)
   299→	checkpointStart := time.Now()
   300→	var files int
   301→	var err error
   302→
   303→	ms := GetMemoryStore()
   304→
   305→	lastCheckpointMu.Lock()
   306→	from := lastCheckpoint
   307→	lastCheckpointMu.Unlock()
   308→
   309→	if Keys.Checkpoints.FileFormat == "wal" {
   310→		// WAL files are deleted per-host inside ToCheckpointWAL workers.
   311→		files, _, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
   312→	} else {
   313→		files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
   314→	}
   315→
   316→	if err != nil {
   317→		cclog.Errorf("[METRICSTORE]> Writing checkpoint failed: %s", err.Error())
   318→	}
   319→	cclog.Infof("[METRICSTORE]> Done! (%d files written in %v, total shutdown: %v)", files, time.Since(checkpointStart), time.Since(totalStart))
   320→}
   321→
   322→// Retention starts a background goroutine that periodically frees old metric data.
   323→//
   324→// This worker runs at half the retention interval and calls Free() to remove buffers
   325→// older than the configured retention time. It respects the NodeProvider to preserve
   326→// data for nodes with active jobs.
   327→//
   328→// Parameters:
   329→//   - wg: WaitGroup to signal completion when context is cancelled
   330→//   - ctx: Context for cancellation signal
   331→//
   332→// The goroutine exits when ctx is cancelled.
   333→func Retention(wg *sync.WaitGroup, ctx context.Context) {
   334→	ms := GetMemoryStore()
   335→
   336→	wg.Go(func() {
   337→		d, err := time.ParseDuration(Keys.RetentionInMemory)
   338→		if err != nil {
   339→			cclog.Fatal(err)
   340→		}
   341→		if d <= 0 {
   342→			return
   343→		}
   344→
   345→		tickInterval := d / 2
   346→		if tickInterval <= 0 {
   347→			return
   348→		}
   349→		ticker := time.NewTicker(tickInterval)

	root         Level
	nodeProvider NodeProvider
}

// Init initializes the metric store from configuration and starts background workers.
//
// This function must be called exactly once before any other metricstore operations.
// It performs the following initialization steps:
//  1. Validates and decodes the metric store configuration
//  2. Configures worker pool size (defaults to NumCPU/2+1, max 10)
//  3. Loads metric configurations from all registered clusters
//  4. Restores checkpoints within the retention window
//  5. Starts background workers for retention, checkpointing, archiving, and monitoring
//  6. Optionally subscribes to NATS for real-time metric ingestion
//
// Parameters:
//   - rawConfig: JSON configuration for the metric store (see MetricStoreConfig)
//   - wg: WaitGroup that will be incremented for each background goroutine started
//
// The function will call cclog.Fatal on critical errors during initialization.
// Use Shutdown() to cleanly stop all background workers started by Init().
//
// Note: Signal handling must be implemented by the caller. Call Shutdown() when
// receiving termination signals to ensure checkpoint data is persisted.
func Init(rawConfig json.RawMessage, metrics map[string]MetricConfig, wg *sync.WaitGroup) {
	startupTime := time.Now()

	if rawConfig != nil {
		config.Validate(configSchema, rawConfig)
		dec := json.NewDecoder(bytes.NewReader(rawConfig))
		dec.DisallowUnknownFields()
		if err := dec.Decode(&Keys); err != nil {
			cclog.Abortf("[METRICSTORE]> Metric Store Config Init: Could not decode config file '%s'.\nError: %s\n", rawConfig, err.Error())
		}
	}

	// Set NumWorkers from config or use default
	if Keys.NumWorkers <= 0 {
		Keys.NumWorkers = min(runtime.NumCPU()/2+1, DefaultMaxWorkers)
	}
	cclog.Debugf("[METRICSTORE]> Using %d workers for checkpoint/archive operations\n", Keys.NumWorkers)

	// Pass the config.MetricStoreKeys
	InitMetrics(metrics)

	ms := GetMemoryStore()

	d, err := time.ParseDuration(Keys.RetentionInMemory)
	if err != nil {
		cclog.Fatal(err)
	}

	restoreFrom := startupTime.Add(-d)
	cclog.Infof("[METRICSTORE]> Loading checkpoints newer than %s\n", restoreFrom.Format(time.RFC3339))
	files, err := ms.FromCheckpointFiles(Keys.Checkpoints.RootDir, restoreFrom.Unix())
	loadedData := ms.SizeInBytes() / 1024 / 1024 // In MB
	if err != nil {
		cclog.Fatalf("[METRICSTORE]> Loading checkpoints failed: %s\n", err.Error())
	} else {
		cclog.Infof("[METRICSTORE]> Checkpoints loaded (%d files, %d MB, that took %fs)\n", files, loadedData, time.Since(startupTime).Seconds())
	}

	// Try to use less memory by forcing a GC run here and then
	// lowering the target percentage. The default of 100 means
	// that only once the ratio of new allocations execeds the
	// previously active heap, a GC is triggered.
	// Forcing a GC here will set the "previously active heap"
	// to a minumum.
	// runtime.GC()

	ctx, shutdown := context.WithCancel(context.Background())

	Retention(wg, ctx)
	Checkpointing(wg, ctx)
	CleanUp(wg, ctx)
	WALStaging(wg, ctx)
	MemoryUsageTracker(wg, ctx)

	// Note: Signal handling has been removed from this function.
	// The caller is responsible for handling shutdown signals and calling
	// the shutdown() function when appropriate.
	// Store the shutdown function for later use by Shutdown()
	shutdownFuncMu.Lock()
	shutdownFunc = shutdown
	shutdownFuncMu.Unlock()

	if Keys.Subscriptions != nil {
		wg.Go(func() {
			if err := ReceiveNats(ms, Keys.NumWorkers, ctx); err != nil {
				cclog.Fatal(err)
			}
		})
	}
}

// InitMetrics initializes the singleton MemoryStore instance with the given metric configurations.
//
// This function must be called before GetMemoryStore() and can only be called once due to
// the singleton pattern. It assigns each metric an internal offset for efficient buffer indexing.
//
// Parameters:
//   - metrics: Map of metric names to their configurations (frequency and aggregation strategy)
//
// Panics if any metric has Frequency == 0, which indicates an invalid configuration.
//
// After this call, the global msInstance is ready for use via GetMemoryStore().
func InitMetrics(metrics map[string]MetricConfig) {
	singleton.Do(func() {
		offset := 0
		for key, cfg := range metrics {
			if cfg.Frequency == 0 {
				panic("[METRICSTORE]> invalid frequency")
			}

			metrics[key] = MetricConfig{
				Frequency:   cfg.Frequency,
				Aggregation: cfg.Aggregation,
				offset:      offset,
			}
			offset += 1
		}

		msInstance = &MemoryStore{
			root: Level{
				metrics:  make([]*buffer, len(metrics)),
				children: make(map[string]*Level),
			},
			Metrics: metrics,
		}
	})
}

// GetMemoryStore returns the singleton MemoryStore instance.
//
// Returns the initialized MemoryStore singleton. Calls cclog.Fatal if InitMetrics() was not called first.
//
// This function is safe for concurrent use after initialization.
func GetMemoryStore() *MemoryStore {
	if msInstance == nil {
		cclog.Warnf("[METRICSTORE]> MemoryStore not initialized!")
	}

	return msInstance
}

func (ms *MemoryStore) GetMetricFrequency(metricName string) (int64, error) {
	if metric, ok := ms.Metrics[metricName]; ok {
		return metric.Frequency, nil
	}
	return 0, fmt.Errorf("[METRICSTORE]> metric %s not found", metricName)
}

// SetNodeProvider sets the NodeProvider implementation for the MemoryStore.
// This must be called during initialization to provide job state information
// for selective buffer retention during Free operations.
// If not set, the Free function will fall back to freeing all buffers.
func (ms *MemoryStore) SetNodeProvider(provider NodeProvider) {
	ms.nodeProvider = provider
}

// Shutdown performs a graceful shutdown of the metric store.
//
// This function cancels all background goroutines started by Init() and writes
// a final checkpoint to disk before returning. It should be called when the
// application receives a termination signal.
//
// The function will:
//  1. Cancel the context to stop all background workers
//  2. Close the WAL messages channel if using WAL format
//  3. Write a final checkpoint to preserve in-memory data
//  4. Log any errors encountered during shutdown
//
// Note: This function blocks until the final checkpoint is written.
func Shutdown() {
	totalStart := time.Now()

	shutdownFuncMu.Lock()
	defer shutdownFuncMu.Unlock()
	if shutdownFunc != nil {
		shutdownFunc()
	}
	cclog.Infof("[METRICSTORE]> Background workers cancelled (%v)", time.Since(totalStart))

	if Keys.Checkpoints.FileFormat == "wal" {
		// Signal producers to stop sending before closing channels,
		// preventing send-on-closed-channel panics from in-flight NATS workers.
		walShuttingDown.Store(true)
		// Brief grace period for in-flight DecodeLine calls to complete.
		time.Sleep(100 * time.Millisecond)

		for _, ch := range walShardChs {
			close(ch)
		}
		drainStart := time.Now()
		WaitForWALStagingDrain()
		cclog.Infof("[METRICSTORE]> WAL staging goroutines exited (%v)", time.Since(drainStart))
	}

	cclog.Infof("[METRICSTORE]> Writing checkpoint to '%s'...", Keys.Checkpoints.RootDir)
	checkpointStart := time.Now()
	var files int
	var err error

	ms := GetMemoryStore()

	lastCheckpointMu.Lock()
	from := lastCheckpoint
	lastCheckpointMu.Unlock()

	if Keys.Checkpoints.FileFormat == "wal" {
		// WAL files are deleted per-host inside ToCheckpointWAL workers.
		files, _, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
	} else {
		files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
	}

	if err != nil {
		cclog.Errorf("[METRICSTORE]> Writing checkpoint failed: %s", err.Error())
	}
	cclog.Infof("[METRICSTORE]> Done! (%d files written in %v, total shutdown: %v)", files, time.Since(checkpointStart), time.Since(totalStart))
}

// Retention starts a background goroutine that periodically frees old metric data.
//
// This worker runs at half the retention interval and calls Free() to remove buffers
// older than the configured retention time. It respects the NodeProvider to preserve
// data for nodes with active jobs.
//
// Parameters:
//   - wg: WaitGroup to signal completion when context is cancelled
//   - ctx: Context for cancellation signal
//
// The goroutine exits when ctx is cancelled.
func Retention(wg *sync.WaitGroup, ctx context.Context) {
	ms := GetMemoryStore()

	wg.Go(func() {
		d, err := time.ParseDuration(Keys.RetentionInMemory)
		if err != nil {
			cclog.Fatal(err)
		}
		if d <= 0 {
			return
		}

		tickInterval := d / 2
		if tickInterval <= 0 {
			return
		}
		ticker := time.NewTicker(tickInterval)


    95→// deleteCheckpoints removes checkpoint files older than `from` across all clusters/hosts.
    96→func deleteCheckpoints(checkpointsDir string, from int64) (int, error) {
    97→	entries1, err := os.ReadDir(checkpointsDir)
    98→	if err != nil {
    99→		return 0, err
   100→	}
   101→
   102→	type workItem struct {
   103→		dir           string
   104→		cluster, host string
   105→	}
   106→
   107→	var wg sync.WaitGroup
   108→	n, errs := int32(0), int32(0)
   109→	work := make(chan workItem, Keys.NumWorkers)
   110→
   111→	wg.Add(Keys.NumWorkers)
   112→	for worker := 0; worker < Keys.NumWorkers; worker++ {
   113→		go func() {
   114→			defer wg.Done()
   115→			for item := range work {
   116→				entries, err := os.ReadDir(item.dir)
   117→				if err != nil {
   118→					cclog.Errorf("error reading %s/%s: %s", item.cluster, item.host, err.Error())
   119→					atomic.AddInt32(&errs, 1)
   120→					continue
   121→				}
   122→
   123→				files, err := findFiles(entries, from, false)
   124→				if err != nil {
   125→					cclog.Errorf("error finding files in %s/%s: %s", item.cluster, item.host, err.Error())
   126→					atomic.AddInt32(&errs, 1)
   127→					continue
   128→				}
   129→
   130→				for _, checkpoint := range files {
   131→					if err := os.Remove(filepath.Join(item.dir, checkpoint)); err != nil {
   132→						cclog.Errorf("error deleting %s/%s/%s: %s", item.cluster, item.host, checkpoint, err.Error())
   133→						atomic.AddInt32(&errs, 1)
   134→					} else {
   135→						atomic.AddInt32(&n, 1)
   136→					}
   137→				}
   138→			}
   139→		}()
   140→	}
   141→
   142→	for _, de1 := range entries1 {
   143→		entries2, e := os.ReadDir(filepath.Join(checkpointsDir, de1.Name()))
   144→		if e != nil {
   145→			err = e
   146→			continue
   147→		}
   148→
   149→		for _, de2 := range entries2 {
   150→			work <- workItem{
   151→				dir:     filepath.Join(checkpointsDir, de1.Name(), de2.Name()),
   152→				cluster: de1.Name(),
   153→				host:    de2.Name(),
   154→			}
   155→		}
   156→	}
   157→
   158→	close(work)
   159→	wg.Wait()
   160→
   161→	if err != nil {
   162→		return int(n), err
   163→	}
   164→	if errs > 0 {
   165→		return int(n), fmt.Errorf("%d errors happened while deleting (%d successes)", errs, n)
   166→	}
   167→	return int(n), nil
   168→}
   169→
   170→// archiveCheckpoints archives checkpoint files to Parquet format.
   171→// Produces one Parquet file per cluster: <cleanupDir>/<cluster>/<timestamp>.parquet
   172→// Workers load checkpoint files from disk and send CheckpointFile trees on a
   173→// back-pressured channel. The main thread streams each tree directly to Parquet
   174→// rows without materializing all rows in memory.

// deleteCheckpoints removes checkpoint files older than `from` across all clusters/hosts.
func deleteCheckpoints(checkpointsDir string, from int64) (int, error) {
	entries1, err := os.ReadDir(checkpointsDir)
	if err != nil {
		return 0, err
	}

	type workItem struct {
		dir           string
		cluster, host string
	}

	var wg sync.WaitGroup
	n, errs := int32(0), int32(0)
	work := make(chan workItem, Keys.NumWorkers)

	wg.Add(Keys.NumWorkers)
	for worker := 0; worker < Keys.NumWorkers; worker++ {
		go func() {
			defer wg.Done()
			for item := range work {
				entries, err := os.ReadDir(item.dir)
				if err != nil {
					cclog.Errorf("error reading %s/%s: %s", item.cluster, item.host, err.Error())
					atomic.AddInt32(&errs, 1)
					continue
				}

				files, err := findFiles(entries, from, false)
				if err != nil {
					cclog.Errorf("error finding files in %s/%s: %s", item.cluster, item.host, err.Error())
					atomic.AddInt32(&errs, 1)
					continue
				}

				for _, checkpoint := range files {
					if err := os.Remove(filepath.Join(item.dir, checkpoint)); err != nil {
						cclog.Errorf("error deleting %s/%s/%s: %s", item.cluster, item.host, checkpoint, err.Error())
						atomic.AddInt32(&errs, 1)
					} else {
						atomic.AddInt32(&n, 1)
					}
				}
			}
		}()
	}

	for _, de1 := range entries1 {
		entries2, e := os.ReadDir(filepath.Join(checkpointsDir, de1.Name()))
		if e != nil {
			err = e
			continue
		}

		for _, de2 := range entries2 {
			work <- workItem{
				dir:     filepath.Join(checkpointsDir, de1.Name(), de2.Name()),
				cluster: de1.Name(),
				host:    de2.Name(),
			}
		}
	}

	close(work)
	wg.Wait()

	if err != nil {
		return int(n), err
	}
	if errs > 0 {
		return int(n), fmt.Errorf("%d errors happened while deleting (%d successes)", errs, n)
	}
	return int(n), nil
}

// archiveCheckpoints archives checkpoint files to Parquet format.
// Produces one Parquet file per cluster: <cleanupDir>/<cluster>/<timestamp>.parquet
// Workers load checkpoint files from disk and send CheckpointFile trees on a
// back-pressured channel. The main thread streams each tree directly to Parquet
// rows without materializing all rows in memory.


Now let me find the `loadFile` and `toCheckpointFile` functions, and `findFiles`.


pkg/metricstore/checkpoint.go:263:func (l *Level) toCheckpointFile(from, to int64, m *MemoryStore) (*CheckpointFile, error) {
pkg/metricstore/checkpoint.go:464:func (l *Level) loadFile(cf *CheckpointFile, m *MemoryStore) error {
pkg/metricstore/checkpoint.go:619:func findFiles(direntries []fs.DirEntry, t int64, findMoreRecentFiles bool) ([]string, error) {


   263→func (l *Level) toCheckpointFile(from, to int64, m *MemoryStore) (*CheckpointFile, error) {
   264→	l.lock.RLock()
   265→	defer l.lock.RUnlock()
   266→
   267→	retval := &CheckpointFile{
   268→		From:     from,
   269→		To:       to,
   270→		Metrics:  make(map[string]*CheckpointMetrics),
   271→		Children: make(map[string]*CheckpointFile),
   272→	}
   273→
   274→	for metric, minfo := range m.Metrics {
   275→		b := l.metrics[minfo.offset]
   276→		if b == nil {
   277→			continue
   278→		}
   279→
   280→		allArchived := true
   281→		b.iterFromTo(from, to, func(b *buffer) error {
   282→			if !b.archived {
   283→				allArchived = false
   284→				return fmt.Errorf("stop") // Early termination signal
   285→			}
   286→			return nil
   287→		})
   288→
   289→		if allArchived {
   290→			continue
   291→		}
   292→
   293→		data := make([]schema.Float, (to-from)/b.frequency+1)
   294→		data, start, end, err := b.read(from, to, data)
   295→		if err != nil {
   296→			return nil, err
   297→		}
   298→
   299→		for i := int((end - start) / b.frequency); i < len(data); i++ {
   300→			data[i] = schema.NaN
   301→		}
   302→
   303→		retval.Metrics[metric] = &CheckpointMetrics{
   304→			Frequency: b.frequency,
   305→			Start:     start,
   306→			Data:      data,
   307→		}
   308→	}
   309→
   310→	for name, child := range l.children {
   311→		val, err := child.toCheckpointFile(from, to, m)
   312→		if err != nil {
   313→			return nil, err
   314→		}
   315→
   316→		if val != nil {
   317→			retval.Children[name] = val
   318→		}
   319→	}
   320→
   321→	if len(retval.Children) == 0 && len(retval.Metrics) == 0 {
   322→		return nil, nil
   323→	}
   324→
   325→	return retval, nil
   326→}
   327→
   328→// toCheckpoint writes a Level's data to a JSON checkpoint file.
   329→// Creates directory if needed. Returns ErrNoNewArchiveData if nothing to save.
   330→func (l *Level) toCheckpoint(dir string, from, to int64, m *MemoryStore) error {
   331→	cf, err := l.toCheckpointFile(from, to, m)
   332→	if err != nil {
   333→		return err
   334→	}
   335→
   336→	if cf == nil {
   337→		return ErrNoNewArchiveData
   338→	}
   339→
   340→	filepath := path.Join(dir, fmt.Sprintf("%d.json", from))
   341→	f, err := os.OpenFile(filepath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
   342→	if err != nil && os.IsNotExist(err) {

func (l *Level) toCheckpointFile(from, to int64, m *MemoryStore) (*CheckpointFile, error) {
	l.lock.RLock()
	defer l.lock.RUnlock()

	retval := &CheckpointFile{
		From:     from,
		To:       to,
		Metrics:  make(map[string]*CheckpointMetrics),
		Children: make(map[string]*CheckpointFile),
	}

	for metric, minfo := range m.Metrics {
		b := l.metrics[minfo.offset]
		if b == nil {
			continue
		}

		allArchived := true
		b.iterFromTo(from, to, func(b *buffer) error {
			if !b.archived {
				allArchived = false
				return fmt.Errorf("stop") // Early termination signal
			}
			return nil
		})

		if allArchived {
			continue
		}

		data := make([]schema.Float, (to-from)/b.frequency+1)
		data, start, end, err := b.read(from, to, data)
		if err != nil {
			return nil, err
		}

		for i := int((end - start) / b.frequency); i < len(data); i++ {
			data[i] = schema.NaN
		}

		retval.Metrics[metric] = &CheckpointMetrics{
			Frequency: b.frequency,
			Start:     start,
			Data:      data,
		}
	}

	for name, child := range l.children {
		val, err := child.toCheckpointFile(from, to, m)
		if err != nil {
			return nil, err
		}

		if val != nil {
			retval.Children[name] = val
		}
	}

	if len(retval.Children) == 0 && len(retval.Metrics) == 0 {
		return nil, nil
	}

	return retval, nil
}

// toCheckpoint writes a Level's data to a JSON checkpoint file.
// Creates directory if needed. Returns ErrNoNewArchiveData if nothing to save.
func (l *Level) toCheckpoint(dir string, from, to int64, m *MemoryStore) error {
	cf, err := l.toCheckpointFile(from, to, m)
	if err != nil {
		return err
	}

	if cf == nil {
		return ErrNoNewArchiveData
	}

	filepath := path.Join(dir, fmt.Sprintf("%d.json", from))
	f, err := os.OpenFile(filepath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil && os.IsNotExist(err) {


   464→func (l *Level) loadFile(cf *CheckpointFile, m *MemoryStore) error {
   465→	for name, metric := range cf.Metrics {
   466→		n := len(metric.Data)
   467→		b := &buffer{
   468→			frequency: metric.Frequency,
   469→			start:     metric.Start,
   470→			data:      metric.Data[0:n:n],
   471→			prev:      nil,
   472→			next:      nil,
   473→			archived:  true,
   474→		}
   475→
   476→		minfo, ok := m.Metrics[name]
   477→		if !ok {
   478→			continue
   479→		}
   480→
   481→		prev := l.metrics[minfo.offset]
   482→		if prev == nil {
   483→			l.metrics[minfo.offset] = b
   484→		} else {
   485→			if prev.start > b.start {
   486→				return fmt.Errorf("[METRICSTORE]> buffer start time %d is before previous buffer start %d", b.start, prev.start)
   487→			}
   488→
   489→			b.prev = prev
   490→			prev.next = b
   491→		}
   492→		l.metrics[minfo.offset] = b
   493→	}
   494→
   495→	if len(cf.Children) > 0 && l.children == nil {
   496→		l.children = make(map[string]*Level)
   497→	}
   498→
   499→	for sel, childCf := range cf.Children {
   500→		child, ok := l.children[sel]
   501→		if !ok {
   502→			child = &Level{
   503→				metrics:  make([]*buffer, len(m.Metrics)),
   504→				children: nil,
   505→			}
   506→			l.children[sel] = child
   507→		}
   508→
   509→		if err := child.loadFile(childCf, m); err != nil {
   510→			return err
   511→		}
   512→	}
   513→
   514→	return nil
   515→}
   516→
   517→// fromCheckpoint loads all checkpoint files (JSON, binary snapshot, WAL) for a
   518→// single host directory. Snapshot files are loaded first (sorted by timestamp),
   519→// then current.wal is replayed on top.
   520→func (l *Level) fromCheckpoint(m *MemoryStore, dir string, from int64) (int, error) {
   521→	direntries, err := os.ReadDir(dir)
   522→	if err != nil {
   523→		if os.IsNotExist(err) {
   524→			return 0, nil
   525→		}
   526→		return 0, err
   527→	}
   528→
   529→	allFiles := make([]fs.DirEntry, 0)
   530→	var walEntry fs.DirEntry
   531→	filesLoaded := 0
   532→
   533→	for _, e := range direntries {
   534→		if e.IsDir() {
   535→			// Legacy: skip subdirectories (only used by old Avro format).
   536→			// These are ignored; their data is not loaded.
   537→			cclog.Debugf("[METRICSTORE]> skipping subdirectory %s in checkpoint dir %s", e.Name(), dir)
   538→			continue
   539→		}
   540→
   541→		name := e.Name()
   542→		if strings.HasSuffix(name, ".json") || strings.HasSuffix(name, ".bin") {
   543→			allFiles = append(allFiles, e)

func (l *Level) loadFile(cf *CheckpointFile, m *MemoryStore) error {
	for name, metric := range cf.Metrics {
		n := len(metric.Data)
		b := &buffer{
			frequency: metric.Frequency,
			start:     metric.Start,
			data:      metric.Data[0:n:n],
			prev:      nil,
			next:      nil,
			archived:  true,
		}

		minfo, ok := m.Metrics[name]
		if !ok {
			continue
		}

		prev := l.metrics[minfo.offset]
		if prev == nil {
			l.metrics[minfo.offset] = b
		} else {
			if prev.start > b.start {
				return fmt.Errorf("[METRICSTORE]> buffer start time %d is before previous buffer start %d", b.start, prev.start)
			}

			b.prev = prev
			prev.next = b
		}
		l.metrics[minfo.offset] = b
	}

	if len(cf.Children) > 0 && l.children == nil {
		l.children = make(map[string]*Level)
	}

	for sel, childCf := range cf.Children {
		child, ok := l.children[sel]
		if !ok {
			child = &Level{
				metrics:  make([]*buffer, len(m.Metrics)),
				children: nil,
			}
			l.children[sel] = child
		}

		if err := child.loadFile(childCf, m); err != nil {
			return err
		}
	}

	return nil
}

// fromCheckpoint loads all checkpoint files (JSON, binary snapshot, WAL) for a
// single host directory. Snapshot files are loaded first (sorted by timestamp),
// then current.wal is replayed on top.
func (l *Level) fromCheckpoint(m *MemoryStore, dir string, from int64) (int, error) {
	direntries, err := os.ReadDir(dir)
	if err != nil {
		if os.IsNotExist(err) {
			return 0, nil
		}
		return 0, err
	}

	allFiles := make([]fs.DirEntry, 0)
	var walEntry fs.DirEntry
	filesLoaded := 0

	for _, e := range direntries {
		if e.IsDir() {
			// Legacy: skip subdirectories (only used by old Avro format).
			// These are ignored; their data is not loaded.
			cclog.Debugf("[METRICSTORE]> skipping subdirectory %s in checkpoint dir %s", e.Name(), dir)
			continue
		}

		name := e.Name()
		if strings.HasSuffix(name, ".json") || strings.HasSuffix(name, ".bin") {
			allFiles = append(allFiles, e)


   619→func findFiles(direntries []fs.DirEntry, t int64, findMoreRecentFiles bool) ([]string, error) {
   620→	nums := map[string]int64{}
   621→	for _, e := range direntries {
   622→		name := e.Name()
   623→		if !strings.HasSuffix(name, ".json") && !strings.HasSuffix(name, ".bin") {
   624→			continue
   625→		}
   626→
   627→		ts, err := parseTimestampFromFilename(name)
   628→		if err != nil {
   629→			return nil, err
   630→		}
   631→		nums[name] = ts
   632→	}
   633→
   634→	sort.Slice(direntries, func(i, j int) bool {
   635→		a, b := direntries[i], direntries[j]
   636→		return nums[a.Name()] < nums[b.Name()]
   637→	})
   638→
   639→	if len(nums) == 0 {
   640→		return nil, nil
   641→	}
   642→
   643→	filenames := make([]string, 0)
   644→
   645→	for i, e := range direntries {
   646→		ts1 := nums[e.Name()]
   647→
   648→		if findMoreRecentFiles && t <= ts1 {
   649→			filenames = append(filenames, e.Name())
   650→		} else if !findMoreRecentFiles && ts1 <= t && ts1 != 0 {
   651→			filenames = append(filenames, e.Name())
   652→		}
   653→
   654→		if i == len(direntries)-1 {
   655→			continue
   656→		}
   657→
   658→		enext := direntries[i+1]
   659→		ts2 := nums[enext.Name()]
   660→
   661→		if findMoreRecentFiles {
   662→			if ts1 < t && t < ts2 {
   663→				filenames = append(filenames, e.Name())
   664→			}
   665→		}
   666→	}
   667→
   668→	return filenames, nil
   669→}
   670→

func findFiles(direntries []fs.DirEntry, t int64, findMoreRecentFiles bool) ([]string, error) {
	nums := map[string]int64{}
	for _, e := range direntries {
		name := e.Name()
		if !strings.HasSuffix(name, ".json") && !strings.HasSuffix(name, ".bin") {
			continue
		}

		ts, err := parseTimestampFromFilename(name)
		if err != nil {
			return nil, err
		}
		nums[name] = ts
	}

	sort.Slice(direntries, func(i, j int) bool {
		a, b := direntries[i], direntries[j]
		return nums[a.Name()] < nums[b.Name()]
	})

	if len(nums) == 0 {
		return nil, nil
	}

	filenames := make([]string, 0)

	for i, e := range direntries {
		ts1 := nums[e.Name()]

		if findMoreRecentFiles && t <= ts1 {
			filenames = append(filenames, e.Name())
		} else if !findMoreRecentFiles && ts1 <= t && ts1 != 0 {
			filenames = append(filenames, e.Name())
		}

		if i == len(direntries)-1 {
			continue
		}

		enext := direntries[i+1]
		ts2 := nums[enext.Name()]

		if findMoreRecentFiles {
			if ts1 < t && t < ts2 {
				filenames = append(filenames, e.Name())
			}
		}
	}

	return filenames, nil
}


Now let me also check how `toCheckpointFile` is used in the binary/WAL path to understand if `archived` is handled differently there.


pkg/metricstore/debug.go:26:	if b.archived {
pkg/metricstore/checkpoint.go:280:		allArchived := true
pkg/metricstore/checkpoint.go:282:			if !b.archived {
pkg/metricstore/checkpoint.go:283:				allArchived = false
pkg/metricstore/checkpoint.go:289:		if allArchived {
pkg/metricstore/buffer.go:187:	b.archived = false


pkg/metricstore/walCheckpoint.go:627:func (m *MemoryStore) ToCheckpointWAL(dir string, from, to int64) (int, []string, error) {


   627→func (m *MemoryStore) ToCheckpointWAL(dir string, from, to int64) (int, []string, error) {
   628→	// Collect all cluster/host pairs.
   629→	m.root.lock.RLock()
   630→	totalHosts := 0
   631→	for _, l1 := range m.root.children {
   632→		l1.lock.RLock()
   633→		totalHosts += len(l1.children)
   634→		l1.lock.RUnlock()
   635→	}
   636→	m.root.lock.RUnlock()
   637→
   638→	levels := make([]*Level, 0, totalHosts)
   639→	selectors := make([][]string, 0, totalHosts)
   640→
   641→	m.root.lock.RLock()
   642→	for sel1, l1 := range m.root.children {
   643→		l1.lock.RLock()
   644→		for sel2, l2 := range l1.children {
   645→			levels = append(levels, l2)
   646→			selectors = append(selectors, []string{sel1, sel2})
   647→		}
   648→		l1.lock.RUnlock()
   649→	}
   650→	m.root.lock.RUnlock()
   651→
   652→	type workItem struct {
   653→		level    *Level
   654→		hostDir  string
   655→		selector []string
   656→	}
   657→
   658→	totalWork := len(levels)
   659→	cclog.Infof("[METRICSTORE]> Starting binary checkpoint for %d hosts with %d workers", totalWork, Keys.NumWorkers)
   660→
   661→	n, errs, completed := int32(0), int32(0), int32(0)
   662→	var successDirs []string
   663→	var successMu sync.Mutex
   664→
   665→	var wg sync.WaitGroup
   666→	wg.Add(Keys.NumWorkers)
   667→	work := make(chan workItem, Keys.NumWorkers*2)
   668→
   669→	// Progress logging goroutine.
   670→	stopProgress := make(chan struct{})
   671→	go func() {
   672→		ticker := time.NewTicker(10 * time.Second)
   673→		defer ticker.Stop()
   674→		for {
   675→			select {
   676→			case <-ticker.C:
   677→				cclog.Infof("[METRICSTORE]> Checkpoint progress: %d/%d hosts (%d written, %d errors)",
   678→					atomic.LoadInt32(&completed), totalWork, atomic.LoadInt32(&n), atomic.LoadInt32(&errs))
   679→			case <-stopProgress:
   680→				return
   681→			}
   682→		}
   683→	}()
   684→
   685→	for range Keys.NumWorkers {
   686→		go func() {
   687→			defer wg.Done()
   688→			for wi := range work {
   689→				err := wi.level.toCheckpointBinary(wi.hostDir, from, to, m)
   690→				if err != nil {
   691→					if err == ErrNoNewArchiveData {
   692→						atomic.AddInt32(&completed, 1)
   693→						continue
   694→					}
   695→					cclog.Errorf("[METRICSTORE]> binary checkpoint error for %s: %v", wi.hostDir, err)
   696→					atomic.AddInt32(&errs, 1)
   697→				} else {
   698→					atomic.AddInt32(&n, 1)
   699→					// Delete WAL immediately after successful snapshot.
   700→					walPath := path.Join(wi.hostDir, "current.wal")
   701→					if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
   702→						cclog.Errorf("[METRICSTORE]> WAL remove %s: %v", walPath, err)
   703→					}
   704→					successMu.Lock()
   705→					successDirs = append(successDirs, wi.hostDir)
   706→					successMu.Unlock()
   707→				}
   708→				atomic.AddInt32(&completed, 1)
   709→			}
   710→		}()
   711→	}
   712→
   713→	for i := range levels {
   714→		hostDir := path.Join(dir, path.Join(selectors[i]...))
   715→		work <- workItem{
   716→			level:    levels[i],
   717→			hostDir:  hostDir,
   718→			selector: selectors[i],
   719→		}
   720→	}
   721→	close(work)
   722→	wg.Wait()
   723→	close(stopProgress)
   724→
   725→	if errs > 0 {
   726→		return int(n), successDirs, fmt.Errorf("[METRICSTORE]> %d errors during binary checkpoint (%d successes)", errs, n)

func (m *MemoryStore) ToCheckpointWAL(dir string, from, to int64) (int, []string, error) {
	// Collect all cluster/host pairs.
	m.root.lock.RLock()
	totalHosts := 0
	for _, l1 := range m.root.children {
		l1.lock.RLock()
		totalHosts += len(l1.children)
		l1.lock.RUnlock()
	}
	m.root.lock.RUnlock()

	levels := make([]*Level, 0, totalHosts)
	selectors := make([][]string, 0, totalHosts)

	m.root.lock.RLock()
	for sel1, l1 := range m.root.children {
		l1.lock.RLock()
		for sel2, l2 := range l1.children {
			levels = append(levels, l2)
			selectors = append(selectors, []string{sel1, sel2})
		}
		l1.lock.RUnlock()
	}
	m.root.lock.RUnlock()

	type workItem struct {
		level    *Level
		hostDir  string
		selector []string
	}

	totalWork := len(levels)
	cclog.Infof("[METRICSTORE]> Starting binary checkpoint for %d hosts with %d workers", totalWork, Keys.NumWorkers)

	n, errs, completed := int32(0), int32(0), int32(0)
	var successDirs []string
	var successMu sync.Mutex

	var wg sync.WaitGroup
	wg.Add(Keys.NumWorkers)
	work := make(chan workItem, Keys.NumWorkers*2)

	// Progress logging goroutine.
	stopProgress := make(chan struct{})
	go func() {
		ticker := time.NewTicker(10 * time.Second)
		defer ticker.Stop()
		for {
			select {
			case <-ticker.C:
				cclog.Infof("[METRICSTORE]> Checkpoint progress: %d/%d hosts (%d written, %d errors)",
					atomic.LoadInt32(&completed), totalWork, atomic.LoadInt32(&n), atomic.LoadInt32(&errs))
			case <-stopProgress:
				return
			}
		}
	}()

	for range Keys.NumWorkers {
		go func() {
			defer wg.Done()
			for wi := range work {
				err := wi.level.toCheckpointBinary(wi.hostDir, from, to, m)
				if err != nil {
					if err == ErrNoNewArchiveData {
						atomic.AddInt32(&completed, 1)
						continue
					}
					cclog.Errorf("[METRICSTORE]> binary checkpoint error for %s: %v", wi.hostDir, err)
					atomic.AddInt32(&errs, 1)
				} else {
					atomic.AddInt32(&n, 1)
					// Delete WAL immediately after successful snapshot.
					walPath := path.Join(wi.hostDir, "current.wal")
					if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
						cclog.Errorf("[METRICSTORE]> WAL remove %s: %v", walPath, err)
					}
					successMu.Lock()
					successDirs = append(successDirs, wi.hostDir)
					successMu.Unlock()
				}
				atomic.AddInt32(&completed, 1)
			}
		}()
	}

	for i := range levels {
		hostDir := path.Join(dir, path.Join(selectors[i]...))
		work <- workItem{
			level:    levels[i],
			hostDir:  hostDir,
			selector: selectors[i],
		}
	}
	close(work)
	wg.Wait()
	close(stopProgress)

	if errs > 0 {
		return int(n), successDirs, fmt.Errorf("[METRICSTORE]> %d errors during binary checkpoint (%d successes)", errs, n)


Now let me check `toCheckpointBinary` to see if it also checks the `archived` flag.


pkg/metricstore/walCheckpoint.go:733:func (l *Level) toCheckpointBinary(dir string, from, to int64, m *MemoryStore) error {


   733→func (l *Level) toCheckpointBinary(dir string, from, to int64, m *MemoryStore) error {
   734→	cf, err := l.toCheckpointFile(from, to, m)
   735→	if err != nil {
   736→		return err
   737→	}
   738→	if cf == nil {
   739→		return ErrNoNewArchiveData
   740→	}
   741→
   742→	if err := os.MkdirAll(dir, CheckpointDirPerms); err != nil {
   743→		return fmt.Errorf("mkdir %s: %w", dir, err)
   744→	}
   745→
   746→	// Write to a temp file first, then rename (atomic on POSIX).
   747→	tmpPath := path.Join(dir, fmt.Sprintf("%d.bin.tmp", from))
   748→	finalPath := path.Join(dir, fmt.Sprintf("%d.bin", from))
   749→
   750→	f, err := os.OpenFile(tmpPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
   751→	if err != nil {
   752→		return fmt.Errorf("open binary snapshot %s: %w", tmpPath, err)
   753→	}
   754→
   755→	bw := bufio.NewWriter(f)
   756→	if err := writeBinarySnapshotFile(bw, cf); err != nil {
   757→		f.Close()
   758→		os.Remove(tmpPath)
   759→		return fmt.Errorf("write binary snapshot: %w", err)
   760→	}
   761→	if err := bw.Flush(); err != nil {
   762→		f.Close()
   763→		os.Remove(tmpPath)
   764→		return err
   765→	}
   766→	f.Close()
   767→
   768→	return os.Rename(tmpPath, finalPath)
   769→}
   770→
   771→// writeBinarySnapshotFile writes the binary snapshot file header and level tree.
   772→func writeBinarySnapshotFile(w io.Writer, cf *CheckpointFile) error {
   773→	if err := binary.Write(w, binary.LittleEndian, snapFileMagic); err != nil {
   774→		return err
   775→	}
   776→	if err := binary.Write(w, binary.LittleEndian, cf.From); err != nil {
   777→		return err
   778→	}
   779→	if err := binary.Write(w, binary.LittleEndian, cf.To); err != nil {
   780→		return err
   781→	}
   782→	return writeBinaryLevel(w, cf)
   783→}
   784→
   785→// writeBinaryLevel recursively writes a CheckpointFile level in binary format.
   786→func writeBinaryLevel(w io.Writer, cf *CheckpointFile) error {
   787→	if err := binary.Write(w, binary.LittleEndian, uint32(len(cf.Metrics))); err != nil {
   788→		return err
   789→	}
   790→
   791→	for name, metric := range cf.Metrics {
   792→		if err := writeString16(w, name); err != nil {
   793→			return err
   794→		}
   795→		if err := binary.Write(w, binary.LittleEndian, metric.Frequency); err != nil {
   796→			return err
   797→		}
   798→		if err := binary.Write(w, binary.LittleEndian, metric.Start); err != nil {
   799→			return err
   800→		}
   801→		if err := binary.Write(w, binary.LittleEndian, uint32(len(metric.Data))); err != nil {
   802→			return err
   803→		}
   804→		for _, v := range metric.Data {
   805→			if err := binary.Write(w, binary.LittleEndian, math.Float32bits(float32(v))); err != nil {
   806→				return err
   807→			}
   808→		}
   809→	}
   810→
   811→	if err := binary.Write(w, binary.LittleEndian, uint32(len(cf.Children))); err != nil {
   812→		return err

func (l *Level) toCheckpointBinary(dir string, from, to int64, m *MemoryStore) error {
	cf, err := l.toCheckpointFile(from, to, m)
	if err != nil {
		return err
	}
	if cf == nil {
		return ErrNoNewArchiveData
	}

	if err := os.MkdirAll(dir, CheckpointDirPerms); err != nil {
		return fmt.Errorf("mkdir %s: %w", dir, err)
	}

	// Write to a temp file first, then rename (atomic on POSIX).
	tmpPath := path.Join(dir, fmt.Sprintf("%d.bin.tmp", from))
	finalPath := path.Join(dir, fmt.Sprintf("%d.bin", from))

	f, err := os.OpenFile(tmpPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil {
		return fmt.Errorf("open binary snapshot %s: %w", tmpPath, err)
	}

	bw := bufio.NewWriter(f)
	if err := writeBinarySnapshotFile(bw, cf); err != nil {
		f.Close()
		os.Remove(tmpPath)
		return fmt.Errorf("write binary snapshot: %w", err)
	}
	if err := bw.Flush(); err != nil {
		f.Close()
		os.Remove(tmpPath)
		return err
	}
	f.Close()

	return os.Rename(tmpPath, finalPath)
}

// writeBinarySnapshotFile writes the binary snapshot file header and level tree.
func writeBinarySnapshotFile(w io.Writer, cf *CheckpointFile) error {
	if err := binary.Write(w, binary.LittleEndian, snapFileMagic); err != nil {
		return err
	}
	if err := binary.Write(w, binary.LittleEndian, cf.From); err != nil {
		return err
	}
	if err := binary.Write(w, binary.LittleEndian, cf.To); err != nil {
		return err
	}
	return writeBinaryLevel(w, cf)
}

// writeBinaryLevel recursively writes a CheckpointFile level in binary format.
func writeBinaryLevel(w io.Writer, cf *CheckpointFile) error {
	if err := binary.Write(w, binary.LittleEndian, uint32(len(cf.Metrics))); err != nil {
		return err
	}

	for name, metric := range cf.Metrics {
		if err := writeString16(w, name); err != nil {
			return err
		}
		if err := binary.Write(w, binary.LittleEndian, metric.Frequency); err != nil {
			return err
		}
		if err := binary.Write(w, binary.LittleEndian, metric.Start); err != nil {
			return err
		}
		if err := binary.Write(w, binary.LittleEndian, uint32(len(metric.Data))); err != nil {
			return err
		}
		for _, v := range metric.Data {
			if err := binary.Write(w, binary.LittleEndian, math.Float32bits(float32(v))); err != nil {
				return err
			}
		}
	}

	if err := binary.Write(w, binary.LittleEndian, uint32(len(cf.Children))); err != nil {
		return err


Good, `toCheckpointBinary` also calls `toCheckpointFile`, which has the `allArchived` skip logic. Both JSON and binary paths are affected.

Now let me also check the JSON checkpoint's `toCheckpoint` for Bug 3 (non-atomic writes).


   328→// toCheckpoint writes a Level's data to a JSON checkpoint file.
   329→// Creates directory if needed. Returns ErrNoNewArchiveData if nothing to save.
   330→func (l *Level) toCheckpoint(dir string, from, to int64, m *MemoryStore) error {
   331→	cf, err := l.toCheckpointFile(from, to, m)
   332→	if err != nil {
   333→		return err
   334→	}
   335→
   336→	if cf == nil {
   337→		return ErrNoNewArchiveData
   338→	}
   339→
   340→	filepath := path.Join(dir, fmt.Sprintf("%d.json", from))
   341→	f, err := os.OpenFile(filepath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
   342→	if err != nil && os.IsNotExist(err) {
   343→		err = os.MkdirAll(dir, CheckpointDirPerms)
   344→		if err == nil {
   345→			f, err = os.OpenFile(filepath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
   346→		}
   347→	}
   348→	if err != nil {
   349→		return err
   350→	}
   351→	defer f.Close()
   352→
   353→	bw := bufio.NewWriter(f)
   354→	if err = json.NewEncoder(bw).Encode(cf); err != nil {
   355→		return err
   356→	}
   357→
   358→	return bw.Flush()
   359→}
   360→
   361→// enqueueCheckpointHosts traverses checkpoint directory and enqueues cluster/host pairs.
   362→// Returns error if directory structure is invalid.
   363→func enqueueCheckpointHosts(dir string, work chan<- [2]string) error {
   364→	clustersDir, err := os.ReadDir(dir)
   365→	if err != nil {
   366→		return err
   367→	}

// toCheckpoint writes a Level's data to a JSON checkpoint file.
// Creates directory if needed. Returns ErrNoNewArchiveData if nothing to save.
func (l *Level) toCheckpoint(dir string, from, to int64, m *MemoryStore) error {
	cf, err := l.toCheckpointFile(from, to, m)
	if err != nil {
		return err
	}

	if cf == nil {
		return ErrNoNewArchiveData
	}

	filepath := path.Join(dir, fmt.Sprintf("%d.json", from))
	f, err := os.OpenFile(filepath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil && os.IsNotExist(err) {
		err = os.MkdirAll(dir, CheckpointDirPerms)
		if err == nil {
			f, err = os.OpenFile(filepath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
		}
	}
	if err != nil {
		return err
	}
	defer f.Close()

	bw := bufio.NewWriter(f)
	if err = json.NewEncoder(bw).Encode(cf); err != nil {
		return err
	}

	return bw.Flush()
}

// enqueueCheckpointHosts traverses checkpoint directory and enqueues cluster/host pairs.
// Returns error if directory structure is invalid.
func enqueueCheckpointHosts(dir string, work chan<- [2]string) error {
	clustersDir, err := os.ReadDir(dir)
	if err != nil {
		return err
	}


Now I have a clear picture. Let me also check the `buffer` struct definition to confirm the `archived` field.


pkg/metricstore/buffer.go:170:type buffer struct {
/Users/jan/tmp/cc-backend/pkg/metricstore/buffer.go-171-	prev      *buffer
/Users/jan/tmp/cc-backend/pkg/metricstore/buffer.go-172-	next      *buffer
/Users/jan/tmp/cc-backend/pkg/metricstore/buffer.go-173-	data      []schema.Float
/Users/jan/tmp/cc-backend/pkg/metricstore/buffer.go-174-	frequency int64
/Users/jan/tmp/cc-backend/pkg/metricstore/buffer.go-175-	start     int64
/Users/jan/tmp/cc-backend/pkg/metricstore/buffer.go-176-	archived  bool
/Users/jan/tmp/cc-backend/pkg/metricstore/buffer.go-177-	closed    bool
/Users/jan/tmp/cc-backend/pkg/metricstore/buffer.go-178-	lastUsed  int64
/Users/jan/tmp/cc-backend/pkg/metricstore/buffer.go-179-}
/Users/jan/tmp/cc-backend/pkg/metricstore/buffer.go-180-


Good. Now I have all the context needed. Let me implement the fixes.

## Fix 1 (Primary): Pass `restoreFrom` to `Checkpointing()` and set `archived: false` on loaded buffers


Edit

func Checkpointing(wg *sync.WaitGroup, ctx context.Context) {
	lastCheckpointMu.Lock()
	lastCheckpoint = time.Now()
	lastCheckpointMu.Unlock()

func Checkpointing(wg *sync.WaitGroup, ctx context.Context, restoreFrom time.Time) {
	lastCheckpointMu.Lock()
	lastCheckpoint = restoreFrom
	lastCheckpointMu.Unlock()