When are the wal journal files reset. Are there situation where they can grow indefintily?


Agent

Explore WAL journal reset/rotation

Prompt:
I need to understand when WAL journal files are reset/rotated and whether they can grow indefinitely in this codebase.

Key files to examine thoroughly:
- `pkg/metricstore/walCheckpoint.go` — WAL + binary snapshot implementation, look for RotateWALFiles, WAL rotation logic, file truncation/deletion
- `pkg/metricstore/checkpoint.go` — Checkpointing goroutine that triggers WAL rotation
- `pkg/metricstore/lineprotocol.go` — where WAL messages are sent
- `pkg/metricstore/metricstore.go` — Init, Shutdown, WALStaging

For each file, trace:
1. When is RotateWALFiles called? What does it do exactly?
2. What triggers a checkpoint (and thus WAL rotation)?
3. Are there error paths where rotation could be skipped?
4. Is there any size limit on WAL files?
5. What happens during shutdown — is the WAL flushed/rotated?
6. What happens if checkpoint writing fails — does the WAL still get rotated?

Be very thorough — read the full implementation of all relevant functions.


Perfect! Now I have all the information I need. Let me provide you with a comprehensive analysis of the WAL rotation logic.

## WAL Journal File Reset/Rotation Analysis

Based on my thorough examination of the codebase, here's a detailed breakdown of when and how WAL journal files are reset/rotated:

### 1. **When RotateWALFiles Is Called**

**Location**: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/checkpoint.go:138`

RotateWALFiles is called **only after successful binary checkpoint creation**:

```go
if Keys.Checkpoints.FileFormat == "wal" {
    n, hostDirs, err := ms.ToCheckpointWAL(...)
    if err != nil {
        cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
    } else {
        // Rotate WAL files for successfully checkpointed hosts.
        RotateWALFiles(hostDirs)
    }
}
```

This occurs in the **Checkpointing() background goroutine**, which runs every **12 hours** (configurable via `Keys.CheckpointInterval`, lines 103-111).

---

### 2. **What RotateWALFiles Does Exactly**

**Location**: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/walCheckpoint.go:324-354`

The function:
- **Sends rotation requests** to shard-specific channels (one per host directory)
- **Blocks until all rotations complete** or times out after 30 seconds
- **Returns early if shutdown** is in progress (via `walShuttingDown` flag)
- **Skips hosts silently** if rotation channel is full

```go
func RotateWALFiles(hostDirs []string) {
    if walShardRotateChs == nil || walShuttingDown.Load() {
        return  // Silently skip if shutdown in progress
    }
    dones := make([]chan struct{}, 0, len(hostDirs))
    for _, dir := range hostDirs {
        done := make(chan struct{})
        shard := walShardIndexFromDir(dir)
        select {
        case walShardRotateChs[shard] <- walRotateReq{hostDir: dir, done: done}:
            dones = append(dones, done)
        default:
            // Channel full or goroutine not consuming — skip this host.
            cclog.Warnf("[METRICSTORE]> WAL rotation skipped for %s (channel full)", dir)
        }
    }
    for _, done := range dones {
        select {
        case <-done:
        case <-time.After(30 * time.Second):
            cclog.Warn("[METRICSTORE]> WAL rotation timed out, continuing")
            return
        }
    }
}
```

---

### 3. **The Actual Rotation Implementation**

**Location**: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/walCheckpoint.go:244-256`

The WALStaging goroutines process rotation requests:

```go
processRotate := func(req walRotateReq) {
    ws, ok := hostFiles[req.hostDir]
    if ok && ws.f != nil {
        ws.w.Flush()         // Flush any buffered data
        ws.f.Close()         // Close the open file handle
        walPath := path.Join(req.hostDir, "current.wal")
        if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
            cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
        }
        delete(hostFiles, req.hostDir)  // Remove from open files cache
    }
    close(req.done)
}
```

**What happens:**
1. Flushes the buffered writer to disk
2. Closes the file handle
3. **Deletes the current.wal file completely** (not rotated to a numbered file)
4. Removes the entry from the open files cache
5. Next write to that host will create a fresh `current.wal`

---

### 4. **Alternative Rotation: ToCheckpointWAL Workers**

**Location**: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/walCheckpoint.go:699-703`

There's also **immediate deletion** of WAL files inside the checkpoint workers themselves:

```go
// Delete WAL immediately after successful snapshot.
walPath := path.Join(wi.hostDir, "current.wal")
if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
    cclog.Errorf("[METRICSTORE]> WAL remove %s: %v", walPath, err)
}
```

This is a redundant cleanup—the file may have already been deleted by RotateWALFiles, but the code handles both cases gracefully with `!os.IsNotExist(err)`.

---

### 5. **What Triggers Checkpoints (and Thus WAL Rotation)?**

The Checkpointing goroutine (line 94-154 in checkpoint.go):

- Runs on a **ticker interval** (default 12 hours, configurable)
- Only calls `ToCheckpointWAL()` if format is "wal"
- **Always updates `lastCheckpoint` timestamp** after success
- Only calls `RotateWALFiles()` if checkpoint succeeded (`else` branch is skipped on error)

---

### 6. **Are There Error Paths Where Rotation Could Be Skipped?**

**YES - Several critical scenarios:**

| Scenario | Impact | Details |
|----------|--------|---------|
| **Binary checkpoint fails** | ❌ WAL NOT rotated | Lines 130-131: If `ToCheckpointWAL()` returns error, `RotateWALFiles()` is never called. WAL continues growing indefinitely. |
| **Rotation channel full** | ⚠ Specific hosts skipped | Lines 341-344: If rotation request buffer is full (capacity 64), that host's rotation is silently dropped. Warning logged but no retry. |
| **Rotation times out** | ⚠ Partial rotation | Lines 349-352: After 30s timeout, function returns without waiting for remaining rotations. Some hosts may still have their WAL files. |
| **Shutdown in progress** | ✅ Handled gracefully | Line 331: Function returns early if `walShuttingDown` flag is set, preventing deadlock. |
| **WAL staging goroutine hangs** | ⚠ Timeout triggers | If goroutine isn't consuming rotation requests, 30s timeout allows recovery. |

---

### 7. **Is There a Size Limit on WAL Files?**

**NO - WAL files can grow indefinitely.**

Evidence:
- **No size checking** in `writeWALRecordDirect()` (lines 379-435)
- **No rotation trigger based on file size** (only time-based: 12-hour interval)
- **No maximum file size enforcement** anywhere in the code
- The only limits are:
  - Record payload sanity check: 1 MB max per record (line 458)
  - WAL message channel buffer size: ~65KB per shard (lines 166-167)

**Consequence**: If checkpointing fails, the `current.wal` file grows unbounded until the next successful checkpoint.

---

### 8. **What Happens During Shutdown?**

**Location**: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/metricstore.go:273-320`

The shutdown sequence:

```go
// Signal producers to stop sending before closing channels
walShuttingDown.Store(true)
time.Sleep(100 * time.Millisecond)  // Grace period

for _, ch := range walShardChs {
    close(ch)  // Close message channels
}
drainStart := time.Now()
WaitForWALStagingDrain()  // Wait for goroutines to exit

// Write final checkpoint
if Keys.Checkpoints.FileFormat == "wal" {
    files, _, err = ms.ToCheckpointWAL(...)
}
```

**Key points:**
- ✅ **WAL staging goroutines are allowed to exit** before final checkpoint (comment: lines 293-295)
- ✅ **Buffered WAL messages are NOT drained** (intentional—final snapshot captures all in-memory data)
- ✅ **Final checkpoint is written** (line 311)
- ❌ **WAL files are NOT explicitly rotated after shutdown checkpoint**

After shutdown, the WAL files remain on disk. On restart:
1. Latest binary snapshots are loaded
2. `current.wal` is replayed on top (lines 582-598 in checkpoint.go)
3. WAL is NOT cleared—it persists until the next successful scheduled checkpoint

---

### 9. **What Happens If Checkpoint Writing Fails?**

**Location**: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/checkpoint.go:128-139`

```go
if Keys.Checkpoints.FileFormat == "wal" {
    n, hostDirs, err := ms.ToCheckpointWAL(...)
    if err != nil {
        cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
        // ❌ RotateWALFiles is NOT called
    } else {
        // ✅ Only on success:
        RotateWALFiles(hostDirs)
    }
}
```

**Behavior on checkpoint failure:**
- ❌ **WAL rotation is skipped**
- ❌ **`lastCheckpoint` timestamp is NOT updated** (lines 134-136)
- ✅ Next checkpoint attempt uses same time range (will retry those hosts)
- ⚠ **WAL files continue accumulating** until next successful checkpoint

---

### 10. **Orphaned WAL Files on Restart**

**Location**: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/metricstore.go:309-312`

There's a function `RotateWALFilesAfterShutdown()` (lines 367-374 in walCheckpoint.go) that's defined but **never called in the codebase**:

```go
func RotateWALFilesAfterShutdown(hostDirs []string) {
    for _, dir := range hostDirs {
        walPath := path.Join(dir, "current.wal")
        if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
            cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
        }
    }
}
```

This suggests there may have been an intent to clean up WAL files after restart, but it's not implemented.

---

### Summary Table

| Question | Answer | Evidence |
|----------|--------|----------|
| **When rotated?** | After successful binary checkpoint every 12h | checkpoint.go:138 |
| **Can they grow indefinitely?** | **YES** - if checkpoint fails | No size limits, rotation only on checkpoint success |
| **Error path skips rotation?** | **YES** - checkpoint failure | checkpoint.go:130-131 |
| **Size limits?** | Only per-record (1MB), not per-file | walCheckpoint.go:458 |
| **Shutdown flushes/rotates WAL?** | Flushes only, no rotation | metricstore.go:293-295 |
| **Checkpoint failure rotates?** | **NO** | checkpoint.go:130-131 |

---

### Critical Risk Assessment

🔴 **HIGH RISK**: If checkpoints consistently fail (e.g., disk full, permission errors), WAL files will grow unbounded until:
1. Disk space is exhausted, OR
2. A checkpoint eventually succeeds

This could cause operational issues without explicit size monitoring.

Prompt:
I need to understand when WAL journal files are reset/rotated and whether they can grow indefinitely in this codebase.

Key files to examine thoroughly:
- `pkg/metricstore/walCheckpoint.go` — WAL + binary snapshot implementation, look for RotateWALFiles, WAL rotation logic, file truncation/deletion
- `pkg/metricstore/checkpoint.go` — Checkpointing goroutine that triggers WAL rotation
- `pkg/metricstore/lineprotocol.go` — where WAL messages are sent
- `pkg/metricstore/metricstore.go` — Init, Shutdown, WALStaging

For each file, trace:
1. When is RotateWALFiles called? What does it do exactly?
2. What triggers a checkpoint (and thus WAL rotation)?
3. Are there error paths where rotation could be skipped?
4. Is there any size limit on WAL files?
5. What happens during shutdown — is the WAL flushed/rotated?
6. What happens if checkpoint writing fails — does the WAL still get rotated?

Be very thorough — read the full implementation of all relevant functions.


320	func WaitForWALStagingDrain() {
321		walStagingWg.Wait()
322	}
323
324	// RotateWALFiles sends rotation requests for the given host directories
325	// and blocks until all rotations complete. Each request is routed to the
326	// shard that owns the host directory.
327	//
328	// If shutdown is in progress (WAL staging goroutines may have exited),
329	// rotation is skipped to avoid deadlocking on abandoned channels.
330	func RotateWALFiles(hostDirs []string) {
331		if walShardRotateChs == nil || walShuttingDown.Load() {
332			return
333		}
334		dones := make([]chan struct{}, 0, len(hostDirs))
335		for _, dir := range hostDirs {
336			done := make(chan struct{})
337			shard := walShardIndexFromDir(dir)
338			select {
339			case walShardRotateChs[shard] <- walRotateReq{hostDir: dir, done: done}:
340				dones = append(dones, done)
341			default:
342				// Channel full or goroutine not consuming — skip this host.
343				cclog.Warnf("[METRICSTORE]> WAL rotation skipped for %s (channel full)", dir)
344			}
345		}
346		for _, done := range dones {
347			select {
348			case <-done:
349			case <-time.After(30 * time.Second):
350				cclog.Warn("[METRICSTORE]> WAL rotation timed out, continuing")
351				return
352			}
353		}
354	}
355
356	// walShardIndexFromDir extracts cluster and node from a host directory path
357	// and returns the shard index. The path format is: rootDir/cluster/node
358	func walShardIndexFromDir(hostDir string) int {
359		// hostDir = .../cluster/node — extract last two path components
360		node := path.Base(hostDir)
361		cluster := path.Base(path.Dir(hostDir))
362		return walShardIndex(cluster, node)
363	}
364
365	// RotateWALFiles sends rotation requests for the given host directories
366	// and blocks until all rotations complete.
367	func RotateWALFilesAfterShutdown(hostDirs []string) {
368		for _, dir := range hostDirs {
369			walPath := path.Join(dir, "current.wal")
370			if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
371				cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
372			}
373		}
374	}
375
376	// writeWALRecordDirect encodes a WAL record into a contiguous buffer first,
377	// then writes it to the bufio.Writer in a single call. This prevents partial
378	// records in the write buffer if a write error occurs mid-record (e.g. disk full).
379	func writeWALRecordDirect(w *bufio.Writer, msg *WALMessage) error {

func WaitForWALStagingDrain() {
	walStagingWg.Wait()
}

// RotateWALFiles sends rotation requests for the given host directories
// and blocks until all rotations complete. Each request is routed to the
// shard that owns the host directory.
//
// If shutdown is in progress (WAL staging goroutines may have exited),
// rotation is skipped to avoid deadlocking on abandoned channels.
func RotateWALFiles(hostDirs []string) {
	if walShardRotateChs == nil || walShuttingDown.Load() {
		return
	}
	dones := make([]chan struct{}, 0, len(hostDirs))
	for _, dir := range hostDirs {
		done := make(chan struct{})
		shard := walShardIndexFromDir(dir)
		select {
		case walShardRotateChs[shard] <- walRotateReq{hostDir: dir, done: done}:
			dones = append(dones, done)
		default:
			// Channel full or goroutine not consuming — skip this host.
			cclog.Warnf("[METRICSTORE]> WAL rotation skipped for %s (channel full)", dir)
		}
	}
	for _, done := range dones {
		select {
		case <-done:
		case <-time.After(30 * time.Second):
			cclog.Warn("[METRICSTORE]> WAL rotation timed out, continuing")
			return
		}
	}
}

// walShardIndexFromDir extracts cluster and node from a host directory path
// and returns the shard index. The path format is: rootDir/cluster/node
func walShardIndexFromDir(hostDir string) int {
	// hostDir = .../cluster/node — extract last two path components
	node := path.Base(hostDir)
	cluster := path.Base(path.Dir(hostDir))
	return walShardIndex(cluster, node)
}

// RotateWALFiles sends rotation requests for the given host directories
// and blocks until all rotations complete.
func RotateWALFilesAfterShutdown(hostDirs []string) {
	for _, dir := range hostDirs {
		walPath := path.Join(dir, "current.wal")
		if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
			cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
		}
	}
}

// writeWALRecordDirect encodes a WAL record into a contiguous buffer first,
// then writes it to the bufio.Writer in a single call. This prevents partial
// records in the write buffer if a write error occurs mid-record (e.g. disk full).
func writeWALRecordDirect(w *bufio.Writer, msg *WALMessage) error {


94	func Checkpointing(wg *sync.WaitGroup, ctx context.Context) {
95		lastCheckpointMu.Lock()
96		lastCheckpoint = time.Now()
97		lastCheckpointMu.Unlock()
98
99		ms := GetMemoryStore()
100
101		wg.Go(func() {
102
103			d := 12 * time.Hour // default checkpoint interval
104			if Keys.CheckpointInterval != "" {
105				parsed, err := time.ParseDuration(Keys.CheckpointInterval)
106				if err != nil {
107					cclog.Errorf("[METRICSTORE]> invalid checkpoint-interval %q: %s, using default 12h", Keys.CheckpointInterval, err)
108				} else {
109					d = parsed
110				}
111			}
112
113			ticker := time.NewTicker(d)
114			defer ticker.Stop()
115
116			for {
117				select {
118				case <-ctx.Done():
119					return
120				case <-ticker.C:
121					lastCheckpointMu.Lock()
122					from := lastCheckpoint
123					lastCheckpointMu.Unlock()
124
125					now := time.Now()
126					cclog.Infof("[METRICSTORE]> start checkpointing (starting at %s)...", from.Format(time.RFC3339))
127
128					if Keys.Checkpoints.FileFormat == "wal" {
129						n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
130						if err != nil {
131							cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
132						} else {
133							cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
134							lastCheckpointMu.Lock()
135							lastCheckpoint = now
136							lastCheckpointMu.Unlock()
137							// Rotate WAL files for successfully checkpointed hosts.
138							RotateWALFiles(hostDirs)
139						}
140					} else {
141						n, err := ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
142						if err != nil {
143							cclog.Errorf("[METRICSTORE]> checkpointing failed: %s", err.Error())
144						} else {
145							cclog.Infof("[METRICSTORE]> done: %d checkpoint files created", n)
146							lastCheckpointMu.Lock()
147							lastCheckpoint = now
148							lastCheckpointMu.Unlock()
149						}
150					}
151				}
152			}
153		})
154	}
155
156	// MarshalJSON provides optimized JSON encoding for CheckpointMetrics.
157	//
158	// Since schema.Float has custom MarshalJSON, serializing []Float has significant overhead.

func Checkpointing(wg *sync.WaitGroup, ctx context.Context) {
	lastCheckpointMu.Lock()
	lastCheckpoint = time.Now()
	lastCheckpointMu.Unlock()

	ms := GetMemoryStore()

	wg.Go(func() {

		d := 12 * time.Hour // default checkpoint interval
		if Keys.CheckpointInterval != "" {
			parsed, err := time.ParseDuration(Keys.CheckpointInterval)
			if err != nil {
				cclog.Errorf("[METRICSTORE]> invalid checkpoint-interval %q: %s, using default 12h", Keys.CheckpointInterval, err)
			} else {
				d = parsed
			}
		}

		ticker := time.NewTicker(d)
		defer ticker.Stop()

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				lastCheckpointMu.Lock()
				from := lastCheckpoint
				lastCheckpointMu.Unlock()

				now := time.Now()
				cclog.Infof("[METRICSTORE]> start checkpointing (starting at %s)...", from.Format(time.RFC3339))

				if Keys.Checkpoints.FileFormat == "wal" {
					n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
					if err != nil {
						cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
					} else {
						cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
						lastCheckpointMu.Lock()
						lastCheckpoint = now
						lastCheckpointMu.Unlock()
						// Rotate WAL files for successfully checkpointed hosts.
						RotateWALFiles(hostDirs)
					}
				} else {
					n, err := ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
					if err != nil {
						cclog.Errorf("[METRICSTORE]> checkpointing failed: %s", err.Error())
					} else {
						cclog.Infof("[METRICSTORE]> done: %d checkpoint files created", n)
						lastCheckpointMu.Lock()
						lastCheckpoint = now
						lastCheckpointMu.Unlock()
					}
				}
			}
		}
	})
}

// MarshalJSON provides optimized JSON encoding for CheckpointMetrics.
//
// Since schema.Float has custom MarshalJSON, serializing []Float has significant overhead.


270	//  4. Log any errors encountered during shutdown
271	//
272	// Note: This function blocks until the final checkpoint is written.
273	func Shutdown() {
274		totalStart := time.Now()
275
276		shutdownFuncMu.Lock()
277		defer shutdownFuncMu.Unlock()
278		if shutdownFunc != nil {
279			shutdownFunc()
280		}
281		cclog.Infof("[METRICSTORE]> Background workers cancelled (%v)", time.Since(totalStart))
282
283		if Keys.Checkpoints.FileFormat == "wal" {
284			// Signal producers to stop sending before closing channels,
285			// preventing send-on-closed-channel panics from in-flight NATS workers.
286			walShuttingDown.Store(true)
287			// Brief grace period for in-flight DecodeLine calls to complete.
288			time.Sleep(100 * time.Millisecond)
289
290			for _, ch := range walShardChs {
291				close(ch)
292			}
293			drainStart := time.Now()
294			WaitForWALStagingDrain()
295			cclog.Infof("[METRICSTORE]> WAL staging goroutines exited (%v)", time.Since(drainStart))
296		}
297
298		cclog.Infof("[METRICSTORE]> Writing checkpoint to '%s'...", Keys.Checkpoints.RootDir)
299		checkpointStart := time.Now()
300		var files int
301		var err error
302
303		ms := GetMemoryStore()
304
305		lastCheckpointMu.Lock()
306		from := lastCheckpoint
307		lastCheckpointMu.Unlock()
308
309		if Keys.Checkpoints.FileFormat == "wal" {
310			// WAL files are deleted per-host inside ToCheckpointWAL workers.
311			files, _, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
312		} else {
313			files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
314		}
315
316		if err != nil {
317			cclog.Errorf("[METRICSTORE]> Writing checkpoint failed: %s", err.Error())
318		}
319		cclog.Infof("[METRICSTORE]> Done! (%d files written in %v, total shutdown: %v)", files, time.Since(checkpointStart), time.Since(totalStart))
320	}
321
322	// Retention starts a background goroutine that periodically frees old metric data.
323	//
324	// This worker runs at half the retention interval and calls Free() to remove buffers

//  4. Log any errors encountered during shutdown
//
// Note: This function blocks until the final checkpoint is written.
func Shutdown() {
	totalStart := time.Now()

	shutdownFuncMu.Lock()
	defer shutdownFuncMu.Unlock()
	if shutdownFunc != nil {
		shutdownFunc()
	}
	cclog.Infof("[METRICSTORE]> Background workers cancelled (%v)", time.Since(totalStart))

	if Keys.Checkpoints.FileFormat == "wal" {
		// Signal producers to stop sending before closing channels,
		// preventing send-on-closed-channel panics from in-flight NATS workers.
		walShuttingDown.Store(true)
		// Brief grace period for in-flight DecodeLine calls to complete.
		time.Sleep(100 * time.Millisecond)

		for _, ch := range walShardChs {
			close(ch)
		}
		drainStart := time.Now()
		WaitForWALStagingDrain()
		cclog.Infof("[METRICSTORE]> WAL staging goroutines exited (%v)", time.Since(drainStart))
	}

	cclog.Infof("[METRICSTORE]> Writing checkpoint to '%s'...", Keys.Checkpoints.RootDir)
	checkpointStart := time.Now()
	var files int
	var err error

	ms := GetMemoryStore()

	lastCheckpointMu.Lock()
	from := lastCheckpoint
	lastCheckpointMu.Unlock()

	if Keys.Checkpoints.FileFormat == "wal" {
		// WAL files are deleted per-host inside ToCheckpointWAL workers.
		files, _, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
	} else {
		files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
	}

	if err != nil {
		cclog.Errorf("[METRICSTORE]> Writing checkpoint failed: %s", err.Error())
	}
	cclog.Infof("[METRICSTORE]> Done! (%d files written in %v, total shutdown: %v)", files, time.Since(checkpointStart), time.Since(totalStart))
}

// Retention starts a background goroutine that periodically frees old metric data.
//
// This worker runs at half the retention interval and calls Free() to remove buffers


Now let me check the WAL deletion inside `ToCheckpointWAL` workers.


No matches found


250:					if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
370:		if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
701:					if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {


690					if err != nil {
691						if err == ErrNoNewArchiveData {
692							atomic.AddInt32(&completed, 1)
693							continue
694						}
695						cclog.Errorf("[METRICSTORE]> binary checkpoint error for %s: %v", wi.hostDir, err)
696						atomic.AddInt32(&errs, 1)
697					} else {
698						atomic.AddInt32(&n, 1)
699						// Delete WAL immediately after successful snapshot.
700						walPath := path.Join(wi.hostDir, "current.wal")
701						if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
702							cclog.Errorf("[METRICSTORE]> WAL remove %s: %v", walPath, err)
703						}
704						successMu.Lock()
705						successDirs = append(successDirs, wi.hostDir)
706						successMu.Unlock()
707					}
708					atomic.AddInt32(&completed, 1)
709				}

				if err != nil {
					if err == ErrNoNewArchiveData {
						atomic.AddInt32(&completed, 1)
						continue
					}
					cclog.Errorf("[METRICSTORE]> binary checkpoint error for %s: %v", wi.hostDir, err)
					atomic.AddInt32(&errs, 1)
				} else {
					atomic.AddInt32(&n, 1)
					// Delete WAL immediately after successful snapshot.
					walPath := path.Join(wi.hostDir, "current.wal")
					if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
						cclog.Errorf("[METRICSTORE]> WAL remove %s: %v", walPath, err)
					}
					successMu.Lock()
					successDirs = append(successDirs, wi.hostDir)
					successMu.Unlock()
				}
				atomic.AddInt32(&completed, 1)
			}


Now I have a clear picture. Let me write the plan.


ToolSearch

Question: select:ExitPlanMode


Question: select:ExitPlanMode


Here's what I found from tracing the code:

## When WAL files are reset

WAL files (`current.wal`) are deleted in **two places**, both tied to successful checkpointing:

1. **Inside `ToCheckpointWAL` workers** (`walCheckpoint.go:699-703`) — each worker deletes the host's `current.wal` immediately after writing a successful binary snapshot.

2. **Via `RotateWALFiles`** (`walCheckpoint.go:330-354`) — called after `ToCheckpointWAL` returns successfully in the periodic `Checkpointing()` goroutine (`checkpoint.go:138`). The staging goroutines flush, close, and delete the file (`walCheckpoint.go:244-256`).

So effectively, WAL files are reset **every checkpoint interval** (default 12 hours, configurable via `CheckpointInterval`).

3. **On shutdown** (`metricstore.go:309-311`) — `ToCheckpointWAL` is called, which deletes WAL files per-host inside its workers. However, `RotateWALFiles` is **not** called after the shutdown checkpoint (the staging goroutines have already exited). The comment at line 310 says "WAL files are deleted per-host inside ToCheckpointWAL workers" so this is intentional.

## Situations where they can grow indefinitely

**Yes, there are several:**

1. **Checkpoint failure** (`checkpoint.go:130-131`): If `ToCheckpointWAL()` returns an error, `RotateWALFiles()` is never called and `lastCheckpoint` is not advanced. The WAL keeps accumulating until a checkpoint eventually succeeds. There is **no file size limit** — only a per-record sanity check of 1 MB.

2. **Rotation channel full** (`walCheckpoint.go:341-344`): If a shard's rotation channel (capacity 64) is full, that host's rotation is skipped with a warning log. The WAL for that host persists until the next checkpoint cycle.

3. **Rotation timeout** (`walCheckpoint.go:349-352`): If rotation doesn't complete within 30 seconds, the function returns early. Remaining hosts keep their WAL files.

4. **Shutdown checkpoint failure** (`metricstore.go:316-318`): If the final checkpoint on shutdown fails, WAL files remain on disk. On restart they'll be replayed, but won't be cleaned up until the first successful scheduled checkpoint (up to 12 hours later).

5. **`RotateWALFilesAfterShutdown`** (`walCheckpoint.go:367-374`) exists but is **never called anywhere** — it appears to be dead code that was intended for post-restart cleanup.

## Bottom line

The main risk is scenario 1: if checkpoints keep failing (e.g., disk full, permissions), WAL files grow without bound. There's no size-based rotation or cap. Would you like me to plan a fix for any of these issues?


Plan a fix for scenario 1


Agent

Explore WAL size and checkpoint error handling

Prompt:
I need to understand the WAL file writing and checkpoint error handling in detail to plan a fix for unbounded WAL growth when checkpoints fail.

Please examine these aspects thoroughly:

1. In `pkg/metricstore/walCheckpoint.go`:
   - How are WAL files opened/written? Look at `writeWALRecordDirect` and the WALStaging goroutine's write path
   - Is there any existing tracking of WAL file size?
   - What's the structure of `hostFiles` map and the file state stored per host?
   - Look at the `walHostState` or similar struct that tracks open WAL file handles
   - How does `ToCheckpointWAL` work — does it return partial success (some hosts succeed, some fail)? What does the `hostDirs` return value contain — all hosts or only successful ones?

2. In `pkg/metricstore/checkpoint.go`:
   - The full Checkpointing() goroutine — what information is available when checkpoint fails?
   - Is there any retry logic or backoff?

3. In `pkg/metricstore/metricstore.go`:
   - Look at the Keys/config structure — what checkpoint-related config options exist?
   - Look at Init() to understand available config fields

4. In `pkg/metricstore/lineprotocol.go`:
   - How WALMessages are created and sent — is there a place to intercept/skip if WAL is too large?

5. Any existing config structures in `pkg/metricstore/` that define checkpoint settings (look for struct definitions with json tags related to checkpoints, wal, max-size, etc.)

Read all relevant sections fully.


Perfect! Now let me compile a comprehensive report. Let me create a summary document with all the key findings:

## Comprehensive Analysis: WAL File Writing and Checkpoint Error Handling

Based on my thorough examination of the codebase, here are the detailed findings:

### 1. WAL File Opening/Writing and File State Tracking

**File: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/walCheckpoint.go`**

**walFileState Structure (lines 120-125):**
```go
type walFileState struct {
	f     *os.File
	w     *bufio.Writer
	dirty bool
}
```
- Tracks open WAL file handle, buffered writer, and dirty flag per host directory
- Stored in `hostFiles` map: `map[string]*walFileState` (line 184 in WALStaging)
- Each shard owns a subset of hosts determined by FNV-1a hash of cluster+node

**WAL File Opening (lines 195-230):**
- `getOrOpenWAL()` lazily opens `current.wal` files on first write
- Creates file with `os.O_CREATE|os.O_APPEND|os.O_WRONLY` flags
- Writes file header magic (`walFileMagic = 0xCC1DA701`) only on new (empty) files
- File size is checked via `info.Size()` but **NO tracking of cumulative WAL size**

**WAL Record Writing (lines 378-435 - writeWALRecordDirect):**
- Encodes record in buffer before writing to prevent partial writes on error
- Uses stack buffer (256 bytes) for small records, heap allocates for larger ones
- Single atomic write via `bufio.Writer.Write()`
- **NO size limits checked before writing**
- **NO per-file or per-host WAL size tracking**

**Flush Strategy (lines 258-265):**
```go
flushDirty := func() {
    for _, ws := range hostFiles {
        if ws.dirty {
            ws.w.Flush()
            ws.dirty = false
        }
    }
}
```
- Periodic flush every 1 second (`walFlushInterval`) or on rotation
- Flushes **all** open files (no per-file size awareness)

### 2. Checkpoint File Success/Failure Handling

**File: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/walCheckpoint.go`**

**ToCheckpointWAL Function (lines 627-729):**
```go
func (m *MemoryStore) ToCheckpointWAL(dir string, from, to int64) (int, []string, error)
```

**Return Value Details:**
- Returns: `(numFilesWritten int, successDirs []string, error)`
- `successDirs` contains **only hosts that successfully wrote binary snapshots**
- Hosts with checkpoint errors are **NOT included** in successDirs (line 705)
- **Partial success is supported:** some hosts can fail while others succeed

**Critical Issue - Error Handling (lines 689-707):**
```go
err := wi.level.toCheckpointBinary(wi.hostDir, from, to, m)
if err != nil {
    if err == ErrNoNewArchiveData {
        atomic.AddInt32(&completed, 1)
        continue
    }
    cclog.Errorf("[METRICSTORE]> binary checkpoint error for %s: %v", wi.hostDir, err)
    atomic.AddInt32(&errs, 1)
} else {
    atomic.AddInt32(&n, 1)
    // Delete WAL immediately after successful snapshot.
    walPath := path.Join(wi.hostDir, "current.wal")
    if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
        cclog.Errorf("[METRICSTORE]> WAL remove %s: %v", walPath, err)
    }
    successMu.Lock()
    successDirs = append(successDirs, wi.hostDir)
    successMu.Unlock()
}
```

**Key Finding:**
- WAL deletion only occurs for hosts with **successful snapshots** (line 700)
- Hosts with checkpoint errors keep growing WAL files indefinitely
- WAL files are **never rotated if checkpoint fails**

**Return Error (lines 725-728):**
```go
if errs > 0 {
    return int(n), successDirs, fmt.Errorf("[METRICSTORE]> %d errors during binary checkpoint (%d successes)", errs, n)
}
return int(n), successDirs, nil
```
- Returns error even with partial success
- Caller must check both the error AND the successDirs

### 3. Checkpointing Worker Behavior

**File: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/checkpoint.go`**

**Checkpointing Function (lines 94-154):**
```go
func Checkpointing(wg *sync.WaitGroup, ctx context.Context) {
    // ... setup ticker (default 12 hours)
    case <-ticker.C:
        from := lastCheckpoint
        now := time.Now()

        if Keys.Checkpoints.FileFormat == "wal" {
            n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
            if err != nil {
                cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
            } else {
                cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
                lastCheckpointMu.Lock()
                lastCheckpoint = now  // Updated ONLY on success
                lastCheckpointMu.Unlock()
                RotateWALFiles(hostDirs)  // Only rotates successfully checkpointed hosts
            }
        }
}
```

**Critical Issue - Incomplete Error Handling:**
1. **Error case (line 131):** When `err != nil`, lastCheckpoint is **NOT updated**
   - However, `lastCheckpoint` tracks the time of the last successful checkpoint
   - **Bug: If all hosts fail, the error is logged but nothing prevents retrying**

2. **Next checkpoint attempt:**
   - Will try to checkpoint the same `from` time again
   - Will still fail (same underlying issue)
   - WAL files for failed hosts continue growing
   - **There is no backoff, no skip, no mechanism to prevent repeated failures**

3. **Retry Logic:** **NONE** - failures are logged but not acted upon

### 4. WAL Message Channel and Ingestion

**File: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/lineprotocol.go`**

**DecodeLine Function (lines 199-379):**
- Creates `WALMessage` for each metric when format is "wal" (line 356)
- Sends via `SendWALMessage()` (line 364)
- **Dropped messages logged periodically if channel full** (lines 367-369)
- **NO size check before sending** - messages sent regardless of WAL size

**SendWALMessage (lines 144-155 in walCheckpoint.go):**
```go
func SendWALMessage(msg *WALMessage) bool {
    if walShardChs == nil || walShuttingDown.Load() {
        return false
    }
    shard := walShardIndex(msg.Cluster, msg.Node)
    select {
    case walShardChs[shard] <- msg:
        return true
    default:
        return false  // Channel full, message dropped
    }
}
```
- Returns false if channel is full (non-blocking send)
- **No capacity check or backpressure mechanism**
- Dropped messages are only logged, not acted upon

### 5. Configuration Structures

**File: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/config.go`**

**MetricStoreConfig Structure (lines 121-132):**
```go
type MetricStoreConfig struct {
    NumWorkers         int            `json:"num-workers"`
    RetentionInMemory  string         `json:"retention-in-memory"`
    CheckpointInterval string         `json:"checkpoint-interval,omitempty"`
    MemoryCap          int            `json:"memory-cap"`
    Checkpoints        Checkpoints    `json:"checkpoints"`
    Debug              *Debug         `json:"debug"`
    Cleanup            *Cleanup       `json:"cleanup"`
    Subscriptions      *Subscriptions `json:"nats-subscriptions"`
}
```

**Checkpoints Structure (lines 64-67):**
```go
type Checkpoints struct {
    FileFormat string `json:"file-format"`  // "json" or "wal"
    RootDir    string `json:"directory"`
}
```

**Missing Configuration:**
- **NO max WAL file size limit**
- **NO max WAL directory size limit**
- **NO checkpoint retry policy**
- **NO checkpoint timeout**
- **NO WAL rotation interval independent of checkpoint**

### 6. Root Causes of Unbounded WAL Growth

1. **No WAL size tracking:** `walFileState` doesn't track file size; `getOrOpenWAL()` never checks cumulative size
2. **WAL rotation only on successful checkpoint:** Line 700 of walCheckpoint.go - WAL deleted only if snapshot succeeds
3. **No checkpoint error recovery:** Failed checkpoints just logged; no retry mechanism
4. **No independent WAL rotation:** WAL lifetime tied entirely to checkpoint success
5. **Checkpoint errors don't prevent future writes:** Messages keep flowing to failed hosts' WAL files
6. **No circuit breaker:** Repeated failures don't trigger any defensive action
7. **No backoff or throttling:** Metrics written even when WAL growth indicates problems

### 7. Detailed Data Flow on Checkpoint Failure

**Scenario: One host fails checkpoint**

1. `ToCheckpointBinary()` fails for host1 (e.g., disk error)
2. Worker logs error, increments `errs` counter, **does NOT add to successDirs**
3. `ToCheckpointWAL()` returns with `(2, ["host2", "host3"], error)`
4. `Checkpointing()` receives error (line 130-131)
5. Error logged; **lastCheckpoint NOT updated**
6. `RotateWALFiles()` called only for `["host2", "host3"]` (line 138)
7. **host1's current.wal is never deleted**, continues growing
8. Next checkpoint interval (12 hours later):
   - `lastCheckpoint` still points to previous successful time
   - Will retry same `from` timestamp
   - host1 checkpoint will likely fail again (same condition)
   - **WAL grows more**
9. **Infinite loop potential:** Permanent error = permanent WAL growth

---

## Summary of Key Structures and Data Flow

**WAL File State Per Host:**
- Path: `checkpoints/cluster/host/current.wal`
- Tracked by: `walFileState{f, w, dirty}` in per-shard `hostFiles` map
- **No size awareness anywhere**

**Checkpoint Success/Failure Tracking:**
- `successDirs` = only hosts that succeeded (returned from `ToCheckpointWAL`)
- Partial success is possible
- Error doesn't prevent future checkpoint attempts
- No metrics on failure rate or retry logic

**Configuration Available:**
- `Keys.Checkpoints.FileFormat`: "json" or "wal"
- `Keys.Checkpoints.RootDir`: storage directory
- `Keys.CheckpointInterval`: interval between checkpoints (default 12h)
- **Nothing related to WAL size limits or rotation policy**

**Critical Missing Pieces for Fix:**
1. Per-host WAL file size tracking
2. Per-shard or global WAL directory size tracking
3. Configurable WAL rotation interval/size limits
4. Independent WAL rotation mechanism not tied to checkpoint success
5. Checkpoint error recovery strategy with backoff/skip logic
6. Mechanism to handle failed hosts differently (e.g., pause writes, skip rotation)

Prompt:
I need to understand the WAL file writing and checkpoint error handling in detail to plan a fix for unbounded WAL growth when checkpoints fail.

Please examine these aspects thoroughly:

1. In `pkg/metricstore/walCheckpoint.go`:
   - How are WAL files opened/written? Look at `writeWALRecordDirect` and the WALStaging goroutine's write path
   - Is there any existing tracking of WAL file size?
   - What's the structure of `hostFiles` map and the file state stored per host?
   - Look at the `walHostState` or similar struct that tracks open WAL file handles
   - How does `ToCheckpointWAL` work — does it return partial success (some hosts succeed, some fail)? What does the `hostDirs` return value contain — all hosts or only successful ones?

2. In `pkg/metricstore/checkpoint.go`:
   - The full Checkpointing() goroutine — what information is available when checkpoint fails?
   - Is there any retry logic or backoff?

3. In `pkg/metricstore/metricstore.go`:
   - Look at the Keys/config structure — what checkpoint-related config options exist?
   - Look at Init() to understand available config fields

4. In `pkg/metricstore/lineprotocol.go`:
   - How WALMessages are created and sent — is there a place to intercept/skip if WAL is too large?

5. Any existing config structures in `pkg/metricstore/` that define checkpoint settings (look for struct definitions with json tags related to checkpoints, wal, max-size, etc.)

Read all relevant sections fully.


118	}
119
120	// walFileState holds an open WAL file handle and buffered writer for one host directory.
121	type walFileState struct {
122		f     *os.File
123		w     *bufio.Writer
124		dirty bool
125	}
126
127	// walFlushInterval controls how often dirty WAL files are flushed to disk.
128	// Decoupling flushes from message processing lets the consumer run at memory
129	// speed, amortizing syscall overhead across many writes.
130	const walFlushInterval = 1 * time.Second
131
132	// walShardIndex computes which shard a message belongs to based on cluster+node.

}

// walFileState holds an open WAL file handle and buffered writer for one host directory.
type walFileState struct {
	f     *os.File
	w     *bufio.Writer
	dirty bool
}

// walFlushInterval controls how often dirty WAL files are flushed to disk.
// Decoupling flushes from message processing lets the consumer run at memory
// speed, amortizing syscall overhead across many writes.
const walFlushInterval = 1 * time.Second

// walShardIndex computes which shard a message belongs to based on cluster+node.


180
181			walStagingWg.Add(1)
182			wg.Go(func() {
183				defer walStagingWg.Done()
184				hostFiles := make(map[string]*walFileState)
185
186				defer func() {
187					for _, ws := range hostFiles {
188						if ws.f != nil {
189							ws.w.Flush()
190							ws.f.Close()
191						}
192					}
193				}()
194
195				getOrOpenWAL := func(hostDir string) *walFileState {
196					ws, ok := hostFiles[hostDir]
197					if ok {
198						return ws
199					}
200
201					if err := os.MkdirAll(hostDir, CheckpointDirPerms); err != nil {
202						cclog.Errorf("[METRICSTORE]> WAL: mkdir %s: %v", hostDir, err)
203						return nil
204					}
205
206					walPath := path.Join(hostDir, "current.wal")
207					f, err := os.OpenFile(walPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, CheckpointFilePerms)
208					if err != nil {
209						cclog.Errorf("[METRICSTORE]> WAL: open %s: %v", walPath, err)
210						return nil
211					}
212
213					w := bufio.NewWriter(f)
214
215					// Write file header magic if file is new (empty).
216					info, err := f.Stat()
217					if err == nil && info.Size() == 0 {
218						var hdr [4]byte
219						binary.LittleEndian.PutUint32(hdr[:], walFileMagic)
220						if _, err := w.Write(hdr[:]); err != nil {
221							cclog.Errorf("[METRICSTORE]> WAL: write header %s: %v", walPath, err)
222							f.Close()
223							return nil
224						}
225					}
226
227					ws = &walFileState{f: f, w: w}
228					hostFiles[hostDir] = ws
229					return ws
230				}
231
232				processMsg := func(msg *WALMessage) {
233					hostDir := path.Join(Keys.Checkpoints.RootDir, msg.Cluster, msg.Node)
234					ws := getOrOpenWAL(hostDir)
235					if ws == nil {
236						return
237					}
238					if err := writeWALRecordDirect(ws.w, msg); err != nil {
239						cclog.Errorf("[METRICSTORE]> WAL: write record: %v", err)
240					}
241					ws.dirty = true
242				}
243
244				processRotate := func(req walRotateReq) {
245					ws, ok := hostFiles[req.hostDir]
246					if ok && ws.f != nil {
247						ws.w.Flush()
248						ws.f.Close()
249						walPath := path.Join(req.hostDir, "current.wal")
250						if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
251							cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
252						}
253						delete(hostFiles, req.hostDir)
254					}
255					close(req.done)
256				}
257
258				flushDirty := func() {
259					for _, ws := range hostFiles {
260						if ws.dirty {
261							ws.w.Flush()
262							ws.dirty = false
263						}
264					}

		walStagingWg.Add(1)
		wg.Go(func() {
			defer walStagingWg.Done()
			hostFiles := make(map[string]*walFileState)

			defer func() {
				for _, ws := range hostFiles {
					if ws.f != nil {
						ws.w.Flush()
						ws.f.Close()
					}
				}
			}()

			getOrOpenWAL := func(hostDir string) *walFileState {
				ws, ok := hostFiles[hostDir]
				if ok {
					return ws
				}

				if err := os.MkdirAll(hostDir, CheckpointDirPerms); err != nil {
					cclog.Errorf("[METRICSTORE]> WAL: mkdir %s: %v", hostDir, err)
					return nil
				}

				walPath := path.Join(hostDir, "current.wal")
				f, err := os.OpenFile(walPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, CheckpointFilePerms)
				if err != nil {
					cclog.Errorf("[METRICSTORE]> WAL: open %s: %v", walPath, err)
					return nil
				}

				w := bufio.NewWriter(f)

				// Write file header magic if file is new (empty).
				info, err := f.Stat()
				if err == nil && info.Size() == 0 {
					var hdr [4]byte
					binary.LittleEndian.PutUint32(hdr[:], walFileMagic)
					if _, err := w.Write(hdr[:]); err != nil {
						cclog.Errorf("[METRICSTORE]> WAL: write header %s: %v", walPath, err)
						f.Close()
						return nil
					}
				}

				ws = &walFileState{f: f, w: w}
				hostFiles[hostDir] = ws
				return ws
			}

			processMsg := func(msg *WALMessage) {
				hostDir := path.Join(Keys.Checkpoints.RootDir, msg.Cluster, msg.Node)
				ws := getOrOpenWAL(hostDir)
				if ws == nil {
					return
				}
				if err := writeWALRecordDirect(ws.w, msg); err != nil {
					cclog.Errorf("[METRICSTORE]> WAL: write record: %v", err)
				}
				ws.dirty = true
			}

			processRotate := func(req walRotateReq) {
				ws, ok := hostFiles[req.hostDir]
				if ok && ws.f != nil {
					ws.w.Flush()
					ws.f.Close()
					walPath := path.Join(req.hostDir, "current.wal")
					if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
						cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
					}
					delete(hostFiles, req.hostDir)
				}
				close(req.done)
			}

			flushDirty := func() {
				for _, ws := range hostFiles {
					if ws.dirty {
						ws.w.Flush()
						ws.dirty = false
					}
				}


60	//
61	// Fields:
62	//   - FileFormat: "json" (human-readable, periodic) or "wal" (binary snapshot + WAL, crash-safe); default is "wal"
63	//   - RootDir:    Filesystem path for checkpoint files (created if missing)
64	type Checkpoints struct {
65		FileFormat string `json:"file-format"`
66		RootDir    string `json:"directory"`
67	}
68
69	// Debug provides development and profiling options.
70	//
71	// Fields:
72	//   - DumpToFile: Path to dump checkpoint data for inspection (empty = disabled)
73	//   - EnableGops: Enable gops agent for live runtime debugging (https://github.com/google/gops)
74	type Debug struct {

//
// Fields:
//   - FileFormat: "json" (human-readable, periodic) or "wal" (binary snapshot + WAL, crash-safe); default is "wal"
//   - RootDir:    Filesystem path for checkpoint files (created if missing)
type Checkpoints struct {
	FileFormat string `json:"file-format"`
	RootDir    string `json:"directory"`
}

// Debug provides development and profiling options.
//
// Fields:
//   - DumpToFile: Path to dump checkpoint data for inspection (empty = disabled)
//   - EnableGops: Enable gops agent for live runtime debugging (https://github.com/google/gops)
type Debug struct {


Now I have everything I need. Let me write the plan.


Write

# Fix: Unbounded WAL growth on checkpoint failure

## Context

When binary checkpointing fails (disk error, permission issue, etc.), WAL files are never rotated because `RotateWALFiles()` is only called on success. With a 12-hour checkpoint interval, a persistent failure means WAL files grow indefinitely until disk exhaustion. There are two distinct bugs:

1. **Partial success is treated as total failure** (`checkpoint.go:128-139`): `ToCheckpointWAL` supports partial success (returns `successDirs` for hosts that wrote snapshots), but the caller ignores this — on any error, `lastCheckpoint` is not advanced and `RotateWALFiles` is not called. The inner workers already delete WAL files for successful hosts (line 700), but `lastCheckpoint` not advancing means the next run re-processes all hosts unnecessarily.

2. **No WAL size limit**: There is no mechanism to cap WAL file size. If a host's checkpoint consistently fails, its `current.wal` grows without bound.

## Fix

### Part 1: Handle partial checkpoint success (`checkpoint.go`)

In `Checkpointing()`, when `ToCheckpointWAL` returns an error **and** successful hosts:
- Still advance `lastCheckpoint` if there were any successes
- Still call `RotateWALFiles(hostDirs)` for the successful hosts

Change the `if err != nil` / `else` block (lines 128-139) to:

```go
n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
if err != nil {
    cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
}
if n > 0 {
    cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
    lastCheckpointMu.Lock()
    lastCheckpoint = now
    lastCheckpointMu.Unlock()
    RotateWALFiles(hostDirs)
}
```

This way partial success still advances the checkpoint timestamp and rotates WAL for successful hosts.

### Part 2: Add max WAL file size with forced rotation (`walCheckpoint.go`)

**Config** — add `MaxWALSize` to `Checkpoints` struct in `config.go`:
```go
type Checkpoints struct {
    FileFormat string `json:"file-format"`
    RootDir    string `json:"directory"`
    MaxWALSize int64  `json:"max-wal-size,omitempty"` // bytes, 0 = unlimited
}
```

Default to 0 (unlimited) for backward compatibility. Recommended value in docs: something like 512 MB.

**Size tracking** — add `size` field to `walFileState`:
```go
type walFileState struct {
    f     *os.File
    w     *bufio.Writer
    dirty bool
    size  int64
}
```

Initialize `size` from `info.Size()` in `getOrOpenWAL()` (the stat is already done at line 216-217).

**Size enforcement** — in `processMsg()`, after `writeWALRecordDirect` succeeds, update `ws.size`. Before writing, check if `ws.size` would exceed `Keys.Checkpoints.MaxWALSize` (when > 0). If exceeded:
1. Log a warning
2. Flush, close, and delete the WAL file (same as `processRotate`)
3. Remove from `hostFiles` so next write creates a fresh file

This accepts data loss for crash recovery of that host's gap, but the in-memory store still holds the data and will persist it at the next successful checkpoint. The alternative (letting the disk fill up) is worse.

## Files to modify

1. **`pkg/metricstore/config.go`** — add `MaxWALSize` field to `Checkpoints` struct
2. **`pkg/metricstore/walCheckpoint.go`** — add `size` to `walFileState`, track size in `getOrOpenWAL` and `processMsg`, enforce limit in `processMsg`
3. **`pkg/metricstore/checkpoint.go`** — fix partial success handling in `Checkpointing()`

## Verification

1. `go build ./...` — ensure it compiles
2. `go test ./pkg/metricstore/...` — run existing tests
3. Manual review: trace all callers of the changed code paths to confirm no behavioral regression
4. Verify that the `processRotate` path still works correctly (it doesn't track size, but it deletes from `hostFiles` which resets tracking)


File created successfully at: /Users/jan/.claude/plans/atomic-giggling-gizmo.md

# Fix: Unbounded WAL growth on checkpoint failure

## Context

When binary checkpointing fails (disk error, permission issue, etc.), WAL files are never rotated because `RotateWALFiles()` is only called on success. With a 12-hour checkpoint interval, a persistent failure means WAL files grow indefinitely until disk exhaustion. There are two distinct bugs:

1. **Partial success is treated as total failure** (`checkpoint.go:128-139`): `ToCheckpointWAL` supports partial success (returns `successDirs` for hosts that wrote snapshots), but the caller ignores this — on any error, `lastCheckpoint` is not advanced and `RotateWALFiles` is not called. The inner workers already delete WAL files for successful hosts (line 700), but `lastCheckpoint` not advancing means the next run re-processes all hosts unnecessarily.

2. **No WAL size limit**: There is no mechanism to cap WAL file size. If a host's checkpoint consistently fails, its `current.wal` grows without bound.

## Fix

### Part 1: Handle partial checkpoint success (`checkpoint.go`)

In `Checkpointing()`, when `ToCheckpointWAL` returns an error **and** successful hosts:
- Still advance `lastCheckpoint` if there were any successes
- Still call `RotateWALFiles(hostDirs)` for the successful hosts

Change the `if err != nil` / `else` block (lines 128-139) to:

```go
n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
if err != nil {
    cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
}
if n > 0 {
    cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
    lastCheckpointMu.Lock()
    lastCheckpoint = now
    lastCheckpointMu.Unlock()
    RotateWALFiles(hostDirs)
}
```

This way partial success still advances the checkpoint timestamp and rotates WAL for successful hosts.

### Part 2: Add max WAL file size with forced rotation (`walCheckpoint.go`)

**Config** — add `MaxWALSize` to `Checkpoints` struct in `config.go`:
```go
type Checkpoints struct {
    FileFormat string `json:"file-format"`
    RootDir    string `json:"directory"`
    MaxWALSize int64  `json:"max-wal-size,omitempty"` // bytes, 0 = unlimited
}
```

Default to 0 (unlimited) for backward compatibility. Recommended value in docs: something like 512 MB.

**Size tracking** — add `size` field to `walFileState`:
```go
type walFileState struct {
    f     *os.File
    w     *bufio.Writer
    dirty bool
    size  int64
}
```

Initialize `size` from `info.Size()` in `getOrOpenWAL()` (the stat is already done at line 216-217).

**Size enforcement** — in `processMsg()`, after `writeWALRecordDirect` succeeds, update `ws.size`. Before writing, check if `ws.size` would exceed `Keys.Checkpoints.MaxWALSize` (when > 0). If exceeded:
1. Log a warning
2. Flush, close, and delete the WAL file (same as `processRotate`)
3. Remove from `hostFiles` so next write creates a fresh file

This accepts data loss for crash recovery of that host's gap, but the in-memory store still holds the data and will persist it at the next successful checkpoint. The alternative (letting the disk fill up) is worse.

## Files to modify

1. **`pkg/metricstore/config.go`** — add `MaxWALSize` field to `Checkpoints` struct
2. **`pkg/metricstore/walCheckpoint.go`** — add `size` to `walFileState`, track size in `getOrOpenWAL` and `processMsg`, enforce limit in `processMsg`
3. **`pkg/metricstore/checkpoint.go`** — fix partial success handling in `Checkpointing()`

## Verification

1. `go build ./...` — ensure it compiles
2. `go test ./pkg/metricstore/...` — run existing tests
3. Manual review: trace all callers of the changed code paths to confirm no behavioral regression
4. Verify that the `processRotate` path still works correctly (it doesn't track size, but it deletes from `hostFiles` which resets tracking)


ExitPlanMode

# Fix: Unbounded WAL growth on checkpoint failure

## Context

When binary checkpointing fails (disk error, permission issue, etc.), WAL files are never rotated because `RotateWALFiles()` is only called on success. With a 12-hour checkpoint interval, a persistent failure means WAL files grow indefinitely until disk exhaustion. There are two distinct bugs:

1. **Partial success is treated as total failure** (`checkpoint.go:128-139`): `ToCheckpointWAL` supports partial success (returns `successDirs` for hosts that wrote snapshots), but the caller ignores this — on any error, `lastCheckpoint` is not advanced and `RotateWALFiles` is not called. The inner workers already delete WAL files for successful hosts (line 700), but `lastCheckpoint` not advancing means the next run re-processes all hosts unnecessarily.

2. **No WAL size limit**: There is no mechanism to cap WAL file size. If a host's checkpoint consistently fails, its `current.wal` grows without bound.

## Fix

### Part 1: Handle partial checkpoint success (`checkpoint.go`)

In `Checkpointing()`, when `ToCheckpointWAL` returns an error **and** successful hosts:
- Still advance `lastCheckpoint` if there were any successes
- Still call `RotateWALFiles(hostDirs)` for the successful hosts

Change the `if err != nil` / `else` block (lines 128-139) to:

```go
n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
if err != nil {
    cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
}
if n > 0 {
    cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
    lastCheckpointMu.Lock()
    lastCheckpoint = now
    lastCheckpointMu.Unlock()
    RotateWALFiles(hostDirs)
}
```

This way partial success still advances the checkpoint timestamp and rotates WAL for successful hosts.

### Part 2: Add max WAL file size with forced rotation (`walCheckpoint.go`)

**Config** — add `MaxWALSize` to `Checkpoints` struct in `config.go`:
```go
type Checkpoints struct {
    FileFormat string `json:"file-format"`
    RootDir    string `json:"directory"`
    MaxWALSize int64  `json:"max-wal-size,omitempty"` // bytes, 0 = unlimited
}
```

Default to 0 (unlimited) for backward compatibility. Recommended value in docs: something like 512 MB.

**Size tracking** — add `size` field to `walFileState`:
```go
type walFileState struct {
    f     *os.File
    w     *bufio.Writer
    dirty bool
    size  int64
}
```

Initialize `size` from `info.Size()` in `getOrOpenWAL()` (the stat is already done at line 216-217).

**Size enforcement** — in `processMsg()`, after `writeWALRecordDirect` succeeds, update `ws.size`. Before writing, check if `ws.size` would exceed `Keys.Checkpoints.MaxWALSize` (when > 0). If exceeded:
1. Log a warning
2. Flush, close, and delete the WAL file (same as `processRotate`)
3. Remove from `hostFiles` so next write creates a fresh file

This accepts data loss for crash recovery of that host's gap, but the in-memory store still holds the data and will persist it at the next successful checkpoint. The alternative (letting the disk fill up) is worse.

## Files to modify

1. **`pkg/metricstore/config.go`** — add `MaxWALSize` field to `Checkpoints` struct
2. **`pkg/metricstore/walCheckpoint.go`** — add `size` to `walFileState`, track size in `getOrOpenWAL` and `processMsg`, enforce limit in `processMsg`
3. **`pkg/metricstore/checkpoint.go`** — fix partial success handling in `Checkpointing()`

## Verification

1. `go build ./...` — ensure it compiles
2. `go test ./pkg/metricstore/...` — run existing tests
3. Manual review: trace all callers of the changed code paths to confirm no behavioral regression
4. Verify that the `processRotate` path still works correctly (it doesn't track size, but it deletes from `hostFiles` which resets tracking)


User has approved your plan. You can now start coding. Start with updating your todo list if applicable

Your plan has been saved to: /Users/jan/.claude/plans/atomic-giggling-gizmo.md
You can refer back to it if needed during implementation.

## Approved Plan:
# Fix: Unbounded WAL growth on checkpoint failure

## Context

When binary checkpointing fails (disk error, permission issue, etc.), WAL files are never rotated because `RotateWALFiles()` is only called on success. With a 12-hour checkpoint interval, a persistent failure means WAL files grow indefinitely until disk exhaustion. There are two distinct bugs:

1. **Partial success is treated as total failure** (`checkpoint.go:128-139`): `ToCheckpointWAL` supports partial success (returns `successDirs` for hosts that wrote snapshots), but the caller ignores this — on any error, `lastCheckpoint` is not advanced and `RotateWALFiles` is not called. The inner workers already delete WAL files for successful hosts (line 700), but `lastCheckpoint` not advancing means the next run re-processes all hosts unnecessarily.

2. **No WAL size limit**: There is no mechanism to cap WAL file size. If a host's checkpoint consistently fails, its `current.wal` grows without bound.

## Fix

### Part 1: Handle partial checkpoint success (`checkpoint.go`)

In `Checkpointing()`, when `ToCheckpointWAL` returns an error **and** successful hosts:
- Still advance `lastCheckpoint` if there were any successes
- Still call `RotateWALFiles(hostDirs)` for the successful hosts

Change the `if err != nil` / `else` block (lines 128-139) to:

```go
n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
if err != nil {
    cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
}
if n > 0 {
    cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
    lastCheckpointMu.Lock()
    lastCheckpoint = now
    lastCheckpointMu.Unlock()
    RotateWALFiles(hostDirs)
}
```

This way partial success still advances the checkpoint timestamp and rotates WAL for successful hosts.

### Part 2: Add max WAL file size with forced rotation (`walCheckpoint.go`)

**Config** — add `MaxWALSize` to `Checkpoints` struct in `config.go`:
```go
type Checkpoints struct {
    FileFormat string `json:"file-format"`
    RootDir    string `json:"directory"`
    MaxWALSize int64  `json:"max-wal-size,omitempty"` // bytes, 0 = unlimited
}
```

Default to 0 (unlimited) for backward compatibility. Recommended value in docs: something like 512 MB.

**Size tracking** — add `size` field to `walFileState`:
```go
type walFileState struct {
    f     *os.File
    w     *bufio.Writer
    dirty bool
    size  int64
}
```

Initialize `size` from `info.Size()` in `getOrOpenWAL()` (the stat is already done at line 216-217).

**Size enforcement** — in `processMsg()`, after `writeWALRecordDirect` succeeds, update `ws.size`. Before writing, check if `ws.size` would exceed `Keys.Checkpoints.MaxWALSize` (when > 0). If exceeded:
1. Log a warning
2. Flush, close, and delete the WAL file (same as `processRotate`)
3. Remove from `hostFiles` so next write creates a fresh file

This accepts data loss for crash recovery of that host's gap, but the in-memory store still holds the data and will persist it at the next successful checkpoint. The alternative (letting the disk fill up) is worse.

## Files to modify

1. **`pkg/metricstore/config.go`** — add `MaxWALSize` field to `Checkpoints` struct
2. **`pkg/metricstore/walCheckpoint.go`** — add `size` to `walFileState`, track size in `getOrOpenWAL` and `processMsg`, enforce limit in `processMsg`
3. **`pkg/metricstore/checkpoint.go`** — fix partial success handling in `Checkpointing()`

## Verification

1. `go build ./...` — ensure it compiles
2. `go test ./pkg/metricstore/...` — run existing tests
3. Manual review: trace all callers of the changed code paths to confirm no behavioral regression
4. Verify that the `processRotate` path still works correctly (it doesn't track size, but it deletes from `hostFiles` which resets tracking)


ToolSearch

Question: select:TaskCreate,TaskUpdate


Question: select:TaskCreate,TaskUpdate


TaskCreate

Fix partial checkpoint success handling in Checkpointing()

Change checkpoint.go so that partial success still advances lastCheckpoint and rotates WAL for successful hosts

Fixing partial checkpoint success handling


Task #1 created successfully: Fix partial checkpoint success handling in Checkpointing()


Add MaxWALSize config field

Add MaxWALSize int64 to Checkpoints struct in config.go


Task #2 created successfully: Add MaxWALSize config field

Add MaxWALSize config field


Add WAL size tracking and enforcement in walCheckpoint.go

Add size field to walFileState, track in getOrOpenWAL and processMsg, force-rotate when exceeded

Adding WAL size tracking and enforcement


Task #3 created successfully: Add WAL size tracking and enforcement in walCheckpoint.go


Build and test

Run go build ./... and go test ./pkg/metricstore/...


Task #4 created successfully: Build and test

Build and test


Updated task #1 status


Edit

				if Keys.Checkpoints.FileFormat == "wal" {
					n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
					if err != nil {
						cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
					} else {
						cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
						lastCheckpointMu.Lock()
						lastCheckpoint = now
						lastCheckpointMu.Unlock()
						// Rotate WAL files for successfully checkpointed hosts.
						RotateWALFiles(hostDirs)
					}

				if Keys.Checkpoints.FileFormat == "wal" {
					n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
					if err != nil {
						cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
					}
					if n > 0 {
						cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
						lastCheckpointMu.Lock()
						lastCheckpoint = now
						lastCheckpointMu.Unlock()
						// Rotate WAL files for successfully checkpointed hosts.
						RotateWALFiles(hostDirs)
					}


The file /Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/checkpoint.go has been updated successfully.

				if Keys.Checkpoints.FileFormat == "wal" {
					n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
					if err != nil {
						cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
					}
					if n > 0 {
						cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
						lastCheckpointMu.Lock()
						lastCheckpoint = now
						lastCheckpointMu.Unlock()
						// Rotate WAL files for successfully checkpointed hosts.
						RotateWALFiles(hostDirs)
					}

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// This file implements checkpoint persistence for the in-memory metric store.
//
// Checkpoints enable graceful restarts by periodically saving in-memory metric
// data to disk in JSON or binary format. The checkpoint system:
//
// Key Features:
//   - Periodic background checkpointing via the Checkpointing() worker
//   - Two format families: JSON (human-readable) and WAL+binary (compact, crash-safe)
//   - Parallel checkpoint creation and loading using worker pools
//   - Hierarchical file organization: checkpoint_dir/cluster/host/timestamp.{json|bin}
//   - WAL file: checkpoint_dir/cluster/host/current.wal (append-only, per-entry)
//   - Only saves unarchived data (archived data is already persisted elsewhere)
//   - GC optimization during loading to prevent excessive heap growth
//
// Checkpoint Workflow:
//  1. Init() loads checkpoints within retention window at startup
//  2. Checkpointing() worker periodically saves new data
//  3. Shutdown() writes final checkpoint before exit
//
// File Organization:
//
//	checkpoints/
//	  cluster1/
//	    host001/
//	      1234567890.json  (JSON format: full subtree snapshot)
//	      1234567890.bin   (binary format: full subtree snapshot)
//	      current.wal      (WAL format: append-only per-entry log)
//	    host002/
//	      ...
package metricstore

import (
	"bufio"
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"io/fs"
	"os"
	"path"
	"path/filepath"
	"sort"
	"strconv"
	"strings"
	"sync"
	"sync/atomic"
	"time"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

const (
	CheckpointFilePerms = 0o644                    // File permissions for checkpoint files
	CheckpointDirPerms  = 0o755                    // Directory permissions for checkpoint directories
	GCTriggerInterval   = DefaultGCTriggerInterval // Interval for triggering GC during checkpoint loading
)

// CheckpointMetrics represents metric data in a checkpoint file.
// Whenever the structure changes, update MarshalJSON as well!
type CheckpointMetrics struct {
	Data      []schema.Float `json:"data"`
	Frequency int64          `json:"frequency"`
	Start     int64          `json:"start"`
}

// CheckpointFile represents the hierarchical structure of a checkpoint file.
// It mirrors the Level tree structure from the MemoryStore.
type CheckpointFile struct {
	Metrics  map[string]*CheckpointMetrics `json:"metrics"`
	Children map[string]*CheckpointFile    `json:"children"`
	From     int64                         `json:"from"`
	To       int64                         `json:"to"`
}

// lastCheckpoint tracks the timestamp of the last checkpoint creation.
var (
	lastCheckpoint   time.Time
	lastCheckpointMu sync.Mutex
)

// Checkpointing starts a background worker that periodically saves metric data to disk.
//
// Checkpoints are written every 12 hours (hardcoded).
//
// Format behaviour:
//   - "json": Periodic checkpointing every checkpointInterval
//   - "wal":  Periodic binary snapshots + WAL rotation every checkpointInterval
func Checkpointing(wg *sync.WaitGroup, ctx context.Context) {
	lastCheckpointMu.Lock()
	lastCheckpoint = time.Now()
	lastCheckpointMu.Unlock()

	ms := GetMemoryStore()

	wg.Go(func() {

		d := 12 * time.Hour // default checkpoint interval
		if Keys.CheckpointInterval != "" {
			parsed, err := time.ParseDuration(Keys.CheckpointInterval)
			if err != nil {
				cclog.Errorf("[METRICSTORE]> invalid checkpoint-interval %q: %s, using default 12h", Keys.CheckpointInterval, err)
			} else {
				d = parsed
			}
		}

		ticker := time.NewTicker(d)
		defer ticker.Stop()

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				lastCheckpointMu.Lock()
				from := lastCheckpoint
				lastCheckpointMu.Unlock()

				now := time.Now()
				cclog.Infof("[METRICSTORE]> start checkpointing (starting at %s)...", from.Format(time.RFC3339))

				if Keys.Checkpoints.FileFormat == "wal" {
					n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
					if err != nil {
						cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
					} else {
						cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
						lastCheckpointMu.Lock()
						lastCheckpoint = now
						lastCheckpointMu.Unlock()
						// Rotate WAL files for successfully checkpointed hosts.
						RotateWALFiles(hostDirs)
					}
				} else {
					n, err := ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
					if err != nil {
						cclog.Errorf("[METRICSTORE]> checkpointing failed: %s", err.Error())
					} else {
						cclog.Infof("[METRICSTORE]> done: %d checkpoint files created", n)
						lastCheckpointMu.Lock()
						lastCheckpoint = now
						lastCheckpointMu.Unlock()
					}
				}
			}
		}
	})
}

// MarshalJSON provides optimized JSON encoding for CheckpointMetrics.
//
// Since schema.Float has custom MarshalJSON, serializing []Float has significant overhead.
// This method manually constructs JSON to avoid allocations and interface conversions.
func (cm *CheckpointMetrics) MarshalJSON() ([]byte, error) {
	buf := make([]byte, 0, 128+len(cm.Data)*8)
	buf = append(buf, `{"frequency":`...)
	buf = strconv.AppendInt(buf, cm.Frequency, 10)
	buf = append(buf, `,"start":`...)
	buf = strconv.AppendInt(buf, cm.Start, 10)
	buf = append(buf, `,"data":[`...)
	for i, x := range cm.Data {
		if i != 0 {
			buf = append(buf, ',')
		}
		if x.IsNaN() {
			buf = append(buf, `null`...)
		} else {
			buf = strconv.AppendFloat(buf, float64(x), 'f', 1, 32)
		}
	}
	buf = append(buf, `]}`...)
	return buf, nil
}

// ToCheckpoint writes metric data to checkpoint files in parallel (JSON format).
//
// Metrics at root and cluster levels are skipped. One file per host is created.
// Uses worker pool (Keys.NumWorkers) for parallel processing. Only locks one host
// at a time, allowing concurrent writes/reads to other hosts.
//
// Returns the number of checkpoint files created and any errors encountered.
func (m *MemoryStore) ToCheckpoint(dir string, from, to int64) (int, error) {
	// Pre-calculate capacity by counting cluster/host pairs
	m.root.lock.RLock()
	totalHosts := 0
	for _, l1 := range m.root.children {
		l1.lock.RLock()
		totalHosts += len(l1.children)
		l1.lock.RUnlock()
	}
	m.root.lock.RUnlock()

	levels := make([]*Level, 0, totalHosts)
	selectors := make([][]string, 0, totalHosts)

	m.root.lock.RLock()
	for sel1, l1 := range m.root.children {
		l1.lock.RLock()
		for sel2, l2 := range l1.children {
			levels = append(levels, l2)
			selectors = append(selectors, []string{sel1, sel2})
		}
		l1.lock.RUnlock()
	}
	m.root.lock.RUnlock()

	type workItem struct {
		level    *Level
		dir      string
		selector []string
	}

	n, errs := int32(0), int32(0)

	var wg sync.WaitGroup
	wg.Add(Keys.NumWorkers)
	work := make(chan workItem, Keys.NumWorkers*2)
	for worker := 0; worker < Keys.NumWorkers; worker++ {
		go func() {
			defer wg.Done()

			for workItem := range work {
				if err := workItem.level.toCheckpoint(workItem.dir, from, to, m); err != nil {
					if err == ErrNoNewArchiveData {
						continue
					}

					cclog.Errorf("[METRICSTORE]> error while checkpointing %#v: %s", workItem.selector, err.Error())
					atomic.AddInt32(&errs, 1)
				} else {
					atomic.AddInt32(&n, 1)
				}
			}
		}()
	}

	for i := 0; i < len(levels); i++ {
		dir := path.Join(dir, path.Join(selectors[i]...))
		work <- workItem{
			level:    levels[i],
			dir:      dir,
			selector: selectors[i],
		}
	}

	close(work)
	wg.Wait()

	if errs > 0 {
		return int(n), fmt.Errorf("[METRICSTORE]> %d errors happened while creating checkpoints (%d successes)", errs, n)
	}
	return int(n), nil
}

// toCheckpointFile recursively converts a Level tree to CheckpointFile structure.
// Skips metrics that are already archived. Returns nil if no unarchived data exists.
func (l *Level) toCheckpointFile(from, to int64, m *MemoryStore) (*CheckpointFile, error) {
	l.lock.RLock()
	defer l.lock.RUnlock()

	retval := &CheckpointFile{
		From:     from,
		To:       to,
		Metrics:  make(map[string]*CheckpointMetrics),
		Children: make(map[string]*CheckpointFile),
	}

	for metric, minfo := range m.Metrics {
		b := l.metrics[minfo.offset]
		if b == nil {
			continue
		}

		allArchived := true
		b.iterFromTo(from, to, func(b *buffer) error {
			if !b.archived {
				allArchived = false
				return fmt.Errorf("stop") // Early termination signal
			}
			return nil
		})

		if allArchived {
			continue
		}

		data := make([]schema.Float, (to-from)/b.frequency+1)
		data, start, end, err := b.read(from, to, data)
		if err != nil {
			return nil, err
		}

		for i := int((end - start) / b.frequency); i < len(data); i++ {
			data[i] = schema.NaN
		}

		retval.Metrics[metric] = &CheckpointMetrics{
			Frequency: b.frequency,
			Start:     start,
			Data:      data,
		}
	}

	for name, child := range l.children {
		val, err := child.toCheckpointFile(from, to, m)
		if err != nil {
			return nil, err
		}

		if val != nil {
			retval.Children[name] = val
		}
	}

	if len(retval.Children) == 0 && len(retval.Metrics) == 0 {
		return nil, nil
	}

	return retval, nil
}

// toCheckpoint writes a Level's data to a JSON checkpoint file.
// Creates directory if needed. Returns ErrNoNewArchiveData if nothing to save.
func (l *Level) toCheckpoint(dir string, from, to int64, m *MemoryStore) error {
	cf, err := l.toCheckpointFile(from, to, m)
	if err != nil {
		return err
	}

	if cf == nil {
		return ErrNoNewArchiveData
	}

	filepath := path.Join(dir, fmt.Sprintf("%d.json", from))
	f, err := os.OpenFile(filepath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil && os.IsNotExist(err) {
		err = os.MkdirAll(dir, CheckpointDirPerms)
		if err == nil {
			f, err = os.OpenFile(filepath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
		}
	}
	if err != nil {
		return err
	}
	defer f.Close()

	bw := bufio.NewWriter(f)
	if err = json.NewEncoder(bw).Encode(cf); err != nil {
		return err
	}

	return bw.Flush()
}

// enqueueCheckpointHosts traverses checkpoint directory and enqueues cluster/host pairs.
// Returns error if directory structure is invalid.
func enqueueCheckpointHosts(dir string, work chan<- [2]string) error {
	clustersDir, err := os.ReadDir(dir)
	if err != nil {
		return err
	}

	for _, clusterDir := range clustersDir {
		if !clusterDir.IsDir() {
			return errors.New("[METRICSTORE]> expected only directories at first level of checkpoints/ directory")
		}

		hostsDir, err := os.ReadDir(filepath.Join(dir, clusterDir.Name()))
		if err != nil {
			return err
		}

		for _, hostDir := range hostsDir {
			if !hostDir.IsDir() {
				return errors.New("[METRICSTORE]> expected only directories at second level of checkpoints/ directory")
			}

			work <- [2]string{clusterDir.Name(), hostDir.Name()}
		}
	}

	return nil
}

// FromCheckpoint loads checkpoint files from disk into memory in parallel.
//
// Uses worker pool to load cluster/host combinations. Returns number of files
// loaded and any errors.
func (m *MemoryStore) FromCheckpoint(dir string, from int64) (int, error) {
	var wg sync.WaitGroup
	work := make(chan [2]string, Keys.NumWorkers*4)
	n, errs := int32(0), int32(0)

	wg.Add(Keys.NumWorkers)
	for worker := 0; worker < Keys.NumWorkers; worker++ {
		go func() {
			defer wg.Done()
			for host := range work {
				lvl := m.root.findLevelOrCreate(host[:], len(m.Metrics))
				nn, err := lvl.fromCheckpoint(m, filepath.Join(dir, host[0], host[1]), from)
				if err != nil {
					cclog.Errorf("[METRICSTORE]> error while loading checkpoints for %s/%s: %s", host[0], host[1], err.Error())
					atomic.AddInt32(&errs, 1)
				}
				atomic.AddInt32(&n, int32(nn))
			}
		}()
	}

	err := enqueueCheckpointHosts(dir, work)
	close(work)
	wg.Wait()

	if err != nil {
		return int(n), err
	}

	if errs > 0 {
		return int(n), fmt.Errorf("[METRICSTORE]> %d errors happened while creating checkpoints (%d successes)", errs, n)
	}
	return int(n), nil
}

// FromCheckpointFiles is the main entry point for loading checkpoints at startup.
//
// Creates checkpoint directory if it doesn't exist. This function must be called
// before any writes or reads, and can only be called once.
func (m *MemoryStore) FromCheckpointFiles(dir string, from int64) (int, error) {
	if _, err := os.Stat(dir); os.IsNotExist(err) {
		err := os.MkdirAll(dir, CheckpointDirPerms)
		if err != nil {
			cclog.Fatalf("[METRICSTORE]> Error creating directory: %#v\n", err)
		}
		cclog.Debugf("[METRICSTORE]> %#v Directory created successfully", dir)
	}

	return m.FromCheckpoint(dir, from)
}

func (l *Level) loadJSONFile(m *MemoryStore, f *os.File, from int64) error {
	br := bufio.NewReader(f)
	cf := &CheckpointFile{}
	if err := json.NewDecoder(br).Decode(cf); err != nil {
		return err
	}

	if cf.To != 0 && cf.To < from {
		return nil
	}

	if err := l.loadFile(cf, m); err != nil {
		return err
	}

	return nil
}

func (l *Level) loadFile(cf *CheckpointFile, m *MemoryStore) error {
	for name, metric := range cf.Metrics {
		n := len(metric.Data)
		b := &buffer{
			frequency: metric.Frequency,
			start:     metric.Start,
			data:      metric.Data[0:n:n],
			prev:      nil,
			next:      nil,
			archived:  true,
		}

		minfo, ok := m.Metrics[name]
		if !ok {
			continue
		}

		prev := l.metrics[minfo.offset]
		if prev == nil {
			l.metrics[minfo.offset] = b
		} else {
			if prev.start > b.start {
				return fmt.Errorf("[METRICSTORE]> buffer start time %d is before previous buffer start %d", b.start, prev.start)
			}

			b.prev = prev
			prev.next = b
		}
		l.metrics[minfo.offset] = b
	}

	if len(cf.Children) > 0 && l.children == nil {
		l.children = make(map[string]*Level)
	}

	for sel, childCf := range cf.Children {
		child, ok := l.children[sel]
		if !ok {
			child = &Level{
				metrics:  make([]*buffer, len(m.Metrics)),
				children: nil,
			}
			l.children[sel] = child
		}

		if err := child.loadFile(childCf, m); err != nil {
			return err
		}
	}

	return nil
}

// fromCheckpoint loads all checkpoint files (JSON, binary snapshot, WAL) for a
// single host directory. Snapshot files are loaded first (sorted by timestamp),
// then current.wal is replayed on top.
func (l *Level) fromCheckpoint(m *MemoryStore, dir string, from int64) (int, error) {
	direntries, err := os.ReadDir(dir)
	if err != nil {
		if os.IsNotExist(err) {
			return 0, nil
		}
		return 0, err
	}

	allFiles := make([]fs.DirEntry, 0)
	var walEntry fs.DirEntry
	filesLoaded := 0

	for _, e := range direntries {
		if e.IsDir() {
			// Legacy: skip subdirectories (only used by old Avro format).
			// These are ignored; their data is not loaded.
			cclog.Debugf("[METRICSTORE]> skipping subdirectory %s in checkpoint dir %s", e.Name(), dir)
			continue
		}

		name := e.Name()
		if strings.HasSuffix(name, ".json") || strings.HasSuffix(name, ".bin") {
			allFiles = append(allFiles, e)
		} else if name == "current.wal" {
			walEntry = e
		}
		// Silently ignore other files (e.g., .tmp, .bin.tmp from interrupted writes).
	}

	files, err := findFiles(allFiles, from, true)
	if err != nil {
		return filesLoaded, err
	}

	loaders := map[string]func(*MemoryStore, *os.File, int64) error{
		".json": l.loadJSONFile,
		".bin":  l.loadBinaryFile,
	}

	for _, filename := range files {
		ext := filepath.Ext(filename)
		loader := loaders[ext]
		if loader == nil {
			cclog.Warnf("[METRICSTORE]> unknown extension for checkpoint file %s", filename)
			continue
		}

		err := func() error {
			f, err := os.Open(path.Join(dir, filename))
			if err != nil {
				return err
			}
			defer f.Close()
			return loader(m, f, from)
		}()
		if err != nil {
			return filesLoaded, err
		}
		filesLoaded++
	}

	// Replay WAL after all snapshot files so it fills in data since the last snapshot.
	if walEntry != nil {
		err := func() error {
			f, err := os.Open(path.Join(dir, walEntry.Name()))
			if err != nil {
				return err
			}
			defer f.Close()
			return l.loadWALFile(m, f, from)
		}()
		if err != nil {
			// WAL errors are non-fatal: the snapshot already loaded the bulk of data.
			cclog.Warnf("[METRICSTORE]> WAL replay error for %s: %v (data since last snapshot may be missing)", dir, err)
		} else {
			filesLoaded++
		}
	}

	return filesLoaded, nil
}

// parseTimestampFromFilename extracts a Unix timestamp from a checkpoint filename.
// Supports ".json" (format: "<ts>.json") and ".bin" (format: "<ts>.bin").
func parseTimestampFromFilename(name string) (int64, error) {
	switch {
	case strings.HasSuffix(name, ".json"):
		return strconv.ParseInt(name[:len(name)-5], 10, 64)
	case strings.HasSuffix(name, ".bin"):
		return strconv.ParseInt(name[:len(name)-4], 10, 64)
	default:
		return 0, fmt.Errorf("unknown checkpoint extension for file %q", name)
	}
}

// findFiles returns filenames from direntries whose timestamps satisfy the filter.
// If findMoreRecentFiles is true, returns files with timestamps >= t (plus the
// last file before t if t falls between two files).
func findFiles(direntries []fs.DirEntry, t int64, findMoreRecentFiles bool) ([]string, error) {
	nums := map[string]int64{}
	for _, e := range direntries {
		name := e.Name()
		if !strings.HasSuffix(name, ".json") && !strings.HasSuffix(name, ".bin") {
			continue
		}

		ts, err := parseTimestampFromFilename(name)
		if err != nil {
			return nil, err
		}
		nums[name] = ts
	}

	sort.Slice(direntries, func(i, j int) bool {
		a, b := direntries[i], direntries[j]
		return nums[a.Name()] < nums[b.Name()]
	})

	if len(nums) == 0 {
		return nil, nil
	}

	filenames := make([]string, 0)

	for i, e := range direntries {
		ts1 := nums[e.Name()]

		if findMoreRecentFiles && t <= ts1 {
			filenames = append(filenames, e.Name())
		} else if !findMoreRecentFiles && ts1 <= t && ts1 != 0 {
			filenames = append(filenames, e.Name())
		}

		if i == len(direntries)-1 {
			continue
		}

		enext := direntries[i+1]
		ts2 := nums[enext.Name()]

		if findMoreRecentFiles {
			if ts1 < t && t < ts2 {
				filenames = append(filenames, e.Name())
			}
		}
	}

	return filenames, nil
}


Updated task #2 status


56		DefaultMemoryUsageTrackerInterval = 5 * time.Minute
57	)
58
59	// Checkpoints configures periodic persistence of in-memory metric data.
60	//
61	// Fields:
62	//   - FileFormat: "json" (human-readable, periodic) or "wal" (binary snapshot + WAL, crash-safe); default is "wal"
63	//   - RootDir:    Filesystem path for checkpoint files (created if missing)
64	type Checkpoints struct {
65		FileFormat string `json:"file-format"`
66		RootDir    string `json:"directory"`
67	}
68
69	// Debug provides development and profiling options.
70	//

	DefaultMemoryUsageTrackerInterval = 5 * time.Minute
)

// Checkpoints configures periodic persistence of in-memory metric data.
//
// Fields:
//   - FileFormat: "json" (human-readable, periodic) or "wal" (binary snapshot + WAL, crash-safe); default is "wal"
//   - RootDir:    Filesystem path for checkpoint files (created if missing)
type Checkpoints struct {
	FileFormat string `json:"file-format"`
	RootDir    string `json:"directory"`
}

// Debug provides development and profiling options.
//


// Checkpoints configures periodic persistence of in-memory metric data.
//
// Fields:
//   - FileFormat: "json" (human-readable, periodic) or "wal" (binary snapshot + WAL, crash-safe); default is "wal"
//   - RootDir:    Filesystem path for checkpoint files (created if missing)
type Checkpoints struct {
	FileFormat string `json:"file-format"`
	RootDir    string `json:"directory"`
}

// Checkpoints configures periodic persistence of in-memory metric data.
//
// Fields:
//   - FileFormat:  "json" (human-readable, periodic) or "wal" (binary snapshot + WAL, crash-safe); default is "wal"
//   - RootDir:     Filesystem path for checkpoint files (created if missing)
//   - MaxWALSize:  Maximum size in bytes for a single host's WAL file; 0 = unlimited (default).
//     When exceeded the WAL is force-rotated to prevent unbounded disk growth.
type Checkpoints struct {
	FileFormat string `json:"file-format"`
	RootDir    string `json:"directory"`
	MaxWALSize int64  `json:"max-wal-size,omitempty"`
}


The file /Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/config.go has been updated successfully.

// Checkpoints configures periodic persistence of in-memory metric data.
//
// Fields:
//   - FileFormat:  "json" (human-readable, periodic) or "wal" (binary snapshot + WAL, crash-safe); default is "wal"
//   - RootDir:     Filesystem path for checkpoint files (created if missing)
//   - MaxWALSize:  Maximum size in bytes for a single host's WAL file; 0 = unlimited (default).
//     When exceeded the WAL is force-rotated to prevent unbounded disk growth.
type Checkpoints struct {
	FileFormat string `json:"file-format"`
	RootDir    string `json:"directory"`
	MaxWALSize int64  `json:"max-wal-size,omitempty"`
}

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package metricstore provides config.go: Configuration structures and metric management.
//
// # Configuration Hierarchy
//
// The metricstore package uses nested configuration structures:
//
//	MetricStoreConfig (Keys)
//	├─ NumWorkers: Parallel checkpoint/archive workers
//	├─ RetentionInMemory: How long to keep data in RAM (also used as cleanup interval)
//	├─ MemoryCap: Memory limit in bytes (triggers forceFree)
//	├─ Checkpoints: Persistence configuration
//	│  ├─ FileFormat: "json" or "wal" (default: "wal")
//	│  └─ RootDir: Checkpoint storage path
//	├─ Cleanup: Long-term storage configuration (interval = RetentionInMemory)
//	│  ├─ RootDir: Archive storage path (archive mode only)
//	│  └─ Mode: "delete" or "archive"
//	├─ Debug: Development/debugging options
//	└─ Subscriptions: NATS topic subscriptions for metric ingestion
//
// # Metric Configuration
//
// Each metric (e.g., "cpu_load", "mem_used") has a MetricConfig entry in the global
// Metrics map, defining:
//
//   - Frequency: Measurement interval in seconds
//   - Aggregation: How to combine values (sum/avg/none) when transforming scopes
//   - offset: Internal index into Level.metrics slice (assigned during Init)
//
// # AggregationStrategy
//
// Determines how to combine metric values when aggregating from finer to coarser scopes:
//
//   - NoAggregation: Do not combine (incompatible scopes)
//   - SumAggregation: Add values (e.g., power consumption: core→socket)
//   - AvgAggregation: Average values (e.g., temperature: core→socket)
package metricstore

import (
	"fmt"
	"time"

	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

const (
	DefaultMaxWorkers                 = 10
	DefaultBufferCapacity             = 512
	DefaultGCTriggerInterval          = 100
	DefaultMemoryUsageTrackerInterval = 5 * time.Minute
)

// Checkpoints configures periodic persistence of in-memory metric data.
//
// Fields:
//   - FileFormat: "json" (human-readable, periodic) or "wal" (binary snapshot + WAL, crash-safe); default is "wal"
//   - RootDir:    Filesystem path for checkpoint files (created if missing)
type Checkpoints struct {
	FileFormat string `json:"file-format"`
	RootDir    string `json:"directory"`
}

// Debug provides development and profiling options.
//
// Fields:
//   - DumpToFile: Path to dump checkpoint data for inspection (empty = disabled)
//   - EnableGops: Enable gops agent for live runtime debugging (https://github.com/google/gops)
type Debug struct {
	DumpToFile string `json:"dump-to-file"`
	EnableGops bool   `json:"gops"`
}

// Cleanup configures long-term storage of old metric data.
//
// Data older than RetentionInMemory is archived to disk or deleted.
// The cleanup interval is always RetentionInMemory.
//
// Fields:
//   - RootDir: Filesystem path for archived data (used in "archive" mode)
//   - Mode:    "delete" (discard old data) or "archive" (write to RootDir)
type Cleanup struct {
	RootDir string `json:"directory"`
	Mode    string `json:"mode"`
}

// Subscriptions defines NATS topics to subscribe to for metric ingestion.
//
// Each subscription receives metrics via NATS messaging, enabling real-time
// data collection from compute nodes.
//
// Fields:
//   - SubscribeTo: NATS subject/channel name (e.g., "metrics.compute.*")
//   - ClusterTag:  Default cluster name for metrics without cluster tag (optional)
type Subscriptions []struct {
	// Channel name
	SubscribeTo string `json:"subscribe-to"`

	// Allow lines without a cluster tag, use this as default, optional
	ClusterTag string `json:"cluster-tag"`
}

// MetricStoreConfig defines the main configuration for the metricstore.
//
// Loaded from cc-backend's config.json "metricstore" section. Controls memory usage,
// persistence, archiving, and metric ingestion.
//
// Fields:
//   - NumWorkers:        Parallel workers for checkpoint/archive (0 = auto: min(NumCPU/2+1, 10))
//   - RetentionInMemory: Duration string (e.g., "48h") for in-memory data retention
//   - MemoryCap:         Max memory in GB for buffer data (0 = unlimited); triggers forceFree when exceeded
//   - Checkpoints:       Periodic persistence configuration
//   - Debug:             Development/profiling options (nil = disabled)
//   - Archive:           Long-term storage configuration (nil = disabled)
//   - Subscriptions:     NATS topics for metric ingestion (nil = polling only)
type MetricStoreConfig struct {
	// Number of concurrent workers for checkpoint and archive operations.
	// If not set or 0, defaults to min(runtime.NumCPU()/2+1, 10)
	NumWorkers         int            `json:"num-workers"`
	RetentionInMemory  string         `json:"retention-in-memory"`
	CheckpointInterval string         `json:"checkpoint-interval,omitempty"`
	MemoryCap          int            `json:"memory-cap"`
	Checkpoints        Checkpoints    `json:"checkpoints"`
	Debug              *Debug         `json:"debug"`
	Cleanup            *Cleanup       `json:"cleanup"`
	Subscriptions      *Subscriptions `json:"nats-subscriptions"`
}

// Keys is the global metricstore configuration instance.
//
// Initialized with defaults, then overwritten by cc-backend's config.json.
// Accessed by Init(), Checkpointing(), and other lifecycle functions.
var Keys MetricStoreConfig = MetricStoreConfig{
	Checkpoints: Checkpoints{
		FileFormat: "wal",
		RootDir:    "./var/checkpoints",
	},
	Cleanup: &Cleanup{
		Mode: "delete",
	},
}

// AggregationStrategy defines how to combine metric values across hierarchy levels.
//
// Used when transforming data from finer-grained scopes (e.g., core) to coarser scopes
// (e.g., socket). This is SPATIAL aggregation, not TEMPORAL (time-based) aggregation.
//
// Values:
//   - NoAggregation:  Do not aggregate (incompatible scopes or non-aggregatable metrics)
//   - SumAggregation: Add values (e.g., power: sum core power → socket power)
//   - AvgAggregation: Average values (e.g., temperature: average core temps → socket temp)
type AggregationStrategy int

const (
	NoAggregation  AggregationStrategy = iota // Do not aggregate
	SumAggregation                            // Sum values (e.g., power, energy)
	AvgAggregation                            // Average values (e.g., temperature, utilization)
)

// AssignAggregationStrategy parses a string into an AggregationStrategy value.
//
// Used when loading metric configurations from JSON/YAML files.
//
// Parameters:
//   - str: "sum", "avg", or "" (empty string for NoAggregation)
//
// Returns:
//   - AggregationStrategy: Parsed value
//   - error:               Non-nil if str is unrecognized
func AssignAggregationStrategy(str string) (AggregationStrategy, error) {
	switch str {
	case "":
		return NoAggregation, nil
	case "sum":
		return SumAggregation, nil
	case "avg":
		return AvgAggregation, nil
	default:
		return NoAggregation, fmt.Errorf("[METRICSTORE]> unknown aggregation strategy: %s", str)
	}
}

// MetricConfig defines configuration for a single metric type.
//
// Stored in the global Metrics map, keyed by metric name (e.g., "cpu_load").
//
// Fields:
//   - Frequency:   Measurement interval in seconds (e.g., 60 for 1-minute granularity)
//   - Aggregation: How to combine values across hierarchy levels (sum/avg/none)
//   - offset:      Internal index into Level.metrics slice (assigned during Init)
type MetricConfig struct {
	// Interval in seconds at which measurements are stored
	Frequency int64

	// Can be 'sum', 'avg' or null. Describes how to aggregate metrics from the same timestep over the hierarchy.
	Aggregation AggregationStrategy

	// Private, used internally...
	offset int
}

func BuildMetricList() map[string]MetricConfig {
	var metrics map[string]MetricConfig = make(map[string]MetricConfig)

	addMetric := func(name string, metric MetricConfig) error {
		if metrics == nil {
			metrics = make(map[string]MetricConfig, 0)
		}

		if existingMetric, ok := metrics[name]; ok {
			if existingMetric.Frequency != metric.Frequency {
				if existingMetric.Frequency < metric.Frequency {
					existingMetric.Frequency = metric.Frequency
					metrics[name] = existingMetric
				}
			}
		} else {
			metrics[name] = metric
		}

		return nil
	}

	// Helper function to add metric configuration
	addMetricConfig := func(mc *schema.MetricConfig) {
		agg, err := AssignAggregationStrategy(mc.Aggregation)
		if err != nil {
			cclog.Warnf("Could not find aggregation strategy for metric config '%s': %s", mc.Name, err.Error())
		}

		addMetric(mc.Name, MetricConfig{
			Frequency:   int64(mc.Timestep),
			Aggregation: agg,
		})
	}
	for _, c := range archive.Clusters {
		for _, mc := range c.MetricConfig {
			addMetricConfig(mc)
		}

		for _, sc := range c.SubClusters {
			for _, mc := range sc.MetricConfig {
				addMetricConfig(mc)
			}
		}
	}

	return metrics
}

-//   - FileFormat: "json" (human-readable, periodic) or "wal" (binary snapshot + WAL, crash-safe); default is "wal"

-//   - RootDir:    Filesystem path for checkpoint files (created if missing)

+//   - FileFormat:  "json" (human-readable, periodic) or "wal" (binary snapshot + WAL, crash-safe); default is "wal"

+//   - RootDir:     Filesystem path for checkpoint files (created if missing)

+//   - MaxWALSize:  Maximum size in bytes for a single host's WAL file; 0 = unlimited (default).

+//     When exceeded the WAL is force-rotated to prevent unbounded disk growth.

+  MaxWALSize int64  `json:"max-wal-size,omitempty"`


Updated task #3 status


Now add `size` tracking to `walFileState` and enforce the limit:


// walFileState holds an open WAL file handle and buffered writer for one host directory.
type walFileState struct {
	f     *os.File
	w     *bufio.Writer
	dirty bool
}

// walFileState holds an open WAL file handle and buffered writer for one host directory.
type walFileState struct {
	f     *os.File
	w     *bufio.Writer
	dirty bool
	size  int64 // approximate bytes written (tracked from open + writes)
}