Previously the throughput of the wal rotation was improved. But it is still not sufficient. It seems that on top of a continous metric instream there are also bursts of metric data. The problem occurs when a checkpoint is written: 3/29/2026, 8:06:02 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1440001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:01 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1430001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:01 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1420001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:01 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1410001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1400001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1390001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1380001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1370001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1360001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1350001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1340001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1330001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1320001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1310001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1300001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1290001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1280001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1270001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1260001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1250001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1240001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1230001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1220001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1210001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1200001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1190001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1180001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1170001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1160001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1150001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1140001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1130001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1120001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1110001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1100001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1090001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1080001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1070001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1060001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1050001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1040001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1030001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1020001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1010001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 1000001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 990001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 980001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 970001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 960001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 950001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 940001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 930001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 920001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 910001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 900001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 890001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 880001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 870001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 860001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 850001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 840001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 830001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 820001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 810001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 800001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 790001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 780001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 770001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 760001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 750001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 740001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 730001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 720001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 710001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 700001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 690001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:06:00 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 680001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 670001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 660001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 580001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 650001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 640001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 630001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 620001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 610001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 600001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 590001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 570001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 560001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 550001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 540001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 530001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 520001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 510001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:59 AMWARN[WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 500001 messages (data safe in memory, next snapshot will capture)
3/29/2026, 8:05:50 AMINFO[INFO]     walCheckpoint.go:690: [METRICSTORE]> Starting binary checkpoint for 1845 hosts with 10 workers . Analyze the problem, improve the throughput and fix the issue.


Let me explore the codebase to understand the current WAL pipeline and identify the bottleneck.


Agent

Explore WAL pipeline architecture

Prompt:
I need to understand the WAL (Write-Ahead Log) pipeline in the metricstore package to diagnose a throughput bottleneck. Specifically:

1. In `pkg/metricstore/lineprotocol.go`: Find the WAL message sending code around line 368, the WAL shard channel setup, the `WALMessage` struct, and how messages are dispatched to shards. Look for channel sizes, the non-blocking send pattern, and the drop/warning logic.

2. In `pkg/metricstore/walCheckpoint.go`: Find the `WALStaging` goroutine (or equivalent shard consumer), the binary checkpoint writing code (around line 690), and the `RotateWALFiles` function. I need to understand what happens during checkpointing — does writing a checkpoint block the WAL consumer?

3. In `pkg/metricstore/metricstore.go`: Find where WAL channels are initialized, their buffer sizes, how many shards/workers are created, and the `Init()` / `Shutdown()` flow.

Read these files thoroughly and report back the complete architecture: channel sizes, number of shards, what blocks what during checkpointing, and where the bottleneck likely is.


Perfect! Now I have all the information I need. Let me create a comprehensive analysis document:

## Complete WAL Pipeline Architecture Analysis

Based on my thorough examination of the three key files, here is the complete architecture of the WAL pipeline and where the throughput bottleneck likely exists:

---

### **1. CHANNEL CONFIGURATION & SHARD DISPATCH**

**File:** `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/lineprotocol.go` (lines 355-371)

**WAL Message Structure** (`lineprotocol.go` line 356-363):
```go
type WALMessage struct {
    MetricName string
    Cluster    string
    Node       string
    Selector   []string
    Value      schema.Float
    Timestamp  int64
}
```

**Sending Pattern** (line 364):
- Non-blocking send using `select` with `default` case
- Returns `false` if shard channel is full
- Dropped messages tracked via `walDropped` atomic counter
- Warning logged every 10,000 drops (line 368)

---

### **2. SHARD CHANNEL SETUP & SIZING**

**File:** `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/walCheckpoint.go` (lines 162-176)

**Initialization** (lines 167-176):
```go
walNumShards = max(Keys.NumWorkers, 1)
chBufSize := max(65536/walNumShards, 1024)

walShardChs = make([]chan *WALMessage, walNumShards)
walShardRotateChs = make([]chan walRotateReq, walNumShards)

for i := range walNumShards {
    walShardChs[i] = make(chan *WALMessage, chBufSize)
    walShardRotateChs[i] = make(chan walRotateReq, 64)
}
```

**Key Parameters:**
- **Number of Shards:** Equal to `Keys.NumWorkers` (configurable, defaults to `min(runtime.NumCPU()/2+1, 10)`)
- **Channel Buffer Size per Shard:** `max(65536/walNumShards, 1024)`
  - Default assumption: if `NumWorkers=4`, then `chBufSize = max(16384, 1024) = 16,384` messages per shard
  - If `NumWorkers=8`, then `chBufSize = max(8192, 1024) = 8,192` messages per shard
  - **Total system capacity:** ~65,536 buffered WAL messages across all shards
- **Rotation Channel Buffer:** 64 (fixed, small - for rotation signaling only)

**Message Routing** (lines 135-141, 145-156):
```go
func SendWALMessage(msg *WALMessage) bool {
    if walShardChs == nil || walShuttingDown.Load() {
        return false
    }
    shard := walShardIndex(msg.Cluster, msg.Node)
    select {
    case walShardChs[shard] <- msg:
        return true
    default:
        return false
    }
}
```
- Uses FNV-1a hash of `cluster+node` to determine shard
- Each host affinity maintained within a shard (good for locality)

---

### **3. WAL STAGING GOROUTINE (Consumer Side)**

**File:** `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/walCheckpoint.go` (lines 162-342)

**Architecture (lines 178-341):**
- Creates `walNumShards` independent goroutines (one per shard)
- Each goroutine:
  - Maintains a `map[string]*walFileState` for open host WAL files
  - Receives messages from its dedicated `msgCh` (non-blocking)
  - Receives rotation requests from `rotateCh`
  - Drains up to 4,096 messages per batch (line 300)
  - Flushes dirty files every `walFlushInterval = 1 second` (line 131)

**Critical Blocking Point - The Checkpoint Write Process:**

```go
cf, err := l.toCheckpointFile(from, to, m)  // Locks Level tree, reads buffers
```

This happens inside:
- `ToCheckpointWAL()` (walCheckpoint.go lines 658-760)
- `toCheckpointBinary()` (walCheckpoint.go lines 764-800)

**Blocking Sequence During Checkpointing:**

1. **toCheckpointFile** (checkpoint.go lines 264-327):
   - Acquires **RLock** on the Level tree for **each host** (line 265)
   - Holds lock while reading buffer data (lines 276-309)
   - For each metric buffer: calls `b.read(from, to, data)` while holding the lock
   - Recursively processes child levels (line 312) - **tree traversal under lock**
   - Lock remains held until entire subtree is serialized to memory

2. **Write to Disk:**
   - No locks held during file I/O (disk writes are async)
   - Uses atomic write pattern: write to `.tmp` then rename

**Worker Pool During Checkpointing:**
- `ToCheckpointWAL()` uses `Keys.NumWorkers` parallel workers (line 697)
- Creates work queue with `Keys.NumWorkers*2` buffer (line 698)
- Each worker calls `level.toCheckpointBinary()` → `level.toCheckpointFile()`

---

### **4. MEMORY STORE & WRITE SIDE LOCKING**

**File:** `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/metricstore.go` (lines 658-686)

**WriteToLevel Function** (lines 658-686):
```go
func (m *MemoryStore) WriteToLevel(l *Level, selector []string, ts int64, metrics []Metric) error {
    l = l.findLevelOrCreate(selector, len(m.Metrics))
    l.lock.Lock()  // ← WRITE LOCK acquired here
    defer l.lock.Unlock()

    for _, metric := range metrics {
        // ... write to buffers ...
    }
    return nil
}
```

**Lock Hierarchy:**
- **Write path:** `WriteToLevel()` takes **Lock()** (exclusive) on target Level
- **Checkpoint read path:** `toCheckpointFile()` takes **RLock()** (shared) on Level
- **Contention point:** When checkpoint is in progress:
  - All writes to that host's Level are **blocked** (waiting for RLock release)
  - RLock holds entire host's Level while serializing all metrics + children

---

### **5. INITIALIZATION & WORKER CONFIGURATION**

**File:** `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/metricstore.go` (lines 124-193)

**Default Worker Count** (config.go lines 52-57, metricstore.go lines 136-140):
```go
const DefaultMaxWorkers = 10

if Keys.NumWorkers <= 0 {
    Keys.NumWorkers = min(runtime.NumCPU()/2+1, DefaultMaxWorkers)
}
```

**Initialization Order** (lines 172-176):
1. `Retention()` - garbage collection goroutine
2. `Checkpointing()` - periodic snapshot creator
3. `CleanUp()` - archive worker
4. `WALStaging()` - **WAL consumer goroutines** ← Started here
5. `MemoryUsageTracker()` - emergency free handler

**Shutdown Flow** (metricstore.go lines 273-320):
- Sets `walShuttingDown.Store(true)` (line 286)
- Sleeps 100ms to allow in-flight DecodeLine calls to complete (line 288)
- Closes all `walShardChs` (line 290-292)
- Waits for `WaitForWALStagingDrain()` (line 294)
- Writes final checkpoint (line 311)

---

### **6. THE BOTTLENECK - DETAILED ANALYSIS**

**Root Cause: Double Locking During Checkpointing**

The throughput bottleneck occurs at the intersection of three factors:

1. **WAL Message Production Rate**
   - Decoder threads send messages to shard channels (non-blocking)
   - Buffer capacity: ~65,536 messages total across all shards
   - When full → messages dropped (logged as warnings)

2. **WAL Staging Consumption Rate**
   - Each shard goroutine processes up to 4,096 messages per batch
   - Flushes every 1 second to disk
   - Normal case: no congestion, messages flow through

3. **Checkpoint Write Contention** ← **THE BOTTLENECK**

   During periodic checkpoint (every 12 hours by default):

   ```
   Checkpointing goroutine:
      ├─ Call ToCheckpointWAL() for all N hosts
      │  ├─ Start Keys.NumWorkers checkpoint workers
      │  └─ Each worker processes host in parallel:
      │     └─ toCheckpointBinary(hostDir)
      │        └─ toCheckpointFile()
      │           └─ l.lock.RLock()           ← BLOCKS new writes to Level
      │              └─ Serialize all metrics   ← Holds lock for duration
      │              └─ Recursive child traversal ← Lock held for entire subtree
      │           └─ l.lock.RUnlock()
      │        └─ File I/O (disk write)
      │
      └─ After completion: RotateWALFiles(hostDirs)
         └─ Sends rotation requests via walShardRotateChs
         └─ WAL staging goroutines process rotation
            └─ Delete current.wal for each host

   DecodeLine threads (in parallel):
      └─ WriteToLevel(cluster, host, ...)
         └─ l.lock.Lock()                      ← BLOCKS waiting for RLock
            (DEADLOCK if checkpoint reader doesn't release)
   ```

   **Lock contention window:** The time `toCheckpointFile()` holds `RLock` on a Level
   - Depends on: number of metrics, size of metric buffers, tree depth
   - Multiply by: number of hosts being checkpointed simultaneously
   - With `Keys.NumWorkers=4`: up to 4 hosts locked simultaneously
   - With `Keys.NumWorkers=8`: up to 8 hosts locked simultaneously

4. **WAL Channel Backpressure**
   - While writes are blocked waiting for checkpoint to complete:
     - `DecodeLine()` threads accumulate in `WriteToLevel()` lock wait queues
     - New WAL messages from NATS are still produced
     - Shard channels fill up (65,536 buffer max)
     - Once full → messages dropped silently
     - Dropped messages logged every 10,000 (walDropped counter)

---

### **7. SPECIFIC BLOCKING SCENARIOS**

**Scenario A: Checkpoint Duration > Message Batch Time**
```
t=0ms:   Checkpoint starts, locks host "node001"
t=50ms:  NATS delivers batch of 10,000 metrics for "node001"
t=50ms:  DecodeLine calls WriteToLevel() → blocks on l.lock.Lock()
t=100ms: Checkpoint finishes, releases RLock
t=100ms: WriteToLevel() acquires lock, writes buffered data
t=100ms: WAL messages finally enqueued to shard channel
Result: 50ms latency added per host per checkpoint
```

**Scenario B: WAL Channel Saturation**
```
t=0ms:   Checkpoint running on hosts 1-4 (4 workers)
t=0ms:   NATS rate spike: 50,000 messages/sec for hosts not being checkpointed
t=10ms:  65,536 shard channel buffers fill
t=10ms:  SendWALMessage() returns false → messages dropped
t=20ms:  Warning logged: "dropped 10,001 messages"
t=120ms: Checkpoint completes
t=120ms: Writer threads resume → WAL messages enqueue normally
Result: Bursty throughput, packet loss during checkpoint windows
```

**Scenario C: Read-Write Priority Inversion**
```
Line 624-636 in walCheckpoint.go:
   lvl.lock.Lock()          ← WAL replay during startup
   b = lvl.metrics[...]
   b.write(timestamp, value) ← Writes to buffer while checkpoint reader holds RLock
   lvl.lock.Unlock()

During startup FromCheckpoint():
   → Replays WAL into buffers (holding exclusive locks per level)
   → Simultaneously: DecodeLine threads try to write new metrics
   → Contention: NewMetrics vs replay writes
```

---

### **8. SUMMARY: WHERE THE BOTTLENECK IS**

| Component | Location | Capacity | Bottleneck Severity |
|-----------|----------|----------|-------------------|
| **WAL Shard Channels** | walCheckpoint.go:174 | 65,536 total msgs | MODERATE - fills during checkpoint |
| **toCheckpointFile RLock** | checkpoint.go:265 | Duration-based | **SEVERE - blocks writes for N*100ms** |
| **Checkpoint Workers** | walCheckpoint.go:697 | Keys.NumWorkers | MODERATE - locks up to N hosts |
| **WAL Flush Interval** | walCheckpoint.go:131 | 1 second | LOW - async to message flow |
| **Rotation Blocking** | walCheckpoint.go:357-384 | 2 min timeout | MODERATE - can deadlock on error |

**PRIMARY BOTTLENECK:**
The `toCheckpointFile()` function's `RLock` on Level nodes during the periodic checkpoint (every 12 hours) blocks **all writes to those hosts**, causing:
1. NATS message ingestion to stall
2. WAL shard channels to fill and drop messages
3. Throughput cliff at checkpoint time
4. Data loss (dropped messages safe only if next binary snapshot captures them)

**RECOMMENDATION:**
To diagnose the exact impact, search logs for:
- `"WAL shard channel full, dropped %d messages"` patterns at checkpoint times
- Gaps in checkpoint intervals (should be 12h, check if extended)
- Correlation between checkpoint windows and metric ingestion drops

Prompt:
I need to understand the WAL (Write-Ahead Log) pipeline in the metricstore package to diagnose a throughput bottleneck. Specifically:

1. In `pkg/metricstore/lineprotocol.go`: Find the WAL message sending code around line 368, the WAL shard channel setup, the `WALMessage` struct, and how messages are dispatched to shards. Look for channel sizes, the non-blocking send pattern, and the drop/warning logic.

2. In `pkg/metricstore/walCheckpoint.go`: Find the `WALStaging` goroutine (or equivalent shard consumer), the binary checkpoint writing code (around line 690), and the `RotateWALFiles` function. I need to understand what happens during checkpointing — does writing a checkpoint block the WAL consumer?

3. In `pkg/metricstore/metricstore.go`: Find where WAL channels are initialized, their buffer sizes, how many shards/workers are created, and the `Init()` / `Shutdown()` flow.

Read these files thoroughly and report back the complete architecture: channel sizes, number of shards, what blocks what during checkpointing, and where the bottleneck likely is.


Let me read the critical code sections directly to verify the analysis and understand the exact implementation.


125		size  int64 // approximate bytes written (tracked from open + writes)
126	}
127
128	// walFlushInterval controls how often dirty WAL files are flushed to disk.
129	// Decoupling flushes from message processing lets the consumer run at memory
130	// speed, amortizing syscall overhead across many writes.
131	const walFlushInterval = 1 * time.Second
132
133	// walShardIndex computes which shard a message belongs to based on cluster+node.
134	// Uses FNV-1a hash for fast, well-distributed mapping.
135	func walShardIndex(cluster, node string) int {
136		h := fnv.New32a()
137		h.Write([]byte(cluster))
138		h.Write([]byte{0})
139		h.Write([]byte(node))
140		return int(h.Sum32()) % walNumShards
141	}
142
143	// SendWALMessage routes a WAL message to the appropriate shard channel.
144	// Returns false if the channel is full or shutdown is in progress.
145	func SendWALMessage(msg *WALMessage) bool {
146		if walShardChs == nil || walShuttingDown.Load() {
147			return false
148		}
149		shard := walShardIndex(msg.Cluster, msg.Node)
150		select {
151		case walShardChs[shard] <- msg:
152			return true
153		default:
154			return false
155		}
156	}
157
158	// WALStaging starts N sharded background goroutines that receive WALMessage items
159	// and append binary WAL records to per-host current.wal files.
160	// Each shard owns a subset of hosts (determined by hash of cluster+node),
161	// parallelizing serialization and I/O while keeping per-host writes serial.
162	func WALStaging(wg *sync.WaitGroup, ctx context.Context) {
163		if Keys.Checkpoints.FileFormat == "json" {
164			return
165		}
166
167		walNumShards = max(Keys.NumWorkers, 1)
168		chBufSize := max(65536/walNumShards, 1024)
169
170		walShardChs = make([]chan *WALMessage, walNumShards)
171		walShardRotateChs = make([]chan walRotateReq, walNumShards)
172
173		for i := range walNumShards {
174			walShardChs[i] = make(chan *WALMessage, chBufSize)
175			walShardRotateChs[i] = make(chan walRotateReq, 64)
176		}
177
178		for i := range walNumShards {
179			msgCh := walShardChs[i]
180			rotateCh := walShardRotateChs[i]
181
182			walStagingWg.Add(1)
183			wg.Go(func() {
184				defer walStagingWg.Done()
185				hostFiles := make(map[string]*walFileState)
186
187				defer func() {
188					for _, ws := range hostFiles {
189						if ws.f != nil {
190							ws.w.Flush()
191							ws.f.Close()
192						}
193					}
194				}()
195
196				getOrOpenWAL := func(hostDir string) *walFileState {
197					ws, ok := hostFiles[hostDir]
198					if ok {
199						return ws
200					}
201
202					if err := os.MkdirAll(hostDir, CheckpointDirPerms); err != nil {
203						cclog.Errorf("[METRICSTORE]> WAL: mkdir %s: %v", hostDir, err)
204						return nil
205					}
206
207					walPath := path.Join(hostDir, "current.wal")
208					f, err := os.OpenFile(walPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, CheckpointFilePerms)
209					if err != nil {
210						cclog.Errorf("[METRICSTORE]> WAL: open %s: %v", walPath, err)
211						return nil
212					}
213
214					w := bufio.NewWriter(f)
215
216					// Write file header magic if file is new (empty).
217					info, err := f.Stat()
218					var fileSize int64
219					if err == nil {
220						fileSize = info.Size()
221					}
222					if err == nil && fileSize == 0 {
223						var hdr [4]byte
224						binary.LittleEndian.PutUint32(hdr[:], walFileMagic)

	size  int64 // approximate bytes written (tracked from open + writes)
}

// walFlushInterval controls how often dirty WAL files are flushed to disk.
// Decoupling flushes from message processing lets the consumer run at memory
// speed, amortizing syscall overhead across many writes.
const walFlushInterval = 1 * time.Second

// walShardIndex computes which shard a message belongs to based on cluster+node.
// Uses FNV-1a hash for fast, well-distributed mapping.
func walShardIndex(cluster, node string) int {
	h := fnv.New32a()
	h.Write([]byte(cluster))
	h.Write([]byte{0})
	h.Write([]byte(node))
	return int(h.Sum32()) % walNumShards
}

// SendWALMessage routes a WAL message to the appropriate shard channel.
// Returns false if the channel is full or shutdown is in progress.
func SendWALMessage(msg *WALMessage) bool {
	if walShardChs == nil || walShuttingDown.Load() {
		return false
	}
	shard := walShardIndex(msg.Cluster, msg.Node)
	select {
	case walShardChs[shard] <- msg:
		return true
	default:
		return false
	}
}

// WALStaging starts N sharded background goroutines that receive WALMessage items
// and append binary WAL records to per-host current.wal files.
// Each shard owns a subset of hosts (determined by hash of cluster+node),
// parallelizing serialization and I/O while keeping per-host writes serial.
func WALStaging(wg *sync.WaitGroup, ctx context.Context) {
	if Keys.Checkpoints.FileFormat == "json" {
		return
	}

	walNumShards = max(Keys.NumWorkers, 1)
	chBufSize := max(65536/walNumShards, 1024)

	walShardChs = make([]chan *WALMessage, walNumShards)
	walShardRotateChs = make([]chan walRotateReq, walNumShards)

	for i := range walNumShards {
		walShardChs[i] = make(chan *WALMessage, chBufSize)
		walShardRotateChs[i] = make(chan walRotateReq, 64)
	}

	for i := range walNumShards {
		msgCh := walShardChs[i]
		rotateCh := walShardRotateChs[i]

		walStagingWg.Add(1)
		wg.Go(func() {
			defer walStagingWg.Done()
			hostFiles := make(map[string]*walFileState)

			defer func() {
				for _, ws := range hostFiles {
					if ws.f != nil {
						ws.w.Flush()
						ws.f.Close()
					}
				}
			}()

			getOrOpenWAL := func(hostDir string) *walFileState {
				ws, ok := hostFiles[hostDir]
				if ok {
					return ws
				}

				if err := os.MkdirAll(hostDir, CheckpointDirPerms); err != nil {
					cclog.Errorf("[METRICSTORE]> WAL: mkdir %s: %v", hostDir, err)
					return nil
				}

				walPath := path.Join(hostDir, "current.wal")
				f, err := os.OpenFile(walPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, CheckpointFilePerms)
				if err != nil {
					cclog.Errorf("[METRICSTORE]> WAL: open %s: %v", walPath, err)
					return nil
				}

				w := bufio.NewWriter(f)

				// Write file header magic if file is new (empty).
				info, err := f.Stat()
				var fileSize int64
				if err == nil {
					fileSize = info.Size()
				}
				if err == nil && fileSize == 0 {
					var hdr [4]byte
					binary.LittleEndian.PutUint32(hdr[:], walFileMagic)


640	}
641
642	// joinSelector builds a cache key from a selector slice using null bytes as separators.
643	func joinSelector(sel []string) string {
644		if len(sel) == 0 {
645			return ""
646		}
647		var result strings.Builder
648		result.WriteString(sel[0])
649		for i := 1; i < len(sel); i++ {
650			result.WriteString("\x00" + sel[i])
651		}
652		return result.String()
653	}
654
655	// ToCheckpointWAL writes binary snapshot files for all hosts in parallel.
656	// Returns the number of files written, the list of host directories that were
657	// successfully checkpointed (for WAL rotation), and any errors.
658	func (m *MemoryStore) ToCheckpointWAL(dir string, from, to int64) (int, []string, error) {
659		// Collect all cluster/host pairs.
660		m.root.lock.RLock()
661		totalHosts := 0
662		for _, l1 := range m.root.children {
663			l1.lock.RLock()
664			totalHosts += len(l1.children)
665			l1.lock.RUnlock()
666		}
667		m.root.lock.RUnlock()
668
669		levels := make([]*Level, 0, totalHosts)
670		selectors := make([][]string, 0, totalHosts)
671
672		m.root.lock.RLock()
673		for sel1, l1 := range m.root.children {
674			l1.lock.RLock()
675			for sel2, l2 := range l1.children {
676				levels = append(levels, l2)
677				selectors = append(selectors, []string{sel1, sel2})
678			}
679			l1.lock.RUnlock()
680		}
681		m.root.lock.RUnlock()
682
683		type workItem struct {
684			level    *Level
685			hostDir  string
686			selector []string
687		}
688
689		totalWork := len(levels)
690		cclog.Infof("[METRICSTORE]> Starting binary checkpoint for %d hosts with %d workers", totalWork, Keys.NumWorkers)
691
692		n, errs, completed := int32(0), int32(0), int32(0)
693		var successDirs []string
694		var successMu sync.Mutex
695
696		var wg sync.WaitGroup
697		wg.Add(Keys.NumWorkers)
698		work := make(chan workItem, Keys.NumWorkers*2)
699
700		// Progress logging goroutine.
701		stopProgress := make(chan struct{})
702		go func() {
703			ticker := time.NewTicker(10 * time.Second)
704			defer ticker.Stop()
705			for {
706				select {
707				case <-ticker.C:
708					cclog.Infof("[METRICSTORE]> Checkpoint progress: %d/%d hosts (%d written, %d errors)",
709						atomic.LoadInt32(&completed), totalWork, atomic.LoadInt32(&n), atomic.LoadInt32(&errs))
710				case <-stopProgress:
711					return
712				}
713			}
714		}()
715
716		for range Keys.NumWorkers {
717			go func() {
718				defer wg.Done()
719				for wi := range work {
720					err := wi.level.toCheckpointBinary(wi.hostDir, from, to, m)
721					if err != nil {
722						if err == ErrNoNewArchiveData {
723							atomic.AddInt32(&completed, 1)
724							continue
725						}
726						cclog.Errorf("[METRICSTORE]> binary checkpoint error for %s: %v", wi.hostDir, err)
727						atomic.AddInt32(&errs, 1)
728					} else {
729						atomic.AddInt32(&n, 1)
730						// Delete WAL immediately after successful snapshot.
731						walPath := path.Join(wi.hostDir, "current.wal")
732						if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
733							cclog.Errorf("[METRICSTORE]> WAL remove %s: %v", walPath, err)
734						}
735						successMu.Lock()
736						successDirs = append(successDirs, wi.hostDir)
737						successMu.Unlock()
738					}
739					atomic.AddInt32(&completed, 1)
740				}
741			}()
742		}
743
744		for i := range levels {
745			hostDir := path.Join(dir, path.Join(selectors[i]...))
746			work <- workItem{
747				level:    levels[i],
748				hostDir:  hostDir,
749				selector: selectors[i],
750			}
751		}
752		close(work)
753		wg.Wait()
754		close(stopProgress)
755
756		if errs > 0 {
757			return int(n), successDirs, fmt.Errorf("[METRICSTORE]> %d errors during binary checkpoint (%d successes)", errs, n)
758		}
759		return int(n), successDirs, nil

}

// joinSelector builds a cache key from a selector slice using null bytes as separators.
func joinSelector(sel []string) string {
	if len(sel) == 0 {
		return ""
	}
	var result strings.Builder
	result.WriteString(sel[0])
	for i := 1; i < len(sel); i++ {
		result.WriteString("\x00" + sel[i])
	}
	return result.String()
}

// ToCheckpointWAL writes binary snapshot files for all hosts in parallel.
// Returns the number of files written, the list of host directories that were
// successfully checkpointed (for WAL rotation), and any errors.
func (m *MemoryStore) ToCheckpointWAL(dir string, from, to int64) (int, []string, error) {
	// Collect all cluster/host pairs.
	m.root.lock.RLock()
	totalHosts := 0
	for _, l1 := range m.root.children {
		l1.lock.RLock()
		totalHosts += len(l1.children)
		l1.lock.RUnlock()
	}
	m.root.lock.RUnlock()

	levels := make([]*Level, 0, totalHosts)
	selectors := make([][]string, 0, totalHosts)

	m.root.lock.RLock()
	for sel1, l1 := range m.root.children {
		l1.lock.RLock()
		for sel2, l2 := range l1.children {
			levels = append(levels, l2)
			selectors = append(selectors, []string{sel1, sel2})
		}
		l1.lock.RUnlock()
	}
	m.root.lock.RUnlock()

	type workItem struct {
		level    *Level
		hostDir  string
		selector []string
	}

	totalWork := len(levels)
	cclog.Infof("[METRICSTORE]> Starting binary checkpoint for %d hosts with %d workers", totalWork, Keys.NumWorkers)

	n, errs, completed := int32(0), int32(0), int32(0)
	var successDirs []string
	var successMu sync.Mutex

	var wg sync.WaitGroup
	wg.Add(Keys.NumWorkers)
	work := make(chan workItem, Keys.NumWorkers*2)

	// Progress logging goroutine.
	stopProgress := make(chan struct{})
	go func() {
		ticker := time.NewTicker(10 * time.Second)
		defer ticker.Stop()
		for {
			select {
			case <-ticker.C:
				cclog.Infof("[METRICSTORE]> Checkpoint progress: %d/%d hosts (%d written, %d errors)",
					atomic.LoadInt32(&completed), totalWork, atomic.LoadInt32(&n), atomic.LoadInt32(&errs))
			case <-stopProgress:
				return
			}
		}
	}()

	for range Keys.NumWorkers {
		go func() {
			defer wg.Done()
			for wi := range work {
				err := wi.level.toCheckpointBinary(wi.hostDir, from, to, m)
				if err != nil {
					if err == ErrNoNewArchiveData {
						atomic.AddInt32(&completed, 1)
						continue
					}
					cclog.Errorf("[METRICSTORE]> binary checkpoint error for %s: %v", wi.hostDir, err)
					atomic.AddInt32(&errs, 1)
				} else {
					atomic.AddInt32(&n, 1)
					// Delete WAL immediately after successful snapshot.
					walPath := path.Join(wi.hostDir, "current.wal")
					if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
						cclog.Errorf("[METRICSTORE]> WAL remove %s: %v", walPath, err)
					}
					successMu.Lock()
					successDirs = append(successDirs, wi.hostDir)
					successMu.Unlock()
				}
				atomic.AddInt32(&completed, 1)
			}
		}()
	}

	for i := range levels {
		hostDir := path.Join(dir, path.Join(selectors[i]...))
		work <- workItem{
			level:    levels[i],
			hostDir:  hostDir,
			selector: selectors[i],
		}
	}
	close(work)
	wg.Wait()
	close(stopProgress)

	if errs > 0 {
		return int(n), successDirs, fmt.Errorf("[METRICSTORE]> %d errors during binary checkpoint (%d successes)", errs, n)
	}
	return int(n), successDirs, nil


330					metric.Value = schema.Float(val.UintV())
331				} else {
332					return fmt.Errorf("host %s: unsupported value type in message: %s", host, val.Kind().String())
333				}
334			}
335
336			if t, err = dec.Time(lineprotocol.Second, t); err != nil {
337				t = time.Now()
338				if t, err = dec.Time(lineprotocol.Millisecond, t); err != nil {
339					t = time.Now()
340					if t, err = dec.Time(lineprotocol.Microsecond, t); err != nil {
341						t = time.Now()
342						if t, err = dec.Time(lineprotocol.Nanosecond, t); err != nil {
343							return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
344						}
345					}
346				}
347			}
348
349			if err != nil {
350				return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
351			}
352
353			time := t.Unix()
354
355			if Keys.Checkpoints.FileFormat == "wal" {
356				msg := &WALMessage{
357					MetricName: string(st.metricBuf),
358					Cluster:    cluster,
359					Node:       host,
360					Selector:   append([]string{}, st.selector...),
361					Value:      metric.Value,
362					Timestamp:  time,
363				}
364				if !SendWALMessage(msg) {
365					// WAL shard channel full — metric is written to memory store but not WAL.
366					// Next binary snapshot will capture it.
367					if dropped := walDropped.Add(1); dropped%10000 == 1 {
368						cclog.Warnf("[METRICSTORE]> WAL shard channel full, dropped %d messages (data safe in memory, next snapshot will capture)", dropped)
369					}
370				}
371			}
372
373			if err := ms.WriteToLevel(lvl, st.selector, time, []Metric{metric}); err != nil {
374				cclog.Warnf("write error for host %s metric %s at ts %d: %s", host, string(st.metricBuf), time, err.Error())
375				continue
376			}
377		}
378		return nil
379	}
380

				metric.Value = schema.Float(val.UintV())
			} else {
				return fmt.Errorf("host %s: unsupported value type in message: %s", host, val.Kind().String())
			}
		}

		if t, err = dec.Time(lineprotocol.Second, t); err != nil {
			t = time.Now()
			if t, err = dec.Time(lineprotocol.Millisecond, t); err != nil {
				t = time.Now()
				if t, err = dec.Time(lineprotocol.Microsecond, t); err != nil {
					t = time.Now()
					if t, err = dec.Time(lineprotocol.Nanosecond, t); err != nil {
						return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
					}
				}
			}
		}

		if err != nil {
			return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
		}

		time := t.Unix()

		if Keys.Checkpoints.FileFormat == "wal" {
			msg := &WALMessage{
				MetricName: string(st.metricBuf),
				Cluster:    cluster,
				Node:       host,
				Selector:   append([]string{}, st.selector...),
				Value:      metric.Value,
				Timestamp:  time,
			}
			if !SendWALMessage(msg) {
				// WAL shard channel full — metric is written to memory store but not WAL.
				// Next binary snapshot will capture it.
				if dropped := walDropped.Add(1); dropped%10000 == 1 {
					cclog.Warnf("[METRICSTORE]> WAL shard channel full, dropped %d messages (data safe in memory, next snapshot will capture)", dropped)
				}
			}
		}

		if err := ms.WriteToLevel(lvl, st.selector, time, []Metric{metric}); err != nil {
			cclog.Warnf("write error for host %s metric %s at ts %d: %s", host, string(st.metricBuf), time, err.Error())
			continue
		}
	}
	return nil
}


760	}
761
762	// toCheckpointBinary writes a binary snapshot file for a single host-level node.
763	// Uses atomic rename (write to .tmp then rename) to avoid partial reads on crash.
764	func (l *Level) toCheckpointBinary(dir string, from, to int64, m *MemoryStore) error {
765		cf, err := l.toCheckpointFile(from, to, m)
766		if err != nil {
767			return err
768		}
769		if cf == nil {
770			return ErrNoNewArchiveData
771		}
772
773		if err := os.MkdirAll(dir, CheckpointDirPerms); err != nil {
774			return fmt.Errorf("mkdir %s: %w", dir, err)
775		}
776
777		// Write to a temp file first, then rename (atomic on POSIX).
778		tmpPath := path.Join(dir, fmt.Sprintf("%d.bin.tmp", from))
779		finalPath := path.Join(dir, fmt.Sprintf("%d.bin", from))
780
781		f, err := os.OpenFile(tmpPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
782		if err != nil {
783			return fmt.Errorf("open binary snapshot %s: %w", tmpPath, err)
784		}
785
786		bw := bufio.NewWriter(f)
787		if err := writeBinarySnapshotFile(bw, cf); err != nil {
788			f.Close()
789			os.Remove(tmpPath)
790			return fmt.Errorf("write binary snapshot: %w", err)
791		}
792		if err := bw.Flush(); err != nil {
793			f.Close()
794			os.Remove(tmpPath)
795			return err
796		}
797		f.Close()
798
799		return os.Rename(tmpPath, finalPath)
800	}
801
802	// writeBinarySnapshotFile writes the binary snapshot file header and level tree.
803	func writeBinarySnapshotFile(w io.Writer, cf *CheckpointFile) error {
804		if err := binary.Write(w, binary.LittleEndian, snapFileMagic); err != nil {
805			return err
806		}
807		if err := binary.Write(w, binary.LittleEndian, cf.From); err != nil {
808			return err
809		}
810		if err := binary.Write(w, binary.LittleEndian, cf.To); err != nil {
811			return err
812		}
813		return writeBinaryLevel(w, cf)
814	}
815
816	// writeBinaryLevel recursively writes a CheckpointFile level in binary format.
817	func writeBinaryLevel(w io.Writer, cf *CheckpointFile) error {
818		if err := binary.Write(w, binary.LittleEndian, uint32(len(cf.Metrics))); err != nil {
819			return err
820		}
821
822		for name, metric := range cf.Metrics {
823			if err := writeString16(w, name); err != nil {
824				return err
825			}
826			if err := binary.Write(w, binary.LittleEndian, metric.Frequency); err != nil {
827				return err
828			}
829			if err := binary.Write(w, binary.LittleEndian, metric.Start); err != nil {
830				return err
831			}
832			if err := binary.Write(w, binary.LittleEndian, uint32(len(metric.Data))); err != nil {
833				return err
834			}
835			for _, v := range metric.Data {
836				if err := binary.Write(w, binary.LittleEndian, math.Float32bits(float32(v))); err != nil {
837					return err
838				}
839			}

}

// toCheckpointBinary writes a binary snapshot file for a single host-level node.
// Uses atomic rename (write to .tmp then rename) to avoid partial reads on crash.
func (l *Level) toCheckpointBinary(dir string, from, to int64, m *MemoryStore) error {
	cf, err := l.toCheckpointFile(from, to, m)
	if err != nil {
		return err
	}
	if cf == nil {
		return ErrNoNewArchiveData
	}

	if err := os.MkdirAll(dir, CheckpointDirPerms); err != nil {
		return fmt.Errorf("mkdir %s: %w", dir, err)
	}

	// Write to a temp file first, then rename (atomic on POSIX).
	tmpPath := path.Join(dir, fmt.Sprintf("%d.bin.tmp", from))
	finalPath := path.Join(dir, fmt.Sprintf("%d.bin", from))

	f, err := os.OpenFile(tmpPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil {
		return fmt.Errorf("open binary snapshot %s: %w", tmpPath, err)
	}

	bw := bufio.NewWriter(f)
	if err := writeBinarySnapshotFile(bw, cf); err != nil {
		f.Close()
		os.Remove(tmpPath)
		return fmt.Errorf("write binary snapshot: %w", err)
	}
	if err := bw.Flush(); err != nil {
		f.Close()
		os.Remove(tmpPath)
		return err
	}
	f.Close()

	return os.Rename(tmpPath, finalPath)
}

// writeBinarySnapshotFile writes the binary snapshot file header and level tree.
func writeBinarySnapshotFile(w io.Writer, cf *CheckpointFile) error {
	if err := binary.Write(w, binary.LittleEndian, snapFileMagic); err != nil {
		return err
	}
	if err := binary.Write(w, binary.LittleEndian, cf.From); err != nil {
		return err
	}
	if err := binary.Write(w, binary.LittleEndian, cf.To); err != nil {
		return err
	}
	return writeBinaryLevel(w, cf)
}

// writeBinaryLevel recursively writes a CheckpointFile level in binary format.
func writeBinaryLevel(w io.Writer, cf *CheckpointFile) error {
	if err := binary.Write(w, binary.LittleEndian, uint32(len(cf.Metrics))); err != nil {
		return err
	}

	for name, metric := range cf.Metrics {
		if err := writeString16(w, name); err != nil {
			return err
		}
		if err := binary.Write(w, binary.LittleEndian, metric.Frequency); err != nil {
			return err
		}
		if err := binary.Write(w, binary.LittleEndian, metric.Start); err != nil {
			return err
		}
		if err := binary.Write(w, binary.LittleEndian, uint32(len(metric.Data))); err != nil {
			return err
		}
		for _, v := range metric.Data {
			if err := binary.Write(w, binary.LittleEndian, math.Float32bits(float32(v))); err != nil {
				return err
			}
		}


255
256		if errs > 0 {
257			return int(n), fmt.Errorf("[METRICSTORE]> %d errors happened while creating checkpoints (%d successes)", errs, n)
258		}
259		return int(n), nil
260	}
261
262	// toCheckpointFile recursively converts a Level tree to CheckpointFile structure.
263	// Skips metrics that are already archived. Returns nil if no unarchived data exists.
264	func (l *Level) toCheckpointFile(from, to int64, m *MemoryStore) (*CheckpointFile, error) {
265		l.lock.RLock()
266		defer l.lock.RUnlock()
267
268		retval := &CheckpointFile{
269			From:     from,
270			To:       to,
271			Metrics:  make(map[string]*CheckpointMetrics),
272			Children: make(map[string]*CheckpointFile),
273		}
274
275		for metric, minfo := range m.Metrics {
276			b := l.metrics[minfo.offset]
277			if b == nil {
278				continue
279			}
280
281			allArchived := true
282			b.iterFromTo(from, to, func(b *buffer) error {
283				if !b.archived {
284					allArchived = false
285					return fmt.Errorf("stop") // Early termination signal
286				}
287				return nil
288			})
289
290			if allArchived {
291				continue
292			}
293
294			data := make([]schema.Float, (to-from)/b.frequency+1)
295			data, start, end, err := b.read(from, to, data)
296			if err != nil {
297				return nil, err
298			}
299
300			for i := int((end - start) / b.frequency); i < len(data); i++ {
301				data[i] = schema.NaN
302			}
303
304			retval.Metrics[metric] = &CheckpointMetrics{
305				Frequency: b.frequency,
306				Start:     start,
307				Data:      data,
308			}
309		}
310
311		for name, child := range l.children {
312			val, err := child.toCheckpointFile(from, to, m)
313			if err != nil {
314				return nil, err
315			}
316
317			if val != nil {
318				retval.Children[name] = val
319			}
320		}
321
322		if len(retval.Children) == 0 && len(retval.Metrics) == 0 {
323			return nil, nil
324		}
325
326		return retval, nil
327	}
328
329	// toCheckpoint writes a Level's data to a JSON checkpoint file.
330	// Creates directory if needed. Returns ErrNoNewArchiveData if nothing to save.
331	func (l *Level) toCheckpoint(dir string, from, to int64, m *MemoryStore) error {
332		cf, err := l.toCheckpointFile(from, to, m)
333		if err != nil {
334			return err

	if errs > 0 {
		return int(n), fmt.Errorf("[METRICSTORE]> %d errors happened while creating checkpoints (%d successes)", errs, n)
	}
	return int(n), nil
}

// toCheckpointFile recursively converts a Level tree to CheckpointFile structure.
// Skips metrics that are already archived. Returns nil if no unarchived data exists.
func (l *Level) toCheckpointFile(from, to int64, m *MemoryStore) (*CheckpointFile, error) {
	l.lock.RLock()
	defer l.lock.RUnlock()

	retval := &CheckpointFile{
		From:     from,
		To:       to,
		Metrics:  make(map[string]*CheckpointMetrics),
		Children: make(map[string]*CheckpointFile),
	}

	for metric, minfo := range m.Metrics {
		b := l.metrics[minfo.offset]
		if b == nil {
			continue
		}

		allArchived := true
		b.iterFromTo(from, to, func(b *buffer) error {
			if !b.archived {
				allArchived = false
				return fmt.Errorf("stop") // Early termination signal
			}
			return nil
		})

		if allArchived {
			continue
		}

		data := make([]schema.Float, (to-from)/b.frequency+1)
		data, start, end, err := b.read(from, to, data)
		if err != nil {
			return nil, err
		}

		for i := int((end - start) / b.frequency); i < len(data); i++ {
			data[i] = schema.NaN
		}

		retval.Metrics[metric] = &CheckpointMetrics{
			Frequency: b.frequency,
			Start:     start,
			Data:      data,
		}
	}

	for name, child := range l.children {
		val, err := child.toCheckpointFile(from, to, m)
		if err != nil {
			return nil, err
		}

		if val != nil {
			retval.Children[name] = val
		}
	}

	if len(retval.Children) == 0 && len(retval.Metrics) == 0 {
		return nil, nil
	}

	return retval, nil
}

// toCheckpoint writes a Level's data to a JSON checkpoint file.
// Creates directory if needed. Returns ErrNoNewArchiveData if nothing to save.
func (l *Level) toCheckpoint(dir string, from, to int64, m *MemoryStore) error {
	cf, err := l.toCheckpointFile(from, to, m)
	if err != nil {
		return err


Let me read the WAL staging processing loop to understand what blocks during checkpoint.


225						if _, err := w.Write(hdr[:]); err != nil {
226							cclog.Errorf("[METRICSTORE]> WAL: write header %s: %v", walPath, err)
227							f.Close()
228							return nil
229						}
230						fileSize = 4
231					}
232
233					ws = &walFileState{f: f, w: w, size: fileSize}
234					hostFiles[hostDir] = ws
235					return ws
236				}
237
238				processMsg := func(msg *WALMessage) {
239					hostDir := path.Join(Keys.Checkpoints.RootDir, msg.Cluster, msg.Node)
240					ws := getOrOpenWAL(hostDir)
241					if ws == nil {
242						return
243					}
244
245					// Enforce max WAL size: force-rotate before writing if limit is exceeded.
246					// The in-memory store still holds the data; only crash-recovery coverage is lost.
247					if maxSize := Keys.Checkpoints.MaxWALSize; maxSize > 0 && ws.size >= maxSize {
248						cclog.Warnf("[METRICSTORE]> WAL: force-rotating %s (size %d >= limit %d)",
249							hostDir, ws.size, maxSize)
250						ws.w.Flush()
251						ws.f.Close()
252						walPath := path.Join(hostDir, "current.wal")
253						if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
254							cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
255						}
256						delete(hostFiles, hostDir)
257						ws = getOrOpenWAL(hostDir)
258						if ws == nil {
259							return
260						}
261					}
262
263					n, err := writeWALRecordDirect(ws.w, msg)
264					if err != nil {
265						cclog.Errorf("[METRICSTORE]> WAL: write record: %v", err)
266					}
267					ws.size += int64(n)
268					ws.dirty = true
269				}
270
271				processRotate := func(req walRotateReq) {
272					ws, ok := hostFiles[req.hostDir]
273					if ok && ws.f != nil {
274						ws.w.Flush()
275						ws.f.Close()
276						walPath := path.Join(req.hostDir, "current.wal")
277						if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
278							cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
279						}
280						delete(hostFiles, req.hostDir)
281					}
282					close(req.done)
283				}
284
285				flushDirty := func() {
286					for _, ws := range hostFiles {
287						if ws.dirty {
288							ws.w.Flush()
289							ws.dirty = false
290						}
291					}
292				}
293
294				ticker := time.NewTicker(walFlushInterval)
295				defer ticker.Stop()
296
297				// drainBatch processes up to 4096 pending messages without blocking.
298				// Returns false if the channel was closed.
299				drainBatch := func() bool {
300					for range 4096 {
301						select {
302						case msg, ok := <-msgCh:
303							if !ok {
304								flushDirty()
305								return false
306							}
307							processMsg(msg)
308						case req := <-rotateCh:
309							processRotate(req)
310						default:
311							return true
312						}
313					}
314					return true
315				}
316
317				for {
318					select {
319					case <-ctx.Done():
320						// On shutdown, skip draining buffered messages — a full binary
321						// checkpoint will be written from in-memory state, making
322						// buffered WAL records redundant.
323						flushDirty()
324						return
325					case msg, ok := <-msgCh:
326						if !ok {
327							return
328						}
329						processMsg(msg)
330						if !drainBatch() {
331							return
332						}
333						// No flush here — timer handles periodic flushing.
334					case <-ticker.C:
335						flushDirty()
336					case req := <-rotateCh:
337						processRotate(req)
338					}
339				}
340			})
341		}
342	}
343
344	// WaitForWALStagingDrain blocks until all WALStaging goroutines have exited.
345	// Must be called after closing walShardChs to ensure all file handles are
346	// flushed and closed before checkpoint writes begin.
347	func WaitForWALStagingDrain() {
348		walStagingWg.Wait()
349	}
350
351	// RotateWALFiles sends rotation requests for the given host directories
352	// and blocks until all rotations complete. Each request is routed to the
353	// shard that owns the host directory.
354	//

					if _, err := w.Write(hdr[:]); err != nil {
						cclog.Errorf("[METRICSTORE]> WAL: write header %s: %v", walPath, err)
						f.Close()
						return nil
					}
					fileSize = 4
				}

				ws = &walFileState{f: f, w: w, size: fileSize}
				hostFiles[hostDir] = ws
				return ws
			}

			processMsg := func(msg *WALMessage) {
				hostDir := path.Join(Keys.Checkpoints.RootDir, msg.Cluster, msg.Node)
				ws := getOrOpenWAL(hostDir)
				if ws == nil {
					return
				}

				// Enforce max WAL size: force-rotate before writing if limit is exceeded.
				// The in-memory store still holds the data; only crash-recovery coverage is lost.
				if maxSize := Keys.Checkpoints.MaxWALSize; maxSize > 0 && ws.size >= maxSize {
					cclog.Warnf("[METRICSTORE]> WAL: force-rotating %s (size %d >= limit %d)",
						hostDir, ws.size, maxSize)
					ws.w.Flush()
					ws.f.Close()
					walPath := path.Join(hostDir, "current.wal")
					if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
						cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
					}
					delete(hostFiles, hostDir)
					ws = getOrOpenWAL(hostDir)
					if ws == nil {
						return
					}
				}

				n, err := writeWALRecordDirect(ws.w, msg)
				if err != nil {
					cclog.Errorf("[METRICSTORE]> WAL: write record: %v", err)
				}
				ws.size += int64(n)
				ws.dirty = true
			}

			processRotate := func(req walRotateReq) {
				ws, ok := hostFiles[req.hostDir]
				if ok && ws.f != nil {
					ws.w.Flush()
					ws.f.Close()
					walPath := path.Join(req.hostDir, "current.wal")
					if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
						cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
					}
					delete(hostFiles, req.hostDir)
				}
				close(req.done)
			}

			flushDirty := func() {
				for _, ws := range hostFiles {
					if ws.dirty {
						ws.w.Flush()
						ws.dirty = false
					}
				}
			}

			ticker := time.NewTicker(walFlushInterval)
			defer ticker.Stop()

			// drainBatch processes up to 4096 pending messages without blocking.
			// Returns false if the channel was closed.
			drainBatch := func() bool {
				for range 4096 {
					select {
					case msg, ok := <-msgCh:
						if !ok {
							flushDirty()
							return false
						}
						processMsg(msg)
					case req := <-rotateCh:
						processRotate(req)
					default:
						return true
					}
				}
				return true
			}

			for {
				select {
				case <-ctx.Done():
					// On shutdown, skip draining buffered messages — a full binary
					// checkpoint will be written from in-memory state, making
					// buffered WAL records redundant.
					flushDirty()
					return
				case msg, ok := <-msgCh:
					if !ok {
						return
					}
					processMsg(msg)
					if !drainBatch() {
						return
					}
					// No flush here — timer handles periodic flushing.
				case <-ticker.C:
					flushDirty()
				case req := <-rotateCh:
					processRotate(req)
				}
			}
		})
	}
}

// WaitForWALStagingDrain blocks until all WALStaging goroutines have exited.
// Must be called after closing walShardChs to ensure all file handles are
// flushed and closed before checkpoint writes begin.
func WaitForWALStagingDrain() {
	walStagingWg.Wait()
}

// RotateWALFiles sends rotation requests for the given host directories
// and blocks until all rotations complete. Each request is routed to the
// shard that owns the host directory.
//


1	// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
2	// All rights reserved. This file is part of cc-backend.
3	// Use of this source code is governed by a MIT-style
4	// license that can be found in the LICENSE file.
5
6	// Package metricstore provides walCheckpoint.go: WAL-based checkpoint implementation.
7	//
8	// This replaces the Avro shadow tree with an append-only Write-Ahead Log (WAL)
9	// per host, eliminating the extra memory overhead of the AvroStore and providing
10	// truly continuous (per-write) crash safety.
11	//
12	// # Architecture
13	//
14	//	Metric write (DecodeLine)
15	//	      │
16	//	      ├─► WriteToLevel()    → main MemoryStore (unchanged)
17	//	      │
18	//	      └─► WALMessages channel
19	//	               │
20	//	               ▼
21	//	        WALStaging goroutine
22	//	               │
23	//	               ▼
24	//	        checkpoints/cluster/host/current.wal  (append-only, binary)
25	//
26	//	Periodic checkpoint (Checkpointing goroutine):
27	//	  1. Write <timestamp>.bin snapshot (column-oriented, from main tree)
28	//	  2. Signal WALStaging to truncate current.wal per host
29	//
30	//	On restart (FromCheckpoint):

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package metricstore provides walCheckpoint.go: WAL-based checkpoint implementation.
//
// This replaces the Avro shadow tree with an append-only Write-Ahead Log (WAL)
// per host, eliminating the extra memory overhead of the AvroStore and providing
// truly continuous (per-write) crash safety.
//
// # Architecture
//
//	Metric write (DecodeLine)
//	      │
//	      ├─► WriteToLevel()    → main MemoryStore (unchanged)
//	      │
//	      └─► WALMessages channel
//	               │
//	               ▼
//	        WALStaging goroutine
//	               │
//	               ▼
//	        checkpoints/cluster/host/current.wal  (append-only, binary)
//
//	Periodic checkpoint (Checkpointing goroutine):
//	  1. Write <timestamp>.bin snapshot (column-oriented, from main tree)
//	  2. Signal WALStaging to truncate current.wal per host
//
//	On restart (FromCheckpoint):


Now let me read the Checkpointing goroutine to see how it calls ToCheckpointWAL and understand the full lifecycle.


130						if err != nil {
131							cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
132						}
133						if n > 0 {
134							cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
135							lastCheckpointMu.Lock()
136							lastCheckpoint = now
137							lastCheckpointMu.Unlock()
138							// Rotate WAL files for successfully checkpointed hosts.
139							RotateWALFiles(hostDirs)
140						}
141					} else {
142						n, err := ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
143						if err != nil {
144							cclog.Errorf("[METRICSTORE]> checkpointing failed: %s", err.Error())
145						} else {
146							cclog.Infof("[METRICSTORE]> done: %d checkpoint files created", n)
147							lastCheckpointMu.Lock()
148							lastCheckpoint = now
149							lastCheckpointMu.Unlock()
150						}
151					}
152				}
153			}
154		})
155	}
156
157	// MarshalJSON provides optimized JSON encoding for CheckpointMetrics.
158	//
159	// Since schema.Float has custom MarshalJSON, serializing []Float has significant overhead.
160	// This method manually constructs JSON to avoid allocations and interface conversions.
161	func (cm *CheckpointMetrics) MarshalJSON() ([]byte, error) {
162		buf := make([]byte, 0, 128+len(cm.Data)*8)
163		buf = append(buf, `{"frequency":`...)
164		buf = strconv.AppendInt(buf, cm.Frequency, 10)
165		buf = append(buf, `,"start":`...)
166		buf = strconv.AppendInt(buf, cm.Start, 10)
167		buf = append(buf, `,"data":[`...)
168		for i, x := range cm.Data {
169			if i != 0 {
170				buf = append(buf, ',')
171			}
172			if x.IsNaN() {
173				buf = append(buf, `null`...)
174			} else {
175				buf = strconv.AppendFloat(buf, float64(x), 'f', 1, 32)
176			}
177		}
178		buf = append(buf, `]}`...)
179		return buf, nil
180	}
181
182	// ToCheckpoint writes metric data to checkpoint files in parallel (JSON format).
183	//
184	// Metrics at root and cluster levels are skipped. One file per host is created.
185	// Uses worker pool (Keys.NumWorkers) for parallel processing. Only locks one host
186	// at a time, allowing concurrent writes/reads to other hosts.
187	//
188	// Returns the number of checkpoint files created and any errors encountered.
189	func (m *MemoryStore) ToCheckpoint(dir string, from, to int64) (int, error) {
190		// Pre-calculate capacity by counting cluster/host pairs
191		m.root.lock.RLock()
192		totalHosts := 0
193		for _, l1 := range m.root.children {
194			l1.lock.RLock()
195			totalHosts += len(l1.children)
196			l1.lock.RUnlock()
197		}
198		m.root.lock.RUnlock()
199
200		levels := make([]*Level, 0, totalHosts)
201		selectors := make([][]string, 0, totalHosts)
202
203		m.root.lock.RLock()
204		for sel1, l1 := range m.root.children {
205			l1.lock.RLock()
206			for sel2, l2 := range l1.children {
207				levels = append(levels, l2)
208				selectors = append(selectors, []string{sel1, sel2})
209			}

					if err != nil {
						cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
					}
					if n > 0 {
						cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
						lastCheckpointMu.Lock()
						lastCheckpoint = now
						lastCheckpointMu.Unlock()
						// Rotate WAL files for successfully checkpointed hosts.
						RotateWALFiles(hostDirs)
					}
				} else {
					n, err := ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
					if err != nil {
						cclog.Errorf("[METRICSTORE]> checkpointing failed: %s", err.Error())
					} else {
						cclog.Infof("[METRICSTORE]> done: %d checkpoint files created", n)
						lastCheckpointMu.Lock()
						lastCheckpoint = now
						lastCheckpointMu.Unlock()
					}
				}
			}
		}
	})
}

// MarshalJSON provides optimized JSON encoding for CheckpointMetrics.
//
// Since schema.Float has custom MarshalJSON, serializing []Float has significant overhead.
// This method manually constructs JSON to avoid allocations and interface conversions.
func (cm *CheckpointMetrics) MarshalJSON() ([]byte, error) {
	buf := make([]byte, 0, 128+len(cm.Data)*8)
	buf = append(buf, `{"frequency":`...)
	buf = strconv.AppendInt(buf, cm.Frequency, 10)
	buf = append(buf, `,"start":`...)
	buf = strconv.AppendInt(buf, cm.Start, 10)
	buf = append(buf, `,"data":[`...)
	for i, x := range cm.Data {
		if i != 0 {
			buf = append(buf, ',')
		}
		if x.IsNaN() {
			buf = append(buf, `null`...)
		} else {
			buf = strconv.AppendFloat(buf, float64(x), 'f', 1, 32)
		}
	}
	buf = append(buf, `]}`...)
	return buf, nil
}

// ToCheckpoint writes metric data to checkpoint files in parallel (JSON format).
//
// Metrics at root and cluster levels are skipped. One file per host is created.
// Uses worker pool (Keys.NumWorkers) for parallel processing. Only locks one host
// at a time, allowing concurrent writes/reads to other hosts.
//
// Returns the number of checkpoint files created and any errors encountered.
func (m *MemoryStore) ToCheckpoint(dir string, from, to int64) (int, error) {
	// Pre-calculate capacity by counting cluster/host pairs
	m.root.lock.RLock()
	totalHosts := 0
	for _, l1 := range m.root.children {
		l1.lock.RLock()
		totalHosts += len(l1.children)
		l1.lock.RUnlock()
	}
	m.root.lock.RUnlock()

	levels := make([]*Level, 0, totalHosts)
	selectors := make([][]string, 0, totalHosts)

	m.root.lock.RLock()
	for sel1, l1 := range m.root.children {
		l1.lock.RLock()
		for sel2, l2 := range l1.children {
			levels = append(levels, l2)
			selectors = append(selectors, []string{sel1, sel2})
		}


100	var walShuttingDown atomic.Bool
101
102	// WALMessage represents a single metric write to be appended to the WAL.
103	// Cluster and Node are NOT stored in the WAL record (inferred from file path).
104	type WALMessage struct {
105		MetricName string
106		Cluster    string
107		Node       string
108		Selector   []string
109		Value      schema.Float
110		Timestamp  int64
111	}
112
113	// walRotateReq requests WAL file rotation for a specific host directory.
114	// The done channel is closed by the WAL goroutine when rotation is complete.
115	type walRotateReq struct {
116		hostDir string
117		done    chan struct{}
118	}
119
120	// walFileState holds an open WAL file handle and buffered writer for one host directory.
121	type walFileState struct {
122		f     *os.File
123		w     *bufio.Writer
124		dirty bool
125		size  int64 // approximate bytes written (tracked from open + writes)
126	}
127
128	// walFlushInterval controls how often dirty WAL files are flushed to disk.
129	// Decoupling flushes from message processing lets the consumer run at memory

var walShuttingDown atomic.Bool

// WALMessage represents a single metric write to be appended to the WAL.
// Cluster and Node are NOT stored in the WAL record (inferred from file path).
type WALMessage struct {
	MetricName string
	Cluster    string
	Node       string
	Selector   []string
	Value      schema.Float
	Timestamp  int64
}

// walRotateReq requests WAL file rotation for a specific host directory.
// The done channel is closed by the WAL goroutine when rotation is complete.
type walRotateReq struct {
	hostDir string
	done    chan struct{}
}

// walFileState holds an open WAL file handle and buffered writer for one host directory.
type walFileState struct {
	f     *os.File
	w     *bufio.Writer
	dirty bool
	size  int64 // approximate bytes written (tracked from open + writes)
}

// walFlushInterval controls how often dirty WAL files are flushed to disk.
// Decoupling flushes from message processing lets the consumer run at memory


Now let me also check if there's an existing pause mechanism and look at the config for numWorkers.


pkg/metricstore/lineprotocol.go:127:// walDropped counts WAL messages dropped due to a full channel.
pkg/metricstore/lineprotocol.go:129:var walDropped atomic.Int64
pkg/metricstore/lineprotocol.go:367:				if dropped := walDropped.Add(1); dropped%10000 == 1 {


85	// walShardChs holds per-shard channels for WAL messages.
86	// Initialized by WALStaging; nil when WAL is not active.
87	var walShardChs []chan *WALMessage
88
89	// walShardRotateChs holds per-shard channels for WAL rotation requests.
90	var walShardRotateChs []chan walRotateReq
91
92	// walNumShards stores the number of shards (set during WALStaging init).
93	var walNumShards int
94
95	// walStagingWg tracks WALStaging goroutine exits for shutdown synchronization.
96	var walStagingWg sync.WaitGroup
97
98	// walShuttingDown is set before closing shard channels to prevent
99	// SendWALMessage from sending on a closed channel (which panics in Go).
100	var walShuttingDown atomic.Bool
101
102	// WALMessage represents a single metric write to be appended to the WAL.
103	// Cluster and Node are NOT stored in the WAL record (inferred from file path).
104	type WALMessage struct {

// walShardChs holds per-shard channels for WAL messages.
// Initialized by WALStaging; nil when WAL is not active.
var walShardChs []chan *WALMessage

// walShardRotateChs holds per-shard channels for WAL rotation requests.
var walShardRotateChs []chan walRotateReq

// walNumShards stores the number of shards (set during WALStaging init).
var walNumShards int

// walStagingWg tracks WALStaging goroutine exits for shutdown synchronization.
var walStagingWg sync.WaitGroup

// walShuttingDown is set before closing shard channels to prevent
// SendWALMessage from sending on a closed channel (which panics in Go).
var walShuttingDown atomic.Bool

// WALMessage represents a single metric write to be appended to the WAL.
// Cluster and Node are NOT stored in the WAL record (inferred from file path).
type WALMessage struct {


Now let me also look at the RotateWALFiles function and check if the WAL deletion in ToCheckpointWAL (line 731) races with the staging goroutine.


350
351	// RotateWALFiles sends rotation requests for the given host directories
352	// and blocks until all rotations complete. Each request is routed to the
353	// shard that owns the host directory.
354	//
355	// If shutdown is in progress (WAL staging goroutines may have exited),
356	// rotation is skipped to avoid deadlocking on abandoned channels.
357	func RotateWALFiles(hostDirs []string) {
358		if walShardRotateChs == nil || walShuttingDown.Load() {
359			return
360		}
361		deadline := time.After(2 * time.Minute)
362		dones := make([]chan struct{}, 0, len(hostDirs))
363		for _, dir := range hostDirs {
364			done := make(chan struct{})
365			shard := walShardIndexFromDir(dir)
366			select {
367			case walShardRotateChs[shard] <- walRotateReq{hostDir: dir, done: done}:
368				dones = append(dones, done)
369			case <-deadline:
370				cclog.Warnf("[METRICSTORE]> WAL rotation send timed out, %d of %d hosts remaining",
371					len(hostDirs)-len(dones), len(hostDirs))
372				goto waitDones
373			}
374		}
375	waitDones:
376		for _, done := range dones {
377			select {
378			case <-done:
379			case <-time.After(30 * time.Second):
380				cclog.Warn("[METRICSTORE]> WAL rotation completion timed out, continuing")
381				return
382			}
383		}
384	}
385
386	// walShardIndexFromDir extracts cluster and node from a host directory path
387	// and returns the shard index. The path format is: rootDir/cluster/node
388	func walShardIndexFromDir(hostDir string) int {
389		// hostDir = .../cluster/node — extract last two path components
390		node := path.Base(hostDir)
391		cluster := path.Base(path.Dir(hostDir))
392		return walShardIndex(cluster, node)
393	}
394
395	// RotateWALFiles sends rotation requests for the given host directories
396	// and blocks until all rotations complete.
397	func RotateWALFilesAfterShutdown(hostDirs []string) {
398		for _, dir := range hostDirs {
399			walPath := path.Join(dir, "current.wal")
400			if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
401				cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
402			}
403		}
404	}
405
406	// writeWALRecordDirect encodes a WAL record into a contiguous buffer first,
407	// then writes it to the bufio.Writer in a single call. This prevents partial
408	// records in the write buffer if a write error occurs mid-record (e.g. disk full).
409	// Returns the number of bytes written and any error.

// RotateWALFiles sends rotation requests for the given host directories
// and blocks until all rotations complete. Each request is routed to the
// shard that owns the host directory.
//
// If shutdown is in progress (WAL staging goroutines may have exited),
// rotation is skipped to avoid deadlocking on abandoned channels.
func RotateWALFiles(hostDirs []string) {
	if walShardRotateChs == nil || walShuttingDown.Load() {
		return
	}
	deadline := time.After(2 * time.Minute)
	dones := make([]chan struct{}, 0, len(hostDirs))
	for _, dir := range hostDirs {
		done := make(chan struct{})
		shard := walShardIndexFromDir(dir)
		select {
		case walShardRotateChs[shard] <- walRotateReq{hostDir: dir, done: done}:
			dones = append(dones, done)
		case <-deadline:
			cclog.Warnf("[METRICSTORE]> WAL rotation send timed out, %d of %d hosts remaining",
				len(hostDirs)-len(dones), len(hostDirs))
			goto waitDones
		}
	}
waitDones:
	for _, done := range dones {
		select {
		case <-done:
		case <-time.After(30 * time.Second):
			cclog.Warn("[METRICSTORE]> WAL rotation completion timed out, continuing")
			return
		}
	}
}

// walShardIndexFromDir extracts cluster and node from a host directory path
// and returns the shard index. The path format is: rootDir/cluster/node
func walShardIndexFromDir(hostDir string) int {
	// hostDir = .../cluster/node — extract last two path components
	node := path.Base(hostDir)
	cluster := path.Base(path.Dir(hostDir))
	return walShardIndex(cluster, node)
}

// RotateWALFiles sends rotation requests for the given host directories
// and blocks until all rotations complete.
func RotateWALFilesAfterShutdown(hostDirs []string) {
	for _, dir := range hostDirs {
		walPath := path.Join(dir, "current.wal")
		if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
			cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
		}
	}
}

// writeWALRecordDirect encodes a WAL record into a contiguous buffer first,
// then writes it to the bufio.Writer in a single call. This prevents partial
// records in the write buffer if a write error occurs mid-record (e.g. disk full).
// Returns the number of bytes written and any error.


Now I have a thorough understanding of the problem. Let me design the plan.


Design WAL checkpoint fix

Prompt:
Design an implementation to fix WAL shard channel saturation during binary checkpointing in a metric store.

## Problem
During periodic binary checkpoints (every 12h), ~1.44M WAL messages are dropped because shard channels fill up. The checkpoint takes 1845 hosts × 10 workers and runs for ~12+ seconds. During this time, incoming metrics keep arriving but WAL staging goroutines can't keep up due to disk I/O contention from checkpoint writes.

## Key Insight
WAL writes during checkpoint are REDUNDANT: the binary snapshot captures ALL in-memory data. After checkpoint, WAL files are rotated (deleted). So any WAL writes during the checkpoint window are wasted work that will be immediately discarded.

## Current Architecture
- `lineprotocol.go:364`: `SendWALMessage(msg)` — non-blocking send to shard channel, drops if full
- `walCheckpoint.go:100`: `var walShuttingDown atomic.Bool` — existing flag for shutdown
- `walCheckpoint.go:145-156`: `SendWALMessage()` — checks `walShuttingDown`, then does select send
- `walCheckpoint.go:658`: `ToCheckpointWAL()` — the binary checkpoint writer
- `checkpoint.go:128-139`: Checkpointing goroutine calls `ToCheckpointWAL()` then `RotateWALFiles()`
- `walCheckpoint.go:730-733`: Inside ToCheckpointWAL, each worker deletes WAL immediately after successful snapshot (race with staging goroutine that still has file open!)

## Proposed Fix: Pause WAL during checkpoint

Add an atomic flag `walCheckpointActive` that causes `SendWALMessage` to skip sending (return true, not false — because data is safe in memory and snapshot will capture it). Set the flag before checkpoint starts, clear it after checkpoint + rotation completes.

### Specific changes:

1. **walCheckpoint.go**: Add `var walCheckpointActive atomic.Bool` next to `walShuttingDown`

2. **walCheckpoint.go:SendWALMessage()**: Add early return if `walCheckpointActive.Load()` — return `true` (not false) since data isn't lost

3. **checkpoint.go**: Set `walCheckpointActive.Store(true)` before calling `ToCheckpointWAL()`, and `walCheckpointActive.Store(false)` after `RotateWALFiles()` completes. Also reset `walDropped` counter.

4. **Also fix the race condition**: ToCheckpointWAL line 730-733 deletes WAL directly from checkpoint worker while staging goroutine has the file open. This should be removed — RotateWALFiles already handles this properly via the staging goroutine.

### Benefits:
- Zero WAL drops during checkpoint (nothing to drop)
- Reduced disk I/O during checkpoint (no WAL writes competing)
- Faster checkpoint completion
- Eliminates the file deletion race condition

### Concerns to address:
- If the process crashes DURING checkpoint (after snapshot written for some hosts but not all), the pre-checkpoint WAL data for un-snapshotted hosts is needed for recovery. But since we're pausing WAL writes for NEW incoming data (which is already in memory), and the existing WAL files are untouched until after snapshot succeeds, crash recovery is fine.

Please evaluate this approach and suggest any improvements or issues.


Now I have a complete picture. Let me provide the evaluation.

---

## Evaluation of the Proposed Approach

The proposal is fundamentally sound. I have verified the code paths and confirmed the key claims. Here is a detailed analysis.

### Confirming the Architecture

The flow is:

1. `DecodeLine()` in `lineprotocol.go` writes each metric to both the in-memory store (`WriteToLevel`) and the WAL (`SendWALMessage`) at line 355-369.
2. `SendWALMessage()` in `walCheckpoint.go:145-156` does a non-blocking send to a shard channel. If the channel is full, it returns `false` and the caller increments `walDropped`.
3. `Checkpointing()` in `checkpoint.go:128-139` calls `ToCheckpointWAL()` then `RotateWALFiles()`.
4. `ToCheckpointWAL()` in `walCheckpoint.go:658-760` writes binary snapshots for all hosts in parallel, and **on line 730-733, each worker deletes the WAL file directly via `os.Remove`**.
5. `RotateWALFiles()` in `walCheckpoint.go:357-384` sends rotation requests through the shard channels to the staging goroutines, which close the file handle, delete the file, and remove it from their map.

### Confirmed Issues

**Issue 1: WAL writes during checkpoint are wasted work.** Correct. The binary snapshot at `toCheckpointBinary` reads directly from the in-memory Level tree (via `toCheckpointFile`), capturing everything currently in memory. After the snapshot, `RotateWALFiles` deletes the WAL. Any WAL records written during this window are immediately discarded.

**Issue 2: Race condition on WAL file deletion (lines 730-733).** This is a real bug. The checkpoint worker calls `os.Remove(walPath)` directly on the WAL file while the staging goroutine in `WALStaging` may still have the file open via its `walFileState` map entry. The staging goroutine's `getOrOpenWAL` caches file handles in `hostFiles`, and nothing tells it the file was deleted. On Linux, the staging goroutine can continue writing to the unlinked fd (data goes nowhere), and when it next opens a new WAL, it creates a fresh file. But on some edge cases, this can cause confusing behavior. More importantly, `RotateWALFiles` is called right after (checkpoint.go:139), which sends a rotation request through the proper channel -- but by then the file is already deleted by the worker, so the staging goroutine tries to remove an already-gone file (benign due to the `!os.IsNotExist` guard) and closes the handle.

The double-delete is wasteful but not catastrophic. However, the staging goroutine still holds a stale `walFileState` entry between the worker's `os.Remove` and the rotation request arriving. During that window, any incoming WAL messages for that host would be written to the deleted fd (on Linux) or fail (on other platforms).

### Evaluation of the Proposed Fix

The proposal to add `walCheckpointActive atomic.Bool` is clean and correct. Here is my assessment of each change:

**Change 1: Add `var walCheckpointActive atomic.Bool`** -- Straightforward. Place it next to `walShuttingDown` at line 100 for clarity.

**Change 2: Early return in `SendWALMessage` returning `true`** -- Correct. Returning `true` is the right choice because the data is not lost; it is in-memory and will be captured by the snapshot. This also prevents the `walDropped` counter from being incremented in `lineprotocol.go:367`, which is the desired behavior (no false alarms).

**Change 3: Set/clear flag around checkpoint + rotation in `checkpoint.go`** -- Correct. The flag should be set before `ToCheckpointWAL()` and cleared after `RotateWALFiles()` completes, so the entire window of snapshot-write + WAL-rotation is covered.

**Change 4: Remove direct WAL deletion in ToCheckpointWAL lines 730-733** -- This is the most important fix. The `os.Remove` on line 731-733 should be removed entirely. `RotateWALFiles` (called from `checkpoint.go:139`) already handles deletion properly through the staging goroutine's `processRotate` function (walCheckpoint.go:271-283), which flushes, closes the handle, removes the file, and cleans up the map entry. The direct delete is both redundant and racy.

### Suggested Improvements

**1. Reset `walDropped` counter after checkpoint.** The proposal mentions this. After clearing the flag, add `walDropped.Store(0)` in `checkpoint.go` so the next checkpoint cycle starts with a clean count. This avoids confusing log messages that show accumulated drops from previous cycles.

**2. Log the pause/resume.** Add a brief info log when the flag is set and cleared, so operators can correlate any timing anomalies:
```
cclog.Infof("[METRICSTORE]> pausing WAL writes during binary checkpoint")
// ... checkpoint + rotate ...
cclog.Infof("[METRICSTORE]> resuming WAL writes after binary checkpoint")
```

**3. Consider a defer for clearing the flag.** If `ToCheckpointWAL` returns an error, the current proposal in `checkpoint.go:128-139` does not call `RotateWALFiles` (it only rotates when `n > 0`). The flag would remain set forever, permanently disabling WAL writes. Use a `defer` or ensure the flag is cleared in all exit paths:

Looking at `checkpoint.go:128-139`:
```go
if Keys.Checkpoints.FileFormat == "wal" {
    n, hostDirs, err := ms.ToCheckpointWAL(...)
    if err != nil {
        cclog.Errorf(...)
    }
    if n > 0 {
        // ...
        RotateWALFiles(hostDirs)
    }
}
```

If `ToCheckpointWAL` returns `n=0` with an error, the flag would never be cleared. The flag must be cleared unconditionally. The safest pattern:

```go
walCheckpointActive.Store(true)
defer walCheckpointActive.Store(false)
```

This should be placed just before the `ToCheckpointWAL` call, within the `if Keys.Checkpoints.FileFormat == "wal"` block.

**4. Drain shard channels after clearing the flag (optional optimization).** When the flag is cleared, the shard channels are empty (no messages were queued during the pause). This is actually beneficial -- the staging goroutines have an empty pipeline and can immediately start processing new messages without backlog. No drain step needed.

**5. Crash safety analysis.** The proposal's crash safety argument is correct. During the checkpoint window:
- Pre-existing WAL files on disk remain untouched until `RotateWALFiles` (which happens after all snapshots succeed).
- New incoming metrics are written to the in-memory store (line 373 in `lineprotocol.go` runs regardless of WAL status).
- If the process crashes mid-checkpoint, on restart `fromCheckpoint` loads the most recent complete `.bin` snapshot plus replays the pre-existing `current.wal`. No data is lost because: (a) snapshots use atomic rename (`.bin.tmp` -> `.bin`), so partial snapshots are invisible; (b) the old WAL is still on disk.
- The only risk: metrics that arrive DURING checkpoint but AFTER the snapshot reads a particular host's in-memory data would not be in either the snapshot or the WAL. This is a very small window (the read lock in `toCheckpointFile` is brief) and would affect at most a few seconds of data. This is the same risk that exists today (those metrics would have been written to WAL, but the WAL would have been deleted post-checkpoint anyway). So the proposal does not make crash safety any worse.

### Implementation Plan

**Step 1**: In `walCheckpoint.go`, add `var walCheckpointActive atomic.Bool` at line ~100, next to `walShuttingDown`.

**Step 2**: In `walCheckpoint.go`, modify `SendWALMessage()` (lines 145-156) to check the new flag:
```go
func SendWALMessage(msg *WALMessage) bool {
    if walShardChs == nil || walShuttingDown.Load() {
        return false
    }
    if walCheckpointActive.Load() {
        return true  // Data safe in memory; snapshot will capture it
    }
    shard := walShardIndex(msg.Cluster, msg.Node)
    select {
    case walShardChs[shard] <- msg:
        return true
    default:
        return false
    }
}
```

**Step 3**: In `walCheckpoint.go`, remove lines 730-733 (the direct `os.Remove` of WAL files inside `ToCheckpointWAL` workers). Keep the `successDirs` tracking since `RotateWALFiles` still needs it.

**Step 4**: In `checkpoint.go`, wrap the WAL checkpoint block (lines 128-139) with the flag:
```go
if Keys.Checkpoints.FileFormat == "wal" {
    walCheckpointActive.Store(true)
    n, hostDirs, err := ms.ToCheckpointWAL(...)
    if err != nil {
        cclog.Errorf(...)
    }
    if n > 0 {
        cclog.Infof(...)
        lastCheckpointMu.Lock()
        lastCheckpoint = now
        lastCheckpointMu.Unlock()
        RotateWALFiles(hostDirs)
    }
    walCheckpointActive.Store(false)
    walDropped.Store(0)
}
```

Note that `walCheckpointActive.Store(false)` is placed outside the `if n > 0` block to ensure it is always cleared, even on full failure.

**Step 5**: Add tests. A unit test should verify that `SendWALMessage` returns `true` (not `false`) when `walCheckpointActive` is set, confirming the caller won't increment `walDropped`.

### Critical Files for Implementation
- /Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/walCheckpoint.go
- /Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/checkpoint.go
- /Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/lineprotocol.go

Prompt:
Design an implementation to fix WAL shard channel saturation during binary checkpointing in a metric store.

## Problem
During periodic binary checkpoints (every 12h), ~1.44M WAL messages are dropped because shard channels fill up. The checkpoint takes 1845 hosts × 10 workers and runs for ~12+ seconds. During this time, incoming metrics keep arriving but WAL staging goroutines can't keep up due to disk I/O contention from checkpoint writes.

## Key Insight
WAL writes during checkpoint are REDUNDANT: the binary snapshot captures ALL in-memory data. After checkpoint, WAL files are rotated (deleted). So any WAL writes during the checkpoint window are wasted work that will be immediately discarded.

## Current Architecture
- `lineprotocol.go:364`: `SendWALMessage(msg)` — non-blocking send to shard channel, drops if full
- `walCheckpoint.go:100`: `var walShuttingDown atomic.Bool` — existing flag for shutdown
- `walCheckpoint.go:145-156`: `SendWALMessage()` — checks `walShuttingDown`, then does select send
- `walCheckpoint.go:658`: `ToCheckpointWAL()` — the binary checkpoint writer
- `checkpoint.go:128-139`: Checkpointing goroutine calls `ToCheckpointWAL()` then `RotateWALFiles()`
- `walCheckpoint.go:730-733`: Inside ToCheckpointWAL, each worker deletes WAL immediately after successful snapshot (race with staging goroutine that still has file open!)

## Proposed Fix: Pause WAL during checkpoint

Add an atomic flag `walCheckpointActive` that causes `SendWALMessage` to skip sending (return true, not false — because data is safe in memory and snapshot will capture it). Set the flag before checkpoint starts, clear it after checkpoint + rotation completes.

### Specific changes:

1. **walCheckpoint.go**: Add `var walCheckpointActive atomic.Bool` next to `walShuttingDown`

2. **walCheckpoint.go:SendWALMessage()**: Add early return if `walCheckpointActive.Load()` — return `true` (not false) since data isn't lost

3. **checkpoint.go**: Set `walCheckpointActive.Store(true)` before calling `ToCheckpointWAL()`, and `walCheckpointActive.Store(false)` after `RotateWALFiles()` completes. Also reset `walDropped` counter.

4. **Also fix the race condition**: ToCheckpointWAL line 730-733 deletes WAL directly from checkpoint worker while staging goroutine has the file open. This should be removed — RotateWALFiles already handles this properly via the staging goroutine.

### Benefits:
- Zero WAL drops during checkpoint (nothing to drop)
- Reduced disk I/O during checkpoint (no WAL writes competing)
- Faster checkpoint completion
- Eliminates the file deletion race condition

### Concerns to address:
- If the process crashes DURING checkpoint (after snapshot written for some hosts but not all), the pre-checkpoint WAL data for un-snapshotted hosts is needed for recovery. But since we're pausing WAL writes for NEW incoming data (which is already in memory), and the existing WAL files are untouched until after snapshot succeeds, crash recovery is fine.

Please evaluate this approach and suggest any improvements or issues.


Good analysis. Let me write the final plan.


Write

# Fix: WAL shard channel saturation during binary checkpoint

## Context

During periodic binary checkpoints (~every 12h), 1.44M+ WAL messages are dropped because shard channels fill up. The checkpoint processes 1845 hosts with 10 workers, taking ~12+ seconds. During this time, incoming metrics keep arriving but WAL staging goroutines can't keep up due to disk I/O contention from checkpoint binary writes.

**Key insight:** WAL writes during checkpoint are completely redundant — the binary snapshot reads directly from the in-memory Level tree, capturing everything. After the snapshot, WAL files are rotated (deleted). Any WAL records written during this window are immediately discarded.

## Changes

### 1. Add `walCheckpointActive` atomic flag (`walCheckpoint.go`)

Add `var walCheckpointActive atomic.Bool` at ~line 100, next to `walShuttingDown`.

### 2. Skip WAL sends during checkpoint (`walCheckpoint.go:SendWALMessage`)

Add early return `true` (not `false` — data is safe in memory) when `walCheckpointActive.Load()` is set. This prevents channel pressure AND avoids incrementing `walDropped`.

```go
func SendWALMessage(msg *WALMessage) bool {
    if walShardChs == nil || walShuttingDown.Load() {
        return false
    }
    if walCheckpointActive.Load() {
        return true // Data safe in memory; snapshot will capture it
    }
    // ... existing select/send logic
}
```

### 3. Remove direct WAL deletion race in `ToCheckpointWAL` (`walCheckpoint.go:730-733`)

Remove the `os.Remove(walPath)` inside the checkpoint worker loop. This races with the WAL staging goroutine that still has the file open. `RotateWALFiles` (called afterward from `checkpoint.go:139`) already handles deletion properly through the staging goroutine's `processRotate`.

### 4. Set/clear flag around checkpoint (`checkpoint.go:~128-139`)

```go
if Keys.Checkpoints.FileFormat == "wal" {
    walCheckpointActive.Store(true)
    n, hostDirs, err := ms.ToCheckpointWAL(...)
    if err != nil {
        cclog.Errorf(...)
    }
    if n > 0 {
        // ... existing success handling ...
        RotateWALFiles(hostDirs)
    }
    walCheckpointActive.Store(false)  // Always clear, even on error
    walDropped.Store(0)               // Reset drop counter
}
```

## Files to modify

- `pkg/metricstore/walCheckpoint.go` — add flag, modify `SendWALMessage`, remove direct WAL delete
- `pkg/metricstore/checkpoint.go` — set/clear flag around checkpoint call
- `pkg/metricstore/lineprotocol.go` — no changes needed (`walDropped` is already exported)

## Crash safety

No regression. During the checkpoint window:
- Pre-existing WAL files remain on disk until `RotateWALFiles` (after all snapshots succeed)
- New metrics are still written to in-memory store (`WriteToLevel` at line 373 runs regardless)
- If crash mid-checkpoint: restart loads latest complete `.bin` + replays old `current.wal` — no data loss since snapshots use atomic rename (`.bin.tmp` → `.bin`)

## Verification

1. `go build ./...` — compiles
2. `go test ./pkg/metricstore/...` — existing tests pass
3. Deploy and confirm no more "WAL shard channel full, dropped N messages" warnings during checkpoint
4. Confirm checkpoint logs show normal "Starting binary checkpoint" → "done: N binary snapshot files created" without drops