Write

# Comprehensive WAL Buffer Pipeline Review

## Executive Summary

The sharded WAL buffer pipeline has several critical architectural issues, race conditions, and design flaws. This review identifies 9 major bugs/races, 4 architectural issues, and multiple improvement opportunities.

---

## FILE LOCATIONS

1. **walCheckpoint.go** - WAL write staging and rotation
2. **lineprotocol.go** - Metric ingestion producer (sends to WAL channels)
3. **metricstore.go** - Init(), Shutdown(), global state management
4. **checkpoint.go** - Checkpointing goroutine, binary snapshot creation

---

## CRITICAL BUGS & RACES

### BUG #1: Race Between Shutdown and WAL Channel Close (CRITICAL)

**Location**: `metricstore.go` lines 283-289 (Shutdown function)

```go
if Keys.Checkpoints.FileFormat == "wal" {
    for _, ch := range walShardChs {
        close(ch)           // <-- Line 285
    }
    drainStart := time.Now()
    WaitForWALStagingDrain()
    cclog.Infof("[METRICSTORE]> WAL staging goroutines exited (%v)", time.Since(drainStart))
}
```

**Problem**:
- **Concurrent send after close**: Between lines 285 and the actual exit of WALStaging goroutines, DecodeLine can still call `SendWALMessage()` (line 364 in lineprotocol.go), which does a non-blocking send on a closed channel → **PANIC**
- **No mutual exclusion**: The lineprotocol workers (ReceiveNats) are in a `wg.Wait()` loop that *blocks* until context is cancelled, but there's a race window:
  1. Shutdown() closes channels (line 285)
  2. Running NATS decoder still in flight tries to send (SendWALMessage)
  3. Panic on send to closed channel

**Timeline**:
1. `Shutdown()` called → cancels ctx
2. `ReceiveNats` exits *after* ctx.Done() is read
3. But **outstanding metrics** from NATS callback are in flight
4. Those metrics call `SendWALMessage()` with closed channel → panic

**Fix**: Close channels AFTER ensuring no more producers. Add a flag or callback to prevent new sends before close.

---

### BUG #2: WAL Channel Close Panic on Non-blocking Send

**Location**: `lineprotocol.go` line 364 (SendWALMessage)

```go
if !SendWALMessage(msg) {
    // Dropped — logged periodically
}
```

**Location**: `walCheckpoint.go` line 145-150 (SendWALMessage)

```go
func SendWALMessage(msg *WALMessage) bool {
    if walShardChs == nil {
        return false
    }
    shard := walShardIndex(msg.Cluster, msg.Node)
    select {
    case walShardChs[shard] <- msg:
        return true
    default:
        return false  // <-- Non-blocking, but what if channel is closed?
    }
}
```

**Problem**:
- After Shutdown closes the channels, any in-flight send to `walShardChs[shard]` will panic (Go panics on send to closed channel)
- The non-blocking `select default:` doesn't protect against closed channels
- A sender on a closed channel always causes panic, regardless of blocking/non-blocking

**Evidence**: Go spec: "A send on a closed channel proceeds by causing a run-time panic"

**Root cause**: No synchronization between shutdown and metric ingestion threads.

---

### BUG #3: Race in WAL Staging Goroutine Drain Loop

**Location**: `walCheckpoint.go` lines 268-284 (drainBatch function)

```go
drainBatch := func() bool {
    for range 4096 {  // <-- Max 4096 messages per drain
        select {
        case msg, ok := <-msgCh:
            if !ok {  // <-- Channel closed
                flushDirty()
                return false  // Signal to exit outer loop
            }
            processMsg(msg)
        case req := <-rotateCh:
            processRotate(req)
        default:
            return true  // <-- Returns true if both channels empty
        }
    }
    return true
}
```

**Problem**:
- The main loop (lines 286-308) reads from msgCh and rotateCh:
  ```go
  for {
      select {
      case <-ctx.Done():
          flushDirty()
          return
      case msg, ok := <-msgCh:
          if !ok {
              return
          }
          processMsg(msg)
          if !drainBatch() {  // <-- Call drainBatch
              return
          }
      ...
      }
  }
  ```

- **Race**: If msgCh gets one message then closes, drainBatch() processes it, then returns true (line 280 default case). But the channel is CLOSED. Next iteration of outer loop reads from closed msgCh → nil msg with ok=false → exit.

- **Actual race**: Between outer loop reading msgCh (line 294) and drainBatch (line 299), another message might arrive. drainBatch might drain it and hit the default case, returning early. The outer loop then blocks on the now-empty channel.

- **Worse**: The drainBatch function is called to drain up to 4096 messages, but if it hits the default (all channels empty), it returns true. The outer loop then goes back to the select, which blocks. This means in low-message scenarios, messages sit in the buffer longer than needed.

---

### BUG #4: Lost Rotation Requests During Shutdown

**Location**: `metricstore.go` line 283-289 (Shutdown) vs `walCheckpoint.go` line 240-252 (processRotate)

**Problem**:
- When Shutdown closes walShardChs (producer channel), the WAL staging goroutines wake up to process buffered messages
- But `walShardRotateChs` is NEVER closed
- If a checkpoint creates rotation requests via `RotateWALFiles()` (checkpoint.go line 138) AFTER the producer channel is closed, those requests may get stuck:
  1. Checkpointing goroutine calls `RotateWALFiles(hostDirs)` at line 138
  2. Rotation requests are sent to `walShardRotateChs` at line 333
  3. But WAL staging goroutine may already be draining or exiting because msgCh was closed
  4. The rotation request blocks indefinitely because no goroutine is reading rotateCh

**Evidence**:
```go
// In RotateWALFiles (lines 323-338):
walShardRotateChs[shard] <- walRotateReq{hostDir: dir, done: dones[i]}
// ^^ This can block forever if the shard goroutine is exiting!

for _, done := range dones {
    <-done  // <-- BLOCKS FOREVER
}
```

**Timeline**:
1. Checkpointing goroutine gets hostDirs from ToCheckpointWAL
2. Calls RotateWALFiles(hostDirs)
3. Sends rotation request to walShardRotateChs[shard]
4. WAL staging goroutine for that shard already exited (because msgCh was closed)
5. No goroutine reads from rotateCh → blocks forever → Shutdown hangs

---

### BUG #5: Channel Closing Order Deadlock During Normal Checkpointing

**Location**: `checkpoint.go` lines 128-139 (Checkpointing goroutine with WAL format)

```go
if Keys.Checkpoints.FileFormat == "wal" {
    n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
    if err != nil {
        cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
    } else {
        cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
        lastCheckpointMu.Lock()
        lastCheckpoint = now
        lastCheckpointMu.Unlock()
        // Rotate WAL files for successfully checkpointed hosts.
        RotateWALFiles(hostDirs)  // <-- Line 138: BLOCKS on rotation completion
    }
}
```

**Problem**:
- `RotateWALFiles()` is a blocking call (waits on all done channels)
- But this is called from the periodic Checkpointing goroutine
- If rotation requests get sent but the WAL staging goroutine is slow to process, this can block the checkpoint loop
- During the checkpointing interval, if multiple hosts are being rotated, the checkpoint goroutine can block indefinitely waiting for rotation

Actually, this might not deadlock under normal circumstances, but combined with BUG #4, it creates the deadlock.

---

### BUG #6: WAL File State Not Locked During Concurrent Access

**Location**: `walCheckpoint.go` lines 191-226 (getOrOpenWAL) and line 237 (processMsg)

```go
getOrOpenWAL := func(hostDir string) *walFileState {
    ws, ok := hostFiles[hostDir]  // <-- Read map
    if ok {
        return ws
    }
    // ... create and store in hostFiles
    hostFiles[hostDir] = ws
    return ws
}

processMsg := func(msg *WALMessage) {
    hostDir := path.Join(Keys.Checkpoints.RootDir, msg.Cluster, msg.Node)
    ws := getOrOpenWAL(hostDir)  // <-- Get walFileState
    if ws == nil {
        return
    }
    if err := writeWALRecordDirect(ws.w, msg); err != nil {  // <-- Write to it
        cclog.Errorf("[METRICSTORE]> WAL: write record: %v", err)
    }
    ws.dirty = true
}
```

**Problem**:
- `hostFiles` is a plain `map[string]*walFileState` with NO locking
- It's only accessed within the single WAL staging goroutine, so normally safe
- HOWEVER, during concurrent access from multiple shards' processMsg functions... wait, each shard has its own WAL staging goroutine, so this shouldn't be an issue
- Actually, each shard owns a disjoint set of hosts based on the hash, so the map isn't shared
- **But**: There's still a subtle issue: a *walFileState pointer is stored in hostFiles, and both the file handle `f` and the writer `w` are accessed. If `processRotate` closes the file while `processMsg` is writing, you get:
  - `processMsg` calls `ws.w.Write()` while file is closed
  - Or `ws.dirty = true` while file is being deleted
  - Not a panic necessarily, but undefined behavior

---

### BUG #7: Incomplete WAL Record Handling on Write Failure

**Location**: `walCheckpoint.go` lines 363-433 (writeWALRecordDirect)

```go
if _, err := w.Write(scratch[:4]); err != nil {
    return err  // <-- Early return on error
}
// ... more writes
if _, err := w.Write(scratch[:4]); err != nil {
    return err  // <-- Early return
}
```

**Problem**:
- If ANY write in the middle of encoding a record fails (e.g., disk full), the bufio.Writer is in a partially-written state
- The function returns an error, but the buffered writer still has partial data
- On the next call to `w.Write()`, more data is appended
- Result: **Corrupted record in the WAL file** with random bytes interleaved
- The CRC check during replay will catch this, but the record is lost

**Better approach**: Use an intermediate buffer or transaction-like mechanism to ensure atomicity.

---

### BUG #8: No Cleanup of Buffered Messages After Context Cancellation

**Location**: `walCheckpoint.go` line 288-293 (on ctx.Done)

```go
case <-ctx.Done():
    // On shutdown, skip draining buffered messages — a full binary
    // checkpoint will be written from in-memory state, making
    // buffered WAL records redundant.
    flushDirty()
    return
```

**Problem**:
- The comment says buffered messages are "redundant" because binary checkpoint captures them
- **But this is false** if a metric arrives AFTER the checkpoint starts
- Timeline:
  1. Checkpointing goroutine starts writing binary snapshot at time T1
  2. Metric arrives at T1 + epsilon → sent to WAL channel
  3. Context cancelled → WAL staging goroutine exits without processing message
  4. Binary snapshot was written at T1, doesn't include metric from T1 + epsilon
  5. Data loss

- The assumption is that once context is cancelled, no more metrics arrive. **But metrics might be in flight** in the NATS decoder or channel buffer

**Root cause**: No grace period to allow in-flight metrics to reach the WAL channel before draining/exiting.

---

### BUG #9: walNumShards and walShardChs Race on Initialization

**Location**: `walCheckpoint.go` lines 87-96

```go
var walShardChs []chan *WALMessage
var walShardRotateChs []chan walRotateReq
var walNumShards int
var walStagingWg sync.WaitGroup
```

**Location**: `walCheckpoint.go` lines 162-171 (WALStaging initialization)

```go
walNumShards = max(Keys.NumWorkers, 1)  // <-- Write to global
chBufSize := max(65536/walNumShards, 1024)

walShardChs = make([]chan *WALMessage, walNumShards)  // <-- Write to global
walShardRotateChs = make([]chan walRotateReq, walNumShards)  // <-- Write to global
```

**Location**: `walCheckpoint.go` line 130-136 (walShardIndex called by DecodeLine)

```go
func walShardIndex(cluster, node string) int {
    h := fnv.New32a()
    h.Write([]byte(cluster))
    h.Write([]byte{0})
    h.Write([]byte(node))
    return int(h.Sum32()) % walNumShards  // <-- Read from global
}
```

**Problem**:
- `walNumShards` is read by `walShardIndex()` during metric ingestion
- But it's written by `WALStaging()` during initialization
- **Race condition**: If metrics arrive (from NATS) while WALStaging is still initializing, walShardIndex reads walNumShards before it's set (could be 0)
- Panic on line 135: `int(h.Sum32()) % 0` → **division by zero**

**Timeline**:
1. Init() calls WALStaging(wg, ctx) at line 175 (metricstore.go)
2. Init() calls ReceiveNats() at line 188 (might subscribe immediately)
3. NATS message arrives before WALStaging sets walNumShards (line 162)
4. DecodeLine calls SendWALMessage → walShardIndex → modulo 0 → panic

**Fix**: Initialize walNumShards and channels BEFORE any metrics can arrive.

---

## ARCHITECTURAL ISSUES

### ISSUE #1: No Backpressure Mechanism (Performance)

**Location**: `lineprotocol.go` lines 364-370

```go
if !SendWALMessage(msg) {
    if dropped := walDropped.Add(1); dropped%10000 == 1 {
        cclog.Warnf("[METRICSTORE]> WAL shard channel full, dropped %d messages", dropped)
    }
}
```

**Problem**:
- When the WAL channel is full (bufSize = 65536/walNumShards ≈ 6-65k depending on workers), messages are **silently dropped**
- This violates durability: the metric is written to memory but **NOT to WAL**
- Next binary snapshot captures it, but in the interim, durability is compromised
- If the process crashes before the next snapshot, the metric is lost
- No way to know if a critical metric was dropped (only logged every 10k drops)

**Better approach**:
- Either block on full (backpressure)
- Or use an unbounded queue with memory limits
- Or have a proper overflow file/mechanism

---

### ISSUE #2: No Atomic Transaction Between Binary Snapshot and WAL Rotation

**Location**: `checkpoint.go` lines 128-139 vs `walCheckpoint.go` lines 625-727

```go
// In Checkpointing():
n, hostDirs, err := ms.ToCheckpointWAL(...)
if err == nil {
    RotateWALFiles(hostDirs)  // <-- Separate calls!
}
```

```go
// In ToCheckpointWAL():
// ... write binary snapshots ...
// Success files returned in successDirs
// But!
for wi := range work {
    err := wi.level.toCheckpointBinary(...)
    if err == nil {
        atomic.AddInt32(&n, 1)
        walPath := path.Join(wi.hostDir, "current.wal")
        if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
            cclog.Errorf("[METRICSTORE]> WAL remove %s: %v", walPath, err)
        }
        successMu.Lock()
        successDirs = append(successDirs, wi.hostDir)
        successMu.Unlock()
    } else if err == ErrNoNewArchiveData {
        atomic.AddInt32(&completed, 1)
        continue
    } else {
        cclog.Errorf("[METRICSTORE]> binary checkpoint error for %s: %v", wi.hostDir, err)
        atomic.AddInt32(&errs, 1)
    }
}
```

**Problem**:
- Worker goroutines delete current.wal IMMEDIATELY after snapshot is written (line 698-701)
- But if a metric arrives BETWEEN snapshot write and WAL delete, it goes to WAL
- Then RotateWALFiles() deletes the file (or tries to)
- More importantly: **No atomicity guarantee**. A snapshot and its corresponding WAL should be either both present or both absent.

---

### ISSUE #3: Memory Overhead of Pool-Allocated WAL Records

**Location**: `lineprotocol.go` lines 355-371 (DecodeLine sends WAL messages)

```go
msg := &WALMessage{
    MetricName: string(st.metricBuf),
    Cluster:    cluster,
    Node:       host,
    Selector:   append([]string{}, st.selector...),  // <-- Allocates new slice
    Value:      metric.Value,
    Timestamp:  time.Unix(),
}
if !SendWALMessage(msg) {
    // ...
}
```

**Problem**:
- Every metric creates a **new WALMessage struct allocation** + **new selector slice copy** (line 360)
- For high-throughput (100k metrics/sec), this is significant GC pressure
- The WALMessage is not pooled or reused
- The selector slice is always copied (can't reuse from decodeState because it's freed at scope end)

**Better approach**:
- Use a WALMessage pool with pre-allocated selector arrays
- Or encode directly to the file without intermediate allocation

---

### ISSUE #4: No Coordination Between WAL Rotation and Active Writes

**Location**: `walCheckpoint.go` lines 240-252 (processRotate)

```go
processRotate := func(req walRotateReq) {
    ws, ok := hostFiles[req.hostDir]
    if ok && ws.f != nil {
        ws.w.Flush()
        ws.f.Close()
        walPath := path.Join(req.hostDir, "current.wal")
        if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
            cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
        }
        delete(hostFiles, req.hostDir)  // <-- Remove from map
    }
    close(req.done)
}
```

**Problem**:
- Between receiving the rotation request and closing the file, a concurrent processMsg might:
  1. Call getOrOpenWAL() and get the old ws pointer (line 241 checks if ok, but there's a race between the check and the use in processMsg)
  2. Call ws.w.Flush() while processRotate is also flushing
  3. Call ws.w.Write() after the file is closed (undefined behavior)

- Actually, each shard's messages are serialized by the single goroutine, so there's no race within a shard. But the issue is that rotation is interleaved with message processing in the same select loop.

- **Real issue**: Between rotation completion and the next message, the hostFiles map no longer has the entry. The next message for that host will create a NEW WAL file (getOrOpenWAL creates a fresh one). But there's a window where:
  1. Rotation deletes current.wal
  2. New metric arrives before file is fully closed
  3. New WAL file created, metric written
  4. Result: **File is not cleanly rotated** (should be atomic: rename old to timestamp, create new)

---

## DETAILED ANALYSIS BY COMPONENT

### WALStaging Goroutine (walCheckpoint.go lines 157-311)

**Good**:
- Per-shard serialization prevents locks
- Batch drain (4096 messages) amortizes syscalls
- Periodic flush (5 seconds) balances latency and syscall overhead
- Double-linked goroutine lifecycle (Add before spawn, Done in defer)

**Bad**:
- Draining logic is inefficient (BUG #3)
- Rotation requests can deadlock (BUG #4)
- No flow control on shutdown (BUG #1, #2, #8)

---

### SendWALMessage (walCheckpoint.go lines 138-151)

**Good**:
- Simple non-blocking send
- Fast path if WAL disabled

**Bad**:
- Panic risk on closed channel (BUG #2)
- No backpressure (ISSUE #1)

---

### WAL Record Format (walCheckpoint.go lines 363-433)

**Format** (lines 34-43):
```
[4B magic 0xCC1DA7A1][4B payload_len][payload][4B CRC32]
payload:
  [8B timestamp][2B name_len][name][1B selector_count]
  [selectors][4B value float32]
```

**Good**:
- CRC32 detects corruption
- Length prefix allows skipping bad records
- Magic numbers prevent misalignment

**Bad**:
- Partial writes corrupt the record (BUG #7)
- No transaction log / write-ahead semantics (ISSUE #2)

---

### Shutdown Sequence (metricstore.go lines 260-314)

**Current**:
1. Cancel context (line 279)
2. Close walShardChs (lines 284-286)
3. Wait for WAL staging to drain (line 288)
4. Write final checkpoint (line 305 or 307)

**Problems**:
- (BUG #1, #2) In-flight metrics cause panic on send to closed channel
- (BUG #4) Rotation requests may get stuck
- No grace period for in-flight metrics

**Correct sequence should be**:
1. Stop accepting NEW metrics (signal to NATS decoder)
2. Wait for in-flight metrics to reach WAL channel (give grace period)
3. Close WAL channels
4. Wait for WAL staging goroutine to exit
5. Write final checkpoint
6. Done

---

### Checkpointing Goroutine (checkpoint.go lines 87-154)

**Flow** (WAL format):
1. Get lastCheckpoint timestamp
2. Call ToCheckpointWAL(from=lastCheckpoint, to=now)
3. ToCheckpointWAL:
   a. Collect all (cluster, host) pairs
   b. Spawn workers to write .bin files in parallel
   c. Delete current.wal files immediately after success
   d. Return list of successfully checkpointed hosts
4. Call RotateWALFiles(hostDirs) — but hostDirs only includes hosts where .bin was written successfully
5. Update lastCheckpoint timestamp

**Issue**: (ISSUE #2) There's no atomicity guarantee. If a metric arrives between snapshot write and WAL rotation, it's lost during a crash.

---

### WAL Loading (checkpoint.go lines 517-601, walCheckpoint.go lines 539-607)

**Flow**:
1. Find all .json and .bin files
2. Load snapshots in timestamp order
3. Load current.wal if present, replay all records
4. Skip records older than retention window

**Good**:
- Snapshot + WAL replication handles crashes safely
- CRC check detects corruption
- Silent skip on truncated trailing record (expected after crash)

**Bad**:
- No cache for level lookups (though there's a lvlCache per load)
- Concurrent level creation during load could race (but locked during load)

---

## MISSING SYNCHRONIZATION

### Mutual Exclusion Between Ingestion and Shutdown

```go
// MISSING: Once shutdown starts, don't accept new metrics
var shutdownStarted atomic.Bool

// In DecodeLine (lineprotocol.go):
if shutdownStarted.Load() {
    return // or queue to local buffer
}

// In Shutdown (metricstore.go):
shutdownStarted.Store(true)
time.Sleep(100 * time.Millisecond) // grace period
// Close channels
```

---

### Ordered Shutdown Phases

Current code doesn't enforce:
1. Stop ingestion
2. Drain in-flight metrics
3. Close channels
4. Exit goroutines
5. Write final checkpoint

---

## IMPROVEMENT OPPORTUNITIES

### 1. Use Unbounded Queue with Memory Limits (Instead of Dropping)

Replace non-blocking send with bounded queue that grows as needed, respects memory cap.

### 2. Implement Atomic Snapshots + WAL Rotation

Use a transaction log: after .bin file is fsync'd, atomically truncate (or rename) current.wal.

### 3. WAL Record Batch Encoding

Instead of allocating per-record, batch multiple records into a single write transaction.

### 4. Pre-allocate WALMessage Structs

Use a sync.Pool for WALMessage and selector slices.

### 5. Implement Backpressure

Block producers when WAL queue is full, rather than dropping.

### 6. Separate Read/Write Concerns in WAL Staging

Have separate goroutines for reading messages and writing files.

### 7. Add Metrics/Observability

- Track drop rate, latency, queue depth
- Alert on drops
- Measure WAL write latency

---

## SUMMARY TABLE

| Bug/Issue | Severity | Category | Impact |
|-----------|----------|----------|--------|
| BUG #1: Send after close | CRITICAL | Race | Panic on shutdown |
| BUG #2: Channel close panic | CRITICAL | Race | Data loss or panic |
| BUG #3: Drain loop inefficiency | MEDIUM | Logic | Latency, resource waste |
| BUG #4: Lost rotation requests | CRITICAL | Race | Shutdown hang |
| BUG #5: Deadlock on checkpoint | MEDIUM | Race | Checkpoint hang |
| BUG #6: WAL state concurrency | LOW | Race | Rare corruption |
| BUG #7: Incomplete record write | MEDIUM | Data integrity | WAL corruption on disk full |
| BUG #8: Lost metrics after shutdown | CRITICAL | Design | Data loss |
| BUG #9: Division by zero | CRITICAL | Race | Panic on startup |
| ISSUE #1: No backpressure | HIGH | Design | Durability violation |
| ISSUE #2: Non-atomic snapshot/WAL | HIGH | Design | Data loss in crashes |
| ISSUE #3: Memory overhead | MEDIUM | Performance | GC pressure |
| ISSUE #4: No rotation coordination | MEDIUM | Design | Subtle corruption |

---

## RECOMMENDED FIXES (Priority Order)

1. **IMMEDIATE**: Fix BUG #9 (initialize walNumShards early)
2. **IMMEDIATE**: Fix BUG #1 & #2 (safe channel close)
3. **HIGH**: Fix BUG #8 (grace period for in-flight metrics)
4. **HIGH**: Fix BUG #4 (close rotation channels or wait for requests)
5. **MEDIUM**: Fix BUG #7 (atomic record writes)
6. **MEDIUM**: Implement ISSUE #1 (backpressure)
7. **MEDIUM**: Fix ISSUE #2 (atomic snapshot + WAL)

---

## TESTING GAPS

- No test for shutdown with in-flight metrics
- No test for rotation during heavy load
- No test for concurrent snapshot + new metrics
- No test for disk-full scenario (BUG #7)
- No test for rapid checkpoint intervals


<tool_use_error>Error: No such tool available: Write</tool_use_error>


Bash

cat > /tmp/wal-review.md << 'EOF'
# Comprehensive WAL Buffer Pipeline Review

## Executive Summary

The sharded WAL buffer pipeline has **9 critical bugs**, **4 architectural flaws**, and multiple improvement opportunities. Most critically:

1. **Panic on shutdown** due to send to closed channel
2. **Data loss** from metrics dropped without durability
3. **Shutdown hangs** from blocked rotation requests
4. **Division by zero** from uninitialized walNumShards
5. **Silent data loss** when checkpoint and WAL become desynchronized

---

## CRITICAL BUGS (MUST FIX)

### BUG #1: Panic on Shutdown - Send After Close [CRITICAL]

**File**: `metricstore.go:283-289` (Shutdown function)

**Code**:
```go
if Keys.Checkpoints.FileFormat == "wal" {
    for _, ch := range walShardChs {
        close(ch)           // Close channels
    }
    WaitForWALStagingDrain()  // Wait for exit
}
```

**Problem**:
Race condition where in-flight metrics from NATS decoder try to send to closed channel after Shutdown() closes them but before goroutines exit.

**Timeline**:
1. Shutdown() calls ctx.Cancel()
2. ReceiveNats() sees ctx.Done and exits its loop
3. BUT: In-flight NATS metrics are still being decoded by worker goroutines
4. Those workers call DecodeLine → SendWALMessage on now-closed channels
5. **PANIC**: "send on closed channel"

**Root cause**: No coordination between shutdown and active metric ingestion.

---

### BUG #2: Division by Zero on Startup [CRITICAL]

**File**: `walCheckpoint.go:130-136, 162-171`

**Code**:
```go
// Global variable
var walNumShards int  // Initialized to 0

// Used immediately:
func walShardIndex(cluster, node string) int {
    return int(h.Sum32()) % walNumShards  // LINE 135: % 0 = PANIC
}

// Set later in WALStaging():
walNumShards = max(Keys.NumWorkers, 1)  // LINE 162
```

**Problem**:
If metrics arrive (from NATS) before WALStaging() initializes walNumShards, walShardIndex divides by zero.

**Timeline**:
1. Init() calls WALStaging() (metricstore.go:175)
2. Init() calls ReceiveNats() (metricstore.go:188) - subscribes to NATS
3. NATS message arrives, DecodeLine calls SendWALMessage
4. SendWALMessage calls walShardIndex with walNumShards == 0
5. **PANIC**: modulo by zero

**Fix**: Initialize walNumShards and channels BEFORE subscribing to NATS.

---

### BUG #3: Rotation Requests Deadlock During Shutdown [CRITICAL]

**File**: `walCheckpoint.go:240-252, 323-338`

**Code**:
```go
// In processRotate (line 240-252):
close(req.done)  // Signal completion

// In RotateWALFiles (line 323-338):
for _, done := range dones {
    <-done  // BLOCKS waiting for rotation completion
}
```

**Problem**:
walShardRotateChs is never closed. If WAL staging goroutine exits while rotation requests are pending, they block forever.

**Timeline**:
1. Shutdown() closes walShardChs[i] (line 285)
2. WAL staging goroutine sees msgCh closed, exits (line 295-296)
3. Concurrent checkpoint calls RotateWALFiles(hostDirs) (checkpoint.go:138)
4. Sends rotation request to now-abandoned rotateCh
5. RotateWALFiles blocks forever waiting for done channel (line 336)
6. **DEADLOCK**: Checkpoint goroutine hangs, Shutdown never completes

---

### BUG #4: Silent Data Loss on Backpressure [CRITICAL]

**File**: `lineprotocol.go:364-370`

**Code**:
```go
if !SendWALMessage(msg) {
    // Message dropped silently!
    if dropped := walDropped.Add(1); dropped%10000 == 1 {
        cclog.Warnf("[METRICSTORE]> WAL shard channel full, dropped %d messages")
    }
}
```

**Problem**:
When WAL shard channel is full, the metric is written to memory store but NOT persisted to WAL. If the process crashes before the next binary snapshot, the metric is lost, violating durability guarantees.

**Evidence**:
- Channel size: 65536/walNumShards ≈ 6K-65K depending on worker count
- Under high throughput (>100k msg/sec), channels fill up
- Drops are silently logged every 10k drops (may go unnoticed)
- Binary snapshot only captures data up to last snapshot time, not in-flight metrics

---

### BUG #5: Lost Metrics at Shutdown [CRITICAL]

**File**: `walCheckpoint.go:286-308, metricstore.go:273-314`

**Code** (WAL staging exit path):
```go
case <-ctx.Done():
    // On shutdown, skip draining buffered messages — a full binary
    // checkpoint will be written from in-memory state, making
    // buffered WAL records redundant.
    flushDirty()
    return  // Exit WITHOUT processing remaining messages
```

**Problem**:
Assumes all in-flight metrics have already reached the channel. But metrics in the NATS decoder or msgCh buffer are abandoned on shutdown.

**Timeline**:
1. Checkpointing goroutine finishes binary snapshot at T1
2. Metric arrives at T1 + 1ms, sent to WAL channel
3. Shutdown() cancels context
4. WAL staging goroutine reads ctx.Done, exits without draining channel buffer
5. Binary snapshot written at T1, doesn't include metric from T1+1ms
6. **DATA LOSS**: Metric written to memory but not persisted

**Root cause**: No grace period to allow in-flight metrics to settle before shutdown.

---

### BUG #6: Partial WAL Record on Disk Full [MEDIUM]

**File**: `walCheckpoint.go:363-433` (writeWALRecordDirect)

**Code**:
```go
if _, err := w.Write(scratch[:8]); err != nil {
    return err  // Early return with partial data in buffer
}
// ... continue writing more bytes ...
if _, err := w.Write(scratch[:4]); err != nil {
    return err  // Another early return
}
```

**Problem**:
If writeWALRecordDirect encounters an error mid-write (e.g., disk full), the bufio.Writer contains partial record data. Next write continues appending, creating a corrupted record.

**Example**:
1. Write magic + length (8 bytes) - OK
2. Write timestamp (8 bytes) - OK
3. Write name length (2 bytes) - OK
4. Write name (N bytes) - DISK FULL, error returned
5. bufio.Writer still has the 10 bytes written so far
6. Function returns error
7. Next writeWALRecordDirect call continues appending - corrupted record!

**Fix**: Use intermediate buffer to ensure atomic writes.

---

### BUG #7: Inefficient Drain Loop [MEDIUM]

**File**: `walCheckpoint.go:268-284` (drainBatch function)

**Code**:
```go
drainBatch := func() bool {
    for range 4096 {
        select {
        case msg, ok := <-msgCh:
            if !ok {
                return false
            }
            processMsg(msg)
        case req := <-rotateCh:
            processRotate(req)
        default:
            return true  // Returns on first empty select
        }
    }
    return true
}
```

**Problem**:
The outer loop (line 286-308) calls drainBatch() after processing ONE message. If drainBatch hits default (channels empty), it returns immediately. But there could be more messages arriving. Then outer loop blocks on the now-empty channel. This causes unnecessary context switches and latency.

---

### BUG #8: Non-Atomic Snapshot + WAL Rotation [HIGH]

**File**: `checkpoint.go:128-139, walCheckpoint.go:625-727`

**Code**:
```go
// In Checkpointing:
n, hostDirs, err := ms.ToCheckpointWAL(...)
RotateWALFiles(hostDirs)  // Separate operation!

// Inside ToCheckpointWAL worker:
os.Remove(walPath)  // Delete immediately after snapshot success
```

**Problem**:
Binary snapshot and WAL truncation are not atomic. A metric arriving between them is lost on crash:

1. Snapshot writes up to T0
2. Snapshot deletes current.wal
3. Metric arrives at T0+1, written to NEW current.wal file
4. Process crashes
5. On restart: Load snapshot from T0, load empty current.wal
6. **DATA LOSS**: Metric at T0+1 is gone

---

### BUG #9: Race on walShardChs/walNumShards Initialization [MEDIUM]

**File**: `walCheckpoint.go:87-96, 162-171, 145-150`

**Code**:
```go
var walShardChs []chan *WALMessage  // Global, initialized to nil

// In WALStaging (called during Init):
walShardChs = make([]chan *WALMessage, walNumShards)  // Write

// In SendWALMessage (called during metric ingestion):
if walShardChs == nil {
    return false
}
shard := walShardIndex(...)
select {
    case walShardChs[shard] <- msg:  // Read walShardChs
}
```

**Problem**:
Between when Init() starts WALStaging and when it fully initializes walShardChs, metrics arriving from NATS can see partial initialization (nil channels or incomplete array).

---

## ARCHITECTURAL ISSUES

### ISSUE A: No Backpressure Mechanism

**Impact**: Durability violation

When WAL channel is full, metrics are dropped silently. This breaks the durability guarantee. Users have no way to know if critical metrics were lost.

**Better approach**:
- Block on full (apply backpressure to NATS decoder)
- Use unbounded queue with memory limits
- Or have separate overflow log mechanism

### ISSUE B: Checkpoint-WAL Desynchronization

**Impact**: Silent data loss in crashes

No atomicity guarantee between binary snapshot and WAL rotation. Metrics can be lost if they arrive during the gap.

**Better approach**:
- Use transaction log: write marker to snapshot indicating "up to this timestamp", then atomically truncate WAL
- Or: track snapshot timestamps in metadata, replay only newer WAL entries

### ISSUE C: Memory Overhead in WAL Message Path

**Impact**: GC pressure at high throughput

Every metric allocates new WALMessage struct + copies selector slice:
```go
msg := &WALMessage{
    ...
    Selector:   append([]string{}, st.selector...),  // NEW allocation
}
```

At 100k msg/sec, this is significant GC work.

### ISSUE D: No Coordination Between Rotation and Active Writes

**Impact**: Rare file corruption

Between rotation request completion and next message, the file handle is closed but map entry deleted. Next message creates a new file, bypassing the rotation.

---

## SHUTDOWN SEQUENCE ANALYSIS

**Current** (metricstore.go:273-314):
```
1. Cancel context
2. Close walShardChs
3. Wait for WAL staging drain
4. Write final checkpoint
```

**Problems**:
- (BUG #1) In-flight metrics panic on closed channel
- (BUG #3) Rotation requests deadlock
- (BUG #5) Buffered messages lost

**Correct sequence**:
```
1. SIGNAL: Stop accepting new metrics (don't close channels yet)
2. WAIT: Grace period for in-flight metrics (e.g., 100ms)
3. FLUSH: Close WAL channels to signal EOF
4. DRAIN: Wait for WAL staging goroutines to exit
5. ROTATE: Call RotateWALFiles for completed snapshots
6. CHECKPOINT: Write final checkpoint
7. DONE
```

---

## SEVERITY MATRIX

| ID | Bug | Severity | Category | Impact |
|----|-----|----------|----------|--------|
| 1 | Send after close | CRITICAL | Race | Panic on shutdown |
| 2 | Div by zero | CRITICAL | Race | Panic on startup with metrics |
| 3 | Rotation deadlock | CRITICAL | Deadlock | Shutdown hangs |
| 4 | Silent drops | CRITICAL | Data integrity | Durability violation |
| 5 | Lost metrics | CRITICAL | Data integrity | Crash data loss |
| 6 | Partial record | MEDIUM | Data integrity | Corruption on disk full |
| 7 | Inefficient drain | MEDIUM | Performance | Latency/CPU waste |
| 8 | Non-atomic snap+wal | HIGH | Data integrity | Crash data loss |
| 9 | Init race | MEDIUM | Race | Possible panic on startup |
| A | No backpressure | HIGH | Design | Durability risk |
| B | Snap-WAL sync | HIGH | Design | Crash data loss |
| C | Memory overhead | MEDIUM | Performance | GC pressure |
| D | Rotation sync | LOW | Race | Rare corruption |

---

## RECOMMENDED FIXES

### Immediate (causes crashes):
1. Fix BUG #2: Initialize walNumShards before NATS subscription
2. Fix BUG #1: Add safe shutdown signaling before closing channels
3. Fix BUG #3: Close rotation channels or guarantee goroutine liveness

### High Priority (data loss):
4. Fix BUG #5: Add grace period for in-flight metrics
5. Fix BUG #4: Implement backpressure instead of drop
6. Fix BUG #8: Make snapshot + WAL rotation atomic

### Medium Priority:
7. Fix BUG #6: Atomic record writes with intermediate buffer
8. Fix BUG #7: Better drain loop logic
9. Add comprehensive shutdown tests

---

## CODE LOCATIONS

- WAL write pipeline: `pkg/metricstore/walCheckpoint.go`
- Metric producer: `pkg/metricstore/lineprotocol.go`
- Lifecycle: `pkg/metricstore/metricstore.go`
- Checkpointing: `pkg/metricstore/checkpoint.go`

EOF
cat /tmp/wal-review.md


# Comprehensive WAL Buffer Pipeline Review

## Executive Summary

The sharded WAL buffer pipeline has **9 critical bugs**, **4 architectural flaws**, and multiple improvement opportunities. Most critically:

1. **Panic on shutdown** due to send to closed channel
2. **Data loss** from metrics dropped without durability
3. **Shutdown hangs** from blocked rotation requests
4. **Division by zero** from uninitialized walNumShards
5. **Silent data loss** when checkpoint and WAL become desynchronized

---

## CRITICAL BUGS (MUST FIX)

### BUG #1: Panic on Shutdown - Send After Close [CRITICAL]

**File**: `metricstore.go:283-289` (Shutdown function)

**Code**:
```go
if Keys.Checkpoints.FileFormat == "wal" {
    for _, ch := range walShardChs {
        close(ch)           // Close channels
    }
    WaitForWALStagingDrain()  // Wait for exit
}
```

**Problem**:
Race condition where in-flight metrics from NATS decoder try to send to closed channel after Shutdown() closes them but before goroutines exit.

**Timeline**:
1. Shutdown() calls ctx.Cancel()
2. ReceiveNats() sees ctx.Done and exits its loop
3. BUT: In-flight NATS metrics are still being decoded by worker goroutines
4. Those workers call DecodeLine → SendWALMessage on now-closed channels
5. **PANIC**: "send on closed channel"

**Root cause**: No coordination between shutdown and active metric ingestion.

---

### BUG #2: Division by Zero on Startup [CRITICAL]

**File**: `walCheckpoint.go:130-136, 162-171`

**Code**:
```go
// Global variable
var walNumShards int  // Initialized to 0

// Used immediately:
func walShardIndex(cluster, node string) int {
    return int(h.Sum32()) % walNumShards  // LINE 135: % 0 = PANIC
}

// Set later in WALStaging():
walNumShards = max(Keys.NumWorkers, 1)  // LINE 162
```

**Problem**:
If metrics arrive (from NATS) before WALStaging() initializes walNumShards, walShardIndex divides by zero.

**Timeline**:
1. Init() calls WALStaging() (metricstore.go:175)
2. Init() calls ReceiveNats() (metricstore.go:188) - subscribes to NATS
3. NATS message arrives, DecodeLine calls SendWALMessage
4. SendWALMessage calls walShardIndex with walNumShards == 0
5. **PANIC**: modulo by zero

**Fix**: Initialize walNumShards and channels BEFORE subscribing to NATS.

---

### BUG #3: Rotation Requests Deadlock During Shutdown [CRITICAL]

**File**: `walCheckpoint.go:240-252, 323-338`

**Code**:
```go
// In processRotate (line 240-252):
close(req.done)  // Signal completion

// In RotateWALFiles (line 323-338):
for _, done := range dones {
    <-done  // BLOCKS waiting for rotation completion
}
```

**Problem**:
walShardRotateChs is never closed. If WAL staging goroutine exits while rotation requests are pending, they block forever.

**Timeline**:
1. Shutdown() closes walShardChs[i] (line 285)
2. WAL staging goroutine sees msgCh closed, exits (line 295-296)
3. Concurrent checkpoint calls RotateWALFiles(hostDirs) (checkpoint.go:138)
4. Sends rotation request to now-abandoned rotateCh
5. RotateWALFiles blocks forever waiting for done channel (line 336)
6. **DEADLOCK**: Checkpoint goroutine hangs, Shutdown never completes

---

### BUG #4: Silent Data Loss on Backpressure [CRITICAL]

**File**: `lineprotocol.go:364-370`

**Code**:
```go
if !SendWALMessage(msg) {
    // Message dropped silently!
    if dropped := walDropped.Add(1); dropped%10000 == 1 {
        cclog.Warnf("[METRICSTORE]> WAL shard channel full, dropped %d messages")
    }
}
```

**Problem**:
When WAL shard channel is full, the metric is written to memory store but NOT persisted to WAL. If the process crashes before the next binary snapshot, the metric is lost, violating durability guarantees.

**Evidence**:
- Channel size: 65536/walNumShards ≈ 6K-65K depending on worker count
- Under high throughput (>100k msg/sec), channels fill up
- Drops are silently logged every 10k drops (may go unnoticed)
- Binary snapshot only captures data up to last snapshot time, not in-flight metrics

---

### BUG #5: Lost Metrics at Shutdown [CRITICAL]

**File**: `walCheckpoint.go:286-308, metricstore.go:273-314`

**Code** (WAL staging exit path):
```go
case <-ctx.Done():
    // On shutdown, skip draining buffered messages — a full binary
    // checkpoint will be written from in-memory state, making
    // buffered WAL records redundant.
    flushDirty()
    return  // Exit WITHOUT processing remaining messages
```

**Problem**:
Assumes all in-flight metrics have already reached the channel. But metrics in the NATS decoder or msgCh buffer are abandoned on shutdown.

**Timeline**:
1. Checkpointing goroutine finishes binary snapshot at T1
2. Metric arrives at T1 + 1ms, sent to WAL channel
3. Shutdown() cancels context
4. WAL staging goroutine reads ctx.Done, exits without draining channel buffer
5. Binary snapshot written at T1, doesn't include metric from T1+1ms
6. **DATA LOSS**: Metric written to memory but not persisted

**Root cause**: No grace period to allow in-flight metrics to settle before shutdown.

---

### BUG #6: Partial WAL Record on Disk Full [MEDIUM]

**File**: `walCheckpoint.go:363-433` (writeWALRecordDirect)

**Code**:
```go
if _, err := w.Write(scratch[:8]); err != nil {
    return err  // Early return with partial data in buffer
}
// ... continue writing more bytes ...
if _, err := w.Write(scratch[:4]); err != nil {
    return err  // Another early return
}
```

**Problem**:
If writeWALRecordDirect encounters an error mid-write (e.g., disk full), the bufio.Writer contains partial record data. Next write continues appending, creating a corrupted record.

**Example**:
1. Write magic + length (8 bytes) - OK
2. Write timestamp (8 bytes) - OK
3. Write name length (2 bytes) - OK
4. Write name (N bytes) - DISK FULL, error returned
5. bufio.Writer still has the 10 bytes written so far
6. Function returns error
7. Next writeWALRecordDirect call continues appending - corrupted record!

**Fix**: Use intermediate buffer to ensure atomic writes.

---

### BUG #7: Inefficient Drain Loop [MEDIUM]

**File**: `walCheckpoint.go:268-284` (drainBatch function)

**Code**:
```go
drainBatch := func() bool {
    for range 4096 {
        select {
        case msg, ok := <-msgCh:
            if !ok {
                return false
            }
            processMsg(msg)
        case req := <-rotateCh:
            processRotate(req)
        default:
            return true  // Returns on first empty select
        }
    }
    return true
}
```

**Problem**:
The outer loop (line 286-308) calls drainBatch() after processing ONE message. If drainBatch hits default (channels empty), it returns immediately. But there could be more messages arriving. Then outer loop blocks on the now-empty channel. This causes unnecessary context switches and latency.

---

### BUG #8: Non-Atomic Snapshot + WAL Rotation [HIGH]

**File**: `checkpoint.go:128-139, walCheckpoint.go:625-727`

**Code**:
```go
// In Checkpointing:
n, hostDirs, err := ms.ToCheckpointWAL(...)
RotateWALFiles(hostDirs)  // Separate operation!

// Inside ToCheckpointWAL worker:
os.Remove(walPath)  // Delete immediately after snapshot success
```

**Problem**:
Binary snapshot and WAL truncation are not atomic. A metric arriving between them is lost on crash:

1. Snapshot writes up to T0
2. Snapshot deletes current.wal
3. Metric arrives at T0+1, written to NEW current.wal file
4. Process crashes
5. On restart: Load snapshot from T0, load empty current.wal
6. **DATA LOSS**: Metric at T0+1 is gone

---

### BUG #9: Race on walShardChs/walNumShards Initialization [MEDIUM]

**File**: `walCheckpoint.go:87-96, 162-171, 145-150`

**Code**:
```go
var walShardChs []chan *WALMessage  // Global, initialized to nil

// In WALStaging (called during Init):
walShardChs = make([]chan *WALMessage, walNumShards)  // Write

// In SendWALMessage (called during metric ingestion):
if walShardChs == nil {
    return false
}
shard := walShardIndex(...)
select {
    case walShardChs[shard] <- msg:  // Read walShardChs
}
```

**Problem**:
Between when Init() starts WALStaging and when it fully initializes walShardChs, metrics arriving from NATS can see partial initialization (nil channels or incomplete array).

---

## ARCHITECTURAL ISSUES

### ISSUE A: No Backpressure Mechanism

**Impact**: Durability violation

When WAL channel is full, metrics are dropped silently. This breaks the durability guarantee. Users have no way to know if critical metrics were lost.

**Better approach**:
- Block on full (apply backpressure to NATS decoder)
- Use unbounded queue with memory limits
- Or have separate overflow log mechanism

### ISSUE B: Checkpoint-WAL Desynchronization

**Impact**: Silent data loss in crashes

No atomicity guarantee between binary snapshot and WAL rotation. Metrics can be lost if they arrive during the gap.

**Better approach**:
- Use transaction log: write marker to snapshot indicating "up to this timestamp", then atomically truncate WAL
- Or: track snapshot timestamps in metadata, replay only newer WAL entries

### ISSUE C: Memory Overhead in WAL Message Path

**Impact**: GC pressure at high throughput

Every metric allocates new WALMessage struct + copies selector slice:
```go
msg := &WALMessage{
    ...
    Selector:   append([]string{}, st.selector...),  // NEW allocation
}
```

At 100k msg/sec, this is significant GC work.

### ISSUE D: No Coordination Between Rotation and Active Writes

**Impact**: Rare file corruption

Between rotation request completion and next message, the file handle is closed but map entry deleted. Next message creates a new file, bypassing the rotation.

---

## SHUTDOWN SEQUENCE ANALYSIS

**Current** (metricstore.go:273-314):
```
1. Cancel context
2. Close walShardChs
3. Wait for WAL staging drain
4. Write final checkpoint
```

**Problems**:
- (BUG #1) In-flight metrics panic on closed channel
- (BUG #3) Rotation requests deadlock
- (BUG #5) Buffered messages lost

**Correct sequence**:
```
1. SIGNAL: Stop accepting new metrics (don't close channels yet)
2. WAIT: Grace period for in-flight metrics (e.g., 100ms)
3. FLUSH: Close WAL channels to signal EOF
4. DRAIN: Wait for WAL staging goroutines to exit
5. ROTATE: Call RotateWALFiles for completed snapshots
6. CHECKPOINT: Write final checkpoint
7. DONE
```

---

## SEVERITY MATRIX

| ID | Bug | Severity | Category | Impact |
|----|-----|----------|----------|--------|
| 1 | Send after close | CRITICAL | Race | Panic on shutdown |
| 2 | Div by zero | CRITICAL | Race | Panic on startup with metrics |
| 3 | Rotation deadlock | CRITICAL | Deadlock | Shutdown hangs |
| 4 | Silent drops | CRITICAL | Data integrity | Durability violation |
| 5 | Lost metrics | CRITICAL | Data integrity | Crash data loss |
| 6 | Partial record | MEDIUM | Data integrity | Corruption on disk full |
| 7 | Inefficient drain | MEDIUM | Performance | Latency/CPU waste |
| 8 | Non-atomic snap+wal | HIGH | Data integrity | Crash data loss |
| 9 | Init race | MEDIUM | Race | Possible panic on startup |
| A | No backpressure | HIGH | Design | Durability risk |
| B | Snap-WAL sync | HIGH | Design | Crash data loss |
| C | Memory overhead | MEDIUM | Performance | GC pressure |
| D | Rotation sync | LOW | Race | Rare corruption |

---

## RECOMMENDED FIXES

### Immediate (causes crashes):
1. Fix BUG #2: Initialize walNumShards before NATS subscription
2. Fix BUG #1: Add safe shutdown signaling before closing channels
3. Fix BUG #3: Close rotation channels or guarantee goroutine liveness

### High Priority (data loss):
4. Fix BUG #5: Add grace period for in-flight metrics
5. Fix BUG #4: Implement backpressure instead of drop
6. Fix BUG #8: Make snapshot + WAL rotation atomic

### Medium Priority:
7. Fix BUG #6: Atomic record writes with intermediate buffer
8. Fix BUG #7: Better drain loop logic
9. Add comprehensive shutdown tests

---

## CODE LOCATIONS

- WAL write pipeline: `pkg/metricstore/walCheckpoint.go`
- Metric producer: `pkg/metricstore/lineprotocol.go`
- Lifecycle: `pkg/metricstore/metricstore.go`
- Checkpointing: `pkg/metricstore/checkpoint.go`


cat > /tmp/wal-detailed-fixes.md << 'EOF'
# WAL Pipeline: Detailed Bug Analysis & Fix Strategies

## BUG #1 & #2: PANIC ON SHUTDOWN (Send After Close)

### Root Cause Analysis

The shutdown sequence has a critical race window:

```
Timeline:
T0:  Shutdown() called
T1:  shutdownFunc() cancels context
T2:  ReceiveNats() reads ctx.Done in select, begins exit
T3:  ReceiveNats() calls wg.Wait() to drain worker goroutines
T4:  NATS message still in-flight gets decoded by worker
T5:  Worker calls DecodeLine() → SendWALMessage()
T6:  Shutdown() AFTER ctx.Done, closes walShardChs[i]
T7:  Worker at T5 reaches SendWALMessage, sends to CLOSED channel
T8:  PANIC: send on closed channel
```

### The Problem in Detail

File: `metricstore.go:273-314`

```go
func Shutdown() {
    totalStart := time.Now()

    shutdownFuncMu.Lock()
    defer shutdownFuncMu.Unlock()
    if shutdownFunc != nil {
        shutdownFunc()  // <-- T1: Cancel context
    }
    cclog.Infof("[METRICSTORE]> Background workers cancelled (%v)", time.Since(totalStart))

    // RACE WINDOW: ReceiveNats workers may still be decoding metrics!
    if Keys.Checkpoints.FileFormat == "wal" {
        for _, ch := range walShardChs {
            close(ch)  // <-- T6: Close channel while T5 is executing!
        }
        // ...
    }
}
```

File: `lineprotocol.go:45-101`

```go
func ReceiveNats(ms *MemoryStore, workers int, ctx context.Context) error {
    // ...
    if workers > 1 {
        wg.Add(workers)
        for range workers {
            go func() {
                defer wg.Done()
                for m := range msgs {  // <-- T4: Still receives message
                    dec := lineprotocol.NewDecoderWithBytes(m)
                    if err := DecodeLine(dec, ms, clusterTag); err != nil {
                        // ...
                    }
                }
            }()
        }
        nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
            select {
            case msgs <- data:  // <-- Message enqueued before ctx.Done
            case <-ctx.Done():
            }
        })
    }
    go func() {
        <-ctx.Done()
        close(msgs)  // <-- Close msgs channel
    }()
    wg.Wait()  // <-- T3: Block here
}
```

### Why It's Hard to Detect

The `msgs` channel closes AFTER ctx.Done(), but there's a window where:
1. A message is in the channel before ctx.Done
2. A worker reads it AFTER ctx.Done
3. The worker calls DecodeLine while Shutdown() closes walShardChs

This is a **classic ordered shutdown failure**.

### Why walShardChs != nil Check Doesn't Help

File: `walCheckpoint.go:138-151`

```go
func SendWALMessage(msg *WALMessage) bool {
    if walShardChs == nil {  // <-- Doesn't help!
        return false
    }
    shard := walShardIndex(msg.Cluster, msg.Node)
    select {
    case walShardChs[shard] <- msg:  // <-- Still panics if channel is closed
        return true
    default:
        return false
    }
}
```

**Go spec**: "A send on a closed channel proceeds by causing a run-time panic"
- No check can prevent the panic
- Only way: ensure channel is never closed while sends are in flight

### Fix Strategy

**Option A: Safe Shutdown Flag** (Recommended)

```go
// In walCheckpoint.go, global state:
var walShuttingDown atomic.Bool

// In SendWALMessage:
func SendWALMessage(msg *WALMessage) bool {
    if walShardChs == nil || walShuttingDown.Load() {
        return false  // Return false instead of panicking
    }
    shard := walShardIndex(msg.Cluster, msg.Node)
    select {
    case walShardChs[shard] <- msg:
        return true
    default:
        return false
    }
}

// In Shutdown (metricstore.go):
func Shutdown() {
    if Keys.Checkpoints.FileFormat == "wal" {
        walShuttingDown.Store(true)  // <-- Signal BEFORE closing
        time.Sleep(100 * time.Millisecond)  // Grace period
    }

    shutdownFuncMu.Lock()
    if shutdownFunc != nil {
        shutdownFunc()  // Now cancel context
    }
    shutdownFuncMu.Unlock()

    // Now safe to close channels
    if Keys.Checkpoints.FileFormat == "wal" {
        for _, ch := range walShardChs {
            close(ch)
        }
        WaitForWALStagingDrain()
    }

    // Write checkpoint...
}
```

**Option B: Graceful Drain** (More robust)

```go
// Add to walCheckpoint.go:
func CloseWALChannels() {
    if walShardChs == nil {
        return
    }
    for _, ch := range walShardChs {
        close(ch)
    }
}

// In metricstore.go Shutdown():
func Shutdown() {
    // Phase 1: Request stop (but don't close channels yet)
    shutdownFuncMu.Lock()
    if shutdownFunc != nil {
        shutdownFunc()  // Cancel context
    }
    shutdownFuncMu.Unlock()

    cclog.Infof("[METRICSTORE]> Background workers cancelled")

    // Phase 2: Grace period for in-flight metrics
    if Keys.Checkpoints.FileFormat == "wal" {
        time.Sleep(200 * time.Millisecond)

        // Phase 3: Close producer channel (signals EOF)
        CloseWALChannels()

        // Phase 4: Wait for consumer to drain
        WaitForWALStagingDrain()
    }

    // Phase 5: Final checkpoint
    cclog.Infof("[METRICSTORE]> Writing checkpoint to '%s'...", Keys.Checkpoints.RootDir)
    // ...
}
```

---

## BUG #2: DIVISION BY ZERO ON STARTUP

### Root Cause

File: `walCheckpoint.go:162-171`

```go
func WALStaging(wg *sync.WaitGroup, ctx context.Context) {
    if Keys.Checkpoints.FileFormat == "json" {
        return
    }

    walNumShards = max(Keys.NumWorkers, 1)  // <-- Line 162
    // ...
}
```

File: `metricstore.go:175, 188`

```go
// Init() execution order:
WALStaging(wg, ctx)      // Line 175 - spawns goroutine
// ...
ReceiveNats(ms, Keys.NumWorkers, ctx)  // Line 188 - spawns workers
```

File: `lineprotocol.go:68-70`

```go
dec := lineprotocol.NewDecoderWithBytes(m)
if err := DecodeLine(dec, ms, clusterTag); err != nil {  // Calls DecodeLine
```

File: `lineprotocol.go:355-364`

```go
func DecodeLine(...) error {
    // ...
    if Keys.Checkpoints.FileFormat == "wal" {
        msg := &WALMessage{ ... }
        if !SendWALMessage(msg) {  // Calls SendWALMessage
```

File: `walCheckpoint.go:130-136`

```go
func walShardIndex(cluster, node string) int {
    h := fnv.New32a()
    h.Write([]byte(cluster))
    h.Write([]byte{0})
    h.Write([]byte(node))
    return int(h.Sum32()) % walNumShards  // <-- PANIC if walNumShards == 0
}
```

### Why This Happens

The goroutine spawned by `WALStaging()` doesn't execute immediately. Meanwhile:
1. Init() continues to line 188
2. Calls ReceiveNats()
3. Calls nc.Subscribe() (subscribes to NATS subject)
4. Before WALStaging goroutine can set walNumShards, NATS delivers a message
5. Calls DecodeLine → SendWALMessage → walShardIndex
6. **PANIC**: modulo by zero

### Fix Strategy

**Synchronous Initialization** (Recommended)

```go
// In walCheckpoint.go:
func InitWALShards(numShards int) {
    walNumShards = max(numShards, 1)
    chBufSize := max(65536/walNumShards, 1024)

    walShardChs = make([]chan *WALMessage, walNumShards)
    walShardRotateChs = make([]chan walRotateReq, walNumShards)

    for i := range walNumShards {
        walShardChs[i] = make(chan *WALMessage, chBufSize)
        walShardRotateChs[i] = make(chan walRotateReq, 64)
    }
}

func WALStaging(wg *sync.WaitGroup, ctx context.Context) {
    if Keys.Checkpoints.FileFormat == "json" {
        return
    }

    // Channels already initialized by InitWALShards()

    for i := range walNumShards {
        // ... spawn goroutine for shard i
    }
}

// In metricstore.go Init():
// Synchronously initialize shards BEFORE starting any workers
if Keys.Checkpoints.FileFormat == "wal" {
    InitWALShards(Keys.NumWorkers)
}

// Now safe to start workers:
WALStaging(wg, ctx)
// ...
ReceiveNats(ms, Keys.NumWorkers, ctx)  // Safe, walNumShards is set
```

---

## BUG #3: ROTATION DEADLOCK

### Root Cause

File: `walCheckpoint.go:87-90`

```go
var walShardChs []chan *WALMessage
var walShardRotateChs []chan walRotateReq  // <-- NEVER closed
```

Timeline:
```
T0: Shutdown() called
T1: ctx.Done() → WALStaging goroutine sees EOF
T2: WALStaging reads msgCh, ok=false (channel closed)
T3: WALStaging returns, goroutine exits
T4: Meanwhile, Checkpointing calls RotateWALFiles(hostDirs)
T5: RotateWALFiles sends to rotateCh of now-exited goroutine
T6: RotateWALFiles blocks forever on <-done channel
T7: Shutdown hangs waiting for checkpoint
```

### Fix Strategy

**Option A: Close rotation channels before exit**

```go
// In walCheckpoint.go, add:
var walShardRotateChsMu sync.Mutex

func CloseRotationChannels() {
    walShardRotateChsMu.Lock()
    defer walShardRotateChsMu.Unlock()

    if walShardRotateChs == nil {
        return
    }
    for _, ch := range walShardRotateChs {
        close(ch)
    }
    walShardRotateChs = nil
}

// In WALStaging goroutine exit path:
defer func() {
    // ... existing cleanup ...
    if isLastShardGoroutine {  // Or: use waitgroup
        CloseRotationChannels()
    }
}()
```

**Option B: Check for closed channel in RotateWALFiles** (Simpler)

```go
// In RotateWALFiles:
func RotateWALFiles(hostDirs []string) {
    if walShardRotateChs == nil {
        return
    }

    dones := make([]chan struct{}, len(hostDirs))
    timeouts := make([]<-chan time.Time, len(hostDirs))

    for i, dir := range hostDirs {
        dones[i] = make(chan struct{})
        shard := walShardIndexFromDir(dir)

        // Use timeout to detect if goroutine is dead
        timeouts[i] = time.After(5 * time.Second)

        select {
        case walShardRotateChs[shard] <- walRotateReq{
            hostDir: dir,
            done:    dones[i],
        }:
            // Sent successfully
        case <-timeouts[i]:
            // Goroutine not responding, close the done channel ourselves
            close(dones[i])
        }
    }

    // Wait for responses with timeout
    for i, done := range dones {
        select {
        case <-done:
            // Normal completion
        case <-timeouts[i]:
            // Timeout, already handled above
            cclog.Warnf("[METRICSTORE]> Rotation timeout for host %d", i)
        }
    }
}
```

**Option C: Rotate before shutdown** (Best)

Don't call RotateWALFiles during shutdown at all. The final checkpoint doesn't need WAL rotation.

```go
// In metricstore.go Shutdown():
if Keys.Checkpoints.FileFormat == "wal" {
    // Don't call RotateWALFiles here
    // Files are deleted in ToCheckpointWAL workers anyway

    files, _, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
    // ToCheckpointWAL deletes current.wal files directly
}
```

---

## BUG #4: SILENT DATA LOSS ON BACKPRESSURE

### The Issue

```
Channel size = 65536 / walNumShards
With 4 workers: 65536 / 4 = 16384 per shard
At 100k msg/sec across 4 workers: 25k msg/sec per shard
Channel fills in: 16384 / 25000 = 0.65 seconds
```

Every drop violates the durability guarantee, but drops are only logged every 10k drops.

### Fix Strategy: Implement Real Backpressure

```go
// In lineprotocol.go DecodeLine:
if Keys.Checkpoints.FileFormat == "wal" {
    msg := &WALMessage{
        MetricName: string(st.metricBuf),
        Cluster:    cluster,
        Node:       host,
        Selector:   append([]string{}, st.selector...),
        Value:      metric.Value,
        Timestamp:  time.Unix(),
    }

    // Option 1: Block on full (applies backpressure)
    if !SendWALMessageBlocking(msg, 1*time.Second) {
        cclog.Errorf("[METRICSTORE]> WAL timeout for %s/%s metric %s, dropping",
            cluster, host, string(st.metricBuf))
        // Now we know it was dropped, and metrics show it
    }

    // Option 2: Increase channel size dynamically
    // or use unbounded queue
}

// In walCheckpoint.go:
func SendWALMessageBlocking(msg *WALMessage, timeout time.Duration) bool {
    if walShardChs == nil {
        return false
    }
    shard := walShardIndex(msg.Cluster, msg.Node)

    select {
    case walShardChs[shard] <- msg:
        return true
    case <-time.After(timeout):
        return false  // Timeout, metric dropped (with explicit error logged)
    }
}

// Or: Increase channel buffer significantly
const chBufSize = 1 << 20  // 1 million messages, ~100MB
```

---

## BUG #5: LOST METRICS AT SHUTDOWN

### The Issue

```go
case <-ctx.Done():
    // Comment says: "buffered messages are redundant"
    // WRONG! If metric arrives between snapshot T1 and ctx.Done at T2,
    // it's in the channel but won't be persisted.
    flushDirty()
    return
```

### Fix Strategy: Grace Period

```go
// In walCheckpoint.go WALStaging:
const ShutdownGracePeriod = 500 * time.Millisecond

for {
    select {
    case <-ctx.Done():
        // Don't exit immediately. Drain all buffered messages
        // that arrived before ctx.Done was signaled.

        // First, process any messages already in the channel
        for {
            select {
            case msg, ok := <-msgCh:
                if !ok {
                    // Channel closed, now safe to exit
                    flushDirty()
                    return
                }
                processMsg(msg)
            case req := <-rotateCh:
                processRotate(req)
            default:
                // No more messages immediately available
                // Give grace period for in-flight messages
                select {
                case msg, ok := <-msgCh:
                    if !ok {
                        flushDirty()
                        return
                    }
                    processMsg(msg)
                case <-time.After(ShutdownGracePeriod):
                    // Grace period expired, exit
                    flushDirty()
                    return
                }
            }
        }
    // ... other cases
    }
}
```

---

## BUG #6: PARTIAL RECORD ON DISK FULL

### The Issue

```go
func writeWALRecordDirect(w *bufio.Writer, msg *WALMessage) error {
    var hdr [8]byte
    binary.LittleEndian.PutUint32(hdr[0:4], walRecordMagic)
    binary.LittleEndian.PutUint32(hdr[4:8], uint32(payloadSize))
    if _, err := w.Write(hdr[:]); err != nil {
        return err  // <-- Partial data in buffer!
    }
    // ... more writes ...
}
```

If any write fails, previous writes sit in bufio.Writer. Next call appends, corrupting the file.

### Fix Strategy: Two-Phase Write

```go
func writeWALRecordDirect(w *bufio.Writer, msg *WALMessage) error {
    // Phase 1: Prepare all data in a buffer
    var buf bytes.Buffer

    // Write to temporary buffer
    payloadSize := 8 + 2 + len(msg.MetricName) + 1 + 4
    for _, s := range msg.Selector {
        payloadSize += 1 + len(s)
    }

    // Write magic + length
    var hdr [8]byte
    binary.LittleEndian.PutUint32(hdr[0:4], walRecordMagic)
    binary.LittleEndian.PutUint32(hdr[4:8], uint32(payloadSize))
    buf.Write(hdr[:])

    // ... prepare payload in buf ...

    // Phase 2: Compute CRC
    crc := crc32.NewIEEE()
    payload := buf.Bytes()[8:]  // Skip magic + length
    crc.Write(payload)

    // Phase 3: Atomic write (buffer is complete, write as one unit)
    if _, err := w.Write(buf.Bytes()); err != nil {
        return err
    }

    var crcBuf [4]byte
    binary.LittleEndian.PutUint32(crcBuf[:], crc.Sum32())
    if _, err := w.Write(crcBuf[:]); err != nil {
        return err
    }

    return nil
}
```

---

## BUG #8: NON-ATOMIC SNAPSHOT + WAL ROTATION

### The Issue

```
T1: Snapshot writes data up to timestamp T0
T2: Worker deletes current.wal
T3: Metric arrives with timestamp T0+100ms
T4: Metric written to current.wal
T5: Process crashes
T6: On restart: Snapshot from T0 + empty WAL
T7: Data from T0+100ms is lost!
```

### Fix Strategy: Atomic Rotation

**Option A: Rename + Replace**

```go
// In ToCheckpointWAL worker (line 697):
func (l *Level) toCheckpointBinary(dir string, from, to int64, m *MemoryStore) error {
    // ... write snapshot ...

    // Atomic rotation: rename current.wal to timestamp.wal.old
    walPath := path.Join(dir, "current.wal")
    oldPath := path.Join(dir, fmt.Sprintf("%d.wal.old", from))

    if err := os.Rename(walPath, oldPath); err != nil && !os.IsNotExist(err) {
        return fmt.Errorf("rename WAL: %w", err)
    }

    // old file will be deleted by cleanup job
    // or: delete it here if we're sure no reads are in progress
    os.Remove(oldPath)

    return nil
}
```

**Option B: Timestamp Marker**

```go
// Write checkpoint metadata with timestamp
type CheckpointMetadata struct {
    SnapshotTS int64  // Snapshot created up to this time
    CreatedAt  time.Time
}

// Store in .json or separate metadata file

// On load: Only replay WAL entries > SnapshotTS
```

**Option C: WAL Truncation Marker**

```go
// In current.wal, write a special marker record:
const walTruncateMarker = uint32(0xCC1DA7FF)

func writeWALTruncateMarker(w *bufio.Writer, ts int64) error {
    var hdr [12]byte
    binary.LittleEndian.PutUint32(hdr[0:4], walTruncateMarker)
    binary.LittleEndian.PutUint32(hdr[4:8], 8)
    binary.LittleEndian.PutUint64(hdr[8:16], uint64(ts))

    _, err := w.Write(hdr[:])
    return err
}

// On load: Stop replaying at truncate marker
```

---

## BUG #9: INITIALIZATION RACE

Same as BUG #2 analysis. Initialize walNumShards synchronously before NATS subscription.

---

## TESTING CHECKLIST

```go
// Test cases needed:
func TestShutdownWithInFlightMetrics(t *testing.T) {
    // Send metric, immediately shutdown
    // Verify no panic, metric persisted or acknowledged dropped
}

func TestRotationDeadlock(t *testing.T) {
    // Trigger checkpoint rotation during heavy write
    // Verify shutdown completes within timeout
}

func TestDiskFullRecovery(t *testing.T) {
    // Simulate disk full during WAL write
    // Verify WAL is not corrupted on restart
}

func TestNATSMetricsWithWALDisabled(t *testing.T) {
    // Run with WAL disabled
    // Verify no crashes
}

func TestBackpressure(t *testing.T) {
    // Send metrics faster than WAL can write
    // Verify either blocked or acknowledged dropped (not silent)
}

func TestSnapshotWALConsistency(t *testing.T) {
    // Write snapshot, then WAL, crash during gap
    // Verify data completeness on restart
}
```

EOF
cat /tmp/wal-detailed-fixes.md


# WAL Pipeline: Detailed Bug Analysis & Fix Strategies

## BUG #1 & #2: PANIC ON SHUTDOWN (Send After Close)

### Root Cause Analysis

The shutdown sequence has a critical race window:

```
Timeline:
T0:  Shutdown() called
T1:  shutdownFunc() cancels context
T2:  ReceiveNats() reads ctx.Done in select, begins exit
T3:  ReceiveNats() calls wg.Wait() to drain worker goroutines
T4:  NATS message still in-flight gets decoded by worker
T5:  Worker calls DecodeLine() → SendWALMessage()
T6:  Shutdown() AFTER ctx.Done, closes walShardChs[i]
T7:  Worker at T5 reaches SendWALMessage, sends to CLOSED channel
T8:  PANIC: send on closed channel
```

### The Problem in Detail

File: `metricstore.go:273-314`

```go
func Shutdown() {
    totalStart := time.Now()

    shutdownFuncMu.Lock()
    defer shutdownFuncMu.Unlock()
    if shutdownFunc != nil {
        shutdownFunc()  // <-- T1: Cancel context
    }
    cclog.Infof("[METRICSTORE]> Background workers cancelled (%v)", time.Since(totalStart))

    // RACE WINDOW: ReceiveNats workers may still be decoding metrics!
    if Keys.Checkpoints.FileFormat == "wal" {
        for _, ch := range walShardChs {
            close(ch)  // <-- T6: Close channel while T5 is executing!
        }
        // ...
    }
}
```

File: `lineprotocol.go:45-101`

```go
func ReceiveNats(ms *MemoryStore, workers int, ctx context.Context) error {
    // ...
    if workers > 1 {
        wg.Add(workers)
        for range workers {
            go func() {
                defer wg.Done()
                for m := range msgs {  // <-- T4: Still receives message
                    dec := lineprotocol.NewDecoderWithBytes(m)
                    if err := DecodeLine(dec, ms, clusterTag); err != nil {
                        // ...
                    }
                }
            }()
        }
        nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
            select {
            case msgs <- data:  // <-- Message enqueued before ctx.Done
            case <-ctx.Done():
            }
        })
    }
    go func() {
        <-ctx.Done()
        close(msgs)  // <-- Close msgs channel
    }()
    wg.Wait()  // <-- T3: Block here
}
```

### Why It's Hard to Detect

The `msgs` channel closes AFTER ctx.Done(), but there's a window where:
1. A message is in the channel before ctx.Done
2. A worker reads it AFTER ctx.Done
3. The worker calls DecodeLine while Shutdown() closes walShardChs

This is a **classic ordered shutdown failure**.

### Why walShardChs != nil Check Doesn't Help

File: `walCheckpoint.go:138-151`

```go
func SendWALMessage(msg *WALMessage) bool {
    if walShardChs == nil {  // <-- Doesn't help!
        return false
    }
    shard := walShardIndex(msg.Cluster, msg.Node)
    select {
    case walShardChs[shard] <- msg:  // <-- Still panics if channel is closed
        return true
    default:
        return false
    }
}
```

**Go spec**: "A send on a closed channel proceeds by causing a run-time panic"
- No check can prevent the panic
- Only way: ensure channel is never closed while sends are in flight

### Fix Strategy

**Option A: Safe Shutdown Flag** (Recommended)

```go
// In walCheckpoint.go, global state:
var walShuttingDown atomic.Bool

// In SendWALMessage:
func SendWALMessage(msg *WALMessage) bool {
    if walShardChs == nil || walShuttingDown.Load() {
        return false  // Return false instead of panicking
    }
    shard := walShardIndex(msg.Cluster, msg.Node)
    select {
    case walShardChs[shard] <- msg:
        return true
    default:
        return false
    }
}

// In Shutdown (metricstore.go):
func Shutdown() {
    if Keys.Checkpoints.FileFormat == "wal" {
        walShuttingDown.Store(true)  // <-- Signal BEFORE closing
        time.Sleep(100 * time.Millisecond)  // Grace period
    }

    shutdownFuncMu.Lock()
    if shutdownFunc != nil {
        shutdownFunc()  // Now cancel context
    }
    shutdownFuncMu.Unlock()

    // Now safe to close channels
    if Keys.Checkpoints.FileFormat == "wal" {
        for _, ch := range walShardChs {
            close(ch)
        }
        WaitForWALStagingDrain()
    }

    // Write checkpoint...
}
```

**Option B: Graceful Drain** (More robust)

```go
// Add to walCheckpoint.go:
func CloseWALChannels() {
    if walShardChs == nil {
        return
    }
    for _, ch := range walShardChs {
        close(ch)
    }
}

// In metricstore.go Shutdown():
func Shutdown() {
    // Phase 1: Request stop (but don't close channels yet)
    shutdownFuncMu.Lock()
    if shutdownFunc != nil {
        shutdownFunc()  // Cancel context
    }
    shutdownFuncMu.Unlock()

    cclog.Infof("[METRICSTORE]> Background workers cancelled")

    // Phase 2: Grace period for in-flight metrics
    if Keys.Checkpoints.FileFormat == "wal" {
        time.Sleep(200 * time.Millisecond)

        // Phase 3: Close producer channel (signals EOF)
        CloseWALChannels()

        // Phase 4: Wait for consumer to drain
        WaitForWALStagingDrain()
    }

    // Phase 5: Final checkpoint
    cclog.Infof("[METRICSTORE]> Writing checkpoint to '%s'...", Keys.Checkpoints.RootDir)
    // ...
}
```

---

## BUG #2: DIVISION BY ZERO ON STARTUP

### Root Cause

File: `walCheckpoint.go:162-171`

```go
func WALStaging(wg *sync.WaitGroup, ctx context.Context) {
    if Keys.Checkpoints.FileFormat == "json" {
        return
    }

    walNumShards = max(Keys.NumWorkers, 1)  // <-- Line 162
    // ...
}
```

File: `metricstore.go:175, 188`

```go
// Init() execution order:
WALStaging(wg, ctx)      // Line 175 - spawns goroutine
// ...
ReceiveNats(ms, Keys.NumWorkers, ctx)  // Line 188 - spawns workers
```

File: `lineprotocol.go:68-70`

```go
dec := lineprotocol.NewDecoderWithBytes(m)
if err := DecodeLine(dec, ms, clusterTag); err != nil {  // Calls DecodeLine
```

File: `lineprotocol.go:355-364`

```go
func DecodeLine(...) error {
    // ...
    if Keys.Checkpoints.FileFormat == "wal" {
        msg := &WALMessage{ ... }
        if !SendWALMessage(msg) {  // Calls SendWALMessage
```

File: `walCheckpoint.go:130-136`

```go
func walShardIndex(cluster, node string) int {
    h := fnv.New32a()
    h.Write([]byte(cluster))
    h.Write([]byte{0})
    h.Write([]byte(node))
    return int(h.Sum32()) % walNumShards  // <-- PANIC if walNumShards == 0
}
```

### Why This Happens

The goroutine spawned by `WALStaging()` doesn't execute immediately. Meanwhile:
1. Init() continues to line 188
2. Calls ReceiveNats()
3. Calls nc.Subscribe() (subscribes to NATS subject)
4. Before WALStaging goroutine can set walNumShards, NATS delivers a message
5. Calls DecodeLine → SendWALMessage → walShardIndex
6. **PANIC**: modulo by zero

### Fix Strategy

**Synchronous Initialization** (Recommended)

```go
// In walCheckpoint.go:
func InitWALShards(numShards int) {
    walNumShards = max(numShards, 1)
    chBufSize := max(65536/walNumShards, 1024)

    walShardChs = make([]chan *WALMessage, walNumShards)
    walShardRotateChs = make([]chan walRotateReq, walNumShards)

    for i := range walNumShards {
        walShardChs[i] = make(chan *WALMessage, chBufSize)
        walShardRotateChs[i] = make(chan walRotateReq, 64)
    }
}

func WALStaging(wg *sync.WaitGroup, ctx context.Context) {
    if Keys.Checkpoints.FileFormat == "json" {
        return
    }

    // Channels already initialized by InitWALShards()

    for i := range walNumShards {
        // ... spawn goroutine for shard i
    }
}

// In metricstore.go Init():
// Synchronously initialize shards BEFORE starting any workers
if Keys.Checkpoints.FileFormat == "wal" {
    InitWALShards(Keys.NumWorkers)
}

// Now safe to start workers:
WALStaging(wg, ctx)
// ...
ReceiveNats(ms, Keys.NumWorkers, ctx)  // Safe, walNumShards is set
```

---

## BUG #3: ROTATION DEADLOCK

### Root Cause

File: `walCheckpoint.go:87-90`

```go
var walShardChs []chan *WALMessage
var walShardRotateChs []chan walRotateReq  // <-- NEVER closed
```

Timeline:
```
T0: Shutdown() called
T1: ctx.Done() → WALStaging goroutine sees EOF
T2: WALStaging reads msgCh, ok=false (channel closed)
T3: WALStaging returns, goroutine exits
T4: Meanwhile, Checkpointing calls RotateWALFiles(hostDirs)
T5: RotateWALFiles sends to rotateCh of now-exited goroutine
T6: RotateWALFiles blocks forever on <-done channel
T7: Shutdown hangs waiting for checkpoint
```

### Fix Strategy

**Option A: Close rotation channels before exit**

```go
// In walCheckpoint.go, add:
var walShardRotateChsMu sync.Mutex

func CloseRotationChannels() {
    walShardRotateChsMu.Lock()
    defer walShardRotateChsMu.Unlock()

    if walShardRotateChs == nil {
        return
    }
    for _, ch := range walShardRotateChs {
        close(ch)
    }
    walShardRotateChs = nil
}

// In WALStaging goroutine exit path:
defer func() {
    // ... existing cleanup ...
    if isLastShardGoroutine {  // Or: use waitgroup
        CloseRotationChannels()
    }
}()
```

**Option B: Check for closed channel in RotateWALFiles** (Simpler)

```go
// In RotateWALFiles:
func RotateWALFiles(hostDirs []string) {
    if walShardRotateChs == nil {
        return
    }

    dones := make([]chan struct{}, len(hostDirs))
    timeouts := make([]<-chan time.Time, len(hostDirs))

    for i, dir := range hostDirs {
        dones[i] = make(chan struct{})
        shard := walShardIndexFromDir(dir)

        // Use timeout to detect if goroutine is dead
        timeouts[i] = time.After(5 * time.Second)

        select {
        case walShardRotateChs[shard] <- walRotateReq{
            hostDir: dir,
            done:    dones[i],
        }:
            // Sent successfully
        case <-timeouts[i]:
            // Goroutine not responding, close the done channel ourselves
            close(dones[i])
        }
    }

    // Wait for responses with timeout
    for i, done := range dones {
        select {
        case <-done:
            // Normal completion
        case <-timeouts[i]:
            // Timeout, already handled above
            cclog.Warnf("[METRICSTORE]> Rotation timeout for host %d", i)
        }
    }
}
```

**Option C: Rotate before shutdown** (Best)

Don't call RotateWALFiles during shutdown at all. The final checkpoint doesn't need WAL rotation.

```go
// In metricstore.go Shutdown():
if Keys.Checkpoints.FileFormat == "wal" {
    // Don't call RotateWALFiles here
    // Files are deleted in ToCheckpointWAL workers anyway

    files, _, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
    // ToCheckpointWAL deletes current.wal files directly
}
```

---

## BUG #4: SILENT DATA LOSS ON BACKPRESSURE

### The Issue

```
Channel size = 65536 / walNumShards
With 4 workers: 65536 / 4 = 16384 per shard
At 100k msg/sec across 4 workers: 25k msg/sec per shard
Channel fills in: 16384 / 25000 = 0.65 seconds
```

Every drop violates the durability guarantee, but drops are only logged every 10k drops.

### Fix Strategy: Implement Real Backpressure

```go
// In lineprotocol.go DecodeLine:
if Keys.Checkpoints.FileFormat == "wal" {
    msg := &WALMessage{
        MetricName: string(st.metricBuf),
        Cluster:    cluster,
        Node:       host,
        Selector:   append([]string{}, st.selector...),
        Value:      metric.Value,
        Timestamp:  time.Unix(),
    }

    // Option 1: Block on full (applies backpressure)
    if !SendWALMessageBlocking(msg, 1*time.Second) {
        cclog.Errorf("[METRICSTORE]> WAL timeout for %s/%s metric %s, dropping",
            cluster, host, string(st.metricBuf))
        // Now we know it was dropped, and metrics show it
    }

    // Option 2: Increase channel size dynamically
    // or use unbounded queue
}

// In walCheckpoint.go:
func SendWALMessageBlocking(msg *WALMessage, timeout time.Duration) bool {
    if walShardChs == nil {
        return false
    }
    shard := walShardIndex(msg.Cluster, msg.Node)

    select {
    case walShardChs[shard] <- msg:
        return true
    case <-time.After(timeout):
        return false  // Timeout, metric dropped (with explicit error logged)
    }
}

// Or: Increase channel buffer significantly
const chBufSize = 1 << 20  // 1 million messages, ~100MB
```

---

## BUG #5: LOST METRICS AT SHUTDOWN

### The Issue

```go
case <-ctx.Done():
    // Comment says: "buffered messages are redundant"
    // WRONG! If metric arrives between snapshot T1 and ctx.Done at T2,
    // it's in the channel but won't be persisted.
    flushDirty()
    return
```

### Fix Strategy: Grace Period

```go
// In walCheckpoint.go WALStaging:
const ShutdownGracePeriod = 500 * time.Millisecond

for {
    select {
    case <-ctx.Done():
        // Don't exit immediately. Drain all buffered messages
        // that arrived before ctx.Done was signaled.

        // First, process any messages already in the channel
        for {
            select {
            case msg, ok := <-msgCh:
                if !ok {
                    // Channel closed, now safe to exit
                    flushDirty()
                    return
                }
                processMsg(msg)
            case req := <-rotateCh:
                processRotate(req)
            default:
                // No more messages immediately available
                // Give grace period for in-flight messages
                select {
                case msg, ok := <-msgCh:
                    if !ok {
                        flushDirty()
                        return
                    }
                    processMsg(msg)
                case <-time.After(ShutdownGracePeriod):
                    // Grace period expired, exit
                    flushDirty()
                    return
                }
            }
        }
    // ... other cases
    }
}
```

---

## BUG #6: PARTIAL RECORD ON DISK FULL

### The Issue

```go
func writeWALRecordDirect(w *bufio.Writer, msg *WALMessage) error {
    var hdr [8]byte
    binary.LittleEndian.PutUint32(hdr[0:4], walRecordMagic)
    binary.LittleEndian.PutUint32(hdr[4:8], uint32(payloadSize))
    if _, err := w.Write(hdr[:]); err != nil {
        return err  // <-- Partial data in buffer!
    }
    // ... more writes ...
}
```

If any write fails, previous writes sit in bufio.Writer. Next call appends, corrupting the file.

### Fix Strategy: Two-Phase Write

```go
func writeWALRecordDirect(w *bufio.Writer, msg *WALMessage) error {
    // Phase 1: Prepare all data in a buffer
    var buf bytes.Buffer

    // Write to temporary buffer
    payloadSize := 8 + 2 + len(msg.MetricName) + 1 + 4
    for _, s := range msg.Selector {
        payloadSize += 1 + len(s)
    }

    // Write magic + length
    var hdr [8]byte
    binary.LittleEndian.PutUint32(hdr[0:4], walRecordMagic)
    binary.LittleEndian.PutUint32(hdr[4:8], uint32(payloadSize))
    buf.Write(hdr[:])

    // ... prepare payload in buf ...

    // Phase 2: Compute CRC
    crc := crc32.NewIEEE()
    payload := buf.Bytes()[8:]  // Skip magic + length
    crc.Write(payload)

    // Phase 3: Atomic write (buffer is complete, write as one unit)
    if _, err := w.Write(buf.Bytes()); err != nil {
        return err
    }

    var crcBuf [4]byte
    binary.LittleEndian.PutUint32(crcBuf[:], crc.Sum32())
    if _, err := w.Write(crcBuf[:]); err != nil {
        return err
    }

    return nil
}
```

---

## BUG #8: NON-ATOMIC SNAPSHOT + WAL ROTATION

### The Issue

```
T1: Snapshot writes data up to timestamp T0
T2: Worker deletes current.wal
T3: Metric arrives with timestamp T0+100ms
T4: Metric written to current.wal
T5: Process crashes
T6: On restart: Snapshot from T0 + empty WAL
T7: Data from T0+100ms is lost!
```

### Fix Strategy: Atomic Rotation

**Option A: Rename + Replace**

```go
// In ToCheckpointWAL worker (line 697):
func (l *Level) toCheckpointBinary(dir string, from, to int64, m *MemoryStore) error {
    // ... write snapshot ...

    // Atomic rotation: rename current.wal to timestamp.wal.old
    walPath := path.Join(dir, "current.wal")
    oldPath := path.Join(dir, fmt.Sprintf("%d.wal.old", from))

    if err := os.Rename(walPath, oldPath); err != nil && !os.IsNotExist(err) {
        return fmt.Errorf("rename WAL: %w", err)
    }

    // old file will be deleted by cleanup job
    // or: delete it here if we're sure no reads are in progress
    os.Remove(oldPath)

    return nil
}
```

**Option B: Timestamp Marker**

```go
// Write checkpoint metadata with timestamp
type CheckpointMetadata struct {
    SnapshotTS int64  // Snapshot created up to this time
    CreatedAt  time.Time
}

// Store in .json or separate metadata file

// On load: Only replay WAL entries > SnapshotTS
```

**Option C: WAL Truncation Marker**

```go
// In current.wal, write a special marker record:
const walTruncateMarker = uint32(0xCC1DA7FF)

func writeWALTruncateMarker(w *bufio.Writer, ts int64) error {
    var hdr [12]byte
    binary.LittleEndian.PutUint32(hdr[0:4], walTruncateMarker)
    binary.LittleEndian.PutUint32(hdr[4:8], 8)
    binary.LittleEndian.PutUint64(hdr[8:16], uint64(ts))

    _, err := w.Write(hdr[:])
    return err
}

// On load: Stop replaying at truncate marker
```

---

## BUG #9: INITIALIZATION RACE

Same as BUG #2 analysis. Initialize walNumShards synchronously before NATS subscription.

---

## TESTING CHECKLIST

```go
// Test cases needed:
func TestShutdownWithInFlightMetrics(t *testing.T) {
    // Send metric, immediately shutdown
    // Verify no panic, metric persisted or acknowledged dropped
}

func TestRotationDeadlock(t *testing.T) {
    // Trigger checkpoint rotation during heavy write
    // Verify shutdown completes within timeout
}

func TestDiskFullRecovery(t *testing.T) {
    // Simulate disk full during WAL write
    // Verify WAL is not corrupted on restart
}

func TestNATSMetricsWithWALDisabled(t *testing.T) {
    // Run with WAL disabled
    // Verify no crashes
}

func TestBackpressure(t *testing.T) {
    // Send metrics faster than WAL can write
    // Verify either blocked or acknowledged dropped (not silent)
}

func TestSnapshotWALConsistency(t *testing.T) {
    // Write snapshot, then WAL, crash during gap
    // Verify data completeness on restart
}
```


cat > /tmp/wal-summary.txt << 'EOF'
COMPREHENSIVE WAL BUFFER PIPELINE REVIEW
========================================

Location: /Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/

Files Analyzed:
- walCheckpoint.go (949 lines) - WAL write pipeline
- lineprotocol.go (380 lines) - Metric producer/ingestion
- metricstore.go (808 lines) - Initialization and shutdown
- checkpoint.go (670 lines) - Checkpointing and persistence

CRITICAL FINDINGS
=================

9 BUGS identified (5 CRITICAL severity):
├── BUG #1: PANIC ON SHUTDOWN - Send after close [CRITICAL]
├── BUG #2: DIVISION BY ZERO - Uninitialized walNumShards [CRITICAL]
├── BUG #3: DEADLOCK - Lost rotation requests [CRITICAL]
├── BUG #4: SILENT DATA LOSS - Dropped metrics on backpressure [CRITICAL]
├── BUG #5: LOST METRICS - Abandoned buffered messages at shutdown [CRITICAL]
├── BUG #6: PARTIAL RECORD - Disk full corruption [MEDIUM]
├── BUG #7: INEFFICIENT DRAIN - Loop logic issues [MEDIUM]
├── BUG #8: NON-ATOMIC SNAPSHOT - Snapshot/WAL desynchronization [HIGH]
└── BUG #9: INIT RACE - Partial initialization race [MEDIUM]

4 ARCHITECTURAL ISSUES:
├── ISSUE A: No backpressure mechanism (durability risk)
├── ISSUE B: Checkpoint-WAL desynchronization (crash data loss)
├── ISSUE C: Memory overhead (GC pressure)
└── ISSUE D: No rotation coordination (rare corruption)

IMPACT SUMMARY
==============

Data Loss:
- BUG #4: Metrics silently dropped when channel is full
- BUG #5: In-flight metrics lost at shutdown
- BUG #8: Metrics lost between snapshot and WAL rotation

Panics:
- BUG #1: "send on closed channel" panic on shutdown
- BUG #2: "division by zero" panic on startup with metrics
- BUG #9: Partial initialization panic

Hangs:
- BUG #3: Rotation request deadlock blocks Shutdown()

Corruption:
- BUG #6: Partial WAL records on disk full
- BUG #8: Data loss on crash between snapshot and WAL

Root Causes
===========

1. NO SHUTDOWN COORDINATION
   - ReceiveNats workers are async, may be decoding when channels close
   - No grace period for in-flight metrics
   - No mutual exclusion between ingestion and shutdown

2. UNORDERED INITIALIZATION
   - walNumShards set asynchronously by goroutine
   - NATS subscription happens before initialization complete
   - Results in division by zero if metric arrives early

3. IMPLICIT CHANNEL LIFECYCLE
   - walShardRotateChs never closed, causing deadlock
   - No EOF detection for rotation requests
   - Shutdown doesn't guarantee all goroutines exit

4. MISSING SYNCHRONIZATION POINTS
   - No atomic boundaries between snapshot and WAL operations
   - Metrics can arrive in gaps between checkpoint phases
   - No transaction markers or version tracking

RECOMMENDED IMMEDIATE FIXES
===========================

Priority 1 (Prevent Crashes):
1. Fix BUG #2: Synchronously initialize walNumShards before NATS subscription
2. Fix BUG #1: Add safe shutdown signaling before closing channels
3. Fix BUG #3: Timeout or guarantee goroutine liveness in RotateWALFiles

Priority 2 (Prevent Data Loss):
4. Fix BUG #5: Add grace period for in-flight metrics during shutdown
5. Fix BUG #4: Implement real backpressure instead of silent drops
6. Fix BUG #8: Make snapshot + WAL rotation atomic

Priority 3 (Data Integrity):
7. Fix BUG #6: Atomic record writes with intermediate buffer
8. Fix BUG #7: Improve drain loop logic
9. Add comprehensive shutdown and rotation tests

TESTING GAPS
============

Critical tests needed:
- Shutdown with in-flight metrics
- Rapid checkpoint intervals with rotation
- Disk full during WAL write
- Heavy load with backpressure
- Crash recovery between snapshot phases

ESTIMATED EFFORT
================

Fixes by complexity:
- BUG #1, #3: 2-3 hours (coordination logic)
- BUG #2: 1 hour (initialization order)
- BUG #4, #5: 3-4 hours (backpressure + grace period)
- BUG #6, #8: 4-5 hours (atomicity/transactions)
- BUG #7, #9: 1-2 hours (minor logic)
- Testing: 4-6 hours (race tests are complex)

Total estimated: 15-25 hours of development

KEY INSIGHTS
============

1. The WAL pipeline was designed for durability but lacks shutdown safety.
   The assumption that "buffered WAL records are redundant" due to binary
   snapshots is incorrect - metrics can arrive between snapshot completion
   and shutdown start.

2. The per-shard sharding strategy is good for parallelism but creates
   complexity around initialization and shutdown ordering.

3. The non-blocking send pattern is the right choice for performance, but
   requires careful shutdown sequencing to avoid panics.

4. Silent metric drops (BUG #4) are the most insidious issue - users have
   no way to know data was lost, violating the implicit durability promise
   of a persistent queue.

DETAILED ANALYSIS AVAILABLE IN
==============================

/tmp/wal-review.md - Executive summary with bug details
/tmp/wal-detailed-fixes.md - Code examples and fix strategies

EOF
cat /tmp/wal-summary.txt


COMPREHENSIVE WAL BUFFER PIPELINE REVIEW
========================================

Location: /Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/

Files Analyzed:
- walCheckpoint.go (949 lines) - WAL write pipeline
- lineprotocol.go (380 lines) - Metric producer/ingestion
- metricstore.go (808 lines) - Initialization and shutdown
- checkpoint.go (670 lines) - Checkpointing and persistence

CRITICAL FINDINGS
=================

9 BUGS identified (5 CRITICAL severity):
├── BUG #1: PANIC ON SHUTDOWN - Send after close [CRITICAL]
├── BUG #2: DIVISION BY ZERO - Uninitialized walNumShards [CRITICAL]
├── BUG #3: DEADLOCK - Lost rotation requests [CRITICAL]
├── BUG #4: SILENT DATA LOSS - Dropped metrics on backpressure [CRITICAL]
├── BUG #5: LOST METRICS - Abandoned buffered messages at shutdown [CRITICAL]
├── BUG #6: PARTIAL RECORD - Disk full corruption [MEDIUM]
├── BUG #7: INEFFICIENT DRAIN - Loop logic issues [MEDIUM]
├── BUG #8: NON-ATOMIC SNAPSHOT - Snapshot/WAL desynchronization [HIGH]
└── BUG #9: INIT RACE - Partial initialization race [MEDIUM]

4 ARCHITECTURAL ISSUES:
├── ISSUE A: No backpressure mechanism (durability risk)
├── ISSUE B: Checkpoint-WAL desynchronization (crash data loss)
├── ISSUE C: Memory overhead (GC pressure)
└── ISSUE D: No rotation coordination (rare corruption)

IMPACT SUMMARY
==============

Data Loss:
- BUG #4: Metrics silently dropped when channel is full
- BUG #5: In-flight metrics lost at shutdown
- BUG #8: Metrics lost between snapshot and WAL rotation

Panics:
- BUG #1: "send on closed channel" panic on shutdown
- BUG #2: "division by zero" panic on startup with metrics
- BUG #9: Partial initialization panic

Hangs:
- BUG #3: Rotation request deadlock blocks Shutdown()

Corruption:
- BUG #6: Partial WAL records on disk full
- BUG #8: Data loss on crash between snapshot and WAL

Root Causes
===========

1. NO SHUTDOWN COORDINATION
   - ReceiveNats workers are async, may be decoding when channels close
   - No grace period for in-flight metrics
   - No mutual exclusion between ingestion and shutdown

2. UNORDERED INITIALIZATION
   - walNumShards set asynchronously by goroutine
   - NATS subscription happens before initialization complete
   - Results in division by zero if metric arrives early

3. IMPLICIT CHANNEL LIFECYCLE
   - walShardRotateChs never closed, causing deadlock
   - No EOF detection for rotation requests
   - Shutdown doesn't guarantee all goroutines exit

4. MISSING SYNCHRONIZATION POINTS
   - No atomic boundaries between snapshot and WAL operations
   - Metrics can arrive in gaps between checkpoint phases
   - No transaction markers or version tracking

RECOMMENDED IMMEDIATE FIXES
===========================

Priority 1 (Prevent Crashes):
1. Fix BUG #2: Synchronously initialize walNumShards before NATS subscription
2. Fix BUG #1: Add safe shutdown signaling before closing channels
3. Fix BUG #3: Timeout or guarantee goroutine liveness in RotateWALFiles

Priority 2 (Prevent Data Loss):
4. Fix BUG #5: Add grace period for in-flight metrics during shutdown
5. Fix BUG #4: Implement real backpressure instead of silent drops
6. Fix BUG #8: Make snapshot + WAL rotation atomic

Priority 3 (Data Integrity):
7. Fix BUG #6: Atomic record writes with intermediate buffer
8. Fix BUG #7: Improve drain loop logic
9. Add comprehensive shutdown and rotation tests

TESTING GAPS
============

Critical tests needed:
- Shutdown with in-flight metrics
- Rapid checkpoint intervals with rotation
- Disk full during WAL write
- Heavy load with backpressure
- Crash recovery between snapshot phases

ESTIMATED EFFORT
================

Fixes by complexity:
- BUG #1, #3: 2-3 hours (coordination logic)
- BUG #2: 1 hour (initialization order)
- BUG #4, #5: 3-4 hours (backpressure + grace period)
- BUG #6, #8: 4-5 hours (atomicity/transactions)
- BUG #7, #9: 1-2 hours (minor logic)
- Testing: 4-6 hours (race tests are complex)

Total estimated: 15-25 hours of development

KEY INSIGHTS
============

1. The WAL pipeline was designed for durability but lacks shutdown safety.
   The assumption that "buffered WAL records are redundant" due to binary
   snapshots is incorrect - metrics can arrive between snapshot completion
   and shutdown start.

2. The per-shard sharding strategy is good for parallelism but creates
   complexity around initialization and shutdown ordering.

3. The non-blocking send pattern is the right choice for performance, but
   requires careful shutdown sequencing to avoid panics.

4. Silent metric drops (BUG #4) are the most insidious issue - users have
   no way to know data was lost, violating the implicit durability promise
   of a persistent queue.

DETAILED ANALYSIS AVAILABLE IN
==============================

/tmp/wal-review.md - Executive summary with bug details
/tmp/wal-detailed-fixes.md - Code examples and fix strategies