Now I have the full picture. Here is the implementation plan.

---

## Implementation Plan: Fix 60-Second Shutdown Timeout

### Problem Summary

The `metricstore.Shutdown()` function calls `shutdownFunc()` (context cancel), then closes WAL shard channels, then calls `ToCheckpointWAL()` which writes binary snapshots for all hosts, then calls `RotateWALFilesAfterShutdown()`. The snapshot write for thousands of hosts exceeds the 60-second deadline in `server.go`. There are three interrelated issues:

1. No logging to diagnose what is slow or how much progress has been made.
2. WAL files are only deleted after ALL snapshots succeed -- a timeout leaves WAL files on disk.
3. There is a race: `Shutdown()` cancels the context and closes WAL channels, but does NOT wait for the WALStaging goroutines to finish draining. The checkpoint write then starts immediately, potentially racing with WAL goroutines that are still flushing and closing files.

### Race Condition Analysis (Requirement 4)

In `Shutdown()` (metricstore.go:273-310):
- Line 277: `shutdownFunc()` cancels context, triggering `ctx.Done()` in each WALStaging goroutine.
- Lines 281-283: Immediately closes all `walShardChs[i]` channels.
- Line 298: Calls `ToCheckpointWAL()` which writes `.bin` files to the same host directories.

Meanwhile, each WALStaging goroutine (walCheckpoint.go:298-317):
- On `ctx.Done()`: calls `drain()`, which reads remaining messages, flushes, then returns.
- The defer at line 177-184 then flushes and closes all file handles.

The problem: closing the channels at line 281-283 can cause the `drainBatch()` loop (line 280-296) to return `false` when it sees the closed channel, which causes the goroutine to exit at line 309 -- **skipping** the `drain()` call entirely. The `select` on `ctx.Done()` at line 300 and the channel read at line 303 are racing. If the channel close wins, the goroutine exits without fully draining via `drain()`. However, the bigger concern is that `ToCheckpointWAL()` starts while WALStaging goroutines are still running their deferred flush/close, potentially writing to the same host directory concurrently.

**Decision**: Yes, we should wait for WALStaging goroutines to finish before starting checkpoint writes. The solution is to add a dedicated `sync.WaitGroup` or `done` channel for WALStaging goroutines that `Shutdown()` waits on before proceeding.

---

### Change 1: WALStaging Drain Synchronization

**File**: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/walCheckpoint.go`

Add a package-level `walStagingDone` channel (or WaitGroup) that signals when all WALStaging goroutines have exited.

**Specific changes**:

1. Add a new package-level variable at line ~93:
   ```go
   var walStagingWg sync.WaitGroup
   ```

2. In `WALStaging()` (line 154), for each shard goroutine, increment the WaitGroup before spawning and defer `Done()` inside. Currently the goroutines are added to the caller's `wg` via `wg.Go()`. Add a parallel tracking:
   ```go
   walStagingWg.Add(1)
   wg.Go(func() {
       defer walStagingWg.Done()
       // ... existing goroutine body ...
   })
   ```

3. Add a new exported function:
   ```go
   func WaitForWALStagingDrain() {
       walStagingWg.Wait()
   }
   ```

**File**: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/metricstore.go`

4. In `Shutdown()`, after closing WAL shard channels (line 283), add:
   ```go
   cclog.Info("[METRICSTORE]> Waiting for WAL staging goroutines to drain...")
   drainStart := time.Now()
   WaitForWALStagingDrain()
   cclog.Infof("[METRICSTORE]> WAL staging drained in %v", time.Since(drainStart))
   ```

This ensures all WALStaging goroutines have flushed, closed files, and exited before `ToCheckpointWAL()` begins.

---

### Change 2: Shutdown Phase Logging with Timings

**File**: `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/server.go`

In `Server.Shutdown()` (line 409), add timing instrumentation:

```go
func (s *Server) Shutdown(ctx context.Context) {
    shutdownStart := time.Now()
    cclog.Info("Shutdown: starting")

    // NATS
    natsStart := time.Now()
    nc := nats.GetClient()
    if nc != nil {
        nc.Close()
    }
    cclog.Infof("Shutdown: NATS closed in %v", time.Since(natsStart))

    // HTTP
    httpStart := time.Now()
    shutdownCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
    defer cancel()
    if err := s.server.Shutdown(shutdownCtx); err != nil {
        cclog.Errorf("Server shutdown error: %v", err)
    }
    cclog.Infof("Shutdown: HTTP server stopped in %v", time.Since(httpStart))

    // Concurrent metricstore + archiver
    storeStart := time.Now()
    done := make(chan struct{})
    go func() {
        defer close(done)
        var wg sync.WaitGroup
        if ms := metricstore.GetMemoryStore(); ms != nil {
            wg.Go(func() {
                metricstore.Shutdown()
            })
        }
        wg.Go(func() {
            if err := archiver.Shutdown(10 * time.Second); err != nil {
                cclog.Warnf("Archiver shutdown: %v", err)
            }
        })
        wg.Wait()
    }()

    select {
    case <-done:
        cclog.Infof("Shutdown: completed in %v", time.Since(shutdownStart))
    case <-time.After(60 * time.Second):
        cclog.Warnf("Shutdown deadline exceeded after %v, forcing exit", time.Since(shutdownStart))
    }
}
```

**File**: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/metricstore.go`

In `Shutdown()` (line 273), add phase timing:

```go
func Shutdown() {
    totalStart := time.Now()

    shutdownFuncMu.Lock()
    defer shutdownFuncMu.Unlock()
    if shutdownFunc != nil {
        shutdownFunc()
    }
    cclog.Infof("[METRICSTORE]> Context cancelled in %v", time.Since(totalStart))

    if Keys.Checkpoints.FileFormat == "wal" {
        for _, ch := range walShardChs {
            close(ch)
        }

        drainStart := time.Now()
        cclog.Info("[METRICSTORE]> Waiting for WAL staging goroutines to drain...")
        WaitForWALStagingDrain()
        cclog.Infof("[METRICSTORE]> WAL staging drained in %v", time.Since(drainStart))
    }

    cclog.Infof("[METRICSTORE]> Writing to '%s'...", Keys.Checkpoints.RootDir)
    checkpointStart := time.Now()
    // ... existing checkpoint code ...
    cclog.Infof("[METRICSTORE]> Done! (%d files written in %v)", files, time.Since(checkpointStart))
    cclog.Infof("[METRICSTORE]> Total shutdown time: %v", time.Since(totalStart))
}
```

---

### Change 3: Per-Host WAL Deletion Inside Worker Loop

**File**: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/walCheckpoint.go`

In `ToCheckpointWAL()` (line 627), modify the worker loop to delete the WAL file immediately after a successful binary snapshot. This replaces the all-at-the-end `RotateWALFilesAfterShutdown()` call.

Change the worker function (lines 667-684) from:

```go
go func() {
    defer wg.Done()
    for wi := range work {
        err := wi.level.toCheckpointBinary(wi.hostDir, from, to, m)
        if err != nil {
            if err == ErrNoNewArchiveData {
                continue
            }
            cclog.Errorf(...)
            atomic.AddInt32(&errs, 1)
        } else {
            atomic.AddInt32(&n, 1)
            successMu.Lock()
            successDirs = append(successDirs, wi.hostDir)
            successMu.Unlock()
        }
    }
}()
```

To:

```go
go func() {
    defer wg.Done()
    for wi := range work {
        err := wi.level.toCheckpointBinary(wi.hostDir, from, to, m)
        if err != nil {
            if err == ErrNoNewArchiveData {
                continue
            }
            cclog.Errorf("[METRICSTORE]> binary checkpoint error for %s: %v", wi.hostDir, err)
            atomic.AddInt32(&errs, 1)
        } else {
            atomic.AddInt32(&n, 1)
            // Delete WAL immediately after successful snapshot
            walPath := path.Join(wi.hostDir, "current.wal")
            if removeErr := os.Remove(walPath); removeErr != nil && !os.IsNotExist(removeErr) {
                cclog.Errorf("[METRICSTORE]> WAL remove %s: %v", walPath, removeErr)
            }
            successMu.Lock()
            successDirs = append(successDirs, wi.hostDir)
            successMu.Unlock()
        }
    }
}()
```

**File**: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/metricstore.go`

In `Shutdown()` (lines 296-301), remove the separate `RotateWALFilesAfterShutdown` call since WAL deletion now happens per-host inside the worker:

```go
// Before:
if Keys.Checkpoints.FileFormat == "wal" {
    var hostDirs []string
    files, hostDirs, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
    if err == nil {
        RotateWALFilesAfterShutdown(hostDirs)
    }
}

// After:
if Keys.Checkpoints.FileFormat == "wal" {
    files, _, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
    // WAL files are deleted per-host inside ToCheckpointWAL workers
}
```

The `RotateWALFilesAfterShutdown` function can be kept for now (it is still useful for the non-shutdown periodic rotation path), but it is no longer called from `Shutdown()`. Alternatively, if it is only used from `Shutdown()`, it can be removed entirely.

---

### Change 4: Progress Logging During Checkpoint

**File**: `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/walCheckpoint.go`

In `ToCheckpointWAL()`, add an atomic counter and a progress-logging goroutine. Modify lines 627-702:

Add after the `successMu` declaration (around line 660):

```go
var completed int32  // atomic counter for progress
totalWork := len(levels)
```

In the worker loop, after each successful snapshot or error, increment the counter:

```go
// After the err check block (both success and error paths, but not ErrNoNewArchiveData skips):
atomic.AddInt32(&completed, 1)
```

Add a progress-logging goroutine before the work-dispatch loop (before line 687):

```go
progressDone := make(chan struct{})
go func() {
    defer close(progressDone)
    ticker := time.NewTicker(10 * time.Second)
    defer ticker.Stop()
    for {
        select {
        case <-ticker.C:
            done := atomic.LoadInt32(&completed)
            written := atomic.LoadInt32(&n)
            errors := atomic.LoadInt32(&errs)
            cclog.Infof("[METRICSTORE]> Checkpoint progress: %d/%d hosts processed (%d written, %d errors)",
                done, totalWork, written, errors)
        case <-progressDone:
            return
        }
    }
}()
```

Wait: that won't work because we close `progressDone` to signal exit but also select on it. Use a different pattern -- stop the goroutine after `wg.Wait()`:

```go
stopProgress := make(chan struct{})
go func() {
    ticker := time.NewTicker(10 * time.Second)
    defer ticker.Stop()
    for {
        select {
        case <-ticker.C:
            done := atomic.LoadInt32(&completed)
            written := atomic.LoadInt32(&n)
            errors := atomic.LoadInt32(&errs)
            cclog.Infof("[METRICSTORE]> Checkpoint progress: %d/%d hosts processed (%d written, %d errors)",
                done, totalWork, written, errors)
        case <-stopProgress:
            return
        }
    }
}()
```

Then after `wg.Wait()` (line 696), add:

```go
close(stopProgress)
```

Also add an initial log line before dispatching work:

```go
cclog.Infof("[METRICSTORE]> Starting binary checkpoint for %d hosts with %d workers", totalWork, Keys.NumWorkers)
```

---

### Change Summary and Sequencing

1. **walCheckpoint.go**: Add `walStagingWg`, track WALStaging goroutine completion, add `WaitForWALStagingDrain()`.
2. **walCheckpoint.go**: In `ToCheckpointWAL()` worker loop, delete WAL per-host after successful snapshot; add progress counter and logging goroutine; add initial host-count log.
3. **metricstore.go**: In `Shutdown()`, wait for WAL staging drain before checkpointing; remove `RotateWALFilesAfterShutdown` call; add phase timing logs.
4. **server.go**: Add timing logs for each shutdown phase (NATS, HTTP, metricstore+archiver).

### Potential Risks

- **WAL deletion in worker**: The `os.Remove` for the WAL file is safe because WALStaging goroutines are guaranteed to have exited (we wait for them in Change 1). Without Change 1, this would be a data race.
- **Progress goroutine**: Minimal overhead (one goroutine, one ticker). Clean shutdown via `close(stopProgress)`.
- **WaitGroup tracking**: Adding `walStagingWg` is purely additive and cannot break existing behavior.

### Critical Files for Implementation

- `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/walCheckpoint.go`
- `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/metricstore.go`
- `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/server.go`
- `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/checkpoint.go`

Prompt:
I need a plan to fix a 60-second shutdown timeout issue in a Go HPC monitoring backend. Here's the full context:

## Problem
The shutdown consistently hits the 60s timeout. The log shows:
```
INFO  main.go:410: Shutdown signal received
INFO  client.go:241: NATS connection closed
INFO  archiveWorker.go:223: Initiating archiver shutdown...
INFO  archiveWorker.go:238: All archive jobs completed successfully
INFO  metricstore.go:286: [METRICSTORE]> Writing to './var/metric-checkpoints'...
INFO  archiveWorker.go:121: Archive channel closed, worker exiting
WARN  server.go:448: Shutdown deadline exceeded, forcing exit
```

The archiver finishes fast. The metricstore checkpoint write takes >60s and never completes within the deadline.

## Architecture
The metricstore keeps all metric data in memory (per cluster, per host, per metric). On shutdown it writes binary snapshot files for every host in parallel via a worker pool.

### Key code paths:

**server.go:409-450** - `Server.Shutdown()`:
- Closes NATS, shuts down HTTP (5s timeout)
- Runs metricstore.Shutdown() and archiver.Shutdown() concurrently via WaitGroup
- 60s overall deadline; if exceeded, forces exit

**metricstore.go:273-310** - `Shutdown()`:
1. Cancels context (stops all background goroutines)
2. Closes WAL shard channels
3. Calls `ToCheckpointWAL()` — writes binary snapshots for ALL hosts using NumWorkers worker pool
4. Calls `RotateWALFilesAfterShutdown()` — deletes WAL files

**walCheckpoint.go:627-702** - `ToCheckpointWAL()`:
- Collects all cluster/host pairs
- Spawns NumWorkers goroutines processing a work channel
- Each work item calls `toCheckpointBinary()` which:
  1. Calls `toCheckpointFile()` (holds RLock, reads all metric data for host)
  2. Writes binary file via `writeBinarySnapshotFile()`

**checkpoint.go:261-326** - `toCheckpointFile()`:
- Holds RLock on the Level
- For each metric, checks if all buffers are archived; if so, skips
- Otherwise reads data from `from` to `to` into []Float slice
- Recurses into children (socket → core → etc.)

**WALStaging goroutines** (walCheckpoint.go:154-320):
- N sharded goroutines writing WAL records to per-host files
- On ctx.Done(): drain remaining messages, flush, close files
- On channel close: also exit
- Shutdown() closes these channels but doesn't wait for them to finish

**RotateWALFilesAfterShutdown** (walCheckpoint.go:353-360):
- Simply does os.Remove() on each host's current.wal
- Only called AFTER ToCheckpointWAL succeeds
- If checkpoint times out, WAL files are NEVER deleted

## User Requirements
1. Add log messages to indicate shutdown phases and timings
2. WAL journals MUST be deleted before shutdown completes
3. Understand and improve what takes so long

## Analysis
The bottleneck is `ToCheckpointWAL()` writing binary snapshots for potentially thousands of hosts. Each host requires:
- RLock traversal of entire subtree (socket/core/accelerator levels)
- Data copy from buffer linked list
- Binary serialization and file I/O

Problems:
1. No timing/progress logging during checkpoint — impossible to diagnose
2. WAL deletion happens after checkpoint; if checkpoint times out, WALs remain
3. No way to know how many hosts or how much data is being written
4. The `from` timestamp may be up to 12h old (last periodic checkpoint), meaning massive data ranges

## Design the plan for:

1. **Shutdown phase logging with timings**: Add timing logs for each phase in server.go Shutdown() and metricstore Shutdown(). Include host count and file count progress.

2. **WAL deletion strategy**: Delete each host's WAL file immediately after its binary snapshot succeeds (inside the worker loop), not all at the end. This ensures WAL files are cleaned up even if the overall checkpoint doesn't finish within the deadline.

3. **Progress logging during checkpoint**: In ToCheckpointWAL, log progress periodically (e.g., every 100 hosts or every 10 seconds).

4. **Consider**: Should we wait for WALStaging goroutines to drain before starting checkpoint writes? Currently there's a potential race.

Please provide a detailed implementation plan with specific code changes needed.


Now let me also check the periodic checkpoint path to see if `RotateWALFiles` (not the "AfterShutdown" variant) is also used.


pkg/metricstore/walCheckpoint.go:322:// RotateWALFiles sends rotation requests for the given host directories
pkg/metricstore/walCheckpoint.go:325:func RotateWALFiles(hostDirs []string) {
pkg/metricstore/walCheckpoint.go:351:// RotateWALFiles sends rotation requests for the given host directories
pkg/metricstore/walCheckpoint.go:353:func RotateWALFilesAfterShutdown(hostDirs []string) {
pkg/metricstore/metricstore.go:300:			RotateWALFilesAfterShutdown(hostDirs)
pkg/metricstore/checkpoint.go:138:						RotateWALFiles(hostDirs)


Good — `RotateWALFiles` (the live version via shard channels) is used during periodic checkpoints, and `RotateWALFilesAfterShutdown` (direct `os.Remove`) is used only during shutdown. Both remain useful; we just move the shutdown deletion into the worker loop.


ToolSearch

Question: select:ExitPlanMode


Question: select:ExitPlanMode


Write

# Fix Shutdown Timeout & Add Shutdown Logging

## Context

The shutdown consistently hits the 60s timeout in `server.go`. The log shows `metricstore.go` starts writing checkpoints but never finishes before the deadline. The archiver completes quickly. The root cause is `ToCheckpointWAL()` writing binary snapshots for potentially thousands of hosts, which exceeds 60s. Additionally:

- No phase/timing logs make it impossible to diagnose what's slow
- WAL files are only deleted **after all** snapshots succeed — a timeout means WAL files are never cleaned up
- WALStaging goroutines aren't waited on before checkpoint writes begin (race condition)

## Changes

### 1. WALStaging drain synchronization (`pkg/metricstore/walCheckpoint.go`)

Add a package-level `walStagingWg sync.WaitGroup` to track WALStaging goroutines independently. In `WALStaging()`, wrap each shard goroutine with `walStagingWg.Add(1)` / `defer walStagingWg.Done()`. Export `WaitForWALStagingDrain()` that calls `walStagingWg.Wait()`.

This ensures all WAL file handles are flushed and closed before checkpoint writes begin.

### 2. Per-host WAL deletion in worker loop (`pkg/metricstore/walCheckpoint.go`)

In `ToCheckpointWAL()` worker loop, delete `current.wal` immediately after each successful binary snapshot write (inside the worker, after `toCheckpointBinary` succeeds). This ensures WAL files are cleaned up incrementally even if the overall checkpoint times out.

### 3. Progress logging in `ToCheckpointWAL()` (`pkg/metricstore/walCheckpoint.go`)

- Log total host count and worker count before dispatching work
- Add atomic `completed` counter, increment after each host (success or error)
- Spawn a goroutine that logs progress every 10s, stopped after `wg.Wait()`

### 4. Phase timing in `Shutdown()` (`pkg/metricstore/metricstore.go`)

- Time context cancellation, WAL drain wait, checkpoint write, and total
- Wait for WAL staging drain before starting checkpoint writes
- Remove `RotateWALFilesAfterShutdown()` call (now handled per-host in change 2)

### 5. Phase timing in `Server.Shutdown()` (`cmd/cc-backend/server.go`)

Add `time.Now()` + `time.Since()` logs for: NATS close, HTTP server stop, metricstore+archiver (concurrent), and overall shutdown.

## Files to modify

1. `pkg/metricstore/walCheckpoint.go` — changes 1, 2, 3
2. `pkg/metricstore/metricstore.go` — change 4
3. `cmd/cc-backend/server.go` — change 5

## Verification

1. `go build ./...` — must compile
2. `go test ./pkg/metricstore/...` — existing tests pass
3. Manual test: start with sample data, send SIGTERM, verify:
   - Phase timing logs appear in order
   - WAL files are deleted during checkpoint (not after)
   - "Graceful shutdown completed" appears (no 60s timeout)


File created successfully at: /Users/jan/.claude/plans/glistening-cuddling-hare.md

# Fix Shutdown Timeout & Add Shutdown Logging

## Context

The shutdown consistently hits the 60s timeout in `server.go`. The log shows `metricstore.go` starts writing checkpoints but never finishes before the deadline. The archiver completes quickly. The root cause is `ToCheckpointWAL()` writing binary snapshots for potentially thousands of hosts, which exceeds 60s. Additionally:

- No phase/timing logs make it impossible to diagnose what's slow
- WAL files are only deleted **after all** snapshots succeed — a timeout means WAL files are never cleaned up
- WALStaging goroutines aren't waited on before checkpoint writes begin (race condition)

## Changes

### 1. WALStaging drain synchronization (`pkg/metricstore/walCheckpoint.go`)

Add a package-level `walStagingWg sync.WaitGroup` to track WALStaging goroutines independently. In `WALStaging()`, wrap each shard goroutine with `walStagingWg.Add(1)` / `defer walStagingWg.Done()`. Export `WaitForWALStagingDrain()` that calls `walStagingWg.Wait()`.

This ensures all WAL file handles are flushed and closed before checkpoint writes begin.

### 2. Per-host WAL deletion in worker loop (`pkg/metricstore/walCheckpoint.go`)

In `ToCheckpointWAL()` worker loop, delete `current.wal` immediately after each successful binary snapshot write (inside the worker, after `toCheckpointBinary` succeeds). This ensures WAL files are cleaned up incrementally even if the overall checkpoint times out.

### 3. Progress logging in `ToCheckpointWAL()` (`pkg/metricstore/walCheckpoint.go`)

- Log total host count and worker count before dispatching work
- Add atomic `completed` counter, increment after each host (success or error)
- Spawn a goroutine that logs progress every 10s, stopped after `wg.Wait()`

### 4. Phase timing in `Shutdown()` (`pkg/metricstore/metricstore.go`)

- Time context cancellation, WAL drain wait, checkpoint write, and total
- Wait for WAL staging drain before starting checkpoint writes
- Remove `RotateWALFilesAfterShutdown()` call (now handled per-host in change 2)

### 5. Phase timing in `Server.Shutdown()` (`cmd/cc-backend/server.go`)

Add `time.Now()` + `time.Since()` logs for: NATS close, HTTP server stop, metricstore+archiver (concurrent), and overall shutdown.

## Files to modify

1. `pkg/metricstore/walCheckpoint.go` — changes 1, 2, 3
2. `pkg/metricstore/metricstore.go` — change 4
3. `cmd/cc-backend/server.go` — change 5

## Verification

1. `go build ./...` — must compile
2. `go test ./pkg/metricstore/...` — existing tests pass
3. Manual test: start with sample data, send SIGTERM, verify:
   - Phase timing logs appear in order
   - WAL files are deleted during checkpoint (not after)
   - "Graceful shutdown completed" appears (no 60s timeout)


ExitPlanMode

# Fix Shutdown Timeout & Add Shutdown Logging

## Context

The shutdown consistently hits the 60s timeout in `server.go`. The log shows `metricstore.go` starts writing checkpoints but never finishes before the deadline. The archiver completes quickly. The root cause is `ToCheckpointWAL()` writing binary snapshots for potentially thousands of hosts, which exceeds 60s. Additionally:

- No phase/timing logs make it impossible to diagnose what's slow
- WAL files are only deleted **after all** snapshots succeed — a timeout means WAL files are never cleaned up
- WALStaging goroutines aren't waited on before checkpoint writes begin (race condition)

## Changes

### 1. WALStaging drain synchronization (`pkg/metricstore/walCheckpoint.go`)

Add a package-level `walStagingWg sync.WaitGroup` to track WALStaging goroutines independently. In `WALStaging()`, wrap each shard goroutine with `walStagingWg.Add(1)` / `defer walStagingWg.Done()`. Export `WaitForWALStagingDrain()` that calls `walStagingWg.Wait()`.

This ensures all WAL file handles are flushed and closed before checkpoint writes begin.

### 2. Per-host WAL deletion in worker loop (`pkg/metricstore/walCheckpoint.go`)

In `ToCheckpointWAL()` worker loop, delete `current.wal` immediately after each successful binary snapshot write (inside the worker, after `toCheckpointBinary` succeeds). This ensures WAL files are cleaned up incrementally even if the overall checkpoint times out.

### 3. Progress logging in `ToCheckpointWAL()` (`pkg/metricstore/walCheckpoint.go`)

- Log total host count and worker count before dispatching work
- Add atomic `completed` counter, increment after each host (success or error)
- Spawn a goroutine that logs progress every 10s, stopped after `wg.Wait()`

### 4. Phase timing in `Shutdown()` (`pkg/metricstore/metricstore.go`)

- Time context cancellation, WAL drain wait, checkpoint write, and total
- Wait for WAL staging drain before starting checkpoint writes
- Remove `RotateWALFilesAfterShutdown()` call (now handled per-host in change 2)

### 5. Phase timing in `Server.Shutdown()` (`cmd/cc-backend/server.go`)

Add `time.Now()` + `time.Since()` logs for: NATS close, HTTP server stop, metricstore+archiver (concurrent), and overall shutdown.

## Files to modify

1. `pkg/metricstore/walCheckpoint.go` — changes 1, 2, 3
2. `pkg/metricstore/metricstore.go` — change 4
3. `cmd/cc-backend/server.go` — change 5

## Verification

1. `go build ./...` — must compile
2. `go test ./pkg/metricstore/...` — existing tests pass
3. Manual test: start with sample data, send SIGTERM, verify:
   - Phase timing logs appear in order
   - WAL files are deleted during checkpoint (not after)
   - "Graceful shutdown completed" appears (no 60s timeout)


Error: The user doesn't want to proceed with this tool use. The tool use was rejected (eg. if it was a file edit, the new_string was NOT written to the file). To tell you how to proceed, the user said:
There are indications that the introduction of WAL sharding caused the long shutdown times. Further investigate if this relates to the issue.


Agent

Investigate WAL sharding shutdown

Prompt:
The user suspects that WAL sharding caused the long shutdown times. I need to investigate how WAL sharding interacts with shutdown.

Key areas to explore:

1. Read `pkg/metricstore/walCheckpoint.go` completely - understand the WAL sharding mechanism, especially:
   - How many shards are created (walNumShards)
   - Channel buffer sizes (chBufSize)
   - What happens when channels are closed during shutdown
   - The `drain()` function behavior
   - How many messages could be buffered in channels at shutdown time

2. Read `pkg/metricstore/lineprotocol.go` - how WAL messages are sent to shard channels. Is there a blocking send? Could large channel buffers cause drain delays?

3. Read `pkg/metricstore/metricstore.go` fully - the Init() and Shutdown() functions, especially how NumWorkers is configured and used.

4. Check git log for when WAL sharding was introduced - `git log --oneline --all -- pkg/metricstore/walCheckpoint.go` and look at the relevant commits.

5. Think about: if there are N shards, each with a buffer of 65536/N messages, and at shutdown time these channels need to be drained (each message triggers a file write), how long could that take?

6. Check if the issue is that Shutdown() closes WAL channels, then WALStaging goroutines drain them (processing potentially thousands of buffered messages with file I/O each), and THEN ToCheckpointWAL runs - meaning the drain itself takes most of the 60 seconds.

Be very thorough - read all relevant code and trace the exact shutdown sequence step by step.


The user suspects that WAL sharding caused the long shutdown times. I need to investigate how WAL sharding interacts with shutdown.

Key areas to explore:

1. Read `pkg/metricstore/walCheckpoint.go` completely - understand the WAL sharding mechanism, especially:
   - How many shards are created (walNumShards)
   - Channel buffer sizes (chBufSize)
   - What happens when channels are closed during shutdown
   - The `drain()` function behavior
   - How many messages could be buffered in channels at shutdown time

2. Read `pkg/metricstore/lineprotocol.go` - how WAL messages are sent to shard channels. Is there a blocking send? Could large channel buffers cause drain delays?

3. Read `pkg/metricstore/metricstore.go` fully - the Init() and Shutdown() functions, especially how NumWorkers is configured and used.

4. Check git log for when WAL sharding was introduced - `git log --oneline --all -- pkg/metricstore/walCheckpoint.go` and look at the relevant commits.

5. Think about: if there are N shards, each with a buffer of 65536/N messages, and at shutdown time these channels need to be drained (each message triggers a file write), how long could that take?

6. Check if the issue is that Shutdown() closes WAL channels, then WALStaging goroutines drain them (processing potentially thousands of buffered messages with file I/O each), and THEN ToCheckpointWAL runs - meaning the drain itself takes most of the 60 seconds.

Be very thorough - read all relevant code and trace the exact shutdown sequence step by step.

Prompt:
The user suspects that WAL sharding caused the long shutdown times. I need to investigate how WAL sharding interacts with shutdown.

Key areas to explore:

1. Read `pkg/metricstore/walCheckpoint.go` completely - understand the WAL sharding mechanism, especially:
   - How many shards are created (walNumShards)
   - Channel buffer sizes (chBufSize)
   - What happens when channels are closed during shutdown
   - The `drain()` function behavior
   - How many messages could be buffered in channels at shutdown time

2. Read `pkg/metricstore/lineprotocol.go` - how WAL messages are sent to shard channels. Is there a blocking send? Could large channel buffers cause drain delays?

3. Read `pkg/metricstore/metricstore.go` fully - the Init() and Shutdown() functions, especially how NumWorkers is configured and used.

4. Check git log for when WAL sharding was introduced - `git log --oneline --all -- pkg/metricstore/walCheckpoint.go` and look at the relevant commits.

5. Think about: if there are N shards, each with a buffer of 65536/N messages, and at shutdown time these channels need to be drained (each message triggers a file write), how long could that take?

6. Check if the issue is that Shutdown() closes WAL channels, then WALStaging goroutines drain them (processing potentially thousands of buffered messages with file I/O each), and THEN ToCheckpointWAL runs - meaning the drain itself takes most of the 60 seconds.

Be very thorough - read all relevant code and trace the exact shutdown sequence step by step.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstore provides walCheckpoint.go: WAL-based checkpoint implementation.
     7→//
     8→// This replaces the Avro shadow tree with an append-only Write-Ahead Log (WAL)
     9→// per host, eliminating the extra memory overhead of the AvroStore and providing
    10→// truly continuous (per-write) crash safety.
    11→//
    12→// # Architecture
    13→//
    14→//	Metric write (DecodeLine)
    15→//	      │
    16→//	      ├─► WriteToLevel()    → main MemoryStore (unchanged)
    17→//	      │
    18→//	      └─► WALMessages channel
    19→//	               │
    20→//	               ▼
    21→//	        WALStaging goroutine
    22→//	               │
    23→//	               ▼
    24→//	        checkpoints/cluster/host/current.wal  (append-only, binary)
    25→//
    26→//	Periodic checkpoint (Checkpointing goroutine):
    27→//	  1. Write <timestamp>.bin snapshot (column-oriented, from main tree)
    28→//	  2. Signal WALStaging to truncate current.wal per host
    29→//
    30→//	On restart (FromCheckpoint):
    31→//	  1. Load most recent <timestamp>.bin snapshot
    32→//	  2. Replay current.wal (overwrite-safe: buffer.write handles duplicate timestamps)
    33→//
    34→// # WAL Record Format
    35→//
    36→//	[4B magic 0xCC1DA7A1][4B payload_len][payload][4B CRC32]
    37→//
    38→//	payload:
    39→//	  [8B timestamp int64]
    40→//	  [2B metric_name_len uint16][N metric name bytes]
    41→//	  [1B selector_count uint8]
    42→//	  per selector: [1B selector_len uint8][M selector bytes]
    43→//	  [4B value float32 bits]
    44→//
    45→// # Binary Snapshot Format
    46→//
    47→//	[4B magic 0xCC5B0001][8B from int64][8B to int64]
    48→//	Level tree (recursive):
    49→//	  [4B num_metrics uint32]
    50→//	  per metric:
    51→//	    [2B name_len uint16][N name bytes]
    52→//	    [8B frequency int64][8B start int64]
    53→//	    [4B num_values uint32][num_values × 4B float32]
    54→//	  [4B num_children uint32]
    55→//	  per child: [2B name_len uint16][N name bytes] + Level (recursive)
    56→package metricstore
    57→
    58→import (
    59→	"bufio"
    60→	"context"
    61→	"encoding/binary"
    62→	"fmt"
    63→	"hash/crc32"
    64→	"hash/fnv"
    65→	"io"
    66→	"math"
    67→	"os"
    68→	"path"
    69→	"strings"
    70→	"sync"
    71→	"sync/atomic"
    72→	"time"
    73→
    74→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    75→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    76→)
    77→
    78→// Magic numbers for binary formats.
    79→const (
    80→	walFileMagic   = uint32(0xCC1DA701) // WAL file header magic
    81→	walRecordMagic = uint32(0xCC1DA7A1) // WAL record magic
    82→	snapFileMagic  = uint32(0xCC5B0001) // Binary snapshot magic
    83→)
    84→
    85→// walShardChs holds per-shard channels for WAL messages.
    86→// Initialized by WALStaging; nil when WAL is not active.
    87→var walShardChs []chan *WALMessage
    88→
    89→// walShardRotateChs holds per-shard channels for WAL rotation requests.
    90→var walShardRotateChs []chan walRotateReq
    91→
    92→// walNumShards stores the number of shards (set during WALStaging init).
    93→var walNumShards int
    94→
    95→// WALMessage represents a single metric write to be appended to the WAL.
    96→// Cluster and Node are NOT stored in the WAL record (inferred from file path).
    97→type WALMessage struct {
    98→	MetricName string
    99→	Cluster    string
   100→	Node       string
   101→	Selector   []string
   102→	Value      schema.Float
   103→	Timestamp  int64
   104→}
   105→
   106→// walRotateReq requests WAL file rotation for a specific host directory.
   107→// The done channel is closed by the WAL goroutine when rotation is complete.
   108→type walRotateReq struct {
   109→	hostDir string
   110→	done    chan struct{}
   111→}
   112→
   113→// walFileState holds an open WAL file handle and buffered writer for one host directory.
   114→type walFileState struct {
   115→	f     *os.File
   116→	w     *bufio.Writer
   117→	dirty bool
   118→}
   119→
   120→// walFlushInterval controls how often dirty WAL files are flushed to disk.
   121→// Decoupling flushes from message processing lets the consumer run at memory
   122→// speed, amortizing syscall overhead across many writes.
   123→const walFlushInterval = 5 * time.Second
   124→
   125→// walShardIndex computes which shard a message belongs to based on cluster+node.
   126→// Uses FNV-1a hash for fast, well-distributed mapping.
   127→func walShardIndex(cluster, node string) int {
   128→	h := fnv.New32a()
   129→	h.Write([]byte(cluster))
   130→	h.Write([]byte{0})
   131→	h.Write([]byte(node))
   132→	return int(h.Sum32()) % walNumShards
   133→}
   134→
   135→// SendWALMessage routes a WAL message to the appropriate shard channel.
   136→// Returns false if the channel is full (message dropped).
   137→func SendWALMessage(msg *WALMessage) bool {
   138→	if walShardChs == nil {
   139→		return false
   140→	}
   141→	shard := walShardIndex(msg.Cluster, msg.Node)
   142→	select {
   143→	case walShardChs[shard] <- msg:
   144→		return true
   145→	default:
   146→		return false
   147→	}
   148→}
   149→
   150→// WALStaging starts N sharded background goroutines that receive WALMessage items
   151→// and append binary WAL records to per-host current.wal files.
   152→// Each shard owns a subset of hosts (determined by hash of cluster+node),
   153→// parallelizing serialization and I/O while keeping per-host writes serial.
   154→func WALStaging(wg *sync.WaitGroup, ctx context.Context) {
   155→	if Keys.Checkpoints.FileFormat == "json" {
   156→		return
   157→	}
   158→
   159→	walNumShards = max(Keys.NumWorkers, 1)
   160→	chBufSize := max(65536/walNumShards, 1024)
   161→
   162→	walShardChs = make([]chan *WALMessage, walNumShards)
   163→	walShardRotateChs = make([]chan walRotateReq, walNumShards)
   164→
   165→	for i := range walNumShards {
   166→		walShardChs[i] = make(chan *WALMessage, chBufSize)
   167→		walShardRotateChs[i] = make(chan walRotateReq, 64)
   168→	}
   169→
   170→	for i := range walNumShards {
   171→		msgCh := walShardChs[i]
   172→		rotateCh := walShardRotateChs[i]
   173→
   174→		wg.Go(func() {
   175→			hostFiles := make(map[string]*walFileState)
   176→
   177→			defer func() {
   178→				for _, ws := range hostFiles {
   179→					if ws.f != nil {
   180→						ws.w.Flush()
   181→						ws.f.Close()
   182→					}
   183→				}
   184→			}()
   185→
   186→			getOrOpenWAL := func(hostDir string) *walFileState {
   187→				ws, ok := hostFiles[hostDir]
   188→				if ok {
   189→					return ws
   190→				}
   191→
   192→				if err := os.MkdirAll(hostDir, CheckpointDirPerms); err != nil {
   193→					cclog.Errorf("[METRICSTORE]> WAL: mkdir %s: %v", hostDir, err)
   194→					return nil
   195→				}
   196→
   197→				walPath := path.Join(hostDir, "current.wal")
   198→				f, err := os.OpenFile(walPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, CheckpointFilePerms)
   199→				if err != nil {
   200→					cclog.Errorf("[METRICSTORE]> WAL: open %s: %v", walPath, err)
   201→					return nil
   202→				}
   203→
   204→				w := bufio.NewWriter(f)
   205→
   206→				// Write file header magic if file is new (empty).
   207→				info, err := f.Stat()
   208→				if err == nil && info.Size() == 0 {
   209→					var hdr [4]byte
   210→					binary.LittleEndian.PutUint32(hdr[:], walFileMagic)
   211→					if _, err := w.Write(hdr[:]); err != nil {
   212→						cclog.Errorf("[METRICSTORE]> WAL: write header %s: %v", walPath, err)
   213→						f.Close()
   214→						return nil
   215→					}
   216→				}
   217→
   218→				ws = &walFileState{f: f, w: w}
   219→				hostFiles[hostDir] = ws
   220→				return ws
   221→			}
   222→
   223→			processMsg := func(msg *WALMessage) {
   224→				hostDir := path.Join(Keys.Checkpoints.RootDir, msg.Cluster, msg.Node)
   225→				ws := getOrOpenWAL(hostDir)
   226→				if ws == nil {
   227→					return
   228→				}
   229→				if err := writeWALRecordDirect(ws.w, msg); err != nil {
   230→					cclog.Errorf("[METRICSTORE]> WAL: write record: %v", err)
   231→				}
   232→				ws.dirty = true
   233→			}
   234→
   235→			processRotate := func(req walRotateReq) {
   236→				ws, ok := hostFiles[req.hostDir]
   237→				if ok && ws.f != nil {
   238→					ws.w.Flush()
   239→					ws.f.Close()
   240→					walPath := path.Join(req.hostDir, "current.wal")
   241→					if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
   242→						cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
   243→					}
   244→					delete(hostFiles, req.hostDir)
   245→				}
   246→				close(req.done)
   247→			}
   248→
   249→			flushDirty := func() {
   250→				for _, ws := range hostFiles {
   251→					if ws.dirty {
   252→						ws.w.Flush()
   253→						ws.dirty = false
   254→					}
   255→				}
   256→			}
   257→
   258→			drain := func() {
   259→				for {
   260→					select {
   261→					case msg, ok := <-msgCh:
   262→						if !ok {
   263→							return
   264→						}
   265→						processMsg(msg)
   266→					case req := <-rotateCh:
   267→						processRotate(req)
   268→					default:
   269→						flushDirty()
   270→						return
   271→					}
   272→				}
   273→			}
   274→
   275→			ticker := time.NewTicker(walFlushInterval)
   276→			defer ticker.Stop()
   277→
   278→			// drainBatch processes up to 4096 pending messages without blocking.
   279→			// Returns false if the channel was closed.
   280→			drainBatch := func() bool {
   281→				for range 4096 {
   282→					select {
   283→					case msg, ok := <-msgCh:
   284→						if !ok {
   285→							flushDirty()
   286→							return false
   287→						}
   288→						processMsg(msg)
   289→					case req := <-rotateCh:
   290→						processRotate(req)
   291→					default:
   292→						return true
   293→					}
   294→				}
   295→				return true
   296→			}
   297→
   298→			for {
   299→				select {
   300→				case <-ctx.Done():
   301→					drain()
   302→					return
   303→				case msg, ok := <-msgCh:
   304→					if !ok {
   305→						return
   306→					}
   307→					processMsg(msg)
   308→					if !drainBatch() {
   309→						return
   310→					}
   311→					// No flush here — timer handles periodic flushing.
   312→				case <-ticker.C:
   313→					flushDirty()
   314→				case req := <-rotateCh:
   315→					processRotate(req)
   316→				}
   317→			}
   318→		})
   319→	}
   320→}
   321→
   322→// RotateWALFiles sends rotation requests for the given host directories
   323→// and blocks until all rotations complete. Each request is routed to the
   324→// shard that owns the host directory.
   325→func RotateWALFiles(hostDirs []string) {
   326→	if walShardRotateChs == nil {
   327→		return
   328→	}
   329→	dones := make([]chan struct{}, len(hostDirs))
   330→	for i, dir := range hostDirs {
   331→		dones[i] = make(chan struct{})
   332→		// Extract cluster/node from hostDir to find the right shard.
   333→		// hostDir = rootDir/cluster/node
   334→		shard := walShardIndexFromDir(dir)
   335→		walShardRotateChs[shard] <- walRotateReq{hostDir: dir, done: dones[i]}
   336→	}
   337→	for _, done := range dones {
   338→		<-done
   339→	}
   340→}
   341→
   342→// walShardIndexFromDir extracts cluster and node from a host directory path
   343→// and returns the shard index. The path format is: rootDir/cluster/node
   344→func walShardIndexFromDir(hostDir string) int {
   345→	// hostDir = .../cluster/node — extract last two path components
   346→	node := path.Base(hostDir)
   347→	cluster := path.Base(path.Dir(hostDir))
   348→	return walShardIndex(cluster, node)
   349→}
   350→
   351→// RotateWALFiles sends rotation requests for the given host directories
   352→// and blocks until all rotations complete.
   353→func RotateWALFilesAfterShutdown(hostDirs []string) {
   354→	for _, dir := range hostDirs {
   355→		walPath := path.Join(dir, "current.wal")
   356→		if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
   357→			cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
   358→		}
   359→	}
   360→}
   361→
   362→// writeWALRecordDirect encodes a WAL record directly into the bufio.Writer,
   363→// avoiding heap allocations by using a stack-allocated scratch buffer for
   364→// the fixed-size header/trailer and computing CRC inline.
   365→func writeWALRecordDirect(w *bufio.Writer, msg *WALMessage) error {
   366→	// Compute payload size.
   367→	payloadSize := 8 + 2 + len(msg.MetricName) + 1 + 4
   368→	for _, s := range msg.Selector {
   369→		payloadSize += 1 + len(s)
   370→	}
   371→
   372→	// Write magic + payload length (8 bytes header).
   373→	var hdr [8]byte
   374→	binary.LittleEndian.PutUint32(hdr[0:4], walRecordMagic)
   375→	binary.LittleEndian.PutUint32(hdr[4:8], uint32(payloadSize))
   376→	if _, err := w.Write(hdr[:]); err != nil {
   377→		return err
   378→	}
   379→
   380→	// We need to compute CRC over the payload as we write it.
   381→	crc := crc32.NewIEEE()
   382→
   383→	// Timestamp (8 bytes).
   384→	var scratch [8]byte
   385→	binary.LittleEndian.PutUint64(scratch[:8], uint64(msg.Timestamp))
   386→	crc.Write(scratch[:8])
   387→	if _, err := w.Write(scratch[:8]); err != nil {
   388→		return err
   389→	}
   390→
   391→	// Metric name length (2 bytes) + metric name.
   392→	binary.LittleEndian.PutUint16(scratch[:2], uint16(len(msg.MetricName)))
   393→	crc.Write(scratch[:2])
   394→	if _, err := w.Write(scratch[:2]); err != nil {
   395→		return err
   396→	}
   397→	nameBytes := []byte(msg.MetricName)
   398→	crc.Write(nameBytes)
   399→	if _, err := w.Write(nameBytes); err != nil {
   400→		return err
   401→	}
   402→
   403→	// Selector count (1 byte).
   404→	scratch[0] = byte(len(msg.Selector))
   405→	crc.Write(scratch[:1])
   406→	if _, err := w.Write(scratch[:1]); err != nil {
   407→		return err
   408→	}
   409→
   410→	// Selectors (1-byte length + bytes each).
   411→	for _, sel := range msg.Selector {
   412→		scratch[0] = byte(len(sel))
   413→		crc.Write(scratch[:1])
   414→		if _, err := w.Write(scratch[:1]); err != nil {
   415→			return err
   416→		}
   417→		selBytes := []byte(sel)
   418→		crc.Write(selBytes)
   419→		if _, err := w.Write(selBytes); err != nil {
   420→			return err
   421→		}
   422→	}
   423→
   424→	// Value (4 bytes, float32 bits).
   425→	binary.LittleEndian.PutUint32(scratch[:4], math.Float32bits(float32(msg.Value)))
   426→	crc.Write(scratch[:4])
   427→	if _, err := w.Write(scratch[:4]); err != nil {
   428→		return err
   429→	}
   430→
   431→	// CRC32 (4 bytes).
   432→	binary.LittleEndian.PutUint32(scratch[:4], crc.Sum32())
   433→	_, err := w.Write(scratch[:4])
   434→	return err
   435→}
   436→
   437→// readWALRecord reads one WAL record from the reader.
   438→// Returns (nil, nil) on clean EOF. Returns error on data corruption.
   439→// A CRC mismatch indicates a truncated trailing record (expected on crash).
   440→func readWALRecord(r io.Reader) (*WALMessage, error) {
   441→	var magic uint32
   442→	if err := binary.Read(r, binary.LittleEndian, &magic); err != nil {
   443→		if err == io.EOF {
   444→			return nil, nil // Clean EOF
   445→		}
   446→		return nil, fmt.Errorf("read record magic: %w", err)
   447→	}
   448→
   449→	if magic != walRecordMagic {
   450→		return nil, fmt.Errorf("invalid record magic 0x%08X (expected 0x%08X)", magic, walRecordMagic)
   451→	}
   452→
   453→	var payloadLen uint32
   454→	if err := binary.Read(r, binary.LittleEndian, &payloadLen); err != nil {
   455→		return nil, fmt.Errorf("read payload length: %w", err)
   456→	}
   457→
   458→	if payloadLen > 1<<20 { // 1 MB sanity limit
   459→		return nil, fmt.Errorf("record payload too large: %d bytes", payloadLen)
   460→	}
   461→
   462→	payload := make([]byte, payloadLen)
   463→	if _, err := io.ReadFull(r, payload); err != nil {
   464→		return nil, fmt.Errorf("read payload: %w", err)
   465→	}
   466→
   467→	var storedCRC uint32
   468→	if err := binary.Read(r, binary.LittleEndian, &storedCRC); err != nil {
   469→		return nil, fmt.Errorf("read CRC: %w", err)
   470→	}
   471→
   472→	if crc32.ChecksumIEEE(payload) != storedCRC {
   473→		return nil, fmt.Errorf("CRC mismatch (truncated write or corruption)")
   474→	}
   475→
   476→	return parseWALPayload(payload)
   477→}
   478→
   479→// parseWALPayload decodes a binary payload into a WALMessage.
   480→func parseWALPayload(payload []byte) (*WALMessage, error) {
   481→	if len(payload) < 8+2+1+4 {
   482→		return nil, fmt.Errorf("payload too short: %d bytes", len(payload))
   483→	}
   484→
   485→	offset := 0
   486→
   487→	// Timestamp (8 bytes)
   488→	ts := int64(binary.LittleEndian.Uint64(payload[offset : offset+8]))
   489→	offset += 8
   490→
   491→	// Metric name (2-byte length + bytes)
   492→	if offset+2 > len(payload) {
   493→		return nil, fmt.Errorf("metric name length overflows payload")
   494→	}
   495→	mLen := int(binary.LittleEndian.Uint16(payload[offset : offset+2]))
   496→	offset += 2
   497→
   498→	if offset+mLen > len(payload) {
   499→		return nil, fmt.Errorf("metric name overflows payload")
   500→	}
   501→	metricName := string(payload[offset : offset+mLen])
   502→	offset += mLen
   503→
   504→	// Selector count (1 byte)
   505→	if offset >= len(payload) {
   506→		return nil, fmt.Errorf("selector count overflows payload")
   507→	}
   508→	selCount := int(payload[offset])
   509→	offset++
   510→
   511→	selectors := make([]string, selCount)
   512→	for i := range selCount {
   513→		if offset >= len(payload) {
   514→			return nil, fmt.Errorf("selector[%d] length overflows payload", i)
   515→		}
   516→		sLen := int(payload[offset])
   517→		offset++
   518→
   519→		if offset+sLen > len(payload) {
   520→			return nil, fmt.Errorf("selector[%d] data overflows payload", i)
   521→		}
   522→		selectors[i] = string(payload[offset : offset+sLen])
   523→		offset += sLen
   524→	}
   525→
   526→	// Value (4 bytes, float32 bits)
   527→	if offset+4 > len(payload) {
   528→		return nil, fmt.Errorf("value overflows payload")
   529→	}
   530→	bits := binary.LittleEndian.Uint32(payload[offset : offset+4])
   531→	value := schema.Float(math.Float32frombits(bits))
   532→
   533→	return &WALMessage{
   534→		MetricName: metricName,
   535→		Timestamp:  ts,
   536→		Selector:   selectors,
   537→		Value:      value,
   538→	}, nil
   539→}
   540→
   541→// loadWALFile reads a WAL file and replays all valid records into the Level tree.
   542→// l is the host-level node. Corrupt or partial trailing records are silently skipped
   543→// (expected on crash). Records older than 'from' are skipped.
   544→func (l *Level) loadWALFile(m *MemoryStore, f *os.File, from int64) error {
   545→	br := bufio.NewReader(f)
   546→
   547→	// Verify file header magic.
   548→	var fileMagic uint32
   549→	if err := binary.Read(br, binary.LittleEndian, &fileMagic); err != nil {
   550→		if err == io.EOF {
   551→			return nil // Empty file, no data
   552→		}
   553→		return fmt.Errorf("[METRICSTORE]> WAL: read file header: %w", err)
   554→	}
   555→
   556→	if fileMagic != walFileMagic {
   557→		return fmt.Errorf("[METRICSTORE]> WAL: invalid file magic 0x%08X (expected 0x%08X)", fileMagic, walFileMagic)
   558→	}
   559→
   560→	// Cache level lookups to avoid repeated tree traversal.
   561→	lvlCache := make(map[string]*Level)
   562→
   563→	for {
   564→		msg, err := readWALRecord(br)
   565→		if err != nil {
   566→			// Truncated trailing record is expected after a crash; stop replaying.
   567→			cclog.Debugf("[METRICSTORE]> WAL: stopping replay at corrupted/partial record: %v", err)
   568→			break
   569→		}
   570→		if msg == nil {
   571→			break // Clean EOF
   572→		}
   573→
   574→		if msg.Timestamp < from {
   575→			continue // Older than retention window
   576→		}
   577→
   578→		minfo, ok := m.Metrics[msg.MetricName]
   579→		if !ok {
   580→			continue // Unknown metric (config may have changed)
   581→		}
   582→
   583→		// Cache key is the null-separated selector path.
   584→		cacheKey := joinSelector(msg.Selector)
   585→		lvl, ok := lvlCache[cacheKey]
   586→		if !ok {
   587→			lvl = l.findLevelOrCreate(msg.Selector, len(m.Metrics))
   588→			lvlCache[cacheKey] = lvl
   589→		}
   590→
   591→		// Write directly to the buffer, same as WriteToLevel but without the
   592→		// global level lookup (we already have the right level).
   593→		lvl.lock.Lock()
   594→		b := lvl.metrics[minfo.offset]
   595→		if b == nil {
   596→			b = newBuffer(msg.Timestamp, minfo.Frequency)
   597→			lvl.metrics[minfo.offset] = b
   598→		}
   599→		nb, writeErr := b.write(msg.Timestamp, msg.Value)
   600→		if writeErr == nil && b != nb {
   601→			lvl.metrics[minfo.offset] = nb
   602→		}
   603→		// Ignore write errors for timestamps before buffer start (can happen when
   604→		// replaying WAL entries that predate a loaded snapshot's start time).
   605→		lvl.lock.Unlock()
   606→	}
   607→
   608→	return nil
   609→}
   610→
   611→// joinSelector builds a cache key from a selector slice using null bytes as separators.
   612→func joinSelector(sel []string) string {
   613→	if len(sel) == 0 {
   614→		return ""
   615→	}
   616→	var result strings.Builder
   617→	result.WriteString(sel[0])
   618→	for i := 1; i < len(sel); i++ {
   619→		result.WriteString("\x00" + sel[i])
   620→	}
   621→	return result.String()
   622→}
   623→
   624→// ToCheckpointWAL writes binary snapshot files for all hosts in parallel.
   625→// Returns the number of files written, the list of host directories that were
   626→// successfully checkpointed (for WAL rotation), and any errors.
   627→func (m *MemoryStore) ToCheckpointWAL(dir string, from, to int64) (int, []string, error) {
   628→	// Collect all cluster/host pairs.
   629→	m.root.lock.RLock()
   630→	totalHosts := 0
   631→	for _, l1 := range m.root.children {
   632→		l1.lock.RLock()
   633→		totalHosts += len(l1.children)
   634→		l1.lock.RUnlock()
   635→	}
   636→	m.root.lock.RUnlock()
   637→
   638→	levels := make([]*Level, 0, totalHosts)
   639→	selectors := make([][]string, 0, totalHosts)
   640→
   641→	m.root.lock.RLock()
   642→	for sel1, l1 := range m.root.children {
   643→		l1.lock.RLock()
   644→		for sel2, l2 := range l1.children {
   645→			levels = append(levels, l2)
   646→			selectors = append(selectors, []string{sel1, sel2})
   647→		}
   648→		l1.lock.RUnlock()
   649→	}
   650→	m.root.lock.RUnlock()
   651→
   652→	type workItem struct {
   653→		level    *Level
   654→		hostDir  string
   655→		selector []string
   656→	}
   657→
   658→	n, errs := int32(0), int32(0)
   659→	var successDirs []string
   660→	var successMu sync.Mutex
   661→
   662→	var wg sync.WaitGroup
   663→	wg.Add(Keys.NumWorkers)
   664→	work := make(chan workItem, Keys.NumWorkers*2)
   665→
   666→	for range Keys.NumWorkers {
   667→		go func() {
   668→			defer wg.Done()
   669→			for wi := range work {
   670→				err := wi.level.toCheckpointBinary(wi.hostDir, from, to, m)
   671→				if err != nil {
   672→					if err == ErrNoNewArchiveData {
   673→						continue
   674→					}
   675→					cclog.Errorf("[METRICSTORE]> binary checkpoint error for %s: %v", wi.hostDir, err)
   676→					atomic.AddInt32(&errs, 1)
   677→				} else {
   678→					atomic.AddInt32(&n, 1)
   679→					successMu.Lock()
   680→					successDirs = append(successDirs, wi.hostDir)
   681→					successMu.Unlock()
   682→				}
   683→			}
   684→		}()
   685→	}
   686→
   687→	for i := range levels {
   688→		hostDir := path.Join(dir, path.Join(selectors[i]...))
   689→		work <- workItem{
   690→			level:    levels[i],
   691→			hostDir:  hostDir,
   692→			selector: selectors[i],
   693→		}
   694→	}
   695→	close(work)
   696→	wg.Wait()
   697→
   698→	if errs > 0 {
   699→		return int(n), successDirs, fmt.Errorf("[METRICSTORE]> %d errors during binary checkpoint (%d successes)", errs, n)
   700→	}
   701→	return int(n), successDirs, nil
   702→}
   703→
   704→// toCheckpointBinary writes a binary snapshot file for a single host-level node.
   705→// Uses atomic rename (write to .tmp then rename) to avoid partial reads on crash.
   706→func (l *Level) toCheckpointBinary(dir string, from, to int64, m *MemoryStore) error {
   707→	cf, err := l.toCheckpointFile(from, to, m)
   708→	if err != nil {
   709→		return err
   710→	}
   711→	if cf == nil {
   712→		return ErrNoNewArchiveData
   713→	}
   714→
   715→	if err := os.MkdirAll(dir, CheckpointDirPerms); err != nil {
   716→		return fmt.Errorf("mkdir %s: %w", dir, err)
   717→	}
   718→
   719→	// Write to a temp file first, then rename (atomic on POSIX).
   720→	tmpPath := path.Join(dir, fmt.Sprintf("%d.bin.tmp", from))
   721→	finalPath := path.Join(dir, fmt.Sprintf("%d.bin", from))
   722→
   723→	f, err := os.OpenFile(tmpPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
   724→	if err != nil {
   725→		return fmt.Errorf("open binary snapshot %s: %w", tmpPath, err)
   726→	}
   727→
   728→	bw := bufio.NewWriter(f)
   729→	if err := writeBinarySnapshotFile(bw, cf); err != nil {
   730→		f.Close()
   731→		os.Remove(tmpPath)
   732→		return fmt.Errorf("write binary snapshot: %w", err)
   733→	}
   734→	if err := bw.Flush(); err != nil {
   735→		f.Close()
   736→		os.Remove(tmpPath)
   737→		return err
   738→	}
   739→	f.Close()
   740→
   741→	return os.Rename(tmpPath, finalPath)
   742→}
   743→
   744→// writeBinarySnapshotFile writes the binary snapshot file header and level tree.
   745→func writeBinarySnapshotFile(w io.Writer, cf *CheckpointFile) error {
   746→	if err := binary.Write(w, binary.LittleEndian, snapFileMagic); err != nil {
   747→		return err
   748→	}
   749→	if err := binary.Write(w, binary.LittleEndian, cf.From); err != nil {
   750→		return err
   751→	}
   752→	if err := binary.Write(w, binary.LittleEndian, cf.To); err != nil {
   753→		return err
   754→	}
   755→	return writeBinaryLevel(w, cf)
   756→}
   757→
   758→// writeBinaryLevel recursively writes a CheckpointFile level in binary format.
   759→func writeBinaryLevel(w io.Writer, cf *CheckpointFile) error {
   760→	if err := binary.Write(w, binary.LittleEndian, uint32(len(cf.Metrics))); err != nil {
   761→		return err
   762→	}
   763→
   764→	for name, metric := range cf.Metrics {
   765→		if err := writeString16(w, name); err != nil {
   766→			return err
   767→		}
   768→		if err := binary.Write(w, binary.LittleEndian, metric.Frequency); err != nil {
   769→			return err
   770→		}
   771→		if err := binary.Write(w, binary.LittleEndian, metric.Start); err != nil {
   772→			return err
   773→		}
   774→		if err := binary.Write(w, binary.LittleEndian, uint32(len(metric.Data))); err != nil {
   775→			return err
   776→		}
   777→		for _, v := range metric.Data {
   778→			if err := binary.Write(w, binary.LittleEndian, math.Float32bits(float32(v))); err != nil {
   779→				return err
   780→			}
   781→		}
   782→	}
   783→
   784→	if err := binary.Write(w, binary.LittleEndian, uint32(len(cf.Children))); err != nil {
   785→		return err
   786→	}
   787→
   788→	for name, child := range cf.Children {
   789→		if err := writeString16(w, name); err != nil {
   790→			return err
   791→		}
   792→		if err := writeBinaryLevel(w, child); err != nil {
   793→			return err
   794→		}
   795→	}
   796→
   797→	return nil
   798→}
   799→
   800→// writeString16 writes a 2-byte length-prefixed string to w.
   801→func writeString16(w io.Writer, s string) error {
   802→	if err := binary.Write(w, binary.LittleEndian, uint16(len(s))); err != nil {
   803→		return err
   804→	}
   805→	_, err := io.WriteString(w, s)
   806→	return err
   807→}
   808→
   809→// loadBinaryFile reads a binary snapshot file and loads data into the Level tree.
   810→// The retention check (from) is applied to the file's 'to' timestamp.
   811→func (l *Level) loadBinaryFile(m *MemoryStore, f *os.File, from int64) error {
   812→	br := bufio.NewReader(f)
   813→
   814→	var magic uint32
   815→	if err := binary.Read(br, binary.LittleEndian, &magic); err != nil {
   816→		return fmt.Errorf("[METRICSTORE]> binary snapshot: read magic: %w", err)
   817→	}
   818→	if magic != snapFileMagic {
   819→		return fmt.Errorf("[METRICSTORE]> binary snapshot: invalid magic 0x%08X (expected 0x%08X)", magic, snapFileMagic)
   820→	}
   821→
   822→	var fileFrom, fileTo int64
   823→	if err := binary.Read(br, binary.LittleEndian, &fileFrom); err != nil {
   824→		return fmt.Errorf("[METRICSTORE]> binary snapshot: read from: %w", err)
   825→	}
   826→	if err := binary.Read(br, binary.LittleEndian, &fileTo); err != nil {
   827→		return fmt.Errorf("[METRICSTORE]> binary snapshot: read to: %w", err)
   828→	}
   829→
   830→	if fileTo != 0 && fileTo < from {
   831→		return nil // File is older than retention window, skip it
   832→	}
   833→
   834→	cf, err := readBinaryLevel(br)
   835→	if err != nil {
   836→		return fmt.Errorf("[METRICSTORE]> binary snapshot: read level tree: %w", err)
   837→	}
   838→	cf.From = fileFrom
   839→	cf.To = fileTo
   840→
   841→	return l.loadFile(cf, m)
   842→}
   843→
   844→// readBinaryLevel recursively reads a level from the binary snapshot format.
   845→func readBinaryLevel(r io.Reader) (*CheckpointFile, error) {
   846→	cf := &CheckpointFile{
   847→		Metrics:  make(map[string]*CheckpointMetrics),
   848→		Children: make(map[string]*CheckpointFile),
   849→	}
   850→
   851→	var numMetrics uint32
   852→	if err := binary.Read(r, binary.LittleEndian, &numMetrics); err != nil {
   853→		return nil, fmt.Errorf("read num_metrics: %w", err)
   854→	}
   855→
   856→	for range numMetrics {
   857→		name, err := readString16(r)
   858→		if err != nil {
   859→			return nil, fmt.Errorf("read metric name: %w", err)
   860→		}
   861→
   862→		var freq, start int64
   863→		if err := binary.Read(r, binary.LittleEndian, &freq); err != nil {
   864→			return nil, fmt.Errorf("read frequency for %s: %w", name, err)
   865→		}
   866→		if err := binary.Read(r, binary.LittleEndian, &start); err != nil {
   867→			return nil, fmt.Errorf("read start for %s: %w", name, err)
   868→		}
   869→
   870→		var numValues uint32
   871→		if err := binary.Read(r, binary.LittleEndian, &numValues); err != nil {
   872→			return nil, fmt.Errorf("read num_values for %s: %w", name, err)
   873→		}
   874→
   875→		data := make([]schema.Float, numValues)
   876→		for i := range numValues {
   877→			var bits uint32
   878→			if err := binary.Read(r, binary.LittleEndian, &bits); err != nil {
   879→				return nil, fmt.Errorf("read value[%d] for %s: %w", i, name, err)
   880→			}
   881→			data[i] = schema.Float(math.Float32frombits(bits))
   882→		}
   883→
   884→		cf.Metrics[name] = &CheckpointMetrics{
   885→			Frequency: freq,
   886→			Start:     start,
   887→			Data:      data,
   888→		}
   889→	}
   890→
   891→	var numChildren uint32
   892→	if err := binary.Read(r, binary.LittleEndian, &numChildren); err != nil {
   893→		return nil, fmt.Errorf("read num_children: %w", err)
   894→	}
   895→
   896→	for range numChildren {
   897→		childName, err := readString16(r)
   898→		if err != nil {
   899→			return nil, fmt.Errorf("read child name: %w", err)
   900→		}
   901→
   902→		child, err := readBinaryLevel(r)
   903→		if err != nil {
   904→			return nil, fmt.Errorf("read child %s: %w", childName, err)
   905→		}
   906→		cf.Children[childName] = child
   907→	}
   908→
   909→	return cf, nil
   910→}
   911→
   912→// readString16 reads a 2-byte length-prefixed string from r.
   913→func readString16(r io.Reader) (string, error) {
   914→	var sLen uint16
   915→	if err := binary.Read(r, binary.LittleEndian, &sLen); err != nil {
   916→		return "", err
   917→	}
   918→	buf := make([]byte, sLen)
   919→	if _, err := io.ReadFull(r, buf); err != nil {
   920→		return "", err
   921→	}
   922→	return string(buf), nil
   923→}
   924→


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// This file implements ingestion of InfluxDB line-protocol metric data received
     7→// over NATS. Each line encodes one metric sample with the following structure:
     8→//
     9→//	<measurement>[,cluster=<c>][,hostname=<h>][,type=<t>][,type-id=<id>][,subtype=<s>][,stype-id=<id>] value=<v> [<timestamp>]
    10→//
    11→// The measurement name identifies the metric (e.g. "cpu_load"). Tags provide
    12→// routing information (cluster, host) and optional sub-device selectors (type,
    13→// subtype). Only one field is expected per line: "value".
    14→//
    15→// After decoding, each sample is:
    16→//  1. Written to the in-memory store via ms.WriteToLevel.
    17→//  2. If the checkpoint format is "wal", also forwarded to the WAL staging
    18→//     goroutine via the WALMessages channel for durable write-ahead logging.
    19→package metricstore
    20→
    21→import (
    22→	"bytes"
    23→	"context"
    24→	"fmt"
    25→	"sync"
    26→	"sync/atomic"
    27→	"time"
    28→
    29→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    30→	"github.com/ClusterCockpit/cc-lib/v2/nats"
    31→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    32→	"github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
    33→)
    34→
    35→// ReceiveNats subscribes to all configured NATS subjects and feeds incoming
    36→// line-protocol messages into the MemoryStore.
    37→//
    38→// When workers > 1 a pool of goroutines drains a shared channel so that
    39→// multiple messages can be decoded in parallel. With workers == 1 the NATS
    40→// callback decodes inline (no channel overhead, lower latency).
    41→//
    42→// The function blocks until ctx is cancelled and all worker goroutines have
    43→// finished. It returns nil when the NATS client is not configured; callers
    44→// should treat that as a no-op rather than an error.
    45→func ReceiveNats(ms *MemoryStore,
    46→	workers int,
    47→	ctx context.Context,
    48→) error {
    49→	nc := nats.GetClient()
    50→
    51→	if nc == nil {
    52→		cclog.Warn("NATS client not initialized")
    53→		return nil
    54→	}
    55→
    56→	var wg sync.WaitGroup
    57→	msgs := make(chan []byte, max(workers*256, 8192))
    58→
    59→	for _, sc := range *Keys.Subscriptions {
    60→		clusterTag := sc.ClusterTag
    61→		if workers > 1 {
    62→			wg.Add(workers)
    63→
    64→			for range workers {
    65→				go func() {
    66→					defer wg.Done()
    67→					for m := range msgs {
    68→						dec := lineprotocol.NewDecoderWithBytes(m)
    69→						if err := DecodeLine(dec, ms, clusterTag); err != nil {
    70→							cclog.Errorf("error: %s", err.Error())
    71→						}
    72→					}
    73→				}()
    74→			}
    75→
    76→			nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
    77→				select {
    78→				case msgs <- data:
    79→				case <-ctx.Done():
    80→				}
    81→			})
    82→		} else {
    83→			nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
    84→				dec := lineprotocol.NewDecoderWithBytes(data)
    85→				if err := DecodeLine(dec, ms, clusterTag); err != nil {
    86→					cclog.Errorf("error: %s", err.Error())
    87→				}
    88→			})
    89→		}
    90→		cclog.Infof("NATS subscription to '%s' established", sc.SubscribeTo)
    91→	}
    92→
    93→	go func() {
    94→		<-ctx.Done()
    95→		close(msgs)
    96→	}()
    97→
    98→	wg.Wait()
    99→
   100→	return nil
   101→}
   102→
   103→// reorder prepends prefix to buf in-place when buf has enough spare capacity,
   104→// avoiding an allocation. Falls back to a regular append otherwise.
   105→//
   106→// It is used to assemble the "type<type-id>" and "subtype<stype-id>" selector
   107→// strings when the type tag arrives before the type-id tag in the line, so the
   108→// two byte slices need to be concatenated in tag-declaration order regardless
   109→// of wire order.
   110→func reorder(buf, prefix []byte) []byte {
   111→	n := len(prefix)
   112→	m := len(buf)
   113→	if cap(buf) < m+n {
   114→		return append(prefix[:n:n], buf...)
   115→	} else {
   116→		buf = buf[:n+m]
   117→		for i := m - 1; i >= 0; i-- {
   118→			buf[i+n] = buf[i]
   119→		}
   120→		for i := range n {
   121→			buf[i] = prefix[i]
   122→		}
   123→		return buf
   124→	}
   125→}
   126→
   127→// walDropped counts WAL messages dropped due to a full channel.
   128→// Logged periodically; not critical since binary snapshots capture the data.
   129→var walDropped atomic.Int64
   130→
   131→// decodeState holds the per-call scratch buffers used by DecodeLine.
   132→// Instances are recycled via decodeStatePool to avoid repeated allocations
   133→// during high-throughput ingestion.
   134→type decodeState struct {
   135→	// metricBuf holds a copy of the current measurement name (line-protocol
   136→	// measurement field). Copied because dec.Measurement() returns a slice
   137→	// that is invalidated by the next decoder call.
   138→	metricBuf []byte
   139→
   140→	// selector is the sub-device path passed to WriteToLevel and WALMessage
   141→	// (e.g. ["socket0"] or ["socket0", "memctrl1"]). Reused across lines.
   142→	selector []string
   143→
   144→	// typeBuf accumulates the concatenated "type"+"type-id" tag value for the
   145→	// current line. Reset at the start of each line's tag-decode loop.
   146→	typeBuf []byte
   147→
   148→	// subTypeBuf accumulates the concatenated "subtype"+"stype-id" tag value.
   149→	// Reset at the start of each line's tag-decode loop.
   150→	subTypeBuf []byte
   151→
   152→	// prevTypeBytes / prevTypeStr cache the last seen typeBuf content and its
   153→	// string conversion. Because consecutive lines in a batch typically address
   154→	// the same sub-device, the cache hit rate is very high and avoids
   155→	// repeated []byte→string allocations.
   156→	prevTypeBytes []byte
   157→	prevTypeStr   string
   158→
   159→	// prevSubTypeBytes / prevSubTypeStr are the same cache for the subtype.
   160→	prevSubTypeBytes []byte
   161→	prevSubTypeStr   string
   162→}
   163→
   164→// decodeStatePool recycles decodeState values across DecodeLine calls to
   165→// reduce GC pressure during sustained metric ingestion.
   166→var decodeStatePool = sync.Pool{
   167→	New: func() any {
   168→		return &decodeState{
   169→			metricBuf:  make([]byte, 0, 16),
   170→			selector:   make([]string, 0, 4),
   171→			typeBuf:    make([]byte, 0, 16),
   172→			subTypeBuf: make([]byte, 0, 16),
   173→		}
   174→	},
   175→}
   176→
   177→// DecodeLine reads all lines from dec (InfluxDB line-protocol) and writes each
   178→// decoded metric sample into ms.
   179→//
   180→// clusterDefault is used as the cluster name for lines that do not carry a
   181→// "cluster" tag. Callers typically supply the ClusterTag value from the NATS
   182→// subscription configuration.
   183→//
   184→// Performance notes:
   185→//   - A decodeState is obtained from decodeStatePool to reuse scratch buffers.
   186→//   - The Level pointer (host-level node in the metric tree) is cached across
   187→//     consecutive lines that share the same cluster+host pair to avoid
   188→//     repeated lock acquisitions on the root and cluster levels.
   189→//   - []byte→string conversions for type/subtype selectors are cached via
   190→//     prevType*/prevSubType* fields because batches typically repeat the same
   191→//     sub-device identifiers.
   192→//   - Timestamp parsing tries Second precision first; if that fails it retries
   193→//     Millisecond, Microsecond, and Nanosecond in turn. A missing timestamp
   194→//     falls back to time.Now().
   195→//
   196→// When the checkpoint format is "wal" each successfully decoded sample is also
   197→// sent to WALMessages so the WAL staging goroutine can persist it durably
   198→// before the next binary snapshot.
   199→func DecodeLine(dec *lineprotocol.Decoder,
   200→	ms *MemoryStore,
   201→	clusterDefault string,
   202→) error {
   203→	// Reduce allocations in loop:
   204→	t := time.Now()
   205→	metric := Metric{}
   206→	st := decodeStatePool.Get().(*decodeState)
   207→	defer decodeStatePool.Put(st)
   208→
   209→	// Optimize for the case where all lines in a "batch" are about the same
   210→	// cluster and host. By using `WriteToLevel` (level = host), we do not need
   211→	// to take the root- and cluster-level lock as often.
   212→	var lvl *Level = nil
   213→	prevCluster, prevHost := "", ""
   214→
   215→	var ok bool
   216→	for dec.Next() {
   217→		rawmeasurement, err := dec.Measurement()
   218→		if err != nil {
   219→			return err
   220→		}
   221→
   222→		// Needs to be copied because another call to dec.* would
   223→		// invalidate the returned slice.
   224→		st.metricBuf = append(st.metricBuf[:0], rawmeasurement...)
   225→
   226→		// The go compiler optimizes map[string(byteslice)] lookups:
   227→		metric.MetricConfig, ok = ms.Metrics[string(rawmeasurement)]
   228→		if !ok {
   229→			continue
   230→		}
   231→
   232→		st.typeBuf, st.subTypeBuf = st.typeBuf[:0], st.subTypeBuf[:0]
   233→		cluster, host := clusterDefault, ""
   234→		for {
   235→			key, val, err := dec.NextTag()
   236→			if err != nil {
   237→				return err
   238→			}
   239→			if key == nil {
   240→				break
   241→			}
   242→
   243→			// The go compiler optimizes string([]byte{...}) == "...":
   244→			switch string(key) {
   245→			case "cluster":
   246→				if string(val) == prevCluster {
   247→					cluster = prevCluster
   248→				} else {
   249→					cluster = string(val)
   250→					lvl = nil
   251→				}
   252→			case "hostname", "host":
   253→				if string(val) == prevHost {
   254→					host = prevHost
   255→				} else {
   256→					host = string(val)
   257→					lvl = nil
   258→				}
   259→			case "type":
   260→				if string(val) == "node" {
   261→					break
   262→				}
   263→
   264→				// We cannot be sure that the "type" tag comes before the "type-id" tag:
   265→				if len(st.typeBuf) == 0 {
   266→					st.typeBuf = append(st.typeBuf, val...)
   267→				} else {
   268→					st.typeBuf = reorder(st.typeBuf, val)
   269→				}
   270→			case "type-id":
   271→				st.typeBuf = append(st.typeBuf, val...)
   272→			case "subtype":
   273→				// We cannot be sure that the "subtype" tag comes before the "stype-id" tag:
   274→				if len(st.subTypeBuf) == 0 {
   275→					st.subTypeBuf = append(st.subTypeBuf, val...)
   276→				} else {
   277→					st.subTypeBuf = reorder(st.subTypeBuf, val)
   278→				}
   279→			case "stype-id":
   280→				st.subTypeBuf = append(st.subTypeBuf, val...)
   281→			default:
   282→			}
   283→		}
   284→
   285→		// If the cluster or host changed, the lvl was set to nil
   286→		if lvl == nil {
   287→			st.selector = st.selector[:2]
   288→			st.selector[0], st.selector[1] = cluster, host
   289→			lvl = ms.GetLevel(st.selector)
   290→			prevCluster, prevHost = cluster, host
   291→		}
   292→
   293→		// subtypes: cache []byte→string conversions; messages in a batch typically
   294→		// share the same type/subtype so the hit rate is very high.
   295→		st.selector = st.selector[:0]
   296→		if len(st.typeBuf) > 0 {
   297→			if !bytes.Equal(st.typeBuf, st.prevTypeBytes) {
   298→				st.prevTypeBytes = append(st.prevTypeBytes[:0], st.typeBuf...)
   299→				st.prevTypeStr = string(st.typeBuf)
   300→			}
   301→			st.selector = append(st.selector, st.prevTypeStr)
   302→			if len(st.subTypeBuf) > 0 {
   303→				if !bytes.Equal(st.subTypeBuf, st.prevSubTypeBytes) {
   304→					st.prevSubTypeBytes = append(st.prevSubTypeBytes[:0], st.subTypeBuf...)
   305→					st.prevSubTypeStr = string(st.subTypeBuf)
   306→				}
   307→				st.selector = append(st.selector, st.prevSubTypeStr)
   308→			}
   309→		}
   310→
   311→		for {
   312→			key, val, err := dec.NextField()
   313→			if err != nil {
   314→				return err
   315→			}
   316→
   317→			if key == nil {
   318→				break
   319→			}
   320→
   321→			if string(key) != "value" {
   322→				return fmt.Errorf("host %s: unknown field: '%s' (value: %#v)", host, string(key), val)
   323→			}
   324→
   325→			if val.Kind() == lineprotocol.Float {
   326→				metric.Value = schema.Float(val.FloatV())
   327→			} else if val.Kind() == lineprotocol.Int {
   328→				metric.Value = schema.Float(val.IntV())
   329→			} else if val.Kind() == lineprotocol.Uint {
   330→				metric.Value = schema.Float(val.UintV())
   331→			} else {
   332→				return fmt.Errorf("host %s: unsupported value type in message: %s", host, val.Kind().String())
   333→			}
   334→		}
   335→
   336→		if t, err = dec.Time(lineprotocol.Second, t); err != nil {
   337→			t = time.Now()
   338→			if t, err = dec.Time(lineprotocol.Millisecond, t); err != nil {
   339→				t = time.Now()
   340→				if t, err = dec.Time(lineprotocol.Microsecond, t); err != nil {
   341→					t = time.Now()
   342→					if t, err = dec.Time(lineprotocol.Nanosecond, t); err != nil {
   343→						return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
   344→					}
   345→				}
   346→			}
   347→		}
   348→
   349→		if err != nil {
   350→			return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
   351→		}
   352→
   353→		time := t.Unix()
   354→
   355→		if Keys.Checkpoints.FileFormat == "wal" {
   356→			msg := &WALMessage{
   357→				MetricName: string(st.metricBuf),
   358→				Cluster:    cluster,
   359→				Node:       host,
   360→				Selector:   append([]string{}, st.selector...),
   361→				Value:      metric.Value,
   362→				Timestamp:  time,
   363→			}
   364→			if !SendWALMessage(msg) {
   365→				// WAL shard channel full — metric is written to memory store but not WAL.
   366→				// Next binary snapshot will capture it.
   367→				if dropped := walDropped.Add(1); dropped%10000 == 1 {
   368→					cclog.Warnf("[METRICSTORE]> WAL shard channel full, dropped %d messages (data safe in memory, next snapshot will capture)", dropped)
   369→				}
   370→			}
   371→		}
   372→
   373→		if err := ms.WriteToLevel(lvl, st.selector, time, []Metric{metric}); err != nil {
   374→			cclog.Warnf("write error for host %s metric %s at ts %d: %s", host, string(st.metricBuf), time, err.Error())
   375→			continue
   376→		}
   377→	}
   378→	return nil
   379→}
   380→