Implement the following plan:

# Fix WAL Channel Saturation (3M+ messages dropped)

## Context

After the initial slow-consumer fixes (multi-worker decode, non-blocking WAL send, 65K channel buffer, 256-batch drain), the WAL channel is still permanently full. 3M+ messages are being dropped — the single `WALStaging` goroutine fundamentally cannot keep up with `NumWorkers` concurrent decode goroutines producing WAL messages.

## Root Cause

The single `WALStaging` goroutine is the bottleneck. For every message it must:
1. `path.Join()` to build hostDir string — **allocates**
2. Map lookup in `hostFiles`
3. `buildWALPayload()` — **allocates a new `[]byte`** per message
4. `writeWALRecord()` — **allocates another `[]byte`** (record = magic+len+payload+crc)
5. `bufio.Write()` — copies into buffer, occasionally flushes to disk

With `NumWorkers` producers (e.g., 5-10 goroutines) each decoding batches of metrics, the single consumer cannot match. The 65K channel buffer fills in seconds, then every message is dropped.

## Plan: Sharded WAL Writers

Replace the single `WALStaging` goroutine with **N sharded writer goroutines**, each owning a subset of hosts. Messages are routed to shards by hashing `cluster+node`. This:
- Parallelizes serialization and I/O across goroutines
- Keeps per-host writes serial (required for correct WAL files)
- Eliminates the single-goroutine bottleneck entirely

Additionally, eliminate per-message heap allocations in `buildWALPayload`/`writeWALRecord` by encoding directly into the `bufio.Writer`.

### Change 1: Sharded WAL writer goroutines
**File:** `pkg/metricstore/walCheckpoint.go`

Replace the single `WALMessages` channel + `WALStaging()` goroutine with:

```go
const walShards = 16  // or NumWorkers

// Per-shard channel + goroutine
var walShardChs []chan *WALMessage

func walShard(msg *WALMessage) int {
    // FNV-1a hash of cluster+node, mod shard count
    h := fnv.New32a()
    h.Write([]byte(msg.Cluster))
    h.Write([]byte{0})
    h.Write([]byte(msg.Node))
    return int(h.Sum32()) % len(walShardChs)
}
```

Each shard goroutine:
- Owns its own `hostFiles` map (no contention)
- Drains its channel with the existing batch-drain pattern
- Handles rotate requests for its hosts

**Shard count:** Use `NumWorkers` (already configured, sensible default).

**Channel buffer per shard:** `65536 / numShards` — same total memory, but each shard only buffers its hosts.

### Change 2: Eliminate per-message allocations
**File:** `pkg/metricstore/walCheckpoint.go`

Replace `buildWALPayload()` → `writeWALRecord()` (2 allocations per message) with `writeWALRecordDirect(w *bufio.Writer, msg *WALMessage)` that encodes directly into the `bufio.Writer` using a small stack-allocated scratch buffer:

```go
func writeWALRecordDirect(w *bufio.Writer, msg *WALMessage) error {
    // Build payload into a pooled or stack buffer, compute CRC inline,
    // write magic+len, then payload, then CRC — all directly to w.
    // Zero heap allocations in the hot path.
    var scratch [128]byte  // stack-allocated, covers most records
    // ... encode payload into scratch or a pooled []byte if >128 ...
}
```

Keep the old `writeWALRecord`/`buildWALPayload` for use in tests or non-hot paths, but the shard goroutines use the direct version.

### Change 3: Route WAL sends to shards from DecodeLine
**File:** `pkg/metricstore/lineprotocol.go:355-372`

Replace:
```go
select {
case WALMessages <- msg:
default:
    // dropped
}
```
With:
```go
shard := walShard(msg)
select {
case walShardChs[shard] <- msg:
default:
    // dropped
}
```

### Change 4: Shard-aware rotation
**File:** `pkg/metricstore/walCheckpoint.go`

`RotateWALFiles` must send rotate requests to the correct shard. Each shard has its own `walRotateCh`. The checkpoint goroutine hashes the hostDir to find the right shard and sends the rotate request there.

### Change 5: Shutdown
**File:** `pkg/metricstore/metricstore.go` (Shutdown function)

Close all shard channels instead of the single `WALMessages` channel. Each shard goroutine drains and flushes on close.

## Files to Modify

- `pkg/metricstore/walCheckpoint.go` — sharded goroutines, direct record writing, shard-aware rotation
- `pkg/metricstore/lineprotocol.go` — route WAL sends to shard channels
- `pkg/metricstore/metricstore.go` — `WALStaging()` call updated, `Shutdown()` closes shard channels

## Verification

1. `go build ./...`
2. `go test ./pkg/metricstore/...`
3. Deploy and verify:
   - WAL drop count stays at 0 (or near-zero during extreme bursts)
   - WAL files are correct (load a checkpoint, verify data)
   - No "WAL channel full" log messages
   - Memory usage roughly same (total channel buffer size unchanged)

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-ClusterCockpit-cc-backend/5eb20a94-f8df-4979-a0ae-2aa99bfad884.jsonl


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstore provides walCheckpoint.go: WAL-based checkpoint implementation.
     7→//
     8→// This replaces the Avro shadow tree with an append-only Write-Ahead Log (WAL)
     9→// per host, eliminating the extra memory overhead of the AvroStore and providing
    10→// truly continuous (per-write) crash safety.
    11→//
    12→// # Architecture
    13→//
    14→//	Metric write (DecodeLine)
    15→//	      │
    16→//	      ├─► WriteToLevel()    → main MemoryStore (unchanged)
    17→//	      │
    18→//	      └─► WALMessages channel
    19→//	               │
    20→//	               ▼
    21→//	        WALStaging goroutine
    22→//	               │
    23→//	               ▼
    24→//	        checkpoints/cluster/host/current.wal  (append-only, binary)
    25→//
    26→//	Periodic checkpoint (Checkpointing goroutine):
    27→//	  1. Write <timestamp>.bin snapshot (column-oriented, from main tree)
    28→//	  2. Signal WALStaging to truncate current.wal per host
    29→//
    30→//	On restart (FromCheckpoint):
    31→//	  1. Load most recent <timestamp>.bin snapshot
    32→//	  2. Replay current.wal (overwrite-safe: buffer.write handles duplicate timestamps)
    33→//
    34→// # WAL Record Format
    35→//
    36→//	[4B magic 0xCC1DA7A1][4B payload_len][payload][4B CRC32]
    37→//
    38→//	payload:
    39→//	  [8B timestamp int64]
    40→//	  [2B metric_name_len uint16][N metric name bytes]
    41→//	  [1B selector_count uint8]
    42→//	  per selector: [1B selector_len uint8][M selector bytes]
    43→//	  [4B value float32 bits]
    44→//
    45→// # Binary Snapshot Format
    46→//
    47→//	[4B magic 0xCC5B0001][8B from int64][8B to int64]
    48→//	Level tree (recursive):
    49→//	  [4B num_metrics uint32]
    50→//	  per metric:
    51→//	    [2B name_len uint16][N name bytes]
    52→//	    [8B frequency int64][8B start int64]
    53→//	    [4B num_values uint32][num_values × 4B float32]
    54→//	  [4B num_children uint32]
    55→//	  per child: [2B name_len uint16][N name bytes] + Level (recursive)
    56→package metricstore
    57→
    58→import (
    59→	"bufio"
    60→	"context"
    61→	"encoding/binary"
    62→	"fmt"
    63→	"hash/crc32"
    64→	"io"
    65→	"math"
    66→	"os"
    67→	"path"
    68→	"strings"
    69→	"sync"
    70→	"sync/atomic"
    71→
    72→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    73→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    74→)
    75→
    76→// Magic numbers for binary formats.
    77→const (
    78→	walFileMagic   = uint32(0xCC1DA701) // WAL file header magic
    79→	walRecordMagic = uint32(0xCC1DA7A1) // WAL record magic
    80→	snapFileMagic  = uint32(0xCC5B0001) // Binary snapshot magic
    81→)
    82→
    83→// WALMessages is the channel for sending metric writes to the WAL staging goroutine.
    84→// Buffered to allow burst writes without blocking the metric ingestion path.
    85→var WALMessages = make(chan *WALMessage, 65536)
    86→
    87→// walRotateCh is used by the checkpoint goroutine to request WAL file rotation
    88→// (close, delete, reopen) after a binary snapshot has been written.
    89→var walRotateCh = make(chan walRotateReq, 256)
    90→
    91→// WALMessage represents a single metric write to be appended to the WAL.
    92→// Cluster and Node are NOT stored in the WAL record (inferred from file path).
    93→type WALMessage struct {
    94→	MetricName string
    95→	Cluster    string
    96→	Node       string
    97→	Selector   []string
    98→	Value      schema.Float
    99→	Timestamp  int64
   100→}
   101→
   102→// walRotateReq requests WAL file rotation for a specific host directory.
   103→// The done channel is closed by the WAL goroutine when rotation is complete.
   104→type walRotateReq struct {
   105→	hostDir string
   106→	done    chan struct{}
   107→}
   108→
   109→// walFileState holds an open WAL file handle and buffered writer for one host directory.
   110→type walFileState struct {
   111→	f *os.File
   112→	w *bufio.Writer
   113→}
   114→
   115→// WALStaging starts a background goroutine that receives WALMessage items
   116→// and appends binary WAL records to per-host current.wal files.
   117→// Also handles WAL rotation requests from the checkpoint goroutine.
   118→func WALStaging(wg *sync.WaitGroup, ctx context.Context) {
   119→	wg.Go(func() {
   120→		if Keys.Checkpoints.FileFormat == "json" {
   121→			return
   122→		}
   123→
   124→		hostFiles := make(map[string]*walFileState)
   125→
   126→		defer func() {
   127→			for _, ws := range hostFiles {
   128→				if ws.f != nil {
   129→					ws.w.Flush()
   130→					ws.f.Close()
   131→				}
   132→			}
   133→		}()
   134→
   135→		getOrOpenWAL := func(hostDir string) *walFileState {
   136→			ws, ok := hostFiles[hostDir]
   137→			if ok {
   138→				return ws
   139→			}
   140→
   141→			if err := os.MkdirAll(hostDir, CheckpointDirPerms); err != nil {
   142→				cclog.Errorf("[METRICSTORE]> WAL: mkdir %s: %v", hostDir, err)
   143→				return nil
   144→			}
   145→
   146→			walPath := path.Join(hostDir, "current.wal")
   147→			f, err := os.OpenFile(walPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, CheckpointFilePerms)
   148→			if err != nil {
   149→				cclog.Errorf("[METRICSTORE]> WAL: open %s: %v", walPath, err)
   150→				return nil
   151→			}
   152→
   153→			w := bufio.NewWriter(f)
   154→
   155→			// Write file header magic if file is new (empty).
   156→			info, err := f.Stat()
   157→			if err == nil && info.Size() == 0 {
   158→				var hdr [4]byte
   159→				binary.LittleEndian.PutUint32(hdr[:], walFileMagic)
   160→				if _, err := w.Write(hdr[:]); err != nil {
   161→					cclog.Errorf("[METRICSTORE]> WAL: write header %s: %v", walPath, err)
   162→					f.Close()
   163→					return nil
   164→				}
   165→			}
   166→
   167→			ws = &walFileState{f: f, w: w}
   168→			hostFiles[hostDir] = ws
   169→			return ws
   170→		}
   171→
   172→		processMsg := func(msg *WALMessage) {
   173→			hostDir := path.Join(Keys.Checkpoints.RootDir, msg.Cluster, msg.Node)
   174→			ws := getOrOpenWAL(hostDir)
   175→			if ws == nil {
   176→				return
   177→			}
   178→			if err := writeWALRecord(ws.w, msg); err != nil {
   179→				cclog.Errorf("[METRICSTORE]> WAL: write record: %v", err)
   180→			}
   181→		}
   182→
   183→		processRotate := func(req walRotateReq) {
   184→			ws, ok := hostFiles[req.hostDir]
   185→			if ok && ws.f != nil {
   186→				ws.w.Flush()
   187→				ws.f.Close()
   188→				walPath := path.Join(req.hostDir, "current.wal")
   189→				if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
   190→					cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
   191→				}
   192→				delete(hostFiles, req.hostDir)
   193→			}
   194→			close(req.done)
   195→		}
   196→
   197→		drain := func() {
   198→			for {
   199→				select {
   200→				case msg, ok := <-WALMessages:
   201→					if !ok {
   202→						return
   203→					}
   204→					processMsg(msg)
   205→				case req := <-walRotateCh:
   206→					processRotate(req)
   207→				default:
   208→					// Flush all buffered writers after draining remaining messages.
   209→					for _, ws := range hostFiles {
   210→						if ws.f != nil {
   211→							ws.w.Flush()
   212→						}
   213→					}
   214→					return
   215→				}
   216→			}
   217→		}
   218→
   219→		for {
   220→			select {
   221→			case <-ctx.Done():
   222→				drain()
   223→				return
   224→			case msg, ok := <-WALMessages:
   225→				if !ok {
   226→					return
   227→				}
   228→				processMsg(msg)
   229→
   230→				// Drain up to 256 more messages without blocking to batch writes.
   231→				for range 256 {
   232→					select {
   233→					case msg, ok := <-WALMessages:
   234→						if !ok {
   235→							return
   236→						}
   237→						processMsg(msg)
   238→					case req := <-walRotateCh:
   239→						processRotate(req)
   240→					default:
   241→						goto flushed
   242→					}
   243→				}
   244→			flushed:
   245→				// Flush all buffered writers after processing the batch.
   246→				for _, ws := range hostFiles {
   247→					if ws.f != nil {
   248→						ws.w.Flush()
   249→					}
   250→				}
   251→			case req := <-walRotateCh:
   252→				processRotate(req)
   253→			}
   254→		}
   255→	})
   256→}
   257→
   258→// RotateWALFiles sends rotation requests for the given host directories
   259→// and blocks until all rotations complete.
   260→func RotateWALFiles(hostDirs []string) {
   261→	dones := make([]chan struct{}, len(hostDirs))
   262→	for i, dir := range hostDirs {
   263→		dones[i] = make(chan struct{})
   264→		walRotateCh <- walRotateReq{hostDir: dir, done: dones[i]}
   265→	}
   266→	for _, done := range dones {
   267→		<-done
   268→	}
   269→}
   270→
   271→// RotateWALFiles sends rotation requests for the given host directories
   272→// and blocks until all rotations complete.
   273→func RotateWALFilesAfterShutdown(hostDirs []string) {
   274→	for _, dir := range hostDirs {
   275→		walPath := path.Join(dir, "current.wal")
   276→		if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
   277→			cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
   278→		}
   279→	}
   280→}
   281→
   282→// buildWALPayload encodes a WALMessage into a binary payload (without magic/length/CRC).
   283→func buildWALPayload(msg *WALMessage) []byte {
   284→	size := 8 + 2 + len(msg.MetricName) + 1 + 4
   285→	for _, s := range msg.Selector {
   286→		size += 1 + len(s)
   287→	}
   288→
   289→	buf := make([]byte, 0, size)
   290→
   291→	// Timestamp (8 bytes, little-endian int64)
   292→	var ts [8]byte
   293→	binary.LittleEndian.PutUint64(ts[:], uint64(msg.Timestamp))
   294→	buf = append(buf, ts[:]...)
   295→
   296→	// Metric name (2-byte length prefix + bytes)
   297→	var mLen [2]byte
   298→	binary.LittleEndian.PutUint16(mLen[:], uint16(len(msg.MetricName)))
   299→	buf = append(buf, mLen[:]...)
   300→	buf = append(buf, msg.MetricName...)
   301→
   302→	// Selector count (1 byte)
   303→	buf = append(buf, byte(len(msg.Selector)))
   304→
   305→	// Selectors (1-byte length prefix + bytes each)
   306→	for _, sel := range msg.Selector {
   307→		buf = append(buf, byte(len(sel)))
   308→		buf = append(buf, sel...)
   309→	}
   310→
   311→	// Value (4 bytes, float32 bit representation)
   312→	var val [4]byte
   313→	binary.LittleEndian.PutUint32(val[:], math.Float32bits(float32(msg.Value)))
   314→	buf = append(buf, val[:]...)
   315→
   316→	return buf
   317→}
   318→
   319→// writeWALRecord appends a binary WAL record to the writer.
   320→// Format: [4B magic][4B payload_len][payload][4B CRC32]
   321→func writeWALRecord(w io.Writer, msg *WALMessage) error {
   322→	payload := buildWALPayload(msg)
   323→	crc := crc32.ChecksumIEEE(payload)
   324→
   325→	record := make([]byte, 0, 4+4+len(payload)+4)
   326→
   327→	var magic [4]byte
   328→	binary.LittleEndian.PutUint32(magic[:], walRecordMagic)
   329→	record = append(record, magic[:]...)
   330→
   331→	var pLen [4]byte
   332→	binary.LittleEndian.PutUint32(pLen[:], uint32(len(payload)))
   333→	record = append(record, pLen[:]...)
   334→
   335→	record = append(record, payload...)
   336→
   337→	var crcBytes [4]byte
   338→	binary.LittleEndian.PutUint32(crcBytes[:], crc)
   339→	record = append(record, crcBytes[:]...)
   340→
   341→	_, err := w.Write(record)
   342→	return err
   343→}
   344→
   345→// readWALRecord reads one WAL record from the reader.
   346→// Returns (nil, nil) on clean EOF. Returns error on data corruption.
   347→// A CRC mismatch indicates a truncated trailing record (expected on crash).
   348→func readWALRecord(r io.Reader) (*WALMessage, error) {
   349→	var magic uint32
   350→	if err := binary.Read(r, binary.LittleEndian, &magic); err != nil {
   351→		if err == io.EOF {
   352→			return nil, nil // Clean EOF
   353→		}
   354→		return nil, fmt.Errorf("read record magic: %w", err)
   355→	}
   356→
   357→	if magic != walRecordMagic {
   358→		return nil, fmt.Errorf("invalid record magic 0x%08X (expected 0x%08X)", magic, walRecordMagic)
   359→	}
   360→
   361→	var payloadLen uint32
   362→	if err := binary.Read(r, binary.LittleEndian, &payloadLen); err != nil {
   363→		return nil, fmt.Errorf("read payload length: %w", err)
   364→	}
   365→
   366→	if payloadLen > 1<<20 { // 1 MB sanity limit
   367→		return nil, fmt.Errorf("record payload too large: %d bytes", payloadLen)
   368→	}
   369→
   370→	payload := make([]byte, payloadLen)
   371→	if _, err := io.ReadFull(r, payload); err != nil {
   372→		return nil, fmt.Errorf("read payload: %w", err)
   373→	}
   374→
   375→	var storedCRC uint32
   376→	if err := binary.Read(r, binary.LittleEndian, &storedCRC); err != nil {
   377→		return nil, fmt.Errorf("read CRC: %w", err)
   378→	}
   379→
   380→	if crc32.ChecksumIEEE(payload) != storedCRC {
   381→		return nil, fmt.Errorf("CRC mismatch (truncated write or corruption)")
   382→	}
   383→
   384→	return parseWALPayload(payload)
   385→}
   386→
   387→// parseWALPayload decodes a binary payload into a WALMessage.
   388→func parseWALPayload(payload []byte) (*WALMessage, error) {
   389→	if len(payload) < 8+2+1+4 {
   390→		return nil, fmt.Errorf("payload too short: %d bytes", len(payload))
   391→	}
   392→
   393→	offset := 0
   394→
   395→	// Timestamp (8 bytes)
   396→	ts := int64(binary.LittleEndian.Uint64(payload[offset : offset+8]))
   397→	offset += 8
   398→
   399→	// Metric name (2-byte length + bytes)
   400→	if offset+2 > len(payload) {
   401→		return nil, fmt.Errorf("metric name length overflows payload")
   402→	}
   403→	mLen := int(binary.LittleEndian.Uint16(payload[offset : offset+2]))
   404→	offset += 2
   405→
   406→	if offset+mLen > len(payload) {
   407→		return nil, fmt.Errorf("metric name overflows payload")
   408→	}
   409→	metricName := string(payload[offset : offset+mLen])
   410→	offset += mLen
   411→
   412→	// Selector count (1 byte)
   413→	if offset >= len(payload) {
   414→		return nil, fmt.Errorf("selector count overflows payload")
   415→	}
   416→	selCount := int(payload[offset])
   417→	offset++
   418→
   419→	selectors := make([]string, selCount)
   420→	for i := range selCount {
   421→		if offset >= len(payload) {
   422→			return nil, fmt.Errorf("selector[%d] length overflows payload", i)
   423→		}
   424→		sLen := int(payload[offset])
   425→		offset++
   426→
   427→		if offset+sLen > len(payload) {
   428→			return nil, fmt.Errorf("selector[%d] data overflows payload", i)
   429→		}
   430→		selectors[i] = string(payload[offset : offset+sLen])
   431→		offset += sLen
   432→	}
   433→
   434→	// Value (4 bytes, float32 bits)
   435→	if offset+4 > len(payload) {
   436→		return nil, fmt.Errorf("value overflows payload")
   437→	}
   438→	bits := binary.LittleEndian.Uint32(payload[offset : offset+4])
   439→	value := schema.Float(math.Float32frombits(bits))
   440→
   441→	return &WALMessage{
   442→		MetricName: metricName,
   443→		Timestamp:  ts,
   444→		Selector:   selectors,
   445→		Value:      value,
   446→	}, nil
   447→}
   448→
   449→// loadWALFile reads a WAL file and replays all valid records into the Level tree.
   450→// l is the host-level node. Corrupt or partial trailing records are silently skipped
   451→// (expected on crash). Records older than 'from' are skipped.
   452→func (l *Level) loadWALFile(m *MemoryStore, f *os.File, from int64) error {
   453→	br := bufio.NewReader(f)
   454→
   455→	// Verify file header magic.
   456→	var fileMagic uint32
   457→	if err := binary.Read(br, binary.LittleEndian, &fileMagic); err != nil {
   458→		if err == io.EOF {
   459→			return nil // Empty file, no data
   460→		}
   461→		return fmt.Errorf("[METRICSTORE]> WAL: read file header: %w", err)
   462→	}
   463→
   464→	if fileMagic != walFileMagic {
   465→		return fmt.Errorf("[METRICSTORE]> WAL: invalid file magic 0x%08X (expected 0x%08X)", fileMagic, walFileMagic)
   466→	}
   467→
   468→	// Cache level lookups to avoid repeated tree traversal.
   469→	lvlCache := make(map[string]*Level)
   470→
   471→	for {
   472→		msg, err := readWALRecord(br)
   473→		if err != nil {
   474→			// Truncated trailing record is expected after a crash; stop replaying.
   475→			cclog.Debugf("[METRICSTORE]> WAL: stopping replay at corrupted/partial record: %v", err)
   476→			break
   477→		}
   478→		if msg == nil {
   479→			break // Clean EOF
   480→		}
   481→
   482→		if msg.Timestamp < from {
   483→			continue // Older than retention window
   484→		}
   485→
   486→		minfo, ok := m.Metrics[msg.MetricName]
   487→		if !ok {
   488→			continue // Unknown metric (config may have changed)
   489→		}
   490→
   491→		// Cache key is the null-separated selector path.
   492→		cacheKey := joinSelector(msg.Selector)
   493→		lvl, ok := lvlCache[cacheKey]
   494→		if !ok {
   495→			lvl = l.findLevelOrCreate(msg.Selector, len(m.Metrics))
   496→			lvlCache[cacheKey] = lvl
   497→		}
   498→
   499→		// Write directly to the buffer, same as WriteToLevel but without the
   500→		// global level lookup (we already have the right level).
   501→		lvl.lock.Lock()
   502→		b := lvl.metrics[minfo.offset]
   503→		if b == nil {
   504→			b = newBuffer(msg.Timestamp, minfo.Frequency)
   505→			lvl.metrics[minfo.offset] = b
   506→		}
   507→		nb, writeErr := b.write(msg.Timestamp, msg.Value)
   508→		if writeErr == nil && b != nb {
   509→			lvl.metrics[minfo.offset] = nb
   510→		}
   511→		// Ignore write errors for timestamps before buffer start (can happen when
   512→		// replaying WAL entries that predate a loaded snapshot's start time).
   513→		lvl.lock.Unlock()
   514→	}
   515→
   516→	return nil
   517→}
   518→
   519→// joinSelector builds a cache key from a selector slice using null bytes as separators.
   520→func joinSelector(sel []string) string {
   521→	if len(sel) == 0 {
   522→		return ""
   523→	}
   524→	var result strings.Builder
   525→	result.WriteString(sel[0])
   526→	for i := 1; i < len(sel); i++ {
   527→		result.WriteString("\x00" + sel[i])
   528→	}
   529→	return result.String()
   530→}
   531→
   532→// ToCheckpointWAL writes binary snapshot files for all hosts in parallel.
   533→// Returns the number of files written, the list of host directories that were
   534→// successfully checkpointed (for WAL rotation), and any errors.
   535→func (m *MemoryStore) ToCheckpointWAL(dir string, from, to int64) (int, []string, error) {
   536→	// Collect all cluster/host pairs.
   537→	m.root.lock.RLock()
   538→	totalHosts := 0
   539→	for _, l1 := range m.root.children {
   540→		l1.lock.RLock()
   541→		totalHosts += len(l1.children)
   542→		l1.lock.RUnlock()
   543→	}
   544→	m.root.lock.RUnlock()
   545→
   546→	levels := make([]*Level, 0, totalHosts)
   547→	selectors := make([][]string, 0, totalHosts)
   548→
   549→	m.root.lock.RLock()
   550→	for sel1, l1 := range m.root.children {
   551→		l1.lock.RLock()
   552→		for sel2, l2 := range l1.children {
   553→			levels = append(levels, l2)
   554→			selectors = append(selectors, []string{sel1, sel2})
   555→		}
   556→		l1.lock.RUnlock()
   557→	}
   558→	m.root.lock.RUnlock()
   559→
   560→	type workItem struct {
   561→		level    *Level
   562→		hostDir  string
   563→		selector []string
   564→	}
   565→
   566→	n, errs := int32(0), int32(0)
   567→	var successDirs []string
   568→	var successMu sync.Mutex
   569→
   570→	var wg sync.WaitGroup
   571→	wg.Add(Keys.NumWorkers)
   572→	work := make(chan workItem, Keys.NumWorkers*2)
   573→
   574→	for range Keys.NumWorkers {
   575→		go func() {
   576→			defer wg.Done()
   577→			for wi := range work {
   578→				err := wi.level.toCheckpointBinary(wi.hostDir, from, to, m)
   579→				if err != nil {
   580→					if err == ErrNoNewArchiveData {
   581→						continue
   582→					}
   583→					cclog.Errorf("[METRICSTORE]> binary checkpoint error for %s: %v", wi.hostDir, err)
   584→					atomic.AddInt32(&errs, 1)
   585→				} else {
   586→					atomic.AddInt32(&n, 1)
   587→					successMu.Lock()
   588→					successDirs = append(successDirs, wi.hostDir)
   589→					successMu.Unlock()
   590→				}
   591→			}
   592→		}()
   593→	}
   594→
   595→	for i := range levels {
   596→		hostDir := path.Join(dir, path.Join(selectors[i]...))
   597→		work <- workItem{
   598→			level:    levels[i],
   599→			hostDir:  hostDir,
   600→			selector: selectors[i],
   601→		}
   602→	}
   603→	close(work)
   604→	wg.Wait()
   605→
   606→	if errs > 0 {
   607→		return int(n), successDirs, fmt.Errorf("[METRICSTORE]> %d errors during binary checkpoint (%d successes)", errs, n)
   608→	}
   609→	return int(n), successDirs, nil
   610→}
   611→
   612→// toCheckpointBinary writes a binary snapshot file for a single host-level node.
   613→// Uses atomic rename (write to .tmp then rename) to avoid partial reads on crash.
   614→func (l *Level) toCheckpointBinary(dir string, from, to int64, m *MemoryStore) error {
   615→	cf, err := l.toCheckpointFile(from, to, m)
   616→	if err != nil {
   617→		return err
   618→	}
   619→	if cf == nil {
   620→		return ErrNoNewArchiveData
   621→	}
   622→
   623→	if err := os.MkdirAll(dir, CheckpointDirPerms); err != nil {
   624→		return fmt.Errorf("mkdir %s: %w", dir, err)
   625→	}
   626→
   627→	// Write to a temp file first, then rename (atomic on POSIX).
   628→	tmpPath := path.Join(dir, fmt.Sprintf("%d.bin.tmp", from))
   629→	finalPath := path.Join(dir, fmt.Sprintf("%d.bin", from))
   630→
   631→	f, err := os.OpenFile(tmpPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
   632→	if err != nil {
   633→		return fmt.Errorf("open binary snapshot %s: %w", tmpPath, err)
   634→	}
   635→
   636→	bw := bufio.NewWriter(f)
   637→	if err := writeBinarySnapshotFile(bw, cf); err != nil {
   638→		f.Close()
   639→		os.Remove(tmpPath)
   640→		return fmt.Errorf("write binary snapshot: %w", err)
   641→	}
   642→	if err := bw.Flush(); err != nil {
   643→		f.Close()
   644→		os.Remove(tmpPath)
   645→		return err
   646→	}
   647→	f.Close()
   648→
   649→	return os.Rename(tmpPath, finalPath)
   650→}
   651→
   652→// writeBinarySnapshotFile writes the binary snapshot file header and level tree.
   653→func writeBinarySnapshotFile(w io.Writer, cf *CheckpointFile) error {
   654→	if err := binary.Write(w, binary.LittleEndian, snapFileMagic); err != nil {
   655→		return err
   656→	}
   657→	if err := binary.Write(w, binary.LittleEndian, cf.From); err != nil {
   658→		return err
   659→	}
   660→	if err := binary.Write(w, binary.LittleEndian, cf.To); err != nil {
   661→		return err
   662→	}
   663→	return writeBinaryLevel(w, cf)
   664→}
   665→
   666→// writeBinaryLevel recursively writes a CheckpointFile level in binary format.
   667→func writeBinaryLevel(w io.Writer, cf *CheckpointFile) error {
   668→	if err := binary.Write(w, binary.LittleEndian, uint32(len(cf.Metrics))); err != nil {
   669→		return err
   670→	}
   671→
   672→	for name, metric := range cf.Metrics {
   673→		if err := writeString16(w, name); err != nil {
   674→			return err
   675→		}
   676→		if err := binary.Write(w, binary.LittleEndian, metric.Frequency); err != nil {
   677→			return err
   678→		}
   679→		if err := binary.Write(w, binary.LittleEndian, metric.Start); err != nil {
   680→			return err
   681→		}
   682→		if err := binary.Write(w, binary.LittleEndian, uint32(len(metric.Data))); err != nil {
   683→			return err
   684→		}
   685→		for _, v := range metric.Data {
   686→			if err := binary.Write(w, binary.LittleEndian, math.Float32bits(float32(v))); err != nil {
   687→				return err
   688→			}
   689→		}
   690→	}
   691→
   692→	if err := binary.Write(w, binary.LittleEndian, uint32(len(cf.Children))); err != nil {
   693→		return err
   694→	}
   695→
   696→	for name, child := range cf.Children {
   697→		if err := writeString16(w, name); err != nil {
   698→			return err
   699→		}
   700→		if err := writeBinaryLevel(w, child); err != nil {
   701→			return err
   702→		}
   703→	}
   704→
   705→	return nil
   706→}
   707→
   708→// writeString16 writes a 2-byte length-prefixed string to w.
   709→func writeString16(w io.Writer, s string) error {
   710→	if err := binary.Write(w, binary.LittleEndian, uint16(len(s))); err != nil {
   711→		return err
   712→	}
   713→	_, err := io.WriteString(w, s)
   714→	return err
   715→}
   716→
   717→// loadBinaryFile reads a binary snapshot file and loads data into the Level tree.
   718→// The retention check (from) is applied to the file's 'to' timestamp.
   719→func (l *Level) loadBinaryFile(m *MemoryStore, f *os.File, from int64) error {
   720→	br := bufio.NewReader(f)
   721→
   722→	var magic uint32
   723→	if err := binary.Read(br, binary.LittleEndian, &magic); err != nil {
   724→		return fmt.Errorf("[METRICSTORE]> binary snapshot: read magic: %w", err)
   725→	}
   726→	if magic != snapFileMagic {
   727→		return fmt.Errorf("[METRICSTORE]> binary snapshot: invalid magic 0x%08X (expected 0x%08X)", magic, snapFileMagic)
   728→	}
   729→
   730→	var fileFrom, fileTo int64
   731→	if err := binary.Read(br, binary.LittleEndian, &fileFrom); err != nil {
   732→		return fmt.Errorf("[METRICSTORE]> binary snapshot: read from: %w", err)
   733→	}
   734→	if err := binary.Read(br, binary.LittleEndian, &fileTo); err != nil {
   735→		return fmt.Errorf("[METRICSTORE]> binary snapshot: read to: %w", err)
   736→	}
   737→
   738→	if fileTo != 0 && fileTo < from {
   739→		return nil // File is older than retention window, skip it
   740→	}
   741→
   742→	cf, err := readBinaryLevel(br)
   743→	if err != nil {
   744→		return fmt.Errorf("[METRICSTORE]> binary snapshot: read level tree: %w", err)
   745→	}
   746→	cf.From = fileFrom
   747→	cf.To = fileTo
   748→
   749→	return l.loadFile(cf, m)
   750→}
   751→
   752→// readBinaryLevel recursively reads a level from the binary snapshot format.
   753→func readBinaryLevel(r io.Reader) (*CheckpointFile, error) {
   754→	cf := &CheckpointFile{
   755→		Metrics:  make(map[string]*CheckpointMetrics),
   756→		Children: make(map[string]*CheckpointFile),
   757→	}
   758→
   759→	var numMetrics uint32
   760→	if err := binary.Read(r, binary.LittleEndian, &numMetrics); err != nil {
   761→		return nil, fmt.Errorf("read num_metrics: %w", err)
   762→	}
   763→
   764→	for range numMetrics {
   765→		name, err := readString16(r)
   766→		if err != nil {
   767→			return nil, fmt.Errorf("read metric name: %w", err)
   768→		}
   769→
   770→		var freq, start int64
   771→		if err := binary.Read(r, binary.LittleEndian, &freq); err != nil {
   772→			return nil, fmt.Errorf("read frequency for %s: %w", name, err)
   773→		}
   774→		if err := binary.Read(r, binary.LittleEndian, &start); err != nil {
   775→			return nil, fmt.Errorf("read start for %s: %w", name, err)
   776→		}
   777→
   778→		var numValues uint32
   779→		if err := binary.Read(r, binary.LittleEndian, &numValues); err != nil {
   780→			return nil, fmt.Errorf("read num_values for %s: %w", name, err)
   781→		}
   782→
   783→		data := make([]schema.Float, numValues)
   784→		for i := range numValues {
   785→			var bits uint32
   786→			if err := binary.Read(r, binary.LittleEndian, &bits); err != nil {
   787→				return nil, fmt.Errorf("read value[%d] for %s: %w", i, name, err)
   788→			}
   789→			data[i] = schema.Float(math.Float32frombits(bits))
   790→		}
   791→
   792→		cf.Metrics[name] = &CheckpointMetrics{
   793→			Frequency: freq,
   794→			Start:     start,
   795→			Data:      data,
   796→		}
   797→	}
   798→
   799→	var numChildren uint32
   800→	if err := binary.Read(r, binary.LittleEndian, &numChildren); err != nil {
   801→		return nil, fmt.Errorf("read num_children: %w", err)
   802→	}
   803→
   804→	for range numChildren {
   805→		childName, err := readString16(r)
   806→		if err != nil {
   807→			return nil, fmt.Errorf("read child name: %w", err)
   808→		}
   809→
   810→		child, err := readBinaryLevel(r)
   811→		if err != nil {
   812→			return nil, fmt.Errorf("read child %s: %w", childName, err)
   813→		}
   814→		cf.Children[childName] = child
   815→	}
   816→
   817→	return cf, nil
   818→}
   819→
   820→// readString16 reads a 2-byte length-prefixed string from r.
   821→func readString16(r io.Reader) (string, error) {
   822→	var sLen uint16
   823→	if err := binary.Read(r, binary.LittleEndian, &sLen); err != nil {
   824→		return "", err
   825→	}
   826→	buf := make([]byte, sLen)
   827→	if _, err := io.ReadFull(r, buf); err != nil {
   828→		return "", err
   829→	}
   830→	return string(buf), nil
   831→}
   832→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package metricstore provides walCheckpoint.go: WAL-based checkpoint implementation.
//
// This replaces the Avro shadow tree with an append-only Write-Ahead Log (WAL)
// per host, eliminating the extra memory overhead of the AvroStore and providing
// truly continuous (per-write) crash safety.
//
// # Architecture
//
//	Metric write (DecodeLine)
//	      │
//	      ├─► WriteToLevel()    → main MemoryStore (unchanged)
//	      │
//	      └─► WALMessages channel
//	               │
//	               ▼
//	        WALStaging goroutine
//	               │
//	               ▼
//	        checkpoints/cluster/host/current.wal  (append-only, binary)
//
//	Periodic checkpoint (Checkpointing goroutine):
//	  1. Write <timestamp>.bin snapshot (column-oriented, from main tree)
//	  2. Signal WALStaging to truncate current.wal per host
//
//	On restart (FromCheckpoint):
//	  1. Load most recent <timestamp>.bin snapshot
//	  2. Replay current.wal (overwrite-safe: buffer.write handles duplicate timestamps)
//
// # WAL Record Format
//
//	[4B magic 0xCC1DA7A1][4B payload_len][payload][4B CRC32]
//
//	payload:
//	  [8B timestamp int64]
//	  [2B metric_name_len uint16][N metric name bytes]
//	  [1B selector_count uint8]
//	  per selector: [1B selector_len uint8][M selector bytes]
//	  [4B value float32 bits]
//
// # Binary Snapshot Format
//
//	[4B magic 0xCC5B0001][8B from int64][8B to int64]
//	Level tree (recursive):
//	  [4B num_metrics uint32]
//	  per metric:
//	    [2B name_len uint16][N name bytes]
//	    [8B frequency int64][8B start int64]
//	    [4B num_values uint32][num_values × 4B float32]
//	  [4B num_children uint32]
//	  per child: [2B name_len uint16][N name bytes] + Level (recursive)
package metricstore

import (
	"bufio"
	"context"
	"encoding/binary"
	"fmt"
	"hash/crc32"
	"io"
	"math"
	"os"
	"path"
	"strings"
	"sync"
	"sync/atomic"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

// Magic numbers for binary formats.
const (
	walFileMagic   = uint32(0xCC1DA701) // WAL file header magic
	walRecordMagic = uint32(0xCC1DA7A1) // WAL record magic
	snapFileMagic  = uint32(0xCC5B0001) // Binary snapshot magic
)

// WALMessages is the channel for sending metric writes to the WAL staging goroutine.
// Buffered to allow burst writes without blocking the metric ingestion path.
var WALMessages = make(chan *WALMessage, 65536)

// walRotateCh is used by the checkpoint goroutine to request WAL file rotation
// (close, delete, reopen) after a binary snapshot has been written.
var walRotateCh = make(chan walRotateReq, 256)

// WALMessage represents a single metric write to be appended to the WAL.
// Cluster and Node are NOT stored in the WAL record (inferred from file path).
type WALMessage struct {
	MetricName string
	Cluster    string
	Node       string
	Selector   []string
	Value      schema.Float
	Timestamp  int64
}

// walRotateReq requests WAL file rotation for a specific host directory.
// The done channel is closed by the WAL goroutine when rotation is complete.
type walRotateReq struct {
	hostDir string
	done    chan struct{}
}

// walFileState holds an open WAL file handle and buffered writer for one host directory.
type walFileState struct {
	f *os.File
	w *bufio.Writer
}

// WALStaging starts a background goroutine that receives WALMessage items
// and appends binary WAL records to per-host current.wal files.
// Also handles WAL rotation requests from the checkpoint goroutine.
func WALStaging(wg *sync.WaitGroup, ctx context.Context) {
	wg.Go(func() {
		if Keys.Checkpoints.FileFormat == "json" {
			return
		}

		hostFiles := make(map[string]*walFileState)

		defer func() {
			for _, ws := range hostFiles {
				if ws.f != nil {
					ws.w.Flush()
					ws.f.Close()
				}
			}
		}()

		getOrOpenWAL := func(hostDir string) *walFileState {
			ws, ok := hostFiles[hostDir]
			if ok {
				return ws
			}

			if err := os.MkdirAll(hostDir, CheckpointDirPerms); err != nil {
				cclog.Errorf("[METRICSTORE]> WAL: mkdir %s: %v", hostDir, err)
				return nil
			}

			walPath := path.Join(hostDir, "current.wal")
			f, err := os.OpenFile(walPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, CheckpointFilePerms)
			if err != nil {
				cclog.Errorf("[METRICSTORE]> WAL: open %s: %v", walPath, err)
				return nil
			}

			w := bufio.NewWriter(f)

			// Write file header magic if file is new (empty).
			info, err := f.Stat()
			if err == nil && info.Size() == 0 {
				var hdr [4]byte
				binary.LittleEndian.PutUint32(hdr[:], walFileMagic)
				if _, err := w.Write(hdr[:]); err != nil {
					cclog.Errorf("[METRICSTORE]> WAL: write header %s: %v", walPath, err)
					f.Close()
					return nil
				}
			}

			ws = &walFileState{f: f, w: w}
			hostFiles[hostDir] = ws
			return ws
		}

		processMsg := func(msg *WALMessage) {
			hostDir := path.Join(Keys.Checkpoints.RootDir, msg.Cluster, msg.Node)
			ws := getOrOpenWAL(hostDir)
			if ws == nil {
				return
			}
			if err := writeWALRecord(ws.w, msg); err != nil {
				cclog.Errorf("[METRICSTORE]> WAL: write record: %v", err)
			}
		}

		processRotate := func(req walRotateReq) {
			ws, ok := hostFiles[req.hostDir]
			if ok && ws.f != nil {
				ws.w.Flush()
				ws.f.Close()
				walPath := path.Join(req.hostDir, "current.wal")
				if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
					cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
				}
				delete(hostFiles, req.hostDir)
			}
			close(req.done)
		}

		drain := func() {
			for {
				select {
				case msg, ok := <-WALMessages:
					if !ok {
						return
					}
					processMsg(msg)
				case req := <-walRotateCh:
					processRotate(req)
				default:
					// Flush all buffered writers after draining remaining messages.
					for _, ws := range hostFiles {
						if ws.f != nil {
							ws.w.Flush()
						}
					}
					return
				}
			}
		}

		for {
			select {
			case <-ctx.Done():
				drain()
				return
			case msg, ok := <-WALMessages:
				if !ok {
					return
				}
				processMsg(msg)

				// Drain up to 256 more messages without blocking to batch writes.
				for range 256 {
					select {
					case msg, ok := <-WALMessages:
						if !ok {
							return
						}
						processMsg(msg)
					case req := <-walRotateCh:
						processRotate(req)
					default:
						goto flushed
					}
				}
			flushed:
				// Flush all buffered writers after processing the batch.
				for _, ws := range hostFiles {
					if ws.f != nil {
						ws.w.Flush()
					}
				}
			case req := <-walRotateCh:
				processRotate(req)
			}
		}
	})
}

// RotateWALFiles sends rotation requests for the given host directories
// and blocks until all rotations complete.
func RotateWALFiles(hostDirs []string) {
	dones := make([]chan struct{}, len(hostDirs))
	for i, dir := range hostDirs {
		dones[i] = make(chan struct{})
		walRotateCh <- walRotateReq{hostDir: dir, done: dones[i]}
	}
	for _, done := range dones {
		<-done
	}
}

// RotateWALFiles sends rotation requests for the given host directories
// and blocks until all rotations complete.
func RotateWALFilesAfterShutdown(hostDirs []string) {
	for _, dir := range hostDirs {
		walPath := path.Join(dir, "current.wal")
		if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
			cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
		}
	}
}

// buildWALPayload encodes a WALMessage into a binary payload (without magic/length/CRC).
func buildWALPayload(msg *WALMessage) []byte {
	size := 8 + 2 + len(msg.MetricName) + 1 + 4
	for _, s := range msg.Selector {
		size += 1 + len(s)
	}

	buf := make([]byte, 0, size)

	// Timestamp (8 bytes, little-endian int64)
	var ts [8]byte
	binary.LittleEndian.PutUint64(ts[:], uint64(msg.Timestamp))
	buf = append(buf, ts[:]...)

	// Metric name (2-byte length prefix + bytes)
	var mLen [2]byte
	binary.LittleEndian.PutUint16(mLen[:], uint16(len(msg.MetricName)))
	buf = append(buf, mLen[:]...)
	buf = append(buf, msg.MetricName...)

	// Selector count (1 byte)
	buf = append(buf, byte(len(msg.Selector)))

	// Selectors (1-byte length prefix + bytes each)
	for _, sel := range msg.Selector {
		buf = append(buf, byte(len(sel)))
		buf = append(buf, sel...)
	}

	// Value (4 bytes, float32 bit representation)
	var val [4]byte
	binary.LittleEndian.PutUint32(val[:], math.Float32bits(float32(msg.Value)))
	buf = append(buf, val[:]...)

	return buf
}

// writeWALRecord appends a binary WAL record to the writer.
// Format: [4B magic][4B payload_len][payload][4B CRC32]
func writeWALRecord(w io.Writer, msg *WALMessage) error {
	payload := buildWALPayload(msg)
	crc := crc32.ChecksumIEEE(payload)

	record := make([]byte, 0, 4+4+len(payload)+4)

	var magic [4]byte
	binary.LittleEndian.PutUint32(magic[:], walRecordMagic)
	record = append(record, magic[:]...)

	var pLen [4]byte
	binary.LittleEndian.PutUint32(pLen[:], uint32(len(payload)))
	record = append(record, pLen[:]...)

	record = append(record, payload...)

	var crcBytes [4]byte
	binary.LittleEndian.PutUint32(crcBytes[:], crc)
	record = append(record, crcBytes[:]...)

	_, err := w.Write(record)
	return err
}

// readWALRecord reads one WAL record from the reader.
// Returns (nil, nil) on clean EOF. Returns error on data corruption.
// A CRC mismatch indicates a truncated trailing record (expected on crash).
func readWALRecord(r io.Reader) (*WALMessage, error) {
	var magic uint32
	if err := binary.Read(r, binary.LittleEndian, &magic); err != nil {
		if err == io.EOF {
			return nil, nil // Clean EOF
		}
		return nil, fmt.Errorf("read record magic: %w", err)
	}

	if magic != walRecordMagic {
		return nil, fmt.Errorf("invalid record magic 0x%08X (expected 0x%08X)", magic, walRecordMagic)
	}

	var payloadLen uint32
	if err := binary.Read(r, binary.LittleEndian, &payloadLen); err != nil {
		return nil, fmt.Errorf("read payload length: %w", err)
	}

	if payloadLen > 1<<20 { // 1 MB sanity limit
		return nil, fmt.Errorf("record payload too large: %d bytes", payloadLen)
	}

	payload := make([]byte, payloadLen)
	if _, err := io.ReadFull(r, payload); err != nil {
		return nil, fmt.Errorf("read payload: %w", err)
	}

	var storedCRC uint32
	if err := binary.Read(r, binary.LittleEndian, &storedCRC); err != nil {
		return nil, fmt.Errorf("read CRC: %w", err)
	}

	if crc32.ChecksumIEEE(payload) != storedCRC {
		return nil, fmt.Errorf("CRC mismatch (truncated write or corruption)")
	}

	return parseWALPayload(payload)
}

// parseWALPayload decodes a binary payload into a WALMessage.
func parseWALPayload(payload []byte) (*WALMessage, error) {
	if len(payload) < 8+2+1+4 {
		return nil, fmt.Errorf("payload too short: %d bytes", len(payload))
	}

	offset := 0

	// Timestamp (8 bytes)
	ts := int64(binary.LittleEndian.Uint64(payload[offset : offset+8]))
	offset += 8

	// Metric name (2-byte length + bytes)
	if offset+2 > len(payload) {
		return nil, fmt.Errorf("metric name length overflows payload")
	}
	mLen := int(binary.LittleEndian.Uint16(payload[offset : offset+2]))
	offset += 2

	if offset+mLen > len(payload) {
		return nil, fmt.Errorf("metric name overflows payload")
	}
	metricName := string(payload[offset : offset+mLen])
	offset += mLen

	// Selector count (1 byte)
	if offset >= len(payload) {
		return nil, fmt.Errorf("selector count overflows payload")
	}
	selCount := int(payload[offset])
	offset++

	selectors := make([]string, selCount)
	for i := range selCount {
		if offset >= len(payload) {
			return nil, fmt.Errorf("selector[%d] length overflows payload", i)
		}
		sLen := int(payload[offset])
		offset++

		if offset+sLen > len(payload) {
			return nil, fmt.Errorf("selector[%d] data overflows payload", i)
		}
		selectors[i] = string(payload[offset : offset+sLen])
		offset += sLen
	}

	// Value (4 bytes, float32 bits)
	if offset+4 > len(payload) {
		return nil, fmt.Errorf("value overflows payload")
	}
	bits := binary.LittleEndian.Uint32(payload[offset : offset+4])
	value := schema.Float(math.Float32frombits(bits))

	return &WALMessage{
		MetricName: metricName,
		Timestamp:  ts,
		Selector:   selectors,
		Value:      value,
	}, nil
}

// loadWALFile reads a WAL file and replays all valid records into the Level tree.
// l is the host-level node. Corrupt or partial trailing records are silently skipped
// (expected on crash). Records older than 'from' are skipped.
func (l *Level) loadWALFile(m *MemoryStore, f *os.File, from int64) error {
	br := bufio.NewReader(f)

	// Verify file header magic.
	var fileMagic uint32
	if err := binary.Read(br, binary.LittleEndian, &fileMagic); err != nil {
		if err == io.EOF {
			return nil // Empty file, no data
		}
		return fmt.Errorf("[METRICSTORE]> WAL: read file header: %w", err)
	}

	if fileMagic != walFileMagic {
		return fmt.Errorf("[METRICSTORE]> WAL: invalid file magic 0x%08X (expected 0x%08X)", fileMagic, walFileMagic)
	}

	// Cache level lookups to avoid repeated tree traversal.
	lvlCache := make(map[string]*Level)

	for {
		msg, err := readWALRecord(br)
		if err != nil {
			// Truncated trailing record is expected after a crash; stop replaying.
			cclog.Debugf("[METRICSTORE]> WAL: stopping replay at corrupted/partial record: %v", err)
			break
		}
		if msg == nil {
			break // Clean EOF
		}

		if msg.Timestamp < from {
			continue // Older than retention window
		}

		minfo, ok := m.Metrics[msg.MetricName]
		if !ok {
			continue // Unknown metric (config may have changed)
		}

		// Cache key is the null-separated selector path.
		cacheKey := joinSelector(msg.Selector)
		lvl, ok := lvlCache[cacheKey]
		if !ok {
			lvl = l.findLevelOrCreate(msg.Selector, len(m.Metrics))
			lvlCache[cacheKey] = lvl
		}

		// Write directly to the buffer, same as WriteToLevel but without the
		// global level lookup (we already have the right level).
		lvl.lock.Lock()
		b := lvl.metrics[minfo.offset]
		if b == nil {
			b = newBuffer(msg.Timestamp, minfo.Frequency)
			lvl.metrics[minfo.offset] = b
		}
		nb, writeErr := b.write(msg.Timestamp, msg.Value)
		if writeErr == nil && b != nb {
			lvl.metrics[minfo.offset] = nb
		}
		// Ignore write errors for timestamps before buffer start (can happen when
		// replaying WAL entries that predate a loaded snapshot's start time).
		lvl.lock.Unlock()
	}

	return nil
}

// joinSelector builds a cache key from a selector slice using null bytes as separators.
func joinSelector(sel []string) string {
	if len(sel) == 0 {
		return ""
	}
	var result strings.Builder
	result.WriteString(sel[0])
	for i := 1; i < len(sel); i++ {
		result.WriteString("\x00" + sel[i])
	}
	return result.String()
}

// ToCheckpointWAL writes binary snapshot files for all hosts in parallel.
// Returns the number of files written, the list of host directories that were
// successfully checkpointed (for WAL rotation), and any errors.
func (m *MemoryStore) ToCheckpointWAL(dir string, from, to int64) (int, []string, error) {
	// Collect all cluster/host pairs.
	m.root.lock.RLock()
	totalHosts := 0
	for _, l1 := range m.root.children {
		l1.lock.RLock()
		totalHosts += len(l1.children)
		l1.lock.RUnlock()
	}
	m.root.lock.RUnlock()

	levels := make([]*Level, 0, totalHosts)
	selectors := make([][]string, 0, totalHosts)

	m.root.lock.RLock()
	for sel1, l1 := range m.root.children {
		l1.lock.RLock()
		for sel2, l2 := range l1.children {
			levels = append(levels, l2)
			selectors = append(selectors, []string{sel1, sel2})
		}
		l1.lock.RUnlock()
	}
	m.root.lock.RUnlock()

	type workItem struct {
		level    *Level
		hostDir  string
		selector []string
	}

	n, errs := int32(0), int32(0)
	var successDirs []string
	var successMu sync.Mutex

	var wg sync.WaitGroup
	wg.Add(Keys.NumWorkers)
	work := make(chan workItem, Keys.NumWorkers*2)

	for range Keys.NumWorkers {
		go func() {
			defer wg.Done()
			for wi := range work {
				err := wi.level.toCheckpointBinary(wi.hostDir, from, to, m)
				if err != nil {
					if err == ErrNoNewArchiveData {
						continue
					}
					cclog.Errorf("[METRICSTORE]> binary checkpoint error for %s: %v", wi.hostDir, err)
					atomic.AddInt32(&errs, 1)
				} else {
					atomic.AddInt32(&n, 1)
					successMu.Lock()
					successDirs = append(successDirs, wi.hostDir)
					successMu.Unlock()
				}
			}
		}()
	}

	for i := range levels {
		hostDir := path.Join(dir, path.Join(selectors[i]...))
		work <- workItem{
			level:    levels[i],
			hostDir:  hostDir,
			selector: selectors[i],
		}
	}
	close(work)
	wg.Wait()

	if errs > 0 {
		return int(n), successDirs, fmt.Errorf("[METRICSTORE]> %d errors during binary checkpoint (%d successes)", errs, n)
	}
	return int(n), successDirs, nil
}

// toCheckpointBinary writes a binary snapshot file for a single host-level node.
// Uses atomic rename (write to .tmp then rename) to avoid partial reads on crash.
func (l *Level) toCheckpointBinary(dir string, from, to int64, m *MemoryStore) error {
	cf, err := l.toCheckpointFile(from, to, m)
	if err != nil {
		return err
	}
	if cf == nil {
		return ErrNoNewArchiveData
	}

	if err := os.MkdirAll(dir, CheckpointDirPerms); err != nil {
		return fmt.Errorf("mkdir %s: %w", dir, err)
	}

	// Write to a temp file first, then rename (atomic on POSIX).
	tmpPath := path.Join(dir, fmt.Sprintf("%d.bin.tmp", from))
	finalPath := path.Join(dir, fmt.Sprintf("%d.bin", from))

	f, err := os.OpenFile(tmpPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
	if err != nil {
		return fmt.Errorf("open binary snapshot %s: %w", tmpPath, err)
	}

	bw := bufio.NewWriter(f)
	if err := writeBinarySnapshotFile(bw, cf); err != nil {
		f.Close()
		os.Remove(tmpPath)
		return fmt.Errorf("write binary snapshot: %w", err)
	}
	if err := bw.Flush(); err != nil {
		f.Close()
		os.Remove(tmpPath)
		return err
	}
	f.Close()

	return os.Rename(tmpPath, finalPath)
}

// writeBinarySnapshotFile writes the binary snapshot file header and level tree.
func writeBinarySnapshotFile(w io.Writer, cf *CheckpointFile) error {
	if err := binary.Write(w, binary.LittleEndian, snapFileMagic); err != nil {
		return err
	}
	if err := binary.Write(w, binary.LittleEndian, cf.From); err != nil {
		return err
	}
	if err := binary.Write(w, binary.LittleEndian, cf.To); err != nil {
		return err
	}
	return writeBinaryLevel(w, cf)
}

// writeBinaryLevel recursively writes a CheckpointFile level in binary format.
func writeBinaryLevel(w io.Writer, cf *CheckpointFile) error {
	if err := binary.Write(w, binary.LittleEndian, uint32(len(cf.Metrics))); err != nil {
		return err
	}

	for name, metric := range cf.Metrics {
		if err := writeString16(w, name); err != nil {
			return err
		}
		if err := binary.Write(w, binary.LittleEndian, metric.Frequency); err != nil {
			return err
		}
		if err := binary.Write(w, binary.LittleEndian, metric.Start); err != nil {
			return err
		}
		if err := binary.Write(w, binary.LittleEndian, uint32(len(metric.Data))); err != nil {
			return err
		}
		for _, v := range metric.Data {
			if err := binary.Write(w, binary.LittleEndian, math.Float32bits(float32(v))); err != nil {
				return err
			}
		}
	}

	if err := binary.Write(w, binary.LittleEndian, uint32(len(cf.Children))); err != nil {
		return err
	}

	for name, child := range cf.Children {
		if err := writeString16(w, name); err != nil {
			return err
		}
		if err := writeBinaryLevel(w, child); err != nil {
			return err
		}
	}

	return nil
}

// writeString16 writes a 2-byte length-prefixed string to w.
func writeString16(w io.Writer, s string) error {
	if err := binary.Write(w, binary.LittleEndian, uint16(len(s))); err != nil {
		return err
	}
	_, err := io.WriteString(w, s)
	return err
}

// loadBinaryFile reads a binary snapshot file and loads data into the Level tree.
// The retention check (from) is applied to the file's 'to' timestamp.
func (l *Level) loadBinaryFile(m *MemoryStore, f *os.File, from int64) error {
	br := bufio.NewReader(f)

	var magic uint32
	if err := binary.Read(br, binary.LittleEndian, &magic); err != nil {
		return fmt.Errorf("[METRICSTORE]> binary snapshot: read magic: %w", err)
	}
	if magic != snapFileMagic {
		return fmt.Errorf("[METRICSTORE]> binary snapshot: invalid magic 0x%08X (expected 0x%08X)", magic, snapFileMagic)
	}

	var fileFrom, fileTo int64
	if err := binary.Read(br, binary.LittleEndian, &fileFrom); err != nil {
		return fmt.Errorf("[METRICSTORE]> binary snapshot: read from: %w", err)
	}
	if err := binary.Read(br, binary.LittleEndian, &fileTo); err != nil {
		return fmt.Errorf("[METRICSTORE]> binary snapshot: read to: %w", err)
	}

	if fileTo != 0 && fileTo < from {
		return nil // File is older than retention window, skip it
	}

	cf, err := readBinaryLevel(br)
	if err != nil {
		return fmt.Errorf("[METRICSTORE]> binary snapshot: read level tree: %w", err)
	}
	cf.From = fileFrom
	cf.To = fileTo

	return l.loadFile(cf, m)
}

// readBinaryLevel recursively reads a level from the binary snapshot format.
func readBinaryLevel(r io.Reader) (*CheckpointFile, error) {
	cf := &CheckpointFile{
		Metrics:  make(map[string]*CheckpointMetrics),
		Children: make(map[string]*CheckpointFile),
	}

	var numMetrics uint32
	if err := binary.Read(r, binary.LittleEndian, &numMetrics); err != nil {
		return nil, fmt.Errorf("read num_metrics: %w", err)
	}

	for range numMetrics {
		name, err := readString16(r)
		if err != nil {
			return nil, fmt.Errorf("read metric name: %w", err)
		}

		var freq, start int64
		if err := binary.Read(r, binary.LittleEndian, &freq); err != nil {
			return nil, fmt.Errorf("read frequency for %s: %w", name, err)
		}
		if err := binary.Read(r, binary.LittleEndian, &start); err != nil {
			return nil, fmt.Errorf("read start for %s: %w", name, err)
		}

		var numValues uint32
		if err := binary.Read(r, binary.LittleEndian, &numValues); err != nil {
			return nil, fmt.Errorf("read num_values for %s: %w", name, err)
		}

		data := make([]schema.Float, numValues)
		for i := range numValues {
			var bits uint32
			if err := binary.Read(r, binary.LittleEndian, &bits); err != nil {
				return nil, fmt.Errorf("read value[%d] for %s: %w", i, name, err)
			}
			data[i] = schema.Float(math.Float32frombits(bits))
		}

		cf.Metrics[name] = &CheckpointMetrics{
			Frequency: freq,
			Start:     start,
			Data:      data,
		}
	}

	var numChildren uint32
	if err := binary.Read(r, binary.LittleEndian, &numChildren); err != nil {
		return nil, fmt.Errorf("read num_children: %w", err)
	}

	for range numChildren {
		childName, err := readString16(r)
		if err != nil {
			return nil, fmt.Errorf("read child name: %w", err)
		}

		child, err := readBinaryLevel(r)
		if err != nil {
			return nil, fmt.Errorf("read child %s: %w", childName, err)
		}
		cf.Children[childName] = child
	}

	return cf, nil
}

// readString16 reads a 2-byte length-prefixed string from r.
func readString16(r io.Reader) (string, error) {
	var sLen uint16
	if err := binary.Read(r, binary.LittleEndian, &sLen); err != nil {
		return "", err
	}
	buf := make([]byte, sLen)
	if _, err := io.ReadFull(r, buf); err != nil {
		return "", err
	}
	return string(buf), nil
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// This file implements ingestion of InfluxDB line-protocol metric data received
     7→// over NATS. Each line encodes one metric sample with the following structure:
     8→//
     9→//	<measurement>[,cluster=<c>][,hostname=<h>][,type=<t>][,type-id=<id>][,subtype=<s>][,stype-id=<id>] value=<v> [<timestamp>]
    10→//
    11→// The measurement name identifies the metric (e.g. "cpu_load"). Tags provide
    12→// routing information (cluster, host) and optional sub-device selectors (type,
    13→// subtype). Only one field is expected per line: "value".
    14→//
    15→// After decoding, each sample is:
    16→//  1. Written to the in-memory store via ms.WriteToLevel.
    17→//  2. If the checkpoint format is "wal", also forwarded to the WAL staging
    18→//     goroutine via the WALMessages channel for durable write-ahead logging.
    19→package metricstore
    20→
    21→import (
    22→	"bytes"
    23→	"context"
    24→	"fmt"
    25→	"sync"
    26→	"sync/atomic"
    27→	"time"
    28→
    29→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    30→	"github.com/ClusterCockpit/cc-lib/v2/nats"
    31→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    32→	"github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
    33→)
    34→
    35→// ReceiveNats subscribes to all configured NATS subjects and feeds incoming
    36→// line-protocol messages into the MemoryStore.
    37→//
    38→// When workers > 1 a pool of goroutines drains a shared channel so that
    39→// multiple messages can be decoded in parallel. With workers == 1 the NATS
    40→// callback decodes inline (no channel overhead, lower latency).
    41→//
    42→// The function blocks until ctx is cancelled and all worker goroutines have
    43→// finished. It returns nil when the NATS client is not configured; callers
    44→// should treat that as a no-op rather than an error.
    45→func ReceiveNats(ms *MemoryStore,
    46→	workers int,
    47→	ctx context.Context,
    48→) error {
    49→	nc := nats.GetClient()
    50→
    51→	if nc == nil {
    52→		cclog.Warn("NATS client not initialized")
    53→		return nil
    54→	}
    55→
    56→	var wg sync.WaitGroup
    57→	msgs := make(chan []byte, max(workers*256, 8192))
    58→
    59→	for _, sc := range *Keys.Subscriptions {
    60→		clusterTag := sc.ClusterTag
    61→		if workers > 1 {
    62→			wg.Add(workers)
    63→
    64→			for range workers {
    65→				go func() {
    66→					defer wg.Done()
    67→					for m := range msgs {
    68→						dec := lineprotocol.NewDecoderWithBytes(m)
    69→						if err := DecodeLine(dec, ms, clusterTag); err != nil {
    70→							cclog.Errorf("error: %s", err.Error())
    71→						}
    72→					}
    73→				}()
    74→			}
    75→
    76→			nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
    77→				select {
    78→				case msgs <- data:
    79→				case <-ctx.Done():
    80→				}
    81→			})
    82→		} else {
    83→			nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
    84→				dec := lineprotocol.NewDecoderWithBytes(data)
    85→				if err := DecodeLine(dec, ms, clusterTag); err != nil {
    86→					cclog.Errorf("error: %s", err.Error())
    87→				}
    88→			})
    89→		}
    90→		cclog.Infof("NATS subscription to '%s' established", sc.SubscribeTo)
    91→	}
    92→
    93→	go func() {
    94→		<-ctx.Done()
    95→		close(msgs)
    96→	}()
    97→
    98→	wg.Wait()
    99→
   100→	return nil
   101→}
   102→
   103→// reorder prepends prefix to buf in-place when buf has enough spare capacity,
   104→// avoiding an allocation. Falls back to a regular append otherwise.
   105→//
   106→// It is used to assemble the "type<type-id>" and "subtype<stype-id>" selector
   107→// strings when the type tag arrives before the type-id tag in the line, so the
   108→// two byte slices need to be concatenated in tag-declaration order regardless
   109→// of wire order.
   110→func reorder(buf, prefix []byte) []byte {
   111→	n := len(prefix)
   112→	m := len(buf)
   113→	if cap(buf) < m+n {
   114→		return append(prefix[:n:n], buf...)
   115→	} else {
   116→		buf = buf[:n+m]
   117→		for i := m - 1; i >= 0; i-- {
   118→			buf[i+n] = buf[i]
   119→		}
   120→		for i := range n {
   121→			buf[i] = prefix[i]
   122→		}
   123→		return buf
   124→	}
   125→}
   126→
   127→// walDropped counts WAL messages dropped due to a full channel.
   128→// Logged periodically; not critical since binary snapshots capture the data.
   129→var walDropped atomic.Int64
   130→
   131→// decodeState holds the per-call scratch buffers used by DecodeLine.
   132→// Instances are recycled via decodeStatePool to avoid repeated allocations
   133→// during high-throughput ingestion.
   134→type decodeState struct {
   135→	// metricBuf holds a copy of the current measurement name (line-protocol
   136→	// measurement field). Copied because dec.Measurement() returns a slice
   137→	// that is invalidated by the next decoder call.
   138→	metricBuf []byte
   139→
   140→	// selector is the sub-device path passed to WriteToLevel and WALMessage
   141→	// (e.g. ["socket0"] or ["socket0", "memctrl1"]). Reused across lines.
   142→	selector []string
   143→
   144→	// typeBuf accumulates the concatenated "type"+"type-id" tag value for the
   145→	// current line. Reset at the start of each line's tag-decode loop.
   146→	typeBuf []byte
   147→
   148→	// subTypeBuf accumulates the concatenated "subtype"+"stype-id" tag value.
   149→	// Reset at the start of each line's tag-decode loop.
   150→	subTypeBuf []byte
   151→
   152→	// prevTypeBytes / prevTypeStr cache the last seen typeBuf content and its
   153→	// string conversion. Because consecutive lines in a batch typically address
   154→	// the same sub-device, the cache hit rate is very high and avoids
   155→	// repeated []byte→string allocations.
   156→	prevTypeBytes []byte
   157→	prevTypeStr   string
   158→
   159→	// prevSubTypeBytes / prevSubTypeStr are the same cache for the subtype.
   160→	prevSubTypeBytes []byte
   161→	prevSubTypeStr   string
   162→}
   163→
   164→// decodeStatePool recycles decodeState values across DecodeLine calls to
   165→// reduce GC pressure during sustained metric ingestion.
   166→var decodeStatePool = sync.Pool{
   167→	New: func() any {
   168→		return &decodeState{
   169→			metricBuf:  make([]byte, 0, 16),
   170→			selector:   make([]string, 0, 4),
   171→			typeBuf:    make([]byte, 0, 16),
   172→			subTypeBuf: make([]byte, 0, 16),
   173→		}
   174→	},
   175→}
   176→
   177→// DecodeLine reads all lines from dec (InfluxDB line-protocol) and writes each
   178→// decoded metric sample into ms.
   179→//
   180→// clusterDefault is used as the cluster name for lines that do not carry a
   181→// "cluster" tag. Callers typically supply the ClusterTag value from the NATS
   182→// subscription configuration.
   183→//
   184→// Performance notes:
   185→//   - A decodeState is obtained from decodeStatePool to reuse scratch buffers.
   186→//   - The Level pointer (host-level node in the metric tree) is cached across
   187→//     consecutive lines that share the same cluster+host pair to avoid
   188→//     repeated lock acquisitions on the root and cluster levels.
   189→//   - []byte→string conversions for type/subtype selectors are cached via
   190→//     prevType*/prevSubType* fields because batches typically repeat the same
   191→//     sub-device identifiers.
   192→//   - Timestamp parsing tries Second precision first; if that fails it retries
   193→//     Millisecond, Microsecond, and Nanosecond in turn. A missing timestamp
   194→//     falls back to time.Now().
   195→//
   196→// When the checkpoint format is "wal" each successfully decoded sample is also
   197→// sent to WALMessages so the WAL staging goroutine can persist it durably
   198→// before the next binary snapshot.
   199→func DecodeLine(dec *lineprotocol.Decoder,
   200→	ms *MemoryStore,
   201→	clusterDefault string,
   202→) error {
   203→	// Reduce allocations in loop:
   204→	t := time.Now()
   205→	metric := Metric{}
   206→	st := decodeStatePool.Get().(*decodeState)
   207→	defer decodeStatePool.Put(st)
   208→
   209→	// Optimize for the case where all lines in a "batch" are about the same
   210→	// cluster and host. By using `WriteToLevel` (level = host), we do not need
   211→	// to take the root- and cluster-level lock as often.
   212→	var lvl *Level = nil
   213→	prevCluster, prevHost := "", ""
   214→
   215→	var ok bool
   216→	for dec.Next() {
   217→		rawmeasurement, err := dec.Measurement()
   218→		if err != nil {
   219→			return err
   220→		}
   221→
   222→		// Needs to be copied because another call to dec.* would
   223→		// invalidate the returned slice.
   224→		st.metricBuf = append(st.metricBuf[:0], rawmeasurement...)
   225→
   226→		// The go compiler optimizes map[string(byteslice)] lookups:
   227→		metric.MetricConfig, ok = ms.Metrics[string(rawmeasurement)]
   228→		if !ok {
   229→			continue
   230→		}
   231→
   232→		st.typeBuf, st.subTypeBuf = st.typeBuf[:0], st.subTypeBuf[:0]
   233→		cluster, host := clusterDefault, ""
   234→		for {
   235→			key, val, err := dec.NextTag()
   236→			if err != nil {
   237→				return err
   238→			}
   239→			if key == nil {
   240→				break
   241→			}
   242→
   243→			// The go compiler optimizes string([]byte{...}) == "...":
   244→			switch string(key) {
   245→			case "cluster":
   246→				if string(val) == prevCluster {
   247→					cluster = prevCluster
   248→				} else {
   249→					cluster = string(val)
   250→					lvl = nil
   251→				}
   252→			case "hostname", "host":
   253→				if string(val) == prevHost {
   254→					host = prevHost
   255→				} else {
   256→					host = string(val)
   257→					lvl = nil
   258→				}
   259→			case "type":
   260→				if string(val) == "node" {
   261→					break
   262→				}
   263→
   264→				// We cannot be sure that the "type" tag comes before the "type-id" tag:
   265→				if len(st.typeBuf) == 0 {
   266→					st.typeBuf = append(st.typeBuf, val...)
   267→				} else {
   268→					st.typeBuf = reorder(st.typeBuf, val)
   269→				}
   270→			case "type-id":
   271→				st.typeBuf = append(st.typeBuf, val...)
   272→			case "subtype":
   273→				// We cannot be sure that the "subtype" tag comes before the "stype-id" tag:
   274→				if len(st.subTypeBuf) == 0 {
   275→					st.subTypeBuf = append(st.subTypeBuf, val...)
   276→				} else {
   277→					st.subTypeBuf = reorder(st.subTypeBuf, val)
   278→				}
   279→			case "stype-id":
   280→				st.subTypeBuf = append(st.subTypeBuf, val...)
   281→			default:
   282→			}
   283→		}
   284→
   285→		// If the cluster or host changed, the lvl was set to nil
   286→		if lvl == nil {
   287→			st.selector = st.selector[:2]
   288→			st.selector[0], st.selector[1] = cluster, host
   289→			lvl = ms.GetLevel(st.selector)
   290→			prevCluster, prevHost = cluster, host
   291→		}
   292→
   293→		// subtypes: cache []byte→string conversions; messages in a batch typically
   294→		// share the same type/subtype so the hit rate is very high.
   295→		st.selector = st.selector[:0]
   296→		if len(st.typeBuf) > 0 {
   297→			if !bytes.Equal(st.typeBuf, st.prevTypeBytes) {
   298→				st.prevTypeBytes = append(st.prevTypeBytes[:0], st.typeBuf...)
   299→				st.prevTypeStr = string(st.typeBuf)
   300→			}
   301→			st.selector = append(st.selector, st.prevTypeStr)
   302→			if len(st.subTypeBuf) > 0 {
   303→				if !bytes.Equal(st.subTypeBuf, st.prevSubTypeBytes) {
   304→					st.prevSubTypeBytes = append(st.prevSubTypeBytes[:0], st.subTypeBuf...)
   305→					st.prevSubTypeStr = string(st.subTypeBuf)
   306→				}
   307→				st.selector = append(st.selector, st.prevSubTypeStr)
   308→			}
   309→		}
   310→
   311→		for {
   312→			key, val, err := dec.NextField()
   313→			if err != nil {
   314→				return err
   315→			}
   316→
   317→			if key == nil {
   318→				break
   319→			}
   320→
   321→			if string(key) != "value" {
   322→				return fmt.Errorf("host %s: unknown field: '%s' (value: %#v)", host, string(key), val)
   323→			}
   324→
   325→			if val.Kind() == lineprotocol.Float {
   326→				metric.Value = schema.Float(val.FloatV())
   327→			} else if val.Kind() == lineprotocol.Int {
   328→				metric.Value = schema.Float(val.IntV())
   329→			} else if val.Kind() == lineprotocol.Uint {
   330→				metric.Value = schema.Float(val.UintV())
   331→			} else {
   332→				return fmt.Errorf("host %s: unsupported value type in message: %s", host, val.Kind().String())
   333→			}
   334→		}
   335→
   336→		if t, err = dec.Time(lineprotocol.Second, t); err != nil {
   337→			t = time.Now()
   338→			if t, err = dec.Time(lineprotocol.Millisecond, t); err != nil {
   339→				t = time.Now()
   340→				if t, err = dec.Time(lineprotocol.Microsecond, t); err != nil {
   341→					t = time.Now()
   342→					if t, err = dec.Time(lineprotocol.Nanosecond, t); err != nil {
   343→						return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
   344→					}
   345→				}
   346→			}
   347→		}
   348→
   349→		if err != nil {
   350→			return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
   351→		}
   352→
   353→		time := t.Unix()
   354→
   355→		if Keys.Checkpoints.FileFormat == "wal" {
   356→			msg := &WALMessage{
   357→				MetricName: string(st.metricBuf),
   358→				Cluster:    cluster,
   359→				Node:       host,
   360→				Selector:   append([]string{}, st.selector...),
   361→				Value:      metric.Value,
   362→				Timestamp:  time,
   363→			}
   364→			select {
   365→			case WALMessages <- msg:
   366→			default:
   367→				// WAL channel full — metric is written to memory store but not WAL.
   368→				// Next binary snapshot will capture it.
   369→				if dropped := walDropped.Add(1); dropped%10000 == 1 {
   370→					cclog.Warnf("[METRICSTORE]> WAL channel full, dropped %d messages (data safe in memory, next snapshot will capture)", dropped)
   371→				}
   372→			}
   373→		}
   374→
   375→		if err := ms.WriteToLevel(lvl, st.selector, time, []Metric{metric}); err != nil {
   376→			cclog.Warnf("write error for host %s metric %s at ts %d: %s", host, string(st.metricBuf), time, err.Error())
   377→			continue
   378→		}
   379→	}
   380→	return nil
   381→}
   382→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// This file implements ingestion of InfluxDB line-protocol metric data received
// over NATS. Each line encodes one metric sample with the following structure:
//
//	<measurement>[,cluster=<c>][,hostname=<h>][,type=<t>][,type-id=<id>][,subtype=<s>][,stype-id=<id>] value=<v> [<timestamp>]
//
// The measurement name identifies the metric (e.g. "cpu_load"). Tags provide
// routing information (cluster, host) and optional sub-device selectors (type,
// subtype). Only one field is expected per line: "value".
//
// After decoding, each sample is:
//  1. Written to the in-memory store via ms.WriteToLevel.
//  2. If the checkpoint format is "wal", also forwarded to the WAL staging
//     goroutine via the WALMessages channel for durable write-ahead logging.
package metricstore

import (
	"bytes"
	"context"
	"fmt"
	"sync"
	"sync/atomic"
	"time"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/nats"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	"github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
)

// ReceiveNats subscribes to all configured NATS subjects and feeds incoming
// line-protocol messages into the MemoryStore.
//
// When workers > 1 a pool of goroutines drains a shared channel so that
// multiple messages can be decoded in parallel. With workers == 1 the NATS
// callback decodes inline (no channel overhead, lower latency).
//
// The function blocks until ctx is cancelled and all worker goroutines have
// finished. It returns nil when the NATS client is not configured; callers
// should treat that as a no-op rather than an error.
func ReceiveNats(ms *MemoryStore,
	workers int,
	ctx context.Context,
) error {
	nc := nats.GetClient()

	if nc == nil {
		cclog.Warn("NATS client not initialized")
		return nil
	}

	var wg sync.WaitGroup
	msgs := make(chan []byte, max(workers*256, 8192))

	for _, sc := range *Keys.Subscriptions {
		clusterTag := sc.ClusterTag
		if workers > 1 {
			wg.Add(workers)

			for range workers {
				go func() {
					defer wg.Done()
					for m := range msgs {
						dec := lineprotocol.NewDecoderWithBytes(m)
						if err := DecodeLine(dec, ms, clusterTag); err != nil {
							cclog.Errorf("error: %s", err.Error())
						}
					}
				}()
			}

			nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
				select {
				case msgs <- data:
				case <-ctx.Done():
				}
			})
		} else {
			nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
				dec := lineprotocol.NewDecoderWithBytes(data)
				if err := DecodeLine(dec, ms, clusterTag); err != nil {
					cclog.Errorf("error: %s", err.Error())
				}
			})
		}
		cclog.Infof("NATS subscription to '%s' established", sc.SubscribeTo)
	}

	go func() {
		<-ctx.Done()
		close(msgs)
	}()

	wg.Wait()

	return nil
}

// reorder prepends prefix to buf in-place when buf has enough spare capacity,
// avoiding an allocation. Falls back to a regular append otherwise.
//
// It is used to assemble the "type<type-id>" and "subtype<stype-id>" selector
// strings when the type tag arrives before the type-id tag in the line, so the
// two byte slices need to be concatenated in tag-declaration order regardless
// of wire order.
func reorder(buf, prefix []byte) []byte {
	n := len(prefix)
	m := len(buf)
	if cap(buf) < m+n {
		return append(prefix[:n:n], buf...)
	} else {
		buf = buf[:n+m]
		for i := m - 1; i >= 0; i-- {
			buf[i+n] = buf[i]
		}
		for i := range n {
			buf[i] = prefix[i]
		}
		return buf
	}
}

// walDropped counts WAL messages dropped due to a full channel.
// Logged periodically; not critical since binary snapshots capture the data.
var walDropped atomic.Int64

// decodeState holds the per-call scratch buffers used by DecodeLine.
// Instances are recycled via decodeStatePool to avoid repeated allocations
// during high-throughput ingestion.
type decodeState struct {
	// metricBuf holds a copy of the current measurement name (line-protocol
	// measurement field). Copied because dec.Measurement() returns a slice
	// that is invalidated by the next decoder call.
	metricBuf []byte

	// selector is the sub-device path passed to WriteToLevel and WALMessage
	// (e.g. ["socket0"] or ["socket0", "memctrl1"]). Reused across lines.
	selector []string

	// typeBuf accumulates the concatenated "type"+"type-id" tag value for the
	// current line. Reset at the start of each line's tag-decode loop.
	typeBuf []byte

	// subTypeBuf accumulates the concatenated "subtype"+"stype-id" tag value.
	// Reset at the start of each line's tag-decode loop.
	subTypeBuf []byte

	// prevTypeBytes / prevTypeStr cache the last seen typeBuf content and its
	// string conversion. Because consecutive lines in a batch typically address
	// the same sub-device, the cache hit rate is very high and avoids
	// repeated []byte→string allocations.
	prevTypeBytes []byte
	prevTypeStr   string

	// prevSubTypeBytes / prevSubTypeStr are the same cache for the subtype.
	prevSubTypeBytes []byte
	prevSubTypeStr   string
}

// decodeStatePool recycles decodeState values across DecodeLine calls to
// reduce GC pressure during sustained metric ingestion.
var decodeStatePool = sync.Pool{
	New: func() any {
		return &decodeState{
			metricBuf:  make([]byte, 0, 16),
			selector:   make([]string, 0, 4),
			typeBuf:    make([]byte, 0, 16),
			subTypeBuf: make([]byte, 0, 16),
		}
	},
}

// DecodeLine reads all lines from dec (InfluxDB line-protocol) and writes each
// decoded metric sample into ms.
//
// clusterDefault is used as the cluster name for lines that do not carry a
// "cluster" tag. Callers typically supply the ClusterTag value from the NATS
// subscription configuration.
//
// Performance notes:
//   - A decodeState is obtained from decodeStatePool to reuse scratch buffers.
//   - The Level pointer (host-level node in the metric tree) is cached across
//     consecutive lines that share the same cluster+host pair to avoid
//     repeated lock acquisitions on the root and cluster levels.
//   - []byte→string conversions for type/subtype selectors are cached via
//     prevType*/prevSubType* fields because batches typically repeat the same
//     sub-device identifiers.
//   - Timestamp parsing tries Second precision first; if that fails it retries
//     Millisecond, Microsecond, and Nanosecond in turn. A missing timestamp
//     falls back to time.Now().
//
// When the checkpoint format is "wal" each successfully decoded sample is also
// sent to WALMessages so the WAL staging goroutine can persist it durably
// before the next binary snapshot.
func DecodeLine(dec *lineprotocol.Decoder,
	ms *MemoryStore,
	clusterDefault string,
) error {
	// Reduce allocations in loop:
	t := time.Now()
	metric := Metric{}
	st := decodeStatePool.Get().(*decodeState)
	defer decodeStatePool.Put(st)

	// Optimize for the case where all lines in a "batch" are about the same
	// cluster and host. By using `WriteToLevel` (level = host), we do not need
	// to take the root- and cluster-level lock as often.
	var lvl *Level = nil
	prevCluster, prevHost := "", ""

	var ok bool
	for dec.Next() {
		rawmeasurement, err := dec.Measurement()
		if err != nil {
			return err
		}

		// Needs to be copied because another call to dec.* would
		// invalidate the returned slice.
		st.metricBuf = append(st.metricBuf[:0], rawmeasurement...)

		// The go compiler optimizes map[string(byteslice)] lookups:
		metric.MetricConfig, ok = ms.Metrics[string(rawmeasurement)]
		if !ok {
			continue
		}

		st.typeBuf, st.subTypeBuf = st.typeBuf[:0], st.subTypeBuf[:0]
		cluster, host := clusterDefault, ""
		for {
			key, val, err := dec.NextTag()
			if err != nil {
				return err
			}
			if key == nil {
				break
			}

			// The go compiler optimizes string([]byte{...}) == "...":
			switch string(key) {
			case "cluster":
				if string(val) == prevCluster {
					cluster = prevCluster
				} else {
					cluster = string(val)
					lvl = nil
				}
			case "hostname", "host":
				if string(val) == prevHost {
					host = prevHost
				} else {
					host = string(val)
					lvl = nil
				}
			case "type":
				if string(val) == "node" {
					break
				}

				// We cannot be sure that the "type" tag comes before the "type-id" tag:
				if len(st.typeBuf) == 0 {
					st.typeBuf = append(st.typeBuf, val...)
				} else {
					st.typeBuf = reorder(st.typeBuf, val)
				}
			case "type-id":
				st.typeBuf = append(st.typeBuf, val...)
			case "subtype":
				// We cannot be sure that the "subtype" tag comes before the "stype-id" tag:
				if len(st.subTypeBuf) == 0 {
					st.subTypeBuf = append(st.subTypeBuf, val...)
				} else {
					st.subTypeBuf = reorder(st.subTypeBuf, val)
				}
			case "stype-id":
				st.subTypeBuf = append(st.subTypeBuf, val...)
			default:
			}
		}

		// If the cluster or host changed, the lvl was set to nil
		if lvl == nil {
			st.selector = st.selector[:2]
			st.selector[0], st.selector[1] = cluster, host
			lvl = ms.GetLevel(st.selector)
			prevCluster, prevHost = cluster, host
		}

		// subtypes: cache []byte→string conversions; messages in a batch typically
		// share the same type/subtype so the hit rate is very high.
		st.selector = st.selector[:0]
		if len(st.typeBuf) > 0 {
			if !bytes.Equal(st.typeBuf, st.prevTypeBytes) {
				st.prevTypeBytes = append(st.prevTypeBytes[:0], st.typeBuf...)
				st.prevTypeStr = string(st.typeBuf)
			}
			st.selector = append(st.selector, st.prevTypeStr)
			if len(st.subTypeBuf) > 0 {
				if !bytes.Equal(st.subTypeBuf, st.prevSubTypeBytes) {
					st.prevSubTypeBytes = append(st.prevSubTypeBytes[:0], st.subTypeBuf...)
					st.prevSubTypeStr = string(st.subTypeBuf)
				}
				st.selector = append(st.selector, st.prevSubTypeStr)
			}
		}

		for {
			key, val, err := dec.NextField()
			if err != nil {
				return err
			}

			if key == nil {
				break
			}

			if string(key) != "value" {
				return fmt.Errorf("host %s: unknown field: '%s' (value: %#v)", host, string(key), val)
			}

			if val.Kind() == lineprotocol.Float {
				metric.Value = schema.Float(val.FloatV())
			} else if val.Kind() == lineprotocol.Int {
				metric.Value = schema.Float(val.IntV())
			} else if val.Kind() == lineprotocol.Uint {
				metric.Value = schema.Float(val.UintV())
			} else {
				return fmt.Errorf("host %s: unsupported value type in message: %s", host, val.Kind().String())
			}
		}

		if t, err = dec.Time(lineprotocol.Second, t); err != nil {
			t = time.Now()
			if t, err = dec.Time(lineprotocol.Millisecond, t); err != nil {
				t = time.Now()
				if t, err = dec.Time(lineprotocol.Microsecond, t); err != nil {
					t = time.Now()
					if t, err = dec.Time(lineprotocol.Nanosecond, t); err != nil {
						return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
					}
				}
			}
		}

		if err != nil {
			return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
		}

		time := t.Unix()

		if Keys.Checkpoints.FileFormat == "wal" {
			msg := &WALMessage{
				MetricName: string(st.metricBuf),
				Cluster:    cluster,
				Node:       host,
				Selector:   append([]string{}, st.selector...),
				Value:      metric.Value,
				Timestamp:  time,
			}
			select {
			case WALMessages <- msg:
			default:
				// WAL channel full — metric is written to memory store but not WAL.
				// Next binary snapshot will capture it.
				if dropped := walDropped.Add(1); dropped%10000 == 1 {
					cclog.Warnf("[METRICSTORE]> WAL channel full, dropped %d messages (data safe in memory, next snapshot will capture)", dropped)
				}
			}
		}

		if err := ms.WriteToLevel(lvl, st.selector, time, []Metric{metric}); err != nil {
			cclog.Warnf("write error for host %s metric %s at ts %d: %s", host, string(st.metricBuf), time, err.Error())
			continue
		}
	}
	return nil
}