There are occasional warnings in the log: [WARNING]  lineprotocol.go:368: [METRICSTORE]> WAL shard channel full, dropped 7220001 messages (data safe in memory, next snapshot will capture). Is this a problem? And how can it be fixed?


Let me explore the relevant code to understand the WAL message flow and the warning.


Agent

Explore WAL message flow

Prompt:
I need to understand the WAL message flow in the metricstore, specifically around WAL shard channels getting full and dropping messages.

Explore these areas thoroughly:

1. In `pkg/metricstore/lineprotocol.go` — find where WAL messages are sent to channels, especially around line 368 where the warning "WAL shard channel full, dropped" is logged. Understand how messages are produced.

2. In `pkg/metricstore/metricstore.go` and `pkg/metricstore/walCheckpoint.go` — find where WAL channels are created (their buffer sizes), how `WALStaging()` or equivalent consumers work, and how messages are consumed/written.

3. Look for any configuration of channel buffer sizes, shard counts, or concurrency settings related to WAL.

Report back:
- The exact channel creation code (buffer sizes)
- The producer code (how messages are sent, what happens when channel is full)
- The consumer code (how messages are drained/written)
- Any batching or flow control mechanisms
- The full context around the warning message


I need to understand the WAL message flow in the metricstore, specifically around WAL shard channels getting full and dropping messages.

Explore these areas thoroughly:

1. In `pkg/metricstore/lineprotocol.go` — find where WAL messages are sent to channels, especially around line 368 where the warning "WAL shard channel full, dropped" is logged. Understand how messages are produced.

2. In `pkg/metricstore/metricstore.go` and `pkg/metricstore/walCheckpoint.go` — find where WAL channels are created (their buffer sizes), how `WALStaging()` or equivalent consumers work, and how messages are consumed/written.

3. Look for any configuration of channel buffer sizes, shard counts, or concurrency settings related to WAL.

Report back:
- The exact channel creation code (buffer sizes)
- The producer code (how messages are sent, what happens when channel is full)
- The consumer code (how messages are drained/written)
- Any batching or flow control mechanisms
- The full context around the warning message

Prompt:
I need to understand the WAL message flow in the metricstore, specifically around WAL shard channels getting full and dropping messages.

Explore these areas thoroughly:

1. In `pkg/metricstore/lineprotocol.go` — find where WAL messages are sent to channels, especially around line 368 where the warning "WAL shard channel full, dropped" is logged. Understand how messages are produced.

2. In `pkg/metricstore/metricstore.go` and `pkg/metricstore/walCheckpoint.go` — find where WAL channels are created (their buffer sizes), how `WALStaging()` or equivalent consumers work, and how messages are consumed/written.

3. Look for any configuration of channel buffer sizes, shard counts, or concurrency settings related to WAL.

Report back:
- The exact channel creation code (buffer sizes)
- The producer code (how messages are sent, what happens when channel is full)
- The consumer code (how messages are drained/written)
- Any batching or flow control mechanisms
- The full context around the warning message


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// This file implements ingestion of InfluxDB line-protocol metric data received
     7→// over NATS. Each line encodes one metric sample with the following structure:
     8→//
     9→//	<measurement>[,cluster=<c>][,hostname=<h>][,type=<t>][,type-id=<id>][,subtype=<s>][,stype-id=<id>] value=<v> [<timestamp>]
    10→//
    11→// The measurement name identifies the metric (e.g. "cpu_load"). Tags provide
    12→// routing information (cluster, host) and optional sub-device selectors (type,
    13→// subtype). Only one field is expected per line: "value".
    14→//
    15→// After decoding, each sample is:
    16→//  1. Written to the in-memory store via ms.WriteToLevel.
    17→//  2. If the checkpoint format is "wal", also forwarded to the WAL staging
    18→//     goroutine via the WALMessages channel for durable write-ahead logging.
    19→package metricstore
    20→
    21→import (
    22→	"bytes"
    23→	"context"
    24→	"fmt"
    25→	"sync"
    26→	"sync/atomic"
    27→	"time"
    28→
    29→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    30→	"github.com/ClusterCockpit/cc-lib/v2/nats"
    31→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    32→	"github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
    33→)
    34→
    35→// ReceiveNats subscribes to all configured NATS subjects and feeds incoming
    36→// line-protocol messages into the MemoryStore.
    37→//
    38→// When workers > 1 a pool of goroutines drains a shared channel so that
    39→// multiple messages can be decoded in parallel. With workers == 1 the NATS
    40→// callback decodes inline (no channel overhead, lower latency).
    41→//
    42→// The function blocks until ctx is cancelled and all worker goroutines have
    43→// finished. It returns nil when the NATS client is not configured; callers
    44→// should treat that as a no-op rather than an error.
    45→func ReceiveNats(ms *MemoryStore,
    46→	workers int,
    47→	ctx context.Context,
    48→) error {
    49→	nc := nats.GetClient()
    50→
    51→	if nc == nil {
    52→		cclog.Warn("NATS client not initialized")
    53→		return nil
    54→	}
    55→
    56→	var wg sync.WaitGroup
    57→	msgs := make(chan []byte, max(workers*256, 8192))
    58→
    59→	for _, sc := range *Keys.Subscriptions {
    60→		clusterTag := sc.ClusterTag
    61→		if workers > 1 {
    62→			wg.Add(workers)
    63→
    64→			for range workers {
    65→				go func() {
    66→					defer wg.Done()
    67→					for m := range msgs {
    68→						dec := lineprotocol.NewDecoderWithBytes(m)
    69→						if err := DecodeLine(dec, ms, clusterTag); err != nil {
    70→							cclog.Errorf("error: %s", err.Error())
    71→						}
    72→					}
    73→				}()
    74→			}
    75→
    76→			nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
    77→				select {
    78→				case msgs <- data:
    79→				case <-ctx.Done():
    80→				}
    81→			})
    82→		} else {
    83→			nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
    84→				dec := lineprotocol.NewDecoderWithBytes(data)
    85→				if err := DecodeLine(dec, ms, clusterTag); err != nil {
    86→					cclog.Errorf("error: %s", err.Error())
    87→				}
    88→			})
    89→		}
    90→		cclog.Infof("NATS subscription to '%s' established", sc.SubscribeTo)
    91→	}
    92→
    93→	go func() {
    94→		<-ctx.Done()
    95→		close(msgs)
    96→	}()
    97→
    98→	wg.Wait()
    99→
   100→	return nil
   101→}
   102→
   103→// reorder prepends prefix to buf in-place when buf has enough spare capacity,
   104→// avoiding an allocation. Falls back to a regular append otherwise.
   105→//
   106→// It is used to assemble the "type<type-id>" and "subtype<stype-id>" selector
   107→// strings when the type tag arrives before the type-id tag in the line, so the
   108→// two byte slices need to be concatenated in tag-declaration order regardless
   109→// of wire order.
   110→func reorder(buf, prefix []byte) []byte {
   111→	n := len(prefix)
   112→	m := len(buf)
   113→	if cap(buf) < m+n {
   114→		return append(prefix[:n:n], buf...)
   115→	} else {
   116→		buf = buf[:n+m]
   117→		for i := m - 1; i >= 0; i-- {
   118→			buf[i+n] = buf[i]
   119→		}
   120→		for i := range n {
   121→			buf[i] = prefix[i]
   122→		}
   123→		return buf
   124→	}
   125→}
   126→
   127→// walDropped counts WAL messages dropped due to a full channel.
   128→// Logged periodically; not critical since binary snapshots capture the data.
   129→var walDropped atomic.Int64
   130→
   131→// decodeState holds the per-call scratch buffers used by DecodeLine.
   132→// Instances are recycled via decodeStatePool to avoid repeated allocations
   133→// during high-throughput ingestion.
   134→type decodeState struct {
   135→	// metricBuf holds a copy of the current measurement name (line-protocol
   136→	// measurement field). Copied because dec.Measurement() returns a slice
   137→	// that is invalidated by the next decoder call.
   138→	metricBuf []byte
   139→
   140→	// selector is the sub-device path passed to WriteToLevel and WALMessage
   141→	// (e.g. ["socket0"] or ["socket0", "memctrl1"]). Reused across lines.
   142→	selector []string
   143→
   144→	// typeBuf accumulates the concatenated "type"+"type-id" tag value for the
   145→	// current line. Reset at the start of each line's tag-decode loop.
   146→	typeBuf []byte
   147→
   148→	// subTypeBuf accumulates the concatenated "subtype"+"stype-id" tag value.
   149→	// Reset at the start of each line's tag-decode loop.
   150→	subTypeBuf []byte
   151→
   152→	// prevTypeBytes / prevTypeStr cache the last seen typeBuf content and its
   153→	// string conversion. Because consecutive lines in a batch typically address
   154→	// the same sub-device, the cache hit rate is very high and avoids
   155→	// repeated []byte→string allocations.
   156→	prevTypeBytes []byte
   157→	prevTypeStr   string
   158→
   159→	// prevSubTypeBytes / prevSubTypeStr are the same cache for the subtype.
   160→	prevSubTypeBytes []byte
   161→	prevSubTypeStr   string
   162→}
   163→
   164→// decodeStatePool recycles decodeState values across DecodeLine calls to
   165→// reduce GC pressure during sustained metric ingestion.
   166→var decodeStatePool = sync.Pool{
   167→	New: func() any {
   168→		return &decodeState{
   169→			metricBuf:  make([]byte, 0, 16),
   170→			selector:   make([]string, 0, 4),
   171→			typeBuf:    make([]byte, 0, 16),
   172→			subTypeBuf: make([]byte, 0, 16),
   173→		}
   174→	},
   175→}
   176→
   177→// DecodeLine reads all lines from dec (InfluxDB line-protocol) and writes each
   178→// decoded metric sample into ms.
   179→//
   180→// clusterDefault is used as the cluster name for lines that do not carry a
   181→// "cluster" tag. Callers typically supply the ClusterTag value from the NATS
   182→// subscription configuration.
   183→//
   184→// Performance notes:
   185→//   - A decodeState is obtained from decodeStatePool to reuse scratch buffers.
   186→//   - The Level pointer (host-level node in the metric tree) is cached across
   187→//     consecutive lines that share the same cluster+host pair to avoid
   188→//     repeated lock acquisitions on the root and cluster levels.
   189→//   - []byte→string conversions for type/subtype selectors are cached via
   190→//     prevType*/prevSubType* fields because batches typically repeat the same
   191→//     sub-device identifiers.
   192→//   - Timestamp parsing tries Second precision first; if that fails it retries
   193→//     Millisecond, Microsecond, and Nanosecond in turn. A missing timestamp
   194→//     falls back to time.Now().
   195→//
   196→// When the checkpoint format is "wal" each successfully decoded sample is also
   197→// sent to WALMessages so the WAL staging goroutine can persist it durably
   198→// before the next binary snapshot.
   199→func DecodeLine(dec *lineprotocol.Decoder,
   200→	ms *MemoryStore,
   201→	clusterDefault string,
   202→) error {
   203→	// Reduce allocations in loop:
   204→	t := time.Now()
   205→	metric := Metric{}
   206→	st := decodeStatePool.Get().(*decodeState)
   207→	defer decodeStatePool.Put(st)
   208→
   209→	// Optimize for the case where all lines in a "batch" are about the same
   210→	// cluster and host. By using `WriteToLevel` (level = host), we do not need
   211→	// to take the root- and cluster-level lock as often.
   212→	var lvl *Level = nil
   213→	prevCluster, prevHost := "", ""
   214→
   215→	var ok bool
   216→	for dec.Next() {
   217→		rawmeasurement, err := dec.Measurement()
   218→		if err != nil {
   219→			return err
   220→		}
   221→
   222→		// Needs to be copied because another call to dec.* would
   223→		// invalidate the returned slice.
   224→		st.metricBuf = append(st.metricBuf[:0], rawmeasurement...)
   225→
   226→		// The go compiler optimizes map[string(byteslice)] lookups:
   227→		metric.MetricConfig, ok = ms.Metrics[string(rawmeasurement)]
   228→		if !ok {
   229→			continue
   230→		}
   231→
   232→		st.typeBuf, st.subTypeBuf = st.typeBuf[:0], st.subTypeBuf[:0]
   233→		cluster, host := clusterDefault, ""
   234→		for {
   235→			key, val, err := dec.NextTag()
   236→			if err != nil {
   237→				return err
   238→			}
   239→			if key == nil {
   240→				break
   241→			}
   242→
   243→			// The go compiler optimizes string([]byte{...}) == "...":
   244→			switch string(key) {
   245→			case "cluster":
   246→				if string(val) == prevCluster {
   247→					cluster = prevCluster
   248→				} else {
   249→					cluster = string(val)
   250→					lvl = nil
   251→				}
   252→			case "hostname", "host":
   253→				if string(val) == prevHost {
   254→					host = prevHost
   255→				} else {
   256→					host = string(val)
   257→					lvl = nil
   258→				}
   259→			case "type":
   260→				if string(val) == "node" {
   261→					break
   262→				}
   263→
   264→				// We cannot be sure that the "type" tag comes before the "type-id" tag:
   265→				if len(st.typeBuf) == 0 {
   266→					st.typeBuf = append(st.typeBuf, val...)
   267→				} else {
   268→					st.typeBuf = reorder(st.typeBuf, val)
   269→				}
   270→			case "type-id":
   271→				st.typeBuf = append(st.typeBuf, val...)
   272→			case "subtype":
   273→				// We cannot be sure that the "subtype" tag comes before the "stype-id" tag:
   274→				if len(st.subTypeBuf) == 0 {
   275→					st.subTypeBuf = append(st.subTypeBuf, val...)
   276→				} else {
   277→					st.subTypeBuf = reorder(st.subTypeBuf, val)
   278→				}
   279→			case "stype-id":
   280→				st.subTypeBuf = append(st.subTypeBuf, val...)
   281→			default:
   282→			}
   283→		}
   284→
   285→		// If the cluster or host changed, the lvl was set to nil
   286→		if lvl == nil {
   287→			st.selector = st.selector[:2]
   288→			st.selector[0], st.selector[1] = cluster, host
   289→			lvl = ms.GetLevel(st.selector)
   290→			prevCluster, prevHost = cluster, host
   291→		}
   292→
   293→		// subtypes: cache []byte→string conversions; messages in a batch typically
   294→		// share the same type/subtype so the hit rate is very high.
   295→		st.selector = st.selector[:0]
   296→		if len(st.typeBuf) > 0 {
   297→			if !bytes.Equal(st.typeBuf, st.prevTypeBytes) {
   298→				st.prevTypeBytes = append(st.prevTypeBytes[:0], st.typeBuf...)
   299→				st.prevTypeStr = string(st.typeBuf)
   300→			}
   301→			st.selector = append(st.selector, st.prevTypeStr)
   302→			if len(st.subTypeBuf) > 0 {
   303→				if !bytes.Equal(st.subTypeBuf, st.prevSubTypeBytes) {
   304→					st.prevSubTypeBytes = append(st.prevSubTypeBytes[:0], st.subTypeBuf...)
   305→					st.prevSubTypeStr = string(st.subTypeBuf)
   306→				}
   307→				st.selector = append(st.selector, st.prevSubTypeStr)
   308→			}
   309→		}
   310→
   311→		for {
   312→			key, val, err := dec.NextField()
   313→			if err != nil {
   314→				return err
   315→			}
   316→
   317→			if key == nil {
   318→				break
   319→			}
   320→
   321→			if string(key) != "value" {
   322→				return fmt.Errorf("host %s: unknown field: '%s' (value: %#v)", host, string(key), val)
   323→			}
   324→
   325→			if val.Kind() == lineprotocol.Float {
   326→				metric.Value = schema.Float(val.FloatV())
   327→			} else if val.Kind() == lineprotocol.Int {
   328→				metric.Value = schema.Float(val.IntV())
   329→			} else if val.Kind() == lineprotocol.Uint {
   330→				metric.Value = schema.Float(val.UintV())
   331→			} else {
   332→				return fmt.Errorf("host %s: unsupported value type in message: %s", host, val.Kind().String())
   333→			}
   334→		}
   335→
   336→		if t, err = dec.Time(lineprotocol.Second, t); err != nil {
   337→			t = time.Now()
   338→			if t, err = dec.Time(lineprotocol.Millisecond, t); err != nil {
   339→				t = time.Now()
   340→				if t, err = dec.Time(lineprotocol.Microsecond, t); err != nil {
   341→					t = time.Now()
   342→					if t, err = dec.Time(lineprotocol.Nanosecond, t); err != nil {
   343→						return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
   344→					}
   345→				}
   346→			}
   347→		}
   348→
   349→		if err != nil {
   350→			return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
   351→		}
   352→
   353→		time := t.Unix()
   354→
   355→		if Keys.Checkpoints.FileFormat == "wal" {
   356→			msg := &WALMessage{
   357→				MetricName: string(st.metricBuf),
   358→				Cluster:    cluster,
   359→				Node:       host,
   360→				Selector:   append([]string{}, st.selector...),
   361→				Value:      metric.Value,
   362→				Timestamp:  time,
   363→			}
   364→			if !SendWALMessage(msg) {
   365→				// WAL shard channel full — metric is written to memory store but not WAL.
   366→				// Next binary snapshot will capture it.
   367→				if dropped := walDropped.Add(1); dropped%10000 == 1 {
   368→					cclog.Warnf("[METRICSTORE]> WAL shard channel full, dropped %d messages (data safe in memory, next snapshot will capture)", dropped)
   369→				}
   370→			}
   371→		}
   372→
   373→		if err := ms.WriteToLevel(lvl, st.selector, time, []Metric{metric}); err != nil {
   374→			cclog.Warnf("write error for host %s metric %s at ts %d: %s", host, string(st.metricBuf), time, err.Error())
   375→			continue
   376→		}
   377→	}
   378→	return nil
   379→}
   380→


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstore provides walCheckpoint.go: WAL-based checkpoint implementation.
     7→//
     8→// This replaces the Avro shadow tree with an append-only Write-Ahead Log (WAL)
     9→// per host, eliminating the extra memory overhead of the AvroStore and providing
    10→// truly continuous (per-write) crash safety.
    11→//
    12→// # Architecture
    13→//
    14→//	Metric write (DecodeLine)
    15→//	      │
    16→//	      ├─► WriteToLevel()    → main MemoryStore (unchanged)
    17→//	      │
    18→//	      └─► WALMessages channel
    19→//	               │
    20→//	               ▼
    21→//	        WALStaging goroutine
    22→//	               │
    23→//	               ▼
    24→//	        checkpoints/cluster/host/current.wal  (append-only, binary)
    25→//
    26→//	Periodic checkpoint (Checkpointing goroutine):
    27→//	  1. Write <timestamp>.bin snapshot (column-oriented, from main tree)
    28→//	  2. Signal WALStaging to truncate current.wal per host
    29→//
    30→//	On restart (FromCheckpoint):
    31→//	  1. Load most recent <timestamp>.bin snapshot
    32→//	  2. Replay current.wal (overwrite-safe: buffer.write handles duplicate timestamps)
    33→//
    34→// # WAL Record Format
    35→//
    36→//	[4B magic 0xCC1DA7A1][4B payload_len][payload][4B CRC32]
    37→//
    38→//	payload:
    39→//	  [8B timestamp int64]
    40→//	  [2B metric_name_len uint16][N metric name bytes]
    41→//	  [1B selector_count uint8]
    42→//	  per selector: [1B selector_len uint8][M selector bytes]
    43→//	  [4B value float32 bits]
    44→//
    45→// # Binary Snapshot Format
    46→//
    47→//	[4B magic 0xCC5B0001][8B from int64][8B to int64]
    48→//	Level tree (recursive):
    49→//	  [4B num_metrics uint32]
    50→//	  per metric:
    51→//	    [2B name_len uint16][N name bytes]
    52→//	    [8B frequency int64][8B start int64]
    53→//	    [4B num_values uint32][num_values × 4B float32]
    54→//	  [4B num_children uint32]
    55→//	  per child: [2B name_len uint16][N name bytes] + Level (recursive)
    56→package metricstore
    57→
    58→import (
    59→	"bufio"
    60→	"context"
    61→	"encoding/binary"
    62→	"fmt"
    63→	"hash/crc32"
    64→	"hash/fnv"
    65→	"io"
    66→	"math"
    67→	"os"
    68→	"path"
    69→	"strings"
    70→	"sync"
    71→	"sync/atomic"
    72→
    73→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    74→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    75→)
    76→
    77→// Magic numbers for binary formats.
    78→const (
    79→	walFileMagic   = uint32(0xCC1DA701) // WAL file header magic
    80→	walRecordMagic = uint32(0xCC1DA7A1) // WAL record magic
    81→	snapFileMagic  = uint32(0xCC5B0001) // Binary snapshot magic
    82→)
    83→
    84→// walShardChs holds per-shard channels for WAL messages.
    85→// Initialized by WALStaging; nil when WAL is not active.
    86→var walShardChs []chan *WALMessage
    87→
    88→// walShardRotateChs holds per-shard channels for WAL rotation requests.
    89→var walShardRotateChs []chan walRotateReq
    90→
    91→// walNumShards stores the number of shards (set during WALStaging init).
    92→var walNumShards int
    93→
    94→// WALMessage represents a single metric write to be appended to the WAL.
    95→// Cluster and Node are NOT stored in the WAL record (inferred from file path).
    96→type WALMessage struct {
    97→	MetricName string
    98→	Cluster    string
    99→	Node       string
   100→	Selector   []string
   101→	Value      schema.Float
   102→	Timestamp  int64
   103→}
   104→
   105→// walRotateReq requests WAL file rotation for a specific host directory.
   106→// The done channel is closed by the WAL goroutine when rotation is complete.
   107→type walRotateReq struct {
   108→	hostDir string
   109→	done    chan struct{}
   110→}
   111→
   112→// walFileState holds an open WAL file handle and buffered writer for one host directory.
   113→type walFileState struct {
   114→	f *os.File
   115→	w *bufio.Writer
   116→}
   117→
   118→// walShardIndex computes which shard a message belongs to based on cluster+node.
   119→// Uses FNV-1a hash for fast, well-distributed mapping.
   120→func walShardIndex(cluster, node string) int {
   121→	h := fnv.New32a()
   122→	h.Write([]byte(cluster))
   123→	h.Write([]byte{0})
   124→	h.Write([]byte(node))
   125→	return int(h.Sum32()) % walNumShards
   126→}
   127→
   128→// SendWALMessage routes a WAL message to the appropriate shard channel.
   129→// Returns false if the channel is full (message dropped).
   130→func SendWALMessage(msg *WALMessage) bool {
   131→	if walShardChs == nil {
   132→		return false
   133→	}
   134→	shard := walShardIndex(msg.Cluster, msg.Node)
   135→	select {
   136→	case walShardChs[shard] <- msg:
   137→		return true
   138→	default:
   139→		return false
   140→	}
   141→}
   142→
   143→// WALStaging starts N sharded background goroutines that receive WALMessage items
   144→// and append binary WAL records to per-host current.wal files.
   145→// Each shard owns a subset of hosts (determined by hash of cluster+node),
   146→// parallelizing serialization and I/O while keeping per-host writes serial.
   147→func WALStaging(wg *sync.WaitGroup, ctx context.Context) {
   148→	if Keys.Checkpoints.FileFormat == "json" {
   149→		return
   150→	}
   151→
   152→	walNumShards = max(Keys.NumWorkers, 1)
   153→	chBufSize := max(65536/walNumShards, 1024)
   154→
   155→	walShardChs = make([]chan *WALMessage, walNumShards)
   156→	walShardRotateChs = make([]chan walRotateReq, walNumShards)
   157→
   158→	for i := range walNumShards {
   159→		walShardChs[i] = make(chan *WALMessage, chBufSize)
   160→		walShardRotateChs[i] = make(chan walRotateReq, 64)
   161→	}
   162→
   163→	for i := range walNumShards {
   164→		msgCh := walShardChs[i]
   165→		rotateCh := walShardRotateChs[i]
   166→
   167→		wg.Go(func() {
   168→			hostFiles := make(map[string]*walFileState)
   169→
   170→			defer func() {
   171→				for _, ws := range hostFiles {
   172→					if ws.f != nil {
   173→						ws.w.Flush()
   174→						ws.f.Close()
   175→					}
   176→				}
   177→			}()
   178→
   179→			getOrOpenWAL := func(hostDir string) *walFileState {
   180→				ws, ok := hostFiles[hostDir]
   181→				if ok {
   182→					return ws
   183→				}
   184→
   185→				if err := os.MkdirAll(hostDir, CheckpointDirPerms); err != nil {
   186→					cclog.Errorf("[METRICSTORE]> WAL: mkdir %s: %v", hostDir, err)
   187→					return nil
   188→				}
   189→
   190→				walPath := path.Join(hostDir, "current.wal")
   191→				f, err := os.OpenFile(walPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, CheckpointFilePerms)
   192→				if err != nil {
   193→					cclog.Errorf("[METRICSTORE]> WAL: open %s: %v", walPath, err)
   194→					return nil
   195→				}
   196→
   197→				w := bufio.NewWriter(f)
   198→
   199→				// Write file header magic if file is new (empty).
   200→				info, err := f.Stat()
   201→				if err == nil && info.Size() == 0 {
   202→					var hdr [4]byte
   203→					binary.LittleEndian.PutUint32(hdr[:], walFileMagic)
   204→					if _, err := w.Write(hdr[:]); err != nil {
   205→						cclog.Errorf("[METRICSTORE]> WAL: write header %s: %v", walPath, err)
   206→						f.Close()
   207→						return nil
   208→					}
   209→				}
   210→
   211→				ws = &walFileState{f: f, w: w}
   212→				hostFiles[hostDir] = ws
   213→				return ws
   214→			}
   215→
   216→			processMsg := func(msg *WALMessage) {
   217→				hostDir := path.Join(Keys.Checkpoints.RootDir, msg.Cluster, msg.Node)
   218→				ws := getOrOpenWAL(hostDir)
   219→				if ws == nil {
   220→					return
   221→				}
   222→				if err := writeWALRecordDirect(ws.w, msg); err != nil {
   223→					cclog.Errorf("[METRICSTORE]> WAL: write record: %v", err)
   224→				}
   225→			}
   226→
   227→			processRotate := func(req walRotateReq) {
   228→				ws, ok := hostFiles[req.hostDir]
   229→				if ok && ws.f != nil {
   230→					ws.w.Flush()
   231→					ws.f.Close()
   232→					walPath := path.Join(req.hostDir, "current.wal")
   233→					if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
   234→						cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
   235→					}
   236→					delete(hostFiles, req.hostDir)
   237→				}
   238→				close(req.done)
   239→			}
   240→
   241→			flushAll := func() {
   242→				for _, ws := range hostFiles {
   243→					if ws.f != nil {
   244→						ws.w.Flush()
   245→					}
   246→				}
   247→			}
   248→
   249→			drain := func() {
   250→				for {
   251→					select {
   252→					case msg, ok := <-msgCh:
   253→						if !ok {
   254→							return
   255→						}
   256→						processMsg(msg)
   257→					case req := <-rotateCh:
   258→						processRotate(req)
   259→					default:
   260→						flushAll()
   261→						return
   262→					}
   263→				}
   264→			}
   265→
   266→			for {
   267→				select {
   268→				case <-ctx.Done():
   269→					drain()
   270→					return
   271→				case msg, ok := <-msgCh:
   272→					if !ok {
   273→						return
   274→					}
   275→					processMsg(msg)
   276→
   277→					// Drain up to 256 more messages without blocking to batch writes.
   278→					for range 256 {
   279→						select {
   280→						case msg, ok := <-msgCh:
   281→							if !ok {
   282→								return
   283→							}
   284→							processMsg(msg)
   285→						case req := <-rotateCh:
   286→							processRotate(req)
   287→						default:
   288→							goto flushed
   289→						}
   290→					}
   291→				flushed:
   292→					flushAll()
   293→				case req := <-rotateCh:
   294→					processRotate(req)
   295→				}
   296→			}
   297→		})
   298→	}
   299→}
   300→
   301→// RotateWALFiles sends rotation requests for the given host directories
   302→// and blocks until all rotations complete. Each request is routed to the
   303→// shard that owns the host directory.
   304→func RotateWALFiles(hostDirs []string) {
   305→	if walShardRotateChs == nil {
   306→		return
   307→	}
   308→	dones := make([]chan struct{}, len(hostDirs))
   309→	for i, dir := range hostDirs {
   310→		dones[i] = make(chan struct{})
   311→		// Extract cluster/node from hostDir to find the right shard.
   312→		// hostDir = rootDir/cluster/node
   313→		shard := walShardIndexFromDir(dir)
   314→		walShardRotateChs[shard] <- walRotateReq{hostDir: dir, done: dones[i]}
   315→	}
   316→	for _, done := range dones {
   317→		<-done
   318→	}
   319→}
   320→
   321→// walShardIndexFromDir extracts cluster and node from a host directory path
   322→// and returns the shard index. The path format is: rootDir/cluster/node
   323→func walShardIndexFromDir(hostDir string) int {
   324→	// hostDir = .../cluster/node — extract last two path components
   325→	node := path.Base(hostDir)
   326→	cluster := path.Base(path.Dir(hostDir))
   327→	return walShardIndex(cluster, node)
   328→}
   329→
   330→// RotateWALFiles sends rotation requests for the given host directories
   331→// and blocks until all rotations complete.
   332→func RotateWALFilesAfterShutdown(hostDirs []string) {
   333→	for _, dir := range hostDirs {
   334→		walPath := path.Join(dir, "current.wal")
   335→		if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
   336→			cclog.Errorf("[METRICSTORE]> WAL: remove %s: %v", walPath, err)
   337→		}
   338→	}
   339→}
   340→
   341→// writeWALRecordDirect encodes a WAL record directly into the bufio.Writer,
   342→// avoiding heap allocations by using a stack-allocated scratch buffer for
   343→// the fixed-size header/trailer and computing CRC inline.
   344→func writeWALRecordDirect(w *bufio.Writer, msg *WALMessage) error {
   345→	// Compute payload size.
   346→	payloadSize := 8 + 2 + len(msg.MetricName) + 1 + 4
   347→	for _, s := range msg.Selector {
   348→		payloadSize += 1 + len(s)
   349→	}
   350→
   351→	// Write magic + payload length (8 bytes header).
   352→	var hdr [8]byte
   353→	binary.LittleEndian.PutUint32(hdr[0:4], walRecordMagic)
   354→	binary.LittleEndian.PutUint32(hdr[4:8], uint32(payloadSize))
   355→	if _, err := w.Write(hdr[:]); err != nil {
   356→		return err
   357→	}
   358→
   359→	// We need to compute CRC over the payload as we write it.
   360→	crc := crc32.NewIEEE()
   361→
   362→	// Timestamp (8 bytes).
   363→	var scratch [8]byte
   364→	binary.LittleEndian.PutUint64(scratch[:8], uint64(msg.Timestamp))
   365→	crc.Write(scratch[:8])
   366→	if _, err := w.Write(scratch[:8]); err != nil {
   367→		return err
   368→	}
   369→
   370→	// Metric name length (2 bytes) + metric name.
   371→	binary.LittleEndian.PutUint16(scratch[:2], uint16(len(msg.MetricName)))
   372→	crc.Write(scratch[:2])
   373→	if _, err := w.Write(scratch[:2]); err != nil {
   374→		return err
   375→	}
   376→	nameBytes := []byte(msg.MetricName)
   377→	crc.Write(nameBytes)
   378→	if _, err := w.Write(nameBytes); err != nil {
   379→		return err
   380→	}
   381→
   382→	// Selector count (1 byte).
   383→	scratch[0] = byte(len(msg.Selector))
   384→	crc.Write(scratch[:1])
   385→	if _, err := w.Write(scratch[:1]); err != nil {
   386→		return err
   387→	}
   388→
   389→	// Selectors (1-byte length + bytes each).
   390→	for _, sel := range msg.Selector {
   391→		scratch[0] = byte(len(sel))
   392→		crc.Write(scratch[:1])
   393→		if _, err := w.Write(scratch[:1]); err != nil {
   394→			return err
   395→		}
   396→		selBytes := []byte(sel)
   397→		crc.Write(selBytes)
   398→		if _, err := w.Write(selBytes); err != nil {
   399→			return err
   400→		}
   401→	}
   402→
   403→	// Value (4 bytes, float32 bits).
   404→	binary.LittleEndian.PutUint32(scratch[:4], math.Float32bits(float32(msg.Value)))
   405→	crc.Write(scratch[:4])
   406→	if _, err := w.Write(scratch[:4]); err != nil {
   407→		return err
   408→	}
   409→
   410→	// CRC32 (4 bytes).
   411→	binary.LittleEndian.PutUint32(scratch[:4], crc.Sum32())
   412→	_, err := w.Write(scratch[:4])
   413→	return err
   414→}
   415→
   416→// buildWALPayload encodes a WALMessage into a binary payload (without magic/length/CRC).
   417→func buildWALPayload(msg *WALMessage) []byte {
   418→	size := 8 + 2 + len(msg.MetricName) + 1 + 4
   419→	for _, s := range msg.Selector {
   420→		size += 1 + len(s)
   421→	}
   422→
   423→	buf := make([]byte, 0, size)
   424→
   425→	// Timestamp (8 bytes, little-endian int64)
   426→	var ts [8]byte
   427→	binary.LittleEndian.PutUint64(ts[:], uint64(msg.Timestamp))
   428→	buf = append(buf, ts[:]...)
   429→
   430→	// Metric name (2-byte length prefix + bytes)
   431→	var mLen [2]byte
   432→	binary.LittleEndian.PutUint16(mLen[:], uint16(len(msg.MetricName)))
   433→	buf = append(buf, mLen[:]...)
   434→	buf = append(buf, msg.MetricName...)
   435→
   436→	// Selector count (1 byte)
   437→	buf = append(buf, byte(len(msg.Selector)))
   438→
   439→	// Selectors (1-byte length prefix + bytes each)
   440→	for _, sel := range msg.Selector {
   441→		buf = append(buf, byte(len(sel)))
   442→		buf = append(buf, sel...)
   443→	}
   444→
   445→	// Value (4 bytes, float32 bit representation)
   446→	var val [4]byte
   447→	binary.LittleEndian.PutUint32(val[:], math.Float32bits(float32(msg.Value)))
   448→	buf = append(buf, val[:]...)
   449→
   450→	return buf
   451→}
   452→
   453→// writeWALRecord appends a binary WAL record to the writer.
   454→// Format: [4B magic][4B payload_len][payload][4B CRC32]
   455→func writeWALRecord(w io.Writer, msg *WALMessage) error {
   456→	payload := buildWALPayload(msg)
   457→	crc := crc32.ChecksumIEEE(payload)
   458→
   459→	record := make([]byte, 0, 4+4+len(payload)+4)
   460→
   461→	var magic [4]byte
   462→	binary.LittleEndian.PutUint32(magic[:], walRecordMagic)
   463→	record = append(record, magic[:]...)
   464→
   465→	var pLen [4]byte
   466→	binary.LittleEndian.PutUint32(pLen[:], uint32(len(payload)))
   467→	record = append(record, pLen[:]...)
   468→
   469→	record = append(record, payload...)
   470→
   471→	var crcBytes [4]byte
   472→	binary.LittleEndian.PutUint32(crcBytes[:], crc)
   473→	record = append(record, crcBytes[:]...)
   474→
   475→	_, err := w.Write(record)
   476→	return err
   477→}
   478→
   479→// readWALRecord reads one WAL record from the reader.
   480→// Returns (nil, nil) on clean EOF. Returns error on data corruption.
   481→// A CRC mismatch indicates a truncated trailing record (expected on crash).
   482→func readWALRecord(r io.Reader) (*WALMessage, error) {
   483→	var magic uint32
   484→	if err := binary.Read(r, binary.LittleEndian, &magic); err != nil {
   485→		if err == io.EOF {
   486→			return nil, nil // Clean EOF
   487→		}
   488→		return nil, fmt.Errorf("read record magic: %w", err)
   489→	}
   490→
   491→	if magic != walRecordMagic {
   492→		return nil, fmt.Errorf("invalid record magic 0x%08X (expected 0x%08X)", magic, walRecordMagic)
   493→	}
   494→
   495→	var payloadLen uint32
   496→	if err := binary.Read(r, binary.LittleEndian, &payloadLen); err != nil {
   497→		return nil, fmt.Errorf("read payload length: %w", err)
   498→	}
   499→
   500→	if payloadLen > 1<<20 { // 1 MB sanity limit
   501→		return nil, fmt.Errorf("record payload too large: %d bytes", payloadLen)
   502→	}
   503→
   504→	payload := make([]byte, payloadLen)
   505→	if _, err := io.ReadFull(r, payload); err != nil {
   506→		return nil, fmt.Errorf("read payload: %w", err)
   507→	}
   508→
   509→	var storedCRC uint32
   510→	if err := binary.Read(r, binary.LittleEndian, &storedCRC); err != nil {
   511→		return nil, fmt.Errorf("read CRC: %w", err)
   512→	}
   513→
   514→	if crc32.ChecksumIEEE(payload) != storedCRC {
   515→		return nil, fmt.Errorf("CRC mismatch (truncated write or corruption)")
   516→	}
   517→
   518→	return parseWALPayload(payload)
   519→}
   520→
   521→// parseWALPayload decodes a binary payload into a WALMessage.
   522→func parseWALPayload(payload []byte) (*WALMessage, error) {
   523→	if len(payload) < 8+2+1+4 {
   524→		return nil, fmt.Errorf("payload too short: %d bytes", len(payload))
   525→	}
   526→
   527→	offset := 0
   528→
   529→	// Timestamp (8 bytes)
   530→	ts := int64(binary.LittleEndian.Uint64(payload[offset : offset+8]))
   531→	offset += 8
   532→
   533→	// Metric name (2-byte length + bytes)
   534→	if offset+2 > len(payload) {
   535→		return nil, fmt.Errorf("metric name length overflows payload")
   536→	}
   537→	mLen := int(binary.LittleEndian.Uint16(payload[offset : offset+2]))
   538→	offset += 2
   539→
   540→	if offset+mLen > len(payload) {
   541→		return nil, fmt.Errorf("metric name overflows payload")
   542→	}
   543→	metricName := string(payload[offset : offset+mLen])
   544→	offset += mLen
   545→
   546→	// Selector count (1 byte)
   547→	if offset >= len(payload) {
   548→		return nil, fmt.Errorf("selector count overflows payload")
   549→	}
   550→	selCount := int(payload[offset])
   551→	offset++
   552→
   553→	selectors := make([]string, selCount)
   554→	for i := range selCount {
   555→		if offset >= len(payload) {
   556→			return nil, fmt.Errorf("selector[%d] length overflows payload", i)
   557→		}
   558→		sLen := int(payload[offset])
   559→		offset++
   560→
   561→		if offset+sLen > len(payload) {
   562→			return nil, fmt.Errorf("selector[%d] data overflows payload", i)
   563→		}
   564→		selectors[i] = string(payload[offset : offset+sLen])
   565→		offset += sLen
   566→	}
   567→
   568→	// Value (4 bytes, float32 bits)
   569→	if offset+4 > len(payload) {
   570→		return nil, fmt.Errorf("value overflows payload")
   571→	}
   572→	bits := binary.LittleEndian.Uint32(payload[offset : offset+4])
   573→	value := schema.Float(math.Float32frombits(bits))
   574→
   575→	return &WALMessage{
   576→		MetricName: metricName,
   577→		Timestamp:  ts,
   578→		Selector:   selectors,
   579→		Value:      value,
   580→	}, nil
   581→}
   582→
   583→// loadWALFile reads a WAL file and replays all valid records into the Level tree.
   584→// l is the host-level node. Corrupt or partial trailing records are silently skipped
   585→// (expected on crash). Records older than 'from' are skipped.
   586→func (l *Level) loadWALFile(m *MemoryStore, f *os.File, from int64) error {
   587→	br := bufio.NewReader(f)
   588→
   589→	// Verify file header magic.
   590→	var fileMagic uint32
   591→	if err := binary.Read(br, binary.LittleEndian, &fileMagic); err != nil {
   592→		if err == io.EOF {
   593→			return nil // Empty file, no data
   594→		}
   595→		return fmt.Errorf("[METRICSTORE]> WAL: read file header: %w", err)
   596→	}
   597→
   598→	if fileMagic != walFileMagic {
   599→		return fmt.Errorf("[METRICSTORE]> WAL: invalid file magic 0x%08X (expected 0x%08X)", fileMagic, walFileMagic)
   600→	}
   601→
   602→	// Cache level lookups to avoid repeated tree traversal.
   603→	lvlCache := make(map[string]*Level)
   604→
   605→	for {
   606→		msg, err := readWALRecord(br)
   607→		if err != nil {
   608→			// Truncated trailing record is expected after a crash; stop replaying.
   609→			cclog.Debugf("[METRICSTORE]> WAL: stopping replay at corrupted/partial record: %v", err)
   610→			break
   611→		}
   612→		if msg == nil {
   613→			break // Clean EOF
   614→		}
   615→
   616→		if msg.Timestamp < from {
   617→			continue // Older than retention window
   618→		}
   619→
   620→		minfo, ok := m.Metrics[msg.MetricName]
   621→		if !ok {
   622→			continue // Unknown metric (config may have changed)
   623→		}
   624→
   625→		// Cache key is the null-separated selector path.
   626→		cacheKey := joinSelector(msg.Selector)
   627→		lvl, ok := lvlCache[cacheKey]
   628→		if !ok {
   629→			lvl = l.findLevelOrCreate(msg.Selector, len(m.Metrics))
   630→			lvlCache[cacheKey] = lvl
   631→		}
   632→
   633→		// Write directly to the buffer, same as WriteToLevel but without the
   634→		// global level lookup (we already have the right level).
   635→		lvl.lock.Lock()
   636→		b := lvl.metrics[minfo.offset]
   637→		if b == nil {
   638→			b = newBuffer(msg.Timestamp, minfo.Frequency)
   639→			lvl.metrics[minfo.offset] = b
   640→		}
   641→		nb, writeErr := b.write(msg.Timestamp, msg.Value)
   642→		if writeErr == nil && b != nb {
   643→			lvl.metrics[minfo.offset] = nb
   644→		}
   645→		// Ignore write errors for timestamps before buffer start (can happen when
   646→		// replaying WAL entries that predate a loaded snapshot's start time).
   647→		lvl.lock.Unlock()
   648→	}
   649→
   650→	return nil
   651→}
   652→
   653→// joinSelector builds a cache key from a selector slice using null bytes as separators.
   654→func joinSelector(sel []string) string {
   655→	if len(sel) == 0 {
   656→		return ""
   657→	}
   658→	var result strings.Builder
   659→	result.WriteString(sel[0])
   660→	for i := 1; i < len(sel); i++ {
   661→		result.WriteString("\x00" + sel[i])
   662→	}
   663→	return result.String()
   664→}
   665→
   666→// ToCheckpointWAL writes binary snapshot files for all hosts in parallel.
   667→// Returns the number of files written, the list of host directories that were
   668→// successfully checkpointed (for WAL rotation), and any errors.
   669→func (m *MemoryStore) ToCheckpointWAL(dir string, from, to int64) (int, []string, error) {
   670→	// Collect all cluster/host pairs.
   671→	m.root.lock.RLock()
   672→	totalHosts := 0
   673→	for _, l1 := range m.root.children {
   674→		l1.lock.RLock()
   675→		totalHosts += len(l1.children)
   676→		l1.lock.RUnlock()
   677→	}
   678→	m.root.lock.RUnlock()
   679→
   680→	levels := make([]*Level, 0, totalHosts)
   681→	selectors := make([][]string, 0, totalHosts)
   682→
   683→	m.root.lock.RLock()
   684→	for sel1, l1 := range m.root.children {
   685→		l1.lock.RLock()
   686→		for sel2, l2 := range l1.children {
   687→			levels = append(levels, l2)
   688→			selectors = append(selectors, []string{sel1, sel2})
   689→		}
   690→		l1.lock.RUnlock()
   691→	}
   692→	m.root.lock.RUnlock()
   693→
   694→	type workItem struct {
   695→		level    *Level
   696→		hostDir  string
   697→		selector []string
   698→	}
   699→
   700→	n, errs := int32(0), int32(0)
   701→	var successDirs []string
   702→	var successMu sync.Mutex
   703→
   704→	var wg sync.WaitGroup
   705→	wg.Add(Keys.NumWorkers)
   706→	work := make(chan workItem, Keys.NumWorkers*2)
   707→
   708→	for range Keys.NumWorkers {
   709→		go func() {
   710→			defer wg.Done()
   711→			for wi := range work {
   712→				err := wi.level.toCheckpointBinary(wi.hostDir, from, to, m)
   713→				if err != nil {
   714→					if err == ErrNoNewArchiveData {
   715→						continue
   716→					}
   717→					cclog.Errorf("[METRICSTORE]> binary checkpoint error for %s: %v", wi.hostDir, err)
   718→					atomic.AddInt32(&errs, 1)
   719→				} else {
   720→					atomic.AddInt32(&n, 1)
   721→					successMu.Lock()
   722→					successDirs = append(successDirs, wi.hostDir)
   723→					successMu.Unlock()
   724→				}
   725→			}
   726→		}()
   727→	}
   728→
   729→	for i := range levels {
   730→		hostDir := path.Join(dir, path.Join(selectors[i]...))
   731→		work <- workItem{
   732→			level:    levels[i],
   733→			hostDir:  hostDir,
   734→			selector: selectors[i],
   735→		}
   736→	}
   737→	close(work)
   738→	wg.Wait()
   739→
   740→	if errs > 0 {
   741→		return int(n), successDirs, fmt.Errorf("[METRICSTORE]> %d errors during binary checkpoint (%d successes)", errs, n)
   742→	}
   743→	return int(n), successDirs, nil
   744→}
   745→
   746→// toCheckpointBinary writes a binary snapshot file for a single host-level node.
   747→// Uses atomic rename (write to .tmp then rename) to avoid partial reads on crash.
   748→func (l *Level) toCheckpointBinary(dir string, from, to int64, m *MemoryStore) error {
   749→	cf, err := l.toCheckpointFile(from, to, m)
   750→	if err != nil {
   751→		return err
   752→	}
   753→	if cf == nil {
   754→		return ErrNoNewArchiveData
   755→	}
   756→
   757→	if err := os.MkdirAll(dir, CheckpointDirPerms); err != nil {
   758→		return fmt.Errorf("mkdir %s: %w", dir, err)
   759→	}
   760→
   761→	// Write to a temp file first, then rename (atomic on POSIX).
   762→	tmpPath := path.Join(dir, fmt.Sprintf("%d.bin.tmp", from))
   763→	finalPath := path.Join(dir, fmt.Sprintf("%d.bin", from))
   764→
   765→	f, err := os.OpenFile(tmpPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, CheckpointFilePerms)
   766→	if err != nil {
   767→		return fmt.Errorf("open binary snapshot %s: %w", tmpPath, err)
   768→	}
   769→
   770→	bw := bufio.NewWriter(f)
   771→	if err := writeBinarySnapshotFile(bw, cf); err != nil {
   772→		f.Close()
   773→		os.Remove(tmpPath)
   774→		return fmt.Errorf("write binary snapshot: %w", err)
   775→	}
   776→	if err := bw.Flush(); err != nil {
   777→		f.Close()
   778→		os.Remove(tmpPath)
   779→		return err
   780→	}
   781→	f.Close()
   782→
   783→	return os.Rename(tmpPath, finalPath)
   784→}
   785→
   786→// writeBinarySnapshotFile writes the binary snapshot file header and level tree.
   787→func writeBinarySnapshotFile(w io.Writer, cf *CheckpointFile) error {
   788→	if err := binary.Write(w, binary.LittleEndian, snapFileMagic); err != nil {
   789→		return err
   790→	}
   791→	if err := binary.Write(w, binary.LittleEndian, cf.From); err != nil {
   792→		return err
   793→	}
   794→	if err := binary.Write(w, binary.LittleEndian, cf.To); err != nil {
   795→		return err
   796→	}
   797→	return writeBinaryLevel(w, cf)
   798→}
   799→
   800→// writeBinaryLevel recursively writes a CheckpointFile level in binary format.
   801→func writeBinaryLevel(w io.Writer, cf *CheckpointFile) error {
   802→	if err := binary.Write(w, binary.LittleEndian, uint32(len(cf.Metrics))); err != nil {
   803→		return err
   804→	}
   805→
   806→	for name, metric := range cf.Metrics {
   807→		if err := writeString16(w, name); err != nil {
   808→			return err
   809→		}
   810→		if err := binary.Write(w, binary.LittleEndian, metric.Frequency); err != nil {
   811→			return err
   812→		}
   813→		if err := binary.Write(w, binary.LittleEndian, metric.Start); err != nil {
   814→			return err
   815→		}
   816→		if err := binary.Write(w, binary.LittleEndian, uint32(len(metric.Data))); err != nil {
   817→			return err
   818→		}
   819→		for _, v := range metric.Data {
   820→			if err := binary.Write(w, binary.LittleEndian, math.Float32bits(float32(v))); err != nil {
   821→				return err
   822→			}
   823→		}
   824→	}
   825→
   826→	if err := binary.Write(w, binary.LittleEndian, uint32(len(cf.Children))); err != nil {
   827→		return err
   828→	}
   829→
   830→	for name, child := range cf.Children {
   831→		if err := writeString16(w, name); err != nil {
   832→			return err
   833→		}
   834→		if err := writeBinaryLevel(w, child); err != nil {
   835→			return err
   836→		}
   837→	}
   838→
   839→	return nil
   840→}
   841→
   842→// writeString16 writes a 2-byte length-prefixed string to w.
   843→func writeString16(w io.Writer, s string) error {
   844→	if err := binary.Write(w, binary.LittleEndian, uint16(len(s))); err != nil {
   845→		return err
   846→	}
   847→	_, err := io.WriteString(w, s)
   848→	return err
   849→}
   850→
   851→// loadBinaryFile reads a binary snapshot file and loads data into the Level tree.
   852→// The retention check (from) is applied to the file's 'to' timestamp.
   853→func (l *Level) loadBinaryFile(m *MemoryStore, f *os.File, from int64) error {
   854→	br := bufio.NewReader(f)
   855→
   856→	var magic uint32
   857→	if err := binary.Read(br, binary.LittleEndian, &magic); err != nil {
   858→		return fmt.Errorf("[METRICSTORE]> binary snapshot: read magic: %w", err)
   859→	}
   860→	if magic != snapFileMagic {
   861→		return fmt.Errorf("[METRICSTORE]> binary snapshot: invalid magic 0x%08X (expected 0x%08X)", magic, snapFileMagic)
   862→	}
   863→
   864→	var fileFrom, fileTo int64
   865→	if err := binary.Read(br, binary.LittleEndian, &fileFrom); err != nil {
   866→		return fmt.Errorf("[METRICSTORE]> binary snapshot: read from: %w", err)
   867→	}
   868→	if err := binary.Read(br, binary.LittleEndian, &fileTo); err != nil {
   869→		return fmt.Errorf("[METRICSTORE]> binary snapshot: read to: %w", err)
   870→	}
   871→
   872→	if fileTo != 0 && fileTo < from {
   873→		return nil // File is older than retention window, skip it
   874→	}
   875→
   876→	cf, err := readBinaryLevel(br)
   877→	if err != nil {
   878→		return fmt.Errorf("[METRICSTORE]> binary snapshot: read level tree: %w", err)
   879→	}
   880→	cf.From = fileFrom
   881→	cf.To = fileTo
   882→
   883→	return l.loadFile(cf, m)
   884→}
   885→
   886→// readBinaryLevel recursively reads a level from the binary snapshot format.
   887→func readBinaryLevel(r io.Reader) (*CheckpointFile, error) {
   888→	cf := &CheckpointFile{
   889→		Metrics:  make(map[string]*CheckpointMetrics),
   890→		Children: make(map[string]*CheckpointFile),
   891→	}
   892→
   893→	var numMetrics uint32
   894→	if err := binary.Read(r, binary.LittleEndian, &numMetrics); err != nil {
   895→		return nil, fmt.Errorf("read num_metrics: %w", err)
   896→	}
   897→
   898→	for range numMetrics {
   899→		name, err := readString16(r)
   900→		if err != nil {
   901→			return nil, fmt.Errorf("read metric name: %w", err)
   902→		}
   903→
   904→		var freq, start int64
   905→		if err := binary.Read(r, binary.LittleEndian, &freq); err != nil {
   906→			return nil, fmt.Errorf("read frequency for %s: %w", name, err)
   907→		}
   908→		if err := binary.Read(r, binary.LittleEndian, &start); err != nil {
   909→			return nil, fmt.Errorf("read start for %s: %w", name, err)
   910→		}
   911→
   912→		var numValues uint32
   913→		if err := binary.Read(r, binary.LittleEndian, &numValues); err != nil {
   914→			return nil, fmt.Errorf("read num_values for %s: %w", name, err)
   915→		}
   916→
   917→		data := make([]schema.Float, numValues)
   918→		for i := range numValues {
   919→			var bits uint32
   920→			if err := binary.Read(r, binary.LittleEndian, &bits); err != nil {
   921→				return nil, fmt.Errorf("read value[%d] for %s: %w", i, name, err)
   922→			}
   923→			data[i] = schema.Float(math.Float32frombits(bits))
   924→		}
   925→
   926→		cf.Metrics[name] = &CheckpointMetrics{
   927→			Frequency: freq,
   928→			Start:     start,
   929→			Data:      data,
   930→		}
   931→	}
   932→
   933→	var numChildren uint32
   934→	if err := binary.Read(r, binary.LittleEndian, &numChildren); err != nil {
   935→		return nil, fmt.Errorf("read num_children: %w", err)
   936→	}
   937→
   938→	for range numChildren {
   939→		childName, err := readString16(r)
   940→		if err != nil {
   941→			return nil, fmt.Errorf("read child name: %w", err)
   942→		}
   943→
   944→		child, err := readBinaryLevel(r)
   945→		if err != nil {
   946→			return nil, fmt.Errorf("read child %s: %w", childName, err)
   947→		}
   948→		cf.Children[childName] = child
   949→	}
   950→
   951→	return cf, nil
   952→}
   953→
   954→// readString16 reads a 2-byte length-prefixed string from r.
   955→func readString16(r io.Reader) (string, error) {
   956→	var sLen uint16
   957→	if err := binary.Read(r, binary.LittleEndian, &sLen); err != nil {
   958→		return "", err
   959→	}
   960→	buf := make([]byte, sLen)
   961→	if _, err := io.ReadFull(r, buf); err != nil {
   962→		return "", err
   963→	}
   964→	return string(buf), nil
   965→}
   966→


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstore provides an efficient in-memory time-series metric storage system
     7→// with support for hierarchical data organization, checkpointing, and archiving.
     8→//
     9→// The package organizes metrics in a tree structure (cluster → host → component) and
    10→// provides concurrent read/write access to metric data with configurable aggregation strategies.
    11→// Background goroutines handle periodic checkpointing (JSON or Avro format), archiving old data,
    12→// and enforcing retention policies.
    13→//
    14→// Key features:
    15→//   - In-memory metric storage with configurable retention
    16→//   - Hierarchical data organization (selectors)
    17→//   - Concurrent checkpoint/archive workers
    18→//   - Support for sum and average aggregation
    19→//   - NATS integration for metric ingestion
    20→package metricstore
    21→
    22→import (
    23→	"bytes"
    24→	"context"
    25→	"encoding/json"
    26→	"errors"
    27→	"fmt"
    28→	"runtime"
    29→	"runtime/debug"
    30→	"slices"
    31→	"sync"
    32→	"time"
    33→
    34→	"github.com/ClusterCockpit/cc-backend/internal/config"
    35→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    36→	"github.com/ClusterCockpit/cc-lib/v2/resampler"
    37→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    38→	"github.com/ClusterCockpit/cc-lib/v2/util"
    39→)
    40→
    41→// GlobalState holds the global state for the metric store with thread-safe access.
    42→type GlobalState struct {
    43→	mu                sync.RWMutex
    44→	lastRetentionTime int64
    45→	selectorsExcluded bool
    46→}
    47→
    48→var (
    49→	singleton  sync.Once
    50→	msInstance *MemoryStore
    51→	// shutdownFunc stores the context cancellation function created in Init
    52→	// and is called during Shutdown to cancel all background goroutines
    53→	shutdownFunc   context.CancelFunc
    54→	shutdownFuncMu sync.Mutex // Protects shutdownFunc from concurrent access
    55→	// Create a global instance
    56→	state = &GlobalState{}
    57→)
    58→
    59→// NodeProvider provides information about nodes currently in use by running jobs.
    60→//
    61→// This interface allows metricstore to query job information without directly
    62→// depending on the repository package, breaking the import cycle.
    63→//
    64→// Implementations should return nodes that are actively processing jobs started
    65→// before the given timestamp. These nodes will be excluded from retention-based
    66→// garbage collection to prevent data loss for jobs that are still running or
    67→// recently completed.
    68→type NodeProvider interface {
    69→	// GetUsedNodes returns a map of cluster names to sorted lists of unique hostnames
    70→	// that are currently in use by jobs that started before the given timestamp.
    71→	//
    72→	// Parameters:
    73→	//   - ts: Unix timestamp threshold - returns nodes with jobs started before this time
    74→	//
    75→	// Returns:
    76→	//   - Map of cluster names to lists of node hostnames that should be excluded from garbage collection
    77→	//   - Error if the query fails
    78→	GetUsedNodes(ts int64) (map[string][]string, error)
    79→}
    80→
    81→// Metric represents a single metric data point to be written to the store.
    82→type Metric struct {
    83→	Name  string
    84→	Value schema.Float
    85→	// MetricConfig contains frequency and aggregation settings for this metric.
    86→	// If Frequency is 0, configuration will be looked up from MemoryStore.Metrics during Write().
    87→	MetricConfig MetricConfig
    88→}
    89→
    90→// MemoryStore is the main in-memory time-series metric storage implementation.
    91→//
    92→// It organizes metrics in a hierarchical tree structure where each level represents
    93→// a component of the system hierarchy (e.g., cluster → host → CPU). Each level can
    94→// store multiple metrics as time-series buffers.
    95→//
    96→// The store is initialized as a singleton via InitMetrics() and accessed via GetMemoryStore().
    97→// All public methods are safe for concurrent use.
    98→type MemoryStore struct {
    99→	Metrics      map[string]MetricConfig
   100→	root         Level
   101→	nodeProvider NodeProvider
   102→}
   103→
   104→// Init initializes the metric store from configuration and starts background workers.
   105→//
   106→// This function must be called exactly once before any other metricstore operations.
   107→// It performs the following initialization steps:
   108→//  1. Validates and decodes the metric store configuration
   109→//  2. Configures worker pool size (defaults to NumCPU/2+1, max 10)
   110→//  3. Loads metric configurations from all registered clusters
   111→//  4. Restores checkpoints within the retention window
   112→//  5. Starts background workers for retention, checkpointing, archiving, and monitoring
   113→//  6. Optionally subscribes to NATS for real-time metric ingestion
   114→//
   115→// Parameters:
   116→//   - rawConfig: JSON configuration for the metric store (see MetricStoreConfig)
   117→//   - wg: WaitGroup that will be incremented for each background goroutine started
   118→//
   119→// The function will call cclog.Fatal on critical errors during initialization.
   120→// Use Shutdown() to cleanly stop all background workers started by Init().
   121→//
   122→// Note: Signal handling must be implemented by the caller. Call Shutdown() when
   123→// receiving termination signals to ensure checkpoint data is persisted.
   124→func Init(rawConfig json.RawMessage, metrics map[string]MetricConfig, wg *sync.WaitGroup) {
   125→	startupTime := time.Now()
   126→
   127→	if rawConfig != nil {
   128→		config.Validate(configSchema, rawConfig)
   129→		dec := json.NewDecoder(bytes.NewReader(rawConfig))
   130→		dec.DisallowUnknownFields()
   131→		if err := dec.Decode(&Keys); err != nil {
   132→			cclog.Abortf("[METRICSTORE]> Metric Store Config Init: Could not decode config file '%s'.\nError: %s\n", rawConfig, err.Error())
   133→		}
   134→	}
   135→
   136→	// Set NumWorkers from config or use default
   137→	if Keys.NumWorkers <= 0 {
   138→		Keys.NumWorkers = min(runtime.NumCPU()/2+1, DefaultMaxWorkers)
   139→	}
   140→	cclog.Debugf("[METRICSTORE]> Using %d workers for checkpoint/archive operations\n", Keys.NumWorkers)
   141→
   142→	// Pass the config.MetricStoreKeys
   143→	InitMetrics(metrics)
   144→
   145→	ms := GetMemoryStore()
   146→
   147→	d, err := time.ParseDuration(Keys.RetentionInMemory)
   148→	if err != nil {
   149→		cclog.Fatal(err)
   150→	}
   151→
   152→	restoreFrom := startupTime.Add(-d)
   153→	cclog.Infof("[METRICSTORE]> Loading checkpoints newer than %s\n", restoreFrom.Format(time.RFC3339))
   154→	files, err := ms.FromCheckpointFiles(Keys.Checkpoints.RootDir, restoreFrom.Unix())
   155→	loadedData := ms.SizeInBytes() / 1024 / 1024 // In MB
   156→	if err != nil {
   157→		cclog.Fatalf("[METRICSTORE]> Loading checkpoints failed: %s\n", err.Error())
   158→	} else {
   159→		cclog.Infof("[METRICSTORE]> Checkpoints loaded (%d files, %d MB, that took %fs)\n", files, loadedData, time.Since(startupTime).Seconds())
   160→	}
   161→
   162→	// Try to use less memory by forcing a GC run here and then
   163→	// lowering the target percentage. The default of 100 means
   164→	// that only once the ratio of new allocations execeds the
   165→	// previously active heap, a GC is triggered.
   166→	// Forcing a GC here will set the "previously active heap"
   167→	// to a minumum.
   168→	// runtime.GC()
   169→
   170→	ctx, shutdown := context.WithCancel(context.Background())
   171→
   172→	Retention(wg, ctx)
   173→	Checkpointing(wg, ctx)
   174→	CleanUp(wg, ctx)
   175→	WALStaging(wg, ctx)
   176→	MemoryUsageTracker(wg, ctx)
   177→
   178→	// Note: Signal handling has been removed from this function.
   179→	// The caller is responsible for handling shutdown signals and calling
   180→	// the shutdown() function when appropriate.
   181→	// Store the shutdown function for later use by Shutdown()
   182→	shutdownFuncMu.Lock()
   183→	shutdownFunc = shutdown
   184→	shutdownFuncMu.Unlock()
   185→
   186→	if Keys.Subscriptions != nil {
   187→		wg.Go(func() {
   188→			if err := ReceiveNats(ms, Keys.NumWorkers, ctx); err != nil {
   189→				cclog.Fatal(err)
   190→			}
   191→		})
   192→	}
   193→}
   194→
   195→// InitMetrics initializes the singleton MemoryStore instance with the given metric configurations.
   196→//
   197→// This function must be called before GetMemoryStore() and can only be called once due to
   198→// the singleton pattern. It assigns each metric an internal offset for efficient buffer indexing.
   199→//
   200→// Parameters:
   201→//   - metrics: Map of metric names to their configurations (frequency and aggregation strategy)
   202→//
   203→// Panics if any metric has Frequency == 0, which indicates an invalid configuration.
   204→//
   205→// After this call, the global msInstance is ready for use via GetMemoryStore().
   206→func InitMetrics(metrics map[string]MetricConfig) {
   207→	singleton.Do(func() {
   208→		offset := 0
   209→		for key, cfg := range metrics {
   210→			if cfg.Frequency == 0 {
   211→				panic("[METRICSTORE]> invalid frequency")
   212→			}
   213→
   214→			metrics[key] = MetricConfig{
   215→				Frequency:   cfg.Frequency,
   216→				Aggregation: cfg.Aggregation,
   217→				offset:      offset,
   218→			}
   219→			offset += 1
   220→		}
   221→
   222→		msInstance = &MemoryStore{
   223→			root: Level{
   224→				metrics:  make([]*buffer, len(metrics)),
   225→				children: make(map[string]*Level),
   226→			},
   227→			Metrics: metrics,
   228→		}
   229→	})
   230→}
   231→
   232→// GetMemoryStore returns the singleton MemoryStore instance.
   233→//
   234→// Returns the initialized MemoryStore singleton. Calls cclog.Fatal if InitMetrics() was not called first.
   235→//
   236→// This function is safe for concurrent use after initialization.
   237→func GetMemoryStore() *MemoryStore {
   238→	if msInstance == nil {
   239→		cclog.Warnf("[METRICSTORE]> MemoryStore not initialized!")
   240→	}
   241→
   242→	return msInstance
   243→}
   244→
   245→func (ms *MemoryStore) GetMetricFrequency(metricName string) (int64, error) {
   246→	if metric, ok := ms.Metrics[metricName]; ok {
   247→		return metric.Frequency, nil
   248→	}
   249→	return 0, fmt.Errorf("[METRICSTORE]> metric %s not found", metricName)
   250→}
   251→
   252→// SetNodeProvider sets the NodeProvider implementation for the MemoryStore.
   253→// This must be called during initialization to provide job state information
   254→// for selective buffer retention during Free operations.
   255→// If not set, the Free function will fall back to freeing all buffers.
   256→func (ms *MemoryStore) SetNodeProvider(provider NodeProvider) {
   257→	ms.nodeProvider = provider
   258→}
   259→
   260→// Shutdown performs a graceful shutdown of the metric store.
   261→//
   262→// This function cancels all background goroutines started by Init() and writes
   263→// a final checkpoint to disk before returning. It should be called when the
   264→// application receives a termination signal.
   265→//
   266→// The function will:
   267→//  1. Cancel the context to stop all background workers
   268→//  2. Close the WAL messages channel if using WAL format
   269→//  3. Write a final checkpoint to preserve in-memory data
   270→//  4. Log any errors encountered during shutdown
   271→//
   272→// Note: This function blocks until the final checkpoint is written.
   273→func Shutdown() {
   274→	shutdownFuncMu.Lock()
   275→	defer shutdownFuncMu.Unlock()
   276→	if shutdownFunc != nil {
   277→		shutdownFunc()
   278→	}
   279→
   280→	if Keys.Checkpoints.FileFormat == "wal" {
   281→		for _, ch := range walShardChs {
   282→			close(ch)
   283→		}
   284→	}
   285→
   286→	cclog.Infof("[METRICSTORE]> Writing to '%s'...\n", Keys.Checkpoints.RootDir)
   287→	var files int
   288→	var err error
   289→
   290→	ms := GetMemoryStore()
   291→
   292→	lastCheckpointMu.Lock()
   293→	from := lastCheckpoint
   294→	lastCheckpointMu.Unlock()
   295→
   296→	if Keys.Checkpoints.FileFormat == "wal" {
   297→		var hostDirs []string
   298→		files, hostDirs, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
   299→		if err == nil {
   300→			RotateWALFilesAfterShutdown(hostDirs)
   301→		}
   302→	} else {
   303→		files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
   304→	}
   305→
   306→	if err != nil {
   307→		cclog.Errorf("[METRICSTORE]> Writing checkpoint failed: %s\n", err.Error())
   308→	}
   309→	cclog.Infof("[METRICSTORE]> Done! (%d files written)\n", files)
   310→}
   311→
   312→// Retention starts a background goroutine that periodically frees old metric data.
   313→//
   314→// This worker runs at half the retention interval and calls Free() to remove buffers
   315→// older than the configured retention time. It respects the NodeProvider to preserve
   316→// data for nodes with active jobs.
   317→//
   318→// Parameters:
   319→//   - wg: WaitGroup to signal completion when context is cancelled
   320→//   - ctx: Context for cancellation signal
   321→//
   322→// The goroutine exits when ctx is cancelled.
   323→func Retention(wg *sync.WaitGroup, ctx context.Context) {
   324→	ms := GetMemoryStore()
   325→
   326→	wg.Go(func() {
   327→		d, err := time.ParseDuration(Keys.RetentionInMemory)
   328→		if err != nil {
   329→			cclog.Fatal(err)
   330→		}
   331→		if d <= 0 {
   332→			return
   333→		}
   334→
   335→		tickInterval := d / 2
   336→		if tickInterval <= 0 {
   337→			return
   338→		}
   339→		ticker := time.NewTicker(tickInterval)
   340→		defer ticker.Stop()
   341→
   342→		for {
   343→			select {
   344→			case <-ctx.Done():
   345→				return
   346→			case <-ticker.C:
   347→				state.mu.Lock()
   348→
   349→				t := time.Now().Add(-d)
   350→
   351→				state.lastRetentionTime = t.Unix()
   352→
   353→				cclog.Infof("[METRICSTORE]> start freeing buffers (older than %s)...\n", t.Format(time.RFC3339))
   354→
   355→				freed, err := Free(ms, t)
   356→				if err != nil {
   357→					cclog.Errorf("[METRICSTORE]> freeing up buffers failed: %s\n", err.Error())
   358→				} else {
   359→					cclog.Infof("[METRICSTORE]> done: %d buffers freed\n", freed)
   360→				}
   361→
   362→				state.mu.Unlock()
   363→
   364→				// Clean up the buffer pool
   365→				bufferPool.Clean(state.lastRetentionTime)
   366→			}
   367→		}
   368→	})
   369→}
   370→
   371→// MemoryUsageTracker starts a background goroutine that monitors memory usage.
   372→//
   373→// This worker checks actual process memory usage (via runtime.MemStats) periodically
   374→// and force-frees buffers if memory exceeds the configured cap. It uses FreeOSMemory()
   375→// to return memory to the OS after freeing buffers, avoiding aggressive GC that causes
   376→// performance issues.
   377→//
   378→// The tracker logs both actual memory usage (heap allocated) and metric data size for
   379→// visibility into memory overhead from Go runtime structures and allocations.
   380→//
   381→// Parameters:
   382→//   - wg: WaitGroup to signal completion when context is cancelled
   383→//   - ctx: Context for cancellation signal
   384→//
   385→// The goroutine exits when ctx is cancelled.
   386→func MemoryUsageTracker(wg *sync.WaitGroup, ctx context.Context) {
   387→	ms := GetMemoryStore()
   388→
   389→	wg.Go(func() {
   390→		normalInterval := DefaultMemoryUsageTrackerInterval
   391→		fastInterval := 30 * time.Second
   392→
   393→		if normalInterval <= 0 {
   394→			return
   395→		}
   396→
   397→		ticker := time.NewTicker(normalInterval)
   398→		defer ticker.Stop()
   399→		currentInterval := normalInterval
   400→
   401→		for {
   402→			select {
   403→			case <-ctx.Done():
   404→				return
   405→			case <-ticker.C:
   406→				var mem runtime.MemStats
   407→				runtime.ReadMemStats(&mem)
   408→				actualMemoryGB := float64(mem.Alloc) / 1e9
   409→				metricDataGB := ms.SizeInGB()
   410→				cclog.Infof("[METRICSTORE]> memory usage: %.2f GB actual (%.2f GB metric data)", actualMemoryGB, metricDataGB)
   411→
   412→				freedExcluded := 0
   413→				freedEmergency := 0
   414→				var err error
   415→
   416→				state.mu.RLock()
   417→				lastRetention := state.lastRetentionTime
   418→				selectorsExcluded := state.selectorsExcluded
   419→				state.mu.RUnlock()
   420→
   421→				if lastRetention != 0 && selectorsExcluded {
   422→					freedExcluded, err = ms.Free(nil, lastRetention)
   423→					if err != nil {
   424→						cclog.Errorf("[METRICSTORE]> error while force-freeing the excluded buffers: %s", err)
   425→					}
   426→
   427→					if freedExcluded > 0 {
   428→						debug.FreeOSMemory()
   429→						cclog.Infof("[METRICSTORE]> done: %d excluded buffers force-freed", freedExcluded)
   430→					}
   431→				}
   432→
   433→				runtime.ReadMemStats(&mem)
   434→				actualMemoryGB = float64(mem.Alloc) / 1e9
   435→
   436→				if actualMemoryGB > float64(Keys.MemoryCap) {
   437→					cclog.Warnf("[METRICSTORE]> memory usage %.2f GB exceeds cap %d GB, starting emergency buffer freeing", actualMemoryGB, Keys.MemoryCap)
   438→
   439→					// Use progressive time-based Free with increasing threshold
   440→					// instead of ForceFree loop — fewer tree traversals, more effective
   441→					d, parseErr := time.ParseDuration(Keys.RetentionInMemory)
   442→					if parseErr != nil {
   443→						cclog.Errorf("[METRICSTORE]> cannot parse retention duration: %s", parseErr)
   444→					} else {
   445→						thresholds := []float64{0.75, 0.5, 0.25}
   446→						for _, fraction := range thresholds {
   447→							threshold := time.Now().Add(-time.Duration(float64(d) * fraction))
   448→							freed, freeErr := ms.Free(nil, threshold.Unix())
   449→							if freeErr != nil {
   450→								cclog.Errorf("[METRICSTORE]> error while freeing buffers at %.0f%% retention: %s", fraction*100, freeErr)
   451→							}
   452→							freedEmergency += freed
   453→
   454→							bufferPool.Clear()
   455→							runtime.GC()
   456→							runtime.ReadMemStats(&mem)
   457→							actualMemoryGB = float64(mem.Alloc) / 1e9
   458→
   459→							if actualMemoryGB < float64(Keys.MemoryCap) {
   460→								break
   461→							}
   462→						}
   463→					}
   464→
   465→					bufferPool.Clear()
   466→					debug.FreeOSMemory()
   467→
   468→					runtime.ReadMemStats(&mem)
   469→					actualMemoryGB = float64(mem.Alloc) / 1e9
   470→
   471→					if actualMemoryGB >= float64(Keys.MemoryCap) {
   472→						cclog.Errorf("[METRICSTORE]> after emergency frees (%d buffers), memory usage %.2f GB still at/above cap %d GB", freedEmergency, actualMemoryGB, Keys.MemoryCap)
   473→					} else {
   474→						cclog.Infof("[METRICSTORE]> emergency freeing complete: %d buffers freed, memory now %.2f GB", freedEmergency, actualMemoryGB)
   475→					}
   476→				}
   477→
   478→				// Adaptive ticker: check more frequently when memory is high
   479→				memoryRatio := actualMemoryGB / float64(Keys.MemoryCap)
   480→				if memoryRatio > 0.8 && currentInterval != fastInterval {
   481→					ticker.Reset(fastInterval)
   482→					currentInterval = fastInterval
   483→					cclog.Infof("[METRICSTORE]> memory at %.0f%% of cap, switching to fast check interval (30s)", memoryRatio*100)
   484→				} else if memoryRatio <= 0.8 && currentInterval != normalInterval {
   485→					ticker.Reset(normalInterval)
   486→					currentInterval = normalInterval
   487→					cclog.Infof("[METRICSTORE]> memory at %.0f%% of cap, switching to normal check interval", memoryRatio*100)
   488→				}
   489→			}
   490→		}
   491→	})
   492→}
   493→
   494→// Free removes metric data older than the given time while preserving data for active nodes.
   495→//
   496→// This function implements intelligent retention by consulting the NodeProvider (if configured)
   497→// to determine which nodes are currently in use by running jobs. Data for these nodes is
   498→// preserved even if older than the retention time.
   499→//
   500→// Parameters:
   501→//   - ms: The MemoryStore instance
   502→//   - t: Time threshold - buffers with data older than this will be freed
   503→//
   504→// Returns:
   505→//   - Number of buffers freed
   506→//   - Error if NodeProvider query fails
   507→//
   508→// Behavior:
   509→//   - If no NodeProvider is set: frees all buffers older than t
   510→//   - If NodeProvider returns empty map: frees all buffers older than t
   511→//   - Otherwise: preserves buffers for nodes returned by GetUsedNodes(), frees others
   512→func Free(ms *MemoryStore, t time.Time) (int, error) {
   513→	// If no NodeProvider is configured, free all buffers older than t
   514→	if ms.nodeProvider == nil {
   515→		return ms.Free(nil, t.Unix())
   516→	}
   517→
   518→	excludeSelectors, err := ms.nodeProvider.GetUsedNodes(t.Unix())
   519→	if err != nil {
   520→		return 0, err
   521→	}
   522→
   523→	switch lenMap := len(excludeSelectors); lenMap {
   524→
   525→	// If the length of the map returned by GetUsedNodes() is 0,
   526→	// then use default Free method with nil selector
   527→	case 0:
   528→		state.selectorsExcluded = false
   529→		return ms.Free(nil, t.Unix())
   530→
   531→	// Else formulate selectors, exclude those from the map
   532→	// and free the rest of the selectors
   533→	default:
   534→		state.selectorsExcluded = true
   535→		selectors := GetSelectors(ms, excludeSelectors)
   536→		return FreeSelected(ms, selectors, t)
   537→	}
   538→}
   539→
   540→// FreeSelected frees buffers for specific selectors while preserving others.
   541→//
   542→// This function is used when we want to retain some specific nodes beyond the retention time.
   543→// It iterates through the provided selectors and frees their associated buffers.
   544→//
   545→// Parameters:
   546→//   - ms: The MemoryStore instance
   547→//   - selectors: List of selector paths to free (e.g., [["cluster1", "node1"], ["cluster2", "node2"]])
   548→//   - t: Time threshold for freeing buffers
   549→//
   550→// Returns the total number of buffers freed and any error encountered.
   551→func FreeSelected(ms *MemoryStore, selectors [][]string, t time.Time) (int, error) {
   552→	freed := 0
   553→
   554→	for _, selector := range selectors {
   555→
   556→		freedBuffers, err := ms.Free(selector, t.Unix())
   557→		if err != nil {
   558→			cclog.Errorf("error while freeing selected buffers: %#v", err)
   559→		}
   560→		freed += freedBuffers
   561→
   562→	}
   563→
   564→	return freed, nil
   565→}
   566→
   567→// GetSelectors returns all selectors at depth 2 (cluster/node level) that are NOT in the exclusion map.
   568→//
   569→// This function generates a list of selectors whose buffers should be freed by excluding
   570→// selectors that correspond to nodes currently in use by running jobs.
   571→//
   572→// Parameters:
   573→//   - ms: The MemoryStore instance
   574→//   - excludeSelectors: Map of cluster names to node hostnames that should NOT be freed
   575→//
   576→// Returns a list of selectors ([]string paths) that can be safely freed.
   577→//
   578→// Example:
   579→//
   580→//	If the tree has paths ["emmy", "node001"] and ["emmy", "node002"],
   581→//	and excludeSelectors contains {"emmy": ["node001"]},
   582→//	then only [["emmy", "node002"]] is returned.
   583→func GetSelectors(ms *MemoryStore, excludeSelectors map[string][]string) [][]string {
   584→	allSelectors := ms.GetPaths(2)
   585→
   586→	filteredSelectors := make([][]string, 0, len(allSelectors))
   587→
   588→	for _, path := range allSelectors {
   589→		if len(path) < 2 {
   590→			continue
   591→		}
   592→
   593→		key := path[0]   // The "Key" (Level 1)
   594→		value := path[1] // The "Value" (Level 2)
   595→
   596→		exclude := false
   597→
   598→		// Check if the key exists in our exclusion map
   599→		if excludedValues, exists := excludeSelectors[key]; exists {
   600→			// The key exists, now check if the specific value is in the exclusion list
   601→			if slices.Contains(excludedValues, value) {
   602→				exclude = true
   603→			}
   604→		}
   605→
   606→		if !exclude {
   607→			filteredSelectors = append(filteredSelectors, path)
   608→		}
   609→	}
   610→
   611→	return filteredSelectors
   612→}
   613→
   614→// GetPaths returns a list of lists (paths) to the specified depth.
   615→func (ms *MemoryStore) GetPaths(targetDepth int) [][]string {
   616→	var results [][]string
   617→
   618→	// Start recursion. Initial path is empty.
   619→	// We treat Root as depth 0.
   620→	ms.root.collectPaths(0, targetDepth, []string{}, &results)
   621→
   622→	return results
   623→}
   624→
   625→// Write all values in `metrics` to the level specified by `selector` for time `ts`.
   626→// Look at `findLevelOrCreate` for how selectors work.
   627→func (m *MemoryStore) Write(selector []string, ts int64, metrics []Metric) error {
   628→	var ok bool
   629→	for i, metric := range metrics {
   630→		if metric.MetricConfig.Frequency == 0 {
   631→			metric.MetricConfig, ok = m.Metrics[metric.Name]
   632→			if !ok {
   633→				cclog.Debugf("[METRICSTORE]> Unknown metric '%s' in Write() - skipping", metric.Name)
   634→				metric.MetricConfig.Frequency = 0
   635→			}
   636→			metrics[i] = metric
   637→		}
   638→	}
   639→
   640→	return m.WriteToLevel(&m.root, selector, ts, metrics)
   641→}
   642→
   643→func (m *MemoryStore) GetLevel(selector []string) *Level {
   644→	return m.root.findLevelOrCreate(selector, len(m.Metrics))
   645→}
   646→
   647→// WriteToLevel assumes that `minfo` in `metrics` is filled in
   648→func (m *MemoryStore) WriteToLevel(l *Level, selector []string, ts int64, metrics []Metric) error {
   649→	l = l.findLevelOrCreate(selector, len(m.Metrics))
   650→	l.lock.Lock()
   651→	defer l.lock.Unlock()
   652→
   653→	for _, metric := range metrics {
   654→		if metric.MetricConfig.Frequency == 0 {
   655→			continue
   656→		}
   657→
   658→		b := l.metrics[metric.MetricConfig.offset]
   659→		if b == nil {
   660→			// First write to this metric and level
   661→			b = newBuffer(ts, metric.MetricConfig.Frequency)
   662→			l.metrics[metric.MetricConfig.offset] = b
   663→		}
   664→
   665→		nb, err := b.write(ts, metric.Value)
   666→		if err != nil {
   667→			return err
   668→		}
   669→
   670→		// Last write created a new buffer...
   671→		if b != nb {
   672→			l.metrics[metric.MetricConfig.offset] = nb
   673→		}
   674→	}
   675→	return nil
   676→}
   677→
   678→// Read returns all values for metric `metric` from `from` to `to` for the selected level(s).
   679→// If the level does not hold the metric itself, the data will be aggregated recursively from the children.
   680→// The second and third return value are the actual from/to for the data. Those can be different from
   681→// the range asked for if no data was available.
   682→func (m *MemoryStore) Read(selector util.Selector, metric string, from, to, resolution int64) ([]schema.Float, int64, int64, int64, error) {
   683→	if from > to {
   684→		return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid time range")
   685→	}
   686→
   687→	minfo, ok := m.Metrics[metric]
   688→	if !ok {
   689→		return nil, 0, 0, 0, errors.New("[METRICSTORE]> unknown metric: " + metric)
   690→	}
   691→
   692→	n, data := 0, make([]schema.Float, (to-from)/minfo.Frequency+1)
   693→
   694→	err := m.root.findBuffers(selector, minfo.offset, func(b *buffer) error {
   695→		cdata, cfrom, cto, err := b.read(from, to, data)
   696→		if err != nil {
   697→			return err
   698→		}
   699→
   700→		if n == 0 {
   701→			from, to = cfrom, cto
   702→		} else if from != cfrom || to != cto || len(data) != len(cdata) {
   703→			missingfront, missingback := int((from-cfrom)/minfo.Frequency), int((to-cto)/minfo.Frequency)
   704→			if missingfront != 0 {
   705→				return ErrDataDoesNotAlign
   706→			}
   707→
   708→			newlen := len(cdata) - missingback
   709→			if newlen < 1 {
   710→				return ErrDataDoesNotAlign
   711→			}
   712→			cdata = cdata[0:newlen]
   713→			if len(cdata) != len(data) {
   714→				return ErrDataDoesNotAlign
   715→			}
   716→
   717→			from, to = cfrom, cto
   718→		}
   719→
   720→		data = cdata
   721→		n += 1
   722→		return nil
   723→	})
   724→
   725→	if err != nil {
   726→		return nil, 0, 0, 0, err
   727→	} else if n == 0 {
   728→		return nil, 0, 0, 0, ErrNoHostOrMetric
   729→	} else if n > 1 {
   730→		if minfo.Aggregation == AvgAggregation {
   731→			normalize := 1. / schema.Float(n)
   732→			for i := 0; i < len(data); i++ {
   733→				data[i] *= normalize
   734→			}
   735→		} else if minfo.Aggregation != SumAggregation {
   736→			return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid aggregation")
   737→		}
   738→	}
   739→
   740→	data, resolution, err = resampler.LargestTriangleThreeBucket(data, minfo.Frequency, resolution)
   741→	if err != nil {
   742→		return nil, 0, 0, 0, err
   743→	}
   744→
   745→	return data, from, to, resolution, nil
   746→}
   747→
   748→// Free releases all buffers for the selected level and all its children that
   749→// contain only values older than `t`.
   750→func (m *MemoryStore) Free(selector []string, t int64) (int, error) {
   751→	return m.GetLevel(selector).free(t)
   752→}
   753→
   754→// ForceFree unconditionally removes the oldest buffer from each metric chain.
   755→func (m *MemoryStore) ForceFree() (int, error) {
   756→	return m.GetLevel(nil).forceFree()
   757→}
   758→
   759→func (m *MemoryStore) FreeAll() error {
   760→	m.root.lock.Lock()
   761→	defer m.root.lock.Unlock()
   762→
   763→	for k := range m.root.children {
   764→		delete(m.root.children, k)
   765→	}
   766→
   767→	return nil
   768→}
   769→
   770→func (m *MemoryStore) SizeInBytes() int64 {
   771→	return m.root.sizeInBytes()
   772→}
   773→
   774→func (m *MemoryStore) SizeInGB() float64 {
   775→	return float64(m.root.sizeInBytes()) / 1e9
   776→}
   777→
   778→// ListChildren , given a selector, returns a list of all children of the level
   779→// selected.
   780→func (m *MemoryStore) ListChildren(selector []string) []string {
   781→	lvl := &m.root
   782→	for lvl != nil && len(selector) != 0 {
   783→		lvl.lock.RLock()
   784→		next := lvl.children[selector[0]]
   785→		lvl.lock.RUnlock()
   786→		lvl = next
   787→		selector = selector[1:]
   788→	}
   789→
   790→	if lvl == nil {
   791→		return nil
   792→	}
   793→
   794→	lvl.lock.RLock()
   795→	defer lvl.lock.RUnlock()
   796→
   797→	children := make([]string, 0, len(lvl.children))
   798→	for child := range lvl.children {
   799→		children = append(children, child)
   800→	}
   801→
   802→	return children
   803→}
   804→


Found 4 files
pkg/metricstore/metricstore.go
pkg/metricstore/walCheckpoint.go
pkg/metricstore/checkpoint.go
pkg/metricstore/config.go


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// This file implements checkpoint persistence for the in-memory metric store.
     7→//
     8→// Checkpoints enable graceful restarts by periodically saving in-memory metric
     9→// data to disk in JSON or binary format. The checkpoint system:
    10→//
    11→// Key Features:
    12→//   - Periodic background checkpointing via the Checkpointing() worker
    13→//   - Two format families: JSON (human-readable) and WAL+binary (compact, crash-safe)
    14→//   - Parallel checkpoint creation and loading using worker pools
    15→//   - Hierarchical file organization: checkpoint_dir/cluster/host/timestamp.{json|bin}
    16→//   - WAL file: checkpoint_dir/cluster/host/current.wal (append-only, per-entry)
    17→//   - Only saves unarchived data (archived data is already persisted elsewhere)
    18→//   - GC optimization during loading to prevent excessive heap growth
    19→//
    20→// Checkpoint Workflow:
    21→//  1. Init() loads checkpoints within retention window at startup
    22→//  2. Checkpointing() worker periodically saves new data
    23→//  3. Shutdown() writes final checkpoint before exit
    24→//
    25→// File Organization:
    26→//
    27→//	checkpoints/
    28→//	  cluster1/
    29→//	    host001/
    30→//	      1234567890.json  (JSON format: full subtree snapshot)
    31→//	      1234567890.bin   (binary format: full subtree snapshot)
    32→//	      current.wal      (WAL format: append-only per-entry log)
    33→//	    host002/
    34→//	      ...
    35→package metricstore
    36→
    37→import (
    38→	"bufio"
    39→	"context"
    40→	"encoding/json"
    41→	"errors"
    42→	"fmt"
    43→	"io/fs"
    44→	"os"
    45→	"path"
    46→	"path/filepath"
    47→	"sort"
    48→	"strconv"
    49→	"strings"
    50→	"sync"
    51→	"sync/atomic"
    52→	"time"
    53→
    54→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    55→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    56→)
    57→
    58→const (
    59→	CheckpointFilePerms = 0o644                    // File permissions for checkpoint files
    60→	CheckpointDirPerms  = 0o755                    // Directory permissions for checkpoint directories
    61→	GCTriggerInterval   = DefaultGCTriggerInterval // Interval for triggering GC during checkpoint loading
    62→)
    63→
    64→// CheckpointMetrics represents metric data in a checkpoint file.
    65→// Whenever the structure changes, update MarshalJSON as well!
    66→type CheckpointMetrics struct {
    67→	Data      []schema.Float `json:"data"`
    68→	Frequency int64          `json:"frequency"`
    69→	Start     int64          `json:"start"`
    70→}
    71→
    72→// CheckpointFile represents the hierarchical structure of a checkpoint file.
    73→// It mirrors the Level tree structure from the MemoryStore.
    74→type CheckpointFile struct {
    75→	Metrics  map[string]*CheckpointMetrics `json:"metrics"`
    76→	Children map[string]*CheckpointFile    `json:"children"`
    77→	From     int64                         `json:"from"`
    78→	To       int64                         `json:"to"`
    79→}
    80→
    81→// lastCheckpoint tracks the timestamp of the last checkpoint creation.
    82→var (
    83→	lastCheckpoint   time.Time
    84→	lastCheckpointMu sync.Mutex
    85→)
    86→
    87→// Checkpointing starts a background worker that periodically saves metric data to disk.
    88→//
    89→// Checkpoints are written every 12 hours (hardcoded).
    90→//
    91→// Format behaviour:
    92→//   - "json": Periodic checkpointing every checkpointInterval
    93→//   - "wal":  Periodic binary snapshots + WAL rotation every checkpointInterval
    94→func Checkpointing(wg *sync.WaitGroup, ctx context.Context) {
    95→	lastCheckpointMu.Lock()
    96→	lastCheckpoint = time.Now()
    97→	lastCheckpointMu.Unlock()
    98→
    99→	ms := GetMemoryStore()
   100→
   101→	wg.Go(func() {
   102→
   103→		d := 12 * time.Hour // default checkpoint interval
   104→		if Keys.CheckpointInterval != "" {
   105→			parsed, err := time.ParseDuration(Keys.CheckpointInterval)
   106→			if err != nil {
   107→				cclog.Errorf("[METRICSTORE]> invalid checkpoint-interval %q: %s, using default 12h", Keys.CheckpointInterval, err)
   108→			} else {
   109→				d = parsed
   110→			}
   111→		}
   112→
   113→		ticker := time.NewTicker(d)
   114→		defer ticker.Stop()
   115→
   116→		for {
   117→			select {
   118→			case <-ctx.Done():
   119→				return
   120→			case <-ticker.C:
   121→				lastCheckpointMu.Lock()
   122→				from := lastCheckpoint
   123→				lastCheckpointMu.Unlock()
   124→
   125→				now := time.Now()
   126→				cclog.Infof("[METRICSTORE]> start checkpointing (starting at %s)...", from.Format(time.RFC3339))
   127→
   128→				if Keys.Checkpoints.FileFormat == "wal" {
   129→					n, hostDirs, err := ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
   130→					if err != nil {
   131→						cclog.Errorf("[METRICSTORE]> binary checkpointing failed: %s", err.Error())
   132→					} else {
   133→						cclog.Infof("[METRICSTORE]> done: %d binary snapshot files created", n)
   134→						lastCheckpointMu.Lock()
   135→						lastCheckpoint = now
   136→						lastCheckpointMu.Unlock()
   137→						// Rotate WAL files for successfully checkpointed hosts.
   138→						RotateWALFiles(hostDirs)
   139→					}
   140→				} else {
   141→					n, err := ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), now.Unix())
   142→					if err != nil {
   143→						cclog.Errorf("[METRICSTORE]> checkpointing failed: %s", err.Error())
   144→					} else {
   145→						cclog.Infof("[METRICSTORE]> done: %d checkpoint files created", n)
   146→						lastCheckpointMu.Lock()
   147→						lastCheckpoint = now
   148→						lastCheckpointMu.Unlock()
   149→					}
   150→				}
   151→			}
   152→		}
   153→	})
   154→}
   155→
   156→// MarshalJSON provides optimized JSON encoding for CheckpointMetrics.
   157→//
   158→// Since schema.Float has custom MarshalJSON, serializing []Float has significant overhead.
   159→// This method manually constructs JSON to avoid allocations and interface conversions.
   160→func (cm *CheckpointMetrics) MarshalJSON() ([]byte, error) {
   161→	buf := make([]byte, 0, 128+len(cm.Data)*8)
   162→	buf = append(buf, `{"frequency":`...)
   163→	buf = strconv.AppendInt(buf, cm.Frequency, 10)
   164→	buf = append(buf, `,"start":`...)
   165→	buf = strconv.AppendInt(buf, cm.Start, 10)
   166→	buf = append(buf, `,"data":[`...)
   167→	for i, x := range cm.Data {
   168→		if i != 0 {
   169→			buf = append(buf, ',')
   170→		}
   171→		if x.IsNaN() {
   172→			buf = append(buf, `null`...)
   173→		} else {
   174→			buf = strconv.AppendFloat(buf, float64(x), 'f', 1, 32)
   175→		}
   176→	}
   177→	buf = append(buf, `]}`...)
   178→	return buf, nil
   179→}
   180→
   181→// ToCheckpoint writes metric data to checkpoint files in parallel (JSON format).
   182→//
   183→// Metrics at root and cluster levels are skipped. One file per host is created.
   184→// Uses worker pool (Keys.NumWorkers) for parallel processing. Only locks one host
   185→// at a time, allowing concurrent writes/reads to other hosts.
   186→//
   187→// Returns the number of checkpoint files created and any errors encountered.
   188→func (m *MemoryStore) ToCheckpoint(dir string, from, to int64) (int, error) {
   189→	// Pre-calculate capacity by counting cluster/host pairs
   190→	m.root.lock.RLock()
   191→	totalHosts := 0
   192→	for _, l1 := range m.root.children {
   193→		l1.lock.RLock()
   194→		totalHosts += len(l1.children)
   195→		l1.lock.RUnlock()
   196→	}
   197→	m.root.lock.RUnlock()
   198→
   199→	levels := make([]*Level, 0, totalHosts)
   200→	selectors := make([][]string, 0, totalHosts)


Perfect! Now I have comprehensive information about the WAL message flow. Let me compile my findings:

## WAL Message Flow Analysis

### 1. **Channel Creation & Buffer Sizing** (lines 147-161 in walCheckpoint.go)

```go
// WALStaging function initializes the channels:
walNumShards = max(Keys.NumWorkers, 1)  // Number of shards = NumWorkers
chBufSize := max(65536/walNumShards, 1024)  // Per-shard buffer size

for i := range walNumShards {
    walShardChs[i] = make(chan *WALMessage, chBufSize)
    walShardRotateChs[i] = make(chan walRotateReq, 64)
}
```

**Buffer Size Logic:**
- Total global buffer: 65,536 messages
- Divided by number of shards (= NumWorkers)
- Minimum per-shard buffer: 1,024 messages
- Example: With 4 workers → 16,384 messages per shard
- Example: With 64+ workers → 1,024 messages per shard (hits minimum)

### 2. **Producer Code** (lines 355-371 in lineprotocol.go)

This is where WAL messages are sent when a metric is decoded:

```go
var walDropped atomic.Int64  // Line 129 - global counter

if Keys.Checkpoints.FileFormat == "wal" {
    msg := &WALMessage{
        MetricName: string(st.metricBuf),
        Cluster:    cluster,
        Node:       host,
        Selector:   append([]string{}, st.selector...),
        Value:      metric.Value,
        Timestamp:  time.Unix(),
    }
    if !SendWALMessage(msg) {
        // Channel is full — metric is in memory but not WAL
        if dropped := walDropped.Add(1); dropped%10000 == 1 {
            cclog.Warnf("[METRICSTORE]> WAL shard channel full, dropped %d messages...", dropped)
        }
    }
}
ms.WriteToLevel(...)  // Always write to memory, regardless of WAL status
```

**SendWALMessage function** (lines 128-141 in walCheckpoint.go):
```go
func SendWALMessage(msg *WALMessage) bool {
    if walShardChs == nil {
        return false
    }
    shard := walShardIndex(msg.Cluster, msg.Node)  // FNV-1a hash
    select {
    case walShardChs[shard] <- msg:
        return true
    default:
        return false  // Non-blocking: returns false if channel full
    }
}
```

**Key Points:**
- Non-blocking send using select with default case
- Returns `false` immediately if channel is full (no retry, no wait)
- Metric still written to memory store regardless
- Counter incremented and logged every 10,000 drops

### 3. **Consumer Code (WAL Staging Goroutines)** (lines 147-299 in walCheckpoint.go)

Each of N shards has its own goroutine that:

**Main message processing loop:**
```go
for {
    select {
    case <-ctx.Done():
        drain()  // Drain pending messages before exit
        return
    case msg, ok := <-msgCh:
        if !ok {
            return
        }
        processMsg(msg)

        // Batch draining: pull up to 256 more messages without blocking
        for range 256 {
            select {
            case msg, ok := <-msgCh:
                if !ok {
                    return
                }
                processMsg(msg)
            case req := <-rotateCh:
                processRotate(req)
            default:
                goto flushed  // Exit inner loop to flush
            }
        }
    flushed:
        flushAll()  // Flush bufio.Writer to all open WAL files

    case req := <-rotateCh:
        processRotate(req)
    }
}
```

**Message processing:**
```go
processMsg := func(msg *WALMessage) {
    hostDir := path.Join(Keys.Checkpoints.RootDir, msg.Cluster, msg.Node)
    ws := getOrOpenWAL(hostDir)  // Lazily open per-host WAL file
    if ws == nil {
        return
    }
    if err := writeWALRecordDirect(ws.w, msg) {
        cclog.Errorf("[METRICSTORE]> WAL: write record: %v", err)
    }
}
```

### 4. **Flow Control & Batching Mechanisms**

**Batch Draining (lines 277-290):**
- After receiving first message, goroutine tries to drain up to 256 more
- Non-blocking attempts prevent stalling on empty channels
- Reduces flush frequency and improves I/O efficiency

**Flushing Strategy:**
- `flushAll()` flushes all open bufio.Writer instances (lines 241-247)
- Called after batch drain or when handling rotation requests
- Uses `bufio.Writer` for efficient buffering

**Rotation Mechanism (lines 304-319):**
- `RotateWALFiles()` sends rotation requests to shard channels
- Waits for completion via done channel
- On rotation: flush file, close WAL, delete current.wal
- Next checkpoint starts fresh WAL file

### 5. **Context Around the Warning Message** (line 368 in lineprotocol.go)

```go
if !SendWALMessage(msg) {
    // WAL shard channel full — metric is written to memory store but not WAL.
    // Next binary snapshot will capture it.
    if dropped := walDropped.Add(1); dropped%10000 == 1 {
        cclog.Warnf("[METRICSTORE]> WAL shard channel full, dropped %d messages "+
                    "(data safe in memory, next snapshot will capture)", dropped)
    }
}
```

**Safety Note:** Messages dropped from WAL channel are NOT lost because:
1. Metric is always written to in-memory store (`ms.WriteToLevel()`)
2. In-memory data is periodically checkpointed to `.bin` snapshot files
3. Snapshot writes capture all metrics regardless of WAL drops
4. Only continuous crash-safety is lost (WAL provides per-write safety)

### Summary Table

| Component | Value | Configuration |
|-----------|-------|----------------|
| **Number of WAL shards** | NumWorkers (1-10 default) | `Keys.NumWorkers` |
| **Per-shard channel size** | 65536/shards (min 1024) | Hardcoded in line 153 |
| **Total channel capacity** | 65,536 messages | Hardcoded |
| **Batch drain size** | 256 messages | Hardcoded in line 278 |
| **Rotation channel size** | 64 | Hardcoded in line 160 |
| **Send behavior** | Non-blocking | Select with default case |
| **Drop counter** | Logged every 10,000 | Modulo check in line 367 |

---

**Absolute File Paths:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/walCheckpoint.go`
- `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/lineprotocol.go`
- `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/metricstore.go`
- `/Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/checkpoint.go`

agentId: a3b1b582c2e5509be (use SendMessage with to: 'a3b1b582c2e5509be' to continue this agent)
<usage>total_tokens: 52693
tool_uses: 5
duration_ms: 24358</usage>

Prompt:
I need to understand the WAL message flow in the metricstore, specifically around WAL shard channels getting full and dropping messages.

Explore these areas thoroughly:

1. In `pkg/metricstore/lineprotocol.go` — find where WAL messages are sent to channels, especially around line 368 where the warning "WAL shard channel full, dropped" is logged. Understand how messages are produced.

2. In `pkg/metricstore/metricstore.go` and `pkg/metricstore/walCheckpoint.go` — find where WAL channels are created (their buffer sizes), how `WALStaging()` or equivalent consumers work, and how messages are consumed/written.

3. Look for any configuration of channel buffer sizes, shard counts, or concurrency settings related to WAL.

Report back:
- The exact channel creation code (buffer sizes)
- The producer code (how messages are sent, what happens when channel is full)
- The consumer code (how messages are drained/written)
- Any batching or flow control mechanisms
- The full context around the warning message


Let me read the key sections to confirm the details.