Implement the following plan:

# Metricstore NATS Slow Consumer Analysis & Improvements

## Context

The metricstore NATS subscription triggers "slow consumer, messages dropped" warnings under load. The root cause is that the NATS message callback blocks for too long, causing the nats.go client to detect a slow consumer and drop messages from its internal pending buffer.

## Root Cause Analysis

There are **three compounding bottlenecks** in the ingestion path:

### 1. Single-threaded inline decoding (primary cause)
- `metricstore.go:187` calls `ReceiveNats(ms, 1, ctx)` — **workers=1**
- With workers=1, `DecodeLine()` runs **directly inside the NATS callback** (`lineprotocol.go:82-87`)
- The NATS client cannot deliver the next message until the callback returns
- Each `DecodeLine` call parses line protocol, acquires locks on the level tree, and may block on the WAL channel

### 2. Blocking WAL channel send
- `lineprotocol.go:351`: `WALMessages <- &WALMessage{...}` is a **blocking send**
- The `WALMessages` channel has a buffer of only 4096 (`walCheckpoint.go:85`)
- If the single WALStaging goroutine can't drain fast enough (file I/O), this blocks `DecodeLine`, which blocks the NATS callback
- Under sustained load, the WAL channel fills up and becomes the dominant bottleneck

### 3. Single WALStaging goroutine with per-message I/O
- `WALStaging()` (`walCheckpoint.go:118-234`) runs as a **single goroutine**
- It processes messages sequentially: open/lookup file handle → serialize → write via bufio
- While bufio batches actual syscalls, the goroutine still handles one message at a time
- File handle lookups (map access) and potential `MkdirAll` calls add latency

### How nats.go detects slow consumers
The nats.go library maintains an internal per-subscription message queue. When the callback blocks, messages accumulate. Once the pending count/bytes exceed limits (default: 65536 messages / 64MB), it drops messages and reports "slow consumer".

## Proposed Improvements

### Change 1: Increase worker count and channel buffer
**File:** `pkg/metricstore/metricstore.go:187` and `pkg/metricstore/lineprotocol.go:56`

- Pass `Keys.NumWorkers` (or a dedicated config field) instead of hardcoded `1`
- Increase the `msgs` channel buffer from `workers*2` to `workers*256` or a dedicated configurable value (default 16384)
- This parallelizes `DecodeLine()` across multiple goroutines and decouples the NATS callback from processing

```go
// metricstore.go — change workers from 1 to NumWorkers
err = ReceiveNats(ms, Keys.NumWorkers, ctx)
```

```go
// lineprotocol.go — larger channel buffer
msgs := make(chan []byte, max(workers*256, 8192))
```

### Change 2: Increase WAL channel buffer
**File:** `pkg/metricstore/walCheckpoint.go:85`

- Increase from 4096 to 65536 or higher
- This gives the WAL staging goroutine more headroom during I/O bursts

```go
var WALMessages = make(chan *WALMessage, 65536)
```

### Change 3: Make WAL send non-blocking with overflow logging
**File:** `pkg/metricstore/lineprotocol.go:350-359`

- Use a non-blocking send with `select/default` and log dropped WAL messages
- The in-memory store write (line 361) already succeeded — WAL is for crash recovery only
- A periodic binary snapshot will capture the data anyway; a few missed WAL entries between snapshots are acceptable

```go
if Keys.Checkpoints.FileFormat == "wal" {
    select {
    case WALMessages <- &WALMessage{...}:
    default:
        // WAL channel full — message written to memory store but not WAL.
        // Next binary snapshot will capture it.
    }
}
```

### Change 4: Batch WAL writes in WALStaging
**File:** `pkg/metricstore/walCheckpoint.go:219-231`

- Drain multiple messages per iteration before going back to `select`
- Periodically flush all writers (e.g., after draining or on a timer)

```go
// After processing one message from the channel, drain up to N more without blocking
for range 256 {
    select {
    case msg := <-WALMessages:
        processMsg(msg)
    default:
        goto flushed
    }
}
flushed:
// Flush all dirty writers
```

### Change 5 (cc-lib, optional): Set subscription pending limits
**File:** cc-lib `nats/client.go` — after `conn.Subscribe()`, call `sub.SetPendingLimits()`

- Increase pending message/byte limits on the subscription object
- This is a cc-lib change and would need a separate PR

## Implementation Priority

| # | Change | Impact | Effort |
|---|--------|--------|--------|
| 1 | Increase workers + channel buffer | **High** — parallelizes decoding, decouples callback | Low |
| 2 | Increase WAL channel buffer | **Medium** — more burst headroom | Trivial |
| 3 | Non-blocking WAL send | **High** — eliminates WAL as backpressure source | Low |
| 4 | Batch WAL drain | **Medium** — reduces select overhead | Low |
| 5 | cc-lib pending limits | **Medium** — increases NATS-side tolerance | Separate repo |

Changes 1-4 are all in cc-backend and can be done together.

## Files to Modify

- `pkg/metricstore/metricstore.go` — worker count parameter (line 187)
- `pkg/metricstore/lineprotocol.go` — channel buffer size (line 56), non-blocking WAL send (lines 350-359)
- `pkg/metricstore/walCheckpoint.go` — WAL channel buffer (line 85), batch drain loop (lines 219-231)

## Verification

1. `go build ./...` — ensure compilation
2. `go test ./pkg/metricstore/...` — run existing tests
3. Deploy to a test cluster and monitor:
   - Absence of "slow consumer" log messages
   - WAL file growth rate (should be similar)
   - Memory usage (larger channel buffers add ~few MB)
   - `pending` and `dropped` counts from the NATS error handler in cc-lib

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-ClusterCockpit-cc-backend/c276bf22-8137-44c9-a3e4-951a700f423f.jsonl


Let me read the files that need modification.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstore provides an efficient in-memory time-series metric storage system
     7→// with support for hierarchical data organization, checkpointing, and archiving.
     8→//
     9→// The package organizes metrics in a tree structure (cluster → host → component) and
    10→// provides concurrent read/write access to metric data with configurable aggregation strategies.
    11→// Background goroutines handle periodic checkpointing (JSON or Avro format), archiving old data,
    12→// and enforcing retention policies.
    13→//
    14→// Key features:
    15→//   - In-memory metric storage with configurable retention
    16→//   - Hierarchical data organization (selectors)
    17→//   - Concurrent checkpoint/archive workers
    18→//   - Support for sum and average aggregation
    19→//   - NATS integration for metric ingestion
    20→package metricstore
    21→
    22→import (
    23→	"bytes"
    24→	"context"
    25→	"encoding/json"
    26→	"errors"
    27→	"fmt"
    28→	"runtime"
    29→	"runtime/debug"
    30→	"slices"
    31→	"sync"
    32→	"time"
    33→
    34→	"github.com/ClusterCockpit/cc-backend/internal/config"
    35→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    36→	"github.com/ClusterCockpit/cc-lib/v2/resampler"
    37→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    38→	"github.com/ClusterCockpit/cc-lib/v2/util"
    39→)
    40→
    41→// GlobalState holds the global state for the metric store with thread-safe access.
    42→type GlobalState struct {
    43→	mu                sync.RWMutex
    44→	lastRetentionTime int64
    45→	selectorsExcluded bool
    46→}
    47→
    48→var (
    49→	singleton  sync.Once
    50→	msInstance *MemoryStore
    51→	// shutdownFunc stores the context cancellation function created in Init
    52→	// and is called during Shutdown to cancel all background goroutines
    53→	shutdownFunc   context.CancelFunc
    54→	shutdownFuncMu sync.Mutex // Protects shutdownFunc from concurrent access
    55→	// Create a global instance
    56→	state = &GlobalState{}
    57→)
    58→
    59→// NodeProvider provides information about nodes currently in use by running jobs.
    60→//
    61→// This interface allows metricstore to query job information without directly
    62→// depending on the repository package, breaking the import cycle.
    63→//
    64→// Implementations should return nodes that are actively processing jobs started
    65→// before the given timestamp. These nodes will be excluded from retention-based
    66→// garbage collection to prevent data loss for jobs that are still running or
    67→// recently completed.
    68→type NodeProvider interface {
    69→	// GetUsedNodes returns a map of cluster names to sorted lists of unique hostnames
    70→	// that are currently in use by jobs that started before the given timestamp.
    71→	//
    72→	// Parameters:
    73→	//   - ts: Unix timestamp threshold - returns nodes with jobs started before this time
    74→	//
    75→	// Returns:
    76→	//   - Map of cluster names to lists of node hostnames that should be excluded from garbage collection
    77→	//   - Error if the query fails
    78→	GetUsedNodes(ts int64) (map[string][]string, error)
    79→}
    80→
    81→// Metric represents a single metric data point to be written to the store.
    82→type Metric struct {
    83→	Name  string
    84→	Value schema.Float
    85→	// MetricConfig contains frequency and aggregation settings for this metric.
    86→	// If Frequency is 0, configuration will be looked up from MemoryStore.Metrics during Write().
    87→	MetricConfig MetricConfig
    88→}
    89→
    90→// MemoryStore is the main in-memory time-series metric storage implementation.
    91→//
    92→// It organizes metrics in a hierarchical tree structure where each level represents
    93→// a component of the system hierarchy (e.g., cluster → host → CPU). Each level can
    94→// store multiple metrics as time-series buffers.
    95→//
    96→// The store is initialized as a singleton via InitMetrics() and accessed via GetMemoryStore().
    97→// All public methods are safe for concurrent use.
    98→type MemoryStore struct {
    99→	Metrics      map[string]MetricConfig
   100→	root         Level
   101→	nodeProvider NodeProvider
   102→}
   103→
   104→// Init initializes the metric store from configuration and starts background workers.
   105→//
   106→// This function must be called exactly once before any other metricstore operations.
   107→// It performs the following initialization steps:
   108→//  1. Validates and decodes the metric store configuration
   109→//  2. Configures worker pool size (defaults to NumCPU/2+1, max 10)
   110→//  3. Loads metric configurations from all registered clusters
   111→//  4. Restores checkpoints within the retention window
   112→//  5. Starts background workers for retention, checkpointing, archiving, and monitoring
   113→//  6. Optionally subscribes to NATS for real-time metric ingestion
   114→//
   115→// Parameters:
   116→//   - rawConfig: JSON configuration for the metric store (see MetricStoreConfig)
   117→//   - wg: WaitGroup that will be incremented for each background goroutine started
   118→//
   119→// The function will call cclog.Fatal on critical errors during initialization.
   120→// Use Shutdown() to cleanly stop all background workers started by Init().
   121→//
   122→// Note: Signal handling must be implemented by the caller. Call Shutdown() when
   123→// receiving termination signals to ensure checkpoint data is persisted.
   124→func Init(rawConfig json.RawMessage, metrics map[string]MetricConfig, wg *sync.WaitGroup) {
   125→	startupTime := time.Now()
   126→
   127→	if rawConfig != nil {
   128→		config.Validate(configSchema, rawConfig)
   129→		dec := json.NewDecoder(bytes.NewReader(rawConfig))
   130→		dec.DisallowUnknownFields()
   131→		if err := dec.Decode(&Keys); err != nil {
   132→			cclog.Abortf("[METRICSTORE]> Metric Store Config Init: Could not decode config file '%s'.\nError: %s\n", rawConfig, err.Error())
   133→		}
   134→	}
   135→
   136→	// Set NumWorkers from config or use default
   137→	if Keys.NumWorkers <= 0 {
   138→		Keys.NumWorkers = min(runtime.NumCPU()/2+1, DefaultMaxWorkers)
   139→	}
   140→	cclog.Debugf("[METRICSTORE]> Using %d workers for checkpoint/archive operations\n", Keys.NumWorkers)
   141→
   142→	// Pass the config.MetricStoreKeys
   143→	InitMetrics(metrics)
   144→
   145→	ms := GetMemoryStore()
   146→
   147→	d, err := time.ParseDuration(Keys.RetentionInMemory)
   148→	if err != nil {
   149→		cclog.Fatal(err)
   150→	}
   151→
   152→	restoreFrom := startupTime.Add(-d)
   153→	cclog.Infof("[METRICSTORE]> Loading checkpoints newer than %s\n", restoreFrom.Format(time.RFC3339))
   154→	files, err := ms.FromCheckpointFiles(Keys.Checkpoints.RootDir, restoreFrom.Unix())
   155→	loadedData := ms.SizeInBytes() / 1024 / 1024 // In MB
   156→	if err != nil {
   157→		cclog.Fatalf("[METRICSTORE]> Loading checkpoints failed: %s\n", err.Error())
   158→	} else {
   159→		cclog.Infof("[METRICSTORE]> Checkpoints loaded (%d files, %d MB, that took %fs)\n", files, loadedData, time.Since(startupTime).Seconds())
   160→	}
   161→
   162→	// Try to use less memory by forcing a GC run here and then
   163→	// lowering the target percentage. The default of 100 means
   164→	// that only once the ratio of new allocations execeds the
   165→	// previously active heap, a GC is triggered.
   166→	// Forcing a GC here will set the "previously active heap"
   167→	// to a minumum.
   168→	// runtime.GC()
   169→
   170→	ctx, shutdown := context.WithCancel(context.Background())
   171→
   172→	Retention(wg, ctx)
   173→	Checkpointing(wg, ctx)
   174→	CleanUp(wg, ctx)
   175→	WALStaging(wg, ctx)
   176→	MemoryUsageTracker(wg, ctx)
   177→
   178→	// Note: Signal handling has been removed from this function.
   179→	// The caller is responsible for handling shutdown signals and calling
   180→	// the shutdown() function when appropriate.
   181→	// Store the shutdown function for later use by Shutdown()
   182→	shutdownFuncMu.Lock()
   183→	shutdownFunc = shutdown
   184→	shutdownFuncMu.Unlock()
   185→
   186→	if Keys.Subscriptions != nil {
   187→		err = ReceiveNats(ms, 1, ctx)
   188→		if err != nil {
   189→			cclog.Fatal(err)
   190→		}
   191→	}
   192→}
   193→
   194→// InitMetrics initializes the singleton MemoryStore instance with the given metric configurations.
   195→//
   196→// This function must be called before GetMemoryStore() and can only be called once due to
   197→// the singleton pattern. It assigns each metric an internal offset for efficient buffer indexing.
   198→//
   199→// Parameters:
   200→//   - metrics: Map of metric names to their configurations (frequency and aggregation strategy)
   201→//
   202→// Panics if any metric has Frequency == 0, which indicates an invalid configuration.
   203→//
   204→// After this call, the global msInstance is ready for use via GetMemoryStore().
   205→func InitMetrics(metrics map[string]MetricConfig) {
   206→	singleton.Do(func() {
   207→		offset := 0
   208→		for key, cfg := range metrics {
   209→			if cfg.Frequency == 0 {
   210→				panic("[METRICSTORE]> invalid frequency")
   211→			}
   212→
   213→			metrics[key] = MetricConfig{
   214→				Frequency:   cfg.Frequency,
   215→				Aggregation: cfg.Aggregation,
   216→				offset:      offset,
   217→			}
   218→			offset += 1
   219→		}
   220→
   221→		msInstance = &MemoryStore{
   222→			root: Level{
   223→				metrics:  make([]*buffer, len(metrics)),
   224→				children: make(map[string]*Level),
   225→			},
   226→			Metrics: metrics,
   227→		}
   228→	})
   229→}
   230→
   231→// GetMemoryStore returns the singleton MemoryStore instance.
   232→//
   233→// Returns the initialized MemoryStore singleton. Calls cclog.Fatal if InitMetrics() was not called first.
   234→//
   235→// This function is safe for concurrent use after initialization.
   236→func GetMemoryStore() *MemoryStore {
   237→	if msInstance == nil {
   238→		cclog.Warnf("[METRICSTORE]> MemoryStore not initialized!")
   239→	}
   240→
   241→	return msInstance
   242→}
   243→
   244→func (ms *MemoryStore) GetMetricFrequency(metricName string) (int64, error) {
   245→	if metric, ok := ms.Metrics[metricName]; ok {
   246→		return metric.Frequency, nil
   247→	}
   248→	return 0, fmt.Errorf("[METRICSTORE]> metric %s not found", metricName)
   249→}
   250→
   251→// SetNodeProvider sets the NodeProvider implementation for the MemoryStore.
   252→// This must be called during initialization to provide job state information
   253→// for selective buffer retention during Free operations.
   254→// If not set, the Free function will fall back to freeing all buffers.
   255→func (ms *MemoryStore) SetNodeProvider(provider NodeProvider) {
   256→	ms.nodeProvider = provider
   257→}
   258→
   259→// Shutdown performs a graceful shutdown of the metric store.
   260→//
   261→// This function cancels all background goroutines started by Init() and writes
   262→// a final checkpoint to disk before returning. It should be called when the
   263→// application receives a termination signal.
   264→//
   265→// The function will:
   266→//  1. Cancel the context to stop all background workers
   267→//  2. Close the WAL messages channel if using WAL format
   268→//  3. Write a final checkpoint to preserve in-memory data
   269→//  4. Log any errors encountered during shutdown
   270→//
   271→// Note: This function blocks until the final checkpoint is written.
   272→func Shutdown() {
   273→	shutdownFuncMu.Lock()
   274→	defer shutdownFuncMu.Unlock()
   275→	if shutdownFunc != nil {
   276→		shutdownFunc()
   277→	}
   278→
   279→	if Keys.Checkpoints.FileFormat == "wal" {
   280→		close(WALMessages)
   281→	}
   282→
   283→	cclog.Infof("[METRICSTORE]> Writing to '%s'...\n", Keys.Checkpoints.RootDir)
   284→	var files int
   285→	var err error
   286→
   287→	ms := GetMemoryStore()
   288→
   289→	lastCheckpointMu.Lock()
   290→	from := lastCheckpoint
   291→	lastCheckpointMu.Unlock()
   292→
   293→	if Keys.Checkpoints.FileFormat == "wal" {
   294→		var hostDirs []string
   295→		files, hostDirs, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
   296→		if err == nil {
   297→			RotateWALFilesAfterShutdown(hostDirs)
   298→		}
   299→	} else {
   300→		files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
   301→	}
   302→
   303→	if err != nil {
   304→		cclog.Errorf("[METRICSTORE]> Writing checkpoint failed: %s\n", err.Error())
   305→	}
   306→	cclog.Infof("[METRICSTORE]> Done! (%d files written)\n", files)
   307→}
   308→
   309→// Retention starts a background goroutine that periodically frees old metric data.
   310→//
   311→// This worker runs at half the retention interval and calls Free() to remove buffers
   312→// older than the configured retention time. It respects the NodeProvider to preserve
   313→// data for nodes with active jobs.
   314→//
   315→// Parameters:
   316→//   - wg: WaitGroup to signal completion when context is cancelled
   317→//   - ctx: Context for cancellation signal
   318→//
   319→// The goroutine exits when ctx is cancelled.
   320→func Retention(wg *sync.WaitGroup, ctx context.Context) {
   321→	ms := GetMemoryStore()
   322→
   323→	wg.Go(func() {
   324→		d, err := time.ParseDuration(Keys.RetentionInMemory)
   325→		if err != nil {
   326→			cclog.Fatal(err)
   327→		}
   328→		if d <= 0 {
   329→			return
   330→		}
   331→
   332→		tickInterval := d / 2
   333→		if tickInterval <= 0 {
   334→			return
   335→		}
   336→		ticker := time.NewTicker(tickInterval)
   337→		defer ticker.Stop()
   338→
   339→		for {
   340→			select {
   341→			case <-ctx.Done():
   342→				return
   343→			case <-ticker.C:
   344→				state.mu.Lock()
   345→
   346→				t := time.Now().Add(-d)
   347→
   348→				state.lastRetentionTime = t.Unix()
   349→
   350→				cclog.Infof("[METRICSTORE]> start freeing buffers (older than %s)...\n", t.Format(time.RFC3339))
   351→
   352→				freed, err := Free(ms, t)
   353→				if err != nil {
   354→					cclog.Errorf("[METRICSTORE]> freeing up buffers failed: %s\n", err.Error())
   355→				} else {
   356→					cclog.Infof("[METRICSTORE]> done: %d buffers freed\n", freed)
   357→				}
   358→
   359→				state.mu.Unlock()
   360→
   361→				// Clean up the buffer pool
   362→				bufferPool.Clean(state.lastRetentionTime)
   363→			}
   364→		}
   365→	})
   366→}
   367→
   368→// MemoryUsageTracker starts a background goroutine that monitors memory usage.
   369→//
   370→// This worker checks actual process memory usage (via runtime.MemStats) periodically
   371→// and force-frees buffers if memory exceeds the configured cap. It uses FreeOSMemory()
   372→// to return memory to the OS after freeing buffers, avoiding aggressive GC that causes
   373→// performance issues.
   374→//
   375→// The tracker logs both actual memory usage (heap allocated) and metric data size for
   376→// visibility into memory overhead from Go runtime structures and allocations.
   377→//
   378→// Parameters:
   379→//   - wg: WaitGroup to signal completion when context is cancelled
   380→//   - ctx: Context for cancellation signal
   381→//
   382→// The goroutine exits when ctx is cancelled.
   383→func MemoryUsageTracker(wg *sync.WaitGroup, ctx context.Context) {
   384→	ms := GetMemoryStore()
   385→
   386→	wg.Go(func() {
   387→		normalInterval := DefaultMemoryUsageTrackerInterval
   388→		fastInterval := 30 * time.Second
   389→
   390→		if normalInterval <= 0 {
   391→			return
   392→		}
   393→
   394→		ticker := time.NewTicker(normalInterval)
   395→		defer ticker.Stop()
   396→		currentInterval := normalInterval
   397→
   398→		for {
   399→			select {
   400→			case <-ctx.Done():
   401→				return
   402→			case <-ticker.C:
   403→				var mem runtime.MemStats
   404→				runtime.ReadMemStats(&mem)
   405→				actualMemoryGB := float64(mem.Alloc) / 1e9
   406→				metricDataGB := ms.SizeInGB()
   407→				cclog.Infof("[METRICSTORE]> memory usage: %.2f GB actual (%.2f GB metric data)", actualMemoryGB, metricDataGB)
   408→
   409→				freedExcluded := 0
   410→				freedEmergency := 0
   411→				var err error
   412→
   413→				state.mu.RLock()
   414→				lastRetention := state.lastRetentionTime
   415→				selectorsExcluded := state.selectorsExcluded
   416→				state.mu.RUnlock()
   417→
   418→				if lastRetention != 0 && selectorsExcluded {
   419→					freedExcluded, err = ms.Free(nil, lastRetention)
   420→					if err != nil {
   421→						cclog.Errorf("[METRICSTORE]> error while force-freeing the excluded buffers: %s", err)
   422→					}
   423→
   424→					if freedExcluded > 0 {
   425→						debug.FreeOSMemory()
   426→						cclog.Infof("[METRICSTORE]> done: %d excluded buffers force-freed", freedExcluded)
   427→					}
   428→				}
   429→
   430→				runtime.ReadMemStats(&mem)
   431→				actualMemoryGB = float64(mem.Alloc) / 1e9
   432→
   433→				if actualMemoryGB > float64(Keys.MemoryCap) {
   434→					cclog.Warnf("[METRICSTORE]> memory usage %.2f GB exceeds cap %d GB, starting emergency buffer freeing", actualMemoryGB, Keys.MemoryCap)
   435→
   436→					// Use progressive time-based Free with increasing threshold
   437→					// instead of ForceFree loop — fewer tree traversals, more effective
   438→					d, parseErr := time.ParseDuration(Keys.RetentionInMemory)
   439→					if parseErr != nil {
   440→						cclog.Errorf("[METRICSTORE]> cannot parse retention duration: %s", parseErr)
   441→					} else {
   442→						thresholds := []float64{0.75, 0.5, 0.25}
   443→						for _, fraction := range thresholds {
   444→							threshold := time.Now().Add(-time.Duration(float64(d) * fraction))
   445→							freed, freeErr := ms.Free(nil, threshold.Unix())
   446→							if freeErr != nil {
   447→								cclog.Errorf("[METRICSTORE]> error while freeing buffers at %.0f%% retention: %s", fraction*100, freeErr)
   448→							}
   449→							freedEmergency += freed
   450→
   451→							bufferPool.Clear()
   452→							runtime.GC()
   453→							runtime.ReadMemStats(&mem)
   454→							actualMemoryGB = float64(mem.Alloc) / 1e9
   455→
   456→							if actualMemoryGB < float64(Keys.MemoryCap) {
   457→								break
   458→							}
   459→						}
   460→					}
   461→
   462→					bufferPool.Clear()
   463→					debug.FreeOSMemory()
   464→
   465→					runtime.ReadMemStats(&mem)
   466→					actualMemoryGB = float64(mem.Alloc) / 1e9
   467→
   468→					if actualMemoryGB >= float64(Keys.MemoryCap) {
   469→						cclog.Errorf("[METRICSTORE]> after emergency frees (%d buffers), memory usage %.2f GB still at/above cap %d GB", freedEmergency, actualMemoryGB, Keys.MemoryCap)
   470→					} else {
   471→						cclog.Infof("[METRICSTORE]> emergency freeing complete: %d buffers freed, memory now %.2f GB", freedEmergency, actualMemoryGB)
   472→					}
   473→				}
   474→
   475→				// Adaptive ticker: check more frequently when memory is high
   476→				memoryRatio := actualMemoryGB / float64(Keys.MemoryCap)
   477→				if memoryRatio > 0.8 && currentInterval != fastInterval {
   478→					ticker.Reset(fastInterval)
   479→					currentInterval = fastInterval
   480→					cclog.Infof("[METRICSTORE]> memory at %.0f%% of cap, switching to fast check interval (30s)", memoryRatio*100)
   481→				} else if memoryRatio <= 0.8 && currentInterval != normalInterval {
   482→					ticker.Reset(normalInterval)
   483→					currentInterval = normalInterval
   484→					cclog.Infof("[METRICSTORE]> memory at %.0f%% of cap, switching to normal check interval", memoryRatio*100)
   485→				}
   486→			}
   487→		}
   488→	})
   489→}
   490→
   491→// Free removes metric data older than the given time while preserving data for active nodes.
   492→//
   493→// This function implements intelligent retention by consulting the NodeProvider (if configured)
   494→// to determine which nodes are currently in use by running jobs. Data for these nodes is
   495→// preserved even if older than the retention time.
   496→//
   497→// Parameters:
   498→//   - ms: The MemoryStore instance
   499→//   - t: Time threshold - buffers with data older than this will be freed
   500→//
   501→// Returns:
   502→//   - Number of buffers freed
   503→//   - Error if NodeProvider query fails
   504→//
   505→// Behavior:
   506→//   - If no NodeProvider is set: frees all buffers older than t
   507→//   - If NodeProvider returns empty map: frees all buffers older than t
   508→//   - Otherwise: preserves buffers for nodes returned by GetUsedNodes(), frees others
   509→func Free(ms *MemoryStore, t time.Time) (int, error) {
   510→	// If no NodeProvider is configured, free all buffers older than t
   511→	if ms.nodeProvider == nil {
   512→		return ms.Free(nil, t.Unix())
   513→	}
   514→
   515→	excludeSelectors, err := ms.nodeProvider.GetUsedNodes(t.Unix())
   516→	if err != nil {
   517→		return 0, err
   518→	}
   519→
   520→	switch lenMap := len(excludeSelectors); lenMap {
   521→
   522→	// If the length of the map returned by GetUsedNodes() is 0,
   523→	// then use default Free method with nil selector
   524→	case 0:
   525→		state.selectorsExcluded = false
   526→		return ms.Free(nil, t.Unix())
   527→
   528→	// Else formulate selectors, exclude those from the map
   529→	// and free the rest of the selectors
   530→	default:
   531→		state.selectorsExcluded = true
   532→		selectors := GetSelectors(ms, excludeSelectors)
   533→		return FreeSelected(ms, selectors, t)
   534→	}
   535→}
   536→
   537→// FreeSelected frees buffers for specific selectors while preserving others.
   538→//
   539→// This function is used when we want to retain some specific nodes beyond the retention time.
   540→// It iterates through the provided selectors and frees their associated buffers.
   541→//
   542→// Parameters:
   543→//   - ms: The MemoryStore instance
   544→//   - selectors: List of selector paths to free (e.g., [["cluster1", "node1"], ["cluster2", "node2"]])
   545→//   - t: Time threshold for freeing buffers
   546→//
   547→// Returns the total number of buffers freed and any error encountered.
   548→func FreeSelected(ms *MemoryStore, selectors [][]string, t time.Time) (int, error) {
   549→	freed := 0
   550→
   551→	for _, selector := range selectors {
   552→
   553→		freedBuffers, err := ms.Free(selector, t.Unix())
   554→		if err != nil {
   555→			cclog.Errorf("error while freeing selected buffers: %#v", err)
   556→		}
   557→		freed += freedBuffers
   558→
   559→	}
   560→
   561→	return freed, nil
   562→}
   563→
   564→// GetSelectors returns all selectors at depth 2 (cluster/node level) that are NOT in the exclusion map.
   565→//
   566→// This function generates a list of selectors whose buffers should be freed by excluding
   567→// selectors that correspond to nodes currently in use by running jobs.
   568→//
   569→// Parameters:
   570→//   - ms: The MemoryStore instance
   571→//   - excludeSelectors: Map of cluster names to node hostnames that should NOT be freed
   572→//
   573→// Returns a list of selectors ([]string paths) that can be safely freed.
   574→//
   575→// Example:
   576→//
   577→//	If the tree has paths ["emmy", "node001"] and ["emmy", "node002"],
   578→//	and excludeSelectors contains {"emmy": ["node001"]},
   579→//	then only [["emmy", "node002"]] is returned.
   580→func GetSelectors(ms *MemoryStore, excludeSelectors map[string][]string) [][]string {
   581→	allSelectors := ms.GetPaths(2)
   582→
   583→	filteredSelectors := make([][]string, 0, len(allSelectors))
   584→
   585→	for _, path := range allSelectors {
   586→		if len(path) < 2 {
   587→			continue
   588→		}
   589→
   590→		key := path[0]   // The "Key" (Level 1)
   591→		value := path[1] // The "Value" (Level 2)
   592→
   593→		exclude := false
   594→
   595→		// Check if the key exists in our exclusion map
   596→		if excludedValues, exists := excludeSelectors[key]; exists {
   597→			// The key exists, now check if the specific value is in the exclusion list
   598→			if slices.Contains(excludedValues, value) {
   599→				exclude = true
   600→			}
   601→		}
   602→
   603→		if !exclude {
   604→			filteredSelectors = append(filteredSelectors, path)
   605→		}
   606→	}
   607→
   608→	return filteredSelectors
   609→}
   610→
   611→// GetPaths returns a list of lists (paths) to the specified depth.
   612→func (ms *MemoryStore) GetPaths(targetDepth int) [][]string {
   613→	var results [][]string
   614→
   615→	// Start recursion. Initial path is empty.
   616→	// We treat Root as depth 0.
   617→	ms.root.collectPaths(0, targetDepth, []string{}, &results)
   618→
   619→	return results
   620→}
   621→
   622→// Write all values in `metrics` to the level specified by `selector` for time `ts`.
   623→// Look at `findLevelOrCreate` for how selectors work.
   624→func (m *MemoryStore) Write(selector []string, ts int64, metrics []Metric) error {
   625→	var ok bool
   626→	for i, metric := range metrics {
   627→		if metric.MetricConfig.Frequency == 0 {
   628→			metric.MetricConfig, ok = m.Metrics[metric.Name]
   629→			if !ok {
   630→				cclog.Debugf("[METRICSTORE]> Unknown metric '%s' in Write() - skipping", metric.Name)
   631→				metric.MetricConfig.Frequency = 0
   632→			}
   633→			metrics[i] = metric
   634→		}
   635→	}
   636→
   637→	return m.WriteToLevel(&m.root, selector, ts, metrics)
   638→}
   639→
   640→func (m *MemoryStore) GetLevel(selector []string) *Level {
   641→	return m.root.findLevelOrCreate(selector, len(m.Metrics))
   642→}
   643→
   644→// WriteToLevel assumes that `minfo` in `metrics` is filled in
   645→func (m *MemoryStore) WriteToLevel(l *Level, selector []string, ts int64, metrics []Metric) error {
   646→	l = l.findLevelOrCreate(selector, len(m.Metrics))
   647→	l.lock.Lock()
   648→	defer l.lock.Unlock()
   649→
   650→	for _, metric := range metrics {
   651→		if metric.MetricConfig.Frequency == 0 {
   652→			continue
   653→		}
   654→
   655→		b := l.metrics[metric.MetricConfig.offset]
   656→		if b == nil {
   657→			// First write to this metric and level
   658→			b = newBuffer(ts, metric.MetricConfig.Frequency)
   659→			l.metrics[metric.MetricConfig.offset] = b
   660→		}
   661→
   662→		nb, err := b.write(ts, metric.Value)
   663→		if err != nil {
   664→			return err
   665→		}
   666→
   667→		// Last write created a new buffer...
   668→		if b != nb {
   669→			l.metrics[metric.MetricConfig.offset] = nb
   670→		}
   671→	}
   672→	return nil
   673→}
   674→
   675→// Read returns all values for metric `metric` from `from` to `to` for the selected level(s).
   676→// If the level does not hold the metric itself, the data will be aggregated recursively from the children.
   677→// The second and third return value are the actual from/to for the data. Those can be different from
   678→// the range asked for if no data was available.
   679→func (m *MemoryStore) Read(selector util.Selector, metric string, from, to, resolution int64) ([]schema.Float, int64, int64, int64, error) {
   680→	if from > to {
   681→		return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid time range")
   682→	}
   683→
   684→	minfo, ok := m.Metrics[metric]
   685→	if !ok {
   686→		return nil, 0, 0, 0, errors.New("[METRICSTORE]> unknown metric: " + metric)
   687→	}
   688→
   689→	n, data := 0, make([]schema.Float, (to-from)/minfo.Frequency+1)
   690→
   691→	err := m.root.findBuffers(selector, minfo.offset, func(b *buffer) error {
   692→		cdata, cfrom, cto, err := b.read(from, to, data)
   693→		if err != nil {
   694→			return err
   695→		}
   696→
   697→		if n == 0 {
   698→			from, to = cfrom, cto
   699→		} else if from != cfrom || to != cto || len(data) != len(cdata) {
   700→			missingfront, missingback := int((from-cfrom)/minfo.Frequency), int((to-cto)/minfo.Frequency)
   701→			if missingfront != 0 {
   702→				return ErrDataDoesNotAlign
   703→			}
   704→
   705→			newlen := len(cdata) - missingback
   706→			if newlen < 1 {
   707→				return ErrDataDoesNotAlign
   708→			}
   709→			cdata = cdata[0:newlen]
   710→			if len(cdata) != len(data) {
   711→				return ErrDataDoesNotAlign
   712→			}
   713→
   714→			from, to = cfrom, cto
   715→		}
   716→
   717→		data = cdata
   718→		n += 1
   719→		return nil
   720→	})
   721→
   722→	if err != nil {
   723→		return nil, 0, 0, 0, err
   724→	} else if n == 0 {
   725→		return nil, 0, 0, 0, ErrNoHostOrMetric
   726→	} else if n > 1 {
   727→		if minfo.Aggregation == AvgAggregation {
   728→			normalize := 1. / schema.Float(n)
   729→			for i := 0; i < len(data); i++ {
   730→				data[i] *= normalize
   731→			}
   732→		} else if minfo.Aggregation != SumAggregation {
   733→			return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid aggregation")
   734→		}
   735→	}
   736→
   737→	data, resolution, err = resampler.LargestTriangleThreeBucket(data, minfo.Frequency, resolution)
   738→	if err != nil {
   739→		return nil, 0, 0, 0, err
   740→	}
   741→
   742→	return data, from, to, resolution, nil
   743→}
   744→
   745→// Free releases all buffers for the selected level and all its children that
   746→// contain only values older than `t`.
   747→func (m *MemoryStore) Free(selector []string, t int64) (int, error) {
   748→	return m.GetLevel(selector).free(t)
   749→}
   750→
   751→// ForceFree unconditionally removes the oldest buffer from each metric chain.
   752→func (m *MemoryStore) ForceFree() (int, error) {
   753→	return m.GetLevel(nil).forceFree()
   754→}
   755→
   756→func (m *MemoryStore) FreeAll() error {
   757→	m.root.lock.Lock()
   758→	defer m.root.lock.Unlock()
   759→
   760→	for k := range m.root.children {
   761→		delete(m.root.children, k)
   762→	}
   763→
   764→	return nil
   765→}
   766→
   767→func (m *MemoryStore) SizeInBytes() int64 {
   768→	return m.root.sizeInBytes()
   769→}
   770→
   771→func (m *MemoryStore) SizeInGB() float64 {
   772→	return float64(m.root.sizeInBytes()) / 1e9
   773→}
   774→
   775→// ListChildren , given a selector, returns a list of all children of the level
   776→// selected.
   777→func (m *MemoryStore) ListChildren(selector []string) []string {
   778→	lvl := &m.root
   779→	for lvl != nil && len(selector) != 0 {
   780→		lvl.lock.RLock()
   781→		next := lvl.children[selector[0]]
   782→		lvl.lock.RUnlock()
   783→		lvl = next
   784→		selector = selector[1:]
   785→	}
   786→
   787→	if lvl == nil {
   788→		return nil
   789→	}
   790→
   791→	lvl.lock.RLock()
   792→	defer lvl.lock.RUnlock()
   793→
   794→	children := make([]string, 0, len(lvl.children))
   795→	for child := range lvl.children {
   796→		children = append(children, child)
   797→	}
   798→
   799→	return children
   800→}
   801→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package metricstore provides an efficient in-memory time-series metric storage system
// with support for hierarchical data organization, checkpointing, and archiving.
//
// The package organizes metrics in a tree structure (cluster → host → component) and
// provides concurrent read/write access to metric data with configurable aggregation strategies.
// Background goroutines handle periodic checkpointing (JSON or Avro format), archiving old data,
// and enforcing retention policies.
//
// Key features:
//   - In-memory metric storage with configurable retention
//   - Hierarchical data organization (selectors)
//   - Concurrent checkpoint/archive workers
//   - Support for sum and average aggregation
//   - NATS integration for metric ingestion
package metricstore

import (
	"bytes"
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"runtime"
	"runtime/debug"
	"slices"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/config"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/resampler"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	"github.com/ClusterCockpit/cc-lib/v2/util"
)

// GlobalState holds the global state for the metric store with thread-safe access.
type GlobalState struct {
	mu                sync.RWMutex
	lastRetentionTime int64
	selectorsExcluded bool
}

var (
	singleton  sync.Once
	msInstance *MemoryStore
	// shutdownFunc stores the context cancellation function created in Init
	// and is called during Shutdown to cancel all background goroutines
	shutdownFunc   context.CancelFunc
	shutdownFuncMu sync.Mutex // Protects shutdownFunc from concurrent access
	// Create a global instance
	state = &GlobalState{}
)

// NodeProvider provides information about nodes currently in use by running jobs.
//
// This interface allows metricstore to query job information without directly
// depending on the repository package, breaking the import cycle.
//
// Implementations should return nodes that are actively processing jobs started
// before the given timestamp. These nodes will be excluded from retention-based
// garbage collection to prevent data loss for jobs that are still running or
// recently completed.
type NodeProvider interface {
	// GetUsedNodes returns a map of cluster names to sorted lists of unique hostnames
	// that are currently in use by jobs that started before the given timestamp.
	//
	// Parameters:
	//   - ts: Unix timestamp threshold - returns nodes with jobs started before this time
	//
	// Returns:
	//   - Map of cluster names to lists of node hostnames that should be excluded from garbage collection
	//   - Error if the query fails
	GetUsedNodes(ts int64) (map[string][]string, error)
}

// Metric represents a single metric data point to be written to the store.
type Metric struct {
	Name  string
	Value schema.Float
	// MetricConfig contains frequency and aggregation settings for this metric.
	// If Frequency is 0, configuration will be looked up from MemoryStore.Metrics during Write().
	MetricConfig MetricConfig
}

// MemoryStore is the main in-memory time-series metric storage implementation.
//
// It organizes metrics in a hierarchical tree structure where each level represents
// a component of the system hierarchy (e.g., cluster → host → CPU). Each level can
// store multiple metrics as time-series buffers.
//
// The store is initialized as a singleton via InitMetrics() and accessed via GetMemoryStore().
// All public methods are safe for concurrent use.
type MemoryStore struct {
	Metrics      map[string]MetricConfig
	root         Level
	nodeProvider NodeProvider
}

// Init initializes the metric store from configuration and starts background workers.
//
// This function must be called exactly once before any other metricstore operations.
// It performs the following initialization steps:
//  1. Validates and decodes the metric store configuration
//  2. Configures worker pool size (defaults to NumCPU/2+1, max 10)
//  3. Loads metric configurations from all registered clusters
//  4. Restores checkpoints within the retention window
//  5. Starts background workers for retention, checkpointing, archiving, and monitoring
//  6. Optionally subscribes to NATS for real-time metric ingestion
//
// Parameters:
//   - rawConfig: JSON configuration for the metric store (see MetricStoreConfig)
//   - wg: WaitGroup that will be incremented for each background goroutine started
//
// The function will call cclog.Fatal on critical errors during initialization.
// Use Shutdown() to cleanly stop all background workers started by Init().
//
// Note: Signal handling must be implemented by the caller. Call Shutdown() when
// receiving termination signals to ensure checkpoint data is persisted.
func Init(rawConfig json.RawMessage, metrics map[string]MetricConfig, wg *sync.WaitGroup) {
	startupTime := time.Now()

	if rawConfig != nil {
		config.Validate(configSchema, rawConfig)
		dec := json.NewDecoder(bytes.NewReader(rawConfig))
		dec.DisallowUnknownFields()
		if err := dec.Decode(&Keys); err != nil {
			cclog.Abortf("[METRICSTORE]> Metric Store Config Init: Could not decode config file '%s'.\nError: %s\n", rawConfig, err.Error())
		}
	}

	// Set NumWorkers from config or use default
	if Keys.NumWorkers <= 0 {
		Keys.NumWorkers = min(runtime.NumCPU()/2+1, DefaultMaxWorkers)
	}
	cclog.Debugf("[METRICSTORE]> Using %d workers for checkpoint/archive operations\n", Keys.NumWorkers)

	// Pass the config.MetricStoreKeys
	InitMetrics(metrics)

	ms := GetMemoryStore()

	d, err := time.ParseDuration(Keys.RetentionInMemory)
	if err != nil {
		cclog.Fatal(err)
	}

	restoreFrom := startupTime.Add(-d)
	cclog.Infof("[METRICSTORE]> Loading checkpoints newer than %s\n", restoreFrom.Format(time.RFC3339))
	files, err := ms.FromCheckpointFiles(Keys.Checkpoints.RootDir, restoreFrom.Unix())
	loadedData := ms.SizeInBytes() / 1024 / 1024 // In MB
	if err != nil {
		cclog.Fatalf("[METRICSTORE]> Loading checkpoints failed: %s\n", err.Error())
	} else {
		cclog.Infof("[METRICSTORE]> Checkpoints loaded (%d files, %d MB, that took %fs)\n", files, loadedData, time.Since(startupTime).Seconds())
	}

	// Try to use less memory by forcing a GC run here and then
	// lowering the target percentage. The default of 100 means
	// that only once the ratio of new allocations execeds the
	// previously active heap, a GC is triggered.
	// Forcing a GC here will set the "previously active heap"
	// to a minumum.
	// runtime.GC()

	ctx, shutdown := context.WithCancel(context.Background())

	Retention(wg, ctx)
	Checkpointing(wg, ctx)
	CleanUp(wg, ctx)
	WALStaging(wg, ctx)
	MemoryUsageTracker(wg, ctx)

	// Note: Signal handling has been removed from this function.
	// The caller is responsible for handling shutdown signals and calling
	// the shutdown() function when appropriate.
	// Store the shutdown function for later use by Shutdown()
	shutdownFuncMu.Lock()
	shutdownFunc = shutdown
	shutdownFuncMu.Unlock()

	if Keys.Subscriptions != nil {
		err = ReceiveNats(ms, 1, ctx)
		if err != nil {
			cclog.Fatal(err)
		}
	}
}

// InitMetrics initializes the singleton MemoryStore instance with the given metric configurations.
//
// This function must be called before GetMemoryStore() and can only be called once due to
// the singleton pattern. It assigns each metric an internal offset for efficient buffer indexing.
//
// Parameters:
//   - metrics: Map of metric names to their configurations (frequency and aggregation strategy)
//
// Panics if any metric has Frequency == 0, which indicates an invalid configuration.
//
// After this call, the global msInstance is ready for use via GetMemoryStore().
func InitMetrics(metrics map[string]MetricConfig) {
	singleton.Do(func() {
		offset := 0
		for key, cfg := range metrics {
			if cfg.Frequency == 0 {
				panic("[METRICSTORE]> invalid frequency")
			}

			metrics[key] = MetricConfig{
				Frequency:   cfg.Frequency,
				Aggregation: cfg.Aggregation,
				offset:      offset,
			}
			offset += 1
		}

		msInstance = &MemoryStore{
			root: Level{
				metrics:  make([]*buffer, len(metrics)),
				children: make(map[string]*Level),
			},
			Metrics: metrics,
		}
	})
}

// GetMemoryStore returns the singleton MemoryStore instance.
//
// Returns the initialized MemoryStore singleton. Calls cclog.Fatal if InitMetrics() was not called first.
//
// This function is safe for concurrent use after initialization.
func GetMemoryStore() *MemoryStore {
	if msInstance == nil {
		cclog.Warnf("[METRICSTORE]> MemoryStore not initialized!")
	}

	return msInstance
}

func (ms *MemoryStore) GetMetricFrequency(metricName string) (int64, error) {
	if metric, ok := ms.Metrics[metricName]; ok {
		return metric.Frequency, nil
	}
	return 0, fmt.Errorf("[METRICSTORE]> metric %s not found", metricName)
}

// SetNodeProvider sets the NodeProvider implementation for the MemoryStore.
// This must be called during initialization to provide job state information
// for selective buffer retention during Free operations.
// If not set, the Free function will fall back to freeing all buffers.
func (ms *MemoryStore) SetNodeProvider(provider NodeProvider) {
	ms.nodeProvider = provider
}

// Shutdown performs a graceful shutdown of the metric store.
//
// This function cancels all background goroutines started by Init() and writes
// a final checkpoint to disk before returning. It should be called when the
// application receives a termination signal.
//
// The function will:
//  1. Cancel the context to stop all background workers
//  2. Close the WAL messages channel if using WAL format
//  3. Write a final checkpoint to preserve in-memory data
//  4. Log any errors encountered during shutdown
//
// Note: This function blocks until the final checkpoint is written.
func Shutdown() {
	shutdownFuncMu.Lock()
	defer shutdownFuncMu.Unlock()
	if shutdownFunc != nil {
		shutdownFunc()
	}

	if Keys.Checkpoints.FileFormat == "wal" {
		close(WALMessages)
	}

	cclog.Infof("[METRICSTORE]> Writing to '%s'...\n", Keys.Checkpoints.RootDir)
	var files int
	var err error

	ms := GetMemoryStore()

	lastCheckpointMu.Lock()
	from := lastCheckpoint
	lastCheckpointMu.Unlock()

	if Keys.Checkpoints.FileFormat == "wal" {
		var hostDirs []string
		files, hostDirs, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
		if err == nil {
			RotateWALFilesAfterShutdown(hostDirs)
		}
	} else {
		files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
	}

	if err != nil {
		cclog.Errorf("[METRICSTORE]> Writing checkpoint failed: %s\n", err.Error())
	}
	cclog.Infof("[METRICSTORE]> Done! (%d files written)\n", files)
}

// Retention starts a background goroutine that periodically frees old metric data.
//
// This worker runs at half the retention interval and calls Free() to remove buffers
// older than the configured retention time. It respects the NodeProvider to preserve
// data for nodes with active jobs.
//
// Parameters:
//   - wg: WaitGroup to signal completion when context is cancelled
//   - ctx: Context for cancellation signal
//
// The goroutine exits when ctx is cancelled.
func Retention(wg *sync.WaitGroup, ctx context.Context) {
	ms := GetMemoryStore()

	wg.Go(func() {
		d, err := time.ParseDuration(Keys.RetentionInMemory)
		if err != nil {
			cclog.Fatal(err)
		}
		if d <= 0 {
			return
		}

		tickInterval := d / 2
		if tickInterval <= 0 {
			return
		}
		ticker := time.NewTicker(tickInterval)
		defer ticker.Stop()

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				state.mu.Lock()

				t := time.Now().Add(-d)

				state.lastRetentionTime = t.Unix()

				cclog.Infof("[METRICSTORE]> start freeing buffers (older than %s)...\n", t.Format(time.RFC3339))

				freed, err := Free(ms, t)
				if err != nil {
					cclog.Errorf("[METRICSTORE]> freeing up buffers failed: %s\n", err.Error())
				} else {
					cclog.Infof("[METRICSTORE]> done: %d buffers freed\n", freed)
				}

				state.mu.Unlock()

				// Clean up the buffer pool
				bufferPool.Clean(state.lastRetentionTime)
			}
		}
	})
}

// MemoryUsageTracker starts a background goroutine that monitors memory usage.
//
// This worker checks actual process memory usage (via runtime.MemStats) periodically
// and force-frees buffers if memory exceeds the configured cap. It uses FreeOSMemory()
// to return memory to the OS after freeing buffers, avoiding aggressive GC that causes
// performance issues.
//
// The tracker logs both actual memory usage (heap allocated) and metric data size for
// visibility into memory overhead from Go runtime structures and allocations.
//
// Parameters:
//   - wg: WaitGroup to signal completion when context is cancelled
//   - ctx: Context for cancellation signal
//
// The goroutine exits when ctx is cancelled.
func MemoryUsageTracker(wg *sync.WaitGroup, ctx context.Context) {
	ms := GetMemoryStore()

	wg.Go(func() {
		normalInterval := DefaultMemoryUsageTrackerInterval
		fastInterval := 30 * time.Second

		if normalInterval <= 0 {
			return
		}

		ticker := time.NewTicker(normalInterval)
		defer ticker.Stop()
		currentInterval := normalInterval

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				var mem runtime.MemStats
				runtime.ReadMemStats(&mem)
				actualMemoryGB := float64(mem.Alloc) / 1e9
				metricDataGB := ms.SizeInGB()
				cclog.Infof("[METRICSTORE]> memory usage: %.2f GB actual (%.2f GB metric data)", actualMemoryGB, metricDataGB)

				freedExcluded := 0
				freedEmergency := 0
				var err error

				state.mu.RLock()
				lastRetention := state.lastRetentionTime
				selectorsExcluded := state.selectorsExcluded
				state.mu.RUnlock()

				if lastRetention != 0 && selectorsExcluded {
					freedExcluded, err = ms.Free(nil, lastRetention)
					if err != nil {
						cclog.Errorf("[METRICSTORE]> error while force-freeing the excluded buffers: %s", err)
					}

					if freedExcluded > 0 {
						debug.FreeOSMemory()
						cclog.Infof("[METRICSTORE]> done: %d excluded buffers force-freed", freedExcluded)
					}
				}

				runtime.ReadMemStats(&mem)
				actualMemoryGB = float64(mem.Alloc) / 1e9

				if actualMemoryGB > float64(Keys.MemoryCap) {
					cclog.Warnf("[METRICSTORE]> memory usage %.2f GB exceeds cap %d GB, starting emergency buffer freeing", actualMemoryGB, Keys.MemoryCap)

					// Use progressive time-based Free with increasing threshold
					// instead of ForceFree loop — fewer tree traversals, more effective
					d, parseErr := time.ParseDuration(Keys.RetentionInMemory)
					if parseErr != nil {
						cclog.Errorf("[METRICSTORE]> cannot parse retention duration: %s", parseErr)
					} else {
						thresholds := []float64{0.75, 0.5, 0.25}
						for _, fraction := range thresholds {
							threshold := time.Now().Add(-time.Duration(float64(d) * fraction))
							freed, freeErr := ms.Free(nil, threshold.Unix())
							if freeErr != nil {
								cclog.Errorf("[METRICSTORE]> error while freeing buffers at %.0f%% retention: %s", fraction*100, freeErr)
							}
							freedEmergency += freed

							bufferPool.Clear()
							runtime.GC()
							runtime.ReadMemStats(&mem)
							actualMemoryGB = float64(mem.Alloc) / 1e9

							if actualMemoryGB < float64(Keys.MemoryCap) {
								break
							}
						}
					}

					bufferPool.Clear()
					debug.FreeOSMemory()

					runtime.ReadMemStats(&mem)
					actualMemoryGB = float64(mem.Alloc) / 1e9

					if actualMemoryGB >= float64(Keys.MemoryCap) {
						cclog.Errorf("[METRICSTORE]> after emergency frees (%d buffers), memory usage %.2f GB still at/above cap %d GB", freedEmergency, actualMemoryGB, Keys.MemoryCap)
					} else {
						cclog.Infof("[METRICSTORE]> emergency freeing complete: %d buffers freed, memory now %.2f GB", freedEmergency, actualMemoryGB)
					}
				}

				// Adaptive ticker: check more frequently when memory is high
				memoryRatio := actualMemoryGB / float64(Keys.MemoryCap)
				if memoryRatio > 0.8 && currentInterval != fastInterval {
					ticker.Reset(fastInterval)
					currentInterval = fastInterval
					cclog.Infof("[METRICSTORE]> memory at %.0f%% of cap, switching to fast check interval (30s)", memoryRatio*100)
				} else if memoryRatio <= 0.8 && currentInterval != normalInterval {
					ticker.Reset(normalInterval)
					currentInterval = normalInterval
					cclog.Infof("[METRICSTORE]> memory at %.0f%% of cap, switching to normal check interval", memoryRatio*100)
				}
			}
		}
	})
}

// Free removes metric data older than the given time while preserving data for active nodes.
//
// This function implements intelligent retention by consulting the NodeProvider (if configured)
// to determine which nodes are currently in use by running jobs. Data for these nodes is
// preserved even if older than the retention time.
//
// Parameters:
//   - ms: The MemoryStore instance
//   - t: Time threshold - buffers with data older than this will be freed
//
// Returns:
//   - Number of buffers freed
//   - Error if NodeProvider query fails
//
// Behavior:
//   - If no NodeProvider is set: frees all buffers older than t
//   - If NodeProvider returns empty map: frees all buffers older than t
//   - Otherwise: preserves buffers for nodes returned by GetUsedNodes(), frees others
func Free(ms *MemoryStore, t time.Time) (int, error) {
	// If no NodeProvider is configured, free all buffers older than t
	if ms.nodeProvider == nil {
		return ms.Free(nil, t.Unix())
	}

	excludeSelectors, err := ms.nodeProvider.GetUsedNodes(t.Unix())
	if err != nil {
		return 0, err
	}

	switch lenMap := len(excludeSelectors); lenMap {

	// If the length of the map returned by GetUsedNodes() is 0,
	// then use default Free method with nil selector
	case 0:
		state.selectorsExcluded = false
		return ms.Free(nil, t.Unix())

	// Else formulate selectors, exclude those from the map
	// and free the rest of the selectors
	default:
		state.selectorsExcluded = true
		selectors := GetSelectors(ms, excludeSelectors)
		return FreeSelected(ms, selectors, t)
	}
}

// FreeSelected frees buffers for specific selectors while preserving others.
//
// This function is used when we want to retain some specific nodes beyond the retention time.
// It iterates through the provided selectors and frees their associated buffers.
//
// Parameters:
//   - ms: The MemoryStore instance
//   - selectors: List of selector paths to free (e.g., [["cluster1", "node1"], ["cluster2", "node2"]])
//   - t: Time threshold for freeing buffers
//
// Returns the total number of buffers freed and any error encountered.
func FreeSelected(ms *MemoryStore, selectors [][]string, t time.Time) (int, error) {
	freed := 0

	for _, selector := range selectors {

		freedBuffers, err := ms.Free(selector, t.Unix())
		if err != nil {
			cclog.Errorf("error while freeing selected buffers: %#v", err)
		}
		freed += freedBuffers

	}

	return freed, nil
}

// GetSelectors returns all selectors at depth 2 (cluster/node level) that are NOT in the exclusion map.
//
// This function generates a list of selectors whose buffers should be freed by excluding
// selectors that correspond to nodes currently in use by running jobs.
//
// Parameters:
//   - ms: The MemoryStore instance
//   - excludeSelectors: Map of cluster names to node hostnames that should NOT be freed
//
// Returns a list of selectors ([]string paths) that can be safely freed.
//
// Example:
//
//	If the tree has paths ["emmy", "node001"] and ["emmy", "node002"],
//	and excludeSelectors contains {"emmy": ["node001"]},
//	then only [["emmy", "node002"]] is returned.
func GetSelectors(ms *MemoryStore, excludeSelectors map[string][]string) [][]string {
	allSelectors := ms.GetPaths(2)

	filteredSelectors := make([][]string, 0, len(allSelectors))

	for _, path := range allSelectors {
		if len(path) < 2 {
			continue
		}

		key := path[0]   // The "Key" (Level 1)
		value := path[1] // The "Value" (Level 2)

		exclude := false

		// Check if the key exists in our exclusion map
		if excludedValues, exists := excludeSelectors[key]; exists {
			// The key exists, now check if the specific value is in the exclusion list
			if slices.Contains(excludedValues, value) {
				exclude = true
			}
		}

		if !exclude {
			filteredSelectors = append(filteredSelectors, path)
		}
	}

	return filteredSelectors
}

// GetPaths returns a list of lists (paths) to the specified depth.
func (ms *MemoryStore) GetPaths(targetDepth int) [][]string {
	var results [][]string

	// Start recursion. Initial path is empty.
	// We treat Root as depth 0.
	ms.root.collectPaths(0, targetDepth, []string{}, &results)

	return results
}

// Write all values in `metrics` to the level specified by `selector` for time `ts`.
// Look at `findLevelOrCreate` for how selectors work.
func (m *MemoryStore) Write(selector []string, ts int64, metrics []Metric) error {
	var ok bool
	for i, metric := range metrics {
		if metric.MetricConfig.Frequency == 0 {
			metric.MetricConfig, ok = m.Metrics[metric.Name]
			if !ok {
				cclog.Debugf("[METRICSTORE]> Unknown metric '%s' in Write() - skipping", metric.Name)
				metric.MetricConfig.Frequency = 0
			}
			metrics[i] = metric
		}
	}

	return m.WriteToLevel(&m.root, selector, ts, metrics)
}

func (m *MemoryStore) GetLevel(selector []string) *Level {
	return m.root.findLevelOrCreate(selector, len(m.Metrics))
}

// WriteToLevel assumes that `minfo` in `metrics` is filled in
func (m *MemoryStore) WriteToLevel(l *Level, selector []string, ts int64, metrics []Metric) error {
	l = l.findLevelOrCreate(selector, len(m.Metrics))
	l.lock.Lock()
	defer l.lock.Unlock()

	for _, metric := range metrics {
		if metric.MetricConfig.Frequency == 0 {
			continue
		}

		b := l.metrics[metric.MetricConfig.offset]
		if b == nil {
			// First write to this metric and level
			b = newBuffer(ts, metric.MetricConfig.Frequency)
			l.metrics[metric.MetricConfig.offset] = b
		}

		nb, err := b.write(ts, metric.Value)
		if err != nil {
			return err
		}

		// Last write created a new buffer...
		if b != nb {
			l.metrics[metric.MetricConfig.offset] = nb
		}
	}
	return nil
}

// Read returns all values for metric `metric` from `from` to `to` for the selected level(s).
// If the level does not hold the metric itself, the data will be aggregated recursively from the children.
// The second and third return value are the actual from/to for the data. Those can be different from
// the range asked for if no data was available.
func (m *MemoryStore) Read(selector util.Selector, metric string, from, to, resolution int64) ([]schema.Float, int64, int64, int64, error) {
	if from > to {
		return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid time range")
	}

	minfo, ok := m.Metrics[metric]
	if !ok {
		return nil, 0, 0, 0, errors.New("[METRICSTORE]> unknown metric: " + metric)
	}

	n, data := 0, make([]schema.Float, (to-from)/minfo.Frequency+1)

	err := m.root.findBuffers(selector, minfo.offset, func(b *buffer) error {
		cdata, cfrom, cto, err := b.read(from, to, data)
		if err != nil {
			return err
		}

		if n == 0 {
			from, to = cfrom, cto
		} else if from != cfrom || to != cto || len(data) != len(cdata) {
			missingfront, missingback := int((from-cfrom)/minfo.Frequency), int((to-cto)/minfo.Frequency)
			if missingfront != 0 {
				return ErrDataDoesNotAlign
			}

			newlen := len(cdata) - missingback
			if newlen < 1 {
				return ErrDataDoesNotAlign
			}
			cdata = cdata[0:newlen]
			if len(cdata) != len(data) {
				return ErrDataDoesNotAlign
			}

			from, to = cfrom, cto
		}

		data = cdata
		n += 1
		return nil
	})

	if err != nil {
		return nil, 0, 0, 0, err
	} else if n == 0 {
		return nil, 0, 0, 0, ErrNoHostOrMetric
	} else if n > 1 {
		if minfo.Aggregation == AvgAggregation {
			normalize := 1. / schema.Float(n)
			for i := 0; i < len(data); i++ {
				data[i] *= normalize
			}
		} else if minfo.Aggregation != SumAggregation {
			return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid aggregation")
		}
	}

	data, resolution, err = resampler.LargestTriangleThreeBucket(data, minfo.Frequency, resolution)
	if err != nil {
		return nil, 0, 0, 0, err
	}

	return data, from, to, resolution, nil
}

// Free releases all buffers for the selected level and all its children that
// contain only values older than `t`.
func (m *MemoryStore) Free(selector []string, t int64) (int, error) {
	return m.GetLevel(selector).free(t)
}

// ForceFree unconditionally removes the oldest buffer from each metric chain.
func (m *MemoryStore) ForceFree() (int, error) {
	return m.GetLevel(nil).forceFree()
}

func (m *MemoryStore) FreeAll() error {
	m.root.lock.Lock()
	defer m.root.lock.Unlock()

	for k := range m.root.children {
		delete(m.root.children, k)
	}

	return nil
}

func (m *MemoryStore) SizeInBytes() int64 {
	return m.root.sizeInBytes()
}

func (m *MemoryStore) SizeInGB() float64 {
	return float64(m.root.sizeInBytes()) / 1e9
}

// ListChildren , given a selector, returns a list of all children of the level
// selected.
func (m *MemoryStore) ListChildren(selector []string) []string {
	lvl := &m.root
	for lvl != nil && len(selector) != 0 {
		lvl.lock.RLock()
		next := lvl.children[selector[0]]
		lvl.lock.RUnlock()
		lvl = next
		selector = selector[1:]
	}

	if lvl == nil {
		return nil
	}

	lvl.lock.RLock()
	defer lvl.lock.RUnlock()

	children := make([]string, 0, len(lvl.children))
	for child := range lvl.children {
		children = append(children, child)
	}

	return children
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// This file implements ingestion of InfluxDB line-protocol metric data received
     7→// over NATS. Each line encodes one metric sample with the following structure:
     8→//
     9→//	<measurement>[,cluster=<c>][,hostname=<h>][,type=<t>][,type-id=<id>][,subtype=<s>][,stype-id=<id>] value=<v> [<timestamp>]
    10→//
    11→// The measurement name identifies the metric (e.g. "cpu_load"). Tags provide
    12→// routing information (cluster, host) and optional sub-device selectors (type,
    13→// subtype). Only one field is expected per line: "value".
    14→//
    15→// After decoding, each sample is:
    16→//  1. Written to the in-memory store via ms.WriteToLevel.
    17→//  2. If the checkpoint format is "wal", also forwarded to the WAL staging
    18→//     goroutine via the WALMessages channel for durable write-ahead logging.
    19→package metricstore
    20→
    21→import (
    22→	"bytes"
    23→	"context"
    24→	"fmt"
    25→	"sync"
    26→	"time"
    27→
    28→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    29→	"github.com/ClusterCockpit/cc-lib/v2/nats"
    30→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    31→	"github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
    32→)
    33→
    34→// ReceiveNats subscribes to all configured NATS subjects and feeds incoming
    35→// line-protocol messages into the MemoryStore.
    36→//
    37→// When workers > 1 a pool of goroutines drains a shared channel so that
    38→// multiple messages can be decoded in parallel. With workers == 1 the NATS
    39→// callback decodes inline (no channel overhead, lower latency).
    40→//
    41→// The function blocks until ctx is cancelled and all worker goroutines have
    42→// finished. It returns nil when the NATS client is not configured; callers
    43→// should treat that as a no-op rather than an error.
    44→func ReceiveNats(ms *MemoryStore,
    45→	workers int,
    46→	ctx context.Context,
    47→) error {
    48→	nc := nats.GetClient()
    49→
    50→	if nc == nil {
    51→		cclog.Warn("NATS client not initialized")
    52→		return nil
    53→	}
    54→
    55→	var wg sync.WaitGroup
    56→	msgs := make(chan []byte, workers*2)
    57→
    58→	for _, sc := range *Keys.Subscriptions {
    59→		clusterTag := sc.ClusterTag
    60→		if workers > 1 {
    61→			wg.Add(workers)
    62→
    63→			for range workers {
    64→				go func() {
    65→					defer wg.Done()
    66→					for m := range msgs {
    67→						dec := lineprotocol.NewDecoderWithBytes(m)
    68→						if err := DecodeLine(dec, ms, clusterTag); err != nil {
    69→							cclog.Errorf("error: %s", err.Error())
    70→						}
    71→					}
    72→				}()
    73→			}
    74→
    75→			nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
    76→				select {
    77→				case msgs <- data:
    78→				case <-ctx.Done():
    79→				}
    80→			})
    81→		} else {
    82→			nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
    83→				dec := lineprotocol.NewDecoderWithBytes(data)
    84→				if err := DecodeLine(dec, ms, clusterTag); err != nil {
    85→					cclog.Errorf("error: %s", err.Error())
    86→				}
    87→			})
    88→		}
    89→		cclog.Infof("NATS subscription to '%s' established", sc.SubscribeTo)
    90→	}
    91→
    92→	go func() {
    93→		<-ctx.Done()
    94→		close(msgs)
    95→	}()
    96→
    97→	wg.Wait()
    98→
    99→	return nil
   100→}
   101→
   102→// reorder prepends prefix to buf in-place when buf has enough spare capacity,
   103→// avoiding an allocation. Falls back to a regular append otherwise.
   104→//
   105→// It is used to assemble the "type<type-id>" and "subtype<stype-id>" selector
   106→// strings when the type tag arrives before the type-id tag in the line, so the
   107→// two byte slices need to be concatenated in tag-declaration order regardless
   108→// of wire order.
   109→func reorder(buf, prefix []byte) []byte {
   110→	n := len(prefix)
   111→	m := len(buf)
   112→	if cap(buf) < m+n {
   113→		return append(prefix[:n:n], buf...)
   114→	} else {
   115→		buf = buf[:n+m]
   116→		for i := m - 1; i >= 0; i-- {
   117→			buf[i+n] = buf[i]
   118→		}
   119→		for i := range n {
   120→			buf[i] = prefix[i]
   121→		}
   122→		return buf
   123→	}
   124→}
   125→
   126→// decodeState holds the per-call scratch buffers used by DecodeLine.
   127→// Instances are recycled via decodeStatePool to avoid repeated allocations
   128→// during high-throughput ingestion.
   129→type decodeState struct {
   130→	// metricBuf holds a copy of the current measurement name (line-protocol
   131→	// measurement field). Copied because dec.Measurement() returns a slice
   132→	// that is invalidated by the next decoder call.
   133→	metricBuf []byte
   134→
   135→	// selector is the sub-device path passed to WriteToLevel and WALMessage
   136→	// (e.g. ["socket0"] or ["socket0", "memctrl1"]). Reused across lines.
   137→	selector []string
   138→
   139→	// typeBuf accumulates the concatenated "type"+"type-id" tag value for the
   140→	// current line. Reset at the start of each line's tag-decode loop.
   141→	typeBuf []byte
   142→
   143→	// subTypeBuf accumulates the concatenated "subtype"+"stype-id" tag value.
   144→	// Reset at the start of each line's tag-decode loop.
   145→	subTypeBuf []byte
   146→
   147→	// prevTypeBytes / prevTypeStr cache the last seen typeBuf content and its
   148→	// string conversion. Because consecutive lines in a batch typically address
   149→	// the same sub-device, the cache hit rate is very high and avoids
   150→	// repeated []byte→string allocations.
   151→	prevTypeBytes []byte
   152→	prevTypeStr   string
   153→
   154→	// prevSubTypeBytes / prevSubTypeStr are the same cache for the subtype.
   155→	prevSubTypeBytes []byte
   156→	prevSubTypeStr   string
   157→}
   158→
   159→// decodeStatePool recycles decodeState values across DecodeLine calls to
   160→// reduce GC pressure during sustained metric ingestion.
   161→var decodeStatePool = sync.Pool{
   162→	New: func() any {
   163→		return &decodeState{
   164→			metricBuf:  make([]byte, 0, 16),
   165→			selector:   make([]string, 0, 4),
   166→			typeBuf:    make([]byte, 0, 16),
   167→			subTypeBuf: make([]byte, 0, 16),
   168→		}
   169→	},
   170→}
   171→
   172→// DecodeLine reads all lines from dec (InfluxDB line-protocol) and writes each
   173→// decoded metric sample into ms.
   174→//
   175→// clusterDefault is used as the cluster name for lines that do not carry a
   176→// "cluster" tag. Callers typically supply the ClusterTag value from the NATS
   177→// subscription configuration.
   178→//
   179→// Performance notes:
   180→//   - A decodeState is obtained from decodeStatePool to reuse scratch buffers.
   181→//   - The Level pointer (host-level node in the metric tree) is cached across
   182→//     consecutive lines that share the same cluster+host pair to avoid
   183→//     repeated lock acquisitions on the root and cluster levels.
   184→//   - []byte→string conversions for type/subtype selectors are cached via
   185→//     prevType*/prevSubType* fields because batches typically repeat the same
   186→//     sub-device identifiers.
   187→//   - Timestamp parsing tries Second precision first; if that fails it retries
   188→//     Millisecond, Microsecond, and Nanosecond in turn. A missing timestamp
   189→//     falls back to time.Now().
   190→//
   191→// When the checkpoint format is "wal" each successfully decoded sample is also
   192→// sent to WALMessages so the WAL staging goroutine can persist it durably
   193→// before the next binary snapshot.
   194→func DecodeLine(dec *lineprotocol.Decoder,
   195→	ms *MemoryStore,
   196→	clusterDefault string,
   197→) error {
   198→	// Reduce allocations in loop:
   199→	t := time.Now()
   200→	metric := Metric{}
   201→	st := decodeStatePool.Get().(*decodeState)
   202→	defer decodeStatePool.Put(st)
   203→
   204→	// Optimize for the case where all lines in a "batch" are about the same
   205→	// cluster and host. By using `WriteToLevel` (level = host), we do not need
   206→	// to take the root- and cluster-level lock as often.
   207→	var lvl *Level = nil
   208→	prevCluster, prevHost := "", ""
   209→
   210→	var ok bool
   211→	for dec.Next() {
   212→		rawmeasurement, err := dec.Measurement()
   213→		if err != nil {
   214→			return err
   215→		}
   216→
   217→		// Needs to be copied because another call to dec.* would
   218→		// invalidate the returned slice.
   219→		st.metricBuf = append(st.metricBuf[:0], rawmeasurement...)
   220→
   221→		// The go compiler optimizes map[string(byteslice)] lookups:
   222→		metric.MetricConfig, ok = ms.Metrics[string(rawmeasurement)]
   223→		if !ok {
   224→			continue
   225→		}
   226→
   227→		st.typeBuf, st.subTypeBuf = st.typeBuf[:0], st.subTypeBuf[:0]
   228→		cluster, host := clusterDefault, ""
   229→		for {
   230→			key, val, err := dec.NextTag()
   231→			if err != nil {
   232→				return err
   233→			}
   234→			if key == nil {
   235→				break
   236→			}
   237→
   238→			// The go compiler optimizes string([]byte{...}) == "...":
   239→			switch string(key) {
   240→			case "cluster":
   241→				if string(val) == prevCluster {
   242→					cluster = prevCluster
   243→				} else {
   244→					cluster = string(val)
   245→					lvl = nil
   246→				}
   247→			case "hostname", "host":
   248→				if string(val) == prevHost {
   249→					host = prevHost
   250→				} else {
   251→					host = string(val)
   252→					lvl = nil
   253→				}
   254→			case "type":
   255→				if string(val) == "node" {
   256→					break
   257→				}
   258→
   259→				// We cannot be sure that the "type" tag comes before the "type-id" tag:
   260→				if len(st.typeBuf) == 0 {
   261→					st.typeBuf = append(st.typeBuf, val...)
   262→				} else {
   263→					st.typeBuf = reorder(st.typeBuf, val)
   264→				}
   265→			case "type-id":
   266→				st.typeBuf = append(st.typeBuf, val...)
   267→			case "subtype":
   268→				// We cannot be sure that the "subtype" tag comes before the "stype-id" tag:
   269→				if len(st.subTypeBuf) == 0 {
   270→					st.subTypeBuf = append(st.subTypeBuf, val...)
   271→				} else {
   272→					st.subTypeBuf = reorder(st.subTypeBuf, val)
   273→				}
   274→			case "stype-id":
   275→				st.subTypeBuf = append(st.subTypeBuf, val...)
   276→			default:
   277→			}
   278→		}
   279→
   280→		// If the cluster or host changed, the lvl was set to nil
   281→		if lvl == nil {
   282→			st.selector = st.selector[:2]
   283→			st.selector[0], st.selector[1] = cluster, host
   284→			lvl = ms.GetLevel(st.selector)
   285→			prevCluster, prevHost = cluster, host
   286→		}
   287→
   288→		// subtypes: cache []byte→string conversions; messages in a batch typically
   289→		// share the same type/subtype so the hit rate is very high.
   290→		st.selector = st.selector[:0]
   291→		if len(st.typeBuf) > 0 {
   292→			if !bytes.Equal(st.typeBuf, st.prevTypeBytes) {
   293→				st.prevTypeBytes = append(st.prevTypeBytes[:0], st.typeBuf...)
   294→				st.prevTypeStr = string(st.typeBuf)
   295→			}
   296→			st.selector = append(st.selector, st.prevTypeStr)
   297→			if len(st.subTypeBuf) > 0 {
   298→				if !bytes.Equal(st.subTypeBuf, st.prevSubTypeBytes) {
   299→					st.prevSubTypeBytes = append(st.prevSubTypeBytes[:0], st.subTypeBuf...)
   300→					st.prevSubTypeStr = string(st.subTypeBuf)
   301→				}
   302→				st.selector = append(st.selector, st.prevSubTypeStr)
   303→			}
   304→		}
   305→
   306→		for {
   307→			key, val, err := dec.NextField()
   308→			if err != nil {
   309→				return err
   310→			}
   311→
   312→			if key == nil {
   313→				break
   314→			}
   315→
   316→			if string(key) != "value" {
   317→				return fmt.Errorf("host %s: unknown field: '%s' (value: %#v)", host, string(key), val)
   318→			}
   319→
   320→			if val.Kind() == lineprotocol.Float {
   321→				metric.Value = schema.Float(val.FloatV())
   322→			} else if val.Kind() == lineprotocol.Int {
   323→				metric.Value = schema.Float(val.IntV())
   324→			} else if val.Kind() == lineprotocol.Uint {
   325→				metric.Value = schema.Float(val.UintV())
   326→			} else {
   327→				return fmt.Errorf("host %s: unsupported value type in message: %s", host, val.Kind().String())
   328→			}
   329→		}
   330→
   331→		if t, err = dec.Time(lineprotocol.Second, t); err != nil {
   332→			t = time.Now()
   333→			if t, err = dec.Time(lineprotocol.Millisecond, t); err != nil {
   334→				t = time.Now()
   335→				if t, err = dec.Time(lineprotocol.Microsecond, t); err != nil {
   336→					t = time.Now()
   337→					if t, err = dec.Time(lineprotocol.Nanosecond, t); err != nil {
   338→						return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
   339→					}
   340→				}
   341→			}
   342→		}
   343→
   344→		if err != nil {
   345→			return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
   346→		}
   347→
   348→		time := t.Unix()
   349→
   350→		if Keys.Checkpoints.FileFormat == "wal" {
   351→			WALMessages <- &WALMessage{
   352→				MetricName: string(st.metricBuf),
   353→				Cluster:    cluster,
   354→				Node:       host,
   355→				Selector:   append([]string{}, st.selector...),
   356→				Value:      metric.Value,
   357→				Timestamp:  time,
   358→			}
   359→		}
   360→
   361→		if err := ms.WriteToLevel(lvl, st.selector, time, []Metric{metric}); err != nil {
   362→			cclog.Warnf("write error for host %s metric %s at ts %d: %s", host, string(st.metricBuf), time, err.Error())
   363→			continue
   364→		}
   365→	}
   366→	return nil
   367→}
   368→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// This file implements ingestion of InfluxDB line-protocol metric data received
// over NATS. Each line encodes one metric sample with the following structure:
//
//	<measurement>[,cluster=<c>][,hostname=<h>][,type=<t>][,type-id=<id>][,subtype=<s>][,stype-id=<id>] value=<v> [<timestamp>]
//
// The measurement name identifies the metric (e.g. "cpu_load"). Tags provide
// routing information (cluster, host) and optional sub-device selectors (type,
// subtype). Only one field is expected per line: "value".
//
// After decoding, each sample is:
//  1. Written to the in-memory store via ms.WriteToLevel.
//  2. If the checkpoint format is "wal", also forwarded to the WAL staging
//     goroutine via the WALMessages channel for durable write-ahead logging.
package metricstore

import (
	"bytes"
	"context"
	"fmt"
	"sync"
	"time"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/nats"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	"github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
)

// ReceiveNats subscribes to all configured NATS subjects and feeds incoming
// line-protocol messages into the MemoryStore.
//
// When workers > 1 a pool of goroutines drains a shared channel so that
// multiple messages can be decoded in parallel. With workers == 1 the NATS
// callback decodes inline (no channel overhead, lower latency).
//
// The function blocks until ctx is cancelled and all worker goroutines have
// finished. It returns nil when the NATS client is not configured; callers
// should treat that as a no-op rather than an error.
func ReceiveNats(ms *MemoryStore,
	workers int,
	ctx context.Context,
) error {
	nc := nats.GetClient()

	if nc == nil {
		cclog.Warn("NATS client not initialized")
		return nil
	}

	var wg sync.WaitGroup
	msgs := make(chan []byte, workers*2)

	for _, sc := range *Keys.Subscriptions {
		clusterTag := sc.ClusterTag
		if workers > 1 {
			wg.Add(workers)

			for range workers {
				go func() {
					defer wg.Done()
					for m := range msgs {
						dec := lineprotocol.NewDecoderWithBytes(m)
						if err := DecodeLine(dec, ms, clusterTag); err != nil {
							cclog.Errorf("error: %s", err.Error())
						}
					}
				}()
			}

			nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
				select {
				case msgs <- data:
				case <-ctx.Done():
				}
			})
		} else {
			nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
				dec := lineprotocol.NewDecoderWithBytes(data)
				if err := DecodeLine(dec, ms, clusterTag); err != nil {
					cclog.Errorf("error: %s", err.Error())
				}
			})
		}
		cclog.Infof("NATS subscription to '%s' established", sc.SubscribeTo)
	}

	go func() {
		<-ctx.Done()
		close(msgs)
	}()

	wg.Wait()

	return nil
}

// reorder prepends prefix to buf in-place when buf has enough spare capacity,
// avoiding an allocation. Falls back to a regular append otherwise.
//
// It is used to assemble the "type<type-id>" and "subtype<stype-id>" selector
// strings when the type tag arrives before the type-id tag in the line, so the
// two byte slices need to be concatenated in tag-declaration order regardless
// of wire order.
func reorder(buf, prefix []byte) []byte {
	n := len(prefix)
	m := len(buf)
	if cap(buf) < m+n {
		return append(prefix[:n:n], buf...)
	} else {
		buf = buf[:n+m]
		for i := m - 1; i >= 0; i-- {
			buf[i+n] = buf[i]
		}
		for i := range n {
			buf[i] = prefix[i]
		}
		return buf
	}
}

// decodeState holds the per-call scratch buffers used by DecodeLine.
// Instances are recycled via decodeStatePool to avoid repeated allocations
// during high-throughput ingestion.
type decodeState struct {
	// metricBuf holds a copy of the current measurement name (line-protocol
	// measurement field). Copied because dec.Measurement() returns a slice
	// that is invalidated by the next decoder call.
	metricBuf []byte

	// selector is the sub-device path passed to WriteToLevel and WALMessage
	// (e.g. ["socket0"] or ["socket0", "memctrl1"]). Reused across lines.
	selector []string

	// typeBuf accumulates the concatenated "type"+"type-id" tag value for the
	// current line. Reset at the start of each line's tag-decode loop.
	typeBuf []byte

	// subTypeBuf accumulates the concatenated "subtype"+"stype-id" tag value.
	// Reset at the start of each line's tag-decode loop.
	subTypeBuf []byte

	// prevTypeBytes / prevTypeStr cache the last seen typeBuf content and its
	// string conversion. Because consecutive lines in a batch typically address
	// the same sub-device, the cache hit rate is very high and avoids
	// repeated []byte→string allocations.
	prevTypeBytes []byte
	prevTypeStr   string

	// prevSubTypeBytes / prevSubTypeStr are the same cache for the subtype.
	prevSubTypeBytes []byte
	prevSubTypeStr   string
}

// decodeStatePool recycles decodeState values across DecodeLine calls to
// reduce GC pressure during sustained metric ingestion.
var decodeStatePool = sync.Pool{
	New: func() any {
		return &decodeState{
			metricBuf:  make([]byte, 0, 16),
			selector:   make([]string, 0, 4),
			typeBuf:    make([]byte, 0, 16),
			subTypeBuf: make([]byte, 0, 16),
		}
	},
}

// DecodeLine reads all lines from dec (InfluxDB line-protocol) and writes each
// decoded metric sample into ms.
//
// clusterDefault is used as the cluster name for lines that do not carry a
// "cluster" tag. Callers typically supply the ClusterTag value from the NATS
// subscription configuration.
//
// Performance notes:
//   - A decodeState is obtained from decodeStatePool to reuse scratch buffers.
//   - The Level pointer (host-level node in the metric tree) is cached across
//     consecutive lines that share the same cluster+host pair to avoid
//     repeated lock acquisitions on the root and cluster levels.
//   - []byte→string conversions for type/subtype selectors are cached via
//     prevType*/prevSubType* fields because batches typically repeat the same
//     sub-device identifiers.
//   - Timestamp parsing tries Second precision first; if that fails it retries
//     Millisecond, Microsecond, and Nanosecond in turn. A missing timestamp
//     falls back to time.Now().
//
// When the checkpoint format is "wal" each successfully decoded sample is also
// sent to WALMessages so the WAL staging goroutine can persist it durably
// before the next binary snapshot.
func DecodeLine(dec *lineprotocol.Decoder,
	ms *MemoryStore,
	clusterDefault string,
) error {
	// Reduce allocations in loop:
	t := time.Now()
	metric := Metric{}
	st := decodeStatePool.Get().(*decodeState)
	defer decodeStatePool.Put(st)

	// Optimize for the case where all lines in a "batch" are about the same
	// cluster and host. By using `WriteToLevel` (level = host), we do not need
	// to take the root- and cluster-level lock as often.
	var lvl *Level = nil
	prevCluster, prevHost := "", ""

	var ok bool
	for dec.Next() {
		rawmeasurement, err := dec.Measurement()
		if err != nil {
			return err
		}

		// Needs to be copied because another call to dec.* would
		// invalidate the returned slice.
		st.metricBuf = append(st.metricBuf[:0], rawmeasurement...)

		// The go compiler optimizes map[string(byteslice)] lookups:
		metric.MetricConfig, ok = ms.Metrics[string(rawmeasurement)]
		if !ok {
			continue
		}

		st.typeBuf, st.subTypeBuf = st.typeBuf[:0], st.subTypeBuf[:0]
		cluster, host := clusterDefault, ""
		for {
			key, val, err := dec.NextTag()
			if err != nil {
				return err
			}
			if key == nil {
				break
			}

			// The go compiler optimizes string([]byte{...}) == "...":
			switch string(key) {
			case "cluster":
				if string(val) == prevCluster {
					cluster = prevCluster
				} else {
					cluster = string(val)
					lvl = nil
				}
			case "hostname", "host":
				if string(val) == prevHost {
					host = prevHost
				} else {
					host = string(val)
					lvl = nil
				}
			case "type":
				if string(val) == "node" {
					break
				}

				// We cannot be sure that the "type" tag comes before the "type-id" tag:
				if len(st.typeBuf) == 0 {
					st.typeBuf = append(st.typeBuf, val...)
				} else {
					st.typeBuf = reorder(st.typeBuf, val)
				}
			case "type-id":
				st.typeBuf = append(st.typeBuf, val...)
			case "subtype":
				// We cannot be sure that the "subtype" tag comes before the "stype-id" tag:
				if len(st.subTypeBuf) == 0 {
					st.subTypeBuf = append(st.subTypeBuf, val...)
				} else {
					st.subTypeBuf = reorder(st.subTypeBuf, val)
				}
			case "stype-id":
				st.subTypeBuf = append(st.subTypeBuf, val...)
			default:
			}
		}

		// If the cluster or host changed, the lvl was set to nil
		if lvl == nil {
			st.selector = st.selector[:2]
			st.selector[0], st.selector[1] = cluster, host
			lvl = ms.GetLevel(st.selector)
			prevCluster, prevHost = cluster, host
		}

		// subtypes: cache []byte→string conversions; messages in a batch typically
		// share the same type/subtype so the hit rate is very high.
		st.selector = st.selector[:0]
		if len(st.typeBuf) > 0 {
			if !bytes.Equal(st.typeBuf, st.prevTypeBytes) {
				st.prevTypeBytes = append(st.prevTypeBytes[:0], st.typeBuf...)
				st.prevTypeStr = string(st.typeBuf)
			}
			st.selector = append(st.selector, st.prevTypeStr)
			if len(st.subTypeBuf) > 0 {
				if !bytes.Equal(st.subTypeBuf, st.prevSubTypeBytes) {
					st.prevSubTypeBytes = append(st.prevSubTypeBytes[:0], st.subTypeBuf...)
					st.prevSubTypeStr = string(st.subTypeBuf)
				}
				st.selector = append(st.selector, st.prevSubTypeStr)
			}
		}

		for {
			key, val, err := dec.NextField()
			if err != nil {
				return err
			}

			if key == nil {
				break
			}

			if string(key) != "value" {
				return fmt.Errorf("host %s: unknown field: '%s' (value: %#v)", host, string(key), val)
			}

			if val.Kind() == lineprotocol.Float {
				metric.Value = schema.Float(val.FloatV())
			} else if val.Kind() == lineprotocol.Int {
				metric.Value = schema.Float(val.IntV())
			} else if val.Kind() == lineprotocol.Uint {
				metric.Value = schema.Float(val.UintV())
			} else {
				return fmt.Errorf("host %s: unsupported value type in message: %s", host, val.Kind().String())
			}
		}

		if t, err = dec.Time(lineprotocol.Second, t); err != nil {
			t = time.Now()
			if t, err = dec.Time(lineprotocol.Millisecond, t); err != nil {
				t = time.Now()
				if t, err = dec.Time(lineprotocol.Microsecond, t); err != nil {
					t = time.Now()
					if t, err = dec.Time(lineprotocol.Nanosecond, t); err != nil {
						return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
					}
				}
			}
		}

		if err != nil {
			return fmt.Errorf("host %s: timestamp : %#v with error : %#v", host, t, err.Error())
		}

		time := t.Unix()

		if Keys.Checkpoints.FileFormat == "wal" {
			WALMessages <- &WALMessage{
				MetricName: string(st.metricBuf),
				Cluster:    cluster,
				Node:       host,
				Selector:   append([]string{}, st.selector...),
				Value:      metric.Value,
				Timestamp:  time,
			}
		}

		if err := ms.WriteToLevel(lvl, st.selector, time, []Metric{metric}); err != nil {
			cclog.Warnf("write error for host %s metric %s at ts %d: %s", host, string(st.metricBuf), time, err.Error())
			continue
		}
	}
	return nil
}