Implement the following plan:

# Metricstore Memory Explosion Fix Plan

## Root Cause Analysis

The memory explosion has two interacting causes:
1. **Emergency free mechanism is broken** — freed buffers go back to pool, pool isn't cleared after the loop, so `mem.Alloc` never decreases. The loop runs 100 iterations uselessly, then waits 1 hour before trying again. Memory grows unchecked.
2. **Batch abort on write error** — a single out-of-order timestamp aborts the entire NATS batch (`lineprotocol.go:361`). This causes data gaps across many metrics/hosts, producing the "does not align" warnings during reads. The same instability that causes timestamp issues causes bursts of data that grow memory.

## Changes

### 1. `pkg/metricstore/metricstore.go` — Fix MemoryUsageTracker (lines 383-476)
- Remove premature `bufferPool.Clear()` + log at lines 431-432
- In emergency loop (lines 439-458): add `bufferPool.Clear()` + `runtime.GC()` before `ReadMemStats` at line 455
- After loop: add `bufferPool.Clear()` + uncomment `debug.FreeOSMemory()` (lines 460-462)
- Add adaptive ticker: if memory > 80% cap, reset ticker to 30s; else normal interval
- Replace progressive `ForceFree` loop with `Free(nil, threshold)` using progressive threshold (fewer tree traversals, more effective)

### 2. `pkg/metricstore/config.go` — Reduce check interval (line 56)
- Change `DefaultMemoryUsageTrackerInterval` from `1 * time.Hour` to `5 * time.Minute`
- Add `MemoryCheckInterval string` field to `MetricStoreConfig`

### 3. `pkg/metricstore/lineprotocol.go` — Don't abort batch on write error (lines 361-363)
- Change `return err` to `cclog.Warn(...)` + `continue` so remaining lines in the NATS message are processed

### 4. `pkg/metricstore/level.go` — Prune empty levels after free
- Modify `free()` to return emptiness flag; parent deletes empty children from map
- Improve `sizeInBytes()` to count buffer struct overhead (not just float data)

### 5. `pkg/metricstore/buffer.go` — Add `bufferCount()` helper
- Count linked-list length for overhead calculation in `sizeInBytes()`

## Verification
```bash
go test ./pkg/metricstore/...
```

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-ClusterCockpit-cc-backend/7e911723-3abe-436f-b672-d1eef82d81c0.jsonl


Let me read the relevant files first.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstore provides an efficient in-memory time-series metric storage system
     7→// with support for hierarchical data organization, checkpointing, and archiving.
     8→//
     9→// The package organizes metrics in a tree structure (cluster → host → component) and
    10→// provides concurrent read/write access to metric data with configurable aggregation strategies.
    11→// Background goroutines handle periodic checkpointing (JSON or Avro format), archiving old data,
    12→// and enforcing retention policies.
    13→//
    14→// Key features:
    15→//   - In-memory metric storage with configurable retention
    16→//   - Hierarchical data organization (selectors)
    17→//   - Concurrent checkpoint/archive workers
    18→//   - Support for sum and average aggregation
    19→//   - NATS integration for metric ingestion
    20→package metricstore
    21→
    22→import (
    23→	"bytes"
    24→	"context"
    25→	"encoding/json"
    26→	"errors"
    27→	"fmt"
    28→	"runtime"
    29→	"runtime/debug"
    30→	"slices"
    31→	"sync"
    32→	"time"
    33→
    34→	"github.com/ClusterCockpit/cc-backend/internal/config"
    35→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    36→	"github.com/ClusterCockpit/cc-lib/v2/resampler"
    37→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    38→	"github.com/ClusterCockpit/cc-lib/v2/util"
    39→)
    40→
    41→// GlobalState holds the global state for the metric store with thread-safe access.
    42→type GlobalState struct {
    43→	mu                sync.RWMutex
    44→	lastRetentionTime int64
    45→	selectorsExcluded bool
    46→}
    47→
    48→var (
    49→	singleton  sync.Once
    50→	msInstance *MemoryStore
    51→	// shutdownFunc stores the context cancellation function created in Init
    52→	// and is called during Shutdown to cancel all background goroutines
    53→	shutdownFunc   context.CancelFunc
    54→	shutdownFuncMu sync.Mutex // Protects shutdownFunc from concurrent access
    55→	// Create a global instance
    56→	state = &GlobalState{}
    57→)
    58→
    59→// NodeProvider provides information about nodes currently in use by running jobs.
    60→//
    61→// This interface allows metricstore to query job information without directly
    62→// depending on the repository package, breaking the import cycle.
    63→//
    64→// Implementations should return nodes that are actively processing jobs started
    65→// before the given timestamp. These nodes will be excluded from retention-based
    66→// garbage collection to prevent data loss for jobs that are still running or
    67→// recently completed.
    68→type NodeProvider interface {
    69→	// GetUsedNodes returns a map of cluster names to sorted lists of unique hostnames
    70→	// that are currently in use by jobs that started before the given timestamp.
    71→	//
    72→	// Parameters:
    73→	//   - ts: Unix timestamp threshold - returns nodes with jobs started before this time
    74→	//
    75→	// Returns:
    76→	//   - Map of cluster names to lists of node hostnames that should be excluded from garbage collection
    77→	//   - Error if the query fails
    78→	GetUsedNodes(ts int64) (map[string][]string, error)
    79→}
    80→
    81→// Metric represents a single metric data point to be written to the store.
    82→type Metric struct {
    83→	Name  string
    84→	Value schema.Float
    85→	// MetricConfig contains frequency and aggregation settings for this metric.
    86→	// If Frequency is 0, configuration will be looked up from MemoryStore.Metrics during Write().
    87→	MetricConfig MetricConfig
    88→}
    89→
    90→// MemoryStore is the main in-memory time-series metric storage implementation.
    91→//
    92→// It organizes metrics in a hierarchical tree structure where each level represents
    93→// a component of the system hierarchy (e.g., cluster → host → CPU). Each level can
    94→// store multiple metrics as time-series buffers.
    95→//
    96→// The store is initialized as a singleton via InitMetrics() and accessed via GetMemoryStore().
    97→// All public methods are safe for concurrent use.
    98→type MemoryStore struct {
    99→	Metrics      map[string]MetricConfig
   100→	root         Level
   101→	nodeProvider NodeProvider
   102→}
   103→
   104→// Init initializes the metric store from configuration and starts background workers.
   105→//
   106→// This function must be called exactly once before any other metricstore operations.
   107→// It performs the following initialization steps:
   108→//  1. Validates and decodes the metric store configuration
   109→//  2. Configures worker pool size (defaults to NumCPU/2+1, max 10)
   110→//  3. Loads metric configurations from all registered clusters
   111→//  4. Restores checkpoints within the retention window
   112→//  5. Starts background workers for retention, checkpointing, archiving, and monitoring
   113→//  6. Optionally subscribes to NATS for real-time metric ingestion
   114→//
   115→// Parameters:
   116→//   - rawConfig: JSON configuration for the metric store (see MetricStoreConfig)
   117→//   - wg: WaitGroup that will be incremented for each background goroutine started
   118→//
   119→// The function will call cclog.Fatal on critical errors during initialization.
   120→// Use Shutdown() to cleanly stop all background workers started by Init().
   121→//
   122→// Note: Signal handling must be implemented by the caller. Call Shutdown() when
   123→// receiving termination signals to ensure checkpoint data is persisted.
   124→func Init(rawConfig json.RawMessage, metrics map[string]MetricConfig, wg *sync.WaitGroup) {
   125→	startupTime := time.Now()
   126→
   127→	if rawConfig != nil {
   128→		config.Validate(configSchema, rawConfig)
   129→		dec := json.NewDecoder(bytes.NewReader(rawConfig))
   130→		dec.DisallowUnknownFields()
   131→		if err := dec.Decode(&Keys); err != nil {
   132→			cclog.Abortf("[METRICSTORE]> Metric Store Config Init: Could not decode config file '%s'.\nError: %s\n", rawConfig, err.Error())
   133→		}
   134→	}
   135→
   136→	// Set NumWorkers from config or use default
   137→	if Keys.NumWorkers <= 0 {
   138→		Keys.NumWorkers = min(runtime.NumCPU()/2+1, DefaultMaxWorkers)
   139→	}
   140→	cclog.Debugf("[METRICSTORE]> Using %d workers for checkpoint/archive operations\n", Keys.NumWorkers)
   141→
   142→	// Pass the config.MetricStoreKeys
   143→	InitMetrics(metrics)
   144→
   145→	ms := GetMemoryStore()
   146→
   147→	d, err := time.ParseDuration(Keys.RetentionInMemory)
   148→	if err != nil {
   149→		cclog.Fatal(err)
   150→	}
   151→
   152→	restoreFrom := startupTime.Add(-d)
   153→	cclog.Infof("[METRICSTORE]> Loading checkpoints newer than %s\n", restoreFrom.Format(time.RFC3339))
   154→	files, err := ms.FromCheckpointFiles(Keys.Checkpoints.RootDir, restoreFrom.Unix())
   155→	loadedData := ms.SizeInBytes() / 1024 / 1024 // In MB
   156→	if err != nil {
   157→		cclog.Fatalf("[METRICSTORE]> Loading checkpoints failed: %s\n", err.Error())
   158→	} else {
   159→		cclog.Infof("[METRICSTORE]> Checkpoints loaded (%d files, %d MB, that took %fs)\n", files, loadedData, time.Since(startupTime).Seconds())
   160→	}
   161→
   162→	// Try to use less memory by forcing a GC run here and then
   163→	// lowering the target percentage. The default of 100 means
   164→	// that only once the ratio of new allocations execeds the
   165→	// previously active heap, a GC is triggered.
   166→	// Forcing a GC here will set the "previously active heap"
   167→	// to a minumum.
   168→	// runtime.GC()
   169→
   170→	ctx, shutdown := context.WithCancel(context.Background())
   171→
   172→	Retention(wg, ctx)
   173→	Checkpointing(wg, ctx)
   174→	CleanUp(wg, ctx)
   175→	WALStaging(wg, ctx)
   176→	MemoryUsageTracker(wg, ctx)
   177→
   178→	// Note: Signal handling has been removed from this function.
   179→	// The caller is responsible for handling shutdown signals and calling
   180→	// the shutdown() function when appropriate.
   181→	// Store the shutdown function for later use by Shutdown()
   182→	shutdownFuncMu.Lock()
   183→	shutdownFunc = shutdown
   184→	shutdownFuncMu.Unlock()
   185→
   186→	if Keys.Subscriptions != nil {
   187→		err = ReceiveNats(ms, 1, ctx)
   188→		if err != nil {
   189→			cclog.Fatal(err)
   190→		}
   191→	}
   192→}
   193→
   194→// InitMetrics initializes the singleton MemoryStore instance with the given metric configurations.
   195→//
   196→// This function must be called before GetMemoryStore() and can only be called once due to
   197→// the singleton pattern. It assigns each metric an internal offset for efficient buffer indexing.
   198→//
   199→// Parameters:
   200→//   - metrics: Map of metric names to their configurations (frequency and aggregation strategy)
   201→//
   202→// Panics if any metric has Frequency == 0, which indicates an invalid configuration.
   203→//
   204→// After this call, the global msInstance is ready for use via GetMemoryStore().
   205→func InitMetrics(metrics map[string]MetricConfig) {
   206→	singleton.Do(func() {
   207→		offset := 0
   208→		for key, cfg := range metrics {
   209→			if cfg.Frequency == 0 {
   210→				panic("[METRICSTORE]> invalid frequency")
   211→			}
   212→
   213→			metrics[key] = MetricConfig{
   214→				Frequency:   cfg.Frequency,
   215→				Aggregation: cfg.Aggregation,
   216→				offset:      offset,
   217→			}
   218→			offset += 1
   219→		}
   220→
   221→		msInstance = &MemoryStore{
   222→			root: Level{
   223→				metrics:  make([]*buffer, len(metrics)),
   224→				children: make(map[string]*Level),
   225→			},
   226→			Metrics: metrics,
   227→		}
   228→	})
   229→}
   230→
   231→// GetMemoryStore returns the singleton MemoryStore instance.
   232→//
   233→// Returns the initialized MemoryStore singleton. Calls cclog.Fatal if InitMetrics() was not called first.
   234→//
   235→// This function is safe for concurrent use after initialization.
   236→func GetMemoryStore() *MemoryStore {
   237→	if msInstance == nil {
   238→		cclog.Warnf("[METRICSTORE]> MemoryStore not initialized!")
   239→	}
   240→
   241→	return msInstance
   242→}
   243→
   244→func (ms *MemoryStore) GetMetricFrequency(metricName string) (int64, error) {
   245→	if metric, ok := ms.Metrics[metricName]; ok {
   246→		return metric.Frequency, nil
   247→	}
   248→	return 0, fmt.Errorf("[METRICSTORE]> metric %s not found", metricName)
   249→}
   250→
   251→// SetNodeProvider sets the NodeProvider implementation for the MemoryStore.
   252→// This must be called during initialization to provide job state information
   253→// for selective buffer retention during Free operations.
   254→// If not set, the Free function will fall back to freeing all buffers.
   255→func (ms *MemoryStore) SetNodeProvider(provider NodeProvider) {
   256→	ms.nodeProvider = provider
   257→}
   258→
   259→// Shutdown performs a graceful shutdown of the metric store.
   260→//
   261→// This function cancels all background goroutines started by Init() and writes
   262→// a final checkpoint to disk before returning. It should be called when the
   263→// application receives a termination signal.
   264→//
   265→// The function will:
   266→//  1. Cancel the context to stop all background workers
   267→//  2. Close the WAL messages channel if using WAL format
   268→//  3. Write a final checkpoint to preserve in-memory data
   269→//  4. Log any errors encountered during shutdown
   270→//
   271→// Note: This function blocks until the final checkpoint is written.
   272→func Shutdown() {
   273→	shutdownFuncMu.Lock()
   274→	defer shutdownFuncMu.Unlock()
   275→	if shutdownFunc != nil {
   276→		shutdownFunc()
   277→	}
   278→
   279→	if Keys.Checkpoints.FileFormat == "wal" {
   280→		close(WALMessages)
   281→	}
   282→
   283→	cclog.Infof("[METRICSTORE]> Writing to '%s'...\n", Keys.Checkpoints.RootDir)
   284→	var files int
   285→	var err error
   286→
   287→	ms := GetMemoryStore()
   288→
   289→	lastCheckpointMu.Lock()
   290→	from := lastCheckpoint
   291→	lastCheckpointMu.Unlock()
   292→
   293→	if Keys.Checkpoints.FileFormat == "wal" {
   294→		var hostDirs []string
   295→		files, hostDirs, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
   296→		if err == nil {
   297→			RotateWALFilesAfterShutdown(hostDirs)
   298→		}
   299→	} else {
   300→		files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
   301→	}
   302→
   303→	if err != nil {
   304→		cclog.Errorf("[METRICSTORE]> Writing checkpoint failed: %s\n", err.Error())
   305→	}
   306→	cclog.Infof("[METRICSTORE]> Done! (%d files written)\n", files)
   307→}
   308→
   309→// Retention starts a background goroutine that periodically frees old metric data.
   310→//
   311→// This worker runs at half the retention interval and calls Free() to remove buffers
   312→// older than the configured retention time. It respects the NodeProvider to preserve
   313→// data for nodes with active jobs.
   314→//
   315→// Parameters:
   316→//   - wg: WaitGroup to signal completion when context is cancelled
   317→//   - ctx: Context for cancellation signal
   318→//
   319→// The goroutine exits when ctx is cancelled.
   320→func Retention(wg *sync.WaitGroup, ctx context.Context) {
   321→	ms := GetMemoryStore()
   322→
   323→	wg.Go(func() {
   324→		d, err := time.ParseDuration(Keys.RetentionInMemory)
   325→		if err != nil {
   326→			cclog.Fatal(err)
   327→		}
   328→		if d <= 0 {
   329→			return
   330→		}
   331→
   332→		tickInterval := d / 2
   333→		if tickInterval <= 0 {
   334→			return
   335→		}
   336→		ticker := time.NewTicker(tickInterval)
   337→		defer ticker.Stop()
   338→
   339→		for {
   340→			select {
   341→			case <-ctx.Done():
   342→				return
   343→			case <-ticker.C:
   344→				state.mu.Lock()
   345→
   346→				t := time.Now().Add(-d)
   347→
   348→				state.lastRetentionTime = t.Unix()
   349→
   350→				cclog.Infof("[METRICSTORE]> start freeing buffers (older than %s)...\n", t.Format(time.RFC3339))
   351→
   352→				freed, err := Free(ms, t)
   353→				if err != nil {
   354→					cclog.Errorf("[METRICSTORE]> freeing up buffers failed: %s\n", err.Error())
   355→				} else {
   356→					cclog.Infof("[METRICSTORE]> done: %d buffers freed\n", freed)
   357→				}
   358→
   359→				state.mu.Unlock()
   360→
   361→				// Clean up the buffer pool
   362→				bufferPool.Clean(state.lastRetentionTime)
   363→			}
   364→		}
   365→	})
   366→}
   367→
   368→// MemoryUsageTracker starts a background goroutine that monitors memory usage.
   369→//
   370→// This worker checks actual process memory usage (via runtime.MemStats) periodically
   371→// and force-frees buffers if memory exceeds the configured cap. It uses FreeOSMemory()
   372→// to return memory to the OS after freeing buffers, avoiding aggressive GC that causes
   373→// performance issues.
   374→//
   375→// The tracker logs both actual memory usage (heap allocated) and metric data size for
   376→// visibility into memory overhead from Go runtime structures and allocations.
   377→//
   378→// Parameters:
   379→//   - wg: WaitGroup to signal completion when context is cancelled
   380→//   - ctx: Context for cancellation signal
   381→//
   382→// The goroutine exits when ctx is cancelled.
   383→func MemoryUsageTracker(wg *sync.WaitGroup, ctx context.Context) {
   384→	ms := GetMemoryStore()
   385→
   386→	wg.Go(func() {
   387→		d := DefaultMemoryUsageTrackerInterval
   388→
   389→		if d <= 0 {
   390→			return
   391→		}
   392→
   393→		ticker := time.NewTicker(d)
   394→		defer ticker.Stop()
   395→
   396→		for {
   397→			select {
   398→			case <-ctx.Done():
   399→				return
   400→			case <-ticker.C:
   401→				var mem runtime.MemStats
   402→				runtime.ReadMemStats(&mem)
   403→				actualMemoryGB := float64(mem.Alloc) / 1e9
   404→				metricDataGB := ms.SizeInGB()
   405→				cclog.Infof("[METRICSTORE]> memory usage: %.2f GB actual (%.2f GB metric data)", actualMemoryGB, metricDataGB)
   406→
   407→				freedExcluded := 0
   408→				freedEmergency := 0
   409→				var err error
   410→
   411→				state.mu.RLock()
   412→				lastRetention := state.lastRetentionTime
   413→				selectorsExcluded := state.selectorsExcluded
   414→				state.mu.RUnlock()
   415→
   416→				if lastRetention != 0 && selectorsExcluded {
   417→					freedExcluded, err = ms.Free(nil, lastRetention)
   418→					if err != nil {
   419→						cclog.Errorf("[METRICSTORE]> error while force-freeing the excluded buffers: %s", err)
   420→					}
   421→
   422→					if freedExcluded > 0 {
   423→						debug.FreeOSMemory()
   424→						cclog.Infof("[METRICSTORE]> done: %d excluded buffers force-freed", freedExcluded)
   425→					}
   426→				}
   427→
   428→				runtime.ReadMemStats(&mem)
   429→				actualMemoryGB = float64(mem.Alloc) / 1e9
   430→
   431→				bufferPool.Clear()
   432→				cclog.Infof("[METRICSTORE]> Cleaned up bufferPool\n")
   433→
   434→				if actualMemoryGB > float64(Keys.MemoryCap) {
   435→					cclog.Warnf("[METRICSTORE]> memory usage %.2f GB exceeds cap %d GB, starting emergency buffer freeing", actualMemoryGB, Keys.MemoryCap)
   436→
   437→					const maxIterations = 100
   438→
   439→					for i := range maxIterations {
   440→						if actualMemoryGB < float64(Keys.MemoryCap) {
   441→							break
   442→						}
   443→
   444→						freed, err := ms.ForceFree()
   445→						if err != nil {
   446→							cclog.Errorf("[METRICSTORE]> error while force-freeing buffers: %s", err)
   447→						}
   448→						if freed == 0 {
   449→							cclog.Errorf("[METRICSTORE]> no more buffers to free after %d emergency frees, memory usage %.2f GB still exceeds cap %d GB", freedEmergency, actualMemoryGB, Keys.MemoryCap)
   450→							break
   451→						}
   452→						freedEmergency += freed
   453→
   454→						if i%10 == 0 && freedEmergency > 0 {
   455→							runtime.ReadMemStats(&mem)
   456→							actualMemoryGB = float64(mem.Alloc) / 1e9
   457→						}
   458→					}
   459→
   460→					// if freedEmergency > 0 {
   461→					// 	debug.FreeOSMemory()
   462→					// }
   463→
   464→					runtime.ReadMemStats(&mem)
   465→					actualMemoryGB = float64(mem.Alloc) / 1e9
   466→
   467→					if actualMemoryGB >= float64(Keys.MemoryCap) {
   468→						cclog.Errorf("[METRICSTORE]> after %d emergency frees, memory usage %.2f GB still at/above cap %d GB", freedEmergency, actualMemoryGB, Keys.MemoryCap)
   469→					} else {
   470→						cclog.Infof("[METRICSTORE]> emergency freeing complete: %d buffers freed, memory now %.2f GB", freedEmergency, actualMemoryGB)
   471→					}
   472→				}
   473→			}
   474→		}
   475→	})
   476→}
   477→
   478→// Free removes metric data older than the given time while preserving data for active nodes.
   479→//
   480→// This function implements intelligent retention by consulting the NodeProvider (if configured)
   481→// to determine which nodes are currently in use by running jobs. Data for these nodes is
   482→// preserved even if older than the retention time.
   483→//
   484→// Parameters:
   485→//   - ms: The MemoryStore instance
   486→//   - t: Time threshold - buffers with data older than this will be freed
   487→//
   488→// Returns:
   489→//   - Number of buffers freed
   490→//   - Error if NodeProvider query fails
   491→//
   492→// Behavior:
   493→//   - If no NodeProvider is set: frees all buffers older than t
   494→//   - If NodeProvider returns empty map: frees all buffers older than t
   495→//   - Otherwise: preserves buffers for nodes returned by GetUsedNodes(), frees others
   496→func Free(ms *MemoryStore, t time.Time) (int, error) {
   497→	// If no NodeProvider is configured, free all buffers older than t
   498→	if ms.nodeProvider == nil {
   499→		return ms.Free(nil, t.Unix())
   500→	}
   501→
   502→	excludeSelectors, err := ms.nodeProvider.GetUsedNodes(t.Unix())
   503→	if err != nil {
   504→		return 0, err
   505→	}
   506→
   507→	switch lenMap := len(excludeSelectors); lenMap {
   508→
   509→	// If the length of the map returned by GetUsedNodes() is 0,
   510→	// then use default Free method with nil selector
   511→	case 0:
   512→		state.selectorsExcluded = false
   513→		return ms.Free(nil, t.Unix())
   514→
   515→	// Else formulate selectors, exclude those from the map
   516→	// and free the rest of the selectors
   517→	default:
   518→		state.selectorsExcluded = true
   519→		selectors := GetSelectors(ms, excludeSelectors)
   520→		return FreeSelected(ms, selectors, t)
   521→	}
   522→}
   523→
   524→// FreeSelected frees buffers for specific selectors while preserving others.
   525→//
   526→// This function is used when we want to retain some specific nodes beyond the retention time.
   527→// It iterates through the provided selectors and frees their associated buffers.
   528→//
   529→// Parameters:
   530→//   - ms: The MemoryStore instance
   531→//   - selectors: List of selector paths to free (e.g., [["cluster1", "node1"], ["cluster2", "node2"]])
   532→//   - t: Time threshold for freeing buffers
   533→//
   534→// Returns the total number of buffers freed and any error encountered.
   535→func FreeSelected(ms *MemoryStore, selectors [][]string, t time.Time) (int, error) {
   536→	freed := 0
   537→
   538→	for _, selector := range selectors {
   539→
   540→		freedBuffers, err := ms.Free(selector, t.Unix())
   541→		if err != nil {
   542→			cclog.Errorf("error while freeing selected buffers: %#v", err)
   543→		}
   544→		freed += freedBuffers
   545→
   546→	}
   547→
   548→	return freed, nil
   549→}
   550→
   551→// GetSelectors returns all selectors at depth 2 (cluster/node level) that are NOT in the exclusion map.
   552→//
   553→// This function generates a list of selectors whose buffers should be freed by excluding
   554→// selectors that correspond to nodes currently in use by running jobs.
   555→//
   556→// Parameters:
   557→//   - ms: The MemoryStore instance
   558→//   - excludeSelectors: Map of cluster names to node hostnames that should NOT be freed
   559→//
   560→// Returns a list of selectors ([]string paths) that can be safely freed.
   561→//
   562→// Example:
   563→//
   564→//	If the tree has paths ["emmy", "node001"] and ["emmy", "node002"],
   565→//	and excludeSelectors contains {"emmy": ["node001"]},
   566→//	then only [["emmy", "node002"]] is returned.
   567→func GetSelectors(ms *MemoryStore, excludeSelectors map[string][]string) [][]string {
   568→	allSelectors := ms.GetPaths(2)
   569→
   570→	filteredSelectors := make([][]string, 0, len(allSelectors))
   571→
   572→	for _, path := range allSelectors {
   573→		if len(path) < 2 {
   574→			continue
   575→		}
   576→
   577→		key := path[0]   // The "Key" (Level 1)
   578→		value := path[1] // The "Value" (Level 2)
   579→
   580→		exclude := false
   581→
   582→		// Check if the key exists in our exclusion map
   583→		if excludedValues, exists := excludeSelectors[key]; exists {
   584→			// The key exists, now check if the specific value is in the exclusion list
   585→			if slices.Contains(excludedValues, value) {
   586→				exclude = true
   587→			}
   588→		}
   589→
   590→		if !exclude {
   591→			filteredSelectors = append(filteredSelectors, path)
   592→		}
   593→	}
   594→
   595→	return filteredSelectors
   596→}
   597→
   598→// GetPaths returns a list of lists (paths) to the specified depth.
   599→func (ms *MemoryStore) GetPaths(targetDepth int) [][]string {
   600→	var results [][]string
   601→
   602→	// Start recursion. Initial path is empty.
   603→	// We treat Root as depth 0.
   604→	ms.root.collectPaths(0, targetDepth, []string{}, &results)
   605→
   606→	return results
   607→}
   608→
   609→// Write all values in `metrics` to the level specified by `selector` for time `ts`.
   610→// Look at `findLevelOrCreate` for how selectors work.
   611→func (m *MemoryStore) Write(selector []string, ts int64, metrics []Metric) error {
   612→	var ok bool
   613→	for i, metric := range metrics {
   614→		if metric.MetricConfig.Frequency == 0 {
   615→			metric.MetricConfig, ok = m.Metrics[metric.Name]
   616→			if !ok {
   617→				cclog.Debugf("[METRICSTORE]> Unknown metric '%s' in Write() - skipping", metric.Name)
   618→				metric.MetricConfig.Frequency = 0
   619→			}
   620→			metrics[i] = metric
   621→		}
   622→	}
   623→
   624→	return m.WriteToLevel(&m.root, selector, ts, metrics)
   625→}
   626→
   627→func (m *MemoryStore) GetLevel(selector []string) *Level {
   628→	return m.root.findLevelOrCreate(selector, len(m.Metrics))
   629→}
   630→
   631→// WriteToLevel assumes that `minfo` in `metrics` is filled in
   632→func (m *MemoryStore) WriteToLevel(l *Level, selector []string, ts int64, metrics []Metric) error {
   633→	l = l.findLevelOrCreate(selector, len(m.Metrics))
   634→	l.lock.Lock()
   635→	defer l.lock.Unlock()
   636→
   637→	for _, metric := range metrics {
   638→		if metric.MetricConfig.Frequency == 0 {
   639→			continue
   640→		}
   641→
   642→		b := l.metrics[metric.MetricConfig.offset]
   643→		if b == nil {
   644→			// First write to this metric and level
   645→			b = newBuffer(ts, metric.MetricConfig.Frequency)
   646→			l.metrics[metric.MetricConfig.offset] = b
   647→		}
   648→
   649→		nb, err := b.write(ts, metric.Value)
   650→		if err != nil {
   651→			return err
   652→		}
   653→
   654→		// Last write created a new buffer...
   655→		if b != nb {
   656→			l.metrics[metric.MetricConfig.offset] = nb
   657→		}
   658→	}
   659→	return nil
   660→}
   661→
   662→// Read returns all values for metric `metric` from `from` to `to` for the selected level(s).
   663→// If the level does not hold the metric itself, the data will be aggregated recursively from the children.
   664→// The second and third return value are the actual from/to for the data. Those can be different from
   665→// the range asked for if no data was available.
   666→func (m *MemoryStore) Read(selector util.Selector, metric string, from, to, resolution int64) ([]schema.Float, int64, int64, int64, error) {
   667→	if from > to {
   668→		return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid time range")
   669→	}
   670→
   671→	minfo, ok := m.Metrics[metric]
   672→	if !ok {
   673→		return nil, 0, 0, 0, errors.New("[METRICSTORE]> unknown metric: " + metric)
   674→	}
   675→
   676→	n, data := 0, make([]schema.Float, (to-from)/minfo.Frequency+1)
   677→
   678→	err := m.root.findBuffers(selector, minfo.offset, func(b *buffer) error {
   679→		cdata, cfrom, cto, err := b.read(from, to, data)
   680→		if err != nil {
   681→			return err
   682→		}
   683→
   684→		if n == 0 {
   685→			from, to = cfrom, cto
   686→		} else if from != cfrom || to != cto || len(data) != len(cdata) {
   687→			missingfront, missingback := int((from-cfrom)/minfo.Frequency), int((to-cto)/minfo.Frequency)
   688→			if missingfront != 0 {
   689→				return ErrDataDoesNotAlign
   690→			}
   691→
   692→			newlen := len(cdata) - missingback
   693→			if newlen < 1 {
   694→				return ErrDataDoesNotAlign
   695→			}
   696→			cdata = cdata[0:newlen]
   697→			if len(cdata) != len(data) {
   698→				return ErrDataDoesNotAlign
   699→			}
   700→
   701→			from, to = cfrom, cto
   702→		}
   703→
   704→		data = cdata
   705→		n += 1
   706→		return nil
   707→	})
   708→
   709→	if err != nil {
   710→		return nil, 0, 0, 0, err
   711→	} else if n == 0 {
   712→		return nil, 0, 0, 0, ErrNoHostOrMetric
   713→	} else if n > 1 {
   714→		if minfo.Aggregation == AvgAggregation {
   715→			normalize := 1. / schema.Float(n)
   716→			for i := 0; i < len(data); i++ {
   717→				data[i] *= normalize
   718→			}
   719→		} else if minfo.Aggregation != SumAggregation {
   720→			return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid aggregation")
   721→		}
   722→	}
   723→
   724→	data, resolution, err = resampler.LargestTriangleThreeBucket(data, minfo.Frequency, resolution)
   725→	if err != nil {
   726→		return nil, 0, 0, 0, err
   727→	}
   728→
   729→	return data, from, to, resolution, nil
   730→}
   731→
   732→// Free releases all buffers for the selected level and all its children that
   733→// contain only values older than `t`.
   734→func (m *MemoryStore) Free(selector []string, t int64) (int, error) {
   735→	return m.GetLevel(selector).free(t)
   736→}
   737→
   738→// ForceFree unconditionally removes the oldest buffer from each metric chain.
   739→func (m *MemoryStore) ForceFree() (int, error) {
   740→	return m.GetLevel(nil).forceFree()
   741→}
   742→
   743→func (m *MemoryStore) FreeAll() error {
   744→	for k := range m.root.children {
   745→		delete(m.root.children, k)
   746→	}
   747→
   748→	return nil
   749→}
   750→
   751→func (m *MemoryStore) SizeInBytes() int64 {
   752→	return m.root.sizeInBytes()
   753→}
   754→
   755→func (m *MemoryStore) SizeInGB() float64 {
   756→	return float64(m.root.sizeInBytes()) / 1e9
   757→}
   758→
   759→// ListChildren , given a selector, returns a list of all children of the level
   760→// selected.
   761→func (m *MemoryStore) ListChildren(selector []string) []string {
   762→	lvl := &m.root
   763→	for lvl != nil && len(selector) != 0 {
   764→		lvl.lock.RLock()
   765→		next := lvl.children[selector[0]]
   766→		lvl.lock.RUnlock()
   767→		lvl = next
   768→		selector = selector[1:]
   769→	}
   770→
   771→	if lvl == nil {
   772→		return nil
   773→	}
   774→
   775→	lvl.lock.RLock()
   776→	defer lvl.lock.RUnlock()
   777→
   778→	children := make([]string, 0, len(lvl.children))
   779→	for child := range lvl.children {
   780→		children = append(children, child)
   781→	}
   782→
   783→	return children
   784→}
   785→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package metricstore provides an efficient in-memory time-series metric storage system
// with support for hierarchical data organization, checkpointing, and archiving.
//
// The package organizes metrics in a tree structure (cluster → host → component) and
// provides concurrent read/write access to metric data with configurable aggregation strategies.
// Background goroutines handle periodic checkpointing (JSON or Avro format), archiving old data,
// and enforcing retention policies.
//
// Key features:
//   - In-memory metric storage with configurable retention
//   - Hierarchical data organization (selectors)
//   - Concurrent checkpoint/archive workers
//   - Support for sum and average aggregation
//   - NATS integration for metric ingestion
package metricstore

import (
	"bytes"
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"runtime"
	"runtime/debug"
	"slices"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/config"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/resampler"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	"github.com/ClusterCockpit/cc-lib/v2/util"
)

// GlobalState holds the global state for the metric store with thread-safe access.
type GlobalState struct {
	mu                sync.RWMutex
	lastRetentionTime int64
	selectorsExcluded bool
}

var (
	singleton  sync.Once
	msInstance *MemoryStore
	// shutdownFunc stores the context cancellation function created in Init
	// and is called during Shutdown to cancel all background goroutines
	shutdownFunc   context.CancelFunc
	shutdownFuncMu sync.Mutex // Protects shutdownFunc from concurrent access
	// Create a global instance
	state = &GlobalState{}
)

// NodeProvider provides information about nodes currently in use by running jobs.
//
// This interface allows metricstore to query job information without directly
// depending on the repository package, breaking the import cycle.
//
// Implementations should return nodes that are actively processing jobs started
// before the given timestamp. These nodes will be excluded from retention-based
// garbage collection to prevent data loss for jobs that are still running or
// recently completed.
type NodeProvider interface {
	// GetUsedNodes returns a map of cluster names to sorted lists of unique hostnames
	// that are currently in use by jobs that started before the given timestamp.
	//
	// Parameters:
	//   - ts: Unix timestamp threshold - returns nodes with jobs started before this time
	//
	// Returns:
	//   - Map of cluster names to lists of node hostnames that should be excluded from garbage collection
	//   - Error if the query fails
	GetUsedNodes(ts int64) (map[string][]string, error)
}

// Metric represents a single metric data point to be written to the store.
type Metric struct {
	Name  string
	Value schema.Float
	// MetricConfig contains frequency and aggregation settings for this metric.
	// If Frequency is 0, configuration will be looked up from MemoryStore.Metrics during Write().
	MetricConfig MetricConfig
}

// MemoryStore is the main in-memory time-series metric storage implementation.
//
// It organizes metrics in a hierarchical tree structure where each level represents
// a component of the system hierarchy (e.g., cluster → host → CPU). Each level can
// store multiple metrics as time-series buffers.
//
// The store is initialized as a singleton via InitMetrics() and accessed via GetMemoryStore().
// All public methods are safe for concurrent use.
type MemoryStore struct {
	Metrics      map[string]MetricConfig
	root         Level
	nodeProvider NodeProvider
}

// Init initializes the metric store from configuration and starts background workers.
//
// This function must be called exactly once before any other metricstore operations.
// It performs the following initialization steps:
//  1. Validates and decodes the metric store configuration
//  2. Configures worker pool size (defaults to NumCPU/2+1, max 10)
//  3. Loads metric configurations from all registered clusters
//  4. Restores checkpoints within the retention window
//  5. Starts background workers for retention, checkpointing, archiving, and monitoring
//  6. Optionally subscribes to NATS for real-time metric ingestion
//
// Parameters:
//   - rawConfig: JSON configuration for the metric store (see MetricStoreConfig)
//   - wg: WaitGroup that will be incremented for each background goroutine started
//
// The function will call cclog.Fatal on critical errors during initialization.
// Use Shutdown() to cleanly stop all background workers started by Init().
//
// Note: Signal handling must be implemented by the caller. Call Shutdown() when
// receiving termination signals to ensure checkpoint data is persisted.
func Init(rawConfig json.RawMessage, metrics map[string]MetricConfig, wg *sync.WaitGroup) {
	startupTime := time.Now()

	if rawConfig != nil {
		config.Validate(configSchema, rawConfig)
		dec := json.NewDecoder(bytes.NewReader(rawConfig))
		dec.DisallowUnknownFields()
		if err := dec.Decode(&Keys); err != nil {
			cclog.Abortf("[METRICSTORE]> Metric Store Config Init: Could not decode config file '%s'.\nError: %s\n", rawConfig, err.Error())
		}
	}

	// Set NumWorkers from config or use default
	if Keys.NumWorkers <= 0 {
		Keys.NumWorkers = min(runtime.NumCPU()/2+1, DefaultMaxWorkers)
	}
	cclog.Debugf("[METRICSTORE]> Using %d workers for checkpoint/archive operations\n", Keys.NumWorkers)

	// Pass the config.MetricStoreKeys
	InitMetrics(metrics)

	ms := GetMemoryStore()

	d, err := time.ParseDuration(Keys.RetentionInMemory)
	if err != nil {
		cclog.Fatal(err)
	}

	restoreFrom := startupTime.Add(-d)
	cclog.Infof("[METRICSTORE]> Loading checkpoints newer than %s\n", restoreFrom.Format(time.RFC3339))
	files, err := ms.FromCheckpointFiles(Keys.Checkpoints.RootDir, restoreFrom.Unix())
	loadedData := ms.SizeInBytes() / 1024 / 1024 // In MB
	if err != nil {
		cclog.Fatalf("[METRICSTORE]> Loading checkpoints failed: %s\n", err.Error())
	} else {
		cclog.Infof("[METRICSTORE]> Checkpoints loaded (%d files, %d MB, that took %fs)\n", files, loadedData, time.Since(startupTime).Seconds())
	}

	// Try to use less memory by forcing a GC run here and then
	// lowering the target percentage. The default of 100 means
	// that only once the ratio of new allocations execeds the
	// previously active heap, a GC is triggered.
	// Forcing a GC here will set the "previously active heap"
	// to a minumum.
	// runtime.GC()

	ctx, shutdown := context.WithCancel(context.Background())

	Retention(wg, ctx)
	Checkpointing(wg, ctx)
	CleanUp(wg, ctx)
	WALStaging(wg, ctx)
	MemoryUsageTracker(wg, ctx)

	// Note: Signal handling has been removed from this function.
	// The caller is responsible for handling shutdown signals and calling
	// the shutdown() function when appropriate.
	// Store the shutdown function for later use by Shutdown()
	shutdownFuncMu.Lock()
	shutdownFunc = shutdown
	shutdownFuncMu.Unlock()

	if Keys.Subscriptions != nil {
		err = ReceiveNats(ms, 1, ctx)
		if err != nil {
			cclog.Fatal(err)
		}
	}
}

// InitMetrics initializes the singleton MemoryStore instance with the given metric configurations.
//
// This function must be called before GetMemoryStore() and can only be called once due to
// the singleton pattern. It assigns each metric an internal offset for efficient buffer indexing.
//
// Parameters:
//   - metrics: Map of metric names to their configurations (frequency and aggregation strategy)
//
// Panics if any metric has Frequency == 0, which indicates an invalid configuration.
//
// After this call, the global msInstance is ready for use via GetMemoryStore().
func InitMetrics(metrics map[string]MetricConfig) {
	singleton.Do(func() {
		offset := 0
		for key, cfg := range metrics {
			if cfg.Frequency == 0 {
				panic("[METRICSTORE]> invalid frequency")
			}

			metrics[key] = MetricConfig{
				Frequency:   cfg.Frequency,
				Aggregation: cfg.Aggregation,
				offset:      offset,
			}
			offset += 1
		}

		msInstance = &MemoryStore{
			root: Level{
				metrics:  make([]*buffer, len(metrics)),
				children: make(map[string]*Level),
			},
			Metrics: metrics,
		}
	})
}

// GetMemoryStore returns the singleton MemoryStore instance.
//
// Returns the initialized MemoryStore singleton. Calls cclog.Fatal if InitMetrics() was not called first.
//
// This function is safe for concurrent use after initialization.
func GetMemoryStore() *MemoryStore {
	if msInstance == nil {
		cclog.Warnf("[METRICSTORE]> MemoryStore not initialized!")
	}

	return msInstance
}

func (ms *MemoryStore) GetMetricFrequency(metricName string) (int64, error) {
	if metric, ok := ms.Metrics[metricName]; ok {
		return metric.Frequency, nil
	}
	return 0, fmt.Errorf("[METRICSTORE]> metric %s not found", metricName)
}

// SetNodeProvider sets the NodeProvider implementation for the MemoryStore.
// This must be called during initialization to provide job state information
// for selective buffer retention during Free operations.
// If not set, the Free function will fall back to freeing all buffers.
func (ms *MemoryStore) SetNodeProvider(provider NodeProvider) {
	ms.nodeProvider = provider
}

// Shutdown performs a graceful shutdown of the metric store.
//
// This function cancels all background goroutines started by Init() and writes
// a final checkpoint to disk before returning. It should be called when the
// application receives a termination signal.
//
// The function will:
//  1. Cancel the context to stop all background workers
//  2. Close the WAL messages channel if using WAL format
//  3. Write a final checkpoint to preserve in-memory data
//  4. Log any errors encountered during shutdown
//
// Note: This function blocks until the final checkpoint is written.
func Shutdown() {
	shutdownFuncMu.Lock()
	defer shutdownFuncMu.Unlock()
	if shutdownFunc != nil {
		shutdownFunc()
	}

	if Keys.Checkpoints.FileFormat == "wal" {
		close(WALMessages)
	}

	cclog.Infof("[METRICSTORE]> Writing to '%s'...\n", Keys.Checkpoints.RootDir)
	var files int
	var err error

	ms := GetMemoryStore()

	lastCheckpointMu.Lock()
	from := lastCheckpoint
	lastCheckpointMu.Unlock()

	if Keys.Checkpoints.FileFormat == "wal" {
		var hostDirs []string
		files, hostDirs, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
		if err == nil {
			RotateWALFilesAfterShutdown(hostDirs)
		}
	} else {
		files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
	}

	if err != nil {
		cclog.Errorf("[METRICSTORE]> Writing checkpoint failed: %s\n", err.Error())
	}
	cclog.Infof("[METRICSTORE]> Done! (%d files written)\n", files)
}

// Retention starts a background goroutine that periodically frees old metric data.
//
// This worker runs at half the retention interval and calls Free() to remove buffers
// older than the configured retention time. It respects the NodeProvider to preserve
// data for nodes with active jobs.
//
// Parameters:
//   - wg: WaitGroup to signal completion when context is cancelled
//   - ctx: Context for cancellation signal
//
// The goroutine exits when ctx is cancelled.
func Retention(wg *sync.WaitGroup, ctx context.Context) {
	ms := GetMemoryStore()

	wg.Go(func() {
		d, err := time.ParseDuration(Keys.RetentionInMemory)
		if err != nil {
			cclog.Fatal(err)
		}
		if d <= 0 {
			return
		}

		tickInterval := d / 2
		if tickInterval <= 0 {
			return
		}
		ticker := time.NewTicker(tickInterval)
		defer ticker.Stop()

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				state.mu.Lock()

				t := time.Now().Add(-d)

				state.lastRetentionTime = t.Unix()

				cclog.Infof("[METRICSTORE]> start freeing buffers (older than %s)...\n", t.Format(time.RFC3339))

				freed, err := Free(ms, t)
				if err != nil {
					cclog.Errorf("[METRICSTORE]> freeing up buffers failed: %s\n", err.Error())
				} else {
					cclog.Infof("[METRICSTORE]> done: %d buffers freed\n", freed)
				}

				state.mu.Unlock()

				// Clean up the buffer pool
				bufferPool.Clean(state.lastRetentionTime)
			}
		}
	})
}

// MemoryUsageTracker starts a background goroutine that monitors memory usage.
//
// This worker checks actual process memory usage (via runtime.MemStats) periodically
// and force-frees buffers if memory exceeds the configured cap. It uses FreeOSMemory()
// to return memory to the OS after freeing buffers, avoiding aggressive GC that causes
// performance issues.
//
// The tracker logs both actual memory usage (heap allocated) and metric data size for
// visibility into memory overhead from Go runtime structures and allocations.
//
// Parameters:
//   - wg: WaitGroup to signal completion when context is cancelled
//   - ctx: Context for cancellation signal
//
// The goroutine exits when ctx is cancelled.
func MemoryUsageTracker(wg *sync.WaitGroup, ctx context.Context) {
	ms := GetMemoryStore()

	wg.Go(func() {
		d := DefaultMemoryUsageTrackerInterval

		if d <= 0 {
			return
		}

		ticker := time.NewTicker(d)
		defer ticker.Stop()

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				var mem runtime.MemStats
				runtime.ReadMemStats(&mem)
				actualMemoryGB := float64(mem.Alloc) / 1e9
				metricDataGB := ms.SizeInGB()
				cclog.Infof("[METRICSTORE]> memory usage: %.2f GB actual (%.2f GB metric data)", actualMemoryGB, metricDataGB)

				freedExcluded := 0
				freedEmergency := 0
				var err error

				state.mu.RLock()
				lastRetention := state.lastRetentionTime
				selectorsExcluded := state.selectorsExcluded
				state.mu.RUnlock()

				if lastRetention != 0 && selectorsExcluded {
					freedExcluded, err = ms.Free(nil, lastRetention)
					if err != nil {
						cclog.Errorf("[METRICSTORE]> error while force-freeing the excluded buffers: %s", err)
					}

					if freedExcluded > 0 {
						debug.FreeOSMemory()
						cclog.Infof("[METRICSTORE]> done: %d excluded buffers force-freed", freedExcluded)
					}
				}

				runtime.ReadMemStats(&mem)
				actualMemoryGB = float64(mem.Alloc) / 1e9

				bufferPool.Clear()
				cclog.Infof("[METRICSTORE]> Cleaned up bufferPool\n")

				if actualMemoryGB > float64(Keys.MemoryCap) {
					cclog.Warnf("[METRICSTORE]> memory usage %.2f GB exceeds cap %d GB, starting emergency buffer freeing", actualMemoryGB, Keys.MemoryCap)

					const maxIterations = 100

					for i := range maxIterations {
						if actualMemoryGB < float64(Keys.MemoryCap) {
							break
						}

						freed, err := ms.ForceFree()
						if err != nil {
							cclog.Errorf("[METRICSTORE]> error while force-freeing buffers: %s", err)
						}
						if freed == 0 {
							cclog.Errorf("[METRICSTORE]> no more buffers to free after %d emergency frees, memory usage %.2f GB still exceeds cap %d GB", freedEmergency, actualMemoryGB, Keys.MemoryCap)
							break
						}
						freedEmergency += freed

						if i%10 == 0 && freedEmergency > 0 {
							runtime.ReadMemStats(&mem)
							actualMemoryGB = float64(mem.Alloc) / 1e9
						}
					}

					// if freedEmergency > 0 {
					// 	debug.FreeOSMemory()
					// }

					runtime.ReadMemStats(&mem)
					actualMemoryGB = float64(mem.Alloc) / 1e9

					if actualMemoryGB >= float64(Keys.MemoryCap) {
						cclog.Errorf("[METRICSTORE]> after %d emergency frees, memory usage %.2f GB still at/above cap %d GB", freedEmergency, actualMemoryGB, Keys.MemoryCap)
					} else {
						cclog.Infof("[METRICSTORE]> emergency freeing complete: %d buffers freed, memory now %.2f GB", freedEmergency, actualMemoryGB)
					}
				}
			}
		}
	})
}

// Free removes metric data older than the given time while preserving data for active nodes.
//
// This function implements intelligent retention by consulting the NodeProvider (if configured)
// to determine which nodes are currently in use by running jobs. Data for these nodes is
// preserved even if older than the retention time.
//
// Parameters:
//   - ms: The MemoryStore instance
//   - t: Time threshold - buffers with data older than this will be freed
//
// Returns:
//   - Number of buffers freed
//   - Error if NodeProvider query fails
//
// Behavior:
//   - If no NodeProvider is set: frees all buffers older than t
//   - If NodeProvider returns empty map: frees all buffers older than t
//   - Otherwise: preserves buffers for nodes returned by GetUsedNodes(), frees others
func Free(ms *MemoryStore, t time.Time) (int, error) {
	// If no NodeProvider is configured, free all buffers older than t
	if ms.nodeProvider == nil {
		return ms.Free(nil, t.Unix())
	}

	excludeSelectors, err := ms.nodeProvider.GetUsedNodes(t.Unix())
	if err != nil {
		return 0, err
	}

	switch lenMap := len(excludeSelectors); lenMap {

	// If the length of the map returned by GetUsedNodes() is 0,
	// then use default Free method with nil selector
	case 0:
		state.selectorsExcluded = false
		return ms.Free(nil, t.Unix())

	// Else formulate selectors, exclude those from the map
	// and free the rest of the selectors
	default:
		state.selectorsExcluded = true
		selectors := GetSelectors(ms, excludeSelectors)
		return FreeSelected(ms, selectors, t)
	}
}

// FreeSelected frees buffers for specific selectors while preserving others.
//
// This function is used when we want to retain some specific nodes beyond the retention time.
// It iterates through the provided selectors and frees their associated buffers.
//
// Parameters:
//   - ms: The MemoryStore instance
//   - selectors: List of selector paths to free (e.g., [["cluster1", "node1"], ["cluster2", "node2"]])
//   - t: Time threshold for freeing buffers
//
// Returns the total number of buffers freed and any error encountered.
func FreeSelected(ms *MemoryStore, selectors [][]string, t time.Time) (int, error) {
	freed := 0

	for _, selector := range selectors {

		freedBuffers, err := ms.Free(selector, t.Unix())
		if err != nil {
			cclog.Errorf("error while freeing selected buffers: %#v", err)
		}
		freed += freedBuffers

	}

	return freed, nil
}

// GetSelectors returns all selectors at depth 2 (cluster/node level) that are NOT in the exclusion map.
//
// This function generates a list of selectors whose buffers should be freed by excluding
// selectors that correspond to nodes currently in use by running jobs.
//
// Parameters:
//   - ms: The MemoryStore instance
//   - excludeSelectors: Map of cluster names to node hostnames that should NOT be freed
//
// Returns a list of selectors ([]string paths) that can be safely freed.
//
// Example:
//
//	If the tree has paths ["emmy", "node001"] and ["emmy", "node002"],
//	and excludeSelectors contains {"emmy": ["node001"]},
//	then only [["emmy", "node002"]] is returned.
func GetSelectors(ms *MemoryStore, excludeSelectors map[string][]string) [][]string {
	allSelectors := ms.GetPaths(2)

	filteredSelectors := make([][]string, 0, len(allSelectors))

	for _, path := range allSelectors {
		if len(path) < 2 {
			continue
		}

		key := path[0]   // The "Key" (Level 1)
		value := path[1] // The "Value" (Level 2)

		exclude := false

		// Check if the key exists in our exclusion map
		if excludedValues, exists := excludeSelectors[key]; exists {
			// The key exists, now check if the specific value is in the exclusion list
			if slices.Contains(excludedValues, value) {
				exclude = true
			}
		}

		if !exclude {
			filteredSelectors = append(filteredSelectors, path)
		}
	}

	return filteredSelectors
}

// GetPaths returns a list of lists (paths) to the specified depth.
func (ms *MemoryStore) GetPaths(targetDepth int) [][]string {
	var results [][]string

	// Start recursion. Initial path is empty.
	// We treat Root as depth 0.
	ms.root.collectPaths(0, targetDepth, []string{}, &results)

	return results
}

// Write all values in `metrics` to the level specified by `selector` for time `ts`.
// Look at `findLevelOrCreate` for how selectors work.
func (m *MemoryStore) Write(selector []string, ts int64, metrics []Metric) error {
	var ok bool
	for i, metric := range metrics {
		if metric.MetricConfig.Frequency == 0 {
			metric.MetricConfig, ok = m.Metrics[metric.Name]
			if !ok {
				cclog.Debugf("[METRICSTORE]> Unknown metric '%s' in Write() - skipping", metric.Name)
				metric.MetricConfig.Frequency = 0
			}
			metrics[i] = metric
		}
	}

	return m.WriteToLevel(&m.root, selector, ts, metrics)
}

func (m *MemoryStore) GetLevel(selector []string) *Level {
	return m.root.findLevelOrCreate(selector, len(m.Metrics))
}

// WriteToLevel assumes that `minfo` in `metrics` is filled in
func (m *MemoryStore) WriteToLevel(l *Level, selector []string, ts int64, metrics []Metric) error {
	l = l.findLevelOrCreate(selector, len(m.Metrics))
	l.lock.Lock()
	defer l.lock.Unlock()

	for _, metric := range metrics {
		if metric.MetricConfig.Frequency == 0 {
			continue
		}

		b := l.metrics[metric.MetricConfig.offset]
		if b == nil {
			// First write to this metric and level
			b = newBuffer(ts, metric.MetricConfig.Frequency)
			l.metrics[metric.MetricConfig.offset] = b
		}

		nb, err := b.write(ts, metric.Value)
		if err != nil {
			return err
		}

		// Last write created a new buffer...
		if b != nb {
			l.metrics[metric.MetricConfig.offset] = nb
		}
	}
	return nil
}

// Read returns all values for metric `metric` from `from` to `to` for the selected level(s).
// If the level does not hold the metric itself, the data will be aggregated recursively from the children.
// The second and third return value are the actual from/to for the data. Those can be different from
// the range asked for if no data was available.
func (m *MemoryStore) Read(selector util.Selector, metric string, from, to, resolution int64) ([]schema.Float, int64, int64, int64, error) {
	if from > to {
		return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid time range")
	}

	minfo, ok := m.Metrics[metric]
	if !ok {
		return nil, 0, 0, 0, errors.New("[METRICSTORE]> unknown metric: " + metric)
	}

	n, data := 0, make([]schema.Float, (to-from)/minfo.Frequency+1)

	err := m.root.findBuffers(selector, minfo.offset, func(b *buffer) error {
		cdata, cfrom, cto, err := b.read(from, to, data)
		if err != nil {
			return err
		}

		if n == 0 {
			from, to = cfrom, cto
		} else if from != cfrom || to != cto || len(data) != len(cdata) {
			missingfront, missingback := int((from-cfrom)/minfo.Frequency), int((to-cto)/minfo.Frequency)
			if missingfront != 0 {
				return ErrDataDoesNotAlign
			}

			newlen := len(cdata) - missingback
			if newlen < 1 {
				return ErrDataDoesNotAlign
			}
			cdata = cdata[0:newlen]
			if len(cdata) != len(data) {
				return ErrDataDoesNotAlign
			}

			from, to = cfrom, cto
		}

		data = cdata
		n += 1
		return nil
	})

	if err != nil {
		return nil, 0, 0, 0, err
	} else if n == 0 {
		return nil, 0, 0, 0, ErrNoHostOrMetric
	} else if n > 1 {
		if minfo.Aggregation == AvgAggregation {
			normalize := 1. / schema.Float(n)
			for i := 0; i < len(data); i++ {
				data[i] *= normalize
			}
		} else if minfo.Aggregation != SumAggregation {
			return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid aggregation")
		}
	}

	data, resolution, err = resampler.LargestTriangleThreeBucket(data, minfo.Frequency, resolution)
	if err != nil {
		return nil, 0, 0, 0, err
	}

	return data, from, to, resolution, nil
}

// Free releases all buffers for the selected level and all its children that
// contain only values older than `t`.
func (m *MemoryStore) Free(selector []string, t int64) (int, error) {
	return m.GetLevel(selector).free(t)
}

// ForceFree unconditionally removes the oldest buffer from each metric chain.
func (m *MemoryStore) ForceFree() (int, error) {
	return m.GetLevel(nil).forceFree()
}

func (m *MemoryStore) FreeAll() error {
	for k := range m.root.children {
		delete(m.root.children, k)
	}

	return nil
}

func (m *MemoryStore) SizeInBytes() int64 {
	return m.root.sizeInBytes()
}

func (m *MemoryStore) SizeInGB() float64 {
	return float64(m.root.sizeInBytes()) / 1e9
}

// ListChildren , given a selector, returns a list of all children of the level
// selected.
func (m *MemoryStore) ListChildren(selector []string) []string {
	lvl := &m.root
	for lvl != nil && len(selector) != 0 {
		lvl.lock.RLock()
		next := lvl.children[selector[0]]
		lvl.lock.RUnlock()
		lvl = next
		selector = selector[1:]
	}

	if lvl == nil {
		return nil
	}

	lvl.lock.RLock()
	defer lvl.lock.RUnlock()

	children := make([]string, 0, len(lvl.children))
	for child := range lvl.children {
		children = append(children, child)
	}

	return children
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstore provides config.go: Configuration structures and metric management.
     7→//
     8→// # Configuration Hierarchy
     9→//
    10→// The metricstore package uses nested configuration structures:
    11→//
    12→//	MetricStoreConfig (Keys)
    13→//	├─ NumWorkers: Parallel checkpoint/archive workers
    14→//	├─ RetentionInMemory: How long to keep data in RAM (also used as cleanup interval)
    15→//	├─ MemoryCap: Memory limit in bytes (triggers forceFree)
    16→//	├─ Checkpoints: Persistence configuration
    17→//	│  ├─ FileFormat: "json" or "wal" (default: "wal")
    18→//	│  └─ RootDir: Checkpoint storage path
    19→//	├─ Cleanup: Long-term storage configuration (interval = RetentionInMemory)
    20→//	│  ├─ RootDir: Archive storage path (archive mode only)
    21→//	│  └─ Mode: "delete" or "archive"
    22→//	├─ Debug: Development/debugging options
    23→//	└─ Subscriptions: NATS topic subscriptions for metric ingestion
    24→//
    25→// # Metric Configuration
    26→//
    27→// Each metric (e.g., "cpu_load", "mem_used") has a MetricConfig entry in the global
    28→// Metrics map, defining:
    29→//
    30→//   - Frequency: Measurement interval in seconds
    31→//   - Aggregation: How to combine values (sum/avg/none) when transforming scopes
    32→//   - offset: Internal index into Level.metrics slice (assigned during Init)
    33→//
    34→// # AggregationStrategy
    35→//
    36→// Determines how to combine metric values when aggregating from finer to coarser scopes:
    37→//
    38→//   - NoAggregation: Do not combine (incompatible scopes)
    39→//   - SumAggregation: Add values (e.g., power consumption: core→socket)
    40→//   - AvgAggregation: Average values (e.g., temperature: core→socket)
    41→package metricstore
    42→
    43→import (
    44→	"fmt"
    45→	"time"
    46→
    47→	"github.com/ClusterCockpit/cc-backend/pkg/archive"
    48→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    49→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    50→)
    51→
    52→const (
    53→	DefaultMaxWorkers                 = 10
    54→	DefaultBufferCapacity             = 512
    55→	DefaultGCTriggerInterval          = 100
    56→	DefaultMemoryUsageTrackerInterval = 1 * time.Hour
    57→)
    58→
    59→// Checkpoints configures periodic persistence of in-memory metric data.
    60→//
    61→// Fields:
    62→//   - FileFormat: "json" (human-readable, periodic) or "wal" (binary snapshot + WAL, crash-safe); default is "wal"
    63→//   - RootDir:    Filesystem path for checkpoint files (created if missing)
    64→type Checkpoints struct {
    65→	FileFormat string `json:"file-format"`
    66→	RootDir    string `json:"directory"`
    67→}
    68→
    69→// Debug provides development and profiling options.
    70→//
    71→// Fields:
    72→//   - DumpToFile: Path to dump checkpoint data for inspection (empty = disabled)
    73→//   - EnableGops: Enable gops agent for live runtime debugging (https://github.com/google/gops)
    74→type Debug struct {
    75→	DumpToFile string `json:"dump-to-file"`
    76→	EnableGops bool   `json:"gops"`
    77→}
    78→
    79→// Cleanup configures long-term storage of old metric data.
    80→//
    81→// Data older than RetentionInMemory is archived to disk or deleted.
    82→// The cleanup interval is always RetentionInMemory.
    83→//
    84→// Fields:
    85→//   - RootDir: Filesystem path for archived data (used in "archive" mode)
    86→//   - Mode:    "delete" (discard old data) or "archive" (write to RootDir)
    87→type Cleanup struct {
    88→	RootDir string `json:"directory"`
    89→	Mode    string `json:"mode"`
    90→}
    91→
    92→// Subscriptions defines NATS topics to subscribe to for metric ingestion.
    93→//
    94→// Each subscription receives metrics via NATS messaging, enabling real-time
    95→// data collection from compute nodes.
    96→//
    97→// Fields:
    98→//   - SubscribeTo: NATS subject/channel name (e.g., "metrics.compute.*")
    99→//   - ClusterTag:  Default cluster name for metrics without cluster tag (optional)
   100→type Subscriptions []struct {
   101→	// Channel name
   102→	SubscribeTo string `json:"subscribe-to"`
   103→
   104→	// Allow lines without a cluster tag, use this as default, optional
   105→	ClusterTag string `json:"cluster-tag"`
   106→}
   107→
   108→// MetricStoreConfig defines the main configuration for the metricstore.
   109→//
   110→// Loaded from cc-backend's config.json "metricstore" section. Controls memory usage,
   111→// persistence, archiving, and metric ingestion.
   112→//
   113→// Fields:
   114→//   - NumWorkers:        Parallel workers for checkpoint/archive (0 = auto: min(NumCPU/2+1, 10))
   115→//   - RetentionInMemory: Duration string (e.g., "48h") for in-memory data retention
   116→//   - MemoryCap:         Max bytes for buffer data (0 = unlimited); triggers forceFree when exceeded
   117→//   - Checkpoints:       Periodic persistence configuration
   118→//   - Debug:             Development/profiling options (nil = disabled)
   119→//   - Archive:           Long-term storage configuration (nil = disabled)
   120→//   - Subscriptions:     NATS topics for metric ingestion (nil = polling only)
   121→type MetricStoreConfig struct {
   122→	// Number of concurrent workers for checkpoint and archive operations.
   123→	// If not set or 0, defaults to min(runtime.NumCPU()/2+1, 10)
   124→	NumWorkers        int            `json:"num-workers"`
   125→	RetentionInMemory string         `json:"retention-in-memory"`
   126→	MemoryCap         int            `json:"memory-cap"`
   127→	Checkpoints       Checkpoints    `json:"checkpoints"`
   128→	Debug             *Debug         `json:"debug"`
   129→	Cleanup           *Cleanup       `json:"cleanup"`
   130→	Subscriptions     *Subscriptions `json:"nats-subscriptions"`
   131→}
   132→
   133→// Keys is the global metricstore configuration instance.
   134→//
   135→// Initialized with defaults, then overwritten by cc-backend's config.json.
   136→// Accessed by Init(), Checkpointing(), and other lifecycle functions.
   137→var Keys MetricStoreConfig = MetricStoreConfig{
   138→	Checkpoints: Checkpoints{
   139→		FileFormat: "wal",
   140→		RootDir:    "./var/checkpoints",
   141→	},
   142→	Cleanup: &Cleanup{
   143→		Mode: "delete",
   144→	},
   145→}
   146→
   147→// AggregationStrategy defines how to combine metric values across hierarchy levels.
   148→//
   149→// Used when transforming data from finer-grained scopes (e.g., core) to coarser scopes
   150→// (e.g., socket). This is SPATIAL aggregation, not TEMPORAL (time-based) aggregation.
   151→//
   152→// Values:
   153→//   - NoAggregation:  Do not aggregate (incompatible scopes or non-aggregatable metrics)
   154→//   - SumAggregation: Add values (e.g., power: sum core power → socket power)
   155→//   - AvgAggregation: Average values (e.g., temperature: average core temps → socket temp)
   156→type AggregationStrategy int
   157→
   158→const (
   159→	NoAggregation  AggregationStrategy = iota // Do not aggregate
   160→	SumAggregation                            // Sum values (e.g., power, energy)
   161→	AvgAggregation                            // Average values (e.g., temperature, utilization)
   162→)
   163→
   164→// AssignAggregationStrategy parses a string into an AggregationStrategy value.
   165→//
   166→// Used when loading metric configurations from JSON/YAML files.
   167→//
   168→// Parameters:
   169→//   - str: "sum", "avg", or "" (empty string for NoAggregation)
   170→//
   171→// Returns:
   172→//   - AggregationStrategy: Parsed value
   173→//   - error:               Non-nil if str is unrecognized
   174→func AssignAggregationStrategy(str string) (AggregationStrategy, error) {
   175→	switch str {
   176→	case "":
   177→		return NoAggregation, nil
   178→	case "sum":
   179→		return SumAggregation, nil
   180→	case "avg":
   181→		return AvgAggregation, nil
   182→	default:
   183→		return NoAggregation, fmt.Errorf("[METRICSTORE]> unknown aggregation strategy: %s", str)
   184→	}
   185→}
   186→
   187→// MetricConfig defines configuration for a single metric type.
   188→//
   189→// Stored in the global Metrics map, keyed by metric name (e.g., "cpu_load").
   190→//
   191→// Fields:
   192→//   - Frequency:   Measurement interval in seconds (e.g., 60 for 1-minute granularity)
   193→//   - Aggregation: How to combine values across hierarchy levels (sum/avg/none)
   194→//   - offset:      Internal index into Level.metrics slice (assigned during Init)
   195→type MetricConfig struct {
   196→	// Interval in seconds at which measurements are stored
   197→	Frequency int64
   198→
   199→	// Can be 'sum', 'avg' or null. Describes how to aggregate metrics from the same timestep over the hierarchy.
   200→	Aggregation AggregationStrategy
   201→
   202→	// Private, used internally...
   203→	offset int
   204→}
   205→
   206→func BuildMetricList() map[string]MetricConfig {
   207→	var metrics map[string]MetricConfig = make(map[string]MetricConfig)
   208→
   209→	addMetric := func(name string, metric MetricConfig) error {
   210→		if metrics == nil {
   211→			metrics = make(map[string]MetricConfig, 0)
   212→		}
   213→
   214→		if existingMetric, ok := metrics[name]; ok {
   215→			if existingMetric.Frequency != metric.Frequency {
   216→				if existingMetric.Frequency < metric.Frequency {
   217→					existingMetric.Frequency = metric.Frequency
   218→					metrics[name] = existingMetric
   219→				}
   220→			}
   221→		} else {
   222→			metrics[name] = metric
   223→		}
   224→
   225→		return nil
   226→	}
   227→
   228→	// Helper function to add metric configuration
   229→	addMetricConfig := func(mc *schema.MetricConfig) {
   230→		agg, err := AssignAggregationStrategy(mc.Aggregation)
   231→		if err != nil {
   232→			cclog.Warnf("Could not find aggregation strategy for metric config '%s': %s", mc.Name, err.Error())
   233→		}
   234→
   235→		addMetric(mc.Name, MetricConfig{
   236→			Frequency:   int64(mc.Timestep),
   237→			Aggregation: agg,
   238→		})
   239→	}
   240→	for _, c := range archive.Clusters {
   241→		for _, mc := range c.MetricConfig {
   242→			addMetricConfig(mc)
   243→		}
   244→
   245→		for _, sc := range c.SubClusters {
   246→			for _, mc := range sc.MetricConfig {
   247→				addMetricConfig(mc)
   248→			}
   249→		}
   250→	}
   251→
   252→	return metrics
   253→}
   254→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package metricstore provides config.go: Configuration structures and metric management.
//
// # Configuration Hierarchy
//
// The metricstore package uses nested configuration structures:
//
//	MetricStoreConfig (Keys)
//	├─ NumWorkers: Parallel checkpoint/archive workers
//	├─ RetentionInMemory: How long to keep data in RAM (also used as cleanup interval)
//	├─ MemoryCap: Memory limit in bytes (triggers forceFree)
//	├─ Checkpoints: Persistence configuration
//	│  ├─ FileFormat: "json" or "wal" (default: "wal")
//	│  └─ RootDir: Checkpoint storage path
//	├─ Cleanup: Long-term storage configuration (interval = RetentionInMemory)
//	│  ├─ RootDir: Archive storage path (archive mode only)
//	│  └─ Mode: "delete" or "archive"
//	├─ Debug: Development/debugging options
//	└─ Subscriptions: NATS topic subscriptions for metric ingestion
//
// # Metric Configuration
//
// Each metric (e.g., "cpu_load", "mem_used") has a MetricConfig entry in the global
// Metrics map, defining:
//
//   - Frequency: Measurement interval in seconds
//   - Aggregation: How to combine values (sum/avg/none) when transforming scopes
//   - offset: Internal index into Level.metrics slice (assigned during Init)
//
// # AggregationStrategy
//
// Determines how to combine metric values when aggregating from finer to coarser scopes:
//
//   - NoAggregation: Do not combine (incompatible scopes)
//   - SumAggregation: Add values (e.g., power consumption: core→socket)
//   - AvgAggregation: Average values (e.g., temperature: core→socket)
package metricstore

import (
	"fmt"
	"time"

	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

const (
	DefaultMaxWorkers                 = 10
	DefaultBufferCapacity             = 512
	DefaultGCTriggerInterval          = 100
	DefaultMemoryUsageTrackerInterval = 1 * time.Hour
)

// Checkpoints configures periodic persistence of in-memory metric data.
//
// Fields:
//   - FileFormat: "json" (human-readable, periodic) or "wal" (binary snapshot + WAL, crash-safe); default is "wal"
//   - RootDir:    Filesystem path for checkpoint files (created if missing)
type Checkpoints struct {
	FileFormat string `json:"file-format"`
	RootDir    string `json:"directory"`
}

// Debug provides development and profiling options.
//
// Fields:
//   - DumpToFile: Path to dump checkpoint data for inspection (empty = disabled)
//   - EnableGops: Enable gops agent for live runtime debugging (https://github.com/google/gops)
type Debug struct {
	DumpToFile string `json:"dump-to-file"`
	EnableGops bool   `json:"gops"`
}

// Cleanup configures long-term storage of old metric data.
//
// Data older than RetentionInMemory is archived to disk or deleted.
// The cleanup interval is always RetentionInMemory.
//
// Fields:
//   - RootDir: Filesystem path for archived data (used in "archive" mode)
//   - Mode:    "delete" (discard old data) or "archive" (write to RootDir)
type Cleanup struct {
	RootDir string `json:"directory"`
	Mode    string `json:"mode"`
}

// Subscriptions defines NATS topics to subscribe to for metric ingestion.
//
// Each subscription receives metrics via NATS messaging, enabling real-time
// data collection from compute nodes.
//
// Fields:
//   - SubscribeTo: NATS subject/channel name (e.g., "metrics.compute.*")
//   - ClusterTag:  Default cluster name for metrics without cluster tag (optional)
type Subscriptions []struct {
	// Channel name
	SubscribeTo string `json:"subscribe-to"`

	// Allow lines without a cluster tag, use this as default, optional
	ClusterTag string `json:"cluster-tag"`
}

// MetricStoreConfig defines the main configuration for the metricstore.
//
// Loaded from cc-backend's config.json "metricstore" section. Controls memory usage,
// persistence, archiving, and metric ingestion.
//
// Fields:
//   - NumWorkers:        Parallel workers for checkpoint/archive (0 = auto: min(NumCPU/2+1, 10))
//   - RetentionInMemory: Duration string (e.g., "48h") for in-memory data retention
//   - MemoryCap:         Max bytes for buffer data (0 = unlimited); triggers forceFree when exceeded
//   - Checkpoints:       Periodic persistence configuration
//   - Debug:             Development/profiling options (nil = disabled)
//   - Archive:           Long-term storage configuration (nil = disabled)
//   - Subscriptions:     NATS topics for metric ingestion (nil = polling only)
type MetricStoreConfig struct {
	// Number of concurrent workers for checkpoint and archive operations.
	// If not set or 0, defaults to min(runtime.NumCPU()/2+1, 10)
	NumWorkers        int            `json:"num-workers"`
	RetentionInMemory string         `json:"retention-in-memory"`
	MemoryCap         int            `json:"memory-cap"`
	Checkpoints       Checkpoints    `json:"checkpoints"`
	Debug             *Debug         `json:"debug"`
	Cleanup           *Cleanup       `json:"cleanup"`
	Subscriptions     *Subscriptions `json:"nats-subscriptions"`
}

// Keys is the global metricstore configuration instance.
//
// Initialized with defaults, then overwritten by cc-backend's config.json.
// Accessed by Init(), Checkpointing(), and other lifecycle functions.
var Keys MetricStoreConfig = MetricStoreConfig{
	Checkpoints: Checkpoints{
		FileFormat: "wal",
		RootDir:    "./var/checkpoints",
	},
	Cleanup: &Cleanup{
		Mode: "delete",
	},
}

// AggregationStrategy defines how to combine metric values across hierarchy levels.
//
// Used when transforming data from finer-grained scopes (e.g., core) to coarser scopes
// (e.g., socket). This is SPATIAL aggregation, not TEMPORAL (time-based) aggregation.
//
// Values:
//   - NoAggregation:  Do not aggregate (incompatible scopes or non-aggregatable metrics)
//   - SumAggregation: Add values (e.g., power: sum core power → socket power)
//   - AvgAggregation: Average values (e.g., temperature: average core temps → socket temp)
type AggregationStrategy int

const (
	NoAggregation  AggregationStrategy = iota // Do not aggregate
	SumAggregation                            // Sum values (e.g., power, energy)
	AvgAggregation                            // Average values (e.g., temperature, utilization)
)

// AssignAggregationStrategy parses a string into an AggregationStrategy value.
//
// Used when loading metric configurations from JSON/YAML files.
//
// Parameters:
//   - str: "sum", "avg", or "" (empty string for NoAggregation)
//
// Returns:
//   - AggregationStrategy: Parsed value
//   - error:               Non-nil if str is unrecognized
func AssignAggregationStrategy(str string) (AggregationStrategy, error) {
	switch str {
	case "":
		return NoAggregation, nil
	case "sum":
		return SumAggregation, nil
	case "avg":
		return AvgAggregation, nil
	default:
		return NoAggregation, fmt.Errorf("[METRICSTORE]> unknown aggregation strategy: %s", str)
	}
}

// MetricConfig defines configuration for a single metric type.
//
// Stored in the global Metrics map, keyed by metric name (e.g., "cpu_load").
//
// Fields:
//   - Frequency:   Measurement interval in seconds (e.g., 60 for 1-minute granularity)
//   - Aggregation: How to combine values across hierarchy levels (sum/avg/none)
//   - offset:      Internal index into Level.metrics slice (assigned during Init)
type MetricConfig struct {
	// Interval in seconds at which measurements are stored
	Frequency int64

	// Can be 'sum', 'avg' or null. Describes how to aggregate metrics from the same timestep over the hierarchy.
	Aggregation AggregationStrategy

	// Private, used internally...
	offset int
}

func BuildMetricList() map[string]MetricConfig {
	var metrics map[string]MetricConfig = make(map[string]MetricConfig)

	addMetric := func(name string, metric MetricConfig) error {
		if metrics == nil {
			metrics = make(map[string]MetricConfig, 0)
		}

		if existingMetric, ok := metrics[name]; ok {
			if existingMetric.Frequency != metric.Frequency {
				if existingMetric.Frequency < metric.Frequency {
					existingMetric.Frequency = metric.Frequency
					metrics[name] = existingMetric
				}
			}
		} else {
			metrics[name] = metric
		}

		return nil
	}

	// Helper function to add metric configuration
	addMetricConfig := func(mc *schema.MetricConfig) {
		agg, err := AssignAggregationStrategy(mc.Aggregation)
		if err != nil {
			cclog.Warnf("Could not find aggregation strategy for metric config '%s': %s", mc.Name, err.Error())
		}

		addMetric(mc.Name, MetricConfig{
			Frequency:   int64(mc.Timestep),
			Aggregation: agg,
		})
	}
	for _, c := range archive.Clusters {
		for _, mc := range c.MetricConfig {
			addMetricConfig(mc)
		}

		for _, sc := range c.SubClusters {
			for _, mc := range sc.MetricConfig {
				addMetricConfig(mc)
			}
		}
	}

	return metrics
}