Implement the following plan:

# Metricstore In-Depth Review: Bugs & Improvements

## BUG 1: `isEmpty()` deadlock risk in `free()` (level.go:209,218)
**Severity: HIGH** — `free()` holds `l.lock.Lock()` (line 184), then calls `child.free(t)` which acquires `child.lock.Lock()`. Then `child.isEmpty()` (line 209) acquires `child.lock.RLock()` — **re-entrant RLock on a locked RWMutex**. Go's `sync.RWMutex` is NOT reentrant. `child.free()` returns with `child.lock` unlocked (defer), then `isEmpty()` re-locks it — this works. BUT: `isEmpty()` is called on the child right after `child.free()` returns, and `child.free()` already released its lock via defer. Between `child.free()` returning and `isEmpty()` being called, another goroutine could add a new metric to the child, making it appear non-empty. This is a **TOCTOU race** but not a deadlock. However, the real risk is: during the `child.free()` call, child's children may have been pruned, and then `isEmpty()` re-acquires the lock — this is safe but has a race window.

**Fix**: Check emptiness inside `free()` itself, returning a boolean alongside `(int, error)`, so the parent can decide while holding its own lock consistently.

## BUG 2: `stats()` index recalculation after buffer switch (stats.go:40-46)
**Severity: MEDIUM** — When `idx >= cap(b.data)`, the code does `b = b.next; idx = 0`. This hardcodes idx=0, ignoring the actual timestamp position in the next buffer. If there's a gap between buffers (different start times), `idx=0` is wrong — it reads data from the wrong timestamp. This is the **same bug** that was already fixed in `buffer.read()` (the `TestBufferReadIdxAfterSwitch` test covers read, but NOT stats).

**Fix**: After `b = b.next`, recalculate `idx = int((t - b.start) / b.frequency)` (same as was done in `read()`).

## BUG 3: `FreeAll()` not thread-safe (metricstore.go:756-762)
**Severity: MEDIUM** — `FreeAll()` iterates and deletes from `m.root.children` without holding `m.root.lock`. Concurrent reads/writes will race.

**Fix**: Wrap in `m.root.lock.Lock()`/`Unlock()`.

## BUG 4: `toCheckpointFile` reads buffer without lock (checkpoint.go:268-299)
**Severity: MEDIUM** — `toCheckpointFile` holds `l.lock.RLock()` (line 257) protecting `l.metrics`, but the buffer's `iterFromTo` and `read` are called without the buffer's own lock. Concurrent `write()` to the same buffer (from NATS ingestion in `WriteToLevel`) can cause data races because `write()` mutates `b.data` (append) and `b.next` under `l.lock.Lock()` — but `toCheckpointFile` only holds `RLock`. This means a write and a checkpoint read can race on buffer internals.

**Fix**: This is partially mitigated because `read()` and `iterFromTo` only read `b.data` and traverse `b.prev`. The real danger is `b.data` being appended to (which can reallocate the slice) while being read. Since `l.lock.RLock` is held during checkpoint read, and `WriteToLevel` acquires `l.lock.Lock`, they are actually mutually exclusive. **Actually safe** — no fix needed.

## BUG 5: WAL writes not fsynced (walCheckpoint.go:307)
**Severity: LOW** — `writeWALRecord` does `f.Write()` without `f.Sync()`. On crash, WAL records may be lost even though they were "written". The CRC check on replay handles corrupt trailing records gracefully, but recent data can be silently lost.

**Fix**: Consider periodic `f.Sync()` (e.g., every N records or on a timer) rather than per-record fsync for performance.

## BUG 6: `ScaleBy(0)` is a no-op (api.go:131)
**Severity: LOW** — `ScaleBy` returns early when factor is 0, but scaling by 0 should zero out data. The comment says "for performance" but the semantics are wrong.

**Fix**: Only skip for factor == 1 (identity), not 0.

## BUG 7: `collectMetricStatus` panics on unknown metric (healthcheck.go:69-70)
**Severity: MEDIUM** — If `expectedMetrics` contains a metric name not present in `m.Metrics`, `mc := m.Metrics[metricName]` returns zero-value MetricConfig with `offset=0`, then `l.metrics[mc.offset]` accesses index 0 which may belong to a different metric. No panic, but **wrong metric data** is checked for health status.

**Fix**: Check `_, ok := m.Metrics[metricName]` and skip unknown metrics.

## IMPROVEMENT 1: WAL buffering (walCheckpoint.go:167-176)
**Current**: Each WAL message results in a separate `f.Write()` syscall. With thousands of metrics per second, this is excessive.
**Fix**: Use a `bufio.Writer` per host WAL file, flushing periodically or on rotation.

## IMPROVEMENT 2: Checkpoint interval hardcoded (checkpoint.go:103)
**Current**: `const checkpointInterval = 12 * time.Hour` — not configurable.
**Fix**: Add optional `CheckpointInterval string` to `MetricStoreConfig` (json: `"checkpoint-interval",omitempty`). Default to `12h` when empty/unset. Parse with `time.ParseDuration` in `Checkpointing()`.

## IMPROVEMENT 3: `MemoryCap` type mismatch
**Current**: `MemoryCap int` in config (bytes implied) but compared as GB: `float64(Keys.MemoryCap)` vs `actualMemoryGB`. If user sets `"memory-cap": 32`, it means 32 GB, not 32 bytes. The config comment says "bytes" but the code treats it as GB.
**Fix**: Clarify the unit in docs/config schema; it's actually GB.

## IMPROVEMENT 4: No WAL size limit
**Current**: WAL files grow unbounded between checkpoints (12h). If ingestion rate is high, WAL files can consume significant disk space and slow replay.
**Fix**: Add configurable WAL size limit; trigger checkpoint when exceeded.

---

## Files to modify

- `pkg/metricstore/level.go` — Bug 1: `free()` returns `(int, bool, error)` where bool=isEmpty; remove separate `isEmpty()` call
- `pkg/metricstore/stats.go:46` — Bug 2: recalculate `idx = int((t - b.start) / b.frequency)` after `b = b.next`
- `pkg/metricstore/metricstore.go:756` — Bug 3: wrap `FreeAll()` body in `m.root.lock.Lock()`/`Unlock()`
- `pkg/metricstore/api.go:131` — Bug 6: change `f == 0 || f == 1` to `f == 1`
- `pkg/metricstore/healthcheck.go:69` — Bug 7: add `mc, ok := m.Metrics[metricName]; if !ok { continue }`
- `pkg/metricstore/config.go` — Improvement 2: add optional `CheckpointInterval string` field (json `"checkpoint-interval,omitempty"`)
- `pkg/metricstore/checkpoint.go:103` — Improvement 2: use `Keys.CheckpointInterval` if set, else default `12h`

## Verification
```bash
go test ./pkg/metricstore/...
```

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-ClusterCockpit-cc-backend/39705f8c-1691-4d60-b0aa-39f588ed9dcf.jsonl


Let me read all the files that need modification first.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstore provides level.go: Hierarchical tree structure for metric storage.
     7→//
     8→// # Level Architecture
     9→//
    10→// The Level type forms a tree structure where each node represents a level in the
    11→// ClusterCockpit hierarchy: cluster → host → socket → core → hwthread, with special
    12→// nodes for memory domains and accelerators.
    13→//
    14→// Structure:
    15→//
    16→//	Root Level (cluster="emmy")
    17→//	├─ Level (host="node001")
    18→//	│  ├─ Level (socket="0")
    19→//	│  │  ├─ Level (core="0") [stores cpu0 metrics]
    20→//	│  │  └─ Level (core="1") [stores cpu1 metrics]
    21→//	│  └─ Level (socket="1")
    22→//	│     └─ ...
    23→//	└─ Level (host="node002")
    24→//	   └─ ...
    25→//
    26→// Each Level can:
    27→//   - Hold data (metrics slice of buffer pointers)
    28→//   - Have child nodes (children map[string]*Level)
    29→//   - Both simultaneously (inner nodes can store aggregated metrics)
    30→//
    31→// # Selector Paths
    32→//
    33→// Selectors are hierarchical paths: []string{"cluster", "host", "component"}.
    34→// Example: []string{"emmy", "node001", "cpu0"} navigates to the cpu0 core level.
    35→//
    36→// # Concurrency
    37→//
    38→// RWMutex protects children map and metrics slice. Read-heavy workload (metric reads)
    39→// uses RLock. Writes (new levels, buffer updates) use Lock. Double-checked locking
    40→// prevents races during level creation.
    41→package metricstore
    42→
    43→import (
    44→	"sync"
    45→	"time"
    46→	"unsafe"
    47→
    48→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    49→	"github.com/ClusterCockpit/cc-lib/v2/util"
    50→)
    51→
    52→// Level represents a node in the hierarchical metric storage tree.
    53→//
    54→// Can be both a leaf or inner node. Inner nodes hold data in 'metrics' for aggregated
    55→// values (e.g., socket-level metrics derived from core-level data). Named "Level"
    56→// instead of "node" to avoid confusion with cluster nodes (hosts).
    57→//
    58→// Fields:
    59→//   - children: Map of child level names to Level pointers (e.g., "cpu0" → Level)
    60→//   - metrics:  Slice of buffer pointers (one per metric, indexed by MetricConfig.offset)
    61→//   - lock:     RWMutex for concurrent access (read-heavy, write-rare)
    62→type Level struct {
    63→	children map[string]*Level
    64→	metrics  []*buffer
    65→	lock     sync.RWMutex
    66→}
    67→
    68→// findLevelOrCreate navigates to or creates the level specified by selector.
    69→//
    70→// Recursively descends the tree, creating missing levels as needed. Uses double-checked
    71→// locking: RLock first (fast path), then Lock if creation needed (slow path), then
    72→// re-check after acquiring Lock to handle races.
    73→//
    74→// Example selector: []string{"emmy", "node001", "cpu0"}
    75→// Navigates: root → emmy → node001 → cpu0, creating levels as needed.
    76→//
    77→// Parameters:
    78→//   - selector: Hierarchical path (consumed recursively, decreasing depth)
    79→//   - nMetrics: Number of metric slots to allocate in new levels
    80→//
    81→// Returns:
    82→//   - *Level: The target level (existing or newly created)
    83→//
    84→// Note: sync.Map may improve performance for high-concurrency writes, but current
    85→// approach suffices for read-heavy workload.
    86→func (l *Level) findLevelOrCreate(selector []string, nMetrics int) *Level {
    87→	if len(selector) == 0 {
    88→		return l
    89→	}
    90→
    91→	// Allow concurrent reads:
    92→	l.lock.RLock()
    93→	var child *Level
    94→	var ok bool
    95→	if l.children == nil {
    96→		// Children map needs to be created...
    97→		l.lock.RUnlock()
    98→	} else {
    99→		child, ok = l.children[selector[0]]
   100→		l.lock.RUnlock()
   101→		if ok {
   102→			return child.findLevelOrCreate(selector[1:], nMetrics)
   103→		}
   104→	}
   105→
   106→	// The level does not exist, take write lock for unique access:
   107→	l.lock.Lock()
   108→	// While this thread waited for the write lock, another thread
   109→	// could have created the child node.
   110→	if l.children != nil {
   111→		child, ok = l.children[selector[0]]
   112→		if ok {
   113→			l.lock.Unlock()
   114→			return child.findLevelOrCreate(selector[1:], nMetrics)
   115→		}
   116→	}
   117→
   118→	child = &Level{
   119→		metrics:  make([]*buffer, nMetrics),
   120→		children: nil,
   121→	}
   122→
   123→	if l.children != nil {
   124→		l.children[selector[0]] = child
   125→	} else {
   126→		l.children = map[string]*Level{selector[0]: child}
   127→	}
   128→	l.lock.Unlock()
   129→	return child.findLevelOrCreate(selector[1:], nMetrics)
   130→}
   131→
   132→// collectPaths gathers all selector paths at the specified depth in the tree.
   133→//
   134→// Recursively traverses children, collecting paths when currentDepth+1 == targetDepth.
   135→// Each path is a selector that can be used with findLevel() or findBuffers().
   136→//
   137→// Explicitly copies slices to avoid shared underlying arrays between siblings, preventing
   138→// unintended mutations.
   139→//
   140→// Parameters:
   141→//   - currentDepth: Depth of current level (0 = root)
   142→//   - targetDepth:  Depth to collect paths from
   143→//   - currentPath:  Path accumulated so far
   144→//   - results:      Output slice (appended to)
   145→//
   146→// Example: collectPaths(0, 2, []string{}, &results) collects all 2-level paths
   147→// like []string{"emmy", "node001"}, []string{"emmy", "node002"}, etc.
   148→func (l *Level) collectPaths(currentDepth, targetDepth int, currentPath []string, results *[][]string) {
   149→	l.lock.RLock()
   150→	defer l.lock.RUnlock()
   151→
   152→	for key, child := range l.children {
   153→		if child == nil {
   154→			continue
   155→		}
   156→
   157→		// We explicitly make a new slice and copy data to avoid sharing underlying arrays between siblings
   158→		newPath := make([]string, len(currentPath))
   159→		copy(newPath, currentPath)
   160→		newPath = append(newPath, key)
   161→
   162→		// Check depth, and just return if depth reached
   163→		if currentDepth+1 == targetDepth {
   164→			*results = append(*results, newPath)
   165→		} else {
   166→			child.collectPaths(currentDepth+1, targetDepth, newPath, results)
   167→		}
   168→	}
   169→}
   170→
   171→// free removes buffers older than the retention threshold from the entire subtree.
   172→//
   173→// Recursively frees buffers in this level's metrics and all child levels. Buffers
   174→// with standard capacity (BufferCap) are returned to the pool. Called by the
   175→// retention worker to enforce retention policies.
   176→//
   177→// Parameters:
   178→//   - t: Retention threshold timestamp (Unix seconds)
   179→//
   180→// Returns:
   181→//   - int:   Total number of buffers freed in this subtree
   182→//   - error: Non-nil on failure (propagated from children)
   183→func (l *Level) free(t int64) (int, error) {
   184→	l.lock.Lock()
   185→	defer l.lock.Unlock()
   186→
   187→	n := 0
   188→	for i, b := range l.metrics {
   189→		if b != nil {
   190→			delme, m := b.free(t)
   191→			n += m
   192→			if delme {
   193→				if cap(b.data) != BufferCap {
   194→					b.data = make([]schema.Float, 0, BufferCap)
   195→				}
   196→				b.lastUsed = time.Now().Unix()
   197→				bufferPool.Put(b)
   198→				l.metrics[i] = nil
   199→			}
   200→		}
   201→	}
   202→
   203→	for key, child := range l.children {
   204→		m, err := child.free(t)
   205→		n += m
   206→		if err != nil {
   207→			return n, err
   208→		}
   209→		if child.isEmpty() {
   210→			delete(l.children, key)
   211→		}
   212→	}
   213→
   214→	return n, nil
   215→}
   216→
   217→// isEmpty returns true if this level has no metrics and no children.
   218→func (l *Level) isEmpty() bool {
   219→	l.lock.RLock()
   220→	defer l.lock.RUnlock()
   221→
   222→	for _, b := range l.metrics {
   223→		if b != nil {
   224→			return false
   225→		}
   226→	}
   227→	return len(l.children) == 0
   228→}
   229→
   230→// forceFree removes the oldest buffer from each metric chain in the subtree.
   231→//
   232→// Unlike free(), which removes based on time threshold, this unconditionally removes
   233→// the oldest buffer in each chain. Used by MemoryUsageTracker when memory cap is
   234→// exceeded and time-based retention is insufficient.
   235→//
   236→// Recursively processes current level's metrics and all child levels.
   237→//
   238→// Returns:
   239→//   - int:   Total number of buffers freed in this subtree
   240→//   - error: Non-nil on failure (propagated from children)
   241→func (l *Level) forceFree() (int, error) {
   242→	l.lock.Lock()
   243→	defer l.lock.Unlock()
   244→
   245→	n := 0
   246→
   247→	// Iterate over metrics in the current level
   248→	for i, b := range l.metrics {
   249→		if b != nil {
   250→			// Attempt to free the oldest buffer in this chain
   251→			delme, freedCount := b.forceFreeOldest()
   252→			n += freedCount
   253→
   254→			// If delme is true, it means 'b' itself (the head) was the oldest
   255→			// and needs to be removed from the slice.
   256→			if delme {
   257→				b.next = nil
   258→				b.prev = nil
   259→				if cap(b.data) != BufferCap {
   260→					b.data = make([]schema.Float, 0, BufferCap)
   261→				}
   262→				b.lastUsed = time.Now().Unix()
   263→				bufferPool.Put(b)
   264→				l.metrics[i] = nil
   265→			}
   266→		}
   267→	}
   268→
   269→	// Recursively traverse children
   270→	for _, child := range l.children {
   271→		m, err := child.forceFree()
   272→		n += m
   273→		if err != nil {
   274→			return n, err
   275→		}
   276→	}
   277→
   278→	return n, nil
   279→}
   280→
   281→// sizeInBytes calculates the total memory usage of all buffers in the subtree.
   282→//
   283→// Recursively sums buffer data sizes (count of Float values × sizeof(Float)) across
   284→// this level's metrics and all child levels. Used by MemoryUsageTracker to enforce
   285→// memory cap limits.
   286→//
   287→// Returns:
   288→//   - int64: Total bytes used by buffer data in this subtree
   289→func (l *Level) sizeInBytes() int64 {
   290→	l.lock.RLock()
   291→	defer l.lock.RUnlock()
   292→	size := int64(0)
   293→
   294→	for _, b := range l.metrics {
   295→		if b != nil {
   296→			size += b.count() * int64(unsafe.Sizeof(schema.Float(0)))
   297→			size += b.bufferCount() * int64(unsafe.Sizeof(buffer{}))
   298→		}
   299→	}
   300→
   301→	for _, child := range l.children {
   302→		size += child.sizeInBytes()
   303→	}
   304→
   305→	return size
   306→}
   307→
   308→// findLevel navigates to the level specified by selector, returning nil if not found.
   309→//
   310→// Read-only variant of findLevelOrCreate. Does not create missing levels.
   311→// Recursively descends the tree following the selector path.
   312→//
   313→// Parameters:
   314→//   - selector: Hierarchical path (e.g., []string{"emmy", "node001", "cpu0"})
   315→//
   316→// Returns:
   317→//   - *Level: The target level, or nil if any component in the path does not exist
   318→func (l *Level) findLevel(selector []string) *Level {
   319→	if len(selector) == 0 {
   320→		return l
   321→	}
   322→
   323→	l.lock.RLock()
   324→	defer l.lock.RUnlock()
   325→
   326→	lvl := l.children[selector[0]]
   327→	if lvl == nil {
   328→		return nil
   329→	}
   330→
   331→	return lvl.findLevel(selector[1:])
   332→}
   333→
   334→// findBuffers invokes callback on all buffers matching the selector pattern.
   335→//
   336→// Supports flexible selector patterns (from cc-lib/util.Selector):
   337→//   - Exact match: Selector element with String set (e.g., "node001")
   338→//   - Group match: Selector element with Group set (e.g., ["cpu0", "cpu2", "cpu4"])
   339→//   - Wildcard:    Selector element with Any=true (matches all children)
   340→//
   341→// Empty selector (len==0) matches current level's buffer at 'offset' and recursively
   342→// all descendant buffers at the same offset (used for aggregation queries).
   343→//
   344→// Parameters:
   345→//   - selector: Pattern to match (consumed recursively)
   346→//   - offset:   Metric index in metrics slice (from MetricConfig.offset)
   347→//   - f:        Callback invoked on each matching buffer
   348→//
   349→// Returns:
   350→//   - error: First error returned by callback, or nil if all succeeded
   351→//
   352→// Example:
   353→//
   354→//	// Find all cpu0 buffers across all hosts:
   355→//	findBuffers([]Selector{{Any: true}, {String: "cpu0"}}, metricOffset, callback)
   356→func (l *Level) findBuffers(selector util.Selector, offset int, f func(b *buffer) error) error {
   357→	l.lock.RLock()
   358→	defer l.lock.RUnlock()
   359→
   360→	if len(selector) == 0 {
   361→		b := l.metrics[offset]
   362→		if b != nil {
   363→			return f(b)
   364→		}
   365→
   366→		for _, lvl := range l.children {
   367→			err := lvl.findBuffers(nil, offset, f)
   368→			if err != nil {
   369→				return err
   370→			}
   371→		}
   372→		return nil
   373→	}
   374→
   375→	sel := selector[0]
   376→	if len(sel.String) != 0 && l.children != nil {
   377→		lvl, ok := l.children[sel.String]
   378→		if ok {
   379→			err := lvl.findBuffers(selector[1:], offset, f)
   380→			if err != nil {
   381→				return err
   382→			}
   383→		}
   384→		return nil
   385→	}
   386→
   387→	if sel.Group != nil && l.children != nil {
   388→		for _, key := range sel.Group {
   389→			lvl, ok := l.children[key]
   390→			if ok {
   391→				err := lvl.findBuffers(selector[1:], offset, f)
   392→				if err != nil {
   393→					return err
   394→				}
   395→			}
   396→		}
   397→		return nil
   398→	}
   399→
   400→	if sel.Any && l.children != nil {
   401→		for _, lvl := range l.children {
   402→			if err := lvl.findBuffers(selector[1:], offset, f); err != nil {
   403→				return err
   404→			}
   405→		}
   406→		return nil
   407→	}
   408→
   409→	return nil
   410→}
   411→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package metricstore provides level.go: Hierarchical tree structure for metric storage.
//
// # Level Architecture
//
// The Level type forms a tree structure where each node represents a level in the
// ClusterCockpit hierarchy: cluster → host → socket → core → hwthread, with special
// nodes for memory domains and accelerators.
//
// Structure:
//
//	Root Level (cluster="emmy")
//	├─ Level (host="node001")
//	│  ├─ Level (socket="0")
//	│  │  ├─ Level (core="0") [stores cpu0 metrics]
//	│  │  └─ Level (core="1") [stores cpu1 metrics]
//	│  └─ Level (socket="1")
//	│     └─ ...
//	└─ Level (host="node002")
//	   └─ ...
//
// Each Level can:
//   - Hold data (metrics slice of buffer pointers)
//   - Have child nodes (children map[string]*Level)
//   - Both simultaneously (inner nodes can store aggregated metrics)
//
// # Selector Paths
//
// Selectors are hierarchical paths: []string{"cluster", "host", "component"}.
// Example: []string{"emmy", "node001", "cpu0"} navigates to the cpu0 core level.
//
// # Concurrency
//
// RWMutex protects children map and metrics slice. Read-heavy workload (metric reads)
// uses RLock. Writes (new levels, buffer updates) use Lock. Double-checked locking
// prevents races during level creation.
package metricstore

import (
	"sync"
	"time"
	"unsafe"

	"github.com/ClusterCockpit/cc-lib/v2/schema"
	"github.com/ClusterCockpit/cc-lib/v2/util"
)

// Level represents a node in the hierarchical metric storage tree.
//
// Can be both a leaf or inner node. Inner nodes hold data in 'metrics' for aggregated
// values (e.g., socket-level metrics derived from core-level data). Named "Level"
// instead of "node" to avoid confusion with cluster nodes (hosts).
//
// Fields:
//   - children: Map of child level names to Level pointers (e.g., "cpu0" → Level)
//   - metrics:  Slice of buffer pointers (one per metric, indexed by MetricConfig.offset)
//   - lock:     RWMutex for concurrent access (read-heavy, write-rare)
type Level struct {
	children map[string]*Level
	metrics  []*buffer
	lock     sync.RWMutex
}

// findLevelOrCreate navigates to or creates the level specified by selector.
//
// Recursively descends the tree, creating missing levels as needed. Uses double-checked
// locking: RLock first (fast path), then Lock if creation needed (slow path), then
// re-check after acquiring Lock to handle races.
//
// Example selector: []string{"emmy", "node001", "cpu0"}
// Navigates: root → emmy → node001 → cpu0, creating levels as needed.
//
// Parameters:
//   - selector: Hierarchical path (consumed recursively, decreasing depth)
//   - nMetrics: Number of metric slots to allocate in new levels
//
// Returns:
//   - *Level: The target level (existing or newly created)
//
// Note: sync.Map may improve performance for high-concurrency writes, but current
// approach suffices for read-heavy workload.
func (l *Level) findLevelOrCreate(selector []string, nMetrics int) *Level {
	if len(selector) == 0 {
		return l
	}

	// Allow concurrent reads:
	l.lock.RLock()
	var child *Level
	var ok bool
	if l.children == nil {
		// Children map needs to be created...
		l.lock.RUnlock()
	} else {
		child, ok = l.children[selector[0]]
		l.lock.RUnlock()
		if ok {
			return child.findLevelOrCreate(selector[1:], nMetrics)
		}
	}

	// The level does not exist, take write lock for unique access:
	l.lock.Lock()
	// While this thread waited for the write lock, another thread
	// could have created the child node.
	if l.children != nil {
		child, ok = l.children[selector[0]]
		if ok {
			l.lock.Unlock()
			return child.findLevelOrCreate(selector[1:], nMetrics)
		}
	}

	child = &Level{
		metrics:  make([]*buffer, nMetrics),
		children: nil,
	}

	if l.children != nil {
		l.children[selector[0]] = child
	} else {
		l.children = map[string]*Level{selector[0]: child}
	}
	l.lock.Unlock()
	return child.findLevelOrCreate(selector[1:], nMetrics)
}

// collectPaths gathers all selector paths at the specified depth in the tree.
//
// Recursively traverses children, collecting paths when currentDepth+1 == targetDepth.
// Each path is a selector that can be used with findLevel() or findBuffers().
//
// Explicitly copies slices to avoid shared underlying arrays between siblings, preventing
// unintended mutations.
//
// Parameters:
//   - currentDepth: Depth of current level (0 = root)
//   - targetDepth:  Depth to collect paths from
//   - currentPath:  Path accumulated so far
//   - results:      Output slice (appended to)
//
// Example: collectPaths(0, 2, []string{}, &results) collects all 2-level paths
// like []string{"emmy", "node001"}, []string{"emmy", "node002"}, etc.
func (l *Level) collectPaths(currentDepth, targetDepth int, currentPath []string, results *[][]string) {
	l.lock.RLock()
	defer l.lock.RUnlock()

	for key, child := range l.children {
		if child == nil {
			continue
		}

		// We explicitly make a new slice and copy data to avoid sharing underlying arrays between siblings
		newPath := make([]string, len(currentPath))
		copy(newPath, currentPath)
		newPath = append(newPath, key)

		// Check depth, and just return if depth reached
		if currentDepth+1 == targetDepth {
			*results = append(*results, newPath)
		} else {
			child.collectPaths(currentDepth+1, targetDepth, newPath, results)
		}
	}
}

// free removes buffers older than the retention threshold from the entire subtree.
//
// Recursively frees buffers in this level's metrics and all child levels. Buffers
// with standard capacity (BufferCap) are returned to the pool. Called by the
// retention worker to enforce retention policies.
//
// Parameters:
//   - t: Retention threshold timestamp (Unix seconds)
//
// Returns:
//   - int:   Total number of buffers freed in this subtree
//   - error: Non-nil on failure (propagated from children)
func (l *Level) free(t int64) (int, error) {
	l.lock.Lock()
	defer l.lock.Unlock()

	n := 0
	for i, b := range l.metrics {
		if b != nil {
			delme, m := b.free(t)
			n += m
			if delme {
				if cap(b.data) != BufferCap {
					b.data = make([]schema.Float, 0, BufferCap)
				}
				b.lastUsed = time.Now().Unix()
				bufferPool.Put(b)
				l.metrics[i] = nil
			}
		}
	}

	for key, child := range l.children {
		m, err := child.free(t)
		n += m
		if err != nil {
			return n, err
		}
		if child.isEmpty() {
			delete(l.children, key)
		}
	}

	return n, nil
}

// isEmpty returns true if this level has no metrics and no children.
func (l *Level) isEmpty() bool {
	l.lock.RLock()
	defer l.lock.RUnlock()

	for _, b := range l.metrics {
		if b != nil {
			return false
		}
	}
	return len(l.children) == 0
}

// forceFree removes the oldest buffer from each metric chain in the subtree.
//
// Unlike free(), which removes based on time threshold, this unconditionally removes
// the oldest buffer in each chain. Used by MemoryUsageTracker when memory cap is
// exceeded and time-based retention is insufficient.
//
// Recursively processes current level's metrics and all child levels.
//
// Returns:
//   - int:   Total number of buffers freed in this subtree
//   - error: Non-nil on failure (propagated from children)
func (l *Level) forceFree() (int, error) {
	l.lock.Lock()
	defer l.lock.Unlock()

	n := 0

	// Iterate over metrics in the current level
	for i, b := range l.metrics {
		if b != nil {
			// Attempt to free the oldest buffer in this chain
			delme, freedCount := b.forceFreeOldest()
			n += freedCount

			// If delme is true, it means 'b' itself (the head) was the oldest
			// and needs to be removed from the slice.
			if delme {
				b.next = nil
				b.prev = nil
				if cap(b.data) != BufferCap {
					b.data = make([]schema.Float, 0, BufferCap)
				}
				b.lastUsed = time.Now().Unix()
				bufferPool.Put(b)
				l.metrics[i] = nil
			}
		}
	}

	// Recursively traverse children
	for _, child := range l.children {
		m, err := child.forceFree()
		n += m
		if err != nil {
			return n, err
		}
	}

	return n, nil
}

// sizeInBytes calculates the total memory usage of all buffers in the subtree.
//
// Recursively sums buffer data sizes (count of Float values × sizeof(Float)) across
// this level's metrics and all child levels. Used by MemoryUsageTracker to enforce
// memory cap limits.
//
// Returns:
//   - int64: Total bytes used by buffer data in this subtree
func (l *Level) sizeInBytes() int64 {
	l.lock.RLock()
	defer l.lock.RUnlock()
	size := int64(0)

	for _, b := range l.metrics {
		if b != nil {
			size += b.count() * int64(unsafe.Sizeof(schema.Float(0)))
			size += b.bufferCount() * int64(unsafe.Sizeof(buffer{}))
		}
	}

	for _, child := range l.children {
		size += child.sizeInBytes()
	}

	return size
}

// findLevel navigates to the level specified by selector, returning nil if not found.
//
// Read-only variant of findLevelOrCreate. Does not create missing levels.
// Recursively descends the tree following the selector path.
//
// Parameters:
//   - selector: Hierarchical path (e.g., []string{"emmy", "node001", "cpu0"})
//
// Returns:
//   - *Level: The target level, or nil if any component in the path does not exist
func (l *Level) findLevel(selector []string) *Level {
	if len(selector) == 0 {
		return l
	}

	l.lock.RLock()
	defer l.lock.RUnlock()

	lvl := l.children[selector[0]]
	if lvl == nil {
		return nil
	}

	return lvl.findLevel(selector[1:])
}

// findBuffers invokes callback on all buffers matching the selector pattern.
//
// Supports flexible selector patterns (from cc-lib/util.Selector):
//   - Exact match: Selector element with String set (e.g., "node001")
//   - Group match: Selector element with Group set (e.g., ["cpu0", "cpu2", "cpu4"])
//   - Wildcard:    Selector element with Any=true (matches all children)
//
// Empty selector (len==0) matches current level's buffer at 'offset' and recursively
// all descendant buffers at the same offset (used for aggregation queries).
//
// Parameters:
//   - selector: Pattern to match (consumed recursively)
//   - offset:   Metric index in metrics slice (from MetricConfig.offset)
//   - f:        Callback invoked on each matching buffer
//
// Returns:
//   - error: First error returned by callback, or nil if all succeeded
//
// Example:
//
//	// Find all cpu0 buffers across all hosts:
//	findBuffers([]Selector{{Any: true}, {String: "cpu0"}}, metricOffset, callback)
func (l *Level) findBuffers(selector util.Selector, offset int, f func(b *buffer) error) error {
	l.lock.RLock()
	defer l.lock.RUnlock()

	if len(selector) == 0 {
		b := l.metrics[offset]
		if b != nil {
			return f(b)
		}

		for _, lvl := range l.children {
			err := lvl.findBuffers(nil, offset, f)
			if err != nil {
				return err
			}
		}
		return nil
	}

	sel := selector[0]
	if len(sel.String) != 0 && l.children != nil {
		lvl, ok := l.children[sel.String]
		if ok {
			err := lvl.findBuffers(selector[1:], offset, f)
			if err != nil {
				return err
			}
		}
		return nil
	}

	if sel.Group != nil && l.children != nil {
		for _, key := range sel.Group {
			lvl, ok := l.children[key]
			if ok {
				err := lvl.findBuffers(selector[1:], offset, f)
				if err != nil {
					return err
				}
			}
		}
		return nil
	}

	if sel.Any && l.children != nil {
		for _, lvl := range l.children {
			if err := lvl.findBuffers(selector[1:], offset, f); err != nil {
				return err
			}
		}
		return nil
	}

	return nil
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package metricstore
     7→
     8→import (
     9→	"errors"
    10→	"math"
    11→
    12→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    13→	"github.com/ClusterCockpit/cc-lib/v2/util"
    14→)
    15→
    16→type Stats struct {
    17→	Samples int
    18→	Avg     schema.Float
    19→	Min     schema.Float
    20→	Max     schema.Float
    21→}
    22→
    23→func (b *buffer) stats(from, to int64) (Stats, int64, int64, error) {
    24→	if from < b.start {
    25→		if b.prev != nil {
    26→			return b.prev.stats(from, to)
    27→		}
    28→		from = b.start
    29→	}
    30→
    31→	// TODO: Check if b.closed and if so and the full buffer is queried,
    32→	// use b.statistics instead of iterating over the buffer.
    33→
    34→	samples := 0
    35→	sum, min, max := 0.0, math.MaxFloat32, -math.MaxFloat32
    36→
    37→	var t int64
    38→	for t = from; t < to; t += b.frequency {
    39→		idx := int((t - b.start) / b.frequency)
    40→		if idx >= cap(b.data) {
    41→			b = b.next
    42→			if b == nil {
    43→				break
    44→			}
    45→			idx = 0
    46→		}
    47→
    48→		if t < b.start || idx >= len(b.data) {
    49→			continue
    50→		}
    51→
    52→		xf := float64(b.data[idx])
    53→		if math.IsNaN(xf) {
    54→			continue
    55→		}
    56→
    57→		samples += 1
    58→		sum += xf
    59→		min = math.Min(min, xf)
    60→		max = math.Max(max, xf)
    61→	}
    62→
    63→	return Stats{
    64→		Samples: samples,
    65→		Avg:     schema.Float(sum) / schema.Float(samples),
    66→		Min:     schema.Float(min),
    67→		Max:     schema.Float(max),
    68→	}, from, t, nil
    69→}
    70→
    71→// Returns statistics for the requested metric on the selected node/level.
    72→// Data is aggregated to the selected level the same way as in `MemoryStore.Read`.
    73→// If `Stats.Samples` is zero, the statistics should not be considered as valid.
    74→func (m *MemoryStore) Stats(selector util.Selector, metric string, from, to int64) (*Stats, int64, int64, error) {
    75→	if from > to {
    76→		return nil, 0, 0, errors.New("invalid time range")
    77→	}
    78→
    79→	minfo, ok := m.Metrics[metric]
    80→	if !ok {
    81→		return nil, 0, 0, errors.New("unknown metric: " + metric)
    82→	}
    83→
    84→	n, samples := 0, 0
    85→	avg, min, max := schema.Float(0), math.MaxFloat32, -math.MaxFloat32
    86→	err := m.root.findBuffers(selector, minfo.offset, func(b *buffer) error {
    87→		stats, cfrom, cto, err := b.stats(from, to)
    88→		if err != nil {
    89→			return err
    90→		}
    91→
    92→		if n == 0 {
    93→			from, to = cfrom, cto
    94→		} else if from != cfrom || to != cto {
    95→			return ErrDataDoesNotAlign
    96→		}
    97→
    98→		samples += stats.Samples
    99→		avg += stats.Avg
   100→		min = math.Min(min, float64(stats.Min))
   101→		max = math.Max(max, float64(stats.Max))
   102→		n += 1
   103→		return nil
   104→	})
   105→	if err != nil {
   106→		return nil, 0, 0, err
   107→	}
   108→
   109→	if n == 0 {
   110→		return nil, 0, 0, ErrNoData
   111→	}
   112→
   113→	if minfo.Aggregation == AvgAggregation {
   114→		avg /= schema.Float(n)
   115→	} else if n > 1 && minfo.Aggregation != SumAggregation {
   116→		return nil, 0, 0, errors.New("invalid aggregation")
   117→	}
   118→
   119→	return &Stats{
   120→		Samples: samples,
   121→		Avg:     avg,
   122→		Min:     schema.Float(min),
   123→		Max:     schema.Float(max),
   124→	}, from, to, nil
   125→}
   126→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package metricstore

import (
	"errors"
	"math"

	"github.com/ClusterCockpit/cc-lib/v2/schema"
	"github.com/ClusterCockpit/cc-lib/v2/util"
)

type Stats struct {
	Samples int
	Avg     schema.Float
	Min     schema.Float
	Max     schema.Float
}

func (b *buffer) stats(from, to int64) (Stats, int64, int64, error) {
	if from < b.start {
		if b.prev != nil {
			return b.prev.stats(from, to)
		}
		from = b.start
	}

	// TODO: Check if b.closed and if so and the full buffer is queried,
	// use b.statistics instead of iterating over the buffer.

	samples := 0
	sum, min, max := 0.0, math.MaxFloat32, -math.MaxFloat32

	var t int64
	for t = from; t < to; t += b.frequency {
		idx := int((t - b.start) / b.frequency)
		if idx >= cap(b.data) {
			b = b.next
			if b == nil {
				break
			}
			idx = 0
		}

		if t < b.start || idx >= len(b.data) {
			continue
		}

		xf := float64(b.data[idx])
		if math.IsNaN(xf) {
			continue
		}

		samples += 1
		sum += xf
		min = math.Min(min, xf)
		max = math.Max(max, xf)
	}

	return Stats{
		Samples: samples,
		Avg:     schema.Float(sum) / schema.Float(samples),
		Min:     schema.Float(min),
		Max:     schema.Float(max),
	}, from, t, nil
}

// Returns statistics for the requested metric on the selected node/level.
// Data is aggregated to the selected level the same way as in `MemoryStore.Read`.
// If `Stats.Samples` is zero, the statistics should not be considered as valid.
func (m *MemoryStore) Stats(selector util.Selector, metric string, from, to int64) (*Stats, int64, int64, error) {
	if from > to {
		return nil, 0, 0, errors.New("invalid time range")
	}

	minfo, ok := m.Metrics[metric]
	if !ok {
		return nil, 0, 0, errors.New("unknown metric: " + metric)
	}

	n, samples := 0, 0
	avg, min, max := schema.Float(0), math.MaxFloat32, -math.MaxFloat32
	err := m.root.findBuffers(selector, minfo.offset, func(b *buffer) error {
		stats, cfrom, cto, err := b.stats(from, to)
		if err != nil {
			return err
		}

		if n == 0 {
			from, to = cfrom, cto
		} else if from != cfrom || to != cto {
			return ErrDataDoesNotAlign
		}

		samples += stats.Samples
		avg += stats.Avg
		min = math.Min(min, float64(stats.Min))
		max = math.Max(max, float64(stats.Max))
		n += 1
		return nil
	})
	if err != nil {
		return nil, 0, 0, err
	}

	if n == 0 {
		return nil, 0, 0, ErrNoData
	}

	if minfo.Aggregation == AvgAggregation {
		avg /= schema.Float(n)
	} else if n > 1 && minfo.Aggregation != SumAggregation {
		return nil, 0, 0, errors.New("invalid aggregation")
	}

	return &Stats{
		Samples: samples,
		Avg:     avg,
		Min:     schema.Float(min),
		Max:     schema.Float(max),
	}, from, to, nil
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstore provides an efficient in-memory time-series metric storage system
     7→// with support for hierarchical data organization, checkpointing, and archiving.
     8→//
     9→// The package organizes metrics in a tree structure (cluster → host → component) and
    10→// provides concurrent read/write access to metric data with configurable aggregation strategies.
    11→// Background goroutines handle periodic checkpointing (JSON or Avro format), archiving old data,
    12→// and enforcing retention policies.
    13→//
    14→// Key features:
    15→//   - In-memory metric storage with configurable retention
    16→//   - Hierarchical data organization (selectors)
    17→//   - Concurrent checkpoint/archive workers
    18→//   - Support for sum and average aggregation
    19→//   - NATS integration for metric ingestion
    20→package metricstore
    21→
    22→import (
    23→	"bytes"
    24→	"context"
    25→	"encoding/json"
    26→	"errors"
    27→	"fmt"
    28→	"runtime"
    29→	"runtime/debug"
    30→	"slices"
    31→	"sync"
    32→	"time"
    33→
    34→	"github.com/ClusterCockpit/cc-backend/internal/config"
    35→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    36→	"github.com/ClusterCockpit/cc-lib/v2/resampler"
    37→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    38→	"github.com/ClusterCockpit/cc-lib/v2/util"
    39→)
    40→
    41→// GlobalState holds the global state for the metric store with thread-safe access.
    42→type GlobalState struct {
    43→	mu                sync.RWMutex
    44→	lastRetentionTime int64
    45→	selectorsExcluded bool
    46→}
    47→
    48→var (
    49→	singleton  sync.Once
    50→	msInstance *MemoryStore
    51→	// shutdownFunc stores the context cancellation function created in Init
    52→	// and is called during Shutdown to cancel all background goroutines
    53→	shutdownFunc   context.CancelFunc
    54→	shutdownFuncMu sync.Mutex // Protects shutdownFunc from concurrent access
    55→	// Create a global instance
    56→	state = &GlobalState{}
    57→)
    58→
    59→// NodeProvider provides information about nodes currently in use by running jobs.
    60→//
    61→// This interface allows metricstore to query job information without directly
    62→// depending on the repository package, breaking the import cycle.
    63→//
    64→// Implementations should return nodes that are actively processing jobs started
    65→// before the given timestamp. These nodes will be excluded from retention-based
    66→// garbage collection to prevent data loss for jobs that are still running or
    67→// recently completed.
    68→type NodeProvider interface {
    69→	// GetUsedNodes returns a map of cluster names to sorted lists of unique hostnames
    70→	// that are currently in use by jobs that started before the given timestamp.
    71→	//
    72→	// Parameters:
    73→	//   - ts: Unix timestamp threshold - returns nodes with jobs started before this time
    74→	//
    75→	// Returns:
    76→	//   - Map of cluster names to lists of node hostnames that should be excluded from garbage collection
    77→	//   - Error if the query fails
    78→	GetUsedNodes(ts int64) (map[string][]string, error)
    79→}
    80→
    81→// Metric represents a single metric data point to be written to the store.
    82→type Metric struct {
    83→	Name  string
    84→	Value schema.Float
    85→	// MetricConfig contains frequency and aggregation settings for this metric.
    86→	// If Frequency is 0, configuration will be looked up from MemoryStore.Metrics during Write().
    87→	MetricConfig MetricConfig
    88→}
    89→
    90→// MemoryStore is the main in-memory time-series metric storage implementation.
    91→//
    92→// It organizes metrics in a hierarchical tree structure where each level represents
    93→// a component of the system hierarchy (e.g., cluster → host → CPU). Each level can
    94→// store multiple metrics as time-series buffers.
    95→//
    96→// The store is initialized as a singleton via InitMetrics() and accessed via GetMemoryStore().
    97→// All public methods are safe for concurrent use.
    98→type MemoryStore struct {
    99→	Metrics      map[string]MetricConfig
   100→	root         Level
   101→	nodeProvider NodeProvider
   102→}
   103→
   104→// Init initializes the metric store from configuration and starts background workers.
   105→//
   106→// This function must be called exactly once before any other metricstore operations.
   107→// It performs the following initialization steps:
   108→//  1. Validates and decodes the metric store configuration
   109→//  2. Configures worker pool size (defaults to NumCPU/2+1, max 10)
   110→//  3. Loads metric configurations from all registered clusters
   111→//  4. Restores checkpoints within the retention window
   112→//  5. Starts background workers for retention, checkpointing, archiving, and monitoring
   113→//  6. Optionally subscribes to NATS for real-time metric ingestion
   114→//
   115→// Parameters:
   116→//   - rawConfig: JSON configuration for the metric store (see MetricStoreConfig)
   117→//   - wg: WaitGroup that will be incremented for each background goroutine started
   118→//
   119→// The function will call cclog.Fatal on critical errors during initialization.
   120→// Use Shutdown() to cleanly stop all background workers started by Init().
   121→//
   122→// Note: Signal handling must be implemented by the caller. Call Shutdown() when
   123→// receiving termination signals to ensure checkpoint data is persisted.
   124→func Init(rawConfig json.RawMessage, metrics map[string]MetricConfig, wg *sync.WaitGroup) {
   125→	startupTime := time.Now()
   126→
   127→	if rawConfig != nil {
   128→		config.Validate(configSchema, rawConfig)
   129→		dec := json.NewDecoder(bytes.NewReader(rawConfig))
   130→		dec.DisallowUnknownFields()
   131→		if err := dec.Decode(&Keys); err != nil {
   132→			cclog.Abortf("[METRICSTORE]> Metric Store Config Init: Could not decode config file '%s'.\nError: %s\n", rawConfig, err.Error())
   133→		}
   134→	}
   135→
   136→	// Set NumWorkers from config or use default
   137→	if Keys.NumWorkers <= 0 {
   138→		Keys.NumWorkers = min(runtime.NumCPU()/2+1, DefaultMaxWorkers)
   139→	}
   140→	cclog.Debugf("[METRICSTORE]> Using %d workers for checkpoint/archive operations\n", Keys.NumWorkers)
   141→
   142→	// Pass the config.MetricStoreKeys
   143→	InitMetrics(metrics)
   144→
   145→	ms := GetMemoryStore()
   146→
   147→	d, err := time.ParseDuration(Keys.RetentionInMemory)
   148→	if err != nil {
   149→		cclog.Fatal(err)
   150→	}
   151→
   152→	restoreFrom := startupTime.Add(-d)
   153→	cclog.Infof("[METRICSTORE]> Loading checkpoints newer than %s\n", restoreFrom.Format(time.RFC3339))
   154→	files, err := ms.FromCheckpointFiles(Keys.Checkpoints.RootDir, restoreFrom.Unix())
   155→	loadedData := ms.SizeInBytes() / 1024 / 1024 // In MB
   156→	if err != nil {
   157→		cclog.Fatalf("[METRICSTORE]> Loading checkpoints failed: %s\n", err.Error())
   158→	} else {
   159→		cclog.Infof("[METRICSTORE]> Checkpoints loaded (%d files, %d MB, that took %fs)\n", files, loadedData, time.Since(startupTime).Seconds())
   160→	}
   161→
   162→	// Try to use less memory by forcing a GC run here and then
   163→	// lowering the target percentage. The default of 100 means
   164→	// that only once the ratio of new allocations execeds the
   165→	// previously active heap, a GC is triggered.
   166→	// Forcing a GC here will set the "previously active heap"
   167→	// to a minumum.
   168→	// runtime.GC()
   169→
   170→	ctx, shutdown := context.WithCancel(context.Background())
   171→
   172→	Retention(wg, ctx)
   173→	Checkpointing(wg, ctx)
   174→	CleanUp(wg, ctx)
   175→	WALStaging(wg, ctx)
   176→	MemoryUsageTracker(wg, ctx)
   177→
   178→	// Note: Signal handling has been removed from this function.
   179→	// The caller is responsible for handling shutdown signals and calling
   180→	// the shutdown() function when appropriate.
   181→	// Store the shutdown function for later use by Shutdown()
   182→	shutdownFuncMu.Lock()
   183→	shutdownFunc = shutdown
   184→	shutdownFuncMu.Unlock()
   185→
   186→	if Keys.Subscriptions != nil {
   187→		err = ReceiveNats(ms, 1, ctx)
   188→		if err != nil {
   189→			cclog.Fatal(err)
   190→		}
   191→	}
   192→}
   193→
   194→// InitMetrics initializes the singleton MemoryStore instance with the given metric configurations.
   195→//
   196→// This function must be called before GetMemoryStore() and can only be called once due to
   197→// the singleton pattern. It assigns each metric an internal offset for efficient buffer indexing.
   198→//
   199→// Parameters:
   200→//   - metrics: Map of metric names to their configurations (frequency and aggregation strategy)
   201→//
   202→// Panics if any metric has Frequency == 0, which indicates an invalid configuration.
   203→//
   204→// After this call, the global msInstance is ready for use via GetMemoryStore().
   205→func InitMetrics(metrics map[string]MetricConfig) {
   206→	singleton.Do(func() {
   207→		offset := 0
   208→		for key, cfg := range metrics {
   209→			if cfg.Frequency == 0 {
   210→				panic("[METRICSTORE]> invalid frequency")
   211→			}
   212→
   213→			metrics[key] = MetricConfig{
   214→				Frequency:   cfg.Frequency,
   215→				Aggregation: cfg.Aggregation,
   216→				offset:      offset,
   217→			}
   218→			offset += 1
   219→		}
   220→
   221→		msInstance = &MemoryStore{
   222→			root: Level{
   223→				metrics:  make([]*buffer, len(metrics)),
   224→				children: make(map[string]*Level),
   225→			},
   226→			Metrics: metrics,
   227→		}
   228→	})
   229→}
   230→
   231→// GetMemoryStore returns the singleton MemoryStore instance.
   232→//
   233→// Returns the initialized MemoryStore singleton. Calls cclog.Fatal if InitMetrics() was not called first.
   234→//
   235→// This function is safe for concurrent use after initialization.
   236→func GetMemoryStore() *MemoryStore {
   237→	if msInstance == nil {
   238→		cclog.Warnf("[METRICSTORE]> MemoryStore not initialized!")
   239→	}
   240→
   241→	return msInstance
   242→}
   243→
   244→func (ms *MemoryStore) GetMetricFrequency(metricName string) (int64, error) {
   245→	if metric, ok := ms.Metrics[metricName]; ok {
   246→		return metric.Frequency, nil
   247→	}
   248→	return 0, fmt.Errorf("[METRICSTORE]> metric %s not found", metricName)
   249→}
   250→
   251→// SetNodeProvider sets the NodeProvider implementation for the MemoryStore.
   252→// This must be called during initialization to provide job state information
   253→// for selective buffer retention during Free operations.
   254→// If not set, the Free function will fall back to freeing all buffers.
   255→func (ms *MemoryStore) SetNodeProvider(provider NodeProvider) {
   256→	ms.nodeProvider = provider
   257→}
   258→
   259→// Shutdown performs a graceful shutdown of the metric store.
   260→//
   261→// This function cancels all background goroutines started by Init() and writes
   262→// a final checkpoint to disk before returning. It should be called when the
   263→// application receives a termination signal.
   264→//
   265→// The function will:
   266→//  1. Cancel the context to stop all background workers
   267→//  2. Close the WAL messages channel if using WAL format
   268→//  3. Write a final checkpoint to preserve in-memory data
   269→//  4. Log any errors encountered during shutdown
   270→//
   271→// Note: This function blocks until the final checkpoint is written.
   272→func Shutdown() {
   273→	shutdownFuncMu.Lock()
   274→	defer shutdownFuncMu.Unlock()
   275→	if shutdownFunc != nil {
   276→		shutdownFunc()
   277→	}
   278→
   279→	if Keys.Checkpoints.FileFormat == "wal" {
   280→		close(WALMessages)
   281→	}
   282→
   283→	cclog.Infof("[METRICSTORE]> Writing to '%s'...\n", Keys.Checkpoints.RootDir)
   284→	var files int
   285→	var err error
   286→
   287→	ms := GetMemoryStore()
   288→
   289→	lastCheckpointMu.Lock()
   290→	from := lastCheckpoint
   291→	lastCheckpointMu.Unlock()
   292→
   293→	if Keys.Checkpoints.FileFormat == "wal" {
   294→		var hostDirs []string
   295→		files, hostDirs, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
   296→		if err == nil {
   297→			RotateWALFilesAfterShutdown(hostDirs)
   298→		}
   299→	} else {
   300→		files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
   301→	}
   302→
   303→	if err != nil {
   304→		cclog.Errorf("[METRICSTORE]> Writing checkpoint failed: %s\n", err.Error())
   305→	}
   306→	cclog.Infof("[METRICSTORE]> Done! (%d files written)\n", files)
   307→}
   308→
   309→// Retention starts a background goroutine that periodically frees old metric data.
   310→//
   311→// This worker runs at half the retention interval and calls Free() to remove buffers
   312→// older than the configured retention time. It respects the NodeProvider to preserve
   313→// data for nodes with active jobs.
   314→//
   315→// Parameters:
   316→//   - wg: WaitGroup to signal completion when context is cancelled
   317→//   - ctx: Context for cancellation signal
   318→//
   319→// The goroutine exits when ctx is cancelled.
   320→func Retention(wg *sync.WaitGroup, ctx context.Context) {
   321→	ms := GetMemoryStore()
   322→
   323→	wg.Go(func() {
   324→		d, err := time.ParseDuration(Keys.RetentionInMemory)
   325→		if err != nil {
   326→			cclog.Fatal(err)
   327→		}
   328→		if d <= 0 {
   329→			return
   330→		}
   331→
   332→		tickInterval := d / 2
   333→		if tickInterval <= 0 {
   334→			return
   335→		}
   336→		ticker := time.NewTicker(tickInterval)
   337→		defer ticker.Stop()
   338→
   339→		for {
   340→			select {
   341→			case <-ctx.Done():
   342→				return
   343→			case <-ticker.C:
   344→				state.mu.Lock()
   345→
   346→				t := time.Now().Add(-d)
   347→
   348→				state.lastRetentionTime = t.Unix()
   349→
   350→				cclog.Infof("[METRICSTORE]> start freeing buffers (older than %s)...\n", t.Format(time.RFC3339))
   351→
   352→				freed, err := Free(ms, t)
   353→				if err != nil {
   354→					cclog.Errorf("[METRICSTORE]> freeing up buffers failed: %s\n", err.Error())
   355→				} else {
   356→					cclog.Infof("[METRICSTORE]> done: %d buffers freed\n", freed)
   357→				}
   358→
   359→				state.mu.Unlock()
   360→
   361→				// Clean up the buffer pool
   362→				bufferPool.Clean(state.lastRetentionTime)
   363→			}
   364→		}
   365→	})
   366→}
   367→
   368→// MemoryUsageTracker starts a background goroutine that monitors memory usage.
   369→//
   370→// This worker checks actual process memory usage (via runtime.MemStats) periodically
   371→// and force-frees buffers if memory exceeds the configured cap. It uses FreeOSMemory()
   372→// to return memory to the OS after freeing buffers, avoiding aggressive GC that causes
   373→// performance issues.
   374→//
   375→// The tracker logs both actual memory usage (heap allocated) and metric data size for
   376→// visibility into memory overhead from Go runtime structures and allocations.
   377→//
   378→// Parameters:
   379→//   - wg: WaitGroup to signal completion when context is cancelled
   380→//   - ctx: Context for cancellation signal
   381→//
   382→// The goroutine exits when ctx is cancelled.
   383→func MemoryUsageTracker(wg *sync.WaitGroup, ctx context.Context) {
   384→	ms := GetMemoryStore()
   385→
   386→	wg.Go(func() {
   387→		normalInterval := DefaultMemoryUsageTrackerInterval
   388→		fastInterval := 30 * time.Second
   389→
   390→		if normalInterval <= 0 {
   391→			return
   392→		}
   393→
   394→		ticker := time.NewTicker(normalInterval)
   395→		defer ticker.Stop()
   396→		currentInterval := normalInterval
   397→
   398→		for {
   399→			select {
   400→			case <-ctx.Done():
   401→				return
   402→			case <-ticker.C:
   403→				var mem runtime.MemStats
   404→				runtime.ReadMemStats(&mem)
   405→				actualMemoryGB := float64(mem.Alloc) / 1e9
   406→				metricDataGB := ms.SizeInGB()
   407→				cclog.Infof("[METRICSTORE]> memory usage: %.2f GB actual (%.2f GB metric data)", actualMemoryGB, metricDataGB)
   408→
   409→				freedExcluded := 0
   410→				freedEmergency := 0
   411→				var err error
   412→
   413→				state.mu.RLock()
   414→				lastRetention := state.lastRetentionTime
   415→				selectorsExcluded := state.selectorsExcluded
   416→				state.mu.RUnlock()
   417→
   418→				if lastRetention != 0 && selectorsExcluded {
   419→					freedExcluded, err = ms.Free(nil, lastRetention)
   420→					if err != nil {
   421→						cclog.Errorf("[METRICSTORE]> error while force-freeing the excluded buffers: %s", err)
   422→					}
   423→
   424→					if freedExcluded > 0 {
   425→						debug.FreeOSMemory()
   426→						cclog.Infof("[METRICSTORE]> done: %d excluded buffers force-freed", freedExcluded)
   427→					}
   428→				}
   429→
   430→				runtime.ReadMemStats(&mem)
   431→				actualMemoryGB = float64(mem.Alloc) / 1e9
   432→
   433→				if actualMemoryGB > float64(Keys.MemoryCap) {
   434→					cclog.Warnf("[METRICSTORE]> memory usage %.2f GB exceeds cap %d GB, starting emergency buffer freeing", actualMemoryGB, Keys.MemoryCap)
   435→
   436→					// Use progressive time-based Free with increasing threshold
   437→					// instead of ForceFree loop — fewer tree traversals, more effective
   438→					d, parseErr := time.ParseDuration(Keys.RetentionInMemory)
   439→					if parseErr != nil {
   440→						cclog.Errorf("[METRICSTORE]> cannot parse retention duration: %s", parseErr)
   441→					} else {
   442→						thresholds := []float64{0.75, 0.5, 0.25}
   443→						for _, fraction := range thresholds {
   444→							threshold := time.Now().Add(-time.Duration(float64(d) * fraction))
   445→							freed, freeErr := ms.Free(nil, threshold.Unix())
   446→							if freeErr != nil {
   447→								cclog.Errorf("[METRICSTORE]> error while freeing buffers at %.0f%% retention: %s", fraction*100, freeErr)
   448→							}
   449→							freedEmergency += freed
   450→
   451→							bufferPool.Clear()
   452→							runtime.GC()
   453→							runtime.ReadMemStats(&mem)
   454→							actualMemoryGB = float64(mem.Alloc) / 1e9
   455→
   456→							if actualMemoryGB < float64(Keys.MemoryCap) {
   457→								break
   458→							}
   459→						}
   460→					}
   461→
   462→					bufferPool.Clear()
   463→					debug.FreeOSMemory()
   464→
   465→					runtime.ReadMemStats(&mem)
   466→					actualMemoryGB = float64(mem.Alloc) / 1e9
   467→
   468→					if actualMemoryGB >= float64(Keys.MemoryCap) {
   469→						cclog.Errorf("[METRICSTORE]> after emergency frees (%d buffers), memory usage %.2f GB still at/above cap %d GB", freedEmergency, actualMemoryGB, Keys.MemoryCap)
   470→					} else {
   471→						cclog.Infof("[METRICSTORE]> emergency freeing complete: %d buffers freed, memory now %.2f GB", freedEmergency, actualMemoryGB)
   472→					}
   473→				}
   474→
   475→				// Adaptive ticker: check more frequently when memory is high
   476→				memoryRatio := actualMemoryGB / float64(Keys.MemoryCap)
   477→				if memoryRatio > 0.8 && currentInterval != fastInterval {
   478→					ticker.Reset(fastInterval)
   479→					currentInterval = fastInterval
   480→					cclog.Infof("[METRICSTORE]> memory at %.0f%% of cap, switching to fast check interval (30s)", memoryRatio*100)
   481→				} else if memoryRatio <= 0.8 && currentInterval != normalInterval {
   482→					ticker.Reset(normalInterval)
   483→					currentInterval = normalInterval
   484→					cclog.Infof("[METRICSTORE]> memory at %.0f%% of cap, switching to normal check interval", memoryRatio*100)
   485→				}
   486→			}
   487→		}
   488→	})
   489→}
   490→
   491→// Free removes metric data older than the given time while preserving data for active nodes.
   492→//
   493→// This function implements intelligent retention by consulting the NodeProvider (if configured)
   494→// to determine which nodes are currently in use by running jobs. Data for these nodes is
   495→// preserved even if older than the retention time.
   496→//
   497→// Parameters:
   498→//   - ms: The MemoryStore instance
   499→//   - t: Time threshold - buffers with data older than this will be freed
   500→//
   501→// Returns:
   502→//   - Number of buffers freed
   503→//   - Error if NodeProvider query fails
   504→//
   505→// Behavior:
   506→//   - If no NodeProvider is set: frees all buffers older than t
   507→//   - If NodeProvider returns empty map: frees all buffers older than t
   508→//   - Otherwise: preserves buffers for nodes returned by GetUsedNodes(), frees others
   509→func Free(ms *MemoryStore, t time.Time) (int, error) {
   510→	// If no NodeProvider is configured, free all buffers older than t
   511→	if ms.nodeProvider == nil {
   512→		return ms.Free(nil, t.Unix())
   513→	}
   514→
   515→	excludeSelectors, err := ms.nodeProvider.GetUsedNodes(t.Unix())
   516→	if err != nil {
   517→		return 0, err
   518→	}
   519→
   520→	switch lenMap := len(excludeSelectors); lenMap {
   521→
   522→	// If the length of the map returned by GetUsedNodes() is 0,
   523→	// then use default Free method with nil selector
   524→	case 0:
   525→		state.selectorsExcluded = false
   526→		return ms.Free(nil, t.Unix())
   527→
   528→	// Else formulate selectors, exclude those from the map
   529→	// and free the rest of the selectors
   530→	default:
   531→		state.selectorsExcluded = true
   532→		selectors := GetSelectors(ms, excludeSelectors)
   533→		return FreeSelected(ms, selectors, t)
   534→	}
   535→}
   536→
   537→// FreeSelected frees buffers for specific selectors while preserving others.
   538→//
   539→// This function is used when we want to retain some specific nodes beyond the retention time.
   540→// It iterates through the provided selectors and frees their associated buffers.
   541→//
   542→// Parameters:
   543→//   - ms: The MemoryStore instance
   544→//   - selectors: List of selector paths to free (e.g., [["cluster1", "node1"], ["cluster2", "node2"]])
   545→//   - t: Time threshold for freeing buffers
   546→//
   547→// Returns the total number of buffers freed and any error encountered.
   548→func FreeSelected(ms *MemoryStore, selectors [][]string, t time.Time) (int, error) {
   549→	freed := 0
   550→
   551→	for _, selector := range selectors {
   552→
   553→		freedBuffers, err := ms.Free(selector, t.Unix())
   554→		if err != nil {
   555→			cclog.Errorf("error while freeing selected buffers: %#v", err)
   556→		}
   557→		freed += freedBuffers
   558→
   559→	}
   560→
   561→	return freed, nil
   562→}
   563→
   564→// GetSelectors returns all selectors at depth 2 (cluster/node level) that are NOT in the exclusion map.
   565→//
   566→// This function generates a list of selectors whose buffers should be freed by excluding
   567→// selectors that correspond to nodes currently in use by running jobs.
   568→//
   569→// Parameters:
   570→//   - ms: The MemoryStore instance
   571→//   - excludeSelectors: Map of cluster names to node hostnames that should NOT be freed
   572→//
   573→// Returns a list of selectors ([]string paths) that can be safely freed.
   574→//
   575→// Example:
   576→//
   577→//	If the tree has paths ["emmy", "node001"] and ["emmy", "node002"],
   578→//	and excludeSelectors contains {"emmy": ["node001"]},
   579→//	then only [["emmy", "node002"]] is returned.
   580→func GetSelectors(ms *MemoryStore, excludeSelectors map[string][]string) [][]string {
   581→	allSelectors := ms.GetPaths(2)
   582→
   583→	filteredSelectors := make([][]string, 0, len(allSelectors))
   584→
   585→	for _, path := range allSelectors {
   586→		if len(path) < 2 {
   587→			continue
   588→		}
   589→
   590→		key := path[0]   // The "Key" (Level 1)
   591→		value := path[1] // The "Value" (Level 2)
   592→
   593→		exclude := false
   594→
   595→		// Check if the key exists in our exclusion map
   596→		if excludedValues, exists := excludeSelectors[key]; exists {
   597→			// The key exists, now check if the specific value is in the exclusion list
   598→			if slices.Contains(excludedValues, value) {
   599→				exclude = true
   600→			}
   601→		}
   602→
   603→		if !exclude {
   604→			filteredSelectors = append(filteredSelectors, path)
   605→		}
   606→	}
   607→
   608→	return filteredSelectors
   609→}
   610→
   611→// GetPaths returns a list of lists (paths) to the specified depth.
   612→func (ms *MemoryStore) GetPaths(targetDepth int) [][]string {
   613→	var results [][]string
   614→
   615→	// Start recursion. Initial path is empty.
   616→	// We treat Root as depth 0.
   617→	ms.root.collectPaths(0, targetDepth, []string{}, &results)
   618→
   619→	return results
   620→}
   621→
   622→// Write all values in `metrics` to the level specified by `selector` for time `ts`.
   623→// Look at `findLevelOrCreate` for how selectors work.
   624→func (m *MemoryStore) Write(selector []string, ts int64, metrics []Metric) error {
   625→	var ok bool
   626→	for i, metric := range metrics {
   627→		if metric.MetricConfig.Frequency == 0 {
   628→			metric.MetricConfig, ok = m.Metrics[metric.Name]
   629→			if !ok {
   630→				cclog.Debugf("[METRICSTORE]> Unknown metric '%s' in Write() - skipping", metric.Name)
   631→				metric.MetricConfig.Frequency = 0
   632→			}
   633→			metrics[i] = metric
   634→		}
   635→	}
   636→
   637→	return m.WriteToLevel(&m.root, selector, ts, metrics)
   638→}
   639→
   640→func (m *MemoryStore) GetLevel(selector []string) *Level {
   641→	return m.root.findLevelOrCreate(selector, len(m.Metrics))
   642→}
   643→
   644→// WriteToLevel assumes that `minfo` in `metrics` is filled in
   645→func (m *MemoryStore) WriteToLevel(l *Level, selector []string, ts int64, metrics []Metric) error {
   646→	l = l.findLevelOrCreate(selector, len(m.Metrics))
   647→	l.lock.Lock()
   648→	defer l.lock.Unlock()
   649→
   650→	for _, metric := range metrics {
   651→		if metric.MetricConfig.Frequency == 0 {
   652→			continue
   653→		}
   654→
   655→		b := l.metrics[metric.MetricConfig.offset]
   656→		if b == nil {
   657→			// First write to this metric and level
   658→			b = newBuffer(ts, metric.MetricConfig.Frequency)
   659→			l.metrics[metric.MetricConfig.offset] = b
   660→		}
   661→
   662→		nb, err := b.write(ts, metric.Value)
   663→		if err != nil {
   664→			return err
   665→		}
   666→
   667→		// Last write created a new buffer...
   668→		if b != nb {
   669→			l.metrics[metric.MetricConfig.offset] = nb
   670→		}
   671→	}
   672→	return nil
   673→}
   674→
   675→// Read returns all values for metric `metric` from `from` to `to` for the selected level(s).
   676→// If the level does not hold the metric itself, the data will be aggregated recursively from the children.
   677→// The second and third return value are the actual from/to for the data. Those can be different from
   678→// the range asked for if no data was available.
   679→func (m *MemoryStore) Read(selector util.Selector, metric string, from, to, resolution int64) ([]schema.Float, int64, int64, int64, error) {
   680→	if from > to {
   681→		return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid time range")
   682→	}
   683→
   684→	minfo, ok := m.Metrics[metric]
   685→	if !ok {
   686→		return nil, 0, 0, 0, errors.New("[METRICSTORE]> unknown metric: " + metric)
   687→	}
   688→
   689→	n, data := 0, make([]schema.Float, (to-from)/minfo.Frequency+1)
   690→
   691→	err := m.root.findBuffers(selector, minfo.offset, func(b *buffer) error {
   692→		cdata, cfrom, cto, err := b.read(from, to, data)
   693→		if err != nil {
   694→			return err
   695→		}
   696→
   697→		if n == 0 {
   698→			from, to = cfrom, cto
   699→		} else if from != cfrom || to != cto || len(data) != len(cdata) {
   700→			missingfront, missingback := int((from-cfrom)/minfo.Frequency), int((to-cto)/minfo.Frequency)
   701→			if missingfront != 0 {
   702→				return ErrDataDoesNotAlign
   703→			}
   704→
   705→			newlen := len(cdata) - missingback
   706→			if newlen < 1 {
   707→				return ErrDataDoesNotAlign
   708→			}
   709→			cdata = cdata[0:newlen]
   710→			if len(cdata) != len(data) {
   711→				return ErrDataDoesNotAlign
   712→			}
   713→
   714→			from, to = cfrom, cto
   715→		}
   716→
   717→		data = cdata
   718→		n += 1
   719→		return nil
   720→	})
   721→
   722→	if err != nil {
   723→		return nil, 0, 0, 0, err
   724→	} else if n == 0 {
   725→		return nil, 0, 0, 0, ErrNoHostOrMetric
   726→	} else if n > 1 {
   727→		if minfo.Aggregation == AvgAggregation {
   728→			normalize := 1. / schema.Float(n)
   729→			for i := 0; i < len(data); i++ {
   730→				data[i] *= normalize
   731→			}
   732→		} else if minfo.Aggregation != SumAggregation {
   733→			return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid aggregation")
   734→		}
   735→	}
   736→
   737→	data, resolution, err = resampler.LargestTriangleThreeBucket(data, minfo.Frequency, resolution)
   738→	if err != nil {
   739→		return nil, 0, 0, 0, err
   740→	}
   741→
   742→	return data, from, to, resolution, nil
   743→}
   744→
   745→// Free releases all buffers for the selected level and all its children that
   746→// contain only values older than `t`.
   747→func (m *MemoryStore) Free(selector []string, t int64) (int, error) {
   748→	return m.GetLevel(selector).free(t)
   749→}
   750→
   751→// ForceFree unconditionally removes the oldest buffer from each metric chain.
   752→func (m *MemoryStore) ForceFree() (int, error) {
   753→	return m.GetLevel(nil).forceFree()
   754→}
   755→
   756→func (m *MemoryStore) FreeAll() error {
   757→	for k := range m.root.children {
   758→		delete(m.root.children, k)
   759→	}
   760→
   761→	return nil
   762→}
   763→
   764→func (m *MemoryStore) SizeInBytes() int64 {
   765→	return m.root.sizeInBytes()
   766→}
   767→
   768→func (m *MemoryStore) SizeInGB() float64 {
   769→	return float64(m.root.sizeInBytes()) / 1e9
   770→}
   771→
   772→// ListChildren , given a selector, returns a list of all children of the level
   773→// selected.
   774→func (m *MemoryStore) ListChildren(selector []string) []string {
   775→	lvl := &m.root
   776→	for lvl != nil && len(selector) != 0 {
   777→		lvl.lock.RLock()
   778→		next := lvl.children[selector[0]]
   779→		lvl.lock.RUnlock()
   780→		lvl = next
   781→		selector = selector[1:]
   782→	}
   783→
   784→	if lvl == nil {
   785→		return nil
   786→	}
   787→
   788→	lvl.lock.RLock()
   789→	defer lvl.lock.RUnlock()
   790→
   791→	children := make([]string, 0, len(lvl.children))
   792→	for child := range lvl.children {
   793→		children = append(children, child)
   794→	}
   795→
   796→	return children
   797→}
   798→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package metricstore provides an efficient in-memory time-series metric storage system
// with support for hierarchical data organization, checkpointing, and archiving.
//
// The package organizes metrics in a tree structure (cluster → host → component) and
// provides concurrent read/write access to metric data with configurable aggregation strategies.
// Background goroutines handle periodic checkpointing (JSON or Avro format), archiving old data,
// and enforcing retention policies.
//
// Key features:
//   - In-memory metric storage with configurable retention
//   - Hierarchical data organization (selectors)
//   - Concurrent checkpoint/archive workers
//   - Support for sum and average aggregation
//   - NATS integration for metric ingestion
package metricstore

import (
	"bytes"
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"runtime"
	"runtime/debug"
	"slices"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/config"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/resampler"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	"github.com/ClusterCockpit/cc-lib/v2/util"
)

// GlobalState holds the global state for the metric store with thread-safe access.
type GlobalState struct {
	mu                sync.RWMutex
	lastRetentionTime int64
	selectorsExcluded bool
}

var (
	singleton  sync.Once
	msInstance *MemoryStore
	// shutdownFunc stores the context cancellation function created in Init
	// and is called during Shutdown to cancel all background goroutines
	shutdownFunc   context.CancelFunc
	shutdownFuncMu sync.Mutex // Protects shutdownFunc from concurrent access
	// Create a global instance
	state = &GlobalState{}
)

// NodeProvider provides information about nodes currently in use by running jobs.
//
// This interface allows metricstore to query job information without directly
// depending on the repository package, breaking the import cycle.
//
// Implementations should return nodes that are actively processing jobs started
// before the given timestamp. These nodes will be excluded from retention-based
// garbage collection to prevent data loss for jobs that are still running or
// recently completed.
type NodeProvider interface {
	// GetUsedNodes returns a map of cluster names to sorted lists of unique hostnames
	// that are currently in use by jobs that started before the given timestamp.
	//
	// Parameters:
	//   - ts: Unix timestamp threshold - returns nodes with jobs started before this time
	//
	// Returns:
	//   - Map of cluster names to lists of node hostnames that should be excluded from garbage collection
	//   - Error if the query fails
	GetUsedNodes(ts int64) (map[string][]string, error)
}

// Metric represents a single metric data point to be written to the store.
type Metric struct {
	Name  string
	Value schema.Float
	// MetricConfig contains frequency and aggregation settings for this metric.
	// If Frequency is 0, configuration will be looked up from MemoryStore.Metrics during Write().
	MetricConfig MetricConfig
}

// MemoryStore is the main in-memory time-series metric storage implementation.
//
// It organizes metrics in a hierarchical tree structure where each level represents
// a component of the system hierarchy (e.g., cluster → host → CPU). Each level can
// store multiple metrics as time-series buffers.
//
// The store is initialized as a singleton via InitMetrics() and accessed via GetMemoryStore().
// All public methods are safe for concurrent use.
type MemoryStore struct {
	Metrics      map[string]MetricConfig
	root         Level
	nodeProvider NodeProvider
}

// Init initializes the metric store from configuration and starts background workers.
//
// This function must be called exactly once before any other metricstore operations.
// It performs the following initialization steps:
//  1. Validates and decodes the metric store configuration
//  2. Configures worker pool size (defaults to NumCPU/2+1, max 10)
//  3. Loads metric configurations from all registered clusters
//  4. Restores checkpoints within the retention window
//  5. Starts background workers for retention, checkpointing, archiving, and monitoring
//  6. Optionally subscribes to NATS for real-time metric ingestion
//
// Parameters:
//   - rawConfig: JSON configuration for the metric store (see MetricStoreConfig)
//   - wg: WaitGroup that will be incremented for each background goroutine started
//
// The function will call cclog.Fatal on critical errors during initialization.
// Use Shutdown() to cleanly stop all background workers started by Init().
//
// Note: Signal handling must be implemented by the caller. Call Shutdown() when
// receiving termination signals to ensure checkpoint data is persisted.
func Init(rawConfig json.RawMessage, metrics map[string]MetricConfig, wg *sync.WaitGroup) {
	startupTime := time.Now()

	if rawConfig != nil {
		config.Validate(configSchema, rawConfig)
		dec := json.NewDecoder(bytes.NewReader(rawConfig))
		dec.DisallowUnknownFields()
		if err := dec.Decode(&Keys); err != nil {
			cclog.Abortf("[METRICSTORE]> Metric Store Config Init: Could not decode config file '%s'.\nError: %s\n", rawConfig, err.Error())
		}
	}

	// Set NumWorkers from config or use default
	if Keys.NumWorkers <= 0 {
		Keys.NumWorkers = min(runtime.NumCPU()/2+1, DefaultMaxWorkers)
	}
	cclog.Debugf("[METRICSTORE]> Using %d workers for checkpoint/archive operations\n", Keys.NumWorkers)

	// Pass the config.MetricStoreKeys
	InitMetrics(metrics)

	ms := GetMemoryStore()

	d, err := time.ParseDuration(Keys.RetentionInMemory)
	if err != nil {
		cclog.Fatal(err)
	}

	restoreFrom := startupTime.Add(-d)
	cclog.Infof("[METRICSTORE]> Loading checkpoints newer than %s\n", restoreFrom.Format(time.RFC3339))
	files, err := ms.FromCheckpointFiles(Keys.Checkpoints.RootDir, restoreFrom.Unix())
	loadedData := ms.SizeInBytes() / 1024 / 1024 // In MB
	if err != nil {
		cclog.Fatalf("[METRICSTORE]> Loading checkpoints failed: %s\n", err.Error())
	} else {
		cclog.Infof("[METRICSTORE]> Checkpoints loaded (%d files, %d MB, that took %fs)\n", files, loadedData, time.Since(startupTime).Seconds())
	}

	// Try to use less memory by forcing a GC run here and then
	// lowering the target percentage. The default of 100 means
	// that only once the ratio of new allocations execeds the
	// previously active heap, a GC is triggered.
	// Forcing a GC here will set the "previously active heap"
	// to a minumum.
	// runtime.GC()

	ctx, shutdown := context.WithCancel(context.Background())

	Retention(wg, ctx)
	Checkpointing(wg, ctx)
	CleanUp(wg, ctx)
	WALStaging(wg, ctx)
	MemoryUsageTracker(wg, ctx)

	// Note: Signal handling has been removed from this function.
	// The caller is responsible for handling shutdown signals and calling
	// the shutdown() function when appropriate.
	// Store the shutdown function for later use by Shutdown()
	shutdownFuncMu.Lock()
	shutdownFunc = shutdown
	shutdownFuncMu.Unlock()

	if Keys.Subscriptions != nil {
		err = ReceiveNats(ms, 1, ctx)
		if err != nil {
			cclog.Fatal(err)
		}
	}
}

// InitMetrics initializes the singleton MemoryStore instance with the given metric configurations.
//
// This function must be called before GetMemoryStore() and can only be called once due to
// the singleton pattern. It assigns each metric an internal offset for efficient buffer indexing.
//
// Parameters:
//   - metrics: Map of metric names to their configurations (frequency and aggregation strategy)
//
// Panics if any metric has Frequency == 0, which indicates an invalid configuration.
//
// After this call, the global msInstance is ready for use via GetMemoryStore().
func InitMetrics(metrics map[string]MetricConfig) {
	singleton.Do(func() {
		offset := 0
		for key, cfg := range metrics {
			if cfg.Frequency == 0 {
				panic("[METRICSTORE]> invalid frequency")
			}

			metrics[key] = MetricConfig{
				Frequency:   cfg.Frequency,
				Aggregation: cfg.Aggregation,
				offset:      offset,
			}
			offset += 1
		}

		msInstance = &MemoryStore{
			root: Level{
				metrics:  make([]*buffer, len(metrics)),
				children: make(map[string]*Level),
			},
			Metrics: metrics,
		}
	})
}

// GetMemoryStore returns the singleton MemoryStore instance.
//
// Returns the initialized MemoryStore singleton. Calls cclog.Fatal if InitMetrics() was not called first.
//
// This function is safe for concurrent use after initialization.
func GetMemoryStore() *MemoryStore {
	if msInstance == nil {
		cclog.Warnf("[METRICSTORE]> MemoryStore not initialized!")
	}

	return msInstance
}

func (ms *MemoryStore) GetMetricFrequency(metricName string) (int64, error) {
	if metric, ok := ms.Metrics[metricName]; ok {
		return metric.Frequency, nil
	}
	return 0, fmt.Errorf("[METRICSTORE]> metric %s not found", metricName)
}

// SetNodeProvider sets the NodeProvider implementation for the MemoryStore.
// This must be called during initialization to provide job state information
// for selective buffer retention during Free operations.
// If not set, the Free function will fall back to freeing all buffers.
func (ms *MemoryStore) SetNodeProvider(provider NodeProvider) {
	ms.nodeProvider = provider
}

// Shutdown performs a graceful shutdown of the metric store.
//
// This function cancels all background goroutines started by Init() and writes
// a final checkpoint to disk before returning. It should be called when the
// application receives a termination signal.
//
// The function will:
//  1. Cancel the context to stop all background workers
//  2. Close the WAL messages channel if using WAL format
//  3. Write a final checkpoint to preserve in-memory data
//  4. Log any errors encountered during shutdown
//
// Note: This function blocks until the final checkpoint is written.
func Shutdown() {
	shutdownFuncMu.Lock()
	defer shutdownFuncMu.Unlock()
	if shutdownFunc != nil {
		shutdownFunc()
	}

	if Keys.Checkpoints.FileFormat == "wal" {
		close(WALMessages)
	}

	cclog.Infof("[METRICSTORE]> Writing to '%s'...\n", Keys.Checkpoints.RootDir)
	var files int
	var err error

	ms := GetMemoryStore()

	lastCheckpointMu.Lock()
	from := lastCheckpoint
	lastCheckpointMu.Unlock()

	if Keys.Checkpoints.FileFormat == "wal" {
		var hostDirs []string
		files, hostDirs, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
		if err == nil {
			RotateWALFilesAfterShutdown(hostDirs)
		}
	} else {
		files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
	}

	if err != nil {
		cclog.Errorf("[METRICSTORE]> Writing checkpoint failed: %s\n", err.Error())
	}
	cclog.Infof("[METRICSTORE]> Done! (%d files written)\n", files)
}

// Retention starts a background goroutine that periodically frees old metric data.
//
// This worker runs at half the retention interval and calls Free() to remove buffers
// older than the configured retention time. It respects the NodeProvider to preserve
// data for nodes with active jobs.
//
// Parameters:
//   - wg: WaitGroup to signal completion when context is cancelled
//   - ctx: Context for cancellation signal
//
// The goroutine exits when ctx is cancelled.
func Retention(wg *sync.WaitGroup, ctx context.Context) {
	ms := GetMemoryStore()

	wg.Go(func() {
		d, err := time.ParseDuration(Keys.RetentionInMemory)
		if err != nil {
			cclog.Fatal(err)
		}
		if d <= 0 {
			return
		}

		tickInterval := d / 2
		if tickInterval <= 0 {
			return
		}
		ticker := time.NewTicker(tickInterval)
		defer ticker.Stop()

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				state.mu.Lock()

				t := time.Now().Add(-d)

				state.lastRetentionTime = t.Unix()

				cclog.Infof("[METRICSTORE]> start freeing buffers (older than %s)...\n", t.Format(time.RFC3339))

				freed, err := Free(ms, t)
				if err != nil {
					cclog.Errorf("[METRICSTORE]> freeing up buffers failed: %s\n", err.Error())
				} else {
					cclog.Infof("[METRICSTORE]> done: %d buffers freed\n", freed)
				}

				state.mu.Unlock()

				// Clean up the buffer pool
				bufferPool.Clean(state.lastRetentionTime)
			}
		}
	})
}

// MemoryUsageTracker starts a background goroutine that monitors memory usage.
//
// This worker checks actual process memory usage (via runtime.MemStats) periodically
// and force-frees buffers if memory exceeds the configured cap. It uses FreeOSMemory()
// to return memory to the OS after freeing buffers, avoiding aggressive GC that causes
// performance issues.
//
// The tracker logs both actual memory usage (heap allocated) and metric data size for
// visibility into memory overhead from Go runtime structures and allocations.
//
// Parameters:
//   - wg: WaitGroup to signal completion when context is cancelled
//   - ctx: Context for cancellation signal
//
// The goroutine exits when ctx is cancelled.
func MemoryUsageTracker(wg *sync.WaitGroup, ctx context.Context) {
	ms := GetMemoryStore()

	wg.Go(func() {
		normalInterval := DefaultMemoryUsageTrackerInterval
		fastInterval := 30 * time.Second

		if normalInterval <= 0 {
			return
		}

		ticker := time.NewTicker(normalInterval)
		defer ticker.Stop()
		currentInterval := normalInterval

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				var mem runtime.MemStats
				runtime.ReadMemStats(&mem)
				actualMemoryGB := float64(mem.Alloc) / 1e9
				metricDataGB := ms.SizeInGB()
				cclog.Infof("[METRICSTORE]> memory usage: %.2f GB actual (%.2f GB metric data)", actualMemoryGB, metricDataGB)

				freedExcluded := 0
				freedEmergency := 0
				var err error

				state.mu.RLock()
				lastRetention := state.lastRetentionTime
				selectorsExcluded := state.selectorsExcluded
				state.mu.RUnlock()

				if lastRetention != 0 && selectorsExcluded {
					freedExcluded, err = ms.Free(nil, lastRetention)
					if err != nil {
						cclog.Errorf("[METRICSTORE]> error while force-freeing the excluded buffers: %s", err)
					}

					if freedExcluded > 0 {
						debug.FreeOSMemory()
						cclog.Infof("[METRICSTORE]> done: %d excluded buffers force-freed", freedExcluded)
					}
				}

				runtime.ReadMemStats(&mem)
				actualMemoryGB = float64(mem.Alloc) / 1e9

				if actualMemoryGB > float64(Keys.MemoryCap) {
					cclog.Warnf("[METRICSTORE]> memory usage %.2f GB exceeds cap %d GB, starting emergency buffer freeing", actualMemoryGB, Keys.MemoryCap)

					// Use progressive time-based Free with increasing threshold
					// instead of ForceFree loop — fewer tree traversals, more effective
					d, parseErr := time.ParseDuration(Keys.RetentionInMemory)
					if parseErr != nil {
						cclog.Errorf("[METRICSTORE]> cannot parse retention duration: %s", parseErr)
					} else {
						thresholds := []float64{0.75, 0.5, 0.25}
						for _, fraction := range thresholds {
							threshold := time.Now().Add(-time.Duration(float64(d) * fraction))
							freed, freeErr := ms.Free(nil, threshold.Unix())
							if freeErr != nil {
								cclog.Errorf("[METRICSTORE]> error while freeing buffers at %.0f%% retention: %s", fraction*100, freeErr)
							}
							freedEmergency += freed

							bufferPool.Clear()
							runtime.GC()
							runtime.ReadMemStats(&mem)
							actualMemoryGB = float64(mem.Alloc) / 1e9

							if actualMemoryGB < float64(Keys.MemoryCap) {
								break
							}
						}
					}

					bufferPool.Clear()
					debug.FreeOSMemory()

					runtime.ReadMemStats(&mem)
					actualMemoryGB = float64(mem.Alloc) / 1e9

					if actualMemoryGB >= float64(Keys.MemoryCap) {
						cclog.Errorf("[METRICSTORE]> after emergency frees (%d buffers), memory usage %.2f GB still at/above cap %d GB", freedEmergency, actualMemoryGB, Keys.MemoryCap)
					} else {
						cclog.Infof("[METRICSTORE]> emergency freeing complete: %d buffers freed, memory now %.2f GB", freedEmergency, actualMemoryGB)
					}
				}

				// Adaptive ticker: check more frequently when memory is high
				memoryRatio := actualMemoryGB / float64(Keys.MemoryCap)
				if memoryRatio > 0.8 && currentInterval != fastInterval {
					ticker.Reset(fastInterval)
					currentInterval = fastInterval
					cclog.Infof("[METRICSTORE]> memory at %.0f%% of cap, switching to fast check interval (30s)", memoryRatio*100)
				} else if memoryRatio <= 0.8 && currentInterval != normalInterval {
					ticker.Reset(normalInterval)
					currentInterval = normalInterval
					cclog.Infof("[METRICSTORE]> memory at %.0f%% of cap, switching to normal check interval", memoryRatio*100)
				}
			}
		}
	})
}

// Free removes metric data older than the given time while preserving data for active nodes.
//
// This function implements intelligent retention by consulting the NodeProvider (if configured)
// to determine which nodes are currently in use by running jobs. Data for these nodes is
// preserved even if older than the retention time.
//
// Parameters:
//   - ms: The MemoryStore instance
//   - t: Time threshold - buffers with data older than this will be freed
//
// Returns:
//   - Number of buffers freed
//   - Error if NodeProvider query fails
//
// Behavior:
//   - If no NodeProvider is set: frees all buffers older than t
//   - If NodeProvider returns empty map: frees all buffers older than t
//   - Otherwise: preserves buffers for nodes returned by GetUsedNodes(), frees others
func Free(ms *MemoryStore, t time.Time) (int, error) {
	// If no NodeProvider is configured, free all buffers older than t
	if ms.nodeProvider == nil {
		return ms.Free(nil, t.Unix())
	}

	excludeSelectors, err := ms.nodeProvider.GetUsedNodes(t.Unix())
	if err != nil {
		return 0, err
	}

	switch lenMap := len(excludeSelectors); lenMap {

	// If the length of the map returned by GetUsedNodes() is 0,
	// then use default Free method with nil selector
	case 0:
		state.selectorsExcluded = false
		return ms.Free(nil, t.Unix())

	// Else formulate selectors, exclude those from the map
	// and free the rest of the selectors
	default:
		state.selectorsExcluded = true
		selectors := GetSelectors(ms, excludeSelectors)
		return FreeSelected(ms, selectors, t)
	}
}

// FreeSelected frees buffers for specific selectors while preserving others.
//
// This function is used when we want to retain some specific nodes beyond the retention time.
// It iterates through the provided selectors and frees their associated buffers.
//
// Parameters:
//   - ms: The MemoryStore instance
//   - selectors: List of selector paths to free (e.g., [["cluster1", "node1"], ["cluster2", "node2"]])
//   - t: Time threshold for freeing buffers
//
// Returns the total number of buffers freed and any error encountered.
func FreeSelected(ms *MemoryStore, selectors [][]string, t time.Time) (int, error) {
	freed := 0

	for _, selector := range selectors {

		freedBuffers, err := ms.Free(selector, t.Unix())
		if err != nil {
			cclog.Errorf("error while freeing selected buffers: %#v", err)
		}
		freed += freedBuffers

	}

	return freed, nil
}

// GetSelectors returns all selectors at depth 2 (cluster/node level) that are NOT in the exclusion map.
//
// This function generates a list of selectors whose buffers should be freed by excluding
// selectors that correspond to nodes currently in use by running jobs.
//
// Parameters:
//   - ms: The MemoryStore instance
//   - excludeSelectors: Map of cluster names to node hostnames that should NOT be freed
//
// Returns a list of selectors ([]string paths) that can be safely freed.
//
// Example:
//
//	If the tree has paths ["emmy", "node001"] and ["emmy", "node002"],
//	and excludeSelectors contains {"emmy": ["node001"]},
//	then only [["emmy", "node002"]] is returned.
func GetSelectors(ms *MemoryStore, excludeSelectors map[string][]string) [][]string {
	allSelectors := ms.GetPaths(2)

	filteredSelectors := make([][]string, 0, len(allSelectors))

	for _, path := range allSelectors {
		if len(path) < 2 {
			continue
		}

		key := path[0]   // The "Key" (Level 1)
		value := path[1] // The "Value" (Level 2)

		exclude := false

		// Check if the key exists in our exclusion map
		if excludedValues, exists := excludeSelectors[key]; exists {
			// The key exists, now check if the specific value is in the exclusion list
			if slices.Contains(excludedValues, value) {
				exclude = true
			}
		}

		if !exclude {
			filteredSelectors = append(filteredSelectors, path)
		}
	}

	return filteredSelectors
}

// GetPaths returns a list of lists (paths) to the specified depth.
func (ms *MemoryStore) GetPaths(targetDepth int) [][]string {
	var results [][]string

	// Start recursion. Initial path is empty.
	// We treat Root as depth 0.
	ms.root.collectPaths(0, targetDepth, []string{}, &results)

	return results
}

// Write all values in `metrics` to the level specified by `selector` for time `ts`.
// Look at `findLevelOrCreate` for how selectors work.
func (m *MemoryStore) Write(selector []string, ts int64, metrics []Metric) error {
	var ok bool
	for i, metric := range metrics {
		if metric.MetricConfig.Frequency == 0 {
			metric.MetricConfig, ok = m.Metrics[metric.Name]
			if !ok {
				cclog.Debugf("[METRICSTORE]> Unknown metric '%s' in Write() - skipping", metric.Name)
				metric.MetricConfig.Frequency = 0
			}
			metrics[i] = metric
		}
	}

	return m.WriteToLevel(&m.root, selector, ts, metrics)
}

func (m *MemoryStore) GetLevel(selector []string) *Level {
	return m.root.findLevelOrCreate(selector, len(m.Metrics))
}

// WriteToLevel assumes that `minfo` in `metrics` is filled in
func (m *MemoryStore) WriteToLevel(l *Level, selector []string, ts int64, metrics []Metric) error {
	l = l.findLevelOrCreate(selector, len(m.Metrics))
	l.lock.Lock()
	defer l.lock.Unlock()

	for _, metric := range metrics {
		if metric.MetricConfig.Frequency == 0 {
			continue
		}

		b := l.metrics[metric.MetricConfig.offset]
		if b == nil {
			// First write to this metric and level
			b = newBuffer(ts, metric.MetricConfig.Frequency)
			l.metrics[metric.MetricConfig.offset] = b
		}

		nb, err := b.write(ts, metric.Value)
		if err != nil {
			return err
		}

		// Last write created a new buffer...
		if b != nb {
			l.metrics[metric.MetricConfig.offset] = nb
		}
	}
	return nil
}

// Read returns all values for metric `metric` from `from` to `to` for the selected level(s).
// If the level does not hold the metric itself, the data will be aggregated recursively from the children.
// The second and third return value are the actual from/to for the data. Those can be different from
// the range asked for if no data was available.
func (m *MemoryStore) Read(selector util.Selector, metric string, from, to, resolution int64) ([]schema.Float, int64, int64, int64, error) {
	if from > to {
		return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid time range")
	}

	minfo, ok := m.Metrics[metric]
	if !ok {
		return nil, 0, 0, 0, errors.New("[METRICSTORE]> unknown metric: " + metric)
	}

	n, data := 0, make([]schema.Float, (to-from)/minfo.Frequency+1)

	err := m.root.findBuffers(selector, minfo.offset, func(b *buffer) error {
		cdata, cfrom, cto, err := b.read(from, to, data)
		if err != nil {
			return err
		}

		if n == 0 {
			from, to = cfrom, cto
		} else if from != cfrom || to != cto || len(data) != len(cdata) {
			missingfront, missingback := int((from-cfrom)/minfo.Frequency), int((to-cto)/minfo.Frequency)
			if missingfront != 0 {
				return ErrDataDoesNotAlign
			}

			newlen := len(cdata) - missingback
			if newlen < 1 {
				return ErrDataDoesNotAlign
			}
			cdata = cdata[0:newlen]
			if len(cdata) != len(data) {
				return ErrDataDoesNotAlign
			}

			from, to = cfrom, cto
		}

		data = cdata
		n += 1
		return nil
	})

	if err != nil {
		return nil, 0, 0, 0, err
	} else if n == 0 {
		return nil, 0, 0, 0, ErrNoHostOrMetric
	} else if n > 1 {
		if minfo.Aggregation == AvgAggregation {
			normalize := 1. / schema.Float(n)
			for i := 0; i < len(data); i++ {
				data[i] *= normalize
			}
		} else if minfo.Aggregation != SumAggregation {
			return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid aggregation")
		}
	}

	data, resolution, err = resampler.LargestTriangleThreeBucket(data, minfo.Frequency, resolution)
	if err != nil {
		return nil, 0, 0, 0, err
	}

	return data, from, to, resolution, nil
}

// Free releases all buffers for the selected level and all its children that
// contain only values older than `t`.
func (m *MemoryStore) Free(selector []string, t int64) (int, error) {
	return m.GetLevel(selector).free(t)
}

// ForceFree unconditionally removes the oldest buffer from each metric chain.
func (m *MemoryStore) ForceFree() (int, error) {
	return m.GetLevel(nil).forceFree()
}

func (m *MemoryStore) FreeAll() error {
	for k := range m.root.children {
		delete(m.root.children, k)
	}

	return nil
}

func (m *MemoryStore) SizeInBytes() int64 {
	return m.root.sizeInBytes()
}

func (m *MemoryStore) SizeInGB() float64 {
	return float64(m.root.sizeInBytes()) / 1e9
}

// ListChildren , given a selector, returns a list of all children of the level
// selected.
func (m *MemoryStore) ListChildren(selector []string) []string {
	lvl := &m.root
	for lvl != nil && len(selector) != 0 {
		lvl.lock.RLock()
		next := lvl.children[selector[0]]
		lvl.lock.RUnlock()
		lvl = next
		selector = selector[1:]
	}

	if lvl == nil {
		return nil
	}

	lvl.lock.RLock()
	defer lvl.lock.RUnlock()

	children := make([]string, 0, len(lvl.children))
	for child := range lvl.children {
		children = append(children, child)
	}

	return children
}