<local-command-caveat>Caveat: The messages below were generated by the user while running local commands. DO NOT respond to these messages or otherwise consider them in your response unless the user explicitly asks you to.</local-command-caveat>


<command-name>/model</command-name>
            <command-message>model</command-message>
            <command-args></command-args>


<local-command-stdout>Set model to opus (claude-opus-4-6)</local-command-stdout>


The buildQueries logic on @pkg/metricstore/query.go  is diverged from the buildQueries logic in @internal/metricstoreclient/cc-metric-store-queries.go . As it appears the latter is the cleaner implementation. Analyse the difference, synchronize and evaluate if the code can be consolidated.


Let me analyze the differences between the two implementations and explore whether they can be consolidated.


I'll explore both implementations and their surrounding context to understand the full picture.


Agent

Explore metricstore package structure

Prompt:
Explore the pkg/metricstore/ package thoroughly. I need to understand:
1. What types are defined (APIQuery, APIQueryRequest, etc.) - check metricstore.go and any type definition files
2. How FetchData works
3. What InternalMetricStore is and how it relates to the metric store interface
4. The full list of files in this package

Focus on understanding the package's role and its types/interfaces.


Explore the pkg/metricstore/ package thoroughly. I need to understand:
1. What types are defined (APIQuery, APIQueryRequest, etc.) - check metricstore.go and any type definition files
2. How FetchData works
3. What InternalMetricStore is and how it relates to the metric store interface
4. The full list of files in this package

Focus on understanding the package's role and its types/interfaces.

Prompt:
Explore the pkg/metricstore/ package thoroughly. I need to understand:
1. What types are defined (APIQuery, APIQueryRequest, etc.) - check metricstore.go and any type definition files
2. How FetchData works
3. What InternalMetricStore is and how it relates to the metric store interface
4. The full list of files in this package

Focus on understanding the package's role and its types/interfaces.


Bash

find /Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore -type f -name "*.go" | sort


Agent

Explore metricstoreclient package

Prompt:
Explore the internal/metricstoreclient/ package thoroughly. I need to understand:
1. All files in this package
2. The CCMetricStore type and its methods (especially LoadData, LoadStats, LoadScopedStats, LoadNodeData, LoadNodeListData)
3. How it defines APIQuery and related types
4. The getTopology and getTopologyByNode helper methods
5. How this package relates to pkg/metricstore/

Focus on understanding the full structure and how it parallels pkg/metricstore/query.go.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstore provides an efficient in-memory time-series metric storage system
     7→// with support for hierarchical data organization, checkpointing, and archiving.
     8→//
     9→// The package organizes metrics in a tree structure (cluster → host → component) and
    10→// provides concurrent read/write access to metric data with configurable aggregation strategies.
    11→// Background goroutines handle periodic checkpointing (JSON or Avro format), archiving old data,
    12→// and enforcing retention policies.
    13→//
    14→// Key features:
    15→//   - In-memory metric storage with configurable retention
    16→//   - Hierarchical data organization (selectors)
    17→//   - Concurrent checkpoint/archive workers
    18→//   - Support for sum and average aggregation
    19→//   - NATS integration for metric ingestion
    20→package metricstore
    21→
    22→import (
    23→	"bytes"
    24→	"context"
    25→	"encoding/json"
    26→	"errors"
    27→	"fmt"
    28→	"runtime"
    29→	"runtime/debug"
    30→	"slices"
    31→	"sync"
    32→	"time"
    33→
    34→	"github.com/ClusterCockpit/cc-backend/internal/config"
    35→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    36→	"github.com/ClusterCockpit/cc-lib/v2/resampler"
    37→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    38→	"github.com/ClusterCockpit/cc-lib/v2/util"
    39→)
    40→
    41→// GlobalState holds the global state for the metric store with thread-safe access.
    42→type GlobalState struct {
    43→	mu                sync.RWMutex
    44→	lastRetentionTime int64
    45→	selectorsExcluded bool
    46→}
    47→
    48→var (
    49→	singleton  sync.Once
    50→	msInstance *MemoryStore
    51→	// shutdownFunc stores the context cancellation function created in Init
    52→	// and is called during Shutdown to cancel all background goroutines
    53→	shutdownFunc   context.CancelFunc
    54→	shutdownFuncMu sync.Mutex // Protects shutdownFunc from concurrent access
    55→	// Create a global instance
    56→	state = &GlobalState{}
    57→)
    58→
    59→// NodeProvider provides information about nodes currently in use by running jobs.
    60→//
    61→// This interface allows metricstore to query job information without directly
    62→// depending on the repository package, breaking the import cycle.
    63→//
    64→// Implementations should return nodes that are actively processing jobs started
    65→// before the given timestamp. These nodes will be excluded from retention-based
    66→// garbage collection to prevent data loss for jobs that are still running or
    67→// recently completed.
    68→type NodeProvider interface {
    69→	// GetUsedNodes returns a map of cluster names to sorted lists of unique hostnames
    70→	// that are currently in use by jobs that started before the given timestamp.
    71→	//
    72→	// Parameters:
    73→	//   - ts: Unix timestamp threshold - returns nodes with jobs started before this time
    74→	//
    75→	// Returns:
    76→	//   - Map of cluster names to lists of node hostnames that should be excluded from garbage collection
    77→	//   - Error if the query fails
    78→	GetUsedNodes(ts int64) (map[string][]string, error)
    79→}
    80→
    81→// Metric represents a single metric data point to be written to the store.
    82→type Metric struct {
    83→	Name  string
    84→	Value schema.Float
    85→	// MetricConfig contains frequency and aggregation settings for this metric.
    86→	// If Frequency is 0, configuration will be looked up from MemoryStore.Metrics during Write().
    87→	MetricConfig MetricConfig
    88→}
    89→
    90→// MemoryStore is the main in-memory time-series metric storage implementation.
    91→//
    92→// It organizes metrics in a hierarchical tree structure where each level represents
    93→// a component of the system hierarchy (e.g., cluster → host → CPU). Each level can
    94→// store multiple metrics as time-series buffers.
    95→//
    96→// The store is initialized as a singleton via InitMetrics() and accessed via GetMemoryStore().
    97→// All public methods are safe for concurrent use.
    98→type MemoryStore struct {
    99→	Metrics      map[string]MetricConfig
   100→	root         Level
   101→	nodeProvider NodeProvider
   102→}
   103→
   104→// Init initializes the metric store from configuration and starts background workers.
   105→//
   106→// This function must be called exactly once before any other metricstore operations.
   107→// It performs the following initialization steps:
   108→//  1. Validates and decodes the metric store configuration
   109→//  2. Configures worker pool size (defaults to NumCPU/2+1, max 10)
   110→//  3. Loads metric configurations from all registered clusters
   111→//  4. Restores checkpoints within the retention window
   112→//  5. Starts background workers for retention, checkpointing, archiving, and monitoring
   113→//  6. Optionally subscribes to NATS for real-time metric ingestion
   114→//
   115→// Parameters:
   116→//   - rawConfig: JSON configuration for the metric store (see MetricStoreConfig)
   117→//   - wg: WaitGroup that will be incremented for each background goroutine started
   118→//
   119→// The function will call cclog.Fatal on critical errors during initialization.
   120→// Use Shutdown() to cleanly stop all background workers started by Init().
   121→//
   122→// Note: Signal handling must be implemented by the caller. Call Shutdown() when
   123→// receiving termination signals to ensure checkpoint data is persisted.
   124→func Init(rawConfig json.RawMessage, metrics map[string]MetricConfig, wg *sync.WaitGroup) {
   125→	startupTime := time.Now()
   126→
   127→	if rawConfig != nil {
   128→		config.Validate(configSchema, rawConfig)
   129→		dec := json.NewDecoder(bytes.NewReader(rawConfig))
   130→		dec.DisallowUnknownFields()
   131→		if err := dec.Decode(&Keys); err != nil {
   132→			cclog.Abortf("[METRICSTORE]> Metric Store Config Init: Could not decode config file '%s'.\nError: %s\n", rawConfig, err.Error())
   133→		}
   134→	}
   135→
   136→	// Set NumWorkers from config or use default
   137→	if Keys.NumWorkers <= 0 {
   138→		Keys.NumWorkers = min(runtime.NumCPU()/2+1, DefaultMaxWorkers)
   139→	}
   140→	cclog.Debugf("[METRICSTORE]> Using %d workers for checkpoint/archive operations\n", Keys.NumWorkers)
   141→
   142→	// Pass the config.MetricStoreKeys
   143→	InitMetrics(metrics)
   144→
   145→	ms := GetMemoryStore()
   146→
   147→	d, err := time.ParseDuration(Keys.RetentionInMemory)
   148→	if err != nil {
   149→		cclog.Fatal(err)
   150→	}
   151→
   152→	restoreFrom := startupTime.Add(-d)
   153→	cclog.Infof("[METRICSTORE]> Loading checkpoints newer than %s\n", restoreFrom.Format(time.RFC3339))
   154→	files, err := ms.FromCheckpointFiles(Keys.Checkpoints.RootDir, restoreFrom.Unix())
   155→	loadedData := ms.SizeInBytes() / 1024 / 1024 // In MB
   156→	if err != nil {
   157→		cclog.Fatalf("[METRICSTORE]> Loading checkpoints failed: %s\n", err.Error())
   158→	} else {
   159→		cclog.Infof("[METRICSTORE]> Checkpoints loaded (%d files, %d MB, that took %fs)\n", files, loadedData, time.Since(startupTime).Seconds())
   160→	}
   161→
   162→	// Try to use less memory by forcing a GC run here and then
   163→	// lowering the target percentage. The default of 100 means
   164→	// that only once the ratio of new allocations execeds the
   165→	// previously active heap, a GC is triggered.
   166→	// Forcing a GC here will set the "previously active heap"
   167→	// to a minumum.
   168→	// runtime.GC()
   169→
   170→	ctx, shutdown := context.WithCancel(context.Background())
   171→
   172→	Retention(wg, ctx)
   173→	Checkpointing(wg, ctx)
   174→	CleanUp(wg, ctx)
   175→	WALStaging(wg, ctx)
   176→	MemoryUsageTracker(wg, ctx)
   177→
   178→	// Note: Signal handling has been removed from this function.
   179→	// The caller is responsible for handling shutdown signals and calling
   180→	// the shutdown() function when appropriate.
   181→	// Store the shutdown function for later use by Shutdown()
   182→	shutdownFuncMu.Lock()
   183→	shutdownFunc = shutdown
   184→	shutdownFuncMu.Unlock()
   185→
   186→	if Keys.Subscriptions != nil {
   187→		err = ReceiveNats(ms, 1, ctx)
   188→		if err != nil {
   189→			cclog.Fatal(err)
   190→		}
   191→	}
   192→}
   193→
   194→// InitMetrics initializes the singleton MemoryStore instance with the given metric configurations.
   195→//
   196→// This function must be called before GetMemoryStore() and can only be called once due to
   197→// the singleton pattern. It assigns each metric an internal offset for efficient buffer indexing.
   198→//
   199→// Parameters:
   200→//   - metrics: Map of metric names to their configurations (frequency and aggregation strategy)
   201→//
   202→// Panics if any metric has Frequency == 0, which indicates an invalid configuration.
   203→//
   204→// After this call, the global msInstance is ready for use via GetMemoryStore().
   205→func InitMetrics(metrics map[string]MetricConfig) {
   206→	singleton.Do(func() {
   207→		offset := 0
   208→		for key, cfg := range metrics {
   209→			if cfg.Frequency == 0 {
   210→				panic("[METRICSTORE]> invalid frequency")
   211→			}
   212→
   213→			metrics[key] = MetricConfig{
   214→				Frequency:   cfg.Frequency,
   215→				Aggregation: cfg.Aggregation,
   216→				offset:      offset,
   217→			}
   218→			offset += 1
   219→		}
   220→
   221→		msInstance = &MemoryStore{
   222→			root: Level{
   223→				metrics:  make([]*buffer, len(metrics)),
   224→				children: make(map[string]*Level),
   225→			},
   226→			Metrics: metrics,
   227→		}
   228→	})
   229→}
   230→
   231→// GetMemoryStore returns the singleton MemoryStore instance.
   232→//
   233→// Returns the initialized MemoryStore singleton. Calls cclog.Fatal if InitMetrics() was not called first.
   234→//
   235→// This function is safe for concurrent use after initialization.
   236→func GetMemoryStore() *MemoryStore {
   237→	if msInstance == nil {
   238→		cclog.Warnf("[METRICSTORE]> MemoryStore not initialized!")
   239→	}
   240→
   241→	return msInstance
   242→}
   243→
   244→func (ms *MemoryStore) GetMetricFrequency(metricName string) (int64, error) {
   245→	if metric, ok := ms.Metrics[metricName]; ok {
   246→		return metric.Frequency, nil
   247→	}
   248→	return 0, fmt.Errorf("[METRICSTORE]> metric %s not found", metricName)
   249→}
   250→
   251→// SetNodeProvider sets the NodeProvider implementation for the MemoryStore.
   252→// This must be called during initialization to provide job state information
   253→// for selective buffer retention during Free operations.
   254→// If not set, the Free function will fall back to freeing all buffers.
   255→func (ms *MemoryStore) SetNodeProvider(provider NodeProvider) {
   256→	ms.nodeProvider = provider
   257→}
   258→
   259→// Shutdown performs a graceful shutdown of the metric store.
   260→//
   261→// This function cancels all background goroutines started by Init() and writes
   262→// a final checkpoint to disk before returning. It should be called when the
   263→// application receives a termination signal.
   264→//
   265→// The function will:
   266→//  1. Cancel the context to stop all background workers
   267→//  2. Close the WAL messages channel if using WAL format
   268→//  3. Write a final checkpoint to preserve in-memory data
   269→//  4. Log any errors encountered during shutdown
   270→//
   271→// Note: This function blocks until the final checkpoint is written.
   272→func Shutdown() {
   273→	shutdownFuncMu.Lock()
   274→	defer shutdownFuncMu.Unlock()
   275→	if shutdownFunc != nil {
   276→		shutdownFunc()
   277→	}
   278→
   279→	if Keys.Checkpoints.FileFormat == "wal" {
   280→		close(WALMessages)
   281→	}
   282→
   283→	cclog.Infof("[METRICSTORE]> Writing to '%s'...\n", Keys.Checkpoints.RootDir)
   284→	var files int
   285→	var err error
   286→
   287→	ms := GetMemoryStore()
   288→
   289→	lastCheckpointMu.Lock()
   290→	from := lastCheckpoint
   291→	lastCheckpointMu.Unlock()
   292→
   293→	if Keys.Checkpoints.FileFormat == "wal" {
   294→		var hostDirs []string
   295→		files, hostDirs, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
   296→		if err == nil {
   297→			RotateWALFilesAfterShutdown(hostDirs)
   298→		}
   299→	} else {
   300→		files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
   301→	}
   302→
   303→	if err != nil {
   304→		cclog.Errorf("[METRICSTORE]> Writing checkpoint failed: %s\n", err.Error())
   305→	}
   306→	cclog.Infof("[METRICSTORE]> Done! (%d files written)\n", files)
   307→}
   308→
   309→// Retention starts a background goroutine that periodically frees old metric data.
   310→//
   311→// This worker runs at half the retention interval and calls Free() to remove buffers
   312→// older than the configured retention time. It respects the NodeProvider to preserve
   313→// data for nodes with active jobs.
   314→//
   315→// Parameters:
   316→//   - wg: WaitGroup to signal completion when context is cancelled
   317→//   - ctx: Context for cancellation signal
   318→//
   319→// The goroutine exits when ctx is cancelled.
   320→func Retention(wg *sync.WaitGroup, ctx context.Context) {
   321→	ms := GetMemoryStore()
   322→
   323→	wg.Go(func() {
   324→		d, err := time.ParseDuration(Keys.RetentionInMemory)
   325→		if err != nil {
   326→			cclog.Fatal(err)
   327→		}
   328→		if d <= 0 {
   329→			return
   330→		}
   331→
   332→		tickInterval := d / 2
   333→		if tickInterval <= 0 {
   334→			return
   335→		}
   336→		ticker := time.NewTicker(tickInterval)
   337→		defer ticker.Stop()
   338→
   339→		for {
   340→			select {
   341→			case <-ctx.Done():
   342→				return
   343→			case <-ticker.C:
   344→				state.mu.Lock()
   345→
   346→				t := time.Now().Add(-d)
   347→
   348→				state.lastRetentionTime = t.Unix()
   349→
   350→				cclog.Infof("[METRICSTORE]> start freeing buffers (older than %s)...\n", t.Format(time.RFC3339))
   351→
   352→				freed, err := Free(ms, t)
   353→				if err != nil {
   354→					cclog.Errorf("[METRICSTORE]> freeing up buffers failed: %s\n", err.Error())
   355→				} else {
   356→					cclog.Infof("[METRICSTORE]> done: %d buffers freed\n", freed)
   357→				}
   358→
   359→				state.mu.Unlock()
   360→
   361→				// Clean up the buffer pool
   362→				bufferPool.Clean(state.lastRetentionTime)
   363→			}
   364→		}
   365→	})
   366→}
   367→
   368→// MemoryUsageTracker starts a background goroutine that monitors memory usage.
   369→//
   370→// This worker checks actual process memory usage (via runtime.MemStats) periodically
   371→// and force-frees buffers if memory exceeds the configured cap. It uses FreeOSMemory()
   372→// to return memory to the OS after freeing buffers, avoiding aggressive GC that causes
   373→// performance issues.
   374→//
   375→// The tracker logs both actual memory usage (heap allocated) and metric data size for
   376→// visibility into memory overhead from Go runtime structures and allocations.
   377→//
   378→// Parameters:
   379→//   - wg: WaitGroup to signal completion when context is cancelled
   380→//   - ctx: Context for cancellation signal
   381→//
   382→// The goroutine exits when ctx is cancelled.
   383→func MemoryUsageTracker(wg *sync.WaitGroup, ctx context.Context) {
   384→	ms := GetMemoryStore()
   385→
   386→	wg.Go(func() {
   387→		d := DefaultMemoryUsageTrackerInterval
   388→
   389→		if d <= 0 {
   390→			return
   391→		}
   392→
   393→		ticker := time.NewTicker(d)
   394→		defer ticker.Stop()
   395→
   396→		for {
   397→			select {
   398→			case <-ctx.Done():
   399→				return
   400→			case <-ticker.C:
   401→				var mem runtime.MemStats
   402→				runtime.ReadMemStats(&mem)
   403→				actualMemoryGB := float64(mem.Alloc) / 1e9
   404→				metricDataGB := ms.SizeInGB()
   405→				cclog.Infof("[METRICSTORE]> memory usage: %.2f GB actual (%.2f GB metric data)", actualMemoryGB, metricDataGB)
   406→
   407→				freedExcluded := 0
   408→				freedEmergency := 0
   409→				var err error
   410→
   411→				state.mu.RLock()
   412→				lastRetention := state.lastRetentionTime
   413→				selectorsExcluded := state.selectorsExcluded
   414→				state.mu.RUnlock()
   415→
   416→				if lastRetention != 0 && selectorsExcluded {
   417→					freedExcluded, err = ms.Free(nil, lastRetention)
   418→					if err != nil {
   419→						cclog.Errorf("[METRICSTORE]> error while force-freeing the excluded buffers: %s", err)
   420→					}
   421→
   422→					if freedExcluded > 0 {
   423→						debug.FreeOSMemory()
   424→						cclog.Infof("[METRICSTORE]> done: %d excluded buffers force-freed", freedExcluded)
   425→					}
   426→				}
   427→
   428→				runtime.ReadMemStats(&mem)
   429→				actualMemoryGB = float64(mem.Alloc) / 1e9
   430→
   431→				bufferPool.Clear()
   432→				cclog.Infof("[METRICSTORE]> Cleaned up bufferPool\n")
   433→
   434→				if actualMemoryGB > float64(Keys.MemoryCap) {
   435→					cclog.Warnf("[METRICSTORE]> memory usage %.2f GB exceeds cap %d GB, starting emergency buffer freeing", actualMemoryGB, Keys.MemoryCap)
   436→
   437→					const maxIterations = 100
   438→
   439→					for i := range maxIterations {
   440→						if actualMemoryGB < float64(Keys.MemoryCap) {
   441→							break
   442→						}
   443→
   444→						freed, err := ms.ForceFree()
   445→						if err != nil {
   446→							cclog.Errorf("[METRICSTORE]> error while force-freeing buffers: %s", err)
   447→						}
   448→						if freed == 0 {
   449→							cclog.Errorf("[METRICSTORE]> no more buffers to free after %d emergency frees, memory usage %.2f GB still exceeds cap %d GB", freedEmergency, actualMemoryGB, Keys.MemoryCap)
   450→							break
   451→						}
   452→						freedEmergency += freed
   453→
   454→						if i%10 == 0 && freedEmergency > 0 {
   455→							runtime.ReadMemStats(&mem)
   456→							actualMemoryGB = float64(mem.Alloc) / 1e9
   457→						}
   458→					}
   459→
   460→					// if freedEmergency > 0 {
   461→					// 	debug.FreeOSMemory()
   462→					// }
   463→
   464→					runtime.ReadMemStats(&mem)
   465→					actualMemoryGB = float64(mem.Alloc) / 1e9
   466→
   467→					if actualMemoryGB >= float64(Keys.MemoryCap) {
   468→						cclog.Errorf("[METRICSTORE]> after %d emergency frees, memory usage %.2f GB still at/above cap %d GB", freedEmergency, actualMemoryGB, Keys.MemoryCap)
   469→					} else {
   470→						cclog.Infof("[METRICSTORE]> emergency freeing complete: %d buffers freed, memory now %.2f GB", freedEmergency, actualMemoryGB)
   471→					}
   472→				}
   473→			}
   474→		}
   475→	})
   476→}
   477→
   478→// Free removes metric data older than the given time while preserving data for active nodes.
   479→//
   480→// This function implements intelligent retention by consulting the NodeProvider (if configured)
   481→// to determine which nodes are currently in use by running jobs. Data for these nodes is
   482→// preserved even if older than the retention time.
   483→//
   484→// Parameters:
   485→//   - ms: The MemoryStore instance
   486→//   - t: Time threshold - buffers with data older than this will be freed
   487→//
   488→// Returns:
   489→//   - Number of buffers freed
   490→//   - Error if NodeProvider query fails
   491→//
   492→// Behavior:
   493→//   - If no NodeProvider is set: frees all buffers older than t
   494→//   - If NodeProvider returns empty map: frees all buffers older than t
   495→//   - Otherwise: preserves buffers for nodes returned by GetUsedNodes(), frees others
   496→func Free(ms *MemoryStore, t time.Time) (int, error) {
   497→	// If no NodeProvider is configured, free all buffers older than t
   498→	if ms.nodeProvider == nil {
   499→		return ms.Free(nil, t.Unix())
   500→	}
   501→
   502→	excludeSelectors, err := ms.nodeProvider.GetUsedNodes(t.Unix())
   503→	if err != nil {
   504→		return 0, err
   505→	}
   506→
   507→	switch lenMap := len(excludeSelectors); lenMap {
   508→
   509→	// If the length of the map returned by GetUsedNodes() is 0,
   510→	// then use default Free method with nil selector
   511→	case 0:
   512→		state.selectorsExcluded = false
   513→		return ms.Free(nil, t.Unix())
   514→
   515→	// Else formulate selectors, exclude those from the map
   516→	// and free the rest of the selectors
   517→	default:
   518→		state.selectorsExcluded = true
   519→		selectors := GetSelectors(ms, excludeSelectors)
   520→		return FreeSelected(ms, selectors, t)
   521→	}
   522→}
   523→
   524→// FreeSelected frees buffers for specific selectors while preserving others.
   525→//
   526→// This function is used when we want to retain some specific nodes beyond the retention time.
   527→// It iterates through the provided selectors and frees their associated buffers.
   528→//
   529→// Parameters:
   530→//   - ms: The MemoryStore instance
   531→//   - selectors: List of selector paths to free (e.g., [["cluster1", "node1"], ["cluster2", "node2"]])
   532→//   - t: Time threshold for freeing buffers
   533→//
   534→// Returns the total number of buffers freed and any error encountered.
   535→func FreeSelected(ms *MemoryStore, selectors [][]string, t time.Time) (int, error) {
   536→	freed := 0
   537→
   538→	for _, selector := range selectors {
   539→
   540→		freedBuffers, err := ms.Free(selector, t.Unix())
   541→		if err != nil {
   542→			cclog.Errorf("error while freeing selected buffers: %#v", err)
   543→		}
   544→		freed += freedBuffers
   545→
   546→	}
   547→
   548→	return freed, nil
   549→}
   550→
   551→// GetSelectors returns all selectors at depth 2 (cluster/node level) that are NOT in the exclusion map.
   552→//
   553→// This function generates a list of selectors whose buffers should be freed by excluding
   554→// selectors that correspond to nodes currently in use by running jobs.
   555→//
   556→// Parameters:
   557→//   - ms: The MemoryStore instance
   558→//   - excludeSelectors: Map of cluster names to node hostnames that should NOT be freed
   559→//
   560→// Returns a list of selectors ([]string paths) that can be safely freed.
   561→//
   562→// Example:
   563→//
   564→//	If the tree has paths ["emmy", "node001"] and ["emmy", "node002"],
   565→//	and excludeSelectors contains {"emmy": ["node001"]},
   566→//	then only [["emmy", "node002"]] is returned.
   567→func GetSelectors(ms *MemoryStore, excludeSelectors map[string][]string) [][]string {
   568→	allSelectors := ms.GetPaths(2)
   569→
   570→	filteredSelectors := make([][]string, 0, len(allSelectors))
   571→
   572→	for _, path := range allSelectors {
   573→		if len(path) < 2 {
   574→			continue
   575→		}
   576→
   577→		key := path[0]   // The "Key" (Level 1)
   578→		value := path[1] // The "Value" (Level 2)
   579→
   580→		exclude := false
   581→
   582→		// Check if the key exists in our exclusion map
   583→		if excludedValues, exists := excludeSelectors[key]; exists {
   584→			// The key exists, now check if the specific value is in the exclusion list
   585→			if slices.Contains(excludedValues, value) {
   586→				exclude = true
   587→			}
   588→		}
   589→
   590→		if !exclude {
   591→			filteredSelectors = append(filteredSelectors, path)
   592→		}
   593→	}
   594→
   595→	return filteredSelectors
   596→}
   597→
   598→// GetPaths returns a list of lists (paths) to the specified depth.
   599→func (ms *MemoryStore) GetPaths(targetDepth int) [][]string {
   600→	var results [][]string
   601→
   602→	// Start recursion. Initial path is empty.
   603→	// We treat Root as depth 0.
   604→	ms.root.collectPaths(0, targetDepth, []string{}, &results)
   605→
   606→	return results
   607→}
   608→
   609→// Write all values in `metrics` to the level specified by `selector` for time `ts`.
   610→// Look at `findLevelOrCreate` for how selectors work.
   611→func (m *MemoryStore) Write(selector []string, ts int64, metrics []Metric) error {
   612→	var ok bool
   613→	for i, metric := range metrics {
   614→		if metric.MetricConfig.Frequency == 0 {
   615→			metric.MetricConfig, ok = m.Metrics[metric.Name]
   616→			if !ok {
   617→				cclog.Debugf("[METRICSTORE]> Unknown metric '%s' in Write() - skipping", metric.Name)
   618→				metric.MetricConfig.Frequency = 0
   619→			}
   620→			metrics[i] = metric
   621→		}
   622→	}
   623→
   624→	return m.WriteToLevel(&m.root, selector, ts, metrics)
   625→}
   626→
   627→func (m *MemoryStore) GetLevel(selector []string) *Level {
   628→	return m.root.findLevelOrCreate(selector, len(m.Metrics))
   629→}
   630→
   631→// WriteToLevel assumes that `minfo` in `metrics` is filled in
   632→func (m *MemoryStore) WriteToLevel(l *Level, selector []string, ts int64, metrics []Metric) error {
   633→	l = l.findLevelOrCreate(selector, len(m.Metrics))
   634→	l.lock.Lock()
   635→	defer l.lock.Unlock()
   636→
   637→	for _, metric := range metrics {
   638→		if metric.MetricConfig.Frequency == 0 {
   639→			continue
   640→		}
   641→
   642→		b := l.metrics[metric.MetricConfig.offset]
   643→		if b == nil {
   644→			// First write to this metric and level
   645→			b = newBuffer(ts, metric.MetricConfig.Frequency)
   646→			l.metrics[metric.MetricConfig.offset] = b
   647→		}
   648→
   649→		nb, err := b.write(ts, metric.Value)
   650→		if err != nil {
   651→			return err
   652→		}
   653→
   654→		// Last write created a new buffer...
   655→		if b != nb {
   656→			l.metrics[metric.MetricConfig.offset] = nb
   657→		}
   658→	}
   659→	return nil
   660→}
   661→
   662→// Read returns all values for metric `metric` from `from` to `to` for the selected level(s).
   663→// If the level does not hold the metric itself, the data will be aggregated recursively from the children.
   664→// The second and third return value are the actual from/to for the data. Those can be different from
   665→// the range asked for if no data was available.
   666→func (m *MemoryStore) Read(selector util.Selector, metric string, from, to, resolution int64) ([]schema.Float, int64, int64, int64, error) {
   667→	if from > to {
   668→		return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid time range")
   669→	}
   670→
   671→	minfo, ok := m.Metrics[metric]
   672→	if !ok {
   673→		return nil, 0, 0, 0, errors.New("[METRICSTORE]> unknown metric: " + metric)
   674→	}
   675→
   676→	n, data := 0, make([]schema.Float, (to-from)/minfo.Frequency+1)
   677→
   678→	err := m.root.findBuffers(selector, minfo.offset, func(b *buffer) error {
   679→		cdata, cfrom, cto, err := b.read(from, to, data)
   680→		if err != nil {
   681→			return err
   682→		}
   683→
   684→		if n == 0 {
   685→			from, to = cfrom, cto
   686→		} else if from != cfrom || to != cto || len(data) != len(cdata) {
   687→			missingfront, missingback := int((from-cfrom)/minfo.Frequency), int((to-cto)/minfo.Frequency)
   688→			if missingfront != 0 {
   689→				return ErrDataDoesNotAlign
   690→			}
   691→
   692→			newlen := len(cdata) - missingback
   693→			if newlen < 1 {
   694→				return ErrDataDoesNotAlign
   695→			}
   696→			cdata = cdata[0:newlen]
   697→			if len(cdata) != len(data) {
   698→				return ErrDataDoesNotAlign
   699→			}
   700→
   701→			from, to = cfrom, cto
   702→		}
   703→
   704→		data = cdata
   705→		n += 1
   706→		return nil
   707→	})
   708→
   709→	if err != nil {
   710→		return nil, 0, 0, 0, err
   711→	} else if n == 0 {
   712→		return nil, 0, 0, 0, ErrNoHostOrMetric
   713→	} else if n > 1 {
   714→		if minfo.Aggregation == AvgAggregation {
   715→			normalize := 1. / schema.Float(n)
   716→			for i := 0; i < len(data); i++ {
   717→				data[i] *= normalize
   718→			}
   719→		} else if minfo.Aggregation != SumAggregation {
   720→			return nil, 0, 0, 0, errors.New("[METRICSTORE]> invalid aggregation")
   721→		}
   722→	}
   723→
   724→	data, resolution, err = resampler.LargestTriangleThreeBucket(data, minfo.Frequency, resolution)
   725→	if err != nil {
   726→		return nil, 0, 0, 0, err
   727→	}
   728→
   729→	return data, from, to, resolution, nil
   730→}
   731→
   732→// Free releases all buffers for the selected level and all its children that
   733→// contain only values older than `t`.
   734→func (m *MemoryStore) Free(selector []string, t int64) (int, error) {
   735→	return m.GetLevel(selector).free(t)
   736→}
   737→
   738→// ForceFree unconditionally removes the oldest buffer from each metric chain.
   739→func (m *MemoryStore) ForceFree() (int, error) {
   740→	return m.GetLevel(nil).forceFree()
   741→}
   742→
   743→func (m *MemoryStore) FreeAll() error {
   744→	for k := range m.root.children {
   745→		delete(m.root.children, k)
   746→	}
   747→
   748→	return nil
   749→}
   750→
   751→func (m *MemoryStore) SizeInBytes() int64 {
   752→	return m.root.sizeInBytes()
   753→}
   754→
   755→func (m *MemoryStore) SizeInGB() float64 {
   756→	return float64(m.root.sizeInBytes()) / 1e9
   757→}
   758→
   759→// ListChildren , given a selector, returns a list of all children of the level
   760→// selected.
   761→func (m *MemoryStore) ListChildren(selector []string) []string {
   762→	lvl := &m.root
   763→	for lvl != nil && len(selector) != 0 {
   764→		lvl.lock.RLock()
   765→		next := lvl.children[selector[0]]
   766→		lvl.lock.RUnlock()
   767→		lvl = next
   768→		selector = selector[1:]
   769→	}
   770→
   771→	if lvl == nil {
   772→		return nil
   773→	}
   774→
   775→	lvl.lock.RLock()
   776→	defer lvl.lock.RUnlock()
   777→
   778→	children := make([]string, 0, len(lvl.children))
   779→	for child := range lvl.children {
   780→		children = append(children, child)
   781→	}
   782→
   783→	return children
   784→}
   785→


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// This file contains the API types and data fetching logic for querying metric data
     7→// from the in-memory metric store. It provides structures for building complex queries
     8→// with support for aggregation, scaling, padding, and statistics computation.
     9→package metricstore
    10→
    11→import (
    12→	"errors"
    13→	"fmt"
    14→	"math"
    15→
    16→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    17→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    18→	"github.com/ClusterCockpit/cc-lib/v2/util"
    19→)
    20→
    21→var (
    22→	// ErrNoHostOrMetric is returned when the metric store does not find the host or the metric
    23→	ErrNoHostOrMetric error = errors.New("[METRICSTORE]> metric or host not found")
    24→	// ErrInvalidTimeRange is returned when a query has 'from' >= 'to'
    25→	ErrInvalidTimeRange = errors.New("[METRICSTORE]> invalid time range: 'from' must be before 'to'")
    26→	// ErrEmptyCluster is returned when a query with ForAllNodes has no cluster specified
    27→	ErrEmptyCluster = errors.New("[METRICSTORE]> cluster name cannot be empty")
    28→)
    29→
    30→// APIMetricData represents the response data for a single metric query.
    31→//
    32→// It contains both the time-series data points and computed statistics (avg, min, max).
    33→// If an error occurred during data retrieval, the Error field will be set and other
    34→// fields may be incomplete.
    35→type APIMetricData struct {
    36→	Error      *string           `json:"error,omitempty"`
    37→	Data       schema.FloatArray `json:"data,omitempty"`
    38→	From       int64             `json:"from"`
    39→	To         int64             `json:"to"`
    40→	Resolution int64             `json:"resolution"`
    41→	Avg        schema.Float      `json:"avg"`
    42→	Min        schema.Float      `json:"min"`
    43→	Max        schema.Float      `json:"max"`
    44→}
    45→
    46→// APIQueryRequest represents a batch query request for metric data.
    47→//
    48→// It supports two modes of operation:
    49→//  1. Explicit queries via the Queries field
    50→//  2. Automatic query generation via ForAllNodes (queries all specified metrics for all nodes in the cluster)
    51→//
    52→// The request can be customized with flags to include/exclude statistics, raw data, and padding.
    53→type APIQueryRequest struct {
    54→	Cluster     string     `json:"cluster"`
    55→	Queries     []APIQuery `json:"queries"`
    56→	ForAllNodes []string   `json:"for-all-nodes"`
    57→	From        int64      `json:"from"`
    58→	To          int64      `json:"to"`
    59→	WithStats   bool       `json:"with-stats"`
    60→	WithData    bool       `json:"with-data"`
    61→	WithPadding bool       `json:"with-padding"`
    62→}
    63→
    64→// APIQueryResponse represents the response to an APIQueryRequest.
    65→//
    66→// Results is a 2D array where each outer element corresponds to a query,
    67→// and each inner element corresponds to a selector within that query
    68→// (e.g., multiple CPUs or cores).
    69→type APIQueryResponse struct {
    70→	Queries []APIQuery        `json:"queries,omitempty"`
    71→	Results [][]APIMetricData `json:"results"`
    72→}
    73→
    74→// APIQuery represents a single metric query with optional hierarchical selectors.
    75→//
    76→// The hierarchical selection works as follows:
    77→//   - Hostname: The node to query
    78→//   - Type + TypeIds: First level of hierarchy (e.g., "cpu" + ["0", "1", "2"])
    79→//   - SubType + SubTypeIds: Second level of hierarchy (e.g., "core" + ["0", "1"])
    80→//
    81→// If Aggregate is true, data from multiple type/subtype IDs will be aggregated according
    82→// to the metric's aggregation strategy. Otherwise, separate results are returned for each combination.
    83→type APIQuery struct {
    84→	Type        *string      `json:"type,omitempty"`
    85→	SubType     *string      `json:"subtype,omitempty"`
    86→	Metric      string       `json:"metric"`
    87→	Hostname    string       `json:"host"`
    88→	Resolution  int64        `json:"resolution"`
    89→	TypeIds     []string     `json:"type-ids,omitempty"`
    90→	SubTypeIds  []string     `json:"subtype-ids,omitempty"`
    91→	ScaleFactor schema.Float `json:"scale-by,omitempty"`
    92→	Aggregate   bool         `json:"aggreg"`
    93→}
    94→
    95→// AddStats computes and populates the Avg, Min, and Max fields from the Data array.
    96→//
    97→// NaN values in the data are ignored during computation. If all values are NaN,
    98→// the statistics fields will be set to NaN.
    99→//
   100→// TODO: Optimize this, just like the stats endpoint!
   101→func (data *APIMetricData) AddStats() {
   102→	n := 0
   103→	sum, min, max := 0.0, math.MaxFloat64, -math.MaxFloat64
   104→	for _, x := range data.Data {
   105→		if x.IsNaN() {
   106→			continue
   107→		}
   108→
   109→		n += 1
   110→		sum += float64(x)
   111→		min = math.Min(min, float64(x))
   112→		max = math.Max(max, float64(x))
   113→	}
   114→
   115→	if n > 0 {
   116→		avg := sum / float64(n)
   117→		data.Avg = schema.Float(avg)
   118→		data.Min = schema.Float(min)
   119→		data.Max = schema.Float(max)
   120→	} else {
   121→		data.Avg, data.Min, data.Max = schema.NaN, schema.NaN, schema.NaN
   122→	}
   123→}
   124→
   125→// ScaleBy multiplies all data points and statistics by the given factor.
   126→//
   127→// This is commonly used for unit conversion (e.g., bytes to gigabytes).
   128→// Scaling by 0 or 1 is a no-op for performance reasons.
   129→func (data *APIMetricData) ScaleBy(f schema.Float) {
   130→	if f == 0 || f == 1 {
   131→		return
   132→	}
   133→
   134→	data.Avg *= f
   135→	data.Min *= f
   136→	data.Max *= f
   137→	for i := 0; i < len(data.Data); i++ {
   138→		data.Data[i] *= f
   139→	}
   140→}
   141→
   142→// PadDataWithNull pads the beginning of the data array with NaN values if needed.
   143→//
   144→// This ensures that the data aligns with the requested 'from' timestamp, even if
   145→// the metric store doesn't have data for the earliest time points. This is useful
   146→// for maintaining consistent array indexing across multiple queries.
   147→//
   148→// Parameters:
   149→//   - ms: MemoryStore instance to lookup metric configuration
   150→//   - from: The requested start timestamp
   151→//   - to: The requested end timestamp (unused but kept for API consistency)
   152→//   - metric: The metric name to lookup frequency information
   153→func (data *APIMetricData) PadDataWithNull(ms *MemoryStore, from, to int64, metric string) {
   154→	minfo, ok := ms.Metrics[metric]
   155→	if !ok {
   156→		return
   157→	}
   158→
   159→	if (data.From / minfo.Frequency) > (from / minfo.Frequency) {
   160→		padfront := int((data.From / minfo.Frequency) - (from / minfo.Frequency))
   161→		ndata := make([]schema.Float, 0, padfront+len(data.Data))
   162→		for range padfront {
   163→			ndata = append(ndata, schema.NaN)
   164→		}
   165→		for j := 0; j < len(data.Data); j++ {
   166→			ndata = append(ndata, data.Data[j])
   167→		}
   168→		data.Data = ndata
   169→	}
   170→}
   171→
   172→// FetchData executes a batch metric query request and returns the results.
   173→//
   174→// This is the primary API for retrieving metric data from the memory store. It supports:
   175→//   - Individual queries via req.Queries
   176→//   - Batch queries for all nodes via req.ForAllNodes
   177→//   - Hierarchical selector construction (cluster → host → type → subtype)
   178→//   - Optional statistics computation (avg, min, max)
   179→//   - Optional data scaling
   180→//   - Optional data padding with NaN values
   181→//
   182→// The function constructs selectors based on the query parameters and calls MemoryStore.Read()
   183→// for each selector. If a query specifies Aggregate=false with multiple type/subtype IDs,
   184→// separate results are returned for each combination.
   185→//
   186→// Parameters:
   187→//   - req: The query request containing queries, time range, and options
   188→//
   189→// Returns:
   190→//   - APIQueryResponse containing results for each query, or error if validation fails
   191→//
   192→// Errors:
   193→//   - ErrInvalidTimeRange if req.From > req.To
   194→//   - ErrEmptyCluster if req.ForAllNodes is used without specifying a cluster
   195→//   - Error if MemoryStore is not initialized
   196→//   - Individual query errors are stored in APIMetricData.Error field
   197→func FetchData(req APIQueryRequest) (*APIQueryResponse, error) {
   198→	if req.From > req.To {
   199→		return nil, ErrInvalidTimeRange
   200→	}
   201→	if req.Cluster == "" && req.ForAllNodes != nil {
   202→		return nil, ErrEmptyCluster
   203→	}
   204→
   205→	req.WithData = true
   206→	ms := GetMemoryStore()
   207→	if ms == nil {
   208→		return nil, fmt.Errorf("[METRICSTORE]> memorystore not initialized")
   209→	}
   210→
   211→	response := APIQueryResponse{
   212→		Results: make([][]APIMetricData, 0, len(req.Queries)),
   213→	}
   214→	if req.ForAllNodes != nil {
   215→		nodes := ms.ListChildren([]string{req.Cluster})
   216→		for _, node := range nodes {
   217→			for _, metric := range req.ForAllNodes {
   218→				q := APIQuery{
   219→					Metric:   metric,
   220→					Hostname: node,
   221→				}
   222→				req.Queries = append(req.Queries, q)
   223→				response.Queries = append(response.Queries, q)
   224→			}
   225→		}
   226→	}
   227→
   228→	for _, query := range req.Queries {
   229→		sels := make([]util.Selector, 0, 1)
   230→		if query.Aggregate || query.Type == nil {
   231→			sel := util.Selector{{String: req.Cluster}, {String: query.Hostname}}
   232→			if query.Type != nil {
   233→				if len(query.TypeIds) == 1 {
   234→					sel = append(sel, util.SelectorElement{String: *query.Type + query.TypeIds[0]})
   235→				} else {
   236→					ids := make([]string, len(query.TypeIds))
   237→					for i, id := range query.TypeIds {
   238→						ids[i] = *query.Type + id
   239→					}
   240→					sel = append(sel, util.SelectorElement{Group: ids})
   241→				}
   242→
   243→				if query.SubType != nil {
   244→					if len(query.SubTypeIds) == 1 {
   245→						sel = append(sel, util.SelectorElement{String: *query.SubType + query.SubTypeIds[0]})
   246→					} else {
   247→						ids := make([]string, len(query.SubTypeIds))
   248→						for i, id := range query.SubTypeIds {
   249→							ids[i] = *query.SubType + id
   250→						}
   251→						sel = append(sel, util.SelectorElement{Group: ids})
   252→					}
   253→				}
   254→			}
   255→			sels = append(sels, sel)
   256→		} else {
   257→			for _, typeID := range query.TypeIds {
   258→				if query.SubType != nil {
   259→					for _, subTypeID := range query.SubTypeIds {
   260→						sels = append(sels, util.Selector{
   261→							{String: req.Cluster},
   262→							{String: query.Hostname},
   263→							{String: *query.Type + typeID},
   264→							{String: *query.SubType + subTypeID},
   265→						})
   266→					}
   267→				} else {
   268→					sels = append(sels, util.Selector{
   269→						{String: req.Cluster},
   270→						{String: query.Hostname},
   271→						{String: *query.Type + typeID},
   272→					})
   273→				}
   274→			}
   275→		}
   276→
   277→		var err error
   278→		res := make([]APIMetricData, 0, len(sels))
   279→		for _, sel := range sels {
   280→			data := APIMetricData{}
   281→
   282→			data.Data, data.From, data.To, data.Resolution, err = ms.Read(sel, query.Metric, req.From, req.To, query.Resolution)
   283→			if err != nil {
   284→				// Skip Error If Just Missing Host or Metric, Continue
   285→				// Empty Return For Metric Handled Gracefully By Frontend
   286→				if err != ErrNoHostOrMetric {
   287→					msg := err.Error()
   288→					data.Error = &msg
   289→					res = append(res, data)
   290→				} else {
   291→					cclog.Warnf("failed to fetch '%s' from host '%s' (cluster: %s): %s", query.Metric, query.Hostname, req.Cluster, err.Error())
   292→				}
   293→				continue
   294→			}
   295→
   296→			if req.WithStats {
   297→				data.AddStats()
   298→			}
   299→			if query.ScaleFactor != 0 {
   300→				data.ScaleBy(query.ScaleFactor)
   301→			}
   302→			if req.WithPadding {
   303→				data.PadDataWithNull(ms, req.From, req.To, query.Metric)
   304→			}
   305→			if !req.WithData {
   306→				data.Data = nil
   307→			}
   308→			res = append(res, data)
   309→		}
   310→		response.Results = append(response.Results, res)
   311→	}
   312→
   313→	return &response, nil
   314→}
   315→


<persisted-output>
Output too large (50.6KB). Full output saved to: /Users/jan/.claude/projects/-Users-jan-prg-ClusterCockpit-cc-backend/eec1fc87-d8f6-42f6-b296-d7a2f79af002/tool-results/REDACTED.txt

Preview (first 2KB):
     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// This file implements high-level query functions for loading job metric data
     7→// with automatic scope transformation and aggregation.
     8→//
     9→// Key Concepts:
    10→//
    11→// Metric Scopes: Metrics are collected at different granularities (native scope):
    12→//   - HWThread: Per hardware thread
    13→//   - Core: Per CPU core
    14→//   - Socket: Per CPU socket
    15→//   - MemoryDomain: Per memory domain (NUMA)
    16→//   - Accelerator: Per GPU/accelerator
    17→//   - Node: Per compute node
    18→//
    19→// Scope Transformation: The buildQueries functions transform between native scope
    20→// and requested scope by:
    21→//   - Aggregating finer-grained data (e.g., HWThread → Core → Socket → Node)
    22→//   - Rejecting requests for finer granularity than available
    23→//   - Handling special cases (e.g., Accelerator metrics)
    24→//
    25→// Query Building: Constructs APIQuery structures with proper selectors (Type, TypeIds)
    26→// based on cluster topology and job resources.
    27→package metricstore
    28→
    29→import (
    30→	"context"
    31→	"fmt"
    32→	"strconv"
    33→	"strings"
    34→	"time"
    35→
    36→	"github.com/ClusterCockpit/cc-backend/pkg/archive"
    37→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    38→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    39→)
    40→
    41→type InternalMetricStore struct{}
    42→
    43→var MetricStoreHandle *InternalMetricStore
    44→
    45→// HealthCheck delegates to the internal MemoryStore's HealthCheck.
    46→func (ccms *InternalMetricStore) HealthCheck(cluster string,
    47→	nodes []string, metrics []string,
    48→) (map[string]HealthCheckResult, error) {
...
</persisted-output>


Explore the internal/metricstoreclient/ package thoroughly. I need to understand:
1. All files in this package
2. The CCMetricStore type and its methods (especially LoadData, LoadStats, LoadScopedStats, LoadNodeData, LoadNodeListData)
3. How it defines APIQuery and related types
4. The getTopology and getTopologyByNode helper methods
5. How this package relates to pkg/metricstore/

Focus on understanding the full structure and how it parallels pkg/metricstore/query.go.

Prompt:
Explore the internal/metricstoreclient/ package thoroughly. I need to understand:
1. All files in this package
2. The CCMetricStore type and its methods (especially LoadData, LoadStats, LoadScopedStats, LoadNodeData, LoadNodeListData)
3. How it defines APIQuery and related types
4. The getTopology and getTopologyByNode helper methods
5. How this package relates to pkg/metricstore/

Focus on understanding the full structure and how it parallels pkg/metricstore/query.go.


Error: EISDIR: illegal operation on a directory, read '/Users/jan/prg/ClusterCockpit/cc-backend/internal/metricstoreclient'


<tool_use_error>Sibling tool call errored</tool_use_error>


Explore metric store interface

Prompt:
Find the MetricDataRepository interface or similar interface that both pkg/metricstore and internal/metricstoreclient implement. Search for:
1. Any shared interface definition (likely in pkg/ or internal/)
2. How the two metric store implementations are registered/used
3. Any shared types between the two packages (APIQuery, etc.)
4. Check if there's a common types file or if each package defines its own

Search in pkg/metricstore/, internal/metricstoreclient/, and any relevant interface definition files. Also check if there's a metricdata or similar package.


.rw-r--r--@ 16k jan  4 Mar 10:09 cc-metric-store-queries.go
.rw-r--r--@ 28k jan  4 Mar 10:09 cc-metric-store.go


Find the MetricDataRepository interface or similar interface that both pkg/metricstore and internal/metricstoreclient implement. Search for:
1. Any shared interface definition (likely in pkg/ or internal/)
2. How the two metric store implementations are registered/used
3. Any shared types between the two packages (APIQuery, etc.)
4. Check if there's a common types file or if each package defines its own

Search in pkg/metricstore/, internal/metricstoreclient/, and any relevant interface definition files. Also check if there's a metricdata or similar package.

Prompt:
Find the MetricDataRepository interface or similar interface that both pkg/metricstore and internal/metricstoreclient implement. Search for:
1. Any shared interface definition (likely in pkg/ or internal/)
2. How the two metric store implementations are registered/used
3. Any shared types between the two packages (APIQuery, etc.)
4. Check if there's a common types file or if each package defines its own

Search in pkg/metricstore/, internal/metricstoreclient/, and any relevant interface definition files. Also check if there's a metricdata or similar package.


    40→
    41→type InternalMetricStore struct{}
    42→
    43→var MetricStoreHandle *InternalMetricStore
    44→
    45→// HealthCheck delegates to the internal MemoryStore's HealthCheck.
    46→func (ccms *InternalMetricStore) HealthCheck(cluster string,
    47→	nodes []string, metrics []string,
    48→) (map[string]HealthCheckResult, error) {
    49→	return GetMemoryStore().HealthCheck(cluster, nodes, metrics)
    50→}
    51→
    52→// TestLoadDataCallback allows tests to override LoadData behavior for testing purposes.
    53→// When set to a non-nil function, LoadData will call this function instead of the default implementation.
    54→var TestLoadDataCallback func(job *schema.Job, metrics []string, scopes []schema.MetricScope, ctx context.Context, resolution int) (schema.JobData, error)
    55→
    56→// LoadData loads metric data for a specific job with automatic scope transformation.
    57→//
    58→// This is the primary function for retrieving job metric data. It handles:
    59→//   - Building queries with scope transformation via buildQueries
    60→//   - Fetching data from the metric store
    61→//   - Organizing results by metric and scope
    62→//   - Converting NaN statistics to 0 for JSON compatibility
    63→//   - Partial error handling (returns data for successful queries even if some fail)
    64→//
    65→// Parameters:
    66→//   - job: Job metadata including cluster, resources, and time range
    67→//   - metrics: List of metric names to load
    68→//   - scopes: Requested metric scopes (will be transformed to match native scopes)
    69→//   - ctx: Context for cancellation (currently unused but reserved for future use)
    70→//   - resolution: Data resolution in seconds (0 for native resolution)
    71→//
    72→// Returns:
    73→//   - JobData: Map of metric → scope → JobMetric with time-series data and statistics
    74→//   - Error: Returns error if query building or fetching fails, or partial error listing failed hosts
    75→//
    76→// Example:
    77→//
    78→//	jobData, err := LoadData(job, []string{"cpu_load", "mem_used"}, []schema.MetricScope{schema.MetricScopeNode}, ctx, 60)
    79→func (ccms *InternalMetricStore) LoadData(
    80→	job *schema.Job,
    81→	metrics []string,
    82→	scopes []schema.MetricScope,
    83→	ctx context.Context,
    84→	resolution int,
    85→) (schema.JobData, error) {
    86→	if TestLoadDataCallback != nil {
    87→		return TestLoadDataCallback(job, metrics, scopes, ctx, resolution)
    88→	}
    89→
    90→	queries, assignedScope, err := buildQueries(job, metrics, scopes, int64(resolution))
    91→	if err != nil {
    92→		cclog.Errorf("Error while building queries for jobId %d, Metrics %v, Scopes %v: %s", job.JobID, metrics, scopes, err.Error())
    93→		return nil, err
    94→	}
    95→
    96→	// Verify assignment is correct - log any inconsistencies for debugging
    97→	if len(queries) != len(assignedScope) {
    98→		cclog.Errorf("Critical error: queries and assignedScope have different lengths after buildQueries: %d vs %d",
    99→			len(queries), len(assignedScope))
   100→	}
   101→
   102→	req := APIQueryRequest{
   103→		Cluster:   job.Cluster,
   104→		From:      job.StartTime,
   105→		To:        job.StartTime + int64(job.Duration),
   106→		Queries:   queries,
   107→		WithStats: true,
   108→		WithData:  true,
   109→	}
   110→
   111→	resBody, err := FetchData(req)
   112→	if err != nil {
   113→		cclog.Errorf("Error while fetching data : %s", err.Error())
   114→		return nil, err
   115→	}
   116→
   117→	var errors []string
   118→	jobData := make(schema.JobData)
   119→
   120→	// Add safety check for potential index out of range errors
   121→	if len(resBody.Results) != len(req.Queries) || len(assignedScope) != len(req.Queries) {
   122→		cclog.Warnf("Mismatch in query results count: queries=%d, results=%d, assignedScope=%d",
   123→			len(req.Queries), len(resBody.Results), len(assignedScope))
   124→		if len(resBody.Results) > len(req.Queries) {
   125→			resBody.Results = resBody.Results[:len(req.Queries)]
   126→		}
   127→		if len(assignedScope) > len(req.Queries) {
   128→			assignedScope = assignedScope[:len(req.Queries)]
   129→		}
   130→	}
   131→
   132→	for i, row := range resBody.Results {
   133→		// Safety check to prevent index out of range errors
   134→		if i >= len(req.Queries) || i >= len(assignedScope) {
   135→			cclog.Warnf("Index out of range prevented: i=%d, queries=%d, assignedScope=%d",
   136→				i, len(req.Queries), len(assignedScope))
   137→			continue
   138→		}
   139→
   140→		query := req.Queries[i]
   141→		metric := query.Metric
   142→		scope := assignedScope[i]
   143→		mc := archive.GetMetricConfig(job.Cluster, metric)
   144→
   145→		if mc == nil {
   146→			cclog.Warnf("Metric config not found for %s on cluster %s", metric, job.Cluster)
   147→			continue
   148→		}
   149→
   150→		if _, ok := jobData[metric]; !ok {
   151→			jobData[metric] = make(map[schema.MetricScope]*schema.JobMetric)
   152→		}
   153→
   154→		res := mc.Timestep
   155→		if len(row) > 0 {
   156→			res = int(row[0].Resolution)
   157→		}
   158→
   159→		jobMetric, ok := jobData[metric][scope]
   160→		if !ok {
   161→			jobMetric = &schema.JobMetric{
   162→				Unit:     mc.Unit,
   163→				Timestep: res,
   164→				Series:   make([]schema.Series, 0),
   165→			}
   166→			jobData[metric][scope] = jobMetric
   167→		}
   168→
   169→		for ndx, res := range row {
   170→			if res.Error != nil {
   171→				/* Build list for "partial errors", if any */
   172→				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
   173→				continue
   174→			}
   175→
   176→			id := (*string)(nil)
   177→			if query.Type != nil {
   178→				// Check if ndx is within the bounds of TypeIds slice
   179→				if ndx < len(query.TypeIds) {
   180→					id = new(string)
   181→					*id = query.TypeIds[ndx]
   182→				} else {
   183→					// Log the error but continue processing
   184→					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
   185→						ndx, len(query.TypeIds), query.Metric, query.Hostname)
   186→				}
   187→			}
   188→
   189→			sanitizeStats(&res)
   190→
   191→			jobMetric.Series = append(jobMetric.Series, schema.Series{
   192→				Hostname: query.Hostname,
   193→				ID:       id,
   194→				Statistics: schema.MetricStatistics{
   195→					Avg: float64(res.Avg),
   196→					Min: float64(res.Min),
   197→					Max: float64(res.Max),
   198→				},
   199→				Data: res.Data,
   200→			})
   201→		}
   202→
   203→		// So that one can later check len(jobData):
   204→		if len(jobMetric.Series) == 0 {
   205→			delete(jobData[metric], scope)
   206→			if len(jobData[metric]) == 0 {
   207→				delete(jobData, metric)
   208→			}
   209→		}
   210→	}
   211→
   212→	if len(errors) != 0 {
   213→		/* Returns list for "partial errors" */
   214→		return jobData, fmt.Errorf("METRICDATA/INTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
   215→	}
   216→	return jobData, nil
   217→}
   218→
   219→// Pre-converted scope strings avoid repeated string(MetricScope) allocations during
   220→// query construction. These are used in APIQuery.Type field throughout buildQueries
   221→// and buildNodeQueries functions. Converting once at package initialization improves
   222→// performance for high-volume query building.
   223→var (
   224→	hwthreadString     = string(schema.MetricScopeHWThread)
   225→	coreString         = string(schema.MetricScopeCore)
   226→	memoryDomainString = string(schema.MetricScopeMemoryDomain)
   227→	socketString       = string(schema.MetricScopeSocket)
   228→	acceleratorString  = string(schema.MetricScopeAccelerator)
   229→)
   230→
   231→// buildQueries constructs APIQuery structures with automatic scope transformation for a job.
   232→//
   233→// This function implements the core scope transformation logic, handling all combinations of
   234→// native metric scopes and requested scopes. It uses the cluster topology to determine which
   235→// hardware IDs to include in each query.
   236→//
   237→// Scope Transformation Rules:
   238→//   - If native scope >= requested scope: Aggregates data (Aggregate=true in APIQuery)
   239→//   - If native scope < requested scope: Returns error (cannot increase granularity)


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstore provides level.go: Hierarchical tree structure for metric storage.
     7→//
     8→// # Level Architecture
     9→//
    10→// The Level type forms a tree structure where each node represents a level in the
    11→// ClusterCockpit hierarchy: cluster → host → socket → core → hwthread, with special
    12→// nodes for memory domains and accelerators.
    13→//
    14→// Structure:
    15→//
    16→//	Root Level (cluster="emmy")
    17→//	├─ Level (host="node001")
    18→//	│  ├─ Level (socket="0")
    19→//	│  │  ├─ Level (core="0") [stores cpu0 metrics]
    20→//	│  │  └─ Level (core="1") [stores cpu1 metrics]
    21→//	│  └─ Level (socket="1")
    22→//	│     └─ ...
    23→//	└─ Level (host="node002")
    24→//	   └─ ...
    25→//
    26→// Each Level can:
    27→//   - Hold data (metrics slice of buffer pointers)
    28→//   - Have child nodes (children map[string]*Level)
    29→//   - Both simultaneously (inner nodes can store aggregated metrics)
    30→//
    31→// # Selector Paths
    32→//
    33→// Selectors are hierarchical paths: []string{"cluster", "host", "component"}.
    34→// Example: []string{"emmy", "node001", "cpu0"} navigates to the cpu0 core level.
    35→//
    36→// # Concurrency
    37→//
    38→// RWMutex protects children map and metrics slice. Read-heavy workload (metric reads)
    39→// uses RLock. Writes (new levels, buffer updates) use Lock. Double-checked locking
    40→// prevents races during level creation.
    41→package metricstore
    42→
    43→import (
    44→	"sync"
    45→	"time"
    46→	"unsafe"
    47→
    48→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    49→	"github.com/ClusterCockpit/cc-lib/v2/util"
    50→)
    51→
    52→// Level represents a node in the hierarchical metric storage tree.
    53→//
    54→// Can be both a leaf or inner node. Inner nodes hold data in 'metrics' for aggregated
    55→// values (e.g., socket-level metrics derived from core-level data). Named "Level"
    56→// instead of "node" to avoid confusion with cluster nodes (hosts).
    57→//
    58→// Fields:
    59→//   - children: Map of child level names to Level pointers (e.g., "cpu0" → Level)
    60→//   - metrics:  Slice of buffer pointers (one per metric, indexed by MetricConfig.offset)
    61→//   - lock:     RWMutex for concurrent access (read-heavy, write-rare)
    62→type Level struct {
    63→	children map[string]*Level
    64→	metrics  []*buffer
    65→	lock     sync.RWMutex
    66→}
    67→
    68→// findLevelOrCreate navigates to or creates the level specified by selector.
    69→//
    70→// Recursively descends the tree, creating missing levels as needed. Uses double-checked
    71→// locking: RLock first (fast path), then Lock if creation needed (slow path), then
    72→// re-check after acquiring Lock to handle races.
    73→//
    74→// Example selector: []string{"emmy", "node001", "cpu0"}
    75→// Navigates: root → emmy → node001 → cpu0, creating levels as needed.
    76→//
    77→// Parameters:
    78→//   - selector: Hierarchical path (consumed recursively, decreasing depth)
    79→//   - nMetrics: Number of metric slots to allocate in new levels
    80→//
    81→// Returns:
    82→//   - *Level: The target level (existing or newly created)
    83→//
    84→// Note: sync.Map may improve performance for high-concurrency writes, but current
    85→// approach suffices for read-heavy workload.
    86→func (l *Level) findLevelOrCreate(selector []string, nMetrics int) *Level {
    87→	if len(selector) == 0 {
    88→		return l
    89→	}
    90→
    91→	// Allow concurrent reads:
    92→	l.lock.RLock()
    93→	var child *Level
    94→	var ok bool
    95→	if l.children == nil {
    96→		// Children map needs to be created...
    97→		l.lock.RUnlock()
    98→	} else {
    99→		child, ok = l.children[selector[0]]
   100→		l.lock.RUnlock()


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstore provides buffer.go: Time-series data buffer implementation.
     7→//
     8→// # Buffer Architecture
     9→//
    10→// Each metric at each hierarchical level (cluster/host/cpu/etc.) uses a linked-list
    11→// chain of fixed-size buffers to store time-series data. This design:
    12→//
    13→//   - Avoids reallocation/copying when growing (new links added instead)
    14→//   - Enables efficient pooling (buffers returned to sync.Pool)
    15→//   - Supports traversal back in time (via prev pointers)
    16→//   - Maintains temporal ordering (newer data in later buffers)
    17→//
    18→// # Buffer Chain Example
    19→//
    20→//	[oldest buffer] <- prev -- [older] <- prev -- [newest buffer (head)]
    21→//	  start=1000               start=1512           start=2024
    22→//	  data=[v0...v511]         data=[v0...v511]     data=[v0...v42]
    23→//
    24→// When the head buffer reaches capacity (BufferCap = 512), a new buffer becomes
    25→// the new head and the old head is linked via prev.
    26→//
    27→// # Pooling Strategy
    28→//
    29→// sync.Pool reduces GC pressure for the common case (BufferCap-sized allocations).
    30→// Non-standard capacity buffers are not pooled (e.g., from checkpoint deserialization).
    31→//
    32→// # Time Alignment
    33→//
    34→// Timestamps are aligned to measurement frequency intervals:
    35→//
    36→//	index = (timestamp - buffer.start) / buffer.frequency
    37→//	actualTime = buffer.start + (frequency / 2) + (index * frequency)
    38→//
    39→// Missing data points are represented as NaN values. The read() function performs
    40→// linear interpolation where possible.
    41→package metricstore
    42→
    43→import (
    44→	"errors"
    45→	"sync"
    46→	"time"
    47→
    48→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    49→)
    50→
    51→// BufferCap is the default buffer capacity.
    52→// buffer.data will only ever grow up to its capacity and a new link
    53→// in the buffer chain will be created if needed so that no copying
    54→// of data or reallocation needs to happen on writes.
    55→const BufferCap int = DefaultBufferCapacity
    56→
    57→// BufferPool is the global instance.
    58→// It is initialized immediately when the package loads.
    59→var bufferPool = NewPersistentBufferPool()
    60→
    61→type PersistentBufferPool struct {
    62→	pool []*buffer
    63→	mu   sync.Mutex
    64→}
    65→
    66→// NewPersistentBufferPool creates a dynamic pool for buffers.
    67→func NewPersistentBufferPool() *PersistentBufferPool {
    68→	return &PersistentBufferPool{
    69→		pool: make([]*buffer, 0),
    70→	}
    71→}
    72→
    73→func (p *PersistentBufferPool) Get() *buffer {
    74→	p.mu.Lock()
    75→	defer p.mu.Unlock()
    76→
    77→	n := len(p.pool)
    78→	if n == 0 {
    79→		// Pool is empty, allocate a new one
    80→		return &buffer{
    81→			data: make([]schema.Float, 0, BufferCap),
    82→		}
    83→	}
    84→
    85→	// Reuse existing buffer from the pool
    86→	b := p.pool[n-1]
    87→	p.pool[n-1] = nil // Avoid memory leak
    88→	p.pool = p.pool[:n-1]
    89→	return b
    90→}
    91→
    92→// Put returns b to the pool. The caller must set b.lastUsed = time.Now().Unix()
    93→// before calling Put so that Clean() can evict idle entries correctly.
    94→func (p *PersistentBufferPool) Put(b *buffer) {
    95→	// Reset the buffer before putting it back
    96→	b.data = b.data[:0]
    97→
    98→	p.mu.Lock()
    99→	defer p.mu.Unlock()
   100→