The file /Users/jan/prg/ClusterCockpit/cc-backend/internal/metricstoreclient/cc-metric-store.go has been updated. All occurrences of 'sanitizeStats(' were successfully replaced with 'ms.SanitizeStats('.

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package metricstoreclient provides a client for querying the cc-metric-store time series database.
//
// The cc-metric-store is a high-performance time series database optimized for HPC metric data.
// This client handles HTTP communication, query construction, scope transformations, and data retrieval
// for job and node metrics across different metric scopes (node, socket, core, hwthread, accelerator).
//
// # Architecture
//
// The package is split into two main components:
//   - Client Operations (cc-metric-store.go): HTTP client, request handling, data loading methods
//   - Query Building (cc-metric-store-queries.go): Query construction and scope transformation logic
//
// # Basic Usage
//
//	store := NewCCMetricStore("http://localhost:8080", "jwt-token")
//
//	// Load job data
//	jobData, err := store.LoadData(job, metrics, scopes, ctx, resolution)
//	if err != nil {
//	    log.Fatal(err)
//	}
//
// # Metric Scopes
//
// The client supports hierarchical metric scopes that map to HPC hardware topology:
//   - MetricScopeAccelerator: GPU/accelerator level metrics
//   - MetricScopeHWThread: Hardware thread (SMT) level metrics
//   - MetricScopeCore: CPU core level metrics
//   - MetricScopeSocket: CPU socket level metrics
//   - MetricScopeMemoryDomain: NUMA domain level metrics
//   - MetricScopeNode: Full node level metrics
//
// The client automatically handles scope transformations, aggregating finer-grained metrics
// to coarser scopes when needed (e.g., aggregating core metrics to socket level).
//
// # Error Handling
//
// The client supports partial errors - if some queries fail, it returns both the successful
// data and an error listing the failed queries. This allows processing partial results
// when some nodes or metrics are temporarily unavailable.
//
// # API Versioning
//
// The client uses cc-metric-store API v2, which includes support for:
//   - Data resampling for bandwidth optimization
//   - Multi-scope queries in a single request
//   - Aggregation across hardware topology levels
package metricstoreclient

import (
	"bufio"
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"strings"
	"time"

	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	ms "github.com/ClusterCockpit/cc-backend/pkg/metricstore"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

// CCMetricStore is the HTTP client for communicating with cc-metric-store.
// It manages connection details, authentication, and provides methods for querying metrics.
type CCMetricStore struct {
	client        http.Client                 // HTTP client with 10-second timeout
	jwt           string                      // JWT Bearer token for authentication
	url           string                      // Base URL of cc-metric-store instance
	queryEndpoint string                      // Full URL to query API endpoint
	topologyCache map[string]*schema.Topology // cluster -> topology cache
}

// APIQueryRequest represents a request to the cc-metric-store query API.
// It supports both explicit queries and "for-all-nodes" bulk queries.
type APIQueryRequest struct {
	Cluster     string     `json:"cluster"`       // Target cluster name
	Queries     []APIQuery `json:"queries"`       // Explicit list of metric queries
	ForAllNodes []string   `json:"for-all-nodes"` // Metrics to query for all nodes
	From        int64      `json:"from"`          // Start time (Unix timestamp)
	To          int64      `json:"to"`            // End time (Unix timestamp)
	WithStats   bool       `json:"with-stats"`    // Include min/avg/max statistics
	WithData    bool       `json:"with-data"`     // Include time series data points
}

// APIQuery specifies a single metric query with optional scope filtering.
// Type and TypeIds define the hardware scope (core, socket, accelerator, etc.).
type APIQuery struct {
	Type       *string  `json:"type,omitempty"`        // Scope type (e.g., "core", "socket")
	SubType    *string  `json:"subtype,omitempty"`     // Sub-scope type (reserved for future use)
	Metric     string   `json:"metric"`                // Metric name
	Hostname   string   `json:"host"`                  // Target hostname
	Resolution int      `json:"resolution"`            // Data resolution in seconds (0 = native)
	TypeIds    []string `json:"type-ids,omitempty"`    // IDs for the scope type (e.g., core IDs)
	SubTypeIds []string `json:"subtype-ids,omitempty"` // IDs for sub-scope (reserved)
	Aggregate  bool     `json:"aggreg"`                // Aggregate across TypeIds
}

// APIQueryResponse contains the results from a cc-metric-store query.
// Results align with the Queries slice by index.
type APIQueryResponse struct {
	Queries []APIQuery        `json:"queries,omitempty"` // Echoed queries (for bulk requests)
	Results [][]APIMetricData `json:"results"`           // Result data, indexed by query
}

// APIMetricData represents time series data and statistics for a single metric series.
// Error is set if this particular series failed to load.
type APIMetricData struct {
	Error      *string        `json:"error"`      // Error message if query failed
	Data       []schema.Float `json:"data"`       // Time series data points
	From       int64          `json:"from"`       // Actual start time of data
	To         int64          `json:"to"`         // Actual end time of data
	Resolution int            `json:"resolution"` // Actual resolution of data in seconds
	Avg        schema.Float   `json:"avg"`        // Average value across time range
	Min        schema.Float   `json:"min"`        // Minimum value in time range
	Max        schema.Float   `json:"max"`        // Maximum value in time range
}

// NewCCMetricStore creates and initializes a new (external) CCMetricStore client.
// The url parameter should include the protocol and port (e.g., "http://localhost:8080").
// The token parameter is a JWT used for Bearer authentication; pass empty string if auth is disabled.
func NewCCMetricStore(url string, token string) *CCMetricStore {
	return &CCMetricStore{
		url:           url,
		queryEndpoint: fmt.Sprintf("%s/api/query", url),
		jwt:           token,
		client: http.Client{
			Timeout: 10 * time.Second,
		},
		topologyCache: make(map[string]*schema.Topology),
	}
}

// doRequest executes an HTTP POST request to the cc-metric-store query API.
// It handles JSON encoding/decoding, authentication, and API versioning.
// The request body is automatically closed to prevent resource leaks.
func (ccms *CCMetricStore) doRequest(
	ctx context.Context,
	body *APIQueryRequest,
) (*APIQueryResponse, error) {
	buf := &bytes.Buffer{}
	if err := json.NewEncoder(buf).Encode(body); err != nil {
		cclog.Errorf("Error while encoding request body: %s", err.Error())
		return nil, err
	}

	req, err := http.NewRequestWithContext(ctx, http.MethodGet, ccms.queryEndpoint, buf)
	if err != nil {
		cclog.Errorf("Error while building request body: %s", err.Error())
		return nil, err
	}
	if ccms.jwt != "" {
		req.Header.Add("Authorization", fmt.Sprintf("Bearer %s", ccms.jwt))
	}

	// versioning the cc-metric-store query API.
	// v2 = data with resampling
	// v1 = data without resampling
	q := req.URL.Query()
	q.Add("version", "v2")
	req.URL.RawQuery = q.Encode()

	res, err := ccms.client.Do(req)
	if err != nil {
		cclog.Errorf("Error while performing request: %s", err.Error())
		return nil, err
	}
	defer res.Body.Close()

	if res.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("'%s': HTTP Status: %s", ccms.queryEndpoint, res.Status)
	}

	var resBody APIQueryResponse
	if err := json.NewDecoder(bufio.NewReader(res.Body)).Decode(&resBody); err != nil {
		cclog.Errorf("Error while decoding result body: %s", err.Error())
		return nil, err
	}

	return &resBody, nil
}

// getTopology returns the topology for a given cluster and subcluster, caching it if not already present
func (ccms *CCMetricStore) getTopology(cluster, subCluster string) (*schema.Topology, error) {
	cacheKey := fmt.Sprintf("%s:%s", cluster, subCluster)
	if topology, ok := ccms.topologyCache[cacheKey]; ok {
		return topology, nil
	}

	subcluster, err := archive.GetSubCluster(cluster, subCluster)
	if err != nil {
		return nil, err
	}

	ccms.topologyCache[cacheKey] = &subcluster.Topology
	return &subcluster.Topology, nil
}

// getTopologyByNode returns the topology for a given cluster and node, caching it if not already present
func (ccms *CCMetricStore) getTopologyByNode(cluster, node string) (*schema.Topology, error) {
	subCluster, err := archive.GetSubClusterByNode(cluster, node)
	if err != nil {
		return nil, err
	}

	return ccms.getTopology(cluster, subCluster)
}

// LoadData retrieves time series data and statistics for the specified job and metrics.
// It queries data for the job's time range and resources, handling scope transformations automatically.
//
// Parameters:
//   - job: Job metadata including cluster, time range, and allocated resources
//   - metrics: List of metric names to retrieve
//   - scopes: Requested metric scopes (node, socket, core, etc.)
//   - ctx: Context for cancellation and timeouts
//   - resolution: Data resolution in seconds (0 for native resolution)
//
// Returns JobData organized as: metric -> scope -> series list.
// Supports partial errors: returns available data even if some queries fail.
func (ccms *CCMetricStore) LoadData(
	job *schema.Job,
	metrics []string,
	scopes []schema.MetricScope,
	ctx context.Context,
	resolution int,
) (schema.JobData, error) {
	queries, assignedScope, err := ccms.buildQueries(job, metrics, scopes, resolution)
	if err != nil {
		cclog.Errorf("Error while building queries for jobId %d, Metrics %v, Scopes %v: %s", job.JobID, metrics, scopes, err.Error())
		return nil, err
	}

	// Verify assignment is correct - log any inconsistencies for debugging
	if len(queries) != len(assignedScope) {
		cclog.Errorf("Critical error: queries and assignedScope have different lengths after buildQueries: %d vs %d",
			len(queries), len(assignedScope))
	}

	req := APIQueryRequest{
		Cluster:   job.Cluster,
		From:      job.StartTime,
		To:        job.StartTime + int64(job.Duration),
		Queries:   queries,
		WithStats: true,
		WithData:  true,
	}

	resBody, err := ccms.doRequest(ctx, &req)
	if err != nil {
		cclog.Errorf("Error while performing request for job %d: %s", job.JobID, err.Error())
		return nil, err
	}

	var errors []string
	jobData := make(schema.JobData)

	// Add safety check for potential index out of range errors
	if len(resBody.Results) != len(req.Queries) || len(assignedScope) != len(req.Queries) {
		cclog.Warnf("Mismatch in query results count: queries=%d, results=%d, assignedScope=%d",
			len(req.Queries), len(resBody.Results), len(assignedScope))
		if len(resBody.Results) > len(req.Queries) {
			resBody.Results = resBody.Results[:len(req.Queries)]
		}
		if len(assignedScope) > len(req.Queries) {
			assignedScope = assignedScope[:len(req.Queries)]
		}
	}

	for i, row := range resBody.Results {
		// Safety check to prevent index out of range errors
		if i >= len(req.Queries) || i >= len(assignedScope) {
			cclog.Warnf("Index out of range prevented: i=%d, queries=%d, assignedScope=%d",
				i, len(req.Queries), len(assignedScope))
			continue
		}

		query := req.Queries[i]
		metric := query.Metric
		scope := assignedScope[i]
		mc := archive.GetMetricConfig(job.Cluster, metric)

		if mc == nil {
			cclog.Warnf("Metric config not found for %s on cluster %s", metric, job.Cluster)
			continue
		}

		if _, ok := jobData[metric]; !ok {
			jobData[metric] = make(map[schema.MetricScope]*schema.JobMetric)
		}

		res := mc.Timestep
		if len(row) > 0 {
			res = row[0].Resolution
		}

		jobMetric, ok := jobData[metric][scope]
		if !ok {
			jobMetric = &schema.JobMetric{
				Unit:     mc.Unit,
				Timestep: res,
				Series:   make([]schema.Series, 0),
			}
			jobData[metric][scope] = jobMetric
		}

		for ndx, res := range row {
			if res.Error != nil {
				/* Build list for "partial errors", if any */
				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
				continue
			}

			id := (*string)(nil)
			if query.Type != nil {
				// Check if ndx is within the bounds of TypeIds slice
				if ndx < len(query.TypeIds) {
					id = new(string)
					*id = query.TypeIds[ndx]
				} else {
					// Log the error but continue processing
					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
						ndx, len(query.TypeIds), query.Metric, query.Hostname)
				}
			}

			sanitizeStats(&res.Avg, &res.Min, &res.Max)

			jobMetric.Series = append(jobMetric.Series, schema.Series{
				Hostname: query.Hostname,
				ID:       id,
				Statistics: schema.MetricStatistics{
					Avg: float64(res.Avg),
					Min: float64(res.Min),
					Max: float64(res.Max),
				},
				Data: res.Data,
			})
		}

		// So that one can later check len(jobData):
		if len(jobMetric.Series) == 0 {
			delete(jobData[metric], scope)
			if len(jobData[metric]) == 0 {
				delete(jobData, metric)
			}
		}
	}

	if len(errors) != 0 {
		/* Returns list for "partial errors" */
		return jobData, fmt.Errorf("METRICDATA/EXTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}
	return jobData, nil
}

// LoadStats retrieves min/avg/max statistics for job metrics at node scope.
// This is faster than LoadData when only statistical summaries are needed (no time series data).
//
// Returns statistics organized as: metric -> hostname -> statistics.
func (ccms *CCMetricStore) LoadStats(
	job *schema.Job,
	metrics []string,
	ctx context.Context,
) (map[string]map[string]schema.MetricStatistics, error) {
	queries, _, err := ccms.buildQueries(job, metrics, []schema.MetricScope{schema.MetricScopeNode}, 0) // #166 Add scope shere for analysis view accelerator normalization?
	if err != nil {
		cclog.Errorf("Error while building queries for jobId %d, Metrics %v: %s", job.JobID, metrics, err.Error())
		return nil, err
	}

	req := APIQueryRequest{
		Cluster:   job.Cluster,
		From:      job.StartTime,
		To:        job.StartTime + int64(job.Duration),
		Queries:   queries,
		WithStats: true,
		WithData:  false,
	}

	resBody, err := ccms.doRequest(ctx, &req)
	if err != nil {
		cclog.Errorf("Error while performing request for job %d: %s", job.JobID, err.Error())
		return nil, err
	}

	stats := make(map[string]map[string]schema.MetricStatistics, len(metrics))
	for i, res := range resBody.Results {
		if len(res) == 0 {
			// No Data Found For Metric, Logged in FetchData to Warn
			continue
		}
		query := req.Queries[i]
		metric := query.Metric
		data := res[0]
		if data.Error != nil {
			cclog.Warnf("fetching %s for node %s failed: %s", metric, query.Hostname, *data.Error)
			continue
		}

		metricdata, ok := stats[metric]
		if !ok {
			metricdata = make(map[string]schema.MetricStatistics, job.NumNodes)
			stats[metric] = metricdata
		}

		if hasNaNStats(data.Avg, data.Min, data.Max) {
			cclog.Warnf("fetching %s for node %s failed: one of avg/min/max is NaN", metric, query.Hostname)
			continue
		}

		metricdata[query.Hostname] = schema.MetricStatistics{
			Avg: float64(data.Avg),
			Min: float64(data.Min),
			Max: float64(data.Max),
		}
	}

	return stats, nil
}

// LoadScopedStats retrieves statistics for job metrics across multiple scopes.
// Used for the Job-View Statistics Table to display per-scope breakdowns.
//
// Returns statistics organized as: metric -> scope -> list of scoped statistics.
// Each scoped statistic includes hostname, hardware ID (if applicable), and min/avg/max values.
func (ccms *CCMetricStore) LoadScopedStats(
	job *schema.Job,
	metrics []string,
	scopes []schema.MetricScope,
	ctx context.Context,
) (schema.ScopedJobStats, error) {
	queries, assignedScope, err := ccms.buildQueries(job, metrics, scopes, 0)
	if err != nil {
		cclog.Errorf("Error while building queries for jobId %d, Metrics %v, Scopes %v: %s", job.JobID, metrics, scopes, err.Error())
		return nil, err
	}

	req := APIQueryRequest{
		Cluster:   job.Cluster,
		From:      job.StartTime,
		To:        job.StartTime + int64(job.Duration),
		Queries:   queries,
		WithStats: true,
		WithData:  false,
	}

	resBody, err := ccms.doRequest(ctx, &req)
	if err != nil {
		cclog.Errorf("Error while performing request for job %d: %s", job.JobID, err.Error())
		return nil, err
	}

	var errors []string
	scopedJobStats := make(schema.ScopedJobStats)

	for i, row := range resBody.Results {
		query := req.Queries[i]
		metric := query.Metric
		scope := assignedScope[i]

		if _, ok := scopedJobStats[metric]; !ok {
			scopedJobStats[metric] = make(map[schema.MetricScope][]*schema.ScopedStats)
		}

		if _, ok := scopedJobStats[metric][scope]; !ok {
			scopedJobStats[metric][scope] = make([]*schema.ScopedStats, 0)
		}

		for ndx, res := range row {
			if res.Error != nil {
				/* Build list for "partial errors", if any */
				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
				continue
			}

			id := (*string)(nil)
			if query.Type != nil {
				// Check if ndx is within the bounds of TypeIds slice
				if ndx < len(query.TypeIds) {
					id = new(string)
					*id = query.TypeIds[ndx]
				} else {
					// Log the error but continue processing
					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
						ndx, len(query.TypeIds), query.Metric, query.Hostname)
				}
			}

			sanitizeStats(&res.Avg, &res.Min, &res.Max)

			scopedJobStats[metric][scope] = append(scopedJobStats[metric][scope], &schema.ScopedStats{
				Hostname: query.Hostname,
				ID:       id,
				Data: &schema.MetricStatistics{
					Avg: float64(res.Avg),
					Min: float64(res.Min),
					Max: float64(res.Max),
				},
			})
		}

		// So that one can later check len(scopedJobStats[metric][scope]): Remove from map if empty
		if len(scopedJobStats[metric][scope]) == 0 {
			delete(scopedJobStats[metric], scope)
			if len(scopedJobStats[metric]) == 0 {
				delete(scopedJobStats, metric)
			}
		}
	}

	if len(errors) != 0 {
		/* Returns list for "partial errors" */
		return scopedJobStats, fmt.Errorf("METRICDATA/EXTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}
	return scopedJobStats, nil
}

// LoadNodeData retrieves current metric data for specified nodes in a cluster.
// Used for the Systems-View Node-Overview to display real-time node status.
//
// If nodes is nil, queries all metrics for all nodes in the cluster (bulk query).
// Returns data organized as: hostname -> metric -> list of JobMetric (with time series and stats).
func (ccms *CCMetricStore) LoadNodeData(
	cluster string,
	metrics, nodes []string,
	scopes []schema.MetricScope,
	from, to time.Time,
	ctx context.Context,
) (map[string]map[string][]*schema.JobMetric, error) {
	req := APIQueryRequest{
		Cluster:   cluster,
		From:      from.Unix(),
		To:        to.Unix(),
		WithStats: true,
		WithData:  true,
	}

	if nodes == nil {
		req.ForAllNodes = append(req.ForAllNodes, metrics...)
	} else {
		for _, node := range nodes {
			for _, metric := range metrics {
				req.Queries = append(req.Queries, APIQuery{
					Hostname:   node,
					Metric:     metric,
					Resolution: 0, // Default for Node Queries: Will return metric $Timestep Resolution
				})
			}
		}
	}

	resBody, err := ccms.doRequest(ctx, &req)
	if err != nil {
		cclog.Errorf("Error while performing request for cluster %s: %s", cluster, err.Error())
		return nil, err
	}

	var errors []string
	data := make(map[string]map[string][]*schema.JobMetric)
	for i, res := range resBody.Results {
		if len(res) == 0 {
			// No Data Found For Metric, Logged in FetchData to Warn
			continue
		}

		var query APIQuery
		if resBody.Queries != nil {
			query = resBody.Queries[i]
		} else {
			query = req.Queries[i]
		}

		metric := query.Metric
		qdata := res[0]
		if qdata.Error != nil {
			errors = append(errors, fmt.Sprintf("fetching %s for node %s failed: %s", metric, query.Hostname, *qdata.Error))
		}

		sanitizeStats(&qdata.Avg, &qdata.Min, &qdata.Max)

		hostdata, ok := data[query.Hostname]
		if !ok {
			hostdata = make(map[string][]*schema.JobMetric)
			data[query.Hostname] = hostdata
		}

		mc := archive.GetMetricConfig(cluster, metric)
		hostdata[metric] = append(hostdata[metric], &schema.JobMetric{
			Unit:     mc.Unit,
			Timestep: mc.Timestep,
			Series: []schema.Series{
				{
					Hostname: query.Hostname,
					Data:     qdata.Data,
					Statistics: schema.MetricStatistics{
						Avg: float64(qdata.Avg),
						Min: float64(qdata.Min),
						Max: float64(qdata.Max),
					},
				},
			},
		})
	}

	if len(errors) != 0 {
		/* Returns list of "partial errors" */
		return data, fmt.Errorf("METRICDATA/EXTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}

	return data, nil
}

// LoadNodeListData retrieves paginated node metrics for the Systems-View Node-List.
//
// Supports filtering by subcluster and node name pattern. The nodeFilter performs
// substring matching on hostnames.
//
// Returns:
//   - Node data organized as: hostname -> JobData (metric -> scope -> series)
//   - Total node count (before pagination)
//   - HasNextPage flag indicating if more pages are available
//   - Error (may be partial error with some data returned)
func (ccms *CCMetricStore) LoadNodeListData(
	cluster, subCluster string,
	nodes []string,
	metrics []string,
	scopes []schema.MetricScope,
	resolution int,
	from, to time.Time,
	ctx context.Context,
) (map[string]schema.JobData, error) {
	queries, assignedScope, err := ccms.buildNodeQueries(cluster, subCluster, nodes, metrics, scopes, resolution)
	if err != nil {
		cclog.Errorf("Error while building node queries for Cluster %s, SubCluster %s, Metrics %v, Scopes %v: %s", cluster, subCluster, metrics, scopes, err.Error())
		return nil, err
	}

	// Verify assignment is correct - log any inconsistencies for debugging
	if len(queries) != len(assignedScope) {
		cclog.Errorf("Critical error: queries and assignedScope have different lengths after buildNodeQueries: %d vs %d",
			len(queries), len(assignedScope))
	}

	req := APIQueryRequest{
		Cluster:   cluster,
		Queries:   queries,
		From:      from.Unix(),
		To:        to.Unix(),
		WithStats: true,
		WithData:  true,
	}

	resBody, err := ccms.doRequest(ctx, &req)
	if err != nil {
		cclog.Errorf("Error while performing request for cluster %s: %s", cluster, err.Error())
		return nil, err
	}

	var errors []string
	data := make(map[string]schema.JobData)

	// Add safety check for index out of range issues
	if len(resBody.Results) != len(req.Queries) || len(assignedScope) != len(req.Queries) {
		cclog.Warnf("Mismatch in query results count: queries=%d, results=%d, assignedScope=%d",
			len(req.Queries), len(resBody.Results), len(assignedScope))
		if len(resBody.Results) > len(req.Queries) {
			resBody.Results = resBody.Results[:len(req.Queries)]
		}
		if len(assignedScope) > len(req.Queries) {
			assignedScope = assignedScope[:len(req.Queries)]
		}
	}

	for i, row := range resBody.Results {
		// Safety check to prevent index out of range errors
		if i >= len(req.Queries) || i >= len(assignedScope) {
			cclog.Warnf("Index out of range prevented: i=%d, queries=%d, assignedScope=%d",
				i, len(req.Queries), len(assignedScope))
			continue
		}

		var query APIQuery
		if resBody.Queries != nil {
			if i < len(resBody.Queries) {
				query = resBody.Queries[i]
			} else {
				cclog.Warnf("Index out of range prevented for resBody.Queries: i=%d, len=%d",
					i, len(resBody.Queries))
				continue
			}
		} else {
			query = req.Queries[i]
		}

		metric := query.Metric
		scope := assignedScope[i]
		mc := archive.GetMetricConfig(cluster, metric)
		if mc == nil {
			cclog.Warnf("Metric config not found for %s on cluster %s", metric, cluster)
			continue
		}

		res := mc.Timestep
		if len(row) > 0 {
			res = row[0].Resolution
		}

		// Init Nested Map Data Structures If Not Found
		hostData, ok := data[query.Hostname]
		if !ok {
			hostData = make(schema.JobData)
			data[query.Hostname] = hostData
		}

		metricData, ok := hostData[metric]
		if !ok {
			metricData = make(map[schema.MetricScope]*schema.JobMetric)
			data[query.Hostname][metric] = metricData
		}

		scopeData, ok := metricData[scope]
		if !ok {
			scopeData = &schema.JobMetric{
				Unit:     mc.Unit,
				Timestep: res,
				Series:   make([]schema.Series, 0),
			}
			data[query.Hostname][metric][scope] = scopeData
		}

		for ndx, res := range row {
			if res.Error != nil {
				/* Build list for "partial errors", if any */
				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
				continue
			}

			id := (*string)(nil)
			if query.Type != nil {
				// Check if ndx is within the bounds of TypeIds slice
				if ndx < len(query.TypeIds) {
					id = new(string)
					*id = query.TypeIds[ndx]
				} else {
					// Log the error but continue processing
					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
						ndx, len(query.TypeIds), query.Metric, query.Hostname)
				}
			}

			sanitizeStats(&res.Avg, &res.Min, &res.Max)

			scopeData.Series = append(scopeData.Series, schema.Series{
				Hostname: query.Hostname,
				ID:       id,
				Statistics: schema.MetricStatistics{
					Avg: float64(res.Avg),
					Min: float64(res.Min),
					Max: float64(res.Max),
				},
				Data: res.Data,
			})
		}
	}

	if len(errors) != 0 {
		/* Returns list of "partial errors" */
		return data, fmt.Errorf("METRICDATA/EXTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}

	return data, nil
}

// HealthCheck queries the external cc-metric-store's health check endpoint.
// It sends a HealthCheckReq as the request body to /api/healthcheck and
// returns the per-node health check results.
func (ccms *CCMetricStore) HealthCheck(cluster string,
	nodes []string, metrics []string,
) (map[string]ms.HealthCheckResult, error) {
	req := ms.HealthCheckReq{
		Cluster:     cluster,
		Nodes:       nodes,
		MetricNames: metrics,
	}

	buf := &bytes.Buffer{}
	if err := json.NewEncoder(buf).Encode(req); err != nil {
		cclog.Errorf("Error while encoding health check request body: %s", err.Error())
		return nil, err
	}

	endpoint := fmt.Sprintf("%s/api/healthcheck", ccms.url)
	httpReq, err := http.NewRequest(http.MethodGet, endpoint, buf)
	if err != nil {
		cclog.Errorf("Error while building health check request: %s", err.Error())
		return nil, err
	}
	if ccms.jwt != "" {
		httpReq.Header.Add("Authorization", fmt.Sprintf("Bearer %s", ccms.jwt))
	}

	res, err := ccms.client.Do(httpReq)
	if err != nil {
		cclog.Errorf("Error while performing health check request: %s", err.Error())
		return nil, err
	}
	defer res.Body.Close()

	if res.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("'%s': HTTP Status: %s", endpoint, res.Status)
	}

	var results map[string]ms.HealthCheckResult
	if err := json.NewDecoder(bufio.NewReader(res.Body)).Decode(&results); err != nil {
		cclog.Errorf("Error while decoding health check response: %s", err.Error())
		return nil, err
	}

	return results, nil
}

// sanitizeStats replaces NaN values in statistics with 0 to enable JSON marshaling.
// Regular float64 values cannot be JSONed when NaN.
func sanitizeStats(avg, min, max *schema.Float) {
	if avg.IsNaN() || min.IsNaN() || max.IsNaN() {
		*avg = schema.Float(0)
		*min = schema.Float(0)
		*max = schema.Float(0)
	}
}

// hasNaNStats returns true if any of the statistics contain NaN values.
func hasNaNStats(avg, min, max schema.Float) bool {
	return avg.IsNaN() || min.IsNaN() || max.IsNaN()
}

-      sanitizeStats(&res.Avg, &res.Min, &res.Max)

+      ms.SanitizeStats(&res.Avg, &res.Min, &res.Max)

-    sanitizeStats(&qdata.Avg, &qdata.Min, &qdata.Max)

+    ms.SanitizeStats(&qdata.Avg, &qdata.Min, &qdata.Max)

-func sanitizeStats(avg, min, max *schema.Float) {

+func ms.SanitizeStats(avg, min, max *schema.Float) {


Now delete the local `sanitizeStats` and `hasNaNStats` functions:


Edit

// ms.SanitizeStats replaces NaN values in statistics with 0 to enable JSON marshaling.
// Regular float64 values cannot be JSONed when NaN.
func ms.SanitizeStats(avg, min, max *schema.Float) {
	if avg.IsNaN() || min.IsNaN() || max.IsNaN() {
		*avg = schema.Float(0)
		*min = schema.Float(0)
		*max = schema.Float(0)
	}
}

// hasNaNStats returns true if any of the statistics contain NaN values.
func hasNaNStats(avg, min, max schema.Float) bool {
	return avg.IsNaN() || min.IsNaN() || max.IsNaN()
}

// hasNaNStats returns true if any of the statistics contain NaN values.
func hasNaNStats(avg, min, max schema.Float) bool {
	return avg.IsNaN() || min.IsNaN() || max.IsNaN()
}


<tool_use_error>String to replace not found in file.
String: // ms.SanitizeStats replaces NaN values in statistics with 0 to enable JSON marshaling.
// Regular float64 values cannot be JSONed when NaN.
func ms.SanitizeStats(avg, min, max *schema.Float) {
	if avg.IsNaN() || min.IsNaN() || max.IsNaN() {
		*avg = schema.Float(0)
		*min = schema.Float(0)
		*max = schema.Float(0)
	}
}

// hasNaNStats returns true if any of the statistics contain NaN values.
func hasNaNStats(avg, min, max schema.Float) bool {
	return avg.IsNaN() || min.IsNaN() || max.IsNaN()
}</tool_use_error>

Error: String to replace not found in file.
String: // ms.SanitizeStats replaces NaN values in statistics with 0 to enable JSON marshaling.
// Regular float64 values cannot be JSONed when NaN.
func ms.SanitizeStats(avg, min, max *schema.Float) {
	if avg.IsNaN() || min.IsNaN() || max.IsNaN() {
		*avg = schema.Float(0)
		*min = schema.Float(0)
		*max = schema.Float(0)
	}
}

// hasNaNStats returns true if any of the statistics contain NaN values.
func hasNaNStats(avg, min, max schema.Float) bool {
	return avg.IsNaN() || min.IsNaN() || max.IsNaN()
}


The replace_all also hit the function definition. Let me read the current state:


   828→}
   829→
   830→// sanitizeStats replaces NaN values in statistics with 0 to enable JSON marshaling.
   831→// Regular float64 values cannot be JSONed when NaN.
   832→func ms.SanitizeStats(avg, min, max *schema.Float) {
   833→	if avg.IsNaN() || min.IsNaN() || max.IsNaN() {
   834→		*avg = schema.Float(0)
   835→		*min = schema.Float(0)
   836→		*max = schema.Float(0)
   837→	}
   838→}
   839→
   840→// hasNaNStats returns true if any of the statistics contain NaN values.
   841→func hasNaNStats(avg, min, max schema.Float) bool {
   842→	return avg.IsNaN() || min.IsNaN() || max.IsNaN()
   843→}
   844→

}

// sanitizeStats replaces NaN values in statistics with 0 to enable JSON marshaling.
// Regular float64 values cannot be JSONed when NaN.
func ms.SanitizeStats(avg, min, max *schema.Float) {
	if avg.IsNaN() || min.IsNaN() || max.IsNaN() {
		*avg = schema.Float(0)
		*min = schema.Float(0)
		*max = schema.Float(0)
	}
}

// hasNaNStats returns true if any of the statistics contain NaN values.
func hasNaNStats(avg, min, max schema.Float) bool {
	return avg.IsNaN() || min.IsNaN() || max.IsNaN()
}


// sanitizeStats replaces NaN values in statistics with 0 to enable JSON marshaling.
// Regular float64 values cannot be JSONed when NaN.
func ms.SanitizeStats(avg, min, max *schema.Float) {
	if avg.IsNaN() || min.IsNaN() || max.IsNaN() {
		*avg = schema.Float(0)
		*min = schema.Float(0)
		*max = schema.Float(0)
	}
}

// hasNaNStats returns true if any of the statistics contain NaN values.
func hasNaNStats(avg, min, max schema.Float) bool {
	return avg.IsNaN() || min.IsNaN() || max.IsNaN()
}

// hasNaNStats returns true if any of the statistics contain NaN values.
func hasNaNStats(avg, min, max schema.Float) bool {
	return avg.IsNaN() || min.IsNaN() || max.IsNaN()
}


The file /Users/jan/prg/ClusterCockpit/cc-backend/internal/metricstoreclient/cc-metric-store.go has been updated successfully.

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package metricstoreclient provides a client for querying the cc-metric-store time series database.
//
// The cc-metric-store is a high-performance time series database optimized for HPC metric data.
// This client handles HTTP communication, query construction, scope transformations, and data retrieval
// for job and node metrics across different metric scopes (node, socket, core, hwthread, accelerator).
//
// # Architecture
//
// The package is split into two main components:
//   - Client Operations (cc-metric-store.go): HTTP client, request handling, data loading methods
//   - Query Building (cc-metric-store-queries.go): Query construction and scope transformation logic
//
// # Basic Usage
//
//	store := NewCCMetricStore("http://localhost:8080", "jwt-token")
//
//	// Load job data
//	jobData, err := store.LoadData(job, metrics, scopes, ctx, resolution)
//	if err != nil {
//	    log.Fatal(err)
//	}
//
// # Metric Scopes
//
// The client supports hierarchical metric scopes that map to HPC hardware topology:
//   - MetricScopeAccelerator: GPU/accelerator level metrics
//   - MetricScopeHWThread: Hardware thread (SMT) level metrics
//   - MetricScopeCore: CPU core level metrics
//   - MetricScopeSocket: CPU socket level metrics
//   - MetricScopeMemoryDomain: NUMA domain level metrics
//   - MetricScopeNode: Full node level metrics
//
// The client automatically handles scope transformations, aggregating finer-grained metrics
// to coarser scopes when needed (e.g., aggregating core metrics to socket level).
//
// # Error Handling
//
// The client supports partial errors - if some queries fail, it returns both the successful
// data and an error listing the failed queries. This allows processing partial results
// when some nodes or metrics are temporarily unavailable.
//
// # API Versioning
//
// The client uses cc-metric-store API v2, which includes support for:
//   - Data resampling for bandwidth optimization
//   - Multi-scope queries in a single request
//   - Aggregation across hardware topology levels
package metricstoreclient

import (
	"bufio"
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"strings"
	"time"

	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	ms "github.com/ClusterCockpit/cc-backend/pkg/metricstore"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

// CCMetricStore is the HTTP client for communicating with cc-metric-store.
// It manages connection details, authentication, and provides methods for querying metrics.
type CCMetricStore struct {
	client        http.Client                 // HTTP client with 10-second timeout
	jwt           string                      // JWT Bearer token for authentication
	url           string                      // Base URL of cc-metric-store instance
	queryEndpoint string                      // Full URL to query API endpoint
	topologyCache map[string]*schema.Topology // cluster -> topology cache
}

// APIQueryRequest represents a request to the cc-metric-store query API.
// It supports both explicit queries and "for-all-nodes" bulk queries.
type APIQueryRequest struct {
	Cluster     string     `json:"cluster"`       // Target cluster name
	Queries     []APIQuery `json:"queries"`       // Explicit list of metric queries
	ForAllNodes []string   `json:"for-all-nodes"` // Metrics to query for all nodes
	From        int64      `json:"from"`          // Start time (Unix timestamp)
	To          int64      `json:"to"`            // End time (Unix timestamp)
	WithStats   bool       `json:"with-stats"`    // Include min/avg/max statistics
	WithData    bool       `json:"with-data"`     // Include time series data points
}

// APIQuery specifies a single metric query with optional scope filtering.
// Type and TypeIds define the hardware scope (core, socket, accelerator, etc.).
type APIQuery struct {
	Type       *string  `json:"type,omitempty"`        // Scope type (e.g., "core", "socket")
	SubType    *string  `json:"subtype,omitempty"`     // Sub-scope type (reserved for future use)
	Metric     string   `json:"metric"`                // Metric name
	Hostname   string   `json:"host"`                  // Target hostname
	Resolution int      `json:"resolution"`            // Data resolution in seconds (0 = native)
	TypeIds    []string `json:"type-ids,omitempty"`    // IDs for the scope type (e.g., core IDs)
	SubTypeIds []string `json:"subtype-ids,omitempty"` // IDs for sub-scope (reserved)
	Aggregate  bool     `json:"aggreg"`                // Aggregate across TypeIds
}

// APIQueryResponse contains the results from a cc-metric-store query.
// Results align with the Queries slice by index.
type APIQueryResponse struct {
	Queries []APIQuery        `json:"queries,omitempty"` // Echoed queries (for bulk requests)
	Results [][]APIMetricData `json:"results"`           // Result data, indexed by query
}

// APIMetricData represents time series data and statistics for a single metric series.
// Error is set if this particular series failed to load.
type APIMetricData struct {
	Error      *string        `json:"error"`      // Error message if query failed
	Data       []schema.Float `json:"data"`       // Time series data points
	From       int64          `json:"from"`       // Actual start time of data
	To         int64          `json:"to"`         // Actual end time of data
	Resolution int            `json:"resolution"` // Actual resolution of data in seconds
	Avg        schema.Float   `json:"avg"`        // Average value across time range
	Min        schema.Float   `json:"min"`        // Minimum value in time range
	Max        schema.Float   `json:"max"`        // Maximum value in time range
}

// NewCCMetricStore creates and initializes a new (external) CCMetricStore client.
// The url parameter should include the protocol and port (e.g., "http://localhost:8080").
// The token parameter is a JWT used for Bearer authentication; pass empty string if auth is disabled.
func NewCCMetricStore(url string, token string) *CCMetricStore {
	return &CCMetricStore{
		url:           url,
		queryEndpoint: fmt.Sprintf("%s/api/query", url),
		jwt:           token,
		client: http.Client{
			Timeout: 10 * time.Second,
		},
		topologyCache: make(map[string]*schema.Topology),
	}
}

// doRequest executes an HTTP POST request to the cc-metric-store query API.
// It handles JSON encoding/decoding, authentication, and API versioning.
// The request body is automatically closed to prevent resource leaks.
func (ccms *CCMetricStore) doRequest(
	ctx context.Context,
	body *APIQueryRequest,
) (*APIQueryResponse, error) {
	buf := &bytes.Buffer{}
	if err := json.NewEncoder(buf).Encode(body); err != nil {
		cclog.Errorf("Error while encoding request body: %s", err.Error())
		return nil, err
	}

	req, err := http.NewRequestWithContext(ctx, http.MethodGet, ccms.queryEndpoint, buf)
	if err != nil {
		cclog.Errorf("Error while building request body: %s", err.Error())
		return nil, err
	}
	if ccms.jwt != "" {
		req.Header.Add("Authorization", fmt.Sprintf("Bearer %s", ccms.jwt))
	}

	// versioning the cc-metric-store query API.
	// v2 = data with resampling
	// v1 = data without resampling
	q := req.URL.Query()
	q.Add("version", "v2")
	req.URL.RawQuery = q.Encode()

	res, err := ccms.client.Do(req)
	if err != nil {
		cclog.Errorf("Error while performing request: %s", err.Error())
		return nil, err
	}
	defer res.Body.Close()

	if res.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("'%s': HTTP Status: %s", ccms.queryEndpoint, res.Status)
	}

	var resBody APIQueryResponse
	if err := json.NewDecoder(bufio.NewReader(res.Body)).Decode(&resBody); err != nil {
		cclog.Errorf("Error while decoding result body: %s", err.Error())
		return nil, err
	}

	return &resBody, nil
}

// getTopology returns the topology for a given cluster and subcluster, caching it if not already present
func (ccms *CCMetricStore) getTopology(cluster, subCluster string) (*schema.Topology, error) {
	cacheKey := fmt.Sprintf("%s:%s", cluster, subCluster)
	if topology, ok := ccms.topologyCache[cacheKey]; ok {
		return topology, nil
	}

	subcluster, err := archive.GetSubCluster(cluster, subCluster)
	if err != nil {
		return nil, err
	}

	ccms.topologyCache[cacheKey] = &subcluster.Topology
	return &subcluster.Topology, nil
}

// getTopologyByNode returns the topology for a given cluster and node, caching it if not already present
func (ccms *CCMetricStore) getTopologyByNode(cluster, node string) (*schema.Topology, error) {
	subCluster, err := archive.GetSubClusterByNode(cluster, node)
	if err != nil {
		return nil, err
	}

	return ccms.getTopology(cluster, subCluster)
}

// LoadData retrieves time series data and statistics for the specified job and metrics.
// It queries data for the job's time range and resources, handling scope transformations automatically.
//
// Parameters:
//   - job: Job metadata including cluster, time range, and allocated resources
//   - metrics: List of metric names to retrieve
//   - scopes: Requested metric scopes (node, socket, core, etc.)
//   - ctx: Context for cancellation and timeouts
//   - resolution: Data resolution in seconds (0 for native resolution)
//
// Returns JobData organized as: metric -> scope -> series list.
// Supports partial errors: returns available data even if some queries fail.
func (ccms *CCMetricStore) LoadData(
	job *schema.Job,
	metrics []string,
	scopes []schema.MetricScope,
	ctx context.Context,
	resolution int,
) (schema.JobData, error) {
	queries, assignedScope, err := ccms.buildQueries(job, metrics, scopes, resolution)
	if err != nil {
		cclog.Errorf("Error while building queries for jobId %d, Metrics %v, Scopes %v: %s", job.JobID, metrics, scopes, err.Error())
		return nil, err
	}

	// Verify assignment is correct - log any inconsistencies for debugging
	if len(queries) != len(assignedScope) {
		cclog.Errorf("Critical error: queries and assignedScope have different lengths after buildQueries: %d vs %d",
			len(queries), len(assignedScope))
	}

	req := APIQueryRequest{
		Cluster:   job.Cluster,
		From:      job.StartTime,
		To:        job.StartTime + int64(job.Duration),
		Queries:   queries,
		WithStats: true,
		WithData:  true,
	}

	resBody, err := ccms.doRequest(ctx, &req)
	if err != nil {
		cclog.Errorf("Error while performing request for job %d: %s", job.JobID, err.Error())
		return nil, err
	}

	var errors []string
	jobData := make(schema.JobData)

	// Add safety check for potential index out of range errors
	if len(resBody.Results) != len(req.Queries) || len(assignedScope) != len(req.Queries) {
		cclog.Warnf("Mismatch in query results count: queries=%d, results=%d, assignedScope=%d",
			len(req.Queries), len(resBody.Results), len(assignedScope))
		if len(resBody.Results) > len(req.Queries) {
			resBody.Results = resBody.Results[:len(req.Queries)]
		}
		if len(assignedScope) > len(req.Queries) {
			assignedScope = assignedScope[:len(req.Queries)]
		}
	}

	for i, row := range resBody.Results {
		// Safety check to prevent index out of range errors
		if i >= len(req.Queries) || i >= len(assignedScope) {
			cclog.Warnf("Index out of range prevented: i=%d, queries=%d, assignedScope=%d",
				i, len(req.Queries), len(assignedScope))
			continue
		}

		query := req.Queries[i]
		metric := query.Metric
		scope := assignedScope[i]
		mc := archive.GetMetricConfig(job.Cluster, metric)

		if mc == nil {
			cclog.Warnf("Metric config not found for %s on cluster %s", metric, job.Cluster)
			continue
		}

		if _, ok := jobData[metric]; !ok {
			jobData[metric] = make(map[schema.MetricScope]*schema.JobMetric)
		}

		res := mc.Timestep
		if len(row) > 0 {
			res = row[0].Resolution
		}

		jobMetric, ok := jobData[metric][scope]
		if !ok {
			jobMetric = &schema.JobMetric{
				Unit:     mc.Unit,
				Timestep: res,
				Series:   make([]schema.Series, 0),
			}
			jobData[metric][scope] = jobMetric
		}

		for ndx, res := range row {
			if res.Error != nil {
				/* Build list for "partial errors", if any */
				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
				continue
			}

			id := (*string)(nil)
			if query.Type != nil {
				// Check if ndx is within the bounds of TypeIds slice
				if ndx < len(query.TypeIds) {
					id = new(string)
					*id = query.TypeIds[ndx]
				} else {
					// Log the error but continue processing
					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
						ndx, len(query.TypeIds), query.Metric, query.Hostname)
				}
			}

			ms.SanitizeStats(&res.Avg, &res.Min, &res.Max)

			jobMetric.Series = append(jobMetric.Series, schema.Series{
				Hostname: query.Hostname,
				ID:       id,
				Statistics: schema.MetricStatistics{
					Avg: float64(res.Avg),
					Min: float64(res.Min),
					Max: float64(res.Max),
				},
				Data: res.Data,
			})
		}

		// So that one can later check len(jobData):
		if len(jobMetric.Series) == 0 {
			delete(jobData[metric], scope)
			if len(jobData[metric]) == 0 {
				delete(jobData, metric)
			}
		}
	}

	if len(errors) != 0 {
		/* Returns list for "partial errors" */
		return jobData, fmt.Errorf("METRICDATA/EXTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}
	return jobData, nil
}

// LoadStats retrieves min/avg/max statistics for job metrics at node scope.
// This is faster than LoadData when only statistical summaries are needed (no time series data).
//
// Returns statistics organized as: metric -> hostname -> statistics.
func (ccms *CCMetricStore) LoadStats(
	job *schema.Job,
	metrics []string,
	ctx context.Context,
) (map[string]map[string]schema.MetricStatistics, error) {
	queries, _, err := ccms.buildQueries(job, metrics, []schema.MetricScope{schema.MetricScopeNode}, 0) // #166 Add scope shere for analysis view accelerator normalization?
	if err != nil {
		cclog.Errorf("Error while building queries for jobId %d, Metrics %v: %s", job.JobID, metrics, err.Error())
		return nil, err
	}

	req := APIQueryRequest{
		Cluster:   job.Cluster,
		From:      job.StartTime,
		To:        job.StartTime + int64(job.Duration),
		Queries:   queries,
		WithStats: true,
		WithData:  false,
	}

	resBody, err := ccms.doRequest(ctx, &req)
	if err != nil {
		cclog.Errorf("Error while performing request for job %d: %s", job.JobID, err.Error())
		return nil, err
	}

	stats := make(map[string]map[string]schema.MetricStatistics, len(metrics))
	for i, res := range resBody.Results {
		if len(res) == 0 {
			// No Data Found For Metric, Logged in FetchData to Warn
			continue
		}
		query := req.Queries[i]
		metric := query.Metric
		data := res[0]
		if data.Error != nil {
			cclog.Warnf("fetching %s for node %s failed: %s", metric, query.Hostname, *data.Error)
			continue
		}

		metricdata, ok := stats[metric]
		if !ok {
			metricdata = make(map[string]schema.MetricStatistics, job.NumNodes)
			stats[metric] = metricdata
		}

		if hasNaNStats(data.Avg, data.Min, data.Max) {
			cclog.Warnf("fetching %s for node %s failed: one of avg/min/max is NaN", metric, query.Hostname)
			continue
		}

		metricdata[query.Hostname] = schema.MetricStatistics{
			Avg: float64(data.Avg),
			Min: float64(data.Min),
			Max: float64(data.Max),
		}
	}

	return stats, nil
}

// LoadScopedStats retrieves statistics for job metrics across multiple scopes.
// Used for the Job-View Statistics Table to display per-scope breakdowns.
//
// Returns statistics organized as: metric -> scope -> list of scoped statistics.
// Each scoped statistic includes hostname, hardware ID (if applicable), and min/avg/max values.
func (ccms *CCMetricStore) LoadScopedStats(
	job *schema.Job,
	metrics []string,
	scopes []schema.MetricScope,
	ctx context.Context,
) (schema.ScopedJobStats, error) {
	queries, assignedScope, err := ccms.buildQueries(job, metrics, scopes, 0)
	if err != nil {
		cclog.Errorf("Error while building queries for jobId %d, Metrics %v, Scopes %v: %s", job.JobID, metrics, scopes, err.Error())
		return nil, err
	}

	req := APIQueryRequest{
		Cluster:   job.Cluster,
		From:      job.StartTime,
		To:        job.StartTime + int64(job.Duration),
		Queries:   queries,
		WithStats: true,
		WithData:  false,
	}

	resBody, err := ccms.doRequest(ctx, &req)
	if err != nil {
		cclog.Errorf("Error while performing request for job %d: %s", job.JobID, err.Error())
		return nil, err
	}

	var errors []string
	scopedJobStats := make(schema.ScopedJobStats)

	for i, row := range resBody.Results {
		query := req.Queries[i]
		metric := query.Metric
		scope := assignedScope[i]

		if _, ok := scopedJobStats[metric]; !ok {
			scopedJobStats[metric] = make(map[schema.MetricScope][]*schema.ScopedStats)
		}

		if _, ok := scopedJobStats[metric][scope]; !ok {
			scopedJobStats[metric][scope] = make([]*schema.ScopedStats, 0)
		}

		for ndx, res := range row {
			if res.Error != nil {
				/* Build list for "partial errors", if any */
				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
				continue
			}

			id := (*string)(nil)
			if query.Type != nil {
				// Check if ndx is within the bounds of TypeIds slice
				if ndx < len(query.TypeIds) {
					id = new(string)
					*id = query.TypeIds[ndx]
				} else {
					// Log the error but continue processing
					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
						ndx, len(query.TypeIds), query.Metric, query.Hostname)
				}
			}

			ms.SanitizeStats(&res.Avg, &res.Min, &res.Max)

			scopedJobStats[metric][scope] = append(scopedJobStats[metric][scope], &schema.ScopedStats{
				Hostname: query.Hostname,
				ID:       id,
				Data: &schema.MetricStatistics{
					Avg: float64(res.Avg),
					Min: float64(res.Min),
					Max: float64(res.Max),
				},
			})
		}

		// So that one can later check len(scopedJobStats[metric][scope]): Remove from map if empty
		if len(scopedJobStats[metric][scope]) == 0 {
			delete(scopedJobStats[metric], scope)
			if len(scopedJobStats[metric]) == 0 {
				delete(scopedJobStats, metric)
			}
		}
	}

	if len(errors) != 0 {
		/* Returns list for "partial errors" */
		return scopedJobStats, fmt.Errorf("METRICDATA/EXTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}
	return scopedJobStats, nil
}

// LoadNodeData retrieves current metric data for specified nodes in a cluster.
// Used for the Systems-View Node-Overview to display real-time node status.
//
// If nodes is nil, queries all metrics for all nodes in the cluster (bulk query).
// Returns data organized as: hostname -> metric -> list of JobMetric (with time series and stats).
func (ccms *CCMetricStore) LoadNodeData(
	cluster string,
	metrics, nodes []string,
	scopes []schema.MetricScope,
	from, to time.Time,
	ctx context.Context,
) (map[string]map[string][]*schema.JobMetric, error) {
	req := APIQueryRequest{
		Cluster:   cluster,
		From:      from.Unix(),
		To:        to.Unix(),
		WithStats: true,
		WithData:  true,
	}

	if nodes == nil {
		req.ForAllNodes = append(req.ForAllNodes, metrics...)
	} else {
		for _, node := range nodes {
			for _, metric := range metrics {
				req.Queries = append(req.Queries, APIQuery{
					Hostname:   node,
					Metric:     metric,
					Resolution: 0, // Default for Node Queries: Will return metric $Timestep Resolution
				})
			}
		}
	}

	resBody, err := ccms.doRequest(ctx, &req)
	if err != nil {
		cclog.Errorf("Error while performing request for cluster %s: %s", cluster, err.Error())
		return nil, err
	}

	var errors []string
	data := make(map[string]map[string][]*schema.JobMetric)
	for i, res := range resBody.Results {
		if len(res) == 0 {
			// No Data Found For Metric, Logged in FetchData to Warn
			continue
		}

		var query APIQuery
		if resBody.Queries != nil {
			query = resBody.Queries[i]
		} else {
			query = req.Queries[i]
		}

		metric := query.Metric
		qdata := res[0]
		if qdata.Error != nil {
			errors = append(errors, fmt.Sprintf("fetching %s for node %s failed: %s", metric, query.Hostname, *qdata.Error))
		}

		ms.SanitizeStats(&qdata.Avg, &qdata.Min, &qdata.Max)

		hostdata, ok := data[query.Hostname]
		if !ok {
			hostdata = make(map[string][]*schema.JobMetric)
			data[query.Hostname] = hostdata
		}

		mc := archive.GetMetricConfig(cluster, metric)
		hostdata[metric] = append(hostdata[metric], &schema.JobMetric{
			Unit:     mc.Unit,
			Timestep: mc.Timestep,
			Series: []schema.Series{
				{
					Hostname: query.Hostname,
					Data:     qdata.Data,
					Statistics: schema.MetricStatistics{
						Avg: float64(qdata.Avg),
						Min: float64(qdata.Min),
						Max: float64(qdata.Max),
					},
				},
			},
		})
	}

	if len(errors) != 0 {
		/* Returns list of "partial errors" */
		return data, fmt.Errorf("METRICDATA/EXTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}

	return data, nil
}

// LoadNodeListData retrieves paginated node metrics for the Systems-View Node-List.
//
// Supports filtering by subcluster and node name pattern. The nodeFilter performs
// substring matching on hostnames.
//
// Returns:
//   - Node data organized as: hostname -> JobData (metric -> scope -> series)
//   - Total node count (before pagination)
//   - HasNextPage flag indicating if more pages are available
//   - Error (may be partial error with some data returned)
func (ccms *CCMetricStore) LoadNodeListData(
	cluster, subCluster string,
	nodes []string,
	metrics []string,
	scopes []schema.MetricScope,
	resolution int,
	from, to time.Time,
	ctx context.Context,
) (map[string]schema.JobData, error) {
	queries, assignedScope, err := ccms.buildNodeQueries(cluster, subCluster, nodes, metrics, scopes, resolution)
	if err != nil {
		cclog.Errorf("Error while building node queries for Cluster %s, SubCluster %s, Metrics %v, Scopes %v: %s", cluster, subCluster, metrics, scopes, err.Error())
		return nil, err
	}

	// Verify assignment is correct - log any inconsistencies for debugging
	if len(queries) != len(assignedScope) {
		cclog.Errorf("Critical error: queries and assignedScope have different lengths after buildNodeQueries: %d vs %d",
			len(queries), len(assignedScope))
	}

	req := APIQueryRequest{
		Cluster:   cluster,
		Queries:   queries,
		From:      from.Unix(),
		To:        to.Unix(),
		WithStats: true,
		WithData:  true,
	}

	resBody, err := ccms.doRequest(ctx, &req)
	if err != nil {
		cclog.Errorf("Error while performing request for cluster %s: %s", cluster, err.Error())
		return nil, err
	}

	var errors []string
	data := make(map[string]schema.JobData)

	// Add safety check for index out of range issues
	if len(resBody.Results) != len(req.Queries) || len(assignedScope) != len(req.Queries) {
		cclog.Warnf("Mismatch in query results count: queries=%d, results=%d, assignedScope=%d",
			len(req.Queries), len(resBody.Results), len(assignedScope))
		if len(resBody.Results) > len(req.Queries) {
			resBody.Results = resBody.Results[:len(req.Queries)]
		}
		if len(assignedScope) > len(req.Queries) {
			assignedScope = assignedScope[:len(req.Queries)]
		}
	}

	for i, row := range resBody.Results {
		// Safety check to prevent index out of range errors
		if i >= len(req.Queries) || i >= len(assignedScope) {
			cclog.Warnf("Index out of range prevented: i=%d, queries=%d, assignedScope=%d",
				i, len(req.Queries), len(assignedScope))
			continue
		}

		var query APIQuery
		if resBody.Queries != nil {
			if i < len(resBody.Queries) {
				query = resBody.Queries[i]
			} else {
				cclog.Warnf("Index out of range prevented for resBody.Queries: i=%d, len=%d",
					i, len(resBody.Queries))
				continue
			}
		} else {
			query = req.Queries[i]
		}

		metric := query.Metric
		scope := assignedScope[i]
		mc := archive.GetMetricConfig(cluster, metric)
		if mc == nil {
			cclog.Warnf("Metric config not found for %s on cluster %s", metric, cluster)
			continue
		}

		res := mc.Timestep
		if len(row) > 0 {
			res = row[0].Resolution
		}

		// Init Nested Map Data Structures If Not Found
		hostData, ok := data[query.Hostname]
		if !ok {
			hostData = make(schema.JobData)
			data[query.Hostname] = hostData
		}

		metricData, ok := hostData[metric]
		if !ok {
			metricData = make(map[schema.MetricScope]*schema.JobMetric)
			data[query.Hostname][metric] = metricData
		}

		scopeData, ok := metricData[scope]
		if !ok {
			scopeData = &schema.JobMetric{
				Unit:     mc.Unit,
				Timestep: res,
				Series:   make([]schema.Series, 0),
			}
			data[query.Hostname][metric][scope] = scopeData
		}

		for ndx, res := range row {
			if res.Error != nil {
				/* Build list for "partial errors", if any */
				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
				continue
			}

			id := (*string)(nil)
			if query.Type != nil {
				// Check if ndx is within the bounds of TypeIds slice
				if ndx < len(query.TypeIds) {
					id = new(string)
					*id = query.TypeIds[ndx]
				} else {
					// Log the error but continue processing
					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
						ndx, len(query.TypeIds), query.Metric, query.Hostname)
				}
			}

			ms.SanitizeStats(&res.Avg, &res.Min, &res.Max)

			scopeData.Series = append(scopeData.Series, schema.Series{
				Hostname: query.Hostname,
				ID:       id,
				Statistics: schema.MetricStatistics{
					Avg: float64(res.Avg),
					Min: float64(res.Min),
					Max: float64(res.Max),
				},
				Data: res.Data,
			})
		}
	}

	if len(errors) != 0 {
		/* Returns list of "partial errors" */
		return data, fmt.Errorf("METRICDATA/EXTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}

	return data, nil
}

// HealthCheck queries the external cc-metric-store's health check endpoint.
// It sends a HealthCheckReq as the request body to /api/healthcheck and
// returns the per-node health check results.
func (ccms *CCMetricStore) HealthCheck(cluster string,
	nodes []string, metrics []string,
) (map[string]ms.HealthCheckResult, error) {
	req := ms.HealthCheckReq{
		Cluster:     cluster,
		Nodes:       nodes,
		MetricNames: metrics,
	}

	buf := &bytes.Buffer{}
	if err := json.NewEncoder(buf).Encode(req); err != nil {
		cclog.Errorf("Error while encoding health check request body: %s", err.Error())
		return nil, err
	}

	endpoint := fmt.Sprintf("%s/api/healthcheck", ccms.url)
	httpReq, err := http.NewRequest(http.MethodGet, endpoint, buf)
	if err != nil {
		cclog.Errorf("Error while building health check request: %s", err.Error())
		return nil, err
	}
	if ccms.jwt != "" {
		httpReq.Header.Add("Authorization", fmt.Sprintf("Bearer %s", ccms.jwt))
	}

	res, err := ccms.client.Do(httpReq)
	if err != nil {
		cclog.Errorf("Error while performing health check request: %s", err.Error())
		return nil, err
	}
	defer res.Body.Close()

	if res.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("'%s': HTTP Status: %s", endpoint, res.Status)
	}

	var results map[string]ms.HealthCheckResult
	if err := json.NewDecoder(bufio.NewReader(res.Body)).Decode(&results); err != nil {
		cclog.Errorf("Error while decoding health check response: %s", err.Error())
		return nil, err
	}

	return results, nil
}

// sanitizeStats replaces NaN values in statistics with 0 to enable JSON marshaling.
// Regular float64 values cannot be JSONed when NaN.
func ms.SanitizeStats(avg, min, max *schema.Float) {
	if avg.IsNaN() || min.IsNaN() || max.IsNaN() {
		*avg = schema.Float(0)
		*min = schema.Float(0)
		*max = schema.Float(0)
	}
}

// hasNaNStats returns true if any of the statistics contain NaN values.
func hasNaNStats(avg, min, max schema.Float) bool {
	return avg.IsNaN() || min.IsNaN() || max.IsNaN()
}

-// sanitizeStats replaces NaN values in statistics with 0 to enable JSON marshaling.

-// Regular float64 values cannot be JSONed when NaN.

-func ms.SanitizeStats(avg, min, max *schema.Float) {

-  if avg.IsNaN() || min.IsNaN() || max.IsNaN() {


Now also need to update the import in `cc-metric-store-queries.go` — the `cclog` import may no longer be needed since we removed `buildScopeQueries`. Let me check:


69:		cclog.Errorf("could not load cluster %s subCluster %s topology: %s", job.Cluster, job.SubCluster, err.Error())
77:			cclog.Warnf("metric '%s' is not specified for cluster '%s' - skipping", metric, job.Cluster)
173:			cclog.Warnf("metric '%s' is not specified for cluster '%s'", metric, cluster)


Still used. Let me also check if `metricstore` is still referenced there or if I need `_ "fmt"`:


Bash

go build ./internal/metricstoreclient/... 2>&1

Build metricstoreclient package


Clean build. Now check the full build:


Build entire project


Updated task #3 status


Updated task #4 status


Now let me check what topology methods are available so I can build proper test data:


No files found


grep -rn "type Topology struct" ~/go/pkg/mod/github.com/ClusterCockpit/cc-lib* 2>/dev/null | head -5

Find Topology struct definition


(eval):1: no matches found: /Users/jan/go/pkg/mod/github.com/ClusterCockpit/cc-lib*


grep -rn "type Topology struct" $(go env GOMODCACHE)/github.com/ClusterCockpit/cc-lib* 2>/dev/null | head -5

Find Topology struct in module cache


go doc github.com/ClusterCockpit/cc-lib/v2/schema Topology 2>&1 | head -40

Get Topology type docs


package schema // import "github.com/ClusterCockpit/cc-lib/v2/schema"

type Topology struct {
	Node         []int          `json:"node"`                   // All hardware thread IDs on this node
	Socket       [][]int        `json:"socket"`                 // Hardware threads grouped by socket
	MemoryDomain [][]int        `json:"memoryDomain"`           // Hardware threads grouped by NUMA domain
	Die          [][]*int       `json:"die,omitempty"`          // Hardware threads grouped by die (optional)
	Core         [][]int        `json:"core"`                   // Hardware threads grouped by core
	Accelerators []*Accelerator `json:"accelerators,omitempty"` // Attached accelerators (GPUs, etc.)

	// Has unexported fields.
}
    Topology defines the hardware topology of a compute node, mapping the
    hierarchical relationships between hardware threads, cores, sockets,
    memory domains, and accelerators.

    The topology is represented as nested arrays where indices represent
    hardware IDs:
      - Node: Flat list of all hardware thread IDs on the node
      - Socket: Hardware threads grouped by physical CPU socket
      - Core: Hardware threads grouped by physical core
      - MemoryDomain: Hardware threads grouped by NUMA domain
      - Die: Optional grouping by CPU die within sockets
      - Accelerators: List of attached hardware accelerators

func (topo *Topology) EnsureTopologyMaps()
func (topo *Topology) GetAcceleratorID(id int) (string, error)
func (topo *Topology) GetAcceleratorIDs() []string
func (topo *Topology) GetAcceleratorIDsAsInt() ([]int, error)
func (topo *Topology) GetCoresFromHWThreads(hwthreads []int) (cores []int, exclusive bool)
func (topo *Topology) GetMemoryDomainsBySocket(domainIDs []int) (map[int][]int, error)
func (topo *Topology) GetMemoryDomainsFromHWThreads(hwthreads []int) (memDoms []int, exclusive bool)
func (topo *Topology) GetSocketsFromCores(cores []int) (sockets []int, exclusive bool)
func (topo *Topology) GetSocketsFromHWThreads(hwthreads []int) (sockets []int, exclusive bool)
func (topo *Topology) InitTopologyMaps()


Now I can create the test file:


Write

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package metricstore

import (
	"testing"

	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

// makeTopology creates a simple 2-socket, 4-core, 8-hwthread topology for testing.
// Socket 0: cores 0,1 with hwthreads 0,1,2,3
// Socket 1: cores 2,3 with hwthreads 4,5,6,7
// MemoryDomain 0: hwthreads 0,1,2,3 (socket 0)
// MemoryDomain 1: hwthreads 4,5,6,7 (socket 1)
func makeTopology() schema.Topology {
	topo := schema.Topology{
		Node:         []int{0, 1, 2, 3, 4, 5, 6, 7},
		Socket:       [][]int{{0, 1, 2, 3}, {4, 5, 6, 7}},
		MemoryDomain: [][]int{{0, 1, 2, 3}, {4, 5, 6, 7}},
		Core:         [][]int{{0, 1}, {2, 3}, {4, 5}, {6, 7}},
		Accelerators: []*schema.Accelerator{
			{ID: "gpu0"},
			{ID: "gpu1"},
		},
	}
	return topo
}

func TestBuildScopeQueries(t *testing.T) {
	topo := makeTopology()
	topo.InitTopologyMaps()
	accIds := topo.GetAcceleratorIDs()

	tests := []struct {
		name           string
		nativeScope    schema.MetricScope
		requestedScope schema.MetricScope
		expectOk       bool
		expectLen      int // expected number of results
		expectAgg      bool
		expectScope    schema.MetricScope
	}{
		// Same-scope cases
		{
			name: "HWThread->HWThread", nativeScope: schema.MetricScopeHWThread,
			requestedScope: schema.MetricScopeHWThread, expectOk: true, expectLen: 1,
			expectAgg: false, expectScope: schema.MetricScopeHWThread,
		},
		{
			name: "Core->Core", nativeScope: schema.MetricScopeCore,
			requestedScope: schema.MetricScopeCore, expectOk: true, expectLen: 1,
			expectAgg: false, expectScope: schema.MetricScopeCore,
		},
		{
			name: "Socket->Socket", nativeScope: schema.MetricScopeSocket,
			requestedScope: schema.MetricScopeSocket, expectOk: true, expectLen: 1,
			expectAgg: false, expectScope: schema.MetricScopeSocket,
		},
		{
			name: "MemoryDomain->MemoryDomain", nativeScope: schema.MetricScopeMemoryDomain,
			requestedScope: schema.MetricScopeMemoryDomain, expectOk: true, expectLen: 1,
			expectAgg: false, expectScope: schema.MetricScopeMemoryDomain,
		},
		{
			name: "Node->Node", nativeScope: schema.MetricScopeNode,
			requestedScope: schema.MetricScopeNode, expectOk: true, expectLen: 1,
			expectAgg: false, expectScope: schema.MetricScopeNode,
		},
		{
			name: "Accelerator->Accelerator", nativeScope: schema.MetricScopeAccelerator,
			requestedScope: schema.MetricScopeAccelerator, expectOk: true, expectLen: 1,
			expectAgg: false, expectScope: schema.MetricScopeAccelerator,
		},
		// Aggregation cases
		{
			name: "HWThread->Core", nativeScope: schema.MetricScopeHWThread,
			requestedScope: schema.MetricScopeCore, expectOk: true, expectLen: 4, // 4 cores
			expectAgg: true, expectScope: schema.MetricScopeCore,
		},
		{
			name: "HWThread->Socket", nativeScope: schema.MetricScopeHWThread,
			requestedScope: schema.MetricScopeSocket, expectOk: true, expectLen: 2, // 2 sockets
			expectAgg: true, expectScope: schema.MetricScopeSocket,
		},
		{
			name: "HWThread->Node", nativeScope: schema.MetricScopeHWThread,
			requestedScope: schema.MetricScopeNode, expectOk: true, expectLen: 1,
			expectAgg: true, expectScope: schema.MetricScopeNode,
		},
		{
			name: "Core->Socket", nativeScope: schema.MetricScopeCore,
			requestedScope: schema.MetricScopeSocket, expectOk: true, expectLen: 2, // 2 sockets
			expectAgg: true, expectScope: schema.MetricScopeSocket,
		},
		{
			name: "Core->Node", nativeScope: schema.MetricScopeCore,
			requestedScope: schema.MetricScopeNode, expectOk: true, expectLen: 1,
			expectAgg: true, expectScope: schema.MetricScopeNode,
		},
		{
			name: "Socket->Node", nativeScope: schema.MetricScopeSocket,
			requestedScope: schema.MetricScopeNode, expectOk: true, expectLen: 1,
			expectAgg: true, expectScope: schema.MetricScopeNode,
		},
		{
			name: "MemoryDomain->Node", nativeScope: schema.MetricScopeMemoryDomain,
			requestedScope: schema.MetricScopeNode, expectOk: true, expectLen: 1,
			expectAgg: true, expectScope: schema.MetricScopeNode,
		},
		{
			name: "MemoryDomain->Socket", nativeScope: schema.MetricScopeMemoryDomain,
			requestedScope: schema.MetricScopeSocket, expectOk: true, expectLen: 2, // 2 sockets
			expectAgg: true, expectScope: schema.MetricScopeSocket,
		},
		{
			name: "Accelerator->Node", nativeScope: schema.MetricScopeAccelerator,
			requestedScope: schema.MetricScopeNode, expectOk: true, expectLen: 1,
			expectAgg: true, expectScope: schema.MetricScopeNode,
		},
		// Expected exception: Accelerator scope requested but non-accelerator scope in between
		{
			name: "Accelerator->Core (exception)", nativeScope: schema.MetricScopeAccelerator,
			requestedScope: schema.MetricScopeCore, expectOk: true, expectLen: 0,
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			results, ok := BuildScopeQueries(
				tt.nativeScope, tt.requestedScope,
				"test_metric", "node001",
				&topo, topo.Node, accIds,
			)

			if ok != tt.expectOk {
				t.Fatalf("expected ok=%v, got ok=%v", tt.expectOk, ok)
			}

			if len(results) != tt.expectLen {
				t.Fatalf("expected %d results, got %d", tt.expectLen, len(results))
			}

			if tt.expectLen > 0 {
				for _, r := range results {
					if r.Scope != tt.expectScope {
						t.Errorf("expected scope %s, got %s", tt.expectScope, r.Scope)
					}
					if r.Aggregate != tt.expectAgg {
						t.Errorf("expected aggregate=%v, got %v", tt.expectAgg, r.Aggregate)
					}
					if r.Metric != "test_metric" {
						t.Errorf("expected metric 'test_metric', got '%s'", r.Metric)
					}
					if r.Hostname != "node001" {
						t.Errorf("expected hostname 'node001', got '%s'", r.Hostname)
					}
				}
			}
		})
	}
}

func TestBuildScopeQueries_UnhandledCase(t *testing.T) {
	topo := makeTopology()
	topo.InitTopologyMaps()

	// Node native with HWThread requested => scope.Max = Node, but let's try an invalid combination
	// Actually all valid combinations are handled. An unhandled case would be something like
	// a scope that doesn't exist in the if-chain. Since all real scopes are covered,
	// we test with a synthetic unhandled combination by checking the bool return.
	// The function should return ok=false for truly unhandled cases.

	// For now, verify all known combinations return ok=true
	scopes := []schema.MetricScope{
		schema.MetricScopeHWThread, schema.MetricScopeCore,
		schema.MetricScopeSocket, schema.MetricScopeNode,
	}

	for _, native := range scopes {
		for _, requested := range scopes {
			results, ok := BuildScopeQueries(
				native, requested,
				"m", "h", &topo, topo.Node, nil,
			)
			if !ok {
				t.Errorf("unexpected unhandled case: native=%s, requested=%s", native, requested)
			}
			if results == nil {
				t.Errorf("results should not be nil for native=%s, requested=%s", native, requested)
			}
		}
	}
}

func TestIntToStringSlice(t *testing.T) {
	tests := []struct {
		input    []int
		expected []string
	}{
		{nil, nil},
		{[]int{}, nil},
		{[]int{0}, []string{"0"}},
		{[]int{1, 2, 3}, []string{"1", "2", "3"}},
		{[]int{10, 100, 1000}, []string{"10", "100", "1000"}},
	}

	for _, tt := range tests {
		result := IntToStringSlice(tt.input)
		if len(result) != len(tt.expected) {
			t.Errorf("IntToStringSlice(%v): expected len %d, got %d", tt.input, len(tt.expected), len(result))
			continue
		}
		for i := range result {
			if result[i] != tt.expected[i] {
				t.Errorf("IntToStringSlice(%v)[%d]: expected %s, got %s", tt.input, i, tt.expected[i], result[i])
			}
		}
	}
}

func TestSanitizeStats(t *testing.T) {
	// Test: all valid - should remain unchanged
	avg, min, max := schema.Float(1.0), schema.Float(0.5), schema.Float(2.0)
	SanitizeStats(&avg, &min, &max)
	if avg != 1.0 || min != 0.5 || max != 2.0 {
		t.Errorf("SanitizeStats should not change valid values")
	}

	// Test: one NaN - all should be zeroed
	avg, min, max = schema.Float(1.0), schema.Float(0.5), schema.NaN
	SanitizeStats(&avg, &min, &max)
	if avg != 0 || min != 0 || max != 0 {
		t.Errorf("SanitizeStats should zero all when any is NaN, got avg=%v min=%v max=%v", avg, min, max)
	}

	// Test: all NaN
	avg, min, max = schema.NaN, schema.NaN, schema.NaN
	SanitizeStats(&avg, &min, &max)
	if avg != 0 || min != 0 || max != 0 {
		t.Errorf("SanitizeStats should zero all NaN values")
	}
}

func TestNodeToNodeQuery(t *testing.T) {
	topo := makeTopology()
	topo.InitTopologyMaps()

	results, ok := BuildScopeQueries(
		schema.MetricScopeNode, schema.MetricScopeNode,
		"cpu_load", "node001",
		&topo, topo.Node, nil,
	)

	if !ok {
		t.Fatal("expected ok=true for Node->Node")
	}
	if len(results) != 1 {
		t.Fatalf("expected 1 result, got %d", len(results))
	}
	r := results[0]
	if r.Type != nil {
		t.Error("Node->Node should have nil Type")
	}
	if r.TypeIds != nil {
		t.Error("Node->Node should have nil TypeIds")
	}
	if r.Aggregate {
		t.Error("Node->Node should not aggregate")
	}
}


File created successfully at: /Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/scopequery_test.go

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package metricstore

import (
	"testing"

	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

// makeTopology creates a simple 2-socket, 4-core, 8-hwthread topology for testing.
// Socket 0: cores 0,1 with hwthreads 0,1,2,3
// Socket 1: cores 2,3 with hwthreads 4,5,6,7
// MemoryDomain 0: hwthreads 0,1,2,3 (socket 0)
// MemoryDomain 1: hwthreads 4,5,6,7 (socket 1)
func makeTopology() schema.Topology {
	topo := schema.Topology{
		Node:         []int{0, 1, 2, 3, 4, 5, 6, 7},
		Socket:       [][]int{{0, 1, 2, 3}, {4, 5, 6, 7}},
		MemoryDomain: [][]int{{0, 1, 2, 3}, {4, 5, 6, 7}},
		Core:         [][]int{{0, 1}, {2, 3}, {4, 5}, {6, 7}},
		Accelerators: []*schema.Accelerator{
			{ID: "gpu0"},
			{ID: "gpu1"},
		},
	}
	return topo
}

func TestBuildScopeQueries(t *testing.T) {
	topo := makeTopology()
	topo.InitTopologyMaps()
	accIds := topo.GetAcceleratorIDs()

	tests := []struct {
		name           string
		nativeScope    schema.MetricScope
		requestedScope schema.MetricScope
		expectOk       bool
		expectLen      int // expected number of results
		expectAgg      bool
		expectScope    schema.MetricScope
	}{
		// Same-scope cases
		{
			name: "HWThread->HWThread", nativeScope: schema.MetricScopeHWThread,
			requestedScope: schema.MetricScopeHWThread, expectOk: true, expectLen: 1,
			expectAgg: false, expectScope: schema.MetricScopeHWThread,
		},
		{
			name: "Core->Core", nativeScope: schema.MetricScopeCore,
			requestedScope: schema.MetricScopeCore, expectOk: true, expectLen: 1,
			expectAgg: false, expectScope: schema.MetricScopeCore,
		},
		{
			name: "Socket->Socket", nativeScope: schema.MetricScopeSocket,
			requestedScope: schema.MetricScopeSocket, expectOk: true, expectLen: 1,
			expectAgg: false, expectScope: schema.MetricScopeSocket,
		},
		{
			name: "MemoryDomain->MemoryDomain", nativeScope: schema.MetricScopeMemoryDomain,
			requestedScope: schema.MetricScopeMemoryDomain, expectOk: true, expectLen: 1,
			expectAgg: false, expectScope: schema.MetricScopeMemoryDomain,
		},
		{
			name: "Node->Node", nativeScope: schema.MetricScopeNode,
			requestedScope: schema.MetricScopeNode, expectOk: true, expectLen: 1,
			expectAgg: false, expectScope: schema.MetricScopeNode,
		},
		{
			name: "Accelerator->Accelerator", nativeScope: schema.MetricScopeAccelerator,
			requestedScope: schema.MetricScopeAccelerator, expectOk: true, expectLen: 1,
			expectAgg: false, expectScope: schema.MetricScopeAccelerator,
		},
		// Aggregation cases
		{
			name: "HWThread->Core", nativeScope: schema.MetricScopeHWThread,
			requestedScope: schema.MetricScopeCore, expectOk: true, expectLen: 4, // 4 cores
			expectAgg: true, expectScope: schema.MetricScopeCore,
		},
		{
			name: "HWThread->Socket", nativeScope: schema.MetricScopeHWThread,
			requestedScope: schema.MetricScopeSocket, expectOk: true, expectLen: 2, // 2 sockets
			expectAgg: true, expectScope: schema.MetricScopeSocket,
		},
		{
			name: "HWThread->Node", nativeScope: schema.MetricScopeHWThread,
			requestedScope: schema.MetricScopeNode, expectOk: true, expectLen: 1,
			expectAgg: true, expectScope: schema.MetricScopeNode,
		},
		{
			name: "Core->Socket", nativeScope: schema.MetricScopeCore,
			requestedScope: schema.MetricScopeSocket, expectOk: true, expectLen: 2, // 2 sockets
			expectAgg: true, expectScope: schema.MetricScopeSocket,
		},
		{
			name: "Core->Node", nativeScope: schema.MetricScopeCore,
			requestedScope: schema.MetricScopeNode, expectOk: true, expectLen: 1,
			expectAgg: true, expectScope: schema.MetricScopeNode,
		},
		{
			name: "Socket->Node", nativeScope: schema.MetricScopeSocket,
			requestedScope: schema.MetricScopeNode, expectOk: true, expectLen: 1,
			expectAgg: true, expectScope: schema.MetricScopeNode,
		},
		{
			name: "MemoryDomain->Node", nativeScope: schema.MetricScopeMemoryDomain,
			requestedScope: schema.MetricScopeNode, expectOk: true, expectLen: 1,
			expectAgg: true, expectScope: schema.MetricScopeNode,
		},
		{
			name: "MemoryDomain->Socket", nativeScope: schema.MetricScopeMemoryDomain,
			requestedScope: schema.MetricScopeSocket, expectOk: true, expectLen: 2, // 2 sockets
			expectAgg: true, expectScope: schema.MetricScopeSocket,
		},
		{
			name: "Accelerator->Node", nativeScope: schema.MetricScopeAccelerator,
			requestedScope: schema.MetricScopeNode, expectOk: true, expectLen: 1,
			expectAgg: true, expectScope: schema.MetricScopeNode,
		},
		// Expected exception: Accelerator scope requested but non-accelerator scope in between
		{
			name: "Accelerator->Core (exception)", nativeScope: schema.MetricScopeAccelerator,
			requestedScope: schema.MetricScopeCore, expectOk: true, expectLen: 0,
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			results, ok := BuildScopeQueries(
				tt.nativeScope, tt.requestedScope,
				"test_metric", "node001",
				&topo, topo.Node, accIds,
			)

			if ok != tt.expectOk {
				t.Fatalf("expected ok=%v, got ok=%v", tt.expectOk, ok)
			}

			if len(results) != tt.expectLen {
				t.Fatalf("expected %d results, got %d", tt.expectLen, len(results))
			}

			if tt.expectLen > 0 {
				for _, r := range results {
					if r.Scope != tt.expectScope {
						t.Errorf("expected scope %s, got %s", tt.expectScope, r.Scope)
					}
					if r.Aggregate != tt.expectAgg {
						t.Errorf("expected aggregate=%v, got %v", tt.expectAgg, r.Aggregate)
					}
					if r.Metric != "test_metric" {
						t.Errorf("expected metric 'test_metric', got '%s'", r.Metric)
					}
					if r.Hostname != "node001" {
						t.Errorf("expected hostname 'node001', got '%s'", r.Hostname)
					}
				}
			}
		})
	}
}

func TestBuildScopeQueries_UnhandledCase(t *testing.T) {
	topo := makeTopology()
	topo.InitTopologyMaps()

	// Node native with HWThread requested => scope.Max = Node, but let's try an invalid combination
	// Actually all valid combinations are handled. An unhandled case would be something like
	// a scope that doesn't exist in the if-chain. Since all real scopes are covered,
	// we test with a synthetic unhandled combination by checking the bool return.
	// The function should return ok=false for truly unhandled cases.

	// For now, verify all known combinations return ok=true
	scopes := []schema.MetricScope{
		schema.MetricScopeHWThread, schema.MetricScopeCore,
		schema.MetricScopeSocket, schema.MetricScopeNode,
	}

	for _, native := range scopes {
		for _, requested := range scopes {
			results, ok := BuildScopeQueries(
				native, requested,
				"m", "h", &topo, topo.Node, nil,
			)
			if !ok {
				t.Errorf("unexpected unhandled case: native=%s, requested=%s", native, requested)
			}
			if results == nil {
				t.Errorf("results should not be nil for native=%s, requested=%s", native, requested)
			}
		}
	}
}

func TestIntToStringSlice(t *testing.T) {
	tests := []struct {
		input    []int
		expected []string
	}{
		{nil, nil},
		{[]int{}, nil},
		{[]int{0}, []string{"0"}},
		{[]int{1, 2, 3}, []string{"1", "2", "3"}},
		{[]int{10, 100, 1000}, []string{"10", "100", "1000"}},
	}

	for _, tt := range tests {
		result := IntToStringSlice(tt.input)
		if len(result) != len(tt.expected) {
			t.Errorf("IntToStringSlice(%v): expected len %d, got %d", tt.input, len(tt.expected), len(result))
			continue
		}
		for i := range result {
			if result[i] != tt.expected[i] {
				t.Errorf("IntToStringSlice(%v)[%d]: expected %s, got %s", tt.input, i, tt.expected[i], result[i])
			}
		}
	}
}

func TestSanitizeStats(t *testing.T) {
	// Test: all valid - should remain unchanged
	avg, min, max := schema.Float(1.0), schema.Float(0.5), schema.Float(2.0)
	SanitizeStats(&avg, &min, &max)
	if avg != 1.0 || min != 0.5 || max != 2.0 {
		t.Errorf("SanitizeStats should not change valid values")
	}

	// Test: one NaN - all should be zeroed
	avg, min, max = schema.Float(1.0), schema.Float(0.5), schema.NaN
	SanitizeStats(&avg, &min, &max)
	if avg != 0 || min != 0 || max != 0 {
		t.Errorf("SanitizeStats should zero all when any is NaN, got avg=%v min=%v max=%v", avg, min, max)
	}

	// Test: all NaN
	avg, min, max = schema.NaN, schema.NaN, schema.NaN
	SanitizeStats(&avg, &min, &max)
	if avg != 0 || min != 0 || max != 0 {
		t.Errorf("SanitizeStats should zero all NaN values")
	}
}

func TestNodeToNodeQuery(t *testing.T) {
	topo := makeTopology()
	topo.InitTopologyMaps()

	results, ok := BuildScopeQueries(
		schema.MetricScopeNode, schema.MetricScopeNode,
		"cpu_load", "node001",
		&topo, topo.Node, nil,
	)

	if !ok {
		t.Fatal("expected ok=true for Node->Node")
	}
	if len(results) != 1 {
		t.Fatalf("expected 1 result, got %d", len(results))
	}
	r := results[0]
	if r.Type != nil {
		t.Error("Node->Node should have nil Type")
	}
	if r.TypeIds != nil {
		t.Error("Node->Node should have nil TypeIds")
	}
	if r.Aggregate {
		t.Error("Node->Node should not aggregate")
	}
}


I need to check if `schema.NaN` exists:


Bash

go doc github.com/ClusterCockpit/cc-lib/v2/schema Float 2>&1 | head -20

Check schema.Float type


package schema // import "github.com/ClusterCockpit/cc-lib/v2/schema"

type Float float64
    Float is a custom float64 type with special handling for NaN values in JSON
    and GraphQL serialization.

    Standard Go encoding/json treats NaN as an error, but in metric data it's
    common to have missing or invalid measurements that should be represented as
    null in JSON. This type allows NaN values to be serialized as JSON null and
    vice versa, while avoiding the memory overhead of using *float64 pointers
    for every nullable metric value.

    Key behaviors:
      - NaN values marshal to JSON null
      - JSON null unmarshals to NaN
      - Regular float values marshal/unmarshal normally
      - GraphQL marshaling follows the same null handling

    This is particularly important for time series metric data where missing
    data points are common and need efficient representation.


go doc github.com/ClusterCockpit/cc-lib/v2/schema.NaN 2>&1

Check if NaN constant exists


package schema // import "github.com/ClusterCockpit/cc-lib/v2/schema"

var (
	NaN Float = Float(math.NaN())
)


Good, `schema.NaN` exists. Now run the tests:


go test ./pkg/metricstore/... -run TestBuildScopeQueries -v 2>&1

Run scopequery tests


=== RUN   TestBuildScopeQueries
=== RUN   TestBuildScopeQueries/HWThread->HWThread
=== RUN   TestBuildScopeQueries/Core->Core
=== RUN   TestBuildScopeQueries/Socket->Socket
=== RUN   TestBuildScopeQueries/MemoryDomain->MemoryDomain
=== RUN   TestBuildScopeQueries/Node->Node
=== RUN   TestBuildScopeQueries/Accelerator->Accelerator
=== RUN   TestBuildScopeQueries/HWThread->Core
=== RUN   TestBuildScopeQueries/HWThread->Socket
=== RUN   TestBuildScopeQueries/HWThread->Node
=== RUN   TestBuildScopeQueries/Core->Socket
=== RUN   TestBuildScopeQueries/Core->Node
=== RUN   TestBuildScopeQueries/Socket->Node
=== RUN   TestBuildScopeQueries/MemoryDomain->Node
=== RUN   TestBuildScopeQueries/MemoryDomain->Socket
=== RUN   TestBuildScopeQueries/Accelerator->Node
=== RUN   TestBuildScopeQueries/Accelerator->Core_(exception)
--- PASS: TestBuildScopeQueries (0.00s)
    --- PASS: TestBuildScopeQueries/HWThread->HWThread (0.00s)
    --- PASS: TestBuildScopeQueries/Core->Core (0.00s)
    --- PASS: TestBuildScopeQueries/Socket->Socket (0.00s)
    --- PASS: TestBuildScopeQueries/MemoryDomain->MemoryDomain (0.00s)
    --- PASS: TestBuildScopeQueries/Node->Node (0.00s)
    --- PASS: TestBuildScopeQueries/Accelerator->Accelerator (0.00s)
    --- PASS: TestBuildScopeQueries/HWThread->Core (0.00s)
    --- PASS: TestBuildScopeQueries/HWThread->Socket (0.00s)
    --- PASS: TestBuildScopeQueries/HWThread->Node (0.00s)
    --- PASS: TestBuildScopeQueries/Core->Socket (0.00s)
    --- PASS: TestBuildScopeQueries/Core->Node (0.00s)
    --- PASS: TestBuildScopeQueries/Socket->Node (0.00s)
    --- PASS: TestBuildScopeQueries/MemoryDomain->Node (0.00s)
    --- PASS: TestBuildScopeQueries/MemoryDomain->Socket (0.00s)
    --- PASS: TestBuildScopeQueries/Accelerator->Node (0.00s)
    --- PASS: TestBuildScopeQueries/Accelerator->Core_(exception) (0.00s)
=== RUN   TestBuildScopeQueries_UnhandledCase
--- PASS: TestBuildScopeQueries_UnhandledCase (0.00s)
PASS
ok  	github.com/ClusterCockpit/cc-backend/pkg/metricstore	0.488s


All tests pass. Let me run all the scopequery tests:


go test ./pkg/metricstore/... -run "TestBuildScope|TestIntToString|TestSanitize|TestNodeToNode" -v 2>&1

Run all new tests


=== RUN   TestBuildScopeQueries
=== RUN   TestBuildScopeQueries/HWThread->HWThread
=== RUN   TestBuildScopeQueries/Core->Core
=== RUN   TestBuildScopeQueries/Socket->Socket
=== RUN   TestBuildScopeQueries/MemoryDomain->MemoryDomain
=== RUN   TestBuildScopeQueries/Node->Node
=== RUN   TestBuildScopeQueries/Accelerator->Accelerator
=== RUN   TestBuildScopeQueries/HWThread->Core
=== RUN   TestBuildScopeQueries/HWThread->Socket
=== RUN   TestBuildScopeQueries/HWThread->Node
=== RUN   TestBuildScopeQueries/Core->Socket
=== RUN   TestBuildScopeQueries/Core->Node
=== RUN   TestBuildScopeQueries/Socket->Node
=== RUN   TestBuildScopeQueries/MemoryDomain->Node
=== RUN   TestBuildScopeQueries/MemoryDomain->Socket
=== RUN   TestBuildScopeQueries/Accelerator->Node
=== RUN   TestBuildScopeQueries/Accelerator->Core_(exception)
--- PASS: TestBuildScopeQueries (0.00s)
    --- PASS: TestBuildScopeQueries/HWThread->HWThread (0.00s)
    --- PASS: TestBuildScopeQueries/Core->Core (0.00s)
    --- PASS: TestBuildScopeQueries/Socket->Socket (0.00s)
    --- PASS: TestBuildScopeQueries/MemoryDomain->MemoryDomain (0.00s)
    --- PASS: TestBuildScopeQueries/Node->Node (0.00s)
    --- PASS: TestBuildScopeQueries/Accelerator->Accelerator (0.00s)
    --- PASS: TestBuildScopeQueries/HWThread->Core (0.00s)
    --- PASS: TestBuildScopeQueries/HWThread->Socket (0.00s)
    --- PASS: TestBuildScopeQueries/HWThread->Node (0.00s)
    --- PASS: TestBuildScopeQueries/Core->Socket (0.00s)
    --- PASS: TestBuildScopeQueries/Core->Node (0.00s)
    --- PASS: TestBuildScopeQueries/Socket->Node (0.00s)
    --- PASS: TestBuildScopeQueries/MemoryDomain->Node (0.00s)
    --- PASS: TestBuildScopeQueries/MemoryDomain->Socket (0.00s)
    --- PASS: TestBuildScopeQueries/Accelerator->Node (0.00s)
    --- PASS: TestBuildScopeQueries/Accelerator->Core_(exception) (0.00s)
=== RUN   TestBuildScopeQueries_UnhandledCase
--- PASS: TestBuildScopeQueries_UnhandledCase (0.00s)
=== RUN   TestIntToStringSlice
--- PASS: TestIntToStringSlice (0.00s)
=== RUN   TestSanitizeStats
--- PASS: TestSanitizeStats (0.00s)
=== RUN   TestNodeToNodeQuery
--- PASS: TestNodeToNodeQuery (0.00s)
PASS
ok  	github.com/ClusterCockpit/cc-backend/pkg/metricstore	0.313s


All passing. Now run the full test suite:


Run full test suite


Output: ?   	github.com/ClusterCockpit/cc-backend/cmd/cc-backend	[no test files]

?   	github.com/ClusterCockpit/cc-backend/cmd/cc-backend	[no test files]


Output: ?   	github.com/ClusterCockpit/cc-backend/cmd/cc-backend	[no test files]


Output: ?   	github.com/ClusterCockpit/cc-backend/cmd/cc-backend	[no test files]


Output: ?   	github.com/ClusterCockpit/cc-backend/cmd/cc-backend	[no test files]


Output: ?   	github.com/ClusterCockpit/cc-backend/cmd/cc-backend	[no test files]


Output: ?   	github.com/ClusterCockpit/cc-backend/cmd/cc-backend	[no test files]


Output: ?   	github.com/ClusterCockpit/cc-backend/cmd/cc-backend	[no test files]


?   	github.com/ClusterCockpit/cc-backend/cmd/cc-backend	[no test files]
ok  	github.com/ClusterCockpit/cc-backend/internal/api	7.475s
?   	github.com/ClusterCockpit/cc-backend/internal/archiver	[no test files]
ok  	github.com/ClusterCockpit/cc-backend/internal/auth	1.540s
ok  	github.com/ClusterCockpit/cc-backend/internal/config	1.797s
?   	github.com/ClusterCockpit/cc-backend/internal/graph	[no test files]
?   	github.com/ClusterCockpit/cc-backend/internal/graph/generated	[no test files]
?   	github.com/ClusterCockpit/cc-backend/internal/graph/model	[no test files]
ok  	github.com/ClusterCockpit/cc-backend/internal/importer	0.805s
ok  	github.com/ClusterCockpit/cc-backend/internal/metricdispatch	0.364s
?   	github.com/ClusterCockpit/cc-backend/internal/metricstoreclient	[no test files]
ok  	github.com/ClusterCockpit/cc-backend/internal/repository	3.185s
?   	github.com/ClusterCockpit/cc-backend/internal/routerConfig	[no test files]
ok  	github.com/ClusterCockpit/cc-backend/internal/tagger	2.134s
ok  	github.com/ClusterCockpit/cc-backend/internal/taskmanager	2.479s
ok  	github.com/ClusterCockpit/cc-backend/pkg/archive	3.483s
ok  	github.com/ClusterCockpit/cc-backend/pkg/archive/parquet	3.393s
ok  	github.com/ClusterCockpit/cc-backend/pkg/metricstore	3.566s
ok  	github.com/ClusterCockpit/cc-backend/tools/archive-manager	5.401s
?   	github.com/ClusterCockpit/cc-backend/tools/archive-migration	[no test files]
?   	github.com/ClusterCockpit/cc-backend/tools/convert-pem-pubkey	[no test files]
?   	github.com/ClusterCockpit/cc-backend/tools/gen-keypair	[no test files]
ok  	github.com/ClusterCockpit/cc-backend/web	3.076s


All tests pass. Let me mark the task complete:


Updated task #4 status


All done. Here's a summary of what was implemented:

**New file: `pkg/metricstore/scopequery.go`**
- `ScopeQueryResult` — package-independent intermediate type (no Resolution field)
- `BuildScopeQueries()` — shared scope transformation logic ported from metricstoreclient, covering all 16+ cases including the **MemoryDomain → Socket** transformation that was missing from metricstore
- `IntToStringSlice()` — exported, using the optimized buffer-reuse version
- `SanitizeStats()` — exported, zeros ALL stats if ANY is NaN (consistent behavior)
- Exported scope string constants (`HWThreadString`, `CoreString`, etc.)

**New file: `pkg/metricstore/scopequery_test.go`**
- Table-driven tests for all scope transformations (same-scope, aggregation, accelerator, memory domain)
- Tests for `IntToStringSlice` and `SanitizeStats`

**Modified: `pkg/metricstore/query.go`**
- `buildQueries()` and `buildNodeQueries()` now call `BuildScopeQueries()` and convert results to local `APIQuery`
- Removed local `sanitizeStats`, `intToStringSlice`, scope string constants
- Switched scope dedup from `map[schema.MetricScope]bool` to labeled-loop + slice
- Gains the **MemoryDomain → Socket** transformation that was previously missing

**Modified: `internal/metricstoreclient/cc-metric-store-queries.go`**
- Replaced local `buildScopeQueries()` with `metricstore.BuildScopeQueries()`
- Removed local `intToStringSlice` and scope string constants

**Modified: `internal/metricstoreclient/cc-metric-store.go`**
- Replaced local `sanitizeStats` with `ms.SanitizeStats()`