Implement the following plan:

# Plan: Improve buildQuery implementation and related code

## Context

The `buildQueries`/`buildNodeQueries` functions and `Load*` methods exist in two packages with near-identical logic:
- `pkg/metricstore/query.go` (internal, in-memory store)
- `internal/metricstoreclient/cc-metric-store.go` + `cc-metric-store-queries.go` (external, HTTP client)

Both share `BuildScopeQueries` and `SanitizeStats` from `pkg/metricstore/scopequery.go`. The review identified bugs, dead code, duplicated patterns, and inconsistencies across both packages.

## Changes

### 1. Bug fix: `LoadNodeData` nil metric config panic (both packages)

`archive.GetMetricConfig()` can return nil, but `LoadNodeData` dereferences `mc.Unit`/`mc.Timestep` without checking. Every other Load function checks.

**Files:**
- `pkg/metricstore/query.go:600` — add nil check + `continue`
- `internal/metricstoreclient/cc-metric-store.go:595` — add nil check + `continue`

### 2. Bug fix: `LoadNodeData` missing `continue` after error (both packages)

When `qdata.Error != nil`, the error is appended but execution falls through to append invalid data. Should `continue`.

**Files:**
- `pkg/metricstore/query.go:589-590`
- `internal/metricstoreclient/cc-metric-store.go:583-585`

### 3. Extract `ExtractTypeID` helper to `scopequery.go`

The ID-extraction block is copy-pasted 6 times (3x in each package). Extract to shared helper in `pkg/metricstore/scopequery.go`.

```go
func ExtractTypeID(queryType *string, typeIds []string, ndx int, metric, hostname string) *string {
    if queryType == nil {
        return nil
    }
    if ndx < len(typeIds) {
        id := typeIds[ndx]
        return &id
    }
    cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
        ndx, len(typeIds), metric, hostname)
    return nil
}
```

**Files to update:**
- `pkg/metricstore/scopequery.go` — add function
- `pkg/metricstore/query.go` — replace 3 occurrences in `LoadData`, `LoadScopedStats`, `LoadNodeListData`
- `internal/metricstoreclient/cc-metric-store.go` — replace 3 occurrences in `LoadData`, `LoadScopedStats`, `LoadNodeListData`

### 4. Extract `IsMetricRemovedForSubCluster` helper to `scopequery.go`

The subcluster-removal check is duplicated 4 times (2x in each package). Extract to shared helper.

```go
func IsMetricRemovedForSubCluster(mc *schema.MetricConfig, subCluster string) bool {
    for _, scConfig := range mc.SubClusters {
        if scConfig.Name == subCluster && scConfig.Remove {
            return true
        }
    }
    return false
}
```

**Files to update:**
- `pkg/metricstore/scopequery.go` — add function
- `pkg/metricstore/query.go` — replace in `buildQueries` and `buildNodeQueries`
- `internal/metricstoreclient/cc-metric-store-queries.go` — replace in `buildQueries` and `buildNodeQueries`

### 5. Fix TODO in production error messages

Replace `"TODO: unhandled case"` with proper error text in all 4 occurrences.

**Files:**
- `pkg/metricstore/query.go:314,901`
- `internal/metricstoreclient/cc-metric-store-queries.go:126,237`

### 6. Remove redundant inner bounds checks in `LoadData` and `LoadNodeListData`

The outer truncation (e.g., lines 120-129 in internal `LoadData`) already ensures bounds safety, making the inner `if i >= len(...)` check dead code. Remove the inner checks.

**Files (4 occurrences — 2 in each package):**
- `pkg/metricstore/query.go` — `LoadData` (lines 131-137), `LoadNodeListData` (lines 696-702)
- `internal/metricstoreclient/cc-metric-store.go` — `LoadData` (lines 278-283), `LoadNodeListData` (lines 682-688)

### 7. Add capacity hint for `assignedScope` in internal `buildQueries`/`buildNodeQueries`

**File:** `pkg/metricstore/query.go:254,822`

Change `assignedScope := []schema.MetricScope{}` to use `make()` with matching capacity.

### 8. Add bounds check to `LoadStats` for consistency (both packages)

`LoadStats` indexes `req.Queries[i]` without checking bounds, unlike all other Load functions. Add a check.

**Files:**
- `pkg/metricstore/query.go:376`
- `internal/metricstoreclient/cc-metric-store.go:395`

## Verification

1. `go build ./...` — ensure compilation succeeds
2. `go test ./pkg/metricstore/...` — run metricstore tests
3. `go test ./internal/metricstoreclient/...` — run client tests
4. `go vet ./pkg/metricstore/... ./internal/metricstoreclient/...` — check for issues

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-ClusterCockpit-cc-backend/f0763fe7-f9bf-47b1-85f7-b16c2b9bffe1.jsonl


Let me start by reading the relevant files to understand the current code before making changes.


I'll start by reading the key files to understand the current code.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// This file implements high-level query functions for loading job metric data
     7→// with automatic scope transformation and aggregation.
     8→//
     9→// Key Concepts:
    10→//
    11→// Metric Scopes: Metrics are collected at different granularities (native scope):
    12→//   - HWThread: Per hardware thread
    13→//   - Core: Per CPU core
    14→//   - Socket: Per CPU socket
    15→//   - MemoryDomain: Per memory domain (NUMA)
    16→//   - Accelerator: Per GPU/accelerator
    17→//   - Node: Per compute node
    18→//
    19→// Scope Transformation: The buildQueries functions transform between native scope
    20→// and requested scope by:
    21→//   - Aggregating finer-grained data (e.g., HWThread → Core → Socket → Node)
    22→//   - Rejecting requests for finer granularity than available
    23→//   - Handling special cases (e.g., Accelerator metrics)
    24→//
    25→// Query Building: Constructs APIQuery structures with proper selectors (Type, TypeIds)
    26→// based on cluster topology and job resources.
    27→package metricstore
    28→
    29→import (
    30→	"context"
    31→	"fmt"
    32→	"strings"
    33→	"time"
    34→
    35→	"github.com/ClusterCockpit/cc-backend/pkg/archive"
    36→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    37→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    38→)
    39→
    40→type InternalMetricStore struct{}
    41→
    42→var MetricStoreHandle *InternalMetricStore
    43→
    44→// HealthCheck delegates to the internal MemoryStore's HealthCheck.
    45→func (ccms *InternalMetricStore) HealthCheck(cluster string,
    46→	nodes []string, metrics []string,
    47→) (map[string]HealthCheckResult, error) {
    48→	return GetMemoryStore().HealthCheck(cluster, nodes, metrics)
    49→}
    50→
    51→// TestLoadDataCallback allows tests to override LoadData behavior for testing purposes.
    52→// When set to a non-nil function, LoadData will call this function instead of the default implementation.
    53→var TestLoadDataCallback func(job *schema.Job, metrics []string, scopes []schema.MetricScope, ctx context.Context, resolution int) (schema.JobData, error)
    54→
    55→// LoadData loads metric data for a specific job with automatic scope transformation.
    56→//
    57→// This is the primary function for retrieving job metric data. It handles:
    58→//   - Building queries with scope transformation via buildQueries
    59→//   - Fetching data from the metric store
    60→//   - Organizing results by metric and scope
    61→//   - Converting NaN statistics to 0 for JSON compatibility
    62→//   - Partial error handling (returns data for successful queries even if some fail)
    63→//
    64→// Parameters:
    65→//   - job: Job metadata including cluster, resources, and time range
    66→//   - metrics: List of metric names to load
    67→//   - scopes: Requested metric scopes (will be transformed to match native scopes)
    68→//   - ctx: Context for cancellation (currently unused but reserved for future use)
    69→//   - resolution: Data resolution in seconds (0 for native resolution)
    70→//
    71→// Returns:
    72→//   - JobData: Map of metric → scope → JobMetric with time-series data and statistics
    73→//   - Error: Returns error if query building or fetching fails, or partial error listing failed hosts
    74→//
    75→// Example:
    76→//
    77→//	jobData, err := LoadData(job, []string{"cpu_load", "mem_used"}, []schema.MetricScope{schema.MetricScopeNode}, ctx, 60)
    78→func (ccms *InternalMetricStore) LoadData(
    79→	job *schema.Job,
    80→	metrics []string,
    81→	scopes []schema.MetricScope,
    82→	ctx context.Context,
    83→	resolution int,
    84→) (schema.JobData, error) {
    85→	if TestLoadDataCallback != nil {
    86→		return TestLoadDataCallback(job, metrics, scopes, ctx, resolution)
    87→	}
    88→
    89→	queries, assignedScope, err := buildQueries(job, metrics, scopes, int64(resolution))
    90→	if err != nil {
    91→		cclog.Errorf("Error while building queries for jobId %d, Metrics %v, Scopes %v: %s", job.JobID, metrics, scopes, err.Error())
    92→		return nil, err
    93→	}
    94→
    95→	// Verify assignment is correct - log any inconsistencies for debugging
    96→	if len(queries) != len(assignedScope) {
    97→		cclog.Errorf("Critical error: queries and assignedScope have different lengths after buildQueries: %d vs %d",
    98→			len(queries), len(assignedScope))
    99→	}
   100→
   101→	req := APIQueryRequest{
   102→		Cluster:   job.Cluster,
   103→		From:      job.StartTime,
   104→		To:        job.StartTime + int64(job.Duration),
   105→		Queries:   queries,
   106→		WithStats: true,
   107→		WithData:  true,
   108→	}
   109→
   110→	resBody, err := FetchData(req)
   111→	if err != nil {
   112→		cclog.Errorf("Error while fetching data : %s", err.Error())
   113→		return nil, err
   114→	}
   115→
   116→	var errors []string
   117→	jobData := make(schema.JobData)
   118→
   119→	// Add safety check for potential index out of range errors
   120→	if len(resBody.Results) != len(req.Queries) || len(assignedScope) != len(req.Queries) {
   121→		cclog.Warnf("Mismatch in query results count: queries=%d, results=%d, assignedScope=%d",
   122→			len(req.Queries), len(resBody.Results), len(assignedScope))
   123→		if len(resBody.Results) > len(req.Queries) {
   124→			resBody.Results = resBody.Results[:len(req.Queries)]
   125→		}
   126→		if len(assignedScope) > len(req.Queries) {
   127→			assignedScope = assignedScope[:len(req.Queries)]
   128→		}
   129→	}
   130→
   131→	for i, row := range resBody.Results {
   132→		// Safety check to prevent index out of range errors
   133→		if i >= len(req.Queries) || i >= len(assignedScope) {
   134→			cclog.Warnf("Index out of range prevented: i=%d, queries=%d, assignedScope=%d",
   135→				i, len(req.Queries), len(assignedScope))
   136→			continue
   137→		}
   138→
   139→		query := req.Queries[i]
   140→		metric := query.Metric
   141→		scope := assignedScope[i]
   142→		mc := archive.GetMetricConfig(job.Cluster, metric)
   143→
   144→		if mc == nil {
   145→			cclog.Warnf("Metric config not found for %s on cluster %s", metric, job.Cluster)
   146→			continue
   147→		}
   148→
   149→		if _, ok := jobData[metric]; !ok {
   150→			jobData[metric] = make(map[schema.MetricScope]*schema.JobMetric)
   151→		}
   152→
   153→		res := mc.Timestep
   154→		if len(row) > 0 {
   155→			res = int(row[0].Resolution)
   156→		}
   157→
   158→		jobMetric, ok := jobData[metric][scope]
   159→		if !ok {
   160→			jobMetric = &schema.JobMetric{
   161→				Unit:     mc.Unit,
   162→				Timestep: res,
   163→				Series:   make([]schema.Series, 0),
   164→			}
   165→			jobData[metric][scope] = jobMetric
   166→		}
   167→
   168→		for ndx, res := range row {
   169→			if res.Error != nil {
   170→				/* Build list for "partial errors", if any */
   171→				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
   172→				continue
   173→			}
   174→
   175→			id := (*string)(nil)
   176→			if query.Type != nil {
   177→				// Check if ndx is within the bounds of TypeIds slice
   178→				if ndx < len(query.TypeIds) {
   179→					id = new(string)
   180→					*id = query.TypeIds[ndx]
   181→				} else {
   182→					// Log the error but continue processing
   183→					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
   184→						ndx, len(query.TypeIds), query.Metric, query.Hostname)
   185→				}
   186→			}
   187→
   188→			SanitizeStats(&res.Avg, &res.Min, &res.Max)
   189→
   190→			jobMetric.Series = append(jobMetric.Series, schema.Series{
   191→				Hostname: query.Hostname,
   192→				ID:       id,
   193→				Statistics: schema.MetricStatistics{
   194→					Avg: float64(res.Avg),
   195→					Min: float64(res.Min),
   196→					Max: float64(res.Max),
   197→				},
   198→				Data: res.Data,
   199→			})
   200→		}
   201→
   202→		// So that one can later check len(jobData):
   203→		if len(jobMetric.Series) == 0 {
   204→			delete(jobData[metric], scope)
   205→			if len(jobData[metric]) == 0 {
   206→				delete(jobData, metric)
   207→			}
   208→		}
   209→	}
   210→
   211→	if len(errors) != 0 {
   212→		/* Returns list for "partial errors" */
   213→		return jobData, fmt.Errorf("METRICDATA/INTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
   214→	}
   215→	return jobData, nil
   216→}
   217→
   218→// buildQueries constructs APIQuery structures with automatic scope transformation for a job.
   219→//
   220→// This function implements the core scope transformation logic, handling all combinations of
   221→// native metric scopes and requested scopes. It uses the cluster topology to determine which
   222→// hardware IDs to include in each query.
   223→//
   224→// Scope Transformation Rules:
   225→//   - If native scope >= requested scope: Aggregates data (Aggregate=true in APIQuery)
   226→//   - If native scope < requested scope: Returns error (cannot increase granularity)
   227→//   - Special handling for Accelerator scope (independent of CPU hierarchy)
   228→//
   229→// The function generates one or more APIQuery per (metric, scope, host) combination:
   230→//   - For non-aggregated queries: One query with all relevant IDs
   231→//   - For aggregated queries: May generate multiple queries (e.g., one per socket/core)
   232→//
   233→// Parameters:
   234→//   - job: Job metadata including cluster, subcluster, and resource allocation
   235→//   - metrics: List of metrics to query
   236→//   - scopes: Requested scopes for each metric
   237→//   - resolution: Data resolution in seconds
   238→//
   239→// Returns:
   240→//   - []APIQuery: List of queries to execute
   241→//   - []schema.MetricScope: Assigned scope for each query (after transformation)
   242→//   - error: Returns error if topology lookup fails or unhandled scope combination encountered
   243→func buildQueries(
   244→	job *schema.Job,
   245→	metrics []string,
   246→	scopes []schema.MetricScope,
   247→	resolution int64,
   248→) ([]APIQuery, []schema.MetricScope, error) {
   249→	if len(job.Resources) == 0 {
   250→		return nil, nil, fmt.Errorf("METRICDATA/INTERNAL-CCMS > no resources allocated for job %d", job.JobID)
   251→	}
   252→
   253→	queries := make([]APIQuery, 0, len(metrics)*len(scopes)*len(job.Resources))
   254→	assignedScope := []schema.MetricScope{}
   255→
   256→	subcluster, scerr := archive.GetSubCluster(job.Cluster, job.SubCluster)
   257→	if scerr != nil {
   258→		return nil, nil, scerr
   259→	}
   260→	topology := subcluster.Topology
   261→
   262→	for _, metric := range metrics {
   263→		mc := archive.GetMetricConfig(job.Cluster, metric)
   264→		if mc == nil {
   265→			cclog.Warnf("metric '%s' is not specified for cluster '%s'", metric, job.Cluster)
   266→			continue
   267→		}
   268→
   269→		// Skip if metric is removed for subcluster
   270→		if len(mc.SubClusters) != 0 {
   271→			isRemoved := false
   272→			for _, scConfig := range mc.SubClusters {
   273→				if scConfig.Name == job.SubCluster && scConfig.Remove {
   274→					isRemoved = true
   275→					break
   276→				}
   277→			}
   278→			if isRemoved {
   279→				continue
   280→			}
   281→		}
   282→
   283→		// Avoid duplicates...
   284→		handledScopes := make([]schema.MetricScope, 0, 3)
   285→
   286→	scopesLoop:
   287→		for _, requestedScope := range scopes {
   288→			nativeScope := mc.Scope
   289→			if nativeScope == schema.MetricScopeAccelerator && job.NumAcc == 0 {
   290→				continue
   291→			}
   292→
   293→			scope := nativeScope.Max(requestedScope)
   294→			for _, s := range handledScopes {
   295→				if scope == s {
   296→					continue scopesLoop
   297→				}
   298→			}
   299→			handledScopes = append(handledScopes, scope)
   300→
   301→			for _, host := range job.Resources {
   302→				hwthreads := host.HWThreads
   303→				if hwthreads == nil {
   304→					hwthreads = topology.Node
   305→				}
   306→
   307→				scopeResults, ok := BuildScopeQueries(
   308→					nativeScope, requestedScope,
   309→					metric, host.Hostname,
   310→					&topology, hwthreads, host.Accelerators,
   311→				)
   312→
   313→				if !ok {
   314→					return nil, nil, fmt.Errorf("METRICDATA/INTERNAL-CCMS > TODO: unhandled case: native-scope=%s, requested-scope=%s", nativeScope, requestedScope)
   315→				}
   316→
   317→				for _, sr := range scopeResults {
   318→					queries = append(queries, APIQuery{
   319→						Metric:     sr.Metric,
   320→						Hostname:   sr.Hostname,
   321→						Aggregate:  sr.Aggregate,
   322→						Type:       sr.Type,
   323→						TypeIds:    sr.TypeIds,
   324→						Resolution: resolution,
   325→					})
   326→					assignedScope = append(assignedScope, sr.Scope)
   327→				}
   328→			}
   329→		}
   330→	}
   331→
   332→	return queries, assignedScope, nil
   333→}
   334→
   335→// LoadStats loads only metric statistics (avg/min/max) for a job at node scope.
   336→//
   337→// This is an optimized version of LoadData that fetches only statistics without
   338→// time-series data, reducing bandwidth and memory usage. Always queries at node scope.
   339→//
   340→// Parameters:
   341→//   - job: Job metadata
   342→//   - metrics: List of metric names
   343→//   - ctx: Context (currently unused)
   344→//
   345→// Returns:
   346→//   - Map of metric → hostname → statistics
   347→//   - Error on query building or fetching failure
   348→func (ccms *InternalMetricStore) LoadStats(
   349→	job *schema.Job,
   350→	metrics []string,
   351→	ctx context.Context,
   352→) (map[string]map[string]schema.MetricStatistics, error) {
   353→	// TODO(#166): Add scope parameter for analysis view accelerator normalization
   354→	queries, _, err := buildQueries(job, metrics, []schema.MetricScope{schema.MetricScopeNode}, 0)
   355→	if err != nil {
   356→		cclog.Errorf("Error while building queries for jobId %d, Metrics %v: %s", job.JobID, metrics, err.Error())
   357→		return nil, err
   358→	}
   359→
   360→	req := APIQueryRequest{
   361→		Cluster:   job.Cluster,
   362→		From:      job.StartTime,
   363→		To:        job.StartTime + int64(job.Duration),
   364→		Queries:   queries,
   365→		WithStats: true,
   366→		WithData:  false,
   367→	}
   368→
   369→	resBody, err := FetchData(req)
   370→	if err != nil {
   371→		cclog.Errorf("Error while fetching data : %s", err.Error())
   372→		return nil, err
   373→	}
   374→
   375→	stats := make(map[string]map[string]schema.MetricStatistics, len(metrics))
   376→	for i, res := range resBody.Results {
   377→		if len(res) == 0 {
   378→			// No Data Found For Metric, Logged in FetchData to Warn
   379→			continue
   380→		}
   381→		query := req.Queries[i]
   382→		metric := query.Metric
   383→		data := res[0]
   384→		if data.Error != nil {
   385→			cclog.Warnf("fetching %s for node %s failed: %s", metric, query.Hostname, *data.Error)
   386→			continue
   387→		}
   388→
   389→		metricdata, ok := stats[metric]
   390→		if !ok {
   391→			metricdata = make(map[string]schema.MetricStatistics, job.NumNodes)
   392→			stats[metric] = metricdata
   393→		}
   394→
   395→		if data.Avg.IsNaN() || data.Min.IsNaN() || data.Max.IsNaN() {
   396→			cclog.Warnf("fetching %s for node %s failed: one of avg/min/max is NaN", metric, query.Hostname)
   397→			continue
   398→		}
   399→
   400→		metricdata[query.Hostname] = schema.MetricStatistics{
   401→			Avg: float64(data.Avg),
   402→			Min: float64(data.Min),
   403→			Max: float64(data.Max),
   404→		}
   405→	}
   406→
   407→	return stats, nil
   408→}
   409→
   410→// LoadScopedStats loads metric statistics for a job with scope-aware grouping.
   411→//
   412→// Similar to LoadStats but supports multiple scopes and returns statistics grouped
   413→// by scope with hardware IDs (e.g., per-core, per-socket statistics).
   414→//
   415→// Parameters:
   416→//   - job: Job metadata
   417→//   - metrics: List of metric names
   418→//   - scopes: Requested metric scopes
   419→//   - ctx: Context (currently unused)
   420→//
   421→// Returns:
   422→//   - ScopedJobStats: Map of metric → scope → []ScopedStats (with hostname and ID)
   423→//   - Error or partial error listing failed queries
   424→func (ccms *InternalMetricStore) LoadScopedStats(
   425→	job *schema.Job,
   426→	metrics []string,
   427→	scopes []schema.MetricScope,
   428→	ctx context.Context,
   429→) (schema.ScopedJobStats, error) {
   430→	queries, assignedScope, err := buildQueries(job, metrics, scopes, 0)
   431→	if err != nil {
   432→		cclog.Errorf("Error while building queries for jobId %d, Metrics %v, Scopes %v: %s", job.JobID, metrics, scopes, err.Error())
   433→		return nil, err
   434→	}
   435→
   436→	req := APIQueryRequest{
   437→		Cluster:   job.Cluster,
   438→		From:      job.StartTime,
   439→		To:        job.StartTime + int64(job.Duration),
   440→		Queries:   queries,
   441→		WithStats: true,
   442→		WithData:  false,
   443→	}
   444→
   445→	resBody, err := FetchData(req)
   446→	if err != nil {
   447→		cclog.Errorf("Error while fetching data : %s", err.Error())
   448→		return nil, err
   449→	}
   450→
   451→	var errors []string
   452→	scopedJobStats := make(schema.ScopedJobStats)
   453→
   454→	for i, row := range resBody.Results {
   455→		if len(row) == 0 {
   456→			// No Data Found For Metric, Logged in FetchData to Warn
   457→			continue
   458→		}
   459→		query := req.Queries[i]
   460→		metric := query.Metric
   461→		scope := assignedScope[i]
   462→
   463→		if _, ok := scopedJobStats[metric]; !ok {
   464→			scopedJobStats[metric] = make(map[schema.MetricScope][]*schema.ScopedStats)
   465→		}
   466→
   467→		if _, ok := scopedJobStats[metric][scope]; !ok {
   468→			scopedJobStats[metric][scope] = make([]*schema.ScopedStats, 0)
   469→		}
   470→
   471→		for ndx, res := range row {
   472→			if res.Error != nil {
   473→				/* Build list for "partial errors", if any */
   474→				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
   475→				continue
   476→			}
   477→
   478→			id := (*string)(nil)
   479→			if query.Type != nil {
   480→				// Check if ndx is within the bounds of TypeIds slice
   481→				if ndx < len(query.TypeIds) {
   482→					id = new(string)
   483→					*id = query.TypeIds[ndx]
   484→				} else {
   485→					// Log the error but continue processing
   486→					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
   487→						ndx, len(query.TypeIds), query.Metric, query.Hostname)
   488→				}
   489→			}
   490→
   491→			SanitizeStats(&res.Avg, &res.Min, &res.Max)
   492→
   493→			scopedJobStats[metric][scope] = append(scopedJobStats[metric][scope], &schema.ScopedStats{
   494→				Hostname: query.Hostname,
   495→				ID:       id,
   496→				Data: &schema.MetricStatistics{
   497→					Avg: float64(res.Avg),
   498→					Min: float64(res.Min),
   499→					Max: float64(res.Max),
   500→				},
   501→			})
   502→		}
   503→
   504→		// So that one can later check len(scopedJobStats[metric][scope]): Remove from map if empty
   505→		if len(scopedJobStats[metric][scope]) == 0 {
   506→			delete(scopedJobStats[metric], scope)
   507→			if len(scopedJobStats[metric]) == 0 {
   508→				delete(scopedJobStats, metric)
   509→			}
   510→		}
   511→	}
   512→
   513→	if len(errors) != 0 {
   514→		/* Returns list for "partial errors" */
   515→		return scopedJobStats, fmt.Errorf("METRICDATA/INTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
   516→	}
   517→	return scopedJobStats, nil
   518→}
   519→
   520→// LoadNodeData loads metric data for specific nodes in a cluster over a time range.
   521→//
   522→// Unlike LoadData which operates on job resources, this function queries arbitrary nodes
   523→// directly. Useful for system monitoring and node status views.
   524→//
   525→// Parameters:
   526→//   - cluster: Cluster name
   527→//   - metrics: List of metric names
   528→//   - nodes: List of node hostnames (nil = all nodes in cluster via ForAllNodes)
   529→//   - scopes: Requested metric scopes (currently unused - always node scope)
   530→//   - from, to: Time range
   531→//   - ctx: Context (currently unused)
   532→//
   533→// Returns:
   534→//   - Map of hostname → metric → []JobMetric
   535→//   - Error or partial error listing failed queries
   536→func (ccms *InternalMetricStore) LoadNodeData(
   537→	cluster string,
   538→	metrics, nodes []string,
   539→	scopes []schema.MetricScope,
   540→	from, to time.Time,
   541→	ctx context.Context,
   542→) (map[string]map[string][]*schema.JobMetric, error) {
   543→	req := APIQueryRequest{
   544→		Cluster:   cluster,
   545→		From:      from.Unix(),
   546→		To:        to.Unix(),
   547→		WithStats: true,
   548→		WithData:  true,
   549→	}
   550→
   551→	if nodes == nil {
   552→		req.ForAllNodes = append(req.ForAllNodes, metrics...)
   553→	} else {
   554→		for _, node := range nodes {
   555→			for _, metric := range metrics {
   556→				req.Queries = append(req.Queries, APIQuery{
   557→					Hostname:   node,
   558→					Metric:     metric,
   559→					Resolution: 0, // Default for Node Queries: Will return metric $Timestep Resolution
   560→				})
   561→			}
   562→		}
   563→	}
   564→
   565→	resBody, err := FetchData(req)
   566→	if err != nil {
   567→		cclog.Errorf("Error while fetching data : %s", err.Error())
   568→		return nil, err
   569→	}
   570→
   571→	var errors []string
   572→	data := make(map[string]map[string][]*schema.JobMetric)
   573→	for i, res := range resBody.Results {
   574→		if len(res) == 0 {
   575→			// No Data Found For Metric, Logged in FetchData to Warn
   576→			continue
   577→		}
   578→
   579→		var query APIQuery
   580→		if resBody.Queries != nil {
   581→			query = resBody.Queries[i]
   582→		} else {
   583→			query = req.Queries[i]
   584→		}
   585→
   586→		metric := query.Metric
   587→		qdata := res[0]
   588→		if qdata.Error != nil {
   589→			errors = append(errors, fmt.Sprintf("fetching %s for node %s failed: %s", metric, query.Hostname, *qdata.Error))
   590→		}
   591→
   592→		SanitizeStats(&qdata.Avg, &qdata.Min, &qdata.Max)
   593→
   594→		hostdata, ok := data[query.Hostname]
   595→		if !ok {
   596→			hostdata = make(map[string][]*schema.JobMetric)
   597→			data[query.Hostname] = hostdata
   598→		}
   599→
   600→		mc := archive.GetMetricConfig(cluster, metric)
   601→		hostdata[metric] = append(hostdata[metric], &schema.JobMetric{
   602→			Unit:     mc.Unit,
   603→			Timestep: mc.Timestep,
   604→			Series: []schema.Series{
   605→				{
   606→					Hostname: query.Hostname,
   607→					Data:     qdata.Data,
   608→					Statistics: schema.MetricStatistics{
   609→						Avg: float64(qdata.Avg),
   610→						Min: float64(qdata.Min),
   611→						Max: float64(qdata.Max),
   612→					},
   613→				},
   614→			},
   615→		})
   616→	}
   617→
   618→	if len(errors) != 0 {
   619→		/* Returns list of "partial errors" */
   620→		return data, fmt.Errorf("METRICDATA/INTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
   621→	}
   622→
   623→	return data, nil
   624→}
   625→
   626→// LoadNodeListData loads metric data for a list of nodes with full scope transformation support.
   627→//
   628→// This is the most flexible node data loading function, supporting arbitrary scopes and
   629→// resolution. Uses buildNodeQueries for proper scope transformation based on topology.
   630→//
   631→// Parameters:
   632→//   - cluster: Cluster name
   633→//   - subCluster: SubCluster name (empty string to infer from node names)
   634→//   - nodes: List of node hostnames
   635→//   - metrics: List of metric names
   636→//   - scopes: Requested metric scopes
   637→//   - resolution: Data resolution in seconds
   638→//   - from, to: Time range
   639→//   - ctx: Context (currently unused)
   640→//
   641→// Returns:
   642→//   - Map of hostname → JobData (metric → scope → JobMetric)
   643→//   - Error or partial error listing failed queries
   644→func (ccms *InternalMetricStore) LoadNodeListData(
   645→	cluster, subCluster string,
   646→	nodes []string,
   647→	metrics []string,
   648→	scopes []schema.MetricScope,
   649→	resolution int,
   650→	from, to time.Time,
   651→	ctx context.Context,
   652→) (map[string]schema.JobData, error) {
   653→	// Note: Order of node data is not guaranteed after this point
   654→	queries, assignedScope, err := buildNodeQueries(cluster, subCluster, nodes, metrics, scopes, int64(resolution))
   655→	if err != nil {
   656→		cclog.Errorf("Error while building node queries for Cluster %s, SubCLuster %s, Metrics %v, Scopes %v: %s", cluster, subCluster, metrics, scopes, err.Error())
   657→		return nil, err
   658→	}
   659→
   660→	// Verify assignment is correct - log any inconsistencies for debugging
   661→	if len(queries) != len(assignedScope) {
   662→		cclog.Errorf("Critical error: queries and assignedScope have different lengths after buildNodeQueries: %d vs %d",
   663→			len(queries), len(assignedScope))
   664→	}
   665→
   666→	req := APIQueryRequest{
   667→		Cluster:   cluster,
   668→		Queries:   queries,
   669→		From:      from.Unix(),
   670→		To:        to.Unix(),
   671→		WithStats: true,
   672→		WithData:  true,
   673→	}
   674→
   675→	resBody, err := FetchData(req)
   676→	if err != nil {
   677→		cclog.Errorf("Error while fetching data : %s", err.Error())
   678→		return nil, err
   679→	}
   680→
   681→	var errors []string
   682→	data := make(map[string]schema.JobData)
   683→
   684→	// Add safety check for index out of range issues
   685→	if len(resBody.Results) != len(req.Queries) || len(assignedScope) != len(req.Queries) {
   686→		cclog.Warnf("Mismatch in query results count: queries=%d, results=%d, assignedScope=%d",
   687→			len(req.Queries), len(resBody.Results), len(assignedScope))
   688→		if len(resBody.Results) > len(req.Queries) {
   689→			resBody.Results = resBody.Results[:len(req.Queries)]
   690→		}
   691→		if len(assignedScope) > len(req.Queries) {
   692→			assignedScope = assignedScope[:len(req.Queries)]
   693→		}
   694→	}
   695→
   696→	for i, row := range resBody.Results {
   697→		// Safety check to prevent index out of range errors
   698→		if i >= len(req.Queries) || i >= len(assignedScope) {
   699→			cclog.Warnf("Index out of range prevented: i=%d, queries=%d, assignedScope=%d",
   700→				i, len(req.Queries), len(assignedScope))
   701→			continue
   702→		}
   703→
   704→		var query APIQuery
   705→		if resBody.Queries != nil {
   706→			if i < len(resBody.Queries) {
   707→				query = resBody.Queries[i]
   708→			} else {
   709→				cclog.Warnf("Index out of range prevented for resBody.Queries: i=%d, len=%d",
   710→					i, len(resBody.Queries))
   711→				continue
   712→			}
   713→		} else {
   714→			query = req.Queries[i]
   715→		}
   716→
   717→		metric := query.Metric
   718→		scope := assignedScope[i]
   719→		mc := archive.GetMetricConfig(cluster, metric)
   720→		if mc == nil {
   721→			cclog.Warnf("Metric config not found for %s on cluster %s", metric, cluster)
   722→			continue
   723→		}
   724→
   725→		res := mc.Timestep
   726→		if len(row) > 0 {
   727→			res = int(row[0].Resolution)
   728→		}
   729→
   730→		// Init Nested Map Data Structures If Not Found
   731→		hostData, ok := data[query.Hostname]
   732→		if !ok {
   733→			hostData = make(schema.JobData)
   734→			data[query.Hostname] = hostData
   735→		}
   736→
   737→		metricData, ok := hostData[metric]
   738→		if !ok {
   739→			metricData = make(map[schema.MetricScope]*schema.JobMetric)
   740→			data[query.Hostname][metric] = metricData
   741→		}
   742→
   743→		scopeData, ok := metricData[scope]
   744→		if !ok {
   745→			scopeData = &schema.JobMetric{
   746→				Unit:     mc.Unit,
   747→				Timestep: res,
   748→				Series:   make([]schema.Series, 0),
   749→			}
   750→			data[query.Hostname][metric][scope] = scopeData
   751→		}
   752→
   753→		for ndx, res := range row {
   754→			if res.Error != nil {
   755→				/* Build list for "partial errors", if any */
   756→				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
   757→				continue
   758→			}
   759→
   760→			id := (*string)(nil)
   761→			if query.Type != nil {
   762→				// Check if ndx is within the bounds of TypeIds slice
   763→				if ndx < len(query.TypeIds) {
   764→					id = new(string)
   765→					*id = query.TypeIds[ndx]
   766→				} else {
   767→					// Log the error but continue processing
   768→					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
   769→						ndx, len(query.TypeIds), query.Metric, query.Hostname)
   770→				}
   771→			}
   772→
   773→			SanitizeStats(&res.Avg, &res.Min, &res.Max)
   774→
   775→			scopeData.Series = append(scopeData.Series, schema.Series{
   776→				Hostname: query.Hostname,
   777→				ID:       id,
   778→				Statistics: schema.MetricStatistics{
   779→					Avg: float64(res.Avg),
   780→					Min: float64(res.Min),
   781→					Max: float64(res.Max),
   782→				},
   783→				Data: res.Data,
   784→			})
   785→		}
   786→	}
   787→
   788→	if len(errors) != 0 {
   789→		/* Returns list of "partial errors" */
   790→		return data, fmt.Errorf("METRICDATA/INTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
   791→	}
   792→
   793→	return data, nil
   794→}
   795→
   796→// buildNodeQueries constructs APIQuery structures for node-based queries with scope transformation.
   797→//
   798→// Similar to buildQueries but operates on node lists rather than job resources.
   799→// Supports dynamic subcluster lookup when subCluster parameter is empty.
   800→//
   801→// Parameters:
   802→//   - cluster: Cluster name
   803→//   - subCluster: SubCluster name (empty = infer from node hostnames)
   804→//   - nodes: List of node hostnames
   805→//   - metrics: List of metric names
   806→//   - scopes: Requested metric scopes
   807→//   - resolution: Data resolution in seconds
   808→//
   809→// Returns:
   810→//   - []APIQuery: List of queries to execute
   811→//   - []schema.MetricScope: Assigned scope for each query
   812→//   - error: Returns error if topology lookup fails or unhandled scope combination
   813→func buildNodeQueries(
   814→	cluster string,
   815→	subCluster string,
   816→	nodes []string,
   817→	metrics []string,
   818→	scopes []schema.MetricScope,
   819→	resolution int64,
   820→) ([]APIQuery, []schema.MetricScope, error) {
   821→	queries := make([]APIQuery, 0, len(metrics)*len(scopes)*len(nodes))
   822→	assignedScope := []schema.MetricScope{}
   823→
   824→	// Get Topol before loop if subCluster given
   825→	var subClusterTopol *schema.SubCluster
   826→	var scterr error
   827→	if subCluster != "" {
   828→		subClusterTopol, scterr = archive.GetSubCluster(cluster, subCluster)
   829→		if scterr != nil {
   830→			cclog.Errorf("could not load cluster %s subCluster %s topology: %s", cluster, subCluster, scterr.Error())
   831→			return nil, nil, scterr
   832→		}
   833→	}
   834→
   835→	for _, metric := range metrics {
   836→		mc := archive.GetMetricConfig(cluster, metric)
   837→		if mc == nil {
   838→			cclog.Warnf("metric '%s' is not specified for cluster '%s'", metric, cluster)
   839→			continue
   840→		}
   841→
   842→		// Skip if metric is removed for subcluster
   843→		if mc.SubClusters != nil {
   844→			isRemoved := false
   845→			for _, scConfig := range mc.SubClusters {
   846→				if scConfig.Name == subCluster && scConfig.Remove {
   847→					isRemoved = true
   848→					break
   849→				}
   850→			}
   851→			if isRemoved {
   852→				continue
   853→			}
   854→		}
   855→
   856→		// Avoid duplicates...
   857→		handledScopes := make([]schema.MetricScope, 0, 3)
   858→
   859→	nodeScopesLoop:
   860→		for _, requestedScope := range scopes {
   861→			nativeScope := mc.Scope
   862→
   863→			scope := nativeScope.Max(requestedScope)
   864→			for _, s := range handledScopes {
   865→				if scope == s {
   866→					continue nodeScopesLoop
   867→				}
   868→			}
   869→			handledScopes = append(handledScopes, scope)
   870→
   871→			for _, hostname := range nodes {
   872→
   873→				// If no subCluster given, get it by node
   874→				if subCluster == "" {
   875→					subClusterName, scnerr := archive.GetSubClusterByNode(cluster, hostname)
   876→					if scnerr != nil {
   877→						return nil, nil, scnerr
   878→					}
   879→					subClusterTopol, scterr = archive.GetSubCluster(cluster, subClusterName)
   880→					if scterr != nil {
   881→						return nil, nil, scterr
   882→					}
   883→				}
   884→
   885→				// Always full node hwthread id list, no partial queries expected
   886→				topology := subClusterTopol.Topology
   887→				acceleratorIds := topology.GetAcceleratorIDs()
   888→
   889→				// Moved check here if metric matches hardware specs
   890→				if nativeScope == schema.MetricScopeAccelerator && len(acceleratorIds) == 0 {
   891→					continue
   892→				}
   893→
   894→				scopeResults, ok := BuildScopeQueries(
   895→					nativeScope, requestedScope,
   896→					metric, hostname,
   897→					&topology, topology.Node, acceleratorIds,
   898→				)
   899→
   900→				if !ok {
   901→					return nil, nil, fmt.Errorf("METRICDATA/INTERNAL-CCMS > TODO: unhandled case: native-scope=%s, requested-scope=%s", nativeScope, requestedScope)
   902→				}
   903→
   904→				for _, sr := range scopeResults {
   905→					queries = append(queries, APIQuery{
   906→						Metric:     sr.Metric,
   907→						Hostname:   sr.Hostname,
   908→						Aggregate:  sr.Aggregate,
   909→						Type:       sr.Type,
   910→						TypeIds:    sr.TypeIds,
   911→						Resolution: resolution,
   912→					})
   913→					assignedScope = append(assignedScope, sr.Scope)
   914→				}
   915→			}
   916→		}
   917→	}
   918→
   919→	return queries, assignedScope, nil
   920→}
   921→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// This file implements high-level query functions for loading job metric data
// with automatic scope transformation and aggregation.
//
// Key Concepts:
//
// Metric Scopes: Metrics are collected at different granularities (native scope):
//   - HWThread: Per hardware thread
//   - Core: Per CPU core
//   - Socket: Per CPU socket
//   - MemoryDomain: Per memory domain (NUMA)
//   - Accelerator: Per GPU/accelerator
//   - Node: Per compute node
//
// Scope Transformation: The buildQueries functions transform between native scope
// and requested scope by:
//   - Aggregating finer-grained data (e.g., HWThread → Core → Socket → Node)
//   - Rejecting requests for finer granularity than available
//   - Handling special cases (e.g., Accelerator metrics)
//
// Query Building: Constructs APIQuery structures with proper selectors (Type, TypeIds)
// based on cluster topology and job resources.
package metricstore

import (
	"context"
	"fmt"
	"strings"
	"time"

	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

type InternalMetricStore struct{}

var MetricStoreHandle *InternalMetricStore

// HealthCheck delegates to the internal MemoryStore's HealthCheck.
func (ccms *InternalMetricStore) HealthCheck(cluster string,
	nodes []string, metrics []string,
) (map[string]HealthCheckResult, error) {
	return GetMemoryStore().HealthCheck(cluster, nodes, metrics)
}

// TestLoadDataCallback allows tests to override LoadData behavior for testing purposes.
// When set to a non-nil function, LoadData will call this function instead of the default implementation.
var TestLoadDataCallback func(job *schema.Job, metrics []string, scopes []schema.MetricScope, ctx context.Context, resolution int) (schema.JobData, error)

// LoadData loads metric data for a specific job with automatic scope transformation.
//
// This is the primary function for retrieving job metric data. It handles:
//   - Building queries with scope transformation via buildQueries
//   - Fetching data from the metric store
//   - Organizing results by metric and scope
//   - Converting NaN statistics to 0 for JSON compatibility
//   - Partial error handling (returns data for successful queries even if some fail)
//
// Parameters:
//   - job: Job metadata including cluster, resources, and time range
//   - metrics: List of metric names to load
//   - scopes: Requested metric scopes (will be transformed to match native scopes)
//   - ctx: Context for cancellation (currently unused but reserved for future use)
//   - resolution: Data resolution in seconds (0 for native resolution)
//
// Returns:
//   - JobData: Map of metric → scope → JobMetric with time-series data and statistics
//   - Error: Returns error if query building or fetching fails, or partial error listing failed hosts
//
// Example:
//
//	jobData, err := LoadData(job, []string{"cpu_load", "mem_used"}, []schema.MetricScope{schema.MetricScopeNode}, ctx, 60)
func (ccms *InternalMetricStore) LoadData(
	job *schema.Job,
	metrics []string,
	scopes []schema.MetricScope,
	ctx context.Context,
	resolution int,
) (schema.JobData, error) {
	if TestLoadDataCallback != nil {
		return TestLoadDataCallback(job, metrics, scopes, ctx, resolution)
	}

	queries, assignedScope, err := buildQueries(job, metrics, scopes, int64(resolution))
	if err != nil {
		cclog.Errorf("Error while building queries for jobId %d, Metrics %v, Scopes %v: %s", job.JobID, metrics, scopes, err.Error())
		return nil, err
	}

	// Verify assignment is correct - log any inconsistencies for debugging
	if len(queries) != len(assignedScope) {
		cclog.Errorf("Critical error: queries and assignedScope have different lengths after buildQueries: %d vs %d",
			len(queries), len(assignedScope))
	}

	req := APIQueryRequest{
		Cluster:   job.Cluster,
		From:      job.StartTime,
		To:        job.StartTime + int64(job.Duration),
		Queries:   queries,
		WithStats: true,
		WithData:  true,
	}

	resBody, err := FetchData(req)
	if err != nil {
		cclog.Errorf("Error while fetching data : %s", err.Error())
		return nil, err
	}

	var errors []string
	jobData := make(schema.JobData)

	// Add safety check for potential index out of range errors
	if len(resBody.Results) != len(req.Queries) || len(assignedScope) != len(req.Queries) {
		cclog.Warnf("Mismatch in query results count: queries=%d, results=%d, assignedScope=%d",
			len(req.Queries), len(resBody.Results), len(assignedScope))
		if len(resBody.Results) > len(req.Queries) {
			resBody.Results = resBody.Results[:len(req.Queries)]
		}
		if len(assignedScope) > len(req.Queries) {
			assignedScope = assignedScope[:len(req.Queries)]
		}
	}

	for i, row := range resBody.Results {
		// Safety check to prevent index out of range errors
		if i >= len(req.Queries) || i >= len(assignedScope) {
			cclog.Warnf("Index out of range prevented: i=%d, queries=%d, assignedScope=%d",
				i, len(req.Queries), len(assignedScope))
			continue
		}

		query := req.Queries[i]
		metric := query.Metric
		scope := assignedScope[i]
		mc := archive.GetMetricConfig(job.Cluster, metric)

		if mc == nil {
			cclog.Warnf("Metric config not found for %s on cluster %s", metric, job.Cluster)
			continue
		}

		if _, ok := jobData[metric]; !ok {
			jobData[metric] = make(map[schema.MetricScope]*schema.JobMetric)
		}

		res := mc.Timestep
		if len(row) > 0 {
			res = int(row[0].Resolution)
		}

		jobMetric, ok := jobData[metric][scope]
		if !ok {
			jobMetric = &schema.JobMetric{
				Unit:     mc.Unit,
				Timestep: res,
				Series:   make([]schema.Series, 0),
			}
			jobData[metric][scope] = jobMetric
		}

		for ndx, res := range row {
			if res.Error != nil {
				/* Build list for "partial errors", if any */
				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
				continue
			}

			id := (*string)(nil)
			if query.Type != nil {
				// Check if ndx is within the bounds of TypeIds slice
				if ndx < len(query.TypeIds) {
					id = new(string)
					*id = query.TypeIds[ndx]
				} else {
					// Log the error but continue processing
					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
						ndx, len(query.TypeIds), query.Metric, query.Hostname)
				}
			}

			SanitizeStats(&res.Avg, &res.Min, &res.Max)

			jobMetric.Series = append(jobMetric.Series, schema.Series{
				Hostname: query.Hostname,
				ID:       id,
				Statistics: schema.MetricStatistics{
					Avg: float64(res.Avg),
					Min: float64(res.Min),
					Max: float64(res.Max),
				},
				Data: res.Data,
			})
		}

		// So that one can later check len(jobData):
		if len(jobMetric.Series) == 0 {
			delete(jobData[metric], scope)
			if len(jobData[metric]) == 0 {
				delete(jobData, metric)
			}
		}
	}

	if len(errors) != 0 {
		/* Returns list for "partial errors" */
		return jobData, fmt.Errorf("METRICDATA/INTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}
	return jobData, nil
}

// buildQueries constructs APIQuery structures with automatic scope transformation for a job.
//
// This function implements the core scope transformation logic, handling all combinations of
// native metric scopes and requested scopes. It uses the cluster topology to determine which
// hardware IDs to include in each query.
//
// Scope Transformation Rules:
//   - If native scope >= requested scope: Aggregates data (Aggregate=true in APIQuery)
//   - If native scope < requested scope: Returns error (cannot increase granularity)
//   - Special handling for Accelerator scope (independent of CPU hierarchy)
//
// The function generates one or more APIQuery per (metric, scope, host) combination:
//   - For non-aggregated queries: One query with all relevant IDs
//   - For aggregated queries: May generate multiple queries (e.g., one per socket/core)
//
// Parameters:
//   - job: Job metadata including cluster, subcluster, and resource allocation
//   - metrics: List of metrics to query
//   - scopes: Requested scopes for each metric
//   - resolution: Data resolution in seconds
//
// Returns:
//   - []APIQuery: List of queries to execute
//   - []schema.MetricScope: Assigned scope for each query (after transformation)
//   - error: Returns error if topology lookup fails or unhandled scope combination encountered
func buildQueries(
	job *schema.Job,
	metrics []string,
	scopes []schema.MetricScope,
	resolution int64,
) ([]APIQuery, []schema.MetricScope, error) {
	if len(job.Resources) == 0 {
		return nil, nil, fmt.Errorf("METRICDATA/INTERNAL-CCMS > no resources allocated for job %d", job.JobID)
	}

	queries := make([]APIQuery, 0, len(metrics)*len(scopes)*len(job.Resources))
	assignedScope := []schema.MetricScope{}

	subcluster, scerr := archive.GetSubCluster(job.Cluster, job.SubCluster)
	if scerr != nil {
		return nil, nil, scerr
	}
	topology := subcluster.Topology

	for _, metric := range metrics {
		mc := archive.GetMetricConfig(job.Cluster, metric)
		if mc == nil {
			cclog.Warnf("metric '%s' is not specified for cluster '%s'", metric, job.Cluster)
			continue
		}

		// Skip if metric is removed for subcluster
		if len(mc.SubClusters) != 0 {
			isRemoved := false
			for _, scConfig := range mc.SubClusters {
				if scConfig.Name == job.SubCluster && scConfig.Remove {
					isRemoved = true
					break
				}
			}
			if isRemoved {
				continue
			}
		}

		// Avoid duplicates...
		handledScopes := make([]schema.MetricScope, 0, 3)

	scopesLoop:
		for _, requestedScope := range scopes {
			nativeScope := mc.Scope
			if nativeScope == schema.MetricScopeAccelerator && job.NumAcc == 0 {
				continue
			}

			scope := nativeScope.Max(requestedScope)
			for _, s := range handledScopes {
				if scope == s {
					continue scopesLoop
				}
			}
			handledScopes = append(handledScopes, scope)

			for _, host := range job.Resources {
				hwthreads := host.HWThreads
				if hwthreads == nil {
					hwthreads = topology.Node
				}

				scopeResults, ok := BuildScopeQueries(
					nativeScope, requestedScope,
					metric, host.Hostname,
					&topology, hwthreads, host.Accelerators,
				)

				if !ok {
					return nil, nil, fmt.Errorf("METRICDATA/INTERNAL-CCMS > TODO: unhandled case: native-scope=%s, requested-scope=%s", nativeScope, requestedScope)
				}

				for _, sr := range scopeResults {
					queries = append(queries, APIQuery{
						Metric:     sr.Metric,
						Hostname:   sr.Hostname,
						Aggregate:  sr.Aggregate,
						Type:       sr.Type,
						TypeIds:    sr.TypeIds,
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, sr.Scope)
				}
			}
		}
	}

	return queries, assignedScope, nil
}

// LoadStats loads only metric statistics (avg/min/max) for a job at node scope.
//
// This is an optimized version of LoadData that fetches only statistics without
// time-series data, reducing bandwidth and memory usage. Always queries at node scope.
//
// Parameters:
//   - job: Job metadata
//   - metrics: List of metric names
//   - ctx: Context (currently unused)
//
// Returns:
//   - Map of metric → hostname → statistics
//   - Error on query building or fetching failure
func (ccms *InternalMetricStore) LoadStats(
	job *schema.Job,
	metrics []string,
	ctx context.Context,
) (map[string]map[string]schema.MetricStatistics, error) {
	// TODO(#166): Add scope parameter for analysis view accelerator normalization
	queries, _, err := buildQueries(job, metrics, []schema.MetricScope{schema.MetricScopeNode}, 0)
	if err != nil {
		cclog.Errorf("Error while building queries for jobId %d, Metrics %v: %s", job.JobID, metrics, err.Error())
		return nil, err
	}

	req := APIQueryRequest{
		Cluster:   job.Cluster,
		From:      job.StartTime,
		To:        job.StartTime + int64(job.Duration),
		Queries:   queries,
		WithStats: true,
		WithData:  false,
	}

	resBody, err := FetchData(req)
	if err != nil {
		cclog.Errorf("Error while fetching data : %s", err.Error())
		return nil, err
	}

	stats := make(map[string]map[string]schema.MetricStatistics, len(metrics))
	for i, res := range resBody.Results {
		if len(res) == 0 {
			// No Data Found For Metric, Logged in FetchData to Warn
			continue
		}
		query := req.Queries[i]
		metric := query.Metric
		data := res[0]
		if data.Error != nil {
			cclog.Warnf("fetching %s for node %s failed: %s", metric, query.Hostname, *data.Error)
			continue
		}

		metricdata, ok := stats[metric]
		if !ok {
			metricdata = make(map[string]schema.MetricStatistics, job.NumNodes)
			stats[metric] = metricdata
		}

		if data.Avg.IsNaN() || data.Min.IsNaN() || data.Max.IsNaN() {
			cclog.Warnf("fetching %s for node %s failed: one of avg/min/max is NaN", metric, query.Hostname)
			continue
		}

		metricdata[query.Hostname] = schema.MetricStatistics{
			Avg: float64(data.Avg),
			Min: float64(data.Min),
			Max: float64(data.Max),
		}
	}

	return stats, nil
}

// LoadScopedStats loads metric statistics for a job with scope-aware grouping.
//
// Similar to LoadStats but supports multiple scopes and returns statistics grouped
// by scope with hardware IDs (e.g., per-core, per-socket statistics).
//
// Parameters:
//   - job: Job metadata
//   - metrics: List of metric names
//   - scopes: Requested metric scopes
//   - ctx: Context (currently unused)
//
// Returns:
//   - ScopedJobStats: Map of metric → scope → []ScopedStats (with hostname and ID)
//   - Error or partial error listing failed queries
func (ccms *InternalMetricStore) LoadScopedStats(
	job *schema.Job,
	metrics []string,
	scopes []schema.MetricScope,
	ctx context.Context,
) (schema.ScopedJobStats, error) {
	queries, assignedScope, err := buildQueries(job, metrics, scopes, 0)
	if err != nil {
		cclog.Errorf("Error while building queries for jobId %d, Metrics %v, Scopes %v: %s", job.JobID, metrics, scopes, err.Error())
		return nil, err
	}

	req := APIQueryRequest{
		Cluster:   job.Cluster,
		From:      job.StartTime,
		To:        job.StartTime + int64(job.Duration),
		Queries:   queries,
		WithStats: true,
		WithData:  false,
	}

	resBody, err := FetchData(req)
	if err != nil {
		cclog.Errorf("Error while fetching data : %s", err.Error())
		return nil, err
	}

	var errors []string
	scopedJobStats := make(schema.ScopedJobStats)

	for i, row := range resBody.Results {
		if len(row) == 0 {
			// No Data Found For Metric, Logged in FetchData to Warn
			continue
		}
		query := req.Queries[i]
		metric := query.Metric
		scope := assignedScope[i]

		if _, ok := scopedJobStats[metric]; !ok {
			scopedJobStats[metric] = make(map[schema.MetricScope][]*schema.ScopedStats)
		}

		if _, ok := scopedJobStats[metric][scope]; !ok {
			scopedJobStats[metric][scope] = make([]*schema.ScopedStats, 0)
		}

		for ndx, res := range row {
			if res.Error != nil {
				/* Build list for "partial errors", if any */
				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
				continue
			}

			id := (*string)(nil)
			if query.Type != nil {
				// Check if ndx is within the bounds of TypeIds slice
				if ndx < len(query.TypeIds) {
					id = new(string)
					*id = query.TypeIds[ndx]
				} else {
					// Log the error but continue processing
					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
						ndx, len(query.TypeIds), query.Metric, query.Hostname)
				}
			}

			SanitizeStats(&res.Avg, &res.Min, &res.Max)

			scopedJobStats[metric][scope] = append(scopedJobStats[metric][scope], &schema.ScopedStats{
				Hostname: query.Hostname,
				ID:       id,
				Data: &schema.MetricStatistics{
					Avg: float64(res.Avg),
					Min: float64(res.Min),
					Max: float64(res.Max),
				},
			})
		}

		// So that one can later check len(scopedJobStats[metric][scope]): Remove from map if empty
		if len(scopedJobStats[metric][scope]) == 0 {
			delete(scopedJobStats[metric], scope)
			if len(scopedJobStats[metric]) == 0 {
				delete(scopedJobStats, metric)
			}
		}
	}

	if len(errors) != 0 {
		/* Returns list for "partial errors" */
		return scopedJobStats, fmt.Errorf("METRICDATA/INTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}
	return scopedJobStats, nil
}

// LoadNodeData loads metric data for specific nodes in a cluster over a time range.
//
// Unlike LoadData which operates on job resources, this function queries arbitrary nodes
// directly. Useful for system monitoring and node status views.
//
// Parameters:
//   - cluster: Cluster name
//   - metrics: List of metric names
//   - nodes: List of node hostnames (nil = all nodes in cluster via ForAllNodes)
//   - scopes: Requested metric scopes (currently unused - always node scope)
//   - from, to: Time range
//   - ctx: Context (currently unused)
//
// Returns:
//   - Map of hostname → metric → []JobMetric
//   - Error or partial error listing failed queries
func (ccms *InternalMetricStore) LoadNodeData(
	cluster string,
	metrics, nodes []string,
	scopes []schema.MetricScope,
	from, to time.Time,
	ctx context.Context,
) (map[string]map[string][]*schema.JobMetric, error) {
	req := APIQueryRequest{
		Cluster:   cluster,
		From:      from.Unix(),
		To:        to.Unix(),
		WithStats: true,
		WithData:  true,
	}

	if nodes == nil {
		req.ForAllNodes = append(req.ForAllNodes, metrics...)
	} else {
		for _, node := range nodes {
			for _, metric := range metrics {
				req.Queries = append(req.Queries, APIQuery{
					Hostname:   node,
					Metric:     metric,
					Resolution: 0, // Default for Node Queries: Will return metric $Timestep Resolution
				})
			}
		}
	}

	resBody, err := FetchData(req)
	if err != nil {
		cclog.Errorf("Error while fetching data : %s", err.Error())
		return nil, err
	}

	var errors []string
	data := make(map[string]map[string][]*schema.JobMetric)
	for i, res := range resBody.Results {
		if len(res) == 0 {
			// No Data Found For Metric, Logged in FetchData to Warn
			continue
		}

		var query APIQuery
		if resBody.Queries != nil {
			query = resBody.Queries[i]
		} else {
			query = req.Queries[i]
		}

		metric := query.Metric
		qdata := res[0]
		if qdata.Error != nil {
			errors = append(errors, fmt.Sprintf("fetching %s for node %s failed: %s", metric, query.Hostname, *qdata.Error))
		}

		SanitizeStats(&qdata.Avg, &qdata.Min, &qdata.Max)

		hostdata, ok := data[query.Hostname]
		if !ok {
			hostdata = make(map[string][]*schema.JobMetric)
			data[query.Hostname] = hostdata
		}

		mc := archive.GetMetricConfig(cluster, metric)
		hostdata[metric] = append(hostdata[metric], &schema.JobMetric{
			Unit:     mc.Unit,
			Timestep: mc.Timestep,
			Series: []schema.Series{
				{
					Hostname: query.Hostname,
					Data:     qdata.Data,
					Statistics: schema.MetricStatistics{
						Avg: float64(qdata.Avg),
						Min: float64(qdata.Min),
						Max: float64(qdata.Max),
					},
				},
			},
		})
	}

	if len(errors) != 0 {
		/* Returns list of "partial errors" */
		return data, fmt.Errorf("METRICDATA/INTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}

	return data, nil
}

// LoadNodeListData loads metric data for a list of nodes with full scope transformation support.
//
// This is the most flexible node data loading function, supporting arbitrary scopes and
// resolution. Uses buildNodeQueries for proper scope transformation based on topology.
//
// Parameters:
//   - cluster: Cluster name
//   - subCluster: SubCluster name (empty string to infer from node names)
//   - nodes: List of node hostnames
//   - metrics: List of metric names
//   - scopes: Requested metric scopes
//   - resolution: Data resolution in seconds
//   - from, to: Time range
//   - ctx: Context (currently unused)
//
// Returns:
//   - Map of hostname → JobData (metric → scope → JobMetric)
//   - Error or partial error listing failed queries
func (ccms *InternalMetricStore) LoadNodeListData(
	cluster, subCluster string,
	nodes []string,
	metrics []string,
	scopes []schema.MetricScope,
	resolution int,
	from, to time.Time,
	ctx context.Context,
) (map[string]schema.JobData, error) {
	// Note: Order of node data is not guaranteed after this point
	queries, assignedScope, err := buildNodeQueries(cluster, subCluster, nodes, metrics, scopes, int64(resolution))
	if err != nil {
		cclog.Errorf("Error while building node queries for Cluster %s, SubCLuster %s, Metrics %v, Scopes %v: %s", cluster, subCluster, metrics, scopes, err.Error())
		return nil, err
	}

	// Verify assignment is correct - log any inconsistencies for debugging
	if len(queries) != len(assignedScope) {
		cclog.Errorf("Critical error: queries and assignedScope have different lengths after buildNodeQueries: %d vs %d",
			len(queries), len(assignedScope))
	}

	req := APIQueryRequest{
		Cluster:   cluster,
		Queries:   queries,
		From:      from.Unix(),
		To:        to.Unix(),
		WithStats: true,
		WithData:  true,
	}

	resBody, err := FetchData(req)
	if err != nil {
		cclog.Errorf("Error while fetching data : %s", err.Error())
		return nil, err
	}

	var errors []string
	data := make(map[string]schema.JobData)

	// Add safety check for index out of range issues
	if len(resBody.Results) != len(req.Queries) || len(assignedScope) != len(req.Queries) {
		cclog.Warnf("Mismatch in query results count: queries=%d, results=%d, assignedScope=%d",
			len(req.Queries), len(resBody.Results), len(assignedScope))
		if len(resBody.Results) > len(req.Queries) {
			resBody.Results = resBody.Results[:len(req.Queries)]
		}
		if len(assignedScope) > len(req.Queries) {
			assignedScope = assignedScope[:len(req.Queries)]
		}
	}

	for i, row := range resBody.Results {
		// Safety check to prevent index out of range errors
		if i >= len(req.Queries) || i >= len(assignedScope) {
			cclog.Warnf("Index out of range prevented: i=%d, queries=%d, assignedScope=%d",
				i, len(req.Queries), len(assignedScope))
			continue
		}

		var query APIQuery
		if resBody.Queries != nil {
			if i < len(resBody.Queries) {
				query = resBody.Queries[i]
			} else {
				cclog.Warnf("Index out of range prevented for resBody.Queries: i=%d, len=%d",
					i, len(resBody.Queries))
				continue
			}
		} else {
			query = req.Queries[i]
		}

		metric := query.Metric
		scope := assignedScope[i]
		mc := archive.GetMetricConfig(cluster, metric)
		if mc == nil {
			cclog.Warnf("Metric config not found for %s on cluster %s", metric, cluster)
			continue
		}

		res := mc.Timestep
		if len(row) > 0 {
			res = int(row[0].Resolution)
		}

		// Init Nested Map Data Structures If Not Found
		hostData, ok := data[query.Hostname]
		if !ok {
			hostData = make(schema.JobData)
			data[query.Hostname] = hostData
		}

		metricData, ok := hostData[metric]
		if !ok {
			metricData = make(map[schema.MetricScope]*schema.JobMetric)
			data[query.Hostname][metric] = metricData
		}

		scopeData, ok := metricData[scope]
		if !ok {
			scopeData = &schema.JobMetric{
				Unit:     mc.Unit,
				Timestep: res,
				Series:   make([]schema.Series, 0),
			}
			data[query.Hostname][metric][scope] = scopeData
		}

		for ndx, res := range row {
			if res.Error != nil {
				/* Build list for "partial errors", if any */
				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
				continue
			}

			id := (*string)(nil)
			if query.Type != nil {
				// Check if ndx is within the bounds of TypeIds slice
				if ndx < len(query.TypeIds) {
					id = new(string)
					*id = query.TypeIds[ndx]
				} else {
					// Log the error but continue processing
					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
						ndx, len(query.TypeIds), query.Metric, query.Hostname)
				}
			}

			SanitizeStats(&res.Avg, &res.Min, &res.Max)

			scopeData.Series = append(scopeData.Series, schema.Series{
				Hostname: query.Hostname,
				ID:       id,
				Statistics: schema.MetricStatistics{
					Avg: float64(res.Avg),
					Min: float64(res.Min),
					Max: float64(res.Max),
				},
				Data: res.Data,
			})
		}
	}

	if len(errors) != 0 {
		/* Returns list of "partial errors" */
		return data, fmt.Errorf("METRICDATA/INTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}

	return data, nil
}

// buildNodeQueries constructs APIQuery structures for node-based queries with scope transformation.
//
// Similar to buildQueries but operates on node lists rather than job resources.
// Supports dynamic subcluster lookup when subCluster parameter is empty.
//
// Parameters:
//   - cluster: Cluster name
//   - subCluster: SubCluster name (empty = infer from node hostnames)
//   - nodes: List of node hostnames
//   - metrics: List of metric names
//   - scopes: Requested metric scopes
//   - resolution: Data resolution in seconds
//
// Returns:
//   - []APIQuery: List of queries to execute
//   - []schema.MetricScope: Assigned scope for each query
//   - error: Returns error if topology lookup fails or unhandled scope combination
func buildNodeQueries(
	cluster string,
	subCluster string,
	nodes []string,
	metrics []string,
	scopes []schema.MetricScope,
	resolution int64,
) ([]APIQuery, []schema.MetricScope, error) {
	queries := make([]APIQuery, 0, len(metrics)*len(scopes)*len(nodes))
	assignedScope := []schema.MetricScope{}

	// Get Topol before loop if subCluster given
	var subClusterTopol *schema.SubCluster
	var scterr error
	if subCluster != "" {
		subClusterTopol, scterr = archive.GetSubCluster(cluster, subCluster)
		if scterr != nil {
			cclog.Errorf("could not load cluster %s subCluster %s topology: %s", cluster, subCluster, scterr.Error())
			return nil, nil, scterr
		}
	}

	for _, metric := range metrics {
		mc := archive.GetMetricConfig(cluster, metric)
		if mc == nil {
			cclog.Warnf("metric '%s' is not specified for cluster '%s'", metric, cluster)
			continue
		}

		// Skip if metric is removed for subcluster
		if mc.SubClusters != nil {
			isRemoved := false
			for _, scConfig := range mc.SubClusters {
				if scConfig.Name == subCluster && scConfig.Remove {
					isRemoved = true
					break
				}
			}
			if isRemoved {
				continue
			}
		}

		// Avoid duplicates...
		handledScopes := make([]schema.MetricScope, 0, 3)

	nodeScopesLoop:
		for _, requestedScope := range scopes {
			nativeScope := mc.Scope

			scope := nativeScope.Max(requestedScope)
			for _, s := range handledScopes {
				if scope == s {
					continue nodeScopesLoop
				}
			}
			handledScopes = append(handledScopes, scope)

			for _, hostname := range nodes {

				// If no subCluster given, get it by node
				if subCluster == "" {
					subClusterName, scnerr := archive.GetSubClusterByNode(cluster, hostname)
					if scnerr != nil {
						return nil, nil, scnerr
					}
					subClusterTopol, scterr = archive.GetSubCluster(cluster, subClusterName)
					if scterr != nil {
						return nil, nil, scterr
					}
				}

				// Always full node hwthread id list, no partial queries expected
				topology := subClusterTopol.Topology
				acceleratorIds := topology.GetAcceleratorIDs()

				// Moved check here if metric matches hardware specs
				if nativeScope == schema.MetricScopeAccelerator && len(acceleratorIds) == 0 {
					continue
				}

				scopeResults, ok := BuildScopeQueries(
					nativeScope, requestedScope,
					metric, hostname,
					&topology, topology.Node, acceleratorIds,
				)

				if !ok {
					return nil, nil, fmt.Errorf("METRICDATA/INTERNAL-CCMS > TODO: unhandled case: native-scope=%s, requested-scope=%s", nativeScope, requestedScope)
				}

				for _, sr := range scopeResults {
					queries = append(queries, APIQuery{
						Metric:     sr.Metric,
						Hostname:   sr.Hostname,
						Aggregate:  sr.Aggregate,
						Type:       sr.Type,
						TypeIds:    sr.TypeIds,
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, sr.Scope)
				}
			}
		}
	}

	return queries, assignedScope, nil
}