Implement the following plan:

# Consolidate buildQueries / Scope Transformation Logic

## Context

`pkg/metricstore/query.go` (InternalMetricStore) and `internal/metricstoreclient/cc-metric-store-queries.go` (CCMetricStore) both implement the `MetricDataRepository` interface. They contain nearly identical scope-transformation logic for building metric queries, but the code has diverged:

- **metricstoreclient** has the cleaner design: a standalone `buildScopeQueries()` function handling all transformations
- **metricstore** has the same logic inlined as a long if-chain inside `buildQueries()` and duplicated in `buildNodeQueries()`
- **metricstore is missing** the `MemoryDomain → Socket` transformation that metricstoreclient has
- Both packages duplicate scope string constants, `intToStringSlice`, and `sanitizeStats`

Goal: Extract shared scope-transformation logic, synchronize behavior, and eliminate duplication.

## Approach

Extract `BuildScopeQueries` as a shared exported function in `pkg/metricstore/` (since metricstoreclient already imports it). Return a package-independent intermediate type (`ScopeQueryResult`) that each package converts to its own `APIQuery` — this avoids coupling the two different `APIQuery` types (they differ in `Resolution` type: `int64` vs `int`, and internal has `ScaleFactor`).

## Steps

### 1. Create `pkg/metricstore/scopequery.go`

New file containing:

- **`ScopeQueryResult`** struct — intermediate type with fields: `Metric`, `Hostname`, `Aggregate`, `Type *string`, `TypeIds []string`, `Scope schema.MetricScope` (no Resolution — callers set it)
- **`BuildScopeQueries()`** — exported function ported from `metricstoreclient/buildScopeQueries()` (lines 257-517), returning `([]ScopeQueryResult, error)`. Returns empty slice for expected exceptions, error for unhandled cases
- **Scope string constants** — `HWThreadString`, `CoreString`, `MemoryDomainString`, `SocketString`, `AcceleratorString` (exported, move from both packages)
- **`IntToStringSlice()`** — exported, using the optimized buffer-reuse version from metricstore
- **`SanitizeStats(avg, min, max *schema.Float)`** — exported, using the metricstoreclient behavior (zero ALL if ANY is NaN)

### 2. Update `pkg/metricstore/query.go`

- Replace inline if-chain in `buildQueries()` (lines 317-534) with call to `BuildScopeQueries()`, converting each `ScopeQueryResult` → local `APIQuery` (adding `Resolution`)
- Replace inline if-chain in `buildNodeQueries()` (lines 1099-1316) with same pattern
- Delete local scope string constants, `intToStringSlice`, local `sanitizeStats`
- Update `sanitizeStats` calls in `LoadData`/`LoadScopedStats`/`LoadNodeData`/`LoadNodeListData` to use `SanitizeStats(&res.Avg, &res.Min, &res.Max)`
- Switch scope dedup from `map[schema.MetricScope]bool` to labeled-loop + slice (matches metricstoreclient, avoids allocation for typical 1-3 scopes)
- This **adds the missing MemoryDomain → Socket transformation** to the internal store

### 3. Update `internal/metricstoreclient/cc-metric-store-queries.go`

- Replace call to local `buildScopeQueries()` with `metricstore.BuildScopeQueries()`, converting each `ScopeQueryResult` → local `APIQuery` (adding `Resolution`)
- Delete local `buildScopeQueries()` function
- Delete local scope string constants and `intToStringSlice`
- Update `sanitizeStats` in `cc-metric-store.go` to call `metricstore.SanitizeStats()`

### 4. Add tests: `pkg/metricstore/scopequery_test.go`

Table-driven tests for `BuildScopeQueries` covering all transformation cases:
- Same-scope (HWThread→HWThread, Core→Core, Socket→Socket, Node→Node, etc.)
- Aggregation (HWThread→Core, HWThread→Socket, HWThread→Node, Core→Socket, Core→Node, etc.)
- Accelerator cases (Accelerator→Accelerator, Accelerator→Node)
- MemoryDomain cases including MemoryDomain→Socket
- Unhandled case returns error

## Files Modified

| File | Action |
|------|--------|
| `pkg/metricstore/scopequery.go` | **New** — shared types and functions |
| `pkg/metricstore/scopequery_test.go` | **New** — tests for BuildScopeQueries |
| `pkg/metricstore/query.go` | Refactor buildQueries/buildNodeQueries to use BuildScopeQueries; remove duplicated helpers |
| `internal/metricstoreclient/cc-metric-store-queries.go` | Replace local buildScopeQueries with shared; remove duplicated helpers |
| `internal/metricstoreclient/cc-metric-store.go` | Update sanitizeStats to use shared SanitizeStats |

## What is NOT shared (intentionally)

- **`APIQuery`/`APIMetricData` types** — different Resolution types, Data types, ScaleFactor field
- **`buildQueries`/`buildNodeQueries` themselves** — different topology access (caching vs inline), different receiver types
- **Load* result-processing functions** — depend on package-specific types and fetch mechanisms

## Verification

1. `go build ./...` — compile check
2. `go test ./pkg/metricstore/...` — existing + new scopequery tests
3. `go test ./internal/metricstoreclient/...` — compile check
4. `make test` — full test suite

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-ClusterCockpit-cc-backend/eec1fc87-d8f6-42f6-b296-d7a2f79af002.jsonl


Let me start by reading the key files to understand the current code before implementing the plan.


I'll start by reading the key source files to understand the current code structure.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstoreclient - Query Building
     7→//
     8→// This file contains the query construction and scope transformation logic for cc-metric-store queries.
     9→// It handles the complex mapping between requested metric scopes and native hardware topology,
    10→// automatically aggregating or filtering metrics as needed.
    11→//
    12→// # Scope Transformations
    13→//
    14→// The buildScopeQueries function implements the core scope transformation algorithm.
    15→// It handles 25+ different transformation cases, mapping between:
    16→//   - Accelerator (GPU) scope
    17→//   - HWThread (hardware thread/SMT) scope
    18→//   - Core (CPU core) scope
    19→//   - Socket (CPU package) scope
    20→//   - MemoryDomain (NUMA domain) scope
    21→//   - Node (full system) scope
    22→//
    23→// Transformations follow these rules:
    24→//   - Same scope: Return data as-is (e.g., Core → Core)
    25→//   - Coarser scope: Aggregate data (e.g., Core → Socket with Aggregate=true)
    26→//   - Finer scope: Error - cannot increase granularity
    27→//
    28→// # Query Building
    29→//
    30→// buildQueries and buildNodeQueries are the main entry points, handling job-specific
    31→// and node-specific query construction respectively. They:
    32→//   - Validate metric configurations
    33→//   - Handle subcluster-specific metric filtering
    34→//   - Detect and skip duplicate scope requests
    35→//   - Call buildScopeQueries for each metric/scope/host combination
    36→package metricstoreclient
    37→
    38→import (
    39→	"fmt"
    40→	"strconv"
    41→
    42→	"github.com/ClusterCockpit/cc-backend/pkg/archive"
    43→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    44→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    45→)
    46→
    47→// Scope string constants used in API queries.
    48→// Pre-converted to avoid repeated allocations during query building.
    49→var (
    50→	hwthreadString     = string(schema.MetricScopeHWThread)
    51→	coreString         = string(schema.MetricScopeCore)
    52→	memoryDomainString = string(schema.MetricScopeMemoryDomain)
    53→	socketString       = string(schema.MetricScopeSocket)
    54→	acceleratorString  = string(schema.MetricScopeAccelerator)
    55→)
    56→
    57→// buildQueries constructs API queries for job-specific metric data.
    58→// It iterates through metrics, scopes, and job resources to build the complete query set.
    59→//
    60→// The function handles:
    61→//   - Metric configuration validation and subcluster filtering
    62→//   - Scope deduplication to avoid redundant queries
    63→//   - Hardware thread list resolution (job-allocated vs full node)
    64→//   - Delegation to buildScopeQueries for scope transformations
    65→//
    66→// Returns queries and their corresponding assigned scopes (which may differ from requested scopes).
    67→func (ccms *CCMetricStore) buildQueries(
    68→	job *schema.Job,
    69→	metrics []string,
    70→	scopes []schema.MetricScope,
    71→	resolution int,
    72→) ([]APIQuery, []schema.MetricScope, error) {
    73→	// Initialize both slices together
    74→	queries := make([]APIQuery, 0, len(metrics)*len(scopes)*len(job.Resources))
    75→	assignedScope := make([]schema.MetricScope, 0, len(metrics)*len(scopes)*len(job.Resources))
    76→
    77→	topology, err := ccms.getTopology(job.Cluster, job.SubCluster)
    78→	if err != nil {
    79→		cclog.Errorf("could not load cluster %s subCluster %s topology: %s", job.Cluster, job.SubCluster, err.Error())
    80→		return nil, nil, err
    81→	}
    82→
    83→	for _, metric := range metrics {
    84→		remoteName := metric
    85→		mc := archive.GetMetricConfig(job.Cluster, metric)
    86→		if mc == nil {
    87→			cclog.Warnf("metric '%s' is not specified for cluster '%s' - skipping", metric, job.Cluster)
    88→			continue
    89→		}
    90→
    91→		// Skip if metric is removed for subcluster
    92→		if len(mc.SubClusters) != 0 {
    93→			isRemoved := false
    94→			for _, scConfig := range mc.SubClusters {
    95→				if scConfig.Name == job.SubCluster && scConfig.Remove {
    96→					isRemoved = true
    97→					break
    98→				}
    99→			}
   100→			if isRemoved {
   101→				continue
   102→			}
   103→		}
   104→
   105→		// Avoid duplicates...
   106→		handledScopes := make([]schema.MetricScope, 0, 3)
   107→
   108→	scopesLoop:
   109→		for _, requestedScope := range scopes {
   110→			nativeScope := mc.Scope
   111→			if nativeScope == schema.MetricScopeAccelerator && job.NumAcc == 0 {
   112→				continue
   113→			}
   114→
   115→			scope := nativeScope.Max(requestedScope)
   116→			for _, s := range handledScopes {
   117→				if scope == s {
   118→					continue scopesLoop
   119→				}
   120→			}
   121→			handledScopes = append(handledScopes, scope)
   122→
   123→			for _, host := range job.Resources {
   124→				hwthreads := host.HWThreads
   125→				if hwthreads == nil {
   126→					hwthreads = topology.Node
   127→				}
   128→
   129→				// Note: Expected exceptions will return as empty slices -> Continue
   130→				hostQueries, hostScopes := buildScopeQueries(
   131→					nativeScope, requestedScope,
   132→					remoteName, host.Hostname,
   133→					topology, hwthreads, host.Accelerators,
   134→					resolution,
   135→				)
   136→
   137→				// Note: Unexpected errors, such as unhandled cases, will return as nils -> Error
   138→				if hostQueries == nil && hostScopes == nil {
   139→					return nil, nil, fmt.Errorf("METRICDATA/EXTERNAL-CCMS > TODO: unhandled case: native-scope=%s, requested-scope=%s", nativeScope, requestedScope)
   140→				}
   141→
   142→				queries = append(queries, hostQueries...)
   143→				assignedScope = append(assignedScope, hostScopes...)
   144→			}
   145→		}
   146→	}
   147→
   148→	return queries, assignedScope, nil
   149→}
   150→
   151→// buildNodeQueries constructs API queries for node-specific metric data (Systems View).
   152→// Similar to buildQueries but uses full node topology instead of job-allocated resources.
   153→//
   154→// The function handles:
   155→//   - SubCluster topology resolution (either pre-loaded or per-node lookup)
   156→//   - Full node hardware thread lists (not job-specific subsets)
   157→//   - All accelerators on each node
   158→//   - Metric configuration validation with subcluster filtering
   159→//
   160→// Returns queries and their corresponding assigned scopes.
   161→func (ccms *CCMetricStore) buildNodeQueries(
   162→	cluster string,
   163→	subCluster string,
   164→	nodes []string,
   165→	metrics []string,
   166→	scopes []schema.MetricScope,
   167→	resolution int,
   168→) ([]APIQuery, []schema.MetricScope, error) {
   169→	// Initialize both slices together
   170→	queries := make([]APIQuery, 0, len(metrics)*len(scopes)*len(nodes))
   171→	assignedScope := make([]schema.MetricScope, 0, len(metrics)*len(scopes)*len(nodes))
   172→
   173→	for _, metric := range metrics {
   174→		remoteName := metric
   175→		mc := archive.GetMetricConfig(cluster, metric)
   176→		if mc == nil {
   177→			cclog.Warnf("metric '%s' is not specified for cluster '%s'", metric, cluster)
   178→			continue
   179→		}
   180→
   181→		// Skip if metric is removed for subcluster
   182→		if mc.SubClusters != nil {
   183→			isRemoved := false
   184→			for _, scConfig := range mc.SubClusters {
   185→				if scConfig.Name == subCluster && scConfig.Remove {
   186→					isRemoved = true
   187→					break
   188→				}
   189→			}
   190→			if isRemoved {
   191→				continue
   192→			}
   193→		}
   194→
   195→		// Avoid duplicates...
   196→		handledScopes := make([]schema.MetricScope, 0, 3)
   197→
   198→	scopesLoop:
   199→		for _, requestedScope := range scopes {
   200→			nativeScope := mc.Scope
   201→
   202→			scope := nativeScope.Max(requestedScope)
   203→			for _, s := range handledScopes {
   204→				if scope == s {
   205→					continue scopesLoop
   206→				}
   207→			}
   208→			handledScopes = append(handledScopes, scope)
   209→
   210→			for _, hostname := range nodes {
   211→				var topology *schema.Topology
   212→				var err error
   213→
   214→				// If no subCluster given, get it by node
   215→				if subCluster == "" {
   216→					topology, err = ccms.getTopologyByNode(cluster, hostname)
   217→				} else {
   218→					topology, err = ccms.getTopology(cluster, subCluster)
   219→				}
   220→
   221→				if err != nil {
   222→					return nil, nil, err
   223→				}
   224→
   225→				// Always full node hwthread id list, no partial queries expected -> Use "topology.Node" directly where applicable
   226→				// Always full accelerator id list, no partial queries expected -> Use "acceleratorIds" directly where applicable
   227→				acceleratorIds := topology.GetAcceleratorIDs()
   228→
   229→				// Moved check here if metric matches hardware specs
   230→				if nativeScope == schema.MetricScopeAccelerator && len(acceleratorIds) == 0 {
   231→					continue scopesLoop
   232→				}
   233→
   234→				nodeQueries, nodeScopes := buildScopeQueries(
   235→					nativeScope, requestedScope,
   236→					remoteName, hostname,
   237→					topology, topology.Node, acceleratorIds,
   238→					resolution,
   239→				)
   240→
   241→				if len(nodeQueries) == 0 && len(nodeScopes) == 0 {
   242→					return nil, nil, fmt.Errorf("METRICDATA/EXTERNAL-CCMS > TODO: unhandled case: native-scope=%s, requested-scope=%s", nativeScope, requestedScope)
   243→				}
   244→
   245→				queries = append(queries, nodeQueries...)
   246→				assignedScope = append(assignedScope, nodeScopes...)
   247→			}
   248→		}
   249→	}
   250→
   251→	return queries, assignedScope, nil
   252→}
   253→
   254→// buildScopeQueries generates API queries for a given scope transformation.
   255→// It returns a slice of queries and corresponding assigned scopes.
   256→// Some transformations (e.g., HWThread -> Core/Socket) may generate multiple queries.
   257→func buildScopeQueries(
   258→	nativeScope, requestedScope schema.MetricScope,
   259→	metric, hostname string,
   260→	topology *schema.Topology,
   261→	hwthreads []int,
   262→	accelerators []string,
   263→	resolution int,
   264→) ([]APIQuery, []schema.MetricScope) {
   265→	scope := nativeScope.Max(requestedScope)
   266→	queries := []APIQuery{}
   267→	scopes := []schema.MetricScope{}
   268→
   269→	hwthreadsStr := intToStringSlice(hwthreads)
   270→
   271→	// Accelerator -> Accelerator (Use "accelerator" scope if requested scope is lower than node)
   272→	if nativeScope == schema.MetricScopeAccelerator && scope.LT(schema.MetricScopeNode) {
   273→		if scope != schema.MetricScopeAccelerator {
   274→			// Expected Exception -> Continue -> Return Empty Slices
   275→			return queries, scopes
   276→		}
   277→
   278→		queries = append(queries, APIQuery{
   279→			Metric:     metric,
   280→			Hostname:   hostname,
   281→			Aggregate:  false,
   282→			Type:       &acceleratorString,
   283→			TypeIds:    accelerators,
   284→			Resolution: resolution,
   285→		})
   286→		scopes = append(scopes, schema.MetricScopeAccelerator)
   287→		return queries, scopes
   288→	}
   289→
   290→	// Accelerator -> Node
   291→	if nativeScope == schema.MetricScopeAccelerator && scope == schema.MetricScopeNode {
   292→		if len(accelerators) == 0 {
   293→			// Expected Exception -> Continue -> Return Empty Slices
   294→			return queries, scopes
   295→		}
   296→
   297→		queries = append(queries, APIQuery{
   298→			Metric:     metric,
   299→			Hostname:   hostname,
   300→			Aggregate:  true,
   301→			Type:       &acceleratorString,
   302→			TypeIds:    accelerators,
   303→			Resolution: resolution,
   304→		})
   305→		scopes = append(scopes, scope)
   306→		return queries, scopes
   307→	}
   308→
   309→	// HWThread -> HWThread
   310→	if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeHWThread {
   311→		queries = append(queries, APIQuery{
   312→			Metric:     metric,
   313→			Hostname:   hostname,
   314→			Aggregate:  false,
   315→			Type:       &hwthreadString,
   316→			TypeIds:    hwthreadsStr,
   317→			Resolution: resolution,
   318→		})
   319→		scopes = append(scopes, scope)
   320→		return queries, scopes
   321→	}
   322→
   323→	// HWThread -> Core
   324→	if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeCore {
   325→		cores, _ := topology.GetCoresFromHWThreads(hwthreads)
   326→		for _, core := range cores {
   327→			queries = append(queries, APIQuery{
   328→				Metric:     metric,
   329→				Hostname:   hostname,
   330→				Aggregate:  true,
   331→				Type:       &hwthreadString,
   332→				TypeIds:    intToStringSlice(topology.Core[core]),
   333→				Resolution: resolution,
   334→			})
   335→			scopes = append(scopes, scope)
   336→		}
   337→		return queries, scopes
   338→	}
   339→
   340→	// HWThread -> Socket
   341→	if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeSocket {
   342→		sockets, _ := topology.GetSocketsFromHWThreads(hwthreads)
   343→		for _, socket := range sockets {
   344→			queries = append(queries, APIQuery{
   345→				Metric:     metric,
   346→				Hostname:   hostname,
   347→				Aggregate:  true,
   348→				Type:       &hwthreadString,
   349→				TypeIds:    intToStringSlice(topology.Socket[socket]),
   350→				Resolution: resolution,
   351→			})
   352→			scopes = append(scopes, scope)
   353→		}
   354→		return queries, scopes
   355→	}
   356→
   357→	// HWThread -> Node
   358→	if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeNode {
   359→		queries = append(queries, APIQuery{
   360→			Metric:     metric,
   361→			Hostname:   hostname,
   362→			Aggregate:  true,
   363→			Type:       &hwthreadString,
   364→			TypeIds:    hwthreadsStr,
   365→			Resolution: resolution,
   366→		})
   367→		scopes = append(scopes, scope)
   368→		return queries, scopes
   369→	}
   370→
   371→	// Core -> Core
   372→	if nativeScope == schema.MetricScopeCore && scope == schema.MetricScopeCore {
   373→		cores, _ := topology.GetCoresFromHWThreads(hwthreads)
   374→		queries = append(queries, APIQuery{
   375→			Metric:     metric,
   376→			Hostname:   hostname,
   377→			Aggregate:  false,
   378→			Type:       &coreString,
   379→			TypeIds:    intToStringSlice(cores),
   380→			Resolution: resolution,
   381→		})
   382→		scopes = append(scopes, scope)
   383→		return queries, scopes
   384→	}
   385→
   386→	// Core -> Socket
   387→	if nativeScope == schema.MetricScopeCore && scope == schema.MetricScopeSocket {
   388→		sockets, _ := topology.GetSocketsFromCores(hwthreads)
   389→		for _, socket := range sockets {
   390→			queries = append(queries, APIQuery{
   391→				Metric:     metric,
   392→				Hostname:   hostname,
   393→				Aggregate:  true,
   394→				Type:       &coreString,
   395→				TypeIds:    intToStringSlice(topology.Socket[socket]),
   396→				Resolution: resolution,
   397→			})
   398→			scopes = append(scopes, scope)
   399→		}
   400→		return queries, scopes
   401→	}
   402→
   403→	// Core -> Node
   404→	if nativeScope == schema.MetricScopeCore && scope == schema.MetricScopeNode {
   405→		cores, _ := topology.GetCoresFromHWThreads(hwthreads)
   406→		queries = append(queries, APIQuery{
   407→			Metric:     metric,
   408→			Hostname:   hostname,
   409→			Aggregate:  true,
   410→			Type:       &coreString,
   411→			TypeIds:    intToStringSlice(cores),
   412→			Resolution: resolution,
   413→		})
   414→		scopes = append(scopes, scope)
   415→		return queries, scopes
   416→	}
   417→
   418→	// MemoryDomain -> MemoryDomain
   419→	if nativeScope == schema.MetricScopeMemoryDomain && scope == schema.MetricScopeMemoryDomain {
   420→		memDomains, _ := topology.GetMemoryDomainsFromHWThreads(hwthreads)
   421→		queries = append(queries, APIQuery{
   422→			Metric:     metric,
   423→			Hostname:   hostname,
   424→			Aggregate:  false,
   425→			Type:       &memoryDomainString,
   426→			TypeIds:    intToStringSlice(memDomains),
   427→			Resolution: resolution,
   428→		})
   429→		scopes = append(scopes, scope)
   430→		return queries, scopes
   431→	}
   432→
   433→	// MemoryDomain -> Node
   434→	if nativeScope == schema.MetricScopeMemoryDomain && scope == schema.MetricScopeNode {
   435→		memDomains, _ := topology.GetMemoryDomainsFromHWThreads(hwthreads)
   436→		queries = append(queries, APIQuery{
   437→			Metric:     metric,
   438→			Hostname:   hostname,
   439→			Aggregate:  true,
   440→			Type:       &memoryDomainString,
   441→			TypeIds:    intToStringSlice(memDomains),
   442→			Resolution: resolution,
   443→		})
   444→		scopes = append(scopes, scope)
   445→		return queries, scopes
   446→	}
   447→
   448→	// MemoryDomain -> Socket
   449→	if nativeScope == schema.MetricScopeMemoryDomain && scope == schema.MetricScopeSocket {
   450→		memDomains, _ := topology.GetMemoryDomainsFromHWThreads(hwthreads)
   451→		socketToDomains, err := topology.GetMemoryDomainsBySocket(memDomains)
   452→		if err != nil {
   453→			cclog.Errorf("Error mapping memory domains to sockets, return unchanged: %v", err)
   454→			// Rare Error Case -> Still Continue -> Return Empty Slices
   455→			return queries, scopes
   456→		}
   457→
   458→		// Create a query for each socket
   459→		for _, domains := range socketToDomains {
   460→			queries = append(queries, APIQuery{
   461→				Metric:     metric,
   462→				Hostname:   hostname,
   463→				Aggregate:  true,
   464→				Type:       &memoryDomainString,
   465→				TypeIds:    intToStringSlice(domains),
   466→				Resolution: resolution,
   467→			})
   468→			// Add scope for each query, not just once
   469→			scopes = append(scopes, scope)
   470→		}
   471→		return queries, scopes
   472→	}
   473→
   474→	// Socket -> Socket
   475→	if nativeScope == schema.MetricScopeSocket && scope == schema.MetricScopeSocket {
   476→		sockets, _ := topology.GetSocketsFromHWThreads(hwthreads)
   477→		queries = append(queries, APIQuery{
   478→			Metric:     metric,
   479→			Hostname:   hostname,
   480→			Aggregate:  false,
   481→			Type:       &socketString,
   482→			TypeIds:    intToStringSlice(sockets),
   483→			Resolution: resolution,
   484→		})
   485→		scopes = append(scopes, scope)
   486→		return queries, scopes
   487→	}
   488→
   489→	// Socket -> Node
   490→	if nativeScope == schema.MetricScopeSocket && scope == schema.MetricScopeNode {
   491→		sockets, _ := topology.GetSocketsFromHWThreads(hwthreads)
   492→		queries = append(queries, APIQuery{
   493→			Metric:     metric,
   494→			Hostname:   hostname,
   495→			Aggregate:  true,
   496→			Type:       &socketString,
   497→			TypeIds:    intToStringSlice(sockets),
   498→			Resolution: resolution,
   499→		})
   500→		scopes = append(scopes, scope)
   501→		return queries, scopes
   502→	}
   503→
   504→	// Node -> Node
   505→	if nativeScope == schema.MetricScopeNode && scope == schema.MetricScopeNode {
   506→		queries = append(queries, APIQuery{
   507→			Metric:     metric,
   508→			Hostname:   hostname,
   509→			Resolution: resolution,
   510→		})
   511→		scopes = append(scopes, scope)
   512→		return queries, scopes
   513→	}
   514→
   515→	// Unhandled Case -> Error -> Return nils
   516→	return nil, nil
   517→}
   518→
   519→// intToStringSlice converts a slice of integers to a slice of strings.
   520→// Used to convert hardware IDs (core IDs, socket IDs, etc.) to the string format required by the API.
   521→func intToStringSlice(is []int) []string {
   522→	ss := make([]string, len(is))
   523→	for i, x := range is {
   524→		ss[i] = strconv.Itoa(x)
   525→	}
   526→	return ss
   527→}
   528→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package metricstoreclient - Query Building
//
// This file contains the query construction and scope transformation logic for cc-metric-store queries.
// It handles the complex mapping between requested metric scopes and native hardware topology,
// automatically aggregating or filtering metrics as needed.
//
// # Scope Transformations
//
// The buildScopeQueries function implements the core scope transformation algorithm.
// It handles 25+ different transformation cases, mapping between:
//   - Accelerator (GPU) scope
//   - HWThread (hardware thread/SMT) scope
//   - Core (CPU core) scope
//   - Socket (CPU package) scope
//   - MemoryDomain (NUMA domain) scope
//   - Node (full system) scope
//
// Transformations follow these rules:
//   - Same scope: Return data as-is (e.g., Core → Core)
//   - Coarser scope: Aggregate data (e.g., Core → Socket with Aggregate=true)
//   - Finer scope: Error - cannot increase granularity
//
// # Query Building
//
// buildQueries and buildNodeQueries are the main entry points, handling job-specific
// and node-specific query construction respectively. They:
//   - Validate metric configurations
//   - Handle subcluster-specific metric filtering
//   - Detect and skip duplicate scope requests
//   - Call buildScopeQueries for each metric/scope/host combination
package metricstoreclient

import (
	"fmt"
	"strconv"

	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

// Scope string constants used in API queries.
// Pre-converted to avoid repeated allocations during query building.
var (
	hwthreadString     = string(schema.MetricScopeHWThread)
	coreString         = string(schema.MetricScopeCore)
	memoryDomainString = string(schema.MetricScopeMemoryDomain)
	socketString       = string(schema.MetricScopeSocket)
	acceleratorString  = string(schema.MetricScopeAccelerator)
)

// buildQueries constructs API queries for job-specific metric data.
// It iterates through metrics, scopes, and job resources to build the complete query set.
//
// The function handles:
//   - Metric configuration validation and subcluster filtering
//   - Scope deduplication to avoid redundant queries
//   - Hardware thread list resolution (job-allocated vs full node)
//   - Delegation to buildScopeQueries for scope transformations
//
// Returns queries and their corresponding assigned scopes (which may differ from requested scopes).
func (ccms *CCMetricStore) buildQueries(
	job *schema.Job,
	metrics []string,
	scopes []schema.MetricScope,
	resolution int,
) ([]APIQuery, []schema.MetricScope, error) {
	// Initialize both slices together
	queries := make([]APIQuery, 0, len(metrics)*len(scopes)*len(job.Resources))
	assignedScope := make([]schema.MetricScope, 0, len(metrics)*len(scopes)*len(job.Resources))

	topology, err := ccms.getTopology(job.Cluster, job.SubCluster)
	if err != nil {
		cclog.Errorf("could not load cluster %s subCluster %s topology: %s", job.Cluster, job.SubCluster, err.Error())
		return nil, nil, err
	}

	for _, metric := range metrics {
		remoteName := metric
		mc := archive.GetMetricConfig(job.Cluster, metric)
		if mc == nil {
			cclog.Warnf("metric '%s' is not specified for cluster '%s' - skipping", metric, job.Cluster)
			continue
		}

		// Skip if metric is removed for subcluster
		if len(mc.SubClusters) != 0 {
			isRemoved := false
			for _, scConfig := range mc.SubClusters {
				if scConfig.Name == job.SubCluster && scConfig.Remove {
					isRemoved = true
					break
				}
			}
			if isRemoved {
				continue
			}
		}

		// Avoid duplicates...
		handledScopes := make([]schema.MetricScope, 0, 3)

	scopesLoop:
		for _, requestedScope := range scopes {
			nativeScope := mc.Scope
			if nativeScope == schema.MetricScopeAccelerator && job.NumAcc == 0 {
				continue
			}

			scope := nativeScope.Max(requestedScope)
			for _, s := range handledScopes {
				if scope == s {
					continue scopesLoop
				}
			}
			handledScopes = append(handledScopes, scope)

			for _, host := range job.Resources {
				hwthreads := host.HWThreads
				if hwthreads == nil {
					hwthreads = topology.Node
				}

				// Note: Expected exceptions will return as empty slices -> Continue
				hostQueries, hostScopes := buildScopeQueries(
					nativeScope, requestedScope,
					remoteName, host.Hostname,
					topology, hwthreads, host.Accelerators,
					resolution,
				)

				// Note: Unexpected errors, such as unhandled cases, will return as nils -> Error
				if hostQueries == nil && hostScopes == nil {
					return nil, nil, fmt.Errorf("METRICDATA/EXTERNAL-CCMS > TODO: unhandled case: native-scope=%s, requested-scope=%s", nativeScope, requestedScope)
				}

				queries = append(queries, hostQueries...)
				assignedScope = append(assignedScope, hostScopes...)
			}
		}
	}

	return queries, assignedScope, nil
}

// buildNodeQueries constructs API queries for node-specific metric data (Systems View).
// Similar to buildQueries but uses full node topology instead of job-allocated resources.
//
// The function handles:
//   - SubCluster topology resolution (either pre-loaded or per-node lookup)
//   - Full node hardware thread lists (not job-specific subsets)
//   - All accelerators on each node
//   - Metric configuration validation with subcluster filtering
//
// Returns queries and their corresponding assigned scopes.
func (ccms *CCMetricStore) buildNodeQueries(
	cluster string,
	subCluster string,
	nodes []string,
	metrics []string,
	scopes []schema.MetricScope,
	resolution int,
) ([]APIQuery, []schema.MetricScope, error) {
	// Initialize both slices together
	queries := make([]APIQuery, 0, len(metrics)*len(scopes)*len(nodes))
	assignedScope := make([]schema.MetricScope, 0, len(metrics)*len(scopes)*len(nodes))

	for _, metric := range metrics {
		remoteName := metric
		mc := archive.GetMetricConfig(cluster, metric)
		if mc == nil {
			cclog.Warnf("metric '%s' is not specified for cluster '%s'", metric, cluster)
			continue
		}

		// Skip if metric is removed for subcluster
		if mc.SubClusters != nil {
			isRemoved := false
			for _, scConfig := range mc.SubClusters {
				if scConfig.Name == subCluster && scConfig.Remove {
					isRemoved = true
					break
				}
			}
			if isRemoved {
				continue
			}
		}

		// Avoid duplicates...
		handledScopes := make([]schema.MetricScope, 0, 3)

	scopesLoop:
		for _, requestedScope := range scopes {
			nativeScope := mc.Scope

			scope := nativeScope.Max(requestedScope)
			for _, s := range handledScopes {
				if scope == s {
					continue scopesLoop
				}
			}
			handledScopes = append(handledScopes, scope)

			for _, hostname := range nodes {
				var topology *schema.Topology
				var err error

				// If no subCluster given, get it by node
				if subCluster == "" {
					topology, err = ccms.getTopologyByNode(cluster, hostname)
				} else {
					topology, err = ccms.getTopology(cluster, subCluster)
				}

				if err != nil {
					return nil, nil, err
				}

				// Always full node hwthread id list, no partial queries expected -> Use "topology.Node" directly where applicable
				// Always full accelerator id list, no partial queries expected -> Use "acceleratorIds" directly where applicable
				acceleratorIds := topology.GetAcceleratorIDs()

				// Moved check here if metric matches hardware specs
				if nativeScope == schema.MetricScopeAccelerator && len(acceleratorIds) == 0 {
					continue scopesLoop
				}

				nodeQueries, nodeScopes := buildScopeQueries(
					nativeScope, requestedScope,
					remoteName, hostname,
					topology, topology.Node, acceleratorIds,
					resolution,
				)

				if len(nodeQueries) == 0 && len(nodeScopes) == 0 {
					return nil, nil, fmt.Errorf("METRICDATA/EXTERNAL-CCMS > TODO: unhandled case: native-scope=%s, requested-scope=%s", nativeScope, requestedScope)
				}

				queries = append(queries, nodeQueries...)
				assignedScope = append(assignedScope, nodeScopes...)
			}
		}
	}

	return queries, assignedScope, nil
}

// buildScopeQueries generates API queries for a given scope transformation.
// It returns a slice of queries and corresponding assigned scopes.
// Some transformations (e.g., HWThread -> Core/Socket) may generate multiple queries.
func buildScopeQueries(
	nativeScope, requestedScope schema.MetricScope,
	metric, hostname string,
	topology *schema.Topology,
	hwthreads []int,
	accelerators []string,
	resolution int,
) ([]APIQuery, []schema.MetricScope) {
	scope := nativeScope.Max(requestedScope)
	queries := []APIQuery{}
	scopes := []schema.MetricScope{}

	hwthreadsStr := intToStringSlice(hwthreads)

	// Accelerator -> Accelerator (Use "accelerator" scope if requested scope is lower than node)
	if nativeScope == schema.MetricScopeAccelerator && scope.LT(schema.MetricScopeNode) {
		if scope != schema.MetricScopeAccelerator {
			// Expected Exception -> Continue -> Return Empty Slices
			return queries, scopes
		}

		queries = append(queries, APIQuery{
			Metric:     metric,
			Hostname:   hostname,
			Aggregate:  false,
			Type:       &acceleratorString,
			TypeIds:    accelerators,
			Resolution: resolution,
		})
		scopes = append(scopes, schema.MetricScopeAccelerator)
		return queries, scopes
	}

	// Accelerator -> Node
	if nativeScope == schema.MetricScopeAccelerator && scope == schema.MetricScopeNode {
		if len(accelerators) == 0 {
			// Expected Exception -> Continue -> Return Empty Slices
			return queries, scopes
		}

		queries = append(queries, APIQuery{
			Metric:     metric,
			Hostname:   hostname,
			Aggregate:  true,
			Type:       &acceleratorString,
			TypeIds:    accelerators,
			Resolution: resolution,
		})
		scopes = append(scopes, scope)
		return queries, scopes
	}

	// HWThread -> HWThread
	if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeHWThread {
		queries = append(queries, APIQuery{
			Metric:     metric,
			Hostname:   hostname,
			Aggregate:  false,
			Type:       &hwthreadString,
			TypeIds:    hwthreadsStr,
			Resolution: resolution,
		})
		scopes = append(scopes, scope)
		return queries, scopes
	}

	// HWThread -> Core
	if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeCore {
		cores, _ := topology.GetCoresFromHWThreads(hwthreads)
		for _, core := range cores {
			queries = append(queries, APIQuery{
				Metric:     metric,
				Hostname:   hostname,
				Aggregate:  true,
				Type:       &hwthreadString,
				TypeIds:    intToStringSlice(topology.Core[core]),
				Resolution: resolution,
			})
			scopes = append(scopes, scope)
		}
		return queries, scopes
	}

	// HWThread -> Socket
	if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeSocket {
		sockets, _ := topology.GetSocketsFromHWThreads(hwthreads)
		for _, socket := range sockets {
			queries = append(queries, APIQuery{
				Metric:     metric,
				Hostname:   hostname,
				Aggregate:  true,
				Type:       &hwthreadString,
				TypeIds:    intToStringSlice(topology.Socket[socket]),
				Resolution: resolution,
			})
			scopes = append(scopes, scope)
		}
		return queries, scopes
	}

	// HWThread -> Node
	if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeNode {
		queries = append(queries, APIQuery{
			Metric:     metric,
			Hostname:   hostname,
			Aggregate:  true,
			Type:       &hwthreadString,
			TypeIds:    hwthreadsStr,
			Resolution: resolution,
		})
		scopes = append(scopes, scope)
		return queries, scopes
	}

	// Core -> Core
	if nativeScope == schema.MetricScopeCore && scope == schema.MetricScopeCore {
		cores, _ := topology.GetCoresFromHWThreads(hwthreads)
		queries = append(queries, APIQuery{
			Metric:     metric,
			Hostname:   hostname,
			Aggregate:  false,
			Type:       &coreString,
			TypeIds:    intToStringSlice(cores),
			Resolution: resolution,
		})
		scopes = append(scopes, scope)
		return queries, scopes
	}

	// Core -> Socket
	if nativeScope == schema.MetricScopeCore && scope == schema.MetricScopeSocket {
		sockets, _ := topology.GetSocketsFromCores(hwthreads)
		for _, socket := range sockets {
			queries = append(queries, APIQuery{
				Metric:     metric,
				Hostname:   hostname,
				Aggregate:  true,
				Type:       &coreString,
				TypeIds:    intToStringSlice(topology.Socket[socket]),
				Resolution: resolution,
			})
			scopes = append(scopes, scope)
		}
		return queries, scopes
	}

	// Core -> Node
	if nativeScope == schema.MetricScopeCore && scope == schema.MetricScopeNode {
		cores, _ := topology.GetCoresFromHWThreads(hwthreads)
		queries = append(queries, APIQuery{
			Metric:     metric,
			Hostname:   hostname,
			Aggregate:  true,
			Type:       &coreString,
			TypeIds:    intToStringSlice(cores),
			Resolution: resolution,
		})
		scopes = append(scopes, scope)
		return queries, scopes
	}

	// MemoryDomain -> MemoryDomain
	if nativeScope == schema.MetricScopeMemoryDomain && scope == schema.MetricScopeMemoryDomain {
		memDomains, _ := topology.GetMemoryDomainsFromHWThreads(hwthreads)
		queries = append(queries, APIQuery{
			Metric:     metric,
			Hostname:   hostname,
			Aggregate:  false,
			Type:       &memoryDomainString,
			TypeIds:    intToStringSlice(memDomains),
			Resolution: resolution,
		})
		scopes = append(scopes, scope)
		return queries, scopes
	}

	// MemoryDomain -> Node
	if nativeScope == schema.MetricScopeMemoryDomain && scope == schema.MetricScopeNode {
		memDomains, _ := topology.GetMemoryDomainsFromHWThreads(hwthreads)
		queries = append(queries, APIQuery{
			Metric:     metric,
			Hostname:   hostname,
			Aggregate:  true,
			Type:       &memoryDomainString,
			TypeIds:    intToStringSlice(memDomains),
			Resolution: resolution,
		})
		scopes = append(scopes, scope)
		return queries, scopes
	}

	// MemoryDomain -> Socket
	if nativeScope == schema.MetricScopeMemoryDomain && scope == schema.MetricScopeSocket {
		memDomains, _ := topology.GetMemoryDomainsFromHWThreads(hwthreads)
		socketToDomains, err := topology.GetMemoryDomainsBySocket(memDomains)
		if err != nil {
			cclog.Errorf("Error mapping memory domains to sockets, return unchanged: %v", err)
			// Rare Error Case -> Still Continue -> Return Empty Slices
			return queries, scopes
		}

		// Create a query for each socket
		for _, domains := range socketToDomains {
			queries = append(queries, APIQuery{
				Metric:     metric,
				Hostname:   hostname,
				Aggregate:  true,
				Type:       &memoryDomainString,
				TypeIds:    intToStringSlice(domains),
				Resolution: resolution,
			})
			// Add scope for each query, not just once
			scopes = append(scopes, scope)
		}
		return queries, scopes
	}

	// Socket -> Socket
	if nativeScope == schema.MetricScopeSocket && scope == schema.MetricScopeSocket {
		sockets, _ := topology.GetSocketsFromHWThreads(hwthreads)
		queries = append(queries, APIQuery{
			Metric:     metric,
			Hostname:   hostname,
			Aggregate:  false,
			Type:       &socketString,
			TypeIds:    intToStringSlice(sockets),
			Resolution: resolution,
		})
		scopes = append(scopes, scope)
		return queries, scopes
	}

	// Socket -> Node
	if nativeScope == schema.MetricScopeSocket && scope == schema.MetricScopeNode {
		sockets, _ := topology.GetSocketsFromHWThreads(hwthreads)
		queries = append(queries, APIQuery{
			Metric:     metric,
			Hostname:   hostname,
			Aggregate:  true,
			Type:       &socketString,
			TypeIds:    intToStringSlice(sockets),
			Resolution: resolution,
		})
		scopes = append(scopes, scope)
		return queries, scopes
	}

	// Node -> Node
	if nativeScope == schema.MetricScopeNode && scope == schema.MetricScopeNode {
		queries = append(queries, APIQuery{
			Metric:     metric,
			Hostname:   hostname,
			Resolution: resolution,
		})
		scopes = append(scopes, scope)
		return queries, scopes
	}

	// Unhandled Case -> Error -> Return nils
	return nil, nil
}

// intToStringSlice converts a slice of integers to a slice of strings.
// Used to convert hardware IDs (core IDs, socket IDs, etc.) to the string format required by the API.
func intToStringSlice(is []int) []string {
	ss := make([]string, len(is))
	for i, x := range is {
		ss[i] = strconv.Itoa(x)
	}
	return ss
}


<persisted-output>
Output too large (50.6KB). Full output saved to: /Users/jan/.claude/projects/-Users-jan-prg-ClusterCockpit-cc-backend/5de2a23c-09cd-4d6c-a97c-fab8657d094c/tool-results/toolu_013RKvera1zE5WTk8Cut1cxk.txt

Preview (first 2KB):
     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// This file implements high-level query functions for loading job metric data
     7→// with automatic scope transformation and aggregation.
     8→//
     9→// Key Concepts:
    10→//
    11→// Metric Scopes: Metrics are collected at different granularities (native scope):
    12→//   - HWThread: Per hardware thread
    13→//   - Core: Per CPU core
    14→//   - Socket: Per CPU socket
    15→//   - MemoryDomain: Per memory domain (NUMA)
    16→//   - Accelerator: Per GPU/accelerator
    17→//   - Node: Per compute node
    18→//
    19→// Scope Transformation: The buildQueries functions transform between native scope
    20→// and requested scope by:
    21→//   - Aggregating finer-grained data (e.g., HWThread → Core → Socket → Node)
    22→//   - Rejecting requests for finer granularity than available
    23→//   - Handling special cases (e.g., Accelerator metrics)
    24→//
    25→// Query Building: Constructs APIQuery structures with proper selectors (Type, TypeIds)
    26→// based on cluster topology and job resources.
    27→package metricstore
    28→
    29→import (
    30→	"context"
    31→	"fmt"
    32→	"strconv"
    33→	"strings"
    34→	"time"
    35→
    36→	"github.com/ClusterCockpit/cc-backend/pkg/archive"
    37→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    38→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    39→)
    40→
    41→type InternalMetricStore struct{}
    42→
    43→var MetricStoreHandle *InternalMetricStore
    44→
    45→// HealthCheck delegates to the internal MemoryStore's HealthCheck.
    46→func (ccms *InternalMetricStore) HealthCheck(cluster string,
    47→	nodes []string, metrics []string,
    48→) (map[string]HealthCheckResult, error) {
...
</persisted-output>

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// This file implements high-level query functions for loading job metric data
// with automatic scope transformation and aggregation.
//
// Key Concepts:
//
// Metric Scopes: Metrics are collected at different granularities (native scope):
//   - HWThread: Per hardware thread
//   - Core: Per CPU core
//   - Socket: Per CPU socket
//   - MemoryDomain: Per memory domain (NUMA)
//   - Accelerator: Per GPU/accelerator
//   - Node: Per compute node
//
// Scope Transformation: The buildQueries functions transform between native scope
// and requested scope by:
//   - Aggregating finer-grained data (e.g., HWThread → Core → Socket → Node)
//   - Rejecting requests for finer granularity than available
//   - Handling special cases (e.g., Accelerator metrics)
//
// Query Building: Constructs APIQuery structures with proper selectors (Type, TypeIds)
// based on cluster topology and job resources.
package metricstore

import (
	"context"
	"fmt"
	"strconv"
	"strings"
	"time"

	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

type InternalMetricStore struct{}

var MetricStoreHandle *InternalMetricStore

// HealthCheck delegates to the internal MemoryStore's HealthCheck.
func (ccms *InternalMetricStore) HealthCheck(cluster string,
	nodes []string, metrics []string,
) (map[string]HealthCheckResult, error) {
	return GetMemoryStore().HealthCheck(cluster, nodes, metrics)
}

// TestLoadDataCallback allows tests to override LoadData behavior for testing purposes.
// When set to a non-nil function, LoadData will call this function instead of the default implementation.
var TestLoadDataCallback func(job *schema.Job, metrics []string, scopes []schema.MetricScope, ctx context.Context, resolution int) (schema.JobData, error)

// LoadData loads metric data for a specific job with automatic scope transformation.
//
// This is the primary function for retrieving job metric data. It handles:
//   - Building queries with scope transformation via buildQueries
//   - Fetching data from the metric store
//   - Organizing results by metric and scope
//   - Converting NaN statistics to 0 for JSON compatibility
//   - Partial error handling (returns data for successful queries even if some fail)
//
// Parameters:
//   - job: Job metadata including cluster, resources, and time range
//   - metrics: List of metric names to load
//   - scopes: Requested metric scopes (will be transformed to match native scopes)
//   - ctx: Context for cancellation (currently unused but reserved for future use)
//   - resolution: Data resolution in seconds (0 for native resolution)
//
// Returns:
//   - JobData: Map of metric → scope → JobMetric with time-series data and statistics
//   - Error: Returns error if query building or fetching fails, or partial error listing failed hosts
//
// Example:
//
//	jobData, err := LoadData(job, []string{"cpu_load", "mem_used"}, []schema.MetricScope{schema.MetricScopeNode}, ctx, 60)
func (ccms *InternalMetricStore) LoadData(
	job *schema.Job,
	metrics []string,
	scopes []schema.MetricScope,
	ctx context.Context,
	resolution int,
) (schema.JobData, error) {
	if TestLoadDataCallback != nil {
		return TestLoadDataCallback(job, metrics, scopes, ctx, resolution)
	}

	queries, assignedScope, err := buildQueries(job, metrics, scopes, int64(resolution))
	if err != nil {
		cclog.Errorf("Error while building queries for jobId %d, Metrics %v, Scopes %v: %s", job.JobID, metrics, scopes, err.Error())
		return nil, err
	}

	// Verify assignment is correct - log any inconsistencies for debugging
	if len(queries) != len(assignedScope) {
		cclog.Errorf("Critical error: queries and assignedScope have different lengths after buildQueries: %d vs %d",
			len(queries), len(assignedScope))
	}

	req := APIQueryRequest{
		Cluster:   job.Cluster,
		From:      job.StartTime,
		To:        job.StartTime + int64(job.Duration),
		Queries:   queries,
		WithStats: true,
		WithData:  true,
	}

	resBody, err := FetchData(req)
	if err != nil {
		cclog.Errorf("Error while fetching data : %s", err.Error())
		return nil, err
	}

	var errors []string
	jobData := make(schema.JobData)

	// Add safety check for potential index out of range errors
	if len(resBody.Results) != len(req.Queries) || len(assignedScope) != len(req.Queries) {
		cclog.Warnf("Mismatch in query results count: queries=%d, results=%d, assignedScope=%d",
			len(req.Queries), len(resBody.Results), len(assignedScope))
		if len(resBody.Results) > len(req.Queries) {
			resBody.Results = resBody.Results[:len(req.Queries)]
		}
		if len(assignedScope) > len(req.Queries) {
			assignedScope = assignedScope[:len(req.Queries)]
		}
	}

	for i, row := range resBody.Results {
		// Safety check to prevent index out of range errors
		if i >= len(req.Queries) || i >= len(assignedScope) {
			cclog.Warnf("Index out of range prevented: i=%d, queries=%d, assignedScope=%d",
				i, len(req.Queries), len(assignedScope))
			continue
		}

		query := req.Queries[i]
		metric := query.Metric
		scope := assignedScope[i]
		mc := archive.GetMetricConfig(job.Cluster, metric)

		if mc == nil {
			cclog.Warnf("Metric config not found for %s on cluster %s", metric, job.Cluster)
			continue
		}

		if _, ok := jobData[metric]; !ok {
			jobData[metric] = make(map[schema.MetricScope]*schema.JobMetric)
		}

		res := mc.Timestep
		if len(row) > 0 {
			res = int(row[0].Resolution)
		}

		jobMetric, ok := jobData[metric][scope]
		if !ok {
			jobMetric = &schema.JobMetric{
				Unit:     mc.Unit,
				Timestep: res,
				Series:   make([]schema.Series, 0),
			}
			jobData[metric][scope] = jobMetric
		}

		for ndx, res := range row {
			if res.Error != nil {
				/* Build list for "partial errors", if any */
				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
				continue
			}

			id := (*string)(nil)
			if query.Type != nil {
				// Check if ndx is within the bounds of TypeIds slice
				if ndx < len(query.TypeIds) {
					id = new(string)
					*id = query.TypeIds[ndx]
				} else {
					// Log the error but continue processing
					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
						ndx, len(query.TypeIds), query.Metric, query.Hostname)
				}
			}

			sanitizeStats(&res)

			jobMetric.Series = append(jobMetric.Series, schema.Series{
				Hostname: query.Hostname,
				ID:       id,
				Statistics: schema.MetricStatistics{
					Avg: float64(res.Avg),
					Min: float64(res.Min),
					Max: float64(res.Max),
				},
				Data: res.Data,
			})
		}

		// So that one can later check len(jobData):
		if len(jobMetric.Series) == 0 {
			delete(jobData[metric], scope)
			if len(jobData[metric]) == 0 {
				delete(jobData, metric)
			}
		}
	}

	if len(errors) != 0 {
		/* Returns list for "partial errors" */
		return jobData, fmt.Errorf("METRICDATA/INTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}
	return jobData, nil
}

// Pre-converted scope strings avoid repeated string(MetricScope) allocations during
// query construction. These are used in APIQuery.Type field throughout buildQueries
// and buildNodeQueries functions. Converting once at package initialization improves
// performance for high-volume query building.
var (
	hwthreadString     = string(schema.MetricScopeHWThread)
	coreString         = string(schema.MetricScopeCore)
	memoryDomainString = string(schema.MetricScopeMemoryDomain)
	socketString       = string(schema.MetricScopeSocket)
	acceleratorString  = string(schema.MetricScopeAccelerator)
)

// buildQueries constructs APIQuery structures with automatic scope transformation for a job.
//
// This function implements the core scope transformation logic, handling all combinations of
// native metric scopes and requested scopes. It uses the cluster topology to determine which
// hardware IDs to include in each query.
//
// Scope Transformation Rules:
//   - If native scope >= requested scope: Aggregates data (Aggregate=true in APIQuery)
//   - If native scope < requested scope: Returns error (cannot increase granularity)
//   - Special handling for Accelerator scope (independent of CPU hierarchy)
//
// The function generates one or more APIQuery per (metric, scope, host) combination:
//   - For non-aggregated queries: One query with all relevant IDs
//   - For aggregated queries: May generate multiple queries (e.g., one per socket/core)
//
// Parameters:
//   - job: Job metadata including cluster, subcluster, and resource allocation
//   - metrics: List of metrics to query
//   - scopes: Requested scopes for each metric
//   - resolution: Data resolution in seconds
//
// Returns:
//   - []APIQuery: List of queries to execute
//   - []schema.MetricScope: Assigned scope for each query (after transformation)
//   - error: Returns error if topology lookup fails or unhandled scope combination encountered
func buildQueries(
	job *schema.Job,
	metrics []string,
	scopes []schema.MetricScope,
	resolution int64,
) ([]APIQuery, []schema.MetricScope, error) {
	if len(job.Resources) == 0 {
		return nil, nil, fmt.Errorf("METRICDATA/INTERNAL-CCMS > no resources allocated for job %d", job.JobID)
	}

	queries := make([]APIQuery, 0, len(metrics)*len(scopes)*len(job.Resources))
	assignedScope := []schema.MetricScope{}

	subcluster, scerr := archive.GetSubCluster(job.Cluster, job.SubCluster)
	if scerr != nil {
		return nil, nil, scerr
	}
	topology := subcluster.Topology

	for _, metric := range metrics {
		mc := archive.GetMetricConfig(job.Cluster, metric)
		if mc == nil {
			cclog.Warnf("metric '%s' is not specified for cluster '%s'", metric, job.Cluster)
			continue
		}

		// Skip if metric is removed for subcluster
		if len(mc.SubClusters) != 0 {
			isRemoved := false
			for _, scConfig := range mc.SubClusters {
				if scConfig.Name == job.SubCluster && scConfig.Remove {
					isRemoved = true
					break
				}
			}
			if isRemoved {
				continue
			}
		}

		// Avoid duplicates using map for O(1) lookup
		handledScopes := make(map[schema.MetricScope]bool, 3)

		for _, requestedScope := range scopes {
			nativeScope := mc.Scope
			if nativeScope == schema.MetricScopeAccelerator && job.NumAcc == 0 {
				continue
			}

			scope := nativeScope.Max(requestedScope)
			if handledScopes[scope] {
				continue
			}
			handledScopes[scope] = true

			for _, host := range job.Resources {
				hwthreads := host.HWThreads
				if hwthreads == nil {
					hwthreads = topology.Node
				}

				// Accelerator -> Accelerator (Use "accelerator" scope if requested scope is lower than node)
				if nativeScope == schema.MetricScopeAccelerator && scope.LT(schema.MetricScopeNode) {
					if scope != schema.MetricScopeAccelerator {
						// Skip all other catched cases
						continue
					}

					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   host.Hostname,
						Aggregate:  false,
						Type:       &acceleratorString,
						TypeIds:    host.Accelerators,
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, schema.MetricScopeAccelerator)
					continue
				}

				// Accelerator -> Node
				if nativeScope == schema.MetricScopeAccelerator && scope == schema.MetricScopeNode {
					if len(host.Accelerators) == 0 {
						continue
					}

					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   host.Hostname,
						Aggregate:  true,
						Type:       &acceleratorString,
						TypeIds:    host.Accelerators,
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// HWThread -> HWThread
				if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeHWThread {
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   host.Hostname,
						Aggregate:  false,
						Type:       &hwthreadString,
						TypeIds:    intToStringSlice(hwthreads),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// HWThread -> Core
				if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeCore {
					cores, _ := topology.GetCoresFromHWThreads(hwthreads)
					for _, core := range cores {
						queries = append(queries, APIQuery{
							Metric:     metric,
							Hostname:   host.Hostname,
							Aggregate:  true,
							Type:       &hwthreadString,
							TypeIds:    intToStringSlice(topology.Core[core]),
							Resolution: resolution,
						})
						assignedScope = append(assignedScope, scope)
					}
					continue
				}

				// HWThread -> Socket
				if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeSocket {
					sockets, _ := topology.GetSocketsFromHWThreads(hwthreads)
					for _, socket := range sockets {
						queries = append(queries, APIQuery{
							Metric:     metric,
							Hostname:   host.Hostname,
							Aggregate:  true,
							Type:       &hwthreadString,
							TypeIds:    intToStringSlice(topology.Socket[socket]),
							Resolution: resolution,
						})
						assignedScope = append(assignedScope, scope)
					}
					continue
				}

				// HWThread -> Node
				if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeNode {
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   host.Hostname,
						Aggregate:  true,
						Type:       &hwthreadString,
						TypeIds:    intToStringSlice(hwthreads),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// Core -> Core
				if nativeScope == schema.MetricScopeCore && scope == schema.MetricScopeCore {
					cores, _ := topology.GetCoresFromHWThreads(hwthreads)
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   host.Hostname,
						Aggregate:  false,
						Type:       &coreString,
						TypeIds:    intToStringSlice(cores),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// Core -> Socket
				if nativeScope == schema.MetricScopeCore && scope == schema.MetricScopeSocket {
					sockets, _ := topology.GetSocketsFromCores(hwthreads)
					for _, socket := range sockets {
						queries = append(queries, APIQuery{
							Metric:     metric,
							Hostname:   host.Hostname,
							Aggregate:  true,
							Type:       &coreString,
							TypeIds:    intToStringSlice(topology.Socket[socket]),
							Resolution: resolution,
						})
						assignedScope = append(assignedScope, scope)
					}
					continue
				}

				// Core -> Node
				if nativeScope == schema.MetricScopeCore && scope == schema.MetricScopeNode {
					cores, _ := topology.GetCoresFromHWThreads(hwthreads)
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   host.Hostname,
						Aggregate:  true,
						Type:       &coreString,
						TypeIds:    intToStringSlice(cores),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// MemoryDomain -> MemoryDomain
				if nativeScope == schema.MetricScopeMemoryDomain && scope == schema.MetricScopeMemoryDomain {
					sockets, _ := topology.GetMemoryDomainsFromHWThreads(hwthreads)
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   host.Hostname,
						Aggregate:  false,
						Type:       &memoryDomainString,
						TypeIds:    intToStringSlice(sockets),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// MemoryDomain -> Node
				if nativeScope == schema.MetricScopeMemoryDomain && scope == schema.MetricScopeNode {
					sockets, _ := topology.GetMemoryDomainsFromHWThreads(hwthreads)
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   host.Hostname,
						Aggregate:  true,
						Type:       &memoryDomainString,
						TypeIds:    intToStringSlice(sockets),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// Socket -> Socket
				if nativeScope == schema.MetricScopeSocket && scope == schema.MetricScopeSocket {
					sockets, _ := topology.GetSocketsFromHWThreads(hwthreads)
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   host.Hostname,
						Aggregate:  false,
						Type:       &socketString,
						TypeIds:    intToStringSlice(sockets),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// Socket -> Node
				if nativeScope == schema.MetricScopeSocket && scope == schema.MetricScopeNode {
					sockets, _ := topology.GetSocketsFromHWThreads(hwthreads)
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   host.Hostname,
						Aggregate:  true,
						Type:       &socketString,
						TypeIds:    intToStringSlice(sockets),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// Node -> Node
				if nativeScope == schema.MetricScopeNode && scope == schema.MetricScopeNode {
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   host.Hostname,
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				return nil, nil, fmt.Errorf("METRICDATA/INTERNAL-CCMS > TODO: unhandled case: native-scope=%s, requested-scope=%s", nativeScope, requestedScope)
			}
		}
	}

	return queries, assignedScope, nil
}

// LoadStats loads only metric statistics (avg/min/max) for a job at node scope.
//
// This is an optimized version of LoadData that fetches only statistics without
// time-series data, reducing bandwidth and memory usage. Always queries at node scope.
//
// Parameters:
//   - job: Job metadata
//   - metrics: List of metric names
//   - ctx: Context (currently unused)
//
// Returns:
//   - Map of metric → hostname → statistics
//   - Error on query building or fetching failure
func (ccms *InternalMetricStore) LoadStats(
	job *schema.Job,
	metrics []string,
	ctx context.Context,
) (map[string]map[string]schema.MetricStatistics, error) {
	// TODO(#166): Add scope parameter for analysis view accelerator normalization
	queries, _, err := buildQueries(job, metrics, []schema.MetricScope{schema.MetricScopeNode}, 0)
	if err != nil {
		cclog.Errorf("Error while building queries for jobId %d, Metrics %v: %s", job.JobID, metrics, err.Error())
		return nil, err
	}

	req := APIQueryRequest{
		Cluster:   job.Cluster,
		From:      job.StartTime,
		To:        job.StartTime + int64(job.Duration),
		Queries:   queries,
		WithStats: true,
		WithData:  false,
	}

	resBody, err := FetchData(req)
	if err != nil {
		cclog.Errorf("Error while fetching data : %s", err.Error())
		return nil, err
	}

	stats := make(map[string]map[string]schema.MetricStatistics, len(metrics))
	for i, res := range resBody.Results {
		if len(res) == 0 {
			// No Data Found For Metric, Logged in FetchData to Warn
			continue
		}
		query := req.Queries[i]
		metric := query.Metric
		data := res[0]
		if data.Error != nil {
			cclog.Warnf("fetching %s for node %s failed: %s", metric, query.Hostname, *data.Error)
			continue
		}

		metricdata, ok := stats[metric]
		if !ok {
			metricdata = make(map[string]schema.MetricStatistics, job.NumNodes)
			stats[metric] = metricdata
		}

		if data.Avg.IsNaN() || data.Min.IsNaN() || data.Max.IsNaN() {
			cclog.Warnf("fetching %s for node %s failed: one of avg/min/max is NaN", metric, query.Hostname)
			continue
		}

		metricdata[query.Hostname] = schema.MetricStatistics{
			Avg: float64(data.Avg),
			Min: float64(data.Min),
			Max: float64(data.Max),
		}
	}

	return stats, nil
}

// LoadScopedStats loads metric statistics for a job with scope-aware grouping.
//
// Similar to LoadStats but supports multiple scopes and returns statistics grouped
// by scope with hardware IDs (e.g., per-core, per-socket statistics).
//
// Parameters:
//   - job: Job metadata
//   - metrics: List of metric names
//   - scopes: Requested metric scopes
//   - ctx: Context (currently unused)
//
// Returns:
//   - ScopedJobStats: Map of metric → scope → []ScopedStats (with hostname and ID)
//   - Error or partial error listing failed queries
func (ccms *InternalMetricStore) LoadScopedStats(
	job *schema.Job,
	metrics []string,
	scopes []schema.MetricScope,
	ctx context.Context,
) (schema.ScopedJobStats, error) {
	queries, assignedScope, err := buildQueries(job, metrics, scopes, 0)
	if err != nil {
		cclog.Errorf("Error while building queries for jobId %d, Metrics %v, Scopes %v: %s", job.JobID, metrics, scopes, err.Error())
		return nil, err
	}

	req := APIQueryRequest{
		Cluster:   job.Cluster,
		From:      job.StartTime,
		To:        job.StartTime + int64(job.Duration),
		Queries:   queries,
		WithStats: true,
		WithData:  false,
	}

	resBody, err := FetchData(req)
	if err != nil {
		cclog.Errorf("Error while fetching data : %s", err.Error())
		return nil, err
	}

	var errors []string
	scopedJobStats := make(schema.ScopedJobStats)

	for i, row := range resBody.Results {
		if len(row) == 0 {
			// No Data Found For Metric, Logged in FetchData to Warn
			continue
		}
		query := req.Queries[i]
		metric := query.Metric
		scope := assignedScope[i]

		if _, ok := scopedJobStats[metric]; !ok {
			scopedJobStats[metric] = make(map[schema.MetricScope][]*schema.ScopedStats)
		}

		if _, ok := scopedJobStats[metric][scope]; !ok {
			scopedJobStats[metric][scope] = make([]*schema.ScopedStats, 0)
		}

		for ndx, res := range row {
			if res.Error != nil {
				/* Build list for "partial errors", if any */
				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
				continue
			}

			id := (*string)(nil)
			if query.Type != nil {
				// Check if ndx is within the bounds of TypeIds slice
				if ndx < len(query.TypeIds) {
					id = new(string)
					*id = query.TypeIds[ndx]
				} else {
					// Log the error but continue processing
					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
						ndx, len(query.TypeIds), query.Metric, query.Hostname)
				}
			}

			sanitizeStats(&res)

			scopedJobStats[metric][scope] = append(scopedJobStats[metric][scope], &schema.ScopedStats{
				Hostname: query.Hostname,
				ID:       id,
				Data: &schema.MetricStatistics{
					Avg: float64(res.Avg),
					Min: float64(res.Min),
					Max: float64(res.Max),
				},
			})
		}

		// So that one can later check len(scopedJobStats[metric][scope]): Remove from map if empty
		if len(scopedJobStats[metric][scope]) == 0 {
			delete(scopedJobStats[metric], scope)
			if len(scopedJobStats[metric]) == 0 {
				delete(scopedJobStats, metric)
			}
		}
	}

	if len(errors) != 0 {
		/* Returns list for "partial errors" */
		return scopedJobStats, fmt.Errorf("METRICDATA/INTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}
	return scopedJobStats, nil
}

// LoadNodeData loads metric data for specific nodes in a cluster over a time range.
//
// Unlike LoadData which operates on job resources, this function queries arbitrary nodes
// directly. Useful for system monitoring and node status views.
//
// Parameters:
//   - cluster: Cluster name
//   - metrics: List of metric names
//   - nodes: List of node hostnames (nil = all nodes in cluster via ForAllNodes)
//   - scopes: Requested metric scopes (currently unused - always node scope)
//   - from, to: Time range
//   - ctx: Context (currently unused)
//
// Returns:
//   - Map of hostname → metric → []JobMetric
//   - Error or partial error listing failed queries
func (ccms *InternalMetricStore) LoadNodeData(
	cluster string,
	metrics, nodes []string,
	scopes []schema.MetricScope,
	from, to time.Time,
	ctx context.Context,
) (map[string]map[string][]*schema.JobMetric, error) {
	req := APIQueryRequest{
		Cluster:   cluster,
		From:      from.Unix(),
		To:        to.Unix(),
		WithStats: true,
		WithData:  true,
	}

	if nodes == nil {
		req.ForAllNodes = append(req.ForAllNodes, metrics...)
	} else {
		for _, node := range nodes {
			for _, metric := range metrics {
				req.Queries = append(req.Queries, APIQuery{
					Hostname:   node,
					Metric:     metric,
					Resolution: 0, // Default for Node Queries: Will return metric $Timestep Resolution
				})
			}
		}
	}

	resBody, err := FetchData(req)
	if err != nil {
		cclog.Errorf("Error while fetching data : %s", err.Error())
		return nil, err
	}

	var errors []string
	data := make(map[string]map[string][]*schema.JobMetric)
	for i, res := range resBody.Results {
		if len(res) == 0 {
			// No Data Found For Metric, Logged in FetchData to Warn
			continue
		}

		var query APIQuery
		if resBody.Queries != nil {
			query = resBody.Queries[i]
		} else {
			query = req.Queries[i]
		}

		metric := query.Metric
		qdata := res[0]
		if qdata.Error != nil {
			errors = append(errors, fmt.Sprintf("fetching %s for node %s failed: %s", metric, query.Hostname, *qdata.Error))
		}

		sanitizeStats(&qdata)

		hostdata, ok := data[query.Hostname]
		if !ok {
			hostdata = make(map[string][]*schema.JobMetric)
			data[query.Hostname] = hostdata
		}

		mc := archive.GetMetricConfig(cluster, metric)
		hostdata[metric] = append(hostdata[metric], &schema.JobMetric{
			Unit:     mc.Unit,
			Timestep: mc.Timestep,
			Series: []schema.Series{
				{
					Hostname: query.Hostname,
					Data:     qdata.Data,
					Statistics: schema.MetricStatistics{
						Avg: float64(qdata.Avg),
						Min: float64(qdata.Min),
						Max: float64(qdata.Max),
					},
				},
			},
		})
	}

	if len(errors) != 0 {
		/* Returns list of "partial errors" */
		return data, fmt.Errorf("METRICDATA/INTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}

	return data, nil
}

// LoadNodeListData loads metric data for a list of nodes with full scope transformation support.
//
// This is the most flexible node data loading function, supporting arbitrary scopes and
// resolution. Uses buildNodeQueries for proper scope transformation based on topology.
//
// Parameters:
//   - cluster: Cluster name
//   - subCluster: SubCluster name (empty string to infer from node names)
//   - nodes: List of node hostnames
//   - metrics: List of metric names
//   - scopes: Requested metric scopes
//   - resolution: Data resolution in seconds
//   - from, to: Time range
//   - ctx: Context (currently unused)
//
// Returns:
//   - Map of hostname → JobData (metric → scope → JobMetric)
//   - Error or partial error listing failed queries
func (ccms *InternalMetricStore) LoadNodeListData(
	cluster, subCluster string,
	nodes []string,
	metrics []string,
	scopes []schema.MetricScope,
	resolution int,
	from, to time.Time,
	ctx context.Context,
) (map[string]schema.JobData, error) {
	// Note: Order of node data is not guaranteed after this point
	queries, assignedScope, err := buildNodeQueries(cluster, subCluster, nodes, metrics, scopes, int64(resolution))
	if err != nil {
		cclog.Errorf("Error while building node queries for Cluster %s, SubCLuster %s, Metrics %v, Scopes %v: %s", cluster, subCluster, metrics, scopes, err.Error())
		return nil, err
	}

	// Verify assignment is correct - log any inconsistencies for debugging
	if len(queries) != len(assignedScope) {
		cclog.Errorf("Critical error: queries and assignedScope have different lengths after buildNodeQueries: %d vs %d",
			len(queries), len(assignedScope))
	}

	req := APIQueryRequest{
		Cluster:   cluster,
		Queries:   queries,
		From:      from.Unix(),
		To:        to.Unix(),
		WithStats: true,
		WithData:  true,
	}

	resBody, err := FetchData(req)
	if err != nil {
		cclog.Errorf("Error while fetching data : %s", err.Error())
		return nil, err
	}

	var errors []string
	data := make(map[string]schema.JobData)

	// Add safety check for index out of range issues
	if len(resBody.Results) != len(req.Queries) || len(assignedScope) != len(req.Queries) {
		cclog.Warnf("Mismatch in query results count: queries=%d, results=%d, assignedScope=%d",
			len(req.Queries), len(resBody.Results), len(assignedScope))
		if len(resBody.Results) > len(req.Queries) {
			resBody.Results = resBody.Results[:len(req.Queries)]
		}
		if len(assignedScope) > len(req.Queries) {
			assignedScope = assignedScope[:len(req.Queries)]
		}
	}

	for i, row := range resBody.Results {
		// Safety check to prevent index out of range errors
		if i >= len(req.Queries) || i >= len(assignedScope) {
			cclog.Warnf("Index out of range prevented: i=%d, queries=%d, assignedScope=%d",
				i, len(req.Queries), len(assignedScope))
			continue
		}

		var query APIQuery
		if resBody.Queries != nil {
			if i < len(resBody.Queries) {
				query = resBody.Queries[i]
			} else {
				cclog.Warnf("Index out of range prevented for resBody.Queries: i=%d, len=%d",
					i, len(resBody.Queries))
				continue
			}
		} else {
			query = req.Queries[i]
		}

		metric := query.Metric
		scope := assignedScope[i]
		mc := archive.GetMetricConfig(cluster, metric)
		if mc == nil {
			cclog.Warnf("Metric config not found for %s on cluster %s", metric, cluster)
			continue
		}

		res := mc.Timestep
		if len(row) > 0 {
			res = int(row[0].Resolution)
		}

		// Init Nested Map Data Structures If Not Found
		hostData, ok := data[query.Hostname]
		if !ok {
			hostData = make(schema.JobData)
			data[query.Hostname] = hostData
		}

		metricData, ok := hostData[metric]
		if !ok {
			metricData = make(map[schema.MetricScope]*schema.JobMetric)
			data[query.Hostname][metric] = metricData
		}

		scopeData, ok := metricData[scope]
		if !ok {
			scopeData = &schema.JobMetric{
				Unit:     mc.Unit,
				Timestep: res,
				Series:   make([]schema.Series, 0),
			}
			data[query.Hostname][metric][scope] = scopeData
		}

		for ndx, res := range row {
			if res.Error != nil {
				/* Build list for "partial errors", if any */
				errors = append(errors, fmt.Sprintf("failed to fetch '%s' from host '%s': %s", query.Metric, query.Hostname, *res.Error))
				continue
			}

			id := (*string)(nil)
			if query.Type != nil {
				// Check if ndx is within the bounds of TypeIds slice
				if ndx < len(query.TypeIds) {
					id = new(string)
					*id = query.TypeIds[ndx]
				} else {
					// Log the error but continue processing
					cclog.Warnf("TypeIds index out of range: %d with length %d for metric %s on host %s",
						ndx, len(query.TypeIds), query.Metric, query.Hostname)
				}
			}

			sanitizeStats(&res)

			scopeData.Series = append(scopeData.Series, schema.Series{
				Hostname: query.Hostname,
				ID:       id,
				Statistics: schema.MetricStatistics{
					Avg: float64(res.Avg),
					Min: float64(res.Min),
					Max: float64(res.Max),
				},
				Data: res.Data,
			})
		}
	}

	if len(errors) != 0 {
		/* Returns list of "partial errors" */
		return data, fmt.Errorf("METRICDATA/INTERNAL-CCMS > Errors: %s", strings.Join(errors, ", "))
	}

	return data, nil
}

// buildNodeQueries constructs APIQuery structures for node-based queries with scope transformation.
//
// Similar to buildQueries but operates on node lists rather than job resources.
// Supports dynamic subcluster lookup when subCluster parameter is empty.
//
// Parameters:
//   - cluster: Cluster name
//   - subCluster: SubCluster name (empty = infer from node hostnames)
//   - nodes: List of node hostnames
//   - metrics: List of metric names
//   - scopes: Requested metric scopes
//   - resolution: Data resolution in seconds
//
// Returns:
//   - []APIQuery: List of queries to execute
//   - []schema.MetricScope: Assigned scope for each query
//   - error: Returns error if topology lookup fails or unhandled scope combination
func buildNodeQueries(
	cluster string,
	subCluster string,
	nodes []string,
	metrics []string,
	scopes []schema.MetricScope,
	resolution int64,
) ([]APIQuery, []schema.MetricScope, error) {
	queries := make([]APIQuery, 0, len(metrics)*len(scopes)*len(nodes))
	assignedScope := []schema.MetricScope{}

	// Get Topol before loop if subCluster given
	var subClusterTopol *schema.SubCluster
	var scterr error
	if subCluster != "" {
		subClusterTopol, scterr = archive.GetSubCluster(cluster, subCluster)
		if scterr != nil {
			cclog.Errorf("could not load cluster %s subCluster %s topology: %s", cluster, subCluster, scterr.Error())
			return nil, nil, scterr
		}
	}

	for _, metric := range metrics {
		mc := archive.GetMetricConfig(cluster, metric)
		if mc == nil {
			cclog.Warnf("metric '%s' is not specified for cluster '%s'", metric, cluster)
			continue
		}

		// Skip if metric is removed for subcluster
		if mc.SubClusters != nil {
			isRemoved := false
			for _, scConfig := range mc.SubClusters {
				if scConfig.Name == subCluster && scConfig.Remove {
					isRemoved = true
					break
				}
			}
			if isRemoved {
				continue
			}
		}

		// Avoid duplicates using map for O(1) lookup
		handledScopes := make(map[schema.MetricScope]bool, 3)

		for _, requestedScope := range scopes {
			nativeScope := mc.Scope

			scope := nativeScope.Max(requestedScope)
			if handledScopes[scope] {
				continue
			}
			handledScopes[scope] = true

			for _, hostname := range nodes {

				// If no subCluster given, get it by node
				if subCluster == "" {
					subClusterName, scnerr := archive.GetSubClusterByNode(cluster, hostname)
					if scnerr != nil {
						return nil, nil, scnerr
					}
					subClusterTopol, scterr = archive.GetSubCluster(cluster, subClusterName)
					if scterr != nil {
						return nil, nil, scterr
					}
				}

				// Always full node hwthread id list, no partial queries expected -> Use "topology.Node" directly where applicable
				// Always full accelerator id list, no partial queries expected -> Use "acceleratorIds" directly where applicable
				topology := subClusterTopol.Topology
				acceleratorIds := topology.GetAcceleratorIDs()

				// Moved check here if metric matches hardware specs
				if nativeScope == schema.MetricScopeAccelerator && len(acceleratorIds) == 0 {
					continue
				}

				// Accelerator -> Accelerator (Use "accelerator" scope if requested scope is lower than node)
				if nativeScope == schema.MetricScopeAccelerator && scope.LT(schema.MetricScopeNode) {
					if scope != schema.MetricScopeAccelerator {
						// Skip all other catched cases
						continue
					}

					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   hostname,
						Aggregate:  false,
						Type:       &acceleratorString,
						TypeIds:    acceleratorIds,
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, schema.MetricScopeAccelerator)
					continue
				}

				// Accelerator -> Node
				if nativeScope == schema.MetricScopeAccelerator && scope == schema.MetricScopeNode {
					if len(acceleratorIds) == 0 {
						continue
					}

					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   hostname,
						Aggregate:  true,
						Type:       &acceleratorString,
						TypeIds:    acceleratorIds,
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// HWThread -> HWThread
				if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeHWThread {
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   hostname,
						Aggregate:  false,
						Type:       &hwthreadString,
						TypeIds:    intToStringSlice(topology.Node),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// HWThread -> Core
				if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeCore {
					cores, _ := topology.GetCoresFromHWThreads(topology.Node)
					for _, core := range cores {
						queries = append(queries, APIQuery{
							Metric:     metric,
							Hostname:   hostname,
							Aggregate:  true,
							Type:       &hwthreadString,
							TypeIds:    intToStringSlice(topology.Core[core]),
							Resolution: resolution,
						})
						assignedScope = append(assignedScope, scope)
					}
					continue
				}

				// HWThread -> Socket
				if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeSocket {
					sockets, _ := topology.GetSocketsFromHWThreads(topology.Node)
					for _, socket := range sockets {
						queries = append(queries, APIQuery{
							Metric:     metric,
							Hostname:   hostname,
							Aggregate:  true,
							Type:       &hwthreadString,
							TypeIds:    intToStringSlice(topology.Socket[socket]),
							Resolution: resolution,
						})
						assignedScope = append(assignedScope, scope)
					}
					continue
				}

				// HWThread -> Node
				if nativeScope == schema.MetricScopeHWThread && scope == schema.MetricScopeNode {
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   hostname,
						Aggregate:  true,
						Type:       &hwthreadString,
						TypeIds:    intToStringSlice(topology.Node),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// Core -> Core
				if nativeScope == schema.MetricScopeCore && scope == schema.MetricScopeCore {
					cores, _ := topology.GetCoresFromHWThreads(topology.Node)
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   hostname,
						Aggregate:  false,
						Type:       &coreString,
						TypeIds:    intToStringSlice(cores),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// Core -> Socket
				if nativeScope == schema.MetricScopeCore && scope == schema.MetricScopeSocket {
					sockets, _ := topology.GetSocketsFromCores(topology.Node)
					for _, socket := range sockets {
						queries = append(queries, APIQuery{
							Metric:     metric,
							Hostname:   hostname,
							Aggregate:  true,
							Type:       &coreString,
							TypeIds:    intToStringSlice(topology.Socket[socket]),
							Resolution: resolution,
						})
						assignedScope = append(assignedScope, scope)
					}
					continue
				}

				// Core -> Node
				if nativeScope == schema.MetricScopeCore && scope == schema.MetricScopeNode {
					cores, _ := topology.GetCoresFromHWThreads(topology.Node)
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   hostname,
						Aggregate:  true,
						Type:       &coreString,
						TypeIds:    intToStringSlice(cores),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// MemoryDomain -> MemoryDomain
				if nativeScope == schema.MetricScopeMemoryDomain && scope == schema.MetricScopeMemoryDomain {
					sockets, _ := topology.GetMemoryDomainsFromHWThreads(topology.Node)
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   hostname,
						Aggregate:  false,
						Type:       &memoryDomainString,
						TypeIds:    intToStringSlice(sockets),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// MemoryDomain -> Node
				if nativeScope == schema.MetricScopeMemoryDomain && scope == schema.MetricScopeNode {
					sockets, _ := topology.GetMemoryDomainsFromHWThreads(topology.Node)
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   hostname,
						Aggregate:  true,
						Type:       &memoryDomainString,
						TypeIds:    intToStringSlice(sockets),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// Socket -> Socket
				if nativeScope == schema.MetricScopeSocket && scope == schema.MetricScopeSocket {
					sockets, _ := topology.GetSocketsFromHWThreads(topology.Node)
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   hostname,
						Aggregate:  false,
						Type:       &socketString,
						TypeIds:    intToStringSlice(sockets),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// Socket -> Node
				if nativeScope == schema.MetricScopeSocket && scope == schema.MetricScopeNode {
					sockets, _ := topology.GetSocketsFromHWThreads(topology.Node)
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   hostname,
						Aggregate:  true,
						Type:       &socketString,
						TypeIds:    intToStringSlice(sockets),
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				// Node -> Node
				if nativeScope == schema.MetricScopeNode && scope == schema.MetricScopeNode {
					queries = append(queries, APIQuery{
						Metric:     metric,
						Hostname:   hostname,
						Resolution: resolution,
					})
					assignedScope = append(assignedScope, scope)
					continue
				}

				return nil, nil, fmt.Errorf("METRICDATA/INTERNAL-CCMS > TODO: unhandled case: native-scope=%s, requested-scope=%s", nativeScope, requestedScope)
			}
		}
	}

	return queries, assignedScope, nil
}

// sanitizeStats converts NaN statistics to zero for JSON compatibility.
//
// schema.Float with NaN values cannot be properly JSON-encoded, so we convert
// NaN to 0. This loses the distinction between "no data" and "zero value",
// but maintains API compatibility.
func sanitizeStats(data *APIMetricData) {
	if data.Avg.IsNaN() {
		data.Avg = schema.Float(0)
	}
	if data.Min.IsNaN() {
		data.Min = schema.Float(0)
	}
	if data.Max.IsNaN() {
		data.Max = schema.Float(0)
	}
}

// intToStringSlice converts a slice of integers to a slice of strings.
// Used to convert hardware thread/core/socket IDs from topology (int) to APIQuery TypeIds (string).
//
// Optimized to reuse a byte buffer for string conversion, reducing allocations.
func intToStringSlice(is []int) []string {
	if len(is) == 0 {
		return nil
	}

	ss := make([]string, len(is))
	buf := make([]byte, 0, 16) // Reusable buffer for integer conversion
	for i, x := range is {
		buf = strconv.AppendInt(buf[:0], int64(x), 10)
		ss[i] = string(buf)
	}
	return ss
}