Excellent! Now I have a comprehensive understanding of the package. Let me create a detailed summary report.

## Comprehensive Exploration Report: `internal/metricstoreclient/` Package

### Overview

The `internal/metricstoreclient/` package is a **client library for querying cc-metric-store**, an external high-performance time series database optimized for HPC metric data. It mirrors the functionality of `pkg/metricstore/query.go` but communicates with an external cc-metric-store service via HTTP instead of using an in-memory store.

**Files in package:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/metricstoreclient/cc-metric-store.go` (28 KB) — Main client operations
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/metricstoreclient/cc-metric-store-queries.go` (16 KB) — Query building logic

---

### 1. Core Type: `CCMetricStore`

**Location:** `cc-metric-store.go`, lines 71-79

```go
type CCMetricStore struct {
    client        http.Client                 // HTTP client with 10-second timeout
    jwt           string                      // JWT Bearer token for authentication
    url           string                      // Base URL of cc-metric-store instance
    queryEndpoint string                      // Full URL to query API endpoint
    topologyCache map[string]*schema.Topology // cluster -> topology cache
}
```

**Purpose:** HTTP client for communicating with an external cc-metric-store service. Manages connection details, authentication (JWT), and caching of cluster topology information.

**Constructor:**
```go
func NewCCMetricStore(url string, token string) *CCMetricStore
```
- Creates client with 10-second timeout
- Initializes empty topology cache
- URL should include protocol and port (e.g., "http://localhost:8080")
- Token is JWT for Bearer authentication; empty string disables auth

---

### 2. API Request/Response Types

#### `APIQueryRequest` (lines 83-91)
Represents a request to the cc-metric-store query API:
```go
type APIQueryRequest struct {
    Cluster     string     // Target cluster name
    Queries     []APIQuery // Explicit list of metric queries
    ForAllNodes []string   // Metrics to query for all nodes (bulk query)
    From        int64      // Start time (Unix timestamp)
    To          int64      // End time (Unix timestamp)
    WithStats   bool       // Include min/avg/max statistics
    WithData    bool       // Include time series data points
}
```

#### `APIQuery` (lines 95-104)
Specifies a single metric query with optional scope filtering:
```go
type APIQuery struct {
    Type       *string  // Scope type (e.g., "core", "socket", "hwthread")
    SubType    *string  // Sub-scope type (reserved for future use)
    Metric     string   // Metric name
    Hostname   string   // Target hostname
    Resolution int      // Data resolution in seconds (0 = native)
    TypeIds    []string // IDs for the scope type (e.g., ["0", "1", "2"] for core IDs)
    SubTypeIds []string // IDs for sub-scope (reserved)
    Aggregate  bool     // Aggregate across TypeIds
}
```

#### `APIQueryResponse` (lines 108-111)
Contains results from a cc-metric-store query:
```go
type APIQueryResponse struct {
    Queries []APIQuery        // Echoed queries (for bulk requests)
    Results [][]APIMetricData // Result data, indexed by query
}
```

#### `APIMetricData` (lines 115-124)
Time series data and statistics for a single metric series:
```go
type APIMetricData struct {
    Error      *string        // Error message if query failed
    Data       []schema.Float // Time series data points
    From       int64          // Actual start time of data
    To         int64          // Actual end time of data
    Resolution int            // Actual resolution of data in seconds
    Avg        schema.Float   // Average value across time range
    Min        schema.Float   // Minimum value in time range
    Max        schema.Float   // Maximum value in time range
}
```

---

### 3. Core Methods on `CCMetricStore`

#### `doRequest()` (lines 144-188)
**Purpose:** Executes HTTP POST request to cc-metric-store query API

**Key behavior:**
- Encodes request body as JSON
- Adds JWT Bearer token if configured
- Adds API versioning (v2 for data with resampling, v1 without)
- Uses GET method with JSON body (unusual but matches cc-metric-store API)
- Returns parsed `APIQueryResponse`
- Properly closes response body to prevent resource leaks

#### `getTopology()` (lines 191-204)
**Purpose:** Returns topology for given cluster and subcluster, caching results

```go
func (ccms *CCMetricStore) getTopology(cluster, subCluster string) (*schema.Topology, error)
```

**Implementation:**
- Uses cache key `"cluster:subcluster"`
- Returns cached topology if available
- Otherwise loads via `archive.GetSubCluster()` and caches it
- Returns pointer to `schema.Topology` from subcluster

#### `getTopologyByNode()` (lines 207-214)
**Purpose:** Returns topology for given cluster and node, caching it

```go
func (ccms *CCMetricStore) getTopologyByNode(cluster, node string) (*schema.Topology, error)
```

**Implementation:**
- Uses `archive.GetSubClusterByNode()` to find subcluster for node
- Delegates to `getTopology()` for cache lookup and loading

---

### 4. Main Data Loading Methods

#### `LoadData()` (lines 228-362)
**Purpose:** Retrieves time series data and statistics for specified job and metrics

**Signature:**
```go
func (ccms *CCMetricStore) LoadData(
    job *schema.Job,
    metrics []string,
    scopes []schema.MetricScope,
    ctx context.Context,
    resolution int,
) (schema.JobData, error)
```

**Returns:** `schema.JobData` organized as: `metric → scope → series list`

**Key steps:**
1. Calls `buildQueries()` to construct API queries with scope transformations
2. Creates `APIQueryRequest` with job time range and resources
3. Calls `doRequest()` to fetch from cc-metric-store
4. Processes results with safety checks for index bounds
5. For each result, extracts metric config and builds `schema.Series` objects
6. Supports partial errors: returns available data even if some queries fail
7. Sanitizes NaN statistics for JSON compatibility

#### `LoadStats()` (lines 368-427)
**Purpose:** Retrieves min/avg/max statistics for job metrics at node scope (faster, no time series data)

**Signature:**
```go
func (ccms *CCMetricStore) LoadStats(
    job *schema.Job,
    metrics []string,
    ctx context.Context,
) (map[string]map[string]schema.MetricStatistics, error)
```

**Returns:** Statistics organized as: `metric → hostname → statistics`

**Key differences from LoadData:**
- Only queries node scope
- Sets `WithData: false` (no time series)
- Sets `WithStats: true` (only statistics)
- Simpler result processing

#### `LoadScopedStats()` (lines 434-524)
**Purpose:** Retrieves statistics for job metrics across multiple scopes (used for Job-View Statistics Table)

**Signature:**
```go
func (ccms *CCMetricStore) LoadScopedStats(
    job *schema.Job,
    metrics []string,
    scopes []schema.MetricScope,
    ctx context.Context,
) (schema.ScopedJobStats, error)
```

**Returns:** Statistics organized as: `metric → scope → list of scoped statistics`

**Key features:**
- Similar structure to `LoadData()` but returns `schema.ScopedStats` instead of full series
- Each scoped stat includes hostname, hardware ID (if applicable), and min/avg/max

#### `LoadNodeData()` (lines 531-619)
**Purpose:** Retrieves current metric data for specified nodes in a cluster (Systems-View Node-Overview)

**Signature:**
```go
func (ccms *CCMetricStore) LoadNodeData(
    cluster string,
    metrics, nodes []string,
    scopes []schema.MetricScope,
    from, to time.Time,
    ctx context.Context,
) (map[string]map[string][]*schema.JobMetric, error)
```

**Returns:** Data organized as: `hostname → metric → []*JobMetric` (list supports multiple scopes)

**Key features:**
- If `nodes == nil`: performs bulk query for all metrics on all nodes
- Otherwise: queries each specified node for each metric
- Simpler query construction (doesn't call `buildQueries()`)
- No scope transformations applied

#### `LoadNodeListData()` (lines 631-780)
**Purpose:** Retrieves paginated node metrics for Systems-View Node-List with filtering

**Signature:**
```go
func (ccms *CCMetricStore) LoadNodeListData(
    cluster, subCluster string,
    nodes []string,
    metrics []string,
    scopes []schema.MetricScope,
    resolution int,
    from, to time.Time,
    ctx context.Context,
) (map[string]schema.JobData, error)
```

**Returns:** Data organized as: `hostname → JobData (metric → scope → series)`

**Key steps:**
1. Calls `buildNodeQueries()` for query construction with scope transformations
2. Uses `getTopology()` or `getTopologyByNode()` depending on subcluster parameter
3. Similar result processing to `LoadData()` with safety checks

#### `HealthCheck()` (lines 785-828)
**Purpose:** Queries external cc-metric-store's health check endpoint

**Key features:**
- Sends `metricstore.HealthCheckReq` with cluster, nodes, metrics
- Uses `/api/healthcheck` endpoint
- Returns per-node health check results

---

### 5. Query Building (cc-metric-store-queries.go)

#### Scope String Constants (lines 49-55)
Pre-converted strings to avoid repeated allocations:
```go
hwthreadString     = "hwthread"
coreString         = "core"
memoryDomainString = "memory_domain"
socketString       = "socket"
acceleratorString  = "accelerator"
```

#### `buildQueries()` (lines 67-149)
**Purpose:** Constructs API queries for job-specific metric data

**Signature:**
```go
func (ccms *CCMetricStore) buildQueries(
    job *schema.Job,
    metrics []string,
    scopes []schema.MetricScope,
    resolution int,
) ([]APIQuery, []schema.MetricScope, error)
```

**Returns:** Both queries and their corresponding assigned scopes (may differ from requested)

**Key logic:**
1. Validates metric configurations via `archive.GetMetricConfig()`
2. Skips metrics removed for subcluster
3. Deduplicates requested scopes to avoid redundant queries
4. For each host in job resources:
   - Uses job-allocated HWThreads (or full node if not specified)
   - Calls `buildScopeQueries()` for scope transformation
   - Accumulates queries and assigned scopes

**Optimization:** Tracks handled scopes to skip redundant queries

#### `buildNodeQueries()` (lines 161-252)
**Purpose:** Constructs API queries for node-specific metric data (Systems View)

**Signature:**
```go
func (ccms *CCMetricStore) buildNodeQueries(
    cluster string,
    subCluster string,
    nodes []string,
    metrics []string,
    scopes []schema.MetricScope,
    resolution int,
) ([]APIQuery, []schema.MetricScope, error)
```

**Key differences from `buildQueries()`:**
- Always uses full node topology (not job-specific subsets)
- Always uses all accelerators on each node
- Handles optional subCluster parameter:
  - If empty: calls `getTopologyByNode()` per node
  - If provided: calls `getTopology()` once and reuses

#### `buildScopeQueries()` (lines 257-517)
**Purpose:** Generates API queries for a given scope transformation

**Signature:**
```go
func buildScopeQueries(
    nativeScope, requestedScope schema.MetricScope,
    metric, hostname string,
    topology *schema.Topology,
    hwthreads []int,
    accelerators []string,
    resolution int,
) ([]APIQuery, []schema.MetricScope)
```

**Returns:** Empty slices for expected exceptions; nil for unexpected unhandled cases

**Scope Transformations Implemented (25+ cases):**

| Native → Requested | Behavior | Result Scope |
|---|---|---|
| Accelerator → Accelerator | Return as-is | Accelerator |
| Accelerator → Node | Aggregate all accelerators | Node |
| HWThread → HWThread | Return as-is | HWThread |
| HWThread → Core | Aggregate per core | Core (one query per core) |
| HWThread → Socket | Aggregate per socket | Socket (one query per socket) |
| HWThread → Node | Aggregate all threads | Node |
| Core → Core | Return as-is | Core |
| Core → Socket | Aggregate per socket | Socket (one query per socket) |
| Core → Node | Aggregate all cores | Node |
| MemoryDomain → MemoryDomain | Return as-is | MemoryDomain |
| MemoryDomain → Node | Aggregate all domains | Node |
| MemoryDomain → Socket | Aggregate domains per socket | Socket (one query per socket) |
| Socket → Socket | Return as-is | Socket |
| Socket → Node | Aggregate all sockets | Node |
| Node → Node | Return as-is | Node |

**Key features:**
- Uses `Aggregate: true` for aggregations, `false` for direct queries
- Sets `Type` and `TypeIds` for scope-specific queries
- Returns nil slices for unhandled cases (triggers error in `buildQueries()`)
- Uses helper functions like `GetCoresFromHWThreads()`, `GetSocketsFromCores()`, etc. from topology

#### `intToStringSlice()` (lines 521-527)
Helper function to convert integer hardware IDs to string format required by API:
```go
func intToStringSlice(is []int) []string
```

---

### 6. Helper Functions

#### `sanitizeStats()` (lines 832-838)
Replaces NaN values in statistics with 0 to enable JSON marshaling:
```go
func sanitizeStats(avg, min, max *schema.Float)
```

#### `hasNaNStats()` (lines 841-843)
Returns true if any of the statistics contain NaN values:
```go
func hasNaNStats(avg, min, max schema.Float) bool
```

---

### 7. Relationship to `pkg/metricstore/query.go`

Both packages implement **parallel interfaces** with nearly identical method signatures:

| Method | Location | Purpose |
|---|---|---|
| `LoadData()` | Both | Load job metric data with scope transformations |
| `LoadStats()` | Both | Load statistics at node scope |
| `LoadScopedStats()` | Both | Load statistics across multiple scopes |
| `HealthCheck()` | Both | Health check for metric store |

**Key differences:**

| Aspect | `pkg/metricstore/` | `internal/metricstoreclient/` |
|---|---|---|
| **Storage** | In-memory store | External cc-metric-store via HTTP |
| **Data fetching** | `FetchData()` (internal) | `doRequest()` (HTTP to cc-metric-store) |
| **Query builder** | `buildQueries()` in query.go | `buildQueries()` + `buildScopeQueries()` in cc-metric-store-queries.go |
| **Authentication** | None (in-process) | JWT Bearer token |
| **Caching** | No caching | Topology caching |
| **API versioning** | Not applicable | Uses cc-metric-store v2 API |
| **Timeout** | Not applicable | 10-second HTTP timeout |
| **Resolution type** | `int64` | `int` |

**Code Similarity:**
- Both use same scope transformation logic
- Both construct queries with metric configs from `archive.GetMetricConfig()`
- Both handle partial errors (return data for successful queries)
- Both sanitize NaN statistics
- Nearly identical result processing loops with identical safety checks

**When to use each:**
- **`pkg/metricstore/`**: When cc-metric-store is co-located or unavailable
- **`internal/metricstoreclient/`**: When cc-metric-store is available externally (production preferred for scalability)

---

### 8. API Endpoint Details

**Query Endpoint:**
- Base: `http://cc-metric-store-host:port`
- Query: `{base}/api/query?version=v2`
- Health: `{base}/api/healthcheck`

**Message Format:**
Uses InfluxDB-like line protocol with JSON payloads in `APIQuery` structures. For job events/node updates via NATS (per CLAUDE.md), uses InfluxDB line protocol directly.

---

### 9. Error Handling

- **Partial errors**: Methods return both data and error; error lists failed queries
- **Index out of range**: Prevented with bounds checking; logged as warnings
- **Invalid scopes**: Return nil slices from `buildScopeQueries()` triggering error
- **Metric config missing**: Logged warning, metric skipped
- **HTTP errors**: Returned from `doRequest()`
- **Context cancellation**: Supported via context parameter in all load methods

---

### Summary

The `internal/metricstoreclient/` package is a **complete HTTP client for external cc-metric-store** that mirrors the functionality of the internal metric store while communicating via HTTP API. It handles complex scope transformations, topology caching, and partial error reporting. The query building logic handles 25+ scope transformation cases, enabling flexible metric data requests across different hardware granularities while maintaining compatibility with the in-memory metric store interface.

agentId: ac0e18c809f60baa6 (for resuming to continue this agent's work if needed)
<usage>total_tokens: 42376
tool_uses: 6
duration_ms: 47035</usage>

Prompt:
Explore the internal/metricstoreclient/ package thoroughly. I need to understand:
1. All files in this package
2. The CCMetricStore type and its methods (especially LoadData, LoadStats, LoadScopedStats, LoadNodeData, LoadNodeListData)
3. How it defines APIQuery and related types
4. The getTopology and getTopologyByNode helper methods
5. How this package relates to pkg/metricstore/

Focus on understanding the full structure and how it parallels pkg/metricstore/query.go.


Perfect! Now I have a comprehensive understanding of the package. Let me compile my findings into a detailed report.

## Comprehensive Exploration of `pkg/metricstore/` Package

I've thoroughly analyzed the `pkg/metricstore/` package. Here's a complete overview:

### Package Overview
The `pkg/metricstore/` is a sophisticated in-memory time-series metric storage system for ClusterCockpit, providing:
- Hierarchical metric organization (cluster → host → component hierarchy)
- Concurrent read/write access with RWMutex protection
- Periodic checkpointing (JSON or WAL format) and archiving
- Memory management with retention policies and emergency freeing
- NATS integration for real-time metric ingestion
- Scope transformation for job metric data (HWThread → Core → Socket → Node aggregation)

### Complete File List (17 files, 7,615 lines total)

1. **metricstore.go** (784 lines) - Core singleton instance and lifecycle
2. **api.go** (314 lines) - API query types and FetchData function
3. **query.go** (1,357 lines) - Job metric loading with scope transformation
4. **level.go** (393 lines) - Hierarchical tree structure
5. **buffer.go** (410 lines) - Time-series data buffers
6. **config.go** (253 lines) - Configuration structures
7. **checkpoint.go** (662 lines) - Persistence to disk
8. **walCheckpoint.go** (797 lines) - WAL format implementation
9. **lineprotocol.go** (366 lines) - NATS line-protocol ingestion
10. **healthcheck.go** (197 lines) - Monitoring state detection
11. **archive.go** (277 lines) - Long-term storage
12. **parquetArchive.go** (213 lines) - Parquet format support
13. **stats.go** (125 lines) - Statistics computation
14. **debug.go** (112 lines) - Debugging utilities
15. **configSchema.go** (81 lines) - JSON schema validation
16. **metricstore_test.go** (1,019 lines) - Unit tests
17. **parquetArchive_test.go** (255 lines) - Parquet tests

---

## 1. TYPE DEFINITIONS

### Core Types

**APIMetricData** (api.go)
```go
type APIMetricData struct {
    Error      *string           // Error message if retrieval failed
    Data       schema.FloatArray // Time-series data points
    From       int64             // Actual start timestamp
    To         int64             // Actual end timestamp
    Resolution int64             // Resolution in seconds
    Avg        schema.Float      // Average of data points
    Min        schema.Float      // Minimum value
    Max        schema.Float      // Maximum value
}
```

**APIQueryRequest** (api.go)
```go
type APIQueryRequest struct {
    Cluster     string     // Cluster name
    Queries     []APIQuery // Individual metric queries
    ForAllNodes []string   // Auto-generate queries for all nodes in cluster (metric names)
    From        int64      // Start timestamp
    To          int64      // End timestamp
    WithStats   bool       // Compute avg/min/max
    WithData    bool       // Include data array
    WithPadding bool       // Pad with NaN at start
}
```

**APIQuery** (api.go)
```go
type APIQuery struct {
    Type        *string      // Scope level (e.g., "hwthread", "core", "socket")
    SubType     *string      // Sub-level scope (e.g., "core" under "socket")
    Metric      string       // Metric name
    Hostname    string       // Node hostname
    Resolution  int64        // Requested resolution
    TypeIds     []string     // IDs for Type level (e.g., ["0", "1", "2"] for cores)
    SubTypeIds  []string     // IDs for SubType level
    ScaleFactor schema.Float // Multiplier for unit conversion
    Aggregate   bool         // Sum/avg across TypeIds/SubTypeIds
}
```

**APIQueryResponse** (api.go)
```go
type APIQueryResponse struct {
    Queries []APIQuery        // Echo of input queries
    Results [][]APIMetricData // Results: [query_idx][selector_idx]
}
```

### Storage Types

**MemoryStore** (metricstore.go)
```go
type MemoryStore struct {
    Metrics      map[string]MetricConfig // Metric config map (name → MetricConfig)
    root         Level                   // Root of hierarchical tree
    nodeProvider NodeProvider            // For job-aware retention
}
```

**Level** (level.go) - Hierarchical tree node
```go
type Level struct {
    children map[string]*Level  // Child levels (e.g., "node001" → Level)
    metrics  []*buffer          // Metric buffers (indexed by MetricConfig.offset)
    lock     sync.RWMutex       // Protects concurrent access
}
```

**buffer** (buffer.go) - Time-series data storage
```go
type buffer struct {
    prev      *buffer      // Older buffer in chain
    next      *buffer      // Newer buffer in chain
    data      []schema.Float // Data points
    frequency int64        // Measurement interval (seconds)
    start     int64        // Start timestamp (adjusted by -frequency/2)
    archived  bool         // Persisted to archive
    closed    bool         // No longer accepting writes
    lastUsed  int64        // Last used timestamp (for pool cleanup)
}
```

### Configuration Types

**MetricConfig** (config.go)
```go
type MetricConfig struct {
    Frequency   int64               // Measurement interval in seconds
    Aggregation AggregationStrategy // How to combine values (sum/avg/none)
    offset      int                 // Internal index into Level.metrics
}
```

**AggregationStrategy** (config.go)
```go
const (
    NoAggregation  AggregationStrategy = iota // Incompatible scopes
    SumAggregation                            // Add values (power, energy)
    AvgAggregation                            // Average values (temperature)
)
```

**MetricStoreConfig** (config.go)
```go
type MetricStoreConfig struct {
    NumWorkers        int           // Workers for checkpoint/archive
    RetentionInMemory string        // Duration (e.g., "48h")
    MemoryCap         int           // Memory cap in bytes
    Checkpoints       Checkpoints   // Checkpoint settings
    Debug             *Debug        // Debug options
    Cleanup           *Cleanup      // Archive settings
    Subscriptions     *Subscriptions // NATS topics
}
```

### Query Building Types

**InternalMetricStore** (query.go)
```go
type InternalMetricStore struct{}
var MetricStoreHandle *InternalMetricStore
```
- Provides public API for job data loading
- Methods: `LoadData()`, `HealthCheck()`
- Delegates to underlying `MemoryStore`

### Health Check Types

**HealthCheckResult** (healthcheck.go)
```go
type HealthCheckResult struct {
    State         schema.MonitoringState // Full/Partial/Failed
    HealthMetrics string                 // JSON: {"missing":[...],"degraded":[...]}
}
```

---

## 2. HOW FetchData WORKS

**Function Signature**: `func FetchData(req APIQueryRequest) (*APIQueryResponse, error)`

**High-Level Flow**:

1. **Validation**
   - Checks `req.From <= req.To`
   - Ensures cluster is specified if `ForAllNodes` is used
   - Verifies MemoryStore is initialized

2. **Query Expansion**
   - If `req.ForAllNodes` is set: auto-generates queries for all nodes in the cluster
   - Calls `ms.ListChildren([]string{cluster})` to get node list
   - Creates one query per (node, metric) combination

3. **Selector Construction** (per query)
   - Base selector: `[cluster, hostname]`
   - If `Aggregate=true` or no Type specified:
     - Creates single aggregated selector
     - If Type is specified and has multiple TypeIds:
       - Uses `SelectorElement{Group: ids}` for multi-selection
     - Similar for SubType
   - If `Aggregate=false`:
     - Creates individual selector for each Type/SubType ID combination
     - Enables non-aggregated per-component results

4. **Data Fetching**
   - For each selector: calls `ms.Read(selector, metric, from, to, resolution)`
   - Returns: `(data, from, to, resolution, error)`

5. **Result Processing** (per query result)
   - **Skips** `ErrNoHostOrMetric` errors (logs warning, continues)
   - **Includes** other errors in `APIMetricData.Error` field
   - If `req.WithStats`: calls `data.AddStats()` to compute avg/min/max
   - If `query.ScaleFactor != 0`: calls `data.ScaleBy()` for unit conversion
   - If `req.WithPadding`: calls `data.PadDataWithNull()` to align timestamps
   - If `!req.WithData`: clears the data array (keeps stats)

6. **Response Assembly**
   - Returns 2D array: `Results[query_idx][selector_idx]`
   - Each inner element is `APIMetricData` with data/stats

**Example Scenario**:
```
Request: Fetch cpu_load for cluster "emmy", node "node001", all cores (0-3)
  Type="core", TypeIds=["0","1","2","3"], Aggregate=false

Result: 4 separate APIMetricData entries, one per core
  (each with separate data points)
```

---

## 3. InternalMetricStore & Its Relationship

### Purpose
`InternalMetricStore` is a **public facade** that abstracts the `MemoryStore` singleton, preventing direct exposure of internal details.

### Key Methods

**`LoadData()`** (query.go)
```go
func (ccms *InternalMetricStore) LoadData(
    job *schema.Job,
    metrics []string,
    scopes []schema.MetricScope,
    ctx context.Context,
    resolution int,
) (schema.JobData, error)
```

**Responsibilities**:
1. Calls `buildQueries()` to construct API queries with scope transformation
2. Determines target scope for each query (e.g., HWThread native → Socket requested)
3. Calls `FetchData()` to retrieve metric data
4. Reorganizes results by metric and scope into `schema.JobData` structure
5. Returns: `map[metricName]map[scope]*JobMetric`

**Scope Transformation Logic** (in `buildQueries()`):
- Native scope >= requested scope: **Aggregate** data (Aggregate=true)
- Native scope < requested scope: **Error** (cannot refine granularity)
- Handles special cases:
  - Accelerator metrics (no aggregation)
  - Memory domain metrics
  - Multi-node jobs (per-node queries)

**`HealthCheck()`** (query.go)
```go
func (ccms *InternalMetricStore) HealthCheck(
    cluster string,
    nodes []string,
    metrics []string,
) (map[string]HealthCheckResult, error)
```
- Delegates to `ms.HealthCheck()`
- Returns monitoring state per node (Full/Partial/Failed)

### Relationship to MemoryStore

| Aspect | InternalMetricStore | MemoryStore |
|--------|---------------------|------------|
| **Role** | Public API facade | Internal storage/retrieval |
| **Instance** | Singleton (var MetricStoreHandle) | Singleton (msInstance) |
| **Access** | Used by external packages | Only via InternalMetricStore |
| **Methods** | High-level: LoadData, HealthCheck | Low-level: Write, Read, Free |
| **Scope** | Job-centric, scope transformation | Metric-centric, raw storage |

---

## 4. KEY DATA STRUCTURES & RELATIONSHIPS

### Hierarchical Tree Organization

```
MemoryStore
└── root (Level)
    └── children["emmy"] (cluster Level)
        └── metrics [buffer, buffer, ...] (node-level aggregates)
        └── children["node001"] (host Level)
            └── metrics [buffer, buffer, ...]
            └── children["socket0"] (Level)
                └── metrics [buffer, buffer, ...]
                └── children["core0"] (Level)
                    └── metrics [buffer, buffer, ...] (actual data)
                └── children["core1"]
                    └── metrics [buffer, buffer, ...]
```

### Buffer Chain (Linked List)

Time ordering: oldest ← prev -- prev -- newest (head)

```
[start=1000] ← [start=1512] ← [start=2024] (head)
 data[512]       data[512]      data[42]
```

### Metrics Indexing

- Each `Level.metrics` is a slice of `*buffer`
- Index = `MetricConfig.offset` (assigned during `InitMetrics()`)
- Example:
  ```
  Metrics map: {"cpu_load": {offset: 0}, "mem_used": {offset: 1}}
  Level.metrics: [*buffer for cpu_load, *buffer for mem_used]
  ```

---

## 5. LIFECYCLE & BACKGROUND WORKERS

### Initialization (Init)
1. Decodes JSON config into `Keys`
2. Calls `InitMetrics()` to create singleton MemoryStore
3. Loads checkpoints from disk within retention window
4. Starts 5 background workers:
   - `Retention()` - Frees old data based on retention policy
   - `Checkpointing()` - Periodic snapshot to disk
   - `CleanUp()` - Archives/deletes data older than retention
   - `WALStaging()` - Writes WAL entries to disk (if WAL format)
   - `MemoryUsageTracker()` - Monitors memory, triggers emergency freeing

### Shutdown
1. Cancels all background workers
2. Closes WAL messages channel (if WAL format)
3. Writes final checkpoint to disk

---

## 6. CHECKPOINT FORMATS

### JSON Format (Legacy)
```
checkpoints/
  cluster1/
    host001/
      1234567890.json
```

Structure:
```go
type CheckpointFile struct {
    Metrics  map[string]*CheckpointMetrics
    Children map[string]*CheckpointFile
    From     int64
    To       int64
}
```

### WAL Format (New, Recommended)
```
checkpoints/
  cluster1/
    host001/
      1234567890.bin    (full binary snapshot)
      current.wal       (append-only write-ahead log)
```

- **Binary snapshot**: Complete tree dump at a point in time
- **WAL file**: Per-write entries, replayed on startup for crash recovery
- **Rotation**: After checkpoint, WAL is truncated

---

## 7. CONCURRENCY STRATEGY

### RWMutex Usage
- **Read-heavy**: Metric reads during `ms.Read()` use `RLock`
- **Write-rare**: Level creation and buffer writes use `Lock`
- **Double-checked locking**: In `findLevelOrCreate()` prevents race conditions

### Channel-based Communication
- `WALMessages` channel: Queues WAL entries for disk write
- `WALStaging()` goroutine: Drains channel, batches to disk

### Buffer Pool
- Sync.Pool reduces GC pressure
- `PersistentBufferPool` with LRU cleanup (based on lastUsed timestamp)

---

## 8. RETENTION & MEMORY MANAGEMENT

### Retention Policy
- Time-based: Data older than `RetentionInMemory` is freed
- **NodeProvider integration**: Preserves data for nodes with active jobs
- Runs at half the retention interval

### Memory Emergency
- Monitors actual heap usage via `runtime.MemStats`
- If exceeds `MemoryCap`: triggers `ForceFree()` up to 100 iterations
- `ForceFree()` removes oldest buffer from each metric chain

### Free Mechanisms
- **Free()**: Removes buffers older than threshold, respecting active nodes
- **FreeSelected()**: Frees specific selectors (excludes others)
- **ForceFree()**: Unconditionally removes oldest buffers

---

## Summary

The `pkg/metricstore/` package is a production-grade time-series storage system with:
- **Hierarchical organization** for efficient per-component metric storage
- **Flexible API** supporting both raw queries and job-centric scope transformation
- **Robust persistence** via JSON or WAL+binary checkpoint formats
- **Intelligent memory management** with retention, emergency freeing, and pooling
- **Real-time ingestion** via NATS line-protocol
- **Health monitoring** with degraded metric detection

The **InternalMetricStore** abstraction provides a clean public interface for job metric loading, handling scope transformation automatically.

agentId: a9c86a3b0a5799ebb (for resuming to continue this agent's work if needed)
<usage>total_tokens: 57113
tool_uses: 15
duration_ms: 57023</usage>

Prompt:
Explore the pkg/metricstore/ package thoroughly. I need to understand:
1. What types are defined (APIQuery, APIQueryRequest, etc.) - check metricstore.go and any type definition files
2. How FetchData works
3. What InternalMetricStore is and how it relates to the metric store interface
4. The full list of files in this package

Focus on understanding the package's role and its types/interfaces.


Now let me read a few key files to understand the exact differences and consolidation opportunities.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// This file contains the API types and data fetching logic for querying metric data
     7→// from the in-memory metric store. It provides structures for building complex queries
     8→// with support for aggregation, scaling, padding, and statistics computation.
     9→package metricstore
    10→
    11→import (
    12→	"errors"
    13→	"fmt"
    14→	"math"
    15→
    16→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    17→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    18→	"github.com/ClusterCockpit/cc-lib/v2/util"
    19→)
    20→
    21→var (
    22→	// ErrNoHostOrMetric is returned when the metric store does not find the host or the metric
    23→	ErrNoHostOrMetric error = errors.New("[METRICSTORE]> metric or host not found")
    24→	// ErrInvalidTimeRange is returned when a query has 'from' >= 'to'
    25→	ErrInvalidTimeRange = errors.New("[METRICSTORE]> invalid time range: 'from' must be before 'to'")
    26→	// ErrEmptyCluster is returned when a query with ForAllNodes has no cluster specified
    27→	ErrEmptyCluster = errors.New("[METRICSTORE]> cluster name cannot be empty")
    28→)
    29→
    30→// APIMetricData represents the response data for a single metric query.
    31→//
    32→// It contains both the time-series data points and computed statistics (avg, min, max).
    33→// If an error occurred during data retrieval, the Error field will be set and other
    34→// fields may be incomplete.
    35→type APIMetricData struct {
    36→	Error      *string           `json:"error,omitempty"`
    37→	Data       schema.FloatArray `json:"data,omitempty"`
    38→	From       int64             `json:"from"`
    39→	To         int64             `json:"to"`
    40→	Resolution int64             `json:"resolution"`
    41→	Avg        schema.Float      `json:"avg"`
    42→	Min        schema.Float      `json:"min"`
    43→	Max        schema.Float      `json:"max"`
    44→}
    45→
    46→// APIQueryRequest represents a batch query request for metric data.
    47→//
    48→// It supports two modes of operation:
    49→//  1. Explicit queries via the Queries field
    50→//  2. Automatic query generation via ForAllNodes (queries all specified metrics for all nodes in the cluster)
    51→//
    52→// The request can be customized with flags to include/exclude statistics, raw data, and padding.
    53→type APIQueryRequest struct {
    54→	Cluster     string     `json:"cluster"`
    55→	Queries     []APIQuery `json:"queries"`
    56→	ForAllNodes []string   `json:"for-all-nodes"`
    57→	From        int64      `json:"from"`
    58→	To          int64      `json:"to"`
    59→	WithStats   bool       `json:"with-stats"`
    60→	WithData    bool       `json:"with-data"`
    61→	WithPadding bool       `json:"with-padding"`
    62→}
    63→
    64→// APIQueryResponse represents the response to an APIQueryRequest.
    65→//
    66→// Results is a 2D array where each outer element corresponds to a query,
    67→// and each inner element corresponds to a selector within that query
    68→// (e.g., multiple CPUs or cores).
    69→type APIQueryResponse struct {
    70→	Queries []APIQuery        `json:"queries,omitempty"`
    71→	Results [][]APIMetricData `json:"results"`
    72→}
    73→
    74→// APIQuery represents a single metric query with optional hierarchical selectors.
    75→//
    76→// The hierarchical selection works as follows:
    77→//   - Hostname: The node to query
    78→//   - Type + TypeIds: First level of hierarchy (e.g., "cpu" + ["0", "1", "2"])
    79→//   - SubType + SubTypeIds: Second level of hierarchy (e.g., "core" + ["0", "1"])
    80→//
    81→// If Aggregate is true, data from multiple type/subtype IDs will be aggregated according
    82→// to the metric's aggregation strategy. Otherwise, separate results are returned for each combination.
    83→type APIQuery struct {
    84→	Type        *string      `json:"type,omitempty"`
    85→	SubType     *string      `json:"subtype,omitempty"`
    86→	Metric      string       `json:"metric"`
    87→	Hostname    string       `json:"host"`
    88→	Resolution  int64        `json:"resolution"`
    89→	TypeIds     []string     `json:"type-ids,omitempty"`
    90→	SubTypeIds  []string     `json:"subtype-ids,omitempty"`
    91→	ScaleFactor schema.Float `json:"scale-by,omitempty"`
    92→	Aggregate   bool         `json:"aggreg"`
    93→}
    94→
    95→// AddStats computes and populates the Avg, Min, and Max fields from the Data array.
    96→//
    97→// NaN values in the data are ignored during computation. If all values are NaN,
    98→// the statistics fields will be set to NaN.
    99→//
   100→// TODO: Optimize this, just like the stats endpoint!
   101→func (data *APIMetricData) AddStats() {
   102→	n := 0
   103→	sum, min, max := 0.0, math.MaxFloat64, -math.MaxFloat64
   104→	for _, x := range data.Data {
   105→		if x.IsNaN() {
   106→			continue
   107→		}
   108→
   109→		n += 1
   110→		sum += float64(x)

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// This file contains the API types and data fetching logic for querying metric data
// from the in-memory metric store. It provides structures for building complex queries
// with support for aggregation, scaling, padding, and statistics computation.
package metricstore

import (
	"errors"
	"fmt"
	"math"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	"github.com/ClusterCockpit/cc-lib/v2/util"
)

var (
	// ErrNoHostOrMetric is returned when the metric store does not find the host or the metric
	ErrNoHostOrMetric error = errors.New("[METRICSTORE]> metric or host not found")
	// ErrInvalidTimeRange is returned when a query has 'from' >= 'to'
	ErrInvalidTimeRange = errors.New("[METRICSTORE]> invalid time range: 'from' must be before 'to'")
	// ErrEmptyCluster is returned when a query with ForAllNodes has no cluster specified
	ErrEmptyCluster = errors.New("[METRICSTORE]> cluster name cannot be empty")
)

// APIMetricData represents the response data for a single metric query.
//
// It contains both the time-series data points and computed statistics (avg, min, max).
// If an error occurred during data retrieval, the Error field will be set and other
// fields may be incomplete.
type APIMetricData struct {
	Error      *string           `json:"error,omitempty"`
	Data       schema.FloatArray `json:"data,omitempty"`
	From       int64             `json:"from"`
	To         int64             `json:"to"`
	Resolution int64             `json:"resolution"`
	Avg        schema.Float      `json:"avg"`
	Min        schema.Float      `json:"min"`
	Max        schema.Float      `json:"max"`
}

// APIQueryRequest represents a batch query request for metric data.
//
// It supports two modes of operation:
//  1. Explicit queries via the Queries field
//  2. Automatic query generation via ForAllNodes (queries all specified metrics for all nodes in the cluster)
//
// The request can be customized with flags to include/exclude statistics, raw data, and padding.
type APIQueryRequest struct {
	Cluster     string     `json:"cluster"`
	Queries     []APIQuery `json:"queries"`
	ForAllNodes []string   `json:"for-all-nodes"`
	From        int64      `json:"from"`
	To          int64      `json:"to"`
	WithStats   bool       `json:"with-stats"`
	WithData    bool       `json:"with-data"`
	WithPadding bool       `json:"with-padding"`
}

// APIQueryResponse represents the response to an APIQueryRequest.
//
// Results is a 2D array where each outer element corresponds to a query,
// and each inner element corresponds to a selector within that query
// (e.g., multiple CPUs or cores).
type APIQueryResponse struct {
	Queries []APIQuery        `json:"queries,omitempty"`
	Results [][]APIMetricData `json:"results"`
}

// APIQuery represents a single metric query with optional hierarchical selectors.
//
// The hierarchical selection works as follows:
//   - Hostname: The node to query
//   - Type + TypeIds: First level of hierarchy (e.g., "cpu" + ["0", "1", "2"])
//   - SubType + SubTypeIds: Second level of hierarchy (e.g., "core" + ["0", "1"])
//
// If Aggregate is true, data from multiple type/subtype IDs will be aggregated according
// to the metric's aggregation strategy. Otherwise, separate results are returned for each combination.
type APIQuery struct {
	Type        *string      `json:"type,omitempty"`
	SubType     *string      `json:"subtype,omitempty"`
	Metric      string       `json:"metric"`
	Hostname    string       `json:"host"`
	Resolution  int64        `json:"resolution"`
	TypeIds     []string     `json:"type-ids,omitempty"`
	SubTypeIds  []string     `json:"subtype-ids,omitempty"`
	ScaleFactor schema.Float `json:"scale-by,omitempty"`
	Aggregate   bool         `json:"aggreg"`
}

// AddStats computes and populates the Avg, Min, and Max fields from the Data array.
//
// NaN values in the data are ignored during computation. If all values are NaN,
// the statistics fields will be set to NaN.
//
// TODO: Optimize this, just like the stats endpoint!
func (data *APIMetricData) AddStats() {
	n := 0
	sum, min, max := 0.0, math.MaxFloat64, -math.MaxFloat64
	for _, x := range data.Data {
		if x.IsNaN() {
			continue
		}

		n += 1
		sum += float64(x)


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package metricstoreclient provides a client for querying the cc-metric-store time series database.
     7→//
     8→// The cc-metric-store is a high-performance time series database optimized for HPC metric data.
     9→// This client handles HTTP communication, query construction, scope transformations, and data retrieval
    10→// for job and node metrics across different metric scopes (node, socket, core, hwthread, accelerator).
    11→//
    12→// # Architecture
    13→//
    14→// The package is split into two main components:
    15→//   - Client Operations (cc-metric-store.go): HTTP client, request handling, data loading methods
    16→//   - Query Building (cc-metric-store-queries.go): Query construction and scope transformation logic
    17→//
    18→// # Basic Usage
    19→//
    20→//	store := NewCCMetricStore("http://localhost:8080", "jwt-token")
    21→//
    22→//	// Load job data
    23→//	jobData, err := store.LoadData(job, metrics, scopes, ctx, resolution)
    24→//	if err != nil {
    25→//	    log.Fatal(err)
    26→//	}
    27→//
    28→// # Metric Scopes
    29→//
    30→// The client supports hierarchical metric scopes that map to HPC hardware topology:
    31→//   - MetricScopeAccelerator: GPU/accelerator level metrics
    32→//   - MetricScopeHWThread: Hardware thread (SMT) level metrics
    33→//   - MetricScopeCore: CPU core level metrics
    34→//   - MetricScopeSocket: CPU socket level metrics
    35→//   - MetricScopeMemoryDomain: NUMA domain level metrics
    36→//   - MetricScopeNode: Full node level metrics
    37→//
    38→// The client automatically handles scope transformations, aggregating finer-grained metrics
    39→// to coarser scopes when needed (e.g., aggregating core metrics to socket level).
    40→//
    41→// # Error Handling
    42→//
    43→// The client supports partial errors - if some queries fail, it returns both the successful
    44→// data and an error listing the failed queries. This allows processing partial results
    45→// when some nodes or metrics are temporarily unavailable.
    46→//
    47→// # API Versioning
    48→//
    49→// The client uses cc-metric-store API v2, which includes support for:
    50→//   - Data resampling for bandwidth optimization
    51→//   - Multi-scope queries in a single request
    52→//   - Aggregation across hardware topology levels
    53→package metricstoreclient
    54→
    55→import (
    56→	"bufio"
    57→	"bytes"
    58→	"context"
    59→	"encoding/json"
    60→	"fmt"
    61→	"net/http"
    62→	"strings"
    63→	"time"
    64→
    65→	"github.com/ClusterCockpit/cc-backend/pkg/archive"
    66→	"github.com/ClusterCockpit/cc-backend/pkg/metricstore"
    67→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    68→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    69→)
    70→
    71→// CCMetricStore is the HTTP client for communicating with cc-metric-store.
    72→// It manages connection details, authentication, and provides methods for querying metrics.
    73→type CCMetricStore struct {
    74→	client        http.Client                 // HTTP client with 10-second timeout
    75→	jwt           string                      // JWT Bearer token for authentication
    76→	url           string                      // Base URL of cc-metric-store instance
    77→	queryEndpoint string                      // Full URL to query API endpoint
    78→	topologyCache map[string]*schema.Topology // cluster -> topology cache
    79→}
    80→
    81→// APIQueryRequest represents a request to the cc-metric-store query API.
    82→// It supports both explicit queries and "for-all-nodes" bulk queries.
    83→type APIQueryRequest struct {
    84→	Cluster     string     `json:"cluster"`       // Target cluster name
    85→	Queries     []APIQuery `json:"queries"`       // Explicit list of metric queries
    86→	ForAllNodes []string   `json:"for-all-nodes"` // Metrics to query for all nodes
    87→	From        int64      `json:"from"`          // Start time (Unix timestamp)
    88→	To          int64      `json:"to"`            // End time (Unix timestamp)
    89→	WithStats   bool       `json:"with-stats"`    // Include min/avg/max statistics
    90→	WithData    bool       `json:"with-data"`     // Include time series data points
    91→}
    92→
    93→// APIQuery specifies a single metric query with optional scope filtering.
    94→// Type and TypeIds define the hardware scope (core, socket, accelerator, etc.).
    95→type APIQuery struct {
    96→	Type       *string  `json:"type,omitempty"`        // Scope type (e.g., "core", "socket")
    97→	SubType    *string  `json:"subtype,omitempty"`     // Sub-scope type (reserved for future use)
    98→	Metric     string   `json:"metric"`                // Metric name
    99→	Hostname   string   `json:"host"`                  // Target hostname
   100→	Resolution int      `json:"resolution"`            // Data resolution in seconds (0 = native)
   101→	TypeIds    []string `json:"type-ids,omitempty"`    // IDs for the scope type (e.g., core IDs)
   102→	SubTypeIds []string `json:"subtype-ids,omitempty"` // IDs for sub-scope (reserved)
   103→	Aggregate  bool     `json:"aggreg"`                // Aggregate across TypeIds
   104→}
   105→
   106→// APIQueryResponse contains the results from a cc-metric-store query.
   107→// Results align with the Queries slice by index.
   108→type APIQueryResponse struct {
   109→	Queries []APIQuery        `json:"queries,omitempty"` // Echoed queries (for bulk requests)
   110→	Results [][]APIMetricData `json:"results"`           // Result data, indexed by query
   111→}
   112→
   113→// APIMetricData represents time series data and statistics for a single metric series.
   114→// Error is set if this particular series failed to load.
   115→type APIMetricData struct {
   116→	Error      *string        `json:"error"`      // Error message if query failed
   117→	Data       []schema.Float `json:"data"`       // Time series data points
   118→	From       int64          `json:"from"`       // Actual start time of data
   119→	To         int64          `json:"to"`         // Actual end time of data
   120→	Resolution int            `json:"resolution"` // Actual resolution of data in seconds
   121→	Avg        schema.Float   `json:"avg"`        // Average value across time range
   122→	Min        schema.Float   `json:"min"`        // Minimum value in time range
   123→	Max        schema.Float   `json:"max"`        // Maximum value in time range
   124→}
   125→
   126→// NewCCMetricStore creates and initializes a new (external) CCMetricStore client.
   127→// The url parameter should include the protocol and port (e.g., "http://localhost:8080").
   128→// The token parameter is a JWT used for Bearer authentication; pass empty string if auth is disabled.
   129→func NewCCMetricStore(url string, token string) *CCMetricStore {
   130→	return &CCMetricStore{

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package metricstoreclient provides a client for querying the cc-metric-store time series database.
//
// The cc-metric-store is a high-performance time series database optimized for HPC metric data.
// This client handles HTTP communication, query construction, scope transformations, and data retrieval
// for job and node metrics across different metric scopes (node, socket, core, hwthread, accelerator).
//
// # Architecture
//
// The package is split into two main components:
//   - Client Operations (cc-metric-store.go): HTTP client, request handling, data loading methods
//   - Query Building (cc-metric-store-queries.go): Query construction and scope transformation logic
//
// # Basic Usage
//
//	store := NewCCMetricStore("http://localhost:8080", "jwt-token")
//
//	// Load job data
//	jobData, err := store.LoadData(job, metrics, scopes, ctx, resolution)
//	if err != nil {
//	    log.Fatal(err)
//	}
//
// # Metric Scopes
//
// The client supports hierarchical metric scopes that map to HPC hardware topology:
//   - MetricScopeAccelerator: GPU/accelerator level metrics
//   - MetricScopeHWThread: Hardware thread (SMT) level metrics
//   - MetricScopeCore: CPU core level metrics
//   - MetricScopeSocket: CPU socket level metrics
//   - MetricScopeMemoryDomain: NUMA domain level metrics
//   - MetricScopeNode: Full node level metrics
//
// The client automatically handles scope transformations, aggregating finer-grained metrics
// to coarser scopes when needed (e.g., aggregating core metrics to socket level).
//
// # Error Handling
//
// The client supports partial errors - if some queries fail, it returns both the successful
// data and an error listing the failed queries. This allows processing partial results
// when some nodes or metrics are temporarily unavailable.
//
// # API Versioning
//
// The client uses cc-metric-store API v2, which includes support for:
//   - Data resampling for bandwidth optimization
//   - Multi-scope queries in a single request
//   - Aggregation across hardware topology levels
package metricstoreclient

import (
	"bufio"
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"strings"
	"time"

	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	"github.com/ClusterCockpit/cc-backend/pkg/metricstore"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

// CCMetricStore is the HTTP client for communicating with cc-metric-store.
// It manages connection details, authentication, and provides methods for querying metrics.
type CCMetricStore struct {
	client        http.Client                 // HTTP client with 10-second timeout
	jwt           string                      // JWT Bearer token for authentication
	url           string                      // Base URL of cc-metric-store instance
	queryEndpoint string                      // Full URL to query API endpoint
	topologyCache map[string]*schema.Topology // cluster -> topology cache
}

// APIQueryRequest represents a request to the cc-metric-store query API.
// It supports both explicit queries and "for-all-nodes" bulk queries.
type APIQueryRequest struct {
	Cluster     string     `json:"cluster"`       // Target cluster name
	Queries     []APIQuery `json:"queries"`       // Explicit list of metric queries
	ForAllNodes []string   `json:"for-all-nodes"` // Metrics to query for all nodes
	From        int64      `json:"from"`          // Start time (Unix timestamp)
	To          int64      `json:"to"`            // End time (Unix timestamp)
	WithStats   bool       `json:"with-stats"`    // Include min/avg/max statistics
	WithData    bool       `json:"with-data"`     // Include time series data points
}

// APIQuery specifies a single metric query with optional scope filtering.
// Type and TypeIds define the hardware scope (core, socket, accelerator, etc.).
type APIQuery struct {
	Type       *string  `json:"type,omitempty"`        // Scope type (e.g., "core", "socket")
	SubType    *string  `json:"subtype,omitempty"`     // Sub-scope type (reserved for future use)
	Metric     string   `json:"metric"`                // Metric name
	Hostname   string   `json:"host"`                  // Target hostname
	Resolution int      `json:"resolution"`            // Data resolution in seconds (0 = native)
	TypeIds    []string `json:"type-ids,omitempty"`    // IDs for the scope type (e.g., core IDs)
	SubTypeIds []string `json:"subtype-ids,omitempty"` // IDs for sub-scope (reserved)
	Aggregate  bool     `json:"aggreg"`                // Aggregate across TypeIds
}

// APIQueryResponse contains the results from a cc-metric-store query.
// Results align with the Queries slice by index.
type APIQueryResponse struct {
	Queries []APIQuery        `json:"queries,omitempty"` // Echoed queries (for bulk requests)
	Results [][]APIMetricData `json:"results"`           // Result data, indexed by query
}

// APIMetricData represents time series data and statistics for a single metric series.
// Error is set if this particular series failed to load.
type APIMetricData struct {
	Error      *string        `json:"error"`      // Error message if query failed
	Data       []schema.Float `json:"data"`       // Time series data points
	From       int64          `json:"from"`       // Actual start time of data
	To         int64          `json:"to"`         // Actual end time of data
	Resolution int            `json:"resolution"` // Actual resolution of data in seconds
	Avg        schema.Float   `json:"avg"`        // Average value across time range
	Min        schema.Float   `json:"min"`        // Minimum value in time range
	Max        schema.Float   `json:"max"`        // Maximum value in time range
}

// NewCCMetricStore creates and initializes a new (external) CCMetricStore client.
// The url parameter should include the protocol and port (e.g., "http://localhost:8080").
// The token parameter is a JWT used for Bearer authentication; pass empty string if auth is disabled.
func NewCCMetricStore(url string, token string) *CCMetricStore {
	return &CCMetricStore{