Design an optimal index strategy for the `job` table in a SQLite database with 20 million rows. The current setup has 79 indexes which is excessive — it confuses the query planner, wastes disk space (~2x overhead per index on 20M rows), and slows inserts/updates.

## Current Problem
- 79 indexes on one table, many redundant
- SQLite query planner picks wrong indexes without ANALYZE (e.g., `jobs_jobstate_energy` instead of `jobs_jobstate_starttime_duration` for ORDER BY start_time queries)
- `IN` clauses with multiple values + `ORDER BY` causes temp B-tree sorts
- Timeouts on filter queries at 20M rows

## Key Constraints
1. **SQLite only** — no partial indexes with expressions on columns (but expression indexes on JSON are supported)
2. Each additional index costs ~200-400MB at 20M rows
3. The `ANALYZE` fix (already implemented) helps the planner, but we still need fewer, better indexes
4. Indexes must be defined in a migration file

## Job Table Columns
```
id (PK), job_id, cluster, subcluster, submit_time, start_time, hpc_user, project,
cluster_partition, array_job_id, duration, walltime, job_state, meta_data (JSON),
resources (JSON), num_nodes, num_hwthreads, num_acc, smt, shared, monitoring_status,
energy, energy_footprint (JSON), footprint (JSON)
UNIQUE(job_id, cluster, start_time) — sqlite_autoindex_job_1
```

## Query Plans WITH ANALYZE (verified working)
These are the plans SQLite ALREADY chooses with ANALYZE and current indexes:

1. **Default job list** (multi-state IN + ORDER BY start_time DESC LIMIT 50): → `SCAN jobs_starttime` ✓
2. **Cluster + state + sort start_time**: → `SEARCH jobs_cluster_starttime_duration (cluster=?)` ✓
3. **User + sort start_time**: → `SEARCH jobs_user_starttime_duration (hpc_user=?)` ✓
4. **Cluster + state aggregation**: → `SEARCH COVERING jobs_cluster_jobstate_duration_starttime` ✓
5. **Unique lookup (job_id, cluster, start_time)**: → `SEARCH sqlite_autoindex_job_1` ✓
6. **Running jobs for cluster**: → `SEARCH jobs_cluster_jobstate_duration_starttime (cluster=? AND job_state=? AND duration>?)` ✓
7. **Time range**: → `SEARCH jobs_starttime (start_time BETWEEN)` ✓ (temp b-tree for multi-col ORDER BY is acceptable)
8. **Group by user with cluster**: → `SEARCH COVERING jobs_cluster_user (cluster=?)` ✓
9. **Concurrent jobs**: → `SEARCH jobs_cluster_starttime_duration (cluster=? AND start_time<?)` ✓
10. **Project IN + state IN + sort**: → `SEARCH jobs_jobstate_project (job_state=? AND project=?)` + TEMP B-TREE ⚠
11. **User + multi-state + sort**: → `SEARCH jobs_user_starttime_duration (hpc_user=?)` ✓
12. **Cluster + state + sort duration**: → `SEARCH jobs_cluster_jobstate_duration_starttime (cluster=? AND job_state=?)` ✓
13. **Cluster + state + sort num_nodes**: → `SEARCH jobs_cluster_jobstate_numnodes (cluster=? AND job_state=?)` ✓
14. **Tag join**: → `SEARCH tags_tagid` + PK lookup ✓
15. **Delete before timestamp**: → `SEARCH COVERING jobs_starttime (start_time<?)` ✓
16. **Non-running jobs list**: → `SCAN COVERING jobs_jobstate_duration_starttime` ✓

## All Query Patterns (28 sites across 9 files)

**High-frequency patterns (every page load):**
- `QueryJobs`: WHERE=(any filter combo from BuildWhereClause) ORDER BY=(start_time|duration|num_nodes|num_hwthreads|num_acc|energy|footprint_json) LIMIT+OFFSET
- `CountJobs`: WHERE=(same filters) COUNT(DISTINCT job.id)
- Security filter always adds: `hpc_user = ?` (user role) or `project IN (...)` (manager role)

**BuildWhereClause filter columns (in priority order):**
1. `cluster` — almost always present (equality)
2. `job_state` — almost always present (IN list, typically 1-3 values)
3. `start_time` — very common (range or relative: last6h/24h/7d/30d)
4. `hpc_user` — added by security for user role (equality)
5. `project` — added by security for manager role (IN list)
6. `duration` — common filter (range)
7. `subcluster` — occasional (equality)
8. `cluster_partition` — occasional (equality)
9. `num_nodes`, `num_hwthreads`, `num_acc` — occasional (range)
10. `energy` — occasional (range)
11. `shared` — occasional (equality)
12. `array_job_id` — rare (equality)
13. JSON columns (footprint, meta_data, resources) — rare, can't be indexed traditionally

**ORDER BY columns (one at a time, ASC or DESC):**
- `start_time` (default, ~80% of queries)
- `duration` (~10%)
- `num_nodes`, `num_hwthreads`, `num_acc`, `energy` (~10% combined)
- JSON footprint fields (rare, always requires temp sort)

**Statistics/aggregation patterns:**
- GROUP BY: `hpc_user`, `project`, `cluster`, `subcluster`
- Aggregates: COUNT, SUM(duration), SUM(num_nodes * duration), etc.

**Lookup patterns:**
- `(job_id, cluster, start_time)` — covered by UNIQUE constraint
- `job.id` (PK) — always indexed

## Design the optimal reduced index set
Requirements:
1. **Target: 20-25 indexes max** (down from 79)
2. Every query pattern above must use an index (no full table scans unless LIMIT is low enough)
3. For each remaining index, explain which query patterns it serves
4. List every dropped index and explain why it's redundant
5. Use composite index prefix properties (an index on (A, B, C) also covers queries on (A) and (A, B))
6. Prioritize covering indexes for aggregation queries where practical
7. Consider that `start_time` is the dominant sort column

Format the output as a concrete migration SQL file with comments explaining each index.

Prompt:
Design an optimal index strategy for the `job` table in a SQLite database with 20 million rows. The current setup has 79 indexes which is excessive — it confuses the query planner, wastes disk space (~2x overhead per index on 20M rows), and slows inserts/updates.

## Current Problem
- 79 indexes on one table, many redundant
- SQLite query planner picks wrong indexes without ANALYZE (e.g., `jobs_jobstate_energy` instead of `jobs_jobstate_starttime_duration` for ORDER BY start_time queries)
- `IN` clauses with multiple values + `ORDER BY` causes temp B-tree sorts
- Timeouts on filter queries at 20M rows

## Key Constraints
1. **SQLite only** — no partial indexes with expressions on columns (but expression indexes on JSON are supported)
2. Each additional index costs ~200-400MB at 20M rows
3. The `ANALYZE` fix (already implemented) helps the planner, but we still need fewer, better indexes
4. Indexes must be defined in a migration file

## Job Table Columns
```
id (PK), job_id, cluster, subcluster, submit_time, start_time, hpc_user, project,
cluster_partition, array_job_id, duration, walltime, job_state, meta_data (JSON),
resources (JSON), num_nodes, num_hwthreads, num_acc, smt, shared, monitoring_status,
energy, energy_footprint (JSON), footprint (JSON)
UNIQUE(job_id, cluster, start_time) — sqlite_autoindex_job_1
```

## Query Plans WITH ANALYZE (verified working)
These are the plans SQLite ALREADY chooses with ANALYZE and current indexes:

1. **Default job list** (multi-state IN + ORDER BY start_time DESC LIMIT 50): → `SCAN jobs_starttime` ✓
2. **Cluster + state + sort start_time**: → `SEARCH jobs_cluster_starttime_duration (cluster=?)` ✓
3. **User + sort start_time**: → `SEARCH jobs_user_starttime_duration (hpc_user=?)` ✓
4. **Cluster + state aggregation**: → `SEARCH COVERING jobs_cluster_jobstate_duration_starttime` ✓
5. **Unique lookup (job_id, cluster, start_time)**: → `SEARCH sqlite_autoindex_job_1` ✓
6. **Running jobs for cluster**: → `SEARCH jobs_cluster_jobstate_duration_starttime (cluster=? AND job_state=? AND duration>?)` ✓
7. **Time range**: → `SEARCH jobs_starttime (start_time BETWEEN)` ✓ (temp b-tree for multi-col ORDER BY is acceptable)
8. **Group by user with cluster**: → `SEARCH COVERING jobs_cluster_user (cluster=?)` ✓
9. **Concurrent jobs**: → `SEARCH jobs_cluster_starttime_duration (cluster=? AND start_time<?)` ✓
10. **Project IN + state IN + sort**: → `SEARCH jobs_jobstate_project (job_state=? AND project=?)` + TEMP B-TREE ⚠
11. **User + multi-state + sort**: → `SEARCH jobs_user_starttime_duration (hpc_user=?)` ✓
12. **Cluster + state + sort duration**: → `SEARCH jobs_cluster_jobstate_duration_starttime (cluster=? AND job_state=?)` ✓
13. **Cluster + state + sort num_nodes**: → `SEARCH jobs_cluster_jobstate_numnodes (cluster=? AND job_state=?)` ✓
14. **Tag join**: → `SEARCH tags_tagid` + PK lookup ✓
15. **Delete before timestamp**: → `SEARCH COVERING jobs_starttime (start_time<?)` ✓
16. **Non-running jobs list**: → `SCAN COVERING jobs_jobstate_duration_starttime` ✓

## All Query Patterns (28 sites across 9 files)

**High-frequency patterns (every page load):**
- `QueryJobs`: WHERE=(any filter combo from BuildWhereClause) ORDER BY=(start_time|duration|num_nodes|num_hwthreads|num_acc|energy|footprint_json) LIMIT+OFFSET
- `CountJobs`: WHERE=(same filters) COUNT(DISTINCT job.id)
- Security filter always adds: `hpc_user = ?` (user role) or `project IN (...)` (manager role)

**BuildWhereClause filter columns (in priority order):**
1. `cluster` — almost always present (equality)
2. `job_state` — almost always present (IN list, typically 1-3 values)
3. `start_time` — very common (range or relative: last6h/24h/7d/30d)
4. `hpc_user` — added by security for user role (equality)
5. `project` — added by security for manager role (IN list)
6. `duration` — common filter (range)
7. `subcluster` — occasional (equality)
8. `cluster_partition` — occasional (equality)
9. `num_nodes`, `num_hwthreads`, `num_acc` — occasional (range)
10. `energy` — occasional (range)
11. `shared` — occasional (equality)
12. `array_job_id` — rare (equality)
13. JSON columns (footprint, meta_data, resources) — rare, can't be indexed traditionally

**ORDER BY columns (one at a time, ASC or DESC):**
- `start_time` (default, ~80% of queries)
- `duration` (~10%)
- `num_nodes`, `num_hwthreads`, `num_acc`, `energy` (~10% combined)
- JSON footprint fields (rare, always requires temp sort)

**Statistics/aggregation patterns:**
- GROUP BY: `hpc_user`, `project`, `cluster`, `subcluster`
- Aggregates: COUNT, SUM(duration), SUM(num_nodes * duration), etc.

**Lookup patterns:**
- `(job_id, cluster, start_time)` — covered by UNIQUE constraint
- `job.id` (PK) — always indexed

## Design the optimal reduced index set
Requirements:
1. **Target: 20-25 indexes max** (down from 79)
2. Every query pattern above must use an index (no full table scans unless LIMIT is low enough)
3. For each remaining index, explain which query patterns it serves
4. List every dropped index and explain why it's redundant
5. Use composite index prefix properties (an index on (A, B, C) also covers queries on (A) and (A, B))
6. Prioritize covering indexes for aggregation queries where practical
7. Consider that `start_time` is the dominant sort column

Format the output as a concrete migration SQL file with comments explaining each index.


     1→CREATE INDEX IF NOT EXISTS job_stats        ON job (cluster,subcluster,user);
     2→CREATE INDEX IF NOT EXISTS job_by_user      ON job (user);
     3→CREATE INDEX IF NOT EXISTS job_by_starttime ON job (start_time);
     4→CREATE INDEX IF NOT EXISTS job_by_job_id    ON job (job_id, cluster, start_time);
     5→CREATE INDEX IF NOT EXISTS job_list         ON job (cluster, job_state);
     6→CREATE INDEX IF NOT EXISTS job_list_user    ON job (user, cluster, job_state);
     7→CREATE INDEX IF NOT EXISTS job_list_users   ON job (user, job_state);
     8→CREATE INDEX IF NOT EXISTS job_list_users_start ON job (start_time, user, job_state);
     9→


     1→DROP INDEX IF EXISTS job_stats;
     2→DROP INDEX IF EXISTS job_by_user;
     3→DROP INDEX IF EXISTS job_by_starttime;
     4→DROP INDEX IF EXISTS job_by_job_id;
     5→DROP INDEX IF EXISTS job_list;
     6→DROP INDEX IF EXISTS job_list_user;
     7→DROP INDEX IF EXISTS job_list_users;
     8→DROP INDEX IF EXISTS job_list_users_start;
     9→
    10→ALTER TABLE job ADD COLUMN energy REAL NOT NULL DEFAULT 0.0;
    11→ALTER TABLE job ADD COLUMN energy_footprint TEXT DEFAULT NULL;
    12→
    13→ALTER TABLE job ADD COLUMN footprint TEXT DEFAULT NULL;
    14→ALTER TABLE tag ADD COLUMN tag_scope TEXT NOT NULL DEFAULT 'global';
    15→
    16→-- Do not use reserved keywords anymore
    17→ALTER TABLE "user" RENAME TO hpc_user;
    18→ALTER TABLE job RENAME COLUMN "user" TO hpc_user;
    19→ALTER TABLE job RENAME COLUMN "partition" TO cluster_partition;
    20→
    21→UPDATE job SET footprint = '{"flops_any_avg": 0.0}';
    22→UPDATE job SET footprint = json_replace(footprint, '$.flops_any_avg', job.flops_any_avg);
    23→UPDATE job SET footprint = json_insert(footprint, '$.mem_bw_avg', job.mem_bw_avg);
    24→UPDATE job SET footprint = json_insert(footprint, '$.mem_used_max', job.mem_used_max);
    25→UPDATE job SET footprint = json_insert(footprint, '$.cpu_load_avg', job.load_avg);
    26→UPDATE job SET footprint = json_insert(footprint, '$.net_bw_avg', job.net_bw_avg) WHERE job.net_bw_avg != 0;
    27→UPDATE job SET footprint = json_insert(footprint, '$.net_data_vol_total', job.net_data_vol_total) WHERE job.net_data_vol_total != 0;
    28→UPDATE job SET footprint = json_insert(footprint, '$.file_bw_avg', job.file_bw_avg) WHERE job.file_bw_avg != 0;
    29→UPDATE job SET footprint = json_insert(footprint, '$.file_data_vol_total', job.file_data_vol_total) WHERE job.file_data_vol_total != 0;
    30→
    31→ALTER TABLE job DROP flops_any_avg;
    32→ALTER TABLE job DROP mem_bw_avg;
    33→ALTER TABLE job DROP mem_used_max;
    34→ALTER TABLE job DROP load_avg;
    35→ALTER TABLE job DROP net_bw_avg;
    36→ALTER TABLE job DROP net_data_vol_total;
    37→ALTER TABLE job DROP file_bw_avg;
    38→ALTER TABLE job DROP file_data_vol_total;
    39→
    40→-- Indices for: Single filters, combined filters, sorting, sorting with filters
    41→-- Cluster Filter
    42→CREATE INDEX IF NOT EXISTS jobs_cluster ON job (cluster);
    43→CREATE INDEX IF NOT EXISTS jobs_cluster_user ON job (cluster, hpc_user);
    44→CREATE INDEX IF NOT EXISTS jobs_cluster_project ON job (cluster, project);
    45→CREATE INDEX IF NOT EXISTS jobs_cluster_subcluster ON job (cluster, subcluster);
    46→-- Cluster Filter Sorting
    47→CREATE INDEX IF NOT EXISTS jobs_cluster_starttime ON job (cluster, start_time);
    48→CREATE INDEX IF NOT EXISTS jobs_cluster_duration ON job (cluster, duration);
    49→CREATE INDEX IF NOT EXISTS jobs_cluster_numnodes ON job (cluster, num_nodes);
    50→CREATE INDEX IF NOT EXISTS jobs_cluster_numhwthreads ON job (cluster, num_hwthreads);
    51→CREATE INDEX IF NOT EXISTS jobs_cluster_numacc ON job (cluster, num_acc);
    52→CREATE INDEX IF NOT EXISTS jobs_cluster_energy ON job (cluster, energy);
    53→
    54→-- Cluster+Partition Filter
    55→CREATE INDEX IF NOT EXISTS jobs_cluster_partition ON job (cluster, cluster_partition);
    56→-- Cluster+Partition Filter Sorting
    57→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_starttime ON job (cluster, cluster_partition, start_time);
    58→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_duration ON job (cluster, cluster_partition, duration);
    59→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numnodes ON job (cluster, cluster_partition, num_nodes);
    60→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numhwthreads ON job (cluster, cluster_partition, num_hwthreads);
    61→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numacc ON job (cluster, cluster_partition, num_acc);
    62→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_energy ON job (cluster, cluster_partition, energy);
    63→
    64→-- Cluster+Partition+Jobstate Filter
    65→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate ON job (cluster, cluster_partition, job_state);
    66→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_user ON job (cluster, cluster_partition, job_state, hpc_user);
    67→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_project ON job (cluster, cluster_partition, job_state, project);
    68→-- Cluster+Partition+Jobstate Filter Sorting
    69→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_starttime ON job (cluster, cluster_partition, job_state, start_time);
    70→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_duration ON job (cluster, cluster_partition, job_state, duration);
    71→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_numnodes ON job (cluster, cluster_partition, job_state, num_nodes);
    72→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_numhwthreads ON job (cluster, cluster_partition, job_state, num_hwthreads);
    73→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_numacc ON job (cluster, cluster_partition, job_state, num_acc);
    74→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_energy ON job (cluster, cluster_partition, job_state, energy);
    75→
    76→-- Cluster+JobState Filter
    77→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate ON job (cluster, job_state);
    78→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user ON job (cluster, job_state, hpc_user);
    79→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project ON job (cluster, job_state, project);
    80→-- Cluster+JobState Filter Sorting
    81→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_starttime ON job (cluster, job_state, start_time);
    82→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_duration ON job (cluster, job_state, duration);
    83→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numnodes ON job (cluster, job_state, num_nodes);
    84→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numhwthreads ON job (cluster, job_state, num_hwthreads);
    85→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numacc ON job (cluster, job_state, num_acc);
    86→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_energy ON job (cluster, job_state, energy);
    87→
    88→-- User Filter
    89→CREATE INDEX IF NOT EXISTS jobs_user ON job (hpc_user);
    90→-- User Filter Sorting
    91→CREATE INDEX IF NOT EXISTS jobs_user_starttime ON job (hpc_user, start_time);
    92→CREATE INDEX IF NOT EXISTS jobs_user_duration ON job (hpc_user, duration);
    93→CREATE INDEX IF NOT EXISTS jobs_user_numnodes ON job (hpc_user, num_nodes);
    94→CREATE INDEX IF NOT EXISTS jobs_user_numhwthreads ON job (hpc_user, num_hwthreads);
    95→CREATE INDEX IF NOT EXISTS jobs_user_numacc ON job (hpc_user, num_acc);
    96→CREATE INDEX IF NOT EXISTS jobs_user_energy ON job (hpc_user, energy);
    97→
    98→-- Project Filter
    99→CREATE INDEX IF NOT EXISTS jobs_project ON job (project);
   100→CREATE INDEX IF NOT EXISTS jobs_project_user ON job (project, hpc_user);
   101→-- Project Filter Sorting
   102→CREATE INDEX IF NOT EXISTS jobs_project_starttime ON job (project, start_time);
   103→CREATE INDEX IF NOT EXISTS jobs_project_duration ON job (project, duration);
   104→CREATE INDEX IF NOT EXISTS jobs_project_numnodes ON job (project, num_nodes);
   105→CREATE INDEX IF NOT EXISTS jobs_project_numhwthreads ON job (project, num_hwthreads);
   106→CREATE INDEX IF NOT EXISTS jobs_project_numacc ON job (project, num_acc);
   107→CREATE INDEX IF NOT EXISTS jobs_project_energy ON job (project, energy);
   108→
   109→-- JobState Filter
   110→CREATE INDEX IF NOT EXISTS jobs_jobstate ON job (job_state);
   111→CREATE INDEX IF NOT EXISTS jobs_jobstate_user ON job (job_state, hpc_user);
   112→CREATE INDEX IF NOT EXISTS jobs_jobstate_project ON job (job_state, project);
   113→CREATE INDEX IF NOT EXISTS jobs_jobstate_cluster ON job (job_state, cluster);
   114→-- JobState Filter Sorting
   115→CREATE INDEX IF NOT EXISTS jobs_jobstate_starttime ON job (job_state, start_time);
   116→CREATE INDEX IF NOT EXISTS jobs_jobstate_duration ON job (job_state, duration);
   117→CREATE INDEX IF NOT EXISTS jobs_jobstate_numnodes ON job (job_state, num_nodes);
   118→CREATE INDEX IF NOT EXISTS jobs_jobstate_numhwthreads ON job (job_state, num_hwthreads);
   119→CREATE INDEX IF NOT EXISTS jobs_jobstate_numacc ON job (job_state, num_acc);
   120→CREATE INDEX IF NOT EXISTS jobs_jobstate_energy ON job (job_state, energy);
   121→
   122→-- ArrayJob Filter
   123→CREATE INDEX IF NOT EXISTS jobs_arrayjobid_starttime ON job (array_job_id, start_time);
   124→CREATE INDEX IF NOT EXISTS jobs_cluster_arrayjobid_starttime ON job (cluster, array_job_id, start_time);
   125→
   126→-- Sorting without active filters
   127→CREATE INDEX IF NOT EXISTS jobs_starttime ON job (start_time);
   128→CREATE INDEX IF NOT EXISTS jobs_duration ON job (duration);
   129→CREATE INDEX IF NOT EXISTS jobs_numnodes ON job (num_nodes);
   130→CREATE INDEX IF NOT EXISTS jobs_numhwthreads ON job (num_hwthreads);
   131→CREATE INDEX IF NOT EXISTS jobs_numacc ON job (num_acc);
   132→CREATE INDEX IF NOT EXISTS jobs_energy ON job (energy);
   133→
   134→-- Single filters with default starttime sorting
   135→CREATE INDEX IF NOT EXISTS jobs_duration_starttime ON job (duration, start_time);
   136→CREATE INDEX IF NOT EXISTS jobs_numnodes_starttime ON job (num_nodes, start_time);
   137→CREATE INDEX IF NOT EXISTS jobs_numhwthreads_starttime ON job (num_hwthreads, start_time);
   138→CREATE INDEX IF NOT EXISTS jobs_numacc_starttime ON job (num_acc, start_time);
   139→CREATE INDEX IF NOT EXISTS jobs_energy_starttime ON job (energy, start_time);
   140→
   141→-- Optimize DB index usage
   142→PRAGMA optimize;
   143→


     1→CREATE TABLE IF NOT EXISTS job (
     2→id                INTEGER PRIMARY KEY,
     3→job_id            BIGINT NOT NULL,
     4→cluster           VARCHAR(255) NOT NULL,
     5→subcluster        VARCHAR(255) NOT NULL,
     6→start_time        BIGINT NOT NULL, -- Unix timestamp
     7→user              VARCHAR(255) NOT NULL,
     8→project           VARCHAR(255) NOT NULL,
     9→partition         VARCHAR(255),
    10→array_job_id      BIGINT,
    11→duration          INT NOT NULL,
    12→walltime          INT NOT NULL,
    13→job_state         VARCHAR(255) NOT NULL
    14→CHECK(job_state IN ('running', 'completed', 'failed', 'canceled', 'stopped', 'timeout')),
    15→meta_data         TEXT,          -- JSON
    16→resources         TEXT NOT NULL, -- JSON
    17→num_nodes         INT NOT NULL,
    18→num_hwthreads     INT,
    19→num_acc           INT,
    20→smt               TINYINT NOT NULL DEFAULT 1 CHECK(smt               IN (0, 1   )),
    21→exclusive         TINYINT NOT NULL DEFAULT 1 CHECK(exclusive         IN (0, 1, 2)),
    22→monitoring_status TINYINT NOT NULL DEFAULT 1 CHECK(monitoring_status IN (0, 1, 2, 3)),
    23→mem_used_max        REAL NOT NULL DEFAULT 0.0,
    24→flops_any_avg       REAL NOT NULL DEFAULT 0.0,
    25→mem_bw_avg          REAL NOT NULL DEFAULT 0.0,
    26→load_avg            REAL NOT NULL DEFAULT 0.0,
    27→net_bw_avg          REAL NOT NULL DEFAULT 0.0,
    28→net_data_vol_total  REAL NOT NULL DEFAULT 0.0,
    29→file_bw_avg         REAL NOT NULL DEFAULT 0.0,
    30→file_data_vol_total REAL NOT NULL DEFAULT 0.0);
    31→
    32→CREATE TABLE IF NOT EXISTS tag (
    33→id       INTEGER PRIMARY KEY,
    34→tag_type VARCHAR(255) NOT NULL,
    35→tag_name VARCHAR(255) NOT NULL,
    36→UNIQUE (tag_type, tag_name));
    37→
    38→CREATE TABLE IF NOT EXISTS jobtag (
    39→job_id INTEGER,
    40→tag_id INTEGER,
    41→PRIMARY KEY (job_id, tag_id),
    42→FOREIGN KEY (job_id) REFERENCES job (id) ON DELETE CASCADE,
    43→FOREIGN KEY (tag_id) REFERENCES tag (id) ON DELETE CASCADE);
    44→
    45→CREATE TABLE IF NOT EXISTS user (
    46→username varchar(255) PRIMARY KEY NOT NULL,
    47→password varchar(255) DEFAULT NULL,
    48→ldap     tinyint      NOT NULL DEFAULT 0, /* col called "ldap" for historic reasons, fills the "AuthSource" */
    49→name     varchar(255) DEFAULT NULL,
    50→roles    varchar(255) NOT NULL DEFAULT "[]",
    51→email    varchar(255) DEFAULT NULL);
    52→
    53→CREATE TABLE IF NOT EXISTS configuration (
    54→username varchar(255),
    55→confkey  varchar(255),
    56→value    varchar(255),
    57→PRIMARY KEY (username, confkey),
    58→FOREIGN KEY (username) REFERENCES user (username) ON DELETE CASCADE ON UPDATE NO ACTION);
    59→
    60→
    61→


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package repository provides job query functionality with filtering, pagination,
     7→// and security controls. This file contains the main query builders and security
     8→// checks for job retrieval operations.
     9→package repository
    10→
    11→import (
    12→	"context"
    13→	"errors"
    14→	"fmt"
    15→	"regexp"
    16→	"strings"
    17→	"time"
    18→
    19→	"github.com/ClusterCockpit/cc-backend/internal/config"
    20→	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
    21→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    22→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    23→	sq "github.com/Masterminds/squirrel"
    24→)
    25→
    26→const (
    27→	// Default initial capacity for job result slices
    28→	defaultJobsCapacity = 50
    29→)
    30→
    31→// QueryJobs retrieves jobs from the database with optional filtering, pagination,
    32→// and sorting. Security controls are automatically applied based on the user context.
    33→//
    34→// Parameters:
    35→//   - ctx: Context containing user authentication information
    36→//   - filters: Optional job filters (cluster, state, user, time ranges, etc.)
    37→//   - page: Optional pagination parameters (page number and items per page)
    38→//   - order: Optional sorting specification (column or footprint field)
    39→//
    40→// Returns a slice of jobs matching the criteria, or an error if the query fails.
    41→// The function enforces role-based access control through SecurityCheck.
    42→func (r *JobRepository) QueryJobs(
    43→	ctx context.Context,
    44→	filters []*model.JobFilter,
    45→	page *model.PageRequest,
    46→	order *model.OrderByInput,
    47→) ([]*schema.Job, error) {
    48→	query, qerr := SecurityCheck(ctx, sq.Select(jobColumns...).From("job"))
    49→	if qerr != nil {
    50→		return nil, qerr
    51→	}
    52→
    53→	if order != nil {
    54→		field := toSnakeCase(order.Field)
    55→		if order.Type == "col" {
    56→			switch order.Order {
    57→			case model.SortDirectionEnumAsc:
    58→				query = query.OrderBy(fmt.Sprintf("job.%s ASC", field))
    59→			case model.SortDirectionEnumDesc:
    60→				query = query.OrderBy(fmt.Sprintf("job.%s DESC", field))
    61→			default:
    62→				return nil, errors.New("invalid sorting order for column")
    63→			}
    64→		} else {
    65→			// Order by footprint JSON field values
    66→			query = query.Where("JSON_VALID(footprint)")
    67→			switch order.Order {
    68→			case model.SortDirectionEnumAsc:
    69→				query = query.OrderBy(fmt.Sprintf("JSON_EXTRACT(footprint, \"$.%s\") ASC", field))
    70→			case model.SortDirectionEnumDesc:
    71→				query = query.OrderBy(fmt.Sprintf("JSON_EXTRACT(footprint, \"$.%s\") DESC", field))
    72→			default:
    73→				return nil, errors.New("invalid sorting order for footprint")
    74→			}
    75→		}
    76→	}
    77→
    78→	if page != nil && page.ItemsPerPage != -1 {
    79→		limit := uint64(page.ItemsPerPage)
    80→		query = query.Offset((uint64(page.Page) - 1) * limit).Limit(limit)
    81→	}
    82→
    83→	for _, f := range filters {
    84→		query = BuildWhereClause(f, query)
    85→	}
    86→
    87→	rows, err := query.RunWith(r.stmtCache).Query()
    88→	if err != nil {
    89→		queryString, queryVars, _ := query.ToSql()
    90→		return nil, fmt.Errorf("query failed [%s] %v: %w", queryString, queryVars, err)
    91→	}
    92→	defer rows.Close()
    93→
    94→	jobs := make([]*schema.Job, 0, defaultJobsCapacity)
    95→	for rows.Next() {
    96→		job, err := scanJob(rows)
    97→		if err != nil {
    98→			cclog.Warnf("Error scanning job row: %v", err)
    99→			return nil, fmt.Errorf("failed to scan job row: %w", err)
   100→		}
   101→		jobs = append(jobs, job)
   102→	}
   103→
   104→	if err := rows.Err(); err != nil {
   105→		return nil, fmt.Errorf("error iterating job rows: %w", err)
   106→	}
   107→
   108→	return jobs, nil
   109→}
   110→
   111→// CountJobs returns the total number of jobs matching the given filters.
   112→// Security controls are automatically applied based on the user context.
   113→// Uses DISTINCT count to handle tag filters correctly (jobs may appear multiple
   114→// times when joined with the tag table).
   115→func (r *JobRepository) CountJobs(
   116→	ctx context.Context,
   117→	filters []*model.JobFilter,
   118→) (int, error) {
   119→	query, qerr := SecurityCheck(ctx, sq.Select("count(DISTINCT job.id)").From("job"))
   120→	if qerr != nil {
   121→		return 0, qerr
   122→	}
   123→
   124→	for _, f := range filters {
   125→		query = BuildWhereClause(f, query)
   126→	}
   127→
   128→	var count int
   129→	if err := query.RunWith(r.DB).Scan(&count); err != nil {
   130→		return 0, fmt.Errorf("failed to count jobs: %w", err)
   131→	}
   132→
   133→	return count, nil
   134→}
   135→
   136→// SecurityCheckWithUser applies role-based access control filters to a job query
   137→// based on the provided user's roles and permissions.
   138→//
   139→// Access rules by role:
   140→//   - API role (exclusive): Full access to all jobs
   141→//   - Admin/Support roles: Full access to all jobs
   142→//   - Manager role: Access to jobs in managed projects plus own jobs
   143→//   - User role: Access only to own jobs
   144→//
   145→// Returns an error if the user is nil or has no recognized roles.
   146→func SecurityCheckWithUser(user *schema.User, query sq.SelectBuilder) (sq.SelectBuilder, error) {
   147→	if user == nil {
   148→		var qnil sq.SelectBuilder
   149→		return qnil, fmt.Errorf("user context is nil")
   150→	}
   151→
   152→	switch {
   153→	case len(user.Roles) == 1 && user.HasRole(schema.RoleAPI):
   154→		return query, nil
   155→	case user.HasAnyRole([]schema.Role{schema.RoleAdmin, schema.RoleSupport}):
   156→		return query, nil
   157→	case user.HasRole(schema.RoleManager):
   158→		if len(user.Projects) != 0 {
   159→			return query.Where(sq.Or{sq.Eq{"job.project": user.Projects}, sq.Eq{"job.hpc_user": user.Username}}), nil
   160→		}
   161→		cclog.Debugf("Manager '%s' has no assigned projects, restricting to personal jobs", user.Username)
   162→		return query.Where("job.hpc_user = ?", user.Username), nil
   163→	case user.HasRole(schema.RoleUser):
   164→		return query.Where("job.hpc_user = ?", user.Username), nil
   165→	default:
   166→		var qnil sq.SelectBuilder
   167→		return qnil, fmt.Errorf("user has no or unknown roles")
   168→	}
   169→}
   170→
   171→// SecurityCheck extracts the user from the context and applies role-based access
   172→// control filters to the query. This is a convenience wrapper around SecurityCheckWithUser.
   173→func SecurityCheck(ctx context.Context, query sq.SelectBuilder) (sq.SelectBuilder, error) {
   174→	user := GetUserFromContext(ctx)
   175→	return SecurityCheckWithUser(user, query)
   176→}
   177→
   178→// BuildWhereClause constructs SQL WHERE conditions from a JobFilter and applies
   179→// them to the query. Supports filtering by job properties (cluster, state, user),
   180→// time ranges, resource usage, tags, and JSON field searches in meta_data,
   181→// footprint, and resources columns.
   182→func BuildWhereClause(filter *model.JobFilter, query sq.SelectBuilder) sq.SelectBuilder {
   183→	// Primary Key
   184→	if filter.DbID != nil {
   185→		dbIDs := make([]string, len(filter.DbID))
   186→		copy(dbIDs, filter.DbID)
   187→		query = query.Where(sq.Eq{"job.id": dbIDs})
   188→	}
   189→	// Explicit indices
   190→	if filter.Cluster != nil {
   191→		query = buildStringCondition("job.cluster", filter.Cluster, query)
   192→	}
   193→	if filter.SubCluster != nil {
   194→		query = buildStringCondition("job.subcluster", filter.SubCluster, query)
   195→	}
   196→	if filter.Partition != nil {
   197→		query = buildStringCondition("job.cluster_partition", filter.Partition, query)
   198→	}
   199→	if filter.State != nil {
   200→		states := make([]string, len(filter.State))
   201→		for i, val := range filter.State {
   202→			states[i] = string(val)
   203→		}
   204→		query = query.Where(sq.Eq{"job.job_state": states})
   205→	}
   206→	if filter.Shared != nil {
   207→		query = query.Where("job.shared = ?", *filter.Shared)
   208→	}
   209→	if filter.Project != nil {
   210→		query = buildStringCondition("job.project", filter.Project, query)
   211→	}
   212→	if filter.User != nil {
   213→		query = buildStringCondition("job.hpc_user", filter.User, query)
   214→	}
   215→	if filter.NumNodes != nil {
   216→		query = buildIntCondition("job.num_nodes", filter.NumNodes, query)
   217→	}
   218→	if filter.NumAccelerators != nil {
   219→		query = buildIntCondition("job.num_acc", filter.NumAccelerators, query)
   220→	}
   221→	if filter.NumHWThreads != nil {
   222→		query = buildIntCondition("job.num_hwthreads", filter.NumHWThreads, query)
   223→	}
   224→	if filter.ArrayJobID != nil {
   225→		query = query.Where("job.array_job_id = ?", *filter.ArrayJobID)
   226→	}
   227→	if filter.StartTime != nil {
   228→		query = buildTimeCondition("job.start_time", filter.StartTime, query)
   229→	}
   230→	if filter.Duration != nil {
   231→		query = buildIntCondition("job.duration", filter.Duration, query)
   232→	}
   233→	if filter.Energy != nil {
   234→		query = buildFloatCondition("job.energy", filter.Energy, query)
   235→	}
   236→	// Indices on Tag Table
   237→	if filter.Tags != nil {
   238→		// This is an OR-Logic query: Returns all distinct jobs with at least one of the requested tags; TODO: AND-Logic query?
   239→		query = query.Join("jobtag ON jobtag.job_id = job.id").Where(sq.Eq{"jobtag.tag_id": filter.Tags}).Distinct()
   240→	}
   241→	// No explicit Indices
   242→	if filter.JobID != nil {
   243→		query = buildStringCondition("job.job_id", filter.JobID, query)
   244→	}
   245→	// Queries Within JSONs
   246→	if filter.MetricStats != nil {
   247→		for _, ms := range filter.MetricStats {
   248→			query = buildFloatJSONCondition(ms.MetricName, ms.Range, query)
   249→		}
   250→	}
   251→	if filter.Node != nil {
   252→		query = buildResourceJSONCondition("hostname", filter.Node, query)
   253→	}
   254→	if filter.JobName != nil {
   255→		query = buildMetaJSONCondition("jobName", filter.JobName, query)
   256→	}
   257→	if filter.Schedule != nil {
   258→		interactiveJobname := "interactive"
   259→		switch *filter.Schedule {
   260→		case "interactive":
   261→			iFilter := model.StringInput{Eq: &interactiveJobname}
   262→			query = buildMetaJSONCondition("jobName", &iFilter, query)
   263→		case "batch":
   264→			sFilter := model.StringInput{Neq: &interactiveJobname}
   265→			query = buildMetaJSONCondition("jobName", &sFilter, query)
   266→		}
   267→	}
   268→
   269→	// Configurable Filter to exclude recently started jobs, see config.go: ShortRunningJobsDuration
   270→	if filter.MinRunningFor != nil {
   271→		now := time.Now().Unix()
   272→		// Only jobs whose start timestamp is more than MinRunningFor seconds in the past
   273→		// If a job completed within the configured timeframe, it will still show up after the start_time matches the condition!
   274→		query = query.Where(sq.Lt{"job.start_time": (now - int64(*filter.MinRunningFor))})
   275→	}
   276→	return query
   277→}
   278→
   279→// buildIntCondition creates clauses for integer range filters, using BETWEEN only if required.
   280→func buildIntCondition(field string, cond *config.IntRange, query sq.SelectBuilder) sq.SelectBuilder {
   281→	if cond.From != 1 && cond.To != 0 {
   282→		return query.Where(field+" BETWEEN ? AND ?", cond.From, cond.To)
   283→	} else if cond.From != 1 && cond.To == 0 {
   284→		return query.Where(field+" >= ?", cond.From)
   285→	} else if cond.From == 1 && cond.To != 0 {
   286→		return query.Where(field+" <= ?", cond.To)
   287→	} else {
   288→		return query
   289→	}
   290→}
   291→
   292→// buildFloatCondition creates a clauses for float range filters, using BETWEEN only if required.
   293→func buildFloatCondition(field string, cond *model.FloatRange, query sq.SelectBuilder) sq.SelectBuilder {
   294→	if cond.From != 1.0 && cond.To != 0.0 {
   295→		return query.Where(field+" BETWEEN ? AND ?", cond.From, cond.To)
   296→	} else if cond.From != 1.0 && cond.To == 0.0 {
   297→		return query.Where(field+" >= ?", cond.From)
   298→	} else if cond.From == 1.0 && cond.To != 0.0 {
   299→		return query.Where(field+" <= ?", cond.To)
   300→	} else {
   301→		return query
   302→	}
   303→}
   304→
   305→// buildTimeCondition creates time range filters supporting absolute timestamps,
   306→// relative time ranges (last6h, last24h, last7d, last30d), or open-ended ranges.
   307→// Reminder: BETWEEN Queries are slower and dont use indices as frequently: Only use if both conditions required
   308→func buildTimeCondition(field string, cond *config.TimeRange, query sq.SelectBuilder) sq.SelectBuilder {
   309→	if cond.From != nil && cond.To != nil {
   310→		return query.Where(field+" BETWEEN ? AND ?", cond.From.Unix(), cond.To.Unix())
   311→	} else if cond.From != nil {
   312→		return query.Where(field+" >= ?", cond.From.Unix())
   313→	} else if cond.To != nil {
   314→		return query.Where(field+" <= ?", cond.To.Unix())
   315→	} else if cond.Range != "" {
   316→		now := time.Now().Unix()
   317→		var then int64
   318→		switch cond.Range {
   319→		case "last6h":
   320→			then = now - (60 * 60 * 6)
   321→		case "last24h":
   322→			then = now - (60 * 60 * 24)
   323→		case "last7d":
   324→			then = now - (60 * 60 * 24 * 7)
   325→		case "last30d":
   326→			then = now - (60 * 60 * 24 * 30)
   327→		default:
   328→			cclog.Debugf("No known named timeRange: startTime.range = %s", cond.Range)
   329→			return query
   330→		}
   331→		return query.Where(field+" >= ?", then)
   332→	} else {
   333→		return query
   334→	}
   335→}
   336→
   337→// buildFloatJSONCondition creates a filter on a numeric field within the footprint JSON column, using BETWEEN only if required.
   338→func buildFloatJSONCondition(jsonField string, cond *model.FloatRange, query sq.SelectBuilder) sq.SelectBuilder {
   339→	query = query.Where("JSON_VALID(footprint)")
   340→	if cond.From != 1.0 && cond.To != 0.0 {
   341→		return query.Where("JSON_EXTRACT(footprint, \"$."+jsonField+"\") BETWEEN ? AND ?", cond.From, cond.To)
   342→	} else if cond.From != 1.0 && cond.To == 0.0 {
   343→		return query.Where("JSON_EXTRACT(footprint, \"$."+jsonField+"\") >= ?", cond.From)
   344→	} else if cond.From == 1.0 && cond.To != 0.0 {
   345→		return query.Where("JSON_EXTRACT(footprint, \"$."+jsonField+"\") <= ?", cond.To)
   346→	} else {
   347→		return query
   348→	}
   349→}
   350→
   351→// buildStringCondition creates filters for string fields supporting equality,
   352→// inequality, prefix, suffix, substring, and IN list matching.
   353→func buildStringCondition(field string, cond *model.StringInput, query sq.SelectBuilder) sq.SelectBuilder {
   354→	if cond.Eq != nil {
   355→		return query.Where(field+" = ?", *cond.Eq)
   356→	}
   357→	if cond.Neq != nil {
   358→		return query.Where(field+" != ?", *cond.Neq)
   359→	}
   360→	if cond.StartsWith != nil {
   361→		return query.Where(field+" LIKE ?", fmt.Sprint(*cond.StartsWith, "%"))
   362→	}
   363→	if cond.EndsWith != nil {
   364→		return query.Where(field+" LIKE ?", fmt.Sprint("%", *cond.EndsWith))
   365→	}
   366→	if cond.Contains != nil {
   367→		return query.Where(field+" LIKE ?", fmt.Sprint("%", *cond.Contains, "%"))
   368→	}
   369→	if cond.In != nil {
   370→		queryElements := make([]string, len(cond.In))
   371→		copy(queryElements, cond.In)
   372→		return query.Where(sq.Or{sq.Eq{field: queryElements}})
   373→	}
   374→	return query
   375→}
   376→
   377→// buildMetaJSONCondition creates filters on fields within the meta_data JSON column.
   378→func buildMetaJSONCondition(jsonField string, cond *model.StringInput, query sq.SelectBuilder) sq.SelectBuilder {
   379→	query = query.Where("JSON_VALID(meta_data)")
   380→	if cond.Eq != nil {
   381→		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") = ?", *cond.Eq)
   382→	}
   383→	if cond.Neq != nil {
   384→		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") != ?", *cond.Neq)
   385→	}
   386→	if cond.StartsWith != nil {
   387→		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") LIKE ?", fmt.Sprint(*cond.StartsWith, "%"))
   388→	}
   389→	if cond.EndsWith != nil {
   390→		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") LIKE ?", fmt.Sprint("%", *cond.EndsWith))
   391→	}
   392→	if cond.Contains != nil {
   393→		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") LIKE ?", fmt.Sprint("%", *cond.Contains, "%"))
   394→	}
   395→	return query
   396→}
   397→
   398→// buildResourceJSONCondition creates filters on fields within the resources JSON array column.
   399→// Uses json_each to search within array elements.
   400→func buildResourceJSONCondition(jsonField string, cond *model.StringInput, query sq.SelectBuilder) sq.SelectBuilder {
   401→	query = query.Where("JSON_VALID(resources)")
   402→	if cond.Eq != nil {
   403→		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\") = ?)", *cond.Eq)
   404→	}
   405→	if cond.Neq != nil { // Currently Unused
   406→		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\") != ?)", *cond.Neq)
   407→	}
   408→	if cond.StartsWith != nil { // Currently Unused
   409→		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\")) LIKE ?)", fmt.Sprint(*cond.StartsWith, "%"))
   410→	}
   411→	if cond.EndsWith != nil { // Currently Unused
   412→		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\") LIKE ?)", fmt.Sprint("%", *cond.EndsWith))
   413→	}
   414→	if cond.Contains != nil {
   415→		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\") LIKE ?)", fmt.Sprint("%", *cond.Contains, "%"))
   416→	}
   417→	return query
   418→}
   419→
   420→var (
   421→	matchFirstCap = regexp.MustCompile("(.)([A-Z][a-z]+)")
   422→	matchAllCap   = regexp.MustCompile("([a-z0-9])([A-Z])")
   423→)
   424→
   425→// toSnakeCase converts camelCase strings to snake_case for SQL column names.
   426→// Includes security checks to prevent SQL injection attempts.
   427→// Panics if potentially dangerous characters are detected.
   428→func toSnakeCase(str string) string {
   429→	for _, c := range str {
   430→		if c == '\'' || c == '\\' || c == '"' || c == ';' || c == '-' || c == ' ' {
   431→			cclog.Panicf("toSnakeCase: potentially dangerous character detected in input: %q", str)
   432→		}
   433→	}
   434→
   435→	snake := matchFirstCap.ReplaceAllString(str, "${1}_${2}")
   436→	snake = matchAllCap.ReplaceAllString(snake, "${1}_${2}")
   437→	return strings.ToLower(snake)
   438→}
   439→


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// This file contains job statistics and histogram generation functionality for the JobRepository.
     7→//
     8→// # Job Statistics
     9→//
    10→// The statistics methods provide aggregated metrics about jobs including total jobs, users,
    11→// walltime, and resource usage (nodes, cores, accelerators). Statistics can be computed:
    12→//   - Overall (JobsStats): Single aggregate across all matching jobs
    13→//   - Grouped (JobsStatsGrouped): Aggregated by user, project, cluster, or subcluster
    14→//   - Counts (JobCountGrouped, AddJobCount): Simple job counts with optional filtering
    15→//
    16→// All statistics methods support filtering via JobFilter and respect security contexts.
    17→//
    18→// # Histograms
    19→//
    20→// Histogram methods generate distribution data for visualization:
    21→//   - Duration, nodes, cores, accelerators (AddHistograms)
    22→//   - Job metrics like CPU load, memory usage (AddMetricHistograms)
    23→//
    24→// Histograms use intelligent binning:
    25→//   - Duration: Variable bin sizes (1m, 10m, 1h, 6h, 12h, 24h) with zero-padding
    26→//   - Resources: Natural value-based bins
    27→//   - Metrics: Normalized to peak values with configurable bin counts
    28→//
    29→// # Running vs. Completed Jobs
    30→//
    31→// Statistics handle running jobs specially:
    32→//   - Duration calculated as (now - start_time) for running jobs
    33→//   - Metric histograms for running jobs load data from metric backend instead of footprint
    34→//   - Job state filtering distinguishes running/completed jobs
    35→//
    36→// # Performance Considerations
    37→//
    38→// - All queries use prepared statements via stmtCache
    39→// - Complex aggregations use SQL for efficiency
    40→// - Histogram pre-initialization ensures consistent bin ranges
    41→// - Metric histogram queries limited to 5000 jobs for running job analysis
    42→
    43→package repository
    44→
    45→import (
    46→	"context"
    47→	"database/sql"
    48→	"fmt"
    49→	"time"
    50→
    51→	"github.com/ClusterCockpit/cc-backend/internal/config"
    52→	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
    53→	"github.com/ClusterCockpit/cc-backend/internal/metricdispatch"
    54→	"github.com/ClusterCockpit/cc-backend/pkg/archive"
    55→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    56→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    57→	sq "github.com/Masterminds/squirrel"
    58→)
    59→
    60→// groupBy2column maps GraphQL Aggregate enum values to their corresponding database column names.
    61→// Used by JobsStatsGrouped and JobCountGrouped to translate user-facing grouping dimensions
    62→// into SQL GROUP BY clauses. GraphQL validation ensures only valid enum values are accepted.
    63→var groupBy2column = map[model.Aggregate]string{
    64→	model.AggregateUser:       "job.hpc_user",
    65→	model.AggregateProject:    "job.project",
    66→	model.AggregateCluster:    "job.cluster",
    67→	model.AggregateSubcluster: "job.subcluster",
    68→}
    69→
    70→// sortBy2column maps GraphQL SortByAggregate enum values to their corresponding computed column names.
    71→// Used by JobsStatsGrouped to translate sort preferences into SQL ORDER BY clauses.
    72→// Column names match the AS aliases used in buildStatsQuery.
    73→var sortBy2column = map[model.SortByAggregate]string{
    74→	model.SortByAggregateTotaljobs:      "totalJobs",
    75→	model.SortByAggregateTotalusers:     "totalUsers",
    76→	model.SortByAggregateTotalwalltime:  "totalWalltime",
    77→	model.SortByAggregateTotalnodes:     "totalNodes",
    78→	model.SortByAggregateTotalnodehours: "totalNodeHours",
    79→	model.SortByAggregateTotalcores:     "totalCores",
    80→	model.SortByAggregateTotalcorehours: "totalCoreHours",
    81→	model.SortByAggregateTotalaccs:      "totalAccs",
    82→	model.SortByAggregateTotalacchours:  "totalAccHours",
    83→}
    84→
    85→// buildCountQuery constructs a SQL query to count jobs with optional grouping and filtering.
    86→//
    87→// Parameters:
    88→//   - filter: Job filters to apply (cluster, user, time range, etc.)
    89→//   - kind: Special filter - "running" for running jobs only, "short" for jobs under threshold
    90→//   - col: Column name to GROUP BY; empty string for total count without grouping
    91→//
    92→// Returns a SelectBuilder that produces either:
    93→//   - Single count: COUNT(job.id) when col is empty
    94→//   - Grouped counts: col, COUNT(job.id) when col is specified
    95→//
    96→// The kind parameter enables counting specific job categories:
    97→//   - "running": Only jobs with job_state = 'running'
    98→//   - "short": Only jobs with duration < ShortRunningJobsDuration config value
    99→//   - empty: All jobs matching filters
   100→func (r *JobRepository) buildCountQuery(
   101→	filter []*model.JobFilter,
   102→	kind string,
   103→	col string,
   104→) sq.SelectBuilder {
   105→	var query sq.SelectBuilder
   106→
   107→	if col != "" {
   108→		query = sq.Select(col, "COUNT(job.id)").From("job").GroupBy(col)
   109→	} else {
   110→		query = sq.Select("COUNT(job.id)").From("job")
   111→	}
   112→
   113→	switch kind {
   114→	case "running":
   115→		query = query.Where("job.job_state = ?", "running")
   116→	case "short":
   117→		query = query.Where("job.duration < ?", config.Keys.ShortRunningJobsDuration)
   118→	}
   119→
   120→	for _, f := range filter {
   121→		query = BuildWhereClause(f, query)
   122→	}
   123→
   124→	return query
   125→}
   126→
   127→// buildStatsQuery constructs a SQL query to compute comprehensive job statistics with optional grouping.
   128→//
   129→// Parameters:
   130→//   - filter: Job filters to apply (cluster, user, time range, etc.)
   131→//   - col: Column name to GROUP BY; empty string for overall statistics without grouping
   132→//
   133→// Returns a SelectBuilder that produces comprehensive statistics:
   134→//   - totalJobs: Count of jobs
   135→//   - totalUsers: Count of distinct users (always 0 when grouping by user)
   136→//   - totalWalltime: Sum of job durations in hours
   137→//   - totalNodes: Sum of nodes used across all jobs
   138→//   - totalNodeHours: Sum of (duration × num_nodes) in hours
   139→//   - totalCores: Sum of hardware threads used across all jobs
   140→//   - totalCoreHours: Sum of (duration × num_hwthreads) in hours
   141→//   - totalAccs: Sum of accelerators used across all jobs
   142→//   - totalAccHours: Sum of (duration × num_acc) in hours
   143→//
   144→// Special handling:
   145→//   - Running jobs: Duration calculated as (now - start_time) instead of stored duration
   146→//   - Grouped queries: Also select grouping column and user's display name from hpc_user table
   147→//   - All time values converted from seconds to hours (÷ 3600) and rounded
   148→func (r *JobRepository) buildStatsQuery(
   149→	filter []*model.JobFilter,
   150→	col string,
   151→) sq.SelectBuilder {
   152→	var query sq.SelectBuilder
   153→
   154→	if col != "" {
   155→		query = sq.Select(
   156→			col,
   157→			"name",
   158→			"COUNT(job.id) as totalJobs",
   159→			"COUNT(DISTINCT job.hpc_user) AS totalUsers",
   160→			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END)) / 3600) as int) as totalWalltime`, time.Now().Unix()),
   161→			`CAST(SUM(job.num_nodes) as int) as totalNodes`,
   162→			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) * job.num_nodes) / 3600) as int) as totalNodeHours`, time.Now().Unix()),
   163→			`CAST(SUM(job.num_hwthreads) as int) as totalCores`,
   164→			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) * job.num_hwthreads) / 3600) as int) as totalCoreHours`, time.Now().Unix()),
   165→			`CAST(SUM(job.num_acc) as int) as totalAccs`,
   166→			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) * job.num_acc) / 3600) as int) as totalAccHours`, time.Now().Unix()),
   167→		).From("job").LeftJoin("hpc_user ON hpc_user.username = job.hpc_user").GroupBy(col)
   168→	} else {
   169→		query = sq.Select(
   170→			"COUNT(job.id) as totalJobs",
   171→			"COUNT(DISTINCT job.hpc_user) AS totalUsers",
   172→			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END)) / 3600) as int)`, time.Now().Unix()),
   173→			`CAST(SUM(job.num_nodes) as int)`,
   174→			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) * job.num_nodes) / 3600) as int)`, time.Now().Unix()),
   175→			`CAST(SUM(job.num_hwthreads) as int)`,
   176→			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) * job.num_hwthreads) / 3600) as int)`, time.Now().Unix()),
   177→			`CAST(SUM(job.num_acc) as int)`,
   178→			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) * job.num_acc) / 3600) as int)`, time.Now().Unix()),
   179→		).From("job")
   180→	}
   181→
   182→	for _, f := range filter {
   183→		query = BuildWhereClause(f, query)
   184→	}
   185→
   186→	return query
   187→}
   188→
   189→// JobsStatsGrouped computes comprehensive job statistics grouped by a dimension (user, project, cluster, or subcluster).
   190→//
   191→// This is the primary method for generating aggregated statistics views in the UI, providing
   192→// metrics like total jobs, walltime, and resource usage broken down by the specified grouping.
   193→//
   194→// Parameters:
   195→//   - ctx: Context for security checks and cancellation
   196→//   - filter: Filters to apply (time range, cluster, job state, etc.)
   197→//   - page: Optional pagination (ItemsPerPage: -1 disables pagination)
   198→//   - sortBy: Optional sort column (totalJobs, totalWalltime, totalCoreHours, etc.)
   199→//   - groupBy: Required grouping dimension (User, Project, Cluster, or SubCluster)
   200→//
   201→// Returns a slice of JobsStatistics, one per group, with:
   202→//   - ID: The group identifier (username, project name, cluster name, etc.)
   203→//   - Name: Display name (for users, from hpc_user.name; empty for other groups)
   204→//   - Statistics: totalJobs, totalUsers, totalWalltime, resource usage metrics
   205→//
   206→// Security: Respects user roles via SecurityCheck - users see only their own data unless admin/support.
   207→// Performance: Results are sorted in SQL and pagination applied before scanning rows.
   208→func (r *JobRepository) JobsStatsGrouped(
   209→	ctx context.Context,
   210→	filter []*model.JobFilter,
   211→	page *model.PageRequest,
   212→	sortBy *model.SortByAggregate,
   213→	groupBy *model.Aggregate,
   214→) ([]*model.JobsStatistics, error) {
   215→	start := time.Now()
   216→	col := groupBy2column[*groupBy]
   217→	query := r.buildStatsQuery(filter, col)
   218→
   219→	query, err := SecurityCheck(ctx, query)
   220→	if err != nil {
   221→		return nil, err
   222→	}
   223→
   224→	if sortBy != nil {
   225→		sortBy := sortBy2column[*sortBy]
   226→		query = query.OrderBy(fmt.Sprintf("%s DESC", sortBy))
   227→	}
   228→	if page != nil && page.ItemsPerPage != -1 {
   229→		limit := uint64(page.ItemsPerPage)
   230→		query = query.Offset((uint64(page.Page) - 1) * limit).Limit(limit)
   231→	}
   232→
   233→	rows, err := query.RunWith(r.DB).Query()
   234→	if err != nil {
   235→		cclog.Warn("Error while querying DB for job statistics")
   236→		return nil, err
   237→	}
   238→	defer rows.Close()
   239→
   240→	stats := make([]*model.JobsStatistics, 0, 100)
   241→
   242→	for rows.Next() {
   243→		var id sql.NullString
   244→		var name sql.NullString
   245→		var jobs, users, walltime, nodes, nodeHours, cores, coreHours, accs, accHours sql.NullInt64
   246→		if err := rows.Scan(&id, &name, &jobs, &users, &walltime, &nodes, &nodeHours, &cores, &coreHours, &accs, &accHours); err != nil {
   247→			cclog.Warnf("Error while scanning rows: %s", err.Error())
   248→			return nil, err
   249→		}
   250→
   251→		if id.Valid {
   252→			var totalJobs, totalUsers, totalWalltime, totalNodes, totalNodeHours, totalCores, totalCoreHours, totalAccs, totalAccHours int
   253→			var personName string
   254→
   255→			if name.Valid {
   256→				personName = name.String
   257→			}
   258→
   259→			if jobs.Valid {
   260→				totalJobs = int(jobs.Int64)
   261→			}
   262→
   263→			if users.Valid {
   264→				totalUsers = int(users.Int64)
   265→			}
   266→
   267→			if walltime.Valid {
   268→				totalWalltime = int(walltime.Int64)
   269→			}
   270→
   271→			if nodes.Valid {
   272→				totalNodes = int(nodes.Int64)
   273→			}
   274→			if cores.Valid {
   275→				totalCores = int(cores.Int64)
   276→			}
   277→			if accs.Valid {
   278→				totalAccs = int(accs.Int64)
   279→			}
   280→
   281→			if nodeHours.Valid {
   282→				totalNodeHours = int(nodeHours.Int64)
   283→			}
   284→			if coreHours.Valid {
   285→				totalCoreHours = int(coreHours.Int64)
   286→			}
   287→			if accHours.Valid {
   288→				totalAccHours = int(accHours.Int64)
   289→			}
   290→
   291→			if col == "job.hpc_user" {
   292→				// name := r.getUserName(ctx, id.String)
   293→				stats = append(stats,
   294→					&model.JobsStatistics{
   295→						ID:             id.String,
   296→						Name:           personName,
   297→						TotalJobs:      totalJobs,
   298→						TotalWalltime:  totalWalltime,
   299→						TotalNodes:     totalNodes,
   300→						TotalNodeHours: totalNodeHours,
   301→						TotalCores:     totalCores,
   302→						TotalCoreHours: totalCoreHours,
   303→						TotalAccs:      totalAccs,
   304→						TotalAccHours:  totalAccHours,
   305→					})
   306→			} else {
   307→				stats = append(stats,
   308→					&model.JobsStatistics{
   309→						ID:             id.String,
   310→						TotalJobs:      totalJobs,
   311→						TotalUsers:     totalUsers,
   312→						TotalWalltime:  totalWalltime,
   313→						TotalNodes:     totalNodes,
   314→						TotalNodeHours: totalNodeHours,
   315→						TotalCores:     totalCores,
   316→						TotalCoreHours: totalCoreHours,
   317→						TotalAccs:      totalAccs,
   318→						TotalAccHours:  totalAccHours,
   319→					})
   320→			}
   321→		}
   322→	}
   323→
   324→	if err := rows.Err(); err != nil {
   325→		return nil, err
   326→	}
   327→
   328→	cclog.Debugf("Timer JobsStatsGrouped %s", time.Since(start))
   329→	return stats, nil
   330→}
   331→
   332→// JobsStats computes overall job statistics across all matching jobs without grouping.
   333→//
   334→// This method provides a single aggregate view of job metrics, useful for dashboard
   335→// summaries and overall system utilization reports.
   336→//
   337→// Parameters:
   338→//   - ctx: Context for security checks and cancellation
   339→//   - filter: Filters to apply (time range, cluster, job state, etc.)
   340→//
   341→// Returns a single-element slice containing aggregate statistics:
   342→//   - totalJobs, totalUsers, totalWalltime
   343→//   - totalNodeHours, totalCoreHours, totalAccHours
   344→//
   345→// Unlike JobsStatsGrouped, this returns overall totals without breaking down by dimension.
   346→// Security checks are applied via SecurityCheck to respect user access levels.
   347→func (r *JobRepository) JobsStats(
   348→	ctx context.Context,
   349→	filter []*model.JobFilter,
   350→) ([]*model.JobsStatistics, error) {
   351→	start := time.Now()
   352→	query := r.buildStatsQuery(filter, "")
   353→	query, err := SecurityCheck(ctx, query)
   354→	if err != nil {
   355→		return nil, err
   356→	}
   357→
   358→	row := query.RunWith(r.DB).QueryRow()
   359→	stats := make([]*model.JobsStatistics, 0, 1)
   360→
   361→	var jobs, users, walltime, nodes, nodeHours, cores, coreHours, accs, accHours sql.NullInt64
   362→	if err := row.Scan(&jobs, &users, &walltime, &nodes, &nodeHours, &cores, &coreHours, &accs, &accHours); err != nil {
   363→		cclog.Warn("Error while scanning rows")
   364→		return nil, err
   365→	}
   366→
   367→	if jobs.Valid {
   368→		var totalNodeHours, totalCoreHours, totalAccHours int
   369→
   370→		if nodeHours.Valid {
   371→			totalNodeHours = int(nodeHours.Int64)
   372→		}
   373→		if coreHours.Valid {
   374→			totalCoreHours = int(coreHours.Int64)
   375→		}
   376→		if accHours.Valid {
   377→			totalAccHours = int(accHours.Int64)
   378→		}
   379→		stats = append(stats,
   380→			&model.JobsStatistics{
   381→				TotalJobs:      int(jobs.Int64),
   382→				TotalUsers:     int(users.Int64),
   383→				TotalWalltime:  int(walltime.Int64),
   384→				TotalNodeHours: totalNodeHours,
   385→				TotalCoreHours: totalCoreHours,
   386→				TotalAccHours:  totalAccHours,
   387→			})
   388→	}
   389→
   390→	cclog.Debugf("Timer JobStats %s", time.Since(start))
   391→	return stats, nil
   392→}
   393→
   394→// LoadJobStat retrieves a specific statistic for a metric from a job's statistics.
   395→// Returns 0.0 if the metric is not found or statType is invalid.
   396→//
   397→// Parameters:
   398→//   - job: Job struct with populated Statistics field
   399→//   - metric: Name of the metric to query (e.g., "cpu_load", "mem_used")
   400→//   - statType: Type of statistic: "avg", "min", or "max"
   401→//
   402→// Returns the requested statistic value or 0.0 if not found.
   403→func LoadJobStat(job *schema.Job, metric string, statType string) float64 {
   404→	if stats, ok := job.Statistics[metric]; ok {
   405→		switch statType {
   406→		case "avg":
   407→			return stats.Avg
   408→		case "max":
   409→			return stats.Max
   410→		case "min":
   411→			return stats.Min
   412→		default:
   413→			cclog.Errorf("Unknown stat type %s", statType)
   414→		}
   415→	}
   416→
   417→	return 0.0
   418→}
   419→
   420→// JobCountGrouped counts jobs grouped by a dimension without computing detailed statistics.
   421→//
   422→// This is a lightweight alternative to JobsStatsGrouped when only job counts are needed,
   423→// avoiding the overhead of calculating walltime and resource usage metrics.
   424→//
   425→// Parameters:
   426→//   - ctx: Context for security checks
   427→//   - filter: Filters to apply
   428→//   - groupBy: Grouping dimension (User, Project, Cluster, or SubCluster)
   429→//
   430→// Returns JobsStatistics with only ID and TotalJobs populated for each group.
   431→func (r *JobRepository) JobCountGrouped(
   432→	ctx context.Context,
   433→	filter []*model.JobFilter,
   434→	groupBy *model.Aggregate,
   435→) ([]*model.JobsStatistics, error) {
   436→	start := time.Now()
   437→	col := groupBy2column[*groupBy]
   438→	query := r.buildCountQuery(filter, "", col)
   439→	query, err := SecurityCheck(ctx, query)
   440→	if err != nil {
   441→		return nil, err
   442→	}
   443→	rows, err := query.RunWith(r.DB).Query()
   444→	if err != nil {
   445→		cclog.Warn("Error while querying DB for job statistics")
   446→		return nil, err
   447→	}
   448→	defer rows.Close()
   449→
   450→	stats := make([]*model.JobsStatistics, 0, 100)
   451→
   452→	for rows.Next() {
   453→		var id sql.NullString
   454→		var cnt sql.NullInt64
   455→		if err := rows.Scan(&id, &cnt); err != nil {
   456→			cclog.Warn("Error while scanning rows")
   457→			return nil, err
   458→		}
   459→		if id.Valid {
   460→			stats = append(stats,
   461→				&model.JobsStatistics{
   462→					ID:        id.String,
   463→					TotalJobs: int(cnt.Int64),
   464→				})
   465→		}
   466→	}
   467→
   468→	if err := rows.Err(); err != nil {
   469→		return nil, err
   470→	}
   471→
   472→	cclog.Debugf("Timer JobCountGrouped %s", time.Since(start))
   473→	return stats, nil
   474→}
   475→
   476→// AddJobCountGrouped augments existing statistics with additional job counts by category.
   477→//
   478→// This method enriches JobsStatistics returned by JobsStatsGrouped or JobCountGrouped
   479→// with counts of running or short-running (based on ShortRunningJobsDuration) jobs, matched by group ID.
   480→//
   481→// Parameters:
   482→//   - ctx: Context for security checks
   483→//   - filter: Filters to apply
   484→//   - groupBy: Grouping dimension (must match the dimension used for stats parameter)
   485→//   - stats: Existing statistics to augment (modified in-place by ID matching)
   486→//   - kind: "running" to add RunningJobs count, "short" to add ShortJobs count
   487→//
   488→// Returns the same stats slice with RunningJobs or ShortJobs fields populated per group.
   489→// Groups without matching jobs will have 0 for the added field.
   490→func (r *JobRepository) AddJobCountGrouped(
   491→	ctx context.Context,
   492→	filter []*model.JobFilter,
   493→	groupBy *model.Aggregate,
   494→	stats []*model.JobsStatistics,
   495→	kind string,
   496→) ([]*model.JobsStatistics, error) {
   497→	start := time.Now()
   498→	col := groupBy2column[*groupBy]
   499→	query := r.buildCountQuery(filter, kind, col)
   500→	query, err := SecurityCheck(ctx, query)
   501→	if err != nil {
   502→		return nil, err
   503→	}
   504→	rows, err := query.RunWith(r.DB).Query()
   505→	if err != nil {
   506→		cclog.Warn("Error while querying DB for job statistics")
   507→		return nil, err
   508→	}
   509→	defer rows.Close()
   510→
   511→	counts := make(map[string]int)
   512→
   513→	for rows.Next() {
   514→		var id sql.NullString
   515→		var cnt sql.NullInt64
   516→		if err := rows.Scan(&id, &cnt); err != nil {
   517→			cclog.Warn("Error while scanning rows")
   518→			return nil, err
   519→		}
   520→		if id.Valid {
   521→			counts[id.String] = int(cnt.Int64)
   522→		}
   523→	}
   524→
   525→	if err := rows.Err(); err != nil {
   526→		return nil, err
   527→	}
   528→
   529→	switch kind {
   530→	case "running":
   531→		for _, s := range stats {
   532→			s.RunningJobs = counts[s.ID]
   533→		}
   534→	case "short":
   535→		for _, s := range stats {
   536→			s.ShortJobs = counts[s.ID]
   537→		}
   538→	}
   539→
   540→	cclog.Debugf("Timer AddJobCountGrouped %s", time.Since(start))
   541→	return stats, nil
   542→}
   543→
   544→// AddJobCount augments existing overall statistics with additional job counts by category.
   545→//
   546→// Similar to AddJobCountGrouped but for ungrouped statistics. Applies the same count
   547→// to all statistics entries (typically just one).
   548→//
   549→// Parameters:
   550→//   - ctx: Context for security checks
   551→//   - filter: Filters to apply
   552→//   - stats: Existing statistics to augment (modified in-place)
   553→//   - kind: "running" to add RunningJobs count, "short" to add ShortJobs count
   554→//
   555→// Returns the same stats slice with RunningJobs or ShortJobs fields set to the total count.
   556→func (r *JobRepository) AddJobCount(
   557→	ctx context.Context,
   558→	filter []*model.JobFilter,
   559→	stats []*model.JobsStatistics,
   560→	kind string,
   561→) ([]*model.JobsStatistics, error) {
   562→	start := time.Now()
   563→	query := r.buildCountQuery(filter, kind, "")
   564→	query, err := SecurityCheck(ctx, query)
   565→	if err != nil {
   566→		return nil, err
   567→	}
   568→	var cnt sql.NullInt64
   569→	if err := query.RunWith(r.DB).QueryRow().Scan(&cnt); err != nil {
   570→		cclog.Warn("Error while querying DB for job count")
   571→		return nil, err
   572→	}
   573→
   574→	count := int(cnt.Int64)
   575→
   576→	switch kind {
   577→	case "running":
   578→		for _, s := range stats {
   579→			s.RunningJobs = count
   580→		}
   581→	case "short":
   582→		for _, s := range stats {
   583→			s.ShortJobs = count
   584→		}
   585→	}
   586→
   587→	cclog.Debugf("Timer AddJobCount %s", time.Since(start))
   588→	return stats, nil
   589→}
   590→
   591→// AddHistograms augments statistics with distribution histograms for job properties.
   592→//
   593→// Generates histogram data for visualization of job duration, node count, core count,
   594→// and accelerator count distributions. Duration histogram uses intelligent binning based
   595→// on the requested resolution.
   596→//
   597→// Parameters:
   598→//   - ctx: Context for security checks
   599→//   - filter: Filters to apply to jobs included in histograms
   600→//   - stat: Statistics struct to augment (modified in-place)
   601→//   - durationBins: Bin size - "1m", "10m", "1h", "6h", "12h", or "24h" (default)
   602→//
   603→// Populates these fields in stat:
   604→//   - HistDuration: Job duration distribution (zero-padded bins)
   605→//   - HistNumNodes: Node count distribution
   606→//   - HistNumCores: Core (hwthread) count distribution
   607→//   - HistNumAccs: Accelerator count distribution
   608→//
   609→// Duration bins are pre-initialized with zeros to ensure consistent ranges for visualization.
   610→// Bin size determines both the width and maximum duration displayed (e.g., "1h" = 48 bins × 1h = 48h max).
   611→func (r *JobRepository) AddHistograms(
   612→	ctx context.Context,
   613→	filter []*model.JobFilter,
   614→	stat *model.JobsStatistics,
   615→	durationBins *string,
   616→) (*model.JobsStatistics, error) {
   617→	start := time.Now()
   618→
   619→	var targetBinCount int
   620→	var targetBinSize int
   621→	switch *durationBins {
   622→	case "1m": // 1 Minute Bins + Max 60 Bins -> Max 60 Minutes
   623→		targetBinCount = 60
   624→		targetBinSize = 60
   625→	case "10m": // 10 Minute Bins + Max 72 Bins -> Max 12 Hours
   626→		targetBinCount = 72
   627→		targetBinSize = 600
   628→	case "1h": // 1 Hour Bins + Max 48 Bins -> Max 48 Hours
   629→		targetBinCount = 48
   630→		targetBinSize = 3600
   631→	case "6h": // 6 Hour Bins + Max 12 Bins -> Max 3 Days
   632→		targetBinCount = 12
   633→		targetBinSize = 21600
   634→	case "12h": // 12 hour Bins + Max 14 Bins -> Max 7 Days
   635→		targetBinCount = 14
   636→		targetBinSize = 43200
   637→	default: // 24h
   638→		targetBinCount = 24
   639→		targetBinSize = 3600
   640→	}
   641→
   642→	var err error
   643→	// Return X-Values always as seconds, will be formatted into minutes and hours in frontend
   644→	value := fmt.Sprintf(`CAST(ROUND(((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) / %d) + 1) as int) as value`, time.Now().Unix(), targetBinSize)
   645→	stat.HistDuration, err = r.jobsDurationStatisticsHistogram(ctx, value, filter, targetBinSize, &targetBinCount)
   646→	if err != nil {
   647→		cclog.Warn("Error while loading job statistics histogram: job duration")
   648→		return nil, err
   649→	}
   650→
   651→	stat.HistNumNodes, err = r.jobsStatisticsHistogram(ctx, "job.num_nodes as value", filter)
   652→	if err != nil {
   653→		cclog.Warn("Error while loading job statistics histogram: num nodes")
   654→		return nil, err
   655→	}
   656→
   657→	stat.HistNumCores, err = r.jobsStatisticsHistogram(ctx, "job.num_hwthreads as value", filter)
   658→	if err != nil {
   659→		cclog.Warn("Error while loading job statistics histogram: num hwthreads")
   660→		return nil, err
   661→	}
   662→
   663→	stat.HistNumAccs, err = r.jobsStatisticsHistogram(ctx, "job.num_acc as value", filter)
   664→	if err != nil {
   665→		cclog.Warn("Error while loading job statistics histogram: num acc")
   666→		return nil, err
   667→	}
   668→
   669→	cclog.Debugf("Timer AddHistograms %s", time.Since(start))
   670→	return stat, nil
   671→}
   672→
   673→// AddMetricHistograms augments statistics with distribution histograms for job metrics.
   674→//
   675→// Generates histogram data for metrics like CPU load, memory usage, etc. Handles running
   676→// and completed jobs differently: running jobs load data from metric backend, completed jobs
   677→// use footprint data from database.
   678→//
   679→// Parameters:
   680→//   - ctx: Context for security checks
   681→//   - filter: Filters to apply (MUST contain State filter for running jobs)
   682→//   - metrics: List of metric names to histogram (e.g., ["cpu_load", "mem_used"])
   683→//   - stat: Statistics struct to augment (modified in-place)
   684→//   - targetBinCount: Number of histogram bins (default: 10)
   685→//
   686→// Populates HistMetrics field in stat with MetricHistoPoints for each metric.
   687→//
   688→// Binning algorithm:
   689→//   - Values normalized to metric's peak value from cluster configuration
   690→//   - Bins evenly distributed from 0 to peak
   691→//   - Pre-initialized with zeros for consistent visualization
   692→//
   693→// Limitations:
   694→//   - Running jobs: Limited to 5000 jobs for performance
   695→//   - Requires valid cluster configuration with metric peak values
   696→//   - Uses footprint statistic (avg/max/min) configured per metric
   697→func (r *JobRepository) AddMetricHistograms(
   698→	ctx context.Context,
   699→	filter []*model.JobFilter,
   700→	metrics []string,
   701→	stat *model.JobsStatistics,
   702→	targetBinCount *int,
   703→) (*model.JobsStatistics, error) {
   704→	start := time.Now()
   705→
   706→	// Running Jobs Only: First query jobdata from sqlite, then query data and make bins
   707→	for _, f := range filter {
   708→		if f.State != nil {
   709→			if len(f.State) == 1 && f.State[0] == "running" {
   710→				stat.HistMetrics = r.runningJobsMetricStatisticsHistogram(ctx, metrics, filter, targetBinCount)
   711→				cclog.Debugf("Timer AddMetricHistograms %s", time.Since(start))
   712→				return stat, nil
   713→			}
   714→		}
   715→	}
   716→
   717→	// All other cases: Query and make bins in sqlite directly
   718→	for _, m := range metrics {
   719→		metricHisto, err := r.jobsMetricStatisticsHistogram(ctx, m, filter, targetBinCount)
   720→		if err != nil {
   721→			cclog.Warnf("Error while loading job metric statistics histogram: %s", m)
   722→			continue
   723→		}
   724→		stat.HistMetrics = append(stat.HistMetrics, metricHisto)
   725→	}
   726→
   727→	cclog.Debugf("Timer AddMetricHistograms %s", time.Since(start))
   728→	return stat, nil
   729→}
   730→
   731→// jobsStatisticsHistogram generates a simple histogram by grouping on a column value.
   732→//
   733→// Used for histograms where the column value directly represents the bin (e.g., node count, core count).
   734→// Unlike duration/metric histograms, this doesn't pre-initialize bins with zeros.
   735→//
   736→// Parameters:
   737→//   - value: SQL expression that produces the histogram value, aliased as "value"
   738→//   - filters: Job filters to apply
   739→//
   740→// Returns histogram points with Value (from column) and Count (number of jobs).
   741→func (r *JobRepository) jobsStatisticsHistogram(
   742→	ctx context.Context,
   743→	value string,
   744→	filters []*model.JobFilter,
   745→) ([]*model.HistoPoint, error) {
   746→	start := time.Now()
   747→	query, qerr := SecurityCheck(ctx,
   748→		sq.Select(value, "COUNT(job.id) AS count").From("job"))
   749→
   750→	if qerr != nil {
   751→		return nil, qerr
   752→	}
   753→
   754→	for _, f := range filters {
   755→		query = BuildWhereClause(f, query)
   756→	}
   757→
   758→	rows, err := query.GroupBy("value").RunWith(r.DB).Query()
   759→	if err != nil {
   760→		cclog.Error("Error while running query")
   761→		return nil, err
   762→	}
   763→	defer rows.Close()
   764→
   765→	points := make([]*model.HistoPoint, 0)
   766→	// is it possible to introduce zero values here? requires info about bincount
   767→	for rows.Next() {
   768→		point := model.HistoPoint{}
   769→		if err := rows.Scan(&point.Value, &point.Count); err != nil {
   770→			cclog.Warn("Error while scanning rows")
   771→			return nil, err
   772→		}
   773→
   774→		points = append(points, &point)
   775→	}
   776→
   777→	if err := rows.Err(); err != nil {
   778→		return nil, err
   779→	}
   780→
   781→	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
   782→	return points, nil
   783→}
   784→
   785→// jobsDurationStatisticsHistogram generates a duration histogram with pre-initialized bins.
   786→//
   787→// Bins are zero-padded to provide consistent ranges for visualization, unlike simple
   788→// histograms which only return bins with data. The value parameter should compute
   789→// the bin number from job duration.
   790→//
   791→// Parameters:
   792→//   - value: SQL expression computing bin number from duration, aliased as "value"
   793→//   - filters: Job filters to apply
   794→//   - binSizeSeconds: Width of each bin in seconds
   795→//   - targetBinCount: Number of bins to pre-initialize
   796→//
   797→// Returns histogram points with Value (bin_number × binSizeSeconds) and Count.
   798→// All bins from 1 to targetBinCount are returned, with Count=0 for empty bins.
   799→//
   800→// Algorithm:
   801→//  1. Pre-initialize targetBinCount bins with zero counts
   802→//  2. Query database for actual counts per bin
   803→//  3. Match query results to pre-initialized bins by value
   804→//  4. Bins without matches remain at zero
   805→func (r *JobRepository) jobsDurationStatisticsHistogram(
   806→	ctx context.Context,
   807→	value string,
   808→	filters []*model.JobFilter,
   809→	binSizeSeconds int,
   810→	targetBinCount *int,
   811→) ([]*model.HistoPoint, error) {
   812→	start := time.Now()
   813→	query, qerr := SecurityCheck(ctx,
   814→		sq.Select(value, "COUNT(job.id) AS count").From("job"))
   815→
   816→	if qerr != nil {
   817→		return nil, qerr
   818→	}
   819→
   820→	// Each bin represents a duration range: bin N = [N*binSizeSeconds, (N+1)*binSizeSeconds)
   821→	// Example: binSizeSeconds=3600 (1 hour), bin 1 = 0-1h, bin 2 = 1-2h, etc.
   822→	points := make([]*model.HistoPoint, 0)
   823→	for i := 1; i <= *targetBinCount; i++ {
   824→		point := model.HistoPoint{Value: i * binSizeSeconds, Count: 0}
   825→		points = append(points, &point)
   826→	}
   827→
   828→	for _, f := range filters {
   829→		query = BuildWhereClause(f, query)
   830→	}
   831→
   832→	rows, err := query.GroupBy("value").RunWith(r.DB).Query()
   833→	if err != nil {
   834→		cclog.Error("Error while running query")
   835→		return nil, err
   836→	}
   837→	defer rows.Close()
   838→
   839→	// Match query results to pre-initialized bins.
   840→	// point.Value from query is the bin number; multiply by binSizeSeconds to match bin.Value.
   841→	for rows.Next() {
   842→		point := model.HistoPoint{}
   843→		if err := rows.Scan(&point.Value, &point.Count); err != nil {
   844→			cclog.Warn("Error while scanning rows")
   845→			return nil, err
   846→		}
   847→
   848→		for _, e := range points {
   849→			if e.Value == (point.Value * binSizeSeconds) {
   850→				e.Count = point.Count
   851→				break
   852→			}
   853→		}
   854→	}
   855→
   856→	if err := rows.Err(); err != nil {
   857→		return nil, err
   858→	}
   859→
   860→	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
   861→	return points, nil
   862→}
   863→
   864→// jobsMetricStatisticsHistogram generates a metric histogram using footprint data from completed jobs.
   865→//
   866→// Values are normalized to the metric's peak value and distributed into bins. The algorithm
   867→// is based on SQL histogram generation techniques, extracting metric values from JSON footprint
   868→// and computing bin assignments in SQL.
   869→//
   870→// Parameters:
   871→//   - metric: Metric name (e.g., "cpu_load", "mem_used")
   872→//   - filters: Job filters to apply
   873→//   - bins: Number of bins to generate
   874→//
   875→// Returns MetricHistoPoints with metric name, unit, footprint stat type, and binned data.
   876→//
   877→// Algorithm:
   878→//  1. Determine peak value from cluster configuration (filtered cluster or max across all)
   879→//  2. Generate SQL that extracts footprint value, normalizes to [0,1], multiplies by bin count
   880→//  3. Pre-initialize bins with min/max ranges based on peak value
   881→//  4. Query database for counts per bin
   882→//  5. Match results to pre-initialized bins
   883→//
   884→// Special handling: Values exactly equal to peak are forced into the last bin by multiplying
   885→// peak by 0.999999999 to avoid creating an extra bin.
   886→func (r *JobRepository) jobsMetricStatisticsHistogram(
   887→	ctx context.Context,
   888→	metric string,
   889→	filters []*model.JobFilter,
   890→	bins *int,
   891→) (*model.MetricHistoPoints, error) {
   892→	// Peak value defines the upper bound for binning: values are distributed across
   893→	// bins from 0 to peak. First try to get peak from filtered cluster, otherwise
   894→	// scan all clusters to find the maximum peak value.
   895→	var metricConfig *schema.MetricConfig
   896→	var peak float64
   897→	var unit string
   898→	var footprintStat string
   899→
   900→	// Try to get metric config from filtered cluster
   901→	for _, f := range filters {
   902→		if f.Cluster != nil {
   903→			metricConfig = archive.GetMetricConfig(*f.Cluster.Eq, metric)
   904→			peak = metricConfig.Peak
   905→			unit = metricConfig.Unit.Prefix + metricConfig.Unit.Base
   906→			footprintStat = metricConfig.Footprint
   907→			cclog.Debugf("Cluster %s filter found with peak %f for %s", *f.Cluster.Eq, peak, metric)
   908→		}
   909→	}
   910→
   911→	// If no cluster filter or peak not found, find largest peak across all clusters
   912→	// This ensures histogram can accommodate all possible values
   913→	if peak == 0.0 {
   914→		for _, c := range archive.Clusters {
   915→			for _, m := range c.MetricConfig {
   916→				if m.Name == metric {
   917→					if m.Peak > peak {
   918→						peak = m.Peak
   919→					}
   920→					if unit == "" {
   921→						unit = m.Unit.Prefix + m.Unit.Base
   922→					}
   923→					if footprintStat == "" {
   924→						footprintStat = m.Footprint
   925→					}
   926→				}
   927→			}
   928→		}
   929→	}
   930→
   931→	// Construct SQL histogram bins using normalized values.
   932→	// Algorithm based on: https://jereze.com/code/sql-histogram/ (modified)
   933→	start := time.Now()
   934→
   935→	// Bin calculation formula:
   936→	//   bin_number = CAST( (value / peak) * num_bins AS INTEGER ) + 1
   937→	// Special case: value == peak would create bin N+1, so we test for equality
   938→	// and multiply peak by 0.999999999 to force it into bin N.
   939→	binQuery := fmt.Sprintf(`CAST(
   940→		((case when json_extract(footprint, "$.%s") = %f then %f*0.999999999 else json_extract(footprint, "$.%s") end) / %f)
   941→		* %v as INTEGER )`,
   942→		(metric + "_" + footprintStat), peak, peak, (metric + "_" + footprintStat), peak, *bins)
   943→
   944→	mainQuery := sq.Select(
   945→		fmt.Sprintf(`%s + 1 as bin`, binQuery),
   946→		`count(*) as count`,
   947→	).From("job").Where(
   948→		"JSON_VALID(footprint)",
   949→	).Where(fmt.Sprintf(`json_extract(footprint, "$.%s") is not null and json_extract(footprint, "$.%s") <= %f`, (metric + "_" + footprintStat), (metric + "_" + footprintStat), peak))
   950→
   951→	mainQuery, qerr := SecurityCheck(ctx, mainQuery)
   952→	if qerr != nil {
   953→		return nil, qerr
   954→	}
   955→
   956→	for _, f := range filters {
   957→		mainQuery = BuildWhereClause(f, mainQuery)
   958→	}
   959→
   960→	mainQuery = mainQuery.GroupBy("bin").OrderBy("bin")
   961→
   962→	rows, err := mainQuery.RunWith(r.DB).Query()
   963→	if err != nil {
   964→		cclog.Errorf("Error while running mainQuery: %s", err)
   965→		return nil, err
   966→	}
   967→	defer rows.Close()
   968→
   969→	// Pre-initialize bins with calculated min/max ranges.
   970→	// Example: peak=1000, bins=10 -> bin 1=[0,100), bin 2=[100,200), ..., bin 10=[900,1000]
   971→	points := make([]*model.MetricHistoPoint, 0)
   972→	binStep := int(peak) / *bins
   973→	for i := 1; i <= *bins; i++ {
   974→		binMin := (binStep * (i - 1))
   975→		binMax := (binStep * i)
   976→		epoint := model.MetricHistoPoint{Bin: &i, Count: 0, Min: &binMin, Max: &binMax}
   977→		points = append(points, &epoint)
   978→	}
   979→
   980→	// Match query results to pre-initialized bins.
   981→	for rows.Next() {
   982→		rpoint := model.MetricHistoPoint{}
   983→		if err := rows.Scan(&rpoint.Bin, &rpoint.Count); err != nil {
   984→			cclog.Warnf("Error while scanning rows for %s", metric)
   985→			return nil, err
   986→		}
   987→
   988→		for _, e := range points {
   989→			if e.Bin != nil && rpoint.Bin != nil && *e.Bin == *rpoint.Bin {
   990→				e.Count = rpoint.Count
   991→				break
   992→			}
   993→		}
   994→	}
   995→
   996→	if err := rows.Err(); err != nil {
   997→		return nil, err
   998→	}
   999→
  1000→	result := model.MetricHistoPoints{Metric: metric, Unit: unit, Stat: &footprintStat, Data: points}
  1001→
  1002→	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
  1003→	return &result, nil
  1004→}
  1005→
  1006→// runningJobsMetricStatisticsHistogram generates metric histograms for running jobs using live data.
  1007→//
  1008→// Unlike completed jobs which use footprint data from the database, running jobs require
  1009→// fetching current metric averages from the metric backend (via metricdispatch).
  1010→//
  1011→// Parameters:
  1012→//   - metrics: List of metric names
  1013→//   - filters: Job filters (should filter to running jobs only)
  1014→//   - bins: Number of histogram bins
  1015→//
  1016→// Returns slice of MetricHistoPoints, one per metric.
  1017→//
  1018→// Limitations:
  1019→//   - Maximum 5000 jobs (returns nil if more jobs match)
  1020→//   - Requires metric backend availability
  1021→//   - Bins based on metric peak values from cluster configuration
  1022→//
  1023→// Algorithm:
  1024→//  1. Query first 5001 jobs to check count limit
  1025→//  2. Load metric averages for all jobs via metricdispatch
  1026→//  3. For each metric, create bins based on peak value
  1027→//  4. Iterate averages and count jobs per bin
  1028→func (r *JobRepository) runningJobsMetricStatisticsHistogram(
  1029→	ctx context.Context,
  1030→	metrics []string,
  1031→	filters []*model.JobFilter,
  1032→	bins *int,
  1033→) []*model.MetricHistoPoints {
  1034→	// Get Jobs
  1035→	jobs, err := r.QueryJobs(ctx, filters, &model.PageRequest{Page: 1, ItemsPerPage: 5000 + 1}, nil)
  1036→	if err != nil {
  1037→		cclog.Errorf("Error while querying jobs for footprint: %s", err)
  1038→		return nil
  1039→	}
  1040→	if len(jobs) > 5000 {
  1041→		cclog.Errorf("too many jobs matched (max: %d)", 5000)
  1042→		return nil
  1043→	}
  1044→
  1045→	// Get AVGs from metric repo
  1046→	avgs := make([][]schema.Float, len(metrics))
  1047→	for i := range avgs {
  1048→		avgs[i] = make([]schema.Float, 0, len(jobs))
  1049→	}
  1050→
  1051→	for _, job := range jobs {
  1052→		if job.MonitoringStatus == schema.MonitoringStatusDisabled || job.MonitoringStatus == schema.MonitoringStatusArchivingFailed {
  1053→			continue
  1054→		}
  1055→
  1056→		if err := metricdispatch.LoadAverages(job, metrics, avgs, ctx); err != nil {
  1057→			cclog.Errorf("Error while loading averages for histogram: %s", err)
  1058→			return nil
  1059→		}
  1060→	}
  1061→
  1062→	// Iterate metrics to fill endresult
  1063→	data := make([]*model.MetricHistoPoints, 0)
  1064→	for idx, metric := range metrics {
  1065→		// Get specific Peak or largest Peak
  1066→		var metricConfig *schema.MetricConfig
  1067→		var peak float64
  1068→		var unit string
  1069→
  1070→		for _, f := range filters {
  1071→			if f.Cluster != nil {
  1072→				metricConfig = archive.GetMetricConfig(*f.Cluster.Eq, metric)
  1073→				peak = metricConfig.Peak
  1074→				unit = metricConfig.Unit.Prefix + metricConfig.Unit.Base
  1075→			}
  1076→		}
  1077→
  1078→		if peak == 0.0 {
  1079→			for _, c := range archive.Clusters {
  1080→				for _, m := range c.MetricConfig {
  1081→					if m.Name == metric {
  1082→						if m.Peak > peak {
  1083→							peak = m.Peak
  1084→						}
  1085→						if unit == "" {
  1086→							unit = m.Unit.Prefix + m.Unit.Base
  1087→						}
  1088→					}
  1089→				}
  1090→			}
  1091→		}
  1092→
  1093→		// Make and fill bins
  1094→		peakBin := int(peak) / *bins
  1095→
  1096→		points := make([]*model.MetricHistoPoint, 0)
  1097→		for b := 0; b < *bins; b++ {
  1098→			count := 0
  1099→			bindex := b + 1
  1100→			bmin := peakBin * b
  1101→			bmax := peakBin * (b + 1)
  1102→
  1103→			// Iterate AVG values for indexed metric and count for bins
  1104→			for _, val := range avgs[idx] {
  1105→				if int(val) >= bmin && int(val) < bmax {
  1106→					count += 1
  1107→				}
  1108→			}
  1109→
  1110→			// Append Bin to Metric Result Array
  1111→			point := model.MetricHistoPoint{Bin: &bindex, Count: count, Min: &bmin, Max: &bmax}
  1112→			points = append(points, &point)
  1113→		}
  1114→
  1115→		// Append Metric Result Array to final results array
  1116→		result := model.MetricHistoPoints{Metric: metric, Unit: unit, Data: points}
  1117→		data = append(data, &result)
  1118→	}
  1119→
  1120→	return data
  1121→}
  1122→