Implement the following plan:

# Fix: buildStatsQuery hangs on `/monitoring/users/?cluster=woody&startTime=last30d`

## Context

The users list page (`/monitoring/users/?cluster=woody&startTime=last30d`) executes a `GROUP BY job.hpc_user` stats query with `WHERE job.cluster = ? AND job.start_time >= ?`. On production databases with hundreds of thousands of jobs in the last 30 days, this query never returns.

**Root cause**: The covering index from migration 12 has column order `(cluster, start_time, hpc_user, ...)`. Since `start_time` is a **range** condition before `hpc_user`, SQLite cannot use index ordering for `GROUP BY hpc_user`. It must collect all matching rows into a **temp B-Tree** and sort them — extremely expensive for large result sets.

## Changes

### 1. Reorder covering index columns (critical fix)

**File**: `internal/repository/migrations/sqlite3/12_stats-covering-index.up.sql`

Change from: `(cluster, start_time, hpc_user, duration, job_state, num_nodes, num_hwthreads, num_acc)`
Change to: `(cluster, hpc_user, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc)`

This lets SQLite:
- Match `cluster = ?` (equality prefix)
- Walk `hpc_user` in index order (GROUP BY without temp B-Tree)
- Within each `(cluster, hpc_user)` group, binary-search `start_time >= ?`
- Read all aggregation columns from the index (covering)

Add `DROP INDEX IF EXISTS jobs_cluster_starttime_user_stats` to handle databases that already applied the old migration 12.

**File**: `internal/repository/migrations/sqlite3/12_stats-covering-index.down.sql`

Update to drop the renamed index `jobs_cluster_user_starttime_stats`.

### 2. Use `COUNT(*)` instead of `COUNT(job.id)`

**File**: `internal/repository/stats.go`

Lines to change: 108, 110, 156, 751, 817

`COUNT(*)` is idiomatic for counting rows and avoids any ambiguity about whether the rowid is available in a covering index scan.

### 3. Fix double-quoted string literals in CASE expressions

**File**: `internal/repository/stats.go`

Lines 147, 647: Change `job.job_state = "running"` to `job.job_state = 'running'`

Double quotes are SQL identifiers, not string literals. SQLite accepts them as a compatibility quirk but proper single quotes are safer for the query planner.

## Not changing

- **`jobs_cluster_starttime` index**: Still needed for job list queries (`ORDER BY start_time`).
- **`jobs_cluster_user` index**: Technically redundant with new covering index prefix, but harmless. Can clean up later.
- **Migration version**: Stays at 12 (same migration number, updated content).

## Verification

```bash
# Run existing tests
go test -v ./internal/repository -run TestJobStats
go test -v ./internal/repository -run TestBuildJobStatsQuery

# On production DB copy, verify query plan no longer shows temp B-Tree:
# EXPLAIN QUERY PLAN SELECT job.hpc_user, COUNT(*) ... FROM job
#   WHERE job.cluster = 'woody' AND job.start_time >= <30d_ago> GROUP BY job.hpc_user;
# Before: "USE TEMP B-TREE FOR GROUP BY"
# After: direct index scan, no temp B-Tree
```

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-ClusterCockpit-cc-backend/3f83174b-0ec7-4466-8bc9-b6b52eb8bbfb.jsonl


Let me read the files to modify.


     1→-- Migration 12: Add covering index for grouped stats queries
     2→-- Covers the common pattern: WHERE cluster=X AND start_time >= Y GROUP BY hpc_user
     3→-- Includes aggregated columns to avoid main table lookups entirely.
     4→
     5→CREATE INDEX IF NOT EXISTS jobs_cluster_starttime_user_stats
     6→  ON job (cluster, start_time, hpc_user, duration, job_state, num_nodes, num_hwthreads, num_acc);
     7→
     8→PRAGMA optimize;
     9→

-- Migration 12: Add covering index for grouped stats queries
-- Covers the common pattern: WHERE cluster=X AND start_time >= Y GROUP BY hpc_user
-- Includes aggregated columns to avoid main table lookups entirely.

CREATE INDEX IF NOT EXISTS jobs_cluster_starttime_user_stats
  ON job (cluster, start_time, hpc_user, duration, job_state, num_nodes, num_hwthreads, num_acc);

PRAGMA optimize;


     1→DROP INDEX IF EXISTS jobs_cluster_starttime_user_stats;
     2→


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// This file contains job statistics and histogram generation functionality for the JobRepository.
     7→//
     8→// # Job Statistics
     9→//
    10→// The statistics methods provide aggregated metrics about jobs including total jobs, users,
    11→// walltime, and resource usage (nodes, cores, accelerators). Statistics can be computed:
    12→//   - Overall (JobsStats): Single aggregate across all matching jobs
    13→//   - Grouped (JobsStatsGrouped): Aggregated by user, project, cluster, or subcluster
    14→//   - Counts (JobCountGrouped, AddJobCount): Simple job counts with optional filtering
    15→//
    16→// All statistics methods support filtering via JobFilter and respect security contexts.
    17→//
    18→// # Histograms
    19→//
    20→// Histogram methods generate distribution data for visualization:
    21→//   - Duration, nodes, cores, accelerators (AddHistograms)
    22→//   - Job metrics like CPU load, memory usage (AddMetricHistograms)
    23→//
    24→// Histograms use intelligent binning:
    25→//   - Duration: Variable bin sizes (1m, 10m, 1h, 6h, 12h, 24h) with zero-padding
    26→//   - Resources: Natural value-based bins
    27→//   - Metrics: Normalized to peak values with configurable bin counts
    28→//
    29→// # Running vs. Completed Jobs
    30→//
    31→// Statistics handle running jobs specially:
    32→//   - Duration calculated as (now - start_time) for running jobs
    33→//   - Metric histograms for running jobs load data from metric backend instead of footprint
    34→//   - Job state filtering distinguishes running/completed jobs
    35→//
    36→// # Performance Considerations
    37→//
    38→// - All queries use prepared statements via stmtCache
    39→// - Complex aggregations use SQL for efficiency
    40→// - Histogram pre-initialization ensures consistent bin ranges
    41→// - Metric histogram queries limited to 5000 jobs for running job analysis
    42→
    43→package repository
    44→
    45→import (
    46→	"context"
    47→	"database/sql"
    48→	"fmt"
    49→	"time"
    50→
    51→	"github.com/ClusterCockpit/cc-backend/internal/config"
    52→	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
    53→	"github.com/ClusterCockpit/cc-backend/internal/metricdispatch"
    54→	"github.com/ClusterCockpit/cc-backend/pkg/archive"
    55→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    56→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    57→	sq "github.com/Masterminds/squirrel"
    58→)
    59→
    60→// groupBy2column maps GraphQL Aggregate enum values to their corresponding database column names.
    61→// Used by JobsStatsGrouped and JobCountGrouped to translate user-facing grouping dimensions
    62→// into SQL GROUP BY clauses. GraphQL validation ensures only valid enum values are accepted.
    63→var groupBy2column = map[model.Aggregate]string{
    64→	model.AggregateUser:       "job.hpc_user",
    65→	model.AggregateProject:    "job.project",
    66→	model.AggregateCluster:    "job.cluster",
    67→	model.AggregateSubcluster: "job.subcluster",
    68→}
    69→
    70→// sortBy2column maps GraphQL SortByAggregate enum values to their corresponding computed column names.
    71→// Used by JobsStatsGrouped to translate sort preferences into SQL ORDER BY clauses.
    72→// Column names match the AS aliases used in buildStatsQuery.
    73→var sortBy2column = map[model.SortByAggregate]string{
    74→	model.SortByAggregateTotaljobs:      "totalJobs",
    75→	model.SortByAggregateTotalusers:     "totalUsers",
    76→	model.SortByAggregateTotalwalltime:  "totalWalltime",
    77→	model.SortByAggregateTotalnodes:     "totalNodes",
    78→	model.SortByAggregateTotalnodehours: "totalNodeHours",
    79→	model.SortByAggregateTotalcores:     "totalCores",
    80→	model.SortByAggregateTotalcorehours: "totalCoreHours",
    81→	model.SortByAggregateTotalaccs:      "totalAccs",
    82→	model.SortByAggregateTotalacchours:  "totalAccHours",
    83→}
    84→
    85→// buildCountQuery constructs a SQL query to count jobs with optional grouping and filtering.
    86→//
    87→// Parameters:
    88→//   - filter: Job filters to apply (cluster, user, time range, etc.)
    89→//   - kind: Special filter - "running" for running jobs only, "short" for jobs under threshold
    90→//   - col: Column name to GROUP BY; empty string for total count without grouping
    91→//
    92→// Returns a SelectBuilder that produces either:
    93→//   - Single count: COUNT(job.id) when col is empty
    94→//   - Grouped counts: col, COUNT(job.id) when col is specified
    95→//
    96→// The kind parameter enables counting specific job categories:
    97→//   - "running": Only jobs with job_state = 'running'
    98→//   - "short": Only jobs with duration < ShortRunningJobsDuration config value
    99→//   - empty: All jobs matching filters
   100→func (r *JobRepository) buildCountQuery(
   101→	filter []*model.JobFilter,
   102→	kind string,
   103→	col string,
   104→) sq.SelectBuilder {
   105→	var query sq.SelectBuilder
   106→
   107→	if col != "" {
   108→		query = sq.Select(col, "COUNT(job.id)").From("job").GroupBy(col)
   109→	} else {
   110→		query = sq.Select("COUNT(job.id)").From("job")
   111→	}
   112→
   113→	switch kind {
   114→	case "running":
   115→		query = query.Where("job.job_state = ?", "running")
   116→	case "short":
   117→		query = query.Where("job.duration < ?", config.Keys.ShortRunningJobsDuration)
   118→	}
   119→
   120→	for _, f := range filter {
   121→		query = BuildWhereClause(f, query)
   122→	}
   123→
   124→	return query
   125→}
   126→
   127→// buildStatsQuery constructs a SQL query to compute job statistics with optional grouping.
   128→// Only requested columns are computed; unrequested columns select 0 as placeholder.
   129→//
   130→// Parameters:
   131→//   - filter: Job filters to apply (cluster, user, time range, etc.)
   132→//   - col: Column name to GROUP BY; empty string for overall statistics without grouping
   133→//   - shortThreshold: Duration threshold in seconds for counting short-running jobs
   134→//   - reqFields: Set of requested field names; nil means compute all fields
   135→func (r *JobRepository) buildStatsQuery(
   136→	filter []*model.JobFilter,
   137→	col string,
   138→	shortThreshold int,
   139→	reqFields map[string]bool,
   140→) sq.SelectBuilder {
   141→	now := time.Now().Unix()
   142→
   143→	// Helper: return real expression if field is requested (or reqFields is nil), else "0 as alias"
   144→	need := func(field string) bool {
   145→		return reqFields == nil || reqFields[field]
   146→	}
   147→	durationExpr := fmt.Sprintf(`(CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END)`, now)
   148→
   149→	// Build column list
   150→	columns := make([]string, 0, 14)
   151→
   152→	if col != "" {
   153→		columns = append(columns, col)
   154→	}
   155→
   156→	columns = append(columns, "COUNT(job.id) as totalJobs")
   157→
   158→	if need("totalUsers") && col != "job.hpc_user" {
   159→		columns = append(columns, "COUNT(DISTINCT job.hpc_user) AS totalUsers")
   160→	} else {
   161→		columns = append(columns, "0 AS totalUsers")
   162→	}
   163→
   164→	if need("totalWalltime") {
   165→		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s) / 3600) as int) as totalWalltime`, durationExpr))
   166→	} else {
   167→		columns = append(columns, "0 as totalWalltime")
   168→	}
   169→
   170→	if need("totalNodes") {
   171→		columns = append(columns, `CAST(SUM(job.num_nodes) as int) as totalNodes`)
   172→	} else {
   173→		columns = append(columns, "0 as totalNodes")
   174→	}
   175→
   176→	if need("totalNodeHours") {
   177→		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s * job.num_nodes) / 3600) as int) as totalNodeHours`, durationExpr))
   178→	} else {
   179→		columns = append(columns, "0 as totalNodeHours")
   180→	}
   181→
   182→	if need("totalCores") {
   183→		columns = append(columns, `CAST(SUM(job.num_hwthreads) as int) as totalCores`)
   184→	} else {
   185→		columns = append(columns, "0 as totalCores")
   186→	}
   187→
   188→	if need("totalCoreHours") {
   189→		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s * job.num_hwthreads) / 3600) as int) as totalCoreHours`, durationExpr))
   190→	} else {
   191→		columns = append(columns, "0 as totalCoreHours")
   192→	}
   193→
   194→	if need("totalAccs") {
   195→		columns = append(columns, `CAST(SUM(job.num_acc) as int) as totalAccs`)
   196→	} else {
   197→		columns = append(columns, "0 as totalAccs")
   198→	}
   199→
   200→	if need("totalAccHours") {
   201→		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s * job.num_acc) / 3600) as int) as totalAccHours`, durationExpr))
   202→	} else {
   203→		columns = append(columns, "0 as totalAccHours")
   204→	}
   205→
   206→	if need("runningJobs") {
   207→		columns = append(columns, `COUNT(CASE WHEN job.job_state = 'running' THEN 1 END) as runningJobs`)
   208→	} else {
   209→		columns = append(columns, "0 as runningJobs")
   210→	}
   211→
   212→	if need("shortJobs") {
   213→		columns = append(columns, fmt.Sprintf(`COUNT(CASE WHEN job.duration < %d THEN 1 END) as shortJobs`, shortThreshold))
   214→	} else {
   215→		columns = append(columns, "0 as shortJobs")
   216→	}
   217→
   218→	query := sq.Select(columns...).From("job")
   219→	if col != "" {
   220→		query = query.GroupBy(col)
   221→	}
   222→
   223→	for _, f := range filter {
   224→		query = BuildWhereClause(f, query)
   225→	}
   226→
   227→	return query
   228→}
   229→
   230→// JobsStatsGrouped computes job statistics grouped by a dimension (user, project, cluster, or subcluster).
   231→// Only columns listed in reqFields are computed; others return 0. User display names are looked up
   232→// in a separate lightweight query to avoid JOIN overhead on the main aggregation.
   233→func (r *JobRepository) JobsStatsGrouped(
   234→	ctx context.Context,
   235→	filter []*model.JobFilter,
   236→	page *model.PageRequest,
   237→	sortBy *model.SortByAggregate,
   238→	groupBy *model.Aggregate,
   239→	reqFields map[string]bool,
   240→) ([]*model.JobsStatistics, error) {
   241→	start := time.Now()
   242→	col := groupBy2column[*groupBy]
   243→	query := r.buildStatsQuery(filter, col, config.Keys.ShortRunningJobsDuration, reqFields)
   244→
   245→	query, err := SecurityCheck(ctx, query)
   246→	if err != nil {
   247→		return nil, err
   248→	}
   249→
   250→	if sortBy != nil {
   251→		sortBy := sortBy2column[*sortBy]
   252→		query = query.OrderBy(fmt.Sprintf("%s DESC", sortBy))
   253→	}
   254→	if page != nil && page.ItemsPerPage != -1 {
   255→		limit := uint64(page.ItemsPerPage)
   256→		query = query.Offset((uint64(page.Page) - 1) * limit).Limit(limit)
   257→	}
   258→
   259→	rows, err := query.RunWith(r.DB).QueryContext(ctx)
   260→	if err != nil {
   261→		cclog.Warn("Error while querying DB for job statistics")
   262→		return nil, err
   263→	}
   264→	defer rows.Close()
   265→
   266→	stats := make([]*model.JobsStatistics, 0, 100)
   267→
   268→	for rows.Next() {
   269→		var id sql.NullString
   270→		var jobs, users, walltime, nodes, nodeHours, cores, coreHours, accs, accHours, runningJobs, shortJobs sql.NullInt64
   271→		if err := rows.Scan(&id, &jobs, &users, &walltime, &nodes, &nodeHours, &cores, &coreHours, &accs, &accHours, &runningJobs, &shortJobs); err != nil {
   272→			cclog.Warnf("Error while scanning rows: %s", err.Error())
   273→			return nil, err
   274→		}
   275→
   276→		if id.Valid {
   277→			stats = append(stats,
   278→				&model.JobsStatistics{
   279→					ID:             id.String,
   280→					TotalJobs:      int(jobs.Int64),
   281→					TotalUsers:     int(users.Int64),
   282→					TotalWalltime:  int(walltime.Int64),
   283→					TotalNodes:     int(nodes.Int64),
   284→					TotalNodeHours: int(nodeHours.Int64),
   285→					TotalCores:     int(cores.Int64),
   286→					TotalCoreHours: int(coreHours.Int64),
   287→					TotalAccs:      int(accs.Int64),
   288→					TotalAccHours:  int(accHours.Int64),
   289→					RunningJobs:    int(runningJobs.Int64),
   290→					ShortJobs:      int(shortJobs.Int64),
   291→				})
   292→		}
   293→	}
   294→
   295→	if err := rows.Err(); err != nil {
   296→		return nil, err
   297→	}
   298→
   299→	// Post-query name lookup for user grouping (avoids LEFT JOIN on aggregation query)
   300→	if col == "job.hpc_user" && len(stats) > 0 {
   301→		usernames := make([]any, len(stats))
   302→		for i, s := range stats {
   303→			usernames[i] = s.ID
   304→		}
   305→
   306→		nameQuery := sq.Select("username", "name").From("hpc_user").Where(sq.Eq{"username": usernames})
   307→		nameRows, err := nameQuery.RunWith(r.DB).QueryContext(ctx)
   308→		if err != nil {
   309→			cclog.Warnf("Error looking up user names: %s", err.Error())
   310→			// Non-fatal: stats are still valid without display names
   311→		} else {
   312→			defer nameRows.Close()
   313→			nameMap := make(map[string]string, len(stats))
   314→			for nameRows.Next() {
   315→				var username, name string
   316→				if err := nameRows.Scan(&username, &name); err == nil {
   317→					nameMap[username] = name
   318→				}
   319→			}
   320→			for _, s := range stats {
   321→				if name, ok := nameMap[s.ID]; ok {
   322→					s.Name = name
   323→				}
   324→			}
   325→		}
   326→	}
   327→
   328→	cclog.Debugf("Timer JobsStatsGrouped %s", time.Since(start))
   329→	return stats, nil
   330→}
   331→
   332→// JobsStats computes overall job statistics across all matching jobs without grouping.
   333→//
   334→// This method provides a single aggregate view of job metrics, useful for dashboard
   335→// summaries and overall system utilization reports.
   336→//
   337→// Parameters:
   338→//   - ctx: Context for security checks and cancellation
   339→//   - filter: Filters to apply (time range, cluster, job state, etc.)
   340→//
   341→// Returns a single-element slice containing aggregate statistics:
   342→//   - totalJobs, totalUsers, totalWalltime
   343→//   - totalNodeHours, totalCoreHours, totalAccHours
   344→//
   345→// Unlike JobsStatsGrouped, this returns overall totals without breaking down by dimension.
   346→// Security checks are applied via SecurityCheck to respect user access levels.
   347→func (r *JobRepository) JobsStats(
   348→	ctx context.Context,
   349→	filter []*model.JobFilter,
   350→	reqFields map[string]bool,
   351→) ([]*model.JobsStatistics, error) {
   352→	start := time.Now()
   353→	query := r.buildStatsQuery(filter, "", config.Keys.ShortRunningJobsDuration, reqFields)
   354→	query, err := SecurityCheck(ctx, query)
   355→	if err != nil {
   356→		return nil, err
   357→	}
   358→
   359→	row := query.RunWith(r.DB).QueryRowContext(ctx)
   360→	stats := make([]*model.JobsStatistics, 0, 1)
   361→
   362→	var jobs, users, walltime, nodes, nodeHours, cores, coreHours, accs, accHours, runningJobs, shortJobs sql.NullInt64
   363→	if err := row.Scan(&jobs, &users, &walltime, &nodes, &nodeHours, &cores, &coreHours, &accs, &accHours, &runningJobs, &shortJobs); err != nil {
   364→		cclog.Warn("Error while scanning rows")
   365→		return nil, err
   366→	}
   367→
   368→	if jobs.Valid {
   369→		var totalNodeHours, totalCoreHours, totalAccHours int
   370→
   371→		if nodeHours.Valid {
   372→			totalNodeHours = int(nodeHours.Int64)
   373→		}
   374→		if coreHours.Valid {
   375→			totalCoreHours = int(coreHours.Int64)
   376→		}
   377→		if accHours.Valid {
   378→			totalAccHours = int(accHours.Int64)
   379→		}
   380→		stats = append(stats,
   381→			&model.JobsStatistics{
   382→				TotalJobs:      int(jobs.Int64),
   383→				TotalUsers:     int(users.Int64),
   384→				TotalWalltime:  int(walltime.Int64),
   385→				TotalNodeHours: totalNodeHours,
   386→				TotalCoreHours: totalCoreHours,
   387→				TotalAccHours:  totalAccHours,
   388→				RunningJobs:    int(runningJobs.Int64),
   389→				ShortJobs:      int(shortJobs.Int64),
   390→			})
   391→	}
   392→
   393→	cclog.Debugf("Timer JobStats %s", time.Since(start))
   394→	return stats, nil
   395→}
   396→
   397→// LoadJobStat retrieves a specific statistic for a metric from a job's statistics.
   398→// Returns 0.0 if the metric is not found or statType is invalid.
   399→//
   400→// Parameters:
   401→//   - job: Job struct with populated Statistics field
   402→//   - metric: Name of the metric to query (e.g., "cpu_load", "mem_used")
   403→//   - statType: Type of statistic: "avg", "min", or "max"
   404→//
   405→// Returns the requested statistic value or 0.0 if not found.
   406→func LoadJobStat(job *schema.Job, metric string, statType string) float64 {
   407→	if stats, ok := job.Statistics[metric]; ok {
   408→		switch statType {
   409→		case "avg":
   410→			return stats.Avg
   411→		case "max":
   412→			return stats.Max
   413→		case "min":
   414→			return stats.Min
   415→		default:
   416→			cclog.Errorf("Unknown stat type %s", statType)
   417→		}
   418→	}
   419→
   420→	return 0.0
   421→}
   422→
   423→// JobCountGrouped counts jobs grouped by a dimension without computing detailed statistics.
   424→//
   425→// This is a lightweight alternative to JobsStatsGrouped when only job counts are needed,
   426→// avoiding the overhead of calculating walltime and resource usage metrics.
   427→//
   428→// Parameters:
   429→//   - ctx: Context for security checks
   430→//   - filter: Filters to apply
   431→//   - groupBy: Grouping dimension (User, Project, Cluster, or SubCluster)
   432→//
   433→// Returns JobsStatistics with only ID and TotalJobs populated for each group.
   434→func (r *JobRepository) JobCountGrouped(
   435→	ctx context.Context,
   436→	filter []*model.JobFilter,
   437→	groupBy *model.Aggregate,
   438→) ([]*model.JobsStatistics, error) {
   439→	start := time.Now()
   440→	col := groupBy2column[*groupBy]
   441→	query := r.buildCountQuery(filter, "", col)
   442→	query, err := SecurityCheck(ctx, query)
   443→	if err != nil {
   444→		return nil, err
   445→	}
   446→	rows, err := query.RunWith(r.DB).QueryContext(ctx)
   447→	if err != nil {
   448→		cclog.Warn("Error while querying DB for job statistics")
   449→		return nil, err
   450→	}
   451→	defer rows.Close()
   452→
   453→	stats := make([]*model.JobsStatistics, 0, 100)
   454→
   455→	for rows.Next() {
   456→		var id sql.NullString
   457→		var cnt sql.NullInt64
   458→		if err := rows.Scan(&id, &cnt); err != nil {
   459→			cclog.Warn("Error while scanning rows")
   460→			return nil, err
   461→		}
   462→		if id.Valid {
   463→			stats = append(stats,
   464→				&model.JobsStatistics{
   465→					ID:        id.String,
   466→					TotalJobs: int(cnt.Int64),
   467→				})
   468→		}
   469→	}
   470→
   471→	if err := rows.Err(); err != nil {
   472→		return nil, err
   473→	}
   474→
   475→	cclog.Debugf("Timer JobCountGrouped %s", time.Since(start))
   476→	return stats, nil
   477→}
   478→
   479→// AddJobCountGrouped augments existing statistics with additional job counts by category.
   480→//
   481→// This method enriches JobsStatistics returned by JobsStatsGrouped or JobCountGrouped
   482→// with counts of running or short-running (based on ShortRunningJobsDuration) jobs, matched by group ID.
   483→//
   484→// Parameters:
   485→//   - ctx: Context for security checks
   486→//   - filter: Filters to apply
   487→//   - groupBy: Grouping dimension (must match the dimension used for stats parameter)
   488→//   - stats: Existing statistics to augment (modified in-place by ID matching)
   489→//   - kind: "running" to add RunningJobs count, "short" to add ShortJobs count
   490→//
   491→// Returns the same stats slice with RunningJobs or ShortJobs fields populated per group.
   492→// Groups without matching jobs will have 0 for the added field.
   493→func (r *JobRepository) AddJobCountGrouped(
   494→	ctx context.Context,
   495→	filter []*model.JobFilter,
   496→	groupBy *model.Aggregate,
   497→	stats []*model.JobsStatistics,
   498→	kind string,
   499→) ([]*model.JobsStatistics, error) {
   500→	start := time.Now()
   501→	col := groupBy2column[*groupBy]
   502→	query := r.buildCountQuery(filter, kind, col)
   503→	query, err := SecurityCheck(ctx, query)
   504→	if err != nil {
   505→		return nil, err
   506→	}
   507→	rows, err := query.RunWith(r.DB).QueryContext(ctx)
   508→	if err != nil {
   509→		cclog.Warn("Error while querying DB for job statistics")
   510→		return nil, err
   511→	}
   512→	defer rows.Close()
   513→
   514→	counts := make(map[string]int)
   515→
   516→	for rows.Next() {
   517→		var id sql.NullString
   518→		var cnt sql.NullInt64
   519→		if err := rows.Scan(&id, &cnt); err != nil {
   520→			cclog.Warn("Error while scanning rows")
   521→			return nil, err
   522→		}
   523→		if id.Valid {
   524→			counts[id.String] = int(cnt.Int64)
   525→		}
   526→	}
   527→
   528→	if err := rows.Err(); err != nil {
   529→		return nil, err
   530→	}
   531→
   532→	switch kind {
   533→	case "running":
   534→		for _, s := range stats {
   535→			s.RunningJobs = counts[s.ID]
   536→		}
   537→	case "short":
   538→		for _, s := range stats {
   539→			s.ShortJobs = counts[s.ID]
   540→		}
   541→	}
   542→
   543→	cclog.Debugf("Timer AddJobCountGrouped %s", time.Since(start))
   544→	return stats, nil
   545→}
   546→
   547→// AddJobCount augments existing overall statistics with additional job counts by category.
   548→//
   549→// Similar to AddJobCountGrouped but for ungrouped statistics. Applies the same count
   550→// to all statistics entries (typically just one).
   551→//
   552→// Parameters:
   553→//   - ctx: Context for security checks
   554→//   - filter: Filters to apply
   555→//   - stats: Existing statistics to augment (modified in-place)
   556→//   - kind: "running" to add RunningJobs count, "short" to add ShortJobs count
   557→//
   558→// Returns the same stats slice with RunningJobs or ShortJobs fields set to the total count.
   559→func (r *JobRepository) AddJobCount(
   560→	ctx context.Context,
   561→	filter []*model.JobFilter,
   562→	stats []*model.JobsStatistics,
   563→	kind string,
   564→) ([]*model.JobsStatistics, error) {
   565→	start := time.Now()
   566→	query := r.buildCountQuery(filter, kind, "")
   567→	query, err := SecurityCheck(ctx, query)
   568→	if err != nil {
   569→		return nil, err
   570→	}
   571→	var cnt sql.NullInt64
   572→	if err := query.RunWith(r.DB).QueryRowContext(ctx).Scan(&cnt); err != nil {
   573→		cclog.Warn("Error while querying DB for job count")
   574→		return nil, err
   575→	}
   576→
   577→	count := int(cnt.Int64)
   578→
   579→	switch kind {
   580→	case "running":
   581→		for _, s := range stats {
   582→			s.RunningJobs = count
   583→		}
   584→	case "short":
   585→		for _, s := range stats {
   586→			s.ShortJobs = count
   587→		}
   588→	}
   589→
   590→	cclog.Debugf("Timer AddJobCount %s", time.Since(start))
   591→	return stats, nil
   592→}
   593→
   594→// AddHistograms augments statistics with distribution histograms for job properties.
   595→//
   596→// Generates histogram data for visualization of job duration, node count, core count,
   597→// and accelerator count distributions. Duration histogram uses intelligent binning based
   598→// on the requested resolution.
   599→//
   600→// Parameters:
   601→//   - ctx: Context for security checks
   602→//   - filter: Filters to apply to jobs included in histograms
   603→//   - stat: Statistics struct to augment (modified in-place)
   604→//   - durationBins: Bin size - "1m", "10m", "1h", "6h", "12h", or "24h" (default)
   605→//
   606→// Populates these fields in stat:
   607→//   - HistDuration: Job duration distribution (zero-padded bins)
   608→//   - HistNumNodes: Node count distribution
   609→//   - HistNumCores: Core (hwthread) count distribution
   610→//   - HistNumAccs: Accelerator count distribution
   611→//
   612→// Duration bins are pre-initialized with zeros to ensure consistent ranges for visualization.
   613→// Bin size determines both the width and maximum duration displayed (e.g., "1h" = 48 bins × 1h = 48h max).
   614→func (r *JobRepository) AddHistograms(
   615→	ctx context.Context,
   616→	filter []*model.JobFilter,
   617→	stat *model.JobsStatistics,
   618→	durationBins *string,
   619→) (*model.JobsStatistics, error) {
   620→	start := time.Now()
   621→
   622→	var targetBinCount int
   623→	var targetBinSize int
   624→	switch *durationBins {
   625→	case "1m": // 1 Minute Bins + Max 60 Bins -> Max 60 Minutes
   626→		targetBinCount = 60
   627→		targetBinSize = 60
   628→	case "10m": // 10 Minute Bins + Max 72 Bins -> Max 12 Hours
   629→		targetBinCount = 72
   630→		targetBinSize = 600
   631→	case "1h": // 1 Hour Bins + Max 48 Bins -> Max 48 Hours
   632→		targetBinCount = 48
   633→		targetBinSize = 3600
   634→	case "6h": // 6 Hour Bins + Max 12 Bins -> Max 3 Days
   635→		targetBinCount = 12
   636→		targetBinSize = 21600
   637→	case "12h": // 12 hour Bins + Max 14 Bins -> Max 7 Days
   638→		targetBinCount = 14
   639→		targetBinSize = 43200
   640→	default: // 24h
   641→		targetBinCount = 24
   642→		targetBinSize = 3600
   643→	}
   644→
   645→	var err error
   646→	// Return X-Values always as seconds, will be formatted into minutes and hours in frontend
   647→	value := fmt.Sprintf(`CAST(ROUND(((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) / %d) + 1) as int) as value`, time.Now().Unix(), targetBinSize)
   648→	stat.HistDuration, err = r.jobsDurationStatisticsHistogram(ctx, value, filter, targetBinSize, &targetBinCount)
   649→	if err != nil {
   650→		cclog.Warn("Error while loading job statistics histogram: job duration")
   651→		return nil, err
   652→	}
   653→
   654→	stat.HistNumNodes, err = r.jobsStatisticsHistogram(ctx, "job.num_nodes as value", filter)
   655→	if err != nil {
   656→		cclog.Warn("Error while loading job statistics histogram: num nodes")
   657→		return nil, err
   658→	}
   659→
   660→	stat.HistNumCores, err = r.jobsStatisticsHistogram(ctx, "job.num_hwthreads as value", filter)
   661→	if err != nil {
   662→		cclog.Warn("Error while loading job statistics histogram: num hwthreads")
   663→		return nil, err
   664→	}
   665→
   666→	stat.HistNumAccs, err = r.jobsStatisticsHistogram(ctx, "job.num_acc as value", filter)
   667→	if err != nil {
   668→		cclog.Warn("Error while loading job statistics histogram: num acc")
   669→		return nil, err
   670→	}
   671→
   672→	cclog.Debugf("Timer AddHistograms %s", time.Since(start))
   673→	return stat, nil
   674→}
   675→
   676→// AddMetricHistograms augments statistics with distribution histograms for job metrics.
   677→//
   678→// Generates histogram data for metrics like CPU load, memory usage, etc. Handles running
   679→// and completed jobs differently: running jobs load data from metric backend, completed jobs
   680→// use footprint data from database.
   681→//
   682→// Parameters:
   683→//   - ctx: Context for security checks
   684→//   - filter: Filters to apply (MUST contain State filter for running jobs)
   685→//   - metrics: List of metric names to histogram (e.g., ["cpu_load", "mem_used"])
   686→//   - stat: Statistics struct to augment (modified in-place)
   687→//   - targetBinCount: Number of histogram bins (default: 10)
   688→//
   689→// Populates HistMetrics field in stat with MetricHistoPoints for each metric.
   690→//
   691→// Binning algorithm:
   692→//   - Values normalized to metric's peak value from cluster configuration
   693→//   - Bins evenly distributed from 0 to peak
   694→//   - Pre-initialized with zeros for consistent visualization
   695→//
   696→// Limitations:
   697→//   - Running jobs: Limited to 5000 jobs for performance
   698→//   - Requires valid cluster configuration with metric peak values
   699→//   - Uses footprint statistic (avg/max/min) configured per metric
   700→func (r *JobRepository) AddMetricHistograms(
   701→	ctx context.Context,
   702→	filter []*model.JobFilter,
   703→	metrics []string,
   704→	stat *model.JobsStatistics,
   705→	targetBinCount *int,
   706→) (*model.JobsStatistics, error) {
   707→	start := time.Now()
   708→
   709→	// Running Jobs Only: First query jobdata from sqlite, then query data and make bins
   710→	for _, f := range filter {
   711→		if f.State != nil {
   712→			if len(f.State) == 1 && f.State[0] == "running" {
   713→				stat.HistMetrics = r.runningJobsMetricStatisticsHistogram(ctx, metrics, filter, targetBinCount)
   714→				cclog.Debugf("Timer AddMetricHistograms %s", time.Since(start))
   715→				return stat, nil
   716→			}
   717→		}
   718→	}
   719→
   720→	// All other cases: Query and make bins in sqlite directly
   721→	for _, m := range metrics {
   722→		metricHisto, err := r.jobsMetricStatisticsHistogram(ctx, m, filter, targetBinCount)
   723→		if err != nil {
   724→			cclog.Warnf("Error while loading job metric statistics histogram: %s", m)
   725→			continue
   726→		}
   727→		stat.HistMetrics = append(stat.HistMetrics, metricHisto)
   728→	}
   729→
   730→	cclog.Debugf("Timer AddMetricHistograms %s", time.Since(start))
   731→	return stat, nil
   732→}
   733→
   734→// jobsStatisticsHistogram generates a simple histogram by grouping on a column value.
   735→//
   736→// Used for histograms where the column value directly represents the bin (e.g., node count, core count).
   737→// Unlike duration/metric histograms, this doesn't pre-initialize bins with zeros.
   738→//
   739→// Parameters:
   740→//   - value: SQL expression that produces the histogram value, aliased as "value"
   741→//   - filters: Job filters to apply
   742→//
   743→// Returns histogram points with Value (from column) and Count (number of jobs).
   744→func (r *JobRepository) jobsStatisticsHistogram(
   745→	ctx context.Context,
   746→	value string,
   747→	filters []*model.JobFilter,
   748→) ([]*model.HistoPoint, error) {
   749→	start := time.Now()
   750→	query, qerr := SecurityCheck(ctx,
   751→		sq.Select(value, "COUNT(job.id) AS count").From("job"))
   752→
   753→	if qerr != nil {
   754→		return nil, qerr
   755→	}
   756→
   757→	for _, f := range filters {
   758→		query = BuildWhereClause(f, query)
   759→	}
   760→
   761→	rows, err := query.GroupBy("value").RunWith(r.DB).QueryContext(ctx)
   762→	if err != nil {
   763→		cclog.Error("Error while running query")
   764→		return nil, err
   765→	}
   766→	defer rows.Close()
   767→
   768→	points := make([]*model.HistoPoint, 0)
   769→	// is it possible to introduce zero values here? requires info about bincount
   770→	for rows.Next() {
   771→		point := model.HistoPoint{}
   772→		if err := rows.Scan(&point.Value, &point.Count); err != nil {
   773→			cclog.Warn("Error while scanning rows")
   774→			return nil, err
   775→		}
   776→
   777→		points = append(points, &point)
   778→	}
   779→
   780→	if err := rows.Err(); err != nil {
   781→		return nil, err
   782→	}
   783→
   784→	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
   785→	return points, nil
   786→}
   787→
   788→// jobsDurationStatisticsHistogram generates a duration histogram with pre-initialized bins.
   789→//
   790→// Bins are zero-padded to provide consistent ranges for visualization, unlike simple
   791→// histograms which only return bins with data. The value parameter should compute
   792→// the bin number from job duration.
   793→//
   794→// Parameters:
   795→//   - value: SQL expression computing bin number from duration, aliased as "value"
   796→//   - filters: Job filters to apply
   797→//   - binSizeSeconds: Width of each bin in seconds
   798→//   - targetBinCount: Number of bins to pre-initialize
   799→//
   800→// Returns histogram points with Value (bin_number × binSizeSeconds) and Count.
   801→// All bins from 1 to targetBinCount are returned, with Count=0 for empty bins.
   802→//
   803→// Algorithm:
   804→//  1. Pre-initialize targetBinCount bins with zero counts
   805→//  2. Query database for actual counts per bin
   806→//  3. Match query results to pre-initialized bins by value
   807→//  4. Bins without matches remain at zero
   808→func (r *JobRepository) jobsDurationStatisticsHistogram(
   809→	ctx context.Context,
   810→	value string,
   811→	filters []*model.JobFilter,
   812→	binSizeSeconds int,
   813→	targetBinCount *int,
   814→) ([]*model.HistoPoint, error) {
   815→	start := time.Now()
   816→	query, qerr := SecurityCheck(ctx,
   817→		sq.Select(value, "COUNT(job.id) AS count").From("job"))
   818→
   819→	if qerr != nil {
   820→		return nil, qerr
   821→	}
   822→
   823→	// Each bin represents a duration range: bin N = [N*binSizeSeconds, (N+1)*binSizeSeconds)
   824→	// Example: binSizeSeconds=3600 (1 hour), bin 1 = 0-1h, bin 2 = 1-2h, etc.
   825→	points := make([]*model.HistoPoint, 0, *targetBinCount)
   826→	for i := 1; i <= *targetBinCount; i++ {
   827→		point := model.HistoPoint{Value: i * binSizeSeconds, Count: 0}
   828→		points = append(points, &point)
   829→	}
   830→
   831→	// Build a map from bin value (seconds) to slice index for O(1) lookup.
   832→	binMap := make(map[int]int, len(points))
   833→	for i, p := range points {
   834→		binMap[p.Value] = i
   835→	}
   836→
   837→	for _, f := range filters {
   838→		query = BuildWhereClause(f, query)
   839→	}
   840→
   841→	rows, err := query.GroupBy("value").RunWith(r.DB).QueryContext(ctx)
   842→	if err != nil {
   843→		cclog.Error("Error while running query")
   844→		return nil, err
   845→	}
   846→	defer rows.Close()
   847→
   848→	// Match query results to pre-initialized bins using map lookup.
   849→	// point.Value from query is the bin number; multiply by binSizeSeconds to get bin key.
   850→	for rows.Next() {
   851→		point := model.HistoPoint{}
   852→		if err := rows.Scan(&point.Value, &point.Count); err != nil {
   853→			cclog.Warn("Error while scanning rows")
   854→			return nil, err
   855→		}
   856→
   857→		if idx, ok := binMap[point.Value*binSizeSeconds]; ok {
   858→			points[idx].Count = point.Count
   859→		}
   860→	}
   861→
   862→	if err := rows.Err(); err != nil {
   863→		return nil, err
   864→	}
   865→
   866→	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
   867→	return points, nil
   868→}
   869→
   870→// jobsMetricStatisticsHistogram generates a metric histogram using footprint data from completed jobs.
   871→//
   872→// Values are normalized to the metric's peak value and distributed into bins. The algorithm
   873→// is based on SQL histogram generation techniques, extracting metric values from JSON footprint
   874→// and computing bin assignments in SQL.
   875→//
   876→// Parameters:
   877→//   - metric: Metric name (e.g., "cpu_load", "mem_used")
   878→//   - filters: Job filters to apply
   879→//   - bins: Number of bins to generate
   880→//
   881→// Returns MetricHistoPoints with metric name, unit, footprint stat type, and binned data.
   882→//
   883→// Algorithm:
   884→//  1. Determine peak value from cluster configuration (filtered cluster or max across all)
   885→//  2. Generate SQL that extracts footprint value, normalizes to [0,1], multiplies by bin count
   886→//  3. Pre-initialize bins with min/max ranges based on peak value
   887→//  4. Query database for counts per bin
   888→//  5. Match results to pre-initialized bins
   889→//
   890→// Special handling: Values exactly equal to peak are forced into the last bin by multiplying
   891→// peak by 0.999999999 to avoid creating an extra bin.
   892→func (r *JobRepository) jobsMetricStatisticsHistogram(
   893→	ctx context.Context,
   894→	metric string,
   895→	filters []*model.JobFilter,
   896→	bins *int,
   897→) (*model.MetricHistoPoints, error) {
   898→	// Peak value defines the upper bound for binning: values are distributed across
   899→	// bins from 0 to peak. First try to get peak from filtered cluster, otherwise
   900→	// scan all clusters to find the maximum peak value.
   901→	var metricConfig *schema.MetricConfig
   902→	var peak float64
   903→	var unit string
   904→	var footprintStat string
   905→
   906→	// Try to get metric config from filtered cluster
   907→	for _, f := range filters {
   908→		if f.Cluster != nil {
   909→			metricConfig = archive.GetMetricConfig(*f.Cluster.Eq, metric)
   910→			peak = metricConfig.Peak
   911→			unit = metricConfig.Unit.Prefix + metricConfig.Unit.Base
   912→			footprintStat = metricConfig.Footprint
   913→			cclog.Debugf("Cluster %s filter found with peak %f for %s", *f.Cluster.Eq, peak, metric)
   914→		}
   915→	}
   916→
   917→	// If no cluster filter or peak not found, find largest peak across all clusters
   918→	// This ensures histogram can accommodate all possible values
   919→	if peak == 0.0 {
   920→		for _, c := range archive.Clusters {
   921→			for _, m := range c.MetricConfig {
   922→				if m.Name == metric {
   923→					if m.Peak > peak {
   924→						peak = m.Peak
   925→					}
   926→					if unit == "" {
   927→						unit = m.Unit.Prefix + m.Unit.Base
   928→					}
   929→					if footprintStat == "" {
   930→						footprintStat = m.Footprint
   931→					}
   932→				}
   933→			}
   934→		}
   935→	}
   936→
   937→	// Construct SQL histogram bins using normalized values.
   938→	// Algorithm based on: https://jereze.com/code/sql-histogram/ (modified)
   939→	start := time.Now()
   940→
   941→	// Bin calculation formula:
   942→	//   bin_number = CAST( (value / peak) * num_bins AS INTEGER ) + 1
   943→	// Special case: value == peak would create bin N+1, so we test for equality
   944→	// and multiply peak by 0.999999999 to force it into bin N.
   945→	binQuery := fmt.Sprintf(`CAST(
   946→		((case when json_extract(footprint, "$.%s") = %f then %f*0.999999999 else json_extract(footprint, "$.%s") end) / %f)
   947→		* %v as INTEGER )`,
   948→		(metric + "_" + footprintStat), peak, peak, (metric + "_" + footprintStat), peak, *bins)
   949→
   950→	mainQuery := sq.Select(
   951→		fmt.Sprintf(`%s + 1 as bin`, binQuery),
   952→		`count(*) as count`,
   953→	).From("job").Where(
   954→		"JSON_VALID(footprint)",
   955→	).Where(fmt.Sprintf(`json_extract(footprint, "$.%s") is not null and json_extract(footprint, "$.%s") <= %f`, (metric + "_" + footprintStat), (metric + "_" + footprintStat), peak))
   956→
   957→	mainQuery, qerr := SecurityCheck(ctx, mainQuery)
   958→	if qerr != nil {
   959→		return nil, qerr
   960→	}
   961→
   962→	for _, f := range filters {
   963→		mainQuery = BuildWhereClause(f, mainQuery)
   964→	}
   965→
   966→	mainQuery = mainQuery.GroupBy("bin").OrderBy("bin")
   967→
   968→	rows, err := mainQuery.RunWith(r.DB).QueryContext(ctx)
   969→	if err != nil {
   970→		cclog.Errorf("Error while running mainQuery: %s", err)
   971→		return nil, err
   972→	}
   973→	defer rows.Close()
   974→
   975→	// Pre-initialize bins with calculated min/max ranges.
   976→	// Example: peak=1000, bins=10 -> bin 1=[0,100), bin 2=[100,200), ..., bin 10=[900,1000]
   977→	points := make([]*model.MetricHistoPoint, 0, *bins)
   978→	binStep := int(peak) / *bins
   979→	for i := 1; i <= *bins; i++ {
   980→		binMin := (binStep * (i - 1))
   981→		binMax := (binStep * i)
   982→		idx := i
   983→		epoint := model.MetricHistoPoint{Bin: &idx, Count: 0, Min: &binMin, Max: &binMax}
   984→		points = append(points, &epoint)
   985→	}
   986→
   987→	// Build a map from bin number to slice index for O(1) lookup.
   988→	binMap := make(map[int]int, len(points))
   989→	for i, p := range points {
   990→		if p.Bin != nil {
   991→			binMap[*p.Bin] = i
   992→		}
   993→	}
   994→
   995→	// Match query results to pre-initialized bins using map lookup.
   996→	for rows.Next() {
   997→		rpoint := model.MetricHistoPoint{}
   998→		if err := rows.Scan(&rpoint.Bin, &rpoint.Count); err != nil {
   999→			cclog.Warnf("Error while scanning rows for %s", metric)
  1000→			return nil, err
  1001→		}
  1002→
  1003→		if rpoint.Bin != nil {
  1004→			if idx, ok := binMap[*rpoint.Bin]; ok {
  1005→				points[idx].Count = rpoint.Count
  1006→			}
  1007→		}
  1008→	}
  1009→
  1010→	if err := rows.Err(); err != nil {
  1011→		return nil, err
  1012→	}
  1013→
  1014→	result := model.MetricHistoPoints{Metric: metric, Unit: unit, Stat: &footprintStat, Data: points}
  1015→
  1016→	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
  1017→	return &result, nil
  1018→}
  1019→
  1020→// runningJobsMetricStatisticsHistogram generates metric histograms for running jobs using live data.
  1021→//
  1022→// Unlike completed jobs which use footprint data from the database, running jobs require
  1023→// fetching current metric averages from the metric backend (via metricdispatch).
  1024→//
  1025→// Parameters:
  1026→//   - metrics: List of metric names
  1027→//   - filters: Job filters (should filter to running jobs only)
  1028→//   - bins: Number of histogram bins
  1029→//
  1030→// Returns slice of MetricHistoPoints, one per metric.
  1031→//
  1032→// Limitations:
  1033→//   - Maximum 5000 jobs (returns nil if more jobs match)
  1034→//   - Requires metric backend availability
  1035→//   - Bins based on metric peak values from cluster configuration
  1036→//
  1037→// Algorithm:
  1038→//  1. Query first 5001 jobs to check count limit
  1039→//  2. Load metric averages for all jobs via metricdispatch
  1040→//  3. For each metric, create bins based on peak value
  1041→//  4. Iterate averages and count jobs per bin
  1042→func (r *JobRepository) runningJobsMetricStatisticsHistogram(
  1043→	ctx context.Context,
  1044→	metrics []string,
  1045→	filters []*model.JobFilter,
  1046→	bins *int,
  1047→) []*model.MetricHistoPoints {
  1048→	// Get Jobs
  1049→	jobs, err := r.QueryJobs(ctx, filters, &model.PageRequest{Page: 1, ItemsPerPage: 5000 + 1}, nil)
  1050→	if err != nil {
  1051→		cclog.Errorf("Error while querying jobs for footprint: %s", err)
  1052→		return nil
  1053→	}
  1054→	if len(jobs) > 5000 {
  1055→		cclog.Errorf("too many jobs matched (max: %d)", 5000)
  1056→		return nil
  1057→	}
  1058→
  1059→	// Get AVGs from metric repo
  1060→	avgs := make([][]schema.Float, len(metrics))
  1061→	for i := range avgs {
  1062→		avgs[i] = make([]schema.Float, 0, len(jobs))
  1063→	}
  1064→
  1065→	for _, job := range jobs {
  1066→		if job.MonitoringStatus == schema.MonitoringStatusDisabled || job.MonitoringStatus == schema.MonitoringStatusArchivingFailed {
  1067→			continue
  1068→		}
  1069→
  1070→		if err := metricdispatch.LoadAverages(job, metrics, avgs, ctx); err != nil {
  1071→			cclog.Errorf("Error while loading averages for histogram: %s", err)
  1072→			return nil
  1073→		}
  1074→	}
  1075→
  1076→	// Iterate metrics to fill endresult
  1077→	data := make([]*model.MetricHistoPoints, 0)
  1078→	for idx, metric := range metrics {
  1079→		// Get specific Peak or largest Peak
  1080→		var metricConfig *schema.MetricConfig
  1081→		var peak float64
  1082→		var unit string
  1083→
  1084→		for _, f := range filters {
  1085→			if f.Cluster != nil {
  1086→				metricConfig = archive.GetMetricConfig(*f.Cluster.Eq, metric)
  1087→				peak = metricConfig.Peak
  1088→				unit = metricConfig.Unit.Prefix + metricConfig.Unit.Base
  1089→			}
  1090→		}
  1091→
  1092→		if peak == 0.0 {
  1093→			for _, c := range archive.Clusters {
  1094→				for _, m := range c.MetricConfig {
  1095→					if m.Name == metric {
  1096→						if m.Peak > peak {
  1097→							peak = m.Peak
  1098→						}
  1099→						if unit == "" {
  1100→							unit = m.Unit.Prefix + m.Unit.Base
  1101→						}
  1102→					}
  1103→				}
  1104→			}
  1105→		}
  1106→
  1107→		// Make and fill bins
  1108→		peakBin := int(peak) / *bins
  1109→
  1110→		points := make([]*model.MetricHistoPoint, 0)
  1111→		for b := 0; b < *bins; b++ {
  1112→			count := 0
  1113→			bindex := b + 1
  1114→			bmin := peakBin * b
  1115→			bmax := peakBin * (b + 1)
  1116→
  1117→			// Iterate AVG values for indexed metric and count for bins
  1118→			for _, val := range avgs[idx] {
  1119→				if int(val) >= bmin && int(val) < bmax {
  1120→					count += 1
  1121→				}
  1122→			}
  1123→
  1124→			// Append Bin to Metric Result Array
  1125→			point := model.MetricHistoPoint{Bin: &bindex, Count: count, Min: &bmin, Max: &bmax}
  1126→			points = append(points, &point)
  1127→		}
  1128→
  1129→		// Append Metric Result Array to final results array
  1130→		result := model.MetricHistoPoints{Metric: metric, Unit: unit, Data: points}
  1131→		data = append(data, &result)
  1132→	}
  1133→
  1134→	return data
  1135→}
  1136→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// This file contains job statistics and histogram generation functionality for the JobRepository.
//
// # Job Statistics
//
// The statistics methods provide aggregated metrics about jobs including total jobs, users,
// walltime, and resource usage (nodes, cores, accelerators). Statistics can be computed:
//   - Overall (JobsStats): Single aggregate across all matching jobs
//   - Grouped (JobsStatsGrouped): Aggregated by user, project, cluster, or subcluster
//   - Counts (JobCountGrouped, AddJobCount): Simple job counts with optional filtering
//
// All statistics methods support filtering via JobFilter and respect security contexts.
//
// # Histograms
//
// Histogram methods generate distribution data for visualization:
//   - Duration, nodes, cores, accelerators (AddHistograms)
//   - Job metrics like CPU load, memory usage (AddMetricHistograms)
//
// Histograms use intelligent binning:
//   - Duration: Variable bin sizes (1m, 10m, 1h, 6h, 12h, 24h) with zero-padding
//   - Resources: Natural value-based bins
//   - Metrics: Normalized to peak values with configurable bin counts
//
// # Running vs. Completed Jobs
//
// Statistics handle running jobs specially:
//   - Duration calculated as (now - start_time) for running jobs
//   - Metric histograms for running jobs load data from metric backend instead of footprint
//   - Job state filtering distinguishes running/completed jobs
//
// # Performance Considerations
//
// - All queries use prepared statements via stmtCache
// - Complex aggregations use SQL for efficiency
// - Histogram pre-initialization ensures consistent bin ranges
// - Metric histogram queries limited to 5000 jobs for running job analysis

package repository

import (
	"context"
	"database/sql"
	"fmt"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/config"
	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
	"github.com/ClusterCockpit/cc-backend/internal/metricdispatch"
	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	sq "github.com/Masterminds/squirrel"
)

// groupBy2column maps GraphQL Aggregate enum values to their corresponding database column names.
// Used by JobsStatsGrouped and JobCountGrouped to translate user-facing grouping dimensions
// into SQL GROUP BY clauses. GraphQL validation ensures only valid enum values are accepted.
var groupBy2column = map[model.Aggregate]string{
	model.AggregateUser:       "job.hpc_user",
	model.AggregateProject:    "job.project",
	model.AggregateCluster:    "job.cluster",
	model.AggregateSubcluster: "job.subcluster",
}

// sortBy2column maps GraphQL SortByAggregate enum values to their corresponding computed column names.
// Used by JobsStatsGrouped to translate sort preferences into SQL ORDER BY clauses.
// Column names match the AS aliases used in buildStatsQuery.
var sortBy2column = map[model.SortByAggregate]string{
	model.SortByAggregateTotaljobs:      "totalJobs",
	model.SortByAggregateTotalusers:     "totalUsers",
	model.SortByAggregateTotalwalltime:  "totalWalltime",
	model.SortByAggregateTotalnodes:     "totalNodes",
	model.SortByAggregateTotalnodehours: "totalNodeHours",
	model.SortByAggregateTotalcores:     "totalCores",
	model.SortByAggregateTotalcorehours: "totalCoreHours",
	model.SortByAggregateTotalaccs:      "totalAccs",
	model.SortByAggregateTotalacchours:  "totalAccHours",
}

// buildCountQuery constructs a SQL query to count jobs with optional grouping and filtering.
//
// Parameters:
//   - filter: Job filters to apply (cluster, user, time range, etc.)
//   - kind: Special filter - "running" for running jobs only, "short" for jobs under threshold
//   - col: Column name to GROUP BY; empty string for total count without grouping
//
// Returns a SelectBuilder that produces either:
//   - Single count: COUNT(job.id) when col is empty
//   - Grouped counts: col, COUNT(job.id) when col is specified
//
// The kind parameter enables counting specific job categories:
//   - "running": Only jobs with job_state = 'running'
//   - "short": Only jobs with duration < ShortRunningJobsDuration config value
//   - empty: All jobs matching filters
func (r *JobRepository) buildCountQuery(
	filter []*model.JobFilter,
	kind string,
	col string,
) sq.SelectBuilder {
	var query sq.SelectBuilder

	if col != "" {
		query = sq.Select(col, "COUNT(job.id)").From("job").GroupBy(col)
	} else {
		query = sq.Select("COUNT(job.id)").From("job")
	}

	switch kind {
	case "running":
		query = query.Where("job.job_state = ?", "running")
	case "short":
		query = query.Where("job.duration < ?", config.Keys.ShortRunningJobsDuration)
	}

	for _, f := range filter {
		query = BuildWhereClause(f, query)
	}

	return query
}

// buildStatsQuery constructs a SQL query to compute job statistics with optional grouping.
// Only requested columns are computed; unrequested columns select 0 as placeholder.
//
// Parameters:
//   - filter: Job filters to apply (cluster, user, time range, etc.)
//   - col: Column name to GROUP BY; empty string for overall statistics without grouping
//   - shortThreshold: Duration threshold in seconds for counting short-running jobs
//   - reqFields: Set of requested field names; nil means compute all fields
func (r *JobRepository) buildStatsQuery(
	filter []*model.JobFilter,
	col string,
	shortThreshold int,
	reqFields map[string]bool,
) sq.SelectBuilder {
	now := time.Now().Unix()

	// Helper: return real expression if field is requested (or reqFields is nil), else "0 as alias"
	need := func(field string) bool {
		return reqFields == nil || reqFields[field]
	}
	durationExpr := fmt.Sprintf(`(CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END)`, now)

	// Build column list
	columns := make([]string, 0, 14)

	if col != "" {
		columns = append(columns, col)
	}

	columns = append(columns, "COUNT(job.id) as totalJobs")

	if need("totalUsers") && col != "job.hpc_user" {
		columns = append(columns, "COUNT(DISTINCT job.hpc_user) AS totalUsers")
	} else {
		columns = append(columns, "0 AS totalUsers")
	}

	if need("totalWalltime") {
		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s) / 3600) as int) as totalWalltime`, durationExpr))
	} else {
		columns = append(columns, "0 as totalWalltime")
	}

	if need("totalNodes") {
		columns = append(columns, `CAST(SUM(job.num_nodes) as int) as totalNodes`)
	} else {
		columns = append(columns, "0 as totalNodes")
	}

	if need("totalNodeHours") {
		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s * job.num_nodes) / 3600) as int) as totalNodeHours`, durationExpr))
	} else {
		columns = append(columns, "0 as totalNodeHours")
	}

	if need("totalCores") {
		columns = append(columns, `CAST(SUM(job.num_hwthreads) as int) as totalCores`)
	} else {
		columns = append(columns, "0 as totalCores")
	}

	if need("totalCoreHours") {
		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s * job.num_hwthreads) / 3600) as int) as totalCoreHours`, durationExpr))
	} else {
		columns = append(columns, "0 as totalCoreHours")
	}

	if need("totalAccs") {
		columns = append(columns, `CAST(SUM(job.num_acc) as int) as totalAccs`)
	} else {
		columns = append(columns, "0 as totalAccs")
	}

	if need("totalAccHours") {
		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s * job.num_acc) / 3600) as int) as totalAccHours`, durationExpr))
	} else {
		columns = append(columns, "0 as totalAccHours")
	}

	if need("runningJobs") {
		columns = append(columns, `COUNT(CASE WHEN job.job_state = 'running' THEN 1 END) as runningJobs`)
	} else {
		columns = append(columns, "0 as runningJobs")
	}

	if need("shortJobs") {
		columns = append(columns, fmt.Sprintf(`COUNT(CASE WHEN job.duration < %d THEN 1 END) as shortJobs`, shortThreshold))
	} else {
		columns = append(columns, "0 as shortJobs")
	}

	query := sq.Select(columns...).From("job")
	if col != "" {
		query = query.GroupBy(col)
	}

	for _, f := range filter {
		query = BuildWhereClause(f, query)
	}

	return query
}

// JobsStatsGrouped computes job statistics grouped by a dimension (user, project, cluster, or subcluster).
// Only columns listed in reqFields are computed; others return 0. User display names are looked up
// in a separate lightweight query to avoid JOIN overhead on the main aggregation.
func (r *JobRepository) JobsStatsGrouped(
	ctx context.Context,
	filter []*model.JobFilter,
	page *model.PageRequest,
	sortBy *model.SortByAggregate,
	groupBy *model.Aggregate,
	reqFields map[string]bool,
) ([]*model.JobsStatistics, error) {
	start := time.Now()
	col := groupBy2column[*groupBy]
	query := r.buildStatsQuery(filter, col, config.Keys.ShortRunningJobsDuration, reqFields)

	query, err := SecurityCheck(ctx, query)
	if err != nil {
		return nil, err
	}

	if sortBy != nil {
		sortBy := sortBy2column[*sortBy]
		query = query.OrderBy(fmt.Sprintf("%s DESC", sortBy))
	}
	if page != nil && page.ItemsPerPage != -1 {
		limit := uint64(page.ItemsPerPage)
		query = query.Offset((uint64(page.Page) - 1) * limit).Limit(limit)
	}

	rows, err := query.RunWith(r.DB).QueryContext(ctx)
	if err != nil {
		cclog.Warn("Error while querying DB for job statistics")
		return nil, err
	}
	defer rows.Close()

	stats := make([]*model.JobsStatistics, 0, 100)

	for rows.Next() {
		var id sql.NullString
		var jobs, users, walltime, nodes, nodeHours, cores, coreHours, accs, accHours, runningJobs, shortJobs sql.NullInt64
		if err := rows.Scan(&id, &jobs, &users, &walltime, &nodes, &nodeHours, &cores, &coreHours, &accs, &accHours, &runningJobs, &shortJobs); err != nil {
			cclog.Warnf("Error while scanning rows: %s", err.Error())
			return nil, err
		}

		if id.Valid {
			stats = append(stats,
				&model.JobsStatistics{
					ID:             id.String,
					TotalJobs:      int(jobs.Int64),
					TotalUsers:     int(users.Int64),
					TotalWalltime:  int(walltime.Int64),
					TotalNodes:     int(nodes.Int64),
					TotalNodeHours: int(nodeHours.Int64),
					TotalCores:     int(cores.Int64),
					TotalCoreHours: int(coreHours.Int64),
					TotalAccs:      int(accs.Int64),
					TotalAccHours:  int(accHours.Int64),
					RunningJobs:    int(runningJobs.Int64),
					ShortJobs:      int(shortJobs.Int64),
				})
		}
	}

	if err := rows.Err(); err != nil {
		return nil, err
	}

	// Post-query name lookup for user grouping (avoids LEFT JOIN on aggregation query)
	if col == "job.hpc_user" && len(stats) > 0 {
		usernames := make([]any, len(stats))
		for i, s := range stats {
			usernames[i] = s.ID
		}

		nameQuery := sq.Select("username", "name").From("hpc_user").Where(sq.Eq{"username": usernames})
		nameRows, err := nameQuery.RunWith(r.DB).QueryContext(ctx)
		if err != nil {
			cclog.Warnf("Error looking up user names: %s", err.Error())
			// Non-fatal: stats are still valid without display names
		} else {
			defer nameRows.Close()
			nameMap := make(map[string]string, len(stats))
			for nameRows.Next() {
				var username, name string
				if err := nameRows.Scan(&username, &name); err == nil {
					nameMap[username] = name
				}
			}
			for _, s := range stats {
				if name, ok := nameMap[s.ID]; ok {
					s.Name = name
				}
			}
		}
	}

	cclog.Debugf("Timer JobsStatsGrouped %s", time.Since(start))
	return stats, nil
}

// JobsStats computes overall job statistics across all matching jobs without grouping.
//
// This method provides a single aggregate view of job metrics, useful for dashboard
// summaries and overall system utilization reports.
//
// Parameters:
//   - ctx: Context for security checks and cancellation
//   - filter: Filters to apply (time range, cluster, job state, etc.)
//
// Returns a single-element slice containing aggregate statistics:
//   - totalJobs, totalUsers, totalWalltime
//   - totalNodeHours, totalCoreHours, totalAccHours
//
// Unlike JobsStatsGrouped, this returns overall totals without breaking down by dimension.
// Security checks are applied via SecurityCheck to respect user access levels.
func (r *JobRepository) JobsStats(
	ctx context.Context,
	filter []*model.JobFilter,
	reqFields map[string]bool,
) ([]*model.JobsStatistics, error) {
	start := time.Now()
	query := r.buildStatsQuery(filter, "", config.Keys.ShortRunningJobsDuration, reqFields)
	query, err := SecurityCheck(ctx, query)
	if err != nil {
		return nil, err
	}

	row := query.RunWith(r.DB).QueryRowContext(ctx)
	stats := make([]*model.JobsStatistics, 0, 1)

	var jobs, users, walltime, nodes, nodeHours, cores, coreHours, accs, accHours, runningJobs, shortJobs sql.NullInt64
	if err := row.Scan(&jobs, &users, &walltime, &nodes, &nodeHours, &cores, &coreHours, &accs, &accHours, &runningJobs, &shortJobs); err != nil {
		cclog.Warn("Error while scanning rows")
		return nil, err
	}

	if jobs.Valid {
		var totalNodeHours, totalCoreHours, totalAccHours int

		if nodeHours.Valid {
			totalNodeHours = int(nodeHours.Int64)
		}
		if coreHours.Valid {
			totalCoreHours = int(coreHours.Int64)
		}
		if accHours.Valid {
			totalAccHours = int(accHours.Int64)
		}
		stats = append(stats,
			&model.JobsStatistics{
				TotalJobs:      int(jobs.Int64),
				TotalUsers:     int(users.Int64),
				TotalWalltime:  int(walltime.Int64),
				TotalNodeHours: totalNodeHours,
				TotalCoreHours: totalCoreHours,
				TotalAccHours:  totalAccHours,
				RunningJobs:    int(runningJobs.Int64),
				ShortJobs:      int(shortJobs.Int64),
			})
	}

	cclog.Debugf("Timer JobStats %s", time.Since(start))
	return stats, nil
}

// LoadJobStat retrieves a specific statistic for a metric from a job's statistics.
// Returns 0.0 if the metric is not found or statType is invalid.
//
// Parameters:
//   - job: Job struct with populated Statistics field
//   - metric: Name of the metric to query (e.g., "cpu_load", "mem_used")
//   - statType: Type of statistic: "avg", "min", or "max"
//
// Returns the requested statistic value or 0.0 if not found.
func LoadJobStat(job *schema.Job, metric string, statType string) float64 {
	if stats, ok := job.Statistics[metric]; ok {
		switch statType {
		case "avg":
			return stats.Avg
		case "max":
			return stats.Max
		case "min":
			return stats.Min
		default:
			cclog.Errorf("Unknown stat type %s", statType)
		}
	}

	return 0.0
}

// JobCountGrouped counts jobs grouped by a dimension without computing detailed statistics.
//
// This is a lightweight alternative to JobsStatsGrouped when only job counts are needed,
// avoiding the overhead of calculating walltime and resource usage metrics.
//
// Parameters:
//   - ctx: Context for security checks
//   - filter: Filters to apply
//   - groupBy: Grouping dimension (User, Project, Cluster, or SubCluster)
//
// Returns JobsStatistics with only ID and TotalJobs populated for each group.
func (r *JobRepository) JobCountGrouped(
	ctx context.Context,
	filter []*model.JobFilter,
	groupBy *model.Aggregate,
) ([]*model.JobsStatistics, error) {
	start := time.Now()
	col := groupBy2column[*groupBy]
	query := r.buildCountQuery(filter, "", col)
	query, err := SecurityCheck(ctx, query)
	if err != nil {
		return nil, err
	}
	rows, err := query.RunWith(r.DB).QueryContext(ctx)
	if err != nil {
		cclog.Warn("Error while querying DB for job statistics")
		return nil, err
	}
	defer rows.Close()

	stats := make([]*model.JobsStatistics, 0, 100)

	for rows.Next() {
		var id sql.NullString
		var cnt sql.NullInt64
		if err := rows.Scan(&id, &cnt); err != nil {
			cclog.Warn("Error while scanning rows")
			return nil, err
		}
		if id.Valid {
			stats = append(stats,
				&model.JobsStatistics{
					ID:        id.String,
					TotalJobs: int(cnt.Int64),
				})
		}
	}

	if err := rows.Err(); err != nil {
		return nil, err
	}

	cclog.Debugf("Timer JobCountGrouped %s", time.Since(start))
	return stats, nil
}

// AddJobCountGrouped augments existing statistics with additional job counts by category.
//
// This method enriches JobsStatistics returned by JobsStatsGrouped or JobCountGrouped
// with counts of running or short-running (based on ShortRunningJobsDuration) jobs, matched by group ID.
//
// Parameters:
//   - ctx: Context for security checks
//   - filter: Filters to apply
//   - groupBy: Grouping dimension (must match the dimension used for stats parameter)
//   - stats: Existing statistics to augment (modified in-place by ID matching)
//   - kind: "running" to add RunningJobs count, "short" to add ShortJobs count
//
// Returns the same stats slice with RunningJobs or ShortJobs fields populated per group.
// Groups without matching jobs will have 0 for the added field.
func (r *JobRepository) AddJobCountGrouped(
	ctx context.Context,
	filter []*model.JobFilter,
	groupBy *model.Aggregate,
	stats []*model.JobsStatistics,
	kind string,
) ([]*model.JobsStatistics, error) {
	start := time.Now()
	col := groupBy2column[*groupBy]
	query := r.buildCountQuery(filter, kind, col)
	query, err := SecurityCheck(ctx, query)
	if err != nil {
		return nil, err
	}
	rows, err := query.RunWith(r.DB).QueryContext(ctx)
	if err != nil {
		cclog.Warn("Error while querying DB for job statistics")
		return nil, err
	}
	defer rows.Close()

	counts := make(map[string]int)

	for rows.Next() {
		var id sql.NullString
		var cnt sql.NullInt64
		if err := rows.Scan(&id, &cnt); err != nil {
			cclog.Warn("Error while scanning rows")
			return nil, err
		}
		if id.Valid {
			counts[id.String] = int(cnt.Int64)
		}
	}

	if err := rows.Err(); err != nil {
		return nil, err
	}

	switch kind {
	case "running":
		for _, s := range stats {
			s.RunningJobs = counts[s.ID]
		}
	case "short":
		for _, s := range stats {
			s.ShortJobs = counts[s.ID]
		}
	}

	cclog.Debugf("Timer AddJobCountGrouped %s", time.Since(start))
	return stats, nil
}

// AddJobCount augments existing overall statistics with additional job counts by category.
//
// Similar to AddJobCountGrouped but for ungrouped statistics. Applies the same count
// to all statistics entries (typically just one).
//
// Parameters:
//   - ctx: Context for security checks
//   - filter: Filters to apply
//   - stats: Existing statistics to augment (modified in-place)
//   - kind: "running" to add RunningJobs count, "short" to add ShortJobs count
//
// Returns the same stats slice with RunningJobs or ShortJobs fields set to the total count.
func (r *JobRepository) AddJobCount(
	ctx context.Context,
	filter []*model.JobFilter,
	stats []*model.JobsStatistics,
	kind string,
) ([]*model.JobsStatistics, error) {
	start := time.Now()
	query := r.buildCountQuery(filter, kind, "")
	query, err := SecurityCheck(ctx, query)
	if err != nil {
		return nil, err
	}
	var cnt sql.NullInt64
	if err := query.RunWith(r.DB).QueryRowContext(ctx).Scan(&cnt); err != nil {
		cclog.Warn("Error while querying DB for job count")
		return nil, err
	}

	count := int(cnt.Int64)

	switch kind {
	case "running":
		for _, s := range stats {
			s.RunningJobs = count
		}
	case "short":
		for _, s := range stats {
			s.ShortJobs = count
		}
	}

	cclog.Debugf("Timer AddJobCount %s", time.Since(start))
	return stats, nil
}

// AddHistograms augments statistics with distribution histograms for job properties.
//
// Generates histogram data for visualization of job duration, node count, core count,
// and accelerator count distributions. Duration histogram uses intelligent binning based
// on the requested resolution.
//
// Parameters:
//   - ctx: Context for security checks
//   - filter: Filters to apply to jobs included in histograms
//   - stat: Statistics struct to augment (modified in-place)
//   - durationBins: Bin size - "1m", "10m", "1h", "6h", "12h", or "24h" (default)
//
// Populates these fields in stat:
//   - HistDuration: Job duration distribution (zero-padded bins)
//   - HistNumNodes: Node count distribution
//   - HistNumCores: Core (hwthread) count distribution
//   - HistNumAccs: Accelerator count distribution
//
// Duration bins are pre-initialized with zeros to ensure consistent ranges for visualization.
// Bin size determines both the width and maximum duration displayed (e.g., "1h" = 48 bins × 1h = 48h max).
func (r *JobRepository) AddHistograms(
	ctx context.Context,
	filter []*model.JobFilter,
	stat *model.JobsStatistics,
	durationBins *string,
) (*model.JobsStatistics, error) {
	start := time.Now()

	var targetBinCount int
	var targetBinSize int
	switch *durationBins {
	case "1m": // 1 Minute Bins + Max 60 Bins -> Max 60 Minutes
		targetBinCount = 60
		targetBinSize = 60
	case "10m": // 10 Minute Bins + Max 72 Bins -> Max 12 Hours
		targetBinCount = 72
		targetBinSize = 600
	case "1h": // 1 Hour Bins + Max 48 Bins -> Max 48 Hours
		targetBinCount = 48
		targetBinSize = 3600
	case "6h": // 6 Hour Bins + Max 12 Bins -> Max 3 Days
		targetBinCount = 12
		targetBinSize = 21600
	case "12h": // 12 hour Bins + Max 14 Bins -> Max 7 Days
		targetBinCount = 14
		targetBinSize = 43200
	default: // 24h
		targetBinCount = 24
		targetBinSize = 3600
	}

	var err error
	// Return X-Values always as seconds, will be formatted into minutes and hours in frontend
	value := fmt.Sprintf(`CAST(ROUND(((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) / %d) + 1) as int) as value`, time.Now().Unix(), targetBinSize)
	stat.HistDuration, err = r.jobsDurationStatisticsHistogram(ctx, value, filter, targetBinSize, &targetBinCount)
	if err != nil {
		cclog.Warn("Error while loading job statistics histogram: job duration")
		return nil, err
	}

	stat.HistNumNodes, err = r.jobsStatisticsHistogram(ctx, "job.num_nodes as value", filter)
	if err != nil {
		cclog.Warn("Error while loading job statistics histogram: num nodes")
		return nil, err
	}

	stat.HistNumCores, err = r.jobsStatisticsHistogram(ctx, "job.num_hwthreads as value", filter)
	if err != nil {
		cclog.Warn("Error while loading job statistics histogram: num hwthreads")
		return nil, err
	}

	stat.HistNumAccs, err = r.jobsStatisticsHistogram(ctx, "job.num_acc as value", filter)
	if err != nil {
		cclog.Warn("Error while loading job statistics histogram: num acc")
		return nil, err
	}

	cclog.Debugf("Timer AddHistograms %s", time.Since(start))
	return stat, nil
}

// AddMetricHistograms augments statistics with distribution histograms for job metrics.
//
// Generates histogram data for metrics like CPU load, memory usage, etc. Handles running
// and completed jobs differently: running jobs load data from metric backend, completed jobs
// use footprint data from database.
//
// Parameters:
//   - ctx: Context for security checks
//   - filter: Filters to apply (MUST contain State filter for running jobs)
//   - metrics: List of metric names to histogram (e.g., ["cpu_load", "mem_used"])
//   - stat: Statistics struct to augment (modified in-place)
//   - targetBinCount: Number of histogram bins (default: 10)
//
// Populates HistMetrics field in stat with MetricHistoPoints for each metric.
//
// Binning algorithm:
//   - Values normalized to metric's peak value from cluster configuration
//   - Bins evenly distributed from 0 to peak
//   - Pre-initialized with zeros for consistent visualization
//
// Limitations:
//   - Running jobs: Limited to 5000 jobs for performance
//   - Requires valid cluster configuration with metric peak values
//   - Uses footprint statistic (avg/max/min) configured per metric
func (r *JobRepository) AddMetricHistograms(
	ctx context.Context,
	filter []*model.JobFilter,
	metrics []string,
	stat *model.JobsStatistics,
	targetBinCount *int,
) (*model.JobsStatistics, error) {
	start := time.Now()

	// Running Jobs Only: First query jobdata from sqlite, then query data and make bins
	for _, f := range filter {
		if f.State != nil {
			if len(f.State) == 1 && f.State[0] == "running" {
				stat.HistMetrics = r.runningJobsMetricStatisticsHistogram(ctx, metrics, filter, targetBinCount)
				cclog.Debugf("Timer AddMetricHistograms %s", time.Since(start))
				return stat, nil
			}
		}
	}

	// All other cases: Query and make bins in sqlite directly
	for _, m := range metrics {
		metricHisto, err := r.jobsMetricStatisticsHistogram(ctx, m, filter, targetBinCount)
		if err != nil {
			cclog.Warnf("Error while loading job metric statistics histogram: %s", m)
			continue
		}
		stat.HistMetrics = append(stat.HistMetrics, metricHisto)
	}

	cclog.Debugf("Timer AddMetricHistograms %s", time.Since(start))
	return stat, nil
}

// jobsStatisticsHistogram generates a simple histogram by grouping on a column value.
//
// Used for histograms where the column value directly represents the bin (e.g., node count, core count).
// Unlike duration/metric histograms, this doesn't pre-initialize bins with zeros.
//
// Parameters:
//   - value: SQL expression that produces the histogram value, aliased as "value"
//   - filters: Job filters to apply
//
// Returns histogram points with Value (from column) and Count (number of jobs).
func (r *JobRepository) jobsStatisticsHistogram(
	ctx context.Context,
	value string,
	filters []*model.JobFilter,
) ([]*model.HistoPoint, error) {
	start := time.Now()
	query, qerr := SecurityCheck(ctx,
		sq.Select(value, "COUNT(job.id) AS count").From("job"))

	if qerr != nil {
		return nil, qerr
	}

	for _, f := range filters {
		query = BuildWhereClause(f, query)
	}

	rows, err := query.GroupBy("value").RunWith(r.DB).QueryContext(ctx)
	if err != nil {
		cclog.Error("Error while running query")
		return nil, err
	}
	defer rows.Close()

	points := make([]*model.HistoPoint, 0)
	// is it possible to introduce zero values here? requires info about bincount
	for rows.Next() {
		point := model.HistoPoint{}
		if err := rows.Scan(&point.Value, &point.Count); err != nil {
			cclog.Warn("Error while scanning rows")
			return nil, err
		}

		points = append(points, &point)
	}

	if err := rows.Err(); err != nil {
		return nil, err
	}

	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
	return points, nil
}

// jobsDurationStatisticsHistogram generates a duration histogram with pre-initialized bins.
//
// Bins are zero-padded to provide consistent ranges for visualization, unlike simple
// histograms which only return bins with data. The value parameter should compute
// the bin number from job duration.
//
// Parameters:
//   - value: SQL expression computing bin number from duration, aliased as "value"
//   - filters: Job filters to apply
//   - binSizeSeconds: Width of each bin in seconds
//   - targetBinCount: Number of bins to pre-initialize
//
// Returns histogram points with Value (bin_number × binSizeSeconds) and Count.
// All bins from 1 to targetBinCount are returned, with Count=0 for empty bins.
//
// Algorithm:
//  1. Pre-initialize targetBinCount bins with zero counts
//  2. Query database for actual counts per bin
//  3. Match query results to pre-initialized bins by value
//  4. Bins without matches remain at zero
func (r *JobRepository) jobsDurationStatisticsHistogram(
	ctx context.Context,
	value string,
	filters []*model.JobFilter,
	binSizeSeconds int,
	targetBinCount *int,
) ([]*model.HistoPoint, error) {
	start := time.Now()
	query, qerr := SecurityCheck(ctx,
		sq.Select(value, "COUNT(job.id) AS count").From("job"))

	if qerr != nil {
		return nil, qerr
	}

	// Each bin represents a duration range: bin N = [N*binSizeSeconds, (N+1)*binSizeSeconds)
	// Example: binSizeSeconds=3600 (1 hour), bin 1 = 0-1h, bin 2 = 1-2h, etc.
	points := make([]*model.HistoPoint, 0, *targetBinCount)
	for i := 1; i <= *targetBinCount; i++ {
		point := model.HistoPoint{Value: i * binSizeSeconds, Count: 0}
		points = append(points, &point)
	}

	// Build a map from bin value (seconds) to slice index for O(1) lookup.
	binMap := make(map[int]int, len(points))
	for i, p := range points {
		binMap[p.Value] = i
	}

	for _, f := range filters {
		query = BuildWhereClause(f, query)
	}

	rows, err := query.GroupBy("value").RunWith(r.DB).QueryContext(ctx)
	if err != nil {
		cclog.Error("Error while running query")
		return nil, err
	}
	defer rows.Close()

	// Match query results to pre-initialized bins using map lookup.
	// point.Value from query is the bin number; multiply by binSizeSeconds to get bin key.
	for rows.Next() {
		point := model.HistoPoint{}
		if err := rows.Scan(&point.Value, &point.Count); err != nil {
			cclog.Warn("Error while scanning rows")
			return nil, err
		}

		if idx, ok := binMap[point.Value*binSizeSeconds]; ok {
			points[idx].Count = point.Count
		}
	}

	if err := rows.Err(); err != nil {
		return nil, err
	}

	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
	return points, nil
}

// jobsMetricStatisticsHistogram generates a metric histogram using footprint data from completed jobs.
//
// Values are normalized to the metric's peak value and distributed into bins. The algorithm
// is based on SQL histogram generation techniques, extracting metric values from JSON footprint
// and computing bin assignments in SQL.
//
// Parameters:
//   - metric: Metric name (e.g., "cpu_load", "mem_used")
//   - filters: Job filters to apply
//   - bins: Number of bins to generate
//
// Returns MetricHistoPoints with metric name, unit, footprint stat type, and binned data.
//
// Algorithm:
//  1. Determine peak value from cluster configuration (filtered cluster or max across all)
//  2. Generate SQL that extracts footprint value, normalizes to [0,1], multiplies by bin count
//  3. Pre-initialize bins with min/max ranges based on peak value
//  4. Query database for counts per bin
//  5. Match results to pre-initialized bins
//
// Special handling: Values exactly equal to peak are forced into the last bin by multiplying
// peak by 0.999999999 to avoid creating an extra bin.
func (r *JobRepository) jobsMetricStatisticsHistogram(
	ctx context.Context,
	metric string,
	filters []*model.JobFilter,
	bins *int,
) (*model.MetricHistoPoints, error) {
	// Peak value defines the upper bound for binning: values are distributed across
	// bins from 0 to peak. First try to get peak from filtered cluster, otherwise
	// scan all clusters to find the maximum peak value.
	var metricConfig *schema.MetricConfig
	var peak float64
	var unit string
	var footprintStat string

	// Try to get metric config from filtered cluster
	for _, f := range filters {
		if f.Cluster != nil {
			metricConfig = archive.GetMetricConfig(*f.Cluster.Eq, metric)
			peak = metricConfig.Peak
			unit = metricConfig.Unit.Prefix + metricConfig.Unit.Base
			footprintStat = metricConfig.Footprint
			cclog.Debugf("Cluster %s filter found with peak %f for %s", *f.Cluster.Eq, peak, metric)
		}
	}

	// If no cluster filter or peak not found, find largest peak across all clusters
	// This ensures histogram can accommodate all possible values
	if peak == 0.0 {
		for _, c := range archive.Clusters {
			for _, m := range c.MetricConfig {
				if m.Name == metric {
					if m.Peak > peak {
						peak = m.Peak
					}
					if unit == "" {
						unit = m.Unit.Prefix + m.Unit.Base
					}
					if footprintStat == "" {
						footprintStat = m.Footprint
					}
				}
			}
		}
	}

	// Construct SQL histogram bins using normalized values.
	// Algorithm based on: https://jereze.com/code/sql-histogram/ (modified)
	start := time.Now()

	// Bin calculation formula:
	//   bin_number = CAST( (value / peak) * num_bins AS INTEGER ) + 1
	// Special case: value == peak would create bin N+1, so we test for equality
	// and multiply peak by 0.999999999 to force it into bin N.
	binQuery := fmt.Sprintf(`CAST(
		((case when json_extract(footprint, "$.%s") = %f then %f*0.999999999 else json_extract(footprint, "$.%s") end) / %f)
		* %v as INTEGER )`,
		(metric + "_" + footprintStat), peak, peak, (metric + "_" + footprintStat), peak, *bins)

	mainQuery := sq.Select(
		fmt.Sprintf(`%s + 1 as bin`, binQuery),
		`count(*) as count`,
	).From("job").Where(
		"JSON_VALID(footprint)",
	).Where(fmt.Sprintf(`json_extract(footprint, "$.%s") is not null and json_extract(footprint, "$.%s") <= %f`, (metric + "_" + footprintStat), (metric + "_" + footprintStat), peak))

	mainQuery, qerr := SecurityCheck(ctx, mainQuery)
	if qerr != nil {
		return nil, qerr
	}

	for _, f := range filters {
		mainQuery = BuildWhereClause(f, mainQuery)
	}

	mainQuery = mainQuery.GroupBy("bin").OrderBy("bin")

	rows, err := mainQuery.RunWith(r.DB).QueryContext(ctx)
	if err != nil {
		cclog.Errorf("Error while running mainQuery: %s", err)
		return nil, err
	}
	defer rows.Close()

	// Pre-initialize bins with calculated min/max ranges.
	// Example: peak=1000, bins=10 -> bin 1=[0,100), bin 2=[100,200), ..., bin 10=[900,1000]
	points := make([]*model.MetricHistoPoint, 0, *bins)
	binStep := int(peak) / *bins
	for i := 1; i <= *bins; i++ {
		binMin := (binStep * (i - 1))
		binMax := (binStep * i)
		idx := i
		epoint := model.MetricHistoPoint{Bin: &idx, Count: 0, Min: &binMin, Max: &binMax}
		points = append(points, &epoint)
	}

	// Build a map from bin number to slice index for O(1) lookup.
	binMap := make(map[int]int, len(points))
	for i, p := range points {
		if p.Bin != nil {
			binMap[*p.Bin] = i
		}
	}

	// Match query results to pre-initialized bins using map lookup.
	for rows.Next() {
		rpoint := model.MetricHistoPoint{}
		if err := rows.Scan(&rpoint.Bin, &rpoint.Count); err != nil {
			cclog.Warnf("Error while scanning rows for %s", metric)
			return nil, err
		}

		if rpoint.Bin != nil {
			if idx, ok := binMap[*rpoint.Bin]; ok {
				points[idx].Count = rpoint.Count
			}
		}
	}

	if err := rows.Err(); err != nil {
		return nil, err
	}

	result := model.MetricHistoPoints{Metric: metric, Unit: unit, Stat: &footprintStat, Data: points}

	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
	return &result, nil
}

// runningJobsMetricStatisticsHistogram generates metric histograms for running jobs using live data.
//
// Unlike completed jobs which use footprint data from the database, running jobs require
// fetching current metric averages from the metric backend (via metricdispatch).
//
// Parameters:
//   - metrics: List of metric names
//   - filters: Job filters (should filter to running jobs only)
//   - bins: Number of histogram bins
//
// Returns slice of MetricHistoPoints, one per metric.
//
// Limitations:
//   - Maximum 5000 jobs (returns nil if more jobs match)
//   - Requires metric backend availability
//   - Bins based on metric peak values from cluster configuration
//
// Algorithm:
//  1. Query first 5001 jobs to check count limit
//  2. Load metric averages for all jobs via metricdispatch
//  3. For each metric, create bins based on peak value
//  4. Iterate averages and count jobs per bin
func (r *JobRepository) runningJobsMetricStatisticsHistogram(
	ctx context.Context,
	metrics []string,
	filters []*model.JobFilter,
	bins *int,
) []*model.MetricHistoPoints {
	// Get Jobs
	jobs, err := r.QueryJobs(ctx, filters, &model.PageRequest{Page: 1, ItemsPerPage: 5000 + 1}, nil)
	if err != nil {
		cclog.Errorf("Error while querying jobs for footprint: %s", err)
		return nil
	}
	if len(jobs) > 5000 {
		cclog.Errorf("too many jobs matched (max: %d)", 5000)
		return nil
	}

	// Get AVGs from metric repo
	avgs := make([][]schema.Float, len(metrics))
	for i := range avgs {
		avgs[i] = make([]schema.Float, 0, len(jobs))
	}

	for _, job := range jobs {
		if job.MonitoringStatus == schema.MonitoringStatusDisabled || job.MonitoringStatus == schema.MonitoringStatusArchivingFailed {
			continue
		}

		if err := metricdispatch.LoadAverages(job, metrics, avgs, ctx); err != nil {
			cclog.Errorf("Error while loading averages for histogram: %s", err)
			return nil
		}
	}

	// Iterate metrics to fill endresult
	data := make([]*model.MetricHistoPoints, 0)
	for idx, metric := range metrics {
		// Get specific Peak or largest Peak
		var metricConfig *schema.MetricConfig
		var peak float64
		var unit string

		for _, f := range filters {
			if f.Cluster != nil {
				metricConfig = archive.GetMetricConfig(*f.Cluster.Eq, metric)
				peak = metricConfig.Peak
				unit = metricConfig.Unit.Prefix + metricConfig.Unit.Base
			}
		}

		if peak == 0.0 {
			for _, c := range archive.Clusters {
				for _, m := range c.MetricConfig {
					if m.Name == metric {
						if m.Peak > peak {
							peak = m.Peak
						}
						if unit == "" {
							unit = m.Unit.Prefix + m.Unit.Base
						}
					}
				}
			}
		}

		// Make and fill bins
		peakBin := int(peak) / *bins

		points := make([]*model.MetricHistoPoint, 0)
		for b := 0; b < *bins; b++ {
			count := 0
			bindex := b + 1
			bmin := peakBin * b
			bmax := peakBin * (b + 1)

			// Iterate AVG values for indexed metric and count for bins
			for _, val := range avgs[idx] {
				if int(val) >= bmin && int(val) < bmax {
					count += 1
				}
			}

			// Append Bin to Metric Result Array
			point := model.MetricHistoPoint{Bin: &bindex, Count: count, Min: &bmin, Max: &bmax}
			points = append(points, &point)
		}

		// Append Metric Result Array to final results array
		result := model.MetricHistoPoints{Metric: metric, Unit: unit, Data: points}
		data = append(data, &result)
	}

	return data
}


Now I'll apply all the changes in parallel.


Write

-- Migration 12: Add covering index for grouped stats queries
-- Column order: cluster (equality), hpc_user (GROUP BY), start_time (range scan)
-- Includes aggregated columns to avoid main table lookups entirely.

DROP INDEX IF EXISTS jobs_cluster_starttime_user_stats;

CREATE INDEX IF NOT EXISTS jobs_cluster_user_starttime_stats
  ON job (cluster, hpc_user, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);

PRAGMA optimize;


The file /Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/migrations/sqlite3/12_stats-covering-index.up.sql has been updated successfully.

-- Migration 12: Add covering index for grouped stats queries
-- Column order: cluster (equality), hpc_user (GROUP BY), start_time (range scan)
-- Includes aggregated columns to avoid main table lookups entirely.

DROP INDEX IF EXISTS jobs_cluster_starttime_user_stats;

CREATE INDEX IF NOT EXISTS jobs_cluster_user_starttime_stats
  ON job (cluster, hpc_user, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);

PRAGMA optimize;

--- Covers the common pattern: WHERE cluster=X AND start_time >= Y GROUP BY hpc_user

+-- Column order: cluster (equality), hpc_user (GROUP BY), start_time (range scan)

-CREATE INDEX IF NOT EXISTS jobs_cluster_starttime_user_stats

-  ON job (cluster, start_time, hpc_user, duration, job_state, num_nodes, num_hwthreads, num_acc);

+DROP INDEX IF EXISTS jobs_cluster_starttime_user_stats;

+CREATE INDEX IF NOT EXISTS jobs_cluster_user_starttime_stats

+  ON job (cluster, hpc_user, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);

-- Migration 12: Add covering index for grouped stats queries
-- Covers the common pattern: WHERE cluster=X AND start_time >= Y GROUP BY hpc_user
-- Includes aggregated columns to avoid main table lookups entirely.

CREATE INDEX IF NOT EXISTS jobs_cluster_starttime_user_stats
  ON job (cluster, start_time, hpc_user, duration, job_state, num_nodes, num_hwthreads, num_acc);

PRAGMA optimize;


DROP INDEX IF EXISTS jobs_cluster_user_starttime_stats;


The file /Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/migrations/sqlite3/12_stats-covering-index.down.sql has been updated successfully.

-DROP INDEX IF EXISTS jobs_cluster_starttime_user_stats;

+DROP INDEX IF EXISTS jobs_cluster_user_starttime_stats;


Edit

durationExpr := fmt.Sprintf(`(CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END)`, now)

durationExpr := fmt.Sprintf(`(CASE WHEN job.job_state = 'running' THEN %d - job.start_time ELSE job.duration END)`, now)