Implement the following plan:

# Inline literal job_state in remaining parameterized sites

## Files to change

- **`internal/repository/stats.go:116`** (`buildCountQuery`) — change `query.Where("job.job_state = ?", "running")` → `query.Where("job.job_state = 'running'")`. Highest impact: called by dashboard count queries.
- **`internal/repository/job.go:1069`** (`GetUsedNodes`) — change `sq.Eq{"job.job_state": "running"}` → `query.Where("job.job_state = 'running'")`. Benefits node allocation queries.

## Sites intentionally NOT changed

- **`jobFind.go:280,285`** (`FindConcurrentJobs`) — equality/inequality on job_state isn't the leading filter here (time overlap + resources dominate), so partial index unlikely to help. Also uses `!=` which doesn't match `WHERE job_state = 'running'` partial indexes.

## Verification
```bash
go build ./cmd/cc-backend && go test ./internal/repository/...
```

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-ClusterCockpit-cc-backend/889ef150-c6ca-4f35-81e5-c19dc95fe399.jsonl


Let me read the two files to see the exact code at those locations.


   110→	} else {
   111→		query = sq.Select("COUNT(*)").From("job")
   112→	}
   113→
   114→	switch kind {
   115→	case "running":
   116→		query = query.Where("job.job_state = ?", "running")
   117→	case "short":
   118→		query = query.Where("job.duration < ?", config.Keys.ShortRunningJobsDuration)
   119→	}
   120→
   121→	for _, f := range filter {
   122→		query = BuildWhereClause(f, query)
   123→	}
   124→

	} else {
		query = sq.Select("COUNT(*)").From("job")
	}

	switch kind {
	case "running":
		query = query.Where("job.job_state = ?", "running")
	case "short":
		query = query.Where("job.duration < ?", config.Keys.ShortRunningJobsDuration)
	}

	for _, f := range filter {
		query = BuildWhereClause(f, query)
	}


  1063→// encounters errors. Individual row parsing errors are logged but don't fail
  1064→// the entire operation.
  1065→func (r *JobRepository) GetUsedNodes(ts int64) (map[string][]string, error) {
  1066→	// Note: Query expects index on (job_state, start_time) for optimal performance
  1067→	q := sq.Select("job.cluster", "job.resources").From("job").
  1068→		Where("job.start_time < ?", ts).
  1069→		Where(sq.Eq{"job.job_state": "running"})
  1070→
  1071→	rows, err := q.RunWith(r.stmtCache).Query()
  1072→	if err != nil {
  1073→		queryString, queryVars, _ := q.ToSql()
  1074→		return nil, fmt.Errorf("query failed [%s] %v: %w", queryString, queryVars, err)
  1075→	}
  1076→	defer rows.Close()
  1077→

// encounters errors. Individual row parsing errors are logged but don't fail
// the entire operation.
func (r *JobRepository) GetUsedNodes(ts int64) (map[string][]string, error) {
	// Note: Query expects index on (job_state, start_time) for optimal performance
	q := sq.Select("job.cluster", "job.resources").From("job").
		Where("job.start_time < ?", ts).
		Where(sq.Eq{"job.job_state": "running"})

	rows, err := q.RunWith(r.stmtCache).Query()
	if err != nil {
		queryString, queryVars, _ := q.ToSql()
		return nil, fmt.Errorf("query failed [%s] %v: %w", queryString, queryVars, err)
	}
	defer rows.Close()


Edit

query = query.Where("job.job_state = ?", "running")

query = query.Where("job.job_state = 'running'")


The file /Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/stats.go has been updated successfully.

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// This file contains job statistics and histogram generation functionality for the JobRepository.
//
// # Job Statistics
//
// The statistics methods provide aggregated metrics about jobs including total jobs, users,
// walltime, and resource usage (nodes, cores, accelerators). Statistics can be computed:
//   - Overall (JobsStats): Single aggregate across all matching jobs
//   - Grouped (JobsStatsGrouped): Aggregated by user, project, cluster, or subcluster
//   - Counts (JobCountGrouped, AddJobCount): Simple job counts with optional filtering
//
// All statistics methods support filtering via JobFilter and respect security contexts.
//
// # Histograms
//
// Histogram methods generate distribution data for visualization:
//   - Duration, nodes, cores, accelerators (AddHistograms)
//   - Job metrics like CPU load, memory usage (AddMetricHistograms)
//
// Histograms use intelligent binning:
//   - Duration: Variable bin sizes (1m, 10m, 1h, 6h, 12h, 24h) with zero-padding
//   - Resources: Natural value-based bins
//   - Metrics: Normalized to peak values with configurable bin counts
//
// # Running vs. Completed Jobs
//
// Statistics handle running jobs specially:
//   - Duration calculated as (now - start_time) for running jobs
//   - Metric histograms for running jobs load data from metric backend instead of footprint
//   - Job state filtering distinguishes running/completed jobs
//
// # Performance Considerations
//
// - All queries use prepared statements via stmtCache
// - Complex aggregations use SQL for efficiency
// - Histogram pre-initialization ensures consistent bin ranges
// - Metric histogram queries limited to 5000 jobs for running job analysis

package repository

import (
	"context"
	"database/sql"
	"fmt"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/config"
	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
	"github.com/ClusterCockpit/cc-backend/internal/metricdispatch"
	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	sq "github.com/Masterminds/squirrel"
	"golang.org/x/sync/errgroup"
)

// groupBy2column maps GraphQL Aggregate enum values to their corresponding database column names.
// Used by JobsStatsGrouped and JobCountGrouped to translate user-facing grouping dimensions
// into SQL GROUP BY clauses. GraphQL validation ensures only valid enum values are accepted.
var groupBy2column = map[model.Aggregate]string{
	model.AggregateUser:       "job.hpc_user",
	model.AggregateProject:    "job.project",
	model.AggregateCluster:    "job.cluster",
	model.AggregateSubcluster: "job.subcluster",
}

// sortBy2column maps GraphQL SortByAggregate enum values to their corresponding computed column names.
// Used by JobsStatsGrouped to translate sort preferences into SQL ORDER BY clauses.
// Column names match the AS aliases used in buildStatsQuery.
var sortBy2column = map[model.SortByAggregate]string{
	model.SortByAggregateTotaljobs:      "totalJobs",
	model.SortByAggregateTotalusers:     "totalUsers",
	model.SortByAggregateTotalwalltime:  "totalWalltime",
	model.SortByAggregateTotalnodes:     "totalNodes",
	model.SortByAggregateTotalnodehours: "totalNodeHours",
	model.SortByAggregateTotalcores:     "totalCores",
	model.SortByAggregateTotalcorehours: "totalCoreHours",
	model.SortByAggregateTotalaccs:      "totalAccs",
	model.SortByAggregateTotalacchours:  "totalAccHours",
}

// buildCountQuery constructs a SQL query to count jobs with optional grouping and filtering.
//
// Parameters:
//   - filter: Job filters to apply (cluster, user, time range, etc.)
//   - kind: Special filter - "running" for running jobs only, "short" for jobs under threshold
//   - col: Column name to GROUP BY; empty string for total count without grouping
//
// Returns a SelectBuilder that produces either:
//   - Single count: COUNT(job.id) when col is empty
//   - Grouped counts: col, COUNT(job.id) when col is specified
//
// The kind parameter enables counting specific job categories:
//   - "running": Only jobs with job_state = 'running'
//   - "short": Only jobs with duration < ShortRunningJobsDuration config value
//   - empty: All jobs matching filters
func (r *JobRepository) buildCountQuery(
	filter []*model.JobFilter,
	kind string,
	col string,
) sq.SelectBuilder {
	var query sq.SelectBuilder

	if col != "" {
		query = sq.Select(col, "COUNT(*)").From("job").GroupBy(col)
	} else {
		query = sq.Select("COUNT(*)").From("job")
	}

	switch kind {
	case "running":
		query = query.Where("job.job_state = ?", "running")
	case "short":
		query = query.Where("job.duration < ?", config.Keys.ShortRunningJobsDuration)
	}

	for _, f := range filter {
		query = BuildWhereClause(f, query)
	}

	return query
}

// buildStatsQuery constructs a SQL query to compute job statistics with optional grouping.
// Only requested columns are computed; unrequested columns select 0 as placeholder.
//
// Parameters:
//   - filter: Job filters to apply (cluster, user, time range, etc.)
//   - col: Column name to GROUP BY; empty string for overall statistics without grouping
//   - shortThreshold: Duration threshold in seconds for counting short-running jobs
//   - reqFields: Set of requested field names; nil means compute all fields
func (r *JobRepository) buildStatsQuery(
	filter []*model.JobFilter,
	col string,
	shortThreshold int,
	reqFields map[string]bool,
) sq.SelectBuilder {
	now := time.Now().Unix()

	// Helper: return real expression if field is requested (or reqFields is nil), else "0 as alias"
	need := func(field string) bool {
		return reqFields == nil || reqFields[field]
	}
	durationExpr := fmt.Sprintf(`(CASE WHEN job.job_state = 'running' THEN %d - job.start_time ELSE job.duration END)`, now)

	// Build column list
	columns := make([]string, 0, 14)

	if col != "" {
		columns = append(columns, col)
	}

	columns = append(columns, "COUNT(*) as totalJobs")

	if need("totalUsers") && col != "job.hpc_user" {
		columns = append(columns, "COUNT(DISTINCT job.hpc_user) AS totalUsers")
	} else {
		columns = append(columns, "0 AS totalUsers")
	}

	if need("totalWalltime") {
		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s) / 3600) as int) as totalWalltime`, durationExpr))
	} else {
		columns = append(columns, "0 as totalWalltime")
	}

	if need("totalNodes") {
		columns = append(columns, `CAST(SUM(job.num_nodes) as int) as totalNodes`)
	} else {
		columns = append(columns, "0 as totalNodes")
	}

	if need("totalNodeHours") {
		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s * job.num_nodes) / 3600) as int) as totalNodeHours`, durationExpr))
	} else {
		columns = append(columns, "0 as totalNodeHours")
	}

	if need("totalCores") {
		columns = append(columns, `CAST(SUM(job.num_hwthreads) as int) as totalCores`)
	} else {
		columns = append(columns, "0 as totalCores")
	}

	if need("totalCoreHours") {
		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s * job.num_hwthreads) / 3600) as int) as totalCoreHours`, durationExpr))
	} else {
		columns = append(columns, "0 as totalCoreHours")
	}

	if need("totalAccs") {
		columns = append(columns, `CAST(SUM(job.num_acc) as int) as totalAccs`)
	} else {
		columns = append(columns, "0 as totalAccs")
	}

	if need("totalAccHours") {
		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s * job.num_acc) / 3600) as int) as totalAccHours`, durationExpr))
	} else {
		columns = append(columns, "0 as totalAccHours")
	}

	if need("runningJobs") {
		columns = append(columns, `COUNT(CASE WHEN job.job_state = 'running' THEN 1 END) as runningJobs`)
	} else {
		columns = append(columns, "0 as runningJobs")
	}

	if need("shortJobs") {
		columns = append(columns, fmt.Sprintf(`COUNT(CASE WHEN job.duration < %d THEN 1 END) as shortJobs`, shortThreshold))
	} else {
		columns = append(columns, "0 as shortJobs")
	}

	query := sq.Select(columns...).From("job")
	if col != "" {
		query = query.GroupBy(col)
	}

	for _, f := range filter {
		query = BuildWhereClause(f, query)
	}

	return query
}

// JobsStatsGrouped computes job statistics grouped by a dimension (user, project, cluster, or subcluster).
// Only columns listed in reqFields are computed; others return 0. User display names are looked up
// in a separate lightweight query to avoid JOIN overhead on the main aggregation.
func (r *JobRepository) JobsStatsGrouped(
	ctx context.Context,
	filter []*model.JobFilter,
	page *model.PageRequest,
	sortBy *model.SortByAggregate,
	groupBy *model.Aggregate,
	reqFields map[string]bool,
) ([]*model.JobsStatistics, error) {
	col := groupBy2column[*groupBy]
	query := r.buildStatsQuery(filter, col, config.Keys.ShortRunningJobsDuration, reqFields)

	query, err := SecurityCheck(ctx, query)
	if err != nil {
		return nil, err
	}

	if sortBy != nil {
		sortBy := sortBy2column[*sortBy]
		query = query.OrderBy(fmt.Sprintf("%s DESC", sortBy))
	}
	if page != nil && page.ItemsPerPage != -1 {
		limit := uint64(page.ItemsPerPage)
		query = query.Offset((uint64(page.Page) - 1) * limit).Limit(limit)
	}

	rows, err := query.RunWith(r.DB).QueryContext(ctx)
	if err != nil {
		cclog.Warn("Error while querying DB for job statistics")
		return nil, err
	}
	defer rows.Close()

	stats := make([]*model.JobsStatistics, 0, 100)

	for rows.Next() {
		var id sql.NullString
		var jobs, users, walltime, nodes, nodeHours, cores, coreHours, accs, accHours, runningJobs, shortJobs sql.NullInt64
		if err := rows.Scan(&id, &jobs, &users, &walltime, &nodes, &nodeHours, &cores, &coreHours, &accs, &accHours, &runningJobs, &shortJobs); err != nil {
			cclog.Warnf("Error while scanning rows: %s", err.Error())
			return nil, err
		}

		if id.Valid {
			stats = append(stats,
				&model.JobsStatistics{
					ID:             id.String,
					TotalJobs:      int(jobs.Int64),
					TotalUsers:     int(users.Int64),
					TotalWalltime:  int(walltime.Int64),
					TotalNodes:     int(nodes.Int64),
					TotalNodeHours: int(nodeHours.Int64),
					TotalCores:     int(cores.Int64),
					TotalCoreHours: int(coreHours.Int64),
					TotalAccs:      int(accs.Int64),
					TotalAccHours:  int(accHours.Int64),
					RunningJobs:    int(runningJobs.Int64),
					ShortJobs:      int(shortJobs.Int64),
				})
		}
	}

	if err := rows.Err(); err != nil {
		return nil, err
	}

	// Post-query name lookup for user grouping (avoids LEFT JOIN on aggregation query)
	if col == "job.hpc_user" && len(stats) > 0 {
		usernames := make([]any, len(stats))
		for i, s := range stats {
			usernames[i] = s.ID
		}

		nameQuery := sq.Select("username", "name").From("hpc_user").Where(sq.Eq{"username": usernames})
		nameRows, err := nameQuery.RunWith(r.DB).QueryContext(ctx)
		if err != nil {
			cclog.Warnf("Error looking up user names: %s", err.Error())
			// Non-fatal: stats are still valid without display names
		} else {
			defer nameRows.Close()
			nameMap := make(map[string]string, len(stats))
			for nameRows.Next() {
				var username, name string
				if err := nameRows.Scan(&username, &name); err == nil {
					nameMap[username] = name
				}
			}
			for _, s := range stats {
				if name, ok := nameMap[s.ID]; ok {
					s.Name = name
				}
			}
		}
	}

	return stats, nil
}

// JobsStats computes overall job statistics across all matching jobs without grouping.
//
// This method provides a single aggregate view of job metrics, useful for dashboard
// summaries and overall system utilization reports.
//
// Parameters:
//   - ctx: Context for security checks and cancellation
//   - filter: Filters to apply (time range, cluster, job state, etc.)
//
// Returns a single-element slice containing aggregate statistics:
//   - totalJobs, totalUsers, totalWalltime
//   - totalNodeHours, totalCoreHours, totalAccHours
//
// Unlike JobsStatsGrouped, this returns overall totals without breaking down by dimension.
// Security checks are applied via SecurityCheck to respect user access levels.
func (r *JobRepository) JobsStats(
	ctx context.Context,
	filter []*model.JobFilter,
	reqFields map[string]bool,
) ([]*model.JobsStatistics, error) {
	start := time.Now()
	query := r.buildStatsQuery(filter, "", config.Keys.ShortRunningJobsDuration, reqFields)
	query, err := SecurityCheck(ctx, query)
	if err != nil {
		return nil, err
	}

	row := query.RunWith(r.DB).QueryRowContext(ctx)
	stats := make([]*model.JobsStatistics, 0, 1)

	var jobs, users, walltime, nodes, nodeHours, cores, coreHours, accs, accHours, runningJobs, shortJobs sql.NullInt64
	if err := row.Scan(&jobs, &users, &walltime, &nodes, &nodeHours, &cores, &coreHours, &accs, &accHours, &runningJobs, &shortJobs); err != nil {
		cclog.Warn("Error while scanning rows")
		return nil, err
	}

	if jobs.Valid {
		var totalNodeHours, totalCoreHours, totalAccHours int

		if nodeHours.Valid {
			totalNodeHours = int(nodeHours.Int64)
		}
		if coreHours.Valid {
			totalCoreHours = int(coreHours.Int64)
		}
		if accHours.Valid {
			totalAccHours = int(accHours.Int64)
		}
		stats = append(stats,
			&model.JobsStatistics{
				TotalJobs:      int(jobs.Int64),
				TotalUsers:     int(users.Int64),
				TotalWalltime:  int(walltime.Int64),
				TotalNodeHours: totalNodeHours,
				TotalCoreHours: totalCoreHours,
				TotalAccHours:  totalAccHours,
				RunningJobs:    int(runningJobs.Int64),
				ShortJobs:      int(shortJobs.Int64),
			})
	}

	cclog.Debugf("Timer JobStats %s", time.Since(start))
	return stats, nil
}

// LoadJobStat retrieves a specific statistic for a metric from a job's statistics.
// Returns 0.0 if the metric is not found or statType is invalid.
//
// Parameters:
//   - job: Job struct with populated Statistics field
//   - metric: Name of the metric to query (e.g., "cpu_load", "mem_used")
//   - statType: Type of statistic: "avg", "min", or "max"
//
// Returns the requested statistic value or 0.0 if not found.
func LoadJobStat(job *schema.Job, metric string, statType string) float64 {
	if stats, ok := job.Statistics[metric]; ok {
		switch statType {
		case "avg":
			return stats.Avg
		case "max":
			return stats.Max
		case "min":
			return stats.Min
		default:
			cclog.Errorf("Unknown stat type %s", statType)
		}
	}

	return 0.0
}

// JobCountGrouped counts jobs grouped by a dimension without computing detailed statistics.
//
// This is a lightweight alternative to JobsStatsGrouped when only job counts are needed,
// avoiding the overhead of calculating walltime and resource usage metrics.
//
// Parameters:
//   - ctx: Context for security checks
//   - filter: Filters to apply
//   - groupBy: Grouping dimension (User, Project, Cluster, or SubCluster)
//
// Returns JobsStatistics with only ID and TotalJobs populated for each group.
func (r *JobRepository) JobCountGrouped(
	ctx context.Context,
	filter []*model.JobFilter,
	groupBy *model.Aggregate,
) ([]*model.JobsStatistics, error) {
	start := time.Now()
	col := groupBy2column[*groupBy]
	query := r.buildCountQuery(filter, "", col)
	query, err := SecurityCheck(ctx, query)
	if err != nil {
		return nil, err
	}
	rows, err := query.RunWith(r.DB).QueryContext(ctx)
	if err != nil {
		cclog.Warn("Error while querying DB for job statistics")
		return nil, err
	}
	defer rows.Close()

	stats := make([]*model.JobsStatistics, 0, 100)

	for rows.Next() {
		var id sql.NullString
		var cnt sql.NullInt64
		if err := rows.Scan(&id, &cnt); err != nil {
			cclog.Warn("Error while scanning rows")
			return nil, err
		}
		if id.Valid {
			stats = append(stats,
				&model.JobsStatistics{
					ID:        id.String,
					TotalJobs: int(cnt.Int64),
				})
		}
	}

	if err := rows.Err(); err != nil {
		return nil, err
	}

	cclog.Debugf("Timer JobCountGrouped %s", time.Since(start))
	return stats, nil
}

// AddJobCountGrouped augments existing statistics with additional job counts by category.
//
// This method enriches JobsStatistics returned by JobsStatsGrouped or JobCountGrouped
// with counts of running or short-running (based on ShortRunningJobsDuration) jobs, matched by group ID.
//
// Parameters:
//   - ctx: Context for security checks
//   - filter: Filters to apply
//   - groupBy: Grouping dimension (must match the dimension used for stats parameter)
//   - stats: Existing statistics to augment (modified in-place by ID matching)
//   - kind: "running" to add RunningJobs count, "short" to add ShortJobs count
//
// Returns the same stats slice with RunningJobs or ShortJobs fields populated per group.
// Groups without matching jobs will have 0 for the added field.
func (r *JobRepository) AddJobCountGrouped(
	ctx context.Context,
	filter []*model.JobFilter,
	groupBy *model.Aggregate,
	stats []*model.JobsStatistics,
	kind string,
) ([]*model.JobsStatistics, error) {
	start := time.Now()
	col := groupBy2column[*groupBy]
	query := r.buildCountQuery(filter, kind, col)
	query, err := SecurityCheck(ctx, query)
	if err != nil {
		return nil, err
	}
	rows, err := query.RunWith(r.DB).QueryContext(ctx)
	if err != nil {
		cclog.Warn("Error while querying DB for job statistics")
		return nil, err
	}
	defer rows.Close()

	counts := make(map[string]int)

	for rows.Next() {
		var id sql.NullString
		var cnt sql.NullInt64
		if err := rows.Scan(&id, &cnt); err != nil {
			cclog.Warn("Error while scanning rows")
			return nil, err
		}
		if id.Valid {
			counts[id.String] = int(cnt.Int64)
		}
	}

	if err := rows.Err(); err != nil {
		return nil, err
	}

	switch kind {
	case "running":
		for _, s := range stats {
			s.RunningJobs = counts[s.ID]
		}
	case "short":
		for _, s := range stats {
			s.ShortJobs = counts[s.ID]
		}
	}

	cclog.Debugf("Timer AddJobCountGrouped %s", time.Since(start))
	return stats, nil
}

// AddJobCount augments existing overall statistics with additional job counts by category.
//
// Similar to AddJobCountGrouped but for ungrouped statistics. Applies the same count
// to all statistics entries (typically just one).
//
// Parameters:
//   - ctx: Context for security checks
//   - filter: Filters to apply
//   - stats: Existing statistics to augment (modified in-place)
//   - kind: "running" to add RunningJobs count, "short" to add ShortJobs count
//
// Returns the same stats slice with RunningJobs or ShortJobs fields set to the total count.
func (r *JobRepository) AddJobCount(
	ctx context.Context,
	filter []*model.JobFilter,
	stats []*model.JobsStatistics,
	kind string,
) ([]*model.JobsStatistics, error) {
	start := time.Now()
	query := r.buildCountQuery(filter, kind, "")
	query, err := SecurityCheck(ctx, query)
	if err != nil {
		return nil, err
	}
	var cnt sql.NullInt64
	if err := query.RunWith(r.DB).QueryRowContext(ctx).Scan(&cnt); err != nil {
		cclog.Warn("Error while querying DB for job count")
		return nil, err
	}

	count := int(cnt.Int64)

	switch kind {
	case "running":
		for _, s := range stats {
			s.RunningJobs = count
		}
	case "short":
		for _, s := range stats {
			s.ShortJobs = count
		}
	}

	cclog.Debugf("Timer AddJobCount %s", time.Since(start))
	return stats, nil
}

// AddHistograms augments statistics with distribution histograms for job properties.
//
// Generates histogram data for visualization of job duration, node count, core count,
// and accelerator count distributions. Duration histogram uses intelligent binning based
// on the requested resolution.
//
// Parameters:
//   - ctx: Context for security checks
//   - filter: Filters to apply to jobs included in histograms
//   - stat: Statistics struct to augment (modified in-place)
//   - durationBins: Bin size - "1m", "10m", "1h", "6h", "12h", or "24h" (default)
//
// Populates these fields in stat:
//   - HistDuration: Job duration distribution (zero-padded bins)
//   - HistNumNodes: Node count distribution
//   - HistNumCores: Core (hwthread) count distribution
//   - HistNumAccs: Accelerator count distribution
//
// Duration bins are pre-initialized with zeros to ensure consistent ranges for visualization.
// Bin size determines both the width and maximum duration displayed (e.g., "1h" = 48 bins × 1h = 48h max).
func (r *JobRepository) AddHistograms(
	ctx context.Context,
	filter []*model.JobFilter,
	stat *model.JobsStatistics,
	durationBins *string,
) (*model.JobsStatistics, error) {
	start := time.Now()

	var targetBinCount int
	var targetBinSize int
	switch *durationBins {
	case "1m": // 1 Minute Bins + Max 60 Bins -> Max 60 Minutes
		targetBinCount = 60
		targetBinSize = 60
	case "10m": // 10 Minute Bins + Max 72 Bins -> Max 12 Hours
		targetBinCount = 72
		targetBinSize = 600
	case "1h": // 1 Hour Bins + Max 48 Bins -> Max 48 Hours
		targetBinCount = 48
		targetBinSize = 3600
	case "6h": // 6 Hour Bins + Max 12 Bins -> Max 3 Days
		targetBinCount = 12
		targetBinSize = 21600
	case "12h": // 12 hour Bins + Max 14 Bins -> Max 7 Days
		targetBinCount = 14
		targetBinSize = 43200
	default: // 24h
		targetBinCount = 24
		targetBinSize = 3600
	}

	// Run all 4 histogram queries in parallel — each writes a distinct struct field.
	g, gctx := errgroup.WithContext(ctx)

	value := fmt.Sprintf(`CAST(ROUND(((CASE WHEN job.job_state = 'running' THEN %d - job.start_time ELSE job.duration END) / %d) + 1) as int) as value`, time.Now().Unix(), targetBinSize)

	g.Go(func() error {
		var err error
		stat.HistDuration, err = r.jobsDurationStatisticsHistogram(gctx, value, filter, targetBinSize, &targetBinCount)
		if err != nil {
			cclog.Warn("Error while loading job statistics histogram: job duration")
		}
		return err
	})
	g.Go(func() error {
		var err error
		stat.HistNumNodes, err = r.jobsStatisticsHistogram(gctx, "job.num_nodes as value", filter)
		if err != nil {
			cclog.Warn("Error while loading job statistics histogram: num nodes")
		}
		return err
	})
	g.Go(func() error {
		var err error
		stat.HistNumCores, err = r.jobsStatisticsHistogram(gctx, "job.num_hwthreads as value", filter)
		if err != nil {
			cclog.Warn("Error while loading job statistics histogram: num hwthreads")
		}
		return err
	})
	g.Go(func() error {
		var err error
		stat.HistNumAccs, err = r.jobsStatisticsHistogram(gctx, "job.num_acc as value", filter)
		if err != nil {
			cclog.Warn("Error while loading job statistics histogram: num acc")
		}
		return err
	})

	if err := g.Wait(); err != nil {
		return nil, err
	}

	cclog.Debugf("Timer AddHistograms %s", time.Since(start))
	return stat, nil
}

// AddMetricHistograms augments statistics with distribution histograms for job metrics.
//
// Generates histogram data for metrics like CPU load, memory usage, etc. Handles running
// and completed jobs differently: running jobs load data from metric backend, completed jobs
// use footprint data from database.
//
// Parameters:
//   - ctx: Context for security checks
//   - filter: Filters to apply (MUST contain State filter for running jobs)
//   - metrics: List of metric names to histogram (e.g., ["cpu_load", "mem_used"])
//   - stat: Statistics struct to augment (modified in-place)
//   - targetBinCount: Number of histogram bins (default: 10)
//
// Populates HistMetrics field in stat with MetricHistoPoints for each metric.
//
// Binning algorithm:
//   - Values normalized to metric's peak value from cluster configuration
//   - Bins evenly distributed from 0 to peak
//   - Pre-initialized with zeros for consistent visualization
//
// Limitations:
//   - Running jobs: Limited to 5000 jobs for performance
//   - Requires valid cluster configuration with metric peak values
//   - Uses footprint statistic (avg/max/min) configured per metric
func (r *JobRepository) AddMetricHistograms(
	ctx context.Context,
	filter []*model.JobFilter,
	metrics []string,
	stat *model.JobsStatistics,
	targetBinCount *int,
) (*model.JobsStatistics, error) {
	start := time.Now()

	// Running Jobs Only: First query jobdata from sqlite, then query data and make bins
	for _, f := range filter {
		if f.State != nil {
			if len(f.State) == 1 && f.State[0] == "running" {
				stat.HistMetrics = r.runningJobsMetricStatisticsHistogram(ctx, metrics, filter, targetBinCount)
				cclog.Debugf("Timer AddMetricHistograms %s", time.Since(start))
				return stat, nil
			}
		}
	}

	// All other cases: Query and make bins in sqlite directly
	for _, m := range metrics {
		metricHisto, err := r.jobsMetricStatisticsHistogram(ctx, m, filter, targetBinCount)
		if err != nil {
			cclog.Warnf("Error while loading job metric statistics histogram: %s", m)
			continue
		}
		stat.HistMetrics = append(stat.HistMetrics, metricHisto)
	}

	cclog.Debugf("Timer AddMetricHistograms %s", time.Since(start))
	return stat, nil
}

// jobsStatisticsHistogram generates a simple histogram by grouping on a column value.
//
// Used for histograms where the column value directly represents the bin (e.g., node count, core count).
// Unlike duration/metric histograms, this doesn't pre-initialize bins with zeros.
//
// Parameters:
//   - value: SQL expression that produces the histogram value, aliased as "value"
//   - filters: Job filters to apply
//
// Returns histogram points with Value (from column) and Count (number of jobs).
func (r *JobRepository) jobsStatisticsHistogram(
	ctx context.Context,
	value string,
	filters []*model.JobFilter,
) ([]*model.HistoPoint, error) {
	start := time.Now()
	query, qerr := SecurityCheck(ctx,
		sq.Select(value, "COUNT(*) AS count").From("job"))

	if qerr != nil {
		return nil, qerr
	}

	for _, f := range filters {
		query = BuildWhereClause(f, query)
	}

	rows, err := query.GroupBy("value").RunWith(r.DB).QueryContext(ctx)
	if err != nil {
		cclog.Error("Error while running query")
		return nil, err
	}
	defer rows.Close()

	points := make([]*model.HistoPoint, 0)
	// is it possible to introduce zero values here? requires info about bincount
	for rows.Next() {
		point := model.HistoPoint{}
		if err := rows.Scan(&point.Value, &point.Count); err != nil {
			cclog.Warn("Error while scanning rows")
			return nil, err
		}

		points = append(points, &point)
	}

	if err := rows.Err(); err != nil {
		return nil, err
	}

	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
	return points, nil
}

// jobsDurationStatisticsHistogram generates a duration histogram with pre-initialized bins.
//
// Bins are zero-padded to provide consistent ranges for visualization, unlike simple
// histograms which only return bins with data. The value parameter should compute
// the bin number from job duration.
//
// Parameters:
//   - value: SQL expression computing bin number from duration, aliased as "value"
//   - filters: Job filters to apply
//   - binSizeSeconds: Width of each bin in seconds
//   - targetBinCount: Number of bins to pre-initialize
//
// Returns histogram points with Value (bin_number × binSizeSeconds) and Count.
// All bins from 1 to targetBinCount are returned, with Count=0 for empty bins.
//
// Algorithm:
//  1. Pre-initialize targetBinCount bins with zero counts
//  2. Query database for actual counts per bin
//  3. Match query results to pre-initialized bins by value
//  4. Bins without matches remain at zero
func (r *JobRepository) jobsDurationStatisticsHistogram(
	ctx context.Context,
	value string,
	filters []*model.JobFilter,
	binSizeSeconds int,
	targetBinCount *int,
) ([]*model.HistoPoint, error) {
	start := time.Now()
	query, qerr := SecurityCheck(ctx,
		sq.Select(value, "COUNT(*) AS count").From("job"))

	if qerr != nil {
		return nil, qerr
	}

	// Each bin represents a duration range: bin N = [N*binSizeSeconds, (N+1)*binSizeSeconds)
	// Example: binSizeSeconds=3600 (1 hour), bin 1 = 0-1h, bin 2 = 1-2h, etc.
	points := make([]*model.HistoPoint, 0, *targetBinCount)
	for i := 1; i <= *targetBinCount; i++ {
		point := model.HistoPoint{Value: i * binSizeSeconds, Count: 0}
		points = append(points, &point)
	}

	// Build a map from bin value (seconds) to slice index for O(1) lookup.
	binMap := make(map[int]int, len(points))
	for i, p := range points {
		binMap[p.Value] = i
	}

	for _, f := range filters {
		query = BuildWhereClause(f, query)
	}

	rows, err := query.GroupBy("value").RunWith(r.DB).QueryContext(ctx)
	if err != nil {
		cclog.Error("Error while running query")
		return nil, err
	}
	defer rows.Close()

	// Match query results to pre-initialized bins using map lookup.
	// point.Value from query is the bin number; multiply by binSizeSeconds to get bin key.
	for rows.Next() {
		point := model.HistoPoint{}
		if err := rows.Scan(&point.Value, &point.Count); err != nil {
			cclog.Warn("Error while scanning rows")
			return nil, err
		}

		if idx, ok := binMap[point.Value*binSizeSeconds]; ok {
			points[idx].Count = point.Count
		}
	}

	if err := rows.Err(); err != nil {
		return nil, err
	}

	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
	return points, nil
}

// jobsMetricStatisticsHistogram generates a metric histogram using footprint data from completed jobs.
//
// Values are normalized to the metric's peak value and distributed into bins. The algorithm
// is based on SQL histogram generation techniques, extracting metric values from JSON footprint
// and computing bin assignments in SQL.
//
// Parameters:
//   - metric: Metric name (e.g., "cpu_load", "mem_used")
//   - filters: Job filters to apply
//   - bins: Number of bins to generate
//
// Returns MetricHistoPoints with metric name, unit, footprint stat type, and binned data.
//
// Algorithm:
//  1. Determine peak value from cluster configuration (filtered cluster or max across all)
//  2. Generate SQL that extracts footprint value, normalizes to [0,1], multiplies by bin count
//  3. Pre-initialize bins with min/max ranges based on peak value
//  4. Query database for counts per bin
//  5. Match results to pre-initialized bins
//
// Special handling: Values exactly equal to peak are forced into the last bin by multiplying
// peak by 0.999999999 to avoid creating an extra bin.
func (r *JobRepository) jobsMetricStatisticsHistogram(
	ctx context.Context,
	metric string,
	filters []*model.JobFilter,
	bins *int,
) (*model.MetricHistoPoints, error) {
	// Peak value defines the upper bound for binning: values are distributed across
	// bins from 0 to peak. First try to get peak from filtered cluster, otherwise
	// scan all clusters to find the maximum peak value.
	var metricConfig *schema.MetricConfig
	var peak float64
	var unit string
	var footprintStat string

	// Try to get metric config from filtered cluster
	for _, f := range filters {
		if f.Cluster != nil {
			metricConfig = archive.GetMetricConfig(*f.Cluster.Eq, metric)
			peak = metricConfig.Peak
			unit = metricConfig.Unit.Prefix + metricConfig.Unit.Base
			footprintStat = metricConfig.Footprint
			cclog.Debugf("Cluster %s filter found with peak %f for %s", *f.Cluster.Eq, peak, metric)
		}
	}

	// If no cluster filter or peak not found, find largest peak across all clusters
	// This ensures histogram can accommodate all possible values
	if peak == 0.0 {
		for _, c := range archive.Clusters {
			for _, m := range c.MetricConfig {
				if m.Name == metric {
					if m.Peak > peak {
						peak = m.Peak
					}
					if unit == "" {
						unit = m.Unit.Prefix + m.Unit.Base
					}
					if footprintStat == "" {
						footprintStat = m.Footprint
					}
				}
			}
		}
	}

	// Construct SQL histogram bins using normalized values.
	// Algorithm based on: https://jereze.com/code/sql-histogram/ (modified)
	start := time.Now()

	// Bin calculation formula:
	//   bin_number = CAST( (value / peak) * num_bins AS INTEGER ) + 1
	// Special case: value == peak would create bin N+1, so we test for equality
	// and multiply peak by 0.999999999 to force it into bin N.
	binQuery := fmt.Sprintf(`CAST(
		((case when json_extract(footprint, "$.%s") = %f then %f*0.999999999 else json_extract(footprint, "$.%s") end) / %f)
		* %v as INTEGER )`,
		(metric + "_" + footprintStat), peak, peak, (metric + "_" + footprintStat), peak, *bins)

	mainQuery := sq.Select(
		fmt.Sprintf(`%s + 1 as bin`, binQuery),
		`count(*) as count`,
	).From("job").Where(
		"JSON_VALID(footprint)",
	).Where(fmt.Sprintf(`json_extract(footprint, "$.%s") is not null and json_extract(footprint, "$.%s") <= %f`, (metric + "_" + footprintStat), (metric + "_" + footprintStat), peak))

	mainQuery, qerr := SecurityCheck(ctx, mainQuery)
	if qerr != nil {
		return nil, qerr
	}

	for _, f := range filters {
		mainQuery = BuildWhereClause(f, mainQuery)
	}

	mainQuery = mainQuery.GroupBy("bin").OrderBy("bin")

	rows, err := mainQuery.RunWith(r.DB).QueryContext(ctx)
	if err != nil {
		cclog.Errorf("Error while running mainQuery: %s", err)
		return nil, err
	}
	defer rows.Close()

	// Pre-initialize bins with calculated min/max ranges.
	// Example: peak=1000, bins=10 -> bin 1=[0,100), bin 2=[100,200), ..., bin 10=[900,1000]
	points := make([]*model.MetricHistoPoint, 0, *bins)
	binStep := int(peak) / *bins
	for i := 1; i <= *bins; i++ {
		binMin := (binStep * (i - 1))
		binMax := (binStep * i)
		idx := i
		epoint := model.MetricHistoPoint{Bin: &idx, Count: 0, Min: &binMin, Max: &binMax}
		points = append(points, &epoint)
	}

	// Build a map from bin number to slice index for O(1) lookup.
	binMap := make(map[int]int, len(points))
	for i, p := range points {
		if p.Bin != nil {
			binMap[*p.Bin] = i
		}
	}

	// Match query results to pre-initialized bins using map lookup.
	for rows.Next() {
		rpoint := model.MetricHistoPoint{}
		if err := rows.Scan(&rpoint.Bin, &rpoint.Count); err != nil {
			cclog.Warnf("Error while scanning rows for %s", metric)
			return nil, err
		}

		if rpoint.Bin != nil {
			if idx, ok := binMap[*rpoint.Bin]; ok {
				points[idx].Count = rpoint.Count
			}
		}
	}

	if err := rows.Err(); err != nil {
		return nil, err
	}

	result := model.MetricHistoPoints{Metric: metric, Unit: unit, Stat: &footprintStat, Data: points}

	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
	return &result, nil
}

// runningJobsMetricStatisticsHistogram generates metric histograms for running jobs using live data.
//
// Unlike completed jobs which use footprint data from the database, running jobs require
// fetching current metric averages from the metric backend (via metricdispatch).
//
// Parameters:
//   - metrics: List of metric names
//   - filters: Job filters (should filter to running jobs only)
//   - bins: Number of histogram bins
//
// Returns slice of MetricHistoPoints, one per metric.
//
// Limitations:
//   - Maximum 5000 jobs (returns nil if more jobs match)
//   - Requires metric backend availability
//   - Bins based on metric peak values from cluster configuration
//
// Algorithm:
//  1. Query first 5001 jobs to check count limit
//  2. Load metric averages for all jobs via metricdispatch
//  3. For each metric, create bins based on peak value
//  4. Iterate averages and count jobs per bin
func (r *JobRepository) runningJobsMetricStatisticsHistogram(
	ctx context.Context,
	metrics []string,
	filters []*model.JobFilter,
	bins *int,
) []*model.MetricHistoPoints {
	// Get Jobs
	jobs, err := r.QueryJobs(ctx, filters, &model.PageRequest{Page: 1, ItemsPerPage: 5000 + 1}, nil)
	if err != nil {
		cclog.Errorf("Error while querying jobs for footprint: %s", err)
		return nil
	}
	if len(jobs) > 5000 {
		cclog.Errorf("too many jobs matched (max: %d)", 5000)
		return nil
	}

	// Get AVGs from metric repo
	avgs := make([][]schema.Float, len(metrics))
	for i := range avgs {
		avgs[i] = make([]schema.Float, 0, len(jobs))
	}

	for _, job := range jobs {
		if job.MonitoringStatus == schema.MonitoringStatusDisabled || job.MonitoringStatus == schema.MonitoringStatusArchivingFailed {
			continue
		}

		if err := metricdispatch.LoadAverages(job, metrics, avgs, ctx); err != nil {
			cclog.Errorf("Error while loading averages for histogram: %s", err)
			return nil
		}
	}

	// Iterate metrics to fill endresult
	data := make([]*model.MetricHistoPoints, 0)
	for idx, metric := range metrics {
		// Get specific Peak or largest Peak
		var metricConfig *schema.MetricConfig
		var peak float64
		var unit string

		for _, f := range filters {
			if f.Cluster != nil {
				metricConfig = archive.GetMetricConfig(*f.Cluster.Eq, metric)
				peak = metricConfig.Peak
				unit = metricConfig.Unit.Prefix + metricConfig.Unit.Base
			}
		}

		if peak == 0.0 {
			for _, c := range archive.Clusters {
				for _, m := range c.MetricConfig {
					if m.Name == metric {
						if m.Peak > peak {
							peak = m.Peak
						}
						if unit == "" {
							unit = m.Unit.Prefix + m.Unit.Base
						}
					}
				}
			}
		}

		// Make and fill bins
		peakBin := int(peak) / *bins

		points := make([]*model.MetricHistoPoint, 0)
		for b := 0; b < *bins; b++ {
			count := 0
			bindex := b + 1
			bmin := peakBin * b
			bmax := peakBin * (b + 1)

			// Iterate AVG values for indexed metric and count for bins
			for _, val := range avgs[idx] {
				if int(val) >= bmin && int(val) < bmax {
					count += 1
				}
			}

			// Append Bin to Metric Result Array
			point := model.MetricHistoPoint{Bin: &bindex, Count: count, Min: &bmin, Max: &bmax}
			points = append(points, &point)
		}

		// Append Metric Result Array to final results array
		result := model.MetricHistoPoints{Metric: metric, Unit: unit, Data: points}
		data = append(data, &result)
	}

	return data
}

-    query = query.Where("job.job_state = ?", "running")

+    query = query.Where("job.job_state = 'running'")


The file /Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/job.go has been updated successfully.

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package repository provides the data access layer for cc-backend using the repository pattern.
//
// The repository pattern abstracts database operations and provides a clean interface for
// data access. Each major entity (Job, User, Node, Tag) has its own repository with CRUD
// operations and specialized queries.
//
// # Database Connection
//
// Initialize the database connection before using any repository:
//
//	repository.Connect("sqlite3", "./var/job.db")
//
// # Configuration
//
// Optional: Configure repository settings before initialization:
//
//	repository.SetConfig(&repository.RepositoryConfig{
//	    CacheSize: 2 * 1024 * 1024,     // 2MB cache
//	    MaxOpenConnections: 8,           // Connection pool size
//	    MinRunningJobDuration: 300,      // Filter threshold
//	})
//
// If not configured, sensible defaults are used automatically.
//
// # Repositories
//
//   - JobRepository: Job lifecycle management and querying
//   - UserRepository: User management and authentication
//   - NodeRepository: Cluster node state tracking
//   - Tags: Job tagging and categorization
//
// # Caching
//
// Repositories use LRU caching to improve performance. Cache keys are constructed
// as "type:id" (e.g., "metadata:123"). Cache is automatically invalidated on
// mutations to maintain consistency.
//
// # Transaction Support
//
// For batch operations, use transactions:
//
//	t, err := jobRepo.TransactionInit()
//	if err != nil {
//	    return err
//	}
//	defer t.Rollback() // Rollback if not committed
//
//	// Perform operations...
//	jobRepo.TransactionAdd(t, query, args...)
//
//	// Commit when done
//	if err := t.Commit(); err != nil {
//	    return err
//	}
package repository

import (
	"database/sql"
	"encoding/json"
	"errors"
	"fmt"
	"maps"
	"math"
	"sort"
	"strconv"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/lrucache"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	sq "github.com/Masterminds/squirrel"
	"github.com/jmoiron/sqlx"
)

var (
	// jobRepoOnce ensures singleton initialization of the JobRepository
	jobRepoOnce sync.Once
	// jobRepoInstance holds the single instance of JobRepository
	jobRepoInstance *JobRepository
)

// JobRepository provides database access for job-related operations.
// It implements the repository pattern to abstract database interactions
// and provides caching for improved performance.
//
// The repository is a singleton initialized via GetJobRepository().
// All database queries use prepared statements via stmtCache for efficiency.
// Frequently accessed data (metadata, energy footprints) is cached in an LRU cache.
type JobRepository struct {
	DB        *sqlx.DB        // Database connection pool
	stmtCache *sq.StmtCache   // Prepared statement cache for query optimization
	cache     *lrucache.Cache // LRU cache for metadata and footprint data
	driver    string          // Database driver name (e.g., "sqlite3")
	Mutex     sync.Mutex      // Mutex for thread-safe operations
}

// GetJobRepository returns the singleton instance of JobRepository.
// The repository is initialized lazily on first access with database connection,
// prepared statement cache, and LRU cache configured from repoConfig.
//
// This function is thread-safe and ensures only one instance is created.
// It must be called after Connect() has established a database connection.
func GetJobRepository() *JobRepository {
	jobRepoOnce.Do(func() {
		db := GetConnection()

		jobRepoInstance = &JobRepository{
			DB:     db.DB,
			driver: db.Driver,

			stmtCache: sq.NewStmtCache(db.DB),
			cache:     lrucache.New(repoConfig.CacheSize),
		}
	})
	return jobRepoInstance
}

// jobColumns defines the standard set of columns selected from the job table.
// Used consistently across all job queries to ensure uniform data retrieval.
var jobColumns []string = []string{
	"job.id", "job.job_id", "job.hpc_user", "job.project", "job.cluster", "job.subcluster",
	"job.start_time", "job.cluster_partition", "job.array_job_id", "job.num_nodes",
	"job.num_hwthreads", "job.num_acc", "job.shared", "job.monitoring_status",
	"job.smt", "job.job_state", "job.duration", "job.walltime", "job.resources",
	"job.footprint", "job.energy",
}

// jobCacheColumns defines columns from the job_cache table, mirroring jobColumns.
// Used for queries against cached job data for performance optimization.
var jobCacheColumns []string = []string{
	"job_cache.id", "job_cache.job_id", "job_cache.hpc_user", "job_cache.project", "job_cache.cluster",
	"job_cache.subcluster", "job_cache.start_time", "job_cache.cluster_partition",
	"job_cache.array_job_id", "job_cache.num_nodes", "job_cache.num_hwthreads",
	"job_cache.num_acc", "job_cache.shared", "job_cache.monitoring_status", "job_cache.smt",
	"job_cache.job_state", "job_cache.duration", "job_cache.walltime", "job_cache.resources",
	"job_cache.footprint", "job_cache.energy",
}

// scanJob converts a database row into a schema.Job struct.
// It handles JSON unmarshaling of resources and footprint fields,
// and calculates accurate duration for running jobs.
//
// Parameters:
//   - row: Database row implementing Scan() interface (sql.Row or sql.Rows)
//
// Returns the populated Job struct or an error if scanning or unmarshaling fails.
func scanJob(row interface{ Scan(...any) error }) (*schema.Job, error) {
	job := &schema.Job{}

	if err := row.Scan(
		&job.ID, &job.JobID, &job.User, &job.Project, &job.Cluster, &job.SubCluster,
		&job.StartTime, &job.Partition, &job.ArrayJobID, &job.NumNodes, &job.NumHWThreads,
		&job.NumAcc, &job.Shared, &job.MonitoringStatus, &job.SMT, &job.State,
		&job.Duration, &job.Walltime, &job.RawResources, &job.RawFootprint, &job.Energy); err != nil {
		cclog.Warnf("Error while scanning rows (Job): %v", err)
		return nil, err
	}

	if err := json.Unmarshal(job.RawResources, &job.Resources); err != nil {
		cclog.Warn("Error while unmarshaling raw resources json")
		return nil, err
	}
	job.RawResources = nil

	if err := json.Unmarshal(job.RawFootprint, &job.Footprint); err != nil {
		cclog.Warnf("Error while unmarshaling raw footprint json: %v", err)
		return nil, err
	}
	job.RawFootprint = nil

	// Always ensure accurate duration for running jobs
	if job.State == schema.JobStateRunning {
		job.Duration = int32(time.Now().Unix() - job.StartTime)
	}

	return job, nil
}

// Optimize performs database optimization by running VACUUM command.
// This reclaims unused space and defragments the database file.
// Should be run periodically during maintenance windows.
func (r *JobRepository) Optimize() error {
	if _, err := r.DB.Exec(`VACUUM`); err != nil {
		cclog.Errorf("Error while executing VACUUM: %v", err)
		return fmt.Errorf("failed to optimize database: %w", err)
	}
	return nil
}

// Flush removes all data from job-related tables (jobtag, tag, job).
// WARNING: This is a destructive operation that deletes all job data.
// Use with extreme caution, typically only for testing or complete resets.
func (r *JobRepository) Flush() error {
	if _, err := r.DB.Exec(`DELETE FROM jobtag`); err != nil {
		cclog.Errorf("Error while deleting from jobtag table: %v", err)
		return fmt.Errorf("failed to flush jobtag table: %w", err)
	}
	if _, err := r.DB.Exec(`DELETE FROM tag`); err != nil {
		cclog.Errorf("Error while deleting from tag table: %v", err)
		return fmt.Errorf("failed to flush tag table: %w", err)
	}
	if _, err := r.DB.Exec(`DELETE FROM job`); err != nil {
		cclog.Errorf("Error while deleting from job table: %v", err)
		return fmt.Errorf("failed to flush job table: %w", err)
	}
	return nil
}

// FetchMetadata retrieves and unmarshals the metadata JSON for a job.
// Metadata is cached with a 24-hour TTL to improve performance.
//
// The metadata field stores arbitrary key-value pairs associated with a job,
// such as tags, labels, or custom attributes added by external systems.
//
// Parameters:
//   - job: Job struct with valid ID field, metadata will be populated in job.MetaData
//
// Returns the metadata map or an error if the job is nil or database query fails.
func (r *JobRepository) FetchMetadata(job *schema.Job) (map[string]string, error) {
	if job == nil {
		return nil, fmt.Errorf("job cannot be nil")
	}

	start := time.Now()
	cachekey := fmt.Sprintf("metadata:%d", *job.ID)
	if cached := r.cache.Get(cachekey, nil); cached != nil {
		job.MetaData = cached.(map[string]string)
		return job.MetaData, nil
	}

	if err := sq.Select("job.meta_data").From("job").Where("job.id = ?", job.ID).
		RunWith(r.stmtCache).QueryRow().Scan(&job.RawMetaData); err != nil {
		cclog.Warnf("Error while scanning for job metadata (ID=%d): %v", *job.ID, err)
		return nil, fmt.Errorf("failed to fetch metadata for job %d: %w", *job.ID, err)
	}

	if len(job.RawMetaData) == 0 {
		return nil, nil
	}

	if err := json.Unmarshal(job.RawMetaData, &job.MetaData); err != nil {
		cclog.Warnf("Error while unmarshaling raw metadata json (ID=%d): %v", *job.ID, err)
		return nil, fmt.Errorf("failed to unmarshal metadata for job %d: %w", *job.ID, err)
	}

	r.cache.Put(cachekey, job.MetaData, len(job.RawMetaData), 24*time.Hour)
	cclog.Debugf("Timer FetchMetadata %s", time.Since(start))
	return job.MetaData, nil
}

// UpdateMetadata adds or updates a single metadata key-value pair for a job.
// The entire metadata map is re-marshaled and stored, and the cache is invalidated.
// Also triggers archive metadata update via archive.UpdateMetadata.
//
// Parameters:
//   - job: Job struct with valid ID, existing metadata will be fetched if not present
//   - key: Metadata key to set
//   - val: Metadata value to set
//
// Returns an error if the job is nil, metadata fetch fails, or database update fails.
func (r *JobRepository) UpdateMetadata(job *schema.Job, key, val string) (err error) {
	if job == nil {
		return fmt.Errorf("job cannot be nil")
	}

	cachekey := fmt.Sprintf("metadata:%d", *job.ID)
	r.cache.Del(cachekey)
	if job.MetaData == nil {
		if _, err = r.FetchMetadata(job); err != nil {
			cclog.Warnf("Error while fetching metadata for job, DB ID '%v'", *job.ID)
			return fmt.Errorf("failed to fetch metadata for job %d: %w", *job.ID, err)
		}
	}

	if job.MetaData != nil {
		cpy := make(map[string]string, len(job.MetaData)+1)
		maps.Copy(cpy, job.MetaData)
		cpy[key] = val
		job.MetaData = cpy
	} else {
		job.MetaData = map[string]string{key: val}
	}

	if job.RawMetaData, err = json.Marshal(job.MetaData); err != nil {
		cclog.Warnf("Error while marshaling metadata for job, DB ID '%v'", *job.ID)
		return fmt.Errorf("failed to marshal metadata for job %d: %w", *job.ID, err)
	}

	if _, err = sq.Update("job").
		Set("meta_data", job.RawMetaData).
		Where("job.id = ?", job.ID).
		RunWith(r.stmtCache).Exec(); err != nil {
		cclog.Warnf("Error while updating metadata for job, DB ID '%v'", *job.ID)
		return fmt.Errorf("failed to update metadata in database for job %d: %w", *job.ID, err)
	}

	r.cache.Put(cachekey, job.MetaData, len(job.RawMetaData), 24*time.Hour)
	return archive.UpdateMetadata(job, job.MetaData)
}

// FetchFootprint retrieves and unmarshals the performance footprint JSON for a job.
// Unlike FetchMetadata, footprints are NOT cached as they can be large and change frequently.
//
// The footprint contains summary statistics (avg/min/max) for monitored metrics,
// stored as JSON with keys like "cpu_load_avg", "mem_used_max", etc.
//
// Parameters:
//   - job: Job struct with valid ID, footprint will be populated in job.Footprint
//
// Returns the footprint map or an error if the job is nil or database query fails.
func (r *JobRepository) FetchFootprint(job *schema.Job) (map[string]float64, error) {
	if job == nil {
		return nil, fmt.Errorf("job cannot be nil")
	}

	start := time.Now()

	if err := sq.Select("job.footprint").From("job").Where("job.id = ?", job.ID).
		RunWith(r.stmtCache).QueryRow().Scan(&job.RawFootprint); err != nil {
		cclog.Warnf("Error while scanning for job footprint (ID=%d): %v", *job.ID, err)
		return nil, fmt.Errorf("failed to fetch footprint for job %d: %w", *job.ID, err)
	}

	if len(job.RawFootprint) == 0 {
		return nil, nil
	}

	if err := json.Unmarshal(job.RawFootprint, &job.Footprint); err != nil {
		cclog.Warnf("Error while unmarshaling raw footprint json (ID=%d): %v", *job.ID, err)
		return nil, fmt.Errorf("failed to unmarshal footprint for job %d: %w", *job.ID, err)
	}

	cclog.Debugf("Timer FetchFootprint %s", time.Since(start))
	return job.Footprint, nil
}

// FetchEnergyFootprint retrieves and unmarshals the energy footprint JSON for a job.
// Energy footprints are cached with a 24-hour TTL as they are frequently accessed but rarely change.
//
// The energy footprint contains calculated energy consumption (in kWh) per metric,
// stored as JSON with keys like "power_avg", "acc_power_avg", etc.
//
// Parameters:
//   - job: Job struct with valid ID, energy footprint will be populated in job.EnergyFootprint
//
// Returns the energy footprint map or an error if the job is nil or database query fails.
func (r *JobRepository) FetchEnergyFootprint(job *schema.Job) (map[string]float64, error) {
	if job == nil {
		return nil, fmt.Errorf("job cannot be nil")
	}

	start := time.Now()
	cachekey := fmt.Sprintf("energyFootprint:%d", *job.ID)
	if cached := r.cache.Get(cachekey, nil); cached != nil {
		job.EnergyFootprint = cached.(map[string]float64)
		return job.EnergyFootprint, nil
	}

	if err := sq.Select("job.energy_footprint").From("job").Where("job.id = ?", job.ID).
		RunWith(r.stmtCache).QueryRow().Scan(&job.RawEnergyFootprint); err != nil {
		cclog.Warnf("Error while scanning for job energy_footprint (ID=%d): %v", *job.ID, err)
		return nil, fmt.Errorf("failed to fetch energy footprint for job %d: %w", *job.ID, err)
	}

	if len(job.RawEnergyFootprint) == 0 {
		return nil, nil
	}

	if err := json.Unmarshal(job.RawEnergyFootprint, &job.EnergyFootprint); err != nil {
		cclog.Warnf("Error while unmarshaling raw energy footprint json (ID=%d): %v", *job.ID, err)
		return nil, fmt.Errorf("failed to unmarshal energy footprint for job %d: %w", *job.ID, err)
	}

	r.cache.Put(cachekey, job.EnergyFootprint, len(job.EnergyFootprint), 24*time.Hour)
	cclog.Debugf("Timer FetchEnergyFootprint %s", time.Since(start))
	return job.EnergyFootprint, nil
}

// DeleteJobsBefore removes jobs older than the specified start time.
// Optionally preserves tagged jobs to protect important data from deletion.
// Cache entries for deleted jobs are automatically invalidated.
//
// This is typically used for data retention policies and cleanup operations.
// WARNING: This is a destructive operation that permanently deletes job records.
//
// Parameters:
//   - startTime: Unix timestamp, jobs with start_time < this value will be deleted
//   - omitTagged: "none" = delete all jobs, "all" = skip any tagged jobs,
//     "user" = skip jobs with user-created tags (not auto-tagger types "app"/"jobClass")
//
// Returns the count of deleted jobs or an error if the operation fails.
func (r *JobRepository) DeleteJobsBefore(startTime int64, omitTagged string) (int, error) {
	var cnt int
	q := sq.Select("count(*)").From("job").Where("job.start_time < ?", startTime)

	switch omitTagged {
	case "all":
		q = q.Where("NOT EXISTS (SELECT 1 FROM jobtag WHERE jobtag.job_id = job.id)")
	case "user":
		q = q.Where("NOT EXISTS (SELECT 1 FROM jobtag JOIN tag ON tag.id = jobtag.tag_id WHERE jobtag.job_id = job.id AND tag.tag_type NOT IN ('app', 'jobClass'))")
	}

	if err := q.RunWith(r.DB).QueryRow().Scan(&cnt); err != nil {
		cclog.Errorf("Error counting jobs before %d: %v", startTime, err)
		return 0, err
	}

	// Invalidate cache for jobs being deleted (get job IDs first)
	if cnt > 0 {
		var jobIds []int64
		selectQuery := sq.Select("id").From("job").Where("job.start_time < ?", startTime)

		switch omitTagged {
		case "all":
			selectQuery = selectQuery.Where("NOT EXISTS (SELECT 1 FROM jobtag WHERE jobtag.job_id = job.id)")
		case "user":
			selectQuery = selectQuery.Where("NOT EXISTS (SELECT 1 FROM jobtag JOIN tag ON tag.id = jobtag.tag_id WHERE jobtag.job_id = job.id AND tag.tag_type NOT IN ('app', 'jobClass'))")
		}

		rows, err := selectQuery.RunWith(r.DB).Query()
		if err == nil {
			defer rows.Close()
			for rows.Next() {
				var id int64
				if err := rows.Scan(&id); err == nil {
					jobIds = append(jobIds, id)
				}
			}
			// Invalidate cache entries
			for _, id := range jobIds {
				r.cache.Del(fmt.Sprintf("metadata:%d", id))
				r.cache.Del(fmt.Sprintf("energyFootprint:%d", id))
			}
		}
	}

	qd := sq.Delete("job").Where("job.start_time < ?", startTime)

	switch omitTagged {
	case "all":
		qd = qd.Where("NOT EXISTS (SELECT 1 FROM jobtag WHERE jobtag.job_id = job.id)")
	case "user":
		qd = qd.Where("NOT EXISTS (SELECT 1 FROM jobtag JOIN tag ON tag.id = jobtag.tag_id WHERE jobtag.job_id = job.id AND tag.tag_type NOT IN ('app', 'jobClass'))")
	}
	_, err := qd.RunWith(r.DB).Exec()

	if err != nil {
		s, _, _ := qd.ToSql()
		cclog.Errorf(" DeleteJobsBefore(%d) with %s: error %#v", startTime, s, err)
	} else {
		cclog.Debugf("DeleteJobsBefore(%d): Deleted %d jobs", startTime, cnt)
	}
	return cnt, err
}

// DeleteJobByID permanently removes a single job by its database ID.
// Cache entries for the deleted job are automatically invalidated.
//
// Parameters:
//   - id: Database ID (primary key) of the job to delete
//
// Returns an error if the deletion fails.
func (r *JobRepository) DeleteJobByID(id int64) error {
	// Invalidate cache entries before deletion
	r.cache.Del(fmt.Sprintf("metadata:%d", id))
	r.cache.Del(fmt.Sprintf("energyFootprint:%d", id))

	qd := sq.Delete("job").Where("job.id = ?", id)
	_, err := qd.RunWith(r.DB).Exec()

	if err != nil {
		s, _, _ := qd.ToSql()
		cclog.Errorf("DeleteJobById(%d) with %s : error %#v", id, s, err)
	} else {
		cclog.Debugf("DeleteJobById(%d): Success", id)
	}
	return err
}

// FindUserOrProjectOrJobname attempts to interpret a search term as a job ID,
// username, project ID, or job name by querying the database.
//
// Search logic (in priority order):
//  1. If searchterm is numeric, treat as job ID (returned immediately)
//  2. Try exact match in job.hpc_user column (username)
//  3. Try LIKE match in hpc_user.name column (real name)
//  4. Try exact match in job.project column (project ID)
//  5. If no matches, return searchterm as jobname for GraphQL query
//
// This powers the searchbar functionality for flexible job searching.
// Requires authenticated user for database lookups (returns empty if user is nil).
//
// Parameters:
//   - user: Authenticated user context, required for database access
//   - searchterm: Search string to interpret
//
// Returns up to one non-empty value among (jobid, username, project, jobname).
func (r *JobRepository) FindUserOrProjectOrJobname(user *schema.User, searchterm string) (jobid string, username string, project string, jobname string) {
	if searchterm == "" {
		return "", "", "", ""
	}

	if _, err := strconv.Atoi(searchterm); err == nil { // Return empty on successful conversion: parent method will redirect for integer jobId
		return searchterm, "", "", ""
	} else { // Has to have letters and logged-in user for other guesses
		if user != nil {
			// Find username by username in job table (match)
			uresult, _ := r.FindColumnValue(user, searchterm, "job", "hpc_user", "hpc_user", false)
			if uresult != "" {
				return "", uresult, "", ""
			}
			// Find username by real name in hpc_user table (like)
			nresult, _ := r.FindColumnValue(user, searchterm, "hpc_user", "username", "name", true)
			if nresult != "" {
				return "", nresult, "", ""
			}
			// Find projectId by projectId in job table (match)
			presult, _ := r.FindColumnValue(user, searchterm, "job", "project", "project", false)
			if presult != "" {
				return "", "", presult, ""
			}
		}
		// Return searchterm if no match before: Forward as jobname query to GQL in handleSearchbar function
		return "", "", "", searchterm
	}
}

var (
	ErrNotFound  = errors.New("no such jobname, project or user")
	ErrForbidden = errors.New("not authorized")
)

// FindColumnValue performs a generic column lookup in a database table with role-based access control.
// Only users with admin, support, or manager roles can execute this query.
//
// Parameters:
//   - user: User context for authorization check
//   - searchterm: Value to search for (exact match or LIKE pattern)
//   - table: Database table name to query
//   - selectColumn: Column name to return in results
//   - whereColumn: Column name to filter on
//   - isLike: If true, use LIKE with wildcards; if false, use exact equality
//
// Returns the first matching value, ErrForbidden if user lacks permission,
// or ErrNotFound if no matches are found.
func (r *JobRepository) FindColumnValue(user *schema.User, searchterm string, table string, selectColumn string, whereColumn string, isLike bool) (result string, err error) {
	if user == nil {
		return "", fmt.Errorf("user cannot be nil")
	}

	compareStr := " = ?"
	query := searchterm
	if isLike {
		compareStr = " LIKE ?"
		query = "%" + searchterm + "%"
	}
	if user.HasAnyRole([]schema.Role{schema.RoleAdmin, schema.RoleSupport, schema.RoleManager}) {
		theQuery := sq.Select(table+"."+selectColumn).Distinct().From(table).
			Where(table+"."+whereColumn+compareStr, query)

		err := theQuery.RunWith(r.stmtCache).QueryRow().Scan(&result)

		if err != nil && err != sql.ErrNoRows {
			cclog.Warnf("Error while querying FindColumnValue (table=%s, column=%s): %v", table, selectColumn, err)
			return "", fmt.Errorf("failed to find column value: %w", err)
		} else if err == nil {
			return result, nil
		}
		return "", ErrNotFound
	} else {
		cclog.Infof("Non-Admin User %s : Requested Query '%s' on table '%s' : Forbidden", user.Name, query, table)
		return "", ErrForbidden
	}
}

// FindColumnValues performs a generic column lookup returning multiple matches with role-based access control.
// Similar to FindColumnValue but returns all matching values instead of just the first.
// Only users with admin, support, or manager roles can execute this query.
//
// Parameters:
//   - user: User context for authorization check
//   - query: Search pattern (always uses LIKE with wildcards)
//   - table: Database table name to query
//   - selectColumn: Column name to return in results
//   - whereColumn: Column name to filter on
//
// Returns a slice of matching values, ErrForbidden if user lacks permission,
// or ErrNotFound if no matches are found.
func (r *JobRepository) FindColumnValues(user *schema.User, query string, table string, selectColumn string, whereColumn string) (results []string, err error) {
	if user == nil {
		return nil, fmt.Errorf("user cannot be nil")
	}

	emptyResult := make([]string, 0)
	if user.HasAnyRole([]schema.Role{schema.RoleAdmin, schema.RoleSupport, schema.RoleManager}) {
		rows, err := sq.Select(table+"."+selectColumn).Distinct().From(table).
			Where(table+"."+whereColumn+" LIKE ?", fmt.Sprint("%", query, "%")).
			RunWith(r.stmtCache).Query()
		if err != nil && err != sql.ErrNoRows {
			cclog.Errorf("Error while querying FindColumnValues (table=%s, column=%s): %v", table, selectColumn, err)
			return emptyResult, fmt.Errorf("failed to find column values: %w", err)
		} else if err == nil {
			defer rows.Close()
			for rows.Next() {
				var result string
				err := rows.Scan(&result)
				if err != nil {
					cclog.Warnf("Error while scanning rows in FindColumnValues: %v", err)
					return emptyResult, fmt.Errorf("failed to scan column value: %w", err)
				}
				results = append(results, result)
			}
			return results, nil
		}
		return emptyResult, ErrNotFound

	} else {
		cclog.Infof("Non-Admin User %s : Requested Query '%s' on table '%s' : Forbidden", user.Name, query, table)
		return emptyResult, ErrForbidden
	}
}

// Partitions returns a list of distinct cluster partitions for a given cluster.
// Results are cached with a 1-hour TTL to improve performance.
//
// Parameters:
//   - cluster: Cluster name to query partitions for
//
// Returns a slice of partition names or an error if the database query fails.
func (r *JobRepository) Partitions(cluster string) ([]string, error) {
	var err error
	start := time.Now()
	partitions := r.cache.Get("partitions:"+cluster, func() (any, time.Duration, int) {
		parts := []string{}
		if err = r.DB.Select(&parts, `SELECT DISTINCT job.cluster_partition FROM job WHERE job.cluster = ?;`, cluster); err != nil {
			return nil, 0, 1000
		}

		return parts, 1 * time.Hour, 1
	})
	if err != nil {
		return nil, err
	}
	cclog.Debugf("Timer Partitions %s", time.Since(start))
	return partitions.([]string), nil
}

// AllocatedNodes returns a map of all subclusters to a map of hostnames to the amount of jobs running on that host.
// Hosts with zero jobs running on them will not show up!
func (r *JobRepository) AllocatedNodes(cluster string) (map[string]map[string]int, error) {
	start := time.Now()
	subclusters := make(map[string]map[string]int)
	rows, err := sq.Select("resources", "subcluster").From("job").
		Where("job.job_state = 'running'").
		Where("job.cluster = ?", cluster).
		RunWith(r.stmtCache).Query()
	if err != nil {
		cclog.Errorf("Error while running AllocatedNodes query for cluster=%s: %v", cluster, err)
		return nil, fmt.Errorf("failed to query allocated nodes for cluster %s: %w", cluster, err)
	}

	var raw []byte
	defer rows.Close()
	for rows.Next() {
		raw = raw[0:0]
		var resources []*schema.Resource
		var subcluster string
		if err := rows.Scan(&raw, &subcluster); err != nil {
			cclog.Warnf("Error while scanning rows in AllocatedNodes: %v", err)
			return nil, fmt.Errorf("failed to scan allocated nodes row: %w", err)
		}
		if err := json.Unmarshal(raw, &resources); err != nil {
			cclog.Warnf("Error while unmarshaling raw resources json in AllocatedNodes: %v", err)
			return nil, fmt.Errorf("failed to unmarshal resources in AllocatedNodes: %w", err)
		}

		hosts, ok := subclusters[subcluster]
		if !ok {
			hosts = make(map[string]int)
			subclusters[subcluster] = hosts
		}

		for _, resource := range resources {
			hosts[resource.Hostname] += 1
		}
	}

	cclog.Debugf("Timer AllocatedNodes %s", time.Since(start))
	return subclusters, nil
}

// StopJobsExceedingWalltimeBy marks running jobs as failed if they exceed their walltime limit.
// This is typically called periodically to clean up stuck or orphaned jobs.
//
// Jobs are marked with:
//   - monitoring_status: MonitoringStatusArchivingFailed
//   - duration: 0
//   - job_state: JobStateFailed
//
// Parameters:
//   - seconds: Grace period beyond walltime before marking as failed
//
// Returns an error if the database update fails.
// Logs the number of jobs marked as failed if any were affected.
func (r *JobRepository) StopJobsExceedingWalltimeBy(seconds int) error {
	start := time.Now()
	currentTime := time.Now().Unix()
	res, err := sq.Update("job").
		Set("monitoring_status", schema.MonitoringStatusArchivingFailed).
		Set("duration", 0).
		Set("job_state", schema.JobStateFailed).
		Where("job.job_state = 'running'").
		Where("job.walltime > 0").
		Where("job.start_time < (? - job.walltime)", currentTime-int64(seconds)).
		RunWith(r.DB).Exec()
	if err != nil {
		cclog.Warnf("Error while stopping jobs exceeding walltime: %v", err)
		return fmt.Errorf("failed to stop jobs exceeding walltime: %w", err)
	}

	rowsAffected, err := res.RowsAffected()
	if err != nil {
		cclog.Warnf("Error while fetching affected rows after stopping due to exceeded walltime: %v", err)
		return fmt.Errorf("failed to get rows affected count: %w", err)
	}

	if rowsAffected > 0 {
		cclog.Infof("%d jobs have been marked as failed due to running too long", rowsAffected)
	}
	cclog.Debugf("Timer StopJobsExceedingWalltimeBy %s", time.Since(start))
	return nil
}

// FindJobIdsByTag returns all job database IDs associated with a specific tag.
//
// Parameters:
//   - tagID: Database ID of the tag to search for
//
// Returns a slice of job IDs or an error if the query fails.
func (r *JobRepository) FindJobIdsByTag(tagID int64) ([]int64, error) {
	query := sq.Select("job.id").From("job").
		Join("jobtag ON jobtag.job_id = job.id").
		Where(sq.Eq{"jobtag.tag_id": tagID}).Distinct()
	rows, err := query.RunWith(r.stmtCache).Query()
	if err != nil {
		cclog.Errorf("Error while running FindJobIdsByTag query for tagID=%d: %v", tagID, err)
		return nil, fmt.Errorf("failed to find job IDs by tag %d: %w", tagID, err)
	}
	defer rows.Close()

	jobIds := make([]int64, 0, 100)

	for rows.Next() {
		var jobID int64

		if err := rows.Scan(&jobID); err != nil {
			cclog.Warnf("Error while scanning rows in FindJobIdsByTag: %v", err)
			return nil, fmt.Errorf("failed to scan job ID in FindJobIdsByTag: %w", err)
		}

		jobIds = append(jobIds, jobID)
	}

	return jobIds, nil
}

// FindRunningJobs returns all currently running jobs for a specific cluster.
// Filters out short-running jobs based on repoConfig.MinRunningJobDuration threshold.
//
// Parameters:
//   - cluster: Cluster name to filter jobs
//
// Returns a slice of running job objects or an error if the query fails.
func (r *JobRepository) FindRunningJobs(cluster string) ([]*schema.Job, error) {
	query := sq.Select(jobColumns...).From("job").
		Where("job.cluster = ?", cluster).
		Where("job.job_state = 'running'").
		Where("job.duration > ?", repoConfig.MinRunningJobDuration)

	rows, err := query.RunWith(r.stmtCache).Query()
	if err != nil {
		cclog.Errorf("Error while running FindRunningJobs query for cluster=%s: %v", cluster, err)
		return nil, fmt.Errorf("failed to find running jobs for cluster %s: %w", cluster, err)
	}
	defer rows.Close()

	jobs := make([]*schema.Job, 0, 50)
	for rows.Next() {
		job, err := scanJob(rows)
		if err != nil {
			cclog.Warnf("Error while scanning rows in FindRunningJobs: %v", err)
			return nil, fmt.Errorf("failed to scan job in FindRunningJobs: %w", err)
		}
		jobs = append(jobs, job)
	}

	cclog.Debugf("JobRepository.FindRunningJobs(): Return job count %d (cluster: %s)", len(jobs), cluster)
	return jobs, nil
}

// UpdateDuration recalculates and updates the duration field for all running jobs.
// Called periodically to keep job durations current without querying individual jobs.
//
// Duration is calculated as: current_time - job.start_time
//
// Returns an error if the database update fails.
func (r *JobRepository) UpdateDuration() error {
	stmnt := sq.Update("job").
		Set("duration", sq.Expr("? - job.start_time", time.Now().Unix())).
		Where("job_state = 'running'")

	_, err := stmnt.RunWith(r.stmtCache).Exec()
	if err != nil {
		cclog.Errorf("Error while updating duration for running jobs: %v", err)
		return fmt.Errorf("failed to update duration for running jobs: %w", err)
	}

	return nil
}

// FindJobsBetween returns jobs within a specified time range.
// If startTimeBegin is 0, returns all jobs before startTimeEnd.
// Optionally excludes tagged jobs from results.
//
// Parameters:
//   - startTimeBegin: Unix timestamp for range start (use 0 for unbounded start)
//   - startTimeEnd: Unix timestamp for range end
//   - omitTagged: "none" = include all jobs, "all" = exclude any tagged jobs,
//     "user" = exclude jobs with user-created tags (not auto-tagger types "app"/"jobClass")
//
// Returns a slice of jobs or an error if the time range is invalid or query fails.
func (r *JobRepository) FindJobsBetween(startTimeBegin int64, startTimeEnd int64, omitTagged string) ([]*schema.Job, error) {
	var query sq.SelectBuilder

	if startTimeBegin == startTimeEnd || startTimeBegin > startTimeEnd {
		return nil, errors.New("startTimeBegin is equal or larger startTimeEnd")
	}

	if startTimeBegin == 0 {
		cclog.Infof("Find jobs before %d", startTimeEnd)
		query = sq.Select(jobColumns...).From("job").Where("job.start_time < ?", startTimeEnd)
	} else {
		cclog.Infof("Find jobs between %d and %d", startTimeBegin, startTimeEnd)
		query = sq.Select(jobColumns...).From("job").Where("job.start_time BETWEEN ? AND ?", startTimeBegin, startTimeEnd)
	}

	switch omitTagged {
	case "all":
		query = query.Where("NOT EXISTS (SELECT 1 FROM jobtag WHERE jobtag.job_id = job.id)")
	case "user":
		query = query.Where("NOT EXISTS (SELECT 1 FROM jobtag JOIN tag ON tag.id = jobtag.tag_id WHERE jobtag.job_id = job.id AND tag.tag_type NOT IN ('app', 'jobClass'))")
	}

	query = query.OrderBy("job.cluster ASC", "job.subcluster ASC", "job.project ASC", "job.start_time ASC")

	rows, err := query.RunWith(r.stmtCache).Query()
	if err != nil {
		cclog.Errorf("Error while running FindJobsBetween query: %v", err)
		return nil, fmt.Errorf("failed to find jobs between %d and %d: %w", startTimeBegin, startTimeEnd, err)
	}
	defer rows.Close()

	jobs := make([]*schema.Job, 0, 50)
	for rows.Next() {
		job, err := scanJob(rows)
		if err != nil {
			cclog.Warnf("Error while scanning rows in FindJobsBetween: %v", err)
			return nil, fmt.Errorf("failed to scan job in FindJobsBetween: %w", err)
		}
		jobs = append(jobs, job)
	}

	cclog.Debugf("JobRepository.FindJobsBetween(): Return job count %d (omitTagged: %v)", len(jobs), omitTagged)
	return jobs, nil
}

// UpdateMonitoringStatus updates the monitoring status for a job and invalidates its cache entries.
// Cache invalidation affects both metadata and energy footprint to ensure consistency.
//
// Parameters:
//   - job: Database ID of the job to update
//   - monitoringStatus: New monitoring status value (see schema.MonitoringStatus constants)
//
// Returns an error if the database update fails.
func (r *JobRepository) UpdateMonitoringStatus(job int64, monitoringStatus int32) (err error) {
	// Invalidate cache entries as monitoring status affects job state
	r.cache.Del(fmt.Sprintf("metadata:%d", job))
	r.cache.Del(fmt.Sprintf("energyFootprint:%d", job))

	stmt := sq.Update("job").
		Set("monitoring_status", monitoringStatus).
		Where("job.id = ?", job)

	if _, err = stmt.RunWith(r.stmtCache).Exec(); err != nil {
		cclog.Errorf("Error while updating monitoring status for job %d: %v", job, err)
		return fmt.Errorf("failed to update monitoring status for job %d: %w", job, err)
	}
	return nil
}

// Execute runs a Squirrel UpdateBuilder statement against the database.
// This is a generic helper for executing pre-built update queries.
//
// Parameters:
//   - stmt: Squirrel UpdateBuilder with prepared update query
//
// Returns an error if the execution fails.
func (r *JobRepository) Execute(stmt sq.UpdateBuilder) error {
	if _, err := stmt.RunWith(r.stmtCache).Exec(); err != nil {
		cclog.Errorf("Error while executing statement: %v", err)
		return fmt.Errorf("failed to execute update statement: %w", err)
	}

	return nil
}

// MarkArchived adds monitoring status update to an existing UpdateBuilder statement.
// This is a builder helper used when constructing multi-field update queries.
//
// Parameters:
//   - stmt: Existing UpdateBuilder to modify
//   - monitoringStatus: Monitoring status value to set
//
// Returns the modified UpdateBuilder for method chaining.
func (r *JobRepository) MarkArchived(
	stmt sq.UpdateBuilder,
	monitoringStatus int32,
) sq.UpdateBuilder {
	return stmt.Set("monitoring_status", monitoringStatus)
}

// UpdateEnergy calculates and updates the energy consumption for a job.
// This is called for running jobs during intermediate updates or when archiving.
//
// Energy calculation formula:
//   - For "power" metrics: Energy (kWh) = (Power_avg * NumNodes * Duration_hours) / 1000
//   - For "energy" metrics: Currently not implemented (would need sum statistics)
//
// The calculation accounts for:
//   - Multi-node jobs: Multiplies by NumNodes to get total cluster energy
//   - Shared jobs: Node average is already based on partial resources, so NumNodes=1
//   - Unit conversion: Watts * hours / 1000 = kilowatt-hours (kWh)
//   - Rounding: Results rounded to 2 decimal places
func (r *JobRepository) UpdateEnergy(
	stmt sq.UpdateBuilder,
	jobMeta *schema.Job,
) (sq.UpdateBuilder, error) {
	sc, err := archive.GetSubCluster(jobMeta.Cluster, jobMeta.SubCluster)
	if err != nil {
		cclog.Errorf("cannot get subcluster: %s", err.Error())
		return stmt, err
	}
	energyFootprint := make(map[string]float64)

	// Accumulate total energy across all energy-related metrics
	totalEnergy := 0.0
	for _, fp := range sc.EnergyFootprint {
		// Calculate energy for this specific metric
		metricEnergy := 0.0
		if i, err := archive.MetricIndex(sc.MetricConfig, fp); err == nil {
			switch sc.MetricConfig[i].Energy {
			case "energy": // Metric already in energy units (Joules or Wh)
				cclog.Warnf("Update EnergyFootprint for Job %d and Metric %s on cluster %s: Set to 'energy' in cluster.json: Not implemented, will return 0.0", jobMeta.JobID, jobMeta.Cluster, fp)
				// FIXME: Needs sum as stats type to accumulate energy values over time
			case "power": // Metric in power units (Watts)
				// Energy (kWh) = Power (W) × Time (h) / 1000
				// Formula: (avg_power_per_node * num_nodes) * (duration_sec / 3600) / 1000
				//
				// Breakdown:
				//   LoadJobStat(jobMeta, fp, "avg") = average power per node (W)
				//   jobMeta.NumNodes = number of nodes (1 for shared jobs)
				//   jobMeta.Duration / 3600.0 = duration in hours
				//   / 1000.0 = convert Wh to kWh
				rawEnergy := ((LoadJobStat(jobMeta, fp, "avg") * float64(jobMeta.NumNodes)) * (float64(jobMeta.Duration) / 3600.0)) / 1000.0
				metricEnergy = math.Round(rawEnergy*100.0) / 100.0 // Round to 2 decimal places
			}
		} else {
			cclog.Warnf("Error while collecting energy metric %s for job, DB ID '%v', return '0.0'", fp, jobMeta.ID)
		}

		energyFootprint[fp] = metricEnergy
		totalEnergy += metricEnergy
	}

	var rawFootprint []byte
	if rawFootprint, err = json.Marshal(energyFootprint); err != nil {
		cclog.Warnf("Error while marshaling energy footprint for job INTO BYTES, DB ID '%v'", jobMeta.ID)
		return stmt, err
	}

	return stmt.Set("energy_footprint", string(rawFootprint)).Set("energy", (math.Round(totalEnergy*100.0) / 100.0)), nil
}

// UpdateFootprint calculates and updates the performance footprint for a job.
// This is called for running jobs during intermediate updates or when archiving.
//
// A footprint is a summary statistic (avg/min/max) for each monitored metric.
// The specific statistic type is defined in the cluster config's Footprint field.
// Results are stored as JSON with keys like "metric_avg", "metric_max", etc.
//
// Example: For a "cpu_load" metric with Footprint="avg", this stores
// the average CPU load across all nodes as "cpu_load_avg": 85.3
func (r *JobRepository) UpdateFootprint(
	stmt sq.UpdateBuilder,
	jobMeta *schema.Job,
) (sq.UpdateBuilder, error) {
	sc, err := archive.GetSubCluster(jobMeta.Cluster, jobMeta.SubCluster)
	if err != nil {
		cclog.Errorf("cannot get subcluster: %s", err.Error())
		return stmt, err
	}
	footprint := make(map[string]float64)

	// Build footprint map with metric_stattype as keys
	for _, fp := range sc.Footprint {
		// Determine which statistic to use: avg, min, or max
		// First check global metric config, then cluster-specific config
		var statType string
		for _, gm := range archive.GlobalMetricList {
			if gm.Name == fp {
				statType = gm.Footprint
			}
		}

		// Validate statistic type
		if statType != "avg" && statType != "min" && statType != "max" {
			cclog.Warnf("unknown statType for footprint update: %s", statType)
			return stmt, fmt.Errorf("unknown statType for footprint update: %s", statType)
		}

		// Override with cluster-specific config if available
		if i, err := archive.MetricIndex(sc.MetricConfig, fp); err != nil {
			statType = sc.MetricConfig[i].Footprint
		}

		// Store as "metric_stattype": value (e.g., "cpu_load_avg": 85.3)
		name := fmt.Sprintf("%s_%s", fp, statType)
		footprint[name] = LoadJobStat(jobMeta, fp, statType)
	}

	var rawFootprint []byte
	if rawFootprint, err = json.Marshal(footprint); err != nil {
		cclog.Warnf("Error while marshaling footprint for job INTO BYTES, DB ID '%v'", jobMeta.ID)
		return stmt, err
	}

	return stmt.Set("footprint", string(rawFootprint)), nil
}

// GetUsedNodes returns a map of cluster names to sorted lists of unique hostnames
// that are currently in use by jobs that started before the given timestamp and
// are still in running state.
//
// The timestamp parameter (ts) is compared against job.start_time to find
// relevant jobs. Returns an error if the database query fails or row iteration
// encounters errors. Individual row parsing errors are logged but don't fail
// the entire operation.
func (r *JobRepository) GetUsedNodes(ts int64) (map[string][]string, error) {
	// Note: Query expects index on (job_state, start_time) for optimal performance
	q := sq.Select("job.cluster", "job.resources").From("job").
		Where("job.start_time < ?", ts).
		Where(sq.Eq{"job.job_state": "running"})

	rows, err := q.RunWith(r.stmtCache).Query()
	if err != nil {
		queryString, queryVars, _ := q.ToSql()
		return nil, fmt.Errorf("query failed [%s] %v: %w", queryString, queryVars, err)
	}
	defer rows.Close()

	// Use a map of sets for efficient deduplication
	nodeSet := make(map[string]map[string]struct{})

	var (
		cluster      string
		rawResources []byte
		resources    []*schema.Resource
		skippedRows  int
	)

	for rows.Next() {
		if err := rows.Scan(&cluster, &rawResources); err != nil {
			cclog.Warnf("Error scanning job row in GetUsedNodes: %v", err)
			skippedRows++
			continue
		}

		resources = resources[:0] // Clear slice, keep capacity
		if err := json.Unmarshal(rawResources, &resources); err != nil {
			cclog.Warnf("Error unmarshaling resources for cluster %s: %v", cluster, err)
			skippedRows++
			continue
		}

		if len(resources) == 0 {
			cclog.Debugf("Job in cluster %s has no resources", cluster)
			continue
		}

		if _, ok := nodeSet[cluster]; !ok {
			nodeSet[cluster] = make(map[string]struct{})
		}

		for _, res := range resources {
			nodeSet[cluster][res.Hostname] = struct{}{}
		}
	}

	if err := rows.Err(); err != nil {
		return nil, fmt.Errorf("error iterating rows: %w", err)
	}

	if skippedRows > 0 {
		cclog.Warnf("GetUsedNodes: Skipped %d rows due to parsing errors", skippedRows)
	}

	// Convert sets to sorted slices
	nodeList := make(map[string][]string, len(nodeSet))
	for cluster, nodes := range nodeSet {
		list := make([]string, 0, len(nodes))
		for node := range nodes {
			list = append(list, node)
		}
		sort.Strings(list)
		nodeList[cluster] = list
	}

	return nodeList, nil
}

-    Where(sq.Eq{"job.job_state": "running"})


Bash

go build ./cmd/cc-backend && go test ./internal/repository/...

Build and test


ok  	github.com/ClusterCockpit/cc-backend/internal/repository	(cached)


go test -count=1 ./internal/repository/...

Re-run tests without cache


ok  	github.com/ClusterCockpit/cc-backend/internal/repository	0.976s


Both changes applied and tests pass:

- **`stats.go:116`** — `query.Where("job.job_state = ?", "running")` → `query.Where("job.job_state = 'running'")`
- **`job.go:1069`** — `sq.Eq{"job.job_state": "running"}` → `Where("job.job_state = 'running'")`

This ensures SQLite can use partial indexes on `job_state = 'running'` for these queries, since the literal must appear directly in the SQL (not as a bound parameter).