Implement the following plan:

# Fix SQLite Memory Not Released After Query Timeout

## Context

On the production 20M-row database, when a query runs into a timeout (due to full-table scan with wrong index), the memory allocated by SQLite is **not released afterwards**. The process stays bloated until restarted. This is caused by three compounding issues in the current SQLite configuration.

## Root Cause Analysis

### 1. `_cache_size=1000000000` is effectively unlimited (~4TB)

**File:** `internal/repository/dbConnection.go:82`

```go
connectionURLParams.Add("_cache_size", "1000000000")
```

SQLite's `cache_size` PRAGMA interprets **positive values as page count** (default page size = 4KB). So 1,000,000,000 pages × 4KB = ~4TB. In practice, this means "never evict cached pages." After a full-table scan of 20M rows, every page touched stays in SQLite's page cache. With 4 connections (`MaxOpenConns=4`), each can independently cache gigabytes.

For comparison, the SQLite archive backend in `pkg/archive/sqliteBackend.go` uses `PRAGMA cache_size=-64000` (64MB — negative = KiB).

### 2. No query context/timeout — queries run indefinitely

**File:** `internal/repository/jobQuery.go:87`

```go
rows, err := query.RunWith(r.stmtCache).Query()  // No context!
```

The `ctx` parameter is available but never passed to the database layer. Squirrel supports `.QueryContext(ctx)` but it's not used. If the HTTP request times out or the client disconnects, the query keeps running and scanning pages into cache.

### 3. No SQLite memory limit — no `soft_heap_limit` or `shrink_memory`

SQLite has built-in memory management PRAGMAs that are not configured:
- **`soft_heap_limit`** — asks SQLite to keep heap usage below N bytes (best-effort, releases cache pages to stay under limit)
- **`hard_heap_limit`** — hard cap, queries fail with SQLITE_NOMEM if exceeded
- **`shrink_memory`** — immediately releases all unused memory back to the OS

None of these are set, so SQLite allocates freely and never releases.

### 4. `temp_store = memory` amplifies the problem

**File:** `internal/repository/dbConnection.go:41`

Temporary B-tree sorts (exactly what happens during ORDER BY on a full-table scan) are stored in RAM. With 20M rows and no sort optimization, this can be gigabytes of temporary memory on top of the page cache.

### 5. Connections live for 1 hour after use

`ConnMaxIdleTime = 1 hour` means a connection that just did a massive full-table scan sits idle in the pool for up to an hour, holding all its cached pages.

## Proposed Changes

### Fix 1: Set reasonable `cache_size` (high impact, low risk)

**File:** `internal/repository/dbConnection.go:82`

Change from `1000000000` (1B pages ≈ 4TB) to `-200000` (200MB in KiB notation, per connection):

```go
connectionURLParams.Add("_cache_size", "-200000")  // 200MB per connection
```

With 4 max connections: 4 × 200MB = 800MB max page cache. This is generous enough for normal queries but prevents runaway memory after full-table scans.

### Fix 2: Add `soft_heap_limit` (high impact, low risk)

**File:** `internal/repository/dbConnection.go`, in `setupSqlite()`:

```go
"soft_heap_limit = 1073741824",  // 1GB soft limit across all connections
```

This is a **process-wide** limit (not per-connection). SQLite will try to release cache pages to stay under 1GB total. It's a soft limit — it won't abort queries, just evicts cache more aggressively.

### Fix 3: Pass context to database queries (medium impact, medium effort)

Change `.Query()` to `.QueryContext(ctx)`, `.QueryRow()` to `.QueryRowContext(ctx)`, and `.Scan()` to `.ScanContext(ctx)` for all query methods that already receive a `ctx` parameter. This allows HTTP request cancellation to stop the SQLite query.

**Note:** The `stmtCache` from squirrel supports `QueryContext`/`QueryRowContext`. Only methods that already have `ctx` are changed — no signature changes needed.

**Call sites to update** (methods that have `ctx` and call `.Query()`/`.QueryRow()`/`.Scan()`):

| File | Method | Line | Change |
|------|--------|------|--------|
| `jobQuery.go` | `QueryJobs` | 87 | `.Query()` → `.QueryContext(ctx)` |
| `jobQuery.go` | `CountJobs` | 129 | `.Scan()` → `.ScanContext(ctx)` |
| `stats.go` | `JobsStatsGrouped` | 233 | `.Query()` → `.QueryContext(ctx)` |
| `stats.go` | `JobsStats` | 358 | `.QueryRow()` → `.QueryRowContext(ctx)` |
| `stats.go` | `JobCountGrouped` | 443 | `.Query()` → `.QueryContext(ctx)` |
| `stats.go` | `AddJobCountGrouped` | 504 | `.Query()` → `.QueryContext(ctx)` |
| `stats.go` | `AddJobCount` | 569 | `.Scan()` → `.ScanContext(ctx)` |
| `stats.go` | `jobsStatisticsHistogram` | 758 | `.Query()` → `.QueryContext(ctx)` |
| `stats.go` | `jobsDurationStatisticsHistogram` | 832 | `.Query()` → `.QueryContext(ctx)` |
| `stats.go` | `jobsMetricStatisticsHistogram` | 962 | `.Query()` → `.QueryContext(ctx)` |
| `jobFind.go` | `FindByID` | 174 | `.QueryRow()` → `.QueryRowContext(ctx)` |
| `jobFind.go` | `FindByJobID` | 220 | `.QueryRow()` → `.QueryRowContext(ctx)` |
| `job.go` | `CountGroupedJobs` | 410 | `.Scan()` → `.ScanContext(ctx)` (needs ctx added to signature) |
| `job.go` | `GetJobList` | 751 | `.Query()` → `.QueryContext(ctx)` (needs ctx added to signature) |

Methods without `ctx` in their signature (e.g., `GetJobList`, `CountGroupedJobs`) can either have `ctx` added or be left for a follow-up. The priority is the methods already accepting `ctx`.

### Fix 4: Reduce `ConnMaxIdleTime`

**File:** `internal/repository/config.go`

Reduce `ConnectionMaxIdleTime` from 1 hour to 10 minutes. Idle connections with cached pages are released sooner.

## Scope — All Four Fixes

All fixes are complementary, independent, and low-risk. No disadvantages to applying all together.

## Files to Modify

- `internal/repository/dbConnection.go` — cache_size, soft_heap_limit (Fixes 1 & 2)
- `internal/repository/jobQuery.go` — QueryContext (Fix 3)
- `internal/repository/job.go` — QueryContext for other query methods (Fix 3)
- `internal/repository/stats.go` — QueryContext for stats queries (Fix 3)
- `internal/repository/config.go` — ConnMaxIdleTime (Fix 4, optional)

## Verification

1. `go build ./...` — compiles
2. `go test ./internal/repository/...` — tests pass
3. Monitor RSS memory before/after on production:
   - Trigger a heavy query, observe memory spike
   - Verify memory drops back down within seconds (not hours)
4. `PRAGMA cache_size;` in sqlite3 CLI on running DB — should show `-200000`
5. `PRAGMA soft_heap_limit;` — should show `1073741824`

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-CC-cc-backend/c31c699a-f492-48f7-bcf0-35d3ceeac243.jsonl


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package repository
     7→
     8→import (
     9→	"database/sql"
    10→	"fmt"
    11→	"net/url"
    12→	"sync"
    13→	"time"
    14→
    15→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    16→	"github.com/jmoiron/sqlx"
    17→	"github.com/mattn/go-sqlite3"
    18→	"github.com/qustavo/sqlhooks/v2"
    19→)
    20→
    21→var (
    22→	dbConnOnce     sync.Once
    23→	dbConnInstance *DBConnection
    24→)
    25→
    26→type DBConnection struct {
    27→	DB     *sqlx.DB
    28→	Driver string
    29→}
    30→
    31→type DatabaseOptions struct {
    32→	URL                   string
    33→	MaxOpenConnections    int
    34→	MaxIdleConnections    int
    35→	ConnectionMaxLifetime time.Duration
    36→	ConnectionMaxIdleTime time.Duration
    37→}
    38→
    39→func setupSqlite(db *sql.DB) error {
    40→	pragmas := []string{
    41→		"temp_store = memory",
    42→	}
    43→
    44→	for _, pragma := range pragmas {
    45→		_, err := db.Exec("PRAGMA " + pragma)
    46→		if err != nil {
    47→			return err
    48→		}
    49→	}
    50→
    51→	// Update query planner statistics so SQLite picks optimal indexes.
    52→	// Without this, SQLite guesses row distributions and often chooses wrong
    53→	// indexes for queries with IN clauses + ORDER BY, causing full table sorts
    54→	// in temp B-trees instead of using covering indexes.
    55→	if _, err := db.Exec("ANALYZE"); err != nil {
    56→		cclog.Warnf("Failed to run ANALYZE: %v", err)
    57→	}
    58→
    59→	return nil
    60→}
    61→
    62→func Connect(db string) {
    63→	var err error
    64→	var dbHandle *sqlx.DB
    65→
    66→	dbConnOnce.Do(func() {
    67→		opts := DatabaseOptions{
    68→			URL:                   db,
    69→			MaxOpenConnections:    repoConfig.MaxOpenConnections,
    70→			MaxIdleConnections:    repoConfig.MaxIdleConnections,
    71→			ConnectionMaxLifetime: repoConfig.ConnectionMaxLifetime,
    72→			ConnectionMaxIdleTime: repoConfig.ConnectionMaxIdleTime,
    73→		}
    74→
    75→		// TODO: Have separate DB handles for Writes and Reads
    76→		// Optimize SQLite connection: https://kerkour.com/sqlite-for-servers
    77→		connectionURLParams := make(url.Values)
    78→		connectionURLParams.Add("_txlock", "immediate")
    79→		connectionURLParams.Add("_journal_mode", "WAL")
    80→		connectionURLParams.Add("_busy_timeout", "5000")
    81→		connectionURLParams.Add("_synchronous", "NORMAL")
    82→		connectionURLParams.Add("_cache_size", "1000000000")
    83→		connectionURLParams.Add("_foreign_keys", "true")
    84→		opts.URL = fmt.Sprintf("file:%s?%s", opts.URL, connectionURLParams.Encode())
    85→
    86→		if cclog.Loglevel() == "debug" {
    87→			sql.Register("sqlite3WithHooks", sqlhooks.Wrap(&sqlite3.SQLiteDriver{}, &Hooks{}))
    88→			dbHandle, err = sqlx.Open("sqlite3WithHooks", opts.URL)
    89→		} else {
    90→			dbHandle, err = sqlx.Open("sqlite3", opts.URL)
    91→		}
    92→
    93→		if err != nil {
    94→			cclog.Abortf("DB Connection: Could not connect to SQLite database with sqlx.Open().\nError: %s\n", err.Error())
    95→		}
    96→
    97→		err = setupSqlite(dbHandle.DB)
    98→		if err != nil {
    99→			cclog.Abortf("Failed sqlite db setup.\nError: %s\n", err.Error())
   100→		}
   101→
   102→		dbHandle.SetMaxOpenConns(opts.MaxOpenConnections)
   103→		dbHandle.SetMaxIdleConns(opts.MaxIdleConnections)
   104→		dbHandle.SetConnMaxLifetime(opts.ConnectionMaxLifetime)
   105→		dbHandle.SetConnMaxIdleTime(opts.ConnectionMaxIdleTime)
   106→
   107→		dbConnInstance = &DBConnection{DB: dbHandle}
   108→		err = checkDBVersion(dbHandle.DB)
   109→		if err != nil {
   110→			cclog.Abortf("DB Connection: Failed DB version check.\nError: %s\n", err.Error())
   111→		}
   112→	})
   113→}
   114→
   115→func GetConnection() *DBConnection {
   116→	if dbConnInstance == nil {
   117→		cclog.Fatalf("Database connection not initialized!")
   118→	}
   119→
   120→	return dbConnInstance
   121→}
   122→
   123→// ResetConnection closes the current database connection and resets the connection state.
   124→// This function is intended for testing purposes only to allow test isolation.
   125→func ResetConnection() error {
   126→	if dbConnInstance != nil && dbConnInstance.DB != nil {
   127→		if err := dbConnInstance.DB.Close(); err != nil {
   128→			return fmt.Errorf("failed to close database connection: %w", err)
   129→		}
   130→	}
   131→
   132→	dbConnInstance = nil
   133→	dbConnOnce = sync.Once{}
   134→	jobRepoInstance = nil
   135→	jobRepoOnce = sync.Once{}
   136→	nodeRepoInstance = nil
   137→	nodeRepoOnce = sync.Once{}
   138→	userRepoInstance = nil
   139→	userRepoOnce = sync.Once{}
   140→	userCfgRepoInstance = nil
   141→	userCfgRepoOnce = sync.Once{}
   142→
   143→	return nil
   144→}
   145→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package repository

import (
	"database/sql"
	"fmt"
	"net/url"
	"sync"
	"time"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/jmoiron/sqlx"
	"github.com/mattn/go-sqlite3"
	"github.com/qustavo/sqlhooks/v2"
)

var (
	dbConnOnce     sync.Once
	dbConnInstance *DBConnection
)

type DBConnection struct {
	DB     *sqlx.DB
	Driver string
}

type DatabaseOptions struct {
	URL                   string
	MaxOpenConnections    int
	MaxIdleConnections    int
	ConnectionMaxLifetime time.Duration
	ConnectionMaxIdleTime time.Duration
}

func setupSqlite(db *sql.DB) error {
	pragmas := []string{
		"temp_store = memory",
	}

	for _, pragma := range pragmas {
		_, err := db.Exec("PRAGMA " + pragma)
		if err != nil {
			return err
		}
	}

	// Update query planner statistics so SQLite picks optimal indexes.
	// Without this, SQLite guesses row distributions and often chooses wrong
	// indexes for queries with IN clauses + ORDER BY, causing full table sorts
	// in temp B-trees instead of using covering indexes.
	if _, err := db.Exec("ANALYZE"); err != nil {
		cclog.Warnf("Failed to run ANALYZE: %v", err)
	}

	return nil
}

func Connect(db string) {
	var err error
	var dbHandle *sqlx.DB

	dbConnOnce.Do(func() {
		opts := DatabaseOptions{
			URL:                   db,
			MaxOpenConnections:    repoConfig.MaxOpenConnections,
			MaxIdleConnections:    repoConfig.MaxIdleConnections,
			ConnectionMaxLifetime: repoConfig.ConnectionMaxLifetime,
			ConnectionMaxIdleTime: repoConfig.ConnectionMaxIdleTime,
		}

		// TODO: Have separate DB handles for Writes and Reads
		// Optimize SQLite connection: https://kerkour.com/sqlite-for-servers
		connectionURLParams := make(url.Values)
		connectionURLParams.Add("_txlock", "immediate")
		connectionURLParams.Add("_journal_mode", "WAL")
		connectionURLParams.Add("_busy_timeout", "5000")
		connectionURLParams.Add("_synchronous", "NORMAL")
		connectionURLParams.Add("_cache_size", "1000000000")
		connectionURLParams.Add("_foreign_keys", "true")
		opts.URL = fmt.Sprintf("file:%s?%s", opts.URL, connectionURLParams.Encode())

		if cclog.Loglevel() == "debug" {
			sql.Register("sqlite3WithHooks", sqlhooks.Wrap(&sqlite3.SQLiteDriver{}, &Hooks{}))
			dbHandle, err = sqlx.Open("sqlite3WithHooks", opts.URL)
		} else {
			dbHandle, err = sqlx.Open("sqlite3", opts.URL)
		}

		if err != nil {
			cclog.Abortf("DB Connection: Could not connect to SQLite database with sqlx.Open().\nError: %s\n", err.Error())
		}

		err = setupSqlite(dbHandle.DB)
		if err != nil {
			cclog.Abortf("Failed sqlite db setup.\nError: %s\n", err.Error())
		}

		dbHandle.SetMaxOpenConns(opts.MaxOpenConnections)
		dbHandle.SetMaxIdleConns(opts.MaxIdleConnections)
		dbHandle.SetConnMaxLifetime(opts.ConnectionMaxLifetime)
		dbHandle.SetConnMaxIdleTime(opts.ConnectionMaxIdleTime)

		dbConnInstance = &DBConnection{DB: dbHandle}
		err = checkDBVersion(dbHandle.DB)
		if err != nil {
			cclog.Abortf("DB Connection: Failed DB version check.\nError: %s\n", err.Error())
		}
	})
}

func GetConnection() *DBConnection {
	if dbConnInstance == nil {
		cclog.Fatalf("Database connection not initialized!")
	}

	return dbConnInstance
}

// ResetConnection closes the current database connection and resets the connection state.
// This function is intended for testing purposes only to allow test isolation.
func ResetConnection() error {
	if dbConnInstance != nil && dbConnInstance.DB != nil {
		if err := dbConnInstance.DB.Close(); err != nil {
			return fmt.Errorf("failed to close database connection: %w", err)
		}
	}

	dbConnInstance = nil
	dbConnOnce = sync.Once{}
	jobRepoInstance = nil
	jobRepoOnce = sync.Once{}
	nodeRepoInstance = nil
	nodeRepoOnce = sync.Once{}
	userRepoInstance = nil
	userRepoOnce = sync.Once{}
	userCfgRepoInstance = nil
	userCfgRepoOnce = sync.Once{}

	return nil
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package repository
     7→
     8→import "time"
     9→
    10→// RepositoryConfig holds configuration for repository operations.
    11→// All fields have sensible defaults, so this configuration is optional.
    12→type RepositoryConfig struct {
    13→	// CacheSize is the LRU cache size in bytes for job metadata and energy footprints.
    14→	// Default: 1MB (1024 * 1024 bytes)
    15→	CacheSize int
    16→
    17→	// MaxOpenConnections is the maximum number of open database connections.
    18→	// Default: 4
    19→	MaxOpenConnections int
    20→
    21→	// MaxIdleConnections is the maximum number of idle database connections.
    22→	// Default: 4
    23→	MaxIdleConnections int
    24→
    25→	// ConnectionMaxLifetime is the maximum amount of time a connection may be reused.
    26→	// Default: 1 hour
    27→	ConnectionMaxLifetime time.Duration
    28→
    29→	// ConnectionMaxIdleTime is the maximum amount of time a connection may be idle.
    30→	// Default: 1 hour
    31→	ConnectionMaxIdleTime time.Duration
    32→
    33→	// MinRunningJobDuration is the minimum duration in seconds for a job to be
    34→	// considered in "running jobs" queries. This filters out very short jobs.
    35→	// Default: 600 seconds (10 minutes)
    36→	MinRunningJobDuration int
    37→}
    38→
    39→// DefaultConfig returns the default repository configuration.
    40→// These values are optimized for typical deployments.
    41→func DefaultConfig() *RepositoryConfig {
    42→	return &RepositoryConfig{
    43→		CacheSize:             1 * 1024 * 1024, // 1MB
    44→		MaxOpenConnections:    4,
    45→		MaxIdleConnections:    4,
    46→		ConnectionMaxLifetime: time.Hour,
    47→		ConnectionMaxIdleTime: time.Hour,
    48→		MinRunningJobDuration: 600, // 10 minutes
    49→	}
    50→}
    51→
    52→// repoConfig is the package-level configuration instance.
    53→// It is initialized with defaults and can be overridden via SetConfig.
    54→var repoConfig *RepositoryConfig = DefaultConfig()
    55→
    56→// SetConfig sets the repository configuration.
    57→// This must be called before any repository initialization (Connect, GetJobRepository, etc.).
    58→// If not called, default values from DefaultConfig() are used.
    59→func SetConfig(cfg *RepositoryConfig) {
    60→	if cfg != nil {
    61→		repoConfig = cfg
    62→	}
    63→}
    64→
    65→// GetConfig returns the current repository configuration.
    66→func GetConfig() *RepositoryConfig {
    67→	return repoConfig
    68→}
    69→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package repository

import "time"

// RepositoryConfig holds configuration for repository operations.
// All fields have sensible defaults, so this configuration is optional.
type RepositoryConfig struct {
	// CacheSize is the LRU cache size in bytes for job metadata and energy footprints.
	// Default: 1MB (1024 * 1024 bytes)
	CacheSize int

	// MaxOpenConnections is the maximum number of open database connections.
	// Default: 4
	MaxOpenConnections int

	// MaxIdleConnections is the maximum number of idle database connections.
	// Default: 4
	MaxIdleConnections int

	// ConnectionMaxLifetime is the maximum amount of time a connection may be reused.
	// Default: 1 hour
	ConnectionMaxLifetime time.Duration

	// ConnectionMaxIdleTime is the maximum amount of time a connection may be idle.
	// Default: 1 hour
	ConnectionMaxIdleTime time.Duration

	// MinRunningJobDuration is the minimum duration in seconds for a job to be
	// considered in "running jobs" queries. This filters out very short jobs.
	// Default: 600 seconds (10 minutes)
	MinRunningJobDuration int
}

// DefaultConfig returns the default repository configuration.
// These values are optimized for typical deployments.
func DefaultConfig() *RepositoryConfig {
	return &RepositoryConfig{
		CacheSize:             1 * 1024 * 1024, // 1MB
		MaxOpenConnections:    4,
		MaxIdleConnections:    4,
		ConnectionMaxLifetime: time.Hour,
		ConnectionMaxIdleTime: time.Hour,
		MinRunningJobDuration: 600, // 10 minutes
	}
}

// repoConfig is the package-level configuration instance.
// It is initialized with defaults and can be overridden via SetConfig.
var repoConfig *RepositoryConfig = DefaultConfig()

// SetConfig sets the repository configuration.
// This must be called before any repository initialization (Connect, GetJobRepository, etc.).
// If not called, default values from DefaultConfig() are used.
func SetConfig(cfg *RepositoryConfig) {
	if cfg != nil {
		repoConfig = cfg
	}
}

// GetConfig returns the current repository configuration.
func GetConfig() *RepositoryConfig {
	return repoConfig
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package repository provides job query functionality with filtering, pagination,
     7→// and security controls. This file contains the main query builders and security
     8→// checks for job retrieval operations.
     9→package repository
    10→
    11→import (
    12→	"context"
    13→	"errors"
    14→	"fmt"
    15→	"regexp"
    16→	"strings"
    17→	"time"
    18→
    19→	"github.com/ClusterCockpit/cc-backend/internal/config"
    20→	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
    21→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    22→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    23→	sq "github.com/Masterminds/squirrel"
    24→)
    25→
    26→const (
    27→	// Default initial capacity for job result slices
    28→	defaultJobsCapacity = 50
    29→)
    30→
    31→// QueryJobs retrieves jobs from the database with optional filtering, pagination,
    32→// and sorting. Security controls are automatically applied based on the user context.
    33→//
    34→// Parameters:
    35→//   - ctx: Context containing user authentication information
    36→//   - filters: Optional job filters (cluster, state, user, time ranges, etc.)
    37→//   - page: Optional pagination parameters (page number and items per page)
    38→//   - order: Optional sorting specification (column or footprint field)
    39→//
    40→// Returns a slice of jobs matching the criteria, or an error if the query fails.
    41→// The function enforces role-based access control through SecurityCheck.
    42→func (r *JobRepository) QueryJobs(
    43→	ctx context.Context,
    44→	filters []*model.JobFilter,
    45→	page *model.PageRequest,
    46→	order *model.OrderByInput,
    47→) ([]*schema.Job, error) {
    48→	query, qerr := SecurityCheck(ctx, sq.Select(jobColumns...).From("job"))
    49→	if qerr != nil {
    50→		return nil, qerr
    51→	}
    52→
    53→	if order != nil {
    54→		field := toSnakeCase(order.Field)
    55→		if order.Type == "col" {
    56→			switch order.Order {
    57→			case model.SortDirectionEnumAsc:
    58→				query = query.OrderBy(fmt.Sprintf("job.%s ASC", field))
    59→			case model.SortDirectionEnumDesc:
    60→				query = query.OrderBy(fmt.Sprintf("job.%s DESC", field))
    61→			default:
    62→				return nil, errors.New("invalid sorting order for column")
    63→			}
    64→		} else {
    65→			// Order by footprint JSON field values
    66→			query = query.Where("JSON_VALID(footprint)")
    67→			switch order.Order {
    68→			case model.SortDirectionEnumAsc:
    69→				query = query.OrderBy(fmt.Sprintf("JSON_EXTRACT(footprint, \"$.%s\") ASC", field))
    70→			case model.SortDirectionEnumDesc:
    71→				query = query.OrderBy(fmt.Sprintf("JSON_EXTRACT(footprint, \"$.%s\") DESC", field))
    72→			default:
    73→				return nil, errors.New("invalid sorting order for footprint")
    74→			}
    75→		}
    76→	}
    77→
    78→	if page != nil && page.ItemsPerPage != -1 {
    79→		limit := uint64(page.ItemsPerPage)
    80→		query = query.Offset((uint64(page.Page) - 1) * limit).Limit(limit)
    81→	}
    82→
    83→	for _, f := range filters {
    84→		query = BuildWhereClause(f, query)
    85→	}
    86→
    87→	rows, err := query.RunWith(r.stmtCache).Query()
    88→	if err != nil {
    89→		queryString, queryVars, _ := query.ToSql()
    90→		return nil, fmt.Errorf("query failed [%s] %v: %w", queryString, queryVars, err)
    91→	}
    92→	defer rows.Close()
    93→
    94→	jobs := make([]*schema.Job, 0, defaultJobsCapacity)
    95→	for rows.Next() {
    96→		job, err := scanJob(rows)
    97→		if err != nil {
    98→			cclog.Warnf("Error scanning job row: %v", err)
    99→			return nil, fmt.Errorf("failed to scan job row: %w", err)
   100→		}
   101→		jobs = append(jobs, job)
   102→	}
   103→
   104→	if err := rows.Err(); err != nil {
   105→		return nil, fmt.Errorf("error iterating job rows: %w", err)
   106→	}
   107→
   108→	return jobs, nil
   109→}
   110→
   111→// CountJobs returns the total number of jobs matching the given filters.
   112→// Security controls are automatically applied based on the user context.
   113→// Uses DISTINCT count to handle tag filters correctly (jobs may appear multiple
   114→// times when joined with the tag table).
   115→func (r *JobRepository) CountJobs(
   116→	ctx context.Context,
   117→	filters []*model.JobFilter,
   118→) (int, error) {
   119→	query, qerr := SecurityCheck(ctx, sq.Select("count(DISTINCT job.id)").From("job"))
   120→	if qerr != nil {
   121→		return 0, qerr
   122→	}
   123→
   124→	for _, f := range filters {
   125→		query = BuildWhereClause(f, query)
   126→	}
   127→
   128→	var count int
   129→	if err := query.RunWith(r.DB).Scan(&count); err != nil {
   130→		return 0, fmt.Errorf("failed to count jobs: %w", err)
   131→	}
   132→
   133→	return count, nil
   134→}
   135→
   136→// SecurityCheckWithUser applies role-based access control filters to a job query
   137→// based on the provided user's roles and permissions.
   138→//
   139→// Access rules by role:
   140→//   - API role (exclusive): Full access to all jobs
   141→//   - Admin/Support roles: Full access to all jobs
   142→//   - Manager role: Access to jobs in managed projects plus own jobs
   143→//   - User role: Access only to own jobs
   144→//
   145→// Returns an error if the user is nil or has no recognized roles.
   146→func SecurityCheckWithUser(user *schema.User, query sq.SelectBuilder) (sq.SelectBuilder, error) {
   147→	if user == nil {
   148→		var qnil sq.SelectBuilder
   149→		return qnil, fmt.Errorf("user context is nil")
   150→	}
   151→
   152→	switch {
   153→	case len(user.Roles) == 1 && user.HasRole(schema.RoleAPI):
   154→		return query, nil
   155→	case user.HasAnyRole([]schema.Role{schema.RoleAdmin, schema.RoleSupport}):
   156→		return query, nil
   157→	case user.HasRole(schema.RoleManager):
   158→		if len(user.Projects) != 0 {
   159→			return query.Where(sq.Or{sq.Eq{"job.project": user.Projects}, sq.Eq{"job.hpc_user": user.Username}}), nil
   160→		}
   161→		cclog.Debugf("Manager '%s' has no assigned projects, restricting to personal jobs", user.Username)
   162→		return query.Where("job.hpc_user = ?", user.Username), nil
   163→	case user.HasRole(schema.RoleUser):
   164→		return query.Where("job.hpc_user = ?", user.Username), nil
   165→	default:
   166→		var qnil sq.SelectBuilder
   167→		return qnil, fmt.Errorf("user has no or unknown roles")
   168→	}
   169→}
   170→
   171→// SecurityCheck extracts the user from the context and applies role-based access
   172→// control filters to the query. This is a convenience wrapper around SecurityCheckWithUser.
   173→func SecurityCheck(ctx context.Context, query sq.SelectBuilder) (sq.SelectBuilder, error) {
   174→	user := GetUserFromContext(ctx)
   175→	return SecurityCheckWithUser(user, query)
   176→}
   177→
   178→// BuildWhereClause constructs SQL WHERE conditions from a JobFilter and applies
   179→// them to the query. Supports filtering by job properties (cluster, state, user),
   180→// time ranges, resource usage, tags, and JSON field searches in meta_data,
   181→// footprint, and resources columns.
   182→func BuildWhereClause(filter *model.JobFilter, query sq.SelectBuilder) sq.SelectBuilder {
   183→	// Primary Key
   184→	if filter.DbID != nil {
   185→		dbIDs := make([]string, len(filter.DbID))
   186→		copy(dbIDs, filter.DbID)
   187→		query = query.Where(sq.Eq{"job.id": dbIDs})
   188→	}
   189→	// Explicit indices
   190→	if filter.Cluster != nil {
   191→		query = buildStringCondition("job.cluster", filter.Cluster, query)
   192→	}
   193→	if filter.SubCluster != nil {
   194→		query = buildStringCondition("job.subcluster", filter.SubCluster, query)
   195→	}
   196→	if filter.Partition != nil {
   197→		query = buildStringCondition("job.cluster_partition", filter.Partition, query)
   198→	}
   199→	if filter.State != nil {
   200→		states := make([]string, len(filter.State))
   201→		for i, val := range filter.State {
   202→			states[i] = string(val)
   203→		}
   204→		query = query.Where(sq.Eq{"job.job_state": states})
   205→	}
   206→	if filter.Shared != nil {
   207→		query = query.Where("job.shared = ?", *filter.Shared)
   208→	}
   209→	if filter.Project != nil {
   210→		query = buildStringCondition("job.project", filter.Project, query)
   211→	}
   212→	if filter.User != nil {
   213→		query = buildStringCondition("job.hpc_user", filter.User, query)
   214→	}
   215→	if filter.NumNodes != nil {
   216→		query = buildIntCondition("job.num_nodes", filter.NumNodes, query)
   217→	}
   218→	if filter.NumAccelerators != nil {
   219→		query = buildIntCondition("job.num_acc", filter.NumAccelerators, query)
   220→	}
   221→	if filter.NumHWThreads != nil {
   222→		query = buildIntCondition("job.num_hwthreads", filter.NumHWThreads, query)
   223→	}
   224→	if filter.ArrayJobID != nil {
   225→		query = query.Where("job.array_job_id = ?", *filter.ArrayJobID)
   226→	}
   227→	if filter.StartTime != nil {
   228→		query = buildTimeCondition("job.start_time", filter.StartTime, query)
   229→	}
   230→	if filter.Duration != nil {
   231→		query = buildIntCondition("job.duration", filter.Duration, query)
   232→	}
   233→	if filter.Energy != nil {
   234→		query = buildFloatCondition("job.energy", filter.Energy, query)
   235→	}
   236→	// Indices on Tag Table
   237→	if filter.Tags != nil {
   238→		// This is an OR-Logic query: Returns all distinct jobs with at least one of the requested tags; TODO: AND-Logic query?
   239→		query = query.Join("jobtag ON jobtag.job_id = job.id").Where(sq.Eq{"jobtag.tag_id": filter.Tags}).Distinct()
   240→	}
   241→	// No explicit Indices
   242→	if filter.JobID != nil {
   243→		query = buildStringCondition("job.job_id", filter.JobID, query)
   244→	}
   245→	// Queries Within JSONs
   246→	if filter.MetricStats != nil {
   247→		for _, ms := range filter.MetricStats {
   248→			query = buildFloatJSONCondition(ms.MetricName, ms.Range, query)
   249→		}
   250→	}
   251→	if filter.Node != nil {
   252→		query = buildResourceJSONCondition("hostname", filter.Node, query)
   253→	}
   254→	if filter.JobName != nil {
   255→		query = buildMetaJSONCondition("jobName", filter.JobName, query)
   256→	}
   257→	if filter.Schedule != nil {
   258→		interactiveJobname := "interactive"
   259→		switch *filter.Schedule {
   260→		case "interactive":
   261→			iFilter := model.StringInput{Eq: &interactiveJobname}
   262→			query = buildMetaJSONCondition("jobName", &iFilter, query)
   263→		case "batch":
   264→			sFilter := model.StringInput{Neq: &interactiveJobname}
   265→			query = buildMetaJSONCondition("jobName", &sFilter, query)
   266→		}
   267→	}
   268→
   269→	// Configurable Filter to exclude recently started jobs, see config.go: ShortRunningJobsDuration
   270→	if filter.MinRunningFor != nil {
   271→		now := time.Now().Unix()
   272→		// Only jobs whose start timestamp is more than MinRunningFor seconds in the past
   273→		// If a job completed within the configured timeframe, it will still show up after the start_time matches the condition!
   274→		query = query.Where(sq.Lt{"job.start_time": (now - int64(*filter.MinRunningFor))})
   275→	}
   276→	return query
   277→}
   278→
   279→// buildIntCondition creates clauses for integer range filters, using BETWEEN only if required.
   280→func buildIntCondition(field string, cond *config.IntRange, query sq.SelectBuilder) sq.SelectBuilder {
   281→	if cond.From != 1 && cond.To != 0 {
   282→		return query.Where(field+" BETWEEN ? AND ?", cond.From, cond.To)
   283→	} else if cond.From != 1 && cond.To == 0 {
   284→		return query.Where(field+" >= ?", cond.From)
   285→	} else if cond.From == 1 && cond.To != 0 {
   286→		return query.Where(field+" <= ?", cond.To)
   287→	} else {
   288→		return query
   289→	}
   290→}
   291→
   292→// buildFloatCondition creates a clauses for float range filters, using BETWEEN only if required.
   293→func buildFloatCondition(field string, cond *model.FloatRange, query sq.SelectBuilder) sq.SelectBuilder {
   294→	if cond.From != 1.0 && cond.To != 0.0 {
   295→		return query.Where(field+" BETWEEN ? AND ?", cond.From, cond.To)
   296→	} else if cond.From != 1.0 && cond.To == 0.0 {
   297→		return query.Where(field+" >= ?", cond.From)
   298→	} else if cond.From == 1.0 && cond.To != 0.0 {
   299→		return query.Where(field+" <= ?", cond.To)
   300→	} else {
   301→		return query
   302→	}
   303→}
   304→
   305→// buildTimeCondition creates time range filters supporting absolute timestamps,
   306→// relative time ranges (last6h, last24h, last7d, last30d), or open-ended ranges.
   307→// Reminder: BETWEEN Queries are slower and dont use indices as frequently: Only use if both conditions required
   308→func buildTimeCondition(field string, cond *config.TimeRange, query sq.SelectBuilder) sq.SelectBuilder {
   309→	if cond.From != nil && cond.To != nil {
   310→		return query.Where(field+" BETWEEN ? AND ?", cond.From.Unix(), cond.To.Unix())
   311→	} else if cond.From != nil {
   312→		return query.Where(field+" >= ?", cond.From.Unix())
   313→	} else if cond.To != nil {
   314→		return query.Where(field+" <= ?", cond.To.Unix())
   315→	} else if cond.Range != "" {
   316→		now := time.Now().Unix()
   317→		var then int64
   318→		switch cond.Range {
   319→		case "last6h":
   320→			then = now - (60 * 60 * 6)
   321→		case "last24h":
   322→			then = now - (60 * 60 * 24)
   323→		case "last7d":
   324→			then = now - (60 * 60 * 24 * 7)
   325→		case "last30d":
   326→			then = now - (60 * 60 * 24 * 30)
   327→		default:
   328→			cclog.Debugf("No known named timeRange: startTime.range = %s", cond.Range)
   329→			return query
   330→		}
   331→		return query.Where(field+" >= ?", then)
   332→	} else {
   333→		return query
   334→	}
   335→}
   336→
   337→// buildFloatJSONCondition creates a filter on a numeric field within the footprint JSON column, using BETWEEN only if required.
   338→func buildFloatJSONCondition(jsonField string, cond *model.FloatRange, query sq.SelectBuilder) sq.SelectBuilder {
   339→	query = query.Where("JSON_VALID(footprint)")
   340→	if cond.From != 1.0 && cond.To != 0.0 {
   341→		return query.Where("JSON_EXTRACT(footprint, \"$."+jsonField+"\") BETWEEN ? AND ?", cond.From, cond.To)
   342→	} else if cond.From != 1.0 && cond.To == 0.0 {
   343→		return query.Where("JSON_EXTRACT(footprint, \"$."+jsonField+"\") >= ?", cond.From)
   344→	} else if cond.From == 1.0 && cond.To != 0.0 {
   345→		return query.Where("JSON_EXTRACT(footprint, \"$."+jsonField+"\") <= ?", cond.To)
   346→	} else {
   347→		return query
   348→	}
   349→}
   350→
   351→// buildStringCondition creates filters for string fields supporting equality,
   352→// inequality, prefix, suffix, substring, and IN list matching.
   353→func buildStringCondition(field string, cond *model.StringInput, query sq.SelectBuilder) sq.SelectBuilder {
   354→	if cond.Eq != nil {
   355→		return query.Where(field+" = ?", *cond.Eq)
   356→	}
   357→	if cond.Neq != nil {
   358→		return query.Where(field+" != ?", *cond.Neq)
   359→	}
   360→	if cond.StartsWith != nil {
   361→		return query.Where(field+" LIKE ?", fmt.Sprint(*cond.StartsWith, "%"))
   362→	}
   363→	if cond.EndsWith != nil {
   364→		return query.Where(field+" LIKE ?", fmt.Sprint("%", *cond.EndsWith))
   365→	}
   366→	if cond.Contains != nil {
   367→		return query.Where(field+" LIKE ?", fmt.Sprint("%", *cond.Contains, "%"))
   368→	}
   369→	if cond.In != nil {
   370→		queryElements := make([]string, len(cond.In))
   371→		copy(queryElements, cond.In)
   372→		return query.Where(sq.Or{sq.Eq{field: queryElements}})
   373→	}
   374→	return query
   375→}
   376→
   377→// buildMetaJSONCondition creates filters on fields within the meta_data JSON column.
   378→func buildMetaJSONCondition(jsonField string, cond *model.StringInput, query sq.SelectBuilder) sq.SelectBuilder {
   379→	query = query.Where("JSON_VALID(meta_data)")
   380→	if cond.Eq != nil {
   381→		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") = ?", *cond.Eq)
   382→	}
   383→	if cond.Neq != nil {
   384→		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") != ?", *cond.Neq)
   385→	}
   386→	if cond.StartsWith != nil {
   387→		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") LIKE ?", fmt.Sprint(*cond.StartsWith, "%"))
   388→	}
   389→	if cond.EndsWith != nil {
   390→		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") LIKE ?", fmt.Sprint("%", *cond.EndsWith))
   391→	}
   392→	if cond.Contains != nil {
   393→		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") LIKE ?", fmt.Sprint("%", *cond.Contains, "%"))
   394→	}
   395→	return query
   396→}
   397→
   398→// buildResourceJSONCondition creates filters on fields within the resources JSON array column.
   399→// Uses json_each to search within array elements.
   400→func buildResourceJSONCondition(jsonField string, cond *model.StringInput, query sq.SelectBuilder) sq.SelectBuilder {
   401→	query = query.Where("JSON_VALID(resources)")
   402→	if cond.Eq != nil {
   403→		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\") = ?)", *cond.Eq)
   404→	}
   405→	if cond.Neq != nil { // Currently Unused
   406→		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\") != ?)", *cond.Neq)
   407→	}
   408→	if cond.StartsWith != nil { // Currently Unused
   409→		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\")) LIKE ?)", fmt.Sprint(*cond.StartsWith, "%"))
   410→	}
   411→	if cond.EndsWith != nil { // Currently Unused
   412→		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\") LIKE ?)", fmt.Sprint("%", *cond.EndsWith))
   413→	}
   414→	if cond.Contains != nil {
   415→		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\") LIKE ?)", fmt.Sprint("%", *cond.Contains, "%"))
   416→	}
   417→	return query
   418→}
   419→
   420→var (
   421→	matchFirstCap = regexp.MustCompile("(.)([A-Z][a-z]+)")
   422→	matchAllCap   = regexp.MustCompile("([a-z0-9])([A-Z])")
   423→)
   424→
   425→// toSnakeCase converts camelCase strings to snake_case for SQL column names.
   426→// Includes security checks to prevent SQL injection attempts.
   427→// Panics if potentially dangerous characters are detected.
   428→func toSnakeCase(str string) string {
   429→	for _, c := range str {
   430→		if c == '\'' || c == '\\' || c == '"' || c == ';' || c == '-' || c == ' ' {
   431→			cclog.Panicf("toSnakeCase: potentially dangerous character detected in input: %q", str)
   432→		}
   433→	}
   434→
   435→	snake := matchFirstCap.ReplaceAllString(str, "${1}_${2}")
   436→	snake = matchAllCap.ReplaceAllString(snake, "${1}_${2}")
   437→	return strings.ToLower(snake)
   438→}
   439→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package repository provides job query functionality with filtering, pagination,
// and security controls. This file contains the main query builders and security
// checks for job retrieval operations.
package repository

import (
	"context"
	"errors"
	"fmt"
	"regexp"
	"strings"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/config"
	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	sq "github.com/Masterminds/squirrel"
)

const (
	// Default initial capacity for job result slices
	defaultJobsCapacity = 50
)

// QueryJobs retrieves jobs from the database with optional filtering, pagination,
// and sorting. Security controls are automatically applied based on the user context.
//
// Parameters:
//   - ctx: Context containing user authentication information
//   - filters: Optional job filters (cluster, state, user, time ranges, etc.)
//   - page: Optional pagination parameters (page number and items per page)
//   - order: Optional sorting specification (column or footprint field)
//
// Returns a slice of jobs matching the criteria, or an error if the query fails.
// The function enforces role-based access control through SecurityCheck.
func (r *JobRepository) QueryJobs(
	ctx context.Context,
	filters []*model.JobFilter,
	page *model.PageRequest,
	order *model.OrderByInput,
) ([]*schema.Job, error) {
	query, qerr := SecurityCheck(ctx, sq.Select(jobColumns...).From("job"))
	if qerr != nil {
		return nil, qerr
	}

	if order != nil {
		field := toSnakeCase(order.Field)
		if order.Type == "col" {
			switch order.Order {
			case model.SortDirectionEnumAsc:
				query = query.OrderBy(fmt.Sprintf("job.%s ASC", field))
			case model.SortDirectionEnumDesc:
				query = query.OrderBy(fmt.Sprintf("job.%s DESC", field))
			default:
				return nil, errors.New("invalid sorting order for column")
			}
		} else {
			// Order by footprint JSON field values
			query = query.Where("JSON_VALID(footprint)")
			switch order.Order {
			case model.SortDirectionEnumAsc:
				query = query.OrderBy(fmt.Sprintf("JSON_EXTRACT(footprint, \"$.%s\") ASC", field))
			case model.SortDirectionEnumDesc:
				query = query.OrderBy(fmt.Sprintf("JSON_EXTRACT(footprint, \"$.%s\") DESC", field))
			default:
				return nil, errors.New("invalid sorting order for footprint")
			}
		}
	}

	if page != nil && page.ItemsPerPage != -1 {
		limit := uint64(page.ItemsPerPage)
		query = query.Offset((uint64(page.Page) - 1) * limit).Limit(limit)
	}

	for _, f := range filters {
		query = BuildWhereClause(f, query)
	}

	rows, err := query.RunWith(r.stmtCache).Query()
	if err != nil {
		queryString, queryVars, _ := query.ToSql()
		return nil, fmt.Errorf("query failed [%s] %v: %w", queryString, queryVars, err)
	}
	defer rows.Close()

	jobs := make([]*schema.Job, 0, defaultJobsCapacity)
	for rows.Next() {
		job, err := scanJob(rows)
		if err != nil {
			cclog.Warnf("Error scanning job row: %v", err)
			return nil, fmt.Errorf("failed to scan job row: %w", err)
		}
		jobs = append(jobs, job)
	}

	if err := rows.Err(); err != nil {
		return nil, fmt.Errorf("error iterating job rows: %w", err)
	}

	return jobs, nil
}

// CountJobs returns the total number of jobs matching the given filters.
// Security controls are automatically applied based on the user context.
// Uses DISTINCT count to handle tag filters correctly (jobs may appear multiple
// times when joined with the tag table).
func (r *JobRepository) CountJobs(
	ctx context.Context,
	filters []*model.JobFilter,
) (int, error) {
	query, qerr := SecurityCheck(ctx, sq.Select("count(DISTINCT job.id)").From("job"))
	if qerr != nil {
		return 0, qerr
	}

	for _, f := range filters {
		query = BuildWhereClause(f, query)
	}

	var count int
	if err := query.RunWith(r.DB).Scan(&count); err != nil {
		return 0, fmt.Errorf("failed to count jobs: %w", err)
	}

	return count, nil
}

// SecurityCheckWithUser applies role-based access control filters to a job query
// based on the provided user's roles and permissions.
//
// Access rules by role:
//   - API role (exclusive): Full access to all jobs
//   - Admin/Support roles: Full access to all jobs
//   - Manager role: Access to jobs in managed projects plus own jobs
//   - User role: Access only to own jobs
//
// Returns an error if the user is nil or has no recognized roles.
func SecurityCheckWithUser(user *schema.User, query sq.SelectBuilder) (sq.SelectBuilder, error) {
	if user == nil {
		var qnil sq.SelectBuilder
		return qnil, fmt.Errorf("user context is nil")
	}

	switch {
	case len(user.Roles) == 1 && user.HasRole(schema.RoleAPI):
		return query, nil
	case user.HasAnyRole([]schema.Role{schema.RoleAdmin, schema.RoleSupport}):
		return query, nil
	case user.HasRole(schema.RoleManager):
		if len(user.Projects) != 0 {
			return query.Where(sq.Or{sq.Eq{"job.project": user.Projects}, sq.Eq{"job.hpc_user": user.Username}}), nil
		}
		cclog.Debugf("Manager '%s' has no assigned projects, restricting to personal jobs", user.Username)
		return query.Where("job.hpc_user = ?", user.Username), nil
	case user.HasRole(schema.RoleUser):
		return query.Where("job.hpc_user = ?", user.Username), nil
	default:
		var qnil sq.SelectBuilder
		return qnil, fmt.Errorf("user has no or unknown roles")
	}
}

// SecurityCheck extracts the user from the context and applies role-based access
// control filters to the query. This is a convenience wrapper around SecurityCheckWithUser.
func SecurityCheck(ctx context.Context, query sq.SelectBuilder) (sq.SelectBuilder, error) {
	user := GetUserFromContext(ctx)
	return SecurityCheckWithUser(user, query)
}

// BuildWhereClause constructs SQL WHERE conditions from a JobFilter and applies
// them to the query. Supports filtering by job properties (cluster, state, user),
// time ranges, resource usage, tags, and JSON field searches in meta_data,
// footprint, and resources columns.
func BuildWhereClause(filter *model.JobFilter, query sq.SelectBuilder) sq.SelectBuilder {
	// Primary Key
	if filter.DbID != nil {
		dbIDs := make([]string, len(filter.DbID))
		copy(dbIDs, filter.DbID)
		query = query.Where(sq.Eq{"job.id": dbIDs})
	}
	// Explicit indices
	if filter.Cluster != nil {
		query = buildStringCondition("job.cluster", filter.Cluster, query)
	}
	if filter.SubCluster != nil {
		query = buildStringCondition("job.subcluster", filter.SubCluster, query)
	}
	if filter.Partition != nil {
		query = buildStringCondition("job.cluster_partition", filter.Partition, query)
	}
	if filter.State != nil {
		states := make([]string, len(filter.State))
		for i, val := range filter.State {
			states[i] = string(val)
		}
		query = query.Where(sq.Eq{"job.job_state": states})
	}
	if filter.Shared != nil {
		query = query.Where("job.shared = ?", *filter.Shared)
	}
	if filter.Project != nil {
		query = buildStringCondition("job.project", filter.Project, query)
	}
	if filter.User != nil {
		query = buildStringCondition("job.hpc_user", filter.User, query)
	}
	if filter.NumNodes != nil {
		query = buildIntCondition("job.num_nodes", filter.NumNodes, query)
	}
	if filter.NumAccelerators != nil {
		query = buildIntCondition("job.num_acc", filter.NumAccelerators, query)
	}
	if filter.NumHWThreads != nil {
		query = buildIntCondition("job.num_hwthreads", filter.NumHWThreads, query)
	}
	if filter.ArrayJobID != nil {
		query = query.Where("job.array_job_id = ?", *filter.ArrayJobID)
	}
	if filter.StartTime != nil {
		query = buildTimeCondition("job.start_time", filter.StartTime, query)
	}
	if filter.Duration != nil {
		query = buildIntCondition("job.duration", filter.Duration, query)
	}
	if filter.Energy != nil {
		query = buildFloatCondition("job.energy", filter.Energy, query)
	}
	// Indices on Tag Table
	if filter.Tags != nil {
		// This is an OR-Logic query: Returns all distinct jobs with at least one of the requested tags; TODO: AND-Logic query?
		query = query.Join("jobtag ON jobtag.job_id = job.id").Where(sq.Eq{"jobtag.tag_id": filter.Tags}).Distinct()
	}
	// No explicit Indices
	if filter.JobID != nil {
		query = buildStringCondition("job.job_id", filter.JobID, query)
	}
	// Queries Within JSONs
	if filter.MetricStats != nil {
		for _, ms := range filter.MetricStats {
			query = buildFloatJSONCondition(ms.MetricName, ms.Range, query)
		}
	}
	if filter.Node != nil {
		query = buildResourceJSONCondition("hostname", filter.Node, query)
	}
	if filter.JobName != nil {
		query = buildMetaJSONCondition("jobName", filter.JobName, query)
	}
	if filter.Schedule != nil {
		interactiveJobname := "interactive"
		switch *filter.Schedule {
		case "interactive":
			iFilter := model.StringInput{Eq: &interactiveJobname}
			query = buildMetaJSONCondition("jobName", &iFilter, query)
		case "batch":
			sFilter := model.StringInput{Neq: &interactiveJobname}
			query = buildMetaJSONCondition("jobName", &sFilter, query)
		}
	}

	// Configurable Filter to exclude recently started jobs, see config.go: ShortRunningJobsDuration
	if filter.MinRunningFor != nil {
		now := time.Now().Unix()
		// Only jobs whose start timestamp is more than MinRunningFor seconds in the past
		// If a job completed within the configured timeframe, it will still show up after the start_time matches the condition!
		query = query.Where(sq.Lt{"job.start_time": (now - int64(*filter.MinRunningFor))})
	}
	return query
}

// buildIntCondition creates clauses for integer range filters, using BETWEEN only if required.
func buildIntCondition(field string, cond *config.IntRange, query sq.SelectBuilder) sq.SelectBuilder {
	if cond.From != 1 && cond.To != 0 {
		return query.Where(field+" BETWEEN ? AND ?", cond.From, cond.To)
	} else if cond.From != 1 && cond.To == 0 {
		return query.Where(field+" >= ?", cond.From)
	} else if cond.From == 1 && cond.To != 0 {
		return query.Where(field+" <= ?", cond.To)
	} else {
		return query
	}
}

// buildFloatCondition creates a clauses for float range filters, using BETWEEN only if required.
func buildFloatCondition(field string, cond *model.FloatRange, query sq.SelectBuilder) sq.SelectBuilder {
	if cond.From != 1.0 && cond.To != 0.0 {
		return query.Where(field+" BETWEEN ? AND ?", cond.From, cond.To)
	} else if cond.From != 1.0 && cond.To == 0.0 {
		return query.Where(field+" >= ?", cond.From)
	} else if cond.From == 1.0 && cond.To != 0.0 {
		return query.Where(field+" <= ?", cond.To)
	} else {
		return query
	}
}

// buildTimeCondition creates time range filters supporting absolute timestamps,
// relative time ranges (last6h, last24h, last7d, last30d), or open-ended ranges.
// Reminder: BETWEEN Queries are slower and dont use indices as frequently: Only use if both conditions required
func buildTimeCondition(field string, cond *config.TimeRange, query sq.SelectBuilder) sq.SelectBuilder {
	if cond.From != nil && cond.To != nil {
		return query.Where(field+" BETWEEN ? AND ?", cond.From.Unix(), cond.To.Unix())
	} else if cond.From != nil {
		return query.Where(field+" >= ?", cond.From.Unix())
	} else if cond.To != nil {
		return query.Where(field+" <= ?", cond.To.Unix())
	} else if cond.Range != "" {
		now := time.Now().Unix()
		var then int64
		switch cond.Range {
		case "last6h":
			then = now - (60 * 60 * 6)
		case "last24h":
			then = now - (60 * 60 * 24)
		case "last7d":
			then = now - (60 * 60 * 24 * 7)
		case "last30d":
			then = now - (60 * 60 * 24 * 30)
		default:
			cclog.Debugf("No known named timeRange: startTime.range = %s", cond.Range)
			return query
		}
		return query.Where(field+" >= ?", then)
	} else {
		return query
	}
}

// buildFloatJSONCondition creates a filter on a numeric field within the footprint JSON column, using BETWEEN only if required.
func buildFloatJSONCondition(jsonField string, cond *model.FloatRange, query sq.SelectBuilder) sq.SelectBuilder {
	query = query.Where("JSON_VALID(footprint)")
	if cond.From != 1.0 && cond.To != 0.0 {
		return query.Where("JSON_EXTRACT(footprint, \"$."+jsonField+"\") BETWEEN ? AND ?", cond.From, cond.To)
	} else if cond.From != 1.0 && cond.To == 0.0 {
		return query.Where("JSON_EXTRACT(footprint, \"$."+jsonField+"\") >= ?", cond.From)
	} else if cond.From == 1.0 && cond.To != 0.0 {
		return query.Where("JSON_EXTRACT(footprint, \"$."+jsonField+"\") <= ?", cond.To)
	} else {
		return query
	}
}

// buildStringCondition creates filters for string fields supporting equality,
// inequality, prefix, suffix, substring, and IN list matching.
func buildStringCondition(field string, cond *model.StringInput, query sq.SelectBuilder) sq.SelectBuilder {
	if cond.Eq != nil {
		return query.Where(field+" = ?", *cond.Eq)
	}
	if cond.Neq != nil {
		return query.Where(field+" != ?", *cond.Neq)
	}
	if cond.StartsWith != nil {
		return query.Where(field+" LIKE ?", fmt.Sprint(*cond.StartsWith, "%"))
	}
	if cond.EndsWith != nil {
		return query.Where(field+" LIKE ?", fmt.Sprint("%", *cond.EndsWith))
	}
	if cond.Contains != nil {
		return query.Where(field+" LIKE ?", fmt.Sprint("%", *cond.Contains, "%"))
	}
	if cond.In != nil {
		queryElements := make([]string, len(cond.In))
		copy(queryElements, cond.In)
		return query.Where(sq.Or{sq.Eq{field: queryElements}})
	}
	return query
}

// buildMetaJSONCondition creates filters on fields within the meta_data JSON column.
func buildMetaJSONCondition(jsonField string, cond *model.StringInput, query sq.SelectBuilder) sq.SelectBuilder {
	query = query.Where("JSON_VALID(meta_data)")
	if cond.Eq != nil {
		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") = ?", *cond.Eq)
	}
	if cond.Neq != nil {
		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") != ?", *cond.Neq)
	}
	if cond.StartsWith != nil {
		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") LIKE ?", fmt.Sprint(*cond.StartsWith, "%"))
	}
	if cond.EndsWith != nil {
		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") LIKE ?", fmt.Sprint("%", *cond.EndsWith))
	}
	if cond.Contains != nil {
		return query.Where("JSON_EXTRACT(meta_data, \"$."+jsonField+"\") LIKE ?", fmt.Sprint("%", *cond.Contains, "%"))
	}
	return query
}

// buildResourceJSONCondition creates filters on fields within the resources JSON array column.
// Uses json_each to search within array elements.
func buildResourceJSONCondition(jsonField string, cond *model.StringInput, query sq.SelectBuilder) sq.SelectBuilder {
	query = query.Where("JSON_VALID(resources)")
	if cond.Eq != nil {
		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\") = ?)", *cond.Eq)
	}
	if cond.Neq != nil { // Currently Unused
		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\") != ?)", *cond.Neq)
	}
	if cond.StartsWith != nil { // Currently Unused
		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\")) LIKE ?)", fmt.Sprint(*cond.StartsWith, "%"))
	}
	if cond.EndsWith != nil { // Currently Unused
		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\") LIKE ?)", fmt.Sprint("%", *cond.EndsWith))
	}
	if cond.Contains != nil {
		return query.Where("EXISTS (SELECT 1 FROM json_each(job.resources) WHERE json_extract(value, \"$."+jsonField+"\") LIKE ?)", fmt.Sprint("%", *cond.Contains, "%"))
	}
	return query
}

var (
	matchFirstCap = regexp.MustCompile("(.)([A-Z][a-z]+)")
	matchAllCap   = regexp.MustCompile("([a-z0-9])([A-Z])")
)

// toSnakeCase converts camelCase strings to snake_case for SQL column names.
// Includes security checks to prevent SQL injection attempts.
// Panics if potentially dangerous characters are detected.
func toSnakeCase(str string) string {
	for _, c := range str {
		if c == '\'' || c == '\\' || c == '"' || c == ';' || c == '-' || c == ' ' {
			cclog.Panicf("toSnakeCase: potentially dangerous character detected in input: %q", str)
		}
	}

	snake := matchFirstCap.ReplaceAllString(str, "${1}_${2}")
	snake = matchAllCap.ReplaceAllString(snake, "${1}_${2}")
	return strings.ToLower(snake)
}