Merge pull request #519 from ClusterCockpit/hotfix

Hotfix
2026-07-07 08:50:37 +02:00 · 2026-03-13 17:39:04 +01:00
parent d0ebba5b4a 045311eec0
commit 128c098865
68 changed files with 2210 additions and 777 deletions
@@ -0,0 +1,29 @@
+# Session Context
+
+Session ID: 28147033-ddc8-4056-b064-e0558fbc614e
+Commit Message: There are memory leaks in production on queries that trigger a full scan
+
+## Prompts
+
+### Prompt 1
+
+There are memory leaks in production on queries that trigger a full scan. Review all queries for missing close and other bugs that could cause memory leaks in the sqlite-3 driver. Example for a query that triggers a massive allocation is a specific jobstate list with sorting on start_time on all jobs.
+
+## Summary
+
+All 31 `.Query()` calls now have proper `rows.Close()`. Build passes, tests pass, vet clean.
+
+### Summary of changes
+
+**`internal/repository/stats.go`** (7 fixes + 1 rewrite):
+- Added `defer rows.Close()` + `rows.Err()` to `JobsStatsGrouped`, `JobCountGrouped`, `AddJobCountGrouped`, `jobsStatisticsHistogram`, `jobsDurationStatisticsHistogram`, `jobsMetricStatisticsHistogram`
+- Rewrote `AddJobCount` to use `QueryRow()` instead of `Query()` (single-row result)
+
+**`internal/repository/tags.go`** (5 fixes):
+- Added `defer xrows.Close()` + `xrows.Err()` for `Queryx` in `CountTags`
+- Added `defer rows.Close()` for second query in `CountTags`
+- Added `defer rows.Close()` + `rows.Err()` to `GetTags`, `GetTagsDirect`, `getArchiveTags`
+
+**`internal/repository/node.go`** (4 fixes + 1 rewrite):
+- Added `defer rows.Close()` + `rows.Err()`, removed manual `rows.Close()` in `QueryNodes`, `QueryNodesWithMeta`, `CountStatesTimed`
+- Rewrote `CountStates` to use `COUNT(*) ... GROUP BY` in SQL instead of fetching every row and counting in Go
@@ -0,0 +1 @@
+There are memory leaks in production on queries that trigger a full scan. Review all queries for missing close and other bugs that could cause memory leaks in the sqlite-3 driver. Example for a query that triggers a massive allocation is a specific jobstate list with sorting on start_time on all jobs.
@@ -0,0 +1,16 @@
+All 31 `.Query()` calls now have proper `rows.Close()`. Build passes, tests pass, vet clean.
+
+### Summary of changes
+
+**`internal/repository/stats.go`** (7 fixes + 1 rewrite):
+- Added `defer rows.Close()` + `rows.Err()` to `JobsStatsGrouped`, `JobCountGrouped`, `AddJobCountGrouped`, `jobsStatisticsHistogram`, `jobsDurationStatisticsHistogram`, `jobsMetricStatisticsHistogram`
+- Rewrote `AddJobCount` to use `QueryRow()` instead of `Query()` (single-row result)
+
+**`internal/repository/tags.go`** (5 fixes):
+- Added `defer xrows.Close()` + `xrows.Err()` for `Queryx` in `CountTags`
+- Added `defer rows.Close()` for second query in `CountTags`
+- Added `defer rows.Close()` + `rows.Err()` to `GetTags`, `GetTagsDirect`, `getArchiveTags`
+
+**`internal/repository/node.go`** (4 fixes + 1 rewrite):
+- Added `defer rows.Close()` + `rows.Err()`, removed manual `rows.Close()` in `QueryNodes`, `QueryNodesWithMeta`, `CountStatesTimed`
+- Rewrote `CountStates` to use `COUNT(*) ... GROUP BY` in SQL instead of fetching every row and counting in Go
@@ -0,0 +1,40 @@
+# Session Context
+
+Session ID: 42401d2e-7d1c-4c0e-abe6-356cb2d48747
+Commit Message: There also have to be bugs in jobQuery.go . Especially the following que
+
+## Prompts
+
+### Prompt 1
+
+There also have to be bugs in jobQuery.go . Especially the following query triggers the memory leak: SELECT * FROM job WHERE job.job_state IN ("completed", "running", "failed") ORDER BY job.start_time DESC LIMIT 1 OFFSET 10; Dig deeper to find the cause. Also investigate why no existing index is used for this query.
+
+## Summary
+
+All passes. Here's a summary of findings and fixes:
+
+---
+
+## Findings
+
+### Bug 1: Missing `defer rows.Close()` in `user.go:GetLdapUsernames()` (line 100)
+**Fixed.** Every call leaked a database connection and its buffers.
+
+### Bug 2: SQLite never runs `ANALYZE` — wrong index selection
+**Fixed.** Added `ANALYZE` to `setupSqlite()` in `dbConnection.go`.
+
+Without statistics in `sqlite_stat1`, SQLite's query planner **guessed** and chose `jobs_jobstate_energy` (a terrible index for this query) instead of scanning `jobs_starttime` in reverse. The result:
+
+| | Before ANALYZE | After ANALYZE |
+|---|---|---|
+| Index used | `jobs_jobstate_energy` | `jobs_starttime` |
+| Sort | **TEMP B-TREE** (materializes ALL rows) | None (index order) |
+| Memory | Proportional to total matching rows | Constant |
+| I/O | Full scan of all matching rows | Stops at OFFSET+LIMIT |
+
+### Bug 3: `IN` clause + `ORDER BY` is fundamentally incompatible with composite indexes
+
+Even with the "correct" index `(job_state, start_time)`, SQLite **cannot** merge-sort across 3 separate index range scans for `IN ('completed','running','failed')`. It always falls back to a temp B-tree sort. The only efficient plan is to use the standalone `jobs_starttime` index — which SQLite does automatically **after ANALYZE** because it realizes the 3 states cover virtually all rows, making the WHERE clause nearly a no-op.
+
+### Observation: 79 indexes on the `job` table
+This is excessive and actively harmful — it confuses the query planner (especially without ANALYZE) and slows writes. The `jobs_jobstate_starttime` index from migration 08 is also missing from the actual DB (only the 3-column `jobs_jobstate_starttime_duration` exists). This is worth investigating separately but is a schema/migration concern, not a code bug.
@@ -0,0 +1 @@
+There also have to be bugs in jobQuery.go . Especially the following query triggers the memory leak: SELECT * FROM job WHERE job.job_state IN ("completed", "running", "failed") ORDER BY job.start_time DESC LIMIT 1 OFFSET 10; Dig deeper to find the cause. Also investigate why no existing index is used for this query.
@@ -0,0 +1,27 @@
+All passes. Here's a summary of findings and fixes:
+
+---
+
+## Findings
+
+### Bug 1: Missing `defer rows.Close()` in `user.go:GetLdapUsernames()` (line 100)
+**Fixed.** Every call leaked a database connection and its buffers.
+
+### Bug 2: SQLite never runs `ANALYZE` — wrong index selection
+**Fixed.** Added `ANALYZE` to `setupSqlite()` in `dbConnection.go`.
+
+Without statistics in `sqlite_stat1`, SQLite's query planner **guessed** and chose `jobs_jobstate_energy` (a terrible index for this query) instead of scanning `jobs_starttime` in reverse. The result:
+
+| | Before ANALYZE | After ANALYZE |
+|---|---|---|
+| Index used | `jobs_jobstate_energy` | `jobs_starttime` |
+| Sort | **TEMP B-TREE** (materializes ALL rows) | None (index order) |
+| Memory | Proportional to total matching rows | Constant |
+| I/O | Full scan of all matching rows | Stops at OFFSET+LIMIT |
+
+### Bug 3: `IN` clause + `ORDER BY` is fundamentally incompatible with composite indexes
+
+Even with the "correct" index `(job_state, start_time)`, SQLite **cannot** merge-sort across 3 separate index range scans for `IN ('completed','running','failed')`. It always falls back to a temp B-tree sort. The only efficient plan is to use the standalone `jobs_starttime` index — which SQLite does automatically **after ANALYZE** because it realizes the 3 states cover virtually all rows, making the WHERE clause nearly a no-op.
+
+### Observation: 79 indexes on the `job` table
+This is excessive and actively harmful — it confuses the query planner (especially without ANALYZE) and slows writes. The `jobs_jobstate_starttime` index from migration 08 is also missing from the actual DB (only the 3-column `jobs_jobstate_starttime_duration` exists). This is worth investigating separately but is a schema/migration concern, not a code bug.
@@ -101,7 +101,7 @@ changelog:
 release:
  draft: false
  footer: |
-    Supports job archive version 3 and database version 10.
+    Supports job archive version 3 and database version 11.
    Please check out the [Release Notes](https://github.com/ClusterCockpit/cc-backend/blob/master/ReleaseNotes.md) for further details on breaking changes.

 # vim: set ts=2 sw=2 tw=0 fo=cnqoj
@@ -1,6 +1,6 @@
 TARGET = ./cc-backend
 FRONTEND = ./web/frontend
-VERSION = 1.5.0
+VERSION = 1.5.1
 GIT_HASH := $(shell git rev-parse --short HEAD || echo 'development')
 CURRENT_TIME = $(shell date +"%Y-%m-%d:T%H:%M:%S")
 LD_FLAGS = '-s -X main.date=${CURRENT_TIME} -X main.version=${VERSION} -X main.commit=${GIT_HASH}'
@@ -133,6 +133,92 @@ ln -s <your-existing-job-archive> ./var/job-archive
 ./cc-backend -help
 ```

+## Database Configuration
+
+cc-backend uses SQLite as its database. For large installations, SQLite memory
+usage can be tuned via the optional `db-config` section in config.json under
+`main`:
+
+```json
+{
+  "main": {
+    "db": "./var/job.db",
+    "db-config": {
+      "cache-size-mb": 2048,
+      "soft-heap-limit-mb": 16384,
+      "max-open-connections": 4,
+      "max-idle-connections": 4,
+      "max-idle-time-minutes": 10
+    }
+  }
+}
+```
+
+All fields are optional. If `db-config` is omitted entirely, built-in defaults
+are used.
+
+### Options
+
+| Option                  | Default | Description                                                                                                                                                                             |
+| ----------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `cache-size-mb`         | 2048    | SQLite page cache size per connection in MB. Maps to `PRAGMA cache_size`. Total cache memory is up to `cache-size-mb × max-open-connections`.                                           |
+| `soft-heap-limit-mb`    | 16384   | Process-wide SQLite soft heap limit in MB. SQLite will try to release cache pages to stay under this limit. Queries won't fail if exceeded, but cache eviction becomes more aggressive. |
+| `max-open-connections`  | 4       | Maximum number of open database connections.                                                                                                                                            |
+| `max-idle-connections`  | 4       | Maximum number of idle database connections kept in the pool.                                                                                                                           |
+| `max-idle-time-minutes` | 10      | Maximum time in minutes a connection can sit idle before being closed.                                                                                                                  |
+
+### Sizing Guidelines
+
+SQLite's `cache_size` is a **per-connection** setting — each connection
+maintains its own independent page cache. With multiple connections, the total
+memory available for caching is the sum across all connections.
+
+In practice, different connections tend to cache **different pages** (e.g., one
+handles a job listing query while another runs a statistics aggregation), so
+their caches naturally spread across the database. The formula
+`DB_size / max-open-connections` gives enough per-connection cache that the
+combined caches can cover the entire database.
+
+However, this is a best-case estimate. Connections running similar queries will
+cache the same pages redundantly. In the worst case (all connections caching
+identical pages), only `cache-size-mb` worth of unique data is cached rather
+than `cache-size-mb × max-open-connections`. For workloads with diverse
+concurrent queries, cache overlap is typically low.
+
+**Rules of thumb:**
+
+- **cache-size-mb**: Set to `DB_size_in_MB / max-open-connections` to allow the
+  entire database to be cached in memory. For example, an 80GB database with 8
+  connections needs at least 10240 MB (10GB) per connection. If your workload
+  has many similar concurrent queries, consider setting it higher to account for
+  cache overlap between connections.
+
+- **soft-heap-limit-mb**: Should be >= `cache-size-mb × max-open-connections` to
+  avoid cache thrashing. This is the total SQLite memory budget for the process.
+- On small installations the defaults work well. On servers with large databases
+  (tens of GB) and plenty of RAM, increasing these values significantly improves
+  query performance by reducing disk I/O.
+
+### Example: Large Server (512GB RAM, 80GB database)
+
+```json
+{
+  "main": {
+    "db-config": {
+      "cache-size-mb": 16384,
+      "soft-heap-limit-mb": 131072,
+      "max-open-connections": 8,
+      "max-idle-time-minutes": 30
+    }
+  }
+}
+```
+
+This allows the entire 80GB database to be cached (8 × 16GB = 128GB page cache)
+with a 128GB soft heap limit, using about 25% of available RAM.
+
+The effective configuration is logged at startup for verification.
+
 ## Project file structure

 - [`.github/`](https://github.com/ClusterCockpit/cc-backend/tree/master/.github)
@@ -1,10 +1,51 @@
-# `cc-backend` version 1.5.0
+# `cc-backend` version 1.5.1

-Supports job archive version 3 and database version 10.
+Supports job archive version 3 and database version 11.

-This is a feature release of `cc-backend`, the API backend and frontend
+This is a bugfix release of `cc-backend`, the API backend and frontend
 implementation of ClusterCockpit.
 For release specific notes visit the [ClusterCockpit Documentation](https://clusterockpit.org/docs/release/).
+If you are upgrading from v1.5.0 you need to do another DB migration. This
+should not take long. For optimal database performance after the migration it is
+recommended to apply the new `optimize-db` flag, which runs the sqlite `ANALYZE`
+and `VACUUM` commands. Depending on your database size (more then 40GB) the
+`VACUUM` may take up to 2h.
+
+## Changes in 1.5.1
+
+### Database
+
+- **New migration (version 11)**: Optimized database index count and added covering indexes for stats queries for significantly improved query performance
+- **Migration 9 fix**: Removed redundant indices from migration 9 that are superseded by migration 11
+- **Optional DB optimization flag**: Added `-optimize-db` CLI flag to run `ANALYZE` on demand; removed automatic ANALYZE on startup
+- **Selective stats queries**: Stats queries are now selective, reducing unnecessary computation
+- **User list paging**: Added paging support to the user list for better scalability
+- **SQLite configuration hardening**: Sanitized SQLite configuration with new configurable options; fixes large heap allocations in the SQLite driver
+- **Query cancellation**: Long-running database queries can now be cancelled
+- **Resource leak fix**: Added missing `defer Close()` calls for all query result sets
+
+### Bug fixes
+
+- **Segfault when taggers misconfigured**: Fixed crash when `enable-job-taggers` is set but tagger rule directories are missing
+- **GroupBy stats query complexity**: Reduced complexity for `groupBy` statistics queries
+- **Ranged filter conditions**: Fixed GT and LT conditions in ranged filters
+- **Energy filter preset**: Reduced energy filter preset to a more practical default
+- **JSON validity check**: Fixed wrong field being checked for JSON validity
+- **Tagger float rounding**: Fixed rounding of floats in tagger messages
+- **Node view null safety**: Added null-safe checks in node view to prevent runtime errors
+- **Public dashboard null safety**: Added null-safe checks in the public dashboard to prevent runtime errors
+
+### Frontend
+
+- **Bumped patch versions**: Updated frontend dependencies to latest patch versions
+
+### Documentation
+
+- **New DB config options**: Added new database configuration options to README
+
+---
+
+_The sections below document all features and changes introduced in the 1.5.0 major release, which 1.5.1 is based on._

 ## Breaking changes

@@ -34,7 +75,7 @@ For release specific notes visit the [ClusterCockpit Documentation](https://clus

 ### Dependency changes

- **cc-lib v2.5.1**: Switched to cc-lib version 2 with updated APIs (currently at v2.5.1)
+- **cc-lib v2.8.0**: Switched to cc-lib version 2 with updated APIs
 - **cclib NATS client**: Now using the cclib NATS client implementation
 - Removed obsolete `util.Float` usage from cclib

@@ -11,7 +11,7 @@ import "flag"

 var (
 	flagReinitDB, flagInit, flagServer, flagSyncLDAP, flagGops, flagMigrateDB, flagRevertDB,
-	flagForceDB, flagDev, flagVersion, flagLogDateTime, flagApplyTags bool
+	flagForceDB, flagDev, flagVersion, flagLogDateTime, flagApplyTags, flagOptimizeDB bool
 	flagNewUser, flagDelUser, flagGenJWT, flagConfigFile, flagImportJob, flagLogLevel string
 )

@@ -27,6 +27,7 @@ func cliInit() {
 	flag.BoolVar(&flagRevertDB, "revert-db", false, "Migrate database to previous version and exit")
 	flag.BoolVar(&flagApplyTags, "apply-tags", false, "Run taggers on all completed jobs and exit")
 	flag.BoolVar(&flagForceDB, "force-db", false, "Force database version, clear dirty flag and exit")
+	flag.BoolVar(&flagOptimizeDB, "optimize-db", false, "Optimize database: run VACUUM to reclaim space, then ANALYZE to update query planner statistics")
 	flag.BoolVar(&flagLogDateTime, "logdate", false, "Set this flag to add date and time to log messages")
 	flag.StringVar(&flagConfigFile, "config", "./config.json", "Specify alternative path to `config.json`")
 	flag.StringVar(&flagNewUser, "add-user", "", "Add a new user. Argument format: <username>:[admin,support,manager,api,user]:<password>")
@@ -108,6 +108,26 @@ func initConfiguration() error {
 }

 func initDatabase() error {
+	if config.Keys.DbConfig != nil {
+		cfg := repository.DefaultConfig()
+		dc := config.Keys.DbConfig
+		if dc.CacheSizeMB > 0 {
+			cfg.DbCacheSizeMB = dc.CacheSizeMB
+		}
+		if dc.SoftHeapLimitMB > 0 {
+			cfg.DbSoftHeapLimitMB = dc.SoftHeapLimitMB
+		}
+		if dc.MaxOpenConnections > 0 {
+			cfg.MaxOpenConnections = dc.MaxOpenConnections
+		}
+		if dc.MaxIdleConnections > 0 {
+			cfg.MaxIdleConnections = dc.MaxIdleConnections
+		}
+		if dc.ConnectionMaxIdleTimeMins > 0 {
+			cfg.ConnectionMaxIdleTime = time.Duration(dc.ConnectionMaxIdleTimeMins) * time.Minute
+		}
+		repository.SetConfig(cfg)
+	}
 	repository.Connect(config.Keys.DB)
 	return nil
 }
@@ -489,6 +509,20 @@ func run() error {
 		return err
 	}

+	// Optimize database if requested
+	if flagOptimizeDB {
+		db := repository.GetConnection()
+		cclog.Print("Running VACUUM to reclaim space and defragment database...")
+		if _, err := db.DB.Exec("VACUUM"); err != nil {
+			return fmt.Errorf("VACUUM failed: %w", err)
+		}
+		cclog.Print("Running ANALYZE to update query planner statistics...")
+		if _, err := db.DB.Exec("ANALYZE"); err != nil {
+			return fmt.Errorf("ANALYZE failed: %w", err)
+		}
+		cclog.Exitf("OptimizeDB Success: Database '%s' optimized (VACUUM + ANALYZE).\n", config.Keys.DB)
+	}
+
 	// Handle user commands (add, delete, sync, JWT)
 	if err := handleUserCommands(); err != nil {
 		return err
@@ -20,6 +20,7 @@ import (
 	"strings"
 	"time"

+	"github.com/99designs/gqlgen/graphql"
 	"github.com/99designs/gqlgen/graphql/handler"
 	"github.com/99designs/gqlgen/graphql/handler/transport"
 	"github.com/99designs/gqlgen/graphql/playground"
@@ -89,6 +90,12 @@ func (s *Server) init() error {

 	graphQLServer.AddTransport(transport.POST{})

+	// Inject a per-request stats cache so that grouped statistics queries
+	// sharing the same (filter, groupBy) pair are executed only once.
+	graphQLServer.AroundOperations(func(ctx context.Context, next graphql.OperationHandler) graphql.ResponseHandler {
+		return next(graph.WithStatsGroupCache(ctx))
+	})
+
 	if os.Getenv(envDebug) != "1" {
 		// Having this handler means that a error message is returned via GraphQL instead of the connection simply beeing closed.
 		// The problem with this is that then, no more stacktrace is printed to stderr.
@@ -8,14 +8,14 @@ tool (
 )

 require (
-	github.com/99designs/gqlgen v0.17.87
-	github.com/ClusterCockpit/cc-lib/v2 v2.8.0
+	github.com/99designs/gqlgen v0.17.88
+	github.com/ClusterCockpit/cc-lib/v2 v2.8.2
 	github.com/ClusterCockpit/cc-line-protocol/v2 v2.4.0
 	github.com/Masterminds/squirrel v1.5.4
-	github.com/aws/aws-sdk-go-v2 v1.41.2
-	github.com/aws/aws-sdk-go-v2/config v1.32.10
-	github.com/aws/aws-sdk-go-v2/credentials v1.19.10
-	github.com/aws/aws-sdk-go-v2/service/s3 v1.96.2
+	github.com/aws/aws-sdk-go-v2 v1.41.3
+	github.com/aws/aws-sdk-go-v2/config v1.32.11
+	github.com/aws/aws-sdk-go-v2/credentials v1.19.11
+	github.com/aws/aws-sdk-go-v2/service/s3 v1.97.0
 	github.com/coreos/go-oidc/v3 v3.17.0
 	github.com/expr-lang/expr v1.17.8
 	github.com/go-chi/chi/v5 v5.2.5
@@ -29,16 +29,17 @@ require (
 	github.com/jmoiron/sqlx v1.4.0
 	github.com/joho/godotenv v1.5.1
 	github.com/mattn/go-sqlite3 v1.14.34
-	github.com/parquet-go/parquet-go v0.28.0
+	github.com/parquet-go/parquet-go v0.29.0
 	github.com/qustavo/sqlhooks/v2 v2.1.0
 	github.com/santhosh-tekuri/jsonschema/v5 v5.3.1
 	github.com/stretchr/testify v1.11.1
 	github.com/swaggo/http-swagger v1.3.4
 	github.com/swaggo/swag v1.16.6
 	github.com/vektah/gqlparser/v2 v2.5.32
-	golang.org/x/crypto v0.48.0
-	golang.org/x/oauth2 v0.35.0
-	golang.org/x/time v0.14.0
+	golang.org/x/crypto v0.49.0
+	golang.org/x/oauth2 v0.36.0
+	golang.org/x/sync v0.20.0
+	golang.org/x/time v0.15.0
 )

 require (
@@ -47,20 +48,20 @@ require (
 	github.com/agnivade/levenshtein v1.2.1 // indirect
 	github.com/andybalholm/brotli v1.2.0 // indirect
 	github.com/apapsch/go-jsonmerge/v2 v2.0.0 // indirect
-	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.5 // indirect
-	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.18 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.18 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.18 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.18 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.5 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.10 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.18 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.18 // indirect
-	github.com/aws/aws-sdk-go-v2/service/signin v1.0.6 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sso v1.30.11 // indirect
-	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.15 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sts v1.41.7 // indirect
+	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.6 // indirect
+	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.19 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.19 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.19 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/ini v1.8.5 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.20 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.6 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.11 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.19 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.19 // indirect
+	github.com/aws/aws-sdk-go-v2/service/signin v1.0.7 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sso v1.30.12 // indirect
+	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.16 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sts v1.41.8 // indirect
 	github.com/aws/smithy-go v1.24.2 // indirect
 	github.com/cpuguy83/go-md2man/v2 v2.0.7 // indirect
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
@@ -96,7 +97,7 @@ require (
 	github.com/oapi-codegen/runtime v1.2.0 // indirect
 	github.com/parquet-go/bitpack v1.0.0 // indirect
 	github.com/parquet-go/jsonlite v1.4.0 // indirect
-	github.com/pierrec/lz4/v4 v4.1.25 // indirect
+	github.com/pierrec/lz4/v4 v4.1.26 // indirect
 	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
 	github.com/robfig/cron/v3 v3.0.1 // indirect
 	github.com/rogpeppe/go-internal v1.10.0 // indirect
@@ -107,16 +108,15 @@ require (
 	github.com/swaggo/files v1.0.1 // indirect
 	github.com/twpayne/go-geom v1.6.1 // indirect
 	github.com/urfave/cli/v2 v2.27.7 // indirect
-	github.com/urfave/cli/v3 v3.6.2 // indirect
+	github.com/urfave/cli/v3 v3.7.0 // indirect
 	github.com/xrash/smetrics v0.0.0-20250705151800-55b8f293f342 // indirect
 	go.yaml.in/yaml/v2 v2.4.3 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
-	golang.org/x/mod v0.33.0 // indirect
-	golang.org/x/net v0.51.0 // indirect
-	golang.org/x/sync v0.19.0 // indirect
-	golang.org/x/sys v0.41.0 // indirect
-	golang.org/x/text v0.34.0 // indirect
-	golang.org/x/tools v0.42.0 // indirect
+	golang.org/x/mod v0.34.0 // indirect
+	golang.org/x/net v0.52.0 // indirect
+	golang.org/x/sys v0.42.0 // indirect
+	golang.org/x/text v0.35.0 // indirect
+	golang.org/x/tools v0.43.0 // indirect
 	google.golang.org/protobuf v1.36.11 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 	sigs.k8s.io/yaml v1.6.0 // indirect
@@ -1,11 +1,11 @@
 filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA=
 filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
-github.com/99designs/gqlgen v0.17.87 h1:pSnCIMhBQezAE8bc1GNmfdLXFmnWtWl1GRDFEE/nHP8=
-github.com/99designs/gqlgen v0.17.87/go.mod h1:fK05f1RqSNfQpd4CfW5qk/810Tqi4/56Wf6Nem0khAg=
+github.com/99designs/gqlgen v0.17.88 h1:neMQDgehMwT1vYIOx/w5ZYPUU/iMNAJzRO44I5Intoc=
+github.com/99designs/gqlgen v0.17.88/go.mod h1:qeqYFEgOeSKqWedOjogPizimp2iu4E23bdPvl4jTYic=
 github.com/Azure/go-ntlmssp v0.1.0 h1:DjFo6YtWzNqNvQdrwEyr/e4nhU3vRiwenz5QX7sFz+A=
 github.com/Azure/go-ntlmssp v0.1.0/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk=
-github.com/ClusterCockpit/cc-lib/v2 v2.8.0 h1:ROduRzRuusi+6kLB991AAu3Pp2AHOasQJFJc7JU/n/E=
-github.com/ClusterCockpit/cc-lib/v2 v2.8.0/go.mod h1:FwD8vnTIbBM3ngeLNKmCvp9FoSjQZm7xnuaVxEKR23o=
+github.com/ClusterCockpit/cc-lib/v2 v2.8.2 h1:rCLZk8wz8yq8xBnBEdVKigvA2ngR8dPmHbEFwxxb3jw=
+github.com/ClusterCockpit/cc-lib/v2 v2.8.2/go.mod h1:FwD8vnTIbBM3ngeLNKmCvp9FoSjQZm7xnuaVxEKR23o=
 github.com/ClusterCockpit/cc-line-protocol/v2 v2.4.0 h1:hIzxgTBWcmCIHtoDKDkSCsKCOCOwUC34sFsbD2wcW0Q=
 github.com/ClusterCockpit/cc-line-protocol/v2 v2.4.0/go.mod h1:y42qUu+YFmu5fdNuUAS4VbbIKxVjxCvbVqFdpdh8ahY=
 github.com/DATA-DOG/go-sqlmock v1.5.2 h1:OcvFkGmslmlZibjAjaHm3L//6LiuBgolP7OputlJIzU=
@@ -39,42 +39,42 @@ github.com/apapsch/go-jsonmerge/v2 v2.0.0 h1:axGnT1gRIfimI7gJifB699GoE/oq+F2MU7D
 github.com/apapsch/go-jsonmerge/v2 v2.0.0/go.mod h1:lvDnEdqiQrp0O42VQGgmlKpxL1AP2+08jFMw88y4klk=
 github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0 h1:jfIu9sQUG6Ig+0+Ap1h4unLjW6YQJpKZVmUzxsD4E/Q=
 github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0/go.mod h1:t2tdKJDJF9BV14lnkjHmOQgcvEKgtqs5a1N3LNdJhGE=
-github.com/aws/aws-sdk-go-v2 v1.41.2 h1:LuT2rzqNQsauaGkPK/7813XxcZ3o3yePY0Iy891T2ls=
-github.com/aws/aws-sdk-go-v2 v1.41.2/go.mod h1:IvvlAZQXvTXznUPfRVfryiG1fbzE2NGK6m9u39YQ+S4=
-github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.5 h1:zWFmPmgw4sveAYi1mRqG+E/g0461cJ5M4bJ8/nc6d3Q=
-github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.5/go.mod h1:nVUlMLVV8ycXSb7mSkcNu9e3v/1TJq2RTlrPwhYWr5c=
-github.com/aws/aws-sdk-go-v2/config v1.32.10 h1:9DMthfO6XWZYLfzZglAgW5Fyou2nRI5CuV44sTedKBI=
-github.com/aws/aws-sdk-go-v2/config v1.32.10/go.mod h1:2rUIOnA2JaiqYmSKYmRJlcMWy6qTj1vuRFscppSBMcw=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.10 h1:EEhmEUFCE1Yhl7vDhNOI5OCL/iKMdkkYFTRpZXNw7m8=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.10/go.mod h1:RnnlFCAlxQCkN2Q379B67USkBMu1PipEEiibzYN5UTE=
-github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.18 h1:Ii4s+Sq3yDfaMLpjrJsqD6SmG/Wq/P5L/hw2qa78UAY=
-github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.18/go.mod h1:6x81qnY++ovptLE6nWQeWrpXxbnlIex+4H4eYYGcqfc=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.18 h1:F43zk1vemYIqPAwhjTjYIz0irU2EY7sOb/F5eJ3HuyM=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.18/go.mod h1:w1jdlZXrGKaJcNoL+Nnrj+k5wlpGXqnNrKoP22HvAug=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.18 h1:xCeWVjj0ki0l3nruoyP2slHsGArMxeiiaoPN5QZH6YQ=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.18/go.mod h1:r/eLGuGCBw6l36ZRWiw6PaZwPXb6YOj+i/7MizNl5/k=
-github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 h1:WKuaxf++XKWlHWu9ECbMlha8WOEGm0OUEZqm4K/Gcfk=
-github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4/go.mod h1:ZWy7j6v1vWGmPReu0iSGvRiise4YI5SkR3OHKTZ6Wuc=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.18 h1:eZioDaZGJ0tMM4gzmkNIO2aAoQd+je7Ug7TkvAzlmkU=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.18/go.mod h1:CCXwUKAJdoWr6/NcxZ+zsiPr6oH/Q5aTooRGYieAyj4=
-github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.5 h1:CeY9LUdur+Dxoeldqoun6y4WtJ3RQtzk0JMP2gfUay0=
-github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.5/go.mod h1:AZLZf2fMaahW5s/wMRciu1sYbdsikT/UHwbUjOdEVTc=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.10 h1:fJvQ5mIBVfKtiyx0AHY6HeWcRX5LGANLpq8SVR+Uazs=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.10/go.mod h1:Kzm5e6OmNH8VMkgK9t+ry5jEih4Y8whqs+1hrkxim1I=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.18 h1:LTRCYFlnnKFlKsyIQxKhJuDuA3ZkrDQMRYm6rXiHlLY=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.18/go.mod h1:XhwkgGG6bHSd00nO/mexWTcTjgd6PjuvWQMqSn2UaEk=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.18 h1:/A/xDuZAVD2BpsS2fftFRo/NoEKQJ8YTnJDEHBy2Gtg=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.18/go.mod h1:hWe9b4f+djUQGmyiGEeOnZv69dtMSgpDRIvNMvuvzvY=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.96.2 h1:M1A9AjcFwlxTLuf0Faj88L8Iqw0n/AJHjpZTQzMMsSc=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.96.2/go.mod h1:KsdTV6Q9WKUZm2mNJnUFmIoXfZux91M3sr/a4REX8e0=
-github.com/aws/aws-sdk-go-v2/service/signin v1.0.6 h1:MzORe+J94I+hYu2a6XmV5yC9huoTv8NRcCrUNedDypQ=
-github.com/aws/aws-sdk-go-v2/service/signin v1.0.6/go.mod h1:hXzcHLARD7GeWnifd8j9RWqtfIgxj4/cAtIVIK7hg8g=
-github.com/aws/aws-sdk-go-v2/service/sso v1.30.11 h1:7oGD8KPfBOJGXiCoRKrrrQkbvCp8N++u36hrLMPey6o=
-github.com/aws/aws-sdk-go-v2/service/sso v1.30.11/go.mod h1:0DO9B5EUJQlIDif+XJRWCljZRKsAFKh3gpFz7UnDtOo=
-github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.15 h1:edCcNp9eGIUDUCrzoCu1jWAXLGFIizeqkdkKgRlJwWc=
-github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.15/go.mod h1:lyRQKED9xWfgkYC/wmmYfv7iVIM68Z5OQ88ZdcV1QbU=
-github.com/aws/aws-sdk-go-v2/service/sts v1.41.7 h1:NITQpgo9A5NrDZ57uOWj+abvXSb83BbyggcUBVksN7c=
-github.com/aws/aws-sdk-go-v2/service/sts v1.41.7/go.mod h1:sks5UWBhEuWYDPdwlnRFn1w7xWdH29Jcpe+/PJQefEs=
+github.com/aws/aws-sdk-go-v2 v1.41.3 h1:4kQ/fa22KjDt13QCy1+bYADvdgcxpfH18f0zP542kZA=
+github.com/aws/aws-sdk-go-v2 v1.41.3/go.mod h1:mwsPRE8ceUUpiTgF7QmQIJ7lgsKUPQOUl3o72QBrE1o=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.6 h1:N4lRUXZpZ1KVEUn6hxtco/1d2lgYhNn1fHkkl8WhlyQ=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.6/go.mod h1:lyw7GFp3qENLh7kwzf7iMzAxDn+NzjXEAGjKS2UOKqI=
+github.com/aws/aws-sdk-go-v2/config v1.32.11 h1:ftxI5sgz8jZkckuUHXfC/wMUc8u3fG1vQS0plr2F2Zs=
+github.com/aws/aws-sdk-go-v2/config v1.32.11/go.mod h1:twF11+6ps9aNRKEDimksp923o44w/Thk9+8YIlzWMmo=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.11 h1:NdV8cwCcAXrCWyxArt58BrvZJ9pZ9Fhf9w6Uh5W3Uyc=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.11/go.mod h1:30yY2zqkMPdrvxBqzI9xQCM+WrlrZKSOpSJEsylVU+8=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.19 h1:INUvJxmhdEbVulJYHI061k4TVuS3jzzthNvjqvVvTKM=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.19/go.mod h1:FpZN2QISLdEBWkayloda+sZjVJL+e9Gl0k1SyTgcswU=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.19 h1:/sECfyq2JTifMI2JPyZ4bdRN77zJmr6SrS1eL3augIA=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.19/go.mod h1:dMf8A5oAqr9/oxOfLkC/c2LU/uMcALP0Rgn2BD5LWn0=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.19 h1:AWeJMk33GTBf6J20XJe6qZoRSJo0WfUhsMdUKhoODXE=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.19/go.mod h1:+GWrYoaAsV7/4pNHpwh1kiNLXkKaSoppxQq9lbH8Ejw=
+github.com/aws/aws-sdk-go-v2/internal/ini v1.8.5 h1:clHU5fm//kWS1C2HgtgWxfQbFbx4b6rx+5jzhgX9HrI=
+github.com/aws/aws-sdk-go-v2/internal/ini v1.8.5/go.mod h1:O3h0IK87yXci+kg6flUKzJnWeziQUKciKrLjcatSNcY=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.20 h1:qi3e/dmpdONhj1RyIZdi6DKKpDXS5Lb8ftr3p7cyHJc=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.20/go.mod h1:V1K+TeJVD5JOk3D9e5tsX2KUdL7BlB+FV6cBhdobN8c=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.6 h1:XAq62tBTJP/85lFD5oqOOe7YYgWxY9LvWq8plyDvDVg=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.6/go.mod h1:x0nZssQ3qZSnIcePWLvcoFisRXJzcTVvYpAAdYX8+GI=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.11 h1:BYf7XNsJMzl4mObARUBUib+j2tf0U//JAAtTnYqvqCw=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.11/go.mod h1:aEUS4WrNk/+FxkBZZa7tVgp4pGH+kFGW40Y8rCPqt5g=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.19 h1:X1Tow7suZk9UCJHE1Iw9GMZJJl0dAnKXXP1NaSDHwmw=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.19/go.mod h1:/rARO8psX+4sfjUQXp5LLifjUt8DuATZ31WptNJTyQA=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.19 h1:JnQeStZvPHFHeyky/7LbMlyQjUa+jIBj36OlWm0pzIk=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.19/go.mod h1:HGyasyHvYdFQeJhvDHfH7HXkHh57htcJGKDZ+7z+I24=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.97.0 h1:zyKY4OxzUImu+DigelJI9o49QQv8CjREs5E1CywjtIA=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.97.0/go.mod h1:NF3JcMGOiARAss1ld3WGORCw71+4ExDD2cbbdKS5PpA=
+github.com/aws/aws-sdk-go-v2/service/signin v1.0.7 h1:Y2cAXlClHsXkkOvWZFXATr34b0hxxloeQu/pAZz2row=
+github.com/aws/aws-sdk-go-v2/service/signin v1.0.7/go.mod h1:idzZ7gmDeqeNrSPkdbtMp9qWMgcBwykA7P7Rzh5DXVU=
+github.com/aws/aws-sdk-go-v2/service/sso v1.30.12 h1:iSsvB9EtQ09YrsmIc44Heqlx5ByGErqhPK1ZQLppias=
+github.com/aws/aws-sdk-go-v2/service/sso v1.30.12/go.mod h1:fEWYKTRGoZNl8tZ77i61/ccwOMJdGxwOhWCkp6TXAr0=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.16 h1:EnUdUqRP1CNzt2DkV67tJx6XDN4xlfBFm+bzeNOQVb0=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.16/go.mod h1:Jic/xv0Rq/pFNCh3WwpH4BEqdbSAl+IyHro8LbibHD8=
+github.com/aws/aws-sdk-go-v2/service/sts v1.41.8 h1:XQTQTF75vnug2TXS8m7CVJfC2nniYPZnO1D4Np761Oo=
+github.com/aws/aws-sdk-go-v2/service/sts v1.41.8/go.mod h1:Xgx+PR1NUOjNmQY+tRMnouRp83JRM8pRMw/vCaVhPkI=
 github.com/aws/smithy-go v1.24.2 h1:FzA3bu/nt/vDvmnkg+R8Xl46gmzEDam6mZ1hzmwXFng=
 github.com/aws/smithy-go v1.24.2/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
@@ -236,10 +236,10 @@ github.com/parquet-go/bitpack v1.0.0 h1:AUqzlKzPPXf2bCdjfj4sTeacrUwsT7NlcYDMUQxP
 github.com/parquet-go/bitpack v1.0.0/go.mod h1:XnVk9TH+O40eOOmvpAVZ7K2ocQFrQwysLMnc6M/8lgs=
 github.com/parquet-go/jsonlite v1.4.0 h1:RTG7prqfO0HD5egejU8MUDBN8oToMj55cgSV1I0zNW4=
 github.com/parquet-go/jsonlite v1.4.0/go.mod h1:nDjpkpL4EOtqs6NQugUsi0Rleq9sW/OtC1NnZEnxzF0=
-github.com/parquet-go/parquet-go v0.28.0 h1:ECyksyv8T2pOrlLsN7aWJIoQakyk/HtxQ2lchgS4els=
-github.com/parquet-go/parquet-go v0.28.0/go.mod h1:navtkAYr2LGoJVp141oXPlO/sxLvaOe3la2JEoD8+rg=
-github.com/pierrec/lz4/v4 v4.1.25 h1:kocOqRffaIbU5djlIBr7Wh+cx82C0vtFb0fOurZHqD0=
-github.com/pierrec/lz4/v4 v4.1.25/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
+github.com/parquet-go/parquet-go v0.29.0 h1:xXlPtFVR51jpSVzf+cgHnNIcb7Xet+iuvkbe0HIm90Y=
+github.com/parquet-go/parquet-go v0.29.0/go.mod h1:navtkAYr2LGoJVp141oXPlO/sxLvaOe3la2JEoD8+rg=
+github.com/pierrec/lz4/v4 v4.1.26 h1:GrpZw1gZttORinvzBdXPUXATeqlJjqUG/D87TKMnhjY=
+github.com/pierrec/lz4/v4 v4.1.26/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
 github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
@@ -288,8 +288,8 @@ github.com/twpayne/go-geom v1.6.1 h1:iLE+Opv0Ihm/ABIcvQFGIiFBXd76oBIar9drAwHFhR4
 github.com/twpayne/go-geom v1.6.1/go.mod h1:Kr+Nly6BswFsKM5sd31YaoWS5PeDDH2NftJTK7Gd028=
 github.com/urfave/cli/v2 v2.27.7 h1:bH59vdhbjLv3LAvIu6gd0usJHgoTTPhCFib8qqOwXYU=
 github.com/urfave/cli/v2 v2.27.7/go.mod h1:CyNAG/xg+iAOg0N4MPGZqVmv2rCoP267496AOXUZjA4=
-github.com/urfave/cli/v3 v3.6.2 h1:lQuqiPrZ1cIz8hz+HcrG0TNZFxU70dPZ3Yl+pSrH9A8=
-github.com/urfave/cli/v3 v3.6.2/go.mod h1:ysVLtOEmg2tOy6PknnYVhDoouyC/6N42TMeoMzskhso=
+github.com/urfave/cli/v3 v3.7.0 h1:AGSnbUyjtLiM+WJUb4dzXKldl/gL+F8OwmRDtVr6g2U=
+github.com/urfave/cli/v3 v3.7.0/go.mod h1:ysVLtOEmg2tOy6PknnYVhDoouyC/6N42TMeoMzskhso=
 github.com/vektah/gqlparser/v2 v2.5.32 h1:k9QPJd4sEDTL+qB4ncPLflqTJ3MmjB9SrVzJrawpFSc=
 github.com/vektah/gqlparser/v2 v2.5.32/go.mod h1:c1I28gSOVNzlfc4WuDlqU7voQnsqI6OG2amkBAFmgts=
 github.com/xrash/smetrics v0.0.0-20250705151800-55b8f293f342 h1:FnBeRrxr7OU4VvAzt5X7s6266i6cSVkkFPS0TuXWbIg=
@@ -305,31 +305,31 @@ go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
 go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
-golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
-golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
+golang.org/x/crypto v0.49.0 h1:+Ng2ULVvLHnJ/ZFEq4KdcDd/cfjrrjjNSXNzxg0Y4U4=
+golang.org/x/crypto v0.49.0/go.mod h1:ErX4dUh2UM+CFYiXZRTcMpEcN8b/1gxEuv3nODoYtCA=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
-golang.org/x/mod v0.33.0 h1:tHFzIWbBifEmbwtGz65eaWyGiGZatSrT9prnU8DbVL8=
-golang.org/x/mod v0.33.0/go.mod h1:swjeQEj+6r7fODbD2cqrnje9PnziFuw4bmLbBZFrQ5w=
+golang.org/x/mod v0.34.0 h1:xIHgNUUnW6sYkcM5Jleh05DvLOtwc6RitGHbDk4akRI=
+golang.org/x/mod v0.34.0/go.mod h1:ykgH52iCZe79kzLLMhyCUzhMci+nQj+0XkbXpNYtVjY=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
-golang.org/x/net v0.51.0 h1:94R/GTO7mt3/4wIKpcR5gkGmRLOuE/2hNGeWq/GBIFo=
-golang.org/x/net v0.51.0/go.mod h1:aamm+2QF5ogm02fjy5Bb7CQ0WMt1/WVM7FtyaTLlA9Y=
-golang.org/x/oauth2 v0.35.0 h1:Mv2mzuHuZuY2+bkyWXIHMfhNdJAdwW3FuWeCPYN5GVQ=
-golang.org/x/oauth2 v0.35.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA=
+golang.org/x/net v0.52.0 h1:He/TN1l0e4mmR3QqHMT2Xab3Aj3L9qjbhRm78/6jrW0=
+golang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw=
+golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
+golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
 golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
-golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
+golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
+golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
-golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
+golang.org/x/sys v0.42.0 h1:omrd2nAlyT5ESRdCLYdm3+fMfNFE/+Rf4bDIQImRJeo=
+golang.org/x/sys v0.42.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
@@ -337,15 +337,15 @@ golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
-golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
-golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
-golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
-golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
+golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
+golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
+golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
+golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
-golang.org/x/tools v0.42.0 h1:uNgphsn75Tdz5Ji2q36v/nsFSfR/9BRFvqhGBaJGd5k=
-golang.org/x/tools v0.42.0/go.mod h1:Ma6lCIwGZvHK6XtgbswSoWroEkhugApmsXyrUmBhfr0=
+golang.org/x/tools v0.43.0 h1:12BdW9CeB3Z+J/I/wj34VMl8X+fEXBxVR90JeMX5E7s=
+golang.org/x/tools v0.43.0/go.mod h1:uHkMso649BX2cZK6+RpuIPXS3ho2hZo4FVwfoy1vIk0=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
 google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
@@ -77,6 +77,17 @@ type ProgramConfig struct {

 	// Node state retention configuration
 	NodeStateRetention *NodeStateRetention `json:"nodestate-retention"`
+
+	// Database tuning configuration
+	DbConfig *DbConfig `json:"db-config"`
+}
+
+type DbConfig struct {
+	CacheSizeMB               int `json:"cache-size-mb"`
+	SoftHeapLimitMB           int `json:"soft-heap-limit-mb"`
+	MaxOpenConnections        int `json:"max-open-connections"`
+	MaxIdleConnections        int `json:"max-idle-connections"`
+	ConnectionMaxIdleTimeMins int `json:"max-idle-time-minutes"`
 }

 type NodeStateRetention struct {
@@ -177,6 +177,32 @@ var configSchema = `
        }
      },
      "required": ["policy"]
+    },
+    "db-config": {
+      "description": "SQLite database tuning configuration.",
+      "type": "object",
+      "properties": {
+        "cache-size-mb": {
+          "description": "SQLite page cache size per connection in MB (default: 2048).",
+          "type": "integer"
+        },
+        "soft-heap-limit-mb": {
+          "description": "Process-wide SQLite soft heap limit in MB (default: 16384).",
+          "type": "integer"
+        },
+        "max-open-connections": {
+          "description": "Maximum number of open database connections (default: 4).",
+          "type": "integer"
+        },
+        "max-idle-connections": {
+          "description": "Maximum number of idle database connections (default: 4).",
+          "type": "integer"
+        },
+        "max-idle-time-minutes": {
+          "description": "Maximum idle time for a connection in minutes (default: 10).",
+          "type": "integer"
+        }
+      }
    }
  }
 }`
@@ -13209,6 +13209,10 @@ func (ec *executionContext) fieldContext___Type_isOneOf(_ context.Context, field

 func (ec *executionContext) unmarshalInputFloatRange(ctx context.Context, obj any) (model.FloatRange, error) {
 	var it model.FloatRange
+	if obj == nil {
+		return it, nil
+	}
+
 	asMap := map[string]any{}
 	for k, v := range obj.(map[string]any) {
 		asMap[k] = v
@@ -13242,6 +13246,10 @@ func (ec *executionContext) unmarshalInputFloatRange(ctx context.Context, obj an

 func (ec *executionContext) unmarshalInputIntRange(ctx context.Context, obj any) (config.IntRange, error) {
 	var it config.IntRange
+	if obj == nil {
+		return it, nil
+	}
+
 	asMap := map[string]any{}
 	for k, v := range obj.(map[string]any) {
 		asMap[k] = v
@@ -13275,6 +13283,10 @@ func (ec *executionContext) unmarshalInputIntRange(ctx context.Context, obj any)

 func (ec *executionContext) unmarshalInputJobFilter(ctx context.Context, obj any) (model.JobFilter, error) {
 	var it model.JobFilter
+	if obj == nil {
+		return it, nil
+	}
+
 	asMap := map[string]any{}
 	for k, v := range obj.(map[string]any) {
 		asMap[k] = v
@@ -13448,6 +13460,10 @@ func (ec *executionContext) unmarshalInputJobFilter(ctx context.Context, obj any

 func (ec *executionContext) unmarshalInputMetricStatItem(ctx context.Context, obj any) (model.MetricStatItem, error) {
 	var it model.MetricStatItem
+	if obj == nil {
+		return it, nil
+	}
+
 	asMap := map[string]any{}
 	for k, v := range obj.(map[string]any) {
 		asMap[k] = v
@@ -13481,6 +13497,10 @@ func (ec *executionContext) unmarshalInputMetricStatItem(ctx context.Context, ob

 func (ec *executionContext) unmarshalInputNodeFilter(ctx context.Context, obj any) (model.NodeFilter, error) {
 	var it model.NodeFilter
+	if obj == nil {
+		return it, nil
+	}
+
 	asMap := map[string]any{}
 	for k, v := range obj.(map[string]any) {
 		asMap[k] = v
@@ -13542,6 +13562,10 @@ func (ec *executionContext) unmarshalInputNodeFilter(ctx context.Context, obj an

 func (ec *executionContext) unmarshalInputOrderByInput(ctx context.Context, obj any) (model.OrderByInput, error) {
 	var it model.OrderByInput
+	if obj == nil {
+		return it, nil
+	}
+
 	asMap := map[string]any{}
 	for k, v := range obj.(map[string]any) {
 		asMap[k] = v
@@ -13586,6 +13610,10 @@ func (ec *executionContext) unmarshalInputOrderByInput(ctx context.Context, obj

 func (ec *executionContext) unmarshalInputPageRequest(ctx context.Context, obj any) (model.PageRequest, error) {
 	var it model.PageRequest
+	if obj == nil {
+		return it, nil
+	}
+
 	asMap := map[string]any{}
 	for k, v := range obj.(map[string]any) {
 		asMap[k] = v
@@ -13619,6 +13647,10 @@ func (ec *executionContext) unmarshalInputPageRequest(ctx context.Context, obj a

 func (ec *executionContext) unmarshalInputStringInput(ctx context.Context, obj any) (model.StringInput, error) {
 	var it model.StringInput
+	if obj == nil {
+		return it, nil
+	}
+
 	asMap := map[string]any{}
 	for k, v := range obj.(map[string]any) {
 		asMap[k] = v
@@ -13680,6 +13712,10 @@ func (ec *executionContext) unmarshalInputStringInput(ctx context.Context, obj a

 func (ec *executionContext) unmarshalInputTimeRange(ctx context.Context, obj any) (config.TimeRange, error) {
 	var it config.TimeRange
+	if obj == nil {
+		return it, nil
+	}
+
 	asMap := map[string]any{}
 	for k, v := range obj.(map[string]any) {
 		asMap[k] = v
@@ -3,7 +3,7 @@ package graph
 // This file will be automatically regenerated based on the schema, any resolver
 // implementations
 // will be copied through when generating and any unknown code will be moved to the end.
-// Code generated by github.com/99designs/gqlgen version v0.17.87
+// Code generated by github.com/99designs/gqlgen version v0.17.88

 import (
 	"context"
@@ -652,18 +652,51 @@ func (r *queryResolver) JobsStatistics(ctx context.Context, filter []*model.JobF
 	defaultDurationBins := "1h"
 	defaultMetricBins := 10

-	if requireField(ctx, "totalJobs") || requireField(ctx, "totalUsers") || requireField(ctx, "totalWalltime") || requireField(ctx, "totalNodes") || requireField(ctx, "totalCores") ||
-		requireField(ctx, "totalAccs") || requireField(ctx, "totalNodeHours") || requireField(ctx, "totalCoreHours") || requireField(ctx, "totalAccHours") {
+	// Build requested fields map for selective column computation
+	statsFields := []string{
+		"totalJobs", "totalUsers", "totalWalltime", "totalNodes", "totalCores",
+		"totalAccs", "totalNodeHours", "totalCoreHours", "totalAccHours", "runningJobs", "shortJobs",
+	}
+	reqFields := make(map[string]bool, len(statsFields))
+	fetchedMainStats := false
+	for _, f := range statsFields {
+		if requireField(ctx, f) {
+			reqFields[f] = true
+			if f != "runningJobs" && f != "shortJobs" {
+				fetchedMainStats = true
+			}
+		}
+	}
+
+	if fetchedMainStats {
 		if groupBy == nil {
-			stats, err = r.Repo.JobsStats(ctx, filter)
+			stats, err = r.Repo.JobsStats(ctx, filter, reqFields)
 		} else {
-			stats, err = r.Repo.JobsStatsGrouped(ctx, filter, page, sortBy, groupBy)
+			startGrouped := time.Now()
+			// Use request-scoped cache: multiple aliases with same (filter, groupBy)
+			// but different sortBy/page hit the DB only once.
+			if cache := getStatsGroupCache(ctx); cache != nil {
+				key := statsCacheKey(filter, groupBy)
+				var allStats []*model.JobsStatistics
+				allStats, err = cache.getOrCompute(key, func() ([]*model.JobsStatistics, error) {
+					return r.Repo.JobsStatsGrouped(ctx, filter, nil, nil, groupBy, nil)
+				})
+				if err == nil {
+					stats = sortAndPageStats(allStats, sortBy, page)
+				}
+			} else {
+				stats, err = r.Repo.JobsStatsGrouped(ctx, filter, page, sortBy, groupBy, reqFields)
+			}
+			cclog.Infof("Timer JobsStatsGrouped call: %s", time.Since(startGrouped))
 		}
 	} else {
 		stats = make([]*model.JobsStatistics, 0, 1)
 		stats = append(stats, &model.JobsStatistics{})
 	}

+	// runningJobs and shortJobs are already inlined in JobsStats/JobsStatsGrouped.
+	// Only run separate count queries if main stats were not fetched.
+	if !fetchedMainStats {
 		if groupBy != nil {
 			if requireField(ctx, "shortJobs") {
 				stats, err = r.Repo.AddJobCountGrouped(ctx, filter, groupBy, stats, "short")
@@ -679,6 +712,7 @@ func (r *queryResolver) JobsStatistics(ctx context.Context, filter []*model.JobF
 				stats, err = r.Repo.AddJobCount(ctx, filter, stats, "running")
 			}
 		}
+	}

 	if err != nil {
 		return nil, err
@@ -0,0 +1,135 @@
+// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
+// All rights reserved. This file is part of cc-backend.
+// Use of this source code is governed by a MIT-style
+// license that can be found in the LICENSE file.
+
+package graph
+
+import (
+	"context"
+	"fmt"
+	"slices"
+	"sync"
+
+	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
+)
+
+// statsGroupCache is a per-request cache for grouped JobsStatistics results.
+// It deduplicates identical (filter+groupBy) SQL queries that arise when the
+// frontend requests multiple sort/page slices of the same underlying data
+// (e.g. topUserJobs, topUserNodes, topUserAccs all group by USER).
+type statsGroupCache struct {
+	mu      sync.Mutex
+	entries map[string]*cacheEntry
+}
+
+type cacheEntry struct {
+	once   sync.Once
+	result []*model.JobsStatistics
+	err    error
+}
+
+type ctxKey int
+
+const statsGroupCacheKey ctxKey = iota
+
+// newStatsGroupCache creates a new empty cache.
+func newStatsGroupCache() *statsGroupCache {
+	return &statsGroupCache{
+		entries: make(map[string]*cacheEntry),
+	}
+}
+
+// WithStatsGroupCache injects a new cache into the context.
+func WithStatsGroupCache(ctx context.Context) context.Context {
+	return context.WithValue(ctx, statsGroupCacheKey, newStatsGroupCache())
+}
+
+// getStatsGroupCache retrieves the cache from context, or nil if absent.
+func getStatsGroupCache(ctx context.Context) *statsGroupCache {
+	if c, ok := ctx.Value(statsGroupCacheKey).(*statsGroupCache); ok {
+		return c
+	}
+	return nil
+}
+
+// cacheKey builds a deterministic string key from filter + groupBy.
+func statsCacheKey(filter []*model.JobFilter, groupBy *model.Aggregate) string {
+	return fmt.Sprintf("%v|%v", filter, *groupBy)
+}
+
+// getOrCompute returns cached results for the given key, computing them on
+// first access via the provided function.
+func (c *statsGroupCache) getOrCompute(
+	key string,
+	compute func() ([]*model.JobsStatistics, error),
+) ([]*model.JobsStatistics, error) {
+	c.mu.Lock()
+	entry, ok := c.entries[key]
+	if !ok {
+		entry = &cacheEntry{}
+		c.entries[key] = entry
+	}
+	c.mu.Unlock()
+
+	entry.once.Do(func() {
+		entry.result, entry.err = compute()
+	})
+	return entry.result, entry.err
+}
+
+// sortAndPageStats sorts a copy of allStats by the given sortBy field (descending)
+// and returns the requested page slice.
+func sortAndPageStats(allStats []*model.JobsStatistics, sortBy *model.SortByAggregate, page *model.PageRequest) []*model.JobsStatistics {
+	// Work on a shallow copy so the cached slice order is not mutated.
+	sorted := make([]*model.JobsStatistics, len(allStats))
+	copy(sorted, allStats)
+
+	if sortBy != nil {
+		getter := statsFieldGetter(*sortBy)
+		slices.SortFunc(sorted, func(a, b *model.JobsStatistics) int {
+			return getter(b) - getter(a) // descending
+		})
+	}
+
+	if page != nil && page.ItemsPerPage != -1 {
+		start := (page.Page - 1) * page.ItemsPerPage
+		if start >= len(sorted) {
+			return nil
+		}
+		end := start + page.ItemsPerPage
+		if end > len(sorted) {
+			end = len(sorted)
+		}
+		sorted = sorted[start:end]
+	}
+
+	return sorted
+}
+
+// statsFieldGetter returns a function that extracts the sortable int field
+// from a JobsStatistics struct for the given sort key.
+func statsFieldGetter(sortBy model.SortByAggregate) func(*model.JobsStatistics) int {
+	switch sortBy {
+	case model.SortByAggregateTotaljobs:
+		return func(s *model.JobsStatistics) int { return s.TotalJobs }
+	case model.SortByAggregateTotalusers:
+		return func(s *model.JobsStatistics) int { return s.TotalUsers }
+	case model.SortByAggregateTotalwalltime:
+		return func(s *model.JobsStatistics) int { return s.TotalWalltime }
+	case model.SortByAggregateTotalnodes:
+		return func(s *model.JobsStatistics) int { return s.TotalNodes }
+	case model.SortByAggregateTotalnodehours:
+		return func(s *model.JobsStatistics) int { return s.TotalNodeHours }
+	case model.SortByAggregateTotalcores:
+		return func(s *model.JobsStatistics) int { return s.TotalCores }
+	case model.SortByAggregateTotalcorehours:
+		return func(s *model.JobsStatistics) int { return s.TotalCoreHours }
+	case model.SortByAggregateTotalaccs:
+		return func(s *model.JobsStatistics) int { return s.TotalAccs }
+	case model.SortByAggregateTotalacchours:
+		return func(s *model.JobsStatistics) int { return s.TotalAccHours }
+	default:
+		return func(s *model.JobsStatistics) int { return s.TotalJobs }
+	}
+}
@@ -27,13 +27,25 @@ type RepositoryConfig struct {
 	ConnectionMaxLifetime time.Duration

 	// ConnectionMaxIdleTime is the maximum amount of time a connection may be idle.
-	// Default: 1 hour
+	// Default: 10 minutes
 	ConnectionMaxIdleTime time.Duration

 	// MinRunningJobDuration is the minimum duration in seconds for a job to be
 	// considered in "running jobs" queries. This filters out very short jobs.
 	// Default: 600 seconds (10 minutes)
 	MinRunningJobDuration int
+
+	// DbCacheSizeMB is the SQLite page cache size per connection in MB.
+	// Uses negative PRAGMA cache_size notation (KiB). With MaxOpenConnections=4
+	// and DbCacheSizeMB=2048, total page cache is up to 8GB.
+	// Default: 2048 (2GB)
+	DbCacheSizeMB int
+
+	// DbSoftHeapLimitMB is the process-wide SQLite soft heap limit in MB.
+	// SQLite will try to release cache pages to stay under this limit.
+	// It's a soft limit — queries won't fail, but cache eviction becomes more aggressive.
+	// Default: 16384 (16GB)
+	DbSoftHeapLimitMB int
 }

 // DefaultConfig returns the default repository configuration.
@@ -44,8 +56,10 @@ func DefaultConfig() *RepositoryConfig {
 		MaxOpenConnections:    4,
 		MaxIdleConnections:    4,
 		ConnectionMaxLifetime: time.Hour,
-		ConnectionMaxIdleTime: time.Hour,
+		ConnectionMaxIdleTime: 10 * time.Minute,
 		MinRunningJobDuration: 600, // 10 minutes
+		DbCacheSizeMB:        2048,  // 2GB per connection
+		DbSoftHeapLimitMB:    16384, // 16GB process-wide
 	}
 }

@@ -36,9 +36,10 @@ type DatabaseOptions struct {
 	ConnectionMaxIdleTime time.Duration
 }

-func setupSqlite(db *sql.DB) error {
+func setupSqlite(db *sql.DB, cfg *RepositoryConfig) error {
 	pragmas := []string{
 		"temp_store = memory",
+		fmt.Sprintf("soft_heap_limit = %d", int64(cfg.DbSoftHeapLimitMB)*1024*1024),
 	}

 	for _, pragma := range pragmas {
@@ -71,7 +72,8 @@ func Connect(db string) {
 		connectionURLParams.Add("_journal_mode", "WAL")
 		connectionURLParams.Add("_busy_timeout", "5000")
 		connectionURLParams.Add("_synchronous", "NORMAL")
-		connectionURLParams.Add("_cache_size", "1000000000")
+		cacheSizeKiB := repoConfig.DbCacheSizeMB * 1024 // Convert MB to KiB
+		connectionURLParams.Add("_cache_size", fmt.Sprintf("-%d", cacheSizeKiB))
 		connectionURLParams.Add("_foreign_keys", "true")
 		opts.URL = fmt.Sprintf("file:%s?%s", opts.URL, connectionURLParams.Encode())

@@ -86,11 +88,14 @@ func Connect(db string) {
 			cclog.Abortf("DB Connection: Could not connect to SQLite database with sqlx.Open().\nError: %s\n", err.Error())
 		}

-		err = setupSqlite(dbHandle.DB)
+		err = setupSqlite(dbHandle.DB, repoConfig)
 		if err != nil {
 			cclog.Abortf("Failed sqlite db setup.\nError: %s\n", err.Error())
 		}

+		cclog.Infof("SQLite config: cache_size=%dMB/conn, soft_heap_limit=%dMB, max_conns=%d",
+			repoConfig.DbCacheSizeMB, repoConfig.DbSoftHeapLimitMB, repoConfig.MaxOpenConnections)
+
 		dbHandle.SetMaxOpenConns(opts.MaxOpenConnections)
 		dbHandle.SetMaxIdleConns(opts.MaxIdleConnections)
 		dbHandle.SetConnMaxLifetime(opts.ConnectionMaxLifetime)
@@ -1066,7 +1066,7 @@ func (r *JobRepository) GetUsedNodes(ts int64) (map[string][]string, error) {
 	// Note: Query expects index on (job_state, start_time) for optimal performance
 	q := sq.Select("job.cluster", "job.resources").From("job").
 		Where("job.start_time < ?", ts).
-		Where(sq.Eq{"job.job_state": "running"})
+		Where("job.job_state = 'running'")

 	rows, err := q.RunWith(r.stmtCache).Query()
 	if err != nil {
@@ -96,13 +96,13 @@ func (r *JobRepository) SyncJobs() ([]*schema.Job, error) {
 	_, err = r.DB.Exec(
 		"INSERT OR IGNORE INTO job (job_id, cluster, subcluster, start_time, hpc_user, project, cluster_partition, array_job_id, num_nodes, num_hwthreads, num_acc, shared, monitoring_status, smt, job_state, duration, walltime, footprint, energy, energy_footprint, resources, meta_data) SELECT job_id, cluster, subcluster, start_time, hpc_user, project, cluster_partition, array_job_id, num_nodes, num_hwthreads, num_acc, shared, monitoring_status, smt, job_state, duration, walltime, footprint, energy, energy_footprint, resources, meta_data FROM job_cache")
 	if err != nil {
-		cclog.Warnf("Error while Job sync: %v", err)
+		cclog.Errorf("Error while Job sync: %v", err)
 		return nil, err
 	}

 	_, err = r.DB.Exec("DELETE FROM job_cache")
 	if err != nil {
-		cclog.Warnf("Error while Job cache clean: %v", err)
+		cclog.Errorf("Error while Job cache clean: %v", err)
 		return nil, err
 	}

@@ -211,4 +211,3 @@ func (r *JobRepository) Stop(
 	_, err = stmt.RunWith(r.stmtCache).Exec()
 	return err
 }
-
@@ -171,7 +171,7 @@ func (r *JobRepository) FindByID(ctx context.Context, jobID int64) (*schema.Job,
 		return nil, qerr
 	}

-	return scanJob(q.RunWith(r.stmtCache).QueryRow())
+	return scanJob(q.RunWith(r.stmtCache).QueryRowContext(ctx))
 }

 // FindByIDWithUser executes a SQL query to find a specific batch job.
@@ -217,7 +217,7 @@ func (r *JobRepository) FindByJobID(ctx context.Context, jobID int64, startTime
 		return nil, qerr
 	}

-	return scanJob(q.RunWith(r.stmtCache).QueryRow())
+	return scanJob(q.RunWith(r.stmtCache).QueryRowContext(ctx))
 }

 // IsJobOwner checks if the specified user owns the batch job identified by jobID,
@@ -63,7 +63,7 @@ func (r *JobRepository) QueryJobs(
 			}
 		} else {
 			// Order by footprint JSON field values
-			query = query.Where("JSON_VALID(meta_data)")
+			query = query.Where("JSON_VALID(footprint)")
 			switch order.Order {
 			case model.SortDirectionEnumAsc:
 				query = query.OrderBy(fmt.Sprintf("JSON_EXTRACT(footprint, \"$.%s\") ASC", field))
@@ -84,7 +84,7 @@ func (r *JobRepository) QueryJobs(
 		query = BuildWhereClause(f, query)
 	}

-	rows, err := query.RunWith(r.stmtCache).Query()
+	rows, err := query.RunWith(r.stmtCache).QueryContext(ctx)
 	if err != nil {
 		queryString, queryVars, _ := query.ToSql()
 		return nil, fmt.Errorf("query failed [%s] %v: %w", queryString, queryVars, err)
@@ -126,7 +126,7 @@ func (r *JobRepository) CountJobs(
 	}

 	var count int
-	if err := query.RunWith(r.DB).Scan(&count); err != nil {
+	if err := query.RunWith(r.DB).QueryRowContext(ctx).Scan(&count); err != nil {
 		return 0, fmt.Errorf("failed to count jobs: %w", err)
 	}

@@ -197,14 +197,22 @@ func BuildWhereClause(filter *model.JobFilter, query sq.SelectBuilder) sq.Select
 		query = buildStringCondition("job.cluster_partition", filter.Partition, query)
 	}
 	if filter.State != nil {
+		if len(filter.State) == 1 {
+			// Inline literal value so SQLite can match partial indexes (WHERE job_state = 'running').
+			// Safe: values come from validated GraphQL enum (model.JobState).
+			singleStat := string(filter.State[0])
+			query = query.Where(fmt.Sprintf("job.job_state = '%s'", singleStat))
+		} else {
 			states := make([]string, len(filter.State))
 			for i, val := range filter.State {
 				states[i] = string(val)
 			}
 			query = query.Where(sq.Eq{"job.job_state": states})
 		}
+	}
 	if filter.Shared != nil {
-		query = query.Where("job.shared = ?", *filter.Shared)
+		// Inline literal value so SQLite can match partial indexes (see above).
+		query = query.Where(fmt.Sprintf("job.shared = '%s'", *filter.Shared))
 	}
 	if filter.Project != nil {
 		query = buildStringCondition("job.project", filter.Project, query)
@@ -222,7 +230,8 @@ func BuildWhereClause(filter *model.JobFilter, query sq.SelectBuilder) sq.Select
 		query = buildIntCondition("job.num_hwthreads", filter.NumHWThreads, query)
 	}
 	if filter.ArrayJobID != nil {
-		query = query.Where("job.array_job_id = ?", *filter.ArrayJobID)
+		// Inline literal value so SQLite can match partial indexes (see above).
+		query = query.Where(fmt.Sprintf("job.array_job_id = %d", *filter.ArrayJobID))
 	}
 	if filter.StartTime != nil {
 		query = buildTimeCondition("job.start_time", filter.StartTime, query)
@@ -276,28 +285,26 @@ func BuildWhereClause(filter *model.JobFilter, query sq.SelectBuilder) sq.Select
 	return query
 }

-// buildIntCondition creates a BETWEEN clause for integer range filters.
-// Reminder: BETWEEN Queries are slower and dont use indices as frequently: Only use if both conditions required
+// buildIntCondition creates clauses for integer range filters, using BETWEEN only if required.
 func buildIntCondition(field string, cond *config.IntRange, query sq.SelectBuilder) sq.SelectBuilder {
-	if cond.From != 0 && cond.To != 0 {
+	if cond.From != 1 && cond.To != 0 {
 		return query.Where(field+" BETWEEN ? AND ?", cond.From, cond.To)
-	} else if cond.From != 0 {
+	} else if cond.From != 1 && cond.To == 0 {
 		return query.Where(field+" >= ?", cond.From)
-	} else if cond.To != 0 {
+	} else if cond.From == 1 && cond.To != 0 {
 		return query.Where(field+" <= ?", cond.To)
 	} else {
 		return query
 	}
 }

-// buildFloatCondition creates a BETWEEN clause for float range filters.
-// Reminder: BETWEEN Queries are slower and dont use indices as frequently: Only use if both conditions required
+// buildFloatCondition creates a clauses for float range filters, using BETWEEN only if required.
 func buildFloatCondition(field string, cond *model.FloatRange, query sq.SelectBuilder) sq.SelectBuilder {
-	if cond.From != 0.0 && cond.To != 0.0 {
+	if cond.From != 1.0 && cond.To != 0.0 {
 		return query.Where(field+" BETWEEN ? AND ?", cond.From, cond.To)
-	} else if cond.From != 0.0 {
+	} else if cond.From != 1.0 && cond.To == 0.0 {
 		return query.Where(field+" >= ?", cond.From)
-	} else if cond.To != 0.0 {
+	} else if cond.From == 1.0 && cond.To != 0.0 {
 		return query.Where(field+" <= ?", cond.To)
 	} else {
 		return query
@@ -336,16 +343,15 @@ func buildTimeCondition(field string, cond *config.TimeRange, query sq.SelectBui
 	}
 }

-// buildFloatJSONCondition creates a filter on a numeric field within the footprint JSON column.
-// Reminder: BETWEEN Queries are slower and dont use indices as frequently: Only use if both conditions required
-func buildFloatJSONCondition(condName string, condRange *model.FloatRange, query sq.SelectBuilder) sq.SelectBuilder {
+// buildFloatJSONCondition creates a filter on a numeric field within the footprint JSON column, using BETWEEN only if required.
+func buildFloatJSONCondition(jsonField string, cond *model.FloatRange, query sq.SelectBuilder) sq.SelectBuilder {
 	query = query.Where("JSON_VALID(footprint)")
-	if condRange.From != 0.0 && condRange.To != 0.0 {
-		return query.Where("JSON_EXTRACT(footprint, \"$."+condName+"\") BETWEEN ? AND ?", condRange.From, condRange.To)
-	} else if condRange.From != 0.0 {
-		return query.Where("JSON_EXTRACT(footprint, \"$."+condName+"\") >= ?", condRange.From)
-	} else if condRange.To != 0.0 {
-		return query.Where("JSON_EXTRACT(footprint, \"$."+condName+"\") <= ?", condRange.To)
+	if cond.From != 1.0 && cond.To != 0.0 {
+		return query.Where("JSON_EXTRACT(footprint, \"$."+jsonField+"\") BETWEEN ? AND ?", cond.From, cond.To)
+	} else if cond.From != 1.0 && cond.To == 0.0 {
+		return query.Where("JSON_EXTRACT(footprint, \"$."+jsonField+"\") >= ?", cond.From)
+	} else if cond.From == 1.0 && cond.To != 0.0 {
+		return query.Where("JSON_EXTRACT(footprint, \"$."+jsonField+"\") <= ?", cond.To)
 	} else {
 		return query
 	}
@@ -21,10 +21,11 @@ import (
 // is added to internal/repository/migrations/sqlite3/.
 //
 // Version history:
-//   - Version 10: Current version
+//   - Version 11: Optimize job table indexes (reduce from ~78 to 48, add covering/partial indexes)
+//   - Version 10: Node table
 //
 // Migration files are embedded at build time from the migrations directory.
-const Version uint = 10
+const Version uint = 11

 //go:embed migrations/*
 var migrationFiles embed.FS
@@ -101,3 +101,5 @@ DROP INDEX IF EXISTS jobs_numnodes_starttime;
 DROP INDEX IF EXISTS jobs_numhwthreads_starttime;
 DROP INDEX IF EXISTS jobs_numacc_starttime;
 DROP INDEX IF EXISTS jobs_energy_starttime;
+
+PRAGMA optimize;
@@ -139,12 +139,6 @@ CREATE INDEX IF NOT EXISTS jobs_cluster_partition_project ON job (cluster, clust
 CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate ON job (cluster, cluster_partition, job_state);
 CREATE INDEX IF NOT EXISTS jobs_cluster_partition_shared ON job (cluster, cluster_partition, shared);

-- Cluster+Partition Filter Sorting
-CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numnodes ON job (cluster, cluster_partition, num_nodes);
-CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numhwthreads ON job (cluster, cluster_partition, num_hwthreads);
-CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numacc ON job (cluster, cluster_partition, num_acc);
-CREATE INDEX IF NOT EXISTS jobs_cluster_partition_energy ON job (cluster, cluster_partition, energy);
-
 -- Cluster+Partition Time Filter Sorting
 CREATE INDEX IF NOT EXISTS jobs_cluster_partition_duration_starttime ON job (cluster, cluster_partition, duration, start_time);
 CREATE INDEX IF NOT EXISTS jobs_cluster_partition_starttime_duration ON job (cluster, cluster_partition, start_time, duration);
@@ -152,11 +146,6 @@ CREATE INDEX IF NOT EXISTS jobs_cluster_partition_starttime_duration ON job (clu
 -- Cluster+JobState Filter
 CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user ON job (cluster, job_state, hpc_user);
 CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project ON job (cluster, job_state, project);
-- Cluster+JobState Filter Sorting
-CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numnodes ON job (cluster, job_state, num_nodes);
-CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numhwthreads ON job (cluster, job_state, num_hwthreads);
-CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numacc ON job (cluster, job_state, num_acc);
-CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_energy ON job (cluster, job_state, energy);

 -- Cluster+JobState Time Filter Sorting
 CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_starttime_duration ON job (cluster, job_state, start_time, duration);
@@ -165,34 +154,18 @@ CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_duration_starttime ON job (clus
 -- Cluster+Shared Filter
 CREATE INDEX IF NOT EXISTS jobs_cluster_shared_user ON job (cluster, shared, hpc_user);
 CREATE INDEX IF NOT EXISTS jobs_cluster_shared_project ON job (cluster, shared, project);
-- Cluster+Shared Filter Sorting
-CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numnodes ON job (cluster, shared, num_nodes);
-CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numhwthreads ON job (cluster, shared, num_hwthreads);
-CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numacc ON job (cluster, shared, num_acc);
-CREATE INDEX IF NOT EXISTS jobs_cluster_shared_energy ON job (cluster, shared, energy);

 -- Cluster+Shared Time Filter Sorting
 CREATE INDEX IF NOT EXISTS jobs_cluster_shared_starttime_duration ON job (cluster, shared, start_time, duration);
 CREATE INDEX IF NOT EXISTS jobs_cluster_shared_duration_starttime ON job (cluster, shared, duration, start_time);

 -- User Filter
-- User Filter Sorting
-CREATE INDEX IF NOT EXISTS jobs_user_numnodes ON job (hpc_user, num_nodes);
-CREATE INDEX IF NOT EXISTS jobs_user_numhwthreads ON job (hpc_user, num_hwthreads);
-CREATE INDEX IF NOT EXISTS jobs_user_numacc ON job (hpc_user, num_acc);
-CREATE INDEX IF NOT EXISTS jobs_user_energy ON job (hpc_user, energy);
-
 -- Cluster+Shared Time Filter Sorting
 CREATE INDEX IF NOT EXISTS jobs_user_starttime_duration ON job (hpc_user, start_time, duration);
 CREATE INDEX IF NOT EXISTS jobs_user_duration_starttime ON job (hpc_user, duration, start_time);

 -- Project Filter
 CREATE INDEX IF NOT EXISTS jobs_project_user ON job (project, hpc_user);
-- Project Filter Sorting
-CREATE INDEX IF NOT EXISTS jobs_project_numnodes ON job (project, num_nodes);
-CREATE INDEX IF NOT EXISTS jobs_project_numhwthreads ON job (project, num_hwthreads);
-CREATE INDEX IF NOT EXISTS jobs_project_numacc ON job (project, num_acc);
-CREATE INDEX IF NOT EXISTS jobs_project_energy ON job (project, energy);

 -- Cluster+Shared Time Filter Sorting
 CREATE INDEX IF NOT EXISTS jobs_project_starttime_duration ON job (project, start_time, duration);
@@ -201,11 +174,6 @@ CREATE INDEX IF NOT EXISTS jobs_project_duration_starttime ON job (project, dura
 -- JobState Filter
 CREATE INDEX IF NOT EXISTS jobs_jobstate_user ON job (job_state, hpc_user);
 CREATE INDEX IF NOT EXISTS jobs_jobstate_project ON job (job_state, project);
-- JobState Filter Sorting
-CREATE INDEX IF NOT EXISTS jobs_jobstate_numnodes ON job (job_state, num_nodes);
-CREATE INDEX IF NOT EXISTS jobs_jobstate_numhwthreads ON job (job_state, num_hwthreads);
-CREATE INDEX IF NOT EXISTS jobs_jobstate_numacc ON job (job_state, num_acc);
-CREATE INDEX IF NOT EXISTS jobs_jobstate_energy ON job (job_state, energy);

 -- Cluster+Shared Time Filter Sorting
 CREATE INDEX IF NOT EXISTS jobs_jobstate_starttime_duration ON job (job_state, start_time, duration);
@@ -214,11 +182,6 @@ CREATE INDEX IF NOT EXISTS jobs_jobstate_duration_starttime ON job (job_state, d
 -- Shared Filter
 CREATE INDEX IF NOT EXISTS jobs_shared_user ON job (shared, hpc_user);
 CREATE INDEX IF NOT EXISTS jobs_shared_project ON job (shared, project);
-- Shared Filter Sorting
-CREATE INDEX IF NOT EXISTS jobs_shared_numnodes ON job (shared, num_nodes);
-CREATE INDEX IF NOT EXISTS jobs_shared_numhwthreads ON job (shared, num_hwthreads);
-CREATE INDEX IF NOT EXISTS jobs_shared_numacc ON job (shared, num_acc);
-CREATE INDEX IF NOT EXISTS jobs_shared_energy ON job (shared, energy);

 -- Cluster+Shared Time Filter Sorting
 CREATE INDEX IF NOT EXISTS jobs_shared_starttime_duration ON job (shared, start_time, duration);
@@ -226,7 +189,6 @@ CREATE INDEX IF NOT EXISTS jobs_shared_duration_starttime ON job (shared, durati

 -- ArrayJob Filter
 CREATE INDEX IF NOT EXISTS jobs_arrayjobid_starttime ON job (array_job_id, start_time);
-CREATE INDEX IF NOT EXISTS jobs_cluster_arrayjobid_starttime ON job (cluster, array_job_id, start_time);

 -- Single filters with default starttime sorting
 CREATE INDEX IF NOT EXISTS jobs_duration_starttime ON job (duration, start_time);
@@ -244,7 +206,6 @@ CREATE INDEX IF NOT EXISTS jobs_energy_duration ON job (energy, duration);

 -- Backup Indices For High Variety Columns
 CREATE INDEX IF NOT EXISTS jobs_starttime ON job (start_time);
-CREATE INDEX IF NOT EXISTS jobs_duration ON job (duration);

 -- Notes:
 -- Cluster+Partition+Jobstate Filter: Tested -> Full Array Of Combinations non-required
@@ -0,0 +1,73 @@
+-- Migration 11 DOWN: Restore indexes from migration 09
+
+-- Drop partial indexes for running jobs
+DROP INDEX IF EXISTS jobs_running_user_stats;
+DROP INDEX IF EXISTS jobs_running_project_stats;
+DROP INDEX IF EXISTS jobs_running_subcluster_stats;
+
+-- Drop covering status indexes, restore 3-col indexes
+DROP INDEX IF EXISTS jobs_cluster_jobstate_user_stats;
+DROP INDEX IF EXISTS jobs_cluster_jobstate_project_stats;
+DROP INDEX IF EXISTS jobs_cluster_jobstate_subcluster_stats;
+
+CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user
+  ON job (cluster, job_state, hpc_user);
+
+CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project
+  ON job (cluster, job_state, project);
+
+-- Drop covering stats indexes
+DROP INDEX IF EXISTS jobs_cluster_user_starttime_stats;
+DROP INDEX IF EXISTS jobs_cluster_project_starttime_stats;
+
+-- Recreate all removed indexes from migration 09
+
+-- Cluster+Partition Filter Sorting
+CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numnodes ON job (cluster, cluster_partition, num_nodes);
+CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numhwthreads ON job (cluster, cluster_partition, num_hwthreads);
+CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numacc ON job (cluster, cluster_partition, num_acc);
+CREATE INDEX IF NOT EXISTS jobs_cluster_partition_energy ON job (cluster, cluster_partition, energy);
+
+-- Cluster+JobState Filter Sorting
+CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numnodes ON job (cluster, job_state, num_nodes);
+CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numhwthreads ON job (cluster, job_state, num_hwthreads);
+CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numacc ON job (cluster, job_state, num_acc);
+CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_energy ON job (cluster, job_state, energy);
+
+-- Cluster+Shared Filter Sorting
+CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numnodes ON job (cluster, shared, num_nodes);
+CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numhwthreads ON job (cluster, shared, num_hwthreads);
+CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numacc ON job (cluster, shared, num_acc);
+CREATE INDEX IF NOT EXISTS jobs_cluster_shared_energy ON job (cluster, shared, energy);
+
+-- User Filter Sorting
+CREATE INDEX IF NOT EXISTS jobs_user_numnodes ON job (hpc_user, num_nodes);
+CREATE INDEX IF NOT EXISTS jobs_user_numhwthreads ON job (hpc_user, num_hwthreads);
+CREATE INDEX IF NOT EXISTS jobs_user_numacc ON job (hpc_user, num_acc);
+CREATE INDEX IF NOT EXISTS jobs_user_energy ON job (hpc_user, energy);
+
+-- Project Filter Sorting
+CREATE INDEX IF NOT EXISTS jobs_project_numnodes ON job (project, num_nodes);
+CREATE INDEX IF NOT EXISTS jobs_project_numhwthreads ON job (project, num_hwthreads);
+CREATE INDEX IF NOT EXISTS jobs_project_numacc ON job (project, num_acc);
+CREATE INDEX IF NOT EXISTS jobs_project_energy ON job (project, energy);
+
+-- JobState Filter Sorting
+CREATE INDEX IF NOT EXISTS jobs_jobstate_numnodes ON job (job_state, num_nodes);
+CREATE INDEX IF NOT EXISTS jobs_jobstate_numhwthreads ON job (job_state, num_hwthreads);
+CREATE INDEX IF NOT EXISTS jobs_jobstate_numacc ON job (job_state, num_acc);
+CREATE INDEX IF NOT EXISTS jobs_jobstate_energy ON job (job_state, energy);
+
+-- Shared Filter Sorting
+CREATE INDEX IF NOT EXISTS jobs_shared_numnodes ON job (shared, num_nodes);
+CREATE INDEX IF NOT EXISTS jobs_shared_numhwthreads ON job (shared, num_hwthreads);
+CREATE INDEX IF NOT EXISTS jobs_shared_numacc ON job (shared, num_acc);
+CREATE INDEX IF NOT EXISTS jobs_shared_energy ON job (shared, energy);
+
+-- ArrayJob Filter
+CREATE INDEX IF NOT EXISTS jobs_cluster_arrayjobid_starttime ON job (cluster, array_job_id, start_time);
+
+-- Backup Indices For High Variety Columns
+CREATE INDEX IF NOT EXISTS jobs_duration ON job (duration);
+
+PRAGMA optimize;
@@ -0,0 +1,110 @@
+-- Migration 11: Optimize job table indexes
+--   - Remove overly specific sorting indexes (reduces ~78 → 48)
+--   - Add covering index for grouped stats queries
+--   - Add covering indexes for status/dashboard queries
+--   - Add partial covering indexes for running jobs (tiny B-tree)
+
+-- ============================================================
+-- Drop SELECTED existing job indexes (from migrations 08/09)
+-- sqlite_autoindex_job_1 (UNIQUE constraint) is kept automatically
+-- ============================================================
+
+-- Cluster+Partition Filter Sorting
+DROP INDEX IF EXISTS jobs_cluster_partition_numnodes;
+DROP INDEX IF EXISTS jobs_cluster_partition_numhwthreads;
+DROP INDEX IF EXISTS jobs_cluster_partition_numacc;
+DROP INDEX IF EXISTS jobs_cluster_partition_energy;
+
+-- Cluster+JobState Filter Sorting
+DROP INDEX IF EXISTS jobs_cluster_jobstate_numnodes;
+DROP INDEX IF EXISTS jobs_cluster_jobstate_numhwthreads;
+DROP INDEX IF EXISTS jobs_cluster_jobstate_numacc;
+DROP INDEX IF EXISTS jobs_cluster_jobstate_energy;
+
+-- Cluster+Shared Filter Sorting
+DROP INDEX IF EXISTS jobs_cluster_shared_numnodes;
+DROP INDEX IF EXISTS jobs_cluster_shared_numhwthreads;
+DROP INDEX IF EXISTS jobs_cluster_shared_numacc;
+DROP INDEX IF EXISTS jobs_cluster_shared_energy;
+
+-- User Filter Sorting
+DROP INDEX IF EXISTS jobs_user_numnodes;
+DROP INDEX IF EXISTS jobs_user_numhwthreads;
+DROP INDEX IF EXISTS jobs_user_numacc;
+DROP INDEX IF EXISTS jobs_user_energy;
+
+-- Project Filter Sorting
+DROP INDEX IF EXISTS jobs_project_numnodes;
+DROP INDEX IF EXISTS jobs_project_numhwthreads;
+DROP INDEX IF EXISTS jobs_project_numacc;
+DROP INDEX IF EXISTS jobs_project_energy;
+
+-- JobState Filter Sorting
+DROP INDEX IF EXISTS jobs_jobstate_numnodes;
+DROP INDEX IF EXISTS jobs_jobstate_numhwthreads;
+DROP INDEX IF EXISTS jobs_jobstate_numacc;
+DROP INDEX IF EXISTS jobs_jobstate_energy;
+
+-- Shared Filter Sorting
+DROP INDEX IF EXISTS jobs_shared_numnodes;
+DROP INDEX IF EXISTS jobs_shared_numhwthreads;
+DROP INDEX IF EXISTS jobs_shared_numacc;
+DROP INDEX IF EXISTS jobs_shared_energy;
+
+-- ArrayJob Filter
+DROP INDEX IF EXISTS jobs_cluster_arrayjobid_starttime;
+
+-- Backup Indices For High Variety Columns
+DROP INDEX IF EXISTS jobs_duration;
+
+-- ============================================================
+-- Covering indexes for grouped stats queries
+-- Column order: cluster (equality), hpc_user/project (GROUP BY), start_time (range scan)
+-- Includes aggregated columns to avoid main table lookups entirely.
+-- ============================================================
+
+CREATE INDEX IF NOT EXISTS jobs_cluster_user_starttime_stats
+  ON job (cluster, hpc_user, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);
+
+CREATE INDEX IF NOT EXISTS jobs_cluster_project_starttime_stats
+  ON job (cluster, project, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);
+
+-- ============================================================
+-- Covering indexes for status/dashboard queries
+-- Column order: cluster (equality), job_state (equality), grouping col, then aggregated columns
+-- These indexes allow the status views to be served entirely from index scans.
+-- ============================================================
+
+-- Drop 3-col indexes that are subsumed by the covering indexes below
+DROP INDEX IF EXISTS jobs_cluster_jobstate_user;
+DROP INDEX IF EXISTS jobs_cluster_jobstate_project;
+
+CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user_stats
+  ON job (cluster, job_state, hpc_user, duration, start_time, num_nodes, num_hwthreads, num_acc);
+
+CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project_stats
+  ON job (cluster, job_state, project, duration, start_time, num_nodes, num_hwthreads, num_acc);
+
+CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_subcluster_stats
+  ON job (cluster, job_state, subcluster, duration, start_time, num_nodes, num_hwthreads, num_acc);
+
+-- ============================================================
+-- Partial covering indexes for running jobs
+-- Only running jobs are in the B-tree, so these indexes are tiny compared to
+-- the full-table indexes above. SQLite uses them when the query contains the
+-- literal `job_state = 'running'` (not a parameter placeholder).
+-- ============================================================
+
+CREATE INDEX IF NOT EXISTS jobs_running_user_stats
+  ON job (cluster, hpc_user, num_nodes, num_hwthreads, num_acc, duration, start_time)
+  WHERE job_state = 'running';
+
+CREATE INDEX IF NOT EXISTS jobs_running_project_stats
+  ON job (cluster, project, num_nodes, num_hwthreads, num_acc, duration, start_time)
+  WHERE job_state = 'running';
+
+CREATE INDEX IF NOT EXISTS jobs_running_subcluster_stats
+  ON job (cluster, subcluster, num_nodes, num_hwthreads, num_acc, duration, start_time)
+  WHERE job_state = 'running';
+
+PRAGMA optimize;
@@ -366,19 +366,23 @@ func (r *NodeRepository) QueryNodes(
 		cclog.Errorf("Error while running query '%s' %v: %v", queryString, queryVars, err)
 		return nil, err
 	}
+	defer rows.Close()

 	nodes := make([]*schema.Node, 0)
 	for rows.Next() {
 		node := schema.Node{}
 		if err := rows.Scan(&node.Hostname, &node.Cluster, &node.SubCluster,
 			&node.NodeState, &node.HealthState); err != nil {
-			rows.Close()
 			cclog.Warn("Error while scanning rows (QueryNodes)")
 			return nil, err
 		}
 		nodes = append(nodes, &node)
 	}

+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+
 	return nodes, nil
 }

@@ -415,6 +419,7 @@ func (r *NodeRepository) QueryNodesWithMeta(
 		cclog.Errorf("Error while running query '%s' %v: %v", queryString, queryVars, err)
 		return nil, err
 	}
+	defer rows.Close()

 	nodes := make([]*schema.Node, 0)
 	for rows.Next() {
@@ -424,7 +429,6 @@ func (r *NodeRepository) QueryNodesWithMeta(

 		if err := rows.Scan(&node.Hostname, &node.Cluster, &node.SubCluster,
 			&node.NodeState, &node.HealthState, &RawMetaData, &RawMetricHealth); err != nil {
-			rows.Close()
 			cclog.Warn("Error while scanning rows (QueryNodes)")
 			return nil, err
 		}
@@ -454,6 +458,10 @@ func (r *NodeRepository) QueryNodesWithMeta(
 		nodes = append(nodes, &node)
 	}

+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+
 	return nodes, nil
 }

@@ -545,10 +553,11 @@ func (r *NodeRepository) MapNodes(cluster string) (map[string]string, error) {

 func (r *NodeRepository) CountStates(ctx context.Context, filters []*model.NodeFilter, column string) ([]*model.NodeStates, error) {
 	query, qerr := AccessCheck(ctx,
-		sq.Select(column).
+		sq.Select(column, "COUNT(*) as count").
 			From("node").
 			Join("node_state ON node_state.node_id = node.id").
-			Where(latestStateCondition()))
+			Where(latestStateCondition()).
+			GroupBy(column))
 	if qerr != nil {
 		return nil, qerr
 	}
@@ -561,23 +570,21 @@ func (r *NodeRepository) CountStates(ctx context.Context, filters []*model.NodeF
 		cclog.Errorf("Error while running query '%s' %v: %v", queryString, queryVars, err)
 		return nil, err
 	}
+	defer rows.Close()

-	stateMap := map[string]int{}
+	nodes := make([]*model.NodeStates, 0)
 	for rows.Next() {
 		var state string
-		if err := rows.Scan(&state); err != nil {
-			rows.Close()
+		var count int
+		if err := rows.Scan(&state, &count); err != nil {
 			cclog.Warn("Error while scanning rows (CountStates)")
 			return nil, err
 		}
-
-		stateMap[state] += 1
+		nodes = append(nodes, &model.NodeStates{State: state, Count: count})
 	}

-	nodes := make([]*model.NodeStates, 0)
-	for state, counts := range stateMap {
-		node := model.NodeStates{State: state, Count: counts}
-		nodes = append(nodes, &node)
+	if err := rows.Err(); err != nil {
+		return nil, err
 	}

 	return nodes, nil
@@ -623,6 +630,7 @@ func (r *NodeRepository) CountStatesTimed(ctx context.Context, filters []*model.
 		cclog.Errorf("Error while running query '%s' %v: %v", queryString, queryVars, err)
 		return nil, err
 	}
+	defer rows.Close()

 	rawData := make(map[string][][]int)
 	for rows.Next() {
@@ -630,7 +638,6 @@ func (r *NodeRepository) CountStatesTimed(ctx context.Context, filters []*model.
 		var timestamp, count int

 		if err := rows.Scan(&state, &timestamp, &count); err != nil {
-			rows.Close()
 			cclog.Warnf("Error while scanning rows (CountStatesTimed) at time '%d'", timestamp)
 			return nil, err
 		}
@@ -643,6 +650,10 @@ func (r *NodeRepository) CountStatesTimed(ctx context.Context, filters []*model.
 		rawData[state][1] = append(rawData[state][1], count)
 	}

+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+
 	timedStates := make([]*model.NodeStatesTimed, 0)
 	for state, data := range rawData {
 		entry := model.NodeStatesTimed{State: state, Times: data[0], Counts: data[1]}
@@ -55,6 +55,7 @@ import (
 	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
 	"github.com/ClusterCockpit/cc-lib/v2/schema"
 	sq "github.com/Masterminds/squirrel"
+	"golang.org/x/sync/errgroup"
 )

 // groupBy2column maps GraphQL Aggregate enum values to their corresponding database column names.
@@ -105,14 +106,14 @@ func (r *JobRepository) buildCountQuery(
 	var query sq.SelectBuilder

 	if col != "" {
-		query = sq.Select(col, "COUNT(job.id)").From("job").GroupBy(col)
+		query = sq.Select(col, "COUNT(*)").From("job").GroupBy(col)
 	} else {
-		query = sq.Select("COUNT(job.id)").From("job")
+		query = sq.Select("COUNT(*)").From("job")
 	}

 	switch kind {
 	case "running":
-		query = query.Where("job.job_state = ?", "running")
+		query = query.Where("job.job_state = 'running'")
 	case "short":
 		query = query.Where("job.duration < ?", config.Keys.ShortRunningJobsDuration)
 	}
@@ -124,59 +125,100 @@ func (r *JobRepository) buildCountQuery(
 	return query
 }

-// buildStatsQuery constructs a SQL query to compute comprehensive job statistics with optional grouping.
+// buildStatsQuery constructs a SQL query to compute job statistics with optional grouping.
+// Only requested columns are computed; unrequested columns select 0 as placeholder.
 //
 // Parameters:
 //   - filter: Job filters to apply (cluster, user, time range, etc.)
 //   - col: Column name to GROUP BY; empty string for overall statistics without grouping
-//
-// Returns a SelectBuilder that produces comprehensive statistics:
-//   - totalJobs: Count of jobs
-//   - totalUsers: Count of distinct users (always 0 when grouping by user)
-//   - totalWalltime: Sum of job durations in hours
-//   - totalNodes: Sum of nodes used across all jobs
-//   - totalNodeHours: Sum of (duration × num_nodes) in hours
-//   - totalCores: Sum of hardware threads used across all jobs
-//   - totalCoreHours: Sum of (duration × num_hwthreads) in hours
-//   - totalAccs: Sum of accelerators used across all jobs
-//   - totalAccHours: Sum of (duration × num_acc) in hours
-//
-// Special handling:
-//   - Running jobs: Duration calculated as (now - start_time) instead of stored duration
-//   - Grouped queries: Also select grouping column and user's display name from hpc_user table
-//   - All time values converted from seconds to hours (÷ 3600) and rounded
+//   - shortThreshold: Duration threshold in seconds for counting short-running jobs
+//   - reqFields: Set of requested field names; nil means compute all fields
 func (r *JobRepository) buildStatsQuery(
 	filter []*model.JobFilter,
 	col string,
+	shortThreshold int,
+	reqFields map[string]bool,
 ) sq.SelectBuilder {
-	var query sq.SelectBuilder
+	now := time.Now().Unix()
+
+	// Helper: return real expression if field is requested (or reqFields is nil), else "0 as alias"
+	need := func(field string) bool {
+		return reqFields == nil || reqFields[field]
+	}
+	durationExpr := fmt.Sprintf(`(CASE WHEN job.job_state = 'running' THEN %d - job.start_time ELSE job.duration END)`, now)
+
+	// Build column list
+	columns := make([]string, 0, 14)

 	if col != "" {
-		query = sq.Select(
-			col,
-			"name",
-			"COUNT(job.id) as totalJobs",
-			"COUNT(DISTINCT job.hpc_user) AS totalUsers",
-			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END)) / 3600) as int) as totalWalltime`, time.Now().Unix()),
-			`CAST(SUM(job.num_nodes) as int) as totalNodes`,
-			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) * job.num_nodes) / 3600) as int) as totalNodeHours`, time.Now().Unix()),
-			`CAST(SUM(job.num_hwthreads) as int) as totalCores`,
-			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) * job.num_hwthreads) / 3600) as int) as totalCoreHours`, time.Now().Unix()),
-			`CAST(SUM(job.num_acc) as int) as totalAccs`,
-			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) * job.num_acc) / 3600) as int) as totalAccHours`, time.Now().Unix()),
-		).From("job").LeftJoin("hpc_user ON hpc_user.username = job.hpc_user").GroupBy(col)
+		columns = append(columns, col)
+	}
+
+	columns = append(columns, "COUNT(*) as totalJobs")
+
+	if need("totalUsers") && col != "job.hpc_user" {
+		columns = append(columns, "COUNT(DISTINCT job.hpc_user) AS totalUsers")
 	} else {
-		query = sq.Select(
-			"COUNT(job.id) as totalJobs",
-			"COUNT(DISTINCT job.hpc_user) AS totalUsers",
-			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END)) / 3600) as int)`, time.Now().Unix()),
-			`CAST(SUM(job.num_nodes) as int)`,
-			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) * job.num_nodes) / 3600) as int)`, time.Now().Unix()),
-			`CAST(SUM(job.num_hwthreads) as int)`,
-			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) * job.num_hwthreads) / 3600) as int)`, time.Now().Unix()),
-			`CAST(SUM(job.num_acc) as int)`,
-			fmt.Sprintf(`CAST(ROUND(SUM((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) * job.num_acc) / 3600) as int)`, time.Now().Unix()),
-		).From("job")
+		columns = append(columns, "0 AS totalUsers")
+	}
+
+	if need("totalWalltime") {
+		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s) / 3600) as int) as totalWalltime`, durationExpr))
+	} else {
+		columns = append(columns, "0 as totalWalltime")
+	}
+
+	if need("totalNodes") {
+		columns = append(columns, `CAST(SUM(job.num_nodes) as int) as totalNodes`)
+	} else {
+		columns = append(columns, "0 as totalNodes")
+	}
+
+	if need("totalNodeHours") {
+		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s * job.num_nodes) / 3600) as int) as totalNodeHours`, durationExpr))
+	} else {
+		columns = append(columns, "0 as totalNodeHours")
+	}
+
+	if need("totalCores") {
+		columns = append(columns, `CAST(SUM(job.num_hwthreads) as int) as totalCores`)
+	} else {
+		columns = append(columns, "0 as totalCores")
+	}
+
+	if need("totalCoreHours") {
+		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s * job.num_hwthreads) / 3600) as int) as totalCoreHours`, durationExpr))
+	} else {
+		columns = append(columns, "0 as totalCoreHours")
+	}
+
+	if need("totalAccs") {
+		columns = append(columns, `CAST(SUM(job.num_acc) as int) as totalAccs`)
+	} else {
+		columns = append(columns, "0 as totalAccs")
+	}
+
+	if need("totalAccHours") {
+		columns = append(columns, fmt.Sprintf(`CAST(ROUND(SUM(%s * job.num_acc) / 3600) as int) as totalAccHours`, durationExpr))
+	} else {
+		columns = append(columns, "0 as totalAccHours")
+	}
+
+	if need("runningJobs") {
+		columns = append(columns, `COUNT(CASE WHEN job.job_state = 'running' THEN 1 END) as runningJobs`)
+	} else {
+		columns = append(columns, "0 as runningJobs")
+	}
+
+	if need("shortJobs") {
+		columns = append(columns, fmt.Sprintf(`COUNT(CASE WHEN job.duration < %d THEN 1 END) as shortJobs`, shortThreshold))
+	} else {
+		columns = append(columns, "0 as shortJobs")
+	}
+
+	query := sq.Select(columns...).From("job")
+	if col != "" {
+		query = query.GroupBy(col)
 	}

 	for _, f := range filter {
@@ -186,35 +228,19 @@ func (r *JobRepository) buildStatsQuery(
 	return query
 }

-// JobsStatsGrouped computes comprehensive job statistics grouped by a dimension (user, project, cluster, or subcluster).
-//
-// This is the primary method for generating aggregated statistics views in the UI, providing
-// metrics like total jobs, walltime, and resource usage broken down by the specified grouping.
-//
-// Parameters:
-//   - ctx: Context for security checks and cancellation
-//   - filter: Filters to apply (time range, cluster, job state, etc.)
-//   - page: Optional pagination (ItemsPerPage: -1 disables pagination)
-//   - sortBy: Optional sort column (totalJobs, totalWalltime, totalCoreHours, etc.)
-//   - groupBy: Required grouping dimension (User, Project, Cluster, or SubCluster)
-//
-// Returns a slice of JobsStatistics, one per group, with:
-//   - ID: The group identifier (username, project name, cluster name, etc.)
-//   - Name: Display name (for users, from hpc_user.name; empty for other groups)
-//   - Statistics: totalJobs, totalUsers, totalWalltime, resource usage metrics
-//
-// Security: Respects user roles via SecurityCheck - users see only their own data unless admin/support.
-// Performance: Results are sorted in SQL and pagination applied before scanning rows.
+// JobsStatsGrouped computes job statistics grouped by a dimension (user, project, cluster, or subcluster).
+// Only columns listed in reqFields are computed; others return 0. User display names are looked up
+// in a separate lightweight query to avoid JOIN overhead on the main aggregation.
 func (r *JobRepository) JobsStatsGrouped(
 	ctx context.Context,
 	filter []*model.JobFilter,
 	page *model.PageRequest,
 	sortBy *model.SortByAggregate,
 	groupBy *model.Aggregate,
+	reqFields map[string]bool,
 ) ([]*model.JobsStatistics, error) {
-	start := time.Now()
 	col := groupBy2column[*groupBy]
-	query := r.buildStatsQuery(filter, col)
+	query := r.buildStatsQuery(filter, col, config.Keys.ShortRunningJobsDuration, reqFields)

 	query, err := SecurityCheck(ctx, query)
 	if err != nil {
@@ -230,97 +256,75 @@ func (r *JobRepository) JobsStatsGrouped(
 		query = query.Offset((uint64(page.Page) - 1) * limit).Limit(limit)
 	}

-	rows, err := query.RunWith(r.DB).Query()
+	rows, err := query.RunWith(r.DB).QueryContext(ctx)
 	if err != nil {
 		cclog.Warn("Error while querying DB for job statistics")
 		return nil, err
 	}
+	defer rows.Close()

 	stats := make([]*model.JobsStatistics, 0, 100)

 	for rows.Next() {
 		var id sql.NullString
-		var name sql.NullString
-		var jobs, users, walltime, nodes, nodeHours, cores, coreHours, accs, accHours sql.NullInt64
-		if err := rows.Scan(&id, &name, &jobs, &users, &walltime, &nodes, &nodeHours, &cores, &coreHours, &accs, &accHours); err != nil {
+		var jobs, users, walltime, nodes, nodeHours, cores, coreHours, accs, accHours, runningJobs, shortJobs sql.NullInt64
+		if err := rows.Scan(&id, &jobs, &users, &walltime, &nodes, &nodeHours, &cores, &coreHours, &accs, &accHours, &runningJobs, &shortJobs); err != nil {
 			cclog.Warnf("Error while scanning rows: %s", err.Error())
 			return nil, err
 		}

 		if id.Valid {
-			var totalJobs, totalUsers, totalWalltime, totalNodes, totalNodeHours, totalCores, totalCoreHours, totalAccs, totalAccHours int
-			var personName string
-
-			if name.Valid {
-				personName = name.String
-			}
-
-			if jobs.Valid {
-				totalJobs = int(jobs.Int64)
-			}
-
-			if users.Valid {
-				totalUsers = int(users.Int64)
-			}
-
-			if walltime.Valid {
-				totalWalltime = int(walltime.Int64)
-			}
-
-			if nodes.Valid {
-				totalNodes = int(nodes.Int64)
-			}
-			if cores.Valid {
-				totalCores = int(cores.Int64)
-			}
-			if accs.Valid {
-				totalAccs = int(accs.Int64)
-			}
-
-			if nodeHours.Valid {
-				totalNodeHours = int(nodeHours.Int64)
-			}
-			if coreHours.Valid {
-				totalCoreHours = int(coreHours.Int64)
-			}
-			if accHours.Valid {
-				totalAccHours = int(accHours.Int64)
-			}
-
-			if col == "job.hpc_user" {
-				// name := r.getUserName(ctx, id.String)
 			stats = append(stats,
 				&model.JobsStatistics{
 					ID:             id.String,
-						Name:           personName,
-						TotalJobs:      totalJobs,
-						TotalWalltime:  totalWalltime,
-						TotalNodes:     totalNodes,
-						TotalNodeHours: totalNodeHours,
-						TotalCores:     totalCores,
-						TotalCoreHours: totalCoreHours,
-						TotalAccs:      totalAccs,
-						TotalAccHours:  totalAccHours,
+					TotalJobs:      int(jobs.Int64),
+					TotalUsers:     int(users.Int64),
+					TotalWalltime:  int(walltime.Int64),
+					TotalNodes:     int(nodes.Int64),
+					TotalNodeHours: int(nodeHours.Int64),
+					TotalCores:     int(cores.Int64),
+					TotalCoreHours: int(coreHours.Int64),
+					TotalAccs:      int(accs.Int64),
+					TotalAccHours:  int(accHours.Int64),
+					RunningJobs:    int(runningJobs.Int64),
+					ShortJobs:      int(shortJobs.Int64),
 				})
+		}
+	}
+
+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+
+	// Post-query name lookup for user grouping (avoids LEFT JOIN on aggregation query)
+	if col == "job.hpc_user" && len(stats) > 0 {
+		usernames := make([]any, len(stats))
+		for i, s := range stats {
+			usernames[i] = s.ID
+		}
+
+		nameQuery := sq.Select("username", "name").From("hpc_user").Where(sq.Eq{"username": usernames})
+		nameRows, err := nameQuery.RunWith(r.DB).QueryContext(ctx)
+		if err != nil {
+			cclog.Warnf("Error looking up user names: %s", err.Error())
+			// Non-fatal: stats are still valid without display names
 		} else {
-				stats = append(stats,
-					&model.JobsStatistics{
-						ID:             id.String,
-						TotalJobs:      totalJobs,
-						TotalUsers:     totalUsers,
-						TotalWalltime:  totalWalltime,
-						TotalNodes:     totalNodes,
-						TotalNodeHours: totalNodeHours,
-						TotalCores:     totalCores,
-						TotalCoreHours: totalCoreHours,
-						TotalAccs:      totalAccs,
-						TotalAccHours:  totalAccHours,
-					})
+			defer nameRows.Close()
+			nameMap := make(map[string]string, len(stats))
+			for nameRows.Next() {
+				var username, name string
+				if err := nameRows.Scan(&username, &name); err == nil {
+					nameMap[username] = name
+				}
+			}
+			for _, s := range stats {
+				if name, ok := nameMap[s.ID]; ok {
+					s.Name = name
+				}
 			}
 		}
 	}

-	cclog.Debugf("Timer JobsStatsGrouped %s", time.Since(start))
 	return stats, nil
 }

@@ -342,19 +346,20 @@ func (r *JobRepository) JobsStatsGrouped(
 func (r *JobRepository) JobsStats(
 	ctx context.Context,
 	filter []*model.JobFilter,
+	reqFields map[string]bool,
 ) ([]*model.JobsStatistics, error) {
 	start := time.Now()
-	query := r.buildStatsQuery(filter, "")
+	query := r.buildStatsQuery(filter, "", config.Keys.ShortRunningJobsDuration, reqFields)
 	query, err := SecurityCheck(ctx, query)
 	if err != nil {
 		return nil, err
 	}

-	row := query.RunWith(r.DB).QueryRow()
+	row := query.RunWith(r.DB).QueryRowContext(ctx)
 	stats := make([]*model.JobsStatistics, 0, 1)

-	var jobs, users, walltime, nodes, nodeHours, cores, coreHours, accs, accHours sql.NullInt64
-	if err := row.Scan(&jobs, &users, &walltime, &nodes, &nodeHours, &cores, &coreHours, &accs, &accHours); err != nil {
+	var jobs, users, walltime, nodes, nodeHours, cores, coreHours, accs, accHours, runningJobs, shortJobs sql.NullInt64
+	if err := row.Scan(&jobs, &users, &walltime, &nodes, &nodeHours, &cores, &coreHours, &accs, &accHours, &runningJobs, &shortJobs); err != nil {
 		cclog.Warn("Error while scanning rows")
 		return nil, err
 	}
@@ -379,6 +384,8 @@ func (r *JobRepository) JobsStats(
 				TotalNodeHours: totalNodeHours,
 				TotalCoreHours: totalCoreHours,
 				TotalAccHours:  totalAccHours,
+				RunningJobs:    int(runningJobs.Int64),
+				ShortJobs:      int(shortJobs.Int64),
 			})
 	}

@@ -435,11 +442,12 @@ func (r *JobRepository) JobCountGrouped(
 	if err != nil {
 		return nil, err
 	}
-	rows, err := query.RunWith(r.DB).Query()
+	rows, err := query.RunWith(r.DB).QueryContext(ctx)
 	if err != nil {
 		cclog.Warn("Error while querying DB for job statistics")
 		return nil, err
 	}
+	defer rows.Close()

 	stats := make([]*model.JobsStatistics, 0, 100)

@@ -459,6 +467,10 @@ func (r *JobRepository) JobCountGrouped(
 		}
 	}

+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+
 	cclog.Debugf("Timer JobCountGrouped %s", time.Since(start))
 	return stats, nil
 }
@@ -491,11 +503,12 @@ func (r *JobRepository) AddJobCountGrouped(
 	if err != nil {
 		return nil, err
 	}
-	rows, err := query.RunWith(r.DB).Query()
+	rows, err := query.RunWith(r.DB).QueryContext(ctx)
 	if err != nil {
 		cclog.Warn("Error while querying DB for job statistics")
 		return nil, err
 	}
+	defer rows.Close()

 	counts := make(map[string]int)

@@ -511,6 +524,10 @@ func (r *JobRepository) AddJobCountGrouped(
 		}
 	}

+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+
 	switch kind {
 	case "running":
 		for _, s := range stats {
@@ -550,23 +567,13 @@ func (r *JobRepository) AddJobCount(
 	if err != nil {
 		return nil, err
 	}
-	rows, err := query.RunWith(r.DB).Query()
-	if err != nil {
-		cclog.Warn("Error while querying DB for job statistics")
-		return nil, err
-	}
-
-	var count int
-
-	for rows.Next() {
 	var cnt sql.NullInt64
-		if err := rows.Scan(&cnt); err != nil {
-			cclog.Warn("Error while scanning rows")
+	if err := query.RunWith(r.DB).QueryRowContext(ctx).Scan(&cnt); err != nil {
+		cclog.Warn("Error while querying DB for job count")
 		return nil, err
 	}

-		count = int(cnt.Int64)
-	}
+	count := int(cnt.Int64)

 	switch kind {
 	case "running":
@@ -634,30 +641,45 @@ func (r *JobRepository) AddHistograms(
 		targetBinSize = 3600
 	}

+	// Run all 4 histogram queries in parallel — each writes a distinct struct field.
+	g, gctx := errgroup.WithContext(ctx)
+
+	value := fmt.Sprintf(`CAST(ROUND(((CASE WHEN job.job_state = 'running' THEN %d - job.start_time ELSE job.duration END) / %d) + 1) as int) as value`, time.Now().Unix(), targetBinSize)
+
+	g.Go(func() error {
 		var err error
-	// Return X-Values always as seconds, will be formatted into minutes and hours in frontend
-	value := fmt.Sprintf(`CAST(ROUND(((CASE WHEN job.job_state = "running" THEN %d - job.start_time ELSE job.duration END) / %d) + 1) as int) as value`, time.Now().Unix(), targetBinSize)
-	stat.HistDuration, err = r.jobsDurationStatisticsHistogram(ctx, value, filter, targetBinSize, &targetBinCount)
+		stat.HistDuration, err = r.jobsDurationStatisticsHistogram(gctx, value, filter, targetBinSize, &targetBinCount)
 		if err != nil {
 			cclog.Warn("Error while loading job statistics histogram: job duration")
-		return nil, err
 		}
-
-	stat.HistNumNodes, err = r.jobsStatisticsHistogram(ctx, "job.num_nodes as value", filter)
+		return err
+	})
+	g.Go(func() error {
+		var err error
+		stat.HistNumNodes, err = r.jobsStatisticsHistogram(gctx, "job.num_nodes as value", filter)
 		if err != nil {
 			cclog.Warn("Error while loading job statistics histogram: num nodes")
-		return nil, err
 		}
-
-	stat.HistNumCores, err = r.jobsStatisticsHistogram(ctx, "job.num_hwthreads as value", filter)
+		return err
+	})
+	g.Go(func() error {
+		var err error
+		stat.HistNumCores, err = r.jobsStatisticsHistogram(gctx, "job.num_hwthreads as value", filter)
 		if err != nil {
 			cclog.Warn("Error while loading job statistics histogram: num hwthreads")
-		return nil, err
 		}
-
-	stat.HistNumAccs, err = r.jobsStatisticsHistogram(ctx, "job.num_acc as value", filter)
+		return err
+	})
+	g.Go(func() error {
+		var err error
+		stat.HistNumAccs, err = r.jobsStatisticsHistogram(gctx, "job.num_acc as value", filter)
 		if err != nil {
 			cclog.Warn("Error while loading job statistics histogram: num acc")
+		}
+		return err
+	})
+
+	if err := g.Wait(); err != nil {
 		return nil, err
 	}

@@ -740,7 +762,7 @@ func (r *JobRepository) jobsStatisticsHistogram(
 ) ([]*model.HistoPoint, error) {
 	start := time.Now()
 	query, qerr := SecurityCheck(ctx,
-		sq.Select(value, "COUNT(job.id) AS count").From("job"))
+		sq.Select(value, "COUNT(*) AS count").From("job"))

 	if qerr != nil {
 		return nil, qerr
@@ -750,11 +772,12 @@ func (r *JobRepository) jobsStatisticsHistogram(
 		query = BuildWhereClause(f, query)
 	}

-	rows, err := query.GroupBy("value").RunWith(r.DB).Query()
+	rows, err := query.GroupBy("value").RunWith(r.DB).QueryContext(ctx)
 	if err != nil {
 		cclog.Error("Error while running query")
 		return nil, err
 	}
+	defer rows.Close()

 	points := make([]*model.HistoPoint, 0)
 	// is it possible to introduce zero values here? requires info about bincount
@@ -767,6 +790,11 @@ func (r *JobRepository) jobsStatisticsHistogram(

 		points = append(points, &point)
 	}
+
+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+
 	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
 	return points, nil
 }
@@ -800,7 +828,7 @@ func (r *JobRepository) jobsDurationStatisticsHistogram(
 ) ([]*model.HistoPoint, error) {
 	start := time.Now()
 	query, qerr := SecurityCheck(ctx,
-		sq.Select(value, "COUNT(job.id) AS count").From("job"))
+		sq.Select(value, "COUNT(*) AS count").From("job"))

 	if qerr != nil {
 		return nil, qerr
@@ -808,24 +836,31 @@ func (r *JobRepository) jobsDurationStatisticsHistogram(

 	// Each bin represents a duration range: bin N = [N*binSizeSeconds, (N+1)*binSizeSeconds)
 	// Example: binSizeSeconds=3600 (1 hour), bin 1 = 0-1h, bin 2 = 1-2h, etc.
-	points := make([]*model.HistoPoint, 0)
+	points := make([]*model.HistoPoint, 0, *targetBinCount)
 	for i := 1; i <= *targetBinCount; i++ {
 		point := model.HistoPoint{Value: i * binSizeSeconds, Count: 0}
 		points = append(points, &point)
 	}

+	// Build a map from bin value (seconds) to slice index for O(1) lookup.
+	binMap := make(map[int]int, len(points))
+	for i, p := range points {
+		binMap[p.Value] = i
+	}
+
 	for _, f := range filters {
 		query = BuildWhereClause(f, query)
 	}

-	rows, err := query.GroupBy("value").RunWith(r.DB).Query()
+	rows, err := query.GroupBy("value").RunWith(r.DB).QueryContext(ctx)
 	if err != nil {
 		cclog.Error("Error while running query")
 		return nil, err
 	}
+	defer rows.Close()

-	// Match query results to pre-initialized bins.
-	// point.Value from query is the bin number; multiply by binSizeSeconds to match bin.Value.
+	// Match query results to pre-initialized bins using map lookup.
+	// point.Value from query is the bin number; multiply by binSizeSeconds to get bin key.
 	for rows.Next() {
 		point := model.HistoPoint{}
 		if err := rows.Scan(&point.Value, &point.Count); err != nil {
@@ -833,12 +868,13 @@ func (r *JobRepository) jobsDurationStatisticsHistogram(
 			return nil, err
 		}

-		for _, e := range points {
-			if e.Value == (point.Value * binSizeSeconds) {
-				e.Count = point.Count
-				break
+		if idx, ok := binMap[point.Value*binSizeSeconds]; ok {
+			points[idx].Count = point.Count
 		}
 	}
+
+	if err := rows.Err(); err != nil {
+		return nil, err
 	}

 	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
@@ -943,24 +979,34 @@ func (r *JobRepository) jobsMetricStatisticsHistogram(

 	mainQuery = mainQuery.GroupBy("bin").OrderBy("bin")

-	rows, err := mainQuery.RunWith(r.DB).Query()
+	rows, err := mainQuery.RunWith(r.DB).QueryContext(ctx)
 	if err != nil {
 		cclog.Errorf("Error while running mainQuery: %s", err)
 		return nil, err
 	}
+	defer rows.Close()

 	// Pre-initialize bins with calculated min/max ranges.
 	// Example: peak=1000, bins=10 -> bin 1=[0,100), bin 2=[100,200), ..., bin 10=[900,1000]
-	points := make([]*model.MetricHistoPoint, 0)
+	points := make([]*model.MetricHistoPoint, 0, *bins)
 	binStep := int(peak) / *bins
 	for i := 1; i <= *bins; i++ {
 		binMin := (binStep * (i - 1))
 		binMax := (binStep * i)
-		epoint := model.MetricHistoPoint{Bin: &i, Count: 0, Min: &binMin, Max: &binMax}
+		idx := i
+		epoint := model.MetricHistoPoint{Bin: &idx, Count: 0, Min: &binMin, Max: &binMax}
 		points = append(points, &epoint)
 	}

-	// Match query results to pre-initialized bins.
+	// Build a map from bin number to slice index for O(1) lookup.
+	binMap := make(map[int]int, len(points))
+	for i, p := range points {
+		if p.Bin != nil {
+			binMap[*p.Bin] = i
+		}
+	}
+
+	// Match query results to pre-initialized bins using map lookup.
 	for rows.Next() {
 		rpoint := model.MetricHistoPoint{}
 		if err := rows.Scan(&rpoint.Bin, &rpoint.Count); err != nil {
@@ -968,14 +1014,17 @@ func (r *JobRepository) jobsMetricStatisticsHistogram(
 			return nil, err
 		}

-		for _, e := range points {
-			if e.Bin != nil && rpoint.Bin != nil && *e.Bin == *rpoint.Bin {
-				e.Count = rpoint.Count
-				break
+		if rpoint.Bin != nil {
+			if idx, ok := binMap[*rpoint.Bin]; ok {
+				points[idx].Count = rpoint.Count
 			}
 		}
 	}

+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+
 	result := model.MetricHistoPoints{Metric: metric, Unit: unit, Stat: &footprintStat, Data: points}

 	cclog.Debugf("Timer jobsStatisticsHistogram %s", time.Since(start))
@@ -14,7 +14,7 @@ import (

 func TestBuildJobStatsQuery(t *testing.T) {
 	r := setup(t)
-	q := r.buildStatsQuery(nil, "USER")
+	q := r.buildStatsQuery(nil, "USER", 300, nil)

 	sql, _, err := q.ToSql()
 	noErr(t, err)
@@ -29,7 +29,7 @@ func TestJobStats(t *testing.T) {
 	err := r.DB.QueryRow(`SELECT COUNT(*) FROM job`).Scan(&expectedCount)
 	noErr(t, err)

-	stats, err := r.JobsStats(getContext(t), []*model.JobFilter{})
+	stats, err := r.JobsStats(getContext(t), []*model.JobFilter{}, nil)
 	noErr(t, err)

 	if stats[0].TotalJobs != expectedCount {
@@ -283,6 +283,7 @@ func (r *JobRepository) CountTags(user *schema.User) (tags []schema.Tag, counts
 	if err != nil {
 		return nil, nil, err
 	}
+	defer xrows.Close()

 	for xrows.Next() {
 		var t schema.Tag
@@ -300,6 +301,10 @@ func (r *JobRepository) CountTags(user *schema.User) (tags []schema.Tag, counts
 		}
 	}

+	if err := xrows.Err(); err != nil {
+		return nil, nil, err
+	}
+
 	// Query and Count Jobs with attached Tags
 	q := sq.Select("t.tag_type, t.tag_name, t.id, count(jt.tag_id)").
 		From("tag t").
@@ -334,6 +339,7 @@ func (r *JobRepository) CountTags(user *schema.User) (tags []schema.Tag, counts
 	if err != nil {
 		return nil, nil, err
 	}
+	defer rows.Close()

 	counts = make(map[string]int)
 	for rows.Next() {
@@ -511,6 +517,7 @@ func (r *JobRepository) GetTags(user *schema.User, job *int64) ([]*schema.Tag, e
 		cclog.Errorf("Error get tags with %s: %v", s, err)
 		return nil, err
 	}
+	defer rows.Close()

 	tags := make([]*schema.Tag, 0)
 	for rows.Next() {
@@ -529,6 +536,10 @@ func (r *JobRepository) GetTags(user *schema.User, job *int64) ([]*schema.Tag, e
 		}
 	}

+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+
 	return tags, nil
 }

@@ -544,6 +555,7 @@ func (r *JobRepository) GetTagsDirect(job *int64) ([]*schema.Tag, error) {
 		cclog.Errorf("Error get tags with %s: %v", s, err)
 		return nil, err
 	}
+	defer rows.Close()

 	tags := make([]*schema.Tag, 0)
 	for rows.Next() {
@@ -555,6 +567,10 @@ func (r *JobRepository) GetTagsDirect(job *int64) ([]*schema.Tag, error) {
 		tags = append(tags, tag)
 	}

+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+
 	return tags, nil
 }

@@ -582,6 +598,7 @@ func (r *JobRepository) getArchiveTags(job *int64) ([]*schema.Tag, error) {
 		cclog.Errorf("Error get tags with %s: %v", s, err)
 		return nil, err
 	}
+	defer rows.Close()

 	tags := make([]*schema.Tag, 0)
 	for rows.Next() {
@@ -593,6 +610,10 @@ func (r *JobRepository) getArchiveTags(job *int64) ([]*schema.Tag, error) {
 		tags = append(tags, tag)
 	}

+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+
 	return tags, nil
 }

@@ -102,6 +102,7 @@ func (r *UserRepository) GetLdapUsernames() ([]string, error) {
 		cclog.Warn("Error while querying usernames")
 		return nil, err
 	}
+	defer rows.Close()

 	for rows.Next() {
 		var username string
@@ -10,6 +10,7 @@ import (
 	"encoding/json"
 	"fmt"
 	"maps"
+	"math"
 	"os"
 	"path/filepath"
 	"strings"
@@ -111,6 +112,29 @@ type JobClassTagger struct {
 	getMetricConfig func(cluster, subCluster string) map[string]*schema.Metric
 }

+// roundEnv returns a copy of env with all float64 values rounded to 2 decimal places.
+// Nested map[string]any and map[string]float64 values are recursed into.
+func roundEnv(env map[string]any) map[string]any {
+	rounded := make(map[string]any, len(env))
+	for k, v := range env {
+		switch val := v.(type) {
+		case float64:
+			rounded[k] = math.Round(val*100) / 100
+		case map[string]any:
+			rounded[k] = roundEnv(val)
+		case map[string]float64:
+			rm := make(map[string]float64, len(val))
+			for mk, mv := range val {
+				rm[mk] = math.Round(mv*100) / 100
+			}
+			rounded[k] = rm
+		default:
+			rounded[k] = v
+		}
+	}
+	return rounded
+}
+
 func (t *JobClassTagger) prepareRule(b []byte, fns string) {
 	var rule RuleFormat
 	if err := json.NewDecoder(bytes.NewReader(b)).Decode(&rule); err != nil {
@@ -408,7 +432,7 @@ func (t *JobClassTagger) Match(job *schema.Job) {

 			// process hint template
 			var msg bytes.Buffer
-			if err := ri.hint.Execute(&msg, env); err != nil {
+			if err := ri.hint.Execute(&msg, roundEnv(env)); err != nil {
 				cclog.Errorf("Template error: %s", err.Error())
 				continue
 			}
@@ -127,7 +127,7 @@ func (data *APIMetricData) AddStats() {
 // This is commonly used for unit conversion (e.g., bytes to gigabytes).
 // Scaling by 0 or 1 is a no-op for performance reasons.
 func (data *APIMetricData) ScaleBy(f schema.Float) {
-	if f == 0 || f == 1 {
+	if f == 1 {
 		return
 	}

@@ -408,3 +408,12 @@ func (b *buffer) count() int64 {
 	}
 	return res
 }
+
+// bufferCount returns the number of buffer nodes in the linked list.
+func (b *buffer) bufferCount() int64 {
+	var n int64
+	for ; b != nil; b = b.prev {
+		n++
+	}
+	return n
+}
@@ -100,8 +100,15 @@ func Checkpointing(wg *sync.WaitGroup, ctx context.Context) {

 	wg.Go(func() {

-		const checkpointInterval = 12 * time.Hour
-		d := checkpointInterval
+		d := 12 * time.Hour // default checkpoint interval
+		if Keys.CheckpointInterval != "" {
+			parsed, err := time.ParseDuration(Keys.CheckpointInterval)
+			if err != nil {
+				cclog.Errorf("[METRICSTORE]> invalid checkpoint-interval %q: %s, using default 12h", Keys.CheckpointInterval, err)
+			} else {
+				d = parsed
+			}
+		}

 		ticker := time.NewTicker(d)
 		defer ticker.Stop()
@@ -53,7 +53,7 @@ const (
 	DefaultMaxWorkers                 = 10
 	DefaultBufferCapacity             = 512
 	DefaultGCTriggerInterval          = 100
-	DefaultMemoryUsageTrackerInterval = 1 * time.Hour
+	DefaultMemoryUsageTrackerInterval = 5 * time.Minute
 )

 // Checkpoints configures periodic persistence of in-memory metric data.
@@ -113,7 +113,7 @@ type Subscriptions []struct {
 // Fields:
 //   - NumWorkers:        Parallel workers for checkpoint/archive (0 = auto: min(NumCPU/2+1, 10))
 //   - RetentionInMemory: Duration string (e.g., "48h") for in-memory data retention
-//   - MemoryCap:         Max bytes for buffer data (0 = unlimited); triggers forceFree when exceeded
+//   - MemoryCap:         Max memory in GB for buffer data (0 = unlimited); triggers forceFree when exceeded
 //   - Checkpoints:       Periodic persistence configuration
 //   - Debug:             Development/profiling options (nil = disabled)
 //   - Archive:           Long-term storage configuration (nil = disabled)
@@ -123,6 +123,7 @@ type MetricStoreConfig struct {
 	// If not set or 0, defaults to min(runtime.NumCPU()/2+1, 10)
 	NumWorkers         int            `json:"num-workers"`
 	RetentionInMemory  string         `json:"retention-in-memory"`
+	CheckpointInterval string         `json:"checkpoint-interval,omitempty"`
 	MemoryCap          int            `json:"memory-cap"`
 	Checkpoints        Checkpoints    `json:"checkpoints"`
 	Debug              *Debug         `json:"debug"`
@@ -54,6 +54,10 @@ const configSchema = `{
      "description": "Keep the metrics within memory for given time interval. Retention for X hours, then the metrics would be freed.",
      "type": "string"
    },
+    "checkpoint-interval": {
+      "description": "Interval between checkpoints as a Go duration string (e.g., '12h', '6h', '30m'). Default is '12h'.",
+      "type": "string"
+    },
    "memory-cap": {
      "description": "Upper memory capacity limit used by metricstore in GB",
      "type": "integer"
@@ -66,7 +66,10 @@ func (l *Level) collectMetricStatus(m *MemoryStore, expectedMetrics []string, he
 		if degraded[metricName] {
 			continue // already degraded, cannot improve
 		}
-		mc := m.Metrics[metricName]
+		mc, ok := m.Metrics[metricName]
+		if !ok {
+			continue // unknown metric, will be reported as missing
+		}
 		b := l.metrics[mc.offset]
 		if b.bufferExists() {
 			if !b.isBufferHealthy() {
@@ -181,6 +181,14 @@ func (l *Level) collectPaths(currentDepth, targetDepth int, currentPath []string
 //   - int:   Total number of buffers freed in this subtree
 //   - error: Non-nil on failure (propagated from children)
 func (l *Level) free(t int64) (int, error) {
+	n, _, err := l.freeAndCheckEmpty(t)
+	return n, err
+}
+
+// freeAndCheckEmpty performs free() and atomically checks if the level is empty
+// while still holding its own lock, avoiding a TOCTOU race between free() and
+// a separate isEmpty() call.
+func (l *Level) freeAndCheckEmpty(t int64) (int, bool, error) {
 	l.lock.Lock()
 	defer l.lock.Unlock()

@@ -200,15 +208,29 @@ func (l *Level) free(t int64) (int, error) {
 		}
 	}

-	for _, l := range l.children {
-		m, err := l.free(t)
+	for key, child := range l.children {
+		m, empty, err := child.freeAndCheckEmpty(t)
 		n += m
 		if err != nil {
-			return n, err
+			return n, false, err
+		}
+		if empty {
+			delete(l.children, key)
 		}
 	}

-	return n, nil
+	// Check emptiness while still holding the lock
+	empty := len(l.children) == 0
+	if empty {
+		for _, b := range l.metrics {
+			if b != nil {
+				empty = false
+				break
+			}
+		}
+	}
+
+	return n, empty, nil
 }

 // forceFree removes the oldest buffer from each metric chain in the subtree.
@@ -278,6 +300,7 @@ func (l *Level) sizeInBytes() int64 {
 	for _, b := range l.metrics {
 		if b != nil {
 			size += b.count() * int64(unsafe.Sizeof(schema.Float(0)))
+			size += b.bufferCount() * int64(unsafe.Sizeof(buffer{}))
 		}
 	}

@@ -359,7 +359,8 @@ func DecodeLine(dec *lineprotocol.Decoder,
 		}

 		if err := ms.WriteToLevel(lvl, st.selector, time, []Metric{metric}); err != nil {
-			return err
+			cclog.Warnf("write error for host %s metric %s at ts %d: %s", host, string(st.metricBuf), time, err.Error())
+			continue
 		}
 	}
 	return nil
@@ -384,14 +384,16 @@ func MemoryUsageTracker(wg *sync.WaitGroup, ctx context.Context) {
 	ms := GetMemoryStore()

 	wg.Go(func() {
-		d := DefaultMemoryUsageTrackerInterval
+		normalInterval := DefaultMemoryUsageTrackerInterval
+		fastInterval := 30 * time.Second

-		if d <= 0 {
+		if normalInterval <= 0 {
 			return
 		}

-		ticker := time.NewTicker(d)
+		ticker := time.NewTicker(normalInterval)
 		defer ticker.Stop()
+		currentInterval := normalInterval

 		for {
 			select {
@@ -428,48 +430,59 @@ func MemoryUsageTracker(wg *sync.WaitGroup, ctx context.Context) {
 				runtime.ReadMemStats(&mem)
 				actualMemoryGB = float64(mem.Alloc) / 1e9

-				bufferPool.Clear()
-				cclog.Infof("[METRICSTORE]> Cleaned up bufferPool\n")
-
 				if actualMemoryGB > float64(Keys.MemoryCap) {
 					cclog.Warnf("[METRICSTORE]> memory usage %.2f GB exceeds cap %d GB, starting emergency buffer freeing", actualMemoryGB, Keys.MemoryCap)

-					const maxIterations = 100
-
-					for i := range maxIterations {
-						if actualMemoryGB < float64(Keys.MemoryCap) {
-							break
-						}
-
-						freed, err := ms.ForceFree()
-						if err != nil {
-							cclog.Errorf("[METRICSTORE]> error while force-freeing buffers: %s", err)
-						}
-						if freed == 0 {
-							cclog.Errorf("[METRICSTORE]> no more buffers to free after %d emergency frees, memory usage %.2f GB still exceeds cap %d GB", freedEmergency, actualMemoryGB, Keys.MemoryCap)
-							break
+					// Use progressive time-based Free with increasing threshold
+					// instead of ForceFree loop — fewer tree traversals, more effective
+					d, parseErr := time.ParseDuration(Keys.RetentionInMemory)
+					if parseErr != nil {
+						cclog.Errorf("[METRICSTORE]> cannot parse retention duration: %s", parseErr)
+					} else {
+						thresholds := []float64{0.75, 0.5, 0.25}
+						for _, fraction := range thresholds {
+							threshold := time.Now().Add(-time.Duration(float64(d) * fraction))
+							freed, freeErr := ms.Free(nil, threshold.Unix())
+							if freeErr != nil {
+								cclog.Errorf("[METRICSTORE]> error while freeing buffers at %.0f%% retention: %s", fraction*100, freeErr)
 							}
 							freedEmergency += freed

-						if i%10 == 0 && freedEmergency > 0 {
+							bufferPool.Clear()
+							runtime.GC()
 							runtime.ReadMemStats(&mem)
 							actualMemoryGB = float64(mem.Alloc) / 1e9
+
+							if actualMemoryGB < float64(Keys.MemoryCap) {
+								break
+							}
 						}
 					}

-					// if freedEmergency > 0 {
-					// 	debug.FreeOSMemory()
-					// }
+					bufferPool.Clear()
+					debug.FreeOSMemory()

 					runtime.ReadMemStats(&mem)
 					actualMemoryGB = float64(mem.Alloc) / 1e9

 					if actualMemoryGB >= float64(Keys.MemoryCap) {
-						cclog.Errorf("[METRICSTORE]> after %d emergency frees, memory usage %.2f GB still at/above cap %d GB", freedEmergency, actualMemoryGB, Keys.MemoryCap)
+						cclog.Errorf("[METRICSTORE]> after emergency frees (%d buffers), memory usage %.2f GB still at/above cap %d GB", freedEmergency, actualMemoryGB, Keys.MemoryCap)
 					} else {
 						cclog.Infof("[METRICSTORE]> emergency freeing complete: %d buffers freed, memory now %.2f GB", freedEmergency, actualMemoryGB)
 					}
 				}
+
+				// Adaptive ticker: check more frequently when memory is high
+				memoryRatio := actualMemoryGB / float64(Keys.MemoryCap)
+				if memoryRatio > 0.8 && currentInterval != fastInterval {
+					ticker.Reset(fastInterval)
+					currentInterval = fastInterval
+					cclog.Infof("[METRICSTORE]> memory at %.0f%% of cap, switching to fast check interval (30s)", memoryRatio*100)
+				} else if memoryRatio <= 0.8 && currentInterval != normalInterval {
+					ticker.Reset(normalInterval)
+					currentInterval = normalInterval
+					cclog.Infof("[METRICSTORE]> memory at %.0f%% of cap, switching to normal check interval", memoryRatio*100)
+				}
 			}
 		}
 	})
@@ -741,6 +754,9 @@ func (m *MemoryStore) ForceFree() (int, error) {
 }

 func (m *MemoryStore) FreeAll() error {
+	m.root.lock.Lock()
+	defer m.root.lock.Unlock()
+
 	for k := range m.root.children {
 		delete(m.root.children, k)
 	}
@@ -42,7 +42,7 @@ func (b *buffer) stats(from, to int64) (Stats, int64, int64, error) {
 			if b == nil {
 				break
 			}
-			idx = 0
+			idx = int((t - b.start) / b.frequency)
 		}

 		if t < b.start || idx >= len(b.data) {
@@ -106,9 +106,10 @@ type walRotateReq struct {
 	done    chan struct{}
 }

-// walFileState holds an open WAL file handle for one host directory.
+// walFileState holds an open WAL file handle and buffered writer for one host directory.
 type walFileState struct {
 	f *os.File
+	w *bufio.Writer
 }

 // WALStaging starts a background goroutine that receives WALMessage items
@@ -125,15 +126,16 @@ func WALStaging(wg *sync.WaitGroup, ctx context.Context) {
 		defer func() {
 			for _, ws := range hostFiles {
 				if ws.f != nil {
+					ws.w.Flush()
 					ws.f.Close()
 				}
 			}
 		}()

-		getOrOpenWAL := func(hostDir string) *os.File {
+		getOrOpenWAL := func(hostDir string) *walFileState {
 			ws, ok := hostFiles[hostDir]
 			if ok {
-				return ws.f
+				return ws
 			}

 			if err := os.MkdirAll(hostDir, CheckpointDirPerms); err != nil {
@@ -148,29 +150,32 @@ func WALStaging(wg *sync.WaitGroup, ctx context.Context) {
 				return nil
 			}

+			w := bufio.NewWriter(f)
+
 			// Write file header magic if file is new (empty).
 			info, err := f.Stat()
 			if err == nil && info.Size() == 0 {
 				var hdr [4]byte
 				binary.LittleEndian.PutUint32(hdr[:], walFileMagic)
-				if _, err := f.Write(hdr[:]); err != nil {
+				if _, err := w.Write(hdr[:]); err != nil {
 					cclog.Errorf("[METRICSTORE]> WAL: write header %s: %v", walPath, err)
 					f.Close()
 					return nil
 				}
 			}

-			hostFiles[hostDir] = &walFileState{f: f}
-			return f
+			ws = &walFileState{f: f, w: w}
+			hostFiles[hostDir] = ws
+			return ws
 		}

 		processMsg := func(msg *WALMessage) {
 			hostDir := path.Join(Keys.Checkpoints.RootDir, msg.Cluster, msg.Node)
-			f := getOrOpenWAL(hostDir)
-			if f == nil {
+			ws := getOrOpenWAL(hostDir)
+			if ws == nil {
 				return
 			}
-			if err := writeWALRecord(f, msg); err != nil {
+			if err := writeWALRecord(ws.w, msg); err != nil {
 				cclog.Errorf("[METRICSTORE]> WAL: write record: %v", err)
 			}
 		}
@@ -178,6 +183,7 @@ func WALStaging(wg *sync.WaitGroup, ctx context.Context) {
 		processRotate := func(req walRotateReq) {
 			ws, ok := hostFiles[req.hostDir]
 			if ok && ws.f != nil {
+				ws.w.Flush()
 				ws.f.Close()
 				walPath := path.Join(req.hostDir, "current.wal")
 				if err := os.Remove(walPath); err != nil && !os.IsNotExist(err) {
@@ -199,6 +205,12 @@ func WALStaging(wg *sync.WaitGroup, ctx context.Context) {
 				case req := <-walRotateCh:
 					processRotate(req)
 				default:
+					// Flush all buffered writers after draining remaining messages.
+					for _, ws := range hostFiles {
+						if ws.f != nil {
+							ws.w.Flush()
+						}
+					}
 					return
 				}
 			}
@@ -282,9 +294,9 @@ func buildWALPayload(msg *WALMessage) []byte {
 	return buf
 }

-// writeWALRecord appends a binary WAL record to the file.
+// writeWALRecord appends a binary WAL record to the writer.
 // Format: [4B magic][4B payload_len][payload][4B CRC32]
-func writeWALRecord(f *os.File, msg *WALMessage) error {
+func writeWALRecord(w io.Writer, msg *WALMessage) error {
 	payload := buildWALPayload(msg)
 	crc := crc32.ChecksumIEEE(payload)

@@ -304,7 +316,7 @@ func writeWALRecord(f *os.File, msg *WALMessage) error {
 	binary.LittleEndian.PutUint32(crcBytes[:], crc)
 	record = append(record, crcBytes[:]...)

-	_, err := f.Write(record)
+	_, err := w.Write(record)
 	return err
 }

@@ -20,14 +20,14 @@
        "wonka": "^6.3.5"
      },
      "devDependencies": {
-        "@rollup/plugin-commonjs": "^29.0.1",
+        "@rollup/plugin-commonjs": "^29.0.2",
        "@rollup/plugin-node-resolve": "^16.0.3",
        "@rollup/plugin-terser": "^1.0.0",
        "@timohausmann/quadtree-js": "^1.2.6",
        "rollup": "^4.59.0",
        "rollup-plugin-css-only": "^4.5.5",
        "rollup-plugin-svelte": "^7.2.3",
-        "svelte": "^5.53.7"
+        "svelte": "^5.53.9"
      }
    },
    "node_modules/@0no-co/graphql.web": {
@@ -126,9 +126,9 @@
      }
    },
    "node_modules/@rollup/plugin-commonjs": {
-      "version": "29.0.1",
-      "resolved": "https://registry.npmjs.org/@rollup/plugin-commonjs/-/plugin-commonjs-29.0.1.tgz",
-      "integrity": "sha512-VUEHINN2rQEWPfNUR3mzidRObM1XZKXMQsaG6qBlDqd6M1qyw91nDZvcSozgyjt3x/QKrgKBc5MdxfdxAy6tdg==",
+      "version": "29.0.2",
+      "resolved": "https://registry.npmjs.org/@rollup/plugin-commonjs/-/plugin-commonjs-29.0.2.tgz",
+      "integrity": "sha512-S/ggWH1LU7jTyi9DxZOKyxpVd4hF/OZ0JrEbeLjXk/DFXwRny0tjD2c992zOUYQobLrVkRVMDdmHP16HKP7GRg==",
      "dev": true,
      "license": "MIT",
      "dependencies": {
@@ -1193,9 +1193,9 @@
      }
    },
    "node_modules/svelte": {
-      "version": "5.53.7",
-      "resolved": "https://registry.npmjs.org/svelte/-/svelte-5.53.7.tgz",
-      "integrity": "sha512-uxck1KI7JWtlfP3H6HOWi/94soAl23jsGJkBzN2BAWcQng0+lTrRNhxActFqORgnO9BHVd1hKJhG+ljRuIUWfQ==",
+      "version": "5.53.9",
+      "resolved": "https://registry.npmjs.org/svelte/-/svelte-5.53.9.tgz",
+      "integrity": "sha512-MwDfWsN8qZzeP0jlQsWF4k/4B3csb3IbzCRggF+L/QqY7T8bbKvnChEo1cPZztF51HJQhilDbevWYl2LvXbquA==",
      "license": "MIT",
      "dependencies": {
        "@jridgewell/remapping": "^2.3.4",
@@ -7,14 +7,14 @@
    "dev": "rollup -c -w"
  },
  "devDependencies": {
-    "@rollup/plugin-commonjs": "^29.0.1",
+    "@rollup/plugin-commonjs": "^29.0.2",
    "@rollup/plugin-node-resolve": "^16.0.3",
    "@rollup/plugin-terser": "^1.0.0",
    "@timohausmann/quadtree-js": "^1.2.6",
    "rollup": "^4.59.0",
    "rollup-plugin-css-only": "^4.5.5",
    "rollup-plugin-svelte": "^7.2.3",
-    "svelte": "^5.53.7"
+    "svelte": "^5.53.9"
  },
  "dependencies": {
    "@rollup/plugin-replace": "^6.0.3",
@@ -149,7 +149,6 @@
        jobsStatistics(
          filter: $jobFilter
          page: $paging
-          sortBy: TOTALJOBS
          groupBy: CLUSTER
        ) {
          id
@@ -185,7 +184,7 @@
      from: from.toISOString(),
      clusterFrom: clusterFrom.toISOString(),
      to: to.toISOString(),
-      jobFilter: [{ state: ["running"] }, { cluster: { eq: presetCluster } }],
+      jobFilter: [{ cluster: { eq: presetCluster } }, { state: ["running"] }],
      nodeFilter: { cluster: { eq: presetCluster }},
      paging: { itemsPerPage: -1, page: 1 }, // Get all: -1
      sorting: { field: "startTime", type: "col", order: "DESC" }
@@ -302,11 +301,11 @@

    if (subclusterData) {
      for (let i = 0; i < subclusterData.length; i++) {
-        const flopsData = subclusterData[i].metrics.find((s) => s.name == "flops_any")
-        const memBwData = subclusterData[i].metrics.find((s) => s.name == "mem_bw")
+        const flopsData = subclusterData[i]?.metrics?.find((s) => s.name == "flops_any")
+        const memBwData = subclusterData[i]?.metrics?.find((s) => s.name == "mem_bw")

-        const f = flopsData.metric.series[0].statistics.avg
-        const m = memBwData.metric.series[0].statistics.avg
+        const f = flopsData?.metric?.series[0]?.statistics?.avg || 0;
+        const m = memBwData?.metric?.series[0]?.statistics?.avg || 0;

        let intensity = f / m
        if (Number.isNaN(intensity) || !Number.isFinite(intensity)) {
@@ -211,7 +211,7 @@
  const orderAndMap = (grouped, inputMetrics) =>
    inputMetrics.map((metric) => ({
      metric: metric,
-      data: grouped.find((group) => group[0].name == metric),
+      data: grouped.find((group) => group[0]?.name == metric),
      availability: checkMetricAvailability(
        globalMetrics,
        metric,
@@ -148,7 +148,7 @@
      showFilter={!showCompare}
      matchedJobs={showCompare? matchedCompareJobs: matchedListJobs}
      applyFilters={(detail) => {
-        selectedCluster = detail.filters[0]?.cluster
+        selectedCluster = detail?.filters[0]?.cluster
          ? detail.filters[0].cluster.eq
          : null;
        selectedSubCluster = detail.filters[1]?.partition
@@ -34,6 +34,7 @@
    formatDurationTime
  } from "./generic/units.js";
  import Filters from "./generic/Filters.svelte";
+  import Pagination from "./generic/joblist/Pagination.svelte";

  /* Svelte 5 Props */
  let {
@@ -51,6 +52,8 @@
  let jobFilters = $state([]);
  let nameFilter = $state("");
  let sorting = $state({ field: "totalJobs", direction: "desc" });
+  let page = $state(1);
+  let itemsPerPage = $state(25);

  /* Derived Vars */
  const fetchRunning = $derived(jobFilters.some(jf => jf?.state?.length == 1 && jf?.state?.includes("running")));
@@ -64,6 +67,12 @@
  const sortedRows = $derived(
    $stats.data ? sort($stats.data.rows, sorting, nameFilter) : []
  );
+  const paginatedRows = $derived(
+    sortedRows.slice((page - 1) * itemsPerPage, page * itemsPerPage)
+  );
+
+  /* Reset page when sorting or filter changes */
+  $effect(() => { sorting; nameFilter; page = 1; });

  let stats = $derived(
    queryStore({
@@ -360,7 +369,7 @@
        >
      </tr>
    {:else if $stats.data}
-      {#each sort($stats.data.rows, sorting, nameFilter) as row (row.id)}
+      {#each paginatedRows as row (row.id)}
        <tr>
          <td>
            {#if type == "USER"}
@@ -402,3 +411,16 @@
    {/if}
  </tbody>
 </Table>
+{#if sortedRows.length > 0}
+  <Pagination
+    {page}
+    {itemsPerPage}
+    totalItems={sortedRows.length}
+    itemText={type === 'USER' ? 'Users' : 'Projects'}
+    pageSizes={[25, 50, 100]}
+    updatePaging={(detail) => {
+      itemsPerPage = detail.itemsPerPage;
+      page = detail.page;
+    }}
+  />
+{/if}
@@ -119,8 +119,8 @@

  const filter = $derived([
    { cluster: { eq: cluster } },
-    { node: { eq: hostname } },
    { state: ["running"] },
+    { node: { eq: hostname } },
  ]);

  const systemUnits = $derived.by(() => {
@@ -167,7 +167,7 @@
      <InputGroup>
        <InputGroupText><Icon name="hdd" /></InputGroupText>
        <InputGroupText>Selected Node</InputGroupText>
-        <Input style="background-color: white;" type="text" value="{hostname} [{cluster} {$nodeMetricsData?.data ? `(${$nodeMetricsData.data.nodeMetrics[0].subCluster})` : ''}]" disabled/>
+        <Input style="background-color: white;" type="text" value="{hostname} [{cluster} {$nodeMetricsData?.data?.nodeMetrics[0] ? `(${$nodeMetricsData.data.nodeMetrics[0].subCluster})` : ''}]" disabled/>
      </InputGroup>
    </Col>
    <!-- State Col -->
@@ -259,7 +259,7 @@
            </CardHeader>
            <CardBody>
              <p>No dataset(s) returned for <b>{item.name}</b></p>
-              <p class="mb-1">Metric has been disabled for subcluster <b>{$nodeMetricsData.data.nodeMetrics[0].subCluster}</b>.</p>
+              <p class="mb-1">Metric has been disabled for subcluster <b>{$nodeMetricsData?.data?.nodeMetrics[0]?.subCluster}</b>.</p>
            </CardBody>
          </Card>
        {:else if item?.metric}
@@ -267,7 +267,7 @@
            metric={item.name}
            timestep={item.metric.timestep}
            cluster={clusterInfos.find((c) => c.name == cluster)}
-            subCluster={$nodeMetricsData.data.nodeMetrics[0].subCluster}
+            subCluster={$nodeMetricsData?.data?.nodeMetrics[0]?.subCluster}
            series={item.metric.series}
            enableFlip
            forNode
@@ -286,17 +286,17 @@
      {/snippet}

      <PlotGrid
-        items={$nodeMetricsData.data.nodeMetrics[0].metrics
+        items={$nodeMetricsData?.data?.nodeMetrics[0]?.metrics
          .map((m) => ({
            ...m,
            availability: checkMetricAvailability(
              globalMetrics,
              m.name,
              cluster,
-              $nodeMetricsData.data.nodeMetrics[0].subCluster,
+              $nodeMetricsData?.data?.nodeMetrics[0]?.subCluster,
            ),
          }))
-          .sort((a, b) => a.name.localeCompare(b.name))}
+          .sort((a, b) => a.name.localeCompare(b.name)) || []}
        itemsPerRow={ccconfig.plotConfiguration_plotsPerRow}
        {gridContent}
      />
@@ -159,7 +159,7 @@
      variables: { jobFilters, selectedHistograms, numDurationBins, numMetricBins },
    })
  );
-  const hasAccHours = $derived($stats?.data?.jobsStatistics[0]?.totalAccHours != 0);
+  const hasAccHours = $derived(($stats?.data?.jobsStatistics[0]?.totalAccHours || 0) != 0);

  /* Functions */
  function resetJobSelection() {
@@ -310,8 +310,8 @@
            <tr>
              <th scope="row">Total Jobs</th>
              <td>
-                <span style="cursor: help;" title="{$stats.data.jobsStatistics[0].totalJobs} Jobs">
-                  {formatNumber($stats.data.jobsStatistics[0].totalJobs)} Jobs
+                <span style="cursor: help;" title="{$stats?.data?.jobsStatistics[0]?.totalJobs || 0} Jobs">
+                  {formatNumber($stats?.data?.jobsStatistics[0]?.totalJobs || 0)} Jobs
                </span>
              </td>
            </tr>
@@ -326,24 +326,24 @@
                </Tooltip>
              </th>
              <td>
-                <span style="cursor: help;" title="{$stats.data.jobsStatistics[0].shortJobs} Jobs">
-                  {formatNumber($stats.data.jobsStatistics[0].shortJobs)} Jobs
+                <span style="cursor: help;" title="{$stats?.data?.jobsStatistics[0]?.shortJobs || 0} Jobs">
+                  {formatNumber($stats?.data?.jobsStatistics[0]?.shortJobs || 0)} Jobs
                </span>
              </td>
            </tr>
            <tr>
              <th scope="row">Total Walltime</th>
              <td>
-                <span style="cursor: help;" title="{$stats.data.jobsStatistics[0].totalWalltime} Hours">
-                  {formatNumber($stats.data.jobsStatistics[0].totalWalltime)} Hours
+                <span style="cursor: help;" title="{$stats?.data?.jobsStatistics[0]?.totalWalltime || 0} Hours">
+                  {formatNumber($stats?.data?.jobsStatistics[0]?.totalWalltime || 0)} Hours
                </span>
              </td>
            </tr>
            <tr>
              <th scope="row">Total Core Hours</th>
              <td>
-                <span style="cursor: help;" title="{$stats.data.jobsStatistics[0].totalCoreHours} Hours">
-                  {formatNumber($stats.data.jobsStatistics[0].totalCoreHours)} Hours
+                <span style="cursor: help;" title="{$stats?.data?.jobsStatistics[0]?.totalCoreHours || 0} Hours">
+                  {formatNumber($stats?.data?.jobsStatistics[0]?.totalCoreHours || 0)} Hours
                </span>
              </td>
            </tr>
@@ -351,8 +351,8 @@
              <tr>
                <th scope="row">Total Accelerator Hours</th>
                <td>
-                  <span style="cursor: help;" title="{$stats.data.jobsStatistics[0].totalAccHours} Hours">
-                    {formatNumber($stats.data.jobsStatistics[0].totalAccHours)} Hours
+                  <span style="cursor: help;" title="{$stats?.data?.jobsStatistics[0]?.totalAccHours || 0} Hours">
+                    {formatNumber($stats?.data?.jobsStatistics[0]?.totalAccHours || 0)} Hours
                  </span>
                </td>
              </tr>
@@ -361,9 +361,9 @@
        </Table>
      </Col>
      <Col class="px-1">
-        {#key $stats.data.jobsStatistics[0].histDuration}
+        {#key $stats?.data?.jobsStatistics[0]?.histDuration}
          <Histogram
-            data={convert2uplot($stats.data.jobsStatistics[0].histDuration)}
+            data={convert2uplot($stats?.data?.jobsStatistics[0]?.histDuration)}
            title="Duration Distribution"
            xlabel="Job Runtimes"
            xunit="Runtime"
@@ -376,9 +376,9 @@
        {/key}
      </Col>
      <Col class="px-1">
-        {#key $stats.data.jobsStatistics[0].histNumNodes}
+        {#key $stats?.data?.jobsStatistics[0]?.histNumNodes}
          <Histogram
-            data={convert2uplot($stats.data.jobsStatistics[0].histNumNodes)}
+            data={convert2uplot($stats?.data?.jobsStatistics[0]?.histNumNodes)}
            title="Number of Nodes Distribution"
            xlabel="Allocated Nodes"
            xunit="Nodes"
@@ -450,9 +450,9 @@
        />
      {/snippet}

-      {#key $stats.data.jobsStatistics[0].histMetrics}
+      {#key $stats?.data?.jobsStatistics[0]?.histMetrics}
        <PlotGrid
-          items={$stats.data.jobsStatistics[0].histMetrics}
+          items={$stats?.data?.jobsStatistics[0]?.histMetrics || []}
          itemsPerRow={3}
          {gridContent}
        />
@@ -29,7 +29,7 @@

  /* Const */
  const minEnergyPreset = 1;
-  const maxEnergyPreset = 1000;
+  const maxEnergyPreset = 100;

  /* Derived */
  // Pending
@@ -167,7 +167,7 @@

  <p class="mb-2">
    {#if job.numNodes == 1}
-      {job.resources[0].hostname}
+      {job?.resources[0]?.hostname}
    {:else}
      {job.numNodes}
    {/if}
@@ -340,7 +340,7 @@
 </script>

 <!-- Define $width Wrapper and NoData Card -->
-{#if data && data[0].length > 0}
+{#if data && data[0]?.length > 0}
  <div bind:this={plotWrapper} bind:clientWidth={width}
        style="background-color: rgba(255, 255, 255, 1.0);" class="rounded"
  ></div>
@@ -155,7 +155,6 @@
        jobsStatistics(
          filter: $jobFilter
          page: $paging
-          sortBy: TOTALJOBS
          groupBy: CLUSTER
        ) {
          id
@@ -190,7 +189,7 @@
      from: from.toISOString(),
      to: to.toISOString(),
      clusterFrom: clusterFrom.toISOString(),
-      jobFilter: [{ state: ["running"] }, { cluster: { eq: presetCluster } }],
+      jobFilter: [{ cluster: { eq: presetCluster } }, { state: ["running"] }],
      paging: { itemsPerPage: -1, page: 1 }, // Get all: -1
      sorting: { field: "startTime", type: "col", order: "DESC" }
    },
@@ -216,7 +215,7 @@
      }
    `,
    variables: {
-      filter: [{ state: ["running"] }, { cluster: { eq: presetCluster} }],
+      filter: [{ cluster: { eq: presetCluster} }, { state: ["running"] }],
      paging: pagingState // Top 10
    },
    requestPolicy: "network-only"
@@ -72,7 +72,7 @@
      }
    `,
    variables: {
-      filter: [{ state: ["running"] }, { cluster: { eq: cluster} }],
+      filter: [{ cluster: { eq: cluster} }, { state: ["running"] }],
      selectedHistograms: selectedHistograms
    },
    requestPolicy: "network-only"
@@ -129,9 +129,9 @@
      />
    {/snippet}
    
-    {#key $metricStatusQuery.data.jobsStatistics[0].histMetrics}
+    {#key $metricStatusQuery?.data?.jobsStatistics[0]?.histMetrics}
      <PlotGrid
-        items={$metricStatusQuery.data.jobsStatistics[0].histMetrics}
+        items={$metricStatusQuery?.data?.jobsStatistics[0]?.histMetrics}
        itemsPerRow={2}
        {gridContent}
      />
@@ -163,7 +163,6 @@
        jobsStatistics(
          filter: $jobFilter
          page: $paging
-          sortBy: TOTALJOBS
          groupBy: SUBCLUSTER
        ) {
          id
@@ -179,7 +178,7 @@
      metrics: ["flops_any", "mem_bw"], // Fixed names for roofline and status bars
      from: from.toISOString(),
      to: to.toISOString(),
-      jobFilter: [{ state: ["running"] }, { cluster: { eq: cluster } }],
+      jobFilter: [{ cluster: { eq: cluster } }, { state: ["running"] }],
      nodeFilter: { cluster: { eq: cluster }},
      paging: { itemsPerPage: -1, page: 1 }, // Get all: -1
      sorting: { field: "startTime", type: "col", order: "DESC" }
@@ -295,8 +294,8 @@
        const flopsData = subclusterData[i].metrics.find((s) => s.name == "flops_any")
        const memBwData = subclusterData[i].metrics.find((s) => s.name == "mem_bw")

-        const f = flopsData.metric.series[0].statistics.avg
-        const m = memBwData.metric.series[0].statistics.avg
+        const f = flopsData?.metric?.series[0]?.statistics?.avg || 0
+        const m = memBwData?.metric?.series[0]?.statistics?.avg || 0

        let intensity = f / m
        if (Number.isNaN(intensity) || !Number.isFinite(intensity)) {
@@ -19,13 +19,9 @@
    Tooltip,
    Input,
    InputGroup,
-    InputGroupText
+    InputGroupText,
  } from "@sveltestrap/sveltestrap";
-  import {
-    queryStore,
-    gql,
-    getContextClient,
-  } from "@urql/svelte";
+  import { queryStore, gql, getContextClient } from "@urql/svelte";
  import {
    scramble,
    scrambleNames,
@@ -49,127 +45,98 @@
  const durationBinOptions = ["1m", "10m", "1h", "6h", "12h"];

  /* State Init */
-  let pagingState = $state({page: 1, itemsPerPage: 10}) // Top 10
-  let selectedHistograms = $state([]) // Dummy For Refresh 
+  let pagingState = $state({ page: 1, itemsPerPage: 10 }); // Top 10
+  let selectedHistograms = $state([]); // Dummy For Refresh
  let colWidthJobs = $state(0);
  let colWidthNodes = $state(0);
  let colWidthAccs = $state(0);
  let numDurationBins = $state("1h");

  /* Derived */
-  const canvasPrefix = $derived(`${presetCluster}-${presetSubCluster ? presetSubCluster : ''}`)
-
-  const statusFilter = $derived(presetSubCluster
-      ? [{ state: ["running"] }, { cluster: { eq: presetCluster} }, { subCluster: { eq: presetSubCluster } }]
-      : [{ state: ["running"] }, { cluster: { eq: presetCluster} }] 
+  const canvasPrefix = $derived(
+    `${presetCluster}-${presetSubCluster ? presetSubCluster : ""}`,
  );
-  const topJobsQuery = $derived(loadMe ? queryStore({
+
+  const statusFilter = $derived(
+    presetSubCluster
+      ? [
+          { cluster: { eq: presetCluster } },
+          { subCluster: { eq: presetSubCluster } },
+          { state: ["running"] },
+        ]
+      : [{ cluster: { eq: presetCluster } }, { state: ["running"] }],
+  );
+  const topStatsQuery = $derived(
+    loadMe
+      ? queryStore({
          client: client,
          query: gql`
-      query (
-        $filter: [JobFilter!]!
-        $paging: PageRequest!
-      ) {
-        topUser: jobsStatistics(
+            query ($filter: [JobFilter!]!) {
+              allUsers: jobsStatistics(
                filter: $filter
-          page: $paging
-          sortBy: TOTALJOBS
                groupBy: USER
              ) {
                id
                name
                totalJobs
-        }
-        topProjects: jobsStatistics(
-          filter: $filter
-          page: $paging
-          sortBy: TOTALJOBS
-          groupBy: PROJECT
-        ) {
-          id
-          totalJobs
-        }
-      }
-    `,
-    variables: {
-      filter: statusFilter,
-      paging: pagingState // Top 10
-    },
-    requestPolicy: "network-only"
-  }) : null);
-
-  const topNodesQuery = $derived(loadMe ? queryStore({
-    client: client,
-    query: gql`
-      query (
-        $filter: [JobFilter!]!
-        $paging: PageRequest!
-      ) {
-        topUser: jobsStatistics(
-          filter: $filter
-          page: $paging
-          sortBy: TOTALNODES
-          groupBy: USER
-        ) {
-          id
-          name
                totalNodes
-        }
-        topProjects: jobsStatistics(
-          filter: $filter
-          page: $paging
-          sortBy: TOTALNODES
-          groupBy: PROJECT
-        ) {
-          id
-          totalNodes
-        }
-      }
-    `,
-    variables: {
-      filter: statusFilter,
-      paging: pagingState
-    },
-    requestPolicy: "network-only"
-  }) : null);
-
-  const topAccsQuery = $derived(loadMe ? queryStore({
-    client: client,
-    query: gql`
-      query (
-        $filter: [JobFilter!]!
-        $paging: PageRequest!
-      ) {
-        topUser: jobsStatistics(
-          filter: $filter
-          page: $paging
-          sortBy: TOTALACCS
-          groupBy: USER
-        ) {
-          id
-          name
                totalAccs
              }
-        topProjects: jobsStatistics(
+              allProjects: jobsStatistics(
                filter: $filter
-          page: $paging
-          sortBy: TOTALACCS
                groupBy: PROJECT
              ) {
                id
+                totalJobs
+                totalNodes
                totalAccs
              }
            }
          `,
          variables: {
            filter: statusFilter,
-      paging: pagingState
          },
-    requestPolicy: "network-only"
-  }): null);
+          requestPolicy: "network-only",
+        })
+      : null,
+  );
+
+  // Sort + slice top-10 from the full results in the frontend
+  const topUserJobs = $derived(
+    $topStatsQuery?.data?.allUsers
+      ?.toSorted((a, b) => b.totalJobs - a.totalJobs)
+      .slice(0, 10) ?? [],
+  );
+  const topProjectJobs = $derived(
+    $topStatsQuery?.data?.allProjects
+      ?.toSorted((a, b) => b.totalJobs - a.totalJobs)
+      .slice(0, 10) ?? [],
+  );
+  const topUserNodes = $derived(
+    $topStatsQuery?.data?.allUsers
+      ?.toSorted((a, b) => b.totalNodes - a.totalNodes)
+      .slice(0, 10) ?? [],
+  );
+  const topProjectNodes = $derived(
+    $topStatsQuery?.data?.allProjects
+      ?.toSorted((a, b) => b.totalNodes - a.totalNodes)
+      .slice(0, 10) ?? [],
+  );
+  const topUserAccs = $derived(
+    $topStatsQuery?.data?.allUsers
+      ?.toSorted((a, b) => b.totalAccs - a.totalAccs)
+      .slice(0, 10) ?? [],
+  );
+  const topProjectAccs = $derived(
+    $topStatsQuery?.data?.allProjects
+      ?.toSorted((a, b) => b.totalAccs - a.totalAccs)
+      .slice(0, 10) ?? [],
+  );

  // Note: nodeMetrics are requested on configured $timestep resolution
-  const nodeStatusQuery = $derived(loadMe ? queryStore({
+  const nodeStatusQuery = $derived(
+    loadMe
+      ? queryStore({
          client: client,
          query: gql`
            query (
@@ -177,7 +144,11 @@
              $selectedHistograms: [String!]
              $numDurationBins: String
            ) {
-        jobsStatistics(filter: $filter, metrics: $selectedHistograms, numDurationBins: $numDurationBins) {
+              jobsStatistics(
+                filter: $filter
+                metrics: $selectedHistograms
+                numDurationBins: $numDurationBins
+              ) {
                histDuration {
                  count
                  value
@@ -198,19 +169,21 @@
            selectedHistograms: selectedHistograms, // No Metrics requested for node hardware stats
            numDurationBins: numDurationBins,
          },
-    requestPolicy: "network-only"
-  }) : null);
+          requestPolicy: "network-only",
+        })
+      : null,
+  );

  /* Functions */
  function legendColors(targetIdx) {
    // Reuses first color if targetIdx overflows
    let c;
    if (useCbColors) {
-        c = [...colors['colorblind']];
+      c = [...colors["colorblind"]];
    } else if (useAltColors) {
-        c = [...colors['alternative']];
+      c = [...colors["alternative"]];
    } else {
-        c = [...colors['default']];
+      c = [...colors["default"]];
    }
    return c[(c.length + targetIdx) % c.length];
  }
@@ -223,9 +196,7 @@
      <InputGroupText>
        <Icon name="bar-chart-line-fill" />
      </InputGroupText>
-      <InputGroupText>
-        Duration Bin Size
-      </InputGroupText>
+      <InputGroupText>Duration Bin Size</InputGroupText>
      <Input type="select" bind:value={numDurationBins}>
        {#each durationBinOptions as dbin}
          <option value={dbin}>{dbin}</option>
@@ -247,14 +218,16 @@
 <hr />

 <!-- Job Duration, Top Users and Projects-->
-{#if $topJobsQuery?.fetching || $nodeStatusQuery?.fetching}
+{#if $topStatsQuery?.fetching || $nodeStatusQuery?.fetching}
  <Spinner />
-{:else if $topJobsQuery?.data && $nodeStatusQuery?.data}
+{:else if $topStatsQuery?.data && $nodeStatusQuery?.data}
  <Row>
    <Col xs="12" lg="4" class="p-2">
-      {#key $nodeStatusQuery.data.jobsStatistics[0].histDuration}
+      {#key $nodeStatusQuery?.data?.jobsStatistics[0]?.histDuration}
        <Histogram
-          data={convert2uplot($nodeStatusQuery.data.jobsStatistics[0].histDuration)}
+          data={convert2uplot(
+            $nodeStatusQuery?.data?.jobsStatistics[0]?.histDuration,
+          )}
          title="Duration Distribution"
          xlabel="Current Job Runtimes"
          xunit="Runtime"
@@ -269,18 +242,18 @@
    </Col>
    <Col xs="6" md="3" lg="2" class="p-2">
      <div bind:clientWidth={colWidthJobs}>
-        <h4 class="text-center">
-          Top Users: Jobs
-        </h4>
+        <h4 class="text-center">Top Users: Jobs</h4>
        <Pie
          {useAltColors}
          canvasId="{canvasPrefix}-hpcpie-jobs-users"
          size={colWidthJobs * 0.75}
          sliceLabel="Jobs"
-          quantities={$topJobsQuery.data.topUser.map(
-            (tu) => tu['totalJobs'],
+          quantities={topUserJobs.map(
+            (tu) => tu["totalJobs"],
+          )}
+          entities={topUserJobs.map((tu) =>
+            scrambleNames ? scramble(tu.id) : tu.id,
          )}
-          entities={$topJobsQuery.data.topUser.map((tu) => scrambleNames ? scramble(tu.id) : tu.id)}
        />
      </div>
    </Col>
@@ -291,11 +264,17 @@
          <th style="padding-left: 0.5rem;">User</th>
          <th>Jobs</th>
        </tr>
-        {#each $topJobsQuery.data.topUser as tu, i}
+        {#each topUserJobs as tu, i}
          <tr>
-            <td><Icon name="circle-fill" style="color: {legendColors(i)};" /></td>
+            <td
+              ><Icon name="circle-fill" style="color: {legendColors(i)};" /></td
+            >
            <td id="{canvasPrefix}-topName-jobs-{tu.id}">
-              <a target="_blank" href="/monitoring/user/{tu.id}?cluster={presetCluster}{presetSubCluster ? '&partition='+presetSubCluster : ''}&state=running"
+              <a
+                target="_blank"
+                href="/monitoring/user/{tu.id}?cluster={presetCluster}{presetSubCluster
+                  ? '&partition=' + presetSubCluster
+                  : ''}&state=running"
                >{scrambleNames ? scramble(tu.id) : tu.id}
              </a>
            </td>
@@ -306,25 +285,25 @@
                >{scrambleNames ? scramble(tu.name) : tu.name}</Tooltip
              >
            {/if}
-            <td>{tu['totalJobs']}</td>
+            <td>{tu["totalJobs"]}</td>
          </tr>
        {/each}
      </Table>
    </Col>

    <Col xs="6" md="3" lg="2" class="p-2">
-      <h4 class="text-center">
-        Top Projects: Jobs
-      </h4>
+      <h4 class="text-center">Top Projects: Jobs</h4>
      <Pie
        {useAltColors}
        canvasId="{canvasPrefix}-hpcpie-jobs-projects"
        size={colWidthJobs * 0.75}
-        sliceLabel={'Jobs'}
-        quantities={$topJobsQuery.data.topProjects.map(
-          (tp) => tp['totalJobs'],
+        sliceLabel={"Jobs"}
+        quantities={topProjectJobs.map(
+          (tp) => tp["totalJobs"],
+        )}
+        entities={topProjectJobs.map((tp) =>
+          scrambleNames ? scramble(tp.id) : tp.id,
        )}
-        entities={$topJobsQuery.data.topProjects.map((tp) => scrambleNames ? scramble(tp.id) : tp.id)}
      />
    </Col>
    <Col xs="6" md="3" lg="2" class="p-2">
@@ -334,34 +313,44 @@
          <th style="padding-left: 0.5rem;">Project</th>
          <th>Jobs</th>
        </tr>
-        {#each $topJobsQuery.data.topProjects as tp, i}
+        {#each topProjectJobs as tp, i}
          <tr>
-            <td><Icon name="circle-fill" style="color: {legendColors(i)};" /></td>
+            <td
+              ><Icon name="circle-fill" style="color: {legendColors(i)};" /></td
+            >
            <td>
-              <a target="_blank" href="/monitoring/jobs/?cluster={presetCluster}{presetSubCluster ? '&partition='+presetSubCluster : ''}&state=running&project={tp.id}&projectMatch=eq"
+              <a
+                target="_blank"
+                href="/monitoring/jobs/?cluster={presetCluster}{presetSubCluster
+                  ? '&partition=' + presetSubCluster
+                  : ''}&state=running&project={tp.id}&projectMatch=eq"
                >{scrambleNames ? scramble(tp.id) : tp.id}
              </a>
            </td>
-            <td>{tp['totalJobs']}</td>
+            <td>{tp["totalJobs"]}</td>
          </tr>
        {/each}
      </Table>
    </Col>
  </Row>
 {:else}
-  <Card class="mx-4" body color="warning">Cannot render job status charts: No data!</Card>
+  <Card class="mx-4" body color="warning"
+    >Cannot render job status charts: No data!</Card
+  >
 {/if}

 <hr />

 <!-- Node Distribution, Top Users and Projects-->
-{#if $topNodesQuery?.fetching || $nodeStatusQuery?.fetching}
+{#if $topStatsQuery?.fetching || $nodeStatusQuery?.fetching}
  <Spinner />
-{:else if $topNodesQuery?.data && $nodeStatusQuery?.data}
+{:else if $topStatsQuery?.data && $nodeStatusQuery?.data}
  <Row>
    <Col xs="12" lg="4" class="p-2">
      <Histogram
-        data={convert2uplot($nodeStatusQuery.data.jobsStatistics[0].histNumNodes)}
+        data={convert2uplot(
+          $nodeStatusQuery?.data?.jobsStatistics[0]?.histNumNodes,
+        )}
        title="Number of Nodes Distribution"
        xlabel="Allocated Nodes"
        xunit="Nodes"
@@ -373,18 +362,18 @@
    </Col>
    <Col xs="6" md="3" lg="2" class="p-2">
      <div bind:clientWidth={colWidthNodes}>
-        <h4 class="text-center">
-          Top Users: Nodes
-        </h4>
+        <h4 class="text-center">Top Users: Nodes</h4>
        <Pie
          {useAltColors}
          canvasId="{canvasPrefix}-hpcpie-nodes-users"
          size={colWidthNodes * 0.75}
          sliceLabel="Nodes"
-          quantities={$topNodesQuery.data.topUser.map(
-            (tu) => tu['totalNodes'],
+          quantities={topUserNodes.map(
+            (tu) => tu["totalNodes"],
+          )}
+          entities={topUserNodes.map((tu) =>
+            scrambleNames ? scramble(tu.id) : tu.id,
          )}
-          entities={$topNodesQuery.data.topUser.map((tu) => scrambleNames ? scramble(tu.id) : tu.id)}
        />
      </div>
    </Col>
@@ -395,11 +384,17 @@
          <th style="padding-left: 0.5rem;">User</th>
          <th>Nodes</th>
        </tr>
-        {#each $topNodesQuery.data.topUser as tu, i}
+        {#each topUserNodes as tu, i}
          <tr>
-            <td><Icon name="circle-fill" style="color: {legendColors(i)};" /></td>
+            <td
+              ><Icon name="circle-fill" style="color: {legendColors(i)};" /></td
+            >
            <td id="{canvasPrefix}-topName-nodes-{tu.id}">
-              <a target="_blank" href="/monitoring/user/{tu.id}?cluster={presetCluster}{presetSubCluster ? '&partition='+presetSubCluster : ''}&state=running"
+              <a
+                target="_blank"
+                href="/monitoring/user/{tu.id}?cluster={presetCluster}{presetSubCluster
+                  ? '&partition=' + presetSubCluster
+                  : ''}&state=running"
                >{scrambleNames ? scramble(tu.id) : tu.id}
              </a>
            </td>
@@ -410,25 +405,25 @@
                >{scrambleNames ? scramble(tu.name) : tu.name}</Tooltip
              >
            {/if}
-            <td>{tu['totalNodes']}</td>
+            <td>{tu["totalNodes"]}</td>
          </tr>
        {/each}
      </Table>
    </Col>

    <Col xs="6" md="3" lg="2" class="p-2">
-      <h4 class="text-center">
-        Top Projects: Nodes
-      </h4>
+      <h4 class="text-center">Top Projects: Nodes</h4>
      <Pie
        {useAltColors}
        canvasId="{canvasPrefix}-hpcpie-nodes-projects"
        size={colWidthNodes * 0.75}
-        sliceLabel={'Nodes'}
-        quantities={$topNodesQuery.data.topProjects.map(
-          (tp) => tp['totalNodes'],
+        sliceLabel={"Nodes"}
+        quantities={topProjectNodes.map(
+          (tp) => tp["totalNodes"],
+        )}
+        entities={topProjectNodes.map((tp) =>
+          scrambleNames ? scramble(tp.id) : tp.id,
        )}
-        entities={$topNodesQuery.data.topProjects.map((tp) => scrambleNames ? scramble(tp.id) : tp.id)}
      />
    </Col>
    <Col xs="6" md="3" lg="2" class="p-2">
@@ -438,34 +433,44 @@
          <th style="padding-left: 0.5rem;">Project</th>
          <th>Nodes</th>
        </tr>
-        {#each $topNodesQuery.data.topProjects as tp, i}
+        {#each topProjectNodes as tp, i}
          <tr>
-            <td><Icon name="circle-fill" style="color: {legendColors(i)};" /></td>
+            <td
+              ><Icon name="circle-fill" style="color: {legendColors(i)};" /></td
+            >
            <td>
-              <a target="_blank" href="/monitoring/jobs/?cluster={presetCluster}{presetSubCluster ? '&partition='+presetSubCluster : ''}&state=running&project={tp.id}&projectMatch=eq"
+              <a
+                target="_blank"
+                href="/monitoring/jobs/?cluster={presetCluster}{presetSubCluster
+                  ? '&partition=' + presetSubCluster
+                  : ''}&state=running&project={tp.id}&projectMatch=eq"
                >{scrambleNames ? scramble(tp.id) : tp.id}
              </a>
            </td>
-            <td>{tp['totalNodes']}</td>
+            <td>{tp["totalNodes"]}</td>
          </tr>
        {/each}
      </Table>
    </Col>
  </Row>
 {:else}
-  <Card class="mx-4" body color="warning">Cannot render node status charts: No data!</Card>
+  <Card class="mx-4" body color="warning"
+    >Cannot render node status charts: No data!</Card
+  >
 {/if}

 <hr />

 <!-- Acc Distribution, Top Users and Projects-->
-{#if $topAccsQuery?.fetching || $nodeStatusQuery?.fetching}
+{#if $topStatsQuery?.fetching || $nodeStatusQuery?.fetching}
  <Spinner />
-{:else if $topAccsQuery?.data && $nodeStatusQuery?.data}
+{:else if $topStatsQuery?.data && $nodeStatusQuery?.data}
  <Row>
    <Col xs="12" lg="4" class="p-2">
      <Histogram
-        data={convert2uplot($nodeStatusQuery.data.jobsStatistics[0].histNumAccs)}
+        data={convert2uplot(
+          $nodeStatusQuery?.data?.jobsStatistics[0]?.histNumAccs,
+        )}
        title="Number of Accelerators Distribution"
        xlabel="Allocated Accs"
        xunit="Accs"
@@ -477,18 +482,18 @@
    </Col>
    <Col xs="6" md="3" lg="2" class="p-2">
      <div bind:clientWidth={colWidthAccs}>
-        <h4 class="text-center">
-          Top Users: GPUs
-        </h4>
+        <h4 class="text-center">Top Users: GPUs</h4>
        <Pie
          {useAltColors}
          canvasId="{canvasPrefix}-hpcpie-accs-users"
          size={colWidthAccs * 0.75}
          sliceLabel="GPUs"
-          quantities={$topAccsQuery.data.topUser.map(
-            (tu) => tu['totalAccs'],
+          quantities={topUserAccs.map(
+            (tu) => tu["totalAccs"],
+          )}
+          entities={topUserAccs.map((tu) =>
+            scrambleNames ? scramble(tu.id) : tu.id,
          )}
-          entities={$topAccsQuery.data.topUser.map((tu) => scrambleNames ? scramble(tu.id) : tu.id)}
        />
      </div>
    </Col>
@@ -499,11 +504,17 @@
          <th style="padding-left: 0.5rem;">User</th>
          <th>GPUs</th>
        </tr>
-        {#each $topAccsQuery.data.topUser as tu, i}
+        {#each topUserAccs as tu, i}
          <tr>
-            <td><Icon name="circle-fill" style="color: {legendColors(i)};" /></td>
+            <td
+              ><Icon name="circle-fill" style="color: {legendColors(i)};" /></td
+            >
            <td id="{canvasPrefix}-topName-accs-{tu.id}">
-              <a target="_blank" href="/monitoring/user/{tu.id}?cluster={presetCluster}{presetSubCluster ? '&partition='+presetSubCluster : ''}&state=running"
+              <a
+                target="_blank"
+                href="/monitoring/user/{tu.id}?cluster={presetCluster}{presetSubCluster
+                  ? '&partition=' + presetSubCluster
+                  : ''}&state=running"
                >{scrambleNames ? scramble(tu.id) : tu.id}
              </a>
            </td>
@@ -514,25 +525,25 @@
                >{scrambleNames ? scramble(tu.name) : tu.name}</Tooltip
              >
            {/if}
-            <td>{tu['totalAccs']}</td>
+            <td>{tu["totalAccs"]}</td>
          </tr>
        {/each}
      </Table>
    </Col>

    <Col xs="6" md="3" lg="2" class="p-2">
-      <h4 class="text-center">
-        Top Projects: GPUs
-      </h4>
+      <h4 class="text-center">Top Projects: GPUs</h4>
      <Pie
        {useAltColors}
        canvasId="{canvasPrefix}-hpcpie-accs-projects"
        size={colWidthAccs * 0.75}
-        sliceLabel={'GPUs'}
-        quantities={$topAccsQuery.data.topProjects.map(
-          (tp) => tp['totalAccs'],
+        sliceLabel={"GPUs"}
+        quantities={topProjectAccs.map(
+          (tp) => tp["totalAccs"],
+        )}
+        entities={topProjectAccs.map((tp) =>
+          scrambleNames ? scramble(tp.id) : tp.id,
        )}
-        entities={$topAccsQuery.data.topProjects.map((tp) => scrambleNames ? scramble(tp.id) : tp.id)}
      />
    </Col>
    <Col xs="6" md="3" lg="2" class="p-2">
@@ -542,20 +553,28 @@
          <th style="padding-left: 0.5rem;">Project</th>
          <th>GPUs</th>
        </tr>
-        {#each $topAccsQuery.data.topProjects as tp, i}
+        {#each topProjectAccs as tp, i}
          <tr>
-            <td><Icon name="circle-fill" style="color: {legendColors(i)};" /></td>
+            <td
+              ><Icon name="circle-fill" style="color: {legendColors(i)};" /></td
+            >
            <td>
-              <a target="_blank" href="/monitoring/jobs/?cluster={presetCluster}{presetSubCluster ? '&partition='+presetSubCluster : ''}&state=running&project={tp.id}&projectMatch=eq"
+              <a
+                target="_blank"
+                href="/monitoring/jobs/?cluster={presetCluster}{presetSubCluster
+                  ? '&partition=' + presetSubCluster
+                  : ''}&state=running&project={tp.id}&projectMatch=eq"
                >{scrambleNames ? scramble(tp.id) : tp.id}
              </a>
            </td>
-            <td>{tp['totalAccs']}</td>
+            <td>{tp["totalAccs"]}</td>
          </tr>
        {/each}
      </Table>
    </Col>
  </Row>
 {:else}
-  <Card class="mx-4" body color="warning">Cannot render accelerator status charts: No data!</Card>
+  <Card class="mx-4" body color="warning"
+    >Cannot render accelerator status charts: No data!</Card
+  >
 {/if}
@@ -175,13 +175,13 @@
          {:else if item?.data}
            <!-- "Empty Series"-Warning included in MetricPlot-Component   -->
            <!-- #key: X-axis keeps last selected timerange otherwise -->
-            {#key item.data[0].metric.series[0].data.length}
+            {#key item?.data[0]?.metric?.series[0]?.data?.length}
              <MetricPlot
-                timestep={item.data[0].metric.timestep}
-                series={item.data[0].metric.series}
-                metric={item.data[0].name}
+                timestep={item?.data[0]?.metric?.timestep || 60}
+                series={item?.data[0]?.metric?.series || []}
+                metric={item?.data[0]?.name || 'unknown'}
                {cluster}
-                subCluster={item.subCluster}
+                subCluster={item?.subCluster || 'unknown'}
                forNode
                enableFlip
              />
@@ -109,7 +109,7 @@
            <Button class="flex-grow-1" color="warning" disabled>
              Missing Metric
            </Button>
-          {:else if nodeJobsData.jobs.count == 1 && nodeJobsData.jobs.items[0].shared == "none"}
+          {:else if nodeJobsData.jobs.count == 1 && nodeJobsData?.jobs?.items[0]?.shared == "none"}
            <InputGroupText class="flex-grow-1 flex-lg-grow-0">
              <Icon name="circle-fill" style="padding-right: 0.5rem;"/>
              <span>Jobs</span>
@@ -117,7 +117,7 @@
            <Button class="flex-grow-1" color="success" disabled>
              Exclusive
            </Button>
-          {:else if nodeJobsData.jobs.count >= 1 && !(nodeJobsData.jobs.items[0].shared == "none")}
+          {:else if nodeJobsData.jobs.count >= 1 && !(nodeJobsData?.jobs?.items[0]?.shared == "none")}
            <InputGroupText class="flex-grow-1 flex-lg-grow-0">
              <Icon name="circle-half" style="padding-right: 0.5rem;"/>
              <span>Jobs</span>
@@ -63,8 +63,8 @@
  /* Derived */
  const filter = $derived([
    { cluster: { eq: cluster } },
-    { node: { contains: nodeData.host } },
    { state: ["running"] },
+    { node: { contains: nodeData.host } },
  ]);
  const nodeJobsData = $derived(queryStore({
      client: client,
				`@@ -0,0 +1 @@`
				`There are memory leaks in production on queries that trigger a full scan. Review all queries for missing close and other bugs that could cause memory leaks in the sqlite-3 driver. Example for a query that triggers a massive allocation is a specific jobstate list with sorting on start_time on all jobs.`
				`@@ -0,0 +1 @@`
				`There also have to be bugs in jobQuery.go . Especially the following query triggers the memory leak: SELECT * FROM job WHERE job.job_state IN ("completed", "running", "failed") ORDER BY job.start_time DESC LIMIT 1 OFFSET 10; Dig deeper to find the cause. Also investigate why no existing index is used for this query.`