Implement the following plan:

# Optimize Job Table Indexes for 20M Row Production Database

## Context

The `job` table has **79 indexes** (created in migrations 08/09), causing:
1. **Wrong index selection** — without `ANALYZE` statistics, SQLite picks wrong indexes (e.g., `jobs_jobstate_energy` instead of `jobs_starttime` for ORDER BY queries), causing full-table temp B-tree sorts on 20M rows → timeouts
2. **Excessive disk/memory overhead** — each index costs ~200-400MB at 20M rows; 79 indexes = ~16-32GB wasted
3. **Slower writes** — every INSERT/UPDATE touches all 79 indexes
4. **Planner confusion** — too many similar indexes make the query planner's cost estimation unreliable

The `ANALYZE` fix (already added to `setupSqlite` in `dbConnection.go`) resolves the planner issue with current indexes, but the index count must be reduced for disk/write performance.

## Approach: Reduce to 20 indexes

The key insight from query plan analysis: with `ANALYZE` and `LIMIT`, a `(filter_col, sort_col)` index is often better than `(filter_col1, filter_col2, sort_col)` because SQLite can scan the index in sort order and cheaply filter non-matching rows, stopping at LIMIT.

### Verified query plans (with ANALYZE, after this change)

| # | Pattern | Index Used | Plan |
|---|---------|-----------|------|
| 1 | Multi-state IN + ORDER BY start_time LIMIT | `jobs_starttime` | SCAN (index order, no sort) |
| 2 | cluster + state + sort start_time | `jobs_cluster_starttime_duration` | SEARCH |
| 3 | hpc_user + sort start_time | `jobs_user_starttime_duration` | SEARCH |
| 4 | cluster + state aggregation | `jobs_cluster_jobstate_duration_starttime` | COVERING SEARCH |
| 5 | Unique lookup (job_id,cluster,start_time) | `sqlite_autoindex_job_1` | SEARCH |
| 6 | Running jobs for cluster + duration > | `jobs_cluster_jobstate_duration_starttime` | SEARCH |
| 7 | start_time BETWEEN range | `jobs_starttime` | SEARCH |
| 8 | GROUP BY user with cluster | `jobs_cluster_user` | COVERING SEARCH |
| 9 | Concurrent jobs (cluster + start_time <) | `jobs_cluster_starttime_duration` | SEARCH |
| 10 | project IN + state IN + sort | `jobs_jobstate_project` | SEARCH + temp sort |
| 11 | user + multi-state + sort start_time | `jobs_user_starttime_duration` | SEARCH |
| 12 | cluster + state + sort duration | `jobs_cluster_jobstate_duration_starttime` | SEARCH |
| 13 | cluster + state + sort num_nodes | `jobs_cluster_numnodes` | SEARCH (state filtered per-row) |
| 14 | Tag join | `tags_tagid` + PK | SEARCH |
| 15 | Delete before timestamp | `jobs_starttime` | COVERING SEARCH |
| 16 | Non-running jobs (GetJobList) | `jobs_jobstate_duration_starttime` | COVERING SCAN |

## Changes Required

### File: `internal/repository/migrations/sqlite3/11_optimize-indexes.up.sql` (new)

```sql
-- Drop all 77 job indexes from migration 09 (sqlite_autoindex_job_1 is UNIQUE, kept)
-- Then create optimized set of 20

-- GROUP 1: Global (1 index)
-- #1 jobs_starttime (start_time)
--    Default sort for unfiltered/multi-state queries, time range, delete-before

-- GROUP 2: Cluster-prefixed (8 indexes)
-- #2 jobs_cluster_starttime_duration (cluster, start_time, duration)
--    Cluster + default sort, concurrent jobs, time range within cluster
-- #3 jobs_cluster_duration_starttime (cluster, duration, start_time)
--    Cluster + sort by duration
-- #4 jobs_cluster_jobstate_duration_starttime (cluster, job_state, duration, start_time)
--    COVERING for cluster+state aggregation; running jobs (cluster, state, duration>?)
-- #5 jobs_cluster_jobstate_starttime_duration (cluster, job_state, start_time, duration)
--    Cluster+state+sort start_time (single state equality)
-- #6 jobs_cluster_user (cluster, hpc_user)
--    COVERING for GROUP BY user with cluster filter
-- #7 jobs_cluster_project (cluster, project)
--    GROUP BY project with cluster filter
-- #8 jobs_cluster_subcluster (cluster, subcluster)
--    GROUP BY subcluster with cluster filter
-- #9 jobs_cluster_numnodes (cluster, num_nodes)
--    Cluster + sort by num_nodes (state filtered per-row, fast with LIMIT)

-- GROUP 3: User-prefixed (1 index)
-- #10 jobs_user_starttime_duration (hpc_user, start_time, duration)
--     Security filter (user role) + default sort

-- GROUP 4: Project-prefixed (1 index)
-- #11 jobs_project_starttime_duration (project, start_time, duration)
--     Security filter (manager role) + default sort

-- GROUP 5: JobState-prefixed (3 indexes)
-- #12 jobs_jobstate_project (job_state, project)
--     State + project filter (for manager security within state query)
-- #13 jobs_jobstate_user (job_state, hpc_user)
--     State + user filter/aggregation
-- #14 jobs_jobstate_duration_starttime (job_state, duration, start_time)
--     COVERING for non-running jobs scan, state + sort duration

-- GROUP 6: Rare filters (1 index)
-- #15 jobs_arrayjobid (array_job_id)
--     Array job lookup (rare but must be indexed)

-- GROUP 7: Secondary sort columns (5 indexes)
-- #16 jobs_cluster_numhwthreads (cluster, num_hwthreads)
-- #17 jobs_cluster_numacc (cluster, num_acc)
-- #18 jobs_cluster_energy (cluster, energy)
-- #19 jobs_cluster_partition_starttime (cluster, cluster_partition, start_time)
--     Cluster+partition + sort start_time
-- #20 jobs_cluster_partition_jobstate (cluster, cluster_partition, job_state)
--     Cluster+partition+state filter
```

### What's dropped and why (59 indexes removed)

| Category | Count | Why redundant |
|----------|-------|---------------|
| cluster+partition sort/filter variants | 8 | Kept only 2 partition indexes (#19, #20); rest use cluster indexes + row filter |
| cluster+shared (all) | 8 | `shared` is rare; cluster index + row filter is fast |
| shared-prefixed (all) | 8 | `shared` alone is never a leading filter |
| cluster+jobstate sort variants (numnodes, hwthreads, acc, energy) | 4 | Replaced by `(cluster, sort_col)` indexes which work for any state combo with LIMIT |
| user sort variants (numnodes, hwthreads, acc, energy, duration) | 5 | User result sets are small; temp sort is fast |
| project sort variants + project_user | 6 | Same reasoning as user |
| jobstate sort variants (numnodes, hwthreads, acc, energy) | 4 | State has low cardinality; cluster+sort indexes handle these |
| single-filter+starttime (5) + single-filter+duration (5) | 10 | Queries always have cluster/user/project filter; standalone rare |
| standalone duration | 1 | Covered by cluster_duration_starttime |
| duplicate arrayjob variants | 1 | Simplified to single-column (array_job_id) |
| redundant cluster_starttime variants | 2 | Consolidated into 2 cluster+time indexes |
| cluster_jobstate_user, cluster_jobstate_project | 2 | Covered by cluster_user/cluster_project + state row filter |

### File: `internal/repository/migrations/sqlite3/11_optimize-indexes.down.sql` (new)

Recreate all 77 indexes from migration 09 for safe rollback.

### File: `internal/repository/migration.go`

Increment `Version` from `10` to `11`.

## Verification

1. `go build ./...` — compiles
2. `go test ./internal/repository/...` — tests pass
3. `cc-backend -migrate-db` on a test copy of production DB
4. After migration, run `ANALYZE;` then verify all 16 query plans match the table above using:
   ```sql
   EXPLAIN QUERY PLAN SELECT * FROM job WHERE job.job_state IN ('completed','running','failed') ORDER BY job.start_time DESC LIMIT 50;
   -- Should show: SCAN job USING INDEX jobs_starttime
   ```
5. Verify index count: `SELECT COUNT(*) FROM sqlite_master WHERE type='index' AND tbl_name='job';` → should be 21 (20 + autoindex)
6. Compare DB file size before/after (expect ~70% reduction in index overhead)

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-CC-cc-backend/42401d2e-7d1c-4c0e-abe6-356cb2d48747.jsonl


Let me start by reading the existing files I need to understand.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package repository
     7→
     8→import (
     9→	"database/sql"
    10→	"embed"
    11→	"fmt"
    12→
    13→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    14→	"github.com/golang-migrate/migrate/v4"
    15→	"github.com/golang-migrate/migrate/v4/database/sqlite3"
    16→	"github.com/golang-migrate/migrate/v4/source/iofs"
    17→)
    18→
    19→// Version is the current database schema version required by this version of cc-backend.
    20→// When the database schema changes, this version is incremented and a new migration file
    21→// is added to internal/repository/migrations/sqlite3/.
    22→//
    23→// Version history:
    24→//   - Version 10: Current version
    25→//
    26→// Migration files are embedded at build time from the migrations directory.
    27→const Version uint = 10
    28→
    29→//go:embed migrations/*
    30→var migrationFiles embed.FS
    31→
    32→// checkDBVersion verifies that the database schema version matches the expected version.
    33→// This is called automatically during Connect() to ensure schema compatibility.
    34→//
    35→// Returns an error if:
    36→//   - Database version is older than expected (needs migration)
    37→//   - Database version is newer than expected (needs app upgrade)
    38→//   - Database is in a dirty state (failed migration)
    39→//
    40→// A "dirty" database indicates a migration was started but not completed successfully.
    41→// This requires manual intervention to fix the database and force the version.
    42→func checkDBVersion(db *sql.DB) error {
    43→	driver, err := sqlite3.WithInstance(db, &sqlite3.Config{})
    44→	if err != nil {
    45→		return err
    46→	}
    47→	d, err := iofs.New(migrationFiles, "migrations/sqlite3")
    48→	if err != nil {
    49→		return err
    50→	}
    51→
    52→	m, err := migrate.NewWithInstance("iofs", d, "sqlite3", driver)
    53→	if err != nil {
    54→		return err
    55→	}
    56→
    57→	v, dirty, err := m.Version()
    58→	if err != nil {
    59→		if err == migrate.ErrNilVersion {
    60→			cclog.Warn("Legacy database without version or missing database file!")
    61→		} else {
    62→			return err
    63→		}
    64→	}
    65→
    66→	if v < Version {
    67→		return fmt.Errorf("unsupported database version %d, need %d.\nPlease backup your database file and run cc-backend -migrate-db", v, Version)
    68→	} else if v > Version {
    69→		return fmt.Errorf("unsupported database version %d, need %d.\nPlease refer to documentation how to downgrade db with external migrate tool", v, Version)
    70→	}
    71→
    72→	if dirty {
    73→		return fmt.Errorf("last migration to version %d has failed, please fix the db manually and force version with -force-db flag", Version)
    74→	}
    75→
    76→	return nil
    77→}
    78→
    79→// getMigrateInstance creates a new migration instance for the given database file.
    80→// This is used internally by MigrateDB, RevertDB, and ForceDB.
    81→func getMigrateInstance(db string) (m *migrate.Migrate, err error) {
    82→	d, err := iofs.New(migrationFiles, "migrations/sqlite3")
    83→	if err != nil {
    84→		return nil, err
    85→	}
    86→
    87→	m, err = migrate.NewWithSourceInstance("iofs", d, fmt.Sprintf("sqlite3://%s?_foreign_keys=on", db))
    88→	if err != nil {
    89→		return nil, err
    90→	}
    91→
    92→	return m, nil
    93→}
    94→
    95→// MigrateDB applies all pending database migrations to bring the schema up to date.
    96→// This should be run with the -migrate-db flag before starting the application
    97→// after upgrading to a new version that requires schema changes.
    98→//
    99→// Process:
   100→//  1. Checks current database version
   101→//  2. Applies all migrations from current version to target Version
   102→//  3. Updates schema_migrations table to track applied migrations
   103→//
   104→// Important:
   105→//   - Always backup your database before running migrations
   106→//   - Migrations are irreversible without manual intervention
   107→//   - If a migration fails, the database is marked "dirty" and requires manual fix
   108→//
   109→// Usage:
   110→//
   111→//	cc-backend -migrate-db
   112→func MigrateDB(db string) error {
   113→	m, err := getMigrateInstance(db)
   114→	if err != nil {
   115→		return err
   116→	}
   117→
   118→	v, dirty, err := m.Version()
   119→	if err != nil {
   120→		if err == migrate.ErrNilVersion {
   121→			cclog.Info("Legacy database without version or missing database file!")
   122→		} else {
   123→			return err
   124→		}
   125→	}
   126→
   127→	if v < Version {
   128→		cclog.Infof("unsupported database version %d, need %d.\nPlease backup your database file and run cc-backend -migrate-db", v, Version)
   129→	}
   130→
   131→	if dirty {
   132→		return fmt.Errorf("last migration to version %d has failed, please fix the db manually and force version with -force-db flag", Version)
   133→	}
   134→
   135→	if err := m.Up(); err != nil {
   136→		if err == migrate.ErrNoChange {
   137→			cclog.Info("DB already up to date!")
   138→		} else {
   139→			return err
   140→		}
   141→	}
   142→
   143→	m.Close()
   144→	return nil
   145→}
   146→
   147→// RevertDB rolls back the database schema to the previous version (Version - 1).
   148→// This is primarily used for testing or emergency rollback scenarios.
   149→//
   150→// Warning:
   151→//   - This may cause data loss if newer schema added columns/tables
   152→//   - Always backup before reverting
   153→//   - Not all migrations are safely reversible
   154→//
   155→// Usage:
   156→//
   157→//	cc-backend -revert-db
   158→func RevertDB(db string) error {
   159→	m, err := getMigrateInstance(db)
   160→	if err != nil {
   161→		return err
   162→	}
   163→
   164→	if err := m.Migrate(Version - 1); err != nil {
   165→		if err == migrate.ErrNoChange {
   166→			cclog.Info("DB already up to date!")
   167→		} else {
   168→			return err
   169→		}
   170→	}
   171→
   172→	m.Close()
   173→	return nil
   174→}
   175→
   176→// ForceDB forces the database schema version to the current Version without running migrations.
   177→// This is only used to recover from failed migrations that left the database in a "dirty" state.
   178→//
   179→// When to use:
   180→//   - After manually fixing a failed migration
   181→//   - When you've manually applied schema changes and need to update the version marker
   182→//
   183→// Warning:
   184→//   - This does NOT apply any schema changes
   185→//   - Only use after manually verifying the schema is correct
   186→//   - Improper use can cause schema/version mismatch
   187→//
   188→// Usage:
   189→//
   190→//	cc-backend -force-db
   191→func ForceDB(db string) error {
   192→	m, err := getMigrateInstance(db)
   193→	if err != nil {
   194→		return err
   195→	}
   196→
   197→	if err := m.Force(int(Version)); err != nil {
   198→		return err
   199→	}
   200→
   201→	m.Close()
   202→	return nil
   203→}
   204→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package repository

import (
	"database/sql"
	"embed"
	"fmt"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/golang-migrate/migrate/v4"
	"github.com/golang-migrate/migrate/v4/database/sqlite3"
	"github.com/golang-migrate/migrate/v4/source/iofs"
)

// Version is the current database schema version required by this version of cc-backend.
// When the database schema changes, this version is incremented and a new migration file
// is added to internal/repository/migrations/sqlite3/.
//
// Version history:
//   - Version 10: Current version
//
// Migration files are embedded at build time from the migrations directory.
const Version uint = 10

//go:embed migrations/*
var migrationFiles embed.FS

// checkDBVersion verifies that the database schema version matches the expected version.
// This is called automatically during Connect() to ensure schema compatibility.
//
// Returns an error if:
//   - Database version is older than expected (needs migration)
//   - Database version is newer than expected (needs app upgrade)
//   - Database is in a dirty state (failed migration)
//
// A "dirty" database indicates a migration was started but not completed successfully.
// This requires manual intervention to fix the database and force the version.
func checkDBVersion(db *sql.DB) error {
	driver, err := sqlite3.WithInstance(db, &sqlite3.Config{})
	if err != nil {
		return err
	}
	d, err := iofs.New(migrationFiles, "migrations/sqlite3")
	if err != nil {
		return err
	}

	m, err := migrate.NewWithInstance("iofs", d, "sqlite3", driver)
	if err != nil {
		return err
	}

	v, dirty, err := m.Version()
	if err != nil {
		if err == migrate.ErrNilVersion {
			cclog.Warn("Legacy database without version or missing database file!")
		} else {
			return err
		}
	}

	if v < Version {
		return fmt.Errorf("unsupported database version %d, need %d.\nPlease backup your database file and run cc-backend -migrate-db", v, Version)
	} else if v > Version {
		return fmt.Errorf("unsupported database version %d, need %d.\nPlease refer to documentation how to downgrade db with external migrate tool", v, Version)
	}

	if dirty {
		return fmt.Errorf("last migration to version %d has failed, please fix the db manually and force version with -force-db flag", Version)
	}

	return nil
}

// getMigrateInstance creates a new migration instance for the given database file.
// This is used internally by MigrateDB, RevertDB, and ForceDB.
func getMigrateInstance(db string) (m *migrate.Migrate, err error) {
	d, err := iofs.New(migrationFiles, "migrations/sqlite3")
	if err != nil {
		return nil, err
	}

	m, err = migrate.NewWithSourceInstance("iofs", d, fmt.Sprintf("sqlite3://%s?_foreign_keys=on", db))
	if err != nil {
		return nil, err
	}

	return m, nil
}

// MigrateDB applies all pending database migrations to bring the schema up to date.
// This should be run with the -migrate-db flag before starting the application
// after upgrading to a new version that requires schema changes.
//
// Process:
//  1. Checks current database version
//  2. Applies all migrations from current version to target Version
//  3. Updates schema_migrations table to track applied migrations
//
// Important:
//   - Always backup your database before running migrations
//   - Migrations are irreversible without manual intervention
//   - If a migration fails, the database is marked "dirty" and requires manual fix
//
// Usage:
//
//	cc-backend -migrate-db
func MigrateDB(db string) error {
	m, err := getMigrateInstance(db)
	if err != nil {
		return err
	}

	v, dirty, err := m.Version()
	if err != nil {
		if err == migrate.ErrNilVersion {
			cclog.Info("Legacy database without version or missing database file!")
		} else {
			return err
		}
	}

	if v < Version {
		cclog.Infof("unsupported database version %d, need %d.\nPlease backup your database file and run cc-backend -migrate-db", v, Version)
	}

	if dirty {
		return fmt.Errorf("last migration to version %d has failed, please fix the db manually and force version with -force-db flag", Version)
	}

	if err := m.Up(); err != nil {
		if err == migrate.ErrNoChange {
			cclog.Info("DB already up to date!")
		} else {
			return err
		}
	}

	m.Close()
	return nil
}

// RevertDB rolls back the database schema to the previous version (Version - 1).
// This is primarily used for testing or emergency rollback scenarios.
//
// Warning:
//   - This may cause data loss if newer schema added columns/tables
//   - Always backup before reverting
//   - Not all migrations are safely reversible
//
// Usage:
//
//	cc-backend -revert-db
func RevertDB(db string) error {
	m, err := getMigrateInstance(db)
	if err != nil {
		return err
	}

	if err := m.Migrate(Version - 1); err != nil {
		if err == migrate.ErrNoChange {
			cclog.Info("DB already up to date!")
		} else {
			return err
		}
	}

	m.Close()
	return nil
}

// ForceDB forces the database schema version to the current Version without running migrations.
// This is only used to recover from failed migrations that left the database in a "dirty" state.
//
// When to use:
//   - After manually fixing a failed migration
//   - When you've manually applied schema changes and need to update the version marker
//
// Warning:
//   - This does NOT apply any schema changes
//   - Only use after manually verifying the schema is correct
//   - Improper use can cause schema/version mismatch
//
// Usage:
//
//	cc-backend -force-db
func ForceDB(db string) error {
	m, err := getMigrateInstance(db)
	if err != nil {
		return err
	}

	if err := m.Force(int(Version)); err != nil {
		return err
	}

	m.Close()
	return nil
}


Bash

List migration files


01_init-schema.down.sql
01_init-schema.up.sql
02_add-index.down.sql
02_add-index.up.sql
03_add-userprojects.down.sql
03_add-userprojects.up.sql
04_add-constraints.down.sql
04_add-constraints.up.sql
05_extend-tags.down.sql
05_extend-tags.up.sql
06_change-config.down.sql
06_change-config.up.sql
07_fix-tag-id.down.sql
07_fix-tag-id.up.sql
08_add-footprint.down.sql
08_add-footprint.up.sql
09_add-job-cache.down.sql
09_add-job-cache.up.sql
10_node-table.down.sql
10_node-table.up.sql


Now let me read the migration files that created the indexes.


     1→CREATE TABLE "job_cache" (
     2→    id INTEGER PRIMARY KEY,
     3→    job_id BIGINT NOT NULL,
     4→    cluster VARCHAR(255) NOT NULL,
     5→    subcluster VARCHAR(255) NOT NULL,
     6→    submit_time BIGINT NOT NULL DEFAULT 0, -- Unix timestamp
     7→    start_time BIGINT NOT NULL DEFAULT 0, -- Unix timestamp
     8→    hpc_user VARCHAR(255) NOT NULL,
     9→    project VARCHAR(255) NOT NULL,
    10→    cluster_partition VARCHAR(255),
    11→    array_job_id BIGINT,
    12→    duration INT NOT NULL,
    13→    walltime INT NOT NULL,
    14→    job_state VARCHAR(255) NOT NULL
    15→    CHECK (job_state IN (
    16→        'boot_fail', 'cancelled', 'completed', 'deadline',
    17→        'failed', 'node_fail', 'out_of_memory', 'pending',
    18→        'preempted', 'running', 'suspended', 'timeout'
    19→    )),
    20→    meta_data TEXT,          -- JSON
    21→    resources TEXT NOT NULL, -- JSON
    22→    num_nodes INT NOT NULL,
    23→    num_hwthreads INT,
    24→    num_acc INT,
    25→    smt TINYINT NOT NULL DEFAULT 1 CHECK (smt IN (0, 1)),
    26→    shared TEXT NOT NULL
    27→    CHECK (shared IN ("none", "single_user", "multi_user")),
    28→    monitoring_status TINYINT NOT NULL DEFAULT 1
    29→    CHECK (monitoring_status IN (0, 1, 2, 3)),
    30→    energy REAL NOT NULL DEFAULT 0.0,
    31→    energy_footprint TEXT DEFAULT NULL,
    32→    footprint TEXT DEFAULT NULL,
    33→    UNIQUE (job_id, cluster, start_time)
    34→);
    35→
    36→CREATE TABLE "job_new" (
    37→    id INTEGER PRIMARY KEY,
    38→    job_id BIGINT NOT NULL,
    39→    cluster TEXT NOT NULL,
    40→    subcluster TEXT NOT NULL,
    41→    submit_time BIGINT NOT NULL DEFAULT 0, -- Unix timestamp
    42→    start_time BIGINT NOT NULL DEFAULT 0, -- Unix timestamp
    43→    hpc_user TEXT NOT NULL,
    44→    project TEXT NOT NULL,
    45→    cluster_partition TEXT,
    46→    array_job_id BIGINT,
    47→    duration INT NOT NULL,
    48→    walltime INT NOT NULL,
    49→    job_state TEXT NOT NULL
    50→    CHECK (job_state IN (
    51→        'boot_fail', 'cancelled', 'completed', 'deadline',
    52→        'failed', 'node_fail', 'out_of_memory', 'pending',
    53→        'preempted', 'running', 'suspended', 'timeout'
    54→    )),
    55→    meta_data TEXT,          -- JSON
    56→    resources TEXT NOT NULL, -- JSON
    57→    num_nodes INT NOT NULL,
    58→    num_hwthreads INT,
    59→    num_acc INT,
    60→    smt INT NOT NULL DEFAULT 1,
    61→    shared TEXT NOT NULL
    62→    CHECK (shared IN ("none", "single_user", "multi_user")),
    63→    monitoring_status TINYINT NOT NULL DEFAULT 1
    64→    CHECK (monitoring_status IN (0, 1, 2, 3)),
    65→    energy REAL NOT NULL DEFAULT 0.0,
    66→    energy_footprint TEXT DEFAULT NULL,
    67→    footprint TEXT DEFAULT NULL,
    68→    UNIQUE (job_id, cluster, start_time)
    69→);
    70→
    71→
    72→CREATE TABLE IF NOT EXISTS lookup_exclusive (
    73→    id INTEGER PRIMARY KEY,
    74→    name TEXT NOT NULL UNIQUE
    75→);
    76→
    77→INSERT INTO lookup_exclusive (id, name) VALUES
    78→(0, 'multi_user'),
    79→(1, 'none'),
    80→(2, 'single_user');
    81→
    82→INSERT INTO job_new (
    83→    id, job_id, cluster, subcluster, submit_time, start_time, hpc_user, project,
    84→    cluster_partition, array_job_id, duration, walltime, job_state, meta_data, resources,
    85→    num_nodes, num_hwthreads, num_acc, smt, shared, monitoring_status, energy,
    86→    energy_footprint, footprint
    87→) SELECT
    88→    id,
    89→    job_id,
    90→    cluster,
    91→    subcluster,
    92→    0,
    93→    start_time,
    94→    hpc_user,
    95→    project,
    96→    cluster_partition,
    97→    array_job_id,
    98→    duration,
    99→    walltime,
   100→    job_state,
   101→    meta_data,
   102→    resources,
   103→    num_nodes,
   104→    num_hwthreads,
   105→    num_acc,
   106→    smt,
   107→    (
   108→        SELECT name FROM lookup_exclusive
   109→        WHERE id = job.exclusive
   110→    ),
   111→    monitoring_status,
   112→    energy,
   113→    energy_footprint,
   114→    footprint
   115→FROM job;
   116→
   117→DROP TABLE lookup_exclusive;
   118→DROP TABLE job; -- Deletes All Existing 'job' Indices; Recreate after Renaming
   119→ALTER TABLE job_new RENAME TO job;
   120→
   121→-- Recreate Indices from 08_add-footprint; include new 'shared' column
   122→-- Cluster Filter
   123→CREATE INDEX IF NOT EXISTS jobs_cluster_user ON job (cluster, hpc_user);
   124→CREATE INDEX IF NOT EXISTS jobs_cluster_project ON job (cluster, project);
   125→CREATE INDEX IF NOT EXISTS jobs_cluster_subcluster ON job (cluster, subcluster);
   126→-- Cluster Filter Sorting
   127→CREATE INDEX IF NOT EXISTS jobs_cluster_numnodes ON job (cluster, num_nodes);
   128→CREATE INDEX IF NOT EXISTS jobs_cluster_numhwthreads ON job (cluster, num_hwthreads);
   129→CREATE INDEX IF NOT EXISTS jobs_cluster_numacc ON job (cluster, num_acc);
   130→CREATE INDEX IF NOT EXISTS jobs_cluster_energy ON job (cluster, energy);
   131→
   132→-- Cluster Time Filter Sorting
   133→CREATE INDEX IF NOT EXISTS jobs_cluster_duration_starttime ON job (cluster, duration, start_time);
   134→CREATE INDEX IF NOT EXISTS jobs_cluster_starttime_duration ON job (cluster, start_time, duration);
   135→
   136→-- Cluster+Partition Filter
   137→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_user ON job (cluster, cluster_partition, hpc_user);
   138→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_project ON job (cluster, cluster_partition, project);
   139→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate ON job (cluster, cluster_partition, job_state);
   140→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_shared ON job (cluster, cluster_partition, shared);
   141→
   142→-- Cluster+Partition Filter Sorting
   143→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numnodes ON job (cluster, cluster_partition, num_nodes);
   144→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numhwthreads ON job (cluster, cluster_partition, num_hwthreads);
   145→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numacc ON job (cluster, cluster_partition, num_acc);
   146→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_energy ON job (cluster, cluster_partition, energy);
   147→
   148→-- Cluster+Partition Time Filter Sorting
   149→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_duration_starttime ON job (cluster, cluster_partition, duration, start_time);
   150→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_starttime_duration ON job (cluster, cluster_partition, start_time, duration);
   151→
   152→-- Cluster+JobState Filter
   153→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user ON job (cluster, job_state, hpc_user);
   154→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project ON job (cluster, job_state, project);
   155→-- Cluster+JobState Filter Sorting
   156→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numnodes ON job (cluster, job_state, num_nodes);
   157→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numhwthreads ON job (cluster, job_state, num_hwthreads);
   158→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numacc ON job (cluster, job_state, num_acc);
   159→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_energy ON job (cluster, job_state, energy);
   160→
   161→-- Cluster+JobState Time Filter Sorting
   162→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_starttime_duration ON job (cluster, job_state, start_time, duration);
   163→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_duration_starttime ON job (cluster, job_state, duration, start_time);
   164→
   165→-- Cluster+Shared Filter
   166→CREATE INDEX IF NOT EXISTS jobs_cluster_shared_user ON job (cluster, shared, hpc_user);
   167→CREATE INDEX IF NOT EXISTS jobs_cluster_shared_project ON job (cluster, shared, project);
   168→-- Cluster+Shared Filter Sorting
   169→CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numnodes ON job (cluster, shared, num_nodes);
   170→CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numhwthreads ON job (cluster, shared, num_hwthreads);
   171→CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numacc ON job (cluster, shared, num_acc);
   172→CREATE INDEX IF NOT EXISTS jobs_cluster_shared_energy ON job (cluster, shared, energy);
   173→
   174→-- Cluster+Shared Time Filter Sorting
   175→CREATE INDEX IF NOT EXISTS jobs_cluster_shared_starttime_duration ON job (cluster, shared, start_time, duration);
   176→CREATE INDEX IF NOT EXISTS jobs_cluster_shared_duration_starttime ON job (cluster, shared, duration, start_time);
   177→
   178→-- User Filter
   179→-- User Filter Sorting
   180→CREATE INDEX IF NOT EXISTS jobs_user_numnodes ON job (hpc_user, num_nodes);
   181→CREATE INDEX IF NOT EXISTS jobs_user_numhwthreads ON job (hpc_user, num_hwthreads);
   182→CREATE INDEX IF NOT EXISTS jobs_user_numacc ON job (hpc_user, num_acc);
   183→CREATE INDEX IF NOT EXISTS jobs_user_energy ON job (hpc_user, energy);
   184→
   185→-- Cluster+Shared Time Filter Sorting
   186→CREATE INDEX IF NOT EXISTS jobs_user_starttime_duration ON job (hpc_user, start_time, duration);
   187→CREATE INDEX IF NOT EXISTS jobs_user_duration_starttime ON job (hpc_user, duration, start_time);
   188→
   189→-- Project Filter
   190→CREATE INDEX IF NOT EXISTS jobs_project_user ON job (project, hpc_user);
   191→-- Project Filter Sorting
   192→CREATE INDEX IF NOT EXISTS jobs_project_numnodes ON job (project, num_nodes);
   193→CREATE INDEX IF NOT EXISTS jobs_project_numhwthreads ON job (project, num_hwthreads);
   194→CREATE INDEX IF NOT EXISTS jobs_project_numacc ON job (project, num_acc);
   195→CREATE INDEX IF NOT EXISTS jobs_project_energy ON job (project, energy);
   196→
   197→-- Cluster+Shared Time Filter Sorting
   198→CREATE INDEX IF NOT EXISTS jobs_project_starttime_duration ON job (project, start_time, duration);
   199→CREATE INDEX IF NOT EXISTS jobs_project_duration_starttime ON job (project, duration, start_time);
   200→
   201→-- JobState Filter
   202→CREATE INDEX IF NOT EXISTS jobs_jobstate_user ON job (job_state, hpc_user);
   203→CREATE INDEX IF NOT EXISTS jobs_jobstate_project ON job (job_state, project);
   204→-- JobState Filter Sorting
   205→CREATE INDEX IF NOT EXISTS jobs_jobstate_numnodes ON job (job_state, num_nodes);
   206→CREATE INDEX IF NOT EXISTS jobs_jobstate_numhwthreads ON job (job_state, num_hwthreads);
   207→CREATE INDEX IF NOT EXISTS jobs_jobstate_numacc ON job (job_state, num_acc);
   208→CREATE INDEX IF NOT EXISTS jobs_jobstate_energy ON job (job_state, energy);
   209→
   210→-- Cluster+Shared Time Filter Sorting
   211→CREATE INDEX IF NOT EXISTS jobs_jobstate_starttime_duration ON job (job_state, start_time, duration);
   212→CREATE INDEX IF NOT EXISTS jobs_jobstate_duration_starttime ON job (job_state, duration, start_time);
   213→
   214→-- Shared Filter
   215→CREATE INDEX IF NOT EXISTS jobs_shared_user ON job (shared, hpc_user);
   216→CREATE INDEX IF NOT EXISTS jobs_shared_project ON job (shared, project);
   217→-- Shared Filter Sorting
   218→CREATE INDEX IF NOT EXISTS jobs_shared_numnodes ON job (shared, num_nodes);
   219→CREATE INDEX IF NOT EXISTS jobs_shared_numhwthreads ON job (shared, num_hwthreads);
   220→CREATE INDEX IF NOT EXISTS jobs_shared_numacc ON job (shared, num_acc);
   221→CREATE INDEX IF NOT EXISTS jobs_shared_energy ON job (shared, energy);
   222→
   223→-- Cluster+Shared Time Filter Sorting
   224→CREATE INDEX IF NOT EXISTS jobs_shared_starttime_duration ON job (shared, start_time, duration);
   225→CREATE INDEX IF NOT EXISTS jobs_shared_duration_starttime ON job (shared, duration, start_time);
   226→
   227→-- ArrayJob Filter
   228→CREATE INDEX IF NOT EXISTS jobs_arrayjobid_starttime ON job (array_job_id, start_time);
   229→CREATE INDEX IF NOT EXISTS jobs_cluster_arrayjobid_starttime ON job (cluster, array_job_id, start_time);
   230→
   231→-- Single filters with default starttime sorting
   232→CREATE INDEX IF NOT EXISTS jobs_duration_starttime ON job (duration, start_time);
   233→CREATE INDEX IF NOT EXISTS jobs_numnodes_starttime ON job (num_nodes, start_time);
   234→CREATE INDEX IF NOT EXISTS jobs_numhwthreads_starttime ON job (num_hwthreads, start_time);
   235→CREATE INDEX IF NOT EXISTS jobs_numacc_starttime ON job (num_acc, start_time);
   236→CREATE INDEX IF NOT EXISTS jobs_energy_starttime ON job (energy, start_time);
   237→
   238→-- Single filters with duration sorting
   239→CREATE INDEX IF NOT EXISTS jobs_starttime_duration ON job (start_time, duration);
   240→CREATE INDEX IF NOT EXISTS jobs_numnodes_duration ON job (num_nodes, duration);
   241→CREATE INDEX IF NOT EXISTS jobs_numhwthreads_duration ON job (num_hwthreads, duration);
   242→CREATE INDEX IF NOT EXISTS jobs_numacc_duration ON job (num_acc, duration);
   243→CREATE INDEX IF NOT EXISTS jobs_energy_duration ON job (energy, duration);
   244→
   245→-- Backup Indices For High Variety Columns
   246→CREATE INDEX IF NOT EXISTS jobs_starttime ON job (start_time);
   247→CREATE INDEX IF NOT EXISTS jobs_duration ON job (duration);
   248→
   249→-- Notes:
   250→-- Cluster+Partition+Jobstate Filter: Tested -> Full Array Of Combinations non-required
   251→-- Cluster+JobState+Shared Filter: Tested -> No further timing improvement
   252→-- JobState+Shared Filter: Tested -> No further timing improvement
   253→
   254→-- Optimize DB index usage
   255→PRAGMA optimize;
   256→
   257→-- Optimize DB size: https://sqlite.org/lang_vacuum.html
   258→-- Not allowed within a migration transaction; Keep command here for documentation and recommendation
   259→-- Command: 'VACUUM;'

CREATE TABLE "job_cache" (
    id INTEGER PRIMARY KEY,
    job_id BIGINT NOT NULL,
    cluster VARCHAR(255) NOT NULL,
    subcluster VARCHAR(255) NOT NULL,
    submit_time BIGINT NOT NULL DEFAULT 0, -- Unix timestamp
    start_time BIGINT NOT NULL DEFAULT 0, -- Unix timestamp
    hpc_user VARCHAR(255) NOT NULL,
    project VARCHAR(255) NOT NULL,
    cluster_partition VARCHAR(255),
    array_job_id BIGINT,
    duration INT NOT NULL,
    walltime INT NOT NULL,
    job_state VARCHAR(255) NOT NULL
    CHECK (job_state IN (
        'boot_fail', 'cancelled', 'completed', 'deadline',
        'failed', 'node_fail', 'out_of_memory', 'pending',
        'preempted', 'running', 'suspended', 'timeout'
    )),
    meta_data TEXT,          -- JSON
    resources TEXT NOT NULL, -- JSON
    num_nodes INT NOT NULL,
    num_hwthreads INT,
    num_acc INT,
    smt TINYINT NOT NULL DEFAULT 1 CHECK (smt IN (0, 1)),
    shared TEXT NOT NULL
    CHECK (shared IN ("none", "single_user", "multi_user")),
    monitoring_status TINYINT NOT NULL DEFAULT 1
    CHECK (monitoring_status IN (0, 1, 2, 3)),
    energy REAL NOT NULL DEFAULT 0.0,
    energy_footprint TEXT DEFAULT NULL,
    footprint TEXT DEFAULT NULL,
    UNIQUE (job_id, cluster, start_time)
);

CREATE TABLE "job_new" (
    id INTEGER PRIMARY KEY,
    job_id BIGINT NOT NULL,
    cluster TEXT NOT NULL,
    subcluster TEXT NOT NULL,
    submit_time BIGINT NOT NULL DEFAULT 0, -- Unix timestamp
    start_time BIGINT NOT NULL DEFAULT 0, -- Unix timestamp
    hpc_user TEXT NOT NULL,
    project TEXT NOT NULL,
    cluster_partition TEXT,
    array_job_id BIGINT,
    duration INT NOT NULL,
    walltime INT NOT NULL,
    job_state TEXT NOT NULL
    CHECK (job_state IN (
        'boot_fail', 'cancelled', 'completed', 'deadline',
        'failed', 'node_fail', 'out_of_memory', 'pending',
        'preempted', 'running', 'suspended', 'timeout'
    )),
    meta_data TEXT,          -- JSON
    resources TEXT NOT NULL, -- JSON
    num_nodes INT NOT NULL,
    num_hwthreads INT,
    num_acc INT,
    smt INT NOT NULL DEFAULT 1,
    shared TEXT NOT NULL
    CHECK (shared IN ("none", "single_user", "multi_user")),
    monitoring_status TINYINT NOT NULL DEFAULT 1
    CHECK (monitoring_status IN (0, 1, 2, 3)),
    energy REAL NOT NULL DEFAULT 0.0,
    energy_footprint TEXT DEFAULT NULL,
    footprint TEXT DEFAULT NULL,
    UNIQUE (job_id, cluster, start_time)
);

CREATE TABLE IF NOT EXISTS lookup_exclusive (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL UNIQUE
);

INSERT INTO lookup_exclusive (id, name) VALUES
(0, 'multi_user'),
(1, 'none'),
(2, 'single_user');

INSERT INTO job_new (
    id, job_id, cluster, subcluster, submit_time, start_time, hpc_user, project,
    cluster_partition, array_job_id, duration, walltime, job_state, meta_data, resources,
    num_nodes, num_hwthreads, num_acc, smt, shared, monitoring_status, energy,
    energy_footprint, footprint
) SELECT
    id,
    job_id,
    cluster,
    subcluster,
    0,
    start_time,
    hpc_user,
    project,
    cluster_partition,
    array_job_id,
    duration,
    walltime,
    job_state,
    meta_data,
    resources,
    num_nodes,
    num_hwthreads,
    num_acc,
    smt,
    (
        SELECT name FROM lookup_exclusive
        WHERE id = job.exclusive
    ),
    monitoring_status,
    energy,
    energy_footprint,
    footprint
FROM job;

DROP TABLE lookup_exclusive;
DROP TABLE job; -- Deletes All Existing 'job' Indices; Recreate after Renaming
ALTER TABLE job_new RENAME TO job;

-- Recreate Indices from 08_add-footprint; include new 'shared' column
-- Cluster Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_user ON job (cluster, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_cluster_project ON job (cluster, project);
CREATE INDEX IF NOT EXISTS jobs_cluster_subcluster ON job (cluster, subcluster);
-- Cluster Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_numnodes ON job (cluster, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_numhwthreads ON job (cluster, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_numacc ON job (cluster, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_energy ON job (cluster, energy);

-- Cluster Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_duration_starttime ON job (cluster, duration, start_time);
CREATE INDEX IF NOT EXISTS jobs_cluster_starttime_duration ON job (cluster, start_time, duration);

-- Cluster+Partition Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_user ON job (cluster, cluster_partition, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_project ON job (cluster, cluster_partition, project);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate ON job (cluster, cluster_partition, job_state);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_shared ON job (cluster, cluster_partition, shared);

-- Cluster+Partition Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numnodes ON job (cluster, cluster_partition, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numhwthreads ON job (cluster, cluster_partition, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numacc ON job (cluster, cluster_partition, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_energy ON job (cluster, cluster_partition, energy);

-- Cluster+Partition Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_duration_starttime ON job (cluster, cluster_partition, duration, start_time);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_starttime_duration ON job (cluster, cluster_partition, start_time, duration);

-- Cluster+JobState Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user ON job (cluster, job_state, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project ON job (cluster, job_state, project);
-- Cluster+JobState Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numnodes ON job (cluster, job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numhwthreads ON job (cluster, job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numacc ON job (cluster, job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_energy ON job (cluster, job_state, energy);

-- Cluster+JobState Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_starttime_duration ON job (cluster, job_state, start_time, duration);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_duration_starttime ON job (cluster, job_state, duration, start_time);

-- Cluster+Shared Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_user ON job (cluster, shared, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_project ON job (cluster, shared, project);
-- Cluster+Shared Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numnodes ON job (cluster, shared, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numhwthreads ON job (cluster, shared, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numacc ON job (cluster, shared, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_energy ON job (cluster, shared, energy);

-- Cluster+Shared Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_starttime_duration ON job (cluster, shared, start_time, duration);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_duration_starttime ON job (cluster, shared, duration, start_time);

-- User Filter
-- User Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_user_numnodes ON job (hpc_user, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_user_numhwthreads ON job (hpc_user, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_user_numacc ON job (hpc_user, num_acc);
CREATE INDEX IF NOT EXISTS jobs_user_energy ON job (hpc_user, energy);

-- Cluster+Shared Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_user_starttime_duration ON job (hpc_user, start_time, duration);
CREATE INDEX IF NOT EXISTS jobs_user_duration_starttime ON job (hpc_user, duration, start_time);

-- Project Filter
CREATE INDEX IF NOT EXISTS jobs_project_user ON job (project, hpc_user);
-- Project Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_project_numnodes ON job (project, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_project_numhwthreads ON job (project, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_project_numacc ON job (project, num_acc);
CREATE INDEX IF NOT EXISTS jobs_project_energy ON job (project, energy);

-- Cluster+Shared Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_project_starttime_duration ON job (project, start_time, duration);
CREATE INDEX IF NOT EXISTS jobs_project_duration_starttime ON job (project, duration, start_time);

-- JobState Filter
CREATE INDEX IF NOT EXISTS jobs_jobstate_user ON job (job_state, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_jobstate_project ON job (job_state, project);
-- JobState Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_jobstate_numnodes ON job (job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numhwthreads ON job (job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numacc ON job (job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_jobstate_energy ON job (job_state, energy);

-- Cluster+Shared Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_jobstate_starttime_duration ON job (job_state, start_time, duration);
CREATE INDEX IF NOT EXISTS jobs_jobstate_duration_starttime ON job (job_state, duration, start_time);

-- Shared Filter
CREATE INDEX IF NOT EXISTS jobs_shared_user ON job (shared, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_shared_project ON job (shared, project);
-- Shared Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_shared_numnodes ON job (shared, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_shared_numhwthreads ON job (shared, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_shared_numacc ON job (shared, num_acc);
CREATE INDEX IF NOT EXISTS jobs_shared_energy ON job (shared, energy);

-- Cluster+Shared Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_shared_starttime_duration ON job (shared, start_time, duration);
CREATE INDEX IF NOT EXISTS jobs_shared_duration_starttime ON job (shared, duration, start_time);

-- ArrayJob Filter
CREATE INDEX IF NOT EXISTS jobs_arrayjobid_starttime ON job (array_job_id, start_time);
CREATE INDEX IF NOT EXISTS jobs_cluster_arrayjobid_starttime ON job (cluster, array_job_id, start_time);

-- Single filters with default starttime sorting
CREATE INDEX IF NOT EXISTS jobs_duration_starttime ON job (duration, start_time);
CREATE INDEX IF NOT EXISTS jobs_numnodes_starttime ON job (num_nodes, start_time);
CREATE INDEX IF NOT EXISTS jobs_numhwthreads_starttime ON job (num_hwthreads, start_time);
CREATE INDEX IF NOT EXISTS jobs_numacc_starttime ON job (num_acc, start_time);
CREATE INDEX IF NOT EXISTS jobs_energy_starttime ON job (energy, start_time);

-- Single filters with duration sorting
CREATE INDEX IF NOT EXISTS jobs_starttime_duration ON job (start_time, duration);
CREATE INDEX IF NOT EXISTS jobs_numnodes_duration ON job (num_nodes, duration);
CREATE INDEX IF NOT EXISTS jobs_numhwthreads_duration ON job (num_hwthreads, duration);
CREATE INDEX IF NOT EXISTS jobs_numacc_duration ON job (num_acc, duration);
CREATE INDEX IF NOT EXISTS jobs_energy_duration ON job (energy, duration);

-- Backup Indices For High Variety Columns
CREATE INDEX IF NOT EXISTS jobs_starttime ON job (start_time);
CREATE INDEX IF NOT EXISTS jobs_duration ON job (duration);

-- Notes:
-- Cluster+Partition+Jobstate Filter: Tested -> Full Array Of Combinations non-required
-- Cluster+JobState+Shared Filter: Tested -> No further timing improvement
-- JobState+Shared Filter: Tested -> No further timing improvement

-- Optimize DB index usage
PRAGMA optimize;

-- Optimize DB size: https://sqlite.org/lang_vacuum.html
-- Not allowed within a migration transaction; Keep command here for documentation and recommendation
-- Command: 'VACUUM;'


     1→DROP INDEX IF EXISTS job_stats;
     2→DROP INDEX IF EXISTS job_by_user;
     3→DROP INDEX IF EXISTS job_by_starttime;
     4→DROP INDEX IF EXISTS job_by_job_id;
     5→DROP INDEX IF EXISTS job_list;
     6→DROP INDEX IF EXISTS job_list_user;
     7→DROP INDEX IF EXISTS job_list_users;
     8→DROP INDEX IF EXISTS job_list_users_start;
     9→
    10→ALTER TABLE job ADD COLUMN energy REAL NOT NULL DEFAULT 0.0;
    11→ALTER TABLE job ADD COLUMN energy_footprint TEXT DEFAULT NULL;
    12→
    13→ALTER TABLE job ADD COLUMN footprint TEXT DEFAULT NULL;
    14→ALTER TABLE tag ADD COLUMN tag_scope TEXT NOT NULL DEFAULT 'global';
    15→
    16→-- Do not use reserved keywords anymore
    17→ALTER TABLE "user" RENAME TO hpc_user;
    18→ALTER TABLE job RENAME COLUMN "user" TO hpc_user;
    19→ALTER TABLE job RENAME COLUMN "partition" TO cluster_partition;
    20→
    21→UPDATE job SET footprint = '{"flops_any_avg": 0.0}';
    22→UPDATE job SET footprint = json_replace(footprint, '$.flops_any_avg', job.flops_any_avg);
    23→UPDATE job SET footprint = json_insert(footprint, '$.mem_bw_avg', job.mem_bw_avg);
    24→UPDATE job SET footprint = json_insert(footprint, '$.mem_used_max', job.mem_used_max);
    25→UPDATE job SET footprint = json_insert(footprint, '$.cpu_load_avg', job.load_avg);
    26→UPDATE job SET footprint = json_insert(footprint, '$.net_bw_avg', job.net_bw_avg) WHERE job.net_bw_avg != 0;
    27→UPDATE job SET footprint = json_insert(footprint, '$.net_data_vol_total', job.net_data_vol_total) WHERE job.net_data_vol_total != 0;
    28→UPDATE job SET footprint = json_insert(footprint, '$.file_bw_avg', job.file_bw_avg) WHERE job.file_bw_avg != 0;
    29→UPDATE job SET footprint = json_insert(footprint, '$.file_data_vol_total', job.file_data_vol_total) WHERE job.file_data_vol_total != 0;
    30→
    31→ALTER TABLE job DROP flops_any_avg;
    32→ALTER TABLE job DROP mem_bw_avg;
    33→ALTER TABLE job DROP mem_used_max;
    34→ALTER TABLE job DROP load_avg;
    35→ALTER TABLE job DROP net_bw_avg;
    36→ALTER TABLE job DROP net_data_vol_total;
    37→ALTER TABLE job DROP file_bw_avg;
    38→ALTER TABLE job DROP file_data_vol_total;
    39→
    40→-- Indices for: Single filters, combined filters, sorting, sorting with filters
    41→-- Cluster Filter
    42→CREATE INDEX IF NOT EXISTS jobs_cluster ON job (cluster);
    43→CREATE INDEX IF NOT EXISTS jobs_cluster_user ON job (cluster, hpc_user);
    44→CREATE INDEX IF NOT EXISTS jobs_cluster_project ON job (cluster, project);
    45→CREATE INDEX IF NOT EXISTS jobs_cluster_subcluster ON job (cluster, subcluster);
    46→-- Cluster Filter Sorting
    47→CREATE INDEX IF NOT EXISTS jobs_cluster_starttime ON job (cluster, start_time);
    48→CREATE INDEX IF NOT EXISTS jobs_cluster_duration ON job (cluster, duration);
    49→CREATE INDEX IF NOT EXISTS jobs_cluster_numnodes ON job (cluster, num_nodes);
    50→CREATE INDEX IF NOT EXISTS jobs_cluster_numhwthreads ON job (cluster, num_hwthreads);
    51→CREATE INDEX IF NOT EXISTS jobs_cluster_numacc ON job (cluster, num_acc);
    52→CREATE INDEX IF NOT EXISTS jobs_cluster_energy ON job (cluster, energy);
    53→
    54→-- Cluster+Partition Filter
    55→CREATE INDEX IF NOT EXISTS jobs_cluster_partition ON job (cluster, cluster_partition);
    56→-- Cluster+Partition Filter Sorting
    57→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_starttime ON job (cluster, cluster_partition, start_time);
    58→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_duration ON job (cluster, cluster_partition, duration);
    59→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numnodes ON job (cluster, cluster_partition, num_nodes);
    60→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numhwthreads ON job (cluster, cluster_partition, num_hwthreads);
    61→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numacc ON job (cluster, cluster_partition, num_acc);
    62→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_energy ON job (cluster, cluster_partition, energy);
    63→
    64→-- Cluster+Partition+Jobstate Filter
    65→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate ON job (cluster, cluster_partition, job_state);
    66→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_user ON job (cluster, cluster_partition, job_state, hpc_user);
    67→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_project ON job (cluster, cluster_partition, job_state, project);
    68→-- Cluster+Partition+Jobstate Filter Sorting
    69→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_starttime ON job (cluster, cluster_partition, job_state, start_time);
    70→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_duration ON job (cluster, cluster_partition, job_state, duration);
    71→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_numnodes ON job (cluster, cluster_partition, job_state, num_nodes);
    72→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_numhwthreads ON job (cluster, cluster_partition, job_state, num_hwthreads);
    73→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_numacc ON job (cluster, cluster_partition, job_state, num_acc);
    74→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_energy ON job (cluster, cluster_partition, job_state, energy);
    75→
    76→-- Cluster+JobState Filter
    77→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate ON job (cluster, job_state);
    78→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user ON job (cluster, job_state, hpc_user);
    79→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project ON job (cluster, job_state, project);
    80→-- Cluster+JobState Filter Sorting
    81→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_starttime ON job (cluster, job_state, start_time);
    82→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_duration ON job (cluster, job_state, duration);
    83→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numnodes ON job (cluster, job_state, num_nodes);
    84→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numhwthreads ON job (cluster, job_state, num_hwthreads);
    85→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numacc ON job (cluster, job_state, num_acc);
    86→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_energy ON job (cluster, job_state, energy);
    87→
    88→-- User Filter
    89→CREATE INDEX IF NOT EXISTS jobs_user ON job (hpc_user);
    90→-- User Filter Sorting
    91→CREATE INDEX IF NOT EXISTS jobs_user_starttime ON job (hpc_user, start_time);
    92→CREATE INDEX IF NOT EXISTS jobs_user_duration ON job (hpc_user, duration);
    93→CREATE INDEX IF NOT EXISTS jobs_user_numnodes ON job (hpc_user, num_nodes);
    94→CREATE INDEX IF NOT EXISTS jobs_user_numhwthreads ON job (hpc_user, num_hwthreads);
    95→CREATE INDEX IF NOT EXISTS jobs_user_numacc ON job (hpc_user, num_acc);
    96→CREATE INDEX IF NOT EXISTS jobs_user_energy ON job (hpc_user, energy);
    97→
    98→-- Project Filter
    99→CREATE INDEX IF NOT EXISTS jobs_project ON job (project);
   100→CREATE INDEX IF NOT EXISTS jobs_project_user ON job (project, hpc_user);
   101→-- Project Filter Sorting
   102→CREATE INDEX IF NOT EXISTS jobs_project_starttime ON job (project, start_time);
   103→CREATE INDEX IF NOT EXISTS jobs_project_duration ON job (project, duration);
   104→CREATE INDEX IF NOT EXISTS jobs_project_numnodes ON job (project, num_nodes);
   105→CREATE INDEX IF NOT EXISTS jobs_project_numhwthreads ON job (project, num_hwthreads);
   106→CREATE INDEX IF NOT EXISTS jobs_project_numacc ON job (project, num_acc);
   107→CREATE INDEX IF NOT EXISTS jobs_project_energy ON job (project, energy);
   108→
   109→-- JobState Filter
   110→CREATE INDEX IF NOT EXISTS jobs_jobstate ON job (job_state);
   111→CREATE INDEX IF NOT EXISTS jobs_jobstate_user ON job (job_state, hpc_user);
   112→CREATE INDEX IF NOT EXISTS jobs_jobstate_project ON job (job_state, project);
   113→CREATE INDEX IF NOT EXISTS jobs_jobstate_cluster ON job (job_state, cluster);
   114→-- JobState Filter Sorting
   115→CREATE INDEX IF NOT EXISTS jobs_jobstate_starttime ON job (job_state, start_time);
   116→CREATE INDEX IF NOT EXISTS jobs_jobstate_duration ON job (job_state, duration);
   117→CREATE INDEX IF NOT EXISTS jobs_jobstate_numnodes ON job (job_state, num_nodes);
   118→CREATE INDEX IF NOT EXISTS jobs_jobstate_numhwthreads ON job (job_state, num_hwthreads);
   119→CREATE INDEX IF NOT EXISTS jobs_jobstate_numacc ON job (job_state, num_acc);
   120→CREATE INDEX IF NOT EXISTS jobs_jobstate_energy ON job (job_state, energy);
   121→
   122→-- ArrayJob Filter
   123→CREATE INDEX IF NOT EXISTS jobs_arrayjobid_starttime ON job (array_job_id, start_time);
   124→CREATE INDEX IF NOT EXISTS jobs_cluster_arrayjobid_starttime ON job (cluster, array_job_id, start_time);
   125→
   126→-- Sorting without active filters
   127→CREATE INDEX IF NOT EXISTS jobs_starttime ON job (start_time);
   128→CREATE INDEX IF NOT EXISTS jobs_duration ON job (duration);
   129→CREATE INDEX IF NOT EXISTS jobs_numnodes ON job (num_nodes);
   130→CREATE INDEX IF NOT EXISTS jobs_numhwthreads ON job (num_hwthreads);
   131→CREATE INDEX IF NOT EXISTS jobs_numacc ON job (num_acc);
   132→CREATE INDEX IF NOT EXISTS jobs_energy ON job (energy);
   133→
   134→-- Single filters with default starttime sorting
   135→CREATE INDEX IF NOT EXISTS jobs_duration_starttime ON job (duration, start_time);
   136→CREATE INDEX IF NOT EXISTS jobs_numnodes_starttime ON job (num_nodes, start_time);
   137→CREATE INDEX IF NOT EXISTS jobs_numhwthreads_starttime ON job (num_hwthreads, start_time);
   138→CREATE INDEX IF NOT EXISTS jobs_numacc_starttime ON job (num_acc, start_time);
   139→CREATE INDEX IF NOT EXISTS jobs_energy_starttime ON job (energy, start_time);
   140→
   141→-- Optimize DB index usage
   142→PRAGMA optimize;
   143→

DROP INDEX IF EXISTS job_stats;
DROP INDEX IF EXISTS job_by_user;
DROP INDEX IF EXISTS job_by_starttime;
DROP INDEX IF EXISTS job_by_job_id;
DROP INDEX IF EXISTS job_list;
DROP INDEX IF EXISTS job_list_user;
DROP INDEX IF EXISTS job_list_users;
DROP INDEX IF EXISTS job_list_users_start;

ALTER TABLE job ADD COLUMN energy REAL NOT NULL DEFAULT 0.0;
ALTER TABLE job ADD COLUMN energy_footprint TEXT DEFAULT NULL;

ALTER TABLE job ADD COLUMN footprint TEXT DEFAULT NULL;
ALTER TABLE tag ADD COLUMN tag_scope TEXT NOT NULL DEFAULT 'global';

-- Do not use reserved keywords anymore
ALTER TABLE "user" RENAME TO hpc_user;
ALTER TABLE job RENAME COLUMN "user" TO hpc_user;
ALTER TABLE job RENAME COLUMN "partition" TO cluster_partition;

UPDATE job SET footprint = '{"flops_any_avg": 0.0}';
UPDATE job SET footprint = json_replace(footprint, '$.flops_any_avg', job.flops_any_avg);
UPDATE job SET footprint = json_insert(footprint, '$.mem_bw_avg', job.mem_bw_avg);
UPDATE job SET footprint = json_insert(footprint, '$.mem_used_max', job.mem_used_max);
UPDATE job SET footprint = json_insert(footprint, '$.cpu_load_avg', job.load_avg);
UPDATE job SET footprint = json_insert(footprint, '$.net_bw_avg', job.net_bw_avg) WHERE job.net_bw_avg != 0;
UPDATE job SET footprint = json_insert(footprint, '$.net_data_vol_total', job.net_data_vol_total) WHERE job.net_data_vol_total != 0;
UPDATE job SET footprint = json_insert(footprint, '$.file_bw_avg', job.file_bw_avg) WHERE job.file_bw_avg != 0;
UPDATE job SET footprint = json_insert(footprint, '$.file_data_vol_total', job.file_data_vol_total) WHERE job.file_data_vol_total != 0;

ALTER TABLE job DROP flops_any_avg;
ALTER TABLE job DROP mem_bw_avg;
ALTER TABLE job DROP mem_used_max;
ALTER TABLE job DROP load_avg;
ALTER TABLE job DROP net_bw_avg;
ALTER TABLE job DROP net_data_vol_total;
ALTER TABLE job DROP file_bw_avg;
ALTER TABLE job DROP file_data_vol_total;

-- Indices for: Single filters, combined filters, sorting, sorting with filters
-- Cluster Filter
CREATE INDEX IF NOT EXISTS jobs_cluster ON job (cluster);
CREATE INDEX IF NOT EXISTS jobs_cluster_user ON job (cluster, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_cluster_project ON job (cluster, project);
CREATE INDEX IF NOT EXISTS jobs_cluster_subcluster ON job (cluster, subcluster);
-- Cluster Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_starttime ON job (cluster, start_time);
CREATE INDEX IF NOT EXISTS jobs_cluster_duration ON job (cluster, duration);
CREATE INDEX IF NOT EXISTS jobs_cluster_numnodes ON job (cluster, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_numhwthreads ON job (cluster, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_numacc ON job (cluster, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_energy ON job (cluster, energy);

-- Cluster+Partition Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_partition ON job (cluster, cluster_partition);
-- Cluster+Partition Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_starttime ON job (cluster, cluster_partition, start_time);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_duration ON job (cluster, cluster_partition, duration);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numnodes ON job (cluster, cluster_partition, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numhwthreads ON job (cluster, cluster_partition, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numacc ON job (cluster, cluster_partition, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_energy ON job (cluster, cluster_partition, energy);

-- Cluster+Partition+Jobstate Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate ON job (cluster, cluster_partition, job_state);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_user ON job (cluster, cluster_partition, job_state, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_project ON job (cluster, cluster_partition, job_state, project);
-- Cluster+Partition+Jobstate Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_starttime ON job (cluster, cluster_partition, job_state, start_time);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_duration ON job (cluster, cluster_partition, job_state, duration);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_numnodes ON job (cluster, cluster_partition, job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_numhwthreads ON job (cluster, cluster_partition, job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_numacc ON job (cluster, cluster_partition, job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate_energy ON job (cluster, cluster_partition, job_state, energy);

-- Cluster+JobState Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate ON job (cluster, job_state);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user ON job (cluster, job_state, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project ON job (cluster, job_state, project);
-- Cluster+JobState Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_starttime ON job (cluster, job_state, start_time);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_duration ON job (cluster, job_state, duration);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numnodes ON job (cluster, job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numhwthreads ON job (cluster, job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numacc ON job (cluster, job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_energy ON job (cluster, job_state, energy);

-- User Filter
CREATE INDEX IF NOT EXISTS jobs_user ON job (hpc_user);
-- User Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_user_starttime ON job (hpc_user, start_time);
CREATE INDEX IF NOT EXISTS jobs_user_duration ON job (hpc_user, duration);
CREATE INDEX IF NOT EXISTS jobs_user_numnodes ON job (hpc_user, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_user_numhwthreads ON job (hpc_user, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_user_numacc ON job (hpc_user, num_acc);
CREATE INDEX IF NOT EXISTS jobs_user_energy ON job (hpc_user, energy);

-- Project Filter
CREATE INDEX IF NOT EXISTS jobs_project ON job (project);
CREATE INDEX IF NOT EXISTS jobs_project_user ON job (project, hpc_user);
-- Project Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_project_starttime ON job (project, start_time);
CREATE INDEX IF NOT EXISTS jobs_project_duration ON job (project, duration);
CREATE INDEX IF NOT EXISTS jobs_project_numnodes ON job (project, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_project_numhwthreads ON job (project, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_project_numacc ON job (project, num_acc);
CREATE INDEX IF NOT EXISTS jobs_project_energy ON job (project, energy);

-- JobState Filter
CREATE INDEX IF NOT EXISTS jobs_jobstate ON job (job_state);
CREATE INDEX IF NOT EXISTS jobs_jobstate_user ON job (job_state, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_jobstate_project ON job (job_state, project);
CREATE INDEX IF NOT EXISTS jobs_jobstate_cluster ON job (job_state, cluster);
-- JobState Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_jobstate_starttime ON job (job_state, start_time);
CREATE INDEX IF NOT EXISTS jobs_jobstate_duration ON job (job_state, duration);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numnodes ON job (job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numhwthreads ON job (job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numacc ON job (job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_jobstate_energy ON job (job_state, energy);

-- ArrayJob Filter
CREATE INDEX IF NOT EXISTS jobs_arrayjobid_starttime ON job (array_job_id, start_time);
CREATE INDEX IF NOT EXISTS jobs_cluster_arrayjobid_starttime ON job (cluster, array_job_id, start_time);

-- Sorting without active filters
CREATE INDEX IF NOT EXISTS jobs_starttime ON job (start_time);
CREATE INDEX IF NOT EXISTS jobs_duration ON job (duration);
CREATE INDEX IF NOT EXISTS jobs_numnodes ON job (num_nodes);
CREATE INDEX IF NOT EXISTS jobs_numhwthreads ON job (num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_numacc ON job (num_acc);
CREATE INDEX IF NOT EXISTS jobs_energy ON job (energy);

-- Single filters with default starttime sorting
CREATE INDEX IF NOT EXISTS jobs_duration_starttime ON job (duration, start_time);
CREATE INDEX IF NOT EXISTS jobs_numnodes_starttime ON job (num_nodes, start_time);
CREATE INDEX IF NOT EXISTS jobs_numhwthreads_starttime ON job (num_hwthreads, start_time);
CREATE INDEX IF NOT EXISTS jobs_numacc_starttime ON job (num_acc, start_time);
CREATE INDEX IF NOT EXISTS jobs_energy_starttime ON job (energy, start_time);

-- Optimize DB index usage
PRAGMA optimize;


Now I have all the information needed. Let me create the three files.


Write

-- Migration 11: Optimize job table indexes
-- Reduces from ~78 indexes to 20 for better write performance,
-- reduced disk usage, and more reliable query planner decisions.
-- Requires ANALYZE to be run after migration (done automatically on startup).

-- ============================================================
-- Drop ALL existing job indexes (from migrations 08/09)
-- sqlite_autoindex_job_1 (UNIQUE constraint) is kept automatically
-- ============================================================

-- Cluster Filter
DROP INDEX IF EXISTS jobs_cluster_user;
DROP INDEX IF EXISTS jobs_cluster_project;
DROP INDEX IF EXISTS jobs_cluster_subcluster;
-- Cluster Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_numnodes;
DROP INDEX IF EXISTS jobs_cluster_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_numacc;
DROP INDEX IF EXISTS jobs_cluster_energy;
-- Cluster Time Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_duration_starttime;
DROP INDEX IF EXISTS jobs_cluster_starttime_duration;

-- Cluster+Partition Filter
DROP INDEX IF EXISTS jobs_cluster_partition_user;
DROP INDEX IF EXISTS jobs_cluster_partition_project;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate;
DROP INDEX IF EXISTS jobs_cluster_partition_shared;
-- Cluster+Partition Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_partition_numnodes;
DROP INDEX IF EXISTS jobs_cluster_partition_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_partition_numacc;
DROP INDEX IF EXISTS jobs_cluster_partition_energy;
-- Cluster+Partition Time Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_partition_duration_starttime;
DROP INDEX IF EXISTS jobs_cluster_partition_starttime_duration;

-- Cluster+JobState Filter
DROP INDEX IF EXISTS jobs_cluster_jobstate_user;
DROP INDEX IF EXISTS jobs_cluster_jobstate_project;
-- Cluster+JobState Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_jobstate_numnodes;
DROP INDEX IF EXISTS jobs_cluster_jobstate_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_jobstate_numacc;
DROP INDEX IF EXISTS jobs_cluster_jobstate_energy;
-- Cluster+JobState Time Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_jobstate_starttime_duration;
DROP INDEX IF EXISTS jobs_cluster_jobstate_duration_starttime;

-- Cluster+Shared Filter
DROP INDEX IF EXISTS jobs_cluster_shared_user;
DROP INDEX IF EXISTS jobs_cluster_shared_project;
-- Cluster+Shared Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_shared_numnodes;
DROP INDEX IF EXISTS jobs_cluster_shared_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_shared_numacc;
DROP INDEX IF EXISTS jobs_cluster_shared_energy;
-- Cluster+Shared Time Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_shared_starttime_duration;
DROP INDEX IF EXISTS jobs_cluster_shared_duration_starttime;

-- User Filter Sorting
DROP INDEX IF EXISTS jobs_user_numnodes;
DROP INDEX IF EXISTS jobs_user_numhwthreads;
DROP INDEX IF EXISTS jobs_user_numacc;
DROP INDEX IF EXISTS jobs_user_energy;
-- User Time Filter Sorting
DROP INDEX IF EXISTS jobs_user_starttime_duration;
DROP INDEX IF EXISTS jobs_user_duration_starttime;

-- Project Filter
DROP INDEX IF EXISTS jobs_project_user;
-- Project Filter Sorting
DROP INDEX IF EXISTS jobs_project_numnodes;
DROP INDEX IF EXISTS jobs_project_numhwthreads;
DROP INDEX IF EXISTS jobs_project_numacc;
DROP INDEX IF EXISTS jobs_project_energy;
-- Project Time Filter Sorting
DROP INDEX IF EXISTS jobs_project_starttime_duration;
DROP INDEX IF EXISTS jobs_project_duration_starttime;

-- JobState Filter
DROP INDEX IF EXISTS jobs_jobstate_user;
DROP INDEX IF EXISTS jobs_jobstate_project;
-- JobState Filter Sorting
DROP INDEX IF EXISTS jobs_jobstate_numnodes;
DROP INDEX IF EXISTS jobs_jobstate_numhwthreads;
DROP INDEX IF EXISTS jobs_jobstate_numacc;
DROP INDEX IF EXISTS jobs_jobstate_energy;
-- JobState Time Filter Sorting
DROP INDEX IF EXISTS jobs_jobstate_starttime_duration;
DROP INDEX IF EXISTS jobs_jobstate_duration_starttime;

-- Shared Filter
DROP INDEX IF EXISTS jobs_shared_user;
DROP INDEX IF EXISTS jobs_shared_project;
-- Shared Filter Sorting
DROP INDEX IF EXISTS jobs_shared_numnodes;
DROP INDEX IF EXISTS jobs_shared_numhwthreads;
DROP INDEX IF EXISTS jobs_shared_numacc;
DROP INDEX IF EXISTS jobs_shared_energy;
-- Shared Time Filter Sorting
DROP INDEX IF EXISTS jobs_shared_starttime_duration;
DROP INDEX IF EXISTS jobs_shared_duration_starttime;

-- ArrayJob Filter
DROP INDEX IF EXISTS jobs_arrayjobid_starttime;
DROP INDEX IF EXISTS jobs_cluster_arrayjobid_starttime;

-- Single filters with default starttime sorting
DROP INDEX IF EXISTS jobs_duration_starttime;
DROP INDEX IF EXISTS jobs_numnodes_starttime;
DROP INDEX IF EXISTS jobs_numhwthreads_starttime;
DROP INDEX IF EXISTS jobs_numacc_starttime;
DROP INDEX IF EXISTS jobs_energy_starttime;

-- Single filters with duration sorting
DROP INDEX IF EXISTS jobs_starttime_duration;
DROP INDEX IF EXISTS jobs_numnodes_duration;
DROP INDEX IF EXISTS jobs_numhwthreads_duration;
DROP INDEX IF EXISTS jobs_numacc_duration;
DROP INDEX IF EXISTS jobs_energy_duration;

-- Backup Indices
DROP INDEX IF EXISTS jobs_starttime;
DROP INDEX IF EXISTS jobs_duration;

-- Legacy indexes from migration 08 (may exist on older DBs)
DROP INDEX IF EXISTS jobs_cluster;
DROP INDEX IF EXISTS jobs_cluster_starttime;
DROP INDEX IF EXISTS jobs_cluster_duration;
DROP INDEX IF EXISTS jobs_cluster_partition;
DROP INDEX IF EXISTS jobs_cluster_partition_starttime;
DROP INDEX IF EXISTS jobs_cluster_partition_duration;
DROP INDEX IF EXISTS jobs_cluster_partition_numnodes;
DROP INDEX IF EXISTS jobs_cluster_partition_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_partition_numacc;
DROP INDEX IF EXISTS jobs_cluster_partition_energy;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_user;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_project;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_starttime;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_duration;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_numnodes;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_numacc;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_energy;
DROP INDEX IF EXISTS jobs_cluster_jobstate;
DROP INDEX IF EXISTS jobs_cluster_jobstate_starttime;
DROP INDEX IF EXISTS jobs_cluster_jobstate_duration;
DROP INDEX IF EXISTS jobs_user;
DROP INDEX IF EXISTS jobs_user_starttime;
DROP INDEX IF EXISTS jobs_user_duration;
DROP INDEX IF EXISTS jobs_project;
DROP INDEX IF EXISTS jobs_project_starttime;
DROP INDEX IF EXISTS jobs_project_duration;
DROP INDEX IF EXISTS jobs_jobstate;
DROP INDEX IF EXISTS jobs_jobstate_cluster;
DROP INDEX IF EXISTS jobs_jobstate_starttime;
DROP INDEX IF EXISTS jobs_jobstate_duration;
DROP INDEX IF EXISTS jobs_numnodes;
DROP INDEX IF EXISTS jobs_numhwthreads;
DROP INDEX IF EXISTS jobs_numacc;
DROP INDEX IF EXISTS jobs_energy;

-- ============================================================
-- Create optimized set of 20 indexes
-- ============================================================

-- GROUP 1: Global sort (1 index)
-- Default sort for unfiltered/multi-state IN queries, time range, delete-before
CREATE INDEX jobs_starttime ON job (start_time);

-- GROUP 2: Cluster-prefixed (8 indexes)
-- Cluster + default sort, concurrent jobs, time range within cluster
CREATE INDEX jobs_cluster_starttime_duration ON job (cluster, start_time, duration);
-- Cluster + sort by duration
CREATE INDEX jobs_cluster_duration_starttime ON job (cluster, duration, start_time);
-- COVERING for cluster+state aggregation; running jobs (cluster, state, duration>?)
CREATE INDEX jobs_cluster_jobstate_duration_starttime ON job (cluster, job_state, duration, start_time);
-- Cluster+state+sort start_time (single state equality)
CREATE INDEX jobs_cluster_jobstate_starttime_duration ON job (cluster, job_state, start_time, duration);
-- COVERING for GROUP BY user with cluster filter
CREATE INDEX jobs_cluster_user ON job (cluster, hpc_user);
-- GROUP BY project with cluster filter
CREATE INDEX jobs_cluster_project ON job (cluster, project);
-- GROUP BY subcluster with cluster filter
CREATE INDEX jobs_cluster_subcluster ON job (cluster, subcluster);
-- Cluster + sort by num_nodes (state filtered per-row, fast with LIMIT)
CREATE INDEX jobs_cluster_numnodes ON job (cluster, num_nodes);

-- GROUP 3: User-prefixed (1 index)
-- Security filter (user role) + default sort
CREATE INDEX jobs_user_starttime_duration ON job (hpc_user, start_time, duration);

-- GROUP 4: Project-prefixed (1 index)
-- Security filter (manager role) + default sort
CREATE INDEX jobs_project_starttime_duration ON job (project, start_time, duration);

-- GROUP 5: JobState-prefixed (3 indexes)
-- State + project filter (for manager security within state query)
CREATE INDEX jobs_jobstate_project ON job (job_state, project);
-- State + user filter/aggregation
CREATE INDEX jobs_jobstate_user ON job (job_state, hpc_user);
-- COVERING for non-running jobs scan, state + sort duration
CREATE INDEX jobs_jobstate_duration_starttime ON job (job_state, duration, start_time);

-- GROUP 6: Rare filters (1 index)
-- Array job lookup
CREATE INDEX jobs_arrayjobid ON job (array_job_id);

-- GROUP 7: Secondary sort columns (5 indexes)
CREATE INDEX jobs_cluster_numhwthreads ON job (cluster, num_hwthreads);
CREATE INDEX jobs_cluster_numacc ON job (cluster, num_acc);
CREATE INDEX jobs_cluster_energy ON job (cluster, energy);
-- Cluster+partition + sort start_time
CREATE INDEX jobs_cluster_partition_starttime ON job (cluster, cluster_partition, start_time);
-- Cluster+partition+state filter
CREATE INDEX jobs_cluster_partition_jobstate ON job (cluster, cluster_partition, job_state);

-- Optimize DB index usage
PRAGMA optimize;


File created successfully at: /Users/jan/prg/CC/cc-backend/internal/repository/migrations/sqlite3/11_optimize-indexes.up.sql

-- Migration 11: Optimize job table indexes
-- Reduces from ~78 indexes to 20 for better write performance,
-- reduced disk usage, and more reliable query planner decisions.
-- Requires ANALYZE to be run after migration (done automatically on startup).

-- ============================================================
-- Drop ALL existing job indexes (from migrations 08/09)
-- sqlite_autoindex_job_1 (UNIQUE constraint) is kept automatically
-- ============================================================

-- Cluster Filter
DROP INDEX IF EXISTS jobs_cluster_user;
DROP INDEX IF EXISTS jobs_cluster_project;
DROP INDEX IF EXISTS jobs_cluster_subcluster;
-- Cluster Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_numnodes;
DROP INDEX IF EXISTS jobs_cluster_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_numacc;
DROP INDEX IF EXISTS jobs_cluster_energy;
-- Cluster Time Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_duration_starttime;
DROP INDEX IF EXISTS jobs_cluster_starttime_duration;

-- Cluster+Partition Filter
DROP INDEX IF EXISTS jobs_cluster_partition_user;
DROP INDEX IF EXISTS jobs_cluster_partition_project;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate;
DROP INDEX IF EXISTS jobs_cluster_partition_shared;
-- Cluster+Partition Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_partition_numnodes;
DROP INDEX IF EXISTS jobs_cluster_partition_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_partition_numacc;
DROP INDEX IF EXISTS jobs_cluster_partition_energy;
-- Cluster+Partition Time Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_partition_duration_starttime;
DROP INDEX IF EXISTS jobs_cluster_partition_starttime_duration;

-- Cluster+JobState Filter
DROP INDEX IF EXISTS jobs_cluster_jobstate_user;
DROP INDEX IF EXISTS jobs_cluster_jobstate_project;
-- Cluster+JobState Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_jobstate_numnodes;
DROP INDEX IF EXISTS jobs_cluster_jobstate_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_jobstate_numacc;
DROP INDEX IF EXISTS jobs_cluster_jobstate_energy;
-- Cluster+JobState Time Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_jobstate_starttime_duration;
DROP INDEX IF EXISTS jobs_cluster_jobstate_duration_starttime;

-- Cluster+Shared Filter
DROP INDEX IF EXISTS jobs_cluster_shared_user;
DROP INDEX IF EXISTS jobs_cluster_shared_project;
-- Cluster+Shared Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_shared_numnodes;
DROP INDEX IF EXISTS jobs_cluster_shared_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_shared_numacc;
DROP INDEX IF EXISTS jobs_cluster_shared_energy;
-- Cluster+Shared Time Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_shared_starttime_duration;
DROP INDEX IF EXISTS jobs_cluster_shared_duration_starttime;

-- User Filter Sorting
DROP INDEX IF EXISTS jobs_user_numnodes;
DROP INDEX IF EXISTS jobs_user_numhwthreads;
DROP INDEX IF EXISTS jobs_user_numacc;
DROP INDEX IF EXISTS jobs_user_energy;
-- User Time Filter Sorting
DROP INDEX IF EXISTS jobs_user_starttime_duration;
DROP INDEX IF EXISTS jobs_user_duration_starttime;

-- Project Filter
DROP INDEX IF EXISTS jobs_project_user;
-- Project Filter Sorting
DROP INDEX IF EXISTS jobs_project_numnodes;
DROP INDEX IF EXISTS jobs_project_numhwthreads;
DROP INDEX IF EXISTS jobs_project_numacc;
DROP INDEX IF EXISTS jobs_project_energy;
-- Project Time Filter Sorting
DROP INDEX IF EXISTS jobs_project_starttime_duration;
DROP INDEX IF EXISTS jobs_project_duration_starttime;

-- JobState Filter
DROP INDEX IF EXISTS jobs_jobstate_user;
DROP INDEX IF EXISTS jobs_jobstate_project;
-- JobState Filter Sorting
DROP INDEX IF EXISTS jobs_jobstate_numnodes;
DROP INDEX IF EXISTS jobs_jobstate_numhwthreads;
DROP INDEX IF EXISTS jobs_jobstate_numacc;
DROP INDEX IF EXISTS jobs_jobstate_energy;
-- JobState Time Filter Sorting
DROP INDEX IF EXISTS jobs_jobstate_starttime_duration;
DROP INDEX IF EXISTS jobs_jobstate_duration_starttime;

-- Shared Filter
DROP INDEX IF EXISTS jobs_shared_user;
DROP INDEX IF EXISTS jobs_shared_project;
-- Shared Filter Sorting
DROP INDEX IF EXISTS jobs_shared_numnodes;
DROP INDEX IF EXISTS jobs_shared_numhwthreads;
DROP INDEX IF EXISTS jobs_shared_numacc;
DROP INDEX IF EXISTS jobs_shared_energy;
-- Shared Time Filter Sorting
DROP INDEX IF EXISTS jobs_shared_starttime_duration;
DROP INDEX IF EXISTS jobs_shared_duration_starttime;

-- ArrayJob Filter
DROP INDEX IF EXISTS jobs_arrayjobid_starttime;
DROP INDEX IF EXISTS jobs_cluster_arrayjobid_starttime;

-- Single filters with default starttime sorting
DROP INDEX IF EXISTS jobs_duration_starttime;
DROP INDEX IF EXISTS jobs_numnodes_starttime;
DROP INDEX IF EXISTS jobs_numhwthreads_starttime;
DROP INDEX IF EXISTS jobs_numacc_starttime;
DROP INDEX IF EXISTS jobs_energy_starttime;

-- Single filters with duration sorting
DROP INDEX IF EXISTS jobs_starttime_duration;
DROP INDEX IF EXISTS jobs_numnodes_duration;
DROP INDEX IF EXISTS jobs_numhwthreads_duration;
DROP INDEX IF EXISTS jobs_numacc_duration;
DROP INDEX IF EXISTS jobs_energy_duration;

-- Backup Indices
DROP INDEX IF EXISTS jobs_starttime;
DROP INDEX IF EXISTS jobs_duration;

-- Legacy indexes from migration 08 (may exist on older DBs)
DROP INDEX IF EXISTS jobs_cluster;
DROP INDEX IF EXISTS jobs_cluster_starttime;
DROP INDEX IF EXISTS jobs_cluster_duration;
DROP INDEX IF EXISTS jobs_cluster_partition;
DROP INDEX IF EXISTS jobs_cluster_partition_starttime;
DROP INDEX IF EXISTS jobs_cluster_partition_duration;
DROP INDEX IF EXISTS jobs_cluster_partition_numnodes;
DROP INDEX IF EXISTS jobs_cluster_partition_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_partition_numacc;
DROP INDEX IF EXISTS jobs_cluster_partition_energy;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_user;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_project;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_starttime;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_duration;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_numnodes;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_numacc;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate_energy;
DROP INDEX IF EXISTS jobs_cluster_jobstate;
DROP INDEX IF EXISTS jobs_cluster_jobstate_starttime;
DROP INDEX IF EXISTS jobs_cluster_jobstate_duration;
DROP INDEX IF EXISTS jobs_user;
DROP INDEX IF EXISTS jobs_user_starttime;
DROP INDEX IF EXISTS jobs_user_duration;
DROP INDEX IF EXISTS jobs_project;
DROP INDEX IF EXISTS jobs_project_starttime;
DROP INDEX IF EXISTS jobs_project_duration;
DROP INDEX IF EXISTS jobs_jobstate;
DROP INDEX IF EXISTS jobs_jobstate_cluster;
DROP INDEX IF EXISTS jobs_jobstate_starttime;
DROP INDEX IF EXISTS jobs_jobstate_duration;
DROP INDEX IF EXISTS jobs_numnodes;
DROP INDEX IF EXISTS jobs_numhwthreads;
DROP INDEX IF EXISTS jobs_numacc;
DROP INDEX IF EXISTS jobs_energy;

-- ============================================================
-- Create optimized set of 20 indexes
-- ============================================================

-- GROUP 1: Global sort (1 index)
-- Default sort for unfiltered/multi-state IN queries, time range, delete-before
CREATE INDEX jobs_starttime ON job (start_time);

-- GROUP 2: Cluster-prefixed (8 indexes)
-- Cluster + default sort, concurrent jobs, time range within cluster
CREATE INDEX jobs_cluster_starttime_duration ON job (cluster, start_time, duration);
-- Cluster + sort by duration
CREATE INDEX jobs_cluster_duration_starttime ON job (cluster, duration, start_time);
-- COVERING for cluster+state aggregation; running jobs (cluster, state, duration>?)
CREATE INDEX jobs_cluster_jobstate_duration_starttime ON job (cluster, job_state, duration, start_time);
-- Cluster+state+sort start_time (single state equality)
CREATE INDEX jobs_cluster_jobstate_starttime_duration ON job (cluster, job_state, start_time, duration);
-- COVERING for GROUP BY user with cluster filter
CREATE INDEX jobs_cluster_user ON job (cluster, hpc_user);
-- GROUP BY project with cluster filter
CREATE INDEX jobs_cluster_project ON job (cluster, project);
-- GROUP BY subcluster with cluster filter
CREATE INDEX jobs_cluster_subcluster ON job (cluster, subcluster);
-- Cluster + sort by num_nodes (state filtered per-row, fast with LIMIT)
CREATE INDEX jobs_cluster_numnodes ON job (cluster, num_nodes);

-- GROUP 3: User-prefixed (1 index)
-- Security filter (user role) + default sort
CREATE INDEX jobs_user_starttime_duration ON job (hpc_user, start_time, duration);

-- GROUP 4: Project-prefixed (1 index)
-- Security filter (manager role) + default sort
CREATE INDEX jobs_project_starttime_duration ON job (project, start_time, duration);

-- GROUP 5: JobState-prefixed (3 indexes)
-- State + project filter (for manager security within state query)
CREATE INDEX jobs_jobstate_project ON job (job_state, project);
-- State + user filter/aggregation
CREATE INDEX jobs_jobstate_user ON job (job_state, hpc_user);
-- COVERING for non-running jobs scan, state + sort duration
CREATE INDEX jobs_jobstate_duration_starttime ON job (job_state, duration, start_time);

-- GROUP 6: Rare filters (1 index)
-- Array job lookup
CREATE INDEX jobs_arrayjobid ON job (array_job_id);

-- GROUP 7: Secondary sort columns (5 indexes)
CREATE INDEX jobs_cluster_numhwthreads ON job (cluster, num_hwthreads);
CREATE INDEX jobs_cluster_numacc ON job (cluster, num_acc);
CREATE INDEX jobs_cluster_energy ON job (cluster, energy);
-- Cluster+partition + sort start_time
CREATE INDEX jobs_cluster_partition_starttime ON job (cluster, cluster_partition, start_time);
-- Cluster+partition+state filter
CREATE INDEX jobs_cluster_partition_jobstate ON job (cluster, cluster_partition, job_state);

-- Optimize DB index usage
PRAGMA optimize;


-- Migration 11 DOWN: Restore all indexes from migration 09
-- Reverts the index optimization by dropping the 20 optimized indexes
-- and recreating the original full set.

-- ============================================================
-- Drop optimized indexes
-- ============================================================

DROP INDEX IF EXISTS jobs_starttime;
DROP INDEX IF EXISTS jobs_cluster_starttime_duration;
DROP INDEX IF EXISTS jobs_cluster_duration_starttime;
DROP INDEX IF EXISTS jobs_cluster_jobstate_duration_starttime;
DROP INDEX IF EXISTS jobs_cluster_jobstate_starttime_duration;
DROP INDEX IF EXISTS jobs_cluster_user;
DROP INDEX IF EXISTS jobs_cluster_project;
DROP INDEX IF EXISTS jobs_cluster_subcluster;
DROP INDEX IF EXISTS jobs_cluster_numnodes;
DROP INDEX IF EXISTS jobs_user_starttime_duration;
DROP INDEX IF EXISTS jobs_project_starttime_duration;
DROP INDEX IF EXISTS jobs_jobstate_project;
DROP INDEX IF EXISTS jobs_jobstate_user;
DROP INDEX IF EXISTS jobs_jobstate_duration_starttime;
DROP INDEX IF EXISTS jobs_arrayjobid;
DROP INDEX IF EXISTS jobs_cluster_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_numacc;
DROP INDEX IF EXISTS jobs_cluster_energy;
DROP INDEX IF EXISTS jobs_cluster_partition_starttime;
DROP INDEX IF EXISTS jobs_cluster_partition_jobstate;

-- ============================================================
-- Recreate all indexes from migration 09
-- ============================================================

-- Cluster Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_user ON job (cluster, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_cluster_project ON job (cluster, project);
CREATE INDEX IF NOT EXISTS jobs_cluster_subcluster ON job (cluster, subcluster);
-- Cluster Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_numnodes ON job (cluster, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_numhwthreads ON job (cluster, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_numacc ON job (cluster, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_energy ON job (cluster, energy);

-- Cluster Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_duration_starttime ON job (cluster, duration, start_time);
CREATE INDEX IF NOT EXISTS jobs_cluster_starttime_duration ON job (cluster, start_time, duration);

-- Cluster+Partition Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_user ON job (cluster, cluster_partition, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_project ON job (cluster, cluster_partition, project);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_jobstate ON job (cluster, cluster_partition, job_state);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_shared ON job (cluster, cluster_partition, shared);

-- Cluster+Partition Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numnodes ON job (cluster, cluster_partition, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numhwthreads ON job (cluster, cluster_partition, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numacc ON job (cluster, cluster_partition, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_energy ON job (cluster, cluster_partition, energy);

-- Cluster+Partition Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_duration_starttime ON job (cluster, cluster_partition, duration, start_time);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_starttime_duration ON job (cluster, cluster_partition, start_time, duration);

-- Cluster+JobState Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user ON job (cluster, job_state, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project ON job (cluster, job_state, project);
-- Cluster+JobState Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numnodes ON job (cluster, job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numhwthreads ON job (cluster, job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numacc ON job (cluster, job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_energy ON job (cluster, job_state, energy);

-- Cluster+JobState Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_starttime_duration ON job (cluster, job_state, start_time, duration);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_duration_starttime ON job (cluster, job_state, duration, start_time);

-- Cluster+Shared Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_user ON job (cluster, shared, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_project ON job (cluster, shared, project);
-- Cluster+Shared Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numnodes ON job (cluster, shared, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numhwthreads ON job (cluster, shared, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numacc ON job (cluster, shared, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_energy ON job (cluster, shared, energy);

-- Cluster+Shared Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_starttime_duration ON job (cluster, shared, start_time, duration);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_duration_starttime ON job (cluster, shared, duration, start_time);

-- User Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_user_numnodes ON job (hpc_user, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_user_numhwthreads ON job (hpc_user, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_user_numacc ON job (hpc_user, num_acc);
CREATE INDEX IF NOT EXISTS jobs_user_energy ON job (hpc_user, energy);

-- User Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_user_starttime_duration ON job (hpc_user, start_time, duration);
CREATE INDEX IF NOT EXISTS jobs_user_duration_starttime ON job (hpc_user, duration, start_time);

-- Project Filter
CREATE INDEX IF NOT EXISTS jobs_project_user ON job (project, hpc_user);
-- Project Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_project_numnodes ON job (project, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_project_numhwthreads ON job (project, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_project_numacc ON job (project, num_acc);
CREATE INDEX IF NOT EXISTS jobs_project_energy ON job (project, energy);

-- Project Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_project_starttime_duration ON job (project, start_time, duration);
CREATE INDEX IF NOT EXISTS jobs_project_duration_starttime ON job (project, duration, start_time);

-- JobState Filter
CREATE INDEX IF NOT EXISTS jobs_jobstate_user ON job (job_state, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_jobstate_project ON job (job_state, project);
-- JobState Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_jobstate_numnodes ON job (job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numhwthreads ON job (job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numacc ON job (job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_jobstate_energy ON job (job_state, energy);

-- JobState Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_jobstate_starttime_duration ON job (job_state, start_time, duration);
CREATE INDEX IF NOT EXISTS jobs_jobstate_duration_starttime ON job (job_state, duration, start_time);

-- Shared Filter
CREATE INDEX IF NOT EXISTS jobs_shared_user ON job (shared, hpc_user);
CREATE INDEX IF NOT EXISTS jobs_shared_project ON job (shared, project);
-- Shared Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_shared_numnodes ON job (shared, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_shared_numhwthreads ON job (shared, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_shared_numacc ON job (shared, num_acc);
CREATE INDEX IF NOT EXISTS jobs_shared_energy ON job (shared, energy);

-- Shared Time Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_shared_starttime_duration ON job (shared, start_time, duration);
CREATE INDEX IF NOT EXISTS jobs_shared_duration_starttime ON job (shared, duration, start_time);

-- ArrayJob Filter
CREATE INDEX IF NOT EXISTS jobs_arrayjobid_starttime ON job (array_job_id, start_time);
CREATE INDEX IF NOT EXISTS jobs_cluster_arrayjobid_starttime ON job (cluster, array_job_id, start_time);

-- Single filters with default starttime sorting
CREATE INDEX IF NOT EXISTS jobs_duration_starttime ON job (duration, start_time);
CREATE INDEX IF NOT EXISTS jobs_numnodes_starttime ON job (num_nodes, start_time);
CREATE INDEX IF NOT EXISTS jobs_numhwthreads_starttime ON job (num_hwthreads, start_time);
CREATE INDEX IF NOT EXISTS jobs_numacc_starttime ON job (num_acc, start_time);
CREATE INDEX IF NOT EXISTS jobs_energy_starttime ON job (energy, start_time);

-- Single filters with duration sorting
CREATE INDEX IF NOT EXISTS jobs_starttime_duration ON job (start_time, duration);
CREATE INDEX IF NOT EXISTS jobs_numnodes_duration ON job (num_nodes, duration);
CREATE INDEX IF NOT EXISTS jobs_numhwthreads_duration ON job (num_hwthreads, duration);
CREATE INDEX IF NOT EXISTS jobs_numacc_duration ON job (num_acc, duration);
CREATE INDEX IF NOT EXISTS jobs_energy_duration ON job (energy, duration);

-- Backup Indices For High Variety Columns
CREATE INDEX IF NOT EXISTS jobs_starttime ON job (start_time);
CREATE INDEX IF NOT EXISTS jobs_duration ON job (duration);

-- Optimize DB index usage
PRAGMA optimize;