<local-command-caveat>Caveat: The messages below were generated by the user while running local commands. DO NOT respond to these messages or otherwise consider them in your response unless the user explicitly asks you to.</local-command-caveat>


<command-name>/clear</command-name>
            <command-message>clear</command-message>
            <command-args></command-args>


<local-command-stdout></local-command-stdout>


<command-name>/model</command-name>
            <command-message>model</command-message>
            <command-args></command-args>


<local-command-stdout>Set model to Default (Sonnet 4.6 · Best for everyday tasks)</local-command-stdout>


Consolidate all migrations after 10 to one migration 11.


     1→-- Migration 11: Remove overly specific table indexes formerly used in sorting
     2→-- When one or two indexed columns are used, sorting usually is fast
     3→-- Reduces from ~78 indexes to 48 for better write performance,
     4→-- reduced disk usage, and more reliable query planner decisions.
     5→-- Requires ANALYZE to be run after migration (done automatically on startup).
     6→
     7→-- ============================================================
     8→-- Drop SELECTED existing job indexes (from migrations 08/09)
     9→-- sqlite_autoindex_job_1 (UNIQUE constraint) is kept automatically
    10→-- ============================================================
    11→
    12→-- Cluster+Partition Filter Sorting
    13→DROP INDEX IF EXISTS jobs_cluster_partition_numnodes;
    14→DROP INDEX IF EXISTS jobs_cluster_partition_numhwthreads;
    15→DROP INDEX IF EXISTS jobs_cluster_partition_numacc;
    16→DROP INDEX IF EXISTS jobs_cluster_partition_energy;
    17→
    18→-- Cluster+JobState Filter Sorting
    19→DROP INDEX IF EXISTS jobs_cluster_jobstate_numnodes;
    20→DROP INDEX IF EXISTS jobs_cluster_jobstate_numhwthreads;
    21→DROP INDEX IF EXISTS jobs_cluster_jobstate_numacc;
    22→DROP INDEX IF EXISTS jobs_cluster_jobstate_energy;
    23→
    24→-- Cluster+Shared Filter Sorting
    25→DROP INDEX IF EXISTS jobs_cluster_shared_numnodes;
    26→DROP INDEX IF EXISTS jobs_cluster_shared_numhwthreads;
    27→DROP INDEX IF EXISTS jobs_cluster_shared_numacc;
    28→DROP INDEX IF EXISTS jobs_cluster_shared_energy;
    29→
    30→-- User Filter Sorting
    31→DROP INDEX IF EXISTS jobs_user_numnodes;
    32→DROP INDEX IF EXISTS jobs_user_numhwthreads;
    33→DROP INDEX IF EXISTS jobs_user_numacc;
    34→DROP INDEX IF EXISTS jobs_user_energy;
    35→
    36→-- Project Filter Sorting
    37→DROP INDEX IF EXISTS jobs_project_numnodes;
    38→DROP INDEX IF EXISTS jobs_project_numhwthreads;
    39→DROP INDEX IF EXISTS jobs_project_numacc;
    40→DROP INDEX IF EXISTS jobs_project_energy;
    41→
    42→-- JobState Filter Sorting
    43→DROP INDEX IF EXISTS jobs_jobstate_numnodes;
    44→DROP INDEX IF EXISTS jobs_jobstate_numhwthreads;
    45→DROP INDEX IF EXISTS jobs_jobstate_numacc;
    46→DROP INDEX IF EXISTS jobs_jobstate_energy;
    47→
    48→-- Shared Filter Sorting
    49→DROP INDEX IF EXISTS jobs_shared_numnodes;
    50→DROP INDEX IF EXISTS jobs_shared_numhwthreads;
    51→DROP INDEX IF EXISTS jobs_shared_numacc;
    52→DROP INDEX IF EXISTS jobs_shared_energy;
    53→
    54→-- ArrayJob Filter
    55→DROP INDEX IF EXISTS jobs_cluster_arrayjobid_starttime;
    56→
    57→-- Backup Indices For High Variety Columns
    58→DROP INDEX IF EXISTS jobs_duration;
    59→
    60→-- Optimize DB index usage
    61→PRAGMA optimize;
    62→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

-- Migration 11: Remove overly specific table indexes formerly used in sorting
-- When one or two indexed columns are used, sorting usually is fast
-- Reduces from ~78 indexes to 48 for better write performance,
-- reduced disk usage, and more reliable query planner decisions.
-- Requires ANALYZE to be run after migration (done automatically on startup).

-- ============================================================
-- Drop SELECTED existing job indexes (from migrations 08/09)
-- sqlite_autoindex_job_1 (UNIQUE constraint) is kept automatically
-- ============================================================

-- Cluster+Partition Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_partition_numnodes;
DROP INDEX IF EXISTS jobs_cluster_partition_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_partition_numacc;
DROP INDEX IF EXISTS jobs_cluster_partition_energy;

-- Cluster+JobState Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_jobstate_numnodes;
DROP INDEX IF EXISTS jobs_cluster_jobstate_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_jobstate_numacc;
DROP INDEX IF EXISTS jobs_cluster_jobstate_energy;

-- Cluster+Shared Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_shared_numnodes;
DROP INDEX IF EXISTS jobs_cluster_shared_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_shared_numacc;
DROP INDEX IF EXISTS jobs_cluster_shared_energy;

-- User Filter Sorting
DROP INDEX IF EXISTS jobs_user_numnodes;
DROP INDEX IF EXISTS jobs_user_numhwthreads;
DROP INDEX IF EXISTS jobs_user_numacc;
DROP INDEX IF EXISTS jobs_user_energy;

-- Project Filter Sorting
DROP INDEX IF EXISTS jobs_project_numnodes;
DROP INDEX IF EXISTS jobs_project_numhwthreads;
DROP INDEX IF EXISTS jobs_project_numacc;
DROP INDEX IF EXISTS jobs_project_energy;

-- JobState Filter Sorting
DROP INDEX IF EXISTS jobs_jobstate_numnodes;
DROP INDEX IF EXISTS jobs_jobstate_numhwthreads;
DROP INDEX IF EXISTS jobs_jobstate_numacc;
DROP INDEX IF EXISTS jobs_jobstate_energy;

-- Shared Filter Sorting
DROP INDEX IF EXISTS jobs_shared_numnodes;
DROP INDEX IF EXISTS jobs_shared_numhwthreads;
DROP INDEX IF EXISTS jobs_shared_numacc;
DROP INDEX IF EXISTS jobs_shared_energy;

-- ArrayJob Filter
DROP INDEX IF EXISTS jobs_cluster_arrayjobid_starttime;

-- Backup Indices For High Variety Columns
DROP INDEX IF EXISTS jobs_duration;

-- Optimize DB index usage
PRAGMA optimize;


     1→-- Migration 12: Add covering index for grouped stats queries
     2→-- Column order: cluster (equality), hpc_user (GROUP BY), start_time (range scan)
     3→-- Includes aggregated columns to avoid main table lookups entirely.
     4→
     5→CREATE INDEX IF NOT EXISTS jobs_cluster_user_starttime_stats
     6→  ON job (cluster, hpc_user, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);
     7→
     8→CREATE INDEX IF NOT EXISTS jobs_cluster_project_starttime_stats
     9→  ON job (cluster, project, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);
    10→
    11→PRAGMA optimize;
    12→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

-- Migration 12: Add covering index for grouped stats queries
-- Column order: cluster (equality), hpc_user (GROUP BY), start_time (range scan)
-- Includes aggregated columns to avoid main table lookups entirely.

CREATE INDEX IF NOT EXISTS jobs_cluster_user_starttime_stats
  ON job (cluster, hpc_user, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);

CREATE INDEX IF NOT EXISTS jobs_cluster_project_starttime_stats
  ON job (cluster, project, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);

PRAGMA optimize;


     1→-- Migration 13: Add covering indexes for status/dashboard queries
     2→-- Column order: cluster (equality), job_state (equality), grouping col, then aggregated columns
     3→-- These indexes allow the status views to be served entirely from index scans.
     4→
     5→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user_stats
     6→  ON job (cluster, job_state, hpc_user, duration, start_time, num_nodes, num_hwthreads, num_acc);
     7→
     8→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project_stats
     9→  ON job (cluster, job_state, project, duration, start_time, num_nodes, num_hwthreads, num_acc);
    10→
    11→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_subcluster_stats
    12→  ON job (cluster, job_state, subcluster, duration, start_time, num_nodes, num_hwthreads, num_acc);
    13→
    14→-- Drop 3-col indexes that are now subsumed by the covering indexes above
    15→DROP INDEX IF EXISTS jobs_cluster_jobstate_user;
    16→DROP INDEX IF EXISTS jobs_cluster_jobstate_project;
    17→
    18→PRAGMA optimize;
    19→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

-- Migration 13: Add covering indexes for status/dashboard queries
-- Column order: cluster (equality), job_state (equality), grouping col, then aggregated columns
-- These indexes allow the status views to be served entirely from index scans.

CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user_stats
  ON job (cluster, job_state, hpc_user, duration, start_time, num_nodes, num_hwthreads, num_acc);

CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project_stats
  ON job (cluster, job_state, project, duration, start_time, num_nodes, num_hwthreads, num_acc);

CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_subcluster_stats
  ON job (cluster, job_state, subcluster, duration, start_time, num_nodes, num_hwthreads, num_acc);

-- Drop 3-col indexes that are now subsumed by the covering indexes above
DROP INDEX IF EXISTS jobs_cluster_jobstate_user;
DROP INDEX IF EXISTS jobs_cluster_jobstate_project;

PRAGMA optimize;


     1→-- Migration 14: Partial covering indexes for running jobs
     2→-- Only running jobs are in the B-tree, so these indexes are tiny compared to
     3→-- the full-table indexes from migration 13. SQLite uses them when the query
     4→-- contains the literal `job_state = 'running'` (not a parameter placeholder).
     5→
     6→CREATE INDEX IF NOT EXISTS jobs_running_user_stats
     7→  ON job (cluster, hpc_user, num_nodes, num_hwthreads, num_acc, duration, start_time)
     8→  WHERE job_state = 'running';
     9→
    10→CREATE INDEX IF NOT EXISTS jobs_running_project_stats
    11→  ON job (cluster, project, num_nodes, num_hwthreads, num_acc, duration, start_time)
    12→  WHERE job_state = 'running';
    13→
    14→CREATE INDEX IF NOT EXISTS jobs_running_subcluster_stats
    15→  ON job (cluster, subcluster, num_nodes, num_hwthreads, num_acc, duration, start_time)
    16→  WHERE job_state = 'running';
    17→
    18→PRAGMA optimize;
    19→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

-- Migration 14: Partial covering indexes for running jobs
-- Only running jobs are in the B-tree, so these indexes are tiny compared to
-- the full-table indexes from migration 13. SQLite uses them when the query
-- contains the literal `job_state = 'running'` (not a parameter placeholder).

CREATE INDEX IF NOT EXISTS jobs_running_user_stats
  ON job (cluster, hpc_user, num_nodes, num_hwthreads, num_acc, duration, start_time)
  WHERE job_state = 'running';

CREATE INDEX IF NOT EXISTS jobs_running_project_stats
  ON job (cluster, project, num_nodes, num_hwthreads, num_acc, duration, start_time)
  WHERE job_state = 'running';

CREATE INDEX IF NOT EXISTS jobs_running_subcluster_stats
  ON job (cluster, subcluster, num_nodes, num_hwthreads, num_acc, duration, start_time)
  WHERE job_state = 'running';

PRAGMA optimize;


Now let me read the down files and migration.go.


     1→-- Migration 11 DOWN: Restore indexes from migration 09
     2→
     3→-- ============================================================
     4→-- Recreate all removed indexes from migration 09
     5→-- ============================================================
     6→
     7→-- Cluster+Partition Filter Sorting
     8→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numnodes ON job (cluster, cluster_partition, num_nodes);
     9→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numhwthreads ON job (cluster, cluster_partition, num_hwthreads);
    10→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numacc ON job (cluster, cluster_partition, num_acc);
    11→CREATE INDEX IF NOT EXISTS jobs_cluster_partition_energy ON job (cluster, cluster_partition, energy);
    12→
    13→-- Cluster+JobState Filter Sorting
    14→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numnodes ON job (cluster, job_state, num_nodes);
    15→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numhwthreads ON job (cluster, job_state, num_hwthreads);
    16→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numacc ON job (cluster, job_state, num_acc);
    17→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_energy ON job (cluster, job_state, energy);
    18→
    19→-- Cluster+Shared Filter Sorting
    20→CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numnodes ON job (cluster, shared, num_nodes);
    21→CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numhwthreads ON job (cluster, shared, num_hwthreads);
    22→CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numacc ON job (cluster, shared, num_acc);
    23→CREATE INDEX IF NOT EXISTS jobs_cluster_shared_energy ON job (cluster, shared, energy);
    24→
    25→-- User Filter Sorting
    26→CREATE INDEX IF NOT EXISTS jobs_user_numnodes ON job (hpc_user, num_nodes);
    27→CREATE INDEX IF NOT EXISTS jobs_user_numhwthreads ON job (hpc_user, num_hwthreads);
    28→CREATE INDEX IF NOT EXISTS jobs_user_numacc ON job (hpc_user, num_acc);
    29→CREATE INDEX IF NOT EXISTS jobs_user_energy ON job (hpc_user, energy);
    30→
    31→-- Project Filter Sorting
    32→CREATE INDEX IF NOT EXISTS jobs_project_numnodes ON job (project, num_nodes);
    33→CREATE INDEX IF NOT EXISTS jobs_project_numhwthreads ON job (project, num_hwthreads);
    34→CREATE INDEX IF NOT EXISTS jobs_project_numacc ON job (project, num_acc);
    35→CREATE INDEX IF NOT EXISTS jobs_project_energy ON job (project, energy);
    36→
    37→-- JobState Filter Sorting
    38→CREATE INDEX IF NOT EXISTS jobs_jobstate_numnodes ON job (job_state, num_nodes);
    39→CREATE INDEX IF NOT EXISTS jobs_jobstate_numhwthreads ON job (job_state, num_hwthreads);
    40→CREATE INDEX IF NOT EXISTS jobs_jobstate_numacc ON job (job_state, num_acc);
    41→CREATE INDEX IF NOT EXISTS jobs_jobstate_energy ON job (job_state, energy);
    42→
    43→-- Shared Filter Sorting
    44→CREATE INDEX IF NOT EXISTS jobs_shared_numnodes ON job (shared, num_nodes);
    45→CREATE INDEX IF NOT EXISTS jobs_shared_numhwthreads ON job (shared, num_hwthreads);
    46→CREATE INDEX IF NOT EXISTS jobs_shared_numacc ON job (shared, num_acc);
    47→CREATE INDEX IF NOT EXISTS jobs_shared_energy ON job (shared, energy);
    48→
    49→-- ArrayJob Filter
    50→CREATE INDEX IF NOT EXISTS jobs_cluster_arrayjobid_starttime ON job (cluster, array_job_id, start_time);
    51→
    52→-- Backup Indices For High Variety Columns
    53→CREATE INDEX IF NOT EXISTS jobs_duration ON job (duration);
    54→
    55→-- Optimize DB index usage
    56→PRAGMA optimize;
    57→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

-- Migration 11 DOWN: Restore indexes from migration 09

-- ============================================================
-- Recreate all removed indexes from migration 09
-- ============================================================

-- Cluster+Partition Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numnodes ON job (cluster, cluster_partition, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numhwthreads ON job (cluster, cluster_partition, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numacc ON job (cluster, cluster_partition, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_energy ON job (cluster, cluster_partition, energy);

-- Cluster+JobState Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numnodes ON job (cluster, job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numhwthreads ON job (cluster, job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numacc ON job (cluster, job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_energy ON job (cluster, job_state, energy);

-- Cluster+Shared Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numnodes ON job (cluster, shared, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numhwthreads ON job (cluster, shared, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numacc ON job (cluster, shared, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_energy ON job (cluster, shared, energy);

-- User Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_user_numnodes ON job (hpc_user, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_user_numhwthreads ON job (hpc_user, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_user_numacc ON job (hpc_user, num_acc);
CREATE INDEX IF NOT EXISTS jobs_user_energy ON job (hpc_user, energy);

-- Project Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_project_numnodes ON job (project, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_project_numhwthreads ON job (project, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_project_numacc ON job (project, num_acc);
CREATE INDEX IF NOT EXISTS jobs_project_energy ON job (project, energy);

-- JobState Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_jobstate_numnodes ON job (job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numhwthreads ON job (job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numacc ON job (job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_jobstate_energy ON job (job_state, energy);

-- Shared Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_shared_numnodes ON job (shared, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_shared_numhwthreads ON job (shared, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_shared_numacc ON job (shared, num_acc);
CREATE INDEX IF NOT EXISTS jobs_shared_energy ON job (shared, energy);

-- ArrayJob Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_arrayjobid_starttime ON job (cluster, array_job_id, start_time);

-- Backup Indices For High Variety Columns
CREATE INDEX IF NOT EXISTS jobs_duration ON job (duration);

-- Optimize DB index usage
PRAGMA optimize;


     1→DROP INDEX IF EXISTS jobs_cluster_user_starttime_stats;
     2→DROP INDEX IF EXISTS jobs_cluster_project_starttime_stats;
     3→
     4→PRAGMA optimize;
     5→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

DROP INDEX IF EXISTS jobs_cluster_user_starttime_stats;
DROP INDEX IF EXISTS jobs_cluster_project_starttime_stats;

PRAGMA optimize;


     1→-- Reverse migration 13: Remove covering status indexes, restore 3-col indexes
     2→
     3→DROP INDEX IF EXISTS jobs_cluster_jobstate_user_stats;
     4→DROP INDEX IF EXISTS jobs_cluster_jobstate_project_stats;
     5→DROP INDEX IF EXISTS jobs_cluster_jobstate_subcluster_stats;
     6→
     7→-- Restore the original 3-col indexes
     8→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user
     9→  ON job (cluster, job_state, hpc_user);
    10→
    11→CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project
    12→  ON job (cluster, job_state, project);
    13→
    14→  PRAGMA optimize;
    15→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

-- Reverse migration 13: Remove covering status indexes, restore 3-col indexes

DROP INDEX IF EXISTS jobs_cluster_jobstate_user_stats;
DROP INDEX IF EXISTS jobs_cluster_jobstate_project_stats;
DROP INDEX IF EXISTS jobs_cluster_jobstate_subcluster_stats;

-- Restore the original 3-col indexes
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user
  ON job (cluster, job_state, hpc_user);

CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project
  ON job (cluster, job_state, project);

  PRAGMA optimize;


     1→-- Reverse migration 14: Drop partial indexes for running jobs
     2→
     3→DROP INDEX IF EXISTS jobs_running_user_stats;
     4→DROP INDEX IF EXISTS jobs_running_project_stats;
     5→DROP INDEX IF EXISTS jobs_running_subcluster_stats;
     6→
     7→PRAGMA optimize;
     8→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

-- Reverse migration 14: Drop partial indexes for running jobs

DROP INDEX IF EXISTS jobs_running_user_stats;
DROP INDEX IF EXISTS jobs_running_project_stats;
DROP INDEX IF EXISTS jobs_running_subcluster_stats;

PRAGMA optimize;


No matches found


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package repository
     7→
     8→import (
     9→	"database/sql"
    10→	"embed"
    11→	"fmt"
    12→
    13→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    14→	"github.com/golang-migrate/migrate/v4"
    15→	"github.com/golang-migrate/migrate/v4/database/sqlite3"
    16→	"github.com/golang-migrate/migrate/v4/source/iofs"
    17→)
    18→
    19→// Version is the current database schema version required by this version of cc-backend.
    20→// When the database schema changes, this version is incremented and a new migration file
    21→// is added to internal/repository/migrations/sqlite3/.
    22→//
    23→// Version history:
    24→//   - Version 14: Partial covering indexes for running jobs (tiny B-tree vs millions)
    25→//   - Version 13: Add covering indexes for status/dashboard queries (cluster, job_state, ...)
    26→//   - Version 12: Add covering index for stats queries (cluster, start_time, hpc_user, ...)
    27→//   - Version 11: Optimize job table indexes (reduce from ~78 to 48)
    28→//   - Version 10: Node table
    29→//
    30→// Migration files are embedded at build time from the migrations directory.
    31→const Version uint = 14
    32→
    33→//go:embed migrations/*
    34→var migrationFiles embed.FS
    35→
    36→// checkDBVersion verifies that the database schema version matches the expected version.
    37→// This is called automatically during Connect() to ensure schema compatibility.
    38→//
    39→// Returns an error if:
    40→//   - Database version is older than expected (needs migration)
    41→//   - Database version is newer than expected (needs app upgrade)
    42→//   - Database is in a dirty state (failed migration)
    43→//
    44→// A "dirty" database indicates a migration was started but not completed successfully.
    45→// This requires manual intervention to fix the database and force the version.
    46→func checkDBVersion(db *sql.DB) error {
    47→	driver, err := sqlite3.WithInstance(db, &sqlite3.Config{})
    48→	if err != nil {
    49→		return err
    50→	}
    51→	d, err := iofs.New(migrationFiles, "migrations/sqlite3")
    52→	if err != nil {
    53→		return err
    54→	}
    55→
    56→	m, err := migrate.NewWithInstance("iofs", d, "sqlite3", driver)
    57→	if err != nil {
    58→		return err
    59→	}
    60→
    61→	v, dirty, err := m.Version()
    62→	if err != nil {
    63→		if err == migrate.ErrNilVersion {
    64→			cclog.Warn("Legacy database without version or missing database file!")
    65→		} else {
    66→			return err
    67→		}
    68→	}
    69→
    70→	if v < Version {
    71→		return fmt.Errorf("unsupported database version %d, need %d.\nPlease backup your database file and run cc-backend -migrate-db", v, Version)
    72→	} else if v > Version {
    73→		return fmt.Errorf("unsupported database version %d, need %d.\nPlease refer to documentation how to downgrade db with external migrate tool", v, Version)
    74→	}
    75→
    76→	if dirty {
    77→		return fmt.Errorf("last migration to version %d has failed, please fix the db manually and force version with -force-db flag", Version)
    78→	}
    79→
    80→	return nil
    81→}
    82→
    83→// getMigrateInstance creates a new migration instance for the given database file.
    84→// This is used internally by MigrateDB, RevertDB, and ForceDB.
    85→func getMigrateInstance(db string) (m *migrate.Migrate, err error) {
    86→	d, err := iofs.New(migrationFiles, "migrations/sqlite3")
    87→	if err != nil {
    88→		return nil, err
    89→	}
    90→
    91→	m, err = migrate.NewWithSourceInstance("iofs", d, fmt.Sprintf("sqlite3://%s?_foreign_keys=on", db))
    92→	if err != nil {
    93→		return nil, err
    94→	}
    95→
    96→	return m, nil
    97→}
    98→
    99→// MigrateDB applies all pending database migrations to bring the schema up to date.
   100→// This should be run with the -migrate-db flag before starting the application
   101→// after upgrading to a new version that requires schema changes.
   102→//
   103→// Process:
   104→//  1. Checks current database version
   105→//  2. Applies all migrations from current version to target Version
   106→//  3. Updates schema_migrations table to track applied migrations
   107→//
   108→// Important:
   109→//   - Always backup your database before running migrations
   110→//   - Migrations are irreversible without manual intervention
   111→//   - If a migration fails, the database is marked "dirty" and requires manual fix
   112→//
   113→// Usage:
   114→//
   115→//	cc-backend -migrate-db
   116→func MigrateDB(db string) error {
   117→	m, err := getMigrateInstance(db)
   118→	if err != nil {
   119→		return err
   120→	}
   121→
   122→	v, dirty, err := m.Version()
   123→	if err != nil {
   124→		if err == migrate.ErrNilVersion {
   125→			cclog.Info("Legacy database without version or missing database file!")
   126→		} else {
   127→			return err
   128→		}
   129→	}
   130→
   131→	if v < Version {
   132→		cclog.Infof("unsupported database version %d, need %d.\nPlease backup your database file and run cc-backend -migrate-db", v, Version)
   133→	}
   134→
   135→	if dirty {
   136→		return fmt.Errorf("last migration to version %d has failed, please fix the db manually and force version with -force-db flag", Version)
   137→	}
   138→
   139→	if err := m.Up(); err != nil {
   140→		if err == migrate.ErrNoChange {
   141→			cclog.Info("DB already up to date!")
   142→		} else {
   143→			return err
   144→		}
   145→	}
   146→
   147→	m.Close()
   148→	return nil
   149→}
   150→
   151→// RevertDB rolls back the database schema to the previous version (Version - 1).
   152→// This is primarily used for testing or emergency rollback scenarios.
   153→//
   154→// Warning:
   155→//   - This may cause data loss if newer schema added columns/tables
   156→//   - Always backup before reverting
   157→//   - Not all migrations are safely reversible
   158→//
   159→// Usage:
   160→//
   161→//	cc-backend -revert-db
   162→func RevertDB(db string) error {
   163→	m, err := getMigrateInstance(db)
   164→	if err != nil {
   165→		return err
   166→	}
   167→
   168→	if err := m.Migrate(Version - 1); err != nil {
   169→		if err == migrate.ErrNoChange {
   170→			cclog.Info("DB already up to date!")
   171→		} else {
   172→			return err
   173→		}
   174→	}
   175→
   176→	m.Close()
   177→	return nil
   178→}
   179→
   180→// ForceDB forces the database schema version to the current Version without running migrations.
   181→// This is only used to recover from failed migrations that left the database in a "dirty" state.
   182→//
   183→// When to use:
   184→//   - After manually fixing a failed migration
   185→//   - When you've manually applied schema changes and need to update the version marker
   186→//
   187→// Warning:
   188→//   - This does NOT apply any schema changes
   189→//   - Only use after manually verifying the schema is correct
   190→//   - Improper use can cause schema/version mismatch
   191→//
   192→// Usage:
   193→//
   194→//	cc-backend -force-db
   195→func ForceDB(db string) error {
   196→	m, err := getMigrateInstance(db)
   197→	if err != nil {
   198→		return err
   199→	}
   200→
   201→	if err := m.Force(int(Version)); err != nil {
   202→		return err
   203→	}
   204→
   205→	m.Close()
   206→	return nil
   207→}
   208→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package repository

import (
	"database/sql"
	"embed"
	"fmt"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/golang-migrate/migrate/v4"
	"github.com/golang-migrate/migrate/v4/database/sqlite3"
	"github.com/golang-migrate/migrate/v4/source/iofs"
)

// Version is the current database schema version required by this version of cc-backend.
// When the database schema changes, this version is incremented and a new migration file
// is added to internal/repository/migrations/sqlite3/.
//
// Version history:
//   - Version 14: Partial covering indexes for running jobs (tiny B-tree vs millions)
//   - Version 13: Add covering indexes for status/dashboard queries (cluster, job_state, ...)
//   - Version 12: Add covering index for stats queries (cluster, start_time, hpc_user, ...)
//   - Version 11: Optimize job table indexes (reduce from ~78 to 48)
//   - Version 10: Node table
//
// Migration files are embedded at build time from the migrations directory.
const Version uint = 14

//go:embed migrations/*
var migrationFiles embed.FS

// checkDBVersion verifies that the database schema version matches the expected version.
// This is called automatically during Connect() to ensure schema compatibility.
//
// Returns an error if:
//   - Database version is older than expected (needs migration)
//   - Database version is newer than expected (needs app upgrade)
//   - Database is in a dirty state (failed migration)
//
// A "dirty" database indicates a migration was started but not completed successfully.
// This requires manual intervention to fix the database and force the version.
func checkDBVersion(db *sql.DB) error {
	driver, err := sqlite3.WithInstance(db, &sqlite3.Config{})
	if err != nil {
		return err
	}
	d, err := iofs.New(migrationFiles, "migrations/sqlite3")
	if err != nil {
		return err
	}

	m, err := migrate.NewWithInstance("iofs", d, "sqlite3", driver)
	if err != nil {
		return err
	}

	v, dirty, err := m.Version()
	if err != nil {
		if err == migrate.ErrNilVersion {
			cclog.Warn("Legacy database without version or missing database file!")
		} else {
			return err
		}
	}

	if v < Version {
		return fmt.Errorf("unsupported database version %d, need %d.\nPlease backup your database file and run cc-backend -migrate-db", v, Version)
	} else if v > Version {
		return fmt.Errorf("unsupported database version %d, need %d.\nPlease refer to documentation how to downgrade db with external migrate tool", v, Version)
	}

	if dirty {
		return fmt.Errorf("last migration to version %d has failed, please fix the db manually and force version with -force-db flag", Version)
	}

	return nil
}

// getMigrateInstance creates a new migration instance for the given database file.
// This is used internally by MigrateDB, RevertDB, and ForceDB.
func getMigrateInstance(db string) (m *migrate.Migrate, err error) {
	d, err := iofs.New(migrationFiles, "migrations/sqlite3")
	if err != nil {
		return nil, err
	}

	m, err = migrate.NewWithSourceInstance("iofs", d, fmt.Sprintf("sqlite3://%s?_foreign_keys=on", db))
	if err != nil {
		return nil, err
	}

	return m, nil
}

// MigrateDB applies all pending database migrations to bring the schema up to date.
// This should be run with the -migrate-db flag before starting the application
// after upgrading to a new version that requires schema changes.
//
// Process:
//  1. Checks current database version
//  2. Applies all migrations from current version to target Version
//  3. Updates schema_migrations table to track applied migrations
//
// Important:
//   - Always backup your database before running migrations
//   - Migrations are irreversible without manual intervention
//   - If a migration fails, the database is marked "dirty" and requires manual fix
//
// Usage:
//
//	cc-backend -migrate-db
func MigrateDB(db string) error {
	m, err := getMigrateInstance(db)
	if err != nil {
		return err
	}

	v, dirty, err := m.Version()
	if err != nil {
		if err == migrate.ErrNilVersion {
			cclog.Info("Legacy database without version or missing database file!")
		} else {
			return err
		}
	}

	if v < Version {
		cclog.Infof("unsupported database version %d, need %d.\nPlease backup your database file and run cc-backend -migrate-db", v, Version)
	}

	if dirty {
		return fmt.Errorf("last migration to version %d has failed, please fix the db manually and force version with -force-db flag", Version)
	}

	if err := m.Up(); err != nil {
		if err == migrate.ErrNoChange {
			cclog.Info("DB already up to date!")
		} else {
			return err
		}
	}

	m.Close()
	return nil
}

// RevertDB rolls back the database schema to the previous version (Version - 1).
// This is primarily used for testing or emergency rollback scenarios.
//
// Warning:
//   - This may cause data loss if newer schema added columns/tables
//   - Always backup before reverting
//   - Not all migrations are safely reversible
//
// Usage:
//
//	cc-backend -revert-db
func RevertDB(db string) error {
	m, err := getMigrateInstance(db)
	if err != nil {
		return err
	}

	if err := m.Migrate(Version - 1); err != nil {
		if err == migrate.ErrNoChange {
			cclog.Info("DB already up to date!")
		} else {
			return err
		}
	}

	m.Close()
	return nil
}

// ForceDB forces the database schema version to the current Version without running migrations.
// This is only used to recover from failed migrations that left the database in a "dirty" state.
//
// When to use:
//   - After manually fixing a failed migration
//   - When you've manually applied schema changes and need to update the version marker
//
// Warning:
//   - This does NOT apply any schema changes
//   - Only use after manually verifying the schema is correct
//   - Improper use can cause schema/version mismatch
//
// Usage:
//
//	cc-backend -force-db
func ForceDB(db string) error {
	m, err := getMigrateInstance(db)
	if err != nil {
		return err
	}

	if err := m.Force(int(Version)); err != nil {
		return err
	}

	m.Close()
	return nil
}


I'll consolidate migrations 11–14 into a single migration 11, then delete 12–14, and update `Version` to 11.


Write

-- Migration 11: Optimize job table indexes
-- Consolidates former migrations 11–14:
--   - Remove overly specific sorting indexes (reduces ~78 → 48)
--   - Add covering index for grouped stats queries
--   - Add covering indexes for status/dashboard queries
--   - Add partial covering indexes for running jobs (tiny B-tree)
-- Requires ANALYZE to be run after migration (done automatically on startup).

-- ============================================================
-- Drop SELECTED existing job indexes (from migrations 08/09)
-- sqlite_autoindex_job_1 (UNIQUE constraint) is kept automatically
-- ============================================================

-- Cluster+Partition Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_partition_numnodes;
DROP INDEX IF EXISTS jobs_cluster_partition_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_partition_numacc;
DROP INDEX IF EXISTS jobs_cluster_partition_energy;

-- Cluster+JobState Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_jobstate_numnodes;
DROP INDEX IF EXISTS jobs_cluster_jobstate_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_jobstate_numacc;
DROP INDEX IF EXISTS jobs_cluster_jobstate_energy;

-- Cluster+Shared Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_shared_numnodes;
DROP INDEX IF EXISTS jobs_cluster_shared_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_shared_numacc;
DROP INDEX IF EXISTS jobs_cluster_shared_energy;

-- User Filter Sorting
DROP INDEX IF EXISTS jobs_user_numnodes;
DROP INDEX IF EXISTS jobs_user_numhwthreads;
DROP INDEX IF EXISTS jobs_user_numacc;
DROP INDEX IF EXISTS jobs_user_energy;

-- Project Filter Sorting
DROP INDEX IF EXISTS jobs_project_numnodes;
DROP INDEX IF EXISTS jobs_project_numhwthreads;
DROP INDEX IF EXISTS jobs_project_numacc;
DROP INDEX IF EXISTS jobs_project_energy;

-- JobState Filter Sorting
DROP INDEX IF EXISTS jobs_jobstate_numnodes;
DROP INDEX IF EXISTS jobs_jobstate_numhwthreads;
DROP INDEX IF EXISTS jobs_jobstate_numacc;
DROP INDEX IF EXISTS jobs_jobstate_energy;

-- Shared Filter Sorting
DROP INDEX IF EXISTS jobs_shared_numnodes;
DROP INDEX IF EXISTS jobs_shared_numhwthreads;
DROP INDEX IF EXISTS jobs_shared_numacc;
DROP INDEX IF EXISTS jobs_shared_energy;

-- ArrayJob Filter
DROP INDEX IF EXISTS jobs_cluster_arrayjobid_starttime;

-- Backup Indices For High Variety Columns
DROP INDEX IF EXISTS jobs_duration;

-- ============================================================
-- Covering indexes for grouped stats queries
-- Column order: cluster (equality), hpc_user/project (GROUP BY), start_time (range scan)
-- Includes aggregated columns to avoid main table lookups entirely.
-- ============================================================

CREATE INDEX IF NOT EXISTS jobs_cluster_user_starttime_stats
  ON job (cluster, hpc_user, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);

CREATE INDEX IF NOT EXISTS jobs_cluster_project_starttime_stats
  ON job (cluster, project, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);

-- ============================================================
-- Covering indexes for status/dashboard queries
-- Column order: cluster (equality), job_state (equality), grouping col, then aggregated columns
-- These indexes allow the status views to be served entirely from index scans.
-- ============================================================

-- Drop 3-col indexes that are subsumed by the covering indexes below
DROP INDEX IF EXISTS jobs_cluster_jobstate_user;
DROP INDEX IF EXISTS jobs_cluster_jobstate_project;

CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user_stats
  ON job (cluster, job_state, hpc_user, duration, start_time, num_nodes, num_hwthreads, num_acc);

CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project_stats
  ON job (cluster, job_state, project, duration, start_time, num_nodes, num_hwthreads, num_acc);

CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_subcluster_stats
  ON job (cluster, job_state, subcluster, duration, start_time, num_nodes, num_hwthreads, num_acc);

-- ============================================================
-- Partial covering indexes for running jobs
-- Only running jobs are in the B-tree, so these indexes are tiny compared to
-- the full-table indexes above. SQLite uses them when the query contains the
-- literal `job_state = 'running'` (not a parameter placeholder).
-- ============================================================

CREATE INDEX IF NOT EXISTS jobs_running_user_stats
  ON job (cluster, hpc_user, num_nodes, num_hwthreads, num_acc, duration, start_time)
  WHERE job_state = 'running';

CREATE INDEX IF NOT EXISTS jobs_running_project_stats
  ON job (cluster, project, num_nodes, num_hwthreads, num_acc, duration, start_time)
  WHERE job_state = 'running';

CREATE INDEX IF NOT EXISTS jobs_running_subcluster_stats
  ON job (cluster, subcluster, num_nodes, num_hwthreads, num_acc, duration, start_time)
  WHERE job_state = 'running';

PRAGMA optimize;


The file /Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/migrations/sqlite3/11_optimize-indexes.up.sql has been updated successfully.

-- Migration 11: Optimize job table indexes
-- Consolidates former migrations 11–14:
--   - Remove overly specific sorting indexes (reduces ~78 → 48)
--   - Add covering index for grouped stats queries
--   - Add covering indexes for status/dashboard queries
--   - Add partial covering indexes for running jobs (tiny B-tree)
-- Requires ANALYZE to be run after migration (done automatically on startup).

-- ============================================================
-- Drop SELECTED existing job indexes (from migrations 08/09)
-- sqlite_autoindex_job_1 (UNIQUE constraint) is kept automatically
-- ============================================================

-- Cluster+Partition Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_partition_numnodes;
DROP INDEX IF EXISTS jobs_cluster_partition_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_partition_numacc;
DROP INDEX IF EXISTS jobs_cluster_partition_energy;

-- Cluster+JobState Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_jobstate_numnodes;
DROP INDEX IF EXISTS jobs_cluster_jobstate_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_jobstate_numacc;
DROP INDEX IF EXISTS jobs_cluster_jobstate_energy;

-- Cluster+Shared Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_shared_numnodes;
DROP INDEX IF EXISTS jobs_cluster_shared_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_shared_numacc;
DROP INDEX IF EXISTS jobs_cluster_shared_energy;

-- User Filter Sorting
DROP INDEX IF EXISTS jobs_user_numnodes;
DROP INDEX IF EXISTS jobs_user_numhwthreads;
DROP INDEX IF EXISTS jobs_user_numacc;
DROP INDEX IF EXISTS jobs_user_energy;

-- Project Filter Sorting
DROP INDEX IF EXISTS jobs_project_numnodes;
DROP INDEX IF EXISTS jobs_project_numhwthreads;
DROP INDEX IF EXISTS jobs_project_numacc;
DROP INDEX IF EXISTS jobs_project_energy;

-- JobState Filter Sorting
DROP INDEX IF EXISTS jobs_jobstate_numnodes;
DROP INDEX IF EXISTS jobs_jobstate_numhwthreads;
DROP INDEX IF EXISTS jobs_jobstate_numacc;
DROP INDEX IF EXISTS jobs_jobstate_energy;

-- Shared Filter Sorting
DROP INDEX IF EXISTS jobs_shared_numnodes;
DROP INDEX IF EXISTS jobs_shared_numhwthreads;
DROP INDEX IF EXISTS jobs_shared_numacc;
DROP INDEX IF EXISTS jobs_shared_energy;

-- ArrayJob Filter
DROP INDEX IF EXISTS jobs_cluster_arrayjobid_starttime;

-- Backup Indices For High Variety Columns
DROP INDEX IF EXISTS jobs_duration;

-- ============================================================
-- Covering indexes for grouped stats queries
-- Column order: cluster (equality), hpc_user/project (GROUP BY), start_time (range scan)
-- Includes aggregated columns to avoid main table lookups entirely.
-- ============================================================

CREATE INDEX IF NOT EXISTS jobs_cluster_user_starttime_stats
  ON job (cluster, hpc_user, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);

CREATE INDEX IF NOT EXISTS jobs_cluster_project_starttime_stats
  ON job (cluster, project, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);

-- ============================================================
-- Covering indexes for status/dashboard queries
-- Column order: cluster (equality), job_state (equality), grouping col, then aggregated columns
-- These indexes allow the status views to be served entirely from index scans.
-- ============================================================

-- Drop 3-col indexes that are subsumed by the covering indexes below
DROP INDEX IF EXISTS jobs_cluster_jobstate_user;
DROP INDEX IF EXISTS jobs_cluster_jobstate_project;

CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user_stats
  ON job (cluster, job_state, hpc_user, duration, start_time, num_nodes, num_hwthreads, num_acc);

CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project_stats
  ON job (cluster, job_state, project, duration, start_time, num_nodes, num_hwthreads, num_acc);

CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_subcluster_stats
  ON job (cluster, job_state, subcluster, duration, start_time, num_nodes, num_hwthreads, num_acc);

-- ============================================================
-- Partial covering indexes for running jobs
-- Only running jobs are in the B-tree, so these indexes are tiny compared to
-- the full-table indexes above. SQLite uses them when the query contains the
-- literal `job_state = 'running'` (not a parameter placeholder).
-- ============================================================

CREATE INDEX IF NOT EXISTS jobs_running_user_stats
  ON job (cluster, hpc_user, num_nodes, num_hwthreads, num_acc, duration, start_time)
  WHERE job_state = 'running';

CREATE INDEX IF NOT EXISTS jobs_running_project_stats
  ON job (cluster, project, num_nodes, num_hwthreads, num_acc, duration, start_time)
  WHERE job_state = 'running';

CREATE INDEX IF NOT EXISTS jobs_running_subcluster_stats
  ON job (cluster, subcluster, num_nodes, num_hwthreads, num_acc, duration, start_time)
  WHERE job_state = 'running';

PRAGMA optimize;

--- Migration 11: Remove overly specific table indexes formerly used in sorting

--- When one or two indexed columns are used, sorting usually is fast

--- Reduces from ~78 indexes to 48 for better write performance,

--- reduced disk usage, and more reliable query planner decisions.

+-- Migration 11: Optimize job table indexes

+-- Consolidates former migrations 11–14:

+--   - Remove overly specific sorting indexes (reduces ~78 → 48)

+--   - Add covering index for grouped stats queries

+--   - Add covering indexes for status/dashboard queries

+--   - Add partial covering indexes for running jobs (tiny B-tree)

+-- Covering indexes for grouped stats queries

+-- Column order: cluster (equality), hpc_user/project (GROUP BY), start_time (range scan)

+-- Includes aggregated columns to avoid main table lookups entirely.

+CREATE INDEX IF NOT EXISTS jobs_cluster_user_starttime_stats

+  ON job (cluster, hpc_user, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);

+CREATE INDEX IF NOT EXISTS jobs_cluster_project_starttime_stats

+  ON job (cluster, project, start_time, duration, job_state, num_nodes, num_hwthreads, num_acc);

+-- Covering indexes for status/dashboard queries

+-- Column order: cluster (equality), job_state (equality), grouping col, then aggregated columns

+-- These indexes allow the status views to be served entirely from index scans.

+-- Drop 3-col indexes that are subsumed by the covering indexes below

+DROP INDEX IF EXISTS jobs_cluster_jobstate_user;

+DROP INDEX IF EXISTS jobs_cluster_jobstate_project;

+CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user_stats

+  ON job (cluster, job_state, hpc_user, duration, start_time, num_nodes, num_hwthreads, num_acc);

+CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project_stats

+  ON job (cluster, job_state, project, duration, start_time, num_nodes, num_hwthreads, num_acc);

+CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_subcluster_stats

+  ON job (cluster, job_state, subcluster, duration, start_time, num_nodes, num_hwthreads, num_acc);

+-- Partial covering indexes for running jobs

+-- Only running jobs are in the B-tree, so these indexes are tiny compared to

+-- the full-table indexes above. SQLite uses them when the query contains the

+-- literal `job_state = 'running'` (not a parameter placeholder).

+CREATE INDEX IF NOT EXISTS jobs_running_user_stats

+  ON job (cluster, hpc_user, num_nodes, num_hwthreads, num_acc, duration, start_time)

+CREATE INDEX IF NOT EXISTS jobs_running_project_stats

+  ON job (cluster, project, num_nodes, num_hwthreads, num_acc, duration, start_time)

+CREATE INDEX IF NOT EXISTS jobs_running_subcluster_stats

+  ON job (cluster, subcluster, num_nodes, num_hwthreads, num_acc, duration, start_time)

-- Migration 11: Remove overly specific table indexes formerly used in sorting
-- When one or two indexed columns are used, sorting usually is fast
-- Reduces from ~78 indexes to 48 for better write performance,
-- reduced disk usage, and more reliable query planner decisions.
-- Requires ANALYZE to be run after migration (done automatically on startup).

-- ============================================================
-- Drop SELECTED existing job indexes (from migrations 08/09)
-- sqlite_autoindex_job_1 (UNIQUE constraint) is kept automatically
-- ============================================================

-- Cluster+Partition Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_partition_numnodes;
DROP INDEX IF EXISTS jobs_cluster_partition_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_partition_numacc;
DROP INDEX IF EXISTS jobs_cluster_partition_energy;

-- Cluster+JobState Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_jobstate_numnodes;
DROP INDEX IF EXISTS jobs_cluster_jobstate_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_jobstate_numacc;
DROP INDEX IF EXISTS jobs_cluster_jobstate_energy;

-- Cluster+Shared Filter Sorting
DROP INDEX IF EXISTS jobs_cluster_shared_numnodes;
DROP INDEX IF EXISTS jobs_cluster_shared_numhwthreads;
DROP INDEX IF EXISTS jobs_cluster_shared_numacc;
DROP INDEX IF EXISTS jobs_cluster_shared_energy;

-- User Filter Sorting
DROP INDEX IF EXISTS jobs_user_numnodes;
DROP INDEX IF EXISTS jobs_user_numhwthreads;
DROP INDEX IF EXISTS jobs_user_numacc;
DROP INDEX IF EXISTS jobs_user_energy;

-- Project Filter Sorting
DROP INDEX IF EXISTS jobs_project_numnodes;
DROP INDEX IF EXISTS jobs_project_numhwthreads;
DROP INDEX IF EXISTS jobs_project_numacc;
DROP INDEX IF EXISTS jobs_project_energy;

-- JobState Filter Sorting
DROP INDEX IF EXISTS jobs_jobstate_numnodes;
DROP INDEX IF EXISTS jobs_jobstate_numhwthreads;
DROP INDEX IF EXISTS jobs_jobstate_numacc;
DROP INDEX IF EXISTS jobs_jobstate_energy;

-- Shared Filter Sorting
DROP INDEX IF EXISTS jobs_shared_numnodes;
DROP INDEX IF EXISTS jobs_shared_numhwthreads;
DROP INDEX IF EXISTS jobs_shared_numacc;
DROP INDEX IF EXISTS jobs_shared_energy;

-- ArrayJob Filter
DROP INDEX IF EXISTS jobs_cluster_arrayjobid_starttime;

-- Backup Indices For High Variety Columns
DROP INDEX IF EXISTS jobs_duration;

-- Optimize DB index usage
PRAGMA optimize;


-- Migration 11 DOWN: Restore indexes from migration 09

-- Drop partial indexes for running jobs
DROP INDEX IF EXISTS jobs_running_user_stats;
DROP INDEX IF EXISTS jobs_running_project_stats;
DROP INDEX IF EXISTS jobs_running_subcluster_stats;

-- Drop covering status indexes, restore 3-col indexes
DROP INDEX IF EXISTS jobs_cluster_jobstate_user_stats;
DROP INDEX IF EXISTS jobs_cluster_jobstate_project_stats;
DROP INDEX IF EXISTS jobs_cluster_jobstate_subcluster_stats;

CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user
  ON job (cluster, job_state, hpc_user);

CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project
  ON job (cluster, job_state, project);

-- Drop covering stats indexes
DROP INDEX IF EXISTS jobs_cluster_user_starttime_stats;
DROP INDEX IF EXISTS jobs_cluster_project_starttime_stats;

-- Recreate all removed indexes from migration 09

-- Cluster+Partition Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numnodes ON job (cluster, cluster_partition, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numhwthreads ON job (cluster, cluster_partition, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numacc ON job (cluster, cluster_partition, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_energy ON job (cluster, cluster_partition, energy);

-- Cluster+JobState Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numnodes ON job (cluster, job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numhwthreads ON job (cluster, job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numacc ON job (cluster, job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_energy ON job (cluster, job_state, energy);

-- Cluster+Shared Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numnodes ON job (cluster, shared, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numhwthreads ON job (cluster, shared, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numacc ON job (cluster, shared, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_energy ON job (cluster, shared, energy);

-- User Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_user_numnodes ON job (hpc_user, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_user_numhwthreads ON job (hpc_user, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_user_numacc ON job (hpc_user, num_acc);
CREATE INDEX IF NOT EXISTS jobs_user_energy ON job (hpc_user, energy);

-- Project Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_project_numnodes ON job (project, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_project_numhwthreads ON job (project, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_project_numacc ON job (project, num_acc);
CREATE INDEX IF NOT EXISTS jobs_project_energy ON job (project, energy);

-- JobState Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_jobstate_numnodes ON job (job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numhwthreads ON job (job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numacc ON job (job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_jobstate_energy ON job (job_state, energy);

-- Shared Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_shared_numnodes ON job (shared, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_shared_numhwthreads ON job (shared, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_shared_numacc ON job (shared, num_acc);
CREATE INDEX IF NOT EXISTS jobs_shared_energy ON job (shared, energy);

-- ArrayJob Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_arrayjobid_starttime ON job (cluster, array_job_id, start_time);

-- Backup Indices For High Variety Columns
CREATE INDEX IF NOT EXISTS jobs_duration ON job (duration);

PRAGMA optimize;


The file /Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/migrations/sqlite3/11_optimize-indexes.down.sql has been updated successfully.

-- Migration 11 DOWN: Restore indexes from migration 09

-- Drop partial indexes for running jobs
DROP INDEX IF EXISTS jobs_running_user_stats;
DROP INDEX IF EXISTS jobs_running_project_stats;
DROP INDEX IF EXISTS jobs_running_subcluster_stats;

-- Drop covering status indexes, restore 3-col indexes
DROP INDEX IF EXISTS jobs_cluster_jobstate_user_stats;
DROP INDEX IF EXISTS jobs_cluster_jobstate_project_stats;
DROP INDEX IF EXISTS jobs_cluster_jobstate_subcluster_stats;

CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user
  ON job (cluster, job_state, hpc_user);

CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project
  ON job (cluster, job_state, project);

-- Drop covering stats indexes
DROP INDEX IF EXISTS jobs_cluster_user_starttime_stats;
DROP INDEX IF EXISTS jobs_cluster_project_starttime_stats;

-- Recreate all removed indexes from migration 09

-- Cluster+Partition Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numnodes ON job (cluster, cluster_partition, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numhwthreads ON job (cluster, cluster_partition, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numacc ON job (cluster, cluster_partition, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_energy ON job (cluster, cluster_partition, energy);

-- Cluster+JobState Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numnodes ON job (cluster, job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numhwthreads ON job (cluster, job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numacc ON job (cluster, job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_energy ON job (cluster, job_state, energy);

-- Cluster+Shared Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numnodes ON job (cluster, shared, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numhwthreads ON job (cluster, shared, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numacc ON job (cluster, shared, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_energy ON job (cluster, shared, energy);

-- User Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_user_numnodes ON job (hpc_user, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_user_numhwthreads ON job (hpc_user, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_user_numacc ON job (hpc_user, num_acc);
CREATE INDEX IF NOT EXISTS jobs_user_energy ON job (hpc_user, energy);

-- Project Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_project_numnodes ON job (project, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_project_numhwthreads ON job (project, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_project_numacc ON job (project, num_acc);
CREATE INDEX IF NOT EXISTS jobs_project_energy ON job (project, energy);

-- JobState Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_jobstate_numnodes ON job (job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numhwthreads ON job (job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numacc ON job (job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_jobstate_energy ON job (job_state, energy);

-- Shared Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_shared_numnodes ON job (shared, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_shared_numhwthreads ON job (shared, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_shared_numacc ON job (shared, num_acc);
CREATE INDEX IF NOT EXISTS jobs_shared_energy ON job (shared, energy);

-- ArrayJob Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_arrayjobid_starttime ON job (cluster, array_job_id, start_time);

-- Backup Indices For High Variety Columns
CREATE INDEX IF NOT EXISTS jobs_duration ON job (duration);

PRAGMA optimize;

+-- Drop partial indexes for running jobs

+DROP INDEX IF EXISTS jobs_running_user_stats;

+DROP INDEX IF EXISTS jobs_running_project_stats;

+DROP INDEX IF EXISTS jobs_running_subcluster_stats;

+-- Drop covering status indexes, restore 3-col indexes

+DROP INDEX IF EXISTS jobs_cluster_jobstate_user_stats;

+DROP INDEX IF EXISTS jobs_cluster_jobstate_project_stats;

+DROP INDEX IF EXISTS jobs_cluster_jobstate_subcluster_stats;

+CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_user

+  ON job (cluster, job_state, hpc_user);

+CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_project

+  ON job (cluster, job_state, project);

+DROP INDEX IF EXISTS jobs_cluster_user_starttime_stats;

+DROP INDEX IF EXISTS jobs_cluster_project_starttime_stats;

-- Migration 11 DOWN: Restore indexes from migration 09

-- ============================================================
-- Recreate all removed indexes from migration 09
-- ============================================================

-- Cluster+Partition Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numnodes ON job (cluster, cluster_partition, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numhwthreads ON job (cluster, cluster_partition, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_numacc ON job (cluster, cluster_partition, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_partition_energy ON job (cluster, cluster_partition, energy);

-- Cluster+JobState Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numnodes ON job (cluster, job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numhwthreads ON job (cluster, job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_numacc ON job (cluster, job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_jobstate_energy ON job (cluster, job_state, energy);

-- Cluster+Shared Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numnodes ON job (cluster, shared, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numhwthreads ON job (cluster, shared, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_numacc ON job (cluster, shared, num_acc);
CREATE INDEX IF NOT EXISTS jobs_cluster_shared_energy ON job (cluster, shared, energy);

-- User Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_user_numnodes ON job (hpc_user, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_user_numhwthreads ON job (hpc_user, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_user_numacc ON job (hpc_user, num_acc);
CREATE INDEX IF NOT EXISTS jobs_user_energy ON job (hpc_user, energy);

-- Project Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_project_numnodes ON job (project, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_project_numhwthreads ON job (project, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_project_numacc ON job (project, num_acc);
CREATE INDEX IF NOT EXISTS jobs_project_energy ON job (project, energy);

-- JobState Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_jobstate_numnodes ON job (job_state, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numhwthreads ON job (job_state, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_jobstate_numacc ON job (job_state, num_acc);
CREATE INDEX IF NOT EXISTS jobs_jobstate_energy ON job (job_state, energy);

-- Shared Filter Sorting
CREATE INDEX IF NOT EXISTS jobs_shared_numnodes ON job (shared, num_nodes);
CREATE INDEX IF NOT EXISTS jobs_shared_numhwthreads ON job (shared, num_hwthreads);
CREATE INDEX IF NOT EXISTS jobs_shared_numacc ON job (shared, num_acc);
CREATE INDEX IF NOT EXISTS jobs_shared_energy ON job (shared, energy);

-- ArrayJob Filter
CREATE INDEX IF NOT EXISTS jobs_cluster_arrayjobid_starttime ON job (cluster, array_job_id, start_time);

-- Backup Indices For High Variety Columns
CREATE INDEX IF NOT EXISTS jobs_duration ON job (duration);

-- Optimize DB index usage
PRAGMA optimize;


Now delete migrations 12–14 and update `Version` to 11:


Bash

rm /Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/migrations/sqlite3/12_stats-covering-index.up.sql \
   /Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/migrations/sqlite3/12_stats-covering-index.down.sql \
   /Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/migrations/sqlite3/13_status-covering-indexes.up.sql \
   /Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/migrations/sqlite3/13_status-covering-indexes.down.sql \
   /Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/migrations/sqlite3/14_running-partial-indexes.up.sql \
   /Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/migrations/sqlite3/14_running-partial-indexes.down.sql

Delete migration files 12, 13, 14


(Bash completed with no output)


Edit

// Version history:
//   - Version 14: Partial covering indexes for running jobs (tiny B-tree vs millions)
//   - Version 13: Add covering indexes for status/dashboard queries (cluster, job_state, ...)
//   - Version 12: Add covering index for stats queries (cluster, start_time, hpc_user, ...)
//   - Version 11: Optimize job table indexes (reduce from ~78 to 48)
//   - Version 10: Node table
//
// Migration files are embedded at build time from the migrations directory.
const Version uint = 14

// Version history:
//   - Version 11: Optimize job table indexes (reduce from ~78 to 48, add covering/partial indexes)
//   - Version 10: Node table
//
// Migration files are embedded at build time from the migrations directory.
const Version uint = 11


The file /Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/migration.go has been updated successfully.

// Version history:
//   - Version 11: Optimize job table indexes (reduce from ~78 to 48, add covering/partial indexes)
//   - Version 10: Node table
//
// Migration files are embedded at build time from the migrations directory.
const Version uint = 11

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package repository

import (
	"database/sql"
	"embed"
	"fmt"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/golang-migrate/migrate/v4"
	"github.com/golang-migrate/migrate/v4/database/sqlite3"
	"github.com/golang-migrate/migrate/v4/source/iofs"
)

// Version is the current database schema version required by this version of cc-backend.
// When the database schema changes, this version is incremented and a new migration file
// is added to internal/repository/migrations/sqlite3/.
//
// Version history:
//   - Version 14: Partial covering indexes for running jobs (tiny B-tree vs millions)
//   - Version 13: Add covering indexes for status/dashboard queries (cluster, job_state, ...)
//   - Version 12: Add covering index for stats queries (cluster, start_time, hpc_user, ...)
//   - Version 11: Optimize job table indexes (reduce from ~78 to 48)
//   - Version 10: Node table
//
// Migration files are embedded at build time from the migrations directory.
const Version uint = 14

//go:embed migrations/*
var migrationFiles embed.FS

// checkDBVersion verifies that the database schema version matches the expected version.
// This is called automatically during Connect() to ensure schema compatibility.
//
// Returns an error if:
//   - Database version is older than expected (needs migration)
//   - Database version is newer than expected (needs app upgrade)
//   - Database is in a dirty state (failed migration)
//
// A "dirty" database indicates a migration was started but not completed successfully.
// This requires manual intervention to fix the database and force the version.
func checkDBVersion(db *sql.DB) error {
	driver, err := sqlite3.WithInstance(db, &sqlite3.Config{})
	if err != nil {
		return err
	}
	d, err := iofs.New(migrationFiles, "migrations/sqlite3")
	if err != nil {
		return err
	}

	m, err := migrate.NewWithInstance("iofs", d, "sqlite3", driver)
	if err != nil {
		return err
	}

	v, dirty, err := m.Version()
	if err != nil {
		if err == migrate.ErrNilVersion {
			cclog.Warn("Legacy database without version or missing database file!")
		} else {
			return err
		}
	}

	if v < Version {
		return fmt.Errorf("unsupported database version %d, need %d.\nPlease backup your database file and run cc-backend -migrate-db", v, Version)
	} else if v > Version {
		return fmt.Errorf("unsupported database version %d, need %d.\nPlease refer to documentation how to downgrade db with external migrate tool", v, Version)
	}

	if dirty {
		return fmt.Errorf("last migration to version %d has failed, please fix the db manually and force version with -force-db flag", Version)
	}

	return nil
}

// getMigrateInstance creates a new migration instance for the given database file.
// This is used internally by MigrateDB, RevertDB, and ForceDB.
func getMigrateInstance(db string) (m *migrate.Migrate, err error) {
	d, err := iofs.New(migrationFiles, "migrations/sqlite3")
	if err != nil {
		return nil, err
	}

	m, err = migrate.NewWithSourceInstance("iofs", d, fmt.Sprintf("sqlite3://%s?_foreign_keys=on", db))
	if err != nil {
		return nil, err
	}

	return m, nil
}

// MigrateDB applies all pending database migrations to bring the schema up to date.
// This should be run with the -migrate-db flag before starting the application
// after upgrading to a new version that requires schema changes.
//
// Process:
//  1. Checks current database version
//  2. Applies all migrations from current version to target Version
//  3. Updates schema_migrations table to track applied migrations
//
// Important:
//   - Always backup your database before running migrations
//   - Migrations are irreversible without manual intervention
//   - If a migration fails, the database is marked "dirty" and requires manual fix
//
// Usage:
//
//	cc-backend -migrate-db
func MigrateDB(db string) error {
	m, err := getMigrateInstance(db)
	if err != nil {
		return err
	}

	v, dirty, err := m.Version()
	if err != nil {
		if err == migrate.ErrNilVersion {
			cclog.Info("Legacy database without version or missing database file!")
		} else {
			return err
		}
	}

	if v < Version {
		cclog.Infof("unsupported database version %d, need %d.\nPlease backup your database file and run cc-backend -migrate-db", v, Version)
	}

	if dirty {
		return fmt.Errorf("last migration to version %d has failed, please fix the db manually and force version with -force-db flag", Version)
	}

	if err := m.Up(); err != nil {
		if err == migrate.ErrNoChange {
			cclog.Info("DB already up to date!")
		} else {
			return err
		}
	}

	m.Close()
	return nil
}

// RevertDB rolls back the database schema to the previous version (Version - 1).
// This is primarily used for testing or emergency rollback scenarios.
//
// Warning:
//   - This may cause data loss if newer schema added columns/tables
//   - Always backup before reverting
//   - Not all migrations are safely reversible
//
// Usage:
//
//	cc-backend -revert-db
func RevertDB(db string) error {
	m, err := getMigrateInstance(db)
	if err != nil {
		return err
	}

	if err := m.Migrate(Version - 1); err != nil {
		if err == migrate.ErrNoChange {
			cclog.Info("DB already up to date!")
		} else {
			return err
		}
	}

	m.Close()
	return nil
}

// ForceDB forces the database schema version to the current Version without running migrations.
// This is only used to recover from failed migrations that left the database in a "dirty" state.
//
// When to use:
//   - After manually fixing a failed migration
//   - When you've manually applied schema changes and need to update the version marker
//
// Warning:
//   - This does NOT apply any schema changes
//   - Only use after manually verifying the schema is correct
//   - Improper use can cause schema/version mismatch
//
// Usage:
//
//	cc-backend -force-db
func ForceDB(db string) error {
	m, err := getMigrateInstance(db)
	if err != nil {
		return err
	}

	if err := m.Force(int(Version)); err != nil {
		return err
	}

	m.Close()
	return nil
}

-//   - Version 14: Partial covering indexes for running jobs (tiny B-tree vs millions)

-//   - Version 13: Add covering indexes for status/dashboard queries (cluster, job_state, ...)

-//   - Version 12: Add covering index for stats queries (cluster, start_time, hpc_user, ...)

-//   - Version 11: Optimize job table indexes (reduce from ~78 to 48)

+//   - Version 11: Optimize job table indexes (reduce from ~78 to 48, add covering/partial indexes)


Done. Migrations 12, 13, and 14 are deleted. Migration 11 now consolidates all their changes:

- **up**: drops old sorting indexes → creates stats covering indexes → creates status covering indexes (dropping the subsumed 3-col ones) → creates partial indexes for running jobs
- **down**: reverses everything in the correct order (running partial → status covering → stats covering → restore old indexes)
- `Version` updated to `11`