Implement the following plan:

# Fix: SQLite Write Contention Causing OOM Crash

## Context

The server crashes daily around 15:00-16:00 with OOM. Logs show "database is locked" errors and LDAP sync failures (`UNIQUE constraint failed: hpc_user.username`) shortly before the crash.

**Root cause:** A burst of concurrent DB writers (NATS job start/stop, node state updates, LDAP sync, scheduled tasks) overwhelm SQLite's 5-second busy_timeout. Failed operations and blocked goroutines accumulate, consuming memory until OOM.

The cascade: burst of job events → DB contention → `SQLITE_BUSY` after 5s → error paths spawn retries/new goroutines → goroutine accumulation → OOM.

---

## Fix 1: Increase busy_timeout (contention relief)

**File:** `internal/repository/dbConnection.go:73`

Change `_busy_timeout` from `5000` to `60000` (60 seconds). Also make it configurable:

**File:** `internal/repository/config.go` — add `BusyTimeoutMs int` field, default 60000.

**File:** `internal/repository/dbConnection.go:73` — use `repoConfig.BusyTimeoutMs` instead of hardcoded value.

**File:** `pkg/archive/sqliteBackend.go:105` — also change hardcoded `busy_timeout=5000` to `busy_timeout=60000`.

With 60s timeout, SQLite retries internally using its backoff mechanism rather than failing fast. This prevents the "database is locked" error cascade.

---

## Fix 2: Batch Node State Updates in a Transaction

Currently each node state update is a separate SELECT + INSERT (2 DB round-trips per node). A message with 1000 nodes = 2000+ individual DB operations, each acquiring/releasing the write lock.

**File:** `internal/repository/node.go` — add `BatchUpdateNodeStates` method:
- Accept a slice of `(hostname, cluster, nodeState)` tuples
- Begin transaction
- For each node: lookup ID (or insert node), then insert state — all within the same transaction
- Commit once

**File:** `internal/api/nats.go:348-364` (`processNodestateEvent`) — build slice of updates, call `BatchUpdateNodeStates` instead of per-node `UpdateNodeState`.

**File:** `internal/api/node.go:119-142` (`updateNodeStates`) — same change for the REST path. Note: the REST handler also does health checks per node before DB inserts, so batch only the DB part (the loop at lines 119-142).

Keep the existing `UpdateNodeState` method for single-node use cases.

---

## Fix 3: Fix LDAP Sync Race Condition

The LDAP sync fails when a user is concurrently created by a login. The error is benign but causes `Sync()` to return early (line 215: `return err`), abandoning remaining users.

**File:** `internal/repository/user.go` — add `AddUserIfNotExists` method using `INSERT OR IGNORE`:
```go
func (r *UserRepository) AddUserIfNotExists(user *schema.User) error {
    rolesJson, _ := json.Marshal(user.Roles)
    projectsJson, _ := json.Marshal(user.Projects)
    cols := "username, name, roles, projects, ldap"
    _, err := r.DB.Exec(
        `INSERT OR IGNORE INTO hpc_user (` + cols + `) VALUES (?, ?, ?, ?, ?)`,
        user.Username, user.Name, string(rolesJson), string(projectsJson), int(user.AuthSource))
    return err
}
```

**File:** `internal/auth/ldap.go:213` — change `ur.AddUser(user)` to `ur.AddUserIfNotExists(user)`.

Note: `AddUserIfNotExists` intentionally skips the deprecated default metrics config insertion that `AddUser` does — LDAP-synced users don't need it, and it adds unnecessary DB writes.

---

## Fix 4: Add Semaphore for NATS Handlers

Without rate limiting, blocked NATS handlers cause unbounded goroutine accumulation.

**File:** `internal/api/nats.go` — add semaphores to `NatsAPI` struct:
```go
type NatsAPI struct {
    JobRepository   *repository.JobRepository
    RepositoryMutex sync.Mutex
    jobSem          chan struct{}
    nodeSem         chan struct{}
}
```

Initialize in `NewNatsAPI`: `jobSem: make(chan struct{}, 8)`, `nodeSem: make(chan struct{}, 2)`.

Add acquire/release in `handleJobEvent` and `handleNodeState`:
```go
api.jobSem <- struct{}{}
defer func() { <-api.jobSem }()
```

This caps concurrent NATS processing at 8 job events + 2 node state updates, preventing goroutine pile-up.

---

## Fix 5: Wrap SyncJobs in a Transaction

**File:** `internal/repository/jobCreate.go:72-126` (`SyncJobs`)

Currently does SELECT + INSERT...SELECT + DELETE as 3 separate operations. Wrap in a DB transaction:
- `tx, err := r.DB.Beginx()`
- Run all three operations on `tx`
- `tx.Commit()`
- Do the read-only ID resolution outside the transaction

This reduces lock acquisitions from 3 to 1 and ensures atomicity.

---

## Implementation Order

1. **Fix 1** (busy_timeout) — quick config change, biggest immediate impact
2. **Fix 3** (LDAP sync) — eliminates error spam, trivial change
3. **Fix 2** (batch node states) — biggest contention reduction
4. **Fix 5** (SyncJobs transaction) — safety + minor contention reduction
5. **Fix 4** (NATS semaphore) — defense in depth against goroutine accumulation

---

## Verification

1. After Fix 1: Check logs for "database is locked" — should be gone or very rare.
2. After Fix 2: check `Timer updateNodeStates, SQLite Inserts` log line — should be much faster.
3. After Fix 3: check logs for "UNIQUE constraint failed" — should be gone.
4. Run `go test ./internal/repository/... ./internal/api/... ./internal/auth/...` after all changes.
5. Load test: simulate burst of 100+ concurrent job start/stop events + node state update with 500 nodes + LDAP sync running simultaneously. Monitor goroutine count and memory.

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-ClusterCockpit-cc-backend/d36fdafb-27dd-4101-b89a-e2bec3f83825.jsonl


I'll start by reading all the files that need to be modified.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package repository
     7→
     8→import (
     9→	"database/sql"
    10→	"fmt"
    11→	"net/url"
    12→	"sync"
    13→	"time"
    14→
    15→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    16→	"github.com/jmoiron/sqlx"
    17→	"github.com/mattn/go-sqlite3"
    18→	"github.com/qustavo/sqlhooks/v2"
    19→)
    20→
    21→var (
    22→	dbConnOnce     sync.Once
    23→	dbConnInstance *DBConnection
    24→)
    25→
    26→type DBConnection struct {
    27→	DB     *sqlx.DB
    28→	Driver string
    29→}
    30→
    31→type DatabaseOptions struct {
    32→	URL                   string
    33→	MaxOpenConnections    int
    34→	MaxIdleConnections    int
    35→	ConnectionMaxLifetime time.Duration
    36→	ConnectionMaxIdleTime time.Duration
    37→}
    38→
    39→func setupSqlite(db *sql.DB, cfg *RepositoryConfig) error {
    40→	pragmas := []string{
    41→		"temp_store = memory",
    42→		fmt.Sprintf("soft_heap_limit = %d", int64(cfg.DbSoftHeapLimitMB)*1024*1024),
    43→	}
    44→
    45→	for _, pragma := range pragmas {
    46→		_, err := db.Exec("PRAGMA " + pragma)
    47→		if err != nil {
    48→			return err
    49→		}
    50→	}
    51→
    52→	return nil
    53→}
    54→
    55→func Connect(db string) {
    56→	var err error
    57→	var dbHandle *sqlx.DB
    58→
    59→	dbConnOnce.Do(func() {
    60→		opts := DatabaseOptions{
    61→			URL:                   db,
    62→			MaxOpenConnections:    repoConfig.MaxOpenConnections,
    63→			MaxIdleConnections:    repoConfig.MaxIdleConnections,
    64→			ConnectionMaxLifetime: repoConfig.ConnectionMaxLifetime,
    65→			ConnectionMaxIdleTime: repoConfig.ConnectionMaxIdleTime,
    66→		}
    67→
    68→		// TODO: Have separate DB handles for Writes and Reads
    69→		// Optimize SQLite connection: https://kerkour.com/sqlite-for-servers
    70→		connectionURLParams := make(url.Values)
    71→		connectionURLParams.Add("_txlock", "immediate")
    72→		connectionURLParams.Add("_journal_mode", "WAL")
    73→		connectionURLParams.Add("_busy_timeout", "5000")
    74→		connectionURLParams.Add("_synchronous", "NORMAL")
    75→		cacheSizeKiB := repoConfig.DbCacheSizeMB * 1024 // Convert MB to KiB
    76→		connectionURLParams.Add("_cache_size", fmt.Sprintf("-%d", cacheSizeKiB))
    77→		connectionURLParams.Add("_foreign_keys", "true")
    78→		opts.URL = fmt.Sprintf("file:%s?%s", opts.URL, connectionURLParams.Encode())
    79→
    80→		if cclog.Loglevel() == "debug" {
    81→			sql.Register("sqlite3WithHooks", sqlhooks.Wrap(&sqlite3.SQLiteDriver{}, &Hooks{}))
    82→			dbHandle, err = sqlx.Open("sqlite3WithHooks", opts.URL)
    83→		} else {
    84→			dbHandle, err = sqlx.Open("sqlite3", opts.URL)
    85→		}
    86→
    87→		if err != nil {
    88→			cclog.Abortf("DB Connection: Could not connect to SQLite database with sqlx.Open().\nError: %s\n", err.Error())
    89→		}
    90→
    91→		err = setupSqlite(dbHandle.DB, repoConfig)
    92→		if err != nil {
    93→			cclog.Abortf("Failed sqlite db setup.\nError: %s\n", err.Error())
    94→		}
    95→
    96→		cclog.Infof("SQLite config: cache_size=%dMB/conn, soft_heap_limit=%dMB, max_conns=%d",
    97→			repoConfig.DbCacheSizeMB, repoConfig.DbSoftHeapLimitMB, repoConfig.MaxOpenConnections)
    98→
    99→		dbHandle.SetMaxOpenConns(opts.MaxOpenConnections)
   100→		dbHandle.SetMaxIdleConns(opts.MaxIdleConnections)
   101→		dbHandle.SetConnMaxLifetime(opts.ConnectionMaxLifetime)
   102→		dbHandle.SetConnMaxIdleTime(opts.ConnectionMaxIdleTime)
   103→
   104→		dbConnInstance = &DBConnection{DB: dbHandle}
   105→		err = checkDBVersion(dbHandle.DB)
   106→		if err != nil {
   107→			cclog.Abortf("DB Connection: Failed DB version check.\nError: %s\n", err.Error())
   108→		}
   109→	})
   110→}
   111→
   112→func GetConnection() *DBConnection {
   113→	if dbConnInstance == nil {
   114→		cclog.Fatalf("Database connection not initialized!")
   115→	}
   116→
   117→	return dbConnInstance
   118→}
   119→
   120→// ResetConnection closes the current database connection and resets the connection state.
   121→// This function is intended for testing purposes only to allow test isolation.
   122→func ResetConnection() error {
   123→	if dbConnInstance != nil && dbConnInstance.DB != nil {
   124→		if err := dbConnInstance.DB.Close(); err != nil {
   125→			return fmt.Errorf("failed to close database connection: %w", err)
   126→		}
   127→	}
   128→
   129→	dbConnInstance = nil
   130→	dbConnOnce = sync.Once{}
   131→	jobRepoInstance = nil
   132→	jobRepoOnce = sync.Once{}
   133→	nodeRepoInstance = nil
   134→	nodeRepoOnce = sync.Once{}
   135→	userRepoInstance = nil
   136→	userRepoOnce = sync.Once{}
   137→	userCfgRepoInstance = nil
   138→	userCfgRepoOnce = sync.Once{}
   139→
   140→	return nil
   141→}
   142→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package repository

import (
	"database/sql"
	"fmt"
	"net/url"
	"sync"
	"time"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/jmoiron/sqlx"
	"github.com/mattn/go-sqlite3"
	"github.com/qustavo/sqlhooks/v2"
)

var (
	dbConnOnce     sync.Once
	dbConnInstance *DBConnection
)

type DBConnection struct {
	DB     *sqlx.DB
	Driver string
}

type DatabaseOptions struct {
	URL                   string
	MaxOpenConnections    int
	MaxIdleConnections    int
	ConnectionMaxLifetime time.Duration
	ConnectionMaxIdleTime time.Duration
}

func setupSqlite(db *sql.DB, cfg *RepositoryConfig) error {
	pragmas := []string{
		"temp_store = memory",
		fmt.Sprintf("soft_heap_limit = %d", int64(cfg.DbSoftHeapLimitMB)*1024*1024),
	}

	for _, pragma := range pragmas {
		_, err := db.Exec("PRAGMA " + pragma)
		if err != nil {
			return err
		}
	}

	return nil
}

func Connect(db string) {
	var err error
	var dbHandle *sqlx.DB

	dbConnOnce.Do(func() {
		opts := DatabaseOptions{
			URL:                   db,
			MaxOpenConnections:    repoConfig.MaxOpenConnections,
			MaxIdleConnections:    repoConfig.MaxIdleConnections,
			ConnectionMaxLifetime: repoConfig.ConnectionMaxLifetime,
			ConnectionMaxIdleTime: repoConfig.ConnectionMaxIdleTime,
		}

		// TODO: Have separate DB handles for Writes and Reads
		// Optimize SQLite connection: https://kerkour.com/sqlite-for-servers
		connectionURLParams := make(url.Values)
		connectionURLParams.Add("_txlock", "immediate")
		connectionURLParams.Add("_journal_mode", "WAL")
		connectionURLParams.Add("_busy_timeout", "5000")
		connectionURLParams.Add("_synchronous", "NORMAL")
		cacheSizeKiB := repoConfig.DbCacheSizeMB * 1024 // Convert MB to KiB
		connectionURLParams.Add("_cache_size", fmt.Sprintf("-%d", cacheSizeKiB))
		connectionURLParams.Add("_foreign_keys", "true")
		opts.URL = fmt.Sprintf("file:%s?%s", opts.URL, connectionURLParams.Encode())

		if cclog.Loglevel() == "debug" {
			sql.Register("sqlite3WithHooks", sqlhooks.Wrap(&sqlite3.SQLiteDriver{}, &Hooks{}))
			dbHandle, err = sqlx.Open("sqlite3WithHooks", opts.URL)
		} else {
			dbHandle, err = sqlx.Open("sqlite3", opts.URL)
		}

		if err != nil {
			cclog.Abortf("DB Connection: Could not connect to SQLite database with sqlx.Open().\nError: %s\n", err.Error())
		}

		err = setupSqlite(dbHandle.DB, repoConfig)
		if err != nil {
			cclog.Abortf("Failed sqlite db setup.\nError: %s\n", err.Error())
		}

		cclog.Infof("SQLite config: cache_size=%dMB/conn, soft_heap_limit=%dMB, max_conns=%d",
			repoConfig.DbCacheSizeMB, repoConfig.DbSoftHeapLimitMB, repoConfig.MaxOpenConnections)

		dbHandle.SetMaxOpenConns(opts.MaxOpenConnections)
		dbHandle.SetMaxIdleConns(opts.MaxIdleConnections)
		dbHandle.SetConnMaxLifetime(opts.ConnectionMaxLifetime)
		dbHandle.SetConnMaxIdleTime(opts.ConnectionMaxIdleTime)

		dbConnInstance = &DBConnection{DB: dbHandle}
		err = checkDBVersion(dbHandle.DB)
		if err != nil {
			cclog.Abortf("DB Connection: Failed DB version check.\nError: %s\n", err.Error())
		}
	})
}

func GetConnection() *DBConnection {
	if dbConnInstance == nil {
		cclog.Fatalf("Database connection not initialized!")
	}

	return dbConnInstance
}

// ResetConnection closes the current database connection and resets the connection state.
// This function is intended for testing purposes only to allow test isolation.
func ResetConnection() error {
	if dbConnInstance != nil && dbConnInstance.DB != nil {
		if err := dbConnInstance.DB.Close(); err != nil {
			return fmt.Errorf("failed to close database connection: %w", err)
		}
	}

	dbConnInstance = nil
	dbConnOnce = sync.Once{}
	jobRepoInstance = nil
	jobRepoOnce = sync.Once{}
	nodeRepoInstance = nil
	nodeRepoOnce = sync.Once{}
	userRepoInstance = nil
	userRepoOnce = sync.Once{}
	userCfgRepoInstance = nil
	userCfgRepoOnce = sync.Once{}

	return nil
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package repository
     7→
     8→import "time"
     9→
    10→// RepositoryConfig holds configuration for repository operations.
    11→// All fields have sensible defaults, so this configuration is optional.
    12→type RepositoryConfig struct {
    13→	// CacheSize is the LRU cache size in bytes for job metadata and energy footprints.
    14→	// Default: 1MB (1024 * 1024 bytes)
    15→	CacheSize int
    16→
    17→	// MaxOpenConnections is the maximum number of open database connections.
    18→	// Default: 4
    19→	MaxOpenConnections int
    20→
    21→	// MaxIdleConnections is the maximum number of idle database connections.
    22→	// Default: 4
    23→	MaxIdleConnections int
    24→
    25→	// ConnectionMaxLifetime is the maximum amount of time a connection may be reused.
    26→	// Default: 1 hour
    27→	ConnectionMaxLifetime time.Duration
    28→
    29→	// ConnectionMaxIdleTime is the maximum amount of time a connection may be idle.
    30→	// Default: 10 minutes
    31→	ConnectionMaxIdleTime time.Duration
    32→
    33→	// MinRunningJobDuration is the minimum duration in seconds for a job to be
    34→	// considered in "running jobs" queries. This filters out very short jobs.
    35→	// Default: 600 seconds (10 minutes)
    36→	MinRunningJobDuration int
    37→
    38→	// DbCacheSizeMB is the SQLite page cache size per connection in MB.
    39→	// Uses negative PRAGMA cache_size notation (KiB). With MaxOpenConnections=4
    40→	// and DbCacheSizeMB=2048, total page cache is up to 8GB.
    41→	// Default: 2048 (2GB)
    42→	DbCacheSizeMB int
    43→
    44→	// DbSoftHeapLimitMB is the process-wide SQLite soft heap limit in MB.
    45→	// SQLite will try to release cache pages to stay under this limit.
    46→	// It's a soft limit — queries won't fail, but cache eviction becomes more aggressive.
    47→	// Default: 16384 (16GB)
    48→	DbSoftHeapLimitMB int
    49→}
    50→
    51→// DefaultConfig returns the default repository configuration.
    52→// These values are optimized for typical deployments.
    53→func DefaultConfig() *RepositoryConfig {
    54→	return &RepositoryConfig{
    55→		CacheSize:             1 * 1024 * 1024, // 1MB
    56→		MaxOpenConnections:    4,
    57→		MaxIdleConnections:    4,
    58→		ConnectionMaxLifetime: time.Hour,
    59→		ConnectionMaxIdleTime: 10 * time.Minute,
    60→		MinRunningJobDuration: 600, // 10 minutes
    61→		DbCacheSizeMB:        2048,  // 2GB per connection
    62→		DbSoftHeapLimitMB:    16384, // 16GB process-wide
    63→	}
    64→}
    65→
    66→// repoConfig is the package-level configuration instance.
    67→// It is initialized with defaults and can be overridden via SetConfig.
    68→var repoConfig *RepositoryConfig = DefaultConfig()
    69→
    70→// SetConfig sets the repository configuration.
    71→// This must be called before any repository initialization (Connect, GetJobRepository, etc.).
    72→// If not called, default values from DefaultConfig() are used.
    73→func SetConfig(cfg *RepositoryConfig) {
    74→	if cfg != nil {
    75→		repoConfig = cfg
    76→	}
    77→}
    78→
    79→// GetConfig returns the current repository configuration.
    80→func GetConfig() *RepositoryConfig {
    81→	return repoConfig
    82→}
    83→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package repository

import "time"

// RepositoryConfig holds configuration for repository operations.
// All fields have sensible defaults, so this configuration is optional.
type RepositoryConfig struct {
	// CacheSize is the LRU cache size in bytes for job metadata and energy footprints.
	// Default: 1MB (1024 * 1024 bytes)
	CacheSize int

	// MaxOpenConnections is the maximum number of open database connections.
	// Default: 4
	MaxOpenConnections int

	// MaxIdleConnections is the maximum number of idle database connections.
	// Default: 4
	MaxIdleConnections int

	// ConnectionMaxLifetime is the maximum amount of time a connection may be reused.
	// Default: 1 hour
	ConnectionMaxLifetime time.Duration

	// ConnectionMaxIdleTime is the maximum amount of time a connection may be idle.
	// Default: 10 minutes
	ConnectionMaxIdleTime time.Duration

	// MinRunningJobDuration is the minimum duration in seconds for a job to be
	// considered in "running jobs" queries. This filters out very short jobs.
	// Default: 600 seconds (10 minutes)
	MinRunningJobDuration int

	// DbCacheSizeMB is the SQLite page cache size per connection in MB.
	// Uses negative PRAGMA cache_size notation (KiB). With MaxOpenConnections=4
	// and DbCacheSizeMB=2048, total page cache is up to 8GB.
	// Default: 2048 (2GB)
	DbCacheSizeMB int

	// DbSoftHeapLimitMB is the process-wide SQLite soft heap limit in MB.
	// SQLite will try to release cache pages to stay under this limit.
	// It's a soft limit — queries won't fail, but cache eviction becomes more aggressive.
	// Default: 16384 (16GB)
	DbSoftHeapLimitMB int
}

// DefaultConfig returns the default repository configuration.
// These values are optimized for typical deployments.
func DefaultConfig() *RepositoryConfig {
	return &RepositoryConfig{
		CacheSize:             1 * 1024 * 1024, // 1MB
		MaxOpenConnections:    4,
		MaxIdleConnections:    4,
		ConnectionMaxLifetime: time.Hour,
		ConnectionMaxIdleTime: 10 * time.Minute,
		MinRunningJobDuration: 600, // 10 minutes
		DbCacheSizeMB:        2048,  // 2GB per connection
		DbSoftHeapLimitMB:    16384, // 16GB process-wide
	}
}

// repoConfig is the package-level configuration instance.
// It is initialized with defaults and can be overridden via SetConfig.
var repoConfig *RepositoryConfig = DefaultConfig()

// SetConfig sets the repository configuration.
// This must be called before any repository initialization (Connect, GetJobRepository, etc.).
// If not called, default values from DefaultConfig() are used.
func SetConfig(cfg *RepositoryConfig) {
	if cfg != nil {
		repoConfig = cfg
	}
}

// GetConfig returns the current repository configuration.
func GetConfig() *RepositoryConfig {
	return repoConfig
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package archive
     7→
     8→import (
     9→	"bytes"
    10→	"compress/gzip"
    11→	"database/sql"
    12→	"encoding/json"
    13→	"fmt"
    14→	"io"
    15→	"math"
    16→	"os"
    17→	"slices"
    18→	"strconv"
    19→	"sync"
    20→	"text/tabwriter"
    21→	"time"
    22→
    23→	"github.com/ClusterCockpit/cc-backend/internal/config"
    24→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    25→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    26→	"github.com/ClusterCockpit/cc-lib/v2/util"
    27→	_ "github.com/mattn/go-sqlite3"
    28→)
    29→
    30→// SqliteArchiveConfig holds the configuration for the SQLite archive backend.
    31→type SqliteArchiveConfig struct {
    32→	DBPath string `json:"db-path"` // Path to SQLite database file
    33→}
    34→
    35→// SqliteArchive implements ArchiveBackend using a SQLite database with BLOB storage.
    36→// Job metadata and data are stored as JSON BLOBs with indexes for fast queries.
    37→//
    38→// Uses WAL (Write-Ahead Logging) mode for better concurrency and a 64MB cache.
    39→type SqliteArchive struct {
    40→	db       *sql.DB  // SQLite database connection
    41→	clusters []string // List of discovered cluster names
    42→}
    43→
    44→// sqliteSchema defines the database schema for SQLite archive backend.
    45→// Jobs table: stores job metadata and data as BLOBs with compression flag
    46→// Clusters table: stores cluster configurations
    47→// Metadata table: stores version and other key-value pairs
    48→const sqliteSchema = `
    49→CREATE TABLE IF NOT EXISTS jobs (
    50→    id INTEGER PRIMARY KEY AUTOINCREMENT,
    51→    job_id INTEGER NOT NULL,
    52→    cluster TEXT NOT NULL,
    53→    start_time INTEGER NOT NULL,
    54→    meta_json BLOB NOT NULL,
    55→    data_json BLOB,
    56→    data_compressed BOOLEAN DEFAULT 0,
    57→    created_at INTEGER NOT NULL,
    58→    updated_at INTEGER NOT NULL,
    59→    UNIQUE(job_id, cluster, start_time)
    60→);
    61→
    62→CREATE INDEX IF NOT EXISTS idx_jobs_cluster ON jobs(cluster);
    63→CREATE INDEX IF NOT EXISTS idx_jobs_start_time ON jobs(start_time);
    64→CREATE INDEX IF NOT EXISTS idx_jobs_order ON jobs(cluster, start_time);
    65→CREATE INDEX IF NOT EXISTS idx_jobs_lookup ON jobs(cluster, job_id, start_time);
    66→
    67→CREATE TABLE IF NOT EXISTS clusters (
    68→    name TEXT PRIMARY KEY,
    69→    config_json BLOB NOT NULL,
    70→    updated_at INTEGER NOT NULL
    71→);
    72→
    73→CREATE TABLE IF NOT EXISTS metadata (
    74→    key TEXT PRIMARY KEY,
    75→    value TEXT NOT NULL
    76→);
    77→`
    78→
    79→func (sa *SqliteArchive) Init(rawConfig json.RawMessage) (uint64, error) {
    80→	var cfg SqliteArchiveConfig
    81→	if err := json.Unmarshal(rawConfig, &cfg); err != nil {
    82→		cclog.Warnf("SqliteArchive Init() > Unmarshal error: %#v", err)
    83→		return 0, err
    84→	}
    85→
    86→	if cfg.DBPath == "" {
    87→		err := fmt.Errorf("SqliteArchive Init(): empty database path")
    88→		cclog.Errorf("SqliteArchive Init() > config error: %v", err)
    89→		return 0, err
    90→	}
    91→
    92→	// Open SQLite database
    93→	db, err := sql.Open("sqlite3", cfg.DBPath)
    94→	if err != nil {
    95→		cclog.Errorf("SqliteArchive Init() > failed to open database: %v", err)
    96→		return 0, err
    97→	}
    98→	sa.db = db
    99→
   100→	// Set pragmas for better performance
   101→	pragmas := []string{
   102→		"PRAGMA journal_mode=WAL",
   103→		"PRAGMA synchronous=NORMAL",
   104→		"PRAGMA cache_size=-64000", // 64MB cache
   105→		"PRAGMA busy_timeout=5000",
   106→	}
   107→	for _, pragma := range pragmas {
   108→		if _, err := sa.db.Exec(pragma); err != nil {
   109→			cclog.Warnf("SqliteArchive Init() > pragma failed: %v", err)
   110→		}
   111→	}
   112→
   113→	// Create schema
   114→	if _, err := sa.db.Exec(sqliteSchema); err != nil {
   115→		cclog.Errorf("SqliteArchive Init() > schema creation failed: %v", err)
   116→		return 0, err
   117→	}
   118→
   119→	// Check/set version
   120→	var versionStr string
   121→	err = sa.db.QueryRow("SELECT value FROM metadata WHERE key = 'version'").Scan(&versionStr)
   122→	if err == sql.ErrNoRows {
   123→		// First time initialization, set version
   124→		_, err = sa.db.Exec("INSERT INTO metadata (key, value) VALUES ('version', ?)", fmt.Sprintf("%d", Version))
   125→		if err != nil {
   126→			cclog.Errorf("SqliteArchive Init() > failed to set version: %v", err)
   127→			return 0, err
   128→		}
   129→		versionStr = fmt.Sprintf("%d", Version)
   130→	} else if err != nil {
   131→		cclog.Errorf("SqliteArchive Init() > failed to read version: %v", err)
   132→		return 0, err
   133→	}
   134→
   135→	version, err := strconv.ParseUint(versionStr, 10, 64)
   136→	if err != nil {
   137→		cclog.Errorf("SqliteArchive Init() > version parse error: %v", err)
   138→		return 0, err
   139→	}
   140→
   141→	if version != Version {
   142→		return version, fmt.Errorf("unsupported version %d, need %d", version, Version)
   143→	}
   144→
   145→	// Discover clusters
   146→	sa.clusters = []string{}
   147→	rows, err := sa.db.Query("SELECT DISTINCT cluster FROM jobs ORDER BY cluster")
   148→	if err != nil {
   149→		cclog.Errorf("SqliteArchive Init() > failed to query clusters: %v", err)
   150→		return 0, err
   151→	}
   152→	defer rows.Close()
   153→
   154→	for rows.Next() {
   155→		var cluster string
   156→		if err := rows.Scan(&cluster); err != nil {
   157→			cclog.Errorf("SqliteArchive Init() > failed to scan cluster: %v", err)
   158→			continue
   159→		}
   160→		sa.clusters = append(sa.clusters, cluster)
   161→	}
   162→
   163→	cclog.Infof("SqliteArchive initialized with database '%s', found %d clusters", cfg.DBPath, len(sa.clusters))
   164→	return version, nil
   165→}
   166→
   167→func (sa *SqliteArchive) Info() {
   168→	fmt.Printf("SQLite Job archive database\n")
   169→
   170→	ci := make(map[string]*clusterInfo)
   171→
   172→	rows, err := sa.db.Query(`
   173→		SELECT cluster, COUNT(*), MIN(start_time), MAX(start_time),
   174→		       SUM(LENGTH(meta_json) + COALESCE(LENGTH(data_json), 0))
   175→		FROM jobs
   176→		GROUP BY cluster
   177→	`)
   178→	if err != nil {
   179→		cclog.Fatalf("SqliteArchive Info() > query failed: %s", err.Error())
   180→	}
   181→	defer rows.Close()
   182→
   183→	for rows.Next() {
   184→		var cluster string
   185→		var numJobs int
   186→		var dateFirst, dateLast int64
   187→		var diskSize int64
   188→
   189→		if err := rows.Scan(&cluster, &numJobs, &dateFirst, &dateLast, &diskSize); err != nil {
   190→			cclog.Errorf("SqliteArchive Info() > scan failed: %v", err)
   191→			continue
   192→		}
   193→
   194→		ci[cluster] = &clusterInfo{
   195→			numJobs:   numJobs,
   196→			dateFirst: dateFirst,
   197→			dateLast:  dateLast,
   198→			diskSize:  float64(diskSize) / (1024 * 1024), // Convert to MB
   199→		}
   200→	}
   201→
   202→	cit := clusterInfo{dateFirst: time.Now().Unix()}
   203→	w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', tabwriter.Debug)
   204→	fmt.Fprintln(w, "cluster\t#jobs\tfrom\tto\tsize (MB)")
   205→	for cluster, clusterInfo := range ci {
   206→		fmt.Fprintf(w, "%s\t%d\t%s\t%s\t%.2f\n", cluster,
   207→			clusterInfo.numJobs,
   208→			time.Unix(clusterInfo.dateFirst, 0),
   209→			time.Unix(clusterInfo.dateLast, 0),
   210→			clusterInfo.diskSize)
   211→
   212→		cit.numJobs += clusterInfo.numJobs
   213→		cit.dateFirst = util.Min(cit.dateFirst, clusterInfo.dateFirst)
   214→		cit.dateLast = util.Max(cit.dateLast, clusterInfo.dateLast)
   215→		cit.diskSize += clusterInfo.diskSize
   216→	}
   217→
   218→	fmt.Fprintf(w, "TOTAL\t%d\t%s\t%s\t%.2f\n",
   219→		cit.numJobs, time.Unix(cit.dateFirst, 0), time.Unix(cit.dateLast, 0), cit.diskSize)
   220→	w.Flush()
   221→}
   222→
   223→func (sa *SqliteArchive) Exists(job *schema.Job) bool {
   224→	var count int
   225→	err := sa.db.QueryRow("SELECT COUNT(*) FROM jobs WHERE job_id = ? AND cluster = ? AND start_time = ?",
   226→		job.JobID, job.Cluster, job.StartTime).Scan(&count)
   227→	return err == nil && count > 0
   228→}
   229→
   230→func (sa *SqliteArchive) LoadJobMeta(job *schema.Job) (*schema.Job, error) {
   231→	var metaBlob []byte
   232→	err := sa.db.QueryRow("SELECT meta_json FROM jobs WHERE job_id = ? AND cluster = ? AND start_time = ?",
   233→		job.JobID, job.Cluster, job.StartTime).Scan(&metaBlob)
   234→	if err != nil {
   235→		cclog.Errorf("SqliteArchive LoadJobMeta() > query error: %v", err)
   236→		return nil, err
   237→	}
   238→
   239→	if config.Keys.Validate {
   240→		if err := schema.Validate(schema.Meta, bytes.NewReader(metaBlob)); err != nil {
   241→			return nil, fmt.Errorf("validate job meta: %v", err)
   242→		}
   243→	}
   244→
   245→	return DecodeJobMeta(bytes.NewReader(metaBlob))
   246→}
   247→
   248→func (sa *SqliteArchive) LoadJobData(job *schema.Job) (schema.JobData, error) {
   249→	var dataBlob []byte
   250→	var compressed bool
   251→	err := sa.db.QueryRow("SELECT data_json, data_compressed FROM jobs WHERE job_id = ? AND cluster = ? AND start_time = ?",
   252→		job.JobID, job.Cluster, job.StartTime).Scan(&dataBlob, &compressed)
   253→	if err != nil {
   254→		cclog.Errorf("SqliteArchive LoadJobData() > query error: %v", err)
   255→		return nil, err
   256→	}
   257→	key := fmt.Sprintf("%s:%d:%d", job.Cluster, job.JobID, job.StartTime)
   258→
   259→	var reader io.Reader = bytes.NewReader(dataBlob)
   260→	if compressed {
   261→		gzipReader, err := gzip.NewReader(reader)
   262→		if err != nil {
   263→			cclog.Errorf("SqliteArchive LoadJobData() > gzip error: %v", err)
   264→			return nil, err
   265→		}
   266→		defer gzipReader.Close()
   267→		reader = gzipReader
   268→	}
   269→
   270→	if config.Keys.Validate {
   271→		data, _ := io.ReadAll(reader)
   272→		if err := schema.Validate(schema.Data, bytes.NewReader(data)); err != nil {
   273→			return schema.JobData{}, fmt.Errorf("validate job data: %v", err)
   274→		}
   275→		return DecodeJobData(bytes.NewReader(data), key)
   276→	}
   277→
   278→	return DecodeJobData(reader, key)
   279→}
   280→
   281→func (sa *SqliteArchive) LoadJobStats(job *schema.Job) (schema.ScopedJobStats, error) {
   282→	var dataBlob []byte
   283→	var compressed bool
   284→	err := sa.db.QueryRow("SELECT data_json, data_compressed FROM jobs WHERE job_id = ? AND cluster = ? AND start_time = ?",
   285→		job.JobID, job.Cluster, job.StartTime).Scan(&dataBlob, &compressed)
   286→	if err != nil {
   287→		cclog.Errorf("SqliteArchive LoadJobStats() > query error: %v", err)
   288→		return nil, err
   289→	}
   290→	key := fmt.Sprintf("%s:%d:%d", job.Cluster, job.JobID, job.StartTime)
   291→
   292→	var reader io.Reader = bytes.NewReader(dataBlob)
   293→	if compressed {
   294→		gzipReader, err := gzip.NewReader(reader)
   295→		if err != nil {
   296→			cclog.Errorf("SqliteArchive LoadJobStats() > gzip error: %v", err)
   297→			return nil, err
   298→		}
   299→		defer gzipReader.Close()
   300→		reader = gzipReader
   301→	}
   302→
   303→	if config.Keys.Validate {
   304→		data, _ := io.ReadAll(reader)
   305→		if err := schema.Validate(schema.Data, bytes.NewReader(data)); err != nil {
   306→			return nil, fmt.Errorf("validate job data: %v", err)
   307→		}
   308→		return DecodeJobStats(bytes.NewReader(data), key)
   309→	}
   310→
   311→	return DecodeJobStats(reader, key)
   312→}
   313→
   314→func (sa *SqliteArchive) LoadClusterCfg(name string) (*schema.Cluster, error) {
   315→	var configBlob []byte
   316→	err := sa.db.QueryRow("SELECT config_json FROM clusters WHERE name = ?", name).Scan(&configBlob)
   317→	if err != nil {
   318→		cclog.Errorf("SqliteArchive LoadClusterCfg() > query error: %v", err)
   319→		return nil, err
   320→	}
   321→
   322→	if config.Keys.Validate {
   323→		if err := schema.Validate(schema.ClusterCfg, bytes.NewReader(configBlob)); err != nil {
   324→			cclog.Warnf("Validate cluster config: %v\n", err)
   325→			return &schema.Cluster{}, fmt.Errorf("validate cluster config: %v", err)
   326→		}
   327→	}
   328→
   329→	return DecodeCluster(bytes.NewReader(configBlob))
   330→}
   331→
   332→func (sa *SqliteArchive) StoreJobMeta(job *schema.Job) error {
   333→	var metaBuf bytes.Buffer
   334→	if err := EncodeJobMeta(&metaBuf, job); err != nil {
   335→		cclog.Error("SqliteArchive StoreJobMeta() > encoding error")
   336→		return err
   337→	}
   338→
   339→	now := time.Now().Unix()
   340→	_, err := sa.db.Exec(`
   341→		INSERT INTO jobs (job_id, cluster, start_time, meta_json, created_at, updated_at)
   342→		VALUES (?, ?, ?, ?, ?, ?)
   343→		ON CONFLICT(job_id, cluster, start_time) DO UPDATE SET
   344→			meta_json = excluded.meta_json,
   345→			updated_at = excluded.updated_at
   346→	`, job.JobID, job.Cluster, job.StartTime, metaBuf.Bytes(), now, now)
   347→	if err != nil {
   348→		cclog.Errorf("SqliteArchive StoreJobMeta() > insert error: %v", err)
   349→		return err
   350→	}
   351→
   352→	return nil
   353→}
   354→
   355→func (sa *SqliteArchive) ImportJob(jobMeta *schema.Job, jobData *schema.JobData) error {
   356→	var metaBuf, dataBuf bytes.Buffer
   357→	if err := EncodeJobMeta(&metaBuf, jobMeta); err != nil {
   358→		cclog.Error("SqliteArchive ImportJob() > encoding meta error")
   359→		return err
   360→	}
   361→	if err := EncodeJobData(&dataBuf, jobData); err != nil {
   362→		cclog.Error("SqliteArchive ImportJob() > encoding data error")
   363→		return err
   364→	}
   365→
   366→	var dataBytes []byte
   367→	var compressed bool
   368→
   369→	if dataBuf.Len() > 2000 {
   370→		var compressedBuf bytes.Buffer
   371→		gzipWriter := gzip.NewWriter(&compressedBuf)
   372→		if _, err := gzipWriter.Write(dataBuf.Bytes()); err != nil {
   373→			cclog.Errorf("SqliteArchive ImportJob() > gzip write error: %v", err)
   374→			return err
   375→		}
   376→		if err := gzipWriter.Close(); err != nil {
   377→			cclog.Errorf("SqliteArchive ImportJob() > gzip close error: %v", err)
   378→			return err
   379→		}
   380→		dataBytes = compressedBuf.Bytes()
   381→		compressed = true
   382→	} else {
   383→		dataBytes = dataBuf.Bytes()
   384→		compressed = false
   385→	}
   386→
   387→	now := time.Now().Unix()
   388→	_, err := sa.db.Exec(`
   389→		INSERT INTO jobs (job_id, cluster, start_time, meta_json, data_json, data_compressed, created_at, updated_at)
   390→		VALUES (?, ?, ?, ?, ?, ?, ?, ?)
   391→		ON CONFLICT(job_id, cluster, start_time) DO UPDATE SET
   392→			meta_json = excluded.meta_json,
   393→			data_json = excluded.data_json,
   394→			data_compressed = excluded.data_compressed,
   395→			updated_at = excluded.updated_at
   396→	`, jobMeta.JobID, jobMeta.Cluster, jobMeta.StartTime, metaBuf.Bytes(), dataBytes, compressed, now, now)
   397→	if err != nil {
   398→		cclog.Errorf("SqliteArchive ImportJob() > insert error: %v", err)
   399→		return err
   400→	}
   401→
   402→	return nil
   403→}
   404→
   405→func (sa *SqliteArchive) GetClusters() []string {
   406→	return sa.clusters
   407→}
   408→
   409→func (sa *SqliteArchive) CleanUp(jobs []*schema.Job) {
   410→	start := time.Now()
   411→	count := 0
   412→
   413→	tx, err := sa.db.Begin()
   414→	if err != nil {
   415→		cclog.Errorf("SqliteArchive CleanUp() > transaction error: %v", err)
   416→		return
   417→	}
   418→	defer tx.Rollback()
   419→
   420→	stmt, err := tx.Prepare("DELETE FROM jobs WHERE job_id = ? AND cluster = ? AND start_time = ?")
   421→	if err != nil {
   422→		cclog.Errorf("SqliteArchive CleanUp() > prepare error: %v", err)
   423→		return
   424→	}
   425→	defer stmt.Close()
   426→
   427→	for _, job := range jobs {
   428→		if job == nil {
   429→			cclog.Errorf("SqliteArchive CleanUp() error: job is nil")
   430→			continue
   431→		}
   432→
   433→		if _, err := stmt.Exec(job.JobID, job.Cluster, job.StartTime); err != nil {
   434→			cclog.Errorf("SqliteArchive CleanUp() > delete error: %v", err)
   435→		} else {
   436→			count++
   437→		}
   438→	}
   439→
   440→	if err := tx.Commit(); err != nil {
   441→		cclog.Errorf("SqliteArchive CleanUp() > commit error: %v", err)
   442→		return
   443→	}
   444→
   445→	cclog.Infof("Retention Service - Remove %d jobs from SQLite in %s", count, time.Since(start))
   446→}
   447→
   448→func (sa *SqliteArchive) Move(jobs []*schema.Job, targetPath string) {
   449→	// For SQLite, "move" means updating the cluster field or similar
   450→	// This is interpretation-dependent; for now we'll just log
   451→	cclog.Warnf("SqliteArchive Move() is not fully implemented - moves within database not applicable")
   452→}
   453→
   454→func (sa *SqliteArchive) Clean(before int64, after int64) {
   455→	if after == 0 {
   456→		after = math.MaxInt64
   457→	}
   458→
   459→	result, err := sa.db.Exec("DELETE FROM jobs WHERE start_time < ? OR start_time > ?", before, after)
   460→	if err != nil {
   461→		cclog.Fatalf("SqliteArchive Clean() > delete error: %s", err.Error())
   462→	}
   463→
   464→	rowsAffected, _ := result.RowsAffected()
   465→	cclog.Infof("SqliteArchive Clean() removed %d jobs", rowsAffected)
   466→}
   467→
   468→func (sa *SqliteArchive) Compress(jobs []*schema.Job) {
   469→	var cnt int
   470→	start := time.Now()
   471→
   472→	tx, err := sa.db.Begin()
   473→	if err != nil {
   474→		cclog.Errorf("SqliteArchive Compress() > transaction error: %v", err)
   475→		return
   476→	}
   477→	defer tx.Rollback()
   478→
   479→	stmt, err := tx.Prepare("UPDATE jobs SET data_json = ?, data_compressed = 1 WHERE job_id = ? AND cluster = ? AND start_time = ?")
   480→	if err != nil {
   481→		cclog.Errorf("SqliteArchive Compress() > prepare error: %v", err)
   482→		return
   483→	}
   484→	defer stmt.Close()
   485→
   486→	for _, job := range jobs {
   487→		var dataBlob []byte
   488→		var compressed bool
   489→		err := sa.db.QueryRow("SELECT data_json, data_compressed FROM jobs WHERE job_id = ? AND cluster = ? AND start_time = ?",
   490→			job.JobID, job.Cluster, job.StartTime).Scan(&dataBlob, &compressed)
   491→		if err != nil || compressed || len(dataBlob) < 2000 {
   492→			continue // Skip if error, already compressed, or too small
   493→		}
   494→
   495→		// Compress the data
   496→		var compressedBuf bytes.Buffer
   497→		gzipWriter := gzip.NewWriter(&compressedBuf)
   498→		if _, err := gzipWriter.Write(dataBlob); err != nil {
   499→			cclog.Errorf("SqliteArchive Compress() > gzip error: %v", err)
   500→			gzipWriter.Close()
   501→			continue
   502→		}
   503→		gzipWriter.Close()
   504→
   505→		if _, err := stmt.Exec(compressedBuf.Bytes(), job.JobID, job.Cluster, job.StartTime); err != nil {
   506→			cclog.Errorf("SqliteArchive Compress() > update error: %v", err)
   507→		} else {
   508→			cnt++
   509→		}
   510→	}
   511→
   512→	if err := tx.Commit(); err != nil {
   513→		cclog.Errorf("SqliteArchive Compress() > commit error: %v", err)
   514→		return
   515→	}
   516→
   517→	cclog.Infof("Compression Service - %d jobs in SQLite took %s", cnt, time.Since(start))
   518→}
   519→
   520→func (sa *SqliteArchive) CompressLast(starttime int64) int64 {
   521→	var lastStr string
   522→	err := sa.db.QueryRow("SELECT value FROM metadata WHERE key = 'compress_last'").Scan(&lastStr)
   523→
   524→	var last int64
   525→	if err == sql.ErrNoRows {
   526→		last = starttime
   527→	} else if err != nil {
   528→		cclog.Errorf("SqliteArchive CompressLast() > query error: %v", err)
   529→		last = starttime
   530→	} else {
   531→		last, err = strconv.ParseInt(lastStr, 10, 64)
   532→		if err != nil {
   533→			cclog.Errorf("SqliteArchive CompressLast() > parse error: %v", err)
   534→			last = starttime
   535→		}
   536→	}
   537→
   538→	cclog.Infof("SqliteArchive CompressLast() - start %d last %d", starttime, last)
   539→
   540→	// Update timestamp
   541→	_, err = sa.db.Exec(`
   542→		INSERT INTO metadata (key, value) VALUES ('compress_last', ?)
   543→		ON CONFLICT(key) DO UPDATE SET value = excluded.value
   544→	`, fmt.Sprintf("%d", starttime))
   545→	if err != nil {
   546→		cclog.Errorf("SqliteArchive CompressLast() > update error: %v", err)
   547→	}
   548→
   549→	return last
   550→}
   551→
   552→type sqliteJobRow struct {
   553→	metaBlob   []byte
   554→	dataBlob   []byte
   555→	compressed bool
   556→}
   557→
   558→func (sa *SqliteArchive) Iter(loadMetricData bool) <-chan JobContainer {
   559→	ch := make(chan JobContainer)
   560→
   561→	go func() {
   562→		defer close(ch)
   563→
   564→		const chunkSize = 1000
   565→		offset := 0
   566→
   567→		var query string
   568→		if loadMetricData {
   569→			query = "SELECT meta_json, data_json, data_compressed FROM jobs ORDER BY cluster, start_time LIMIT ? OFFSET ?"
   570→		} else {
   571→			query = "SELECT meta_json FROM jobs ORDER BY cluster, start_time LIMIT ? OFFSET ?"
   572→		}
   573→
   574→		numWorkers := 4
   575→		jobRows := make(chan sqliteJobRow, numWorkers*2)
   576→		var wg sync.WaitGroup
   577→
   578→		for range numWorkers {
   579→			wg.Go(func() {
   580→				for row := range jobRows {
   581→					job, err := DecodeJobMeta(bytes.NewReader(row.metaBlob))
   582→					if err != nil {
   583→						cclog.Errorf("SqliteArchive Iter() > decode meta error: %v", err)
   584→						continue
   585→					}
   586→
   587→					if loadMetricData && row.dataBlob != nil {
   588→						var reader io.Reader = bytes.NewReader(row.dataBlob)
   589→						if row.compressed {
   590→							gzipReader, err := gzip.NewReader(reader)
   591→							if err != nil {
   592→								cclog.Errorf("SqliteArchive Iter() > gzip error: %v", err)
   593→								ch <- JobContainer{Meta: job, Data: nil}
   594→								continue
   595→							}
   596→							decompressed, err := io.ReadAll(gzipReader)
   597→							gzipReader.Close()
   598→							if err != nil {
   599→								cclog.Errorf("SqliteArchive Iter() > decompress error: %v", err)
   600→								ch <- JobContainer{Meta: job, Data: nil}
   601→								continue
   602→							}
   603→							reader = bytes.NewReader(decompressed)
   604→						}
   605→
   606→						key := fmt.Sprintf("%s:%d:%d", job.Cluster, job.JobID, job.StartTime)
   607→						jobData, err := DecodeJobData(reader, key)
   608→						if err != nil {
   609→							cclog.Errorf("SqliteArchive Iter() > decode data error: %v", err)
   610→							ch <- JobContainer{Meta: job, Data: nil}
   611→						} else {
   612→							ch <- JobContainer{Meta: job, Data: &jobData}
   613→						}
   614→					} else {
   615→						ch <- JobContainer{Meta: job, Data: nil}
   616→					}
   617→				}
   618→			})
   619→		}
   620→
   621→		for {
   622→			rows, err := sa.db.Query(query, chunkSize, offset)
   623→			if err != nil {
   624→				cclog.Fatalf("SqliteArchive Iter() > query error: %s", err.Error())
   625→			}
   626→
   627→			rowCount := 0
   628→			for rows.Next() {
   629→				var row sqliteJobRow
   630→
   631→				if loadMetricData {
   632→					if err := rows.Scan(&row.metaBlob, &row.dataBlob, &row.compressed); err != nil {
   633→						cclog.Errorf("SqliteArchive Iter() > scan error: %v", err)
   634→						continue
   635→					}
   636→				} else {
   637→					if err := rows.Scan(&row.metaBlob); err != nil {
   638→						cclog.Errorf("SqliteArchive Iter() > scan error: %v", err)
   639→						continue
   640→					}
   641→					row.dataBlob = nil
   642→					row.compressed = false
   643→				}
   644→
   645→				jobRows <- row
   646→				rowCount++
   647→			}
   648→			rows.Close()
   649→
   650→			if rowCount < chunkSize {
   651→				break
   652→			}
   653→
   654→			offset += chunkSize
   655→		}
   656→
   657→		close(jobRows)
   658→		wg.Wait()
   659→	}()
   660→
   661→	return ch
   662→}
   663→
   664→func (sa *SqliteArchive) StoreClusterCfg(name string, config *schema.Cluster) error {
   665→	var configBuf bytes.Buffer
   666→	if err := EncodeCluster(&configBuf, config); err != nil {
   667→		cclog.Error("SqliteArchive StoreClusterCfg() > encoding error")
   668→		return err
   669→	}
   670→
   671→	now := time.Now().Unix()
   672→	_, err := sa.db.Exec(`
   673→		INSERT INTO clusters (name, config_json, updated_at)
   674→		VALUES (?, ?, ?)
   675→		ON CONFLICT(name) DO UPDATE SET
   676→			config_json = excluded.config_json,
   677→			updated_at = excluded.updated_at
   678→	`, name, configBuf.Bytes(), now)
   679→	if err != nil {
   680→		cclog.Errorf("SqliteArchive StoreClusterCfg() > insert error: %v", err)
   681→		return err
   682→	}
   683→
   684→	// Update clusters list if new
   685→	found := slices.Contains(sa.clusters, name)
   686→	if !found {
   687→		sa.clusters = append(sa.clusters, name)
   688→	}
   689→
   690→	return nil
   691→}
   692→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package archive

import (
	"bytes"
	"compress/gzip"
	"database/sql"
	"encoding/json"
	"fmt"
	"io"
	"math"
	"os"
	"slices"
	"strconv"
	"sync"
	"text/tabwriter"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/config"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	"github.com/ClusterCockpit/cc-lib/v2/util"
	_ "github.com/mattn/go-sqlite3"
)

// SqliteArchiveConfig holds the configuration for the SQLite archive backend.
type SqliteArchiveConfig struct {
	DBPath string `json:"db-path"` // Path to SQLite database file
}

// SqliteArchive implements ArchiveBackend using a SQLite database with BLOB storage.
// Job metadata and data are stored as JSON BLOBs with indexes for fast queries.
//
// Uses WAL (Write-Ahead Logging) mode for better concurrency and a 64MB cache.
type SqliteArchive struct {
	db       *sql.DB  // SQLite database connection
	clusters []string // List of discovered cluster names
}

// sqliteSchema defines the database schema for SQLite archive backend.
// Jobs table: stores job metadata and data as BLOBs with compression flag
// Clusters table: stores cluster configurations
// Metadata table: stores version and other key-value pairs
const sqliteSchema = `
CREATE TABLE IF NOT EXISTS jobs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    job_id INTEGER NOT NULL,
    cluster TEXT NOT NULL,
    start_time INTEGER NOT NULL,
    meta_json BLOB NOT NULL,
    data_json BLOB,
    data_compressed BOOLEAN DEFAULT 0,
    created_at INTEGER NOT NULL,
    updated_at INTEGER NOT NULL,
    UNIQUE(job_id, cluster, start_time)
);

CREATE INDEX IF NOT EXISTS idx_jobs_cluster ON jobs(cluster);
CREATE INDEX IF NOT EXISTS idx_jobs_start_time ON jobs(start_time);
CREATE INDEX IF NOT EXISTS idx_jobs_order ON jobs(cluster, start_time);
CREATE INDEX IF NOT EXISTS idx_jobs_lookup ON jobs(cluster, job_id, start_time);

CREATE TABLE IF NOT EXISTS clusters (
    name TEXT PRIMARY KEY,
    config_json BLOB NOT NULL,
    updated_at INTEGER NOT NULL
);

CREATE TABLE IF NOT EXISTS metadata (
    key TEXT PRIMARY KEY,
    value TEXT NOT NULL
);
`

func (sa *SqliteArchive) Init(rawConfig json.RawMessage) (uint64, error) {
	var cfg SqliteArchiveConfig
	if err := json.Unmarshal(rawConfig, &cfg); err != nil {
		cclog.Warnf("SqliteArchive Init() > Unmarshal error: %#v", err)
		return 0, err
	}

	if cfg.DBPath == "" {
		err := fmt.Errorf("SqliteArchive Init(): empty database path")
		cclog.Errorf("SqliteArchive Init() > config error: %v", err)
		return 0, err
	}

	// Open SQLite database
	db, err := sql.Open("sqlite3", cfg.DBPath)
	if err != nil {
		cclog.Errorf("SqliteArchive Init() > failed to open database: %v", err)
		return 0, err
	}
	sa.db = db

	// Set pragmas for better performance
	pragmas := []string{
		"PRAGMA journal_mode=WAL",
		"PRAGMA synchronous=NORMAL",
		"PRAGMA cache_size=-64000", // 64MB cache
		"PRAGMA busy_timeout=5000",
	}
	for _, pragma := range pragmas {
		if _, err := sa.db.Exec(pragma); err != nil {
			cclog.Warnf("SqliteArchive Init() > pragma failed: %v", err)
		}
	}

	// Create schema
	if _, err := sa.db.Exec(sqliteSchema); err != nil {
		cclog.Errorf("SqliteArchive Init() > schema creation failed: %v", err)
		return 0, err
	}

	// Check/set version
	var versionStr string
	err = sa.db.QueryRow("SELECT value FROM metadata WHERE key = 'version'").Scan(&versionStr)
	if err == sql.ErrNoRows {
		// First time initialization, set version
		_, err = sa.db.Exec("INSERT INTO metadata (key, value) VALUES ('version', ?)", fmt.Sprintf("%d", Version))
		if err != nil {
			cclog.Errorf("SqliteArchive Init() > failed to set version: %v", err)
			return 0, err
		}
		versionStr = fmt.Sprintf("%d", Version)
	} else if err != nil {
		cclog.Errorf("SqliteArchive Init() > failed to read version: %v", err)
		return 0, err
	}

	version, err := strconv.ParseUint(versionStr, 10, 64)
	if err != nil {
		cclog.Errorf("SqliteArchive Init() > version parse error: %v", err)
		return 0, err
	}

	if version != Version {
		return version, fmt.Errorf("unsupported version %d, need %d", version, Version)
	}

	// Discover clusters
	sa.clusters = []string{}
	rows, err := sa.db.Query("SELECT DISTINCT cluster FROM jobs ORDER BY cluster")
	if err != nil {
		cclog.Errorf("SqliteArchive Init() > failed to query clusters: %v", err)
		return 0, err
	}
	defer rows.Close()

	for rows.Next() {
		var cluster string
		if err := rows.Scan(&cluster); err != nil {
			cclog.Errorf("SqliteArchive Init() > failed to scan cluster: %v", err)
			continue
		}
		sa.clusters = append(sa.clusters, cluster)
	}

	cclog.Infof("SqliteArchive initialized with database '%s', found %d clusters", cfg.DBPath, len(sa.clusters))
	return version, nil
}

func (sa *SqliteArchive) Info() {
	fmt.Printf("SQLite Job archive database\n")

	ci := make(map[string]*clusterInfo)

	rows, err := sa.db.Query(`
		SELECT cluster, COUNT(*), MIN(start_time), MAX(start_time),
		       SUM(LENGTH(meta_json) + COALESCE(LENGTH(data_json), 0))
		FROM jobs
		GROUP BY cluster
	`)
	if err != nil {
		cclog.Fatalf("SqliteArchive Info() > query failed: %s", err.Error())
	}
	defer rows.Close()

	for rows.Next() {
		var cluster string
		var numJobs int
		var dateFirst, dateLast int64
		var diskSize int64

		if err := rows.Scan(&cluster, &numJobs, &dateFirst, &dateLast, &diskSize); err != nil {
			cclog.Errorf("SqliteArchive Info() > scan failed: %v", err)
			continue
		}

		ci[cluster] = &clusterInfo{
			numJobs:   numJobs,
			dateFirst: dateFirst,
			dateLast:  dateLast,
			diskSize:  float64(diskSize) / (1024 * 1024), // Convert to MB
		}
	}

	cit := clusterInfo{dateFirst: time.Now().Unix()}
	w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', tabwriter.Debug)
	fmt.Fprintln(w, "cluster\t#jobs\tfrom\tto\tsize (MB)")
	for cluster, clusterInfo := range ci {
		fmt.Fprintf(w, "%s\t%d\t%s\t%s\t%.2f\n", cluster,
			clusterInfo.numJobs,
			time.Unix(clusterInfo.dateFirst, 0),
			time.Unix(clusterInfo.dateLast, 0),
			clusterInfo.diskSize)

		cit.numJobs += clusterInfo.numJobs
		cit.dateFirst = util.Min(cit.dateFirst, clusterInfo.dateFirst)
		cit.dateLast = util.Max(cit.dateLast, clusterInfo.dateLast)
		cit.diskSize += clusterInfo.diskSize
	}

	fmt.Fprintf(w, "TOTAL\t%d\t%s\t%s\t%.2f\n",
		cit.numJobs, time.Unix(cit.dateFirst, 0), time.Unix(cit.dateLast, 0), cit.diskSize)
	w.Flush()
}

func (sa *SqliteArchive) Exists(job *schema.Job) bool {
	var count int
	err := sa.db.QueryRow("SELECT COUNT(*) FROM jobs WHERE job_id = ? AND cluster = ? AND start_time = ?",
		job.JobID, job.Cluster, job.StartTime).Scan(&count)
	return err == nil && count > 0
}

func (sa *SqliteArchive) LoadJobMeta(job *schema.Job) (*schema.Job, error) {
	var metaBlob []byte
	err := sa.db.QueryRow("SELECT meta_json FROM jobs WHERE job_id = ? AND cluster = ? AND start_time = ?",
		job.JobID, job.Cluster, job.StartTime).Scan(&metaBlob)
	if err != nil {
		cclog.Errorf("SqliteArchive LoadJobMeta() > query error: %v", err)
		return nil, err
	}

	if config.Keys.Validate {
		if err := schema.Validate(schema.Meta, bytes.NewReader(metaBlob)); err != nil {
			return nil, fmt.Errorf("validate job meta: %v", err)
		}
	}

	return DecodeJobMeta(bytes.NewReader(metaBlob))
}

func (sa *SqliteArchive) LoadJobData(job *schema.Job) (schema.JobData, error) {
	var dataBlob []byte
	var compressed bool
	err := sa.db.QueryRow("SELECT data_json, data_compressed FROM jobs WHERE job_id = ? AND cluster = ? AND start_time = ?",
		job.JobID, job.Cluster, job.StartTime).Scan(&dataBlob, &compressed)
	if err != nil {
		cclog.Errorf("SqliteArchive LoadJobData() > query error: %v", err)
		return nil, err
	}
	key := fmt.Sprintf("%s:%d:%d", job.Cluster, job.JobID, job.StartTime)

	var reader io.Reader = bytes.NewReader(dataBlob)
	if compressed {
		gzipReader, err := gzip.NewReader(reader)
		if err != nil {
			cclog.Errorf("SqliteArchive LoadJobData() > gzip error: %v", err)
			return nil, err
		}
		defer gzipReader.Close()
		reader = gzipReader
	}

	if config.Keys.Validate {
		data, _ := io.ReadAll(reader)
		if err := schema.Validate(schema.Data, bytes.NewReader(data)); err != nil {
			return schema.JobData{}, fmt.Errorf("validate job data: %v", err)
		}
		return DecodeJobData(bytes.NewReader(data), key)
	}

	return DecodeJobData(reader, key)
}

func (sa *SqliteArchive) LoadJobStats(job *schema.Job) (schema.ScopedJobStats, error) {
	var dataBlob []byte
	var compressed bool
	err := sa.db.QueryRow("SELECT data_json, data_compressed FROM jobs WHERE job_id = ? AND cluster = ? AND start_time = ?",
		job.JobID, job.Cluster, job.StartTime).Scan(&dataBlob, &compressed)
	if err != nil {
		cclog.Errorf("SqliteArchive LoadJobStats() > query error: %v", err)
		return nil, err
	}
	key := fmt.Sprintf("%s:%d:%d", job.Cluster, job.JobID, job.StartTime)

	var reader io.Reader = bytes.NewReader(dataBlob)
	if compressed {
		gzipReader, err := gzip.NewReader(reader)
		if err != nil {
			cclog.Errorf("SqliteArchive LoadJobStats() > gzip error: %v", err)
			return nil, err
		}
		defer gzipReader.Close()
		reader = gzipReader
	}

	if config.Keys.Validate {
		data, _ := io.ReadAll(reader)
		if err := schema.Validate(schema.Data, bytes.NewReader(data)); err != nil {
			return nil, fmt.Errorf("validate job data: %v", err)
		}
		return DecodeJobStats(bytes.NewReader(data), key)
	}

	return DecodeJobStats(reader, key)
}

func (sa *SqliteArchive) LoadClusterCfg(name string) (*schema.Cluster, error) {
	var configBlob []byte
	err := sa.db.QueryRow("SELECT config_json FROM clusters WHERE name = ?", name).Scan(&configBlob)
	if err != nil {
		cclog.Errorf("SqliteArchive LoadClusterCfg() > query error: %v", err)
		return nil, err
	}

	if config.Keys.Validate {
		if err := schema.Validate(schema.ClusterCfg, bytes.NewReader(configBlob)); err != nil {
			cclog.Warnf("Validate cluster config: %v\n", err)
			return &schema.Cluster{}, fmt.Errorf("validate cluster config: %v", err)
		}
	}

	return DecodeCluster(bytes.NewReader(configBlob))
}

func (sa *SqliteArchive) StoreJobMeta(job *schema.Job) error {
	var metaBuf bytes.Buffer
	if err := EncodeJobMeta(&metaBuf, job); err != nil {
		cclog.Error("SqliteArchive StoreJobMeta() > encoding error")
		return err
	}

	now := time.Now().Unix()
	_, err := sa.db.Exec(`
		INSERT INTO jobs (job_id, cluster, start_time, meta_json, created_at, updated_at)
		VALUES (?, ?, ?, ?, ?, ?)
		ON CONFLICT(job_id, cluster, start_time) DO UPDATE SET
			meta_json = excluded.meta_json,
			updated_at = excluded.updated_at
	`, job.JobID, job.Cluster, job.StartTime, metaBuf.Bytes(), now, now)
	if err != nil {
		cclog.Errorf("SqliteArchive StoreJobMeta() > insert error: %v", err)
		return err
	}

	return nil
}

func (sa *SqliteArchive) ImportJob(jobMeta *schema.Job, jobData *schema.JobData) error {
	var metaBuf, dataBuf bytes.Buffer
	if err := EncodeJobMeta(&metaBuf, jobMeta); err != nil {
		cclog.Error("SqliteArchive ImportJob() > encoding meta error")
		return err
	}
	if err := EncodeJobData(&dataBuf, jobData); err != nil {
		cclog.Error("SqliteArchive ImportJob() > encoding data error")
		return err
	}

	var dataBytes []byte
	var compressed bool

	if dataBuf.Len() > 2000 {
		var compressedBuf bytes.Buffer
		gzipWriter := gzip.NewWriter(&compressedBuf)
		if _, err := gzipWriter.Write(dataBuf.Bytes()); err != nil {
			cclog.Errorf("SqliteArchive ImportJob() > gzip write error: %v", err)
			return err
		}
		if err := gzipWriter.Close(); err != nil {
			cclog.Errorf("SqliteArchive ImportJob() > gzip close error: %v", err)
			return err
		}
		dataBytes = compressedBuf.Bytes()
		compressed = true
	} else {
		dataBytes = dataBuf.Bytes()
		compressed = false
	}

	now := time.Now().Unix()
	_, err := sa.db.Exec(`
		INSERT INTO jobs (job_id, cluster, start_time, meta_json, data_json, data_compressed, created_at, updated_at)
		VALUES (?, ?, ?, ?, ?, ?, ?, ?)
		ON CONFLICT(job_id, cluster, start_time) DO UPDATE SET
			meta_json = excluded.meta_json,
			data_json = excluded.data_json,
			data_compressed = excluded.data_compressed,
			updated_at = excluded.updated_at
	`, jobMeta.JobID, jobMeta.Cluster, jobMeta.StartTime, metaBuf.Bytes(), dataBytes, compressed, now, now)
	if err != nil {
		cclog.Errorf("SqliteArchive ImportJob() > insert error: %v", err)
		return err
	}

	return nil
}

func (sa *SqliteArchive) GetClusters() []string {
	return sa.clusters
}

func (sa *SqliteArchive) CleanUp(jobs []*schema.Job) {
	start := time.Now()
	count := 0

	tx, err := sa.db.Begin()
	if err != nil {
		cclog.Errorf("SqliteArchive CleanUp() > transaction error: %v", err)
		return
	}
	defer tx.Rollback()

	stmt, err := tx.Prepare("DELETE FROM jobs WHERE job_id = ? AND cluster = ? AND start_time = ?")
	if err != nil {
		cclog.Errorf("SqliteArchive CleanUp() > prepare error: %v", err)
		return
	}
	defer stmt.Close()

	for _, job := range jobs {
		if job == nil {
			cclog.Errorf("SqliteArchive CleanUp() error: job is nil")
			continue
		}

		if _, err := stmt.Exec(job.JobID, job.Cluster, job.StartTime); err != nil {
			cclog.Errorf("SqliteArchive CleanUp() > delete error: %v", err)
		} else {
			count++
		}
	}

	if err := tx.Commit(); err != nil {
		cclog.Errorf("SqliteArchive CleanUp() > commit error: %v", err)
		return
	}

	cclog.Infof("Retention Service - Remove %d jobs from SQLite in %s", count, time.Since(start))
}

func (sa *SqliteArchive) Move(jobs []*schema.Job, targetPath string) {
	// For SQLite, "move" means updating the cluster field or similar
	// This is interpretation-dependent; for now we'll just log
	cclog.Warnf("SqliteArchive Move() is not fully implemented - moves within database not applicable")
}

func (sa *SqliteArchive) Clean(before int64, after int64) {
	if after == 0 {
		after = math.MaxInt64
	}

	result, err := sa.db.Exec("DELETE FROM jobs WHERE start_time < ? OR start_time > ?", before, after)
	if err != nil {
		cclog.Fatalf("SqliteArchive Clean() > delete error: %s", err.Error())
	}

	rowsAffected, _ := result.RowsAffected()
	cclog.Infof("SqliteArchive Clean() removed %d jobs", rowsAffected)
}

func (sa *SqliteArchive) Compress(jobs []*schema.Job) {
	var cnt int
	start := time.Now()

	tx, err := sa.db.Begin()
	if err != nil {
		cclog.Errorf("SqliteArchive Compress() > transaction error: %v", err)
		return
	}
	defer tx.Rollback()

	stmt, err := tx.Prepare("UPDATE jobs SET data_json = ?, data_compressed = 1 WHERE job_id = ? AND cluster = ? AND start_time = ?")
	if err != nil {
		cclog.Errorf("SqliteArchive Compress() > prepare error: %v", err)
		return
	}
	defer stmt.Close()

	for _, job := range jobs {
		var dataBlob []byte
		var compressed bool
		err := sa.db.QueryRow("SELECT data_json, data_compressed FROM jobs WHERE job_id = ? AND cluster = ? AND start_time = ?",
			job.JobID, job.Cluster, job.StartTime).Scan(&dataBlob, &compressed)
		if err != nil || compressed || len(dataBlob) < 2000 {
			continue // Skip if error, already compressed, or too small
		}

		// Compress the data
		var compressedBuf bytes.Buffer
		gzipWriter := gzip.NewWriter(&compressedBuf)
		if _, err := gzipWriter.Write(dataBlob); err != nil {
			cclog.Errorf("SqliteArchive Compress() > gzip error: %v", err)
			gzipWriter.Close()
			continue
		}
		gzipWriter.Close()

		if _, err := stmt.Exec(compressedBuf.Bytes(), job.JobID, job.Cluster, job.StartTime); err != nil {
			cclog.Errorf("SqliteArchive Compress() > update error: %v", err)
		} else {
			cnt++
		}
	}

	if err := tx.Commit(); err != nil {
		cclog.Errorf("SqliteArchive Compress() > commit error: %v", err)
		return
	}

	cclog.Infof("Compression Service - %d jobs in SQLite took %s", cnt, time.Since(start))
}

func (sa *SqliteArchive) CompressLast(starttime int64) int64 {
	var lastStr string
	err := sa.db.QueryRow("SELECT value FROM metadata WHERE key = 'compress_last'").Scan(&lastStr)

	var last int64
	if err == sql.ErrNoRows {
		last = starttime
	} else if err != nil {
		cclog.Errorf("SqliteArchive CompressLast() > query error: %v", err)
		last = starttime
	} else {
		last, err = strconv.ParseInt(lastStr, 10, 64)
		if err != nil {
			cclog.Errorf("SqliteArchive CompressLast() > parse error: %v", err)
			last = starttime
		}
	}

	cclog.Infof("SqliteArchive CompressLast() - start %d last %d", starttime, last)

	// Update timestamp
	_, err = sa.db.Exec(`
		INSERT INTO metadata (key, value) VALUES ('compress_last', ?)
		ON CONFLICT(key) DO UPDATE SET value = excluded.value
	`, fmt.Sprintf("%d", starttime))
	if err != nil {
		cclog.Errorf("SqliteArchive CompressLast() > update error: %v", err)
	}

	return last
}

type sqliteJobRow struct {
	metaBlob   []byte
	dataBlob   []byte
	compressed bool
}

func (sa *SqliteArchive) Iter(loadMetricData bool) <-chan JobContainer {
	ch := make(chan JobContainer)

	go func() {
		defer close(ch)

		const chunkSize = 1000
		offset := 0

		var query string
		if loadMetricData {
			query = "SELECT meta_json, data_json, data_compressed FROM jobs ORDER BY cluster, start_time LIMIT ? OFFSET ?"
		} else {
			query = "SELECT meta_json FROM jobs ORDER BY cluster, start_time LIMIT ? OFFSET ?"
		}

		numWorkers := 4
		jobRows := make(chan sqliteJobRow, numWorkers*2)
		var wg sync.WaitGroup

		for range numWorkers {
			wg.Go(func() {
				for row := range jobRows {
					job, err := DecodeJobMeta(bytes.NewReader(row.metaBlob))
					if err != nil {
						cclog.Errorf("SqliteArchive Iter() > decode meta error: %v", err)
						continue
					}

					if loadMetricData && row.dataBlob != nil {
						var reader io.Reader = bytes.NewReader(row.dataBlob)
						if row.compressed {
							gzipReader, err := gzip.NewReader(reader)
							if err != nil {
								cclog.Errorf("SqliteArchive Iter() > gzip error: %v", err)
								ch <- JobContainer{Meta: job, Data: nil}
								continue
							}
							decompressed, err := io.ReadAll(gzipReader)
							gzipReader.Close()
							if err != nil {
								cclog.Errorf("SqliteArchive Iter() > decompress error: %v", err)
								ch <- JobContainer{Meta: job, Data: nil}
								continue
							}
							reader = bytes.NewReader(decompressed)
						}

						key := fmt.Sprintf("%s:%d:%d", job.Cluster, job.JobID, job.StartTime)
						jobData, err := DecodeJobData(reader, key)
						if err != nil {
							cclog.Errorf("SqliteArchive Iter() > decode data error: %v", err)
							ch <- JobContainer{Meta: job, Data: nil}
						} else {
							ch <- JobContainer{Meta: job, Data: &jobData}
						}
					} else {
						ch <- JobContainer{Meta: job, Data: nil}
					}
				}
			})
		}

		for {
			rows, err := sa.db.Query(query, chunkSize, offset)
			if err != nil {
				cclog.Fatalf("SqliteArchive Iter() > query error: %s", err.Error())
			}

			rowCount := 0
			for rows.Next() {
				var row sqliteJobRow

				if loadMetricData {
					if err := rows.Scan(&row.metaBlob, &row.dataBlob, &row.compressed); err != nil {
						cclog.Errorf("SqliteArchive Iter() > scan error: %v", err)
						continue
					}
				} else {
					if err := rows.Scan(&row.metaBlob); err != nil {
						cclog.Errorf("SqliteArchive Iter() > scan error: %v", err)
						continue
					}
					row.dataBlob = nil
					row.compressed = false
				}

				jobRows <- row
				rowCount++
			}
			rows.Close()

			if rowCount < chunkSize {
				break
			}

			offset += chunkSize
		}

		close(jobRows)
		wg.Wait()
	}()

	return ch
}

func (sa *SqliteArchive) StoreClusterCfg(name string, config *schema.Cluster) error {
	var configBuf bytes.Buffer
	if err := EncodeCluster(&configBuf, config); err != nil {
		cclog.Error("SqliteArchive StoreClusterCfg() > encoding error")
		return err
	}

	now := time.Now().Unix()
	_, err := sa.db.Exec(`
		INSERT INTO clusters (name, config_json, updated_at)
		VALUES (?, ?, ?)
		ON CONFLICT(name) DO UPDATE SET
			config_json = excluded.config_json,
			updated_at = excluded.updated_at
	`, name, configBuf.Bytes(), now)
	if err != nil {
		cclog.Errorf("SqliteArchive StoreClusterCfg() > insert error: %v", err)
		return err
	}

	// Update clusters list if new
	found := slices.Contains(sa.clusters, name)
	if !found {
		sa.clusters = append(sa.clusters, name)
	}

	return nil
}