The file /Users/jan/.claude/plans/this-application-uses-a-shiny-flame.md has been updated successfully. (file state is current in your context — no need to Read it back)

# Code Review — Initialization, Startup & Shutdown (critical/severe focus)

## Context

Requested: a thorough review of cc-backend focused on **critical and severe**
issues, starting with the initialization / startup / shutdown paths
(`cmd/cc-backend/main.go`, `cmd/cc-backend/server.go`, and the subsystem
`Init`/`Start`/`Shutdown` functions). Three exploration passes were run; their
findings were then **verified against the actual code** — many reported items were
theoretical or did not occur in the real call flow and are listed as rejected at
the bottom so they are not re-investigated.

This file is a review report plus a prioritized remediation plan. The previously
applied WAL-on-shutdown fix (`metricstore.Shutdown` now rotates WAL via
`RotateWALFilesAfterShutdown`) is already merged and is **not** re-listed.

---

## VERIFIED findings

### F1 — CRITICAL: `disable-authentication: true` aborts the server on startup
The server unconditionally requires an initialized auth instance, but auth is
only initialized when authentication is **enabled**.

- `auth.Init(...)` is called *only* inside the `if !config.Keys.DisableAuthentication`
  block — `cmd/cc-backend/main.go:175-181` (the only two call sites, confirmed by grep).
- `auth.GetAuthInstance()` is called **unconditionally** during server init:
  - `cmd/cc-backend/server.go:123` (`authHandle := auth.GetAuthInstance()`)
  - `internal/api/rest.go:72` via `api.New()` (`cmd/cc-backend/server.go:184`)
- `GetAuthInstance()` calls `cclog.Fatal("Authentication module not initialized!")`
  when `authInstance == nil` — `internal/auth/auth.go:258-264`.

Result: with `disable-authentication: true` and `-server`, the process hard-aborts
at startup. The flag is otherwise honored in 7 places in `server.go`, so it is a
supported configuration that is currently broken. Default deployments
(auth enabled) are unaffected, which is why it has gone unnoticed.

**Fix direction**: ensure auth is always initialized, or guard the unconditional
callers. Cleanest: when `config.Keys.DisableAuthentication` is true, still call
`auth.Init(nil)` (it sets up a session store and returns early without
authenticators — `auth.go:181-206`), OR make `GetAuthInstance()` callers tolerate
a nil instance when auth is disabled. Preferred: call `auth.Init(nil)` for the
disabled case in `handleUserCommands`/startup so `authInstance` is non-nil. Verify
no auth-enabled-only middleware is wired when disabled (it already is guarded).

### F2 — SEVERE: startup-error path skips graceful shutdown
In `runServer` (`cmd/cc-backend/main.go:392-463`), if `srv.Start` returns an error
(e.g. port in use, `tls.LoadX509KeyPair` failure, privileged-port bind), the error
is delivered on `errChan`, the `select` returns it, `run()` returns, and `main()`
calls `os.Exit(1)` (`main.go:589-594`).

- The graceful-shutdown sequence (`metricstore.Shutdown()` final checkpoint + WAL
  rotation, `archiver.Shutdown()`, `taskmanager.Shutdown()`) lives only in the
  signal-handler goroutine (`main.go:406-418`), which is unblocked via `ctx.Done()`
  from the deferred `cancel()` (`main.go:584`) — but `os.Exit(1)` races ahead and
  the process dies before that goroutine runs.
- The metricstore background goroutines started in `metricstore.Init`
  (`Retention`, `Checkpointing`, `WALStaging`, `ReceiveNats`) are tied to the
  store's own internal context and are never cancelled on this path.

Practical data-loss risk is low (the listener fails at startup before any ingest),
but "graceful shutdown" is silently not performed on the error path, and it is a
latent correctness gap if `Serve` ever returns mid-run.

**Fix direction**: on the error return from `runServer`, invoke the same shutdown
sequence (or trigger the signal goroutine and wait for `waitDone`) before
returning, so the final checkpoint runs.

---

## Lower-severity (worth fixing, not blocking)

- **M1 — port-80 redirect goroutine swallows its error** (`server.go:417-419`):
  `go func(){ http.ListenAndServe(":80", ...) }()` ignores the returned error; a
  failed bind on :80 is completely silent. Log it.
- **M2 — archiver shutdown leaks its waiter goroutine and skips a `Done()` on
  timeout** (`internal/archiver/archiveWorker.go:222-249`, `:115-117`): on the
  timeout branch `shutdownCancel()` makes the worker `return` without
  `archivePending.Done()` for the in-flight job, so the
  `go func(){ archivePending.Wait(); close(done) }()` goroutine never returns.
  Only happens during shutdown timeout (process exiting), so impact is bounded.
- **M3 — NATS API workers/channels not closed on shutdown**
  (`internal/api/nats.go` — `jobWorker`/`nodeWorker` loop over `jobCh`/`nodeCh`,
  never closed; created at `server.go:330-333`): goroutines block forever. This is
  a one-time leak at process shutdown (not per-request/per-restart), so low impact,
  but buffered job/node events in those channels are dropped silently.
- **M4 — `metricstore.Shutdown()` is not idempotent** (`metricstore.go:274-321`):
  `shutdownFunc` is not nil'd after use and a second call would re-run
  `ToCheckpointWAL` + `RotateWALFilesAfterShutdown`. Currently only called once
  (`server.go:484`), so latent only.

---

## Remediation plan (recommended order)

1. **F1** — initialize auth in the disabled case (call `auth.Init(nil)` on the
   startup path when `DisableAuthentication` is true), then smoke-test
   `-server -dev` with `disable-authentication: true`. Critical, small change in
   `cmd/cc-backend/main.go`.
2. **F2** — run the graceful-shutdown sequence on the `runServer` error path before
   returning. Small change in `cmd/cc-backend/main.go`.
3. **M1–M4** — opportunistic hardening (log ignored errors; close NATS API
   channels with a stop signal; balance archiver `Done()`; nil `shutdownFunc`).

Each fix must respect the change-impact rules in CLAUDE.md (trace callers, check
goroutine/channel lifecycle).

## Verification

- Build: `go build ./...`; `go vet ./...`.
- F1: start with a config containing `"disable-authentication": true` and
  `-server -dev`; confirm the server starts and serves `/` without the
  "Authentication module not initialized!" fatal. Re-confirm a normal
  auth-enabled start still works (login flow).
- F2: force a startup failure (e.g. bind an already-used `addr`, or point
  `https-cert-file` at a missing file) and confirm the metricstore writes its
  final checkpoint / rotates WAL before exit (check logs + `var/checkpoints`).
- Regression: `go test ./...` (esp. `./pkg/metricstore/...`, `./internal/archiver/...`).

---

## Rejected during verification (false positives — do not chase)

- **"DB migration runs before Connect(), causing multiple connections / lock
  contention."** `handleDatabaseCommands` (`main.go:139-168`) only acts when
  `-migrate-db`/`-revert-db`/`-force-db` is set, and each path ends in
  `cclog.Exitf` (process exits). In normal server startup it is a no-op. Not an issue.
- **"`GetUserRepository()`/`GetConnection()` called before `Connect()`."** Order in
  `run()` is correct: `initDatabase()` (`main.go:512`) precedes both
  `flagOptimizeDB` (`:518`) and `handleUserCommands` (`:531`).
- **Archiver "double-Start", "TriggerArchiving before Start", "nil `workerDone`
  panic".** `Start` is called exactly once (`main.go:377`) before any trigger; the
  worker `Done()` accounting on the normal/error paths is actually correct
  (`archiveWorker.go:128-168`). Only the timeout edge (M2) is real.
- **"NatsAPI 10 goroutine leaks per restart cycle."** There is one process
  lifetime per run; it is a single shutdown-time leak (M3), not per-restart.
- **NATS-close-before-ctx-cancel "panic / mass data loss".** `nc.Close()`
  (`server.go:461`) stops callbacks before `metricstore.Shutdown()` runs; the
  `walShuttingDown` guard in `SendWALMessage` (`walCheckpoint.go:150-164`) prevents
  send-on-closed-channel, and buffered `msgs` are drained when the ctx is cancelled.
  Minor at most, not critical.

-# Fix: WAL files are never reset on shutdown (continuous growth across restarts)

+# Code Review — Initialization, Startup & Shutdown (critical/severe focus)

-The metric store (`pkg/metricstore`) uses a WAL + periodic binary-snapshot

-checkpoint scheme for crash recovery. Each host has an append-only

-`checkpoints/<cluster>/<host>/current.wal` that records every metric write, plus

-periodic `<ts>.bin` full-tree snapshots. On startup the newest `.bin` is loaded

+Requested: a thorough review of cc-backend focused on **critical and severe**

+issues, starting with the initialization / startup / shutdown paths

+(`cmd/cc-backend/main.go`, `cmd/cc-backend/server.go`, and the subsystem

+`Init`/`Start`/`Shutdown` functions). Three exploration passes were run; their

+findings were then **verified against the actual code** — many reported items were

+theoretical or did not occur in the real call flow and are listed as rejected at

+the bottom so they are not re-investigated.

-A `current.wal` is only meant to hold writes *since the last snapshot*. Once a

-snapshot is written, the WAL is redundant and must be rotated (deleted). The user

-reports that WAL files are never reset and grow without bound after a start.

+This file is a review report plus a prioritized remediation plan. The previously

+applied WAL-on-shutdown fix (`metricstore.Shutdown` now rotates WAL via

+`RotateWALFilesAfterShutdown`) is already merged and is **not** re-listed.

-WAL rotation happens in exactly these places today:

-1. **Periodic checkpoint** — `Checkpointing()` in `pkg/metricstore/checkpoint.go:128-145`

-   writes snapshots via `ToCheckpointWAL`, then calls `RotateWALFiles(hostDirs)`

-   for the hosts that got a fresh snapshot. This path works, but the default

-   `checkpoint-interval` is **12h**, so the WAL grows for up to 12h between resets.

-2. **Force-rotate on size** — in `WALStaging`/`processMsg`

-   (`pkg/metricstore/walCheckpoint.go:255`) when `ws.size >= MaxWALSize`. But

-   `MaxWALSize` (`config.go:69`) defaults to `0` = unlimited, so this never fires

-3. **Shutdown** — `Shutdown()` in `pkg/metricstore/metricstore.go:310-315`.

+### F1 — CRITICAL: `disable-authentication: true` aborts the server on startup

+The server unconditionally requires an initialized auth instance, but auth is

+only initialized when authentication is **enabled**.

-The **shutdown path is broken**. It writes a final snapshot but never rotates the

+- `auth.Init(...)` is called *only* inside the `if !config.Keys.DisableAuthentication`

+  block — `cmd/cc-backend/main.go:175-181` (the only two call sites, confirmed by grep).

+- `auth.GetAuthInstance()` is called **unconditionally** during server init:

+  - `cmd/cc-backend/server.go:123` (`authHandle := auth.GetAuthInstance()`)

+  - `internal/api/rest.go:72` via `api.New()` (`cmd/cc-backend/server.go:184`)

+- `GetAuthInstance()` calls `cclog.Fatal("Authentication module not initialized!")`

+  when `authInstance == nil` — `internal/auth/auth.go:258-264`.

-if Keys.Checkpoints.FileFormat == "wal" {

-    // WAL files are deleted per-host inside ToCheckpointWAL workers.   <-- FALSE

-    files, _, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())

+Result: with `disable-authentication: true` and `-server`, the process hard-aborts

+at startup. The flag is otherwise honored in 7 places in `server.go`, so it is a

+supported configuration that is currently broken. Default deployments

+(auth enabled) are unaffected, which is why it has gone unnoticed.

-- The comment is incorrect: `ToCheckpointWAL` / `toCheckpointBinary`

-  (`walCheckpoint.go:667`, `:768`) only write `.bin` snapshots; they do **not**

-- The second return value (`successDirs` — hosts that got a fresh snapshot) is

-- The purpose-built helper `RotateWALFilesAfterShutdown(hostDirs)`

-  (`walCheckpoint.go:406`), which directly `os.Remove`s `current.wal` files after

-  the staging goroutines have exited, is **defined but never called** (verified by

-  grep across `cmd/`, `internal/`, `pkg/`).

+**Fix direction**: ensure auth is always initialized, or guard the unconditional

+callers. Cleanest: when `config.Keys.DisableAuthentication` is true, still call

+`auth.Init(nil)` (it sets up a session store and returns early without

+authenticators — `auth.go:181-206`), OR make `GetAuthInstance()` callers tolerate

+a nil instance when auth is disabled. Preferred: call `auth.Init(nil)` for the

+disabled case in `handleUserCommands`/startup so `authInstance` is non-nil. Verify

+no auth-enabled-only middleware is wired when disabled (it already is guarded).

-Consequence: after each clean shutdown the `current.wal` files remain on disk. On

-the next startup they are replayed (redundantly, since the `.bin` already has that

-data) and then **appended to again** (the staging goroutine opens the existing

-file with `O_APPEND`). Across restarts the WAL grows monotonically and is only

-ever reset if a 12h periodic checkpoint happens to run. This is the reported

-"not reset / continuously grows after a start" behavior.

+### F2 — SEVERE: startup-error path skips graceful shutdown

+In `runServer` (`cmd/cc-backend/main.go:392-463`), if `srv.Start` returns an error

+(e.g. port in use, `tls.LoadX509KeyPair` failure, privileged-port bind), the error

+is delivered on `errChan`, the `select` returns it, `run()` returns, and `main()`

-Note on ordering: in `Shutdown()` the WAL staging goroutines are already stopped

-(channels closed + `WaitForWALStagingDrain()`) *before* the final

-`ToCheckpointWAL`. So the channel-based `RotateWALFiles` is unsafe here — that is

-exactly why the direct-removal `RotateWALFilesAfterShutdown` variant exists.

+- The graceful-shutdown sequence (`metricstore.Shutdown()` final checkpoint + WAL

+  rotation, `archiver.Shutdown()`, `taskmanager.Shutdown()`) lives only in the

+  signal-handler goroutine (`main.go:406-418`), which is unblocked via `ctx.Done()`

+  from the deferred `cancel()` (`main.go:584`) — but `os.Exit(1)` races ahead and

+  the process dies before that goroutine runs.

+- The metricstore background goroutines started in `metricstore.Init`

+  (`Retention`, `Checkpointing`, `WALStaging`, `ReceiveNats`) are tied to the

+  store's own internal context and are never cancelled on this path.

+Practical data-loss risk is low (the listener fails at startup before any ingest),

+but "graceful shutdown" is silently not performed on the error path, and it is a

+latent correctness gap if `Serve` ever returns mid-run.

-In `pkg/metricstore/metricstore.go`, `Shutdown()` (the `FileFormat == "wal"`

-branch, ~line 310): capture the `successDirs` return from `ToCheckpointWAL` and

-rotate those WAL files after the snapshot is durably written.

+**Fix direction**: on the error return from `runServer`, invoke the same shutdown

+sequence (or trigger the signal goroutine and wait for `waitDone`) before

+returning, so the final checkpoint runs.

-if Keys.Checkpoints.FileFormat == "wal" {

-    files, successDirs, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())

-    // Staging goroutines have already exited, so remove the now-redundant

-    // current.wal files directly (channel-based RotateWALFiles is unsafe here).

-    RotateWALFilesAfterShutdown(successDirs)

-    files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())

-Rotate **only `successDirs`** (hosts that received a fresh snapshot), mirroring the

-periodic path. A host that produced no snapshot (`ErrNoNewArchiveData`) had no new

-unarchived data for the window, so its WAL is left untouched to avoid dropping any

-record not yet captured by a snapshot. Remove/replace the misleading comment.

+## Lower-severity (worth fixing, not blocking)

-- `pkg/metricstore/metricstore.go` — `Shutdown()` WAL branch (only change needed).

+- **M1 — port-80 redirect goroutine swallows its error** (`server.go:417-419`):

+  `go func(){ http.ListenAndServe(":80", ...) }()` ignores the returned error; a

+  failed bind on :80 is completely silent. Log it.

+- **M2 — archiver shutdown leaks its waiter goroutine and skips a `Done()` on

+  timeout** (`internal/archiver/archiveWorker.go:222-249`, `:115-117`): on the

+  timeout branch `shutdownCancel()` makes the worker `return` without

+  `archivePending.Done()` for the in-flight job, so the

+  `go func(){ archivePending.Wait(); close(done) }()` goroutine never returns.

+  Only happens during shutdown timeout (process exiting), so impact is bounded.

+- **M3 — NATS API workers/channels not closed on shutdown**

+  (`internal/api/nats.go` — `jobWorker`/`nodeWorker` loop over `jobCh`/`nodeCh`,

+  never closed; created at `server.go:330-333`): goroutines block forever. This is

+  a one-time leak at process shutdown (not per-request/per-restart), so low impact,

+  but buffered job/node events in those channels are dropped silently.

+- **M4 — `metricstore.Shutdown()` is not idempotent** (`metricstore.go:274-321`):

+  `shutdownFunc` is not nil'd after use and a second call would re-run

+  `ToCheckpointWAL` + `RotateWALFilesAfterShutdown`. Currently only called once

-- `RotateWALFilesAfterShutdown([]string)` — `pkg/metricstore/walCheckpoint.go:406`

-  (already implemented; just needs to be wired in).

-- `successDirs` return of `ToCheckpointWAL` — `walCheckpoint.go:667`.

+## Remediation plan (recommended order)

+1. **F1** — initialize auth in the disabled case (call `auth.Init(nil)` on the

+   startup path when `DisableAuthentication` is true), then smoke-test

+   `-server -dev` with `disable-authentication: true`. Critical, small change in

+2. **F2** — run the graceful-shutdown sequence on the `runServer` error path before

+   returning. Small change in `cmd/cc-backend/main.go`.

+3. **M1–M4** — opportunistic hardening (log ignored errors; close NATS API

+   channels with a stop signal; balance archiver `Done()`; nil `shutdownFunc`).

+Each fix must respect the change-impact rules in CLAUDE.md (trace callers, check

-2. Targeted unit test (add to `pkg/metricstore`): write a few metrics so a host

-   WAL exists, run a checkpoint + `Shutdown()` equivalent, then assert

-   `current.wal` no longer exists (or is recreated empty) for hosts that were

-   snapshotted. Run `go test ./pkg/metricstore/...`.

-   - Configure `checkpoints.file-format: "wal"` (default) with a short

-     `checkpoint-interval` is *not* required for this test.

-   - Start `cc-backend`, ingest some metrics, confirm

-     `var/checkpoints/<cluster>/<host>/current.wal` grows.

-   - Stop the server (graceful shutdown) and confirm a new `<ts>.bin` appears and

-     `current.wal` is gone for active hosts.

-   - Restart, ingest more, stop again — confirm the WAL does not accumulate the

-     previous run's bytes (size resets each clean shutdown rather than growing

+- F1: start with a config containing `"disable-authentication": true` and

+  `-server -dev`; confirm the server starts and serves `/` without the

+  "Authentication module not initialized!" fatal. Re-confirm a normal

+  auth-enabled start still works (login flow).

+- F2: force a startup failure (e.g. bind an already-used `addr`, or point

+  `https-cert-file` at a missing file) and confirm the metricstore writes its

+  final checkpoint / rotates WAL before exit (check logs + `var/checkpoints`).

+- Regression: `go test ./...` (esp. `./pkg/metricstore/...`, `./internal/archiver/...`).

-## Out of scope / follow-ups (mention only)

-- The 12h default `checkpoint-interval` still bounds in-run WAL growth; operators

-  who want tighter bounds can lower it or set `checkpoints.max-wal-size`. No code

-  change proposed here unless requested.

+## Rejected during verification (false positives — do not chase)

+- **"DB migration runs before Connect(), causing multiple connections / lock

+  contention."** `handleDatabaseCommands` (`main.go:139-168`) only acts when

+  `-migrate-db`/`-revert-db`/`-force-db` is set, and each path ends in

+  `cclog.Exitf` (process exits). In normal server startup it is a no-op. Not an issue.

+- **"`GetUserRepository()`/`GetConnection()` called before `Connect()`."** Order in

+  `run()` is correct: `initDatabase()` (`main.go:512`) precedes both

+  `flagOptimizeDB` (`:518`) and `handleUserCommands` (`:531`).

+- **Archiver "double-Start", "TriggerArchiving before Start", "nil `workerDone`

+  panic".** `Start` is called exactly once (`main.go:377`) before any trigger; the

+  worker `Done()` accounting on the normal/error paths is actually correct

+  (`archiveWorker.go:128-168`). Only the timeout edge (M2) is real.

+- **"NatsAPI 10 goroutine leaks per restart cycle."** There is one process

+  lifetime per run; it is a single shutdown-time leak (M3), not per-restart.

+- **NATS-close-before-ctx-cancel "panic / mass data loss".** `nc.Close()`

+  (`server.go:461`) stops callbacks before `metricstore.Shutdown()` runs; the

+  `walShuttingDown` guard in `SendWALMessage` (`walCheckpoint.go:150-164`) prevents

+  send-on-closed-channel, and buffered `msgs` are drained when the ctx is cancelled.

# Fix: WAL files are never reset on shutdown (continuous growth across restarts)

## Context

The metric store (`pkg/metricstore`) uses a WAL + periodic binary-snapshot
checkpoint scheme for crash recovery. Each host has an append-only
`checkpoints/<cluster>/<host>/current.wal` that records every metric write, plus
periodic `<ts>.bin` full-tree snapshots. On startup the newest `.bin` is loaded
and `current.wal` is replayed on top.

A `current.wal` is only meant to hold writes *since the last snapshot*. Once a
snapshot is written, the WAL is redundant and must be rotated (deleted). The user
reports that WAL files are never reset and grow without bound after a start.

## Root cause

WAL rotation happens in exactly these places today:

1. **Periodic checkpoint** — `Checkpointing()` in `pkg/metricstore/checkpoint.go:128-145`
   writes snapshots via `ToCheckpointWAL`, then calls `RotateWALFiles(hostDirs)`
   for the hosts that got a fresh snapshot. This path works, but the default
   `checkpoint-interval` is **12h**, so the WAL grows for up to 12h between resets.
2. **Force-rotate on size** — in `WALStaging`/`processMsg`
   (`pkg/metricstore/walCheckpoint.go:255`) when `ws.size >= MaxWALSize`. But
   `MaxWALSize` (`config.go:69`) defaults to `0` = unlimited, so this never fires
   unless explicitly configured.
3. **Shutdown** — `Shutdown()` in `pkg/metricstore/metricstore.go:310-315`.

The **shutdown path is broken**. It writes a final snapshot but never rotates the
WAL:

```go
if Keys.Checkpoints.FileFormat == "wal" {
    // WAL files are deleted per-host inside ToCheckpointWAL workers.   <-- FALSE
    files, _, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
}
```

- The comment is incorrect: `ToCheckpointWAL` / `toCheckpointBinary`
  (`walCheckpoint.go:667`, `:768`) only write `.bin` snapshots; they do **not**
  touch `current.wal`.
- The second return value (`successDirs` — hosts that got a fresh snapshot) is
  discarded with `_`.
- The purpose-built helper `RotateWALFilesAfterShutdown(hostDirs)`
  (`walCheckpoint.go:406`), which directly `os.Remove`s `current.wal` files after
  the staging goroutines have exited, is **defined but never called** (verified by
  grep across `cmd/`, `internal/`, `pkg/`).

Consequence: after each clean shutdown the `current.wal` files remain on disk. On
the next startup they are replayed (redundantly, since the `.bin` already has that
data) and then **appended to again** (the staging goroutine opens the existing
file with `O_APPEND`). Across restarts the WAL grows monotonically and is only
ever reset if a 12h periodic checkpoint happens to run. This is the reported
"not reset / continuously grows after a start" behavior.

Note on ordering: in `Shutdown()` the WAL staging goroutines are already stopped
(channels closed + `WaitForWALStagingDrain()`) *before* the final
`ToCheckpointWAL`. So the channel-based `RotateWALFiles` is unsafe here — that is
exactly why the direct-removal `RotateWALFilesAfterShutdown` variant exists.

## Fix

In `pkg/metricstore/metricstore.go`, `Shutdown()` (the `FileFormat == "wal"`
branch, ~line 310): capture the `successDirs` return from `ToCheckpointWAL` and
rotate those WAL files after the snapshot is durably written.

```go
if Keys.Checkpoints.FileFormat == "wal" {
    var successDirs []string
    files, successDirs, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
    // Staging goroutines have already exited, so remove the now-redundant
    // current.wal files directly (channel-based RotateWALFiles is unsafe here).
    RotateWALFilesAfterShutdown(successDirs)
} else {
    files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
}
```

Rotate **only `successDirs`** (hosts that received a fresh snapshot), mirroring the
periodic path. A host that produced no snapshot (`ErrNoNewArchiveData`) had no new
unarchived data for the window, so its WAL is left untouched to avoid dropping any
record not yet captured by a snapshot. Remove/replace the misleading comment.

### Files
- `pkg/metricstore/metricstore.go` — `Shutdown()` WAL branch (only change needed).

### Reused existing code
- `RotateWALFilesAfterShutdown([]string)` — `pkg/metricstore/walCheckpoint.go:406`
  (already implemented; just needs to be wired in).
- `successDirs` return of `ToCheckpointWAL` — `walCheckpoint.go:667`.

## Verification

1. Build: `go build ./...`
2. Targeted unit test (add to `pkg/metricstore`): write a few metrics so a host
   WAL exists, run a checkpoint + `Shutdown()` equivalent, then assert
   `current.wal` no longer exists (or is recreated empty) for hosts that were
   snapshotted. Run `go test ./pkg/metricstore/...`.
3. Manual end-to-end:
   - Configure `checkpoints.file-format: "wal"` (default) with a short
     `checkpoint-interval` is *not* required for this test.
   - Start `cc-backend`, ingest some metrics, confirm
     `var/checkpoints/<cluster>/<host>/current.wal` grows.
   - Stop the server (graceful shutdown) and confirm a new `<ts>.bin` appears and
     `current.wal` is gone for active hosts.
   - Restart, ingest more, stop again — confirm the WAL does not accumulate the
     previous run's bytes (size resets each clean shutdown rather than growing
     monotonically).

## Out of scope / follow-ups (mention only)
- The 12h default `checkpoint-interval` still bounds in-run WAL growth; operators
  who want tighter bounds can lower it or set `checkpoints.max-wal-size`. No code
  change proposed here unless requested.


Do a thorough code  review with a focus on critical and severe issues. Initially focus on the initialization, startup and shutdown.


I completed the focused review. Here's the summary.

## Code review: init / startup / shutdown

I ran three exploration passes and then **verified every high-impact claim against the code** — the agents over-reported, so I've separated what's real from what isn't.

### 🔴 F1 — CRITICAL: `disable-authentication: true` aborts the server on startup
Auth is initialized **only when authentication is enabled** (`auth.Init` is inside `if !DisableAuthentication`, `main.go:175-181`), but the server **unconditionally** requires it:
- `server.go:123` → `auth.GetAuthInstance()`
- `server.go:184` → `api.New()` → `rest.go:72` → `auth.GetAuthInstance()`

`GetAuthInstance()` does `cclog.Fatal(...)` when the instance is nil (`auth.go:258-264`). So with `disable-authentication: true` + `-server`, the process hard-aborts. The flag is honored in 7 other places in `server.go`, so it's a supported-but-broken config. Default (auth-enabled) deployments are fine — which is why it slipped through.

### 🟠 F2 — SEVERE: startup-error path skips graceful shutdown
If `srv.Start` fails (port in use, bad TLS cert, privileged bind), `runServer` returns the error and `main()` calls `os.Exit(1)` — racing ahead of the signal-handler goroutine that performs `metricstore.Shutdown()` (final checkpoint + WAL rotation), `archiver.Shutdown()`, `taskmanager.Shutdown()`. So no graceful shutdown happens on the error path. Practical data loss is low (fails before ingest), but it's a real correctness gap.

### 🟡 Lower severity (hardening)
- **M1** `server.go:417-419` — :80 redirect goroutine swallows its `ListenAndServe` error (silent bind failure).
- **M2** `archiveWorker.go:115-117,222-249` — shutdown-timeout branch skips an `archivePending.Done()` and leaks the waiter goroutine (shutdown-only).
- **M3** `internal/api/nats.go` — NATS API workers/`jobCh`/`nodeCh` never closed; one-time leak + silent drop of buffered events at shutdown.
- **M4** `metricstore.go:274-321` — `Shutdown()` not idempotent (`shutdownFunc` not nil'd); latent, only called once today.

### Rejected as false positives (verified, don't chase)
DB-migration-before-Connect (flag-gated, exits), `GetUserRepository` before `Connect` (ordering is correct), archiver double-Start/nil-channel panics (Start called once), "10 goroutine leaks per restart" (one process lifetime), and NATS-close panic/mass-loss (guarded by `walShuttingDown`).

Full report with file:line evidence and a remediation plan is in the plan file.


ExitPlanMode

# Code Review — Initialization, Startup & Shutdown (critical/severe focus)

## Context

Requested: a thorough review of cc-backend focused on **critical and severe**
issues, starting with the initialization / startup / shutdown paths
(`cmd/cc-backend/main.go`, `cmd/cc-backend/server.go`, and the subsystem
`Init`/`Start`/`Shutdown` functions). Three exploration passes were run; their
findings were then **verified against the actual code** — many reported items were
theoretical or did not occur in the real call flow and are listed as rejected at
the bottom so they are not re-investigated.

This file is a review report plus a prioritized remediation plan. The previously
applied WAL-on-shutdown fix (`metricstore.Shutdown` now rotates WAL via
`RotateWALFilesAfterShutdown`) is already merged and is **not** re-listed.

---

## VERIFIED findings

### F1 — CRITICAL: `disable-authentication: true` aborts the server on startup
The server unconditionally requires an initialized auth instance, but auth is
only initialized when authentication is **enabled**.

- `auth.Init(...)` is called *only* inside the `if !config.Keys.DisableAuthentication`
  block — `cmd/cc-backend/main.go:175-181` (the only two call sites, confirmed by grep).
- `auth.GetAuthInstance()` is called **unconditionally** during server init:
  - `cmd/cc-backend/server.go:123` (`authHandle := auth.GetAuthInstance()`)
  - `internal/api/rest.go:72` via `api.New()` (`cmd/cc-backend/server.go:184`)
- `GetAuthInstance()` calls `cclog.Fatal("Authentication module not initialized!")`
  when `authInstance == nil` — `internal/auth/auth.go:258-264`.

Result: with `disable-authentication: true` and `-server`, the process hard-aborts
at startup. The flag is otherwise honored in 7 places in `server.go`, so it is a
supported configuration that is currently broken. Default deployments
(auth enabled) are unaffected, which is why it has gone unnoticed.

**Fix direction**: ensure auth is always initialized, or guard the unconditional
callers. Cleanest: when `config.Keys.DisableAuthentication` is true, still call
`auth.Init(nil)` (it sets up a session store and returns early without
authenticators — `auth.go:181-206`), OR make `GetAuthInstance()` callers tolerate
a nil instance when auth is disabled. Preferred: call `auth.Init(nil)` for the
disabled case in `handleUserCommands`/startup so `authInstance` is non-nil. Verify
no auth-enabled-only middleware is wired when disabled (it already is guarded).

### F2 — SEVERE: startup-error path skips graceful shutdown
In `runServer` (`cmd/cc-backend/main.go:392-463`), if `srv.Start` returns an error
(e.g. port in use, `tls.LoadX509KeyPair` failure, privileged-port bind), the error
is delivered on `errChan`, the `select` returns it, `run()` returns, and `main()`
calls `os.Exit(1)` (`main.go:589-594`).

- The graceful-shutdown sequence (`metricstore.Shutdown()` final checkpoint + WAL
  rotation, `archiver.Shutdown()`, `taskmanager.Shutdown()`) lives only in the
  signal-handler goroutine (`main.go:406-418`), which is unblocked via `ctx.Done()`
  from the deferred `cancel()` (`main.go:584`) — but `os.Exit(1)` races ahead and
  the process dies before that goroutine runs.
- The metricstore background goroutines started in `metricstore.Init`
  (`Retention`, `Checkpointing`, `WALStaging`, `ReceiveNats`) are tied to the
  store's own internal context and are never cancelled on this path.

Practical data-loss risk is low (the listener fails at startup before any ingest),
but "graceful shutdown" is silently not performed on the error path, and it is a
latent correctness gap if `Serve` ever returns mid-run.

**Fix direction**: on the error return from `runServer`, invoke the same shutdown
sequence (or trigger the signal goroutine and wait for `waitDone`) before
returning, so the final checkpoint runs.

---

## Lower-severity (worth fixing, not blocking)

- **M1 — port-80 redirect goroutine swallows its error** (`server.go:417-419`):
  `go func(){ http.ListenAndServe(":80", ...) }()` ignores the returned error; a
  failed bind on :80 is completely silent. Log it.
- **M2 — archiver shutdown leaks its waiter goroutine and skips a `Done()` on
  timeout** (`internal/archiver/archiveWorker.go:222-249`, `:115-117`): on the
  timeout branch `shutdownCancel()` makes the worker `return` without
  `archivePending.Done()` for the in-flight job, so the
  `go func(){ archivePending.Wait(); close(done) }()` goroutine never returns.
  Only happens during shutdown timeout (process exiting), so impact is bounded.
- **M3 — NATS API workers/channels not closed on shutdown**
  (`internal/api/nats.go` — `jobWorker`/`nodeWorker` loop over `jobCh`/`nodeCh`,
  never closed; created at `server.go:330-333`): goroutines block forever. This is
  a one-time leak at process shutdown (not per-request/per-restart), so low impact,
  but buffered job/node events in those channels are dropped silently.
- **M4 — `metricstore.Shutdown()` is not idempotent** (`metricstore.go:274-321`):
  `shutdownFunc` is not nil'd after use and a second call would re-run
  `ToCheckpointWAL` + `RotateWALFilesAfterShutdown`. Currently only called once
  (`server.go:484`), so latent only.

---

## Remediation plan (recommended order)

1. **F1** — initialize auth in the disabled case (call `auth.Init(nil)` on the
   startup path when `DisableAuthentication` is true), then smoke-test
   `-server -dev` with `disable-authentication: true`. Critical, small change in
   `cmd/cc-backend/main.go`.
2. **F2** — run the graceful-shutdown sequence on the `runServer` error path before
   returning. Small change in `cmd/cc-backend/main.go`.
3. **M1–M4** — opportunistic hardening (log ignored errors; close NATS API
   channels with a stop signal; balance archiver `Done()`; nil `shutdownFunc`).

Each fix must respect the change-impact rules in CLAUDE.md (trace callers, check
goroutine/channel lifecycle).

## Verification

- Build: `go build ./...`; `go vet ./...`.
- F1: start with a config containing `"disable-authentication": true` and
  `-server -dev`; confirm the server starts and serves `/` without the
  "Authentication module not initialized!" fatal. Re-confirm a normal
  auth-enabled start still works (login flow).
- F2: force a startup failure (e.g. bind an already-used `addr`, or point
  `https-cert-file` at a missing file) and confirm the metricstore writes its
  final checkpoint / rotates WAL before exit (check logs + `var/checkpoints`).
- Regression: `go test ./...` (esp. `./pkg/metricstore/...`, `./internal/archiver/...`).

---

## Rejected during verification (false positives — do not chase)

- **"DB migration runs before Connect(), causing multiple connections / lock
  contention."** `handleDatabaseCommands` (`main.go:139-168`) only acts when
  `-migrate-db`/`-revert-db`/`-force-db` is set, and each path ends in
  `cclog.Exitf` (process exits). In normal server startup it is a no-op. Not an issue.
- **"`GetUserRepository()`/`GetConnection()` called before `Connect()`."** Order in
  `run()` is correct: `initDatabase()` (`main.go:512`) precedes both
  `flagOptimizeDB` (`:518`) and `handleUserCommands` (`:531`).
- **Archiver "double-Start", "TriggerArchiving before Start", "nil `workerDone`
  panic".** `Start` is called exactly once (`main.go:377`) before any trigger; the
  worker `Done()` accounting on the normal/error paths is actually correct
  (`archiveWorker.go:128-168`). Only the timeout edge (M2) is real.
- **"NatsAPI 10 goroutine leaks per restart cycle."** There is one process
  lifetime per run; it is a single shutdown-time leak (M3), not per-restart.
- **NATS-close-before-ctx-cancel "panic / mass data loss".** `nc.Close()`
  (`server.go:461`) stops callbacks before `metricstore.Shutdown()` runs; the
  `walShuttingDown` guard in `SendWALMessage` (`walCheckpoint.go:150-164`) prevents
  send-on-closed-channel, and buffered `msgs` are drained when the ctx is cancelled.
  Minor at most, not critical.


Error: The user doesn't want to proceed with this tool use. The tool use was rejected (eg. if it was a file edit, the new_string was NOT written to the file). To tell you how to proceed, the user said:
Fix all issues in the suggested order


I'll fix the issues in order. Let me start with F1 and F2, then the hardening items. Let me read the exact code sections I need to modify.


1	// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
2	// All rights reserved. This file is part of cc-backend.
3	// Use of this source code is governed by a MIT-style
4	// license that can be found in the LICENSE file.
5
6	package api
7
8	import (
9		"database/sql"
10		"encoding/json"
11		"maps"
12		"strings"
13		"sync"
14		"time"
15
16		"github.com/ClusterCockpit/cc-backend/internal/archiver"
17		"github.com/ClusterCockpit/cc-backend/internal/config"
18		"github.com/ClusterCockpit/cc-backend/internal/importer"
19		"github.com/ClusterCockpit/cc-backend/internal/metricdispatch"
20		"github.com/ClusterCockpit/cc-backend/internal/repository"
21		"github.com/ClusterCockpit/cc-backend/pkg/archive"
22		"github.com/ClusterCockpit/cc-backend/pkg/metricstore"
23		cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
24		lp "github.com/ClusterCockpit/cc-lib/v2/ccMessage"
25		"github.com/ClusterCockpit/cc-lib/v2/nats"
26		"github.com/ClusterCockpit/cc-lib/v2/receivers"
27		"github.com/ClusterCockpit/cc-lib/v2/schema"
28		influx "github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
29	)
30
31	// natsMessage wraps a raw NATS message with its subject for channel-based processing.
32	type natsMessage struct {
33		subject string
34		data    []byte
35	}
36
37	// NatsAPI provides NATS subscription-based handlers for Job and Node operations.
38	// It mirrors the functionality of the REST API but uses NATS messaging with
39	// InfluxDB line protocol as the message format.
40	//
41	// # Message Format
42	//
43	// All NATS messages use InfluxDB line protocol format (https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/)
44	// with the following structure:
45	//
46	//	measurement,tag1=value1,tag2=value2 field1=value1,field2=value2 timestamp
47	//
48	// # Job Events
49	//
50	// Job start/stop events use the "job" measurement with a "function" tag to distinguish operations:
51	//
52	//	job,function=start_job event="{...JSON payload...}" <timestamp>
53	//	job,function=stop_job event="{...JSON payload...}" <timestamp>
54	//
55	// The JSON payload in the "event" field follows the schema.Job or StopJobAPIRequest structure.
56	//
57	// Example job start message:
58	//
59	//	job,function=start_job event="{\"jobId\":1001,\"user\":\"testuser\",\"cluster\":\"testcluster\",...}" 1234567890000000000
60	//
61	// # Node State Events
62	//
63	// Node state updates use the "nodestate" measurement with cluster information:
64	//
65	//	nodestate event="{...JSON payload...}" <timestamp>
66	//
67	// The JSON payload follows the UpdateNodeStatesRequest structure.
68	//
69	// Example node state message:
70	//
71	//	nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"node01\",\"states\":[\"idle\"]}]}" 1234567890000000000
72	type NatsAPI struct {
73		JobRepository *repository.JobRepository
74		// RepositoryMutex protects job creation operations from race conditions
75		// when checking for duplicate jobs during startJob calls.
76		RepositoryMutex sync.Mutex
77		// jobCh receives job event messages for processing by worker goroutines.
78		jobCh chan natsMessage
79		// nodeCh receives node state messages for processing by worker goroutines.
80		nodeCh chan natsMessage
81	}
82
83	// NewNatsAPI creates a new NatsAPI instance with channel-based worker pools.
84	// Concurrency is configured via NATSConfig (defaults: JobConcurrency=8, NodeConcurrency=2).
85	func NewNatsAPI() *NatsAPI {
86		jobConc := 8
87		nodeConc := 2
88
89		if s := config.Keys.APISubjects; s != nil {
90			if s.JobConcurrency > 0 {
91				jobConc = s.JobConcurrency
92			}
93			if s.NodeConcurrency > 0 {
94				nodeConc = s.NodeConcurrency
95			}
96		}
97
98		api := &NatsAPI{
99			JobRepository: repository.GetJobRepository(),
100			jobCh:         make(chan natsMessage, jobConc),
101			nodeCh:        make(chan natsMessage, nodeConc),
102		}
103
104		// Start worker goroutines
105		for range jobConc {
106			go api.jobWorker()
107		}
108		for range nodeConc {
109			go api.nodeWorker()
110		}
111
112		return api
113	}
114
115	// jobWorker processes job event messages from the job channel.
116	func (api *NatsAPI) jobWorker() {
117		for msg := range api.jobCh {
118			api.handleJobEvent(msg.subject, msg.data)
119		}
120	}
121
122	// nodeWorker processes node state messages from the node channel.
123	func (api *NatsAPI) nodeWorker() {
124		for msg := range api.nodeCh {
125			api.handleNodeState(msg.subject, msg.data)
126		}
127	}
128
129	// StartSubscriptions registers all NATS subscriptions for Job and Node APIs.
130	// Messages are delivered to buffered channels and processed by worker goroutines.
131	// Returns an error if the NATS client is not available or subscription fails.
132	func (api *NatsAPI) StartSubscriptions() error {
133		client := nats.GetClient()
134		if client == nil {
135			cclog.Warn("NATS client not available, skipping API subscriptions")
136			return nil
137		}
138
139		if config.Keys.APISubjects != nil {
140			s := config.Keys.APISubjects
141
142			if err := client.Subscribe(s.SubjectJobEvent, func(subject string, data []byte) {
143				api.jobCh <- natsMessage{subject: subject, data: data}
144			}); err != nil {
145				return err
146			}
147
148			if err := client.Subscribe(s.SubjectNodeState, func(subject string, data []byte) {
149				api.nodeCh <- natsMessage{subject: subject, data: data}
150			}); err != nil {
151				return err
152			}
153
154			cclog.Warnf("NATS API subscriptions started on subjects %q and %q — these are UNAUTHENTICATED: "+
155				"anyone with publish rights on the broker can start/stop jobs and update node state. "+
156				"Restrict publish ACLs on the NATS broker to trusted producers only.",
157				s.SubjectJobEvent, s.SubjectNodeState)
158		}
159		return nil
160	}
161
162	// processJobEvent routes job event messages to the appropriate handler based on the "function" tag.
163	// Validates that required tags and fields are present before processing.
164	func (api *NatsAPI) processJobEvent(msg lp.CCMessage) {
165		function, ok := msg.GetTag("function")
166		if !ok {
167			cclog.Errorf("Job event is missing required tag 'function': measurement=%s", msg.Name())
168			return
169		}
170
171		switch function {
172		case "start_job":
173			v, ok := msg.GetEventValue()
174			if !ok {
175				cclog.Errorf("Job start event is missing event field with JSON payload")

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package api

import (
	"database/sql"
	"encoding/json"
	"maps"
	"strings"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/archiver"
	"github.com/ClusterCockpit/cc-backend/internal/config"
	"github.com/ClusterCockpit/cc-backend/internal/importer"
	"github.com/ClusterCockpit/cc-backend/internal/metricdispatch"
	"github.com/ClusterCockpit/cc-backend/internal/repository"
	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	"github.com/ClusterCockpit/cc-backend/pkg/metricstore"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	lp "github.com/ClusterCockpit/cc-lib/v2/ccMessage"
	"github.com/ClusterCockpit/cc-lib/v2/nats"
	"github.com/ClusterCockpit/cc-lib/v2/receivers"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	influx "github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
)

// natsMessage wraps a raw NATS message with its subject for channel-based processing.
type natsMessage struct {
	subject string
	data    []byte
}

// NatsAPI provides NATS subscription-based handlers for Job and Node operations.
// It mirrors the functionality of the REST API but uses NATS messaging with
// InfluxDB line protocol as the message format.
//
// # Message Format
//
// All NATS messages use InfluxDB line protocol format (https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/)
// with the following structure:
//
//	measurement,tag1=value1,tag2=value2 field1=value1,field2=value2 timestamp
//
// # Job Events
//
// Job start/stop events use the "job" measurement with a "function" tag to distinguish operations:
//
//	job,function=start_job event="{...JSON payload...}" <timestamp>
//	job,function=stop_job event="{...JSON payload...}" <timestamp>
//
// The JSON payload in the "event" field follows the schema.Job or StopJobAPIRequest structure.
//
// Example job start message:
//
//	job,function=start_job event="{\"jobId\":1001,\"user\":\"testuser\",\"cluster\":\"testcluster\",...}" 1234567890000000000
//
// # Node State Events
//
// Node state updates use the "nodestate" measurement with cluster information:
//
//	nodestate event="{...JSON payload...}" <timestamp>
//
// The JSON payload follows the UpdateNodeStatesRequest structure.
//
// Example node state message:
//
//	nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"node01\",\"states\":[\"idle\"]}]}" 1234567890000000000
type NatsAPI struct {
	JobRepository *repository.JobRepository
	// RepositoryMutex protects job creation operations from race conditions
	// when checking for duplicate jobs during startJob calls.
	RepositoryMutex sync.Mutex
	// jobCh receives job event messages for processing by worker goroutines.
	jobCh chan natsMessage
	// nodeCh receives node state messages for processing by worker goroutines.
	nodeCh chan natsMessage
}

// NewNatsAPI creates a new NatsAPI instance with channel-based worker pools.
// Concurrency is configured via NATSConfig (defaults: JobConcurrency=8, NodeConcurrency=2).
func NewNatsAPI() *NatsAPI {
	jobConc := 8
	nodeConc := 2

	if s := config.Keys.APISubjects; s != nil {
		if s.JobConcurrency > 0 {
			jobConc = s.JobConcurrency
		}
		if s.NodeConcurrency > 0 {
			nodeConc = s.NodeConcurrency
		}
	}

	api := &NatsAPI{
		JobRepository: repository.GetJobRepository(),
		jobCh:         make(chan natsMessage, jobConc),
		nodeCh:        make(chan natsMessage, nodeConc),
	}

	// Start worker goroutines
	for range jobConc {
		go api.jobWorker()
	}
	for range nodeConc {
		go api.nodeWorker()
	}

	return api
}

// jobWorker processes job event messages from the job channel.
func (api *NatsAPI) jobWorker() {
	for msg := range api.jobCh {
		api.handleJobEvent(msg.subject, msg.data)
	}
}

// nodeWorker processes node state messages from the node channel.
func (api *NatsAPI) nodeWorker() {
	for msg := range api.nodeCh {
		api.handleNodeState(msg.subject, msg.data)
	}
}

// StartSubscriptions registers all NATS subscriptions for Job and Node APIs.
// Messages are delivered to buffered channels and processed by worker goroutines.
// Returns an error if the NATS client is not available or subscription fails.
func (api *NatsAPI) StartSubscriptions() error {
	client := nats.GetClient()
	if client == nil {
		cclog.Warn("NATS client not available, skipping API subscriptions")
		return nil
	}

	if config.Keys.APISubjects != nil {
		s := config.Keys.APISubjects

		if err := client.Subscribe(s.SubjectJobEvent, func(subject string, data []byte) {
			api.jobCh <- natsMessage{subject: subject, data: data}
		}); err != nil {
			return err
		}

		if err := client.Subscribe(s.SubjectNodeState, func(subject string, data []byte) {
			api.nodeCh <- natsMessage{subject: subject, data: data}
		}); err != nil {
			return err
		}

		cclog.Warnf("NATS API subscriptions started on subjects %q and %q — these are UNAUTHENTICATED: "+
			"anyone with publish rights on the broker can start/stop jobs and update node state. "+
			"Restrict publish ACLs on the NATS broker to trusted producers only.",
			s.SubjectJobEvent, s.SubjectNodeState)
	}
	return nil
}

// processJobEvent routes job event messages to the appropriate handler based on the "function" tag.
// Validates that required tags and fields are present before processing.
func (api *NatsAPI) processJobEvent(msg lp.CCMessage) {
	function, ok := msg.GetTag("function")
	if !ok {
		cclog.Errorf("Job event is missing required tag 'function': measurement=%s", msg.Name())
		return
	}

	switch function {
	case "start_job":
		v, ok := msg.GetEventValue()
		if !ok {
			cclog.Errorf("Job start event is missing event field with JSON payload")


Let me read the top of `auth.go` (imports + Keys) and `rest.go` New context to design F1 properly.


1	// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
2	// All rights reserved. This file is part of cc-backend.
3	// Use of this source code is governed by a MIT-style
4	// license that can be found in the LICENSE file.
5
6	// Package auth implements various authentication methods
7	package auth
8
9	import (
10		"bytes"
11		"context"
12		"database/sql"
13		"encoding/base64"
14		"encoding/json"
15		"errors"
16		"fmt"
17		"net"
18		"net/http"
19		"net/url"
20		"os"
21		"sync"
22		"time"
23
24		"golang.org/x/time/rate"
25
26		"github.com/ClusterCockpit/cc-backend/internal/config"
27		"github.com/ClusterCockpit/cc-backend/internal/repository"
28		cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
29		"github.com/ClusterCockpit/cc-lib/v2/schema"
30		"github.com/ClusterCockpit/cc-lib/v2/util"
31		"github.com/gorilla/sessions"
32	)
33
34	// Authenticator is the interface for all authentication methods.
35	// Each authenticator determines if it can handle a login request (CanLogin)
36	// and performs the actual authentication (Login).
37	type Authenticator interface {
38		// CanLogin determines if this authenticator can handle the login request.
39		// It returns the user object if available and a boolean indicating if this
40		// authenticator should attempt the login. This method should not perform
41		// expensive operations or actual authentication.
42		CanLogin(user *schema.User, username string, rw http.ResponseWriter, r *http.Request) (*schema.User, bool)
43
44		// Login performs the actually authentication for the user.
45		// It returns the authenticated user or an error if authentication fails.
46		// The user parameter may be nil if the user doesn't exist in the database yet.
47		Login(user *schema.User, rw http.ResponseWriter, r *http.Request) (*schema.User, error)
48	}
49
50	var (
51		initOnce     sync.Once
52		authInstance *Authentication
53	)
54
55	// rateLimiterEntry tracks a rate limiter and its last use time for cleanup
56	type rateLimiterEntry struct {
57		limiter  *rate.Limiter
58		lastUsed time.Time
59	}
60
61	var ipUserLimiters sync.Map
62
63	// getIPUserLimiter returns a rate limiter for the given IP and username combination.
64	// Rate limiters are created on demand and track 5 attempts per 15 minutes.
65	func getIPUserLimiter(ip, username string) *rate.Limiter {
66		key := ip + ":" + username
67		now := time.Now()
68
69		if entry, ok := ipUserLimiters.Load(key); ok {
70			rle := entry.(*rateLimiterEntry)
71			rle.lastUsed = now
72			return rle.limiter
73		}
74
75		// More aggressive rate limiting: 5 attempts per 15 minutes
76		newLimiter := rate.NewLimiter(rate.Every(15*time.Minute/5), 5)
77		ipUserLimiters.Store(key, &rateLimiterEntry{
78			limiter:  newLimiter,
79			lastUsed: now,
80		})
81		return newLimiter
82	}
83
84	// cleanupOldRateLimiters removes rate limiters that haven't been used recently
85	func cleanupOldRateLimiters(olderThan time.Time) {
86		ipUserLimiters.Range(func(key, value any) bool {
87			entry := value.(*rateLimiterEntry)
88			if entry.lastUsed.Before(olderThan) {
89				ipUserLimiters.Delete(key)
90				cclog.Debugf("Cleaned up rate limiter for %v", key)
91			}
92			return true
93		})
94	}
95
96	// startRateLimiterCleanup starts a background goroutine to clean up old rate limiters
97	func startRateLimiterCleanup() {
98		go func() {
99			ticker := time.NewTicker(1 * time.Hour)
100			defer ticker.Stop()
101			for range ticker.C {
102				// Clean up limiters not used in the last 24 hours
103				cleanupOldRateLimiters(time.Now().Add(-24 * time.Hour))
104			}
105		}()
106	}
107
108	// AuthConfig contains configuration for all authentication methods
109	type AuthConfig struct {
110		LdapConfig   *LdapConfig    `json:"ldap"`
111		JwtConfig    *JWTAuthConfig `json:"jwts"`
112		OpenIDConfig *OpenIDConfig  `json:"oidc"`
113	}
114
115	// Keys holds the global authentication configuration
116	var Keys AuthConfig
117
118	// Authentication manages all authentication methods and session handling
119	type Authentication struct {
120		sessionStore   *sessions.CookieStore

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package auth implements various authentication methods
package auth

import (
	"bytes"
	"context"
	"database/sql"
	"encoding/base64"
	"encoding/json"
	"errors"
	"fmt"
	"net"
	"net/http"
	"net/url"
	"os"
	"sync"
	"time"

	"golang.org/x/time/rate"

	"github.com/ClusterCockpit/cc-backend/internal/config"
	"github.com/ClusterCockpit/cc-backend/internal/repository"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	"github.com/ClusterCockpit/cc-lib/v2/util"
	"github.com/gorilla/sessions"
)

// Authenticator is the interface for all authentication methods.
// Each authenticator determines if it can handle a login request (CanLogin)
// and performs the actual authentication (Login).
type Authenticator interface {
	// CanLogin determines if this authenticator can handle the login request.
	// It returns the user object if available and a boolean indicating if this
	// authenticator should attempt the login. This method should not perform
	// expensive operations or actual authentication.
	CanLogin(user *schema.User, username string, rw http.ResponseWriter, r *http.Request) (*schema.User, bool)

	// Login performs the actually authentication for the user.
	// It returns the authenticated user or an error if authentication fails.
	// The user parameter may be nil if the user doesn't exist in the database yet.
	Login(user *schema.User, rw http.ResponseWriter, r *http.Request) (*schema.User, error)
}

var (
	initOnce     sync.Once
	authInstance *Authentication
)

// rateLimiterEntry tracks a rate limiter and its last use time for cleanup
type rateLimiterEntry struct {
	limiter  *rate.Limiter
	lastUsed time.Time
}

var ipUserLimiters sync.Map

// getIPUserLimiter returns a rate limiter for the given IP and username combination.
// Rate limiters are created on demand and track 5 attempts per 15 minutes.
func getIPUserLimiter(ip, username string) *rate.Limiter {
	key := ip + ":" + username
	now := time.Now()

	if entry, ok := ipUserLimiters.Load(key); ok {
		rle := entry.(*rateLimiterEntry)
		rle.lastUsed = now
		return rle.limiter
	}

	// More aggressive rate limiting: 5 attempts per 15 minutes
	newLimiter := rate.NewLimiter(rate.Every(15*time.Minute/5), 5)
	ipUserLimiters.Store(key, &rateLimiterEntry{
		limiter:  newLimiter,
		lastUsed: now,
	})
	return newLimiter
}

// cleanupOldRateLimiters removes rate limiters that haven't been used recently
func cleanupOldRateLimiters(olderThan time.Time) {
	ipUserLimiters.Range(func(key, value any) bool {
		entry := value.(*rateLimiterEntry)
		if entry.lastUsed.Before(olderThan) {
			ipUserLimiters.Delete(key)
			cclog.Debugf("Cleaned up rate limiter for %v", key)
		}
		return true
	})
}

// startRateLimiterCleanup starts a background goroutine to clean up old rate limiters
func startRateLimiterCleanup() {
	go func() {
		ticker := time.NewTicker(1 * time.Hour)
		defer ticker.Stop()
		for range ticker.C {
			// Clean up limiters not used in the last 24 hours
			cleanupOldRateLimiters(time.Now().Add(-24 * time.Hour))
		}
	}()
}

// AuthConfig contains configuration for all authentication methods
type AuthConfig struct {
	LdapConfig   *LdapConfig    `json:"ldap"`
	JwtConfig    *JWTAuthConfig `json:"jwts"`
	OpenIDConfig *OpenIDConfig  `json:"oidc"`
}

// Keys holds the global authentication configuration
var Keys AuthConfig

// Authentication manages all authentication methods and session handling
type Authentication struct {
	sessionStore   *sessions.CookieStore


Now I'll implement all fixes in order. Let me set up tracking and start with F1.


ToolSearch

Question: select:TaskCreate


Question: select:TaskCreate


I'll proceed directly with the edits. Starting with **F1** (auth.go):


Edit

import (
	"bytes"
	"context"
	"database/sql"
	"encoding/base64"
	"encoding/json"
	"errors"
	"fmt"
	"net"

import (
	"bytes"
	"context"
	"crypto/rand"
	"database/sql"
	"encoding/base64"
	"encoding/json"
	"errors"
	"fmt"
	"net"


The file /Users/jan/prg/ClusterCockpit/cc-backend/internal/auth/auth.go has been updated successfully. (file state is current in your context — no need to Read it back)

import (
	"bytes"
	"context"
	"database/sql"
	"encoding/base64"
	"encoding/json"
	"errors"
	"fmt"
	"net"

import (
	"bytes"
	"context"
	"crypto/rand"
	"database/sql"
	"encoding/base64"
	"encoding/json"
	"errors"
	"fmt"
	"net"

import (
"bytes"
"context"
+  "crypto/rand"
"database/sql"
"encoding/base64"
"encoding/json"


		sessKey := os.Getenv("SESSION_KEY")
		if sessKey == "" {
			cclog.Fatal("environment variable 'SESSION_KEY' not set: refusing to start with an ephemeral session key. " +
				"Set SESSION_KEY in .env (base64-encoded 32 random bytes); a random key would invalidate all sessions on every restart " +
				"and prevent sessions from validating across replicas.")
		}
		keyBytes, err := base64.StdEncoding.DecodeString(sessKey)
		if err != nil {
			cclog.Fatal("Error while initializing authentication -> decoding session key failed")
		}
		authInstance.sessionStore = sessions.NewCookieStore(keyBytes)

		if d, err := time.ParseDuration(config.Keys.SessionMaxAge); err == nil {
			authInstance.SessionMaxAge = d
		}

		if authCfg == nil {
			return
		}

		sessKey := os.Getenv("SESSION_KEY")
		if sessKey == "" {
			if !config.Keys.DisableAuthentication {
				cclog.Fatal("environment variable 'SESSION_KEY' not set: refusing to start with an ephemeral session key. " +
					"Set SESSION_KEY in .env (base64-encoded 32 random bytes); a random key would invalidate all sessions on every restart " +
					"and prevent sessions from validating across replicas.")
			}
			// Authentication is disabled: no user sessions are issued, so an
			// ephemeral random key is sufficient and SESSION_KEY is not required.
			ephemeralKey := make([]byte, 32)
			if _, err := rand.Read(ephemeralKey); err != nil {
				cclog.Fatalf("Error while initializing authentication -> generating ephemeral session key failed: %v", err)
			}
			authInstance.sessionStore = sessions.NewCookieStore(ephemeralKey)
		} else {
			keyBytes, err := base64.StdEncoding.DecodeString(sessKey)
			if err != nil {
				cclog.Fatal("Error while initializing authentication -> decoding session key failed")
			}
			authInstance.sessionStore = sessions.NewCookieStore(keyBytes)
		}

		if d, err := time.ParseDuration(config.Keys.SessionMaxAge); err == nil {
			authInstance.SessionMaxAge = d
		}

		// When authentication is disabled no authenticators are required; the
		// session store created above is enough for the server to run with a
		// valid (non-nil) auth instance.
		if config.Keys.DisableAuthentication {
			return
		}

		if authCfg == nil {
			return
		}


		sessKey := os.Getenv("SESSION_KEY")
		if sessKey == "" {
			cclog.Fatal("environment variable 'SESSION_KEY' not set: refusing to start with an ephemeral session key. " +
				"Set SESSION_KEY in .env (base64-encoded 32 random bytes); a random key would invalidate all sessions on every restart " +
				"and prevent sessions from validating across replicas.")
		}
		keyBytes, err := base64.StdEncoding.DecodeString(sessKey)
		if err != nil {
			cclog.Fatal("Error while initializing authentication -> decoding session key failed")
		}
		authInstance.sessionStore = sessions.NewCookieStore(keyBytes)

		if d, err := time.ParseDuration(config.Keys.SessionMaxAge); err == nil {
			authInstance.SessionMaxAge = d
		}

		if authCfg == nil {
			return
		}

		sessKey := os.Getenv("SESSION_KEY")
		if sessKey == "" {
			if !config.Keys.DisableAuthentication {
				cclog.Fatal("environment variable 'SESSION_KEY' not set: refusing to start with an ephemeral session key. " +
					"Set SESSION_KEY in .env (base64-encoded 32 random bytes); a random key would invalidate all sessions on every restart " +
					"and prevent sessions from validating across replicas.")
			}
			// Authentication is disabled: no user sessions are issued, so an
			// ephemeral random key is sufficient and SESSION_KEY is not required.
			ephemeralKey := make([]byte, 32)
			if _, err := rand.Read(ephemeralKey); err != nil {
				cclog.Fatalf("Error while initializing authentication -> generating ephemeral session key failed: %v", err)
			}
			authInstance.sessionStore = sessions.NewCookieStore(ephemeralKey)
		} else {
			keyBytes, err := base64.StdEncoding.DecodeString(sessKey)
			if err != nil {
				cclog.Fatal("Error while initializing authentication -> decoding session key failed")
			}
			authInstance.sessionStore = sessions.NewCookieStore(keyBytes)
		}

		if d, err := time.ParseDuration(config.Keys.SessionMaxAge); err == nil {
			authInstance.SessionMaxAge = d
		}

		// When authentication is disabled no authenticators are required; the
		// session store created above is enough for the server to run with a
		// valid (non-nil) auth instance.
		if config.Keys.DisableAuthentication {
			return
		}

		if authCfg == nil {
			return
		}

-      cclog.Fatal("environment variable 'SESSION_KEY' not set: refusing to start with an ephemeral session key. " +

-        "Set SESSION_KEY in .env (base64-encoded 32 random bytes); a random key would invalidate all sessions on every restart " +

-        "and prevent sessions from validating across replicas.")

+      if !config.Keys.DisableAuthentication {

+        cclog.Fatal("environment variable 'SESSION_KEY' not set: refusing to start with an ephemeral session key. " +

+      // Authentication is disabled: no user sessions are issued, so an

+      // ephemeral random key is sufficient and SESSION_KEY is not required.

+      if _, err := rand.Read(ephemeralKey); err != nil {

+        cclog.Fatalf("Error while initializing authentication -> generating ephemeral session key failed: %v", err)

+      authInstance.sessionStore = sessions.NewCookieStore(ephemeralKey)

+      keyBytes, err := base64.StdEncoding.DecodeString(sessKey)

+        cclog.Fatal("Error while initializing authentication -> decoding session key failed")

+      authInstance.sessionStore = sessions.NewCookieStore(keyBytes)

-    keyBytes, err := base64.StdEncoding.DecodeString(sessKey)

-      cclog.Fatal("Error while initializing authentication -> decoding session key failed")

-    authInstance.sessionStore = sessions.NewCookieStore(keyBytes)

+    // When authentication is disabled no authenticators are required; the

+    // session store created above is enough for the server to run with a

+    if config.Keys.DisableAuthentication {


Now **F1 part B** — always initialize auth in `main.go`:


	if !config.Keys.DisableAuthentication {
		if cfg := ccconf.GetPackageConfig("auth"); cfg != nil {
			auth.Init(&cfg)
		} else {
			cclog.Warn("Authentication disabled due to missing configuration")
			auth.Init(nil)
		}

		// Check for default security keys
		checkDefaultSecurityKeys()

		if flagNewUser != "" {

	// Always initialize the auth subsystem so the HTTP server and REST API have a
	// valid (non-nil) auth instance, even when authentication is disabled. With
	// authentication disabled, Init only sets up an ephemeral session store and
	// registers no authenticators (see auth.Init).
	if cfg := ccconf.GetPackageConfig("auth"); cfg != nil {
		auth.Init(&cfg)
	} else {
		if !config.Keys.DisableAuthentication {
			cclog.Warn("Authentication enabled but no auth configuration found")
		}
		auth.Init(nil)
	}

	if !config.Keys.DisableAuthentication {
		// Check for default security keys
		checkDefaultSecurityKeys()

		if flagNewUser != "" {


The file /Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/main.go has been updated successfully. (file state is current in your context — no need to Read it back)

	if !config.Keys.DisableAuthentication {
		if cfg := ccconf.GetPackageConfig("auth"); cfg != nil {
			auth.Init(&cfg)
		} else {
			cclog.Warn("Authentication disabled due to missing configuration")
			auth.Init(nil)
		}

		// Check for default security keys
		checkDefaultSecurityKeys()

		if flagNewUser != "" {

	// Always initialize the auth subsystem so the HTTP server and REST API have a
	// valid (non-nil) auth instance, even when authentication is disabled. With
	// authentication disabled, Init only sets up an ephemeral session store and
	// registers no authenticators (see auth.Init).
	if cfg := ccconf.GetPackageConfig("auth"); cfg != nil {
		auth.Init(&cfg)
	} else {
		if !config.Keys.DisableAuthentication {
			cclog.Warn("Authentication enabled but no auth configuration found")
		}
		auth.Init(nil)
	}

	if !config.Keys.DisableAuthentication {
		// Check for default security keys
		checkDefaultSecurityKeys()

		if flagNewUser != "" {

return fmt.Errorf("--add-user and --del-user can only be used if authentication is enabled")

-  if !config.Keys.DisableAuthentication {

-    if cfg := ccconf.GetPackageConfig("auth"); cfg != nil {

-      cclog.Warn("Authentication disabled due to missing configuration")

+  // Always initialize the auth subsystem so the HTTP server and REST API have a

+  // valid (non-nil) auth instance, even when authentication is disabled. With

+  // authentication disabled, Init only sets up an ephemeral session store and

+  // registers no authenticators (see auth.Init).

+  if cfg := ccconf.GetPackageConfig("auth"); cfg != nil {

+    if !config.Keys.DisableAuthentication {

+      cclog.Warn("Authentication enabled but no auth configuration found")


Verify the F1 block structure is intact, then apply **F2** (graceful shutdown on startup error in `runServer`):


170	func handleUserCommands() error {
171		if config.Keys.DisableAuthentication && (flagNewUser != "" || flagDelUser != "") {
172			return fmt.Errorf("--add-user and --del-user can only be used if authentication is enabled")
173		}
174
175		// Always initialize the auth subsystem so the HTTP server and REST API have a
176		// valid (non-nil) auth instance, even when authentication is disabled. With
177		// authentication disabled, Init only sets up an ephemeral session store and
178		// registers no authenticators (see auth.Init).
179		if cfg := ccconf.GetPackageConfig("auth"); cfg != nil {
180			auth.Init(&cfg)
181		} else {
182			if !config.Keys.DisableAuthentication {
183				cclog.Warn("Authentication enabled but no auth configuration found")
184			}
185			auth.Init(nil)
186		}
187
188		if !config.Keys.DisableAuthentication {
189			// Check for default security keys
190			checkDefaultSecurityKeys()
191
192			if flagNewUser != "" {
193				if err := addUser(flagNewUser); err != nil {
194					return err

func handleUserCommands() error {
	if config.Keys.DisableAuthentication && (flagNewUser != "" || flagDelUser != "") {
		return fmt.Errorf("--add-user and --del-user can only be used if authentication is enabled")
	}

	// Always initialize the auth subsystem so the HTTP server and REST API have a
	// valid (non-nil) auth instance, even when authentication is disabled. With
	// authentication disabled, Init only sets up an ephemeral session store and
	// registers no authenticators (see auth.Init).
	if cfg := ccconf.GetPackageConfig("auth"); cfg != nil {
		auth.Init(&cfg)
	} else {
		if !config.Keys.DisableAuthentication {
			cclog.Warn("Authentication enabled but no auth configuration found")
		}
		auth.Init(nil)
	}

	if !config.Keys.DisableAuthentication {
		// Check for default security keys
		checkDefaultSecurityKeys()

		if flagNewUser != "" {
			if err := addUser(flagNewUser); err != nil {
				return err


F1 looks correct. Now **F2** — add a cancelable context to `runServer` and restructure the wait logic:


func runServer(ctx context.Context) error {
	var wg sync.WaitGroup

func runServer(ctx context.Context) error {
	// Derive a cancelable context so the startup-error path below can trigger the
	// same graceful-shutdown sequence as a signal (via the signal handler that
	// waits on ctx.Done()).
	ctx, cancel := context.WithCancel(ctx)
	defer cancel()

	var wg sync.WaitGroup


func runServer(ctx context.Context) error {
	var wg sync.WaitGroup

func runServer(ctx context.Context) error {
	// Derive a cancelable context so the startup-error path below can trigger the
	// same graceful-shutdown sequence as a signal (via the signal handler that
	// waits on ctx.Done()).
	ctx, cancel := context.WithCancel(ctx)
	defer cancel()

	var wg sync.WaitGroup

+  // Derive a cancelable context so the startup-error path below can trigger the

+  // same graceful-shutdown sequence as a signal (via the signal handler that

+  ctx, cancel := context.WithCancel(ctx)

// Initialize metric store if configuration is provided


	// Wait for either:
	// 1. An error from server startup
	// 2. Completion of all goroutines (normal shutdown or crash)
	select {
	case err := <-errChan:
		// errChan will be closed when waitDone is closed, which happens
		// when all goroutines complete (either from normal shutdown or error)
		if err != nil {
			return err
		}
	case <-time.After(100 * time.Millisecond):
		// Give the server 100ms to start and report any immediate startup errors
		// After that, just wait for normal shutdown completion
		select {
		case err := <-errChan:
			if err != nil {
				return err
			}
		case <-waitDone:
			// Normal shutdown completed
		}
	}

	cclog.Print("Graceful shutdown completed!")
	return nil
}

	// Wait for either:
	// 1. An error from server startup
	// 2. Completion of all goroutines (normal shutdown or crash)
	var runErr error
	select {
	case runErr = <-errChan:
		// errChan will be closed when waitDone is closed, which happens
		// when all goroutines complete (either from normal shutdown or error)
	case <-time.After(100 * time.Millisecond):
		// Give the server 100ms to start and report any immediate startup errors
		// After that, just wait for normal shutdown completion
		select {
		case runErr = <-errChan:
		case <-waitDone:
			// Normal shutdown completed
		}
	}

	if runErr != nil {
		// A subsystem failed (e.g. the HTTP server could not bind). Trigger the
		// graceful-shutdown path for the subsystems that were already started
		// (metricstore checkpoint, archiver flush, taskmanager) by cancelling the
		// context the signal handler waits on, then wait for it to finish so we
		// don't exit before the final checkpoint is written.
		cancel()
		<-waitDone
		return runErr
	}

	cclog.Print("Graceful shutdown completed!")
	return nil
}


	// Wait for either:
	// 1. An error from server startup
	// 2. Completion of all goroutines (normal shutdown or crash)
	select {
	case err := <-errChan:
		// errChan will be closed when waitDone is closed, which happens
		// when all goroutines complete (either from normal shutdown or error)
		if err != nil {
			return err
		}
	case <-time.After(100 * time.Millisecond):
		// Give the server 100ms to start and report any immediate startup errors
		// After that, just wait for normal shutdown completion
		select {
		case err := <-errChan:
			if err != nil {
				return err
			}
		case <-waitDone:
			// Normal shutdown completed
		}
	}

	cclog.Print("Graceful shutdown completed!")
	return nil
}

	// Wait for either:
	// 1. An error from server startup
	// 2. Completion of all goroutines (normal shutdown or crash)
	var runErr error
	select {
	case runErr = <-errChan:
		// errChan will be closed when waitDone is closed, which happens
		// when all goroutines complete (either from normal shutdown or error)
	case <-time.After(100 * time.Millisecond):
		// Give the server 100ms to start and report any immediate startup errors
		// After that, just wait for normal shutdown completion
		select {
		case runErr = <-errChan:
		case <-waitDone:
			// Normal shutdown completed
		}
	}

	if runErr != nil {
		// A subsystem failed (e.g. the HTTP server could not bind). Trigger the
		// graceful-shutdown path for the subsystems that were already started
		// (metricstore checkpoint, archiver flush, taskmanager) by cancelling the
		// context the signal handler waits on, then wait for it to finish so we
		// don't exit before the final checkpoint is written.
		cancel()
		<-waitDone
		return runErr
	}

	cclog.Print("Graceful shutdown completed!")
	return nil
}

+    // A subsystem failed (e.g. the HTTP server could not bind). Trigger the

+    // graceful-shutdown path for the subsystems that were already started

+    // (metricstore checkpoint, archiver flush, taskmanager) by cancelling the

+    // context the signal handler waits on, then wait for it to finish so we

+    // don't exit before the final checkpoint is written.


Now **M1** (log :80 redirect error) and **M3** (wire NATS API shutdown) in `server.go`:


	if !strings.HasSuffix(config.Keys.Addr, ":80") && config.Keys.RedirectHTTPTo != "" {
		go func() {
			http.ListenAndServe(":80", http.RedirectHandler(config.Keys.RedirectHTTPTo, http.StatusMovedPermanently))
		}()
	}

	if !strings.HasSuffix(config.Keys.Addr, ":80") && config.Keys.RedirectHTTPTo != "" {
		go func() {
			if err := http.ListenAndServe(":80", http.RedirectHandler(config.Keys.RedirectHTTPTo, http.StatusMovedPermanently)); err != nil {
				cclog.Errorf("HTTP-to-HTTPS redirect listener on :80 failed: %v", err)
			}
		}()
	}


The file /Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/server.go has been updated successfully. (file state is current in your context — no need to Read it back)

	if !strings.HasSuffix(config.Keys.Addr, ":80") && config.Keys.RedirectHTTPTo != "" {
		go func() {
			http.ListenAndServe(":80", http.RedirectHandler(config.Keys.RedirectHTTPTo, http.StatusMovedPermanently))
		}()
	}

	if !strings.HasSuffix(config.Keys.Addr, ":80") && config.Keys.RedirectHTTPTo != "" {
		go func() {
			if err := http.ListenAndServe(":80", http.RedirectHandler(config.Keys.RedirectHTTPTo, http.StatusMovedPermanently)); err != nil {
				cclog.Errorf("HTTP-to-HTTPS redirect listener on :80 failed: %v", err)
			}
		}()
	}

-      http.ListenAndServe(":80", http.RedirectHandler(config.Keys.RedirectHTTPTo, http.StatusMovedPermanently))

+      if err := http.ListenAndServe(":80", http.RedirectHandler(config.Keys.RedirectHTTPTo, http.StatusMovedPermanently)); err != nil {

+        cclog.Errorf("HTTP-to-HTTPS redirect listener on :80 failed: %v", err)


	natsStart := time.Now()
	nc := nats.GetClient()
	if nc != nil {
		nc.Close()
	}
	cclog.Infof("Shutdown: NATS closed (%v)", time.Since(natsStart))

	natsStart := time.Now()
	nc := nats.GetClient()
	if nc != nil {
		nc.Close()
	}
	// Stop the NATS API worker goroutines after the client is closed (no more
	// subscription callbacks can enqueue once the connection is down).
	if s.natsAPIHandle != nil {
		s.natsAPIHandle.Shutdown()
	}
	cclog.Infof("Shutdown: NATS closed (%v)", time.Since(natsStart))


	natsStart := time.Now()
	nc := nats.GetClient()
	if nc != nil {
		nc.Close()
	}
	cclog.Infof("Shutdown: NATS closed (%v)", time.Since(natsStart))

	natsStart := time.Now()
	nc := nats.GetClient()
	if nc != nil {
		nc.Close()
	}
	// Stop the NATS API worker goroutines after the client is closed (no more
	// subscription callbacks can enqueue once the connection is down).
	if s.natsAPIHandle != nil {
		s.natsAPIHandle.Shutdown()
	}
	cclog.Infof("Shutdown: NATS closed (%v)", time.Since(natsStart))

+  // Stop the NATS API worker goroutines after the client is closed (no more

+  // subscription callbacks can enqueue once the connection is down).


Now **M3** in `nats.go` — add a stop channel, make workers/callbacks respect it, and add `Shutdown()`:


	// jobCh receives job event messages for processing by worker goroutines.
	jobCh chan natsMessage
	// nodeCh receives node state messages for processing by worker goroutines.
	nodeCh chan natsMessage
}

	// jobCh receives job event messages for processing by worker goroutines.
	jobCh chan natsMessage
	// nodeCh receives node state messages for processing by worker goroutines.
	nodeCh chan natsMessage
	// stop signals worker goroutines and subscription callbacks to stop.
	// Closing it (via Shutdown) makes workers exit and callbacks drop further
	// messages instead of blocking; the channels are never closed so in-flight
	// callbacks can never send on a closed channel.
	stop     chan struct{}
	stopOnce sync.Once
}


The file /Users/jan/prg/ClusterCockpit/cc-backend/internal/api/nats.go has been updated successfully. (file state is current in your context — no need to Read it back)

	// jobCh receives job event messages for processing by worker goroutines.
	jobCh chan natsMessage
	// nodeCh receives node state messages for processing by worker goroutines.
	nodeCh chan natsMessage
}

	// jobCh receives job event messages for processing by worker goroutines.
	jobCh chan natsMessage
	// nodeCh receives node state messages for processing by worker goroutines.
	nodeCh chan natsMessage
	// stop signals worker goroutines and subscription callbacks to stop.
	// Closing it (via Shutdown) makes workers exit and callbacks drop further
	// messages instead of blocking; the channels are never closed so in-flight
	// callbacks can never send on a closed channel.
	stop     chan struct{}
	stopOnce sync.Once
}

+  // stop signals worker goroutines and subscription callbacks to stop.

+  // Closing it (via Shutdown) makes workers exit and callbacks drop further

+  // messages instead of blocking; the channels are never closed so in-flight

+  // callbacks can never send on a closed channel.

// NewNatsAPI creates a new NatsAPI instance with channel-based worker pools.


s.natsAPIHandle.Shutdown undefined (type *api.NatsAPI has no field or method Shutdown)


	api := &NatsAPI{
		JobRepository: repository.GetJobRepository(),
		jobCh:         make(chan natsMessage, jobConc),
		nodeCh:        make(chan natsMessage, nodeConc),
	}

	// Start worker goroutines
	for range jobConc {
		go api.jobWorker()
	}
	for range nodeConc {
		go api.nodeWorker()
	}

	return api
}

// jobWorker processes job event messages from the job channel.
func (api *NatsAPI) jobWorker() {
	for msg := range api.jobCh {
		api.handleJobEvent(msg.subject, msg.data)
	}
}

// nodeWorker processes node state messages from the node channel.
func (api *NatsAPI) nodeWorker() {
	for msg := range api.nodeCh {
		api.handleNodeState(msg.subject, msg.data)
	}
}

	api := &NatsAPI{
		JobRepository: repository.GetJobRepository(),
		jobCh:         make(chan natsMessage, jobConc),
		nodeCh:        make(chan natsMessage, nodeConc),
		stop:          make(chan struct{}),
	}

	// Start worker goroutines
	for range jobConc {
		go api.jobWorker()
	}
	for range nodeConc {
		go api.nodeWorker()
	}

	return api
}

// Shutdown stops the worker goroutines and tells subscription callbacks to stop
// enqueueing. It is safe to call multiple times. Callers must ensure the NATS
// client is closed first so no new callbacks are invoked.
func (api *NatsAPI) Shutdown() {
	api.stopOnce.Do(func() {
		close(api.stop)
	})
}

// jobWorker processes job event messages from the job channel.
func (api *NatsAPI) jobWorker() {
	for {
		select {
		case <-api.stop:
			return
		case msg := <-api.jobCh:
			api.handleJobEvent(msg.subject, msg.data)
		}
	}
}

// nodeWorker processes node state messages from the node channel.
func (api *NatsAPI) nodeWorker() {
	for {
		select {
		case <-api.stop:
			return
		case msg := <-api.nodeCh:
			api.handleNodeState(msg.subject, msg.data)
		}
	}
}


	api := &NatsAPI{
		JobRepository: repository.GetJobRepository(),
		jobCh:         make(chan natsMessage, jobConc),
		nodeCh:        make(chan natsMessage, nodeConc),
	}

	// Start worker goroutines
	for range jobConc {
		go api.jobWorker()
	}
	for range nodeConc {
		go api.nodeWorker()
	}

	return api
}

// jobWorker processes job event messages from the job channel.
func (api *NatsAPI) jobWorker() {
	for msg := range api.jobCh {
		api.handleJobEvent(msg.subject, msg.data)
	}
}

// nodeWorker processes node state messages from the node channel.
func (api *NatsAPI) nodeWorker() {
	for msg := range api.nodeCh {
		api.handleNodeState(msg.subject, msg.data)
	}
}

	api := &NatsAPI{
		JobRepository: repository.GetJobRepository(),
		jobCh:         make(chan natsMessage, jobConc),
		nodeCh:        make(chan natsMessage, nodeConc),
		stop:          make(chan struct{}),
	}

	// Start worker goroutines
	for range jobConc {
		go api.jobWorker()
	}
	for range nodeConc {
		go api.nodeWorker()
	}

	return api
}

// Shutdown stops the worker goroutines and tells subscription callbacks to stop
// enqueueing. It is safe to call multiple times. Callers must ensure the NATS
// client is closed first so no new callbacks are invoked.
func (api *NatsAPI) Shutdown() {
	api.stopOnce.Do(func() {
		close(api.stop)
	})
}

// jobWorker processes job event messages from the job channel.
func (api *NatsAPI) jobWorker() {
	for {
		select {
		case <-api.stop:
			return
		case msg := <-api.jobCh:
			api.handleJobEvent(msg.subject, msg.data)
		}
	}
}

// nodeWorker processes node state messages from the node channel.
func (api *NatsAPI) nodeWorker() {
	for {
		select {
		case <-api.stop:
			return
		case msg := <-api.nodeCh:
			api.handleNodeState(msg.subject, msg.data)
		}
	}
}

jobCh:         make(chan natsMessage, jobConc),

+    stop:          make(chan struct{}),

+// Shutdown stops the worker goroutines and tells subscription callbacks to stop

+// enqueueing. It is safe to call multiple times. Callers must ensure the NATS

+// client is closed first so no new callbacks are invoked.

-    api.handleJobEvent(msg.subject, msg.data)

+      api.handleJobEvent(msg.subject, msg.data)

-    api.handleNodeState(msg.subject, msg.data)

+      api.handleNodeState(msg.subject, msg.data)


bufio.Scanner "scanner" is used in Scan loop at line 106 without final check of scanner.Err()


Now update the subscription callbacks to respect `stop`:


		if err := client.Subscribe(s.SubjectJobEvent, func(subject string, data []byte) {
			api.jobCh <- natsMessage{subject: subject, data: data}
		}); err != nil {
			return err
		}

		if err := client.Subscribe(s.SubjectNodeState, func(subject string, data []byte) {
			api.nodeCh <- natsMessage{subject: subject, data: data}
		}); err != nil {
			return err
		}

		if err := client.Subscribe(s.SubjectJobEvent, func(subject string, data []byte) {
			select {
			case api.jobCh <- natsMessage{subject: subject, data: data}:
			case <-api.stop:
			}
		}); err != nil {
			return err
		}

		if err := client.Subscribe(s.SubjectNodeState, func(subject string, data []byte) {
			select {
			case api.nodeCh <- natsMessage{subject: subject, data: data}:
			case <-api.stop:
			}
		}); err != nil {
			return err
		}


		if err := client.Subscribe(s.SubjectJobEvent, func(subject string, data []byte) {
			api.jobCh <- natsMessage{subject: subject, data: data}
		}); err != nil {
			return err
		}

		if err := client.Subscribe(s.SubjectNodeState, func(subject string, data []byte) {
			api.nodeCh <- natsMessage{subject: subject, data: data}
		}); err != nil {
			return err
		}

		if err := client.Subscribe(s.SubjectJobEvent, func(subject string, data []byte) {
			select {
			case api.jobCh <- natsMessage{subject: subject, data: data}:
			case <-api.stop:
			}
		}); err != nil {
			return err
		}

		if err := client.Subscribe(s.SubjectNodeState, func(subject string, data []byte) {
			select {
			case api.nodeCh <- natsMessage{subject: subject, data: data}:
			case <-api.stop:
			}
		}); err != nil {
			return err
		}

-      api.jobCh <- natsMessage{subject: subject, data: data}

+      case api.jobCh <- natsMessage{subject: subject, data: data}:

-      api.nodeCh <- natsMessage{subject: subject, data: data}

+      case api.nodeCh <- natsMessage{subject: subject, data: data}:


Now **M2** — guard archiver `Shutdown` against an un-started archiver:


func Shutdown(timeout time.Duration) error {
	cclog.Info("Initiating archiver shutdown...")

	// Close channel to signal no more jobs will be accepted
	close(archiveChannel)

func Shutdown(timeout time.Duration) error {
	cclog.Info("Initiating archiver shutdown...")

	// Guard against Shutdown being called when Start was never run: closing a nil
	// channel and receiving from a nil workerDone would panic/block forever.
	if archiveChannel == nil {
		cclog.Warn("Archiver shutdown called but archiver was never started")
		return nil
	}

	// Close channel to signal no more jobs will be accepted
	close(archiveChannel)


The file /Users/jan/prg/ClusterCockpit/cc-backend/internal/archiver/archiveWorker.go has been updated successfully. (file state is current in your context — no need to Read it back)

func Shutdown(timeout time.Duration) error {
	cclog.Info("Initiating archiver shutdown...")

	// Guard against Shutdown being called when Start was never run: closing a nil
	// channel and receiving from a nil workerDone would panic/block forever.
	if archiveChannel == nil {
		cclog.Warn("Archiver shutdown called but archiver was never started")
		return nil
	}

	// Close channel to signal no more jobs will be accepted
	close(archiveChannel)

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package archiver provides asynchronous job archiving functionality for ClusterCockpit.
//
// The archiver runs a background worker goroutine that processes job archiving requests
// from a buffered channel. When jobs complete, their metric data is archived from the
// metric store to the configured archive backend (filesystem, S3, etc.).
//
// # Architecture
//
// The archiver uses a producer-consumer pattern:
//   - Producer: TriggerArchiving() sends jobs to archiveChannel
//   - Consumer: archivingWorker() processes jobs from the channel
//   - Coordination: sync.WaitGroup tracks pending archive operations
//
// # Lifecycle
//
//  1. Start(repo, ctx) - Initialize worker with context for cancellation
//  2. TriggerArchiving(job) - Queue job for archiving (called when job stops)
//  3. archivingWorker() - Background goroutine processes jobs
//  4. Shutdown(timeout) - Graceful shutdown with timeout
//
// # Graceful Shutdown
//
// The archiver supports graceful shutdown with configurable timeout:
//   - Closes channel to reject new jobs
//   - Waits for pending jobs to complete (up to timeout)
//   - Cancels context if timeout exceeded
//   - Ensures worker goroutine exits cleanly
//
// # Example Usage
//
//	// Initialize archiver
//	ctx, cancel := context.WithCancel(context.Background())
//	defer cancel()
//	archiver.Start(jobRepository, ctx)
//
//	// Trigger archiving when job completes
//	archiver.TriggerArchiving(job)
//
//	// Graceful shutdown with 10 second timeout
//	if err := archiver.Shutdown(10 * time.Second); err != nil {
//	    log.Printf("Archiver shutdown timeout: %v", err)
//	}
package archiver

import (
	"context"
	"fmt"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/repository"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	sq "github.com/Masterminds/squirrel"
)

var (
	archivePending sync.WaitGroup
	archiveChannel chan *schema.Job
	jobRepo        *repository.JobRepository
	shutdownCtx    context.Context
	shutdownCancel context.CancelFunc
	workerDone     chan struct{}
)

// Start initializes the archiver and starts the background worker goroutine.
//
// The archiver processes job archiving requests asynchronously via a buffered channel.
// Jobs are sent to the channel using TriggerArchiving() and processed by the worker.
//
// Parameters:
//   - r: JobRepository instance for database operations
//   - ctx: Context for cancellation (shutdown signal propagation)
//
// The worker goroutine will run until:
//   - ctx is cancelled (via parent shutdown)
//   - archiveChannel is closed (via Shutdown())
//
// Must be called before TriggerArchiving(). Safe to call only once.
func Start(r *repository.JobRepository, ctx context.Context) {
	shutdownCtx, shutdownCancel = context.WithCancel(ctx)
	archiveChannel = make(chan *schema.Job, 128)
	workerDone = make(chan struct{})
	jobRepo = r

	go archivingWorker()
}

// archivingWorker is the background goroutine that processes job archiving requests.
//
// The worker loop:
//  1. Blocks waiting for jobs on archiveChannel or shutdown signal
//  2. Fetches job metadata from repository
//  3. Archives job data to configured backend (calls ArchiveJob)
//  4. Updates job footprint and energy metrics in database
//  5. Marks job as successfully archived
//  6. Calls job stop hooks
//
// The worker exits when:
//   - shutdownCtx is cancelled (timeout during shutdown)
//   - archiveChannel is closed (normal shutdown)
//
// Errors during archiving are logged and the job is marked as failed,
// but the worker continues processing other jobs.
func archivingWorker() {
	defer close(workerDone)

	for {
		select {
		case <-shutdownCtx.Done():
			cclog.Info("Archive worker received shutdown signal")
			return

		case job, ok := <-archiveChannel:
			if !ok {
				cclog.Info("Archive channel closed, worker exiting")
				return
			}

			start := time.Now()
			// not using meta data, called to load JobMeta into Cache?
			// will fail if job meta not in repository
			if _, err := jobRepo.FetchMetadata(job); err != nil {
				cclog.Errorf("archiving job (dbid: %d) failed at check metadata step: %s", *job.ID, err.Error())
				jobRepo.UpdateMonitoringStatus(*job.ID, schema.MonitoringStatusArchivingFailed)
				archivePending.Done()
				continue
			}

			// ArchiveJob will fetch all the data from a MetricDataRepository and push into configured archive backend
			// Use shutdown context to allow cancellation
			jobMeta, err := ArchiveJob(job, shutdownCtx)
			if err != nil {
				cclog.Errorf("archiving job (dbid: %d) failed at archiving job step: %s", *job.ID, err.Error())
				jobRepo.UpdateMonitoringStatus(*job.ID, schema.MonitoringStatusArchivingFailed)
				archivePending.Done()
				continue
			}

			stmt := sq.Update("job").Where("job.id = ?", job.ID)

			if stmt, err = jobRepo.UpdateFootprint(stmt, jobMeta); err != nil {
				cclog.Errorf("archiving job (dbid: %d) failed at update Footprint step: %s", *job.ID, err.Error())
				archivePending.Done()
				continue
			}
			if stmt, err = jobRepo.UpdateEnergy(stmt, jobMeta); err != nil {
				cclog.Errorf("archiving job (dbid: %d) failed at update Energy step: %s", *job.ID, err.Error())
				archivePending.Done()
				continue
			}
			// Update the jobs database entry one last time:
			stmt = jobRepo.MarkArchived(stmt, schema.MonitoringStatusArchivingSuccessful)
			if err := jobRepo.Execute(stmt); err != nil {
				cclog.Errorf("archiving job (dbid: %d) failed at db execute: %s", *job.ID, err.Error())
				archivePending.Done()
				continue
			}
			cclog.Debugf("archiving job %d took %s", job.JobID, time.Since(start))
			cclog.Infof("archiving job (dbid: %d) successful", *job.ID)

			repository.CallJobStopHooks(job)
			archivePending.Done()
		}
	}
}

// TriggerArchiving queues a job for asynchronous archiving.
//
// This function should be called when a job completes (stops) to archive its
// metric data from the metric store to the configured archive backend.
//
// The function:
//  1. Increments the pending job counter (WaitGroup)
//  2. Sends the job to the archiving channel (buffered, capacity 128)
//  3. Returns immediately (non-blocking unless channel is full)
//
// The actual archiving is performed asynchronously by the worker goroutine.
// Upon completion, the worker will decrement the pending counter.
//
// Panics if Start() has not been called first.
func TriggerArchiving(job *schema.Job) {
	if archiveChannel == nil {
		cclog.Fatal("Cannot archive without archiving channel. Did you Start the archiver?")
	}

	archivePending.Add(1)
	archiveChannel <- job
}

// Shutdown performs a graceful shutdown of the archiver with a configurable timeout.
//
// The shutdown process:
//  1. Closes archiveChannel - no new jobs will be accepted
//  2. Waits for pending jobs to complete (up to timeout duration)
//  3. If timeout is exceeded:
//     - Cancels shutdownCtx to interrupt ongoing ArchiveJob operations
//     - Returns error indicating timeout
//  4. Waits for worker goroutine to exit cleanly
//
// Parameters:
//   - timeout: Maximum duration to wait for pending jobs to complete
//     (recommended: 10-30 seconds for production)
//
// Returns:
//   - nil if all jobs completed within timeout
//   - error if timeout was exceeded (some jobs may not have been archived)
//
// Jobs that don't complete within the timeout will be marked as failed.
// The function always ensures the worker goroutine exits before returning.
//
// Example:
//
//	if err := archiver.Shutdown(10 * time.Second); err != nil {
//	    log.Printf("Some jobs did not complete: %v", err)
//	}
func Shutdown(timeout time.Duration) error {
	cclog.Info("Initiating archiver shutdown...")

	// Close channel to signal no more jobs will be accepted
	close(archiveChannel)

	// Create a channel to signal when all jobs are done
	done := make(chan struct{})
	go func() {
		archivePending.Wait()
		close(done)
	}()

	// Wait for jobs to complete or timeout
	select {
	case <-done:
		cclog.Info("All archive jobs completed successfully")
		// Wait for worker to exit
		<-workerDone
		return nil
	case <-time.After(timeout):
		cclog.Warn("Archiver shutdown timeout exceeded, cancelling remaining operations")
		// Cancel any ongoing operations
		shutdownCancel()
		// Wait for worker to exit
		<-workerDone
		return fmt.Errorf("archiver shutdown timeout after %v", timeout)
	}
}

+  // Guard against Shutdown being called when Start was never run: closing a nil

+  // channel and receiving from a nil workerDone would panic/block forever.

+    cclog.Warn("Archiver shutdown called but archiver was never started")


Now **M4** — make `metricstore.Shutdown()` idempotent and release the mutex early:


	shutdownFuncMu.Lock()
	defer shutdownFuncMu.Unlock()
	if shutdownFunc != nil {
		shutdownFunc()
	}
	cclog.Infof("[METRICSTORE]> Background workers cancelled (%v)", time.Since(totalStart))

	shutdownFuncMu.Lock()
	if shutdownFunc == nil {
		// Already shut down (or never initialized): nothing to do. This keeps
		// Shutdown idempotent so it is safe to call from more than one path.
		shutdownFuncMu.Unlock()
		return
	}
	shutdownFunc()
	shutdownFunc = nil
	shutdownFuncMu.Unlock()
	cclog.Infof("[METRICSTORE]> Background workers cancelled (%v)", time.Since(totalStart))


The file /Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/metricstore.go has been updated successfully. (file state is current in your context — no need to Read it back)

	shutdownFuncMu.Lock()
	defer shutdownFuncMu.Unlock()
	if shutdownFunc != nil {
		shutdownFunc()
	}
	cclog.Infof("[METRICSTORE]> Background workers cancelled (%v)", time.Since(totalStart))

	shutdownFuncMu.Lock()
	if shutdownFunc == nil {
		// Already shut down (or never initialized): nothing to do. This keeps
		// Shutdown idempotent so it is safe to call from more than one path.
		shutdownFuncMu.Unlock()
		return
	}
	shutdownFunc()
	shutdownFunc = nil
	shutdownFuncMu.Unlock()
	cclog.Infof("[METRICSTORE]> Background workers cancelled (%v)", time.Since(totalStart))

+    // Already shut down (or never initialized): nothing to do. This keeps

+    // Shutdown idempotent so it is safe to call from more than one path.

if Keys.Checkpoints.FileFormat == "wal" {


Now let me build and vet everything:


Bash

go build ./... 2>&1 | head -30 && echo "=== BUILD DONE ===" && go vet ./cmd/... ./internal/auth/... ./internal/api/... ./internal/archiver/... ./pkg/metricstore/... 2>&1 | head -30 && echo "=== VET DONE ==="


=== BUILD DONE ===
=== VET DONE ===


Build and vet pass cleanly. Let me run the relevant tests:


go test ./internal/auth/... ./internal/api/... ./internal/archiver/... ./pkg/metricstore/... 2>&1 | tail -30


ok  	github.com/ClusterCockpit/cc-backend/internal/auth	0.458s
ok  	github.com/ClusterCockpit/cc-backend/internal/api	6.994s
?   	github.com/ClusterCockpit/cc-backend/internal/archiver	[no test files]
ok  	github.com/ClusterCockpit/cc-backend/pkg/metricstore	1.001s


All tests pass. Let me do a final verification of the two structural changes (the `handleUserCommands` guard and the `runServer` tail) to be sure the logic reads correctly: