Perfect. Now I have all the information I need. Let me compile the comprehensive security/stability review with exact file:line references and code quotes.

## CRITICAL CODE REVIEW: NATS Client & Metricstore Shutdown Race Conditions

I have identified multiple **CRITICAL** and **SEVERE** issues in the shutdown sequence involving NATS client closure and metricstore/API operations. These are not mere logging issues—they represent **data loss**, **panic risks**, and **resource leaks** under load.

---

### ISSUE 1: SUBSCRIPTION CALLBACKS ACTIVE DURING NATS CONNECTION CLOSE

**Severity:** CRITICAL

**Problem:** When `nats.GetClient().Close()` is called in `Server.Shutdown()` (line 461, cmd/cc-backend/server.go), the NATS subscription callbacks are **still executing**. This creates a race condition where the NATS client connection is closed while callbacks are in flight.

**Evidence:**

1. **Line 459-462, cmd/cc-backend/server.go:**
```go
nc := nats.GetClient()
if nc != nil {
    nc.Close()  // <-- NATS connection closes HERE
}
```

2. **ReceiveNats subscriptions still active** (lines 77, 84, pkg/metricstore/lineprotocol.go):
```go
nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
    select {
    case msgs <- data:
    case <-ctx.Done():
    }
})
```

3. **Context not cancelled until later** (lines 279-281, pkg/metricstore/metricstore.go):
```go
if shutdownFunc != nil {
    shutdownFunc()  // <-- Context cancelled AFTER nats.Close()
}
```

**Timeline:**
```
1. nats.Close() called (line 461, server.go) — closes TCP connection
2. NATS subscription callbacks are executing (could be in DecodeLine)
3. 60+ ms later: HTTP server shuts down (lines 465-471)
4. Much later: metricstore.Shutdown() calls shutdownFunc() (line 280)
   — this cancels ctx that ReceiveNats is waiting on
```

**Consequence:** Between `nats.Close()` and `shutdownFunc()`, in-flight NATS subscription callbacks in `ReceiveNats` (lines 77-82) or `NatsAPI.StartSubscriptions` (lines 142-152) may experience:
- Callbacks attempting to send to closed NATS connection internals
- Panics if the NATS library tries to access freed memory
- Unpredictable error handling in callback contexts

---

### ISSUE 2: NATSAPI WORKER GOROUTINES LEAK ON SHUTDOWN

**Severity:** CRITICAL

**Problem:** The `NatsAPI` instantiated in `server.go:330-333` creates persistent worker goroutines that are **never stopped** during shutdown.

**Evidence:**

1. **Lines 105-110, internal/api/nats.go - NatsAPI created with persistent workers:**
```go
api := &NatsAPI{
    JobRepository: repository.GetJobRepository(),
    jobCh:         make(chan natsMessage, jobConc),  // buffered channel
    nodeCh:        make(chan natsMessage, nodeConc),  // buffered channel
}

// Start worker goroutines
for range jobConc {
    go api.jobWorker()  // <-- Spins forever
}
for range nodeConc {
    go api.nodeWorker()  // <-- Spins forever
}
```

2. **Lines 116-127, internal/api/nats.go - Workers block on channel receive:**
```go
func (api *NatsAPI) jobWorker() {
    for msg := range api.jobCh {  // <-- Blocks forever on closed channel
        api.handleJobEvent(msg.subject, msg.data)
    }
}

func (api *NatsAPI) nodeWorker() {
    for msg := range api.nodeCh {  // <-- Blocks forever on closed channel
        api.handleNodeState(msg.subject, msg.data)
    }
}
```

3. **server.go creates NatsAPI but never closes it:**
```go
if config.Keys.APISubjects != nil {
    s.natsAPIHandle = api.NewNatsAPI()  // <-- Created
    if err := s.natsAPIHandle.StartSubscriptions(); err != nil {
        return fmt.Errorf("starting NATS subscriptions: %w", err)
    }
}
// NO CLEANUP during shutdown
```

**Consequence:**
- **8 job workers + 2 node workers = 10 goroutine leaks per restart cycle**
- Channels `api.jobCh` and `api.nodeCh` are **never closed**, so workers block forever on channel receive
- No graceful shutdown, no cleanup of buffered messages in the channels
- Long-lived processes will accumulate thousands of leaked goroutines over days/weeks

---

### ISSUE 3: METRICSTORE CONTEXT CANCELLED WHILE DECODELINE IN FLIGHT

**Severity:** SEVERE

**Problem:** The metricstore context is cancelled in `Shutdown()` (line 280) **before** ensuring `ReceiveNats` goroutine has fully exited. This creates a race where `DecodeLine` is in the middle of writing to the store while the context cancels.

**Evidence:**

1. **Lines 187-193, pkg/metricstore/metricstore.go - ReceiveNats goroutine spawned:**
```go
if Keys.Subscriptions != nil {
    wg.Go(func() {
        if err := ReceiveNats(ms, Keys.NumWorkers, ctx); err != nil {
            cclog.Fatal(err)
        }
    })
}
```

2. **Lines 279-282, pkg/metricstore/metricstore.go - Context cancelled:**
```go
shutdownFuncMu.Lock()
if shutdownFunc != nil {
    shutdownFunc()  // <-- Cancels ctx immediately
}
cclog.Infof("[METRICSTORE]> Background workers cancelled...")
```

3. **No synchronous wait for ReceiveNats to exit before proceeding:**
The `wg.Go()` adds to the WaitGroup, but there's **no `wg.Wait()` after `shutdownFunc()`**. The function proceeds immediately to checkpoint writing (line 299) without waiting for ReceiveNats to flush and exit.

**Consequence:**
- `DecodeLine` at lines 395-398 (ms.WriteToLevel) can race with metricstore state cleanup
- In-flight metrics between context cancellation and actual goroutine exit may not be checkpointed
- If `WriteToLevel` holds the Level lock when context is cancelled, callbacks still executing could deadlock

---

### ISSUE 4: WAL MESSAGE LOSS WINDOW DURING SHUTDOWN

**Severity:** SEVERE

**Problem:** The shutdown sequence has a **100ms grace period** (line 289) before closing WAL channels, but this is insufficient when `DecodeLine` calls are concurrent.

**Evidence:**

1. **Lines 284-296, pkg/metricstore/metricstore.go:**
```go
if Keys.Checkpoints.FileFormat == "wal" {
    // Signal producers to stop sending before closing channels,
    // preventing send-on-closed-channel panics from in-flight NATS workers.
    walShuttingDown.Store(true)
    // Brief grace period for in-flight DecodeLine calls to complete.
    time.Sleep(100 * time.Millisecond)  // <-- ONLY 100ms!

    for _, ch := range walShardChs {
        close(ch)
    }
    drainStart := time.Now()
    WaitForWALStagingDrain()
```

2. **Lines 386-393, pkg/metricstore/lineprotocol.go - SendWALMessage can race:**
```go
if !SendWALMessage(msg) {
    // WAL shard channel full — metric is written to memory store but not WAL.
    // Next binary snapshot will capture it.
    if dropped := walDropped.Add(1); dropped%10000 == 1 {
        cclog.Warnf("[METRICSTORE]> WAL shard channel full, dropped %d messages...")
    }
}
```

3. **The comment on line 288-289 acknowledges the issue but does nothing to fix it:**
The grace period of 100ms is **hardcoded** and provides no guarantee. Under high load with many workers, `DecodeLine` calls from NATS callbacks can easily take >100ms to complete, especially if they hit slow disk I/O for previous checkpoints.

**Race window:**
```
1. walShuttingDown.Store(true) [line 287]
2. sleep(100ms) [line 289]
3. In-flight DecodeLine may call SendWALMessage() at line 386
4. SendWALMessage checks walShuttingDown.Load() [line 151]
5. If called AFTER 100ms but BEFORE channels close, message is dropped silently
6. Later CheckWALMessages dropped % 10000 == 1, so batch losses are not logged
```

**Consequence:**
- Metrics in flight are silently dropped (not written to WAL)
- The comment claims "data safe in memory" but the binary checkpoint hasn't been written yet
- If crash occurs between lines 289 and 312 (before checkpoint written), loss is undetected

---

### ISSUE 5: DOUBLE SHUTDOWN VULNERABILITY IN METRICSTORE.SHUTDOWN()

**Severity:** MEDIUM

**Problem:** `Shutdown()` has minimal protection against being called twice, which could happen if multiple goroutines trigger shutdown concurrently.

**Evidence:**

1. **Lines 274-281, pkg/metricstore/metricstore.go:**
```go
func Shutdown() {
    totalStart := time.Now()

    shutdownFuncMu.Lock()
    defer shutdownFuncMu.Unlock()
    if shutdownFunc != nil {
        shutdownFunc()  // <-- Calls context.CancelFunc
        // NO nil assignment here, so second call would call cancelled context again
    }
```

2. **After first call, `shutdownFunc` is NOT set to nil**, so if `Shutdown()` is called again:
   - The mutex protects the lock, but `shutdownFunc()` would be called twice
   - Go's `context.CancelFunc` is safe to call multiple times (it's idempotent)
   - BUT the flag `walShuttingDown.Store(true)` on line 287 would execute twice unnecessarily

3. **More problematic: If `Init()` is never called** (e.g., during initialization failure):
```go
if metricstore.Keys.Subscriptions != nil {
    // suppose Init fails before line 187
}
// ... later ...
metricstore.Shutdown()  // Called when shutdownFunc is still nil
```
This is safe, but it's fragile.

**Consequence:**
- Not a panic risk (Go's context is idempotent), but indicates weak defensive programming
- In a concurrent shutdown scenario, race conditions could occur if the second call gets through the mutex check at a bad time

---

### ISSUE 6: NO DRAINING OF BUFFERED NATSAPI CHANNELS

**Severity:** MEDIUM

**Problem:** When `nats.Close()` stops delivering messages, the `NatsAPI` channels (`jobCh`, `nodeCh`) are left with potentially buffered unprocessed messages that are **never drained or logged**.

**Evidence:**

1. **Lines 100-101, internal/api/nats.go - Channels created with buffers:**
```go
jobCh:         make(chan natsMessage, jobConc),
nodeCh:        make(chan natsMessage, nodeConc),
```
With default `jobConc=8` and `nodeConc=2`, these channels can hold 8 and 2 messages respectively.

2. **Subscription callbacks send to these channels (lines 142-152):**
```go
if err := client.Subscribe(s.SubjectJobEvent, func(subject string, data []byte) {
    api.jobCh <- natsMessage{subject: subject, data: data}  // <-- Enqueues
}); err != nil {
    return err
}
```

3. **No cleanup on shutdown:**
There is no code that:
- Closes the channels to signal workers to exit
- Drains buffered messages before closing
- Logs how many buffered messages were discarded

**Consequence:**
- Job start/stop messages in the buffer during shutdown are silently dropped
- Worker goroutines remain blocked on channel receive indefinitely
- Silent data loss of API events

---

### ISSUE 7: UNSAFE METRICSTORE GETMEMORYSTORE() AFTER SHUTDOWN

**Severity:** MEDIUM (but could promote to HIGH in concurrent scenarios)

**Problem:** `GetMemoryStore()` can be called after shutdown while the store's context is cancelled, leading to undefined behavior if code relies on context semantics.

**Evidence:**

1. **Lines 238-244, pkg/metricstore/metricstore.go:**
```go
func GetMemoryStore() *MemoryStore {
    if msInstance == nil {
        cclog.Warnf("[METRICSTORE]> MemoryStore not initialized!")
    }
    return msInstance  // <-- Returns even if nil, even after shutdown
}
```

2. **Line 304, pkg/metricstore/metricstore.go - Called during shutdown:**
```go
ms := GetMemoryStore()  // <-- Could be nil or in shutdown state
```

3. **In server.go:482, also called during shutdown:**
```go
if ms := metricstore.GetMemoryStore(); ms != nil {
    wg.Go(func() {
        metricstore.Shutdown()
    })
}
```

**Consequence:**
- If `GetMemoryStore()` returns nil (initialization failed), the nil check at server.go:482 prevents the panic, but there's no logging of this condition
- More subtly: calls to `ms.Write()` or `ms.WriteToLevel()` after shutdown could execute on a valid pointer but with a cancelled context, leading to undefined semantics

---

### ISSUE 8: INCOMING NATS MESSAGES AFTER NCCLOSE - BEHAVIOR UNDEFINED

**Severity:** SEVERE

**Problem:** The NATS client's internal state after `nc.Close()` is implementation-dependent. If any subscription callbacks are queued for execution, they could panic or access freed memory.

**Evidence:**

1. **Lines 459-463, cmd/cc-backend/server.go:**
```go
nc := nats.GetClient()
if nc != nil {
    nc.Close()  // <-- Synchronously closes connection
}
cclog.Infof("Shutdown: NATS closed (%v)", time.Since(natsStart))
```

2. **The NATS library (cc-lib/v2/nats) callback mechanism is not examined here, but typical NATS client behavior:**
   - Subscription callbacks are executed asynchronously by internal NATS goroutines
   - Closing the connection does NOT guarantee all pending callbacks have been executed
   - Some NATS libraries queue callbacks and execute them on a background goroutine

3. **Example of how a race could manifest** (from DecodeLine):
```go
// In a NATS callback (lines 77-82, lineprotocol.go):
nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
    select {
    case msgs <- data:  // <-- This goroutine is executing
    case <-ctx.Done():
    }
})

// Meanwhile in server.go:
nc.Close()  // <-- Closes connection before callback finishes

// If the NATS library accesses connection state after Close():
// PANIC: access to freed/nil connection pointer
```

**Consequence:**
- Undefined behavior in the NATS library's callback execution context
- Potential panics in NATS library code (not in cc-backend, but triggered by it)
- No graceful degradation

---

### SUMMARY TABLE

| Issue | File:Line | Type | Severity | Problem |
|-------|-----------|------|----------|---------|
| 1 | server.go:459-461, metricstore.go:279-281 | Race | CRITICAL | NATS closed before subscriptions drained |
| 2 | nats.go:105-127, server.go:330-333 | Leak | CRITICAL | NatsAPI workers never stopped, channels never closed |
| 3 | metricstore.go:187-193, 279-282 | Race | SEVERE | Context cancelled without waiting for ReceiveNats |
| 4 | metricstore.go:289, lineprotocol.go:386-393 | Loss | SEVERE | 100ms grace insufficient for concurrent DecodeLine |
| 5 | metricstore.go:274-281 | Safety | MEDIUM | shutdownFunc not reset to nil after call |
| 6 | nats.go:100-101, 142-152 | Loss | MEDIUM | Buffered API messages discarded on shutdown |
| 7 | metricstore.go:238-244, 304 | Safety | MEDIUM | GetMemoryStore() unsafe after shutdown |
| 8 | server.go:459-463 | Panic | SEVERE | NATS Close() doesn't drain async callbacks |

---

### RECOMMENDED INVESTIGATION POINTS

1. **Test with high-load metric ingestion during shutdown** to trigger Issue 4 (WAL loss)
2. **Check cc-lib/v2/nats source** to understand callback execution semantics after `Close()`
3. **Add goroutine profiling** to confirm Issue 2 (NatsAPI worker leaks) under repeated start/stop cycles
4. **Stress-test the 100ms grace period** (Issue 4) with concurrent `DecodeLine` calls from multiple workers
5. **Verify if messages are ever lost** between `nats.Close()` and the final binary checkpoint write

Prompt:
I'm doing a code review of the cc-backend Go project (ClusterCockpit) focused on CRITICAL and SEVERE issues in initialization, startup, and shutdown.

Focus on the interaction between the NATS client and the metricstore's NATS metric ingestion during startup and shutdown. Report exact code with file:line references.

Context: In `cmd/cc-backend/main.go`, `nats.Init()` + `nats.Connect()` happen early. In `cmd/cc-backend/server.go` `Server.Shutdown()`, `nats.GetClient().Close()` is called FIRST, then HTTP server shutdown, then `metricstore.Shutdown()` runs concurrently with `archiver.Shutdown()`.

Investigate:
1. `pkg/metricstore/metricstore.go` `Init()` starts a goroutine `ReceiveNats(ms, numWorkers, ctx)` when `Keys.Subscriptions != nil`. Find and read `ReceiveNats` (likely in `pkg/metricstore/lineprotocol.go` or similar). How does it subscribe to NATS? What nats client does it use? When the global nats client is `Close()`d while `ReceiveNats` is still running (before metricstore ctx is cancelled), what happens? Can the message handler call `SendWALMessage` or `Write` on a store mid-shutdown? Is there a panic risk (send on closed channel) or use of a closed nats connection?

2. `internal/api/nats.go` — `NatsAPI.StartSubscriptions()` (started in server.go init). Same questions: what happens to these subscriptions when the nats client is closed at the start of Server.Shutdown? Are subscriptions drained/unsubscribed cleanly?

3. In `pkg/metricstore/metricstore.go`, examine `Shutdown()`: the `shutdownFunc` global and `shutdownFuncMu`. Can `Shutdown()` be called twice safely? Can it be called if Init was never completed (shutdownFunc nil)? What if `GetMemoryStore()` returns nil?

4. The ordering: nats closed -> http stopped -> metricstore.Shutdown cancels its ctx. Between nats.Close() and metricstore ctx cancellation, is there a window where the metricstore NATS subscription goroutine errors out, logs spam, or loses in-flight metric writes (not yet checkpointed)?

Quote relevant code with file:line. Only report verifiable issues with evidence. Do not propose fixes — identify and explain with evidence.


Do a thorough code  review with a focus on critical and severe issues. Initially focus on the initialization, startup and shutdown.


Now I have a clear picture. Let me create a comprehensive report with the findings:

## CRITICAL CODE REVIEW: Initialization and Startup Ordering Issues

Based on my detailed analysis of the cc-backend initialization sequence, I have identified **SEVERE issues** with startup ordering and uninitialized state access. Here are the findings with exact code references:

---

### ISSUE 1: CRITICAL - Unconditional auth.GetAuthInstance() Call When auth.Init May Never Execute

**File: `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/server.go`, line 123**

```go
authHandle := auth.GetAuthInstance()  // Line 123
```

**Context: Main flow in `cmd/cc-backend/main.go`**

- **Lines 175-181** in `handleUserCommands()`: `auth.Init()` is ONLY called when `!config.Keys.DisableAuthentication` is true
- **Lines 530-537** in `run()`: `handleUserCommands()` executes, but may NOT initialize auth if DisableAuthentication is true
- **Lines 386-389** in `runServer()`: Calls `NewServer()` which immediately calls `s.init()` at line 85
- **Lines 92-123** in `server.go init()`: Line 123 calls `auth.GetAuthInstance()` UNCONDITIONALLY

**The Problem:**

When `config.Keys.DisableAuthentication == true`:
1. `handleUserCommands()` skips the entire `if !config.Keys.DisableAuthentication` block (lines 175-211)
2. `auth.Init()` is **NEVER** called
3. `authInstance` remains `nil` (see `/Users/jan/prg/ClusterCockpit/cc-backend/internal/auth/auth.go` line 52: `authInstance *Authentication`)
4. Later, `server.go` line 123 calls `auth.GetAuthInstance()`
5. `GetAuthInstance()` checks `if authInstance == nil` and calls `cclog.Fatal()` (line 260 in auth.go):

```go
func GetAuthInstance() *Authentication {
	if authInstance == nil {
		cclog.Fatal("Authentication module not initialized!")  // Line 260
	}
	return authInstance
}
```

**Result:** Process aborts with `Fatal` error when authentication is disabled.

**Evidence:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/main.go` lines 170-214 (handleUserCommands)
- `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/server.go` line 123 (unconditional call)
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/auth/auth.go` lines 50-53, 258-264

---

### ISSUE 2: SEVERE - auth.Keys Access Before auth.Init() With DisableAuthentication

**File: `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/server.go`, line 189**

```go
if auth.Keys.OpenIDConfig != nil {  // Line 189
    openIDConnect := auth.NewOIDC(authHandle)
    openIDConnect.RegisterEndpoints(s.router)
    info["hasOpenIDConnect"] = true
}
```

**The Problem:**

1. `auth.Keys` is a module-level variable (line 116 in `/Users/jan/prg/ClusterCockpit/cc-backend/internal/auth/auth.go`):
   ```go
   var Keys AuthConfig  // Line 116
   ```

2. `auth.Keys` is ONLY populated inside `auth.Init()` (lines 204-248 in auth.go):
   ```go
   if authCfg == nil {
       return  // Lines 204-206: Early return, Keys never populated
   }
   config.Validate(configSchema, *authCfg)
   dec := json.NewDecoder(bytes.NewReader(*authCfg))
   dec.DisallowUnknownFields()
   if err := dec.Decode(&Keys); err != nil {  // Line 211: Keys populated here
       cclog.Errorf("error while decoding ldap config: %v", err)
   }
   ```

3. When `DisableAuthentication == true`, `auth.Init()` is never called, so `auth.Keys` remains at zero value
4. Line 189 checks `if auth.Keys.OpenIDConfig != nil` — this is safe (nil check on zero-value struct field)
5. BUT: If OpenID config somehow exists in zero-value (which it won't, but the code assumes it doesn't), line 190 would dereference `authHandle` which is NOT nil (from line 123's fatal call or if init somehow succeeded)

**Actually Safe But Fragile:** The nil-check at line 189 prevents dereferencing, BUT:
- The entire initialization depends on `auth.GetAuthInstance()` at line 123 succeeding
- If that call doesn't fatal, then line 190 (`NewOIDC(authHandle)`) will execute with potentially uninitialized `auth.Keys`

**Evidence:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/auth/auth.go` lines 181-256 (Init function with conditional early return)
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/auth/auth.go` line 116 (Keys declaration)
- `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/server.go` lines 189-193

---

### ISSUE 3: CRITICAL - Database Migration Commands Execute Before Connect()

**File: `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/main.go`, lines 507-514**

```go
// Handle database migration (migrate, revert, force)
if err := handleDatabaseCommands(); err != nil {  // Line 507
    return err
}

// Initialize database
if err := initDatabase(); err != nil {  // Line 512
    return err
}
```

**The Problem:**

1. `handleDatabaseCommands()` (lines 139-168) calls:
   - `repository.MigrateDB(config.Keys.DB)` (line 141)
   - `repository.RevertDB(config.Keys.DB)` (line 150)
   - `repository.ForceDB(config.Keys.DB)` (line 159)

2. These functions in `/Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/migration.go` (lines 113-204):
   ```go
   func MigrateDB(db string) error {
       m, err := getMigrateInstance(db)  // Line 114
       if err != nil {
           return err
       }
       // ...
       if err := m.Up(); err != nil {  // Line 136
   ```

   Each creates a NEW database connection via `getMigrateInstance()` which calls:
   ```go
   m, err = migrate.NewWithSourceInstance("iofs", d,
       fmt.Sprintf("sqlite3://%s?_foreign_keys=on", db))  // Line 88
   ```

3. These migrations open THEIR OWN database connections (not using `repository.Connect()`)

4. THEN `initDatabase()` (lines 111-137) calls:
   ```go
   repository.SetConfig(cfg)  // Line 133
   repository.Connect(config.Keys.DB)  // Line 135
   ```

**Issue:** Multiple independent database connections open to the same SQLite file. SQLite with `_journal_mode=WAL` can handle multiple readers, but:
- Migration might hold a write lock while `Connect()` tries to initialize
- The `repository.Connect()` uses different SQLite pragmas (lines 39-53 in dbConnection.go) than the migration's `_foreign_keys=on` only
- No ordering guarantees that migrations complete before Connect() proceeds

**Evidence:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/main.go` lines 139-168 (handleDatabaseCommands)
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/migration.go` lines 82-94 (getMigrateInstance opens own connection)
- `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/main.go` lines 111-137 (initDatabase)
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/dbConnection.go` lines 55-110 (Connect with different pragmas)

---

### ISSUE 4: CRITICAL - GetUserRepository() and GetJobRepository() Can Be Called Before Connect()

**File: `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/main.go`, line 232**

```go
ur := repository.GetUserRepository()  // Called in addUser() at line 232
```

**Call Chain:**
1. Line 531: `handleUserCommands()` is called
2. Lines 186-189: `addUser(flagNewUser)` is called AFTER auth.Init but only if `!DisableAuthentication`
3. Line 232 (addUser): `repository.GetUserRepository()` is called

**The Problem:**

`GetUserRepository()` in `/Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/user.go` lines 59-69:

```go
func GetUserRepository() *UserRepository {
    userRepoOnce.Do(func() {
        db := GetConnection()  // Line 61

        userRepoInstance = &UserRepository{
            DB:     db.DB,
            driver: db.Driver,
        }
    })
    return userRepoInstance
}
```

It calls `GetConnection()` (line 61) which, in `/Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/dbConnection.go` lines 112-118:

```go
func GetConnection() *DBConnection {
    if dbConnInstance == nil {
        cclog.Fatalf("Database connection not initialized!")  // Line 114
    }
    return dbConnInstance
}
```

**Will fatal if `Connect()` was never called.**

BUT: The order in `main.go run()` is:
- Line 512: `initDatabase()` calls `repository.Connect()`
- Line 531: `handleUserCommands()` calls `addUser()` which calls `GetUserRepository()`

**Actually Safe in Current Flow** - but `handleUserCommands()` reads from user repos (lines 232, 247, 269) which depend on `Connect()` having executed at line 512.

**However:** No synchronization prevents races if `Connect()` fails silently. More critically:

**At line 518** (flagOptimizeDB):
```go
db := repository.GetConnection()  // Line 518
```

This is called BEFORE `handleUserCommands()`. If `initDatabase()` fails but doesn't return (e.g., returns partial error), this could call GetConnection before Connect succeeded.

**Evidence:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/user.go` lines 59-69
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/dbConnection.go` lines 112-118
- `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/main.go` lines 517-528 (GetConnection before handleUserCommands)

---

### ISSUE 5: SEVERE - Connect() Uses sync.Once Without Error Handling

**File: `/Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/dbConnection.go`, lines 55-110**

```go
func Connect(db string) {
    var err error
    var dbHandle *sqlx.DB

    dbConnOnce.Do(func() {  // Line 59: sync.Once closure
        // ... setup connection ...
        if err != nil {
            cclog.Abortf("DB Connection: Could not connect to SQLite database...")  // Lines 87-88
        }
        // ...
        if err != nil {
            cclog.Abortf("Failed sqlite db setup...")  // Lines 92-93
        }
        // ...
        if err != nil {
            cclog.Abortf("DB Connection: Failed DB version check...")  // Lines 106-107
        }
    })
}
```

**The Problem:**

1. `Connect()` has NO return value - it's `func Connect(db string)` (line 55)
2. All errors are handled via `cclog.Abortf()` which terminates the process
3. If ANY error occurs inside the sync.Once closure, the process terminates
4. BUT: The sync.Once ensures the function body only runs ONCE
5. If someone calls `Connect()` twice with different `db` parameters:
   - First call: Initializes to db1
   - Second call: sync.Once.Do() doesn't execute, so db1 remains active
   - No way to know this silently happened

**This is not a crash scenario, but a silent failure scenario:**

Example problematic code (doesn't exist but could):
```go
repository.Connect("/path1/db.sqlite")
repository.Connect("/path2/db.sqlite")  // Silently ignored
// Now using /path1/db.sqlite when expecting /path2/db.sqlite
```

**More Critical:** No way to verify Connect actually succeeded. Code at line 518 assumes it worked:

```go
db := repository.GetConnection()  // Assumes Connect succeeded
if _, err := db.DB.Exec("VACUUM"); err != nil {  // Could be operating on wrong DB
```

**Evidence:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/dbConnection.go` lines 55-110
- Uses `sync.Once` with no return value or error channel
- All error handling via `cclog.Abortf()` which fatals

---

### ISSUE 6: Repository State Access When Connect() Calls cclog.Abort

**File: `/Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/dbConnection.go`, lines 88, 93, 107**

When errors occur during `Connect()`:
```go
cclog.Abortf("DB Connection: Could not connect...")  // Line 88
cclog.Abortf("Failed sqlite db setup...")            // Line 93
cclog.Abortf("DB Connection: Failed DB version check...")  // Line 107
```

These are called BEFORE `dbConnInstance` is assigned (line 104):
```go
dbConnInstance = &DBConnection{DB: dbHandle}  // Line 104
```

If an error occurs at lines 87-88 or 92-93, `dbConnInstance` remains `nil`.

Later calls to `GetConnection()` at line 518 or 232 will fatal with "Database connection not initialized!"

**But cclog.Abort likely terminates the process anyway**, so this is a cascading fatal scenario.

**Evidence:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/repository/dbConnection.go` lines 55-110
- Error handling order: errors checked BEFORE dbConnInstance assignment

---

### ISSUE 7: Repository Repositories Never Initialized When DisableAuthentication=true

When `DisableAuthentication == true`:

1. `handleUserCommands()` returns immediately (lines 175-181)
2. `auth.Init()` is never called
3. BUT `server.go` line 123 REQUIRES `auth.GetAuthInstance()` to work
4. This will fatal at that point

**But if it somehow passed**, consider:
- User repositories are lazily initialized via `sync.Once` in `GetUserRepository()`
- If auth never initialized and user commands never run, repositories might not be initialized at all
- Later HTTP handler access might trigger lazy init, which could fail

**Likelihood:** Server crashes before handlers even load due to Issue #1.

---

### Summary of Severity

| Issue | Severity | Impact |
|-------|----------|--------|
| Issue #1 | CRITICAL | Process fatal aborts when DisableAuthentication=true |
| Issue #2 | SEVERE | Zero-value access to auth.Keys when auth not initialized (blocked by #1) |
| Issue #3 | CRITICAL | Multiple DB connections with conflicting pragmas; potential lock contention |
| Issue #4 | SEVERE | Nil-dereference risk in GetConnection; dependent on Connect order |
| Issue #5 | SEVERE | sync.Once with no error return; silent failure mode possible |
| Issue #6 | CRITICAL | Cascading fatal errors when Connect fails before dbConnInstance assignment |
| Issue #7 | SEVERE | Lazy initialization timing assumptions in multi-state scenarios |

The **primary blocker is Issue #1**: The code unconditionally calls `auth.GetAuthInstance()` at server initialization time (line 123 of server.go) but only calls `auth.Init()` when DisableAuthentication is false. This will always cause a fatal error when DisableAuthentication is true, making that configuration path completely broken.

Prompt:
I'm doing a code review of the cc-backend Go project (ClusterCockpit) focused on CRITICAL and SEVERE issues in initialization and startup ordering. Report exact code with file:line references.

Investigate the initialization of these subsystems as called from `cmd/cc-backend/main.go` `run()`:

1. `internal/config` `Init(cfg)` — does it panic/fatal on bad config? Are `config.Keys` safe to read after? Any global mutable state with concurrent access?

2. `internal/repository` `Connect(dbPath)` and `SetConfig`, `GetConnection`, `GetJobRepository`, `GetUserRepository`. Is `Connect` a `sync.Once` singleton? What happens if `GetJobRepository()` / `GetUserRepository()` is called before `Connect()`? (In main.go, `handleUserCommands` calls `GetUserRepository()` — verify Connect ran first.) Does Connect call cclog.Fatal on error (aborting process)? Any nil-deref risk if connection failed?

3. `internal/auth` `Init(cfg)` — singleton? What happens if `GetAuthInstance()` is called when Init was never called (e.g., when `config.Keys.DisableAuthentication` is true, auth.Init is skipped in main.go handleUserCommands, but server.go init calls `auth.GetAuthInstance()` unconditionally)? Trace whether GetAuthInstance can return nil and whether that nil is dereferenced in server.go.

4. In `main.go`, note that when `DisableAuthentication` is true, `auth.Init` is NEVER called (it's inside the `if !config.Keys.DisableAuthentication` block in handleUserCommands). But `server.go` `init()` calls `authHandle := auth.GetAuthInstance()` unconditionally at the top, and later uses `authHandle` only inside `if !config.Keys.DisableAuthentication` guards. Verify: does `auth.GetAuthInstance()` return nil safely in that case, or does it log/panic? Is `auth.Keys.OpenIDConfig` accessed (server.go line ~189) when auth was never initialized — would that be a nil/zero-value access?

5. Check `repository.Connect` ordering vs `handleDatabaseCommands` (migrate/revert/force) which run BEFORE `initDatabase()` in main.go run(). Do the migrate commands open their own DB connection, or rely on Connect? Is there a double-open or lock issue?

Quote relevant code with file:line. Only report verifiable issues with evidence. Do not propose fixes — identify and explain with evidence.


Perfect. Now I have a complete picture. Let me create the comprehensive report.

---

## CRITICAL AND SEVERE LIFECYCLE ISSUES IN ARCHIVER AND TASKMANAGER

Based on thorough code review of `internal/archiver/archiveWorker.go`, `internal/archiver/archiver.go`, `internal/taskmanager/taskManager.go`, and their integration in `cmd/cc-backend/main.go` and `cmd/cc-backend/server.go`, I've identified several critical issues:

---

### **ARCHIVER SUBSYSTEM ISSUES**

#### **ISSUE 1: CRITICAL - Potential Nil Channel Send/Receive (Race Condition on Start)**

**File/Line References:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/archiver/archiveWorker.go:62-69` (package-level globals)
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/archiver/archiveWorker.go:85-92` (Start function)
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/archiver/archiveWorker.go:187-194` (TriggerArchiving)

**Problem:**
The archiver uses uninitialized package-level variables. These globals are initialized in `Start()`:

```go
var (
    archivePending sync.WaitGroup        // line 63 - ZERO VALUE AT PACKAGE INIT
    archiveChannel chan *schema.Job      // line 64 - NIL AT PACKAGE INIT
    jobRepo        *repository.JobRepository  // line 65 - NIL AT PACKAGE INIT
    shutdownCtx    context.Context       // line 66 - NIL (ZERO VALUE)
    shutdownCancel context.CancelFunc    // line 67 - NIL
    workerDone     chan struct{}         // line 68 - NIL AT PACKAGE INIT
)

func Start(r *repository.JobRepository, ctx context.Context) {  // line 85
    shutdownCtx, shutdownCancel = context.WithCancel(ctx)
    archiveChannel = make(chan *schema.Job, 128)
    workerDone = make(chan struct{})
    jobRepo = r
    go archivingWorker()                 // line 91
}
```

**Failure Scenario:**
If `TriggerArchiving()` is called before `Start()` completes, or if there's memory visibility issue across goroutines (Go memory model guarantees ordering only within synchronization primitives), the send on line 193 could fail:

```go
func TriggerArchiving(job *schema.Job) {
    if archiveChannel == nil {  // line 188 - CHECK
        cclog.Fatal("Cannot archive without archiving channel. Did you Start the archiver?")
    }
    archivePending.Add(1)       // line 192 - RACES WITH START
    archiveChannel <- job       // line 193 - COULD SEND ON NIL IF RACE EXISTS
}
```

The nil-channel check happens AFTER goroutines spawn in `Start()` (line 91). If a job calls `TriggerArchiving()` during `Start()` execution and before line 87 completes, the channel is still nil but the check passed (check-then-act race).

**Root Cause:** No synchronization (mutex/sync.Once) protecting initialization of these globals.

---

#### **ISSUE 2: CRITICAL - Double-Close Panic on Multiple Start() Calls**

**File/Line References:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/archiver/archiveWorker.go:85-92`
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/archiver/archiveWorker.go:226`

**Problem:**
`Start()` is documented as "Safe to call only once" but has no enforcement. If called twice:

**First Start():**
```go
archiveChannel = make(chan *schema.Job, 128)  // line 87: creates channel
workerDone = make(chan struct{})              // line 88: creates channel
go archivingWorker()                          // line 91: spawns goroutine
```

**Second Start():**
```go
archiveChannel = make(chan *schema.Job, 128)  // OVERWRITES - old channel still has worker reading it
workerDone = make(chan struct{})              // OVERWRITES - old defer close(workerDone) will panic
go archivingWorker()                          // spawns SECOND worker
```

Then on `Shutdown()`:
```go
close(archiveChannel)  // line 226: PANICS - already closed by old worker's defer on line 111
```

The old worker's `defer close(workerDone)` (line 111) will panic when the new `Start()` overwrites `workerDone` and closes it twice.

**Root Cause:** No guard against re-initialization; `Start()` directly overwrites globals without cleanup.

---

#### **ISSUE 3: CRITICAL - Double-Close Panic on archiveChannel in Shutdown**

**File/Line References:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/archiver/archiveWorker.go:110-122`
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/archiver/archiveWorker.go:222-226`

**Problem:**
Multiple execution paths can close `archiveChannel`:

1. **Normal shutdown path** (line 226 in `Shutdown()`):
   ```go
   close(archiveChannel)  // line 226
   ```

2. **Worker exit path** when channel naturally closes (line 119-122 in `archivingWorker()`):
   ```go
   case job, ok := <-archiveChannel:  // line 119
       if !ok {                         // line 120 - channel closed
           cclog.Info("Archive channel closed, worker exiting")
           return                       // line 122 - exits cleanly
       }
   ```

3. **If worker goroutine crashes or is never started**, `Shutdown()` still calls `close(archiveChannel)` on an already-closed channel from a previous execution.

**Critical scenario:** If `Start()` never completes successfully (e.g., context cancelled before line 91), `archiveChannel` is created (line 87) but worker never starts. Then `Shutdown()` tries to close a channel with no consumer, which is safe. BUT if `Start()` is called twice and the first worker's cleanup races with the second `Shutdown()`, both workers and shutdown try to close the same channel.

**Root Cause:** No state tracking to know if channels are initialized and whether worker is running.

---

#### **ISSUE 4: SEVERE - Goroutine Leak and Stuck Reader on Shutdown Timeout**

**File/Line References:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/archiver/archiveWorker.go:229-249`
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/archiver/archiveWorker.go:113-122`

**Problem:**
The shutdown mechanism has a subtle deadlock risk:

```go
func Shutdown(timeout time.Duration) error {
    close(archiveChannel)  // line 226: stops accepting jobs

    done := make(chan struct{})
    go func() {                            // line 230: NEW goroutine
        archivePending.Wait()              // line 231: waits for all jobs to complete
        close(done)                        // line 232
    }()

    select {
    case <-done:
        cclog.Info("All archive jobs completed successfully")
        <-workerDone                       // line 240: waits for FIRST worker
        return nil
    case <-time.After(timeout):            // line 242
        cclog.Warn("Archiver shutdown timeout exceeded, cancelling remaining operations")
        shutdownCancel()                   // line 245: cancels shutdown context
        <-workerDone                       // line 247: waits for worker to exit
        return fmt.Errorf("archiver shutdown timeout after %v", timeout)
    }
}
```

**Failure Scenario 1: Deadlock on Pending Jobs**

If an archiving job is stuck (e.g., blocked on `ArchiveJob()` line 137 waiting for metric data that never arrives):
- `archivePending.Wait()` (line 231) blocks forever
- `done` channel never closes
- Timeout fires (line 242)
- `shutdownCancel()` cancels context (line 245)
- Worker receives `shutdownCtx.Done()` signal (line 115)
- But if job is stuck reading from a channel that ALSO depends on the context, we have a mutual wait

**Failure Scenario 2: Goroutine Leak**

If `Start()` was never called:
- `workerDone` is nil (line 68 uninitialized)
- `Shutdown()` tries to read `<-workerDone` (line 240 or 247)
- **PANIC: send on nil channel** (reading from nil panics)

Code at line 188-189 handles nil `archiveChannel` but **`workerDone` is never checked**:

```go
if archiveChannel == nil {                 // line 188: checked
    cclog.Fatal("Cannot archive without archiving channel. Did you Start the archiver?")
}
// BUT: no check for workerDone being nil
```

**Root Cause:** No validation that `Start()` was actually called before `Shutdown()`; WaitGroup.Wait() can block indefinitely if jobs never complete.

---

#### **ISSUE 5: SEVERE - Data Loss of Queued Archive Jobs on Timeout**

**File/Line References:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/archiver/archiveWorker.go:125-169`
- `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/server.go:489`

**Problem:**
When shutdown timeout is exceeded, in-flight jobs are abandoned:

```go
func archivingWorker() {
    for {
        select {
        case <-shutdownCtx.Done():         // line 115: context cancelled on timeout
            cclog.Info("Archive worker received shutdown signal")
            return                         // line 117: EXITS, abandoning active job
        // ... job processing ...
        }
    }
}
```

On timeout (server.go line 500, 60-second limit):
```go
select {
case <-done:
    cclog.Infof("Shutdown: metricstore + archiver completed (%v)", time.Since(storeStart))
case <-time.After(60 * time.Second):  // line 500
    cclog.Warnf("Shutdown deadline exceeded after %v, forcing exit", time.Since(shutdownStart))
    // Server exits here - any job being processed in ArchiveJob() is lost
}
```

A job that's 50 seconds into a 70-second archive operation will:
1. Have `archivePending.Done()` **never called** (not even in error path on context cancel)
2. Never be marked as `MonitoringStatusArchivingFailed` (no error handling for context timeout)
3. Remain in database as `MonitoringStatusRunning` or previous state

The job's metric data is **partially archived or not archived at all**, and the database state is inconsistent.

**Root Cause:** Context cancellation during in-flight `ArchiveJob()` (line 137) doesn't trigger `archivePending.Done()` cleanup if context times out mid-operation.

---

#### **ISSUE 6: MODERATE - Missing archivePending.Done() in Timeout Path**

**File/Line References:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/archiver/archiveWorker.go:110-170`

**Problem:**
Every code path that adds to `archivePending.Add(1)` (line 192) must call `Done()`. Missing calls:

```go
func archivingWorker() {
    for {
        select {
        case <-shutdownCtx.Done():  // line 115 - TIMEOUT PATH
            cclog.Info("Archive worker received shutdown signal")
            return                  // line 117: NO archivePending.Done() CALLED

        case job, ok := <-archiveChannel:
            if !ok {
                return              // line 122: NO Done() - but channel closed naturally
            }
            // Multiple Done() calls on error paths - CORRECT
            if _, err := jobRepo.FetchMetadata(job); err {
                archivePending.Done()  // line 131 - OK
                continue
            }
            // ... more error paths with Done() ...
        }
    }
}
```

When `shutdownCtx` is cancelled (line 245 in `Shutdown()`) while a job is being processed:
- Line 115 fires
- `archivingWorker()` exits (line 117)
- **The job being processed has `archivePending.Add(1)` but never gets `Done()`**
- `archivePending.Wait()` (line 231) hangs forever

This directly causes **ISSUE 4 (deadlock on Shutdown timeout)**.

**Root Cause:** Context cancellation path doesn't decrement the pending counter for the job that was in-flight.

---

### **TASKMANAGER SUBSYSTEM ISSUES**

#### **ISSUE 7: CRITICAL - Nil Scheduler Panic if Shutdown Called Without Start**

**File/Line References:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/taskmanager/taskManager.go:52-57` (globals)
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/taskmanager/taskManager.go:105-145` (Start)
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/taskmanager/taskManager.go:165-169` (Shutdown)

**Problem:**
TaskManager stores a scheduler in a package-level variable with zero value:

```go
var (
    s       gocron.Scheduler          // line 53: ZERO VALUE (nil interface{})
    jobRepo *repository.JobRepository // line 54: NIL
    Keys    CronFrequency             // line 56: ZERO STRUCT
)

func Start(cronCfg, archiveConfig json.RawMessage) {  // line 105
    var err error
    jobRepo = repository.GetJobRepository()
    s, err = gocron.NewScheduler()    // line 108: initializes global s
    if err != nil {
        cclog.Abortf("Taskmanager Start: Could not create gocron scheduler.\nError: %s\n", err.Error())
    }
    // ... register jobs ...
    s.Start()                         // line 144
}

func Shutdown() {  // line 165
    if s != nil {  // line 166: CHECK FOR NIL
        s.Shutdown()
    }
}
```

**Failure Scenario:**

1. If `Start()` is never called, `s` remains nil
2. If `Shutdown()` is called, it safely checks `s != nil` (line 166) - **THIS CHECK WORKS**
3. **BUT** if `Start()` is called, fails at line 110 (context cancelled or config error), and then `Shutdown()` is called, the nil check passes but `s` is partially initialized (could be in an invalid state)

More critically: **If `Start()` is called twice** (no re-initialization guard):

```go
// First Start()
s, err = gocron.NewScheduler()  // line 108: creates scheduler S1
RegisterFootprintWorker()        // line 136: adds job to S1
s.Start()                        // line 144: S1 starts

// Code attempts to re-initialize (e.g., hot reload config)
// Second Start()
s, err = gocron.NewScheduler()  // line 108: OVERWRITES with S2
// ... registers jobs to S2 ...
s.Start()                        // line 144: S2 starts, but S1 is still running!
```

Result: **Two schedulers running concurrently**, each with separate job lists, leading to duplicate job executions or resource exhaustion.

**Root Cause:** No `sync.Once` guard on `Start()`; global `s` can be reinitialized.

---

#### **ISSUE 8: SEVERE - Resource Leak on Failed Start (Aborted Process)**

**File/Line References:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/taskmanager/taskManager.go:108-111`

**Problem:**
If scheduler creation fails:

```go
s, err = gocron.NewScheduler()  // line 108
if err != nil {
    cclog.Abortf("Taskmanager Start: Could not create gocron scheduler.\nError: %s\n", err.Error())  // line 110
}
```

`cclog.Abortf()` **kills the entire process** without cleanup:
- Active archiver goroutine is left running
- Archiver has queued jobs in `archiveChannel` (if it was started before taskmanager)
- HTTP server goroutine is still running
- No graceful shutdown of other subsystems

The process dies abruptly, potentially leaving:
- Incomplete database transactions
- Stale lock files
- Queued archive jobs never persisted to database

**Root Cause:** `Abort()` call prevents graceful shutdown sequence; should return error instead.

---

#### **ISSUE 9: MODERATE - Missing Validation of Start Preconditions**

**File/Line References:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/taskmanager/taskManager.go:105-145`

**Problem:**
`Start()` depends on global state being initialized:

```go
func Start(cronCfg, archiveConfig json.RawMessage) {  // line 105
    var err error
    jobRepo = repository.GetJobRepository()           // line 107: assumes config.Keys is set

    dec := json.NewDecoder(bytes.NewReader(cronCfg))  // line 117: uses cronCfg
    // ...
    if config.Keys.StopJobsExceedingWalltime > 0 { // line 113: reads global config
        RegisterStopJobsExceedTime()
    }
```

If `config.Init()` was never called (in main.go line 107), `config.Keys` is zero-valued and all config checks return incorrect defaults. No validation ensures prerequisites are met before starting taskmanager.

---

### **INTEGRATION AND SHUTDOWN SEQUENCING ISSUES**

#### **ISSUE 10: SEVERE - No Archiver Shutdown in main.go Signal Handler**

**File/Line References:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/main.go:402-418`
- `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/server.go:455-505`

**Problem:**
The signal handler in main.go calls `taskmanager.Shutdown()` (line 417) but **NOT** `archiver.Shutdown()`:

```go
// main.go, signal handler (line 406-418)
go func() {
    defer wg.Done()
    select {
    case <-sigs:
        cclog.Info("Shutdown signal received")
    case <-ctx.Done():
    }

    runtime.SystemdNotify(false, "Shutting down ...")
    srv.Shutdown(ctx)              // line 415: HTTP server
    util.FsWatcherShutdown()        // line 416: file watcher
    taskmanager.Shutdown()          // line 417: task manager
    // MISSING: archiver.Shutdown() HERE
}()
```

**BUT** `server.Shutdown()` DOES call `archiver.Shutdown()` (server.go line 489):

```go
// server.go, Server.Shutdown() method (line 488-492)
wg.Go(func() {
    if err := archiver.Shutdown(60 * time.Second); err != nil {
        cclog.Warnf("Archiver shutdown: %v", err)
    }
})
```

**Race Condition:**
1. Signal handler calls `srv.Shutdown(ctx)` (main.go line 415)
2. Server.Shutdown() calls `archiver.Shutdown()` (server.go line 489)
3. But signal handler returns and `wg.Done()` is called (main.go line 407)
4. If `srv.Shutdown()` completes before `archiver.Shutdown()` completes, the server goroutine may exit
5. Then main waits on `wg` to complete (main.go line 430), but if archiver is still shutting down, the timing is off

**More critically:** The `server.Shutdown()` is passed `ctx`, but at line 466:

```go
shutdownCtx, cancel := context.WithTimeout(ctx, 60*time.Second)  // line 466
```

A **new 60-second timeout** is created. But `archiver.Shutdown()` at line 489 also gets a 60-second timeout **independently**. If both hit their timeouts, the process can be stuck for 120 seconds.

**Root Cause:** Archiver shutdown is buried inside `Server.Shutdown()` instead of being in the main signal handler alongside other subsystems.

---

#### **ISSUE 11: MODERATE - No Timeout on taskmanager.Shutdown()**

**File/Line References:**
- `/Users/jan/prg/ClusterCockpit/cc-backend/cmd/cc-backend/main.go:417`
- `/Users/jan/prg/ClusterCockpit/cc-backend/internal/taskmanager/taskManager.go:165-169`

**Problem:**
`taskmanager.Shutdown()` has no timeout mechanism:

```go
func Shutdown() {  // line 165 - NO timeout parameter
    if s != nil {
        s.Shutdown()  // line 167: no timeout, blocks until scheduler fully stopped
    }
}
```

If a cron job hangs (e.g., database deadlock in `RegisterFootprintWorker` at line 147), `s.Shutdown()` will block indefinitely. The signal handler in main.go (line 417) will hang, and the process won't exit.

Contrast with `archiver.Shutdown(timeout)` which takes a timeout parameter.

**Root Cause:** TaskManager's shutdown doesn't support graceful timeout like archiver does.

---

### **SUMMARY TABLE**

| Issue | Severity | Component | Impact | File:Line |
|-------|----------|-----------|--------|-----------|
| Nil channel send race | CRITICAL | Archiver | Send panic if TriggerArchiving() called during Start() | archiveWorker.go:87-92,193 |
| Double-close on reinitialization | CRITICAL | Archiver | Panic on Shutdown if Start called twice | archiveWorker.go:85-92,226 |
| Double-close on shutdown | CRITICAL | Archiver | Panic if channels already closed | archiveWorker.go:110-122,226 |
| Nil workerDone panic | CRITICAL | Archiver | Panic on Shutdown if Start never called | archiveWorker.go:68,240,247 |
| Data loss on timeout | SEVERE | Archiver | Incomplete archives, inconsistent DB state | archiveWorker.go:115-117,137 |
| Missing Done() on context cancel | SEVERE | Archiver | Deadlock in Shutdown waiting for pending jobs | archiveWorker.go:115-117,231 |
| Double-scheduler leak | CRITICAL | TaskManager | Duplicate cron executions, resource waste | taskManager.go:108,144 |
| Process abort on init failure | SEVERE | TaskManager | No graceful shutdown if scheduler creation fails | taskManager.go:110 |
| Signal handler missing archiver | SEVERE | Integration | Archiver shutdown timing issues | main.go:417 |
| No taskmanager timeout | MODERATE | TaskManager | Process can hang if cron job stalls | taskManager.go:167 |
| Unvalidated preconditions | MODERATE | Both | Silent failures if initialization order violated | main.go:376-378 |

Prompt:
I'm doing a code review of the cc-backend Go project focused on CRITICAL and SEVERE issues in initialization, startup, and shutdown.

Investigate the lifecycle of the archiver and taskmanager subsystems. Read these and report exact code with file:line references:

1. `internal/archiver/` — the `Start(jobRepo, ctx)` function and `Shutdown(timeout)` function. How are goroutines/channels started? Is there a worker pool? What happens to in-flight or queued archive jobs on Shutdown? Can Shutdown be called when Start was never called (nil channel / closed channel)? Can `archiver.Shutdown` deadlock or panic? Is there a risk of send-on-closed-channel or double-close?

2. `internal/taskmanager/` — `Start(cronCfg, archiveCfg)` and `Shutdown()`. What does it manage (cron jobs)? Does Shutdown wait for running tasks? Any global/singleton state that breaks if Start not called?

3. Look for any package-level singletons, `sync.Once`, or global variables in these packages that affect re-initialization or shutdown safety.

Report concrete problems: nil-channel sends, double-close panics, goroutine leaks, missing waits, data loss of queued work on shutdown, ignored errors. Quote the relevant code with file:line. Be thorough but only report what you can verify in the code. Do not propose fixes — just identify and explain issues with evidence.


The agents produced many findings, but agent reports tend to include false positives and theoretical issues that don't occur in the real call flow. I need to verify the high-impact claims myself before reporting them. Let me check the most consequential ones: the auth-disabled fatal, the archiver shutdown accounting, and the NATS shutdown ordering.


180
181	func Init(authCfg *json.RawMessage) {
182		initOnce.Do(func() {
183			authInstance = &Authentication{}
184
185			// Start background cleanup of rate limiters
186			startRateLimiterCleanup()
187
188			sessKey := os.Getenv("SESSION_KEY")
189			if sessKey == "" {
190				cclog.Fatal("environment variable 'SESSION_KEY' not set: refusing to start with an ephemeral session key. " +
191					"Set SESSION_KEY in .env (base64-encoded 32 random bytes); a random key would invalidate all sessions on every restart " +
192					"and prevent sessions from validating across replicas.")
193			}
194			keyBytes, err := base64.StdEncoding.DecodeString(sessKey)
195			if err != nil {
196				cclog.Fatal("Error while initializing authentication -> decoding session key failed")
197			}
198			authInstance.sessionStore = sessions.NewCookieStore(keyBytes)
199
200			if d, err := time.ParseDuration(config.Keys.SessionMaxAge); err == nil {
201				authInstance.SessionMaxAge = d
202			}
203
204			if authCfg == nil {
205				return
206			}
207
208			config.Validate(configSchema, *authCfg)
209			dec := json.NewDecoder(bytes.NewReader(*authCfg))
210			dec.DisallowUnknownFields()
211			if err := dec.Decode(&Keys); err != nil {
212				cclog.Errorf("error while decoding ldap config: %v", err)
213			}
214
215			if Keys.LdapConfig != nil {
216				ldapAuth := &LdapAuthenticator{}
217				if err := ldapAuth.Init(); err != nil {
218					cclog.Warn("Error while initializing authentication -> ldapAuth init failed")
219				} else {
220					authInstance.LdapAuth = ldapAuth
221					authInstance.authenticators = append(authInstance.authenticators, authInstance.LdapAuth)
222				}
223			} else {
224				cclog.Info("Missing LDAP configuration: No LDAP support!")
225			}
226
227			if Keys.JwtConfig != nil {
228				authInstance.JwtAuth = &JWTAuthenticator{}
229				if err := authInstance.JwtAuth.Init(); err != nil {
230					cclog.Fatal("Error while initializing authentication -> jwtAuth init failed")
231				}
232
233				jwtSessionAuth := &JWTSessionAuthenticator{}
234				if err := jwtSessionAuth.Init(); err != nil {
235					cclog.Info("jwtSessionAuth init failed: No JWT login support!")
236				} else {
237					authInstance.authenticators = append(authInstance.authenticators, jwtSessionAuth)
238				}
239
240				jwtCookieSessionAuth := &JWTCookieSessionAuthenticator{}
241				if err := jwtCookieSessionAuth.Init(); err != nil {
242					cclog.Info("jwtCookieSessionAuth init failed: No JWT cookie login support!")
243				} else {
244					authInstance.authenticators = append(authInstance.authenticators, jwtCookieSessionAuth)
245				}
246			} else {
247				cclog.Info("Missing JWT configuration: No JWT token support!")
248			}
249
250			authInstance.LocalAuth = &LocalAuthenticator{}
251			if err := authInstance.LocalAuth.Init(); err != nil {
252				cclog.Fatal("Error while initializing authentication -> localAuth init failed")
253			}
254			authInstance.authenticators = append(authInstance.authenticators, authInstance.LocalAuth)
255		})
256	}
257
258	func GetAuthInstance() *Authentication {
259		if authInstance == nil {
260			cclog.Fatal("Authentication module not initialized!")
261		}
262
263		return authInstance
264	}
265
266	// handleUserSync syncs or updates a user in the database based on configuration.
267	// This is used for LDAP, JWT and OIDC authentications when syncUserOnLogin or updateUserOnLogin is enabled.
268	func handleUserSync(user *schema.User, syncUserOnLogin, updateUserOnLogin bool) {
269		r := repository.GetUserRepository()

func Init(authCfg *json.RawMessage) {
	initOnce.Do(func() {
		authInstance = &Authentication{}

		// Start background cleanup of rate limiters
		startRateLimiterCleanup()

		sessKey := os.Getenv("SESSION_KEY")
		if sessKey == "" {
			cclog.Fatal("environment variable 'SESSION_KEY' not set: refusing to start with an ephemeral session key. " +
				"Set SESSION_KEY in .env (base64-encoded 32 random bytes); a random key would invalidate all sessions on every restart " +
				"and prevent sessions from validating across replicas.")
		}
		keyBytes, err := base64.StdEncoding.DecodeString(sessKey)
		if err != nil {
			cclog.Fatal("Error while initializing authentication -> decoding session key failed")
		}
		authInstance.sessionStore = sessions.NewCookieStore(keyBytes)

		if d, err := time.ParseDuration(config.Keys.SessionMaxAge); err == nil {
			authInstance.SessionMaxAge = d
		}

		if authCfg == nil {
			return
		}

		config.Validate(configSchema, *authCfg)
		dec := json.NewDecoder(bytes.NewReader(*authCfg))
		dec.DisallowUnknownFields()
		if err := dec.Decode(&Keys); err != nil {
			cclog.Errorf("error while decoding ldap config: %v", err)
		}

		if Keys.LdapConfig != nil {
			ldapAuth := &LdapAuthenticator{}
			if err := ldapAuth.Init(); err != nil {
				cclog.Warn("Error while initializing authentication -> ldapAuth init failed")
			} else {
				authInstance.LdapAuth = ldapAuth
				authInstance.authenticators = append(authInstance.authenticators, authInstance.LdapAuth)
			}
		} else {
			cclog.Info("Missing LDAP configuration: No LDAP support!")
		}

		if Keys.JwtConfig != nil {
			authInstance.JwtAuth = &JWTAuthenticator{}
			if err := authInstance.JwtAuth.Init(); err != nil {
				cclog.Fatal("Error while initializing authentication -> jwtAuth init failed")
			}

			jwtSessionAuth := &JWTSessionAuthenticator{}
			if err := jwtSessionAuth.Init(); err != nil {
				cclog.Info("jwtSessionAuth init failed: No JWT login support!")
			} else {
				authInstance.authenticators = append(authInstance.authenticators, jwtSessionAuth)
			}

			jwtCookieSessionAuth := &JWTCookieSessionAuthenticator{}
			if err := jwtCookieSessionAuth.Init(); err != nil {
				cclog.Info("jwtCookieSessionAuth init failed: No JWT cookie login support!")
			} else {
				authInstance.authenticators = append(authInstance.authenticators, jwtCookieSessionAuth)
			}
		} else {
			cclog.Info("Missing JWT configuration: No JWT token support!")
		}

		authInstance.LocalAuth = &LocalAuthenticator{}
		if err := authInstance.LocalAuth.Init(); err != nil {
			cclog.Fatal("Error while initializing authentication -> localAuth init failed")
		}
		authInstance.authenticators = append(authInstance.authenticators, authInstance.LocalAuth)
	})
}

func GetAuthInstance() *Authentication {
	if authInstance == nil {
		cclog.Fatal("Authentication module not initialized!")
	}

	return authInstance
}

// handleUserSync syncs or updates a user in the database based on configuration.
// This is used for LDAP, JWT and OIDC authentications when syncUserOnLogin or updateUserOnLogin is enabled.
func handleUserSync(user *schema.User, syncUserOnLogin, updateUserOnLogin bool) {
	r := repository.GetUserRepository()


1	// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
2	// All rights reserved. This file is part of cc-backend.
3	// Use of this source code is governed by a MIT-style
4	// license that can be found in the LICENSE file.
5
6	// Package archiver provides asynchronous job archiving functionality for ClusterCockpit.
7	//
8	// The archiver runs a background worker goroutine that processes job archiving requests
9	// from a buffered channel. When jobs complete, their metric data is archived from the
10	// metric store to the configured archive backend (filesystem, S3, etc.).
11	//
12	// # Architecture
13	//
14	// The archiver uses a producer-consumer pattern:
15	//   - Producer: TriggerArchiving() sends jobs to archiveChannel
16	//   - Consumer: archivingWorker() processes jobs from the channel
17	//   - Coordination: sync.WaitGroup tracks pending archive operations
18	//
19	// # Lifecycle
20	//
21	//  1. Start(repo, ctx) - Initialize worker with context for cancellation
22	//  2. TriggerArchiving(job) - Queue job for archiving (called when job stops)
23	//  3. archivingWorker() - Background goroutine processes jobs
24	//  4. Shutdown(timeout) - Graceful shutdown with timeout
25	//
26	// # Graceful Shutdown
27	//
28	// The archiver supports graceful shutdown with configurable timeout:
29	//   - Closes channel to reject new jobs
30	//   - Waits for pending jobs to complete (up to timeout)
31	//   - Cancels context if timeout exceeded
32	//   - Ensures worker goroutine exits cleanly
33	//
34	// # Example Usage
35	//
36	//	// Initialize archiver
37	//	ctx, cancel := context.WithCancel(context.Background())
38	//	defer cancel()
39	//	archiver.Start(jobRepository, ctx)
40	//
41	//	// Trigger archiving when job completes
42	//	archiver.TriggerArchiving(job)
43	//
44	//	// Graceful shutdown with 10 second timeout
45	//	if err := archiver.Shutdown(10 * time.Second); err != nil {
46	//	    log.Printf("Archiver shutdown timeout: %v", err)
47	//	}
48	package archiver
49
50	import (
51		"context"
52		"fmt"
53		"sync"
54		"time"
55
56		"github.com/ClusterCockpit/cc-backend/internal/repository"
57		cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
58		"github.com/ClusterCockpit/cc-lib/v2/schema"
59		sq "github.com/Masterminds/squirrel"
60	)
61
62	var (
63		archivePending sync.WaitGroup
64		archiveChannel chan *schema.Job
65		jobRepo        *repository.JobRepository
66		shutdownCtx    context.Context
67		shutdownCancel context.CancelFunc
68		workerDone     chan struct{}
69	)
70
71	// Start initializes the archiver and starts the background worker goroutine.
72	//
73	// The archiver processes job archiving requests asynchronously via a buffered channel.
74	// Jobs are sent to the channel using TriggerArchiving() and processed by the worker.
75	//
76	// Parameters:
77	//   - r: JobRepository instance for database operations
78	//   - ctx: Context for cancellation (shutdown signal propagation)
79	//
80	// The worker goroutine will run until:
81	//   - ctx is cancelled (via parent shutdown)
82	//   - archiveChannel is closed (via Shutdown())
83	//
84	// Must be called before TriggerArchiving(). Safe to call only once.
85	func Start(r *repository.JobRepository, ctx context.Context) {
86		shutdownCtx, shutdownCancel = context.WithCancel(ctx)
87		archiveChannel = make(chan *schema.Job, 128)
88		workerDone = make(chan struct{})
89		jobRepo = r
90
91		go archivingWorker()
92	}
93
94	// archivingWorker is the background goroutine that processes job archiving requests.
95	//
96	// The worker loop:
97	//  1. Blocks waiting for jobs on archiveChannel or shutdown signal
98	//  2. Fetches job metadata from repository
99	//  3. Archives job data to configured backend (calls ArchiveJob)
100	//  4. Updates job footprint and energy metrics in database
101	//  5. Marks job as successfully archived
102	//  6. Calls job stop hooks
103	//
104	// The worker exits when:
105	//   - shutdownCtx is cancelled (timeout during shutdown)
106	//   - archiveChannel is closed (normal shutdown)
107	//
108	// Errors during archiving are logged and the job is marked as failed,
109	// but the worker continues processing other jobs.
110	func archivingWorker() {
111		defer close(workerDone)
112
113		for {
114			select {
115			case <-shutdownCtx.Done():
116				cclog.Info("Archive worker received shutdown signal")
117				return
118
119			case job, ok := <-archiveChannel:
120				if !ok {
121					cclog.Info("Archive channel closed, worker exiting")
122					return
123				}
124
125				start := time.Now()
126				// not using meta data, called to load JobMeta into Cache?
127				// will fail if job meta not in repository
128				if _, err := jobRepo.FetchMetadata(job); err != nil {
129					cclog.Errorf("archiving job (dbid: %d) failed at check metadata step: %s", *job.ID, err.Error())
130					jobRepo.UpdateMonitoringStatus(*job.ID, schema.MonitoringStatusArchivingFailed)
131					archivePending.Done()
132					continue
133				}
134
135				// ArchiveJob will fetch all the data from a MetricDataRepository and push into configured archive backend
136				// Use shutdown context to allow cancellation
137				jobMeta, err := ArchiveJob(job, shutdownCtx)
138				if err != nil {
139					cclog.Errorf("archiving job (dbid: %d) failed at archiving job step: %s", *job.ID, err.Error())
140					jobRepo.UpdateMonitoringStatus(*job.ID, schema.MonitoringStatusArchivingFailed)
141					archivePending.Done()
142					continue
143				}
144
145				stmt := sq.Update("job").Where("job.id = ?", job.ID)
146
147				if stmt, err = jobRepo.UpdateFootprint(stmt, jobMeta); err != nil {
148					cclog.Errorf("archiving job (dbid: %d) failed at update Footprint step: %s", *job.ID, err.Error())
149					archivePending.Done()
150					continue
151				}
152				if stmt, err = jobRepo.UpdateEnergy(stmt, jobMeta); err != nil {
153					cclog.Errorf("archiving job (dbid: %d) failed at update Energy step: %s", *job.ID, err.Error())
154					archivePending.Done()
155					continue
156				}
157				// Update the jobs database entry one last time:
158				stmt = jobRepo.MarkArchived(stmt, schema.MonitoringStatusArchivingSuccessful)
159				if err := jobRepo.Execute(stmt); err != nil {
160					cclog.Errorf("archiving job (dbid: %d) failed at db execute: %s", *job.ID, err.Error())
161					archivePending.Done()
162					continue
163				}
164				cclog.Debugf("archiving job %d took %s", job.JobID, time.Since(start))
165				cclog.Infof("archiving job (dbid: %d) successful", *job.ID)
166
167				repository.CallJobStopHooks(job)
168				archivePending.Done()
169			}
170		}
171	}
172
173	// TriggerArchiving queues a job for asynchronous archiving.
174	//
175	// This function should be called when a job completes (stops) to archive its
176	// metric data from the metric store to the configured archive backend.
177	//
178	// The function:
179	//  1. Increments the pending job counter (WaitGroup)
180	//  2. Sends the job to the archiving channel (buffered, capacity 128)
181	//  3. Returns immediately (non-blocking unless channel is full)
182	//
183	// The actual archiving is performed asynchronously by the worker goroutine.
184	// Upon completion, the worker will decrement the pending counter.
185	//
186	// Panics if Start() has not been called first.
187	func TriggerArchiving(job *schema.Job) {
188		if archiveChannel == nil {
189			cclog.Fatal("Cannot archive without archiving channel. Did you Start the archiver?")
190		}
191
192		archivePending.Add(1)
193		archiveChannel <- job
194	}
195
196	// Shutdown performs a graceful shutdown of the archiver with a configurable timeout.
197	//
198	// The shutdown process:
199	//  1. Closes archiveChannel - no new jobs will be accepted
200	//  2. Waits for pending jobs to complete (up to timeout duration)
201	//  3. If timeout is exceeded:
202	//     - Cancels shutdownCtx to interrupt ongoing ArchiveJob operations
203	//     - Returns error indicating timeout
204	//  4. Waits for worker goroutine to exit cleanly
205	//
206	// Parameters:
207	//   - timeout: Maximum duration to wait for pending jobs to complete
208	//     (recommended: 10-30 seconds for production)
209	//
210	// Returns:
211	//   - nil if all jobs completed within timeout
212	//   - error if timeout was exceeded (some jobs may not have been archived)
213	//
214	// Jobs that don't complete within the timeout will be marked as failed.
215	// The function always ensures the worker goroutine exits before returning.
216	//
217	// Example:
218	//
219	//	if err := archiver.Shutdown(10 * time.Second); err != nil {
220	//	    log.Printf("Some jobs did not complete: %v", err)
221	//	}
222	func Shutdown(timeout time.Duration) error {
223		cclog.Info("Initiating archiver shutdown...")
224
225		// Close channel to signal no more jobs will be accepted
226		close(archiveChannel)
227
228		// Create a channel to signal when all jobs are done
229		done := make(chan struct{})
230		go func() {
231			archivePending.Wait()
232			close(done)
233		}()
234
235		// Wait for jobs to complete or timeout
236		select {
237		case <-done:
238			cclog.Info("All archive jobs completed successfully")
239			// Wait for worker to exit
240			<-workerDone
241			return nil
242		case <-time.After(timeout):
243			cclog.Warn("Archiver shutdown timeout exceeded, cancelling remaining operations")
244			// Cancel any ongoing operations
245			shutdownCancel()
246			// Wait for worker to exit
247			<-workerDone
248			return fmt.Errorf("archiver shutdown timeout after %v", timeout)
249		}
250	}
251

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package archiver provides asynchronous job archiving functionality for ClusterCockpit.
//
// The archiver runs a background worker goroutine that processes job archiving requests
// from a buffered channel. When jobs complete, their metric data is archived from the
// metric store to the configured archive backend (filesystem, S3, etc.).
//
// # Architecture
//
// The archiver uses a producer-consumer pattern:
//   - Producer: TriggerArchiving() sends jobs to archiveChannel
//   - Consumer: archivingWorker() processes jobs from the channel
//   - Coordination: sync.WaitGroup tracks pending archive operations
//
// # Lifecycle
//
//  1. Start(repo, ctx) - Initialize worker with context for cancellation
//  2. TriggerArchiving(job) - Queue job for archiving (called when job stops)
//  3. archivingWorker() - Background goroutine processes jobs
//  4. Shutdown(timeout) - Graceful shutdown with timeout
//
// # Graceful Shutdown
//
// The archiver supports graceful shutdown with configurable timeout:
//   - Closes channel to reject new jobs
//   - Waits for pending jobs to complete (up to timeout)
//   - Cancels context if timeout exceeded
//   - Ensures worker goroutine exits cleanly
//
// # Example Usage
//
//	// Initialize archiver
//	ctx, cancel := context.WithCancel(context.Background())
//	defer cancel()
//	archiver.Start(jobRepository, ctx)
//
//	// Trigger archiving when job completes
//	archiver.TriggerArchiving(job)
//
//	// Graceful shutdown with 10 second timeout
//	if err := archiver.Shutdown(10 * time.Second); err != nil {
//	    log.Printf("Archiver shutdown timeout: %v", err)
//	}
package archiver

import (
	"context"
	"fmt"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/repository"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	sq "github.com/Masterminds/squirrel"
)

var (
	archivePending sync.WaitGroup
	archiveChannel chan *schema.Job
	jobRepo        *repository.JobRepository
	shutdownCtx    context.Context
	shutdownCancel context.CancelFunc
	workerDone     chan struct{}
)

// Start initializes the archiver and starts the background worker goroutine.
//
// The archiver processes job archiving requests asynchronously via a buffered channel.
// Jobs are sent to the channel using TriggerArchiving() and processed by the worker.
//
// Parameters:
//   - r: JobRepository instance for database operations
//   - ctx: Context for cancellation (shutdown signal propagation)
//
// The worker goroutine will run until:
//   - ctx is cancelled (via parent shutdown)
//   - archiveChannel is closed (via Shutdown())
//
// Must be called before TriggerArchiving(). Safe to call only once.
func Start(r *repository.JobRepository, ctx context.Context) {
	shutdownCtx, shutdownCancel = context.WithCancel(ctx)
	archiveChannel = make(chan *schema.Job, 128)
	workerDone = make(chan struct{})
	jobRepo = r

	go archivingWorker()
}

// archivingWorker is the background goroutine that processes job archiving requests.
//
// The worker loop:
//  1. Blocks waiting for jobs on archiveChannel or shutdown signal
//  2. Fetches job metadata from repository
//  3. Archives job data to configured backend (calls ArchiveJob)
//  4. Updates job footprint and energy metrics in database
//  5. Marks job as successfully archived
//  6. Calls job stop hooks
//
// The worker exits when:
//   - shutdownCtx is cancelled (timeout during shutdown)
//   - archiveChannel is closed (normal shutdown)
//
// Errors during archiving are logged and the job is marked as failed,
// but the worker continues processing other jobs.
func archivingWorker() {
	defer close(workerDone)

	for {
		select {
		case <-shutdownCtx.Done():
			cclog.Info("Archive worker received shutdown signal")
			return

		case job, ok := <-archiveChannel:
			if !ok {
				cclog.Info("Archive channel closed, worker exiting")
				return
			}

			start := time.Now()
			// not using meta data, called to load JobMeta into Cache?
			// will fail if job meta not in repository
			if _, err := jobRepo.FetchMetadata(job); err != nil {
				cclog.Errorf("archiving job (dbid: %d) failed at check metadata step: %s", *job.ID, err.Error())
				jobRepo.UpdateMonitoringStatus(*job.ID, schema.MonitoringStatusArchivingFailed)
				archivePending.Done()
				continue
			}

			// ArchiveJob will fetch all the data from a MetricDataRepository and push into configured archive backend
			// Use shutdown context to allow cancellation
			jobMeta, err := ArchiveJob(job, shutdownCtx)
			if err != nil {
				cclog.Errorf("archiving job (dbid: %d) failed at archiving job step: %s", *job.ID, err.Error())
				jobRepo.UpdateMonitoringStatus(*job.ID, schema.MonitoringStatusArchivingFailed)
				archivePending.Done()
				continue
			}

			stmt := sq.Update("job").Where("job.id = ?", job.ID)

			if stmt, err = jobRepo.UpdateFootprint(stmt, jobMeta); err != nil {
				cclog.Errorf("archiving job (dbid: %d) failed at update Footprint step: %s", *job.ID, err.Error())
				archivePending.Done()
				continue
			}
			if stmt, err = jobRepo.UpdateEnergy(stmt, jobMeta); err != nil {
				cclog.Errorf("archiving job (dbid: %d) failed at update Energy step: %s", *job.ID, err.Error())
				archivePending.Done()
				continue
			}
			// Update the jobs database entry one last time:
			stmt = jobRepo.MarkArchived(stmt, schema.MonitoringStatusArchivingSuccessful)
			if err := jobRepo.Execute(stmt); err != nil {
				cclog.Errorf("archiving job (dbid: %d) failed at db execute: %s", *job.ID, err.Error())
				archivePending.Done()
				continue
			}
			cclog.Debugf("archiving job %d took %s", job.JobID, time.Since(start))
			cclog.Infof("archiving job (dbid: %d) successful", *job.ID)

			repository.CallJobStopHooks(job)
			archivePending.Done()
		}
	}
}

// TriggerArchiving queues a job for asynchronous archiving.
//
// This function should be called when a job completes (stops) to archive its
// metric data from the metric store to the configured archive backend.
//
// The function:
//  1. Increments the pending job counter (WaitGroup)
//  2. Sends the job to the archiving channel (buffered, capacity 128)
//  3. Returns immediately (non-blocking unless channel is full)
//
// The actual archiving is performed asynchronously by the worker goroutine.
// Upon completion, the worker will decrement the pending counter.
//
// Panics if Start() has not been called first.
func TriggerArchiving(job *schema.Job) {
	if archiveChannel == nil {
		cclog.Fatal("Cannot archive without archiving channel. Did you Start the archiver?")
	}

	archivePending.Add(1)
	archiveChannel <- job
}

// Shutdown performs a graceful shutdown of the archiver with a configurable timeout.
//
// The shutdown process:
//  1. Closes archiveChannel - no new jobs will be accepted
//  2. Waits for pending jobs to complete (up to timeout duration)
//  3. If timeout is exceeded:
//     - Cancels shutdownCtx to interrupt ongoing ArchiveJob operations
//     - Returns error indicating timeout
//  4. Waits for worker goroutine to exit cleanly
//
// Parameters:
//   - timeout: Maximum duration to wait for pending jobs to complete
//     (recommended: 10-30 seconds for production)
//
// Returns:
//   - nil if all jobs completed within timeout
//   - error if timeout was exceeded (some jobs may not have been archived)
//
// Jobs that don't complete within the timeout will be marked as failed.
// The function always ensures the worker goroutine exits before returning.
//
// Example:
//
//	if err := archiver.Shutdown(10 * time.Second); err != nil {
//	    log.Printf("Some jobs did not complete: %v", err)
//	}
func Shutdown(timeout time.Duration) error {
	cclog.Info("Initiating archiver shutdown...")

	// Close channel to signal no more jobs will be accepted
	close(archiveChannel)

	// Create a channel to signal when all jobs are done
	done := make(chan struct{})
	go func() {
		archivePending.Wait()
		close(done)
	}()

	// Wait for jobs to complete or timeout
	select {
	case <-done:
		cclog.Info("All archive jobs completed successfully")
		// Wait for worker to exit
		<-workerDone
		return nil
	case <-time.After(timeout):
		cclog.Warn("Archiver shutdown timeout exceeded, cancelling remaining operations")
		// Cancel any ongoing operations
		shutdownCancel()
		// Wait for worker to exit
		<-workerDone
		return fmt.Errorf("archiver shutdown timeout after %v", timeout)
	}
}


Now let me verify the two most consequential claims: whether the auth-disabled path actually fatals at server startup, and how the NATS ingestion shutdown ordering actually behaves.


Bash

cd /Users/jan/prg/ClusterCockpit/cc-backend; echo "=== auth.Init call sites ==="; grep -rn "auth.Init\|auth\.GetAuthInstance" cmd/ internal/ | grep -v _test; echo; echo "=== DisableAuthentication usages ==="; grep -rn "DisableAuthentication" cmd/ internal/ | grep -v _test


=== auth.Init call sites ===
cmd/cc-backend/main.go:177:			auth.Init(&cfg)
cmd/cc-backend/main.go:180:			auth.Init(nil)
cmd/cc-backend/main.go:198:		authHandle := auth.GetAuthInstance()
cmd/cc-backend/server.go:123:	authHandle := auth.GetAuthInstance()
internal/taskmanager/ldapSyncService.go:24:	auth := auth.GetAuthInstance()
internal/api/rest.go:72:		Authentication:  auth.GetAuthInstance(),

=== DisableAuthentication usages ===
cmd/cc-backend/server.go:209:	if !config.Keys.DisableAuthentication {
cmd/cc-backend/server.go:248:		if !config.Keys.DisableAuthentication {
cmd/cc-backend/server.go:279:			if !config.Keys.DisableAuthentication {
cmd/cc-backend/server.go:289:			if !config.Keys.DisableAuthentication {
cmd/cc-backend/server.go:300:		if !config.Keys.DisableAuthentication {
cmd/cc-backend/server.go:311:		if !config.Keys.DisableAuthentication {
cmd/cc-backend/server.go:321:		if !config.Keys.DisableAuthentication {
cmd/cc-backend/main.go:171:	if config.Keys.DisableAuthentication && (flagNewUser != "" || flagDelUser != "") {
cmd/cc-backend/main.go:175:	if !config.Keys.DisableAuthentication {
internal/config/config.go:32:	DisableAuthentication bool `json:"disable-authentication"`


1	// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
2	// All rights reserved. This file is part of cc-backend.
3	// Use of this source code is governed by a MIT-style
4	// license that can be found in the LICENSE file.
5
6	// This file implements ingestion of InfluxDB line-protocol metric data received
7	// over NATS. Each line encodes one metric sample with the following structure:
8	//
9	//	<measurement>[,cluster=<c>][,hostname=<h>][,type=<t>][,type-id=<id>][,stype=<s>][,stype-id=<id>] value=<v> [<timestamp>]
10	//
11	// The measurement name identifies the metric (e.g. "cpu_load"). Tags provide
12	// routing information (cluster, host) and optional sub-device selectors (type,
13	// stype). Only one field is expected per line: "value".
14	//
15	// After decoding, each sample is:
16	//  1. Written to the in-memory store via ms.WriteToLevel.
17	//  2. If the checkpoint format is "wal", also forwarded to the WAL staging
18	//     goroutine via the WALMessages channel for durable write-ahead logging.
19	package metricstore
20
21	import (
22		"bytes"
23		"context"
24		"fmt"
25		"strings"
26		"sync"
27		"sync/atomic"
28		"time"
29
30		cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
31		"github.com/ClusterCockpit/cc-lib/v2/nats"
32		"github.com/ClusterCockpit/cc-lib/v2/schema"
33		"github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
34	)
35
36	// ReceiveNats subscribes to all configured NATS subjects and feeds incoming
37	// line-protocol messages into the MemoryStore.
38	//
39	// When workers > 1 a pool of goroutines drains a shared channel so that
40	// multiple messages can be decoded in parallel. With workers == 1 the NATS
41	// callback decodes inline (no channel overhead, lower latency).
42	//
43	// The function blocks until ctx is cancelled and all worker goroutines have
44	// finished. It returns nil when the NATS client is not configured; callers
45	// should treat that as a no-op rather than an error.
46	func ReceiveNats(ms *MemoryStore,
47		workers int,
48		ctx context.Context,
49	) error {
50		nc := nats.GetClient()
51
52		if nc == nil {
53			cclog.Warn("NATS client not initialized")
54			return nil
55		}
56
57		var wg sync.WaitGroup
58		msgs := make(chan []byte, max(workers*256, 8192))
59
60		for _, sc := range *Keys.Subscriptions {
61			clusterTag := sc.ClusterTag
62			if workers > 1 {
63				wg.Add(workers)
64
65				for range workers {
66					go func() {
67						defer wg.Done()
68						for m := range msgs {
69							dec := lineprotocol.NewDecoderWithBytes(m)
70							if err := DecodeLine(dec, ms, clusterTag); err != nil {
71								cclog.Errorf("error: %s", err.Error())
72							}
73						}
74					}()
75				}
76
77				nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
78					select {
79					case msgs <- data:
80					case <-ctx.Done():
81					}
82				})
83			} else {
84				nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
85					dec := lineprotocol.NewDecoderWithBytes(data)
86					if err := DecodeLine(dec, ms, clusterTag); err != nil {
87						cclog.Errorf("error: %s", err.Error())
88					}
89				})
90			}
91			cclog.Infof("NATS subscription to '%s' established", sc.SubscribeTo)
92		}
93
94		go func() {
95			<-ctx.Done()
96			close(msgs)
97		}()
98
99		wg.Wait()
100
101		return nil
102	}
103
104	// reorder prepends prefix to buf in-place when buf has enough spare capacity,
105	// avoiding an allocation. Falls back to a regular append otherwise.
106	//
107	// It is used to assemble the "type<type-id>" and "stype<stype-id>" selector
108	// strings when the type tag arrives before the type-id tag in the line, so the
109	// two byte slices need to be concatenated in tag-declaration order regardless
110	// of wire order.
111	func reorder(buf, prefix []byte) []byte {
112		n := len(prefix)
113		m := len(buf)
114		if cap(buf) < m+n {
115			return append(prefix[:n:n], buf...)
116		} else {
117			buf = buf[:n+m]
118			for i := m - 1; i >= 0; i-- {
119				buf[i+n] = buf[i]
120			}
121			for i := range n {
122				buf[i] = prefix[i]
123			}
124			return buf
125		}
126	}
127
128	// walDropped counts WAL messages dropped due to a full channel.
129	// Logged periodically; not critical since binary snapshots capture the data.
130	var walDropped atomic.Int64

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// This file implements ingestion of InfluxDB line-protocol metric data received
// over NATS. Each line encodes one metric sample with the following structure:
//
//	<measurement>[,cluster=<c>][,hostname=<h>][,type=<t>][,type-id=<id>][,stype=<s>][,stype-id=<id>] value=<v> [<timestamp>]
//
// The measurement name identifies the metric (e.g. "cpu_load"). Tags provide
// routing information (cluster, host) and optional sub-device selectors (type,
// stype). Only one field is expected per line: "value".
//
// After decoding, each sample is:
//  1. Written to the in-memory store via ms.WriteToLevel.
//  2. If the checkpoint format is "wal", also forwarded to the WAL staging
//     goroutine via the WALMessages channel for durable write-ahead logging.
package metricstore

import (
	"bytes"
	"context"
	"fmt"
	"strings"
	"sync"
	"sync/atomic"
	"time"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/nats"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	"github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
)

// ReceiveNats subscribes to all configured NATS subjects and feeds incoming
// line-protocol messages into the MemoryStore.
//
// When workers > 1 a pool of goroutines drains a shared channel so that
// multiple messages can be decoded in parallel. With workers == 1 the NATS
// callback decodes inline (no channel overhead, lower latency).
//
// The function blocks until ctx is cancelled and all worker goroutines have
// finished. It returns nil when the NATS client is not configured; callers
// should treat that as a no-op rather than an error.
func ReceiveNats(ms *MemoryStore,
	workers int,
	ctx context.Context,
) error {
	nc := nats.GetClient()

	if nc == nil {
		cclog.Warn("NATS client not initialized")
		return nil
	}

	var wg sync.WaitGroup
	msgs := make(chan []byte, max(workers*256, 8192))

	for _, sc := range *Keys.Subscriptions {
		clusterTag := sc.ClusterTag
		if workers > 1 {
			wg.Add(workers)

			for range workers {
				go func() {
					defer wg.Done()
					for m := range msgs {
						dec := lineprotocol.NewDecoderWithBytes(m)
						if err := DecodeLine(dec, ms, clusterTag); err != nil {
							cclog.Errorf("error: %s", err.Error())
						}
					}
				}()
			}

			nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
				select {
				case msgs <- data:
				case <-ctx.Done():
				}
			})
		} else {
			nc.Subscribe(sc.SubscribeTo, func(subject string, data []byte) {
				dec := lineprotocol.NewDecoderWithBytes(data)
				if err := DecodeLine(dec, ms, clusterTag); err != nil {
					cclog.Errorf("error: %s", err.Error())
				}
			})
		}
		cclog.Infof("NATS subscription to '%s' established", sc.SubscribeTo)
	}

	go func() {
		<-ctx.Done()
		close(msgs)
	}()

	wg.Wait()

	return nil
}

// reorder prepends prefix to buf in-place when buf has enough spare capacity,
// avoiding an allocation. Falls back to a regular append otherwise.
//
// It is used to assemble the "type<type-id>" and "stype<stype-id>" selector
// strings when the type tag arrives before the type-id tag in the line, so the
// two byte slices need to be concatenated in tag-declaration order regardless
// of wire order.
func reorder(buf, prefix []byte) []byte {
	n := len(prefix)
	m := len(buf)
	if cap(buf) < m+n {
		return append(prefix[:n:n], buf...)
	} else {
		buf = buf[:n+m]
		for i := m - 1; i >= 0; i-- {
			buf[i+n] = buf[i]
		}
		for i := range n {
			buf[i] = prefix[i]
		}
		return buf
	}
}

// walDropped counts WAL messages dropped due to a full channel.
// Logged periodically; not critical since binary snapshots capture the data.
var walDropped atomic.Int64


I've now verified the key claims directly. The agents over-reported (many findings are theoretical or flag-gated non-issues), but two are real and important. Let me confirm one more detail — the unconditional auth dependency in `api.New()`.


60		// RepositoryMutex protects job creation operations from race conditions
61		// when checking for duplicate jobs during startJob API calls.
62		// It prevents concurrent job starts with the same jobId/cluster/startTime
63		// from creating duplicate entries in the database.
64		RepositoryMutex sync.Mutex
65	}
66
67	// New creates and initializes a new RestAPI instance with configured dependencies.
68	func New() *RestAPI {
69		return &RestAPI{
70			JobRepository:   repository.GetJobRepository(),
71			MachineStateDir: config.Keys.MachineStateDir,
72			Authentication:  auth.GetAuthInstance(),
73		}
74	}
75
76	// MountAPIRoutes registers REST API endpoints for job and cluster management.
77	// These routes use JWT token authentication via the X-Auth-Token header.
78	func (api *RestAPI) MountAPIRoutes(r chi.Router) {
79		// REST API Uses TokenAuth

	// RepositoryMutex protects job creation operations from race conditions
	// when checking for duplicate jobs during startJob API calls.
	// It prevents concurrent job starts with the same jobId/cluster/startTime
	// from creating duplicate entries in the database.
	RepositoryMutex sync.Mutex
}

// New creates and initializes a new RestAPI instance with configured dependencies.
func New() *RestAPI {
	return &RestAPI{
		JobRepository:   repository.GetJobRepository(),
		MachineStateDir: config.Keys.MachineStateDir,
		Authentication:  auth.GetAuthInstance(),
	}
}

// MountAPIRoutes registers REST API endpoints for job and cluster management.
// These routes use JWT token authentication via the X-Auth-Token header.
func (api *RestAPI) MountAPIRoutes(r chi.Router) {
	// REST API Uses TokenAuth


Verified. Now I'll write the review report and remediation plan to the plan file.


Write

# Code Review — Initialization, Startup & Shutdown (critical/severe focus)

## Context

Requested: a thorough review of cc-backend focused on **critical and severe**
issues, starting with the initialization / startup / shutdown paths
(`cmd/cc-backend/main.go`, `cmd/cc-backend/server.go`, and the subsystem
`Init`/`Start`/`Shutdown` functions). Three exploration passes were run; their
findings were then **verified against the actual code** — many reported items were
theoretical or did not occur in the real call flow and are listed as rejected at
the bottom so they are not re-investigated.

This file is a review report plus a prioritized remediation plan. The previously
applied WAL-on-shutdown fix (`metricstore.Shutdown` now rotates WAL via
`RotateWALFilesAfterShutdown`) is already merged and is **not** re-listed.

---

## VERIFIED findings

### F1 — CRITICAL: `disable-authentication: true` aborts the server on startup
The server unconditionally requires an initialized auth instance, but auth is
only initialized when authentication is **enabled**.

- `auth.Init(...)` is called *only* inside the `if !config.Keys.DisableAuthentication`
  block — `cmd/cc-backend/main.go:175-181` (the only two call sites, confirmed by grep).
- `auth.GetAuthInstance()` is called **unconditionally** during server init:
  - `cmd/cc-backend/server.go:123` (`authHandle := auth.GetAuthInstance()`)
  - `internal/api/rest.go:72` via `api.New()` (`cmd/cc-backend/server.go:184`)
- `GetAuthInstance()` calls `cclog.Fatal("Authentication module not initialized!")`
  when `authInstance == nil` — `internal/auth/auth.go:258-264`.

Result: with `disable-authentication: true` and `-server`, the process hard-aborts
at startup. The flag is otherwise honored in 7 places in `server.go`, so it is a
supported configuration that is currently broken. Default deployments
(auth enabled) are unaffected, which is why it has gone unnoticed.

**Fix direction**: ensure auth is always initialized, or guard the unconditional
callers. Cleanest: when `config.Keys.DisableAuthentication` is true, still call
`auth.Init(nil)` (it sets up a session store and returns early without
authenticators — `auth.go:181-206`), OR make `GetAuthInstance()` callers tolerate
a nil instance when auth is disabled. Preferred: call `auth.Init(nil)` for the
disabled case in `handleUserCommands`/startup so `authInstance` is non-nil. Verify
no auth-enabled-only middleware is wired when disabled (it already is guarded).

### F2 — SEVERE: startup-error path skips graceful shutdown
In `runServer` (`cmd/cc-backend/main.go:392-463`), if `srv.Start` returns an error
(e.g. port in use, `tls.LoadX509KeyPair` failure, privileged-port bind), the error
is delivered on `errChan`, the `select` returns it, `run()` returns, and `main()`
calls `os.Exit(1)` (`main.go:589-594`).

- The graceful-shutdown sequence (`metricstore.Shutdown()` final checkpoint + WAL
  rotation, `archiver.Shutdown()`, `taskmanager.Shutdown()`) lives only in the
  signal-handler goroutine (`main.go:406-418`), which is unblocked via `ctx.Done()`
  from the deferred `cancel()` (`main.go:584`) — but `os.Exit(1)` races ahead and
  the process dies before that goroutine runs.
- The metricstore background goroutines started in `metricstore.Init`
  (`Retention`, `Checkpointing`, `WALStaging`, `ReceiveNats`) are tied to the
  store's own internal context and are never cancelled on this path.

Practical data-loss risk is low (the listener fails at startup before any ingest),
but "graceful shutdown" is silently not performed on the error path, and it is a
latent correctness gap if `Serve` ever returns mid-run.

**Fix direction**: on the error return from `runServer`, invoke the same shutdown
sequence (or trigger the signal goroutine and wait for `waitDone`) before
returning, so the final checkpoint runs.

---

## Lower-severity (worth fixing, not blocking)

- **M1 — port-80 redirect goroutine swallows its error** (`server.go:417-419`):
  `go func(){ http.ListenAndServe(":80", ...) }()` ignores the returned error; a
  failed bind on :80 is completely silent. Log it.
- **M2 — archiver shutdown leaks its waiter goroutine and skips a `Done()` on
  timeout** (`internal/archiver/archiveWorker.go:222-249`, `:115-117`): on the
  timeout branch `shutdownCancel()` makes the worker `return` without
  `archivePending.Done()` for the in-flight job, so the
  `go func(){ archivePending.Wait(); close(done) }()` goroutine never returns.
  Only happens during shutdown timeout (process exiting), so impact is bounded.
- **M3 — NATS API workers/channels not closed on shutdown**
  (`internal/api/nats.go` — `jobWorker`/`nodeWorker` loop over `jobCh`/`nodeCh`,
  never closed; created at `server.go:330-333`): goroutines block forever. This is
  a one-time leak at process shutdown (not per-request/per-restart), so low impact,
  but buffered job/node events in those channels are dropped silently.
- **M4 — `metricstore.Shutdown()` is not idempotent** (`metricstore.go:274-321`):
  `shutdownFunc` is not nil'd after use and a second call would re-run
  `ToCheckpointWAL` + `RotateWALFilesAfterShutdown`. Currently only called once
  (`server.go:484`), so latent only.

---

## Remediation plan (recommended order)

1. **F1** — initialize auth in the disabled case (call `auth.Init(nil)` on the
   startup path when `DisableAuthentication` is true), then smoke-test
   `-server -dev` with `disable-authentication: true`. Critical, small change in
   `cmd/cc-backend/main.go`.
2. **F2** — run the graceful-shutdown sequence on the `runServer` error path before
   returning. Small change in `cmd/cc-backend/main.go`.
3. **M1–M4** — opportunistic hardening (log ignored errors; close NATS API
   channels with a stop signal; balance archiver `Done()`; nil `shutdownFunc`).

Each fix must respect the change-impact rules in CLAUDE.md (trace callers, check
goroutine/channel lifecycle).

## Verification

- Build: `go build ./...`; `go vet ./...`.
- F1: start with a config containing `"disable-authentication": true` and
  `-server -dev`; confirm the server starts and serves `/` without the
  "Authentication module not initialized!" fatal. Re-confirm a normal
  auth-enabled start still works (login flow).
- F2: force a startup failure (e.g. bind an already-used `addr`, or point
  `https-cert-file` at a missing file) and confirm the metricstore writes its
  final checkpoint / rotates WAL before exit (check logs + `var/checkpoints`).
- Regression: `go test ./...` (esp. `./pkg/metricstore/...`, `./internal/archiver/...`).

---

## Rejected during verification (false positives — do not chase)

- **"DB migration runs before Connect(), causing multiple connections / lock
  contention."** `handleDatabaseCommands` (`main.go:139-168`) only acts when
  `-migrate-db`/`-revert-db`/`-force-db` is set, and each path ends in
  `cclog.Exitf` (process exits). In normal server startup it is a no-op. Not an issue.
- **"`GetUserRepository()`/`GetConnection()` called before `Connect()`."** Order in
  `run()` is correct: `initDatabase()` (`main.go:512`) precedes both
  `flagOptimizeDB` (`:518`) and `handleUserCommands` (`:531`).
- **Archiver "double-Start", "TriggerArchiving before Start", "nil `workerDone`
  panic".** `Start` is called exactly once (`main.go:377`) before any trigger; the
  worker `Done()` accounting on the normal/error paths is actually correct
  (`archiveWorker.go:128-168`). Only the timeout edge (M2) is real.
- **"NatsAPI 10 goroutine leaks per restart cycle."** There is one process
  lifetime per run; it is a single shutdown-time leak (M3), not per-restart.
- **NATS-close-before-ctx-cancel "panic / mass data loss".** `nc.Close()`
  (`server.go:461`) stops callbacks before `metricstore.Shutdown()` runs; the
  `walShuttingDown` guard in `SendWALMessage` (`walCheckpoint.go:150-164`) prevents
  send-on-closed-channel, and buffered `msgs` are drained when the ctx is cancelled.
  Minor at most, not critical.