MemoryUsageTracker freed all buffers older than the retention cutoff on
every tick whenever the last retention pass had excluded used nodes,
regardless of memory pressure — discarding the history kept in memory
for long-running jobs. Buffers are now only reclaimed by the existing
emergency path once memory usage exceeds the configured cap. The
now-unused selectorsExcluded state is removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Setting the provider after Init left the checkpoint restore inside Init
without job information and forced callers to pre-call InitMetrics and
SetNodeProvider in the right order. Init now receives the provider as a
parameter and injects it before the restore, so the ordering is enforced
by the signature. SetNodeProvider remains for callers that do not run
Init (tests, -cleanup-checkpoints).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The -cleanup-checkpoints CLI flag called CleanupCheckpoints without
initializing the MemoryStore singleton or setting its NodeProvider,
so it silently fell back to legacy clean-everything behavior and
could delete checkpoints for hosts with running jobs. Inject the job
repository as NodeProvider the same way runServer does, so the CLI
path also skips used hosts. Also update SetNodeProvider's doc comment,
which only mentioned Free but is now also consulted by FromCheckpoint
and CleanupCheckpoints.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- auth: do not abort the server when authentication is disabled. auth.Init
is now always called; with disable-authentication it sets up an ephemeral
session store (SESSION_KEY not required) and registers no authenticators,
so the unconditional auth.GetAuthInstance() callers (server init,
api.New()) always get a valid instance.
- main: run the graceful-shutdown sequence on the startup-error path. runServer
derives a cancelable context and, on a server-start failure, cancels it and
waits so the metricstore final checkpoint / WAL rotation, archiver flush and
taskmanager shutdown actually run before exit.
- server: log the :80 HTTP->HTTPS redirect listener error instead of dropping it.
- archiver: guard Shutdown against being called when Start never ran
(avoids close(nil) panic / blocking on a nil workerDone).
- nats API: stop worker goroutines on shutdown via a stop channel + idempotent
Shutdown(); workers and subscription callbacks select on stop and the
channels are never closed, so no send-on-closed-channel can occur. Wired
into Server.Shutdown after the NATS client is closed.
- metricstore: make Shutdown idempotent (nil shutdownFunc, early return) and
release shutdownFuncMu before the checkpoint write.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 3c179f9caa8f
- Fix MemoryUsageTracker: remove premature bufferPool.Clear() that prevented
mem.Alloc from decreasing, replace broken ForceFree loop (100 iterations
with no GC) with progressive time-based Free at 75%/50%/25% retention,
add bufferPool.Clear()+GC between steps so memory stats update correctly
- Enable debug.FreeOSMemory() after emergency freeing to return memory to OS
- Add adaptive ticker: 30s checks when memory >80% of cap, normal otherwise
- Reduce default memory check interval from 1h to 5min
- Don't abort entire NATS batch on single write error (out-of-order timestamp),
log warning and continue processing remaining lines
- Prune empty levels from tree after free() to reduce overhead
- Include buffer struct overhead in sizeInBytes() for more accurate reporting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: 7ce28627fc1d
Aditya Ujeniya
moebiusband
