Commit Graph

29 Commits

Author SHA1 Message Date
Aditya Ujeniya ea2bfbb6aa fix: wire NodeProvider into CLI checkpoint cleanup path
The -cleanup-checkpoints CLI flag called CleanupCheckpoints without
initializing the MemoryStore singleton or setting its NodeProvider,
so it silently fell back to legacy clean-everything behavior and
could delete checkpoints for hosts with running jobs. Inject the job
repository as NodeProvider the same way runServer does, so the CLI
path also skips used hosts. Also update SetNodeProvider's doc comment,
which only mentioned Free but is now also consulted by FromCheckpoint
and CleanupCheckpoints.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 14:31:26 +02:00
Aditya Ujeniya 97ab828d60 feat: add isNodeUsed helper for NodeProvider lookups
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 14:01:37 +02:00
moebiusband 5d8d4e228e Merge branch 'main' into feature/526-average-resample 2026-06-17 06:30:54 +02:00
moebiusband 1b72b0b5ad Fix critical/severe issues in init, startup and shutdown
- auth: do not abort the server when authentication is disabled. auth.Init
  is now always called; with disable-authentication it sets up an ephemeral
  session store (SESSION_KEY not required) and registers no authenticators,
  so the unconditional auth.GetAuthInstance() callers (server init,
  api.New()) always get a valid instance.
- main: run the graceful-shutdown sequence on the startup-error path. runServer
  derives a cancelable context and, on a server-start failure, cancels it and
  waits so the metricstore final checkpoint / WAL rotation, archiver flush and
  taskmanager shutdown actually run before exit.
- server: log the :80 HTTP->HTTPS redirect listener error instead of dropping it.
- archiver: guard Shutdown against being called when Start never ran
  (avoids close(nil) panic / blocking on a nil workerDone).
- nats API: stop worker goroutines on shutdown via a stop channel + idempotent
  Shutdown(); workers and subscription callbacks select on stop and the
  channels are never closed, so no send-on-closed-channel can occur. Wired
  into Server.Shutdown after the NATS client is closed.
- metricstore: make Shutdown idempotent (nil shutdownFunc, early return) and
  release shutdownFuncMu before the checkpoint write.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 3c179f9caa8f
2026-06-05 10:16:28 +02:00
moebiusband 8733497223 Fix bug that prevents WAL cleanup on shutdown
Entire-Checkpoint: 5b79628feea4
2026-06-05 08:39:49 +02:00
moebiusband 66707bbf15 Update metricstore documentation
Entire-Checkpoint: 99f20c1edd90
2026-03-29 21:38:04 +02:00
Aditya Ujeniya 6e97ac8b28 Verbose logs for DataDoesNotAlign error in CCMS 2026-03-26 14:13:12 +01:00
moebiusband 97d65a9e5c Fix bugs in WAL journal pipeline
Entire-Checkpoint: 8fe0de4e6ac2
2026-03-26 07:25:36 +01:00
moebiusband e759810051 Add shutdown timings. Do not drain WAL buffers on shutdown
Entire-Checkpoint: d4b497002f54
2026-03-26 07:02:37 +01:00
moebiusband c0d2d65f96 Introduce average resampler support
Fixes #526

Entire-Checkpoint: 893a1de325b5
2026-03-19 21:16:48 +01:00
moebiusband 8b132ed7f8 fix: Blocking ReceiveNats call
Entire-Checkpoint: 38a235c86ceb
2026-03-18 06:47:45 +01:00
moebiusband bf1a8a174e fix: Shard WAL consumer for higher throughput
Entire-Checkpoint: e583b7b11439
2026-03-18 06:32:14 +01:00
moebiusband 50aed595cf fix: metricstore NATS contention
Entire-Checkpoint: 7e68050cab59
2026-03-18 06:14:15 +01:00
moebiusband 39ab12784c Make checkpointInterval an option config option again.
Also applies small fixes

Entire-Checkpoint: c11d1a65fae4
2026-03-13 09:07:38 +01:00
moebiusband 8234ad3126 fix: Fix metricstore memory explosion from broken emergency free and batch aborts
- Fix MemoryUsageTracker: remove premature bufferPool.Clear() that prevented
  mem.Alloc from decreasing, replace broken ForceFree loop (100 iterations
  with no GC) with progressive time-based Free at 75%/50%/25% retention,
  add bufferPool.Clear()+GC between steps so memory stats update correctly
- Enable debug.FreeOSMemory() after emergency freeing to return memory to OS
- Add adaptive ticker: 30s checks when memory >80% of cap, normal otherwise
- Reduce default memory check interval from 1h to 5min
- Don't abort entire NATS batch on single write error (out-of-order timestamp),
  log warning and continue processing remaining lines
- Prune empty levels from tree after free() to reduce overhead
- Include buffer struct overhead in sizeInBytes() for more accurate reporting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: 7ce28627fc1d
2026-03-13 07:57:35 +01:00
moebiusband 26982088c3 Consolidate code for external and internal ccms buildQueries function
Entire-Checkpoint: fc3be444ef4c
2026-03-04 16:43:05 +01:00
Aditya Ujeniya a243e17499 Update to shutdown worker for WAL checkpointing mode 2026-03-02 15:27:06 +01:00
moebiusband 1ec41d8389 Review and improve buffer pool implmentation. Add unit tests. 2026-02-28 19:34:33 +01:00
Aditya Ujeniya 07b989cb81 Add new bufferPool implementation 2026-02-27 14:44:32 +01:00
moebiusband a418abc7d5 Run go fix 2026-02-27 14:40:26 +01:00
moebiusband ca0f9a42c7 Introduce metric store binary checkpoints with write ahead log 2026-02-26 10:08:40 +01:00
Aditya Ujeniya 7101d2bb3b Handle the metric/host not found case differently 2026-01-28 17:47:38 +01:00
moebiusband 0d857b49a2 Disable explicit GC calls 2026-01-28 11:21:27 +01:00
moebiusband eb5aa9ad02 Disable explicit GC calls 2026-01-28 11:21:02 +01:00
moebiusband 9d15a87c88 Take into account the real allocated heap memory in MemoryUsageTracker 2026-01-27 18:23:09 +01:00
moebiusband bbde91a1f9 Move wg increment inside goroutines. Make GC calls less aggressive 2026-01-27 17:25:29 +01:00
moebiusband 752e19c276 Pull out metric List build from metricstore Init 2026-01-27 17:06:52 +01:00
moebiusband c782043c64 Upgrade cclib and remove usage of obsolete util.Float 2026-01-26 08:38:53 +01:00
moebiusband f41301036b Move metricstore from internal to pkg 2026-01-23 07:49:47 +01:00