- auth: do not abort the server when authentication is disabled. auth.Init
is now always called; with disable-authentication it sets up an ephemeral
session store (SESSION_KEY not required) and registers no authenticators,
so the unconditional auth.GetAuthInstance() callers (server init,
api.New()) always get a valid instance.
- main: run the graceful-shutdown sequence on the startup-error path. runServer
derives a cancelable context and, on a server-start failure, cancels it and
waits so the metricstore final checkpoint / WAL rotation, archiver flush and
taskmanager shutdown actually run before exit.
- server: log the :80 HTTP->HTTPS redirect listener error instead of dropping it.
- archiver: guard Shutdown against being called when Start never ran
(avoids close(nil) panic / blocking on a nil workerDone).
- nats API: stop worker goroutines on shutdown via a stop channel + idempotent
Shutdown(); workers and subscription callbacks select on stop and the
channels are never closed, so no send-on-closed-channel can occur. Wired
into Server.Shutdown after the NATS client is closed.
- metricstore: make Shutdown idempotent (nil shutdownFunc, early return) and
release shutdownFuncMu before the checkpoint write.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 3c179f9caa8f
Addresses the remaining medium findings from the second-pass audit:
- DoS hardening: bound GraphQL query cost with FixedComplexityLimit, and
reject non-positive items-per-page / page values so uint64 conversion
cannot underflow into an unbounded LIMIT/OFFSET. The -1 "load all"
sentinel stays valid for dashboards; REST now returns 400 for bad input.
- Security headers: add X-Content-Type-Options, X-Frame-Options,
Referrer-Policy and a conservative CSP (frame-ancestors/object-src/
base-uri) that hardens against clickjacking and base-tag injection
without restricting the self-hosted SPA's inline scripts.
- Stored XSS: render job.metaData.message as escaped text instead of
{@html ...} in Job.root and JobFootprint, preserving line breaks via
white-space: pre-wrap.
- SQL injection hardening: parameterize the tag-scope IN list and the
manager project subquery in CountTags instead of interpolating
user.Username / user.Projects (externally sourced via OIDC/LDAP).
- CSRF defense-in-depth: reject cross-site state-changing requests via
Sec-Fetch-Site, failing open for non-browser clients, on top of the
existing SameSite=Lax session cookie.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: de7d47a85c7c
A second-pass audit surfaced three severe issues missed by the previous
review, each a sibling code path of a bug class that was only partially
fixed before:
- auth: JWT session login (jwtSession.go) registered its authenticator
even when CROSS_LOGIN_JWT_HS512_KEY was unset, leaving an empty HMAC
key. golang-jwt verifies any HS256/HS512 signature against an empty
key, allowing unauthenticated admin token forgery. Init() now refuses
to register without a key, with a defense-in-depth empty-key guard in
the keyfunc.
- repository: metric names from GraphQL ([String!]) were interpolated
raw into json_extract(footprint, "$.<name>") SQL. SQLite parses
double-quoted strings as literals, enabling SQL injection by any
authenticated user. Validate metric names against ^[a-zA-Z0-9_]+$ in
jobsMetricStatisticsHistogram and buildFloatJSONCondition.
- metricstore: cluster/host line-protocol tags flowed unvalidated into
path.Join(RootDir, cluster, host) for checkpoint/WAL files, allowing
arbitrary file write outside the checkpoint root via NATS
(unauthenticated) or POST /api/write. Reject path-traversal sequences
in DecodeLine before the tags become path components.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: b57246993ec1
WAL writes during checkpoint are redundant since the binary snapshot
captures all in-memory data. Pausing eliminates channel saturation
(1.4M+ dropped messages) caused by disk I/O contention between
checkpoint writes and WAL staging. Also removes direct WAL file
deletion in checkpoint workers that raced with the staging goroutine.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: 34d698f40bac
RotateWALFiles used a non-blocking send (select/default) on rotation
channels buffered at 64. With thousands of nodes and few shards, the
channel fills instantly and nearly all hosts are skipped, leaving WAL
files unrotated indefinitely.
Replace with a blocking send using a shared 2-minute deadline so the
checkpoint goroutine waits for the staging goroutine to drain the
channel instead of immediately giving up.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: a1ec897216fa
Stream CheckpointFile trees directly to parquet rows instead of
materializing all rows in a giant intermediate slice. This eliminates
~1.9GB per host of redundant allocations (repeated string headers)
and removes the expensive sort on millions of 104-byte structs.
Key changes:
- Replace flattenCheckpointFile + sortParquetRows + WriteHostRows with
streaming WriteCheckpointFile that walks the tree with sorted keys
- Reduce results channel buffer from len(hostEntries) to 2 for
back-pressure (at most NumWorkers+2 results in flight)
- Workers send CheckpointFile trees instead of []ParquetMetricRow
- Write rows in small 1024-element batches via reusable buffer
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: f31dc1847539
These conversation transcript files were committed before the gitignore
rule existed. They are now properly ignored via .entire/.gitignore.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add migration 14: partial covering indexes WHERE job_state='running'
for user/project/subcluster groupings (tiny B-tree vs full table)
- Inline literal state value in BuildWhereClause so SQLite matches
partial indexes instead of parameterized placeholders
- Add per-request statsGroupCache (sync.Once per filter+groupBy key)
so identical grouped stats queries execute only once per GQL operation
- Parallelize 4 histogram queries in AddHistograms using errgroup
- Consolidate frontend from 6 GQL aliases to 2, sort+slice top-10
client-side via $derived
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: 5b26a6e5ff10
Merge three separate queries (topJobsQuery, topNodesQuery, topAccsQuery)
into one topStatsQuery with 6 aliased jobsStatistics fields, reducing
3 HTTP round trips to 1 on the status dashboard.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: 40d806a3240c
Adds composite covering indexes on (cluster, job_state, <group_col>, ...)
for user, project, and subcluster groupings to enable index-only scans
for status views. Drops subsumed 3-column indexes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: 3d8def28e96e
WAL writes now go through bufio.Writer instead of raw syscalls per record,
reducing I/O overhead. Buffers are flushed on rotate, drain, and shutdown.
Fixed misleading MemoryCap comment ("Max bytes" → "Max memory in GB").
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: b38dc35e5334
- Fix MemoryUsageTracker: remove premature bufferPool.Clear() that prevented
mem.Alloc from decreasing, replace broken ForceFree loop (100 iterations
with no GC) with progressive time-based Free at 75%/50%/25% retention,
add bufferPool.Clear()+GC between steps so memory stats update correctly
- Enable debug.FreeOSMemory() after emergency freeing to return memory to OS
- Add adaptive ticker: 30s checks when memory >80% of cap, normal otherwise
- Reduce default memory check interval from 1h to 5min
- Don't abort entire NATS batch on single write error (out-of-order timestamp),
log warning and continue processing remaining lines
- Prune empty levels from tree after free() to reduce overhead
- Include buffer struct overhead in sizeInBytes() for more accurate reporting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: 7ce28627fc1d
moebiusbandandClaude Opus 4.8