cc-backend/ReleaseNotes.md

cc-backend version 1.6.0

Supports job archive version 3 and database version 12.

This release replaces the browser session implementation and requires a database migration to version 12. Run ./cc-backend -migrate-db after upgrading; the new sessions table is created automatically (a fresh -init-db also creates it). Existing login sessions are invalidated by the upgrade, so users have to log in again once.

Two configuration changes are mandatory before starting 1.6.0:

• The .env file is gone. Move every secret into the auth section of config.json (or export the corresponding environment variable).
• The main.resampling section no longer accepts trigger, resolutions and minimum-points. cc-backend rejects unknown keys inside main and aborts on startup if they are still present. Replace them with default-policy and default-algo.

See the behavior changes below for the details, as well as for the session cookie Secure flag and the removal of the SESSION_KEY environment variable. For release specific notes visit the ClusterCockpit Documentation.

Changes in 1.6.0

Behavior changes

• Session backend replaced (gorilla/sessions → alexedwards/scs): Browser sessions are now managed by github.com/alexedwards/scs/v2 with server-side storage in the SQLite database (new sessions table, database version 12) instead of gorilla/sessions client-side cookie storage. Only an opaque random token is stored in the cookie; the session payload lives server-side. Sessions still survive backend restarts. This requires the database migration noted above and invalidates existing sessions.
• Session cookie Secure flag: The Secure attribute on the session cookie is now derived from the server configuration — it is set when cc-backend terminates TLS itself (https-cert-file and https-key-file configured) and unset otherwise. Previously the flag was effectively never set. Deployments that terminate TLS at a reverse proxy and want the Secure flag should serve cc-backend over HTTPS directly for now.
• SESSION_KEY removed: The SESSION_KEY environment variable is no longer used and can be dropped from your environment. Server-side sessions use random tokens, so no cookie-signing secret is required. It is ignored if left in place.
• .env support removed (breaking): The godotenv bootstrap, the .env file and configs/env-template.txt are gone. Secrets (JWT public/private key, LDAP sync password, OIDC client id/secret, cross-login keys) now live in config.json under auth next to the option that uses them. Each secret can still be supplied through its existing environment variable (JWT_PUBLIC_KEY, JWT_PRIVATE_KEY, LDAP_ADMIN_PASSWORD, OID_CLIENT_ID, OID_CLIENT_SECRET, CROSS_LOGIN_JWT_PUBLIC_KEY, CROSS_LOGIN_JWT_HS512_KEY), which takes precedence over the config value. -init now writes the demo JWT keys into config.json.
• Resampling configuration replaced by policies (breaking): The explicit trigger, resolutions and minimum-points keys under main.resampling are removed; main rejects unknown keys, so a config still carrying them aborts startup. Configure default-policy (low ≈ 200, medium ≈ 500, high ≈ 1000 target points) and default-algo instead. Both are also exposed as per-user settings, and the policy is now the single source for the requested resolution and for the resampler's minimum-points threshold.
• Default downsampling algorithm is now average: Previously an unset algorithm fell through to cc-lib's LTTB. Average consolidation makes every plotted point the true arithmetic mean of its interval; LTTB deliberately keeps extremes and cannot be read as interval means. Set main.resampling.default-algo to lttb to restore the old behavior.
• OIDC roles require an explicit mapping (breaking for role-based setups): The hardcoded translation of token roles (which only recognised the literal names user/admin/manager/support and silently dropped api) is replaced by auth.oidc.role-mapping. The mapping is the sole source of roles: unmapped token roles are ignored and users without a mapped role get the base user role. Deployments whose IdP emits literal CC role names must now map them explicitly (e.g. "admin": "admin"). Mapping targets are validated at startup.
• LDAP can assign elevated roles: The new optional auth.ldap.role-filters map (role → LDAP filter) grants admin/support/api/manager to accounts matching a filter. LDAP is authoritative for the roles listed there — sync and login both add and remove them to match group membership — while roles not listed are preserved (a managed manager with assigned projects is never stripped). Without role-filters the behavior is unchanged.
• Unrecognized top-level config sections are reported: cc-backend now logs a warning for unknown top-level keys in config.json, which most often means a setting was nested at the wrong level (e.g. resampling next to main instead of inside it) and was silently ignored before.
• The auth section is validated again: Its JSON schema was missing the enclosing type/properties wrapper, so it accepted every input and no auth option was ever checked. With the schema fixed, an incomplete auth section now aborts startup — jwts requires max-age, oidc requires provider, and ldap requires url, user-base, search-dn, user-bind and user-filter. Each of the three subsections stays optional on its own.
• archive.retention target keys: The retention target is configured with target-kind and target-path (or the target-* S3 keys). The location key used by the old example configuration was never read by the code, so a move or copy policy configured that way ran against an empty target path.

New features

• Configurable footer links: main.footer-links lets the Imprint and Privacy Policy footer entries point at internal pages (default) or external URLs. External http(s) targets open in a new tab; empty or unset values fall back to the built-in /imprint and /privacy routes, and the existing ./var/*.tmpl override mechanism keeps working.
• Display smoothing for metric plots: A centered, NaN-aware moving average can be applied in the plot layer, after whatever downsampling the backend performed. It preserves series length and index alignment, skips NaN/null samples, keeps all-NaN windows as gaps, and shrinks the window at the series edges. The window is a per-user setting in data points with a site default in ui.plot-configuration.smoothing-window (0 disables it). It is display-only: reported statistics and job footprints are unaffected.
• Metric tooltips: metricConfig entries in cluster.json accept an optional tooltip that is rendered in the metric selection dialog.
• Metric store statistics from cached aggregates: Buffers maintain running sum/min/max/count while data is written and compute them eagerly on checkpoint load, so a full-buffer statistics request is served from the cached aggregate instead of rescanning the samples.
• Checkpoint retention aware of running jobs: The metric store receives a NodeProvider (the job repository) at Init. Checkpoint cleanup now skips nodes that still have running jobs, the full checkpoint history is loaded for those nodes at startup, and this also applies to the -cleanup-checkpoints CLI path.

Bug fixes

• Metric plots and the policy-based resample config: MetricPlot still read the removed trigger/resolutions values, so the array-based resolution branch was unreachable and the zoom guard always fired. The zoom trigger is now derived from the policy's target point count.
• Resample algorithm for running jobs: The selected algorithm was not forwarded to the metric store when loading data for running jobs.
• Partially covered buffers: Requests that reach beyond the stored data now get NaN padding at the start/end instead of misaligned series (dataNotAligned).
• Data race in the statistics read path: Reading buffer statistics no longer mutates the buffer, and the fast path is guarded against empty buffers.
• Empty node levels leaked: The retention free path now prunes node levels that no longer hold any data, and no longer force-frees buffers that are retained for running jobs.
• Plot render options in the wrong place: PlotRenderOptions is only shown in the user settings again.
• Invisible navbar: Fixed a stuck animation that hid the navbar with certain browser settings.
• Subcluster dropdown: The subcluster selection is size-limited and scrolls instead of overflowing.
• OIDC login session handling: Fixed session handling on the OIDC login route.

Dependencies

• Added github.com/alexedwards/scs/v2 and github.com/alexedwards/scs/sqlite3store.
• Removed github.com/gorilla/sessions (and github.com/gorilla/securecookie), and github.com/joho/godotenv together with the .env support.
• Upgraded cc-lib (GraphQL code regenerated) and refreshed the remaining Go module dependencies.

Changes in 1.5.4

Supports job archive version 3 and database version 11.

This was a security and bugfix release of cc-backend, the API backend and frontend implementation of ClusterCockpit. If you are upgrading from v1.5.1 no database migration is required. If you are upgrading from v1.5.0 you need to do another DB migration. This should not take long. For optimal database performance after the migration it is recommended to apply the new optimize-db flag, which runs the sqlite ANALYZE and VACUUM commands. Depending on your database size (more then 40GB) the VACUUM may take up to 2h. You can also run the ANALYZE command manually. While we are confident that the memory issue with the metricstore cleanup move policy is fixed, it is still recommended to use delete policy for cleanup. This is also the default.

Security fixes

• JWT HMAC empty-key bypass (critical): jwtSession.go now refuses to register when CROSS_LOGIN_JWT_HS512_KEY is unset. Previously, an empty HMAC key allowed unauthenticated admin token forgery because golang-jwt verifies any HS256/HS512 signature against an empty key.
• SQL injection via metric names (critical): Metric names supplied through GraphQL ([String!]) were interpolated raw into json_extract SQL expressions. Names are now validated against ^[a-zA-Z0-9_]+$ in jobsMetricStatisticsHistogram and buildFloatJSONCondition.
• Path traversal via line-protocol tags (critical): cluster and host tags from the metric line protocol flowed unvalidated into path.Join for checkpoint/WAL file paths, enabling arbitrary file writes outside the checkpoint root via NATS (unauthenticated) or POST /api/write. Path-traversal sequences are now rejected in DecodeLine before the tags become path components.
• CORS AllowCredentials disabled: CORS middleware no longer sets AllowCredentials: true, which was incompatible with AllowedOrigins: ["*"] and could enable cross-origin credential theft.
• HSTS header added: Strict-Transport-Security is now set for all HTTPS connections.
• Security response headers: Added X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Referrer-Policy: same-origin, and a conservative Content-Security-Policy (blocks frame-ancestors, object-src, base-uri) to harden against clickjacking and base-tag injection.
• Stored XSS in job message: job.metaData.message is now rendered as escaped text (CSS white-space: pre-wrap) instead of raw {@html ...} in Job.root and JobFootprint.
• SQL injection in tag queries: The tag-scope IN list and manager project subquery in CountTags are now parameterized instead of interpolating user.Username / user.Projects values sourced from OIDC/LDAP.
• GraphQL DoS hardening: Query cost is bounded with FixedComplexityLimit (5000). Non-positive items-per-page and page values are rejected with HTTP 400 to prevent integer underflow into unbounded LIMIT/OFFSET queries.
• CSRF defense-in-depth: State-changing requests with a cross-site Sec-Fetch-Site header are now rejected (fails open for non-browser clients), complementing the existing SameSite=Lax session cookie.
• NATS API security warning: A startup warning is now logged when NATS subscriptions are enabled, reminding operators that the NATS API has no application-layer authentication and that publish ACLs must be restricted at the broker.

Bug fixes

• Roofline legend placement: Roofline plot legends now use fixed coordinates instead of dynamic placement, preventing the legend from overlapping data points or being rendered off-canvas (#546).
• Subcluster usage tab labels: Subcluster names in the status dashboard usage tabs are no longer force-capitalized; the original cluster-defined casing is preserved.
• NodeListRow host filter: The running-jobs query in the node list row now filters by exact node hostname (eq) instead of substring match (contains), avoiding incorrect matches when one hostname is a prefix of another.
• WAL files not reset on shutdown: On graceful shutdown the metricstore wrote a final binary snapshot but never rotated the per-host current.wal files (the RotateWALFilesAfterShutdown helper was defined but never called). The stale WAL files were replayed and then appended to again on the next start, so they grew without bound across restarts and were only ever reset at the next periodic checkpoint. Shutdown now rotates the WAL files for all successfully snapshotted hosts.

Dependencies

• Go module upgrades: Refreshed Go module dependencies to their latest compatible versions.

Changes in 1.5.3

Bug fixes

• OIDC role extraction: Fixed role extraction from OIDC tokens where roles were not correctly parsed from the token claims. Roles are now always requested from the token regardless of other configuration.

• OIDC user sync role changes: SyncUser and UpdateUser callbacks now allow all role changes, removing a restriction that prevented role updates during OIDC-driven user synchronization.

• OIDC projects array: Projects array from the OIDC token is now submitted and applied when syncing user attributes.

• WAL message drops during checkpoint: WAL writes are now paused during binary checkpoint creation. Previously, disk I/O contention between checkpoint writes and WAL staging caused over 1.4 million dropped messages per checkpoint cycle.

• WAL rotation skipped for all nodes: RotateWALFiles used a non-blocking send on a small channel. With thousands of nodes, the channel filled instantly and nearly all hosts were skipped, leaving WAL files unrotated. Replaced with a blocking send using a shared 2-minute deadline.

• Log viewer auto-refresh: Fixed the log viewer component not auto-refreshing correctly.

• SameSite cookie setting: Relaxed the SameSite cookie attribute to improve compatibility with OIDC redirect flows.

• WAL not rotated on partial checkpoint failure: When binary checkpointing failed for some hosts, WAL files for successfully checkpointed hosts were not rotated and the checkpoint timestamp was not advanced. Partial successes now correctly advance the checkpoint and rotate WAL files for completed hosts.

• Unbounded WAL file growth: If binary checkpointing consistently failed for a host, its current.wal file grew without limit until disk exhaustion. A new max-wal-size configuration option (in the checkpoints block) allows setting a per-host WAL size cap in bytes. When exceeded, the WAL is force-rotated. Defaults to 0 (unlimited) for backward compatibility.

• Doubleranged filter fixes: Range filters now correctly handle zero as a boundary value. Improved validation and UI text for "more than equal" and "less than equal" range selections.

• Lineprotocol body parsing interrupted: Switched from ReadTimeout to ReadHeaderTimeout so that long-running metric submissions are no longer cut off mid-stream.

• Checkpoint archiving continues on error: A single cluster's archiving failure no longer aborts the entire cleanup operation. Errors are collected and reported per cluster.

• Parquet row group overflow: Added periodic flush during checkpoint archiving to prevent exceeding the parquet-go 32k column-write limit.

• Removed metrics excluded from subcluster config: Metrics removed from a subcluster are no longer returned by GetMetricConfigSubCluster.

MetricStore performance

• WAL writer throughput: Decoupled WAL file flushing from message processing using a periodic 5-second batch flush (up to 4096 messages per cycle), significantly increasing metric ingestion throughput.
• Improved shutdown time: HTTP shutdown timeout reduced; metricstore and archiver now shut down concurrently. Overall shutdown deadline raised to 60 seconds.

New features

• Manual checkpoint cleanup flag: New -cleanup-checkpoints CLI flag triggers checkpoint cleanup without starting the server, useful for maintenance windows or automated cleanup scripts.
• Explicit node state queries in node view: Node health and scheduler state are now fetched independently from metric data for fresher status information.

Development tooling

• Make targets for formatting and linting: New make fmt and make lint targets using gofumpt and golangci-lint. Configuration added in .golangci.yml and gopls.json.

New tools

• binaryCheckpointReader: New utility tool (tools/binaryCheckpointReader) that reads .wal or .bin checkpoint files produced by the metricstore WAL/snapshot system and dumps their contents to a human-readable .txt file. Useful for debugging and inspecting checkpoint data. Usage: go run ./tools/binaryCheckpointReader <file.wal|file.bin>

Logging improvements

• Reduced tagger log noise: Missing metrics and expression evaluation errors in the job classification tagger are now logged at debug level instead of error level.

Changes in 1.5.2

Bug fixes

• Memory spike in parquet writer: Fixed memory spikes when using the metricstore move (archive) policy with the parquet writer. The writer now processes data in a streaming fashion to avoid accumulating large allocations.
• Top list query fixes: Fixed top list queries in analysis and dashboard views.
• Exclude down nodes from HealthCheck: Down nodes are now excluded from health checks in both the REST and NATS handlers.
• Node state priority order: Node state determination now enforces a priority order. Exception: idle+down results in idle.
• Blocking ReceiveNats call: Fixed a blocking NATS receive call in the metricstore.

Database performance

• Reduced insert pressure: Bulk insert operations (node state updates, user and job cache syncs) now use explicit transactions and deferred inserts, significantly reducing write contention on the SQLite database.
• SyncJobs wrapped in transaction: SyncJobs now runs inside a transaction for better consistency and reduced lock contention.
• Configurable busy timeout: New busy-timeout configuration option for the SQLite connection. This allows tuning how long the driver waits for a locked database before returning an error, which improves resilience under concurrent write load.
• Increased default SQLite timeout: The default SQLite connection timeout has been raised to reduce spurious timeout errors under load.
• Optimized stats queries: Improved sortby handling in stats queries, fixed cache key passing, and simplified a stats query condition that caused an expensive unnecessary subquery.

MetricStore performance

• Sharded WAL consumer: The WAL consumer is now sharded for significantly higher write throughput.
• NATS contention fix: Fixed contention in the metricstore NATS ingestion path.

NATS API

• Nodestate health checks in NATS API: The NATS node state handler now performs the same metric health checks as the REST API handler, including per-subcluster health checks and MonitoringStateFailed fallback for nodes without health data.

Logging improvements

• Better error context: Several repository functions now include the calling function name in error messages for easier diagnosis.
• Reduced log noise: ErrNoRows (no results found) is no longer logged as an error in scanRow; common "no rows" paths are now silent.
• Debug-level missing metrics: Warning about missing metrics in the metric store has been downgraded to debug level to reduce log noise in normal operation.
• Checkpoint archiving log: Added an informational log message when the metricstore checkpoint archiving process runs.
• Auth failure context: Auth failure log messages now include more context information.

Behavior changes

• DB-based metricHealth: Replaced heuristic-based metric health with DB-based metric health for the node view, providing more accurate health status information.
• Removed minRunningFor filter remnants: Cleaned up remaining minRunningFor references from the GraphQL schema and query builder.

Frontend

• Streamlined statsSeries: Unified stats series calculation and rendering across plot components.
• Clarified plot titles: Improved titles in dashboard and health views.
• Bumped frontend dependencies: Updated frontend dependencies to latest versions.

Dependencies

• cc-lib upgraded: Updated to latest cc-lib version.

Known issues (as of 1.5.2)

These were the open issues at the time of the 1.5.2 release and are kept here for that release's record; they are not necessarily still open. Note that the ui-config section referenced below is now called ui.

• The new dynamic memory management is not bullet proof yet across restarts. Buffers that are kept outside the retention period may be lost across a restart. We will fix that in a subsequent patch release.
• To use the new log viewer (which is only working when starting cc-backend with systemd) in the admin interface the user under which the cc-backend process is running has to be allowed to execute the journalctl command.
• The user configuration keys for the ui have changed. Therefore old user configuration persisted in the database is not used anymore. It is recommended to configure the metrics shown in the ui-config section and remove all records in the table after the update.
• Currently energy footprint metrics of type energy are ignored for calculating total energy.
• With energy footprint metrics of type power the unit is ignored and it is assumed the metric has the unit Watt.

Changes in 1.5.1

Database

• New migration (version 11): Optimized database index count and added covering indexes for stats queries for significantly improved query performance
• Migration 9 fix: Removed redundant indices from migration 9 that are superseded by migration 11
• Optional DB optimization flag: Added -optimize-db CLI flag to run ANALYZE on demand; removed automatic ANALYZE on startup
• Selective stats queries: Stats queries are now selective, reducing unnecessary computation
• User list paging: Added paging support to the user list for better scalability
• SQLite configuration hardening: Sanitized SQLite configuration with new configurable options; fixes large heap allocations in the SQLite driver
• Query cancellation: Long-running database queries can now be cancelled
• Resource leak fix: Added missing defer Close() calls for all query result sets

Bug fixes

• Segfault when taggers misconfigured: Fixed crash when enable-job-taggers is set but tagger rule directories are missing
• GroupBy stats query complexity: Reduced complexity for groupBy statistics queries
• Ranged filter conditions: Fixed GT and LT conditions in ranged filters
• Energy filter preset: Reduced energy filter preset to a more practical default
• JSON validity check: Fixed wrong field being checked for JSON validity
• Tagger float rounding: Fixed rounding of floats in tagger messages
• Node view null safety: Added null-safe checks in node view to prevent runtime errors
• Public dashboard null safety: Added null-safe checks in the public dashboard to prevent runtime errors

Frontend

• Bumped patch versions: Updated frontend dependencies to latest patch versions

Documentation

• New DB config options: Added new database configuration options to README

---

The sections below document all features and changes introduced in the 1.5.0 major release, which 1.5.1 is based on.

Breaking changes

Configuration changes

• JSON attribute naming: All JSON configuration attributes now use kebab-case style consistently (e.g., api-allowed-ips instead of apiAllowedIPs). Update your config.json accordingly.
• Removed disable-archive option: This obsolete configuration option has been removed.
• Removed clusters config section: The separate clusters configuration section has been removed. Cluster information is now derived from the job archive.
• apiAllowedIPs is now optional: If not specified, defaults to not restricted.

Architecture changes

• Web framework replaced: Migrated from gorilla/mux to chi as the HTTP router. This should be transparent to users but affects how middleware and routes are composed. A proper 404 handler is now in place.
• MetricStore moved: The metricstore package has been moved from internal/ to pkg/ as it is now part of the public API.
• MySQL/MariaDB support removed: Only SQLite is now supported as the database backend.
• Archive to Cleanup renaming: Archive-related functions have been refactored and renamed to "Cleanup" for clarity.
• minRunningFor filter removed: This undocumented filter has been removed from the API and frontend.

Dependency changes

• cc-lib v2.8.0: Switched to cc-lib version 2 with updated APIs
• cclib NATS client: Now using the cclib NATS client implementation
• Removed obsolete util.Float usage from cclib

Major new features

NATS API Integration

• Real-time job events: Subscribe to job start/stop events via NATS
• Node state updates: Receive real-time node state changes via NATS
• Configurable subjects: NATS API subjects are now configurable via api-subjects
• Deadlock fixes: Improved NATS client stability and graceful shutdown

Public Dashboard

• Public-facing interface: New public dashboard route for external users
• DoubleMetricPlot component: New visualization component for comparing metrics
• Improved layout: Reviewed and optimized dashboard layouts for better readability

Enhanced Node Management

• Node state tracking: New node table in database with timestamp tracking
• Node state filtering: Filter jobs by node state in systems view
• Node list enhancements: Improved paging, filtering, and continuous scroll support
• Nodestate retention and archiving: Node state data is now subject to configurable retention policies and can be archived to Parquet format for long-term storage
• Faulty node metric tracking: Faulty node state metric lists are persisted to the database

Health Monitoring

• Health status dashboard: New dedicated "Health" tab in the status details view showing per-node metric health across the cluster
• CCMS health check: Support for querying health status of external cc-metric-store (CCMS) instances via the API
• GraphQL health endpoints: New GraphQL queries and resolvers for health data
• Cluster/subcluster filter: Filter health status view by cluster or subcluster

Log Viewer

• Web-based log viewer: New log viewer page in the admin interface for inspecting backend log output directly from the browser without shell access
• Accessible from header: Quick access link from the navigation header

MetricStore Improvements

• Memory tracking worker: New worker for CCMS memory usage tracking
• Dynamic retention: Support for job specific dynamic retention times
• Improved compression: Transparent compression for job archive imports
• Parallel processing: Parallelized Iter function in all archive backends

Job Tagging System

• Job tagger option: Enable automatic job tagging via configuration flag
• Application detection: Automatic detection of applications (MATLAB, GROMACS, etc.)
• Job classification: Automatic detection of pathological jobs
• omit-tagged: Option to exclude tagged jobs from retention/cleanup operations (none, all, or user)
• Admin UI trigger: Taggers can be run on-demand from the admin web interface without restarting the backend

Archive Backends

• Parquet archive format: New Parquet file format for job archiving, providing columnar storage with efficient compression for analytical workloads
• S3 backend: Full support for S3-compatible object storage
• SQLite backend: Full support for SQLite backend using blobs
• Performance improvements: Fixed performance bugs in archive backends
• Better error handling: Improved error messages and fallback handling
• Zstd compression: Parquet writers use zstd compression for better compression ratios compared to the previous snappy default
• Optimized sort order: Job and nodestate Parquet files are sorted by cluster, subcluster, and start time for efficient range queries

Unified Archive Retention and Format Conversion

• Uniform retention policy: Job archive retention now supports both JSON and Parquet as target formats under a single, consistent policy configuration
• Archive manager tool: The tools/archive-manager utility now supports format conversion between JSON and Parquet job archives
• Parquet reader: Full Parquet archive reader implementation for reading back archived job data

New features and improvements

Frontend

• Loading indicators: Added loading indicators to status detail and job lists
• Job info layout: Reviewed and improved job info row layout
• Metric selection: Enhanced metric selection with drag-and-drop fixes
• Filter presets: Move list filter preset to URL for easy sharing
• Job comparison: Improved job comparison views and plots
• Subcluster reactivity: Job list now reacts to subcluster filter changes
• Short jobs quick selection: New "Short jobs" quick-filter button in job lists replaces the removed undocumented minRunningFor filter
• Row plot cursor sync: Cursor position is now synchronized across all metric plots in a job list row for easier cross-metric comparison
• Disabled metrics handling: Improved handling and display of disabled metrics across job view, node view, and list rows
• "Not configured" info cards: Informational cards shown when optional features are not yet configured
• Frontend dependencies: Bumped frontend dependencies to latest versions
• Svelte 5 compatibility: Fixed Svelte state warnings and compatibility issues

Backend

• Progress bars: Import function now shows progress during long operations
• Better logging: Improved logging with appropriate log levels throughout
• Graceful shutdown: Fixed shutdown timeout bugs and hanging issues
• Configuration defaults: Sensible defaults for most configuration options
• Documentation: Extensive documentation improvements across packages
• Server flag in systemd unit: Example systemd unit now includes the -server flag

Security

• LDAP security hardening: Improved input validation, connection handling, and error reporting in the LDAP authenticator
• OIDC security hardening: Stricter token validation and improved error handling in the OIDC authenticator
• Auth schema extensions: Additional schema fields for improved auth configuration

API improvements

• Role-based metric visibility: Metrics can now have role-based access control
• Job exclusivity filter: New filter for exclusive vs. shared jobs
• Improved error messages: Better error messages and documentation in REST API
• GraphQL enhancements: Improved GraphQL queries and resolvers
• Stop job lookup order: Reversed lookup order in stop job requests for more reliable job matching (cluster+jobId first, then jobId alone)

Performance

• Database indices: Optimized SQLite indices for better query performance
• Job cache: Introduced caching table for faster job inserts
• Parallel imports: Archive imports now run in parallel where possible
• External tool integration: Optimized use of external tools (fd) for better performance
• Node repository queries: Reviewed and optimized node repository SQL queries
• Buffer pool: Resized and pooled internal buffers for better memory reuse

Developer experience

• AI agent guidelines: Added documentation for AI coding agents (AGENTS.md, CLAUDE.md)
• Example API payloads: Added example JSON API payloads for testing
• Unit tests: Added more unit tests for NATS API, node repository, and other components
• Test improvements: Better test coverage; test DB is now copied before unit tests to avoid state pollution between test runs
• Parquet writer tests: Comprehensive tests for Parquet archive writing and conversion

Bug fixes

• Fixed nodelist paging issues
• Fixed metric select drag and drop functionality
• Fixed render race conditions in nodeList
• Fixed tag count grouping including type
• Fixed wrong metricstore schema (missing comma)
• Fixed configuration issues causing shutdown hangs
• Fixed deadlock when NATS is not configured
• Fixed archive backend performance bugs
• Fixed continuous scroll buildup on refresh
• Improved footprint calculation logic
• Fixed polar plot data query decoupling
• Fixed missing resolution parameter handling
• Fixed node table initialization fallback
• Fixed reactivity key placement in nodeList
• Fixed nodeList resolver data handling and increased nodestate filter cutoff
• Fixed job always being transferred to main job table before archiving
• Fixed AppTagger error handling and logging
• Fixed log endpoint formatting and correctness
• Fixed automatic refresh in metric status tab
• Fixed NULL value handling in health_state and health_metrics columns
• Fixed bugs related to job_cache IDs being used in the main job table
• Fixed SyncJobs bug causing start job hooks to be called with wrong (cache) IDs
• Fixed 404 handler route for sub-routers

Configuration changes

New configuration options

{
  "main": {
    "enable-job-taggers": true,
    "resampling": {
      "minimum-points": 600,
      "trigger": 180,
      "resolutions": [240, 60]
    },
    "api-subjects": {
      "subject-job-event": "cc.job.event",
      "subject-node-state": "cc.node.state"
    }
  },
  "nats": {
    "address": "nats://0.0.0.0:4222",
    "username": "root",
    "password": "root"
  },
  "cron": {
    "commit-job-worker": "1m",
    "duration-worker": "5m",
    "footprint-worker": "10m"
  },
  "metric-store": {
    "cleanup": {
      "mode": "archive",
      "interval": "48h",
      "directory": "./var/archive"
    }
  },
  "archive": {
    "retention": {
      "policy": "delete",
      "age": "6months",
      "target-format": "parquet"
    }
  },
  "nodestate": {
    "retention": {
      "policy": "archive",
      "age": "30d",
      "archive-path": "./var/nodestate-archive"
    }
  }
}

Migration notes

• Review and update your config.json to use kebab-case attribute names
• If using NATS, configure the new nats and api-subjects sections
• If using S3 archive backend, configure the new archive section options
• Test the new public dashboard at /public route
• Review cron worker configuration if you need different frequencies
• If using the archive retention feature, configure the target-format option to choose between json (default) and parquet output formats
• Consider enabling nodestate retention if you track node states over time