# `cc-backend` version 1.6.0

Supports job archive version 3 and database version 12.

This release replaces the browser session implementation and requires a database
migration to version 12. Run `./cc-backend -migrate-db` after upgrading; the new
`sessions` table is created automatically (a fresh `-init-db` also creates it).
Existing login sessions are invalidated by the upgrade, so users have to log in
again once.

**Two configuration changes are mandatory before starting 1.6.0:**

- The `.env` file is gone. Move every secret into the `auth` section of
  `config.json` (or export the corresponding environment variable).
- The `main.resampling` section no longer accepts `trigger`, `resolutions` and
  `minimum-points`. cc-backend rejects unknown keys inside `main` and aborts on
  startup if they are still present. Replace them with `default-policy` and
  `default-algo`.

See the behavior changes below for the details, as well as for the session
cookie `Secure` flag and the removal of the `SESSION_KEY` environment variable.
For release specific notes visit the [ClusterCockpit Documentation](https://clustercockpit.org/docs/release/).

## Changes in 1.6.0

### Behavior changes

- **Session backend replaced (`gorilla/sessions` → `alexedwards/scs`)**: Browser
  sessions are now managed by `github.com/alexedwards/scs/v2` with server-side
  storage in the SQLite database (new `sessions` table, database version 12)
  instead of `gorilla/sessions` client-side cookie storage. Only an opaque random
  token is stored in the cookie; the session payload lives server-side. Sessions
  still survive backend restarts. This requires the database migration noted above
  and invalidates existing sessions.
- **Session cookie `Secure` flag**: The `Secure` attribute on the session cookie
  is now derived from the server configuration — it is set when cc-backend
  terminates TLS itself (`https-cert-file` and `https-key-file` configured) and
  unset otherwise. Previously the flag was effectively never set. Deployments that
  terminate TLS at a reverse proxy and want the `Secure` flag should serve
  cc-backend over HTTPS directly for now.
- **`SESSION_KEY` removed**: The `SESSION_KEY` environment variable is no longer
  used and can be dropped from your environment. Server-side sessions use random
  tokens, so no cookie-signing secret is required. It is ignored if left in place.
- **`.env` support removed (breaking)**: The `godotenv` bootstrap, the `.env`
  file and `configs/env-template.txt` are gone. Secrets (JWT public/private key,
  LDAP sync password, OIDC client id/secret, cross-login keys) now live in
  `config.json` under `auth` next to the option that uses them. Each secret can
  still be supplied through its existing environment variable
  (`JWT_PUBLIC_KEY`, `JWT_PRIVATE_KEY`, `LDAP_ADMIN_PASSWORD`, `OID_CLIENT_ID`,
  `OID_CLIENT_SECRET`, `CROSS_LOGIN_JWT_PUBLIC_KEY`, `CROSS_LOGIN_JWT_HS512_KEY`),
  which takes precedence over the config value. `-init` now writes the demo JWT
  keys into `config.json`.
- **Resampling configuration replaced by policies (breaking)**: The explicit
  `trigger`, `resolutions` and `minimum-points` keys under `main.resampling` are
  removed; `main` rejects unknown keys, so a config still carrying them aborts
  startup. Configure `default-policy` (`low` ≈ 200, `medium` ≈ 500, `high` ≈ 1000
  target points) and `default-algo` instead. Both are also exposed as per-user
  settings, and the policy is now the single source for the requested resolution
  and for the resampler's minimum-points threshold.
- **Default downsampling algorithm is now `average`**: Previously an unset
  algorithm fell through to cc-lib's LTTB. Average consolidation makes every
  plotted point the true arithmetic mean of its interval; LTTB deliberately keeps
  extremes and cannot be read as interval means. Set
  `main.resampling.default-algo` to `lttb` to restore the old behavior.
- **OIDC roles require an explicit mapping (breaking for role-based setups)**:
  The hardcoded translation of token roles (which only recognised the literal
  names `user`/`admin`/`manager`/`support` and silently dropped `api`) is
  replaced by `auth.oidc.role-mapping`. The mapping is the sole source of roles:
  unmapped token roles are ignored and users without a mapped role get the base
  `user` role. Deployments whose IdP emits literal CC role names must now map
  them explicitly (e.g. `"admin": "admin"`). Mapping targets are validated at
  startup.
- **LDAP can assign elevated roles**: The new optional `auth.ldap.role-filters`
  map (role → LDAP filter) grants `admin`/`support`/`api`/`manager` to accounts
  matching a filter. LDAP is authoritative for the roles listed there — sync and
  login both add *and remove* them to match group membership — while roles not
  listed are preserved (a managed `manager` with assigned projects is never
  stripped). Without `role-filters` the behavior is unchanged.
- **Unrecognized top-level config sections are reported**: cc-backend now logs a
  warning for unknown top-level keys in `config.json`, which most often means a
  setting was nested at the wrong level (e.g. `resampling` next to `main`
  instead of inside it) and was silently ignored before.
- **The `auth` section is validated again**: Its JSON schema was missing the
  enclosing `type`/`properties` wrapper, so it accepted every input and no auth
  option was ever checked. With the schema fixed, an incomplete auth section now
  aborts startup — `jwts` requires `max-age`, `oidc` requires `provider`, and
  `ldap` requires `url`, `user-base`, `search-dn`, `user-bind` and `user-filter`.
  Each of the three subsections stays optional on its own.
- **`archive.retention` target keys**: The retention target is configured with
  `target-kind` and `target-path` (or the `target-*` S3 keys). The `location` key
  used by the old example configuration was never read by the code, so a `move`
  or `copy` policy configured that way ran against an empty target path.

### New features

- **Configurable footer links**: `main.footer-links` lets the Imprint and
  Privacy Policy footer entries point at internal pages (default) or external
  URLs. External `http(s)` targets open in a new tab; empty or unset values fall
  back to the built-in `/imprint` and `/privacy` routes, and the existing
  `./var/*.tmpl` override mechanism keeps working.
- **Display smoothing for metric plots**: A centered, NaN-aware moving average
  can be applied in the plot layer, after whatever downsampling the backend
  performed. It preserves series length and index alignment, skips NaN/null
  samples, keeps all-NaN windows as gaps, and shrinks the window at the series
  edges. The window is a per-user setting in data points with a site default in
  `ui.plot-configuration.smoothing-window` (0 disables it). It is display-only:
  reported statistics and job footprints are unaffected.
- **Metric tooltips**: `metricConfig` entries in `cluster.json` accept an
  optional `tooltip` that is rendered in the metric selection dialog.
- **Metric store statistics from cached aggregates**: Buffers maintain running
  sum/min/max/count while data is written and compute them eagerly on checkpoint
  load, so a full-buffer statistics request is served from the cached aggregate
  instead of rescanning the samples.
- **Checkpoint retention aware of running jobs**: The metric store receives a
  `NodeProvider` (the job repository) at `Init`. Checkpoint cleanup now skips
  nodes that still have running jobs, the full checkpoint history is loaded for
  those nodes at startup, and this also applies to the `-cleanup-checkpoints`
  CLI path.

### Bug fixes

- **Metric plots and the policy-based resample config**: `MetricPlot` still read
  the removed `trigger`/`resolutions` values, so the array-based resolution
  branch was unreachable and the zoom guard always fired. The zoom trigger is now
  derived from the policy's target point count.
- **Resample algorithm for running jobs**: The selected algorithm was not
  forwarded to the metric store when loading data for running jobs.
- **Partially covered buffers**: Requests that reach beyond the stored data now
  get NaN padding at the start/end instead of misaligned series
  (`dataNotAligned`).
- **Data race in the statistics read path**: Reading buffer statistics no longer
  mutates the buffer, and the fast path is guarded against empty buffers.
- **Empty node levels leaked**: The retention free path now prunes node levels
  that no longer hold any data, and no longer force-frees buffers that are
  retained for running jobs.
- **Plot render options in the wrong place**: `PlotRenderOptions` is only shown
  in the user settings again.
- **Invisible navbar**: Fixed a stuck animation that hid the navbar with certain
  browser settings.
- **Subcluster dropdown**: The subcluster selection is size-limited and scrolls
  instead of overflowing.
- **OIDC login session handling**: Fixed session handling on the OIDC login
  route.

### Dependencies

- **Added** `github.com/alexedwards/scs/v2` and
  `github.com/alexedwards/scs/sqlite3store`.
- **Removed** `github.com/gorilla/sessions` (and `github.com/gorilla/securecookie`),
  and `github.com/joho/godotenv` together with the `.env` support.
- **Upgraded** `cc-lib` (GraphQL code regenerated) and refreshed the remaining Go
  module dependencies.

## Changes in 1.5.4

Supports job archive version 3 and database version 11.

This was a security and bugfix release of `cc-backend`, the API backend and
frontend implementation of ClusterCockpit.
If you are upgrading from v1.5.1 no database migration is required.
If you are upgrading from v1.5.0 you need to do another DB migration. This
should not take long. For optimal database performance after the migration it is
recommended to apply the new `optimize-db` flag, which runs the sqlite `ANALYZE`
and `VACUUM` commands. Depending on your database size (more then 40GB) the
`VACUUM` may take up to 2h. You can also run the `ANALYZE` command manually.
While we are confident that the memory issue with the metricstore cleanup move
policy is fixed, it is still recommended to use delete policy for cleanup.
This is also the default.

### Security fixes

- **JWT HMAC empty-key bypass (critical)**: `jwtSession.go` now refuses to
  register when `CROSS_LOGIN_JWT_HS512_KEY` is unset. Previously, an empty HMAC
  key allowed unauthenticated admin token forgery because `golang-jwt` verifies
  any HS256/HS512 signature against an empty key.
- **SQL injection via metric names (critical)**: Metric names supplied through
  GraphQL (`[String!]`) were interpolated raw into `json_extract` SQL expressions.
  Names are now validated against `^[a-zA-Z0-9_]+$` in
  `jobsMetricStatisticsHistogram` and `buildFloatJSONCondition`.
- **Path traversal via line-protocol tags (critical)**: `cluster` and `host`
  tags from the metric line protocol flowed unvalidated into `path.Join` for
  checkpoint/WAL file paths, enabling arbitrary file writes outside the
  checkpoint root via NATS (unauthenticated) or `POST /api/write`. Path-traversal
  sequences are now rejected in `DecodeLine` before the tags become path
  components.
- **CORS `AllowCredentials` disabled**: CORS middleware no longer sets
  `AllowCredentials: true`, which was incompatible with `AllowedOrigins: ["*"]`
  and could enable cross-origin credential theft.
- **HSTS header added**: `Strict-Transport-Security` is now set for all
  HTTPS connections.
- **Security response headers**: Added `X-Content-Type-Options: nosniff`,
  `X-Frame-Options: DENY`, `Referrer-Policy: same-origin`, and a conservative
  `Content-Security-Policy` (blocks `frame-ancestors`, `object-src`, `base-uri`)
  to harden against clickjacking and base-tag injection.
- **Stored XSS in job message**: `job.metaData.message` is now rendered as
  escaped text (CSS `white-space: pre-wrap`) instead of raw `{@html ...}` in
  `Job.root` and `JobFootprint`.
- **SQL injection in tag queries**: The tag-scope `IN` list and manager project
  subquery in `CountTags` are now parameterized instead of interpolating
  `user.Username` / `user.Projects` values sourced from OIDC/LDAP.
- **GraphQL DoS hardening**: Query cost is bounded with `FixedComplexityLimit`
  (5000). Non-positive `items-per-page` and `page` values are rejected with HTTP
  400 to prevent integer underflow into unbounded `LIMIT`/`OFFSET` queries.
- **CSRF defense-in-depth**: State-changing requests with a cross-site
  `Sec-Fetch-Site` header are now rejected (fails open for non-browser clients),
  complementing the existing `SameSite=Lax` session cookie.
- **NATS API security warning**: A startup warning is now logged when NATS
  subscriptions are enabled, reminding operators that the NATS API has no
  application-layer authentication and that publish ACLs must be restricted at
  the broker.

### Bug fixes

- **Roofline legend placement**: Roofline plot legends now use fixed
  coordinates instead of dynamic placement, preventing the legend from
  overlapping data points or being rendered off-canvas (#546).
- **Subcluster usage tab labels**: Subcluster names in the status dashboard
  usage tabs are no longer force-capitalized; the original cluster-defined
  casing is preserved.
- **NodeListRow host filter**: The running-jobs query in the node list row now
  filters by exact node hostname (`eq`) instead of substring match
  (`contains`), avoiding incorrect matches when one hostname is a prefix of
  another.
- **WAL files not reset on shutdown**: On graceful shutdown the metricstore wrote
  a final binary snapshot but never rotated the per-host `current.wal` files (the
  `RotateWALFilesAfterShutdown` helper was defined but never called). The stale
  WAL files were replayed and then appended to again on the next start, so they
  grew without bound across restarts and were only ever reset at the next periodic
  checkpoint. `Shutdown` now rotates the WAL files for all successfully
  snapshotted hosts.

### Dependencies

- **Go module upgrades**: Refreshed Go module dependencies to their latest
  compatible versions.

## Changes in 1.5.3

### Bug fixes

- **OIDC role extraction**: Fixed role extraction from OIDC tokens where roles
  were not correctly parsed from the token claims. Roles are now always
  requested from the token regardless of other configuration.
- **OIDC user sync role changes**: `SyncUser` and `UpdateUser` callbacks now
  allow all role changes, removing a restriction that prevented role updates
  during OIDC-driven user synchronization.
- **OIDC projects array**: Projects array from the OIDC token is now submitted
  and applied when syncing user attributes.
- **WAL message drops during checkpoint**: WAL writes are now paused during
  binary checkpoint creation. Previously, disk I/O contention between
  checkpoint writes and WAL staging caused over 1.4 million dropped messages
  per checkpoint cycle.
- **WAL rotation skipped for all nodes**: `RotateWALFiles` used a non-blocking
  send on a small channel. With thousands of nodes, the channel filled instantly
  and nearly all hosts were skipped, leaving WAL files unrotated. Replaced with
  a blocking send using a shared 2-minute deadline.
- **Log viewer auto-refresh**: Fixed the log viewer component not auto-refreshing
  correctly.
- **SameSite cookie setting**: Relaxed the SameSite cookie attribute to improve
  compatibility with OIDC redirect flows.
- **WAL not rotated on partial checkpoint failure**: When binary checkpointing
  failed for some hosts, WAL files for successfully checkpointed hosts were not
  rotated and the checkpoint timestamp was not advanced. Partial successes now
  correctly advance the checkpoint and rotate WAL files for completed hosts.
- **Unbounded WAL file growth**: If binary checkpointing consistently failed for
  a host, its `current.wal` file grew without limit until disk exhaustion. A new
  `max-wal-size` configuration option (in the `checkpoints` block) allows setting
  a per-host WAL size cap in bytes. When exceeded, the WAL is force-rotated.
  Defaults to 0 (unlimited) for backward compatibility.

- **Doubleranged filter fixes**: Range filters now correctly handle zero as a
  boundary value. Improved validation and UI text for "more than equal" and
  "less than equal" range selections.
- **Lineprotocol body parsing interrupted**: Switched from `ReadTimeout` to
  `ReadHeaderTimeout` so that long-running metric submissions are no longer
  cut off mid-stream.
- **Checkpoint archiving continues on error**: A single cluster's archiving
  failure no longer aborts the entire cleanup operation. Errors are collected
  and reported per cluster.
- **Parquet row group overflow**: Added periodic flush during checkpoint
  archiving to prevent exceeding the parquet-go 32k column-write limit.
- **Removed metrics excluded from subcluster config**: Metrics removed from a
  subcluster are no longer returned by `GetMetricConfigSubCluster`.

### MetricStore performance

- **WAL writer throughput**: Decoupled WAL file flushing from message processing
  using a periodic 5-second batch flush (up to 4096 messages per cycle),
  significantly increasing metric ingestion throughput.
- **Improved shutdown time**: HTTP shutdown timeout reduced; metricstore and
  archiver now shut down concurrently. Overall shutdown deadline raised to
  60 seconds.

### New features

- **Manual checkpoint cleanup flag**: New `-cleanup-checkpoints` CLI flag
  triggers checkpoint cleanup without starting the server, useful for
  maintenance windows or automated cleanup scripts.
- **Explicit node state queries in node view**: Node health and scheduler state
  are now fetched independently from metric data for fresher status information.

### Development tooling

- **Make targets for formatting and linting**: New `make fmt` and `make lint`
  targets using `gofumpt` and `golangci-lint`. Configuration added in
  `.golangci.yml` and `gopls.json`.

### New tools

- **binaryCheckpointReader**: New utility tool (`tools/binaryCheckpointReader`)
  that reads `.wal` or `.bin` checkpoint files produced by the metricstore
  WAL/snapshot system and dumps their contents to a human-readable `.txt` file.
  Useful for debugging and inspecting checkpoint data. Usage:
  `go run ./tools/binaryCheckpointReader <file.wal|file.bin>`

### Logging improvements

- **Reduced tagger log noise**: Missing metrics and expression evaluation errors
  in the job classification tagger are now logged at debug level instead of
  error level.

## Changes in 1.5.2

### Bug fixes

- **Memory spike in parquet writer**: Fixed memory spikes when using the
  metricstore move (archive) policy with the parquet writer. The writer now
  processes data in a streaming fashion to avoid accumulating large allocations.
- **Top list query fixes**: Fixed top list queries in analysis and dashboard
  views.
- **Exclude down nodes from HealthCheck**: Down nodes are now excluded from
  health checks in both the REST and NATS handlers.
- **Node state priority order**: Node state determination now enforces a
  priority order. Exception: idle+down results in idle.
- **Blocking ReceiveNats call**: Fixed a blocking NATS receive call in the
  metricstore.

### Database performance

- **Reduced insert pressure**: Bulk insert operations (node state updates, user
  and job cache syncs) now use explicit transactions and deferred inserts,
  significantly reducing write contention on the SQLite database.
- **SyncJobs wrapped in transaction**: `SyncJobs` now runs inside a transaction
  for better consistency and reduced lock contention.
- **Configurable busy timeout**: New `busy-timeout` configuration option for the
  SQLite connection. This allows tuning how long the driver waits for a locked
  database before returning an error, which improves resilience under concurrent
  write load.
- **Increased default SQLite timeout**: The default SQLite connection timeout
  has been raised to reduce spurious timeout errors under load.
- **Optimized stats queries**: Improved sortby handling in stats queries, fixed
  cache key passing, and simplified a stats query condition that caused an
  expensive unnecessary subquery.

### MetricStore performance

- **Sharded WAL consumer**: The WAL consumer is now sharded for significantly
  higher write throughput.
- **NATS contention fix**: Fixed contention in the metricstore NATS ingestion
  path.

### NATS API

- **Nodestate health checks in NATS API**: The NATS node state handler now
  performs the same metric health checks as the REST API handler, including
  per-subcluster health checks and `MonitoringStateFailed` fallback for nodes
  without health data.

### Logging improvements

- **Better error context**: Several repository functions now include the calling
  function name in error messages for easier diagnosis.
- **Reduced log noise**: `ErrNoRows` (no results found) is no longer logged as
  an error in `scanRow`; common "no rows" paths are now silent.
- **Debug-level missing metrics**: Warning about missing metrics in the metric
  store has been downgraded to debug level to reduce log noise in normal
  operation.
- **Checkpoint archiving log**: Added an informational log message when the
  metricstore checkpoint archiving process runs.
- **Auth failure context**: Auth failure log messages now include more context
  information.

### Behavior changes

- **DB-based metricHealth**: Replaced heuristic-based metric health with
  DB-based metric health for the node view, providing more accurate health
  status information.
- **Removed minRunningFor filter remnants**: Cleaned up remaining `minRunningFor`
  references from the GraphQL schema and query builder.

### Frontend

- **Streamlined statsSeries**: Unified stats series calculation and rendering
  across plot components.
- **Clarified plot titles**: Improved titles in dashboard and health views.
- **Bumped frontend dependencies**: Updated frontend dependencies to latest
  versions.

### Dependencies

- **cc-lib upgraded**: Updated to latest cc-lib version.

## Known issues (as of 1.5.2)

These were the open issues at the time of the 1.5.2 release and are kept here for
that release's record; they are not necessarily still open. Note that the
`ui-config` section referenced below is now called `ui`.

- The new dynamic memory management is not bullet proof yet across restarts.
  Buffers that are kept outside the retention period may be lost across a
  restart. We will fix that in a subsequent patch release.
- To use the new log viewer (which is only working when starting cc-backend with
  systemd) in the admin interface the user under which the cc-backend process is
  running has to be allowed to execute the journalctl command.
- The user configuration keys for the ui have changed. Therefore old user
  configuration persisted in the database is not used anymore. It is recommended
  to configure the metrics shown in the ui-config section and remove all records
  in the table after the update.
- Currently energy footprint metrics of type energy are ignored for calculating
  total energy.
- With energy footprint metrics of type power the unit is ignored and it is
  assumed the metric has the unit Watt.

## Changes in 1.5.1

### Database

- **New migration (version 11)**: Optimized database index count and added covering indexes for stats queries for significantly improved query performance
- **Migration 9 fix**: Removed redundant indices from migration 9 that are superseded by migration 11
- **Optional DB optimization flag**: Added `-optimize-db` CLI flag to run `ANALYZE` on demand; removed automatic ANALYZE on startup
- **Selective stats queries**: Stats queries are now selective, reducing unnecessary computation
- **User list paging**: Added paging support to the user list for better scalability
- **SQLite configuration hardening**: Sanitized SQLite configuration with new configurable options; fixes large heap allocations in the SQLite driver
- **Query cancellation**: Long-running database queries can now be cancelled
- **Resource leak fix**: Added missing `defer Close()` calls for all query result sets

### Bug fixes

- **Segfault when taggers misconfigured**: Fixed crash when `enable-job-taggers` is set but tagger rule directories are missing
- **GroupBy stats query complexity**: Reduced complexity for `groupBy` statistics queries
- **Ranged filter conditions**: Fixed GT and LT conditions in ranged filters
- **Energy filter preset**: Reduced energy filter preset to a more practical default
- **JSON validity check**: Fixed wrong field being checked for JSON validity
- **Tagger float rounding**: Fixed rounding of floats in tagger messages
- **Node view null safety**: Added null-safe checks in node view to prevent runtime errors
- **Public dashboard null safety**: Added null-safe checks in the public dashboard to prevent runtime errors

### Frontend

- **Bumped patch versions**: Updated frontend dependencies to latest patch versions

### Documentation

- **New DB config options**: Added new database configuration options to README

---

_The sections below document all features and changes introduced in the 1.5.0 major release, which 1.5.1 is based on._

## Breaking changes

### Configuration changes

- **JSON attribute naming**: All JSON configuration attributes now use `kebab-case`
  style consistently (e.g., `api-allowed-ips` instead of `apiAllowedIPs`).
  Update your `config.json` accordingly.
- **Removed `disable-archive` option**: This obsolete configuration option has been removed.
- **Removed `clusters` config section**: The separate clusters configuration section
  has been removed. Cluster information is now derived from the job archive.
- **`apiAllowedIPs` is now optional**: If not specified, defaults to not
  restricted.

### Architecture changes

- **Web framework replaced**: Migrated from `gorilla/mux` to `chi` as the HTTP
  router. This should be transparent to users but affects how middleware and
  routes are composed. A proper 404 handler is now in place.
- **MetricStore moved**: The `metricstore` package has been moved from `internal/`
  to `pkg/` as it is now part of the public API.
- **MySQL/MariaDB support removed**: Only SQLite is now supported as the database backend.
- **Archive to Cleanup renaming**: Archive-related functions have been refactored
  and renamed to "Cleanup" for clarity.
- **`minRunningFor` filter removed**: This undocumented filter has been removed
  from the API and frontend.

### Dependency changes

- **cc-lib v2.8.0**: Switched to cc-lib version 2 with updated APIs
- **cclib NATS client**: Now using the cclib NATS client implementation
- Removed obsolete `util.Float` usage from cclib

## Major new features

### NATS API Integration

- **Real-time job events**: Subscribe to job start/stop events via NATS
- **Node state updates**: Receive real-time node state changes via NATS
- **Configurable subjects**: NATS API subjects are now configurable via `api-subjects`
- **Deadlock fixes**: Improved NATS client stability and graceful shutdown

### Public Dashboard

- **Public-facing interface**: New public dashboard route for external users
- **DoubleMetricPlot component**: New visualization component for comparing metrics
- **Improved layout**: Reviewed and optimized dashboard layouts for better readability

### Enhanced Node Management

- **Node state tracking**: New node table in database with timestamp tracking
- **Node state filtering**: Filter jobs by node state in systems view
- **Node list enhancements**: Improved paging, filtering, and continuous scroll support
- **Nodestate retention and archiving**: Node state data is now subject to configurable
  retention policies and can be archived to Parquet format for long-term storage
- **Faulty node metric tracking**: Faulty node state metric lists are persisted to the database

### Health Monitoring

- **Health status dashboard**: New dedicated "Health" tab in the status details view
  showing per-node metric health across the cluster
- **CCMS health check**: Support for querying health status of external
  cc-metric-store (CCMS) instances via the API
- **GraphQL health endpoints**: New GraphQL queries and resolvers for health data
- **Cluster/subcluster filter**: Filter health status view by cluster or subcluster

### Log Viewer

- **Web-based log viewer**: New log viewer page in the admin interface for inspecting
  backend log output directly from the browser without shell access
- **Accessible from header**: Quick access link from the navigation header

### MetricStore Improvements

- **Memory tracking worker**: New worker for CCMS memory usage tracking
- **Dynamic retention**: Support for job specific dynamic retention times
- **Improved compression**: Transparent compression for job archive imports
- **Parallel processing**: Parallelized Iter function in all archive backends

### Job Tagging System

- **Job tagger option**: Enable automatic job tagging via configuration flag
- **Application detection**: Automatic detection of applications (MATLAB, GROMACS, etc.)
- **Job classification**: Automatic detection of pathological jobs
- **omit-tagged**: Option to exclude tagged jobs from retention/cleanup operations (`none`, `all`, or `user`)
- **Admin UI trigger**: Taggers can be run on-demand from the admin web interface
  without restarting the backend

### Archive Backends

- **Parquet archive format**: New Parquet file format for job archiving, providing
  columnar storage with efficient compression for analytical workloads
- **S3 backend**: Full support for S3-compatible object storage
- **SQLite backend**: Full support for SQLite backend using blobs
- **Performance improvements**: Fixed performance bugs in archive backends
- **Better error handling**: Improved error messages and fallback handling
- **Zstd compression**: Parquet writers use zstd compression for better
  compression ratios compared to the previous snappy default
- **Optimized sort order**: Job and nodestate Parquet files are sorted by
  cluster, subcluster, and start time for efficient range queries

### Unified Archive Retention and Format Conversion

- **Uniform retention policy**: Job archive retention now supports both JSON and
  Parquet as target formats under a single, consistent policy configuration
- **Archive manager tool**: The `tools/archive-manager` utility now supports
  format conversion between JSON and Parquet job archives
- **Parquet reader**: Full Parquet archive reader implementation for reading back
  archived job data

## New features and improvements

### Frontend

- **Loading indicators**: Added loading indicators to status detail and job lists
- **Job info layout**: Reviewed and improved job info row layout
- **Metric selection**: Enhanced metric selection with drag-and-drop fixes
- **Filter presets**: Move list filter preset to URL for easy sharing
- **Job comparison**: Improved job comparison views and plots
- **Subcluster reactivity**: Job list now reacts to subcluster filter changes
- **Short jobs quick selection**: New "Short jobs" quick-filter button in job lists
  replaces the removed undocumented `minRunningFor` filter
- **Row plot cursor sync**: Cursor position is now synchronized across all metric
  plots in a job list row for easier cross-metric comparison
- **Disabled metrics handling**: Improved handling and display of disabled metrics
  across job view, node view, and list rows
- **"Not configured" info cards**: Informational cards shown when optional features
  are not yet configured
- **Frontend dependencies**: Bumped frontend dependencies to latest versions
- **Svelte 5 compatibility**: Fixed Svelte state warnings and compatibility issues

### Backend

- **Progress bars**: Import function now shows progress during long operations
- **Better logging**: Improved logging with appropriate log levels throughout
- **Graceful shutdown**: Fixed shutdown timeout bugs and hanging issues
- **Configuration defaults**: Sensible defaults for most configuration options
- **Documentation**: Extensive documentation improvements across packages
- **Server flag in systemd unit**: Example systemd unit now includes the `-server` flag

### Security

- **LDAP security hardening**: Improved input validation, connection handling, and
  error reporting in the LDAP authenticator
- **OIDC security hardening**: Stricter token validation and improved error handling
  in the OIDC authenticator
- **Auth schema extensions**: Additional schema fields for improved auth configuration

### API improvements

- **Role-based metric visibility**: Metrics can now have role-based access control
- **Job exclusivity filter**: New filter for exclusive vs. shared jobs
- **Improved error messages**: Better error messages and documentation in REST API
- **GraphQL enhancements**: Improved GraphQL queries and resolvers
- **Stop job lookup order**: Reversed lookup order in stop job requests for
  more reliable job matching (cluster+jobId first, then jobId alone)

### Performance

- **Database indices**: Optimized SQLite indices for better query performance
- **Job cache**: Introduced caching table for faster job inserts
- **Parallel imports**: Archive imports now run in parallel where possible
- **External tool integration**: Optimized use of external tools (fd) for better performance
- **Node repository queries**: Reviewed and optimized node repository SQL queries
- **Buffer pool**: Resized and pooled internal buffers for better memory reuse

### Developer experience

- **AI agent guidelines**: Added documentation for AI coding agents (AGENTS.md, CLAUDE.md)
- **Example API payloads**: Added example JSON API payloads for testing
- **Unit tests**: Added more unit tests for NATS API, node repository, and other components
- **Test improvements**: Better test coverage; test DB is now copied before unit tests
  to avoid state pollution between test runs
- **Parquet writer tests**: Comprehensive tests for Parquet archive writing and conversion

## Bug fixes

- Fixed nodelist paging issues
- Fixed metric select drag and drop functionality
- Fixed render race conditions in nodeList
- Fixed tag count grouping including type
- Fixed wrong metricstore schema (missing comma)
- Fixed configuration issues causing shutdown hangs
- Fixed deadlock when NATS is not configured
- Fixed archive backend performance bugs
- Fixed continuous scroll buildup on refresh
- Improved footprint calculation logic
- Fixed polar plot data query decoupling
- Fixed missing resolution parameter handling
- Fixed node table initialization fallback
- Fixed reactivity key placement in nodeList
- Fixed nodeList resolver data handling and increased nodestate filter cutoff
- Fixed job always being transferred to main job table before archiving
- Fixed AppTagger error handling and logging
- Fixed log endpoint formatting and correctness
- Fixed automatic refresh in metric status tab
- Fixed NULL value handling in `health_state` and `health_metrics` columns
- Fixed bugs related to `job_cache` IDs being used in the main job table
- Fixed SyncJobs bug causing start job hooks to be called with wrong (cache) IDs
- Fixed 404 handler route for sub-routers

## Configuration changes

### New configuration options

```json
{
  "main": {
    "enable-job-taggers": true,
    "resampling": {
      "minimum-points": 600,
      "trigger": 180,
      "resolutions": [240, 60]
    },
    "api-subjects": {
      "subject-job-event": "cc.job.event",
      "subject-node-state": "cc.node.state"
    }
  },
  "nats": {
    "address": "nats://0.0.0.0:4222",
    "username": "root",
    "password": "root"
  },
  "cron": {
    "commit-job-worker": "1m",
    "duration-worker": "5m",
    "footprint-worker": "10m"
  },
  "metric-store": {
    "cleanup": {
      "mode": "archive",
      "interval": "48h",
      "directory": "./var/archive"
    }
  },
  "archive": {
    "retention": {
      "policy": "delete",
      "age": "6months",
      "target-format": "parquet"
    }
  },
  "nodestate": {
    "retention": {
      "policy": "archive",
      "age": "30d",
      "archive-path": "./var/nodestate-archive"
    }
  }
}
```

## Migration notes

- Review and update your `config.json` to use kebab-case attribute names
- If using NATS, configure the new `nats` and `api-subjects` sections
- If using S3 archive backend, configure the new `archive` section options
- Test the new public dashboard at `/public` route
- Review cron worker configuration if you need different frequencies
- If using the archive retention feature, configure the `target-format` option
  to choose between `json` (default) and `parquet` output formats
- Consider enabling nodestate retention if you track node states over time
