The file /Users/jan/prg/ClusterCockpit/cc-backend/pkg/metricstore/metricstore.go has been updated successfully. (file state is current in your context — no need to Read it back)

	if Keys.Checkpoints.FileFormat == "wal" {
		// WAL files are deleted per-host inside ToCheckpointWAL workers.
		files, _, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
	} else {

	if Keys.Checkpoints.FileFormat == "wal" {
		var successDirs []string
		files, successDirs, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
		// The final binary snapshot now captures all in-memory data for these
		// hosts, making their current.wal redundant. The staging goroutines have
		// already exited, so remove the WAL files directly (the channel-based
		// RotateWALFiles is no longer safe to call). Without this, current.wal
		// files survive shutdown and keep growing across restarts.
		RotateWALFilesAfterShutdown(successDirs)
	} else {

-    // WAL files are deleted per-host inside ToCheckpointWAL workers.

-    files, _, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())

+    files, successDirs, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())

+    // The final binary snapshot now captures all in-memory data for these

+    // hosts, making their current.wal redundant. The staging goroutines have

+    // already exited, so remove the WAL files directly (the channel-based

+    // RotateWALFiles is no longer safe to call). Without this, current.wal

+    // files survive shutdown and keep growing across restarts.

+    RotateWALFilesAfterShutdown(successDirs)

files, err = ms.ToCheckpoint(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())


Bash

go build ./pkg/metricstore/... && echo "BUILD OK" && go vet ./pkg/metricstore/ 2>&1 | head


BUILD OK


Inefficient string concatenation in call to WriteString


Build passes. (The `walCheckpoint.go:659` lint warning is pre-existing in `joinSelector`, unrelated to this change.)

## Done

The fix is in `pkg/metricstore/metricstore.go`, `Shutdown()`:

```go
if Keys.Checkpoints.FileFormat == "wal" {
    var successDirs []string
    files, successDirs, err = ms.ToCheckpointWAL(Keys.Checkpoints.RootDir, from.Unix(), time.Now().Unix())
    // ... remove now-redundant current.wal files directly ...
    RotateWALFilesAfterShutdown(successDirs)
} else {
    ...
}
```

**What was wrong:** On graceful shutdown the final `.bin` snapshot was written, but the `successDirs` return was discarded (`_`) and the WAL was never rotated. The old comment claimed `ToCheckpointWAL` deletes WAL files — it doesn't. The purpose-built `RotateWALFilesAfterShutdown` helper existed but was never called. So `current.wal` survived every clean shutdown, got replayed *and re-appended to* on the next start, and only ever reset at the 12h periodic checkpoint — hence unbounded growth across restarts.

**The change:** capture `successDirs` and rotate exactly those hosts' WAL files (mirroring the periodic-checkpoint path; staging goroutines are already stopped, so the direct-removal variant is correct here). Hosts with no fresh snapshot are deliberately left untouched so no uncaptured records are dropped.

You can verify end-to-end per the plan: start → ingest → stop (confirm `<ts>.bin` appears and `current.wal` is gone for active hosts) → restart/ingest/stop again → confirm WAL size resets each clean shutdown instead of accumulating.


Goal was fixing unbounded WAL file growth in the metric store. I found and fixed the bug: shutdown never rotated WAL files. The change is applied and builds. Next, verify it end-to-end by starting the server, ingesting metrics, and confirming current.wal resets on shutdown. (disable recaps in /config)


Update the releasenotes with this bugfix. Leave the version as is.


1	# `cc-backend` version 1.5.4
2
3	Supports job archive version 3 and database version 11.
4
5	This is a security and bugfix release of `cc-backend`, the API backend and
6	frontend implementation of ClusterCockpit.
7	For release specific notes visit the [ClusterCockpit Documentation](https://clusterockpit.org/docs/release/).
8	If you are upgrading from v1.5.1 no database migration is required.
9	If you are upgrading from v1.5.0 you need to do another DB migration. This
10	should not take long. For optimal database performance after the migration it is
11	recommended to apply the new `optimize-db` flag, which runs the sqlite `ANALYZE`
12	and `VACUUM` commands. Depending on your database size (more then 40GB) the
13	`VACUUM` may take up to 2h. You can also run the `ANALYZE` command manually.
14	While we are confident that the memory issue with the metricstore cleanup move
15	policy is fixed, it is still recommended to use delete policy for cleanup.
16	This is also the default.
17
18	## Changes in 1.5.4
19
20	### Security fixes
21
22	- **JWT HMAC empty-key bypass (critical)**: `jwtSession.go` now refuses to
23	  register when `CROSS_LOGIN_JWT_HS512_KEY` is unset. Previously, an empty HMAC
24	  key allowed unauthenticated admin token forgery because `golang-jwt` verifies
25	  any HS256/HS512 signature against an empty key.
26	- **SQL injection via metric names (critical)**: Metric names supplied through
27	  GraphQL (`[String!]`) were interpolated raw into `json_extract` SQL expressions.
28	  Names are now validated against `^[a-zA-Z0-9_]+$` in
29	  `jobsMetricStatisticsHistogram` and `buildFloatJSONCondition`.
30	- **Path traversal via line-protocol tags (critical)**: `cluster` and `host`
31	  tags from the metric line protocol flowed unvalidated into `path.Join` for
32	  checkpoint/WAL file paths, enabling arbitrary file writes outside the
33	  checkpoint root via NATS (unauthenticated) or `POST /api/write`. Path-traversal
34	  sequences are now rejected in `DecodeLine` before the tags become path
35	  components.
36	- **CORS `AllowCredentials` disabled**: CORS middleware no longer sets
37	  `AllowCredentials: true`, which was incompatible with `AllowedOrigins: ["*"]`
38	  and could enable cross-origin credential theft.
39	- **HSTS header added**: `Strict-Transport-Security` is now set for all
40	  HTTPS connections.
41	- **Security response headers**: Added `X-Content-Type-Options: nosniff`,
42	  `X-Frame-Options: DENY`, `Referrer-Policy: same-origin`, and a conservative
43	  `Content-Security-Policy` (blocks `frame-ancestors`, `object-src`, `base-uri`)
44	  to harden against clickjacking and base-tag injection.
45	- **Stored XSS in job message**: `job.metaData.message` is now rendered as
46	  escaped text (CSS `white-space: pre-wrap`) instead of raw `{@html ...}` in
47	  `Job.root` and `JobFootprint`.
48	- **SQL injection in tag queries**: The tag-scope `IN` list and manager project
49	  subquery in `CountTags` are now parameterized instead of interpolating
50	  `user.Username` / `user.Projects` values sourced from OIDC/LDAP.
51	- **GraphQL DoS hardening**: Query cost is bounded with `FixedComplexityLimit`
52	  (5000). Non-positive `items-per-page` and `page` values are rejected with HTTP
53	  400 to prevent integer underflow into unbounded `LIMIT`/`OFFSET` queries.
54	- **CSRF defense-in-depth**: State-changing requests with a cross-site
55	  `Sec-Fetch-Site` header are now rejected (fails open for non-browser clients),
56	  complementing the existing `SameSite=Lax` session cookie.
57	- **NATS API security warning**: A startup warning is now logged when NATS
58	  subscriptions are enabled, reminding operators that the NATS API has no
59	  application-layer authentication and that publish ACLs must be restricted at
60	  the broker.

# `cc-backend` version 1.5.4

Supports job archive version 3 and database version 11.

This is a security and bugfix release of `cc-backend`, the API backend and
frontend implementation of ClusterCockpit.
For release specific notes visit the [ClusterCockpit Documentation](https://clusterockpit.org/docs/release/).
If you are upgrading from v1.5.1 no database migration is required.
If you are upgrading from v1.5.0 you need to do another DB migration. This
should not take long. For optimal database performance after the migration it is
recommended to apply the new `optimize-db` flag, which runs the sqlite `ANALYZE`
and `VACUUM` commands. Depending on your database size (more then 40GB) the
`VACUUM` may take up to 2h. You can also run the `ANALYZE` command manually.
While we are confident that the memory issue with the metricstore cleanup move
policy is fixed, it is still recommended to use delete policy for cleanup.
This is also the default.

## Changes in 1.5.4

### Security fixes

- **JWT HMAC empty-key bypass (critical)**: `jwtSession.go` now refuses to
  register when `CROSS_LOGIN_JWT_HS512_KEY` is unset. Previously, an empty HMAC
  key allowed unauthenticated admin token forgery because `golang-jwt` verifies
  any HS256/HS512 signature against an empty key.
- **SQL injection via metric names (critical)**: Metric names supplied through
  GraphQL (`[String!]`) were interpolated raw into `json_extract` SQL expressions.
  Names are now validated against `^[a-zA-Z0-9_]+$` in
  `jobsMetricStatisticsHistogram` and `buildFloatJSONCondition`.
- **Path traversal via line-protocol tags (critical)**: `cluster` and `host`
  tags from the metric line protocol flowed unvalidated into `path.Join` for
  checkpoint/WAL file paths, enabling arbitrary file writes outside the
  checkpoint root via NATS (unauthenticated) or `POST /api/write`. Path-traversal
  sequences are now rejected in `DecodeLine` before the tags become path
  components.
- **CORS `AllowCredentials` disabled**: CORS middleware no longer sets
  `AllowCredentials: true`, which was incompatible with `AllowedOrigins: ["*"]`
  and could enable cross-origin credential theft.
- **HSTS header added**: `Strict-Transport-Security` is now set for all
  HTTPS connections.
- **Security response headers**: Added `X-Content-Type-Options: nosniff`,
  `X-Frame-Options: DENY`, `Referrer-Policy: same-origin`, and a conservative
  `Content-Security-Policy` (blocks `frame-ancestors`, `object-src`, `base-uri`)
  to harden against clickjacking and base-tag injection.
- **Stored XSS in job message**: `job.metaData.message` is now rendered as
  escaped text (CSS `white-space: pre-wrap`) instead of raw `{@html ...}` in
  `Job.root` and `JobFootprint`.
- **SQL injection in tag queries**: The tag-scope `IN` list and manager project
  subquery in `CountTags` are now parameterized instead of interpolating
  `user.Username` / `user.Projects` values sourced from OIDC/LDAP.
- **GraphQL DoS hardening**: Query cost is bounded with `FixedComplexityLimit`
  (5000). Non-positive `items-per-page` and `page` values are rejected with HTTP
  400 to prevent integer underflow into unbounded `LIMIT`/`OFFSET` queries.
- **CSRF defense-in-depth**: State-changing requests with a cross-site
  `Sec-Fetch-Site` header are now rejected (fails open for non-browser clients),
  complementing the existing `SameSite=Lax` session cookie.
- **NATS API security warning**: A startup warning is now logged when NATS
  subscriptions are enabled, reminding operators that the NATS API has no
  application-layer authentication and that publish ACLs must be restricted at
  the broker.


60	  the broker.
61
62	### Bug fixes
63
64	- **Roofline legend placement**: Roofline plot legends now use fixed
65	  coordinates instead of dynamic placement, preventing the legend from
66	  overlapping data points or being rendered off-canvas (#546).
67	- **Subcluster usage tab labels**: Subcluster names in the status dashboard
68	  usage tabs are no longer force-capitalized; the original cluster-defined
69	  casing is preserved.
70	- **NodeListRow host filter**: The running-jobs query in the node list row now
71	  filters by exact node hostname (`eq`) instead of substring match
72	  (`contains`), avoiding incorrect matches when one hostname is a prefix of
73	  another.
74
75	### Dependencies
76
77	- **Go module upgrades**: Refreshed Go module dependencies to their latest
78	  compatible versions.
79
80	## Changes in 1.5.3
81
82	### Bug fixes
83
84	- **OIDC role extraction**: Fixed role extraction from OIDC tokens where roles
85	  were not correctly parsed from the token claims. Roles are now always
86	  requested from the token regardless of other configuration.
87	- **OIDC user sync role changes**: `SyncUser` and `UpdateUser` callbacks now
88	  allow all role changes, removing a restriction that prevented role updates
89	  during OIDC-driven user synchronization.
90	- **OIDC projects array**: Projects array from the OIDC token is now submitted
91	  and applied when syncing user attributes.
92	- **WAL message drops during checkpoint**: WAL writes are now paused during
93	  binary checkpoint creation. Previously, disk I/O contention between
94	  checkpoint writes and WAL staging caused over 1.4 million dropped messages
95	  per checkpoint cycle.
96	- **WAL rotation skipped for all nodes**: `RotateWALFiles` used a non-blocking
97	  send on a small channel. With thousands of nodes, the channel filled instantly
98	  and nearly all hosts were skipped, leaving WAL files unrotated. Replaced with
99	  a blocking send using a shared 2-minute deadline.
100	- **Log viewer auto-refresh**: Fixed the log viewer component not auto-refreshing
101	  correctly.
102	- **SameSite cookie setting**: Relaxed the SameSite cookie attribute to improve
103	  compatibility with OIDC redirect flows.
104	- **WAL not rotated on partial checkpoint failure**: When binary checkpointing
105	  failed for some hosts, WAL files for successfully checkpointed hosts were not
106	  rotated and the checkpoint timestamp was not advanced. Partial successes now
107	  correctly advance the checkpoint and rotate WAL files for completed hosts.
108	- **Unbounded WAL file growth**: If binary checkpointing consistently failed for
109	  a host, its `current.wal` file grew without limit until disk exhaustion. A new
110	  `max-wal-size` configuration option (in the `checkpoints` block) allows setting
111	  a per-host WAL size cap in bytes. When exceeded, the WAL is force-rotated.
112	  Defaults to 0 (unlimited) for backward compatibility.
113
114	- **Doubleranged filter fixes**: Range filters now correctly handle zero as a
115	  boundary value. Improved validation and UI text for "more than equal" and
116	  "less than equal" range selections.
117	- **Lineprotocol body parsing interrupted**: Switched from `ReadTimeout` to
118	  `ReadHeaderTimeout` so that long-running metric submissions are no longer
119	  cut off mid-stream.

  the broker.

### Bug fixes

- **Roofline legend placement**: Roofline plot legends now use fixed
  coordinates instead of dynamic placement, preventing the legend from
  overlapping data points or being rendered off-canvas (#546).
- **Subcluster usage tab labels**: Subcluster names in the status dashboard
  usage tabs are no longer force-capitalized; the original cluster-defined
  casing is preserved.
- **NodeListRow host filter**: The running-jobs query in the node list row now
  filters by exact node hostname (`eq`) instead of substring match
  (`contains`), avoiding incorrect matches when one hostname is a prefix of
  another.

### Dependencies

- **Go module upgrades**: Refreshed Go module dependencies to their latest
  compatible versions.

## Changes in 1.5.3

### Bug fixes

- **OIDC role extraction**: Fixed role extraction from OIDC tokens where roles
  were not correctly parsed from the token claims. Roles are now always
  requested from the token regardless of other configuration.
- **OIDC user sync role changes**: `SyncUser` and `UpdateUser` callbacks now
  allow all role changes, removing a restriction that prevented role updates
  during OIDC-driven user synchronization.
- **OIDC projects array**: Projects array from the OIDC token is now submitted
  and applied when syncing user attributes.
- **WAL message drops during checkpoint**: WAL writes are now paused during
  binary checkpoint creation. Previously, disk I/O contention between
  checkpoint writes and WAL staging caused over 1.4 million dropped messages
  per checkpoint cycle.
- **WAL rotation skipped for all nodes**: `RotateWALFiles` used a non-blocking
  send on a small channel. With thousands of nodes, the channel filled instantly
  and nearly all hosts were skipped, leaving WAL files unrotated. Replaced with
  a blocking send using a shared 2-minute deadline.
- **Log viewer auto-refresh**: Fixed the log viewer component not auto-refreshing
  correctly.
- **SameSite cookie setting**: Relaxed the SameSite cookie attribute to improve
  compatibility with OIDC redirect flows.
- **WAL not rotated on partial checkpoint failure**: When binary checkpointing
  failed for some hosts, WAL files for successfully checkpointed hosts were not
  rotated and the checkpoint timestamp was not advanced. Partial successes now
  correctly advance the checkpoint and rotate WAL files for completed hosts.
- **Unbounded WAL file growth**: If binary checkpointing consistently failed for
  a host, its `current.wal` file grew without limit until disk exhaustion. A new
  `max-wal-size` configuration option (in the `checkpoints` block) allows setting
  a per-host WAL size cap in bytes. When exceeded, the WAL is force-rotated.
  Defaults to 0 (unlimited) for backward compatibility.

- **Doubleranged filter fixes**: Range filters now correctly handle zero as a
  boundary value. Improved validation and UI text for "more than equal" and
  "less than equal" range selections.
- **Lineprotocol body parsing interrupted**: Switched from `ReadTimeout` to
  `ReadHeaderTimeout` so that long-running metric submissions are no longer
  cut off mid-stream.


I'll add the bugfix under the `1.5.4` "Bug fixes" section, matching the existing WAL-fix style.


Edit

- **NodeListRow host filter**: The running-jobs query in the node list row now
  filters by exact node hostname (`eq`) instead of substring match
  (`contains`), avoiding incorrect matches when one hostname is a prefix of
  another.

### Dependencies

- **NodeListRow host filter**: The running-jobs query in the node list row now
  filters by exact node hostname (`eq`) instead of substring match
  (`contains`), avoiding incorrect matches when one hostname is a prefix of
  another.
- **WAL files not reset on shutdown**: On graceful shutdown the metricstore wrote
  a final binary snapshot but never rotated the per-host `current.wal` files (the
  `RotateWALFilesAfterShutdown` helper was defined but never called). The stale
  WAL files were replayed and then appended to again on the next start, so they
  grew without bound across restarts and were only ever reset at the next periodic
  checkpoint. `Shutdown` now rotates the WAL files for all successfully
  snapshotted hosts.

### Dependencies


The file /Users/jan/prg/ClusterCockpit/cc-backend/ReleaseNotes.md has been updated successfully. (file state is current in your context — no need to Read it back)

- **NodeListRow host filter**: The running-jobs query in the node list row now
  filters by exact node hostname (`eq`) instead of substring match
  (`contains`), avoiding incorrect matches when one hostname is a prefix of
  another.

### Dependencies

- **NodeListRow host filter**: The running-jobs query in the node list row now
  filters by exact node hostname (`eq`) instead of substring match
  (`contains`), avoiding incorrect matches when one hostname is a prefix of
  another.
- **WAL files not reset on shutdown**: On graceful shutdown the metricstore wrote
  a final binary snapshot but never rotated the per-host `current.wal` files (the
  `RotateWALFilesAfterShutdown` helper was defined but never called). The stale
  WAL files were replayed and then appended to again on the next start, so they
  grew without bound across restarts and were only ever reset at the next periodic
  checkpoint. `Shutdown` now rotates the WAL files for all successfully
  snapshotted hosts.

### Dependencies

+- **WAL files not reset on shutdown**: On graceful shutdown the metricstore wrote

+  a final binary snapshot but never rotated the per-host `current.wal` files (the

+  `RotateWALFilesAfterShutdown` helper was defined but never called). The stale

+  WAL files were replayed and then appended to again on the next start, so they

+  grew without bound across restarts and were only ever reset at the next periodic

+  checkpoint. `Shutdown` now rotates the WAL files for all successfully


Done. I added the bugfix to the `1.5.4` "Bug fixes" section (version left unchanged), matching the existing WAL-fix entry style:

> **WAL files not reset on shutdown**: On graceful shutdown the metricstore wrote a final binary snapshot but never rotated the per-host `current.wal` files (the `RotateWALFilesAfterShutdown` helper was defined but never called). The stale WAL files were replayed and then appended to again on the next start, so they grew without bound across restarts and were only ever reset at the next periodic checkpoint. `Shutdown` now rotates the WAL files for all successfully snapshotted hosts.


Goal was fixing unbounded WAL file growth in the metricstore. The shutdown fix is applied, builds clean, and the release notes are updated. Next: verify end-to-end by restarting cc-backend and confirming current.wal resets on clean shutdown. (disable recaps in /config)