135 Commits
Author SHA1 Message Date
moebiusbandandClaude Opus 5 6e49b0a332 fix(archive): make cluster.json test fixtures schema-valid
The alex and fritz fixtures used "Flops/s" and "packets/s" as unit bases, which
are not in the enum of unit.schema.json ("F/s" is). They are copied into real
deployments, where an invalid cluster.json only fails at runtime once
main.validate is enabled.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-27 16:02:41 +02:00
moebiusbandandClaude Opus 5 668a5bf877 fix(config): update example configurations to match implementation
The shipped examples had drifted from the code that parses them:

- configs/cluster.json was still in the pre-v3 format: no top-level "name",
  "numberOfNodes" instead of the required "nodes" node-list expression, plain
  numbers instead of MetricValue objects for flopRateScalar/flopRateSimd/
  memoryBandwidth, unit strings instead of unit objects, "aggregation": null,
  and hwthread 72 missing from the core topology. It neither validated against
  cluster.schema.json nor decoded into schema.Cluster. Rewritten in the current
  format and extended with the accelerator metrics the GPU subclusters need.
- archive.retention.location is not a field of taskmanager.Retention. That
  decode is not strict, so the key was silently dropped and a "move" policy ran
  with an empty target path. Replaced with target-kind/target-path.
- "ui-file" pointed at ui-config.json while the shipped file is uiConfig.json,
  so ccConfig logged a load error and the UI fell back to defaults.
- Added the options introduced since the examples were written: checkpoint
  file-format/interval/max-wal-size, api-subjects concurrency, db-config,
  nodestate-retention, archive retention/compression, smoothing-window and the
  resample policy/algo defaults.

The config written by -init had no metric-store section, so a server started
right after -init aborted with "missing metricstore configuration".

internal/configexample guards all of this: it resolves "-file" references the
way ccConfig does, strict-decodes every section into the struct that actually
parses it, and validates the cluster configs against the cc-lib schema.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-27 16:02:32 +02:00
moebiusbandandClaude Opus 5 4cfbecd818 Merge branch 'feat/565-add-metric-tooltip' into metric-store-tickets
Resolve conflicts in the generated GraphQL code by regenerating it against
the merged schema. cc-lib v2.13.0 adds Tooltip to schema.MetricConfig and
schema.GlobalMetricListItem, so gqlgen now binds the tooltip field directly
and the hand-written globalMetricListItem/metricConfig resolvers introduced
on the tooltip branch are no longer needed.

Also migrate to the cc-lib v2.13.0 metric container types, which changed
from bare maps to structs carrying array-valued metric groups:

  schema.JobData          map -> {Metrics, Groups}
  schema.ScopedJobStats   map -> {Metrics, Groups}
  job.Statistics          map -> schema.JobStatisticsSet{Metrics, Groups}

Callers index .Metrics, return the zero struct instead of nil, and
deepCopy/DecodeJobStats now also carry the Groups payload through.
archive.GetStatistics returns the full JobStatisticsSet so group
statistics survive the round trip.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-27 10:22:22 +02:00
Christoph Kluge 053c44d7b5 add tooltips to metricConfig, rendered in metricSelect, rework availability display 2026-07-28 17:36:29 +02:00
moebiusband 641dc0e3b8 Run gofumpt 2026-03-30 16:49:27 +02:00
moebiusband 01ec70baa8 Iterate over subCluster MetricConfig directly so that removed metrics are not included
Entire-Checkpoint: efb6f0a96069
2026-03-20 11:39:34 +01:00
moebiusband 51517f8031 Reduce insert pressure in db. Increase sqlite timeout value
Entire-Checkpoint: a1e2931d4deb
2026-03-16 11:17:47 +01:00
Christoph Kluge 9672903d41 fix panic caused by concurrent map writes 2026-03-04 15:54:18 +01:00
moebiusband a418abc7d5 Run go fix 2026-02-27 14:40:26 +01:00
moebiusband 03c65e06f6 Allow finer control for omit tagged jobs in retention policies 2026-02-23 08:46:47 +01:00
moebiusband 2da35909c1 Optimize sort order in job parquet files 2026-02-18 08:13:00 +01:00
moebiusband 2e24fde430 Optimize sort order in nodestate parquet files 2026-02-18 08:06:00 +01:00
moebiusband 757be60b22 Switch to zstd compression for parquet writers 2026-02-18 07:55:28 +01:00
moebiusband 6035b62734 Run go fix 2026-02-17 21:04:17 +01:00
moebiusband 7d8b305cd9 Add format conversion to archive manager 2026-02-13 13:54:10 +01:00
moebiusband 2c8608f5a4 Update job archive retention to uniform policy with json and parquet target format 2026-02-13 12:19:31 +01:00
moebiusband 54ea5d7900 Add nodestate retention and archiving 2026-02-12 09:21:44 +01:00
moebiusband 8d6c6b819b Update and port to cc-lib 2026-02-11 07:06:06 +01:00
moebiusband 035ac2384e Refactor GlobalMetricLists 2026-02-09 21:56:41 +01:00
moebiusband c920c57f5d Add parquet file job archiving target 2026-02-07 10:51:56 +01:00
moebiusband 1d4c79c821 Unify JSON attribute naming ot use kebab style case. Cleanup configuration. 2026-01-20 09:47:13 +01:00
moebiusband 5281f3bb60 Remove obsolete config option disable-archive 2026-01-19 16:42:30 +01:00
moebiusband e1efc68476 Update dependencies. Rebuild graphql and swagger 2026-01-15 08:32:06 +01:00
moebiusband c6465ad9e5 Add s3 configuration options 2026-01-14 15:28:34 +01:00
moebiusband 211d4fae54 Refactor s3 backend and suppress checksum warning 2026-01-14 15:10:20 +01:00
moebiusband 754f7e16f6 Reformat with gofumpt 2026-01-13 09:52:31 +01:00
moebiusband 8576ae458d Switch to cc-lib v2 2025-12-24 09:24:18 +01:00
moebiusband 6e74fa294a Add role-based visibility for metrics
Fixes #387
2025-12-18 15:47:30 +01:00
moebiusband b8fdfc30c0 Fix performance bugs in sqlite archive backend 2025-12-17 10:12:49 +01:00
moebiusband 7fce6fa401 Parallelize the Iter function in all archive backends 2025-12-16 09:11:09 +01:00
moebiusband 0306723307 Introduce transparent compression for importJob function in all archive backends 2025-12-16 08:55:31 +01:00
moebiusband 10aa2bfbd3 Add support for ClusterConfig 2025-12-15 11:24:12 +01:00
moebiusband 6cfed989ff Fix bugs in sqlite backend 2025-12-15 11:23:53 +01:00
moebiusband 48b68d3410 Fix aws endpoint deprecation 2025-12-04 06:20:19 +01:00
moebiusband 78530029ef Reformat 2025-12-03 14:54:48 +01:00
moebiusband f141ca926f Increase archive version. Fix unit tests. 2025-11-20 14:28:06 +01:00
moebiusband 6239e7f19b Merge branch 'dev' into ai-review 2025-11-20 08:59:52 +01:00
moebiusband e1c7583670 Add sqlite and s3 job archive backend
Add documentation
Extend config
2025-11-19 17:00:11 +01:00
moebiusband ae5d202661 Remove S3Backend stub 2025-10-23 15:14:28 +02:00
moebiusband bc43c844fc Fix memoryStore Init and move MetricConfig init 2025-10-20 10:22:40 +02:00
moebiusband 4fc78bc382 Refactor variable namings and doc comments 2025-09-27 09:27:36 +02:00
moebiusband eaca187032 Fix testdata for new schema 2025-09-09 15:04:25 +02:00
Aditya Ujeniya 62565b9ae2 Combined metricstore api and functions 2025-09-08 11:29:27 +02:00
moebiusband 86453e7e11 Port to new job structs
Backup commit: Does not build.
2025-08-05 10:23:54 +02:00
moebiusband 0754ba5292 Port configuration to ccConfig scheme
Decentralize config validation
Modularize configuration handling
2025-07-07 13:09:12 +02:00
moebiusband 4bd73450b5 Temporary disable archive clean test 2025-06-30 13:00:07 +02:00
moebiusband 639e1b9c6d Port to cc-lib. Extend legal header. 2025-06-30 12:06:35 +02:00
moebiusband dca25cc601 Saveguard changes to archive 2025-06-27 12:15:42 +02:00
moebiusband 3efee22536 Remove jobMeta and use job struct everywhere 2025-05-28 15:59:21 +02:00
moebiusband 0aecea6de2 Refactor. Add Subcluster get metric list helper routine. 2025-05-27 09:23:28 +02:00