ClusterCockpit/cc-backend
1
0
Fork 0
mirror of https://github.com/ClusterCockpit/cc-backend synced 2026-07-27 16:57:15 +02:00
Code Issues Packages Releases Activity
2,693 Commits 9 Branches 18 Tags
6855d62bf25b61c62fb7ade51e827913f9e95531
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
moebiusband 6855d62bf2 Make log in scanRow more descriptive. No log for common no rows error 
Entire-Checkpoint: 858b34ef56b8
 2026-03-16 20:03:27 +01:00
moebiusband 7f3eb443d9 Include calling function in error message 
Entire-Checkpoint: a4948d0fe7a3
 2026-03-16 15:42:38 +01:00
moebiusband bab6eb4c3a Convert Warn message on missing metrics to Debug level 2026-03-16 15:35:24 +01:00
moebiusband 09d0ba71d2 Provide idential nodestate functionality in NATS API 
Entire-Checkpoint: 3a40b75edd68
 2026-03-16 12:13:14 +01:00
moebiusband df93dbed63 Add busyTimeout config setting 
Entire-Checkpoint: 81097a6c52a2
 2026-03-16 11:30:21 +01:00
moebiusband e4f3fa9ba0 Wrap SyncJobs in transaction 
Entire-Checkpoint: d4f6c79a8dc1
 2026-03-16 11:25:49 +01:00
moebiusband 51517f8031 Reduce insert pressure in db. Increase sqlite timeout value 
Entire-Checkpoint: a1e2931d4deb
 2026-03-16 11:17:47 +01:00
moebiusband 0aad8f01c8 Upgrade cc-lib 
Fixes panic in AddNodeScope

Entire-Checkpoint: afef27e07ec9
 2026-03-16 08:55:56 +01:00
moebiusband 973ca87bd1 Extend known issues in ReleaseNotes 2026-03-15 07:02:54 +01:00
moebiusband 045311eec0 Prepare release 1.5.1 
Entire-Checkpoint: baed7fbee099
 2026-03-13 17:30:03 +01:00
moebiusband e38396a081 Upgrade dependencies. Rebuild GraphQL. 
Entire-Checkpoint: f770853c9fa0
 2026-03-13 17:22:34 +01:00
moebiusband e83bd2babd Consolidate migrations 
Entire-Checkpoint: a3dba4105838
 2026-03-13 17:14:13 +01:00
moebiusband f15f1452cc Inline jobstate literal in query 
Entire-Checkpoint: 35f06df74b51
 2026-03-13 15:16:07 +01:00
moebiusband df2a13def2 Merge branch 'hotfix' of github.com:ClusterCockpit/cc-backend into hotfix 2026-03-13 14:34:11 +01:00
moebiusbandandClaude Opus 4.6 d586fe4b43 Optimize usage dashboard: partial indexes, request cache, parallel histograms 
- Add migration 14: partial covering indexes WHERE job_state='running'
  for user/project/subcluster groupings (tiny B-tree vs full table)
- Inline literal state value in BuildWhereClause so SQLite matches
  partial indexes instead of parameterized placeholders
- Add per-request statsGroupCache (sync.Once per filter+groupBy key)
  so identical grouped stats queries execute only once per GQL operation
- Parallelize 4 histogram queries in AddHistograms using errgroup
- Consolidate frontend from 6 GQL aliases to 2, sort+slice top-10
  client-side via $derived

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: 5b26a6e5ff10
 2026-03-13 14:31:37 +01:00
moebiusband cbe46c3524 Merge branch 'hotfix' of github.com:ClusterCockpit/cc-backend into hotfix 2026-03-13 13:17:34 +01:00
moebiusbandandClaude Opus 4.6 0037d969b2 Consolidate UsageDash into single GraphQL query 
Merge three separate queries (topJobsQuery, topNodesQuery, topAccsQuery)
into one topStatsQuery with 6 aliased jobsStatistics fields, reducing
3 HTTP round trips to 1 on the status dashboard.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: 40d806a3240c
 2026-03-13 13:14:29 +01:00
moebiusbandandClaude Opus 4.6 dd3e5427f4 Add covering indexes for status/dashboard queries (migration 13) 
Adds composite covering indexes on (cluster, job_state, <group_col>, ...)
for user, project, and subcluster groupings to enable index-only scans
for status views. Drops subsumed 3-column indexes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: 3d8def28e96e
 2026-03-13 13:12:54 +01:00
moebiusband 39ab12784c Make checkpointInterval an option config option again. 
Also applies small fixes

Entire-Checkpoint: c11d1a65fae4
 2026-03-13 09:07:38 +01:00
moebiusbandandClaude Opus 4.6 b214e1755a Add buffered I/O to WAL writes and fix MemoryCap comment 
WAL writes now go through bufio.Writer instead of raw syscalls per record,
reducing I/O overhead. Buffers are flushed on rotate, drain, and shutdown.
Fixed misleading MemoryCap comment ("Max bytes" → "Max memory in GB").

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: b38dc35e5334
 2026-03-13 09:05:24 +01:00
moebiusband a4f9ba6975 Apply correct log level 
Entire-Checkpoint: 8288af281b94
 2026-03-13 07:58:57 +01:00
moebiusbandandClaude Opus 4.6 8234ad3126 fix: Fix metricstore memory explosion from broken emergency free and batch aborts 
- Fix MemoryUsageTracker: remove premature bufferPool.Clear() that prevented
  mem.Alloc from decreasing, replace broken ForceFree loop (100 iterations
  with no GC) with progressive time-based Free at 75%/50%/25% retention,
  add bufferPool.Clear()+GC between steps so memory stats update correctly
- Enable debug.FreeOSMemory() after emergency freeing to return memory to OS
- Add adaptive ticker: 30s checks when memory >80% of cap, normal otherwise
- Reduce default memory check interval from 1h to 5min
- Don't abort entire NATS batch on single write error (out-of-order timestamp),
  log warning and continue processing remaining lines
- Prune empty levels from tree after free() to reduce overhead
- Include buffer struct overhead in sizeInBytes() for more accurate reporting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entire-Checkpoint: 7ce28627fc1d
 2026-03-13 07:57:35 +01:00
moebiusband 126f65879a Update Release informations 
Entire-Checkpoint: 9f282c3d9570
 2026-03-13 06:23:33 +01:00
moebiusband 3aacc669b6 Remove debug timer 2026-03-13 06:18:21 +01:00
moebiusband 96fc44a649 fix: Optimize project stat query 2026-03-13 06:06:38 +01:00
moebiusband 8e86e8720d Make stats query selective. Add stats index. Add paging to user list. 
Entire-Checkpoint: d42431eee30d
 2026-03-12 20:16:55 +01:00
moebiusband 4555fb8a86 Merge branch 'hotfix' of github.com:ClusterCockpit/cc-backend into hotfix 2026-03-12 20:15:54 +01:00
moebiusband 0e27624d73 Add flag to optimize db. Remove ANALYZE on startup. 
Entire-Checkpoint: d49917ff4b10
 2026-03-12 20:12:49 +01:00
moebiusband 7f069f1ec1 Prepare bugfix release 1.5.1 
Entire-Checkpoint: 15cc90a0347a
 2026-03-12 06:40:36 +01:00
moebiusband 2506a92cdf Remove entire log 2026-03-12 06:14:11 +01:00
moebiusband af78f06ced fix: Reduce complexity for groupBy stats queries 
Entire-Checkpoint: fc899a70a751
 2026-03-11 15:14:59 +01:00
moebiusband 6e0fe62566 Add new db config options to README 2026-03-11 14:30:41 +01:00
moebiusband e70310dcbc fix: Segvault when taggers are enabled but rule directories missing 2026-03-11 11:15:08 +01:00
moebiusband 00d2f97c4c fix: Large heap allocations in sqlite driver. Sanitize sqlite config and make it configurablex. Allow to cancel queries. 2026-03-11 11:14:37 +01:00
moebiusband c8d8f7084a Merge branch 'hotfix' of github.com:ClusterCockpit/cc-backend into hotfix 2026-03-11 07:50:55 +01:00
moebiusband dc7407d0f0 fix: prevent segvault if enable-job-taggers option is tru but tagger config directories are missing 
Entire-Checkpoint: 9ec86e3669e1
 2026-03-11 07:50:53 +01:00
moebiusband eba3995610 Add Analyse on db startup 
Entire-Checkpoint: ea70a955214d
 2026-03-11 05:28:52 +01:00
moebiusband f8831e7040 Fixed merge errors 
Entire-Checkpoint: ddd4fa4a7bbb
 2026-03-11 05:09:38 +01:00
moebiusband 1cf99206a9 Merge branch 'hotfix' of github.com:ClusterCockpit/cc-backend into hotfix 2026-03-11 05:06:26 +01:00
moebiusband 5d3d77620e fix: Add defer.close for all queries 2026-03-11 05:04:20 +01:00
moebiusband 282197ebef fix: Round floats in tagger message 
Entire-Checkpoint: b68850c6fcff
 2026-03-10 06:01:31 +01:00
moebiusband 70fea39d03 Add note on dynamic memory management for restarts 2026-03-06 10:56:23 +01:00
moebiusband ddda341e10 Safeguard metricstore shutdown if internal metricstore is not initialized 2026-03-05 10:37:33 +01:00
moebiusband 47181330e9 Update to latest cc-lib 2026-03-04 17:39:46 +01:00
moebiusband 845d0111af Further consolidate and improve ccms query builder 
Entire-Checkpoint: d10e6221ee4f
 2026-03-04 17:31:36 +01:00
moebiusband 26982088c3 Consolidate code for external and internal ccms buildQueries function 
Entire-Checkpoint: fc3be444ef4c
 2026-03-04 16:43:05 +01:00
moebiusband 67a17b5306 Reduce noise in info log 2026-03-04 15:14:35 +01:00
moebiusband 75591eb034 Merge branch 'dev' of github.com:ClusterCockpit/cc-backend into dev 2026-03-04 15:04:56 +01:00
moebiusband 33ec755422 Fix typo in job high memory tagger 2026-03-04 15:04:53 +01:00
moebiusband 2b74ac822f Merge branch 'master' into dev 2026-03-04 14:33:23 +01:00
moebiusband cc0403e2a4 Fix goreleaser config 
Entire-Checkpoint: a204a44fa885
 2026-03-04 14:32:40 +01:00
moebiusband 87425c0b09 Cleanup and update example config files 2026-03-04 10:41:52 +01:00
moebiusband 39635ea123 Cleanup metricstore options 
Entire-Checkpoint: 2f9a4e1c2e87
 2026-03-04 10:37:43 +01:00
moebiusband 6da41982ce Further optimize queries for better index usage 2026-03-03 15:54:51 +01:00
moebiusband 763e0c8d7c Upgrade dependencies 2026-03-03 15:43:38 +01:00
moebiusband b13e1628a0 Merge branch 'dev' of github.com:ClusterCockpit/cc-backend into dev 2026-03-03 15:41:51 +01:00
moebiusband 5669eb5818 Optimize queries for existing indices 2026-03-03 15:41:44 +01:00
moebiusband 688ad507a2 Merge branch 'optimize-checkpoint-wal' into dev 2026-03-03 06:58:28 +01:00
moebiusband 15be664ad8 Add entire gitignore 2026-03-03 06:58:03 +01:00
moebiusband 22c442db5b Enable entire integration 2026-03-02 18:47:45 +01:00
moebiusband 3d5a124321 Refine patterns. Do not match commented lines. 2026-03-02 15:01:44 +01:00
moebiusband 1ec41d8389 Review and improve buffer pool implmentation. Add unit tests. 2026-02-28 19:34:33 +01:00
moebiusband 888d7fb235 Merge branch 'optimize-checkpoint-wal' of github.com:ClusterCockpit/cc-backend into optimize-checkpoint-wal 2026-02-27 17:40:34 +01:00
moebiusband adebffd251 Replace the old zip archive options for the metricstore node data by parquet files 2026-02-27 17:40:32 +01:00
moebiusband a418abc7d5 Run go fix 2026-02-27 14:40:26 +01:00
moebiusband a1db8263d7 Document line protocol. Optimize REST writeMetric path 2026-02-27 12:30:27 +01:00
moebiusband 4c3cd8e66a Merge branch 'dev' into optimize-checkpoint-wal 2026-02-27 09:30:32 +01:00
moebiusband 6ecb934967 Switch to CC line-protocol package. Update cc-lib. 2026-02-27 08:55:33 +01:00
moebiusband ca0f9a42c7 Introduce metric store binary checkpoints with write ahead log 2026-02-26 10:08:40 +01:00
moebiusband 92236b1d1d Merge branch 'dev' of github.com:ClusterCockpit/cc-backend into dev 2026-02-25 18:54:52 +01:00
moebiusband cc21e0e62c Make json the default checkpoint format 2026-02-25 07:38:19 +01:00
moebiusband 45a1bc78b7 Add csv export for user/prject list 
Fixes #389
 2026-02-24 20:26:12 +01:00
moebiusband 5d136634a2 Update jobclass rules 2026-02-24 07:00:15 +01:00
moebiusband 998aff2345 Fix bug in taggers. Allow to add multiple tags to one job 2026-02-24 06:44:17 +01:00
moebiusband dadcb983e7 Merge branch 'dev' into add_GetMemoryDomainsBySocket_2026 2026-02-23 18:47:03 +01:00
moebiusband 8989b7a410 Update cclib 2026-02-23 18:45:41 +01:00
moebiusband ab55ce91a1 Change omit tagged key to kebab case 2026-02-23 09:02:36 +01:00
moebiusband 03c65e06f6 Allow finer control for omit tagged jobs in retention policies 2026-02-23 08:46:47 +01:00
moebiusband defa8fa994 Update ReleaseNotes for v1.5.0 2026-02-23 08:45:49 +01:00
moebiusband c9d8de0d56 Fix and extend jobclass rules 2026-02-22 13:27:51 +01:00
moebiusband 86fbecc679 Update frontend deps 2026-02-22 09:45:48 +01:00
moebiusband 8ee6c09e9b Enable to run taggers from within admin web interface 2026-02-22 09:42:53 +01:00
moebiusband fc1ba1f5b3 Merge branch 'dev' of github.com:ClusterCockpit/cc-backend into dev 2026-02-21 13:52:14 +01:00
moebiusband 82e79b074a Reverse Lookup order in stop job request 2026-02-21 13:51:31 +01:00
moebiusband e1c1148160 Fix more bugs related to job_cache ids used in job table 2026-02-20 09:20:18 +01:00
moebiusband dc161ec421 Fix bug in SyncJobs causing start job hooks being called with job_cache ID 2026-02-20 08:29:41 +01:00
moebiusband abd11d783b Use alternative solution to deal with NULL values in health_state column 2026-02-20 08:12:15 +01:00
moebiusband 064aa0a238 Add logging for AppTagger 2026-02-20 07:44:00 +01:00
moebiusband f00f9fcee0 Cleanup debug printf 2026-02-19 17:42:45 +01:00
moebiusband 90b52f997d Cleanup and handle error in AppTagger 2026-02-19 08:24:39 +01:00
moebiusband 745c0357f3 Handle NULL values on health_metrics column 2026-02-19 08:04:45 +01:00
moebiusband 57536d982c Add log output and fix bugs in AppTagger 2026-02-18 16:00:50 +01:00
moebiusband 7bd79dcc3c Add nodestate retention in example config 2026-02-18 10:28:41 +01:00
moebiusband 2a659915a4 Update config schema 2026-02-18 10:18:33 +01:00
moebiusband d8ad6dd3f0 Upgrade cc-lib to 2.5.1 2026-02-18 09:21:05 +01:00
moebiusband cd8b574cca Rename nodestate retention policy 2026-02-18 08:38:22 +01:00
moebiusband 2da35909c1 Optimize sort order in job parquet files 2026-02-18 08:13:00 +01:00
moebiusband 2e24fde430 Optimize sort order in nodestate parquet files 2026-02-18 08:06:00 +01:00
moebiusband 757be60b22 Switch to zstd compression for parquet writers 2026-02-18 07:55:28 +01:00
moebiusband 589149790f Upgrade dependencies 2026-02-18 07:45:45 +01:00