ClusterCockpit/cc-metric-store
1
0
Fork 0
mirror of https://github.com/ClusterCockpit/cc-metric-store.git synced 2024-12-26 00:49:05 +01:00
Code Issues Packages Projects Releases Wiki Activity

Update Todos

Browse Source
This commit is contained in:
Jan Eitzinger 2024-06-28 07:31:41 +02:00
parent 362adab938
commit 19a5c6cc1d
Signed by: moebiusband
GPG Key ID: 2574BA29B90D6DD5
1 changed files with 27 additions and 9 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

36
TODO.md
View File

@ -1,15 +1,33 @@
# TODOs # Possible Tasks and Improvements
- Improve checkpoints/archives - Benchmarking
- Benchmark and compare common timeseries DBs with our data and our queries
- Memory management
- To overcome garbage collection overhead: Reimplement in Rust
- Request memory directly batchwise via mmap (started in branch)
- Archive
- S3 backend for archive
- Store information in each buffer if already archived - Store information in each buffer if already archived
- Do not create new checkpoint if all buffers already archived - Do not create new checkpoint if all buffers already archived
- Missing Testcases: - Checkpoints
- S3 backend for checkpoints
- Combine checkpoints into larger files
- Binary checkpoints (started in branch)
- API
- Redesign query interface
- Introduce JWT authentication for REST and NATS
- Testing
- General tests - General tests
- Check for corner cases that should fail gracefully - Check for corner cases that should fail gracefully
- Write a more realistic `ToArchive`/`FromArchive` tests - Write a more realistic `ToArchive`/`FromArchive` Tests
- Optimization: Once a buffer is full, calculate min, max and avg - Aggregation
- Calculate averages buffer-wise, average weighted by length of buffer - Calculate averages buffer-wise as soon as full, average weighted by length of buffer
- Only the head-buffer needs to be fully traversed - Only the head-buffer needs to be fully traversed
- Optimization: If aggregating over hwthreads/cores/sockets cache those results - If aggregating over hwthreads/cores/sockets cache those results and reuse
and reuse some of that for new queres aggregating only over the newer data some of that for new queries aggregating only over the newer data
- ... - Compression
- Enable compression for http API requests
- Enable compression for checkpoints/archive
- Sampling
- Support data re sampling to reduce data points
- Support re sampling algorithms that preserve min/max as far as possible