cc-metric-store/TODO.md

# Possible Tasks and Improvements

Importance:

- **I** Important
- **N** Nice to have
- **W** Won't do. Probably not necessary.

- Benchmarking
  - Benchmark and compare common timeseries DBs with our data and our queries (N)
- Web interface
  - Provide simple http endpoint with a status and debug view (Start with Basic
    Authentication)
- Configuration
  - Consolidate configuration with cc-backend, remove redundant information
  - Support to receive configuration via NATS channel
- Memory management
  - To overcome garbage collection overhead: Reimplement in Rust (N)
  - Request memory directly batchwise via mmap (started in branch) (W)
- Archive
  - S3 backend for archive (I)
  - Store information in each buffer if already archived (N)
  - Do not create new checkpoint if all buffers already archived (N)
- Checkpoints
  - S3 backend for checkpoints (I)
  - Combine checkpoints into larger files (I)
  - Binary checkpoints (started in branch) (W)
- API
  - Redesign query interface (N)
  - Provide an endpoint for node health based on received metric data (I)
  - Introduce JWT authentication for REST and NATS (I)
- Testing
  - General tests (I)
  - Test data generator for regression tests (I)
  - Check for corner cases that should fail gracefully (N)
  - Write a more realistic `ToArchive`/`FromArchive` Tests (N)
- Aggregation
  - Calculate averages buffer-wise as soon as full, average weighted by length of buffer (N)
  - Only the head-buffer needs to be fully traversed (N)
  - If aggregating over hwthreads/cores/sockets cache those results and reuse
    some of that for new queries aggregating only over the newer data (W)
- Core functionality
  - Implement a health checker component that provides information to the web
    interface and REST API (I)
  - Support units for metrics including to request unit conversions (I)
- Compression
  - Enable compression for http API requests (N)
  - Enable compression for checkpoints/archive (I)
- Sampling
  - Support data re sampling to reduce data points (I)
  - Use re sampling algorithms that preserve min/max as far as possible (I)
Update Todos 2024-06-28 07:31:41 +02:00			`# Possible Tasks and Improvements`
Initial checkin 2021-06-09 06:03:31 +02:00
Update todos. Cleanup 2024-06-28 08:44:24 +02:00			`Importance:`

			`- I Important`
			`- N Nice to have`
			`- W Won't do. Probably not necessary.`

Update Todos 2024-06-28 07:31:41 +02:00			`- Benchmarking`
Update todos. Cleanup 2024-06-28 08:44:24 +02:00			`- Benchmark and compare common timeseries DBs with our data and our queries (N)`
Update Todo 2024-06-28 09:33:16 +02:00			`- Web interface`
			`- Provide simple http endpoint with a status and debug view (Start with Basic`
			`Authentication)`
			`- Configuration`
			`- Consolidate configuration with cc-backend, remove redundant information`
			`- Support to receive configuration via NATS channel`
Update Todos 2024-06-28 07:31:41 +02:00			`- Memory management`
Update todos. Cleanup 2024-06-28 08:44:24 +02:00			`- To overcome garbage collection overhead: Reimplement in Rust (N)`
			`- Request memory directly batchwise via mmap (started in branch) (W)`
Update Todos 2024-06-28 07:31:41 +02:00			`- Archive`
Update todos. Cleanup 2024-06-28 08:44:24 +02:00			`- S3 backend for archive (I)`
			`- Store information in each buffer if already archived (N)`
			`- Do not create new checkpoint if all buffers already archived (N)`
Update Todos 2024-06-28 07:31:41 +02:00			`- Checkpoints`
Update todos. Cleanup 2024-06-28 08:44:24 +02:00			`- S3 backend for checkpoints (I)`
			`- Combine checkpoints into larger files (I)`
			`- Binary checkpoints (started in branch) (W)`
Update Todos 2024-06-28 07:31:41 +02:00			`- API`
Update todos. Cleanup 2024-06-28 08:44:24 +02:00			`- Redesign query interface (N)`
Update Todo 2024-06-28 09:33:16 +02:00			`- Provide an endpoint for node health based on received metric data (I)`
Update todos. Cleanup 2024-06-28 08:44:24 +02:00			`- Introduce JWT authentication for REST and NATS (I)`
Update Todos 2024-06-28 07:31:41 +02:00			`- Testing`
Update todos. Cleanup 2024-06-28 08:44:24 +02:00			`- General tests (I)`
			`- Test data generator for regression tests (I)`
			`- Check for corner cases that should fail gracefully (N)`
			- Write a more realistic `ToArchive`/`FromArchive` Tests (N)
Update Todos 2024-06-28 07:31:41 +02:00			`- Aggregation`
Update todos. Cleanup 2024-06-28 08:44:24 +02:00			`- Calculate averages buffer-wise as soon as full, average weighted by length of buffer (N)`
			`- Only the head-buffer needs to be fully traversed (N)`
Update Todos 2024-06-28 07:31:41 +02:00			`- If aggregating over hwthreads/cores/sockets cache those results and reuse`
Update todos. Cleanup 2024-06-28 08:44:24 +02:00			`some of that for new queries aggregating only over the newer data (W)`
More Todos 2024-06-28 09:36:28 +02:00			`- Core functionality`
			`- Implement a health checker component that provides information to the web`
			`interface and REST API (I)`
			`- Support units for metrics including to request unit conversions (I)`
Update Todos 2024-06-28 07:31:41 +02:00			`- Compression`
Update todos. Cleanup 2024-06-28 08:44:24 +02:00			`- Enable compression for http API requests (N)`
			`- Enable compression for checkpoints/archive (I)`
Update Todos 2024-06-28 07:31:41 +02:00			`- Sampling`
Update todos. Cleanup 2024-06-28 08:44:24 +02:00			`- Support data re sampling to reduce data points (I)`
			`- Use re sampling algorithms that preserve min/max as far as possible (I)`