cc-metric-store/README.md

142 lines
6.5 KiB
Markdown
Raw Normal View History

2021-06-09 06:03:31 +02:00
# ClusterCockpit Metric Store
2021-09-01 08:48:35 +02:00
[![Build & Test](https://github.com/ClusterCockpit/cc-metric-store/actions/workflows/test.yml/badge.svg)](https://github.com/ClusterCockpit/cc-metric-store/actions/workflows/test.yml)
2021-08-31 15:18:06 +02:00
2022-01-07 08:49:13 +01:00
The cc-metric-store provides a simple in-memory time series database for storing metrics of cluster nodes at preconfigured intervals. It is meant to be used as part of the [ClusterCockpit suite](https://github.com/ClusterCockpit). As all data is kept in-memory (but written to disk as compressed JSON for long term storage), accessing it is very fast. It also provides aggregations over time *and* nodes/sockets/cpus.
2022-01-31 10:53:06 +01:00
There are major limitations: Data only gets written to disk at periodic checkpoints, not as soon as it is received.
Go look at the `TODO.md` file and the [GitHub Issues](https://github.com/ClusterCockpit/cc-metric-store/issues) for a progress overview. Things work, but are not properly tested.
2021-10-12 13:26:54 +02:00
The [NATS.io](https://nats.io/) based writing endpoint consumes messages in [this format of the InfluxDB line protocol](https://github.com/ClusterCockpit/cc-specifications/blob/master/metrics/lineprotocol_alternative.md).
2021-08-24 10:41:30 +02:00
### REST API Endpoints
2022-01-20 10:43:10 +01:00
The REST API is documented in [openapi.yaml](./openapi.yaml) in the OpenAPI 3.0 format.
2021-08-24 10:41:30 +02:00
### Run tests
2021-08-31 15:18:06 +02:00
Some benchmarks concurrently access the `MemoryStore`, so enabling the
[Race Detector](https://golang.org/doc/articles/race_detector) might be useful.
The benchmarks also work as tests as they do check if the returned values are as
expected.
2021-08-24 10:41:30 +02:00
```sh
2021-08-31 15:18:06 +02:00
# Tests only
go test -v ./...
# Benchmarks as well
go test -bench=. -race -v ./...
2021-08-24 10:41:30 +02:00
```
2022-01-20 10:43:10 +01:00
### What are these selectors mentioned in the code?
2021-08-31 15:18:06 +02:00
Tags in InfluxDB are used to build indexes over the stored data. InfluxDB-Tags have no
relation to each other, they do not depend on each other and have no hierarchy.
2021-09-01 08:48:35 +02:00
Different tags build up different indexes (I am no expert at all, but this is how i think they work).
2021-08-31 15:18:06 +02:00
This project also works as a time-series database and uses the InfluxDB line protocol.
Unlike InfluxDB, the data is indexed by one single strictly hierarchical tree structure.
A selector is build out of the tags in the InfluxDB line protocol, and can be used to select
a node (not in the sense of a compute node, can also be a socket, cpu, ...) in that tree.
The implementation calls those nodes `level` to avoid confusion.
It is impossible to access data only by knowing the *socket* or *cpu* tag, all higher up levels have to be specified as well.
2021-08-31 15:18:06 +02:00
This is what the hierarchy currently looks like:
2021-08-31 15:18:06 +02:00
- cluster1
- host1
- socket0
- socket1
2021-09-08 12:26:22 +02:00
- ...
- cpu1
- cpu2
- cpu3
- cpu4
2021-08-31 15:18:06 +02:00
- ...
- host2
- ...
- cluster2
- ...
Example selectors:
1. `["cluster1", "host1", "cpu0"]`: Select only the cpu0 of host1 in cluster1
2. `["cluster1", "host1", ["cpu4", "cpu5", "cpu6", "cpu7"]]`: Select only CPUs 4-7 of host1 in cluster1
3. `["cluster1", "host1"]`: Select the complete node. If querying for a CPU-specific metric such as floats, all CPUs are implied
2021-08-31 15:18:06 +02:00
### Config file
2022-01-31 10:53:06 +01:00
All durations are specified as string that will be parsed [like this](https://pkg.go.dev/time#ParseDuration) (Allowed suffixes: `s`, `m`, `h`, ...).
2021-08-31 15:18:06 +02:00
- `metrics`: Map of metric-name to objects with the following properties
2022-01-31 10:53:06 +01:00
- `frequency`: Timestep/Interval/Resolution of this metric
2021-09-01 08:48:35 +02:00
- `aggregation`: Can be `"sum"`, `"avg"` or `null`
- `null` means aggregation across nodes is forbidden for this metric
2021-08-31 15:18:06 +02:00
- `"sum"` means that values from the child levels are summed up for the parent level
- `"avg"` means that values from the child levels are averaged for the parent level
2022-01-31 10:53:06 +01:00
- `scope`: Unused at the moment, should be something like `"node"`, `"socket"` or `"hwthread"`
- `nats`:
- `address`: Url of NATS.io server, example: "nats://localhost:4222"
2022-02-04 08:52:53 +01:00
- `username` and `password`: Optional, if provided use those for the connection
- `subscribe-to`: Where to expect the measurements to be published
- `http-api`:
- `address`: Address to bind to, for example `0.0.0.0:8080`
2022-02-04 08:52:53 +01:00
- `https-cert-file` and `https-key-file`: Optional, if provided enable HTTPS using those files as certificate/key
- `jwt-public-key`: Base64 encoded string, use this to verify requests to the HTTP API
2022-01-31 10:53:06 +01:00
- `retention-on-memory`: Keep all values in memory for at least that amount of time
- `checkpoints`:
- `interval`: Do checkpoints every X seconds/minutes/hours
- `directory`: Path to a directory
- `restore`: After a restart, load the last X seconds/minutes/hours of data back into memory
- `archive`:
- `interval`: Move and compress all checkpoints not needed anymore every X seconds/minutes/hours
- `directory`: Path to a directory
2021-08-31 15:18:06 +02:00
2021-08-24 10:41:30 +02:00
### Test the complete setup (excluding ClusterCockpit itself)
2022-01-31 10:53:06 +01:00
There are two ways for sending data to the cc-metric-store, both of which are supported by the [cc-metric-collector](https://github.com/ClusterCockpit/cc-metric-collector). This example uses Nats, the alternative is to use HTTP.
2021-08-24 10:41:30 +02:00
```sh
# Only needed once, downloads the docker image
docker pull nats:latest
# Start the NATS server
docker run -p 4222:4222 -ti nats:latest
```
2022-01-31 10:53:06 +01:00
Second, build and start the [cc-metric-collector](https://github.com/ClusterCockpit/cc-metric-collector) using the following as Sink-Config:
2021-08-24 10:41:30 +02:00
```json
{
2022-01-31 10:53:06 +01:00
"type": "nats",
"host": "localhost",
"port": "4222",
"database": "updates"
2021-08-24 10:41:30 +02:00
}
```
2021-08-31 15:18:06 +02:00
Third, build and start the metric store. For this example here, the `config.json` file
already in the repository should work just fine.
2021-08-24 10:41:30 +02:00
```sh
# Assuming you have a clone of this repo in ./cc-metric-store:
cd cc-metric-store
go get
go build
./cc-metric-store
```
And finally, use the API to fetch some data. The API is protected by JWT based authentication if `jwt-public-key` is set in `config.json`. You can use this JWT for testing: `eyJ0eXAiOiJKV1QiLCJhbGciOiJFZERTQSJ9.eyJ1c2VyIjoiYWRtaW4iLCJyb2xlcyI6WyJST0xFX0FETUlOIiwiUk9MRV9BTkFMWVNUIiwiUk9MRV9VU0VSIl19.d-3_3FZTsadPjDEdsWrrQ7nS0edMAR4zjl-eK7rJU3HziNBfI9PDHDIpJVHTNN5E5SlLGLFXctWyKAkwhXL-Dw`
2021-08-24 10:41:30 +02:00
```sh
JWT="eyJ0eXAiOiJKV1QiLCJhbGciOiJFZERTQSJ9.eyJ1c2VyIjoiYWRtaW4iLCJyb2xlcyI6WyJST0xFX0FETUlOIiwiUk9MRV9BTkFMWVNUIiwiUk9MRV9VU0VSIl19.d-3_3FZTsadPjDEdsWrrQ7nS0edMAR4zjl-eK7rJU3HziNBfI9PDHDIpJVHTNN5E5SlLGLFXctWyKAkwhXL-Dw"
2021-08-24 10:41:30 +02:00
# If the collector and store and nats-server have been running for at least 60 seconds on the same host, you may run:
2022-01-20 10:43:10 +01:00
curl -H "Authorization: Bearer $JWT" -D - "http://localhost:8080/api/query" -d "{ \"cluster\": \"testcluster\", \"from\": $(expr $(date +%s) - 60), \"to\": $(date +%s), \"queries\": [{
\"metric\": \"load_one\",
\"host\": \"$(hostname)\"
2022-01-20 10:43:10 +01:00
}] }"
2021-08-24 10:41:30 +02:00
# ...
```