ClusterCockpit/cc-metric-collector
1
0
Fork 0
mirror of https://github.com/ClusterCockpit/cc-metric-collector.git synced 2024-12-25 15:09:05 +01:00
Code Issues Packages Projects Releases Wiki Activity

Update READMEs

Browse Source
This commit is contained in:
Thomas Roehl 2021-05-19 01:46:52 +02:00
parent da11523448
commit 99f3f3b815
2 changed files with 31 additions and 15 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

44
README.md
View File

@ -13,15 +13,6 @@ Supported metrics are documented [here](https://github.com/ClusterCockpit/cc-spe
``` json
{
"sink": {
"user": "admin",
"password": "12345",
"host": "localhost",
"port": "8080",
"database": "testdb",
"organisation": "testorg",
"type": "stdout"
},
"interval" : 3,
"duration" : 1,
"collectors": [
@ -35,6 +26,18 @@ Supported metrics are documented [here](https://github.com/ClusterCockpit/cc-spe
"cpustat",
"nvidia"
],
"sink": {
"user": "admin",
"password": "12345",
"host": "localhost",
"port": "8080",
"database": "testdb",
"organisation": "testorg",
"type": "stdout"
},
"default_tags": {
"cluster": "testcluster"
},
"receiver": {
"type": "none",
"address": "127.0.0.1",
@ -44,7 +47,15 @@ Supported metrics are documented [here](https://github.com/ClusterCockpit/cc-spe
}
```
All available collectors are listed in the above JSON. There are currently three sinks supported `influxdb`, `nats` and `stdout`. The `interval` defines how often the metrics should be read and send to the sink. The `duration` tells collectors how long one measurement has to take. An example for this is the `likwid` collector which starts the hardware performance counter, waits for `duration` seconds and stops the counters again. For most systems, the `likwid` collector has to do two measurements, thus `interval` must be larger than two times `duration`. With `receiver`, the collector can be used as a router by receiving metrics and forwarding them to the configured sink. There are currently only types `none` (for no receiver) and `nats`.
The `interval` defines how often the metrics should be read and send to the sink. The `duration` tells collectors how long one measurement has to take. An example for this is the `likwid` collector which starts the hardware performance counter, waits for `duration` seconds and stops the counters again. For most systems, the `likwid` collector has to do two measurements, thus `interval` must be larger than two times `duration`.
All available collectors are listed in the above JSON. A more detailed list can be found in the [README for collectors](./collectors/README.md).
There are currently three sinks supported `influxdb`, `nats` and `stdout`. See [README for sinks](./sinks/README.md).
In the `default_tags` section, one can define key-value-pairs (only strings) that are added to each sent out metric. This can be useful for cluster names like in the example JSON or information like rank or island for orientation.
With `receiver`, the collector can be used as a router by receiving metrics and forwarding them to the configured sink. There are currently only types `none` (for no receiver) and `nats`. For more information see the [README in receivers](./receivers/README.md).
# Installation
@ -52,7 +63,7 @@ All available collectors are listed in the above JSON. There are currently three
$ git clone git@github.com:ClusterCockpit/cc-metric-collector.git
$ cd cc-metric-collector/collectors
$ edit Makefile (for LIKWID collector)
$ make (downloads LIKWID, builds it as static library and copies all required files for the collector)
$ make (downloads LIKWID, builds it as static library and copies all required files for the collector. Uses sudo in case of own accessdaemon)
$ cd ..
$ go get (requires at least golang 1.13)
$ go build metric-collector
@ -64,14 +75,19 @@ $ go build metric-collector
$ ./metric-collector --help
Usage of metric-collector:
-config string
Path to configuration file (default "./config.json")
Path to configuration file (default "./config.json")
-log string
Path for logfile (default "stderr")
Path for logfile (default "stderr")
```
# Internals
The metric collector sends (and receives) metric in the [InfluxDB line protocol](https://docs.influxdata.com/influxdb/cloud/reference/syntax/line-protocol/) as it provides flexibility while providing a separation between tags (like index columns in relational databases) and fields (like data columns).
There is a single timer loop that triggers all collectors serially, collects the collectors' data and sends the metrics to the sink. The sinks currently use blocking APIs.
There is a single timer loop that triggers all collectors serially, collects the collectors' data and sends the metrics to the sink. This is done as all data is submitted with a single time stamp. The sinks currently use mostly blocking APIs.
The receiver runs as a go routine side-by-side with the timer loop and asynchronously forwards received metrics to the sink.
# Todos
- [ ] Use only non-blocking APIs for the sinks
- [ ] Collector specific configuration in global JSON file? Changing the configuration inside the Go code is not user-friendly.

2
sinks/README.md
View File

@ -5,7 +5,7 @@ The base class/configuration is located in `metricSink.go`.
# Sinks
* `stdoutSink.go`: Writes all metrics to `stdout` in InfluxDB line protocol. The sink does not use https://github.com/influxdata/line-protocol to reduce the executed code for debugging
* `influxSink.go`: Writes all metrics to an InfluxDB database instance using a blocking writer. It uses https://github.com/influxdata/influxdb-client-go . Configuration for the server, port, user, password and database name are in the global configuration file
* `influxSink.go`: Writes all metrics to an InfluxDB database instance using a blocking writer. It uses https://github.com/influxdata/influxdb-client-go . Configuration for the server, port, user, password, database name and organisation are in the global configuration file. It uses the v2 API of Influx.
* `natsSink.go`: Sends all metrics to an NATS server using the InfluxDB line protocol as encoding. It uses https://github.com/nats-io/nats.go . Configuration for the server, port, user, password and database name are in the global configuration file. The database name is used as subject for the NATS messages.
# Installation