From d462db7d1dc7a068c7adbc74254a5aa7046c4843 Mon Sep 17 00:00:00 2001 From: Thomas Roehl Date: Thu, 3 Feb 2022 18:47:39 +0100 Subject: [PATCH] Add documentation --- sinks/README.md | 126 +++++++++++++++++++++---------------------- sinks/gangliaSink.md | 21 ++++++++ sinks/httpSink.md | 25 +++++++++ sinks/influxSink.md | 32 +++++++++++ sinks/natsSink.md | 28 ++++++++++ sinks/stdoutSink.md | 22 ++++++++ 6 files changed, 188 insertions(+), 66 deletions(-) create mode 100644 sinks/gangliaSink.md create mode 100644 sinks/httpSink.md create mode 100644 sinks/influxSink.md create mode 100644 sinks/natsSink.md create mode 100644 sinks/stdoutSink.md diff --git a/sinks/README.md b/sinks/README.md index 8fac8e5..8af2a8a 100644 --- a/sinks/README.md +++ b/sinks/README.md @@ -2,17 +2,24 @@ This folder contains the SinkManager and sink implementations for the cc-metric-collector. +# Available sinks: +- `stdout`: Print all metrics to `stdout`, `stderr` or a file +- `http`: Send metrics to an HTTP server as POST requests +- `influxdb`: Send metrics to an InfluxDB database +- `nats`: Publish metrics to the NATS network overlay system +- `ganglia`: Publish metrics in the [Ganglia Monitoring System](http://ganglia.info/) + # Configuration The configuration file for the sinks is a list of configurations. The `type` field in each specifies which sink to initialize. ```json [ - { + "mystdout" : { "type" : "stdout", "meta_as_tags" : false }, - { + "metricstore" : { "type" : "http", "host" : "localhost", "port" : "4123", @@ -22,74 +29,12 @@ The configuration file for the sinks is a list of configurations. The `type` fie ] ``` -This example initializes two sinks, the `stdout` sink printing all metrics to the STDOUT and the `http` sink with the given `host`, `port`, `database` and `password`. - -If `meta_as_tags` is set, all meta information attached to CCMetric are printed out as tags. - -## Type `stdout` - -```json -{ - "type" : "stdout", - "meta_as_tags" : -} -``` - -The `stdout` sink dumps all metrics to the STDOUT. - -## Type `http` - -```json -{ - "type" : "http", - "host" : "", - "port" : "", - "database" : "", - "password" : "", - "meta_as_tags" : -} -``` -The sink uses POST requests to send metrics to `http://:/` using the JWT token as a JWT in the 'Authorization' header. - -## Type `nats` - -```json -{ - "type" : "nats", - "host" : "", - "port" : "", - "user" : "", - "password" : "", - "database" : "" - "meta_as_tags" : -} -``` - -This sink publishes the CCMetric in a NATS environment using `host`, `port`, `user` and `password` for connecting. The metrics are published using the topic `database`. - -## Type `influxdb` - -```json -{ - "type" : "influxdb", - "host" : "", - "port" : "", - "user" : "", - "password" : "", - "database" : "" - "organization": "", - "ssl" : , - "meta_as_tags" : -} -``` - -This sink submits the CCMetrics to an InfluxDB time-series database. It uses `host`, `port` and `ssl` for connecting. For authentification, it uses either `user:password` if `user` is set and only `password` as API key. The `organization` and `database` are used for writing to the correct database. # Contributing own sinks -A sink contains three functions and is derived from the type `Sink`: -* `Init(config SinkConfig) error` +A sink contains four functions and is derived from the type `sink`: +* `Init(config json.RawMessage) error` * `Write(point CCMetric) error` * `Flush() error` * `Close()` @@ -97,3 +42,52 @@ A sink contains three functions and is derived from the type `Sink`: The data structures should be set up in `Init()` like opening a file or server connection. The `Write()` function writes/sends the data. For non-blocking sinks, the `Flush()` method tells the sink to drain its internal buffers. The `Close()` function should tear down anything created in `Init()`. Finally, the sink needs to be registered in the `sinkManager.go`. There is a list of sinks called `AvailableSinks` which is a map (`sink_type_string` -> `pointer to sink interface`). Add a new entry with a descriptive name and the new sink. + +## Sample sink + +```go +package sinks + +import ( + "encoding/json" + "log" + lp "github.com/ClusterCockpit/cc-metric-collector/internal/ccMetric" +) + +type SampleSinkConfig struct { + defaultSinkConfig // defines JSON tags for 'name' and 'meta_as_tags' +} + +type SampleSink struct { + sink // declarate 'name' and 'meta_as_tags' + config StdoutSinkConfig // entry point to the SampleSinkConfig +} + +// Initialize the sink by giving it a name and reading in the config JSON +func (s *SampleSink) Init(config json.RawMessage) error { + s.name = "SampleSink" // Always specify a name here + // Read in the config JSON + if len(config) > 0 { + err := json.Unmarshal(config, &s.config) + if err != nil { + return err + } + } + return nil +} + +// Code to submit a single CCMetric to the sink +func (s *SampleSink) Write(point lp.CCMetric) error { + log.Print(point) + return nil +} + +// If the sink uses batched sends internally, you can tell to flush its buffers +func (s *SampleSink) Flush() error { + return nil +} + + +// Close sink: close network connection, close files, close libraries, ... +func (s *SampleSink) Close() {} +``` \ No newline at end of file diff --git a/sinks/gangliaSink.md b/sinks/gangliaSink.md new file mode 100644 index 0000000..9b77ac9 --- /dev/null +++ b/sinks/gangliaSink.md @@ -0,0 +1,21 @@ +## `ganglia` sink + +The `ganglia` sink uses the `gmetric` tool of the [Ganglia Monitoring System](http://ganglia.info/) to submit the metrics + +### Configuration structure + +```json +{ + "": { + "type": "ganglia", + "meta_as_tags" : true, + "gmetric_path" : "/path/to/gmetric", + "add_ganglia_group" : true + } +} +``` + +- `type`: makes the sink an `ganglia` sink +- `meta_as_tags`: print all meta information as tags in the output (optional) +- `gmetric_path`: Path to `gmetric` executable (optional). If not given, the sink searches in `$PATH` for `gmetric`. +- `add_ganglia_group`: Add `--group=X` based on meta information to the `gmetric` call. Some old versions of `gmetric` do not support the `--group` option. \ No newline at end of file diff --git a/sinks/httpSink.md b/sinks/httpSink.md new file mode 100644 index 0000000..3c9dfd9 --- /dev/null +++ b/sinks/httpSink.md @@ -0,0 +1,25 @@ +## `http` sink + +The `http` sink uses POST requests to a HTTP server to submit the metrics in the InfluxDB line-protocol format. It uses JSON web tokens for authentification. The sink creates batches of metrics before sending, to reduce the HTTP traffic. + +### Configuration structure + +```json +{ + "": { + "type": "http", + "meta_as_tags" : true, + "database" : "mymetrics", + "host": "dbhost.example.com", + "port": "4222", + "jwt" : "0x0000q231", + } +} +``` + +- `type`: makes the sink an `http` sink +- `meta_as_tags`: print all meta information as tags in the output (optional) +- `database`: All metrics are written to this bucket +- `host`: Hostname of the InfluxDB database server +- `port`: Portnumber (as string) of the InfluxDB database server +- `jwt`: JSON web tokens for authentification \ No newline at end of file diff --git a/sinks/influxSink.md b/sinks/influxSink.md new file mode 100644 index 0000000..2624034 --- /dev/null +++ b/sinks/influxSink.md @@ -0,0 +1,32 @@ +## `influxdb` sink + +The `influxdb` sink uses the official [InfluxDB golang client](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2) to write the metrics to an InfluxDB database. It provides only support for V2 write endpoints (InfluxDB 1.8.0 or later). + + +### Configuration structure + +```json +{ + "": { + "type": "influxdb", + "meta_as_tags" : true, + "database" : "mymetrics", + "host": "dbhost.example.com", + "port": "4222", + "user": "exampleuser", + "password" : "examplepw", + "organization": "myorg", + "ssl": true, + } +} +``` + +- `type`: makes the sink an `influxdb` sink +- `meta_as_tags`: print all meta information as tags in the output (optional) +- `database`: All metrics are written to this bucket +- `host`: Hostname of the InfluxDB database server +- `port`: Portnumber (as string) of the InfluxDB database server +- `user`: Username for basic authentification +- `password`: Password for basic authentification +- `organization`: Organization in the InfluxDB +- `ssl`: Use SSL connection \ No newline at end of file diff --git a/sinks/natsSink.md b/sinks/natsSink.md new file mode 100644 index 0000000..7a53f27 --- /dev/null +++ b/sinks/natsSink.md @@ -0,0 +1,28 @@ +## `nats` sink + +The `nats` sink publishes all metrics into a NATS network. The publishing key is the database name provided in the configuration file + + +### Configuration structure + +```json +{ + "": { + "type": "nats", + "meta_as_tags" : true, + "database" : "mymetrics", + "host": "dbhost.example.com", + "port": "4222", + "user": "exampleuser", + "password" : "examplepw" + } +} +``` + +- `type`: makes the sink an `nats` sink +- `meta_as_tags`: print all meta information as tags in the output (optional) +- `database`: All metrics are published with this subject +- `host`: Hostname of the NATS server +- `port`: Portnumber (as string) of the NATS server +- `user`: Username for basic authentification +- `password`: Password for basic authentification \ No newline at end of file diff --git a/sinks/stdoutSink.md b/sinks/stdoutSink.md new file mode 100644 index 0000000..317ca3f --- /dev/null +++ b/sinks/stdoutSink.md @@ -0,0 +1,22 @@ +## `stdout` sink + +The `stdout` sink is the most simple sink provided by cc-metric-collector. It writes all metrics in InfluxDB line-procol format to the configurable output file or the common special files `stdout` and `stderr`. + + +### Configuration structure + +```json +{ + "": { + "type": "stdout", + "meta_as_tags" : true, + "output_file" : "mylogfile.log" + } +} +``` + +- `type`: makes the sink an `stdout` sink +- `meta_as_tags`: print all meta information as tags in the output (optional) +- `output_file`: Write all data to the selected file (optional). There are two 'special' files: `stdout` and `stderr`. If this option is not provided, the default value is `stdout` + +