Modularize the whole thing (#16)

* Use channels, add a metric router, split up configuration and use extended version of Influx line protocol internally * Use central timer for collectors and router. Add expressions to router * Add expression to router config * Update entry points * Start with README * Update README for CCMetric * Formatting * Update README.md * Add README for MultiChanTicker * Add README for MultiChanTicker * Update README.md * Add README to metric router * Update main README * Remove SinkEntity type * Update README for sinks * Update go files * Update README for receivers * Update collectors README * Update collectors README * Use seperate page per collector * Fix for tempstat page * Add docs for customcmd collector * Add docs for ipmistat collector * Add docs for topprocs collector * Update customCmdMetric.md * Use seconds when calculating LIKWID metrics * Add IB metrics ib_recv_pkts and ib_xmit_pkts * Drop domain part of host name * Updated to latest stable version of likwid * Define source code dependencies in Makefile * Add GPFS / IBM Spectrum Scale collector * Add vet and staticcheck make targets * Add vet and staticcheck make targets * Avoid go vet warning: struct field tag `json:"..., omitempty"` not compatible with reflect.StructTag.Get: suspicious space in struct tag value struct field tag `json:"...", omitempty` not compatible with reflect.StructTag.Get: key:"value" pairs not separated by spaces * Add sample collector to README.md * Add CPU frequency collector * Avoid staticcheck warning: redundant return statement * Avoid staticcheck warning: unnecessary assignment to the blank identifier * Simplified code * Add CPUFreqCollectorCpuinfo a metric collector to measure the current frequency of the CPUs as obtained from /proc/cpuinfo Only measure on the first hyperthread * Add collector for NFS clients * Move publication of metrics into Flush() for NatsSink * Update GitHub actions * Refactoring * Avoid vet warning: Println arg list ends with redundant newline * Avoid vet warning struct field commands has json tag but is not exported * Avoid vet warning: return copies lock value. * Corrected typo * Refactoring * Add go sources in internal/... * Bad separator in Makefile * Fix Infiniband collector Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
2025-07-19 03:11:41 +02:00 · 2022-01-25 15:37:43 +01:00
parent 222862af32
commit 200af84c54
60 changed files with 2596 additions and 1105 deletions
--- a/sinks/README.md
+++ b/sinks/README.md
@@ -1,65 +1,99 @@
-This folder contains the sinks for the cc-metric-collector.
+# CCMetric sinks

-# `metricSink.go`
-The base class/configuration is located in `metricSink.go`.
+This folder contains the SinkManager and sink implementations for the cc-metric-collector.

-# Sinks
-* `stdoutSink.go`: Writes all metrics to `stdout` in InfluxDB line protocol. The sink does not use https://github.com/influxdata/line-protocol to reduce the executed code for debugging
-* `influxSink.go`: Writes all metrics to an InfluxDB database instance using a blocking writer. It uses https://github.com/influxdata/influxdb-client-go . Configuration for the server, port, ssl, password, database name and organisation are in the global configuration file. The 'password' is used for the token and the 'database' for the bucket. It uses the v2 API of Influx.
-* `natsSink.go`: Sends all metrics to an NATS server using the InfluxDB line protocol as encoding. It uses https://github.com/nats-io/nats.go . Configuration for the server, port, user, password and database name are in the global configuration file. The database name is used as subject for the NATS messages.
-* `httpSink.go`: Sends all metrics to an HTTP endpoint `http://<host>:<port>/<database>` using a POST request. The body of the request will consist of lines in the InfluxDB line protocol. In case password is specified, that password is used as a JWT in the 'Authorization' header.
+# Configuration

-# Installation
-Nothing to do, all sinks are pure Go code
-
-# Sink configuration
+The configuration file for the sinks is a list of configurations. The `type` field in each specifies which sink to initialize.

 ```json
-  "sink": {
-    "user": "testuser",
-    "password": "testpass",
-    "host": "127.0.0.1",
-    "port": "9090",
-    "database": "testdb",
-    "organization": "testorg",
-    "ssl": false
-    "type": "stdout"
+[
+  {
+    "type" : "stdout",
+    "meta_as_tags" : false
+  },
+  {
+    "type" : "http",
+    "host" : "localhost",
+    "port" : "4123",
+    "database" : "ccmetric",
+    "password" : "<jwt token>"
  }
+]
 ```

-## `stdout`
-When configuring `type = stdout`, all metrics are printed to stdout. No further configuration is required or touched, so you can leave your other-sink-config in there and just change the `type` for debugging purposes
+This example initializes two sinks, the `stdout` sink printing all metrics to the STDOUT and the `http` sink with the given `host`, `port`, `database` and `password`.

-## `influxdb`
-The InfluxDB sink uses blocking write operations to write to an InfluxDB database using the v2 API. It uses the following configuration options:
-* `host`: Hostname of the database instance
-* `port`: Portnumber (as string) of the database
-* `database`: Name of the database, called 'bucket' in InfluxDB v2
-* `organization`: The InfluxDB v2 API uses organizations to separate database instances running on the same host
-* `ssl`: Boolean to activate SSL/TLS
-* `user`: Although the v2 API uses API keys instead of username and password, this field can be used if the sink should authentificate with `username:password`. If you want to use an API key, leave this field empty.
-* `password`: API key for the InfluxDB v2 API or password if `user` is set
+If `meta_as_tags` is set, all meta information attached to CCMetric are printed out as tags.

-## `nats`
-* `host`: Hostname of the NATS server
-* `port`: Portnumber (as string) of the NATS server
-* `user`: Username for authentification in the NATS transport system
-* `password`: Password for authentification in the NATS transport system
+## Type `stdout`
+
+```json
+{
+  "type" : "stdout",
+  "meta_as_tags" : <true|false>
+}
+```
+
+The `stdout` sink dumps all metrics to the STDOUT.
+
+## Type `http`
+
+```json
+{
+  "type" : "http",
+  "host" : "<hostname>",
+  "port" : "<portnumber>",
+  "database" : "<database name>",
+  "password" : "<jwt token>",
+  "meta_as_tags" : <true|false>
+}
+```
+The sink uses POST requests to send metrics to `http://<host>:<port>/<database>` using the JWT token as a JWT in the 'Authorization' header.
+
+## Type `nats`
+
+```json
+{
+  "type" : "nats",
+  "host" : "<hostname>",
+  "port" : "<portnumber>",
+  "user" : "<username>",
+  "password" : "<password>",
+  "database" : "<database name>"
+  "meta_as_tags" : <true|false>
+}
+```
+
+This sink publishes the CCMetric in a NATS environment using `host`, `port`, `user` and `password` for connecting. The metrics are published using the topic `database`. 
+
+## Type `influxdb`
+
+```json
+{
+  "type" : "influxdb",
+  "host" : "<hostname>",
+  "port" : "<portnumber>",
+  "user" : "<username>",
+  "password" : "<password or API key>",
+  "database" : "<database name>"
+  "organization": "<InfluxDB v2 organization>",
+  "ssl" : <true|false>,
+  "meta_as_tags" : <true|false>
+}
+```
+
+This sink submits the CCMetrics to an InfluxDB time-series database. It uses `host`, `port` and `ssl` for connecting. For authentification, it uses either `user:password` if `user` is set and only `password` as API key. The `organization` and `database` are used for writing to the correct database.

-## `http`
-* `host`: Hostname of the HTTP server
-* `port`: Portnumber (as string) of the HTTP server
-* `database`: Endpoint to write to. HTTP POST requests are performed on `http://<host>:<port>/<database>`
-* `password`: JSON Web token used for authentification


 # Contributing own sinks
-A sink contains three functions and is derived from the type `Sink` (in `metricSink.go`):
+A sink contains three functions and is derived from the type `Sink`:
 * `Init(config SinkConfig) error`
-* `Write(measurement string, tags map[string]string, fields map[string]interface{}, t time.Time) error`
+* `Write(point CCMetric) error`
 * `Flush() error`
 * `Close()`

-The data structures should be set up in `Init()` like opening a file or server connection. The `Write()` function takes a measurement, tags, fields and a timestamp and writes/sends the data. For non-blocking sinks, the `Flush()` method tells the sink to drain its internal buffers. The `Close()` function should tear down anything created in `Init()`.
+The data structures should be set up in `Init()` like opening a file or server connection. The `Write()` function writes/sends the data. For non-blocking sinks, the `Flush()` method tells the sink to drain its internal buffers. The `Close()` function should tear down anything created in `Init()`.

-Finally, the sink needs to be registered in the `metric-collector.go`. There is a list of sinks called `Sinks` which is a map (sink_type_string -> pointer to sink). Add a new entry with a descriptive name and the new sink.
+Finally, the sink needs to be registered in the `sinkManager.go`. There is a list of sinks called `AvailableSinks` which is a map (`sink_type_string` -> `pointer to sink interface`). Add a new entry with a descriptive name and the new sink.
--- a/sinks/httpSink.go
+++ b/sinks/httpSink.go
@@ -7,19 +7,21 @@ import (
 	"net/http"
 	"time"

-	lp "github.com/influxdata/line-protocol"
+	lp "github.com/ClusterCockpit/cc-metric-collector/internal/ccMetric"
+	influx "github.com/influxdata/line-protocol"
 )

 type HttpSink struct {
-	Sink
+	sink
 	client   *http.Client
 	url, jwt string
-	encoder  *lp.Encoder
+	encoder  *influx.Encoder
 	buffer   *bytes.Buffer
 }

-func (s *HttpSink) Init(config SinkConfig) error {
-	if len(config.Host) == 0 || len(config.Port) == 0 {
+func (s *HttpSink) Init(config sinkConfig) error {
+	s.name = "HttpSink"
+	if len(config.Host) == 0 || len(config.Port) == 0 || len(config.Database) == 0 {
 		return errors.New("`host`, `port` and `database` config options required for TCP sink")
 	}

@@ -28,13 +30,13 @@ func (s *HttpSink) Init(config SinkConfig) error {
 	s.port = config.Port
 	s.jwt = config.Password
 	s.buffer = &bytes.Buffer{}
-	s.encoder = lp.NewEncoder(s.buffer)
+	s.encoder = influx.NewEncoder(s.buffer)
 	s.encoder.SetPrecision(time.Second)

 	return nil
 }

-func (s *HttpSink) Write(point lp.MutableMetric) error {
+func (s *HttpSink) Write(point lp.CCMetric) error {
 	_, err := s.encoder.Encode(point)
 	return err
 }
--- a/sinks/influxSink.go
+++ b/sinks/influxSink.go
@@ -5,15 +5,14 @@ import (
 	"crypto/tls"
 	"errors"
 	"fmt"
-
+	lp "github.com/ClusterCockpit/cc-metric-collector/internal/ccMetric"
 	influxdb2 "github.com/influxdata/influxdb-client-go/v2"
 	influxdb2Api "github.com/influxdata/influxdb-client-go/v2/api"
-	lp "github.com/influxdata/line-protocol"
 	"log"
 )

 type InfluxSink struct {
-	Sink
+	sink
 	client    influxdb2.Client
 	writeApi  influxdb2Api.WriteAPIBlocking
 	retPolicy string
@@ -39,7 +38,8 @@ func (s *InfluxSink) connect() error {
 	return nil
 }

-func (s *InfluxSink) Init(config SinkConfig) error {
+func (s *InfluxSink) Init(config sinkConfig) error {
+	s.name = "InfluxSink"
 	if len(config.Host) == 0 ||
 		len(config.Port) == 0 ||
 		len(config.Database) == 0 ||
@@ -54,15 +54,21 @@ func (s *InfluxSink) Init(config SinkConfig) error {
 	s.user = config.User
 	s.password = config.Password
 	s.ssl = config.SSL
+	s.meta_as_tags = config.MetaAsTags
 	return s.connect()
 }

-func (s *InfluxSink) Write(point lp.MutableMetric) error {
+func (s *InfluxSink) Write(point lp.CCMetric) error {
 	tags := map[string]string{}
 	fields := map[string]interface{}{}
 	for _, t := range point.TagList() {
 		tags[t.Key] = t.Value
 	}
+	if s.meta_as_tags {
+		for _, m := range point.MetaList() {
+			tags[m.Key] = m.Value
+		}
+	}
 	for _, f := range point.FieldList() {
 		fields[f.Key] = f.Value
 	}
--- a/sinks/metricSink.go
+++ b/sinks/metricSink.go
@@ -2,21 +2,22 @@ package sinks

 import (
 	//	"time"
-	lp "github.com/influxdata/line-protocol"
+	lp "github.com/ClusterCockpit/cc-metric-collector/internal/ccMetric"
 )

-type SinkConfig struct {
-	Host         string `json:"host"`
-	Port         string `json:"port"`
-	Database     string `json:"database"`
-	User         string `json:"user"`
-	Password     string `json:"password"`
-	Organization string `json:"organization"`
+type sinkConfig struct {
 	Type         string `json:"type"`
-	SSL          bool   `json:"ssl"`
+	Host         string `json:"host,omitempty"`
+	Port         string `json:"port,omitempty"`
+	Database     string `json:"database,omitempty"`
+	User         string `json:"user,omitempty"`
+	Password     string `json:"password,omitempty"`
+	Organization string `json:"organization,omitempty"`
+	SSL          bool   `json:"ssl,omitempty"`
+	MetaAsTags   bool   `json:"meta_as_tags,omitempty"`
 }

-type Sink struct {
+type sink struct {
 	host         string
 	port         string
 	user         string
@@ -24,11 +25,18 @@ type Sink struct {
 	database     string
 	organization string
 	ssl          bool
+	meta_as_tags bool
+	name         string
 }

-type SinkFuncs interface {
-	Init(config SinkConfig) error
-	Write(point lp.MutableMetric) error
+type Sink interface {
+	Init(config sinkConfig) error
+	Write(point lp.CCMetric) error
 	Flush() error
 	Close()
+	Name() string
+}
+
+func (s *sink) Name() string {
+	return s.name
 }
--- a/sinks/natsSink.go
+++ b/sinks/natsSink.go
@@ -4,16 +4,17 @@ import (
 	"bytes"
 	"errors"
 	"fmt"
-	lp "github.com/influxdata/line-protocol"
+	lp "github.com/ClusterCockpit/cc-metric-collector/internal/ccMetric"
+	influx "github.com/influxdata/line-protocol"
 	nats "github.com/nats-io/nats.go"
 	"log"
 	"time"
 )

 type NatsSink struct {
-	Sink
+	sink
 	client  *nats.Conn
-	encoder *lp.Encoder
+	encoder *influx.Encoder
 	buffer  *bytes.Buffer
 }

@@ -31,7 +32,8 @@ func (s *NatsSink) connect() error {
 	return nil
 }

-func (s *NatsSink) Init(config SinkConfig) error {
+func (s *NatsSink) Init(config sinkConfig) error {
+	s.name = "NatsSink"
 	if len(config.Host) == 0 ||
 		len(config.Port) == 0 ||
 		len(config.Database) == 0 {
@@ -46,40 +48,31 @@ func (s *NatsSink) Init(config SinkConfig) error {
 	// Setup Influx line protocol
 	s.buffer = &bytes.Buffer{}
 	s.buffer.Grow(1025)
-	s.encoder = lp.NewEncoder(s.buffer)
+	s.encoder = influx.NewEncoder(s.buffer)
 	s.encoder.SetPrecision(time.Second)
 	s.encoder.SetMaxLineBytes(1024)
 	// Setup infos for connection
 	return s.connect()
 }

-func (s *NatsSink) Write(point lp.MutableMetric) error {
+func (s *NatsSink) Write(point lp.CCMetric) error {
 	if s.client != nil {
-		//	    var tags map[string]string
-		//        var fields map[string]interface{}
-		//        for _, t := range point.TagList() {
-		//            tags[t.Key] = t.Value
-		//        }
-		//        for _, f := range point.FieldList() {
-		//            fields[f.Key] = f.Value
-		//        }
-		//		m, err := protocol.New(point.Name(), tags, fields, point.Time())
-		//		if err != nil {
-		//			log.Print(err)
-		//			return err
-		//		}
 		_, err := s.encoder.Encode(point)
 		if err != nil {
 			log.Print(err)
 			return err
 		}
-		s.client.Publish(s.database, s.buffer.Bytes())
-		s.buffer.Reset()
 	}
 	return nil
 }

 func (s *NatsSink) Flush() error {
+	if s.client != nil {
+		if err := s.client.Publish(s.database, s.buffer.Bytes()); err != nil {
+			return err
+		}
+		s.buffer.Reset()
+	}
 	return nil
 }

--- a/sinks/sinkManager.go
+++ b/sinks/sinkManager.go
@@ -0,0 +1,141 @@
+package sinks
+
+import (
+	"encoding/json"
+	"log"
+	"os"
+	"sync"
+
+	lp "github.com/ClusterCockpit/cc-metric-collector/internal/ccMetric"
+)
+
+var AvailableSinks = map[string]Sink{
+	"influxdb": &InfluxSink{},
+	"stdout":   &StdoutSink{},
+	"nats":     &NatsSink{},
+	"http":     &HttpSink{},
+}
+
+type sinkManager struct {
+	input   chan lp.CCMetric
+	outputs []Sink
+	done    chan bool
+	wg      *sync.WaitGroup
+	config  []sinkConfig
+}
+
+type SinkManager interface {
+	Init(wg *sync.WaitGroup, sinkConfigFile string) error
+	AddInput(input chan lp.CCMetric)
+	AddOutput(config json.RawMessage) error
+	Start()
+	Close()
+}
+
+func (sm *sinkManager) Init(wg *sync.WaitGroup, sinkConfigFile string) error {
+	sm.input = nil
+	sm.outputs = make([]Sink, 0)
+	sm.done = make(chan bool)
+	sm.wg = wg
+	sm.config = make([]sinkConfig, 0)
+	if len(sinkConfigFile) > 0 {
+		configFile, err := os.Open(sinkConfigFile)
+		if err != nil {
+			log.Print("[SinkManager] ", err.Error())
+			return err
+		}
+		defer configFile.Close()
+		jsonParser := json.NewDecoder(configFile)
+		var rawConfigs []json.RawMessage
+		err = jsonParser.Decode(&rawConfigs)
+		if err != nil {
+			log.Print("[SinkManager] ", err.Error())
+			return err
+		}
+		for _, raw := range rawConfigs {
+			err = sm.AddOutput(raw)
+			if err != nil {
+				continue
+			}
+		}
+	}
+	return nil
+}
+
+func (sm *sinkManager) Start() {
+	sm.wg.Add(1)
+	batchcount := 20
+	go func() {
+		for {
+		SinkManagerLoop:
+			select {
+			case <-sm.done:
+				for _, s := range sm.outputs {
+					s.Close()
+				}
+				log.Print("[SinkManager] DONE\n")
+				sm.wg.Done()
+				break SinkManagerLoop
+			case p := <-sm.input:
+				log.Print("[SinkManager] WRITE ", p)
+				for _, s := range sm.outputs {
+					s.Write(p)
+				}
+				if batchcount == 0 {
+					log.Print("[SinkManager] FLUSH")
+					for _, s := range sm.outputs {
+						s.Flush()
+					}
+					batchcount = 20
+				}
+				batchcount--
+			default:
+			}
+		}
+		log.Print("[SinkManager] EXIT\n")
+	}()
+	log.Print("[SinkManager] STARTED\n")
+}
+
+func (sm *sinkManager) AddInput(input chan lp.CCMetric) {
+	sm.input = input
+}
+
+func (sm *sinkManager) AddOutput(rawConfig json.RawMessage) error {
+	var err error
+	var config sinkConfig
+	if len(rawConfig) > 3 {
+		err = json.Unmarshal(rawConfig, &config)
+		if err != nil {
+			log.Print("[SinkManager] SKIP ", config.Type, " JSON config error: ", err.Error())
+			return err
+		}
+	}
+	if _, found := AvailableSinks[config.Type]; !found {
+		log.Print("[SinkManager] SKIP ", config.Type, " unknown sink: ", err.Error())
+		return err
+	}
+	s := AvailableSinks[config.Type]
+	err = s.Init(config)
+	if err != nil {
+		log.Print("[SinkManager] SKIP ", s.Name(), " initialization failed: ", err.Error())
+		return err
+	}
+	sm.outputs = append(sm.outputs, s)
+	sm.config = append(sm.config, config)
+	return nil
+}
+
+func (sm *sinkManager) Close() {
+	sm.done <- true
+	log.Print("[SinkManager] CLOSE")
+}
+
+func New(wg *sync.WaitGroup, sinkConfigFile string) (SinkManager, error) {
+	sm := &sinkManager{}
+	err := sm.Init(wg, sinkConfigFile)
+	if err != nil {
+		return nil, err
+	}
+	return sm, err
+}
--- a/sinks/stdoutSink.go
+++ b/sinks/stdoutSink.go
@@ -6,23 +6,30 @@ import (
 	"strings"

 	//	"time"
-	lp "github.com/influxdata/line-protocol"
+	lp "github.com/ClusterCockpit/cc-metric-collector/internal/ccMetric"
 )

 type StdoutSink struct {
-	Sink
+	sink
 }

-func (s *StdoutSink) Init(config SinkConfig) error {
+func (s *StdoutSink) Init(config sinkConfig) error {
+	s.name = "StdoutSink"
+	s.meta_as_tags = config.MetaAsTags
 	return nil
 }

-func (s *StdoutSink) Write(point lp.MutableMetric) error {
+func (s *StdoutSink) Write(point lp.CCMetric) error {
 	var tagsstr []string
 	var fieldstr []string
 	for _, t := range point.TagList() {
 		tagsstr = append(tagsstr, fmt.Sprintf("%s=%s", t.Key, t.Value))
 	}
+	if s.meta_as_tags {
+		for _, m := range point.MetaList() {
+			tagsstr = append(tagsstr, fmt.Sprintf("%s=%s", m.Key, m.Value))
+		}
+	}
 	for _, f := range point.FieldList() {
 		switch f.Value.(type) {
 		case float64: