A node agent for measuring, processing and forwarding node level metrics
Go to file
2022-02-21 17:16:07 +01:00
.github Remove CentOS8 from RPMBUILD action. End of Life 2022-02-21 17:16:07 +01:00
collectors Remove doubled import and remove merge artifacts 2022-02-21 14:50:11 +01:00
internal Allow concurrent access to condition map 2022-02-16 14:30:11 +01:00
receivers Use receiver names from config (#34) 2022-02-21 12:45:08 +01:00
scripts Set version in RPM spec file 2022-02-14 15:21:01 +01:00
sinks Remove -lganglia ldflag and check dl lib 2022-02-21 17:01:13 +01:00
.gitignore Initial commit 2021-02-16 16:24:11 +01:00
.gitmodules Ganglia sink using libganglia.so directly (#35) 2022-02-16 18:33:46 +01:00
collectors.json Report maximum and critical temperature 2022-02-18 16:56:41 +01:00
config.json Modularize the whole thing (#16) 2022-01-25 15:37:43 +01:00
go.mod Add gval to go files 2022-01-30 15:13:12 +01:00
go.sum Add gval to go files 2022-01-30 15:13:12 +01:00
LICENSE Initial commit 2021-02-16 16:24:11 +01:00
Makefile Ganglia sink using libganglia.so directly (#35) 2022-02-16 18:33:46 +01:00
metric-collector.go Removed PID file creation, as not required by systemd 2022-02-07 13:54:00 +01:00
README.md Add ganglia build tag to README 2022-02-17 15:06:55 +01:00
receivers.json Use receiver names from config (#34) 2022-02-21 12:45:08 +01:00
router.json Modularize the whole thing (#16) 2022-01-25 15:37:43 +01:00
sinks.json Sink specific configuration maps (#25) 2022-02-04 18:12:24 +01:00

cc-metric-collector

A node agent for measuring, processing and forwarding node level metrics. It is part of the ClusterCockpit ecosystem.

The metric collector sends (and receives) metric in the InfluxDB line protocol as it provides flexibility while providing a separation between tags (like index columns in relational databases) and fields (like data columns).

There is a single timer loop that triggers all collectors serially, collects the collectors' data and sends the metrics to the sink. This is done as all data is submitted with a single time stamp. The sinks currently use mostly blocking APIs.

The receiver runs as a go routine side-by-side with the timer loop and asynchronously forwards received metrics to the sink.

Configuration

Configuration is implemented using a single json document that is distributed over network and may be persisted as file. Supported metrics are documented here.

There is a main configuration file with basic settings that point to the other configuration files for the different components.

{
  "sinks": "sinks.json",
  "collectors" : "collectors.json",
  "receivers" : "receivers.json",
  "router" : "router.json",
  "interval": 10,
  "duration": 1
}

The interval defines how often the metrics should be read and send to the sink. The duration tells collectors how long one measurement has to take. This is important for some collectors, like the likwid collector.

See the component READMEs for their configuration:

Installation

$ git clone git@github.com:ClusterCockpit/cc-metric-collector.git
$ make (downloads LIKWID, builds it as static library with 'direct' accessmode and copies all required files for the collector)
$ go get (requires at least golang 1.16)
$ make tags
Available tags:
ganglia
[...]
$ make # calls go build (-tags ganglia,...) -o cc-metric-collector

ganglia build tag

If you want support for the Ganglia Monitoring System, you have to add -tags ganglia to the build command line. This enables two metric sinks. One is using the command line application gmetric (see ganglia sink), the other one interacts directly with libganglia the main Ganglia library that is commonly installed on each compute node (see libganglia sink). The later one requires configuration before building, so use make instead of go build directly.

Running

$ ./cc-metric-collector --help
Usage of metric-collector:
  -config string
    	Path to configuration file (default "./config.json")
  -log string
    	Path for logfile (default "stderr")
  -once
    	Run all collectors only once

Contributing

The ClusterCockpit ecosystem is designed to be used by different HPC computing centers. Since configurations and setups differ between the centers, the centers likely have to put some work into the cc-metric-collector to gather all desired metrics.

You are free to open an issue to request a collector but we would also be happy about PRs.

Contact