mirror of
https://github.com/ClusterCockpit/cc-metric-collector.git
synced 2026-07-11 04:00:38 +02:00
moebiusband
3d4c464166
On nodes with >300 cores, one interval's burst of per-hwthread metrics overran the fixed 200-slot channels. With blocking sends at every hop, sink back-pressure propagated to the collectors, the collection round exceeded the interval, and time.Ticker silently dropped the missed ticks - whole intervals were skipped without any log message. - multiChanTicker: deliver ticks non-blockingly and warn when a consumer misses a tick instead of stalling all consumers; guard the channel list with a mutex (data race with AddChannel) - collectorManager: run the collection round detached from the tick loop, skip-and-warn when a round is still running, log per-collector and per-round durations at debug level, close serial collectors on shutdown - metricRouter: buffer the interval timestamp channel and drain it before stamping, so metrics never carry the previous interval's timestamp; warn when the collector input channel is full at tick time - main: scale the inter-manager channels to max(200, 24*NumCPU), overridable with the new optional channel_buffer_size config option - add first unit tests for ticker, collector manager and router Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
125 lines
4.6 KiB
Markdown
125 lines
4.6 KiB
Markdown
<!--
|
|
---
|
|
title: cc-metric-collector
|
|
description: Metric collecting node agent
|
|
categories: [cc-metric-collector]
|
|
tags: ['Admin']
|
|
weight: 2
|
|
hugo_path: docs/reference/cc-metric-collector/_index.md
|
|
---
|
|
-->
|
|
|
|
# cc-metric-collector
|
|
|
|
A node agent for measuring, processing and forwarding node level metrics. It is part of the [ClusterCockpit ecosystem](https://clustercockpit.org/docs/overview/).
|
|
|
|
The `cc-metric-collector` sends (and maybe receives) metrics in the [InfluxDB line protocol](https://docs.influxdata.com/influxdb/cloud/reference/syntax/line-protocol/) as it provides flexibility while providing a separation between tags (like index columns in relational databases) and fields (like data columns). The `cc-metric-collector` consists of 4 components: collectors, router, sinks and receivers. The collectors read data from the current system and submit metrics to the router. The router can be configured to manipulate the metrics before forwarding them to the sinks. The receivers are also attached to the router like the collectors but they receive data from external source like other `cc-metric-collector` instances.
|
|
|
|
|
|
[](https://doi.org/10.5281/zenodo.7438287)
|
|
|
|
|
|
# Configuration
|
|
|
|
Configuration is implemented using a single json document that is distributed over network and may be persisted as file.
|
|
Supported metrics are documented [here](https://github.com/ClusterCockpit/cc-specifications/blob/master/interfaces/lineprotocol/README.md).
|
|
|
|
There is a main configuration file with basic settings that point to the other configuration files for the different components.
|
|
|
|
``` json
|
|
{
|
|
"sinks-file": "sinks.json",
|
|
"collectors-file" : "collectors.json",
|
|
"receivers-file" : "receivers.json",
|
|
"router-file" : "router.json",
|
|
"main": {
|
|
"interval": "10s",
|
|
"duration": "1s"
|
|
}
|
|
}
|
|
```
|
|
|
|
The `interval` defines how often the metrics should be read and send to the sink(s). The `duration` tells the collectors how long one measurement has to take. This is important for some collectors, like the `likwid` collector. For more information, see [here](./docs/configuration.md).
|
|
|
|
The optional `channel_buffer_size` sets the capacity of the internal channels between the components. If unset, it defaults to `max(200, 24 * number of CPUs)` so that one interval's burst of per-hwthread metrics fits without back-pressuring the collectors on nodes with many cores.
|
|
|
|
See the component READMEs for their configuration:
|
|
|
|
* [`collectors`](./collectors/README.md)
|
|
* [`sinks`](https://github.com/ClusterCockpit/cc-lib/blob/main/sinks/README.md)
|
|
* [`receivers`](https://github.com/ClusterCockpit/cc-lib/blob/main/receivers/README.md)
|
|
* [`router`](./internal/metricRouter/README.md)
|
|
|
|
# Installation
|
|
|
|
```
|
|
$ git clone git@github.com:ClusterCockpit/cc-metric-collector.git
|
|
$ make (downloads LIKWID, builds it as static library with 'direct' accessmode and copies all required files for the collector)
|
|
$ go get
|
|
$ make
|
|
```
|
|
|
|
For more information, see [here](./docs/building.md).
|
|
|
|
# Running
|
|
|
|
```
|
|
$ ./cc-metric-collector --help
|
|
Usage of ./cc-metric-collector:
|
|
-config string
|
|
Path to configuration file (default "./config.json")
|
|
-log string
|
|
Path for logfile (default "stderr")
|
|
-loglevel string
|
|
Set log level (default "info")
|
|
-once
|
|
Run all collectors only once
|
|
```
|
|
|
|
# Scenarios
|
|
|
|
The metric collector was designed with flexibility in mind, so it can be used in many scenarios. Here are a few:
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
subgraph a ["Cluster A"]
|
|
nodeA[NodeA with CC collector]
|
|
nodeB[NodeB with CC collector]
|
|
nodeC[NodeC with CC collector]
|
|
end
|
|
a --> db[(Database)]
|
|
db <--> ccweb("Webfrontend")
|
|
```
|
|
|
|
``` mermaid
|
|
flowchart TD
|
|
subgraph a [ClusterA]
|
|
direction LR
|
|
nodeA[NodeA with CC collector]
|
|
nodeB[NodeB with CC collector]
|
|
nodeC[NodeC with CC collector]
|
|
end
|
|
subgraph b [ClusterB]
|
|
direction LR
|
|
nodeD[NodeD with CC collector]
|
|
nodeE[NodeE with CC collector]
|
|
nodeF[NodeF with CC collector]
|
|
end
|
|
a --> ccrecv{"CC collector as receiver"}
|
|
b --> ccrecv
|
|
ccrecv --> db[("Database1")]
|
|
ccrecv -.-> db2[("Database2")]
|
|
db <-.-> ccweb("Webfrontend")
|
|
```
|
|
|
|
# Contributing
|
|
|
|
The ClusterCockpit ecosystem is designed to be used by different HPC computing centers. Since configurations and setups differ between the centers, the centers likely have to put some work into `cc-metric-collector` to gather all desired metrics.
|
|
|
|
You are free to open an issue to request a collector but we would also be happy about PRs.
|
|
|
|
# Contact
|
|
|
|
* [Matrix.org ClusterCockpit General chat](https://matrix.to/#/#clustercockpit-dev:matrix.org)
|
|
* [Matrix.org ClusterCockpit Development chat](https://matrix.to/#/#clustercockpit:matrix.org)
|