5d263addde
struct field tag `json:"..., omitempty"` not compatible with reflect.StructTag.Get: suspicious space in struct tag value |
||
---|---|---|
.. | ||
cpustatMetric.go | ||
customCmdMetric.go | ||
diskstatMetric.go | ||
gpfs.go | ||
infinibandMetric.go | ||
ipmiMetric.go | ||
likwidMetric.go | ||
loadavgMetric.go | ||
lustreMetric.go | ||
Makefile | ||
memstatMetric.go | ||
metricCollector.go | ||
netstatMetric.go | ||
nvidiaMetric.go | ||
README.md | ||
tempMetric.go | ||
topprocsMetric.go |
This folder contains the collectors for the cc-metric-collector.
metricCollector.go
The base class/configuration is located in metricCollector.go
.
Collectors
memstatMetric.go
: Reads/proc/meminfo
to calculate node metrics. It also combines values to the metricmem_used
loadavgMetric.go
: Reads/proc/loadavg
and submits node metrics:netstatMetric.go
: Reads/proc/net/dev
and submits for all network devices as the node metrics.lustreMetric.go
: Reads Lustre's stats files and submits node metrics:infinibandMetric.go
: Reads InfiniBand metrics. It uses theperfquery
command to read the node metrics but can fallback to sysfs counters in caseperfquery
does not work.likwidMetric.go
: Reads hardware performance events using LIKWID. It submits socket and cpu metricscpustatMetric.go
: Read CPU specific values from/proc/stat
topprocsMetric.go
: Reads the TopX processes by their CPU usage. X is configurablenvidiaMetric.go
: Read data about Nvidia GPUs using the NVML librarytempMetric.go
: Read temperature data from/sys/class/hwmon/hwmon*
ipmiMetric.go
: Collect data fromipmitool
or as fallbackipmi-sensors
customCmdMetric.go
: Run commands or read files and submit the output (output has to be in InfluxDB line protocol!)
If any of the collectors cannot be initialized, it is excluded from all further reads. Like if the Lustre stat file is not a valid path, no Lustre specific metrics will be recorded.
Collector configuration
"collectors": [
"tempstat"
],
"collect_config": {
"tempstat": {
"tag_override": {
"hwmon0" : {
"type" : "socket",
"type-id" : "0"
},
"hwmon1" : {
"type" : "socket",
"type-id" : "1"
}
}
}
}
The configuration of the collectors in the main config files consists of two parts: active collectors (collectors
) and collector configuration (collect_config
). At startup, all collectors in the collectors
list is initialized and, if successfully initialized, added to the active collectors for metric retrieval. At initialization the collector-specific configuration from the collect_config
section is handed over. Each collector has own configuration options, check at the collector-specific section.
memstat
"memstat": {
"exclude_metrics": [
"mem_used"
]
}
The memstat
collector reads data from /proc/meminfo
and outputs a handful node metrics. If a metric is not required, it can be excluded from forwarding it to the sink.
Metrics:
mem_total
mem_sreclaimable
mem_slab
mem_free
mem_buffers
mem_cached
mem_available
mem_shared
swap_total
swap_free
mem_used
=mem_total
- (mem_free
+mem_buffers
+mem_cached
)
loadavg
"loadavg": {
"exclude_metrics": [
"proc_run"
]
}
The loadavg
collector reads data from /proc/loadavg
and outputs a handful node metrics. If a metric is not required, it can be excluded from forwarding it to the sink.
Metrics:
load_one
load_five
load_fifteen
proc_run
proc_total
netstat
"netstat": {
"exclude_devices": [
"lo"
]
}
The netstat
collector reads data from /proc/net/dev
and outputs a handful node metrics. If a device is not required, it can be excluded from forwarding it to the sink. Commonly the lo
device should be excluded.
Metrics:
bytes_in
bytes_out
pkts_in
pkts_out
The device name is added as tag device
.
diskstat
"diskstat": {
"exclude_metrics": [
"read_ms"
],
}
The netstat
collector reads data from /proc/net/dev
and outputs a handful node metrics. If a metric is not required, it can be excluded from forwarding it to the sink.
Metrics:
reads
reads_merged
read_sectors
read_ms
writes
writes_merged
writes_sectors
writes_ms
ioops
ioops_ms
ioops_weighted_ms
discards
discards_merged
discards_sectors
discards_ms
flushes
flushes_ms
The device name is added as tag device
.
cpustat
"netstat": {
"exclude_metrics": [
"cpu_idle"
]
}
The cpustat
collector reads data from /proc/stats
and outputs a handful node and hwthread metrics. If a metric is not required, it can be excluded from forwarding it to the sink.
Metrics:
cpu_user
cpu_nice
cpu_system
cpu_idle
cpu_iowait
cpu_irq
cpu_softirq
cpu_steal
cpu_guest
cpu_guest_nice
likwid
"likwid": {
"eventsets": [
{
"events": {
"FIXC1": "ACTUAL_CPU_CLOCK",
"FIXC2": "MAX_CPU_CLOCK",
"PMC0": "RETIRED_INSTRUCTIONS",
"PMC1": "CPU_CLOCKS_UNHALTED",
"PMC2": "RETIRED_SSE_AVX_FLOPS_ALL",
"PMC3": "MERGE",
"DFC0": "DRAM_CHANNEL_0",
"DFC1": "DRAM_CHANNEL_1",
"DFC2": "DRAM_CHANNEL_2",
"DFC3": "DRAM_CHANNEL_3"
},
"metrics": [
{
"name": "ipc",
"calc": "PMC0/PMC1",
"socket_scope": false,
"publish": true
},
{
"name": "flops_any",
"calc": "0.000001*PMC2/time",
"socket_scope": false,
"publish": true
},
{
"name": "clock_mhz",
"calc": "0.000001*(FIXC1/FIXC2)/inverseClock",
"socket_scope": false,
"publish": true
},
{
"name": "mem1",
"calc": "0.000001*(DFC0+DFC1+DFC2+DFC3)*64.0/time",
"socket_scope": true,
"publish": false
}
]
},
{
"events": {
"DFC0": "DRAM_CHANNEL_4",
"DFC1": "DRAM_CHANNEL_5",
"DFC2": "DRAM_CHANNEL_6",
"DFC3": "DRAM_CHANNEL_7",
"PWR0": "RAPL_CORE_ENERGY",
"PWR1": "RAPL_PKG_ENERGY"
},
"metrics": [
{
"name": "pwr_core",
"calc": "PWR0/time",
"socket_scope": false,
"publish": true
},
{
"name": "pwr_pkg",
"calc": "PWR1/time",
"socket_scope": true,
"publish": true
},
{
"name": "mem2",
"calc": "0.000001*(DFC0+DFC1+DFC2+DFC3)*64.0/time",
"socket_scope": true,
"publish": false
}
]
}
],
"globalmetrics": [
{
"name": "mem_bw",
"calc": "mem1+mem2",
"socket_scope": true,
"publish": true
}
]
}
Example config suitable for AMD Zen3
The likwid
collector reads hardware performance counters at a hwthread and socket level. The configuration looks quite complicated but it is basically copy&paste from LIKWID's performance groups. The collector made multiple iterations and tried to use the performance groups but it lacked flexibility. The current way of configuration provides most flexibility.
The logic is as following: There are multiple eventsets, each consisting of a list of counters+events and a list of metrics. If you compare a common performance group with the example setting above, there is not much difference:
EVENTSET -> "events": {
FIXC1 ACTUAL_CPU_CLOCK -> "FIXC1": "ACTUAL_CPU_CLOCK",
FIXC2 MAX_CPU_CLOCK -> "FIXC2": "MAX_CPU_CLOCK",
PMC0 RETIRED_INSTRUCTIONS -> "PMC0" : "RETIRED_INSTRUCTIONS",
PMC1 CPU_CLOCKS_UNHALTED -> "PMC1" : "CPU_CLOCKS_UNHALTED",
PMC2 RETIRED_SSE_AVX_FLOPS_ALL -> "PMC2": "RETIRED_SSE_AVX_FLOPS_ALL",
PMC3 MERGE -> "PMC3": "MERGE",
-> }
The metrics are following the same procedure:
METRICS -> "metrics": [
IPC PMC0/PMC1 -> {
-> "name" : "IPC",
-> "calc" : "PMC0/PMC1",
-> "socket_scope": false,
-> "publish": true
-> }
-> ]
The socket_scope
option tells whether it is submitted per socket or per hwthread. If a metric is only used for internal calculations, you can set publish = false
.
Since some metrics can only be gathered in multiple measurements (like the memory bandwidth on AMD Zen3 chips), configure multiple eventsets like in the example config and use the globalmetrics
section to combine them. Be aware that the combination might be misleading because the "behavior" of a metric changes over time and the multiple measurements might count different computing phases.
Todos
- Exclude devices for
diskstat
collector - Aggreate metrics to higher topology entity (sum hwthread metrics to socket metric, ...). Needs to be configurable
Contributing own collectors
A collector reads data from any source, parses it to metrics and submits these metrics to the metric-collector
. A collector provides three function:
Init(config []byte) error
: Initializes the collector using the given collector-specific config in JSON.Read(duration time.Duration, out *[]lp.MutableMetric) error
: Read, parse and submit data to theout
list. If the collector has to measure anything for some duration, use the provided function argumentduration
.Close()
: Closes down the collector.
It is recommanded to call setup()
in the Init()
function.
Finally, the collector needs to be registered in the metric-collector.go
. There is a list of collectors called Collectors
which is a map (string -> pointer to collector). Add a new entry with a descriptive name and the new collector.
Sample collector
package collectors
import (
"encoding/json"
lp "github.com/influxdata/line-protocol"
"time"
)
// Struct for the collector-specific JSON config
type SampleCollectorConfig struct {
ExcludeMetrics []string `json:"exclude_metrics"`
}
type SampleCollector struct {
MetricCollector
config SampleCollectorConfig
}
func (m *SampleCollector) Init(config []byte) error {
m.name = "SampleCollector"
m.setup()
if len(config) > 0 {
err := json.Unmarshal(config, &m.config)
if err != nil {
return err
}
}
m.init = true
return nil
}
func (m *SampleCollector) Read(interval time.Duration, out *[]lp.MutableMetric) {
if !m.init {
return
}
// tags for the metric, if type != node use proper type and type-id
tags := map[string][string]{"type" : "node"}
// Each metric has exactly one field: value !
value := map[string]interface{}{"value": int(x)}
y, err := lp.New("sample_metric", tags, value, time.Now())
if err == nil {
*out = append(*out, y)
}
}
func (m *SampleCollector) Close() {
m.init = false
return
}