On nodes with >300 cores, one interval's burst of per-hwthread metrics
overran the fixed 200-slot channels. With blocking sends at every hop,
sink back-pressure propagated to the collectors, the collection round
exceeded the interval, and time.Ticker silently dropped the missed
ticks - whole intervals were skipped without any log message.
- multiChanTicker: deliver ticks non-blockingly and warn when a
consumer misses a tick instead of stalling all consumers; guard the
channel list with a mutex (data race with AddChannel)
- collectorManager: run the collection round detached from the tick
loop, skip-and-warn when a round is still running, log per-collector
and per-round durations at debug level, close serial collectors on
shutdown
- metricRouter: buffer the interval timestamp channel and drain it
before stamping, so metrics never carry the previous interval's
timestamp; warn when the collector input channel is full at tick time
- main: scale the inter-manager channels to max(200, 24*NumCPU),
overridable with the new optional channel_buffer_size config option
- add first unit tests for ticker, collector manager and router
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fix energy and time overflow handling. Send the metric as
"lenovo_node_power" per default. This should avoid clashes by default.
If you want the old behavior, use a router rule to rename the metric.
* Add information about the used infiniband counters
* Change datatype from int64 to uint64
* uint64 subtraction handles wraparound automatically
* Compute total rates by summing up the xmit and recv rates.
This avoids overflows in the raw counters
* Check for cases where the current counter can not be saved as last state
* Use golang variable naming convention (camelCase)
moebiusband
Michael Panzlaff
Thomas Roehl
brinkcoder
Michael Schwarz
Holger Obermaier
Thomas Roehl