mirror of
https://github.com/ClusterCockpit/cc-metric-collector.git
synced 2026-07-11 12:10:37 +02:00
moebiusband
3d4c464166
On nodes with >300 cores, one interval's burst of per-hwthread metrics overran the fixed 200-slot channels. With blocking sends at every hop, sink back-pressure propagated to the collectors, the collection round exceeded the interval, and time.Ticker silently dropped the missed ticks - whole intervals were skipped without any log message. - multiChanTicker: deliver ticks non-blockingly and warn when a consumer misses a tick instead of stalling all consumers; guard the channel list with a mutex (data race with AddChannel) - collectorManager: run the collection round detached from the tick loop, skip-and-warn when a round is still running, log per-collector and per-round durations at debug level, close serial collectors on shutdown - metricRouter: buffer the interval timestamp channel and drain it before stamping, so metrics never carry the previous interval's timestamp; warn when the collector input channel is full at tick time - main: scale the inter-manager channels to max(200, 24*NumCPU), overridable with the new optional channel_buffer_size config option - add first unit tests for ticker, collector manager and router Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
51 lines
1.5 KiB
Markdown
51 lines
1.5 KiB
Markdown
<!--
|
|
---
|
|
title: Multi-channel Ticker
|
|
description: Timer ticker that sends out the tick to multiple channels
|
|
categories: [cc-metric-collector]
|
|
tags: ['Developer']
|
|
weight: 1
|
|
hugo_path: docs/reference/cc-metric-collector/pkg/multichanticker/_index.md
|
|
---
|
|
-->
|
|
|
|
# MultiChanTicker
|
|
|
|
The idea of this ticker is to multiply the output channels. The original Golang `time.Ticker` provides only a single output channel, so the signal can only be received by a single other class. This ticker allows to add multiple channels which get all notified about the time tick.
|
|
|
|
```golang
|
|
type MultiChanTicker interface {
|
|
Init(duration time.Duration)
|
|
AddChannel(chan time.Time)
|
|
}
|
|
```
|
|
|
|
The MultiChanTicker is created similarly to the common `time.Ticker`:
|
|
|
|
```golang
|
|
NewTicker(duration time.Duration) MultiChanTicker
|
|
```
|
|
|
|
Afterwards, you can add channels:
|
|
|
|
```golang
|
|
t := MultiChanTicker(duration)
|
|
c1 := make(chan time.Time, 1)
|
|
c2 := make(chan time.Time, 1)
|
|
t.AddChannel(c1)
|
|
t.AddChannel(c2)
|
|
|
|
for {
|
|
select {
|
|
case t1 := <- c1:
|
|
log.Print(t1)
|
|
case t2 := <- c2:
|
|
log.Print(t2)
|
|
}
|
|
}
|
|
```
|
|
|
|
The result should be the same `time.Time` output in both channels, notified "simultaneously".
|
|
|
|
Ticks are delivered with a non-blocking send: a consumer that has not yet read the previous tick does not stall the ticker (which would silently drop `time.Ticker` fires for all consumers); instead, the tick for that consumer is skipped and a warning is logged. Register buffered channels (capacity 1) so a consumer that is briefly busy at tick time does not lose the tick.
|