Commit Graph

709 Commits

Author SHA1 Message Date
oscarminus
8a3446a596 cpustatMetric.go: Use derived values instead of absolute values (#83)
* cpustatMetric.go: Use derived values instead of absolute values

  The values in /proc/stat are absolute counters related to the boot
  time of the system. To obtain a utilization of the CPU, the changes
  in the counters must be derived according to time. To take only the
  absolute values leads to the fact that changes in the utilization,
  straight with larger values, do not become visible.

* Add new collector for /proc/schedstat

  The `schedstat` collector reads data from /proc/schedstat and calculates
  a load value, separated by hwthread. This might be useful to detect bad
  cpu pinning on shared nodes etc.

Co-authored-by: Michael Schwarz <post@michael-schwarz.name>
2022-09-07 14:13:06 +02:00
oscarminus
88fabc2e83
cpustatMetric.go: Use derived values instead of absolute values (#83)
* cpustatMetric.go: Use derived values instead of absolute values

  The values in /proc/stat are absolute counters related to the boot
  time of the system. To obtain a utilization of the CPU, the changes
  in the counters must be derived according to time. To take only the
  absolute values leads to the fact that changes in the utilization,
  straight with larger values, do not become visible.

* Add new collector for /proc/schedstat

  The `schedstat` collector reads data from /proc/schedstat and calculates
  a load value, separated by hwthread. This might be useful to detect bad
  cpu pinning on shared nodes etc.

Co-authored-by: Michael Schwarz <post@michael-schwarz.name>
2022-09-07 14:09:29 +02:00
Holger Obermaier
503705d442 Allow multiple hosts to share the same client configuration 2022-08-26 11:55:53 +02:00
Holger Obermaier
7ccbf1ebe2 Allow global configuration for redfish devices username, password and endpoint. 2022-08-25 16:47:44 +02:00
Holger Obermaier
60ef0ed116 Fix for servers, which do not set status.state for thermals or powercontrols 2022-08-17 17:37:24 +02:00
Holger Obermaier
a8beec29cc Skip non existing processor metrics URLs 2022-08-17 15:11:21 +02:00
Holger Obermaier
0dd430e7e9 Refactor redfishReceiver. 2022-08-16 15:14:20 +02:00
Holger Obermaier
f7b39d027b url.JoinPath requires go 1.19. For now stay compatible with go 1.18 2022-08-15 15:25:59 +02:00
Holger Obermaier
eaf8b1941d ioutils is depreceated 2022-08-15 15:25:20 +02:00
Holger Obermaier
62f6e4151a Added readProcessorMetrics to read read thermal an power metrics per CPU / GPU 2022-08-15 15:11:29 +02:00
Holger Obermaier
acd092a977 Add redfish receiver documentation 2022-08-11 15:36:18 +02:00
Holger Obermaier
6eb8e3a1f5 Corrected comments. Added additional check 2022-08-10 17:00:47 +02:00
Holger Obermaier
8ba33568a6 Add reading of fan speeds 2022-08-10 16:24:21 +02:00
Holger Obermaier
2ca0359744 Add support to read thermal metrics 2022-08-10 10:30:59 +02:00
Thomas Roehl
a2f0bc37d4 Add runonce job for Golang 1.19 2022-08-03 17:06:28 +02:00
Thomas Roehl
cfcde9b23b Mark code parts as bash 2022-07-28 16:25:32 +02:00
Thomas Roehl
c7d692e27f Use newlines in install lines for readability 2022-07-28 16:24:21 +02:00
Thomas Roehl
c312093d2b Add --owner and --group to install lines 2022-07-28 16:22:39 +02:00
Thomas Roehl
7438b9d245 Add rules files for DEB package 2022-07-27 18:08:15 +02:00
Thomas Roehl
32bb9c5fc0 Update ccMetric README and FromMetric copy 2022-07-27 18:06:41 +02:00
Thomas Roehl
f5ad45e49f Fix old entries in sample scripts 2022-07-27 17:52:36 +02:00
Thomas Roehl
ea33d45d8e Fix link to docs of NumastatsCollector 2022-07-27 17:50:15 +02:00
Thomas Roehl
251ae8e879 Update link to cc-specifications repo with line protocol 2022-07-27 17:46:27 +02:00
Thomas Roehl
edd33d5810 Add docs to README 2022-07-27 17:45:13 +02:00
Thomas Roehl
88b3fe1e41 Add some documentation about building 2022-07-27 17:38:51 +02:00
Thomas Roehl
96b4a2aec1 Merge branch 'develop' of github.com:ClusterCockpit/cc-metric-collector into develop 2022-07-26 14:59:27 +02:00
Thomas Roehl
4b5c2f4e37 Some introduction to CC Metric Collector and the other components 2022-07-26 14:59:08 +02:00
Holger Obermaier
f818bf4c11 Read durations as string from json config 2022-07-22 17:48:11 +02:00
Holger Obermaier
aedc1be277 Set HTTP timeout for redfish device connections 2022-07-22 12:06:02 +02:00
Holger Obermaier
c75d394e11 Corrected json syntax for interval and duration 2022-07-14 16:07:45 +02:00
Holger Obermaier
bae36473f7 Minimum requirement: golang version >= 1.18 2022-07-14 13:55:48 +02:00
Thomas Gruber
b3c27e0af5
Merge latest development changes (#80)
* Cleanup: Remove unused code

* Use Golang duration parser for 'interval' and 'duration'
 in main config

* Update handling of LIKWID headers. Download only if not already present in the system. Fixes #73

* Units with cc-units (#64)

* Add option to normalize units with cc-unit

* Add unit conversion to router

* Add option to change unit prefix in the router

* Add to MetricRouter README

* Add order of operations in router to README

* Use second add_tags/del_tags only if metric gets renamed

* Skip disks in DiskstatCollector that have size=0

* Check readability of sensor files in TempCollector

* Fix for --once option

* Rename `cpu` type to `hwthread` (#69)

* Rename 'cpu' type to 'hwthread' to avoid naming clashes with MetricStore and CC-Webfrontend

* Collectors in parallel (#74)

* Provide info to CollectorManager whether the collector can be executed in parallel with others

* Split serial and parallel collectors. Read in parallel first

* Update NvidiaCollector with new metrics, MIG and NvLink support (#75)

* CC topology module update (#76)

* Rename CPU to hardware thread, write some comments

* Do renaming in other parts

* Remove CpuList and SocketList function from metricCollector. Available in ccTopology

* Option to use MIG UUID as subtype-id in NvidiaCollector

* Option to use MIG slice name as subtype-id in NvidiaCollector

* MetricRouter: Fix JSON in README

* Fix for Github Action to really use the selected version

* Remove Ganglia installation in runonce Action and add Go 1.18

* Fix daemon options in init script

* Add separate go.mod files to use it with deprecated 1.16

* Minor updates for Makefiles

* fix string comparison

* AMD ROCm SMI collector (#77)

* Add collector for AMD ROCm SMI metrics

* Fix import path

* Fix imports

* Remove Board Number

* store GPU index explicitly

* Remove board number from description

* Use http instead of ftp to download likwid

* Fix serial number in rocmCollector

* Improved http sink (#78)

* automatic flush in NatsSink

* tweak default options of HttpSink

* shorter cirt. section and retries for HttpSink

* fix error handling

* Remove file added by mistake.

* Use http instead of ftp to download likwid

* Fix serial number in rocmCollector

Co-authored-by: Thomas Roehl <thomas.roehl@fau.de>

* Fix: When sending metrics failed the batch size could be exceeded

* Improved dropping of metrics failed to send

* Add memstats and topprocs metric

* Updated to latest modules

* Check that at least one sink is running

* Add drop rate, when send buffer is full

* Allow only one timer at a time

* Use mutex to ensure only on flush timer is running

* Fix for NvidiaCollector when devices are not in MiG mode

* Remove Golang version 1.16 an 1.17 from Action. Latest commits require Golang 1.18

* Use Golang 1.18 in Release action to build RPMs

* Change unit of CpufreqCollector to Hz. That's what the sysfs outputs

* Make wget quiet in Release action to reduce log size

Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
Co-authored-by: Lou <lou.knauer@gmx.de>
2022-07-13 10:09:49 +02:00
Thomas Roehl
09b740b82e Make wget quiet in Release action to reduce log size 2022-07-12 12:37:10 +02:00
Thomas Roehl
2adf9484a3 Redo fix for NvidiaCollector and MiG. Got lost somehow 2022-07-12 12:31:24 +02:00
Thomas Roehl
27f17b88af Merge branch 'develop' of github.com:ClusterCockpit/cc-metric-collector into develop 2022-07-12 11:58:50 +02:00
Thomas Roehl
b2bc7b95d3 Change unit of CpufreqCollector to Hz. That's what the sysfs outputs 2022-07-12 11:58:37 +02:00
Thomas Gruber
f79b7b5e2b
Merge branch 'main' into develop 2022-07-12 11:36:46 +02:00
Thomas Roehl
b16343e5e2 Use Golang 1.18 in Release action to build RPMs 2022-07-12 11:30:27 +02:00
Thomas Roehl
4fa37a58f2 Remove Golang version 1.16 an 1.17 from Action. Latest commits require Golang 1.18 2022-07-11 16:01:45 +02:00
Thomas Roehl
addbfd40a1 Fix for NvidiaCollector when devices are not in MiG mode 2022-07-11 13:05:15 +02:00
Holger Obermaier
04819d9db2 Use mutex to ensure only on flush timer is running 2022-06-24 09:08:20 +02:00
Holger Obermaier
9ccc5a6ca7 Allow only one timer at a time 2022-06-23 21:53:02 +02:00
Holger Obermaier
b7dcbaebcf Add drop rate, when send buffer is full 2022-06-23 18:27:03 +02:00
Holger Obermaier
a3ac8f2ead Check that at least one sink is running 2022-06-23 15:44:02 +02:00
Holger Obermaier
8e7143a20a Updated to latest modules 2022-06-23 11:49:18 +02:00
Holger Obermaier
3a10f7cfdb Add memstats and topprocs metric 2022-06-23 11:44:06 +02:00
Holger Obermaier
0ca6d1a794 Improved dropping of metrics failed to send 2022-06-21 07:59:24 +02:00
Holger Obermaier
580d21d8bb Fix: When sending metrics failed the batch size could be exceeded 2022-06-20 18:06:27 +02:00
Thomas Roehl
31a38bc17d Update release action 2022-06-09 14:36:25 +02:00
Thomas Roehl
dbdec1eab8 Merge branch 'main' of github.com:ClusterCockpit/cc-metric-collector into main 2022-06-09 12:46:47 +02:00