ClusterCockpit/cc-metric-collector
1
0
Fork 0
mirror of https://github.com/ClusterCockpit/cc-metric-collector.git synced 2026-07-26 10:50:37 +02:00
Code Issues Packages Projects Releases Wiki Activity
499 Commits 37 Branches 23 Tags
a3ac8f2eadb9f9777cdaca72d375c7eef8d7fe28
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Thomas GruberandGitHub e13695307f AMD ROCm SMI collector (#77) 
* Add collector for AMD ROCm SMI metrics

* Fix import path

* Fix imports

* Remove Board Number

* store GPU index explicitly

* Remove board number from description
 2022-05-25 15:55:43 +02:00
Thomas Roehl 500685672b Option to use MIG slice name as subtype-id in NvidiaCollector 2022-05-13 15:26:47 +02:00
Thomas Roehl d4c89a4206 Option to use MIG UUID as subtype-id in NvidiaCollector 2022-05-13 14:34:32 +02:00
Thomas GruberandGitHub 826f364772 CC topology module update (#76) 
* Rename CPU to hardware thread, write some comments

* Do renaming in other parts

* Remove CpuList and SocketList function from metricCollector. Available in ccTopology
 2022-05-13 14:28:07 +02:00
Thomas GruberandGitHub 5df550b208 Update NvidiaCollector with new metrics, MIG and NvLink support (#75) 2022-05-13 14:11:55 +02:00
Thomas GruberandGitHub 5c34805918 Collectors in parallel (#74) 
* Provide info to CollectorManager whether the collector can be executed in parallel with others

* Split serial and parallel collectors. Read in parallel first
 2022-05-13 14:10:39 +02:00
Thomas GruberandGitHub 1db5f3b29a Rename cpu type to hwthread (#69) 
* Rename 'cpu' type to 'hwthread' to avoid naming clashes with MetricStore and CC-Webfrontend
 2022-05-13 14:09:45 +02:00
Thomas Roehl 0623691bab Fix for --once option 2022-05-13 13:50:19 +02:00
Thomas Roehl 9886f14d14 Check readability of sensor files in TempCollector 2022-05-13 13:32:54 +02:00
Thomas Roehl 857903be2b Skip disks in DiskstatCollector that have size=0 2022-05-13 13:31:22 +02:00
Thomas GruberandGitHub 80d92d6d28 Units with cc-units (#64) 
* Add option to normalize units with cc-unit

* Add unit conversion to router

* Add option to change unit prefix in the router

* Add to MetricRouter README

* Add order of operations in router to README

* Use second add_tags/del_tags only if metric gets renamed
 2022-05-13 13:30:02 +02:00
Thomas Roehl 8068e59818 Update handling of LIKWID headers. Download only if not already present in the system. Fixes #73 2022-05-13 13:14:47 +02:00
Thomas Roehl 8abedac0fe Use Golang duration parser for 'interval' and 'duration' 
in main config
 2022-05-13 12:33:33 +02:00
Thomas Roehl 38d4e0a730 Merge branch 'develop' of github.com:ClusterCockpit/cc-metric-collector into develop 2022-05-04 11:54:55 +02:00
Thomas Roehl 54d14519ca Skip mount points in DiskstatCollector if statfs() call does not work (bind mounts, ...) 2022-05-04 11:54:34 +02:00
Thomas Roehl 7b098e0b1b Fix for missing metrics in LikwidCollector is hwthread is inactive 2022-04-04 15:16:11 +02:00
Thomas Roehl 70a9530aba Set WriteFailedCallback to get some error message 2022-04-04 11:48:54 +02:00
Thomas Roehl 69f7c19659 InfluxAsyncSink: Add custom flush mechanism 2022-04-04 02:56:23 +02:00
Thomas Roehl ecdb4c1bcf Add debug message when updating interval_timestep 2022-04-04 02:55:44 +02:00
Thomas Roehl 4d5b1adbc8 Fix for interval_timestamp option 2022-04-04 02:26:04 +02:00
Thomas Roehl 28348bd108 InfluxSink: Use batch&flush logic from HttpSink 2022-04-01 18:37:45 +02:00
Thomas Roehl a3b9d8a90b HttpSink: Use sink name in error outputs 2022-04-01 18:36:54 +02:00
Thomas Roehl 7e43e9171e Use default options. Overwrite if anything is configured differently. Use seconds as precision 2022-04-01 17:26:56 +02:00
Thomas Roehl 5d25a7bf12 Add units to InfiniBandCollector 2022-04-01 17:14:26 +02:00
Thomas GruberandGitHub 2a014b6fba Read unit of values from /proc/meminfo (#68) 2022-03-31 11:56:31 +02:00
Thomas Roehl 50479f9325 Move all LIKWID related stuff to late initialization routine 2022-03-24 18:12:23 +01:00
Thomas Roehl e0e91844bc Use late initialization of LIKWID and catch access daemon death. Fixes #70 and fixes #71. 2022-03-24 17:56:51 +01:00
Thomas Roehl 296225f3a8 Always export all metrics in NfsCollectors 2022-03-24 13:50:35 +01:00
Thomas Roehl 622e94ae0e Fix DieList() if system does not support dies. Explicitly set entries in CpuData list 2022-03-22 15:58:10 +01:00
Thomas Roehl c506114480 Add processing order to MetricRouter README and add missing options 2022-03-18 12:29:00 +01:00
Thomas Roehl 657543dded Ensure max_forward is at least 1 2022-03-18 12:28:52 +01:00
Thomas Roehl b66fdd1436 Add missing socket->thread_id map for LikwidCollector 2022-03-16 19:04:39 +01:00
Thomas GruberandGitHub c182d295f4 Fix staticcheck warnings (#66) 2022-03-15 16:38:20 +01:00
Thomas GruberandGitHub 57629a2e0a Meta to tags list and map for sinks (#63) 
* Change ccMetric->Influx functions

* Use a meta_as_tags string list in config but create a lookup map afterwards

* Add meta as tag logic to sampleSink
 2022-03-15 16:16:26 +01:00
Thomas GruberGitHubHolger Obermaier
 		aa1afd745e Derived metrics (#65) 
* Add time-based derivatived (e.g. bandwidth) to some collectors

* Add documentation

* Add comments

* Fix: Only compute rates with a valid previous state

* Only compute rates with a valid previous state

* Define const values for net/dev fields

* Set default config values

* Add comments

* Refactor: Consolidate data structures

* Refactor: Consolidate data structures

* Refactor: Avoid struct deep copy

* Refactor: Avoid redundant tag maps

* Refactor: Use int64 type for absolut values

* Update LustreCollector

Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
 2022-03-15 16:09:47 +01:00
Thomas GruberGitHubHolger Obermaier
 		f6dae7c013 Derived metrics (#57) 
* Add time-based derivatived (e.g. bandwidth) to some collectors

* Add documentation

* Add comments

* Fix: Only compute rates with a valid previous state

* Only compute rates with a valid previous state

* Define const values for net/dev fields

* Set default config values

* Add comments

* Refactor: Consolidate data structures

* Refactor: Consolidate data structures

* Refactor: Avoid struct deep copy

* Refactor: Avoid redundant tag maps

* Refactor: Use int64 type for absolut values

Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
 2022-03-11 13:48:18 +01:00
Thomas GruberandGitHub 1de3dda7be Use old metric name in Ganglia if rename has happened in the router (#60) 
* Use old metric name if rename has happened in the router

* Also check for Ganglia renames for the oldname
 2022-03-11 13:44:32 +01:00
Thomas GruberandGitHub 73f22c1041 Refactoring of LikwidCollector and metric units (#62) 
* Reduce complexity of LikwidCollector and allow metric units

* Add unit to LikwidCollector docu and fix some typos

* Make library path configurable
 2022-03-11 13:43:17 +01:00
Thomas GruberandGitHub c9b8fcdaa7 Add config options for retry intervals of InfluxDB clients (#59) 2022-03-11 13:43:03 +01:00
Thomas Roehl 17f37583fc Use a single line for bash loop in make clean 2022-03-11 13:41:13 +01:00
Thomas GruberandGitHub fb9ea992ea Makefile target to build binary-only Debian packages (#61) 
* Add 'install' and 'DEB' make targets to build binary-only Debian packages

* Add control file for DEB builds
 2022-03-11 13:39:09 +01:00
Thomas GruberandGitHub 21edca5f88 Single release action (#55) 
Building all RPMs and releasing in a single workflow
 2022-03-11 13:37:47 +01:00
Thomas Roehl 3cf2f69a07 Make maxForward configurable, save old name in meta in rename metrics and make the hostname tag key configurable 2022-03-09 11:23:17 +01:00
Thomas Roehl e7f7e68095 Use GBytes as unit for large memory numbers 2022-03-09 11:05:26 +01:00
Thomas GruberandGitHub c0e600269a Add HttpReceiver as counterpart to the HttpSink (#49) 2022-03-05 17:30:55 +01:00
Thomas GruberandGitHub f2486abeab Just download LIKWID to get the headers (#54) 
* Just download LIKWID to get the headers

* Remove perl-Data-Dumper from BuildRequires, only required by LIKWID build
 2022-03-05 17:30:40 +01:00
Thomas GruberandGitHub 21864e0ac4 Change default GpfsCollector command to mmpmon (#53) 
* Set default cmd to 'mmpmon'

* Reuse looked up path

* Cast const to string
 2022-03-05 14:42:04 +01:00
Thomas GruberandGitHub 8c668fcc6f Update configuration.md 
Add an additional receiver to have better alignment of components
 2022-03-04 18:33:57 +01:00
Thomas Roehl 1961edc659 Add documentation to help configuring the CC metric collector 2022-03-04 15:42:25 +01:00
Thomas Roehl 7f62975a68 Set proper user for files 2022-03-04 11:52:48 +01:00
Thomas Roehl 5e64c01c9d Fix name for ClusterCockpit user 2022-03-04 11:51:21 +01:00
Thomas Roehl f1d2828e1d Fix error print in LustreCollector 2022-03-04 11:32:10 +01:00
Thomas Roehl 07af660e50 Merge branch 'main' into develop 2022-03-03 17:33:13 +01:00
Thomas Roehl 31f10b3163 Fix user creation type 2022-03-03 17:31:15 +01:00
Thomas Roehl 9ece27eec6 Use right systemd macro to create the user 2022-03-03 17:26:32 +01:00
Thomas Roehl af4c468ab5 Merge branch 'develop' of github.com:ClusterCockpit/cc-metric-collector into develop 2022-03-03 17:24:43 +01:00
Thomas Roehl b3030a8d44 Use right systemd macro to create the user 2022-03-03 17:24:32 +01:00
Thomas Roehl fdbdb79527 Merge branch 'develop' into main 2022-03-03 13:45:51 +01:00
Thomas Roehl 948c34d74d Add user creation in RPM 2022-03-03 13:43:43 +01:00
Thomas Roehl 60de21c41e Switch access mode of LikwidCollector in config file 2022-03-03 13:03:58 +01:00
Thomas Roehl 276c00442a Add option to LustreCollector to call lctl with sudo 2022-03-03 13:02:00 +01:00
Thomas Roehl c61b8d2877 Set name change in Makefile 2022-03-03 11:03:51 +01:00
Thomas Roehl 6023abd028 Rename main file to match with executable name 2022-03-03 11:02:37 +01:00
Thomas Roehl 0753c81156 Add/Remove clustercockpit user and group in RPM 2022-03-02 15:10:14 +01:00
Thomas Roehl 092e7f6a71 Add section how to temporarly disable LIKWID access to page 2022-03-02 13:54:43 +01:00
Thomas Roehl f7e8b52667 Run RPM build actions only on tag push 2022-03-02 13:21:54 +01:00
Thomas Roehl 02baef8c71 Merge branch 'develop' into main 2022-03-02 10:38:17 +01:00
Thomas Roehl d40163cf8f Update README and receiver-specific pages 2022-02-28 17:26:28 +01:00
Thomas GruberGitHubHolger ObermaierHolger Obermaier
 		d98076c792 Merge current development version into main (#48) 
* DiskstatCollector: cast part_max_used metric to int

* Add uint types to GangliaSink and LibgangliaSink

* Use new sink instances to allow multiple of same sink type

* Update sink README and SampleSink

* Use new receiver instances to allow multiple of same receiver type

* Fix metric scope in likwid configuration script

* Mention likwid config script in LikwidCollector README

* Refactor: Embed Init() into New() function

* Refactor: Embed Init() into New() function

* Fix: MetricReceiver uses uninitialized values, when initialization fails

* Use Ganglia configuration (#44)

* Copy all metric configurations from original Ganglia code

* Use metric configurations from Ganglia for some metrics

* Format value string also for known metrics

* Numa-aware memstat collector (#45)

* Add samples for collectors, sinks and receivers

* Ping InfluxDB server after connecting to recognize faulty connections

* Add sink for Prometheus monitoring system (#46)

* Add sink for Prometheus monitoring system

* Add prometheus sink to README

* Add scraper for Prometheus clients (#47)

Co-authored-by: Holger Obermaier <holgerob@gmx.de>
Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
 2022-02-25 14:49:49 +01:00
Thomas GruberandGitHub a203370aaa Add scraper for Prometheus clients (#47) 2022-02-25 14:46:29 +01:00
Thomas GruberandGitHub f099a311a0 Add sink for Prometheus monitoring system (#46) 
* Add sink for Prometheus monitoring system

* Add prometheus sink to README
 2022-02-25 14:33:20 +01:00
Thomas Roehl fe3a8d59b0 Ping InfluxDB server after connecting to recognize faulty connections 2022-02-25 13:51:52 +01:00
Thomas Roehl bac1f18b1d Add samples for collectors, sinks and receivers 2022-02-25 13:47:19 +01:00
Thomas GruberandGitHub c8bca59de4 Numa-aware memstat collector (#45) 2022-02-24 18:27:05 +01:00
Thomas GruberandGitHub 16c03d2aa2 Use Ganglia configuration (#44) 
* Copy all metric configurations from original Ganglia code

* Use metric configurations from Ganglia for some metrics

* Format value string also for known metrics
 2022-02-24 18:22:20 +01:00
Thomas Roehl d542f32baa Mention likwid config script in LikwidCollector README 2022-02-22 17:46:44 +01:00
Thomas Roehl 6b6566b0aa Fix metric scope in likwid configuration script 2022-02-22 17:46:17 +01:00
Thomas Roehl 3598aed090 Use new receiver instances to allow multiple of same receiver type 2022-02-22 16:33:38 +01:00
Thomas Roehl 24e12ccc57 Update sink README and SampleSink 2022-02-22 16:19:46 +01:00
Thomas Roehl 18a226183c Use new sink instances to allow multiple of same sink type 2022-02-22 16:15:25 +01:00
Thomas Roehl 9cfbe10247 Add uint types to GangliaSink and LibgangliaSink 2022-02-22 15:51:08 +01:00
Thomas Roehl 66275ecf74 DiskstatCollector: cast part_max_used metric to int 2022-02-22 15:50:49 +01:00
Thomas GruberandGitHub b4cc6d54ea Update README.md 2022-02-22 15:10:27 +01:00
Thomas GruberandGitHub 45714fe337 Update README.md 2022-02-22 15:09:12 +01:00
Thomas Roehl 5742721cbd Proper tag name in release 2022-02-21 18:58:13 +01:00
Thomas Roehl 888292dbef Remove ganglia build tag documentation, not needed anymore 2022-02-21 18:53:03 +01:00
Thomas Roehl 1425463a26 Upload RPM not SRPM 2022-02-21 18:49:39 +01:00
Thomas GruberandGitHub 5cd7ac2541 Update rpmbuild.yml 2022-02-21 18:28:07 +01:00
Thomas GruberandGitHub afa831aff8 Update rpmbuild.yml 2022-02-21 18:17:12 +01:00
Thomas Roehl a1328b30e9 Remove CentOS8 from RPMBUILD action. End of Life 2022-02-21 17:16:07 +01:00
Thomas Roehl 842395682c Remove -lganglia ldflag and check dl lib 2022-02-21 17:01:13 +01:00
Thomas Roehl 3ade75490c Add RHEL UBI 8 to rpmbuild action 2022-02-21 15:09:42 +01:00
Thomas Roehl ae64eddcc8 Remove doubled import 2022-02-21 14:50:53 +01:00
Thomas Roehl eed9cd227c Remove doubled import and remove merge artifacts 2022-02-21 14:50:11 +01:00
Thomas Roehl 24a2c9992f Merge branch 'develop' into main 2022-02-21 14:32:24 +01:00
Thomas GruberandGitHub f683f2e6da Dynamically load liblikwid (#40) 
* Check whether LIKWID library is present

* Generalize nan_to_zero option to invalid_to_zero including +Inf,+Inf and NaN

* Remove double error printing and return if measurements do not work
 2022-02-21 13:29:33 +01:00
Thomas GruberandGitHub ea5b3bdbd6 Use receiver names from config (#34) 
* Use common configuration format of 'receiver_name' -> 'receiver_config'

* Adjust receiver configuration files
 2022-02-21 12:45:08 +01:00
Thomas GruberandGitHub 435528fa97 Split diskstat Collector (#38) 
* Split diskstats (free, total space) and iostats (reads, writes, ...

* Add iostat Collector to CollectorManager
 2022-02-21 12:44:26 +01:00
Thomas Roehl d9a81501e5 Use common functions in both Ganglia sinks 2022-02-18 15:05:45 +01:00
Thomas Roehl e2f78fe1c0 Add linker flag -Wl,--unresolved-symbols=ignore-in-object-files to build without library. Remove build tags 2022-02-18 11:41:15 +01:00