Commit Graph
100 Commits
Author SHA1 Message Date
Thomas GruberandGitHub 16c03d2aa2 Use Ganglia configuration (#44)
* Copy all metric configurations from original Ganglia code

* Use metric configurations from Ganglia for some metrics

* Format value string also for known metrics
2022-02-24 18:22:20 +01:00
Thomas Roehl d542f32baa Mention likwid config script in LikwidCollector README 2022-02-22 17:46:44 +01:00
Thomas Roehl 6b6566b0aa Fix metric scope in likwid configuration script 2022-02-22 17:46:17 +01:00
Thomas Roehl 3598aed090 Use new receiver instances to allow multiple of same receiver type 2022-02-22 16:33:38 +01:00
Thomas Roehl 24e12ccc57 Update sink README and SampleSink 2022-02-22 16:19:46 +01:00
Thomas Roehl 18a226183c Use new sink instances to allow multiple of same sink type 2022-02-22 16:15:25 +01:00
Thomas Roehl 9cfbe10247 Add uint types to GangliaSink and LibgangliaSink 2022-02-22 15:51:08 +01:00
Thomas Roehl 66275ecf74 DiskstatCollector: cast part_max_used metric to int 2022-02-22 15:50:49 +01:00
Thomas GruberandGitHub b4cc6d54ea Update README.md 2022-02-22 15:10:27 +01:00
Thomas GruberandGitHub 45714fe337 Update README.md 2022-02-22 15:09:12 +01:00
Thomas Roehl 5742721cbd Proper tag name in release 2022-02-21 18:58:13 +01:00
Thomas Roehl 888292dbef Remove ganglia build tag documentation, not needed anymore 2022-02-21 18:53:03 +01:00
Thomas Roehl 1425463a26 Upload RPM not SRPM 2022-02-21 18:49:39 +01:00
Thomas GruberandGitHub 5cd7ac2541 Update rpmbuild.yml 2022-02-21 18:28:07 +01:00
Thomas GruberandGitHub afa831aff8 Update rpmbuild.yml 2022-02-21 18:17:12 +01:00
Thomas Roehl a1328b30e9 Remove CentOS8 from RPMBUILD action. End of Life 2022-02-21 17:16:07 +01:00
Thomas Roehl 842395682c Remove -lganglia ldflag and check dl lib 2022-02-21 17:01:13 +01:00
Thomas Roehl 3ade75490c Add RHEL UBI 8 to rpmbuild action 2022-02-21 15:09:42 +01:00
Thomas Roehl eed9cd227c Remove doubled import and remove merge artifacts 2022-02-21 14:50:11 +01:00
Thomas Roehl 24a2c9992f Merge branch 'develop' into main 2022-02-21 14:32:24 +01:00
Thomas GruberandGitHub f683f2e6da Dynamically load liblikwid (#40)
* Check whether LIKWID library is present

* Generalize nan_to_zero option to invalid_to_zero including +Inf,+Inf and NaN

* Remove double error printing and return if measurements do not work
2022-02-21 13:29:33 +01:00
Thomas GruberandGitHub ea5b3bdbd6 Use receiver names from config (#34)
* Use common configuration format of 'receiver_name' -> 'receiver_config'

* Adjust receiver configuration files
2022-02-21 12:45:08 +01:00
Thomas GruberandGitHub 435528fa97 Split diskstat Collector (#38)
* Split diskstats (free, total space) and iostats (reads, writes, ...

* Add iostat Collector to CollectorManager
2022-02-21 12:44:26 +01:00
Thomas Roehl d9a81501e5 Use common functions in both Ganglia sinks 2022-02-18 15:05:45 +01:00
Thomas Roehl e2f78fe1c0 Add linker flag -Wl,--unresolved-symbols=ignore-in-object-files to build without library. Remove build tags 2022-02-18 11:41:15 +01:00
Thomas Roehl 4e8ee59211 Update NetstatCollector to derive bandwidths and use an include list 2022-02-18 02:25:23 +01:00
Thomas GruberandGitHub 0152c0dc1e Update CpustatCollector (#36)
* Update cpustat collector

* Update CpustatCollector to use percentages and add 'num_cpus' metric
2022-02-17 15:46:06 +01:00
Thomas Roehl 873befb41f Add ganglia build tag to README 2022-02-17 15:06:55 +01:00
Thomas Roehl 7be6d2a338 Use only readable files when searching for libganglia.so 2022-02-16 23:23:07 +01:00
Thomas Roehl e5585eaaa0 Add additional lines for build tags 2022-02-16 19:07:00 +01:00
Thomas GruberandGitHub e28c1fb30b Ganglia sink using libganglia.so directly (#35)
* Add sink directly using libganglia.so

* Remove unneeded confuse header

* add submodule init to build action

* add submodule init to runonce action

* add installation og ganglia to runonce

* add installation of ganglia to runonce

* add installation of ganglia to runonce

* libconfuse not required

* Remove ganglia submodule

* Remove ganglia.h

* Add Makefile to help creating the libganglia.so link

* Fix cgo header

* Rename new Ganglia sink to 'libgangliaSink'

* Add documentation for libgangliaSink

* Extend make buildsystem with find&symlink helper for libgangliaSink

* Add metric renaming function

* Add build tag 'ganglia' and create corresponding files
2022-02-16 18:33:46 +01:00
Thomas Roehl 416a94aceb Merge branch 'develop' of github.com:ClusterCockpit/cc-metric-collector into develop 2022-02-14 18:12:59 +01:00
Thomas Roehl 247fb23de1 Try to operate on multiple metrics if channels if filled 2022-02-14 18:12:50 +01:00
Thomas GruberandGitHub 95a58c29e2 Set version in RPM spec file 2022-02-14 15:21:01 +01:00
Thomas GruberandGitHub 56363a6636 Update cc-metric-collector.spec 2022-02-14 15:20:38 +01:00
Thomas Roehl bd246bdacf Fix group for netstat collector 2022-02-11 18:18:10 +01:00
Thomas Roehl 23d13b2ceb Fix group for netstat collector 2022-02-11 18:09:39 +01:00
Thomas Roehl dc1e4f28a0 Check for Ganglia group explicitly in tags and meta info 2022-02-11 16:10:59 +01:00
Thomas Roehl b15fdf72b9 Exclude metrics and devices in Init() for NvidiaCollector 2022-02-11 14:20:06 +01:00
Thomas Roehl 184d60cc58 Locking in MetricCache 2022-02-10 15:21:26 +01:00
Thomas GruberandGitHub 1ea63332d3 Update README.md 2022-02-08 13:49:48 +01:00
Thomas GruberandGitHub e1cf682989 Add other collectors to README 2022-02-08 13:22:20 +01:00
Thomas GruberandGitHub d1e66201a6 Add non-blocking InfluxDB sink (#29)
* Add non-blocking InfluxDB sink

* Add configurable batch size
2022-02-07 16:51:46 +01:00
Thomas GruberandGitHub 5263a974d1 Split NfsCollector in Nfs3Collector and Nfs4Collector (#28)
* Split NfsCollector in Nfs3Collector and Nfs4Collector

* Add documentation
2022-02-07 15:43:01 +01:00
Thomas Roehl bb87046501 Merge branch 'develop' of github.com:ClusterCockpit/cc-metric-collector into develop 2022-02-07 15:42:33 +01:00
Thomas Roehl 0bd638f211 Add script to help create configs for LikwidCollector 2022-02-07 15:42:25 +01:00
Thomas Roehl b7ee125942 Merge branch 'develop' of github.com:ClusterCockpit/cc-metric-collector into develop 2022-02-07 13:47:06 +01:00
Thomas Roehl 52458ce5a1 Fix for LustreCollector. Check for root user 2022-02-07 13:27:35 +01:00
Thomas Roehl c313055570 Merge branch 'develop' of github.com:ClusterCockpit/cc-metric-collector into develop 2022-02-07 13:22:34 +01:00
Thomas Roehl 25bb395f02 Fix for NumaDomain getter in ccTopology 2022-02-07 13:22:26 +01:00
Thomas GruberandGitHub fdb58b0be2 Sink specific configuration maps (#25)
* Use sink-specific configurations to have more flexibility. Adjust sample sink configuration files

* Add documentation

* Add links to individual sink readmes

* Fix link in README

* HTTPS for HttpSink

* If no CPU die id available, use the socket id instead
2022-02-04 18:12:24 +01:00
Thomas Roehl 66b9a25a88 Prefix metrics from NetstatCollector with 'net' 2022-02-04 12:39:59 +01:00
Thomas Roehl 02cd21abe2 HTTPS for HttpSink 2022-02-04 12:39:25 +01:00
Thomas Roehl db02c89683 Update LustreCollector to use lctl. Sysfs version is commented out 2022-02-03 22:05:16 +01:00
Thomas GruberandGitHub 92d4a9c2b9 Split MetricRouter and MetricAggregator (#24)
* Split MetricRouter and MetricAggregator

* Missing change in MetricCache

* Add README for MetricAggregator
2022-02-03 16:52:55 +01:00
Thomas Roehl 2806b1e7cc Remove debugging artifacts 2022-02-02 17:14:29 +01:00
Thomas Roehl e59852be03 Fix LikwidCollector, merge artifact causes problems 2022-02-02 16:55:15 +01:00
Thomas Roehl 6f399d5f08 Add scope guidelines in LikwidCollector page 2022-02-02 16:46:35 +01:00
Thomas Roehl 5bf538bf97 Update LikwidCollector page 2022-02-02 16:40:20 +01:00
Thomas Roehl 1222f7a32f Configuration option to disable MetricCache completely 2022-02-02 15:30:14 +01:00
Thomas Roehl 2c13cecf13 Fix link in MetricRouter README 2022-02-02 14:52:19 +01:00
Thomas Roehl ed62e952ce Use MetricAggregator to calculate metrics in LIKWID collector. 2022-02-02 14:52:07 +01:00
Thomas Roehl 4633c8f58d Merge branch 'likwid_collector_gval_and_fix' into develop 2022-02-01 18:35:03 +01:00
Thomas Roehl af8654d325 Update MetricRouter README 2022-02-01 18:28:20 +01:00
Thomas Roehl a4bd141786 Use the MetricAggregator for all calculations in the MetricRouter 2022-02-01 18:27:59 +01:00
Thomas Roehl 64a12b80bb Add and export SetName() function for CCMetric 2022-02-01 18:27:16 +01:00
Thomas Roehl 8319d3de43 Add some more helper functions to ccTopology 2022-02-01 18:26:54 +01:00
Thomas Roehl e550226416 Use gval in LikwidCollector 2022-02-01 16:01:31 +01:00
6ff6cb7219 Change CCMetric's internal data structure (#22)
* package ccmetric rewrite

* Create deep copy in New() to avoid access conflicts

* Renamed TagMap() -> Tags(), MetaMap() -> Meta

Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
2022-02-01 14:54:34 +01:00
Thomas GruberandGitHub 862630a218 Extend workflow to test Go 1.16 and 1.17 2022-01-31 14:42:19 +01:00
Thomas GruberandGitHub 1f55aa247f Run rpmbuild workflow only for new tags 2022-01-31 13:29:14 +01:00
Thomas Roehl 3e329c3324 Move defer after checking error 2022-01-30 22:05:27 +01:00
Thomas Roehl 011218ab80 Adjust ganglia sink to CCMetric 2022-01-30 15:25:57 +01:00
Thomas GruberandGitHub d915bcc02d Add sink to add metrics to Ganglia through gmetric (#15) 2022-01-30 15:21:24 +01:00
Thomas Roehl df31df149b Fix for missing math.MaxInt in go 1.16 2022-01-30 15:16:46 +01:00
Thomas Roehl 70ebd2f36d Add gval to go files 2022-01-30 15:13:12 +01:00
Thomas Roehl 18c5d0eb34 Add example interval aggregation to MetricRouter config for CI 2022-01-30 15:04:31 +01:00
Thomas GruberandGitHub cf810b1c0c Add Cache and Aggregator to MetricRouter (#21)
* Add Cache and Aggregator to MetricRouter

* Close done channel in MetricCache
2022-01-30 15:03:21 +01:00
Thomas GruberandGitHub 11844d9d5d Add common topology module for MetricCollectors and MetricRouter (#20) 2022-01-30 14:59:26 +01:00
Thomas GruberandGitHub 6abbc5f77e Fix Github Actions (#18)
* Fix config for Github Actions

* Fix paths

* Add CentOS Latest and AlmaLinux 8.5 to RPM action

* Fix ID

* Reduce min Go version to 1.16 and use time.Unix in gpfsMetric
2022-01-30 14:54:36 +01:00
Thomas Roehl e4a2927b96 Merge branch 'develop' of github.com:ClusterCockpit/cc-metric-collector into develop 2022-01-30 14:31:19 +01:00
Thomas Roehl d3f5611541 Add functions to get the fields of a CCMetric and export some more CCMetric functions 2022-01-30 14:30:06 +01:00
Thomas Roehl 4541e50bea Minor fixes in ccLogger 2022-01-30 14:29:25 +01:00
Thomas Roehl 76884c3380 Prefix Nvidia metrics with 'nv_' 2022-01-26 18:45:23 +01:00
Thomas Roehl 86e9b55bc9 Fix for documentation 2022-01-26 18:41:25 +01:00
Thomas Roehl 78834337b0 Fix for documentation 2022-01-26 18:37:59 +01:00
200af84c54 Modularize the whole thing (#16)
* Use channels, add a metric router, split up configuration and use extended version of Influx line protocol internally

* Use central timer for collectors and router. Add expressions to router

* Add expression to router config

* Update entry points

* Start with README

* Update README for CCMetric

* Formatting

* Update README.md

* Add README for MultiChanTicker

* Add README for MultiChanTicker

* Update README.md

* Add README to metric router

* Update main README

* Remove SinkEntity type

* Update README for sinks

* Update go files

* Update README for receivers

* Update collectors README

* Update collectors README

* Use seperate page per collector

* Fix for tempstat page

* Add docs for customcmd collector

* Add docs for ipmistat collector

* Add docs for topprocs collector

* Update customCmdMetric.md

* Use seconds when calculating LIKWID metrics

* Add IB metrics ib_recv_pkts and ib_xmit_pkts

* Drop domain part of host name

* Updated to latest stable version of likwid

* Define source code dependencies in Makefile

* Add GPFS / IBM Spectrum Scale collector

* Add vet and staticcheck make targets

* Add vet and staticcheck make targets

* Avoid go vet warning:
struct field tag `json:"..., omitempty"` not compatible with reflect.StructTag.Get: suspicious space in struct tag value
struct field tag `json:"...", omitempty` not compatible with reflect.StructTag.Get: key:"value" pairs not separated by spaces

* Add sample collector to README.md

* Add CPU frequency collector

* Avoid staticcheck warning: redundant return statement

* Avoid staticcheck warning: unnecessary assignment to the blank identifier

* Simplified code

* Add CPUFreqCollectorCpuinfo
a metric collector to measure the current frequency of the CPUs
as obtained from /proc/cpuinfo
Only measure on the first hyperthread

* Add collector for NFS clients

* Move publication of metrics into Flush() for NatsSink

* Update GitHub actions

* Refactoring

* Avoid vet warning: Println arg list ends with redundant newline

* Avoid vet warning struct field commands has json tag but is not exported

* Avoid vet warning: return copies lock value.

* Corrected typo

* Refactoring

* Add go sources in internal/...

* Bad separator in Makefile

* Fix Infiniband collector

Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
2022-01-25 15:37:43 +01:00
Thomas GruberandGitHub 15cf16b46f Add Github Action to build RPM (#14)
* Update SPEC file

* Add clean target to Makefile

* add rpmbuild action

* Add missing build dependency

* Fix package name of golang for CentOS8

* Change outputs

* Fix file list and build requires

* Fix for file list

* Also upload SRPM
2021-11-29 16:04:50 +01:00
Thomas Roehl 6d14e66505 Fix missing colon in netstat collector 2021-11-26 19:02:09 +01:00
Thomas Roehl 3a13a41a15 Catch panic in Nvidia Go Bindings 2021-11-26 19:01:47 +01:00
Thomas Roehl 75f5ac8980 Catch panicing collectors (hello Nvidia Go Bindings) late. 2021-11-26 19:01:31 +01:00
Thomas Roehl 7a7ce0c1b3 Catch panicing collectors (hello Nvidia Go Bindings) late. 2021-11-26 19:01:12 +01:00
Thomas Roehl cb8221e73c Read config JSON only if there is any 2021-11-26 18:19:45 +01:00
Thomas Roehl 8d643bb5e8 Set default for topprocs collector to Top2 processes if no config is given 2021-11-26 18:19:26 +01:00
Thomas Roehl f5b7da5046 Use configuration with all supported collectors for CI. Fixes #13 2021-11-26 18:15:04 +01:00
Thomas Roehl bf43f6d975 Updated README.md for collectors. Fix TODO 2021-11-26 15:58:59 +01:00
Thomas Roehl 734d9e5b8f Updated README.md for collectors. Not finished 2021-11-26 15:57:11 +01:00
Thomas Roehl 325d97df96 Updated contributing section for sinks 2021-11-26 15:56:52 +01:00
Thomas Roehl 09e15c9949 Update receiver README.md 2021-11-26 14:19:07 +01:00
Thomas Roehl 32d5a5b98f Update sink README.md 2021-11-26 14:18:59 +01:00