ClusterCockpit/cc-metric-collector
1
0
Fork 0
mirror of https://github.com/ClusterCockpit/cc-metric-collector.git synced 2024-09-20 15:37:27 +02:00
Code Issues Packages Projects Releases Wiki Activity
661 Commits 18 Branches 14 Tags 19 MiB
34436ac261
Commit Graph

257 Commits

Author SHA1 Message Date
Holger Obermaier
fcfb58c31c Use slice element of m.gpus without slice index 2022-02-15 09:23:57 +01:00
Holger Obermaier
5060497abd Cleanup 2022-02-14 22:14:06 +01:00
Holger Obermaier
342f09fabf Cleanup 2022-02-14 11:19:19 +01:00
Holger Obermaier
09b1ea130e Add error handling. Cleanup. 2022-02-14 10:46:05 +01:00
Holger Obermaier
6b12baff6e Use sensor name and sensor label as metric name 2022-02-12 10:13:38 +01:00
Thomas Roehl
bd246bdacf Fix group for netstat collector 2022-02-11 18:18:10 +01:00
Thomas Roehl
23d13b2ceb Fix group for netstat collector 2022-02-11 18:09:39 +01:00
Holger Obermaier
cfc5279958 Move sensor detection to Init() 2022-02-11 17:17:25 +01:00
Thomas Roehl
b15fdf72b9 Exclude metrics and devices in Init() for NvidiaCollector 2022-02-11 14:20:06 +01:00
Holger Obermaier
82138df48e Refactor: Replace readOneLine() by ioutil.ReadFile() 2022-02-10 09:28:06 +01:00
Thomas Gruber
1ea63332d3
Update README.md 2022-02-08 13:49:48 +01:00
Thomas Roehl
7e4c35e224 Merge branch 'develop' of github.com:ClusterCockpit/cc-metric-collector into develop 2022-02-08 13:46:48 +01:00
Thomas Roehl
fcc25f7d30 Add collector documentation 2022-02-08 13:46:44 +01:00
Thomas Roehl
cc86fc00a0 Add missing error check in InfiniBandPerfQueryMetric 2022-02-08 13:46:19 +01:00
Thomas Roehl
9e73dcd437 Fix type tag for numastat 2022-02-08 13:40:27 +01:00
Thomas Roehl
006b9f91f6 Excluding NaN values in Likwid metrics from sending 2022-02-08 13:39:58 +01:00
Thomas Gruber
e1cf682989
Add other collectors to README 2022-02-08 13:22:20 +01:00
Holger Obermaier
4e0782d66b Use FromInfluxMetric() to convert influx to cc metric 2022-02-08 10:58:53 +01:00
Thomas Roehl
a6bec61b1e LikwidCollector: Filter out NaNs or set them to zero if 'nan_to_zero' option is set 2022-02-07 18:35:08 +01:00
Thomas Roehl
7182b339b9 Respect the publish option in the LikwidCollector 2022-02-07 17:41:35 +01:00
Thomas Roehl
d8ab3b0eb0 Use LookPath in IpmiCollector 2022-02-07 15:44:29 +01:00
Thomas Roehl
b19ae7a4db Fix initialization of InfinibandCollector 2022-02-07 15:43:57 +01:00
Thomas Gruber
5263a974d1
Split NfsCollector in Nfs3Collector and Nfs4Collector (#28) 
* Split NfsCollector in Nfs3Collector and Nfs4Collector

* Add documentation
 2022-02-07 15:43:01 +01:00
Thomas Roehl
b7ee125942 Merge branch 'develop' of github.com:ClusterCockpit/cc-metric-collector into develop 2022-02-07 13:47:06 +01:00
Holger Obermaier
ead7117cad Add skip_filesystem configuration 2022-02-07 13:30:42 +01:00
Thomas Roehl
52458ce5a1 Fix for LustreCollector. Check for root user 2022-02-07 13:27:35 +01:00
Holger Obermaier
a534f16685 Add documentation for GPFS metric 2022-02-07 11:37:34 +01:00
Holger Obermaier
25c2ae4910 Avoid int -> int64 conversions 2022-02-07 11:12:03 +01:00
Holger Obermaier
3c10c6b340 Add error handling to Read() 2022-02-07 10:02:38 +01:00
Holger Obermaier
79b25ddbee Add markdown documentation for metric collector ibstat_perfquery 2022-02-07 09:46:19 +01:00
Holger Obermaier
5ac3af895d Moved documentation to markdown file 2022-02-07 09:22:59 +01:00
Holger Obermaier
9ab7a6424b Moved check which metric to skip to Init() 2022-02-04 19:22:42 +01:00
Holger Obermaier
f719f1915c Add error handling 2022-02-04 16:11:56 +01:00
Holger Obermaier
76b69c59b4 Switched to cclog.ComponentError() for error reporting in Read() 2022-02-04 14:42:53 +01:00
Thomas Roehl
66b9a25a88 Prefix metrics from NetstatCollector with 'net' 2022-02-04 12:39:59 +01:00
Thomas Roehl
db02c89683 Update LustreCollector to use lctl. Sysfs version is commented out 2022-02-03 22:05:16 +01:00
Thomas Gruber
92d4a9c2b9
Split MetricRouter and MetricAggregator (#24) 
* Split MetricRouter and MetricAggregator

* Missing change in MetricCache

* Add README for MetricAggregator
 2022-02-03 16:52:55 +01:00
Holger Obermaier
d5ff5b83ce Add NUMA metric collector 2022-02-03 16:19:45 +01:00
Holger Obermaier
a016483012 Add NUMA metric collector. 2022-02-03 15:02:13 +01:00
Thomas Roehl
2806b1e7cc Remove debugging artifacts 2022-02-02 17:14:29 +01:00
Thomas Roehl
e59852be03 Fix LikwidCollector, merge artifact causes problems 2022-02-02 16:55:15 +01:00
Thomas Roehl
6f399d5f08 Add scope guidelines in LikwidCollector page 2022-02-02 16:46:35 +01:00
Thomas Roehl
5bf538bf97 Update LikwidCollector page 2022-02-02 16:40:20 +01:00
Thomas Roehl
ed62e952ce Use MetricAggregator to calculate metrics in LIKWID collector. 2022-02-02 14:52:07 +01:00
Thomas Roehl
e550226416 Use gval in LikwidCollector 2022-02-01 16:01:31 +01:00
Holger Obermaier
9e99e47d73 Wait for close of done channel, to ensure manager finished. 2022-01-30 12:08:33 +01:00
Holger Obermaier
8df58c051f Lower minimum required golang version to 1.16. 2022-01-29 10:04:31 +01:00
Holger Obermaier
4e408f9490 Add documentation 2022-01-28 15:16:58 +01:00
Holger Obermaier
82f5c1c5d0 Minimum requirement go version 1.17 2022-01-28 09:42:19 +01:00
Holger Obermaier
db5b4e4f65 Add type=node to gpf metric tags 2022-01-28 09:14:25 +01:00
Holger Obermaier
aea3e2c6b1 Place wait group Add() and Done() near to each other 2022-01-27 20:45:22 +01:00
Holger Obermaier
b9236dcc31 Handle shutdown sequentially 2022-01-27 17:43:00 +01:00
Holger Obermaier
e1d0aacd1e Moved as much work as possible to Init() 2022-01-27 11:08:27 +01:00
Holger Obermaier
7077452a5d Split InfiniBand metric collector, one using 
/sys filesystem reads and one using perfquery.
 2022-01-26 20:18:47 +01:00
Thomas Roehl
76884c3380 Prefix Nvidia metrics with 'nv_' 2022-01-26 18:45:23 +01:00
Thomas Roehl
86e9b55bc9 Fix for documentation 2022-01-26 18:41:25 +01:00
Thomas Roehl
78834337b0 Fix for documentation 2022-01-26 18:37:59 +01:00
Thomas Roehl
babd7a9af8 Use non-blocking send at close 2022-01-26 16:52:56 +01:00
Holger Obermaier
09b7538479 Avoid labels in collector manager loop 2022-01-26 15:54:49 +01:00
Holger Obermaier
c193b80083 Add documentation 2022-01-26 12:31:04 +01:00
Thomas Roehl
2925ad9f40 Use ccLogger anywhere 2022-01-25 17:43:10 +01:00
Holger Obermaier
b4fde31626 Add documentation 2022-01-25 17:20:20 +01:00
Thomas Roehl
bafc6322e6 Change to own Logger 2022-01-25 16:40:02 +01:00
Thomas Gruber
200af84c54
Modularize the whole thing (#16) 
* Use channels, add a metric router, split up configuration and use extended version of Influx line protocol internally

* Use central timer for collectors and router. Add expressions to router

* Add expression to router config

* Update entry points

* Start with README

* Update README for CCMetric

* Formatting

* Update README.md

* Add README for MultiChanTicker

* Add README for MultiChanTicker

* Update README.md

* Add README to metric router

* Update main README

* Remove SinkEntity type

* Update README for sinks

* Update go files

* Update README for receivers

* Update collectors README

* Update collectors README

* Use seperate page per collector

* Fix for tempstat page

* Add docs for customcmd collector

* Add docs for ipmistat collector

* Add docs for topprocs collector

* Update customCmdMetric.md

* Use seconds when calculating LIKWID metrics

* Add IB metrics ib_recv_pkts and ib_xmit_pkts

* Drop domain part of host name

* Updated to latest stable version of likwid

* Define source code dependencies in Makefile

* Add GPFS / IBM Spectrum Scale collector

* Add vet and staticcheck make targets

* Add vet and staticcheck make targets

* Avoid go vet warning:
struct field tag `json:"..., omitempty"` not compatible with reflect.StructTag.Get: suspicious space in struct tag value
struct field tag `json:"...", omitempty` not compatible with reflect.StructTag.Get: key:"value" pairs not separated by spaces

* Add sample collector to README.md

* Add CPU frequency collector

* Avoid staticcheck warning: redundant return statement

* Avoid staticcheck warning: unnecessary assignment to the blank identifier

* Simplified code

* Add CPUFreqCollectorCpuinfo
a metric collector to measure the current frequency of the CPUs
as obtained from /proc/cpuinfo
Only measure on the first hyperthread

* Add collector for NFS clients

* Move publication of metrics into Flush() for NatsSink

* Update GitHub actions

* Refactoring

* Avoid vet warning: Println arg list ends with redundant newline

* Avoid vet warning struct field commands has json tag but is not exported

* Avoid vet warning: return copies lock value.

* Corrected typo

* Refactoring

* Add go sources in internal/...

* Bad separator in Makefile

* Fix Infiniband collector

Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
 2022-01-25 15:37:43 +01:00
Holger Obermaier
d903fc6daa Avoid vet warning struct field commands has json tag but is not exported 2022-01-25 11:16:46 +01:00
Holger Obermaier
222862af32 Avoid vet warning struct field commands has json tag but is not exported 2022-01-25 11:15:36 +01:00
Holger Obermaier
9f8d3ddbd3 Avoid vet warning: Println arg list ends with redundant newline 2022-01-25 10:34:02 +01:00
Holger Obermaier
df77c3fd60 Avoid vet warning: Println arg list ends with redundant newline 2022-01-25 10:33:20 +01:00
Holger Obermaier
ae6ffd4974 Refactoring 2022-01-25 09:48:22 +01:00
Holger Obermaier
e095e4f202 Refactoring 2022-01-25 09:47:24 +01:00
Holger Obermaier
3d377760b8 Refactoring 2022-01-24 22:04:05 +01:00
Holger Obermaier
be8c92676a Refactoring 2022-01-24 22:03:13 +01:00
Holger Obermaier
9157fdbab2 Fixed topology detection 2022-01-24 20:23:24 +01:00
Holger Obermaier
2026c3acd9 Fixed topology detection 2022-01-24 20:22:08 +01:00
Holger Obermaier
f84f7de05c Add CPUFreqCollectorCpuinfo 
a metric collector to measure the current frequency of the CPUs
as obtained from /proc/cpuinfo
Only measure on the first hyperthread
 2022-01-24 13:12:25 +01:00
Holger Obermaier
8d314ecb19 Add CPUFreqCollectorCpuinfo 
a metric collector to measure the current frequency of the CPUs
as obtained from /proc/cpuinfo
Only measure on the first hyperthread
 2022-01-24 13:10:33 +01:00
Holger Obermaier
bcce471b27 Simplified code 2022-01-24 11:33:04 +01:00
Holger Obermaier
daa7c6bf99 Simplified code 2022-01-24 11:31:45 +01:00
Holger Obermaier
5987901005 Avoid staticcheck warning: unnecessary assignment to the blank identifier 2022-01-21 15:25:13 +01:00
Holger Obermaier
25b9268b24 Avoid staticcheck warning: unnecessary assignment to the blank identifier 2022-01-21 15:20:53 +01:00
Holger Obermaier
72722eff31 Avoid staticcheck warning: redundant return statement 2022-01-21 14:52:45 +01:00
Holger Obermaier
5dd2af4e8f Avoid staticcheck warning: redundant return statement 2022-01-21 14:35:52 +01:00
Holger Obermaier
611ac0fcb0 Add CPU frequency collector 2022-01-21 13:52:27 +01:00
Holger Obermaier
83b784e6f0 Add CPU frequency collector 2022-01-21 09:59:57 +01:00
Holger Obermaier
3a9ea0042c Correct go syntax in README.md 2022-01-20 16:33:21 +01:00
Holger Obermaier
0feb880c3b Correct go syntax in README.md 2022-01-20 16:32:10 +01:00
Holger Obermaier
458f6dec08 Avoid go vet warning: 
struct field tag `json:"..., omitempty"` not compatible with reflect.StructTag.Get: suspicious space in struct tag value
 2022-01-20 12:59:43 +01:00
Holger Obermaier
5d263addde Avoid go vet warning: 
struct field tag `json:"..., omitempty"` not compatible with reflect.StructTag.Get: suspicious space in struct tag value
 2022-01-20 12:38:52 +01:00
Holger Obermaier
f17719113d Updated to latest stable version of likwid 2022-01-19 15:55:48 +01:00
Holger Obermaier
caebca5609 Updated to latest stable version of likwid 2022-01-19 15:55:15 +01:00
Holger Obermaier
82b10b365e Fix to work with golang 1.16 2022-01-19 14:52:37 +01:00
Holger Obermaier
38cba10fb6 Fix to work with golang 1.16 2022-01-19 14:47:59 +01:00
Holger Obermaier
b97c588660 Add GPFS / IBM Spectrum Scale collector 2022-01-19 14:25:58 +01:00
Holger Obermaier
a6cc914b99 Add GPFS / IBM Spectrum Scale collector 2022-01-19 14:25:24 +01:00
Holger Obermaier
bd831060c7 Add IB metrics ib_recv_pkts and ib_xmit_pkts 2022-01-19 10:26:38 +01:00
Holger Obermaier
11e40c6ee3 Add IB metrics ib_recv_pkts and ib_xmit_pkts 2022-01-19 10:15:41 +01:00
Thomas Roehl
830b7de9ea Cast collector measurement duration to seconds. Thanks to KIT 2022-01-06 15:26:51 +01:00
Thomas Roehl
43a8ea683d Cast collector measurement duration to seconds. Thanks to KIT 2022-01-06 15:25:51 +01:00
Thomas Roehl
28163f5d2e Format fixes 2021-11-29 15:32:58 +01:00
Thomas Roehl
6d14e66505 Fix missing colon in netstat collector 2021-11-26 19:02:09 +01:00
Thomas Roehl
3a13a41a15 Catch panic in Nvidia Go Bindings 2021-11-26 19:01:47 +01:00
Thomas Roehl
75f5ac8980 Catch panicing collectors (hello Nvidia Go Bindings) late. 2021-11-26 19:01:31 +01:00
Thomas Roehl
cb8221e73c Read config JSON only if there is any 2021-11-26 18:19:45 +01:00
Thomas Roehl
8d643bb5e8 Set default for topprocs collector to Top2 processes if no config is given 2021-11-26 18:19:26 +01:00
Thomas Roehl
bf43f6d975 Updated README.md for collectors. Fix TODO 2021-11-26 15:58:59 +01:00
Thomas Roehl
734d9e5b8f Updated README.md for collectors. Not finished 2021-11-26 15:57:11 +01:00
Thomas Gruber
3997984714
Temp collector from sysfs hwmon (#8) 
* Add collector for hwmon temperature

* Comment out local tag overrides

* Add temperature collector to README

* Update temperature collector with own config parser
 2021-11-25 18:19:09 +01:00
Thomas Gruber
1e7a75598e
Add collector to read data from ipmitool or ipmi-sensors (fallback) (#9) 
* Add collector to read data from ipmitool or ipmi-sensors

* Update IPMI collector to use own config

* Add 'ipmitool sensor' parser
 2021-11-25 18:15:56 +01:00
Thomas Roehl
59ac123662 Add Makefile, update LIKWID Makefile and fix Github Action 2021-11-25 17:51:34 +01:00
Thomas Roehl
156664d78f Make Infiniband collector more configurable 2021-11-25 16:25:20 +01:00
Thomas Roehl
512fa44a74 Remove unneeded stuff 2021-11-25 16:20:52 +01:00
Thomas Roehl
f6a707ac31 Remove unneeded function and const values 2021-11-25 16:01:16 +01:00
Thomas Roehl
9232dd9732 Formatting 2021-11-25 15:11:39 +01:00
Thomas Roehl
51b8c62d4d Collector-specific configuration. LIKWID collector derives metrics itself, Run once CLI option 2021-11-25 14:04:03 +01:00
Thomas Roehl
7044858c2c Change CPI metric to IPC and inverse measurement 2021-11-02 15:50:18 +01:00
Thomas Roehl
dc4b8d13c2 Add collectors for custom commands and /proc/diskstat. Per default add a hostname tag to all measurements 2021-10-08 13:29:57 +02:00
Thomas Roehl
4688997071 Add two helper functions to collector definition 2021-10-08 13:28:02 +02:00
Thomas Roehl
7f1694adc7 Fix format of infiniband collector 2021-10-08 13:27:36 +02:00
Thomas Roehl
cdc253b569 Return proper error for init() of memstat collector 2021-10-08 13:27:15 +02:00
Thomas Gruber
d7ef32de18
Merge branch 'main' into alternate_storage 2021-10-04 15:49:46 +02:00
Thomas Roehl
486df91a9d Fix format 2021-10-04 15:47:03 +02:00
Thomas Roehl
558bbaba59 Change storage format 2021-10-04 15:23:43 +02:00
Thomas Roehl
34585d88df Skip InfiniBand collector if perfquery does not exist 2021-06-30 14:14:04 +02:00
Thomas Roehl
ac41f02dfc Update READMEs 2021-05-18 15:42:11 +02:00
Thomas Roehl
9a205717f7 Unify collector names 2021-05-18 15:30:11 +02:00
Thomas Roehl
36f34e13c7 Don't use absolute path for 'ps' 2021-05-18 15:14:37 +02:00
Thomas Roehl
cda6ebf884 Properly check error codes in LikwidCollector 2021-05-18 15:14:11 +02:00
Thomas Roehl
6e7babe084 Adding Nvidia NVML collector 2021-05-14 19:22:59 +02:00
Thomas Roehl
6b6e28d4c7 Return error at Init for skip collectors early 2021-05-14 19:21:16 +02:00
Thomas Roehl
9b59429a93 Add some more memory metrics to memstat collector 2021-05-12 18:09:15 +02:00
Thomas Roehl
fd6f19ea07 Add proc_total and proc_run to loadavg collector 2021-05-12 18:08:43 +02:00
Thomas Roehl
14bc29f766 Add collector for the top5 processes in %cpu 2021-05-12 17:47:12 +02:00
Thomas Roehl
e8b2bbdae1 Fix format for cpustat collector 2021-05-12 17:45:41 +02:00
Thomas Roehl
e35a9f6b86 Add Cpustat collector for /proc/stat 2021-05-12 16:57:18 +02:00
Thomas Roehl
1d3a11016e Update README with infos for InfiniBand and Lustre collector and updates to LIKWID Makefile 2021-05-12 14:41:54 +02:00
Thomas Roehl
fc01781ce5 Update Makefile to use central LIKWID installation 2021-05-12 14:28:10 +02:00
Thomas Roehl
eab45102ea Update collectors' README 2021-03-29 15:34:32 +02:00
Thomas Roehl
aa5c05dd6a Update collectors' README 2021-03-29 15:23:55 +02:00
Thomas Roehl
a501aa20be Update collectors' README 2021-03-29 15:21:56 +02:00
Thomas Roehl
162f0f14d6 Fixing flops_any metric in LikwidCollector 2021-03-27 13:50:25 +01:00
Thomas Roehl
bc5d4b288e Add READMEs and Makefile to build and integrate LIKWID 2021-03-27 13:39:43 +01:00
Thomas Roehl
5c5e8b8f2d Remove prebuild LIKWID and add Makefile that prepares LIKWID 2021-03-27 12:44:11 +01:00
Jan Eitzinger
e7d933e60f Reformat. Cosmetic changes in structure 2021-03-26 10:19:54 +01:00
Thomas Roehl
f32ff9d006 Fixing format with gofmt 2021-03-25 17:47:08 +01:00
Thomas Roehl
e9a74a4c9c Add LustreCollector 2021-03-25 16:52:28 +01:00
Thomas Roehl
642314c102 Add InfinibandCollector 2021-03-25 15:55:06 +01:00
Thomas Roehl
5659607993 Add network device to metric names to NetstatCollector 2021-03-25 15:54:17 +01:00
Thomas Roehl
0110547b68 Add load_five and load_fifteen to LoadavgCollector 2021-03-25 15:53:43 +01:00
Thomas Roehl
73fd59537e Add simpler collectors for load, mem_used and network 2021-03-25 14:47:35 +01:00
Thomas Roehl
a6ac0c5373 Add likwid collector 2021-03-25 14:47:10 +01:00