* Add cpu_used (all-cpu_idle) to CpustatCollector
* Update cc-metric-collector.init
* Allow selection of timestamp precision in HttpSink
* Add comment about precision requirement for cc-metric-store
* Fix for API changes in gofish@v0.15.0
* Update requirements to latest version
* Read sensors through redfish
* Update golang toolchain to 1.21
* Remove stray error check
* Update main config in configuration.md
* Update Release action to use golang 1.22 stable release, no golang RPMs anymore
* Update runonce action to use golang 1.22 stable release, no golang RPMs anymore
* Update README.md
Use right JSON type in configuration
* Update sink's README
* Test whether ipmitool or ipmi-sensors can be executed without errors
---------
Co-authored-by: Holger Obermaier <Holger.Obermaier@kit.edu>
Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
* Add cpu_used (all-cpu_idle) to CpustatCollector
* Update to line-protocol/v2
* Update runonce.yml with Golang 1.20
* Update fsnotify in LIKWID Collector
* Use not a pointer to line-protocol.Encoder
* Simplify Makefile
* Use only as many arguments as required
* Allow sum function to handle non float types
* Allow values to be a slice of type float64, float32, int, int64, int32, bool
* Use generic function to simplify code
* Add missing case for type []int32
* Use generic function to compute minimum
* Use generic function to compute maximum
* Use generic function to compute average
* Add error value to sumAnyType
* Use generic function to compute median
* For older versions of go slices is not part of the installation
* Remove old entries from go.sum
* Use simpler sort function
* Compute metrics ib_total and ib_total_pkts
* Add aggregated metrics.
Add missing units
* Update likwidMetric.go
Fixes a potential bug when `fsnotify.NewWatcher()` fails with an error
* Completly avoid memory allocations in infinibandMetric read()
* Fixed initialization: Initalization and measurements should run in the same thread
* Add safe.directory to Release action
* Fix path after installation to /usr/bin after installation
* ioutil.ReadFile is deprecated: As of Go 1.16, this function simply calls os.ReadFile
* Switch to package slices from the golang 1.21 default library
* Read file line by line
* Read file line by line
* Read file line by line
* Use CamelCase
* Use CamelCase
* Fix function getNumaDomain, it always returned 0
* Avoid type conversion by using Atoi
Avoid copying structs by using pointer access
Increase readability with CamelCase variable names
* Add caching
* Cache CpuData
* Cleanup
* Use init function to initalize cache structure to avoid multi threading problems
* Reuse information from /proc/cpuinfo
* Avoid slice cloning. Directly use the cache
* Add DieList
* Add NumaDomainList and SMTList
* Cleanup
* Add comment
* Lookup core ID from /sys/devices/system/cpu, /proc/cpuinfo is not portable
* Lookup all information from /sys/devices/system/cpu, /proc/cpuinfo is not portable
* Correctly handle lists from /sys
* Add Simultaneous Multithreading siblings
* Replace deprecated thread_siblings_list by core_cpus_list
* Reduce number of required slices
* Allow to send total values per core, socket and node
* Send all metrics with same time stamp
calcEventsetMetrics does only computiation, counter measurement is done before
* Input parameters should be float64 when evaluating to float64
* Send all metrics with same time stamp
calcGlobalMetrics does only computiation, counter measurement is done before
* Remove unused variable gmresults
* Add comments
* Updated go packages
* Add build with golang 1.21
* Switch to checkout action version 4
* Switch to setup-go action version 4
* Add workflow_dispatch to allow manual run of workflow
* Add workflow_dispatch to allow manual run of workflow
* Add release build jobs to runonce.yml
* Switch to golang 1.20 for RHEL based distributions
* Use dnf to download golang
* Remove golang versions before 1.20
* Upgrade Ubuntu focal -> jammy
* Pipe golang tar package directly to tar
* Update golang version
* Fix Ubuntu version number
* Add links to ipmi and redfish receivers
* Fix http server addr format
* github.com/influxdata/line-protocol -> github.com/influxdata/line-protocol/v2/lineprotocol
* Corrected spelling
* Add some comments
* github.com/influxdata/line-protocol -> github.com/influxdata/line-protocol/v2/lineprotocol
* Allow other fields not only field "value"
* Add some basic debugging documentation
* Add some basic debugging documentation
* Use a lock for the flush timer
* Add tags in lexical order as required by AddTag()
* Only access meta data, when it gets used as tag
* Use slice to store lexialicly orderd key value pairs
* Increase golang version requirement to 1.20.
* Avoid package cmp to allow builds with golang v1.20
* Fix: Error NVML library not found did crash
cc-metric-collector with "SIGSEGV: segmentation violation"
* Add config option idle_timeout
* Add basic authentication support
* Add basic authentication support
* Avoid unneccessary memory allocations
* Add documentation for send_*_total values
* Use generic package maps to clone maps
* Reuse flush timer
* Add Influx client options
* Reuse ccTopology functionality
* Do not store unused topology information
* Add batch_size config
* Cleanup
* Use stype and stype-id for the NIC in NetstatCollector
* Wait for concurrent flush operations to finish
* Be more verbose in error messages
* Reverted previous changes.
Made the code to complex without much advantages
* Use line protocol encoder
* Go pkg update
* Stop flush timer, when immediatelly flushing
* Fix: Corrected unlock access to batch slice
* Add config option to specify whether to use GZip compression in influx write requests
* Add asynchron send of encoder metrics
* Use DefaultServeMux instead of github.com/gorilla/mux
* Add config option for HTTP keep-alives
* Be more strict, when parsing json
* Add config option for HTTP request timeout and Retry interval
* Allow more then one background send operation
* Fix %sysusers_create_package args (#108)
%sysusers_create_package requires two arguments. See: https://github.com/systemd/systemd/blob/main/src/rpm/macros.systemd.in#L165
* Add nfsiostat to list of collectors
---------
Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
Co-authored-by: Holger Obermaier <holgerob@gmx.de>
Co-authored-by: Obihörnchen <obihoernchende@gmail.com>
* Add cpu_used (all-cpu_idle) to CpustatCollector
* Update to line-protocol/v2
* Update runonce.yml with Golang 1.20
* Update fsnotify in LIKWID Collector
* Use not a pointer to line-protocol.Encoder
* Simplify Makefile
* Use only as many arguments as required
* Allow sum function to handle non float types
* Allow values to be a slice of type float64, float32, int, int64, int32, bool
* Use generic function to simplify code
* Add missing case for type []int32
* Use generic function to compute minimum
* Use generic function to compute maximum
* Use generic function to compute average
* Add error value to sumAnyType
* Use generic function to compute median
* For older versions of go slices is not part of the installation
* Remove old entries from go.sum
* Use simpler sort function
* Compute metrics ib_total and ib_total_pkts
* Add aggregated metrics.
Add missing units
* Update likwidMetric.go
Fixes a potential bug when `fsnotify.NewWatcher()` fails with an error
* Completly avoid memory allocations in infinibandMetric read()
* Fixed initialization: Initalization and measurements should run in the same thread
---------
Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
* Merge develop and main (#99)
* InfiniBandCollector: Scale raw readings from octets to bytes
* Fix clock frequency coming from LikwidCollector and update docs
* Build DEB package for Ubuntu 20.04 for releases
* Fix memstat collector with numa_stats option
* Remove useless prints from MemstatCollector
* Replace ioutils with os and io (#87)
* Use lower case for error strings in RocmSmiCollector
* move maybe-usable-by-other-cc-components to pkg. Fix all files to use the new paths (#88)
* Add collector for monitoring the execution of cc-metric-collector itself (#81)
* Add collector to monitor execution of cc-metric-collector itself
* Register SelfCollector
* Fix import paths for moved packages
* Check if at least one CPU with frequency information was detected
* Correct type: /proc/stats -> /proc/stat
* Update README.md
* Run ipmitool asynchron. Improved error handling.
* Corrected some typos
* Add running average power limit (RAPL) metric collector
* Add running average power limit (RAPL) metric collector
* Do not mess up with the orignal configuration
* * Corrected json config in numastatsMetric.md
* Added some debug output to numastatsMetric.go
* Fixed computing number of physical packages for non continous physical package IDs (e.g. on Ampere Altra Q80-30)
* Fix kernel panic for receiver config with missing receiver type
* Add receiver to gather remote IPMI sensor metrics
* Added config option to add ipmi-sensors command line options
* Add documentaion for IPMI receiver
* Update to latest version of included go modules
* Add go.mod to App dependency
* Try to use common metric tags across hardware vendors
* Add IPMI metric: current
* remove prefix enumeration like 01-...
* Add IPMI receiver example configuration to receivers.json
* Minimal formating changes
* Add hostlist package
* Added tests for hostlist Expand()
* Use package hostlist to expand a host list
* Use package hostlist to expand a host list
* Some servers return "ConsumedPowerWatt":65535 instead of "ConsumedPowerWatt":null
* Updated to latest package versions
* Do not allow unknown fields in JSON configuration file
* Add workflow to customize packages to docs
* NFS I/O Stats Collector (#91)
* Initial version
* Delete values for vanished mount points and comments
* Fix for Likwid collector (#95)
* Run LIKWID in separate thread and check metric type
* Change LIKWID collector documentation to use 'type' instead of 'scope'
* Re-initialize LIKWID after one read is missing due to lock toggle
* Register cc-metric-collector at Zenodo (#93)
* Add initial version of Zenodo project file
* Orcid ID added
* Update .zenodo.json
Co-authored-by: Holger Obermaier <holger.obermaier@kit.edu>
* Update ipmiMetric.go
* Use latest LIKWID version for builds
* Update README.md
* Remove development stuff from Makefile
* Add Requires(pre) to RPM SPEC file
* Use curly brackets in packaging make targets
* Fix for LIKWID collector with separate measurement thread and inotify watcher on the LIKWID lock (#97)
Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
Co-authored-by: Holger Obermaier <Holger.Obermaier@kit.edu>
* Update likwid_perfgroup_to_cc_config.py
* Use customcmd commands if they did not error.
---------
Co-authored-by: Thomas Gruber <Thomas.Roehl@googlemail.com>
Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
Co-authored-by: Holger Obermaier <Holger.Obermaier@kit.edu>
* InfiniBandCollector: Scale raw readings from octets to bytes
* Fix clock frequency coming from LikwidCollector and update docs
* Build DEB package for Ubuntu 20.04 for releases
* Fix memstat collector with numa_stats option
* Remove useless prints from MemstatCollector
* Replace ioutils with os and io (#87)
* Use lower case for error strings in RocmSmiCollector
* move maybe-usable-by-other-cc-components to pkg. Fix all files to use the new paths (#88)
* Add collector for monitoring the execution of cc-metric-collector itself (#81)
* Add collector to monitor execution of cc-metric-collector itself
* Register SelfCollector
* Fix import paths for moved packages
* Check if at least one CPU with frequency information was detected
* Correct type: /proc/stats -> /proc/stat
* Update README.md
* Run ipmitool asynchron. Improved error handling.
* Corrected some typos
* Add running average power limit (RAPL) metric collector
* Add running average power limit (RAPL) metric collector
* Do not mess up with the orignal configuration
* * Corrected json config in numastatsMetric.md
* Added some debug output to numastatsMetric.go
* Fixed computing number of physical packages for non continous physical package IDs (e.g. on Ampere Altra Q80-30)
* Fix kernel panic for receiver config with missing receiver type
* Add receiver to gather remote IPMI sensor metrics
* Added config option to add ipmi-sensors command line options
* Add documentaion for IPMI receiver
* Update to latest version of included go modules
* Add go.mod to App dependency
* Try to use common metric tags across hardware vendors
* Add IPMI metric: current
* remove prefix enumeration like 01-...
* Add IPMI receiver example configuration to receivers.json
* Minimal formating changes
* Add hostlist package
* Added tests for hostlist Expand()
* Use package hostlist to expand a host list
* Use package hostlist to expand a host list
* Some servers return "ConsumedPowerWatt":65535 instead of "ConsumedPowerWatt":null
* Updated to latest package versions
* Do not allow unknown fields in JSON configuration file
* Add workflow to customize packages to docs
* NFS I/O Stats Collector (#91)
* Initial version
* Delete values for vanished mount points and comments
* Fix for Likwid collector (#95)
* Run LIKWID in separate thread and check metric type
* Change LIKWID collector documentation to use 'type' instead of 'scope'
* Re-initialize LIKWID after one read is missing due to lock toggle
* Register cc-metric-collector at Zenodo (#93)
* Add initial version of Zenodo project file
* Orcid ID added
* Update .zenodo.json
Co-authored-by: Holger Obermaier <holger.obermaier@kit.edu>
* Update ipmiMetric.go
Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
Co-authored-by: Holger Obermaier <Holger.Obermaier@kit.edu>
* Run LIKWID in separate thread and check metric type
* Change LIKWID collector documentation to use 'type' instead of 'scope'
* Re-initialize LIKWID after one read is missing due to lock toggle
* InfiniBandCollector: Scale raw readings from octets to bytes
* Fix clock frequency coming from LikwidCollector and update docs
* Build DEB package for Ubuntu 20.04 for releases
* Fix memstat collector with numa_stats option
* Remove useless prints from MemstatCollector
* Replace ioutils with os and io (#87)
* Use lower case for error strings in RocmSmiCollector
* move maybe-usable-by-other-cc-components to pkg. Fix all files to use the new paths (#88)
* Add collector for monitoring the execution of cc-metric-collector itself (#81)
* Add collector to monitor execution of cc-metric-collector itself
* Register SelfCollector
* Fix import paths for moved packages
* cpustatMetric.go: Use derived values instead of absolute values
The values in /proc/stat are absolute counters related to the boot
time of the system. To obtain a utilization of the CPU, the changes
in the counters must be derived according to time. To take only the
absolute values leads to the fact that changes in the utilization,
straight with larger values, do not become visible.
* Add new collector for /proc/schedstat
The `schedstat` collector reads data from /proc/schedstat and calculates
a load value, separated by hwthread. This might be useful to detect bad
cpu pinning on shared nodes etc.
Co-authored-by: Michael Schwarz <post@michael-schwarz.name>
* Add collector for AMD ROCm SMI metrics
* Fix import path
* Fix imports
* Remove Board Number
* store GPU index explicitly
* Remove board number from description
* Rename CPU to hardware thread, write some comments
* Do renaming in other parts
* Remove CpuList and SocketList function from metricCollector. Available in ccTopology
* Provide info to CollectorManager whether the collector can be executed in parallel with others
* Split serial and parallel collectors. Read in parallel first
