Commit Graph

77 Commits

Author SHA1 Message Date
Thomas Gruber
3f76947f54
Merge latest developments into main (#67)
* Update configuration.md

Add an additional receiver to have better alignment of components

* Change default GpfsCollector command to `mmpmon` (#53)

* Set default cmd to 'mmpmon'

* Reuse looked up path

* Cast const to string

* Just download LIKWID to get the headers (#54)

* Just download LIKWID to get the headers

* Remove perl-Data-Dumper from BuildRequires, only required by LIKWID build

* Add HttpReceiver as counterpart to the HttpSink (#49)

* Use GBytes as unit for large memory numbers

* Make maxForward configurable, save old name in meta in rename metrics and make the hostname tag key configurable

* Single release action (#55)

Building all RPMs and releasing in a single workflow

* Makefile target to build binary-only Debian packages (#61)

* Add 'install' and 'DEB' make targets to build binary-only Debian packages

* Add control file for DEB builds

* Use a single line for bash loop in make clean

* Add config options for retry intervals of InfluxDB clients (#59)

* Refactoring of LikwidCollector and metric units (#62)

* Reduce complexity of LikwidCollector and allow metric units

* Add unit to LikwidCollector docu and fix some typos

* Make library path configurable

* Use old metric name in Ganglia if rename has happened in the router (#60)

* Use old metric name if rename has happened in the router

* Also check for Ganglia renames for the oldname

* Derived metrics (#57)

* Add time-based derivatived (e.g. bandwidth) to some collectors

* Add documentation

* Add comments

* Fix: Only compute rates with a valid previous state

* Only compute rates with a valid previous state

* Define const values for net/dev fields

* Set default config values

* Add comments

* Refactor: Consolidate data structures

* Refactor: Consolidate data structures

* Refactor: Avoid struct deep copy

* Refactor: Avoid redundant tag maps

* Refactor: Use int64 type for absolut values

Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>

* Simplified iota usage

* Move unit tag to meta data tags

* Derived metrics (#65)

* Add time-based derivatived (e.g. bandwidth) to some collectors

* Add documentation

* Add comments

* Fix: Only compute rates with a valid previous state

* Only compute rates with a valid previous state

* Define const values for net/dev fields

* Set default config values

* Add comments

* Refactor: Consolidate data structures

* Refactor: Consolidate data structures

* Refactor: Avoid struct deep copy

* Refactor: Avoid redundant tag maps

* Refactor: Use int64 type for absolut values

* Update LustreCollector

Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>

* Meta to tags list and map for sinks (#63)

* Change ccMetric->Influx functions

* Use a meta_as_tags string list in config but create a lookup map afterwards

* Add meta as tag logic to sampleSink

* Fix staticcheck warnings (#66)

Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
2022-03-15 16:41:11 +01:00
Holger Obermaier
33fec95eac Additional comments 2022-02-28 12:16:48 +01:00
Holger Obermaier
a2f9b23e85 Additional comments 2022-02-28 09:39:59 +01:00
Thomas Gruber
f099a311a0
Add sink for Prometheus monitoring system (#46)
* Add sink for Prometheus monitoring system

* Add prometheus sink to README
2022-02-25 14:33:20 +01:00
Thomas Roehl
fe3a8d59b0 Ping InfluxDB server after connecting to recognize faulty connections 2022-02-25 13:51:52 +01:00
Thomas Roehl
bac1f18b1d Add samples for collectors, sinks and receivers 2022-02-25 13:47:19 +01:00
Thomas Gruber
16c03d2aa2
Use Ganglia configuration (#44)
* Copy all metric configurations from original Ganglia code

* Use metric configurations from Ganglia for some metrics

* Format value string also for known metrics
2022-02-24 18:22:20 +01:00
Holger Obermaier
73981527d3 Refactor: Embed Init() into New() function 2022-02-23 14:56:29 +01:00
Thomas Roehl
24e12ccc57 Update sink README and SampleSink 2022-02-22 16:19:46 +01:00
Thomas Roehl
18a226183c Use new sink instances to allow multiple of same sink type 2022-02-22 16:15:25 +01:00
Thomas Roehl
9cfbe10247 Add uint types to GangliaSink and LibgangliaSink 2022-02-22 15:51:08 +01:00
Holger Obermaier
a97c705f4c Do not create link to libganglia.so.
libganglia.so is now loaded during runtime by dlopen
and no longer required during link time
2022-02-21 20:55:14 +01:00
Holger Obermaier
54c693532f Do not create fake libganglia.so.
libganglia.so is now loaded during runtime by dlopen
and no longer required during link time
2022-02-21 17:28:54 +01:00
Thomas Roehl
842395682c Remove -lganglia ldflag and check dl lib 2022-02-21 17:01:13 +01:00
Holger Obermaier
e8adf5b909 Create an empty libganglia.so stub if none exists 2022-02-18 15:44:12 +01:00
Thomas Roehl
d9a81501e5 Use common functions in both Ganglia sinks 2022-02-18 15:05:45 +01:00
Thomas Roehl
e2f78fe1c0 Add linker flag -Wl,--unresolved-symbols=ignore-in-object-files to build without library. Remove build tags 2022-02-18 11:41:15 +01:00
Holger Obermaier
bb9c6be741 Do not create libganglia link if libganglia is not installed 2022-02-17 08:03:55 +01:00
Thomas Roehl
3f8a2cb9b8 Fix default gmond config file path for libgangliaSink 2022-02-17 01:33:38 +01:00
Thomas Roehl
7be6d2a338 Use only readable files when searching for libganglia.so 2022-02-16 23:23:07 +01:00
Thomas Roehl
e5585eaaa0 Add additional lines for build tags 2022-02-16 19:07:00 +01:00
Thomas Gruber
e28c1fb30b
Ganglia sink using libganglia.so directly (#35)
* Add sink directly using libganglia.so

* Remove unneeded confuse header

* add submodule init to build action

* add submodule init to runonce action

* add installation og ganglia to runonce

* add installation of ganglia to runonce

* add installation of ganglia to runonce

* libconfuse not required

* Remove ganglia submodule

* Remove ganglia.h

* Add Makefile to help creating the libganglia.so link

* Fix cgo header

* Rename new Ganglia sink to 'libgangliaSink'

* Add documentation for libgangliaSink

* Extend make buildsystem with find&symlink helper for libgangliaSink

* Add metric renaming function

* Add build tag 'ganglia' and create corresponding files
2022-02-16 18:33:46 +01:00
Holger Obermaier
a3ad9d0cb0 Move toTheSinks out of for loop 2022-02-14 18:53:37 +01:00
Thomas Roehl
247fb23de1 Try to operate on multiple metrics if channels if filled 2022-02-14 18:12:50 +01:00
Holger Obermaier
e4285f02c5 Avoid one append 2022-02-11 17:17:25 +01:00
Thomas Roehl
dc1e4f28a0 Check for Ganglia group explicitly in tags and meta info 2022-02-11 16:10:59 +01:00
Lou
442e512f2d
Automatically flush batched writes in the HTTP sink (#31)
* Add error handling for Sink.Write

* simplify HttpSink config

* HttpSink: dynamically sized batches flushed after timer

* fix panic if sink type does not exist
2022-02-10 13:12:32 +01:00
Holger Obermaier
2aa8c812a6 Add config option flush_interval 2022-02-10 09:43:02 +01:00
Holger Obermaier
acf5db543e Fix: Reset counter 2022-02-09 23:22:54 +01:00
Holger Obermaier
a0e97d216a Move all flush operations to the sinks 2022-02-09 19:47:49 +01:00
Holger Obermaier
1d299be3ea Add comments 2022-02-09 11:08:50 +01:00
Holger Obermaier
b4d7643c25 Add comments 2022-02-09 10:09:03 +01:00
Thomas Roehl
7f78a5baf2 Add timeout options to httpSink 2022-02-08 18:06:07 +01:00
Thomas Roehl
fec3c5981d Fix for gangliaSink 2022-02-08 18:04:08 +01:00
Thomas Roehl
7e4c35e224 Merge branch 'develop' of github.com:ClusterCockpit/cc-metric-collector into develop 2022-02-08 13:46:48 +01:00
Thomas Roehl
8c744617be Remove logging and enable command execution for GangliaSink 2022-02-08 13:45:41 +01:00
Holger Obermaier
377f85111a Refactoring: Use ccmetric function ToLineProtocol() in stdout sink 2022-02-08 13:38:35 +01:00
Holger Obermaier
6d55c376bd Refactoring: Remove all *List() functions from CCMetric 2022-02-08 11:23:19 +01:00
Holger Obermaier
d98b678399 Refactoring: Replace FieldList() -> Fields() 2022-02-08 11:05:07 +01:00
Holger Obermaier
e1a7379c2e Generate influxDB point for data type ccMetric 2022-02-08 09:31:08 +01:00
Holger Obermaier
af051b5e7e Replace FieldList() by Fields() 2022-02-07 22:52:39 +01:00
Holger Obermaier
7b104ebe90 Use cclog.ComponentDebug. Avoid copying point.Fields() 2022-02-07 18:00:17 +01:00
Thomas Gruber
d1e66201a6
Add non-blocking InfluxDB sink (#29)
* Add non-blocking InfluxDB sink

* Add configurable batch size
2022-02-07 16:51:46 +01:00
Thomas Gruber
fdb58b0be2
Sink specific configuration maps (#25)
* Use sink-specific configurations to have more flexibility. Adjust sample sink configuration files

* Add documentation

* Add links to individual sink readmes

* Fix link in README

* HTTPS for HttpSink

* If no CPU die id available, use the socket id instead
2022-02-04 18:12:24 +01:00
Thomas Roehl
02cd21abe2 HTTPS for HttpSink 2022-02-04 12:39:25 +01:00
Thomas Gruber
6ff6cb7219
Change CCMetric's internal data structure (#22)
* package ccmetric rewrite

* Create deep copy in New() to avoid access conflicts

* Renamed TagMap() -> Tags(), MetaMap() -> Meta

Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
2022-02-01 14:54:34 +01:00
Thomas Roehl
011218ab80 Adjust ganglia sink to CCMetric 2022-01-30 15:25:57 +01:00
Thomas Gruber
d915bcc02d
Add sink to add metrics to Ganglia through gmetric (#15) 2022-01-30 15:21:24 +01:00
Holger Obermaier
9e99e47d73 Wait for close of done channel, to ensure manager finished. 2022-01-30 12:08:33 +01:00
Holger Obermaier
4e408f9490 Add documentation 2022-01-28 15:16:58 +01:00