Commit Graph

107 Commits

Author SHA1 Message Date
Thomas Roehl
e5585eaaa0 Add additional lines for build tags 2022-02-16 19:07:00 +01:00
Thomas Gruber
e28c1fb30b
Ganglia sink using libganglia.so directly (#35)
* Add sink directly using libganglia.so

* Remove unneeded confuse header

* add submodule init to build action

* add submodule init to runonce action

* add installation og ganglia to runonce

* add installation of ganglia to runonce

* add installation of ganglia to runonce

* libconfuse not required

* Remove ganglia submodule

* Remove ganglia.h

* Add Makefile to help creating the libganglia.so link

* Fix cgo header

* Rename new Ganglia sink to 'libgangliaSink'

* Add documentation for libgangliaSink

* Extend make buildsystem with find&symlink helper for libgangliaSink

* Add metric renaming function

* Add build tag 'ganglia' and create corresponding files
2022-02-16 18:33:46 +01:00
Holger Obermaier
a3ad9d0cb0 Move toTheSinks out of for loop 2022-02-14 18:53:37 +01:00
Thomas Roehl
247fb23de1 Try to operate on multiple metrics if channels if filled 2022-02-14 18:12:50 +01:00
Holger Obermaier
e4285f02c5 Avoid one append 2022-02-11 17:17:25 +01:00
Thomas Roehl
dc1e4f28a0 Check for Ganglia group explicitly in tags and meta info 2022-02-11 16:10:59 +01:00
Lou
442e512f2d
Automatically flush batched writes in the HTTP sink (#31)
* Add error handling for Sink.Write

* simplify HttpSink config

* HttpSink: dynamically sized batches flushed after timer

* fix panic if sink type does not exist
2022-02-10 13:12:32 +01:00
Holger Obermaier
2aa8c812a6 Add config option flush_interval 2022-02-10 09:43:02 +01:00
Holger Obermaier
acf5db543e Fix: Reset counter 2022-02-09 23:22:54 +01:00
Holger Obermaier
a0e97d216a Move all flush operations to the sinks 2022-02-09 19:47:49 +01:00
Holger Obermaier
1d299be3ea Add comments 2022-02-09 11:08:50 +01:00
Holger Obermaier
b4d7643c25 Add comments 2022-02-09 10:09:03 +01:00
Thomas Roehl
7f78a5baf2 Add timeout options to httpSink 2022-02-08 18:06:07 +01:00
Thomas Roehl
fec3c5981d Fix for gangliaSink 2022-02-08 18:04:08 +01:00
Thomas Roehl
7e4c35e224 Merge branch 'develop' of github.com:ClusterCockpit/cc-metric-collector into develop 2022-02-08 13:46:48 +01:00
Thomas Roehl
8c744617be Remove logging and enable command execution for GangliaSink 2022-02-08 13:45:41 +01:00
Holger Obermaier
377f85111a Refactoring: Use ccmetric function ToLineProtocol() in stdout sink 2022-02-08 13:38:35 +01:00
Holger Obermaier
6d55c376bd Refactoring: Remove all *List() functions from CCMetric 2022-02-08 11:23:19 +01:00
Holger Obermaier
d98b678399 Refactoring: Replace FieldList() -> Fields() 2022-02-08 11:05:07 +01:00
Holger Obermaier
e1a7379c2e Generate influxDB point for data type ccMetric 2022-02-08 09:31:08 +01:00
Holger Obermaier
af051b5e7e Replace FieldList() by Fields() 2022-02-07 22:52:39 +01:00
Holger Obermaier
7b104ebe90 Use cclog.ComponentDebug. Avoid copying point.Fields() 2022-02-07 18:00:17 +01:00
Thomas Gruber
d1e66201a6
Add non-blocking InfluxDB sink (#29)
* Add non-blocking InfluxDB sink

* Add configurable batch size
2022-02-07 16:51:46 +01:00
Thomas Gruber
fdb58b0be2
Sink specific configuration maps (#25)
* Use sink-specific configurations to have more flexibility. Adjust sample sink configuration files

* Add documentation

* Add links to individual sink readmes

* Fix link in README

* HTTPS for HttpSink

* If no CPU die id available, use the socket id instead
2022-02-04 18:12:24 +01:00
Thomas Roehl
02cd21abe2 HTTPS for HttpSink 2022-02-04 12:39:25 +01:00
Thomas Gruber
6ff6cb7219
Change CCMetric's internal data structure (#22)
* package ccmetric rewrite

* Create deep copy in New() to avoid access conflicts

* Renamed TagMap() -> Tags(), MetaMap() -> Meta

Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
2022-02-01 14:54:34 +01:00
Thomas Roehl
011218ab80 Adjust ganglia sink to CCMetric 2022-01-30 15:25:57 +01:00
Thomas Gruber
d915bcc02d
Add sink to add metrics to Ganglia through gmetric (#15) 2022-01-30 15:21:24 +01:00
Holger Obermaier
9e99e47d73 Wait for close of done channel, to ensure manager finished. 2022-01-30 12:08:33 +01:00
Holger Obermaier
4e408f9490 Add documentation 2022-01-28 15:16:58 +01:00
Holger Obermaier
b9236dcc31 Handle shutdown sequentially 2022-01-27 17:43:00 +01:00
Thomas Roehl
3fd77e6887 Use non-blocking send at close, use common done function and remove default case 2022-01-26 16:54:51 +01:00
Thomas Roehl
2925ad9f40 Use ccLogger anywhere 2022-01-25 17:43:10 +01:00
Thomas Gruber
200af84c54
Modularize the whole thing (#16)
* Use channels, add a metric router, split up configuration and use extended version of Influx line protocol internally

* Use central timer for collectors and router. Add expressions to router

* Add expression to router config

* Update entry points

* Start with README

* Update README for CCMetric

* Formatting

* Update README.md

* Add README for MultiChanTicker

* Add README for MultiChanTicker

* Update README.md

* Add README to metric router

* Update main README

* Remove SinkEntity type

* Update README for sinks

* Update go files

* Update README for receivers

* Update collectors README

* Update collectors README

* Use seperate page per collector

* Fix for tempstat page

* Add docs for customcmd collector

* Add docs for ipmistat collector

* Add docs for topprocs collector

* Update customCmdMetric.md

* Use seconds when calculating LIKWID metrics

* Add IB metrics ib_recv_pkts and ib_xmit_pkts

* Drop domain part of host name

* Updated to latest stable version of likwid

* Define source code dependencies in Makefile

* Add GPFS / IBM Spectrum Scale collector

* Add vet and staticcheck make targets

* Add vet and staticcheck make targets

* Avoid go vet warning:
struct field tag `json:"..., omitempty"` not compatible with reflect.StructTag.Get: suspicious space in struct tag value
struct field tag `json:"...", omitempty` not compatible with reflect.StructTag.Get: key:"value" pairs not separated by spaces

* Add sample collector to README.md

* Add CPU frequency collector

* Avoid staticcheck warning: redundant return statement

* Avoid staticcheck warning: unnecessary assignment to the blank identifier

* Simplified code

* Add CPUFreqCollectorCpuinfo
a metric collector to measure the current frequency of the CPUs
as obtained from /proc/cpuinfo
Only measure on the first hyperthread

* Add collector for NFS clients

* Move publication of metrics into Flush() for NatsSink

* Update GitHub actions

* Refactoring

* Avoid vet warning: Println arg list ends with redundant newline

* Avoid vet warning struct field commands has json tag but is not exported

* Avoid vet warning: return copies lock value.

* Corrected typo

* Refactoring

* Add go sources in internal/...

* Bad separator in Makefile

* Fix Infiniband collector

Co-authored-by: Holger Obermaier <40787752+ho-ob@users.noreply.github.com>
2022-01-25 15:37:43 +01:00
Holger Obermaier
5dd2af4e8f Avoid staticcheck warning: redundant return statement 2022-01-21 14:35:52 +01:00
Thomas Roehl
325d97df96 Updated contributing section for sinks 2021-11-26 15:56:52 +01:00
Thomas Roehl
32d5a5b98f Update sink README.md 2021-11-26 14:18:59 +01:00
Thomas Roehl
24bf9590ee Update sink README.md 2021-11-26 14:10:22 +01:00
Lou Knauer
485223c590 Add new httpSink
This sink is compatible with the HTTP API of cc-metric-store.
Example config.json section:

```
"sink": {
    "type": "http",
    "host": "localhost",
    "port": "8080",
    "database": "api/write",
    "password": "<JWT>"
},
```

The password/JWT can be omitted.
2021-10-12 13:44:38 +02:00
Lou Knauer
cdc1811576 Add Flush method to sink interface 2021-10-12 13:43:58 +02:00
Thomas Gruber
d7ef32de18
Merge branch 'main' into alternate_storage 2021-10-04 15:49:46 +02:00
Thomas Roehl
b06572c865 Reset buffer in nats sink 2021-10-04 15:46:51 +02:00
Thomas Roehl
558bbaba59 Change storage format 2021-10-04 15:23:43 +02:00
Lou Knauer
1cc9491eca Reset encoder buffer in nats sink 2021-08-20 12:17:46 +02:00
Thomas Roehl
e456497956 Update sink README with SSL for Influx sink 2021-06-30 17:03:56 +02:00
Thomas Roehl
586c6c12ac Add SSL to InfluxDB sink 2021-06-30 16:56:47 +02:00
Thomas Roehl
99f3f3b815 Update READMEs 2021-05-19 01:46:52 +02:00
Thomas Roehl
2e9cec5488 Update README for sinks and rename file 2021-05-18 15:39:02 +02:00
Thomas Roehl
1b9cb8955c Hand over full config to Sink and Receiver 2021-05-18 15:16:10 +02:00
Thomas Roehl
b84dc08cd6 Enable strings as field values in stdout sink 2021-05-12 17:46:16 +02:00
Thomas Roehl
1f0c1b4e76 Update sinks' README 2021-03-29 15:33:32 +02:00
Thomas Roehl
df15fc9889 Update sinks' README 2021-03-29 15:31:18 +02:00
Thomas Roehl
0a70ca1f13 Add NatsSink to send data via the NATS messaging system 2021-03-29 14:14:38 +02:00
Thomas Roehl
bc5d4b288e Add READMEs and Makefile to build and integrate LIKWID 2021-03-27 13:39:43 +01:00
Thomas Roehl
fe87b2c7cb Fix format of StdoutSink 2021-03-26 17:08:00 +01:00
Thomas Roehl
913c3719a9 Add StdoutSink for debugging purposes 2021-03-26 17:03:46 +01:00
Thomas Roehl
f822f00cdc Add sink for InfluxDB (with the original InfluxDB client) 2021-03-26 16:48:09 +01:00