From 6205680fb80e09c26a3b941496aae97d6f5fc7fd Mon Sep 17 00:00:00 2001 From: Jan Eitzinger Date: Wed, 19 Feb 2025 14:52:15 +0100 Subject: [PATCH 1/3] Initial rewrite messaging spec --- interfaces/lineprotocol/README.md | 62 ++++++++++++++++++++++--------- 1 file changed, 45 insertions(+), 17 deletions(-) diff --git a/interfaces/lineprotocol/README.md b/interfaces/lineprotocol/README.md index df97182..8ff3dbc 100644 --- a/interfaces/lineprotocol/README.md +++ b/interfaces/lineprotocol/README.md @@ -2,25 +2,56 @@ ## Overview -ClusterCockpit uses the [InfluxData line-protocol](https://docs.influxdata.com/influxdb/v2.1/reference/syntax/line-protocol/) for transferring messages between its components. The line-protocol is a text-based representation of a metric/event with a value, time and describing tags. All metrics/events have the following format (if written to `stdout`): +ClusterCockpit uses the +[InfluxData line-protocol](https://docs.influxdata.com/influxdb/v2.1/reference/syntax/line-protocol/) +for transferring messages between its components. The line-protocol is a +text-based representation of a metric/event with a value, time and describing +tags. All metrics/events have the following format (if written to `stdout`): -``` +```txt , ``` -where `` and `` are comma-separated lists of `key=value` +Where `` and `` are comma-separated lists of `key=value` entries. In a mind-model, think about tags as `indices` in the database for faster lookup and the `` as values. **Remark**: In the first iteration, we only sent metrics (number values) but we -had to extend the specification to messages with different meanings. The below -text was changes accordingly. The update is downward-compatible, so for metrics -(number values), nothing changed. +extended the specification to messages with different purposes. The below +text was changed accordingly. The update is backward-compatible, for metrics +(number values), nothing has changed. -## Line-protocol in the ClusterCockpit ecosystem +## Message categories + +There exist the following line line-protocol message flavors: + +* Metric: The field key is `value` +* Event: The field key is `event` +* Log message: The field key is `log` +* Control message: The field key is `control` + +## Messaging + +ClusterCockpit uses the NATS messaging network, with the option to support other +messaging frameworks in the future. To distinguish between different message +types and easily filter the types an application is interested in the following +subject hierarchy tree is used: + +```txt +. | + --- metrics + | + --- events.[job] + | + --- log.[ccb, ccms, ccmc, ccem, ccnc] + | + --- control +``` + +## Metric messages In ClusterCockpit we limit the flexibility of the InfluxData line-protocol -slightly. The idea is to keep the format evaluatable by different components. +slightly. The idea is to keep the format usable by different components. Each message is identifiable by the `measurement` (= metric name), the `hostname`, the `type` and, if required, a `type-id`. @@ -49,15 +80,6 @@ stack above will recognize `stype` (= "sub type") and `stype-id`. So `filesystem=/homes` should be better specified as `stype=filesystem,stype-id=/homes`. -### Mandatory fields per measurement - -* Metric: The field key is always `value` -* Event: The field key is always `event` -* Log message: The field key is always `log` -* Control message: The field key is always `log` - -No other field keys are evaluated by the ClusterCockpit ecosystem. - ### Message types There exist different message types in the ClusterCockpit ecosystem, all @@ -88,6 +110,12 @@ For the whole list, see [job-data schema](../../datastructures/job-data.schema.j #### Events **Identification:** `event="X"` field with `"X"` being a string +The name (measurement) of the event message can further specialize the purpose +(similar to REST endpoints), e.g. `start_job`, and `stop_job` for events of type +job. + +Example start job event: +TBD #### Controls From 917e17d9bf12a12efdbdd7e8a3541049c23a7665 Mon Sep 17 00:00:00 2001 From: Jan Eitzinger Date: Thu, 20 Feb 2025 05:30:46 +0100 Subject: [PATCH 2/3] Continue on lineprotocol ccMessage specs --- interfaces/lineprotocol/README.md | 64 ++++++++++++++++--------------- 1 file changed, 34 insertions(+), 30 deletions(-) diff --git a/interfaces/lineprotocol/README.md b/interfaces/lineprotocol/README.md index 8ff3dbc..0d34791 100644 --- a/interfaces/lineprotocol/README.md +++ b/interfaces/lineprotocol/README.md @@ -16,6 +16,9 @@ Where `` and `` are comma-separated lists of `key=value` entries. In a mind-model, think about tags as `indices` in the database for faster lookup and the `` as values. +We are using the tag set to add metadata information and the field for the +payload. + **Remark**: In the first iteration, we only sent metrics (number values) but we extended the specification to messages with different purposes. The below text was changed accordingly. The update is backward-compatible, for metrics @@ -25,10 +28,11 @@ text was changed accordingly. The update is backward-compatible, for metrics There exist the following line line-protocol message flavors: -* Metric: The field key is `value` -* Event: The field key is `event` -* Log message: The field key is `log` -* Control message: The field key is `control` +- Metric: The field key is `value`, measurement = metric name +- Event: The field key is `event`, Events are actionable informations, measurement = event subtype (job, phases, ?? ), Additional tag function= +- Log message: The field key is `log`. Log messages are purely informational, + measurement = [ccb, ccms, ccmc, ccem, ccnc], Additional tag loglevel +- Control message: The field key is `control`, measurement = knob name (rapl, freq, prefetcher, topology, config), Additional tags: method=[get,put] ## Messaging @@ -45,10 +49,10 @@ subject hierarchy tree is used: | --- log.[ccb, ccms, ccmc, ccem, ccnc] | - --- control + --- control.[get,put] ``` -## Metric messages +## Points generic for all message categories In ClusterCockpit we limit the flexibility of the InfluxData line-protocol slightly. The idea is to keep the format usable by different components. @@ -58,17 +62,17 @@ Each message is identifiable by the `measurement` (= metric name), the ### Mandatory tags per message -* `hostname` -* `type` - * `node` - * `socket` - * `die` - * `memoryDomain` - * `llc` - * `core` - * `hwthread` - * `accelerator` -* `type-id` for further specifying the type like CPU socket or HW Thread identifier +- `hostname` +- `type` + - `node` + - `socket` + - `die` + - `memoryDomain` + - `llc` + - `core` + - `hwthread` + - `accelerator` +- `type-id` for further specifying the type like CPU socket or HW Thread identifier Although no `type-id` is required if `type=node`, it is recommended to send `type=node,type-id=0`. @@ -93,17 +97,17 @@ While the measurements (metric names) can be chosen freely, there is a basic set of measurements which should be present as long as you navigate in the ClusterCockpit ecosystem -* `flops_sp`: Single-precision floating point rate in `Flops/s` -* `flops_dp`: Double-precision floating point rate in `Flops/s` -* `flops_any`: Combined floating point rate in `Flops/s` (often `(flops_dp * 2) + flops_sp`) -* `cpu_load`: The 1m load of the system (see `/proc/loadavg`) -* `mem_used`: The amount of memory used by applications (see `/proc/meminfo`) -* `ipc`: instructions-per-cycle metric -* `mem_bw`: Main memory bandwidth (read and write) in `MByte/s` -* `cpu_power`: Power consumption of the whole CPU package -* `mem_power`: Power consumption of the memory subsystem -* `clock`: CPU clock in `MHz` -* ... +- `flops_sp`: Single-precision floating point rate in `Flops/s` +- `flops_dp`: Double-precision floating point rate in `Flops/s` +- `flops_any`: Combined floating point rate in `Flops/s` (often `(flops_dp * 2) + flops_sp`) +- `cpu_load`: The 1m load of the system (see `/proc/loadavg`) +- `mem_used`: The amount of memory used by applications (see `/proc/meminfo`) +- `ipc`: instructions-per-cycle metric +- `mem_bw`: Main memory bandwidth (read and write) in `MByte/s` +- `cpu_power`: Power consumption of the whole CPU package +- `mem_power`: Power consumption of the memory subsystem +- `clock`: CPU clock in `MHz` +- ... For the whole list, see [job-data schema](../../datastructures/job-data.schema.json) @@ -121,8 +125,8 @@ TBD **Identification:** -* `control="X"` field with `"X"` being a string -* `method` tag is either `GET` or `PUT` +- `control="X"` field with `"X"` being a string +- `method` tag is either `GET` or `PUT` #### Logs From 4594c99395b0a2884b26976659f57b53a98e70be Mon Sep 17 00:00:00 2001 From: Jan Eitzinger Date: Thu, 20 Feb 2025 06:30:00 +0100 Subject: [PATCH 3/3] Initial complete draft of updated line protocol specs --- interfaces/lineprotocol/README.md | 79 +++++++++++++++++++++---------- 1 file changed, 53 insertions(+), 26 deletions(-) diff --git a/interfaces/lineprotocol/README.md b/interfaces/lineprotocol/README.md index 0d34791..77b6cd8 100644 --- a/interfaces/lineprotocol/README.md +++ b/interfaces/lineprotocol/README.md @@ -15,8 +15,9 @@ tags. All metrics/events have the following format (if written to `stdout`): Where `` and `` are comma-separated lists of `key=value` entries. In a mind-model, think about tags as `indices` in the database for faster lookup and the `` as values. +The timestamp is UNIX epoch time in seconds! -We are using the tag set to add metadata information and the field for the +We are using the tag set to add metadata information and one field for the payload. **Remark**: In the first iteration, we only sent metrics (number values) but we @@ -26,15 +27,22 @@ text was changed accordingly. The update is backward-compatible, for metrics ## Message categories -There exist the following line line-protocol message flavors: +There are four line-protocol message flavors: -- Metric: The field key is `value`, measurement = metric name -- Event: The field key is `event`, Events are actionable informations, measurement = event subtype (job, phases, ?? ), Additional tag function= -- Log message: The field key is `log`. Log messages are purely informational, - measurement = [ccb, ccms, ccmc, ccem, ccnc], Additional tag loglevel -- Control message: The field key is `control`, measurement = knob name (rapl, freq, prefetcher, topology, config), Additional tags: method=[get,put] +- **Metric**: The `field` key is `value`, the `measurement` is the metric name +- **Event**: The `field` key is `event`. Events are actionable informations. The +`measurement` is set to an event class (job, slurm, status, phases, ?? ). Additional tag +`function` to indicate the purpose, similar to a REST endpoint (for the job +class this can be start_job and stop_job). +- **Log**: The `field` key is `log`. Log messages are purely informational. + The `measurement` is set to the component identifier [ccb, ccms, ccmc, ccem, +ccnc]. Additional tag `loglevel` to set the log level (debug, info, warn, +error). +- **Control**: The `field` key is `control`, the `measurement` is set to a +control class (rapl, freq, prefetcher, topology, config). Additional tag +`method` with on of [GET,PUT]. -## Messaging +## Messaging subjects ClusterCockpit uses the NATS messaging network, with the option to support other messaging frameworks in the future. To distinguish between different message @@ -45,19 +53,16 @@ subject hierarchy tree is used: . | --- metrics | - --- events.[job] + --- events.[job, slurm] | --- log.[ccb, ccms, ccmc, ccem, ccnc] | - --- control.[get,put] + --- control.[get, put] ``` -## Points generic for all message categories +## Rules valid for all message categories -In ClusterCockpit we limit the flexibility of the InfluxData line-protocol -slightly. The idea is to keep the format usable by different components. - -Each message is identifiable by the `measurement` (= metric name), the +Each message is identifiable by the `measurement`, and the tags `hostname`, the `type` and, if required, a `type-id`. ### Mandatory tags per message @@ -76,7 +81,7 @@ Each message is identifiable by the `measurement` (= metric name), the Although no `type-id` is required if `type=node`, it is recommended to send `type=node,type-id=0`. -#### Optional tags depending on the message +#### Optional tags depending on the message type In some cases, optional tags are required like `filesystem`, `device` or `version`. While you are free to do that, the ClusterCockpit components in the @@ -109,25 +114,47 @@ ClusterCockpit ecosystem - `clock`: CPU clock in `MHz` - ... +FIXME: What about the unit?? For the whole list, see [job-data schema](../../datastructures/job-data.schema.json) +Example: + +```txt +flops_any,hostname=e1208,type=core,type-id=23 value=1203.3 1740027951 +``` + #### Events -**Identification:** `event="X"` field with `"X"` being a string -The name (measurement) of the event message can further specialize the purpose -(similar to REST endpoints), e.g. `start_job`, and `stop_job` for events of type -job. +**Identification:** Field `event="X"` with `"X"` being the payload string. +The name (measurement) of the event message indicates the event +class. The function tag specifies the purpose (similar to REST endpoints), e.g. +`start_job`, and `stop_job` for events of class job. -Example start job event: -TBD +Example: + +```txt +job,hostname=mngmt02,type=node,type-id=0,function=stop_job event={"jobId": 69, "cluster": "ccfront", "stopTime": 1738842306, "jobState": "completed"} 1740027951 +``` #### Controls -**Identification:** +**Identification:** Field `control="X"` with `"X"` being the control request. `measurement` is +set to a control class, the tag `method` is either `GET` or `PUT`. -- `control="X"` field with `"X"` being a string -- `method` tag is either `GET` or `PUT` +Example: + +```txt +rapl,hostname=e1208,type=socket,type-id=2,method=GET control=intel.pkg.energy_status 1740027951 +``` #### Logs -**Identification:** `log="X"` field with `"X"` being a string +**Identification:** `log="X"` field with `"X"` being the log message. The `measurement` is +set to source component id, the tag `loglevel` is one of debug, info, warn, +error. + +Example: + +```txt +ccb,hostname=server01,type=node,type-id=1,loglevel=info log="component: archiver cluster: alex jobId: 232383 - archiving finished" 1740027951 +```