Backend for ClusterCockpit Monitoring Framework. GitHub Mirror
Go to file
Christoph Kluge adc1d94e3f subcluster in metricSelect, add infobox to systems
- Default: Show Clusters next to Metrics
- New: If Cluster filter activem show subclusters instead (reactive)
- add note to analysis view
- systems view now with info boxes if metric is removed for subcluster
2023-03-31 17:18:16 +02:00
.github/workflows Update test.yml 2022-03-16 12:06:56 +01:00
api Add scope to Metric List. Add numberOfNodes. 2023-03-24 15:10:23 +01:00
cmd/cc-backend Swagger definition URL now based on config.json Addr field 2022-09-27 10:33:36 +02:00
configs Fix issue for a non gpu system 2022-10-18 07:32:34 +02:00
docs Add validate option. Backport startDemo 2023-03-22 07:15:41 +01:00
init Add docs for deployment 2022-09-19 17:37:04 +02:00
internal subcluster in metricSelect, add infobox to systems 2023-03-31 17:18:16 +02:00
pkg Remove comments 2023-03-30 15:23:31 +02:00
test Disable test 2023-03-29 14:37:42 +02:00
tools Update job.go 2023-03-29 06:47:19 +02:00
web subcluster in metricSelect, add infobox to systems 2023-03-31 17:18:16 +02:00
.gitignore Update .gitignore 2023-03-27 15:36:57 +02:00
go.mod GraphQL remove toplevel thresholds and add Unit 2023-03-10 12:13:40 +01:00
go.sum GraphQL remove toplevel thresholds and add Unit 2023-03-10 12:13:40 +01:00
gqlgen.yml GraphQL remove toplevel thresholds and add Unit 2023-03-10 12:13:40 +01:00
LICENSE Add copyright and license header. Update license year 2022-07-29 06:29:21 +02:00
Makefile Add info output tp Makefile 2022-09-26 13:51:03 +02:00
README.md Update README.md 2022-09-27 07:46:18 +02:00
startDemo.sh Add validate option. Backport startDemo 2023-03-22 07:15:41 +01:00
tools.go Reconfigure Swagger integration. Regenerate API docs 2022-09-16 06:09:55 +02:00

ClusterCockpit REST and GraphQL API backend

Build

This is a Golang backend implementation for a REST and GraphQL API according to the ClusterCockpit specifications. It also includes a web interface for ClusterCockpit. This implementation replaces the previous PHP Symfony based ClusterCockpit web-interface. Here is a discussion of the reasons why we switched from PHP Symfony to a Golang based solution.

Overview

This is a golang web backend for the ClusterCockpit job-specific performance monitoring framework. It provides a REST API for integrating ClusterCockpit with a HPC cluster batch system and external analysis scripts. Data exchange between the web frontend and backend is based on a GraphQL API. The web frontend is also served by the backend using Svelte components. Layout and styling is based on Bootstrap 5 using Bootstrap Icons. The backend uses SQLite 3 as relational SQL database by default. It can optionally use a MySQL/MariaDB database server. Finished batch jobs are stored in a file-based job archive following this specification. The backend supports authentication using local accounts or an external LDAP directory. Authorization for APIs is implemented using JWT tokens created with public/private key encryption.

You find more detailed information here:

  • ./configs/README.md: Infos about configuration and setup of cc-backend.
  • ./init/README.md: Infos on how to setup cc-backend as systemd service on Linux.
  • ./tools/README.md: Infos on the JWT authorizatin token workflows in ClusterCockpit.

Demo Setup

We provide a shell skript that downloads demo data and automatically builds and starts cc-backend. You need wget, go, and yarn in your path to start the demo. The demo will download 32MB of data (223MB on disk).

git clone git@github.com:ClusterCockpit/cc-backend.git
cd ./cc-backend
./startDemo.sh

You can access the web interface at http://localhost:8080. Credentials for login: demo:AdminDev. Please note that some views do not work without a metric backend (e.g., the Systems and Status view).

Howto Build and Run

There is a Makefile to automate the build of cc-backend. The Makefile supports the following targets:

  • $ make: Initialize var directory and build svelte frontend and backend binary. Please note that there is no proper prerequesite handling. Any change of frontend source files will trigger a complete rebuild.
  • $ make clean: Clean go build cache and remove binary
  • $ make test: Run the tests that are also run in the GitHub workflow setup.

A common workflow to setup cc-backend fron scratch is:

git clone git@github.com:ClusterCockpit/cc-backend.git

# Build binary
cd ./cc-backend/
make

# EDIT THE .env FILE BEFORE YOU DEPLOY (Change the secrets)!
# If authentication is disabled, it can be empty.
cp configs/env-template.txt  .env
vim ./.env

cp configs/config.json ./
vim ./config.json

#Optional: Link an existing job archive:
ln -s <your-existing-job-archive> ./var/job-archive

# This will first initialize the job.db database by traversing all
# `meta.json` files in the job-archive and add a new user. `--no-server` will cause the
# executable to stop once it has done that instead of starting a server.
./cc-backend --init-db --add-user <your-username>:admin:<your-password>

# Start a HTTP server (HTTPS can be enabled, the default port is 8080).
# The --dev flag enables GraphQL Playground (http://localhost:8080/playground) and Swagger UI (http://localhost:8080/swagger).
./cc-backend --server  --dev

# Show other options:
./cc-backend --help

Run as systemd daemon

In order to run this program as a daemon, cc-backend ships with an example systemd setup.

Configuration and Setup

cc-backend can be used as a local web-interface for an existing job archive or as a general web-interface server for a live ClusterCockpit Monitoring framework.

Create your job-archive according to this specification. At least one cluster with a valid cluster.json file is required. Having no jobs in the job-archive at all is fine.

Configuration

A config file in the JSON format has to be provided using --config to override the defaults. By default, if there is a config.json file in the current directory of the cc-backend process, it will be loaded even without the --config flag. You find documentation of all supported configuration and command line options here.

Development

In case the REST or GraphQL API is changed the according code generators have to be used.

Update GraphQL schema

This project uses gqlgen for the GraphQL API. The schema can be found in ./api/schema.graphqls. After changing it, you need to run go run github.com/99designs/gqlgen which will update ./internal/graph/model. In case new resolvers are needed, they will be inserted into ./internal/graph/schema.resolvers.go, where you will need to implement them. If you start cc-backend with flag --dev the GraphQL Playground UI is available at http://localhost:8080/playground .

Update Swagger UI

This project integrates swagger ui to document and test its REST API. The swagger doc files can be found in ./api/. You can generate the configuration of swagger-ui by running go run github.com/swaggo/swag/cmd/swag init -d ./internal/api,./pkg/schema -g rest.go -o ./api . You need to move the generated ./api/doc.go to ./internal/api/doc.go. If you start cc-backend with flag --dev the Swagger UI is available at http://localhost:8080/swagger .

Project Structure

  • api/ contains the API schema files for the REST and GraphQL APIs. The REST API is documented in the OpenAPI 3.0 format in ./api/openapi.yaml.
  • cmd/cc-backend contains main.go for the main application.
  • cmd/gen-keypair contains is a small application to generate a compatible JWT keypair includin a README about JWT setup in ClusterCockpit.
  • configs/ contains documentation about configuration and command line options and required environment variables. An example configuration file is provided.
  • init/ contains an example systemd setup for production use.
  • internal/ contains library source code that is not intended to be used by others.
  • pkg/ contains go packages that can also be used by other projects.
  • test/ Test apps and test data.
  • web/ Server side templates and frontend related files:
    • templates Serverside go templates
    • frontend Svelte components and static assets for frontend UI
  • gqlgen.yml configures the behaviour and generation of gqlgen.
  • startDemo.sh is a shell script that sets up demo data, and builds and starts cc-backend.