.github/workflows | ||
api | ||
auth | ||
config | ||
frontend@239bf19c9a | ||
graph | ||
log | ||
metricdata | ||
repository | ||
schema | ||
templates | ||
test | ||
utils | ||
.env | ||
.gitignore | ||
.gitmodules | ||
go.mod | ||
go.sum | ||
gqlgen.yml | ||
LICENSE | ||
README.md | ||
routes.go | ||
runtimeSetup.go | ||
server.go | ||
startDemo.sh |
ClusterCockpit REST and GraphQL API backend
This is a Golang backend implementation for a REST and GraphQL API according to the ClusterCockpit specifications. It also includes a web interface for ClusterCockpit based on the components implemented in cc-frontend, which is included as a git submodule. This implementation replaces the previous PHP Symfony based ClusterCockpit web-interface.
Overview
This is a golang web backend for the ClusterCockpit job-specific performance monitoring framework. It provides a REST API for integrating ClusterCockpit with a HPC cluster batch system and external analysis scripts. Data exchange between the web frontend and backend is based on a GraphQL API. The web frontend is also served by the backend using Svelte components implemented in cc-frontend. Layout and styling is based on Bootstrap 5 using Bootstrap Icons. The backend uses SQLite 3 as relational SQL database by default. It can optionally use a MySQL/MariaDB database server. Finished batch jobs are stored in a so called job archive following this specification. The backend supports authentication using local accounts or an external LDAP directory. Authorization for APIs is implemented using JWT tokens created with public/private key encryption.
Demo Setup
We provide a shell skript that downloads demo data and automatically builds and starts cc-backend.
You need wget
, go
, and yarn
in your path to start the demo. The demo will download 32MB of data (223MB on disk).
# The frontend is a submodule, so use `--recursive`
git clone --recursive git@github.com:ClusterCockpit/cc-backend.git
./startDemo.sh
You can access the web interface at http://localhost:8080. Credentials for login: demo:AdminDev
. Please note that some views do not work without a metric backend (e.g., the Systems view).
Howto Build and Run
# The frontend is a submodule, so use `--recursive`
git clone --recursive git@github.com:ClusterCockpit/cc-backend.git
# Prepare frontend
cd ./cc-backend/frontend
yarn install
yarn build
cd ..
go get
go build
# The job-archive directory must be organised the same way as
# as for the regular ClusterCockpit.
ln -s <your-existing-job-archive> ./var/job-archive
# Create empty job.db (Will be initialized as SQLite3 database)
touch ./var/job.db
# EDIT THE .env FILE BEFORE YOU DEPLOY (Change the secrets)!
# If authentication is disabled, it can be empty.
vim ./.env
# This will first initialize the job.db database by traversing all
# `meta.json` files in the job-archive and add a new user. `--no-server` will cause the
# executable to stop once it has done that instead of starting a server.
./cc-backend --init-db --add-user <your-username>:admin:<your-password> --no-server
# Start a HTTP server (HTTPS can be enabled, the default port is 8080):
./cc-backend
# Show other options:
./cc-backend --help
Run as systemd daemon
In order to run this program as a daemon, look at utils/systemd/README.md where a systemd unit file and more explanation is provided.
Configuration and Setup
cc-backend can be used as a local web-interface for an existing job archive or as a general web-interface server for a live ClusterCockpit Monitoring framework.
Create your job-archive according to this specification. At least
one cluster with a valid cluster.json
file is required. Having no jobs in the
job-archive at all is fine. You may use the sample job-archive available for
download in cc-docker/develop.
Configuration
A config file in the JSON format can be provided using --config
to override the defaults.
Look at the beginning of server.go
for the defaults and consequently the format of the configuration file.
Update GraphQL schema
This project uses gqlgen for the GraphQL
API. The schema can be found in ./graph/schema.graphqls
. After changing it,
you need to run go run github.com/99designs/gqlgen
which will update
graph/model
. In case new resolvers are needed, they will be inserted into
graph/schema.resolvers.go
, where you will need to implement them.
Project Structure
api/
contains the REST API. The routes defined there should be called whenever a job starts/stops. The API is documented in the OpenAPI 3.0 format in ./api/openapi.yaml.auth/
is where the (optional) authentication middleware can be found, which adds the currently authenticated user to the request context. Theuser
table is created and managed here as well.auth/ldap.go
contains everything to do with automatically syncing and authenticating users form an LDAP server.
config
handles thecluster.json
files and the user-specific configurations (changeable via GraphQL) for the Web-UI such as the selected metrics etc.frontend
is a submodule, this is where the Svelte based frontend resides.graph/generated
should not be touched.graph/model
contains all types defined in the GraphQL schema not manually defined inschema/
. Manually defined types have to be listed ingqlgen.yml
.graph/schema.graphqls
contains the GraphQL schema. Whenever you change it, you should callgo run github.com/99designs/gqlgen
.graph/
contains the resolvers and handlers for the GraphQL API. Function signatures ingraph/schema.resolvers.go
are automatically generated.metricdata/
handles getting and archiving the metrics associated with a job.metricdata/metricdata.go
defines the interfaceMetricDataRepository
and provides functions to the GraphQL and REST API for accessing a jobs metrics which automatically take care of selecting the source for the metrics (the archive or one of the metric data repositories).metricdata/archive.go
provides functions for fetching metrics from the job-archive and archiving a job to the job-archive.metricdata/cc-metric-store.go
contains an implementation of theMetricDataRepository
interface which can fetch data from an cc-metric-storemetricdata/influxdb-v2
contains an implementation of theMetricDataRepository
interface which can fetch data from an InfluxDBv2 database. It is currently disabled and out of date and can not be used as of writing.
repository/
all SQL related stuff.repository/init.go
initializes thejob
(andtag
andjobtag
) table if the--init-db
flag is provided. Not only is the table created in the correct schema, but the job-archive is traversed as well.schema/
contains type definitions used all over this project extracted in this package as Go disallows cyclic dependencies between packages.schema/float.go
contains a customfloat64
type which overwrites JSON and GraphQL Marshaling/Unmarshalling. This is needed because a regular optionalFloat
in GraphQL will map to*float64
types in Go. Wrapping every single metric value in an allocation would be a lot of overhead.schema/job.go
provides the types representing a job and its resources. Those can be used as type for ameta.json
file and/or a row in thejob
table.
templates/
is mostly full of HTML templates and a small helper go module.utils/systemd
describes how to deploy/install this as a systemd servicetest/
rudimentery tests.utils/
.env
must be changed before you deploy this. It contains a Base64 encoded Ed25519 key-pair, the secret used for sessions and the password to the LDAP server if LDAP authentication is enabled.gqlgen.yml
configures the behaviour and generation of gqlgen.server.go
contains the main function and starts the actual http server.