ClusterCockpit/cc-backend
1
0
Fork 0
mirror of https://github.com/ClusterCockpit/cc-backend synced 2024-11-10 00:47:26 +01:00
Code Issues Packages Releases Activity

Cleanup README

Browse Source
This commit is contained in:
Jan Eitzinger 2024-02-13 11:55:00 +01:00
parent 0c7f55ff8d
commit e7ae9dd06d
1 changed files with 81 additions and 117 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

198
README.md
View File

@ -1,12 +1,16 @@
# NOTE # NOTE
Please have a look at the [Release Notes](https://github.com/ClusterCockpit/cc-backend/blob/master/ReleaseNotes.md) for breaking changes!
Please have a look at the [Release
Notes](https://github.com/ClusterCockpit/cc-backend/blob/master/ReleaseNotes.md)
for breaking changes!
# ClusterCockpit REST and GraphQL API backend # ClusterCockpit REST and GraphQL API backend
[![Build](https://github.com/ClusterCockpit/cc-backend/actions/workflows/test.yml/badge.svg)](https://github.com/ClusterCockpit/cc-backend/actions/workflows/test.yml) [![Build](https://github.com/ClusterCockpit/cc-backend/actions/workflows/test.yml/badge.svg)](https://github.com/ClusterCockpit/cc-backend/actions/workflows/test.yml)
This is a Golang backend implementation for a REST and GraphQL API according to This is a Golang backend implementation for a REST and GraphQL API according to
the [ClusterCockpit specifications](https://github.com/ClusterCockpit/cc-specifications). It also the [ClusterCockpit
specifications](https://github.com/ClusterCockpit/cc-specifications). It also
includes a web interface for ClusterCockpit. This implementation replaces the includes a web interface for ClusterCockpit. This implementation replaces the
previous PHP Symfony based ClusterCockpit web interface. The reasons for previous PHP Symfony based ClusterCockpit web interface. The reasons for
switching from PHP Symfony to a Golang based solution are explained switching from PHP Symfony to a Golang based solution are explained
@ -14,31 +18,31 @@ switching from PHP Symfony to a Golang based solution are explained
## Overview ## Overview
This is a Golang web backend for the ClusterCockpit job-specific performance
monitoring framework. It provides a REST API for integrating ClusterCockpit with
an HPC cluster batch system and external analysis scripts. Data exchange between
the web front-end and the back-end is based on a GraphQL API. The web frontend
is also served by the backend using [Svelte](https://svelte.dev/) components.
Layout and styling are based on [Bootstrap 5](https://getbootstrap.com/) using
[Bootstrap Icons](https://icons.getbootstrap.com/).
This is a Golang web backend for the ClusterCockpit job-specific performance monitoring framework. The backend uses [SQLite 3](https://sqlite.org/) as a relational SQL database by
It provides a REST API for integrating ClusterCockpit with an HPC cluster batch system and external analysis scripts. default. Optionally it can use a MySQL/MariaDB database server. While there are
Data exchange between the web front-end and the back-end is based on a GraphQL API. metric data backends for the InfluxDB and Prometheus time series databases, the
The web frontend is also served by the backend using [Svelte](https://svelte.dev/) components. only tested and supported setup is to use cc-metric-store as the metric data
Layout and styling are based on [Bootstrap 5](https://getbootstrap.com/) using [Bootstrap Icons](https://icons.getbootstrap.com/). backend. Documentation on how to integrate ClusterCockpit with other time series
databases will be added in the future.
The backend uses [SQLite 3](https://sqlite.org/) as a relational SQL database by default.
Optionally it can use a MySQL/MariaDB database server.
While there are metric data backends for the InfluxDB and Prometheus time series databases, the only tested and supported setup is to use cc-metric-store as the metric data backend.
Documentation on how to integrate ClusterCockpit with other time series databases will be added in the future.
Completed batch jobs are stored in a file-based job archive according to Completed batch jobs are stored in a file-based job archive according to
[this specification] (https://github.com/ClusterCockpit/cc-specifications/tree/master/job-archive). [this specification](https://github.com/ClusterCockpit/cc-specifications/tree/master/job-archive).
The backend supports authentication via local accounts, an external LDAP The backend supports authentication via local accounts, an external LDAP
directory, and JWT tokens. Authorization for APIs is implemented with directory, and JWT tokens. Authorization for APIs is implemented with
[JWT](https://jwt.io/) tokens created with public/private key encryption. [JWT](https://jwt.io/) tokens created with public/private key encryption.
You find more detailed information here: You find a detailed documentation on the [ClusterCockpit
* `./configs/README.md`: Infos about configuration and setup of cc-backend. Webpage](https://clustercockpit.org).
* `./init/README.md`: Infos on how to setup cc-backend as systemd service on Linux.
* `./tools/README.md`: Infos on the JWT authorizatin token workflows in ClusterCockpit.
* `./docs`: You can find further documentation here. There is also a Hands-on tutorial that is recommended to get familiar with the ClusterCockpit setup.
**NOTE** ## Build requirements
ClusterCockpit requires a current version of the golang toolchain and node.js. ClusterCockpit requires a current version of the golang toolchain and node.js.
You can check `go.mod` to see what is the current minimal golang version needed. You can check `go.mod` to see what is the current minimal golang version needed.
@ -49,7 +53,7 @@ on the Go standard library, it is crucial for security and performance to use a
current version of golang. In addition, an old golang toolchain may limit the supported current version of golang. In addition, an old golang toolchain may limit the supported
versions of third-party packages. versions of third-party packages.
## How to try ClusterCockpit with a demo setup. ## How to try ClusterCockpit with a demo setup
We provide a shell script that downloads demo data and automatically starts the We provide a shell script that downloads demo data and automatically starts the
cc-backend. You will need `wget`, `go`, `node`, `npm` in your path to cc-backend. You will need `wget`, `go`, `node`, `npm` in your path to
@ -64,28 +68,32 @@ cd ./cc-backend
You can also try the demo using the lates release binary. You can also try the demo using the lates release binary.
Create a folder and put the release binary `cc-backend` into this folder. Create a folder and put the release binary `cc-backend` into this folder.
Execute the following steps: Execute the following steps:
```
$ ./cc-backend -init ``` shell
$ vim config.json (Add a second cluster entry and name the clusters alex and fritz) ./cc-backend -init
$ wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/job-archive-demo.tar vim config.json (Add a second cluster entry and name the clusters alex and fritz)
$ tar xf job-archive-demo.tar wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/job-archive-demo.tar
$ ./cc-backend -init-db -add-user demo:admin:demo -loglevel info tar xf job-archive-demo.tar
$ ./cc-backend -server -dev -loglevel info ./cc-backend -init-db -add-user demo:admin:demo -loglevel info
./cc-backend -server -dev -loglevel info
``` ```
You can access the web interface at http://localhost:8080. You can access the web interface at [http://localhost:8080](http://localhost:8080).
Credentials for login are `demo:demo`. Credentials for login are `demo:demo`.
Please note that some views do not work without a metric backend (e.g., the Please note that some views do not work without a metric backend (e.g., the
Analysis, Systems and Status views). Analysis, Systems and Status views).
## Howto build and run ## How to build and run
There is a Makefile to automate the build of cc-backend. The Makefile supports the following targets: There is a Makefile to automate the build of cc-backend. The Makefile supports
* `$ make`: Initialize `var` directory and build svelte frontend and backend binary. Note that there is no proper prerequesite handling. Any change of frontend source files will result in a complete rebuild. the following targets:
* `$ make clean`: Clean go build cache and remove binary.
* `$ make test`: Run the tests that are also run in the GitHub workflow setup. * `make`: Initialize `var` directory and build svelte frontend and backend binary. Note that there is no proper prerequesite handling. Any change of frontend source files will result in a complete rebuild.
* `make clean`: Clean go build cache and remove binary.
* `make test`: Run the tests that are also run in the GitHub workflow setup.
A common workflow for setting up cc-backend from scratch is: A common workflow for setting up cc-backend from scratch is:
```sh ```sh
git clone https://github.com/ClusterCockpit/cc-backend.git git clone https://github.com/ClusterCockpit/cc-backend.git
@ -116,89 +124,45 @@ ln -s <your-existing-job-archive> ./var/job-archive
./cc-backend -help ./cc-backend -help
``` ```
### Run as systemd daemon
To run this program as a daemon, cc-backend comes with a [example systemd setup](./init/README.md).
## Configuration and setup
cc-backend can be used as a local web interface for an existing job archive or
as a server for the ClusterCockpit monitoring framework.
Create your job archive according to [this specification] (https://github.com/ClusterCockpit/cc-specifications/tree/master/job-archive).
At least one cluster directory with a valid `cluster.json` file is required. If
you configure the job archive from scratch, you must also create the job
archive version file that contains the job archive version as an integer.
You can retrieve the currently supported version by running the following
command:
```
$ ./cc-backend -version
```
It is ok to have no jobs in the job archive.
### Configuration
A configuration file in JSON format must be specified with `-config` to override the default settings.
By default, a `config.json` file located in the current directory of the `cc-backend` process will be loaded even without the `-config` flag.
Documentation of all supported configuration and command line options can be found [here](./configs/README.md).
## Database initialization and migration
Each `cc-backend` version supports a specific database version.
At startup, the version of the sqlite database is checked and `cc-backend` terminates if the version does not match.
`cc-backend` supports the migration of the database schema to the required version with the command line option `-migrate-db`.
If the database file does not exist yet, it will be created and initialized with the command line option `-migrate-db`.
If you want to use a newer database version with an older version of cc-backend, you can downgrade a database with the external tool [migrate](https://github.com/golang-migrate/migrate).
In this case, you must specify the path to the migration files in a current source tree: `./internal/repository/migrations/`.
## Development and testing
When making changes to the REST or GraphQL API, the appropriate code generators must be used.
You must always rebuild `cc-backend` after updating the API files.
### Update GraphQL schema
This project uses [gqlgen](https://github.com/99designs/gqlgen) for the GraphQL API.
The schema can be found in `./api/schema.graphqls`.
After changing it, you need to run `go run github.com/99designs/gqlgen`, which will update `./internal/graph/model`.
If new resolvers are needed, they will be added to `./internal/graph/schema.resolvers.go`, where you will then need to implement them.
If you start `cc-backend` with the `-dev` flag, the GraphQL Playground UI is available at http://localhost:8080/playground.
### Update Swagger UI
This project integrates [swagger ui] (https://swagger.io/tools/swagger-ui/) to document and test its REST API.
The swagger documentation files can be found in `./api/`.
You can generate the swagger-ui configuration by running `go run github.com/swaggo/swag/cmd/swag init -d ./internal/api,./pkg/schema -g rest.go -o ./api `.
You need to move the created `./api/docs.go` to `./internal/api/docs.go`.
If you start cc-backend with the `-dev` flag, the Swagger interface is available
at http://localhost:8080/swagger/.
You must enter a JWT key for a user with the API role.
**NOTE**
The user who owns the JWT key must not be logged into the same browser (have a
running session), or the Swagger requests will not work. It is recommended to
create a separate user that has only the API role.
## Development and testing
In case the REST or GraphQL API is changed the according code generators have to be used.
## Project file structure ## Project file structure
- [`api/`](https://github.com/ClusterCockpit/cc-backend/tree/master/api) contains the API schema files for the REST and GraphQL APIs. The REST API is documented in the OpenAPI 3.0 format in [./api/openapi.yaml](./api/openapi.yaml). * [`api/`](https://github.com/ClusterCockpit/cc-backend/tree/master/api)
- [`cmd/cc-backend`](https://github.com/ClusterCockpit/cc-backend/tree/master/cmd/cc-backend) contains `main.go` for the main application. contains the API schema files for the REST and GraphQL APIs. The REST API is
- [`configs/`](https://github.com/ClusterCockpit/cc-backend/tree/master/configs) contains documentation about configuration and command line options and required environment variables. A sample configuration file is provided. documented in the OpenAPI 3.0 format in
- [`docs/`](https://github.com/ClusterCockpit/cc-backend/tree/master/docs) contains more in-depth documentation. [./api/openapi.yaml](./api/openapi.yaml).
- [`init/`](https://github.com/ClusterCockpit/cc-backend/tree/master/init) contains an example of setting up systemd for production use. * [`cmd/cc-backend`](https://github.com/ClusterCockpit/cc-backend/tree/master/cmd/cc-backend)
- [`internal/`](https://github.com/ClusterCockpit/cc-backend/tree/master/internal) contains library source code that is not intended for use by others. contains `main.go` for the main application.
- [`pkg/`](https://github.com/ClusterCockpit/cc-backend/tree/master/pkg) contains Go packages that can be used by other projects. * [`configs/`](https://github.com/ClusterCockpit/cc-backend/tree/master/configs)
- [`tools/`](https://github.com/ClusterCockpit/cc-backend/tree/master/tools) Additional command line helper tools. contains documentation about configuration and command line options and required
- [`archive-manager`](https://github.com/ClusterCockpit/cc-backend/tree/master/tools/archive-manager) Commands for getting infos about and existing job archive. environment variables. A sample configuration file is provided.
- [`archive-migration`](https://github.com/ClusterCockpit/cc-backend/tree/master/tools/archive-migration) Tool to migrate from previous to current job archive version. * [`docs/`](https://github.com/ClusterCockpit/cc-backend/tree/master/docs)
- [`convert-pem-pubkey`](https://github.com/ClusterCockpit/cc-backend/tree/master/tools/convert-pem-pubkey) Tool to convert external pubkey for use in `cc-backend`. contains more in-depth documentation.
- [`gen-keypair`](https://github.com/ClusterCockpit/cc-backend/tree/master/tools/gen-keypair) contains a small application to generate a compatible JWT keypair. You find documentation on how to use it [here](https://github.com/ClusterCockpit/cc-backend/blob/master/docs/JWT-Handling.md). * [`init/`](https://github.com/ClusterCockpit/cc-backend/tree/master/init)
- [`web/`](https://github.com/ClusterCockpit/cc-backend/tree/master/web) Server-side templates and frontend-related files: contains an example of setting up systemd for production use.
- [`frontend`](https://github.com/ClusterCockpit/cc-backend/tree/master/web/frontend) Svelte components and static assets for the frontend UI * [`internal/`](https://github.com/ClusterCockpit/cc-backend/tree/master/internal)
- [`templates`](https://github.com/ClusterCockpit/cc-backend/tree/master/web/templates) Server-side Go templates contains library source code that is not intended for use by others.
- [`gqlgen.yml`](https://github.com/ClusterCockpit/cc-backend/blob/master/gqlgen.yml) Configures the behaviour and generation of [gqlgen](https://github.com/99designs/gqlgen). * [`pkg/`](https://github.com/ClusterCockpit/cc-backend/tree/master/pkg)
- [`startDemo.sh`](https://github.com/ClusterCockpit/cc-backend/blob/master/startDemo.sh) is a shell script that sets up demo data, and builds and starts `cc-backend`. contains Go packages that can be used by other projects.
* [`tools/`](https://github.com/ClusterCockpit/cc-backend/tree/master/tools)
Additional command line helper tools.
* [`archive-manager`](https://github.com/ClusterCockpit/cc-backend/tree/master/tools/archive-manager)
Commands for getting infos about and existing job archive.
* [`archive-migration`](https://github.com/ClusterCockpit/cc-backend/tree/master/tools/archive-migration)
Tool to migrate from previous to current job archive version.
* [`convert-pem-pubkey`](https://github.com/ClusterCockpit/cc-backend/tree/master/tools/convert-pem-pubkey)
Tool to convert external pubkey for use in `cc-backend`.
* [`gen-keypair`](https://github.com/ClusterCockpit/cc-backend/tree/master/tools/gen-keypair)
contains a small application to generate a compatible JWT keypair. You find
documentation on how to use it
[here](https://github.com/ClusterCockpit/cc-backend/blob/master/docs/JWT-Handling.md).
* [`web/`](https://github.com/ClusterCockpit/cc-backend/tree/master/web)
Server-side templates and frontend-related files:
* [`frontend`](https://github.com/ClusterCockpit/cc-backend/tree/master/web/frontend)
Svelte components and static assets for the frontend UI
* [`templates`](https://github.com/ClusterCockpit/cc-backend/tree/master/web/templates)
Server-side Go templates
* [`gqlgen.yml`](https://github.com/ClusterCockpit/cc-backend/blob/master/gqlgen.yml)
Configures the behaviour and generation of
[gqlgen](https://github.com/99designs/gqlgen).
* [`startDemo.sh`](https://github.com/ClusterCockpit/cc-backend/blob/master/startDemo.sh)
is a shell script that sets up demo data, and builds and starts `cc-backend`.