ClusterCockpit/cc-backend
1
0
Fork 0
mirror of https://github.com/ClusterCockpit/cc-backend synced 2025-11-04 01:25:06 +01:00
Code Issues Packages Releases Activity

Merge branch 'hotfix' of https://github.com/ClusterCockpit/cc-backend into hotfix

Browse Source
This commit is contained in:
Christoph Kluge
2023-06-13 16:43:34 +02:00
parent cdd25eaa15 cc1ae28fb3
commit b7fa4f17d2
8 changed files with 121 additions and 33 deletions

4
README.md

@@ -30,7 +30,7 @@ You find more detailed information here:
**NOTICE**
ClusterCockpit requires a recent version of the golang toolchain.
ClusterCockpit requires a recent version of the golang toolchain and node.js.
You can check in `go.mod` what is the current minimal golang version required.
Homebrew and Archlinux usually have up to date golang versions. For other Linux
distros this often means you have to install the golang compiler yourself.
@@ -107,7 +107,7 @@ Having no jobs in the job-archive at all is fine.
A config file in the JSON format has to be provided using `--config` to override the defaults.
By default, if there is a `config.json` file in the current directory of the `cc-backend` process, it will be loaded even without the `--config` flag.
You find documentation of all supported configuration and command line options [here](./configs.README.md).
You find documentation of all supported configuration and command line options [here](./configs/README.md).
## Database initialization and migration

0
docs/config.json → configs/config-demo.json

48
docs/ConfigurationManagement.md

@@ -1,37 +1,37 @@
# Release versioning
Releases are numbered with a integer id starting with 1.
Every release embeds the following assets into the binary:
* Web-frontend including javascript files and all static assets
* Golang template files for server-side rendering
* JSON schema files for validation
Releases are numbered with an integer ID, starting with 1.
Each release embeds the following assets in the binary:
* Web front-end with Javascript files and all static assets.
* Golang template files for server-side rendering.
* JSON schema files for validation.
* Database migration files
Remaining external assets are:
The remaining external assets are:
* The SQL database used
* The job archive
* The configuration file `config.json`
Both external assets are also versioned using integer ids.
This means every release binary is tied to specific versions for the SQL
database and job archive.
A command line switch `--migrate-db` is provided to migrate the SQL database
from a previous to the most recent version.
We provide a separate tool `archive-migration` to migrate an existing job
archive from the previous to the most recent version.
Both external assets are also versioned with integer IDs.
This means that each release binary is bound to specific versions of the SQL
database and the job archive.
The configuration file is validated against the current schema on startup.
The command line switch `-migrate-db` can be used to upgrade the SQL database
to migrate from a previous version to the latest one.
We offer a separate tool `archive-migration` to migrate an existing job archive
archive from the previous to the latest version.
# Versioning of APIs
cc-backend provides two API backends:
* A REST API for querying jobs
* A GraphQL API used for data exchange between web frontend and cc-backend
* A GraphQL API for data exchange between web frontend and cc-backend
Both APIs will also be versioned. We still need to decide if we also support
older REST API version using versioning of the endpoint URLs.
# How to build a specific release
# How to migrate the SQL database
# How to migrate the job archive
Both APIs will also be versioned. We still need to decide wether we will also support
older REST API version by versioning the endpoint URLs.
# How to build
Please always build `cc-backend` with the supplied Makefile. This will ensure
that the frontend is also built correctly and that the version in the binary file is coded
in the binary.

13
docs/adm-customization.md Normal file

@@ -0,0 +1,13 @@
# Overview
Customizing `cc-backend` means changing the logo and certain legal texts
instead of the placeholders. To change the logo displayed in the navigation bar, the
file `web/frontend/public/img/logo.png` in the source tree must be replaced
and cc-backend must be rebuild.
# Replace legal texts
To replace the `imprint.tmpl` and `privacy.tmpl` legal texts, you can place your
version in `./var/`. At startup `cc-backend` will check if `./var/imprint.tmpl` and/or
`./var/privacy.tmpl` exist and use them instead of the built-in placeholders.
You can use the placeholders in `web/templates` as a blueprint.

78
docs/adm-upgrade.md Normal file

@@ -0,0 +1,78 @@
In general, an upgrade is nothing more than a replacement of the binary file.
All the necessary files, except the database file, the configuration file and
the job archive, are embedded in the binary file. It is recommended to use a
directory where the file names of the binary files are named with a version
indicator. This can be, for example, the date or the Unix epoch time. A symbolic
link points to the version to be used. This makes it easier to switch to earlier
versions.
The database and the job archive are versioned. Each release binary supports
specific versions of the database and job archive. If a version mismatch is
detected, the application is terminated and migration is required.
**IMPORTANT NOTE**
It is recommended to make a backup copy of the database before each update. This
is mandatory in case the database needs to be migrated. In the case of sqlite,
this means to stopping `cc-backend` and copying the sqlite database file
somewhere.
# Migrating the database
After you have backed up the database, run the following command to migrate the
database to the latest version:
```
$ ./cc-backend -migrate-db
```
The migration files are embedded in the binary and can also be viewed in the cc
backend [source tree](https://github.com/ClusterCockpit/cc-backend/tree/master/internal/repository/migrations).
There are separate migration files for both supported
database backends.
We use the [migrate library](https://github.com/golang-migrate/migrate).
If something goes wrong, you can check the status and get the current schema
(here for sqlite):
```
$ sqlite3 var/job.db
```
In the sqlite console execute:
```
.schema
```
to get the current databse schema.
You can query the current version and whether the migration failed with:
```
SELECT * FROM schema_migrations;
```
The first column indicates the current database version and the second column is
a dirty flag indicating whether the migration was successful.
# Migrating the job archive
Job archive migration requires a separate tool (`archive-migration`), which is
part of the cc-backend source tree (build with `go build ./tools/archive-migration`)
and is also provided as part of the releases.
Migration is supported only between two successive releases. The migration tool
migrates the existing job archive to a new job archive. This means that there
must be enough disk space for two complete job archives. If the tool is called
without options:
```
$ ./archive-migration
```
it is assumed that a job archive exists in `./var/job-archive`. The new job
archive is written to `./var/job-archive-new`. Since execution is threaded in case
of a fatal error, it is impossible to determine in which job the error occurred.
In this case, you can run the tool in debug mode (with the `-debug` flag). In
debug mode, threading is disabled and the job ID of each migrated job is output.
Jobs with empty files will be skipped. Between multiple runs of the tools, the
`job-archive-new` directory must be moved or deleted.
The `cluster.json` files in `job-archive-new` must be checked for errors, especially
whether the aggregation attribute is set correctly for all metrics.
Migration takes several hours for relatively large job archives (several hundred
GB). A versioned job archive contains a version.txt file in the root directory
of the job archive. This file contains the version as an unsigned integer.

2
internal/config/config_test.go

@@ -17,7 +17,7 @@ func TestInit(t *testing.T) {
}
func TestInitMinimal(t *testing.T) {
fp := "../../docs/config.json"
fp := "../../configs/config-demo.json"
Init(fp)
if Keys.Addr != "127.0.0.1:8080" {
t.Errorf("wrong addr\ngot: %s \nwant: 127.0.0.1:8080", Keys.Addr)

2
startDemo.sh

@@ -10,7 +10,7 @@ else
rm ./job-archive-demo.tar
cp ./configs/env-template.txt .env
cp ./docs/config.json config.json
cp ./configs/config-demo.json config.json
./cc-backend --migrate-db
./cc-backend --server --dev --init-db --add-user demo:admin:demo

7
web/frontend/README.md

@@ -12,18 +12,15 @@ Builds on:
## Get started
[Yarn](https://yarnpkg.com/) is recommended for package management.
Due to an issue with Yarn v2 you have to stick to Yarn v1.
Install the dependencies...
```bash
yarn install
npm install
```
...then build using [Rollup](https://rollupjs.org):
```bash
yarn build
npm run build
```