mirror of
https://github.com/ClusterCockpit/cc-backend
synced 2024-12-25 21:09:05 +01:00
Jan Eitzinger
GitHub
commit
445fe14ac8
@ -1,37 +1,37 @@
|
||||
# Release versioning
|
||||
|
||||
Releases are numbered with a integer id starting with 1.
|
||||
Every release embeds the following assets into the binary:
|
||||
* Web-frontend including javascript files and all static assets
|
||||
* Golang template files for server-side rendering
|
||||
* JSON schema files for validation
|
||||
Releases are numbered with an integer ID, starting with 1.
|
||||
Each release embeds the following assets in the binary:
|
||||
* Web front-end with Javascript files and all static assets.
|
||||
* Golang template files for server-side rendering.
|
||||
* JSON schema files for validation.
|
||||
* Database migration files
|
||||
|
||||
Remaining external assets are:
|
||||
The remaining external assets are:
|
||||
* The SQL database used
|
||||
* The job archive
|
||||
* The configuration file `config.json`
|
||||
|
||||
Both external assets are also versioned using integer ids.
|
||||
This means every release binary is tied to specific versions for the SQL
|
||||
database and job archive.
|
||||
A command line switch `--migrate-db` is provided to migrate the SQL database
|
||||
from a previous to the most recent version.
|
||||
We provide a separate tool `archive-migration` to migrate an existing job
|
||||
archive from the previous to the most recent version.
|
||||
Both external assets are also versioned with integer IDs.
|
||||
This means that each release binary is bound to specific versions of the SQL
|
||||
database and the job archive.
|
||||
The configuration file is validated against the current schema on startup.
|
||||
The command line switch `-migrate-db` can be used to upgrade the SQL database
|
||||
to migrate from a previous version to the latest one.
|
||||
We offer a separate tool `archive-migration` to migrate an existing job archive
|
||||
archive from the previous to the latest version.
|
||||
|
||||
# Versioning of APIs
|
||||
|
||||
cc-backend provides two API backends:
|
||||
* A REST API for querying jobs
|
||||
* A GraphQL API used for data exchange between web frontend and cc-backend
|
||||
* A GraphQL API for data exchange between web frontend and cc-backend
|
||||
|
||||
Both APIs will also be versioned. We still need to decide if we also support
|
||||
older REST API version using versioning of the endpoint URLs.
|
||||
|
||||
# How to build a specific release
|
||||
|
||||
|
||||
# How to migrate the SQL database
|
||||
|
||||
|
||||
# How to migrate the job archive
|
||||
Both APIs will also be versioned. We still need to decide wether we will also support
|
||||
older REST API version by versioning the endpoint URLs.
|
||||
|
||||
# How to build
|
||||
|
||||
Please always build `cc-backend` with the supplied Makefile. This will ensure
|
||||
that the frontend is also built correctly and that the version in the binary file is coded
|
||||
in the binary.
|
||||
|
||||
@ -1,13 +1,13 @@
|
||||
# Overview
|
||||
|
||||
To customize `cc-backend` means to change the logo and add specific legal texts
|
||||
in place of the placeholders. To change the logo shown in the navigation bar the
|
||||
file `web/frontend/public/img/logo.png` has to be replaced in the source tree
|
||||
and cc-backend has to be rebuild.
|
||||
Customizing `cc-backend` means changing the logo and certain legal texts
|
||||
instead of the placeholders. To change the logo displayed in the navigation bar, the
|
||||
file `web/frontend/public/img/logo.png` in the source tree must be replaced
|
||||
and cc-backend must be rebuild.
|
||||
|
||||
# Replace legal texts
|
||||
|
||||
To replace the legal texts `imprint.tmpl` and `privacy.tmpl` you can place your
|
||||
version in `./var/`. On startup `cc-backend` will check if `./var/imprint.tmpl` and/or
|
||||
`./var/privacy.tmpl` exist and use those instead of the builtin placeholders.
|
||||
You may use the placeholders in `web/templates` as blueprint.
|
||||
To replace the `imprint.tmpl` and `privacy.tmpl` legal texts, you can place your
|
||||
version in `./var/`. At startup `cc-backend` will check if `./var/imprint.tmpl` and/or
|
||||
`./var/privacy.tmpl` exist and use them instead of the built-in placeholders.
|
||||
You can use the placeholders in `web/templates` as a blueprint.
|
||||
|
||||
@ -1,33 +1,37 @@
|
||||
In general an upgrade is not more then replacing the binary. All files that are
|
||||
required, apart from the database file, the configuration file and the job
|
||||
archive are embedded into the binary. It is recommended to use a directory where
|
||||
the binary filenames are named using some version tag. This may be, e.g., the date or
|
||||
unix epoch time. A symbolic link points to the version to be used. This allows
|
||||
to easily switch to previous versions.
|
||||
In general, an upgrade is nothing more than a replacement of the binary file.
|
||||
All the necessary files, except the database file, the configuration file and
|
||||
the job archive, are embedded in the binary file. It is recommended to use a
|
||||
directory where the file names of the binary files are named with a version
|
||||
indicator. This can be, for example, the date or the Unix epoch time. A symbolic
|
||||
link points to the version to be used. This makes it easier to switch to earlier
|
||||
versions.
|
||||
|
||||
The database and the job archive are versioned. Every release binary supports
|
||||
specific versions of the database and the job archive. In case a version
|
||||
mismatch is detected the application will exit and a migration is required.
|
||||
The database and the job archive are versioned. Each release binary supports
|
||||
specific versions of the database and job archive. If a version mismatch is
|
||||
detected, the application is terminated and migration is required.
|
||||
|
||||
**IMPORTANT NOTICE**
|
||||
**IMPORTANT NOTE**
|
||||
|
||||
It is recommended to backup the database before any upgrade. It is mandatory
|
||||
to backup in case the database is migrated. Using sqlite this means to stop
|
||||
`cc-backend` and copy the sqlite database file somewhere.
|
||||
It is recommended to make a backup copy of the database before each update. This
|
||||
is mandatory in case the database needs to be migrated. In the case of sqlite,
|
||||
this means to stopping `cc-backend` and copying the sqlite database file
|
||||
somewhere.
|
||||
|
||||
# Migrating the database
|
||||
After backing up the database execute the the following command to migrate the
|
||||
databse to the most recent version:
|
||||
|
||||
After you have backed up the database, run the following command to migrate the
|
||||
database to the latest version:
|
||||
```
|
||||
$ ./cc-backend -migrate-db
|
||||
```
|
||||
|
||||
The migration files are embedded into the binary but can be reviewed in the
|
||||
cc-backend [source tree](https://github.com/ClusterCockpit/cc-backend/tree/master/internal/repository/migrations).
|
||||
There are separate migration files for both supported database backends.
|
||||
The migration files are embedded in the binary and can also be viewed in the cc
|
||||
backend [source tree](https://github.com/ClusterCockpit/cc-backend/tree/master/internal/repository/migrations).
|
||||
There are separate migration files for both supported
|
||||
database backends.
|
||||
We use the [migrate library](https://github.com/golang-migrate/migrate).
|
||||
|
||||
In case something goes wrong you can check the state and get the current schema
|
||||
If something goes wrong, you can check the status and get the current schema
|
||||
(here for sqlite):
|
||||
```
|
||||
$ sqlite3 var/job.db
|
||||
@ -37,38 +41,38 @@ In the sqlite console execute:
|
||||
.schema
|
||||
```
|
||||
to get the current databse schema.
|
||||
You can query the current version and if the migration failed with:
|
||||
You can query the current version and whether the migration failed with:
|
||||
```
|
||||
SELECT * FROM schema_migrations;
|
||||
```
|
||||
The first column indicates the current database version and the second column is
|
||||
a dirty flag indicating if the migration was successful.
|
||||
a dirty flag indicating whether the migration was successful.
|
||||
|
||||
# Migrating the job archive
|
||||
|
||||
To migrate the job archive a separate tool (`archive-migration`) is required that is part of the
|
||||
`cc-backend` source tree (build with `go build ./tools/archive-migration`) and also provided as part of releases.
|
||||
Job archive migration requires a separate tool (`archive-migration`), which is
|
||||
part of the cc-backend source tree (build with `go build ./tools/archive-migration`)
|
||||
and is also provided as part of the releases.
|
||||
|
||||
Migration is only supported between two subsequent versions. The migration tool
|
||||
will migrate the existing job archive into a new job archive. This means there
|
||||
has to be enough disk space for two complete job-archives. If the tool is
|
||||
called without options:
|
||||
Migration is supported only between two successive releases. The migration tool
|
||||
migrates the existing job archive to a new job archive. This means that there
|
||||
must be enough disk space for two complete job archives. If the tool is called
|
||||
without options:
|
||||
```
|
||||
$ ./archive-migration
|
||||
```
|
||||
|
||||
it is assumed that there is a job archive in `./var/job-archive`. The new job
|
||||
archive will be written to `./var/job-archive-new`. Because execution is
|
||||
threaded in case of a fatal error it is impossible to pinpoint in which job the
|
||||
error occured. In this case you can run the tool in debug mode (using the flag
|
||||
`-debug`). In debug mode threading will be disabled and the job id of every
|
||||
migrated job is printed. Jobs with empty files are skipped. Between multiple runs
|
||||
of the tools the directory `job-archive-new` has to be moved or deleted.
|
||||
it is assumed that a job archive exists in `./var/job-archive`. The new job
|
||||
archive is written to `./var/job-archive-new`. Since execution is threaded in case
|
||||
of a fatal error, it is impossible to determine in which job the error occurred.
|
||||
In this case, you can run the tool in debug mode (with the `-debug` flag). In
|
||||
debug mode, threading is disabled and the job ID of each migrated job is output.
|
||||
Jobs with empty files will be skipped. Between multiple runs of the tools, the
|
||||
`job-archive-new` directory must be moved or deleted.
|
||||
|
||||
The cluster.json files in `job-archive-new` need to be reviewed for errors, in
|
||||
particular it has to be checked if the aggregation attribute is set correctly
|
||||
for all metrics.
|
||||
The `cluster.json` files in `job-archive-new` must be checked for errors, especially
|
||||
whether the aggregation attribute is set correctly for all metrics.
|
||||
|
||||
The migration will take in the order of hours for fairly large (several hundred
|
||||
GB) job archives. A versioned job archive contains a file `version.txt` in the
|
||||
job archive root directory. This file contains the version as unsigned integer.
|
||||
Migration takes several hours for relatively large job archives (several hundred
|
||||
GB). A versioned job archive contains a version.txt file in the root directory
|
||||
of the job archive. This file contains the version as an unsigned integer.
|
||||
|
||||
@ -12,18 +12,15 @@ Builds on:
|
||||
|
||||
## Get started
|
||||
|
||||
[Yarn](https://yarnpkg.com/) is recommended for package management.
|
||||
Due to an issue with Yarn v2 you have to stick to Yarn v1.
|
||||
|
||||
Install the dependencies...
|
||||
|
||||
```bash
|
||||
yarn install
|
||||
npm install
|
||||
```
|
||||
|
||||
...then build using [Rollup](https://rollupjs.org):
|
||||
|
||||
```bash
|
||||
yarn build
|
||||
npm run build
|
||||
```
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user