From 18bbd11c5e80eb531788195154d3ca9677a608f6 Mon Sep 17 00:00:00 2001 From: Jan Eitzinger Date: Tue, 13 Jun 2023 14:35:57 +0200 Subject: [PATCH 1/6] Update documentation --- README.md | 4 +- docs/config.json => configs/config-demo.json | 0 docs/adm-customization.md | 13 ++++ docs/adm-upgrade.md | 73 ++++++++++++++++++++ startDemo.sh | 2 +- 5 files changed, 89 insertions(+), 3 deletions(-) rename docs/config.json => configs/config-demo.json (100%) create mode 100644 docs/adm-customization.md create mode 100644 docs/adm-upgrade.md diff --git a/README.md b/README.md index b09f6fd..c8cd60b 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ You find more detailed information here: **NOTICE** -ClusterCockpit requires a recent version of the golang toolchain. +ClusterCockpit requires a recent version of the golang toolchain and node.js. You can check in `go.mod` what is the current minimal golang version required. Homebrew and Archlinux usually have up to date golang versions. For other Linux distros this often means you have to install the golang compiler yourself. @@ -107,7 +107,7 @@ Having no jobs in the job-archive at all is fine. A config file in the JSON format has to be provided using `--config` to override the defaults. By default, if there is a `config.json` file in the current directory of the `cc-backend` process, it will be loaded even without the `--config` flag. -You find documentation of all supported configuration and command line options [here](./configs.README.md). +You find documentation of all supported configuration and command line options [here](./configs/README.md). ## Database initialization and migration diff --git a/docs/config.json b/configs/config-demo.json similarity index 100% rename from docs/config.json rename to configs/config-demo.json diff --git a/docs/adm-customization.md b/docs/adm-customization.md new file mode 100644 index 0000000..5c8801f --- /dev/null +++ b/docs/adm-customization.md @@ -0,0 +1,13 @@ +# Overview + +To customize `cc-backend` means to change the logo and add specific legal texts +in place of the placeholders. To change the logo shown in the navigation bar the +file `web/frontend/public/img/logo.png` has to be replaced in the source tree +and cc-backend has to be rebuild. + +# Replace legal texts + +To replace the legal texts `imprint.tmpl` and `privacy.tmpl` you can place your +version in `./var/`. On startup `cc-backend` will check if `./var/imprint.tmpl` and/or +`./var/privacy.tmpl` exist and use those instead of the builtin placeholders. +You may use the placeholders in `web/templates` as blueprint. diff --git a/docs/adm-upgrade.md b/docs/adm-upgrade.md new file mode 100644 index 0000000..e7e2791 --- /dev/null +++ b/docs/adm-upgrade.md @@ -0,0 +1,73 @@ +In general an upgrade is not more then replacing the binary. All files that are +required, apart from the database file, the configuration file and the job +archive are embedded into the binary. It is recommended to use a directory where +the binary filenames are named using some version tag. This may be, e.g., the date or +unix epoch time. A symbolic link points to the version to be used. This allows +to easily switch to previous versions. + +The database and the job archive are versioned. Every release binary supports +specific versions of the database and the job archive. In case a version +mismatch is detected the application will exit and a migration is required. + +**IMPORTANT NOTICE** + +It is recommended to backup the database before any upgrade. This is mandatory +in case the database needs to be migrated. In case of sqlite this means to stop +`cc-backend` and copy the sqlite databse file somewhere. + +# Migrating the database +After backing up the database execute the the following command to migrate the +databse to the most recent version: +``` +$ ./cc-backend -migrate-db +``` + +The migration files are embedded into the binary but can be reviewed in the +cc-backend [source tree](https://github.com/ClusterCockpit/cc-backend/tree/master/internal/repository/migrations). +There are separate migration files for both supported database backends. +We use the [migrate library](https://github.com/golang-migrate/migrate). + +In case something goes wrong you can check the state and get the current schema +(here for sqlite): +``` +$ sqlite3 var/job.db +``` +In the sqlite console execute: +``` +.schema +``` +to get the current databse schema. +You can query the current version and if the migration failed with: +``` +SELECT * FROM schema_migrations; +``` +The first column indicates the current database version and the second column is +a dirty flag indicating if the migration was successful. + +# Migrating the job archive + +To migrate the job archive a separate tool (`archive-migration`) is required that is part of the +`cc-backend` source tree (build with `go build ./tools/archive-migration`) and also provided as part of releases. + +Migration is only supported between two subsequent versions. The migration tool +will migrate the existing job archive into a new job archive. If the tool is +called without options: +``` +$ ./archive-migration +``` + +it is assumed that there is a job archive in `./var/job-archive`. The new job +archive will be written to `./var/job-archive-new`. Because execution is +threaded in case of a fatal error it is impossible to pinpoint in which job the +error occured. In this case you can run the tool in debug mode (using the flag +`-debug`). In debug mode threading will be disabled and the job id of every +migrated job is printed. Jobs with empty files are skipped. Between multiple runs +of the tools the directory `job-archive-new` has to be moved or deleted. + +The cluster.json files in `job-archive-new` need to be reviewed for errors, in +particular it has to be checked if the aggregation attribute is set correctly +for all metrics. + +The migration will take in the order of hours for fairly large (several hundred +GB) job archives. A versioned job archive contains a file `version.txt` in the +job archive root directory. This file contains the version as unsigned integer. diff --git a/startDemo.sh b/startDemo.sh index 447bdcc..1cfe11c 100755 --- a/startDemo.sh +++ b/startDemo.sh @@ -10,7 +10,7 @@ else rm ./job-archive-demo.tar cp ./configs/env-template.txt .env - cp ./docs/config.json config.json + cp ./configs/config-demo.json config.json ./cc-backend --migrate-db ./cc-backend --server --dev --init-db --add-user demo:admin:demo From 857ce637c077d58c45eb4af085a9c4133f044445 Mon Sep 17 00:00:00 2001 From: Jan Eitzinger Date: Tue, 13 Jun 2023 14:39:11 +0200 Subject: [PATCH 2/6] Update adm-upgrade.md --- docs/adm-upgrade.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/adm-upgrade.md b/docs/adm-upgrade.md index e7e2791..57270ed 100644 --- a/docs/adm-upgrade.md +++ b/docs/adm-upgrade.md @@ -50,7 +50,8 @@ To migrate the job archive a separate tool (`archive-migration`) is required tha `cc-backend` source tree (build with `go build ./tools/archive-migration`) and also provided as part of releases. Migration is only supported between two subsequent versions. The migration tool -will migrate the existing job archive into a new job archive. If the tool is +will migrate the existing job archive into a new job archive. This means there +has to be enough disk space for two complete job-archives. If the tool is called without options: ``` $ ./archive-migration From 3766121aef2f5193423cafa75ca109aa5292f328 Mon Sep 17 00:00:00 2001 From: Jan Eitzinger Date: Tue, 13 Jun 2023 14:40:23 +0200 Subject: [PATCH 3/6] Fix test --- internal/config/config_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 8f089db..dc9d065 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -17,7 +17,7 @@ func TestInit(t *testing.T) { } func TestInitMinimal(t *testing.T) { - fp := "../../docs/config.json" + fp := "../../configs/config-demo.json" Init(fp) if Keys.Addr != "127.0.0.1:8080" { t.Errorf("wrong addr\ngot: %s \nwant: 127.0.0.1:8080", Keys.Addr) From 3b3a561ebf97f884e392e5649edd8ece38e16309 Mon Sep 17 00:00:00 2001 From: Jan Eitzinger Date: Tue, 13 Jun 2023 14:44:03 +0200 Subject: [PATCH 4/6] Update adm-upgrade.md --- docs/adm-upgrade.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/adm-upgrade.md b/docs/adm-upgrade.md index 57270ed..6a6763f 100644 --- a/docs/adm-upgrade.md +++ b/docs/adm-upgrade.md @@ -11,9 +11,9 @@ mismatch is detected the application will exit and a migration is required. **IMPORTANT NOTICE** -It is recommended to backup the database before any upgrade. This is mandatory -in case the database needs to be migrated. In case of sqlite this means to stop -`cc-backend` and copy the sqlite databse file somewhere. +It is recommended to backup the database before any upgrade. It is mandatory +to backup in case the database is migrated. Using sqlite this means to stop +`cc-backend` and copy the sqlite database file somewhere. # Migrating the database After backing up the database execute the the following command to migrate the From afadc30329950f319b02b234db9199c102168d32 Mon Sep 17 00:00:00 2001 From: Jan Eitzinger Date: Tue, 13 Jun 2023 14:49:00 +0200 Subject: [PATCH 5/6] Update adm-upgrade.md --- docs/adm-upgrade.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adm-upgrade.md b/docs/adm-upgrade.md index 57270ed..2ac39ee 100644 --- a/docs/adm-upgrade.md +++ b/docs/adm-upgrade.md @@ -16,13 +16,13 @@ in case the database needs to be migrated. In case of sqlite this means to stop `cc-backend` and copy the sqlite databse file somewhere. # Migrating the database -After backing up the database execute the the following command to migrate the +After doing a database backup execute the following command to migrate the databse to the most recent version: ``` $ ./cc-backend -migrate-db ``` -The migration files are embedded into the binary but can be reviewed in the +The migration files are embedded into the binary and can also be reviewed in the cc-backend [source tree](https://github.com/ClusterCockpit/cc-backend/tree/master/internal/repository/migrations). There are separate migration files for both supported database backends. We use the [migrate library](https://github.com/golang-migrate/migrate). From addeeea3957a77d06441ea674c6ac29aaca05c5a Mon Sep 17 00:00:00 2001 From: Jan Eitzinger Date: Tue, 13 Jun 2023 15:14:19 +0200 Subject: [PATCH 6/6] Update documentation and streamline language --- docs/ConfigurationManagement.md | 48 +++++++++---------- docs/adm-customization.md | 16 +++---- docs/adm-upgrade.md | 84 +++++++++++++++++---------------- web/frontend/README.md | 7 +-- 4 files changed, 78 insertions(+), 77 deletions(-) diff --git a/docs/ConfigurationManagement.md b/docs/ConfigurationManagement.md index 6c565a3..e36c136 100644 --- a/docs/ConfigurationManagement.md +++ b/docs/ConfigurationManagement.md @@ -1,37 +1,37 @@ # Release versioning -Releases are numbered with a integer id starting with 1. -Every release embeds the following assets into the binary: -* Web-frontend including javascript files and all static assets -* Golang template files for server-side rendering -* JSON schema files for validation +Releases are numbered with an integer ID, starting with 1. +Each release embeds the following assets in the binary: +* Web front-end with Javascript files and all static assets. +* Golang template files for server-side rendering. +* JSON schema files for validation. +* Database migration files -Remaining external assets are: +The remaining external assets are: * The SQL database used * The job archive +* The configuration file `config.json` -Both external assets are also versioned using integer ids. -This means every release binary is tied to specific versions for the SQL -database and job archive. -A command line switch `--migrate-db` is provided to migrate the SQL database -from a previous to the most recent version. -We provide a separate tool `archive-migration` to migrate an existing job -archive from the previous to the most recent version. +Both external assets are also versioned with integer IDs. +This means that each release binary is bound to specific versions of the SQL +database and the job archive. +The configuration file is validated against the current schema on startup. +The command line switch `-migrate-db` can be used to upgrade the SQL database +to migrate from a previous version to the latest one. +We offer a separate tool `archive-migration` to migrate an existing job archive +archive from the previous to the latest version. # Versioning of APIs + cc-backend provides two API backends: * A REST API for querying jobs -* A GraphQL API used for data exchange between web frontend and cc-backend +* A GraphQL API for data exchange between web frontend and cc-backend -Both APIs will also be versioned. We still need to decide if we also support -older REST API version using versioning of the endpoint URLs. - -# How to build a specific release - - -# How to migrate the SQL database - - -# How to migrate the job archive +Both APIs will also be versioned. We still need to decide wether we will also support +older REST API version by versioning the endpoint URLs. +# How to build +Please always build `cc-backend` with the supplied Makefile. This will ensure +that the frontend is also built correctly and that the version in the binary file is coded +in the binary. diff --git a/docs/adm-customization.md b/docs/adm-customization.md index 5c8801f..1f9b477 100644 --- a/docs/adm-customization.md +++ b/docs/adm-customization.md @@ -1,13 +1,13 @@ # Overview -To customize `cc-backend` means to change the logo and add specific legal texts -in place of the placeholders. To change the logo shown in the navigation bar the -file `web/frontend/public/img/logo.png` has to be replaced in the source tree -and cc-backend has to be rebuild. +Customizing `cc-backend` means changing the logo and certain legal texts +instead of the placeholders. To change the logo displayed in the navigation bar, the +file `web/frontend/public/img/logo.png` in the source tree must be replaced +and cc-backend must be rebuild. # Replace legal texts -To replace the legal texts `imprint.tmpl` and `privacy.tmpl` you can place your -version in `./var/`. On startup `cc-backend` will check if `./var/imprint.tmpl` and/or -`./var/privacy.tmpl` exist and use those instead of the builtin placeholders. -You may use the placeholders in `web/templates` as blueprint. +To replace the `imprint.tmpl` and `privacy.tmpl` legal texts, you can place your +version in `./var/`. At startup `cc-backend` will check if `./var/imprint.tmpl` and/or +`./var/privacy.tmpl` exist and use them instead of the built-in placeholders. +You can use the placeholders in `web/templates` as a blueprint. diff --git a/docs/adm-upgrade.md b/docs/adm-upgrade.md index 57270ed..bfe2933 100644 --- a/docs/adm-upgrade.md +++ b/docs/adm-upgrade.md @@ -1,33 +1,37 @@ -In general an upgrade is not more then replacing the binary. All files that are -required, apart from the database file, the configuration file and the job -archive are embedded into the binary. It is recommended to use a directory where -the binary filenames are named using some version tag. This may be, e.g., the date or -unix epoch time. A symbolic link points to the version to be used. This allows -to easily switch to previous versions. +In general, an upgrade is nothing more than a replacement of the binary file. +All the necessary files, except the database file, the configuration file and +the job archive, are embedded in the binary file. It is recommended to use a +directory where the file names of the binary files are named with a version +indicator. This can be, for example, the date or the Unix epoch time. A symbolic +link points to the version to be used. This makes it easier to switch to earlier +versions. -The database and the job archive are versioned. Every release binary supports -specific versions of the database and the job archive. In case a version -mismatch is detected the application will exit and a migration is required. +The database and the job archive are versioned. Each release binary supports +specific versions of the database and job archive. If a version mismatch is +detected, the application is terminated and migration is required. -**IMPORTANT NOTICE** +**IMPORTANT NOTE** -It is recommended to backup the database before any upgrade. This is mandatory -in case the database needs to be migrated. In case of sqlite this means to stop -`cc-backend` and copy the sqlite databse file somewhere. +It is recommended to make a backup copy of the database before each update. This +is mandatory in case the database needs to be migrated. In the case of sqlite, +this means to stopping `cc-backend` and copying the sqlite database file +somewhere. # Migrating the database -After backing up the database execute the the following command to migrate the -databse to the most recent version: + +After you have backed up the database, run the following command to migrate the +database to the latest version: ``` $ ./cc-backend -migrate-db ``` -The migration files are embedded into the binary but can be reviewed in the -cc-backend [source tree](https://github.com/ClusterCockpit/cc-backend/tree/master/internal/repository/migrations). -There are separate migration files for both supported database backends. +The migration files are embedded in the binary and can also be viewed in the cc +backend [source tree](https://github.com/ClusterCockpit/cc-backend/tree/master/internal/repository/migrations). +There are separate migration files for both supported +database backends. We use the [migrate library](https://github.com/golang-migrate/migrate). -In case something goes wrong you can check the state and get the current schema +If something goes wrong, you can check the status and get the current schema (here for sqlite): ``` $ sqlite3 var/job.db @@ -37,38 +41,38 @@ In the sqlite console execute: .schema ``` to get the current databse schema. -You can query the current version and if the migration failed with: +You can query the current version and whether the migration failed with: ``` SELECT * FROM schema_migrations; ``` The first column indicates the current database version and the second column is -a dirty flag indicating if the migration was successful. +a dirty flag indicating whether the migration was successful. # Migrating the job archive -To migrate the job archive a separate tool (`archive-migration`) is required that is part of the -`cc-backend` source tree (build with `go build ./tools/archive-migration`) and also provided as part of releases. +Job archive migration requires a separate tool (`archive-migration`), which is +part of the cc-backend source tree (build with `go build ./tools/archive-migration`) +and is also provided as part of the releases. -Migration is only supported between two subsequent versions. The migration tool -will migrate the existing job archive into a new job archive. This means there -has to be enough disk space for two complete job-archives. If the tool is -called without options: +Migration is supported only between two successive releases. The migration tool +migrates the existing job archive to a new job archive. This means that there +must be enough disk space for two complete job archives. If the tool is called +without options: ``` $ ./archive-migration ``` -it is assumed that there is a job archive in `./var/job-archive`. The new job -archive will be written to `./var/job-archive-new`. Because execution is -threaded in case of a fatal error it is impossible to pinpoint in which job the -error occured. In this case you can run the tool in debug mode (using the flag -`-debug`). In debug mode threading will be disabled and the job id of every -migrated job is printed. Jobs with empty files are skipped. Between multiple runs -of the tools the directory `job-archive-new` has to be moved or deleted. +it is assumed that a job archive exists in `./var/job-archive`. The new job +archive is written to `./var/job-archive-new`. Since execution is threaded in case +of a fatal error, it is impossible to determine in which job the error occurred. +In this case, you can run the tool in debug mode (with the `-debug` flag). In +debug mode, threading is disabled and the job ID of each migrated job is output. +Jobs with empty files will be skipped. Between multiple runs of the tools, the +`job-archive-new` directory must be moved or deleted. -The cluster.json files in `job-archive-new` need to be reviewed for errors, in -particular it has to be checked if the aggregation attribute is set correctly -for all metrics. +The `cluster.json` files in `job-archive-new` must be checked for errors, especially +whether the aggregation attribute is set correctly for all metrics. -The migration will take in the order of hours for fairly large (several hundred -GB) job archives. A versioned job archive contains a file `version.txt` in the -job archive root directory. This file contains the version as unsigned integer. +Migration takes several hours for relatively large job archives (several hundred +GB). A versioned job archive contains a version.txt file in the root directory +of the job archive. This file contains the version as an unsigned integer. diff --git a/web/frontend/README.md b/web/frontend/README.md index 0317b1a..d61d302 100644 --- a/web/frontend/README.md +++ b/web/frontend/README.md @@ -12,18 +12,15 @@ Builds on: ## Get started -[Yarn](https://yarnpkg.com/) is recommended for package management. -Due to an issue with Yarn v2 you have to stick to Yarn v1. - Install the dependencies... ```bash -yarn install +npm install ``` ...then build using [Rollup](https://rollupjs.org): ```bash -yarn build +npm run build ```