From afadc30329950f319b02b234db9199c102168d32 Mon Sep 17 00:00:00 2001 From: Jan Eitzinger Date: Tue, 13 Jun 2023 14:49:00 +0200 Subject: [PATCH 1/2] Update adm-upgrade.md --- docs/adm-upgrade.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adm-upgrade.md b/docs/adm-upgrade.md index 57270ed..2ac39ee 100644 --- a/docs/adm-upgrade.md +++ b/docs/adm-upgrade.md @@ -16,13 +16,13 @@ in case the database needs to be migrated. In case of sqlite this means to stop `cc-backend` and copy the sqlite databse file somewhere. # Migrating the database -After backing up the database execute the the following command to migrate the +After doing a database backup execute the following command to migrate the databse to the most recent version: ``` $ ./cc-backend -migrate-db ``` -The migration files are embedded into the binary but can be reviewed in the +The migration files are embedded into the binary and can also be reviewed in the cc-backend [source tree](https://github.com/ClusterCockpit/cc-backend/tree/master/internal/repository/migrations). There are separate migration files for both supported database backends. We use the [migrate library](https://github.com/golang-migrate/migrate). From addeeea3957a77d06441ea674c6ac29aaca05c5a Mon Sep 17 00:00:00 2001 From: Jan Eitzinger Date: Tue, 13 Jun 2023 15:14:19 +0200 Subject: [PATCH 2/2] Update documentation and streamline language --- docs/ConfigurationManagement.md | 48 +++++++++---------- docs/adm-customization.md | 16 +++---- docs/adm-upgrade.md | 84 +++++++++++++++++---------------- web/frontend/README.md | 7 +-- 4 files changed, 78 insertions(+), 77 deletions(-) diff --git a/docs/ConfigurationManagement.md b/docs/ConfigurationManagement.md index 6c565a3..e36c136 100644 --- a/docs/ConfigurationManagement.md +++ b/docs/ConfigurationManagement.md @@ -1,37 +1,37 @@ # Release versioning -Releases are numbered with a integer id starting with 1. -Every release embeds the following assets into the binary: -* Web-frontend including javascript files and all static assets -* Golang template files for server-side rendering -* JSON schema files for validation +Releases are numbered with an integer ID, starting with 1. +Each release embeds the following assets in the binary: +* Web front-end with Javascript files and all static assets. +* Golang template files for server-side rendering. +* JSON schema files for validation. +* Database migration files -Remaining external assets are: +The remaining external assets are: * The SQL database used * The job archive +* The configuration file `config.json` -Both external assets are also versioned using integer ids. -This means every release binary is tied to specific versions for the SQL -database and job archive. -A command line switch `--migrate-db` is provided to migrate the SQL database -from a previous to the most recent version. -We provide a separate tool `archive-migration` to migrate an existing job -archive from the previous to the most recent version. +Both external assets are also versioned with integer IDs. +This means that each release binary is bound to specific versions of the SQL +database and the job archive. +The configuration file is validated against the current schema on startup. +The command line switch `-migrate-db` can be used to upgrade the SQL database +to migrate from a previous version to the latest one. +We offer a separate tool `archive-migration` to migrate an existing job archive +archive from the previous to the latest version. # Versioning of APIs + cc-backend provides two API backends: * A REST API for querying jobs -* A GraphQL API used for data exchange between web frontend and cc-backend +* A GraphQL API for data exchange between web frontend and cc-backend -Both APIs will also be versioned. We still need to decide if we also support -older REST API version using versioning of the endpoint URLs. - -# How to build a specific release - - -# How to migrate the SQL database - - -# How to migrate the job archive +Both APIs will also be versioned. We still need to decide wether we will also support +older REST API version by versioning the endpoint URLs. +# How to build +Please always build `cc-backend` with the supplied Makefile. This will ensure +that the frontend is also built correctly and that the version in the binary file is coded +in the binary. diff --git a/docs/adm-customization.md b/docs/adm-customization.md index 5c8801f..1f9b477 100644 --- a/docs/adm-customization.md +++ b/docs/adm-customization.md @@ -1,13 +1,13 @@ # Overview -To customize `cc-backend` means to change the logo and add specific legal texts -in place of the placeholders. To change the logo shown in the navigation bar the -file `web/frontend/public/img/logo.png` has to be replaced in the source tree -and cc-backend has to be rebuild. +Customizing `cc-backend` means changing the logo and certain legal texts +instead of the placeholders. To change the logo displayed in the navigation bar, the +file `web/frontend/public/img/logo.png` in the source tree must be replaced +and cc-backend must be rebuild. # Replace legal texts -To replace the legal texts `imprint.tmpl` and `privacy.tmpl` you can place your -version in `./var/`. On startup `cc-backend` will check if `./var/imprint.tmpl` and/or -`./var/privacy.tmpl` exist and use those instead of the builtin placeholders. -You may use the placeholders in `web/templates` as blueprint. +To replace the `imprint.tmpl` and `privacy.tmpl` legal texts, you can place your +version in `./var/`. At startup `cc-backend` will check if `./var/imprint.tmpl` and/or +`./var/privacy.tmpl` exist and use them instead of the built-in placeholders. +You can use the placeholders in `web/templates` as a blueprint. diff --git a/docs/adm-upgrade.md b/docs/adm-upgrade.md index 57270ed..bfe2933 100644 --- a/docs/adm-upgrade.md +++ b/docs/adm-upgrade.md @@ -1,33 +1,37 @@ -In general an upgrade is not more then replacing the binary. All files that are -required, apart from the database file, the configuration file and the job -archive are embedded into the binary. It is recommended to use a directory where -the binary filenames are named using some version tag. This may be, e.g., the date or -unix epoch time. A symbolic link points to the version to be used. This allows -to easily switch to previous versions. +In general, an upgrade is nothing more than a replacement of the binary file. +All the necessary files, except the database file, the configuration file and +the job archive, are embedded in the binary file. It is recommended to use a +directory where the file names of the binary files are named with a version +indicator. This can be, for example, the date or the Unix epoch time. A symbolic +link points to the version to be used. This makes it easier to switch to earlier +versions. -The database and the job archive are versioned. Every release binary supports -specific versions of the database and the job archive. In case a version -mismatch is detected the application will exit and a migration is required. +The database and the job archive are versioned. Each release binary supports +specific versions of the database and job archive. If a version mismatch is +detected, the application is terminated and migration is required. -**IMPORTANT NOTICE** +**IMPORTANT NOTE** -It is recommended to backup the database before any upgrade. This is mandatory -in case the database needs to be migrated. In case of sqlite this means to stop -`cc-backend` and copy the sqlite databse file somewhere. +It is recommended to make a backup copy of the database before each update. This +is mandatory in case the database needs to be migrated. In the case of sqlite, +this means to stopping `cc-backend` and copying the sqlite database file +somewhere. # Migrating the database -After backing up the database execute the the following command to migrate the -databse to the most recent version: + +After you have backed up the database, run the following command to migrate the +database to the latest version: ``` $ ./cc-backend -migrate-db ``` -The migration files are embedded into the binary but can be reviewed in the -cc-backend [source tree](https://github.com/ClusterCockpit/cc-backend/tree/master/internal/repository/migrations). -There are separate migration files for both supported database backends. +The migration files are embedded in the binary and can also be viewed in the cc +backend [source tree](https://github.com/ClusterCockpit/cc-backend/tree/master/internal/repository/migrations). +There are separate migration files for both supported +database backends. We use the [migrate library](https://github.com/golang-migrate/migrate). -In case something goes wrong you can check the state and get the current schema +If something goes wrong, you can check the status and get the current schema (here for sqlite): ``` $ sqlite3 var/job.db @@ -37,38 +41,38 @@ In the sqlite console execute: .schema ``` to get the current databse schema. -You can query the current version and if the migration failed with: +You can query the current version and whether the migration failed with: ``` SELECT * FROM schema_migrations; ``` The first column indicates the current database version and the second column is -a dirty flag indicating if the migration was successful. +a dirty flag indicating whether the migration was successful. # Migrating the job archive -To migrate the job archive a separate tool (`archive-migration`) is required that is part of the -`cc-backend` source tree (build with `go build ./tools/archive-migration`) and also provided as part of releases. +Job archive migration requires a separate tool (`archive-migration`), which is +part of the cc-backend source tree (build with `go build ./tools/archive-migration`) +and is also provided as part of the releases. -Migration is only supported between two subsequent versions. The migration tool -will migrate the existing job archive into a new job archive. This means there -has to be enough disk space for two complete job-archives. If the tool is -called without options: +Migration is supported only between two successive releases. The migration tool +migrates the existing job archive to a new job archive. This means that there +must be enough disk space for two complete job archives. If the tool is called +without options: ``` $ ./archive-migration ``` -it is assumed that there is a job archive in `./var/job-archive`. The new job -archive will be written to `./var/job-archive-new`. Because execution is -threaded in case of a fatal error it is impossible to pinpoint in which job the -error occured. In this case you can run the tool in debug mode (using the flag -`-debug`). In debug mode threading will be disabled and the job id of every -migrated job is printed. Jobs with empty files are skipped. Between multiple runs -of the tools the directory `job-archive-new` has to be moved or deleted. +it is assumed that a job archive exists in `./var/job-archive`. The new job +archive is written to `./var/job-archive-new`. Since execution is threaded in case +of a fatal error, it is impossible to determine in which job the error occurred. +In this case, you can run the tool in debug mode (with the `-debug` flag). In +debug mode, threading is disabled and the job ID of each migrated job is output. +Jobs with empty files will be skipped. Between multiple runs of the tools, the +`job-archive-new` directory must be moved or deleted. -The cluster.json files in `job-archive-new` need to be reviewed for errors, in -particular it has to be checked if the aggregation attribute is set correctly -for all metrics. +The `cluster.json` files in `job-archive-new` must be checked for errors, especially +whether the aggregation attribute is set correctly for all metrics. -The migration will take in the order of hours for fairly large (several hundred -GB) job archives. A versioned job archive contains a file `version.txt` in the -job archive root directory. This file contains the version as unsigned integer. +Migration takes several hours for relatively large job archives (several hundred +GB). A versioned job archive contains a version.txt file in the root directory +of the job archive. This file contains the version as an unsigned integer. diff --git a/web/frontend/README.md b/web/frontend/README.md index 0317b1a..d61d302 100644 --- a/web/frontend/README.md +++ b/web/frontend/README.md @@ -12,18 +12,15 @@ Builds on: ## Get started -[Yarn](https://yarnpkg.com/) is recommended for package management. -Due to an issue with Yarn v2 you have to stick to Yarn v1. - Install the dependencies... ```bash -yarn install +npm install ``` ...then build using [Rollup](https://rollupjs.org): ```bash -yarn build +npm run build ```