cc-backend/docs/adm-upgrade.md
2023-06-13 14:44:03 +02:00

3.3 KiB

In general an upgrade is not more then replacing the binary. All files that are required, apart from the database file, the configuration file and the job archive are embedded into the binary. It is recommended to use a directory where the binary filenames are named using some version tag. This may be, e.g., the date or unix epoch time. A symbolic link points to the version to be used. This allows to easily switch to previous versions.

The database and the job archive are versioned. Every release binary supports specific versions of the database and the job archive. In case a version mismatch is detected the application will exit and a migration is required.

IMPORTANT NOTICE

It is recommended to backup the database before any upgrade. It is mandatory to backup in case the database is migrated. Using sqlite this means to stop cc-backend and copy the sqlite database file somewhere.

Migrating the database

After backing up the database execute the the following command to migrate the databse to the most recent version:

$ ./cc-backend -migrate-db

The migration files are embedded into the binary but can be reviewed in the cc-backend source tree. There are separate migration files for both supported database backends. We use the migrate library.

In case something goes wrong you can check the state and get the current schema (here for sqlite):

$ sqlite3 var/job.db

In the sqlite console execute:

.schema

to get the current databse schema. You can query the current version and if the migration failed with:

SELECT * FROM schema_migrations;

The first column indicates the current database version and the second column is a dirty flag indicating if the migration was successful.

Migrating the job archive

To migrate the job archive a separate tool (archive-migration) is required that is part of the cc-backend source tree (build with go build ./tools/archive-migration) and also provided as part of releases.

Migration is only supported between two subsequent versions. The migration tool will migrate the existing job archive into a new job archive. This means there has to be enough disk space for two complete job-archives. If the tool is called without options:

$ ./archive-migration

it is assumed that there is a job archive in ./var/job-archive. The new job archive will be written to ./var/job-archive-new. Because execution is threaded in case of a fatal error it is impossible to pinpoint in which job the error occured. In this case you can run the tool in debug mode (using the flag -debug). In debug mode threading will be disabled and the job id of every migrated job is printed. Jobs with empty files are skipped. Between multiple runs of the tools the directory job-archive-new has to be moved or deleted.

The cluster.json files in job-archive-new need to be reviewed for errors, in particular it has to be checked if the aggregation attribute is set correctly for all metrics.

The migration will take in the order of hours for fairly large (several hundred GB) job archives. A versioned job archive contains a file version.txt in the job archive root directory. This file contains the version as unsigned integer.