The archive-manager tool manages ClusterCockpit job archives. It supports inspecting archives, validating jobs, removing jobs by date range, importing jobs between archive backends, and converting archives between JSON and Parquet formats.

Features

Archive Info: Display statistics about an existing job archive
Validation: Validate job archives against the JSON schema
Cleanup: Remove jobs by date range
Import: Copy jobs between archive backends (file, S3, SQLite) with parallel processing
Convert: Convert archives between JSON and Parquet formats (both directions)
Progress Reporting: Real-time progress display with ETA and throughput metrics
Graceful Interruption: CTRL-C stops processing after finishing current jobs

Usage

Build

go build ./tools/archive-manager/

Archive Info

Display statistics about a job archive:

./archive-manager -s ./var/job-archive

Validate Archive

./archive-manager -s ./var/job-archive --validate --config ./config.json

Remove Jobs by Date

# Remove jobs started before a date
./archive-manager -s ./var/job-archive --remove-before 2023-Jan-01 --config ./config.json

# Remove jobs started after a date
./archive-manager -s ./var/job-archive --remove-after 2024-Dec-31 --config ./config.json

Import Between Backends

Import jobs from one archive backend to another (e.g., file to S3, file to SQLite):

./archive-manager --import \
  --src-config '{"kind":"file","path":"./var/job-archive"}' \
  --dst-config '{"kind":"s3","endpoint":"https://s3.example.com","bucket":"archive","access-key":"...","secret-key":"..."}'

Convert JSON to Parquet

Convert a JSON job archive to Parquet format:

./archive-manager --convert --format parquet \
  --src-config '{"kind":"file","path":"./var/job-archive"}' \
  --dst-config '{"kind":"file","path":"./var/parquet-archive"}'

The source (--src-config) is a standard archive backend config (file, S3, or SQLite). The destination (--dst-config) specifies where to write parquet files.

Convert Parquet to JSON

Convert a Parquet archive back to JSON format:

./archive-manager --convert --format json \
  --src-config '{"kind":"file","path":"./var/parquet-archive"}' \
  --dst-config '{"kind":"file","path":"./var/json-archive"}'

The source (--src-config) points to a directory or S3 bucket containing parquet files organized by cluster. The destination (--dst-config) is a standard archive backend config.

S3 Source/Destination Example

Both conversion directions support S3:

# JSON (S3) -> Parquet (local)
./archive-manager --convert --format parquet \
  --src-config '{"kind":"s3","endpoint":"https://s3.example.com","bucket":"json-archive","accessKey":"...","secretKey":"..."}' \
  --dst-config '{"kind":"file","path":"./var/parquet-archive"}'

# Parquet (local) -> JSON (S3)
./archive-manager --convert --format json \
  --src-config '{"kind":"file","path":"./var/parquet-archive"}' \
  --dst-config '{"kind":"s3","endpoint":"https://s3.example.com","bucket":"json-archive","access-key":"...","secret-key":"..."}'

Command-Line Options

Flag	Default	Description
`-s`	`./var/job-archive`	Source job archive path (for info/validate/remove modes)
`--config`	`./config.json`	Path to config.json
`--loglevel`	`info`	Logging level: debug, info, warn, err, fatal, crit
`--logdate`	`false`	Add timestamps to log messages
`--validate`	`false`	Validate archive against JSON schema
`--remove-before`		Remove jobs started before date (Format: 2006-Jan-02)
`--remove-after`		Remove jobs started after date (Format: 2006-Jan-02)
`--import`	`false`	Import jobs between archive backends
`--convert`	`false`	Convert archive between JSON and Parquet formats
`--format`	`json`	Output format for conversion: `json` or `parquet`
`--max-file-size`	`512`	Max parquet file size in MB (only for parquet output)
`--src-config`		Source config JSON (required for import/convert)
`--dst-config`		Destination config JSON (required for import/convert)

Parquet Archive Layout

When converting to Parquet, the output is organized by cluster:

parquet-archive/
  clusterA/
    cluster.json
    cc-archive-2025-01-20-001.parquet
    cc-archive-2025-01-20-002.parquet
  clusterB/
    cluster.json
    cc-archive-2025-01-20-001.parquet

Each parquet file contains job metadata and gzip-compressed metric data. The cluster.json file preserves the cluster configuration from the source archive.

Round-Trip Conversion

Archives can be converted from JSON to Parquet and back without data loss:

# Original JSON archive
./archive-manager --convert --format parquet \
  --src-config '{"kind":"file","path":"./var/job-archive"}' \
  --dst-config '{"kind":"file","path":"./var/parquet-archive"}'

# Convert back to JSON
./archive-manager --convert --format json \
  --src-config '{"kind":"file","path":"./var/parquet-archive"}' \
  --dst-config '{"kind":"file","path":"./var/json-archive"}'