| .. | ||
| README.md | ||
File based archive specification for HPC jobs
This is a json files based exchange format for HPC job meta and performance metric data.
It consists of two parts:
- a sqlite database schema for job meta data and performance statistics
- a json file format together with a directory hierarchy specification
By using an open, portable and simple specification based on files it is possible to exchange job performance data for research and analysis purposes as well as a robust way for archiving job performance data on disk.
Directory hierarchy and file specification
The job archive has top-level directories named after the clusters. In every
cluster directory there must be one file named cluster.json describing the
cluster. The json schema for this file is described here. Within this directory
a three-level directory tree is used to organize job files.
To manage the number of directories within a single directory a tree approach is used splitting the integer job ID. The job id is split in junks of 1000 each.
For a 2 layer schema this can be achieved with (code example in Perl):
$level1 = $jobID/1000;
$level2 = $jobID%1000;
$dstPath = sprintf("%s/%s/%d/%03d", $trunk, $destdir, $level1, $level2);
The last directory level is the unix epoch timestamp in seconds to allow for overflowing job ids.
Example:
For the job ID 1034871 the directory path is ./1034/871/<timestamp>/.
The job data consists of two files:
- meta.json: Contains job meta information and job statistics.
- data.json: Contains complete job data with time series
The description of the json format specification is available as json schema.
Metric time series data is stored with fixed time step. The time step can be set per metric. If no value is available for a metric time series data timestamp null must be entered.