# HPCJobDatabase A standardized interface and reference implementation for HPC job data # Dependencies * Getopt::Long * Pod::Usage * DateTime::Format::Strptime * DBD::SQLite # Setup ``` sqlite3 jobDB < initDB.sql ``` # Helper Scripts For all scripts apart from `acQuery.pl` the advice *use the source Luke* holds. Help text for acQuery: ``` Usage: acQuery.pl [options] -- Help Options: --help Show help text --man Show man page --hasprofile Only show jobs with timerseries metric data --mode Set the operation mode --user Search for jobs of specific user --project Search for jobs of specific project --numnodes Specify range for number of nodes of job --starttime Specify range for start time of jobs --duration Specify duration range of jobs --mem_used Specify range for average main memory capacity of job --mem_bandwidth Specify range for average main memory bandwidth of job --flops_any Specify range for average flop any rate of job Options: --help Show a brief help information. --man Read the manual, with examples --hasprofile [true|false] Only show jobs with or without timerseries metric data --mode [ids|query|count|list|stat|perf] Specify output mode. Mode can be one of: ids - Print list of job ids matching conditions. One job id per line. query - Print the query string and then exit. count - Only output the number of jobs matching the conditions. (Default mode) list - Output a record of every job matching the conditions. stat - Output job statistic for all jobs matching the conditions. perf - Output job performance footprint statistic for all jobs matching the conditions. --user Search job for a specific user id. --project Search job for a specific project. --duration Specify condition for job duration. This option takes two arguments: If both arguments are positive integers the condition is duration between first argument and second argument. If the second argument is zero condition is duration smaller than first argument. If first argument is zero condition is duration larger than second argument. Duration can be in seconds, minutes (append m) or hours (append h). --numnodes Specify condition for number of node range of job. This option takes two arguments: If both arguments are positive integers the condition is number of nodes between first argument and second argument. If the second argument is zero condition is number of nodes smaller than first argument. If first argument is zero condition is number of nodes larger than second argument. --starttime Specify condition for the starttime of job. This option takes two arguments: If both arguments are positive integers the condition is start time between first argument and second argument. If the second argument is zero condition is start time smaller than first argument. If first argument is zero condition is start time larger than second argument. Start time must be given as date in the following format: %d.%m.%Y/%H:%M. --mem_used Specify condition for average main memory capacity used by job. This option takes two arguments: If both arguments are positive integers the condition is memory used is between first argument and second argument. If the second argument is zero condition is memory used is smaller than first argument. If first argument is zero condition is memory used is larger than second argument. --mem_bandwidth Specify condition for average main memory bandwidth used by job. This option takes two arguments: If both arguments are positive integers the condition is memory bandwidth is between first argument and second argument. If the second argument is zero condition is memory bandwidth is smaller than first argument. If first argument is zero condition is memory bandwidth is larger than second argument. --flops_any Specify condition for average flops any of job. This option takes two arguments: If both arguments are positive integers the condition is flops any is between first argument and second argument. If the second argument is zero condition is flops any is smaller than first argument. If first argument is zero condition is flops any is larger than second argument. ``` # Examples Query jobs with conditions: ``` [HPCJobDatabase] ./acQuery.pl --duration 20h 24h --starttime 01.08.2018/12:00 01.03.2019/12:00 COUNT 6476 ``` Query jobs from alternative database file (default is jobDB): ``` [HPCJobDatabase] ./acQuery.pl --project project_30 --starttime 01.08.2018/12:00 01.03.2019/12:00 -- jobDB-anon-emmy COUNT 21560 ``` Get job statistics output: ``` [HPCJobDatabase] ./acQuery.pl --project project_30 --mode stat --duration 0 20h --starttime 01.08.2018/12:00 01.03.2019/12:00 -- jobDB-anon-emmy ================================= Job count: 747 Total walltime [h]: 16334 Total node hours [h]: 78966 Histogram: Number of nodes nodes count 1 54 **** 2 1 3 1 4 36 **** 5 522 ******* 6 118 ***** 7 15 *** Histogram: Walltime hours count 20 250 ****** 21 200 ****** 22 114 ***** 23 183 ****** ``` Get job performance statistics: ``` [HPCJobDatabase] ./acQuery.pl --project project_30 --mode perf --duration 0 20h --numnodes 1 4 --starttime 01.08.2018/12:00 01.03.2019/12:00 -- jobDB-anon-emmy ================================= Job count: 92 Jobs with performance profile: 48 Total walltime [h]: 2070 Total node hours [h]: 4332 Histogram: Mem used Mem count 2 3 ** 3 4 ** 18 2 * 19 3 ** 20 2 * 21 1 22 2 * 23 5 ** 24 2 * 25 1 26 1 27 3 ** 29 1 30 2 * 31 1 34 1 35 1 36 1 41 1 42 2 * 43 2 * 44 1 49 1 50 2 * 51 1 52 1 53 1 Histogram: Memory bandwidth BW count 1 1 2 9 *** 3 1 4 1 5 4 ** 6 2 * 7 10 *** 8 9 *** 9 11 *** Histogram: Flops any flops count 1 3 ** 2 1 3 4 ** 4 3 ** 5 9 *** 6 10 *** 7 11 *** 85 1 225 1 236 1 240 2 * 244 2 * ```