diff --git a/utils/README.md b/utils/README.md index dfafba6..fabc774 100644 --- a/utils/README.md +++ b/utils/README.md @@ -1,226 +1 @@ -# HPCJobDatabase -A standardized interface and reference implementation for HPC job data. -The DB and json schema specification is available in the [wiki](https://github.com/RRZE-HPC/HPCJobDatabase/wiki). - -# Dependencies - - * Getopt::Long - * Pod::Usage - * DateTime::Format::Strptime - * DBD::SQLite - -# Setup - -``` -sqlite3 jobDB < initDB.sql -``` - -# Helper Scripts - -For all scripts apart from `acQuery.pl` the advice *use the source Luke* holds. - -Help text for acQuery: -``` -Usage: - acQuery.pl [options] -- - - Help Options: - --help Show help text - --man Show man page - --hasprofile Only show jobs with timerseries metric data - --mode Set the operation mode - --user Search for jobs of specific user - --project Search for jobs of specific project - --numnodes Specify range for number of nodes of job - --starttime Specify range for start time of jobs - --duration Specify duration range of jobs - --mem_used Specify range for average main memory capacity of job - --mem_bandwidth Specify range for average main memory bandwidth of job - --flops_any Specify range for average flop any rate of job - -Options: - --help Show a brief help information. - - --man Read the manual, with examples - - --hasprofile [true|false] Only show jobs with or without timerseries - metric data - - --mode [ids|query|count|list|stat|perf] Specify output mode. Mode can be - one of: - - ids - Print list of job ids matching conditions. One job id per - line. - - query - Print the query string and then exit. - count - Only output the number of jobs matching the conditions. - (Default mode) - - list - Output a record of every job matching the conditions. - - stat - Output job statistic for all jobs matching the - conditions. - - perf - Output job performance footprint statistic for all jobs - matching the conditions. - - --user Search job for a specific user id. - - --project Search job for a specific project. - - --duration Specify condition for job duration. This option takes two - arguments: If both arguments are positive integers the condition is - duration between first argument and second argument. If the second - argument is zero condition is duration smaller than first argument. If - first argument is zero condition is duration larger than second - argument. Duration can be in seconds, minutes (append m) or hours - (append h). - - --numnodes Specify condition for number of node range of job. This - option takes two arguments: If both arguments are positive integers the - condition is number of nodes between first argument and second argument. - If the second argument is zero condition is number of nodes smaller than - first argument. If first argument is zero condition is number of nodes - larger than second argument. - - --starttime Specify condition for the starttime of job. This option - takes two arguments: If both arguments are positive integers the - condition is start time between first argument and second argument. If - the second argument is zero condition is start time smaller than first - argument. If first argument is zero condition is start time larger than - second argument. Start time must be given as date in the following - format: %d.%m.%Y/%H:%M. - - --mem_used Specify condition for average main memory capacity used by - job. This option takes two arguments: If both arguments are positive - integers the condition is memory used is between first argument and - second argument. If the second argument is zero condition is memory used - is smaller than first argument. If first argument is zero condition is - memory used is larger than second argument. - - --mem_bandwidth Specify condition for average main memory bandwidth used - by job. This option takes two arguments: If both arguments are positive - integers the condition is memory bandwidth is between first argument and - second argument. If the second argument is zero condition is memory - bandwidth is smaller than first argument. If first argument is zero - condition is memory bandwidth is larger than second argument. - - --flops_any Specify condition for average flops any of job. This option - takes two arguments: If both arguments are positive integers the - condition is flops any is between first argument and second argument. If - the second argument is zero condition is flops any is smaller than first - argument. If first argument is zero condition is flops any is larger - than second argument. - -``` - -# Examples - -Query jobs with conditions: - -``` -[HPCJobDatabase] ./acQuery.pl --duration 20h 24h --starttime 01.08.2018/12:00 01.03.2019/12:00 -COUNT 6476 -``` - -Query jobs from alternative database file (default is jobDB): - -``` -[HPCJobDatabase] ./acQuery.pl --project project_30 --starttime 01.08.2018/12:00 01.03.2019/12:00 -- jobDB-anon-emmy -COUNT 21560 -``` - -Get job statistics output: - -``` -[HPCJobDatabase] ./acQuery.pl --project project_30 --mode stat --duration 0 20h --starttime 01.08.2018/12:00 01.03.2019/12:00 -- jobDB-anon-emmy -================================= -Job count: 747 -Total walltime [h]: 16334 -Total node hours [h]: 78966 - -Histogram: Number of nodes -nodes count -1 54 **** -2 1 -3 1 -4 36 **** -5 522 ******* -6 118 ***** -7 15 *** - -Histogram: Walltime -hours count -20 250 ****** -21 200 ****** -22 114 ***** -23 183 ****** -``` - -Get job performance statistics: - -``` -[HPCJobDatabase] ./acQuery.pl --project project_30 --mode perf --duration 0 20h --numnodes 1 4 --starttime 01.08.2018/12:00 01.03.2019/12:00 -- jobDB-anon-emmy -================================= -Job count: 92 -Jobs with performance profile: 48 -Total walltime [h]: 2070 -Total node hours [h]: 4332 - -Histogram: Mem used -Mem count -2 3 ** -3 4 ** -18 2 * -19 3 ** -20 2 * -21 1 -22 2 * -23 5 ** -24 2 * -25 1 -26 1 -27 3 ** -29 1 -30 2 * -31 1 -34 1 -35 1 -36 1 -41 1 -42 2 * -43 2 * -44 1 -49 1 -50 2 * -51 1 -52 1 -53 1 - -Histogram: Memory bandwidth -BW count -1 1 -2 9 *** -3 1 -4 1 -5 4 ** -6 2 * -7 10 *** -8 9 *** -9 11 *** - -Histogram: Flops any -flops count -1 3 ** -2 1 -3 4 ** -4 3 ** -5 9 *** -6 10 *** -7 11 *** -85 1 -225 1 -236 1 -240 2 * -244 2 * -``` +# Helper scripts