cc-backend/docs/Hands-on.md

CC-HANDSON - Setup ClusterCockpit from scratch (w/o docker)

Prerequisites

• Perl
• Yarn
• Go
• Optional: curl
• Script migrateTimestamp.pl

Documentation

You find READMEs or api docs in

• ./cc-backend/configs
• ./cc-backend/init
• ./cc-backend/api

ClusterCockpit configuration files

cc-backend

• ./.env Passwords and Tokens set in the environment
• ./config.json Configuration options for cc-backend

cc-metric-store

• ./config.json Optional to overwrite configuration options

cc-metric-collector

Not yet included in the hands-on setup.

Setup Components

Start by creating a base folder for all of the following steps.

• mkdir clustercockpit
• cd clustercockpit

Setup cc-backend

• Clone Repository
  • git clone https://github.com/ClusterCockpit/cc-backend.git
  • cd cc-backend
  • git checkout dev-job-archive-module Will be merged soon into master
• Setup Frontend
  • cd ./web/frontend
  • yarn install
  • yarn build
  • cd ../..
• Build Go Executable
  • go build ./cmd/cc-backend/
• Prepare Datafolder and Database file
  • mkdir var
  • touch var/job.db
• Activate & Config environment for cc-backend
  • cp configs/env-template.txt .env
  • Optional: Have a look via vim ./.env
  • Copy the config.json file included in this tarball into the root directory of cc-backend: cp ../../config.json ./
• Back to toplevel clustercockpit
  • cd ..

Setup cc-metric-store

• Clone Repository
  • git clone https://github.com/ClusterCockpit/cc-metric-store.git
  • cd cc-metric-store
• Build Go Executable
  • go get
  • go build
• Prepare Datafolders
  • mkdir -p var/checkpoints
  • mkdir -p var/archive
• Update Config
  • vim config.json
  • Exchange existing setting in metrics with the following:

"clock":      { "frequency": 60, "aggregation": null },
"cpi":        { "frequency": 60, "aggregation": null },
"cpu_load":   { "frequency": 60, "aggregation": null },
"flops_any":  { "frequency": 60, "aggregation": null },
"flops_dp":   { "frequency": 60, "aggregation": null },
"flops_sp":   { "frequency": 60, "aggregation": null },
"ib_bw":      { "frequency": 60, "aggregation": null },
"lustre_bw":  { "frequency": 60, "aggregation": null },
"mem_bw":     { "frequency": 60, "aggregation": null },
"mem_used":   { "frequency": 60, "aggregation": null },
"rapl_power": { "frequency": 60, "aggregation": null }

• Back to toplevel clustercockpit
  • cd ..

Setup Demo Data

• mkdir source-data
• cd source-data
• Download JobArchive-Source:
  • wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/job-archive-dev.tar.xz
  • tar xJf job-archive-dev.tar.xz
  • mv ./job-archive ./job-archive-source
  • rm ./job-archive-dev.tar.xz
• Download CC-Metric-Store Checkpoints:
  • mkdir -p cc-metric-store-source/checkpoints
  • cd cc-metric-store-source/checkpoints
  • wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/cc-metric-store-checkpoints.tar.xz
  • tar xf cc-metric-store-checkpoints.tar.xz
  • rm cc-metric-store-checkpoints.tar.xz
• Back to source-data
  • cd ../..
• Run timestamp migration script. This may take tens of minutes!
  • cp ../migrateTimestamps.pl .
  • ./migrateTimestamps.pl
  • Expected output:

Starting to update start- and stoptimes in job-archive for emmy
Starting to update start- and stoptimes in job-archive for woody
Done for job-archive
Starting to update checkpoint filenames and data starttimes for emmy
Starting to update checkpoint filenames and data starttimes for woody
Done for checkpoints

• Copy cluster.json files from source to migrated folders
  • cp source-data/job-archive-source/emmy/cluster.json cc-backend/var/job-archive/emmy/
  • cp source-data/job-archive-source/woody/cluster.json cc-backend/var/job-archive/woody/
• Initialize Job-Archive in SQLite3 job.db and add demo user
  • cd cc-backend
  • ./cc-backend --init-db --add-user demo:admin:AdminDev
  • Expected output:

<6>[INFO]    new user "demo" created (roles: ["admin"], auth-source: 0)
<6>[INFO]    Building job table...
<6>[INFO]    A total of 3936 jobs have been registered in 1.791 seconds.

• Back to toplevel clustercockpit
  • cd ..

Startup both Apps

• In cc-backend root: $./cc-backend --server --dev
  • Starts Clustercockpit at http:localhost:8080
    • Log: <6>[INFO] HTTP server listening at :8080...
  • Use local internet browser to access interface
    • You should see and be able to browse finished Jobs
    • Metadata is read from SQLite3 database
    • Metricdata is read from job-archive/JSON-Files
  • Create User in settings (top-right corner)
    • Name apiuser
    • Username apiuser
    • Role API
    • Submit & Refresh Page
  • Create JTW for apiuser
    • In Userlist, press Gen. JTW for apiuser
    • Save JWT for later use
• In cc-metric-store root: $./cc-metric-store
  • Start the cc-metric-store on http:localhost:8081, Log:

2022/07/15 17:17:42 Loading checkpoints newer than 2022-07-13T17:17:42+02:00
2022/07/15 17:17:45 Checkpoints loaded (5621 files, 319 MB, that took 3.034652s)
2022/07/15 17:17:45 API http endpoint listening on '0.0.0.0:8081'

- Does *not* have a graphical interface
- Otpional: Test function by executing:

$ curl -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJFZERTQSJ9.eyJ1c2VyIjoiYWRtaW4iLCJyb2xlcyI6WyJST0xFX0FETUlOIiwiUk9MRV9BTkFMWVNUIiwiUk9MRV9VU0VSIl19.d-3_3FZTsadPjDEdsWrrQ7nS0edMAR4zjl-eK7rJU3HziNBfI9PDHDIpJVHTNN5E5SlLGLFXctWyKAkwhXL-Dw" -D - "http://localhost:8081/api/query" -d "{ \"cluster\": \"emmy\", \"from\": $(expr $(date +%s) - 60), \"to\": $(date +%s), \"queries\": [{
  \"metric\": \"flops_any\",
  \"host\": \"e1111\"
}] }"

HTTP/1.1 200 OK
Content-Type: application/json
Date: Fri, 15 Jul 2022 13:57:22 GMT
Content-Length: 119
{"results":[[JSON-DATA-ARRAY]]}

Development API web interfaces

The --dev flag enables web interfaces to document and test the apis:

• http://localhost:8080/playground - A GraphQL playground. To use it you must have a authenticated session in the same browser.
• http://localhost:8080/swagger - A Swagger UI. To use it you have to be logged out, so no user session in the same browser. Use the JWT token with role Api generate previously to authenticate via http header.

Use cc-backend API to start job

• Enter the URL http://localhost:8080/swagger/index.html in your browser.
• Enter your JWT token you generated for the API user by clicking the green Authorize button in the upper right part of the window.
• Click the /job/start_job endpoint and click the Try it out button.
• Enter the following json into the request body text area and fill in a recent start timestamp by executing date +%s.:

{
    "jobId":         100000,
    "arrayJobId":    0,
    "user":          "ccdemouser",
    "subCluster":    "main",
    "cluster":       "emmy",
    "startTime":    <date +%s>,
    "project":       "ccdemoproject",
    "resources":  [
        {"hostname":  "e0601"},
        {"hostname":  "e0823"},
        {"hostname":  "e0337"},
        {"hostname": "e1111"}],
    "numNodes":      4,
    "numHwthreads":  80,
    "walltime":      86400
}

• The response body should be the database id of the started job, for example:

{
  "id": 3937
}

• Check in ClusterCockpit
  • User ccdemouser should appear in Users-Tab with one running job
  • It could take up to 5 Minutes until the Job is displayed with some current data (5 Min Short-Job Filter)
  • Job then is marked with a green running tag
  • Metricdata displayed is read from cc-metric-store!

Use cc-backend API to stop job

• Enter the URL http://localhost:8080/swagger/index.html in your browser.
• Enter your JWT token you generated for the API user by clicking the green Authorize button in the upper right part of the window.
• Click the /job/stop_job/{id} endpoint and click the Try it out button.
• Enter the database id at id that was returned by start_job and copy the following into the request body. Replace the timestamp with a recent one:

{
  "cluster": "emmy",
  "jobState": "completed",
  "stopTime": <RECENT TS>
}

• On success a json document with the job meta data is returned.

• Check in ClusterCockpit

  • User ccdemouser should appear in Users-Tab with one completed job
  • Job is no longer marked with a green running tag -> Completed!
  • Metricdata displayed is now read from job-archive!

• Check in job-archive

  • cd ./cc-backend/var/job-archive/emmy/100/000
  • cd $STARTTIME
  • Inspect meta.json and data.json

Helper scripts

• In this tarball you can find the perl script generate_subcluster.pl that helps to generate the subcluster section for your system. Usage:
• Log into an exclusive cluster node.
• The LIKWID tools likwid-topology and likwid-bench must be in the PATH!
• $./generate_subcluster.pl outputs the subcluster section on stdout

Please be aware that

• You have to enter the name and node list for the subCluster manually.
• GPU detection only works if LIKWID was build with Cuda avalable and you run likwid-topology also with Cuda loaded.
• Do not blindly trust the measured peakflops values.
• Because the script blindly relies on the CSV format output by likwid-topology this is a fragile undertaking!