Finished testing docker services and updated README

README.md

@@ -12,7 +12,7 @@ It includes the following containers:
 |Slurm Worker|node01|6818|
 |MariaDB service|mariadb|3306|
 |InfluxDB serice|influxdb|8086|
-|NATS service|nats|4222|
+|NATS service|nats|4222, 6222, 8222|
 |cc-metric-store service|cc-metric-store|8084|
 |OpenLDAP|openldap|389, 636|
 
@@ -62,23 +62,23 @@ Credentials for the preconfigured demo user are:
 * User: `demo`
 * Password: `demo`
 
+Credentials for the preconfigured LDAP user are:
+* User: `ldapuser`
+* Password: `ldapuser`
+
 You can also login as regular user using any credential in the LDAP user directory at `./data/ldap/users.ldif`.
 
-## Post-Setup adjustment for using `cc-metric-store`
+## Preconfigured setup between docker services and ClusterCockpit components
 
-When using `influxdb` as a metric database, one must adjust the following files:
-* `cc-backend/var/job-archive/fritz/cluster.json`
-* `cc-backend/var/job-archive/alex/cluster.json`
+When you are done cloning the cc-backend repo and once you execute `setupDev.sh` file, it will copy a preconfigured `config.json` from `misc/config.json` and replace the `cc-backend/config.json`, which will be used by cc-backend, once you start the server. 
+The preconfigured config.json attaches to:
+#### 1. MariaDB docker service on port 3306 (database: ccbackend)
+#### 2. OpenLDAP docker service on port 389
+#### 3. cc-metric-store docker service on port 8084
 
-In the JSON (cc-backend/config.json), exchange the content of the `metricDataRepository`-Entry (By default configured for `cc-metric-store`) with:
-```
-"metricDataRepository": 
-{
-        "kind": "cc-metric-store",
-        "url": "http://localhost:8082",
-        "token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJFZERTQSJ9.eyJ1c2VyIjoiYWRtaW4iLCJyb2xlcyI6WyJST0xFX0FETUlOIiwiUk9MRV9BTkFMWVNUIiwiUk9MRV9VU0VSIl19.d-3_3FZTsadPjDEdsWrrQ7nS0edMAR4zjl-eK7rJU3HziNBfI9PDHDIpJVHTNN5E5SlLGLFXctWyKAkwhXL-Dw"      
-}
-```
+cc-metric-store also has a preconfigured `config.json` in `cc-metric-store/config.json` which attaches to NATS docker service on port 4222 and subscribes to topic 'hpc-nats'.
+
+Basically, all the ClusterCockpit components and the docker services attach to each other like lego pieces.
 
 ## Docker commands to access the services
 

dataGenerationScript.sh

@@ -6,6 +6,11 @@ echo "| Starting file required by docker services in data/
 echo "|--------------------------------------------------------------------------------------|"
 
 # Download unedited checkpoint files to ./data/cc-metric-store-source/checkpoints
+# After this, migrateTimestamp.pl will run from setupDev.sh. This will update the timestamps
+# for all the checkpoint files, which then can be read by cc-metric-store.
+# cc-metric-store reads only data upto certain time, like 48 hours of data.
+# These checkpoint files have timestamp older than 48 hours and needs to be updated with
+# migrateTimestamp.pl file, which will be automatically invoked from setupDev.sh.
 if [ ! -d data/cc-metric-store-source ]; then
     mkdir -p data/cc-metric-store-source/checkpoints
     cd data/cc-metric-store-source/checkpoints
@@ -17,6 +22,10 @@ else
     echo "'data/cc-metric-store-source' already exists!"
 fi
 
+# A simple configuration file for mariadb docker service.
+# Required because you can specify only one database per docker service.
+# This file mentions the database to be created for cc-backend.
+# This file automatically picked by mariadb after the docker service starts.
 if [ ! -d data/mariadb ]; then
     mkdir -p data/mariadb
     cat > data/mariadb/01.databases.sql <<EOF
@@ -26,6 +35,9 @@ else
     echo "'data/mariadb' already exists!"
 fi
 
+# A simple configuration file for openldap docker service.
+# Creates a simple user 'ldapuser' with password 'ldapuser'.
+# This file automatically picked by openldap after the docker service starts.
 if [ ! -d data/ldap ]; then
     mkdir -p data/ldap
     cat > data/ldap/add_users.ldif <<EOF
@@ -50,6 +62,11 @@ else
     echo "'data/ldap' already exists!"
 fi
 
+# A simple configuration file for nats docker service.
+# Required because we need to execute custom commands after nats docker service starts.
+# This file automatically executed when the nats docker service starts.
+# After docker service starts, there is an infinite while loop that publises data for 'fritz' and 'alex' cluster 
+# to subject 'hpc-nats' every 1 minute. Random data is generated only for node level metrics, not hardware level metrics.
 if [ ! -d data/nats ]; then
     mkdir -p data/nats
     cat > data/nats/docker-entrypoint.sh <<EOF
@@ -100,7 +117,7 @@ else
     echo "'data/nats' already exists!"
 fi
 
-# prepare folders for influxdb2
+# prepare folders for influxdb3
 if [ ! -d data/influxdb ]; then
     mkdir -p data/influxdb/data
     mkdir -p data/influxdb/config

setupDev.sh

@@ -16,7 +16,7 @@ echo "| 'sudo' keyword.
 echo "|--------------------------------------------------------------------------------------|"
 echo ""
 
-# Check cc-backend, touch job.db if exists
+# Check cc-backend if exists
 if [ ! -d cc-backend ]; then
     echo "'cc-backend' not yet prepared! Please clone cc-backend repository before starting this script."
     echo -n "Stopped."
@@ -44,14 +44,20 @@ else
     fi
 fi
 
+# Creates data directory if it does not exists.
+# Contains all the mount points required by all the docker services
+# and their static files.
 if [ ! -d data ]; then
     mkdir -m777 data
 fi
 
+# Invokes the dataGenerationScript.sh, which then populates the required
+# static files by the docker services. These static files are required by docker services after startup.
 chmod u+x dataGenerationScript.sh
 ./dataGenerationScript.sh
 
-# Update timestamps
+# Update timestamps for all the checkpoints in data/cc-metric-store-source
+# and dumps new files in data/cc-metric-store.
 perl ./migrateTimestamps.pl
 
 # Create archive folder for rewritten ccms checkpoints
@@ -64,13 +70,18 @@ if [ -d data/cc-metric-store-source ]; then
     rm -r data/cc-metric-store-source
 fi
 
+# Just in case user forgot manually shutdown the docker services.
 docker-compose down
 docker-compose down --remove-orphans
 
+# This automatically builds the base docker image for slurm.
+# All the slurm docker service in docker-compose.yml refer to
+# the base image created from this directory.
 cd slurm/base/
 make
 cd ../..
 
+# Starts all the docker services from docker-compose.yml.
 docker-compose build
 docker-compose up -d