Remove ccbackend autobuild, remove ccms/influx splitting, add mariadb

2024-12-25 19:09:04 +01:00 · 2022-06-09 11:38:26 +02:00 · 2022-06-09 11:38:26 +02:00 · 2761102adf
commit 2761102adf
parent ae6d8228fa
9 changed files with 110 additions and 179 deletions
--- a/README.md
+++ b/README.md
@ -7,10 +7,11 @@ This is a `docker-compose` setup which provides a quickly started environment fo
 It includes the following containers:
 * nats (Default)
 * cc-metric-store (Default)
-* influxdb (Optional)
+* influxdb (Default)
 * openldap (Default)
-* mysql (Optional, Manual)
-* phpmyadmin (Optional, Manual)
+* mysql (Optional)
+* mariadb (Optional)
+* phpmyadmin (Optional)

 The setup comes with fixture data for a Job archive, cc-metric-store checkpoints, InfluxDB, MySQL, and a LDAP user directory.

@ -23,19 +24,16 @@ The setup comes with fixture data for a Job archive, cc-metric-store checkpoints
 ## Configuration Templates

 Located in `./templates`  
-* `docker-compose.yml.ccms`: Docker-Compose file to setup cc-metric-store metric database and LDAP containers (Default). Used in `setupDev.sh`.
-* `docker-compose.yml.influxdb`: Docker-Compose file to setup influxDB metric database and LDAP containers. Used in `setupDev.sh`.
+* `docker-compose.yml.default`: Docker-Compose file to setup cc-metric-store, InfluxDB, and LDAP containers (Default). Used in `setupDev.sh`.
 * `docker-compose.yml.mysql`: Docker-Compose configuration template if additional MySQL and phpmyadmin containers are desired.
-* `env.ccms`: Environment variables for setup with cc-metric-store metric database and LDAP containers (Default). Used in `setupDev.sh`.
-* `env.influxdb`: Environment variables for setup with influxDB metric database and LDAP containers. Used in `setupDev.sh`.
+* `docker-compose.yml.mariadb`: Docker-Compose configuration template if additional MariaDB and phpmyadmin containers are desired.
+* `env.default`: Environment variables for setup with cc-metric-store, InfluxDB, and LDAP containers (Default). Used in `setupDev.sh`.
 * `env.mysql`: Additional environment variables required if additional MySQL and phpmyadmin containers are desired.
+* `env.mariadb`: Additional environment variables required if additional MariaDB and phpmyadmin containers are desired.

 ## Setup

-1. `$ ./setupDev.sh [help|ccms|influxdb]`:  **NOTICE** The script will download files of a total size of 338MB (mostly for the InfluxDB data).  
-    * `help`: Displays help.
-    * `ccms`: Copies according docker-compose.yml and env-file to root directory, downloads cc-metric-store checkpoint data, and builds containers.
-    * `influxdb`: Copies according docker-compose.yml and env-file to root directory, downloads influxDB data, and builds containers.
+1. `$ ./setupDev.sh`:  **NOTICE** The script will download files of a total size of 338MB (mostly for the InfluxDB data).  

 2. After that from the root of the cc-docker sandbox you can start up the containers and launch cc-backend with: `$ ./startDev.sh`

@ -43,7 +41,7 @@ Located in `./templates`

 ## Post-Setup Adjustment for using `influxdb`

-When using `influxdb` containers, one must adjust the following files:  
+When using `influxdb` as a metric database, one must adjust the following files:  
 * `cc-backend/var/job-archive/emmy/cluster.json`
 * `cc-backend/var/job-archive/woody/cluster.json`

--- a/setupDev.sh
+++ b/setupDev.sh
@ -1,152 +1,71 @@
 #!/bin/bash

-if [ -z "$1" ]; then
-    echo "No argument supplied. Use 'help', 'ccms' (Default), or 'influxdb'."
-    exit
-elif [ "$1" == "help" ]; then
-    echo "Script to setup cc-backend devel environment. Use 'help', 'ccms' (Default), or 'influxdb' as argument."
-    echo "'help' : This help."
-    echo "'ccms' : Setup cc-metric-store example data and build docker container."
-    echo "'influxdb' : Setup influxdb example data and build docker container. Requires additional configuration afterwards."
-    exit
-else
-    echo "Starting setup for '$1' ..."
-fi
-
 # Download data for influxdb2
-if [ "$1" == "influxdb" ]; then
-    if [ ! -d data/influxdb ]; then
+if [ ! -d data/influxdb ]; then
+    mkdir -p data/influxdb/data
+    cd data/influxdb/data
+    wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/influxdbv2-data.tar.xz
+    tar xJf influxdbv2-data.tar.xz
+    rm influxdbv2-data.tar.xz
+    cd ../../../
+else
+    echo "'data/influxdb' already exists!"
+    echo -n "Remove existing folder and redownload? [yes to redownload / no to continue]  "
+    read -r answer
+    if [ "$answer" == "yes" ]; then
+        echo "Removing 'data/influxdb' ..."
+        rm -rf data/influxdb
+        echo "Reinstall 'data/influxdb'..."
        mkdir -p data/influxdb/data
        cd data/influxdb/data
        wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/influxdbv2-data.tar.xz
        tar xJf influxdbv2-data.tar.xz
        rm influxdbv2-data.tar.xz
        cd ../../../
+        echo "done."
    else
-        echo "'data/influxdb' already exists!"
-        echo -n "Remove existing folder and redownload? [yes to redownload / no to continue]  "
-        read -r answer
-        if [ "$answer" == "yes" ]; then
-            echo "Removing 'data/influxdb' ..."
-            rm -rf data/influxdb
-            echo "Reinstall 'data/influxdb'..."
-            mkdir -p data/influxdb/data
-            cd data/influxdb/data
-            wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/influxdbv2-data.tar.xz
-            tar xJf influxdbv2-data.tar.xz
-            rm influxdbv2-data.tar.xz
-            cd ../../../
-            echo "done."
-        else
-            echo "'data/influxdb' unchanged."
-        fi
+        echo "'data/influxdb' unchanged."
    fi
 fi

 # Download checkpoint files for cc-metric-store
-if [ "$1" == "ccms" ]; then
-    if [ ! -d data/cc-metric-store ]; then
-      mkdir -p data/cc-metric-store/checkpoints
-      mkdir -p data/cc-metric-store/archive
-      cd data/cc-metric-store/checkpoints
-      wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/cc-metric-store-checkpoints.tar.xz
-      tar xf cc-metric-store-checkpoints.tar.xz
-      rm cc-metric-store-checkpoints.tar.xz
-      cd ../../../
-    else
-        echo "'data/cc-metric-store' already exists!"
-        echo -n "Remove existing folder and redownload? [yes to redownload / no to continue]  "
-        read -r answer
-        if [ "$answer" == "yes" ]; then
-            echo "Removing 'data/cc-metric-store' ..."
-            rm -rf data/cc-metric-store
-            echo "Reinstall 'data/cc-metric-store'..."
-            mkdir -p data/cc-metric-store/checkpoints
-            mkdir -p data/cc-metric-store/archive
-            cd data/cc-metric-store/checkpoints
-            wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/cc-metric-store-checkpoints.tar.xz
-            tar xf cc-metric-store-checkpoints.tar.xz
-            rm cc-metric-store-checkpoints.tar.xz
-            cd ../../../
-            echo "done."
-        else
-            echo "'data/cc-metric-store' unchanged."
-        fi
-    fi
-fi
-
-# Download && Setup cc-backend
-if [ ! -d cc-backend ]; then
-    ## Get backend git [can use --recursive to load frontend via ssh directly]
-    git clone https://github.com/ClusterCockpit/cc-backend.git
-    cd cc-backend
-    ## Get frontend git [http variant]
-    cd frontend
-    ### Comment ths if --recursive
-    git clone https://github.com/ClusterCockpit/cc-frontend.git .
-    yarn install
-    yarn build
-    cd ..
-    ## Download Demo-Archive and prepare SQLite DB
-    mkdir ./var
-    cd ./var
-    wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/job-archive.tar.xz
-    tar xJf job-archive.tar.xz
-    rm ./job-archive.tar.xz
-    touch ./job.db
-    cd ..
-    ## Install backend
-    go get
-    go build
-    ## initialize job archive and SQLite
-    ./cc-backend --init-db --add-user demo:admin:AdminDev --no-server
-    cd ..
+if [ ! -d data/cc-metric-store ]; then
+  mkdir -p data/cc-metric-store/checkpoints
+  mkdir -p data/cc-metric-store/archive
+  cd data/cc-metric-store/checkpoints
+  wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/cc-metric-store-checkpoints.tar.xz
+  tar xf cc-metric-store-checkpoints.tar.xz
+  rm cc-metric-store-checkpoints.tar.xz
+  cd ../../../
 else
-    echo "'cc-backend' already exists!"
-    echo -n "Remove existing folder and reinstall? [yes to reinstall / no to continue]  "
+    echo "'data/cc-metric-store' already exists!"
+    echo -n "Remove existing folder and redownload? [yes to redownload / no to continue]  "
    read -r answer
    if [ "$answer" == "yes" ]; then
-        echo "Removing 'cc-backend' ..."
-        rm -rf cc-backend
-        echo "Reinstall 'cc-backend'..."
-        ## Get backend git [can use --recursive to load frontend via ssh directly]
-        git clone https://github.com/ClusterCockpit/cc-backend.git
-        cd cc-backend
-        ## Get frontend git [http variant]
-        cd frontend
-        ### Comment ths if --recursive
-        git clone https://github.com/ClusterCockpit/cc-frontend.git .
-        yarn install
-        yarn build
-        cd ..
-        ## Download Demo-Archive and prepare SQLite DB
-        mkdir ./var
-        cd ./var
-        wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/job-archive.tar.xz
-        tar xJf job-archive.tar.xz
-        rm ./job-archive.tar.xz
-        touch ./job.db
-        cd ..
-        ## Install backend
-        go get
-        go build
-        ## initialize job archive and SQLite
-        ./cc-backend --init-db --add-user demo:admin:AdminDev --no-server
-        cd ..
+        echo "Removing 'data/cc-metric-store' ..."
+        rm -rf data/cc-metric-store
+        echo "Reinstall 'data/cc-metric-store'..."
+        mkdir -p data/cc-metric-store/checkpoints
+        mkdir -p data/cc-metric-store/archive
+        cd data/cc-metric-store/checkpoints
+        wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/cc-metric-store-checkpoints.tar.xz
+        tar xf cc-metric-store-checkpoints.tar.xz
+        rm cc-metric-store-checkpoints.tar.xz
+        cd ../../../
+        echo "done."
    else
-        echo "'cc-backend' unchanged."
+        echo "'data/cc-metric-store' unchanged."
    fi
 fi

-
 # Check dotenv-file and docker-compose-yml, copy accordingly if not present and build docker services
 # !! By default, this decides which metric database is used based on the selected argument !!
 if [ ! -d .env ]; then
-    cp templates/env.$1 ./.env
+    cp templates/env.default ./.env
 fi

 if [ ! -d docker-compose.yml ]; then
-    cp templates/docker-compose.yml.$1 ./docker-compose.yml
+    cp templates/docker-compose.yml.default ./docker-compose.yml
 fi

 docker-compose build
--- a/startDev.sh
+++ b/startDev.sh
@ -1,6 +1,3 @@
 #!/bin/bash

 docker-compose up -d
-
-cd cc-backend
-./cc-backend
--- a/templates/docker-compose.yml.ccms
+++ b/templates/docker-compose.yml.ccms
@ -1,29 +0,0 @@
-services:
-  nats:
-    container_name: cc-nats
-    image: nats:alpine
-    ports:
-      - "4222:4222"
-      - "8222:8222"
-
-  cc-metric-store:
-    container_name: cc-metric-store
-    build:
-      context: ./cc-metric-store
-    ports:
-      - "8084:8084"
-    volumes:
-      - ${DATADIR}/cc-metric-store:/data
-    depends_on:
-      - nats
-
-  openldap:
-    container_name: cc-ldap
-    image: osixia/openldap:1.5.0
-    command: --copy-service --loglevel debug
-    environment:
-      - LDAP_ADMIN_PASSWORD=${LDAP_ADMIN_PASSWORD}
-      - LDAP_ORGANISATION=${LDAP_ORGANISATION}
-      - LDAP_DOMAIN=${LDAP_DOMAIN}
-    volumes:
-      - ${DATADIR}/ldap:/container/service/slapd/assets/config/bootstrap/ldif/custom
--- a/templates/docker-compose.yml.influxdb
+++ b/templates/docker-compose.yml.influxdb
@ -1,4 +1,22 @@
 services:
+  nats:
+    container_name: cc-nats
+    image: nats:alpine
+    ports:
+      - "4222:4222"
+      - "8222:8222"
+
+  cc-metric-store:
+    container_name: cc-metric-store
+    build:
+      context: ./cc-metric-store
+    ports:
+      - "8084:8084"
+    volumes:
+      - ${DATADIR}/cc-metric-store:/data
+    depends_on:
+      - nats
+
  influxdb:
    container_name: cc-influxdb
    image: influxdb
--- a/templates/docker-compose.yml.mariadb
+++ b/templates/docker-compose.yml.mariadb
@ -0,0 +1,28 @@
+services:
+  db:
+    container_name: cc-db
+    image: mariadb:latest
+    command: ["--default-authentication-plugin=mysql_native_password"]
+    environment:
+      MARIADB_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
+      MARIADB_DATABASE: ${MARIADB_DATABASE}
+      MARIADB_USER: ${MARIADB_USER}
+      MARIADB_PASSWORD: ${MARIADB_PASSWORD}
+    ports:
+      - "127.0.0.1:3336:3306"
+    volumes:
+      - ${DATADIR}/sql-init:/docker-entrypoint-initdb.d
+    cap_add:
+      - SYS_NICE
+
+  phpmyadmin:
+    container_name: cc-phpmyadmin
+    image: phpmyadmin
+    environment:
+      - PMA_HOST=cc-db
+      - PMA_USER=root
+      - PMA_PASSWORD=${MARIADB_ROOT_PASSWORD}
+    ports:
+      - "127.0.0.1:${PHPMYADMIN_PORT}:80"
+    depends_on:
+      - db
--- a/templates/env.influxdb
+++ b/templates/env.influxdb
@ -1,5 +1,5 @@
 ########################################################################
-# CCBACKEND DEVEL DOCKER SETTINGS FOR INFLUXDB AND LDAP
+# CCBACKEND DEVEL DOCKER SETTINGS
 ########################################################################

 ########################################################################
@ -13,7 +13,7 @@ INFLUXDB_BUCKET=ClusterCockpit
 # Whether or not to check SSL Cert in Symfony Client, Default: false
 INFLUXDB_SSL=false

-########################################################################
+#########################################
 # LDAP
 ########################################################################
 LDAP_ADMIN_PASSWORD=mashup
--- a/templates/env.mariadb
+++ b/templates/env.mariadb
@ -1,15 +1,16 @@
 ########################################################################
-# CCBACKEND DEVEL DOCKER SETTINGS FOR LDAP (CCMS-Devel)
+# ADDITIONAL ENV VARIABLES FOR MARIABD AND PHPMYADMIN CONTAINERS
 ########################################################################

-#########################################
-# LDAP
 ########################################################################
-LDAP_ADMIN_PASSWORD=mashup
-LDAP_ORGANISATION=NHR@FAU
-LDAP_DOMAIN=rrze.uni-erlangen.de
+# MARIADB
+########################################################################
+MARIADB_ROOT_PASSWORD=root
+MARIADB_DATABASE=ClusterCockpit
+MARIADB_USER=clustercockpit
+MARIADB_PASSWORD=clustercockpit

 ########################################################################
-# INTERNAL SETTINGS
+# PHPMyAdmin
 ########################################################################
-DATADIR=./data
+PHPMYADMIN_PORT=8081
--- a/templates/env.mysql
+++ b/templates/env.mysql
@ -4,12 +4,11 @@

 ########################################################################
 # MySQL
-# The SQL port is not exposed outside the container
 ########################################################################
 MYSQL_ROOT_PASSWORD=root
 MYSQL_DATABASE=ClusterCockpit
-MYSQL_USER=symfony
-MYSQL_PASSWORD=symfony
+MYSQL_USER=clustercockpit
+MYSQL_PASSWORD=clustercockpit

 ########################################################################
 # PHPMyAdmin