mirror of
https://github.com/ClusterCockpit/cc-docker.git
synced 2024-12-25 19:19:05 +01:00
GitHub
commit
db0571dab1
8
.gitignore
vendored
8
.gitignore
vendored
@ -1,6 +1,10 @@
|
||||
data/job-archive
|
||||
data/job-archive/**
|
||||
data/symfony
|
||||
data/symfony/**
|
||||
data/influxdb
|
||||
data/sqldata
|
||||
data/cc-metric-store
|
||||
cc-backend
|
||||
cc-backend/**
|
||||
.vscode
|
||||
docker-compose.yml
|
||||
.env
|
||||
|
||||
108
README.md
108
README.md
@ -1,89 +1,69 @@
|
||||
# cc-docker
|
||||
|
||||
This is a `docker compose` setup to try out the complete ClusterCockpit Application Stack including all external components. This docker setup can be easily configured to be used as demo or as a development environment.
|
||||
For a docker setup targeted to server environment you may have a look at https://github.com/ClusterCockpit/cc-docker-server .
|
||||
**Please note: This repo is under ongoing construction**
|
||||
|
||||
This is a `docker-compose` setup which provides a quickly started environment for ClusterCockpit development and testing, using the modules `cc-backend` (GoLang) and `cc-frontend` (Svelte). A number of services is readily available as docker container (nats, cc-metric-store, InfluxDB, LDAP), or easily added by manual configuration (MySQL).
|
||||
|
||||
It includes the following containers:
|
||||
* mysql
|
||||
* php-fpm
|
||||
* nginx
|
||||
* redis
|
||||
* openldap
|
||||
* influxdb
|
||||
* phpmyadmin
|
||||
* nats (Default)
|
||||
* cc-metric-store (Default)
|
||||
* influxdb (Default)
|
||||
* openldap (Default)
|
||||
* mysql (Optional)
|
||||
* mariadb (Optional)
|
||||
* phpmyadmin (Optional)
|
||||
|
||||
Settings are configured in `.env`.
|
||||
The setup comes with fixture data for a Job archive, InfluxDB, MySQL, and a LDAP user directory.
|
||||
The setup comes with fixture data for a Job archive, cc-metric-store checkpoints, InfluxDB, MySQL, and a LDAP user directory.
|
||||
|
||||
## Known Issues
|
||||
|
||||
* `docker-compose` installed on Ubuntu (18.04, 20.04) via `apt-get` can not correctly parse `docker-compose.yml` due to version differences. Install latest version of `docker-compose` from https://docs.docker.com/compose/install/ instead.
|
||||
* You need to ensure that no other web server is running on port 80 (e.g. Apache2). If port 80 is already in use, edit NGINX_PORT environment variable in `.env`.
|
||||
* You need to ensure that no other web server is running on ports 8080 (cc-backend), 8081 (phpmyadmin), 8084 (cc-metric-store), 8086 (nfluxDB), 4222 and 8222 (Nats), or 3306 (MySQL). If one or more ports are already in use, you habe to adapt the related config accordingly.
|
||||
* Existing VPN connections sometimes cause problems with docker. If `docker-compose` does not start up correctly, try disabling any active VPN connection. Refer to https://stackoverflow.com/questions/45692255/how-make-openvpn-work-with-docker for further information.
|
||||
|
||||
## Configuration
|
||||
## Configuration Templates
|
||||
|
||||
The main branch of this repository will work with the latest ClusterCockpit release (main branch) including the fixture data.
|
||||
For ClusterCockpit development on the develop branch use the cc-docker develop branch.
|
||||
|
||||
While many aspects of this docker compose setup can be configured you usually only need to adapt the following three settings in `.env`:
|
||||
* `CLUSTERCOCKPIT_BRANCH`: The branch to checkout from ClusterCockpit git repository. May also be a tag. This should not be changed as it may be that the fixture data may not be compatible between stable and develop.
|
||||
* `APP_CLUSTERCOCKPIT_INIT` (Default: true): Wether the Symfony tree (located at `./data/symfony`) should be deleted and freshly cloned and initialized on every container startup.
|
||||
* `APP_ENVIRONMENT` (Default: `dev`): The Symfony app environment. With `dev` you get the symfony debug toolbar and more extensive error handling. The `prod` environment is a setup for productions use.
|
||||
Located in `./templates`
|
||||
* `docker-compose.yml.default`: Docker-Compose file to setup cc-metric-store, InfluxDB, MariaDB, PhpMyadmin, and LDAP containers (Default). Used in `setupDev.sh`.
|
||||
* `docker-compose.yml.mysql`: Docker-Compose configuration template if MySQL is desired instead of MariaDB.
|
||||
* `env.default`: Environment variables for setup with cc-metric-store, InfluxDB, MariaDB, PhpMyadmin, and LDAP containers (Default). Used in `setupDev.sh`.
|
||||
* `env.mysql`: Additional environment variables required if MySQL is desired instead of MariaDB.
|
||||
|
||||
## Setup
|
||||
|
||||
* `$ cd data`
|
||||
* `$ ./init.sh`: **NOTICE** The script will download files of a total size of 338MB (mostly for the InfluxDB data).
|
||||
1. Clone `cc-backend` repository in chosen base folder: `$> git clone https://github.com/ClusterCockpit/cc-backend.git`
|
||||
|
||||
If you want to test the REST API and also write to the job archive from Cluster Cockpit you have to comment out the following lines in `./data/init.sh`:
|
||||
2. Run `$ ./setupDev.sh`: **NOTICE** The script will download files of a total size of 338MB (mostly for the InfluxDB data).
|
||||
|
||||
3. The setup-script launches the supporting container stack in the background automatically if everything went well. Run `$> ./cc-backend/cc-backend` to start `cc-backend.`
|
||||
|
||||
4. By default, you can access `cc-backend` in your browser at `http://localhost:8080`. You can shut down the cc-backend server by pressing `CTRL-C`, remember to also shut down all containers via `$> docker-compose down` afterwards.
|
||||
|
||||
5. You can restart the containers with: `$> docker-compose up -d`.
|
||||
|
||||
## Post-Setup Adjustment for using `influxdb`
|
||||
|
||||
When using `influxdb` as a metric database, one must adjust the following files:
|
||||
* `cc-backend/var/job-archive/emmy/cluster.json`
|
||||
* `cc-backend/var/job-archive/woody/cluster.json`
|
||||
|
||||
In the JSON, exchange the content of the `metricDataRepository`-Entry (By default configured for `cc-metric-store`) with:
|
||||
```
|
||||
echo "This script needs to chown the job-archive directory so that the application can write to it:"
|
||||
sudo chown -R 82:82 ./job-archive
|
||||
"metricDataRepository": {
|
||||
"kind": "influxdb",
|
||||
"url": "http://localhost:8086",
|
||||
"token": "egLfcf7fx0FESqFYU3RpAAbj",
|
||||
"bucket": "ClusterCockpit",
|
||||
"org": "ClusterCockpit",
|
||||
"skiptls": false
|
||||
}
|
||||
```
|
||||
|
||||
After that from the root of the cc-docker sandbox you can start up the containers with:
|
||||
* `$ docker-compose up`
|
||||
* Wait... and wait a little longer
|
||||
|
||||
Before you can use ClusterCockpit the following disclaimer must be shown. To download and build all ClusterCockpit components may take up to several minutes:
|
||||
```
|
||||
-------------------- ---------------------------------
|
||||
Symfony
|
||||
-------------------- ---------------------------------
|
||||
Version 5.3.7
|
||||
Long-Term Support No
|
||||
End of maintenance 01/2022 (in +140 days)
|
||||
End of life 01/2022 (in +140 days)
|
||||
-------------------- ---------------------------------
|
||||
Kernel
|
||||
-------------------- ---------------------------------
|
||||
Type App\Kernel
|
||||
Environment dev
|
||||
Debug true
|
||||
Charset UTF-8
|
||||
Cache directory ./var/cache/dev (6.5 MiB)
|
||||
Build directory ./var/cache/dev (6.5 MiB)
|
||||
Log directory ./var/log (249 B)
|
||||
-------------------- ---------------------------------
|
||||
PHP
|
||||
-------------------- ---------------------------------
|
||||
Version 8.0.10
|
||||
Architecture 64 bits
|
||||
Intl locale n/a
|
||||
Timezone UTC (2021-09-13T09:41:33+00:00)
|
||||
OPcache true
|
||||
APCu false
|
||||
Xdebug false
|
||||
-------------------- ---------------------------------
|
||||
```
|
||||
|
||||
By default, you can access ClusterCockpit in your browser at `http://localhost`. If the `NGINX_PORT` environment variable was changed, you have to use `http://localhost:$PORT` . You can shutdown the containers by pressing `CTRL-C`. Refer to the common docker documentation how to start the environment in the background.
|
||||
|
||||
## Usage
|
||||
|
||||
Credentials for the preconfigured admin user are:
|
||||
* User: `admin`
|
||||
Credentials for the preconfigured demo user are:
|
||||
* User: `demo`
|
||||
* Password: `AdminDev`
|
||||
|
||||
You can also login as regular user using any credential in the LDAP user directory at `./data/ldap/users.ldif`.
|
||||
|
||||
19
cc-metric-store/Dockerfile
Normal file
19
cc-metric-store/Dockerfile
Normal file
@ -0,0 +1,19 @@
|
||||
FROM golang:1.17
|
||||
|
||||
RUN apt-get update
|
||||
RUN apt-get -y install git
|
||||
|
||||
RUN git clone https://github.com/ClusterCockpit/cc-metric-store.git /cc-metric-store
|
||||
RUN cd /cc-metric-store && go build
|
||||
|
||||
# Reactivate when latest commit is available
|
||||
#RUN go get -d -v github.com/ClusterCockpit/cc-metric-store
|
||||
#RUN go install -v github.com/ClusterCockpit/cc-metric-store@latest
|
||||
|
||||
RUN mv /cc-metric-store/cc-metric-store /go/bin
|
||||
COPY config.json /go/bin
|
||||
|
||||
VOLUME /data
|
||||
|
||||
WORKDIR /go/bin
|
||||
CMD ["./cc-metric-store"]
|
||||
28
cc-metric-store/config.json
Normal file
28
cc-metric-store/config.json
Normal file
@ -0,0 +1,28 @@
|
||||
{
|
||||
"metrics": {
|
||||
"clock": { "frequency": 60, "aggregation": null, "scope": "node" },
|
||||
"cpi": { "frequency": 60, "aggregation": null, "scope": "node" },
|
||||
"cpu_load": { "frequency": 60, "aggregation": null, "scope": "node" },
|
||||
"flops_any": { "frequency": 60, "aggregation": null, "scope": "node" },
|
||||
"flops_dp": { "frequency": 60, "aggregation": null, "scope": "node" },
|
||||
"flops_sp": { "frequency": 60, "aggregation": null, "scope": "node" },
|
||||
"ib_bw": { "frequency": 60, "aggregation": null, "scope": "node" },
|
||||
"lustre_bw": { "frequency": 60, "aggregation": null, "scope": "node" },
|
||||
"mem_bw": { "frequency": 60, "aggregation": null, "scope": "node" },
|
||||
"mem_used": { "frequency": 60, "aggregation": null, "scope": "node" },
|
||||
"rapl_power": { "frequency": 60, "aggregation": null, "scope": "node" }
|
||||
},
|
||||
"checkpoints": {
|
||||
"interval": 100000000000,
|
||||
"directory": "/data/checkpoints",
|
||||
"restore": 100000000000
|
||||
},
|
||||
"archive": {
|
||||
"interval": 100000000000,
|
||||
"directory": "/data/archive"
|
||||
},
|
||||
"retention-in-memory": 100000000000,
|
||||
"http-api-address": "0.0.0.0:8081",
|
||||
"nats": "nats://cc-nats:4222",
|
||||
"jwt-public-key": "kzfYrYy+TzpanWZHJ5qSdMj5uKUWgq74BWhQG6copP0="
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@ -1,112 +0,0 @@
|
||||
services:
|
||||
db:
|
||||
container_name: cc-db
|
||||
image: mysql:8.0.22
|
||||
command: ["--default-authentication-plugin=mysql_native_password"]
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
|
||||
MYSQL_DATABASE: ${MYSQL_DATABASE}
|
||||
MYSQL_USER: ${MYSQL_USER}
|
||||
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
|
||||
volumes:
|
||||
- ${DATADIR}/sql-init:/docker-entrypoint-initdb.d
|
||||
# - ${DATADIR}/sqldata:/var/lib/mysql
|
||||
cap_add:
|
||||
- SYS_NICE
|
||||
|
||||
influxdb:
|
||||
container_name: cc-influxdb
|
||||
image: influxdb
|
||||
command: ["--reporting-disabled"]
|
||||
environment:
|
||||
DOCKER_INFLUXDB_INIT_MODE: setup
|
||||
DOCKER_INFLUXDB_INIT_USERNAME: symfony
|
||||
DOCKER_INFLUXDB_INIT_PASSWORD: ${INFLUXDB_PASSWORD}
|
||||
DOCKER_INFLUXDB_INIT_ORG: ${INFLUXDB_ORG}
|
||||
DOCKER_INFLUXDB_INIT_BUCKET: ${INFLUXDB_BUCKET}
|
||||
DOCKER_INFLUXDB_INIT_RETENTION: 100w
|
||||
DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: ${INFLUXDB_ADMIN_TOKEN}
|
||||
ports:
|
||||
- "127.0.0.1:${INFLUXDB_PORT}:8086"
|
||||
volumes:
|
||||
- ${DATADIR}/influxdb/data:/var/lib/influxdb2
|
||||
- ${DATADIR}/influxdb/config:/etc/influxdb2
|
||||
|
||||
openldap:
|
||||
container_name: cc-ldap
|
||||
image: osixia/openldap:1.5.0
|
||||
command: --copy-service --loglevel debug
|
||||
environment:
|
||||
- LDAP_ADMIN_PASSWORD=${LDAP_ADMIN_PASSWORD}
|
||||
- LDAP_ORGANISATION=${LDAP_ORGANISATION}
|
||||
- LDAP_DOMAIN=${LDAP_DOMAIN}
|
||||
volumes:
|
||||
- ${DATADIR}/ldap:/container/service/slapd/assets/config/bootstrap/ldif/custom
|
||||
|
||||
redis:
|
||||
container_name: cc-redis
|
||||
image: redis
|
||||
command: [
|
||||
"redis-server",
|
||||
"--save", "",
|
||||
"--maxmemory", "2gb",
|
||||
"--maxmemory-policy", "allkeys-lru"]
|
||||
|
||||
php:
|
||||
container_name: cc-php
|
||||
build:
|
||||
context: ./php-fpm
|
||||
args:
|
||||
PHP_XDEBUG_INIT: ${PHP_XDEBUG_INIT}
|
||||
PHP_XDEBUG_MODE: ${PHP_XDEBUG_MODE}
|
||||
PHP_XDEBUG_CLIENT_PORT: ${PHP_XDEBUG_CLIENT_PORT}
|
||||
PHP_XDEBUG_CLIENT_HOST: ${PHP_XDEBUG_CLIENT_HOST}
|
||||
SYMFONY_CLI_VERSION: 4.23.2
|
||||
MYSQL_DATABASE: ${MYSQL_DATABASE}
|
||||
MYSQL_USER: ${MYSQL_USER}
|
||||
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
|
||||
LDAP_PASSWORD: ${LDAP_ADMIN_PASSWORD}
|
||||
INFLUXDB_PASSWORD: ${INFLUXDB_PASSWORD}
|
||||
INFLUXDB_PORT: ${INFLUXDB_PORT}
|
||||
INFLUXDB_ADMIN_TOKEN: ${INFLUXDB_ADMIN_TOKEN}
|
||||
INFLUXDB_ORG: ${INFLUXDB_ORG}
|
||||
INFLUXDB_BUCKET: ${INFLUXDB_BUCKET}
|
||||
INFLUXDB_SSL: ${INFLUXDB_SSL}
|
||||
APP_ENVIRONMENT: ${APP_ENVIRONMENT}
|
||||
environment:
|
||||
- APP_CLUSTERCOCKPIT_INIT=${APP_CLUSTERCOCKPIT_INIT}
|
||||
- CLUSTERCOCKPIT_BRANCH=${CLUSTERCOCKPIT_BRANCH}
|
||||
- APP_JWT_PUB_KEY=${APP_JWT_PUB_KEY}
|
||||
- APP_JWT_PRIV_KEY=${APP_JWT_PRIV_KEY}
|
||||
volumes:
|
||||
- ${DATADIR}/symfony:/var/www/symfony:cached
|
||||
- ${DATADIR}/job-archive:/var/lib/job-archive:cached
|
||||
depends_on:
|
||||
- db
|
||||
- redis
|
||||
- influxdb
|
||||
|
||||
nginx:
|
||||
container_name: cc-nginx
|
||||
build:
|
||||
context: ./nginx
|
||||
ports:
|
||||
- "127.0.0.1:${NGINX_PORT}:80"
|
||||
depends_on:
|
||||
- php
|
||||
environment:
|
||||
- NGINX_ENVSUBST_OUTPUT_DIR=/etc/nginx/conf.d
|
||||
- NGINX_ENVSUBST_TEMPLATE_DIR=/etc/nginx/templates
|
||||
- NGINX_ENVSUBST_TEMPLATE_SUFFIX=.template
|
||||
volumes:
|
||||
- ${DATADIR}/symfony:/var/www/symfony:cached
|
||||
|
||||
phpmyadmin:
|
||||
container_name: cc-phpmyadmin
|
||||
image: phpmyadmin
|
||||
environment:
|
||||
- PMA_HOST=cc-db
|
||||
- PMA_USER=root
|
||||
- PMA_PASSWORD=${MYSQL_ROOT_PASSWORD}
|
||||
ports:
|
||||
- "127.0.0.1:${PHPMYADMIN_PORT}:80"
|
||||
171
migrateTimestamps.pl
Executable file
171
migrateTimestamps.pl
Executable file
@ -0,0 +1,171 @@
|
||||
#!/usr/bin/env perl
|
||||
use strict;
|
||||
use warnings;
|
||||
use utf8;
|
||||
|
||||
use File::Path qw( make_path rmtree );
|
||||
use Cpanel::JSON::XS qw( decode_json encode_json );
|
||||
use File::Slurp;
|
||||
use Data::Dumper;
|
||||
use Time::Piece;
|
||||
use Sort::Versions;
|
||||
use REST::Client;
|
||||
|
||||
### JOB-ARCHIVE
|
||||
my $localtime = localtime;
|
||||
my $epochtime = $localtime->epoch;
|
||||
my $archiveTarget = './cc-backend/var/job-archive';
|
||||
my $archiveSrc = './data/job-archive-source';
|
||||
my @ArchiveClusters;
|
||||
|
||||
# Get clusters by job-archive/$subfolder
|
||||
opendir my $dh, $archiveSrc or die "can't open directory: $!";
|
||||
while ( readdir $dh ) {
|
||||
chomp; next if $_ eq '.' or $_ eq '..' or $_ eq 'job-archive';
|
||||
|
||||
my $cluster = $_;
|
||||
push @ArchiveClusters, $cluster;
|
||||
}
|
||||
|
||||
# start for jobarchive
|
||||
foreach my $cluster ( @ArchiveClusters ) {
|
||||
print "Starting to update start- and stoptimes in job-archive for $cluster\n";
|
||||
|
||||
opendir my $dhLevel1, "$archiveSrc/$cluster" or die "can't open directory: $!";
|
||||
while ( readdir $dhLevel1 ) {
|
||||
chomp; next if $_ eq '.' or $_ eq '..';
|
||||
my $level1 = $_;
|
||||
|
||||
if ( -d "$archiveSrc/$cluster/$level1" ) {
|
||||
opendir my $dhLevel2, "$archiveSrc/$cluster/$level1" or die "can't open directory: $!";
|
||||
while ( readdir $dhLevel2 ) {
|
||||
chomp; next if $_ eq '.' or $_ eq '..';
|
||||
my $level2 = $_;
|
||||
my $jobSource = "$archiveSrc/$cluster/$level1/$level2";
|
||||
my $jobTarget = "$archiveTarget/$cluster/$level1/$level2/";
|
||||
my $jobOrigin = $jobSource;
|
||||
# check if files are directly accessible (old format) else get subfolders as file and update path
|
||||
if ( ! -e "$jobSource/meta.json") {
|
||||
my @folders = read_dir($jobSource);
|
||||
if (!@folders) {
|
||||
next;
|
||||
}
|
||||
# Only use first subfolder for now TODO
|
||||
$jobSource = "$jobSource/".$folders[0];
|
||||
}
|
||||
# check if subfolder contains file, else remove source and skip
|
||||
if ( ! -e "$jobSource/meta.json") {
|
||||
# rmtree $jobOrigin;
|
||||
next;
|
||||
}
|
||||
|
||||
my $rawstr = read_file("$jobSource/meta.json");
|
||||
my $json = decode_json($rawstr);
|
||||
|
||||
# NOTE Start meta.json iteration here
|
||||
# my $random_number = int(rand(UPPERLIMIT)) + LOWERLIMIT;
|
||||
# Set new startTime: Between 5 days and 1 day before now
|
||||
|
||||
# Remove id from attributes
|
||||
$json->{startTime} = $epochtime - (int(rand(432000)) + 86400);
|
||||
$json->{stopTime} = $json->{startTime} + $json->{duration};
|
||||
|
||||
# Add starttime subfolder to target path
|
||||
$jobTarget .= $json->{startTime};
|
||||
|
||||
# target is not directory
|
||||
if ( not -d $jobTarget ){
|
||||
# print "Writing files\n";
|
||||
# print "$cluster/$level1/$level2\n";
|
||||
make_path($jobTarget);
|
||||
|
||||
my $outstr = encode_json($json);
|
||||
write_file("$jobTarget/meta.json", $outstr);
|
||||
|
||||
my $datstr = read_file("$jobSource/data.json");
|
||||
write_file("$jobTarget/data.json", $datstr);
|
||||
} else {
|
||||
# rmtree $jobSource;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
print "Done for job-archive\n";
|
||||
sleep(1);
|
||||
|
||||
## CHECKPOINTS
|
||||
chomp(my $checkpointStart=`date --date 'TZ="Europe/Berlin" 0:00 7 days ago' +%s`);
|
||||
my $halfday = 43200;
|
||||
my $checkpTarget = './data/cc-metric-store/checkpoints';
|
||||
my $checkpSource = './data/cc-metric-store-source/checkpoints';
|
||||
my @CheckpClusters;
|
||||
|
||||
# Get clusters by cc-metric-store/$subfolder
|
||||
opendir my $dhc, $checkpSource or die "can't open directory: $!";
|
||||
while ( readdir $dhc ) {
|
||||
chomp; next if $_ eq '.' or $_ eq '..' or $_ eq 'job-archive';
|
||||
|
||||
my $cluster = $_;
|
||||
push @CheckpClusters, $cluster;
|
||||
}
|
||||
|
||||
# start for checkpoints
|
||||
foreach my $cluster ( @CheckpClusters ) {
|
||||
print "Starting to update checkpoint filenames and data starttimes for $cluster\n";
|
||||
|
||||
opendir my $dhLevel1, "$checkpSource/$cluster" or die "can't open directory: $!";
|
||||
while ( readdir $dhLevel1 ) {
|
||||
chomp; next if $_ eq '.' or $_ eq '..';
|
||||
# Nodename as level1-folder
|
||||
my $level1 = $_;
|
||||
|
||||
if ( -d "$checkpSource/$cluster/$level1" ) {
|
||||
|
||||
my $nodeSource = "$checkpSource/$cluster/$level1/";
|
||||
my $nodeTarget = "$checkpTarget/$cluster/$level1/";
|
||||
my $nodeOrigin = $nodeSource;
|
||||
my @files;
|
||||
|
||||
if ( -e "$nodeSource/1609459200.json") { # 1609459200 == First Checkpoint time in latest dump
|
||||
@files = read_dir($nodeSource);
|
||||
my $length = @files;
|
||||
if (!@files || $length != 14) { # needs 14 files == 7 days worth of data
|
||||
next;
|
||||
}
|
||||
} else {
|
||||
# rmtree $nodeOrigin;
|
||||
next;
|
||||
}
|
||||
|
||||
my @sortedFiles = sort { versioncmp($a,$b) } @files; # sort alphanumerically: _Really_ start with index == 0 == 1609459200.json
|
||||
|
||||
if ( not -d $nodeTarget ){
|
||||
# print "processing files for $level1 \n";
|
||||
make_path($nodeTarget);
|
||||
|
||||
while (my ($index, $file) = each(@sortedFiles)) {
|
||||
# print "$file\n";
|
||||
my $rawstr = read_file("$nodeSource/$file");
|
||||
my $json = decode_json($rawstr);
|
||||
|
||||
my $newTimestamp = $checkpointStart + ($index * $halfday);
|
||||
# Get Diff from old Timestamp
|
||||
my $timeDiff = $newTimestamp - $json->{from};
|
||||
# Set new timestamp
|
||||
$json->{from} = $newTimestamp;
|
||||
|
||||
foreach my $metric (keys %{$json->{metrics}}) {
|
||||
$json->{metrics}->{$metric}->{start} += $timeDiff;
|
||||
}
|
||||
|
||||
my $outstr = encode_json($json);
|
||||
write_file("$nodeTarget/$newTimestamp.json", $outstr);
|
||||
}
|
||||
} else {
|
||||
# rmtree $nodeSource;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
print "Done for checkpoints\n";
|
||||
@ -1,8 +0,0 @@
|
||||
FROM nginx:mainline-alpine
|
||||
|
||||
RUN mkdir -p /etc/nginx/templates
|
||||
COPY templates/* /etc/nginx/templates/
|
||||
COPY nginx.conf /etc/nginx/
|
||||
|
||||
CMD ["nginx"]
|
||||
EXPOSE 80
|
||||
@ -1,48 +0,0 @@
|
||||
user nginx;
|
||||
worker_processes 4;
|
||||
pid /var/run/nginx.pid;
|
||||
|
||||
events {
|
||||
worker_connections 2048;
|
||||
multi_accept on;
|
||||
use epoll;
|
||||
}
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
|
||||
client_body_buffer_size 256k;
|
||||
client_header_buffer_size 1k;
|
||||
client_max_body_size 8m;
|
||||
large_client_header_buffers 2 1k;
|
||||
client_body_temp_path /tmp 1 2;
|
||||
client_body_in_file_only off;
|
||||
|
||||
keepalive_timeout 90;
|
||||
send_timeout 120;
|
||||
reset_timedout_connection on;
|
||||
|
||||
open_file_cache max=2000 inactive=20s;
|
||||
open_file_cache_valid 30s;
|
||||
open_file_cache_min_uses 2;
|
||||
open_file_cache_errors on;
|
||||
|
||||
server_tokens off;
|
||||
sendfile on;
|
||||
tcp_nopush on;
|
||||
tcp_nodelay on;
|
||||
types_hash_max_size 2048;
|
||||
|
||||
access_log off;
|
||||
error_log off;
|
||||
|
||||
gzip on;
|
||||
gzip_comp_level 9;
|
||||
gzip_min_length 200;
|
||||
gzip_types text/plain text/html text/css application/json;
|
||||
|
||||
include /etc/nginx/conf.d/*.conf;
|
||||
}
|
||||
|
||||
daemon off;
|
||||
@ -1,23 +0,0 @@
|
||||
server {
|
||||
server_name localhost;
|
||||
root /var/www/symfony/public;
|
||||
|
||||
location / {
|
||||
try_files $uri /index.php$is_args$args;
|
||||
}
|
||||
|
||||
location ~ ^/index\.php(/|$) {
|
||||
fastcgi_pass php-upstream;
|
||||
fastcgi_split_path_info ^(.+\.php)(/.*)$;
|
||||
include fastcgi_params;
|
||||
fastcgi_param HTTPS off;
|
||||
fastcgi_read_timeout 300;
|
||||
fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
|
||||
fastcgi_param DOCUMENT_ROOT $realpath_root;
|
||||
internal;
|
||||
}
|
||||
|
||||
location ~ \.php$ {
|
||||
return 404;
|
||||
}
|
||||
}
|
||||
@ -1,3 +0,0 @@
|
||||
upstream php-upstream {
|
||||
server php:9001;
|
||||
}
|
||||
@ -1,99 +0,0 @@
|
||||
FROM php:8.0-fpm
|
||||
|
||||
RUN apt-get update && apt-get install -y \
|
||||
$PHPIZE_DEPS \
|
||||
git \
|
||||
wget \
|
||||
zip \
|
||||
gettext \
|
||||
bash \
|
||||
libldb-dev \
|
||||
libldap-2.4-2 \
|
||||
libldap-common \
|
||||
libldap2-dev \
|
||||
npm \
|
||||
nodejs
|
||||
|
||||
RUN apt-get clean
|
||||
|
||||
RUN npm install --global yarn
|
||||
|
||||
RUN docker-php-ext-install ldap \
|
||||
mysqli \
|
||||
pdo_mysql \
|
||||
opcache
|
||||
|
||||
# Enable php8-xdebug if $PHP_XDEBUG_INIT is true
|
||||
ARG PHP_XDEBUG_INIT="false"
|
||||
ARG PHP_XDEBUG_MODE=off
|
||||
ARG PHP_XDEBUG_CLIENT_PORT=5902
|
||||
ARG PHP_XDEBUG_CLIENT_HOST=host.docker.internal
|
||||
COPY xdebug.ini /etc/php8/conf.d/xdebug.ini.template
|
||||
COPY error_reporting.ini /usr/local/etc/php/conf.d/error_reporting.ini
|
||||
RUN if [[ "$PHP_XDEBUG_INIT" == "true" ]]; then \
|
||||
pecl install xdebug-3.0.4; \
|
||||
docker-php-ext-enable xdebug; \
|
||||
export PHP_XDEBUG_MODE=$PHP_XDEBUG_MODE; \
|
||||
export PHP_XDEBUG_CLIENT_PORT=$PHP_XDEBUG_CLIENT_PORT; \
|
||||
export PHP_XDEBUG_CLIENT_HOST=$PHP_XDEBUG_CLIENT_HOST; \
|
||||
envsubst < /etc/php8/conf.d/xdebug.ini.template > /etc/php8/conf.d/xdebug.ini; \
|
||||
cp /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini.back; \
|
||||
cp /etc/php8/conf.d/xdebug.ini /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini; \
|
||||
rm -f /etc/php8/conf.d/xdebug.ini.template; \
|
||||
fi
|
||||
|
||||
RUN curl -sS https://getcomposer.org/installer | tee composer-setup.php \
|
||||
&& php composer-setup.php && rm composer-setup.php* \
|
||||
&& chmod +x composer.phar && mv composer.phar /usr/bin/composer
|
||||
|
||||
ARG SYMFONY_CLI_VERSION
|
||||
RUN wget https://github.com/symfony/cli/releases/download/v$SYMFONY_CLI_VERSION/symfony_linux_amd64.gz \
|
||||
&& gzip -d symfony_linux_amd64.gz \
|
||||
&& mv symfony_linux_amd64 symfony \
|
||||
&& chmod +x symfony \
|
||||
&& mv symfony /usr/local/bin/
|
||||
|
||||
RUN mkdir -p /var/lib/job-archive
|
||||
RUN mkdir -p /var/www/symfony
|
||||
VOLUME /var/www/symfony /var/lib/job-archive
|
||||
|
||||
COPY php.ini /usr/local/etc/php/
|
||||
COPY symfony.ini /usr/local/etc/php/conf.d/
|
||||
COPY symfony.ini /usr/local/etc/php/cli/conf.d/
|
||||
COPY symfony.pool.conf /usr/local/etc/php/php-fpm.d/
|
||||
COPY entrypoint.sh /entrypoint.sh
|
||||
RUN chmod +x /entrypoint.sh
|
||||
|
||||
ARG APP_ENVIRONMENT
|
||||
ENV APP_ENV=${APP_ENVIRONMENT}
|
||||
ENV APP_SECRET=${APP_SECRET}
|
||||
ENV APP_JWT_PUB_KEY="${APP_JWT_PUB_KEY}"
|
||||
ENV APP_JWT_PRIV_KEY="${APP_JWT_PRIV_KEY}"
|
||||
ENV APP_DEBUG=1
|
||||
ENV REDIS_URL=redis://cc-redis
|
||||
ENV LDAP_URL=ldap://cc-ldap
|
||||
ARG INFLUXDB_PORT
|
||||
ARG INFLUXDB_PASSWORD
|
||||
ARG INFLUXDB_ADMIN_TOKEN
|
||||
ARG INFLUXDB_ORG
|
||||
ARG INFLUXDB_BUCKET
|
||||
ARG INFLUXDB_SSL
|
||||
ENV INFLUXDB_URL=http://cc-influxdb:${INFLUXDB_PORT}
|
||||
ENV INFLUXDB_SSL=${INFLUXDB_SSL}
|
||||
ENV INFLUXDB_TOKEN=${INFLUXDB_ADMIN_TOKEN}
|
||||
ENV INFLUXDB_ORG=${INFLUXDB_ORG}
|
||||
ENV INFLUXDB_BUCKET=${INFLUXDB_BUCKET}
|
||||
ARG LDAP_PASSWORD
|
||||
ENV LDAP_PW=${LDAP_PASSWORD}
|
||||
ARG MYSQL_USER
|
||||
ARG MYSQL_PASSWORD
|
||||
ARG MYSQL_DATABASE
|
||||
ENV DATABASE_URL=mysql://${MYSQL_USER}:${MYSQL_PASSWORD}@cc-db:3306/${MYSQL_DATABASE}
|
||||
ENV CORS_ALLOW_ORIGIN=^https?://(localhost|127\\.0\\.0\\.1)(:[0-9]+)?$
|
||||
|
||||
WORKDIR /var/www/symfony
|
||||
|
||||
ENTRYPOINT ["/entrypoint.sh"]
|
||||
CMD ["php-fpm", "-F", "-y/usr/local/etc/php/php-fpm.d/symfony.pool.conf"]
|
||||
|
||||
EXPOSE 9001
|
||||
@ -1,24 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
if [ "$APP_CLUSTERCOCKPIT_INIT" = true ]; then
|
||||
rm -rf /var/www/symfony/* /var/www/symfony/.??*
|
||||
git clone -b $CLUSTERCOCKPIT_BRANCH https://github.com/ClusterCockpit/ClusterCockpit .
|
||||
|
||||
if [ "$APP_ENV" = dev ]; then
|
||||
composer install --no-progress --optimize-autoloader
|
||||
yarn install
|
||||
yarn encore dev
|
||||
else
|
||||
composer install --no-dev --no-progress --optimize-autoloader
|
||||
yarn install
|
||||
yarn encore production
|
||||
fi
|
||||
|
||||
ln -s /var/lib/job-archive var/job-archive
|
||||
chown -R www-data:www-data /var/www/symfony/* /var/www/symfony/.??*
|
||||
fi
|
||||
|
||||
# Reports php environment on container startup
|
||||
php bin/console about
|
||||
|
||||
exec "$@"
|
||||
@ -1 +0,0 @@
|
||||
error_reporting=E_ALL
|
||||
1944
php-fpm/php.ini
1944
php-fpm/php.ini
File diff suppressed because it is too large
Load Diff
@ -1 +0,0 @@
|
||||
date.timezone = UTC
|
||||
@ -1,100 +0,0 @@
|
||||
; Start a new pool named 'symfony'.
|
||||
; the variable $pool can be used in any directive and will be replaced by the
|
||||
; pool name ('symfony' here)
|
||||
[symfony]
|
||||
|
||||
; Unix user/group of processes
|
||||
; Note: The user is mandatory. If the group is not set, the default user's group
|
||||
; will be used.
|
||||
user = www-data
|
||||
group = www-data
|
||||
|
||||
; The address on which to accept FastCGI requests.
|
||||
; Valid syntaxes are:
|
||||
; 'ip.add.re.ss:port' - to listen on a TCP socket to a specific address on
|
||||
; a specific port;
|
||||
; 'port' - to listen on a TCP socket to all addresses on a
|
||||
; specific port;
|
||||
; '/path/to/unix/socket' - to listen on a unix socket.
|
||||
; Note: This value is mandatory.
|
||||
listen = 0.0.0.0:9001
|
||||
|
||||
; Choose how the process manager will control the number of child processes.
|
||||
; Possible Values:
|
||||
; static - a fixed number (pm.max_children) of child processes;
|
||||
; dynamic - the number of child processes are set dynamically based on the
|
||||
; following directives. With this process management, there will be
|
||||
; always at least 1 children.
|
||||
; pm.max_children - the maximum number of children that can
|
||||
; be alive at the same time.
|
||||
; pm.start_servers - the number of children created on startup.
|
||||
; pm.min_spare_servers - the minimum number of children in 'idle'
|
||||
; state (waiting to process). If the number
|
||||
; of 'idle' processes is less than this
|
||||
; number then some children will be created.
|
||||
; pm.max_spare_servers - the maximum number of children in 'idle'
|
||||
; state (waiting to process). If the number
|
||||
; of 'idle' processes is greater than this
|
||||
; number then some children will be killed.
|
||||
; ondemand - no children are created at startup. Children will be forked when
|
||||
; new requests will connect. The following parameter are used:
|
||||
; pm.max_children - the maximum number of children that
|
||||
; can be alive at the same time.
|
||||
; pm.process_idle_timeout - The number of seconds after which
|
||||
; an idle process will be killed.
|
||||
; Note: This value is mandatory.
|
||||
pm = dynamic
|
||||
|
||||
; The number of child processes to be created when pm is set to 'static' and the
|
||||
; maximum number of child processes when pm is set to 'dynamic' or 'ondemand'.
|
||||
; This value sets the limit on the number of simultaneous requests that will be
|
||||
; served. Equivalent to the ApacheMaxClients directive with mpm_prefork.
|
||||
; Equivalent to the PHP_FCGI_CHILDREN environment variable in the original PHP
|
||||
; CGI. The below defaults are based on a server without much resources. Don't
|
||||
; forget to tweak pm.* to fit your needs.
|
||||
; Note: Used when pm is set to 'static', 'dynamic' or 'ondemand'
|
||||
; Note: This value is mandatory.
|
||||
pm.max_children = 20
|
||||
|
||||
; The number of child processes created on startup.
|
||||
; Note: Used only when pm is set to 'dynamic'
|
||||
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
|
||||
pm.start_servers = 2
|
||||
|
||||
; The desired minimum number of idle server processes.
|
||||
; Note: Used only when pm is set to 'dynamic'
|
||||
; Note: Mandatory when pm is set to 'dynamic'
|
||||
pm.min_spare_servers = 1
|
||||
|
||||
; The desired maximum number of idle server processes.
|
||||
; Note: Used only when pm is set to 'dynamic'
|
||||
; Note: Mandatory when pm is set to 'dynamic'
|
||||
pm.max_spare_servers = 3
|
||||
|
||||
;---------------------
|
||||
|
||||
; Make specific Docker environment variables available to PHP
|
||||
env[APP_ENV] = $APP_ENV
|
||||
env[APP_SECRET] = $APP_SECRET
|
||||
env[APP_JWT_PUB_KEY] = $APP_JWT_PUB_KEY
|
||||
env[APP_JWT_PRIV_KEY] = $APP_JWT_PRIV_KEY
|
||||
env[APP_DEBUG] = $APP_DEBUG
|
||||
env[INFLUXDB_URL] = $INFLUXDB_URL
|
||||
env[INFLUXDB_TOKEN] = $INFLUXDB_TOKEN
|
||||
env[INFLUXDB_ORG] = $INFLUXDB_ORG
|
||||
env[INFLUXDB_BUCKET] = $INFLUXDB_BUCKET
|
||||
env[INFLUXDB_SSL] = $INFLUXDB_SSL
|
||||
env[DATABASE_URL] = $DATABASE_URL
|
||||
env[REDIS_URL] = $REDIS_URL
|
||||
env[LDAP_URL] = $LDAP_URL
|
||||
env[LDAP_PW] = $LDAP_PW
|
||||
env[CORS_ALLOW_ORIGIN] = $CORS_ALLOW_ORIGIN
|
||||
|
||||
; Catch worker output
|
||||
|
||||
catch_workers_output = yes
|
||||
|
||||
; Increase PHP memory limit (Default: 128M)
|
||||
; Note: Required for loading large jobs from InfluxDB (>16 Nodes && >12h Duration)
|
||||
|
||||
php_admin_value[memory_limit] = 1024M
|
||||
@ -1,7 +0,0 @@
|
||||
zend_extension=xdebug.so
|
||||
|
||||
[Xdebug]
|
||||
xdebug.mode=${PHP_XDEBUG_MODE}
|
||||
xdebug.client_port=${PHP_XDEBUG_CLIENT_PORT}
|
||||
xdebug.client_host=${PHP_XDEBUG_CLIENT_HOST}
|
||||
xdebug.start_with_request=yes
|
||||
100
scripts/checkpointsToInflux.pl
Normal file
100
scripts/checkpointsToInflux.pl
Normal file
@ -0,0 +1,100 @@
|
||||
#!/usr/bin/env perl
|
||||
use strict;
|
||||
use warnings;
|
||||
use utf8;
|
||||
|
||||
use File::Path qw( make_path rmtree );
|
||||
use Cpanel::JSON::XS qw( decode_json encode_json );
|
||||
use File::Slurp;
|
||||
use Data::Dumper;
|
||||
use Time::Piece;
|
||||
use Sort::Versions;
|
||||
use REST::Client;
|
||||
|
||||
### INFLUXDB
|
||||
my $newCheckpoints = './data/cc-metric-store/checkpoints';
|
||||
my @CheckpClusters;
|
||||
my $verbose = 1;
|
||||
my $restClient = REST::Client->new();
|
||||
$restClient->setHost('http://localhost:8086'); # Adapt port here!
|
||||
$restClient->addHeader('Authorization', "Token 74008ea2a8dad5e6f856838a90c6392e"); # compare .env file
|
||||
$restClient->addHeader('Content-Type', 'text/plain; charset=utf-8');
|
||||
$restClient->addHeader('Accept', 'application/json');
|
||||
$restClient->getUseragent()->ssl_opts(SSL_verify_mode => 0); # Temporary: Disable Cert Check
|
||||
$restClient->getUseragent()->ssl_opts(verify_hostname => 0); # Temporary: Disable Cert Check
|
||||
|
||||
# Get clusters by cc-metric-store/$subfolder
|
||||
opendir my $dhc, $newCheckpoints or die "can't open directory: $!";
|
||||
while ( readdir $dhc ) {
|
||||
chomp; next if $_ eq '.' or $_ eq '..' or $_ eq 'job-archive';
|
||||
|
||||
my $cluster = $_;
|
||||
push @CheckpClusters, $cluster;
|
||||
}
|
||||
|
||||
# start to read checkpoints for influx
|
||||
foreach my $cluster ( @CheckpClusters ) {
|
||||
print "Starting to read updated checkpoint-files into influx for $cluster\n";
|
||||
|
||||
opendir my $dhLevel1, "$newCheckpoints/$cluster" or die "can't open directory: $!";
|
||||
while ( readdir $dhLevel1 ) {
|
||||
chomp; next if $_ eq '.' or $_ eq '..';
|
||||
my $level1 = $_;
|
||||
|
||||
if ( -d "$newCheckpoints/$cluster/$level1" ) {
|
||||
my $nodeSource = "$newCheckpoints/$cluster/$level1/";
|
||||
my @files = read_dir($nodeSource);
|
||||
my $length = @files;
|
||||
if (!@files || $length != 14) { # needs 14 files == 7 days worth of data
|
||||
next;
|
||||
}
|
||||
my @sortedFiles = sort { versioncmp($a,$b) } @files; # sort alphanumerically: _Really_ start with index == 0 == 1609459200.json
|
||||
my $nodeMeasurement;
|
||||
|
||||
foreach my $file (@sortedFiles) {
|
||||
# print "$file\n";
|
||||
my $rawstr = read_file("$nodeSource/$file");
|
||||
my $json = decode_json($rawstr);
|
||||
my $fileMeasurement;
|
||||
|
||||
foreach my $metric (keys %{$json->{metrics}}) {
|
||||
my $start = $json->{metrics}->{$metric}->{start};
|
||||
my $timestep = $json->{metrics}->{$metric}->{frequency};
|
||||
my $data = $json->{metrics}->{$metric}->{data};
|
||||
my $length = @$data;
|
||||
my $measurement;
|
||||
|
||||
while (my ($index, $value) = each(@$data)) {
|
||||
if ($value) {
|
||||
my $timestamp = $start + ($timestep * $index);
|
||||
$measurement .= "$metric,cluster=$cluster,hostname=$level1,type=node value=".$value." $timestamp"."\n";
|
||||
}
|
||||
}
|
||||
# Use v2 API for Influx2
|
||||
if ($measurement) {
|
||||
# print "Adding: #VALUES $length KEY $metric"."\n";
|
||||
$fileMeasurement .= $measurement;
|
||||
}
|
||||
}
|
||||
if ($fileMeasurement) {
|
||||
$nodeMeasurement .= $fileMeasurement;
|
||||
}
|
||||
}
|
||||
|
||||
$restClient->POST("/api/v2/write?org=ClusterCockpit&bucket=ClusterCockpit&precision=s", "$nodeMeasurement"); # compare .env for bucket and org
|
||||
my $responseCode = $restClient->responseCode();
|
||||
|
||||
if ( $responseCode eq '204') {
|
||||
if ( $verbose ) {
|
||||
print "INFLUX API WRITE: CLUSTER $cluster HOST $level1"."\n";
|
||||
};
|
||||
} else {
|
||||
if ( $responseCode ne '422' ) { # Exclude High Frequency Error 422 - Temporary!
|
||||
my $response = $restClient->responseContent();
|
||||
print "INFLUX API WRITE ERROR CODE ".$responseCode.": ".$response."\n";
|
||||
};
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
print "Done for influx\n";
|
||||
12
scripts/sqliteToMariadb.sh
Normal file
12
scripts/sqliteToMariadb.sh
Normal file
@ -0,0 +1,12 @@
|
||||
#!/bin/bash
|
||||
|
||||
echo "Will run prerequisites 'apt install python3-pip' and 'pip install sqlite3-to-mysql'"
|
||||
|
||||
sudo apt install python3-pip
|
||||
|
||||
pip install sqlite3-to-mysql
|
||||
|
||||
echo "'sqlite3mysql' requires running DB container, will fail otherwise."
|
||||
|
||||
# -f FILE -d DBNAME -u USER -h HOST -P PORT
|
||||
~/.local/bin/sqlite3mysql -f job.db -d ClusterCockpit -u root --mysql-password root -h localhost -P 3306
|
||||
82
setupDev.sh
Executable file
82
setupDev.sh
Executable file
@ -0,0 +1,82 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Check cc-backend, touch job.db if exists
|
||||
if [ ! -d cc-backend ]; then
|
||||
echo "'cc-backend' not yet prepared! Please clone cc-backend repository before starting this script."
|
||||
echo -n "Stopped."
|
||||
exit
|
||||
else
|
||||
cd cc-backend
|
||||
if [ ! -d var ]; then
|
||||
mkdir var
|
||||
touch var/job.db
|
||||
else
|
||||
echo "'cc-backend/var' exists. Cautiously exiting."
|
||||
echo -n "Stopped."
|
||||
exit
|
||||
fi
|
||||
fi
|
||||
|
||||
|
||||
# Download unedited job-archive to ./data/job-archive-source
|
||||
if [ ! -d data/job-archive-source ]; then
|
||||
cd data
|
||||
wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/job-archive.tar.xz
|
||||
tar xJf job-archive.tar.xz
|
||||
mv ./job-archive ./job-archive-source
|
||||
rm ./job-archive.tar.xz
|
||||
cd ..
|
||||
else
|
||||
echo "'data/job-archive-source' already exists!"
|
||||
fi
|
||||
|
||||
# Download unedited checkpoint files to ./data/cc-metric-store-source/checkpoints
|
||||
if [ ! -d data/cc-metric-store-source ]; then
|
||||
mkdir -p data/cc-metric-store-source/checkpoints
|
||||
cd data/cc-metric-store-source/checkpoints
|
||||
wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/cc-metric-store-checkpoints.tar.xz
|
||||
tar xf cc-metric-store-checkpoints.tar.xz
|
||||
rm cc-metric-store-checkpoints.tar.xz
|
||||
cd ../../../
|
||||
else
|
||||
echo "'data/cc-metric-store-source' already exists!"
|
||||
fi
|
||||
|
||||
# Update timestamps
|
||||
perl ./migrateTimestamps.pl
|
||||
|
||||
# Create archive folder for rewrtitten ccms checkpoints
|
||||
if [ ! -d data/cc-metric-store/archive ]; then
|
||||
mkdir -p data/cc-metric-store/archive
|
||||
fi
|
||||
|
||||
# cleanup sources
|
||||
# rm -r ./data/job-archive-source
|
||||
# rm -r ./data/cc-metric-store-source
|
||||
|
||||
# prepare folders for influxdb2
|
||||
if [ ! -d data/influxdb ]; then
|
||||
mkdir -p data/influxdb/data
|
||||
mkdir -p data/influxdb/config/influx-configs
|
||||
else
|
||||
echo "'data/influxdb' already exists!"
|
||||
fi
|
||||
|
||||
# Check dotenv-file and docker-compose-yml, copy accordingly if not present and build docker services
|
||||
if [ ! -d .env ]; then
|
||||
cp templates/env.default ./.env
|
||||
fi
|
||||
|
||||
if [ ! -d docker-compose.yml ]; then
|
||||
cp templates/docker-compose.yml.default ./docker-compose.yml
|
||||
fi
|
||||
|
||||
docker-compose build
|
||||
./cc-backend/cc-backend --init-db --add-user demo:admin:AdminDev
|
||||
docker-compose up -d
|
||||
|
||||
echo ""
|
||||
echo "Setup complete, containers are up by default: Shut down with 'docker-compose down'."
|
||||
echo "Use './cc-backend/cc-backend' to start cc-backend."
|
||||
echo "Use scripts in /scripts to load data into influx or mariadb."
|
||||
# ./cc-backend/cc-backend
|
||||
75
templates/docker-compose.yml.default
Normal file
75
templates/docker-compose.yml.default
Normal file
@ -0,0 +1,75 @@
|
||||
services:
|
||||
nats:
|
||||
container_name: cc-nats
|
||||
image: nats:alpine
|
||||
ports:
|
||||
- "4222:4222"
|
||||
- "8222:8222"
|
||||
|
||||
cc-metric-store:
|
||||
container_name: cc-metric-store
|
||||
build:
|
||||
context: ./cc-metric-store
|
||||
ports:
|
||||
- "8084:8084"
|
||||
volumes:
|
||||
- ${DATADIR}/cc-metric-store:/data
|
||||
depends_on:
|
||||
- nats
|
||||
|
||||
influxdb:
|
||||
container_name: cc-influxdb
|
||||
image: influxdb
|
||||
command: ["--reporting-disabled"]
|
||||
environment:
|
||||
DOCKER_INFLUXDB_INIT_MODE: setup
|
||||
DOCKER_INFLUXDB_INIT_USERNAME: devel
|
||||
DOCKER_INFLUXDB_INIT_PASSWORD: ${INFLUXDB_PASSWORD}
|
||||
DOCKER_INFLUXDB_INIT_ORG: ${INFLUXDB_ORG}
|
||||
DOCKER_INFLUXDB_INIT_BUCKET: ${INFLUXDB_BUCKET}
|
||||
DOCKER_INFLUXDB_INIT_RETENTION: 100w
|
||||
DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: ${INFLUXDB_ADMIN_TOKEN}
|
||||
ports:
|
||||
- "127.0.0.1:${INFLUXDB_PORT}:8086"
|
||||
volumes:
|
||||
- ${DATADIR}/influxdb/data:/var/lib/influxdb2
|
||||
- ${DATADIR}/influxdb/config:/etc/influxdb2
|
||||
|
||||
openldap:
|
||||
container_name: cc-ldap
|
||||
image: osixia/openldap:1.5.0
|
||||
command: --copy-service --loglevel debug
|
||||
environment:
|
||||
- LDAP_ADMIN_PASSWORD=${LDAP_ADMIN_PASSWORD}
|
||||
- LDAP_ORGANISATION=${LDAP_ORGANISATION}
|
||||
- LDAP_DOMAIN=${LDAP_DOMAIN}
|
||||
volumes:
|
||||
- ${DATADIR}/ldap:/container/service/slapd/assets/config/bootstrap/ldif/custom
|
||||
|
||||
db:
|
||||
container_name: cc-db
|
||||
image: mariadb:latest
|
||||
command: ["--default-authentication-plugin=mysql_native_password"]
|
||||
environment:
|
||||
MARIADB_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
|
||||
MARIADB_DATABASE: ${MARIADB_DATABASE}
|
||||
MARIADB_USER: ${MARIADB_USER}
|
||||
MARIADB_PASSWORD: ${MARIADB_PASSWORD}
|
||||
ports:
|
||||
- "127.0.0.1:${MARIADB_PORT}:3306"
|
||||
# volumes:
|
||||
# - ${DATADIR}/sql-init:/docker-entrypoint-initdb.d
|
||||
cap_add:
|
||||
- SYS_NICE
|
||||
|
||||
phpmyadmin:
|
||||
container_name: cc-phpmyadmin
|
||||
image: phpmyadmin
|
||||
environment:
|
||||
- PMA_HOST=cc-db
|
||||
- PMA_USER=root
|
||||
- PMA_PASSWORD=${MARIADB_ROOT_PASSWORD}
|
||||
ports:
|
||||
- "127.0.0.1:${PHPMYADMIN_PORT}:80"
|
||||
depends_on:
|
||||
- db
|
||||
29
templates/docker-compose.yml.mysql
Normal file
29
templates/docker-compose.yml.mysql
Normal file
@ -0,0 +1,29 @@
|
||||
services:
|
||||
db:
|
||||
container_name: cc-db
|
||||
image: mysql:8.0.22
|
||||
command: ["--default-authentication-plugin=mysql_native_password"]
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
|
||||
MYSQL_DATABASE: ${MYSQL_DATABASE}
|
||||
MYSQL_USER: ${MYSQL_USER}
|
||||
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
|
||||
ports:
|
||||
- "127.0.0.1:${MYSQL_PORT}:3306"
|
||||
# volumes:
|
||||
# - ${DATADIR}/sql-init:/docker-entrypoint-initdb.d
|
||||
# - ${DATADIR}/sqldata:/var/lib/mysql
|
||||
cap_add:
|
||||
- SYS_NICE
|
||||
|
||||
phpmyadmin:
|
||||
container_name: cc-phpmyadmin
|
||||
image: phpmyadmin
|
||||
environment:
|
||||
- PMA_HOST=cc-db
|
||||
- PMA_USER=root
|
||||
- PMA_PASSWORD=${MYSQL_ROOT_PASSWORD}
|
||||
ports:
|
||||
- "127.0.0.1:${PHPMYADMIN_PORT}:80"
|
||||
depends_on:
|
||||
- db
|
||||
40
templates/env.default
Normal file
40
templates/env.default
Normal file
@ -0,0 +1,40 @@
|
||||
########################################################################
|
||||
# CCBACKEND DEVEL DOCKER SETTINGS
|
||||
########################################################################
|
||||
|
||||
########################################################################
|
||||
# INFLUXDB
|
||||
########################################################################
|
||||
INFLUXDB_PORT=8086
|
||||
INFLUXDB_PASSWORD=1bc8777daad29d2f05eb77b7571fd8a1
|
||||
INFLUXDB_ADMIN_TOKEN=74008ea2a8dad5e6f856838a90c6392e
|
||||
INFLUXDB_ORG=ClusterCockpit
|
||||
INFLUXDB_BUCKET=ClusterCockpit
|
||||
# Whether or not to check SSL Cert in Symfony Client, Default: false
|
||||
INFLUXDB_SSL=false
|
||||
|
||||
########################################################################
|
||||
# MARIADB
|
||||
########################################################################
|
||||
MARIADB_ROOT_PASSWORD=root
|
||||
MARIADB_DATABASE=ClusterCockpit
|
||||
MARIADB_USER=clustercockpit
|
||||
MARIADB_PASSWORD=clustercockpit
|
||||
MARIADB_PORT=3306
|
||||
|
||||
#########################################
|
||||
# LDAP
|
||||
########################################################################
|
||||
LDAP_ADMIN_PASSWORD=mashup
|
||||
LDAP_ORGANISATION=NHR@FAU
|
||||
LDAP_DOMAIN=rrze.uni-erlangen.de
|
||||
|
||||
########################################################################
|
||||
# PHPMyAdmin
|
||||
########################################################################
|
||||
PHPMYADMIN_PORT=8081
|
||||
|
||||
########################################################################
|
||||
# INTERNAL SETTINGS
|
||||
########################################################################
|
||||
DATADIR=./data
|
||||
17
templates/env.mysql
Normal file
17
templates/env.mysql
Normal file
@ -0,0 +1,17 @@
|
||||
########################################################################
|
||||
# ADDITIONAL ENV VARIABLES FOR MYSQL AND PHPMYADMIN CONTAINERS
|
||||
########################################################################
|
||||
|
||||
########################################################################
|
||||
# MySQL
|
||||
########################################################################
|
||||
MYSQL_ROOT_PASSWORD=root
|
||||
MYSQL_DATABASE=ClusterCockpit
|
||||
MYSQL_USER=clustercockpit
|
||||
MYSQL_PASSWORD=clustercockpit
|
||||
MYSQL_PORT=3306
|
||||
|
||||
########################################################################
|
||||
# PHPMyAdmin
|
||||
########################################################################
|
||||
PHPMYADMIN_PORT=8081
|
||||
Loading…
Reference in New Issue
Block a user