cc-docker/README.md

# cc-docker

This is a `docker-compose` setup which provides a quickly started environment
for ClusterCockpit development and testing, using `cc-backend`. A number of
services is readily available as docker container (nats, cc-metric-store,
InfluxDB, LDAP, SLURM), or easily added by manual configuration (MariaDB).

It includes the following containers:

| Service full name | docker service name | port             |
| ----------------- | ------------------- | ---------------- |
| Slurm Controller  | slurmctld           | 6818             |
| Slurm Database    | slurmdbd            | 6817             |
| Slurm Rest (JWT)  | slurmrestd          | 6820             |
| Slurm Worker      | node01              | 6818             |
| NATS service      | nats                | 4222, 6222, 8222 |
| cc-metric-store   | cc-metric-store     | 8084             |
| KeyCloak          | keycloak            | 8080             |
| OpenLDAP          | openldap            | 389, 636         |

The setup comes with fixture data for a Job archive, cc-metric-store
checkpoints, and a LDAP user directory.

## Prerequisites

For all the docker services to work correctly, you will need the following tools
installed:

1. `docker` and `docker-compose`
2. `golang` (for compiling cc-metric-store)
3. `perl` (for migrateTimestamp.pl) with Cpanel::JSON::XS, Data::Dumper, Time::Piece, Sort::Versions and File::Slurp perl modules.
4. `npm` (for cc-backend)
5. `make` (for building slurm base image)

It is also recommended to add docker service to sudo user group since the
setupDev.sh script assumes sudo permissions for docker and docker-compose
services.

You can use:

```
sudo groupadd docker
sudo usermod -aG docker $USER

# restart after adding your docker with your user to sudo group
sudo shutdown -r -t 0
```

Note: You can install all these dependencies via predefined installation steps
in `prerequisite_installation_script.sh`.

If you are using different Linux flavors, you will have to adapt
`prerequisite_installation_script.sh` as well as `setupDev.sh`.

## Setup Procedure

1. Clone `cc-backend` repository in chosen base folder: `$> git clone https://github.com/ClusterCockpit/cc-backend.git`

2. Run the setup bash file: `$> ./setupDev.sh`: **NOTICE** The script will
   download files of a total size of 338MB (mostly for the cc-metric-store
   data).

3. The setup-script launches the supporting container stack in the background
   automatically if everything went well. Run

```bash
./cc-backend/cc-backend -server -dev
```

to start `cc-backend`.

1. By default, you can access `cc-backend` in your browser at
   `http://localhost:8088` (port 8080 is taken by the KeyCloak container). You
   can shut down the cc-backend server by pressing
   `CTRL-C`, remember to also shut down all containers via `$> docker-compose down`
   afterwards.

2. You can restart the containers with: `$> docker-compose up -d`.

## Credentials for logging into clustercockpit

Credentials for the preconfigured demo user are:

- User: `demo`
- Password: `demo`

The LDAP user directory is defined in `./data/ldap/add_users.ldif`. Every
account's password equals its user name:

| User        | Group        | Role in ClusterCockpit |
| ----------- | ------------ | ---------------------- |
| `ldapuser`  | –            | `user`                 |
| `ccuser`    | –            | `user`                 |
| `ccadmin`   | `cc-admin`   | `user`, `admin`        |
| `ccsupport` | `cc-support` | `user`                 |
| `ccmanager` | `cc-manager` | `user`                 |
| `ccapi`     | `cc-api`     | `user`                 |

### LDAP role sync

cc-backend can derive elevated roles from LDAP group membership via
`auth.ldap.role-filters` in `config.json`. Each entry maps a role to an LDAP
filter that is evaluated against the *user* entry; the preconfigured setup maps
the `cc-admin` group to the `admin` role:

```json
"role-filters": {
  "admin": "(memberOf=cn=cc-admin,ou=groups,dc=example,dc=com)"
}
```

The `memberOf` attribute is maintained automatically by the `memberof` overlay
of the openldap image, which is configured for `groupOfUniqueNames` /
`uniqueMember` — the group entries in the ldif use those classes accordingly.

LDAP is authoritative for every role listed in `role-filters`: a role is granted
when the filter matches and revoked when it no longer does. Roles that are not
listed are never touched. To exercise the other groups, add them as well:

```json
"role-filters": {
  "admin": "(memberOf=cn=cc-admin,ou=groups,dc=example,dc=com)",
  "support": "(memberOf=cn=cc-support,ou=groups,dc=example,dc=com)",
  "manager": "(memberOf=cn=cc-manager,ou=groups,dc=example,dc=com)",
  "api": "(memberOf=cn=cc-api,ou=groups,dc=example,dc=com)"
}
```

Roles are applied when the account is first synced (`sync-user-on-login`) and
refreshed on every subsequent login (`update-user-on-login`), as well as by the
periodic sync task (`sync-interval`).

> Note: The ldif is only applied when the openldap container initialises its
> database. After changing it, recreate the container and its volumes with
> `docker compose rm -sfv openldap`, or load the changes into the running
> directory manually with `ldapadd`.

### OIDC login via KeyCloak

The KeyCloak container imports the realm `clustercockpit` from
`./keycloak/import/clustercockpit-realm.json` on first start. It defines the
confidential client `cc-backend`, the realm roles `cc-admin`, `cc-support`,
`cc-manager` and `cc-api`, and these accounts (password equals user name):

| User        | Realm role   | Role in ClusterCockpit |
| ----------- | ------------ | ---------------------- |
| `kcuser`    | –            | `user`                 |
| `kcadmin`   | `cc-admin`   | `admin`                |
| `kcsupport` | `cc-support` | `user`                 |
| `kcmanager` | `cc-manager` | `user`                 |
| `kcapi`     | `cc-api`     | `user`                 |

The KeyCloak admin console is at `http://localhost:8080` (`admin` / `admin`).

The `auth.oidc.role-mapping` section of `config.json` translates realm roles
into ClusterCockpit roles. The preconfigured setup maps `cc-admin` to `admin`:

```json
"oidc": {
  "provider": "http://localhost:8080/realms/clustercockpit",
  "client-id": "cc-backend",
  "client-secret": "cc-backend-dev-secret",
  "sync-user-on-login": true,
  "update-user-on-login": true,
  "role-mapping": {
    "cc-admin": "admin"
  }
}
```

Unlike the LDAP role filters, `role-mapping` is the *sole* source of roles for
OIDC logins: a role in the token grants a ClusterCockpit role only if it is
listed here, and unmapped roles are ignored — literal role names such as `admin`
would also have to be mapped explicitly. Accounts without a mapped role get
`user`. Add the remaining groups to exercise them:

```json
"role-mapping": {
  "cc-admin": "admin",
  "cc-support": "support",
  "cc-manager": "manager",
  "cc-api": "api"
}
```

Roles are read from the `realm_access.roles` claim of the **ID token**. The
built-in KeyCloak `roles` client scope only adds that claim to the *access*
token, so the imported client carries its own `realm roles in id token`
protocol mapper with `id.token.claim` enabled. Keep that mapper when editing
the realm, otherwise every OIDC login falls back to the plain `user` role.

> Note: `cc-backend` resolves the OIDC provider at startup and aborts if it is
> unreachable, so the KeyCloak container has to be up before starting the
> server. The realm is only imported while the KeyCloak database is empty —
> to re-import after editing the realm file, recreate KeyCloak *and* its
> database with `docker compose rm -sf keycloak postgres`.

## Preconfigured setup between docker services and ClusterCockpit components

When you are done cloning the cc-backend repo and once you execute `setupDev.sh` file, it will copy a preconfigured `config.json` from `misc/config.json` and replace the `cc-backend/config.json`, which will be used by cc-backend, once you start the server.
The preconfigured config.json attaches to:

### 1. OpenLDAP docker service on port 389

### 2. cc-metric-store docker service on port 8084

### 3. cc-slurm-adapter is running on slurmctld docker service

cc-metric-store also has a preconfigured `config.json` in
`cc-metric-store/config.json` which attaches to NATS docker service on port 4222
and subscribes to topic 'hpc-nats'.

Basically, all the ClusterCockpit components and the docker services attach to
each other like lego pieces.

## Docker commands to access the services

> Note: You need to be in cc-docker directory in order to execute any docker command

You can view all docker processes running on either of the VM instance by using
this command:

```
docker ps
```

Now that you can see the docker services, and if you want to manually access the
docker services, you have to run **`bash`** command in those running services.

> **`Example`**: You want to run slurm commands like `sinfo` or `squeue` or
> `scontrol` on slurm controller, you cannot directly access it.

You need to open a **`bash`** session in the running service by using the following command:

```
$ docker exec -it <docker service name> bash

#example
$ docker exec -it slurmctld bash

#or
$ docker exec -it cc-metric-store bash
```

Once you start a **`bash`** on any docker service, then you may execute any
service related commands in that **`bash`**.

But for Cluster Cockpit development, you only need ports to access these docker
services. You have to use `localhost:<port>` when trying to access any docker
service. You may need to configure the `cc-backend/config.json` based on these
docker services and ports.

## Slurm setup in cc-docker

### 1. Slurm controller

Currently slurm controller is aware of the 1 node that we have setup in our mini
cluster i.e. node01.

In order to execute slurm commands, you may need to **`bash`** into the
**`slurmctld`** docker service.

```bash
docker exec -it slurmctld bash
```

Then you may be able to run slurm controller commands. A few examples without
output are:

```bash
sinfo
```

or

```bash
squeue
```

or

```bash
scontrol show nodes
```

### 2. Slurm rest service

You do not need to **`bash`** into the slurmrestd service but can directly
access the rest API via localhost:6820. A simple example on how to CURL to the
slurm rest API is given in the `curl_slurmrestd.sh`.

You can directly use `curl_slurmrestd.sh` with a never expiring JWT token ( can
be found in /data/slurm/secret/jwt_token.txt )

You may also use the never expiring token directly from the file for any of your
custom CURL commands.

## Known Issues

- `docker-compose` installed on Ubuntu (18.04, 20.04) via `apt-get` can not correctly parse `docker-compose.yml` due to version differences. Install latest version of `docker-compose` from <https://docs.docker.com/compose/install/> instead.
- You need to ensure that no other web server is running on ports 8088 (cc-backend), 8080 (KeyCloak), 8084 (cc-metric-store), 4222 and 8222 (Nats). If one or more ports are already in use, you have to adapt the related config accordingly.
- Existing VPN connections sometimes cause problems with docker. If `docker-compose` does not start up correctly, try disabling any active VPN connection. Refer to <https://stackoverflow.com/questions/45692255/how-make-openvpn-work-with-docker> for further information.

## Docker services and restarting the services

You can find all the docker services in `docker-compose.yml`. Feel free to
modify it.

Whenever you modify it, please use

```bash
docker compose down
```

in order to shut down all the services in all the VM’s (maininstance,
nodeinstance, nodeinstance2) and then start all the services by using

```bash
docker compose up
```

TODO: Update job archive and all other metric data.
The job archive with 1867 jobs originates from the second half of 2020.
Roughly 2700 jobs from the first week of 2021 are loaded with data from InfluxDB.
Some views of ClusterCockpit (e.g. the Users view) show the last week or month.
To show some data there you have to set the filter to time periods with jobs
(August 2020 to January 2021).