Merge pull request #16 from ClusterCockpit/refactor-structure-dev

Refactor directory structure according to Golang common standards.
Add more documentation in form of READMEs.

.gitignore

@@ -1,5 +1,4 @@
-# executable:
+/cc-backend
-cc-backend
 
 /var/job-archive
 /var/*.db
@@ -8,3 +7,5 @@ cc-backend
 /.env
 /config.json
 
+/web/frontend/public/build
+/web/frontend/node_modules

.gitmodules

@@ -1,5 +0,0 @@
-[submodule "frontend"]
-	path = frontend
-	url = git@github.com:ClusterCockpit/cc-frontend.git
-    branch = main
-    update = merge

README.md

@@ -3,21 +3,27 @@
 [![Build](https://github.com/ClusterCockpit/cc-backend/actions/workflows/test.yml/badge.svg)](https://github.com/ClusterCockpit/cc-backend/actions/workflows/test.yml)
 
 This is a Golang backend implementation for a REST and GraphQL API according to the [ClusterCockpit specifications](https://github.com/ClusterCockpit/cc-specifications).
-It also includes a web interface for ClusterCockpit based on the components implemented in
+It also includes a web interface for ClusterCockpit.
-[cc-frontend](https://github.com/ClusterCockpit/cc-frontend), which is included as a git submodule.
 This implementation replaces the previous PHP Symfony based ClusterCockpit web-interface.
+[Here](https://github.com/ClusterCockpit/ClusterCockpit/wiki/Why-we-switched-from-PHP-Symfony-to-a-Golang-based-solution) is a discussion of the reasons why we switched from PHP Symfony to a Golang based solution.
 
 ## Overview
 
 This is a golang web backend for the ClusterCockpit job-specific performance monitoring framework.
 It provides a REST API for integrating ClusterCockpit with a HPC cluster batch system and external analysis scripts.
 Data exchange between the web frontend and backend is based on a GraphQL API.
-The web frontend is also served by the backend using [Svelte](https://svelte.dev/) components implemented in [cc-frontend](https://github.com/ClusterCockpit/cc-frontend).
+The web frontend is also served by the backend using [Svelte](https://svelte.dev/) components.
 Layout and styling is based on [Bootstrap 5](https://getbootstrap.com/) using [Bootstrap Icons](https://icons.getbootstrap.com/).
-The backend uses [SQLite 3](https://sqlite.org/) as relational SQL database by default. It can optionally use a MySQL/MariaDB database server.
+The backend uses [SQLite 3](https://sqlite.org/) as relational SQL database by default.
-Finished batch jobs are stored in a so called job archive following [this specification](https://github.com/ClusterCockpit/cc-specifications/tree/master/job-archive).
+It can optionally use a MySQL/MariaDB database server.
+Finished batch jobs are stored in a file-based job archive following [this specification](https://github.com/ClusterCockpit/cc-specifications/tree/master/job-archive).
 The backend supports authentication using local accounts or an external LDAP directory.
-Authorization for APIs is implemented using [JWT](https://jwt.io/) tokens created with  public/private key encryption.
+Authorization for APIs is implemented using [JWT](https://jwt.io/) tokens created with public/private key encryption.
+
+You find more detailed information here:
+* `./configs/README.md`: Infos about configuration and setup of cc-backend.
+* `./init/README.md`: Infos on how to setup cc-backend as systemd service on Linux.
+* `./tools/README.md`: Infos on the JWT authorizatin token workflows in ClusterCockpit.
 
 ## Demo Setup
 
@@ -25,30 +31,28 @@ We provide a shell skript that downloads demo data and automatically builds and
 You need `wget`, `go`, and `yarn` in your path to start the demo. The demo will download 32MB of data (223MB on disk).
 
 ```sh
-# The frontend is a submodule, so use `--recursive`
+git clone git@github.com:ClusterCockpit/cc-backend.git
-git clone --recursive git@github.com:ClusterCockpit/cc-backend.git
 
 ./startDemo.sh
 ```
-You can access the web interface at http://localhost:8080. Credentials for login: `demo:AdminDev`. Please note that some views do not work without a metric backend (e.g., the Systems view).
+You can access the web interface at http://localhost:8080.
+Credentials for login: `demo:AdminDev`.
+Please note that some views do not work without a metric backend (e.g., the Systems and Status view).
 
 ## Howto Build and Run
 
 ```sh
-# The frontend is a submodule, so use `--recursive`
+git clone git@github.com:ClusterCockpit/cc-backend.git
-git clone --recursive git@github.com:ClusterCockpit/cc-backend.git
 
 # Prepare frontend
-cd ./cc-backend/frontend
+cd ./cc-backend/web/frontend
 yarn install
 yarn build
 
 cd ..
 go get
-go build
+go build cmd/cc-backend
 
-# The job-archive directory must be organised the same way as
-# as for the regular ClusterCockpit.
 ln -s <your-existing-job-archive> ./var/job-archive
 
 # Create empty job.db (Will be initialized as SQLite3 database)
@@ -56,6 +60,7 @@ touch ./var/job.db
 
 # EDIT THE .env FILE BEFORE YOU DEPLOY (Change the secrets)!
 # If authentication is disabled, it can be empty.
+cp configs/env-template.txt  .env
 vim ./.env
 
 # This will first initialize the job.db database by traversing all
@@ -71,57 +76,40 @@ vim ./.env
 ```
 ### Run as systemd daemon
 
-In order to run this program as a daemon, look at [utils/systemd/README.md](./utils/systemd/README.md) where a systemd unit file and more explanation is provided.
+In order to run this program as a daemon, cc-backend ships with an [example systemd setup](./init/README.md).
 
 ## Configuration and Setup
 
-cc-backend can be used as a local web-interface for an existing job archive or
+cc-backend can be used as a local web-interface for an existing job archive or as a general web-interface server for a live ClusterCockpit Monitoring framework.
-as a general web-interface server for a live ClusterCockpit Monitoring
-framework.
 
-Create your job-archive according to [this specification](https://github.com/ClusterCockpit/cc-specifications). At least
+Create your job-archive according to [this specification](https://github.com/ClusterCockpit/cc-specifications/tree/master/job-archive).
-one cluster with a valid `cluster.json` file is required. Having no jobs in the
+At least one cluster with a valid `cluster.json` file is required.
-job-archive at all is fine. You may use the sample job-archive available for
+Having no jobs in the job-archive at all is fine.
-download [in cc-docker/develop](https://github.com/ClusterCockpit/cc-docker/tree/develop).
 
 ### Configuration
 
 A config file in the JSON format can be provided using `--config` to override the defaults.
-Look at the beginning of `server.go` for the defaults and consequently the format of the configuration file.
+You find documentation of all supported configuration and command line options [here](./configs.README.md).
 
 ### Update GraphQL schema
 
-This project uses [gqlgen](https://github.com/99designs/gqlgen) for the GraphQL
+This project uses [gqlgen](https://github.com/99designs/gqlgen) for the GraphQL API.
-API. The schema can be found in `./graph/schema.graphqls`. After changing it,
+The schema can be found in `./api/schema.graphqls`.
-you need to run `go run github.com/99designs/gqlgen` which will update
+After changing it, you need to run `go run github.com/99designs/gqlgen` which will update `./internal/graph/model`.
-`graph/model`. In case new resolvers are needed, they will be inserted into
+In case new resolvers are needed, they will be inserted into `./internal/graph/schema.resolvers.go`, where you will need to implement them.
-`graph/schema.resolvers.go`, where you will need to implement them.
 
 ## Project Structure
 
-- `api/` contains the REST API. The routes defined there should be called whenever a job starts/stops. The API is documented in the OpenAPI 3.0 format in [./api/openapi.yaml](./api/openapi.yaml).
+- `api/` contains the API schema files for the REST and GraphQL APIs. The REST API is documented in the OpenAPI 3.0 format in [./api/openapi.yaml](./api/openapi.yaml).
-- `auth/` is where the (optional) authentication middleware can be found, which adds the currently authenticated user to the request context. The `user` table is created and managed here as well.
+- `cmd/cc-backend` contains `main.go` for the main application.
-  - `auth/ldap.go` contains everything to do with automatically syncing and authenticating users form an LDAP server.
+- `configs/` contains documentation about configuration and command line options and required environment variables. An example configuration file is provided.
-- `config` handles the `cluster.json` files and the user-specific configurations (changeable via GraphQL) for the Web-UI such as the selected metrics etc.
+- `init/` contains an example systemd setup for production use.
-- `frontend` is a submodule, this is where the Svelte based frontend resides.
+- `internal/` contains library source code that is not intended to be used by others.
-- `graph/generated` should *not* be touched.
+- `pkg/` contains go packages that can also be used by other projects.
-- `graph/model` contains all types defined in the GraphQL schema not manually defined in `schema/`. Manually defined types have to be listed in `gqlgen.yml`.
+- `test/` Test apps and test data.
-- `graph/schema.graphqls` contains the GraphQL schema. Whenever you change it, you should call `go run github.com/99designs/gqlgen`.
+- `tools/` contains supporting tools for cc-backend. At the moment this is a small application to generate a compatible JWT keypair includin a README about JWT setup in ClusterCockpit.
-- `graph/` contains the resolvers and handlers for the GraphQL API. Function signatures in `graph/schema.resolvers.go` are automatically generated.
+- `web/` Server side templates and frontend related files:
-- `metricdata/` handles getting and archiving the metrics associated with a job.
+   - `templates` Serverside go templates
-  - `metricdata/metricdata.go` defines the interface `MetricDataRepository` and provides functions to the GraphQL and REST API for accessing a jobs metrics which automatically take care of selecting the source for the metrics (the archive or one of the metric data repositories).
+   - `frontend` Svelte components and static assets for frontend UI
-  - `metricdata/archive.go` provides functions for fetching metrics from the job-archive and archiving a job to the job-archive.
-  - `metricdata/cc-metric-store.go` contains an implementation of the `MetricDataRepository` interface which can fetch data from an [cc-metric-store](https://github.com/ClusterCockpit/cc-metric-store)
-  - `metricdata/influxdb-v2` contains an implementation of the `MetricDataRepository` interface which can fetch data from an InfluxDBv2 database. It is currently disabled and out of date and can not be used as of writing.
-- `repository/` all SQL related stuff.
-- `repository/init.go` initializes the `job` (and `tag` and `jobtag`) table if the `--init-db` flag is provided. Not only is the table created in the correct schema, but the job-archive is traversed as well.
-- `schema/` contains type definitions used all over this project extracted in this package as Go disallows cyclic dependencies between packages.
-  - `schema/float.go` contains a custom `float64` type which overwrites JSON and GraphQL Marshaling/Unmarshalling. This is needed because a regular optional `Float` in GraphQL will map to `*float64` types in Go. Wrapping every single metric value in an allocation would be a lot of overhead.
-  - `schema/job.go` provides the types representing a job and its resources. Those can be used as type for a `meta.json` file and/or a row in the `job` table.
-- `templates/` is mostly full of HTML templates and a small helper go module.
-- `utils/systemd` describes how to deploy/install this as a systemd service
-- `test/` rudimentery tests.
-- `utils/`
-- `.env` *must* be changed before you deploy this. It contains a Base64 encoded [Ed25519](https://en.wikipedia.org/wiki/EdDSA) key-pair, the secret used for sessions and the password to the LDAP server if LDAP authentication is enabled.
 - `gqlgen.yml` configures the behaviour and generation of [gqlgen](https://github.com/99designs/gqlgen).
-- `server.go` contains the main function and starts the actual http server.
+- `startDemo.sh` is a shell script that sets up demo data, and builds and starts cc-backend.

cmd/cc-backend/main.go

@@ -22,26 +22,25 @@ import (
 
 	"github.com/99designs/gqlgen/graphql/handler"
 	"github.com/99designs/gqlgen/graphql/playground"
-	"github.com/ClusterCockpit/cc-backend/api"
+	"github.com/ClusterCockpit/cc-backend/internal/api"
-	"github.com/ClusterCockpit/cc-backend/auth"
+	"github.com/ClusterCockpit/cc-backend/internal/auth"
-	"github.com/ClusterCockpit/cc-backend/config"
+	"github.com/ClusterCockpit/cc-backend/internal/config"
-	"github.com/ClusterCockpit/cc-backend/graph"
+	"github.com/ClusterCockpit/cc-backend/internal/graph"
-	"github.com/ClusterCockpit/cc-backend/graph/generated"
+	"github.com/ClusterCockpit/cc-backend/internal/graph/generated"
-	"github.com/ClusterCockpit/cc-backend/log"
+	"github.com/ClusterCockpit/cc-backend/internal/metricdata"
-	"github.com/ClusterCockpit/cc-backend/metricdata"
+	"github.com/ClusterCockpit/cc-backend/internal/repository"
-	"github.com/ClusterCockpit/cc-backend/repository"
+	"github.com/ClusterCockpit/cc-backend/internal/routerConfig"
-	"github.com/ClusterCockpit/cc-backend/templates"
+	"github.com/ClusterCockpit/cc-backend/internal/runtimeEnv"
+	"github.com/ClusterCockpit/cc-backend/internal/templates"
+	"github.com/ClusterCockpit/cc-backend/pkg/log"
 	"github.com/google/gops/agent"
 	"github.com/gorilla/handlers"
 	"github.com/gorilla/mux"
-	"github.com/jmoiron/sqlx"
 
 	_ "github.com/go-sql-driver/mysql"
 	_ "github.com/mattn/go-sqlite3"
 )
 
-var jobRepo *repository.JobRepository
-
 // Format of the configurartion (file). See below for the defaults.
 type ProgramConfig struct {
 	// Address where the http (or https) server will listen on (for example: 'localhost:80').
@@ -70,7 +69,7 @@ type ProgramConfig struct {
 	// do not write to the job-archive.
 	DisableArchive bool `json:"disable-archive"`
 
-	// For LDAP Authentication and user syncronisation.
+	// For LDAP Authentication and user synchronisation.
 	LdapConfig *auth.LdapConfig `json:"ldap"`
 
 	// Specifies for how long a session or JWT shall be valid
@@ -94,14 +93,14 @@ type ProgramConfig struct {
 	// Where to store MachineState files
 	MachineStateDir string `json:"machine-state-dir"`
 
-	// If not zero, automatically mark jobs as stopped running X seconds longer than theire walltime.
+	// If not zero, automatically mark jobs as stopped running X seconds longer than their walltime.
 	StopJobsExceedingWalltime int `json:"stop-jobs-exceeding-walltime"`
 }
 
 var programConfig ProgramConfig = ProgramConfig{
 	Addr:                  ":8080",
 	DisableAuthentication: false,
-	StaticFiles:           "./frontend/public",
+	StaticFiles:           "./web/frontend/public",
 	DBDriver:              "sqlite3",
 	DB:                    "./var/job.db",
 	JobArchive:            "./var/job-archive",
@@ -119,7 +118,7 @@ var programConfig ProgramConfig = ProgramConfig{
 		"plot_general_colorscheme":           []string{"#00bfff", "#0000ff", "#ff00ff", "#ff0000", "#ff8000", "#ffff00", "#80ff00"},
 		"plot_general_lineWidth":             3,
 		"plot_list_hideShortRunningJobs":     5 * 60,
-		"plot_list_jobsPerPage":              10,
+		"plot_list_jobsPerPage":              50,
 		"plot_list_selectedMetrics":          []string{"cpu_load", "ipc", "mem_used", "flops_any", "mem_bw"},
 		"plot_view_plotsPerRow":              3,
 		"plot_view_showPolarplot":            true,
@@ -127,7 +126,7 @@ var programConfig ProgramConfig = ProgramConfig{
 		"plot_view_showStatTable":            true,
 		"system_view_selectedMetric":         "cpu_load",
 	},
-	StopJobsExceedingWalltime: -1,
+	StopJobsExceedingWalltime: 0,
 }
 
 func main() {
@@ -152,7 +151,7 @@ func main() {
 		}
 	}
 
-	if err := loadEnv("./.env"); err != nil && !os.IsNotExist(err) {
+	if err := runtimeEnv.LoadEnv("./.env"); err != nil && !os.IsNotExist(err) {
 		log.Fatalf("parsing './.env' file failed: %s", err.Error())
 	}
 
@@ -178,28 +177,8 @@ func main() {
 	}
 
 	var err error
-	var db *sqlx.DB
+	repository.Connect(programConfig.DBDriver, programConfig.DB)
-	if programConfig.DBDriver == "sqlite3" {
+	db := repository.GetConnection()
-		db, err = sqlx.Open("sqlite3", fmt.Sprintf("%s?_foreign_keys=on", programConfig.DB))
-		if err != nil {
-			log.Fatal(err)
-		}
-
-		// sqlite does not multithread. Having more than one connection open would just mean
-		// waiting for locks.
-		db.SetMaxOpenConns(1)
-	} else if programConfig.DBDriver == "mysql" {
-		db, err = sqlx.Open("mysql", fmt.Sprintf("%s?multiStatements=true", programConfig.DB))
-		if err != nil {
-			log.Fatal(err)
-		}
-
-		db.SetConnMaxLifetime(time.Minute * 3)
-		db.SetMaxOpenConns(10)
-		db.SetMaxIdleConns(10)
-	} else {
-		log.Fatalf("unsupported database driver: %s", programConfig.DBDriver)
-	}
 
 	// Initialize sub-modules and handle all command line flags.
 	// The order here is important! For example, the metricdata package
@@ -215,7 +194,7 @@ func main() {
 			authentication.JwtMaxAge = d
 		}
 
-		if err := authentication.Init(db, programConfig.LdapConfig); err != nil {
+		if err := authentication.Init(db.DB, programConfig.LdapConfig); err != nil {
 			log.Fatal(err)
 		}
 
@@ -257,7 +236,7 @@ func main() {
 		log.Fatal("arguments --add-user and --del-user can only be used if authentication is enabled")
 	}
 
-	if err := config.Init(db, !programConfig.DisableAuthentication, programConfig.UiDefaults, programConfig.JobArchive); err != nil {
+	if err := config.Init(db.DB, !programConfig.DisableAuthentication, programConfig.UiDefaults, programConfig.JobArchive); err != nil {
 		log.Fatal(err)
 	}
 
@@ -266,15 +245,12 @@ func main() {
 	}
 
 	if flagReinitDB {
-		if err := repository.InitDB(db, programConfig.JobArchive); err != nil {
+		if err := repository.InitDB(db.DB, programConfig.JobArchive); err != nil {
 			log.Fatal(err)
 		}
 	}
 
-	jobRepo = &repository.JobRepository{DB: db}
+	jobRepo := repository.GetRepository()
-	if err := jobRepo.Init(); err != nil {
-		log.Fatal(err)
-	}
 
 	if flagImportJob != "" {
 		if err := jobRepo.HandleImportFlag(flagImportJob); err != nil {
@@ -288,7 +264,7 @@ func main() {
 
 	// Setup the http.Handler/Router used by the server
 
-	resolver := &graph.Resolver{DB: db, Repo: jobRepo}
+	resolver := &graph.Resolver{DB: db.DB, Repo: jobRepo}
 	graphQLEndpoint := handler.NewDefaultServer(generated.NewExecutableSchema(generated.Config{Resolvers: resolver}))
 	if os.Getenv("DEBUG") != "1" {
 		// Having this handler means that a error message is returned via GraphQL instead of the connection simply beeing closed.
@@ -394,7 +370,7 @@ func main() {
 	})
 
 	// Mount all /monitoring/... and /api/... routes.
-	setupRoutes(secured, routes)
+	routerConfig.SetupRoutes(secured)
 	api.MountRoutes(secured)
 
 	r.PathPrefix("/").Handler(http.FileServer(http.Dir(programConfig.StaticFiles)))
@@ -461,7 +437,7 @@ func main() {
 	// Because this program will want to bind to a privileged port (like 80), the listener must
 	// be established first, then the user can be changed, and after that,
 	// the actuall http server can be started.
-	if err := dropPrivileges(); err != nil {
+	if err := runtimeEnv.DropPrivileges(programConfig.Group, programConfig.User); err != nil {
 		log.Fatalf("error while changing user: %s", err.Error())
 	}
 
@@ -479,7 +455,7 @@ func main() {
 	go func() {
 		defer wg.Done()
 		<-sigs
-		systemdNotifiy(false, "shutting down")
+		runtimeEnv.SystemdNotifiy(false, "shutting down")
 
 		// First shut down the server gracefully (waiting for all ongoing requests)
 		server.Shutdown(context.Background())
@@ -503,7 +479,7 @@ func main() {
 	if os.Getenv("GOGC") == "" {
 		debug.SetGCPercent(25)
 	}
-	systemdNotifiy(true, "running")
+	runtimeEnv.SystemdNotifiy(true, "running")
 	wg.Wait()
 	log.Print("Gracefull shutdown completed!")
 }

configs/README.md

@@ -0,0 +1,56 @@
+## Intro
+
+cc-backend can be used without a configuration file. In this case the default
+options documented below are used. To overwrite the defaults specify a json
+config file location using the command line option `--config <filepath>`.
+All security relevant configuration. e.g., keys and passwords, are set using environment variables. It is supported to specify these by means of an `.env` file located in the project root.
+
+## Configuration Options
+* `addr`: Type string.  Address where the http (or https) server will listen on (for example: 'localhost:80'). Default `:8080`.
+* `user`: Type string. Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.
+* `group`: Type string.  Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.
+* `disable-authentication`: Type bool.  Disable authentication (for everything: API, Web-UI, ...). Default `false`.
+* `static-files`: Type string. Folder where static assets can be found, those will be served directly. Default `./frontend/public`.
+* `db-driver`: Type string. 'sqlite3' or 'mysql' (mysql will work for mariadb as well). Default `sqlite3`.
+* `db`: Type string. For sqlite3 a filename, for mysql a DSN in this format: https://github.com/go-sql-driver/mysql#dsn-data-source-name (Without query parameters!). Default: `./var/job.db`.
+* `job-archive`: Type string. Path to the job-archive. Default: `./var/job-archive`.
+* `disable-archive`: Type bool. Keep all metric data in the metric data repositories, do not write to the job-archive. Default `false`.
+* `"session-max-age`: Type string. Specifies for how long a session shall be valid  as a string parsable by time.ParseDuration(). If 0 or empty, the session/token does not expire! Default `168h`.
+* `"jwt-max-age`: Type string. Specifies for how long a JWT token shall be valid  as a string parsable by time.ParseDuration(). If 0 or empty, the session/token does not expire! Default `0`.
+* `https-cert-file` and `https-key-file`: Type string. If both those options are not empty, use HTTPS using those certificates.
+* `redirect-http-to`: Type string. If not the empty string and `addr` does not end in ":80", redirect every request incoming at port 80 to that url.
+* `machine-state-dir`: Type string. Where to store MachineState files. TODO: Explain in more detail!
+* `"stop-jobs-exceeding-walltime`: Type int. If not zero, automatically mark jobs as stopped running X seconds longer than their walltime. Only applies if walltime is set for job. Default `0`;
+* `ldap`: Type object. For LDAP Authentication and user synchronisation. Default `nil`.
+   - `url`: Type string.  URL of LDAP directory server.
+   - `user_base`: Type string. Base DN of user tree root.
+   - `search_dn`: Type string. DN for authenticating LDAP admin account with fgeneral read rights.
+   - `user_bind`: Type string. Expression used to authenticate users via LDAP bind. Must contain `uid={username}`.
+   - `user_filter`: Type string. Filter to extract users for syncing.
+   - `sync_interval`: Type string. Interval used for syncing local user table with LDAP directory. Parsed using time.ParseDuration.
+   - `sync_del_old_users`: Type bool. Delete obsolete users in database.
+* `ui-defaults`: Type object. Default configuration for ui views. If overwriten, all options  must be provided! Most options can be overwritten by the user via the web interface.
+   - `analysis_view_histogramMetrics`: Type string array. Metrics to show as job count histograms in analysis view. Default `["flops_any", "mem_bw", "mem_used"]`.
+   - `analysis_view_scatterPlotMetrics`: Type array of string array. Initial scatter plto configuration in analysis view. Default `[["flops_any", "mem_bw"], ["flops_any", "cpu_load"], ["cpu_load", "mem_bw"]]`.
+   - `job_view_nodestats_selectedMetrics`: Type string array. Initial metrics shown in node statistics table of single job view. Default `["flops_any", "mem_bw", "mem_used"]`.
+   - `job_view_polarPlotMetrics`: Type string array. Metrics shown in polar plot of single job view. Default `["flops_any", "mem_bw", "mem_used", "net_bw", "file_bw"]`.
+   - `job_view_selectedMetrics`: Type string array. ??. Default `["flops_any", "mem_bw", "mem_used"]`.
+   - `plot_general_colorBackground`: Type bool. Color plot background according to job average threshold limits. Default `true`.
+   - `plot_general_colorscheme`: Type string array. Initial color scheme. Default `"#00bfff", "#0000ff", "#ff00ff", "#ff0000", "#ff8000", "#ffff00", "#80ff00"`.
+   - `plot_general_lineWidth`: Type int. Initial linewidth. Default `3`.
+   - `plot_list_hideShortRunningJobs`: Type int. Do not show running jobs shorter than X seconds. Default `300`.
+   - `plot_list_jobsPerPage`: Type int. Jobs shown per page in job lists. Default `50`.
+   - `plot_list_selectedMetrics`: Type string array. Initial metric plots shown in jobs lists. Default `"cpu_load", "ipc", "mem_used", "flops_any", "mem_bw"`.
+   - `plot_view_plotsPerRow`: Type int. Number of plots per row in single job view. Default `3`.
+   - `plot_view_showPolarplot`: Type bool. Option to toggle polar plot in single job view. Default `true`.
+   - `plot_view_showRoofline`: Type bool. Option to toggle roofline plot in single job view. Default `true`.
+   - `plot_view_showStatTable`: Type bool. Option to toggle the node statistic table in single job view. Default `true`.
+   - `system_view_selectedMetric`: Type string. Initial metric shown in system view. Default `cpu_load`.
+
+## Environment Variables
+
+An example env file is found in this directory. Copy it to `.env` in the project root and adapt it for your needs.
+
+* `JWT_PUBLIC_KEY` and `JWT_PRIVATE_KEY`:  Base64 encoded Ed25519 keys used for JSON Web Token (JWT) authentication . TODO: Details! You can generate your own keypair using `go run utils/gen-keypair.go`
+* `SESSION_KEY`: Some random bytes used as secret for cookie-based sessions.
+* `LDAP_ADMIN_PASSWORD`: The LDAP admin user password (optional).

configs/config.json

@@ -0,0 +1,14 @@
+{
+    "addr":            "0.0.0.0:443",
+    "ldap": {
+		"url":        "ldaps://hpcldap.rrze.uni-erlangen.de",
+		"user_base":   "ou=people,ou=hpc,dc=rrze,dc=uni-erlangen,dc=de",
+		"search_dn":   "cn=hpcmonitoring,ou=roadm,ou=profile,ou=hpc,dc=rrze,dc=uni-erlangen,dc=de",
+		"user_bind":   "uid={username},ou=people,ou=hpc,dc=rrze,dc=uni-erlangen,dc=de",
+		"user_filter": "(&(objectclass=posixAccount)(uid=*))"
+	},
+    "https-cert-file": "/etc/letsencrypt/live/monitoring.nhr.fau.de/fullchain.pem",
+    "https-key-file":  "/etc/letsencrypt/live/monitoring.nhr.fau.de/privkey.pem",
+    "user":            "clustercockpit",
+    "group":           "clustercockpit"
+}

frontend

@@ -1 +0,0 @@
-Subproject commit 94ef11aa9fc3c194f1df497e3e06c60a7125883d

go.mod

@@ -12,7 +12,6 @@ require (
 	github.com/gorilla/handlers v1.5.1
 	github.com/gorilla/mux v1.8.0
 	github.com/gorilla/sessions v1.2.1
-	github.com/iamlouk/lrucache v0.2.1
 	github.com/influxdata/influxdb-client-go/v2 v2.8.1
 	github.com/jmoiron/sqlx v1.3.4
 	github.com/mattn/go-sqlite3 v1.14.12

gqlgen.yml

@@ -1,10 +1,10 @@
 # Where are all the schema files located? globs are supported eg  src/**/*.graphqls
 schema:
-  - graph/*.graphqls
+  - api/*.graphqls
 
 # Where should the generated server code go?
 exec:
-  filename: graph/generated/generated.go
+  filename: internal/graph/generated/generated.go
   package: generated
 
 # Uncomment to enable federation
@@ -14,7 +14,7 @@ exec:
 
 # Where should any generated models go?
 model:
-  filename: graph/model/models_gen.go
+  filename: internal/graph/model/models_gen.go
   package: model
 
 # Where should the resolver implementations go?
@@ -75,5 +75,3 @@ models:
   Series: { model: "github.com/ClusterCockpit/cc-backend/schema.Series" }
   MetricStatistics: { model: "github.com/ClusterCockpit/cc-backend/schema.MetricStatistics" }
   StatsSeries: { model: "github.com/ClusterCockpit/cc-backend/schema.StatsSeries" }
-
-

init/README.md

@@ -0,0 +1,38 @@
+# How to run this as a systemd deamon
+
+The files in this directory assume that you install ClusterCockpit to `/opt/monitoring`.
+Of course you can choose any other location, but make sure to replace all paths that begin with `/opt/monitoring` in the `clustercockpit.service` file!
+
+If you have not installed [yarn](https://yarnpkg.com/getting-started/install) and [go](https://go.dev/doc/install) already, do that (Golang is available in most package managers).
+It is recommended and easy to install the most recent stable version of Golang as every version also improves the Golang standard library.
+
+The `config.json` can have the optional fields *user* and *group*.
+If provided, the application will call [setuid](https://man7.org/linux/man-pages/man2/setuid.2.html) and [setgid](https://man7.org/linux/man-pages/man2/setgid.2.html) after having read the config file and having bound to a TCP port (so that it can take a privileged port), but before it starts accepting any connections.
+This is good for security, but means that the directories `web/frontend/public`, `var/` and `web/templates/` must be readable by that user and `var/` writable as well (All paths relative to the repos root).
+The `.env` and `config.json` files might contain secrets and should not be readable by that user.
+If those files are changed, the server has to be restarted.
+
+```sh
+# 1.: Clone this repository to /opt/monitoring
+git clone git@github.com:ClusterCockpit/cc-backend.git /opt/monitoring
+
+# 2.: Install all dependencies and build everything
+cd /mnt/monitoring
+go get && go build cmd/cc-backend && (cd ./web/frontend && yarn install && yarn build)
+
+# 3.: Modify the `./config.json` and env-template.txt file from the configs directory to your liking and put it in the repo root
+cp ./configs/config.json ./config.json
+cp ./configs/env-template.txt ./.env
+vim ./config.json # do your thing...
+vim ./.env # do your thing...
+
+# 4.: Add the systemd service unit file (in case /opt/ is mounted on another file system it may be better to copy the file to /etc)
+sudo ln -s /mnt/monitoring/init/clustercockpit.service /etc/systemd/system/clustercockpit.service
+
+# 5.: Enable and start the server
+sudo systemctl enable clustercockpit.service # optional (if done, (re-)starts automatically)
+sudo systemctl start clustercockpit.service
+
+# Check whats going on:
+sudo journalctl -u clustercockpit.service
+```

init/clustercockpit.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=ClusterCockpit Web Server (Go edition)
+Documentation=https://github.com/ClusterCockpit/cc-backend
+Wants=network-online.target
+After=network-online.target
+After=mariadb.service mysql.service
+
+[Service]
+WorkingDirectory=/opt/monitoring/cc-backend
+Type=notify
+NotifyAccess=all
+Restart=on-failure
+RestartSec=30
+TimeoutStopSec=100
+ExecStart=/opt/monitoring/cc-backend/cc-backend --config ./config.json
+
+[Install]
+WantedBy=multi-user.target

internal/api/rest.go

@@ -16,14 +16,14 @@ import (
 	"sync"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/auth"
+	"github.com/ClusterCockpit/cc-backend/internal/auth"
-	"github.com/ClusterCockpit/cc-backend/config"
+	"github.com/ClusterCockpit/cc-backend/internal/config"
-	"github.com/ClusterCockpit/cc-backend/graph"
+	"github.com/ClusterCockpit/cc-backend/internal/graph"
-	"github.com/ClusterCockpit/cc-backend/graph/model"
+	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
-	"github.com/ClusterCockpit/cc-backend/log"
+	"github.com/ClusterCockpit/cc-backend/internal/metricdata"
-	"github.com/ClusterCockpit/cc-backend/metricdata"
+	"github.com/ClusterCockpit/cc-backend/internal/repository"
-	"github.com/ClusterCockpit/cc-backend/repository"
+	"github.com/ClusterCockpit/cc-backend/pkg/log"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 	"github.com/gorilla/mux"
 )
 

internal/auth/auth.go

@@ -14,8 +14,8 @@ import (
 	"strings"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/graph/model"
+	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
-	"github.com/ClusterCockpit/cc-backend/log"
+	"github.com/ClusterCockpit/cc-backend/pkg/log"
 	sq "github.com/Masterminds/squirrel"
 	"github.com/golang-jwt/jwt/v4"
 	"github.com/gorilla/sessions"

internal/auth/ldap.go

@@ -6,8 +6,7 @@ import (
 	"strings"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/log"
+	"github.com/ClusterCockpit/cc-backend/pkg/log"
-
 	"github.com/go-ldap/ldap/v3"
 )
 

internal/config/config.go

@@ -11,10 +11,10 @@ import (
 	"sync"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/auth"
+	"github.com/ClusterCockpit/cc-backend/internal/auth"
-	"github.com/ClusterCockpit/cc-backend/graph/model"
+	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/lrucache"
-	"github.com/iamlouk/lrucache"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 	"github.com/jmoiron/sqlx"
 )
 

internal/config/nodelist.go

@@ -5,7 +5,7 @@ import (
 	"strconv"
 	"strings"
 
-	"github.com/ClusterCockpit/cc-backend/log"
+	"github.com/ClusterCockpit/cc-backend/pkg/log"
 )
 
 type NodeList [][]interface {

internal/graph/generated/generated.go

@@ -13,8 +13,8 @@ import (
 
 	"github.com/99designs/gqlgen/graphql"
 	"github.com/99designs/gqlgen/graphql/introspection"
-	"github.com/ClusterCockpit/cc-backend/graph/model"
+	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 	gqlparser "github.com/vektah/gqlparser/v2"
 	"github.com/vektah/gqlparser/v2/ast"
 )

internal/graph/model/models_gen.go

@@ -8,7 +8,7 @@ import (
 	"strconv"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 )
 
 type Accelerator struct {

internal/graph/resolver.go

@@ -1,7 +1,7 @@
 package graph
 
 import (
-	"github.com/ClusterCockpit/cc-backend/repository"
+	"github.com/ClusterCockpit/cc-backend/internal/repository"
 	"github.com/jmoiron/sqlx"
 )
 

internal/graph/schema.resolvers.go

@@ -10,12 +10,12 @@ import (
 	"strconv"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/auth"
+	"github.com/ClusterCockpit/cc-backend/internal/auth"
-	"github.com/ClusterCockpit/cc-backend/config"
+	"github.com/ClusterCockpit/cc-backend/internal/config"
-	"github.com/ClusterCockpit/cc-backend/graph/generated"
+	"github.com/ClusterCockpit/cc-backend/internal/graph/generated"
-	"github.com/ClusterCockpit/cc-backend/graph/model"
+	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
-	"github.com/ClusterCockpit/cc-backend/metricdata"
+	"github.com/ClusterCockpit/cc-backend/internal/metricdata"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 )
 
 func (r *clusterResolver) Partitions(ctx context.Context, obj *model.Cluster) ([]string, error) {

internal/graph/stats.go

@@ -9,11 +9,11 @@ import (
 	"time"
 
 	"github.com/99designs/gqlgen/graphql"
-	"github.com/ClusterCockpit/cc-backend/config"
+	"github.com/ClusterCockpit/cc-backend/internal/config"
-	"github.com/ClusterCockpit/cc-backend/graph/model"
+	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
-	"github.com/ClusterCockpit/cc-backend/metricdata"
+	"github.com/ClusterCockpit/cc-backend/internal/metricdata"
-	"github.com/ClusterCockpit/cc-backend/repository"
+	"github.com/ClusterCockpit/cc-backend/internal/repository"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 	sq "github.com/Masterminds/squirrel"
 )
 

internal/metricdata/archive.go

@@ -13,8 +13,8 @@ import (
 	"strconv"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/config"
+	"github.com/ClusterCockpit/cc-backend/internal/config"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 )
 
 // For a given job, return the path of the `data.json`/`meta.json` file.

internal/metricdata/cc-metric-store.go

@@ -11,8 +11,8 @@ import (
 	"strings"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/config"
+	"github.com/ClusterCockpit/cc-backend/internal/config"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 )
 
 type CCMetricStoreConfig struct {

internal/metricdata/influxdb-v2.go

@@ -0,0 +1,308 @@
+package metricdata
+
+import (
+	"context"
+	"crypto/tls"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"log"
+	"strings"
+	"time"
+
+	"github.com/ClusterCockpit/cc-backend/internal/config"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
+	influxdb2 "github.com/influxdata/influxdb-client-go/v2"
+	influxdb2Api "github.com/influxdata/influxdb-client-go/v2/api"
+)
+
+type InfluxDBv2DataRepositoryConfig struct {
+	Url     string `json:"url"`
+	Token   string `json:"token"`
+	Bucket  string `json:"bucket"`
+	Org     string `json:"org"`
+	SkipTls bool   `json:"skiptls"`
+}
+
+type InfluxDBv2DataRepository struct {
+	client              influxdb2.Client
+	queryClient         influxdb2Api.QueryAPI
+	bucket, measurement string
+}
+
+func (idb *InfluxDBv2DataRepository) Init(rawConfig json.RawMessage) error {
+	var config InfluxDBv2DataRepositoryConfig
+	if err := json.Unmarshal(rawConfig, &config); err != nil {
+		return err
+	}
+
+	idb.client = influxdb2.NewClientWithOptions(config.Url, config.Token, influxdb2.DefaultOptions().SetTLSConfig(&tls.Config{InsecureSkipVerify: config.SkipTls}))
+	idb.queryClient = idb.client.QueryAPI(config.Org)
+	idb.bucket = config.Bucket
+
+	return nil
+}
+
+func (idb *InfluxDBv2DataRepository) formatTime(t time.Time) string {
+	return t.Format(time.RFC3339) // Like “2006-01-02T15:04:05Z07:00”
+}
+
+func (idb *InfluxDBv2DataRepository) epochToTime(epoch int64) time.Time {
+	return time.Unix(epoch, 0)
+}
+
+func (idb *InfluxDBv2DataRepository) LoadData(job *schema.Job, metrics []string, scopes []schema.MetricScope, ctx context.Context) (schema.JobData, error) {
+
+	measurementsConds := make([]string, 0, len(metrics))
+	for _, m := range metrics {
+		measurementsConds = append(measurementsConds, fmt.Sprintf(`r["_measurement"] == "%s"`, m))
+	}
+	measurementsCond := strings.Join(measurementsConds, " or ")
+
+	hostsConds := make([]string, 0, len(job.Resources))
+	for _, h := range job.Resources {
+		if h.HWThreads != nil || h.Accelerators != nil {
+			// TODO
+			return nil, errors.New("the InfluxDB metric data repository does not yet support HWThreads or Accelerators")
+		}
+		hostsConds = append(hostsConds, fmt.Sprintf(`r["hostname"] == "%s"`, h.Hostname))
+	}
+	hostsCond := strings.Join(hostsConds, " or ")
+
+	jobData := make(schema.JobData) // Empty Schema: map[<string>FIELD]map[<MetricScope>SCOPE]<*JobMetric>METRIC
+	// Requested Scopes
+	for _, scope := range scopes {
+		query := ""
+		switch scope {
+		case "node":
+			// Get Finest Granularity, Groupy By Measurement and Hostname (== Metric / Node), Calculate Mean for 60s windows
+			// log.Println("Note: Scope 'node' requested. ")
+			query = fmt.Sprintf(`
+								from(bucket: "%s")
+								|> range(start: %s, stop: %s)
+								|> filter(fn: (r) => (%s) and (%s) )
+								|> drop(columns: ["_start", "_stop"])
+								|> group(columns: ["hostname", "_measurement"])
+		            |> aggregateWindow(every: 60s, fn: mean)
+								|> drop(columns: ["_time"])`,
+				idb.bucket,
+				idb.formatTime(job.StartTime), idb.formatTime(idb.epochToTime(job.StartTimeUnix+int64(job.Duration)+int64(1))),
+				measurementsCond, hostsCond)
+		case "socket":
+			log.Println("Note: Scope 'socket' requested, but not yet supported: Will return 'node' scope only. ")
+			continue
+		case "core":
+			log.Println("Note: Scope 'core' requested, but not yet supported: Will return 'node' scope only. ")
+			continue
+			// Get Finest Granularity only, Set NULL to 0.0
+			// query = fmt.Sprintf(`
+			//  	from(bucket: "%s")
+			//  	|> range(start: %s, stop: %s)
+			//  	|> filter(fn: (r) => %s )
+			//  	|> filter(fn: (r) => %s )
+			//  	|> drop(columns: ["_start", "_stop", "cluster"])
+			//  	|> map(fn: (r) => (if exists r._value then {r with _value: r._value} else {r with _value: 0.0}))`,
+			//  	idb.bucket,
+			//  	idb.formatTime(job.StartTime), idb.formatTime(idb.epochToTime(job.StartTimeUnix + int64(job.Duration) + int64(1) )),
+			//  	measurementsCond, hostsCond)
+		default:
+			log.Println("Note: Unknown Scope requested: Will return 'node' scope. ")
+			continue
+			// return nil, errors.New("the InfluxDB metric data repository does not yet support other scopes than 'node'")
+		}
+
+		rows, err := idb.queryClient.Query(ctx, query)
+		if err != nil {
+			return nil, err
+		}
+
+		// Init Metrics: Only Node level now -> TODO: Matching /check on scope level ...
+		for _, metric := range metrics {
+			jobMetric, ok := jobData[metric]
+			if !ok {
+				mc := config.GetMetricConfig(job.Cluster, metric)
+				jobMetric = map[schema.MetricScope]*schema.JobMetric{
+					scope: { // uses scope var from above!
+						Unit:             mc.Unit,
+						Scope:            scope,
+						Timestep:         mc.Timestep,
+						Series:           make([]schema.Series, 0, len(job.Resources)),
+						StatisticsSeries: nil, // Should be: &schema.StatsSeries{},
+					},
+				}
+			}
+			jobData[metric] = jobMetric
+		}
+
+		// Process Result: Time-Data
+		field, host, hostSeries := "", "", schema.Series{}
+		// typeId := 0
+		switch scope {
+		case "node":
+			for rows.Next() {
+				row := rows.Record()
+				if host == "" || host != row.ValueByKey("hostname").(string) || rows.TableChanged() {
+					if host != "" {
+						// Append Series before reset
+						jobData[field][scope].Series = append(jobData[field][scope].Series, hostSeries)
+					}
+					field, host = row.Measurement(), row.ValueByKey("hostname").(string)
+					hostSeries = schema.Series{
+						Hostname:   host,
+						Statistics: nil,
+						Data:       make([]schema.Float, 0),
+					}
+				}
+				val, ok := row.Value().(float64)
+				if ok {
+					hostSeries.Data = append(hostSeries.Data, schema.Float(val))
+				} else {
+					hostSeries.Data = append(hostSeries.Data, schema.Float(0))
+				}
+			}
+		case "socket":
+			continue
+		case "core":
+			continue
+			// Include Series.Id in hostSeries
+			// for rows.Next() {
+			// 		row := rows.Record()
+			// 		if ( host == "" || host != row.ValueByKey("hostname").(string) || typeId != row.ValueByKey("type-id").(int) || rows.TableChanged() ) {
+			// 		 		if ( host != "" ) {
+			// 						// Append Series before reset
+			// 		 		  	jobData[field][scope].Series = append(jobData[field][scope].Series, hostSeries)
+			// 		 		}
+			// 		 		field, host, typeId = row.Measurement(), row.ValueByKey("hostname").(string), row.ValueByKey("type-id").(int)
+			// 		 		hostSeries  = schema.Series{
+			// 		 				Hostname:   host,
+			// 						Id:					&typeId,
+			// 		 				Statistics: nil,
+			// 		 				Data:       make([]schema.Float, 0),
+			// 		 		}
+			// 		}
+			// 		val := row.Value().(float64)
+			// 		hostSeries.Data = append(hostSeries.Data, schema.Float(val))
+			// }
+		default:
+			continue
+			// return nil, errors.New("the InfluxDB metric data repository does not yet support other scopes than 'node, core'")
+		}
+		// Append last Series
+		jobData[field][scope].Series = append(jobData[field][scope].Series, hostSeries)
+	}
+
+	// Get Stats
+	stats, err := idb.LoadStats(job, metrics, ctx)
+	if err != nil {
+		return nil, err
+	}
+
+	for _, scope := range scopes {
+		if scope == "node" { // No 'socket/core' support yet
+			for metric, nodes := range stats {
+				// log.Println(fmt.Sprintf("<< Add Stats for : Field %s >>", metric))
+				for node, stats := range nodes {
+					// log.Println(fmt.Sprintf("<< Add Stats for : Host %s : Min %.2f, Max %.2f, Avg %.2f >>", node, stats.Min, stats.Max, stats.Avg ))
+					for index, _ := range jobData[metric][scope].Series {
+						// log.Println(fmt.Sprintf("<< Try to add Stats to Series in Position %d >>", index))
+						if jobData[metric][scope].Series[index].Hostname == node {
+							// log.Println(fmt.Sprintf("<< Match for Series in Position %d : Host %s >>", index, jobData[metric][scope].Series[index].Hostname))
+							jobData[metric][scope].Series[index].Statistics = &schema.MetricStatistics{Avg: stats.Avg, Min: stats.Min, Max: stats.Max}
+							// log.Println(fmt.Sprintf("<< Result Inner: Min %.2f, Max %.2f, Avg %.2f >>", jobData[metric][scope].Series[index].Statistics.Min, jobData[metric][scope].Series[index].Statistics.Max, jobData[metric][scope].Series[index].Statistics.Avg))
+						}
+					}
+				}
+			}
+		}
+	}
+
+	// DEBUG:
+	// for _, scope := range scopes {
+	// 		for _, met := range metrics {
+	// 		   for _, series := range jobData[met][scope].Series {
+	// 		   log.Println(fmt.Sprintf("<< Result: %d data points for metric %s on %s with scope %s, Stats: Min %.2f, Max %.2f, Avg %.2f >>",
+	// 				 	len(series.Data), met, series.Hostname, scope,
+	// 					series.Statistics.Min, series.Statistics.Max, series.Statistics.Avg))
+	// 		   }
+	// 		}
+	// }
+
+	return jobData, nil
+}
+
+func (idb *InfluxDBv2DataRepository) LoadStats(job *schema.Job, metrics []string, ctx context.Context) (map[string]map[string]schema.MetricStatistics, error) {
+
+	stats := map[string]map[string]schema.MetricStatistics{}
+
+	hostsConds := make([]string, 0, len(job.Resources))
+	for _, h := range job.Resources {
+		if h.HWThreads != nil || h.Accelerators != nil {
+			// TODO
+			return nil, errors.New("the InfluxDB metric data repository does not yet support HWThreads or Accelerators")
+		}
+		hostsConds = append(hostsConds, fmt.Sprintf(`r["hostname"] == "%s"`, h.Hostname))
+	}
+	hostsCond := strings.Join(hostsConds, " or ")
+
+	// lenMet := len(metrics)
+
+	for _, metric := range metrics {
+		// log.Println(fmt.Sprintf("<< You are here: %s (Index %d of %d metrics)", metric, index, lenMet))
+
+		query := fmt.Sprintf(`
+				  data = from(bucket: "%s")
+				  |> range(start: %s, stop: %s)
+				  |> filter(fn: (r) => r._measurement == "%s" and r._field == "value" and (%s))
+				  union(tables: [data |> mean(column: "_value") |> set(key: "_field", value: "avg"),
+				                 data |>  min(column: "_value") |> set(key: "_field", value: "min"),
+				                 data |>  max(column: "_value") |> set(key: "_field", value: "max")])
+				  |> pivot(rowKey: ["hostname"], columnKey: ["_field"], valueColumn: "_value")
+				  |> group()`,
+			idb.bucket,
+			idb.formatTime(job.StartTime), idb.formatTime(idb.epochToTime(job.StartTimeUnix+int64(job.Duration)+int64(1))),
+			metric, hostsCond)
+
+		rows, err := idb.queryClient.Query(ctx, query)
+		if err != nil {
+			return nil, err
+		}
+
+		nodes := map[string]schema.MetricStatistics{}
+		for rows.Next() {
+			row := rows.Record()
+			host := row.ValueByKey("hostname").(string)
+
+			avg, avgok := row.ValueByKey("avg").(float64)
+			if !avgok {
+				// log.Println(fmt.Sprintf(">> Assertion error for metric %s, statistic AVG. Expected 'float64', got %v", metric, avg))
+				avg = 0.0
+			}
+			min, minok := row.ValueByKey("min").(float64)
+			if !minok {
+				// log.Println(fmt.Sprintf(">> Assertion error for metric %s, statistic MIN. Expected 'float64', got %v", metric, min))
+				min = 0.0
+			}
+			max, maxok := row.ValueByKey("max").(float64)
+			if !maxok {
+				// log.Println(fmt.Sprintf(">> Assertion error for metric %s, statistic MAX. Expected 'float64', got %v", metric, max))
+				max = 0.0
+			}
+
+			nodes[host] = schema.MetricStatistics{
+				Avg: avg,
+				Min: min,
+				Max: max,
+			}
+		}
+		stats[metric] = nodes
+	}
+
+	return stats, nil
+}
+
+func (idb *InfluxDBv2DataRepository) LoadNodeData(cluster string, metrics, nodes []string, scopes []schema.MetricScope, from, to time.Time, ctx context.Context) (map[string]map[string][]*schema.JobMetric, error) {
+	// TODO : Implement to be used in Analysis- und System/Node-View
+	log.Println(fmt.Sprintf("LoadNodeData unimplemented for InfluxDBv2DataRepository, Args: cluster %s, metrics %v, nodes %v, scopes %v", cluster, metrics, nodes, scopes))
+
+	return nil, errors.New("unimplemented for InfluxDBv2DataRepository")
+}

internal/metricdata/metricdata.go

@@ -6,10 +6,10 @@ import (
 	"fmt"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/config"
+	"github.com/ClusterCockpit/cc-backend/internal/config"
-	"github.com/ClusterCockpit/cc-backend/log"
+	"github.com/ClusterCockpit/cc-backend/pkg/log"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/lrucache"
-	"github.com/iamlouk/lrucache"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 )
 
 type MetricDataRepository interface {

internal/metricdata/utils.go

@@ -5,7 +5,7 @@ import (
 	"encoding/json"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 )
 
 var TestLoadDataCallback func(job *schema.Job, metrics []string, scopes []schema.MetricScope, ctx context.Context) (schema.JobData, error) = func(job *schema.Job, metrics []string, scopes []schema.MetricScope, ctx context.Context) (schema.JobData, error) {

internal/repository/dbConnection.go

@@ -0,0 +1,58 @@
+package repository
+
+import (
+	"fmt"
+	"log"
+	"sync"
+	"time"
+
+	"github.com/jmoiron/sqlx"
+)
+
+var (
+	dbConnOnce     sync.Once
+	dbConnInstance *DBConnection
+)
+
+type DBConnection struct {
+	DB *sqlx.DB
+}
+
+func Connect(driver string, db string) {
+	var err error
+	var dbHandle *sqlx.DB
+
+	dbConnOnce.Do(func() {
+		if driver == "sqlite3" {
+			dbHandle, err = sqlx.Open("sqlite3", fmt.Sprintf("%s?_foreign_keys=on", db))
+			if err != nil {
+				log.Fatal(err)
+			}
+
+			// sqlite does not multithread. Having more than one connection open would just mean
+			// waiting for locks.
+			dbHandle.SetMaxOpenConns(1)
+		} else if driver == "mysql" {
+			dbHandle, err = sqlx.Open("mysql", fmt.Sprintf("%s?multiStatements=true", db))
+			if err != nil {
+				log.Fatal(err)
+			}
+
+			dbHandle.SetConnMaxLifetime(time.Minute * 3)
+			dbHandle.SetMaxOpenConns(10)
+			dbHandle.SetMaxIdleConns(10)
+		} else {
+			log.Fatalf("unsupported database driver: %s", driver)
+		}
+
+		dbConnInstance = &DBConnection{DB: dbHandle}
+	})
+}
+
+func GetConnection() *DBConnection {
+	if dbConnInstance == nil {
+		log.Fatalf("Database connection not initialized!")
+	}
+
+	return dbConnInstance
+}

internal/repository/import.go

@@ -9,10 +9,10 @@ import (
 	"strings"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/config"
+	"github.com/ClusterCockpit/cc-backend/internal/config"
-	"github.com/ClusterCockpit/cc-backend/log"
+	"github.com/ClusterCockpit/cc-backend/internal/metricdata"
-	"github.com/ClusterCockpit/cc-backend/metricdata"
+	"github.com/ClusterCockpit/cc-backend/pkg/log"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 )
 
 const NamedJobInsert string = `INSERT INTO job (

internal/repository/init.go

@@ -8,8 +8,8 @@ import (
 	"path/filepath"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/log"
+	"github.com/ClusterCockpit/cc-backend/pkg/log"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 	"github.com/jmoiron/sqlx"
 )
 

internal/repository/job.go

@@ -7,17 +7,23 @@ import (
 	"errors"
 	"fmt"
 	"strconv"
+	"sync"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/auth"
+	"github.com/ClusterCockpit/cc-backend/internal/auth"
-	"github.com/ClusterCockpit/cc-backend/graph/model"
+	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
-	"github.com/ClusterCockpit/cc-backend/log"
+	"github.com/ClusterCockpit/cc-backend/pkg/log"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/lrucache"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 	sq "github.com/Masterminds/squirrel"
-	"github.com/iamlouk/lrucache"
 	"github.com/jmoiron/sqlx"
 )
 
+var (
+	jobRepoOnce     sync.Once
+	jobRepoInstance *JobRepository
+)
+
 type JobRepository struct {
 	DB *sqlx.DB
 
@@ -25,10 +31,18 @@ type JobRepository struct {
 	cache     *lrucache.Cache
 }
 
-func (r *JobRepository) Init() error {
+func GetRepository() *JobRepository {
-	r.stmtCache = sq.NewStmtCache(r.DB)
+	jobRepoOnce.Do(func() {
-	r.cache = lrucache.New(1024 * 1024)
+		db := GetConnection()
-	return nil
+
+		jobRepoInstance = &JobRepository{
+			DB:        db.DB,
+			stmtCache: sq.NewStmtCache(db.DB),
+			cache:     lrucache.New(1024 * 1024),
+		}
+	})
+
+	return jobRepoInstance
 }
 
 var jobColumns []string = []string{

internal/repository/job_test.go

@@ -11,22 +11,11 @@ import (
 var db *sqlx.DB
 
 func init() {
-	var err error
+	Connect("sqlite3", "../../test/test.db")
-	db, err = sqlx.Open("sqlite3", "../test/test.db")
-	if err != nil {
-		fmt.Println(err)
-	}
 }
 
 func setup(t *testing.T) *JobRepository {
-	r := &JobRepository{
+	return GetRepository()
-		DB: db,
-	}
-	if err := r.Init(); err != nil {
-		t.Fatal(err)
-	}
-
-	return r
 }
 
 func TestFind(t *testing.T) {

internal/repository/query.go

@@ -8,10 +8,10 @@ import (
 	"strings"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/auth"
+	"github.com/ClusterCockpit/cc-backend/internal/auth"
-	"github.com/ClusterCockpit/cc-backend/graph/model"
+	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
-	"github.com/ClusterCockpit/cc-backend/log"
+	"github.com/ClusterCockpit/cc-backend/pkg/log"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 	sq "github.com/Masterminds/squirrel"
 )
 

internal/repository/tags.go

@@ -1,8 +1,8 @@
 package repository
 
 import (
-	"github.com/ClusterCockpit/cc-backend/metricdata"
+	"github.com/ClusterCockpit/cc-backend/internal/metricdata"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 	sq "github.com/Masterminds/squirrel"
 )
 

internal/routerConfig/routes.go

@@ -1,4 +1,4 @@
-package main
+package routerConfig
 
 import (
 	"fmt"
@@ -8,13 +8,14 @@ import (
 	"strings"
 	"time"
 
-	"github.com/ClusterCockpit/cc-backend/auth"
+	"github.com/ClusterCockpit/cc-backend/internal/auth"
-	"github.com/ClusterCockpit/cc-backend/config"
+	"github.com/ClusterCockpit/cc-backend/internal/config"
-	"github.com/ClusterCockpit/cc-backend/graph"
+	"github.com/ClusterCockpit/cc-backend/internal/graph"
-	"github.com/ClusterCockpit/cc-backend/graph/model"
+	"github.com/ClusterCockpit/cc-backend/internal/graph/model"
-	"github.com/ClusterCockpit/cc-backend/log"
+	"github.com/ClusterCockpit/cc-backend/internal/repository"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/internal/templates"
-	"github.com/ClusterCockpit/cc-backend/templates"
+	"github.com/ClusterCockpit/cc-backend/pkg/log"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 	"github.com/gorilla/mux"
 )
 
@@ -50,6 +51,7 @@ func setupHomeRoute(i InfoType, r *http.Request) InfoType {
 		TotalJobs       int
 		RecentShortJobs int
 	}
+	jobRepo := repository.GetRepository()
 
 	runningJobs, err := jobRepo.CountGroupedJobs(r.Context(), model.AggregateCluster, []*model.JobFilter{{
 		State: []schema.JobState{schema.JobStateRunning},
@@ -93,6 +95,7 @@ func setupJobRoute(i InfoType, r *http.Request) InfoType {
 }
 
 func setupUserRoute(i InfoType, r *http.Request) InfoType {
+	jobRepo := repository.GetRepository()
 	username := mux.Vars(r)["id"]
 	i["id"] = username
 	i["username"] = username
@@ -135,6 +138,7 @@ func setupAnalysisRoute(i InfoType, r *http.Request) InfoType {
 
 func setupTaglistRoute(i InfoType, r *http.Request) InfoType {
 	var username *string = nil
+	jobRepo := repository.GetRepository()
 	if user := auth.GetUser(r.Context()); user != nil && !user.HasRole(auth.RoleAdmin) {
 		username = &user.Username
 	}
@@ -245,7 +249,7 @@ func buildFilterPresets(query url.Values) map[string]interface{} {
 	return filterPresets
 }
 
-func setupRoutes(router *mux.Router, routes []Route) {
+func SetupRoutes(router *mux.Router) {
 	for _, route := range routes {
 		route := route
 		router.HandleFunc(route.Route, func(rw http.ResponseWriter, r *http.Request) {

internal/runtimeEnv/setup.go

@@ -1,4 +1,4 @@
-package main
+package runtimeEnv
 
 import (
 	"bufio"
@@ -15,7 +15,7 @@ import (
 // Very simple and limited .env file reader.
 // All variable definitions found are directly
 // added to the processes environment.
-func loadEnv(file string) error {
+func LoadEnv(file string) error {
 	f, err := os.Open(file)
 	if err != nil {
 		return err
@@ -81,9 +81,9 @@ func loadEnv(file string) error {
 // specified in the config.json. The go runtime
 // takes care of all threads (and not only the calling one)
 // executing the underlying systemcall.
-func dropPrivileges() error {
+func DropPrivileges(username string, group string) error {
-	if programConfig.Group != "" {
+	if group != "" {
-		g, err := user.LookupGroup(programConfig.Group)
+		g, err := user.LookupGroup(group)
 		if err != nil {
 			return err
 		}
@@ -94,8 +94,8 @@ func dropPrivileges() error {
 		}
 	}
 
-	if programConfig.User != "" {
+	if username != "" {
-		u, err := user.Lookup(programConfig.User)
+		u, err := user.Lookup(username)
 		if err != nil {
 			return err
 		}
@@ -111,7 +111,7 @@ func dropPrivileges() error {
 
 // If started via systemd, inform systemd that we are running:
 // https://www.freedesktop.org/software/systemd/man/sd_notify.html
-func systemdNotifiy(ready bool, status string) {
+func SystemdNotifiy(ready bool, status string) {
 	if os.Getenv("NOTIFY_SOCKET") == "" {
 		// Not started using systemd
 		return

internal/templates/templates.go

@@ -5,8 +5,8 @@ import (
 	"net/http"
 	"os"
 
-	"github.com/ClusterCockpit/cc-backend/config"
+	"github.com/ClusterCockpit/cc-backend/internal/config"
-	"github.com/ClusterCockpit/cc-backend/log"
+	"github.com/ClusterCockpit/cc-backend/pkg/log"
 )
 
 var templatesDir string
@@ -36,7 +36,7 @@ func init() {
 	if ebp != "" {
 		bp = ebp
 	}
-	templatesDir = bp + "templates/"
+	templatesDir = bp + "web/templates/"
 	base := template.Must(template.ParseFiles(templatesDir + "base.tmpl"))
 	files := []string{
 		"home.tmpl", "404.tmpl", "login.tmpl",

metricdata/influxdb-v2.go

@@ -1,308 +0,0 @@
-package metricdata
-
-import (
-	"context"
-	"errors"
-	"fmt"
-	"log"
-	"strings"
-	"time"
-	"crypto/tls"
-	"encoding/json"
-
-	"github.com/ClusterCockpit/cc-backend/config"
-	"github.com/ClusterCockpit/cc-backend/schema"
-	influxdb2 "github.com/influxdata/influxdb-client-go/v2"
-	influxdb2Api "github.com/influxdata/influxdb-client-go/v2/api"
-)
-
-type InfluxDBv2DataRepositoryConfig struct {
-	Url   string `json:"url"`
-	Token string `json:"token"`
-	Bucket string `json:"bucket"`
-	Org string `json:"org"`
-	SkipTls bool `json:"skiptls"`
-}
-
-type InfluxDBv2DataRepository struct {
-	client              influxdb2.Client
-	queryClient         influxdb2Api.QueryAPI
-	bucket, measurement string
-}
-
-func (idb *InfluxDBv2DataRepository) Init(rawConfig json.RawMessage) error {
-	var config InfluxDBv2DataRepositoryConfig
-	if err := json.Unmarshal(rawConfig, &config); err != nil {
-		return err
-	}
-
-	idb.client 			= influxdb2.NewClientWithOptions(config.Url, config.Token, influxdb2.DefaultOptions().SetTLSConfig(&tls.Config {InsecureSkipVerify: config.SkipTls,} ))
-	idb.queryClient = idb.client.QueryAPI(config.Org)
-	idb.bucket      = config.Bucket
-
-	return nil
-}
-
-func (idb *InfluxDBv2DataRepository) formatTime(t time.Time) string {
-	return t.Format(time.RFC3339) // Like “2006-01-02T15:04:05Z07:00”
-}
-
-func (idb *InfluxDBv2DataRepository) epochToTime(epoch int64) time.Time {
-	return time.Unix(epoch, 0)
-}
-
-func (idb *InfluxDBv2DataRepository) LoadData(job *schema.Job, metrics []string, scopes []schema.MetricScope, ctx context.Context) (schema.JobData, error) {
-
-	measurementsConds := make([]string, 0, len(metrics))
-	for _, m := range metrics {
-		measurementsConds = append(measurementsConds, fmt.Sprintf(`r["_measurement"] == "%s"`, m))
-	}
-	measurementsCond := strings.Join(measurementsConds, " or ")
-
-	hostsConds := make([]string, 0, len(job.Resources))
-	for _, h := range job.Resources {
-		if h.HWThreads != nil || h.Accelerators != nil {
-			// TODO
-			return nil, errors.New("the InfluxDB metric data repository does not yet support HWThreads or Accelerators")
-		}
-		hostsConds = append(hostsConds, fmt.Sprintf(`r["hostname"] == "%s"`, h.Hostname))
-	}
-	hostsCond := strings.Join(hostsConds, " or ")
-
-	jobData := make(schema.JobData) // Empty Schema: map[<string>FIELD]map[<MetricScope>SCOPE]<*JobMetric>METRIC
-	// Requested Scopes
-	for _, scope := range scopes {
-			query := ""
-			switch scope {
-					case "node":
-							// Get Finest Granularity, Groupy By Measurement and Hostname (== Metric / Node), Calculate Mean for 60s windows
-							// log.Println("Note: Scope 'node' requested. ")
-							query = fmt.Sprintf(`
-								from(bucket: "%s")
-								|> range(start: %s, stop: %s)
-								|> filter(fn: (r) => (%s) and (%s) )
-								|> drop(columns: ["_start", "_stop"])
-								|> group(columns: ["hostname", "_measurement"])
-		            |> aggregateWindow(every: 60s, fn: mean)
-								|> drop(columns: ["_time"])`,
-								idb.bucket,
-								idb.formatTime(job.StartTime), idb.formatTime(idb.epochToTime(job.StartTimeUnix + int64(job.Duration) + int64(1) )),
-								measurementsCond, hostsCond)
-					case "socket":
-							log.Println("Note: Scope 'socket' requested, but not yet supported: Will return 'node' scope only. ")
-							continue
-					case "core":
-						  log.Println("Note: Scope 'core' requested, but not yet supported: Will return 'node' scope only. ")
-							continue
-							// Get Finest Granularity only, Set NULL to 0.0
-							// query = fmt.Sprintf(`
-							//  	from(bucket: "%s")
-							//  	|> range(start: %s, stop: %s)
-							//  	|> filter(fn: (r) => %s )
-							//  	|> filter(fn: (r) => %s )
-							//  	|> drop(columns: ["_start", "_stop", "cluster"])
-							//  	|> map(fn: (r) => (if exists r._value then {r with _value: r._value} else {r with _value: 0.0}))`,
-							//  	idb.bucket,
-							//  	idb.formatTime(job.StartTime), idb.formatTime(idb.epochToTime(job.StartTimeUnix + int64(job.Duration) + int64(1) )),
-							//  	measurementsCond, hostsCond)
-					default:
-							log.Println("Note: Unknown Scope requested: Will return 'node' scope. ")
-							continue
-							// return nil, errors.New("the InfluxDB metric data repository does not yet support other scopes than 'node'")
-			}
-
-			rows, err := idb.queryClient.Query(ctx, query)
-			if err != nil {
-				return nil, err
-			}
-
-			// Init Metrics: Only Node level now -> TODO: Matching /check on scope level ...
-			for _, metric := range metrics {
-					jobMetric, ok := jobData[metric]
-					if !ok {
-							mc 		:= config.GetMetricConfig(job.Cluster, metric)
-							jobMetric = map[schema.MetricScope]*schema.JobMetric{
-									scope: { // uses scope var from above!
-											Unit:     mc.Unit,
-											Scope:    scope,
-											Timestep: mc.Timestep,
-											Series:   make([]schema.Series, 0, len(job.Resources)),
-											StatisticsSeries: nil, // Should be: &schema.StatsSeries{},
-									},
-							}
-					}
-					jobData[metric] = jobMetric
-			}
-
-			// Process Result: Time-Data
-		  field, host, hostSeries := "", "", schema.Series{}
-			// typeId := 0
-			switch scope {
-					case "node":
-							for rows.Next() {
-									row := rows.Record()
-									if ( host == "" || host != row.ValueByKey("hostname").(string) || rows.TableChanged() ) {
-									 		if ( host != "" ) {
-													// Append Series before reset
-									 		  	jobData[field][scope].Series = append(jobData[field][scope].Series, hostSeries)
-									 		}
-									 		field, host = row.Measurement(), row.ValueByKey("hostname").(string)
-									 		hostSeries  = schema.Series{
-									 				Hostname:   host,
-									 				Statistics: nil,
-									 				Data:       make([]schema.Float, 0),
-									 		}
-									}
-									val, ok := row.Value().(float64)
-									if ok {
-											hostSeries.Data = append(hostSeries.Data, schema.Float(val))
-									} else {
-											hostSeries.Data = append(hostSeries.Data, schema.Float(0))
-									}
-							}
-					case "socket":
-							continue
-					case "core":
-							continue
-						  // Include Series.Id in hostSeries
-							// for rows.Next() {
-							// 		row := rows.Record()
-							// 		if ( host == "" || host != row.ValueByKey("hostname").(string) || typeId != row.ValueByKey("type-id").(int) || rows.TableChanged() ) {
-							// 		 		if ( host != "" ) {
-							// 						// Append Series before reset
-							// 		 		  	jobData[field][scope].Series = append(jobData[field][scope].Series, hostSeries)
-							// 		 		}
-							// 		 		field, host, typeId = row.Measurement(), row.ValueByKey("hostname").(string), row.ValueByKey("type-id").(int)
-							// 		 		hostSeries  = schema.Series{
-							// 		 				Hostname:   host,
-							// 						Id:					&typeId,
-							// 		 				Statistics: nil,
-							// 		 				Data:       make([]schema.Float, 0),
-							// 		 		}
-							// 		}
-							// 		val := row.Value().(float64)
-							// 		hostSeries.Data = append(hostSeries.Data, schema.Float(val))
-							// }
-					default:
-							continue
-							// return nil, errors.New("the InfluxDB metric data repository does not yet support other scopes than 'node, core'")
-			}
-			// Append last Series
-		  jobData[field][scope].Series = append(jobData[field][scope].Series, hostSeries)
-	}
-
-	// Get Stats
-	stats, err := idb.LoadStats(job, metrics, ctx)
-	if err != nil {
-		return nil, err
-	}
-
-	for _, scope := range scopes {
-		if scope == "node" { // No 'socket/core' support yet
-			for metric, nodes := range stats {
-				// log.Println(fmt.Sprintf("<< Add Stats for : Field %s >>", metric))
-				for node, stats := range nodes {
-					// log.Println(fmt.Sprintf("<< Add Stats for : Host %s : Min %.2f, Max %.2f, Avg %.2f >>", node, stats.Min, stats.Max, stats.Avg ))
-					for index, _ := range jobData[metric][scope].Series {
-						// log.Println(fmt.Sprintf("<< Try to add Stats to Series in Position %d >>", index))
-						if jobData[metric][scope].Series[index].Hostname == node {
-							// log.Println(fmt.Sprintf("<< Match for Series in Position %d : Host %s >>", index, jobData[metric][scope].Series[index].Hostname))
-							jobData[metric][scope].Series[index].Statistics = &schema.MetricStatistics{Avg: stats.Avg, Min: stats.Min, Max: stats.Max}
-							// log.Println(fmt.Sprintf("<< Result Inner: Min %.2f, Max %.2f, Avg %.2f >>", jobData[metric][scope].Series[index].Statistics.Min, jobData[metric][scope].Series[index].Statistics.Max, jobData[metric][scope].Series[index].Statistics.Avg))
-						}
-					}
-				}
-			}
-		}
-	}
-
-	// DEBUG:
-	// for _, scope := range scopes {
-	// 		for _, met := range metrics {
-	// 		   for _, series := range jobData[met][scope].Series {
-	// 		   log.Println(fmt.Sprintf("<< Result: %d data points for metric %s on %s with scope %s, Stats: Min %.2f, Max %.2f, Avg %.2f >>",
-	// 				 	len(series.Data), met, series.Hostname, scope,
-	// 					series.Statistics.Min, series.Statistics.Max, series.Statistics.Avg))
-	// 		   }
-	// 		}
-	// }
-
-	return jobData, nil
-}
-
-func (idb *InfluxDBv2DataRepository) LoadStats(job *schema.Job, metrics []string, ctx context.Context) (map[string]map[string]schema.MetricStatistics, error) {
-
-	stats := map[string]map[string]schema.MetricStatistics{}
-
-	hostsConds := make([]string, 0, len(job.Resources))
-	for _, h := range job.Resources {
-			if h.HWThreads != nil || h.Accelerators != nil {
-				// TODO
-				return nil, errors.New("the InfluxDB metric data repository does not yet support HWThreads or Accelerators")
-			}
-			hostsConds = append(hostsConds, fmt.Sprintf(`r["hostname"] == "%s"`, h.Hostname))
-	}
-	hostsCond := strings.Join(hostsConds, " or ")
-
-	// lenMet := len(metrics)
-
-	for _, metric := range metrics {
-			// log.Println(fmt.Sprintf("<< You are here: %s (Index %d of %d metrics)", metric, index, lenMet))
-
-			query := fmt.Sprintf(`
-				  data = from(bucket: "%s")
-				  |> range(start: %s, stop: %s)
-				  |> filter(fn: (r) => r._measurement == "%s" and r._field == "value" and (%s))
-				  union(tables: [data |> mean(column: "_value") |> set(key: "_field", value: "avg"),
-				                 data |>  min(column: "_value") |> set(key: "_field", value: "min"),
-				                 data |>  max(column: "_value") |> set(key: "_field", value: "max")])
-				  |> pivot(rowKey: ["hostname"], columnKey: ["_field"], valueColumn: "_value")
-				  |> group()`,
-					idb.bucket,
-					idb.formatTime(job.StartTime), idb.formatTime(idb.epochToTime(job.StartTimeUnix + int64(job.Duration) + int64(1) )),
-					metric, hostsCond)
-
-			rows, err := idb.queryClient.Query(ctx, query)
-			if err != nil {
-				return nil, err
-			}
-
-			nodes := map[string]schema.MetricStatistics{}
-			for rows.Next() {
-					row  := rows.Record()
-					host := row.ValueByKey("hostname").(string)
-
-					avg, avgok  := row.ValueByKey("avg").(float64)
-					if !avgok {
-						// log.Println(fmt.Sprintf(">> Assertion error for metric %s, statistic AVG. Expected 'float64', got %v", metric, avg))
-						avg = 0.0
-					}
-					min, minok  := row.ValueByKey("min").(float64)
-					if !minok {
-						// log.Println(fmt.Sprintf(">> Assertion error for metric %s, statistic MIN. Expected 'float64', got %v", metric, min))
-						min = 0.0
-					}
-					max, maxok  := row.ValueByKey("max").(float64)
-					if !maxok {
-						// log.Println(fmt.Sprintf(">> Assertion error for metric %s, statistic MAX. Expected 'float64', got %v", metric, max))
-						max = 0.0
-					}
-
-					nodes[host] = schema.MetricStatistics{
-						Avg: avg,
-						Min: min,
-						Max: max,
-					}
-			}
-			stats[metric] = nodes
-	}
-
-	return stats, nil
-}
-
-func (idb *InfluxDBv2DataRepository) LoadNodeData(cluster string, metrics, nodes []string, scopes []schema.MetricScope, from, to time.Time, ctx context.Context) (map[string]map[string][]*schema.JobMetric, error) {
-	// TODO : Implement to be used in Analysis- und System/Node-View
-	log.Println(fmt.Sprintf("LoadNodeData unimplemented for InfluxDBv2DataRepository, Args: cluster %s, metrics %v, nodes %v, scopes %v", cluster, metrics, nodes, scopes))
-
-	return nil, errors.New("unimplemented for InfluxDBv2DataRepository")
-}

pkg/lrucache/README.md

@@ -0,0 +1,121 @@
+# In-Memory LRU Cache for Golang Applications
+
+[![](https://pkg.go.dev/badge/github.com/iamlouk/lrucache?utm_source=godoc)](https://pkg.go.dev/github.com/iamlouk/lrucache)
+
+This library can be embedded into your existing go applications
+and play the role *Memcached* or *Redis* might play for others.
+It is inspired by [PHP Symfony's Cache Components](https://symfony.com/doc/current/components/cache/adapters/array_cache_adapter.html),
+having a similar API. This library can not be used for persistance,
+is not properly tested yet and a bit special in a few ways described
+below (Especially with regards to the memory usage/`size`).
+
+In addition to the interface described below, a `http.Handler` that can be used as middleware is provided as well.
+
+- Advantages:
+    - Anything (`interface{}`) can be stored as value
+    - As it lives in the application itself, no serialization or de-serialization is needed
+    - As it lives in the application itself, no memory moving/networking is needed
+    - The computation of a new value for a key does __not__ block the full cache (only the key)
+- Disadvantages:
+    - You have to provide a size estimate for every value
+    - __This size estimate should not change (i.e. values should not mutate)__
+    - The cache can only be accessed by one application
+
+## Example
+
+```go
+// Go look at the godocs and ./cache_test.go for more documentation and examples
+
+maxMemory := 1000
+cache := lrucache.New(maxMemory)
+
+bar = cache.Get("foo", func () (value interface{}, ttl time.Duration, size int) {
+	return "bar", 10 * time.Second, len("bar")
+}).(string)
+
+// bar == "bar"
+
+bar = cache.Get("foo", func () (value interface{}, ttl time.Duration, size int) {
+	panic("will not be called")
+}).(string)
+```
+
+## Why does `cache.Get` take a function as argument?
+
+*Using the mechanism described below is optional, the second argument to `Get` can be `nil` and there is a `Put` function as well.*
+
+Because this library is meant to be used by multi threaded applications and the following would
+result in the same data being fetched twice if both goroutines run in parallel:
+
+```go
+// This code shows what could happen with other cache libraries
+c := lrucache.New(MAX_CACHE_ENTRIES)
+
+for i := 0; i < 2; i++ {
+    go func(){
+        // This code will run twice in different goroutines,
+        // it could overlap. As `fetchData` probably does some
+        // I/O and takes a long time, the probability of both
+        // goroutines calling `fetchData` is very high!
+        url := "http://example.com/foo"
+        contents := c.Get(url)
+        if contents == nil {
+            contents = fetchData(url)
+            c.Set(url, contents)
+        }
+
+        handleData(contents.([]byte))
+    }()
+}
+
+```
+
+Here, if one wanted to make sure that only one of both goroutines fetches the data,
+the programmer would need to build his own synchronization. That would suck!
+
+```go
+c := lrucache.New(MAX_CACHE_SIZE)
+
+for i := 0; i < 2; i++ {
+    go func(){
+        url := "http://example.com/foo"
+        contents := c.Get(url, func()(interface{}, time.Time, int) {
+            // This closure will only be called once!
+            // If another goroutine calls `c.Get` while this closure
+            // is still being executed, it will wait.
+            buf := fetchData(url)
+            return buf, 100 * time.Second, len(buf)
+        })
+
+        handleData(contents.([]byte))
+    }()
+}
+
+```
+
+This is much better as less resources are wasted and synchronization is handled by
+the library. If it gets called, the call to the closure happens synchronously. While
+it is being executed, all other cache keys can still be accessed without having to wait
+for the execution to be done.
+
+## How `Get` works
+
+The closure passed to `Get` will be called if the value asked for is not cached or
+expired. It should return the following values:
+
+- The value corresponding to that key and to be stored in the cache
+- The time to live for that value (how long until it expires and needs to be recomputed)
+- A size estimate
+
+When `maxMemory` is reached, cache entries need to be evicted. Theoretically,
+it would be possible to use reflection on every value placed in the cache
+to get its exact size in bytes. This would be very expansive and slow though.
+Also, size can change. Instead of this library calculating the size in bytes, you, the user,
+have to provide a size for every value in whatever unit you like (as long as it is the same unit everywhere).
+
+Suggestions on what to use as size: `len(str)` for strings, `len(slice) * size_of_slice_type`, etc.. It is possible
+to use `1` as size for every entry, in that case at most `maxMemory` entries will be in the cache at the same time.
+
+## Affects on GC
+
+Because of the way a garbage collector decides when to run ([explained in the runtime package](https://pkg.go.dev/runtime)), having large amounts of data sitting in your cache might increase the memory consumption of your process by two times the maximum size of the cache. You can decrease the *target percentage* to reduce the effect, but then you might have negative performance effects when your cache is not filled.

pkg/lrucache/cache.go

@@ -0,0 +1,288 @@
+package lrucache
+
+import (
+	"sync"
+	"time"
+)
+
+// Type of the closure that must be passed to `Get` to
+// compute the value in case it is not cached.
+//
+// returned values are the computed value to be stored in the cache,
+// the duration until this value will expire and a size estimate.
+type ComputeValue func() (value interface{}, ttl time.Duration, size int)
+
+type cacheEntry struct {
+	key   string
+	value interface{}
+
+	expiration            time.Time
+	size                  int
+	waitingForComputation int
+
+	next, prev *cacheEntry
+}
+
+type Cache struct {
+	mutex                 sync.Mutex
+	cond                  *sync.Cond
+	maxmemory, usedmemory int
+	entries               map[string]*cacheEntry
+	head, tail            *cacheEntry
+}
+
+// Return a new instance of a LRU In-Memory Cache.
+// Read [the README](./README.md) for more information
+// on what is going on with `maxmemory`.
+func New(maxmemory int) *Cache {
+	cache := &Cache{
+		maxmemory: maxmemory,
+		entries:   map[string]*cacheEntry{},
+	}
+	cache.cond = sync.NewCond(&cache.mutex)
+	return cache
+}
+
+// Return the cached value for key `key` or call `computeValue` and
+// store its return value in the cache. If called, the closure will be
+// called synchronous and __shall not call methods on the same cache__
+// or a deadlock might ocure. If `computeValue` is nil, the cache is checked
+// and if no entry was found, nil is returned. If another goroutine is currently
+// computing that value, the result is waited for.
+func (c *Cache) Get(key string, computeValue ComputeValue) interface{} {
+	now := time.Now()
+
+	c.mutex.Lock()
+	if entry, ok := c.entries[key]; ok {
+		// The expiration not being set is what shows us that
+		// the computation of that value is still ongoing.
+		for entry.expiration.IsZero() {
+			entry.waitingForComputation += 1
+			c.cond.Wait()
+			entry.waitingForComputation -= 1
+		}
+
+		if now.After(entry.expiration) {
+			if !c.evictEntry(entry) {
+				if entry.expiration.IsZero() {
+					panic("cache entry that shoud have been waited for could not be evicted.")
+				}
+				c.mutex.Unlock()
+				return entry.value
+			}
+		} else {
+			if entry != c.head {
+				c.unlinkEntry(entry)
+				c.insertFront(entry)
+			}
+			c.mutex.Unlock()
+			return entry.value
+		}
+	}
+
+	if computeValue == nil {
+		c.mutex.Unlock()
+		return nil
+	}
+
+	entry := &cacheEntry{
+		key:                   key,
+		waitingForComputation: 1,
+	}
+
+	c.entries[key] = entry
+
+	hasPaniced := true
+	defer func() {
+		if hasPaniced {
+			c.mutex.Lock()
+			delete(c.entries, key)
+			entry.expiration = now
+			entry.waitingForComputation -= 1
+		}
+		c.mutex.Unlock()
+	}()
+
+	c.mutex.Unlock()
+	value, ttl, size := computeValue()
+	c.mutex.Lock()
+	hasPaniced = false
+
+	entry.value = value
+	entry.expiration = now.Add(ttl)
+	entry.size = size
+	entry.waitingForComputation -= 1
+
+	// Only broadcast if other goroutines are actually waiting
+	// for a result.
+	if entry.waitingForComputation > 0 {
+		// TODO: Have more than one condition variable so that there are
+		// less unnecessary wakeups.
+		c.cond.Broadcast()
+	}
+
+	c.usedmemory += size
+	c.insertFront(entry)
+
+	// Evict only entries with a size of more than zero.
+	// This is the only loop in the implementation outside of the `Keys`
+	// method.
+	evictionCandidate := c.tail
+	for c.usedmemory > c.maxmemory && evictionCandidate != nil {
+		nextCandidate := evictionCandidate.prev
+		if (evictionCandidate.size > 0 || now.After(evictionCandidate.expiration)) &&
+			evictionCandidate.waitingForComputation == 0 {
+			c.evictEntry(evictionCandidate)
+		}
+		evictionCandidate = nextCandidate
+	}
+
+	return value
+}
+
+// Put a new value in the cache. If another goroutine is calling `Get` and
+// computing the value, this function waits for the computation to be done
+// before it overwrites the value.
+func (c *Cache) Put(key string, value interface{}, size int, ttl time.Duration) {
+	now := time.Now()
+	c.mutex.Lock()
+	defer c.mutex.Unlock()
+
+	if entry, ok := c.entries[key]; ok {
+		for entry.expiration.IsZero() {
+			entry.waitingForComputation += 1
+			c.cond.Wait()
+			entry.waitingForComputation -= 1
+		}
+
+		c.usedmemory -= entry.size
+		entry.expiration = now.Add(ttl)
+		entry.size = size
+		entry.value = value
+		c.usedmemory += entry.size
+
+		c.unlinkEntry(entry)
+		c.insertFront(entry)
+		return
+	}
+
+	entry := &cacheEntry{
+		key:        key,
+		value:      value,
+		expiration: now.Add(ttl),
+	}
+	c.entries[key] = entry
+	c.insertFront(entry)
+}
+
+// Remove the value at key `key` from the cache.
+// Return true if the key was in the cache and false
+// otherwise. It is possible that true is returned even
+// though the value already expired.
+// It is possible that false is returned even though the value
+// will show up in the cache if this function is called on a key
+// while that key is beeing computed.
+func (c *Cache) Del(key string) bool {
+	c.mutex.Lock()
+	defer c.mutex.Unlock()
+
+	if entry, ok := c.entries[key]; ok {
+		return c.evictEntry(entry)
+	}
+	return false
+}
+
+// Call f for every entry in the cache. Some sanity checks
+// and eviction of expired keys are done as well.
+// The cache is fully locked for the complete duration of this call!
+func (c *Cache) Keys(f func(key string, val interface{})) {
+	c.mutex.Lock()
+	defer c.mutex.Unlock()
+
+	now := time.Now()
+
+	size := 0
+	for key, e := range c.entries {
+		if key != e.key {
+			panic("key mismatch")
+		}
+
+		if now.After(e.expiration) {
+			if c.evictEntry(e) {
+				continue
+			}
+		}
+
+		if e.prev != nil {
+			if e.prev.next != e {
+				panic("list corrupted")
+			}
+		}
+
+		if e.next != nil {
+			if e.next.prev != e {
+				panic("list corrupted")
+			}
+		}
+
+		size += e.size
+		f(key, e.value)
+	}
+
+	if size != c.usedmemory {
+		panic("size calculations failed")
+	}
+
+	if c.head != nil {
+		if c.tail == nil || c.head.prev != nil {
+			panic("head/tail corrupted")
+		}
+	}
+
+	if c.tail != nil {
+		if c.head == nil || c.tail.next != nil {
+			panic("head/tail corrupted")
+		}
+	}
+}
+
+func (c *Cache) insertFront(e *cacheEntry) {
+	e.next = c.head
+	c.head = e
+
+	e.prev = nil
+	if e.next != nil {
+		e.next.prev = e
+	}
+
+	if c.tail == nil {
+		c.tail = e
+	}
+}
+
+func (c *Cache) unlinkEntry(e *cacheEntry) {
+	if e == c.head {
+		c.head = e.next
+	}
+	if e.prev != nil {
+		e.prev.next = e.next
+	}
+	if e.next != nil {
+		e.next.prev = e.prev
+	}
+	if e == c.tail {
+		c.tail = e.prev
+	}
+}
+
+func (c *Cache) evictEntry(e *cacheEntry) bool {
+	if e.waitingForComputation != 0 {
+		// panic("cannot evict this entry as other goroutines need the value")
+		return false
+	}
+
+	c.unlinkEntry(e)
+	c.usedmemory -= e.size
+	delete(c.entries, e.key)
+	return true
+}

pkg/lrucache/cache_test.go

@@ -0,0 +1,219 @@
+package lrucache
+
+import (
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+)
+
+func TestBasics(t *testing.T) {
+	cache := New(123)
+
+	value1 := cache.Get("foo", func() (interface{}, time.Duration, int) {
+		return "bar", 1 * time.Second, 0
+	})
+
+	if value1.(string) != "bar" {
+		t.Error("cache returned wrong value")
+	}
+
+	value2 := cache.Get("foo", func() (interface{}, time.Duration, int) {
+		t.Error("value should be cached")
+		return "", 0, 0
+	})
+
+	if value2.(string) != "bar" {
+		t.Error("cache returned wrong value")
+	}
+
+	existed := cache.Del("foo")
+	if !existed {
+		t.Error("delete did not work as expected")
+	}
+
+	value3 := cache.Get("foo", func() (interface{}, time.Duration, int) {
+		return "baz", 1 * time.Second, 0
+	})
+
+	if value3.(string) != "baz" {
+		t.Error("cache returned wrong value")
+	}
+
+	cache.Keys(func(key string, value interface{}) {
+		if key != "foo" || value.(string) != "baz" {
+			t.Error("cache corrupted")
+		}
+	})
+}
+
+func TestExpiration(t *testing.T) {
+	cache := New(123)
+
+	failIfCalled := func() (interface{}, time.Duration, int) {
+		t.Error("Value should be cached!")
+		return "", 0, 0
+	}
+
+	val1 := cache.Get("foo", func() (interface{}, time.Duration, int) {
+		return "bar", 5 * time.Millisecond, 0
+	})
+	val2 := cache.Get("bar", func() (interface{}, time.Duration, int) {
+		return "foo", 20 * time.Millisecond, 0
+	})
+
+	val3 := cache.Get("foo", failIfCalled).(string)
+	val4 := cache.Get("bar", failIfCalled).(string)
+
+	if val1 != val3 || val3 != "bar" || val2 != val4 || val4 != "foo" {
+		t.Error("Wrong values returned")
+	}
+
+	time.Sleep(10 * time.Millisecond)
+
+	val5 := cache.Get("foo", func() (interface{}, time.Duration, int) {
+		return "baz", 0, 0
+	})
+	val6 := cache.Get("bar", failIfCalled)
+
+	if val5.(string) != "baz" || val6.(string) != "foo" {
+		t.Error("unexpected values")
+	}
+
+	cache.Keys(func(key string, val interface{}) {
+		if key != "bar" || val.(string) != "foo" {
+			t.Error("wrong value expired")
+		}
+	})
+
+	time.Sleep(15 * time.Millisecond)
+	cache.Keys(func(key string, val interface{}) {
+		t.Error("cache should be empty now")
+	})
+}
+
+func TestEviction(t *testing.T) {
+	c := New(100)
+	failIfCalled := func() (interface{}, time.Duration, int) {
+		t.Error("Value should be cached!")
+		return "", 0, 0
+	}
+
+	v1 := c.Get("foo", func() (interface{}, time.Duration, int) {
+		return "bar", 1 * time.Second, 1000
+	})
+
+	v2 := c.Get("foo", func() (interface{}, time.Duration, int) {
+		return "baz", 1 * time.Second, 1000
+	})
+
+	if v1.(string) != "bar" || v2.(string) != "baz" {
+		t.Error("wrong values returned")
+	}
+
+	c.Keys(func(key string, val interface{}) {
+		t.Error("cache should be empty now")
+	})
+
+	_ = c.Get("A", func() (interface{}, time.Duration, int) {
+		return "a", 1 * time.Second, 50
+	})
+
+	_ = c.Get("B", func() (interface{}, time.Duration, int) {
+		return "b", 1 * time.Second, 50
+	})
+
+	_ = c.Get("A", failIfCalled)
+	_ = c.Get("B", failIfCalled)
+	_ = c.Get("C", func() (interface{}, time.Duration, int) {
+		return "c", 1 * time.Second, 50
+	})
+
+	_ = c.Get("B", failIfCalled)
+	_ = c.Get("C", failIfCalled)
+
+	v4 := c.Get("A", func() (interface{}, time.Duration, int) {
+		return "evicted", 1 * time.Second, 25
+	})
+
+	if v4.(string) != "evicted" {
+		t.Error("value should have been evicted")
+	}
+
+	c.Keys(func(key string, val interface{}) {
+		if key != "A" && key != "C" {
+			t.Errorf("'%s' was not expected", key)
+		}
+	})
+}
+
+// I know that this is a shity test,
+// time is relative and unreliable.
+func TestConcurrency(t *testing.T) {
+	c := New(100)
+	var wg sync.WaitGroup
+
+	numActions := 20000
+	numThreads := 4
+	wg.Add(numThreads)
+
+	var concurrentModifications int32 = 0
+
+	for i := 0; i < numThreads; i++ {
+		go func() {
+			for j := 0; j < numActions; j++ {
+				_ = c.Get("key", func() (interface{}, time.Duration, int) {
+					m := atomic.AddInt32(&concurrentModifications, 1)
+					if m != 1 {
+						t.Error("only one goroutine at a time should calculate a value for the same key")
+					}
+
+					time.Sleep(1 * time.Millisecond)
+					atomic.AddInt32(&concurrentModifications, -1)
+					return "value", 3 * time.Millisecond, 1
+				})
+			}
+
+			wg.Done()
+		}()
+	}
+
+	wg.Wait()
+
+	c.Keys(func(key string, val interface{}) {})
+}
+
+func TestPanic(t *testing.T) {
+	c := New(100)
+
+	c.Put("bar", "baz", 3, 1*time.Minute)
+
+	testpanic := func() {
+		defer func() {
+			if r := recover(); r != nil {
+				if r.(string) != "oops" {
+					t.Fatal("unexpected panic value")
+				}
+			}
+		}()
+
+		_ = c.Get("foo", func() (value interface{}, ttl time.Duration, size int) {
+			panic("oops")
+		})
+
+		t.Fatal("should have paniced!")
+	}
+
+	testpanic()
+
+	v := c.Get("bar", func() (value interface{}, ttl time.Duration, size int) {
+		t.Fatal("should not be called!")
+		return nil, 0, 0
+	})
+
+	if v.(string) != "baz" {
+		t.Fatal("unexpected value")
+	}
+
+	testpanic()
+}

pkg/lrucache/handler.go

@@ -0,0 +1,120 @@
+package lrucache
+
+import (
+	"bytes"
+	"net/http"
+	"strconv"
+	"time"
+)
+
+// HttpHandler is can be used as HTTP Middleware in order to cache requests,
+// for example static assets. By default, the request's raw URI is used as key and nothing else.
+// Results with a status code other than 200 are cached with a TTL of zero seconds,
+// so basically re-fetched as soon as the current fetch is done and a new request
+// for that URI is done.
+type HttpHandler struct {
+	cache      *Cache
+	fetcher    http.Handler
+	defaultTTL time.Duration
+
+	// Allows overriding the way the cache key is extracted
+	// from the http request. The defailt is to use the RequestURI.
+	CacheKey func(*http.Request) string
+}
+
+var _ http.Handler = (*HttpHandler)(nil)
+
+type cachedResponseWriter struct {
+	w          http.ResponseWriter
+	statusCode int
+	buf        bytes.Buffer
+}
+
+type cachedResponse struct {
+	headers    http.Header
+	statusCode int
+	data       []byte
+	fetched    time.Time
+}
+
+var _ http.ResponseWriter = (*cachedResponseWriter)(nil)
+
+func (crw *cachedResponseWriter) Header() http.Header {
+	return crw.w.Header()
+}
+
+func (crw *cachedResponseWriter) Write(bytes []byte) (int, error) {
+	return crw.buf.Write(bytes)
+}
+
+func (crw *cachedResponseWriter) WriteHeader(statusCode int) {
+	crw.statusCode = statusCode
+}
+
+// Returns a new caching HttpHandler. If no entry in the cache is found or it was too old, `fetcher` is called with
+// a modified http.ResponseWriter and the response is stored in the cache. If `fetcher` sets the "Expires" header,
+// the ttl is set appropriately (otherwise, the default ttl passed as argument here is used).
+// `maxmemory` should be in the unit bytes.
+func NewHttpHandler(maxmemory int, ttl time.Duration, fetcher http.Handler) *HttpHandler {
+	return &HttpHandler{
+		cache:      New(maxmemory),
+		defaultTTL: ttl,
+		fetcher:    fetcher,
+		CacheKey: func(r *http.Request) string {
+			return r.RequestURI
+		},
+	}
+}
+
+// gorilla/mux style middleware:
+func NewMiddleware(maxmemory int, ttl time.Duration) func(http.Handler) http.Handler {
+	return func(next http.Handler) http.Handler {
+		return NewHttpHandler(maxmemory, ttl, next)
+	}
+}
+
+// Tries to serve a response to r from cache or calls next and stores the response to the cache for the next time.
+func (h *HttpHandler) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
+	if r.Method != http.MethodGet {
+		h.ServeHTTP(rw, r)
+		return
+	}
+
+	cr := h.cache.Get(h.CacheKey(r), func() (interface{}, time.Duration, int) {
+		crw := &cachedResponseWriter{
+			w:          rw,
+			statusCode: 200,
+			buf:        bytes.Buffer{},
+		}
+
+		h.fetcher.ServeHTTP(crw, r)
+
+		cr := &cachedResponse{
+			headers:    rw.Header().Clone(),
+			statusCode: crw.statusCode,
+			data:       crw.buf.Bytes(),
+			fetched:    time.Now(),
+		}
+		cr.headers.Set("Content-Length", strconv.Itoa(len(cr.data)))
+
+		ttl := h.defaultTTL
+		if cr.statusCode != http.StatusOK {
+			ttl = 0
+		} else if cr.headers.Get("Expires") != "" {
+			if expires, err := http.ParseTime(cr.headers.Get("Expires")); err == nil {
+				ttl = time.Until(expires)
+			}
+		}
+
+		return cr, ttl, len(cr.data)
+	}).(*cachedResponse)
+
+	for key, val := range cr.headers {
+		rw.Header()[key] = val
+	}
+
+	cr.headers.Set("Age", strconv.Itoa(int(time.Since(cr.fetched).Seconds())))
+
+	rw.WriteHeader(cr.statusCode)
+	rw.Write(cr.data)
+}

pkg/lrucache/handler_test.go

@@ -0,0 +1,71 @@
+package lrucache
+
+import (
+	"bytes"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+	"time"
+)
+
+func TestHandlerBasics(t *testing.T) {
+	r := httptest.NewRequest(http.MethodGet, "/test1", nil)
+	rw := httptest.NewRecorder()
+	shouldBeCalled := true
+
+	handler := NewHttpHandler(1000, time.Second, http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
+		rw.Write([]byte("Hello World!"))
+
+		if !shouldBeCalled {
+			t.Fatal("fetcher expected to be called")
+		}
+	}))
+
+	handler.ServeHTTP(rw, r)
+
+	if rw.Code != 200 {
+		t.Fatal("unexpected status code")
+	}
+
+	if !bytes.Equal(rw.Body.Bytes(), []byte("Hello World!")) {
+		t.Fatal("unexpected body")
+	}
+
+	rw = httptest.NewRecorder()
+	shouldBeCalled = false
+	handler.ServeHTTP(rw, r)
+
+	if rw.Code != 200 {
+		t.Fatal("unexpected status code")
+	}
+
+	if !bytes.Equal(rw.Body.Bytes(), []byte("Hello World!")) {
+		t.Fatal("unexpected body")
+	}
+}
+
+// func TestHandlerExpiration(t *testing.T) {
+// 	r := httptest.NewRequest(http.MethodGet, "/test1", nil)
+// 	rw := httptest.NewRecorder()
+// 	i := 1
+// 	now := time.Now()
+
+// 	handler := NewHttpHandler(1000, 1*time.Second, http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
+// 		rw.Header().Set("Expires", now.Add(10*time.Millisecond).Format(http.TimeFormat))
+// 		rw.Write([]byte(strconv.Itoa(i)))
+// 	}))
+
+// 	handler.ServeHTTP(rw, r)
+// 	if !(rw.Body.String() == strconv.Itoa(1)) {
+// 		t.Fatal("unexpected body")
+// 	}
+
+// 	i += 1
+
+// 	time.Sleep(11 * time.Millisecond)
+// 	rw = httptest.NewRecorder()
+// 	handler.ServeHTTP(rw, r)
+// 	if !(rw.Body.String() == strconv.Itoa(1)) {
+// 		t.Fatal("unexpected body")
+// 	}
+// }

startDemo.sh

@@ -8,13 +8,13 @@ tar xJf job-archive.tar.xz
 rm ./job-archive.tar.xz
 
 touch ./job.db
-cd ../frontend
+cd ../web/frontend
 yarn install
 yarn build
 
-cd ..
+cd ../..
+cp ./configs/env-template.txt .env
 go get
-go build
+go build ./cmd/cc-backend
 
-./cc-backend --init-db --add-user demo:admin:AdminDev --no-server
+./cc-backend --init-db --add-user demo:admin:AdminDev
-./cc-backend

test/api_test.go

@@ -4,7 +4,6 @@ import (
 	"bytes"
 	"context"
 	"encoding/json"
-	"fmt"
 	"net/http"
 	"net/http/httptest"
 	"os"
@@ -14,14 +13,13 @@ import (
 	"strings"
 	"testing"
 
-	"github.com/ClusterCockpit/cc-backend/api"
+	"github.com/ClusterCockpit/cc-backend/internal/api"
-	"github.com/ClusterCockpit/cc-backend/config"
+	"github.com/ClusterCockpit/cc-backend/internal/config"
-	"github.com/ClusterCockpit/cc-backend/graph"
+	"github.com/ClusterCockpit/cc-backend/internal/graph"
-	"github.com/ClusterCockpit/cc-backend/metricdata"
+	"github.com/ClusterCockpit/cc-backend/internal/metricdata"
-	"github.com/ClusterCockpit/cc-backend/repository"
+	"github.com/ClusterCockpit/cc-backend/internal/repository"
-	"github.com/ClusterCockpit/cc-backend/schema"
+	"github.com/ClusterCockpit/cc-backend/pkg/schema"
 	"github.com/gorilla/mux"
-	"github.com/jmoiron/sqlx"
 
 	_ "github.com/mattn/go-sqlite3"
 )
@@ -95,17 +93,14 @@ func setup(t *testing.T) *api.RestApi {
 	}
 	f.Close()
 
-	db, err := sqlx.Open("sqlite3", fmt.Sprintf("%s?_foreign_keys=on", dbfilepath))
+	repository.Connect("sqlite3", dbfilepath)
-	if err != nil {
+	db := repository.GetConnection()
+
+	if _, err := db.DB.Exec(repository.JobsDBSchema); err != nil {
 		t.Fatal(err)
 	}
 
-	db.SetMaxOpenConns(1)
+	if err := config.Init(db.DB, false, map[string]interface{}{}, jobarchive); err != nil {
-	if _, err := db.Exec(repository.JobsDBSchema); err != nil {
-		t.Fatal(err)
-	}
-
-	if err := config.Init(db, false, map[string]interface{}{}, jobarchive); err != nil {
 		t.Fatal(err)
 	}
 
@@ -113,10 +108,8 @@ func setup(t *testing.T) *api.RestApi {
 		t.Fatal(err)
 	}
 
-	resolver := &graph.Resolver{DB: db, Repo: &repository.JobRepository{DB: db}}
+	jobRepo := repository.GetRepository()
-	if err := resolver.Repo.Init(); err != nil {
+	resolver := &graph.Resolver{DB: db.DB, Repo: jobRepo}
-		t.Fatal(err)
-	}
 
 	return &api.RestApi{
 		JobRepository: resolver.Repo,

tools/README.md

@@ -0,0 +1,46 @@
+## Introduction
+
+ClusterCockpit uses JSON Web Tokens (JWT) for authorization of its APIs.
+JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object.
+This information can be verified and trusted because it is digitally signed.
+In ClusterCockpit JWTs are signed using a public/private key pair using ECDSA.
+Because tokens are signed using public/private key pairs, the signature also certifies that only the party holding the private key is the one that signed it.
+Currently JWT tokens not yet expire.
+
+## JWT Payload
+
+You may view the payload of a JWT token at [https://jwt.io/#debugger-io](https://jwt.io/#debugger-io).
+Currently ClusterCockpit sets the following claims:
+* `iat`: Issued at claim. The “iat” claim is used to identify the the time at which the JWT was issued. This claim can be used to determine the age of the JWT.
+* `sub`: Subject claim. Identifies the subject of the JWT, in our case this is the username.
+* `roles`: An array of strings specifying the roles set for the subject.
+
+## Workflow
+
+1. Create a new ECDSA Public/private keypair:
+```
+$ go build ./tools/gen-keypair.go
+$ ./gen-keypair
+```
+2. Add keypair in your `.env` file. A template can be found in `./configs`.
+
+There are two usage scenarios:
+* The APIs are used during a browser session. In this case on login a JWT token is issued on login, that is used by the web frontend to authorize against the GraphQL and REST APIs.
+* The REST API is used outside a browser session, e.g. by scripts. In this case you have to issue a token manually. This possible from within the configuration view or on the command line. It is recommended to issue a JWT token in this case for a special user that only has the `api` role. By using different users for different purposes a fine grained access control and access revocation management is possible.
+
+The token is commonly specified in the Authorization HTTP header using the Bearer schema.
+
+## Setup user and JWT token for REST API authorization
+
+1. Create user:
+```
+$ ./cc-backend --add-user <username>:api:<Password> --no-server
+```
+2. Issue token for user:
+```
+$ ./cc-backend -jwt <username>  -no-server
+```
+3. Use issued token token on client side:
+```
+$ curl -X GET "<API ENDPOINT>" -H "accept: application/json"  -H "Content-Type: application/json"  -H "Authorization: Bearer <JWT TOKEN>"
+```

tools/gen-keypair.go

@@ -0,0 +1,22 @@
+package main
+
+import (
+	"crypto/ed25519"
+	"crypto/rand"
+	"encoding/base64"
+	"fmt"
+	"os"
+)
+
+func main() {
+	// rand.Reader uses /dev/urandom on Linux
+	pub, priv, err := ed25519.GenerateKey(rand.Reader)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "error: %s\n", err.Error())
+		os.Exit(1)
+	}
+
+	fmt.Fprintf(os.Stdout, "JWT_PUBLIC_KEY=%#v\nJWT_PRIVATE_KEY=%#v\n",
+		base64.StdEncoding.EncodeToString(pub),
+		base64.StdEncoding.EncodeToString(priv))
+}

web/frontend/README.md

@@ -0,0 +1,31 @@
+# cc-svelte-datatable
+
+[![Build](https://github.com/ClusterCockpit/cc-svelte-datatable/actions/workflows/build.yml/badge.svg)](https://github.com/ClusterCockpit/cc-svelte-datatable/actions/workflows/build.yml)
+
+A frontend for [ClusterCockpit](https://github.com/ClusterCockpit/ClusterCockpit) and [cc-backend](https://github.com/ClusterCockpit/cc-backend). Backend specific configuration can de done using the constants defined in the `intro` section in `./rollup.config.js`.
+
+Builds on:
+* [Svelte](https://svelte.dev/)
+* [SvelteStrap](https://sveltestrap.js.org/)
+* [Bootstrap 5](https://getbootstrap.com/)
+* [urql](https://github.com/FormidableLabs/urql)
+
+## Get started
+
+[Yarn](https://yarnpkg.com/) is recommended for package management.
+Due to an issue with Yarn v2 you have to stick to Yarn v1.
+
+Install the dependencies...
+
+```bash
+yarn install
+```
+
+...then start [Rollup](https://rollupjs.org):
+
+```bash
+yarn run dev
+```
+
+Edit a component file in `src`, save it, and reload the page to see your changes.
+

web/frontend/package.json

@@ -0,0 +1,25 @@
+{
+  "name": "svelte-app",
+  "version": "1.0.0",
+  "scripts": {
+    "build": "rollup -c",
+    "dev": "rollup -c -w"
+  },
+  "devDependencies": {
+    "@rollup/plugin-commonjs": "^17.0.0",
+    "@rollup/plugin-node-resolve": "^11.0.0",
+    "rollup": "^2.3.4",
+    "rollup-plugin-css-only": "^3.1.0",
+    "rollup-plugin-svelte": "^7.0.0",
+    "rollup-plugin-terser": "^7.0.0",
+    "svelte": "^3.42.6"
+  },
+  "dependencies": {
+    "@rollup/plugin-replace": "^2.4.1",
+    "@urql/svelte": "^1.3.0",
+    "graphql": "^15.6.0",
+    "sveltestrap": "^5.6.1",
+    "uplot": "^1.6.7",
+    "wonka": "^4.0.15"
+  }
+}

web/frontend/public/global.css

@@ -0,0 +1,54 @@
+html, body {
+	position: relative;
+	width: 100%;
+	height: 100%;
+}
+
+body {
+	color: #333;
+	margin: 0;
+	padding: 8px;
+	box-sizing: border-box;
+	font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif;
+}
+
+.container {
+	max-width: 100vw;
+}
+
+.site {
+    display: flex;
+    flex-direction: column;
+    height: 100%;
+}
+
+.site-content {
+    flex: 1 0 auto;
+    margin-top: 80px;
+}
+
+.site-footer {
+    flex: none;
+}
+
+footer {
+    width: 100%;
+    padding: 0.1rem 1.0rem;
+    line-height: 1.5;
+}
+
+.footer-list {
+    list-style-type: none;
+    padding-left: 0;
+    width: 100%;
+    display: flex;
+    flex-wrap: wrap;
+    justify-content: center;
+    margin-top: 5px;
+    margin-bottom: 5px;
+}
+
+.footer-list-item {
+    margin: 0rem 0.8rem;
+    white-space: nowrap;
+}

web/frontend/public/uPlot.min.css

@@ -0,0 +1 @@
+../node_modules/uplot/dist/uPlot.min.css

web/frontend/rollup.config.js

@@ -0,0 +1,70 @@
+import svelte from 'rollup-plugin-svelte';
+import replace from "@rollup/plugin-replace";
+import commonjs from '@rollup/plugin-commonjs';
+import resolve from '@rollup/plugin-node-resolve';
+import { terser } from 'rollup-plugin-terser';
+import css from 'rollup-plugin-css-only';
+
+const production = !process.env.ROLLUP_WATCH;
+
+const plugins = [
+	svelte({
+		compilerOptions: {
+			// enable run-time checks when not in production
+			dev: !production
+		}
+	}),
+
+	// If you have external dependencies installed from
+	// npm, you'll most likely need these plugins. In
+	// some cases you'll need additional configuration -
+	// consult the documentation for details:
+	// https://github.com/rollup/plugins/tree/master/packages/commonjs
+	resolve({
+		browser: true,
+		dedupe: ['svelte']
+	}),
+	commonjs(),
+
+	// If we're building for production (npm run build
+	// instead of npm run dev), minify
+	production && terser(),
+
+	replace({
+		"process.env.NODE_ENV": JSON.stringify("development"),
+		preventAssignment: true
+	})
+];
+
+const entrypoint = (name, path) => ({
+	input: path,
+	output: {
+		sourcemap: false,
+		format: 'iife',
+		name: 'app',
+		file: `public/build/${name}.js`
+	},
+	plugins: [
+		...plugins,
+
+		// we'll extract any component CSS out into
+		// a separate file - better for performance
+		css({ output: `${name}.css` }),
+	],
+	watch: {
+		clearScreen: false
+	}
+});
+
+export default [
+	entrypoint('header',   'src/header.entrypoint.js'),
+	entrypoint('jobs',     'src/jobs.entrypoint.js'),
+	entrypoint('user',     'src/user.entrypoint.js'),
+	entrypoint('list',     'src/list.entrypoint.js'),
+	entrypoint('job',      'src/job.entrypoint.js'),
+	entrypoint('systems',  'src/systems.entrypoint.js'),
+	entrypoint('node',     'src/node.entrypoint.js'),
+	entrypoint('analysis', 'src/analysis.entrypoint.js'),
+	entrypoint('status',   'src/status.entrypoint.js')
+];
+

web/frontend/src/Analysis.root.svelte

@@ -0,0 +1,265 @@
+<script>
+    import { init } from './utils.js'
+    import { getContext, onMount } from 'svelte'
+    import { operationStore, query } from '@urql/svelte'
+    import { Row, Col, Spinner, Card, Table } from 'sveltestrap'
+    import Filters from './filters/Filters.svelte'
+    import PlotSelection from './PlotSelection.svelte'
+    import Histogram, { binsFromFootprint } from './plots/Histogram.svelte'
+    import ScatterPlot from './plots/Scatter.svelte'
+    import PlotTable from './PlotTable.svelte'
+    import Roofline from './plots/Roofline.svelte'
+
+    const { query: initq } = init()
+
+    export let filterPresets
+
+    // By default, look at the jobs of the last 6 hours:
+    if (filterPresets?.startTime == null) {
+        if (filterPresets == null)
+            filterPresets = {}
+
+        let now = new Date(Date.now())
+        let hourAgo = new Date(now)
+        hourAgo.setHours(hourAgo.getHours() - 6)
+        filterPresets.startTime = { from: hourAgo.toISOString(), to: now.toISOString() }
+    }
+
+    let cluster
+    let filters
+    let rooflineMaxY
+    let colWidth
+    let numBins = 50
+    const ccconfig = getContext('cc-config'),
+          metricConfig = getContext('metrics')
+
+    let metricsInHistograms = ccconfig.analysis_view_histogramMetrics,
+        metricsInScatterplots = ccconfig.analysis_view_scatterPlotMetrics
+
+    $: metrics = [...new Set([...metricsInHistograms, ...metricsInScatterplots.flat()])]
+
+    getContext('on-init')(({ data }) => {
+        if (data != null) {
+            cluster = data.clusters.find(c => c.name == filterPresets.cluster)
+            console.assert(cluster != null, `This cluster could not be found: ${filterPresets.cluster}`)
+
+            rooflineMaxY = cluster.subClusters.reduce((max, part) => Math.max(max, part.flopRateSimd), 0)
+            $rooflineQuery.variables.maxY = rooflineMaxY
+            $rooflineQuery.context.pause = false
+            $rooflineQuery.reexecute()
+        }
+    })
+
+    const statsQuery = operationStore(`
+        query($filter: [JobFilter!]!) {
+            stats: jobsStatistics(filter: $filter) {
+                totalJobs
+                shortJobs
+                totalWalltime
+                totalCoreHours
+                histDuration { count, value }
+                histNumNodes { count, value }
+            }
+
+            topUsers: jobsCount(filter: $filter, groupBy: USER, weight: NODE_HOURS, limit: 5) { name, count }
+        }
+    `, { filter: [] }, { pause: true })
+
+    const footprintsQuery = operationStore(`
+        query($filter: [JobFilter!]!, $metrics: [String!]!) {
+            footprints: jobsFootprints(filter: $filter, metrics: $metrics) {
+                nodehours,
+                metrics { metric, data }
+            }
+        }
+    `, { filter: [], metrics }, { pause: true })
+    $: $footprintsQuery.variables = { ...$footprintsQuery.variables, metrics }
+
+    const rooflineQuery = operationStore(`
+        query($filter: [JobFilter!]!, $rows: Int!, $cols: Int!,
+                $minX: Float!, $minY: Float!, $maxX: Float!, $maxY: Float!) {
+            rooflineHeatmap(filter: $filter, rows: $rows, cols: $cols,
+                    minX: $minX, minY: $minY, maxX: $maxX, maxY: $maxY)
+        }
+    `, {
+        filter: [],
+        rows: 50, cols: 50,
+        minX: 0.01, minY: 1., maxX: 1000., maxY: -1
+    }, { pause: true });
+
+    query(statsQuery)
+    query(footprintsQuery)
+    query(rooflineQuery)
+    onMount(() => filters.update())
+</script>
+
+<Row>
+    {#if $initq.fetching || $statsQuery.fetching || $footprintsQuery.fetching}
+        <Col xs="auto">
+            <Spinner />
+        </Col>
+    {/if}
+    <Col xs="auto">
+        {#if $initq.error}
+            <Card body color="danger">{$initq.error.message}</Card>
+        {:else if cluster}
+            <PlotSelection
+                availableMetrics={cluster.metricConfig.map(mc => mc.name)}
+                bind:metricsInHistograms={metricsInHistograms}
+                bind:metricsInScatterplots={metricsInScatterplots} />
+        {/if}
+    </Col>
+    <Col xs="auto">
+        <Filters
+            bind:this={filters}
+            filterPresets={filterPresets}
+            disableClusterSelection={true}
+            startTimeQuickSelect={true}
+            on:update={({ detail }) => {
+                $statsQuery.context.pause = false
+                $statsQuery.variables = { filter: detail.filters }
+                $footprintsQuery.context.pause = false
+                $footprintsQuery.variables = { metrics, filter: detail.filters }
+                $rooflineQuery.variables = { ...$rooflineQuery.variables, filter: detail.filters }
+            }} />
+    </Col>
+</Row>
+
+<br/>
+{#if $statsQuery.error}
+    <Row>
+        <Col>
+            <Card body color="danger">{$statsQuery.error.message}</Card>
+        </Col>
+    </Row>
+{:else if $statsQuery.data}
+    <Row>
+        <div class="col-3" bind:clientWidth={colWidth}>
+            <div style="height: 40%">
+                <Table>
+                    <tr>
+                        <th scope="col">Total Jobs</th>
+                        <td>{$statsQuery.data.stats[0].totalJobs}</td>
+                    </tr>
+                    <tr>
+                        <th scope="col">Short Jobs (&#60; 2m)</th>
+                        <td>{$statsQuery.data.stats[0].shortJobs}</td>
+                    </tr>
+                    <tr>
+                        <th scope="col">Total Walltime</th>
+                        <td>{$statsQuery.data.stats[0].totalWalltime}</td>
+                    </tr>
+                    <tr>
+                        <th scope="col">Total Core Hours</th>
+                        <td>{$statsQuery.data.stats[0].totalCoreHours}</td>
+                    </tr>
+                </Table>
+            </div>
+            <div style="height: 60%;">
+                {#key $statsQuery.data.topUsers}
+                    <h4>Top Users (by node hours)</h4>
+                    <Histogram
+                        width={colWidth - 25} height={300 * 0.5}
+                        data={$statsQuery.data.topUsers.sort((a, b) => b.count - a.count).map(({ count }, idx) => ({ count, value: idx }))}
+                        label={(x) => x < $statsQuery.data.topUsers.length ? $statsQuery.data.topUsers[Math.floor(x)].name : '0'} />
+                {/key}
+            </div>
+        </div>
+        <div class="col-3">
+            {#key $statsQuery.data.stats[0].histDuration}
+                <h4>Walltime Distribution</h4>
+                <Histogram
+                    width={colWidth - 25} height={300}
+                    data={$statsQuery.data.stats[0].histDuration} />
+            {/key}
+        </div>
+        <div class="col-3">
+            {#key $statsQuery.data.stats[0].histNumNodes}
+                <h4>Number of Nodes Distribution</h4>
+                <Histogram
+                    width={colWidth - 25} height={300}
+                    data={$statsQuery.data.stats[0].histNumNodes} />
+            {/key}
+        </div>
+        <div class="col-3">
+            {#if $rooflineQuery.fetching}
+                <Spinner />
+            {:else if $rooflineQuery.error}
+                <Card body color="danger">{$rooflineQuery.error.message}</Card>
+            {:else if $rooflineQuery.data && cluster}
+                {#key $rooflineQuery.data}
+                    <Roofline
+                        width={colWidth - 25} height={300}
+                        tiles={$rooflineQuery.data.rooflineHeatmap}
+                        cluster={cluster.subClusters.length == 1 ? cluster.subClusters[0] : null}
+                        maxY={rooflineMaxY} />
+                {/key}
+            {/if}
+        </div>
+    </Row>
+{/if}
+
+<br/>
+{#if $footprintsQuery.error}
+    <Row>
+        <Col>
+            <Card body color="danger">{$footprintsQuery.error.message}</Card>
+        </Col>
+    </Row>
+{:else if $footprintsQuery.data && $initq.data}
+    <Row>
+        <Col>
+            <Card body>
+                These histograms show the distribution of the averages of all jobs matching the filters. Each job/average is weighted by its node hours.
+            </Card>
+            <br/>
+        </Col>
+    </Row>
+    <Row>
+        <Col>
+            <PlotTable
+                let:item
+                let:width
+                items={metricsInHistograms.map(metric => ({ metric, ...binsFromFootprint(
+                    $footprintsQuery.data.footprints.nodehours,
+                    $footprintsQuery.data.footprints.metrics.find(f => f.metric == metric).data, numBins) }))}
+                itemsPerRow={ccconfig.plot_view_plotsPerRow}>
+                <h4>{item.metric} [{metricConfig(cluster.name, item.metric)?.unit}]</h4>
+
+                <Histogram
+                    width={width} height={250}
+                    min={item.min} max={item.max}
+                    data={item.bins} label={item.label} />
+            </PlotTable>
+        </Col>
+    </Row>
+    <br/>
+    <Row>
+        <Col>
+            <Card body>
+                Each circle represents one job. The size of a circle is proportional to its node hours. Darker circles mean multiple jobs have the same averages for the respective metrics.
+            </Card>
+            <br/>
+        </Col>
+    </Row>
+    <Row>
+        <Col>
+            <PlotTable
+                let:item
+                let:width
+                items={metricsInScatterplots.map(([m1, m2]) => ({
+                    m1, f1: $footprintsQuery.data.footprints.metrics.find(f => f.metric == m1).data,
+                    m2, f2: $footprintsQuery.data.footprints.metrics.find(f => f.metric == m2).data }))}
+                itemsPerRow={ccconfig.plot_view_plotsPerRow}>
+
+                <ScatterPlot
+                    width={width} height={250} color={"rgba(0, 102, 204, 0.33)"}
+                    xLabel={`${item.m1} [${metricConfig(cluster.name, item.m1)?.unit}]`}
+                    yLabel={`${item.m2} [${metricConfig(cluster.name, item.m2)?.unit}]`}
+                    X={item.f1} Y={item.f2} S={$footprintsQuery.data.footprints.nodehours} />
+            </PlotTable>
+        </Col>
+    </Row>
+{/if}
+
+

web/frontend/src/Header.svelte

@@ -0,0 +1,73 @@
+<script>
+    import { Icon, Button, InputGroup, Input, Collapse,
+             Navbar, NavbarBrand, Nav, NavItem, NavLink, NavbarToggler,
+             Dropdown, DropdownToggle, DropdownMenu, DropdownItem } from 'sveltestrap'
+
+    export let username // empty string if auth. is disabled, otherwise the username as string
+    export let isAdmin  // boolean
+    export let clusters // array of names
+
+    let isOpen = false
+
+    const views = [
+        isAdmin
+            ? { title: 'Jobs',     adminOnly: false, href: '/monitoring/jobs/',            icon: 'card-list' }
+            : { title: 'My Jobs',  adminOnly: false, href: `/monitoring/user/${username}`, icon: 'bar-chart-line-fill' },
+        { title: 'Users',    adminOnly: true,  href: '/monitoring/users/',           icon: 'people-fill' },
+        { title: 'Projects', adminOnly: true,  href: '/monitoring/projects/',        icon: 'folder' },
+        { title: 'Tags',     adminOnly: false, href: '/monitoring/tags/',            icon: 'tags' }
+    ]
+    const viewsPerCluster = [
+        { title: 'Analysis', adminOnly: true,  href: '/monitoring/analysis/', icon: 'graph-up' },
+        { title: 'Systems',  adminOnly: true,  href: '/monitoring/systems/',  icon: 'cpu' },
+        { title: 'Status',   adminOnly: true,  href: '/monitoring/status/',   icon: 'cpu' },
+    ]
+</script>
+
+<Navbar color="light" light expand="lg" fixed="top">
+    <NavbarBrand href="/">
+        <img alt="ClusterCockpit Logo" src="/img/logo.png" height="25rem">
+    </NavbarBrand>
+    <NavbarToggler on:click={() => (isOpen = !isOpen)} />
+    <Collapse {isOpen} navbar expand="lg" on:update={({ detail }) => (isOpen = detail.isOpen)}>
+        <Nav pills>
+            {#each views.filter(item => isAdmin || !item.adminOnly) as item}
+                <NavLink href={item.href} active={window.location.pathname == item.href}><Icon name={item.icon}/> {item.title}</NavLink>
+            {/each}
+            {#each viewsPerCluster.filter(item => !item.adminOnly || isAdmin) as item}
+                <NavItem>
+                    <Dropdown nav inNavbar>
+                        <DropdownToggle nav caret>
+                            <Icon name={item.icon}/> {item.title}
+                        </DropdownToggle>
+                        <DropdownMenu>
+                            {#each clusters as cluster}
+                                <DropdownItem href={item.href + cluster} active={window.location.pathname == item.href + cluster}>
+                                    {cluster}
+                                </DropdownItem>
+                            {/each}
+                        </DropdownMenu>
+                    </Dropdown>
+                </NavItem>
+            {/each}
+        </Nav>
+    </Collapse>
+    <div class="d-flex">
+        <form method="GET" action="/search">
+            <InputGroup>
+                <Input type="text" placeholder={isAdmin ? "Search jobId / username" : "Search jobId"} name="searchId"/>
+                <Button outline type="submit"><Icon name="search"/></Button>
+            </InputGroup>
+        </form>
+        {#if username}
+            <form method="POST" action="/logout">
+                <Button outline color="success" type="submit" style="margin-left: 10px;">
+                    <Icon name="box-arrow-right"/> Logout {username}
+                </Button>
+            </form>
+        {/if}
+        <Button outline on:click={() => window.location.href = '/config'} style="margin-left: 10px;">
+            <Icon name="gear"/>
+        </Button>
+    </div>
+</Navbar>

web/frontend/src/Job.root.svelte

@@ -0,0 +1,224 @@
+<script>
+    import { init, groupByScope, fetchMetricsStore } from './utils.js'
+    import { Row, Col, Card, Spinner, TabContent, TabPane,
+             CardBody, CardHeader, CardTitle, Button, Icon } from 'sveltestrap'
+    import PlotTable from './PlotTable.svelte'
+    import Metric from './Metric.svelte'
+    import PolarPlot from './plots/Polar.svelte'
+    import Roofline from './plots/Roofline.svelte'
+    import JobInfo from './joblist/JobInfo.svelte'
+    import TagManagement from './TagManagement.svelte'
+    import MetricSelection from './MetricSelection.svelte'
+    import Zoom from './Zoom.svelte'
+    import StatsTable from './StatsTable.svelte'
+    import { getContext } from 'svelte'
+
+    export let dbid
+
+    const { query: initq } = init(`
+        job(id: "${dbid}") {
+            id, jobId, user, project, cluster, startTime,
+            duration, numNodes, numHWThreads, numAcc,
+            SMT, exclusive, partition, subCluster, arrayJobId,
+            monitoringStatus, state, walltime,
+            tags { id, type, name },
+            resources { hostname, hwthreads, accelerators },
+            metaData
+            userData { name, email }
+        }
+    `)
+
+    const ccconfig = getContext('cc-config'),
+          clusters = getContext('clusters')
+
+    let isMetricsSelectionOpen = false, selectedMetrics = []
+    const [jobMetrics, startFetching] = fetchMetricsStore()
+    getContext('on-init')(() => {
+        let job = $initq.data.job
+        if (!job)
+            return
+
+        startFetching(job, null, job.numNodes > 2 ? ["node"] : ["node", "core"])
+
+        // TODO: Do not even fetch metrics that are not one of the following: flops_any, mem_bw, job_view_selectedMetrics, job_view_nodestats_selectedMetrics
+        selectedMetrics = ccconfig[`job_view_selectedMetrics:${job.cluster}`]
+            || clusters.find(c => c.name == job.cluster).metricConfig.map(mc => mc.name)
+    })
+
+    let plots = {}, jobTags, fullWidth, statsTable
+    $: polarPlotSize = Math.min(fullWidth / 3 - 10, 300)
+    $: document.title = $initq.fetching ? 'Loading...' : ($initq.error ? 'Error' : `Job ${$initq.data.job.jobId} - ClusterCockpit`)
+
+    let missingMetrics = [], missingHosts = [], somethingMissing = false
+    $: if ($initq.data && $jobMetrics.data) {
+        let job = $initq.data.job,
+            metrics = $jobMetrics.data.jobMetrics,
+            metricNames = clusters.find(c => c.name == job.cluster).metricConfig.map(mc => mc.name)
+
+        missingMetrics = metricNames.filter(metric => !metrics.some(jm => jm.name == metric))
+        missingHosts = job.resources.map(({ hostname }) => ({
+            hostname: hostname,
+            metrics: metricNames.filter(metric => !metrics.some(jm => jm.metric.scope == 'node' && jm.metric.series.some(series => series.hostname == hostname)))
+        })).filter(({ metrics }) => metrics.length > 0)
+        somethingMissing = missingMetrics.length > 0 || missingHosts.length > 0
+    }
+
+    const orderAndMap = (grouped, selectedMetrics) => selectedMetrics.map(metric => ({ metric: metric, data: grouped.find((group) => group[0].name == metric) }))
+</script>
+
+<div class="row" bind:clientWidth={fullWidth}></div>
+<Row>
+    <Col>
+        {#if $initq.error}
+            <Card body color="danger">{$initq.error.message}</Card>
+        {:else if $initq.data}
+            <JobInfo job={$initq.data.job} jobTags={jobTags}/>
+        {:else}
+            <Spinner secondary/>
+        {/if}
+    </Col>
+    {#if $jobMetrics.data && $initq.data}
+        <Col>
+            <PolarPlot
+                width={polarPlotSize} height={polarPlotSize}
+                metrics={ccconfig.job_view_polarPlotMetrics}
+                cluster={$initq.data.job.cluster}
+                jobMetrics={$jobMetrics.data.jobMetrics} />
+        </Col>
+        <Col>
+            <Roofline
+                width={fullWidth / 3 - 10} height={polarPlotSize}
+                cluster={clusters
+                    .find(c => c.name == $initq.data.job.cluster).subClusters
+                    .find(sc => sc.name == $initq.data.job.subCluster)}
+                flopsAny={$jobMetrics.data.jobMetrics.find(m => m.name == 'flops_any' && m.metric.scope == 'node').metric}
+                memBw={$jobMetrics.data.jobMetrics.find(m => m.name == 'mem_bw' && m.metric.scope == 'node').metric} />
+        </Col>
+    {:else}
+        <Col></Col>
+        <Col></Col>
+    {/if}
+</Row>
+<br/>
+<Row>
+    <Col xs="auto">
+        {#if $initq.data}
+            <TagManagement job={$initq.data.job} bind:jobTags={jobTags}/>
+        {/if}
+    </Col>
+    <Col xs="auto">
+        {#if $initq.data}
+            <Button outline
+                on:click={() => (isMetricsSelectionOpen = true)}>
+                <Icon name="graph-up"/> Metrics
+            </Button>
+        {/if}
+    </Col>
+    <Col xs="auto">
+        <Zoom timeseriesPlots={plots} />
+    </Col>
+</Row>
+<br/>
+<Row>
+    <Col>
+        {#if $jobMetrics.error}
+            {#if $initq.data.job.monitoringStatus == 0 || $initq.data.job.monitoringStatus == 2}
+                <Card body color="warning">Not monitored or archiving failed</Card>
+                <br/>
+            {/if}
+            <Card body color="danger">{$jobMetrics.error.message}</Card>
+        {:else if $jobMetrics.fetching}
+            <Spinner secondary/>
+        {:else if $jobMetrics.data && $initq.data}
+            <PlotTable
+                let:item
+                let:width
+                items={orderAndMap(groupByScope($jobMetrics.data.jobMetrics), selectedMetrics)}
+                itemsPerRow={ccconfig.plot_view_plotsPerRow}>
+                {#if item.data}
+                    <Metric
+                        bind:this={plots[item.metric]}
+                        on:more-loaded={({ detail }) => statsTable.moreLoaded(detail)}
+                        job={$initq.data.job}
+                        metric={item.metric}
+                        scopes={item.data.map(x => x.metric)}
+                        width={width}/>
+                {:else}
+                    <Card body color="warning">No data for <code>{item.metric}</code></Card>
+                {/if}
+            </PlotTable>
+        {/if}
+    </Col>
+</Row>
+<br/>
+<Row>
+    <Col>
+        {#if $initq.data}
+        <TabContent>
+            {#if somethingMissing}
+                <TabPane tabId="resources" tab="Resources" active={somethingMissing}>
+                    <div style="margin: 10px;"><Card color="warning">
+                        <CardHeader>
+                            <CardTitle>Missing Metrics/Reseources</CardTitle>
+                        </CardHeader>
+                        <CardBody>
+                            {#if missingMetrics.length > 0}
+                                <p>No data at all is available for the metrics: {missingMetrics.join(', ')}</p>
+                            {/if}
+                            {#if missingHosts.length > 0}
+                                <p>Some metrics are missing for the following hosts:</p>
+                                <ul>
+                                    {#each missingHosts as missing}
+                                        <li>{missing.hostname}: {missing.metrics.join(', ')}</li>
+                                    {/each}
+                                </ul>
+                            {/if}
+                        </CardBody>
+                    </Card></div>
+                </TabPane>
+            {/if}
+            <TabPane tabId="stats" tab="Statistics Table" active={!somethingMissing}>
+                {#if $jobMetrics.data}
+                    <StatsTable bind:this={statsTable} job={$initq.data.job} jobMetrics={$jobMetrics.data.jobMetrics} />
+                {/if}
+            </TabPane>
+            <TabPane tabId="job-script" tab="Job Script">
+                <div class="pre-wrapper">
+                    {#if $initq.data.job.metaData?.jobScript}
+                        <pre><code>{$initq.data.job.metaData?.jobScript}</code></pre>
+                    {:else}
+                        <Card body color="warning">No job script available</Card>
+                    {/if}
+                </div>
+            </TabPane>
+            <TabPane tabId="slurm-info" tab="Slurm Info">
+                <div class="pre-wrapper">
+                    {#if $initq.data.job.metaData?.slurmInfo}
+                        <pre><code>{$initq.data.job.metaData?.slurmInfo}</code></pre>
+                    {:else}
+                        <Card body color="warning">No additional slurm information available</Card>
+                    {/if}
+                </div>
+            </TabPane>
+        </TabContent>
+        {/if}
+    </Col>
+</Row>
+
+{#if $initq.data}
+    <MetricSelection
+        cluster={$initq.data.job.cluster}
+        configName="job_view_selectedMetrics"
+        bind:metrics={selectedMetrics}
+        bind:isOpen={isMetricsSelectionOpen} />
+{/if}
+
+<style>
+    .pre-wrapper {
+        font-size: 1.1rem;
+        margin: 10px;
+        border: 1px solid #bbb;
+        border-radius: 5px;
+        padding: 5px;
+    }
+</style>

web/frontend/src/Jobs.root.svelte

@@ -0,0 +1,88 @@
+<script>
+    import { onMount, getContext } from 'svelte'
+    import { init } from './utils.js'
+    import { Row, Col, Button, Icon, Card, Spinner } from 'sveltestrap'
+    import Filters from './filters/Filters.svelte'
+    import JobList from './joblist/JobList.svelte'
+    import Refresher from './joblist/Refresher.svelte'
+    import Sorting from './joblist/SortSelection.svelte'
+    import MetricSelection from './MetricSelection.svelte'
+    import UserOrProject from './filters/UserOrProject.svelte'
+
+    const { query: initq } = init()
+
+    const ccconfig = getContext('cc-config')
+
+    export let filterPresets = {}
+
+    let filters, jobList, matchedJobs = null
+    let sorting = { field: 'startTime', order: 'DESC' }, isSortingOpen = false, isMetricsSelectionOpen = false
+    let metrics = filterPresets.cluster
+        ? ccconfig[`plot_list_selectedMetrics:${filterPresets.cluster}`] || ccconfig.plot_list_selectedMetrics
+        : ccconfig.plot_list_selectedMetrics
+
+    // The filterPresets are handled by the Filters component,
+    // so we need to wait for it to be ready before we can start a query.
+    // This is also why JobList component starts out with a paused query.
+    onMount(() => filters.update())
+</script>
+
+<Row>
+    {#if $initq.fetching}
+        <Col xs="auto">
+            <Spinner/>
+        </Col>
+    {:else if $initq.error}
+        <Col xs="auto">
+            <Card body color="danger">{$initq.error.message}</Card>
+        </Col>
+    {/if}
+</Row>
+<Row>
+    <Col xs="auto">
+        <Button
+            outline color="primary"
+            on:click={() => (isSortingOpen = true)}>
+            <Icon name="sort-up"/> Sorting
+        </Button>
+        <Button
+            outline color="primary"
+            on:click={() => (isMetricsSelectionOpen = true)}>
+            <Icon name="graph-up"/> Metrics
+        </Button>
+        <Button disabled outline>{matchedJobs == null ? 'Loading...' : `${matchedJobs} jobs`}</Button>
+    </Col>
+    <Col xs="auto">
+        <Filters
+            filterPresets={filterPresets}
+            bind:this={filters}
+            on:update={({ detail }) => jobList.update(detail.filters)} />
+    </Col>
+
+    <Col xs="3" style="margin-left: auto;">
+        <UserOrProject on:update={({ detail }) => filters.update(detail)}/>
+    </Col>
+    <Col xs="2">
+        <Refresher on:reload={() => jobList.update()} />
+    </Col>
+</Row>
+<br/>
+<Row>
+    <Col>
+        <JobList
+            bind:metrics={metrics}
+            bind:sorting={sorting}
+            bind:matchedJobs={matchedJobs}
+            bind:this={jobList} />
+    </Col>
+</Row>
+
+<Sorting
+    bind:sorting={sorting}
+    bind:isOpen={isSortingOpen} />
+
+<MetricSelection
+    cluster={filterPresets.cluster}
+    configName="plot_list_selectedMetrics"
+    bind:metrics={metrics}
+    bind:isOpen={isMetricsSelectionOpen} />

web/frontend/src/List.root.svelte

@@ -0,0 +1,151 @@
+<!-- 
+    @component List of users or projects
+ -->
+<script>
+    import { onMount } from 'svelte'
+    import { init } from './utils.js'
+    import { Row, Col, Button, Icon, Table, Card, Spinner,
+             InputGroup, Input } from 'sveltestrap'
+    import Filters from './filters/Filters.svelte'
+    import { operationStore, query } from '@urql/svelte';
+    import { scramble, scrambleNames } from './joblist/JobInfo.svelte'
+
+    const { } = init()
+
+    export let type
+    export let filterPresets
+
+    console.assert(type == 'USER' || type == 'PROJECT', 'Invalid list type provided!')
+
+    const stats = operationStore(`query($filter: [JobFilter!]!) {
+        rows: jobsStatistics(filter: $filter, groupBy: ${type}) {
+            id
+            totalJobs
+            totalWalltime
+            totalCoreHours
+        }
+    }`, {
+        filter: []
+    }, {
+        pause: true
+    })
+
+    query(stats)
+
+    let filters
+    let nameFilter = ''
+    let sorting = { field: 'totalJobs', direction: 'down' }
+
+    function changeSorting(event, field) {
+        let target = event.target
+        while (target.tagName != 'BUTTON')
+            target = target.parentElement
+
+        let direction = target.children[0].className.includes('up') ? 'down' : 'up'
+        target.children[0].className = `bi-sort-numeric-${direction}`
+        sorting = { field, direction }
+    }
+
+    function sort(stats, sorting, nameFilter) {
+        const cmp = sorting.field == 'id'
+            ? (sorting.direction == 'up'
+                ? (a, b) => a.id < b.id
+                : (a, b) => a.id > b.id)
+            : (sorting.direction == 'up'
+                ? (a, b) => a[sorting.field] - b[sorting.field]
+                : (a, b) => b[sorting.field] - a[sorting.field])
+        
+        return stats.filter(u => u.id.includes(nameFilter)).sort(cmp)
+    }
+
+    onMount(() => filters.update())
+</script>
+
+<Row>
+    <Col xs="auto">
+        <InputGroup>
+            <Button disabled outline>
+                Search {type.toLowerCase()}s
+            </Button>
+            <Input bind:value={nameFilter} placeholder="Filter by {({ USER: 'username', PROJECT: 'project' })[type]}" />
+        </InputGroup>
+    </Col>
+    <Col xs="auto">
+        <Filters
+            bind:this={filters}
+            filterPresets={filterPresets}
+            startTimeQuickSelect={true}
+            menuText="Only {type.toLowerCase()}s with jobs that match the filters will show up"
+            on:update={({ detail }) => {
+                $stats.variables = { filter: detail.filters }
+                $stats.context.pause = false
+                $stats.reexecute()
+            }} />
+    </Col>
+</Row>
+<Table>
+    <thead>
+        <tr>
+            <th scope="col">
+                {({ USER: 'Username', PROJECT: 'Project Name' })[type]}
+                <Button color="{sorting.field == 'id' ? 'primary' : 'light'}"
+                    size="sm" on:click={e => changeSorting(e, 'id')}>
+                    <Icon name="sort-numeric-down" />
+                </Button>
+            </th>
+            <th scope="col">
+                Total Jobs
+                <Button color="{sorting.field == 'totalJobs' ? 'primary' : 'light'}"
+                    size="sm" on:click={e => changeSorting(e, 'totalJobs')}>
+                    <Icon name="sort-numeric-down" />
+                </Button>
+            </th>
+            <th scope="col">
+                Total Walltime
+                <Button color="{sorting.field == 'totalWalltime' ? 'primary' : 'light'}"
+                    size="sm" on:click={e => changeSorting(e, 'totalWalltime')}>
+                    <Icon name="sort-numeric-down" />
+                </Button>
+            </th>
+            <th scope="col">
+                Total Core Hours
+                <Button color="{sorting.field == 'totalCoreHours' ? 'primary' : 'light'}"
+                    size="sm" on:click={e => changeSorting(e, 'totalCoreHours')}>
+                    <Icon name="sort-numeric-down" />
+                </Button>
+            </th>
+        </tr>
+    </thead>
+    <tbody>
+        {#if $stats.fetching}
+            <tr>
+                <td colspan="4" style="text-align: center;"><Spinner secondary/></td>
+            </tr>
+        {:else if $stats.error}
+            <tr>
+                <td colspan="4"><Card body color="danger" class="mb-3">{$stats.error.message}</Card></td>
+            </tr>
+        {:else if $stats.data}
+            {#each sort($stats.data.rows, sorting, nameFilter) as row (row.id)}
+                <tr>
+                    <td>
+                        {#if type == 'USER'}
+                            <a href="/monitoring/user/{row.id}">{scrambleNames ? scramble(row.id) : row.id}</a>
+                        {:else if type == 'PROJECT'}
+                            <a href="/monitoring/jobs/?project={row.id}">{row.id}</a>
+                        {:else}
+                            {row.id}
+                        {/if}
+                    </td>
+                    <td>{row.totalJobs}</td>
+                    <td>{row.totalWalltime}</td>
+                    <td>{row.totalCoreHours}</td>
+                </tr>
+            {:else}
+                <tr>
+                    <td colspan="4"><i>No {type.toLowerCase()}s/jobs found</i></td>
+                </tr>
+            {/each}
+        {/if}
+    </tbody>
+</Table>

web/frontend/src/Metric.svelte

@@ -0,0 +1,88 @@
+<script>
+    import { getContext, createEventDispatcher } from 'svelte'
+    import Timeseries from './plots/MetricPlot.svelte'
+    import { InputGroup, InputGroupText, Spinner, Card } from 'sveltestrap'
+    import { fetchMetrics, minScope } from './utils'
+
+    export let job
+    export let metric
+    export let scopes
+    export let width
+
+    const dispatch = createEventDispatcher()
+    const cluster = getContext('clusters').find(cluster => cluster.name == job.cluster)
+    const subCluster = cluster.subClusters.find(subCluster => subCluster.name == job.subCluster)
+    const metricConfig = cluster.metricConfig.find(metricConfig => metricConfig.name == metric)
+
+    let selectedScope = minScope(scopes.map(s => s.scope)), selectedHost = null, plot, fetching = false, error = null
+
+    $: avaliableScopes = scopes.map(metric => metric.scope)
+    $: data = scopes.find(metric => metric.scope == selectedScope)
+    $: series = data?.series.filter(series => selectedHost == null || series.hostname == selectedHost)
+
+    let from = null, to = null
+    export function setTimeRange(f, t) {
+        from = f, to = t
+    }
+
+    $: if (plot != null) plot.setTimeRange(from, to)
+
+    export async function loadMore() {
+        fetching = true
+        let response = await fetchMetrics(job, [metric], ["core"])
+        fetching = false
+
+        if (response.error) {
+            error = response.error
+            return
+        }
+
+        for (let jm of response.data.jobMetrics) {
+            if (jm.metric.scope != "node") {
+                scopes.push(jm.metric)
+                selectedScope = jm.metric.scope
+                dispatch('more-loaded', jm)
+                if (!avaliableScopes.includes(selectedScope))
+                    avaliableScopes = [...avaliableScopes, selectedScope]
+            }
+        }
+    }
+
+    $: if (selectedScope == "load-more") loadMore()
+</script>
+<InputGroup>
+    <InputGroupText style="min-width: 150px;">
+        {metric} ({metricConfig?.unit})
+    </InputGroupText>
+    <select class="form-select" bind:value={selectedScope}>
+        {#each avaliableScopes as scope}
+            <option value={scope}>{scope}</option>
+        {/each}
+        {#if avaliableScopes.length == 1 && metricConfig?.scope != "node"}
+            <option value={"load-more"}>Load more...</option>
+        {/if}
+    </select>
+    {#if job.resources.length > 1}
+        <select class="form-select" bind:value={selectedHost}>
+            <option value={null}>All Hosts</option>
+            {#each job.resources as { hostname }}
+                <option value={hostname}>{hostname}</option>
+            {/each}
+        </select>
+    {/if}
+</InputGroup>
+{#key series}
+    {#if fetching == true}
+        <Spinner/>
+    {:else if error != null}
+        <Card body color="danger">{error.message}</Card>
+    {:else if series != null}
+        <Timeseries
+            bind:this={plot}
+            width={width} height={300}
+            cluster={cluster} subCluster={subCluster}
+            timestep={data.timestep}
+            scope={selectedScope} metric={metric}
+            series={series} />
+    {/if}
+{/key}

web/frontend/src/MetricSelection.svelte

@@ -0,0 +1,126 @@
+<!-- 
+    @component
+
+    Properties:
+    - metrics:    [String] (changes from inside, needs to be initialised, list of selected metrics)
+    - isOpen:     Boolean  (can change from inside and outside)
+    - configName: String   (constant)
+ -->
+
+<script>
+    import  { Modal, ModalBody, ModalHeader, ModalFooter, Button, ListGroup } from 'sveltestrap'
+    import { getContext } from 'svelte'
+    import { mutation } from '@urql/svelte'
+
+    export let metrics
+    export let isOpen
+    export let configName
+    export let allMetrics = null
+    export let cluster = null
+
+    const clusters = getContext('clusters'),
+          onInit = getContext('on-init')
+
+    let newMetricsOrder = []
+    let unorderedMetrics = [...metrics]
+
+    onInit(() => {
+        if (allMetrics == null) {
+            allMetrics = new Set()
+            for (let c of clusters)
+                if (cluster == null || c.name == cluster)
+                    for (let metric of c.metricConfig)
+                        allMetrics.add(metric.name)
+        }
+
+        newMetricsOrder = [...allMetrics].filter(m => !metrics.includes(m))
+        newMetricsOrder.unshift(...metrics)
+    })
+
+    const updateConfiguration = mutation({
+        query: `mutation($name: String!, $value: String!) {
+            updateConfiguration(name: $name, value: $value)
+        }`
+    })
+
+    let columnHovering = null
+
+    function columnsDragStart(event, i) {
+        event.dataTransfer.effectAllowed = 'move'
+        event.dataTransfer.dropEffect = 'move'
+        event.dataTransfer.setData('text/plain', i)
+    }
+
+    function columnsDrag(event, target) {
+        event.dataTransfer.dropEffect = 'move'
+        const start = Number.parseInt(event.dataTransfer.getData("text/plain"))
+        if (start < target) {
+            newMetricsOrder.splice(target + 1, 0, newMetricsOrder[start])
+            newMetricsOrder.splice(start, 1)
+        } else {
+            newMetricsOrder.splice(target, 0, newMetricsOrder[start])
+            newMetricsOrder.splice(start + 1, 1)
+        }
+        columnHovering = null
+    }
+
+    function closeAndApply() {
+        metrics = newMetricsOrder.filter(m => unorderedMetrics.includes(m))
+        isOpen = false
+
+        updateConfiguration({
+                name: cluster == null ? configName : `${configName}:${cluster}`,
+                value: JSON.stringify(metrics)
+            })
+            .then(res => {
+                if (res.error)
+                    console.error(res.error)
+            })
+    }
+</script>
+
+<style>
+    li.cc-config-column {
+        display: block;
+        cursor: grab;
+    }
+
+    li.cc-config-column.is-active {
+        background-color: #3273dc;
+        color: #fff;
+        cursor: grabbing;
+    }
+</style>
+
+<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
+    <ModalHeader>
+        Configure columns
+    </ModalHeader>
+    <ModalBody>
+        <ListGroup>
+            {#each newMetricsOrder as metric, index (metric)}
+                <li class="cc-config-column list-group-item"
+                    draggable={true} ondragover="return false"
+                    on:dragstart={event => columnsDragStart(event, index)}
+                    on:drop|preventDefault={event => columnsDrag(event, index)}
+                    on:dragenter={() => columnHovering = index}
+                    class:is-active={columnHovering === index}>
+                    {#if unorderedMetrics.includes(metric)}
+                        <input type="checkbox" bind:group={unorderedMetrics} value={metric} checked>
+                    {:else}
+                        <input type="checkbox" bind:group={unorderedMetrics} value={metric}>
+                    {/if}
+                    {metric}
+                    <span style="float: right;">
+                        {cluster == null ? clusters
+                            .filter(cluster => cluster.metricConfig.find(m => m.name == metric) != null)
+                            .map(cluster => cluster.name).join(', ') : ''}
+                    </span>
+                </li>
+            {/each}
+        </ListGroup>
+    </ModalBody>
+    <ModalFooter>
+        <Button color="primary" on:click={closeAndApply}>Close & Apply</Button>
+    </ModalFooter>
+</Modal>

web/frontend/src/Node.root.svelte

@@ -0,0 +1,94 @@
+<script>
+    import { init } from './utils.js'
+    import { Row, Col, InputGroup, InputGroupText, Icon, Spinner, Card } from 'sveltestrap'
+    import { operationStore, query } from '@urql/svelte'
+    import TimeSelection from './filters/TimeSelection.svelte'
+    import PlotTable from './PlotTable.svelte'
+    import MetricPlot from './plots/MetricPlot.svelte'
+    import { getContext } from 'svelte'
+
+    export let cluster
+    export let hostname
+    export let from = null
+    export let to = null
+
+    const { query: initq } = init()
+
+    if (from == null || to == null) {
+        to = new Date(Date.now())
+        from = new Date(to.getTime())
+        from.setMinutes(from.getMinutes() - 30)
+    }
+
+    const ccconfig = getContext('cc-config'), clusters = getContext('clusters')
+
+    const nodesQuery = operationStore(`query($cluster: String!, $nodes: [String!], $from: Time!, $to: Time!) {
+        nodeMetrics(cluster: $cluster, nodes: $nodes, from: $from, to: $to) {
+            host, subCluster
+            metrics {
+                name,
+                metric {
+                    timestep
+                    scope
+                    series {
+                        statistics { min, avg, max }
+                        data
+                    }
+                }
+            }
+        }
+    }`, {
+        cluster: cluster,
+        nodes: [hostname],
+        from: from.toISOString(),
+        to: to.toISOString()
+    })
+
+    $: $nodesQuery.variables = { cluster, nodes: [hostname], from: from.toISOString(), to: to.toISOString() }
+
+    query(nodesQuery)
+
+    $: console.log($nodesQuery?.data?.nodeMetrics[0].metrics)
+</script>
+
+<Row>
+    {#if $initq.error}
+        <Card body color="danger">{$initq.error.message}</Card>
+    {:else if $initq.fetching}
+        <Spinner/>
+    {:else}
+        <Col>
+            <InputGroup>
+                <InputGroupText><Icon name="hdd"/></InputGroupText>
+                <InputGroupText>{hostname} ({cluster})</InputGroupText>
+            </InputGroup>
+        </Col>
+        <Col>
+            <TimeSelection
+                bind:from={from}
+                bind:to={to} />
+        </Col>
+    {/if}
+</Row>
+<br/>
+<Row>
+    <Col>
+        {#if $nodesQuery.error}
+            <Card body color="danger">{$nodesQuery.error.message}</Card>
+        {:else if $nodesQuery.fetching || $initq.fetching}
+            <Spinner/>
+        {:else}
+            <PlotTable
+                let:item
+                let:width
+                itemsPerRow={ccconfig.plot_view_plotsPerRow}
+                items={$nodesQuery.data.nodeMetrics[0].metrics.sort((a, b) => a.name.localeCompare(b.name))}>
+                <h4 style="text-align: center;">{item.name}</h4>
+                <MetricPlot
+                    width={width} height={300} metric={item.name} timestep={item.metric.timestep}
+                    cluster={clusters.find(c => c.name == cluster)} subCluster={$nodesQuery.data.nodeMetrics[0].subCluster}
+                    series={item.metric.series} />
+            </PlotTable>
+        {/if}
+    </Col>
+</Row>

web/frontend/src/PlotSelection.svelte

@@ -0,0 +1,133 @@
+<script>
+    import { Modal, ModalBody, ModalHeader, ModalFooter, InputGroup,
+             Button, ListGroup, ListGroupItem, Icon } from 'sveltestrap'
+    import { mutation } from '@urql/svelte'
+
+    export let availableMetrics
+    export let metricsInHistograms
+    export let metricsInScatterplots
+
+    const updateConfigurationMutation = mutation({
+        query: `mutation($name: String!, $value: String!) {
+            updateConfiguration(name: $name, value: $value)
+        }`
+    })
+
+    let isHistogramConfigOpen = false, isScatterPlotConfigOpen = false
+    let selectedMetric1 = null, selectedMetric2 = null
+
+    function updateConfiguration(data) {
+        updateConfigurationMutation({
+                name: data.name,
+                value: JSON.stringify(data.value)
+            })
+            .then(res => {
+                if (res.error)
+                    console.error(res.error)
+            });
+    }
+</script>
+
+<Button outline
+    on:click={() => (isHistogramConfigOpen = true)}>
+    <Icon name=""/>
+    Select Plots for Histograms
+</Button>
+
+<Button outline
+    on:click={() => (isScatterPlotConfigOpen = true)}>
+    <Icon name=""/>
+    Select Plots in Scatter Plots
+</Button>
+
+<Modal isOpen={isHistogramConfigOpen}
+    toggle={() => (isHistogramConfigOpen = !isHistogramConfigOpen)}>
+    <ModalHeader>
+        Select metrics presented in histograms
+    </ModalHeader>
+    <ModalBody>
+        <ListGroup>
+            {#each availableMetrics as metric (metric)}
+                <ListGroupItem>
+                    <input type="checkbox" bind:group={metricsInHistograms}
+                        value={metric}
+                        on:change={() => updateConfiguration({
+                            name: 'analysis_view_histogramMetrics',
+                            value: metricsInHistograms
+                        })} />
+
+                    {metric}
+                </ListGroupItem>
+            {/each}
+        </ListGroup>
+    </ModalBody>
+    <ModalFooter>
+        <Button color="primary"
+            on:click={() => (isHistogramConfigOpen = false)}>
+            Close
+        </Button>
+    </ModalFooter>
+</Modal>
+
+<Modal isOpen={isScatterPlotConfigOpen}
+    toggle={() => (isScatterPlotConfigOpen = !isScatterPlotConfigOpen)}>
+    <ModalHeader>
+        Select metric pairs presented in scatter plots
+    </ModalHeader>
+    <ModalBody>
+        <ListGroup>
+            {#each metricsInScatterplots as pair}
+                <ListGroupItem>
+                    <b>{pair[0]}</b> / <b>{pair[1]}</b>
+
+                    <Button style="float: right;" outline color="danger"
+                        on:click={() => {
+                            metricsInScatterplots = metricsInScatterplots.filter(p => pair != p)
+                            updateConfiguration({
+                                name: 'analysis_view_scatterPlotMetrics',
+                                value: metricsInScatterplots
+                            });
+                        }}>
+                        <Icon name="x" />
+                    </Button>
+                </ListGroupItem>
+            {/each}
+        </ListGroup>
+
+        <br/>
+
+        <InputGroup>
+            <select bind:value={selectedMetric1} class="form-group form-select">
+                <option value={null}>Choose Metric for X Axis</option>
+                {#each availableMetrics as metric}
+                    <option value={metric}>{metric}</option>
+                {/each}
+            </select>
+            <select bind:value={selectedMetric2} class="form-group form-select">
+                <option value={null}>Choose Metric for Y Axis</option>
+                {#each availableMetrics as metric}
+                    <option value={metric}>{metric}</option>
+                {/each}
+            </select>
+            <Button outline disabled={selectedMetric1 == null || selectedMetric2 == null}
+                on:click={() => {
+                    metricsInScatterplots = [...metricsInScatterplots, [selectedMetric1, selectedMetric2]]
+                    selectedMetric1 = null
+                    selectedMetric2 = null
+                    updateConfiguration({
+                        name: 'analysis_view_scatterPlotMetrics',
+                        value: metricsInScatterplots
+                    })
+                }}>
+                Add Plot
+            </Button>
+        </InputGroup>
+
+    </ModalBody>
+    <ModalFooter>
+        <Button color="primary"
+            on:click={() => (isScatterPlotConfigOpen = false)}>
+            Close
+        </Button>
+    </ModalFooter>
+</Modal>

web/frontend/src/PlotTable.svelte

@@ -0,0 +1,50 @@
+<!-- 
+    @component
+
+    Properties:
+    - itemsPerRow: Number
+    - items:       [Any]
+ -->
+
+<script>
+    export let itemsPerRow
+    export let items
+    export let padding = 10
+
+    let tableWidth = 0
+    const PLACEHOLDER = { magic: 'object' }
+
+    function tile(items, itemsPerRow) {
+        const rows = []
+        for (let ri = 0; ri < items.length; ri += itemsPerRow) {
+            const row = []
+            for (let ci = 0; ci < itemsPerRow; ci += 1) {
+                if (ri + ci < items.length)
+                    row.push(items[ri + ci])
+                else
+                    row.push(PLACEHOLDER)
+            }
+
+            rows.push(row)
+        }
+
+        return rows
+    }
+
+    $: rows = tile(items, itemsPerRow)
+    $: plotWidth = (tableWidth / itemsPerRow) - (padding * itemsPerRow)
+</script>
+
+<table bind:clientWidth={tableWidth} style="width: 100%; table-layout: fixed;">
+    {#each rows as row}
+        <tr>
+            {#each row as item (item)}
+                <td>
+                    {#if item != PLACEHOLDER && plotWidth > 0}
+                        <slot item={item} width={plotWidth}></slot>
+                    {/if}
+                </td>
+            {/each}
+        </tr>
+    {/each}
+</table>

web/frontend/src/StatsTable.svelte

@@ -0,0 +1,122 @@
+<script>
+    import { getContext } from 'svelte'
+    import { Button, Table, InputGroup, InputGroupText, Icon } from 'sveltestrap'
+    import MetricSelection from './MetricSelection.svelte'
+    import StatsTableEntry from './StatsTableEntry.svelte'
+    import { maxScope } from './utils.js'
+
+    export let job
+    export let jobMetrics
+
+    const allMetrics = [...new Set(jobMetrics.map(m => m.name))].sort(),
+          scopesForMetric = (metric) => jobMetrics
+            .filter(jm => jm.name == metric)
+            .map(jm => jm.metric.scope)
+
+    let hosts = job.resources.map(r => r.hostname).sort(),
+        selectedScopes = {},
+        sorting = {},
+        isMetricSelectionOpen = false,
+        selectedMetrics = getContext('cc-config').job_view_nodestats_selectedMetrics
+
+    for (let metric of allMetrics) {
+        selectedScopes[metric] = maxScope(scopesForMetric(metric))
+        sorting[metric] = {
+            min: { dir: 'up', active: false },
+            avg: { dir: 'up', active: false },
+            max: { dir: 'up', active: false },
+        }
+    }
+
+    export function sortBy(metric, stat) {
+        let s = sorting[metric][stat]
+        if (s.active) {
+            s.dir = s.dir == 'up' ? 'down' : 'up'
+        } else {
+            for (let metric in sorting)
+                for (let stat in sorting[metric])
+                    sorting[metric][stat].active = false
+            s.active = true
+        }
+
+        let series = jobMetrics.find(jm => jm.name == metric && jm.metric.scope == 'node')?.metric.series
+        sorting = {...sorting}
+        hosts = hosts.sort((h1, h2) => {
+            let s1 = series.find(s => s.hostname == h1)?.statistics
+            let s2 = series.find(s => s.hostname == h2)?.statistics
+            if (s1 == null || s2 == null)
+                return -1
+
+            return s.dir != 'up' ? s1[stat] - s2[stat] : s2[stat] - s1[stat]         
+        })
+    }
+
+    export function moreLoaded(jobMetric) {
+        jobMetrics = [...jobMetrics, jobMetric]
+    }
+</script>
+
+<Table>
+    <thead>
+        <tr>
+            <th>
+                <Button outline on:click={() => (isMetricSelectionOpen = true, console.log(isMetricSelectionOpen))}>
+                    Metrics
+                </Button>
+            </th>
+            {#each selectedMetrics as metric}
+                <th colspan={selectedScopes[metric] == 'node' ? 3 : 4}>
+                    <InputGroup>
+                        <InputGroupText>
+                            {metric}
+                        </InputGroupText>
+                        <select class="form-select"
+                            bind:value={selectedScopes[metric]}
+                            disabled={scopesForMetric(metric, jobMetrics).length == 1}>
+                            {#each scopesForMetric(metric, jobMetrics) as scope}
+                                <option value={scope}>{scope}</option>
+                            {/each}
+                        </select>
+                    </InputGroup>
+                </th>
+            {/each}
+        </tr>
+        <tr>
+            <th>Node</th>
+            {#each selectedMetrics as metric}
+                {#if selectedScopes[metric] != 'node'}
+                    <th>Id</th>
+                {/if}
+                {#each ['min', 'avg', 'max'] as stat}
+                    <th on:click={() => sortBy(metric, stat)}>
+                        {stat}
+                        {#if selectedScopes[metric] == 'node'}
+                            <Icon name="caret-{sorting[metric][stat].dir}{sorting[metric][stat].active ? '-fill' : ''}" />
+                        {/if}
+                    </th>
+                {/each}
+            {/each}
+        </tr>
+    </thead>
+    <tbody>
+        {#each hosts as host (host)}
+            <tr>
+                <th scope="col">{host}</th>
+                {#each selectedMetrics as metric (metric)}
+                    <StatsTableEntry
+                        host={host} metric={metric}
+                        scope={selectedScopes[metric]}
+                        jobMetrics={jobMetrics} />
+                {/each}
+            </tr>
+        {/each}
+    </tbody>
+</Table>
+
+<br/>
+
+<MetricSelection
+    configName='job_view_nodestats_selectedMetrics'
+    allMetrics={allMetrics}
+    bind:metrics={selectedMetrics}
+    bind:isOpen={isMetricSelectionOpen} />

web/frontend/src/StatsTableEntry.svelte

@@ -0,0 +1,37 @@
+<script>
+    export let host
+    export let metric
+    export let scope
+    export let jobMetrics
+
+    $: series = jobMetrics
+        .find(jm => jm.name == metric && jm.metric.scope == scope)
+        ?.metric.series.filter(s => s.hostname == host && s.statistics != null)
+</script>
+
+{#if series == null || series.length == 0}
+    <td colspan={scope == 'node' ? 3 : 4}><i>No data</i></td>    
+{:else if series.length == 1 && scope == 'node'}
+    <td>
+        {series[0].statistics.min}
+    </td>
+    <td>
+        {series[0].statistics.avg}
+    </td>
+    <td>
+        {series[0].statistics.max}
+    </td>
+{:else}
+    <td colspan="4">
+        <table style="width: 100%;">
+            {#each series as s, i}
+                <tr>
+                    <th>{s.id ?? i}</th>
+                    <td>{s.statistics.min}</td>
+                    <td>{s.statistics.avg}</td>
+                    <td>{s.statistics.max}</td>
+                </tr>
+            {/each}
+        </table>
+    </td>
+{/if}

web/frontend/src/Status.root.svelte

@@ -0,0 +1,184 @@
+<script>
+    import Refresher from './joblist/Refresher.svelte'
+    import Roofline, { transformPerNodeData } from './plots/Roofline.svelte'
+    import Histogram from './plots/Histogram.svelte'
+    import { Row, Col, Spinner, Card, Table, Progress } from 'sveltestrap'
+    import { init } from './utils.js'
+    import { operationStore, query } from '@urql/svelte'
+
+    const { query: initq } = init()
+
+    export let cluster
+
+    let plotWidths = [], colWidth1 = 0, colWidth2
+
+    let from = new Date(Date.now() - 5 * 60 * 1000), to = new Date(Date.now())
+    const mainQuery = operationStore(`query($cluster: String!, $filter: [JobFilter!]!, $metrics: [String!], $from: Time!, $to: Time!) {
+        nodeMetrics(cluster: $cluster, metrics: $metrics, from: $from, to: $to) {
+            host,
+            subCluster,
+            metrics {
+                name,
+                metric {
+                    scope
+                    timestep,
+                    series { data }
+                }
+            }
+        }
+
+        stats: jobsStatistics(filter: $filter) {
+            histDuration { count, value }
+            histNumNodes { count, value }
+        }
+
+        allocatedNodes(cluster: $cluster)                                                        { name, count }
+        topUsers:    jobsCount(filter: $filter, groupBy: USER,    weight: NODE_COUNT, limit: 10) { name, count }
+        topProjects: jobsCount(filter: $filter, groupBy: PROJECT, weight: NODE_COUNT, limit: 10) { name, count }
+    }`, {
+        cluster: cluster,
+        metrics: ['flops_any', 'mem_bw'],
+        from: from.toISOString(),
+        to: to.toISOString(),
+        filter: [{ state: ['running'] }, { cluster: { eq: cluster } }]
+    })
+
+    const sumUp = (data, subcluster, metric) => data.reduce((sum, node) => node.subCluster == subcluster
+        ? sum + (node.metrics.find(m => m.name == metric)?.metric.series.reduce((sum, series) => sum + series.data[series.data.length - 1], 0) || 0)
+        : sum, 0)
+
+    let allocatedNodes = {}, flopRate = {}, memBwRate = {}
+    $: if ($initq.data && $mainQuery.data) {
+        let subClusters = $initq.data.clusters.find(c => c.name == cluster).subClusters
+        for (let subCluster of subClusters) {
+            allocatedNodes[subCluster.name] = $mainQuery.data.allocatedNodes.find(({ name }) => name == subCluster.name)?.count || 0
+            flopRate[subCluster.name] = Math.floor(sumUp($mainQuery.data.nodeMetrics, subCluster.name, 'flops_any') * 100) / 100
+            memBwRate[subCluster.name] = Math.floor(sumUp($mainQuery.data.nodeMetrics, subCluster.name, 'mem_bw') * 100) / 100
+        }
+    }
+
+    query(mainQuery)
+</script>
+
+<Row>
+    <Col xs="auto">
+        {#if $initq.fetching || $mainQuery.fetching}
+            <Spinner/>
+        {:else if $initq.error}
+            <Card body color="danger">{$initq.error.message}</Card>
+        {:else}
+            <!-- ... -->
+        {/if}
+    </Col>
+    <Col xs="auto" style="margin-left: auto;">
+        <Refresher initially={120} on:reload={() => {
+            console.log('reload...')
+
+            from = new Date(Date.now() - 5 * 60 * 1000)
+            to = new Date(Date.now())
+
+            $mainQuery.variables = { ...$mainQuery.variables, from: from, to: to }
+            $mainQuery.reexecute({ requestPolicy: 'network-only' })
+        }} />
+    </Col>
+</Row>
+{#if $mainQuery.error}
+    <Row>
+        <Col>
+            <Card body color="danger">{$mainQuery.error.message}</Card>
+        </Col>
+    </Row>
+{/if}
+{#if $initq.data && $mainQuery.data}
+    {#each $initq.data.clusters.find(c => c.name == cluster).subClusters as subCluster, i}
+        <Row>
+            <Col xs="3">
+                <Table>
+                    <tr>
+                        <th scope="col">SubCluster</th>
+                        <td colspan="2">{subCluster.name}</td>
+                    </tr>
+                    <tr>
+                        <th scope="col">Allocated Nodes</th>
+                        <td style="min-width: 75px;"><div class="col"><Progress value={allocatedNodes[subCluster.name]} max={subCluster.numberOfNodes}/></div></td>
+                        <td>({allocatedNodes[subCluster.name]} / {subCluster.numberOfNodes})</td>
+                    </tr>
+                    <tr>
+                        <th scope="col">Flop Rate</th>
+                        <td style="min-width: 75px;"><div class="col"><Progress value={flopRate[subCluster.name]} max={subCluster.flopRateSimd * subCluster.numberOfNodes}/></div></td>
+                        <td>({flopRate[subCluster.name]} / {subCluster.flopRateSimd * subCluster.numberOfNodes})</td>
+                    </tr>
+                    <tr>
+                        <th scope="col">MemBw Rate</th>
+                        <td style="min-width: 75px;"><div class="col"><Progress value={memBwRate[subCluster.name]} max={subCluster.memoryBandwidth * subCluster.numberOfNodes}/></div></td>
+                        <td>({memBwRate[subCluster.name]} / {subCluster.memoryBandwidth * subCluster.numberOfNodes})</td>
+                    </tr>
+                </Table>
+            </Col>
+            <div class="col-9" bind:clientWidth={plotWidths[i]}>
+                {#key $mainQuery.data.nodeMetrics}
+                    <Roofline
+                        width={plotWidths[i] - 10} height={300} colorDots={false} cluster={subCluster}
+                        data={transformPerNodeData($mainQuery.data.nodeMetrics.filter(data => data.subCluster == subCluster.name))} />
+                {/key}
+            </div>
+        </Row>
+    {/each}
+    <Row>
+        <div class="col-4" bind:clientWidth={colWidth1}>
+            <h4>Top Users</h4>
+            {#key $mainQuery.data}
+                <Histogram
+                    width={colWidth1 - 25} height={300}
+                    data={$mainQuery.data.topUsers.sort((a, b) => b.count - a.count).map(({ count }, idx) => ({ count, value: idx }))}
+                    label={(x) => x < $mainQuery.data.topUsers.length ? $mainQuery.data.topUsers[Math.floor(x)].name : '0'} />
+            {/key}
+        </div>
+        <div class="col-2">
+            <Table>
+                <tr><th>Name</th><th>Number of Nodes</th></tr>
+                {#each $mainQuery.data.topUsers.sort((a, b) => b.count - a.count) as { name, count }}
+                    <tr>
+                        <th scope="col"><a href="/monitoring/user/{name}">{name}</a></th>
+                        <td>{count}</td>
+                    </tr>
+                {/each}
+            </Table>
+        </div>
+        <div class="col-4">
+            <h4>Top Projects</h4>
+            {#key $mainQuery.data}
+                <Histogram
+                    width={colWidth1 - 25} height={300}
+                    data={$mainQuery.data.topProjects.sort((a, b) => b.count - a.count).map(({ count }, idx) => ({ count, value: idx }))}
+                    label={(x) => x < $mainQuery.data.topProjects.length ? $mainQuery.data.topProjects[Math.floor(x)].name : '0'} />
+            {/key}
+        </div>
+        <div class="col-2">
+            <Table>
+                <tr><th>Name</th><th>Number of Nodes</th></tr>
+                {#each $mainQuery.data.topProjects.sort((a, b) => b.count - a.count) as { name, count }}
+                    <tr><th scope="col">{name}</th><td>{count}</td></tr>
+                {/each}
+            </Table>
+        </div>
+    </Row>
+    <Row>
+        <div class="col" bind:clientWidth={colWidth2}>
+            <h4>Duration Distribution</h4>
+            {#key $mainQuery.data.stats}
+                <Histogram
+                    width={colWidth2 - 25} height={300}
+                    data={$mainQuery.data.stats[0].histDuration} />
+            {/key}
+        </div>
+        <div class="col">
+            <h4>Number of Nodes Distribution</h4>
+            {#key $mainQuery.data.stats}
+                <Histogram
+                    width={colWidth2 - 25} height={300}
+                    data={$mainQuery.data.stats[0].histNumNodes} />
+            {/key}
+        </div>
+    </Row>
+{/if}

web/frontend/src/Systems.root.svelte

@@ -0,0 +1,118 @@
+<script>
+    import { init } from './utils.js'
+    import { Row, Col, Input, InputGroup, InputGroupText, Icon, Spinner, Card } from 'sveltestrap'
+    import { operationStore, query } from '@urql/svelte'
+    import TimeSelection from './filters/TimeSelection.svelte'
+    import PlotTable from './PlotTable.svelte'
+    import MetricPlot from './plots/MetricPlot.svelte'
+    import { getContext } from 'svelte'
+
+    export let cluster
+    export let from = null
+    export let to = null
+
+    const { query: initq } = init()
+
+    if (from == null || to == null) {
+        to = new Date(Date.now())
+        from = new Date(to.getTime())
+        from.setMinutes(from.getMinutes() - 30)
+    }
+
+    const clusters = getContext('clusters')
+    const ccconfig = getContext('cc-config')
+
+    let plotHeight = 300
+    let hostnameFilter = ''
+    let selectedMetric = ccconfig.system_view_selectedMetric
+
+    const nodesQuery = operationStore(`query($cluster: String!, $metrics: [String!], $from: Time!, $to: Time!) {
+        nodeMetrics(cluster: $cluster, metrics: $metrics, from: $from, to: $to) {
+            host,
+            subCluster
+            metrics {
+                name,
+                metric {
+                    scope
+                    timestep,
+                    series {
+                        statistics { min, avg, max }
+                        data
+                    }
+                }
+            }
+        }
+    }`, {
+        cluster: cluster,
+        metrics: [],
+        from: from.toISOString(),
+        to: to.toISOString()
+    })
+
+    $: $nodesQuery.variables = { cluster, metrics: [selectedMetric], from: from.toISOString(), to: to.toISOString() }
+
+    query(nodesQuery)
+</script>
+
+<Row>
+    {#if $initq.error}
+        <Card body color="danger">{$initq.error.message}</Card>
+    {:else if $initq.fetching}
+        <Spinner/>
+    {:else}
+        <Col>
+            <TimeSelection
+                bind:from={from}
+                bind:to={to} />
+        </Col>
+        <Col>
+            <InputGroup>
+                <InputGroupText><Icon name="graph-up" /></InputGroupText>
+                <InputGroupText>Metric</InputGroupText>
+                <select class="form-select" bind:value={selectedMetric}>
+                    {#each clusters.find(c => c.name == cluster).metricConfig as metric}
+                        <option value={metric.name}>{metric.name} ({metric.unit})</option>
+                    {/each}
+                </select>
+            </InputGroup>
+        </Col>
+        <Col>
+            <InputGroup>
+                <InputGroupText><Icon name="hdd" /></InputGroupText>
+                <InputGroupText>Find Node</InputGroupText>
+                <Input placeholder="hostname..." type="text" bind:value={hostnameFilter} />
+            </InputGroup>
+        </Col>
+    {/if}
+</Row>
+<br/>
+<Row>
+    <Col>
+        {#if $nodesQuery.error}
+            <Card body color="danger">{$nodesQuery.error.message}</Card>
+        {:else if $nodesQuery.fetching || $initq.fetching}
+            <Spinner/>
+        {:else}
+            <PlotTable
+                let:item
+                let:width
+                itemsPerRow={ccconfig.plot_view_plotsPerRow}
+                items={$nodesQuery.data.nodeMetrics
+                    .filter(h => h.host.includes(hostnameFilter) && h.metrics.some(m => m.name == selectedMetric && m.metric.scope == 'node'))
+                    .map(h => ({ host: h.host, subCluster: h.subCluster, data: h.metrics.find(m => m.name == selectedMetric && m.metric.scope == 'node') }))
+                    .sort((a, b) => a.host.localeCompare(b.host))}>
+
+                <h4 style="width: 100%; text-align: center;"><a href="/monitoring/node/{cluster}/{item.host}">{item.host} ({item.subCluster})</a></h4>
+                <MetricPlot
+                    width={width}
+                    height={plotHeight}
+                    timestep={item.data.metric.timestep}
+                    series={item.data.metric.series}
+                    metric={item.data.name}
+                    cluster={clusters.find(c => c.name == cluster)}
+                    subCluster={item.subCluster} />
+            </PlotTable>
+        {/if}
+    </Col>
+</Row>
+

web/frontend/src/Tag.svelte

@@ -0,0 +1,44 @@
+<!-- 
+    @component
+
+    Properties:
+    - id: ID! (if the tag-id is known but not the tag type/name, this can be used)
+    - tag: { id: ID!, type: String, name: String }
+    - clickable: Boolean (default is true)
+ -->
+
+<script>
+    import { getContext } from 'svelte'
+    const allTags = getContext('tags'),
+          initialized = getContext('initialized')
+
+    export let id = null
+    export let tag = null
+    export let clickable = true
+
+    if (tag != null && id == null)
+        id = tag.id
+
+    $: {
+        if ($initialized && tag == null)
+            tag = allTags.find(tag => tag.id == id)
+    }
+</script>
+
+<style>
+    a {
+        margin-left: 0.5rem;
+        line-height: 2;
+    }
+    span {
+        font-size: 0.9rem;
+    }
+</style>
+
+<a target={clickable ? "_blank" : null} href={clickable ? `/monitoring/jobs/?tag=${id}` : null}>
+    {#if tag}
+        <span class="badge bg-warning text-dark">{tag.type}: {tag.name}</span>
+    {:else}
+        Loading...
+    {/if}
+</a>

web/frontend/src/TagManagement.svelte

@@ -0,0 +1,173 @@
+<script>
+    import { getContext } from 'svelte'
+    import { mutation } from '@urql/svelte'
+    import { Icon, Button, ListGroupItem, Spinner, Modal, Input,
+             ModalBody, ModalHeader, ModalFooter, Alert } from 'sveltestrap'
+    import { fuzzySearchTags } from './utils.js'
+    import Tag from './Tag.svelte'
+
+    export let job
+    export let jobTags = job.tags
+
+    let allTags = getContext('tags'), initialized = getContext('initialized')
+    let newTagType = '', newTagName = ''
+    let filterTerm = ''
+    let pendingChange = false
+    let isOpen = false
+
+    const createTagMutation = mutation({
+        query: `mutation($type: String!, $name: String!) {
+            createTag(type: $type, name: $name) { id, type, name }
+        }`
+    })
+
+    const addTagsToJobMutation = mutation({
+        query: `mutation($job: ID!, $tagIds: [ID!]!) {
+            addTagsToJob(job: $job, tagIds: $tagIds) { id, type, name }
+        }`
+    })
+
+    const removeTagsFromJobMutation = mutation({
+        query: `mutation($job: ID!, $tagIds: [ID!]!) {
+            removeTagsFromJob(job: $job, tagIds: $tagIds) { id, type, name }
+        }`
+    })
+
+    let allTagsFiltered // $initialized is in there because when it becomes true, allTags is initailzed.
+    $: allTagsFiltered = ($initialized, fuzzySearchTags(filterTerm, allTags))
+
+    $: {
+        newTagType = '';
+        newTagName = '';
+        let parts = filterTerm.split(':').map(s => s.trim())
+        if (parts.length == 2 && parts.every(s => s.length > 0)) {
+            newTagType = parts[0]
+            newTagName = parts[1]
+        }
+    }
+
+    function isNewTag(type, name) {
+        for (let tag of allTagsFiltered)
+            if (tag.type == type && tag.name == name)
+                return false
+        return true
+    }
+
+    function createTag(type, name) {
+        pendingChange = true
+        return createTagMutation({ type: type, name: name })
+            .then(res => {
+                if (res.error)
+                    throw res.error
+
+                pendingChange = false
+                allTags = [...allTags, res.data.createTag]
+                newTagType = ''
+                newTagName = ''
+                return res.data.createTag
+            }, err => console.error(err))
+    }
+
+    function addTagToJob(tag) {
+        pendingChange = tag.id
+        addTagsToJobMutation({ job: job.id, tagIds: [tag.id] })
+            .then(res => {
+                if (res.error)
+                    throw res.error
+
+                jobTags = job.tags = res.data.addTagsToJob;
+                pendingChange = false;
+            })
+            .catch(err => console.error(err))
+    }
+
+    function removeTagFromJob(tag) {
+        pendingChange = tag.id
+        removeTagsFromJobMutation({ job: job.id, tagIds: [tag.id] })
+            .then(res => {
+                if (res.error)
+                    throw res.error
+
+                jobTags = job.tags = res.data.removeTagsFromJob
+                pendingChange = false
+            })
+            .catch(err => console.error(err))
+    }
+</script>
+
+<style>
+    ul.list-group {
+        max-height: 450px;
+        margin-bottom: 10px;
+        overflow: scroll;
+    }
+</style>
+
+<Modal {isOpen} toggle={() => (isOpen = !isOpen)}>
+    <ModalHeader>
+        Manage Tags
+        {#if pendingChange !== false}
+            <Spinner size="sm" secondary />
+        {:else}
+            <Icon name="tags" />
+        {/if}
+    </ModalHeader>
+    <ModalBody>
+        <Input style="width: 100%;"
+            type="text" placeholder="Search Tags"
+            bind:value={filterTerm} />
+
+        <br/>
+
+        <Alert color="info">
+            Search using "<code>type: name</code>". If no tag matches your search,
+            a button for creating a new one will appear.
+        </Alert>
+
+        <ul class="list-group">
+            {#each allTagsFiltered as tag}
+                <ListGroupItem>
+                    <Tag tag={tag}/>
+
+                    <span style="float: right;">
+                        {#if pendingChange === tag.id}
+                            <Spinner size="sm" secondary />
+                        {:else if job.tags.find(t => t.id == tag.id)}
+                            <Button size="sm" outline color="danger"
+                                on:click={() => removeTagFromJob(tag)}>
+                                <Icon name="x" />
+                            </Button>
+                        {:else}
+                            <Button size="sm" outline color="success"
+                                on:click={() => addTagToJob(tag)}>
+                                <Icon name="plus" />
+                            </Button>
+                        {/if}
+                    </span>
+                </ListGroupItem>
+            {:else}
+                <ListGroupItem disabled>
+                    <i>No tags matching</i>
+                </ListGroupItem>
+            {/each}
+        </ul>
+        <br/>
+        {#if newTagType && newTagName && isNewTag(newTagType, newTagName)}
+            <Button outline color="success"
+                on:click={e => (e.preventDefault(), createTag(newTagType, newTagName))
+                    .then(tag => addTagToJob(tag))}>
+                Create & Add Tag:
+                <Tag tag={({ type: newTagType, name: newTagName })} clickable={false}/>
+            </Button>
+        {:else if allTagsFiltered.length == 0}
+            <Alert>Search Term is not a valid Tag (<code>type: name</code>)</Alert>
+        {/if}
+    </ModalBody>
+    <ModalFooter>
+        <Button color="primary" on:click={() => (isOpen = false)}>Close</Button>
+    </ModalFooter>
+</Modal>
+
+<Button outline on:click={() => (isOpen = true)}>
+    Manage Tags <Icon name="tags" />
+</Button>

web/frontend/src/User.root.svelte

@@ -0,0 +1,172 @@
+<script>
+    import { onMount, getContext } from 'svelte'
+    import { init } from './utils.js'
+    import { Table, Row, Col, Button, Icon, Card, Spinner } from 'sveltestrap'
+    import { operationStore, query } from '@urql/svelte'
+    import Filters from './filters/Filters.svelte'
+    import JobList from './joblist/JobList.svelte'
+    import Sorting from './joblist/SortSelection.svelte'
+    import Refresher from './joblist/Refresher.svelte'
+    import Histogram from './plots/Histogram.svelte'
+    import MetricSelection from './MetricSelection.svelte'
+    import { scramble, scrambleNames } from './joblist/JobInfo.svelte'
+
+    const { query: initq } = init()
+
+    const ccconfig = getContext('cc-config')
+
+    export let user
+    export let filterPresets
+
+    let filters, jobList
+    let sorting = { field: 'startTime', order: 'DESC' }, isSortingOpen = false
+    let metrics = ccconfig.plot_list_selectedMetrics, isMetricsSelectionOpen = false
+    let w1, w2, histogramHeight = 250
+
+    const stats = operationStore(`
+    query($filter: [JobFilter!]!) {
+        jobsStatistics(filter: $filter) {
+            totalJobs
+            shortJobs
+            totalWalltime
+            totalCoreHours
+            histDuration { count, value }
+            histNumNodes { count, value }
+        }
+    }
+    `, {
+        filter: []
+    }, {
+        pause: true
+    })
+
+    query(stats)
+
+    onMount(() => filters.update())
+</script>
+
+<Row>
+    {#if $initq.fetching}
+        <Col>
+            <Spinner/>
+        </Col>
+    {:else if $initq.error}
+        <Col xs="auto">
+            <Card body color="danger">{$initq.error.message}</Card>
+        </Col>
+    {/if}
+
+    <Col xs="auto">
+        <Button
+            outline color="primary"
+            on:click={() => (isSortingOpen = true)}>
+            <Icon name="sort-up"/> Sorting
+        </Button>
+
+        <Button
+            outline color="primary"
+            on:click={() => (isMetricsSelectionOpen = true)}>
+            <Icon name="graph-up"/> Metrics
+        </Button>
+    </Col>
+    <Col xs="auto">
+        <Filters
+            filterPresets={filterPresets}
+            startTimeQuickSelect={true}
+            bind:this={filters}
+            on:update={({ detail }) => {
+                let filters = [...detail.filters, { user: { eq: user.username } }]
+                $stats.variables = { filter: filters }
+                $stats.context.pause = false
+                $stats.reexecute()
+                jobList.update(filters)
+            }} />
+    </Col>
+    <Col xs="auto" style="margin-left: auto;">
+        <Refresher on:reload={() => jobList.update()} />
+    </Col>
+</Row>
+<br/>
+<Row>
+    {#if $stats.error}
+        <Col>
+            <Card body color="danger">{$stats.error.message}</Card>
+        </Col>
+    {:else if !$stats.data}
+        <Col>
+            <Spinner secondary />
+        </Col>
+    {:else}
+        <Col xs="4">
+            <Table>
+                <tbody>
+                    <tr>
+                        <th scope="row">Username</th>
+                        <td>{scrambleNames ? scramble(user.username) : user.username}</td>
+                    </tr>
+                    {#if user.name}
+                        <tr>
+                            <th scope="row">Name</th>
+                            <td>{scrambleNames ? scramble(user.name) : user.name}</td>
+                        </tr>
+                    {/if}
+                    {#if user.email}
+                        <tr>
+                            <th scope="row">Email</th>
+                            <td>{user.email}</td>
+                        </tr>
+                    {/if}
+                    <tr>
+                        <th scope="row">Total Jobs</th>
+                        <td>{$stats.data.jobsStatistics[0].totalJobs}</td>
+                    </tr>
+                    <tr>
+                        <th scope="row">Short Jobs</th>
+                        <td>{$stats.data.jobsStatistics[0].shortJobs}</td>
+                    </tr>
+                    <tr>
+                        <th scope="row">Total Walltime</th>
+                        <td>{$stats.data.jobsStatistics[0].totalWalltime}</td>
+                    </tr>
+                    <tr>
+                        <th scope="row">Total Core Hours</th>
+                        <td>{$stats.data.jobsStatistics[0].totalCoreHours}</td>
+                    </tr>
+                </tbody>
+            </Table>
+        </Col>
+        <div class="col-4" style="text-align: center;" bind:clientWidth={w1}>
+            <b>Walltime</b>
+            {#key $stats.data.jobsStatistics[0].histDuration}
+                <Histogram
+                    data={$stats.data.jobsStatistics[0].histDuration}
+                    width={w1 - 25} height={histogramHeight} />
+            {/key}
+        </div>
+        <div class="col-4" style="text-align: center;" bind:clientWidth={w2}>
+            <b>Number of Nodes</b>
+            {#key $stats.data.jobsStatistics[0].histNumNodes}
+                <Histogram
+                    data={$stats.data.jobsStatistics[0].histNumNodes}
+                    width={w2 - 25} height={histogramHeight} />
+            {/key}
+        </div>
+    {/if}
+</Row>
+<br/>
+<Row>
+    <Col>
+        <JobList
+            bind:metrics={metrics}
+            bind:sorting={sorting}
+            bind:this={jobList} />
+    </Col>
+</Row>
+
+<Sorting
+    bind:sorting={sorting}
+    bind:isOpen={isSortingOpen} />
+
+<MetricSelection configName="plot_list_selectedMetrics"
+    bind:metrics={metrics}
+    bind:isOpen={isMetricsSelectionOpen} />

web/frontend/src/Zoom.svelte

@@ -0,0 +1,60 @@
+<script>
+    import { Icon, InputGroup, InputGroupText } from 'sveltestrap';
+
+    export let timeseriesPlots;
+
+    let windowSize = 100; // Goes from 0 to 100
+    let windowPosition = 50; // Goes from 0 to 100
+
+    function updatePlots() {
+        let ws = windowSize / (100 * 2),
+            wp = windowPosition / 100;
+        let from = (wp - ws),
+            to = (wp + ws);
+        Object
+            .values(timeseriesPlots)
+            .forEach(plot => plot.setTimeRange(from, to));
+    }
+
+    // Rendering a big job can take a long time, so we
+    // throttle the rerenders to every 100ms here.
+    let timeoutId = null;
+    function requestUpdatePlots() {
+        if (timeoutId != null)
+            window.cancelAnimationFrame(timeoutId);
+
+        timeoutId = window.requestAnimationFrame(() => {
+            updatePlots();
+            timeoutId = null;
+        }, 100);
+    }
+
+    $: requestUpdatePlots(windowSize, windowPosition);
+</script>
+
+<div>
+    <InputGroup>
+        <InputGroupText>
+            <Icon name="zoom-in"/>
+        </InputGroupText>
+        <InputGroupText>
+            Window Size:
+            <input
+                style="margin: 0em 0em 0em 1em"
+                type="range"
+                bind:value={windowSize}
+                min=1 max=100 step=1 />
+            <span style="width: 5em;">
+                ({windowSize}%)
+            </span>
+        </InputGroupText>
+        <InputGroupText>
+            Window Position:
+            <input
+                style="margin: 0em 0em 0em 1em"
+                type="range"
+                bind:value={windowPosition}
+                min=0 max=100 step=1 />
+        </InputGroupText>
+    </InputGroup>
+</div>

web/frontend/src/analysis.entrypoint.js

@@ -0,0 +1,14 @@
+import {} from './header.entrypoint.js'
+import Analysis from './Analysis.root.svelte'
+
+filterPresets.cluster = cluster
+
+new Analysis({
+    target: document.getElementById('svelte-app'),
+    props: {
+        filterPresets: filterPresets
+    },
+    context: new Map([
+            ['cc-config', clusterCockpitConfig]
+    ])
+})

web/frontend/src/cache-exchange.js

@@ -0,0 +1,72 @@
+import { filter, map, merge, pipe, share, tap } from 'wonka';
+
+/*
+ * Alternative to the default cacheExchange from urql (A GraphQL client).
+ * Mutations do not invalidate cached results, so in that regard, this
+ * implementation is inferior to the default one. Most people should probably
+ * use the standard cacheExchange and @urql/exchange-request-policy. This cache
+ * also ignores the 'network-and-cache' request policy.
+ *
+ * Options:
+ *    ttl: How long queries are allowed to be cached (in milliseconds)
+ *    maxSize: Max number of results cached. The oldest queries are removed first.
+ */
+export const expiringCacheExchange = ({ ttl, maxSize }) => ({ forward }) => {
+    const cache = new Map();
+    const isCached = (operation) => {
+        if (operation.kind !== 'query' || operation.context.requestPolicy === 'network-only')
+            return false;
+
+        if (!cache.has(operation.key))
+            return false;
+
+        let cacheEntry = cache.get(operation.key);
+        return Date.now() < cacheEntry.expiresAt;
+    };
+
+    return operations => {
+        let shared = share(operations);
+        return merge([
+            pipe(
+                shared,
+                filter(operation => isCached(operation)),
+                map(operation => cache.get(operation.key).response)
+            ),
+            pipe(
+                shared,
+                filter(operation => !isCached(operation)),
+                forward,
+                tap(response => {
+                    if (!response.operation || response.operation.kind !== 'query')
+                        return;
+
+                    if (!response.data)
+                        return;
+
+                    let now = Date.now();
+                    for (let cacheEntry of cache.values()) {
+                        if (cacheEntry.expiresAt < now) {
+                            cache.delete(cacheEntry.response.operation.key);
+                        }
+                    }
+
+                    if (cache.size > maxSize) {
+                        let n = cache.size - maxSize + 1;
+                        for (let key of cache.keys()) {
+                            if (n-- == 0)
+                                break;
+
+                            cache.delete(key);
+                        }
+                    }
+
+                    cache.set(response.operation.key, {
+                        expiresAt: now + ttl,
+                        response: response
+                    });
+                })
+            )
+        ]);
+    };
+};
+

web/frontend/src/filters/Cluster.svelte

@@ -0,0 +1,77 @@
+<script>
+    import { createEventDispatcher, getContext } from 'svelte'
+    import { Button, ListGroup, ListGroupItem,
+             Modal, ModalBody, ModalHeader, ModalFooter } from 'sveltestrap'
+
+    const clusters = getContext('clusters'),
+          initialized = getContext('initialized'),
+          dispatch = createEventDispatcher()
+
+    export let disableClusterSelection = false
+    export let isModified = false
+    export let isOpen = false
+    export let cluster = null
+    export let partition = null
+    let pendingCluster = cluster, pendingPartition = partition
+    $: isModified = pendingCluster != cluster || pendingPartition != partition
+</script>
+
+<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
+    <ModalHeader>
+        Select Cluster & Slurm Partition
+    </ModalHeader>
+    <ModalBody>
+        {#if $initialized}
+            <h4>Cluster</h4>
+            <ListGroup>
+                <ListGroupItem
+                    disabled={disableClusterSelection}
+                    active={pendingCluster == null}
+                    on:click={() => (pendingCluster = null, pendingPartition = null)}>
+                    Any Cluster
+                </ListGroupItem>
+                {#each clusters as cluster}
+                    <ListGroupItem
+                        disabled={disableClusterSelection}
+                        active={pendingCluster == cluster.name}
+                        on:click={() => (pendingCluster = cluster.name, pendingPartition = null)}>
+                        {cluster.name}
+                    </ListGroupItem>
+                {/each}
+            </ListGroup>        
+        {/if}
+        {#if $initialized && pendingCluster != null}
+            <br/>
+            <h4>Partiton</h4>
+            <ListGroup>
+                <ListGroupItem
+                    active={pendingPartition == null}
+                    on:click={() => (pendingPartition = null)}>
+                    Any Partition
+                </ListGroupItem>
+                {#each clusters.find(c => c.name == pendingCluster).partitions as partition}
+                    <ListGroupItem
+                        active={pendingPartition == partition}
+                        on:click={() => (pendingPartition = partition)}>
+                        {partition}
+                    </ListGroupItem>
+                {/each}
+            </ListGroup>
+        {/if}
+    </ModalBody>
+    <ModalFooter>
+        <Button color="primary" on:click={() => {
+            isOpen = false
+            cluster = pendingCluster
+            partition = pendingPartition
+            dispatch('update', { cluster, partition })
+        }}>Close & Apply</Button>
+        <Button color="danger" on:click={() => {
+            isOpen = false
+            cluster = pendingCluster = null
+            partition = pendingPartition = null
+            dispatch('update', { cluster, partition })
+        }}>Reset</Button>
+        <Button on:click={() => (isOpen = false)}>Close</Button>
+    </ModalFooter>
+</Modal>

web/frontend/src/filters/DoubleRangeSlider.svelte

@@ -0,0 +1,302 @@
+<!--
+Copyright (c) 2021 Michael Keller
+Originally created by Michael Keller (https://github.com/mhkeller/svelte-double-range-slider)
+Changes: remove dependency, text inputs, configurable value ranges, on:change event
+-->
+<!-- 
+	@component
+
+	Properties:
+	- min:          Number
+	- max:          Number
+	- firstSlider:  Number (Starting position of slider #1)
+	- secondSlider: Number (Starting position of slider #2)
+	Events:
+	- `change`: [Number, Number] (Positions of the two sliders)
+ -->
+<script>
+	import { createEventDispatcher } from "svelte";
+
+	export let min;
+	export let max;
+	export let firstSlider;
+	export let secondSlider;
+
+	const dispatch = createEventDispatcher();
+
+	let values;
+	let start, end; /* Positions of sliders from 0 to 1 */
+	$: values = [firstSlider, secondSlider]; /* Avoid feedback loop */
+	$: start = Math.max(((firstSlider  == null ? min : firstSlider)  - min) / (max - min), 0);
+	$: end =   Math.min(((secondSlider == null ? min : secondSlider) - min) / (max - min), 1);
+
+	let leftHandle;
+	let body;
+	let slider;
+	let inputFieldFrom, inputFieldTo;
+
+	let timeoutId = null;
+	function queueChangeEvent() {
+		if (timeoutId !== null) {
+			clearTimeout(timeoutId);
+		}
+
+		timeoutId = setTimeout(() => {
+			timeoutId = null;
+
+			// Show selection but avoid feedback loop
+			if (values[0] != null && inputFieldFrom.value != values[0].toString())
+				inputFieldFrom.value = values[0].toString();
+			if (values[1] != null && inputFieldTo.value != values[1].toString())
+				inputFieldTo.value = values[1].toString();
+
+			dispatch('change', values);
+		}, 250);
+	}
+
+	function update() {
+		values = [
+			Math.floor(min + start  * (max - min)),
+			Math.floor(min + end * (max - min))
+		];
+		queueChangeEvent();
+	}
+
+	function inputChanged(idx, event) {
+		let val = Number.parseInt(event.target.value);
+		if (Number.isNaN(val) || val < min) {
+			event.target.classList.add('bad');
+			return;
+		}
+
+		values[idx] = val;
+		event.target.classList.remove('bad');
+		if (idx == 0)
+			start = clamp((val - min) / (max - min), 0., 1.);
+		else
+			end = clamp((val - min) / (max - min), 0., 1.);
+
+		queueChangeEvent();
+	}
+
+	function clamp(x, min, max) {
+		return x < min
+			? min
+			: (x < max ? x : max);
+	}
+
+	function draggable(node) {
+		let x;
+		let y;
+
+		function handleMousedown(event) {
+			if (event.type === 'touchstart') {
+				event = event.touches[0];
+			}
+			x = event.clientX;
+			y = event.clientY;
+
+			node.dispatchEvent(new CustomEvent('dragstart', {
+				detail: { x, y }
+			}));
+
+			window.addEventListener('mousemove', handleMousemove);
+			window.addEventListener('mouseup', handleMouseup);
+
+			window.addEventListener('touchmove', handleMousemove);
+			window.addEventListener('touchend', handleMouseup);
+		}
+
+		function handleMousemove(event) {
+			if (event.type === 'touchmove') {
+				event = event.changedTouches[0];
+			}
+
+			const dx = event.clientX - x;
+			const dy = event.clientY - y;
+
+			x = event.clientX;
+			y = event.clientY;
+
+			node.dispatchEvent(new CustomEvent('dragmove', {
+				detail: { x, y, dx, dy }
+			}));
+		}
+
+		function handleMouseup(event) {
+			x = event.clientX;
+			y = event.clientY;
+
+			node.dispatchEvent(new CustomEvent('dragend', {
+				detail: { x, y }
+			}));
+
+			window.removeEventListener('mousemove', handleMousemove);
+			window.removeEventListener('mouseup', handleMouseup);
+
+			window.removeEventListener('touchmove', handleMousemove);
+			window.removeEventListener('touchend', handleMouseup);
+		}
+
+		node.addEventListener('mousedown', handleMousedown);
+		node.addEventListener('touchstart', handleMousedown);
+
+		return {
+			destroy() {
+				node.removeEventListener('mousedown', handleMousedown);
+				node.removeEventListener('touchstart', handleMousedown);
+			}
+		};
+	}
+
+	function setHandlePosition (which) {
+		return function (evt) {
+			const { left, right } = slider.getBoundingClientRect();
+			const parentWidth = right - left;
+
+			const p = Math.min(Math.max((evt.detail.x - left) / parentWidth, 0), 1);
+
+			if (which === 'start') {
+				start = p;
+				end = Math.max(end, p);
+			} else {
+				start = Math.min(p, start);
+				end = p;
+			}
+
+			update();
+		}
+	}
+
+	function setHandlesFromBody (evt) {
+		const { width } = body.getBoundingClientRect();
+		const { left, right } = slider.getBoundingClientRect();
+
+		const parentWidth = right - left;
+
+		const leftHandleLeft = leftHandle.getBoundingClientRect().left;
+
+		const pxStart = clamp((leftHandleLeft + event.detail.dx) - left, 0, parentWidth - width);
+		const pxEnd = clamp(pxStart + width, width, parentWidth);
+
+		const pStart = pxStart / parentWidth;
+		const pEnd = pxEnd / parentWidth;
+
+		start = pStart;
+		end = pEnd;
+		update();
+	}
+</script>
+
+<div class="double-range-container">
+	<div class="header">
+		<input class="form-control" type="text" placeholder="from..." bind:this={inputFieldFrom}
+			on:input={(e) => inputChanged(0, e)} />
+
+		<span>Full Range: <b> {min} </b> - <b> {max} </b></span>
+
+		<input class="form-control" type="text" placeholder="to..." bind:this={inputFieldTo}
+			on:input={(e) => inputChanged(1, e)} />
+	</div>
+	<div class="slider" bind:this={slider}>
+		<div
+			class="body"
+			bind:this={body}
+			use:draggable
+			on:dragmove|preventDefault|stopPropagation="{setHandlesFromBody}"
+			style="
+				left: {100 * start}%;
+				right: {100 * (1 - end)}%;
+			"
+			></div>
+		<div
+			class="handle"
+			bind:this={leftHandle}
+			data-which="start"
+			use:draggable
+			on:dragmove|preventDefault|stopPropagation="{setHandlePosition('start')}"
+			style="
+				left: {100 * start}%
+			"
+		></div>
+		<div
+			class="handle"
+			data-which="end"
+			use:draggable
+			on:dragmove|preventDefault|stopPropagation="{setHandlePosition('end')}"
+			style="
+				left: {100 * end}%
+			"
+		></div>
+	</div>
+</div>
+
+<style>
+	.header {
+		width: 100%;
+		display: flex;
+		justify-content: space-between;
+		margin-bottom: -5px;
+	}
+	.header :nth-child(2) {
+		padding-top: 10px;
+	}
+	.header input {
+		height: 25px;
+		border-radius: 5px;
+		width: 100px;
+	}
+
+	:global(.double-range-container .header input[type="text"].bad) {
+		color: #ff5c33;
+		border-color: #ff5c33;
+	}
+
+	.double-range-container {
+		width: 100%;
+		height: 50px;
+		user-select: none;
+		box-sizing: border-box;
+		white-space: nowrap
+	}
+	.slider {
+		position: relative;
+		width: 100%;
+		height: 6px;
+		top: 10px;
+		transform: translate(0, -50%);
+		background-color: #e2e2e2;
+		box-shadow: inset 0 7px 10px -5px #4a4a4a, inset 0 -1px 0px 0px #9c9c9c;
+		border-radius: 6px;
+	}
+	.handle {
+		position: absolute;
+		top: 50%;
+		width: 0;
+		height: 0;
+	}
+	.handle:after {
+		content: ' ';
+		box-sizing: border-box;
+		position: absolute;
+		border-radius: 50%;
+		width: 16px;
+		height: 16px;
+		background-color: #fdfdfd;
+		border: 1px solid #7b7b7b;
+		transform: translate(-50%, -50%)
+	}
+	/* .handle[data-which="end"]:after{
+		transform: translate(-100%, -50%);
+	} */
+	.handle:active:after {
+		background-color: #ddd;
+		z-index: 9;
+	}
+	.body {
+		top: 0;
+		position: absolute;
+		background-color: #34a1ff;
+		bottom: 0;
+	}
+</style>

web/frontend/src/filters/Duration.svelte

@@ -0,0 +1,95 @@
+<script>
+    import { createEventDispatcher } from 'svelte'
+    import { Row, Col, Button, Modal, ModalBody, ModalHeader, ModalFooter, FormGroup } from 'sveltestrap'
+
+    const dispatch = createEventDispatcher()
+
+    export let isOpen = false
+    export let from = null
+    export let to = null
+
+    let pendingFrom, pendingTo
+
+    function reset() {
+        pendingFrom = from == null ? { hours: 0, mins: 0 } : secsToHoursAndMins(from)
+        pendingTo   = to   == null ? { hours: 0, mins: 0 } : secsToHoursAndMins(to)
+    }
+
+    reset()
+
+    function secsToHoursAndMins(duration) {
+        const hours = Math.floor(duration / 3600)
+        duration -= hours * 3600
+        const mins = Math.floor(duration / 60)
+        return { hours, mins }
+    }
+
+    function hoursAndMinsToSecs({ hours, mins }) {
+        return hours * 3600 + mins * 60
+    }
+</script>
+
+<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
+    <ModalHeader>
+        Select Start Time
+    </ModalHeader>
+    <ModalBody>
+        <h4>Between</h4>
+        <Row>
+            <Col>
+                <div class="input-group mb-2 mr-sm-2">
+                    <input type="number" class="form-control"  bind:value={pendingFrom.hours}>
+                    <div class="input-group-append">
+                        <div class="input-group-text">h</div>
+                    </div>
+                </div>
+            </Col>
+            <Col>
+                <div class="input-group mb-2 mr-sm-2">
+                    <input type="number" class="form-control" bind:value={pendingFrom.mins}>
+                    <div class="input-group-append">
+                        <div class="input-group-text">m</div>
+                    </div>
+                </div>
+            </Col>
+        </Row>
+        <h4>and</h4>
+        <Row>
+            <Col>
+                <div class="input-group mb-2 mr-sm-2">
+                    <input type="number" class="form-control" bind:value={pendingTo.hours}>
+                    <div class="input-group-append">
+                        <div class="input-group-text">h</div>
+                    </div>
+                </div>
+            </Col>
+            <Col>
+                <div class="input-group mb-2 mr-sm-2">
+                    <input type="number" class="form-control" bind:value={pendingTo.mins}>
+                    <div class="input-group-append">
+                        <div class="input-group-text">m</div>
+                    </div>
+                </div>
+            </Col>
+        </Row>
+    </ModalBody>
+    <ModalFooter>
+        <Button color="primary"
+            on:click={() => {
+                isOpen = false
+                from = hoursAndMinsToSecs(pendingFrom)
+                to = hoursAndMinsToSecs(pendingTo)
+                dispatch('update', { from, to })
+            }}>
+            Close & Apply
+        </Button>
+        <Button color="danger" on:click={() => {
+            isOpen = false
+            from = null
+            to = null
+            reset()
+            dispatch('update', { from, to })
+        }}>Reset</Button>
+        <Button on:click={() => (isOpen = false)}>Close</Button>
+    </ModalFooter>
+</Modal>

web/frontend/src/filters/Filters.svelte

@@ -0,0 +1,323 @@
+<!--
+    @component
+
+    Properties:
+    - menuText:      String? (Optional text to show in the dropdown menu)
+    - filterPresets: Object? (Optional predefined filter values)
+    Events:
+    - 'update': The detail's 'filters' prop are new filter items to be applied
+    Functions:
+    - void update(additionalFilters: Object?): Triggers an update
+ -->
+<script>
+    import { Row, Col, DropdownItem, DropdownMenu,
+             DropdownToggle, ButtonDropdown, Icon } from 'sveltestrap'
+    import { createEventDispatcher } from 'svelte'
+    import Info from './InfoBox.svelte'
+    import Cluster from './Cluster.svelte'
+    import JobStates, { allJobStates } from './JobStates.svelte'
+    import StartTime from './StartTime.svelte'
+    import Tags from './Tags.svelte'
+    import Tag from '../Tag.svelte'
+    import Duration from './Duration.svelte'
+    import Resources from './Resources.svelte'
+    import Statistics from './Stats.svelte'
+    // import TimeSelection from './TimeSelection.svelte'
+
+    const dispatch = createEventDispatcher()
+
+    export let menuText = null
+    export let filterPresets = {}
+    export let disableClusterSelection = false
+    export let startTimeQuickSelect = false
+
+    let filters = {
+        projectMatch: filterPresets.projectMatch || 'contains',
+        userMatch:    filterPresets.userMatch    || 'contains',
+
+        cluster:    filterPresets.cluster    || null,
+        partition:  filterPresets.partition  || null,
+        states:     filterPresets.states     || filterPresets.state ? [filterPresets.state].flat() : allJobStates,
+        startTime:  filterPresets.startTime  || { from: null, to: null },
+        tags:       filterPresets.tags       || [],
+        duration:   filterPresets.duration   || { from: null, to: null },
+        jobId:      filterPresets.jobId      || '',
+        arrayJobId: filterPresets.arrayJobId || null,
+        user:       filterPresets.user       || '',
+        project:    filterPresets.project    || '',
+
+        numNodes:         filterPresets.numNodes         || { from: null, to: null },
+        numHWThreads:     filterPresets.numHWThreads     || { from: null, to: null },
+        numAccelerators:  filterPresets.numAccelerators  || { from: null, to: null },
+
+        stats: [],
+    }
+
+    let isClusterOpen = false,
+        isJobStatesOpen = false,
+        isStartTimeOpen = false,
+        isTagsOpen = false,
+        isDurationOpen = false,
+        isResourcesOpen = false,
+        isStatsOpen = false
+
+    // Can be called from the outside to trigger a 'update' event from this component.
+    export function update(additionalFilters = null) {
+        if (additionalFilters != null)
+            for (let key in additionalFilters)
+            filters[key] = additionalFilters[key]
+
+        let items = []
+        if (filters.cluster)
+            items.push({ cluster: { eq: filters.cluster } })
+        if (filters.partition)
+            items.push({ partition: { eq: filters.partition } })
+        if (filters.states.length != allJobStates.length)
+            items.push({ state: filters.states })
+        if (filters.startTime.from || filters.startTime.to)
+            items.push({ startTime: { from: filters.startTime.from, to: filters.startTime.to } })
+        if (filters.tags.length != 0)
+            items.push({ tags: filters.tags })
+        if (filters.duration.from || filters.duration.to)
+            items.push({ duration: { from: filters.duration.from, to: filters.duration.to } })
+        if (filters.jobId)
+            items.push({ jobId: { eq: filters.jobId } })
+        if (filters.arrayJobId != null)
+            items.push({ arrayJobId: filters.arrayJobId })
+        if (filters.numNodes.from != null || filters.numNodes.to != null)
+            items.push({ numNodes: { from: filters.numNodes.from, to: filters.numNodes.to } })
+        if (filters.numHWThreads.from != null || filters.numHWThreads.to != null)
+            items.push({ numHWThreads: { from: filters.numHWThreads.from, to: filters.numHWThreads.to } })
+        if (filters.numAccelerators.from != null || filters.numAccelerators.to != null)
+            items.push({ numAccelerators: { from: filters.numAccelerators.from, to: filters.numAccelerators.to } })
+        if (filters.user)
+            items.push({ user: { [filters.userMatch]: filters.user } })
+        if (filters.project)
+            items.push({ project: { [filters.projectMatch]: filters.project } })
+        for (let stat of filters.stats)
+            items.push({ [stat.field]: { from: stat.from, to: stat.to } })
+
+        dispatch('update', { filters: items })
+        changeURL()
+        return items
+    }
+
+    function changeURL() {
+        const dateToUnixEpoch = (rfc3339) => Math.floor(Date.parse(rfc3339) / 1000)
+
+        let opts = []
+        if (filters.cluster)
+            opts.push(`cluster=${filters.cluster}`)
+        if (filters.partition)
+            opts.push(`partition=${filters.partition}`)
+        if (filters.states.length != allJobStates.length)
+            for (let state of filters.states)
+                opts.push(`state=${state}`)
+        if (filters.startTime.from && filters.startTime.to)
+            opts.push(`startTime=${dateToUnixEpoch(filters.startTime.from)}-${dateToUnixEpoch(filters.startTime.to)}`)
+        for (let tag of filters.tags)  
+            opts.push(`tag=${tag}`)
+        if (filters.duration.from && filters.duration.to)
+            opts.push(`duration=${filters.duration.from}-${filters.duration.to}`)
+        if (filters.numNodes.from && filters.numNodes.to)
+            opts.push(`numNodes=${filters.numNodes.from}-${filters.numNodes.to}`)
+        if (filters.numAccelerators.from && filters.numAccelerators.to)
+            opts.push(`numAccelerators=${filters.numAccelerators.from}-${filters.numAccelerators.to}`)
+        if (filters.user)
+            opts.push(`user=${filters.user}`)
+        if (filters.userMatch != 'contains')
+            opts.push(`userMatch=${filters.userMatch}`)
+        if (filters.project)
+            opts.push(`project=${filters.project}`)
+        if (filters.projectMatch != 'contains')
+            opts.push(`projectMatch=${filters.projectMatch}`)
+
+        if (opts.length == 0 && window.location.search.length <= 1)
+            return
+
+        let newurl = `${window.location.pathname}?${opts.join('&')}`
+        window.history.replaceState(null, '', newurl)
+    }
+</script>
+
+<Row>
+    <Col xs="auto">
+        <ButtonDropdown class="cc-dropdown-on-hover">
+            <DropdownToggle outline caret color="success">
+                <Icon name="sliders"/>
+                Filters
+            </DropdownToggle>
+            <DropdownMenu>
+                <DropdownItem header>
+                    Manage Filters
+                </DropdownItem>
+                {#if menuText}
+                    <DropdownItem disabled>{menuText}</DropdownItem>
+                    <DropdownItem divider />
+                {/if}
+                <DropdownItem on:click={() => (isClusterOpen = true)}>
+                    <Icon name="cpu"/> Cluster/Partition
+                </DropdownItem>
+                <DropdownItem on:click={() => (isJobStatesOpen = true)}>
+                    <Icon name="gear-fill"/> Job States
+                </DropdownItem>
+                <DropdownItem on:click={() => (isStartTimeOpen = true)}>
+                    <Icon name="calendar-range"/> Start Time
+                </DropdownItem>
+                <DropdownItem on:click={() => (isDurationOpen = true)}>
+                    <Icon name="stopwatch"/> Duration
+                </DropdownItem>
+                <DropdownItem on:click={() => (isTagsOpen = true)}>
+                    <Icon name="tags"/> Tags
+                </DropdownItem>
+                <DropdownItem on:click={() => (isResourcesOpen = true)}>
+                    <Icon name="hdd-stack"/> Nodes/Accelerators
+                </DropdownItem>
+                <DropdownItem on:click={() => (isStatsOpen = true)}>
+                    <Icon name="bar-chart" on:click={() => (isStatsOpen = true)}/> Statistics
+                </DropdownItem>
+                {#if startTimeQuickSelect}
+                    <DropdownItem divider/>
+                    <DropdownItem disabled>Start Time Qick Selection</DropdownItem>
+                    {#each [
+                        { text: 'Last 6hrs',    seconds: 6*60*60 },
+                        { text: 'Last 12hrs',   seconds: 12*60*60 },
+                        { text: 'Last 24hrs',   seconds: 24*60*60 },
+                        { text: 'Last 48hrs',   seconds: 48*60*60 },
+                        { text: 'Last 7 days',  seconds: 7*24*60*60 },
+                        { text: 'Last 30 days', seconds: 30*24*60*60 }
+                    ] as {text, seconds}}
+                        <DropdownItem on:click={() => {
+                            filters.startTime.from = (new Date(Date.now() - seconds * 1000)).toISOString()
+                            filters.startTime.to = (new Date(Date.now())).toISOString()
+                            update()
+                        }}>
+                            <Icon name="calendar-range"/> {text}
+                        </DropdownItem>
+                    {/each}
+                {/if}
+            </DropdownMenu>
+        </ButtonDropdown>
+    </Col>
+    <!-- {#if startTimeQuickSelect}
+        <Col xs="auto">
+            <TimeSelection customEnabled={false} anyEnabled={true}
+                from={filters.startTime.from ? new Date(filters.startTime.from) : null}
+                to={filters.startTime.to ? new Date(filters.startTime.to) : null}
+                options={{
+                    'Last 6hrs': 6*60*60,
+                    'Last 12hrs': 12*60*60,
+                    'Last 24hrs': 24*60*60,
+                    'Last 48hrs': 48*60*60,
+                    'Last 7 days': 7*24*60*60,
+                    'Last 30 days': 30*24*60*60}}
+                on:change={({ detail: { from, to } }) => {
+                    filters.startTime.from = from?.toISOString()
+                    filters.startTime.to = to?.toISOString()
+                    console.log(filters.startTime)
+                    update()
+                }}
+                />
+        </Col>
+    {/if} -->
+    <Col xs="auto">
+        {#if filters.cluster}
+            <Info icon="cpu" on:click={() => (isClusterOpen = true)}>
+                {filters.cluster}
+                {#if filters.partition}
+                    ({filters.partition})
+                {/if}
+            </Info>
+        {/if}
+
+        {#if filters.states.length != allJobStates.length}
+            <Info icon="gear-fill" on:click={() => (isJobStatesOpen = true)}>
+                {filters.states.join(', ')}
+            </Info>
+        {/if}
+
+        {#if filters.startTime.from || filters.startTime.to}
+            <Info icon="calendar-range" on:click={() => (isStartTimeOpen = true)}>
+                {new Date(filters.startTime.from).toLocaleString()} - {new Date(filters.startTime.to).toLocaleString()}
+            </Info>
+        {/if}
+
+        {#if filters.duration.from || filters.duration.to}
+            <Info icon="stopwatch" on:click={() => (isDurationOpen = true)}>
+                {Math.floor(filters.duration.from / 3600)}h:{Math.floor(filters.duration.from % 3600 / 60)}m
+                -
+                {Math.floor(filters.duration.to / 3600)}h:{Math.floor(filters.duration.to % 3600 / 60)}m
+            </Info>
+        {/if}
+
+        {#if filters.tags.length != 0}
+            <Info icon="tags" on:click={() => (isTagsOpen = true)}>
+                {#each filters.tags as tagId}
+                    <Tag id={tagId} clickable={false} />
+                {/each}
+            </Info>
+        {/if}
+
+        {#if filters.numNodes.from != null || filters.numNodes.to != null}
+            <Info icon="hdd-stack" on:click={() => (isResourcesOpen = true)}>
+                Nodes: {filters.numNodes.from} - {filters.numNodes.to}
+            </Info>
+        {/if}
+
+        {#if filters.stats.length > 0}
+            <Info icon="bar-chart" on:click={() => (isStatsOpen = true)}>
+                {filters.stats.map(stat => `${stat.text}: ${stat.from} - ${stat.to}`).join(', ')}
+            </Info>
+        {/if}
+    </Col>
+</Row>
+
+<Cluster
+    disableClusterSelection={disableClusterSelection}
+    bind:isOpen={isClusterOpen}
+    bind:cluster={filters.cluster}
+    bind:partition={filters.partition}
+    on:update={() => update()} />
+
+<JobStates
+    bind:isOpen={isJobStatesOpen}
+    bind:states={filters.states}
+    on:update={() => update()} />
+
+<StartTime
+    bind:isOpen={isStartTimeOpen}
+    bind:from={filters.startTime.from}
+    bind:to={filters.startTime.to}
+    on:update={() => update()} />
+
+<Duration
+    bind:isOpen={isDurationOpen}
+    bind:from={filters.duration.from}
+    bind:to={filters.duration.to}
+    on:update={() => update()} />
+
+<Tags
+    bind:isOpen={isTagsOpen}
+    bind:tags={filters.tags}
+    on:update={() => update()} />
+
+<Resources cluster={filters.cluster}
+    bind:isOpen={isResourcesOpen}
+    bind:numNodes={filters.numNodes}
+    bind:numHWThreads={filters.numHWThreads}
+    bind:numAccelerators={filters.numAccelerators}
+    on:update={() => update()} />
+
+<Statistics cluster={filters.cluster}
+    bind:isOpen={isStatsOpen}
+    bind:stats={filters.stats}
+    on:update={() => update()} />
+
+<style>
+    :global(.cc-dropdown-on-hover:hover .dropdown-menu) {
+        display: block;
+        margin-top: 0px;
+        padding-top: 0px;
+        transform: none !important;
+    }
+</style>

web/frontend/src/filters/InfoBox.svelte

@@ -0,0 +1,11 @@
+<script>
+    import { Button, Icon } from 'sveltestrap'
+
+    export let icon
+    export let modified = false
+</script>
+
+<Button outline color={modified ? 'warning' : 'primary'} on:click>
+    <Icon name={icon}/>
+    <slot />
+</Button>

web/frontend/src/filters/JobStates.svelte

@@ -0,0 +1,47 @@
+<script context="module">
+    export const allJobStates = [ 'running', 'completed', 'failed', 'cancelled', 'stopped', 'timeout', 'preempted', 'out_of_memory' ]
+</script>
+<script>
+    import { createEventDispatcher } from 'svelte'
+    import { Button, ListGroup, ListGroupItem,
+             Modal, ModalBody, ModalHeader, ModalFooter } from 'sveltestrap'
+
+    const dispatch = createEventDispatcher()
+
+    export let isModified = false
+    export let isOpen = false
+    export let states = [...allJobStates]
+
+    let pendingStates = [...states]
+    $: isModified = states.length != pendingStates.length || !states.every(state => pendingStates.includes(state))
+</script>
+
+<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
+    <ModalHeader>
+        Select Job States
+    </ModalHeader>
+    <ModalBody>
+        <ListGroup>
+            {#each allJobStates as state}
+                <ListGroupItem>
+                    <input type=checkbox bind:group={pendingStates} name="flavours" value={state}>
+                    {state}
+                </ListGroupItem>
+            {/each}
+        </ListGroup>        
+    </ModalBody>
+    <ModalFooter>
+        <Button color="primary" disabled={pendingStates.length == 0} on:click={() => {
+            isOpen = false
+            states = [...pendingStates]
+            dispatch('update', { states })
+        }}>Close & Apply</Button>
+        <Button color="danger" on:click={() => {
+            isOpen = false
+            states = [...allJobStates]
+            pendingStates = [...allJobStates]
+            dispatch('update', { states })
+        }}>Reset</Button>
+        <Button on:click={() => (isOpen = false)}>Close</Button>
+    </ModalFooter>
+</Modal>

web/frontend/src/filters/Resources.svelte

@@ -0,0 +1,99 @@
+<script>
+    import { createEventDispatcher, getContext } from 'svelte'
+    import { Button, Modal, ModalBody, ModalHeader, ModalFooter } from 'sveltestrap'
+    import DoubleRangeSlider from './DoubleRangeSlider.svelte'
+
+    const clusters = getContext('clusters'),
+          initialized = getContext('initialized'),
+          dispatch = createEventDispatcher()
+
+    export let cluster = null
+    export let isModified = false
+    export let isOpen = false
+    export let numNodes = { from: null, to: null }
+    export let numHWThreads = { from: null, to: null }
+    export let numAccelerators = { from: null, to: null }
+
+    let pendingNumNodes = numNodes, pendingNumHWThreads = numHWThreads, pendingNumAccelerators = numAccelerators
+    $: isModified = pendingNumNodes.from != numNodes.from || pendingNumNodes.to != numNodes.to
+        || pendingNumHWThreads.from != numHWThreads.from || pendingNumHWThreads.to != numHWThreads.to
+        || pendingNumAccelerators.from != numAccelerators.from || pendingNumAccelerators.to != numAccelerators.to
+
+    const findMaxNumAccels = clusters => clusters.reduce((max, cluster) => Math.max(max,
+        cluster.subClusters.reduce((max, sc) => Math.max(max, sc.topology.accelerators?.length || 0), 0)), 0)
+
+    let minNumNodes = 1, maxNumNodes = 0, minNumHWThreads = 1, maxNumHWThreads = 0, minNumAccelerators = 0, maxNumAccelerators = 0
+    $: {
+        if ($initialized) {
+            if (cluster != null) {
+                const { filterRanges, subClusters } = clusters.find(c => c.name == cluster)
+                minNumNodes = filterRanges.numNodes.from
+                maxNumNodes = filterRanges.numNodes.to
+                maxNumAccelerators = findMaxNumAccels([{ subClusters }])
+            } else if (clusters.length > 0) {
+                const { filterRanges } = clusters[0]
+                minNumNodes = filterRanges.numNodes.from
+                maxNumNodes = filterRanges.numNodes.to
+                maxNumAccelerators = findMaxNumAccels(clusters)
+                for (let cluster of clusters) {
+                    const { filterRanges } = cluster
+                    minNumNodes = Math.min(minNumNodes, filterRanges.numNodes.from)
+                    maxNumNodes = Math.max(maxNumNodes, filterRanges.numNodes.to)
+                }
+            }
+        }
+    }
+
+    $: {
+        if (isOpen && $initialized && pendingNumNodes.from == null && pendingNumNodes.to == null) {
+            pendingNumNodes = { from: 0, to: maxNumNodes }
+        }
+    }
+</script>
+
+<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
+    <ModalHeader>
+        Select Number of Nodes, HWThreads and Accelerators
+    </ModalHeader>
+    <ModalBody>
+        <h4>Number of Nodes</h4>
+        <DoubleRangeSlider
+            on:change={({ detail }) => (pendingNumNodes = { from: detail[0], to: detail[1] })}
+            min={minNumNodes} max={maxNumNodes}
+            firstSlider={pendingNumNodes.from} secondSlider={pendingNumNodes.to} />
+        <!-- <DoubleRangeSlider
+            on:change={({ detail }) => (pendingNumHWThreads = { from: detail[0], to: detail[1] })}
+            min={minNumHWThreads} max={maxNumHWThreads}
+            firstSlider={pendingNumHWThreads.from} secondSlider={pendingNumHWThreads.to} /> -->
+        {#if maxNumAccelerators != null && maxNumAccelerators > 1}
+            <DoubleRangeSlider
+                on:change={({ detail }) => (pendingNumAccelerators = { from: detail[0], to: detail[1] })}
+                min={minNumAccelerators} max={maxNumAccelerators}
+                firstSlider={pendingNumAccelerators.from} secondSlider={pendingNumAccelerators.to} />
+        {/if}
+    </ModalBody>
+    <ModalFooter>
+        <Button color="primary"
+            disabled={pendingNumNodes.from == null || pendingNumNodes.to == null}
+            on:click={() => {
+                isOpen = false
+                numNodes = { from: pendingNumNodes.from, to: pendingNumNodes.to }
+                numHWThreads = { from: pendingNumHWThreads.from, to: pendingNumHWThreads.to }
+                numAccelerators = { from: pendingNumAccelerators.from, to: pendingNumAccelerators.to }
+                dispatch('update', { numNodes, numHWThreads, numAccelerators })
+            }}>
+            Close & Apply
+        </Button>
+        <Button color="danger" on:click={() => {
+            isOpen = false
+            pendingNumNodes = { from: null, to: null }
+            pendingNumHWThreads = { from: null, to: null }
+            pendingNumAccelerators = { from: null, to: null }
+            numNodes = { from: pendingNumNodes.from, to: pendingNumNodes.to }
+            numHWThreads = { from: pendingNumHWThreads.from, to: pendingNumHWThreads.to }
+            numAccelerators = { from: pendingNumAccelerators.from, to: pendingNumAccelerators.to }
+            dispatch('update', { numNodes, numHWThreads, numAccelerators })
+        }}>Reset</Button>
+        <Button on:click={() => (isOpen = false)}>Close</Button>
+    </ModalFooter>
+</Modal>

web/frontend/src/filters/StartTime.svelte

@@ -0,0 +1,90 @@
+<script>
+    import { createEventDispatcher, getContext } from 'svelte'
+    import { Row, Button, Input, Modal, ModalBody, ModalHeader, ModalFooter, FormGroup } from 'sveltestrap'
+
+    const dispatch = createEventDispatcher()
+
+    export let isModified = false
+    export let isOpen = false
+    export let from = null
+    export let to = null
+
+    let pendingFrom, pendingTo
+
+    function reset() {
+        pendingFrom = from == null ? { date: '0000-00-00', time: '00:00' } : fromRFC3339(from)
+        pendingTo   = to   == null ? { date: '0000-00-00', time: '00:00' } : fromRFC3339(to)
+    }
+
+    reset()
+
+    function toRFC3339({ date, time }, secs = 0) {
+        const dparts = date.split('-')
+        const tparts = time.split(':')
+        const d = new Date(
+            Number.parseInt(dparts[0]),
+            Number.parseInt(dparts[1]) - 1,
+            Number.parseInt(dparts[2]),
+            Number.parseInt(tparts[0]),
+            Number.parseInt(tparts[1]), secs)
+        return d.toISOString()
+    }
+
+    function fromRFC3339(rfc3339) {
+        const d = new Date(rfc3339)
+        const pad = (n) => n.toString().padStart(2, '0')
+        const date = `${d.getFullYear()}-${pad(d.getMonth() + 1)}-${pad(d.getDate())}`
+        const time = `${pad(d.getHours())}:${pad(d.getMinutes())}`
+        return { date, time }
+    }
+
+    $: isModified = (from != toRFC3339(pendingFrom) || to != toRFC3339(pendingTo, 59))
+        && !(from == null && pendingFrom.date == '0000-00-00' && pendingFrom.time == '00:00')
+        && !(to == null && pendingTo.date == '0000-00-00' && pendingTo.time == '00:00')
+</script>
+
+<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
+    <ModalHeader>
+        Select Start Time
+    </ModalHeader>
+    <ModalBody>
+        <h4>From</h4>
+        <Row>
+            <FormGroup class="col">
+                <Input type="date" bind:value={pendingFrom.date}/>
+            </FormGroup>
+            <FormGroup class="col">
+                <Input type="time" bind:value={pendingFrom.time}/>
+            </FormGroup>
+        </Row>
+        <h4>To</h4>
+        <Row>
+            <FormGroup class="col">
+                <Input type="date" bind:value={pendingTo.date}/>
+            </FormGroup>
+            <FormGroup class="col">
+                <Input type="time" bind:value={pendingTo.time}/>
+            </FormGroup>
+        </Row>
+    </ModalBody>
+    <ModalFooter>
+        <Button color="primary"
+            disabled={pendingFrom.date == '0000-00-00' || pendingTo.date == '0000-00-00'}
+            on:click={() => {
+                isOpen = false
+                from = toRFC3339(pendingFrom)
+                to = toRFC3339(pendingTo, 59)
+                dispatch('update', { from, to })
+            }}>
+            Close & Apply
+        </Button>
+        <Button color="danger" on:click={() => {
+            isOpen = false
+            from = null
+            to = null
+            reset()
+            dispatch('update', { from, to })
+        }}>Reset</Button>
+        <Button on:click={() => (isOpen = false)}>Close</Button>
+    </ModalFooter>
+</Modal>

web/frontend/src/filters/Stats.svelte

@@ -0,0 +1,113 @@
+<script>
+    import { createEventDispatcher, getContext } from 'svelte'
+    import { Button, Modal, ModalBody, ModalHeader, ModalFooter } from 'sveltestrap'
+    import DoubleRangeSlider from './DoubleRangeSlider.svelte'
+
+    const clusters = getContext('clusters'),
+          initialized = getContext('initialized'),
+          dispatch = createEventDispatcher()
+
+    export let cluster = null
+    export let isModified = false
+    export let isOpen = false
+    export let stats = []
+
+    let statistics = [
+        {
+            field: 'flopsAnyAvg',
+            text: 'FLOPs (Avg.)',
+            metric: 'flops_any',
+            from: 0, to: 0, peak: 0,
+            enabled: false
+        },
+        {
+            field: 'memBwAvg',
+            text: 'Mem. Bw. (Avg.)',
+            metric: 'mem_bw',
+            from: 0, to: 0, peak: 0,
+            enabled: false
+        },
+        {
+            field: 'loadAvg',
+            text: 'Load (Avg.)',
+            metric: 'cpu_load',
+            from: 0, to: 0, peak: 0,
+            enabled: false
+        },
+        {
+            field: 'memUsedMax',
+            text: 'Mem. used (Max.)',
+            metric: 'mem_used',
+            from: 0, to: 0, peak: 0,
+            enabled: false
+        }
+    ]
+
+    $: isModified = !statistics.every(a => {
+        let b = stats.find(s => s.field == a.field)
+        if (b == null)
+            return !a.enabled
+
+        return a.from == b.from && a.to == b.to
+    })
+
+    function getPeak(cluster, metric) {
+        const mc = cluster.metricConfig.find(mc => mc.name == metric)
+        return mc ? mc.peak : 0
+    }
+
+    function resetRange(isInitialized, cluster) {
+        if (!isInitialized)
+            return
+
+        if (cluster != null) {
+            let c = clusters.find(c => c.name == cluster)
+            for (let stat of statistics) {
+                stat.peak = getPeak(c, stat.metric)
+                stat.from = 0
+                stat.to = stat.peak
+            }
+        } else {
+            for (let stat of statistics) {
+                for (let c of clusters) {
+                    stat.peak = Math.max(stat.peak, getPeak(c, stat.metric))
+                }
+                stat.from = 0
+                stat.to = stat.peak
+            }
+        }
+
+        statistics = [...statistics]
+    }
+
+    $: resetRange($initialized, cluster)
+</script>
+
+<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
+    <ModalHeader>
+        Filter based on statistics (of non-running jobs)
+    </ModalHeader>
+    <ModalBody>
+        {#each statistics as stat}
+            <h4>{stat.text}</h4>
+            <DoubleRangeSlider
+                on:change={({ detail }) => (stat.from = detail[0], stat.to = detail[1], stat.enabled = true)}
+                min={0} max={stat.peak}
+                firstSlider={stat.from} secondSlider={stat.to} />
+        {/each}
+    </ModalBody>
+    <ModalFooter>
+        <Button color="primary" on:click={() => {
+            isOpen = false
+            stats = statistics.filter(stat => stat.enabled)
+            dispatch('update', { stats })
+        }}>Close & Apply</Button>
+        <Button color="danger" on:click={() => {
+            isOpen = false
+            statistics.forEach(stat => stat.enabled = false)
+            stats = []
+            dispatch('update', { stats })
+        }}>Reset</Button>
+        <Button on:click={() => (isOpen = false)}>Close</Button>
+    </ModalFooter>
+</Modal>

web/frontend/src/filters/Tags.svelte

@@ -0,0 +1,67 @@
+<script>
+    import { createEventDispatcher, getContext } from 'svelte'
+    import { Button, ListGroup, ListGroupItem, Input,
+             Modal, ModalBody, ModalHeader, ModalFooter, Icon } from 'sveltestrap'
+    import { fuzzySearchTags } from '../utils.js'
+    import Tag from '../Tag.svelte'
+
+    const allTags = getContext('tags'),
+          initialized = getContext('initialized'),
+          dispatch = createEventDispatcher()
+
+    export let isModified = false
+    export let isOpen = false
+    export let tags = []
+
+    let pendingTags = [...tags]
+    $: isModified = tags.length != pendingTags.length || !tags.every(tagId => pendingTags.includes(tagId))
+
+    let searchTerm = ''
+</script>
+
+<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
+    <ModalHeader>
+        Select Tags
+    </ModalHeader>
+    <ModalBody>
+        <Input type="text" placeholder="Search" bind:value={searchTerm} />
+        <br/>
+        <ListGroup>
+            {#if $initialized}
+                {#each fuzzySearchTags(searchTerm, allTags) as tag (tag)}
+                    <ListGroupItem>
+                        {#if pendingTags.includes(tag.id)}
+                            <Button outline color="danger"
+                                on:click={() => (pendingTags = pendingTags.filter(id => id != tag.id))}>
+                                <Icon name="dash-circle" />
+                            </Button>
+                        {:else}
+                            <Button outline color="success"
+                                on:click={() => (pendingTags = [...pendingTags, tag.id])}>
+                                <Icon name="plus-circle" />
+                            </Button>
+                        {/if}
+
+                        <Tag tag={tag} />
+                    </ListGroupItem>
+                {:else}
+                    <ListGroupItem disabled>No Tags</ListGroupItem>
+                {/each}
+            {/if}
+        </ListGroup>
+    </ModalBody>
+    <ModalFooter>
+        <Button color="primary" on:click={() => {
+            isOpen = false
+            tags = [...pendingTags]
+            dispatch('update', { tags })
+        }}>Close & Apply</Button>
+        <Button color="danger" on:click={() => {
+            isOpen = false
+            tags = []
+            pendingTags = []
+            dispatch('update', { tags })
+        }}>Reset</Button>
+        <Button on:click={() => (isOpen = false)}>Close</Button>
+    </ModalFooter>
+</Modal>

web/frontend/src/filters/TimeSelection.svelte

@@ -0,0 +1,80 @@
+<script>
+    import { Icon, Input, InputGroup, InputGroupText } from 'sveltestrap'
+    import { createEventDispatcher } from "svelte"
+
+    export let from
+    export let to
+    export let customEnabled = true
+    export let anyEnabled = false
+    export let options = {
+        'Last half hour': 30*60,
+        'Last hour': 60*60,
+        'Last 2hrs': 2*60*60,
+        'Last 4hrs': 4*60*60,
+        'Last 12hrs': 12*60*60,
+        'Last 24hrs': 24*60*60
+    }
+
+    $: pendingFrom = from
+    $: pendingTo = to
+
+    const dispatch = createEventDispatcher()
+    let timeRange = to && from
+        ? (to.getTime() - from.getTime()) / 1000
+        : (anyEnabled ? -2 : -1)
+
+    function updateTimeRange(event) {
+        if (timeRange == -1) {
+            pendingFrom = null
+            pendingTo = null
+            return
+        }
+        if (timeRange == -2) {
+            from = pendingFrom = null
+            to = pendingTo = null
+            dispatch('change', { from, to })
+            return
+        }
+
+        let now = Date.now(), t = timeRange * 1000
+        from = pendingFrom = new Date(now - t)
+        to = pendingTo = new Date(now)
+        dispatch('change', { from, to })
+    }
+
+    function updateExplicitTimeRange(type, event) {
+        let d = new Date(Date.parse(event.target.value));
+        if (type == 'from') pendingFrom = d
+        else                pendingTo = d
+
+        if (pendingFrom != null && pendingTo != null) {
+            from = pendingFrom
+            to = pendingTo
+            dispatch('change', { from, to })
+        }
+    }
+</script>
+
+<InputGroup class="inline-from">
+    <InputGroupText><Icon name="clock-history"/></InputGroupText>
+    <!-- <InputGroupText>
+        Time
+    </InputGroupText> -->
+    <select class="form-select" bind:value={timeRange} on:change={updateTimeRange}>
+        {#if customEnabled}
+            <option value={-1}>Custom</option>            
+        {/if}
+        {#if anyEnabled}
+            <option value={-2}>Any</option>
+        {/if}
+        {#each Object.entries(options) as [name, seconds]}
+            <option value={seconds}>{name}</option>
+        {/each}
+    </select>
+    {#if timeRange == -1}
+        <InputGroupText>from</InputGroupText>
+        <Input type="datetime-local" on:change={(event) => updateExplicitTimeRange('from', event)}></Input>
+        <InputGroupText>to</InputGroupText>
+        <Input type="datetime-local" on:change={(event) => updateExplicitTimeRange('to', event)}></Input>
+    {/if}
+</InputGroup>

web/frontend/src/filters/UserOrProject.svelte

@@ -0,0 +1,51 @@
+<script>
+    import { InputGroup, Input } from 'sveltestrap'
+    import { createEventDispatcher } from 'svelte'
+
+    const dispatch = createEventDispatcher()
+
+    export let user = ''
+    export let project = ''
+    let mode = 'user', term = ''
+    const throttle = 500
+
+    function modeChanged() {
+        if (mode == 'user') {
+            project = term
+            term = user
+        } else {
+            user = term
+            term = project
+        }
+        termChanged(0)
+    }
+
+    let timeoutId = null
+    function termChanged(sleep = throttle) {
+        if (mode == 'user')
+            user = term
+        else
+            project = term
+
+        if (timeoutId != null)
+            clearTimeout(timeoutId)
+
+        timeoutId = setTimeout(() => {
+            dispatch('update', {
+                user,
+                project
+            })
+        }, sleep)
+    }
+</script>
+
+<InputGroup>
+    <select style="max-width: 175px;" class="form-select"
+        bind:value={mode} on:change={modeChanged}>
+        <option value={'user'}>Search User</option>
+        <option value={'project'}>Search Project</option>
+    </select>
+    <Input
+        type="text" bind:value={term} on:change={() => termChanged()} on:keyup={(event) => termChanged(event.key == 'Enter' ? 0 : throttle)}
+        placeholder={mode == 'user' ? 'filter username...' : 'filter project...'} />
+</InputGroup>

web/frontend/src/header.entrypoint.js

@@ -0,0 +1,10 @@
+import Header from './Header.svelte'
+
+const headerDomTarget = document.getElementById('svelte-header')
+
+if (headerDomTarget != null) {
+    new Header({
+        target: headerDomTarget,
+        props: { ...header },
+    })
+}

web/frontend/src/job.entrypoint.js

@@ -0,0 +1,12 @@
+import {} from './header.entrypoint.js'
+import Job from './Job.root.svelte'
+
+new Job({
+    target: document.getElementById('svelte-app'),
+    props: {
+        dbid: jobInfos.id
+    },
+    context: new Map([
+            ['cc-config', clusterCockpitConfig]
+    ])
+})

web/frontend/src/joblist/JobInfo.svelte

@@ -0,0 +1,88 @@
+<!-- 
+    @component
+
+    Properties:
+    - job: GraphQL.Job
+    - jobTags: Defaults to job.tags, usefull for dynamically updating the tags.
+ -->
+<script context="module">
+    export const scrambleNames = window.localStorage.getItem("cc-scramble-names")
+    export const scramble = (str) => [...str].reduce((x, c, i) => x * 7 + c.charCodeAt(0) * i * 21, 5).toString(32)
+</script>
+<script>
+    import Tag from '../Tag.svelte';
+    import { Badge, Icon } from 'sveltestrap';
+
+    export let job;
+    export let jobTags = job.tags;
+
+    function formatDuration(duration) {
+        const hours = Math.floor(duration / 3600);
+        duration -= hours * 3600;
+        const minutes = Math.floor(duration / 60);
+        duration -= minutes * 60;
+        const seconds = duration;
+        return `${hours}:${('0' + minutes).slice(-2)}:${('0' + seconds).slice(-2)}`;
+    }
+</script>
+
+<div>
+    <p>
+        <span class="fw-bold"><a href="/monitoring/job/{job.id}" target="_blank">{job.jobId}</a> ({job.cluster})</span>
+        {#if job.metaData?.jobName}
+            <br/>
+            {job.metaData.jobName}
+        {/if}
+        {#if job.arrayJobId}
+            Array Job: <a href="/monitoring/jobs/?arrayJobId={job.arrayJobId}&cluster={job.cluster}" target="_blank">#{job.arrayJobId}</a>
+        {/if}
+    </p>
+
+    <p>
+        <Icon name="person-fill"/>
+        <a class="fst-italic" href="/monitoring/user/{job.user}" target="_blank">
+            {scrambleNames ? scramble(job.user) : job.user}
+        </a>
+        {#if job.userData && job.userData.name}
+            ({scrambleNames ? scramble(job.userData.name) : job.userData.name})
+        {/if}
+        {#if job.project && job.project != 'no project'}
+            <br/>
+            <Icon name="people-fill"/> {job.project}
+        {/if}
+    </p>
+
+    <p>
+        {job.numNodes} <Icon name="pc-horizontal"/>
+        {#if job.exclusive != 1}
+            (shared)
+        {/if}
+        {#if job.numAcc > 0}
+            , {job.numAcc} <Icon name="gpu-card"/>
+        {/if}
+        {#if job.numHWThreads > 0}
+            , {job.numHWThreads} <Icon name="cpu"/>
+        {/if}
+    </p>
+
+    <p>
+        Start: <span class="fw-bold">{(new Date(job.startTime)).toLocaleString()}</span>
+        <br/>
+        Duration: <span class="fw-bold">{formatDuration(job.duration)}</span>
+        {#if job.state == 'running'}
+            <Badge color="success">running</Badge>
+        {:else if job.state != 'completed'}
+            <Badge color="danger">{job.state}</Badge>
+        {/if}
+        {#if job.walltime}
+            <br/>
+            Walltime: <span class="fw-bold">{formatDuration(job.walltime)}</span>
+        {/if}
+    </p>
+
+    <p>
+        {#each jobTags as tag}
+            <Tag tag={tag}/>
+        {/each}
+    </p>
+</div>

web/frontend/src/joblist/JobList.svelte

@@ -0,0 +1,190 @@
+<!--
+    @component
+
+    Properties:
+    - metrics:     [String] (can change from outside)
+    - sorting:     { field: String, order: "DESC" | "ASC" } (can change from outside)
+    - matchedJobs: Number (changes from inside)
+    Functions:
+    - update(filters?: [JobFilter])
+ -->
+<script>
+    import { operationStore, query, mutation } from '@urql/svelte'
+    import { getContext } from 'svelte';
+    import { Row, Table, Card, Spinner } from 'sveltestrap'
+    import Pagination from './Pagination.svelte'
+    import JobListRow from './Row.svelte'
+    import { stickyHeader } from '../utils.js'
+
+    const ccconfig = getContext('cc-config'),
+          clusters = getContext('clusters'),
+          initialized = getContext('initialized')
+
+    export let sorting = { field: "startTime", order: "DESC" }
+    export let matchedJobs = 0
+    export let metrics = ccconfig.plot_list_selectedMetrics
+
+    let itemsPerPage = ccconfig.plot_list_jobsPerPage
+    let page = 1
+    let paging = { itemsPerPage, page }
+
+    const jobs = operationStore(`
+    query($filter: [JobFilter!]!, $sorting: OrderByInput!, $paging: PageRequest! ){
+        jobs(filter: $filter, order: $sorting, page: $paging) {
+            items {
+                id, jobId, user, project, cluster, subCluster, startTime,
+                duration, numNodes, numHWThreads, numAcc, walltime,
+                SMT, exclusive, partition, arrayJobId,
+                monitoringStatus, state,
+                tags { id, type, name }
+                userData { name }
+                metaData
+            }
+            count
+        }
+    }`, {
+        paging,
+        sorting,
+        filter: []
+    }, {
+        pause: true
+    })
+
+    const updateConfiguration = mutation({
+        query: `mutation($name: String!, $value: String!) {
+            updateConfiguration(name: $name, value: $value)
+        }`
+    })
+
+    $: $jobs.variables = { ...$jobs.variables, sorting, paging }
+    $: matchedJobs = $jobs.data != null ? $jobs.data.jobs.count : 0
+
+    // (Re-)query and optionally set new filters.
+    export function update(filters) {
+        if (filters != null) {
+            let minRunningFor = ccconfig.plot_list_hideShortRunningJobs
+            if (minRunningFor && minRunningFor > 0) {
+                filters.push({ minRunningFor })
+            }
+
+            $jobs.variables.filter = filters
+            console.log('filters:', ...filters.map(f => Object.entries(f)).flat(2))
+        }
+
+        page = 1
+        $jobs.variables.paging = paging = { page, itemsPerPage };
+        $jobs.context.pause = false
+        $jobs.reexecute({ requestPolicy: 'network-only' })
+    }
+
+    query(jobs)
+
+    let tableWidth = null
+    let jobInfoColumnWidth = 250
+    $: plotWidth = Math.floor((tableWidth - jobInfoColumnWidth) / metrics.length - 10)
+
+    let headerPaddingTop = 0
+    stickyHeader('.cc-table-wrapper > table.table >thead > tr > th.position-sticky:nth-child(1)', (x) => (headerPaddingTop = x))
+</script>
+
+<Row>
+    <div class="col cc-table-wrapper" bind:clientWidth={tableWidth}>
+        <Table cellspacing="0px" cellpadding="0px">
+            <thead>
+                <tr>
+                    <th class="position-sticky top-0" scope="col" style="width: {jobInfoColumnWidth}px; padding-top: {headerPaddingTop}px">
+                        Job Info
+                    </th>
+                    {#each metrics as metric (metric)}
+                        <th class="position-sticky top-0 text-center" scope="col" style="width: {plotWidth}px; padding-top: {headerPaddingTop}px">
+                            {metric}
+                            {#if $initialized}
+                                ({clusters
+                                    .map(cluster => cluster.metricConfig.find(m => m.name == metric))
+                                    .filter(m => m != null).map(m => m.unit)
+                                    .reduce((arr, unit) => arr.includes(unit) ? arr : [...arr, unit], [])
+                                    .join(', ')})
+                            {/if}
+                        </th>
+                    {/each}
+                </tr>
+            </thead>
+            <tbody>
+                {#if $jobs.error}
+                    <tr>
+                        <td colspan="{metrics.length + 1}">
+                            <Card body color="danger" class="mb-3"><h2>{$jobs.error.message}</h2></Card>
+                        </td>
+                    </tr>
+                {:else if $jobs.fetching || !$jobs.data}
+                    <tr>
+                        <td colspan="{metrics.length + 1}">
+                            <Spinner secondary />
+                        </td>
+                    </tr>
+                {:else if $jobs.data && $initialized}
+                    {#each $jobs.data.jobs.items as job (job)}
+                        <JobListRow
+                            job={job}
+                            metrics={metrics}
+                            plotWidth={plotWidth} />
+                    {:else}
+                    <tr>
+                        <td colspan="{metrics.length + 1}">
+                            No jobs found
+                        </td>
+                    </tr>
+                    {/each}
+                {/if}
+            </tbody>
+        </Table>
+    </div>
+</Row>
+
+<Pagination
+    bind:page={page}
+    {itemsPerPage}
+    itemText="Jobs"
+    totalItems={matchedJobs}
+    on:update={({ detail }) => {
+        if (detail.itemsPerPage != itemsPerPage) {
+            itemsPerPage = detail.itemsPerPage
+            updateConfiguration({
+                name: "plot_list_jobsPerPage",
+                value: itemsPerPage.toString()
+            }).then(res => {
+                if (res.error)
+                    console.error(res.error);
+            })
+        }
+
+        paging = { itemsPerPage: detail.itemsPerPage, page: detail.page }
+    }} />
+
+<style>
+    .cc-table-wrapper {
+        overflow: initial;
+    }
+
+    .cc-table-wrapper > :global(table) {
+        border-collapse: separate;
+        border-spacing: 0px;
+        table-layout: fixed;
+    }
+
+    .cc-table-wrapper :global(button) {
+        margin-bottom: 0px;
+    }
+
+    .cc-table-wrapper > :global(table > tbody > tr > td) {
+        margin: 0px;
+        padding-left: 5px;
+        padding-right: 0px;
+    }
+
+    th.position-sticky.top-0 {
+        background-color: white;
+        z-index: 10;
+        border-bottom: 1px solid black;
+    }
+</style>

web/frontend/src/joblist/Pagination.svelte

@@ -0,0 +1,230 @@
+<!-- 
+    @component
+
+    Properties:
+    - page:         Number (changes from inside)
+    - itemsPerPage: Number (changes from inside)
+    - totalItems:   Number (only displayed)
+    Events:
+    - "update": { page: Number, itemsPerPage: Number }
+      - Dispatched once immediately and then each time page or itemsPerPage changes
+ -->
+
+<div class="cc-pagination" >
+    <div class="cc-pagination-left">
+        <label for="cc-pagination-select">{ itemText } per page:</label>
+        <div class="cc-pagination-select-wrapper">
+            <select on:blur|preventDefault={reset} bind:value={itemsPerPage} id="cc-pagination-select" class="cc-pagination-select">
+                {#each pageSizes as size}
+                    <option value="{size}">{size}</option>
+                {/each}
+            </select>
+            <span class="focus"></span>
+        </div>
+        <span class="cc-pagination-text">
+            { (page - 1) * itemsPerPage } - { Math.min((page - 1) * itemsPerPage + itemsPerPage, totalItems) } of { totalItems } { itemText }
+        </span>
+    </div>
+    <div class="cc-pagination-right">
+        {#if !backButtonDisabled}
+        <button class="reset nav" type="button"
+            on:click|preventDefault="{reset}"></button>
+        <button class="left nav" type="button"
+            on:click|preventDefault="{() => { page -= 1; }}"></button>
+        {/if}
+        {#if !nextButtonDisabled}
+        <button class="right nav" type="button"
+            on:click|preventDefault="{() => { page += 1; }}"></button>
+        {/if}
+    </div>
+</div>
+
+<script>
+    import { createEventDispatcher } from "svelte";
+    export let page = 1;
+    export let itemsPerPage = 10;
+    export let totalItems = 0;
+    export let itemText = "items";
+    export let pageSizes = [10,25,50];
+
+    let backButtonDisabled, nextButtonDisabled;
+
+    const dispatch = createEventDispatcher();
+
+    $: {
+        if (typeof page !== "number") {
+            page = Number(page);
+        }
+
+        if (typeof itemsPerPage !== "number") {
+            itemsPerPage = Number(itemsPerPage);
+        }
+
+        dispatch("update", { itemsPerPage, page });
+    }
+    $: backButtonDisabled = (page === 1);
+    $: nextButtonDisabled = (page >= (totalItems / itemsPerPage));
+
+    function reset ( event ) {
+        page = 1;
+    }
+</script>
+
+<style>
+    *, *::before, *::after {
+        box-sizing: border-box;
+    }
+
+    div {
+        display: flex;
+        align-items: center;
+        vertical-align: baseline;
+        box-sizing: border-box;
+    }
+
+    label, select, button {
+        margin: 0;
+        padding: 0;
+        vertical-align: baseline;
+        color: #525252;
+    }
+
+    button {
+        position: relative;
+        border: none;
+        border-left: 1px solid #e0e0e0;
+        height: 3rem;
+        width: 3rem;
+        background: 0 0;
+        transition: all 70ms;
+    }
+
+    button:hover {
+        background-color: #dde1e6;
+    }
+
+    button:focus {
+        top: -1px;
+        left: -1px;
+        right: -1px;
+        bottom: -1px;
+        border: 1px solid blue;
+        border-radius: inherit;
+    }
+
+    .nav::after {
+        content: "";
+        width: 0.9em;
+        height: 0.8em;
+        background-color: #777;
+        z-index: 1;
+        position: absolute;
+        top: 50%;
+        left: 50%;
+    }
+
+    .nav:disabled {
+        background-color: #fff;
+        cursor: no-drop;
+    }
+
+    .reset::after {
+        clip-path: polygon(100% 0%, 75% 50%, 100% 100%, 25% 100%, 0% 50%, 25% 0%);
+        margin-top: -0.3em;
+        margin-left: -0.5em;
+    }
+
+    .right::after {
+        clip-path: polygon(100% 50%, 50% 0, 50% 100%);
+        margin-top: -0.3em;
+        margin-left: -0.5em;
+    }
+
+    .left::after {
+        clip-path: polygon(50% 0, 0 50%, 50% 100%);
+        margin-top: -0.3em;
+        margin-left: -0.3em;
+    }
+
+    .cc-pagination-select-wrapper::after {
+        content: "";
+        width: 0.8em;
+        height: 0.5em;
+        background-color: #777;
+        clip-path: polygon(100% 0%, 0 0%, 50% 100%);
+        justify-self: end;
+    }
+
+    .cc-pagination {
+        width: 100%;
+        justify-content: space-between;
+        border-top: 1px solid #e0e0e0;
+    }
+
+    .cc-pagination-text {
+        color: #525252;
+        margin-left: 1rem;
+    }
+
+    .cc-pagination-text {
+        color: #525252;
+        margin-right: 1rem;
+    }
+
+    .cc-pagination-left {
+        padding: 0 1rem;
+        height: 3rem;
+    }
+
+    .cc-pagination-select-wrapper {
+        display: grid;
+        grid-template-areas: "select";
+        align-items: center;
+        position: relative;
+        padding: 0 0.5em;
+        min-width: 3em;
+        max-width: 6em;
+        border-right: 1px solid #e0e0e0;
+        cursor: pointer;
+        transition: all 70ms;
+    }
+
+    .cc-pagination-select-wrapper:hover {
+        background-color: #dde1e6;
+    }
+
+    select,
+    .cc-pagination-select-wrapper::after {
+        grid-area: select;
+    }
+
+    .cc-pagination-select {
+        height: 3rem;
+        appearance: none;
+        background-color: transparent;
+        padding: 0 1em 0 0;
+        margin: 0;
+        border: none;
+        width: 100%;
+        font-family: inherit;
+        font-size: inherit;
+        cursor: inherit;
+        line-height: inherit;
+        z-index: 1;
+        outline: none;
+    }
+
+    select:focus + .focus {
+        position: absolute;
+        top: -1px;
+        left: -1px;
+        right: -1px;
+        bottom: -1px;
+        border: 1px solid blue;
+        border-radius: inherit;
+    }
+
+    .cc-pagination-right {
+        height: 3rem;
+    }
+</style>

web/frontend/src/joblist/Refresher.svelte

@@ -0,0 +1,43 @@
+<!-- 
+    @component
+
+    Events:
+    - 'reload': When fired, the parent component shoud refresh its contents
+ -->
+<script>
+    import { createEventDispatcher } from 'svelte'
+    import { Button, Icon, InputGroup } from 'sveltestrap'
+
+    const dispatch = createEventDispatcher()
+
+    let refreshInterval = null;
+    let refreshIntervalId = null;
+    function refreshIntervalChanged() {
+        if (refreshIntervalId != null)
+            clearInterval(refreshIntervalId);
+
+        if (refreshInterval == null)
+            return;
+
+        refreshIntervalId = setInterval(() => dispatch("reload"), refreshInterval);
+    }
+
+    export let initially = null
+    if (initially != null) {
+        refreshInterval = initially * 1000
+        refreshIntervalChanged()
+    }
+</script>
+
+<InputGroup>
+    <Button outline on:click={() => dispatch("reload")} disabled={refreshInterval != null}>
+        <Icon name="arrow-clockwise" /> Reload
+    </Button>
+    <select class="form-select" bind:value={refreshInterval} on:change={refreshIntervalChanged}>
+        <option value={null}>No periodic reload</option>
+        <option value={    30 * 1000}>Update every 30 seconds</option>
+        <option value={    60 * 1000}>Update every minute</option>
+        <option value={2 * 60 * 1000}>Update every two minutes</option>
+        <option value={5 * 60 * 1000}>Update every 5 minutes</option>
+    </select>
+</InputGroup>

web/frontend/src/joblist/Row.svelte

@@ -0,0 +1,101 @@
+<!-- 
+    @component
+
+    Properties:
+    - job:        GraphQL.Job (constant/key)
+    - metrics:    [String]    (can change)
+    - plotWidth:  Number
+    - plotHeight: Number
+ -->
+
+<script>
+    import { operationStore, query } from '@urql/svelte'
+    import { getContext } from 'svelte'
+    import { Card, Spinner } from 'sveltestrap'
+    import MetricPlot from '../plots/MetricPlot.svelte'
+    import JobInfo from './JobInfo.svelte'
+    import { maxScope } from '../utils.js'
+
+    export let job
+    export let metrics
+    export let plotWidth
+    export let plotHeight = 275
+
+    let scopes = [job.numNodes == 1 ? 'core' : 'node']
+
+    const cluster = getContext('clusters').find(c => c.name == job.cluster)
+
+    const metricsQuery = operationStore(`query($id: ID!, $metrics: [String!]!, $scopes: [MetricScope!]!) {
+        jobMetrics(id: $id, metrics: $metrics, scopes: $scopes) {
+            name
+            metric {
+                unit, scope, timestep
+                statisticsSeries { min, mean, max }
+                series {
+                    hostname, id, data
+                    statistics { min, avg, max }
+                }
+            }
+        }
+    }`, {
+        id: job.id,
+        metrics,
+        scopes
+    })
+
+    const selectScope = (jobMetrics) => jobMetrics.reduce(
+        (a, b) => maxScope([a.metric.scope, b.metric.scope]) == a.metric.scope
+            ? (job.numNodes > 1 ? a : b)
+            : (job.numNodes > 1 ? b : a), jobMetrics[0])
+
+    const sortAndSelectScope = (jobMetrics) => metrics
+        .map(name => jobMetrics.filter(jobMetric => jobMetric.name == name))
+        .map(jobMetrics => jobMetrics.length > 0 ? selectScope(jobMetrics) : null)
+
+    $: metricsQuery.variables = { id: job.id, metrics, scopes }
+
+    if (job.monitoringStatus)
+        query(metricsQuery)
+</script>
+
+<tr>
+    <td>
+        <JobInfo job={job}/>
+    </td>
+    {#if job.monitoringStatus == 0 || job.monitoringStatus == 2}
+        <td colspan="{metrics.length}">
+            <Card body color="warning">Not monitored or archiving failed</Card>
+        </td>
+    {:else if $metricsQuery.fetching}
+        <td colspan="{metrics.length}" style="text-align: center;">
+            <Spinner secondary />
+        </td>
+    {:else if $metricsQuery.error}
+        <td colspan="{metrics.length}">
+            <Card body color="danger" class="mb-3">
+                {$metricsQuery.error.message.length > 500
+                    ? $metricsQuery.error.message.substring(0, 499)+'...'
+                    : $metricsQuery.error.message}
+            </Card>
+        </td>
+    {:else}
+        {#each sortAndSelectScope($metricsQuery.data.jobMetrics) as metric, i (metric || i)}
+            <td>
+            {#if metric != null}
+                <MetricPlot
+                    width={plotWidth}
+                    height={plotHeight}
+                    timestep={metric.metric.timestep}
+                    scope={metric.metric.scope}
+                    series={metric.metric.series}
+                    statisticsSeries={metric.metric.statisticsSeries}
+                    metric={metric.name}
+                    cluster={cluster}
+                    subCluster={job.subCluster} />
+            {:else}
+                <Card body color="warning">Missing Data</Card>
+            {/if}
+            </td>
+        {/each}
+    {/if}
+</tr>