ClusterCockpit/cc-backend
1
0
Fork 0
mirror of https://github.com/ClusterCockpit/cc-backend synced 2024-12-25 12:59:06 +01:00
Code Issues Packages Releases Activity

Merge pull request #16 from ClusterCockpit/refactor-structure-dev

Browse Source 
Refactor directory structure according to Golang common standards.
Add more documentation in form of READMEs.
This commit is contained in:
Jan Eitzinger 2022-06-24 10:41:57 +02:00 committed by GitHub
commit 293efefb98
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
130 changed files with 8235 additions and 548 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
.gitignore vendored
View File

@ -1,5 +1,4 @@
# executable:
cc-backend
/cc-backend
/var/job-archive
/var/*.db
@ -8,3 +7,5 @@ cc-backend
/.env
/config.json
/web/frontend/public/build
/web/frontend/node_modules

5
.gitmodules vendored
View File

@ -1,5 +0,0 @@
[submodule "frontend"]
path = frontend
url = git@github.com:ClusterCockpit/cc-frontend.git
branch = main
update = merge

96
README.md
View File

@ -3,21 +3,27 @@
[![Build](https://github.com/ClusterCockpit/cc-backend/actions/workflows/test.yml/badge.svg)](https://github.com/ClusterCockpit/cc-backend/actions/workflows/test.yml)
This is a Golang backend implementation for a REST and GraphQL API according to the [ClusterCockpit specifications](https://github.com/ClusterCockpit/cc-specifications).
It also includes a web interface for ClusterCockpit based on the components implemented in
[cc-frontend](https://github.com/ClusterCockpit/cc-frontend), which is included as a git submodule.
It also includes a web interface for ClusterCockpit.
This implementation replaces the previous PHP Symfony based ClusterCockpit web-interface.
[Here](https://github.com/ClusterCockpit/ClusterCockpit/wiki/Why-we-switched-from-PHP-Symfony-to-a-Golang-based-solution) is a discussion of the reasons why we switched from PHP Symfony to a Golang based solution.
## Overview
This is a golang web backend for the ClusterCockpit job-specific performance monitoring framework.
It provides a REST API for integrating ClusterCockpit with a HPC cluster batch system and external analysis scripts.
Data exchange between the web frontend and backend is based on a GraphQL API.
The web frontend is also served by the backend using [Svelte](https://svelte.dev/) components implemented in [cc-frontend](https://github.com/ClusterCockpit/cc-frontend).
The web frontend is also served by the backend using [Svelte](https://svelte.dev/) components.
Layout and styling is based on [Bootstrap 5](https://getbootstrap.com/) using [Bootstrap Icons](https://icons.getbootstrap.com/).
The backend uses [SQLite 3](https://sqlite.org/) as relational SQL database by default. It can optionally use a MySQL/MariaDB database server.
Finished batch jobs are stored in a so called job archive following [this specification](https://github.com/ClusterCockpit/cc-specifications/tree/master/job-archive).
The backend uses [SQLite 3](https://sqlite.org/) as relational SQL database by default.
It can optionally use a MySQL/MariaDB database server.
Finished batch jobs are stored in a file-based job archive following [this specification](https://github.com/ClusterCockpit/cc-specifications/tree/master/job-archive).
The backend supports authentication using local accounts or an external LDAP directory.
Authorization for APIs is implemented using [JWT](https://jwt.io/) tokens created with public/private key encryption.
Authorization for APIs is implemented using [JWT](https://jwt.io/) tokens created with public/private key encryption.
You find more detailed information here:
* `./configs/README.md`: Infos about configuration and setup of cc-backend.
* `./init/README.md`: Infos on how to setup cc-backend as systemd service on Linux.
* `./tools/README.md`: Infos on the JWT authorizatin token workflows in ClusterCockpit.
## Demo Setup
@ -25,30 +31,28 @@ We provide a shell skript that downloads demo data and automatically builds and
You need `wget`, `go`, and `yarn` in your path to start the demo. The demo will download 32MB of data (223MB on disk).
```sh
# The frontend is a submodule, so use `--recursive`
git clone --recursive git@github.com:ClusterCockpit/cc-backend.git
git clone git@github.com:ClusterCockpit/cc-backend.git
./startDemo.sh
```
You can access the web interface at http://localhost:8080. Credentials for login: `demo:AdminDev`. Please note that some views do not work without a metric backend (e.g., the Systems view).
You can access the web interface at http://localhost:8080.
Credentials for login: `demo:AdminDev`.
Please note that some views do not work without a metric backend (e.g., the Systems and Status view).
## Howto Build and Run
```sh
# The frontend is a submodule, so use `--recursive`
git clone --recursive git@github.com:ClusterCockpit/cc-backend.git
git clone git@github.com:ClusterCockpit/cc-backend.git
# Prepare frontend
cd ./cc-backend/frontend
cd ./cc-backend/web/frontend
yarn install
yarn build
cd ..
go get
go build
go build cmd/cc-backend
# The job-archive directory must be organised the same way as
# as for the regular ClusterCockpit.
ln -s <your-existing-job-archive> ./var/job-archive
# Create empty job.db (Will be initialized as SQLite3 database)
@ -56,6 +60,7 @@ touch ./var/job.db
# EDIT THE .env FILE BEFORE YOU DEPLOY (Change the secrets)!
# If authentication is disabled, it can be empty.
cp configs/env-template.txt .env
vim ./.env
# This will first initialize the job.db database by traversing all
@ -71,57 +76,40 @@ vim ./.env
```
### Run as systemd daemon
In order to run this program as a daemon, look at [utils/systemd/README.md](./utils/systemd/README.md) where a systemd unit file and more explanation is provided.
In order to run this program as a daemon, cc-backend ships with an [example systemd setup](./init/README.md).
## Configuration and Setup
cc-backend can be used as a local web-interface for an existing job archive or
as a general web-interface server for a live ClusterCockpit Monitoring
framework.
cc-backend can be used as a local web-interface for an existing job archive or as a general web-interface server for a live ClusterCockpit Monitoring framework.
Create your job-archive according to [this specification](https://github.com/ClusterCockpit/cc-specifications). At least
one cluster with a valid `cluster.json` file is required. Having no jobs in the
job-archive at all is fine. You may use the sample job-archive available for
download [in cc-docker/develop](https://github.com/ClusterCockpit/cc-docker/tree/develop).
Create your job-archive according to [this specification](https://github.com/ClusterCockpit/cc-specifications/tree/master/job-archive).
At least one cluster with a valid `cluster.json` file is required.
Having no jobs in the job-archive at all is fine.
### Configuration
A config file in the JSON format can be provided using `--config` to override the defaults.
Look at the beginning of `server.go` for the defaults and consequently the format of the configuration file.
You find documentation of all supported configuration and command line options [here](./configs.README.md).
### Update GraphQL schema
This project uses [gqlgen](https://github.com/99designs/gqlgen) for the GraphQL
API. The schema can be found in `./graph/schema.graphqls`. After changing it,
you need to run `go run github.com/99designs/gqlgen` which will update
`graph/model`. In case new resolvers are needed, they will be inserted into
`graph/schema.resolvers.go`, where you will need to implement them.
This project uses [gqlgen](https://github.com/99designs/gqlgen) for the GraphQL API.
The schema can be found in `./api/schema.graphqls`.
After changing it, you need to run `go run github.com/99designs/gqlgen` which will update `./internal/graph/model`.
In case new resolvers are needed, they will be inserted into `./internal/graph/schema.resolvers.go`, where you will need to implement them.
## Project Structure
- `api/` contains the REST API. The routes defined there should be called whenever a job starts/stops. The API is documented in the OpenAPI 3.0 format in [./api/openapi.yaml](./api/openapi.yaml).
- `auth/` is where the (optional) authentication middleware can be found, which adds the currently authenticated user to the request context. The `user` table is created and managed here as well.
- `auth/ldap.go` contains everything to do with automatically syncing and authenticating users form an LDAP server.
- `config` handles the `cluster.json` files and the user-specific configurations (changeable via GraphQL) for the Web-UI such as the selected metrics etc.
- `frontend` is a submodule, this is where the Svelte based frontend resides.
- `graph/generated` should *not* be touched.
- `graph/model` contains all types defined in the GraphQL schema not manually defined in `schema/`. Manually defined types have to be listed in `gqlgen.yml`.
- `graph/schema.graphqls` contains the GraphQL schema. Whenever you change it, you should call `go run github.com/99designs/gqlgen`.
- `graph/` contains the resolvers and handlers for the GraphQL API. Function signatures in `graph/schema.resolvers.go` are automatically generated.
- `metricdata/` handles getting and archiving the metrics associated with a job.
- `metricdata/metricdata.go` defines the interface `MetricDataRepository` and provides functions to the GraphQL and REST API for accessing a jobs metrics which automatically take care of selecting the source for the metrics (the archive or one of the metric data repositories).
- `metricdata/archive.go` provides functions for fetching metrics from the job-archive and archiving a job to the job-archive.
- `metricdata/cc-metric-store.go` contains an implementation of the `MetricDataRepository` interface which can fetch data from an [cc-metric-store](https://github.com/ClusterCockpit/cc-metric-store)
- `metricdata/influxdb-v2` contains an implementation of the `MetricDataRepository` interface which can fetch data from an InfluxDBv2 database. It is currently disabled and out of date and can not be used as of writing.
- `repository/` all SQL related stuff.
- `repository/init.go` initializes the `job` (and `tag` and `jobtag`) table if the `--init-db` flag is provided. Not only is the table created in the correct schema, but the job-archive is traversed as well.
- `schema/` contains type definitions used all over this project extracted in this package as Go disallows cyclic dependencies between packages.
- `schema/float.go` contains a custom `float64` type which overwrites JSON and GraphQL Marshaling/Unmarshalling. This is needed because a regular optional `Float` in GraphQL will map to `*float64` types in Go. Wrapping every single metric value in an allocation would be a lot of overhead.
- `schema/job.go` provides the types representing a job and its resources. Those can be used as type for a `meta.json` file and/or a row in the `job` table.
- `templates/` is mostly full of HTML templates and a small helper go module.
- `utils/systemd` describes how to deploy/install this as a systemd service
- `test/` rudimentery tests.
- `utils/`
- `.env` *must* be changed before you deploy this. It contains a Base64 encoded [Ed25519](https://en.wikipedia.org/wiki/EdDSA) key-pair, the secret used for sessions and the password to the LDAP server if LDAP authentication is enabled.
- `api/` contains the API schema files for the REST and GraphQL APIs. The REST API is documented in the OpenAPI 3.0 format in [./api/openapi.yaml](./api/openapi.yaml).
- `cmd/cc-backend` contains `main.go` for the main application.
- `configs/` contains documentation about configuration and command line options and required environment variables. An example configuration file is provided.
- `init/` contains an example systemd setup for production use.
- `internal/` contains library source code that is not intended to be used by others.
- `pkg/` contains go packages that can also be used by other projects.
- `test/` Test apps and test data.
- `tools/` contains supporting tools for cc-backend. At the moment this is a small application to generate a compatible JWT keypair includin a README about JWT setup in ClusterCockpit.
- `web/` Server side templates and frontend related files:
- `templates` Serverside go templates
- `frontend` Svelte components and static assets for frontend UI
- `gqlgen.yml` configures the behaviour and generation of [gqlgen](https://github.com/99designs/gqlgen).
- `server.go` contains the main function and starts the actual http server.
- `startDemo.sh` is a shell script that sets up demo data, and builds and starts cc-backend.

0
graph/schema.graphqls → api/schema.graphqls
View File

80
server.go → cmd/cc-backend/main.go
View File

@ -22,26 +22,25 @@ import (
"github.com/99designs/gqlgen/graphql/handler"
"github.com/99designs/gqlgen/graphql/playground"
"github.com/ClusterCockpit/cc-backend/api"
"github.com/ClusterCockpit/cc-backend/auth"
"github.com/ClusterCockpit/cc-backend/config"
"github.com/ClusterCockpit/cc-backend/graph"
"github.com/ClusterCockpit/cc-backend/graph/generated"
"github.com/ClusterCockpit/cc-backend/log"
"github.com/ClusterCockpit/cc-backend/metricdata"
"github.com/ClusterCockpit/cc-backend/repository"
"github.com/ClusterCockpit/cc-backend/templates"
"github.com/ClusterCockpit/cc-backend/internal/api"
"github.com/ClusterCockpit/cc-backend/internal/auth"
"github.com/ClusterCockpit/cc-backend/internal/config"
"github.com/ClusterCockpit/cc-backend/internal/graph"
"github.com/ClusterCockpit/cc-backend/internal/graph/generated"
"github.com/ClusterCockpit/cc-backend/internal/metricdata"
"github.com/ClusterCockpit/cc-backend/internal/repository"
"github.com/ClusterCockpit/cc-backend/internal/routerConfig"
"github.com/ClusterCockpit/cc-backend/internal/runtimeEnv"
"github.com/ClusterCockpit/cc-backend/internal/templates"
"github.com/ClusterCockpit/cc-backend/pkg/log"
"github.com/google/gops/agent"
"github.com/gorilla/handlers"
"github.com/gorilla/mux"
"github.com/jmoiron/sqlx"
_ "github.com/go-sql-driver/mysql"
_ "github.com/mattn/go-sqlite3"
)
var jobRepo *repository.JobRepository
// Format of the configurartion (file). See below for the defaults.
type ProgramConfig struct {
// Address where the http (or https) server will listen on (for example: 'localhost:80').
@ -70,7 +69,7 @@ type ProgramConfig struct {
// do not write to the job-archive.
DisableArchive bool `json:"disable-archive"`
// For LDAP Authentication and user syncronisation.
// For LDAP Authentication and user synchronisation.
LdapConfig *auth.LdapConfig `json:"ldap"`
// Specifies for how long a session or JWT shall be valid
@ -94,14 +93,14 @@ type ProgramConfig struct {
// Where to store MachineState files
MachineStateDir string `json:"machine-state-dir"`
// If not zero, automatically mark jobs as stopped running X seconds longer than theire walltime.
// If not zero, automatically mark jobs as stopped running X seconds longer than their walltime.
StopJobsExceedingWalltime int `json:"stop-jobs-exceeding-walltime"`
}
var programConfig ProgramConfig = ProgramConfig{
Addr: ":8080",
DisableAuthentication: false,
StaticFiles: "./frontend/public",
StaticFiles: "./web/frontend/public",
DBDriver: "sqlite3",
DB: "./var/job.db",
JobArchive: "./var/job-archive",
@ -119,7 +118,7 @@ var programConfig ProgramConfig = ProgramConfig{
"plot_general_colorscheme": []string{"#00bfff", "#0000ff", "#ff00ff", "#ff0000", "#ff8000", "#ffff00", "#80ff00"},
"plot_general_lineWidth": 3,
"plot_list_hideShortRunningJobs": 5 * 60,
"plot_list_jobsPerPage": 10,
"plot_list_jobsPerPage": 50,
"plot_list_selectedMetrics": []string{"cpu_load", "ipc", "mem_used", "flops_any", "mem_bw"},
"plot_view_plotsPerRow": 3,
"plot_view_showPolarplot": true,
@ -127,7 +126,7 @@ var programConfig ProgramConfig = ProgramConfig{
"plot_view_showStatTable": true,
"system_view_selectedMetric": "cpu_load",
},
StopJobsExceedingWalltime: -1,
StopJobsExceedingWalltime: 0,
}
func main() {
@ -152,7 +151,7 @@ func main() {
}
}
if err := loadEnv("./.env"); err != nil && !os.IsNotExist(err) {
if err := runtimeEnv.LoadEnv("./.env"); err != nil && !os.IsNotExist(err) {
log.Fatalf("parsing './.env' file failed: %s", err.Error())
}
@ -178,28 +177,8 @@ func main() {
}
var err error
var db *sqlx.DB
if programConfig.DBDriver == "sqlite3" {
db, err = sqlx.Open("sqlite3", fmt.Sprintf("%s?_foreign_keys=on", programConfig.DB))
if err != nil {
log.Fatal(err)
}
// sqlite does not multithread. Having more than one connection open would just mean
// waiting for locks.
db.SetMaxOpenConns(1)
} else if programConfig.DBDriver == "mysql" {
db, err = sqlx.Open("mysql", fmt.Sprintf("%s?multiStatements=true", programConfig.DB))
if err != nil {
log.Fatal(err)
}
db.SetConnMaxLifetime(time.Minute * 3)
db.SetMaxOpenConns(10)
db.SetMaxIdleConns(10)
} else {
log.Fatalf("unsupported database driver: %s", programConfig.DBDriver)
}
repository.Connect(programConfig.DBDriver, programConfig.DB)
db := repository.GetConnection()
// Initialize sub-modules and handle all command line flags.
// The order here is important! For example, the metricdata package
@ -215,7 +194,7 @@ func main() {
authentication.JwtMaxAge = d
}
if err := authentication.Init(db, programConfig.LdapConfig); err != nil {
if err := authentication.Init(db.DB, programConfig.LdapConfig); err != nil {
log.Fatal(err)
}
@ -257,7 +236,7 @@ func main() {
log.Fatal("arguments --add-user and --del-user can only be used if authentication is enabled")
}
if err := config.Init(db, !programConfig.DisableAuthentication, programConfig.UiDefaults, programConfig.JobArchive); err != nil {
if err := config.Init(db.DB, !programConfig.DisableAuthentication, programConfig.UiDefaults, programConfig.JobArchive); err != nil {
log.Fatal(err)
}
@ -266,15 +245,12 @@ func main() {
}
if flagReinitDB {
if err := repository.InitDB(db, programConfig.JobArchive); err != nil {
if err := repository.InitDB(db.DB, programConfig.JobArchive); err != nil {
log.Fatal(err)
}
}
jobRepo = &repository.JobRepository{DB: db}
if err := jobRepo.Init(); err != nil {
log.Fatal(err)
}
jobRepo := repository.GetRepository()
if flagImportJob != "" {
if err := jobRepo.HandleImportFlag(flagImportJob); err != nil {
@ -288,7 +264,7 @@ func main() {
// Setup the http.Handler/Router used by the server
resolver := &graph.Resolver{DB: db, Repo: jobRepo}
resolver := &graph.Resolver{DB: db.DB, Repo: jobRepo}
graphQLEndpoint := handler.NewDefaultServer(generated.NewExecutableSchema(generated.Config{Resolvers: resolver}))
if os.Getenv("DEBUG") != "1" {
// Having this handler means that a error message is returned via GraphQL instead of the connection simply beeing closed.
@ -394,7 +370,7 @@ func main() {
})
// Mount all /monitoring/... and /api/... routes.
setupRoutes(secured, routes)
routerConfig.SetupRoutes(secured)
api.MountRoutes(secured)
r.PathPrefix("/").Handler(http.FileServer(http.Dir(programConfig.StaticFiles)))
@ -461,7 +437,7 @@ func main() {
// Because this program will want to bind to a privileged port (like 80), the listener must
// be established first, then the user can be changed, and after that,
// the actuall http server can be started.
if err := dropPrivileges(); err != nil {
if err := runtimeEnv.DropPrivileges(programConfig.Group, programConfig.User); err != nil {
log.Fatalf("error while changing user: %s", err.Error())
}
@ -479,7 +455,7 @@ func main() {
go func() {
defer wg.Done()
<-sigs
systemdNotifiy(false, "shutting down")
runtimeEnv.SystemdNotifiy(false, "shutting down")
// First shut down the server gracefully (waiting for all ongoing requests)
server.Shutdown(context.Background())
@ -503,7 +479,7 @@ func main() {
if os.Getenv("GOGC") == "" {
debug.SetGCPercent(25)
}
systemdNotifiy(true, "running")
runtimeEnv.SystemdNotifiy(true, "running")
wg.Wait()
log.Print("Gracefull shutdown completed!")
}

56
configs/README.md Normal file
View File

@ -0,0 +1,56 @@
## Intro
cc-backend can be used without a configuration file. In this case the default
options documented below are used. To overwrite the defaults specify a json
config file location using the command line option `--config <filepath>`.
All security relevant configuration. e.g., keys and passwords, are set using environment variables. It is supported to specify these by means of an `.env` file located in the project root.
## Configuration Options
* `addr`: Type string. Address where the http (or https) server will listen on (for example: 'localhost:80'). Default `:8080`.
* `user`: Type string. Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.
* `group`: Type string. Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.
* `disable-authentication`: Type bool. Disable authentication (for everything: API, Web-UI, ...). Default `false`.
* `static-files`: Type string. Folder where static assets can be found, those will be served directly. Default `./frontend/public`.
* `db-driver`: Type string. 'sqlite3' or 'mysql' (mysql will work for mariadb as well). Default `sqlite3`.
* `db`: Type string. For sqlite3 a filename, for mysql a DSN in this format: https://github.com/go-sql-driver/mysql#dsn-data-source-name (Without query parameters!). Default: `./var/job.db`.
* `job-archive`: Type string. Path to the job-archive. Default: `./var/job-archive`.
* `disable-archive`: Type bool. Keep all metric data in the metric data repositories, do not write to the job-archive. Default `false`.
* `"session-max-age`: Type string. Specifies for how long a session shall be valid as a string parsable by time.ParseDuration(). If 0 or empty, the session/token does not expire! Default `168h`.
* `"jwt-max-age`: Type string. Specifies for how long a JWT token shall be valid as a string parsable by time.ParseDuration(). If 0 or empty, the session/token does not expire! Default `0`.
* `https-cert-file` and `https-key-file`: Type string. If both those options are not empty, use HTTPS using those certificates.
* `redirect-http-to`: Type string. If not the empty string and `addr` does not end in ":80", redirect every request incoming at port 80 to that url.
* `machine-state-dir`: Type string. Where to store MachineState files. TODO: Explain in more detail!
* `"stop-jobs-exceeding-walltime`: Type int. If not zero, automatically mark jobs as stopped running X seconds longer than their walltime. Only applies if walltime is set for job. Default `0`;
* `ldap`: Type object. For LDAP Authentication and user synchronisation. Default `nil`.
- `url`: Type string. URL of LDAP directory server.
- `user_base`: Type string. Base DN of user tree root.
- `search_dn`: Type string. DN for authenticating LDAP admin account with fgeneral read rights.
- `user_bind`: Type string. Expression used to authenticate users via LDAP bind. Must contain `uid={username}`.
- `user_filter`: Type string. Filter to extract users for syncing.
- `sync_interval`: Type string. Interval used for syncing local user table with LDAP directory. Parsed using time.ParseDuration.
- `sync_del_old_users`: Type bool. Delete obsolete users in database.
* `ui-defaults`: Type object. Default configuration for ui views. If overwriten, all options must be provided! Most options can be overwritten by the user via the web interface.
- `analysis_view_histogramMetrics`: Type string array. Metrics to show as job count histograms in analysis view. Default `["flops_any", "mem_bw", "mem_used"]`.
- `analysis_view_scatterPlotMetrics`: Type array of string array. Initial scatter plto configuration in analysis view. Default `[["flops_any", "mem_bw"], ["flops_any", "cpu_load"], ["cpu_load", "mem_bw"]]`.
- `job_view_nodestats_selectedMetrics`: Type string array. Initial metrics shown in node statistics table of single job view. Default `["flops_any", "mem_bw", "mem_used"]`.
- `job_view_polarPlotMetrics`: Type string array. Metrics shown in polar plot of single job view. Default `["flops_any", "mem_bw", "mem_used", "net_bw", "file_bw"]`.
- `job_view_selectedMetrics`: Type string array. ??. Default `["flops_any", "mem_bw", "mem_used"]`.
- `plot_general_colorBackground`: Type bool. Color plot background according to job average threshold limits. Default `true`.
- `plot_general_colorscheme`: Type string array. Initial color scheme. Default `"#00bfff", "#0000ff", "#ff00ff", "#ff0000", "#ff8000", "#ffff00", "#80ff00"`.
- `plot_general_lineWidth`: Type int. Initial linewidth. Default `3`.
- `plot_list_hideShortRunningJobs`: Type int. Do not show running jobs shorter than X seconds. Default `300`.
- `plot_list_jobsPerPage`: Type int. Jobs shown per page in job lists. Default `50`.
- `plot_list_selectedMetrics`: Type string array. Initial metric plots shown in jobs lists. Default `"cpu_load", "ipc", "mem_used", "flops_any", "mem_bw"`.
- `plot_view_plotsPerRow`: Type int. Number of plots per row in single job view. Default `3`.
- `plot_view_showPolarplot`: Type bool. Option to toggle polar plot in single job view. Default `true`.
- `plot_view_showRoofline`: Type bool. Option to toggle roofline plot in single job view. Default `true`.
- `plot_view_showStatTable`: Type bool. Option to toggle the node statistic table in single job view. Default `true`.
- `system_view_selectedMetric`: Type string. Initial metric shown in system view. Default `cpu_load`.
## Environment Variables
An example env file is found in this directory. Copy it to `.env` in the project root and adapt it for your needs.
* `JWT_PUBLIC_KEY` and `JWT_PRIVATE_KEY`: Base64 encoded Ed25519 keys used for JSON Web Token (JWT) authentication . TODO: Details! You can generate your own keypair using `go run utils/gen-keypair.go`
* `SESSION_KEY`: Some random bytes used as secret for cookie-based sessions.
* `LDAP_ADMIN_PASSWORD`: The LDAP admin user password (optional).

14
configs/config.json Normal file
View File

@ -0,0 +1,14 @@
{
"addr": "0.0.0.0:443",
"ldap": {
"url": "ldaps://hpcldap.rrze.uni-erlangen.de",
"user_base": "ou=people,ou=hpc,dc=rrze,dc=uni-erlangen,dc=de",
"search_dn": "cn=hpcmonitoring,ou=roadm,ou=profile,ou=hpc,dc=rrze,dc=uni-erlangen,dc=de",
"user_bind": "uid={username},ou=people,ou=hpc,dc=rrze,dc=uni-erlangen,dc=de",
"user_filter": "(&(objectclass=posixAccount)(uid=*))"
},
"https-cert-file": "/etc/letsencrypt/live/monitoring.nhr.fau.de/fullchain.pem",
"https-key-file": "/etc/letsencrypt/live/monitoring.nhr.fau.de/privkey.pem",
"user": "clustercockpit",
"group": "clustercockpit"
}

0
.env → configs/env-template.txt
View File

1
frontend

@ -1 +0,0 @@
Subproject commit 94ef11aa9fc3c194f1df497e3e06c60a7125883d

1
go.mod
View File

@ -12,7 +12,6 @@ require (
github.com/gorilla/handlers v1.5.1
github.com/gorilla/mux v1.8.0
github.com/gorilla/sessions v1.2.1
github.com/iamlouk/lrucache v0.2.1
github.com/influxdata/influxdb-client-go/v2 v2.8.1
github.com/jmoiron/sqlx v1.3.4
github.com/mattn/go-sqlite3 v1.14.12

8
gqlgen.yml
View File

@ -1,10 +1,10 @@
# Where are all the schema files located? globs are supported eg src/**/*.graphqls
schema:
- graph/*.graphqls
- api/*.graphqls
# Where should the generated server code go?
exec:
filename: graph/generated/generated.go
filename: internal/graph/generated/generated.go
package: generated
# Uncomment to enable federation
@ -14,7 +14,7 @@ exec:
# Where should any generated models go?
model:
filename: graph/model/models_gen.go
filename: internal/graph/model/models_gen.go
package: model
# Where should the resolver implementations go?
@ -75,5 +75,3 @@ models:
Series: { model: "github.com/ClusterCockpit/cc-backend/schema.Series" }
MetricStatistics: { model: "github.com/ClusterCockpit/cc-backend/schema.MetricStatistics" }
StatsSeries: { model: "github.com/ClusterCockpit/cc-backend/schema.StatsSeries" }

38
init/README.md Normal file
View File

@ -0,0 +1,38 @@
# How to run this as a systemd deamon
The files in this directory assume that you install ClusterCockpit to `/opt/monitoring`.
Of course you can choose any other location, but make sure to replace all paths that begin with `/opt/monitoring` in the `clustercockpit.service` file!
If you have not installed [yarn](https://yarnpkg.com/getting-started/install) and [go](https://go.dev/doc/install) already, do that (Golang is available in most package managers).
It is recommended and easy to install the most recent stable version of Golang as every version also improves the Golang standard library.
The `config.json` can have the optional fields *user* and *group*.
If provided, the application will call [setuid](https://man7.org/linux/man-pages/man2/setuid.2.html) and [setgid](https://man7.org/linux/man-pages/man2/setgid.2.html) after having read the config file and having bound to a TCP port (so that it can take a privileged port), but before it starts accepting any connections.
This is good for security, but means that the directories `web/frontend/public`, `var/` and `web/templates/` must be readable by that user and `var/` writable as well (All paths relative to the repos root).
The `.env` and `config.json` files might contain secrets and should not be readable by that user.
If those files are changed, the server has to be restarted.
```sh
# 1.: Clone this repository to /opt/monitoring
git clone git@github.com:ClusterCockpit/cc-backend.git /opt/monitoring
# 2.: Install all dependencies and build everything
cd /mnt/monitoring
go get && go build cmd/cc-backend && (cd ./web/frontend && yarn install && yarn build)
# 3.: Modify the `./config.json` and env-template.txt file from the configs directory to your liking and put it in the repo root
cp ./configs/config.json ./config.json
cp ./configs/env-template.txt ./.env
vim ./config.json # do your thing...
vim ./.env # do your thing...
# 4.: Add the systemd service unit file (in case /opt/ is mounted on another file system it may be better to copy the file to /etc)
sudo ln -s /mnt/monitoring/init/clustercockpit.service /etc/systemd/system/clustercockpit.service
# 5.: Enable and start the server
sudo systemctl enable clustercockpit.service # optional (if done, (re-)starts automatically)
sudo systemctl start clustercockpit.service
# Check whats going on:
sudo journalctl -u clustercockpit.service
```

18
init/clustercockpit.service Normal file
View File

@ -0,0 +1,18 @@
[Unit]
Description=ClusterCockpit Web Server (Go edition)
Documentation=https://github.com/ClusterCockpit/cc-backend
Wants=network-online.target
After=network-online.target
After=mariadb.service mysql.service
[Service]
WorkingDirectory=/opt/monitoring/cc-backend
Type=notify
NotifyAccess=all
Restart=on-failure
RestartSec=30
TimeoutStopSec=100
ExecStart=/opt/monitoring/cc-backend/cc-backend --config ./config.json
[Install]
WantedBy=multi-user.target

16
api/rest.go → internal/api/rest.go
View File

@ -16,14 +16,14 @@ import (
"sync"
"time"
"github.com/ClusterCockpit/cc-backend/auth"
"github.com/ClusterCockpit/cc-backend/config"
"github.com/ClusterCockpit/cc-backend/graph"
"github.com/ClusterCockpit/cc-backend/graph/model"
"github.com/ClusterCockpit/cc-backend/log"
"github.com/ClusterCockpit/cc-backend/metricdata"
"github.com/ClusterCockpit/cc-backend/repository"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/internal/auth"
"github.com/ClusterCockpit/cc-backend/internal/config"
"github.com/ClusterCockpit/cc-backend/internal/graph"
"github.com/ClusterCockpit/cc-backend/internal/graph/model"
"github.com/ClusterCockpit/cc-backend/internal/metricdata"
"github.com/ClusterCockpit/cc-backend/internal/repository"
"github.com/ClusterCockpit/cc-backend/pkg/log"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
"github.com/gorilla/mux"
)

4
auth/auth.go → internal/auth/auth.go
View File

@ -14,8 +14,8 @@ import (
"strings"
"time"
"github.com/ClusterCockpit/cc-backend/graph/model"
"github.com/ClusterCockpit/cc-backend/log"
"github.com/ClusterCockpit/cc-backend/internal/graph/model"
"github.com/ClusterCockpit/cc-backend/pkg/log"
sq "github.com/Masterminds/squirrel"
"github.com/golang-jwt/jwt/v4"
"github.com/gorilla/sessions"

3
auth/ldap.go → internal/auth/ldap.go
View File

@ -6,8 +6,7 @@ import (
"strings"
"time"
"github.com/ClusterCockpit/cc-backend/log"
"github.com/ClusterCockpit/cc-backend/pkg/log"
"github.com/go-ldap/ldap/v3"
)

8
config/config.go → internal/config/config.go
View File

@ -11,10 +11,10 @@ import (
"sync"
"time"
"github.com/ClusterCockpit/cc-backend/auth"
"github.com/ClusterCockpit/cc-backend/graph/model"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/iamlouk/lrucache"
"github.com/ClusterCockpit/cc-backend/internal/auth"
"github.com/ClusterCockpit/cc-backend/internal/graph/model"
"github.com/ClusterCockpit/cc-backend/pkg/lrucache"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
"github.com/jmoiron/sqlx"
)

2
config/nodelist.go → internal/config/nodelist.go
View File

@ -5,7 +5,7 @@ import (
"strconv"
"strings"
"github.com/ClusterCockpit/cc-backend/log"
"github.com/ClusterCockpit/cc-backend/pkg/log"
)
type NodeList [][]interface {

0
config/nodelist_test.go → internal/config/nodelist_test.go
View File

4
graph/generated/generated.go → internal/graph/generated/generated.go
View File

@ -13,8 +13,8 @@ import (
"github.com/99designs/gqlgen/graphql"
"github.com/99designs/gqlgen/graphql/introspection"
"github.com/ClusterCockpit/cc-backend/graph/model"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/internal/graph/model"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
gqlparser "github.com/vektah/gqlparser/v2"
"github.com/vektah/gqlparser/v2/ast"
)

0
graph/model/models.go → internal/graph/model/models.go
View File

2
graph/model/models_gen.go → internal/graph/model/models_gen.go
View File

@ -8,7 +8,7 @@ import (
"strconv"
"time"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
)
type Accelerator struct {

2
graph/resolver.go → internal/graph/resolver.go
View File

@ -1,7 +1,7 @@
package graph
import (
"github.com/ClusterCockpit/cc-backend/repository"
"github.com/ClusterCockpit/cc-backend/internal/repository"
"github.com/jmoiron/sqlx"
)

12
graph/schema.resolvers.go → internal/graph/schema.resolvers.go
View File

@ -10,12 +10,12 @@ import (
"strconv"
"time"
"github.com/ClusterCockpit/cc-backend/auth"
"github.com/ClusterCockpit/cc-backend/config"
"github.com/ClusterCockpit/cc-backend/graph/generated"
"github.com/ClusterCockpit/cc-backend/graph/model"
"github.com/ClusterCockpit/cc-backend/metricdata"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/internal/auth"
"github.com/ClusterCockpit/cc-backend/internal/config"
"github.com/ClusterCockpit/cc-backend/internal/graph/generated"
"github.com/ClusterCockpit/cc-backend/internal/graph/model"
"github.com/ClusterCockpit/cc-backend/internal/metricdata"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
)
func (r *clusterResolver) Partitions(ctx context.Context, obj *model.Cluster) ([]string, error) {

10
graph/stats.go → internal/graph/stats.go
View File

@ -9,11 +9,11 @@ import (
"time"
"github.com/99designs/gqlgen/graphql"
"github.com/ClusterCockpit/cc-backend/config"
"github.com/ClusterCockpit/cc-backend/graph/model"
"github.com/ClusterCockpit/cc-backend/metricdata"
"github.com/ClusterCockpit/cc-backend/repository"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/internal/config"
"github.com/ClusterCockpit/cc-backend/internal/graph/model"
"github.com/ClusterCockpit/cc-backend/internal/metricdata"
"github.com/ClusterCockpit/cc-backend/internal/repository"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
sq "github.com/Masterminds/squirrel"
)

4
metricdata/archive.go → internal/metricdata/archive.go
View File

@ -13,8 +13,8 @@ import (
"strconv"
"time"
"github.com/ClusterCockpit/cc-backend/config"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/internal/config"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
)
// For a given job, return the path of the `data.json`/`meta.json` file.

4
metricdata/cc-metric-store.go → internal/metricdata/cc-metric-store.go
View File

@ -11,8 +11,8 @@ import (
"strings"
"time"
"github.com/ClusterCockpit/cc-backend/config"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/internal/config"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
)
type CCMetricStoreConfig struct {

308
internal/metricdata/influxdb-v2.go Normal file
View File

@ -0,0 +1,308 @@
package metricdata
import (
"context"
"crypto/tls"
"encoding/json"
"errors"
"fmt"
"log"
"strings"
"time"
"github.com/ClusterCockpit/cc-backend/internal/config"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
influxdb2 "github.com/influxdata/influxdb-client-go/v2"
influxdb2Api "github.com/influxdata/influxdb-client-go/v2/api"
)
type InfluxDBv2DataRepositoryConfig struct {
Url string `json:"url"`
Token string `json:"token"`
Bucket string `json:"bucket"`
Org string `json:"org"`
SkipTls bool `json:"skiptls"`
}
type InfluxDBv2DataRepository struct {
client influxdb2.Client
queryClient influxdb2Api.QueryAPI
bucket, measurement string
}
func (idb *InfluxDBv2DataRepository) Init(rawConfig json.RawMessage) error {
var config InfluxDBv2DataRepositoryConfig
if err := json.Unmarshal(rawConfig, &config); err != nil {
return err
}
idb.client = influxdb2.NewClientWithOptions(config.Url, config.Token, influxdb2.DefaultOptions().SetTLSConfig(&tls.Config{InsecureSkipVerify: config.SkipTls}))
idb.queryClient = idb.client.QueryAPI(config.Org)
idb.bucket = config.Bucket
return nil
}
func (idb *InfluxDBv2DataRepository) formatTime(t time.Time) string {
return t.Format(time.RFC3339) // Like “2006-01-02T15:04:05Z07:00”
}
func (idb *InfluxDBv2DataRepository) epochToTime(epoch int64) time.Time {
return time.Unix(epoch, 0)
}
func (idb *InfluxDBv2DataRepository) LoadData(job *schema.Job, metrics []string, scopes []schema.MetricScope, ctx context.Context) (schema.JobData, error) {
measurementsConds := make([]string, 0, len(metrics))
for _, m := range metrics {
measurementsConds = append(measurementsConds, fmt.Sprintf(`r["_measurement"] == "%s"`, m))
}
measurementsCond := strings.Join(measurementsConds, " or ")
hostsConds := make([]string, 0, len(job.Resources))
for _, h := range job.Resources {
if h.HWThreads != nil || h.Accelerators != nil {
// TODO
return nil, errors.New("the InfluxDB metric data repository does not yet support HWThreads or Accelerators")
}
hostsConds = append(hostsConds, fmt.Sprintf(`r["hostname"] == "%s"`, h.Hostname))
}
hostsCond := strings.Join(hostsConds, " or ")
jobData := make(schema.JobData) // Empty Schema: map[<string>FIELD]map[<MetricScope>SCOPE]<*JobMetric>METRIC
// Requested Scopes
for _, scope := range scopes {
query := ""
switch scope {
case "node":
// Get Finest Granularity, Groupy By Measurement and Hostname (== Metric / Node), Calculate Mean for 60s windows
// log.Println("Note: Scope 'node' requested. ")
query = fmt.Sprintf(`
from(bucket: "%s")
|> range(start: %s, stop: %s)
|> filter(fn: (r) => (%s) and (%s) )
|> drop(columns: ["_start", "_stop"])
|> group(columns: ["hostname", "_measurement"])
|> aggregateWindow(every: 60s, fn: mean)
|> drop(columns: ["_time"])`,
idb.bucket,
idb.formatTime(job.StartTime), idb.formatTime(idb.epochToTime(job.StartTimeUnix+int64(job.Duration)+int64(1))),
measurementsCond, hostsCond)
case "socket":
log.Println("Note: Scope 'socket' requested, but not yet supported: Will return 'node' scope only. ")
continue
case "core":
log.Println("Note: Scope 'core' requested, but not yet supported: Will return 'node' scope only. ")
continue
// Get Finest Granularity only, Set NULL to 0.0
// query = fmt.Sprintf(`
// from(bucket: "%s")
// |> range(start: %s, stop: %s)
// |> filter(fn: (r) => %s )
// |> filter(fn: (r) => %s )
// |> drop(columns: ["_start", "_stop", "cluster"])
// |> map(fn: (r) => (if exists r._value then {r with _value: r._value} else {r with _value: 0.0}))`,
// idb.bucket,
// idb.formatTime(job.StartTime), idb.formatTime(idb.epochToTime(job.StartTimeUnix + int64(job.Duration) + int64(1) )),
// measurementsCond, hostsCond)
default:
log.Println("Note: Unknown Scope requested: Will return 'node' scope. ")
continue
// return nil, errors.New("the InfluxDB metric data repository does not yet support other scopes than 'node'")
}
rows, err := idb.queryClient.Query(ctx, query)
if err != nil {
return nil, err
}
// Init Metrics: Only Node level now -> TODO: Matching /check on scope level ...
for _, metric := range metrics {
jobMetric, ok := jobData[metric]
if !ok {
mc := config.GetMetricConfig(job.Cluster, metric)
jobMetric = map[schema.MetricScope]*schema.JobMetric{
scope: { // uses scope var from above!
Unit: mc.Unit,
Scope: scope,
Timestep: mc.Timestep,
Series: make([]schema.Series, 0, len(job.Resources)),
StatisticsSeries: nil, // Should be: &schema.StatsSeries{},
},
}
}
jobData[metric] = jobMetric
}
// Process Result: Time-Data
field, host, hostSeries := "", "", schema.Series{}
// typeId := 0
switch scope {
case "node":
for rows.Next() {
row := rows.Record()
if host == "" || host != row.ValueByKey("hostname").(string) || rows.TableChanged() {
if host != "" {
// Append Series before reset
jobData[field][scope].Series = append(jobData[field][scope].Series, hostSeries)
}
field, host = row.Measurement(), row.ValueByKey("hostname").(string)
hostSeries = schema.Series{
Hostname: host,
Statistics: nil,
Data: make([]schema.Float, 0),
}
}
val, ok := row.Value().(float64)
if ok {
hostSeries.Data = append(hostSeries.Data, schema.Float(val))
} else {
hostSeries.Data = append(hostSeries.Data, schema.Float(0))
}
}
case "socket":
continue
case "core":
continue
// Include Series.Id in hostSeries
// for rows.Next() {
// row := rows.Record()
// if ( host == "" || host != row.ValueByKey("hostname").(string) || typeId != row.ValueByKey("type-id").(int) || rows.TableChanged() ) {
// if ( host != "" ) {
// // Append Series before reset
// jobData[field][scope].Series = append(jobData[field][scope].Series, hostSeries)
// }
// field, host, typeId = row.Measurement(), row.ValueByKey("hostname").(string), row.ValueByKey("type-id").(int)
// hostSeries = schema.Series{
// Hostname: host,
// Id: &typeId,
// Statistics: nil,
// Data: make([]schema.Float, 0),
// }
// }
// val := row.Value().(float64)
// hostSeries.Data = append(hostSeries.Data, schema.Float(val))
// }
default:
continue
// return nil, errors.New("the InfluxDB metric data repository does not yet support other scopes than 'node, core'")
}
// Append last Series
jobData[field][scope].Series = append(jobData[field][scope].Series, hostSeries)
}
// Get Stats
stats, err := idb.LoadStats(job, metrics, ctx)
if err != nil {
return nil, err
}
for _, scope := range scopes {
if scope == "node" { // No 'socket/core' support yet
for metric, nodes := range stats {
// log.Println(fmt.Sprintf("<< Add Stats for : Field %s >>", metric))
for node, stats := range nodes {
// log.Println(fmt.Sprintf("<< Add Stats for : Host %s : Min %.2f, Max %.2f, Avg %.2f >>", node, stats.Min, stats.Max, stats.Avg ))
for index, _ := range jobData[metric][scope].Series {
// log.Println(fmt.Sprintf("<< Try to add Stats to Series in Position %d >>", index))
if jobData[metric][scope].Series[index].Hostname == node {
// log.Println(fmt.Sprintf("<< Match for Series in Position %d : Host %s >>", index, jobData[metric][scope].Series[index].Hostname))
jobData[metric][scope].Series[index].Statistics = &schema.MetricStatistics{Avg: stats.Avg, Min: stats.Min, Max: stats.Max}
// log.Println(fmt.Sprintf("<< Result Inner: Min %.2f, Max %.2f, Avg %.2f >>", jobData[metric][scope].Series[index].Statistics.Min, jobData[metric][scope].Series[index].Statistics.Max, jobData[metric][scope].Series[index].Statistics.Avg))
}
}
}
}
}
}
// DEBUG:
// for _, scope := range scopes {
// for _, met := range metrics {
// for _, series := range jobData[met][scope].Series {
// log.Println(fmt.Sprintf("<< Result: %d data points for metric %s on %s with scope %s, Stats: Min %.2f, Max %.2f, Avg %.2f >>",
// len(series.Data), met, series.Hostname, scope,
// series.Statistics.Min, series.Statistics.Max, series.Statistics.Avg))
// }
// }
// }
return jobData, nil
}
func (idb *InfluxDBv2DataRepository) LoadStats(job *schema.Job, metrics []string, ctx context.Context) (map[string]map[string]schema.MetricStatistics, error) {
stats := map[string]map[string]schema.MetricStatistics{}
hostsConds := make([]string, 0, len(job.Resources))
for _, h := range job.Resources {
if h.HWThreads != nil || h.Accelerators != nil {
// TODO
return nil, errors.New("the InfluxDB metric data repository does not yet support HWThreads or Accelerators")
}
hostsConds = append(hostsConds, fmt.Sprintf(`r["hostname"] == "%s"`, h.Hostname))
}
hostsCond := strings.Join(hostsConds, " or ")
// lenMet := len(metrics)
for _, metric := range metrics {
// log.Println(fmt.Sprintf("<< You are here: %s (Index %d of %d metrics)", metric, index, lenMet))
query := fmt.Sprintf(`
data = from(bucket: "%s")
|> range(start: %s, stop: %s)
|> filter(fn: (r) => r._measurement == "%s" and r._field == "value" and (%s))
union(tables: [data |> mean(column: "_value") |> set(key: "_field", value: "avg"),
data |> min(column: "_value") |> set(key: "_field", value: "min"),
data |> max(column: "_value") |> set(key: "_field", value: "max")])
|> pivot(rowKey: ["hostname"], columnKey: ["_field"], valueColumn: "_value")
|> group()`,
idb.bucket,
idb.formatTime(job.StartTime), idb.formatTime(idb.epochToTime(job.StartTimeUnix+int64(job.Duration)+int64(1))),
metric, hostsCond)
rows, err := idb.queryClient.Query(ctx, query)
if err != nil {
return nil, err
}
nodes := map[string]schema.MetricStatistics{}
for rows.Next() {
row := rows.Record()
host := row.ValueByKey("hostname").(string)
avg, avgok := row.ValueByKey("avg").(float64)
if !avgok {
// log.Println(fmt.Sprintf(">> Assertion error for metric %s, statistic AVG. Expected 'float64', got %v", metric, avg))
avg = 0.0
}
min, minok := row.ValueByKey("min").(float64)
if !minok {
// log.Println(fmt.Sprintf(">> Assertion error for metric %s, statistic MIN. Expected 'float64', got %v", metric, min))
min = 0.0
}
max, maxok := row.ValueByKey("max").(float64)
if !maxok {
// log.Println(fmt.Sprintf(">> Assertion error for metric %s, statistic MAX. Expected 'float64', got %v", metric, max))
max = 0.0
}
nodes[host] = schema.MetricStatistics{
Avg: avg,
Min: min,
Max: max,
}
}
stats[metric] = nodes
}
return stats, nil
}
func (idb *InfluxDBv2DataRepository) LoadNodeData(cluster string, metrics, nodes []string, scopes []schema.MetricScope, from, to time.Time, ctx context.Context) (map[string]map[string][]*schema.JobMetric, error) {
// TODO : Implement to be used in Analysis- und System/Node-View
log.Println(fmt.Sprintf("LoadNodeData unimplemented for InfluxDBv2DataRepository, Args: cluster %s, metrics %v, nodes %v, scopes %v", cluster, metrics, nodes, scopes))
return nil, errors.New("unimplemented for InfluxDBv2DataRepository")
}

8
metricdata/metricdata.go → internal/metricdata/metricdata.go
View File

@ -6,10 +6,10 @@ import (
"fmt"
"time"
"github.com/ClusterCockpit/cc-backend/config"
"github.com/ClusterCockpit/cc-backend/log"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/iamlouk/lrucache"
"github.com/ClusterCockpit/cc-backend/internal/config"
"github.com/ClusterCockpit/cc-backend/pkg/log"
"github.com/ClusterCockpit/cc-backend/pkg/lrucache"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
)
type MetricDataRepository interface {

2
metricdata/utils.go → internal/metricdata/utils.go
View File

@ -5,7 +5,7 @@ import (
"encoding/json"
"time"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
)
var TestLoadDataCallback func(job *schema.Job, metrics []string, scopes []schema.MetricScope, ctx context.Context) (schema.JobData, error) = func(job *schema.Job, metrics []string, scopes []schema.MetricScope, ctx context.Context) (schema.JobData, error) {

58
internal/repository/dbConnection.go Normal file
View File

@ -0,0 +1,58 @@
package repository
import (
"fmt"
"log"
"sync"
"time"
"github.com/jmoiron/sqlx"
)
var (
dbConnOnce sync.Once
dbConnInstance *DBConnection
)
type DBConnection struct {
DB *sqlx.DB
}
func Connect(driver string, db string) {
var err error
var dbHandle *sqlx.DB
dbConnOnce.Do(func() {
if driver == "sqlite3" {
dbHandle, err = sqlx.Open("sqlite3", fmt.Sprintf("%s?_foreign_keys=on", db))
if err != nil {
log.Fatal(err)
}
// sqlite does not multithread. Having more than one connection open would just mean
// waiting for locks.
dbHandle.SetMaxOpenConns(1)
} else if driver == "mysql" {
dbHandle, err = sqlx.Open("mysql", fmt.Sprintf("%s?multiStatements=true", db))
if err != nil {
log.Fatal(err)
}
dbHandle.SetConnMaxLifetime(time.Minute * 3)
dbHandle.SetMaxOpenConns(10)
dbHandle.SetMaxIdleConns(10)
} else {
log.Fatalf("unsupported database driver: %s", driver)
}
dbConnInstance = &DBConnection{DB: dbHandle}
})
}
func GetConnection() *DBConnection {
if dbConnInstance == nil {
log.Fatalf("Database connection not initialized!")
}
return dbConnInstance
}

8
repository/import.go → internal/repository/import.go
View File

@ -9,10 +9,10 @@ import (
"strings"
"time"
"github.com/ClusterCockpit/cc-backend/config"
"github.com/ClusterCockpit/cc-backend/log"
"github.com/ClusterCockpit/cc-backend/metricdata"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/internal/config"
"github.com/ClusterCockpit/cc-backend/internal/metricdata"
"github.com/ClusterCockpit/cc-backend/pkg/log"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
)
const NamedJobInsert string = `INSERT INTO job (

4
repository/init.go → internal/repository/init.go
View File

@ -8,8 +8,8 @@ import (
"path/filepath"
"time"
"github.com/ClusterCockpit/cc-backend/log"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/pkg/log"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
"github.com/jmoiron/sqlx"
)

32
repository/job.go → internal/repository/job.go
View File

@ -7,17 +7,23 @@ import (
"errors"
"fmt"
"strconv"
"sync"
"time"
"github.com/ClusterCockpit/cc-backend/auth"
"github.com/ClusterCockpit/cc-backend/graph/model"
"github.com/ClusterCockpit/cc-backend/log"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/internal/auth"
"github.com/ClusterCockpit/cc-backend/internal/graph/model"
"github.com/ClusterCockpit/cc-backend/pkg/log"
"github.com/ClusterCockpit/cc-backend/pkg/lrucache"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
sq "github.com/Masterminds/squirrel"
"github.com/iamlouk/lrucache"
"github.com/jmoiron/sqlx"
)
var (
jobRepoOnce sync.Once
jobRepoInstance *JobRepository
)
type JobRepository struct {
DB *sqlx.DB
@ -25,10 +31,18 @@ type JobRepository struct {
cache *lrucache.Cache
}
func (r *JobRepository) Init() error {
r.stmtCache = sq.NewStmtCache(r.DB)
r.cache = lrucache.New(1024 * 1024)
return nil
func GetRepository() *JobRepository {
jobRepoOnce.Do(func() {
db := GetConnection()
jobRepoInstance = &JobRepository{
DB: db.DB,
stmtCache: sq.NewStmtCache(db.DB),
cache: lrucache.New(1024 * 1024),
}
})
return jobRepoInstance
}
var jobColumns []string = []string{

15
repository/job_test.go → internal/repository/job_test.go
View File

@ -11,22 +11,11 @@ import (
var db *sqlx.DB
func init() {
var err error
db, err = sqlx.Open("sqlite3", "../test/test.db")
if err != nil {
fmt.Println(err)
}
Connect("sqlite3", "../../test/test.db")
}
func setup(t *testing.T) *JobRepository {
r := &JobRepository{
DB: db,
}
if err := r.Init(); err != nil {
t.Fatal(err)
}
return r
return GetRepository()
}
func TestFind(t *testing.T) {

8
repository/query.go → internal/repository/query.go
View File

@ -8,10 +8,10 @@ import (
"strings"
"time"
"github.com/ClusterCockpit/cc-backend/auth"
"github.com/ClusterCockpit/cc-backend/graph/model"
"github.com/ClusterCockpit/cc-backend/log"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/internal/auth"
"github.com/ClusterCockpit/cc-backend/internal/graph/model"
"github.com/ClusterCockpit/cc-backend/pkg/log"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
sq "github.com/Masterminds/squirrel"
)

4
repository/tags.go → internal/repository/tags.go
View File

@ -1,8 +1,8 @@
package repository
import (
"github.com/ClusterCockpit/cc-backend/metricdata"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/internal/metricdata"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
sq "github.com/Masterminds/squirrel"
)

22
routes.go → internal/routerConfig/routes.go
View File

@ -1,4 +1,4 @@
package main
package routerConfig
import (
"fmt"
@ -8,13 +8,14 @@ import (
"strings"
"time"
"github.com/ClusterCockpit/cc-backend/auth"
"github.com/ClusterCockpit/cc-backend/config"
"github.com/ClusterCockpit/cc-backend/graph"
"github.com/ClusterCockpit/cc-backend/graph/model"
"github.com/ClusterCockpit/cc-backend/log"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/templates"
"github.com/ClusterCockpit/cc-backend/internal/auth"
"github.com/ClusterCockpit/cc-backend/internal/config"
"github.com/ClusterCockpit/cc-backend/internal/graph"
"github.com/ClusterCockpit/cc-backend/internal/graph/model"
"github.com/ClusterCockpit/cc-backend/internal/repository"
"github.com/ClusterCockpit/cc-backend/internal/templates"
"github.com/ClusterCockpit/cc-backend/pkg/log"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
"github.com/gorilla/mux"
)
@ -50,6 +51,7 @@ func setupHomeRoute(i InfoType, r *http.Request) InfoType {
TotalJobs int
RecentShortJobs int
}
jobRepo := repository.GetRepository()
runningJobs, err := jobRepo.CountGroupedJobs(r.Context(), model.AggregateCluster, []*model.JobFilter{{
State: []schema.JobState{schema.JobStateRunning},
@ -93,6 +95,7 @@ func setupJobRoute(i InfoType, r *http.Request) InfoType {
}
func setupUserRoute(i InfoType, r *http.Request) InfoType {
jobRepo := repository.GetRepository()
username := mux.Vars(r)["id"]
i["id"] = username
i["username"] = username
@ -135,6 +138,7 @@ func setupAnalysisRoute(i InfoType, r *http.Request) InfoType {
func setupTaglistRoute(i InfoType, r *http.Request) InfoType {
var username *string = nil
jobRepo := repository.GetRepository()
if user := auth.GetUser(r.Context()); user != nil && !user.HasRole(auth.RoleAdmin) {
username = &user.Username
}
@ -245,7 +249,7 @@ func buildFilterPresets(query url.Values) map[string]interface{} {
return filterPresets
}
func setupRoutes(router *mux.Router, routes []Route) {
func SetupRoutes(router *mux.Router) {
for _, route := range routes {
route := route
router.HandleFunc(route.Route, func(rw http.ResponseWriter, r *http.Request) {

16
runtimeSetup.go → internal/runtimeEnv/setup.go
View File

@ -1,4 +1,4 @@
package main
package runtimeEnv
import (
"bufio"
@ -15,7 +15,7 @@ import (
// Very simple and limited .env file reader.
// All variable definitions found are directly
// added to the processes environment.
func loadEnv(file string) error {
func LoadEnv(file string) error {
f, err := os.Open(file)
if err != nil {
return err
@ -81,9 +81,9 @@ func loadEnv(file string) error {
// specified in the config.json. The go runtime
// takes care of all threads (and not only the calling one)
// executing the underlying systemcall.
func dropPrivileges() error {
if programConfig.Group != "" {
g, err := user.LookupGroup(programConfig.Group)
func DropPrivileges(username string, group string) error {
if group != "" {
g, err := user.LookupGroup(group)
if err != nil {
return err
}
@ -94,8 +94,8 @@ func dropPrivileges() error {
}
}
if programConfig.User != "" {
u, err := user.Lookup(programConfig.User)
if username != "" {
u, err := user.Lookup(username)
if err != nil {
return err
}
@ -111,7 +111,7 @@ func dropPrivileges() error {
// If started via systemd, inform systemd that we are running:
// https://www.freedesktop.org/software/systemd/man/sd_notify.html
func systemdNotifiy(ready bool, status string) {
func SystemdNotifiy(ready bool, status string) {
if os.Getenv("NOTIFY_SOCKET") == "" {
// Not started using systemd
return

6
templates/templates.go → internal/templates/templates.go
View File

@ -5,8 +5,8 @@ import (
"net/http"
"os"
"github.com/ClusterCockpit/cc-backend/config"
"github.com/ClusterCockpit/cc-backend/log"
"github.com/ClusterCockpit/cc-backend/internal/config"
"github.com/ClusterCockpit/cc-backend/pkg/log"
)
var templatesDir string
@ -36,7 +36,7 @@ func init() {
if ebp != "" {
bp = ebp
}
templatesDir = bp + "templates/"
templatesDir = bp + "web/templates/"
base := template.Must(template.ParseFiles(templatesDir + "base.tmpl"))
files := []string{
"home.tmpl", "404.tmpl", "login.tmpl",

308
metricdata/influxdb-v2.go
View File

@ -1,308 +0,0 @@
package metricdata
import (
"context"
"errors"
"fmt"
"log"
"strings"
"time"
"crypto/tls"
"encoding/json"
"github.com/ClusterCockpit/cc-backend/config"
"github.com/ClusterCockpit/cc-backend/schema"
influxdb2 "github.com/influxdata/influxdb-client-go/v2"
influxdb2Api "github.com/influxdata/influxdb-client-go/v2/api"
)
type InfluxDBv2DataRepositoryConfig struct {
Url string `json:"url"`
Token string `json:"token"`
Bucket string `json:"bucket"`
Org string `json:"org"`
SkipTls bool `json:"skiptls"`
}
type InfluxDBv2DataRepository struct {
client influxdb2.Client
queryClient influxdb2Api.QueryAPI
bucket, measurement string
}
func (idb *InfluxDBv2DataRepository) Init(rawConfig json.RawMessage) error {
var config InfluxDBv2DataRepositoryConfig
if err := json.Unmarshal(rawConfig, &config); err != nil {
return err
}
idb.client = influxdb2.NewClientWithOptions(config.Url, config.Token, influxdb2.DefaultOptions().SetTLSConfig(&tls.Config {InsecureSkipVerify: config.SkipTls,} ))
idb.queryClient = idb.client.QueryAPI(config.Org)
idb.bucket = config.Bucket
return nil
}
func (idb *InfluxDBv2DataRepository) formatTime(t time.Time) string {
return t.Format(time.RFC3339) // Like “2006-01-02T15:04:05Z07:00”
}
func (idb *InfluxDBv2DataRepository) epochToTime(epoch int64) time.Time {
return time.Unix(epoch, 0)
}
func (idb *InfluxDBv2DataRepository) LoadData(job *schema.Job, metrics []string, scopes []schema.MetricScope, ctx context.Context) (schema.JobData, error) {
measurementsConds := make([]string, 0, len(metrics))
for _, m := range metrics {
measurementsConds = append(measurementsConds, fmt.Sprintf(`r["_measurement"] == "%s"`, m))
}
measurementsCond := strings.Join(measurementsConds, " or ")
hostsConds := make([]string, 0, len(job.Resources))
for _, h := range job.Resources {
if h.HWThreads != nil || h.Accelerators != nil {
// TODO
return nil, errors.New("the InfluxDB metric data repository does not yet support HWThreads or Accelerators")
}
hostsConds = append(hostsConds, fmt.Sprintf(`r["hostname"] == "%s"`, h.Hostname))
}
hostsCond := strings.Join(hostsConds, " or ")
jobData := make(schema.JobData) // Empty Schema: map[<string>FIELD]map[<MetricScope>SCOPE]<*JobMetric>METRIC
// Requested Scopes
for _, scope := range scopes {
query := ""
switch scope {
case "node":
// Get Finest Granularity, Groupy By Measurement and Hostname (== Metric / Node), Calculate Mean for 60s windows
// log.Println("Note: Scope 'node' requested. ")
query = fmt.Sprintf(`
from(bucket: "%s")
|> range(start: %s, stop: %s)
|> filter(fn: (r) => (%s) and (%s) )
|> drop(columns: ["_start", "_stop"])
|> group(columns: ["hostname", "_measurement"])
|> aggregateWindow(every: 60s, fn: mean)
|> drop(columns: ["_time"])`,
idb.bucket,
idb.formatTime(job.StartTime), idb.formatTime(idb.epochToTime(job.StartTimeUnix + int64(job.Duration) + int64(1) )),
measurementsCond, hostsCond)
case "socket":
log.Println("Note: Scope 'socket' requested, but not yet supported: Will return 'node' scope only. ")
continue
case "core":
log.Println("Note: Scope 'core' requested, but not yet supported: Will return 'node' scope only. ")
continue
// Get Finest Granularity only, Set NULL to 0.0
// query = fmt.Sprintf(`
// from(bucket: "%s")
// |> range(start: %s, stop: %s)
// |> filter(fn: (r) => %s )
// |> filter(fn: (r) => %s )
// |> drop(columns: ["_start", "_stop", "cluster"])
// |> map(fn: (r) => (if exists r._value then {r with _value: r._value} else {r with _value: 0.0}))`,
// idb.bucket,
// idb.formatTime(job.StartTime), idb.formatTime(idb.epochToTime(job.StartTimeUnix + int64(job.Duration) + int64(1) )),
// measurementsCond, hostsCond)
default:
log.Println("Note: Unknown Scope requested: Will return 'node' scope. ")
continue
// return nil, errors.New("the InfluxDB metric data repository does not yet support other scopes than 'node'")
}
rows, err := idb.queryClient.Query(ctx, query)
if err != nil {
return nil, err
}
// Init Metrics: Only Node level now -> TODO: Matching /check on scope level ...
for _, metric := range metrics {
jobMetric, ok := jobData[metric]
if !ok {
mc := config.GetMetricConfig(job.Cluster, metric)
jobMetric = map[schema.MetricScope]*schema.JobMetric{
scope: { // uses scope var from above!
Unit: mc.Unit,
Scope: scope,
Timestep: mc.Timestep,
Series: make([]schema.Series, 0, len(job.Resources)),
StatisticsSeries: nil, // Should be: &schema.StatsSeries{},
},
}
}
jobData[metric] = jobMetric
}
// Process Result: Time-Data
field, host, hostSeries := "", "", schema.Series{}
// typeId := 0
switch scope {
case "node":
for rows.Next() {
row := rows.Record()
if ( host == "" || host != row.ValueByKey("hostname").(string) || rows.TableChanged() ) {
if ( host != "" ) {
// Append Series before reset
jobData[field][scope].Series = append(jobData[field][scope].Series, hostSeries)
}
field, host = row.Measurement(), row.ValueByKey("hostname").(string)
hostSeries = schema.Series{
Hostname: host,
Statistics: nil,
Data: make([]schema.Float, 0),
}
}
val, ok := row.Value().(float64)
if ok {
hostSeries.Data = append(hostSeries.Data, schema.Float(val))
} else {
hostSeries.Data = append(hostSeries.Data, schema.Float(0))
}
}
case "socket":
continue
case "core":
continue
// Include Series.Id in hostSeries
// for rows.Next() {
// row := rows.Record()
// if ( host == "" || host != row.ValueByKey("hostname").(string) || typeId != row.ValueByKey("type-id").(int) || rows.TableChanged() ) {
// if ( host != "" ) {
// // Append Series before reset
// jobData[field][scope].Series = append(jobData[field][scope].Series, hostSeries)
// }
// field, host, typeId = row.Measurement(), row.ValueByKey("hostname").(string), row.ValueByKey("type-id").(int)
// hostSeries = schema.Series{
// Hostname: host,
// Id: &typeId,
// Statistics: nil,
// Data: make([]schema.Float, 0),
// }
// }
// val := row.Value().(float64)
// hostSeries.Data = append(hostSeries.Data, schema.Float(val))
// }
default:
continue
// return nil, errors.New("the InfluxDB metric data repository does not yet support other scopes than 'node, core'")
}
// Append last Series
jobData[field][scope].Series = append(jobData[field][scope].Series, hostSeries)
}
// Get Stats
stats, err := idb.LoadStats(job, metrics, ctx)
if err != nil {
return nil, err
}
for _, scope := range scopes {
if scope == "node" { // No 'socket/core' support yet
for metric, nodes := range stats {
// log.Println(fmt.Sprintf("<< Add Stats for : Field %s >>", metric))
for node, stats := range nodes {
// log.Println(fmt.Sprintf("<< Add Stats for : Host %s : Min %.2f, Max %.2f, Avg %.2f >>", node, stats.Min, stats.Max, stats.Avg ))
for index, _ := range jobData[metric][scope].Series {
// log.Println(fmt.Sprintf("<< Try to add Stats to Series in Position %d >>", index))
if jobData[metric][scope].Series[index].Hostname == node {
// log.Println(fmt.Sprintf("<< Match for Series in Position %d : Host %s >>", index, jobData[metric][scope].Series[index].Hostname))
jobData[metric][scope].Series[index].Statistics = &schema.MetricStatistics{Avg: stats.Avg, Min: stats.Min, Max: stats.Max}
// log.Println(fmt.Sprintf("<< Result Inner: Min %.2f, Max %.2f, Avg %.2f >>", jobData[metric][scope].Series[index].Statistics.Min, jobData[metric][scope].Series[index].Statistics.Max, jobData[metric][scope].Series[index].Statistics.Avg))
}
}
}
}
}
}
// DEBUG:
// for _, scope := range scopes {
// for _, met := range metrics {
// for _, series := range jobData[met][scope].Series {
// log.Println(fmt.Sprintf("<< Result: %d data points for metric %s on %s with scope %s, Stats: Min %.2f, Max %.2f, Avg %.2f >>",
// len(series.Data), met, series.Hostname, scope,
// series.Statistics.Min, series.Statistics.Max, series.Statistics.Avg))
// }
// }
// }
return jobData, nil
}
func (idb *InfluxDBv2DataRepository) LoadStats(job *schema.Job, metrics []string, ctx context.Context) (map[string]map[string]schema.MetricStatistics, error) {
stats := map[string]map[string]schema.MetricStatistics{}
hostsConds := make([]string, 0, len(job.Resources))
for _, h := range job.Resources {
if h.HWThreads != nil || h.Accelerators != nil {
// TODO
return nil, errors.New("the InfluxDB metric data repository does not yet support HWThreads or Accelerators")
}
hostsConds = append(hostsConds, fmt.Sprintf(`r["hostname"] == "%s"`, h.Hostname))
}
hostsCond := strings.Join(hostsConds, " or ")
// lenMet := len(metrics)
for _, metric := range metrics {
// log.Println(fmt.Sprintf("<< You are here: %s (Index %d of %d metrics)", metric, index, lenMet))
query := fmt.Sprintf(`
data = from(bucket: "%s")
|> range(start: %s, stop: %s)
|> filter(fn: (r) => r._measurement == "%s" and r._field == "value" and (%s))
union(tables: [data |> mean(column: "_value") |> set(key: "_field", value: "avg"),
data |> min(column: "_value") |> set(key: "_field", value: "min"),
data |> max(column: "_value") |> set(key: "_field", value: "max")])
|> pivot(rowKey: ["hostname"], columnKey: ["_field"], valueColumn: "_value")
|> group()`,
idb.bucket,
idb.formatTime(job.StartTime), idb.formatTime(idb.epochToTime(job.StartTimeUnix + int64(job.Duration) + int64(1) )),
metric, hostsCond)
rows, err := idb.queryClient.Query(ctx, query)
if err != nil {
return nil, err
}
nodes := map[string]schema.MetricStatistics{}
for rows.Next() {
row := rows.Record()
host := row.ValueByKey("hostname").(string)
avg, avgok := row.ValueByKey("avg").(float64)
if !avgok {
// log.Println(fmt.Sprintf(">> Assertion error for metric %s, statistic AVG. Expected 'float64', got %v", metric, avg))
avg = 0.0
}
min, minok := row.ValueByKey("min").(float64)
if !minok {
// log.Println(fmt.Sprintf(">> Assertion error for metric %s, statistic MIN. Expected 'float64', got %v", metric, min))
min = 0.0
}
max, maxok := row.ValueByKey("max").(float64)
if !maxok {
// log.Println(fmt.Sprintf(">> Assertion error for metric %s, statistic MAX. Expected 'float64', got %v", metric, max))
max = 0.0
}
nodes[host] = schema.MetricStatistics{
Avg: avg,
Min: min,
Max: max,
}
}
stats[metric] = nodes
}
return stats, nil
}
func (idb *InfluxDBv2DataRepository) LoadNodeData(cluster string, metrics, nodes []string, scopes []schema.MetricScope, from, to time.Time, ctx context.Context) (map[string]map[string][]*schema.JobMetric, error) {
// TODO : Implement to be used in Analysis- und System/Node-View
log.Println(fmt.Sprintf("LoadNodeData unimplemented for InfluxDBv2DataRepository, Args: cluster %s, metrics %v, nodes %v, scopes %v", cluster, metrics, nodes, scopes))
return nil, errors.New("unimplemented for InfluxDBv2DataRepository")
}

0
log/log.go → pkg/log/log.go
View File

121
pkg/lrucache/README.md Normal file
View File

@ -0,0 +1,121 @@
# In-Memory LRU Cache for Golang Applications
[![](https://pkg.go.dev/badge/github.com/iamlouk/lrucache?utm_source=godoc)](https://pkg.go.dev/github.com/iamlouk/lrucache)
This library can be embedded into your existing go applications
and play the role *Memcached* or *Redis* might play for others.
It is inspired by [PHP Symfony's Cache Components](https://symfony.com/doc/current/components/cache/adapters/array_cache_adapter.html),
having a similar API. This library can not be used for persistance,
is not properly tested yet and a bit special in a few ways described
below (Especially with regards to the memory usage/`size`).
In addition to the interface described below, a `http.Handler` that can be used as middleware is provided as well.
- Advantages:
- Anything (`interface{}`) can be stored as value
- As it lives in the application itself, no serialization or de-serialization is needed
- As it lives in the application itself, no memory moving/networking is needed
- The computation of a new value for a key does __not__ block the full cache (only the key)
- Disadvantages:
- You have to provide a size estimate for every value
- __This size estimate should not change (i.e. values should not mutate)__
- The cache can only be accessed by one application
## Example
```go
// Go look at the godocs and ./cache_test.go for more documentation and examples
maxMemory := 1000
cache := lrucache.New(maxMemory)
bar = cache.Get("foo", func () (value interface{}, ttl time.Duration, size int) {
return "bar", 10 * time.Second, len("bar")
}).(string)
// bar == "bar"
bar = cache.Get("foo", func () (value interface{}, ttl time.Duration, size int) {
panic("will not be called")
}).(string)
```
## Why does `cache.Get` take a function as argument?
*Using the mechanism described below is optional, the second argument to `Get` can be `nil` and there is a `Put` function as well.*
Because this library is meant to be used by multi threaded applications and the following would
result in the same data being fetched twice if both goroutines run in parallel:
```go
// This code shows what could happen with other cache libraries
c := lrucache.New(MAX_CACHE_ENTRIES)
for i := 0; i < 2; i++ {
go func(){
// This code will run twice in different goroutines,
// it could overlap. As `fetchData` probably does some
// I/O and takes a long time, the probability of both
// goroutines calling `fetchData` is very high!
url := "http://example.com/foo"
contents := c.Get(url)
if contents == nil {
contents = fetchData(url)
c.Set(url, contents)
}
handleData(contents.([]byte))
}()
}
```
Here, if one wanted to make sure that only one of both goroutines fetches the data,
the programmer would need to build his own synchronization. That would suck!
```go
c := lrucache.New(MAX_CACHE_SIZE)
for i := 0; i < 2; i++ {
go func(){
url := "http://example.com/foo"
contents := c.Get(url, func()(interface{}, time.Time, int) {
// This closure will only be called once!
// If another goroutine calls `c.Get` while this closure
// is still being executed, it will wait.
buf := fetchData(url)
return buf, 100 * time.Second, len(buf)
})
handleData(contents.([]byte))
}()
}
```
This is much better as less resources are wasted and synchronization is handled by
the library. If it gets called, the call to the closure happens synchronously. While
it is being executed, all other cache keys can still be accessed without having to wait
for the execution to be done.
## How `Get` works
The closure passed to `Get` will be called if the value asked for is not cached or
expired. It should return the following values:
- The value corresponding to that key and to be stored in the cache
- The time to live for that value (how long until it expires and needs to be recomputed)
- A size estimate
When `maxMemory` is reached, cache entries need to be evicted. Theoretically,
it would be possible to use reflection on every value placed in the cache
to get its exact size in bytes. This would be very expansive and slow though.
Also, size can change. Instead of this library calculating the size in bytes, you, the user,
have to provide a size for every value in whatever unit you like (as long as it is the same unit everywhere).
Suggestions on what to use as size: `len(str)` for strings, `len(slice) * size_of_slice_type`, etc.. It is possible
to use `1` as size for every entry, in that case at most `maxMemory` entries will be in the cache at the same time.
## Affects on GC
Because of the way a garbage collector decides when to run ([explained in the runtime package](https://pkg.go.dev/runtime)), having large amounts of data sitting in your cache might increase the memory consumption of your process by two times the maximum size of the cache. You can decrease the *target percentage* to reduce the effect, but then you might have negative performance effects when your cache is not filled.

288
pkg/lrucache/cache.go Normal file
View File

@ -0,0 +1,288 @@
package lrucache
import (
"sync"
"time"
)
// Type of the closure that must be passed to `Get` to
// compute the value in case it is not cached.
//
// returned values are the computed value to be stored in the cache,
// the duration until this value will expire and a size estimate.
type ComputeValue func() (value interface{}, ttl time.Duration, size int)
type cacheEntry struct {
key string
value interface{}
expiration time.Time
size int
waitingForComputation int
next, prev *cacheEntry
}
type Cache struct {
mutex sync.Mutex
cond *sync.Cond
maxmemory, usedmemory int
entries map[string]*cacheEntry
head, tail *cacheEntry
}
// Return a new instance of a LRU In-Memory Cache.
// Read [the README](./README.md) for more information
// on what is going on with `maxmemory`.
func New(maxmemory int) *Cache {
cache := &Cache{
maxmemory: maxmemory,
entries: map[string]*cacheEntry{},
}
cache.cond = sync.NewCond(&cache.mutex)
return cache
}
// Return the cached value for key `key` or call `computeValue` and
// store its return value in the cache. If called, the closure will be
// called synchronous and __shall not call methods on the same cache__
// or a deadlock might ocure. If `computeValue` is nil, the cache is checked
// and if no entry was found, nil is returned. If another goroutine is currently
// computing that value, the result is waited for.
func (c *Cache) Get(key string, computeValue ComputeValue) interface{} {
now := time.Now()
c.mutex.Lock()
if entry, ok := c.entries[key]; ok {
// The expiration not being set is what shows us that
// the computation of that value is still ongoing.
for entry.expiration.IsZero() {
entry.waitingForComputation += 1
c.cond.Wait()
entry.waitingForComputation -= 1
}
if now.After(entry.expiration) {
if !c.evictEntry(entry) {
if entry.expiration.IsZero() {
panic("cache entry that shoud have been waited for could not be evicted.")
}
c.mutex.Unlock()
return entry.value
}
} else {
if entry != c.head {
c.unlinkEntry(entry)
c.insertFront(entry)
}
c.mutex.Unlock()
return entry.value
}
}
if computeValue == nil {
c.mutex.Unlock()
return nil
}
entry := &cacheEntry{
key: key,
waitingForComputation: 1,
}
c.entries[key] = entry
hasPaniced := true
defer func() {
if hasPaniced {
c.mutex.Lock()
delete(c.entries, key)
entry.expiration = now
entry.waitingForComputation -= 1
}
c.mutex.Unlock()
}()
c.mutex.Unlock()
value, ttl, size := computeValue()
c.mutex.Lock()
hasPaniced = false
entry.value = value
entry.expiration = now.Add(ttl)
entry.size = size
entry.waitingForComputation -= 1
// Only broadcast if other goroutines are actually waiting
// for a result.
if entry.waitingForComputation > 0 {
// TODO: Have more than one condition variable so that there are
// less unnecessary wakeups.
c.cond.Broadcast()
}
c.usedmemory += size
c.insertFront(entry)
// Evict only entries with a size of more than zero.
// This is the only loop in the implementation outside of the `Keys`
// method.
evictionCandidate := c.tail
for c.usedmemory > c.maxmemory && evictionCandidate != nil {
nextCandidate := evictionCandidate.prev
if (evictionCandidate.size > 0 || now.After(evictionCandidate.expiration)) &&
evictionCandidate.waitingForComputation == 0 {
c.evictEntry(evictionCandidate)
}
evictionCandidate = nextCandidate
}
return value
}
// Put a new value in the cache. If another goroutine is calling `Get` and
// computing the value, this function waits for the computation to be done
// before it overwrites the value.
func (c *Cache) Put(key string, value interface{}, size int, ttl time.Duration) {
now := time.Now()
c.mutex.Lock()
defer c.mutex.Unlock()
if entry, ok := c.entries[key]; ok {
for entry.expiration.IsZero() {
entry.waitingForComputation += 1
c.cond.Wait()
entry.waitingForComputation -= 1
}
c.usedmemory -= entry.size
entry.expiration = now.Add(ttl)
entry.size = size
entry.value = value
c.usedmemory += entry.size
c.unlinkEntry(entry)
c.insertFront(entry)
return
}
entry := &cacheEntry{
key: key,
value: value,
expiration: now.Add(ttl),
}
c.entries[key] = entry
c.insertFront(entry)
}
// Remove the value at key `key` from the cache.
// Return true if the key was in the cache and false
// otherwise. It is possible that true is returned even
// though the value already expired.
// It is possible that false is returned even though the value
// will show up in the cache if this function is called on a key
// while that key is beeing computed.
func (c *Cache) Del(key string) bool {
c.mutex.Lock()
defer c.mutex.Unlock()
if entry, ok := c.entries[key]; ok {
return c.evictEntry(entry)
}
return false
}
// Call f for every entry in the cache. Some sanity checks
// and eviction of expired keys are done as well.
// The cache is fully locked for the complete duration of this call!
func (c *Cache) Keys(f func(key string, val interface{})) {
c.mutex.Lock()
defer c.mutex.Unlock()
now := time.Now()
size := 0
for key, e := range c.entries {
if key != e.key {
panic("key mismatch")
}
if now.After(e.expiration) {
if c.evictEntry(e) {
continue
}
}
if e.prev != nil {
if e.prev.next != e {
panic("list corrupted")
}
}
if e.next != nil {
if e.next.prev != e {
panic("list corrupted")
}
}
size += e.size
f(key, e.value)
}
if size != c.usedmemory {
panic("size calculations failed")
}
if c.head != nil {
if c.tail == nil || c.head.prev != nil {
panic("head/tail corrupted")
}
}
if c.tail != nil {
if c.head == nil || c.tail.next != nil {
panic("head/tail corrupted")
}
}
}
func (c *Cache) insertFront(e *cacheEntry) {
e.next = c.head
c.head = e
e.prev = nil
if e.next != nil {
e.next.prev = e
}
if c.tail == nil {
c.tail = e
}
}
func (c *Cache) unlinkEntry(e *cacheEntry) {
if e == c.head {
c.head = e.next
}
if e.prev != nil {
e.prev.next = e.next
}
if e.next != nil {
e.next.prev = e.prev
}
if e == c.tail {
c.tail = e.prev
}
}
func (c *Cache) evictEntry(e *cacheEntry) bool {
if e.waitingForComputation != 0 {
// panic("cannot evict this entry as other goroutines need the value")
return false
}
c.unlinkEntry(e)
c.usedmemory -= e.size
delete(c.entries, e.key)
return true
}

219
pkg/lrucache/cache_test.go Normal file
View File

@ -0,0 +1,219 @@
package lrucache
import (
"sync"
"sync/atomic"
"testing"
"time"
)
func TestBasics(t *testing.T) {
cache := New(123)
value1 := cache.Get("foo", func() (interface{}, time.Duration, int) {
return "bar", 1 * time.Second, 0
})
if value1.(string) != "bar" {
t.Error("cache returned wrong value")
}
value2 := cache.Get("foo", func() (interface{}, time.Duration, int) {
t.Error("value should be cached")
return "", 0, 0
})
if value2.(string) != "bar" {
t.Error("cache returned wrong value")
}
existed := cache.Del("foo")
if !existed {
t.Error("delete did not work as expected")
}
value3 := cache.Get("foo", func() (interface{}, time.Duration, int) {
return "baz", 1 * time.Second, 0
})
if value3.(string) != "baz" {
t.Error("cache returned wrong value")
}
cache.Keys(func(key string, value interface{}) {
if key != "foo" || value.(string) != "baz" {
t.Error("cache corrupted")
}
})
}
func TestExpiration(t *testing.T) {
cache := New(123)
failIfCalled := func() (interface{}, time.Duration, int) {
t.Error("Value should be cached!")
return "", 0, 0
}
val1 := cache.Get("foo", func() (interface{}, time.Duration, int) {
return "bar", 5 * time.Millisecond, 0
})
val2 := cache.Get("bar", func() (interface{}, time.Duration, int) {
return "foo", 20 * time.Millisecond, 0
})
val3 := cache.Get("foo", failIfCalled).(string)
val4 := cache.Get("bar", failIfCalled).(string)
if val1 != val3 || val3 != "bar" || val2 != val4 || val4 != "foo" {
t.Error("Wrong values returned")
}
time.Sleep(10 * time.Millisecond)
val5 := cache.Get("foo", func() (interface{}, time.Duration, int) {
return "baz", 0, 0
})
val6 := cache.Get("bar", failIfCalled)
if val5.(string) != "baz" || val6.(string) != "foo" {
t.Error("unexpected values")
}
cache.Keys(func(key string, val interface{}) {
if key != "bar" || val.(string) != "foo" {
t.Error("wrong value expired")
}
})
time.Sleep(15 * time.Millisecond)
cache.Keys(func(key string, val interface{}) {
t.Error("cache should be empty now")
})
}
func TestEviction(t *testing.T) {
c := New(100)
failIfCalled := func() (interface{}, time.Duration, int) {
t.Error("Value should be cached!")
return "", 0, 0
}
v1 := c.Get("foo", func() (interface{}, time.Duration, int) {
return "bar", 1 * time.Second, 1000
})
v2 := c.Get("foo", func() (interface{}, time.Duration, int) {
return "baz", 1 * time.Second, 1000
})
if v1.(string) != "bar" || v2.(string) != "baz" {
t.Error("wrong values returned")
}
c.Keys(func(key string, val interface{}) {
t.Error("cache should be empty now")
})
_ = c.Get("A", func() (interface{}, time.Duration, int) {
return "a", 1 * time.Second, 50
})
_ = c.Get("B", func() (interface{}, time.Duration, int) {
return "b", 1 * time.Second, 50
})
_ = c.Get("A", failIfCalled)
_ = c.Get("B", failIfCalled)
_ = c.Get("C", func() (interface{}, time.Duration, int) {
return "c", 1 * time.Second, 50
})
_ = c.Get("B", failIfCalled)
_ = c.Get("C", failIfCalled)
v4 := c.Get("A", func() (interface{}, time.Duration, int) {
return "evicted", 1 * time.Second, 25
})
if v4.(string) != "evicted" {
t.Error("value should have been evicted")
}
c.Keys(func(key string, val interface{}) {
if key != "A" && key != "C" {
t.Errorf("'%s' was not expected", key)
}
})
}
// I know that this is a shity test,
// time is relative and unreliable.
func TestConcurrency(t *testing.T) {
c := New(100)
var wg sync.WaitGroup
numActions := 20000
numThreads := 4
wg.Add(numThreads)
var concurrentModifications int32 = 0
for i := 0; i < numThreads; i++ {
go func() {
for j := 0; j < numActions; j++ {
_ = c.Get("key", func() (interface{}, time.Duration, int) {
m := atomic.AddInt32(&concurrentModifications, 1)
if m != 1 {
t.Error("only one goroutine at a time should calculate a value for the same key")
}
time.Sleep(1 * time.Millisecond)
atomic.AddInt32(&concurrentModifications, -1)
return "value", 3 * time.Millisecond, 1
})
}
wg.Done()
}()
}
wg.Wait()
c.Keys(func(key string, val interface{}) {})
}
func TestPanic(t *testing.T) {
c := New(100)
c.Put("bar", "baz", 3, 1*time.Minute)
testpanic := func() {
defer func() {
if r := recover(); r != nil {
if r.(string) != "oops" {
t.Fatal("unexpected panic value")
}
}
}()
_ = c.Get("foo", func() (value interface{}, ttl time.Duration, size int) {
panic("oops")
})
t.Fatal("should have paniced!")
}
testpanic()
v := c.Get("bar", func() (value interface{}, ttl time.Duration, size int) {
t.Fatal("should not be called!")
return nil, 0, 0
})
if v.(string) != "baz" {
t.Fatal("unexpected value")
}
testpanic()
}

120
pkg/lrucache/handler.go Normal file
View File

@ -0,0 +1,120 @@
package lrucache
import (
"bytes"
"net/http"
"strconv"
"time"
)
// HttpHandler is can be used as HTTP Middleware in order to cache requests,
// for example static assets. By default, the request's raw URI is used as key and nothing else.
// Results with a status code other than 200 are cached with a TTL of zero seconds,
// so basically re-fetched as soon as the current fetch is done and a new request
// for that URI is done.
type HttpHandler struct {
cache *Cache
fetcher http.Handler
defaultTTL time.Duration
// Allows overriding the way the cache key is extracted
// from the http request. The defailt is to use the RequestURI.
CacheKey func(*http.Request) string
}
var _ http.Handler = (*HttpHandler)(nil)
type cachedResponseWriter struct {
w http.ResponseWriter
statusCode int
buf bytes.Buffer
}
type cachedResponse struct {
headers http.Header
statusCode int
data []byte
fetched time.Time
}
var _ http.ResponseWriter = (*cachedResponseWriter)(nil)
func (crw *cachedResponseWriter) Header() http.Header {
return crw.w.Header()
}
func (crw *cachedResponseWriter) Write(bytes []byte) (int, error) {
return crw.buf.Write(bytes)
}
func (crw *cachedResponseWriter) WriteHeader(statusCode int) {
crw.statusCode = statusCode
}
// Returns a new caching HttpHandler. If no entry in the cache is found or it was too old, `fetcher` is called with
// a modified http.ResponseWriter and the response is stored in the cache. If `fetcher` sets the "Expires" header,
// the ttl is set appropriately (otherwise, the default ttl passed as argument here is used).
// `maxmemory` should be in the unit bytes.
func NewHttpHandler(maxmemory int, ttl time.Duration, fetcher http.Handler) *HttpHandler {
return &HttpHandler{
cache: New(maxmemory),
defaultTTL: ttl,
fetcher: fetcher,
CacheKey: func(r *http.Request) string {
return r.RequestURI
},
}
}
// gorilla/mux style middleware:
func NewMiddleware(maxmemory int, ttl time.Duration) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return NewHttpHandler(maxmemory, ttl, next)
}
}
// Tries to serve a response to r from cache or calls next and stores the response to the cache for the next time.
func (h *HttpHandler) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
h.ServeHTTP(rw, r)
return
}
cr := h.cache.Get(h.CacheKey(r), func() (interface{}, time.Duration, int) {
crw := &cachedResponseWriter{
w: rw,
statusCode: 200,
buf: bytes.Buffer{},
}
h.fetcher.ServeHTTP(crw, r)
cr := &cachedResponse{
headers: rw.Header().Clone(),
statusCode: crw.statusCode,
data: crw.buf.Bytes(),
fetched: time.Now(),
}
cr.headers.Set("Content-Length", strconv.Itoa(len(cr.data)))
ttl := h.defaultTTL
if cr.statusCode != http.StatusOK {
ttl = 0
} else if cr.headers.Get("Expires") != "" {
if expires, err := http.ParseTime(cr.headers.Get("Expires")); err == nil {
ttl = time.Until(expires)
}
}
return cr, ttl, len(cr.data)
}).(*cachedResponse)
for key, val := range cr.headers {
rw.Header()[key] = val
}
cr.headers.Set("Age", strconv.Itoa(int(time.Since(cr.fetched).Seconds())))
rw.WriteHeader(cr.statusCode)
rw.Write(cr.data)
}

71
pkg/lrucache/handler_test.go Normal file
View File

@ -0,0 +1,71 @@
package lrucache
import (
"bytes"
"net/http"
"net/http/httptest"
"testing"
"time"
)
func TestHandlerBasics(t *testing.T) {
r := httptest.NewRequest(http.MethodGet, "/test1", nil)
rw := httptest.NewRecorder()
shouldBeCalled := true
handler := NewHttpHandler(1000, time.Second, http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
rw.Write([]byte("Hello World!"))
if !shouldBeCalled {
t.Fatal("fetcher expected to be called")
}
}))
handler.ServeHTTP(rw, r)
if rw.Code != 200 {
t.Fatal("unexpected status code")
}
if !bytes.Equal(rw.Body.Bytes(), []byte("Hello World!")) {
t.Fatal("unexpected body")
}
rw = httptest.NewRecorder()
shouldBeCalled = false
handler.ServeHTTP(rw, r)
if rw.Code != 200 {
t.Fatal("unexpected status code")
}
if !bytes.Equal(rw.Body.Bytes(), []byte("Hello World!")) {
t.Fatal("unexpected body")
}
}
// func TestHandlerExpiration(t *testing.T) {
// r := httptest.NewRequest(http.MethodGet, "/test1", nil)
// rw := httptest.NewRecorder()
// i := 1
// now := time.Now()
// handler := NewHttpHandler(1000, 1*time.Second, http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
// rw.Header().Set("Expires", now.Add(10*time.Millisecond).Format(http.TimeFormat))
// rw.Write([]byte(strconv.Itoa(i)))
// }))
// handler.ServeHTTP(rw, r)
// if !(rw.Body.String() == strconv.Itoa(1)) {
// t.Fatal("unexpected body")
// }
// i += 1
// time.Sleep(11 * time.Millisecond)
// rw = httptest.NewRecorder()
// handler.ServeHTTP(rw, r)
// if !(rw.Body.String() == strconv.Itoa(1)) {
// t.Fatal("unexpected body")
// }
// }

0
schema/float.go → pkg/schema/float.go
View File

0
schema/job.go → pkg/schema/job.go
View File

0
schema/metrics.go → pkg/schema/metrics.go
View File

10
startDemo.sh
View File

@ -8,13 +8,13 @@ tar xJf job-archive.tar.xz
rm ./job-archive.tar.xz
touch ./job.db
cd ../frontend
cd ../web/frontend
yarn install
yarn build
cd ..
cd ../..
cp ./configs/env-template.txt .env
go get
go build
go build ./cmd/cc-backend
./cc-backend --init-db --add-user demo:admin:AdminDev --no-server
./cc-backend
./cc-backend --init-db --add-user demo:admin:AdminDev

33
test/api_test.go
View File

@ -4,7 +4,6 @@ import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
@ -14,14 +13,13 @@ import (
"strings"
"testing"
"github.com/ClusterCockpit/cc-backend/api"
"github.com/ClusterCockpit/cc-backend/config"
"github.com/ClusterCockpit/cc-backend/graph"
"github.com/ClusterCockpit/cc-backend/metricdata"
"github.com/ClusterCockpit/cc-backend/repository"
"github.com/ClusterCockpit/cc-backend/schema"
"github.com/ClusterCockpit/cc-backend/internal/api"
"github.com/ClusterCockpit/cc-backend/internal/config"
"github.com/ClusterCockpit/cc-backend/internal/graph"
"github.com/ClusterCockpit/cc-backend/internal/metricdata"
"github.com/ClusterCockpit/cc-backend/internal/repository"
"github.com/ClusterCockpit/cc-backend/pkg/schema"
"github.com/gorilla/mux"
"github.com/jmoiron/sqlx"
_ "github.com/mattn/go-sqlite3"
)
@ -95,17 +93,14 @@ func setup(t *testing.T) *api.RestApi {
}
f.Close()
db, err := sqlx.Open("sqlite3", fmt.Sprintf("%s?_foreign_keys=on", dbfilepath))
if err != nil {
repository.Connect("sqlite3", dbfilepath)
db := repository.GetConnection()
if _, err := db.DB.Exec(repository.JobsDBSchema); err != nil {
t.Fatal(err)
}
db.SetMaxOpenConns(1)
if _, err := db.Exec(repository.JobsDBSchema); err != nil {
t.Fatal(err)
}
if err := config.Init(db, false, map[string]interface{}{}, jobarchive); err != nil {
if err := config.Init(db.DB, false, map[string]interface{}{}, jobarchive); err != nil {
t.Fatal(err)
}
@ -113,10 +108,8 @@ func setup(t *testing.T) *api.RestApi {
t.Fatal(err)
}
resolver := &graph.Resolver{DB: db, Repo: &repository.JobRepository{DB: db}}
if err := resolver.Repo.Init(); err != nil {
t.Fatal(err)
}
jobRepo := repository.GetRepository()
resolver := &graph.Resolver{DB: db.DB, Repo: jobRepo}
return &api.RestApi{
JobRepository: resolver.Repo,

46
tools/README.md Normal file
View File

@ -0,0 +1,46 @@
## Introduction
ClusterCockpit uses JSON Web Tokens (JWT) for authorization of its APIs.
JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object.
This information can be verified and trusted because it is digitally signed.
In ClusterCockpit JWTs are signed using a public/private key pair using ECDSA.
Because tokens are signed using public/private key pairs, the signature also certifies that only the party holding the private key is the one that signed it.
Currently JWT tokens not yet expire.
## JWT Payload
You may view the payload of a JWT token at [https://jwt.io/#debugger-io](https://jwt.io/#debugger-io).
Currently ClusterCockpit sets the following claims:
* `iat`: Issued at claim. The “iat” claim is used to identify the the time at which the JWT was issued. This claim can be used to determine the age of the JWT.
* `sub`: Subject claim. Identifies the subject of the JWT, in our case this is the username.
* `roles`: An array of strings specifying the roles set for the subject.
## Workflow
1. Create a new ECDSA Public/private keypair:
```
$ go build ./tools/gen-keypair.go
$ ./gen-keypair
```
2. Add keypair in your `.env` file. A template can be found in `./configs`.
There are two usage scenarios:
* The APIs are used during a browser session. In this case on login a JWT token is issued on login, that is used by the web frontend to authorize against the GraphQL and REST APIs.
* The REST API is used outside a browser session, e.g. by scripts. In this case you have to issue a token manually. This possible from within the configuration view or on the command line. It is recommended to issue a JWT token in this case for a special user that only has the `api` role. By using different users for different purposes a fine grained access control and access revocation management is possible.
The token is commonly specified in the Authorization HTTP header using the Bearer schema.
## Setup user and JWT token for REST API authorization
1. Create user:
```
$ ./cc-backend --add-user <username>:api:<Password> --no-server
```
2. Issue token for user:
```
$ ./cc-backend -jwt <username> -no-server
```
3. Use issued token token on client side:
```
$ curl -X GET "<API ENDPOINT>" -H "accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer <JWT TOKEN>"
```

22
tools/gen-keypair.go Normal file
View File

@ -0,0 +1,22 @@
package main
import (
"crypto/ed25519"
"crypto/rand"
"encoding/base64"
"fmt"
"os"
)
func main() {
// rand.Reader uses /dev/urandom on Linux
pub, priv, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
fmt.Fprintf(os.Stderr, "error: %s\n", err.Error())
os.Exit(1)
}
fmt.Fprintf(os.Stdout, "JWT_PUBLIC_KEY=%#v\nJWT_PRIVATE_KEY=%#v\n",
base64.StdEncoding.EncodeToString(pub),
base64.StdEncoding.EncodeToString(priv))
}

31
web/frontend/README.md Normal file
View File

@ -0,0 +1,31 @@
# cc-svelte-datatable
[![Build](https://github.com/ClusterCockpit/cc-svelte-datatable/actions/workflows/build.yml/badge.svg)](https://github.com/ClusterCockpit/cc-svelte-datatable/actions/workflows/build.yml)
A frontend for [ClusterCockpit](https://github.com/ClusterCockpit/ClusterCockpit) and [cc-backend](https://github.com/ClusterCockpit/cc-backend). Backend specific configuration can de done using the constants defined in the `intro` section in `./rollup.config.js`.
Builds on:
* [Svelte](https://svelte.dev/)
* [SvelteStrap](https://sveltestrap.js.org/)
* [Bootstrap 5](https://getbootstrap.com/)
* [urql](https://github.com/FormidableLabs/urql)
## Get started
[Yarn](https://yarnpkg.com/) is recommended for package management.
Due to an issue with Yarn v2 you have to stick to Yarn v1.
Install the dependencies...
```bash
yarn install
```
...then start [Rollup](https://rollupjs.org):
```bash
yarn run dev
```
Edit a component file in `src`, save it, and reload the page to see your changes.

25
web/frontend/package.json Normal file
View File

@ -0,0 +1,25 @@
{
"name": "svelte-app",
"version": "1.0.0",
"scripts": {
"build": "rollup -c",
"dev": "rollup -c -w"
},
"devDependencies": {
"@rollup/plugin-commonjs": "^17.0.0",
"@rollup/plugin-node-resolve": "^11.0.0",
"rollup": "^2.3.4",
"rollup-plugin-css-only": "^3.1.0",
"rollup-plugin-svelte": "^7.0.0",
"rollup-plugin-terser": "^7.0.0",
"svelte": "^3.42.6"
},
"dependencies": {
"@rollup/plugin-replace": "^2.4.1",
"@urql/svelte": "^1.3.0",
"graphql": "^15.6.0",
"sveltestrap": "^5.6.1",
"uplot": "^1.6.7",
"wonka": "^4.0.15"
}
}

BIN
web/frontend/public/favicon.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

54
web/frontend/public/global.css Normal file
View File

@ -0,0 +1,54 @@
html, body {
position: relative;
width: 100%;
height: 100%;
}
body {
color: #333;
margin: 0;
padding: 8px;
box-sizing: border-box;
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif;
}
.container {
max-width: 100vw;
}
.site {
display: flex;
flex-direction: column;
height: 100%;
}
.site-content {
flex: 1 0 auto;
margin-top: 80px;
}
.site-footer {
flex: none;
}
footer {
width: 100%;
padding: 0.1rem 1.0rem;
line-height: 1.5;
}
.footer-list {
list-style-type: none;
padding-left: 0;
width: 100%;
display: flex;
flex-wrap: wrap;
justify-content: center;
margin-top: 5px;
margin-bottom: 5px;
}
.footer-list-item {
margin: 0rem 0.8rem;
white-space: nowrap;
}

BIN
web/frontend/public/img/logo.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

1
web/frontend/public/uPlot.min.css vendored Symbolic link
View File

@ -0,0 +1 @@
../node_modules/uplot/dist/uPlot.min.css

70
web/frontend/rollup.config.js Normal file
View File

@ -0,0 +1,70 @@
import svelte from 'rollup-plugin-svelte';
import replace from "@rollup/plugin-replace";
import commonjs from '@rollup/plugin-commonjs';
import resolve from '@rollup/plugin-node-resolve';
import { terser } from 'rollup-plugin-terser';
import css from 'rollup-plugin-css-only';
const production = !process.env.ROLLUP_WATCH;
const plugins = [
svelte({
compilerOptions: {
// enable run-time checks when not in production
dev: !production
}
}),
// If you have external dependencies installed from
// npm, you'll most likely need these plugins. In
// some cases you'll need additional configuration -
// consult the documentation for details:
// https://github.com/rollup/plugins/tree/master/packages/commonjs
resolve({
browser: true,
dedupe: ['svelte']
}),
commonjs(),
// If we're building for production (npm run build
// instead of npm run dev), minify
production && terser(),
replace({
"process.env.NODE_ENV": JSON.stringify("development"),
preventAssignment: true
})
];
const entrypoint = (name, path) => ({
input: path,
output: {
sourcemap: false,
format: 'iife',
name: 'app',
file: `public/build/${name}.js`
},
plugins: [
...plugins,
// we'll extract any component CSS out into
// a separate file - better for performance
css({ output: `${name}.css` }),
],
watch: {
clearScreen: false
}
});
export default [
entrypoint('header', 'src/header.entrypoint.js'),
entrypoint('jobs', 'src/jobs.entrypoint.js'),
entrypoint('user', 'src/user.entrypoint.js'),
entrypoint('list', 'src/list.entrypoint.js'),
entrypoint('job', 'src/job.entrypoint.js'),
entrypoint('systems', 'src/systems.entrypoint.js'),
entrypoint('node', 'src/node.entrypoint.js'),
entrypoint('analysis', 'src/analysis.entrypoint.js'),
entrypoint('status', 'src/status.entrypoint.js')
];

265
web/frontend/src/Analysis.root.svelte Normal file
View File

@ -0,0 +1,265 @@
<script>
import { init } from './utils.js'
import { getContext, onMount } from 'svelte'
import { operationStore, query } from '@urql/svelte'
import { Row, Col, Spinner, Card, Table } from 'sveltestrap'
import Filters from './filters/Filters.svelte'
import PlotSelection from './PlotSelection.svelte'
import Histogram, { binsFromFootprint } from './plots/Histogram.svelte'
import ScatterPlot from './plots/Scatter.svelte'
import PlotTable from './PlotTable.svelte'
import Roofline from './plots/Roofline.svelte'
const { query: initq } = init()
export let filterPresets
// By default, look at the jobs of the last 6 hours:
if (filterPresets?.startTime == null) {
if (filterPresets == null)
filterPresets = {}
let now = new Date(Date.now())
let hourAgo = new Date(now)
hourAgo.setHours(hourAgo.getHours() - 6)
filterPresets.startTime = { from: hourAgo.toISOString(), to: now.toISOString() }
}
let cluster
let filters
let rooflineMaxY
let colWidth
let numBins = 50
const ccconfig = getContext('cc-config'),
metricConfig = getContext('metrics')
let metricsInHistograms = ccconfig.analysis_view_histogramMetrics,
metricsInScatterplots = ccconfig.analysis_view_scatterPlotMetrics
$: metrics = [...new Set([...metricsInHistograms, ...metricsInScatterplots.flat()])]
getContext('on-init')(({ data }) => {
if (data != null) {
cluster = data.clusters.find(c => c.name == filterPresets.cluster)
console.assert(cluster != null, `This cluster could not be found: ${filterPresets.cluster}`)
rooflineMaxY = cluster.subClusters.reduce((max, part) => Math.max(max, part.flopRateSimd), 0)
$rooflineQuery.variables.maxY = rooflineMaxY
$rooflineQuery.context.pause = false
$rooflineQuery.reexecute()
}
})
const statsQuery = operationStore(`
query($filter: [JobFilter!]!) {
stats: jobsStatistics(filter: $filter) {
totalJobs
shortJobs
totalWalltime
totalCoreHours
histDuration { count, value }
histNumNodes { count, value }
}
topUsers: jobsCount(filter: $filter, groupBy: USER, weight: NODE_HOURS, limit: 5) { name, count }
}
`, { filter: [] }, { pause: true })
const footprintsQuery = operationStore(`
query($filter: [JobFilter!]!, $metrics: [String!]!) {
footprints: jobsFootprints(filter: $filter, metrics: $metrics) {
nodehours,
metrics { metric, data }
}
}
`, { filter: [], metrics }, { pause: true })
$: $footprintsQuery.variables = { ...$footprintsQuery.variables, metrics }
const rooflineQuery = operationStore(`
query($filter: [JobFilter!]!, $rows: Int!, $cols: Int!,
$minX: Float!, $minY: Float!, $maxX: Float!, $maxY: Float!) {
rooflineHeatmap(filter: $filter, rows: $rows, cols: $cols,
minX: $minX, minY: $minY, maxX: $maxX, maxY: $maxY)
}
`, {
filter: [],
rows: 50, cols: 50,
minX: 0.01, minY: 1., maxX: 1000., maxY: -1
}, { pause: true });
query(statsQuery)
query(footprintsQuery)
query(rooflineQuery)
onMount(() => filters.update())
</script>
<Row>
{#if $initq.fetching || $statsQuery.fetching || $footprintsQuery.fetching}
<Col xs="auto">
<Spinner />
</Col>
{/if}
<Col xs="auto">
{#if $initq.error}
<Card body color="danger">{$initq.error.message}</Card>
{:else if cluster}
<PlotSelection
availableMetrics={cluster.metricConfig.map(mc => mc.name)}
bind:metricsInHistograms={metricsInHistograms}
bind:metricsInScatterplots={metricsInScatterplots} />
{/if}
</Col>
<Col xs="auto">
<Filters
bind:this={filters}
filterPresets={filterPresets}
disableClusterSelection={true}
startTimeQuickSelect={true}
on:update={({ detail }) => {
$statsQuery.context.pause = false
$statsQuery.variables = { filter: detail.filters }
$footprintsQuery.context.pause = false
$footprintsQuery.variables = { metrics, filter: detail.filters }
$rooflineQuery.variables = { ...$rooflineQuery.variables, filter: detail.filters }
}} />
</Col>
</Row>
<br/>
{#if $statsQuery.error}
<Row>
<Col>
<Card body color="danger">{$statsQuery.error.message}</Card>
</Col>
</Row>
{:else if $statsQuery.data}
<Row>
<div class="col-3" bind:clientWidth={colWidth}>
<div style="height: 40%">
<Table>
<tr>
<th scope="col">Total Jobs</th>
<td>{$statsQuery.data.stats[0].totalJobs}</td>
</tr>
<tr>
<th scope="col">Short Jobs (&#60; 2m)</th>
<td>{$statsQuery.data.stats[0].shortJobs}</td>
</tr>
<tr>
<th scope="col">Total Walltime</th>
<td>{$statsQuery.data.stats[0].totalWalltime}</td>
</tr>
<tr>
<th scope="col">Total Core Hours</th>
<td>{$statsQuery.data.stats[0].totalCoreHours}</td>
</tr>
</Table>
</div>
<div style="height: 60%;">
{#key $statsQuery.data.topUsers}
<h4>Top Users (by node hours)</h4>
<Histogram
width={colWidth - 25} height={300 * 0.5}
data={$statsQuery.data.topUsers.sort((a, b) => b.count - a.count).map(({ count }, idx) => ({ count, value: idx }))}
label={(x) => x < $statsQuery.data.topUsers.length ? $statsQuery.data.topUsers[Math.floor(x)].name : '0'} />
{/key}
</div>
</div>
<div class="col-3">
{#key $statsQuery.data.stats[0].histDuration}
<h4>Walltime Distribution</h4>
<Histogram
width={colWidth - 25} height={300}
data={$statsQuery.data.stats[0].histDuration} />
{/key}
</div>
<div class="col-3">
{#key $statsQuery.data.stats[0].histNumNodes}
<h4>Number of Nodes Distribution</h4>
<Histogram
width={colWidth - 25} height={300}
data={$statsQuery.data.stats[0].histNumNodes} />
{/key}
</div>
<div class="col-3">
{#if $rooflineQuery.fetching}
<Spinner />
{:else if $rooflineQuery.error}
<Card body color="danger">{$rooflineQuery.error.message}</Card>
{:else if $rooflineQuery.data && cluster}
{#key $rooflineQuery.data}
<Roofline
width={colWidth - 25} height={300}
tiles={$rooflineQuery.data.rooflineHeatmap}
cluster={cluster.subClusters.length == 1 ? cluster.subClusters[0] : null}
maxY={rooflineMaxY} />
{/key}
{/if}
</div>
</Row>
{/if}
<br/>
{#if $footprintsQuery.error}
<Row>
<Col>
<Card body color="danger">{$footprintsQuery.error.message}</Card>
</Col>
</Row>
{:else if $footprintsQuery.data && $initq.data}
<Row>
<Col>
<Card body>
These histograms show the distribution of the averages of all jobs matching the filters. Each job/average is weighted by its node hours.
</Card>
<br/>
</Col>
</Row>
<Row>
<Col>
<PlotTable
let:item
let:width
items={metricsInHistograms.map(metric => ({ metric, ...binsFromFootprint(
$footprintsQuery.data.footprints.nodehours,
$footprintsQuery.data.footprints.metrics.find(f => f.metric == metric).data, numBins) }))}
itemsPerRow={ccconfig.plot_view_plotsPerRow}>
<h4>{item.metric} [{metricConfig(cluster.name, item.metric)?.unit}]</h4>
<Histogram
width={width} height={250}
min={item.min} max={item.max}
data={item.bins} label={item.label} />
</PlotTable>
</Col>
</Row>
<br/>
<Row>
<Col>
<Card body>
Each circle represents one job. The size of a circle is proportional to its node hours. Darker circles mean multiple jobs have the same averages for the respective metrics.
</Card>
<br/>
</Col>
</Row>
<Row>
<Col>
<PlotTable
let:item
let:width
items={metricsInScatterplots.map(([m1, m2]) => ({
m1, f1: $footprintsQuery.data.footprints.metrics.find(f => f.metric == m1).data,
m2, f2: $footprintsQuery.data.footprints.metrics.find(f => f.metric == m2).data }))}
itemsPerRow={ccconfig.plot_view_plotsPerRow}>
<ScatterPlot
width={width} height={250} color={"rgba(0, 102, 204, 0.33)"}
xLabel={`${item.m1} [${metricConfig(cluster.name, item.m1)?.unit}]`}
yLabel={`${item.m2} [${metricConfig(cluster.name, item.m2)?.unit}]`}
X={item.f1} Y={item.f2} S={$footprintsQuery.data.footprints.nodehours} />
</PlotTable>
</Col>
</Row>
{/if}

73
web/frontend/src/Header.svelte Normal file
View File

@ -0,0 +1,73 @@
<script>
import { Icon, Button, InputGroup, Input, Collapse,
Navbar, NavbarBrand, Nav, NavItem, NavLink, NavbarToggler,
Dropdown, DropdownToggle, DropdownMenu, DropdownItem } from 'sveltestrap'
export let username // empty string if auth. is disabled, otherwise the username as string
export let isAdmin // boolean
export let clusters // array of names
let isOpen = false
const views = [
isAdmin
? { title: 'Jobs', adminOnly: false, href: '/monitoring/jobs/', icon: 'card-list' }
: { title: 'My Jobs', adminOnly: false, href: `/monitoring/user/${username}`, icon: 'bar-chart-line-fill' },
{ title: 'Users', adminOnly: true, href: '/monitoring/users/', icon: 'people-fill' },
{ title: 'Projects', adminOnly: true, href: '/monitoring/projects/', icon: 'folder' },
{ title: 'Tags', adminOnly: false, href: '/monitoring/tags/', icon: 'tags' }
]
const viewsPerCluster = [
{ title: 'Analysis', adminOnly: true, href: '/monitoring/analysis/', icon: 'graph-up' },
{ title: 'Systems', adminOnly: true, href: '/monitoring/systems/', icon: 'cpu' },
{ title: 'Status', adminOnly: true, href: '/monitoring/status/', icon: 'cpu' },
]
</script>
<Navbar color="light" light expand="lg" fixed="top">
<NavbarBrand href="/">
<img alt="ClusterCockpit Logo" src="/img/logo.png" height="25rem">
</NavbarBrand>
<NavbarToggler on:click={() => (isOpen = !isOpen)} />
<Collapse {isOpen} navbar expand="lg" on:update={({ detail }) => (isOpen = detail.isOpen)}>
<Nav pills>
{#each views.filter(item => isAdmin || !item.adminOnly) as item}
<NavLink href={item.href} active={window.location.pathname == item.href}><Icon name={item.icon}/> {item.title}</NavLink>
{/each}
{#each viewsPerCluster.filter(item => !item.adminOnly || isAdmin) as item}
<NavItem>
<Dropdown nav inNavbar>
<DropdownToggle nav caret>
<Icon name={item.icon}/> {item.title}
</DropdownToggle>
<DropdownMenu>
{#each clusters as cluster}
<DropdownItem href={item.href + cluster} active={window.location.pathname == item.href + cluster}>
{cluster}
</DropdownItem>
{/each}
</DropdownMenu>
</Dropdown>
</NavItem>
{/each}
</Nav>
</Collapse>
<div class="d-flex">
<form method="GET" action="/search">
<InputGroup>
<Input type="text" placeholder={isAdmin ? "Search jobId / username" : "Search jobId"} name="searchId"/>
<Button outline type="submit"><Icon name="search"/></Button>
</InputGroup>
</form>
{#if username}
<form method="POST" action="/logout">
<Button outline color="success" type="submit" style="margin-left: 10px;">
<Icon name="box-arrow-right"/> Logout {username}
</Button>
</form>
{/if}
<Button outline on:click={() => window.location.href = '/config'} style="margin-left: 10px;">
<Icon name="gear"/>
</Button>
</div>
</Navbar>

224
web/frontend/src/Job.root.svelte Normal file
View File

@ -0,0 +1,224 @@
<script>
import { init, groupByScope, fetchMetricsStore } from './utils.js'
import { Row, Col, Card, Spinner, TabContent, TabPane,
CardBody, CardHeader, CardTitle, Button, Icon } from 'sveltestrap'
import PlotTable from './PlotTable.svelte'
import Metric from './Metric.svelte'
import PolarPlot from './plots/Polar.svelte'
import Roofline from './plots/Roofline.svelte'
import JobInfo from './joblist/JobInfo.svelte'
import TagManagement from './TagManagement.svelte'
import MetricSelection from './MetricSelection.svelte'
import Zoom from './Zoom.svelte'
import StatsTable from './StatsTable.svelte'
import { getContext } from 'svelte'
export let dbid
const { query: initq } = init(`
job(id: "${dbid}") {
id, jobId, user, project, cluster, startTime,
duration, numNodes, numHWThreads, numAcc,
SMT, exclusive, partition, subCluster, arrayJobId,
monitoringStatus, state, walltime,
tags { id, type, name },
resources { hostname, hwthreads, accelerators },
metaData
userData { name, email }
}
`)
const ccconfig = getContext('cc-config'),
clusters = getContext('clusters')
let isMetricsSelectionOpen = false, selectedMetrics = []
const [jobMetrics, startFetching] = fetchMetricsStore()
getContext('on-init')(() => {
let job = $initq.data.job
if (!job)
return
startFetching(job, null, job.numNodes > 2 ? ["node"] : ["node", "core"])
// TODO: Do not even fetch metrics that are not one of the following: flops_any, mem_bw, job_view_selectedMetrics, job_view_nodestats_selectedMetrics
selectedMetrics = ccconfig[`job_view_selectedMetrics:${job.cluster}`]
|| clusters.find(c => c.name == job.cluster).metricConfig.map(mc => mc.name)
})
let plots = {}, jobTags, fullWidth, statsTable
$: polarPlotSize = Math.min(fullWidth / 3 - 10, 300)
$: document.title = $initq.fetching ? 'Loading...' : ($initq.error ? 'Error' : `Job ${$initq.data.job.jobId} - ClusterCockpit`)
let missingMetrics = [], missingHosts = [], somethingMissing = false
$: if ($initq.data && $jobMetrics.data) {
let job = $initq.data.job,
metrics = $jobMetrics.data.jobMetrics,
metricNames = clusters.find(c => c.name == job.cluster).metricConfig.map(mc => mc.name)
missingMetrics = metricNames.filter(metric => !metrics.some(jm => jm.name == metric))
missingHosts = job.resources.map(({ hostname }) => ({
hostname: hostname,
metrics: metricNames.filter(metric => !metrics.some(jm => jm.metric.scope == 'node' && jm.metric.series.some(series => series.hostname == hostname)))
})).filter(({ metrics }) => metrics.length > 0)
somethingMissing = missingMetrics.length > 0 || missingHosts.length > 0
}
const orderAndMap = (grouped, selectedMetrics) => selectedMetrics.map(metric => ({ metric: metric, data: grouped.find((group) => group[0].name == metric) }))
</script>
<div class="row" bind:clientWidth={fullWidth}></div>
<Row>
<Col>
{#if $initq.error}
<Card body color="danger">{$initq.error.message}</Card>
{:else if $initq.data}
<JobInfo job={$initq.data.job} jobTags={jobTags}/>
{:else}
<Spinner secondary/>
{/if}
</Col>
{#if $jobMetrics.data && $initq.data}
<Col>
<PolarPlot
width={polarPlotSize} height={polarPlotSize}
metrics={ccconfig.job_view_polarPlotMetrics}
cluster={$initq.data.job.cluster}
jobMetrics={$jobMetrics.data.jobMetrics} />
</Col>
<Col>
<Roofline
width={fullWidth / 3 - 10} height={polarPlotSize}
cluster={clusters
.find(c => c.name == $initq.data.job.cluster).subClusters
.find(sc => sc.name == $initq.data.job.subCluster)}
flopsAny={$jobMetrics.data.jobMetrics.find(m => m.name == 'flops_any' && m.metric.scope == 'node').metric}
memBw={$jobMetrics.data.jobMetrics.find(m => m.name == 'mem_bw' && m.metric.scope == 'node').metric} />
</Col>
{:else}
<Col></Col>
<Col></Col>
{/if}
</Row>
<br/>
<Row>
<Col xs="auto">
{#if $initq.data}
<TagManagement job={$initq.data.job} bind:jobTags={jobTags}/>
{/if}
</Col>
<Col xs="auto">
{#if $initq.data}
<Button outline
on:click={() => (isMetricsSelectionOpen = true)}>
<Icon name="graph-up"/> Metrics
</Button>
{/if}
</Col>
<Col xs="auto">
<Zoom timeseriesPlots={plots} />
</Col>
</Row>
<br/>
<Row>
<Col>
{#if $jobMetrics.error}
{#if $initq.data.job.monitoringStatus == 0 || $initq.data.job.monitoringStatus == 2}
<Card body color="warning">Not monitored or archiving failed</Card>
<br/>
{/if}
<Card body color="danger">{$jobMetrics.error.message}</Card>
{:else if $jobMetrics.fetching}
<Spinner secondary/>
{:else if $jobMetrics.data && $initq.data}
<PlotTable
let:item
let:width
items={orderAndMap(groupByScope($jobMetrics.data.jobMetrics), selectedMetrics)}
itemsPerRow={ccconfig.plot_view_plotsPerRow}>
{#if item.data}
<Metric
bind:this={plots[item.metric]}
on:more-loaded={({ detail }) => statsTable.moreLoaded(detail)}
job={$initq.data.job}
metric={item.metric}
scopes={item.data.map(x => x.metric)}
width={width}/>
{:else}
<Card body color="warning">No data for <code>{item.metric}</code></Card>
{/if}
</PlotTable>
{/if}
</Col>
</Row>
<br/>
<Row>
<Col>
{#if $initq.data}
<TabContent>
{#if somethingMissing}
<TabPane tabId="resources" tab="Resources" active={somethingMissing}>
<div style="margin: 10px;"><Card color="warning">
<CardHeader>
<CardTitle>Missing Metrics/Reseources</CardTitle>
</CardHeader>
<CardBody>
{#if missingMetrics.length > 0}
<p>No data at all is available for the metrics: {missingMetrics.join(', ')}</p>
{/if}
{#if missingHosts.length > 0}
<p>Some metrics are missing for the following hosts:</p>
<ul>
{#each missingHosts as missing}
<li>{missing.hostname}: {missing.metrics.join(', ')}</li>
{/each}
</ul>
{/if}
</CardBody>
</Card></div>
</TabPane>
{/if}
<TabPane tabId="stats" tab="Statistics Table" active={!somethingMissing}>
{#if $jobMetrics.data}
<StatsTable bind:this={statsTable} job={$initq.data.job} jobMetrics={$jobMetrics.data.jobMetrics} />
{/if}
</TabPane>
<TabPane tabId="job-script" tab="Job Script">
<div class="pre-wrapper">
{#if $initq.data.job.metaData?.jobScript}
<pre><code>{$initq.data.job.metaData?.jobScript}</code></pre>
{:else}
<Card body color="warning">No job script available</Card>
{/if}
</div>
</TabPane>
<TabPane tabId="slurm-info" tab="Slurm Info">
<div class="pre-wrapper">
{#if $initq.data.job.metaData?.slurmInfo}
<pre><code>{$initq.data.job.metaData?.slurmInfo}</code></pre>
{:else}
<Card body color="warning">No additional slurm information available</Card>
{/if}
</div>
</TabPane>
</TabContent>
{/if}
</Col>
</Row>
{#if $initq.data}
<MetricSelection
cluster={$initq.data.job.cluster}
configName="job_view_selectedMetrics"
bind:metrics={selectedMetrics}
bind:isOpen={isMetricsSelectionOpen} />
{/if}
<style>
.pre-wrapper {
font-size: 1.1rem;
margin: 10px;
border: 1px solid #bbb;
border-radius: 5px;
padding: 5px;
}
</style>

88
web/frontend/src/Jobs.root.svelte Normal file
View File

@ -0,0 +1,88 @@
<script>
import { onMount, getContext } from 'svelte'
import { init } from './utils.js'
import { Row, Col, Button, Icon, Card, Spinner } from 'sveltestrap'
import Filters from './filters/Filters.svelte'
import JobList from './joblist/JobList.svelte'
import Refresher from './joblist/Refresher.svelte'
import Sorting from './joblist/SortSelection.svelte'
import MetricSelection from './MetricSelection.svelte'
import UserOrProject from './filters/UserOrProject.svelte'
const { query: initq } = init()
const ccconfig = getContext('cc-config')
export let filterPresets = {}
let filters, jobList, matchedJobs = null
let sorting = { field: 'startTime', order: 'DESC' }, isSortingOpen = false, isMetricsSelectionOpen = false
let metrics = filterPresets.cluster
? ccconfig[`plot_list_selectedMetrics:${filterPresets.cluster}`] || ccconfig.plot_list_selectedMetrics
: ccconfig.plot_list_selectedMetrics
// The filterPresets are handled by the Filters component,
// so we need to wait for it to be ready before we can start a query.
// This is also why JobList component starts out with a paused query.
onMount(() => filters.update())
</script>
<Row>
{#if $initq.fetching}
<Col xs="auto">
<Spinner/>
</Col>
{:else if $initq.error}
<Col xs="auto">
<Card body color="danger">{$initq.error.message}</Card>
</Col>
{/if}
</Row>
<Row>
<Col xs="auto">
<Button
outline color="primary"
on:click={() => (isSortingOpen = true)}>
<Icon name="sort-up"/> Sorting
</Button>
<Button
outline color="primary"
on:click={() => (isMetricsSelectionOpen = true)}>
<Icon name="graph-up"/> Metrics
</Button>
<Button disabled outline>{matchedJobs == null ? 'Loading...' : `${matchedJobs} jobs`}</Button>
</Col>
<Col xs="auto">
<Filters
filterPresets={filterPresets}
bind:this={filters}
on:update={({ detail }) => jobList.update(detail.filters)} />
</Col>
<Col xs="3" style="margin-left: auto;">
<UserOrProject on:update={({ detail }) => filters.update(detail)}/>
</Col>
<Col xs="2">
<Refresher on:reload={() => jobList.update()} />
</Col>
</Row>
<br/>
<Row>
<Col>
<JobList
bind:metrics={metrics}
bind:sorting={sorting}
bind:matchedJobs={matchedJobs}
bind:this={jobList} />
</Col>
</Row>
<Sorting
bind:sorting={sorting}
bind:isOpen={isSortingOpen} />
<MetricSelection
cluster={filterPresets.cluster}
configName="plot_list_selectedMetrics"
bind:metrics={metrics}
bind:isOpen={isMetricsSelectionOpen} />

151
web/frontend/src/List.root.svelte Normal file
View File

@ -0,0 +1,151 @@
<!--
@component List of users or projects
-->
<script>
import { onMount } from 'svelte'
import { init } from './utils.js'
import { Row, Col, Button, Icon, Table, Card, Spinner,
InputGroup, Input } from 'sveltestrap'
import Filters from './filters/Filters.svelte'
import { operationStore, query } from '@urql/svelte';
import { scramble, scrambleNames } from './joblist/JobInfo.svelte'
const { } = init()
export let type
export let filterPresets
console.assert(type == 'USER' || type == 'PROJECT', 'Invalid list type provided!')
const stats = operationStore(`query($filter: [JobFilter!]!) {
rows: jobsStatistics(filter: $filter, groupBy: ${type}) {
id
totalJobs
totalWalltime
totalCoreHours
}
}`, {
filter: []
}, {
pause: true
})
query(stats)
let filters
let nameFilter = ''
let sorting = { field: 'totalJobs', direction: 'down' }
function changeSorting(event, field) {
let target = event.target
while (target.tagName != 'BUTTON')
target = target.parentElement
let direction = target.children[0].className.includes('up') ? 'down' : 'up'
target.children[0].className = `bi-sort-numeric-${direction}`
sorting = { field, direction }
}
function sort(stats, sorting, nameFilter) {
const cmp = sorting.field == 'id'
? (sorting.direction == 'up'
? (a, b) => a.id < b.id
: (a, b) => a.id > b.id)
: (sorting.direction == 'up'
? (a, b) => a[sorting.field] - b[sorting.field]
: (a, b) => b[sorting.field] - a[sorting.field])
return stats.filter(u => u.id.includes(nameFilter)).sort(cmp)
}
onMount(() => filters.update())
</script>
<Row>
<Col xs="auto">
<InputGroup>
<Button disabled outline>
Search {type.toLowerCase()}s
</Button>
<Input bind:value={nameFilter} placeholder="Filter by {({ USER: 'username', PROJECT: 'project' })[type]}" />
</InputGroup>
</Col>
<Col xs="auto">
<Filters
bind:this={filters}
filterPresets={filterPresets}
startTimeQuickSelect={true}
menuText="Only {type.toLowerCase()}s with jobs that match the filters will show up"
on:update={({ detail }) => {
$stats.variables = { filter: detail.filters }
$stats.context.pause = false
$stats.reexecute()
}} />
</Col>
</Row>
<Table>
<thead>
<tr>
<th scope="col">
{({ USER: 'Username', PROJECT: 'Project Name' })[type]}
<Button color="{sorting.field == 'id' ? 'primary' : 'light'}"
size="sm" on:click={e => changeSorting(e, 'id')}>
<Icon name="sort-numeric-down" />
</Button>
</th>
<th scope="col">
Total Jobs
<Button color="{sorting.field == 'totalJobs' ? 'primary' : 'light'}"
size="sm" on:click={e => changeSorting(e, 'totalJobs')}>
<Icon name="sort-numeric-down" />
</Button>
</th>
<th scope="col">
Total Walltime
<Button color="{sorting.field == 'totalWalltime' ? 'primary' : 'light'}"
size="sm" on:click={e => changeSorting(e, 'totalWalltime')}>
<Icon name="sort-numeric-down" />
</Button>
</th>
<th scope="col">
Total Core Hours
<Button color="{sorting.field == 'totalCoreHours' ? 'primary' : 'light'}"
size="sm" on:click={e => changeSorting(e, 'totalCoreHours')}>
<Icon name="sort-numeric-down" />
</Button>
</th>
</tr>
</thead>
<tbody>
{#if $stats.fetching}
<tr>
<td colspan="4" style="text-align: center;"><Spinner secondary/></td>
</tr>
{:else if $stats.error}
<tr>
<td colspan="4"><Card body color="danger" class="mb-3">{$stats.error.message}</Card></td>
</tr>
{:else if $stats.data}
{#each sort($stats.data.rows, sorting, nameFilter) as row (row.id)}
<tr>
<td>
{#if type == 'USER'}
<a href="/monitoring/user/{row.id}">{scrambleNames ? scramble(row.id) : row.id}</a>
{:else if type == 'PROJECT'}
<a href="/monitoring/jobs/?project={row.id}">{row.id}</a>
{:else}
{row.id}
{/if}
</td>
<td>{row.totalJobs}</td>
<td>{row.totalWalltime}</td>
<td>{row.totalCoreHours}</td>
</tr>
{:else}
<tr>
<td colspan="4"><i>No {type.toLowerCase()}s/jobs found</i></td>
</tr>
{/each}
{/if}
</tbody>
</Table>

88
web/frontend/src/Metric.svelte Normal file
View File

@ -0,0 +1,88 @@
<script>
import { getContext, createEventDispatcher } from 'svelte'
import Timeseries from './plots/MetricPlot.svelte'
import { InputGroup, InputGroupText, Spinner, Card } from 'sveltestrap'
import { fetchMetrics, minScope } from './utils'
export let job
export let metric
export let scopes
export let width
const dispatch = createEventDispatcher()
const cluster = getContext('clusters').find(cluster => cluster.name == job.cluster)
const subCluster = cluster.subClusters.find(subCluster => subCluster.name == job.subCluster)
const metricConfig = cluster.metricConfig.find(metricConfig => metricConfig.name == metric)
let selectedScope = minScope(scopes.map(s => s.scope)), selectedHost = null, plot, fetching = false, error = null
$: avaliableScopes = scopes.map(metric => metric.scope)
$: data = scopes.find(metric => metric.scope == selectedScope)
$: series = data?.series.filter(series => selectedHost == null || series.hostname == selectedHost)
let from = null, to = null
export function setTimeRange(f, t) {
from = f, to = t
}
$: if (plot != null) plot.setTimeRange(from, to)
export async function loadMore() {
fetching = true
let response = await fetchMetrics(job, [metric], ["core"])
fetching = false
if (response.error) {
error = response.error
return
}
for (let jm of response.data.jobMetrics) {
if (jm.metric.scope != "node") {
scopes.push(jm.metric)
selectedScope = jm.metric.scope
dispatch('more-loaded', jm)
if (!avaliableScopes.includes(selectedScope))
avaliableScopes = [...avaliableScopes, selectedScope]
}
}
}
$: if (selectedScope == "load-more") loadMore()
</script>
<InputGroup>
<InputGroupText style="min-width: 150px;">
{metric} ({metricConfig?.unit})
</InputGroupText>
<select class="form-select" bind:value={selectedScope}>
{#each avaliableScopes as scope}
<option value={scope}>{scope}</option>
{/each}
{#if avaliableScopes.length == 1 && metricConfig?.scope != "node"}
<option value={"load-more"}>Load more...</option>
{/if}
</select>
{#if job.resources.length > 1}
<select class="form-select" bind:value={selectedHost}>
<option value={null}>All Hosts</option>
{#each job.resources as { hostname }}
<option value={hostname}>{hostname}</option>
{/each}
</select>
{/if}
</InputGroup>
{#key series}
{#if fetching == true}
<Spinner/>
{:else if error != null}
<Card body color="danger">{error.message}</Card>
{:else if series != null}
<Timeseries
bind:this={plot}
width={width} height={300}
cluster={cluster} subCluster={subCluster}
timestep={data.timestep}
scope={selectedScope} metric={metric}
series={series} />
{/if}
{/key}

126
web/frontend/src/MetricSelection.svelte Normal file
View File

@ -0,0 +1,126 @@
<!--
@component
Properties:
- metrics: [String] (changes from inside, needs to be initialised, list of selected metrics)
- isOpen: Boolean (can change from inside and outside)
- configName: String (constant)
-->
<script>
import { Modal, ModalBody, ModalHeader, ModalFooter, Button, ListGroup } from 'sveltestrap'
import { getContext } from 'svelte'
import { mutation } from '@urql/svelte'
export let metrics
export let isOpen
export let configName
export let allMetrics = null
export let cluster = null
const clusters = getContext('clusters'),
onInit = getContext('on-init')
let newMetricsOrder = []
let unorderedMetrics = [...metrics]
onInit(() => {
if (allMetrics == null) {
allMetrics = new Set()
for (let c of clusters)
if (cluster == null || c.name == cluster)
for (let metric of c.metricConfig)
allMetrics.add(metric.name)
}
newMetricsOrder = [...allMetrics].filter(m => !metrics.includes(m))
newMetricsOrder.unshift(...metrics)
})
const updateConfiguration = mutation({
query: `mutation($name: String!, $value: String!) {
updateConfiguration(name: $name, value: $value)
}`
})
let columnHovering = null
function columnsDragStart(event, i) {
event.dataTransfer.effectAllowed = 'move'
event.dataTransfer.dropEffect = 'move'
event.dataTransfer.setData('text/plain', i)
}
function columnsDrag(event, target) {
event.dataTransfer.dropEffect = 'move'
const start = Number.parseInt(event.dataTransfer.getData("text/plain"))
if (start < target) {
newMetricsOrder.splice(target + 1, 0, newMetricsOrder[start])
newMetricsOrder.splice(start, 1)
} else {
newMetricsOrder.splice(target, 0, newMetricsOrder[start])
newMetricsOrder.splice(start + 1, 1)
}
columnHovering = null
}
function closeAndApply() {
metrics = newMetricsOrder.filter(m => unorderedMetrics.includes(m))
isOpen = false
updateConfiguration({
name: cluster == null ? configName : `${configName}:${cluster}`,
value: JSON.stringify(metrics)
})
.then(res => {
if (res.error)
console.error(res.error)
})
}
</script>
<style>
li.cc-config-column {
display: block;
cursor: grab;
}
li.cc-config-column.is-active {
background-color: #3273dc;
color: #fff;
cursor: grabbing;
}
</style>
<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
<ModalHeader>
Configure columns
</ModalHeader>
<ModalBody>
<ListGroup>
{#each newMetricsOrder as metric, index (metric)}
<li class="cc-config-column list-group-item"
draggable={true} ondragover="return false"
on:dragstart={event => columnsDragStart(event, index)}
on:drop|preventDefault={event => columnsDrag(event, index)}
on:dragenter={() => columnHovering = index}
class:is-active={columnHovering === index}>
{#if unorderedMetrics.includes(metric)}
<input type="checkbox" bind:group={unorderedMetrics} value={metric} checked>
{:else}
<input type="checkbox" bind:group={unorderedMetrics} value={metric}>
{/if}
{metric}
<span style="float: right;">
{cluster == null ? clusters
.filter(cluster => cluster.metricConfig.find(m => m.name == metric) != null)
.map(cluster => cluster.name).join(', ') : ''}
</span>
</li>
{/each}
</ListGroup>
</ModalBody>
<ModalFooter>
<Button color="primary" on:click={closeAndApply}>Close & Apply</Button>
</ModalFooter>
</Modal>

94
web/frontend/src/Node.root.svelte Normal file
View File

@ -0,0 +1,94 @@
<script>
import { init } from './utils.js'
import { Row, Col, InputGroup, InputGroupText, Icon, Spinner, Card } from 'sveltestrap'
import { operationStore, query } from '@urql/svelte'
import TimeSelection from './filters/TimeSelection.svelte'
import PlotTable from './PlotTable.svelte'
import MetricPlot from './plots/MetricPlot.svelte'
import { getContext } from 'svelte'
export let cluster
export let hostname
export let from = null
export let to = null
const { query: initq } = init()
if (from == null || to == null) {
to = new Date(Date.now())
from = new Date(to.getTime())
from.setMinutes(from.getMinutes() - 30)
}
const ccconfig = getContext('cc-config'), clusters = getContext('clusters')
const nodesQuery = operationStore(`query($cluster: String!, $nodes: [String!], $from: Time!, $to: Time!) {
nodeMetrics(cluster: $cluster, nodes: $nodes, from: $from, to: $to) {
host, subCluster
metrics {
name,
metric {
timestep
scope
series {
statistics { min, avg, max }
data
}
}
}
}
}`, {
cluster: cluster,
nodes: [hostname],
from: from.toISOString(),
to: to.toISOString()
})
$: $nodesQuery.variables = { cluster, nodes: [hostname], from: from.toISOString(), to: to.toISOString() }
query(nodesQuery)
$: console.log($nodesQuery?.data?.nodeMetrics[0].metrics)
</script>
<Row>
{#if $initq.error}
<Card body color="danger">{$initq.error.message}</Card>
{:else if $initq.fetching}
<Spinner/>
{:else}
<Col>
<InputGroup>
<InputGroupText><Icon name="hdd"/></InputGroupText>
<InputGroupText>{hostname} ({cluster})</InputGroupText>
</InputGroup>
</Col>
<Col>
<TimeSelection
bind:from={from}
bind:to={to} />
</Col>
{/if}
</Row>
<br/>
<Row>
<Col>
{#if $nodesQuery.error}
<Card body color="danger">{$nodesQuery.error.message}</Card>
{:else if $nodesQuery.fetching || $initq.fetching}
<Spinner/>
{:else}
<PlotTable
let:item
let:width
itemsPerRow={ccconfig.plot_view_plotsPerRow}
items={$nodesQuery.data.nodeMetrics[0].metrics.sort((a, b) => a.name.localeCompare(b.name))}>
<h4 style="text-align: center;">{item.name}</h4>
<MetricPlot
width={width} height={300} metric={item.name} timestep={item.metric.timestep}
cluster={clusters.find(c => c.name == cluster)} subCluster={$nodesQuery.data.nodeMetrics[0].subCluster}
series={item.metric.series} />
</PlotTable>
{/if}
</Col>
</Row>

133
web/frontend/src/PlotSelection.svelte Normal file
View File

@ -0,0 +1,133 @@
<script>
import { Modal, ModalBody, ModalHeader, ModalFooter, InputGroup,
Button, ListGroup, ListGroupItem, Icon } from 'sveltestrap'
import { mutation } from '@urql/svelte'
export let availableMetrics
export let metricsInHistograms
export let metricsInScatterplots
const updateConfigurationMutation = mutation({
query: `mutation($name: String!, $value: String!) {
updateConfiguration(name: $name, value: $value)
}`
})
let isHistogramConfigOpen = false, isScatterPlotConfigOpen = false
let selectedMetric1 = null, selectedMetric2 = null
function updateConfiguration(data) {
updateConfigurationMutation({
name: data.name,
value: JSON.stringify(data.value)
})
.then(res => {
if (res.error)
console.error(res.error)
});
}
</script>
<Button outline
on:click={() => (isHistogramConfigOpen = true)}>
<Icon name=""/>
Select Plots for Histograms
</Button>
<Button outline
on:click={() => (isScatterPlotConfigOpen = true)}>
<Icon name=""/>
Select Plots in Scatter Plots
</Button>
<Modal isOpen={isHistogramConfigOpen}
toggle={() => (isHistogramConfigOpen = !isHistogramConfigOpen)}>
<ModalHeader>
Select metrics presented in histograms
</ModalHeader>
<ModalBody>
<ListGroup>
{#each availableMetrics as metric (metric)}
<ListGroupItem>
<input type="checkbox" bind:group={metricsInHistograms}
value={metric}
on:change={() => updateConfiguration({
name: 'analysis_view_histogramMetrics',
value: metricsInHistograms
})} />
{metric}
</ListGroupItem>
{/each}
</ListGroup>
</ModalBody>
<ModalFooter>
<Button color="primary"
on:click={() => (isHistogramConfigOpen = false)}>
Close
</Button>
</ModalFooter>
</Modal>
<Modal isOpen={isScatterPlotConfigOpen}
toggle={() => (isScatterPlotConfigOpen = !isScatterPlotConfigOpen)}>
<ModalHeader>
Select metric pairs presented in scatter plots
</ModalHeader>
<ModalBody>
<ListGroup>
{#each metricsInScatterplots as pair}
<ListGroupItem>
<b>{pair[0]}</b> / <b>{pair[1]}</b>
<Button style="float: right;" outline color="danger"
on:click={() => {
metricsInScatterplots = metricsInScatterplots.filter(p => pair != p)
updateConfiguration({
name: 'analysis_view_scatterPlotMetrics',
value: metricsInScatterplots
});
}}>
<Icon name="x" />
</Button>
</ListGroupItem>
{/each}
</ListGroup>
<br/>
<InputGroup>
<select bind:value={selectedMetric1} class="form-group form-select">
<option value={null}>Choose Metric for X Axis</option>
{#each availableMetrics as metric}
<option value={metric}>{metric}</option>
{/each}
</select>
<select bind:value={selectedMetric2} class="form-group form-select">
<option value={null}>Choose Metric for Y Axis</option>
{#each availableMetrics as metric}
<option value={metric}>{metric}</option>
{/each}
</select>
<Button outline disabled={selectedMetric1 == null || selectedMetric2 == null}
on:click={() => {
metricsInScatterplots = [...metricsInScatterplots, [selectedMetric1, selectedMetric2]]
selectedMetric1 = null
selectedMetric2 = null
updateConfiguration({
name: 'analysis_view_scatterPlotMetrics',
value: metricsInScatterplots
})
}}>
Add Plot
</Button>
</InputGroup>
</ModalBody>
<ModalFooter>
<Button color="primary"
on:click={() => (isScatterPlotConfigOpen = false)}>
Close
</Button>
</ModalFooter>
</Modal>

50
web/frontend/src/PlotTable.svelte Normal file
View File

@ -0,0 +1,50 @@
<!--
@component
Properties:
- itemsPerRow: Number
- items: [Any]
-->
<script>
export let itemsPerRow
export let items
export let padding = 10
let tableWidth = 0
const PLACEHOLDER = { magic: 'object' }
function tile(items, itemsPerRow) {
const rows = []
for (let ri = 0; ri < items.length; ri += itemsPerRow) {
const row = []
for (let ci = 0; ci < itemsPerRow; ci += 1) {
if (ri + ci < items.length)
row.push(items[ri + ci])
else
row.push(PLACEHOLDER)
}
rows.push(row)
}
return rows
}
$: rows = tile(items, itemsPerRow)
$: plotWidth = (tableWidth / itemsPerRow) - (padding * itemsPerRow)
</script>
<table bind:clientWidth={tableWidth} style="width: 100%; table-layout: fixed;">
{#each rows as row}
<tr>
{#each row as item (item)}
<td>
{#if item != PLACEHOLDER && plotWidth > 0}
<slot item={item} width={plotWidth}></slot>
{/if}
</td>
{/each}
</tr>
{/each}
</table>

122
web/frontend/src/StatsTable.svelte Normal file
View File

@ -0,0 +1,122 @@
<script>
import { getContext } from 'svelte'
import { Button, Table, InputGroup, InputGroupText, Icon } from 'sveltestrap'
import MetricSelection from './MetricSelection.svelte'
import StatsTableEntry from './StatsTableEntry.svelte'
import { maxScope } from './utils.js'
export let job
export let jobMetrics
const allMetrics = [...new Set(jobMetrics.map(m => m.name))].sort(),
scopesForMetric = (metric) => jobMetrics
.filter(jm => jm.name == metric)
.map(jm => jm.metric.scope)
let hosts = job.resources.map(r => r.hostname).sort(),
selectedScopes = {},
sorting = {},
isMetricSelectionOpen = false,
selectedMetrics = getContext('cc-config').job_view_nodestats_selectedMetrics
for (let metric of allMetrics) {
selectedScopes[metric] = maxScope(scopesForMetric(metric))
sorting[metric] = {
min: { dir: 'up', active: false },
avg: { dir: 'up', active: false },
max: { dir: 'up', active: false },
}
}
export function sortBy(metric, stat) {
let s = sorting[metric][stat]
if (s.active) {
s.dir = s.dir == 'up' ? 'down' : 'up'
} else {
for (let metric in sorting)
for (let stat in sorting[metric])
sorting[metric][stat].active = false
s.active = true
}
let series = jobMetrics.find(jm => jm.name == metric && jm.metric.scope == 'node')?.metric.series
sorting = {...sorting}
hosts = hosts.sort((h1, h2) => {
let s1 = series.find(s => s.hostname == h1)?.statistics
let s2 = series.find(s => s.hostname == h2)?.statistics
if (s1 == null || s2 == null)
return -1
return s.dir != 'up' ? s1[stat] - s2[stat] : s2[stat] - s1[stat]
})
}
export function moreLoaded(jobMetric) {
jobMetrics = [...jobMetrics, jobMetric]
}
</script>
<Table>
<thead>
<tr>
<th>
<Button outline on:click={() => (isMetricSelectionOpen = true, console.log(isMetricSelectionOpen))}>
Metrics
</Button>
</th>
{#each selectedMetrics as metric}
<th colspan={selectedScopes[metric] == 'node' ? 3 : 4}>
<InputGroup>
<InputGroupText>
{metric}
</InputGroupText>
<select class="form-select"
bind:value={selectedScopes[metric]}
disabled={scopesForMetric(metric, jobMetrics).length == 1}>
{#each scopesForMetric(metric, jobMetrics) as scope}
<option value={scope}>{scope}</option>
{/each}
</select>
</InputGroup>
</th>
{/each}
</tr>
<tr>
<th>Node</th>
{#each selectedMetrics as metric}
{#if selectedScopes[metric] != 'node'}
<th>Id</th>
{/if}
{#each ['min', 'avg', 'max'] as stat}
<th on:click={() => sortBy(metric, stat)}>
{stat}
{#if selectedScopes[metric] == 'node'}
<Icon name="caret-{sorting[metric][stat].dir}{sorting[metric][stat].active ? '-fill' : ''}" />
{/if}
</th>
{/each}
{/each}
</tr>
</thead>
<tbody>
{#each hosts as host (host)}
<tr>
<th scope="col">{host}</th>
{#each selectedMetrics as metric (metric)}
<StatsTableEntry
host={host} metric={metric}
scope={selectedScopes[metric]}
jobMetrics={jobMetrics} />
{/each}
</tr>
{/each}
</tbody>
</Table>
<br/>
<MetricSelection
configName='job_view_nodestats_selectedMetrics'
allMetrics={allMetrics}
bind:metrics={selectedMetrics}
bind:isOpen={isMetricSelectionOpen} />

37
web/frontend/src/StatsTableEntry.svelte Normal file
View File

@ -0,0 +1,37 @@
<script>
export let host
export let metric
export let scope
export let jobMetrics
$: series = jobMetrics
.find(jm => jm.name == metric && jm.metric.scope == scope)
?.metric.series.filter(s => s.hostname == host && s.statistics != null)
</script>
{#if series == null || series.length == 0}
<td colspan={scope == 'node' ? 3 : 4}><i>No data</i></td>
{:else if series.length == 1 && scope == 'node'}
<td>
{series[0].statistics.min}
</td>
<td>
{series[0].statistics.avg}
</td>
<td>
{series[0].statistics.max}
</td>
{:else}
<td colspan="4">
<table style="width: 100%;">
{#each series as s, i}
<tr>
<th>{s.id ?? i}</th>
<td>{s.statistics.min}</td>
<td>{s.statistics.avg}</td>
<td>{s.statistics.max}</td>
</tr>
{/each}
</table>
</td>
{/if}

184
web/frontend/src/Status.root.svelte Normal file
View File

@ -0,0 +1,184 @@
<script>
import Refresher from './joblist/Refresher.svelte'
import Roofline, { transformPerNodeData } from './plots/Roofline.svelte'
import Histogram from './plots/Histogram.svelte'
import { Row, Col, Spinner, Card, Table, Progress } from 'sveltestrap'
import { init } from './utils.js'
import { operationStore, query } from '@urql/svelte'
const { query: initq } = init()
export let cluster
let plotWidths = [], colWidth1 = 0, colWidth2
let from = new Date(Date.now() - 5 * 60 * 1000), to = new Date(Date.now())
const mainQuery = operationStore(`query($cluster: String!, $filter: [JobFilter!]!, $metrics: [String!], $from: Time!, $to: Time!) {
nodeMetrics(cluster: $cluster, metrics: $metrics, from: $from, to: $to) {
host,
subCluster,
metrics {
name,
metric {
scope
timestep,
series { data }
}
}
}
stats: jobsStatistics(filter: $filter) {
histDuration { count, value }
histNumNodes { count, value }
}
allocatedNodes(cluster: $cluster) { name, count }
topUsers: jobsCount(filter: $filter, groupBy: USER, weight: NODE_COUNT, limit: 10) { name, count }
topProjects: jobsCount(filter: $filter, groupBy: PROJECT, weight: NODE_COUNT, limit: 10) { name, count }
}`, {
cluster: cluster,
metrics: ['flops_any', 'mem_bw'],
from: from.toISOString(),
to: to.toISOString(),
filter: [{ state: ['running'] }, { cluster: { eq: cluster } }]
})
const sumUp = (data, subcluster, metric) => data.reduce((sum, node) => node.subCluster == subcluster
? sum + (node.metrics.find(m => m.name == metric)?.metric.series.reduce((sum, series) => sum + series.data[series.data.length - 1], 0) || 0)
: sum, 0)
let allocatedNodes = {}, flopRate = {}, memBwRate = {}
$: if ($initq.data && $mainQuery.data) {
let subClusters = $initq.data.clusters.find(c => c.name == cluster).subClusters
for (let subCluster of subClusters) {
allocatedNodes[subCluster.name] = $mainQuery.data.allocatedNodes.find(({ name }) => name == subCluster.name)?.count || 0
flopRate[subCluster.name] = Math.floor(sumUp($mainQuery.data.nodeMetrics, subCluster.name, 'flops_any') * 100) / 100
memBwRate[subCluster.name] = Math.floor(sumUp($mainQuery.data.nodeMetrics, subCluster.name, 'mem_bw') * 100) / 100
}
}
query(mainQuery)
</script>
<Row>
<Col xs="auto">
{#if $initq.fetching || $mainQuery.fetching}
<Spinner/>
{:else if $initq.error}
<Card body color="danger">{$initq.error.message}</Card>
{:else}
<!-- ... -->
{/if}
</Col>
<Col xs="auto" style="margin-left: auto;">
<Refresher initially={120} on:reload={() => {
console.log('reload...')
from = new Date(Date.now() - 5 * 60 * 1000)
to = new Date(Date.now())
$mainQuery.variables = { ...$mainQuery.variables, from: from, to: to }
$mainQuery.reexecute({ requestPolicy: 'network-only' })
}} />
</Col>
</Row>
{#if $mainQuery.error}
<Row>
<Col>
<Card body color="danger">{$mainQuery.error.message}</Card>
</Col>
</Row>
{/if}
{#if $initq.data && $mainQuery.data}
{#each $initq.data.clusters.find(c => c.name == cluster).subClusters as subCluster, i}
<Row>
<Col xs="3">
<Table>
<tr>
<th scope="col">SubCluster</th>
<td colspan="2">{subCluster.name}</td>
</tr>
<tr>
<th scope="col">Allocated Nodes</th>
<td style="min-width: 75px;"><div class="col"><Progress value={allocatedNodes[subCluster.name]} max={subCluster.numberOfNodes}/></div></td>
<td>({allocatedNodes[subCluster.name]} / {subCluster.numberOfNodes})</td>
</tr>
<tr>
<th scope="col">Flop Rate</th>
<td style="min-width: 75px;"><div class="col"><Progress value={flopRate[subCluster.name]} max={subCluster.flopRateSimd * subCluster.numberOfNodes}/></div></td>
<td>({flopRate[subCluster.name]} / {subCluster.flopRateSimd * subCluster.numberOfNodes})</td>
</tr>
<tr>
<th scope="col">MemBw Rate</th>
<td style="min-width: 75px;"><div class="col"><Progress value={memBwRate[subCluster.name]} max={subCluster.memoryBandwidth * subCluster.numberOfNodes}/></div></td>
<td>({memBwRate[subCluster.name]} / {subCluster.memoryBandwidth * subCluster.numberOfNodes})</td>
</tr>
</Table>
</Col>
<div class="col-9" bind:clientWidth={plotWidths[i]}>
{#key $mainQuery.data.nodeMetrics}
<Roofline
width={plotWidths[i] - 10} height={300} colorDots={false} cluster={subCluster}
data={transformPerNodeData($mainQuery.data.nodeMetrics.filter(data => data.subCluster == subCluster.name))} />
{/key}
</div>
</Row>
{/each}
<Row>
<div class="col-4" bind:clientWidth={colWidth1}>
<h4>Top Users</h4>
{#key $mainQuery.data}
<Histogram
width={colWidth1 - 25} height={300}
data={$mainQuery.data.topUsers.sort((a, b) => b.count - a.count).map(({ count }, idx) => ({ count, value: idx }))}
label={(x) => x < $mainQuery.data.topUsers.length ? $mainQuery.data.topUsers[Math.floor(x)].name : '0'} />
{/key}
</div>
<div class="col-2">
<Table>
<tr><th>Name</th><th>Number of Nodes</th></tr>
{#each $mainQuery.data.topUsers.sort((a, b) => b.count - a.count) as { name, count }}
<tr>
<th scope="col"><a href="/monitoring/user/{name}">{name}</a></th>
<td>{count}</td>
</tr>
{/each}
</Table>
</div>
<div class="col-4">
<h4>Top Projects</h4>
{#key $mainQuery.data}
<Histogram
width={colWidth1 - 25} height={300}
data={$mainQuery.data.topProjects.sort((a, b) => b.count - a.count).map(({ count }, idx) => ({ count, value: idx }))}
label={(x) => x < $mainQuery.data.topProjects.length ? $mainQuery.data.topProjects[Math.floor(x)].name : '0'} />
{/key}
</div>
<div class="col-2">
<Table>
<tr><th>Name</th><th>Number of Nodes</th></tr>
{#each $mainQuery.data.topProjects.sort((a, b) => b.count - a.count) as { name, count }}
<tr><th scope="col">{name}</th><td>{count}</td></tr>
{/each}
</Table>
</div>
</Row>
<Row>
<div class="col" bind:clientWidth={colWidth2}>
<h4>Duration Distribution</h4>
{#key $mainQuery.data.stats}
<Histogram
width={colWidth2 - 25} height={300}
data={$mainQuery.data.stats[0].histDuration} />
{/key}
</div>
<div class="col">
<h4>Number of Nodes Distribution</h4>
{#key $mainQuery.data.stats}
<Histogram
width={colWidth2 - 25} height={300}
data={$mainQuery.data.stats[0].histNumNodes} />
{/key}
</div>
</Row>
{/if}

118
web/frontend/src/Systems.root.svelte Normal file
View File

@ -0,0 +1,118 @@
<script>
import { init } from './utils.js'
import { Row, Col, Input, InputGroup, InputGroupText, Icon, Spinner, Card } from 'sveltestrap'
import { operationStore, query } from '@urql/svelte'
import TimeSelection from './filters/TimeSelection.svelte'
import PlotTable from './PlotTable.svelte'
import MetricPlot from './plots/MetricPlot.svelte'
import { getContext } from 'svelte'
export let cluster
export let from = null
export let to = null
const { query: initq } = init()
if (from == null || to == null) {
to = new Date(Date.now())
from = new Date(to.getTime())
from.setMinutes(from.getMinutes() - 30)
}
const clusters = getContext('clusters')
const ccconfig = getContext('cc-config')
let plotHeight = 300
let hostnameFilter = ''
let selectedMetric = ccconfig.system_view_selectedMetric
const nodesQuery = operationStore(`query($cluster: String!, $metrics: [String!], $from: Time!, $to: Time!) {
nodeMetrics(cluster: $cluster, metrics: $metrics, from: $from, to: $to) {
host,
subCluster
metrics {
name,
metric {
scope
timestep,
series {
statistics { min, avg, max }
data
}
}
}
}
}`, {
cluster: cluster,
metrics: [],
from: from.toISOString(),
to: to.toISOString()
})
$: $nodesQuery.variables = { cluster, metrics: [selectedMetric], from: from.toISOString(), to: to.toISOString() }
query(nodesQuery)
</script>
<Row>
{#if $initq.error}
<Card body color="danger">{$initq.error.message}</Card>
{:else if $initq.fetching}
<Spinner/>
{:else}
<Col>
<TimeSelection
bind:from={from}
bind:to={to} />
</Col>
<Col>
<InputGroup>
<InputGroupText><Icon name="graph-up" /></InputGroupText>
<InputGroupText>Metric</InputGroupText>
<select class="form-select" bind:value={selectedMetric}>
{#each clusters.find(c => c.name == cluster).metricConfig as metric}
<option value={metric.name}>{metric.name} ({metric.unit})</option>
{/each}
</select>
</InputGroup>
</Col>
<Col>
<InputGroup>
<InputGroupText><Icon name="hdd" /></InputGroupText>
<InputGroupText>Find Node</InputGroupText>
<Input placeholder="hostname..." type="text" bind:value={hostnameFilter} />
</InputGroup>
</Col>
{/if}
</Row>
<br/>
<Row>
<Col>
{#if $nodesQuery.error}
<Card body color="danger">{$nodesQuery.error.message}</Card>
{:else if $nodesQuery.fetching || $initq.fetching}
<Spinner/>
{:else}
<PlotTable
let:item
let:width
itemsPerRow={ccconfig.plot_view_plotsPerRow}
items={$nodesQuery.data.nodeMetrics
.filter(h => h.host.includes(hostnameFilter) && h.metrics.some(m => m.name == selectedMetric && m.metric.scope == 'node'))
.map(h => ({ host: h.host, subCluster: h.subCluster, data: h.metrics.find(m => m.name == selectedMetric && m.metric.scope == 'node') }))
.sort((a, b) => a.host.localeCompare(b.host))}>
<h4 style="width: 100%; text-align: center;"><a href="/monitoring/node/{cluster}/{item.host}">{item.host} ({item.subCluster})</a></h4>
<MetricPlot
width={width}
height={plotHeight}
timestep={item.data.metric.timestep}
series={item.data.metric.series}
metric={item.data.name}
cluster={clusters.find(c => c.name == cluster)}
subCluster={item.subCluster} />
</PlotTable>
{/if}
</Col>
</Row>

44
web/frontend/src/Tag.svelte Normal file
View File

@ -0,0 +1,44 @@
<!--
@component
Properties:
- id: ID! (if the tag-id is known but not the tag type/name, this can be used)
- tag: { id: ID!, type: String, name: String }
- clickable: Boolean (default is true)
-->
<script>
import { getContext } from 'svelte'
const allTags = getContext('tags'),
initialized = getContext('initialized')
export let id = null
export let tag = null
export let clickable = true
if (tag != null && id == null)
id = tag.id
$: {
if ($initialized && tag == null)
tag = allTags.find(tag => tag.id == id)
}
</script>
<style>
a {
margin-left: 0.5rem;
line-height: 2;
}
span {
font-size: 0.9rem;
}
</style>
<a target={clickable ? "_blank" : null} href={clickable ? `/monitoring/jobs/?tag=${id}` : null}>
{#if tag}
<span class="badge bg-warning text-dark">{tag.type}: {tag.name}</span>
{:else}
Loading...
{/if}
</a>

173
web/frontend/src/TagManagement.svelte Normal file
View File

@ -0,0 +1,173 @@
<script>
import { getContext } from 'svelte'
import { mutation } from '@urql/svelte'
import { Icon, Button, ListGroupItem, Spinner, Modal, Input,
ModalBody, ModalHeader, ModalFooter, Alert } from 'sveltestrap'
import { fuzzySearchTags } from './utils.js'
import Tag from './Tag.svelte'
export let job
export let jobTags = job.tags
let allTags = getContext('tags'), initialized = getContext('initialized')
let newTagType = '', newTagName = ''
let filterTerm = ''
let pendingChange = false
let isOpen = false
const createTagMutation = mutation({
query: `mutation($type: String!, $name: String!) {
createTag(type: $type, name: $name) { id, type, name }
}`
})
const addTagsToJobMutation = mutation({
query: `mutation($job: ID!, $tagIds: [ID!]!) {
addTagsToJob(job: $job, tagIds: $tagIds) { id, type, name }
}`
})
const removeTagsFromJobMutation = mutation({
query: `mutation($job: ID!, $tagIds: [ID!]!) {
removeTagsFromJob(job: $job, tagIds: $tagIds) { id, type, name }
}`
})
let allTagsFiltered // $initialized is in there because when it becomes true, allTags is initailzed.
$: allTagsFiltered = ($initialized, fuzzySearchTags(filterTerm, allTags))
$: {
newTagType = '';
newTagName = '';
let parts = filterTerm.split(':').map(s => s.trim())
if (parts.length == 2 && parts.every(s => s.length > 0)) {
newTagType = parts[0]
newTagName = parts[1]
}
}
function isNewTag(type, name) {
for (let tag of allTagsFiltered)
if (tag.type == type && tag.name == name)
return false
return true
}
function createTag(type, name) {
pendingChange = true
return createTagMutation({ type: type, name: name })
.then(res => {
if (res.error)
throw res.error
pendingChange = false
allTags = [...allTags, res.data.createTag]
newTagType = ''
newTagName = ''
return res.data.createTag
}, err => console.error(err))
}
function addTagToJob(tag) {
pendingChange = tag.id
addTagsToJobMutation({ job: job.id, tagIds: [tag.id] })
.then(res => {
if (res.error)
throw res.error
jobTags = job.tags = res.data.addTagsToJob;
pendingChange = false;
})
.catch(err => console.error(err))
}
function removeTagFromJob(tag) {
pendingChange = tag.id
removeTagsFromJobMutation({ job: job.id, tagIds: [tag.id] })
.then(res => {
if (res.error)
throw res.error
jobTags = job.tags = res.data.removeTagsFromJob
pendingChange = false
})
.catch(err => console.error(err))
}
</script>
<style>
ul.list-group {
max-height: 450px;
margin-bottom: 10px;
overflow: scroll;
}
</style>
<Modal {isOpen} toggle={() => (isOpen = !isOpen)}>
<ModalHeader>
Manage Tags
{#if pendingChange !== false}
<Spinner size="sm" secondary />
{:else}
<Icon name="tags" />
{/if}
</ModalHeader>
<ModalBody>
<Input style="width: 100%;"
type="text" placeholder="Search Tags"
bind:value={filterTerm} />
<br/>
<Alert color="info">
Search using "<code>type: name</code>". If no tag matches your search,
a button for creating a new one will appear.
</Alert>
<ul class="list-group">
{#each allTagsFiltered as tag}
<ListGroupItem>
<Tag tag={tag}/>
<span style="float: right;">
{#if pendingChange === tag.id}
<Spinner size="sm" secondary />
{:else if job.tags.find(t => t.id == tag.id)}
<Button size="sm" outline color="danger"
on:click={() => removeTagFromJob(tag)}>
<Icon name="x" />
</Button>
{:else}
<Button size="sm" outline color="success"
on:click={() => addTagToJob(tag)}>
<Icon name="plus" />
</Button>
{/if}
</span>
</ListGroupItem>
{:else}
<ListGroupItem disabled>
<i>No tags matching</i>
</ListGroupItem>
{/each}
</ul>
<br/>
{#if newTagType && newTagName && isNewTag(newTagType, newTagName)}
<Button outline color="success"
on:click={e => (e.preventDefault(), createTag(newTagType, newTagName))
.then(tag => addTagToJob(tag))}>
Create & Add Tag:
<Tag tag={({ type: newTagType, name: newTagName })} clickable={false}/>
</Button>
{:else if allTagsFiltered.length == 0}
<Alert>Search Term is not a valid Tag (<code>type: name</code>)</Alert>
{/if}
</ModalBody>
<ModalFooter>
<Button color="primary" on:click={() => (isOpen = false)}>Close</Button>
</ModalFooter>
</Modal>
<Button outline on:click={() => (isOpen = true)}>
Manage Tags <Icon name="tags" />
</Button>

172
web/frontend/src/User.root.svelte Normal file
View File

@ -0,0 +1,172 @@
<script>
import { onMount, getContext } from 'svelte'
import { init } from './utils.js'
import { Table, Row, Col, Button, Icon, Card, Spinner } from 'sveltestrap'
import { operationStore, query } from '@urql/svelte'
import Filters from './filters/Filters.svelte'
import JobList from './joblist/JobList.svelte'
import Sorting from './joblist/SortSelection.svelte'
import Refresher from './joblist/Refresher.svelte'
import Histogram from './plots/Histogram.svelte'
import MetricSelection from './MetricSelection.svelte'
import { scramble, scrambleNames } from './joblist/JobInfo.svelte'
const { query: initq } = init()
const ccconfig = getContext('cc-config')
export let user
export let filterPresets
let filters, jobList
let sorting = { field: 'startTime', order: 'DESC' }, isSortingOpen = false
let metrics = ccconfig.plot_list_selectedMetrics, isMetricsSelectionOpen = false
let w1, w2, histogramHeight = 250
const stats = operationStore(`
query($filter: [JobFilter!]!) {
jobsStatistics(filter: $filter) {
totalJobs
shortJobs
totalWalltime
totalCoreHours
histDuration { count, value }
histNumNodes { count, value }
}
}
`, {
filter: []
}, {
pause: true
})
query(stats)
onMount(() => filters.update())
</script>
<Row>
{#if $initq.fetching}
<Col>
<Spinner/>
</Col>
{:else if $initq.error}
<Col xs="auto">
<Card body color="danger">{$initq.error.message}</Card>
</Col>
{/if}
<Col xs="auto">
<Button
outline color="primary"
on:click={() => (isSortingOpen = true)}>
<Icon name="sort-up"/> Sorting
</Button>
<Button
outline color="primary"
on:click={() => (isMetricsSelectionOpen = true)}>
<Icon name="graph-up"/> Metrics
</Button>
</Col>
<Col xs="auto">
<Filters
filterPresets={filterPresets}
startTimeQuickSelect={true}
bind:this={filters}
on:update={({ detail }) => {
let filters = [...detail.filters, { user: { eq: user.username } }]
$stats.variables = { filter: filters }
$stats.context.pause = false
$stats.reexecute()
jobList.update(filters)
}} />
</Col>
<Col xs="auto" style="margin-left: auto;">
<Refresher on:reload={() => jobList.update()} />
</Col>
</Row>
<br/>
<Row>
{#if $stats.error}
<Col>
<Card body color="danger">{$stats.error.message}</Card>
</Col>
{:else if !$stats.data}
<Col>
<Spinner secondary />
</Col>
{:else}
<Col xs="4">
<Table>
<tbody>
<tr>
<th scope="row">Username</th>
<td>{scrambleNames ? scramble(user.username) : user.username}</td>
</tr>
{#if user.name}
<tr>
<th scope="row">Name</th>
<td>{scrambleNames ? scramble(user.name) : user.name}</td>
</tr>
{/if}
{#if user.email}
<tr>
<th scope="row">Email</th>
<td>{user.email}</td>
</tr>
{/if}
<tr>
<th scope="row">Total Jobs</th>
<td>{$stats.data.jobsStatistics[0].totalJobs}</td>
</tr>
<tr>
<th scope="row">Short Jobs</th>
<td>{$stats.data.jobsStatistics[0].shortJobs}</td>
</tr>
<tr>
<th scope="row">Total Walltime</th>
<td>{$stats.data.jobsStatistics[0].totalWalltime}</td>
</tr>
<tr>
<th scope="row">Total Core Hours</th>
<td>{$stats.data.jobsStatistics[0].totalCoreHours}</td>
</tr>
</tbody>
</Table>
</Col>
<div class="col-4" style="text-align: center;" bind:clientWidth={w1}>
<b>Walltime</b>
{#key $stats.data.jobsStatistics[0].histDuration}
<Histogram
data={$stats.data.jobsStatistics[0].histDuration}
width={w1 - 25} height={histogramHeight} />
{/key}
</div>
<div class="col-4" style="text-align: center;" bind:clientWidth={w2}>
<b>Number of Nodes</b>
{#key $stats.data.jobsStatistics[0].histNumNodes}
<Histogram
data={$stats.data.jobsStatistics[0].histNumNodes}
width={w2 - 25} height={histogramHeight} />
{/key}
</div>
{/if}
</Row>
<br/>
<Row>
<Col>
<JobList
bind:metrics={metrics}
bind:sorting={sorting}
bind:this={jobList} />
</Col>
</Row>
<Sorting
bind:sorting={sorting}
bind:isOpen={isSortingOpen} />
<MetricSelection configName="plot_list_selectedMetrics"
bind:metrics={metrics}
bind:isOpen={isMetricsSelectionOpen} />

60
web/frontend/src/Zoom.svelte Normal file
View File

@ -0,0 +1,60 @@
<script>
import { Icon, InputGroup, InputGroupText } from 'sveltestrap';
export let timeseriesPlots;
let windowSize = 100; // Goes from 0 to 100
let windowPosition = 50; // Goes from 0 to 100
function updatePlots() {
let ws = windowSize / (100 * 2),
wp = windowPosition / 100;
let from = (wp - ws),
to = (wp + ws);
Object
.values(timeseriesPlots)
.forEach(plot => plot.setTimeRange(from, to));
}
// Rendering a big job can take a long time, so we
// throttle the rerenders to every 100ms here.
let timeoutId = null;
function requestUpdatePlots() {
if (timeoutId != null)
window.cancelAnimationFrame(timeoutId);
timeoutId = window.requestAnimationFrame(() => {
updatePlots();
timeoutId = null;
}, 100);
}
$: requestUpdatePlots(windowSize, windowPosition);
</script>
<div>
<InputGroup>
<InputGroupText>
<Icon name="zoom-in"/>
</InputGroupText>
<InputGroupText>
Window Size:
<input
style="margin: 0em 0em 0em 1em"
type="range"
bind:value={windowSize}
min=1 max=100 step=1 />
<span style="width: 5em;">
({windowSize}%)
</span>
</InputGroupText>
<InputGroupText>
Window Position:
<input
style="margin: 0em 0em 0em 1em"
type="range"
bind:value={windowPosition}
min=0 max=100 step=1 />
</InputGroupText>
</InputGroup>
</div>

14
web/frontend/src/analysis.entrypoint.js Normal file
View File

@ -0,0 +1,14 @@
import {} from './header.entrypoint.js'
import Analysis from './Analysis.root.svelte'
filterPresets.cluster = cluster
new Analysis({
target: document.getElementById('svelte-app'),
props: {
filterPresets: filterPresets
},
context: new Map([
['cc-config', clusterCockpitConfig]
])
})

72
web/frontend/src/cache-exchange.js Normal file
View File

@ -0,0 +1,72 @@
import { filter, map, merge, pipe, share, tap } from 'wonka';
/*
* Alternative to the default cacheExchange from urql (A GraphQL client).
* Mutations do not invalidate cached results, so in that regard, this
* implementation is inferior to the default one. Most people should probably
* use the standard cacheExchange and @urql/exchange-request-policy. This cache
* also ignores the 'network-and-cache' request policy.
*
* Options:
* ttl: How long queries are allowed to be cached (in milliseconds)
* maxSize: Max number of results cached. The oldest queries are removed first.
*/
export const expiringCacheExchange = ({ ttl, maxSize }) => ({ forward }) => {
const cache = new Map();
const isCached = (operation) => {
if (operation.kind !== 'query' || operation.context.requestPolicy === 'network-only')
return false;
if (!cache.has(operation.key))
return false;
let cacheEntry = cache.get(operation.key);
return Date.now() < cacheEntry.expiresAt;
};
return operations => {
let shared = share(operations);
return merge([
pipe(
shared,
filter(operation => isCached(operation)),
map(operation => cache.get(operation.key).response)
),
pipe(
shared,
filter(operation => !isCached(operation)),
forward,
tap(response => {
if (!response.operation || response.operation.kind !== 'query')
return;
if (!response.data)
return;
let now = Date.now();
for (let cacheEntry of cache.values()) {
if (cacheEntry.expiresAt < now) {
cache.delete(cacheEntry.response.operation.key);
}
}
if (cache.size > maxSize) {
let n = cache.size - maxSize + 1;
for (let key of cache.keys()) {
if (n-- == 0)
break;
cache.delete(key);
}
}
cache.set(response.operation.key, {
expiresAt: now + ttl,
response: response
});
})
)
]);
};
};

77
web/frontend/src/filters/Cluster.svelte Normal file
View File

@ -0,0 +1,77 @@
<script>
import { createEventDispatcher, getContext } from 'svelte'
import { Button, ListGroup, ListGroupItem,
Modal, ModalBody, ModalHeader, ModalFooter } from 'sveltestrap'
const clusters = getContext('clusters'),
initialized = getContext('initialized'),
dispatch = createEventDispatcher()
export let disableClusterSelection = false
export let isModified = false
export let isOpen = false
export let cluster = null
export let partition = null
let pendingCluster = cluster, pendingPartition = partition
$: isModified = pendingCluster != cluster || pendingPartition != partition
</script>
<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
<ModalHeader>
Select Cluster & Slurm Partition
</ModalHeader>
<ModalBody>
{#if $initialized}
<h4>Cluster</h4>
<ListGroup>
<ListGroupItem
disabled={disableClusterSelection}
active={pendingCluster == null}
on:click={() => (pendingCluster = null, pendingPartition = null)}>
Any Cluster
</ListGroupItem>
{#each clusters as cluster}
<ListGroupItem
disabled={disableClusterSelection}
active={pendingCluster == cluster.name}
on:click={() => (pendingCluster = cluster.name, pendingPartition = null)}>
{cluster.name}
</ListGroupItem>
{/each}
</ListGroup>
{/if}
{#if $initialized && pendingCluster != null}
<br/>
<h4>Partiton</h4>
<ListGroup>
<ListGroupItem
active={pendingPartition == null}
on:click={() => (pendingPartition = null)}>
Any Partition
</ListGroupItem>
{#each clusters.find(c => c.name == pendingCluster).partitions as partition}
<ListGroupItem
active={pendingPartition == partition}
on:click={() => (pendingPartition = partition)}>
{partition}
</ListGroupItem>
{/each}
</ListGroup>
{/if}
</ModalBody>
<ModalFooter>
<Button color="primary" on:click={() => {
isOpen = false
cluster = pendingCluster
partition = pendingPartition
dispatch('update', { cluster, partition })
}}>Close & Apply</Button>
<Button color="danger" on:click={() => {
isOpen = false
cluster = pendingCluster = null
partition = pendingPartition = null
dispatch('update', { cluster, partition })
}}>Reset</Button>
<Button on:click={() => (isOpen = false)}>Close</Button>
</ModalFooter>
</Modal>

302
web/frontend/src/filters/DoubleRangeSlider.svelte Normal file
View File

@ -0,0 +1,302 @@
<!--
Copyright (c) 2021 Michael Keller
Originally created by Michael Keller (https://github.com/mhkeller/svelte-double-range-slider)
Changes: remove dependency, text inputs, configurable value ranges, on:change event
-->
<!--
@component
Properties:
- min: Number
- max: Number
- firstSlider: Number (Starting position of slider #1)
- secondSlider: Number (Starting position of slider #2)
Events:
- `change`: [Number, Number] (Positions of the two sliders)
-->
<script>
import { createEventDispatcher } from "svelte";
export let min;
export let max;
export let firstSlider;
export let secondSlider;
const dispatch = createEventDispatcher();
let values;
let start, end; /* Positions of sliders from 0 to 1 */
$: values = [firstSlider, secondSlider]; /* Avoid feedback loop */
$: start = Math.max(((firstSlider == null ? min : firstSlider) - min) / (max - min), 0);
$: end = Math.min(((secondSlider == null ? min : secondSlider) - min) / (max - min), 1);
let leftHandle;
let body;
let slider;
let inputFieldFrom, inputFieldTo;
let timeoutId = null;
function queueChangeEvent() {
if (timeoutId !== null) {
clearTimeout(timeoutId);
}
timeoutId = setTimeout(() => {
timeoutId = null;
// Show selection but avoid feedback loop
if (values[0] != null && inputFieldFrom.value != values[0].toString())
inputFieldFrom.value = values[0].toString();
if (values[1] != null && inputFieldTo.value != values[1].toString())
inputFieldTo.value = values[1].toString();
dispatch('change', values);
}, 250);
}
function update() {
values = [
Math.floor(min + start * (max - min)),
Math.floor(min + end * (max - min))
];
queueChangeEvent();
}
function inputChanged(idx, event) {
let val = Number.parseInt(event.target.value);
if (Number.isNaN(val) || val < min) {
event.target.classList.add('bad');
return;
}
values[idx] = val;
event.target.classList.remove('bad');
if (idx == 0)
start = clamp((val - min) / (max - min), 0., 1.);
else
end = clamp((val - min) / (max - min), 0., 1.);
queueChangeEvent();
}
function clamp(x, min, max) {
return x < min
? min
: (x < max ? x : max);
}
function draggable(node) {
let x;
let y;
function handleMousedown(event) {
if (event.type === 'touchstart') {
event = event.touches[0];
}
x = event.clientX;
y = event.clientY;
node.dispatchEvent(new CustomEvent('dragstart', {
detail: { x, y }
}));
window.addEventListener('mousemove', handleMousemove);
window.addEventListener('mouseup', handleMouseup);
window.addEventListener('touchmove', handleMousemove);
window.addEventListener('touchend', handleMouseup);
}
function handleMousemove(event) {
if (event.type === 'touchmove') {
event = event.changedTouches[0];
}
const dx = event.clientX - x;
const dy = event.clientY - y;
x = event.clientX;
y = event.clientY;
node.dispatchEvent(new CustomEvent('dragmove', {
detail: { x, y, dx, dy }
}));
}
function handleMouseup(event) {
x = event.clientX;
y = event.clientY;
node.dispatchEvent(new CustomEvent('dragend', {
detail: { x, y }
}));
window.removeEventListener('mousemove', handleMousemove);
window.removeEventListener('mouseup', handleMouseup);
window.removeEventListener('touchmove', handleMousemove);
window.removeEventListener('touchend', handleMouseup);
}
node.addEventListener('mousedown', handleMousedown);
node.addEventListener('touchstart', handleMousedown);
return {
destroy() {
node.removeEventListener('mousedown', handleMousedown);
node.removeEventListener('touchstart', handleMousedown);
}
};
}
function setHandlePosition (which) {
return function (evt) {
const { left, right } = slider.getBoundingClientRect();
const parentWidth = right - left;
const p = Math.min(Math.max((evt.detail.x - left) / parentWidth, 0), 1);
if (which === 'start') {
start = p;
end = Math.max(end, p);
} else {
start = Math.min(p, start);
end = p;
}
update();
}
}
function setHandlesFromBody (evt) {
const { width } = body.getBoundingClientRect();
const { left, right } = slider.getBoundingClientRect();
const parentWidth = right - left;
const leftHandleLeft = leftHandle.getBoundingClientRect().left;
const pxStart = clamp((leftHandleLeft + event.detail.dx) - left, 0, parentWidth - width);
const pxEnd = clamp(pxStart + width, width, parentWidth);
const pStart = pxStart / parentWidth;
const pEnd = pxEnd / parentWidth;
start = pStart;
end = pEnd;
update();
}
</script>
<div class="double-range-container">
<div class="header">
<input class="form-control" type="text" placeholder="from..." bind:this={inputFieldFrom}
on:input={(e) => inputChanged(0, e)} />
<span>Full Range: <b> {min} </b> - <b> {max} </b></span>
<input class="form-control" type="text" placeholder="to..." bind:this={inputFieldTo}
on:input={(e) => inputChanged(1, e)} />
</div>
<div class="slider" bind:this={slider}>
<div
class="body"
bind:this={body}
use:draggable
on:dragmove|preventDefault|stopPropagation="{setHandlesFromBody}"
style="
left: {100 * start}%;
right: {100 * (1 - end)}%;
"
></div>
<div
class="handle"
bind:this={leftHandle}
data-which="start"
use:draggable
on:dragmove|preventDefault|stopPropagation="{setHandlePosition('start')}"
style="
left: {100 * start}%
"
></div>
<div
class="handle"
data-which="end"
use:draggable
on:dragmove|preventDefault|stopPropagation="{setHandlePosition('end')}"
style="
left: {100 * end}%
"
></div>
</div>
</div>
<style>
.header {
width: 100%;
display: flex;
justify-content: space-between;
margin-bottom: -5px;
}
.header :nth-child(2) {
padding-top: 10px;
}
.header input {
height: 25px;
border-radius: 5px;
width: 100px;
}
:global(.double-range-container .header input[type="text"].bad) {
color: #ff5c33;
border-color: #ff5c33;
}
.double-range-container {
width: 100%;
height: 50px;
user-select: none;
box-sizing: border-box;
white-space: nowrap
}
.slider {
position: relative;
width: 100%;
height: 6px;
top: 10px;
transform: translate(0, -50%);
background-color: #e2e2e2;
box-shadow: inset 0 7px 10px -5px #4a4a4a, inset 0 -1px 0px 0px #9c9c9c;
border-radius: 6px;
}
.handle {
position: absolute;
top: 50%;
width: 0;
height: 0;
}
.handle:after {
content: ' ';
box-sizing: border-box;
position: absolute;
border-radius: 50%;
width: 16px;
height: 16px;
background-color: #fdfdfd;
border: 1px solid #7b7b7b;
transform: translate(-50%, -50%)
}
/* .handle[data-which="end"]:after{
transform: translate(-100%, -50%);
} */
.handle:active:after {
background-color: #ddd;
z-index: 9;
}
.body {
top: 0;
position: absolute;
background-color: #34a1ff;
bottom: 0;
}
</style>

95
web/frontend/src/filters/Duration.svelte Normal file
View File

@ -0,0 +1,95 @@
<script>
import { createEventDispatcher } from 'svelte'
import { Row, Col, Button, Modal, ModalBody, ModalHeader, ModalFooter, FormGroup } from 'sveltestrap'
const dispatch = createEventDispatcher()
export let isOpen = false
export let from = null
export let to = null
let pendingFrom, pendingTo
function reset() {
pendingFrom = from == null ? { hours: 0, mins: 0 } : secsToHoursAndMins(from)
pendingTo = to == null ? { hours: 0, mins: 0 } : secsToHoursAndMins(to)
}
reset()
function secsToHoursAndMins(duration) {
const hours = Math.floor(duration / 3600)
duration -= hours * 3600
const mins = Math.floor(duration / 60)
return { hours, mins }
}
function hoursAndMinsToSecs({ hours, mins }) {
return hours * 3600 + mins * 60
}
</script>
<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
<ModalHeader>
Select Start Time
</ModalHeader>
<ModalBody>
<h4>Between</h4>
<Row>
<Col>
<div class="input-group mb-2 mr-sm-2">
<input type="number" class="form-control" bind:value={pendingFrom.hours}>
<div class="input-group-append">
<div class="input-group-text">h</div>
</div>
</div>
</Col>
<Col>
<div class="input-group mb-2 mr-sm-2">
<input type="number" class="form-control" bind:value={pendingFrom.mins}>
<div class="input-group-append">
<div class="input-group-text">m</div>
</div>
</div>
</Col>
</Row>
<h4>and</h4>
<Row>
<Col>
<div class="input-group mb-2 mr-sm-2">
<input type="number" class="form-control" bind:value={pendingTo.hours}>
<div class="input-group-append">
<div class="input-group-text">h</div>
</div>
</div>
</Col>
<Col>
<div class="input-group mb-2 mr-sm-2">
<input type="number" class="form-control" bind:value={pendingTo.mins}>
<div class="input-group-append">
<div class="input-group-text">m</div>
</div>
</div>
</Col>
</Row>
</ModalBody>
<ModalFooter>
<Button color="primary"
on:click={() => {
isOpen = false
from = hoursAndMinsToSecs(pendingFrom)
to = hoursAndMinsToSecs(pendingTo)
dispatch('update', { from, to })
}}>
Close & Apply
</Button>
<Button color="danger" on:click={() => {
isOpen = false
from = null
to = null
reset()
dispatch('update', { from, to })
}}>Reset</Button>
<Button on:click={() => (isOpen = false)}>Close</Button>
</ModalFooter>
</Modal>

323
web/frontend/src/filters/Filters.svelte Normal file
View File

@ -0,0 +1,323 @@
<!--
@component
Properties:
- menuText: String? (Optional text to show in the dropdown menu)
- filterPresets: Object? (Optional predefined filter values)
Events:
- 'update': The detail's 'filters' prop are new filter items to be applied
Functions:
- void update(additionalFilters: Object?): Triggers an update
-->
<script>
import { Row, Col, DropdownItem, DropdownMenu,
DropdownToggle, ButtonDropdown, Icon } from 'sveltestrap'
import { createEventDispatcher } from 'svelte'
import Info from './InfoBox.svelte'
import Cluster from './Cluster.svelte'
import JobStates, { allJobStates } from './JobStates.svelte'
import StartTime from './StartTime.svelte'
import Tags from './Tags.svelte'
import Tag from '../Tag.svelte'
import Duration from './Duration.svelte'
import Resources from './Resources.svelte'
import Statistics from './Stats.svelte'
// import TimeSelection from './TimeSelection.svelte'
const dispatch = createEventDispatcher()
export let menuText = null
export let filterPresets = {}
export let disableClusterSelection = false
export let startTimeQuickSelect = false
let filters = {
projectMatch: filterPresets.projectMatch || 'contains',
userMatch: filterPresets.userMatch || 'contains',
cluster: filterPresets.cluster || null,
partition: filterPresets.partition || null,
states: filterPresets.states || filterPresets.state ? [filterPresets.state].flat() : allJobStates,
startTime: filterPresets.startTime || { from: null, to: null },
tags: filterPresets.tags || [],
duration: filterPresets.duration || { from: null, to: null },
jobId: filterPresets.jobId || '',
arrayJobId: filterPresets.arrayJobId || null,
user: filterPresets.user || '',
project: filterPresets.project || '',
numNodes: filterPresets.numNodes || { from: null, to: null },
numHWThreads: filterPresets.numHWThreads || { from: null, to: null },
numAccelerators: filterPresets.numAccelerators || { from: null, to: null },
stats: [],
}
let isClusterOpen = false,
isJobStatesOpen = false,
isStartTimeOpen = false,
isTagsOpen = false,
isDurationOpen = false,
isResourcesOpen = false,
isStatsOpen = false
// Can be called from the outside to trigger a 'update' event from this component.
export function update(additionalFilters = null) {
if (additionalFilters != null)
for (let key in additionalFilters)
filters[key] = additionalFilters[key]
let items = []
if (filters.cluster)
items.push({ cluster: { eq: filters.cluster } })
if (filters.partition)
items.push({ partition: { eq: filters.partition } })
if (filters.states.length != allJobStates.length)
items.push({ state: filters.states })
if (filters.startTime.from || filters.startTime.to)
items.push({ startTime: { from: filters.startTime.from, to: filters.startTime.to } })
if (filters.tags.length != 0)
items.push({ tags: filters.tags })
if (filters.duration.from || filters.duration.to)
items.push({ duration: { from: filters.duration.from, to: filters.duration.to } })
if (filters.jobId)
items.push({ jobId: { eq: filters.jobId } })
if (filters.arrayJobId != null)
items.push({ arrayJobId: filters.arrayJobId })
if (filters.numNodes.from != null || filters.numNodes.to != null)
items.push({ numNodes: { from: filters.numNodes.from, to: filters.numNodes.to } })
if (filters.numHWThreads.from != null || filters.numHWThreads.to != null)
items.push({ numHWThreads: { from: filters.numHWThreads.from, to: filters.numHWThreads.to } })
if (filters.numAccelerators.from != null || filters.numAccelerators.to != null)
items.push({ numAccelerators: { from: filters.numAccelerators.from, to: filters.numAccelerators.to } })
if (filters.user)
items.push({ user: { [filters.userMatch]: filters.user } })
if (filters.project)
items.push({ project: { [filters.projectMatch]: filters.project } })
for (let stat of filters.stats)
items.push({ [stat.field]: { from: stat.from, to: stat.to } })
dispatch('update', { filters: items })
changeURL()
return items
}
function changeURL() {
const dateToUnixEpoch = (rfc3339) => Math.floor(Date.parse(rfc3339) / 1000)
let opts = []
if (filters.cluster)
opts.push(`cluster=${filters.cluster}`)
if (filters.partition)
opts.push(`partition=${filters.partition}`)
if (filters.states.length != allJobStates.length)
for (let state of filters.states)
opts.push(`state=${state}`)
if (filters.startTime.from && filters.startTime.to)
opts.push(`startTime=${dateToUnixEpoch(filters.startTime.from)}-${dateToUnixEpoch(filters.startTime.to)}`)
for (let tag of filters.tags)
opts.push(`tag=${tag}`)
if (filters.duration.from && filters.duration.to)
opts.push(`duration=${filters.duration.from}-${filters.duration.to}`)
if (filters.numNodes.from && filters.numNodes.to)
opts.push(`numNodes=${filters.numNodes.from}-${filters.numNodes.to}`)
if (filters.numAccelerators.from && filters.numAccelerators.to)
opts.push(`numAccelerators=${filters.numAccelerators.from}-${filters.numAccelerators.to}`)
if (filters.user)
opts.push(`user=${filters.user}`)
if (filters.userMatch != 'contains')
opts.push(`userMatch=${filters.userMatch}`)
if (filters.project)
opts.push(`project=${filters.project}`)
if (filters.projectMatch != 'contains')
opts.push(`projectMatch=${filters.projectMatch}`)
if (opts.length == 0 && window.location.search.length <= 1)
return
let newurl = `${window.location.pathname}?${opts.join('&')}`
window.history.replaceState(null, '', newurl)
}
</script>
<Row>
<Col xs="auto">
<ButtonDropdown class="cc-dropdown-on-hover">
<DropdownToggle outline caret color="success">
<Icon name="sliders"/>
Filters
</DropdownToggle>
<DropdownMenu>
<DropdownItem header>
Manage Filters
</DropdownItem>
{#if menuText}
<DropdownItem disabled>{menuText}</DropdownItem>
<DropdownItem divider />
{/if}
<DropdownItem on:click={() => (isClusterOpen = true)}>
<Icon name="cpu"/> Cluster/Partition
</DropdownItem>
<DropdownItem on:click={() => (isJobStatesOpen = true)}>
<Icon name="gear-fill"/> Job States
</DropdownItem>
<DropdownItem on:click={() => (isStartTimeOpen = true)}>
<Icon name="calendar-range"/> Start Time
</DropdownItem>
<DropdownItem on:click={() => (isDurationOpen = true)}>
<Icon name="stopwatch"/> Duration
</DropdownItem>
<DropdownItem on:click={() => (isTagsOpen = true)}>
<Icon name="tags"/> Tags
</DropdownItem>
<DropdownItem on:click={() => (isResourcesOpen = true)}>
<Icon name="hdd-stack"/> Nodes/Accelerators
</DropdownItem>
<DropdownItem on:click={() => (isStatsOpen = true)}>
<Icon name="bar-chart" on:click={() => (isStatsOpen = true)}/> Statistics
</DropdownItem>
{#if startTimeQuickSelect}
<DropdownItem divider/>
<DropdownItem disabled>Start Time Qick Selection</DropdownItem>
{#each [
{ text: 'Last 6hrs', seconds: 6*60*60 },
{ text: 'Last 12hrs', seconds: 12*60*60 },
{ text: 'Last 24hrs', seconds: 24*60*60 },
{ text: 'Last 48hrs', seconds: 48*60*60 },
{ text: 'Last 7 days', seconds: 7*24*60*60 },
{ text: 'Last 30 days', seconds: 30*24*60*60 }
] as {text, seconds}}
<DropdownItem on:click={() => {
filters.startTime.from = (new Date(Date.now() - seconds * 1000)).toISOString()
filters.startTime.to = (new Date(Date.now())).toISOString()
update()
}}>
<Icon name="calendar-range"/> {text}
</DropdownItem>
{/each}
{/if}
</DropdownMenu>
</ButtonDropdown>
</Col>
<!-- {#if startTimeQuickSelect}
<Col xs="auto">
<TimeSelection customEnabled={false} anyEnabled={true}
from={filters.startTime.from ? new Date(filters.startTime.from) : null}
to={filters.startTime.to ? new Date(filters.startTime.to) : null}
options={{
'Last 6hrs': 6*60*60,
'Last 12hrs': 12*60*60,
'Last 24hrs': 24*60*60,
'Last 48hrs': 48*60*60,
'Last 7 days': 7*24*60*60,
'Last 30 days': 30*24*60*60}}
on:change={({ detail: { from, to } }) => {
filters.startTime.from = from?.toISOString()
filters.startTime.to = to?.toISOString()
console.log(filters.startTime)
update()
}}
/>
</Col>
{/if} -->
<Col xs="auto">
{#if filters.cluster}
<Info icon="cpu" on:click={() => (isClusterOpen = true)}>
{filters.cluster}
{#if filters.partition}
({filters.partition})
{/if}
</Info>
{/if}
{#if filters.states.length != allJobStates.length}
<Info icon="gear-fill" on:click={() => (isJobStatesOpen = true)}>
{filters.states.join(', ')}
</Info>
{/if}
{#if filters.startTime.from || filters.startTime.to}
<Info icon="calendar-range" on:click={() => (isStartTimeOpen = true)}>
{new Date(filters.startTime.from).toLocaleString()} - {new Date(filters.startTime.to).toLocaleString()}
</Info>
{/if}
{#if filters.duration.from || filters.duration.to}
<Info icon="stopwatch" on:click={() => (isDurationOpen = true)}>
{Math.floor(filters.duration.from / 3600)}h:{Math.floor(filters.duration.from % 3600 / 60)}m
-
{Math.floor(filters.duration.to / 3600)}h:{Math.floor(filters.duration.to % 3600 / 60)}m
</Info>
{/if}
{#if filters.tags.length != 0}
<Info icon="tags" on:click={() => (isTagsOpen = true)}>
{#each filters.tags as tagId}
<Tag id={tagId} clickable={false} />
{/each}
</Info>
{/if}
{#if filters.numNodes.from != null || filters.numNodes.to != null}
<Info icon="hdd-stack" on:click={() => (isResourcesOpen = true)}>
Nodes: {filters.numNodes.from} - {filters.numNodes.to}
</Info>
{/if}
{#if filters.stats.length > 0}
<Info icon="bar-chart" on:click={() => (isStatsOpen = true)}>
{filters.stats.map(stat => `${stat.text}: ${stat.from} - ${stat.to}`).join(', ')}
</Info>
{/if}
</Col>
</Row>
<Cluster
disableClusterSelection={disableClusterSelection}
bind:isOpen={isClusterOpen}
bind:cluster={filters.cluster}
bind:partition={filters.partition}
on:update={() => update()} />
<JobStates
bind:isOpen={isJobStatesOpen}
bind:states={filters.states}
on:update={() => update()} />
<StartTime
bind:isOpen={isStartTimeOpen}
bind:from={filters.startTime.from}
bind:to={filters.startTime.to}
on:update={() => update()} />
<Duration
bind:isOpen={isDurationOpen}
bind:from={filters.duration.from}
bind:to={filters.duration.to}
on:update={() => update()} />
<Tags
bind:isOpen={isTagsOpen}
bind:tags={filters.tags}
on:update={() => update()} />
<Resources cluster={filters.cluster}
bind:isOpen={isResourcesOpen}
bind:numNodes={filters.numNodes}
bind:numHWThreads={filters.numHWThreads}
bind:numAccelerators={filters.numAccelerators}
on:update={() => update()} />
<Statistics cluster={filters.cluster}
bind:isOpen={isStatsOpen}
bind:stats={filters.stats}
on:update={() => update()} />
<style>
:global(.cc-dropdown-on-hover:hover .dropdown-menu) {
display: block;
margin-top: 0px;
padding-top: 0px;
transform: none !important;
}
</style>

11
web/frontend/src/filters/InfoBox.svelte Normal file
View File

@ -0,0 +1,11 @@
<script>
import { Button, Icon } from 'sveltestrap'
export let icon
export let modified = false
</script>
<Button outline color={modified ? 'warning' : 'primary'} on:click>
<Icon name={icon}/>
<slot />
</Button>

47
web/frontend/src/filters/JobStates.svelte Normal file
View File

@ -0,0 +1,47 @@
<script context="module">
export const allJobStates = [ 'running', 'completed', 'failed', 'cancelled', 'stopped', 'timeout', 'preempted', 'out_of_memory' ]
</script>
<script>
import { createEventDispatcher } from 'svelte'
import { Button, ListGroup, ListGroupItem,
Modal, ModalBody, ModalHeader, ModalFooter } from 'sveltestrap'
const dispatch = createEventDispatcher()
export let isModified = false
export let isOpen = false
export let states = [...allJobStates]
let pendingStates = [...states]
$: isModified = states.length != pendingStates.length || !states.every(state => pendingStates.includes(state))
</script>
<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
<ModalHeader>
Select Job States
</ModalHeader>
<ModalBody>
<ListGroup>
{#each allJobStates as state}
<ListGroupItem>
<input type=checkbox bind:group={pendingStates} name="flavours" value={state}>
{state}
</ListGroupItem>
{/each}
</ListGroup>
</ModalBody>
<ModalFooter>
<Button color="primary" disabled={pendingStates.length == 0} on:click={() => {
isOpen = false
states = [...pendingStates]
dispatch('update', { states })
}}>Close & Apply</Button>
<Button color="danger" on:click={() => {
isOpen = false
states = [...allJobStates]
pendingStates = [...allJobStates]
dispatch('update', { states })
}}>Reset</Button>
<Button on:click={() => (isOpen = false)}>Close</Button>
</ModalFooter>
</Modal>

99
web/frontend/src/filters/Resources.svelte Normal file
View File

@ -0,0 +1,99 @@
<script>
import { createEventDispatcher, getContext } from 'svelte'
import { Button, Modal, ModalBody, ModalHeader, ModalFooter } from 'sveltestrap'
import DoubleRangeSlider from './DoubleRangeSlider.svelte'
const clusters = getContext('clusters'),
initialized = getContext('initialized'),
dispatch = createEventDispatcher()
export let cluster = null
export let isModified = false
export let isOpen = false
export let numNodes = { from: null, to: null }
export let numHWThreads = { from: null, to: null }
export let numAccelerators = { from: null, to: null }
let pendingNumNodes = numNodes, pendingNumHWThreads = numHWThreads, pendingNumAccelerators = numAccelerators
$: isModified = pendingNumNodes.from != numNodes.from || pendingNumNodes.to != numNodes.to
|| pendingNumHWThreads.from != numHWThreads.from || pendingNumHWThreads.to != numHWThreads.to
|| pendingNumAccelerators.from != numAccelerators.from || pendingNumAccelerators.to != numAccelerators.to
const findMaxNumAccels = clusters => clusters.reduce((max, cluster) => Math.max(max,
cluster.subClusters.reduce((max, sc) => Math.max(max, sc.topology.accelerators?.length || 0), 0)), 0)
let minNumNodes = 1, maxNumNodes = 0, minNumHWThreads = 1, maxNumHWThreads = 0, minNumAccelerators = 0, maxNumAccelerators = 0
$: {
if ($initialized) {
if (cluster != null) {
const { filterRanges, subClusters } = clusters.find(c => c.name == cluster)
minNumNodes = filterRanges.numNodes.from
maxNumNodes = filterRanges.numNodes.to
maxNumAccelerators = findMaxNumAccels([{ subClusters }])
} else if (clusters.length > 0) {
const { filterRanges } = clusters[0]
minNumNodes = filterRanges.numNodes.from
maxNumNodes = filterRanges.numNodes.to
maxNumAccelerators = findMaxNumAccels(clusters)
for (let cluster of clusters) {
const { filterRanges } = cluster
minNumNodes = Math.min(minNumNodes, filterRanges.numNodes.from)
maxNumNodes = Math.max(maxNumNodes, filterRanges.numNodes.to)
}
}
}
}
$: {
if (isOpen && $initialized && pendingNumNodes.from == null && pendingNumNodes.to == null) {
pendingNumNodes = { from: 0, to: maxNumNodes }
}
}
</script>
<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
<ModalHeader>
Select Number of Nodes, HWThreads and Accelerators
</ModalHeader>
<ModalBody>
<h4>Number of Nodes</h4>
<DoubleRangeSlider
on:change={({ detail }) => (pendingNumNodes = { from: detail[0], to: detail[1] })}
min={minNumNodes} max={maxNumNodes}
firstSlider={pendingNumNodes.from} secondSlider={pendingNumNodes.to} />
<!-- <DoubleRangeSlider
on:change={({ detail }) => (pendingNumHWThreads = { from: detail[0], to: detail[1] })}
min={minNumHWThreads} max={maxNumHWThreads}
firstSlider={pendingNumHWThreads.from} secondSlider={pendingNumHWThreads.to} /> -->
{#if maxNumAccelerators != null && maxNumAccelerators > 1}
<DoubleRangeSlider
on:change={({ detail }) => (pendingNumAccelerators = { from: detail[0], to: detail[1] })}
min={minNumAccelerators} max={maxNumAccelerators}
firstSlider={pendingNumAccelerators.from} secondSlider={pendingNumAccelerators.to} />
{/if}
</ModalBody>
<ModalFooter>
<Button color="primary"
disabled={pendingNumNodes.from == null || pendingNumNodes.to == null}
on:click={() => {
isOpen = false
numNodes = { from: pendingNumNodes.from, to: pendingNumNodes.to }
numHWThreads = { from: pendingNumHWThreads.from, to: pendingNumHWThreads.to }
numAccelerators = { from: pendingNumAccelerators.from, to: pendingNumAccelerators.to }
dispatch('update', { numNodes, numHWThreads, numAccelerators })
}}>
Close & Apply
</Button>
<Button color="danger" on:click={() => {
isOpen = false
pendingNumNodes = { from: null, to: null }
pendingNumHWThreads = { from: null, to: null }
pendingNumAccelerators = { from: null, to: null }
numNodes = { from: pendingNumNodes.from, to: pendingNumNodes.to }
numHWThreads = { from: pendingNumHWThreads.from, to: pendingNumHWThreads.to }
numAccelerators = { from: pendingNumAccelerators.from, to: pendingNumAccelerators.to }
dispatch('update', { numNodes, numHWThreads, numAccelerators })
}}>Reset</Button>
<Button on:click={() => (isOpen = false)}>Close</Button>
</ModalFooter>
</Modal>

90
web/frontend/src/filters/StartTime.svelte Normal file
View File

@ -0,0 +1,90 @@
<script>
import { createEventDispatcher, getContext } from 'svelte'
import { Row, Button, Input, Modal, ModalBody, ModalHeader, ModalFooter, FormGroup } from 'sveltestrap'
const dispatch = createEventDispatcher()
export let isModified = false
export let isOpen = false
export let from = null
export let to = null
let pendingFrom, pendingTo
function reset() {
pendingFrom = from == null ? { date: '0000-00-00', time: '00:00' } : fromRFC3339(from)
pendingTo = to == null ? { date: '0000-00-00', time: '00:00' } : fromRFC3339(to)
}
reset()
function toRFC3339({ date, time }, secs = 0) {
const dparts = date.split('-')
const tparts = time.split(':')
const d = new Date(
Number.parseInt(dparts[0]),
Number.parseInt(dparts[1]) - 1,
Number.parseInt(dparts[2]),
Number.parseInt(tparts[0]),
Number.parseInt(tparts[1]), secs)
return d.toISOString()
}
function fromRFC3339(rfc3339) {
const d = new Date(rfc3339)
const pad = (n) => n.toString().padStart(2, '0')
const date = `${d.getFullYear()}-${pad(d.getMonth() + 1)}-${pad(d.getDate())}`
const time = `${pad(d.getHours())}:${pad(d.getMinutes())}`
return { date, time }
}
$: isModified = (from != toRFC3339(pendingFrom) || to != toRFC3339(pendingTo, 59))
&& !(from == null && pendingFrom.date == '0000-00-00' && pendingFrom.time == '00:00')
&& !(to == null && pendingTo.date == '0000-00-00' && pendingTo.time == '00:00')
</script>
<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
<ModalHeader>
Select Start Time
</ModalHeader>
<ModalBody>
<h4>From</h4>
<Row>
<FormGroup class="col">
<Input type="date" bind:value={pendingFrom.date}/>
</FormGroup>
<FormGroup class="col">
<Input type="time" bind:value={pendingFrom.time}/>
</FormGroup>
</Row>
<h4>To</h4>
<Row>
<FormGroup class="col">
<Input type="date" bind:value={pendingTo.date}/>
</FormGroup>
<FormGroup class="col">
<Input type="time" bind:value={pendingTo.time}/>
</FormGroup>
</Row>
</ModalBody>
<ModalFooter>
<Button color="primary"
disabled={pendingFrom.date == '0000-00-00' || pendingTo.date == '0000-00-00'}
on:click={() => {
isOpen = false
from = toRFC3339(pendingFrom)
to = toRFC3339(pendingTo, 59)
dispatch('update', { from, to })
}}>
Close & Apply
</Button>
<Button color="danger" on:click={() => {
isOpen = false
from = null
to = null
reset()
dispatch('update', { from, to })
}}>Reset</Button>
<Button on:click={() => (isOpen = false)}>Close</Button>
</ModalFooter>
</Modal>

113
web/frontend/src/filters/Stats.svelte Normal file
View File

@ -0,0 +1,113 @@
<script>
import { createEventDispatcher, getContext } from 'svelte'
import { Button, Modal, ModalBody, ModalHeader, ModalFooter } from 'sveltestrap'
import DoubleRangeSlider from './DoubleRangeSlider.svelte'
const clusters = getContext('clusters'),
initialized = getContext('initialized'),
dispatch = createEventDispatcher()
export let cluster = null
export let isModified = false
export let isOpen = false
export let stats = []
let statistics = [
{
field: 'flopsAnyAvg',
text: 'FLOPs (Avg.)',
metric: 'flops_any',
from: 0, to: 0, peak: 0,
enabled: false
},
{
field: 'memBwAvg',
text: 'Mem. Bw. (Avg.)',
metric: 'mem_bw',
from: 0, to: 0, peak: 0,
enabled: false
},
{
field: 'loadAvg',
text: 'Load (Avg.)',
metric: 'cpu_load',
from: 0, to: 0, peak: 0,
enabled: false
},
{
field: 'memUsedMax',
text: 'Mem. used (Max.)',
metric: 'mem_used',
from: 0, to: 0, peak: 0,
enabled: false
}
]
$: isModified = !statistics.every(a => {
let b = stats.find(s => s.field == a.field)
if (b == null)
return !a.enabled
return a.from == b.from && a.to == b.to
})
function getPeak(cluster, metric) {
const mc = cluster.metricConfig.find(mc => mc.name == metric)
return mc ? mc.peak : 0
}
function resetRange(isInitialized, cluster) {
if (!isInitialized)
return
if (cluster != null) {
let c = clusters.find(c => c.name == cluster)
for (let stat of statistics) {
stat.peak = getPeak(c, stat.metric)
stat.from = 0
stat.to = stat.peak
}
} else {
for (let stat of statistics) {
for (let c of clusters) {
stat.peak = Math.max(stat.peak, getPeak(c, stat.metric))
}
stat.from = 0
stat.to = stat.peak
}
}
statistics = [...statistics]
}
$: resetRange($initialized, cluster)
</script>
<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
<ModalHeader>
Filter based on statistics (of non-running jobs)
</ModalHeader>
<ModalBody>
{#each statistics as stat}
<h4>{stat.text}</h4>
<DoubleRangeSlider
on:change={({ detail }) => (stat.from = detail[0], stat.to = detail[1], stat.enabled = true)}
min={0} max={stat.peak}
firstSlider={stat.from} secondSlider={stat.to} />
{/each}
</ModalBody>
<ModalFooter>
<Button color="primary" on:click={() => {
isOpen = false
stats = statistics.filter(stat => stat.enabled)
dispatch('update', { stats })
}}>Close & Apply</Button>
<Button color="danger" on:click={() => {
isOpen = false
statistics.forEach(stat => stat.enabled = false)
stats = []
dispatch('update', { stats })
}}>Reset</Button>
<Button on:click={() => (isOpen = false)}>Close</Button>
</ModalFooter>
</Modal>

67
web/frontend/src/filters/Tags.svelte Normal file
View File

@ -0,0 +1,67 @@
<script>
import { createEventDispatcher, getContext } from 'svelte'
import { Button, ListGroup, ListGroupItem, Input,
Modal, ModalBody, ModalHeader, ModalFooter, Icon } from 'sveltestrap'
import { fuzzySearchTags } from '../utils.js'
import Tag from '../Tag.svelte'
const allTags = getContext('tags'),
initialized = getContext('initialized'),
dispatch = createEventDispatcher()
export let isModified = false
export let isOpen = false
export let tags = []
let pendingTags = [...tags]
$: isModified = tags.length != pendingTags.length || !tags.every(tagId => pendingTags.includes(tagId))
let searchTerm = ''
</script>
<Modal isOpen={isOpen} toggle={() => (isOpen = !isOpen)}>
<ModalHeader>
Select Tags
</ModalHeader>
<ModalBody>
<Input type="text" placeholder="Search" bind:value={searchTerm} />
<br/>
<ListGroup>
{#if $initialized}
{#each fuzzySearchTags(searchTerm, allTags) as tag (tag)}
<ListGroupItem>
{#if pendingTags.includes(tag.id)}
<Button outline color="danger"
on:click={() => (pendingTags = pendingTags.filter(id => id != tag.id))}>
<Icon name="dash-circle" />
</Button>
{:else}
<Button outline color="success"
on:click={() => (pendingTags = [...pendingTags, tag.id])}>
<Icon name="plus-circle" />
</Button>
{/if}
<Tag tag={tag} />
</ListGroupItem>
{:else}
<ListGroupItem disabled>No Tags</ListGroupItem>
{/each}
{/if}
</ListGroup>
</ModalBody>
<ModalFooter>
<Button color="primary" on:click={() => {
isOpen = false
tags = [...pendingTags]
dispatch('update', { tags })
}}>Close & Apply</Button>
<Button color="danger" on:click={() => {
isOpen = false
tags = []
pendingTags = []
dispatch('update', { tags })
}}>Reset</Button>
<Button on:click={() => (isOpen = false)}>Close</Button>
</ModalFooter>
</Modal>

80
web/frontend/src/filters/TimeSelection.svelte Normal file
View File

@ -0,0 +1,80 @@
<script>
import { Icon, Input, InputGroup, InputGroupText } from 'sveltestrap'
import { createEventDispatcher } from "svelte"
export let from
export let to
export let customEnabled = true
export let anyEnabled = false
export let options = {
'Last half hour': 30*60,
'Last hour': 60*60,
'Last 2hrs': 2*60*60,
'Last 4hrs': 4*60*60,
'Last 12hrs': 12*60*60,
'Last 24hrs': 24*60*60
}
$: pendingFrom = from
$: pendingTo = to
const dispatch = createEventDispatcher()
let timeRange = to && from
? (to.getTime() - from.getTime()) / 1000
: (anyEnabled ? -2 : -1)
function updateTimeRange(event) {
if (timeRange == -1) {
pendingFrom = null
pendingTo = null
return
}
if (timeRange == -2) {
from = pendingFrom = null
to = pendingTo = null
dispatch('change', { from, to })
return
}
let now = Date.now(), t = timeRange * 1000
from = pendingFrom = new Date(now - t)
to = pendingTo = new Date(now)
dispatch('change', { from, to })
}
function updateExplicitTimeRange(type, event) {
let d = new Date(Date.parse(event.target.value));
if (type == 'from') pendingFrom = d
else pendingTo = d
if (pendingFrom != null && pendingTo != null) {
from = pendingFrom
to = pendingTo
dispatch('change', { from, to })
}
}
</script>
<InputGroup class="inline-from">
<InputGroupText><Icon name="clock-history"/></InputGroupText>
<!-- <InputGroupText>
Time
</InputGroupText> -->
<select class="form-select" bind:value={timeRange} on:change={updateTimeRange}>
{#if customEnabled}
<option value={-1}>Custom</option>
{/if}
{#if anyEnabled}
<option value={-2}>Any</option>
{/if}
{#each Object.entries(options) as [name, seconds]}
<option value={seconds}>{name}</option>
{/each}
</select>
{#if timeRange == -1}
<InputGroupText>from</InputGroupText>
<Input type="datetime-local" on:change={(event) => updateExplicitTimeRange('from', event)}></Input>
<InputGroupText>to</InputGroupText>
<Input type="datetime-local" on:change={(event) => updateExplicitTimeRange('to', event)}></Input>
{/if}
</InputGroup>

51
web/frontend/src/filters/UserOrProject.svelte Normal file
View File

@ -0,0 +1,51 @@
<script>
import { InputGroup, Input } from 'sveltestrap'
import { createEventDispatcher } from 'svelte'
const dispatch = createEventDispatcher()
export let user = ''
export let project = ''
let mode = 'user', term = ''
const throttle = 500
function modeChanged() {
if (mode == 'user') {
project = term
term = user
} else {
user = term
term = project
}
termChanged(0)
}
let timeoutId = null
function termChanged(sleep = throttle) {
if (mode == 'user')
user = term
else
project = term
if (timeoutId != null)
clearTimeout(timeoutId)
timeoutId = setTimeout(() => {
dispatch('update', {
user,
project
})
}, sleep)
}
</script>
<InputGroup>
<select style="max-width: 175px;" class="form-select"
bind:value={mode} on:change={modeChanged}>
<option value={'user'}>Search User</option>
<option value={'project'}>Search Project</option>
</select>
<Input
type="text" bind:value={term} on:change={() => termChanged()} on:keyup={(event) => termChanged(event.key == 'Enter' ? 0 : throttle)}
placeholder={mode == 'user' ? 'filter username...' : 'filter project...'} />
</InputGroup>

10
web/frontend/src/header.entrypoint.js Normal file
View File

@ -0,0 +1,10 @@
import Header from './Header.svelte'
const headerDomTarget = document.getElementById('svelte-header')
if (headerDomTarget != null) {
new Header({
target: headerDomTarget,
props: { ...header },
})
}

12
web/frontend/src/job.entrypoint.js Normal file
View File

@ -0,0 +1,12 @@
import {} from './header.entrypoint.js'
import Job from './Job.root.svelte'
new Job({
target: document.getElementById('svelte-app'),
props: {
dbid: jobInfos.id
},
context: new Map([
['cc-config', clusterCockpitConfig]
])
})

88
web/frontend/src/joblist/JobInfo.svelte Normal file
View File

@ -0,0 +1,88 @@
<!--
@component
Properties:
- job: GraphQL.Job
- jobTags: Defaults to job.tags, usefull for dynamically updating the tags.
-->
<script context="module">
export const scrambleNames = window.localStorage.getItem("cc-scramble-names")
export const scramble = (str) => [...str].reduce((x, c, i) => x * 7 + c.charCodeAt(0) * i * 21, 5).toString(32)
</script>
<script>
import Tag from '../Tag.svelte';
import { Badge, Icon } from 'sveltestrap';
export let job;
export let jobTags = job.tags;
function formatDuration(duration) {
const hours = Math.floor(duration / 3600);
duration -= hours * 3600;
const minutes = Math.floor(duration / 60);
duration -= minutes * 60;
const seconds = duration;
return `${hours}:${('0' + minutes).slice(-2)}:${('0' + seconds).slice(-2)}`;
}
</script>
<div>
<p>
<span class="fw-bold"><a href="/monitoring/job/{job.id}" target="_blank">{job.jobId}</a> ({job.cluster})</span>
{#if job.metaData?.jobName}
<br/>
{job.metaData.jobName}
{/if}
{#if job.arrayJobId}
Array Job: <a href="/monitoring/jobs/?arrayJobId={job.arrayJobId}&cluster={job.cluster}" target="_blank">#{job.arrayJobId}</a>
{/if}
</p>
<p>
<Icon name="person-fill"/>
<a class="fst-italic" href="/monitoring/user/{job.user}" target="_blank">
{scrambleNames ? scramble(job.user) : job.user}
</a>
{#if job.userData && job.userData.name}
({scrambleNames ? scramble(job.userData.name) : job.userData.name})
{/if}
{#if job.project && job.project != 'no project'}
<br/>
<Icon name="people-fill"/> {job.project}
{/if}
</p>
<p>
{job.numNodes} <Icon name="pc-horizontal"/>
{#if job.exclusive != 1}
(shared)
{/if}
{#if job.numAcc > 0}
, {job.numAcc} <Icon name="gpu-card"/>
{/if}
{#if job.numHWThreads > 0}
, {job.numHWThreads} <Icon name="cpu"/>
{/if}
</p>
<p>
Start: <span class="fw-bold">{(new Date(job.startTime)).toLocaleString()}</span>
<br/>
Duration: <span class="fw-bold">{formatDuration(job.duration)}</span>
{#if job.state == 'running'}
<Badge color="success">running</Badge>
{:else if job.state != 'completed'}
<Badge color="danger">{job.state}</Badge>
{/if}
{#if job.walltime}
<br/>
Walltime: <span class="fw-bold">{formatDuration(job.walltime)}</span>
{/if}
</p>
<p>
{#each jobTags as tag}
<Tag tag={tag}/>
{/each}
</p>
</div>

190
web/frontend/src/joblist/JobList.svelte Normal file
View File

@ -0,0 +1,190 @@
<!--
@component
Properties:
- metrics: [String] (can change from outside)
- sorting: { field: String, order: "DESC" | "ASC" } (can change from outside)
- matchedJobs: Number (changes from inside)
Functions:
- update(filters?: [JobFilter])
-->
<script>
import { operationStore, query, mutation } from '@urql/svelte'
import { getContext } from 'svelte';
import { Row, Table, Card, Spinner } from 'sveltestrap'
import Pagination from './Pagination.svelte'
import JobListRow from './Row.svelte'
import { stickyHeader } from '../utils.js'
const ccconfig = getContext('cc-config'),
clusters = getContext('clusters'),
initialized = getContext('initialized')
export let sorting = { field: "startTime", order: "DESC" }
export let matchedJobs = 0
export let metrics = ccconfig.plot_list_selectedMetrics
let itemsPerPage = ccconfig.plot_list_jobsPerPage
let page = 1
let paging = { itemsPerPage, page }
const jobs = operationStore(`
query($filter: [JobFilter!]!, $sorting: OrderByInput!, $paging: PageRequest! ){
jobs(filter: $filter, order: $sorting, page: $paging) {
items {
id, jobId, user, project, cluster, subCluster, startTime,
duration, numNodes, numHWThreads, numAcc, walltime,
SMT, exclusive, partition, arrayJobId,
monitoringStatus, state,
tags { id, type, name }
userData { name }
metaData
}
count
}
}`, {
paging,
sorting,
filter: []
}, {
pause: true
})
const updateConfiguration = mutation({
query: `mutation($name: String!, $value: String!) {
updateConfiguration(name: $name, value: $value)
}`
})
$: $jobs.variables = { ...$jobs.variables, sorting, paging }
$: matchedJobs = $jobs.data != null ? $jobs.data.jobs.count : 0
// (Re-)query and optionally set new filters.
export function update(filters) {
if (filters != null) {
let minRunningFor = ccconfig.plot_list_hideShortRunningJobs
if (minRunningFor && minRunningFor > 0) {
filters.push({ minRunningFor })
}
$jobs.variables.filter = filters
console.log('filters:', ...filters.map(f => Object.entries(f)).flat(2))
}
page = 1
$jobs.variables.paging = paging = { page, itemsPerPage };
$jobs.context.pause = false
$jobs.reexecute({ requestPolicy: 'network-only' })
}
query(jobs)
let tableWidth = null
let jobInfoColumnWidth = 250
$: plotWidth = Math.floor((tableWidth - jobInfoColumnWidth) / metrics.length - 10)
let headerPaddingTop = 0
stickyHeader('.cc-table-wrapper > table.table >thead > tr > th.position-sticky:nth-child(1)', (x) => (headerPaddingTop = x))
</script>
<Row>
<div class="col cc-table-wrapper" bind:clientWidth={tableWidth}>
<Table cellspacing="0px" cellpadding="0px">
<thead>
<tr>
<th class="position-sticky top-0" scope="col" style="width: {jobInfoColumnWidth}px; padding-top: {headerPaddingTop}px">
Job Info
</th>
{#each metrics as metric (metric)}
<th class="position-sticky top-0 text-center" scope="col" style="width: {plotWidth}px; padding-top: {headerPaddingTop}px">
{metric}
{#if $initialized}
({clusters
.map(cluster => cluster.metricConfig.find(m => m.name == metric))
.filter(m => m != null).map(m => m.unit)
.reduce((arr, unit) => arr.includes(unit) ? arr : [...arr, unit], [])
.join(', ')})
{/if}
</th>
{/each}
</tr>
</thead>
<tbody>
{#if $jobs.error}
<tr>
<td colspan="{metrics.length + 1}">
<Card body color="danger" class="mb-3"><h2>{$jobs.error.message}</h2></Card>
</td>
</tr>
{:else if $jobs.fetching || !$jobs.data}
<tr>
<td colspan="{metrics.length + 1}">
<Spinner secondary />
</td>
</tr>
{:else if $jobs.data && $initialized}
{#each $jobs.data.jobs.items as job (job)}
<JobListRow
job={job}
metrics={metrics}
plotWidth={plotWidth} />
{:else}
<tr>
<td colspan="{metrics.length + 1}">
No jobs found
</td>
</tr>
{/each}
{/if}
</tbody>
</Table>
</div>
</Row>
<Pagination
bind:page={page}
{itemsPerPage}
itemText="Jobs"
totalItems={matchedJobs}
on:update={({ detail }) => {
if (detail.itemsPerPage != itemsPerPage) {
itemsPerPage = detail.itemsPerPage
updateConfiguration({
name: "plot_list_jobsPerPage",
value: itemsPerPage.toString()
}).then(res => {
if (res.error)
console.error(res.error);
})
}
paging = { itemsPerPage: detail.itemsPerPage, page: detail.page }
}} />
<style>
.cc-table-wrapper {
overflow: initial;
}
.cc-table-wrapper > :global(table) {
border-collapse: separate;
border-spacing: 0px;
table-layout: fixed;
}
.cc-table-wrapper :global(button) {
margin-bottom: 0px;
}
.cc-table-wrapper > :global(table > tbody > tr > td) {
margin: 0px;
padding-left: 5px;
padding-right: 0px;
}
th.position-sticky.top-0 {
background-color: white;
z-index: 10;
border-bottom: 1px solid black;
}
</style>

230
web/frontend/src/joblist/Pagination.svelte Normal file
View File

@ -0,0 +1,230 @@
<!--
@component
Properties:
- page: Number (changes from inside)
- itemsPerPage: Number (changes from inside)
- totalItems: Number (only displayed)
Events:
- "update": { page: Number, itemsPerPage: Number }
- Dispatched once immediately and then each time page or itemsPerPage changes
-->
<div class="cc-pagination" >
<div class="cc-pagination-left">
<label for="cc-pagination-select">{ itemText } per page:</label>
<div class="cc-pagination-select-wrapper">
<select on:blur|preventDefault={reset} bind:value={itemsPerPage} id="cc-pagination-select" class="cc-pagination-select">
{#each pageSizes as size}
<option value="{size}">{size}</option>
{/each}
</select>
<span class="focus"></span>
</div>
<span class="cc-pagination-text">
{ (page - 1) * itemsPerPage } - { Math.min((page - 1) * itemsPerPage + itemsPerPage, totalItems) } of { totalItems } { itemText }
</span>
</div>
<div class="cc-pagination-right">
{#if !backButtonDisabled}
<button class="reset nav" type="button"
on:click|preventDefault="{reset}"></button>
<button class="left nav" type="button"
on:click|preventDefault="{() => { page -= 1; }}"></button>
{/if}
{#if !nextButtonDisabled}
<button class="right nav" type="button"
on:click|preventDefault="{() => { page += 1; }}"></button>
{/if}
</div>
</div>
<script>
import { createEventDispatcher } from "svelte";
export let page = 1;
export let itemsPerPage = 10;
export let totalItems = 0;
export let itemText = "items";
export let pageSizes = [10,25,50];
let backButtonDisabled, nextButtonDisabled;
const dispatch = createEventDispatcher();
$: {
if (typeof page !== "number") {
page = Number(page);
}
if (typeof itemsPerPage !== "number") {
itemsPerPage = Number(itemsPerPage);
}
dispatch("update", { itemsPerPage, page });
}
$: backButtonDisabled = (page === 1);
$: nextButtonDisabled = (page >= (totalItems / itemsPerPage));
function reset ( event ) {
page = 1;
}
</script>
<style>
*, *::before, *::after {
box-sizing: border-box;
}
div {
display: flex;
align-items: center;
vertical-align: baseline;
box-sizing: border-box;
}
label, select, button {
margin: 0;
padding: 0;
vertical-align: baseline;
color: #525252;
}
button {
position: relative;
border: none;
border-left: 1px solid #e0e0e0;
height: 3rem;
width: 3rem;
background: 0 0;
transition: all 70ms;
}
button:hover {
background-color: #dde1e6;
}
button:focus {
top: -1px;
left: -1px;
right: -1px;
bottom: -1px;
border: 1px solid blue;
border-radius: inherit;
}
.nav::after {
content: "";
width: 0.9em;
height: 0.8em;
background-color: #777;
z-index: 1;
position: absolute;
top: 50%;
left: 50%;
}
.nav:disabled {
background-color: #fff;
cursor: no-drop;
}
.reset::after {
clip-path: polygon(100% 0%, 75% 50%, 100% 100%, 25% 100%, 0% 50%, 25% 0%);
margin-top: -0.3em;
margin-left: -0.5em;
}
.right::after {
clip-path: polygon(100% 50%, 50% 0, 50% 100%);
margin-top: -0.3em;
margin-left: -0.5em;
}
.left::after {
clip-path: polygon(50% 0, 0 50%, 50% 100%);
margin-top: -0.3em;
margin-left: -0.3em;
}
.cc-pagination-select-wrapper::after {
content: "";
width: 0.8em;
height: 0.5em;
background-color: #777;
clip-path: polygon(100% 0%, 0 0%, 50% 100%);
justify-self: end;
}
.cc-pagination {
width: 100%;
justify-content: space-between;
border-top: 1px solid #e0e0e0;
}
.cc-pagination-text {
color: #525252;
margin-left: 1rem;
}
.cc-pagination-text {
color: #525252;
margin-right: 1rem;
}
.cc-pagination-left {
padding: 0 1rem;
height: 3rem;
}
.cc-pagination-select-wrapper {
display: grid;
grid-template-areas: "select";
align-items: center;
position: relative;
padding: 0 0.5em;
min-width: 3em;
max-width: 6em;
border-right: 1px solid #e0e0e0;
cursor: pointer;
transition: all 70ms;
}
.cc-pagination-select-wrapper:hover {
background-color: #dde1e6;
}
select,
.cc-pagination-select-wrapper::after {
grid-area: select;
}
.cc-pagination-select {
height: 3rem;
appearance: none;
background-color: transparent;
padding: 0 1em 0 0;
margin: 0;
border: none;
width: 100%;
font-family: inherit;
font-size: inherit;
cursor: inherit;
line-height: inherit;
z-index: 1;
outline: none;
}
select:focus + .focus {
position: absolute;
top: -1px;
left: -1px;
right: -1px;
bottom: -1px;
border: 1px solid blue;
border-radius: inherit;
}
.cc-pagination-right {
height: 3rem;
}
</style>

43
web/frontend/src/joblist/Refresher.svelte Normal file
View File

@ -0,0 +1,43 @@
<!--
@component
Events:
- 'reload': When fired, the parent component shoud refresh its contents
-->
<script>
import { createEventDispatcher } from 'svelte'
import { Button, Icon, InputGroup } from 'sveltestrap'
const dispatch = createEventDispatcher()
let refreshInterval = null;
let refreshIntervalId = null;
function refreshIntervalChanged() {
if (refreshIntervalId != null)
clearInterval(refreshIntervalId);
if (refreshInterval == null)
return;
refreshIntervalId = setInterval(() => dispatch("reload"), refreshInterval);
}
export let initially = null
if (initially != null) {
refreshInterval = initially * 1000
refreshIntervalChanged()
}
</script>
<InputGroup>
<Button outline on:click={() => dispatch("reload")} disabled={refreshInterval != null}>
<Icon name="arrow-clockwise" /> Reload
</Button>
<select class="form-select" bind:value={refreshInterval} on:change={refreshIntervalChanged}>
<option value={null}>No periodic reload</option>
<option value={ 30 * 1000}>Update every 30 seconds</option>
<option value={ 60 * 1000}>Update every minute</option>
<option value={2 * 60 * 1000}>Update every two minutes</option>
<option value={5 * 60 * 1000}>Update every 5 minutes</option>
</select>
</InputGroup>

101
web/frontend/src/joblist/Row.svelte Normal file
View File

@ -0,0 +1,101 @@
<!--
@component
Properties:
- job: GraphQL.Job (constant/key)
- metrics: [String] (can change)
- plotWidth: Number
- plotHeight: Number
-->
<script>
import { operationStore, query } from '@urql/svelte'
import { getContext } from 'svelte'
import { Card, Spinner } from 'sveltestrap'
import MetricPlot from '../plots/MetricPlot.svelte'
import JobInfo from './JobInfo.svelte'
import { maxScope } from '../utils.js'
export let job
export let metrics
export let plotWidth
export let plotHeight = 275
let scopes = [job.numNodes == 1 ? 'core' : 'node']
const cluster = getContext('clusters').find(c => c.name == job.cluster)
const metricsQuery = operationStore(`query($id: ID!, $metrics: [String!]!, $scopes: [MetricScope!]!) {
jobMetrics(id: $id, metrics: $metrics, scopes: $scopes) {
name
metric {
unit, scope, timestep
statisticsSeries { min, mean, max }
series {
hostname, id, data
statistics { min, avg, max }
}
}
}
}`, {
id: job.id,
metrics,
scopes
})
const selectScope = (jobMetrics) => jobMetrics.reduce(
(a, b) => maxScope([a.metric.scope, b.metric.scope]) == a.metric.scope
? (job.numNodes > 1 ? a : b)
: (job.numNodes > 1 ? b : a), jobMetrics[0])
const sortAndSelectScope = (jobMetrics) => metrics
.map(name => jobMetrics.filter(jobMetric => jobMetric.name == name))
.map(jobMetrics => jobMetrics.length > 0 ? selectScope(jobMetrics) : null)
$: metricsQuery.variables = { id: job.id, metrics, scopes }
if (job.monitoringStatus)
query(metricsQuery)
</script>
<tr>
<td>
<JobInfo job={job}/>
</td>
{#if job.monitoringStatus == 0 || job.monitoringStatus == 2}
<td colspan="{metrics.length}">
<Card body color="warning">Not monitored or archiving failed</Card>
</td>
{:else if $metricsQuery.fetching}
<td colspan="{metrics.length}" style="text-align: center;">
<Spinner secondary />
</td>
{:else if $metricsQuery.error}
<td colspan="{metrics.length}">
<Card body color="danger" class="mb-3">
{$metricsQuery.error.message.length > 500
? $metricsQuery.error.message.substring(0, 499)+'...'
: $metricsQuery.error.message}
</Card>
</td>
{:else}
{#each sortAndSelectScope($metricsQuery.data.jobMetrics) as metric, i (metric || i)}
<td>
{#if metric != null}
<MetricPlot
width={plotWidth}
height={plotHeight}
timestep={metric.metric.timestep}
scope={metric.metric.scope}
series={metric.metric.series}
statisticsSeries={metric.metric.statisticsSeries}
metric={metric.name}
cluster={cluster}
subCluster={job.subCluster} />
{:else}
<Card body color="warning">Missing Data</Card>
{/if}
</td>
{/each}
{/if}
</tr>

Some files were not shown because too many files have changed in this diff Show More