Implement the following plan:

# NATS API Throughput Improvements: Configurable Semaphores, Queue Groups, Channel-Based Subscriptions

## Context

The NATS API has hardcoded semaphore sizes (8 job, 2 node), uses basic `Subscribe()` instead of queue groups, and directly processes messages in NATS callbacks. These limit throughput and scalability.

## Changes

### 1. Configurable semaphore sizes — `internal/config/config.go`

Add `JobConcurrency` and `NodeConcurrency` fields to `NATSConfig`:

```go
type NATSConfig struct {
    SubjectJobEvent  string `json:"subject-job-event"`
    SubjectNodeState string `json:"subject-node-state"`
    QueueGroup       string `json:"queue-group"`
    JobConcurrency   int    `json:"job-concurrency"`
    NodeConcurrency  int    `json:"node-concurrency"`
}
```

Defaults: `JobConcurrency=8`, `NodeConcurrency=2`, `QueueGroup=""` (empty = no queue group for backward compat).

### 2. Queue groups — `internal/api/nats.go`

In `StartSubscriptions()`, use `SubscribeQueue()` when `QueueGroup` is configured, otherwise fall back to `Subscribe()`. This enables horizontal scaling across multiple cc-backend instances.

### 3. Channel-based subscriptions with worker pool — `internal/api/nats.go`

Replace direct `Subscribe`/`SubscribeQueue` callback model with `SubscribeChan()` + buffered channels + worker goroutines. This:
- Adds a Go-side buffer (configurable via semaphore size) between NATS delivery and processing
- Decouples NATS message delivery from processing
- Workers draw from the channel, applying the same semaphore-like concurrency via channel capacity + worker count

Architecture:
```
NATS server → SubscribeChan(buffered chan) → N worker goroutines → process message
```

The number of workers = `JobConcurrency` / `NodeConcurrency`. The channel buffer provides additional absorb capacity for bursts.

### Files to modify

- `internal/config/config.go:117-120` — add fields to `NATSConfig`
- `internal/api/nats.go` — rework `NewNatsAPI`, `StartSubscriptions`, replace semaphore pattern with channel workers
- `internal/api/nats_test.go` — update `setupNatsTest` to set new config, tests should still pass

### Verification

- `go build ./...`
- `go test ./internal/api/...` — all existing tests must pass
- `go vet ./internal/api/...`

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-ClusterCockpit-cc-backend/96740167-cb62-4e9b-a401-4dce5f9ae536.jsonl


Let me start by reading the relevant files to understand the current implementation.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→// Package config implements the program configuration data structures, validation and parsing
     7→package config
     8→
     9→import (
    10→	"bytes"
    11→	"encoding/json"
    12→	"time"
    13→
    14→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    15→	"github.com/ClusterCockpit/cc-lib/v2/resampler"
    16→)
    17→
    18→type ProgramConfig struct {
    19→	// Address where the http (or https) server will listen on (for example: 'localhost:80').
    20→	Addr string `json:"addr"`
    21→
    22→	// Addresses from which secured admin API endpoints can be reached, can be wildcard "*"
    23→	APIAllowedIPs []string `json:"api-allowed-ips"`
    24→
    25→	APISubjects *NATSConfig `json:"api-subjects"`
    26→
    27→	// Drop root permissions once .env was read and the port was taken.
    28→	User  string `json:"user"`
    29→	Group string `json:"group"`
    30→
    31→	// Disable authentication (for everything: API, Web-UI, ...)
    32→	DisableAuthentication bool `json:"disable-authentication"`
    33→
    34→	// If `embed-static-files` is true (default), the frontend files are directly
    35→	// embeded into the go binary and expected to be in web/frontend. Only if
    36→	// it is false the files in `static-files` are served instead.
    37→	EmbedStaticFiles bool   `json:"embed-static-files"`
    38→	StaticFiles      string `json:"static-files"`
    39→
    40→	// Path to SQLite database file
    41→	DB string `json:"db"`
    42→
    43→	EnableJobTaggers bool `json:"enable-job-taggers"`
    44→
    45→	// Validate json input against schema
    46→	Validate bool `json:"validate"`
    47→
    48→	// If 0 or empty, the session does not expire!
    49→	SessionMaxAge string `json:"session-max-age"`
    50→
    51→	// If both those options are not empty, use HTTPS using those certificates.
    52→	HTTPSCertFile string `json:"https-cert-file"`
    53→	HTTPSKeyFile  string `json:"https-key-file"`
    54→
    55→	// If not the empty string and `addr` does not end in ":80",
    56→	// redirect every request incoming at port 80 to that url.
    57→	RedirectHTTPTo string `json:"redirect-http-to"`
    58→
    59→	// Where to store MachineState files
    60→	MachineStateDir string `json:"machine-state-dir"`
    61→
    62→	// If not zero, automatically mark jobs as stopped running X seconds longer than their walltime.
    63→	StopJobsExceedingWalltime int `json:"stop-jobs-exceeding-walltime"`
    64→
    65→	// Defines time X in seconds in which jobs are considered to be "short" and will be filtered in specific views.
    66→	ShortRunningJobsDuration int `json:"short-running-jobs-duration"`
    67→
    68→	// Energy Mix CO2 Emission Constant [g/kWh]
    69→	// If entered, displays estimated CO2 emission for job based on jobs totalEnergy
    70→	EmissionConstant int `json:"emission-constant"`
    71→
    72→	// If exists, will enable dynamic zoom in frontend metric plots using the configured values
    73→	EnableResampling *ResampleConfig `json:"resampling"`
    74→
    75→	// Systemd unit name for log viewer (default: "clustercockpit")
    76→	SystemdUnit string `json:"systemd-unit"`
    77→
    78→	// Node state retention configuration
    79→	NodeStateRetention *NodeStateRetention `json:"nodestate-retention"`
    80→
    81→	// Database tuning configuration
    82→	DbConfig *DbConfig `json:"db-config"`
    83→}
    84→
    85→type DbConfig struct {
    86→	CacheSizeMB               int `json:"cache-size-mb"`
    87→	SoftHeapLimitMB           int `json:"soft-heap-limit-mb"`
    88→	MaxOpenConnections        int `json:"max-open-connections"`
    89→	MaxIdleConnections        int `json:"max-idle-connections"`
    90→	ConnectionMaxIdleTimeMins int `json:"max-idle-time-minutes"`
    91→	BusyTimeoutMs             int `json:"busy-timeout-ms"`
    92→}
    93→
    94→type NodeStateRetention struct {
    95→	Policy             string `json:"policy"`      // "delete" or "move"
    96→	Age                int    `json:"age"`         // hours, default 24
    97→	TargetKind         string `json:"target-kind"` // "file" or "s3"
    98→	TargetPath         string `json:"target-path"`
    99→	TargetEndpoint     string `json:"target-endpoint"`
   100→	TargetBucket       string `json:"target-bucket"`
   101→	TargetAccessKey    string `json:"target-access-key"`
   102→	TargetSecretKey    string `json:"target-secret-key"`
   103→	TargetRegion       string `json:"target-region"`
   104→	TargetUsePathStyle bool   `json:"target-use-path-style"`
   105→	MaxFileSizeMB      int    `json:"max-file-size-mb"`
   106→}
   107→
   108→type ResampleConfig struct {
   109→	// Minimum number of points to trigger resampling of data
   110→	MinimumPoints int `json:"minimum-points"`
   111→	// Array of resampling target resolutions, in seconds; Example: [600,300,60]
   112→	Resolutions []int `json:"resolutions"`
   113→	// Trigger next zoom level at less than this many visible datapoints
   114→	Trigger int `json:"trigger"`
   115→}
   116→
   117→type NATSConfig struct {
   118→	SubjectJobEvent  string `json:"subject-job-event"`
   119→	SubjectNodeState string `json:"subject-node-state"`
   120→}
   121→
   122→type IntRange struct {
   123→	From int `json:"from"`
   124→	To   int `json:"to"`
   125→}
   126→
   127→type TimeRange struct {
   128→	From  *time.Time `json:"from"`
   129→	To    *time.Time `json:"to"`
   130→	Range string     `json:"range,omitempty"`
   131→}
   132→
   133→type FilterRanges struct {
   134→	Duration  *IntRange  `json:"duration"`
   135→	NumNodes  *IntRange  `json:"num-nodes"`
   136→	StartTime *TimeRange `json:"start-time"`
   137→}
   138→
   139→var Keys ProgramConfig = ProgramConfig{
   140→	Addr:                      "localhost:8080",
   141→	EmbedStaticFiles:          true,
   142→	DB:                        "./var/job.db",
   143→	SessionMaxAge:             "168h",
   144→	StopJobsExceedingWalltime: 0,
   145→	ShortRunningJobsDuration:  5 * 60,
   146→}
   147→
   148→func Init(mainConfig json.RawMessage) {
   149→	Validate(configSchema, mainConfig)
   150→	dec := json.NewDecoder(bytes.NewReader(mainConfig))
   151→	dec.DisallowUnknownFields()
   152→	if err := dec.Decode(&Keys); err != nil {
   153→		cclog.Abortf("Config Init: Could not decode config file '%s'.\nError: %s\n", mainConfig, err.Error())
   154→	}
   155→
   156→	if Keys.EnableResampling != nil && Keys.EnableResampling.MinimumPoints > 0 {
   157→		resampler.SetMinimumRequiredPoints(Keys.EnableResampling.MinimumPoints)
   158→	}
   159→}
   160→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

// Package config implements the program configuration data structures, validation and parsing
package config

import (
	"bytes"
	"encoding/json"
	"time"

	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/resampler"
)

type ProgramConfig struct {
	// Address where the http (or https) server will listen on (for example: 'localhost:80').
	Addr string `json:"addr"`

	// Addresses from which secured admin API endpoints can be reached, can be wildcard "*"
	APIAllowedIPs []string `json:"api-allowed-ips"`

	APISubjects *NATSConfig `json:"api-subjects"`

	// Drop root permissions once .env was read and the port was taken.
	User  string `json:"user"`
	Group string `json:"group"`

	// Disable authentication (for everything: API, Web-UI, ...)
	DisableAuthentication bool `json:"disable-authentication"`

	// If `embed-static-files` is true (default), the frontend files are directly
	// embeded into the go binary and expected to be in web/frontend. Only if
	// it is false the files in `static-files` are served instead.
	EmbedStaticFiles bool   `json:"embed-static-files"`
	StaticFiles      string `json:"static-files"`

	// Path to SQLite database file
	DB string `json:"db"`

	EnableJobTaggers bool `json:"enable-job-taggers"`

	// Validate json input against schema
	Validate bool `json:"validate"`

	// If 0 or empty, the session does not expire!
	SessionMaxAge string `json:"session-max-age"`

	// If both those options are not empty, use HTTPS using those certificates.
	HTTPSCertFile string `json:"https-cert-file"`
	HTTPSKeyFile  string `json:"https-key-file"`

	// If not the empty string and `addr` does not end in ":80",
	// redirect every request incoming at port 80 to that url.
	RedirectHTTPTo string `json:"redirect-http-to"`

	// Where to store MachineState files
	MachineStateDir string `json:"machine-state-dir"`

	// If not zero, automatically mark jobs as stopped running X seconds longer than their walltime.
	StopJobsExceedingWalltime int `json:"stop-jobs-exceeding-walltime"`

	// Defines time X in seconds in which jobs are considered to be "short" and will be filtered in specific views.
	ShortRunningJobsDuration int `json:"short-running-jobs-duration"`

	// Energy Mix CO2 Emission Constant [g/kWh]
	// If entered, displays estimated CO2 emission for job based on jobs totalEnergy
	EmissionConstant int `json:"emission-constant"`

	// If exists, will enable dynamic zoom in frontend metric plots using the configured values
	EnableResampling *ResampleConfig `json:"resampling"`

	// Systemd unit name for log viewer (default: "clustercockpit")
	SystemdUnit string `json:"systemd-unit"`

	// Node state retention configuration
	NodeStateRetention *NodeStateRetention `json:"nodestate-retention"`

	// Database tuning configuration
	DbConfig *DbConfig `json:"db-config"`
}

type DbConfig struct {
	CacheSizeMB               int `json:"cache-size-mb"`
	SoftHeapLimitMB           int `json:"soft-heap-limit-mb"`
	MaxOpenConnections        int `json:"max-open-connections"`
	MaxIdleConnections        int `json:"max-idle-connections"`
	ConnectionMaxIdleTimeMins int `json:"max-idle-time-minutes"`
	BusyTimeoutMs             int `json:"busy-timeout-ms"`
}

type NodeStateRetention struct {
	Policy             string `json:"policy"`      // "delete" or "move"
	Age                int    `json:"age"`         // hours, default 24
	TargetKind         string `json:"target-kind"` // "file" or "s3"
	TargetPath         string `json:"target-path"`
	TargetEndpoint     string `json:"target-endpoint"`
	TargetBucket       string `json:"target-bucket"`
	TargetAccessKey    string `json:"target-access-key"`
	TargetSecretKey    string `json:"target-secret-key"`
	TargetRegion       string `json:"target-region"`
	TargetUsePathStyle bool   `json:"target-use-path-style"`
	MaxFileSizeMB      int    `json:"max-file-size-mb"`
}

type ResampleConfig struct {
	// Minimum number of points to trigger resampling of data
	MinimumPoints int `json:"minimum-points"`
	// Array of resampling target resolutions, in seconds; Example: [600,300,60]
	Resolutions []int `json:"resolutions"`
	// Trigger next zoom level at less than this many visible datapoints
	Trigger int `json:"trigger"`
}

type NATSConfig struct {
	SubjectJobEvent  string `json:"subject-job-event"`
	SubjectNodeState string `json:"subject-node-state"`
}

type IntRange struct {
	From int `json:"from"`
	To   int `json:"to"`
}

type TimeRange struct {
	From  *time.Time `json:"from"`
	To    *time.Time `json:"to"`
	Range string     `json:"range,omitempty"`
}

type FilterRanges struct {
	Duration  *IntRange  `json:"duration"`
	NumNodes  *IntRange  `json:"num-nodes"`
	StartTime *TimeRange `json:"start-time"`
}

var Keys ProgramConfig = ProgramConfig{
	Addr:                      "localhost:8080",
	EmbedStaticFiles:          true,
	DB:                        "./var/job.db",
	SessionMaxAge:             "168h",
	StopJobsExceedingWalltime: 0,
	ShortRunningJobsDuration:  5 * 60,
}

func Init(mainConfig json.RawMessage) {
	Validate(configSchema, mainConfig)
	dec := json.NewDecoder(bytes.NewReader(mainConfig))
	dec.DisallowUnknownFields()
	if err := dec.Decode(&Keys); err != nil {
		cclog.Abortf("Config Init: Could not decode config file '%s'.\nError: %s\n", mainConfig, err.Error())
	}

	if Keys.EnableResampling != nil && Keys.EnableResampling.MinimumPoints > 0 {
		resampler.SetMinimumRequiredPoints(Keys.EnableResampling.MinimumPoints)
	}
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package api
     7→
     8→import (
     9→	"database/sql"
    10→	"encoding/json"
    11→	"strings"
    12→	"sync"
    13→	"time"
    14→
    15→	"github.com/ClusterCockpit/cc-backend/internal/archiver"
    16→	"github.com/ClusterCockpit/cc-backend/internal/config"
    17→	"github.com/ClusterCockpit/cc-backend/internal/importer"
    18→	"github.com/ClusterCockpit/cc-backend/internal/repository"
    19→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    20→	lp "github.com/ClusterCockpit/cc-lib/v2/ccMessage"
    21→	"github.com/ClusterCockpit/cc-lib/v2/nats"
    22→	"github.com/ClusterCockpit/cc-lib/v2/receivers"
    23→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    24→	influx "github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
    25→)
    26→
    27→// NatsAPI provides NATS subscription-based handlers for Job and Node operations.
    28→// It mirrors the functionality of the REST API but uses NATS messaging with
    29→// InfluxDB line protocol as the message format.
    30→//
    31→// # Message Format
    32→//
    33→// All NATS messages use InfluxDB line protocol format (https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/)
    34→// with the following structure:
    35→//
    36→//	measurement,tag1=value1,tag2=value2 field1=value1,field2=value2 timestamp
    37→//
    38→// # Job Events
    39→//
    40→// Job start/stop events use the "job" measurement with a "function" tag to distinguish operations:
    41→//
    42→//	job,function=start_job event="{...JSON payload...}" <timestamp>
    43→//	job,function=stop_job event="{...JSON payload...}" <timestamp>
    44→//
    45→// The JSON payload in the "event" field follows the schema.Job or StopJobAPIRequest structure.
    46→//
    47→// Example job start message:
    48→//
    49→//	job,function=start_job event="{\"jobId\":1001,\"user\":\"testuser\",\"cluster\":\"testcluster\",...}" 1234567890000000000
    50→//
    51→// # Node State Events
    52→//
    53→// Node state updates use the "nodestate" measurement with cluster information:
    54→//
    55→//	nodestate event="{...JSON payload...}" <timestamp>
    56→//
    57→// The JSON payload follows the UpdateNodeStatesRequest structure.
    58→//
    59→// Example node state message:
    60→//
    61→//	nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"node01\",\"states\":[\"idle\"]}]}" 1234567890000000000
    62→type NatsAPI struct {
    63→	JobRepository *repository.JobRepository
    64→	// RepositoryMutex protects job creation operations from race conditions
    65→	// when checking for duplicate jobs during startJob calls.
    66→	RepositoryMutex sync.Mutex
    67→	// jobSem limits concurrent NATS job event processing to prevent goroutine accumulation
    68→	// when the database is under contention.
    69→	jobSem chan struct{}
    70→	// nodeSem limits concurrent NATS node state processing.
    71→	nodeSem chan struct{}
    72→}
    73→
    74→// NewNatsAPI creates a new NatsAPI instance with default dependencies.
    75→func NewNatsAPI() *NatsAPI {
    76→	return &NatsAPI{
    77→		JobRepository: repository.GetJobRepository(),
    78→		jobSem:        make(chan struct{}, 8),
    79→		nodeSem:       make(chan struct{}, 2),
    80→	}
    81→}
    82→
    83→// StartSubscriptions registers all NATS subscriptions for Job and Node APIs.
    84→// Returns an error if the NATS client is not available or subscription fails.
    85→func (api *NatsAPI) StartSubscriptions() error {
    86→	client := nats.GetClient()
    87→	if client == nil {
    88→		cclog.Warn("NATS client not available, skipping API subscriptions")
    89→		return nil
    90→	}
    91→
    92→	if config.Keys.APISubjects != nil {
    93→
    94→		s := config.Keys.APISubjects
    95→
    96→		if err := client.Subscribe(s.SubjectJobEvent, api.handleJobEvent); err != nil {
    97→			return err
    98→		}
    99→
   100→		if err := client.Subscribe(s.SubjectNodeState, api.handleNodeState); err != nil {
   101→			return err
   102→		}
   103→
   104→		cclog.Info("NATS API subscriptions started")
   105→	}
   106→	return nil
   107→}
   108→
   109→// processJobEvent routes job event messages to the appropriate handler based on the "function" tag.
   110→// Validates that required tags and fields are present before processing.
   111→func (api *NatsAPI) processJobEvent(msg lp.CCMessage) {
   112→	function, ok := msg.GetTag("function")
   113→	if !ok {
   114→		cclog.Errorf("Job event is missing required tag 'function': measurement=%s", msg.Name())
   115→		return
   116→	}
   117→
   118→	switch function {
   119→	case "start_job":
   120→		v, ok := msg.GetEventValue()
   121→		if !ok {
   122→			cclog.Errorf("Job start event is missing event field with JSON payload")
   123→			return
   124→		}
   125→		api.handleStartJob(v)
   126→
   127→	case "stop_job":
   128→		v, ok := msg.GetEventValue()
   129→		if !ok {
   130→			cclog.Errorf("Job stop event is missing event field with JSON payload")
   131→			return
   132→		}
   133→		api.handleStopJob(v)
   134→
   135→	default:
   136→		cclog.Warnf("Unknown job event function '%s', expected 'start_job' or 'stop_job'", function)
   137→	}
   138→}
   139→
   140→// handleJobEvent processes job-related messages received via NATS using InfluxDB line protocol.
   141→// The message must be in line protocol format with measurement="job" and include:
   142→//   - tag "function" with value "start_job" or "stop_job"
   143→//   - field "event" containing JSON payload (schema.Job or StopJobAPIRequest)
   144→//
   145→// Example: job,function=start_job event="{\"jobId\":1001,...}" 1234567890000000000
   146→func (api *NatsAPI) handleJobEvent(subject string, data []byte) {
   147→	api.jobSem <- struct{}{}
   148→	defer func() { <-api.jobSem }()
   149→
   150→	if len(data) == 0 {
   151→		cclog.Warnf("NATS %s: received empty message", subject)
   152→		return
   153→	}
   154→
   155→	d := influx.NewDecoderWithBytes(data)
   156→
   157→	for d.Next() {
   158→		m, err := receivers.DecodeInfluxMessage(d)
   159→		if err != nil {
   160→			cclog.Errorf("NATS %s: failed to decode InfluxDB line protocol message: %v", subject, err)
   161→			return
   162→		}
   163→
   164→		if !m.IsEvent() {
   165→			cclog.Debugf("NATS %s: received non-event message, skipping", subject)
   166→			continue
   167→		}
   168→
   169→		if m.Name() == "job" {
   170→			api.processJobEvent(m)
   171→		} else {
   172→			cclog.Debugf("NATS %s: unexpected measurement name '%s', expected 'job'", subject, m.Name())
   173→		}
   174→	}
   175→}
   176→
   177→// handleStartJob processes job start messages received via NATS.
   178→// The payload parameter contains JSON following the schema.Job structure.
   179→// Jobs are validated, checked for duplicates, and inserted into the database.
   180→func (api *NatsAPI) handleStartJob(payload string) {
   181→	if payload == "" {
   182→		cclog.Error("NATS start job: payload is empty")
   183→		return
   184→	}
   185→	req := schema.Job{
   186→		Shared:           "none",
   187→		MonitoringStatus: schema.MonitoringStatusRunningOrArchiving,
   188→	}
   189→
   190→	dec := json.NewDecoder(strings.NewReader(payload))
   191→	dec.DisallowUnknownFields()
   192→	if err := dec.Decode(&req); err != nil {
   193→		cclog.Errorf("NATS start job: parsing request failed: %v", err)
   194→		return
   195→	}
   196→
   197→	cclog.Debugf("NATS start job: %s", req.GoString())
   198→	req.State = schema.JobStateRunning
   199→
   200→	if err := importer.SanityChecks(&req); err != nil {
   201→		cclog.Errorf("NATS start job: sanity check failed: %v", err)
   202→		return
   203→	}
   204→
   205→	var unlockOnce sync.Once
   206→	api.RepositoryMutex.Lock()
   207→	defer unlockOnce.Do(api.RepositoryMutex.Unlock)
   208→
   209→	jobs, err := api.JobRepository.FindAll(&req.JobID, &req.Cluster, nil)
   210→	if err != nil && err != sql.ErrNoRows {
   211→		cclog.Errorf("NATS start job: checking for duplicate failed: %v", err)
   212→		return
   213→	}
   214→	if err == nil {
   215→		for _, job := range jobs {
   216→			if (req.StartTime - job.StartTime) < secondsPerDay {
   217→				cclog.Errorf("NATS start job: job with jobId %d, cluster %s already exists (dbid: %d)",
   218→					req.JobID, req.Cluster, job.ID)
   219→				return
   220→			}
   221→		}
   222→	}
   223→
   224→	// When tags are present, insert directly into the job table so that the
   225→	// returned ID can be used with AddTagOrCreate (which queries the job table).
   226→	var id int64
   227→	if len(req.Tags) > 0 {
   228→		id, err = api.JobRepository.StartDirect(&req)
   229→	} else {
   230→		id, err = api.JobRepository.Start(&req)
   231→	}
   232→	if err != nil {
   233→		cclog.Errorf("NATS start job: insert into database failed: %v", err)
   234→		return
   235→	}
   236→	unlockOnce.Do(api.RepositoryMutex.Unlock)
   237→
   238→	for _, tag := range req.Tags {
   239→		if _, err := api.JobRepository.AddTagOrCreate(nil, id, tag.Type, tag.Name, tag.Scope); err != nil {
   240→			cclog.Errorf("NATS start job: adding tag to new job %d failed: %v", id, err)
   241→			return
   242→		}
   243→	}
   244→
   245→	cclog.Infof("NATS: new job (id: %d): cluster=%s, jobId=%d, user=%s, startTime=%d",
   246→		id, req.Cluster, req.JobID, req.User, req.StartTime)
   247→}
   248→
   249→// handleStopJob processes job stop messages received via NATS.
   250→// The payload parameter contains JSON following the StopJobAPIRequest structure.
   251→// The job is marked as stopped in the database and archiving is triggered if monitoring is enabled.
   252→func (api *NatsAPI) handleStopJob(payload string) {
   253→	if payload == "" {
   254→		cclog.Error("NATS stop job: payload is empty")
   255→		return
   256→	}
   257→	var req StopJobAPIRequest
   258→
   259→	dec := json.NewDecoder(strings.NewReader(payload))
   260→	dec.DisallowUnknownFields()
   261→	if err := dec.Decode(&req); err != nil {
   262→		cclog.Errorf("NATS job stop: parsing request failed: %v", err)
   263→		return
   264→	}
   265→
   266→	if req.JobID == nil {
   267→		cclog.Errorf("NATS job stop: the field 'jobId' is required")
   268→		return
   269→	}
   270→
   271→	isCached := false
   272→	job, err := api.JobRepository.FindCached(req.JobID, req.Cluster, req.StartTime)
   273→	if err != nil {
   274→		// Not in cache, try main job table
   275→		job, err = api.JobRepository.Find(req.JobID, req.Cluster, req.StartTime)
   276→		if err != nil {
   277→			cclog.Errorf("NATS job stop: finding job failed: %v", err)
   278→			return
   279→		}
   280→	} else {
   281→		isCached = true
   282→	}
   283→
   284→	if job.State != schema.JobStateRunning {
   285→		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: job has already been stopped (state is: %s)",
   286→			job.JobID, job.ID, job.Cluster, job.State)
   287→		return
   288→	}
   289→
   290→	if job.StartTime > req.StopTime {
   291→		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: stopTime %d must be >= startTime %d",
   292→			job.JobID, job.ID, job.Cluster, req.StopTime, job.StartTime)
   293→		return
   294→	}
   295→
   296→	if req.State != "" && !req.State.Valid() {
   297→		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: invalid job state: %#v",
   298→			job.JobID, job.ID, job.Cluster, req.State)
   299→		return
   300→	} else if req.State == "" {
   301→		req.State = schema.JobStateCompleted
   302→	}
   303→
   304→	job.Duration = int32(req.StopTime - job.StartTime)
   305→	job.State = req.State
   306→	api.JobRepository.Mutex.Lock()
   307→	defer api.JobRepository.Mutex.Unlock()
   308→
   309→	// If the job is still in job_cache, transfer it to the job table first
   310→	if isCached {
   311→		newID, err := api.JobRepository.TransferCachedJobToMain(*job.ID)
   312→		if err != nil {
   313→			cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: transferring cached job failed: %v",
   314→				job.JobID, *job.ID, job.Cluster, err)
   315→			return
   316→		}
   317→		cclog.Infof("NATS: transferred cached job to main table: old id %d -> new id %d (jobId=%d)", *job.ID, newID, job.JobID)
   318→		job.ID = &newID
   319→	}
   320→
   321→	if err := api.JobRepository.Stop(*job.ID, job.Duration, job.State, job.MonitoringStatus); err != nil {
   322→		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: marking job as '%s' failed: %v",
   323→			job.JobID, *job.ID, job.Cluster, job.State, err)
   324→		return
   325→	}
   326→
   327→	cclog.Infof("NATS: archiving job (dbid: %d): cluster=%s, jobId=%d, user=%s, startTime=%d, duration=%d, state=%s",
   328→		*job.ID, job.Cluster, job.JobID, job.User, job.StartTime, job.Duration, job.State)
   329→
   330→	if job.MonitoringStatus == schema.MonitoringStatusDisabled {
   331→		return
   332→	}
   333→
   334→	archiver.TriggerArchiving(job)
   335→}
   336→
   337→// processNodestateEvent extracts and processes node state data from the InfluxDB message.
   338→// Updates node states in the repository for all nodes in the payload.
   339→func (api *NatsAPI) processNodestateEvent(msg lp.CCMessage) {
   340→	v, ok := msg.GetEventValue()
   341→	if !ok {
   342→		cclog.Errorf("Nodestate event is missing event field with JSON payload")
   343→		return
   344→	}
   345→
   346→	var req UpdateNodeStatesRequest
   347→
   348→	dec := json.NewDecoder(strings.NewReader(v))
   349→	dec.DisallowUnknownFields()
   350→	if err := dec.Decode(&req); err != nil {
   351→		cclog.Errorf("NATS nodestate: parsing request failed: %v", err)
   352→		return
   353→	}
   354→
   355→	repo := repository.GetNodeRepository()
   356→	requestReceived := time.Now().Unix()
   357→
   358→	updates := make([]repository.NodeStateUpdate, 0, len(req.Nodes))
   359→	for _, node := range req.Nodes {
   360→		state := determineState(node.States)
   361→		nodeState := schema.NodeStateDB{
   362→			TimeStamp:       requestReceived,
   363→			NodeState:       state,
   364→			CpusAllocated:   node.CpusAllocated,
   365→			MemoryAllocated: node.MemoryAllocated,
   366→			GpusAllocated:   node.GpusAllocated,
   367→			HealthState:     schema.MonitoringStateFull,
   368→			JobsRunning:     node.JobsRunning,
   369→		}
   370→		updates = append(updates, repository.NodeStateUpdate{
   371→			Hostname:  node.Hostname,
   372→			Cluster:   req.Cluster,
   373→			NodeState: &nodeState,
   374→		})
   375→	}
   376→
   377→	if err := repo.BatchUpdateNodeStates(updates); err != nil {
   378→		cclog.Errorf("NATS nodestate: batch update for cluster %s failed: %v", req.Cluster, err)
   379→	}
   380→
   381→	cclog.Debugf("NATS nodestate: updated %d node states for cluster %s", len(req.Nodes), req.Cluster)
   382→}
   383→
   384→// handleNodeState processes node state update messages received via NATS using InfluxDB line protocol.
   385→// The message must be in line protocol format with measurement="nodestate" and include:
   386→//   - field "event" containing JSON payload (UpdateNodeStatesRequest)
   387→//
   388→// Example: nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[...]}" 1234567890000000000
   389→func (api *NatsAPI) handleNodeState(subject string, data []byte) {
   390→	api.nodeSem <- struct{}{}
   391→	defer func() { <-api.nodeSem }()
   392→
   393→	if len(data) == 0 {
   394→		cclog.Warnf("NATS %s: received empty message", subject)
   395→		return
   396→	}
   397→
   398→	d := influx.NewDecoderWithBytes(data)
   399→
   400→	for d.Next() {
   401→		m, err := receivers.DecodeInfluxMessage(d)
   402→		if err != nil {
   403→			cclog.Errorf("NATS %s: failed to decode InfluxDB line protocol message: %v", subject, err)
   404→			return
   405→		}
   406→
   407→		if !m.IsEvent() {
   408→			cclog.Warnf("NATS %s: received non-event message, skipping", subject)
   409→			continue
   410→		}
   411→
   412→		if m.Name() == "nodestate" {
   413→			api.processNodestateEvent(m)
   414→		} else {
   415→			cclog.Warnf("NATS %s: unexpected measurement name '%s', expected 'nodestate'", subject, m.Name())
   416→		}
   417→	}
   418→}
   419→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package api

import (
	"database/sql"
	"encoding/json"
	"strings"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/archiver"
	"github.com/ClusterCockpit/cc-backend/internal/config"
	"github.com/ClusterCockpit/cc-backend/internal/importer"
	"github.com/ClusterCockpit/cc-backend/internal/repository"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	lp "github.com/ClusterCockpit/cc-lib/v2/ccMessage"
	"github.com/ClusterCockpit/cc-lib/v2/nats"
	"github.com/ClusterCockpit/cc-lib/v2/receivers"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	influx "github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
)

// NatsAPI provides NATS subscription-based handlers for Job and Node operations.
// It mirrors the functionality of the REST API but uses NATS messaging with
// InfluxDB line protocol as the message format.
//
// # Message Format
//
// All NATS messages use InfluxDB line protocol format (https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/)
// with the following structure:
//
//	measurement,tag1=value1,tag2=value2 field1=value1,field2=value2 timestamp
//
// # Job Events
//
// Job start/stop events use the "job" measurement with a "function" tag to distinguish operations:
//
//	job,function=start_job event="{...JSON payload...}" <timestamp>
//	job,function=stop_job event="{...JSON payload...}" <timestamp>
//
// The JSON payload in the "event" field follows the schema.Job or StopJobAPIRequest structure.
//
// Example job start message:
//
//	job,function=start_job event="{\"jobId\":1001,\"user\":\"testuser\",\"cluster\":\"testcluster\",...}" 1234567890000000000
//
// # Node State Events
//
// Node state updates use the "nodestate" measurement with cluster information:
//
//	nodestate event="{...JSON payload...}" <timestamp>
//
// The JSON payload follows the UpdateNodeStatesRequest structure.
//
// Example node state message:
//
//	nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"node01\",\"states\":[\"idle\"]}]}" 1234567890000000000
type NatsAPI struct {
	JobRepository *repository.JobRepository
	// RepositoryMutex protects job creation operations from race conditions
	// when checking for duplicate jobs during startJob calls.
	RepositoryMutex sync.Mutex
	// jobSem limits concurrent NATS job event processing to prevent goroutine accumulation
	// when the database is under contention.
	jobSem chan struct{}
	// nodeSem limits concurrent NATS node state processing.
	nodeSem chan struct{}
}

// NewNatsAPI creates a new NatsAPI instance with default dependencies.
func NewNatsAPI() *NatsAPI {
	return &NatsAPI{
		JobRepository: repository.GetJobRepository(),
		jobSem:        make(chan struct{}, 8),
		nodeSem:       make(chan struct{}, 2),
	}
}

// StartSubscriptions registers all NATS subscriptions for Job and Node APIs.
// Returns an error if the NATS client is not available or subscription fails.
func (api *NatsAPI) StartSubscriptions() error {
	client := nats.GetClient()
	if client == nil {
		cclog.Warn("NATS client not available, skipping API subscriptions")
		return nil
	}

	if config.Keys.APISubjects != nil {

		s := config.Keys.APISubjects

		if err := client.Subscribe(s.SubjectJobEvent, api.handleJobEvent); err != nil {
			return err
		}

		if err := client.Subscribe(s.SubjectNodeState, api.handleNodeState); err != nil {
			return err
		}

		cclog.Info("NATS API subscriptions started")
	}
	return nil
}

// processJobEvent routes job event messages to the appropriate handler based on the "function" tag.
// Validates that required tags and fields are present before processing.
func (api *NatsAPI) processJobEvent(msg lp.CCMessage) {
	function, ok := msg.GetTag("function")
	if !ok {
		cclog.Errorf("Job event is missing required tag 'function': measurement=%s", msg.Name())
		return
	}

	switch function {
	case "start_job":
		v, ok := msg.GetEventValue()
		if !ok {
			cclog.Errorf("Job start event is missing event field with JSON payload")
			return
		}
		api.handleStartJob(v)

	case "stop_job":
		v, ok := msg.GetEventValue()
		if !ok {
			cclog.Errorf("Job stop event is missing event field with JSON payload")
			return
		}
		api.handleStopJob(v)

	default:
		cclog.Warnf("Unknown job event function '%s', expected 'start_job' or 'stop_job'", function)
	}
}

// handleJobEvent processes job-related messages received via NATS using InfluxDB line protocol.
// The message must be in line protocol format with measurement="job" and include:
//   - tag "function" with value "start_job" or "stop_job"
//   - field "event" containing JSON payload (schema.Job or StopJobAPIRequest)
//
// Example: job,function=start_job event="{\"jobId\":1001,...}" 1234567890000000000
func (api *NatsAPI) handleJobEvent(subject string, data []byte) {
	api.jobSem <- struct{}{}
	defer func() { <-api.jobSem }()

	if len(data) == 0 {
		cclog.Warnf("NATS %s: received empty message", subject)
		return
	}

	d := influx.NewDecoderWithBytes(data)

	for d.Next() {
		m, err := receivers.DecodeInfluxMessage(d)
		if err != nil {
			cclog.Errorf("NATS %s: failed to decode InfluxDB line protocol message: %v", subject, err)
			return
		}

		if !m.IsEvent() {
			cclog.Debugf("NATS %s: received non-event message, skipping", subject)
			continue
		}

		if m.Name() == "job" {
			api.processJobEvent(m)
		} else {
			cclog.Debugf("NATS %s: unexpected measurement name '%s', expected 'job'", subject, m.Name())
		}
	}
}

// handleStartJob processes job start messages received via NATS.
// The payload parameter contains JSON following the schema.Job structure.
// Jobs are validated, checked for duplicates, and inserted into the database.
func (api *NatsAPI) handleStartJob(payload string) {
	if payload == "" {
		cclog.Error("NATS start job: payload is empty")
		return
	}
	req := schema.Job{
		Shared:           "none",
		MonitoringStatus: schema.MonitoringStatusRunningOrArchiving,
	}

	dec := json.NewDecoder(strings.NewReader(payload))
	dec.DisallowUnknownFields()
	if err := dec.Decode(&req); err != nil {
		cclog.Errorf("NATS start job: parsing request failed: %v", err)
		return
	}

	cclog.Debugf("NATS start job: %s", req.GoString())
	req.State = schema.JobStateRunning

	if err := importer.SanityChecks(&req); err != nil {
		cclog.Errorf("NATS start job: sanity check failed: %v", err)
		return
	}

	var unlockOnce sync.Once
	api.RepositoryMutex.Lock()
	defer unlockOnce.Do(api.RepositoryMutex.Unlock)

	jobs, err := api.JobRepository.FindAll(&req.JobID, &req.Cluster, nil)
	if err != nil && err != sql.ErrNoRows {
		cclog.Errorf("NATS start job: checking for duplicate failed: %v", err)
		return
	}
	if err == nil {
		for _, job := range jobs {
			if (req.StartTime - job.StartTime) < secondsPerDay {
				cclog.Errorf("NATS start job: job with jobId %d, cluster %s already exists (dbid: %d)",
					req.JobID, req.Cluster, job.ID)
				return
			}
		}
	}

	// When tags are present, insert directly into the job table so that the
	// returned ID can be used with AddTagOrCreate (which queries the job table).
	var id int64
	if len(req.Tags) > 0 {
		id, err = api.JobRepository.StartDirect(&req)
	} else {
		id, err = api.JobRepository.Start(&req)
	}
	if err != nil {
		cclog.Errorf("NATS start job: insert into database failed: %v", err)
		return
	}
	unlockOnce.Do(api.RepositoryMutex.Unlock)

	for _, tag := range req.Tags {
		if _, err := api.JobRepository.AddTagOrCreate(nil, id, tag.Type, tag.Name, tag.Scope); err != nil {
			cclog.Errorf("NATS start job: adding tag to new job %d failed: %v", id, err)
			return
		}
	}

	cclog.Infof("NATS: new job (id: %d): cluster=%s, jobId=%d, user=%s, startTime=%d",
		id, req.Cluster, req.JobID, req.User, req.StartTime)
}

// handleStopJob processes job stop messages received via NATS.
// The payload parameter contains JSON following the StopJobAPIRequest structure.
// The job is marked as stopped in the database and archiving is triggered if monitoring is enabled.
func (api *NatsAPI) handleStopJob(payload string) {
	if payload == "" {
		cclog.Error("NATS stop job: payload is empty")
		return
	}
	var req StopJobAPIRequest

	dec := json.NewDecoder(strings.NewReader(payload))
	dec.DisallowUnknownFields()
	if err := dec.Decode(&req); err != nil {
		cclog.Errorf("NATS job stop: parsing request failed: %v", err)
		return
	}

	if req.JobID == nil {
		cclog.Errorf("NATS job stop: the field 'jobId' is required")
		return
	}

	isCached := false
	job, err := api.JobRepository.FindCached(req.JobID, req.Cluster, req.StartTime)
	if err != nil {
		// Not in cache, try main job table
		job, err = api.JobRepository.Find(req.JobID, req.Cluster, req.StartTime)
		if err != nil {
			cclog.Errorf("NATS job stop: finding job failed: %v", err)
			return
		}
	} else {
		isCached = true
	}

	if job.State != schema.JobStateRunning {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: job has already been stopped (state is: %s)",
			job.JobID, job.ID, job.Cluster, job.State)
		return
	}

	if job.StartTime > req.StopTime {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: stopTime %d must be >= startTime %d",
			job.JobID, job.ID, job.Cluster, req.StopTime, job.StartTime)
		return
	}

	if req.State != "" && !req.State.Valid() {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: invalid job state: %#v",
			job.JobID, job.ID, job.Cluster, req.State)
		return
	} else if req.State == "" {
		req.State = schema.JobStateCompleted
	}

	job.Duration = int32(req.StopTime - job.StartTime)
	job.State = req.State
	api.JobRepository.Mutex.Lock()
	defer api.JobRepository.Mutex.Unlock()

	// If the job is still in job_cache, transfer it to the job table first
	if isCached {
		newID, err := api.JobRepository.TransferCachedJobToMain(*job.ID)
		if err != nil {
			cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: transferring cached job failed: %v",
				job.JobID, *job.ID, job.Cluster, err)
			return
		}
		cclog.Infof("NATS: transferred cached job to main table: old id %d -> new id %d (jobId=%d)", *job.ID, newID, job.JobID)
		job.ID = &newID
	}

	if err := api.JobRepository.Stop(*job.ID, job.Duration, job.State, job.MonitoringStatus); err != nil {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: marking job as '%s' failed: %v",
			job.JobID, *job.ID, job.Cluster, job.State, err)
		return
	}

	cclog.Infof("NATS: archiving job (dbid: %d): cluster=%s, jobId=%d, user=%s, startTime=%d, duration=%d, state=%s",
		*job.ID, job.Cluster, job.JobID, job.User, job.StartTime, job.Duration, job.State)

	if job.MonitoringStatus == schema.MonitoringStatusDisabled {
		return
	}

	archiver.TriggerArchiving(job)
}

// processNodestateEvent extracts and processes node state data from the InfluxDB message.
// Updates node states in the repository for all nodes in the payload.
func (api *NatsAPI) processNodestateEvent(msg lp.CCMessage) {
	v, ok := msg.GetEventValue()
	if !ok {
		cclog.Errorf("Nodestate event is missing event field with JSON payload")
		return
	}

	var req UpdateNodeStatesRequest

	dec := json.NewDecoder(strings.NewReader(v))
	dec.DisallowUnknownFields()
	if err := dec.Decode(&req); err != nil {
		cclog.Errorf("NATS nodestate: parsing request failed: %v", err)
		return
	}

	repo := repository.GetNodeRepository()
	requestReceived := time.Now().Unix()

	updates := make([]repository.NodeStateUpdate, 0, len(req.Nodes))
	for _, node := range req.Nodes {
		state := determineState(node.States)
		nodeState := schema.NodeStateDB{
			TimeStamp:       requestReceived,
			NodeState:       state,
			CpusAllocated:   node.CpusAllocated,
			MemoryAllocated: node.MemoryAllocated,
			GpusAllocated:   node.GpusAllocated,
			HealthState:     schema.MonitoringStateFull,
			JobsRunning:     node.JobsRunning,
		}
		updates = append(updates, repository.NodeStateUpdate{
			Hostname:  node.Hostname,
			Cluster:   req.Cluster,
			NodeState: &nodeState,
		})
	}

	if err := repo.BatchUpdateNodeStates(updates); err != nil {
		cclog.Errorf("NATS nodestate: batch update for cluster %s failed: %v", req.Cluster, err)
	}

	cclog.Debugf("NATS nodestate: updated %d node states for cluster %s", len(req.Nodes), req.Cluster)
}

// handleNodeState processes node state update messages received via NATS using InfluxDB line protocol.
// The message must be in line protocol format with measurement="nodestate" and include:
//   - field "event" containing JSON payload (UpdateNodeStatesRequest)
//
// Example: nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[...]}" 1234567890000000000
func (api *NatsAPI) handleNodeState(subject string, data []byte) {
	api.nodeSem <- struct{}{}
	defer func() { <-api.nodeSem }()

	if len(data) == 0 {
		cclog.Warnf("NATS %s: received empty message", subject)
		return
	}

	d := influx.NewDecoderWithBytes(data)

	for d.Next() {
		m, err := receivers.DecodeInfluxMessage(d)
		if err != nil {
			cclog.Errorf("NATS %s: failed to decode InfluxDB line protocol message: %v", subject, err)
			return
		}

		if !m.IsEvent() {
			cclog.Warnf("NATS %s: received non-event message, skipping", subject)
			continue
		}

		if m.Name() == "nodestate" {
			api.processNodestateEvent(m)
		} else {
			cclog.Warnf("NATS %s: unexpected measurement name '%s', expected 'nodestate'", subject, m.Name())
		}
	}
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→package api
     6→
     7→import (
     8→	"context"
     9→	"database/sql"
    10→	"encoding/json"
    11→	"fmt"
    12→	"os"
    13→	"path/filepath"
    14→	"testing"
    15→	"time"
    16→
    17→	"github.com/ClusterCockpit/cc-backend/internal/archiver"
    18→	"github.com/ClusterCockpit/cc-backend/internal/auth"
    19→	"github.com/ClusterCockpit/cc-backend/internal/config"
    20→	"github.com/ClusterCockpit/cc-backend/internal/graph"
    21→	"github.com/ClusterCockpit/cc-backend/internal/repository"
    22→	"github.com/ClusterCockpit/cc-backend/pkg/archive"
    23→	"github.com/ClusterCockpit/cc-backend/pkg/metricstore"
    24→	ccconf "github.com/ClusterCockpit/cc-lib/v2/ccConfig"
    25→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    26→	lp "github.com/ClusterCockpit/cc-lib/v2/ccMessage"
    27→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    28→
    29→	_ "github.com/mattn/go-sqlite3"
    30→)
    31→
    32→func setupNatsTest(t *testing.T) *NatsAPI {
    33→	repository.ResetConnection()
    34→
    35→	const testconfig = `{
    36→		"main": {
    37→	"addr":            "0.0.0.0:8080",
    38→	"validate": false,
    39→  "api-allowed-ips": [
    40→    "*"
    41→  ]
    42→	},
    43→	"archive": {
    44→		"kind": "file",
    45→		"path": "./var/job-archive"
    46→	},
    47→	"auth": {
    48→  "jwts": {
    49→      "max-age": "2m"
    50→  }
    51→	}
    52→}`
    53→	const testclusterJSON = `{
    54→        "name": "testcluster",
    55→		"subClusters": [
    56→			{
    57→				"name": "sc1",
    58→				"nodes": "host123,host124,host125",
    59→				"processorType": "Intel Core i7-4770",
    60→				"socketsPerNode": 1,
    61→				"coresPerSocket": 4,
    62→				"threadsPerCore": 2,
    63→                "flopRateScalar": {
    64→                  "unit": {
    65→                    "prefix": "G",
    66→                    "base": "F/s"
    67→                  },
    68→                  "value": 14
    69→                },
    70→                "flopRateSimd": {
    71→                  "unit": {
    72→                    "prefix": "G",
    73→                    "base": "F/s"
    74→                  },
    75→                  "value": 112
    76→                },
    77→                "memoryBandwidth": {
    78→                  "unit": {
    79→                    "prefix": "G",
    80→                    "base": "B/s"
    81→                  },
    82→                  "value": 24
    83→                },
    84→                "numberOfNodes": 70,
    85→				"topology": {
    86→					"node": [0, 1, 2, 3, 4, 5, 6, 7],
    87→					"socket": [[0, 1, 2, 3, 4, 5, 6, 7]],
    88→					"memoryDomain": [[0, 1, 2, 3, 4, 5, 6, 7]],
    89→					"die": [[0, 1, 2, 3, 4, 5, 6, 7]],
    90→					"core": [[0], [1], [2], [3], [4], [5], [6], [7]]
    91→				}
    92→			}
    93→		],
    94→		"metricConfig": [
    95→			{
    96→				"name": "load_one",
    97→			    "unit": { "base": ""},
    98→				"scope": "node",
    99→				"timestep": 60,
   100→                "aggregation": "avg",
   101→				"peak": 8,
   102→				"normal": 0,
   103→				"caution": 0,
   104→				"alert": 0
   105→			}
   106→		]
   107→	}`
   108→
   109→	cclog.Init("info", true)
   110→	tmpdir := t.TempDir()
   111→	jobarchive := filepath.Join(tmpdir, "job-archive")
   112→	if err := os.Mkdir(jobarchive, 0o777); err != nil {
   113→		t.Fatal(err)
   114→	}
   115→
   116→	if err := os.WriteFile(filepath.Join(jobarchive, "version.txt"), fmt.Appendf(nil, "%d", 3), 0o666); err != nil {
   117→		t.Fatal(err)
   118→	}
   119→
   120→	if err := os.Mkdir(filepath.Join(jobarchive, "testcluster"), 0o777); err != nil {
   121→		t.Fatal(err)
   122→	}
   123→
   124→	if err := os.WriteFile(filepath.Join(jobarchive, "testcluster", "cluster.json"), []byte(testclusterJSON), 0o666); err != nil {
   125→		t.Fatal(err)
   126→	}
   127→
   128→	dbfilepath := filepath.Join(tmpdir, "test.db")
   129→	err := repository.MigrateDB(dbfilepath)
   130→	if err != nil {
   131→		t.Fatal(err)
   132→	}
   133→
   134→	cfgFilePath := filepath.Join(tmpdir, "config.json")
   135→	if err := os.WriteFile(cfgFilePath, []byte(testconfig), 0o666); err != nil {
   136→		t.Fatal(err)
   137→	}
   138→
   139→	ccconf.Init(cfgFilePath)
   140→
   141→	// Load and check main configuration
   142→	if cfg := ccconf.GetPackageConfig("main"); cfg != nil {
   143→		config.Init(cfg)
   144→	} else {
   145→		cclog.Abort("Main configuration must be present")
   146→	}
   147→	archiveCfg := fmt.Sprintf("{\"kind\": \"file\",\"path\": \"%s\"}", jobarchive)
   148→
   149→	repository.Connect(dbfilepath)
   150→
   151→	if err := archive.Init(json.RawMessage(archiveCfg)); err != nil {
   152→		t.Fatal(err)
   153→	}
   154→
   155→	// metricstore initialization removed - it's initialized via callback in tests
   156→
   157→	archiver.Start(repository.GetJobRepository(), context.Background())
   158→
   159→	if cfg := ccconf.GetPackageConfig("auth"); cfg != nil {
   160→		auth.Init(&cfg)
   161→	} else {
   162→		cclog.Warn("Authentication disabled due to missing configuration")
   163→		auth.Init(nil)
   164→	}
   165→
   166→	graph.Init()
   167→
   168→	return NewNatsAPI()
   169→}
   170→
   171→func cleanupNatsTest() {
   172→	if err := archiver.Shutdown(5 * time.Second); err != nil {
   173→		cclog.Warnf("Archiver shutdown timeout in tests: %v", err)
   174→	}
   175→}
   176→
   177→func TestNatsHandleStartJob(t *testing.T) {
   178→	natsAPI := setupNatsTest(t)
   179→	t.Cleanup(cleanupNatsTest)
   180→
   181→	tests := []struct {
   182→		name          string
   183→		payload       string
   184→		expectError   bool
   185→		validateJob   func(t *testing.T, job *schema.Job)
   186→		shouldFindJob bool
   187→	}{
   188→		{
   189→			name: "valid job start",
   190→			payload: `{
   191→				"jobId": 1001,
   192→				"user": "testuser1",
   193→				"project": "testproj1",
   194→				"cluster": "testcluster",
   195→				"partition": "main",
   196→				"walltime": 7200,
   197→				"numNodes": 1,
   198→				"numHwthreads": 8,
   199→				"numAcc": 0,
   200→				"shared": "none",
   201→				"monitoringStatus": 1,
   202→				"smt": 1,
   203→				"resources": [
   204→					{
   205→						"hostname": "host123",
   206→						"hwthreads": [0, 1, 2, 3, 4, 5, 6, 7]
   207→					}
   208→				],
   209→				"startTime": 1234567890
   210→			}`,
   211→			expectError:   false,
   212→			shouldFindJob: true,
   213→			validateJob: func(t *testing.T, job *schema.Job) {
   214→				if job.JobID != 1001 {
   215→					t.Errorf("expected JobID 1001, got %d", job.JobID)
   216→				}
   217→				if job.User != "testuser1" {
   218→					t.Errorf("expected user testuser1, got %s", job.User)
   219→				}
   220→				if job.State != schema.JobStateRunning {
   221→					t.Errorf("expected state running, got %s", job.State)
   222→				}
   223→			},
   224→		},
   225→		{
   226→			name: "invalid JSON",
   227→			payload: `{
   228→				"jobId": "not a number",
   229→				"user": "testuser2"
   230→			}`,
   231→			expectError:   true,
   232→			shouldFindJob: false,
   233→		},
   234→		{
   235→			name: "missing required fields",
   236→			payload: `{
   237→				"jobId": 1002
   238→			}`,
   239→			expectError:   true,
   240→			shouldFindJob: false,
   241→		},
   242→		{
   243→			name: "job with unknown fields (should fail due to DisallowUnknownFields)",
   244→			payload: `{
   245→				"jobId": 1003,
   246→				"user": "testuser3",
   247→				"project": "testproj3",
   248→				"cluster": "testcluster",
   249→				"partition": "main",
   250→				"walltime": 3600,
   251→				"numNodes": 1,
   252→				"numHwthreads": 8,
   253→				"unknownField": "should cause error",
   254→				"startTime": 1234567900
   255→			}`,
   256→			expectError:   true,
   257→			shouldFindJob: false,
   258→		},
   259→		{
   260→			name: "job with tags",
   261→			payload: `{
   262→				"jobId": 1004,
   263→				"user": "testuser4",
   264→				"project": "testproj4",
   265→				"cluster": "testcluster",
   266→				"partition": "main",
   267→				"walltime": 3600,
   268→				"numNodes": 1,
   269→				"numHwthreads": 8,
   270→				"numAcc": 0,
   271→				"shared": "none",
   272→				"monitoringStatus": 1,
   273→				"smt": 1,
   274→				"resources": [
   275→					{
   276→						"hostname": "host123",
   277→						"hwthreads": [0, 1, 2, 3]
   278→					}
   279→				],
   280→				"tags": [
   281→					{
   282→						"type": "test",
   283→						"name": "testtag",
   284→						"scope": "testuser4"
   285→					}
   286→				],
   287→				"startTime": 1234567910
   288→			}`,
   289→			expectError:   false,
   290→			shouldFindJob: true,
   291→			validateJob: func(t *testing.T, job *schema.Job) {
   292→				if job.JobID != 1004 {
   293→					t.Errorf("expected JobID 1004, got %d", job.JobID)
   294→				}
   295→			},
   296→		},
   297→	}
   298→
   299→	for _, tt := range tests {
   300→		t.Run(tt.name, func(t *testing.T) {
   301→			natsAPI.handleStartJob(tt.payload)
   302→			natsAPI.JobRepository.SyncJobs()
   303→
   304→			// Allow some time for async operations
   305→			time.Sleep(100 * time.Millisecond)
   306→
   307→			if tt.shouldFindJob {
   308→				// Extract jobId from payload
   309→				var payloadMap map[string]any
   310→				json.Unmarshal([]byte(tt.payload), &payloadMap)
   311→				jobID := int64(payloadMap["jobId"].(float64))
   312→				cluster := payloadMap["cluster"].(string)
   313→				startTime := int64(payloadMap["startTime"].(float64))
   314→
   315→				job, err := natsAPI.JobRepository.Find(&jobID, &cluster, &startTime)
   316→				if err != nil {
   317→					if !tt.expectError {
   318→						t.Fatalf("expected to find job, but got error: %v", err)
   319→					}
   320→					return
   321→				}
   322→
   323→				if tt.validateJob != nil {
   324→					tt.validateJob(t, job)
   325→				}
   326→			}
   327→		})
   328→	}
   329→}
   330→
   331→func TestNatsHandleStopJob(t *testing.T) {
   332→	natsAPI := setupNatsTest(t)
   333→	t.Cleanup(cleanupNatsTest)
   334→
   335→	// First, create a running job
   336→	startPayload := `{
   337→		"jobId": 2001,
   338→		"user": "testuser",
   339→		"project": "testproj",
   340→		"cluster": "testcluster",
   341→		"partition": "main",
   342→		"walltime": 3600,
   343→		"numNodes": 1,
   344→		"numHwthreads": 8,
   345→		"numAcc": 0,
   346→		"shared": "none",
   347→		"monitoringStatus": 1,
   348→		"smt": 1,
   349→		"resources": [
   350→			{
   351→				"hostname": "host123",
   352→				"hwthreads": [0, 1, 2, 3, 4, 5, 6, 7]
   353→			}
   354→		],
   355→		"startTime": 1234567890
   356→	}`
   357→
   358→	natsAPI.handleStartJob(startPayload)
   359→	natsAPI.JobRepository.SyncJobs()
   360→	time.Sleep(100 * time.Millisecond)
   361→
   362→	tests := []struct {
   363→		name         string
   364→		payload      string
   365→		expectError  bool
   366→		validateJob  func(t *testing.T, job *schema.Job)
   367→		setupJobFunc func() // Optional: create specific test job
   368→	}{
   369→		{
   370→			name: "valid job stop - completed",
   371→			payload: `{
   372→				"jobId": 2001,
   373→				"cluster": "testcluster",
   374→				"startTime": 1234567890,
   375→				"jobState": "completed",
   376→				"stopTime": 1234571490
   377→			}`,
   378→			expectError: false,
   379→			validateJob: func(t *testing.T, job *schema.Job) {
   380→				if job.State != schema.JobStateCompleted {
   381→					t.Errorf("expected state completed, got %s", job.State)
   382→				}
   383→				expectedDuration := int32(1234571490 - 1234567890)
   384→				if job.Duration != expectedDuration {
   385→					t.Errorf("expected duration %d, got %d", expectedDuration, job.Duration)
   386→				}
   387→			},
   388→		},
   389→		{
   390→			name: "valid job stop - failed",
   391→			setupJobFunc: func() {
   392→				startPayloadFailed := `{
   393→					"jobId": 2002,
   394→					"user": "testuser",
   395→					"project": "testproj",
   396→					"cluster": "testcluster",
   397→					"partition": "main",
   398→					"walltime": 3600,
   399→					"numNodes": 1,
   400→					"numHwthreads": 8,
   401→					"numAcc": 0,
   402→					"shared": "none",
   403→					"monitoringStatus": 1,
   404→					"smt": 1,
   405→					"resources": [
   406→						{
   407→							"hostname": "host123",
   408→							"hwthreads": [0, 1, 2, 3]
   409→						}
   410→					],
   411→					"startTime": 1234567900
   412→				}`
   413→				natsAPI.handleStartJob(startPayloadFailed)
   414→				natsAPI.JobRepository.SyncJobs()
   415→				time.Sleep(100 * time.Millisecond)
   416→			},
   417→			payload: `{
   418→				"jobId": 2002,
   419→				"cluster": "testcluster",
   420→				"startTime": 1234567900,
   421→				"jobState": "failed",
   422→				"stopTime": 1234569900
   423→			}`,
   424→			expectError: false,
   425→			validateJob: func(t *testing.T, job *schema.Job) {
   426→				if job.State != schema.JobStateFailed {
   427→					t.Errorf("expected state failed, got %s", job.State)
   428→				}
   429→			},
   430→		},
   431→		{
   432→			name: "invalid JSON",
   433→			payload: `{
   434→				"jobId": "not a number"
   435→			}`,
   436→			expectError: true,
   437→		},
   438→		{
   439→			name: "missing jobId",
   440→			payload: `{
   441→				"cluster": "testcluster",
   442→				"jobState": "completed",
   443→				"stopTime": 1234571490
   444→			}`,
   445→			expectError: true,
   446→		},
   447→		{
   448→			name: "invalid job state",
   449→			setupJobFunc: func() {
   450→				startPayloadInvalid := `{
   451→					"jobId": 2003,
   452→					"user": "testuser",
   453→					"project": "testproj",
   454→					"cluster": "testcluster",
   455→					"partition": "main",
   456→					"walltime": 3600,
   457→					"numNodes": 1,
   458→					"numHwthreads": 8,
   459→					"numAcc": 0,
   460→					"shared": "none",
   461→					"monitoringStatus": 1,
   462→					"smt": 1,
   463→					"resources": [
   464→						{
   465→							"hostname": "host123",
   466→							"hwthreads": [0, 1]
   467→						}
   468→					],
   469→					"startTime": 1234567910
   470→				}`
   471→				natsAPI.handleStartJob(startPayloadInvalid)
   472→				natsAPI.JobRepository.SyncJobs()
   473→				time.Sleep(100 * time.Millisecond)
   474→			},
   475→			payload: `{
   476→				"jobId": 2003,
   477→				"cluster": "testcluster",
   478→				"startTime": 1234567910,
   479→				"jobState": "invalid_state",
   480→				"stopTime": 1234571510
   481→			}`,
   482→			expectError: true,
   483→		},
   484→		{
   485→			name: "stopTime before startTime",
   486→			setupJobFunc: func() {
   487→				startPayloadTime := `{
   488→					"jobId": 2004,
   489→					"user": "testuser",
   490→					"project": "testproj",
   491→					"cluster": "testcluster",
   492→					"partition": "main",
   493→					"walltime": 3600,
   494→					"numNodes": 1,
   495→					"numHwthreads": 8,
   496→					"numAcc": 0,
   497→					"shared": "none",
   498→					"monitoringStatus": 1,
   499→					"smt": 1,
   500→					"resources": [
   501→						{
   502→							"hostname": "host123",
   503→							"hwthreads": [0]
   504→						}
   505→					],
   506→					"startTime": 1234567920
   507→				}`
   508→				natsAPI.handleStartJob(startPayloadTime)
   509→				natsAPI.JobRepository.SyncJobs()
   510→				time.Sleep(100 * time.Millisecond)
   511→			},
   512→			payload: `{
   513→				"jobId": 2004,
   514→				"cluster": "testcluster",
   515→				"startTime": 1234567920,
   516→				"jobState": "completed",
   517→				"stopTime": 1234567900
   518→			}`,
   519→			expectError: true,
   520→		},
   521→		{
   522→			name: "job not found",
   523→			payload: `{
   524→				"jobId": 99999,
   525→				"cluster": "testcluster",
   526→				"startTime": 1234567890,
   527→				"jobState": "completed",
   528→				"stopTime": 1234571490
   529→			}`,
   530→			expectError: true,
   531→		},
   532→	}
   533→
   534→	testData := schema.JobData{
   535→		"load_one": map[schema.MetricScope]*schema.JobMetric{
   536→			schema.MetricScopeNode: {
   537→				Unit:     schema.Unit{Base: "load"},
   538→				Timestep: 60,
   539→				Series: []schema.Series{
   540→					{
   541→						Hostname:   "host123",
   542→						Statistics: schema.MetricStatistics{Min: 0.1, Avg: 0.2, Max: 0.3},
   543→						Data:       []schema.Float{0.1, 0.1, 0.1, 0.2, 0.2, 0.2, 0.3, 0.3, 0.3},
   544→					},
   545→				},
   546→			},
   547→		},
   548→	}
   549→
   550→	metricstore.TestLoadDataCallback = func(job *schema.Job, metrics []string, scopes []schema.MetricScope, ctx context.Context, resolution int) (schema.JobData, error) {
   551→		return testData, nil
   552→	}
   553→
   554→	for _, tt := range tests {
   555→		t.Run(tt.name, func(t *testing.T) {
   556→			if tt.setupJobFunc != nil {
   557→				tt.setupJobFunc()
   558→			}
   559→
   560→			natsAPI.handleStopJob(tt.payload)
   561→
   562→			// Allow some time for async operations
   563→			time.Sleep(100 * time.Millisecond)
   564→
   565→			if !tt.expectError && tt.validateJob != nil {
   566→				// Extract job details from payload
   567→				var payloadMap map[string]any
   568→				json.Unmarshal([]byte(tt.payload), &payloadMap)
   569→				jobID := int64(payloadMap["jobId"].(float64))
   570→				cluster := payloadMap["cluster"].(string)
   571→
   572→				var startTime *int64
   573→				if st, ok := payloadMap["startTime"]; ok {
   574→					t := int64(st.(float64))
   575→					startTime = &t
   576→				}
   577→
   578→				job, err := natsAPI.JobRepository.Find(&jobID, &cluster, startTime)
   579→				if err != nil {
   580→					t.Fatalf("expected to find job, but got error: %v", err)
   581→				}
   582→
   583→				tt.validateJob(t, job)
   584→			}
   585→		})
   586→	}
   587→}
   588→
   589→func TestNatsHandleNodeState(t *testing.T) {
   590→	natsAPI := setupNatsTest(t)
   591→	t.Cleanup(cleanupNatsTest)
   592→
   593→	tests := []struct {
   594→		name        string
   595→		data        []byte
   596→		expectError bool
   597→		validateFn  func(t *testing.T)
   598→	}{
   599→		{
   600→			name:        "valid node state update",
   601→			data:        []byte(`nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"host123\",\"states\":[\"allocated\"],\"cpusAllocated\":8,\"memoryAllocated\":16384,\"gpusAllocated\":0,\"jobsRunning\":1}]}" 1234567890000000000`),
   602→			expectError: false,
   603→			validateFn: func(t *testing.T) {
   604→				// In a full test, we would verify the node state was updated in the database
   605→				// For now, just ensure no error occurred
   606→			},
   607→		},
   608→		{
   609→			name:        "multiple nodes",
   610→			data:        []byte(`nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"host123\",\"states\":[\"idle\"],\"cpusAllocated\":0,\"memoryAllocated\":0,\"gpusAllocated\":0,\"jobsRunning\":0},{\"hostname\":\"host124\",\"states\":[\"allocated\"],\"cpusAllocated\":4,\"memoryAllocated\":8192,\"gpusAllocated\":1,\"jobsRunning\":1}]}" 1234567890000000000`),
   611→			expectError: false,
   612→		},
   613→		{
   614→			name:        "invalid JSON in event field",
   615→			data:        []byte(`nodestate event="{\"cluster\":\"testcluster\",\"nodes\":\"not an array\"}" 1234567890000000000`),
   616→			expectError: true,
   617→		},
   618→		{
   619→			name:        "empty nodes array",
   620→			data:        []byte(`nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[]}" 1234567890000000000`),
   621→			expectError: false, // Empty array should not cause error
   622→		},
   623→		{
   624→			name:        "invalid line protocol format",
   625→			data:        []byte(`invalid line protocol format`),
   626→			expectError: true,
   627→		},
   628→		{
   629→			name:        "empty data",
   630→			data:        []byte(``),
   631→			expectError: false, // Should be handled gracefully with warning
   632→		},
   633→	}
   634→
   635→	for _, tt := range tests {
   636→		t.Run(tt.name, func(t *testing.T) {
   637→			natsAPI.handleNodeState("test.subject", tt.data)
   638→
   639→			// Allow some time for async operations
   640→			time.Sleep(50 * time.Millisecond)
   641→
   642→			if tt.validateFn != nil {
   643→				tt.validateFn(t)
   644→			}
   645→		})
   646→	}
   647→}
   648→
   649→func TestNatsProcessJobEvent(t *testing.T) {
   650→	natsAPI := setupNatsTest(t)
   651→	t.Cleanup(cleanupNatsTest)
   652→
   653→	msgStartJob, err := lp.NewMessage(
   654→		"job",
   655→		map[string]string{"function": "start_job"},
   656→		nil,
   657→		map[string]any{
   658→			"event": `{
   659→				"jobId": 3001,
   660→				"user": "testuser",
   661→				"project": "testproj",
   662→				"cluster": "testcluster",
   663→				"partition": "main",
   664→				"walltime": 3600,
   665→				"numNodes": 1,
   666→				"numHwthreads": 8,
   667→				"numAcc": 0,
   668→				"shared": "none",
   669→				"monitoringStatus": 1,
   670→				"smt": 1,
   671→				"resources": [
   672→					{
   673→						"hostname": "host123",
   674→						"hwthreads": [0, 1, 2, 3]
   675→					}
   676→				],
   677→				"startTime": 1234567890
   678→			}`,
   679→		},
   680→		time.Now(),
   681→	)
   682→	if err != nil {
   683→		t.Fatalf("failed to create test message: %v", err)
   684→	}
   685→
   686→	msgMissingTag, err := lp.NewMessage(
   687→		"job",
   688→		map[string]string{},
   689→		nil,
   690→		map[string]any{
   691→			"event": `{}`,
   692→		},
   693→		time.Now(),
   694→	)
   695→	if err != nil {
   696→		t.Fatalf("failed to create test message: %v", err)
   697→	}
   698→
   699→	msgUnknownFunc, err := lp.NewMessage(
   700→		"job",
   701→		map[string]string{"function": "unknown_function"},
   702→		nil,
   703→		map[string]any{
   704→			"event": `{}`,
   705→		},
   706→		time.Now(),
   707→	)
   708→	if err != nil {
   709→		t.Fatalf("failed to create test message: %v", err)
   710→	}
   711→
   712→	tests := []struct {
   713→		name        string
   714→		message     lp.CCMessage
   715→		expectError bool
   716→	}{
   717→		{
   718→			name:        "start_job function",
   719→			message:     msgStartJob,
   720→			expectError: false,
   721→		},
   722→		{
   723→			name:        "missing function tag",
   724→			message:     msgMissingTag,
   725→			expectError: true,
   726→		},
   727→		{
   728→			name:        "unknown function",
   729→			message:     msgUnknownFunc,
   730→			expectError: false,
   731→		},
   732→	}
   733→
   734→	for _, tt := range tests {
   735→		t.Run(tt.name, func(t *testing.T) {
   736→			natsAPI.processJobEvent(tt.message)
   737→			time.Sleep(50 * time.Millisecond)
   738→		})
   739→	}
   740→}
   741→
   742→func TestNatsHandleJobEvent(t *testing.T) {
   743→	natsAPI := setupNatsTest(t)
   744→	t.Cleanup(cleanupNatsTest)
   745→
   746→	tests := []struct {
   747→		name        string
   748→		data        []byte
   749→		expectError bool
   750→	}{
   751→		{
   752→			name:        "valid influx line protocol",
   753→			data:        []byte(`job,function=start_job event="{\"jobId\":4001,\"user\":\"testuser\",\"project\":\"testproj\",\"cluster\":\"testcluster\",\"partition\":\"main\",\"walltime\":3600,\"numNodes\":1,\"numHwthreads\":8,\"numAcc\":0,\"shared\":\"none\",\"monitoringStatus\":1,\"smt\":1,\"resources\":[{\"hostname\":\"host123\",\"hwthreads\":[0,1,2,3]}],\"startTime\":1234567890}" 1234567890000000000`),
   754→			expectError: false,
   755→		},
   756→		{
   757→			name:        "invalid influx line protocol",
   758→			data:        []byte(`invalid line protocol format`),
   759→			expectError: true,
   760→		},
   761→		{
   762→			name:        "empty data",
   763→			data:        []byte(``),
   764→			expectError: false, // Decoder should handle empty input gracefully
   765→		},
   766→	}
   767→
   768→	for _, tt := range tests {
   769→		t.Run(tt.name, func(t *testing.T) {
   770→			// HandleJobEvent doesn't return errors, it logs them
   771→			// We're just ensuring it doesn't panic
   772→			natsAPI.handleJobEvent("test.subject", tt.data)
   773→			time.Sleep(50 * time.Millisecond)
   774→		})
   775→	}
   776→}
   777→
   778→func TestNatsHandleJobEventEdgeCases(t *testing.T) {
   779→	natsAPI := setupNatsTest(t)
   780→	t.Cleanup(cleanupNatsTest)
   781→
   782→	tests := []struct {
   783→		name        string
   784→		data        []byte
   785→		expectError bool
   786→		description string
   787→	}{
   788→		{
   789→			name:        "non-event message (metric data)",
   790→			data:        []byte(`job,function=start_job value=123.45 1234567890000000000`),
   791→			expectError: false,
   792→			description: "Should skip non-event messages gracefully",
   793→		},
   794→		{
   795→			name:        "wrong measurement name",
   796→			data:        []byte(`wrongmeasurement,function=start_job event="{}" 1234567890000000000`),
   797→			expectError: false,
   798→			description: "Should warn about unexpected measurement but not fail",
   799→		},
   800→		{
   801→			name:        "missing event field",
   802→			data:        []byte(`job,function=start_job other_field="value" 1234567890000000000`),
   803→			expectError: true,
   804→			description: "Should error when event field is missing",
   805→		},
   806→		{
   807→			name:        "multiple measurements in one message",
   808→			data:        []byte("job,function=start_job event=\"{}\" 1234567890000000000\njob,function=stop_job event=\"{}\" 1234567890000000000"),
   809→			expectError: false,
   810→			description: "Should process multiple lines",
   811→		},
   812→		{
   813→			name:        "escaped quotes in JSON payload",
   814→			data:        []byte(`job,function=start_job event="{\"jobId\":6001,\"user\":\"test\\\"user\",\"cluster\":\"test\"}" 1234567890000000000`),
   815→			expectError: true,
   816→			description: "Should handle escaped quotes (though JSON parsing may fail)",
   817→		},
   818→	}
   819→
   820→	for _, tt := range tests {
   821→		t.Run(tt.name, func(t *testing.T) {
   822→			natsAPI.handleJobEvent("test.subject", tt.data)
   823→			time.Sleep(50 * time.Millisecond)
   824→		})
   825→	}
   826→}
   827→
   828→func TestNatsHandleNodeStateEdgeCases(t *testing.T) {
   829→	natsAPI := setupNatsTest(t)
   830→	t.Cleanup(cleanupNatsTest)
   831→
   832→	tests := []struct {
   833→		name        string
   834→		data        []byte
   835→		expectError bool
   836→		description string
   837→	}{
   838→		{
   839→			name:        "missing cluster field in JSON",
   840→			data:        []byte(`nodestate event="{\"nodes\":[]}" 1234567890000000000`),
   841→			expectError: true,
   842→			description: "Should fail when cluster is missing",
   843→		},
   844→		{
   845→			name:        "malformed JSON with unescaped quotes",
   846→			data:        []byte(`nodestate event="{\"cluster\":\"test"cluster\",\"nodes\":[]}" 1234567890000000000`),
   847→			expectError: true,
   848→			description: "Should fail on malformed JSON",
   849→		},
   850→		{
   851→			name:        "unicode characters in hostname",
   852→			data:        []byte(`nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"host-ñ123\",\"states\":[\"idle\"],\"cpusAllocated\":0,\"memoryAllocated\":0,\"gpusAllocated\":0,\"jobsRunning\":0}]}" 1234567890000000000`),
   853→			expectError: false,
   854→			description: "Should handle unicode characters",
   855→		},
   856→		{
   857→			name:        "very large node count",
   858→			data:        []byte(`nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"node1\",\"states\":[\"idle\"],\"cpusAllocated\":0,\"memoryAllocated\":0,\"gpusAllocated\":0,\"jobsRunning\":0},{\"hostname\":\"node2\",\"states\":[\"idle\"],\"cpusAllocated\":0,\"memoryAllocated\":0,\"gpusAllocated\":0,\"jobsRunning\":0},{\"hostname\":\"node3\",\"states\":[\"idle\"],\"cpusAllocated\":0,\"memoryAllocated\":0,\"gpusAllocated\":0,\"jobsRunning\":0}]}" 1234567890000000000`),
   859→			expectError: false,
   860→			description: "Should handle multiple nodes efficiently",
   861→		},
   862→		{
   863→			name:        "timestamp in past",
   864→			data:        []byte(`nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[]}" 1000000000000000000`),
   865→			expectError: false,
   866→			description: "Should accept any valid timestamp",
   867→		},
   868→	}
   869→
   870→	for _, tt := range tests {
   871→		t.Run(tt.name, func(t *testing.T) {
   872→			natsAPI.handleNodeState("test.subject", tt.data)
   873→			time.Sleep(50 * time.Millisecond)
   874→		})
   875→	}
   876→}
   877→
   878→func TestNatsHandleStartJobDuplicatePrevention(t *testing.T) {
   879→	natsAPI := setupNatsTest(t)
   880→	t.Cleanup(cleanupNatsTest)
   881→
   882→	// Start a job
   883→	payload := `{
   884→		"jobId": 5001,
   885→		"user": "testuser",
   886→		"project": "testproj",
   887→		"cluster": "testcluster",
   888→		"partition": "main",
   889→		"walltime": 3600,
   890→		"numNodes": 1,
   891→		"numHwthreads": 8,
   892→		"numAcc": 0,
   893→		"shared": "none",
   894→		"monitoringStatus": 1,
   895→		"smt": 1,
   896→		"resources": [
   897→			{
   898→				"hostname": "host123",
   899→				"hwthreads": [0, 1, 2, 3]
   900→			}
   901→		],
   902→		"startTime": 1234567890
   903→	}`
   904→
   905→	natsAPI.handleStartJob(payload)
   906→	natsAPI.JobRepository.SyncJobs()
   907→	time.Sleep(100 * time.Millisecond)
   908→
   909→	// Try to start the same job again (within 24 hours)
   910→	duplicatePayload := `{
   911→		"jobId": 5001,
   912→		"user": "testuser",
   913→		"project": "testproj",
   914→		"cluster": "testcluster",
   915→		"partition": "main",
   916→		"walltime": 3600,
   917→		"numNodes": 1,
   918→		"numHwthreads": 8,
   919→		"numAcc": 0,
   920→		"shared": "none",
   921→		"monitoringStatus": 1,
   922→		"smt": 1,
   923→		"resources": [
   924→			{
   925→				"hostname": "host123",
   926→				"hwthreads": [0, 1, 2, 3]
   927→			}
   928→		],
   929→		"startTime": 1234567900
   930→	}`
   931→
   932→	natsAPI.handleStartJob(duplicatePayload)
   933→	natsAPI.JobRepository.SyncJobs()
   934→	time.Sleep(100 * time.Millisecond)
   935→
   936→	// Verify only one job exists
   937→	jobID := int64(5001)
   938→	cluster := "testcluster"
   939→	jobs, err := natsAPI.JobRepository.FindAll(&jobID, &cluster, nil)
   940→	if err != nil && err != sql.ErrNoRows {
   941→		t.Fatalf("unexpected error: %v", err)
   942→	}
   943→
   944→	if len(jobs) != 1 {
   945→		t.Errorf("expected 1 job, got %d", len(jobs))
   946→	}
   947→}
   948→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package api

import (
	"context"
	"database/sql"
	"encoding/json"
	"fmt"
	"os"
	"path/filepath"
	"testing"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/archiver"
	"github.com/ClusterCockpit/cc-backend/internal/auth"
	"github.com/ClusterCockpit/cc-backend/internal/config"
	"github.com/ClusterCockpit/cc-backend/internal/graph"
	"github.com/ClusterCockpit/cc-backend/internal/repository"
	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	"github.com/ClusterCockpit/cc-backend/pkg/metricstore"
	ccconf "github.com/ClusterCockpit/cc-lib/v2/ccConfig"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	lp "github.com/ClusterCockpit/cc-lib/v2/ccMessage"
	"github.com/ClusterCockpit/cc-lib/v2/schema"

	_ "github.com/mattn/go-sqlite3"
)

func setupNatsTest(t *testing.T) *NatsAPI {
	repository.ResetConnection()

	const testconfig = `{
		"main": {
	"addr":            "0.0.0.0:8080",
	"validate": false,
  "api-allowed-ips": [
    "*"
  ]
	},
	"archive": {
		"kind": "file",
		"path": "./var/job-archive"
	},
	"auth": {
  "jwts": {
      "max-age": "2m"
  }
	}
}`
	const testclusterJSON = `{
        "name": "testcluster",
		"subClusters": [
			{
				"name": "sc1",
				"nodes": "host123,host124,host125",
				"processorType": "Intel Core i7-4770",
				"socketsPerNode": 1,
				"coresPerSocket": 4,
				"threadsPerCore": 2,
                "flopRateScalar": {
                  "unit": {
                    "prefix": "G",
                    "base": "F/s"
                  },
                  "value": 14
                },
                "flopRateSimd": {
                  "unit": {
                    "prefix": "G",
                    "base": "F/s"
                  },
                  "value": 112
                },
                "memoryBandwidth": {
                  "unit": {
                    "prefix": "G",
                    "base": "B/s"
                  },
                  "value": 24
                },
                "numberOfNodes": 70,
				"topology": {
					"node": [0, 1, 2, 3, 4, 5, 6, 7],
					"socket": [[0, 1, 2, 3, 4, 5, 6, 7]],
					"memoryDomain": [[0, 1, 2, 3, 4, 5, 6, 7]],
					"die": [[0, 1, 2, 3, 4, 5, 6, 7]],
					"core": [[0], [1], [2], [3], [4], [5], [6], [7]]
				}
			}
		],
		"metricConfig": [
			{
				"name": "load_one",
			    "unit": { "base": ""},
				"scope": "node",
				"timestep": 60,
                "aggregation": "avg",
				"peak": 8,
				"normal": 0,
				"caution": 0,
				"alert": 0
			}
		]
	}`

	cclog.Init("info", true)
	tmpdir := t.TempDir()
	jobarchive := filepath.Join(tmpdir, "job-archive")
	if err := os.Mkdir(jobarchive, 0o777); err != nil {
		t.Fatal(err)
	}

	if err := os.WriteFile(filepath.Join(jobarchive, "version.txt"), fmt.Appendf(nil, "%d", 3), 0o666); err != nil {
		t.Fatal(err)
	}

	if err := os.Mkdir(filepath.Join(jobarchive, "testcluster"), 0o777); err != nil {
		t.Fatal(err)
	}

	if err := os.WriteFile(filepath.Join(jobarchive, "testcluster", "cluster.json"), []byte(testclusterJSON), 0o666); err != nil {
		t.Fatal(err)
	}

	dbfilepath := filepath.Join(tmpdir, "test.db")
	err := repository.MigrateDB(dbfilepath)
	if err != nil {
		t.Fatal(err)
	}

	cfgFilePath := filepath.Join(tmpdir, "config.json")
	if err := os.WriteFile(cfgFilePath, []byte(testconfig), 0o666); err != nil {
		t.Fatal(err)
	}

	ccconf.Init(cfgFilePath)

	// Load and check main configuration
	if cfg := ccconf.GetPackageConfig("main"); cfg != nil {
		config.Init(cfg)
	} else {
		cclog.Abort("Main configuration must be present")
	}
	archiveCfg := fmt.Sprintf("{\"kind\": \"file\",\"path\": \"%s\"}", jobarchive)

	repository.Connect(dbfilepath)

	if err := archive.Init(json.RawMessage(archiveCfg)); err != nil {
		t.Fatal(err)
	}

	// metricstore initialization removed - it's initialized via callback in tests

	archiver.Start(repository.GetJobRepository(), context.Background())

	if cfg := ccconf.GetPackageConfig("auth"); cfg != nil {
		auth.Init(&cfg)
	} else {
		cclog.Warn("Authentication disabled due to missing configuration")
		auth.Init(nil)
	}

	graph.Init()

	return NewNatsAPI()
}

func cleanupNatsTest() {
	if err := archiver.Shutdown(5 * time.Second); err != nil {
		cclog.Warnf("Archiver shutdown timeout in tests: %v", err)
	}
}

func TestNatsHandleStartJob(t *testing.T) {
	natsAPI := setupNatsTest(t)
	t.Cleanup(cleanupNatsTest)

	tests := []struct {
		name          string
		payload       string
		expectError   bool
		validateJob   func(t *testing.T, job *schema.Job)
		shouldFindJob bool
	}{
		{
			name: "valid job start",
			payload: `{
				"jobId": 1001,
				"user": "testuser1",
				"project": "testproj1",
				"cluster": "testcluster",
				"partition": "main",
				"walltime": 7200,
				"numNodes": 1,
				"numHwthreads": 8,
				"numAcc": 0,
				"shared": "none",
				"monitoringStatus": 1,
				"smt": 1,
				"resources": [
					{
						"hostname": "host123",
						"hwthreads": [0, 1, 2, 3, 4, 5, 6, 7]
					}
				],
				"startTime": 1234567890
			}`,
			expectError:   false,
			shouldFindJob: true,
			validateJob: func(t *testing.T, job *schema.Job) {
				if job.JobID != 1001 {
					t.Errorf("expected JobID 1001, got %d", job.JobID)
				}
				if job.User != "testuser1" {
					t.Errorf("expected user testuser1, got %s", job.User)
				}
				if job.State != schema.JobStateRunning {
					t.Errorf("expected state running, got %s", job.State)
				}
			},
		},
		{
			name: "invalid JSON",
			payload: `{
				"jobId": "not a number",
				"user": "testuser2"
			}`,
			expectError:   true,
			shouldFindJob: false,
		},
		{
			name: "missing required fields",
			payload: `{
				"jobId": 1002
			}`,
			expectError:   true,
			shouldFindJob: false,
		},
		{
			name: "job with unknown fields (should fail due to DisallowUnknownFields)",
			payload: `{
				"jobId": 1003,
				"user": "testuser3",
				"project": "testproj3",
				"cluster": "testcluster",
				"partition": "main",
				"walltime": 3600,
				"numNodes": 1,
				"numHwthreads": 8,
				"unknownField": "should cause error",
				"startTime": 1234567900
			}`,
			expectError:   true,
			shouldFindJob: false,
		},
		{
			name: "job with tags",
			payload: `{
				"jobId": 1004,
				"user": "testuser4",
				"project": "testproj4",
				"cluster": "testcluster",
				"partition": "main",
				"walltime": 3600,
				"numNodes": 1,
				"numHwthreads": 8,
				"numAcc": 0,
				"shared": "none",
				"monitoringStatus": 1,
				"smt": 1,
				"resources": [
					{
						"hostname": "host123",
						"hwthreads": [0, 1, 2, 3]
					}
				],
				"tags": [
					{
						"type": "test",
						"name": "testtag",
						"scope": "testuser4"
					}
				],
				"startTime": 1234567910
			}`,
			expectError:   false,
			shouldFindJob: true,
			validateJob: func(t *testing.T, job *schema.Job) {
				if job.JobID != 1004 {
					t.Errorf("expected JobID 1004, got %d", job.JobID)
				}
			},
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			natsAPI.handleStartJob(tt.payload)
			natsAPI.JobRepository.SyncJobs()

			// Allow some time for async operations
			time.Sleep(100 * time.Millisecond)

			if tt.shouldFindJob {
				// Extract jobId from payload
				var payloadMap map[string]any
				json.Unmarshal([]byte(tt.payload), &payloadMap)
				jobID := int64(payloadMap["jobId"].(float64))
				cluster := payloadMap["cluster"].(string)
				startTime := int64(payloadMap["startTime"].(float64))

				job, err := natsAPI.JobRepository.Find(&jobID, &cluster, &startTime)
				if err != nil {
					if !tt.expectError {
						t.Fatalf("expected to find job, but got error: %v", err)
					}
					return
				}

				if tt.validateJob != nil {
					tt.validateJob(t, job)
				}
			}
		})
	}
}

func TestNatsHandleStopJob(t *testing.T) {
	natsAPI := setupNatsTest(t)
	t.Cleanup(cleanupNatsTest)

	// First, create a running job
	startPayload := `{
		"jobId": 2001,
		"user": "testuser",
		"project": "testproj",
		"cluster": "testcluster",
		"partition": "main",
		"walltime": 3600,
		"numNodes": 1,
		"numHwthreads": 8,
		"numAcc": 0,
		"shared": "none",
		"monitoringStatus": 1,
		"smt": 1,
		"resources": [
			{
				"hostname": "host123",
				"hwthreads": [0, 1, 2, 3, 4, 5, 6, 7]
			}
		],
		"startTime": 1234567890
	}`

	natsAPI.handleStartJob(startPayload)
	natsAPI.JobRepository.SyncJobs()
	time.Sleep(100 * time.Millisecond)

	tests := []struct {
		name         string
		payload      string
		expectError  bool
		validateJob  func(t *testing.T, job *schema.Job)
		setupJobFunc func() // Optional: create specific test job
	}{
		{
			name: "valid job stop - completed",
			payload: `{
				"jobId": 2001,
				"cluster": "testcluster",
				"startTime": 1234567890,
				"jobState": "completed",
				"stopTime": 1234571490
			}`,
			expectError: false,
			validateJob: func(t *testing.T, job *schema.Job) {
				if job.State != schema.JobStateCompleted {
					t.Errorf("expected state completed, got %s", job.State)
				}
				expectedDuration := int32(1234571490 - 1234567890)
				if job.Duration != expectedDuration {
					t.Errorf("expected duration %d, got %d", expectedDuration, job.Duration)
				}
			},
		},
		{
			name: "valid job stop - failed",
			setupJobFunc: func() {
				startPayloadFailed := `{
					"jobId": 2002,
					"user": "testuser",
					"project": "testproj",
					"cluster": "testcluster",
					"partition": "main",
					"walltime": 3600,
					"numNodes": 1,
					"numHwthreads": 8,
					"numAcc": 0,
					"shared": "none",
					"monitoringStatus": 1,
					"smt": 1,
					"resources": [
						{
							"hostname": "host123",
							"hwthreads": [0, 1, 2, 3]
						}
					],
					"startTime": 1234567900
				}`
				natsAPI.handleStartJob(startPayloadFailed)
				natsAPI.JobRepository.SyncJobs()
				time.Sleep(100 * time.Millisecond)
			},
			payload: `{
				"jobId": 2002,
				"cluster": "testcluster",
				"startTime": 1234567900,
				"jobState": "failed",
				"stopTime": 1234569900
			}`,
			expectError: false,
			validateJob: func(t *testing.T, job *schema.Job) {
				if job.State != schema.JobStateFailed {
					t.Errorf("expected state failed, got %s", job.State)
				}
			},
		},
		{
			name: "invalid JSON",
			payload: `{
				"jobId": "not a number"
			}`,
			expectError: true,
		},
		{
			name: "missing jobId",
			payload: `{
				"cluster": "testcluster",
				"jobState": "completed",
				"stopTime": 1234571490
			}`,
			expectError: true,
		},
		{
			name: "invalid job state",
			setupJobFunc: func() {
				startPayloadInvalid := `{
					"jobId": 2003,
					"user": "testuser",
					"project": "testproj",
					"cluster": "testcluster",
					"partition": "main",
					"walltime": 3600,
					"numNodes": 1,
					"numHwthreads": 8,
					"numAcc": 0,
					"shared": "none",
					"monitoringStatus": 1,
					"smt": 1,
					"resources": [
						{
							"hostname": "host123",
							"hwthreads": [0, 1]
						}
					],
					"startTime": 1234567910
				}`
				natsAPI.handleStartJob(startPayloadInvalid)
				natsAPI.JobRepository.SyncJobs()
				time.Sleep(100 * time.Millisecond)
			},
			payload: `{
				"jobId": 2003,
				"cluster": "testcluster",
				"startTime": 1234567910,
				"jobState": "invalid_state",
				"stopTime": 1234571510
			}`,
			expectError: true,
		},
		{
			name: "stopTime before startTime",
			setupJobFunc: func() {
				startPayloadTime := `{
					"jobId": 2004,
					"user": "testuser",
					"project": "testproj",
					"cluster": "testcluster",
					"partition": "main",
					"walltime": 3600,
					"numNodes": 1,
					"numHwthreads": 8,
					"numAcc": 0,
					"shared": "none",
					"monitoringStatus": 1,
					"smt": 1,
					"resources": [
						{
							"hostname": "host123",
							"hwthreads": [0]
						}
					],
					"startTime": 1234567920
				}`
				natsAPI.handleStartJob(startPayloadTime)
				natsAPI.JobRepository.SyncJobs()
				time.Sleep(100 * time.Millisecond)
			},
			payload: `{
				"jobId": 2004,
				"cluster": "testcluster",
				"startTime": 1234567920,
				"jobState": "completed",
				"stopTime": 1234567900
			}`,
			expectError: true,
		},
		{
			name: "job not found",
			payload: `{
				"jobId": 99999,
				"cluster": "testcluster",
				"startTime": 1234567890,
				"jobState": "completed",
				"stopTime": 1234571490
			}`,
			expectError: true,
		},
	}

	testData := schema.JobData{
		"load_one": map[schema.MetricScope]*schema.JobMetric{
			schema.MetricScopeNode: {
				Unit:     schema.Unit{Base: "load"},
				Timestep: 60,
				Series: []schema.Series{
					{
						Hostname:   "host123",
						Statistics: schema.MetricStatistics{Min: 0.1, Avg: 0.2, Max: 0.3},
						Data:       []schema.Float{0.1, 0.1, 0.1, 0.2, 0.2, 0.2, 0.3, 0.3, 0.3},
					},
				},
			},
		},
	}

	metricstore.TestLoadDataCallback = func(job *schema.Job, metrics []string, scopes []schema.MetricScope, ctx context.Context, resolution int) (schema.JobData, error) {
		return testData, nil
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			if tt.setupJobFunc != nil {
				tt.setupJobFunc()
			}

			natsAPI.handleStopJob(tt.payload)

			// Allow some time for async operations
			time.Sleep(100 * time.Millisecond)

			if !tt.expectError && tt.validateJob != nil {
				// Extract job details from payload
				var payloadMap map[string]any
				json.Unmarshal([]byte(tt.payload), &payloadMap)
				jobID := int64(payloadMap["jobId"].(float64))
				cluster := payloadMap["cluster"].(string)

				var startTime *int64
				if st, ok := payloadMap["startTime"]; ok {
					t := int64(st.(float64))
					startTime = &t
				}

				job, err := natsAPI.JobRepository.Find(&jobID, &cluster, startTime)
				if err != nil {
					t.Fatalf("expected to find job, but got error: %v", err)
				}

				tt.validateJob(t, job)
			}
		})
	}
}

func TestNatsHandleNodeState(t *testing.T) {
	natsAPI := setupNatsTest(t)
	t.Cleanup(cleanupNatsTest)

	tests := []struct {
		name        string
		data        []byte
		expectError bool
		validateFn  func(t *testing.T)
	}{
		{
			name:        "valid node state update",
			data:        []byte(`nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"host123\",\"states\":[\"allocated\"],\"cpusAllocated\":8,\"memoryAllocated\":16384,\"gpusAllocated\":0,\"jobsRunning\":1}]}" 1234567890000000000`),
			expectError: false,
			validateFn: func(t *testing.T) {
				// In a full test, we would verify the node state was updated in the database
				// For now, just ensure no error occurred
			},
		},
		{
			name:        "multiple nodes",
			data:        []byte(`nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"host123\",\"states\":[\"idle\"],\"cpusAllocated\":0,\"memoryAllocated\":0,\"gpusAllocated\":0,\"jobsRunning\":0},{\"hostname\":\"host124\",\"states\":[\"allocated\"],\"cpusAllocated\":4,\"memoryAllocated\":8192,\"gpusAllocated\":1,\"jobsRunning\":1}]}" 1234567890000000000`),
			expectError: false,
		},
		{
			name:        "invalid JSON in event field",
			data:        []byte(`nodestate event="{\"cluster\":\"testcluster\",\"nodes\":\"not an array\"}" 1234567890000000000`),
			expectError: true,
		},
		{
			name:        "empty nodes array",
			data:        []byte(`nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[]}" 1234567890000000000`),
			expectError: false, // Empty array should not cause error
		},
		{
			name:        "invalid line protocol format",
			data:        []byte(`invalid line protocol format`),
			expectError: true,
		},
		{
			name:        "empty data",
			data:        []byte(``),
			expectError: false, // Should be handled gracefully with warning
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			natsAPI.handleNodeState("test.subject", tt.data)

			// Allow some time for async operations
			time.Sleep(50 * time.Millisecond)

			if tt.validateFn != nil {
				tt.validateFn(t)
			}
		})
	}
}

func TestNatsProcessJobEvent(t *testing.T) {
	natsAPI := setupNatsTest(t)
	t.Cleanup(cleanupNatsTest)

	msgStartJob, err := lp.NewMessage(
		"job",
		map[string]string{"function": "start_job"},
		nil,
		map[string]any{
			"event": `{
				"jobId": 3001,
				"user": "testuser",
				"project": "testproj",
				"cluster": "testcluster",
				"partition": "main",
				"walltime": 3600,
				"numNodes": 1,
				"numHwthreads": 8,
				"numAcc": 0,
				"shared": "none",
				"monitoringStatus": 1,
				"smt": 1,
				"resources": [
					{
						"hostname": "host123",
						"hwthreads": [0, 1, 2, 3]
					}
				],
				"startTime": 1234567890
			}`,
		},
		time.Now(),
	)
	if err != nil {
		t.Fatalf("failed to create test message: %v", err)
	}

	msgMissingTag, err := lp.NewMessage(
		"job",
		map[string]string{},
		nil,
		map[string]any{
			"event": `{}`,
		},
		time.Now(),
	)
	if err != nil {
		t.Fatalf("failed to create test message: %v", err)
	}

	msgUnknownFunc, err := lp.NewMessage(
		"job",
		map[string]string{"function": "unknown_function"},
		nil,
		map[string]any{
			"event": `{}`,
		},
		time.Now(),
	)
	if err != nil {
		t.Fatalf("failed to create test message: %v", err)
	}

	tests := []struct {
		name        string
		message     lp.CCMessage
		expectError bool
	}{
		{
			name:        "start_job function",
			message:     msgStartJob,
			expectError: false,
		},
		{
			name:        "missing function tag",
			message:     msgMissingTag,
			expectError: true,
		},
		{
			name:        "unknown function",
			message:     msgUnknownFunc,
			expectError: false,
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			natsAPI.processJobEvent(tt.message)
			time.Sleep(50 * time.Millisecond)
		})
	}
}

func TestNatsHandleJobEvent(t *testing.T) {
	natsAPI := setupNatsTest(t)
	t.Cleanup(cleanupNatsTest)

	tests := []struct {
		name        string
		data        []byte
		expectError bool
	}{
		{
			name:        "valid influx line protocol",
			data:        []byte(`job,function=start_job event="{\"jobId\":4001,\"user\":\"testuser\",\"project\":\"testproj\",\"cluster\":\"testcluster\",\"partition\":\"main\",\"walltime\":3600,\"numNodes\":1,\"numHwthreads\":8,\"numAcc\":0,\"shared\":\"none\",\"monitoringStatus\":1,\"smt\":1,\"resources\":[{\"hostname\":\"host123\",\"hwthreads\":[0,1,2,3]}],\"startTime\":1234567890}" 1234567890000000000`),
			expectError: false,
		},
		{
			name:        "invalid influx line protocol",
			data:        []byte(`invalid line protocol format`),
			expectError: true,
		},
		{
			name:        "empty data",
			data:        []byte(``),
			expectError: false, // Decoder should handle empty input gracefully
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			// HandleJobEvent doesn't return errors, it logs them
			// We're just ensuring it doesn't panic
			natsAPI.handleJobEvent("test.subject", tt.data)
			time.Sleep(50 * time.Millisecond)
		})
	}
}

func TestNatsHandleJobEventEdgeCases(t *testing.T) {
	natsAPI := setupNatsTest(t)
	t.Cleanup(cleanupNatsTest)

	tests := []struct {
		name        string
		data        []byte
		expectError bool
		description string
	}{
		{
			name:        "non-event message (metric data)",
			data:        []byte(`job,function=start_job value=123.45 1234567890000000000`),
			expectError: false,
			description: "Should skip non-event messages gracefully",
		},
		{
			name:        "wrong measurement name",
			data:        []byte(`wrongmeasurement,function=start_job event="{}" 1234567890000000000`),
			expectError: false,
			description: "Should warn about unexpected measurement but not fail",
		},
		{
			name:        "missing event field",
			data:        []byte(`job,function=start_job other_field="value" 1234567890000000000`),
			expectError: true,
			description: "Should error when event field is missing",
		},
		{
			name:        "multiple measurements in one message",
			data:        []byte("job,function=start_job event=\"{}\" 1234567890000000000\njob,function=stop_job event=\"{}\" 1234567890000000000"),
			expectError: false,
			description: "Should process multiple lines",
		},
		{
			name:        "escaped quotes in JSON payload",
			data:        []byte(`job,function=start_job event="{\"jobId\":6001,\"user\":\"test\\\"user\",\"cluster\":\"test\"}" 1234567890000000000`),
			expectError: true,
			description: "Should handle escaped quotes (though JSON parsing may fail)",
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			natsAPI.handleJobEvent("test.subject", tt.data)
			time.Sleep(50 * time.Millisecond)
		})
	}
}

func TestNatsHandleNodeStateEdgeCases(t *testing.T) {
	natsAPI := setupNatsTest(t)
	t.Cleanup(cleanupNatsTest)

	tests := []struct {
		name        string
		data        []byte
		expectError bool
		description string
	}{
		{
			name:        "missing cluster field in JSON",
			data:        []byte(`nodestate event="{\"nodes\":[]}" 1234567890000000000`),
			expectError: true,
			description: "Should fail when cluster is missing",
		},
		{
			name:        "malformed JSON with unescaped quotes",
			data:        []byte(`nodestate event="{\"cluster\":\"test"cluster\",\"nodes\":[]}" 1234567890000000000`),
			expectError: true,
			description: "Should fail on malformed JSON",
		},
		{
			name:        "unicode characters in hostname",
			data:        []byte(`nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"host-ñ123\",\"states\":[\"idle\"],\"cpusAllocated\":0,\"memoryAllocated\":0,\"gpusAllocated\":0,\"jobsRunning\":0}]}" 1234567890000000000`),
			expectError: false,
			description: "Should handle unicode characters",
		},
		{
			name:        "very large node count",
			data:        []byte(`nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"node1\",\"states\":[\"idle\"],\"cpusAllocated\":0,\"memoryAllocated\":0,\"gpusAllocated\":0,\"jobsRunning\":0},{\"hostname\":\"node2\",\"states\":[\"idle\"],\"cpusAllocated\":0,\"memoryAllocated\":0,\"gpusAllocated\":0,\"jobsRunning\":0},{\"hostname\":\"node3\",\"states\":[\"idle\"],\"cpusAllocated\":0,\"memoryAllocated\":0,\"gpusAllocated\":0,\"jobsRunning\":0}]}" 1234567890000000000`),
			expectError: false,
			description: "Should handle multiple nodes efficiently",
		},
		{
			name:        "timestamp in past",
			data:        []byte(`nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[]}" 1000000000000000000`),
			expectError: false,
			description: "Should accept any valid timestamp",
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			natsAPI.handleNodeState("test.subject", tt.data)
			time.Sleep(50 * time.Millisecond)
		})
	}
}

func TestNatsHandleStartJobDuplicatePrevention(t *testing.T) {
	natsAPI := setupNatsTest(t)
	t.Cleanup(cleanupNatsTest)

	// Start a job
	payload := `{
		"jobId": 5001,
		"user": "testuser",
		"project": "testproj",
		"cluster": "testcluster",
		"partition": "main",
		"walltime": 3600,
		"numNodes": 1,
		"numHwthreads": 8,
		"numAcc": 0,
		"shared": "none",
		"monitoringStatus": 1,
		"smt": 1,
		"resources": [
			{
				"hostname": "host123",
				"hwthreads": [0, 1, 2, 3]
			}
		],
		"startTime": 1234567890
	}`

	natsAPI.handleStartJob(payload)
	natsAPI.JobRepository.SyncJobs()
	time.Sleep(100 * time.Millisecond)

	// Try to start the same job again (within 24 hours)
	duplicatePayload := `{
		"jobId": 5001,
		"user": "testuser",
		"project": "testproj",
		"cluster": "testcluster",
		"partition": "main",
		"walltime": 3600,
		"numNodes": 1,
		"numHwthreads": 8,
		"numAcc": 0,
		"shared": "none",
		"monitoringStatus": 1,
		"smt": 1,
		"resources": [
			{
				"hostname": "host123",
				"hwthreads": [0, 1, 2, 3]
			}
		],
		"startTime": 1234567900
	}`

	natsAPI.handleStartJob(duplicatePayload)
	natsAPI.JobRepository.SyncJobs()
	time.Sleep(100 * time.Millisecond)

	// Verify only one job exists
	jobID := int64(5001)
	cluster := "testcluster"
	jobs, err := natsAPI.JobRepository.FindAll(&jobID, &cluster, nil)
	if err != nil && err != sql.ErrNoRows {
		t.Fatalf("unexpected error: %v", err)
	}

	if len(jobs) != 1 {
		t.Errorf("expected 1 job, got %d", len(jobs))
	}
}


Let me check the NATS client interface to understand available methods.