Implement the following plan:

# Plan: Add Health Check to NATS Node State Handler

## Context

The NATS `processNodestateEvent` handler in `internal/api/nats.go` is missing the health check logic that the REST `updateNodeStates` handler in `internal/api/node.go` has. Currently, the NATS handler hardcodes `HealthState: schema.MonitoringStateFull` and leaves `HealthMetrics` empty, while the REST handler:

1. Groups nodes by subcluster
2. Fetches metric configurations per subcluster
3. Runs health checks against the metric store
4. Uses health check results per node (defaulting to `MonitoringStateFailed` if unavailable)

This means node state updates via NATS always report full health regardless of actual metric availability, producing inconsistent data compared to the REST path.

## Changes

### File: `internal/api/nats.go`

**1. Add missing imports:**
- `"maps"`
- `"github.com/ClusterCockpit/cc-backend/internal/metricdispatch"`
- `"github.com/ClusterCockpit/cc-backend/pkg/archive"`
- `"github.com/ClusterCockpit/cc-backend/pkg/metricstore"`

**2. Rewrite `processNodestateEvent` (lines 382-425)** to match the REST handler logic:

- After parsing the request, add the same health check flow from `updateNodeStates` (node.go lines 82-114):
  - Build `map[subcluster][]hostname` using `archive.GetSubClusterByNode()`
  - Build `metricNames` per subcluster using `archive.GetMetricConfigSubCluster()` + `metricListToNames()`
  - Get health check repo via `metricdispatch.GetHealthCheckRepo(req.Cluster)`
  - Run `healthRepo.HealthCheck()` per subcluster, merge results
- When building `NodeStateDB`, default `HealthState` to `MonitoringStateFailed` (not `MonitoringStateFull`) and use health check results when available — identical to REST handler (node.go lines 120-143)

### Reused existing functions (no new code needed):
- `determineState()` — already shared (`internal/api/node.go:38`)
- `metricListToNames()` — already shared (`internal/api/node.go:29`)
- `archive.GetSubClusterByNode()` — from `pkg/archive`
- `archive.GetMetricConfigSubCluster()` — from `pkg/archive`
- `metricdispatch.GetHealthCheckRepo()` — from `internal/metricdispatch`
- `repo.BatchUpdateNodeStates()` — already used

## Verification

1. `go build ./...` — ensure it compiles
2. `go test ./internal/api/...` — run existing NATS tests
3. Review that `TestNatsHandleNodeState` and `TestNatsHandleNodeStateEdgeCases` still pass (they don't assert on HealthState values, so should be fine)
4. Manual comparison: the node-building loop in NATS should now be identical to REST (node.go lines 119-143)

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-ClusterCockpit-cc-backend/aac4ddda-7b73-4e7a-ab7b-75bbf24e26b3.jsonl


Let me start by reading the relevant files to understand the current code.


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package api
     7→
     8→import (
     9→	"database/sql"
    10→	"encoding/json"
    11→	"strings"
    12→	"sync"
    13→	"time"
    14→
    15→	"github.com/ClusterCockpit/cc-backend/internal/archiver"
    16→	"github.com/ClusterCockpit/cc-backend/internal/config"
    17→	"github.com/ClusterCockpit/cc-backend/internal/importer"
    18→	"github.com/ClusterCockpit/cc-backend/internal/repository"
    19→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    20→	lp "github.com/ClusterCockpit/cc-lib/v2/ccMessage"
    21→	"github.com/ClusterCockpit/cc-lib/v2/nats"
    22→	"github.com/ClusterCockpit/cc-lib/v2/receivers"
    23→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    24→	influx "github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
    25→)
    26→
    27→// natsMessage wraps a raw NATS message with its subject for channel-based processing.
    28→type natsMessage struct {
    29→	subject string
    30→	data    []byte
    31→}
    32→
    33→// NatsAPI provides NATS subscription-based handlers for Job and Node operations.
    34→// It mirrors the functionality of the REST API but uses NATS messaging with
    35→// InfluxDB line protocol as the message format.
    36→//
    37→// # Message Format
    38→//
    39→// All NATS messages use InfluxDB line protocol format (https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/)
    40→// with the following structure:
    41→//
    42→//	measurement,tag1=value1,tag2=value2 field1=value1,field2=value2 timestamp
    43→//
    44→// # Job Events
    45→//
    46→// Job start/stop events use the "job" measurement with a "function" tag to distinguish operations:
    47→//
    48→//	job,function=start_job event="{...JSON payload...}" <timestamp>
    49→//	job,function=stop_job event="{...JSON payload...}" <timestamp>
    50→//
    51→// The JSON payload in the "event" field follows the schema.Job or StopJobAPIRequest structure.
    52→//
    53→// Example job start message:
    54→//
    55→//	job,function=start_job event="{\"jobId\":1001,\"user\":\"testuser\",\"cluster\":\"testcluster\",...}" 1234567890000000000
    56→//
    57→// # Node State Events
    58→//
    59→// Node state updates use the "nodestate" measurement with cluster information:
    60→//
    61→//	nodestate event="{...JSON payload...}" <timestamp>
    62→//
    63→// The JSON payload follows the UpdateNodeStatesRequest structure.
    64→//
    65→// Example node state message:
    66→//
    67→//	nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"node01\",\"states\":[\"idle\"]}]}" 1234567890000000000
    68→type NatsAPI struct {
    69→	JobRepository *repository.JobRepository
    70→	// RepositoryMutex protects job creation operations from race conditions
    71→	// when checking for duplicate jobs during startJob calls.
    72→	RepositoryMutex sync.Mutex
    73→	// jobCh receives job event messages for processing by worker goroutines.
    74→	jobCh chan natsMessage
    75→	// nodeCh receives node state messages for processing by worker goroutines.
    76→	nodeCh chan natsMessage
    77→}
    78→
    79→// NewNatsAPI creates a new NatsAPI instance with channel-based worker pools.
    80→// Concurrency is configured via NATSConfig (defaults: JobConcurrency=8, NodeConcurrency=2).
    81→func NewNatsAPI() *NatsAPI {
    82→	jobConc := 8
    83→	nodeConc := 2
    84→
    85→	if s := config.Keys.APISubjects; s != nil {
    86→		if s.JobConcurrency > 0 {
    87→			jobConc = s.JobConcurrency
    88→		}
    89→		if s.NodeConcurrency > 0 {
    90→			nodeConc = s.NodeConcurrency
    91→		}
    92→	}
    93→
    94→	api := &NatsAPI{
    95→		JobRepository: repository.GetJobRepository(),
    96→		jobCh:         make(chan natsMessage, jobConc),
    97→		nodeCh:        make(chan natsMessage, nodeConc),
    98→	}
    99→
   100→	// Start worker goroutines
   101→	for range jobConc {
   102→		go api.jobWorker()
   103→	}
   104→	for range nodeConc {
   105→		go api.nodeWorker()
   106→	}
   107→
   108→	return api
   109→}
   110→
   111→// jobWorker processes job event messages from the job channel.
   112→func (api *NatsAPI) jobWorker() {
   113→	for msg := range api.jobCh {
   114→		api.handleJobEvent(msg.subject, msg.data)
   115→	}
   116→}
   117→
   118→// nodeWorker processes node state messages from the node channel.
   119→func (api *NatsAPI) nodeWorker() {
   120→	for msg := range api.nodeCh {
   121→		api.handleNodeState(msg.subject, msg.data)
   122→	}
   123→}
   124→
   125→// StartSubscriptions registers all NATS subscriptions for Job and Node APIs.
   126→// Messages are delivered to buffered channels and processed by worker goroutines.
   127→// Returns an error if the NATS client is not available or subscription fails.
   128→func (api *NatsAPI) StartSubscriptions() error {
   129→	client := nats.GetClient()
   130→	if client == nil {
   131→		cclog.Warn("NATS client not available, skipping API subscriptions")
   132→		return nil
   133→	}
   134→
   135→	if config.Keys.APISubjects != nil {
   136→		s := config.Keys.APISubjects
   137→
   138→		if err := client.Subscribe(s.SubjectJobEvent, func(subject string, data []byte) {
   139→			api.jobCh <- natsMessage{subject: subject, data: data}
   140→		}); err != nil {
   141→			return err
   142→		}
   143→
   144→		if err := client.Subscribe(s.SubjectNodeState, func(subject string, data []byte) {
   145→			api.nodeCh <- natsMessage{subject: subject, data: data}
   146→		}); err != nil {
   147→			return err
   148→		}
   149→
   150→		cclog.Info("NATS API subscriptions started")
   151→	}
   152→	return nil
   153→}
   154→
   155→// processJobEvent routes job event messages to the appropriate handler based on the "function" tag.
   156→// Validates that required tags and fields are present before processing.
   157→func (api *NatsAPI) processJobEvent(msg lp.CCMessage) {
   158→	function, ok := msg.GetTag("function")
   159→	if !ok {
   160→		cclog.Errorf("Job event is missing required tag 'function': measurement=%s", msg.Name())
   161→		return
   162→	}
   163→
   164→	switch function {
   165→	case "start_job":
   166→		v, ok := msg.GetEventValue()
   167→		if !ok {
   168→			cclog.Errorf("Job start event is missing event field with JSON payload")
   169→			return
   170→		}
   171→		api.handleStartJob(v)
   172→
   173→	case "stop_job":
   174→		v, ok := msg.GetEventValue()
   175→		if !ok {
   176→			cclog.Errorf("Job stop event is missing event field with JSON payload")
   177→			return
   178→		}
   179→		api.handleStopJob(v)
   180→
   181→	default:
   182→		cclog.Warnf("Unknown job event function '%s', expected 'start_job' or 'stop_job'", function)
   183→	}
   184→}
   185→
   186→// handleJobEvent processes job-related messages received via NATS using InfluxDB line protocol.
   187→// The message must be in line protocol format with measurement="job" and include:
   188→//   - tag "function" with value "start_job" or "stop_job"
   189→//   - field "event" containing JSON payload (schema.Job or StopJobAPIRequest)
   190→//
   191→// Example: job,function=start_job event="{\"jobId\":1001,...}" 1234567890000000000
   192→func (api *NatsAPI) handleJobEvent(subject string, data []byte) {
   193→	if len(data) == 0 {
   194→		cclog.Warnf("NATS %s: received empty message", subject)
   195→		return
   196→	}
   197→
   198→	d := influx.NewDecoderWithBytes(data)
   199→
   200→	for d.Next() {
   201→		m, err := receivers.DecodeInfluxMessage(d)
   202→		if err != nil {
   203→			cclog.Errorf("NATS %s: failed to decode InfluxDB line protocol message: %v", subject, err)
   204→			return
   205→		}
   206→
   207→		if !m.IsEvent() {
   208→			cclog.Debugf("NATS %s: received non-event message, skipping", subject)
   209→			continue
   210→		}
   211→
   212→		if m.Name() == "job" {
   213→			api.processJobEvent(m)
   214→		} else {
   215→			cclog.Debugf("NATS %s: unexpected measurement name '%s', expected 'job'", subject, m.Name())
   216→		}
   217→	}
   218→}
   219→
   220→// handleStartJob processes job start messages received via NATS.
   221→// The payload parameter contains JSON following the schema.Job structure.
   222→// Jobs are validated, checked for duplicates, and inserted into the database.
   223→func (api *NatsAPI) handleStartJob(payload string) {
   224→	if payload == "" {
   225→		cclog.Error("NATS start job: payload is empty")
   226→		return
   227→	}
   228→	req := schema.Job{
   229→		Shared:           "none",
   230→		MonitoringStatus: schema.MonitoringStatusRunningOrArchiving,
   231→	}
   232→
   233→	dec := json.NewDecoder(strings.NewReader(payload))
   234→	dec.DisallowUnknownFields()
   235→	if err := dec.Decode(&req); err != nil {
   236→		cclog.Errorf("NATS start job: parsing request failed: %v", err)
   237→		return
   238→	}
   239→
   240→	cclog.Debugf("NATS start job: %s", req.GoString())
   241→	req.State = schema.JobStateRunning
   242→
   243→	if err := importer.SanityChecks(&req); err != nil {
   244→		cclog.Errorf("NATS start job: sanity check failed: %v", err)
   245→		return
   246→	}
   247→
   248→	var unlockOnce sync.Once
   249→	api.RepositoryMutex.Lock()
   250→	defer unlockOnce.Do(api.RepositoryMutex.Unlock)
   251→
   252→	jobs, err := api.JobRepository.FindAll(&req.JobID, &req.Cluster, nil)
   253→	if err != nil && err != sql.ErrNoRows {
   254→		cclog.Errorf("NATS start job: checking for duplicate failed: %v", err)
   255→		return
   256→	}
   257→	if err == nil {
   258→		for _, job := range jobs {
   259→			if (req.StartTime - job.StartTime) < secondsPerDay {
   260→				cclog.Errorf("NATS start job: job with jobId %d, cluster %s already exists (dbid: %d)",
   261→					req.JobID, req.Cluster, job.ID)
   262→				return
   263→			}
   264→		}
   265→	}
   266→
   267→	// When tags are present, insert directly into the job table so that the
   268→	// returned ID can be used with AddTagOrCreate (which queries the job table).
   269→	var id int64
   270→	if len(req.Tags) > 0 {
   271→		id, err = api.JobRepository.StartDirect(&req)
   272→	} else {
   273→		id, err = api.JobRepository.Start(&req)
   274→	}
   275→	if err != nil {
   276→		cclog.Errorf("NATS start job: insert into database failed: %v", err)
   277→		return
   278→	}
   279→	unlockOnce.Do(api.RepositoryMutex.Unlock)
   280→
   281→	for _, tag := range req.Tags {
   282→		if _, err := api.JobRepository.AddTagOrCreate(nil, id, tag.Type, tag.Name, tag.Scope); err != nil {
   283→			cclog.Errorf("NATS start job: adding tag to new job %d failed: %v", id, err)
   284→			return
   285→		}
   286→	}
   287→
   288→	cclog.Infof("NATS: new job (id: %d): cluster=%s, jobId=%d, user=%s, startTime=%d",
   289→		id, req.Cluster, req.JobID, req.User, req.StartTime)
   290→}
   291→
   292→// handleStopJob processes job stop messages received via NATS.
   293→// The payload parameter contains JSON following the StopJobAPIRequest structure.
   294→// The job is marked as stopped in the database and archiving is triggered if monitoring is enabled.
   295→func (api *NatsAPI) handleStopJob(payload string) {
   296→	if payload == "" {
   297→		cclog.Error("NATS stop job: payload is empty")
   298→		return
   299→	}
   300→	var req StopJobAPIRequest
   301→
   302→	dec := json.NewDecoder(strings.NewReader(payload))
   303→	dec.DisallowUnknownFields()
   304→	if err := dec.Decode(&req); err != nil {
   305→		cclog.Errorf("NATS job stop: parsing request failed: %v", err)
   306→		return
   307→	}
   308→
   309→	if req.JobID == nil {
   310→		cclog.Errorf("NATS job stop: the field 'jobId' is required")
   311→		return
   312→	}
   313→
   314→	isCached := false
   315→	job, err := api.JobRepository.FindCached(req.JobID, req.Cluster, req.StartTime)
   316→	if err != nil {
   317→		// Not in cache, try main job table
   318→		job, err = api.JobRepository.Find(req.JobID, req.Cluster, req.StartTime)
   319→		if err != nil {
   320→			cclog.Errorf("NATS job stop: finding job failed: %v", err)
   321→			return
   322→		}
   323→	} else {
   324→		isCached = true
   325→	}
   326→
   327→	if job.State != schema.JobStateRunning {
   328→		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: job has already been stopped (state is: %s)",
   329→			job.JobID, job.ID, job.Cluster, job.State)
   330→		return
   331→	}
   332→
   333→	if job.StartTime > req.StopTime {
   334→		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: stopTime %d must be >= startTime %d",
   335→			job.JobID, job.ID, job.Cluster, req.StopTime, job.StartTime)
   336→		return
   337→	}
   338→
   339→	if req.State != "" && !req.State.Valid() {
   340→		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: invalid job state: %#v",
   341→			job.JobID, job.ID, job.Cluster, req.State)
   342→		return
   343→	} else if req.State == "" {
   344→		req.State = schema.JobStateCompleted
   345→	}
   346→
   347→	job.Duration = int32(req.StopTime - job.StartTime)
   348→	job.State = req.State
   349→	api.JobRepository.Mutex.Lock()
   350→	defer api.JobRepository.Mutex.Unlock()
   351→
   352→	// If the job is still in job_cache, transfer it to the job table first
   353→	if isCached {
   354→		newID, err := api.JobRepository.TransferCachedJobToMain(*job.ID)
   355→		if err != nil {
   356→			cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: transferring cached job failed: %v",
   357→				job.JobID, *job.ID, job.Cluster, err)
   358→			return
   359→		}
   360→		cclog.Infof("NATS: transferred cached job to main table: old id %d -> new id %d (jobId=%d)", *job.ID, newID, job.JobID)
   361→		job.ID = &newID
   362→	}
   363→
   364→	if err := api.JobRepository.Stop(*job.ID, job.Duration, job.State, job.MonitoringStatus); err != nil {
   365→		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: marking job as '%s' failed: %v",
   366→			job.JobID, *job.ID, job.Cluster, job.State, err)
   367→		return
   368→	}
   369→
   370→	cclog.Infof("NATS: archiving job (dbid: %d): cluster=%s, jobId=%d, user=%s, startTime=%d, duration=%d, state=%s",
   371→		*job.ID, job.Cluster, job.JobID, job.User, job.StartTime, job.Duration, job.State)
   372→
   373→	if job.MonitoringStatus == schema.MonitoringStatusDisabled {
   374→		return
   375→	}
   376→
   377→	archiver.TriggerArchiving(job)
   378→}
   379→
   380→// processNodestateEvent extracts and processes node state data from the InfluxDB message.
   381→// Updates node states in the repository for all nodes in the payload.
   382→func (api *NatsAPI) processNodestateEvent(msg lp.CCMessage) {
   383→	v, ok := msg.GetEventValue()
   384→	if !ok {
   385→		cclog.Errorf("Nodestate event is missing event field with JSON payload")
   386→		return
   387→	}
   388→
   389→	var req UpdateNodeStatesRequest
   390→
   391→	dec := json.NewDecoder(strings.NewReader(v))
   392→	dec.DisallowUnknownFields()
   393→	if err := dec.Decode(&req); err != nil {
   394→		cclog.Errorf("NATS nodestate: parsing request failed: %v", err)
   395→		return
   396→	}
   397→
   398→	repo := repository.GetNodeRepository()
   399→	requestReceived := time.Now().Unix()
   400→
   401→	updates := make([]repository.NodeStateUpdate, 0, len(req.Nodes))
   402→	for _, node := range req.Nodes {
   403→		state := determineState(node.States)
   404→		nodeState := schema.NodeStateDB{
   405→			TimeStamp:       requestReceived,
   406→			NodeState:       state,
   407→			CpusAllocated:   node.CpusAllocated,
   408→			MemoryAllocated: node.MemoryAllocated,
   409→			GpusAllocated:   node.GpusAllocated,
   410→			HealthState:     schema.MonitoringStateFull,
   411→			JobsRunning:     node.JobsRunning,
   412→		}
   413→		updates = append(updates, repository.NodeStateUpdate{
   414→			Hostname:  node.Hostname,
   415→			Cluster:   req.Cluster,
   416→			NodeState: &nodeState,
   417→		})
   418→	}
   419→
   420→	if err := repo.BatchUpdateNodeStates(updates); err != nil {
   421→		cclog.Errorf("NATS nodestate: batch update for cluster %s failed: %v", req.Cluster, err)
   422→	}
   423→
   424→	cclog.Debugf("NATS nodestate: updated %d node states for cluster %s", len(req.Nodes), req.Cluster)
   425→}
   426→
   427→// handleNodeState processes node state update messages received via NATS using InfluxDB line protocol.
   428→// The message must be in line protocol format with measurement="nodestate" and include:
   429→//   - field "event" containing JSON payload (UpdateNodeStatesRequest)
   430→//
   431→// Example: nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[...]}" 1234567890000000000
   432→func (api *NatsAPI) handleNodeState(subject string, data []byte) {
   433→	if len(data) == 0 {
   434→		cclog.Warnf("NATS %s: received empty message", subject)
   435→		return
   436→	}
   437→
   438→	d := influx.NewDecoderWithBytes(data)
   439→
   440→	for d.Next() {
   441→		m, err := receivers.DecodeInfluxMessage(d)
   442→		if err != nil {
   443→			cclog.Errorf("NATS %s: failed to decode InfluxDB line protocol message: %v", subject, err)
   444→			return
   445→		}
   446→
   447→		if !m.IsEvent() {
   448→			cclog.Warnf("NATS %s: received non-event message, skipping", subject)
   449→			continue
   450→		}
   451→
   452→		if m.Name() == "nodestate" {
   453→			api.processNodestateEvent(m)
   454→		} else {
   455→			cclog.Warnf("NATS %s: unexpected measurement name '%s', expected 'nodestate'", subject, m.Name())
   456→		}
   457→	}
   458→}
   459→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package api

import (
	"database/sql"
	"encoding/json"
	"strings"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/archiver"
	"github.com/ClusterCockpit/cc-backend/internal/config"
	"github.com/ClusterCockpit/cc-backend/internal/importer"
	"github.com/ClusterCockpit/cc-backend/internal/repository"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	lp "github.com/ClusterCockpit/cc-lib/v2/ccMessage"
	"github.com/ClusterCockpit/cc-lib/v2/nats"
	"github.com/ClusterCockpit/cc-lib/v2/receivers"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	influx "github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
)

// natsMessage wraps a raw NATS message with its subject for channel-based processing.
type natsMessage struct {
	subject string
	data    []byte
}

// NatsAPI provides NATS subscription-based handlers for Job and Node operations.
// It mirrors the functionality of the REST API but uses NATS messaging with
// InfluxDB line protocol as the message format.
//
// # Message Format
//
// All NATS messages use InfluxDB line protocol format (https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/)
// with the following structure:
//
//	measurement,tag1=value1,tag2=value2 field1=value1,field2=value2 timestamp
//
// # Job Events
//
// Job start/stop events use the "job" measurement with a "function" tag to distinguish operations:
//
//	job,function=start_job event="{...JSON payload...}" <timestamp>
//	job,function=stop_job event="{...JSON payload...}" <timestamp>
//
// The JSON payload in the "event" field follows the schema.Job or StopJobAPIRequest structure.
//
// Example job start message:
//
//	job,function=start_job event="{\"jobId\":1001,\"user\":\"testuser\",\"cluster\":\"testcluster\",...}" 1234567890000000000
//
// # Node State Events
//
// Node state updates use the "nodestate" measurement with cluster information:
//
//	nodestate event="{...JSON payload...}" <timestamp>
//
// The JSON payload follows the UpdateNodeStatesRequest structure.
//
// Example node state message:
//
//	nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"node01\",\"states\":[\"idle\"]}]}" 1234567890000000000
type NatsAPI struct {
	JobRepository *repository.JobRepository
	// RepositoryMutex protects job creation operations from race conditions
	// when checking for duplicate jobs during startJob calls.
	RepositoryMutex sync.Mutex
	// jobCh receives job event messages for processing by worker goroutines.
	jobCh chan natsMessage
	// nodeCh receives node state messages for processing by worker goroutines.
	nodeCh chan natsMessage
}

// NewNatsAPI creates a new NatsAPI instance with channel-based worker pools.
// Concurrency is configured via NATSConfig (defaults: JobConcurrency=8, NodeConcurrency=2).
func NewNatsAPI() *NatsAPI {
	jobConc := 8
	nodeConc := 2

	if s := config.Keys.APISubjects; s != nil {
		if s.JobConcurrency > 0 {
			jobConc = s.JobConcurrency
		}
		if s.NodeConcurrency > 0 {
			nodeConc = s.NodeConcurrency
		}
	}

	api := &NatsAPI{
		JobRepository: repository.GetJobRepository(),
		jobCh:         make(chan natsMessage, jobConc),
		nodeCh:        make(chan natsMessage, nodeConc),
	}

	// Start worker goroutines
	for range jobConc {
		go api.jobWorker()
	}
	for range nodeConc {
		go api.nodeWorker()
	}

	return api
}

// jobWorker processes job event messages from the job channel.
func (api *NatsAPI) jobWorker() {
	for msg := range api.jobCh {
		api.handleJobEvent(msg.subject, msg.data)
	}
}

// nodeWorker processes node state messages from the node channel.
func (api *NatsAPI) nodeWorker() {
	for msg := range api.nodeCh {
		api.handleNodeState(msg.subject, msg.data)
	}
}

// StartSubscriptions registers all NATS subscriptions for Job and Node APIs.
// Messages are delivered to buffered channels and processed by worker goroutines.
// Returns an error if the NATS client is not available or subscription fails.
func (api *NatsAPI) StartSubscriptions() error {
	client := nats.GetClient()
	if client == nil {
		cclog.Warn("NATS client not available, skipping API subscriptions")
		return nil
	}

	if config.Keys.APISubjects != nil {
		s := config.Keys.APISubjects

		if err := client.Subscribe(s.SubjectJobEvent, func(subject string, data []byte) {
			api.jobCh <- natsMessage{subject: subject, data: data}
		}); err != nil {
			return err
		}

		if err := client.Subscribe(s.SubjectNodeState, func(subject string, data []byte) {
			api.nodeCh <- natsMessage{subject: subject, data: data}
		}); err != nil {
			return err
		}

		cclog.Info("NATS API subscriptions started")
	}
	return nil
}

// processJobEvent routes job event messages to the appropriate handler based on the "function" tag.
// Validates that required tags and fields are present before processing.
func (api *NatsAPI) processJobEvent(msg lp.CCMessage) {
	function, ok := msg.GetTag("function")
	if !ok {
		cclog.Errorf("Job event is missing required tag 'function': measurement=%s", msg.Name())
		return
	}

	switch function {
	case "start_job":
		v, ok := msg.GetEventValue()
		if !ok {
			cclog.Errorf("Job start event is missing event field with JSON payload")
			return
		}
		api.handleStartJob(v)

	case "stop_job":
		v, ok := msg.GetEventValue()
		if !ok {
			cclog.Errorf("Job stop event is missing event field with JSON payload")
			return
		}
		api.handleStopJob(v)

	default:
		cclog.Warnf("Unknown job event function '%s', expected 'start_job' or 'stop_job'", function)
	}
}

// handleJobEvent processes job-related messages received via NATS using InfluxDB line protocol.
// The message must be in line protocol format with measurement="job" and include:
//   - tag "function" with value "start_job" or "stop_job"
//   - field "event" containing JSON payload (schema.Job or StopJobAPIRequest)
//
// Example: job,function=start_job event="{\"jobId\":1001,...}" 1234567890000000000
func (api *NatsAPI) handleJobEvent(subject string, data []byte) {
	if len(data) == 0 {
		cclog.Warnf("NATS %s: received empty message", subject)
		return
	}

	d := influx.NewDecoderWithBytes(data)

	for d.Next() {
		m, err := receivers.DecodeInfluxMessage(d)
		if err != nil {
			cclog.Errorf("NATS %s: failed to decode InfluxDB line protocol message: %v", subject, err)
			return
		}

		if !m.IsEvent() {
			cclog.Debugf("NATS %s: received non-event message, skipping", subject)
			continue
		}

		if m.Name() == "job" {
			api.processJobEvent(m)
		} else {
			cclog.Debugf("NATS %s: unexpected measurement name '%s', expected 'job'", subject, m.Name())
		}
	}
}

// handleStartJob processes job start messages received via NATS.
// The payload parameter contains JSON following the schema.Job structure.
// Jobs are validated, checked for duplicates, and inserted into the database.
func (api *NatsAPI) handleStartJob(payload string) {
	if payload == "" {
		cclog.Error("NATS start job: payload is empty")
		return
	}
	req := schema.Job{
		Shared:           "none",
		MonitoringStatus: schema.MonitoringStatusRunningOrArchiving,
	}

	dec := json.NewDecoder(strings.NewReader(payload))
	dec.DisallowUnknownFields()
	if err := dec.Decode(&req); err != nil {
		cclog.Errorf("NATS start job: parsing request failed: %v", err)
		return
	}

	cclog.Debugf("NATS start job: %s", req.GoString())
	req.State = schema.JobStateRunning

	if err := importer.SanityChecks(&req); err != nil {
		cclog.Errorf("NATS start job: sanity check failed: %v", err)
		return
	}

	var unlockOnce sync.Once
	api.RepositoryMutex.Lock()
	defer unlockOnce.Do(api.RepositoryMutex.Unlock)

	jobs, err := api.JobRepository.FindAll(&req.JobID, &req.Cluster, nil)
	if err != nil && err != sql.ErrNoRows {
		cclog.Errorf("NATS start job: checking for duplicate failed: %v", err)
		return
	}
	if err == nil {
		for _, job := range jobs {
			if (req.StartTime - job.StartTime) < secondsPerDay {
				cclog.Errorf("NATS start job: job with jobId %d, cluster %s already exists (dbid: %d)",
					req.JobID, req.Cluster, job.ID)
				return
			}
		}
	}

	// When tags are present, insert directly into the job table so that the
	// returned ID can be used with AddTagOrCreate (which queries the job table).
	var id int64
	if len(req.Tags) > 0 {
		id, err = api.JobRepository.StartDirect(&req)
	} else {
		id, err = api.JobRepository.Start(&req)
	}
	if err != nil {
		cclog.Errorf("NATS start job: insert into database failed: %v", err)
		return
	}
	unlockOnce.Do(api.RepositoryMutex.Unlock)

	for _, tag := range req.Tags {
		if _, err := api.JobRepository.AddTagOrCreate(nil, id, tag.Type, tag.Name, tag.Scope); err != nil {
			cclog.Errorf("NATS start job: adding tag to new job %d failed: %v", id, err)
			return
		}
	}

	cclog.Infof("NATS: new job (id: %d): cluster=%s, jobId=%d, user=%s, startTime=%d",
		id, req.Cluster, req.JobID, req.User, req.StartTime)
}

// handleStopJob processes job stop messages received via NATS.
// The payload parameter contains JSON following the StopJobAPIRequest structure.
// The job is marked as stopped in the database and archiving is triggered if monitoring is enabled.
func (api *NatsAPI) handleStopJob(payload string) {
	if payload == "" {
		cclog.Error("NATS stop job: payload is empty")
		return
	}
	var req StopJobAPIRequest

	dec := json.NewDecoder(strings.NewReader(payload))
	dec.DisallowUnknownFields()
	if err := dec.Decode(&req); err != nil {
		cclog.Errorf("NATS job stop: parsing request failed: %v", err)
		return
	}

	if req.JobID == nil {
		cclog.Errorf("NATS job stop: the field 'jobId' is required")
		return
	}

	isCached := false
	job, err := api.JobRepository.FindCached(req.JobID, req.Cluster, req.StartTime)
	if err != nil {
		// Not in cache, try main job table
		job, err = api.JobRepository.Find(req.JobID, req.Cluster, req.StartTime)
		if err != nil {
			cclog.Errorf("NATS job stop: finding job failed: %v", err)
			return
		}
	} else {
		isCached = true
	}

	if job.State != schema.JobStateRunning {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: job has already been stopped (state is: %s)",
			job.JobID, job.ID, job.Cluster, job.State)
		return
	}

	if job.StartTime > req.StopTime {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: stopTime %d must be >= startTime %d",
			job.JobID, job.ID, job.Cluster, req.StopTime, job.StartTime)
		return
	}

	if req.State != "" && !req.State.Valid() {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: invalid job state: %#v",
			job.JobID, job.ID, job.Cluster, req.State)
		return
	} else if req.State == "" {
		req.State = schema.JobStateCompleted
	}

	job.Duration = int32(req.StopTime - job.StartTime)
	job.State = req.State
	api.JobRepository.Mutex.Lock()
	defer api.JobRepository.Mutex.Unlock()

	// If the job is still in job_cache, transfer it to the job table first
	if isCached {
		newID, err := api.JobRepository.TransferCachedJobToMain(*job.ID)
		if err != nil {
			cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: transferring cached job failed: %v",
				job.JobID, *job.ID, job.Cluster, err)
			return
		}
		cclog.Infof("NATS: transferred cached job to main table: old id %d -> new id %d (jobId=%d)", *job.ID, newID, job.JobID)
		job.ID = &newID
	}

	if err := api.JobRepository.Stop(*job.ID, job.Duration, job.State, job.MonitoringStatus); err != nil {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: marking job as '%s' failed: %v",
			job.JobID, *job.ID, job.Cluster, job.State, err)
		return
	}

	cclog.Infof("NATS: archiving job (dbid: %d): cluster=%s, jobId=%d, user=%s, startTime=%d, duration=%d, state=%s",
		*job.ID, job.Cluster, job.JobID, job.User, job.StartTime, job.Duration, job.State)

	if job.MonitoringStatus == schema.MonitoringStatusDisabled {
		return
	}

	archiver.TriggerArchiving(job)
}

// processNodestateEvent extracts and processes node state data from the InfluxDB message.
// Updates node states in the repository for all nodes in the payload.
func (api *NatsAPI) processNodestateEvent(msg lp.CCMessage) {
	v, ok := msg.GetEventValue()
	if !ok {
		cclog.Errorf("Nodestate event is missing event field with JSON payload")
		return
	}

	var req UpdateNodeStatesRequest

	dec := json.NewDecoder(strings.NewReader(v))
	dec.DisallowUnknownFields()
	if err := dec.Decode(&req); err != nil {
		cclog.Errorf("NATS nodestate: parsing request failed: %v", err)
		return
	}

	repo := repository.GetNodeRepository()
	requestReceived := time.Now().Unix()

	updates := make([]repository.NodeStateUpdate, 0, len(req.Nodes))
	for _, node := range req.Nodes {
		state := determineState(node.States)
		nodeState := schema.NodeStateDB{
			TimeStamp:       requestReceived,
			NodeState:       state,
			CpusAllocated:   node.CpusAllocated,
			MemoryAllocated: node.MemoryAllocated,
			GpusAllocated:   node.GpusAllocated,
			HealthState:     schema.MonitoringStateFull,
			JobsRunning:     node.JobsRunning,
		}
		updates = append(updates, repository.NodeStateUpdate{
			Hostname:  node.Hostname,
			Cluster:   req.Cluster,
			NodeState: &nodeState,
		})
	}

	if err := repo.BatchUpdateNodeStates(updates); err != nil {
		cclog.Errorf("NATS nodestate: batch update for cluster %s failed: %v", req.Cluster, err)
	}

	cclog.Debugf("NATS nodestate: updated %d node states for cluster %s", len(req.Nodes), req.Cluster)
}

// handleNodeState processes node state update messages received via NATS using InfluxDB line protocol.
// The message must be in line protocol format with measurement="nodestate" and include:
//   - field "event" containing JSON payload (UpdateNodeStatesRequest)
//
// Example: nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[...]}" 1234567890000000000
func (api *NatsAPI) handleNodeState(subject string, data []byte) {
	if len(data) == 0 {
		cclog.Warnf("NATS %s: received empty message", subject)
		return
	}

	d := influx.NewDecoderWithBytes(data)

	for d.Next() {
		m, err := receivers.DecodeInfluxMessage(d)
		if err != nil {
			cclog.Errorf("NATS %s: failed to decode InfluxDB line protocol message: %v", subject, err)
			return
		}

		if !m.IsEvent() {
			cclog.Warnf("NATS %s: received non-event message, skipping", subject)
			continue
		}

		if m.Name() == "nodestate" {
			api.processNodestateEvent(m)
		} else {
			cclog.Warnf("NATS %s: unexpected measurement name '%s', expected 'nodestate'", subject, m.Name())
		}
	}
}


     1→// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
     2→// All rights reserved. This file is part of cc-backend.
     3→// Use of this source code is governed by a MIT-style
     4→// license that can be found in the LICENSE file.
     5→
     6→package api
     7→
     8→import (
     9→	"fmt"
    10→	"maps"
    11→	"net/http"
    12→	"strings"
    13→	"time"
    14→
    15→	"github.com/ClusterCockpit/cc-backend/internal/metricdispatch"
    16→	"github.com/ClusterCockpit/cc-backend/internal/repository"
    17→	"github.com/ClusterCockpit/cc-backend/pkg/archive"
    18→	"github.com/ClusterCockpit/cc-backend/pkg/metricstore"
    19→	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
    20→	"github.com/ClusterCockpit/cc-lib/v2/schema"
    21→)
    22→
    23→type UpdateNodeStatesRequest struct {
    24→	Nodes   []schema.NodePayload `json:"nodes"`
    25→	Cluster string               `json:"cluster" example:"fritz"`
    26→}
    27→
    28→// metricListToNames converts a map of metric configurations to a list of metric names
    29→func metricListToNames(metricList map[string]*schema.Metric) []string {
    30→	names := make([]string, 0, len(metricList))
    31→	for name := range metricList {
    32→		names = append(names, name)
    33→	}
    34→	return names
    35→}
    36→
    37→// this routine assumes that only one of them exists per node
    38→func determineState(states []string) schema.SchedulerState {
    39→	for _, state := range states {
    40→		switch strings.ToLower(state) {
    41→		case "allocated":
    42→			return schema.NodeStateAllocated
    43→		case "reserved":
    44→			return schema.NodeStateReserved
    45→		case "idle":
    46→			return schema.NodeStateIdle
    47→		case "down":
    48→			return schema.NodeStateDown
    49→		case "mixed":
    50→			return schema.NodeStateMixed
    51→		}
    52→	}
    53→
    54→	return schema.NodeStateUnknown
    55→}
    56→
    57→// updateNodeStates godoc
    58→// @summary     Deliver updated Slurm node states
    59→// @tags Nodestates
    60→// @description Returns a JSON-encoded list of users.
    61→// @description Required query-parameter defines if all users or only users with additional special roles are returned.
    62→// @produce     json
    63→// @param       request body UpdateNodeStatesRequest true "Request body containing nodes and their states"
    64→// @success     200     {object} api.DefaultAPIResponse "Success message"
    65→// @failure     400     {object} api.ErrorResponse      "Bad Request"
    66→// @failure     401     {object} api.ErrorResponse      "Unauthorized"
    67→// @failure     403     {object} api.ErrorResponse      "Forbidden"
    68→// @failure     500     {object} api.ErrorResponse      "Internal Server Error"
    69→// @security    ApiKeyAuth
    70→// @router      /api/nodestats/ [post]
    71→func (api *RestAPI) updateNodeStates(rw http.ResponseWriter, r *http.Request) {
    72→	// Parse request body
    73→	req := UpdateNodeStatesRequest{}
    74→	if err := decode(r.Body, &req); err != nil {
    75→		handleError(fmt.Errorf("parsing request body failed: %w", err),
    76→			http.StatusBadRequest, rw)
    77→		return
    78→	}
    79→	requestReceived := time.Now().Unix()
    80→	repo := repository.GetNodeRepository()
    81→
    82→	m := make(map[string][]string)
    83→	metricNames := make(map[string][]string)
    84→	healthResults := make(map[string]metricstore.HealthCheckResult)
    85→
    86→	startMs := time.Now()
    87→
    88→	// Step 1: Build nodeList and metricList per subcluster
    89→	for _, node := range req.Nodes {
    90→		if sc, err := archive.GetSubClusterByNode(req.Cluster, node.Hostname); err == nil {
    91→			m[sc] = append(m[sc], node.Hostname)
    92→		}
    93→	}
    94→
    95→	for sc := range m {
    96→		if sc != "" {
    97→			metricList := archive.GetMetricConfigSubCluster(req.Cluster, sc)
    98→			metricNames[sc] = metricListToNames(metricList)
    99→		}
   100→	}
   101→
   102→	// Step 2: Determine which metric store to query and perform health check
   103→	healthRepo, err := metricdispatch.GetHealthCheckRepo(req.Cluster)
   104→	if err != nil {
   105→		cclog.Warnf("updateNodeStates: no metric store for cluster %s, skipping health check: %v", req.Cluster, err)
   106→	} else {
   107→		for sc, nl := range m {
   108→			if sc != "" {
   109→				if results, err := healthRepo.HealthCheck(req.Cluster, nl, metricNames[sc]); err == nil {
   110→					maps.Copy(healthResults, results)
   111→				}
   112→			}
   113→		}
   114→	}
   115→
   116→	cclog.Debugf("Timer updateNodeStates, MemStore HealthCheck: %s", time.Since(startMs))
   117→	startDB := time.Now()
   118→
   119→	updates := make([]repository.NodeStateUpdate, 0, len(req.Nodes))
   120→	for _, node := range req.Nodes {
   121→		state := determineState(node.States)
   122→		healthState := schema.MonitoringStateFailed
   123→		var healthMetrics string
   124→		if result, ok := healthResults[node.Hostname]; ok {
   125→			healthState = result.State
   126→			healthMetrics = result.HealthMetrics
   127→		}
   128→		nodeState := schema.NodeStateDB{
   129→			TimeStamp:       requestReceived,
   130→			NodeState:       state,
   131→			CpusAllocated:   node.CpusAllocated,
   132→			MemoryAllocated: node.MemoryAllocated,
   133→			GpusAllocated:   node.GpusAllocated,
   134→			HealthState:     healthState,
   135→			HealthMetrics:   healthMetrics,
   136→			JobsRunning:     node.JobsRunning,
   137→		}
   138→		updates = append(updates, repository.NodeStateUpdate{
   139→			Hostname:  node.Hostname,
   140→			Cluster:   req.Cluster,
   141→			NodeState: &nodeState,
   142→		})
   143→	}
   144→
   145→	if err := repo.BatchUpdateNodeStates(updates); err != nil {
   146→		cclog.Errorf("updateNodeStates: batch update for cluster %s failed: %v", req.Cluster, err)
   147→	}
   148→
   149→	cclog.Debugf("Timer updateNodeStates, SQLite Inserts: %s", time.Since(startDB))
   150→}
   151→

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package api

import (
	"fmt"
	"maps"
	"net/http"
	"strings"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/metricdispatch"
	"github.com/ClusterCockpit/cc-backend/internal/repository"
	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	"github.com/ClusterCockpit/cc-backend/pkg/metricstore"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
)

type UpdateNodeStatesRequest struct {
	Nodes   []schema.NodePayload `json:"nodes"`
	Cluster string               `json:"cluster" example:"fritz"`
}

// metricListToNames converts a map of metric configurations to a list of metric names
func metricListToNames(metricList map[string]*schema.Metric) []string {
	names := make([]string, 0, len(metricList))
	for name := range metricList {
		names = append(names, name)
	}
	return names
}

// this routine assumes that only one of them exists per node
func determineState(states []string) schema.SchedulerState {
	for _, state := range states {
		switch strings.ToLower(state) {
		case "allocated":
			return schema.NodeStateAllocated
		case "reserved":
			return schema.NodeStateReserved
		case "idle":
			return schema.NodeStateIdle
		case "down":
			return schema.NodeStateDown
		case "mixed":
			return schema.NodeStateMixed
		}
	}

	return schema.NodeStateUnknown
}

// updateNodeStates godoc
// @summary     Deliver updated Slurm node states
// @tags Nodestates
// @description Returns a JSON-encoded list of users.
// @description Required query-parameter defines if all users or only users with additional special roles are returned.
// @produce     json
// @param       request body UpdateNodeStatesRequest true "Request body containing nodes and their states"
// @success     200     {object} api.DefaultAPIResponse "Success message"
// @failure     400     {object} api.ErrorResponse      "Bad Request"
// @failure     401     {object} api.ErrorResponse      "Unauthorized"
// @failure     403     {object} api.ErrorResponse      "Forbidden"
// @failure     500     {object} api.ErrorResponse      "Internal Server Error"
// @security    ApiKeyAuth
// @router      /api/nodestats/ [post]
func (api *RestAPI) updateNodeStates(rw http.ResponseWriter, r *http.Request) {
	// Parse request body
	req := UpdateNodeStatesRequest{}
	if err := decode(r.Body, &req); err != nil {
		handleError(fmt.Errorf("parsing request body failed: %w", err),
			http.StatusBadRequest, rw)
		return
	}
	requestReceived := time.Now().Unix()
	repo := repository.GetNodeRepository()

	m := make(map[string][]string)
	metricNames := make(map[string][]string)
	healthResults := make(map[string]metricstore.HealthCheckResult)

	startMs := time.Now()

	// Step 1: Build nodeList and metricList per subcluster
	for _, node := range req.Nodes {
		if sc, err := archive.GetSubClusterByNode(req.Cluster, node.Hostname); err == nil {
			m[sc] = append(m[sc], node.Hostname)
		}
	}

	for sc := range m {
		if sc != "" {
			metricList := archive.GetMetricConfigSubCluster(req.Cluster, sc)
			metricNames[sc] = metricListToNames(metricList)
		}
	}

	// Step 2: Determine which metric store to query and perform health check
	healthRepo, err := metricdispatch.GetHealthCheckRepo(req.Cluster)
	if err != nil {
		cclog.Warnf("updateNodeStates: no metric store for cluster %s, skipping health check: %v", req.Cluster, err)
	} else {
		for sc, nl := range m {
			if sc != "" {
				if results, err := healthRepo.HealthCheck(req.Cluster, nl, metricNames[sc]); err == nil {
					maps.Copy(healthResults, results)
				}
			}
		}
	}

	cclog.Debugf("Timer updateNodeStates, MemStore HealthCheck: %s", time.Since(startMs))
	startDB := time.Now()

	updates := make([]repository.NodeStateUpdate, 0, len(req.Nodes))
	for _, node := range req.Nodes {
		state := determineState(node.States)
		healthState := schema.MonitoringStateFailed
		var healthMetrics string
		if result, ok := healthResults[node.Hostname]; ok {
			healthState = result.State
			healthMetrics = result.HealthMetrics
		}
		nodeState := schema.NodeStateDB{
			TimeStamp:       requestReceived,
			NodeState:       state,
			CpusAllocated:   node.CpusAllocated,
			MemoryAllocated: node.MemoryAllocated,
			GpusAllocated:   node.GpusAllocated,
			HealthState:     healthState,
			HealthMetrics:   healthMetrics,
			JobsRunning:     node.JobsRunning,
		}
		updates = append(updates, repository.NodeStateUpdate{
			Hostname:  node.Hostname,
			Cluster:   req.Cluster,
			NodeState: &nodeState,
		})
	}

	if err := repo.BatchUpdateNodeStates(updates); err != nil {
		cclog.Errorf("updateNodeStates: batch update for cluster %s failed: %v", req.Cluster, err)
	}

	cclog.Debugf("Timer updateNodeStates, SQLite Inserts: %s", time.Since(startDB))
}


Now I have both files. I need to:
1. Add missing imports to `nats.go`
2. Rewrite `processNodestateEvent` to include health check logic


Edit

	"database/sql"
	"encoding/json"
	"strings"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/archiver"
	"github.com/ClusterCockpit/cc-backend/internal/config"
	"github.com/ClusterCockpit/cc-backend/internal/importer"
	"github.com/ClusterCockpit/cc-backend/internal/repository"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"

	"database/sql"
	"encoding/json"
	"maps"
	"strings"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/archiver"
	"github.com/ClusterCockpit/cc-backend/internal/config"
	"github.com/ClusterCockpit/cc-backend/internal/importer"
	"github.com/ClusterCockpit/cc-backend/internal/metricdispatch"
	"github.com/ClusterCockpit/cc-backend/internal/repository"
	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	"github.com/ClusterCockpit/cc-backend/pkg/metricstore"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"


The file /Users/jan/prg/ClusterCockpit/cc-backend/internal/api/nats.go has been updated successfully.

	"database/sql"
	"encoding/json"
	"maps"
	"strings"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/archiver"
	"github.com/ClusterCockpit/cc-backend/internal/config"
	"github.com/ClusterCockpit/cc-backend/internal/importer"
	"github.com/ClusterCockpit/cc-backend/internal/metricdispatch"
	"github.com/ClusterCockpit/cc-backend/internal/repository"
	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	"github.com/ClusterCockpit/cc-backend/pkg/metricstore"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package api

import (
	"database/sql"
	"encoding/json"
	"strings"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/archiver"
	"github.com/ClusterCockpit/cc-backend/internal/config"
	"github.com/ClusterCockpit/cc-backend/internal/importer"
	"github.com/ClusterCockpit/cc-backend/internal/repository"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	lp "github.com/ClusterCockpit/cc-lib/v2/ccMessage"
	"github.com/ClusterCockpit/cc-lib/v2/nats"
	"github.com/ClusterCockpit/cc-lib/v2/receivers"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	influx "github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
)

// natsMessage wraps a raw NATS message with its subject for channel-based processing.
type natsMessage struct {
	subject string
	data    []byte
}

// NatsAPI provides NATS subscription-based handlers for Job and Node operations.
// It mirrors the functionality of the REST API but uses NATS messaging with
// InfluxDB line protocol as the message format.
//
// # Message Format
//
// All NATS messages use InfluxDB line protocol format (https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/)
// with the following structure:
//
//	measurement,tag1=value1,tag2=value2 field1=value1,field2=value2 timestamp
//
// # Job Events
//
// Job start/stop events use the "job" measurement with a "function" tag to distinguish operations:
//
//	job,function=start_job event="{...JSON payload...}" <timestamp>
//	job,function=stop_job event="{...JSON payload...}" <timestamp>
//
// The JSON payload in the "event" field follows the schema.Job or StopJobAPIRequest structure.
//
// Example job start message:
//
//	job,function=start_job event="{\"jobId\":1001,\"user\":\"testuser\",\"cluster\":\"testcluster\",...}" 1234567890000000000
//
// # Node State Events
//
// Node state updates use the "nodestate" measurement with cluster information:
//
//	nodestate event="{...JSON payload...}" <timestamp>
//
// The JSON payload follows the UpdateNodeStatesRequest structure.
//
// Example node state message:
//
//	nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"node01\",\"states\":[\"idle\"]}]}" 1234567890000000000
type NatsAPI struct {
	JobRepository *repository.JobRepository
	// RepositoryMutex protects job creation operations from race conditions
	// when checking for duplicate jobs during startJob calls.
	RepositoryMutex sync.Mutex
	// jobCh receives job event messages for processing by worker goroutines.
	jobCh chan natsMessage
	// nodeCh receives node state messages for processing by worker goroutines.
	nodeCh chan natsMessage
}

// NewNatsAPI creates a new NatsAPI instance with channel-based worker pools.
// Concurrency is configured via NATSConfig (defaults: JobConcurrency=8, NodeConcurrency=2).
func NewNatsAPI() *NatsAPI {
	jobConc := 8
	nodeConc := 2

	if s := config.Keys.APISubjects; s != nil {
		if s.JobConcurrency > 0 {
			jobConc = s.JobConcurrency
		}
		if s.NodeConcurrency > 0 {
			nodeConc = s.NodeConcurrency
		}
	}

	api := &NatsAPI{
		JobRepository: repository.GetJobRepository(),
		jobCh:         make(chan natsMessage, jobConc),
		nodeCh:        make(chan natsMessage, nodeConc),
	}

	// Start worker goroutines
	for range jobConc {
		go api.jobWorker()
	}
	for range nodeConc {
		go api.nodeWorker()
	}

	return api
}

// jobWorker processes job event messages from the job channel.
func (api *NatsAPI) jobWorker() {
	for msg := range api.jobCh {
		api.handleJobEvent(msg.subject, msg.data)
	}
}

// nodeWorker processes node state messages from the node channel.
func (api *NatsAPI) nodeWorker() {
	for msg := range api.nodeCh {
		api.handleNodeState(msg.subject, msg.data)
	}
}

// StartSubscriptions registers all NATS subscriptions for Job and Node APIs.
// Messages are delivered to buffered channels and processed by worker goroutines.
// Returns an error if the NATS client is not available or subscription fails.
func (api *NatsAPI) StartSubscriptions() error {
	client := nats.GetClient()
	if client == nil {
		cclog.Warn("NATS client not available, skipping API subscriptions")
		return nil
	}

	if config.Keys.APISubjects != nil {
		s := config.Keys.APISubjects

		if err := client.Subscribe(s.SubjectJobEvent, func(subject string, data []byte) {
			api.jobCh <- natsMessage{subject: subject, data: data}
		}); err != nil {
			return err
		}

		if err := client.Subscribe(s.SubjectNodeState, func(subject string, data []byte) {
			api.nodeCh <- natsMessage{subject: subject, data: data}
		}); err != nil {
			return err
		}

		cclog.Info("NATS API subscriptions started")
	}
	return nil
}

// processJobEvent routes job event messages to the appropriate handler based on the "function" tag.
// Validates that required tags and fields are present before processing.
func (api *NatsAPI) processJobEvent(msg lp.CCMessage) {
	function, ok := msg.GetTag("function")
	if !ok {
		cclog.Errorf("Job event is missing required tag 'function': measurement=%s", msg.Name())
		return
	}

	switch function {
	case "start_job":
		v, ok := msg.GetEventValue()
		if !ok {
			cclog.Errorf("Job start event is missing event field with JSON payload")
			return
		}
		api.handleStartJob(v)

	case "stop_job":
		v, ok := msg.GetEventValue()
		if !ok {
			cclog.Errorf("Job stop event is missing event field with JSON payload")
			return
		}
		api.handleStopJob(v)

	default:
		cclog.Warnf("Unknown job event function '%s', expected 'start_job' or 'stop_job'", function)
	}
}

// handleJobEvent processes job-related messages received via NATS using InfluxDB line protocol.
// The message must be in line protocol format with measurement="job" and include:
//   - tag "function" with value "start_job" or "stop_job"
//   - field "event" containing JSON payload (schema.Job or StopJobAPIRequest)
//
// Example: job,function=start_job event="{\"jobId\":1001,...}" 1234567890000000000
func (api *NatsAPI) handleJobEvent(subject string, data []byte) {
	if len(data) == 0 {
		cclog.Warnf("NATS %s: received empty message", subject)
		return
	}

	d := influx.NewDecoderWithBytes(data)

	for d.Next() {
		m, err := receivers.DecodeInfluxMessage(d)
		if err != nil {
			cclog.Errorf("NATS %s: failed to decode InfluxDB line protocol message: %v", subject, err)
			return
		}

		if !m.IsEvent() {
			cclog.Debugf("NATS %s: received non-event message, skipping", subject)
			continue
		}

		if m.Name() == "job" {
			api.processJobEvent(m)
		} else {
			cclog.Debugf("NATS %s: unexpected measurement name '%s', expected 'job'", subject, m.Name())
		}
	}
}

// handleStartJob processes job start messages received via NATS.
// The payload parameter contains JSON following the schema.Job structure.
// Jobs are validated, checked for duplicates, and inserted into the database.
func (api *NatsAPI) handleStartJob(payload string) {
	if payload == "" {
		cclog.Error("NATS start job: payload is empty")
		return
	}
	req := schema.Job{
		Shared:           "none",
		MonitoringStatus: schema.MonitoringStatusRunningOrArchiving,
	}

	dec := json.NewDecoder(strings.NewReader(payload))
	dec.DisallowUnknownFields()
	if err := dec.Decode(&req); err != nil {
		cclog.Errorf("NATS start job: parsing request failed: %v", err)
		return
	}

	cclog.Debugf("NATS start job: %s", req.GoString())
	req.State = schema.JobStateRunning

	if err := importer.SanityChecks(&req); err != nil {
		cclog.Errorf("NATS start job: sanity check failed: %v", err)
		return
	}

	var unlockOnce sync.Once
	api.RepositoryMutex.Lock()
	defer unlockOnce.Do(api.RepositoryMutex.Unlock)

	jobs, err := api.JobRepository.FindAll(&req.JobID, &req.Cluster, nil)
	if err != nil && err != sql.ErrNoRows {
		cclog.Errorf("NATS start job: checking for duplicate failed: %v", err)
		return
	}
	if err == nil {
		for _, job := range jobs {
			if (req.StartTime - job.StartTime) < secondsPerDay {
				cclog.Errorf("NATS start job: job with jobId %d, cluster %s already exists (dbid: %d)",
					req.JobID, req.Cluster, job.ID)
				return
			}
		}
	}

	// When tags are present, insert directly into the job table so that the
	// returned ID can be used with AddTagOrCreate (which queries the job table).
	var id int64
	if len(req.Tags) > 0 {
		id, err = api.JobRepository.StartDirect(&req)
	} else {
		id, err = api.JobRepository.Start(&req)
	}
	if err != nil {
		cclog.Errorf("NATS start job: insert into database failed: %v", err)
		return
	}
	unlockOnce.Do(api.RepositoryMutex.Unlock)

	for _, tag := range req.Tags {
		if _, err := api.JobRepository.AddTagOrCreate(nil, id, tag.Type, tag.Name, tag.Scope); err != nil {
			cclog.Errorf("NATS start job: adding tag to new job %d failed: %v", id, err)
			return
		}
	}

	cclog.Infof("NATS: new job (id: %d): cluster=%s, jobId=%d, user=%s, startTime=%d",
		id, req.Cluster, req.JobID, req.User, req.StartTime)
}

// handleStopJob processes job stop messages received via NATS.
// The payload parameter contains JSON following the StopJobAPIRequest structure.
// The job is marked as stopped in the database and archiving is triggered if monitoring is enabled.
func (api *NatsAPI) handleStopJob(payload string) {
	if payload == "" {
		cclog.Error("NATS stop job: payload is empty")
		return
	}
	var req StopJobAPIRequest

	dec := json.NewDecoder(strings.NewReader(payload))
	dec.DisallowUnknownFields()
	if err := dec.Decode(&req); err != nil {
		cclog.Errorf("NATS job stop: parsing request failed: %v", err)
		return
	}

	if req.JobID == nil {
		cclog.Errorf("NATS job stop: the field 'jobId' is required")
		return
	}

	isCached := false
	job, err := api.JobRepository.FindCached(req.JobID, req.Cluster, req.StartTime)
	if err != nil {
		// Not in cache, try main job table
		job, err = api.JobRepository.Find(req.JobID, req.Cluster, req.StartTime)
		if err != nil {
			cclog.Errorf("NATS job stop: finding job failed: %v", err)
			return
		}
	} else {
		isCached = true
	}

	if job.State != schema.JobStateRunning {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: job has already been stopped (state is: %s)",
			job.JobID, job.ID, job.Cluster, job.State)
		return
	}

	if job.StartTime > req.StopTime {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: stopTime %d must be >= startTime %d",
			job.JobID, job.ID, job.Cluster, req.StopTime, job.StartTime)
		return
	}

	if req.State != "" && !req.State.Valid() {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: invalid job state: %#v",
			job.JobID, job.ID, job.Cluster, req.State)
		return
	} else if req.State == "" {
		req.State = schema.JobStateCompleted
	}

	job.Duration = int32(req.StopTime - job.StartTime)
	job.State = req.State
	api.JobRepository.Mutex.Lock()
	defer api.JobRepository.Mutex.Unlock()

	// If the job is still in job_cache, transfer it to the job table first
	if isCached {
		newID, err := api.JobRepository.TransferCachedJobToMain(*job.ID)
		if err != nil {
			cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: transferring cached job failed: %v",
				job.JobID, *job.ID, job.Cluster, err)
			return
		}
		cclog.Infof("NATS: transferred cached job to main table: old id %d -> new id %d (jobId=%d)", *job.ID, newID, job.JobID)
		job.ID = &newID
	}

	if err := api.JobRepository.Stop(*job.ID, job.Duration, job.State, job.MonitoringStatus); err != nil {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: marking job as '%s' failed: %v",
			job.JobID, *job.ID, job.Cluster, job.State, err)
		return
	}

	cclog.Infof("NATS: archiving job (dbid: %d): cluster=%s, jobId=%d, user=%s, startTime=%d, duration=%d, state=%s",
		*job.ID, job.Cluster, job.JobID, job.User, job.StartTime, job.Duration, job.State)

	if job.MonitoringStatus == schema.MonitoringStatusDisabled {
		return
	}

	archiver.TriggerArchiving(job)
}

// processNodestateEvent extracts and processes node state data from the InfluxDB message.
// Updates node states in the repository for all nodes in the payload.
func (api *NatsAPI) processNodestateEvent(msg lp.CCMessage) {
	v, ok := msg.GetEventValue()
	if !ok {
		cclog.Errorf("Nodestate event is missing event field with JSON payload")
		return
	}

	var req UpdateNodeStatesRequest

	dec := json.NewDecoder(strings.NewReader(v))
	dec.DisallowUnknownFields()
	if err := dec.Decode(&req); err != nil {
		cclog.Errorf("NATS nodestate: parsing request failed: %v", err)
		return
	}

	repo := repository.GetNodeRepository()
	requestReceived := time.Now().Unix()

	updates := make([]repository.NodeStateUpdate, 0, len(req.Nodes))
	for _, node := range req.Nodes {
		state := determineState(node.States)
		nodeState := schema.NodeStateDB{
			TimeStamp:       requestReceived,
			NodeState:       state,
			CpusAllocated:   node.CpusAllocated,
			MemoryAllocated: node.MemoryAllocated,
			GpusAllocated:   node.GpusAllocated,
			HealthState:     schema.MonitoringStateFull,
			JobsRunning:     node.JobsRunning,
		}
		updates = append(updates, repository.NodeStateUpdate{
			Hostname:  node.Hostname,
			Cluster:   req.Cluster,
			NodeState: &nodeState,
		})
	}

	if err := repo.BatchUpdateNodeStates(updates); err != nil {
		cclog.Errorf("NATS nodestate: batch update for cluster %s failed: %v", req.Cluster, err)
	}

	cclog.Debugf("NATS nodestate: updated %d node states for cluster %s", len(req.Nodes), req.Cluster)
}

// handleNodeState processes node state update messages received via NATS using InfluxDB line protocol.
// The message must be in line protocol format with measurement="nodestate" and include:
//   - field "event" containing JSON payload (UpdateNodeStatesRequest)
//
// Example: nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[...]}" 1234567890000000000
func (api *NatsAPI) handleNodeState(subject string, data []byte) {
	if len(data) == 0 {
		cclog.Warnf("NATS %s: received empty message", subject)
		return
	}

	d := influx.NewDecoderWithBytes(data)

	for d.Next() {
		m, err := receivers.DecodeInfluxMessage(d)
		if err != nil {
			cclog.Errorf("NATS %s: failed to decode InfluxDB line protocol message: %v", subject, err)
			return
		}

		if !m.IsEvent() {
			cclog.Warnf("NATS %s: received non-event message, skipping", subject)
			continue
		}

		if m.Name() == "nodestate" {
			api.processNodestateEvent(m)
		} else {
			cclog.Warnf("NATS %s: unexpected measurement name '%s', expected 'nodestate'", subject, m.Name())
		}
	}
}

import (
"database/sql"
"encoding/json"
+  "maps"
"strings"
"sync"
"time"


	repo := repository.GetNodeRepository()
	requestReceived := time.Now().Unix()

	updates := make([]repository.NodeStateUpdate, 0, len(req.Nodes))
	for _, node := range req.Nodes {
		state := determineState(node.States)
		nodeState := schema.NodeStateDB{
			TimeStamp:       requestReceived,
			NodeState:       state,
			CpusAllocated:   node.CpusAllocated,
			MemoryAllocated: node.MemoryAllocated,
			GpusAllocated:   node.GpusAllocated,
			HealthState:     schema.MonitoringStateFull,
			JobsRunning:     node.JobsRunning,
		}
		updates = append(updates, repository.NodeStateUpdate{
			Hostname:  node.Hostname,
			Cluster:   req.Cluster,
			NodeState: &nodeState,
		})
	}

	if err := repo.BatchUpdateNodeStates(updates); err != nil {
		cclog.Errorf("NATS nodestate: batch update for cluster %s failed: %v", req.Cluster, err)
	}

	cclog.Debugf("NATS nodestate: updated %d node states for cluster %s", len(req.Nodes), req.Cluster)

	repo := repository.GetNodeRepository()
	requestReceived := time.Now().Unix()

	// Build nodeList per subcluster for health check
	m := make(map[string][]string)
	metricNames := make(map[string][]string)
	healthResults := make(map[string]metricstore.HealthCheckResult)

	for _, node := range req.Nodes {
		if sc, err := archive.GetSubClusterByNode(req.Cluster, node.Hostname); err == nil {
			m[sc] = append(m[sc], node.Hostname)
		}
	}

	for sc := range m {
		if sc != "" {
			metricList := archive.GetMetricConfigSubCluster(req.Cluster, sc)
			metricNames[sc] = metricListToNames(metricList)
		}
	}

	// Perform health check against metric store
	healthRepo, err := metricdispatch.GetHealthCheckRepo(req.Cluster)
	if err != nil {
		cclog.Warnf("NATS nodestate: no metric store for cluster %s, skipping health check: %v", req.Cluster, err)
	} else {
		for sc, nl := range m {
			if sc != "" {
				if results, err := healthRepo.HealthCheck(req.Cluster, nl, metricNames[sc]); err == nil {
					maps.Copy(healthResults, results)
				}
			}
		}
	}

	updates := make([]repository.NodeStateUpdate, 0, len(req.Nodes))
	for _, node := range req.Nodes {
		state := determineState(node.States)
		healthState := schema.MonitoringStateFailed
		var healthMetrics string
		if result, ok := healthResults[node.Hostname]; ok {
			healthState = result.State
			healthMetrics = result.HealthMetrics
		}
		nodeState := schema.NodeStateDB{
			TimeStamp:       requestReceived,
			NodeState:       state,
			CpusAllocated:   node.CpusAllocated,
			MemoryAllocated: node.MemoryAllocated,
			GpusAllocated:   node.GpusAllocated,
			HealthState:     healthState,
			HealthMetrics:   healthMetrics,
			JobsRunning:     node.JobsRunning,
		}
		updates = append(updates, repository.NodeStateUpdate{
			Hostname:  node.Hostname,
			Cluster:   req.Cluster,
			NodeState: &nodeState,
		})
	}

	if err := repo.BatchUpdateNodeStates(updates); err != nil {
		cclog.Errorf("NATS nodestate: batch update for cluster %s failed: %v", req.Cluster, err)
	}

	cclog.Debugf("NATS nodestate: updated %d node states for cluster %s", len(req.Nodes), req.Cluster)


	repo := repository.GetNodeRepository()
	requestReceived := time.Now().Unix()

	// Build nodeList per subcluster for health check
	m := make(map[string][]string)
	metricNames := make(map[string][]string)
	healthResults := make(map[string]metricstore.HealthCheckResult)

	for _, node := range req.Nodes {
		if sc, err := archive.GetSubClusterByNode(req.Cluster, node.Hostname); err == nil {
			m[sc] = append(m[sc], node.Hostname)
		}
	}

	for sc := range m {
		if sc != "" {
			metricList := archive.GetMetricConfigSubCluster(req.Cluster, sc)
			metricNames[sc] = metricListToNames(metricList)
		}
	}

	// Perform health check against metric store
	healthRepo, err := metricdispatch.GetHealthCheckRepo(req.Cluster)
	if err != nil {
		cclog.Warnf("NATS nodestate: no metric store for cluster %s, skipping health check: %v", req.Cluster, err)
	} else {
		for sc, nl := range m {
			if sc != "" {
				if results, err := healthRepo.HealthCheck(req.Cluster, nl, metricNames[sc]); err == nil {
					maps.Copy(healthResults, results)
				}
			}
		}
	}

	updates := make([]repository.NodeStateUpdate, 0, len(req.Nodes))
	for _, node := range req.Nodes {
		state := determineState(node.States)
		healthState := schema.MonitoringStateFailed
		var healthMetrics string
		if result, ok := healthResults[node.Hostname]; ok {
			healthState = result.State
			healthMetrics = result.HealthMetrics
		}
		nodeState := schema.NodeStateDB{
			TimeStamp:       requestReceived,
			NodeState:       state,
			CpusAllocated:   node.CpusAllocated,
			MemoryAllocated: node.MemoryAllocated,
			GpusAllocated:   node.GpusAllocated,
			HealthState:     healthState,
			HealthMetrics:   healthMetrics,
			JobsRunning:     node.JobsRunning,
		}
		updates = append(updates, repository.NodeStateUpdate{
			Hostname:  node.Hostname,
			Cluster:   req.Cluster,
			NodeState: &nodeState,
		})
	}

	if err := repo.BatchUpdateNodeStates(updates); err != nil {
		cclog.Errorf("NATS nodestate: batch update for cluster %s failed: %v", req.Cluster, err)
	}

	cclog.Debugf("NATS nodestate: updated %d node states for cluster %s", len(req.Nodes), req.Cluster)

// Copyright (C) NHR@FAU, University Erlangen-Nuremberg.
// All rights reserved. This file is part of cc-backend.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package api

import (
	"database/sql"
	"encoding/json"
	"maps"
	"strings"
	"sync"
	"time"

	"github.com/ClusterCockpit/cc-backend/internal/archiver"
	"github.com/ClusterCockpit/cc-backend/internal/config"
	"github.com/ClusterCockpit/cc-backend/internal/importer"
	"github.com/ClusterCockpit/cc-backend/internal/metricdispatch"
	"github.com/ClusterCockpit/cc-backend/internal/repository"
	"github.com/ClusterCockpit/cc-backend/pkg/archive"
	"github.com/ClusterCockpit/cc-backend/pkg/metricstore"
	cclog "github.com/ClusterCockpit/cc-lib/v2/ccLogger"
	lp "github.com/ClusterCockpit/cc-lib/v2/ccMessage"
	"github.com/ClusterCockpit/cc-lib/v2/nats"
	"github.com/ClusterCockpit/cc-lib/v2/receivers"
	"github.com/ClusterCockpit/cc-lib/v2/schema"
	influx "github.com/ClusterCockpit/cc-line-protocol/v2/lineprotocol"
)

// natsMessage wraps a raw NATS message with its subject for channel-based processing.
type natsMessage struct {
	subject string
	data    []byte
}

// NatsAPI provides NATS subscription-based handlers for Job and Node operations.
// It mirrors the functionality of the REST API but uses NATS messaging with
// InfluxDB line protocol as the message format.
//
// # Message Format
//
// All NATS messages use InfluxDB line protocol format (https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/)
// with the following structure:
//
//	measurement,tag1=value1,tag2=value2 field1=value1,field2=value2 timestamp
//
// # Job Events
//
// Job start/stop events use the "job" measurement with a "function" tag to distinguish operations:
//
//	job,function=start_job event="{...JSON payload...}" <timestamp>
//	job,function=stop_job event="{...JSON payload...}" <timestamp>
//
// The JSON payload in the "event" field follows the schema.Job or StopJobAPIRequest structure.
//
// Example job start message:
//
//	job,function=start_job event="{\"jobId\":1001,\"user\":\"testuser\",\"cluster\":\"testcluster\",...}" 1234567890000000000
//
// # Node State Events
//
// Node state updates use the "nodestate" measurement with cluster information:
//
//	nodestate event="{...JSON payload...}" <timestamp>
//
// The JSON payload follows the UpdateNodeStatesRequest structure.
//
// Example node state message:
//
//	nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[{\"hostname\":\"node01\",\"states\":[\"idle\"]}]}" 1234567890000000000
type NatsAPI struct {
	JobRepository *repository.JobRepository
	// RepositoryMutex protects job creation operations from race conditions
	// when checking for duplicate jobs during startJob calls.
	RepositoryMutex sync.Mutex
	// jobCh receives job event messages for processing by worker goroutines.
	jobCh chan natsMessage
	// nodeCh receives node state messages for processing by worker goroutines.
	nodeCh chan natsMessage
}

// NewNatsAPI creates a new NatsAPI instance with channel-based worker pools.
// Concurrency is configured via NATSConfig (defaults: JobConcurrency=8, NodeConcurrency=2).
func NewNatsAPI() *NatsAPI {
	jobConc := 8
	nodeConc := 2

	if s := config.Keys.APISubjects; s != nil {
		if s.JobConcurrency > 0 {
			jobConc = s.JobConcurrency
		}
		if s.NodeConcurrency > 0 {
			nodeConc = s.NodeConcurrency
		}
	}

	api := &NatsAPI{
		JobRepository: repository.GetJobRepository(),
		jobCh:         make(chan natsMessage, jobConc),
		nodeCh:        make(chan natsMessage, nodeConc),
	}

	// Start worker goroutines
	for range jobConc {
		go api.jobWorker()
	}
	for range nodeConc {
		go api.nodeWorker()
	}

	return api
}

// jobWorker processes job event messages from the job channel.
func (api *NatsAPI) jobWorker() {
	for msg := range api.jobCh {
		api.handleJobEvent(msg.subject, msg.data)
	}
}

// nodeWorker processes node state messages from the node channel.
func (api *NatsAPI) nodeWorker() {
	for msg := range api.nodeCh {
		api.handleNodeState(msg.subject, msg.data)
	}
}

// StartSubscriptions registers all NATS subscriptions for Job and Node APIs.
// Messages are delivered to buffered channels and processed by worker goroutines.
// Returns an error if the NATS client is not available or subscription fails.
func (api *NatsAPI) StartSubscriptions() error {
	client := nats.GetClient()
	if client == nil {
		cclog.Warn("NATS client not available, skipping API subscriptions")
		return nil
	}

	if config.Keys.APISubjects != nil {
		s := config.Keys.APISubjects

		if err := client.Subscribe(s.SubjectJobEvent, func(subject string, data []byte) {
			api.jobCh <- natsMessage{subject: subject, data: data}
		}); err != nil {
			return err
		}

		if err := client.Subscribe(s.SubjectNodeState, func(subject string, data []byte) {
			api.nodeCh <- natsMessage{subject: subject, data: data}
		}); err != nil {
			return err
		}

		cclog.Info("NATS API subscriptions started")
	}
	return nil
}

// processJobEvent routes job event messages to the appropriate handler based on the "function" tag.
// Validates that required tags and fields are present before processing.
func (api *NatsAPI) processJobEvent(msg lp.CCMessage) {
	function, ok := msg.GetTag("function")
	if !ok {
		cclog.Errorf("Job event is missing required tag 'function': measurement=%s", msg.Name())
		return
	}

	switch function {
	case "start_job":
		v, ok := msg.GetEventValue()
		if !ok {
			cclog.Errorf("Job start event is missing event field with JSON payload")
			return
		}
		api.handleStartJob(v)

	case "stop_job":
		v, ok := msg.GetEventValue()
		if !ok {
			cclog.Errorf("Job stop event is missing event field with JSON payload")
			return
		}
		api.handleStopJob(v)

	default:
		cclog.Warnf("Unknown job event function '%s', expected 'start_job' or 'stop_job'", function)
	}
}

// handleJobEvent processes job-related messages received via NATS using InfluxDB line protocol.
// The message must be in line protocol format with measurement="job" and include:
//   - tag "function" with value "start_job" or "stop_job"
//   - field "event" containing JSON payload (schema.Job or StopJobAPIRequest)
//
// Example: job,function=start_job event="{\"jobId\":1001,...}" 1234567890000000000
func (api *NatsAPI) handleJobEvent(subject string, data []byte) {
	if len(data) == 0 {
		cclog.Warnf("NATS %s: received empty message", subject)
		return
	}

	d := influx.NewDecoderWithBytes(data)

	for d.Next() {
		m, err := receivers.DecodeInfluxMessage(d)
		if err != nil {
			cclog.Errorf("NATS %s: failed to decode InfluxDB line protocol message: %v", subject, err)
			return
		}

		if !m.IsEvent() {
			cclog.Debugf("NATS %s: received non-event message, skipping", subject)
			continue
		}

		if m.Name() == "job" {
			api.processJobEvent(m)
		} else {
			cclog.Debugf("NATS %s: unexpected measurement name '%s', expected 'job'", subject, m.Name())
		}
	}
}

// handleStartJob processes job start messages received via NATS.
// The payload parameter contains JSON following the schema.Job structure.
// Jobs are validated, checked for duplicates, and inserted into the database.
func (api *NatsAPI) handleStartJob(payload string) {
	if payload == "" {
		cclog.Error("NATS start job: payload is empty")
		return
	}
	req := schema.Job{
		Shared:           "none",
		MonitoringStatus: schema.MonitoringStatusRunningOrArchiving,
	}

	dec := json.NewDecoder(strings.NewReader(payload))
	dec.DisallowUnknownFields()
	if err := dec.Decode(&req); err != nil {
		cclog.Errorf("NATS start job: parsing request failed: %v", err)
		return
	}

	cclog.Debugf("NATS start job: %s", req.GoString())
	req.State = schema.JobStateRunning

	if err := importer.SanityChecks(&req); err != nil {
		cclog.Errorf("NATS start job: sanity check failed: %v", err)
		return
	}

	var unlockOnce sync.Once
	api.RepositoryMutex.Lock()
	defer unlockOnce.Do(api.RepositoryMutex.Unlock)

	jobs, err := api.JobRepository.FindAll(&req.JobID, &req.Cluster, nil)
	if err != nil && err != sql.ErrNoRows {
		cclog.Errorf("NATS start job: checking for duplicate failed: %v", err)
		return
	}
	if err == nil {
		for _, job := range jobs {
			if (req.StartTime - job.StartTime) < secondsPerDay {
				cclog.Errorf("NATS start job: job with jobId %d, cluster %s already exists (dbid: %d)",
					req.JobID, req.Cluster, job.ID)
				return
			}
		}
	}

	// When tags are present, insert directly into the job table so that the
	// returned ID can be used with AddTagOrCreate (which queries the job table).
	var id int64
	if len(req.Tags) > 0 {
		id, err = api.JobRepository.StartDirect(&req)
	} else {
		id, err = api.JobRepository.Start(&req)
	}
	if err != nil {
		cclog.Errorf("NATS start job: insert into database failed: %v", err)
		return
	}
	unlockOnce.Do(api.RepositoryMutex.Unlock)

	for _, tag := range req.Tags {
		if _, err := api.JobRepository.AddTagOrCreate(nil, id, tag.Type, tag.Name, tag.Scope); err != nil {
			cclog.Errorf("NATS start job: adding tag to new job %d failed: %v", id, err)
			return
		}
	}

	cclog.Infof("NATS: new job (id: %d): cluster=%s, jobId=%d, user=%s, startTime=%d",
		id, req.Cluster, req.JobID, req.User, req.StartTime)
}

// handleStopJob processes job stop messages received via NATS.
// The payload parameter contains JSON following the StopJobAPIRequest structure.
// The job is marked as stopped in the database and archiving is triggered if monitoring is enabled.
func (api *NatsAPI) handleStopJob(payload string) {
	if payload == "" {
		cclog.Error("NATS stop job: payload is empty")
		return
	}
	var req StopJobAPIRequest

	dec := json.NewDecoder(strings.NewReader(payload))
	dec.DisallowUnknownFields()
	if err := dec.Decode(&req); err != nil {
		cclog.Errorf("NATS job stop: parsing request failed: %v", err)
		return
	}

	if req.JobID == nil {
		cclog.Errorf("NATS job stop: the field 'jobId' is required")
		return
	}

	isCached := false
	job, err := api.JobRepository.FindCached(req.JobID, req.Cluster, req.StartTime)
	if err != nil {
		// Not in cache, try main job table
		job, err = api.JobRepository.Find(req.JobID, req.Cluster, req.StartTime)
		if err != nil {
			cclog.Errorf("NATS job stop: finding job failed: %v", err)
			return
		}
	} else {
		isCached = true
	}

	if job.State != schema.JobStateRunning {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: job has already been stopped (state is: %s)",
			job.JobID, job.ID, job.Cluster, job.State)
		return
	}

	if job.StartTime > req.StopTime {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: stopTime %d must be >= startTime %d",
			job.JobID, job.ID, job.Cluster, req.StopTime, job.StartTime)
		return
	}

	if req.State != "" && !req.State.Valid() {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: invalid job state: %#v",
			job.JobID, job.ID, job.Cluster, req.State)
		return
	} else if req.State == "" {
		req.State = schema.JobStateCompleted
	}

	job.Duration = int32(req.StopTime - job.StartTime)
	job.State = req.State
	api.JobRepository.Mutex.Lock()
	defer api.JobRepository.Mutex.Unlock()

	// If the job is still in job_cache, transfer it to the job table first
	if isCached {
		newID, err := api.JobRepository.TransferCachedJobToMain(*job.ID)
		if err != nil {
			cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: transferring cached job failed: %v",
				job.JobID, *job.ID, job.Cluster, err)
			return
		}
		cclog.Infof("NATS: transferred cached job to main table: old id %d -> new id %d (jobId=%d)", *job.ID, newID, job.JobID)
		job.ID = &newID
	}

	if err := api.JobRepository.Stop(*job.ID, job.Duration, job.State, job.MonitoringStatus); err != nil {
		cclog.Errorf("NATS job stop: jobId %d (id %d) on %s: marking job as '%s' failed: %v",
			job.JobID, *job.ID, job.Cluster, job.State, err)
		return
	}

	cclog.Infof("NATS: archiving job (dbid: %d): cluster=%s, jobId=%d, user=%s, startTime=%d, duration=%d, state=%s",
		*job.ID, job.Cluster, job.JobID, job.User, job.StartTime, job.Duration, job.State)

	if job.MonitoringStatus == schema.MonitoringStatusDisabled {
		return
	}

	archiver.TriggerArchiving(job)
}

// processNodestateEvent extracts and processes node state data from the InfluxDB message.
// Updates node states in the repository for all nodes in the payload.
func (api *NatsAPI) processNodestateEvent(msg lp.CCMessage) {
	v, ok := msg.GetEventValue()
	if !ok {
		cclog.Errorf("Nodestate event is missing event field with JSON payload")
		return
	}

	var req UpdateNodeStatesRequest

	dec := json.NewDecoder(strings.NewReader(v))
	dec.DisallowUnknownFields()
	if err := dec.Decode(&req); err != nil {
		cclog.Errorf("NATS nodestate: parsing request failed: %v", err)
		return
	}

	repo := repository.GetNodeRepository()
	requestReceived := time.Now().Unix()

	updates := make([]repository.NodeStateUpdate, 0, len(req.Nodes))
	for _, node := range req.Nodes {
		state := determineState(node.States)
		nodeState := schema.NodeStateDB{
			TimeStamp:       requestReceived,
			NodeState:       state,
			CpusAllocated:   node.CpusAllocated,
			MemoryAllocated: node.MemoryAllocated,
			GpusAllocated:   node.GpusAllocated,
			HealthState:     schema.MonitoringStateFull,
			JobsRunning:     node.JobsRunning,
		}
		updates = append(updates, repository.NodeStateUpdate{
			Hostname:  node.Hostname,
			Cluster:   req.Cluster,
			NodeState: &nodeState,
		})
	}

	if err := repo.BatchUpdateNodeStates(updates); err != nil {
		cclog.Errorf("NATS nodestate: batch update for cluster %s failed: %v", req.Cluster, err)
	}

	cclog.Debugf("NATS nodestate: updated %d node states for cluster %s", len(req.Nodes), req.Cluster)
}

// handleNodeState processes node state update messages received via NATS using InfluxDB line protocol.
// The message must be in line protocol format with measurement="nodestate" and include:
//   - field "event" containing JSON payload (UpdateNodeStatesRequest)
//
// Example: nodestate event="{\"cluster\":\"testcluster\",\"nodes\":[...]}" 1234567890000000000
func (api *NatsAPI) handleNodeState(subject string, data []byte) {
	if len(data) == 0 {
		cclog.Warnf("NATS %s: received empty message", subject)
		return
	}

	d := influx.NewDecoderWithBytes(data)

	for d.Next() {
		m, err := receivers.DecodeInfluxMessage(d)
		if err != nil {
			cclog.Errorf("NATS %s: failed to decode InfluxDB line protocol message: %v", subject, err)
			return
		}

		if !m.IsEvent() {
			cclog.Warnf("NATS %s: received non-event message, skipping", subject)
			continue
		}

		if m.Name() == "nodestate" {
			api.processNodestateEvent(m)
		} else {
			cclog.Warnf("NATS %s: unexpected measurement name '%s', expected 'nodestate'", subject, m.Name())
		}
	}
}

+  // Build nodeList per subcluster for health check

+  metricNames := make(map[string][]string)

+  healthResults := make(map[string]metricstore.HealthCheckResult)

+    if sc, err := archive.GetSubClusterByNode(req.Cluster, node.Hostname); err == nil {

+      m[sc] = append(m[sc], node.Hostname)

+      metricList := archive.GetMetricConfigSubCluster(req.Cluster, sc)

+      metricNames[sc] = metricListToNames(metricList)

+  // Perform health check against metric store

+  healthRepo, err := metricdispatch.GetHealthCheckRepo(req.Cluster)

+    cclog.Warnf("NATS nodestate: no metric store for cluster %s, skipping health check: %v", req.Cluster, err)

+        if results, err := healthRepo.HealthCheck(req.Cluster, nl, metricNames[sc]); err == nil {

+          maps.Copy(healthResults, results)

+    healthState := schema.MonitoringStateFailed

+    if result, ok := healthResults[node.Hostname]; ok {

+      healthMetrics = result.HealthMetrics

-      HealthState:     schema.MonitoringStateFull,


The diagnostics show unused imports, likely because the LSP hasn't caught up yet. Let me verify by building.


Bash

Build to verify compilation


(Bash completed with no output)


Build succeeds. Now let me run the NATS tests.


go test ./internal/api/ -run TestNats -v 2>&1

Run NATS-related tests