fix: Add defer.close for all queries

This commit is contained in:
2026-03-11 05:04:20 +01:00
parent f9aa47ea1c
commit 5d3d77620e
12 changed files with 709 additions and 35 deletions

17
.entire/logs/entire.log Normal file
View File

@@ -0,0 +1,17 @@
{"time":"2026-03-10T21:13:26.857514+01:00","level":"INFO","msg":"session-start","component":"lifecycle","agent":"claude-code","event":"SessionStart","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","session_ref":"/Users/jan/.claude/projects/-Users-jan-prg-CC-cc-backend/28147033-ddc8-4056-b064-e0558fbc614e.jsonl"}
{"time":"2026-03-10T21:15:58.745665+01:00","level":"INFO","msg":"turn-start","component":"lifecycle","agent":"claude-code","event":"TurnStart","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","session_ref":"/Users/jan/.claude/projects/-Users-jan-prg-CC-cc-backend/28147033-ddc8-4056-b064-e0558fbc614e.jsonl"}
{"time":"2026-03-10T21:15:59.052384+01:00","level":"INFO","msg":"phase transition","component":"session","agent":"claude-code","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","event":"TurnStart","from":"","to":"active"}
{"time":"2026-03-10T21:15:59.247089+01:00","level":"INFO","msg":"initialized shadow session","component":"hooks","agent":"claude-code","session_id":"28147033-ddc8-4056-b064-e0558fbc614e"}
{"time":"2026-03-10T21:16:11.273515+01:00","level":"INFO","msg":"subagent started","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","component":"lifecycle","agent":"claude-code","event":"SubagentStart","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","tool_use_id":"toolu_019bWLMJ1nzVAT8Fg2W5qRmi","transcript":"/Users/jan/.claude/projects/-Users-jan-prg-CC-cc-backend/28147033-ddc8-4056-b064-e0558fbc614e.jsonl"}
{"time":"2026-03-10T21:17:50.542621+01:00","level":"INFO","msg":"subagent completed","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","component":"lifecycle","agent":"claude-code","event":"SubagentEnd","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","tool_use_id":"toolu_019bWLMJ1nzVAT8Fg2W5qRmi","agent_id":"a03483f2df2e8cef0"}
{"time":"2026-03-10T21:17:50.755441+01:00","level":"INFO","msg":"no file changes detected, skipping task checkpoint","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","component":"lifecycle","agent":"claude-code"}
{"time":"2026-03-10T21:28:03.798482+01:00","level":"INFO","msg":"turn-end","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","component":"lifecycle","agent":"claude-code","event":"TurnEnd","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","session_ref":"/Users/jan/.claude/projects/-Users-jan-prg-CC-cc-backend/28147033-ddc8-4056-b064-e0558fbc614e.jsonl"}
{"time":"2026-03-10T21:28:04.579301+01:00","level":"INFO","msg":"created shadow branch and committed changes","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","component":"checkpoint","agent":"claude-code","shadow_branch":"entire/70fea39-e3b0c4"}
{"time":"2026-03-10T21:28:04.579308+01:00","level":"INFO","msg":"checkpoint saved","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","component":"checkpoint","agent":"claude-code","strategy":"manual-commit","checkpoint_type":"session","checkpoint_count":1,"modified_files":3,"new_files":0,"deleted_files":0,"shadow_branch":"entire/70fea39-e3b0c4","branch_created":true}
{"time":"2026-03-10T21:28:04.579367+01:00","level":"INFO","msg":"phase transition","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","component":"session","agent":"claude-code","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","event":"TurnEnd","from":"active","to":"idle"}
{"time":"2026-03-10T21:28:04.656228+01:00","level":"INFO","msg":"session-end","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","component":"lifecycle","agent":"claude-code","event":"SessionEnd","session_id":"28147033-ddc8-4056-b064-e0558fbc614e"}
{"time":"2026-03-10T21:28:04.656276+01:00","level":"INFO","msg":"phase transition","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","component":"session","agent":"claude-code","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","event":"SessionStop","from":"idle","to":"ended"}
{"time":"2026-03-10T21:28:04.7209+01:00","level":"INFO","msg":"session-start","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","component":"lifecycle","agent":"claude-code","event":"SessionStart","session_id":"42401d2e-7d1c-4c0e-abe6-356cb2d48747","session_ref":"/Users/jan/.claude/projects/-Users-jan-prg-CC-cc-backend/42401d2e-7d1c-4c0e-abe6-356cb2d48747.jsonl"}
{"time":"2026-03-10T21:28:44.262904+01:00","level":"INFO","msg":"turn-end","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","component":"lifecycle","agent":"claude-code","event":"TurnEnd","session_id":"42401d2e-7d1c-4c0e-abe6-356cb2d48747","session_ref":"/Users/jan/.claude/projects/-Users-jan-prg-CC-cc-backend/42401d2e-7d1c-4c0e-abe6-356cb2d48747.jsonl"}
{"time":"2026-03-10T21:28:44.697919+01:00","level":"INFO","msg":"committed changes to shadow branch","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","component":"checkpoint","agent":"claude-code","shadow_branch":"entire/70fea39-e3b0c4"}
{"time":"2026-03-10T21:28:44.697926+01:00","level":"INFO","msg":"checkpoint saved","session_id":"28147033-ddc8-4056-b064-e0558fbc614e","component":"checkpoint","agent":"claude-code","strategy":"manual-commit","checkpoint_type":"session","checkpoint_count":1,"modified_files":3,"new_files":1,"deleted_files":0,"shadow_branch":"entire/70fea39-e3b0c4","branch_created":false}

View File

@@ -0,0 +1,29 @@
# Session Context
Session ID: 28147033-ddc8-4056-b064-e0558fbc614e
Commit Message: There are memory leaks in production on queries that trigger a full scan
## Prompts
### Prompt 1
There are memory leaks in production on queries that trigger a full scan. Review all queries for missing close and other bugs that could cause memory leaks in the sqlite-3 driver. Example for a query that triggers a massive allocation is a specific jobstate list with sorting on start_time on all jobs.
## Summary
All 31 `.Query()` calls now have proper `rows.Close()`. Build passes, tests pass, vet clean.
### Summary of changes
**`internal/repository/stats.go`** (7 fixes + 1 rewrite):
- Added `defer rows.Close()` + `rows.Err()` to `JobsStatsGrouped`, `JobCountGrouped`, `AddJobCountGrouped`, `jobsStatisticsHistogram`, `jobsDurationStatisticsHistogram`, `jobsMetricStatisticsHistogram`
- Rewrote `AddJobCount` to use `QueryRow()` instead of `Query()` (single-row result)
**`internal/repository/tags.go`** (5 fixes):
- Added `defer xrows.Close()` + `xrows.Err()` for `Queryx` in `CountTags`
- Added `defer rows.Close()` for second query in `CountTags`
- Added `defer rows.Close()` + `rows.Err()` to `GetTags`, `GetTagsDirect`, `getArchiveTags`
**`internal/repository/node.go`** (4 fixes + 1 rewrite):
- Added `defer rows.Close()` + `rows.Err()`, removed manual `rows.Close()` in `QueryNodes`, `QueryNodesWithMeta`, `CountStatesTimed`
- Rewrote `CountStates` to use `COUNT(*) ... GROUP BY` in SQL instead of fetching every row and counting in Go

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1 @@
There are memory leaks in production on queries that trigger a full scan. Review all queries for missing close and other bugs that could cause memory leaks in the sqlite-3 driver. Example for a query that triggers a massive allocation is a specific jobstate list with sorting on start_time on all jobs.

View File

@@ -0,0 +1,16 @@
All 31 `.Query()` calls now have proper `rows.Close()`. Build passes, tests pass, vet clean.
### Summary of changes
**`internal/repository/stats.go`** (7 fixes + 1 rewrite):
- Added `defer rows.Close()` + `rows.Err()` to `JobsStatsGrouped`, `JobCountGrouped`, `AddJobCountGrouped`, `jobsStatisticsHistogram`, `jobsDurationStatisticsHistogram`, `jobsMetricStatisticsHistogram`
- Rewrote `AddJobCount` to use `QueryRow()` instead of `Query()` (single-row result)
**`internal/repository/tags.go`** (5 fixes):
- Added `defer xrows.Close()` + `xrows.Err()` for `Queryx` in `CountTags`
- Added `defer rows.Close()` for second query in `CountTags`
- Added `defer rows.Close()` + `rows.Err()` to `GetTags`, `GetTagsDirect`, `getArchiveTags`
**`internal/repository/node.go`** (4 fixes + 1 rewrite):
- Added `defer rows.Close()` + `rows.Err()`, removed manual `rows.Close()` in `QueryNodes`, `QueryNodesWithMeta`, `CountStatesTimed`
- Rewrote `CountStates` to use `COUNT(*) ... GROUP BY` in SQL instead of fetching every row and counting in Go

View File

@@ -0,0 +1,114 @@
# Session Context
Session ID: 42401d2e-7d1c-4c0e-abe6-356cb2d48747
Commit Message: Implement the following plan:
# Fix Missing `rows.Close()` Memory Leaks
## Prompts
### Prompt 1
Implement the following plan:
# Fix Missing `rows.Close()` Memory Leaks in SQLite3 Queries
## Context
Production memory leaks traced to queries that do full table scans (e.g., job state list sorted by `start_time` on all jobs). The root cause is `sql.Rows` objects not being closed after query execution. In Go's `database/sql`, every `rows` returned by `.Query()` holds a database connection and associated buffers until `rows.Close()` is called. Without `defer rows.Close()`, these leak on every code path (both success and error returns).
## Findings
**22 total `.Query()` calls** across the repository layer. **15 have `defer rows.Close()`**. **7 do not** (listed below). Additionally, 1 `Queryx` call in `tags.go` is also missing close.
In `node.go`, `QueryNodes` and `QueryNodesWithMeta` have partial `rows.Close()` only in error paths but **not on the success path** and not via `defer`.
`CountStates` and `CountStatesTimed` in `node.go` also lack `defer rows.Close()` (same partial pattern as above for CountStates, none at all for CountStatesTimed).
## Changes Required
### 1. `internal/repository/stats.go` — 6 functions missing `defer rows.Close()`
Add `defer rows.Close()` immediately after the `if err != nil` check for each:
| Line | Function |
|------|----------|
| 233 | `JobsStatsGrouped` |
| 438 | `JobCountGrouped` |
| 494 | `AddJobCountGrouped` |
| 553 | `AddJobCount` |
| 753 | `jobsStatisticsHistogram` |
| 821 | `jobsDurationStatisticsHistogram` |
| 946 | `jobsMetricStatisticsHistogram` |
Pattern — after each `Query()` error check, add:
```go
rows, err := query.RunWith(r.DB).Query()
if err != nil {
...
return nil, err
}
defer rows.Close() // <-- ADD THIS
```
### 2. `internal/repository/tags.go` — 2 leaks in `CountTags()`
**Line 282**: `xrows` from `r.DB.Queryx(...)` — add `defer xrows.Close()` after error check.
**Line 333**: `rows` from `q.RunWith(r.stmtCache).Query()` — add `defer rows.Close()` after error check.
### 3. `internal/repository/tags.go` — 3 leaks in `GetTags`, `GetTagsDirect`, `getArchiveTags`
**Line 508** (`GetTags`): add `defer rows.Close()` after error check.
**Line 541** (`GetTagsDirect`): add `defer rows.Close()` after error check.
**Line 579** (`getArchiveTags`): add `defer rows.Close()` after error check.
### 4. `internal/repository/node.go` — 4 functions missing `defer rows.Close()`
**Line 363** (`QueryNodes`): Replace the manual `rows.Close()` in the error path with `defer rows.Close()` immediately after the error check. Remove the explicit `rows.Close()` call on line 375.
**Line 412** (`QueryNodesWithMeta`): Same pattern — add `defer rows.Close()` after error check, remove explicit `rows.Close()` on line 427.
**Line 558** (`CountStates`): Add `defer rows.Close()` after error check. Remove explicit `rows.Close()` on line 569.
**Line 620** (`CountStatesTimed`): Add `defer rows.Close()` after error check. Remove explicit `rows.Close()` on line 633.
## Summary of All Edits
| File | Function | Action |
|------|----------|--------|
| `stats.go:237` | `JobsStatsGrouped` | Add `defer rows.Close()` |
| `stats.go:442` | `JobCountGrouped` | Add `defer rows.Close()` |
| `stats.go:498` | `AddJobCountGrouped` | Add `defer rows.Close()` |
| `stats.go:557` | `AddJobCount` | Add `defer rows.Close()` |
| `stats.go:757` | `jobsStatisticsHistogram` | Add `defer rows.Close()` |
| `stats.go:825` | `jobsDurationStatisticsHistogram` | Add `defer rows.Close()` |
| `stats.go:950` | `jobsMetricStatisticsHistogram` | Add `defer rows.Close()` |
| `tags.go:284` | `CountTags` (xrows) | Add `defer xrows.Close()` |
| `tags.go:336` | `CountTags` (rows) | Add `defer rows.Close()` |
| `tags.go:513` | `GetTags` | Add `defer rows.Close()` |
| `tags.go:546` | `GetTagsDirect` | Add `defer rows.Close()` |
| `tags.go:584` | `getArchiveTags` | Add `defer rows.Close()` |
| `node.go:368` | `QueryNodes` | Add `defer rows.Close()`, remove manual close on L375 |
| `node.go:417` | `QueryNodesWithMeta` | Add `defer rows.Close()`, remove manual close on L427 |
| `node.go:563` | `CountStates` | Add `defer rows.Close()`, remove manual close on L569 |
| `node.go:625` | `CountStatesTimed` | Add `defer rows.Close()`, remove manual close on L633 |
## Verification
1. `go build ./...` — ensure it compiles
2. `go test ./internal/repository/...` — run repository tests
3. `go vet ./internal/repository/...` — static analysis
If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-CC-cc-backend/28147033-ddc8-4056-b064-e0558fbc614e.jsonl
## Summary
All changes are already in place and verified:
- **Build**: passes
- **go vet**: passes
- **Tests**: pass
The changes are currently unstaged. All 16 locations from the plan have been fixed — `defer rows.Close()` added after each `Query()` error check, manual `rows.Close()` calls in error paths removed, and `rows.Err()` checks added where missing. The `AddJobCount` function was also simplified from `Query` + loop to `QueryRow().Scan()` since it returns a single value.

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,93 @@
Implement the following plan:
# Fix Missing `rows.Close()` Memory Leaks in SQLite3 Queries
## Context
Production memory leaks traced to queries that do full table scans (e.g., job state list sorted by `start_time` on all jobs). The root cause is `sql.Rows` objects not being closed after query execution. In Go's `database/sql`, every `rows` returned by `.Query()` holds a database connection and associated buffers until `rows.Close()` is called. Without `defer rows.Close()`, these leak on every code path (both success and error returns).
## Findings
**22 total `.Query()` calls** across the repository layer. **15 have `defer rows.Close()`**. **7 do not** (listed below). Additionally, 1 `Queryx` call in `tags.go` is also missing close.
In `node.go`, `QueryNodes` and `QueryNodesWithMeta` have partial `rows.Close()` only in error paths but **not on the success path** and not via `defer`.
`CountStates` and `CountStatesTimed` in `node.go` also lack `defer rows.Close()` (same partial pattern as above for CountStates, none at all for CountStatesTimed).
## Changes Required
### 1. `internal/repository/stats.go` — 6 functions missing `defer rows.Close()`
Add `defer rows.Close()` immediately after the `if err != nil` check for each:
| Line | Function |
|------|----------|
| 233 | `JobsStatsGrouped` |
| 438 | `JobCountGrouped` |
| 494 | `AddJobCountGrouped` |
| 553 | `AddJobCount` |
| 753 | `jobsStatisticsHistogram` |
| 821 | `jobsDurationStatisticsHistogram` |
| 946 | `jobsMetricStatisticsHistogram` |
Pattern — after each `Query()` error check, add:
```go
rows, err := query.RunWith(r.DB).Query()
if err != nil {
...
return nil, err
}
defer rows.Close() // <-- ADD THIS
```
### 2. `internal/repository/tags.go` — 2 leaks in `CountTags()`
**Line 282**: `xrows` from `r.DB.Queryx(...)` — add `defer xrows.Close()` after error check.
**Line 333**: `rows` from `q.RunWith(r.stmtCache).Query()` — add `defer rows.Close()` after error check.
### 3. `internal/repository/tags.go` — 3 leaks in `GetTags`, `GetTagsDirect`, `getArchiveTags`
**Line 508** (`GetTags`): add `defer rows.Close()` after error check.
**Line 541** (`GetTagsDirect`): add `defer rows.Close()` after error check.
**Line 579** (`getArchiveTags`): add `defer rows.Close()` after error check.
### 4. `internal/repository/node.go` — 4 functions missing `defer rows.Close()`
**Line 363** (`QueryNodes`): Replace the manual `rows.Close()` in the error path with `defer rows.Close()` immediately after the error check. Remove the explicit `rows.Close()` call on line 375.
**Line 412** (`QueryNodesWithMeta`): Same pattern — add `defer rows.Close()` after error check, remove explicit `rows.Close()` on line 427.
**Line 558** (`CountStates`): Add `defer rows.Close()` after error check. Remove explicit `rows.Close()` on line 569.
**Line 620** (`CountStatesTimed`): Add `defer rows.Close()` after error check. Remove explicit `rows.Close()` on line 633.
## Summary of All Edits
| File | Function | Action |
|------|----------|--------|
| `stats.go:237` | `JobsStatsGrouped` | Add `defer rows.Close()` |
| `stats.go:442` | `JobCountGrouped` | Add `defer rows.Close()` |
| `stats.go:498` | `AddJobCountGrouped` | Add `defer rows.Close()` |
| `stats.go:557` | `AddJobCount` | Add `defer rows.Close()` |
| `stats.go:757` | `jobsStatisticsHistogram` | Add `defer rows.Close()` |
| `stats.go:825` | `jobsDurationStatisticsHistogram` | Add `defer rows.Close()` |
| `stats.go:950` | `jobsMetricStatisticsHistogram` | Add `defer rows.Close()` |
| `tags.go:284` | `CountTags` (xrows) | Add `defer xrows.Close()` |
| `tags.go:336` | `CountTags` (rows) | Add `defer rows.Close()` |
| `tags.go:513` | `GetTags` | Add `defer rows.Close()` |
| `tags.go:546` | `GetTagsDirect` | Add `defer rows.Close()` |
| `tags.go:584` | `getArchiveTags` | Add `defer rows.Close()` |
| `node.go:368` | `QueryNodes` | Add `defer rows.Close()`, remove manual close on L375 |
| `node.go:417` | `QueryNodesWithMeta` | Add `defer rows.Close()`, remove manual close on L427 |
| `node.go:563` | `CountStates` | Add `defer rows.Close()`, remove manual close on L569 |
| `node.go:625` | `CountStatesTimed` | Add `defer rows.Close()`, remove manual close on L633 |
## Verification
1. `go build ./...` — ensure it compiles
2. `go test ./internal/repository/...` — run repository tests
3. `go vet ./internal/repository/...` — static analysis
If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: /Users/jan/.claude/projects/-Users-jan-prg-CC-cc-backend/28147033-ddc8-4056-b064-e0558fbc614e.jsonl

View File

@@ -0,0 +1,7 @@
All changes are already in place and verified:
- **Build**: passes
- **go vet**: passes
- **Tests**: pass
The changes are currently unstaged. All 16 locations from the plan have been fixed — `defer rows.Close()` added after each `Query()` error check, manual `rows.Close()` calls in error paths removed, and `rows.Err()` checks added where missing. The `AddJobCount` function was also simplified from `Query` + loop to `QueryRow().Scan()` since it returns a single value.