mirror of
https://github.com/ClusterCockpit/cc-backend
synced 2026-03-20 23:07:29 +01:00
Remove tracked .entire/metadata/ files from git
These conversation transcript files were committed before the gitignore rule existed. They are now properly ignored via .entire/.gitignore. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,29 +0,0 @@
|
||||
# Session Context
|
||||
|
||||
Session ID: 28147033-ddc8-4056-b064-e0558fbc614e
|
||||
Commit Message: There are memory leaks in production on queries that trigger a full scan
|
||||
|
||||
## Prompts
|
||||
|
||||
### Prompt 1
|
||||
|
||||
There are memory leaks in production on queries that trigger a full scan. Review all queries for missing close and other bugs that could cause memory leaks in the sqlite-3 driver. Example for a query that triggers a massive allocation is a specific jobstate list with sorting on start_time on all jobs.
|
||||
|
||||
## Summary
|
||||
|
||||
All 31 `.Query()` calls now have proper `rows.Close()`. Build passes, tests pass, vet clean.
|
||||
|
||||
### Summary of changes
|
||||
|
||||
**`internal/repository/stats.go`** (7 fixes + 1 rewrite):
|
||||
- Added `defer rows.Close()` + `rows.Err()` to `JobsStatsGrouped`, `JobCountGrouped`, `AddJobCountGrouped`, `jobsStatisticsHistogram`, `jobsDurationStatisticsHistogram`, `jobsMetricStatisticsHistogram`
|
||||
- Rewrote `AddJobCount` to use `QueryRow()` instead of `Query()` (single-row result)
|
||||
|
||||
**`internal/repository/tags.go`** (5 fixes):
|
||||
- Added `defer xrows.Close()` + `xrows.Err()` for `Queryx` in `CountTags`
|
||||
- Added `defer rows.Close()` for second query in `CountTags`
|
||||
- Added `defer rows.Close()` + `rows.Err()` to `GetTags`, `GetTagsDirect`, `getArchiveTags`
|
||||
|
||||
**`internal/repository/node.go`** (4 fixes + 1 rewrite):
|
||||
- Added `defer rows.Close()` + `rows.Err()`, removed manual `rows.Close()` in `QueryNodes`, `QueryNodesWithMeta`, `CountStatesTimed`
|
||||
- Rewrote `CountStates` to use `COUNT(*) ... GROUP BY` in SQL instead of fetching every row and counting in Go
|
||||
File diff suppressed because one or more lines are too long
@@ -1 +0,0 @@
|
||||
There are memory leaks in production on queries that trigger a full scan. Review all queries for missing close and other bugs that could cause memory leaks in the sqlite-3 driver. Example for a query that triggers a massive allocation is a specific jobstate list with sorting on start_time on all jobs.
|
||||
@@ -1,16 +0,0 @@
|
||||
All 31 `.Query()` calls now have proper `rows.Close()`. Build passes, tests pass, vet clean.
|
||||
|
||||
### Summary of changes
|
||||
|
||||
**`internal/repository/stats.go`** (7 fixes + 1 rewrite):
|
||||
- Added `defer rows.Close()` + `rows.Err()` to `JobsStatsGrouped`, `JobCountGrouped`, `AddJobCountGrouped`, `jobsStatisticsHistogram`, `jobsDurationStatisticsHistogram`, `jobsMetricStatisticsHistogram`
|
||||
- Rewrote `AddJobCount` to use `QueryRow()` instead of `Query()` (single-row result)
|
||||
|
||||
**`internal/repository/tags.go`** (5 fixes):
|
||||
- Added `defer xrows.Close()` + `xrows.Err()` for `Queryx` in `CountTags`
|
||||
- Added `defer rows.Close()` for second query in `CountTags`
|
||||
- Added `defer rows.Close()` + `rows.Err()` to `GetTags`, `GetTagsDirect`, `getArchiveTags`
|
||||
|
||||
**`internal/repository/node.go`** (4 fixes + 1 rewrite):
|
||||
- Added `defer rows.Close()` + `rows.Err()`, removed manual `rows.Close()` in `QueryNodes`, `QueryNodesWithMeta`, `CountStatesTimed`
|
||||
- Rewrote `CountStates` to use `COUNT(*) ... GROUP BY` in SQL instead of fetching every row and counting in Go
|
||||
@@ -1,40 +0,0 @@
|
||||
# Session Context
|
||||
|
||||
Session ID: 42401d2e-7d1c-4c0e-abe6-356cb2d48747
|
||||
Commit Message: There also have to be bugs in jobQuery.go . Especially the following que
|
||||
|
||||
## Prompts
|
||||
|
||||
### Prompt 1
|
||||
|
||||
There also have to be bugs in jobQuery.go . Especially the following query triggers the memory leak: SELECT * FROM job WHERE job.job_state IN ("completed", "running", "failed") ORDER BY job.start_time DESC LIMIT 1 OFFSET 10; Dig deeper to find the cause. Also investigate why no existing index is used for this query.
|
||||
|
||||
## Summary
|
||||
|
||||
All passes. Here's a summary of findings and fixes:
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
### Bug 1: Missing `defer rows.Close()` in `user.go:GetLdapUsernames()` (line 100)
|
||||
**Fixed.** Every call leaked a database connection and its buffers.
|
||||
|
||||
### Bug 2: SQLite never runs `ANALYZE` — wrong index selection
|
||||
**Fixed.** Added `ANALYZE` to `setupSqlite()` in `dbConnection.go`.
|
||||
|
||||
Without statistics in `sqlite_stat1`, SQLite's query planner **guessed** and chose `jobs_jobstate_energy` (a terrible index for this query) instead of scanning `jobs_starttime` in reverse. The result:
|
||||
|
||||
| | Before ANALYZE | After ANALYZE |
|
||||
|---|---|---|
|
||||
| Index used | `jobs_jobstate_energy` | `jobs_starttime` |
|
||||
| Sort | **TEMP B-TREE** (materializes ALL rows) | None (index order) |
|
||||
| Memory | Proportional to total matching rows | Constant |
|
||||
| I/O | Full scan of all matching rows | Stops at OFFSET+LIMIT |
|
||||
|
||||
### Bug 3: `IN` clause + `ORDER BY` is fundamentally incompatible with composite indexes
|
||||
|
||||
Even with the "correct" index `(job_state, start_time)`, SQLite **cannot** merge-sort across 3 separate index range scans for `IN ('completed','running','failed')`. It always falls back to a temp B-tree sort. The only efficient plan is to use the standalone `jobs_starttime` index — which SQLite does automatically **after ANALYZE** because it realizes the 3 states cover virtually all rows, making the WHERE clause nearly a no-op.
|
||||
|
||||
### Observation: 79 indexes on the `job` table
|
||||
This is excessive and actively harmful — it confuses the query planner (especially without ANALYZE) and slows writes. The `jobs_jobstate_starttime` index from migration 08 is also missing from the actual DB (only the 3-column `jobs_jobstate_starttime_duration` exists). This is worth investigating separately but is a schema/migration concern, not a code bug.
|
||||
File diff suppressed because one or more lines are too long
@@ -1 +0,0 @@
|
||||
There also have to be bugs in jobQuery.go . Especially the following query triggers the memory leak: SELECT * FROM job WHERE job.job_state IN ("completed", "running", "failed") ORDER BY job.start_time DESC LIMIT 1 OFFSET 10; Dig deeper to find the cause. Also investigate why no existing index is used for this query.
|
||||
@@ -1,27 +0,0 @@
|
||||
All passes. Here's a summary of findings and fixes:
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
### Bug 1: Missing `defer rows.Close()` in `user.go:GetLdapUsernames()` (line 100)
|
||||
**Fixed.** Every call leaked a database connection and its buffers.
|
||||
|
||||
### Bug 2: SQLite never runs `ANALYZE` — wrong index selection
|
||||
**Fixed.** Added `ANALYZE` to `setupSqlite()` in `dbConnection.go`.
|
||||
|
||||
Without statistics in `sqlite_stat1`, SQLite's query planner **guessed** and chose `jobs_jobstate_energy` (a terrible index for this query) instead of scanning `jobs_starttime` in reverse. The result:
|
||||
|
||||
| | Before ANALYZE | After ANALYZE |
|
||||
|---|---|---|
|
||||
| Index used | `jobs_jobstate_energy` | `jobs_starttime` |
|
||||
| Sort | **TEMP B-TREE** (materializes ALL rows) | None (index order) |
|
||||
| Memory | Proportional to total matching rows | Constant |
|
||||
| I/O | Full scan of all matching rows | Stops at OFFSET+LIMIT |
|
||||
|
||||
### Bug 3: `IN` clause + `ORDER BY` is fundamentally incompatible with composite indexes
|
||||
|
||||
Even with the "correct" index `(job_state, start_time)`, SQLite **cannot** merge-sort across 3 separate index range scans for `IN ('completed','running','failed')`. It always falls back to a temp B-tree sort. The only efficient plan is to use the standalone `jobs_starttime` index — which SQLite does automatically **after ANALYZE** because it realizes the 3 states cover virtually all rows, making the WHERE clause nearly a no-op.
|
||||
|
||||
### Observation: 79 indexes on the `job` table
|
||||
This is excessive and actively harmful — it confuses the query planner (especially without ANALYZE) and slows writes. The `jobs_jobstate_starttime` index from migration 08 is also missing from the actual DB (only the 3-column `jobs_jobstate_starttime_duration` exists). This is worth investigating separately but is a schema/migration concern, not a code bug.
|
||||
Reference in New Issue
Block a user