Old event rows nobody can reach any more are deleted in small nightly batches #12026

Old event rows are now deleted in small batches when no reader can reach them any more. The 15-second sampler tick that already prunes machine_sample now also runs pruneEvents once per tick.

The keep rule (src/system/retention_events.ts). The header names every reader of event and how far back it reads, in three groups:

• Readers limited by age. The longest is flow/estimate_base.ts at 90 days. KEEP_MS is 180 days, twice that.
• Feeds, audit views and replay cursors. These page back through everything: the events page and API, the MCP events tool, the TUI screen, Atom, the home timeline, SSE Last-Event-ID replay and cloud secret reads. The rule now sets how far back they go: they answer the last 180 days, and nothing newer changes.
• State read back out of the log at any age. Examples: repo.detached, user.created, the keeper's notes it says only once, issue and pull timelines, assignees, splits, the last push. Their kinds are listed in KEPT_PREFIXES and KEPT_KINDS and are never pruned.

A row is deleted only if it is older than 180 days, its kind is not kept, and it has no webhook delivery still pending. Deleting an age-only cut, as the issue sketched, would have changed the answers of the readers in the third group.

pruneEvents(db, now, rowCap)

• Returns {deleted, more, lockMs}.
• It walks rows oldest-first from where the last batch stopped. Deletes run as 1,000-row transactions, and lockMs is the longest single hold on the write lock.
• When it reaches a row inside the 180 days, the pass ends and waits a day before starting again.
• The tick asks for 1,000 rows (TICK_ROWS), a few ms each time. At that rate 2.4M old rows take about ten hours.
• If the database is busy, the tick skips the prune and tries again on the next one.

Migration. webhook_delivery(event_id) gets an index. Deliveries are deleted along with their event, and without the index each deleted event scanned the whole deliveries table.

Tests (test/retention-events.test.ts)

• Seeds old and new rows, including kept kinds and a pending delivery. Feeds, replay, estimate, the security window, the keeper's notes, splits and last push return the same rows before and after.
• Capped batches and the one-day wait.
• One 5,000-row batch holds the lock under 50 ms. It is split into 1,000-row transactions: as one transaction it took about 45 ms in this sandbox, too close to the limit.
• A real sampler tick prunes an old row and keeps a new one.

Passing: bun scripts/gate.ts, bun x tsc --noEmit, the new test, test/mcp-size.test.ts, the sampler, schema and stall-scan tests, and the events-page, SSE and webhook tests. scripts/fast.ts named 1,668 test files because of the migration, so the full suite was left to the forge's check.

Not done

• I couldn't see production's row counts by kind. If most of the 578 MB is issue., pull. or push rows, this frees little, and a follow-up would narrow the kept kinds, starting with issue.commented.
• No vacuum, so the file only shrinks once SQLite reuses the freed pages.
• I didn't call keep_plan or estimate: neither tool was available in this session.

🤖 Generated with Claude Code

Closes #12008

Old event rows are now deleted in small batches when no reader can reach them any more. The 15-second sampler tick that already prunes machine_sample now also runs pruneEvents once per tick.

The keep rule (src/system/retention_events.ts). The header names every reader of event and how far back it reads, in three groups:

• Readers limited by age. The longest is flow/estimate_base.ts at 90 days. KEEP_MS is 180 days, twice that.
• Feeds, audit views and replay cursors. These page back through everything: the events page and API, the MCP events tool, the TUI screen, Atom, the home timeline, SSE Last-Event-ID replay and cloud secret reads. The rule now sets how far back they go: they answer the last 180 days, and nothing newer changes.
• State read back out of the log at any age. Examples: repo.detached, user.created, the keeper's notes it says only once, issue and pull timelines, assignees, splits, the last push. Their kinds are listed in KEPT_PREFIXES and KEPT_KINDS and are never pruned.

A row is deleted only if it is older than 180 days, its kind is not kept, and it has no webhook delivery still pending. Deleting an age-only cut, as the issue sketched, would have changed the answers of the readers in the third group.

pruneEvents(db, now, rowCap)

• Returns {deleted, more, lockMs}.
• It walks rows oldest-first from where the last batch stopped. Deletes run as 1,000-row transactions, and lockMs is the longest single hold on the write lock.
• When it reaches a row inside the 180 days, the pass ends and waits a day before starting again.
• The tick asks for 1,000 rows (TICK_ROWS), a few ms each time. At that rate 2.4M old rows take about ten hours.
• If the database is busy, the tick skips the prune and tries again on the next one.

Migration. webhook_delivery(event_id) gets an index. Deliveries are deleted along with their event, and without the index each deleted event scanned the whole deliveries table.

Tests (test/retention-events.test.ts)

• Seeds old and new rows, including kept kinds and a pending delivery. Feeds, replay, estimate, the security window, the keeper's notes, splits and last push return the same rows before and after.
• Capped batches and the one-day wait.
• One 5,000-row batch holds the lock under 50 ms. It is split into 1,000-row transactions: as one transaction it took about 45 ms in this sandbox, too close to the limit.
• A real sampler tick prunes an old row and keeps a new one.

Passing: bun scripts/gate.ts, bun x tsc --noEmit, the new test, test/mcp-size.test.ts, the sampler, schema and stall-scan tests, and the events-page, SSE and webhook tests. scripts/fast.ts named 1,668 test files because of the migration, so the full suite was left to the forge's check.

Not done

• I couldn't see production's row counts by kind. If most of the 578 MB is issue., pull. or push rows, this frees little, and a follow-up would narrow the kept kinds, starting with issue.commented.
• No vacuum, so the file only shrinks once SQLite reuses the freed pages.
• I didn't call keep_plan or estimate: neither tool was available in this session.

🤖 Generated with Claude Code

Old event rows are now deleted in small batches when no reader can reach them any more. The 15-second sampler tick that already prunes machine_sample now also runs pruneEvents once per tick.

The keep rule (src/system/retention_events.ts). The header names every reader of event and how far back it reads, in three groups:

• Readers limited by age. The longest is flow/estimate_base.ts at 90 days. KEEP_MS is 180 days, twice that.
• Feeds, audit views and replay cursors. These page back through everything: the events page and API, the MCP events tool, the TUI screen, Atom, the home timeline, SSE Last-Event-ID replay and cloud secret reads. The rule now sets how far back they go: they answer the last 180 days, and nothing newer changes.
• State read back out of the log at any age. Examples: repo.detached, user.created, the keeper's notes it says only once, issue and pull timelines, assignees, splits, the last push. Their kinds are listed in KEPT_PREFIXES and KEPT_KINDS and are never pruned.

A row is deleted only if it is older than 180 days, its kind is not kept, and it has no webhook delivery still pending. Deleting an age-only cut, as the issue sketched, would have changed the answers of the readers in the third group.

pruneEvents(db, now, rowCap)

• Returns {deleted, more, lockMs}.
• It walks rows oldest-first from where the last batch stopped. Deletes run as 1,000-row transactions, and lockMs is the longest single hold on the write lock.
• When it reaches a row inside the 180 days, the pass ends and waits a day before starting again.
• The tick asks for 1,000 rows (TICK_ROWS), a few ms each time. At that rate 2.4M old rows take about ten hours.
• If the database is busy, the tick skips the prune and tries again on the next one.

Migration. webhook_delivery(event_id) gets an index. Deliveries are deleted along with their event, and without the index each deleted event scanned the whole deliveries table.

Tests (test/retention-events.test.ts)

• Seeds old and new rows, including kept kinds and a pending delivery. Feeds, replay, estimate, the security window, the keeper's notes, splits and last push return the same rows before and after.
• Capped batches and the one-day wait.
• One 5,000-row batch holds the lock under 50 ms. It is split into 1,000-row transactions: as one transaction it took about 45 ms in the sandbox, too close to the limit.
• A real sampler tick prunes an old row and keeps a new one.

The second commit fixes the gate failure: it rewraps three comment lines to 100 columns and adds the Try: line the gate requires. Passing here: the gate run against the merge base, bun x tsc --noEmit, and the retention, sampler and test/mcp-size.test.ts tests. Earlier in the run the schema, stall-scan, events-page, SSE and webhook tests also passed. The migration makes scripts/fast.ts name most of the suite, so the full run is left to the forge's check.

Not done

• I couldn't see production's row counts by kind. If most of the 578 MB is issue., pull. or push rows, this frees little, and a follow-up would narrow the kept kinds, starting with issue.commented.
• No vacuum, so the file only shrinks once SQLite reuses the freed pages.
• keep_plan and estimate weren't available in this session, so the issue has no plan or estimate recorded.

Try: open /events on the preview. The newest events still list, and rows older than 180 days of windowed kinds, such as secret.read, are gone after a few sampler ticks.

🤖 Generated with Claude Code