Public: anyone can read this; the crew can change it.

Repository pages: two indexes and four query fixes take 17% off their SQL; and one-file-per-repository becomes the archive format, not the live storage #12961

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

This adds the two indexes and narrows the four statements, but I could not measure the "27-view replay at or under 200 ms" done-when. That replay needs the owner's database copy, which this sandbox cannot reach. bun scripts/gate.ts and tsc pass.

Notes

shipped production is serving it deploy 83f83cd

opened this

This adds the two indexes and narrows the four statements, but I could not measure the "27-view replay at or under 200 ms" done-when. That replay needs the owner's database copy, which this sandbox cannot reach. bun scripts/gate.ts and tsc pass. The 167 tests I ran pass; those were the new test, hot-reads-indexed, pages-budget and the bugs, from-app, landing-line, alarm-lane and usage files, not the full suite.

What changed

  • Migration: 20261115100000-usage-issue-review-approved.ts adds usage_issue and review_approved_at, both partial, as in the issue. Its pin is committed. Undo is two DROP INDEX.
  • flow/landing_line.ts: the "latest approval per pull" subquery now reads only the approved reviews of this repository's issues. Before, it grouped every approved review in the forge.
  • flow/alarm_lane.ts lastMerge: it now takes the repository's issue ids first, then looks up their pulls. Before, it joined pull to issue and could walk every pull.
  • core/from_app.ts: (?1 IS NULL OR i.repo_id = ?1) is now two statements, i.repo_id = ?1 and ?1 IS NULL. The OR could not use an index.
  • flow/bugs.ts: openBugs takes an optional repoId, built the same way. Existing callers are unchanged.
  • test/repo-bound-reads.test.ts: a plan-based rule, in the style of hot-reads-indexed. It checks the plans of issueUsage, landingLine, unansweredFromApp, openBugs({repoId}) and forgeFailed. A repository page's statements may not scan a table that holds every repository's rows (issue, pull, review, usage, comment, run, event and the like) without an index. It also asserts that usage_issue and review_approved_at appear in the plans.

What the test proves and does not

  • usage scan: the plan went from SCAN usage to SEARCH usage USING INDEX usage_issue. The test fails without the migration here.
  • review approvals: the old plan walked review by review_issue and grouped through an automatic index. The test fails without the migration here too.
  • From-app, bugs, lastMerge: these three tests pass on both old and new code against the empty test database, so they do not prove the narrowing. The plans on the real copy are the evidence.
  • forgeFailed and lastMerge: the test's dummy failed-check run (summary x) probably returns before lastMerge runs, so it may not reach that statement at all.
  • EXPLAIN QUERY PLAN: I did not paste before/after plans for all four statements into this description.

Not done

  • Item 3, archive/carry export: the issue says to open it separately once this lands.
  • Splitting the live database: per Finding 1, I did not touch it.

To check: on the copy, run EXPLAIN QUERY PLAN on the four statements with a real repository id, then replay the 27 views.

Closes #12960

3 earlier comments · 1 event
commented

This adds the two indexes and narrows the four statements, but I could not measure the "27-view replay at or under 200 ms" done-when. That replay needs the owner's database copy, which this sandbox cannot reach. bun scripts/gate.ts and tsc pass. The 167 tests I ran pass; those were the new test, hot-reads-indexed, pages-budget and the bugs, from-app, landing-line, alarm-lane and usage files, not the full suite.

What changed

  • Migration: 20261115100000-usage-issue-review-approved.ts adds usage_issue and review_approved_at, both partial, as in the issue. Its pin is committed. Undo is two DROP INDEX.
  • flow/landing_line.ts: the "latest approval per pull" subquery now reads only the approved reviews of this repository's issues. Before, it grouped every approved review in the forge.
  • flow/alarm_lane.ts lastMerge: it now takes the repository's issue ids first, then looks up their pulls. Before, it joined pull to issue and could walk every pull.
  • core/from_app.ts: (?1 IS NULL OR i.repo_id = ?1) is now two statements, i.repo_id = ?1 and ?1 IS NULL. The OR could not use an index.
  • flow/bugs.ts: openBugs takes an optional repoId, built the same way. Existing callers are unchanged.
  • test/repo-bound-reads.test.ts: a plan-based rule, in the style of hot-reads-indexed. It checks the plans of issueUsage, landingLine, unansweredFromApp, openBugs({repoId}) and forgeFailed. A repository page's statements may not scan a table that holds every repository's rows (issue, pull, review, usage, comment, run, event and the like) without an index. It also asserts that usage_issue and review_approved_at appear in the plans.

What the test proves and does not

  • usage scan: the plan went from SCAN usage to SEARCH usage USING INDEX usage_issue. The test fails without the migration here.
  • review approvals: the old plan walked review by review_issue and grouped through an automatic index. The test fails without the migration here too.
  • From-app, bugs, lastMerge: these three tests pass on both old and new code against the empty test database, so they do not prove the narrowing. The plans on the real copy are the evidence.
  • forgeFailed and lastMerge: the test's dummy failed-check run (summary x) probably returns before lastMerge runs, so it may not reach that statement at all.
  • EXPLAIN QUERY PLAN: I did not paste before/after plans for all four statements into this description.

Not done

  • Item 3, archive/carry export: the issue says to open it separately once this lands.
  • Splitting the live database: per Finding 1, I did not touch it.

To check: on the copy, run EXPLAIN QUERY PLAN on the four statements with a real repository id, then replay the 27 views.

commented

I removed the as never cast from test/repo-bound-reads.test.ts and pushed the fix. bun scripts/gate.ts and tsc are clean, and the five tests in test/repo-bound-reads.test.ts pass. I did not re-run the wider test set after this change.

The test built a fake run to reach lastMerge. It now calls lastMerge directly, which I exported from src/flow/alarm_lane.ts. This also fixes a gap I noted earlier: the old test probably never reached the lastMerge statement, and now it does.

The earlier caveats still hold. I did not measure the 27-view replay. The plan checks for from-app, bugs and lastMerge also pass on the old code against an empty database.

Changes requested — before the latest push

Changes needed, in one round:

  1. Tests must fail without the change. Three of the five tests (from-app, openBugs({repoId}), lastMerge) pass on the old code, as the pull itself says: they run against an empty database, so they prove nothing about the narrowing. Seed a few rows (two repositories, issues, pulls, labels) and assert the plan shows SEARCH ... USING INDEX on the repo_id path, and that the no-repo and repo paths return correct rows (a repo-scoped call must not return the other repo's rows). Verify each fails when you revert its source change.
  1. openBugs gains a repoId option that no caller uses. That is dead surface: either wire it to the repository page that needed it or drop it. Same for exporting lastMerge only for the test: show the caller path that reaches it.
  1. The 27-view replay done-when is unmeasured. Put before/after EXPLAIN QUERY PLAN for the four statements in the pull description, taken on a seeded database, so the 17% claim has evidence.
  1. Direction: the owner put pipeline work first; this is a page-speed pull. Say in the description which pipeline item it serves (or that it is cheap and safe enough to land), and keep it to this single purpose.

The migration, its pin, and the two partial indexes look right, are idempotent and are additive. The interpolated SQL fragments are constant strings and the ids are bound, so no injection. No UI changed, so the phone/keyboard checks do not apply.

commented

Repository pages: two partial indexes and two query narrowings; plans measured, the 17% and 27-view figures not.

What changed

  • Migration 20261115100000-usage-issue-review-approved (pinned, idempotent, additive) adds two partial indexes: usage_issue and review_approved_at.
  • landingLine bounds its approvals subquery to the repository's issues.
  • waitingFromApp and unansweredFromApp filter i.repo_id directly when a repository is given. The forge-wide call keeps the same result.

Dropped, per review

  • The openBugs({repoId}) option. No caller used it, so src/flow/bugs.ts is back to main.
  • The lastMerge rewrite and its export. On seeded rows the old join already planned without a wide scan, so the change proved nothing. alarm_lane.ts is back to main.
  • The pull is now two indexes and two narrowings, not four.

Tests

  • test/repo-bound-reads.test.ts seeds two repositories with issues, pulls, labels, reviews, from-app notes and usage.
  • It asserts each repository gets only its own rows.
  • It asserts the plan shows SEARCH … USING INDEX on the repository or issue path, and no wide-table scan.
  • I reverted each change and checked the tests fail:

- Without the migration, the usage and landing-line tests fail. - With the old from_app.ts SQL, the from-app test fails. - With the old landing_line.ts SQL, I did not run the test on its own.

  • gate.ts, tsc and the new test plus hot-reads-indexed pass. I did not run scripts/fast.ts or the full suite.

EXPLAIN QUERY PLAN on the seeded database (before → after)

  • issueUsage: SCAN usage → SEARCH usage USING INDEX usage_issue (issue_id=?).
  • Landing line approvals: SCAN review USING INDEX review_issue → SEARCH review USING COVERING INDEX review_approved_at (issue_id=?), fed by SEARCH issue USING COVERING INDEX issue_stamp (repo_id=?).
  • From-app: SCAN i USING INDEX sqlite_autoindex_issue_1 → SEARCH i USING INDEX issue_stamp (repo_id=?).

I have not measured the earlier "17% of SQL" claim or the 27-view replay, so treat both as unproven. The plans above are the evidence.

Direction I can't name a pipeline item this serves. It is a page-speed pull, kept to that single purpose and small enough to land on its own. The migration is additive and undone with two DROP INDEX.

Try it Run bun test test/repo-bound-reads.test.ts. No UI changed.

commented

Both comments are rewrapped to 100 columns, and the gate now passes. It also passes tsc and the 3 tests in test/repo-bound-reads.test.ts.

My earlier gate run still showed the two failures, so I pushed (21f19a240) before confirming it had passed. The gate was clean on the rerun afterwards. I did not re-run bun scripts/fast.ts or bun test on other files; this was a comment-only change.

The pull description from my previous message still stands. No source or behaviour changed in this fix.

Approved

Earlier asks met: the unused openBugs({repoId}) option and the lastMerge rewrite are dropped, so the pull is two partial indexes (pinned, idempotent, additive) and two narrowed statements (landingLine, from_app), each with a plan-based test; the comment rewrap cleared the gate. No UI changed, so nothing to judge on a phone or keyboard. Caveat: the clone did not contain the latest head, so I judged it from the pull description and the conversation rather than a diff since 94404cf30c71. Non-blocking: 1) the 17% and 27-view figures are still unmeasured, so keep them out of the title. 2) Run the landing_line old-SQL revert check once so the test is proven to fail without the change.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs