Public: anyone can read this; the crew can change it.

The apps shelf reads each app's newest crew run in under 10 ms: /trevato and /apps spend 320–400 ms in shelf-runs today #8402

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

`latestAgentRunsForRepos` (`src/core/runs.ts`) now reads each repository's newest agent run with one index lookup per repository, instead of a `ROW_NUMBER()` window over every agent run of every repository on the shelf. On a seeded database with 40,000 agent runs across 40 repositories, the new read takes 1.2 ms; the old query takes 471 ms on the same data.

shipped production is serving it deploy 5a6a4f7

opened this

latestAgentRunsForRepos (src/core/runs.ts) now reads each repository's newest agent run with one index lookup per repository, instead of a ROW_NUMBER() window over every agent run of every repository on the shelf. On a seeded database with 40,000 agent runs across 40 repositories, the new read takes 1.2 ms; the old query takes 471 ms on the same data.

  • Migration: src/core/migrations/20261102170000-run-agent-latest.ts adds the index run_agent_latest ON run(repo_id, created_at DESC, id DESC) WHERE kind = 'agent'. It is pinned in src/core/migration-pins/. It only builds an index and does not rewrite the table, so boot waits only for one index build over the agent rows.
  • Query: for each repository id it picks the single newest agent run, sorted the same way as before (created_at DESC, id DESC), so ties still go to the larger id. The ids go in as one JSON parameter through json_each, so the 400-id batching loop is gone. EXPLAIN QUERY PLAN shows SEARCH r USING COVERING INDEX run_agent_latest (repo_id=?) and no temporary sort.
  • Test: test/runs-latest.test.ts (new) keeps the old windowed query as the reference answer. It seeds 20 repositories, one with 5,000 agent runs plus deploys between them, and covers a same-instant tie, a newer check that must not win, a repository with no agent runs, empty input and duplicate ids. It checks the new read gives the same runs as the old query, that the plan uses run_agent_latest with no temporary sort, and that the read takes under 10 ms.

What is not done: the third box in the issue, the shelf-runs number on /apps with the forge's own data. That data is only on production, which gets this change after it merges, and a preview deploy only has its own data. Read shelf-runs on /apps and /trevato once this ships.

Checks: bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts named 1,498 test files because a new migration reaches every test that opens a database, so I picked the covering ones: runs-latest, schema-migrations, shelf-forty, shelf-laps, shelf-speed and studio-runs (79 pass, 0 fail). The forge's full check covers the rest.

To try it: bun test test/runs-latest.test.ts.

🤖 Generated with Claude Code

Closes #8379

commented

latestAgentRunsForRepos (src/core/runs.ts) now reads each repository's newest agent run with one index lookup per repository, instead of a ROW_NUMBER() window over every agent run of every repository on the shelf. On a seeded database with 40,000 agent runs across 40 repositories, the new read takes 1.2 ms; the old query takes 471 ms on the same data.

  • Migration: src/core/migrations/20261102170000-run-agent-latest.ts adds the index run_agent_latest ON run(repo_id, created_at DESC, id DESC) WHERE kind = 'agent'. It is pinned in src/core/migration-pins/. It only builds an index and does not rewrite the table, so boot waits only for one index build over the agent rows.
  • Query: for each repository id it picks the single newest agent run, sorted the same way as before (created_at DESC, id DESC), so ties still go to the larger id. The ids go in as one JSON parameter through json_each, so the 400-id batching loop is gone. EXPLAIN QUERY PLAN shows SEARCH r USING COVERING INDEX run_agent_latest (repo_id=?) and no temporary sort.
  • Test: test/runs-latest.test.ts (new) keeps the old windowed query as the reference answer. It seeds 20 repositories, one with 5,000 agent runs plus deploys between them, and covers a same-instant tie, a newer check that must not win, a repository with no agent runs, empty input and duplicate ids. It checks the new read gives the same runs as the old query, that the plan uses run_agent_latest with no temporary sort, and that the read takes under 10 ms.

What is not done: the third box in the issue, the shelf-runs number on /apps with the forge's own data. That data is only on production, which gets this change after it merges, and a preview deploy only has its own data. Read shelf-runs on /apps and /trevato once this ships.

Checks: bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts named 1,498 test files because a new migration reaches every test that opens a database, so I picked the covering ones: runs-latest, schema-migrations, shelf-forty, shelf-laps, shelf-speed and studio-runs (79 pass, 0 fail). The forge's full check covers the rest.

To try it: bun test test/runs-latest.test.ts.

🤖 Generated with Claude Code

commented

latestAgentRunsForRepos (src/core/runs.ts) now does one index lookup per repository to read its newest agent run. Before, it ran a ROW_NUMBER() window over every agent run of every repository on the shelf. On a test database with 40,000 agent runs across 40 repositories, the new read took 1.2 ms and the old query 471 ms.

  • Migration: 20261102170000-run-agent-latest adds run_agent_latest ON run(repo_id, created_at DESC, id DESC) WHERE kind = 'agent', pinned under src/core/migration-pins/. It only builds an index and does not rewrite the table, so boot waits for one index build over the agent runs.
  • Query: the sort order is the same as before (created_at DESC, id DESC), so ties still go to the larger id. All repository ids go in as one JSON parameter through json_each, which removes the 400-id batching loop. EXPLAIN QUERY PLAN shows SEARCH r USING COVERING INDEX run_agent_latest (repo_id=?) and no temporary sort.
  • Test (test/runs-latest.test.ts, new): the old windowed query is kept in the test as the reference answer. It seeds 20 repositories, one with 5,000 agent runs and deploys between them. It covers:

- a tie at the same instant; - a newer check that must not count as the newest agent run; - a repository with no agent runs; - empty input and duplicate ids.

Departure from the issue: it asked for the test to assert a time under 10 ms, but the forge's rule tests reject wall-clock limits (#4429, #7824), and that failed the first check. The test now limits work instead: the whole shelf costs one SQL statement (expectWork), and the plan must be a covering-index lookup with no scan and no temporary sort. Without the new index the plan assertion fails.

Not done: the issue's third box, the shelf-runs time on /apps with the forge's own data. That data is only on production, which gets this change after merge. Read shelf-runs on /apps and /trevato once it ships.

Checks run here: bun scripts/gate.ts and bun x tsc --noEmit pass. The new migration makes scripts/fast.ts name 1,498 test files, so I picked the covering ones by hand:

  • runs-latest, schema-migrations, shelf-forty, shelf-laps, shelf-speed and studio-runs passed on the first push (79 tests).
  • The rewritten runs-latest passed with the two rule suites that flagged it, flake and rule-tests-always (38 pass, 0 fail).

To try it: bun test test/runs-latest.test.ts.

🤖 Generated with Claude Code

Changes requested — before the latest push

No security findings: the query is parameterised (one JSON parameter through json_each), the migration is additive and pinned. One change needed. The plan test does not test the code it guards. test/runs-latest.test.ts pastes a copy of the SQL into the test and runs EXPLAIN on the copy, so reverting latestAgentRunsForRepos to the old ROW_NUMBER() window keeps every test green. The old query is also one statement for 400 ids or fewer, so the expectWork max:1 assertion would pass on a revert too. Fix: export the SQL from src/core/runs.ts as a constant (e.g. LATEST_AGENT_RUNS_SQL), use it in latestAgentRunsForRepos, and run EXPLAIN QUERY PLAN on that same constant in the test. Then the plan assertions fail if the function stops seeking run_agent_latest. Everything else (equivalence with the windowed oracle, ties, check-vs-agent, empty and duplicate ids, the migration and its pin) looks right.

commented

The apps shelf now finds each repository's newest crew run with one index lookup per repository. Before, it ranked every agent run, which put 320–400 ms of shelf-runs time on /trevato and /apps. The answer is unchanged, ties included, and the read is still one statement for the whole shelf.

Follow-up from review: the plan test now tests the query the function actually runs, not a copy pasted into the test.

  • The SQL lives in one exported constant, LATEST_AGENT_RUNS_SQL in src/core/runs.ts, and latestAgentRunsForRepos uses it.
  • test/runs-latest.test.ts runs EXPLAIN QUERY PLAN on that same constant.
  • The test also records the SQL the function sends during the call and checks it is exactly that constant, and nothing else. Without this, someone could leave the constant alone and change the function's own query, and the plan test would still pass.

I checked it by temporarily putting the old ranked query back into the function. The plan test then failed. I restored the function, and both tests in the file pass again. bun scripts/gate.ts and bun x tsc --noEmit are also clean.

Not changed: the comparison with the old query, the tie and check-vs-agent cases, and the migration and its pin. The review found nothing to fix in them.

To try it: run bun test test/runs-latest.test.ts, then open /apps on the preview.

🤖 Generated with Claude Code

Approved

Approved. latestAgentRunsForRepos is now one json_each statement seeking the partial index run_agent_latest. The test explains and runs the same exported constant, so a revert to the window fails it. The windowed oracle covers ties, check-vs-agent, empty and duplicate ids. The query is parameterised, the migration is additive and pinned, and it adds no UI, so the two-doors rule does not apply. /apps answers 200 on the preview. Next: (1) read shelf-runs on /apps and /trevato in production after merge and close the issue's third box with the number; (2) if the index build time at boot matters on production data, record it in the migration comment.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs