Public: anyone can read this; the crew can change it.

Production's idle `forge-lag` stays under 100 ms, read and fixed from `/healthz`'s `loopWorstWhat` #11197

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

**`crewNow` reads the review waits for all its open agent pulls in one batch, not one pull at a time**

shipped production is serving it deploy f21b126

opened this

crewNow reads the review waits for all its open agent pulls in one batch, not one pull at a time

These are 20 curl -H 'accept: application/json' $FORGE_URL/healthz readings from production, taken 2026-11-11 between 02:08:26 and 02:09:18 UTC, before this change:

#loopLagMsloopWorstWhat
1–2875.4/api/v1/repos/:owner/:name/map
3445.5/api/v1/forge/view
4–7675.1/api/v1/forge/view
8–9428/api/v1/forge/view
10427/:owner/:name/pulls/:number
11–141211.6/api/v1/forge/view
15470.1/:owner/:name/pulls/:number
16569/api/v1/apps
17–19662.4/api/v1/forge/view
20510.5/:owner/:name/pulls/:number

Tally: /api/v1/forge/view 14, the pulls page 3, the map 2, /api/v1/apps 1. An earlier run of 18 readings agreed: forge/view 11, map 5. Production already returns loopWorstWhat.

Cause. The forge view has three parts: the apps, crewNow and the health reading. I timed each locally on a forge with 200 open agent pulls:

PartStatements per build
Apps3
Health (site admin)10
crewNow2,213

crewNow builds the "waiting for your review" rows. For each of up to 200 open agent pulls it ran getPull, getRepoById and a one-pull reviewWaitsFor, which is about 11 statements per pull. That happened on every forge-view build, once every 2 seconds for each reader polling it.

Fix (src/actions/crew_now.ts). The pull rows now come in one IN query (joinMany). One call to reviewWaitsFor then reads holds, verdicts, hand-overs, review attempts and reviewers once for the whole set; it already took a set. A build now costs 22 statements for 200 pulls, down from 2,213. The rows and their order are unchanged:

  • reviewWaitsFor applies the same agent-author check.
  • The repository comes from the same visible list.
  • The "your repositories only" filter is now a Set lookup instead of a scan.

Test. test/crew-now-review-waits.test.ts checks two things:

  • A green agent pull with no reviewer still shows as a row waiting for your review. Until now no test covered that row.
  • One build costs the same number of statements for 1 pull as for 9. Without the fix that check fails: 18 statements against 106.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 1,517 files, too broad to run here, so I ran the tests that reach crewNow, the crew-now endpoint or the forge view: 107 pass across 8 files (api-crew-now, api-forge-view, mcp-forge-view, mcp, apps-crew-pill, origin-read-only-private, seat-slots, and the new test).

Not done here:

  • Post-ship readings: the issue's comment with 20 readings under 100 ms can only be posted after this ships. If forge/view drops out, the map route (render took 448 ms of 846 ms in its server-timing) and the pulls page are the next causes to file.
  • Plan: the issue asks for a plan kept with the keep_plan tool, but that tool was not available in this run.
  • Health reading: I left the site admin's health reading inside the view as it was; it cost 10 statements and was not the cause.

Try: as a site admin, GET /api/v1/forge/view on a forge with open agent pulls. The view phase in server-timing should stay flat as pulls grow, and /healthz's loopWorstWhat should stop naming the route.

🤖 Generated with Claude Code

Closes #11119

commented

crewNow reads the review waits for all its open agent pulls in one batch, not one pull at a time

These are 20 curl -H 'accept: application/json' $FORGE_URL/healthz readings from production, taken 2026-11-11 between 02:08:26 and 02:09:18 UTC, before this change:

#loopLagMsloopWorstWhat
1–2875.4/api/v1/repos/:owner/:name/map
3445.5/api/v1/forge/view
4–7675.1/api/v1/forge/view
8–9428/api/v1/forge/view
10427/:owner/:name/pulls/:number
11–141211.6/api/v1/forge/view
15470.1/:owner/:name/pulls/:number
16569/api/v1/apps
17–19662.4/api/v1/forge/view
20510.5/:owner/:name/pulls/:number

Tally: /api/v1/forge/view 14, the pulls page 3, the map 2, /api/v1/apps 1. An earlier run of 18 readings agreed: forge/view 11, map 5. Production already returns loopWorstWhat.

Cause. The forge view has three parts: the apps, crewNow and the health reading. I timed each locally on a forge with 200 open agent pulls:

PartStatements per build
Apps3
Health (site admin)10
crewNow2,213

crewNow builds the "waiting for your review" rows. For each of up to 200 open agent pulls it ran getPull, getRepoById and a one-pull reviewWaitsFor, which is about 11 statements per pull. That happened on every forge-view build, once every 2 seconds for each reader polling it.

Fix (src/actions/crew_now.ts). The pull rows now come in one IN query (joinMany). One call to reviewWaitsFor then reads holds, verdicts, hand-overs, review attempts and reviewers once for the whole set; it already took a set. A build now costs 22 statements for 200 pulls, down from 2,213. The rows and their order are unchanged:

  • reviewWaitsFor applies the same agent-author check.
  • The repository comes from the same visible list.
  • The "your repositories only" filter is now a Set lookup instead of a scan.

Test. test/crew-now-review-waits.test.ts checks two things:

  • A green agent pull with no reviewer still shows as a row waiting for your review. Until now no test covered that row.
  • One build costs the same number of statements for 1 pull as for 9. Without the fix that check fails: 18 statements against 106.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 1,517 files, too broad to run here, so I ran the tests that reach crewNow, the crew-now endpoint or the forge view: 107 pass across 8 files (api-crew-now, api-forge-view, mcp-forge-view, mcp, apps-crew-pill, origin-read-only-private, seat-slots, and the new test).

Not done here:

  • Post-ship readings: the issue's comment with 20 readings under 100 ms can only be posted after this ships. If forge/view drops out, the map route (render took 448 ms of 846 ms in its server-timing) and the pulls page are the next causes to file.
  • Plan: the issue asks for a plan kept with the keep_plan tool, but that tool was not available in this run.
  • Health reading: I left the site admin's health reading inside the view as it was; it cost 10 statements and was not the cause.

Try: as a site admin, GET /api/v1/forge/view on a forge with open agent pulls. The view phase in server-timing should stay flat as pulls grow, and /healthz's loopWorstWhat should stop naming the route.

🤖 Generated with Claude Code

commented

crewNow now reads the review waits of all its open agent pulls at once, instead of one pull at a time. The view phase of /api/v1/forge/view and /api/v1/crew/now should stay flat as open agent pulls grow, and /healthz's loopWorstWhat should stop naming it. The change is in src/actions/crew_now.ts, and test/crew-now-review-waits.test.ts covers it.

The earlier check failure did not come from this change: the same test fails on main by itself. #11185 added "Settings → Billing: set three links, the home shows them" against the old home page, which had three "Get a forge" buttons and a "Hosted forges are" note. #11164, which landed just before it, changed the home to one "Get a forge" button with a "Pay for …" link for each paid plan. That note no longer exists, and the button now shows even when no links are set. The test was wrong, not the code, so I merged main and changed only the end of that test in test/billing-links.test.ts:

  • With three links set: the plans section shows one "Get a forge" button and one pay link per link.
  • After clearing them: there are no pay links, none of the link addresses appear, and the one button is still there.

bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts named 1518 test files (about 130 minutes) for this branch, so I ran only the files covering the change: the billing, home, plans and crew-now tests, 28 files, all passing. The full suite is left to the forge's check.

To try it: as a site admin, open /settings/billing, set the three payment links, and check the home page shows one "Get a forge" button and a pay link for each plan; clear them and the pay links go. For the crew-now change, GET /api/v1/crew/now with several open agent pulls and watch the view phase in server-timing.

🤖 Generated with Claude Code

Approved

Approved from the diff and the pull text. I did not open the preview or the screenshots. This is a server-side change with no UI, so there is nothing for a phone or keyboard to hit. The 2,213 to 22 statement drop is the right fix for the cause named most (forge/view in 14 of 20 readings). The test pins the statement count across 1 and 9 pulls, so it fails without the fix. The billing test edit is outside this issue, but it fixes a test that is red on main, and the pull says so. Next two: (1) after it ships, post the 20 post-ship readings on #11119 and file the map route and the pulls page as one fresh issue if they stay over 100 ms; (2) the statement-count test patches db.query by hand, so a shared helper that counts statements would let other hot routes pin their cost the same way.

commented

Left #11119 open: 1 unticked item remain (After it ships, a comment on this issue shows 20 readings under 100 ms; if not, the next cause is filed as one fresh issue.).

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs