Displays read git once per repository, and the picker reads fresh crew pulls once per pass #10534

Displays read git once per repository instead of once per build per request

What was slow: every request for a display page spawned git for each source repository's pin listing, and the page did it two or three times: in displayItems, again in selectionKeys, and again in choosableBuilds for a writer. On top of that, demoFactsAt spawned 2 to 4 git processes for every build whose demo names no launch. So page time grew with the number of builds.

The fix:

• src/git/pins_kept.ts keeps each repository's pin listing in memory. A listing is dropped when this process writes pins or notes (pinBuild, rewritePin, appendNote, deletePin in src/git/builds.ts), on any push, merge, repo. or build. event (one subscription in src/app.ts), and after 30 s in any case. It holds at most 2,000 repositories.
• demoFactsAt keeps a commit's facts by oid, since what a commit holds never changes. Facts borrowed from the branch tip, for a build whose commit has no pages of its own, are kept for 30 s only. It holds at most 4,096 commits.

Before and after, from the new fixture in test/display-warm-reads.test.ts: one display whose source is an owner with 10 repositories × 3 showcased builds, a warm second request, Server-Timing as reported.

The writer's one remaining git read is the branch tip in the save form, which is read fresh on purpose so a save against a moved branch is refused. /displays was already cached in display/known.ts; I didn't record its before number. The test counts git spawns rather than timing them, as test/helpers/work.ts asks, and checks that a new pin shows on the next load. I couldn't measure the real /trevato/games-shelf or /trevato/realty-desk: the production instance my shell can reach doesn't hold them (both answer 404).

Not in this pull: the other two parts of #10445 are now their own issues, and #10445 waits on both:

• #10535: a test that walks every GET page against a 150 ms budget, naming the slowest phase.
• #10536: real-user timings rolled up per page on the ops view, one line per page over budget.

Checks: scripts/gate.ts and tsc --noEmit are clean. The display, displays, builds, pin, demo, arcade, notes and lineage suites pass: 253 tests across 35 files. By mistake, one run had no files listed and started the whole suite. The failures it showed were in sandbox, OAuth, network and preview suites, none touching pins, displays or builds; the forge's full check is the verdict on those.

To try it: open a display whose display.json names an owner with many pins, reload, and read the items phase in Server-Timing.

🤖 Generated with Claude Code

Closes #10445

Displays read git once per repository instead of once per build per request

What was slow: every request for a display page spawned git for each source repository's pin listing, and the page did it two or three times: in displayItems, again in selectionKeys, and again in choosableBuilds for a writer. On top of that, demoFactsAt spawned 2 to 4 git processes for every build whose demo names no launch. So page time grew with the number of builds.

The fix:

• src/git/pins_kept.ts keeps each repository's pin listing in memory. A listing is dropped when this process writes pins or notes (pinBuild, rewritePin, appendNote, deletePin in src/git/builds.ts), on any push, merge, repo. or build. event (one subscription in src/app.ts), and after 30 s in any case. It holds at most 2,000 repositories.
• demoFactsAt keeps a commit's facts by oid, since what a commit holds never changes. Facts borrowed from the branch tip, for a build whose commit has no pages of its own, are kept for 30 s only. It holds at most 4,096 commits.

Before and after, from the new fixture in test/display-warm-reads.test.ts: one display whose source is an owner with 10 repositories × 3 showcased builds, a warm second request, Server-Timing as reported.

The writer's one remaining git read is the branch tip in the save form, which is read fresh on purpose so a save against a moved branch is refused. /displays was already cached in display/known.ts; I didn't record its before number. The test counts git spawns rather than timing them, as test/helpers/work.ts asks, and checks that a new pin shows on the next load. I couldn't measure the real /trevato/games-shelf or /trevato/realty-desk: the production instance my shell can reach doesn't hold them (both answer 404).

Not in this pull: the other two parts of #10445 are now their own issues, and #10445 waits on both:

• #10535: a test that walks every GET page against a 150 ms budget, naming the slowest phase.
• #10536: real-user timings rolled up per page on the ops view, one line per page over budget.

Checks: scripts/gate.ts and tsc --noEmit are clean. The display, displays, builds, pin, demo, arcade, notes and lineage suites pass: 253 tests across 35 files. By mistake, one run had no files listed and started the whole suite. The failures it showed were in sandbox, OAuth, network and preview suites, none touching pins, displays or builds; the forge's full check is the verdict on those.

To try it: open a display whose display.json names an owner with many pins, reload, and read the items phase in Server-Timing.

🤖 Generated with Claude Code

The picker now reads a repository's recent crew pulls once per pass, not once per ready issue per builder. The freshPullFor cost from the 03:35Z logs should mostly go away: the logs put it at about 1,300 queries a minute and 11 s of every minute of SQL time. I have not measured it on the running forge.

• src/crew/fresh_pull.ts: split into two parts.

- freshPulls(db, repoId) runs one query for the pulls opened in the last ten minutes. +i.repo_id keeps the query planner on the existing issue_created index, so it reads only that window and not every issue in the repository. No new index or migration was needed. - pullFor(fresh, issueNumber, usernames) matches an issue in memory against <user>/<n> or <user>/<n>-…. The old LIKE escaping isn't needed any more. - freshPullFor stays as a thin wrapper, so the dispatcher's one-off call in dispatch.ts is unchanged.

• src/crew/held.ts: heldBy reads freshPulls once before the loop (and skips it when there are no builders), then holdOf calls pullFor for each issue.
• test/fresh-pull.test.ts (new) covers:

- exact and suffixed branch matches; - lookalikes that must not match (mason/120 against issue 12, ma_on/12 against mason); - several builders; - the ten-minute window; - the query plan using issue_created.

I did not rerun the median check time. If the 6-minute median comes from somewhere else, it needs its own look.

Checks run: scripts/gate.ts and tsc --noEmit pass, and so do test/fresh-pull.test.ts, test/crew-held.test.ts and test/team-queue.test.ts. I picked those suites myself: scripts/fast.ts named 2,075 test files for this branch's diff against main, far over its budget.

🤖 Generated with Claude Code

Displays read git once per repository, and the picker asks SQL once per pass

Displays: a warm display page used to spawn git for every build on every request. Measured with Server-Timing on the fixture in test/display-warm-reads.test.ts (one owner, 10 repositories × 3 builds), a warm request went from 41 git processes to 0 (from 64 to 1 for a writer).

• src/git/pins_kept.ts keeps each repository's pin listing in memory. A listing is dropped when this process writes pins or notes, on any push, merge, repo. or build. event, and after 30 s in any case.
• demoFactsAt keeps a commit's facts by oid, since a commit never changes.

Fresh crew pulls: freshPullFor used to run one query per ready issue per builder, from held.ts. The 03:35Z logs counted about 1,300 queries a minute, costing 11 s of every minute of SQL time.

• src/crew/fresh_pull.ts now has freshPulls: one query per repository per pass. +i.repo_id keeps it on the issue_created index, so it reads only the last ten minutes.
• pullFor matches each issue in memory.
• test/fresh-pull.test.ts covers the branch matching, the time window and the query plan. Whether this is why the median check time is 6 minutes has not been measured.

Red on main as well, fixed here:

• Crew shell sandbox (src/crew/sandbox.ts): an agent could lose read access to its own working directory. Since 636aa0355, a run's temp directory moves to the machine's temp directory when the agent's path is too long for a socket. tempDenied then denied every other entry of the machine's temp directory, and that included the data directory whenever it sat inside temp, as it does in every check shard. A data directory inside temp is now handled first: only the forge's own pieces of temp are denied there, plus other runs' fr- temp directories. This fixes the three crew-sandbox tests and the crew-shell test in outsiders-member. In test/crew-sandbox.test.ts I changed one assertion that was wrong: it assumed the run's temp is always under the agent's directory; it now asks runTmpDir where it should be.
• test/child-quota.test.ts: the test was wrong: it assumed a small forge needs less than 2000 MiB, but it needs 2048. It now computes the limit from SIZES.
• scripts/unreached-allow.json: removed src/kernel/why_now.ts, a file that no longer exists.

Not fixed: the test "a real child forge > boots in its own forge:<name> cell…" still fails. It also fails on #10523 and #10510, which are on the same base. It needs a real cell (bwrap), which this sandbox can't create, and the check log was cut off before that shard's output, so I couldn't see the cause.

Checks run: scripts/gate.ts and tsc --noEmit pass. These test files pass: fresh-pull, crew-held, team-queue, crew-sandbox, unreached, child-quota, outsiders-member and spawn-cell (in spawn-cell, the tests that need a cell skip here).

🤖 Generated with Claude Code

I merged main into scout/10445-fast-everywhere-displays-first-then-a-ti and pushed. Both sides had made the same fix in the three conflicting files (#10541: a crew shell keeps its working directory when the data directory sits in the machine's temp), so each conflict keeps one version of it plus anything only one side had.

• src/crew/sandbox.ts: kept main's tempDenied. It denies what the branch's version denied: Claude Code's own temp, the fc- entries, and every other run's temp directory. It is just written more tidily.
• test/child-quota.test.ts: kept main's version. It works out the room for one more small forge from SIZES and also checks that large needs more memory than small.
• test/crew-sandbox.test.ts: kept main's comment on runTmpDir and the branch's check that a neighbouring fr- run temp directory is denied to this run.

bun scripts/gate.ts and bun x tsc --noEmit pass. The tests for the conflicting files (crew-sandbox, child-quota, 36 tests) pass, as do the display and picker tests (28). bun scripts/fast.ts named 2077 of 2449 test files for this merge (about 177 minutes), so I didn't run that set and left the full suite to the forge's check on the push.

🤖 Generated with Claude Code

Displays read git once per repository, and the picker reads fresh crew pulls once per pass

Serves #7553 (the pipeline is the engine). The freshPulls change cuts picker SQL time. freshPullFor ran about 1,300 queries a minute (one per ready issue per builder) and took about 11 s of every minute of SQL time. Now it runs one query per repository per pass, on issue_created, and each issue is matched in memory. The display cache is the other half of #10445, from the owner's report that displays are slow, and serves no pipeline issue. The per-page time budget and the real-user rollup moved to #10535 and #10536. This pull does not deliver them.

What changed

• src/crew/fresh_pull.ts: freshPulls reads the repository's pulls from the last ten minutes. +i.repo_id keeps the planner on issue_created. pullFor matches <user>/<n> exactly, or <user>/<n>-… ignoring ASCII case. That is what the old = and LIKE did, so behaviour is unchanged, and a test covers it. held.ts reads once per pass.
• src/git/pins_kept.ts: each repository's pin listing is kept in memory for up to 30 s. It is dropped when this process writes that repository's pins or notes (pinsMoved). Every listing is dropped on a push, pull.merged, repo. or build. event on the bus (dropAllPins, subscribed in app.ts). At most 2,000 repositories are kept.
• demoFactsAt keeps each commit's facts by oid, since a commit never changes. Facts borrowed from the branch tip are read again after PINS_FRESH_MS.
• Before and after: a warm /:owner/:name/display over 30 builds spawned 41 git processes for a reader and 64 for a writer. It now spawns 0 for a reader and 1 for a writer (the form's tip, read fresh to guard the save). /displays spawns 0.

Review round

• The EXPLAIN test runs the shipped FRESH_PULLS_SQL with its parameters bound. It fails if +i.repo_id is removed.
• New test/display-cache.test.ts has one test per cache path:

- the 30 s TTL; - each of push, pull.merged, repo.created and build.pinned through app.bus (an issue.opened keeps the listing); - facts kept by oid, answered with the repository moved out of reach; - borrowed facts held within the window and re-read after it.

I removed each behaviour in turn and the matching test failed.

• The test/crew-sandbox.test.ts change is dropped, and the console.log calls are gone from test/display-warm-reads.test.ts.

Not done here: route-wide budgets and real-user timings (#10535, #10536). A pin ref moved by something the forge never sees (a raw git update-ref on the server) stays stale for up to 30 s.

To try it: bun test test/display-cache.test.ts test/display-warm-reads.test.ts test/fresh-pull.test.ts. The gate and tsc --noEmit pass, as do 111 tests across 15 files: the three above plus the display, pin, picker, dispatch and crew-sandbox suites. scripts/fast.ts named 2,078 files for this diff, far over its budget, so I chose these files myself.

🤖 Generated with Claude Code