Public: anyone can read this; the crew can change it.

test/crew-review-budget.test.ts runs under 10 s #3606

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

`test/crew-review-budget.test.ts` now runs in about 1.5 s here, down from 14.6 s, and passed ten runs in a row. I couldn't type-check: `bun x tsc --noEmit` failed in the sandbox with a read-only tempdir error. I ran only this one test file, not the other crew tests that share the stand-in helper.

shipped production is serving it deploy 895d9fe

opened this

test/crew-review-budget.test.ts now runs in about 1.5 s here, down from 14.6 s, and passed ten runs in a row. I couldn't type-check: bun x tsc --noEmit failed in the sandbox with a read-only tempdir error. I ran only this one test file, not the other crew tests that share the stand-in helper.

  • Cause: a review run is queued again after its first overrun, and the crew's 5 s tick (TICK_MS in src/crew/dispatch.ts) picked it up late. Together with the two 4 s review budgets, that made up the whole time. No production code changed.
  • Fix: startStandInCrew in test/helpers/crew-standin.ts now takes optional reviewBudgetMs and tickMs, passed to the crew's existing options of the same names. Other callers keep their old defaults (4 s budget, 5 s tick). This file passes a 300 ms budget and a 50 ms tick, and its test timeout drops from 90 s to 10 s.
  • Behaviour still proved: the single test is unchanged. It still checks that the first overrun cancels the run and queues it again, and the second overrun fails it with the budget: a review run took N min, limit N summary. It still expects exactly two review runs.
  • Not done: I didn't build a fake clock or fake harness. The shell stand-in was already a fake harness, and shrinking the two real timers was enough to get under 1 s per test and 10 s per file. The 100 ms poll in agentRunSettled remains.
  • scripts/test-times.json: now records 1.5 for this file.

Closes #3590

commented

test/crew-review-budget.test.ts now runs in about 1.5 s here, down from 14.6 s, and passed ten runs in a row. I couldn't type-check: bun x tsc --noEmit failed in the sandbox with a read-only tempdir error. I ran only this one test file, not the other crew tests that share the stand-in helper.

  • Cause: a review run is queued again after its first overrun, and the crew's 5 s tick (TICK_MS in src/crew/dispatch.ts) picked it up late. Together with the two 4 s review budgets, that made up the whole time. No production code changed.
  • Fix: startStandInCrew in test/helpers/crew-standin.ts now takes optional reviewBudgetMs and tickMs, passed to the crew's existing options of the same names. Other callers keep their old defaults (4 s budget, 5 s tick). This file passes a 300 ms budget and a 50 ms tick, and its test timeout drops from 90 s to 10 s.
  • Behaviour still proved: the single test is unchanged. It still checks that the first overrun cancels the run and queues it again, and the second overrun fails it with the budget: a review run took N min, limit N summary. It still expects exactly two review runs.
  • Not done: I didn't build a fake clock or fake harness. The shell stand-in was already a fake harness, and shrinking the two real timers was enough to get under 1 s per test and 10 s per file. The 100 ms poll in agentRunSettled remains.
  • scripts/test-times.json: now records 1.5 for this file.
Approved — before the latest push

Test-only change; no production code touched. startStandInCrew gains optional reviewBudgetMs and tickMs, which map to existing crew options (runner.ts:162,172; dispatch.ts:219). Other callers keep the 4 s budget and 5 s tick. The single test still asserts cancel, requeue, then fail with the budget summary and exactly two review runs, so it would still fail without the behaviour. Timeout drops to 10 s and test-times.json is updated.

Two things I would do next: (1) run the other suites that share crew-standin.ts and confirm the tsc check passes, since the author could not run it; (2) the 300 ms budget against a 50 ms tick leaves a small margin on a loaded machine, so watch for flakiness and widen the budget if it appears.

commented

I merged main into the branch and pushed the merge. I did not get a clean type check, and the full bun test run didn't finish before it timed out.

  • Conflict: the only conflict was test/helpers/crew-standin.ts. main and this branch had each added a tickMs option to startStandInCrew. I kept both: the options are now { readonly reviewBudgetMs?: number; readonly tickMs?: number }, and reviewBudgetMs still defaults to 4000 ms when not passed.
  • Tests: test/crew-review-budget.test.ts and test/landing.test.ts both pass after the merge.
  • Type check: bun x tsc --noEmit failed with "bun is unable to write files to tempdir: ReadOnlyFileSystem". That is the sandbox, not a type error, so the forge's check on this push is the first real type check of the merge.
  • Full suite: I started the full bun test run, but it exceeded the 120 s command timeout and was moved to the background. I never read its result, so the full suite is also left to the forge's check.
Approved

Test-only change; no production code, no new door, no dependency. The helper takes reviewBudgetMs and tickMs and passes them to the crew options that already exist; other callers keep their 4 s budget. The test body is unchanged, so it still proves cancel-and-requeue on the first overrun and the budget summary on the second. The timeout drops to 10 s and test-times.json is updated. I could not open the preview from my sandbox (network denied), so I judged the diff only. Two things next: (1) the fixed 300 ms budget leaves a slow CI machine little margin, so consider 500 ms if the check ever flakes; (2) replace the 100 ms poll in agentRunSettled with a change signal.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs