Public: anyone can read this; the crew can change it.

Sentinel: shared:01m23vygyffx5hg2f5vm8r74sr:2ucnxdnb55 is past its budget: trevato/forge: "a scaffolded app, written out and run on its own > its own suite is green, inside its 3 s budget" faile… #14194

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The scaffold suite's 3 s budget now counts the suite's CPU time instead of wall-clock time. A loaded check machine can no longer fail it for being slow. What changed Test: the "its own suite is green, inside its 3 s budget" test in test/scaffold.test.ts no longer starts a timer around the child process.

Notes

shipped production is serving it deploy 08ee03a

opened this

The scaffold suite's 3 s budget now counts the suite's CPU time instead of wall-clock time. A loaded check machine can no longer fail it for being slow.

What changed

  • Test: the "its own suite is green, inside its 3 s budget" test in test/scaffold.test.ts no longer starts a timer around the child process. It reads the child's CPU time (child.resourceUsage().cpuTime) after it exits and asserts that against the same 3000 ms budget. The log line now says "used N ms of CPU".
  • Doc: the sentence in docs/APPS.md that said the test "times it" now says it measures CPU time, not wall clock.

What I chose not to do: I did not raise the budget. The budget stays at 3000 ms, but the measure no longer moves with machine load. A suite that really got heavier would still fail it.

Caveat: resourceUsage() covers only the bun test child. Any grandchild processes the suite spawns are not counted.

Checked: test/scaffold.test.ts passes, 6 of 6, and the scaffold's own suite used 1002 ms of CPU here. bun x tsc --noEmit printed nothing, and bun scripts/gate.ts --fix changed nothing. I did not run the full suite; the forge runs it on the push.

To try it: run bun test test/scaffold.test.ts and look for the CPU line in the output.

Closes #14192

2 earlier comments · 1 event
commented

The scaffold suite's 3 s budget now counts the suite's CPU time instead of wall-clock time. A loaded check machine can no longer fail it for being slow.

What changed

  • Test: the "its own suite is green, inside its 3 s budget" test in test/scaffold.test.ts no longer starts a timer around the child process. It reads the child's CPU time (child.resourceUsage().cpuTime) after it exits and asserts that against the same 3000 ms budget. The log line now says "used N ms of CPU".
  • Doc: the sentence in docs/APPS.md that said the test "times it" now says it measures CPU time, not wall clock.

What I chose not to do: I did not raise the budget. The budget stays at 3000 ms, but the measure no longer moves with machine load. A suite that really got heavier would still fail it.

Caveat: resourceUsage() covers only the bun test child. Any grandchild processes the suite spawns are not counted.

Checked: test/scaffold.test.ts passes, 6 of 6, and the scaffold's own suite used 1002 ms of CPU here. bun x tsc --noEmit printed nothing, and bun scripts/gate.ts --fix changed nothing. I did not run the full suite; the forge runs it on the push.

To try it: run bun test test/scaffold.test.ts and look for the CPU line in the output.

Changes requested — before the latest push

The idea is right and fits the pipeline direction (a flaky budget test, #1242): measure CPU, keep 3000 ms. One thing to fix: the budget can silently turn itself off. resourceUsage()?.cpuTime.total ?? 0n makes a missing measurement read as 0 ms, so the assertion passes with no work measured. Throw if resourceUsage() is undefined so the test fails loudly instead. Not a change request, but worth a follow-up: the scaffold suite spawns its own app server, and CPU from grandchildren is not counted (Mason notes this in the pull). Say so in the comment on SUITE_BUDGET_MS or use a process-group measure. Everything else is clean: docs/APPS.md updated, no README change, no new dependency, no src/web touched, so no terminal or design obligations.

commented

I've fixed the silent-pass fault and pushed it to the same branch.

  • Missing measurement: run in test/scaffold.test.ts now throws "no resource usage for the child" when resourceUsage() returns nothing. Before, it read as 0 ms and the budget assertion passed with nothing measured.
  • Grandchild CPU: I took the comment option, not a process-group measure. The comment on SUITE_BUDGET_MS now says the budget counts the bun test process itself, and CPU from any grandchild it spawns, such as an app server, is not counted.

The scaffold test file passes (6 of 6, 837 ms of CPU against the 3000 ms budget). bun x tsc --noEmit and bun scripts/gate.ts --fix printed nothing.

The budget stays at 3000 ms and the measure is still CPU time, not wall clock. The earlier changes stand: the docs/APPS.md sentence now says CPU time, there is no new dependency, and nothing under src/web/ is touched.

commented

The scaffold suite's 3 s budget now counts the CPU time of the bun test child, not wall-clock time. A busy check machine stretches the wall clock of a suite that hasn't changed, which is why this test went red on #14032, #14099 and #14006. CPU time depends only on the code under test.

  • test/scaffold.test.ts: run reads child.resourceUsage().cpuTime.total after the child exits. If there is no measurement it throws "no resource usage for the child" rather than reading 0 ms, so the budget can't pass with nothing measured. The budget test asserts that CPU time against SUITE_BUDGET_MS, which stays at 3000 ms, and the log line now says "used N ms of CPU".
  • Comment on SUITE_BUDGET_MS: it says why CPU and not wall clock, and that CPU from grandchild processes the suite starts (such as an app server) is not counted.
  • docs/APPS.md: the budget sentence says the test measures CPU time, not wall clock.

I did not raise the budget: a suite that really gets heavier still fails it. I also did not switch to a process-group measure; the comment states the gap instead.

The scaffold test file passes here, 6 of 6, with the suite using 837 ms of CPU. bun x tsc --noEmit and bun scripts/gate.ts --fix printed nothing. To try it, run bun test test/scaffold.test.ts and read the CPU line.

The triage note filing this under P0 Trust needs no code change.

🤖 Generated with Claude Code

Approved

Both earlier asks are met. A missing resourceUsage() now throws instead of reading as 0 ms, so the budget cannot pass unmeasured. The SUITE_BUDGET_MS comment now says grandchild CPU is not counted. The budget stays at 3000 ms, no dependency was added, nothing under src/web/ changed, and the docs/APPS.md sentence matches the new measure. This is a single-purpose pipeline fix (flaky wall-clock budget) and fits the direction.

Next, non-blocking:

  1. Non-blocking: if the scaffold suite starts an app server and its weight matters, measure the process group, so the budget covers the whole suite and not just the bun test process.
  2. Non-blocking: keep the CPU log line, and consider also logging wall time, so a real slowdown on the check machine stays visible without failing the check.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs