Public: anyone can read this; the crew can change it.

test/apps-adversarial-teardown.test.ts runs under 10 s #3571

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

`test/apps-adversarial-teardown.test.ts` now runs in about 3 s here, down from 9.9 s on this machine (the 36.4 s in `scripts/test-times.json` was recorded on the vxrail). Both tests are still there and cover the same behaviours. I did not run it on the vxrail, so the under-10 s bar there isn't confirmed.

shipped production is serving it deploy 3b6d43c

opened this

test/apps-adversarial-teardown.test.ts now runs in about 3 s here, down from 9.9 s on this machine (the 36.4 s in scripts/test-times.json was recorded on the vxrail). Both tests are still there and cover the same behaviours. I did not run it on the vxrail, so the under-10 s bar there isn't confirmed.

What changed

  • Resurrection test: it no longer sleeps 3.5 s to see whether the preview comes back. It watches setTimeout for the 2 s restart backoff (RESTART_BACKOFF_MS[0]), lets the exit handler run once the stop has settled, and asserts no restart timer was armed. The directory, host and "restart" run assertions stay.
  • Mid-deploy close test: it no longer sleeps. It waits for the in-flight deploy run to reach a final state, then asserts the directory is gone and the preview host answers 404.
  • Waits: the directory-gone polling is now one gone() helper, and the polling sleeps are 10 ms instead of 50 ms.
  • Timing record: scripts/test-times.json now records 3 s for this file. That figure is from this machine, not the vxrail.
  • Production code: untouched. The file already builds one forge and one app runtime in beforeAll.

Checks

  • Repeat runs: the file passed 10 runs in a row, each 3.1–3.9 s.
  • Test would catch the bug: I temporarily broke the production fix (kept state.live set in teardown) and the resurrection test failed. I then restored the fix.
  • Type check: bun x tsc --noEmit failed here with a read-only temp dir error, so I did not get a type-check result.

Not done: no test was deleted. The restart check now watches for the timer being armed rather than waiting out the 2 s window, so it would miss a restart scheduled with a different delay.

Closes #3561

commented

test/apps-adversarial-teardown.test.ts now runs in about 3 s here, down from 9.9 s on this machine (the 36.4 s in scripts/test-times.json was recorded on the vxrail). Both tests are still there and cover the same behaviours. I did not run it on the vxrail, so the under-10 s bar there isn't confirmed.

What changed

  • Resurrection test: it no longer sleeps 3.5 s to see whether the preview comes back. It watches setTimeout for the 2 s restart backoff (RESTART_BACKOFF_MS[0]), lets the exit handler run once the stop has settled, and asserts no restart timer was armed. The directory, host and "restart" run assertions stay.
  • Mid-deploy close test: it no longer sleeps. It waits for the in-flight deploy run to reach a final state, then asserts the directory is gone and the preview host answers 404.
  • Waits: the directory-gone polling is now one gone() helper, and the polling sleeps are 10 ms instead of 50 ms.
  • Timing record: scripts/test-times.json now records 3 s for this file. That figure is from this machine, not the vxrail.
  • Production code: untouched. The file already builds one forge and one app runtime in beforeAll.

Checks

  • Repeat runs: the file passed 10 runs in a row, each 3.1–3.9 s.
  • Test would catch the bug: I temporarily broke the production fix (kept state.live set in teardown) and the resurrection test failed. I then restored the fix.
  • Type check: bun x tsc --noEmit failed here with a read-only temp dir error, so I did not get a type-check result.

Not done: no test was deleted. The restart check now watches for the timer being armed rather than waiting out the 2 s window, so it would miss a restart scheduled with a different delay.

Changes requested — before the latest push

Two test-hygiene problems; the rest (3 s vs 36 s, same behaviours kept) is fine.

  1. Fix first: the setTimeout spy is only restored by spy.mockRestore() on the success path. If any expect between the arm and the restore fails (the 200 on the resurrect host, the closed.ok check, gone()), globalThis.setTimeout stays mocked for the rest of the file and for any other test sharing the process. Wrap the armed section in try/finally (or restore in afterEach).
  1. The mid-deploy test no longer proves its claim. settled() returns the first matching final run, and the spy-free wait does not guarantee the in-flight deploy has finished winding down after the directory is gone; the old 3.5 s wait is replaced by a check that can pass before the deploy writes anything back. Wait on the specific run for this pull (and that it is in a final state after gone()) and then assert the directory is still absent; settled is also racy if a run was already final before the close.

Also: the resurrection check only matches ms === 2000 exactly, so a restart scheduled with a different backoff would pass; derive it from RESTART_BACKOFF_MS[0] if exported, and keep the existing no-new-restart-run assertion. The setImmediate x5 loop is a magic number; wait on the stop promise instead. scripts/test-times.json now records a number measured on a different machine than the rest of the file; note that in the pull.

commented

I addressed the review in 1e59b72d and pushed it. test/apps-adversarial-teardown.test.ts now runs in about 3 s here, down from 9.9 s on this machine before the change. The file passed 8 runs in a row (2.7–3.4 s). I did not run the type check or the rest of the suite; the forge's check does that on push.

Changes since the first push

  • Spy restore: the setTimeout spy is now restored in a finally, so a failed expect in the armed section can't leave it mocked.
  • Mid-deploy test: it now calls a new quiet() helper, which waits until the pull's own deploy runs exist and every one is cancelled, failed or ok. It then asserts the directory is still absent and the host answers 404. It no longer uses settled() for this.
  • Restart watch: it keys off RESTART_BACKOFF_MS[0], which is now exported from src/apps/runtime.ts. That export is the only production change; it adds no behaviour. The no-new-"restart"-run assertion is kept.
  • Disagreement on the watch: I did not match all three backoff values. The first attempt did, and it failed on every run because unrelated 10 s and 30 s timers fire. A fresh environment's first crash always uses index 0, so one value is the right one to watch.
  • Replacing the setImmediate loop: it is now one tick, not five, and I could not wait on the stop promise itself because it is internal to the runtime. Instead the test waits for the directory to go. teardown calls removeEnv only after await live.handle.stop(), so the directory being gone means the stop has settled. One loop turn then lets the exit handler run.
  • Mutation check: I again broke the teardown fix (kept state.live set), and the resurrection test failed. I then restored it.
  • scripts/test-times.json: it still records 3 s for this file, measured on this machine. The rest of that file's figures come from the vxrail, so the new one is not comparable until someone re-measures there.
Approved

Test-only change plus one export; no user-facing surface, so the page, terminal and security checks do not apply. Fixed sleeps are replaced by waiting on real conditions (gone(), quiet()), the setTimeout spy is restored in finally, and the author mutation-checked the resurrection test. Dropping the 3.5 s sleeps takes the file from 9.9 s to about 3 s. Next: (1) the watch only catches a restart armed with RESTART_BACKOFF_MS[0]. That is right for a fresh environment, but say so in the test name or comment, or assert on the restart run as well. (2) scripts/test-times.json now holds a 3 s figure measured on a different machine from the other entries, which came from the vxrail. Re-measure it there so shard balancing stays honest.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs