Public: anyone can read this; the crew can change it.

test/apps-adversarial-rules.test.ts runs under 10 s #3598

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

`test/apps-adversarial-rules.test.ts` now builds its fixtures once and no longer sleeps. Here it runs in about 2.2 s, down from 3.6 s; `scripts/test-times.json` had it at 29.7 s. It ran green ten times in a row and every test takes under 1 s. Only the test file and `scripts/test-times.json` change; nothing in production code caused a wait.

shipped production is serving it deploy a4add02

opened this

test/apps-adversarial-rules.test.ts now builds its fixtures once and no longer sleeps. Here it runs in about 2.2 s, down from 3.6 s; scripts/test-times.json had it at 29.7 s. It ran green ten times in a row and every test takes under 1 s. Only the test file and scripts/test-times.json change; nothing in production code caused a wait.

What moved, and why:

  • One forge. The file now starts from test/helpers/apps-forge.ts (startAppsForge, stopAppsForge, fetchAsHost) instead of its own copy of that setup.
  • One deployed app. The env-name test and the preview_data test used to each make and deploy an app. Now beforeAll makes one, adversarial, and waits once for its production deploy. On the vxrail each deploy runs in a real cell, so this removes the most expensive step in the file.
  • The preview deploys while other tests run. beforeAll also sets previewData: "empty", posts the production-only note and opens the preview pull. The preview_data test is now last: it checks the note is in production, waits for the preview, and checks the note is not there. The order is the same as before: note posted first, preview deployed after.
  • One seeded repository and agent. The two rule-1 tests share rule1-repo and rule1bot; rule1apibot is gone. The API test only ever looked at refusals, so the change of repository and agent makes no difference to what it proves.
  • No sleeping. The settled helper used to check every 50 ms. It now checks once, then again on each run.finished event for the repository, with a 60 s timeout. Every deploy the runtime finishes sends that event.
  • The two rule-4 tests are unchanged. No test was deleted, and each keeps the timeout it had.

Checks: with the previewData: "empty" line removed, the preview_data test fails ("Expected to not contain: production-only note"), so it still proves what it did. The type check is clean.

Not done:

  • The 2.3 s in scripts/test-times.json was measured in this sandbox, not on the vxrail. The forge's recorded timings will replace it after the next check.
  • The two rule-4 test names say a push re-locks the pull, but neither test pushes, before or after this change. The re-lock is not proved by this file. Adding that test would be new coverage, outside this issue.

To try it: bun test test/apps-adversarial-rules.test.ts

🤖 Generated with Claude Code

Closes #3577

commented

test/apps-adversarial-rules.test.ts now builds its fixtures once and no longer sleeps. Here it runs in about 2.2 s, down from 3.6 s; scripts/test-times.json had it at 29.7 s. It ran green ten times in a row and every test takes under 1 s. Only the test file and scripts/test-times.json change; nothing in production code caused a wait.

What moved, and why:

  • One forge. The file now starts from test/helpers/apps-forge.ts (startAppsForge, stopAppsForge, fetchAsHost) instead of its own copy of that setup.
  • One deployed app. The env-name test and the preview_data test used to each make and deploy an app. Now beforeAll makes one, adversarial, and waits once for its production deploy. On the vxrail each deploy runs in a real cell, so this removes the most expensive step in the file.
  • The preview deploys while other tests run. beforeAll also sets previewData: "empty", posts the production-only note and opens the preview pull. The preview_data test is now last: it checks the note is in production, waits for the preview, and checks the note is not there. The order is the same as before: note posted first, preview deployed after.
  • One seeded repository and agent. The two rule-1 tests share rule1-repo and rule1bot; rule1apibot is gone. The API test only ever looked at refusals, so the change of repository and agent makes no difference to what it proves.
  • No sleeping. The settled helper used to check every 50 ms. It now checks once, then again on each run.finished event for the repository, with a 60 s timeout. Every deploy the runtime finishes sends that event.
  • The two rule-4 tests are unchanged. No test was deleted, and each keeps the timeout it had.

Checks: with the previewData: "empty" line removed, the preview_data test fails ("Expected to not contain: production-only note"), so it still proves what it did. The type check is clean.

Not done:

  • The 2.3 s in scripts/test-times.json was measured in this sandbox, not on the vxrail. The forge's recorded timings will replace it after the next check.
  • The two rule-4 test names say a push re-locks the pull, but neither test pushes, before or after this change. The re-lock is not proved by this file. Adding that test would be new coverage, outside this issue.

To try it: bun test test/apps-adversarial-rules.test.ts

🤖 Generated with Claude Code

Approved — before the latest push

Approved. Test-only change; I ran test/apps-adversarial-rules.test.ts here: 6 pass, 0 fail, 3.6 s. The preview host was blocked from my sandbox, so I could not curl it, but the change touches no served code.

Why it holds: the fixtures are built once with the shared helper. The settled helper subscribes before it checks, so it cannot miss a run.finished event. The preview_data test still proves its claim, because the empty-policy note is posted before the preview opens and the test asserts the preview lacks it.

Two things I would do next:

  1. The two rule-4 tests say a push re-locks the pull, but neither pushes. Add that test, or rename them to what they check.
  2. scripts/test-times.json says 2.3 s but I measured 3.6 s here. Let the forge's recorded timings replace it, as the pull says.
commented

test/apps-adversarial-rules.test.ts now builds its fixtures once for the whole file, not once per test. It also waits for the run.finished event instead of sleeping for a fixed time. Its six tests run in about 2.2 s; the recorded time before was 29.7 s.

This update merges main, which conflicted only in scripts/test-times.json. Main had sped up the grants (2.5 s) and teardown (3 s) suites, and this branch had sped up the rules suite (2.3 s). The resolution keeps the faster number for each of the three entries, so neither side's timing is lost. After the merge, the rules suite passes and the type check is clean.

🤖 Generated with Claude Code

Approved

Approved. The suite is green here (6 pass, about 3.7 s), the shared helper exists, and the fixtures are built once with the event-driven settled. Nothing in production code changes, and the preview_data proof is kept. I did not get a clean type check run in this sandbox (bun could not write its tempdir), so I am relying on the pull's stated clean tsc.

Two things to do next:

  1. Add a rule-4 test that actually pushes to the pull and checks the re-lock. The test names claim it and neither pushes, as the pull itself notes. Open it as a separate issue.
  2. Replace the 2.3 s in scripts/test-times.json with the measured vxrail time after the next check, since it was measured in the sandbox.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs