Public: anyone can read this; the crew can change it.

The test runner has a looks lane: chromium test files run only where chromium starts, named *.looks.test.ts, out of the gate budget #12506

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

test/.looks.test.ts files now run in a looks lane of their own, only where a Chromium actually starts. The gate tiers no longer see them.

Notes

shipped production is serving it deploy 7612a18

opened this

test/*.looks.test.ts files now run in a looks lane of their own, only where a Chromium actually starts. The gate tiers no longer see them. Gate, tsc, the gate and shard tests and the two moved files all pass; the two moved files' browser tests skipped in this sandbox because no Chromium starts here, so the lane running them for real is not exercised here. I did not run the full suite — the forge's check runs it.

  • Runner (scripts/test-shards.ts): on --gate with the full tier, the runner appends the looks files if findChromium() finds a binary and chromiumRuns() shows it starts. It prints either looks lane, N file(s) with <binary> or looks lane skipped: no Chromium that starts on this machine. The lane is exempt from the per-test budget. The fast tier and a box without a working browser leave the lane out.
  • Gate (scripts/gate.ts):

- presentTests() skips looks files, so the exclusion and SLOW audits and the fast tier never count them. - New looksTests() and isLooksFile() list them. - I removed the two SLOW rows instead of renaming them, because a SLOW row for a file the gate no longer lists is itself an audit failure.

  • Moves: test/oauth-connect-browser.test.ts and test/loading-chrome.test.ts are renamed to *.looks.test.ts with their contents unchanged, so no assertion is dropped. loading-chrome fits the lane because it drives a real Chromium too. The docs/ORIGINS.md reference is updated.
  • Tests (test/test-budget-run.test.ts): three new tests prove the lane:

- With a stand-in Chromium that starts, the gate runs a looks file that is over the budget, and it stays green. - With none that starts, the lane is skipped and the skip is named. - The fast tier and presentTests leave looks files out.

I did not give the lane its own time budget; it is simply unbudgeted. outsideFiles and the scheduled run do not include looks files either, since they derive from presentTests.

To try it: bun scripts/test-shards.ts --gate and look for the looks-lane line near the top. FORGE_CHROMIUM=/path/to/chromium points it at a particular binary.

Closes #12496

3 earlier comments
commented

test/*.looks.test.ts files now run in a looks lane of their own, only where a Chromium actually starts. The gate tiers no longer see them. Gate, tsc, the gate and shard tests and the two moved files all pass; the two moved files' browser tests skipped in this sandbox because no Chromium starts here, so the lane running them for real is not exercised here. I did not run the full suite — the forge's check runs it.

  • Runner (scripts/test-shards.ts): on --gate with the full tier, the runner appends the looks files if findChromium() finds a binary and chromiumRuns() shows it starts. It prints either looks lane, N file(s) with <binary> or looks lane skipped: no Chromium that starts on this machine. The lane is exempt from the per-test budget. The fast tier and a box without a working browser leave the lane out.
  • Gate (scripts/gate.ts):

- presentTests() skips looks files, so the exclusion and SLOW audits and the fast tier never count them. - New looksTests() and isLooksFile() list them. - I removed the two SLOW rows instead of renaming them, because a SLOW row for a file the gate no longer lists is itself an audit failure.

  • Moves: test/oauth-connect-browser.test.ts and test/loading-chrome.test.ts are renamed to *.looks.test.ts with their contents unchanged, so no assertion is dropped. loading-chrome fits the lane because it drives a real Chromium too. The docs/ORIGINS.md reference is updated.
  • Tests (test/test-budget-run.test.ts): three new tests prove the lane:

- With a stand-in Chromium that starts, the gate runs a looks file that is over the budget, and it stays green. - With none that starts, the lane is skipped and the skip is named. - The fast tier and presentTests leave looks files out.

I did not give the lane its own time budget; it is simply unbudgeted. outsideFiles and the scheduled run do not include looks files either, since they derive from presentTests.

To try it: bun scripts/test-shards.ts --gate and look for the looks-lane line near the top. FORGE_CHROMIUM=/path/to/chromium points it at a particular binary.

commented

Pushed a fix for two of the four failures; I can't see the other two, so the next check will show whether they're gone. Locally the two fixed tests now pass, as do the tree-scan, review-rules, release-checkout, spawn-host-push and crew-sandbox-escape tests. bun scripts/gate.ts is clean.

  • Cause: the two failing tests, test/check-list.test.ts and test/flake.test.ts, each built their own list of test files from the directory. That list included the two renamed *.looks.test.ts files, which the gate now leaves out, so the lists disagreed with gateFiles(). Both tests now skip .looks.test.ts when they build the list. The gate's behaviour is unchanged.
  • Not mine: the check log shows test/boundary-tenancy-cell.test.ts is red on main too. It failed with an egress refusal and is the file with the "two hosted forges on one machine…" test, so I left it alone.
  • Unknown: the log was cut off after the third failure name. If a fourth still fails on this push, I'll need its name from the new log.
commented

The test runner now has a looks lane: *.looks.test.ts files run only where Chromium starts, outside the gate tiers and the test budget. The two Chromium tests moved into it, and the gate-list tests list present files as the gate does, leaving the looks lane's files aside. I merged the current main into the branch and pushed; the check has not yet run on the merged branch.

I found no failure that this pull caused:

  • test/deadends.test.ts ("every instruction names a script or a tool that exists"): it fails with tool one at src/http/mcp.ts:287 after the merge. It fails the same way on a plain origin/main, so it is not this pull's. The check log also says test/deadends.test.ts is red on main.
  • "a stranger calling every listed tool…": it lives in test/mcp-refusal-hints.test.ts, which the check log reports as red on main. I never located or ran that test file, so I did not confirm it.
  • test/explore-rank.test.ts: this was the one file the check log said is green on main and red here. It passes here, both before and after the merge. It measures cold and warm request times, so I suspect load on the check machine, but I did not confirm that.
  • test/holds.test.ts: passes here.

Before pushing, bun scripts/gate.ts and bun x tsc --noEmit were clean, and explore-rank and holds passed.

I did not touch the main failures, since fixing them is outside this change.

commented

The test runner now has a looks lane: *.looks.test.ts files run only where a Chromium starts, outside the gate tiers and the per-test budget. loading-chrome and oauth-connect-browser moved into it.

The failing check did not come from this change, and I changed no code to fix it. The one test the log blames on this pull is test/spawn-stranger-cell.test.ts, "a stranger claims a small child…". It builds a whole forge inside a cell. The log shows that forge's requests refused: egress refused: kid--forge.test:443 is not in this repository's egress, and 127.0.0.1:… is a private destination. Those are cell-egress failures, not anything the runner or the renamed files touch. I couldn't run that test here, because it is skipped under FORGE_TEST_SOCKET.

The other failures in the list were red on main too: mirror-worker, snapshot-incremental, snapshot-repositories, boundary-tenancy-cell and studio-app-timing.

To get a fresh check run, I merged current main into the branch and pushed. Before pushing, bun scripts/gate.ts and bun x tsc --noEmit were clean. test/test-budget-run.test.ts, test/check-list.test.ts and test/flake.test.ts pass, 63 tests.

If spawn-stranger-cell fails again with the same egress lines, the cause is the check box's egress for cell children, not this pull. A person would need to look at that.

Try it: bun scripts/test-shards.ts --gate prints test-shards: looks lane, N file(s) with <chromium> where a browser starts. Elsewhere it prints looks lane skipped: no Chromium that starts on this machine.

🤖 Generated with Claude Code

Approved

Clean. The looks lane runs only on the full gate where findChromium() and chromiumRuns() say a browser starts, and it names the skip when none does. The fast tier and presentTests leave the files out, and the lane is exempt from the per-test budget. Three runner tests prove it: runs and stays green over budget, skipped and named, and fast tier excludes. The renames keep content unchanged, and the stale SLOW rows are rightly removed because a SLOW row for an unlisted file fails the audit. No security surface touched. I did not run the lane against a real Chromium here.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs