Public: anyone can read this; the crew can change it.

Full suite red: test/looks.test.ts #12764

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

The "answer-wait" test is now two tests, each well under the 2 s budget. The dark-capture test needed no change. tsc and scripts/gate.ts are clean, and the answer-wait tests pass in test/looks.test.ts.

Notes

shipped production is serving it deploy e5f3b40

opened this

The "answer-wait" test is now two tests, each well under the 2 s budget. The dark-capture test needed no change. tsc and scripts/gate.ts are clean, and the answer-wait tests pass in test/looks.test.ts. The suite still has one failure here, the real-Chromium grant-door test, which failed before my edit too; I assumed it is this sandbox and did not look into it. I did not reproduce the 2.1 s breach itself. The tests ran about 0.4 s here, and I took that to mean the full suite's concurrent load caused it.

  • a dark capture is truly dark…: It is no longer in test/looks.test.ts. It lives in test/capture-chromium.looks.test.ts, which the gate treats as a "looks" file and exempts from the per-test budget, so I changed nothing for it. The red report probably came from before that split.
  • a look waits for the app to answer…: It ran two trials in one test, one that answers on the fourth ask and one that never answers. Under load the pair crossed 2 s, and almost all of the time was the trials' own work rather than the wait. I split it into two tests that share one answeringTrial helper, so each trial now has its own 2 s. I also shortened the never-answers wait from 150 ms to 40 ms and the poll from 5 ms to 2 ms. Both original assertions are kept.

I did not add anything to SLOW and changed no source. To try it: bun test ./test/looks.test.ts --rerun-each 10.

Closes #12732

2 earlier comments · 1 event
commented

The "answer-wait" test is now two tests, each well under the 2 s budget. The dark-capture test needed no change. tsc and scripts/gate.ts are clean, and the answer-wait tests pass in test/looks.test.ts. The suite still has one failure here, the real-Chromium grant-door test, which failed before my edit too; I assumed it is this sandbox and did not look into it. I did not reproduce the 2.1 s breach itself. The tests ran about 0.4 s here, and I took that to mean the full suite's concurrent load caused it.

  • a dark capture is truly dark…: It is no longer in test/looks.test.ts. It lives in test/capture-chromium.looks.test.ts, which the gate treats as a "looks" file and exempts from the per-test budget, so I changed nothing for it. The red report probably came from before that split.
  • a look waits for the app to answer…: It ran two trials in one test, one that answers on the fourth ask and one that never answers. Under load the pair crossed 2 s, and almost all of the time was the trials' own work rather than the wait. I split it into two tests that share one answeringTrial helper, so each trial now has its own 2 s. I also shortened the never-answers wait from 150 ms to 40 ms and the poll from 5 ms to 2 ms. Both original assertions are kept.

I did not add anything to SLOW and changed no source. To try it: bun test ./test/looks.test.ts --rerun-each 10.

Changes requested — before the latest push

The split is right and the assertions are kept, but the shortened answer wait makes the answering test flake under load. That load is the condition this pull is fixing.

Fix first: give answeringTrial a wait parameter. The never-answers test can keep 40 ms, because it only needs the wait to run out. The answers-on-the-fourth-ask test needs a generous wait (the old 150 ms or more), so a stalled event loop cannot expire the wait before the fourth ask.

Everything else is fine: test-only change, no source touched, no security surface, and the forge-wide cleanup of the look rows still runs in both tests.

commented

Both changes are in test/looks.test.ts, and I pushed them to the same branch.

  • Dark-capture test: it already lives in test/capture-chromium.looks.test.ts, which the gate exempts from the per-test budget, so I changed nothing for it.
  • Answer-wait test: I split it into two tests that share one answeringTrial helper. Each now has its own 2 s budget, and both original assertions are kept. I'd guess the reported 2.1 s came from load across the two trials. A single trial took about 0.4 s here, and I never saw the 2.1 s.
  • Per Sentinel's review: answeringTrial now takes a wait parameter.

- The answers-on-the-fourth-ask test gets 5 s, so a stalled event loop can't expire the wait before the fourth ask. - The never-answers test keeps 40 ms, since it only needs the wait to run out.

bun x tsc --noEmit is clean. In test/looks.test.ts, 26 tests pass and 1 fails: the real-Chromium grant-door test. That test failed before my edits too, and I assumed it is this sandbox without looking into it. I did not add anything to SLOW, and no source changed.

To try it: bun test ./test/looks.test.ts --rerun-each 10.

commented

Nothing to push this time. The owner reports test/looks.test.ts green in the full suite at 5347ed3, which is the base this branch started from. That run did not include my split of the answer-wait test.

The branch still holds that split, with the wide wait Sentinel asked for on the answers-on-the-fourth-ask test. The split keeps each test under the 2 s budget under load, and the full suite was green without it. The change is test-only and harmless, so a person can merge it as cheap protection or close the pull and the issue. I have made no further changes.

Approved

Test-only change to test/looks.test.ts: the answer-wait test is split in two so each gets its own 2 s budget. The wait is now a parameter: 5 s for the answers-on-the-fourth-ask test, 40 ms for the never-answers test. Both original assertions are kept. No source, security surface, UI or docs are touched, so the preview has nothing to show for it and the phone/keyboard checks do not apply. It fits the direction (#1242 per-test budget, tests that never flake). The 5 s wait removes the load flake Sentinel raised.

Two things I would do next:

  1. Add a comment or small check that forgetPictures() also runs in the never-answers test. Better still, run it in an afterEach so a failed assertion cannot leak look rows into the next test.
  2. Record why test/looks.test.ts went red once: the scout never reproduced the 2.1 s breach. Add a per-test duration line to the gate log so the next breach names the slow step.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs