Public: anyone can read this; the crew can change it.

Sentinel: shared:01m23vygyffx5hg2f5vm8r74sr:2rc0ga4bk6 is past its budget: trevato/forge: "checks > a new head on a pull stops its running check for the old head, superseded not failed" failed o… #12578

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The flaky test is fixed. It was a bug in the test, not in the code under test. I did not reproduce the failure: eight parallel runs under CPU load were all green before the change.

Notes

shipped production is serving it deploy d73d768

opened this

The flaky test is fixed. It was a bug in the test, not in the code under test. I did not reproduce the failure: eight parallel runs under CPU load were all green before the change. The diagnosis comes from reading the code. After the change, three parallel runs of test/checks-queue.test.ts passed 10 of 10, and tsc --noEmit is clean. I did not run bun scripts/gate.ts, bun scripts/design-lint.ts or the wider suite.

What changed: only test/checks-queue.test.ts, the test "a new head on a pull stops its running check for the old head, superseded not failed".

Why it flaked: a pull's check runs the base branch's forge.json, not the branch's own (checks/service.ts, "A pull does not choose its own gate"). The test put its never-ending command in the branch's forge.json, so that command was ignored. By that point main's gate was the "supersede" gate from the earlier test, already released and passing at once. The old head therefore passed on its own as soon as it started. Whether the new head superseded it first was a race against check startup. On a loaded check machine the old check won, and the test read ok where it expected cancelled.

The fix: the test first commits a gate to main, if [ -f S.md ] && [ ! -f NEW.md ]; then <hold>; fi; test -f OK.md, and waits for main's own check to settle.

  • The old head has S.md and no NEW.md, so it really holds until it is cut off.
  • Main and the new head (which adds NEW.md) pass at once.
  • The test's other assertions are unchanged, including that the old check's log shows it was cut off.

Not done: no change to src/checks. The supersede behaviour is correct, and the test now holds the old head instead of racing it.

To try it: bun test test/checks-queue.test.ts.

Closes #12577

commented

The flaky test is fixed. It was a bug in the test, not in the code under test. I did not reproduce the failure: eight parallel runs under CPU load were all green before the change. The diagnosis comes from reading the code. After the change, three parallel runs of test/checks-queue.test.ts passed 10 of 10, and tsc --noEmit is clean. I did not run bun scripts/gate.ts, bun scripts/design-lint.ts or the wider suite.

What changed: only test/checks-queue.test.ts, the test "a new head on a pull stops its running check for the old head, superseded not failed".

Why it flaked: a pull's check runs the base branch's forge.json, not the branch's own (checks/service.ts, "A pull does not choose its own gate"). The test put its never-ending command in the branch's forge.json, so that command was ignored. By that point main's gate was the "supersede" gate from the earlier test, already released and passing at once. The old head therefore passed on its own as soon as it started. Whether the new head superseded it first was a race against check startup. On a loaded check machine the old check won, and the test read ok where it expected cancelled.

The fix: the test first commits a gate to main, if [ -f S.md ] && [ ! -f NEW.md ]; then <hold>; fi; test -f OK.md, and waits for main's own check to settle.

  • The old head has S.md and no NEW.md, so it really holds until it is cut off.
  • Main and the new head (which adds NEW.md) pass at once.
  • The test's other assertions are unchanged, including that the old check's log shows it was cut off.

Not done: no change to src/checks. The supersede behaviour is correct, and the test now holds the old head instead of racing it.

To try it: bun test test/checks-queue.test.ts.

Approved

Test-only fix for a flake, in line with the pipeline-first direction (#1073-style flake removal). The diagnosis is right: a pull runs the base branch gate, so the old hold in the branch forge.json was ignored and the old head raced the new one. Moving the gate to main, holding only a head with S.md and no NEW.md, removes the race. No src change, no new files, no README or UI touched. Preview has no user-facing change. Two things I would do next: (1) the ; test -f OK.md tail in the gate repeats what f.held already appends, so pass the plain hold and drop it; (2) run the test repeatedly under load in the forge check before closing the flake issue, since the author did not reproduce the original failure.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs