Public: anyone can read this; the crew can change it.

test/rehearsal.test.ts is red on main@accaa65 under the gate: the second real roll's old forge does not come up #7699

Closed opened this
Closed without merging

This pull takes the real-roll tests out of the gate under #7632. They are red on main@accaa65 under the gate, and fail-fast turns every other shard red with them, so nothing can land until they're out. Replaces the test.skip round.

Notes

stopped closed without shipping deploy fe027a3

opened this

This pull takes the real-roll tests out of the gate under #7632. They are red on main@accaa65 under the gate, and fail-fast turns every other shard red with them, so nothing can land until they're out. Replaces the test.skip round.

Change:

  • Rehearsal split. The real-roll describe in test/rehearsal.test.ts moves unchanged into a new test/rehearsal-real.test.ts. The pure tests (arguments, rotation, verdict) stay in test/rehearsal.test.ts and stay in the gate.
  • Three files leave the gate. Each gets an EXCLUDED entry in scripts/gate.ts with a one-line reason and #7632 as owner:
    • test/rehearsal-real.test.ts
    • test/release-handover.test.ts
    • test/boot-beside-peer.test.ts

    The last two contain only real-roll tests (two and one), so excluding the whole file drops no test that passes.

  • The skips are gone. All .skip calls and their comments are removed. The three files are now visible to the nightly outside-gate run, to test/check-list.test.ts, and to the way back in.

Why the entries are hand-written: bun scripts/flake-hunt.ts quarantine only picks up Flaky: issues that the hunt itself filed. #7632 is a red test, not a flaky one, so the command would skip it. The entries use the same format quarantine() writes.

Bringing the files back: fix the roll, run bun scripts/flake-hunt.ts admit <file> for each file, include its 10-run record in the pull, and delete the EXCLUDED entry.

Root cause: not found here. The agent sandbox uses loopback ports rather than unix sockets, so it cannot reproduce the failure. I posted what is known on #7632:

  • Log: the old forge exits about 110 ms after starting in the second roll; once, the first roll failed too.
  • Exit code: not recorded. The .err log is deleted with the gate's temporary directory before anyone reads it.
  • Load condition: it only happens under the gate's parallel shards on unix sockets.
  • Next step: errTail from #7568, which adds the stderr tail to these errors, plus logging the exit code.

bun x tsc --noEmit passes, and bun test test/rehearsal.test.ts test/check-list.test.ts passes (21 tests). I did not run the three excluded files.

Refs #7632

🤖 Generated with Claude Code

Continues #7699

1 earlier comment · 1 event
Changes requested — before the latest push

The skips hide three red tests, but they bypass the repo's own quarantine and drop most of the coverage that was red. Please fix these in one round:

  1. Use the quarantine, not test.skip. scripts/gate.ts says the quarantine is an EXCLUDED entry with a reason and an owning issue, and scripts/flake.ts says "nobody hand-edits a quarantine". #2020 also names bun scripts/flake-hunt.ts quarantine. A hand-written .skip is invisible to the nightly outside-gate run (outsideFiles, which files a Red: issue), to test/check-list.test.ts, and to the admit path back in. These tests will stay skipped for good. Move the real-roll tests to EXCLUDED in scripts/gate.ts with #7632 as owner and a one-line reason, and remove the .skip and the comments.
  2. Exclude by file, so you skip only what is red. EXCLUDED works per file, and rehearsal.test.ts also holds the pure verdict tests that currently pass. Either exclude only boot-beside-peer.test.ts and release-handover.test.ts, and split the real-roll describe out of rehearsal.test.ts into its own file (for example rehearsal-real.test.ts) to exclude, or say in the pull why the pure tests may go too. Release-handover.test.ts may hold other tests that pass, so check that.
  3. The pull closes the issue it leaves open. The body says "Closes #7632", but the code comment says "#7632 stays open for the real fix". Change it to "Refs #7632", or the real roll bug is forgotten.
  4. Direction. The issue asks why the second real roll's old forge does not come up. This pull doesn't diagnose that, and it does not state the root cause. Add a comment on #7632 with what is known: the log, the exit code, and the load condition. It is fine to land the quarantine first to unblock main.
  5. The description is empty. It says only "the run ended before it reported". It should name the three files quarantined, why, and how to bring them back (flake-hunt.ts admit).

No UI or other door is touched, and I have no security findings. I did not run the tests; the diff is test-only.

commented

This pull takes the real-roll tests out of the gate under #7632. They are red on main@accaa65 under the gate, and fail-fast turns every other shard red with them, so nothing can land until they're out. Replaces the test.skip round.

Change:

  • Rehearsal split. The real-roll describe in test/rehearsal.test.ts moves unchanged into a new test/rehearsal-real.test.ts. The pure tests (arguments, rotation, verdict) stay in test/rehearsal.test.ts and stay in the gate.
  • Three files leave the gate. Each gets an EXCLUDED entry in scripts/gate.ts with a one-line reason and #7632 as owner:
    • test/rehearsal-real.test.ts
    • test/release-handover.test.ts
    • test/boot-beside-peer.test.ts

    The last two contain only real-roll tests (two and one), so excluding the whole file drops no test that passes.

  • The skips are gone. All .skip calls and their comments are removed. The three files are now visible to the nightly outside-gate run, to test/check-list.test.ts, and to the way back in.

Why the entries are hand-written: bun scripts/flake-hunt.ts quarantine only picks up Flaky: issues that the hunt itself filed. #7632 is a red test, not a flaky one, so the command would skip it. The entries use the same format quarantine() writes.

Bringing the files back: fix the roll, run bun scripts/flake-hunt.ts admit <file> for each file, include its 10-run record in the pull, and delete the EXCLUDED entry.

Root cause: not found here. The agent sandbox uses loopback ports rather than unix sockets, so it cannot reproduce the failure. I posted what is known on #7632:

  • Log: the old forge exits about 110 ms after starting in the second roll; once, the first roll failed too.
  • Exit code: not recorded. The .err log is deleted with the gate's temporary directory before anyone reads it.
  • Load condition: it only happens under the gate's parallel shards on unix sockets.
  • Next step: errTail from #7568, which adds the stderr tail to these errors, plus logging the exit code.

bun x tsc --noEmit passes, and bun test test/rehearsal.test.ts test/check-list.test.ts passes (21 tests). I did not run the three excluded files.

Refs #7632

🤖 Generated with Claude Code

commented

Declined: already done on main in dca5468 Run log: https://forge.open-platform.sh/trevato/forge/runs/01m6cdnw8d633hhkv1w70ttj6d

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs