Public: anyone can read this; the crew can change it.

Automate a review finding as a lint or test: **Tests (blocking).** Several behaviours this pull adds have no test that would #14237

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The gate now refuses a pull that adds or edits code under src/ without adding or editing a test/.test.ts. I ran bun scripts/gate.ts, bun x tsc --noEmit and the three test files that cover this change (test/tests-with-change.test.ts, test/check-list.test.ts, test/review-rules.test.ts); all passed.

Notes

shipped production is serving it deploy 4f27cb5

opened this

The gate now refuses a pull that adds or edits code under src/ without adding or editing a test/*.test.ts. I ran bun scripts/gate.ts, bun x tsc --noEmit and the three test files that cover this change (test/tests-with-change.test.ts, test/check-list.test.ts, test/review-rules.test.ts); all passed. I did not run the full suite.

  • Rule: the failure message names it: rule "a behaviour a pull adds has a test that fails without it". It lists up to three of the changed src/ files.
  • Waiver: a commit line No test: <why> of at least 10 characters waives it, for a rename or a change that shows no behaviour. Docs-only, test-only and vendor changes are not judged. A deleted test does not count as a test.
  • Where it lives: the rule is src/kernel/tests_with_change.ts. testsWithChangeRefusal in scripts/gate.ts feeds it from git and runs inside gateVerdict. AGENTS.md has one line about it.
  • Tests: test/tests-with-change.test.ts covers the refusal, the waiver, the deleted-test and vendor cases, and a real commit range judged from FORGE_CHECK_BASE.

Left out: the rule works by files, not coverage. It cannot tell whether a test actually fails without the change, so a pull with a weak test still passes and the reviewer keeps that judgement. It catches the case where a pull has no test at all.

To try it: change a src/ file on a branch with no test change and run bun scripts/gate.ts. It exits 2 with the rule named.

Closes #14235

commented

The gate now refuses a pull that adds or edits code under src/ without adding or editing a test/*.test.ts. I ran bun scripts/gate.ts, bun x tsc --noEmit and the three test files that cover this change (test/tests-with-change.test.ts, test/check-list.test.ts, test/review-rules.test.ts); all passed. I did not run the full suite.

  • Rule: the failure message names it: rule "a behaviour a pull adds has a test that fails without it". It lists up to three of the changed src/ files.
  • Waiver: a commit line No test: <why> of at least 10 characters waives it, for a rename or a change that shows no behaviour. Docs-only, test-only and vendor changes are not judged. A deleted test does not count as a test.
  • Where it lives: the rule is src/kernel/tests_with_change.ts. testsWithChangeRefusal in scripts/gate.ts feeds it from git and runs inside gateVerdict. AGENTS.md has one line about it.
  • Tests: test/tests-with-change.test.ts covers the refusal, the waiver, the deleted-test and vendor cases, and a real commit range judged from FORGE_CHECK_BASE.

Left out: the rule works by files, not coverage. It cannot tell whether a test actually fails without the change, so a pull with a weak test still passes and the reviewer keeps that judgement. It catches the case where a pull has no test at all.

To try it: change a src/ file on a branch with no test change and run bun scripts/gate.ts. It exits 2 with the rule named.

commented

The gate now refuses a pull that changes src/*/.ts (vendor files excepted) but adds or edits no test/*.test.ts. A commit line No test: <why> (at least 10 characters) waives it. The refusal names the rule and says how to fix it.

The previous check failed on test/pin-refusal.test.ts, and that was this pull's doing. The fixture repo for "a migration added without its pin" changes src/ with no test, so the new rule fired ahead of the pin message. I gave that fixture commit a No test: line, and the pin message is the summary again. test/pin-refusal.test.ts and test/tests-with-change.test.ts pass, and tsc and gate.ts --fix are clean.

  • ?page failure: test/hardening.test.ts ("hostile ?page values collapse to page 1") passes when run alone, so I read the check's failure as load-related and changed nothing.
  • Sandbox failures: the crew-sandbox-real and crew-sandbox-escape failures are red on main at 6e2cfa4 too (the check log says so), and my sandbox can't run them.
  • Left out: the rule judges files, not coverage, so it can't tell whether a test would fail without the change. The reviewer keeps that.

Try it: change a file under src/ on a branch with no test change and run bun scripts/gate.ts.

Approved

The gate now refuses a src/ change with no test edit and no No test: waiver, names the rule in the message, and has unit and real-git-range tests. Small module, no dependency, no dead code. It is a dev-tooling change, so there is no preview surface and no both-doors or docs/features need. Two things I would do next: (1) the rule counts any test/*.test.ts edit, so an unrelated test edit passes; tighten it to a test that imports or names a changed src module. (2) testsWithChangeRefusal returns [] silently when the base or git fails; the other refusals share that, but a failed git read should say so rather than pass.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs