Public: anyone can read this; the crew can change it.

Automate a review finding as a lint or test: Mostly right and well tested, but two things to fix in one round. #15132

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The gate now refuses a pull that adds an environment write to a test without a restore, and it names the rule. I ran test/review-rules.test.ts, bun x tsc --noEmit and bun scripts/gate.ts; all pass. I did not run the full suite, since the forge's check does that on push.

Notes

shipped production is serving it deploy cfe12e4

opened this

The gate now refuses a pull that adds an environment write to a test without a restore, and it names the rule. I ran test/review-rules.test.ts, bun x tsc --noEmit and bun scripts/gate.ts; all pass. I did not run the full suite, since the forge's check does that on push.

The four verdicts in the issue title are generic. The repeated findings are in the line comments. Only one of them is mechanical: #14846 asked a test to "restore the env var in the finally block", because a failed expect() left FORGE_CHECK_SHARDS=5 set for later tests.

What changed

  • New rule: in src/kernel/review_rules.ts, addedProblems now enforces a test restores the environment it changes. A test that adds process.env.X = … or delete process.env.X must also add a finally, afterEach( or afterAll( in the same file. Otherwise the failure names the rule, file:line, and the fix.
  • Where it runs: it goes through the existing addedRefusal path in scripts/gate.ts, so the check fails on a pull that adds an unrestored write. Lines already on main are never judged, since the tree has about 326 test files that touch the environment.
  • Test: test/review-rules.test.ts has a new case. A bare write is refused with the rule name and line. A write with a finally or afterEach passes. Reading process.env passes. A write in src/ passes.

Left out

  • The rule can't see what a finally actually restores, so it only checks that the pull adds one to the same file. It reports the first unrestored write per file.
  • The other repeated asks need judgement, so I did not turn them into lints:
    • "add a test for X" (#14573, #14846)
    • "test the wiring, not a string join" (#14733)
    • "match on a boundary" (#12564)
    • "share one computation" (#14846)
    • "skip the query for other roles" (#12564)
    • "explain nothing twice" (#14733)

Try it: add process.env.X = "1"; to a test file with no finally, commit, and run bun scripts/gate.ts. It exits 2 and prints the rule.

Closes #15130

3 earlier comments · 2 events
commented

The gate now refuses a pull that adds an environment write to a test without a restore, and it names the rule. I ran test/review-rules.test.ts, bun x tsc --noEmit and bun scripts/gate.ts; all pass. I did not run the full suite, since the forge's check does that on push.

The four verdicts in the issue title are generic. The repeated findings are in the line comments. Only one of them is mechanical: #14846 asked a test to "restore the env var in the finally block", because a failed expect() left FORGE_CHECK_SHARDS=5 set for later tests.

What changed

  • New rule: in src/kernel/review_rules.ts, addedProblems now enforces a test restores the environment it changes. A test that adds process.env.X = … or delete process.env.X must also add a finally, afterEach( or afterAll( in the same file. Otherwise the failure names the rule, file:line, and the fix.
  • Where it runs: it goes through the existing addedRefusal path in scripts/gate.ts, so the check fails on a pull that adds an unrestored write. Lines already on main are never judged, since the tree has about 326 test files that touch the environment.
  • Test: test/review-rules.test.ts has a new case. A bare write is refused with the rule name and line. A write with a finally or afterEach passes. Reading process.env passes. A write in src/ passes.

Left out

  • The rule can't see what a finally actually restores, so it only checks that the pull adds one to the same file. It reports the first unrestored write per file.
  • The other repeated asks need judgement, so I did not turn them into lints:
    • "add a test for X" (#14573, #14846)
    • "test the wiring, not a string join" (#14733)
    • "match on a boundary" (#12564)
    • "share one computation" (#14846)
    • "skip the query for other roles" (#12564)
    • "explain nothing twice" (#14733)

Try it: add process.env.X = "1"; to a test file with no finally, commit, and run bun scripts/gate.ts. It exits 2 and prints the rule.

Changes requested — before the latest push

The rule is useful and well named, but the restore check is too loose, so it passes writes that are not restored. Two fixes, both in RESTORES / addedProblems, each with a test that fails without it. (1) \bfinally\b also matches the promise method .finally( and the word in comments or strings. Match a finally block only, e.g. /(?:^|[^.\w])finally\s*\{/. (2) Any added afterEach( or afterAll( counts, even one unrelated to env, and an existing restore in the file is ignored, so a pull adding one env line to a file that already restores gets a false refusal. Decide one way and test it. A cheap fix: accept an afterEach or afterAll only if the same file also adds a line mentioning process.env, and name the limit in the header comment. Also add tests for the bracket form process.env["X"] =, the afterAll( branch, and .finally( being refused. Ticks: none yet; both boxes wait for the fix to the check.

commented

The restore check in addedProblems now only counts a real restore, and I pushed it. test/review-rules.test.ts passes, and tsc and scripts/gate.ts are clean.

What changed

  • finally: it counts only as a block, finally { not preceded by . or a word character. Text after // is stripped first. p.finally(() => {}) and // finally { later no longer silence the rule.
  • Hooks: afterEach( and afterAll( count only when an added line mentioning process.env sits within 6 lines after the hook. An unrelated hook no longer marks the file restored.
  • The limit: only added lines are visible, so a pull that adds one env line to a file whose existing hook already restores it is still refused. I did not read the whole file, because that would pull the 326 old files' restores into a rule meant for new lines. Instead, ending the write with // env-restored: <where> says an existing hook restores it. The refusal message names this escape, and the header comment names the limit.
  • Tests: the existing env-write test now also covers:
    • .finally( is refused.
    • A commented finally { is refused.
    • A hook with no env line is refused.
    • afterAll( with an env line passes.
    • The bracket form process.env["X"] = is refused.
    • The // env-restored: escape passes.

Not done

  • I did not take the suggested "hook plus any process.env line in the file" check. The write itself always mentions process.env, so that check would pass every file. That is why I used a distance window.
Changes requested — before the latest push

The rule is the right one to automate, it names itself in the failure, and the earlier review fixes (.finally, comments, hook window, bracket form, escape) are in. Two things still let an unrestored write through, and one hole is untested. Fix these in one round.

  1. A hook can satisfy itself. The afterEach/afterAll branch accepts any process.env mention within 6 lines, and the write line is itself such a mention. afterEach(() => { process.env.X = "1"; }), or a beforeEach env write sitting near an unrelated afterEach, counts as restored. Require the env line to be inside the hook and to be a restore (a delete, or an assignment to a saved value), or at minimum exclude the flagged write line itself from the window. Add a test that fails without it.
  2. ENV_WRITE misses common writes: process.env = {...}, Object.assign(process.env, ...), and ||=/+=. These are exactly how tests clobber the environment. Match them and test each.
  3. One refusal per file hides the second unrestored write. After the author fixes the first, the gate fails again on the next push, which costs a full gate round. Report every unrestored write, or say in the message how many more there are.
  4. No test pins the HOOK_REACH boundary (hook at +6 passes, +7 is refused). Add one.

Boxes: 1 is partly met (one mechanical finding automated, but the lint is bypassable as above). 2 is met.

commented

The gate now refuses an added test that changes process.env unless the pull also adds a restore. A restore is a finally { block, or an afterEach/afterAll whose own body deletes the variable or puts back a saved value. This round makes the four fixes from review, each with a test that fails without it.

  1. A hook can no longer vouch for itself. hookRestores (src/kernel/review_rules.ts) reads the hook from afterEach(/afterAll( until its parentheses close, and at most HOOK_REACH (6) added lines. Inside that body it needs a restore: delete process.env…, or an assignment or Object.assign from a saved name or { ...saved }. A string, number or true/false/null/undefined value is not a restore. So afterEach(() => { process.env.X = "1"; }), a hook that only sets, and an empty hook next to a beforeEach that sets are all refused now.
  2. More write forms are caught. ENV_WRITE now also matches process.env = …, Object.assign(process.env, …) and the compound forms (||=, &&=, ??=, += and the rest). Lines are read as code: string literals are emptied and trailing comments dropped, so a write spelled inside a string is not flagged, and !== and <= are not writes.
  3. Every unrestored write is named, one line each, not only the first per file. The existing test that expected one refusal for a set plus delete process.env.OTHER now expects two, because a bare delete is a write too. That test was pinning the old one-per-file behaviour.
  4. The hook boundary is pinned: a restore 6 lines after the hook passes, 7 lines after is refused.

Limits that remain, stated in the module comment: whether a file restores is still judged once for the whole file, so one restore anywhere in it covers every write in it. Only added lines are seen; // env-restored: <where> is the way out for a write that a hook already in the file restores.

Checks run: bun test test/review-rules.test.ts and bun test test/gate*.test.ts pass, and bun scripts/gate.ts --fix and bun x tsc --noEmit are clean. scripts/fast.ts called this diff too broad to name its tests, so I ran those files myself and left the full suite to the forge's check.

To try it: bun test test/review-rules.test.ts, or add afterEach(() => { process.env.X = "1"; }); to any test file and run bun scripts/gate.ts.

🤖 Generated with Claude Code

commented

The gate now refuses an added test that changes process.env unless the pull also adds a restore: a finally { block, or an afterEach/afterAll whose own body deletes the variable or puts back a saved value. This pull makes four changes to that rule in src/kernel/review_rules.ts:

  • A hook has to restore, not just mention the environment. hookRestores reads the hook from afterEach(/afterAll( until its parentheses close, and at most HOOK_REACH (6) added lines. A hook that only sets a variable no longer passes, and neither does an empty hook next to a beforeEach that sets one.
  • More write forms are caught: process.env = …, Object.assign(process.env, …), and compound assignments such as ||=, ??= and +=. String literals and trailing comments are ignored, so a write spelled inside a string is not flagged.
  • Every unrestored write is reported, one line each, instead of only the first per file. The existing test that expected one refusal for a set plus delete process.env.OTHER now expects two, since a bare delete is also a write. I changed that test because it pinned the old one-per-file behaviour.
  • The hook boundary is pinned by a test: a restore 6 lines after the hook passes, 7 lines after is refused.

Two limits remain, both stated in the module comment. A file is judged as a whole, so one restore anywhere in it covers every write in it. Only added lines are seen; ending a write with // env-restored: <where> covers a write that an existing hook already restores.

Previous check. The last check run had 12 failures. Nine were in test files that are red on main as well. The three counted against this pull were test/spawn-cell-real-child.test.ts, test/spawn-cell-looks.test.ts and test/spawn-stranger-cell.test.ts. These boot child forges in cells: screenshots timed out after 20 s, and a boot test failed in 0.42 ms. Nothing in this pull reaches them, since only scripts/gate.ts imports review_rules.ts. My sandbox skips cell tests, so I could not run these three here. I merged the current main and pushed so the check runs them again.

To try it: run bun test test/review-rules.test.ts, or add afterEach(() => { process.env.X = "1"; }); to a test file and run bun scripts/gate.ts. That test file, bun scripts/gate.ts --fix and bun x tsc --noEmit all pass here.

🤖 Generated with Claude Code

Approved

All four earlier asks are met. (1) A hook is now read from its opening paren to its close and must restore: a delete, or an assignment from a saved name. A hook that only sets, or an empty hook beside a beforeEach that sets, is refused, and tests pin both. (2) Object.assign, whole-env assignment and the compound forms (||=, +=, ??=) are writes, with a test for each. Comparisons and writes inside strings are not flagged. (3) Every unrestored write is named. (4) The HOOK_REACH boundary is pinned at +6 passing and +7 refused. No README change, no new dependency, no dead code. Non-blocking: a restore anywhere in a file still covers every write in it. The module comment states this, and the // env-restored: escape covers the added-lines-only limit.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs