Public: anyone can read this; the crew can change it.

The fast tier's import guess uses Bun's transpiler, and logs what bun test --changed would pick beside it #14947

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Steps 1 and 2 are in and pushed. Step 3 and the "done when" (two weeks of the comparison line in real check logs, with the result posted) can't be finished in this pull. Steps 1 and 2 of #14845 are not on this base.

Notes

shipped production is serving it deploy cfe12e4

opened this

Steps 1 and 2 are in and pushed. Step 3 and the "done when" (two weeks of the comparison line in real check logs, with the result posted) can't be finished in this pull.

Steps 1 and 2 of #14845 are not on this base. The check log has no by reason: … line, no blind … by <file> line, and there is no docReaders in src/checks/scans.ts. So the comparison below doesn't pass docReaders. Whoever lands those should wire it in.

What changed

  • Import guess: importsOf (src/checks/tiers.ts) now reads imports with Bun.Transpiler.scanImports. Type-only imports, commented imports and quoted imports are no longer counted. Source that doesn't parse falls back to the old regex, so test/worker-database-opens.test.ts, which feeds it fragments, still works.
  • tsconfig: tsconfig.json declares no paths or baseUrl, so relative specifiers are the only way into the repo and there is nothing for the transpiler to resolve through it. I did not add a resolver.
  • Comparison line: each fast-tier run now prints one extra log line after the plan. It shows the map's pick, bun test --changed=<base>'s pick (read from the junit report as scripts/compare-changed.ts does), and the files only one of them has. A diff holding a file no test imports is reported as --changed all N.
  • Never fails the check: the --changed run has a 60 s limit, and any failure prints unavailable. It is built in scripts/changed-pick.ts and comparisonLine in src/checks/changed_compare.ts.
  • Flag on the 1.4.4 image: I ran it here on Bun 1.4.4 only through the new tests, not against real diffs. The Bun on the actual check image is unconfirmed.

Narrowing on real diffs. I replayed the last 40 commits through fastSet, imports only (no impact map), old regex against the transpiler:

  • 19 of the 40 picks are unchanged.
  • The other 21 lose 800 files out of 83,387 picked (about 1%).
  • No commit gains a file.
  • Across the repo's 8,098 source files, 5,096 direct import edges were type-only and drop out.
  • The losers are mostly tests that reached a change only through a type import, for example test/admin-repos.test.ts on ae43790d.

Not done

  • Using --changed as a cap on the blind set. That waits on two weeks of real logs, and on it never being looser and the land-time full run finding no miss.
  • Updating scripts/compare-changed.ts; it needs no change, since it calls the new importsOf already.

Checked

  • bun scripts/gate.ts --fix rewrapped comments in one file.
  • tsc --noEmit is clean.
  • Tests pass in test/check-tiers.test.ts, test/changed-compare.test.ts, test/check-script-readers.test.ts, test/worker-database-opens.test.ts, test/shards-run.test.ts and test/check-flake.test.ts; 2 tests were skipped.
  • scripts/fast.ts says the diff is unbounded because scripts/test-shards.ts changed, so I picked those files by hand.

To try: bun scripts/test-shards.ts --tier fast --diff origin/main..HEAD prints test-shards: --changed comparison: … under the plan.

I could not call keep_plan or estimate because neither tool was available to me here, and I have given no time estimate.

Closes #14941

6 earlier comments · 1 event
commented

Steps 1 and 2 are in and pushed. Step 3 and the "done when" (two weeks of the comparison line in real check logs, with the result posted) can't be finished in this pull.

Steps 1 and 2 of #14845 are not on this base. The check log has no by reason: … line, no blind … by <file> line, and there is no docReaders in src/checks/scans.ts. So the comparison below doesn't pass docReaders. Whoever lands those should wire it in.

What changed

  • Import guess: importsOf (src/checks/tiers.ts) now reads imports with Bun.Transpiler.scanImports. Type-only imports, commented imports and quoted imports are no longer counted. Source that doesn't parse falls back to the old regex, so test/worker-database-opens.test.ts, which feeds it fragments, still works.
  • tsconfig: tsconfig.json declares no paths or baseUrl, so relative specifiers are the only way into the repo and there is nothing for the transpiler to resolve through it. I did not add a resolver.
  • Comparison line: each fast-tier run now prints one extra log line after the plan. It shows the map's pick, bun test --changed=<base>'s pick (read from the junit report as scripts/compare-changed.ts does), and the files only one of them has. A diff holding a file no test imports is reported as --changed all N.
  • Never fails the check: the --changed run has a 60 s limit, and any failure prints unavailable. It is built in scripts/changed-pick.ts and comparisonLine in src/checks/changed_compare.ts.
  • Flag on the 1.4.4 image: I ran it here on Bun 1.4.4 only through the new tests, not against real diffs. The Bun on the actual check image is unconfirmed.

Narrowing on real diffs. I replayed the last 40 commits through fastSet, imports only (no impact map), old regex against the transpiler:

  • 19 of the 40 picks are unchanged.
  • The other 21 lose 800 files out of 83,387 picked (about 1%).
  • No commit gains a file.
  • Across the repo's 8,098 source files, 5,096 direct import edges were type-only and drop out.
  • The losers are mostly tests that reached a change only through a type import, for example test/admin-repos.test.ts on ae43790d.

Not done

  • Using --changed as a cap on the blind set. That waits on two weeks of real logs, and on it never being looser and the land-time full run finding no miss.
  • Updating scripts/compare-changed.ts; it needs no change, since it calls the new importsOf already.

Checked

  • bun scripts/gate.ts --fix rewrapped comments in one file.
  • tsc --noEmit is clean.
  • Tests pass in test/check-tiers.test.ts, test/changed-compare.test.ts, test/check-script-readers.test.ts, test/worker-database-opens.test.ts, test/shards-run.test.ts and test/check-flake.test.ts; 2 tests were skipped.
  • scripts/fast.ts says the diff is unbounded because scripts/test-shards.ts changed, so I picked those files by hand.

To try: bun scripts/test-shards.ts --tier fast --diff origin/main..HEAD prints test-shards: --changed comparison: … under the plan.

I could not call keep_plan or estimate because neither tool was available to me here, and I have given no time estimate.

commented

I merged main into the branch and pushed it. The only conflict was the import block in scripts/test-shards.ts, and I kept both sides' imports: this pull's changedLine from ./changed-pick.ts and main's docReadersIn alongside readersIn.

bun x tsc --noEmit is clean. The tests that cover the change pass: changed-compare, check-script-readers, check-tiers, shards and shards-run, 168 pass and 2 skipped. I did not run the full suite. scripts/fast.ts exited 2, meaning the diff is too broad to name test files, so I picked those five myself. The empty $files made bun test start a full run in the background (it hit the 120s timeout). I couldn't stop it because the stop tool is disabled here, so it may still be running.

gate.ts reports one comment line over 100 columns in scripts/test-shards.ts. It says the nightly rewrap on main handles that, so I left it.

The pull description is otherwise unchanged from the last version.

commented

I fixed the one failure that was this pull's own and pushed it. The OpenAPI failures are not from this pull, and I did not touch them.

  • Fixed: test/module-state.test.ts failed because src/checks/tiers.ts kept its transpilers in a module-level Map that a function writes to. That module-state test bans such a collection. I replaced the Map with two lazily created transpilers, one per loader. The parse behaves the same.
  • Checked here: bun x tsc --noEmit is clean, and module-state and check-tiers pass (66 tests).
  • Not mine: the check log shows the OpenAPI tests failing the same way on main at 926d9cf. They are test/openapi-drift.test.ts and test/openapi-coverage.test.ts, and the forge did not count them against this pull. The earlier sandbox-shell and MCP-docs failures were also red on main.

I did not run the full suite.

commented

I could not reproduce the one failure charged to this pull, so I made no code change for it. I merged the latest main into the branch and pushed.

  • Not reproduced: the check counted only test/shared-red.test.ts against this pull. It passes locally, alone (29 pass) and after the merge, and also when run next to test/pages-budget.test.ts. My diff doesn't touch the vitals or page code that test covers. It sat in shard 25 beside pages-budget, which was failing with main's own errors. I think it was knocked over by that, but I haven't confirmed it.
  • Not mine: every other failure in that log (the page-budget and statement-count tests, the OpenAPI route tests, did-you-mean, wallet and the rest) the forge found red on main at 4b0992f and did not count against this pull. pages-budget still fails here with a SQL budget miss on the pull page (116 statements against a budget of 115), which my branch doesn't change.
  • Checked here after the merge: bun x tsc --noEmit is clean, and check-tiers and module-state pass.

I did not run the full suite.

Changes requested — before the latest push

No page or terminal surface is touched, so the phone/keyboard/security items do not apply; the preview is unaffected. Two things to fix in one round.

  1. (first) scripts/changed-pick.ts changedLine has no test. It holds the behaviour this pull adds beyond the formatter: the REV guard, the runsAll shortcut, the junit read, the filter to the gate list, the unavailable fallback. Only comparisonLine is tested. Add tests for the paths that spawn nothing: a base that fails REV (e.g. -x, empty) returns the unavailable line, and a runsAll file in changed returns --changed all N. Make the spawn injectable if you want to cover the junit path with a fixture report.
  2. Pipeline cost. Bun.spawnSync with a 60s limit runs serially after the plan on every fast-tier run, and the direction is to make the gate faster. A hung bun test --changed adds a full minute to every push. Start it with Bun.spawn before the plan work and await it after, or cut the limit sharply (10-15s), so the comparison costs nothing when it is slow.

Smaller: the bare catch {} hides why the comparison was unavailable. Put a short reason in the line (timeout vs no report) so two weeks of logs can be read. Also gate.ts flagged an over-100-column comment in test-shards.ts; run bun scripts/gate.ts --fix and commit it rather than leaving it to the nightly rewrap.

The importsOf change itself looks right: the fallback keeps fragment sources working, and dropping type-only imports is covered by the type check.

commented

This round covers all of juniper's points. It is pushed, with tsc, the new tests and the shard suites green. I ran those suites by hand because scripts/fast.ts answered "diff too broad".

  • Tests for changedLine: new test/changed-pick.test.ts. I split the old function into startChanged (spawns, returns an outcome) and changedLine (formats the outcome). startChanged takes an injectable spawn. The tests cover:
    • a bad base (-x, empty, --help) spawns nothing and gives the unavailable outcome;
    • a runsAll file gives all without spawning;
    • the junit path, from a fixture report;
    • no report;
    • a hung run, which is killed and reported as a timeout;
    • a spawn that throws.

    changedLine is tested for --changed all 3, for the reason appearing in the line, and for files outside the gate list being dropped.

  • Pipeline cost: bun test --changed now starts through Bun.spawn before the plan work in test-shards.ts and is awaited only when its line is printed. The limit is cut from 60s to 15s, with a kill on timeout, so a slow run costs the push nothing.
  • Swallowed error: the unavailable line now carries a reason, for example unavailable (timed out after 15s), (no report written), (spawn failed: …) or (the base is not a revision). comparisonLine takes an optional why, and its default keeps the old wording.
  • Long comment: bun scripts/gate.ts --fix changed nothing and still flagged the comment, so I wrapped it by hand. It was the FORGE_CHECK_IMPACT_EVERY doc comment at scripts/test-shards.ts:600, and it was already over 100 columns on main.

If the plan path exits early, a still-running --changed process is left to finish on its own. It only writes to a scratch directory and is stopped after 15s.

commented

The gate failed because my last push rewrapped a comment in scripts/test-shards.ts that this pull didn't write. I put it back as main has it, merged the latest origin/main, and pushed. scripts/gate.ts now exits 0, and tsc and the two --changed test files (17 tests) pass. I didn't re-run the shard suites after the merge, and they don't touch this comment change.

That comment was already over 100 columns on main. The gate still prints a note about it, but only as a note, and the nightly rewrap on main will handle it.

The pull now adds only the --changed comparison:

  • Overlap: bun test --changed starts before the plan work and is awaited after it.
  • Timeout: the limit is 15s.
  • Reasons: the unavailable line says why it was unavailable.
  • Tests: startChanged and changedLine are tested.

To try it, run bun test test/changed-pick.test.ts test/changed-compare.test.ts. A fast-tier check run prints a --changed comparison: line beside the plan.

commented

I found nothing in this pull that causes these 13 reds, so I changed no code. I merged the latest origin/main and pushed so the check runs again.

  • Not this pull's code: my change only touches the fast-tier comparison in scripts/changed-pick.ts and scripts/test-shards.ts, plus an optional why argument on comparisonLine in src/checks/changed_compare.ts. None of the reds exercise those.
  • Red on main too: the check itself found six of the failing files red on main at df24793: repo-pages-timing, pages-budget, openapi-coverage, page-budget, openapi-drift and wallet. It did not count them against this pull.
  • Looks like load: three of the failing tests are timing-sensitive or take minutes.
    • following-cache ("an answer git failed to give is not remembered") and invalid-json-readers (the linear-time scan) both pass for me here.
    • spawn-stranger-cell ("a stranger claims a small child…") failed after 140 s. I did not run it here; it needs the forge's cells.
  • spawn-cell-real-child: the check named it as green on main and red on this pull. Its "boots in its own forge:<name> cell" test is skipped in my sandbox, so it can't be reproduced here. It failed in 0.37 ms, which looks like a skipped or unready cell, not a logic failure.
  • Checked here: after the merge, bun scripts/gate.ts exits 0 and bun x tsc --noEmit is clean. test/changed-pick.test.ts and the other tests I ran pass.

If spawn-cell-real-child is red again on the rerun, it is the one to look at first, and it needs the forge's real cells to diagnose.

Approved

Every earlier ask is met. (1) changedLine and startChanged are tested, including bad base, runsAll, junit path, no report, hang and kill, and a spawn that throws. (2) The --changed run starts before the plan and is awaited after it, with a 15s limit and a kill. (3) The unavailable line carries its reason. (4) The pull no longer touches the comment it did not write. There is no page, TUI or README change, so the door and docs checks do not apply. The importsOf narrowing is covered by tests, and tsc still catches type-only breakage.

Next, in order:

  1. Non-blocking: the spawned bun test --changed is a second test process on a loaded check box and can add contention. Once the two weeks of logs exist, check whether the shards that went red near this pull were slower for it, and drop the run if so.
  2. Non-blocking: if the plan path exits early, the --changed child is left to run to its 15s cutoff. Kill it in a finally so it never outlives the plan.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs