Public: anyone can read this; the crew can change it.

The landsim corpus test runs under the 2 s per-test budget, and test/landsim.test.ts leaves SLOW #12488

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Every test in `test/landsim.test.ts` is now under 2 s alone here, and the `SLOW` entry is gone from `scripts/gate.ts`. `bun scripts/gate.ts` and `bun x tsc --noEmit` are green, and the file's 20 tests pass. I did not run the full suite; the forge does that on push.

shipped production is serving it deploy 9cda780

opened this

Every test in test/landsim.test.ts is now under 2 s alone here, and the SLOW entry is gone from scripts/gate.ts. bun scripts/gate.ts and bun x tsc --noEmit are green, and the file's 20 tests pass. I did not run the full suite; the forge does that on push.

What changed

  • One test a seed: the single 18 s corpus test is now one test per seed. The additive-merge pair check is one test per landing order, and the order-independence check is its own test.
  • Determinism: it runs seed 34 twice, one run per test, and the second run compares signature and final tree against the first. This replaces seed 2.
  • Quick seeds: they are now 1, 3, 5, 8 instead of 1 to 4, chosen because they land fastest. Seed 2 took about 2.3–3 s alone. Seeds 1 and 5 cover additive merges and seed 8 covers order independence.
  • Landed check: the old corpus.landed > 0 check is now a last test that sums what the seed tests landed.
  • Helper change: checkAdditiveMerges in test/support/landsim.ts takes an optional list of orders, defaulting to both, so existing callers behave the same.
  • Wide run: LANDSIM_SEEDS=n still runs seeds 1..n for the nightly job. Some seeds there (6, 7, 13 and others) take over 2 s alone, and the nightly has no per-test budget.
  • Unchanged: the #12104 reproducer stays in the gate as before.

Not done

  • Timings: these are from this sandbox, not the gate box. The slowest run (a quick seed's own test) came in under 1.8 s, so there is less margin there. Check the budget on the gate box.
  • Determinism seeds: the nightly no longer checks determinism on every seed with seed % 16 == 2; it runs seed 34 only. I judged that enough, but it is less than before.

To try it: bun test test/landsim.test.ts, or LANDSIM_SEEDS=24 bun test test/landsim.test.ts for the long run.

Closes #12447

2 earlier comments · 1 event
commented

Every test in test/landsim.test.ts is now under 2 s alone here, and the SLOW entry is gone from scripts/gate.ts. bun scripts/gate.ts and bun x tsc --noEmit are green, and the file's 20 tests pass. I did not run the full suite; the forge does that on push.

What changed

  • One test a seed: the single 18 s corpus test is now one test per seed. The additive-merge pair check is one test per landing order, and the order-independence check is its own test.
  • Determinism: it runs seed 34 twice, one run per test, and the second run compares signature and final tree against the first. This replaces seed 2.
  • Quick seeds: they are now 1, 3, 5, 8 instead of 1 to 4, chosen because they land fastest. Seed 2 took about 2.3–3 s alone. Seeds 1 and 5 cover additive merges and seed 8 covers order independence.
  • Landed check: the old corpus.landed > 0 check is now a last test that sums what the seed tests landed.
  • Helper change: checkAdditiveMerges in test/support/landsim.ts takes an optional list of orders, defaulting to both, so existing callers behave the same.
  • Wide run: LANDSIM_SEEDS=n still runs seeds 1..n for the nightly job. Some seeds there (6, 7, 13 and others) take over 2 s alone, and the nightly has no per-test budget.
  • Unchanged: the #12104 reproducer stays in the gate as before.

Not done

  • Timings: these are from this sandbox, not the gate box. The slowest run (a quick seed's own test) came in under 1.8 s, so there is less margin there. Check the budget on the gate box.
  • Determinism seeds: the nightly no longer checks determinism on every seed with seed % 16 == 2; it runs seed 34 only. I judged that enough, but it is less than before.

To try it: bun test test/landsim.test.ts, or LANDSIM_SEEDS=24 bun test test/landsim.test.ts for the long run.

commented

I rewrapped the three comment lines in test/landsim.test.ts that were over 100 columns, and pushed the fix to the pull. bun scripts/gate.ts is green and bun x tsc --noEmit is clean. All 20 tests in test/landsim.test.ts passed when I ran them, before I committed the rewrap.

One of the three lines was the one the check named, at line 64. The other two were an older header comment and a comment in the shrinker test. The gate only reads committed changes, so it kept reporting line 64 until I committed the rewrap.

What changed in the pull

  • The single 18 s corpus test is now one test per seed.
  • The additive-merge pair check is split into one test per landing order.
  • The order-independence check is its own test.
  • Determinism runs seed 34 twice, one run per test.
  • The quick seeds are now 1, 3, 5 and 8, picked because they land fastest. Seeds 1 and 5 cover additive merges and seed 8 covers order independence.
  • The SLOW entry for test/landsim.test.ts is gone from scripts/gate.ts.
  • checkAdditiveMerges in test/support/landsim.ts takes an optional list of orders. It defaults to both, so existing callers behave as before.
  • The #12104 reproducer stays in the gate.

Not done

  • Timings come from this sandbox, not the gate box. In the last full runs the slowest test was about 1.8 s, so check the 2 s budget on the gate box.
  • LANDSIM_SEEDS=n still runs seeds 1..n for the nightly job. Some of those seeds take over 2 s alone, which only matters if the nightly has a per-test budget.
  • The nightly no longer checks determinism on every seed with seed % 16 == 2; it runs seed 34 only.

To try it: bun test test/landsim.test.ts, or LANDSIM_SEEDS=24 bun test test/landsim.test.ts for the long run.

Changes requested — before the latest push

Test-only change, no page or terminal surface, so the preview is not affected. Splitting the 18 s corpus test into one test per seed is the right move for #1242, and dropping the SLOW entry is correct. Changes needed, all in test/landsim.test.ts, in one round:

  1. Vacuous assertion (fix first). The test "determinism: a seed runs once" asserts first.violations.length >= 0, which is always true. A test that cannot fail does not meet the rule that every behaviour has a test that would fail without it. Either assert something real there (for example that the run produced a signature and no unowned violations, via unowned(first.violations) equal to []), or fold it away.
  1. Order-coupled tests. The determinism second test and "the seeds landed something" both depend on state set by earlier tests (first, landed). Run alone, with -t or a re-run of one test, they fail or pass for the wrong reason (landed stays 0 and expect(first).toBeDefined() fails). Make each self-contained. For example, have the second determinism test run the seed twice if that fits the 2 s budget, or compute and compare in a shared beforeAll. Replace the landed counter with an assertion inside a seed test, such as out.landed for a known-landing seed.
  1. Coverage lost without a stated owner. Determinism was checked on seed % 16 == 2 across the nightly 24 seeds and is now checked on seed 34 only. The header comment still says the nightly runs 24 seeds, and the own-parts test no longer asserts determinism is covered. Say in the test (and in the pull) that this reduction is deliberate, or keep nightly determinism by running it for seeds where seed % 16 == 2 when LANDSIM_SEEDS is set. Also confirm 34 is itself under the 2 s budget on the gate box: the pull reports only sandbox timings and about 1.8 s of margin for the slowest quick seed.
  1. Budget not enforced. The 300_000 ms per-test timeout defeats the purpose of the 2 s budget for these tests. Use whatever the repo's per-test budget convention is so a regression back to a slow seed fails the file rather than passing silently.
commented

The 18 s landsim corpus test is now one test per seed, and test/landsim.test.ts no longer needs a SLOW entry. bun test test/landsim.test.ts passes (19 tests, 22 s in my sandbox), and scripts/gate.ts and tsc --noEmit are clean. All changes are in test/landsim.test.ts.

Changes, against the four review points:

  1. Vacuous assertion: the >= 0 check is gone. The first determinism test now asserts a non-empty signature and unowned(violations) equal to [].
  2. Order coupling: determinism now sits in a describe(\determinism, seed N\) group. Its beforeAll runs the seed twice, and two self-contained tests read the results, so each also passes under -t. Each seed test now asserts out.landed > 0, and the shared landed counter and the "seeds landed something" test are removed.
  3. Determinism coverage: it is not reduced any more. Quick runs check seed 34. With LANDSIM_SEEDS set (the nightly's 24 seeds), it checks every seed where seed % 16 == 2, as before. I updated the header comment to say so.
  4. Budget: I disagree that a 2 s timeout belongs on these tests. The 2 s limit is enforced by the gate, not by per-test timeouts. scripts/test-budget.ts reads each test's time from the JUnit report and fails any test over 2 s unless the file is in SLOW, and this file no longer is. The repo's usual timeout is 30_000 as a safety net, so I replaced 300_000 with that. A regression to a slow seed now fails the gate.

Timings: The slowest seed tests in my sandbox are 1.6 s (seed 1), 1.5 s (seed 8 order independence) and 1.5 s (seed 3). I can't see how long seed 34 takes on the gate box. Its two runs are in a beforeAll, and the budget counts only each test's own time, not its hooks. The 20 s file budget does still count them: the whole file took 22 s here, which is over, and I have no gate-box time to tell whether it is over there. If the gate box turns out over 20 s, the way out is to cut a quick seed or run determinism on a cheaper seed.

Left out: I did not change the seed choice or the scenario generator.

Approved

Earlier asks: the vacuous check is gone (determinism now asserts a non-empty signature and no unowned violations); order coupling is fixed (the beforeAll group with self-contained tests, and a per-seed landed>0); nightly determinism coverage is back (seed%16==2 under LANDSIM_SEEDS, seed 34 in the quick run); the 2 s timeout disagreement is fair, since the gate budget enforces it and 30_000 is the repo norm. The SLOW entry is removed and the helper change is backward compatible. I could not diff against 6a79de9 (commit not in the clone), so I read the full change.

Non-blocking: 1) The file took about 22 s locally against the 20 s per-file budget, and hooks count there; if the gate box goes over, drop one quick seed or move determinism to a cheaper seed. 2) Quick seeds have only about 0.2 s of margin under 2 s on the sandbox, so watch the first gate run.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs