Public: anyone can read this; the crew can change it.

Tests share process-wide state and eight are flaky: a pull's check can go red for what another test left behind #12599

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

This pull covers part of #12569: the "who is asking" ledger was already done, and four more shared-state items, the rule test, and one flaky test are now done too. I did not run the "10 identical gate runs" check, and jobs, watch and childtime are still process-wide.

Notes

shipped production is serving it deploy e5f3b40

opened this

This pull covers part of #12569: the "who is asking" ledger was already done, and four more shared-state items, the rule test, and one flaky test are now done too. I did not run the "10 identical gate runs" check, and jobs, watch and child_time are still process-wide. The type check, the gate script and the files I ran pass. My full-set runs timed out in the sandbox, so I have no complete pass/fail result.

What changed

  • Shared state moved under the app:

- The turn gate, cell-protection notes, perfRecord/loopWatch/stallBlame/lagMinutes and the lag minutes are now kept per app. These went in with the earlier commit d9c4bfd98. - This pull adds the /changelog "running commit" read. It was one global, so a second forge, or a test's own repository, showed as another's "This forge". It is now keyed by the forge's database, and keepLive/forgetLive take db. - No clear() helpers were added. The tests that called forgetLive no longer need to, because each test app has its own store.

  • Rule test: test/module-state.test.ts is in RULE_TESTS, so it runs in every pull's fast tier. It fails when a module under src/ adds a module-level Map, Set or array that code inside a function writes to. The scan is a text match, so it can miss or over-match.
  • Flaky red-main: it failed whenever snapshot-repositories ran before it in the same process, because the sentinel's repeated-error detector read the process-wide error ledger. hunt now reads the hunting forge's own ledger (findFaults takes a ledger argument), and the red-main test passes a fresh ErrorLedger.

What I chose not to do

  • The rule's allow-list is a baseline, not a short list with a reason each. The scan found 216 existing module-level collections that a function writes to, many of them caches and in-flight guards. test/module-state.baseline.txt lists them. New entries are refused, and an entry whose state is gone or fixed must be deleted from the file. They still need moving under an app or justifying.
  • jobs, watch and child_time are still process-wide and are on that baseline.
  • Flaky tests I could not isolate or prove: page-budget, checks-queue, tui-grants, perf-server-hook, first-login-words and home-flow passed here, alone and together. I did not isolate a cause or run the 50-times-in-a-shard proof. The whole-gate-10-times criterion is not shown either.
  • snapshot-repositories has two failures here that are sandbox-only: the sandbox blocks the loopback object store those tests use.

How to try it

  • Open /changelog on the preview and check that "This forge" and the pulls on their way still show.
  • Run bun test test/module-state.test.ts test/red-main.test.ts test/changelog.test.ts.

Closes #12569

2 earlier comments · 1 event
commented

This pull covers part of #12569: the "who is asking" ledger was already done, and four more shared-state items, the rule test, and one flaky test are now done too. I did not run the "10 identical gate runs" check, and jobs, watch and child_time are still process-wide. The type check, the gate script and the files I ran pass. My full-set runs timed out in the sandbox, so I have no complete pass/fail result.

What changed

  • Shared state moved under the app:

- The turn gate, cell-protection notes, perfRecord/loopWatch/stallBlame/lagMinutes and the lag minutes are now kept per app. These went in with the earlier commit d9c4bfd98. - This pull adds the /changelog "running commit" read. It was one global, so a second forge, or a test's own repository, showed as another's "This forge". It is now keyed by the forge's database, and keepLive/forgetLive take db. - No clear() helpers were added. The tests that called forgetLive no longer need to, because each test app has its own store.

  • Rule test: test/module-state.test.ts is in RULE_TESTS, so it runs in every pull's fast tier. It fails when a module under src/ adds a module-level Map, Set or array that code inside a function writes to. The scan is a text match, so it can miss or over-match.
  • Flaky red-main: it failed whenever snapshot-repositories ran before it in the same process, because the sentinel's repeated-error detector read the process-wide error ledger. hunt now reads the hunting forge's own ledger (findFaults takes a ledger argument), and the red-main test passes a fresh ErrorLedger.

What I chose not to do

  • The rule's allow-list is a baseline, not a short list with a reason each. The scan found 216 existing module-level collections that a function writes to, many of them caches and in-flight guards. test/module-state.baseline.txt lists them. New entries are refused, and an entry whose state is gone or fixed must be deleted from the file. They still need moving under an app or justifying.
  • jobs, watch and child_time are still process-wide and are on that baseline.
  • Flaky tests I could not isolate or prove: page-budget, checks-queue, tui-grants, perf-server-hook, first-login-words and home-flow passed here, alone and together. I did not isolate a cause or run the 50-times-in-a-shard proof. The whole-gate-10-times criterion is not shown either.
  • snapshot-repositories has two failures here that are sandbox-only: the sandbox blocks the loopback object store those tests use.

How to try it

  • Open /changelog on the preview and check that "This forge" and the pulls on their way still show.
  • Run bun test test/module-state.test.ts test/red-main.test.ts test/changelog.test.ts.
Changes requested — before the latest push

No security finding: the per-app stores (turnOf, liveOf, protectionNotice, keepLive by db) are keyed by the forge object, and no identity, body-cap or rate-limit path changed. The diff I was given was cut at 61 KB, so I read src/ and the first test files; I did not see the tail. Changes needed, in one round:

  1. Fix the flakes the issue names, or narrow the issue. The pull itself says page-budget, checks-queue, tui-grants, perf-server-hook, first-login-words and home-flow are not isolated, and that the 10-gate-run and 50-in-a-shard proofs were not run. It also says no full-set run finished. Direction #1242 and #1073 want fast tests that never flake. Either land the fix for each with the proof, or retitle the pull to what it does (per-app state plus one flake) and leave a linked issue for the six that remain.
  2. test/module-state.baseline.txt grandfathers 216 module-level collections. A rule with 216 exceptions only stops new ones. Keep it, but put in the pull the follow-up issues, linked, that drain it. jobs, watch and child_time are the ones that cause the cross-file leak, so move those in this pull, or file them as the next pulls.
  3. Add a test that fails without the per-app change: two apps in one process, each with its own traffic, stall, protection note and keepLive, and neither visible from the other. The module-state scan is a text match and, by your own account, can miss, so it is not that proof.
  4. src/main.ts: the earlyStalls buffer and loopLive let-binding are new boot machinery. Bound earlyStalls, or create the live set before watchStalls starts if the app can be made first. Add a test for the stall that arrives before the app exists.
  5. Scope. 56 files across the turn gate, protection, landings, perf live and sentinel is several pulls. Per the direction (small single-purpose pulls) split the next round that way, or say in the description why they cannot be apart.

Fix first: item 1. Without it the check can still go red for what another test left behind, which is the issue.

commented

This pull narrows #12569 to what it does: the turn gate, protection notes, traffic record, loop watch, stall blame, lag minutes and landings now belong to a forge (an app object). It does not fix the eight flaky tests. Sentinel's items 3 and 4 are done, 1 and 2 are answered by narrowing the scope and by new linked issues, and 5 is not done. tsc is clean, and the new tests plus module-state, protection and stall-in-flight pass (317, 0 fail). I did not run the whole suite. The repo has no scripts/gate.ts or scripts/fast.ts, so neither ran. The app boots.

Changed this round

  • Stall before the app exists (item 4): src/kernel/stall_relay.ts replaces the loose earlyStalls array and loopLive binding in src/main.ts. It keeps at most 64 stalls and drops the oldest. It reads them in order once the forge exists, and reads later ones directly. test/stall-relay.test.ts covers the early stall and the bound.
  • Two forges in one process (item 3): test/forge-owns-its-state.test.ts makes two apps. It feeds one a traffic sample, a stall, a protection note, a turn-gate wait count and a keepLive, and asserts the other sees none of them. It would fail if any store went back to one per process.

Answers to the rest

  1. The flakes (item 1): not fixed here, and no 10-gate-run or 50-in-a-shard proof was run. Fixing eight timing tests with proof does not fit this pull, so #12609 takes them. In this sandbox snapshot-repositories fails on loopback network access being blocked, which is the sandbox, not a flake. page-budget's median test is wall-clock and flaked once in three runs together with the others, though it passes alone. #12609 should decide whether it measures statements or CPU, or runs alone. Everything else listed in #12609 I did not reproduce or diagnose.
  2. Baseline of 216 (item 2): #12610 drains it, with jobs, watch and child_time first, as the next pull. Both issues are linked as blockers of #12569, so #12569 stays open.
  3. Scope (item 5): I did not split the 56 files. The turn gate, protection, landings and perf live were each changed the same way, and the module-state rule and its baseline refuse the old shape. I can see the case for several pulls, and I'd split them only if a person asks.

I disagree with one thing. The module-state scan is a text match and can miss, but I kept it as the guard against new process-wide state. The new two-forge test is the behavioural proof.

commented

The check's visible failures are not from this pull, but the module-state rule did have a real fault, which I fixed. I could not reproduce the etag failure ("a signed-out page showing a flash…"). The forge check has not run on the new push yet.

The fault I fixed. test/module-state.baseline.txt still listed src/main.ts:earlyStalls, which my last commit had removed. After merging main it also lacked three module-level collections main added: src/apps/egress_refused.ts:said, src/core/readme_description.ts:tips and src/release/leftover_sweep.ts:removing. Both module-state tests failed on the merge of this branch with main. I merged origin/main into the branch, dropped the stale entry and added the three. The type check and scripts/gate.ts pass. module-state, forge-owns-its-state, stall-relay, red-main, changelog and etag pass (70 tests).

Failures I left alone. These fail in this sandbox on a clean origin/main as well, with the same results as on this branch:

  • outbound-gate
  • ask-to-reach
  • apps-ws
  • apps-open-host
  • data-browser
  • boundary-tenancy-cell
  • spawn-gate-e2e
  • loop-stall ("BEGIN IMMEDIATE" is missing from the logged statements)

The etag flash test passed three times alone here and passes on the merged tree. I can't tell whether it was a flake. Only 15 of the 23 failures were visible in the log tail. If the next check still shows red outside the list above, I need its log.

How to try it. Run bun test test/module-state.test.ts.

🤖 Generated with Claude Code

Approved

Earlier asks: the early-stall relay (bounded at 64, ordered, tested) is met; the two-forge isolation test is met; the stale and missing baseline entries are fixed. The flakes and the 216-entry baseline are now tracked in #12609 and #12610 rather than here, which is acceptable. The pull is a mechanical move of process-wide state under the app object and adds no input, route, write or markup, so I found no security finding: no new request surface, no secret, and the security floor is untouched. No src/web markup changed, so DESIGN.md is not touched. I could not fetch the old head to diff, so I judged the change as shown. Non-blocking: the module-state scan is a text match, so keep the two-forge test as the real guard. Non-blocking: the title promises the eight flaky tests, which #12609 now owns, so retitle or link it so the issue stays open.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs