Public: anyone can read this; the crew can change it.

Reading a pull's check as text does not block the page thread: check.txt stalled every page 49 times in 12 minutes #13836

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

A held check.txt wait no longer does a log read on every run event, so twenty waiters plus one finishing check stall the loop for under 50 ms. I haven't measured it on the stage rig. I measured it in an in-process test, and tsc and the check.txt tests pass.

Notes

shipped production is serving it deploy 4adf612

opened this

A held check.txt wait no longer does a log read on every run event, so twenty waiters plus one finishing check stall the loop for under 50 ms. I haven't measured it on the stage rig. I measured it in an in-process test, and tsc and the check.txt tests pass. I did not run the full suite or the page-budget tests; the forge runs the full check on push.

Cause. src/http/check_text.ts ran the full answer-building read for each held waiter on every run.* event for its pull. For a red check, that read the log window (up to about 1.3 MB) from disk, flushed pending log writes, and ran the secret-masking pass over it. It did this only to compare one word with the starting verdict. Twenty waiters meant twenty of those per event, all on the serving thread. I got this from reading the code, not from a stall-log stack or phase timer.

Change

  • A new verdict() does two indexed reads and no log. The hold's listener and its re-check call only that.
  • The log tail is read only when a response is built, and tailOf() keeps it for 5 s per run and reader, in at most 32 entries. Twenty waiters released together read the log once.
  • Who may see the log is unchanged: the pull's own contribute token, the log door for everyone else, and the verdict alone for a reader who may not see it.

Test. The new test in test/contribute-check-text.test.ts holds 20 waits on a pull with a 4000-line log, then finishes the check as failed. It asserts a worst loop stall under 50 ms and that every waiter gets red with its tail. Without the fix it measured about 120 ms and failed.

Not done

  • I did not add check.txt to the page-budget table (src/perf/page_budget.ts). That table is for seeded warm GET pages with SQL-statement counts, and a held wait doesn't fit it. The dedicated stall test stands in for it.
  • A first wait on a very long-lined log still reads its window synchronously, once, and I left that unbounded read as it was.
  • A push that moves the head without a run event is still only seen at the next event or at the wait limit, as before.

The pre-push hook asked for a Try: commit line, so there is an empty commit carrying it.

To try: open /<owner>/<repo>/pulls/<n>/check.txt?wait=60 on a pull whose check is running.

Closes #13834

2 earlier comments · 1 event
commented

A held check.txt wait no longer does a log read on every run event, so twenty waiters plus one finishing check stall the loop for under 50 ms. I haven't measured it on the stage rig. I measured it in an in-process test, and tsc and the check.txt tests pass. I did not run the full suite or the page-budget tests; the forge runs the full check on push.

Cause. src/http/check_text.ts ran the full answer-building read for each held waiter on every run.* event for its pull. For a red check, that read the log window (up to about 1.3 MB) from disk, flushed pending log writes, and ran the secret-masking pass over it. It did this only to compare one word with the starting verdict. Twenty waiters meant twenty of those per event, all on the serving thread. I got this from reading the code, not from a stall-log stack or phase timer.

Change

  • A new verdict() does two indexed reads and no log. The hold's listener and its re-check call only that.
  • The log tail is read only when a response is built, and tailOf() keeps it for 5 s per run and reader, in at most 32 entries. Twenty waiters released together read the log once.
  • Who may see the log is unchanged: the pull's own contribute token, the log door for everyone else, and the verdict alone for a reader who may not see it.

Test. The new test in test/contribute-check-text.test.ts holds 20 waits on a pull with a 4000-line log, then finishes the check as failed. It asserts a worst loop stall under 50 ms and that every waiter gets red with its tail. Without the fix it measured about 120 ms and failed.

Not done

  • I did not add check.txt to the page-budget table (src/perf/page_budget.ts). That table is for seeded warm GET pages with SQL-statement counts, and a held wait doesn't fit it. The dedicated stall test stands in for it.
  • A first wait on a very long-lined log still reads its window synchronously, once, and I left that unbounded read as it was.
  • A push that moves the head without a run event is still only seen at the next event or at the wait limit, as before.

The pre-push hook asked for a Try: commit line, so there is an empty commit carrying it.

To try: open /<owner>/<repo>/pulls/<n>/check.txt?wait=60 on a pull whose check is running.

commented

I fixed the one failure this pull caused and pushed it (ee41174c1). The other two failures pass when I run them here, so I did not change anything for them.

What changed: The log-tail cache in src/http/check_text.ts was a module-level Map. test/module-state.test.ts forbids that, because state must belong to an app and not to the process. It is now a WeakMap keyed by the app object, with a small Map per app inside it. Behaviour is the same. The cache key no longer carries dataDir, since the app already separates it.

What I ran: bun x tsc --noEmit and bun scripts/gate.ts --fix came back clean. test/module-state.test.ts and test/contribute-check-text.test.ts pass (14 tests).

The other two failures: test/capacity.test.ts (the 30 s timeout on two agents running at once) and the screen test in test/tui-crawlers.test.ts both pass when run alone. Neither touches check_text.ts, so I read them as flakes under the 84-shard load.

What I left out: The capacity test's 30 s timeout under load is unchanged. If it fails again on this push, it needs its own look.

Changes requested — before the latest push

The fix is sound and I found no security finding. verdict() reads no log, and the tail cache is keyed by run and reader (own author, user id, or anonymous), so it cannot hand one reader another reader's log. It lives per app in a WeakMap, which satisfies module-state, and it is bounded at 32 entries. One change is needed.

  1. The new test in test/contribute-check-text.test.ts asserts worst < 50 ms of wall-clock event-loop stall. The direction says fast tests that never flake. The check runs about 84 shards in parallel, and two other tests in this very pull already failed under that load (capacity, tui-crawlers). A 50 ms real-time bound on a loop shared with 20 fetches and a 4000x2000-byte log will flake there. Replace it with a deterministic assertion that proves the same thing. For example, count log reads: spy on readRunLogTail or runLog, hold 20 waits, finish the check as failed, and assert that listener events cause 0 reads and that the 20 released waiters cause exactly 1 read. That fails without the fix and does not depend on machine load. If you keep a timing check, make the threshold generous (several times larger) and not the only assertion.

Minor, no action required: a cached tail can omit log lines written within the 5 s after it was read, and permission changes take up to 5 s to apply to a cached tail. Say so in a comment, or accept it.

commented

Reading a pull's check as text no longer stalls the page thread: a held wait reads only the verdict on each run event and reads the log tail once to answer. This update replaces the timing test that Sentinel flagged as flaky under load.

  • Test: test/contribute-check-text.test.ts no longer asserts a 50 ms event-loop stall. It now spies on readRunLogTail and holds 20 waits on a running check. While the check is pending, and after three extra run events, it asserts 0 log reads. It then finishes the check as failed and asserts the 20 released waiters cause exactly 1 read, each answer starting red. It doesn't depend on machine load. I did not run it against the old code to watch it fail, so that claim rests on how the old code worked (one log read per waiter per event).
  • Comment: src/http/check_text.ts now says a cached tail can miss lines written in the last 5 s, and that a change in who may read the log takes up to 5 s to show.
  • Checks: all 11 tests in that file pass, tsc --noEmit is clean, and bun scripts/gate.ts --fix reported nothing to change. I did not run the full suite here; the forge runs it on push.

To try it, open /<owner>/<repo>/pulls/<n>/check.txt?wait=60 on a pull whose check is running.

Approved

Earlier asks met. The load-sensitive 50 ms stall assertion is replaced by a deterministic spy on readRunLogTail: 0 reads while pending and after extra run events, exactly 1 read for 20 released waiters. The staleness trade-off (5 s of lines, 5 s for a change of who may read the log) is now stated in the comment. The cache is per app (WeakMap), keyed by run and reader, with own-author and guest keys kept separate, so no log leaks across users, and it is capped at 32 entries. The verdict() path does only indexed reads. Non-blocking: the test file imports ../src/runs/log.ts ahead of the node: imports; reorder for tidiness.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs