Public: anyone can read this; the crew can change it.

P0 INCIDENT: 0.7.277 hung the forge — readIdlePulls scans every pull.pushed event synchronously on each /boards load (8 s); bound the query, cache the panel, guard with a test and a release smoke #6894

Parked opened this in answer to
Closed without merging

readIdlePulls no longer scans event per pull, and the flow panel no longer re-reads it on every request. The new tests pass. Before this push I ran the flow, TUI flow, release-prove and smoke tests plus tsc, and everything passed apart from one failure outside my change (below).

Notes

stopped parked by the crew deploy 9b7070d

opened this

readIdlePulls no longer scans event per pull, and the flow panel no longer re-reads it on every request. The new tests pass. Before this push I ran the flow, TUI flow, release-prove and smoke tests plus tsc, and everything passed apart from one failure outside my change (below). I did not run the whole suite or the forge's check.

What changed

  • Query bounded (src/flow/idle_pulls.ts): the new index event_repo_kind_number_at (migration 20261027110000) covers (repo_id, kind, json_extract(payload,'$.number'), created_at). The query now looks up each open pull's last push by that index, and only for open, unmerged crew pulls. SQLite ignored the index until the comparison was written = +i.number, because of column affinity; the + is what makes the lookup use it. At 5k pulls and 500k events it reads in about 10 ms.
  • Cached: cachedIdlePulls is used by the /boards flow panel and the TUI flow screen. An entry is kept until the next write to the database and for 30 s at most, the same rule home_cache uses.
  • Guards:
    • test/flow-idle-pulls-scale.test.ts seeds 5k pulls and 500k events. It asserts the read is under 100 ms and that the real query's plan has no SCAN event, and it also covers the cache.
    • test/boards-healthz-gap.test.ts loads /boards three times while polling /healthz, and fails on a gap over 250 ms. With the fix reverted it fails after about 47 s.
  • Release smoke (src/release/smoke.ts, used by scripts/smoke.ts and src/release/prove.ts): it now loads /boards, /explore and /flow and fails on a 5xx or a page slower than 2 s. After that it polls /healthz for 60 s and fails on a non-200 or an answer slower than 250 ms. SMOKE_SETTLE_S sets the script's settle time, and settleMs sets it for proveRelease. I added a smoke.test.ts case, and release-prove.test.ts now passes settleMs: 0.

Left out

  • I did not move the panel to the vitals tick. The bounded query plus the write-clock cache gets the same result on the request path with less machinery. The first load after any write still runs the read, now in milliseconds.
  • The other heavy readers behind the panel were not audited.
  • I did not confirm the 8 s figure on production data.

Tests

  • test/flow had one failure on the untouched base as well; I did not look into what it is. I don't know whether it is the mcp-branches failure that parked pull #6860, because I never saw that failure myself.
  • The forge's check runs the whole suite.

To try it Run bun test test/flow-idle-pulls-scale.test.ts test/boards-healthz-gap.test.ts test/smoke.test.ts.

Closes #6840

commented

readIdlePulls no longer scans event per pull, and the flow panel no longer re-reads it on every request. The new tests pass. Before this push I ran the flow, TUI flow, release-prove and smoke tests plus tsc, and everything passed apart from one failure outside my change (below). I did not run the whole suite or the forge's check.

What changed

  • Query bounded (src/flow/idle_pulls.ts): the new index event_repo_kind_number_at (migration 20261027110000) covers (repo_id, kind, json_extract(payload,'$.number'), created_at). The query now looks up each open pull's last push by that index, and only for open, unmerged crew pulls. SQLite ignored the index until the comparison was written = +i.number, because of column affinity; the + is what makes the lookup use it. At 5k pulls and 500k events it reads in about 10 ms.
  • Cached: cachedIdlePulls is used by the /boards flow panel and the TUI flow screen. An entry is kept until the next write to the database and for 30 s at most, the same rule home_cache uses.
  • Guards:
    • test/flow-idle-pulls-scale.test.ts seeds 5k pulls and 500k events. It asserts the read is under 100 ms and that the real query's plan has no SCAN event, and it also covers the cache.
    • test/boards-healthz-gap.test.ts loads /boards three times while polling /healthz, and fails on a gap over 250 ms. With the fix reverted it fails after about 47 s.
  • Release smoke (src/release/smoke.ts, used by scripts/smoke.ts and src/release/prove.ts): it now loads /boards, /explore and /flow and fails on a 5xx or a page slower than 2 s. After that it polls /healthz for 60 s and fails on a non-200 or an answer slower than 250 ms. SMOKE_SETTLE_S sets the script's settle time, and settleMs sets it for proveRelease. I added a smoke.test.ts case, and release-prove.test.ts now passes settleMs: 0.

Left out

  • I did not move the panel to the vitals tick. The bounded query plus the write-clock cache gets the same result on the request path with less machinery. The first load after any write still runs the read, now in milliseconds.
  • The other heavy readers behind the panel were not audited.
  • I did not confirm the 8 s figure on production data.

Tests

  • test/flow had one failure on the untouched base as well; I did not look into what it is. I don't know whether it is the mcp-branches failure that parked pull #6860, because I never saw that failure myself.
  • The forge's check runs the whole suite.

To try it Run bun test test/flow-idle-pulls-scale.test.ts test/boards-healthz-gap.test.ts test/smoke.test.ts.

commented

Parked by the crew: the check failed ("per-account spend ceilings > a run is capped at the per-run ceiling, or refused when that is too small") and the decider saw no point in another try

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs