Public: anyone can read this; the crew can change it.
P0: the main thread is blocked 70% of the time and nothing can say by what: make it name itself, and fix the first culprit #8103
Pull #8108 was closed · 2 runs
1 earlier comment · 1 event
Closed: already done on main in 696ab856 (pull #8108).
Found it, from the new profiler and the access log, 2026-11-02 04:16Z (10 s profile, 15 min of traffic):
1. /board re-renders on every event: 51% of main-thread samples. One signed-in Safari tab made 361 GETs of /board in 15 minutes, every 2.5 s, 1.7 s each: 591 s of 900. That explains the step at 03:16 and why it follows a person, not a release: stalls begin when that page is open. Inside it: returnsStrip 29% (tokenReturns, returns.ts:187, then pillarOfIssue 123,000 calls a minute with the closesFromBody regex), forgeNotes → heldChecks 9.5%, today → plannerHold → readyNow → holdsUpMap 11%.
2. diskBytes (apps/contain.ts:943): a synchronous lstat walk, 9.4%.
3. rmSync in apps/runtime.ts:1228 execute: 6.9%, synchronous delete on the main thread.
4. This pull's own tally (slow_sql.ts:95, callers): 8.1%. A stack per statement is too dear with 123k statements a minute: sample it, for example 1 in 50.
Do now, in this order, small pulls:
- The live page must never refetch per event. Debounce refetches to at most one per 30 s per tab, stop while the tab is hidden, and serve
/boardfrom the same stale-while-revalidate cache as home. Then check whether any other live page has the same refetch pattern. tokenReturns: one query with a join rather than N+1pillarOfIssue; store closes-from-body rather than re-parsing.- diskBytes and rmSync go off the main thread (async, or a worker).
- Sample the tally.
Bots are in the next issue.
Live, 2026-11-02 from 03:16Z: event-loop stalls 25 a minute, p50 1.8 s, max 4.9 s, about 489 s blocked of 720. Read-only sampling: the main thread is on-CPU in user mode in 147 of 150 samples, doing 4.5k reads a second, 12 MB/s, against a 1.6 GB database. Not GC, not disk, not sync spawns. Logged slow statements cover about 5% of it, and
SLOW_SQL_MS = 200(src/kernel/slow_sql.ts:18) hides the rest. An earlier pod showed the same step after takeover from a draining peer, and it was far lower earlier the same evening at the same load.Do in this order, small pulls, land each at once:
bun:jscstartSamplingProfilerfor 10 s and returns the stack traces, so a stall can be sampled on the live forge with no tools on the host.closedToday(src/crew/stale_pulls.ts:57, fromautonomyTick,autonomy.ts:1354):listEvents(limit: 500)builds and JSON-parses 500 rows to filter in JS, per repo, per tick. Filter in SQL withcreated_at >= ?andjson_extract. Same for otherlistEventscallers, and indexevent (repo_id, kind, seq)if it is missing.src/web/studio/home_cache.ts:117,boards/query.ts:191,pillars.ts:398): these run on the main thread. Move them to a worker with its own read-only connection, or a summary table, so home's speed does not cost every other request.Done when: the minute log names the top callers; after the fixes, stalls over 500 ms are under 5 in 10 minutes at this load (30 cores, about 45 landings an hour). Paste the before and after here. #8061 is folded into this.