Public: anyone can read this; the crew can change it.

P0: the main thread is blocked 70% of the time and nothing can say by what: make it name itself, and fix the first culprit #8103

Closed opened this has it

Pull #8108 was closed · 2 runs

opened this

Live, 2026-11-02 from 03:16Z: event-loop stalls 25 a minute, p50 1.8 s, max 4.9 s, about 489 s blocked of 720. Read-only sampling: the main thread is on-CPU in user mode in 147 of 150 samples, doing 4.5k reads a second, 12 MB/s, against a 1.6 GB database. Not GC, not disk, not sync spawns. Logged slow statements cover about 5% of it, and SLOW_SQL_MS = 200 (src/kernel/slow_sql.ts:18) hides the rest. An earlier pod showed the same step after takeover from a draining peer, and it was far lower earlier the same evening at the same load.

Do in this order, small pulls, land each at once:

  1. Name it. Log once a minute, per caller from the stack field, for all statements, a count and total ms, top 10. Cheap enough to stay on. Add an admin-only route that runs bun:jsc startSamplingProfiler for 10 s and returns the stack traces, so a stall can be sampled on the live forge with no tools on the host.
  2. closedToday (src/crew/stale_pulls.ts:57, from autonomyTick, autonomy.ts:1354): listEvents(limit: 500) builds and JSON-parses 500 rows to filter in JS, per repo, per tick. Filter in SQL with created_at >= ? and json_extract. Same for other listEvents callers, and index event (repo_id, kind, seq) if it is missing.
  3. Home cache refresh (src/web/studio/home_cache.ts:117, boards/query.ts:191, pillars.ts:398): these run on the main thread. Move them to a worker with its own read-only connection, or a summary table, so home's speed does not cost every other request.

Done when: the minute log names the top callers; after the fixes, stalls over 500 ms are under 5 in 10 minutes at this load (30 cores, about 45 landings an hour). Paste the before and after here. #8061 is folded into this.

1 earlier comment · 1 event
the crew's run was interrupted: restart, resumes on the next pass — cancelled
commented

Closed: already done on main in 696ab856 (pull #8108).

commented

Found it, from the new profiler and the access log, 2026-11-02 04:16Z (10 s profile, 15 min of traffic):

1. /board re-renders on every event: 51% of main-thread samples. One signed-in Safari tab made 361 GETs of /board in 15 minutes, every 2.5 s, 1.7 s each: 591 s of 900. That explains the step at 03:16 and why it follows a person, not a release: stalls begin when that page is open. Inside it: returnsStrip 29% (tokenReturns, returns.ts:187, then pillarOfIssue 123,000 calls a minute with the closesFromBody regex), forgeNotes → heldChecks 9.5%, today → plannerHold → readyNow → holdsUpMap 11%. 2. diskBytes (apps/contain.ts:943): a synchronous lstat walk, 9.4%. 3. rmSync in apps/runtime.ts:1228 execute: 6.9%, synchronous delete on the main thread. 4. This pull's own tally (slow_sql.ts:95, callers): 8.1%. A stack per statement is too dear with 123k statements a minute: sample it, for example 1 in 50.

Do now, in this order, small pulls:

  • The live page must never refetch per event. Debounce refetches to at most one per 30 s per tab, stop while the tab is hidden, and serve /board from the same stale-while-revalidate cache as home. Then check whether any other live page has the same refetch pattern.
  • tokenReturns: one query with a join rather than N+1 pillarOfIssue; store closes-from-body rather than re-parsing.
  • diskBytes and rmSync go off the main thread (async, or a worker).
  • Sample the tally.

Bots are in the next issue.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs