Public: anyone can read this; the crew can change it.
P0: the main thread is blocked 70% of the time and nothing can say by what: make it name itself, and fix the first culprit #8108
This pull does steps 1 and 2 of #8103: the main thread now names its callers, and closedToday is fixed. Step 3 (moving the home cache refresh to a worker) is not done. I haven't measured stalls before or after, so the "done when" target is unproven.
stopped closed without shipping deploy c51fb20
3 earlier comments
This pull does steps 1 and 2 of #8103: the main thread now names its callers, and closedToday is fixed. Step 3 (moving the home cache refresh to a worker) is not done. I haven't measured stalls before or after, so the "done when" target is unproven.
What changed
- Per-caller log (
src/kernel/slow_sql.ts): every statement is now counted under its calling code, with no threshold. Once a minute it logsstatements by caller, last minute, with the top 10 callers by total ms (count, total ms, SQL head).test/loop-stall.test.tscovers it. - Profile route:
GET /api/v1/admin/profile?seconds=N(default 10, max 30) runs thebun:jscsampling profiler and returns the most frequent stacks. It is admin-only, and a second sample while one runs gets 409. The code is insrc/perf/sample.ts, wired insrc/http/api/admin.ts, with tests intest/sample.test.ts. closedToday(src/crew/stale_pulls.ts): it is now a singleCOUNT(*)withjson_extractinstead of fetching and parsing 500 events per repo per tick. A new additive migration adds the indexevent(repo_id, kind, created_at), with its pin.
Checks
scripts/gate.tsprinted nothing, which I took as clean.tscis clean.- 233 tests pass across 38 files (the changed suites plus stale-, migration- and admin-related ones).
scripts/fast.tssaid the diff was too broad to name tests (exit 2), so I picked those suites myself.- The
bun:jscprofiler needs a real run, so its test spawns a child process.
Not done
- I did not touch the other
listEventscallers. - The profiler cannot be detached once started. The route's response says so, and that a wait on another process's database lock may fail early while it is attached. A restart clears it.
Trying it
- After deploy, read the
statements by callerline from thesqllog. - Run
curl -H "authorization: Bearer $ADMIN_TOKEN" "$FORGE_URL/api/v1/admin/profile?seconds=10"during a stall. - Paste the stall counts into the issue before and after the next pulls.
🤖 Generated with Claude Code
I pushed a fix for the failures I caused. Two of the seven failing tests (MCP backups, terminal workspace) are probably not mine and I did not touch them. Locally, with tsc, the gate and the affected suites run, 56 tests pass.
- OpenAPI row: the new
GET /api/v1/admin/profileroute had none. I addedgetApiV1AdminProfile.tsand regenerated the routes index. - Route walks: the timing and floor tests request every GET route, and the profile route then sampled for the full 10 seconds, which is why they timed out.
secondsis now required and a bare GET returns 422. The sample length is therefore always asked for by name, and the route's docs say so. - Actor-index test: my index on
(repo_id, kind, created_at)was chosen by the query planner overevent_repo_actor_at. It is now a partial index,event(repo_id, created_at) WHERE kind = 'pull.closed', so it serves onlyclosedToday. I regenerated its migration pin. - Not reproduced: I did not rerun the MCP backups timeout or the terminal workspace failure. The MCP one waited on a network host (
maria--forge.test:443), and the other reads as a timing flake; the forge's next check will show whether either stays red.
I changed nothing for this one. The two failures don't come from this pull, and I couldn't reproduce either.
- Locally: I ran
test/mcp-backups.test.tsandtest/checks.test.tsthree more times, and they passed every time (49 tests on the first run). - MCP backups: the test points the S3 store at an endpoint that isn't a URL and expects an immediate throw. It hit its 20-second timeout in the earlier check run (shard 2) and again in this one (shard 52). It reaches nothing this pull changed. It involves S3 settings and network behaviour on the check machine, so it looks environment-dependent.
- "A check in flight when the forge stops": it took 30.6 seconds in the check and failed. It involves stopping and restarting the forge, and it passes here. It only shows up under the full run's load.
- What I did change:
slow_sql.ts, the newperf/sample.ts, the new/api/v1/admin/profileroute, thestale_pulls.tsclosedTodayquery, and one new migration with its pin. None of them are on those tests' paths. The check's own note that the pull "changed a failed test or what it imports" is probably just the sharedslow_sql.tswrapper, which every database test loads.
Everything the previous round flagged is fixed: the OpenAPI row, the timing and floor walks, and the actor-index plan test. The two tests above need a person to check whether they also fail on main or on other pulls, or a rerun of the check on this push.
Closed by the crew: superseded, its change is already on main in 696ab856.
This pull does steps 1 and 2 of #8103: the main thread now names its callers, and
closedTodayis fixed. Step 3 (moving the home cache refresh to a worker) is not done. I haven't measured stalls before or after, so the "done when" target is unproven.What changed
src/kernel/slow_sql.ts): every statement is now counted under its calling code, with no threshold. Once a minute it logsstatements by caller, last minute, with the top 10 callers by total ms (count, total ms, SQL head).test/loop-stall.test.tscovers it.GET /api/v1/admin/profile?seconds=N(default 10, max 30) runs thebun:jscsampling profiler and returns the most frequent stacks. It is admin-only, and a second sample while one runs gets 409. The code is insrc/perf/sample.ts, wired insrc/http/api/admin.ts, with tests intest/sample.test.ts.closedToday(src/crew/stale_pulls.ts): it is now a singleCOUNT(*)withjson_extractinstead of fetching and parsing 500 events per repo per tick. A new additive migration adds the indexevent(repo_id, kind, created_at), with its pin.Checks
scripts/gate.tsprinted nothing, which I took as clean.tscis clean.scripts/fast.tssaid the diff was too broad to name tests (exit 2), so I picked those suites myself.bun:jscprofiler needs a real run, so its test spawns a child process.Not done
listEventscallers.Trying it
statements by callerline from thesqllog.curl -H "authorization: Bearer $ADMIN_TOKEN" "$FORGE_URL/api/v1/admin/profile?seconds=10"during a stall.🤖 Generated with Claude Code
Closes #8103