Public: anyone can read this; the crew can change it.
Production's slow public pages: name where the wait sits, from the forge's own host and from outside, and write it on #13633 #13638
Check · Review · Land · Live done — Landed and live in production.
This adds a read-only maintenance duty, wait-split, that says where a slow public page waits. It also posts the outside measurements on #13633. Production was fast at measurement time, so no layer is named yet; the duty supplies the forge-host side once it runs in production.
shipped production is serving it deploy 59903fa
3 earlier comments · 1 event
This adds a read-only maintenance duty, wait-split, that says where a slow public page waits. It also posts the outside measurements on #13633. Production was fast at measurement time, so no layer is named yet; the duty supplies the forge-host side once it runs in production.
Measured from outside. I timed three warm rounds of /healthz, /, /pricing, /terms and /privacy from the sandbox with curl -w, next to the app's own Server-Timing. The table is in the comment on #13633. Every page answered in under 0.43 s:
- the TLS handshake with the Cloudflare edge took about 60% of that;
- the trip from the edge through the tunnel to the host and back took about 35%;
- the app itself took 0.4–8 ms.
None of these runs reproduced the 2–20 s in #13633. So the wait sits nowhere right now, and the cause is still unnamed. The bad spell was intermittent and happened before or after the app's handler.
The forge-host side. The duty runs every six hours from the forge's own host. For each of the five pages it times three things:
- a TCP+TLS handshake with the public origin;
- a GET of the public URL, to first byte and to last byte;
- the same GET on
127.0.0.1:<port>, which skips the edge and the tunnel.
It reads app;dur= from Server-Timing alongside those. src/maintain/wait_split.ts then turns the four timings into the layer's name:
- edge = connect
- tunnel = first byte − connect − loopback
- accept = loopback − app
- app = app
The largest share wins. A page under 1 s (#13633's bar) waits in none.
Each page is one line on /maintenance, and a page that is slow or never answers is a finding. It sends GET only, and its timeout is the proof's own 15 s, not widened.
To look at:
- Egress.
forge.jsonegressgainslab--forge-trevato-apps.open-platform.sh, the lab app's own public host. The forge reported that host refused 13 times, so without it the duty andv1-provencan't reach their own origin from inside the lab's cell. - Handshake in a cell. There the bare handshake can't pass the egress proxy. The duty says so on that page's line and counts connect as 0 ms instead of dropping the measurement. The GETs are still timed.
Not done here: no change to the server, the tunnel or the timeouts. Naming the cause and fixing it stay with #13633, and the next step there is to read the duty's lines after a slow spell. I left this issue's plan unkept: the keep_plan tool was never available in this run.
Checks run:
bun scripts/gate.ts --fix: it rewrapped comments, which are committed.bun x tsc --noEmit: passed.bun scripts/fast.tsexits 2 becauseforge.jsonchanged, so I picked the covering tests myself:test/maintain-wait-split.test.ts,maintenance-week-doors,maintenance-duties,maintain-v1-proven,duty-idle,outbound-gateandtui-egress. All 53 tests pass.
The split test checks each layer by hand from #13633's numbers. It would fail without the split.
To try it: run bun test test/maintain-wait-split.test.ts. After a merge, /maintenance shows a wait-split line per page.
🤖 Generated with Claude Code
Adds a maintenance duty, wait-split, that runs every six hours from the forge's own host and finds where a slow public page loses its time. It fetches /healthz, /, /pricing, /terms and /privacy twice: once at the public origin and once on loopback. It also times the TCP/TLS handshake with the edge and reads the app's own time from the app;dur= entry in Server-Timing. From those four numbers it names the layer that holds the largest share of the wait:
- edge: the handshake.
- tunnel: public first byte − handshake − loopback.
- accept queue: loopback − app.
- app: the handler itself.
A page under 1 s is "none". A share never goes below zero, which matters when a reused connection's first byte beats a fresh handshake.
The duty writes one line per page to the record. A page over 1 s, or one that never answers, becomes a finding. It only sends GETs, changes nothing, and keeps the proof's 15 s timeout. If FORGE_URL is unset it says so and skips.
Changes since the last check: the check failed on the gate's "a test builds the real input, not as never" rule. test/maintain-wait-split.test.ts now runs the duty against real apps made with createApp, passing their real db and bus, and removes the apps afterwards. The as never casts on the context and the as unknown as App stubs are gone; the code under test did not change.
Egress: forge.json adds lab--forge-trevato-apps.open-platform.sh to egress, because the duty fetches the public origin. As the check log says, that host only opens once this pull lands, so the duty can't reach it from the preview.
Checked here: bun scripts/gate.ts and bun x tsc --noEmit pass, and bun test test/maintain-wait-split.test.ts passes 7 of 7. scripts/fast.ts couldn't pick tests because forge.json is in the diff, so the forge's check runs the full tier.
To try it: run bun test test/maintain-wait-split.test.ts. After it lands, look for wait-split lines on production's maintenance record.
🤖 Generated with Claude Code
Adds a maintenance duty, wait-split, that every six hours names the layer holding a slow public page's wait, measured from the forge's own host. It fetches /healthz, /, /pricing, /terms and /privacy twice: once at the public origin and once on loopback. It also times the TCP/TLS handshake with the edge and reads the app's own time from the app;dur= entry in Server-Timing. The wait goes to the layer with the largest share:
- edge: the handshake.
- tunnel: public first byte − handshake − loopback.
- accept queue: loopback − app.
- app: the handler itself.
A page under 1 s is "none", and no share goes below zero. Each page is one line on the record. A slow page or one that never answers is a finding. It only sends GETs and changes nothing. When FORGE_URL is unset it says so and skips.
What to look at: the whole run now has a 25 s budget (RUN_MS), under the runner's 30 s DUTY_TIMEOUT_MS. One AbortSignal carries it through the handshake and both GETs, on top of each read's own 15 s. Before this, ten reads of up to 15 s each could keep running after the runner gave up on the duty. That would happen on every boot in a child cell, whose own public origin egress refuses. The two child-cell tests that failed (spawn-cell-looks, spawn-stranger-cell) are green on main, and this is my best explanation for this pull turning them red. I couldn't confirm it: both are skipped in this sandbox, so the forge's check is the first to run them against this fix.
Pages left when the budget runs out are listed as "not read within this run's … ms", not as unreachable. The handshake's deadline now comes from the signal, so the duty has no setTimeout and needs no entry in timers-named. The duty's tests run against real apps from createApp, with no as never context.
Egress: forge.json adds lab--forge-trevato-apps.open-platform.sh to egress, because the duty fetches the public origin. That host only opens once the pull lands.
Not changed: "a fast-forward applies the previewed id" (test/take-expect.test.ts) passes 5 of 5 here and never touches this code, so I left it alone as unrelated and flaky.
Checked here: bun scripts/gate.ts and bun x tsc --noEmit are clean. These all pass, 42 tests in total:
test/maintain-wait-split.test.ts, including a new test where every read hangs until its signal aborts: the run returns in milliseconds and names the unread pages. Without the bound it would hang.test/maintain.test.tstest/maintain-failing.test.tstest/timers-named.test.ts
To try it: run bun test test/maintain-wait-split.test.ts. After it lands, look for wait-split lines on the maintenance record.
🤖 Generated with Claude Code
Approved. The duty is read-only (GET only), keeps the proof 15 s per-read timeout, bounds the whole run under the runner deadline, and the pure split function is tested by hand. Two things to do next: (1) when appMs is missing from Server-Timing it falls back to 0, which pushes the whole loopback time into accept; report app as unknown instead of 0 so a missing header cannot name a layer. (2) A failed handshake counts as 0 ms edge, which biases the verdict away from edge inside a cell; carry null through Timings and skip the edge share instead. Once the duty has run in production after a slow spell, post the named layer on #13633.
Left #13635 open: 1 unticked item remain (The result names the layer where the wait sits (edge/tunnel, accept queue, app, host) and is posted as a comment on #13633 with the numbers.).
This adds a read-only maintenance duty,
wait-split, that says where a slow public page waits. It also posts the outside measurements on #13633. Production was fast at measurement time, so no layer is named yet; the duty supplies the forge-host side once it runs in production.Measured from outside. I timed three warm rounds of
/healthz,/,/pricing,/termsand/privacyfrom the sandbox withcurl -w, next to the app's ownServer-Timing. The table is in the comment on #13633. Every page answered in under 0.43 s:None of these runs reproduced the 2–20 s in #13633. So the wait sits nowhere right now, and the cause is still unnamed. The bad spell was intermittent and happened before or after the app's handler.
The forge-host side. The duty runs every six hours from the forge's own host. For each of the five pages it times three things:
127.0.0.1:<port>, which skips the edge and the tunnel.It reads
app;dur=fromServer-Timingalongside those.src/maintain/wait_split.tsthen turns the four timings into the layer's name:The largest share wins. A page under 1 s (#13633's bar) waits in
none.Each page is one line on
/maintenance, and a page that is slow or never answers is a finding. It sends GET only, and its timeout is the proof's own 15 s, not widened.To look at:
forge.jsonegressgainslab--forge-trevato-apps.open-platform.sh, the lab app's own public host. The forge reported that host refused 13 times, so without it the duty andv1-provencan't reach their own origin from inside the lab's cell.Not done here: no change to the server, the tunnel or the timeouts. Naming the cause and fixing it stay with #13633, and the next step there is to read the duty's lines after a slow spell. I left this issue's plan unkept: the
keep_plantool was never available in this run.Checks run:
bun scripts/gate.ts --fix: it rewrapped comments, which are committed.bun x tsc --noEmit: passed.bun scripts/fast.tsexits 2 becauseforge.jsonchanged, so I picked the covering tests myself:test/maintain-wait-split.test.ts,maintenance-week-doors,maintenance-duties,maintain-v1-proven,duty-idle,outbound-gateandtui-egress. All 53 tests pass.The split test checks each layer by hand from #13633's numbers. It would fail without the split.
To try it: run
bun test test/maintain-wait-split.test.ts. After a merge,/maintenanceshows await-splitline per page.🤖 Generated with Claude Code
Closes #13635