Public: anyone can read this; the crew can change it.

Slow public pages alarm by name: past one second warm, the forge says which layer and which setting holds the limit #14034

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The wait-split duty now raises an alarm when a public page is slow. It already timed /healthz, /, /pricing, /terms and /privacy from the forge's own host and recorded which layer held the wait. Now, when a warm GET takes over 1 s, it also files an issue for that layer.

Notes

shipped production is serving it deploy 4adf612

opened this

The wait-split duty now raises an alarm when a public page is slow. It already timed /healthz, /, /pricing, /terms and /privacy from the forge's own host and recorded which layer held the wait. Now, when a warm GET takes over 1 s, it also files an issue for that layer. The issue names the setting that holds that layer's limit, with its reason beside it, so a repeat is loud instead of a line on /maintenance that nobody reads. It works like the disk-free duty: one open issue stands for a layer, however many runs see it.

  • src/maintain/wait_limits.ts (new): a table from layer to the setting that holds its limit. Each entry has the setting, where it is set, its value and the reason. Values are read from the code itself, so the issue never shows a stale number.
    • tunnel → FORGE_ORIGIN
    • accept → SLICE_MS
    • app → SLOW_SQL_MS
    • edge → no forge setting exists: the row says so and points at the Cloudflare zone, rather than inventing one.
  • src/maintain/duties/wait_split.ts: groups the slow pages by layer. For each layer it files one issue in the forge's own repository, titled "A public page waits past 1 s in the <layer>". The body has the setting, its value, the reason and each page's measured line.
  • test/maintain-wait-alarm.test.ts: fast pages file nothing. Slow pages file one issue per layer with the right setting and reason, and a second run files nothing new. Without the alarm, this test fails.

Checks: the new test and the existing wait-split tests pass, with the other maintain suites and the repository-shape lints (218 tests across 31 files). bun x tsc --noEmit is clean and bun scripts/gate.ts --fix was run (its rewrap is in the commit). scripts/fast.ts selected 140 test files (about 11 minutes); I ran only that subset and left the full suite to the forge's check.

Left out:

  • A page that gets no answer at all is still only a finding: with no timings there is no layer to name.
  • No change to how fast anything is. The cause is still unknown, so there was nothing to fix yet.

Still to prove: the second done item (bun scripts/v1-proven.ts against production passing #11140 and #13116) can only be checked after merge. A curl from the sandbox just now was still intermittently slow: /healthz 1.30 s, / 1.88 s, /privacy 4.06 s, /pricing 0.36 s, /terms 0.70 s. Once this is live, the issues it files will name the layer; that and the passing probe go on #13633 before it closes.

To try it: bun test test/maintain-wait-alarm.test.ts. On production, the duty runs every six hours, and any issue it files appears in the forge's own repository.

🤖 Generated with Claude Code

Closes #13636

2 earlier comments · 2 events
commented

The wait-split duty now raises an alarm when a public page is slow. It already timed /healthz, /, /pricing, /terms and /privacy from the forge's own host and recorded which layer held the wait. Now, when a warm GET takes over 1 s, it also files an issue for that layer. The issue names the setting that holds that layer's limit, with its reason beside it, so a repeat is loud instead of a line on /maintenance that nobody reads. It works like the disk-free duty: one open issue stands for a layer, however many runs see it.

  • src/maintain/wait_limits.ts (new): a table from layer to the setting that holds its limit. Each entry has the setting, where it is set, its value and the reason. Values are read from the code itself, so the issue never shows a stale number.
    • tunnel → FORGE_ORIGIN
    • accept → SLICE_MS
    • app → SLOW_SQL_MS
    • edge → no forge setting exists: the row says so and points at the Cloudflare zone, rather than inventing one.
  • src/maintain/duties/wait_split.ts: groups the slow pages by layer. For each layer it files one issue in the forge's own repository, titled "A public page waits past 1 s in the <layer>". The body has the setting, its value, the reason and each page's measured line.
  • test/maintain-wait-alarm.test.ts: fast pages file nothing. Slow pages file one issue per layer with the right setting and reason, and a second run files nothing new. Without the alarm, this test fails.

Checks: the new test and the existing wait-split tests pass, with the other maintain suites and the repository-shape lints (218 tests across 31 files). bun x tsc --noEmit is clean and bun scripts/gate.ts --fix was run (its rewrap is in the commit). scripts/fast.ts selected 140 test files (about 11 minutes); I ran only that subset and left the full suite to the forge's check.

Left out:

  • A page that gets no answer at all is still only a finding: with no timings there is no layer to name.
  • No change to how fast anything is. The cause is still unknown, so there was nothing to fix yet.

Still to prove: the second done item (bun scripts/v1-proven.ts against production passing #11140 and #13116) can only be checked after merge. A curl from the sandbox just now was still intermittently slow: /healthz 1.30 s, / 1.88 s, /privacy 4.06 s, /pricing 0.36 s, /terms 0.70 s. Once this is live, the issues it files will name the layer; that and the passing probe go on #13633 before it closes.

To try it: bun test test/maintain-wait-alarm.test.ts. On production, the duty runs every six hours, and any issue it files appears in the forge's own repository.

🤖 Generated with Claude Code

Changes requested — before the latest push

The alarm is wired cleanly and the test fails without it. One miss against box 1: the settings it names do not bound the wait. (1) tunnel: FORGE_ORIGIN is where cloudflared forwards, not a limit on how long it waits, so changing it cannot shorten a slow tunnel hop. (2) app: SLOW_SQL_MS only logs a slow query; it bounds nothing about the page. (3) edge: no setting, said honestly, but then the card has no setting and no fix. Fix: name for each layer the setting that actually bounds time in that layer (a timeout, a budget or a queue limit the forge or cloudflared config holds), or say plainly in the card that the layer has no forge-held limit and what to check instead, and word the issue so it does not claim a setting holds the limit when it only logs or routes. Keep the test asserting the named setting. Also: box 2 (v1-proven against production) can only be proven after merge, so it stays left. Minor: the test sets process.env.FORGE_SENTINEL_REPO at module scope, which leaks into other test files in the same bun process; set and restore it inside the tests.

commented

When a public page takes more than 1 s on a warm request, the alarm card now only names a setting if that setting actually limits time in the layer. If no forge setting does, the card says so and says what to check instead.

The card is built in src/maintain/wait_limits.ts. Each layer now has one of two kinds of entry:

  • A real limit (kind: "bound"): the card gives the setting, where it is set, its current value and the reason. It reads "The setting that bounds this layer's time: …".
  • No limit (kind: "none"): the card gives the reason and what to look at. It reads "No forge-held setting bounds this layer's time: … Check instead: …".

What each layer's card says now:

  • edge: no limit. The TLS handshake (the secure-connection setup) ends at Cloudflare before any byte reaches the host. Check Cloudflare's status page and the zone's analytics.
  • tunnel: no limit. The cloudflared config the forge generates sets no origin-request timeouts, so cloudflared's own defaults apply. FORGE_ORIGIN is only where the tunnel forwards, so the card no longer names it. Check journalctl -u cloudflared, and whether the host answered quickly on loopback at the same moment.
  • accept: a real limit, SLICE_MS in src/kernel/breathe.ts. It is the longest a background job may hold the single serving thread before a waiting request gets answered. The review did not object to this one, so it is unchanged.
  • app: no limit. Nothing caps one handler's time below the server's 120 s idleTimeout in src/http/server.ts, which only closes a silent connection and is far past 1 s. SLOW_SQL_MS is now named only as the threshold for the slow-query log, together with the page's Server-Timing: app value.

The test checks that the tunnel card says no limit is held, no longer mentions FORGE_ORIGIN, and points to the cloudflared log. It checks that the accept card names SLICE_MS with its value. For every layer it checks that the card either names the setting and reason or says no limit is held and what to check. It also checks that SLOW_SQL_MS is never called the app's limit. With the source change removed, two of the three tests fail. The test now sets FORGE_SENTINEL_REPO in beforeAll and restores the old value in afterAll, so it no longer leaks into other test files.

I chose not to add timeouts to the generated cloudflared config. That would change how the live tunnel behaves and belongs in its own issue. Proving the alarm against production also stays open, since it can only be done after this merges.

Run bun test test/maintain-wait-alarm.test.ts. bun scripts/gate.ts (which rewrapped one comment, now committed) and bun x tsc --noEmit passed before the push.

🤖 Generated with Claude Code

Approved

Earlier ask met: the tunnel card no longer calls FORGE_ORIGIN a limit. Layers with no forge-held bound say so and name what to check. SLOW_SQL_MS appears only as a log threshold, and the test leaks no env. The duty only reads and files one issue per layer through fileIssueOnce, so there is no new input, write door or dependency. The tests would fail without the alarm. Non-blocking: a page with no answer still files no issue, which the pull states. Box 2 can only be proved after merge, so it stays open.

commented

Left #13636 open: 1 unticked item remain (bun scripts/v1-proven.ts against production no longer times out; #11140 and #13116 pass there. Do not close #13633 before that probe passes.).

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs