Observability 4: /boards, /map and home answer fast - boards 8-10 s and /map 34.7 s cold today #3710

I cut /boards, a cold /map and a cold home to between 40 and 360 ms on a fixture sized like production; at production's size they took 17–38 s before. Types are clean and the 27 related test files pass (373 tests), including the five cross files CLAUDE.md names. The forge's check on the push hasn't reported yet.

/map cold (34.7 s). The time was in countCrossings in src/web/pages/map.ts, which compared every pair of edge segments on every ordering pass. It now counts each gap between two levels with a Fenwick-tree inversion count, so it no longer compares every pair. Only segments that point up or skip levels (which only a cycle produces) still use the pairwise test. On production's real graph (1,314 issues, 2,266 edges, fetched from the map API) the layout went from 30.2 s to about 0.3 s, with exactly the same crossing count (684,525). A seeded random test checks the new count against the old pairwise one on 200 graphs that include cycles.

/boards (8–10 s). The time was in the flow panel's SQL:

• The "landed" stage and the ask→live median scanned every run in a repository once per merged pull, looking for the next production deploy.
• Each stage's outflow, conflicted pulls and today's crew spend each scanned the whole run table.
• The "ready" check for open asks scanned every open issue.

One additive migration (20261021100000-flow-indexes.ts) adds mostly partial indexes on run, plus indexes on pull, page_vital and session. The "ready" subquery uses CROSS JOIN so SQLite keeps the indexed order; the answer is unchanged. While testing, a plain run(kind, created_at) index made the planner drop the deploy index, so the spend index is limited to agent runs. The forge never runs ANALYZE, so production plans these queries the way the tests do. /boards now reports session, boards, flow and render in its Server-Timing.

Home (kpis 1,032 ms). With the indexes, p75 reads a week of page vitals once instead of counting and sorting twice per column. It still picks the value at rank ⌈0.75·n⌉; a hand-worked unit test checks that.

Tests. test/observability-budget.test.ts builds the fixture: 100,000 runs, 1,500 merged pulls, 400 open asks, 2,266 edges and 90,000 page views. It holds /boards, a cold /map and a cold home (cache missed on purpose) to a multiple of one full pass over the run table, timed in the same run. The wall clock is never the measure (#3069).

test/map-layout-budget.test.ts holds the layout on its own to under three of its own pairwise counts. The old code needed about 50.

Not done:

• I didn't stream the page shell first. Each page now answers well under its target in one render, so streaming would add complexity for no visible gain.
• The fixture doesn't account for production's home time beyond kpis (9.2 s in one run). If it comes back, the Server-Timing phase names will show where.
• The migration builds its indexes on run once at the first boot after deploy. That costs a few seconds on a large table.

To try: bun test test/observability-budget.test.ts test/map-layout-budget.test.ts. On the preview, signed in as an admin, compare the Server-Timing headers on /boards, /map and / with the numbers in the issue.

🤖 Generated with Claude Code

Closes #3693

I cut /boards, a cold /map and a cold home to between 40 and 360 ms on a fixture sized like production; at production's size they took 17–38 s before. Types are clean and the 27 related test files pass (373 tests), including the five cross files CLAUDE.md names. The forge's check on the push hasn't reported yet.

/map cold (34.7 s). The time was in countCrossings in src/web/pages/map.ts, which compared every pair of edge segments on every ordering pass. It now counts each gap between two levels with a Fenwick-tree inversion count, so it no longer compares every pair. Only segments that point up or skip levels (which only a cycle produces) still use the pairwise test. On production's real graph (1,314 issues, 2,266 edges, fetched from the map API) the layout went from 30.2 s to about 0.3 s, with exactly the same crossing count (684,525). A seeded random test checks the new count against the old pairwise one on 200 graphs that include cycles.

/boards (8–10 s). The time was in the flow panel's SQL:

• The "landed" stage and the ask→live median scanned every run in a repository once per merged pull, looking for the next production deploy.
• Each stage's outflow, conflicted pulls and today's crew spend each scanned the whole run table.
• The "ready" check for open asks scanned every open issue.

One additive migration (20261021100000-flow-indexes.ts) adds mostly partial indexes on run, plus indexes on pull, page_vital and session. The "ready" subquery uses CROSS JOIN so SQLite keeps the indexed order; the answer is unchanged. While testing, a plain run(kind, created_at) index made the planner drop the deploy index, so the spend index is limited to agent runs. The forge never runs ANALYZE, so production plans these queries the way the tests do. /boards now reports session, boards, flow and render in its Server-Timing.

Home (kpis 1,032 ms). With the indexes, p75 reads a week of page vitals once instead of counting and sorting twice per column. It still picks the value at rank ⌈0.75·n⌉; a hand-worked unit test checks that.

Tests. test/observability-budget.test.ts builds the fixture: 100,000 runs, 1,500 merged pulls, 400 open asks, 2,266 edges and 90,000 page views. It holds /boards, a cold /map and a cold home (cache missed on purpose) to a multiple of one full pass over the run table, timed in the same run. The wall clock is never the measure (#3069).

test/map-layout-budget.test.ts holds the layout on its own to under three of its own pairwise counts. The old code needed about 50.

Not done:

• I didn't stream the page shell first. Each page now answers well under its target in one render, so streaming would add complexity for no visible gain.
• The fixture doesn't account for production's home time beyond kpis (9.2 s in one run). If it comes back, the Server-Timing phase names will show where.
• The migration builds its indexes on run once at the first boot after deploy. That costs a few seconds on a large table.

To try: bun test test/observability-budget.test.ts test/map-layout-budget.test.ts. On the preview, signed in as an admin, compare the Server-Timing headers on /boards, /map and / with the numbers in the issue.

🤖 Generated with Claude Code

Fixes three slow pages: /boards and home now read the flow through indexes, and the map's layout no longer slows down quadratically as the graph grows.

• /map layout (src/web/pages/map.ts): countCrossings used to compare every pair of hops on every ordering pass. It now counts crossings in each gap between levels as inversions with a Fenwick tree. Hops that point upward or skip a level only occur in cycles, and they keep the old pairwise test. The crossing counts and the layout are the same as before. Production's graph now lays out in about 0.3 s instead of 30 s.
• /boards and home (src/flow/pipeline.ts, src/flow/reach.ts, src/web/pages/boards-view.ts): the "landed" stage and ask-to-live used to walk every run of a repository once per merged pull. Stage outflow, conflicted pulls, today's crew spend and the ready-ask branch check each scanned a whole table. One additive migration (20261021100000-flow-indexes.ts) adds indexes for what they read. A CROSS JOIN pins the order of READY's subquery so it uses them. Home's p75 reads a week of vitals in one pass. On a fixture sized like production this goes from 30 s to 40 ms. /boards now reports its phases in Server-Timing as session, boards, flow and render; the existing names are unchanged.
• Tests: test/observability-budget.test.ts holds /boards, a cold /map and a cold home to a multiple of one full pass over 100,000 runs, timed in the same run, so they don't depend on wall-clock speed in loaded cells (#3069). test/map-layout-budget.test.ts checks that the new crossing count matches the pairwise one and keeps layout within its budget. The fixture is in test/helpers/flow-fixture.ts. Both suites pass (9 tests).

Not done: the issue also asks to stream the page shell first and to cache by data version. With these fixes the pages are already well under their targets, so I left both out; they can follow if production numbers say otherwise. I also haven't touched the map's 2.6 MB response size.

The issue carries the "Later" label because no pillar is named. Someone needs to name one before it moves; nothing in the code depends on that.

To try it, open /boards, /map (cold) and / on the preview and read the Server-Timing header. Or run bun test test/observability-budget.test.ts test/map-layout-budget.test.ts.

🤖 Generated with Claude Code

The full gate on land was red; the pull is open again. Red tests:

• (unnamed)
• (unnamed) [4122.80ms] [shard 511]
• (unnamed) [41.18ms] [shard 524]
• (unnamed) [0.09ms] [shard 524]

Fixes three slow pages: /boards and home now read the flow through indexes, and the map's layout no longer slows down quadratically as the graph grows. I couldn't find the land gate's red, and this push includes no fix for it — details below.

• /map layout (src/web/pages/map.ts): countCrossings used to compare every pair of hops on every ordering pass. It now counts crossings in each gap between levels as inversions with a Fenwick tree. Hops that point upward or skip a level only occur in cycles, and they keep the pairwise test. The crossing counts and the layout are the same as before. On production's graph, layout drops from 30 s to 0.3 s.
• /boards and home (src/flow/pipeline.ts, src/flow/reach.ts, src/web/pages/boards-view.ts): one additive migration (20261021100000-flow-indexes.ts) indexes what the pipeline stages, ask-to-live, conflicted pulls, crew spend and the ready-ask branch check read. A CROSS JOIN makes READY's subquery use those indexes. Home's p75 reads a week of vitals in one pass. On a fixture sized like production this goes from 30 s to 40 ms. /boards now reports session, boards, flow and render in its Server-Timing.
• This push:

- main is merged in. - Both of juniper's review notes are applied. A gap's width is now found with a reduce instead of Math.max(...hops), so a very wide gap can't hit the argument limit. - pairwiseCrossings is gone from the module. The test now writes the pairwise definition out itself, so the oracle shares no code with what it checks.

• Land gate red: it listed only "(unnamed)" failures, which are errors or failed hooks outside any test, and I couldn't find that run's log. I ran every test/*.test.ts on the merged tree, one process per file. Seven files failed: axe, looks, crew-sandbox-real, crew-interrupted, parity, scaffold and snapshot-destination. Each fails the same way on origin/main in this sandbox (no browser, no ripgrep, loopback refused), so none of them comes from this pull. Both budget tests also pass under --coverage, as the gate's --impact mode runs them. If the gate goes red again, I need its log to say more.
• Same-second migration: #3730 also has a migration stamped 20261021100000. Different names are allowed and they run in either order. I didn't rename mine: a preview database has it recorded already, and under a new name its CREATE INDEX would run again and fail.

Not done: streaming the page shell first, caching by data version, and the map's 2.6 MB response size. The pages already answer well under their targets, so these can follow if production says otherwise. The issue is still under "Later" until someone names a pillar.

To try it, open /boards, /map (cold) and / on the preview and read the Server-Timing header. Or run bun test test/observability-budget.test.ts test/map-layout-budget.test.ts.

🤖 Generated with Claude Code

/boards breaks its Server-Timing header into named phases, and the map and the home page get the same treatment. This branch now merges main cleanly.

The merge had one conflict, in src/web/pages/boards-view.ts (boardsList):

• main side: flowPanelCss and the page's styleCss were removed, because the flow panel's styles moved into the design system (#3823).
• This branch: the flow panel is rendered inside timePhase(ctx.request, "flow", …), so /boards reports a flow phase alongside session, boards and render.
• Resolution: both are kept. The flow panel is still timed as the flow phase, and the page no longer passes its own styleCss.

The type check and the boards, flow, timing, map and home tests pass (440 tests).

🤖 Generated with Claude Code

The full gate on land was red; the pull is open again. Red tests:

• (unnamed)
• (unnamed) [4533.99ms] [shard 540]
• (unnamed) [5.30ms] [shard 554]
• (unnamed) [0.17ms] [shard 554]
• (unnamed) [5.37ms] [shard 555]
• (unnamed) [0.17ms] [shard 555]

The /boards, /map and home speed-ups are unchanged. This push is aimed at the red land gate, and the fix is a likely cause I couldn't reproduce or confirm from the gate's own output.

What I could confirm. I couldn't get the land gate's log, so the failures below are worked out from the red-test list.

• What "(unnamed)" means: bun prints it for a failed beforeAll. When an afterAll then throws on state the failed setup never filled in, a second "(unnamed)" follows.
• Which pulls went red: among the 58 pulls I checked, this is the only one whose land gate went red that way, and it happened both times.
• What the failures look like: the failures form two shapes:

- one setup failing about 4.1 s in the first time and 4.5 s the second; - setups in other shards failing within 5 to 41 ms, each followed by its afterAll crash.

• What I ran locally: the full gate (767 files, one process each, under coverage, as --gate --impact runs them) passes on this branch merged with today's main.

Probable cause. test/observability-budget.test.ts seeds a fixture sized like production. The seed takes about 4.2 s and left 114 MB on disk, half of it in the WAL, on a disk the gate shares across suites running side by side. A seed failing as it finishes, with neighbours unable to create their databases at that moment, matches both shapes above. That is consistent with a full disk, but not proven.

What changed:

• test/helpers/flow-fixture.ts:

- It truncates the WAL once the seed is in (PRAGMA wal_checkpoint(TRUNCATE)). - It writes 30,000 page views instead of 90,000; the count is now a size parameter. - The suite now holds 49 MB instead of 114 MB. - The 100,000 runs, 1,500 pulls and 2,266 edges stay. The map's budget is measured against a full pass over the run table, so it needs that size: at 20,000 runs a cold /map went over its limit.

• test/observability-budget.test.ts: afterAll no longer throws when setup didn't finish, so a setup failure is reported once, with its own error.
• main merged in: a clean merge.

The type check is clean and the budget suite passes under coverage. No test assertion changed.

If the gate goes red again with only "(unnamed)", I need the lines bun printed above those entries in the log. The gate also seems worth checking for how full its disk gets by shard ~500 (temporary directories that suites don't clean up would explain it). That is outside this pull.

From juniper's review: the issue still names no pillar. Streaming the shell, keying the cache by data version and the map's 2.6 MB response stay deferred until production numbers call for them.

To try it: bun test --coverage test/observability-budget.test.ts test/map-layout-budget.test.ts, or open /boards, a cold /map and / on the preview and read the Server-Timing header.

🤖 Generated with Claude Code

The full gate on land was red and named no test; the pull is open again.

The full gate on land was red and named no test; the pull is open again.

Parked by the crew: the check failed in the place it ran, not in the change, and failed the same way when it was run again

Speeds up the boards view, the map and the home page, with a merge of the latest main so the pull applies cleanly again (#3693). The type check is clean, and these suites pass: observability-budget, map-layout-budget, schema-migrations and boards-sampler. The full check has not been run here.

What the branch does:

• Database indexes: a new migration, flow-indexes, adds indexes for the flow queries. The flow code that uses them is in src/flow/pipeline.ts and src/flow/reach.ts.
• Pages: changes to src/web/pages/boards-view.ts and src/web/pages/map.ts.
• Speed tests: test/observability-budget.test.ts and test/map-layout-budget.test.ts set time budgets for these pages.
• Test data location: the larger test data is written to /.test-scratch/ in the checkout, not the check cell's 256 MiB /tmp (test/helpers/scratch.ts).

Merge resolution:

• .gitignore: keeps this branch's /.test-scratch/ and main's /.tmpx/.
• Migration renamed: the new migration was 20261021100000-flow-indexes, which shared its timestamp with main's already-released 20261021100000-migrate-mirror-target. Its new name is 20261022120000-flow-indexes, so it sorts after every migration already on main. Migrations run in name order, so a production database that has already applied main's migrations runs this one after them. This migration hasn't been released, so no pinned history changes.

To try it: open /boards and /map on the preview at https://lab--3710--forge-trevato-apps.open-platform.sh, or run bun test test/observability-budget.test.ts test/map-layout-budget.test.ts.

🤖 Generated with Claude Code

This pull speeds up the boards view, the map and the home page (#3693). With this change, /boards names session and render in its Server-Timing header.

The check failed on its last run because test/server-timing-sweep.test.ts still listed /boards as a page that does not name its timings yet. That test also fails when a listed page has started naming them, so it fails until each page that now does is taken off the list. This commit removes the /boards entry (it was filed under #3390). The test was out of date, not the code.

test/server-timing-sweep.test.ts and test/observability-budget.test.ts pass and the type check is clean. The full check was not run here; the forge runs it on this push.

What the branch does:

• Database indexes: a new migration, 20261022120000-flow-indexes, adds indexes for the flow queries in src/flow/pipeline.ts and src/flow/reach.ts.
• Pages: changes to src/web/pages/boards-view.ts and src/web/pages/map.ts.
• Speed tests: test/observability-budget.test.ts and test/map-layout-budget.test.ts set time budgets for these pages.
• Test data location: the larger test data is written to /.test-scratch/ in the checkout, not the check cell's 256 MiB /tmp (test/helpers/scratch.ts).

From the earlier merge of main:

• .gitignore: keeps this branch's /.test-scratch/ and main's /.tmpx/.
• Migration renamed: the new migration was first named 20261021100000-flow-indexes. That shared a timestamp with main's already-released 20261021100000-migrate-mirror-target. Migrations run in name order, and the new name sorts after every migration already on main. This one hasn't been released, so no recorded migration history changes.

To try it: curl -sI https://lab--3710--forge-trevato-apps.open-platform.sh/boards | grep -i server-timing, or run bun test test/server-timing-sweep.test.ts test/observability-budget.test.ts test/map-layout-budget.test.ts.

🤖 Generated with Claude Code