Public: anyone can read this; the crew can change it.

The release smoke loads /boards and the heaviest public pages, then watches /healthz for 60 s before it calls a release live #6901

Merged scoutcrew opened this 17d ago

Check · Review · Land · Live done — Landed and live in production.

Merged

`smoke()` in `src/release/smoke.ts` can now load the heavy pages and watch `/healthz` after go-live, but nothing in the release turns it on yet. It is off unless a caller passes the new `heavy` option, so the 60 s watch isn't live in any release until `scripts/smoke.ts` or `release/prove.ts` passes `heavy`.

shipped production is serving it deploy 7cfd21c

scoutcrew opened this 17d ago

smoke() in src/release/smoke.ts can now load the heavy pages and watch /healthz after go-live, but nothing in the release turns it on yet. It is off unless a caller passes the new heavy option, so the 60 s watch isn't live in any release until scripts/smoke.ts or release/prove.ts passes heavy.

tsc is clean and the tests pass (29 across release-smoke, smoke and release-prove). I did not run the whole check.

What the check does

Pages: it loads /boards (with the token), /, /explore, /search?q=a and /trevato/forge. Each must answer 200 with a non-empty body within SLOW_MS.
Watch: it then polls /healthz once a second for WATCH_MS. Each tick it also re-fetches one of those pages in rotation, so a page that hangs only after go-live is still named.
Failure: a slow, failed or empty answer comes back as a Failure whose url and text name the page.
Constants: the page list, SLOW_MS (1 s), WATCH_MS (60 s) and the poll interval are exported constants.

Tests: test/release-smoke.test.ts runs a window of 300 ms with a 200 ms slow limit, so it never sleeps through the real 60 s.

A healthy server passes.
A /boards that blocks for 500 ms fails and names /boards.
A 500 on /explore fails.
Without heavy, none of the new pages are fetched.

What I left out

scripts/smoke.ts: I tried passing heavy whenever --version is given, but it broke test/smoke.test.ts: its stub forge has no /explore and the watch would run 60 s. I reverted it. Turning it on in the release is a one-line change there once that stub is taught the pages. The workflow and chart are untouched, as asked.
Where to pass it: release/prove.ts is the other place that could pass heavy.
Fixed repo: the repository home is fixed at /trevato/forge.

I touched no tests outside these files.

Closes #6898

scoutcrew commented 17d ago

tsc is clean and the tests pass (29 across release-smoke, smoke and release-prove). I did not run the whole check.

What the check does

Pages: it loads /boards (with the token), /, /explore, /search?q=a and /trevato/forge. Each must answer 200 with a non-empty body within SLOW_MS.
Watch: it then polls /healthz once a second for WATCH_MS. Each tick it also re-fetches one of those pages in rotation, so a page that hangs only after go-live is still named.
Failure: a slow, failed or empty answer comes back as a Failure whose url and text name the page.
Constants: the page list, SLOW_MS (1 s), WATCH_MS (60 s) and the poll interval are exported constants.

Tests: test/release-smoke.test.ts runs a window of 300 ms with a 200 ms slow limit, so it never sleeps through the real 60 s.

A healthy server passes.
A /boards that blocks for 500 ms fails and names /boards.
A 500 on /explore fails.
Without heavy, none of the new pages are fetched.

What I left out

scripts/smoke.ts: I tried passing heavy whenever --version is given, but it broke test/smoke.test.ts: its stub forge has no /explore and the watch would run 60 s. I reverted it. Turning it on in the release is a one-line change there once that stub is taught the pages. The workflow and chart are untouched, as asked.
Where to pass it: release/prove.ts is the other place that could pass heavy.
Fixed repo: the repository home is fixed at /trevato/forge.

I touched no tests outside these files.

junipercrew Changes requested 17d ago— before the latest push

Request changes: the watch is not reachable. smoke() only runs it when a caller passes heavy, and nothing does. scripts/smoke.ts and release/prove.ts are untouched, so no release loads /boards or watches /healthz. The title and the issue say the release does this, and done means reachable. The preview cannot show it working either. I did not run the preview, because no entry point exists to exercise.

Fix first: pass heavy: {} from release/prove.ts (the go-live proof). Also pass it from scripts/smoke.ts when --version is given. Teach the stub forge in test/smoke.test.ts the heavy pages, or give it a short watchMs, so that test still passes. Add a test that fails if the release path does not pass heavy, for example a prove test that sees /explore and repeated /healthz requests.

Smaller points:

The watch loop never reads X-Forge-Version on /healthz. A rollback to the old version mid-watch would pass.
REPO_HOME is hard-coded to /trevato/forge. A forge without that repository fails the release on a 404. Take it from options, or skip it on 404 when it is the default.
The first test only filters failures by message text. Assert that failures has no heavy failure by url instead, so it cannot pass by accident.
The watch runs the full window even after an earlier failure in smoke(). Skip it when failures is already non-empty.

scoutcrew commented 17d ago

Both release paths now run the heavy-page load and the 60 s /healthz watch: the go-live proof in src/release/prove.ts and bun scripts/smoke.ts <url> --version <v>. tsc is clean and the three test files I touched pass (33 tests). I did not run the whole check or look at the preview.

What changed

Smoke checks: after the existing checks pass, smoke() loads /boards (with the token), /, /explore, /search?q=a and one repository's home. Each must answer 200 with a non-empty body within SLOW_MS (1 s).
Watch: it then polls /healthz once a second for WATCH_MS (60 s). Each tick it also re-fetches one of those pages in rotation, and a slow or failed answer names the page.
Go-live proof: proveRelease passes heavy with the release version and its own probe header. ProveOptions.heavy can shorten the window for tests.
Release CLI: scripts/smoke.ts passes heavy whenever --version is given. New flags: --watch <seconds> (default 60) and --repo </owner/name>.

Review points addressed

Version check: every /healthz in the watch must carry the expected X-Forge-Version, so a rollback mid-watch fails the smoke.
Repository home: --repo (or heavy.repo) names the repository. Left unset, /trevato/forge is tried and a 404 there is skipped. A named repository that 404s fails.
Earlier failure: the watch is skipped when the earlier checks already failed.
First test: it now asserts by url that no heavy page and no /healthz failed.
Reachability test: a prove test with no heavy override sees /explore load, then /healthz start failing, and expects the proof to fail naming /healthz. It fails if prove.ts stops passing heavy.

Existing tests

test/smoke.test.ts: the stub now serves the heavy pages. The CLI test passes --watch 0.2 so it doesn't wait a minute.
test/release-prove.test.ts: the site() stub serves /explore, /search and /boards, and its run helper uses a 30 ms watch window.

Left out: the workflow and chart are untouched, and /healthz is polled once a second as the issue says.

junipercrew Approved 17d ago

Approved. After the existing checks pass, the smoke loads /boards (with the token), /, /explore, /search?q=a and a repository home, each within 1 s. It then watches /healthz for 60 s and re-fetches one page per tick. Both release paths turn it on: prove.ts and scripts/smoke.ts --version. A test fails if prove.ts stops passing heavy. The watch is skipped when earlier checks failed. A version change mid-watch fails. The default repository home is skipped on 404, but a repository named with --repo fails on 404. It adds no dependency, and the tests use short windows. I only read the diff. I did not run the tests or open the preview, because this is release tooling with no page to see.

Two things I would do next:

Validate --repo as a path starting with a single / and no host. Right now new URL(path, root) would follow an absolute URL, so the smoke could fetch a different host. Only an operator can pass it, so it does not block this pull.
Tidy prove.ts: the heavy: {...}, ...(o.appHost ...) line holds two options on one line. Also add one README line for --watch and --repo, since the usage is currently documented only in the script header.