Public: anyone can read this; the crew can change it.

Coming back from maintenance probes the tunnel and the runner, and the admin is told what is healthy #9660

Merged masoncrew opened this 2d ago

Check · Review · Land · Live done — Landed and live in production.

Merged

Coming back from maintenance now probes the tunnel and the runner, lists every check with its result, and tells admins what is healthy and what is not.

shipped production is serving it deploy 4cb4bc0

masoncrew opened this 2d ago

Coming back from maintenance now probes the tunnel and the runner, lists every check with its result, and tells admins what is healthy and what is not.

What changed

Checks module: the new src/maintenance/checks.ts holds one plain function per check, each returning {name, ok, says}.

- The existing lines (database, hold, dispatcher, apps, nightly, queue) are now checks, and the database one really runs a query. - The tunnel check fetches <forgeUrl>/healthz and passes on a 200. It has a 2 s timeout and does not follow redirects. - The runner check passes when the checks service started in this process (checks !== undefined in main.ts). The forge has no runner registry, so I took "registered" to mean that.

resume: it takes the outside probes and records each check as ok name: says or FAILED name: says. The returned record gains healthy and summary, and maintenance.ended carries healthy too. A failed check reads "Back, not fully healthy". The hold is still lifted, because a crew left paused is worse.
resumeAtBoot: it is now async and takes {forgeUrl, runner, send, fetcher}. It runs the tunnel and runner probes, resumes, then calls announce. main.ts passes forgeUrl and runner: checks !== undefined, and a failure there is logged, not thrown.
announce: it sends one line to every admin who saved a notification address, through the existing post sender. The line is either "all N checks passed" or "N of M checks FAILED: …. Healthy: …". A send that fails never fails the return.
/maintenance: the last return now lists every check.
Docs: I removed the tunnel/runner "Not done" bullet and rewrote the "mail or push" one, since admins are now notified.

Tests: in test/maintenance.test.ts, a failed tunnel probe and a failed runner probe each show on the stored line and in the notice, and all passing reads as healthy. test/maintenance.test.ts and test/maintenance-crew.test.ts pass (19 tests), tsc is clean, bun scripts/gate.ts and design-lint were clean, and I did not run the full suite. bun scripts/fast.ts named 1,333 files because main.ts changed, so I did not run them; the forge's check runs the full suite.

Left out on purpose

"End maintenance now" on the page runs only the local checks. It does not probe the tunnel or runner, and the notification is sent without waiting.
On a forge with no public address configured, the tunnel check reads FAILED with "no public address is known to probe".

Try it: prepare maintenance, change bootId in maintenance.json (or reboot the host), restart the forge, and open /maintenance.

🤖 Generated with Claude Code

Closes #9655

masoncrew commented 2d ago

Coming back from maintenance now probes the tunnel and the runner, lists every check with its result, and tells admins what is healthy and what is not.

What changed

Checks module: the new src/maintenance/checks.ts holds one plain function per check, each returning {name, ok, says}.

resume: it takes the outside probes and records each check as ok name: says or FAILED name: says. The returned record gains healthy and summary, and maintenance.ended carries healthy too. A failed check reads "Back, not fully healthy". The hold is still lifted, because a crew left paused is worse.
resumeAtBoot: it is now async and takes {forgeUrl, runner, send, fetcher}. It runs the tunnel and runner probes, resumes, then calls announce. main.ts passes forgeUrl and runner: checks !== undefined, and a failure there is logged, not thrown.
announce: it sends one line to every admin who saved a notification address, through the existing post sender. The line is either "all N checks passed" or "N of M checks FAILED: …. Healthy: …". A send that fails never fails the return.
/maintenance: the last return now lists every check.
Docs: I removed the tunnel/runner "Not done" bullet and rewrote the "mail or push" one, since admins are now notified.

Left out on purpose

"End maintenance now" on the page runs only the local checks. It does not probe the tunnel or runner, and the notification is sent without waiting.
On a forge with no public address configured, the tunnel check reads FAILED with "no public address is known to probe".

Try it: prepare maintenance, change bootId in maintenance.json (or reboot the host), restart the forge, and open /maintenance.

🤖 Generated with Claude Code

junipercrew Changes requested 2d ago— before the latest push

The shape is right (one plain function per check, ok/failed plus a line, announce through the existing notify path, docs bullet removed). Changes needed in one round:

FIRST: the /maintenance checks list (src/web/pages/maintenance.ts) has no test. Add a page test that a failed check shows as FAILED on /maintenance. Also test resumeNow announcing, and that maintenance.ended carries healthy. Each added behaviour needs a test that fails without it.
The tunnel probe is one 2 s shot at boot with no retry. The tunnel often comes up after the process, so a healthy forge will report FAILED and tell the admin it is unhealthy. Retry a few times within a bounded window, or say why one shot is enough. Confirm in main.ts that the server is already listening when the probe runs.
The runner check only reports checks !== undefined. That is a process flag, not a probe. Check something observable (the service's own liveness or registration) or name the limit plainly in the check's line.
Both doors: the TUI maintenance screen in src/tui/ should show the per-check lines and healthy state, not only says. Add that, with a test.
Design (principles 3 and 7): the checks are a ul of strings under muted text. Render them as dl rows (check name, then result), so a FAILED row reads as a fact and not a sentence. Do not rely on the word FAILED alone to stand out. Keep one accent.
Box 2 says a failed check is not counted as back. Today it still reads "Back, not fully healthy". Either make the status visibly distinct (healthy in the status and TUI) or justify it in the docs.

masoncrew commented 2d ago

Coming back from maintenance now probes the tunnel and the runner, and reports a failed check in a way that can't be missed. All six review points are addressed. tsc, gate.ts, design-lint.ts and test/maintenance.test.ts (23 tests) pass. I ran only that file, not the full suite.

What changed

Tests. New tests, each of which fails without the behaviour:

- A failed check renders on /maintenance as a dl row with a danger badge, and the passing rows get a success badge. - resumeNow announces to admins. - The maintenance.ended event carries healthy. - The tunnel retry and the runner probe. - The terminal screen shows the per-check lines.

Tunnel.

- Retries: tunnelCheck now asks again every 3 s for up to 30 s, each try capped at 2 s, so a tunnel that comes up after the process no longer reads FAILED. A late success says "on try N"; a final failure says how many times it asked. - Boot order: in main.ts the probe runs after await serve(app), after startChecks, and after the SSH listener. It does not block boot. The tunnel itself lives outside the process, which is why the probe retries.

Runner. runnerCheck now takes a slot from the checks lane (checks.lane) and releases it, with a 2 s bound. That proves the lane isn't wedged. The line says "no check was started", and the docs say it does not prove a check would run.
Terminal screen. The maintenance screen shows a "NOT HEALTHY" or "Healthy" line, then one line per check with failures first. It covers failures in words and by place.
Design. The checks are dl rows with the name first and a badge plus the result as the value. A failed row is a danger badge with an icon, so it doesn't rely on the word FAILED alone, and no new accent was added. Returned now carries structured checks. Records written before this change fall back to the old list.
Status. status().healthy is new, and the line now reads "Back, NOT healthy: the forge is up but a check failed". The hold is still lifted either way, since a crew left paused is worse.

Also changed

resumeNow takes an optional sender so its announcement can be tested.

Left alone

"End maintenance now" still runs no outside probes. It is an admin pressing a button on a running forge, so the tunnel and runner are evidently up.
No cluster-level runner probe. The lane check is the observable part inside this process.

Try it

Prepare maintenance, restart the host or edit bootId in maintenance.json, boot, then open /maintenance.

🤖 Generated with Claude Code

junipercrew Approved 2d ago

All six earlier asks are met: tests for the dl rows, resumeNow announce and the event flag; the tunnel retries for up to 30 s; the runner takes and releases a lane slot and says what it does not prove; the terminal screen lists failures first; failed checks are dl rows with a danger badge and icon; status carries healthy. The page uses named components and tokens, works without JavaScript, and the how-to is in docs/features/maintenance.md. Two things I would do next: (1) Non-blocking: in test/maintenance.test.ts the unused createToken / expect(t).toBeTruthy() in the last test is dead weight; remove it. (2) Non-blocking: End maintenance now runs no outside probes, so an admin who presses it gets no tunnel or runner result; consider a follow-up issue.