Maintenance: shut down for a move and come back by itself (#9453)
Prepare. An admin opens /maintenance (linked from Resilience), the terminal's Maintenance command
(src/tui/screens/maintenance.ts: p prepare, c check again, e end; admins only), or calls
POST /api/v1/admin/maintenance/prepare (admin-scope token) or MCP
maintenance {action: "prepare"}. The forge then:
- holds releases (an owner hold,
keeper/hold.ts); - pauses the crew: the dispatcher starts nothing while
maintenance.json
stands in the data directory; vitals reads the queue as held by "planned maintenance";
- takes a fresh snapshot (
actions/backups_now.ts); - reports the drain:
GET /api/v1/admin/maintenancesaysdrainingwhile runs
or checks still go, ready when the backup is taken and nothing runs. Power
off on ready; a SIGTERM drains the rest as in a release.
The page says plainly what stops: releases, new crew runs, then the forge, its apps and previews when the host powers off.
The page. /maintenance answers one question first: one status line, "Healthy: nothing needs you" or
"Needs you" with what waits (maintenance on, a failed return, overdue or paused duties, stale pulls; needsYou
in src/web/pages/maintenance.ts). Below it, each part is a disclosure with a count in its summary:
Duties, Budgets, Prepare for a power-off (or Maintenance state while on), Last return. A fold opens by
itself only when something in it needs a person. Pulls untouched for 14 days are a fold too, open, with their
count, only when there are some; /system keeps them as a section under its own heading. While maintenance is
on, "End maintenance now" sits under the status line. The forge's looks picture it as the owner (the slot was
/repos'). Tests: test/maintenance-calm.test.ts, test/maintain-stale-web.test.ts.
Duties. /maintenance, the terminal's Maintenance screen and the duties of MCP maintenance {action: "status"}
list each nightly job and the radar in one line (src/maintenance/duties.ts, one function for all three doors):
when it last ran (UTC), what it found ("nothing found" when it filed nothing), and "overdue". A duty is
overdue when now is past its last slot plus one day plus 6 hours of grace (a night may run long); one
that never ran says so and gives its slot, and is not overdue.
Come back. On boot resumeAtBoot acts only when the host has restarted since the prepare
(/proc/sys/kernel/random/boot_id differs from the one stored); a release roll of the process on
the same host boot keeps what the admin prepared. It then lifts the hold, unpauses the crew and
records a line (shown on /maintenance, in the status, and as the
maintenance.ended event) with every check and its result (src/maintenance/checks.ts,
one plain function each: database, hold, dispatcher, apps, nightly lines, queue, and at boot
the tunnel, which must answer /healthz on the public address (asked again every 3 s for up to
30 s, since it often comes up after the process; the listener is already up when this runs), and the
runner, whose checks lane must hand out a slot within 2 s: that proves the lane is not wedged, not that
a check would run). A failed check reads FAILED as a badge on a dl row of /maintenance, first
in the terminal screen; the status reads "Back, NOT healthy" with healthy: false; the hold is lifted
regardless. Admins with a
notification address are told what is healthy and what is not. An admin can also press
"End maintenance now" (no outside probes then). The nightly jobs show on Resilience.
No Kubernetes is asked anything; the state is one file.
While it is off. bun scripts/standby.ts 8080 serves the calm 503 page
(src/maintenance/page.ts) from the droplet; point the tunnel's or
Cloudflare's origin at it for the move, and back after. A Cloudflare Worker can
return the same HTML with status 503 and Retry-After.
Is the standby up? Set MAINTENANCE_STANDBY_URL (an http(s) URL, e.g. http://127.0.0.1:8080/) in
the forge's environment. /maintenance and GET /api/v1/admin/maintenance (field standby) then fetch
that one URL server-side (2 s timeout, no redirects) and say whether it answers 503 with the maintenance
page. The URL is never taken from a request. When ready, the line reads "Ready, standby answering",
"Ready, standby not answering" or "Ready, no standby". Unset, the page says how to start one:
bun scripts/standby.ts 8080 on the droplet, then point the tunnel or Cloudflare origin at it.
Size of a standby. For reads or a whole forge: disk for the data (38 GB now, apps 13 GB) plus one snapshot and headroom, so 64 GB; 4 GB of memory free for the forge alone, 8 GB with apps running. The 16 GB droplet cannot hold it.
Take over reads (#9657). On a second machine of that size, with the backup
store's variables (S3_*, or FORGE_SNAPSHOT_DIR) and FORGE_SNAPSHOT_PASSPHRASE:
bun scripts/standby-restore.ts latest /srv/standby # or a snapshot name; the directory must be empty FORGE_READ_ONLY=1 FORGE_DATA=/srv/standby PORT=3000 bun run src/main.ts
The script restores the snapshot (src/snapshot/restore.ts), runs SQLite's
integrity check and confirms every repository the database lists is on disk;
it exits 1 and names what is wrong otherwise. With FORGE_READ_ONLY=1 the forge
serves pages, the API's reads and git clone/fetch from that directory. One guard
at the front of the request path (src/http/read_only.ts) answers every other
method, a WebSocket upgrade and the push door (git-receive-pack) with a plain
503 "read-only standby", so a route added later is covered. Nothing past the
listener starts: no terminal, apps, crew, checks, releases, mirrors, snapshots
or nightly jobs, and not even the boot's reconcile or any timer. The
standby knows no one: a session or token of the main forge is read as signed
out, so it shows public pages only, and sign-in (a POST) is refused. Opening
the database still migrates it forward, on the restored copy only. Point the tunnel's origin at it for the move.
Test: test/standby-restore.test.ts.
Not done
- Writes on the standby, and switching back: the standby is a copy as of its
snapshot, so anything written on the main forge after it is not there, and nothing carries standby-time changes back (there are none; it takes no writes). Nothing replicates continuously; re-restore into a new directory to catch up.
- The page is not pushed to Cloudflare automatically;
standby.tsis started by hand (the forge only checks it). - The admin is told on the page, event and status, and by the notification address they saved (
/account); no mail. - Where the kernel gives no boot id, any restart ends maintenance.
Stale issues (#13592)
The stale-issues duty runs once a day and files one issue, "Open issues untouched for 60 days with no
pull and no blocker", naming up to 20 open issues of the forge's own repository whose last comment or
edit is over 60 days old, that no open pull says it closes (Closes #n) and that wait for no open
blocker (oldest first; the rest are counted). While that report is open, later nights file none; closing
it lets the next night file a fresh list. The duty never comments on, labels or closes an issue it names.
It shows as the stale-issues row on /maintenance. Test: test/maintain-stale-issues.test.ts.
Flaky tests (#13937)
The flaky-tests duty runs once a day and names each test file whose tests failed and then passed
alone on the same commit in the last 7 days, with the count, distinct tests and commits. It reads
check_flake, which stores a test name, its file and the commit (no finer per-test data), so the report
is per file. A quiet week reports nothing; it files no issue. It shows as the flaky-tests row on
/maintenance. Test: test/maintain-flaky-tests.test.ts.