Public: anyone can read this; the crew can change it.
The forge box comes back by itself after a killed process and a reboot, and a drill proves it #11899
Check · Review · Land · Live done — Landed and live in production.
`scripts/box-drill.sh` is a new drill that shows the forge box comes back on its own after its process is killed and after a reboot. It has two modes:
shipped production is serving it deploy 4e6f862
scripts/box-drill.sh is a new drill that shows the forge box comes back on its own after its process is killed and after a reboot. It has two modes:
kill(run as root): sends SIGKILL to the forge's main process and waits until a new process answers/with a 2xx or 3xx. It prints how many seconds that took. It fails if/isn't answering before the kill, or doesn't answer again withinDRILL_LIMITseconds (default 120).boot(run afterreboot): fails unless the unit isenabled, active and answering/. It then prints how many seconds after the boot the forge came up, read fromActiveEnterTimestampMonotonic.
Both modes take an optional URL; the default is http://127.0.0.1:${PORT:-3000}/.
infra/forge-box/forge.service already had Restart=always, RestartSec=2, WantedBy=multi-user.target and StartLimitIntervalSec=0, so the unit is unchanged. test/forge-box-unit.test.ts now also requires RestartSec to be between 1 and 10 seconds; it already checked the other two settings.
test/box-drill.test.ts runs the script with stand-ins for systemctl, curl, id and uptime. The kill test kills a real sleep process. The other tests cover the usage message, / not answering before the kill, a passing boot check, and a unit that isn't enabled at boot.
docs/EXTRACT.md, under "The box beside production", now has a section saying how to run the drill and how to check after a reboot.
Not done: the third item, posting output from a real kill and reboot on the box, still needs someone with root on the box. This sandbox cannot reach it. Leave the issue open until both outputs are posted on it.
Checks: bun scripts/gate.ts and bun x tsc --noEmit pass, as do test/box-drill.test.ts and test/forge-box-unit.test.ts. scripts/fast.ts said the diff was too broad to name tests (exit 2), so I picked those two files myself.
To try it, on the box as root in /opt/forge:
scripts/box-drill.sh kill
reboot
scripts/box-drill.sh boot🤖 Generated with Claude Code
Approved. The unit already has Restart=always, RestartSec=2 and WantedBy=multi-user.target. The test now bounds RestartSec to 1-10 s. The drill script is strict (set -euo pipefail), requires a new pid that answers 2xx/3xx, and is bounded by DRILL_LIMIT. The tests drive both modes against stubs, including the failure paths. The how-to is in docs/EXTRACT.md, not README. No UI is touched, so there is nothing to check for keyboard, 360px or no-JS. Next: (1) run the drill on the box and post both outputs on the issue; (2) add a boot-mode check that the unit was not started by hand, e.g. that ExecMainStartTimestamp falls within the boot window.
Left #9047 open: 1 unticked item remain (Output from one run on the box (kill and reboot) is posted on this issue.).
scripts/box-drill.shis a new drill that shows the forge box comes back on its own after its process is killed and after a reboot. It has two modes:kill(run as root): sends SIGKILL to the forge's main process and waits until a new process answers/with a 2xx or 3xx. It prints how many seconds that took. It fails if/isn't answering before the kill, or doesn't answer again withinDRILL_LIMITseconds (default 120).boot(run afterreboot): fails unless the unit isenabled, active and answering/. It then prints how many seconds after the boot the forge came up, read fromActiveEnterTimestampMonotonic.Both modes take an optional URL; the default is
http://127.0.0.1:${PORT:-3000}/.infra/forge-box/forge.servicealready hadRestart=always,RestartSec=2,WantedBy=multi-user.targetandStartLimitIntervalSec=0, so the unit is unchanged.test/forge-box-unit.test.tsnow also requiresRestartSecto be between 1 and 10 seconds; it already checked the other two settings.test/box-drill.test.tsruns the script with stand-ins forsystemctl,curl,idanduptime. The kill test kills a realsleepprocess. The other tests cover the usage message,/not answering before the kill, a passing boot check, and a unit that isn't enabled at boot.docs/EXTRACT.md, under "The box beside production", now has a section saying how to run the drill and how to check after a reboot.Not done: the third item, posting output from a real kill and reboot on the box, still needs someone with root on the box. This sandbox cannot reach it. Leave the issue open until both outputs are posted on it.
Checks:
bun scripts/gate.tsandbun x tsc --noEmitpass, as dotest/box-drill.test.tsandtest/forge-box-unit.test.ts.scripts/fast.tssaid the diff was too broad to name tests (exit 2), so I picked those two files myself.To try it, on the box as root in
/opt/forge:🤖 Generated with Claude Code
Closes #9047