A busy forge is not a dead one: the liveness probe restarted the forge under load tonight, losing three runs #1221

Tonight the live forge was restarted by its own liveness probe at 05:22:07Z: /healthz answered 503, kubelet killed the container ("Container forge failed liveness probe, will be restarted"), three crew runs were interrupted and re-queued, the in-flight work of each was lost. At 04:27Z the same probe had already timed out once ("context deadline exceeded") without killing. The pod was busy, not broken: one full check, four crew runs, 1.9 cores and 3.7 GiB in use on a 4-core droplet while the host ran a restore drill beside it.

A probe that restarts a busy process makes the busy moment worse: every restart re-queues the runs and re-reads the repositories. Alive and ready are different questions.

Done looks like:

• [ ] /healthz answers 200 whenever the process is up and the database opens, however slow the moment; readiness (/readyz or a field) is what says "do not route to me yet" and never restarts anything.
• [ ] The chart's liveness probe (charts/app, and the droplet values in mitosis scripts/standby/values/trevato-apps--forge--prod.yaml if they set probes) gets timeoutSeconds ≥ 10 and failureThreshold ≥ 6, so a minute of slowness is tolerated; readiness may stay quick.
• [ ] When the forge answers 503 on purpose (shutting down, preflight hold), the log says why in one line at that moment; tonight's log has no line between "run queued again" and "forge shutting down".
• [ ] test/health.test.ts: a slow database query (fake) does not turn /healthz into 503; a shutdown does, with the log line.

Evidence: kubectl -n trevato-apps--forge--prod get events 05:22Z; pod restarts=2 tonight (04:02 OOM at checks=2, 05:22 liveness).

Happened again on 0.7.71, pod forge-f9bf4b474-76pqc (limits cpu 3, memory 5Gi): Liveness probe failed: Get http://10.42.0.17:8080/healthz: context deadline exceeded at about 17:40–18:09Z, then Container forge failed liveness probe, will be restarted. The forge drained cleanly (18:09:46Z forge draining sshSessions=1, crew draining runs=3; three runs re-queued "stopped for a restart"; forge shutting down 18:12:48Z, exit 0) and the new container started about 18:14Z, so roughly 1–2 minutes unavailable after three minutes of drain. Eighteen minutes later the readiness probe timed out once more on the fresh container, so the cause is load, not a leak: /healthz is answered by the same event loop as the check lane, the crew runs and the pages, and when that loop is busy the probe times out and kubelet kills a healthy forge. The origins log was also repeating origin token refresh failed reason=invalid_response every few seconds in the minute before. Acceptance to add: /healthz answers within the probe timeout under a full lane and three crew runs (a probe that reads a heartbeat the event loop updates, or a dedicated listener), and a busy forge is never killed for being busy.