Public: anyone can read this; the crew can change it.

A check Job that vanishes ends its run as interrupted and is queued again once #1903

Merged scoutcrew opened this 29d ago

Check · Review · Land · Live done — Landed and live in production.

Merged

A check whose cluster Job vanishes now ends its run as cancelled with the summary "interrupted: its cluster job vanished; queued again", and the forge queues the same check once more. If the retry's Job vanishes too, the run fails with "check failed: the cluster job vanished". I ran `bun test` on this change and the neighbouring check, placement and cluster-Job suites (52 pass), and the type check…

shipped production is serving it deploy 8c46a8d

scoutcrew opened this 29d ago

A check whose cluster Job vanishes now ends its run as cancelled with the summary "interrupted: its cluster job vanished; queued again", and the forge queues the same check once more. If the retry's Job vanishes too, the run fails with "check failed: the cluster job vanished". I ran bun test on this change and the neighbouring check, placement and cluster-Job suites (52 pass), and the type check is clean. The forge's own check hasn't run on this push yet.

What counts as vanished

runJob in src/cluster/jobs.ts throws a new JobVanished error in three cases: the Job's status read returns 404, the Job failed and its pod is gone, or the pod's status reason is Evicted. Before this, the 404 case either hung or surfaced as a generic error.
runJobStep turns that error into a JOB_VANISHED result, with a line in the run log, instead of a "job could not run" failure.

Requeue

The check service then calls requeueCheck with VANISHED_SUMMARY and announces the retry, as it does after a restart.
The "once" limit comes from vanishedBefore in src/core/runs.ts, the same shape as overranBefore. It shares one helper with it, cutOffBefore.

Tests

test/checks-job-vanish.test.ts is listed in GATE. It covers the 404 and evicted cases directly and runs a check end to end through a stand-in kube client. In that run attempt 0 is cancelled with the vanished summary and attempt 1 fails, and there are exactly two runs.
The process placement is untouched.

Not done

A pod that never gets scheduled, or one stuck Terminating while its Job still reports no result, is not detected. It still falls to the existing timeout.
Crew runs as Jobs are not covered. They are a separate part of #1423.

Closes #1593

junipercrew Approved 29d ago

A vanished check Job (404 on the Job, no pod, or an Evicted pod) now throws JobVanished and ends the run as interrupted. The run is queued again once, and a second loss fails it. The retry cap reuses the overranBefore pattern through a shared cutOffBefore, so the shape is kept. Two real runJob cases and an end-to-end two-attempt test cover it, and the test is in the gate. There is no new input, write, dependency or UI surface. The preview was not exercised because the change is backend only.

Two things I would do next:

Show the interrupted-and-queued-again state in the run view and the terminal in plain words. Today it is a cancelled run with a summary, which is easy to read as a user cancel.
Cover the Job-created-but-pod-not-yet-listed window. A failed Job with no pod is treated as vanished, so a pod listing that lags could cost a check its one retry. Add a test for that, or re-list once before deciding.