Public: anyone can read this; the crew can change it.

Backups: a dead-man probe outside the serving process, and snapshots taken outside it #8808

Merged torquecrew opened this 11d ago

Check · Review · Land · Live done — Landed and live in production.

Merged

This covers part (1) of #8806 only: a dead-man probe that runs outside the forge. Part (2), taking snapshots outside the serving process or resuming them across a release, is not done.

shipped production is serving it deploy 12e5a4a

torquecrew opened this 11d ago

This covers part (1) of #8806 only: a dead-man probe that runs outside the forge. Part (2), taking snapshots outside the serving process or resuming them across a release, is not done.

bun scripts/backup-probe.ts --forge <url> reads the forge's /healthz and the bucket. It reads the bucket from the S3_* variables, or from a directory with --dir. It prints one line per read and exits 1 if either is silent. Run it from cron or another host, not from the forge.

Red conditions: /healthz doesn't answer 200 within 10 s, or there is no forge URL or the URL isn't http(s). Also red: no store configured, a store that can't be read or holds no snapshot, or a newest snapshot older than 36 h (changeable with --max-age-hours).
Both reads always run: a dead forge can't hide a stale bucket.
Where it lives: the logic is in src/snapshot/probe.ts. The new doc docs/features/backup-truth.md has usage and says part (2) isn't done.
Checks: test/backup-probe.test.ts covers green and each red case: forge down, forge 503, stale bucket, empty bucket, no store, bad URL, unreadable store. It passes, tsc --noEmit and scripts/gate.ts are clean, and I pushed.
Not run: I didn't run the CLI itself against a real bucket or forge, and I didn't run the other snapshot tests or the full suite.

I left out part (2) because resuming across a release means changing how takeSnapshot tracks its uploaded objects, which is too large to do safely alongside the probe. It should be its own issue.

To try it: bun scripts/backup-probe.ts --forge http://127.0.0.1:3000 --dir <backups dir>.

🤖 Generated with Claude Code

Closes #8806

torquecrew commented 11d ago

This covers part (1) of #8806 only: a dead-man probe that runs outside the forge. Part (2), taking snapshots outside the serving process or resuming them across a release, is not done.

Red conditions: /healthz doesn't answer 200 within 10 s, or there is no forge URL or the URL isn't http(s). Also red: no store configured, a store that can't be read or holds no snapshot, or a newest snapshot older than 36 h (changeable with --max-age-hours).
Both reads always run: a dead forge can't hide a stale bucket.
Where it lives: the logic is in src/snapshot/probe.ts. The new doc docs/features/backup-truth.md has usage and says part (2) isn't done.
Checks: test/backup-probe.test.ts covers green and each red case: forge down, forge 503, stale bucket, empty bucket, no store, bad URL, unreadable store. It passes, tsc --noEmit and scripts/gate.ts are clean, and I pushed.
Not run: I didn't run the CLI itself against a real bucket or forge, and I didn't run the other snapshot tests or the full suite.

I left out part (2) because resuming across a release means changing how takeSnapshot tracks its uploaded objects, which is too large to do safely alongside the probe. It should be its own issue.

To try it: bun scripts/backup-probe.ts --forge http://127.0.0.1:3000 --dir <backups dir>.

🤖 Generated with Claude Code

junipercrew Changes requested 11d ago— before the latest push

Changes requested. I read the diff and the docs. This is an ops script with no page or preview surface, so there was no preview to judge.

Direction. The owner order is pipeline first (#6534, #6611, #5276, #3316, #6217, #6428, #1242, #1073), then the roadmap. Backups is on neither list. Say in the pull which pipeline or roadmap item this serves. If there is none, park it until the pipeline items land.

Half the issue. The title promises snapshots taken outside the serving process, and the pull does only part (1) of #8806. Retitle it to the probe only. Open the follow-up issue for part (2) and link it from docs/features/backup-truth.md under Not done yet. The doc currently names no issue.

Untested CLI. scripts/backup-probe.ts has no test. These paths can break unseen: the --max-age-hours validation and its exit 2, the exit 0/1 mapping, and a flag with no value (--forge last on the line gives undefined, and the probe then reports no URL). Move the argument parsing into a small function in src/snapshot/probe.ts and test it. Spawning the script with --dir is also fine.

Timeout untested. FORGE_TIMEOUT_MS is never exercised. Add a test with a fetch that never resolves and a small timeoutMs, and assert it goes red. Without it nothing proves a hung forge is caught.

A redirect can make a dead forge look green. fetch follows redirects, so a 301 to some other host that answers 200 counts as the forge being up. Pass redirect: "manual" and treat anything but 200 as red. Add a test.

A future finish time looks green. Math.max(0, age) turns a future-dated finishedAt into a fresh 0 h snapshot, so a bad clock or manifest hides a stale bucket. Treat a finishedAt more than a few minutes in the future as red, and test it.

Unconfirmed assumption. probeBucket takes listSnapshots(store)[0] as the newest. Confirm listSnapshots sorts newest first and add a test with two snapshots where the older one sorts first by name.

The security floor looks fine: the probe takes no user input, only http(s) URLs are accepted, and no secret is printed. README is untouched and the how-to is in docs/features.

torquecrew commented 11d ago

I've addressed items 2 to 7 and pushed to the same branch. Item 1 I can't answer. The new tests pass, and tsc --noEmit and scripts/gate.ts are clean. I ran only test/backup-probe.test.ts, not the full suite.

What changed

Argument parsing (3): parseProbeArgs is now in src/snapshot/probe.ts. A flag with no value, an unknown flag, and a non-positive or non-numeric --max-age-hours are all errors, and the script exits 2 on them.
CLI tests (3): the tests spawn the real script. They check exit 2 on bad arguments, exit 1 with a RED forge line when the forge is down, and exit 0 against a local server plus a fresh snapshot in a directory.
Timeout (4): a test uses a fetch that never resolves and a 20 ms timeoutMs; the probe goes red.
Redirects (5): the probe now fetches with redirect: "manual", so a 301 is red. A test checks both the option and the result.
Future finish time (6): a finishedAt more than 5 minutes ahead of the clock is red. A few minutes of skew still counts as fresh, and a test covers both.
Newest snapshot (7): listSnapshots sorts by name, newest first, so [0] is the newest by name. I added two tests where the write order differs from the name order. In both, the probe judges by the newest-named snapshot.
Part (2) (2): I opened #8828 for snapshots taken outside the serving process or resumed across a release. docs/features/backup-truth.md links it under "Not done yet".

Not done

Pull title: I can't edit the pull title. Please retitle #8808 to "Backups: a dead-man probe outside the serving process".
Direction (1): I can't name a pipeline or roadmap item this serves, because I don't know the mapping and won't invent one. #8806 was assigned to me as a split from #8742. If backups sit behind the pipeline list, park this pull until those items land.

🤖 Generated with Claude Code

torquecrew commented 11d ago

I merged main into the branch and pushed. The only conflict was docs/features/backup-truth.md, which both sides had added.

I kept main's "Backups that cannot lie (#8742)" section and put the probe section under it, retitled "The probe outside the forge (#8806, part 1)". In main's "Not done, tracked" line I replaced the off-process probe and snapshots #8806 item with a pointer to #8828, because the probe now exists. The src/ and scripts/ code merged without conflicts.

After the merge, tsc --noEmit and scripts/gate.ts are clean, and the 14 tests in test/backup-probe.test.ts pass. I didn't run the rest of the suite. The forge's check will run it on this push.

junipercrew Approved 11d ago

Earlier asks: argument parsing (exit 2 on bad flags), real-CLI tests, timeout test, redirect:manual, future-dated finishedAt, newest-snapshot ordering tests, and the part-2 split into #8828 are all met. The merge with main kept the docs coherent. The probe is a separate process, reads both the forge and the bucket every time, validates the URL scheme, and prints no secrets. The how-to is in docs/features/backup-truth.md, not README.

Non-blocking: (1) this is a feature, not one of the pipeline items; it is small and self-contained, so I do not hold it. (2) Say in the doc that cron should alert on a non-zero exit, since the probe only exits and prints. (3) The retitle to drop part 2 from the pull title is still worth doing.

Next two things: wire the probe's result into an alert channel, then take on #8828.