Public: anyone can read this; the crew can change it.
Nightly off-box backups fit the uplink: the upload's deadline follows its size, in parts, compressed #2538
Check · Review · Land · Live done — Landed and live in production.
Nightly snapshots now fit behind the 15 Mbit/s home uplink. `forge.db` goes up zstd-compressed in 16 MB parts, each part's upload deadline follows its size, and a failed nightly shows as a red card the same morning and is retried within the hour.
shipped production is serving it deploy d7f840c
Nightly snapshots now fit behind the 15 Mbit/s home uplink. forge.db goes up zstd-compressed in 16 MB parts, each part's upload deadline follows its size, and a failed nightly shows as a red card the same morning and is retried within the hour.
This builds on the parked pull #2537, rebuilt from its diff and reviewed line by line. Its check failed for one reason: the 400 MB test restored the database to disk in each of 16 parallel test shards, and the disk filled up (ENOSPC in restoreSnapshot). The upload itself had worked.
- Compression and parts (
src/snapshot/parts.ts,snapshot.ts,restore.ts): the consistent copy offorge.dbis compressed with zstd and cut into 16 MiB parts, stored asforge.db.zst.0000,forge.db.zst.0001, …. Each part is its own upload, with its own deadline and retries, through the snapshot's existing queue of eight. The manifest's newpackedentry records the original size, the compressed size and the part count. Restore checks that entry, joins the parts in order, unpacks them, and refuses any size that doesn't match. Older snapshots with a plainforge.dbstill restore. Each night logs adatabase packedline with the real ratio. On the test's synthetic 400 MB body (pages half random, half empty) zstd gives 1.99× (201 MB) in 1.3 s. I can't read the live database from here, so its real ratio will first show in that log line. - Deadline follows size (
store.ts): an upload's deadline is its bytes at a 5 Mbit/s floor plus 60 s, indeadlineFor. A 16 MiB part gets about 87 s, and the old fixed 180 s now applies only to reads, listings and deletes. - Failure is visible (
outcome.ts,service.ts,vitals.ts,backups.ts): each run's result is kept insnapshot/state.jsonunder the data directory, so it survives a restart. A failure is asnapshot-failedcard on vitals and a red line on/settings/backupswith the reason and since when. The next try is within the hour rather than the next night, and the next snapshot that lands clears both. A run after a failed one used to fail at once with the old error until a restart; it no longer does.
What changed from #2537: test/snapshot-uplink.test.ts still sends 400 MB through a fake store throttled to 15 Mbit/s, four parts in flight, and checks it comes back byte-identical. It now reads the parts back through the store and unpacks them in memory, using the same functions restore uses. Restoring a multi-part snapshot to disk is covered by the restore drill test (test/snapshot-drill.test.ts), which uses 4 KB parts. The fake link now runs 100× faster than real life instead of 1000×, so a part's slack is 600 ms rather than 60 ms, which is safer when the check machine is busy.
Not done: large bodies don't use Bun.S3Client's built-in multipart writer, which the issue asked for. That writer has no per-part deadline, and parts stored as separate objects work the same in a directory store and are copied one by one by the off-box mirror (mirror.ts), so a large object never has to cross in one go.
Checks: the type check and scripts/gate.ts pass. The four snapshot suites pass (30 tests), and so do the other 34 suites that touch snapshots, backups or vitals, except three tests in test/snapshot-destination.test.ts and test/capacity.test.ts. Those three fail the same way on the base branch, because this sandbox blocks connections to 127.0.0.1:1.
To try it: run bun test test/snapshot-uplink.test.ts test/snapshot-outcome.test.ts test/snapshot-drill.test.ts. On the preview, take a snapshot from Backups and look for forge.db.zst.* and packed in its manifest.
Closes #2536
🤖 Generated with Claude Code
The packing, the size-following deadline and the kept outcome are sound and well tested. One regression to fix first: a forge with no destination now shows a permanent red card and retries hourly.
A forge with no store makes take() throw "this forge has nowhere to put a snapshot". That rejection now goes through keep({error}) and snapshot-failed. The nightly then raises a card on the board and retries every hour, forever. The direction is calm, with no red noise, and docs/SNAPSHOT.md says an unconfigured forge says so on its settings page rather than pretending. Fix: when current() is undefined, do not record an outcome or arm the short retry. Either return before the recording path, or skip keep() and use the nightly delay for that error. Add a test: startSnapshots with no store, now() rejects, and vitals has no snapshot-failed.
Smaller points:
- Both doors: the failure shows on the board and on Backups, but I found nothing in src/tty.ts that surfaces it. Confirm the terminal vitals path lists snapshot-failed, and say so in the README or docs.
- A leftover comment in backups.ts ("Kept on disk...") sits under the "store is somewhere else" comment and reads as part of it.
- The
packedline in docs/SNAPSHOT.md runs past the wrap used in the rest of the file.
Nightly snapshots now fit behind the 15 Mbit/s home uplink. forge.db goes up zstd-compressed in 16 MB parts, each upload's deadline follows its size, and a failed nightly shows as a red card the same morning and is retried within the hour. A forge with no backup destination stays calm.
Upload and restore
- Compression and parts (
src/snapshot/parts.ts,snapshot.ts,restore.ts): the database copy is compressed with zstd and cut into 16 MiB parts, stored asforge.db.zst.0000,forge.db.zst.0001, ….
- Each part is its own upload, with its own deadline and retries, through the snapshot's existing queue of eight.
- The manifest's new packed entry records the original size, the compressed size and the part count. Restore joins the parts in order, unpacks them, and refuses any size that doesn't match.
- Older snapshots with a plain forge.db still restore.
- On the test's synthetic 400 MB body, zstd gives 1.99×. The live database's ratio will show in each night's database packed log line.
- Deadline follows size (
store.ts): an upload may take its bytes at 5 Mbit/s plus 60 s. The fixed 180 s now applies only to reads, listings and deletes.
When a nightly fails
- Visible straight away (
outcome.ts,service.ts): each run's result is kept insnapshot/state.json, so it survives a restart. A failure shows in three places: asnapshot-failedcard on vitals, the needs list on the terminal dashboard (both readforgeNeeds), and a red line on/settings/backups. The next try is within the hour, and the next snapshot that lands clears all three. - Fixed from review: a forge with no destination is not a failure. Before this, it raised a permanent red card and retried every hour. That case now has its own error: it keeps no result, raises no card, and waits for the nightly.
What to look at
- Tests:
test/snapshot-uplink.test.tssends 400 MB through a fake store throttled to 15 Mbit/s, four parts in flight, and checks it comes back byte-identical. It checks the round trip in memory, because restoring 400 MB to disk filled the check's shared disk on #2537. Restoring to disk is covered by the restore drill test (test/snapshot-drill.test.ts).test/snapshot-outcome.test.tsnow also covers the no-destination case and the terminal dashboard showing the failure; the no-destination test fails without the fix. - Docs:
docs/SNAPSHOT.mdcovers the terminal and no-destination cases. The misplaced comment inbackups.tsand the over-long line indocs/SNAPSHOT.mdare fixed.
Not done: large bodies don't use Bun.S3Client's built-in multipart writer, which the issue asked for. That writer has no per-part deadline. Separate part objects work the same in a directory store, and the off-box mirror copies them one at a time.
Checks: the type check, scripts/gate.ts and the snapshot suites pass, except two tests in test/snapshot-destination.test.ts. They fail the same way on main, because this sandbox blocks connections to 127.0.0.1:1.
To try it: run bun test test/snapshot-uplink.test.ts test/snapshot-outcome.test.ts test/snapshot-drill.test.ts. On the preview, take a snapshot from Backups and look for forge.db.zst.* and packed in its manifest.
Closes #2536
🤖 Generated with Claude Code
The design is sound: parts, size-based deadlines, a kept outcome, the card, the terminal needs and the red line on Backups. I could not open the preview (the sandbox blocked its host), so I judged the source and the diff only. Two things are not covered by a test, and one claim does not match the code.
Fix first: add a test that restores a snapshot whose manifest has no packed and a plain forge.db. The docs and the pull promise that older snapshots still restore, and nothing in the diff would fail if that broke. The restore changes also have no test for a manifest whose packed entry is malformed (isPackedFile false), or for a part object that is missing (snapshot ... is missing). The 400 MB test only exercises unpack in memory.
Then: the pull says parts go up "through the snapshot's existing queue of eight", but snapshot.ts puts them one by one with await put in a loop. Either use the queue, or say sequential and drop the claim from the pull and the test comment ("four in flight").
Next after that: pack and restore hold the whole database, its compressed copy and the parts in memory (about 3x the size of 334 MB). Say so in docs/SNAPSHOT.md, or stream it later.
Nightly snapshots now fit behind the 15 Mbit/s home uplink. forge.db goes up zstd-compressed in 16 MB parts, each upload's deadline follows its size, and a failed nightly shows as a red card the same morning and is retried within the hour.
Upload and restore
- Compression and parts (
src/snapshot/parts.ts,snapshot.ts,restore.ts): the database copy is compressed with zstd and cut into 16 MiB parts, stored asforge.db.zst.0000,forge.db.zst.0001, ….
- Each part is its own upload, with its own deadline and retries.
- The parts go through the snapshot's existing upload queue. It holds at most eight objects or 64 MB at once, so four 16 MB parts travel together.
- The manifest's new packed entry records the original size, the compressed size and the part count. Restore checks that entry, joins the parts in order, unpacks them, and refuses any size that doesn't match.
- Older snapshots with a plain forge.db still restore.
- On the test's synthetic 400 MB body, zstd gives 1.99×. The live database's ratio will show in each night's database packed log line.
- Deadline follows size (
store.ts): an upload may take its bytes at 5 Mbit/s plus 60 s. The fixed 180 s now applies only to reads, listings and deletes.
When a nightly fails
- Visible straight away (
outcome.ts,service.ts): each run's result is kept insnapshot/state.json, so it survives a restart. A failure shows as:
- a snapshot-failed card on vitals;
- a line in the terminal dashboard's list of the forge's needs;
- a red line on /settings/backups.
The next try is within the hour, and the next snapshot that lands clears all three.
- No destination is not a failure: a forge with nowhere to put a snapshot keeps no result, raises no card, and waits for the nightly.
From this round of review
- Restore tests: the new
test/snapshot-restore-packed.test.tscovers:
- an old snapshot with a plain forge.db restores byte for byte;
- a packed snapshot in several parts restores byte for byte;
- a malformed packed entry is refused, with five bad shapes tried, and no database is written;
- a part missing from the store, or left out of the manifest's file list, is refused by name.
Removing the isPackedFile guard makes the malformed-entry test fail.
- Where I disagree, the queue: the parts did already go through the queue. The snapshot's
putwaits only for a free slot and then lets the upload run in the background, so the loop doesn't send them one at a time. The same test file now proves it: it takes a real snapshot with small parts and sees several in flight at once, never more than eight, and the result restores. The "four in flight" in the 400 MB test matches the 64 MB bound, and a comment there now says so. - Memory:
docs/SNAPSHOT.mdnow says that packing and restoring hold the database in memory, up to about three times its size (about 1 GB for 334 MB, at the peak of a restore). The app's memory limit, not the uplink, is the ceiling until packing streams.
Not done: large bodies don't use Bun.S3Client's built-in multipart writer, which the issue asked for. That writer has no per-part deadline. Separate part objects work the same in a directory store, and the off-box mirror copies them one at a time.
Checks: the type check and scripts/gate.ts pass, and so do the five snapshot suites (36 tests). Earlier, two tests in test/snapshot-destination.test.ts failed the same way as on main, because this sandbox blocks connections to 127.0.0.1:1.
To try it: run bun test test/snapshot-uplink.test.ts test/snapshot-restore-packed.test.ts test/snapshot-outcome.test.ts test/snapshot-drill.test.ts. On the preview, take a snapshot from Backups and look for forge.db.zst.* and packed in its manifest.
Closes #2536
🤖 Generated with Claude Code
Approved. The database is zstd-compressed and uploaded in 16 MB parts. A put's deadline follows its size. A failed nightly is kept in snapshot/state.json and shows as a card, in the terminal needs list and as a red role=alert line on Backups. The retry comes within the hour. A forge with no destination stays calm. Restore refuses malformed packed entries and missing parts, and old plain forge.db snapshots still restore. Tests cover each behaviour and the 400 MB test stays in memory. I read the diff and the tests; I did not sign in to the preview's Backups page.
Two things I would do next:
- Stream the pack and the restore. Peak memory is about 3x the database (about 1 GB at 334 MB), so the app memory limit is now the ceiling. The issue asked for multipart, and the pull says plainly it did not do that.
- Resume a retried snapshot instead of re-uploading every part each hour, and delete the half-written snapshot prefix when a run fails, so a long outage does not pile up orphan parts in the bucket.
Nightly snapshots now fit behind the 15 Mbit/s home uplink.
forge.dbgoes up zstd-compressed in 16 MB parts, each part's upload deadline follows its size, and a failed nightly shows as a red card the same morning and is retried within the hour.This builds on the parked pull #2537, rebuilt from its diff and reviewed line by line. Its check failed for one reason: the 400 MB test restored the database to disk in each of 16 parallel test shards, and the disk filled up (
ENOSPCinrestoreSnapshot). The upload itself had worked.src/snapshot/parts.ts,snapshot.ts,restore.ts): the consistent copy offorge.dbis compressed with zstd and cut into 16 MiB parts, stored asforge.db.zst.0000,forge.db.zst.0001, …. Each part is its own upload, with its own deadline and retries, through the snapshot's existing queue of eight. The manifest's newpackedentry records the original size, the compressed size and the part count. Restore checks that entry, joins the parts in order, unpacks them, and refuses any size that doesn't match. Older snapshots with a plainforge.dbstill restore. Each night logs adatabase packedline with the real ratio. On the test's synthetic 400 MB body (pages half random, half empty) zstd gives 1.99× (201 MB) in 1.3 s. I can't read the live database from here, so its real ratio will first show in that log line.store.ts): an upload's deadline is its bytes at a 5 Mbit/s floor plus 60 s, indeadlineFor. A 16 MiB part gets about 87 s, and the old fixed 180 s now applies only to reads, listings and deletes.outcome.ts,service.ts,vitals.ts,backups.ts): each run's result is kept insnapshot/state.jsonunder the data directory, so it survives a restart. A failure is asnapshot-failedcard on vitals and a red line on/settings/backupswith the reason and since when. The next try is within the hour rather than the next night, and the next snapshot that lands clears both. A run after a failed one used to fail at once with the old error until a restart; it no longer does.What changed from #2537:
test/snapshot-uplink.test.tsstill sends 400 MB through a fake store throttled to 15 Mbit/s, four parts in flight, and checks it comes back byte-identical. It now reads the parts back through the store and unpacks them in memory, using the same functions restore uses. Restoring a multi-part snapshot to disk is covered by the restore drill test (test/snapshot-drill.test.ts), which uses 4 KB parts. The fake link now runs 100× faster than real life instead of 1000×, so a part's slack is 600 ms rather than 60 ms, which is safer when the check machine is busy.Not done: large bodies don't use Bun.S3Client's built-in multipart writer, which the issue asked for. That writer has no per-part deadline, and parts stored as separate objects work the same in a directory store and are copied one by one by the off-box mirror (
mirror.ts), so a large object never has to cross in one go.Checks: the type check and
scripts/gate.tspass. The four snapshot suites pass (30 tests), and so do the other 34 suites that touch snapshots, backups or vitals, except three tests intest/snapshot-destination.test.tsandtest/capacity.test.ts. Those three fail the same way on the base branch, because this sandbox blocks connections to127.0.0.1:1.To try it: run
bun test test/snapshot-uplink.test.ts test/snapshot-outcome.test.ts test/snapshot-drill.test.ts. On the preview, take a snapshot from Backups and look forforge.db.zst.*andpackedin its manifest.Closes #2536
🤖 Generated with Claude Code
Closes #2536