Public: anyone can read this; the crew can change it.
A backup finishes on a slow line: no fixed time budget for the whole snapshot, fewer chunks at once, and a failed run's chunks count #13745
Check · Review · Land · Live done — Landed and live in production.
A put's deadline and the number of puts at once now follow the rate the line is giving, and the Backups page no longer says "No backup" beside a new sealed lane. I did not run the "Done when" test: a 2 GB forge behind a 1 Mbit/s uplink on the stage rig.
shipped production is serving it deploy 8e1361f
7 earlier comments · 1 event
A put's deadline and the number of puts at once now follow the rate the line is giving, and the Backups page no longer says "No backup" beside a new sealed lane. I did not run the "Done when" test: a 2 GB forge behind a 1 Mbit/s uplink on the stage rig. The new tests use fake slow stores and a metered directory store, and the bun x tsc --noEmit check and the backup and snapshot tests pass here. I did not run the full suite. scripts/fast.ts named 2,313 of 3,523 test files for this diff, so I ran the snapshot, backup, restore, Resilience and settings tests instead.
Cause of the failures. The 66 s and 63 s in the log are the fixed floor rule, 60 s of slack plus 4 MiB at 5 Mbit/s. It is not the whole-run budget shrinking. withRetries only cuts deadlines to a run budget through bounded, which only the restore drill uses.
What changed
- Deadline. A put's deadline is now the longer of the floor and three times its bytes at the measured rate, over its share of the line. The new
src/snapshot/line.tsmeasures that rate as bytes landed over the time any put was in flight, andstore.tsand the sealed store use it. A put that moves nothing still times out at that deadline. I could not time out on "no bytes moving" itself, becauseBun.S3Client.writereports no progress. - Puts at once. The queue in
takeSnapshotallows one per megabit of measured rate, between 1 and 8, and 2 until the line has been measured. On 1.2 to 1.75 Mbit/s that is one. - Resume. This was already true for the sealed lane. Every run lists the lane's chunks, under their hidden names, and sends only the missing ones. A new test fails a run partway, then checks that the next run sends only the rest and reports "N of M chunks already there". The counters still start from zero each run, so it still can't be told from the old log.
- Progress. The
in progresslog line, the Backups page, the Resilience screen and the live events now say chunks sent, "N of M chunks already there", bytes to go, the rate and when it will be done. Bytes to go is an estimate from the last snapshot's size. A first sealed run takes it from the unsealed lane's newest snapshot, and it is blank when none is known. - Earlier snapshots. A sealed destination now knows the unsealed store beside it (
Store.before,src/snapshot/before_sealing.ts). Until the sealed lane holds a snapshot of its own, the verdict counts the earlier ones, so the line reads "Backup 13 h old …" and not "No backup in the bucket". If the earlier ones are over 26 h old, the verdict is still red.
- The lane says "14 from before sealing, newest 13 h ago, restorable". - The Backups page and the Resilience screen list them. - They restore with sealing off.
- One run at a time. The nightly timer skipped the
activecheck, so it queued a second run behind one still sending. It now returns when a run is in flight, and the run that ends arms the next. The new test fails with the check removed.
I also added docs/features/slow-line-backups.md and a line in docs/SNAPSHOT.md.
Not done
- The two puts-at-once limits are constants (about 1 Mbit/s per put, 2 before measuring). They are not settings.
- Unsealed snapshots are still not pruned once sealing is on. The doc already says to delete them by hand after a sealed drill.
takeSnapshot's own 3-try wrapper around the store's 5 attempts is unchanged.
To try it. Open /settings/backups while a snapshot runs. With FORGE_SNAPSHOT_SEAL=1 on a bucket that holds unsealed snapshots and no sealed one, the page lists the earlier ones. The tests are in test/snapshot-slow-line.test.ts and test/snapshot-one-core.test.ts.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
🤖 Generated with Claude Code
<!-- look-pages d6f1db23d38b --> Look pages for d6f1db2, from the gate:
look pages (named): this pull changes src/web/pages/backups.ts; a commit names /resilience
This push fixes the three gate failures from the check. The gate now passes locally, tsc --noEmit is clean, and the two test files I touched pass (14 tests). I didn't run the full suite, since the forge runs that on push.
- Real timers: I ended each flagged sleep with a
// real-timer: <why>comment. These aretest/snapshot-slow-line.test.tslines 63 and 122, andtest/snapshot-one-core.test.tslines 142, 158 and 160. The sleeps stay as they are. The tests simulate a slow line and poll a real background run, so I didn't inject a clock. - Both doors: the commit message now says "No door", with the reason. The slow line shows up on the existing
/resilienceterminal screen, which this pull already changes, so no new screen is needed. - Pipeline first: the commit message now carries a "Direction:" line saying that a backup that can't finish on a slow line loses data, so this goes ahead of pipeline items.
The remaining gate line, "look pages (named)", is informational: the commit message names /resilience.
The pull itself changes snapshot behaviour on slow lines: the deadline and puts in flight follow the measured rate, and earlier snapshots count beside a new sealed lane. The nightly backup never doubles a run.
<!-- look-pages effd5c8e04e4 --> Look pages for effd5c8, from the gate:
look pages (named): this pull changes src/web/pages/backups.ts; a commit names /resilience
No code change is needed for this feedback, and I pushed nothing.
The gate's "look pages (named)" line is a pass. It says this pull changes src/web/pages/backups.ts and that a commit names /resilience, which the "No door" commit message from my last push does. It asks for no fix, and the check's only failures were the three I fixed last time.
Request changes. The slow-line deadline and fewer-puts-at-once work is sound and tested. Security, escaping and the terminal door (Resilience screen) look fine. Two things block it, and I list the rest so they go in one round.
- Direction and scope (fix first). The repository says pipeline first, and small single-purpose pulls. This pull bundles four changes: measured deadlines and concurrency, progress and ETA wording, a new 'before sealing' lane view (verdict, page, TUI,
Store.before, and a change toresolveSnapshots), and a nightly-timer guard. Split it. Keep the line meter, the deadline, puts-at-once and progress here. Move the before-sealing display and the nightly guard to their own pulls, and say in each why it should go ahead of the pipeline items. 'Backups fail on a slow line' is a good reason; it has to be argued in the pull, not only in a commit message.
- Design principles 1 and 8 on /settings/backups. The new 'None sealed yet' paragraph repeats what the verdict and the row meta already say ('before sealing', 'restorable'). The progress sentence packs objects, chunks, 'to go', Mbit/s and ETA into one line of jargon. Say it once, in plain words, as docs/WRITING.md asks. Drop the repeat and put the rate and ETA in a
dlrow rather than a long sentence (principle 7).
- Rows in the earlier-snapshots list use
nav: truewith no link target, so they look like links and go nowhere. Make them plain rows, or link them to something real.
- docs/SNAPSHOT.md: the edited sentence is now one very long line and reads as a run-on. Split it.
- Hygiene: the diff rewraps many unrelated comments in files this pull barely touches (sealed_store, resilience, service, bucket). Keep the rewrap only where the gate forces it, and commit it separately so the real change is readable.
Next after these: expose the two concurrency constants as settings only if someone needs them; and add a test that the TUI Resilience screen shows the earlier snapshots.
I split the pull as asked, and this one now keeps only the slow-line work. The slow-line tests and the related snapshot, backups-page and resilience tests pass (80), and tsc and design-lint are clean. I didn't run the whole suite: the diff reaches 2,294 test files, which is too many to name, so the forge's check runs it.
What's in this pull
- The line meter, the deadline that follows the measured rate, puts in flight that follow the line, and the progress wording.
- A failed run keeping the chunks it sent is covered by a test.
What moved out (point 1)
- The before-sealing view is gone from this branch. That is
before_sealing.ts,Store.before, the change toresolveSnapshots, the verdict, page and terminal changes, andrecentSnapshots. It was the same change as your point 3, so thenav: truerows go with it. - The nightly-timer guard and its test are gone too.
- I can't open pulls. Those two changes need their own pulls.
- Arguing "Backups fail on a slow line" for them is still to do; the argument for this pull is in its commit message, not yet in the description.
Backups page (point 2)
- The "None sealed yet" paragraph is gone along with the before-sealing view.
- The existing sentence ("A snapshot is being taken: N objects, X so far, S s in") stays as it was, because existing tests match it.
- The rest goes into a
dlrow list with these rows: Already there, Still to send, Line speed, Done in about. A row is left out when it isn't known yet. - The log and terminal get one short sentence: "X sent, Y left, done in about Z".
Docs and hygiene (points 4 and 5)
docs/SNAPSHOT.mdis back to main, so the run-on sentence is gone.- The comment rewraps are reverted in every file. The gate only notes long comments and says the nightly rewrap handles them on main, so I made no rewrap commit.
- I also took the earlier-snapshots part out of
docs/features/slow-line-backups.md.
I left both follow-ups you listed for later: the concurrency constants as settings, and a test that the terminal Resilience screen shows the earlier snapshots. The second one belongs with the before-sealing pull.
<!-- look-pages 48d43dd1d67c --> Look pages for 48d43dd, from the gate:
look pages (named): this pull changes src/web/pages/backups.ts; a commit names /resilience
No code changed and nothing was pushed. The gate comment is a note, not a failure.
- What the note says: the gate found
src/web/pages/backups.tsin the diff and saw that a commit namesLooks: /resilience, a page inapp.looks. It reports that aslook pages (named). The rule insrc/kernel/look_rule.tsreturns that line whenever a named page is inapp.looks, and it never fails the check. - Why the named page is a weak match: the Backups page is served at
/settings/backups, which is not inapp.looks. I did not trace what/resiliencepictures. My commits named it because the progress sentence also shows on the Resilience terminal screen (src/tui/screens/resilience.ts), which this pull changes. That page is the nearest one inapp.looks. - What I chose not to do: I did not add
/settings/backupstoapp.looksinforge.json. That adds a before/after picture the forge would capture on every pull, so it is a call for the reviewer, not something to fold into this pull.
I ran bun scripts/gate.ts. Its only output was that look-pages note and one about long comment lines in src/snapshot/ files. The gate says the nightly rewrap handles those on main, and I left them alone because the earlier commit had deliberately backed out comment rewraps.
Earlier asks: the split is met (the before-sealing view and the nightly-timer guard are gone from this branch). The Backups page uses a dl of status-rows, which principle 7 asks for. The README is untouched and the how-to is in docs/features/slow-line-backups.md. The comment rewraps are reverted, and the gate only notes the long comments. The tests and the gate are green.
The slow-line work is sound. The deadline follows the measured rate and a stalled put still times out. Puts at once follow the line. A failed run keeps its chunks. The progress words reach the log, the page and the terminal, and the terminal door is covered through the Resilience screen.
Two things to do next, neither blocking:
- Non-blocking: add a test that renders the Backups page while a run is in flight and asserts the #snapshot-progress rows. Add one for the Resilience line too. Today only progressFacts and progressWords are tested, so the page and terminal wiring could be removed without a red test.
- Non-blocking: backups.ts calls progressFacts(p) twice. Bind it once. Also open the follow-up issue for the before-sealing view and the nightly guard, since Mason cannot open pulls.
A put's deadline and the number of puts at once now follow the rate the line is giving, and the Backups page no longer says "No backup" beside a new sealed lane. I did not run the "Done when" test: a 2 GB forge behind a 1 Mbit/s uplink on the stage rig. The new tests use fake slow stores and a metered directory store, and the
bun x tsc --noEmitcheck and the backup and snapshot tests pass here. I did not run the full suite.scripts/fast.tsnamed 2,313 of 3,523 test files for this diff, so I ran the snapshot, backup, restore, Resilience and settings tests instead.Cause of the failures. The 66 s and 63 s in the log are the fixed floor rule, 60 s of slack plus 4 MiB at 5 Mbit/s. It is not the whole-run budget shrinking.
withRetriesonly cuts deadlines to a run budget throughbounded, which only the restore drill uses.What changed
src/snapshot/line.tsmeasures that rate as bytes landed over the time any put was in flight, andstore.tsand the sealed store use it. A put that moves nothing still times out at that deadline. I could not time out on "no bytes moving" itself, becauseBun.S3Client.writereports no progress.takeSnapshotallows one per megabit of measured rate, between 1 and 8, and 2 until the line has been measured. On 1.2 to 1.75 Mbit/s that is one.in progresslog line, the Backups page, the Resilience screen and the live events now say chunks sent, "N of M chunks already there", bytes to go, the rate and when it will be done. Bytes to go is an estimate from the last snapshot's size. A first sealed run takes it from the unsealed lane's newest snapshot, and it is blank when none is known.Store.before,src/snapshot/before_sealing.ts). Until the sealed lane holds a snapshot of its own, the verdict counts the earlier ones, so the line reads "Backup 13 h old …" and not "No backup in the bucket". If the earlier ones are over 26 h old, the verdict is still red.- The lane says "14 from before sealing, newest 13 h ago, restorable". - The Backups page and the Resilience screen list them. - They restore with sealing off.
activecheck, so it queued a second run behind one still sending. It now returns when a run is in flight, and the run that ends arms the next. The new test fails with the check removed.I also added
docs/features/slow-line-backups.mdand a line indocs/SNAPSHOT.md.Not done
takeSnapshot's own 3-try wrapper around the store's 5 attempts is unchanged.To try it. Open
/settings/backupswhile a snapshot runs. WithFORGE_SNAPSHOT_SEAL=1on a bucket that holds unsealed snapshots and no sealed one, the page lists the earlier ones. The tests are intest/snapshot-slow-line.test.tsandtest/snapshot-one-core.test.ts.Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
🤖 Generated with Claude Code
Closes #13743