Public: anyone can read this; the crew can change it.

The restore drill restores one hosted forge from its backup file and proves its database and repositories open #14181

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The restore drill restores one hosted forge alone from the newest snapshot. It proves the forge opens: its ledger names it, its database passes SQLite's integrity check, and each repository passes git fsck.

Notes

shipped production is serving it deploy 4adf612

opened this

The restore drill restores one hosted forge alone from the newest snapshot. It proves the forge opens: its ledger names it, its database passes SQLite's integrity check, and each repository passes git fsck. The last drill then names that forge in three places: the Backups card at /settings/backups, the "Last drill:" line on the terminal's Resilience screen, and the recovery comment. All three use one sentence:

"Hosted forge NAME (1 of N, SIZE) restored and opened: its database passed the integrity check and its K repositories git fsck."

Merge with main. Main landed the same restore step separately in #13923 (src/snapshot/drill_hosted.ts, step forge, HostedProof). The merge keeps main's version:

  • drill.ts, drills.ts, restore.ts, docs/RECOVERY.md and the Worker database-opens list are main's.
  • This branch's own hosted_drill.ts and its test are removed, along with the repoDirs field it had added to restore.ts.

What this branch still adds:

  • Shared sentence: hostedProofWords in drill_hosted.ts builds the sentence above from main's HostedProof, and all three places use it.
  • Fix to main: verdict.ts now accepts the forge step when reading drill/last.json. Main's list of steps left it out, so a drill that failed restoring a hosted forge was dropped when the red-drill card was read back.
  • Docs: docs/features/hosted-backup.md gets one line under "Where it shows" instead of a second section describing the drill.

What the sentence can't say: main's HostedProof has no duration and no "skipped" state. So the sentence has no seconds, and a drill whose snapshot held no hosted forge shows nothing. A forge that fails is named in the drill's own reason at step forge, as main does.

Checks: bun x tsc --noEmit and the design lint pass. These tests pass: test/drill-hosted-words.test.ts (new: the sentence, the Resilience line, the comment, and reading back a drill red at forge), plus snapshot-hosted-chunks, worker-database-opens, tui-resilience, drill-verdict, snapshot-drill, snapshot-drill-page, drill-card-words, recovery-red-drill, tui-resilience-drill, backups-drill-cost, statement and mcp-backups. bun scripts/gate.ts --fix rewrapped comments in src/spawn/host.ts and test/statement.test.ts, and those rewraps are committed.

To try it: open /settings/backups on the preview and press Run a drill. When it ends green, the last-drill line names the hosted forge it restored. In the terminal, open the Resilience screen; the "Last drill:" line under the verdict shows the same sentence.

🤖 Generated with Claude Code

Continues #14181

6 earlier comments
commented

The restore drill restores one hosted forge alone from the newest snapshot. It proves the forge opens: its ledger names it, its database passes SQLite's integrity check, and each repository passes git fsck. The last drill then names that forge in three places: the Backups card at /settings/backups, the "Last drill:" line on the terminal's Resilience screen, and the recovery comment. All three use one sentence:

"Hosted forge NAME (1 of N, SIZE) restored and opened: its database passed the integrity check and its K repositories git fsck."

Merge with main. Main landed the same restore step separately in #13923 (src/snapshot/drill_hosted.ts, step forge, HostedProof). The merge keeps main's version:

  • drill.ts, drills.ts, restore.ts, docs/RECOVERY.md and the Worker database-opens list are main's.
  • This branch's own hosted_drill.ts and its test are removed, along with the repoDirs field it had added to restore.ts.

What this branch still adds:

  • Shared sentence: hostedProofWords in drill_hosted.ts builds the sentence above from main's HostedProof, and all three places use it.
  • Fix to main: verdict.ts now accepts the forge step when reading drill/last.json. Main's list of steps left it out, so a drill that failed restoring a hosted forge was dropped when the red-drill card was read back.
  • Docs: docs/features/hosted-backup.md gets one line under "Where it shows" instead of a second section describing the drill.

What the sentence can't say: main's HostedProof has no duration and no "skipped" state. So the sentence has no seconds, and a drill whose snapshot held no hosted forge shows nothing. A forge that fails is named in the drill's own reason at step forge, as main does.

Checks: bun x tsc --noEmit and the design lint pass. These tests pass: test/drill-hosted-words.test.ts (new: the sentence, the Resilience line, the comment, and reading back a drill red at forge), plus snapshot-hosted-chunks, worker-database-opens, tui-resilience, drill-verdict, snapshot-drill, snapshot-drill-page, drill-card-words, recovery-red-drill, tui-resilience-drill, backups-drill-cost, statement and mcp-backups. bun scripts/gate.ts --fix rewrapped comments in src/spawn/host.ts and test/statement.test.ts, and those rewraps are committed.

To try it: open /settings/backups on the preview and press Run a drill. When it ends green, the last-drill line names the hosted forge it restored. In the terminal, open the Resilience screen; the "Last drill:" line under the verdict shows the same sentence.

🤖 Generated with Claude Code

commented

While this pull was open, main landed the hosted-forge restore itself in 936cb90d1 (#13923, src/snapshot/drill_hosted.ts). That commit restores one forge alone, runs the integrity check, runs git fsck, and fails the drill at the step forge naming the forge. This pull now merges main and drops its own parallel hosted_drill.ts. It adds what the issue asks for and main doesn't have: the restore time in seconds on the card, a one-line skip message, and paging when the hosted step fails.

What changed

  • Restore time: runDrill (src/snapshot/drill.ts) times the hosted step and keeps it as hosted.ms.
  • Skip: when every earlier step passed but the snapshot held no whole hosted forge, the drill sets hostedSkipped: true and stays green.
  • Saved with the drill: both fields are stored in the drill row's JSON (no migration) and read back defensively by src/snapshot/drills.ts.
  • One line, three places: the new src/snapshot/drill_hosted_words.ts writes the line once, and the Backups card (data-drill-hosted), the Resilience screen's "Last drill:" line and the recovery comment's - hosted: all use it. It reads "Hosted forge glint (3 MiB, 1 of 4) restored alone in 3.2 s: its database passed the integrity check and its 2 repositories git fsck.", or says the hosted restore was skipped. Control characters in a forge name print as ?.
  • Paging fix (a bug on main): the step list in src/snapshot/verdict.ts didn't include forge. A drill that failed at the hosted step was read back from drill/last.json as "no drill", so it never paged. forge is now on the list, so that failure pages once like any other red drill.

Tests

  • New test/drill-hosted-card.test.ts covers the words, the stored fields surviving a reload, the card and terminal lines, the skip line, and a drill red at forge paging once with the forge named.
  • test/snapshot-drill.test.ts now checks that a forge with no hosted forges records the skip.
  • One test was out of date: main's test/snapshot-hosted-chunks.test.ts compared the proof with an exact toEqual, which fails now that it carries ms. It now checks the four fields as before, checks ms separately, and checks the card line.
  • tsc, the gate and design-lint are clean, and 113 drill, backup, resilience, verdict and restore test files pass. scripts/fast.ts named 2384 files (about 202 minutes), so I picked those by hand.

What I chose not to do

  • Which forge is drilled: I kept main's drillPick, which takes the next forge by day and week, rather than the issue's "oldest drilled date". Both reach every forge in turn, and changing it would mean keeping another record on disk.
  • Failure wording: a hosted failure is reported as step forge with the cause in the reason (for example "missing chunk … of forge.db" or "does not pass git fsck"), not as separate sub-steps.

Process notes

  • Another run of mine had pushed its own merge of main to the fork (04c28d73f), with no restore time and no skip line. This branch merges it with its own content kept and drops its duplicate words test.
  • One push (4eb86ac04) went up before the gate passed. It flagged two comment rewraps on main files and three as never casts in my test; 1456ee5a6 fixes both.

To try it: on the preview, open /settings/backups and press "Run a drill". When it ends, the last-drill line names the hosted forge with its size and seconds, or says the hosted restore was skipped. The terminal's Resilience screen shows the same line under the verdict.

🤖 Generated with Claude Code

commented

This pull shows the hosted restore on the drill card. The card names the hosted forge it restored, with the restore's size and time in seconds. If the hosted restore was skipped, the card says so. A drill that fails at its forge step now pages.

  • Card: the drill's hosted restore comes from main's drill_hosted.ts (#13923). It replaces this branch's earlier hosted_drill.ts.
  • Resilience screen: it shows the same detail as the card. The recovery comment also names the forge that was restored.
  • Diff: comments from main are left as main wrote them. The hosted card test runs against the real app, not a cast stand-in.

Nothing was pushed this round. The head is still 1456ee5a6 and matches origin. The forge cut the last check run or is running it again, and none of this head's checks is recorded as failed, so there was nothing to fix.

To try it, open the preview at https://lab--14181--forge-trevato-apps.open-platform.sh, run a restore drill, and look at the drill card and the Resilience screen.

🤖 Generated with Claude Code

commented

I made no changes this run. Nothing new was said on the issue, the two commits on main don't touch these files, the fork's branch is still at 1456ee5a6, and the pull is mergeable with no conflicts. The forge's check on that head was still running at the time of writing.

Description

While this pull was open, main landed the hosted-forge restore itself in 936cb90d1 (#13923, src/snapshot/drill_hosted.ts). That commit restores one forge alone, runs the integrity check, runs git fsck, and fails the drill at the step forge naming the forge. This pull now merges main and drops its own parallel hosted_drill.ts. It adds what the issue asks for and main doesn't have: the restore time in seconds on the card, a one-line skip message, and paging when the hosted step fails.

What changed

  • Restore time: runDrill (src/snapshot/drill.ts) times the hosted step and keeps it as hosted.ms.
  • Skip: when every earlier step passed but the snapshot held no whole hosted forge, the drill sets hostedSkipped: true and stays green.
  • Saved with the drill: both fields are stored in the drill row's JSON (no migration) and read back defensively by src/snapshot/drills.ts.
  • One line, three places: the new src/snapshot/drill_hosted_words.ts writes the line once, and the Backups card (data-drill-hosted), the Resilience screen's "Last drill:" line and the recovery comment's - hosted: all use it. It reads "Hosted forge glint (3 MiB, 1 of 4) restored alone in 3.2 s: its database passed the integrity check and its 2 repositories git fsck.", or says the hosted restore was skipped. Control characters in a forge name print as ?.
  • Paging fix (a bug on main): the step list in src/snapshot/verdict.ts didn't include forge. A drill that failed at the hosted step was read back from drill/last.json as "no drill", so it never paged. forge is now on the list, so that failure pages once like any other red drill.

Tests

  • New test/drill-hosted-card.test.ts covers the words, the stored fields surviving a reload, the card and terminal lines, the skip line, and a drill red at forge paging once with the forge named.
  • test/snapshot-drill.test.ts now checks that a forge with no hosted forges records the skip.
  • One test was out of date: main's test/snapshot-hosted-chunks.test.ts compared the proof with an exact toEqual, which fails now that it carries ms. It now checks the four fields as before, checks ms separately, and checks the card line.
  • tsc, the gate and design-lint are clean, and 113 drill, backup, resilience, verdict and restore test files pass. scripts/fast.ts named 2384 files (about 202 minutes), so I picked those by hand.

What I chose not to do

  • Which forge is drilled: I kept main's drillPick, which takes the next forge by day and week, rather than the issue's "oldest drilled date". Both reach every forge in turn, and changing it would mean keeping another record on disk.
  • Failure wording: a hosted failure is reported as step forge with the cause in the reason (for example "missing chunk … of forge.db" or "does not pass git fsck"), not as separate sub-steps.

Process notes

  • Another run of mine had pushed its own merge of main to the fork (04c28d73f), with no restore time and no skip line. This branch merges it with its own content kept and drops its duplicate words test.
  • One push (4eb86ac04) went up before the gate passed. It flagged two comment rewraps on main files and three as never casts in my test; 1456ee5a6 fixes both.

To try it: on the preview, open /settings/backups and press "Run a drill". When it ends, the last-drill line names the hosted forge with its size and seconds, or says the hosted restore was skipped. The terminal's Resilience screen shows the same line under the verdict.

🤖 Generated with Claude Code

commented

I made no changes this run, so there was nothing new to push. The fork's branch is still at 1456ee5a6, and the pull is mergeable with no conflicts. Its check on that head was still queued when I looked, so I have no result for it.

Nothing that moved touches this pull. Main gained two commits, neither touching these files. The one new related issue is #14182, a full-suite failure on main (ssh-port-taken, apps-idle, and snapshot-hosted-inside running slower than its 2 s limit); none of those are files this pull changes.

Description

While this pull was open, main landed the hosted-forge restore itself in 936cb90d1 (#13923, src/snapshot/drill_hosted.ts). That commit restores one forge alone, runs the integrity check, runs git fsck, and fails the drill at the step forge naming the forge. This pull now merges main and drops its own parallel hosted_drill.ts. It adds what the issue asks for and main doesn't have: the restore time in seconds on the card, a one-line skip message, and paging when the hosted step fails.

What changed

  • Restore time: runDrill (src/snapshot/drill.ts) times the hosted step and keeps it as hosted.ms.
  • Skip: when every earlier step passed but the snapshot held no whole hosted forge, the drill sets hostedSkipped: true and stays green.
  • Saved with the drill: both fields are stored in the drill row's JSON (no migration) and read back defensively by src/snapshot/drills.ts.
  • One line, three places: the new src/snapshot/drill_hosted_words.ts writes the line once, and the Backups card (data-drill-hosted), the Resilience screen's "Last drill:" line and the recovery comment's - hosted: all use it. It reads "Hosted forge glint (3 MiB, 1 of 4) restored alone in 3.2 s: its database passed the integrity check and its 2 repositories git fsck.", or says the hosted restore was skipped. Control characters in a forge name print as ?.
  • Paging fix (a bug on main): the step list in src/snapshot/verdict.ts didn't include forge. A drill that failed at the hosted step was read back from drill/last.json as "no drill", so it never paged. forge is now on the list, so that failure pages once like any other red drill.

Tests

  • New test/drill-hosted-card.test.ts covers the words, the stored fields surviving a reload, the card and terminal lines, the skip line, and a drill red at forge paging once with the forge named.
  • test/snapshot-drill.test.ts now checks that a forge with no hosted forges records the skip.
  • One test was out of date: main's test/snapshot-hosted-chunks.test.ts compared the proof with an exact toEqual, which fails now that it carries ms. It now checks the four fields as before, checks ms separately, and checks the card line.
  • tsc, the gate and design-lint are clean, and 113 drill, backup, resilience, verdict and restore test files pass. scripts/fast.ts named 2384 files (about 202 minutes), so I picked those by hand.

What I chose not to do

  • Which forge is drilled: I kept main's drillPick, which takes the next forge by day and week, rather than the issue's "oldest drilled date". Both reach every forge in turn, and changing it would mean keeping another record on disk.
  • Failure wording: a hosted failure is reported as step forge with the cause in the reason (for example "missing chunk … of forge.db" or "does not pass git fsck"), not as separate sub-steps.

Process notes

  • Another run of mine had pushed its own merge of main to the fork (04c28d73f), with no restore time and no skip line. This branch merges it with its own content kept and drops its duplicate words test.
  • One push (4eb86ac04) went up before the gate passed. It flagged two comment rewraps on main files and three as never casts in my test; 1456ee5a6 fixes both.

To try it: on the preview, open /settings/backups and press "Run a drill". When it ends, the last-drill line names the hosted forge with its size and seconds, or says the hosted restore was skipped. The terminal's Resilience screen shows the same line under the verdict.

🤖 Generated with Claude Code

commented

I made no changes this run. That also fits the hold on trevato/forge ("Releasing trevato/forge: production is rolling, hold changes to it until it settles."): nothing new was pushed while production rolls. The hold doesn't touch this change anyway, since the pull reaches production only after a person merges it. The fork's branch is still at 1456ee5a6, the pull is mergeable with no conflicts, and main hasn't changed these files. Its check on that head was still queued when I looked, so I have no result for it.

Description

While this pull was open, main landed the hosted-forge restore itself in 936cb90d1 (#13923, src/snapshot/drill_hosted.ts). That commit restores one forge alone, runs the integrity check, runs git fsck, and fails the drill at the step forge naming the forge. This pull now merges main and drops its own parallel hosted_drill.ts. It adds what the issue asks for and main doesn't have: the restore time in seconds on the card, a one-line skip message, and paging when the hosted step fails.

What changed

  • Restore time: runDrill (src/snapshot/drill.ts) times the hosted step and keeps it as hosted.ms.
  • Skip: when every earlier step passed but the snapshot held no whole hosted forge, the drill sets hostedSkipped: true and stays green.
  • Saved with the drill: both fields are stored in the drill row's JSON (no migration) and read back defensively by src/snapshot/drills.ts.
  • One line, three places: the new src/snapshot/drill_hosted_words.ts writes the line once, and the Backups card (data-drill-hosted), the Resilience screen's "Last drill:" line and the recovery comment's - hosted: all use it. It reads "Hosted forge glint (3 MiB, 1 of 4) restored alone in 3.2 s: its database passed the integrity check and its 2 repositories git fsck.", or says the hosted restore was skipped. Control characters in a forge name print as ?.
  • Paging fix (a bug on main): the step list in src/snapshot/verdict.ts didn't include forge. A drill that failed at the hosted step was read back from drill/last.json as "no drill", so it never paged. forge is now on the list, so that failure pages once like any other red drill.

Tests

  • New test/drill-hosted-card.test.ts covers the words, the stored fields surviving a reload, the card and terminal lines, the skip line, and a drill red at forge paging once with the forge named.
  • test/snapshot-drill.test.ts now checks that a forge with no hosted forges records the skip.
  • One test was out of date: main's test/snapshot-hosted-chunks.test.ts compared the proof with an exact toEqual, which fails now that it carries ms. It now checks the four fields as before, checks ms separately, and checks the card line.
  • tsc, the gate and design-lint are clean, and 113 drill, backup, resilience, verdict and restore test files pass. scripts/fast.ts named 2384 files (about 202 minutes), so I picked those by hand.

What I chose not to do

  • Which forge is drilled: I kept main's drillPick, which takes the next forge by day and week, rather than the issue's "oldest drilled date". Both reach every forge in turn, and changing it would mean keeping another record on disk.
  • Failure wording: a hosted failure is reported as step forge with the cause in the reason (for example "missing chunk … of forge.db" or "does not pass git fsck"), not as separate sub-steps.

Process notes

  • Another run of mine had pushed its own merge of main to the fork (04c28d73f), with no restore time and no skip line. This branch merges it with its own content kept and drops its duplicate words test.
  • One push (4eb86ac04) went up before the gate passed. It flagged two comment rewraps on main files and three as never casts in my test; 1456ee5a6 fixes both.
  • The desk's hold "Releasing trevato/forge: production is rolling" doesn't touch this change. Nothing was pushed while it stood, and the pull reaches production only after a person merges it.

To try it: on the preview, open /settings/backups and press "Run a drill". When it ends, the last-drill line names the hosted forge with its size and seconds, or says the hosted restore was skipped. The terminal's Resilience screen shows the same line under the verdict.

🤖 Generated with Claude Code

commented

I made no changes this run; nothing that moved touches this pull. The fork's branch is still at 1456ee5a6, the pull is mergeable with no conflicts, and main hasn't changed any of the files it touches. The check on that head was still queued when I looked, so I have no result for it.

Description

While this pull was open, main landed the hosted-forge restore itself in 936cb90d1 (#13923, src/snapshot/drill_hosted.ts). That commit restores one forge alone, runs the integrity check, runs git fsck, and fails the drill at the step forge naming the forge. This pull now merges main and drops its own parallel hosted_drill.ts. It adds what the issue asks for and main doesn't have: the restore time in seconds on the card, a one-line skip message, and paging when the hosted step fails.

What changed

  • Restore time: runDrill (src/snapshot/drill.ts) times the hosted step and keeps it as hosted.ms.
  • Skip: when every earlier step passed but the snapshot held no whole hosted forge, the drill sets hostedSkipped: true and stays green.
  • Saved with the drill: both fields are stored in the drill row's JSON (no migration) and read back defensively by src/snapshot/drills.ts.
  • One line, three places: the new src/snapshot/drill_hosted_words.ts writes the line once, and the Backups card (data-drill-hosted), the Resilience screen's "Last drill:" line and the recovery comment's - hosted: all use it. It reads "Hosted forge glint (3 MiB, 1 of 4) restored alone in 3.2 s: its database passed the integrity check and its 2 repositories git fsck.", or says the hosted restore was skipped. Control characters in a forge name print as ?.
  • Paging fix (a bug on main): the step list in src/snapshot/verdict.ts didn't include forge. A drill that failed at the hosted step was read back from drill/last.json as "no drill", so it never paged. forge is now on the list, so that failure pages once like any other red drill.

Tests

  • New test/drill-hosted-card.test.ts covers the words, the stored fields surviving a reload, the card and terminal lines, the skip line, and a drill red at forge paging once with the forge named.
  • test/snapshot-drill.test.ts now checks that a forge with no hosted forges records the skip.
  • One test was out of date: main's test/snapshot-hosted-chunks.test.ts compared the proof with an exact toEqual, which fails now that it carries ms. It now checks the four fields as before, checks ms separately, and checks the card line.
  • tsc, the gate and design-lint are clean, and 113 drill, backup, resilience, verdict and restore test files pass. scripts/fast.ts named 2384 files (about 202 minutes), so I picked those by hand.

What I chose not to do

  • Which forge is drilled: I kept main's drillPick, which takes the next forge by day and week, rather than the issue's "oldest drilled date". Both reach every forge in turn, and changing it would mean keeping another record on disk.
  • Failure wording: a hosted failure is reported as step forge with the cause in the reason (for example "missing chunk … of forge.db" or "does not pass git fsck"), not as separate sub-steps.

Process notes

  • Another run of mine had pushed its own merge of main to the fork (04c28d73f), with no restore time and no skip line. This branch merges it with its own content kept and drops its duplicate words test.
  • One push (4eb86ac04) went up before the gate passed. It flagged two comment rewraps on main files and three as never casts in my test; 1456ee5a6 fixes both.
  • The desk's hold "Releasing trevato/forge: production is rolling" doesn't touch this change. Nothing was pushed while it stood, and the pull reaches production only after a person merges it.

To try it: on the preview, open /settings/backups and press "Run a drill". When it ends, the last-drill line names the hosted forge with its size and seconds, or says the hosted restore was skipped. The terminal's Resilience screen shows the same line under the verdict.

🤖 Generated with Claude Code

Approved

Approved. It adds the restore time, the one-line skip and the forge step in the verdict reader (so a red hosted drill now pages). One sentence is shared by the card, the Resilience screen and the recovery comment. The words are escaped on the page and control characters are neutralised for the terminal. Tests would fail without each behaviour, and the how-to lives in docs/features/hosted-backup.md, not the README. This is not pipeline work, but it is small, single-purpose, and fixes a real paging gap. I could not press Run a drill on the preview without a session, so the card is judged from source and the test that fetches it.

Two things I would do next:

  1. In src/web/pages/backups.ts the hosted sentence is appended inside the already long Last drill paragraph. Give it its own dl row (like drill-cost) so it reads as a fact, not as more prose (principles 1, 7).
  2. In src/snapshot/verdict.ts verdictComment calls hostedDrillWords twice; compute it once. Also the docs bullet says (SIZE, 1 of N) always, but the line omits 1 of N when only one forge is held; say so.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs