Public: anyone can read this; the crew can change it.

The restore drill's cost ceiling is a setting with its reason beside it, and a drill past it stops and says so #15177

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The restore drill's memory ceiling is now a limit an owner can set: drill.memorymb, labelled "Memory a restore drill may hold" on /admin/limits, with its reason shown beside it. Before this, the ceiling was a constant worked out once at boot.

Notes

shipped production is serving it deploy 7a0490b

opened this

The restore drill's memory ceiling is now a limit an owner can set: drill.memory_mb, labelled "Memory a restore drill may hold" on /admin/limits, with its reason shown beside it. Before this, the ceiling was a constant worked out once at boot. Stopping a drill past the ceiling already worked: it ended red with the cause given as memory: the drill process held N MiB, over its M MiB limit. What was missing was that nobody could change the number, so the new test proves the stop through the setting.

What moved:

  • src/settings/limits.ts: the new limit, in MiB, allowed from 512 to 1,048,576. Its default comes from drillMemoryDefaultMiB, which keeps the old rule: 60% of the machine's memory, kept between 3 and 16 GiB, or FORGE_DRILL_MEMORY_MIB. The rule moved here from drill_job.ts.
  • src/snapshot/drill_job.ts: the boot-time DRILL_MEMORY_BYTES constant is gone. drillMemoryLimit now just returns the setting's default in bytes, and is only used when a caller passes no limit.
  • src/snapshot/service.ts: the setting's value is read at each drill and handed to the drill's own process, so a change applies to the next drill without a restart. There are two new test hooks, drillScript and drillWatchMs.
  • src/snapshot/drill_cost.ts: the Backups page's Limit row now names the setting and /admin/limits before giving the default rule.
  • docs/features/backup-truth.md: says the same.

One test was wrong and I changed it: test/backups-drill-cost.test.ts checked the old Limit wording word for word. Only its expected string changed.

The new test, test/drill-cost-setting.test.ts, sets the limit to 512 MiB and runs a drill through the snapshot service with a child process that fills 700 MiB. The drill ends red at the restore step, its cause says over its 512 MiB limit, and its recorded cost keeps the 512 MiB ceiling. It fails without the change.

Not done: the time ceiling (DRILL_MAX_MS) is still a constant, because the issue only covers memory. The weekly drill's time budget is unchanged.

Checks: bun scripts/gate.ts --fix and bun x tsc --noEmit are clean. These test files pass: drill-cost-setting, drill-job, backups-drill-cost, limits, mcp-limits, tui-limits, admin-limits-etag, admin-limits-timing, boundary-tenancy-limits, drill-inert, mcp-restore-drill, tui-drill, snapshot-drill-page, drill-weekly, health-broken, forges-memory and helpers. scripts/fast.ts named 3,201 files, about 4.5 hours of tests, so I ran only those that cover the change. I didn't run bun scripts/design-lint.ts because nothing under src/web/ changed.

To try it: open /admin/limits and set "Memory a restore drill may hold". Then start a drill from the Backups page. Its Limit row shows the new value, and a drill past it ends red with memory named as the cause.

🤖 Generated with Claude Code

Closes #15168

commented

The restore drill's memory ceiling is now a limit an owner can set: drill.memory_mb, labelled "Memory a restore drill may hold" on /admin/limits, with its reason shown beside it. Before this, the ceiling was a constant worked out once at boot. Stopping a drill past the ceiling already worked: it ended red with the cause given as memory: the drill process held N MiB, over its M MiB limit. What was missing was that nobody could change the number, so the new test proves the stop through the setting.

What moved:

  • src/settings/limits.ts: the new limit, in MiB, allowed from 512 to 1,048,576. Its default comes from drillMemoryDefaultMiB, which keeps the old rule: 60% of the machine's memory, kept between 3 and 16 GiB, or FORGE_DRILL_MEMORY_MIB. The rule moved here from drill_job.ts.
  • src/snapshot/drill_job.ts: the boot-time DRILL_MEMORY_BYTES constant is gone. drillMemoryLimit now just returns the setting's default in bytes, and is only used when a caller passes no limit.
  • src/snapshot/service.ts: the setting's value is read at each drill and handed to the drill's own process, so a change applies to the next drill without a restart. There are two new test hooks, drillScript and drillWatchMs.
  • src/snapshot/drill_cost.ts: the Backups page's Limit row now names the setting and /admin/limits before giving the default rule.
  • docs/features/backup-truth.md: says the same.

One test was wrong and I changed it: test/backups-drill-cost.test.ts checked the old Limit wording word for word. Only its expected string changed.

The new test, test/drill-cost-setting.test.ts, sets the limit to 512 MiB and runs a drill through the snapshot service with a child process that fills 700 MiB. The drill ends red at the restore step, its cause says over its 512 MiB limit, and its recorded cost keeps the 512 MiB ceiling. It fails without the change.

Not done: the time ceiling (DRILL_MAX_MS) is still a constant, because the issue only covers memory. The weekly drill's time budget is unchanged.

Checks: bun scripts/gate.ts --fix and bun x tsc --noEmit are clean. These test files pass: drill-cost-setting, drill-job, backups-drill-cost, limits, mcp-limits, tui-limits, admin-limits-etag, admin-limits-timing, boundary-tenancy-limits, drill-inert, mcp-restore-drill, tui-drill, snapshot-drill-page, drill-weekly, health-broken, forges-memory and helpers. scripts/fast.ts named 3,201 files, about 4.5 hours of tests, so I ran only those that cover the change. I didn't run bun scripts/design-lint.ts because nothing under src/web/ changed.

To try it: open /admin/limits and set "Memory a restore drill may hold". Then start a drill from the Backups page. Its Limit row shows the new value, and a drill past it ends red with memory named as the cause.

🤖 Generated with Claude Code

Approved

Clean, small change: the drill ceiling is now the drill.memory_mb limit with its reason, read at each drill in service.ts and passed to the drill process; the old boot-time constant is gone, the existing memory kill is proven through the setting, and the one edited test only had its expected wording changed. Nothing here adds a write, input or output surface beyond the existing validated /admin/limits path. Note: the new test fills 700 MiB in a child, so watch it for flakiness on small runners.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs