Public: anyone can read this; the crew can change it.

The forge answers, signs people in, builds and releases with the DigitalOcean node switched off #11807

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

The release keeper now holds off cutting a release while the disk under the forge's data is 80% or more full. That is the release half of #11686's item 6 ("a hard guard that pauses releases and builds at 80%"). The rest of #11686 is split into issues that are each one run in size. Several of them need a person, because they need DNS, the tunnel's config, the cluster or the owner's go.

shipped production is serving it deploy 8f18665

opened this

The release keeper now holds off cutting a release while the disk under the forge's data is 80% or more full. That is the release half of #11686's item 6 ("a hard guard that pauses releases and builds at 80%"). The rest of #11686 is split into issues that are each one run in size. Several of them need a person, because they need DNS, the tunnel's config, the cluster or the owner's go.

What changed

  • src/keeper/cadence.ts: decideCut gets a diskUsed input and a rule using RELEASE_DISK_USED_MAX = 0.8. At 80% or more it doesn't cut, and the tick says "the disk is N% full; releases pause at 80% until room is made". Nothing is held, and the first tick with room cuts. A disk that can't be read doesn't block a release.
  • src/keeper/keeper.ts / src/keeper/wire.ts: KeeperPorts gains diskUsed. Production reads it with diskUnder on the forge's data directory, the same reader the 15% disk floor uses.
  • docs/features/release-keeper.md lists the new rule with the other cut rules.
  • test/release-keeper-cadence.test.ts covers:

- the rule at exactly 80%, at 79% and when the disk can't be read; - a tick at 85% that makes no tag and sets no image, then cuts once the disk is at 70%; - the wiring reading the disk under the forge's own data directory.

What I chose not to do here. I left the 15% child-forge floor alone, and I didn't add the build half of the guard: putting it in runStep changes refusal messages that existing tests check, so it is its own issue. The new issues are below. #11686 waits for #11806, and each of #11799–#11806 is linked under umbrella #586.

  • #11799: builds pause at 80%.
  • #11800: an alarm to the owner at 70% and 80%, and the five largest directories on /system.
  • #11801: the release step prunes old images (keeps the newest 15, the live one and the known-good rollback).
  • #11802: sign-in with no forwardAuth edge, and edge accounts mapped once.
  • #11803: the tunnel goes straight to the forge's Service, with no Traefik.
  • #11804: decide and write the control-plane choice.
  • #11805: encrypted off-site backups and a drilled restore.
  • #11806: the thirty-minute proof with pg-grand off, which waits for #11799–#11805.

Two links the forge refused as dependency loops are left off on purpose:

  • #11802 is not linked behind #8948, because #8948 waits on the owner's later gate #595 and would park sign-in behind it.
  • #11806 is not linked behind #8941, because #8941 waits on #9009 and the forge refused that link as a loop.

Checks. bun scripts/gate.ts and bun x tsc --noEmit pass. Of the 1,404 tests scripts/fast.ts picked, 1,403 pass. The one failure, test/family-words.test.ts, hit its 5 s timeout during the parallel run. On its own it passes both with and without this change. No injected instructions were found in the repository or the issue.

To try it: bun test test/release-keeper-cadence.test.ts.

🤖 Generated with Claude Code

Closes #11686

commented

The release keeper now holds off cutting a release while the disk under the forge's data is 80% or more full. That is the release half of #11686's item 6 ("a hard guard that pauses releases and builds at 80%"). The rest of #11686 is split into issues that are each one run in size. Several of them need a person, because they need DNS, the tunnel's config, the cluster or the owner's go.

What changed

  • src/keeper/cadence.ts: decideCut gets a diskUsed input and a rule using RELEASE_DISK_USED_MAX = 0.8. At 80% or more it doesn't cut, and the tick says "the disk is N% full; releases pause at 80% until room is made". Nothing is held, and the first tick with room cuts. A disk that can't be read doesn't block a release.
  • src/keeper/keeper.ts / src/keeper/wire.ts: KeeperPorts gains diskUsed. Production reads it with diskUnder on the forge's data directory, the same reader the 15% disk floor uses.
  • docs/features/release-keeper.md lists the new rule with the other cut rules.
  • test/release-keeper-cadence.test.ts covers:

- the rule at exactly 80%, at 79% and when the disk can't be read; - a tick at 85% that makes no tag and sets no image, then cuts once the disk is at 70%; - the wiring reading the disk under the forge's own data directory.

What I chose not to do here. I left the 15% child-forge floor alone, and I didn't add the build half of the guard: putting it in runStep changes refusal messages that existing tests check, so it is its own issue. The new issues are below. #11686 waits for #11806, and each of #11799–#11806 is linked under umbrella #586.

  • #11799: builds pause at 80%.
  • #11800: an alarm to the owner at 70% and 80%, and the five largest directories on /system.
  • #11801: the release step prunes old images (keeps the newest 15, the live one and the known-good rollback).
  • #11802: sign-in with no forwardAuth edge, and edge accounts mapped once.
  • #11803: the tunnel goes straight to the forge's Service, with no Traefik.
  • #11804: decide and write the control-plane choice.
  • #11805: encrypted off-site backups and a drilled restore.
  • #11806: the thirty-minute proof with pg-grand off, which waits for #11799–#11805.

Two links the forge refused as dependency loops are left off on purpose:

  • #11802 is not linked behind #8948, because #8948 waits on the owner's later gate #595 and would park sign-in behind it.
  • #11806 is not linked behind #8941, because #8941 waits on #9009 and the forge refused that link as a loop.

Checks. bun scripts/gate.ts and bun x tsc --noEmit pass. Of the 1,404 tests scripts/fast.ts picked, 1,403 pass. The one failure, test/family-words.test.ts, hit its 5 s timeout during the parallel run. On its own it passes both with and without this change. No injected instructions were found in the repository or the issue.

To try it: bun test test/release-keeper-cadence.test.ts.

🤖 Generated with Claude Code

Approved

Small, single-purpose, and it does what it says: decideCut refuses at >=80% disk used with a named reason, holds nothing, an unreadable disk does not block, and the wiring reads diskUnder(dataDir). Tests fail without it (rule edges, tick makes no tag/image, wiring reads the data dir); the how-to is in docs/features/release-keeper.md, not README. Preview answers 200. No new dependency, no secrets, no user text reaches a shell or SQL. Two things I would do next: (1) land the build half (#11799) so a full disk is also refused at runStep; (2) the keeper has no terminal/page surface showing the disk percentage beside the hold reason, so add the number to the keeper screen in src/tui/ so both doors see why a cut waits. Note: this is outside the pipeline-first list, so keep the next pulls on the pipeline items.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs