Public: anyone can read this; the crew can change it.

The release proof boots a 1-CPU, 512 MiB forge, runs the first-hour journey and fails when memory at rest is over budget #12579

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The release proof now boots a second forge limited to 1 CPU / 512 MiB and walks its first hour.

Notes

shipped production is serving it deploy 28b6a90

opened this

The release proof now boots a second forge limited to 1 CPU / 512 MiB and walks its first hour. If that forge's memory at rest is over BUDGET.restBytes, the proof fails with the number in the message (for example "the 1 CPU / 512 MiB forge held 300.0 MiB at rest; the budget is 256.0 MiB"), and /system shows the reading. The gate, tsc --noEmit, boot-lazy, unused-exports, unreached, hollow, system-parts, test-budget, footprint, release-prove and small-forge tests all pass (122 tests). Nobody has run it yet under a real kernel limit (see the last paragraph).

The likeliest thing to go red: in the sandbox, a forge run from source held about 337 MiB at rest, and the earlier dev-machine measurements in docs/features/footprint.md say 265–271 MiB. Both are over 256 MiB. #12092 measured 170 MiB on the release image. So whether the first release proof passes depends on how production runs the forge; if it fails, the reading is real, not a bug in the proof.

  • src/release/small_forge.ts (new): walkSmallForge starts src/main.ts in a cgroup of its own capped at 512 MiB and one core, on a scratch directory under <data>/release/small. It then checks, in order:

- the forge answers /readyz and the home page; - a first admin can be made, a repository created and a commit pushed (its forge.json declares check.run: "true"); - that commit's check comes back ok.

It reads the process's memory 3 s after ready, before any request, and again after the walk, plus the cgroup's peak and any OOM kills. Afterwards it stops the forge and removes the cgroup and the directory.

  • When it runs: only if the pod has room for 512 MiB beyond the forge's own use (cellMemoryCeilingMiB, #12519) and the kernel can actually hold the cap. Otherwise it is skipped, the reading says why, and the skip does not fail the proof. A forge without the cap would say nothing about 512 MiB, and running one uncapped could kill the forge beside it.
  • What the second forge gets: only the environment listed in smallEnv: no token or secret from this forge, no SSH terminal, no apps or crew, loopback only. The push token reaches git through its environment, so it never shows in a process list.
  • src/release/prove.ts: the small forge runs last, after the smoke checks and the stranger's journey pass, and its reading is kept as small in proof.json. Over budget at rest fails the proof as an alarm. An OOM kill or a step that does not finish also fails it, but is retried sooner, like a slow answer.
  • src/system/footprint.ts: the footprint carries the reading, and /system gets a row "On 1 CPU, 512 MiB" with memory at rest, after the walk, and peak, or why it was not measured. Over budget at rest becomes the section's verdict.
  • Other files:

- src/main.ts passes walkSmallForge to the proof, the only change outside the issue's file list; without it nothing calls the new step. - scripts/gate.ts lists test/small-forge.test.ts in SLOW under #12532, because it really boots a forge (about 12 s). - docs/features/footprint.md has a new section on all of this.

I did not touch the release workflow or the chart. The walk does not stop early when the proof is aborted as hung; it is bounded by its own limits (60 s to boot, 120 s for the check), well inside the proof's 30-minute ceiling. I couldn't record a plan on the issue with keep_plan, because the tool wasn't available in this run.

To try it: bun test test/small-forge.test.ts test/release-prove.test.ts test/footprint.test.ts. The last test in small-forge does a real boot and walk, with a plain directory standing in for the cgroup, because the sandbox has no cgroup. The test of the kernel actually holding 512 MiB is the first release proof on a host that allows it. It shows on /system under "What the forge costs" → "On 1 CPU, 512 MiB".

🤖 Generated with Claude Code

Closes #12532

8 earlier comments · 2 events
commented

The release proof now boots a second forge limited to 1 CPU / 512 MiB and walks its first hour. If that forge's memory at rest is over BUDGET.restBytes, the proof fails with the number in the message (for example "the 1 CPU / 512 MiB forge held 300.0 MiB at rest; the budget is 256.0 MiB"), and /system shows the reading. The gate, tsc --noEmit, boot-lazy, unused-exports, unreached, hollow, system-parts, test-budget, footprint, release-prove and small-forge tests all pass (122 tests). Nobody has run it yet under a real kernel limit (see the last paragraph).

The likeliest thing to go red: in the sandbox, a forge run from source held about 337 MiB at rest, and the earlier dev-machine measurements in docs/features/footprint.md say 265–271 MiB. Both are over 256 MiB. #12092 measured 170 MiB on the release image. So whether the first release proof passes depends on how production runs the forge; if it fails, the reading is real, not a bug in the proof.

  • src/release/small_forge.ts (new): walkSmallForge starts src/main.ts in a cgroup of its own capped at 512 MiB and one core, on a scratch directory under <data>/release/small. It then checks, in order:

- the forge answers /readyz and the home page; - a first admin can be made, a repository created and a commit pushed (its forge.json declares check.run: "true"); - that commit's check comes back ok.

It reads the process's memory 3 s after ready, before any request, and again after the walk, plus the cgroup's peak and any OOM kills. Afterwards it stops the forge and removes the cgroup and the directory.

  • When it runs: only if the pod has room for 512 MiB beyond the forge's own use (cellMemoryCeilingMiB, #12519) and the kernel can actually hold the cap. Otherwise it is skipped, the reading says why, and the skip does not fail the proof. A forge without the cap would say nothing about 512 MiB, and running one uncapped could kill the forge beside it.
  • What the second forge gets: only the environment listed in smallEnv: no token or secret from this forge, no SSH terminal, no apps or crew, loopback only. The push token reaches git through its environment, so it never shows in a process list.
  • src/release/prove.ts: the small forge runs last, after the smoke checks and the stranger's journey pass, and its reading is kept as small in proof.json. Over budget at rest fails the proof as an alarm. An OOM kill or a step that does not finish also fails it, but is retried sooner, like a slow answer.
  • src/system/footprint.ts: the footprint carries the reading, and /system gets a row "On 1 CPU, 512 MiB" with memory at rest, after the walk, and peak, or why it was not measured. Over budget at rest becomes the section's verdict.
  • Other files:

- src/main.ts passes walkSmallForge to the proof, the only change outside the issue's file list; without it nothing calls the new step. - scripts/gate.ts lists test/small-forge.test.ts in SLOW under #12532, because it really boots a forge (about 12 s). - docs/features/footprint.md has a new section on all of this.

I did not touch the release workflow or the chart. The walk does not stop early when the proof is aborted as hung; it is bounded by its own limits (60 s to boot, 120 s for the check), well inside the proof's 30-minute ceiling. I couldn't record a plan on the issue with keep_plan, because the tool wasn't available in this run.

To try it: bun test test/small-forge.test.ts test/release-prove.test.ts test/footprint.test.ts. The last test in small-forge does a real boot and walk, with a plain directory standing in for the cgroup, because the sandbox has no cgroup. The test of the kernel actually holding 512 MiB is the first release proof on a host that allows it. It shows on /system under "What the forge costs" → "On 1 CPU, 512 MiB".

🤖 Generated with Claude Code

commented

This pull makes the release proof boot a second forge limited to 1 CPU and 512 MiB, take it through its first hour (home page, first admin, a repository, a push and a check) and record its memory at rest and after that hour. The proof fails when the memory at rest is over budget (#12532). The reading is kept on the release record as small (in src/release/prove.ts), and smallWords in src/system/footprint.ts turns it into a sentence. The boot and the walk are in src/release/small_forge.ts.

The latest commit only rewraps five comment lines that went past the gate's 100-column limit:

  • src/release/prove.ts
  • src/release/small_forge.ts (two lines)
  • src/system/footprint.ts
  • test/small-forge.test.ts

No code changed.

Checks run here: bun scripts/gate.ts and bun x tsc --noEmit both pass. bun scripts/fast.ts said the change was too broad to name a short list: it picked 2,059 test files, about 175 minutes, well over its 5-minute budget. So I ran the suites that cover these modules myself, and they pass: test/small-forge.test.ts (8 tests), and test/footprint.test.ts plus the release-prove suite (53 tests).

🤖 Generated with Claude Code

commented

The release proof now boots a forge held to 1 CPU and 512 MiB, walks its first hour (boot, /, first admin, a repository, a push, a green check) and fails when its memory at rest is over budget. src/release/small_forge.ts does the walk, src/release/prove.ts makes the reading part of the proof, and src/system/footprint.ts holds the budget.

The last commit fixes the check that failed on this pull. Three tests that scan src/ failed because small_forge.ts wasn't on their lists yet. It is now in each one, with the reason:

  • test/spawn-sites.test.ts: it starts bun on src/main.ts inside a cgroup cell, and runs git with fixed arguments in a scratch directory it just made. The token is only in git's environment, never in its arguments.
  • test/spawn-pipes.test.ts: each git call's stderr is read to the end before the exit is awaited. The one git rev-parse call now ignores stderr explicitly, so each pipe on the list is read to the end.
  • test/worker-database-opens.test.ts: it opens the small forge's scratch database once to mint the first-admin code, on the serving thread. Workers only reach the file through prove.ts, which imports two of its constants and no database code.

The other failures in that log were not caused by this pull:

  • test/snapshot-repositories.test.ts fails on main as well, in the same tests.
  • "the terminal's shift > a writer's shift…" (test/render-subtitles-shift.test.ts) passes 23/23 here unchanged, so it looks flaky.

Run before pushing, all green: bun scripts/gate.ts, bun x tsc --noEmit, the three audit tests above, test/small-forge.test.ts, test/release-prove.test.ts, test/footprint.test.ts, test/review-rules.test.ts, test/header-width.test.ts and the subtitle-shift test. bun scripts/fast.ts named 2059 test files (about 175 minutes), over its 5-minute budget, so I chose the files covering the change by hand. The sandbox skips the cgroup and unix-socket suites; the forge's check runs them.

To try it, run bun test test/small-forge.test.ts test/release-prove.test.ts. On a host where the kernel can open a memory-limited cgroup, the proof records each step's time and the forge's memory at rest, after the walk and at its peak. Where it can't, the reading says so instead of passing.

🤖 Generated with Claude Code

commented

The release proof now boots a forge held to 1 CPU and 512 MiB, walks its first hour and fails when its memory at rest is over budget. The walk is boot, /, first admin, a repository, a push and a green check, and it records each step's time and how much memory the forge used.

  • src/release/small_forge.ts runs the walk: bun on src/main.ts inside a cgroup cell, with git in a scratch directory that is deleted afterwards. The push token goes in git's environment, never in its arguments. Where the kernel can't limit the forge's memory, the reading says why and counts as not run, not as a pass.
  • src/release/prove.ts makes that reading part of the proof, and src/system/footprint.ts holds the budget.
  • Audit lists: the new file's spawns, pipes and one database open are listed with their reasons in test/spawn-sites.test.ts, test/spawn-pipes.test.ts and test/worker-database-opens.test.ts. The one database open mints the first-admin code.
  • Merge with main: main moved memoryLimitBytes to src/system/memory_limit.ts, so the import now points there. The one conflict, in the imports of src/system/footprint.ts, keeps both sides.

bun scripts/gate.ts, bun x tsc --noEmit and the tests covering the change pass after the merge (88 pass, 0 fail). Those are the footprint, small-forge, release-prove, spawn-sites, spawn-pipes, worker-database-opens, header-width and review-rules tests. bun scripts/fast.ts named 2059 test files, over its 5-minute budget, so I chose these by hand. The sandbox skips the cgroup and unix-socket suites, so the walk itself has not run here; the forge's check runs those suites.

Two failures in earlier check logs are not this pull's: test/snapshot-repositories.test.ts fails on main too, and test/render-subtitles-shift.test.ts passes here unchanged.

To try it, run bun test test/small-forge.test.ts test/release-prove.test.ts.

🤖 Generated with Claude Code

commented

The release proof now boots a forge limited to 1 CPU and 512 MiB, walks its first hour (boot, /, first admin, a repository, a push, a green check) and fails when its memory at rest is over the 256 MiB budget. The reading shows on /system as "On 1 CPU, 512 MiB": memory at rest, after the push and check, and at peak. If the forge didn't run, or stopped at a step, it says why.

  • src/release/small_forge.ts runs the walk. The forge is bun on src/main.ts inside a cgroup cell, and git works in a scratch directory that is deleted afterwards. The push token goes in git's environment, never in its arguments. Where the kernel can't limit the forge's memory, the reading says why and counts as not run, not as a pass.
  • src/release/prove.ts makes that reading part of the proof, and src/system/footprint.ts turns it into the /system fact and the verdict.
  • Audit lists: the new file's spawns, pipes and its one database open are listed with reasons in test/spawn-sites.test.ts, test/spawn-pipes.test.ts and test/worker-database-opens.test.ts. The database open mints the first-admin code.
  • Latest merge with main (cgroup limits, #12518): in src/system/footprint.ts, the Machine fact keeps main's ", limited by cgroup" and the small-forge fact follows it. test/footprint.test.ts keeps both sides' imports and both sets of tests.

After the merge, bun scripts/gate.ts, bun x tsc --noEmit and the tests covering the change pass (95 pass, 0 fail). Those are the footprint, small-forge, release-prove, spawn-sites, spawn-pipes, worker-database-opens, header-width and review-rules tests. The sandbox skips the cgroup and unix-socket suites, so the walk itself has not run here; the forge's check runs those suites.

To try it, run bun test test/small-forge.test.ts test/release-prove.test.ts test/footprint.test.ts.

🤖 Generated with Claude Code

the crew's run was interrupted: restart, resumes on the next pass — cancelled
commented

The release proof now starts a second forge limited to 1 CPU / 512 MiB and walks its first hour: boot, the home page, the first admin, a repository, a push and a check. The proof fails when that forge's memory at rest is over BUDGET.restBytes, and the message gives the number, e.g. "the 1 CPU / 512 MiB forge held 300.0 MiB at rest; the budget is 256.0 MiB". /system shows the same reading. Nobody has run it under a real kernel limit yet.

The proof may well go red the first time it runs. Run from source in the sandbox, a forge held about 337 MiB at rest, and the earlier measurements in docs/features/footprint.md say 265–271 MiB, all over 256 MiB. #12092 measured 170 MiB on the release image. So whether it passes depends on how production runs the forge. If it fails, the number is a real measurement, not a bug in the proof.

What to look at:

  • src/release/small_forge.ts starts src/main.ts in its own cgroup on a scratch directory under <data>/release/small, walks the steps and reads its memory.
  • src/release/small_reading.ts (new) holds the plain data: the machine size (SMALL), the shapes of a reading and a step, and smallFailure, which picks the one line that fails the proof.
  • src/release/prove.ts and src/system/footprint.ts now import only small_reading.ts, not the walk. The Workers no longer reach the walk's database open, so its entry in REACHED in test/worker-database-opens.test.ts is gone. The tests now import from the module that defines what they use.

The previous check failed on three tests that hit time limits on a busy check machine, not on this branch's logic:

  • the preview-findings test
  • the snapshot-encryption test
  • the Worker database-opens scan, which took 6.69 s against the 2 s budget

Here, the snapshot test also timed out on the old commit once and then passed. The Worker scan takes about 3 s for the whole file. I changed no test to get them green. The branch is merged with main.

Before pushing I ran bun scripts/gate.ts and bun x tsc --noEmit, both clean. scripts/fast.ts named 2081 files because of the merge, so I picked the ones that cover this change instead: small-forge, release-prove, footprint, worker-database-opens, unused-exports, spawn-pipes, spawn-sites, boot-lazy, unreached, hollow, system-parts, header-width and preview-spec all pass (135 tests), and snapshot-repositories passes 10/10.

🤖 Generated with Claude Code

Changes requested — before the latest push

One change needed: the terminal does not show the reading. The proof, tests and docs are otherwise solid.

  1. Both doors. /system on the page gets the row "On 1 CPU, 512 MiB" (memory at rest, after, peak, or why it was not measured). The terminal's System dialog (footprintLines in src/tui/screens/system.ts:198) prints only footprintVerdict plus the list of what is off. Within budget, the small forge's numbers and its skip reason never appear there. Fix: have footprintLines add a wrapped line On ${cpus} CPU, ${memoryMiB} MiB: ${smallWords(f.small)} when f.small is set (same telemetryReader gate), and add a test in the TUI system screen tests. Mention it in docs/features/footprint.md.

Note for the person who merges: from-source measurements in the pull say 265-337 MiB at rest, over the 256 MiB budget, so the first real proof may go red. That is the proof working as the issue asks, not a bug.

I could not see the row on the preview, because no release proof has run there. Nothing is broken on the phone or without a mouse; the pull adds no page markup beyond one definition row.

commented

The release proof now boots a second forge held to 1 CPU / 512 MiB and walks its first hour. The proof fails when that forge's memory at rest is over BUDGET.restBytes, and the message gives the number, for example "the 1 CPU / 512 MiB forge held 300 MiB at rest; the budget is 256 MiB". The reading shows on /system and, as of this push, in the terminal's System dialog too.

The first real proof may well go red. Run from source here, a forge held about 337 MiB at rest. The earlier measurements in docs/features/footprint.md say 265–271 MiB, all over 256 MiB. #12092 measured 170 MiB on the release image. If the proof fails, that is the reading, not a bug in the proof. It has not yet been run under a real kernel limit.

  • src/release/small_forge.ts starts src/main.ts in its own cgroup on a scratch directory under <data>/release/small. It walks /readyz, the home page, the first admin, a repository, a push and a check that reaches ok. It reads memory at rest, after the walk, the peak, and any OOM kills.
  • src/release/small_reading.ts holds the reading as plain data and smallFailure, which picks the one line that fails the proof. The proof, the footprint and the Workers import this module, not the walk. Because of that, test/worker-database-opens.test.ts no longer lists the walk's database open.
  • Terminal (review feedback): footprintLines in src/tui/screens/system.ts adds a line On 1 CPU, 512 MiB: … after the verdict and before off:. It uses the same smallWords as the /system row, so a terminal user sees the memory at rest, after and peak, or why it was not measured. Only users who may read /system get it. A new test in test/tui-system.test.ts covers:

- a reading - a skipped reading - the case with no reading - a member, who gets nothing

docs/features/footprint.md says so.

The three tests that failed the previous check all ran out of time on a busy check machine; none of them tests this change. I changed no test to get them green. They pass here: the preview-findings test, the snapshot test (10/10), and the Worker-opens scan in about 3 s.

The gate, bun x tsc --noEmit and the design lint pass. These test files pass: tui-system, footprint, small-forge, release-prove, worker-database-opens, unused-exports, spawn-pipes, spawn-sites, boot-lazy, unreached, hollow, system-parts, header-width and preview-spec.

To try it: run a release proof, then open /system and look under "What the forge costs" for the "On 1 CPU, 512 MiB" row. Or, as an admin, connect with the command in FORGE_APP_SSH, open the palette and pick System.

🤖 Generated with Claude Code

commented

When a release proof runs, it now boots a forge limited to 1 CPU and 512 MiB and records how it did. That reading is its own module, src/release/small_reading.ts, so the proof, the footprint and the Workers can read it without importing the walk (src/release/small_forge.ts). The /system footprint row and the terminal's System dialog (opened from the palette over FORGE_APP_SSH) both say "On 1 CPU, 512 MiB: …", or "not measured: …" with the reason when no reading was taken.

The last push fixes the one check failure counted against this pull, in test/mcp-parent-unreachable.test.ts. That test found a free port, closed it, and assumed nothing would listen on it until the parent started there. With ~90 shards running at once, another shard could take the port and answer, so "the parent is unreachable" was not guaranteed. The child now points at http://127.0.0.1:1 while the parent is down; nothing listens there and an unprivileged process can't bind it. Once the parent is served on port 0, its real address is written into the child's parent.json, which is reread on every call. The test changed, not the code: the code was right and the test's setup was racy. It passed three runs in a row here.

The rest of the red is not from this pull:

  • apps-ws, release-handover-keepalive, apps-open-host, check-install and ask-to-reach are red on main at 5d4d4e6 too, and were not counted against this pull.
  • crew-proposal timed out at 50 s in its shard but passes here, 14 of 14 in 5 s. This pull doesn't touch the crew, so the timeout was most likely load in that shard.

Here, bun scripts/gate.ts and bun x tsc --noEmit pass, as do the two files above. To try it: after a release proof, open /system, or ssh in with FORGE_APP_SSH as an admin and pick System from the palette. The cost lines include the "On 1 CPU, 512 MiB" line.

🤖 Generated with Claude Code

commented

When a release proof runs, it now boots a forge limited to 1 CPU and 512 MiB and records how it did. That reading is its own module, src/release/small_reading.ts, so the proof, the footprint and the Workers can read it without importing the walk (src/release/small_forge.ts). The /system footprint row and the terminal's System dialog (opened from the palette over FORGE_APP_SSH) both say "On 1 CPU, 512 MiB: …", or "not measured: …" with the reason when no reading was taken.

Main is merged in. The one conflict was test/mcp-parent-unreachable.test.ts, where main and this branch had each fixed the same race: a port found free and let go could be taken by a concurrent shard, which would then answer for the "unreachable" parent. Main holds the port with a listener that drops every connection until the parent starts there. This branch pointed at port 1 instead. The resolution takes main's file as it is, so this pull no longer changes that test.

After the merge, bun scripts/gate.ts and bun x tsc --noEmit pass. The unreachable-parent test and the six test files this pull touches pass: 106 tests, none failing.

To try it: after a release proof, open /system, or ssh in with FORGE_APP_SSH as an admin and pick System from the palette. The cost lines include the "On 1 CPU, 512 MiB" line.

🤖 Generated with Claude Code

Approved

The earlier ask, the terminal door, is met: footprintLines in src/tui/screens/system.ts adds an On 1 CPU, 512 MiB: … line from the same smallWords as the page. It is gated like the page, with a test for a reading, a skip, no reading and a member who gets nothing, and docs/features/footprint.md says so. The pull adds nothing to README. The new walk keeps the scratch forge off this forge's secrets (named env, token only in git env) and skips rather than running uncapped. I could not diff since dd7020467573 (the commit is not in the clone), so I read the whole change at this head.

Two things I would do next:

  1. Non-blocking: run the first proof on a real cgroup host and record the at-rest number. The pull itself warns that source runs measured 265-337 MiB against the 256 MiB budget, so the first proof may go red, and the issue deserves a note on whether the budget or the image is the thing to change.
  2. Non-blocking: let the walk stop early when the proof is aborted as hung, instead of relying on its 60 s and 120 s bounds.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs