Public: anyone can read this; the crew can change it.

The limits that bound a surge of new forges are constants and env values, not settings, and one of them is silent #13835

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

Births now go through one admission rule, and most of the surge limits are settings an admin can change without a restart. I only ran the spawn, limits, sleep and first-ten-minutes suites (they pass), plus tsc and the gate. The forge runs the full check on this push.

Notes

shipped production is serving it deploy 4ec2c16

opened this

Births now go through one admission rule, and most of the surge limits are settings an admin can change without a restart. I only ran the spawn, limits, sleep and first-ten-minutes suites (they pass), plus tsc and the gate. The forge runs the full check on this push.

What changed

  • Births: src/spawn/births.ts is new. A spawn needs three things:
    • fewer than spawn.births_at_once forges being made (default 8);
    • spawn.birth_memory_mb (1,536) free above forge.memory_floor_mb;
    • the disk above its existing floor.

    A spawn that doesn't fit waits in a first-in-first-out line for up to 15 s. Past that it gets 503 {error:"queued", place, aboutS, retryAfter, why} instead of a bare refusal. A birth keeps its place until the forge answers, or for 5 minutes at most. aboutS comes from a running mean of measured births, starting at 4 minutes.

  • Door: src/spawn/door.ts takes a place before it makes the forge. It gives the place back at once if no forge was made.
  • Settings (all on /admin/limits):
    • spawn.anon_per_day (100) and spawn.anon_global_per_day (2,000), which replace the constants in rate.ts.
    • spawn.timers_max (20), which replaces MAX_WATCHERS.
    • forges.awake_max (8) and forges.wakes_per_forge_hour (20). FORGE_CHILD_AWAKE_MAX and FORGE_CHILD_WAKES_PER_HOUR still apply while the setting is unset.
    • spawn.births_at_once and spawn.birth_memory_mb, the two new ones.
  • Untimed spawns: a spawn past the timer ceiling now logs "made but not timed". startWatch also takes an onSettled callback, which is how a place is given back when the forge answers.
  • Tests and docs: test/spawn-births.test.ts is new. The anonymous-spawn, live and release-prober tests now read the settings, and spawn-anon has a door-level 503 test. AGENTS.md and docs/features/forge-sleep.md describe the births state.

What I did not do

  • Idle time, NEW_FORGE_MS and held wake requests (wakeLimits) are still constants. I left them alone because the idle timing is tangled with the cell sweep.
  • No queue ticket. A spawn that waits 15 s and gets the 503 loses its place and rejoins at the back when it retries. The place is real while the request is held, and the 503 reports it.
  • Under test only, the births count applies only where a test sets it, and the memory reading is off unless a test provides one. The suite's probes never answer, so every spawn would otherwise hold its place for the full 5 minutes.
  • No "being made" page. The browser page for a queued spawn is unchanged; the 503 body and message are what carry the place.
  • No Retry-After header on the 503: the door has no way to set headers, so retryAfter is only in the JSON body.

To try it, set spawn.births_at_once to 1 on /admin/limits, then post two spawns at /api/v1/forges. The second waits up to 15 s, then answers 503 with place: 1 and aboutS.

🤖 Generated with Claude Code

Closes #13822

Show 19 earlier items

7 earlier comments · 1 event
commented

The limits that bound a surge of new forges are now settings on /admin/limits instead of constants and environment values. One rule decides whether a new forge may start, using the machine's own figures. A spawn that runs without a timer used to be silent and is now logged.

  • Admission (births.ts): a new forge starts only when a place is free under spawn.births_at_once and the memory left covers birth_memory_mb. Each start already under way counts its full birth_memory_mb against that memory. Before this, one reading of free memory could let several starts through on memory only one of them would get.
  • Waiting line: it holds at most 4 × births_at_once. People who are not signed in may use only a quarter of it, and signed-in people go first. If no timer starts for a start, its place is given back. The machine's memory figures are not shown to people who are not signed in.
  • Queued answer: the API and the new-forge form send Retry-After set to the seconds named in the body (before, the JSON fallback always said 60). The start form on the home page and the terminal start screen say in words the place in line and the wait. They read only whole numbers from the query.
  • Counts: /admin/limits, the terminal limits screen and the MCP limits tool show how many forges are starting and how many are waiting, next to the limits that decide it.
  • Looks: /admin/limits is now in both app.looks and app.looksAsOwner in forge.json. The gate had refused this pull's Looks: /admin/limits line because app.looks did not list the page. It is an admin page like /system, so it is pictured as the owner. test/looks-owner.test.ts now checks that it answers the preview admin with no credential in the page, and the list of non-top-level looks in test/forge-looks.test.ts names it.

I did not add a new door for the counts: they sit on the existing limits screen. The how-to is in docs/features/forge-sleep.md.

bun scripts/gate.ts and bun x tsc --noEmit pass. test/looks-owner.test.ts, test/forge-looks.test.ts and test/refresh-meta.test.ts pass (65 tests). The rest of the suite is left to the forge's check on this push.

To try it on the preview: open /admin/limits as an admin and read the line above the limits table. Then POST /api/v1/forges more times than spawn.births_at_once allows and read Retry-After on the queued answer.

🤖 Generated with Claude Code

Approved — before the latest push

Earlier asks are met. (1) Memory rule: whyNot now subtracts inFlight × birth_memory_mb, with a test. (2) Queued answer: Retry-After comes from doorHeaders, and the home, /agents, new-forge and terminal start screens say the place and wait in words, with the query read as bounded whole numbers only. (3) Both doors: the births line is on /admin/limits, the terminal limits screen and the MCP limits tool, and the terminal screen is tested. (4) Direction: the pull names P4 (#590); a person should confirm it as the pull says. (5) Seams: one each, not widened. The latest change adds /admin/limits to app.looks and looksAsOwner, with tests. The preview answers 200. I could not diff against 0dbc1125fbb6 because the commit is not in the clone, so I judged the change as given. Non-blocking: the docs/features how-to sits in forge-sleep.md rather than its own page, which is acceptable.

commented

<!-- look-pages e74cd400ebb7 --> Look pages for e74cd40, from the gate:

  • look pages (named): this pull changes src/web/pages/admin-limits.ts, src/web/pages/new-forge.ts, src/web/pages/start-forge.ts; a commit names /admin/limits
commented

The limits that bound a surge of new forges are now settings on /admin/limits instead of constants and environment values. One rule decides whether a new forge may start, using the machine's own figures. A spawn that runs without a timer used to be silent and is now logged.

  • Admission (src/spawn/births.ts): a new forge starts only when a place is free under spawn.births_at_once and the memory left covers birth_memory_mb. Each start already under way counts its full birth_memory_mb against that memory. Before, one reading of free memory could let several starts through on memory only one of them would get.
  • Waiting line: it holds at most 4 × births_at_once. People who are not signed in may fill only a quarter of it, and signed-in people go first. If a forge starts but no timer is set for it, its place is given back. The machine's memory figures are not shown to people who are not signed in.
  • Queued answer: the API and the new-forge form send Retry-After set to the seconds named in the body (before, the JSON fallback always said 60). The home start form and the terminal start screen say in words the place in line and the wait. They read only whole numbers from the query.
  • Counts: /admin/limits, the terminal limits screen and the MCP limits tool show how many forges are starting and how many are waiting, next to the limits that decide it.
  • Looks: /admin/limits is in both app.looks and app.looksAsOwner in forge.json. It is an admin page like /system. test/looks-owner.test.ts checks it answers the preview admin with no credential in the page, and test/forge-looks.test.ts lists it among the looks below the top level.

The last check (run 01m7hykfdzev30n0bpqnzgcqk8) failed on tests this pull does not touch. These 107 tests pass on the merge with current main, and three copies of the first three below also passed when run four at a time:

  • attention, tui-crawlers, media:
    • All pass here, after the merge and when run four at a time.
    • The one failure whose assertion survives in the log is the media test. It got "The forge is busy…" back: src/http/crawl.ts sends that when signed-out requests have waited too long, which happens on a loaded runner. That runner also had about 13% disk free.
    • The log's start is cut off, so the attention and crawler assertions are not in it.
  • Preview-seed test in looks-owner: over its 2 s budget (2.41 s in the check). It runs about 2 s on main too: 1.98 s and 2.26 s here, against 1.50 s and 1.38 s on this branch.

I checked that this pull's preload additions add no import cost: births.ts and door.ts add 0 ms once the modules main already preloads are in. This push merges current main, so the check runs on a fresh base. Load-related red runs like these are what #14187 is working on. I did not add the seed test to SLOW or change any of those assertions.

bun scripts/gate.ts --fix and bun x tsc --noEmit pass.

To try it on the preview, open /admin/limits as an admin and read the line above the limits table. Then POST /api/v1/forges more times than spawn.births_at_once allows and read Retry-After on the queued answer.

🤖 Generated with Claude Code

commented

<!-- look-pages 46fcd33a8617 --> Look pages for 46fcd33, from the gate:

  • look pages (named): this pull changes src/web/pages/admin-limits.ts, src/web/pages/new-forge.ts, src/web/pages/start-forge.ts; a commit names /admin/limits
commented

I fixed one real cause of the failed check and pushed, but I couldn't reproduce most of the seven failures, so I can't promise the next check is green.

What I fixed: test/spawn-births.test.ts reset the disk reader with provideDiskForTest(undefined) after each test. That handed every later test file in the same shard the runner's real disk, which was 13.6% free (under the 15% floor), so tests that start forges, agents or cells could be refused. It now restores the suite's roomy reading (roomyDisk, now exported from test/helpers/roomy-disk.ts) and deletes the data directories it creates. A new test fails if the real reading leaks again.

Not fixed:

  • Six failures: web-ui, shared-red, hardening, the memory sampler, proveHosted and the preview seed's 2.07 s timing all pass locally, alone and grouped with the spawn tests. A low disk forced in front of web-ui and shared-red didn't break them either. The log stops before those shards' details, so I couldn't find their cause.
  • One failure is already on main: the crew-concurrency test in test/capacity.test.ts fails about one run in five when grouped with other files, with main's preload as well as this pull's.
  • Same reset elsewhere: test/maintain-evidence.test.ts and test/no-alarm-banner.test.ts also call provideDiskForTest(undefined). They're outside this pull, so I left them.

Gate and tsc pass. The covering files (spawn, limits, capacity and the named failing files) pass apart from that intermittent crew-concurrency test. I ran named files only; the forge runs the full suite on this push.

---

Births now go through one admission rule, and most of the limits on a surge of new forges are settings an admin can change on /admin/limits without a restart.

What changed

  • Births: src/spawn/births.ts is new. A spawn needs three things:
    • fewer than spawn.births_at_once forges being made at once (default 8);
    • spawn.birth_memory_mb (1,536) free above forge.memory_floor_mb, counting what births already in flight will still take;
    • the disk above its existing floor.

    A spawn that doesn't fit waits in a first-in-first-out line for up to 15 s, with signed-in people ahead of strangers. The line holds at most four times the births limit, and strangers may fill only a quarter of it. Past the wait, the spawn gets 503 {error:"queued", place, aboutS, retryAfter, why} with a Retry-After header. Strangers aren't shown the machine's figures. A birth keeps its place until the new forge answers, or 5 minutes at most. aboutS comes from a running average of measured births, starting at 4 minutes.

  • Door: src/spawn/door.ts takes a place before making the forge and gives it back at once if no forge was made or no watch started. The start page shows a queued spawn as words.
  • Settings (all on /admin/limits, which also shows births in flight and spawns waiting, as does the terminal's limits screen):
    • spawn.anon_per_day (100) and spawn.anon_global_per_day (2,000) replace the constants in rate.ts.
    • spawn.timers_max (20) replaces MAX_WATCHERS. A spawn past it is now logged as made but not timed, so it is no longer silent.
    • forges.awake_max (8) and forges.wakes_per_forge_hour (20). FORGE_CHILD_AWAKE_MAX and FORGE_CHILD_WAKES_PER_HOUR still apply while the setting is unset.
    • spawn.births_at_once and spawn.birth_memory_mb, the two new ones.
  • Test preload: gives the suite 64 GiB of free memory for births and makes new child forges answer their first check at once. Without that, every spawn in the suite would hold its place for the full 5 minutes.
  • /admin/limits in the forge's screenshots: it is now captured as the owner.

What I did not do

  • Idle time, NEW_FORGE_MS and held wake requests (wakeLimits) are still constants, because the idle timing is tangled with the cell sweep.
  • No queue ticket: a spawn that gets the 503 rejoins at the back when it retries.
  • No "being made" page: the queued state is carried by the 503 and the start page's words.

To try it: set spawn.births_at_once to 1 on /admin/limits, then post two spawns to /api/v1/forges. The second waits up to 15 s, then answers 503 with place: 1, aboutS and a Retry-After header.

🤖 Generated with Claude Code

Closes #13822

commented

Merged main into this branch again. Only src/spawn/host.ts conflicted, in two lines of wakeGate, and the fix keeps both sides:

  • From main: wakingResponse no longer takes parent, so both of its calls now begin wakingResponse(request, name, …).
  • From this branch: the new-forge limit still passes process.env, parent.db to rates.refusal, so the configured limits still bound a surge of new forges.

bun x tsc --noEmit is clean. bun scripts/gate.ts --fix changed nothing. scripts/fast.ts said the diff was too broad to name tests, so I picked 12 files that cover the spawn gate, waking, new-forge timing and the limits pages (spawn-births, spawn-limits-listed, spawn-anon, spawn-live, spawn-host, spawn-gate-e2e, spawn-person-limits, wake-budget, waking-plain, limits, admin-limits-words, home-start-forge). All 74 tests in them pass. Tests that need unix sockets or cells are skipped in this sandbox; the forge's check runs them on this push.

To try it, open the preview at https://lab--13835--forge-trevato-apps.open-platform.sh, check that /admin/limits lists the new-forge limits, and start forges past the limit to see the refusal page.

🤖 Generated with Claude Code

commented

<!-- look-pages 2d9ed8f961e0 --> Look pages for 2d9ed8f, from the gate:

  • look pages (named): this pull changes src/web/pages/admin-limits.ts, src/web/pages/new-forge.ts, src/web/pages/start-forge.ts; a commit names /admin/limits
commented

Nothing to change, so I made no commit and the branch stays at 2d9ed8f. The gate's "look pages" note is the satisfied form: it is named, not inferred or none.

Pull description:

The limits that bound a surge of new forges are now settings the owner can see and change on /admin/limits. Before, they were constants and env values, and one of them refused without a word. A queued spawn now says its place and wait as words, with a Retry-After header.

  • Admission (src/spawn/): new forges are let in by one rule based on the machine's figures. Forges still being made count against the memory they haven't taken yet. The waiting line has a bound. The machine's figures are kept from strangers, and a spawn that starts with no timer is logged.
  • Pages (src/web/pages/):
    • /admin/limits shows forges being made and spawns waiting at both limits.
    • The new-forge form and the home and /agents start form pass the queue's wait and Retry-After through on a refusal.
  • Merge: main was merged in. The conflict in wakeGate (src/spawn/host.ts) keeps main's shorter wakingResponse(request, …) and this branch's rates.refusal(…, process.env, parent.db).

Pictures: the forge pictures /admin/limits, light and dark, as owner and visitor. It is in app.looks and looksAsOwner, and commits e74cd400e and 47e093b6e name it in their Looks: /admin/limits lines.

I chose not to name a page for new-forge.ts and start-forge.ts. Their changes only show in the response to a refused POST, when a spawn is queued, and no GET page in app.looks can show that. test/home-start-forge.test.ts and test/spawn-births.test.ts cover those words and headers instead.

To try it: sign in as owner on the preview at https://lab--13835--forge-trevato-apps.open-platform.sh and open /admin/limits to see the new-forge limits. Start forges past the limit to see the queued reply, with its place and wait in words.

🤖 Generated with Claude Code

Approved

Approved. The earlier asks stand met: the memory rule counts births in flight, a queued 503 carries Retry-After and is worded on the home, agents and terminal start paths, and births in flight and waiting show on /admin/limits and the terminal limits screen. The queue is capped and strangers get no machine figures. A place is released if no timer starts. Changes since my last look are a merge with main and the test fix that restores the roomy disk reading after spawn-births, which is test-only. The commit e74cd400ebb7 is not in my clone, so I read the final diff rather than the incremental one. Non-blocking: the direction (pipeline first) does not name this pull. The pillar (P4, #590) needs a person to confirm it before it lands.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs