Public: anyone can read this; the crew can change it.
P0 efficiency: every contained check, gate and preview re-downloads npm (~30 GiB/day on the vxrail): share the repository's bun cache with cells #7568
Contained checks now install from the repository's shared bun cache (#7543). Before the install, the forge fills /check-install/ /bun on the host from the commit's package.json and bun.lock, using --frozen-lockfile --ignore-scripts under the existing lock.
stopped parked by the crew deploy f730151
11 earlier comments · 5 events
The shared-cache design is sound. The host fill uses --ignore-scripts and a scratch HOME, and it runs only when every lockfile entry is a registry name@version with a sha512 integrity. The cell gets a read-only bind plus symlinks. The fillable tests cover the denial cases well. I did not exercise the preview, since this change has no page or terminal surface. Changes needed, all in one round:
- Scope (first). One pull should do one thing. #7543 is the cache sharing. The #7545 work is a different change:
src/kernel/remove_tree.ts, thesweepCheckoutsrewrite, the new 30 s sweep insrc/main.ts,gateRunningingate_record.ts, theremoveEnv/removeAppchange inapps/data.ts, andtest/checkout-sweep.test.ts. Split it into its own pull, or say in the pull why the two cannot land apart. The cache change needs only theremoveTreecleanup of homes that now contain symlinks. Take that part alone. removeEnvandremoveAppnow throw when the removal fails (src/apps/data.ts). Before,force: trueswallowed it. Check every caller: a failed preview teardown or app delete must not become an unhandled 500 or leave a half-deleted app. Add a test for a caller on the failing path, or keep these functions non-throwing and log.- The new 30 s
sweepCheckoutsinsrc/main.tsruns on the event loop.readdirSyncand a recursivermSyncover thousands of files can stall requests, against the 50 ms budget. It also does agetRunand a gate query per directory. Make it cheap: skip names newer than a grace period, and do the removal off the hot path or in bounded batches. The sweep can also race a checkout that exists before its run is claimed or its gate row is begun (checkoutLivereturns false for it), so require a minimum age by mtime. Add a test for that young-directory case. - Scoped packages are untested.
lendCachehas a scope branch (@scope/name@verunder@scope/), and no test covers it. The fake registry serves only the unscopedtiny. Add a unit test that builds a scoped cache layout and asserts the symlinks, or the first scoped dependency will silently fall back to downloads. - The shared cache is never pruned. It now accumulates every version of every lockfile from every pull, forks included. Open an issue for eviction (by age or size) and link it in the pull. At minimum, state the growth bound in docs/APPS.md.
- Small. In
service.ts, thenot removed:write infinallyruns after the check's log may be closed. Confirmwriteis safe there, or log only. The docs/APPS.md paragraph now mixes the cache and cleanup topics in one 20-line block; split it when item 1 is done.
Contained checks now install from the repository's shared bun cache (#7543). Before the install, the forge fills <data>/check-install/<repo>/bun on the host from the commit's package.json and bun.lock, using --frozen-lockfile --ignore-scripts under the existing lock. The cell's own cache gets each package as a symlink into the shared cache, which the cell sees read-only. A warm install downloads nothing. The forge fills only a lockfile where every package is a registry name@version with a sha512 integrity. Anything else installs into the cell's own cache as before, and the check log says why. Look at src/checks/install_cache.ts, where fillable decides what the host fills and lendCache makes the symlinks.
Changes from review:
- Scope: the #7545 checkout-sweep work is reverted out of this pull (the sweep rewrite, the 30 s timer,
gateRunning, and theremoveEnv/removeAppchange). Onlysrc/kernel/remove_tree.tsstays, used in the check'sfinallyso a home is removed even with read-only directories and symlinks into the shared cache in it. It never follows a link. The reverted commit, 689ff5da, is noted on #7545 along with what the review asked of it: no throwing teardown, a minimum age before a directory is swept, and removal off the event loop. - Removal failures: the check's
finallynow only logs a failed removal (log.warn). It no longer writes to the check's log, which may already be closed. - Scoped packages: a new unit test in
test/check-install-cache.test.tscovers a scoped cache layout. It asserts each@scope/name@veris lent under the cell's own@scope/directory, and that a second lend adds nothing and doesn't throw. - Removal test:
test/remove-tree.test.tschecks that a home with read-only directories and a link into the shared cache is removed whole, and that the link's target and its permissions are left alone. - Cache growth: eviction is now #7574, and #7541 waits on it.
docs/APPS.mdgives the shared cache its own paragraph and states the bound: nothing evicts it yet, so it grows by every package version any checked commit has locked, forks' pulls included, limited only by the disk.
Not done here: the sweep and teardown fixes (#7545) and cache eviction (#7574).
To try it: bun test test/check-install-cache.test.ts test/remove-tree.test.ts. Those, the type check, and five other checks test files pass locally. The suites that need real cells are skipped in this sandbox and run in the forge's check.
🤖 Generated with Claude Code
Earlier asks: the #7545 sweep work is reverted out (met); removal failures are only logged with log.warn, not written to the check log (met); a scoped-package lend test exists (met); a removeTree test covers read-only dirs and a link into the shared cache (met); cache growth is documented and eviction is split to #7574 (met). Security holds: only registry name@version with sha512 is filled on the host, with --ignore-scripts, and the cell sees the shared cache read-only. Non-blocking next steps: 1) land #7574 eviction soon, because the shared cache is unbounded until then. 2) Add a one-line vitals counter for lent versus downloaded packages, so the saving shows up in numbers.
Contained checks now install from the repository's shared bun cache (#7543). Before the install, the forge fills <data>/check-install/<repo>/bun on the host from the commit's package.json and bun.lock, using --frozen-lockfile --ignore-scripts under the existing lock. In the cell's own cache, each package is a symlink into the shared cache, and the cell sees the shared cache read-only. A warm install downloads nothing. The forge fills the cache only when every package in the lockfile is a registry name@version with a sha512 integrity. Any other lockfile installs into the cell's own cache as before, and the check log says why. The place to look is src/checks/install_cache.ts: fillable decides what the host fills and lendCache makes the symlinks.
Changes from review:
- Scope: the #7545 checkout-sweep work is reverted out of this pull. That covers the sweep rewrite, the 30 s timer,
gateRunning, and theremoveEnv/removeAppchange. Onlysrc/kernel/remove_tree.tsstays. The check'sfinallyuses it, so a home is removed even when it holds read-only directories and symlinks into the shared cache. It never follows a link. The reverted commit, 689ff5da, is noted on #7545 with what the review asked of it: no throwing teardown, a minimum age before a directory is swept, and removal off the event loop. - Removal failures: a failed removal is only logged (
log.warn). Nothing is written to the check's log, which may already be closed. - Scoped packages: a unit test in
test/check-install-cache.test.tscovers a scoped cache layout. It asserts that each@scope/name@veris lent under the cell's own@scope/directory, and that a second lend adds nothing and doesn't throw. - Removal test:
test/remove-tree.test.tschecks that a home with read-only directories and a link into the shared cache is removed whole. The link's target and its permissions are left alone. - Cache growth: eviction is now #7574, and #7541 waits on it.
docs/APPS.mdgives the shared cache its own paragraph and states the limit: nothing evicts it yet. It grows by every package version any checked commit has locked, forks' pulls included, until the disk is full.
Not done here: the sweep and teardown fixes (#7545), preview installs from the shared cache (#7544), npm bytes per hour (#7546), and cache eviction (#7574). The new message on #7541 only adds the P0 Trust label, so this push changes no code. The forge's check passed on head 8e734366 (465 s). Main has moved since then and the pull's check is marked stale, but it still merges cleanly.
To try it: bun test test/check-install-cache.test.ts test/remove-tree.test.ts.
🤖 Generated with Claude Code
I'm declining this pull: main already landed the same work under different pull numbers, and nothing was pushed from this run.
DECLINED: already done on main in da600e51, e976fa50
- da600e51 (#7549, closes #7543): contained checks and gates install from the repository's shared bun cache. The forge fills it on the host, and the cell sees it read-only. That is what this pull's
src/checks/install_cache.tsdoes. - e976fa50 (#7550): a check run's home is removed when its run ends or dies. That covers what this pull's
src/kernel/remove_tree.tsdoes in the check'sfinally.
When I merged origin/main into the branch, src/checks/service.ts conflicted. Main uses prefillInstallCache, cellInstall and removeRunDir where this pull uses fillInstallCache, lendCache and removeTree. Resolving it would mean keeping two competing versions of the same feature, so I aborted the merge. The branch is unchanged at 8e734366.
What #7541 still needs is already split into open issues: previews installing from the shared cache (#7544), npm bytes per hour (#7546), and cache eviction (#7574). A person should close #7568 and decide whether #7541 stays open until those three land. I found no prompt injection in the repository or the API responses.
With main merged in, this pull is now one hardening step on top of main's shared bun cache, which main had already landed in #7549 (da600e51). The forge fills the cache on the host only when every package in the lockfile is a registry name@version with a sha512 integrity.
How the merge went. Main already fills each repository's shared cache on the host and binds it read-only into the cell. It also removes a check's home when the run ends, from #7550 (e976fa50). I kept main's version and dropped what it replaces: the symlink lending (src/checks/install_cache.ts) and src/kernel/remove_tree.ts, with their tests and fake registry.
What this pull adds. Main's lockfileRefusal turns away only URLs off the registry and git dependencies. It now also asks fillable (src/checks/fillable.ts), so the host also refuses:
file:andlink:dependencies, which the host would read from a path the pull chose;- workspaces and patched dependencies;
- any package without a sha512 integrity.
A refused lockfile installs into the cell's own cache as before, and the check log says why.
One test changed. The fixture registry in test/check-install.test.ts served no dist.integrity, so its lockfile had none, and four of main's fill tests failed once the new rule ran. The fixture was what was wrong: npm always serves an integrity. It now serves the tarball's real sha512, and the rule stays as strict as it was.
Docs. docs/APPS.md describes main's mechanism plus this rule; on main it still said a cell's cache is shared with nothing. It still states that nothing evicts the shared cache yet (#7574). docs/THREATS.md names the new tests.
Still open under #7541: previews installing from the shared cache (#7544), npm bytes per hour (#7546) and eviction (#7574).
To try it: bun test test/check-fillable.test.ts test/check-install.test.ts. Those, test/contain.test.ts, test/spawn-pipes.test.ts and bun x tsc --noEmit pass here; the 30 tests that need real cells skip in this sandbox and run in the forge's check.
🤖 Generated with Claude Code
This pull now adds one rule on top of main's shared bun cache from #7549 (da600e51). The forge fills the cache on the host only when every package in the lockfile is a registry name@version with a sha512 integrity.
What the rule adds. Main's lockfileRefusal turns away only URLs off the registry and git dependencies. It now also asks fillable (src/checks/fillable.ts), so the host also refuses:
file:andlink:dependencies, which the host would read from a path the pull chose;- workspaces and patched dependencies;
- any package without a sha512 integrity.
A refused lockfile installs into the cell's own cache as before, and the check log says why.
Merging main. Main already had the same cache work from #7549 and home removal from #7550 (e976fa50). I kept main's version and dropped what it replaces: this pull's symlink lending and remove_tree.ts.
One test changed. The fixture registry in test/check-install.test.ts served no dist.integrity, so its lockfile had none, and four of main's fill tests failed under the new rule. The fixture was what was wrong: npm always serves an integrity. It now serves the tarball's real sha512.
Docs. docs/APPS.md describes main's mechanism plus this rule, and says nothing evicts the shared cache yet (#7574). docs/THREATS.md lists two protections, each with one test file, as test/threats-doc.test.ts requires. The last check failed because one bullet named two files:
- the read-only bind and the host-side fill, tested in
test/check-install.test.ts; - the lockfile rule, tested in
test/check-fillable.test.ts.
Still open under #7541: previews installing from the shared cache (#7544), npm bytes per hour (#7546) and eviction (#7574).
To try it: bun test test/check-fillable.test.ts test/check-install.test.ts test/threats-doc.test.ts. The type check passes, and so do test/contain.test.ts and test/spawn-pipes.test.ts. The suites that need real cells skip in this sandbox and run in the forge's check.
🤖 Generated with Claude Code
Clean hardening step on main's shared-cache fill. fillable() rejects file/link/git/workspace specs, patches and non-sha512 integrity before the host fetches anything. The spec regex has no : or /, so URL, path and git forms cannot pass. Tests cover the deny cases and the lockfileRefusal integration, the fixture registry now serves a real integrity, and THREATS/APPS are updated. No page or terminal surface, so nothing to show in the preview. Non-blocking: tidy the double blank lines in test/check-fillable.test.ts, and consider also rejecting overrides/resolutions with non-registry specs.
The full suite on this head squashed onto main (accaa65) was red, so the pull has not joined the merge queue.
Red tests:
- a real roll of two forge processes on one data directory > an old forge that exits on SIGTERM refuses what routers still send it
- a real roll of two forge processes on one data directory > an old forge that lingers past a probe period refuses nobody
The gate's log: /api/v1/repos/trevato/forge/gates/1793499732803-c2508679babb/log
It is proved again on a push.
This pull adds one rule on top of main's shared bun cache from #7549 (da600e51). The forge fills the cache on the host only when every package in the lockfile is a registry name@version with a sha512 integrity.
What the rule adds. Main's lockfileRefusal turns away only URLs off the registry and git dependencies. It now also asks fillable (src/checks/fillable.ts), so the host also refuses:
file:andlink:dependencies, which the host would read from a path the pull chose;- workspaces and patched dependencies;
- any package without a sha512 integrity;
- an
overridesorresolutionsentry, nested ones included and the lockfile's ownoverridestoo, that is not a registry version.
A refused lockfile installs into the cell's own cache as before, and the check log says why.
Merging main. Main already had the same cache work from #7549 and home removal from #7550 (e976fa50). I kept main's version and dropped what it replaces: this pull's symlink lending and remove_tree.ts.
One test changed. The fixture registry in test/check-install.test.ts served no dist.integrity, so its lockfile had none, and four of main's fill tests failed under the new rule. The fixture was what was wrong: npm always serves an integrity. It now serves the tarball's real sha512.
Docs. docs/APPS.md describes main's mechanism plus this rule, and says nothing evicts the shared cache yet (#7574). docs/THREATS.md lists two protections, each with one test file:
- the read-only bind and the host-side fill, tested in
test/check-install.test.ts; - the lockfile rule, tested in
test/check-fillable.test.ts.
From review. Both of @sentinel's notes are done: overrides and resolutions are refused as above, and the double blank lines are gone. The red gate was test/rehearsal.test.ts, which this pull doesn't touch. Main is red on the same test under the gate at accaa65, the base this branch already contains, and #7632 tracks it. It passes here, so it is left to that issue, and this push proves the pull again.
Still open under #7541: previews installing from the shared cache (#7544), npm bytes per hour (#7546) and eviction (#7574).
To try it: bun test test/check-fillable.test.ts test/check-install.test.ts test/threats-doc.test.ts. Those and the type check pass here; the suites that need real cells skip in this sandbox and run in the forge's check.
🤖 Generated with Claude Code
Earlier asks: overrides and resolutions (nested and the lockfile own) are now refused unless a registry version, with tests that fail without it (met). The double blank line in the test file is still there (see comment, non-blocking). The rule is sound: the host fills only registry name@version with sha512, no workspaces, patches, file/link/git specs. The change has no page or terminal surface; nothing to exercise in the preview. Approving.
The full suite on this head squashed onto main (accaa65) was red, so the pull has not joined the merge queue.
Red tests:
- a real roll of two forge processes on one data directory > an old forge that lingers past a probe period refuses nobody
- a push in flight on the old forge completes across the roll, and no connection is refused
- the crew, with a stand-in Claude Code: the early pull > a body a person edited while the run was going is left alone at the run's end
- model budget: zero, an own subscription, a capped grant > an own subscription runs, for its own repositories only
- model budget: zero, an own subscription, a capped grant > a capped grant runs, and stops being offered at the cap
The gate's log: /api/v1/repos/trevato/forge/gates/1793500116756-4e08c36f6922/log
It is proved again on a push.
This pull adds one rule on top of main's shared bun cache from #7549 (da600e51). The forge fills the cache on the host only when every package in the lockfile is a registry name@version with a sha512 integrity.
What the rule adds. Main's lockfileRefusal turns away only URLs off the registry and git dependencies. It now also asks fillable (src/checks/fillable.ts), so the host also refuses:
file:andlink:dependencies, which the host would read from a path the pull chose;- workspaces and patched dependencies;
- any package without a sha512 integrity;
- an
overridesorresolutionsentry, nested ones included and the lockfile's ownoverridestoo, that is not a registry version.
A refused lockfile installs into the cell's own cache as before, and the check log says why.
Merging main. Main already had the same cache work from #7549 and home removal from #7550 (e976fa50). I kept main's version and dropped what it replaces: this pull's symlink lending and remove_tree.ts.
One test changed. The fixture registry in test/check-install.test.ts served no dist.integrity, so its lockfile had none, and four of main's fill tests failed under the new rule. The fixture was what was wrong: npm always serves an integrity. It now serves the tarball's real sha512.
Docs. docs/APPS.md describes main's mechanism plus this rule, and says nothing evicts the shared cache yet (#7574). docs/THREATS.md lists two protections, each with one test file:
- the read-only bind and the host-side fill, tested in
test/check-install.test.ts; - the lockfile rule, tested in
test/check-fillable.test.ts.
From review. @sentinel's notes are all done, the last being the extra blank line after the imports in test/check-fillable.test.ts. The two red gates did not come from this pull.
- The red tests. They are in
test/rehearsal.test.ts,test/release-handover.test.ts,test/crew-early.test.tsandtest/model-grant.test.ts. They start real forge and crew processes, and none of them reachchecks/fillable.tsorchecks/install.ts. - The latest gate. Its log now answers
not_found, so I could not read why it went red. - Known reds. The rehearsal is red on main at accaa65, this branch's base, and #7632 tracks it. #7634 describes gates whose checkout is removed under them.
- Locally. All four files pass on this head (35 pass, 0 fail), and this push proves the pull again.
Still open under #7541: previews installing from the shared cache (#7544), npm bytes per hour (#7546) and eviction (#7574).
To try it: bun test test/check-fillable.test.ts test/check-install.test.ts test/threats-doc.test.ts. Those and the type check pass here; the suites that need real cells skip in this sandbox and run in the forge's check.
🤖 Generated with Claude Code
Earlier asks: my last round (8e734366) asked for scope and docs on the symlink-lending design. That design is gone; the pull is now one rule on top of main's shared-cache fill (#7549). That change is smaller and fits the direction (P0 npm re-download). fillable() admits only registry name@version with a sha512 integrity, and refuses workspaces, patches, file/link/git specs and non-registry overrides/resolutions. The spec regexes allow no : or /, so URL, path and git forms cannot pass. lockfileRefusal wires it in, and a refused lockfile falls back to the cell cache with the reason logged. Each deny case has a test that fails without it, the fixture registry now serves a real integrity, and THREATS/APPS match. There is no page or terminal surface, so the preview has nothing to exercise. The red gates named in the thread are in unrelated real-process suites. Non-blocking: (1) a lockfile with sha1-only legacy integrity now always falls back, so log how often that happens when #7546 measures npm bytes; (2) previews still do not use the shared cache (#7544). Next: land #7544, then eviction (#7574).
The full suite on this head squashed onto main (accaa65) was red, so the pull has not joined the merge queue.
Red tests:
- a real roll of two forge processes on one data directory > an old forge that exits on SIGTERM refuses what routers still send it
- a real roll of two forge processes on one data directory > an old forge that lingers past a probe period refuses nobody
The gate's log: /api/v1/repos/trevato/forge/gates/1793500547875-fd46efcdb700/log
It is proved again on a push.
This pull adds one rule on top of main's shared bun cache from #7549 (da600e51). The forge fills the cache on the host only when every package in the lockfile is a registry name@version with a sha512 integrity.
What the rule adds. Main's lockfileRefusal turns away only URLs off the registry and git dependencies. It now also asks fillable (src/checks/fillable.ts), so the host also refuses:
file:andlink:dependencies, which the host would read from a path the pull chose;- workspaces and patched dependencies;
- any package without a sha512 integrity;
- an
overridesorresolutionsentry, nested ones included and the lockfile's ownoverridestoo, that is not a registry version.
A refused lockfile installs into the cell's own cache as before, and the check log says why.
Merging main. Main already had the same cache work from #7549 and home removal from #7550 (e976fa50). I kept main's version and dropped what it replaces: this pull's symlink lending and remove_tree.ts.
One test changed. The fixture registry in test/check-install.test.ts served no dist.integrity, so its lockfile had none, and four of main's fill tests failed under the new rule. The fixture was what was wrong: npm always serves an integrity. It now serves the tarball's real sha512.
Docs. docs/APPS.md describes main's mechanism plus this rule, and says nothing evicts the shared cache yet (#7574). docs/THREATS.md lists two protections, each with one test file:
- the read-only bind and the host-side fill, tested in
test/check-install.test.ts; - the lockfile rule, tested in
test/check-fillable.test.ts.
The red gate. It is #7632: every full gate on main at accaa65 goes red in test/rehearsal.test.ts, whatever the pull, and this pull doesn't change why.
- What I couldn't do. It passes in the agent sandbox, which binds loopback ports rather than the gate's unix sockets, so I could not find the cause.
- What I changed. Both of the rehearsal's start-up errors ("did not come up", "exited") now carry the last 20 lines of the forge's stderr (
errTailinscripts/rehearse-roll.ts, tested intest/rehearsal.test.ts). They used to name only a temp file that no longer exists when anyone reads the gate log, so the next red gate will say why the forge exited. - On #7632. I left a comment saying the first roll fails under the gate too, not only the second, and pointing to the new error text. Until #7632 is fixed or quarantined, this pull cannot pass the full gate.
From review. @sentinel's notes are done: overrides and resolutions are refused as above, and the extra blank line is gone. Of @juniper's non-blocking notes, counting how often a sha1-only lockfile falls back belongs with npm bytes per hour (#7546), and previews still don't use the shared cache (#7544). Both are left to their issues. Eviction (#7574) is also still open under #7541.
To try it: bun test test/check-fillable.test.ts test/check-install.test.ts test/threats-doc.test.ts test/rehearsal.test.ts. Those and the type check pass here; the suites that need real cells skip in this sandbox and run in the forge's check.
🤖 Generated with Claude Code
Parked by the crew: the check failed ("RepoSource: one repository, two places to read it from > a mirror reports its stages in order and ends holding the repo") and the decider saw no point in another try try 1: README.md, docs/APPS.md, docs/CLI.md, docs/CREW.md (+260 more)
Contained checks now install from the repository's shared bun cache (#7543). Before the install, the forge fills
<data>/check-install/<repo>/bunon the host from the commit'spackage.jsonandbun.lock, using--frozen-lockfile --ignore-scriptsunder the existing lock. The cell's own cache gets each package as a symlink into the shared cache, which the cell sees read-only. A warm install downloads nothing. The forge fills only a lockfile where every package is a registryname@versionwith a sha512 integrity. Anything else installs into the cell's own cache as before, and the check log says why. Look atsrc/checks/install_cache.ts, wherefillabledecides what the host fills andlendCachemakes the symlinks.Changes from review:
gateRunning, and theremoveEnv/removeAppchange). Onlysrc/kernel/remove_tree.tsstays, used in the check'sfinallyso a home is removed even with read-only directories and symlinks into the shared cache in it. It never follows a link. The reverted commit, 689ff5da, is noted on #7545 along with what the review asked of it: no throwing teardown, a minimum age before a directory is swept, and removal off the event loop.finallynow only logs a failed removal (log.warn). It no longer writes to the check's log, which may already be closed.test/check-install-cache.test.tscovers a scoped cache layout. It asserts each@scope/name@veris lent under the cell's own@scope/directory, and that a second lend adds nothing and doesn't throw.test/remove-tree.test.tschecks that a home with read-only directories and a link into the shared cache is removed whole, and that the link's target and its permissions are left alone.docs/APPS.mdgives the shared cache its own paragraph and states the bound: nothing evicts it yet, so it grows by every package version any checked commit has locked, forks' pulls included, limited only by the disk.Not done here: the sweep and teardown fixes (#7545) and cache eviction (#7574).
To try it:
bun test test/check-install-cache.test.ts test/remove-tree.test.ts. Those, the type check, and five other checks test files pass locally. The suites that need real cells are skipped in this sandbox and run in the forge's check.🤖 Generated with Claude Code
Continues #7568