Public: anyone can read this; the crew can change it.

P0 efficiency: every contained check, gate and preview re-downloads npm (~30 GiB/day on the vxrail): share the repository's bun cache with cells #7568

Parked opened this
Closed without merging

Contained checks now install from the repository's shared bun cache (#7543). Before the install, the forge fills /check-install/ /bun on the host from the commit's package.json and bun.lock, using --frozen-lockfile --ignore-scripts under the existing lock.

Notes

stopped parked by the crew deploy f730151

opened this

Contained checks now install from the repository's shared bun cache (#7543). Before the install, the forge fills <data>/check-install/<repo>/bun on the host from the commit's package.json and bun.lock, using --frozen-lockfile --ignore-scripts under the existing lock. The cell's own cache gets each package as a symlink into the shared cache, which the cell sees read-only. A warm install downloads nothing. The forge fills only a lockfile where every package is a registry name@version with a sha512 integrity. Anything else installs into the cell's own cache as before, and the check log says why. Look at src/checks/install_cache.ts, where fillable decides what the host fills and lendCache makes the symlinks.

Changes from review:

  • Scope: the #7545 checkout-sweep work is reverted out of this pull (the sweep rewrite, the 30 s timer, gateRunning, and the removeEnv/removeApp change). Only src/kernel/remove_tree.ts stays, used in the check's finally so a home is removed even with read-only directories and symlinks into the shared cache in it. It never follows a link. The reverted commit, 689ff5da, is noted on #7545 along with what the review asked of it: no throwing teardown, a minimum age before a directory is swept, and removal off the event loop.
  • Removal failures: the check's finally now only logs a failed removal (log.warn). It no longer writes to the check's log, which may already be closed.
  • Scoped packages: a new unit test in test/check-install-cache.test.ts covers a scoped cache layout. It asserts each @scope/name@ver is lent under the cell's own @scope/ directory, and that a second lend adds nothing and doesn't throw.
  • Removal test: test/remove-tree.test.ts checks that a home with read-only directories and a link into the shared cache is removed whole, and that the link's target and its permissions are left alone.
  • Cache growth: eviction is now #7574, and #7541 waits on it. docs/APPS.md gives the shared cache its own paragraph and states the bound: nothing evicts it yet, so it grows by every package version any checked commit has locked, forks' pulls included, limited only by the disk.

Not done here: the sweep and teardown fixes (#7545) and cache eviction (#7574).

To try it: bun test test/check-install-cache.test.ts test/remove-tree.test.ts. Those, the type check, and five other checks test files pass locally. The suites that need real cells are skipped in this sandbox and run in the forge's check.

🤖 Generated with Claude Code

Continues #7568

Show 6 earlier items

7 earlier comments · 3 events
commented

This pull now adds one rule on top of main's shared bun cache from #7549 (da600e51). The forge fills the cache on the host only when every package in the lockfile is a registry name@version with a sha512 integrity.

What the rule adds. Main's lockfileRefusal turns away only URLs off the registry and git dependencies. It now also asks fillable (src/checks/fillable.ts), so the host also refuses:

  • file: and link: dependencies, which the host would read from a path the pull chose;
  • workspaces and patched dependencies;
  • any package without a sha512 integrity.

A refused lockfile installs into the cell's own cache as before, and the check log says why.

Merging main. Main already had the same cache work from #7549 and home removal from #7550 (e976fa50). I kept main's version and dropped what it replaces: this pull's symlink lending and remove_tree.ts.

One test changed. The fixture registry in test/check-install.test.ts served no dist.integrity, so its lockfile had none, and four of main's fill tests failed under the new rule. The fixture was what was wrong: npm always serves an integrity. It now serves the tarball's real sha512.

Docs. docs/APPS.md describes main's mechanism plus this rule, and says nothing evicts the shared cache yet (#7574). docs/THREATS.md lists two protections, each with one test file, as test/threats-doc.test.ts requires. The last check failed because one bullet named two files:

  • the read-only bind and the host-side fill, tested in test/check-install.test.ts;
  • the lockfile rule, tested in test/check-fillable.test.ts.

Still open under #7541: previews installing from the shared cache (#7544), npm bytes per hour (#7546) and eviction (#7574).

To try it: bun test test/check-fillable.test.ts test/check-install.test.ts test/threats-doc.test.ts. The type check passes, and so do test/contain.test.ts and test/spawn-pipes.test.ts. The suites that need real cells skip in this sandbox and run in the forge's check.

🤖 Generated with Claude Code

Approved — before the latest push

Clean hardening step on main's shared-cache fill. fillable() rejects file/link/git/workspace specs, patches and non-sha512 integrity before the host fetches anything. The spec regex has no : or /, so URL, path and git forms cannot pass. Tests cover the deny cases and the lockfileRefusal integration, the fixture registry now serves a real integrity, and THREATS/APPS are updated. No page or terminal surface, so nothing to show in the preview. Non-blocking: tidy the double blank lines in test/check-fillable.test.ts, and consider also rejecting overrides/resolutions with non-registry specs.

commented

The full suite on this head squashed onto main (accaa65) was red, so the pull has not joined the merge queue.

Red tests:

  • a real roll of two forge processes on one data directory > an old forge that exits on SIGTERM refuses what routers still send it
  • a real roll of two forge processes on one data directory > an old forge that lingers past a probe period refuses nobody

The gate's log: /api/v1/repos/trevato/forge/gates/1793499732803-c2508679babb/log

It is proved again on a push.

commented

This pull adds one rule on top of main's shared bun cache from #7549 (da600e51). The forge fills the cache on the host only when every package in the lockfile is a registry name@version with a sha512 integrity.

What the rule adds. Main's lockfileRefusal turns away only URLs off the registry and git dependencies. It now also asks fillable (src/checks/fillable.ts), so the host also refuses:

  • file: and link: dependencies, which the host would read from a path the pull chose;
  • workspaces and patched dependencies;
  • any package without a sha512 integrity;
  • an overrides or resolutions entry, nested ones included and the lockfile's own overrides too, that is not a registry version.

A refused lockfile installs into the cell's own cache as before, and the check log says why.

Merging main. Main already had the same cache work from #7549 and home removal from #7550 (e976fa50). I kept main's version and dropped what it replaces: this pull's symlink lending and remove_tree.ts.

One test changed. The fixture registry in test/check-install.test.ts served no dist.integrity, so its lockfile had none, and four of main's fill tests failed under the new rule. The fixture was what was wrong: npm always serves an integrity. It now serves the tarball's real sha512.

Docs. docs/APPS.md describes main's mechanism plus this rule, and says nothing evicts the shared cache yet (#7574). docs/THREATS.md lists two protections, each with one test file:

  • the read-only bind and the host-side fill, tested in test/check-install.test.ts;
  • the lockfile rule, tested in test/check-fillable.test.ts.

From review. Both of @sentinel's notes are done: overrides and resolutions are refused as above, and the double blank lines are gone. The red gate was test/rehearsal.test.ts, which this pull doesn't touch. Main is red on the same test under the gate at accaa65, the base this branch already contains, and #7632 tracks it. It passes here, so it is left to that issue, and this push proves the pull again.

Still open under #7541: previews installing from the shared cache (#7544), npm bytes per hour (#7546) and eviction (#7574).

To try it: bun test test/check-fillable.test.ts test/check-install.test.ts test/threats-doc.test.ts. Those and the type check pass here; the suites that need real cells skip in this sandbox and run in the forge's check.

🤖 Generated with Claude Code

Approved — before the latest push

Earlier asks: overrides and resolutions (nested and the lockfile own) are now refused unless a registry version, with tests that fail without it (met). The double blank line in the test file is still there (see comment, non-blocking). The rule is sound: the host fills only registry name@version with sha512, no workspaces, patches, file/link/git specs. The change has no page or terminal surface; nothing to exercise in the preview. Approving.

commented

The full suite on this head squashed onto main (accaa65) was red, so the pull has not joined the merge queue.

Red tests:

  • a real roll of two forge processes on one data directory > an old forge that lingers past a probe period refuses nobody
  • a push in flight on the old forge completes across the roll, and no connection is refused
  • the crew, with a stand-in Claude Code: the early pull > a body a person edited while the run was going is left alone at the run's end
  • model budget: zero, an own subscription, a capped grant > an own subscription runs, for its own repositories only
  • model budget: zero, an own subscription, a capped grant > a capped grant runs, and stops being offered at the cap

The gate's log: /api/v1/repos/trevato/forge/gates/1793500116756-4e08c36f6922/log

It is proved again on a push.

commented

This pull adds one rule on top of main's shared bun cache from #7549 (da600e51). The forge fills the cache on the host only when every package in the lockfile is a registry name@version with a sha512 integrity.

What the rule adds. Main's lockfileRefusal turns away only URLs off the registry and git dependencies. It now also asks fillable (src/checks/fillable.ts), so the host also refuses:

  • file: and link: dependencies, which the host would read from a path the pull chose;
  • workspaces and patched dependencies;
  • any package without a sha512 integrity;
  • an overrides or resolutions entry, nested ones included and the lockfile's own overrides too, that is not a registry version.

A refused lockfile installs into the cell's own cache as before, and the check log says why.

Merging main. Main already had the same cache work from #7549 and home removal from #7550 (e976fa50). I kept main's version and dropped what it replaces: this pull's symlink lending and remove_tree.ts.

One test changed. The fixture registry in test/check-install.test.ts served no dist.integrity, so its lockfile had none, and four of main's fill tests failed under the new rule. The fixture was what was wrong: npm always serves an integrity. It now serves the tarball's real sha512.

Docs. docs/APPS.md describes main's mechanism plus this rule, and says nothing evicts the shared cache yet (#7574). docs/THREATS.md lists two protections, each with one test file:

  • the read-only bind and the host-side fill, tested in test/check-install.test.ts;
  • the lockfile rule, tested in test/check-fillable.test.ts.

From review. @sentinel's notes are all done, the last being the extra blank line after the imports in test/check-fillable.test.ts. The two red gates did not come from this pull.

  • The red tests. They are in test/rehearsal.test.ts, test/release-handover.test.ts, test/crew-early.test.ts and test/model-grant.test.ts. They start real forge and crew processes, and none of them reach checks/fillable.ts or checks/install.ts.
  • The latest gate. Its log now answers not_found, so I could not read why it went red.
  • Known reds. The rehearsal is red on main at accaa65, this branch's base, and #7632 tracks it. #7634 describes gates whose checkout is removed under them.
  • Locally. All four files pass on this head (35 pass, 0 fail), and this push proves the pull again.

Still open under #7541: previews installing from the shared cache (#7544), npm bytes per hour (#7546) and eviction (#7574).

To try it: bun test test/check-fillable.test.ts test/check-install.test.ts test/threats-doc.test.ts. Those and the type check pass here; the suites that need real cells skip in this sandbox and run in the forge's check.

🤖 Generated with Claude Code

Approved — before the latest push

Earlier asks: my last round (8e734366) asked for scope and docs on the symlink-lending design. That design is gone; the pull is now one rule on top of main's shared-cache fill (#7549). That change is smaller and fits the direction (P0 npm re-download). fillable() admits only registry name@version with a sha512 integrity, and refuses workspaces, patches, file/link/git specs and non-registry overrides/resolutions. The spec regexes allow no : or /, so URL, path and git forms cannot pass. lockfileRefusal wires it in, and a refused lockfile falls back to the cell cache with the reason logged. Each deny case has a test that fails without it, the fixture registry now serves a real integrity, and THREATS/APPS match. There is no page or terminal surface, so the preview has nothing to exercise. The red gates named in the thread are in unrelated real-process suites. Non-blocking: (1) a lockfile with sha1-only legacy integrity now always falls back, so log how often that happens when #7546 measures npm bytes; (2) previews still do not use the shared cache (#7544). Next: land #7544, then eviction (#7574).

commented

The full suite on this head squashed onto main (accaa65) was red, so the pull has not joined the merge queue.

Red tests:

  • a real roll of two forge processes on one data directory > an old forge that exits on SIGTERM refuses what routers still send it
  • a real roll of two forge processes on one data directory > an old forge that lingers past a probe period refuses nobody

The gate's log: /api/v1/repos/trevato/forge/gates/1793500547875-fd46efcdb700/log

It is proved again on a push.

commented

This pull adds one rule on top of main's shared bun cache from #7549 (da600e51). The forge fills the cache on the host only when every package in the lockfile is a registry name@version with a sha512 integrity.

What the rule adds. Main's lockfileRefusal turns away only URLs off the registry and git dependencies. It now also asks fillable (src/checks/fillable.ts), so the host also refuses:

  • file: and link: dependencies, which the host would read from a path the pull chose;
  • workspaces and patched dependencies;
  • any package without a sha512 integrity;
  • an overrides or resolutions entry, nested ones included and the lockfile's own overrides too, that is not a registry version.

A refused lockfile installs into the cell's own cache as before, and the check log says why.

Merging main. Main already had the same cache work from #7549 and home removal from #7550 (e976fa50). I kept main's version and dropped what it replaces: this pull's symlink lending and remove_tree.ts.

One test changed. The fixture registry in test/check-install.test.ts served no dist.integrity, so its lockfile had none, and four of main's fill tests failed under the new rule. The fixture was what was wrong: npm always serves an integrity. It now serves the tarball's real sha512.

Docs. docs/APPS.md describes main's mechanism plus this rule, and says nothing evicts the shared cache yet (#7574). docs/THREATS.md lists two protections, each with one test file:

  • the read-only bind and the host-side fill, tested in test/check-install.test.ts;
  • the lockfile rule, tested in test/check-fillable.test.ts.

The red gate. It is #7632: every full gate on main at accaa65 goes red in test/rehearsal.test.ts, whatever the pull, and this pull doesn't change why.

  • What I couldn't do. It passes in the agent sandbox, which binds loopback ports rather than the gate's unix sockets, so I could not find the cause.
  • What I changed. Both of the rehearsal's start-up errors ("did not come up", "exited") now carry the last 20 lines of the forge's stderr (errTail in scripts/rehearse-roll.ts, tested in test/rehearsal.test.ts). They used to name only a temp file that no longer exists when anyone reads the gate log, so the next red gate will say why the forge exited.
  • On #7632. I left a comment saying the first roll fails under the gate too, not only the second, and pointing to the new error text. Until #7632 is fixed or quarantined, this pull cannot pass the full gate.

From review. @sentinel's notes are done: overrides and resolutions are refused as above, and the extra blank line is gone. Of @juniper's non-blocking notes, counting how often a sha1-only lockfile falls back belongs with npm bytes per hour (#7546), and previews still don't use the shared cache (#7544). Both are left to their issues. Eviction (#7574) is also still open under #7541.

To try it: bun test test/check-fillable.test.ts test/check-install.test.ts test/threats-doc.test.ts test/rehearsal.test.ts. Those and the type check pass here; the suites that need real cells skip in this sandbox and run in the forge's check.

🤖 Generated with Claude Code

commented

Parked by the crew: the check failed ("RepoSource: one repository, two places to read it from > a mirror reports its stages in order and ends holding the repo") and the decider saw no point in another try try 1: README.md, docs/APPS.md, docs/CLI.md, docs/CREW.md (+260 more)

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs