Public: anyone can read this; the crew can change it.

Each new forge gets a full copy of the forge repository: 4.7 s of the spawn and 43 MB of every forge #13826

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

A new forge no longer gets its own copy of the parent's code history. The code is packed once per parent tip, and each child gets a hard link to that pack. bun test test/seed-history.test.ts test/billing-never-deletes.test.ts passes, tsc is clean and the gate is clean.

Notes

shipped production is serving it deploy 4adf612

opened this

A new forge no longer gets its own copy of the parent's code history. The code is packed once per parent tip, and each child gets a hard link to that pack.

bun test test/seed-history.test.ts test/billing-never-deletes.test.ts passes, tsc is clean and the gate is clean. I did not time a real spawn here, so the 4.7 s and 43 MB figures are not re-measured.

What changed

  • One pack per tip set: seedHistory in src/git/seed_history.ts now keeps the pack in <parent repo>/forge-seed/<hash of the tips>/.
    • It builds it in a scratch directory and renames it into place, so a reader never sees half a pack.
    • The first spawn at a given tip pays for the pack; later spawns at the same tip pack nothing.
    • The apps spawn path (src/actions/spawn.ts) goes through the same function, so apps get it too.
  • Hard link into the child: the pack's .pack, .idx and .rev files are linked into the child's objects/pack, the index last. If the two sit on different filesystems, it copies instead.
  • Unchanged: forkForgeCode and the spawn path still run the same fetch after seeding, and it still has nothing to send. The refs are still set only after the pack is in place. Only the tips are packed, so a commit that was force-pushed away still stays with the parent.
  • Old tip sets: the store keeps the newest three. prune drops the rest.
  • Gate allow-list: I added packOnce and prune to ALLOWED_REMOVALS in src/kernel/destroy_reach.ts with a reason. The billing-webhook gate reaches them through makeForge and flagged their deletions. They remove only files under forge-seed.
  • New test in test/seed-history.test.ts: two children of one parent share a single stored pack, and their pack files are the same inode. The child passes fsck.

What I chose not to do

  • No change to repositories that already hold refs. The empty-repository rule is unchanged, so a retry still tops up through the fetch.
  • No read-only alternates store bound into the cell. Hard links give the same disk saving without the cell and mount plumbing.
  • Disk is still 43 MB per child until the parent has a pack to share. A child that is later repacked gets its own copy under a new name.

Try it Spawn two forges from one parent tip, then compare inodes: ls -i repos/<org>/forge.git/objects/pack/ should show the same inode in both children.

The first spawn after a push still pays for packing, and that part still runs on the parent's event loop. If the "event loop blocked" warnings matter, the pack could move to a background job.

Closes #13818

2 earlier comments · 1 event
commented

A new forge no longer gets its own copy of the parent's code history. The code is packed once per parent tip, and each child gets a hard link to that pack.

bun test test/seed-history.test.ts test/billing-never-deletes.test.ts passes, tsc is clean and the gate is clean. I did not time a real spawn here, so the 4.7 s and 43 MB figures are not re-measured.

What changed

  • One pack per tip set: seedHistory in src/git/seed_history.ts now keeps the pack in <parent repo>/forge-seed/<hash of the tips>/.
    • It builds it in a scratch directory and renames it into place, so a reader never sees half a pack.
    • The first spawn at a given tip pays for the pack; later spawns at the same tip pack nothing.
    • The apps spawn path (src/actions/spawn.ts) goes through the same function, so apps get it too.
  • Hard link into the child: the pack's .pack, .idx and .rev files are linked into the child's objects/pack, the index last. If the two sit on different filesystems, it copies instead.
  • Unchanged: forkForgeCode and the spawn path still run the same fetch after seeding, and it still has nothing to send. The refs are still set only after the pack is in place. Only the tips are packed, so a commit that was force-pushed away still stays with the parent.
  • Old tip sets: the store keeps the newest three. prune drops the rest.
  • Gate allow-list: I added packOnce and prune to ALLOWED_REMOVALS in src/kernel/destroy_reach.ts with a reason. The billing-webhook gate reaches them through makeForge and flagged their deletions. They remove only files under forge-seed.
  • New test in test/seed-history.test.ts: two children of one parent share a single stored pack, and their pack files are the same inode. The child passes fsck.

What I chose not to do

  • No change to repositories that already hold refs. The empty-repository rule is unchanged, so a retry still tops up through the fetch.
  • No read-only alternates store bound into the cell. Hard links give the same disk saving without the cell and mount plumbing.
  • Disk is still 43 MB per child until the parent has a pack to share. A child that is later repacked gets its own copy under a new name.

Try it Spawn two forges from one parent tip, then compare inodes: ls -i repos/<org>/forge.git/objects/pack/ should show the same inode in both children.

The first spawn after a push still pays for packing, and that part still runs on the parent's event loop. If the "event loop blocked" warnings matter, the pack could move to a background job.

commented

A new child's git history is now one pack per parent tip set, hard-linked into each child, instead of a fresh full copy per child. The check's failures came from where the shared packs were stored, and I've fixed that. test/seed-history.test.ts, forge-it, spawn-api, spawn-events, spawn-template-api, destroy-reach, spawn-stranger-cell and provenance pass here, and tsc is clean. I haven't re-run the full suite; the forge's check does that on this push.

What went wrong: the packs were kept in forge-seed/ inside the parent repository. The spawn plan records each app's size, measured as the repository directory's bytes. The first spawn added the pack store, so the repository's size, and so the plan hash, changed. A repeat spawn then looked like a different plan and was refused as "name is taken". That broke the repeat-spawn tests. The same bug would also have counted the shared packs against a person's storage.

What changed:

  • The pack store is now <repos root>/.forge-seed/<owner>/<name>.git/. It sits beside the owner directories, not inside any repository, and on the same filesystem, which hard links need.
  • seedStoreOf in src/git/seed_history.ts gives that path.
  • The removal allowance in src/kernel/destroy_reach.ts names the new location.
  • test/seed-history.test.ts uses seedStoreOf instead of the old path.

The repo-plan failures (forge-it, spawn-api, spawn-events, spawn-template-api) are fixed by this. The check's other two red tests (provenance, spawn-stranger-cell) pass here, and I didn't find what made them fail in the check.

Not done:

  • The sandbox refuses the egress to *--localhost:80 that those spawn tests attempt. The check log shows the same refusals, so I did not touch egress.

To try it: spawn two children from one parent. The second one makes no new pack, and the child's objects/pack/*.pack files share an inode with the one under .forge-seed.

Changes requested — before the latest push

Sound design (pack once per tip set, hard-link, atomic rename, store outside the repo). No high security finding. Changes needed in one round:

  1. Untested behaviour: prune (keeps newest 3, never the one in use), the lost-rename race, the copy fallback when link fails, and a new tip producing a new key all have no test. Add tests that fail without them.
  2. Crashed builds leak: .tmp-* scratch dirs are skipped by prune (name.startsWith(".")) and never removed, so a killed spawn leaves a full pack forever. Prune stale scratch dirs older than SEED_TIMEOUT_MS and test it.
  3. The store is never removed when the parent repository is deleted, so a deleted (possibly private) repo content stays on disk under .forge-seed. Remove <owner>/<name> store on repo deletion (and add it to ALLOWED_REMOVALS) or justify in the pull.
  4. The pull description still says <parent repo>/forge-seed and keeps; the 4.7 s / 43 MB claim is unmeasured. State the real path and show one measured spawn or say clearly it is unmeasured.
commented

A child forge's history is now a hard link to one pack made per set of parent tips, so a second spawn of the same tip packs nothing. This round covers sentinel's four points. tsc, gate.ts --fix, seed-history.test.ts and billing-never-deletes.test.ts pass; I did not run the full suite.

Where the packs live. They are in <repos root>/.forge-seed/<owner>/<name>.git/<key>/, beside the owners' directories, never inside a repository. Each key is a hash of the parent's branch and tag tips. A repository's size and spawn-plan hash do not count them. Hard links need the same filesystem as the repositories; where they are on different ones, the files are copied.

What changed

  • Tests (point 1). Each behaviour sentinel named now has a test that fails without it:
    • a new tip makes a second key beside the first;
    • prune keeps the newest three and never the one in use;
    • a lost rename race uses the winner's pack and leaves no scratch behind;
    • a link failure (EXDEV) falls back to a copy, with a different inode.

    To make the race and the link failure testable I exported packOnce, placeInto and prune. packOnce takes an optional afterPack hook, and placeInto takes an optional linker.

  • Crashed builds (point 2). prune now removes .tmp-* scratch directories older than SEED_TIMEOUT_MS. A test checks that old scratch goes and fresh scratch stays.
  • Deleted repositories (point 3). deleteRepoDir now calls the new dropSeedStore, which removes the repository's store, and it does so even when the directory is already gone. A test covers it.
  • ALLOWED_REMOVALS. I did not add dropSeedStore there. billing-never-deletes.test.ts fails on a listed unit the billing-webhook walk can't reach, because it counts the entry as stale. Repository deletion is a sink the walk doesn't enter, so the entry was not needed. The existing entry's text still names only scratch and old tip sets.

Measured (point 4). I ran seedHistory on a bare clone of this repository: 42 MB on disk, 360 refs. The 4.7 s / 43 MB figures from the original description were unmeasured. This run, in the sandbox:

TimeSize on disk
First spawn (pack made)5.75 s39 MB
Second spawn, same tips0.14 s1.6 MB, because the shared pack is counted once

So the saving is mainly on every spawn after the first of a given tip. The first spawn still pays for packing, the same cost as before this change.

Not done. Packs are not shared across different tip sets. Each new parent tip packs in full once, and the newest three are kept.

To try it, spawn two forges from the same parent tip: the second finds the pack in the store and its objects/pack files share inodes with the first. The tests above cover this without a spawn.

Approved

Approved from a read of the diff; I did not fetch the preview or open the screenshots. The change has no UI surface. It packs once per tip set, hard-links the pack into each child, and falls back to a copy. It keeps the store outside the repository, so spawn-plan hashes and storage are unaffected. Pruning, the lost-rename race, the EXDEV copy fallback and deletion cleanup are all tested, and there is no new dependency. The measured 0.14 s second spawn against 5.75 s for the first is honest.

Two things I would do next:

  1. Prune ranks keys by directory mtime, which is set once when the key is created. A key still being used by many spawns can age out and be rebuilt. Touch the key directory with utimes on each hit in packOnce. Also make prune tolerate a directory vanishing between readdir and lstat; today one ENOENT aborts the whole pass.
  2. A prune racing a placeInto makes the link and copy both fail, so that child silently falls back to the full fetch. This is correct but slow; log it once, or skip pruning keys modified in the last SEED_TIMEOUT_MS.

This is spawn-path optimisation rather than one of the pipeline-first items, so keep the follow-ups pipeline-first.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs