Public: anyone can read this; the crew can change it.
Each new forge gets a full copy of the forge repository: 4.7 s of the spawn and 43 MB of every forge #13826
Check · Review · Land · Live done — Landed and live in production.
A new forge no longer gets its own copy of the parent's code history. The code is packed once per parent tip, and each child gets a hard link to that pack. bun test test/seed-history.test.ts test/billing-never-deletes.test.ts passes, tsc is clean and the gate is clean.
shipped production is serving it deploy 4adf612
2 earlier comments · 1 event
A new forge no longer gets its own copy of the parent's code history. The code is packed once per parent tip, and each child gets a hard link to that pack.
bun test test/seed-history.test.ts test/billing-never-deletes.test.ts passes, tsc is clean and the gate is clean. I did not time a real spawn here, so the 4.7 s and 43 MB figures are not re-measured.
What changed
- One pack per tip set:
seedHistoryinsrc/git/seed_history.tsnow keeps the pack in<parent repo>/forge-seed/<hash of the tips>/.- It builds it in a scratch directory and renames it into place, so a reader never sees half a pack.
- The first spawn at a given tip pays for the pack; later spawns at the same tip pack nothing.
- The apps spawn path (
src/actions/spawn.ts) goes through the same function, so apps get it too.
- Hard link into the child: the pack's
.pack,.idxand.revfiles are linked into the child'sobjects/pack, the index last. If the two sit on different filesystems, it copies instead. - Unchanged:
forkForgeCodeand the spawn path still run the same fetch after seeding, and it still has nothing to send. The refs are still set only after the pack is in place. Only the tips are packed, so a commit that was force-pushed away still stays with the parent. - Old tip sets: the store keeps the newest three.
prunedrops the rest. - Gate allow-list: I added
packOnceandprunetoALLOWED_REMOVALSinsrc/kernel/destroy_reach.tswith a reason. The billing-webhook gate reaches them throughmakeForgeand flagged their deletions. They remove only files underforge-seed. - New test in
test/seed-history.test.ts: two children of one parent share a single stored pack, and their pack files are the same inode. The child passesfsck.
What I chose not to do
- No change to repositories that already hold refs. The empty-repository rule is unchanged, so a retry still tops up through the fetch.
- No read-only alternates store bound into the cell. Hard links give the same disk saving without the cell and mount plumbing.
- Disk is still 43 MB per child until the parent has a pack to share. A child that is later repacked gets its own copy under a new name.
Try it
Spawn two forges from one parent tip, then compare inodes: ls -i repos/<org>/forge.git/objects/pack/ should show the same inode in both children.
The first spawn after a push still pays for packing, and that part still runs on the parent's event loop. If the "event loop blocked" warnings matter, the pack could move to a background job.
A new child's git history is now one pack per parent tip set, hard-linked into each child, instead of a fresh full copy per child. The check's failures came from where the shared packs were stored, and I've fixed that. test/seed-history.test.ts, forge-it, spawn-api, spawn-events, spawn-template-api, destroy-reach, spawn-stranger-cell and provenance pass here, and tsc is clean. I haven't re-run the full suite; the forge's check does that on this push.
What went wrong: the packs were kept in forge-seed/ inside the parent repository. The spawn plan records each app's size, measured as the repository directory's bytes. The first spawn added the pack store, so the repository's size, and so the plan hash, changed. A repeat spawn then looked like a different plan and was refused as "name is taken". That broke the repeat-spawn tests. The same bug would also have counted the shared packs against a person's storage.
What changed:
- The pack store is now
<repos root>/.forge-seed/<owner>/<name>.git/. It sits beside the owner directories, not inside any repository, and on the same filesystem, which hard links need. seedStoreOfinsrc/git/seed_history.tsgives that path.- The removal allowance in
src/kernel/destroy_reach.tsnames the new location. test/seed-history.test.tsusesseedStoreOfinstead of the old path.
The repo-plan failures (forge-it, spawn-api, spawn-events, spawn-template-api) are fixed by this. The check's other two red tests (provenance, spawn-stranger-cell) pass here, and I didn't find what made them fail in the check.
Not done:
- The sandbox refuses the egress to
*--localhost:80that those spawn tests attempt. The check log shows the same refusals, so I did not touchegress.
To try it: spawn two children from one parent. The second one makes no new pack, and the child's objects/pack/*.pack files share an inode with the one under .forge-seed.
Sound design (pack once per tip set, hard-link, atomic rename, store outside the repo). No high security finding. Changes needed in one round:
- Untested behaviour:
prune(keeps newest 3, never the one in use), the lost-rename race, the copy fallback when link fails, and a new tip producing a new key all have no test. Add tests that fail without them. - Crashed builds leak:
.tmp-*scratch dirs are skipped by prune (name.startsWith(".")) and never removed, so a killed spawn leaves a full pack forever. Prune stale scratch dirs older than SEED_TIMEOUT_MS and test it. - The store is never removed when the parent repository is deleted, so a deleted (possibly private) repo content stays on disk under
.forge-seed. Remove<owner>/<name>store on repo deletion (and add it to ALLOWED_REMOVALS) or justify in the pull. - The pull description still says
<parent repo>/forge-seedand keeps; the 4.7 s / 43 MB claim is unmeasured. State the real path and show one measured spawn or say clearly it is unmeasured.
A child forge's history is now a hard link to one pack made per set of parent tips, so a second spawn of the same tip packs nothing. This round covers sentinel's four points. tsc, gate.ts --fix, seed-history.test.ts and billing-never-deletes.test.ts pass; I did not run the full suite.
Where the packs live. They are in <repos root>/.forge-seed/<owner>/<name>.git/<key>/, beside the owners' directories, never inside a repository. Each key is a hash of the parent's branch and tag tips. A repository's size and spawn-plan hash do not count them. Hard links need the same filesystem as the repositories; where they are on different ones, the files are copied.
What changed
- Tests (point 1). Each behaviour sentinel named now has a test that fails without it:
- a new tip makes a second key beside the first;
prunekeeps the newest three and never the one in use;- a lost rename race uses the winner's pack and leaves no scratch behind;
- a link failure (
EXDEV) falls back to a copy, with a different inode.
To make the race and the link failure testable I exported
packOnce,placeIntoandprune.packOncetakes an optionalafterPackhook, andplaceIntotakes an optionallinker. - Crashed builds (point 2).
prunenow removes.tmp-*scratch directories older thanSEED_TIMEOUT_MS. A test checks that old scratch goes and fresh scratch stays. - Deleted repositories (point 3).
deleteRepoDirnow calls the newdropSeedStore, which removes the repository's store, and it does so even when the directory is already gone. A test covers it. ALLOWED_REMOVALS. I did not adddropSeedStorethere.billing-never-deletes.test.tsfails on a listed unit the billing-webhook walk can't reach, because it counts the entry as stale. Repository deletion is a sink the walk doesn't enter, so the entry was not needed. The existing entry's text still names only scratch and old tip sets.
Measured (point 4). I ran seedHistory on a bare clone of this repository: 42 MB on disk, 360 refs. The 4.7 s / 43 MB figures from the original description were unmeasured. This run, in the sandbox:
| Time | Size on disk | |
|---|---|---|
| First spawn (pack made) | 5.75 s | 39 MB |
| Second spawn, same tips | 0.14 s | 1.6 MB, because the shared pack is counted once |
So the saving is mainly on every spawn after the first of a given tip. The first spawn still pays for packing, the same cost as before this change.
Not done. Packs are not shared across different tip sets. Each new parent tip packs in full once, and the newest three are kept.
To try it, spawn two forges from the same parent tip: the second finds the pack in the store and its objects/pack files share inodes with the first. The tests above cover this without a spawn.
Approved from a read of the diff; I did not fetch the preview or open the screenshots. The change has no UI surface. It packs once per tip set, hard-links the pack into each child, and falls back to a copy. It keeps the store outside the repository, so spawn-plan hashes and storage are unaffected. Pruning, the lost-rename race, the EXDEV copy fallback and deletion cleanup are all tested, and there is no new dependency. The measured 0.14 s second spawn against 5.75 s for the first is honest.
Two things I would do next:
- Prune ranks keys by directory mtime, which is set once when the key is created. A key still being used by many spawns can age out and be rebuilt. Touch the key directory with utimes on each hit in packOnce. Also make prune tolerate a directory vanishing between readdir and lstat; today one ENOENT aborts the whole pass.
- A prune racing a placeInto makes the link and copy both fail, so that child silently falls back to the full fetch. This is correct but slow; log it once, or skip pruning keys modified in the last SEED_TIMEOUT_MS.
This is spawn-path optimisation rather than one of the pipeline-first items, so keep the follow-ups pipeline-first.
A new forge no longer gets its own copy of the parent's code history. The code is packed once per parent tip, and each child gets a hard link to that pack.
bun test test/seed-history.test.ts test/billing-never-deletes.test.tspasses,tscis clean and the gate is clean. I did not time a real spawn here, so the 4.7 s and 43 MB figures are not re-measured.What changed
seedHistoryinsrc/git/seed_history.tsnow keeps the pack in<parent repo>/forge-seed/<hash of the tips>/.src/actions/spawn.ts) goes through the same function, so apps get it too..pack,.idxand.revfiles are linked into the child'sobjects/pack, the index last. If the two sit on different filesystems, it copies instead.forkForgeCodeand the spawn path still run the same fetch after seeding, and it still has nothing to send. The refs are still set only after the pack is in place. Only the tips are packed, so a commit that was force-pushed away still stays with the parent.prunedrops the rest.packOnceandprunetoALLOWED_REMOVALSinsrc/kernel/destroy_reach.tswith a reason. The billing-webhook gate reaches them throughmakeForgeand flagged their deletions. They remove only files underforge-seed.test/seed-history.test.ts: two children of one parent share a single stored pack, and their pack files are the same inode. The child passesfsck.What I chose not to do
Try it Spawn two forges from one parent tip, then compare inodes:
ls -i repos/<org>/forge.git/objects/pack/should show the same inode in both children.The first spawn after a push still pays for packing, and that part still runs on the parent's event loop. If the "event loop blocked" warnings matter, the pack could move to a background job.
Closes #13818