trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns15g rEventsFlows

P0 efficiency: every contained check, gate and preview re-downloads npm (~30 GiB/day on the vxrail): share the repository's bun cache with cells #8372

Merged torquecrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/7541-p0-efficiency-every-contained-check-gate → main 13 commits fork 1 thread · 1 unresolved

  • Merged. torque/7541-p0-efficiency-every-contained-check-gate was merged into main as 5475310.

Review

Sign in to review this change.
9 files changed +357 −49
docs/APPS.md +26 −13
@@ -713,20 +713,33 @@ queued again, and the forge that carries on runs it from the start.
713713 or `ECONNRESET` is tried once more after 2 s. If it fails that way again,
714714 the check is `failed: infrastructure`. It is queued again, and the merge
715715 queue counts it as cut off, not red. If the re-run fails the same way, it
716 fails and says `infrastructure`. A check in a cell installs into its own
717 home, so its bun cache is shared with nothing. An uncontained check uses
718 its repository's cache (`<data>/check-install/<repo>/bun`), and installs
719 into it take turns under a lock file, so two never extract one package
720 into the same directory. That cache is bounded (`src/checks/install_evict.ts`,
721 #7574): a fill, or an uncontained install, marks the packages its `bun.lock`
722 names as lent, and under the same lock, in the background and at most hourly,
723 the forge removes entries not lent for 30 days (`FORGE_CHECK_CACHE_DAYS`), then
724 the oldest until the repository's cache is under 2 GB (`FORGE_CHECK_CACHE_MB`).
716 fails and says `infrastructure`. An uncontained check uses its
717 repository's cache (`<data>/check-install/<repo>/bun`), and installs into
718 it take turns under a lock file, so two never extract one package into the
719 same directory. A `job` check installs in its own pod with its own cache
720 and is not retried.
721
722 **A contained check installs from the shared cache**
723 (`src/checks/install.ts`, #7543). A check in a cell reads its repository's
724 cache and never writes it: before the install, the forge fills it on the
725 host from a scratch copy of the commit's checkout (no `bunfig.toml` or
726 `.npmrc` of the pull's, `--frozen-lockfile --ignore-scripts`, under the
727 same lock), and binds it read-only into the cell, which installs from it
728 copying. A warm install downloads nothing. The forge fills only a lockfile
729 whose every package is a plain `name@version` from the pinned registry
730 with a sha512 integrity (`src/checks/fillable.ts`); anything else
731 (workspaces, git, file or tarball URLs, patches, an override or
732 resolution to any of those, a binary `bun.lockb`)
733 installs into the cell's own cache as before, and the log says why.
734
735 That cache is bounded (`src/checks/install_evict.ts`, #7574): a fill, or an
736 uncontained install, marks the packages its `bun.lock` names as lent, and
737 under the same lock, in the background and at most hourly, the forge
738 removes entries not lent for 30 days (`FORGE_CHECK_CACHE_DAYS`), then the
739 oldest until the repository's cache is under 2 GB (`FORGE_CHECK_CACHE_MB`).
725740 An evicted package is fetched again by the next install. Only the text
726 `bun.lock` is read: a repository on a binary `bun.lockb` lends nothing, so its
727 packages age out after 30 days and are fetched again (a cell never fills from
728 one at all). A `job` check installs in its own pod and is not
729 retried.
741 `bun.lock` is read: a repository on a binary `bun.lockb` lends nothing, so
742 its packages age out after 30 days and are fetched again.
730743
731744 **An app environment installs from that same cache** (`src/apps/process.ts`,
732745 #7544). Before a cold checkout's `install`, production or a preview, the
docs/THREATS.md +15
@@ -94,6 +94,21 @@ person approved.
9494 and only this repository's directory re-allowed, so a cache made later is
9595 denied too, and deleting the repository removes it.
9696 Test: `test/crew-install-cache.test.ts`
97 - A check's cell reads its repository's shared bun cache
98 (`check-install/<repo id>/bun`) through a read-only bind and cannot write
99 it, so one pull cannot plant a package the gate on main installs. The
100 forge fills it on the host with `--ignore-scripts`, and reads no
101 `bunfig.toml` or `.npmrc` of the pull's.
102 Test: `test/check-install.test.ts`
103 - An app environment's cell (#7544) reads that same cache read-only and
104 copies from it, so a preview cannot plant a package production installs;
105 the forge fills it through the same lockfile rule as a check's.
106 Test: `test/app-install-cache.test.ts`
107 - The forge fills that cache only from a lockfile whose every package is a
108 registry `name@version` with a sha512 integrity (`checks/fillable.ts`):
109 the host never fetches a URL a pull chose and reads no file, link or
110 workspace a pull points at, an override's or resolution's included.
111 Test: `test/check-fillable.test.ts`
97112 - The same policy holds against a real bubblewrap, not only as a string.
98113 Test: `test/crew-sandbox-real.test.ts`
99114 - An agent cannot push to the default branch or merge; a person merges.
src/checks/fillable.ts added +96
@@ -0,0 +1,96 @@
1 /**
2 * Whether the forge may fill a repository's shared bun cache from a commit's
3 * lockfile (#7543, #7541). The fill runs on the host, outside a cell's
4 * egress (`checks/install.ts`, `prefillInstallCache`), so a lockfile is
5 * filled only when every package in it is a plain `name@version` from the
6 * pinned registry with a sha512 integrity bun checks: no tarball URL, git,
7 * file, link or workspace dependency, and no patches. Anything else installs
8 * as before, into the cell's own cache.
9 */
10
11 import { err, ok, type Result } from "../kernel/result.ts";
12
13 /** `[email protected]` or `@scope/[email protected]+build`: a registry package at one version. */
14 const REGISTRY_ID = /^(@[a-z0-9][\w.-]*\/)?[a-z0-9][\w.-]*@\d+\.\d+\.\d+[\w.+-]*$/i;
15
16 /** A dependency range or tag, optionally an `npm:` alias: never a path, URL, git or workspace spec. */
17 const REGISTRY_SPEC = /^(npm:(@[\w.-]+\/)?[\w.-]+@)?[\w\s.^~<>=|*+-]*$/;
18
19 const DEPENDENCY_FIELDS = ["dependencies", "devDependencies", "optionalDependencies", "peerDependencies"];
20
21 /**
22 * Whether the forge may fill the shared cache from this `package.json` and
23 * `bun.lock`: an Err names what keeps it out. Every package must come from
24 * `registry` by name and version, with a sha512 integrity bun checks.
25 */
26 export function fillable(packageJson: string, lock: string, registry: string): Result<void, string> {
27 let manifest: unknown;
28 let parsed: unknown;
29 try {
30 manifest = JSON.parse(packageJson);
31 parsed = Bun.JSONC.parse(lock);
32 } catch {
33 return err("package.json or bun.lock does not parse");
34 }
35 if (!isRecord(manifest) || !isRecord(parsed)) return err("package.json or bun.lock is not an object");
36 if (manifest["workspaces"] !== undefined) return err("workspaces");
37 for (const field of DEPENDENCY_FIELDS) {
38 const deps = manifest[field];
39 if (deps === undefined) continue;
40 if (!isRecord(deps)) return err(`${field} is not an object`);
41 for (const [name, spec] of Object.entries(deps))
42 if (typeof spec !== "string" || !REGISTRY_SPEC.test(spec))
43 return err(`${name} is not a registry dependency`);
44 }
45 if (parsed["patchedDependencies"] !== undefined) return err("patched dependencies");
46 // An override replaces a package wherever it is locked: a path, URL or git
47 // spec in one is as much the host's fetch as a dependency's would be.
48 for (const [where, field] of [[manifest, "overrides"], [manifest, "resolutions"], [parsed, "overrides"]] as const) {
49 const bad = field in where ? nonRegistry(where[field]) : undefined;
50 if (bad !== undefined) return err(`${field}: ${bad} is not a registry version`);
51 }
52 const workspaces = parsed["workspaces"];
53 if (!isRecord(workspaces) || Object.keys(workspaces).some((key) => key !== ""))
54 return err("workspaces");
55 const packages = parsed["packages"] ?? {};
56 if (!isRecord(packages)) return err("bun.lock has no packages");
57 let origin: string;
58 try {
59 origin = new URL(registry).origin;
60 } catch {
61 return err("the registry is not a URL");
62 }
63 for (const [key, entry] of Object.entries(packages)) {
64 if (!Array.isArray(entry) || entry.length !== 4) return err(`${key} is not a registry package`);
65 const [id, url, , integrity] = entry as unknown[];
66 if (typeof id !== "string" || !REGISTRY_ID.test(id)) return err(`${key} is not a registry package`);
67 if (url !== "" && !(typeof url === "string" && sameOrigin(url, origin)))
68 return err(`${key} comes from another registry`);
69 if (typeof integrity !== "string" || !integrity.startsWith("sha512-"))
70 return err(`${key} has no sha512 integrity`);
71 }
72 return ok(undefined);
73 }
74
75 /** The first override spec that is not a registry range, nested ones included; undefined when all are. */
76 function nonRegistry(value: unknown, name = "overrides"): string | undefined {
77 if (typeof value === "string") return REGISTRY_SPEC.test(value) ? undefined : name;
78 if (!isRecord(value)) return name;
79 for (const [key, inner] of Object.entries(value)) {
80 const bad = nonRegistry(inner, key);
81 if (bad !== undefined) return bad;
82 }
83 return undefined;
84 }
85
86 function sameOrigin(url: string, origin: string): boolean {
87 try {
88 return new URL(url).origin === origin;
89 } catch {
90 return false;
91 }
92 }
93
94 function isRecord(value: unknown): value is Record<string, unknown> {
95 return typeof value === "object" && value !== null && !Array.isArray(value);
96 }
src/checks/install.ts +6 −1
@@ -21,6 +21,7 @@ import { tmpdir } from "node:os";
2121 import { dirname, join } from "node:path";
2222 import { formatBytes } from "../core/storage.ts";
2323 import { err, ok, type Result } from "../kernel/result.ts";
24 import { fillable } from "./fillable.ts";
2425 import { lend } from "./install_evict.ts";
2526
2627 /** What an install prints when the machine, not the code, failed it. */
@@ -109,7 +110,11 @@ export function lockfileRefusal(dir: string, registry: string): string | undefin
109110 if (!url.startsWith(registry)) return `the lockfile names ${url.slice(0, 80)}, not the registry`;
110111 if (/[@"'/](?:github|gitlab|bitbucket|git|ssh|git\+\w+):/i.test(text))
111112 return "the lockfile has a git dependency";
112 return undefined;
113 // Every package by name and version with a sha512 integrity, or no fill:
114 // a file, link or workspace dependency, or a patch, is read on the host.
115 const manifest = existsSync(join(dir, "package.json")) ? readFileSync(join(dir, "package.json"), "utf8") : "{}";
116 const strict = fillable(manifest, text, registry);
117 return strict.ok ? undefined : strict.error;
113118 }
114119
115120 /**
test/app-install-cache.test.ts +2 −31
@@ -12,6 +12,7 @@ import { tmpdir } from "node:os";
1212 import { join } from "node:path";
1313 import type { AppManifest } from "../src/apps/manifest.ts";
1414 import { installPlan, startProcess, type StartInput } from "../src/apps/process.ts";
15 import { fixtureRegistry } from "./helpers/fixture-registry.ts";
1516
1617 setDefaultTimeout(60_000);
1718
@@ -29,37 +30,7 @@ const manifest: AppManifest = {
2930 const SERVER = `Bun.serve({ port: Number(process.env.PORT), fetch: () => new Response("ok") });\n`;
3031
3132 /** A registry on loopback serving one package, `[email protected]`; counts its requests. */
32 async function registry(dir: string): Promise<{ url: string; hits: string[]; stop: () => void }> {
33 mkdirSync(join(dir, "pkg", "package"), { recursive: true });
34 writeFileSync(join(dir, "pkg", "package", "package.json"), JSON.stringify({ name: "fixture-pkg", version: "1.0.0" }));
35 const made = Bun.spawnSync(["tar", "-czf", join(dir, "fixture.tgz"), "-C", join(dir, "pkg"), "package"]);
36 if (made.exitCode !== 0) throw new Error("tar is needed for the fixture registry");
37 const tarball = new Uint8Array(await Bun.file(join(dir, "fixture.tgz")).arrayBuffer());
38 const hits: string[] = [];
39 const server: ReturnType<typeof Bun.serve> = Bun.serve({
40 port: 0,
41 hostname: "127.0.0.1",
42 fetch(req): Response {
43 const path = new URL(req.url).pathname;
44 hits.push(path);
45 if (path === "/fixture-pkg/-/fixture-pkg-1.0.0.tgz") return new Response(tarball);
46 if (path === "/fixture-pkg")
47 return Response.json({
48 name: "fixture-pkg",
49 "dist-tags": { latest: "1.0.0" },
50 versions: {
51 "1.0.0": {
52 name: "fixture-pkg",
53 version: "1.0.0",
54 dist: { tarball: `http://127.0.0.1:${server.port}/fixture-pkg/-/fixture-pkg-1.0.0.tgz` },
55 },
56 },
57 });
58 return new Response("not found", { status: 404 });
59 },
60 });
61 return { url: `http://127.0.0.1:${server.port}/`, hits, stop: () => void server.stop(true) };
62 }
33 const registry = fixtureRegistry;
6334
6435 /** An app's checkout depending on the fixture, with the lockfile bun writes against it. */
6536 async function checkout(dir: string, reg: string): Promise<string> {
test/apps-adversarial-test-step.test.ts +39 −3
@@ -26,6 +26,7 @@ import { createUser, type User } from "../src/core/users.ts";
2626 import { serve, type Servers } from "../src/http/server.ts";
2727 import { checkInstallCache } from "../src/checks/install.ts";
2828 import { readRunLog } from "../src/runs/log.ts";
29 import { fixtureLock, fixtureRegistry } from "./helpers/fixture-registry.ts";
2930
3031 const DOMAIN = "apps.test";
3132
@@ -181,9 +182,43 @@ describe("apps adversarial: the install and test steps", () => {
181182 }, 20_000);
182183
183184 test("a preview's install reads its repository's shared cache (#7544)", async () => {
184 // A local dependency: a real bun.lock the fill trusts, and nothing fetched.
185 // A registry package on loopback, with the lockfile bun writes against
186 // it: one the fill trusts (`checks/fillable.ts`), and the forge's own
187 // registry setting, as an operator's mirror would be.
188 const reg = await fixtureRegistry(join(work, "registry"));
189 const before = process.env["BUN_CONFIG_REGISTRY"];
190 process.env["BUN_CONFIG_REGISTRY"] = reg.url;
191 try {
192 const { packageJson, lock } = await fixtureLock(join(work, "registry"), reg.url);
193 const pull = await pullWith(
194 "shared-cache",
195 {
196 install: `bun install --frozen-lockfile && echo "install cache: \${BUN_INSTALL_CACHE_DIR:-none}"`,
197 run: "bun run src/server.ts",
198 },
199 [
200 { path: "package.json", content: packageJson },
201 { path: "bun.lock", content: lock },
202 ],
203 );
204 const run = await settled(
205 (r) => r.env === `pull-${pull.number}` && r.commitOid === pull.commitOid,
206 );
207 const log = readRunLog(dataDir, run.id).lines;
208 expect(run.state).toBe("ok");
209 expect(log).toContain(`install cache: ${checkInstallCache(dataDir, repo.id)}`);
210 } finally {
211 if (before === undefined) delete process.env["BUN_CONFIG_REGISTRY"];
212 else process.env["BUN_CONFIG_REGISTRY"] = before;
213 reg.stop();
214 }
215 }, 30_000);
216
217 test("a preview with a file: dependency installs without the shared cache (#7541)", async () => {
218 // The fill runs on the host: a path dependency would be read there, so
219 // the fill refuses it and the install runs into the environment's home.
185220 const pull = await pullWith(
186 "shared-cache",
221 "file-dep",
187222 {
188223 install: `bun install --frozen-lockfile && echo "install cache: \${BUN_INSTALL_CACHE_DIR:-none}"`,
189224 run: "bun run src/server.ts",
@@ -220,7 +255,8 @@ describe("apps adversarial: the install and test steps", () => {
220255 );
221256 const log = readRunLog(dataDir, run.id).lines;
222257 expect(run.state).toBe("ok");
223 expect(log).toContain(`install cache: ${checkInstallCache(dataDir, repo.id)}`);
258 expect(log.some((line) => line.includes("cache not filled"))).toBe(true);
259 expect(log).not.toContain(`install cache: ${checkInstallCache(dataDir, repo.id)}`);
224260 }, 20_000);
225261
226262 test("closing a pull whose deploy already answered kills its still-running test step — no orphan `bun test`", async () => {
test/check-fillable.test.ts added +105
@@ -0,0 +1,105 @@
1 /**
2 * What the forge fills its shared bun cache from, on the host (#7543,
3 * #7541): only a lockfile whose every package is a registry `name@version`
4 * with a sha512 integrity; anything else installs into the cell's own cache.
5 */
6 import { describe, expect, test } from "bun:test";
7 import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { fillable } from "../src/checks/fillable.ts";
11 import { lockfileRefusal } from "../src/checks/install.ts";
12
13 const lockOf = (packages: Record<string, unknown>, workspaces: Record<string, unknown> = { "": {} }): string =>
14 JSON.stringify({ lockfileVersion: 1, workspaces, packages });
15 const SHA = "sha512-AAAA";
16
17 describe("fillable: what the forge fills on the host", () => {
18 const registry = "https://registry.npmjs.org/";
19
20 test("plain registry packages by name and version are filled, scoped and aliased too", () => {
21 const lock = `{
22 "lockfileVersion": 1,
23 "workspaces": { "": { "name": "p", "dependencies": { "a": "^1.0.0" }, }, },
24 "packages": {
25 "a": ["[email protected]", "", {}, "${SHA}"],
26 "@s/b": ["@s/[email protected]", "", { "os": "linux" }, "${SHA}"],
27 "ts-api": ["[email protected]", "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", {}, "${SHA}"],
28 },
29 }`;
30 const manifest = '{"dependencies":{"a":"^1.0.0","@s/b":"2.0.0-rc.1"},"devDependencies":{"ts-api":"npm:typescript@^5.9.0","c":"latest"}}';
31 expect(fillable(manifest, lock, registry)).toEqual({ ok: true, value: undefined });
32 });
33
34 test("anything the registry does not serve by name keeps the fill off the host", () => {
35 const deny = (manifest: string, lock: string): string => {
36 const result = fillable(manifest, lock, registry);
37 if (result.ok) throw new Error("filled");
38 return result.error;
39 };
40 const plain = "{}";
41 expect(deny(plain, lockOf({ x: ["[email protected]", "http://10.0.0.1/x.tgz", {}, SHA] }))).toContain("another registry");
42 expect(deny(plain, lockOf({ x: ["x@github:o/r#abc", {}, "o-r-abc"] }))).toContain("not a registry package");
43 expect(deny(plain, lockOf({ x: ["x@file:../../etc", {}] }))).toContain("not a registry package");
44 expect(deny(plain, lockOf({ x: ["[email protected]", "", {}, "sha1-abc"] }))).toContain("sha512");
45 expect(deny(plain, lockOf({}, { "": {}, "packages/a": {} }))).toBe("workspaces");
46 expect(deny('{"workspaces":["packages/*"]}', lockOf({}))).toBe("workspaces");
47 expect(deny('{"dependencies":{"x":"file:../x"}}', lockOf({}))).toContain("x is not a registry dependency");
48 expect(deny('{"dependencies":{"x":"o/r"}}', lockOf({}))).toContain("x is not a registry dependency");
49 expect(deny('{"dependencies":{"x":"git+https://h/r.git"}}', lockOf({}))).toContain("x is not");
50 expect(deny(plain, "not json")).toContain("does not parse");
51 expect(deny(plain, JSON.stringify({ lockfileVersion: 1, workspaces: { "": {} }, patchedDependencies: {} }))).toBe(
52 "patched dependencies",
53 );
54 });
55
56 test("an override or resolution that is not a registry version keeps the fill off the host", () => {
57 const deny = (manifest: string, lock: string): string => {
58 const result = fillable(manifest, lock, registry);
59 if (result.ok) throw new Error("filled");
60 return result.error;
61 };
62 expect(deny('{"overrides":{"x":"file:../../etc"}}', lockOf({}))).toBe("overrides: x is not a registry version");
63 expect(deny('{"overrides":{"a":{"x":"github:o/r"}}}', lockOf({}))).toBe("overrides: x is not a registry version");
64 expect(deny('{"resolutions":{"x":"https://h/x.tgz"}}', lockOf({}))).toBe("resolutions: x is not a registry version");
65 const locked = JSON.stringify({ lockfileVersion: 1, workspaces: { "": {} }, overrides: { x: "link:x" }, packages: {} });
66 expect(deny("{}", locked)).toBe("overrides: x is not a registry version");
67 const pinned = '{"overrides":{"x":"1.2.3","a":{"y":"^2.0.0"}},"resolutions":{"z":"~3.1.0"}}';
68 expect(fillable(pinned, lockOf({}), registry)).toEqual({ ok: true, value: undefined });
69 });
70 });
71
72 describe("lockfileRefusal: the fill asks fillable too", () => {
73 const registry = "https://registry.npmjs.org/";
74 const checkout = (manifest: string, lock: string): string => {
75 const dir = mkdtempSync(join(tmpdir(), "fillable-"));
76 writeFileSync(join(dir, "package.json"), manifest);
77 writeFileSync(join(dir, "bun.lock"), lock);
78 return dir;
79 };
80
81 test("a registry lockfile with sha512 integrities is filled", () => {
82 const dir = checkout('{"dependencies":{"a":"1.2.3"}}', lockOf({ a: ["[email protected]", "", {}, SHA] }));
83 try {
84 expect(lockfileRefusal(dir, registry)).toBeUndefined();
85 } finally {
86 rmSync(dir, { recursive: true, force: true });
87 }
88 });
89
90 test("no URL and no git, yet a file dependency, a workspace or a weak integrity is still refused", () => {
91 const cases: [string, string, string][] = [
92 ['{"dependencies":{"x":"file:../../etc"}}', lockOf({ x: ["x@file:../../etc", {}] }), "not a registry"],
93 ['{"workspaces":["packages/*"]}', lockOf({}), "workspaces"],
94 ["{}", lockOf({ x: ["[email protected]", "", {}, "sha1-abc"] }), "sha512"],
95 ];
96 for (const [manifest, lock, why] of cases) {
97 const dir = checkout(manifest, lock);
98 try {
99 expect(lockfileRefusal(dir, registry)).toContain(why);
100 } finally {
101 rmSync(dir, { recursive: true, force: true });
102 }
103 }
104 });
105 });
test/check-install.test.ts +3 −1
@@ -268,6 +268,8 @@ describe("prefillInstallCache (#7543)", () => {
268268 const made = Bun.spawnSync(["tar", "-czf", join(dir, "fixture.tgz"), "-C", join(dir, "pkg"), "package"]);
269269 if (made.exitCode !== 0) throw new Error("tar is needed for the fixture registry");
270270 const tarball = new Uint8Array(await Bun.file(join(dir, "fixture.tgz")).arrayBuffer());
271 // As npm serves it: the fill trusts only a lockfile with sha512 integrities (`checks/fillable.ts`).
272 const integrity = `sha512-${new Bun.CryptoHasher("sha512").update(tarball).digest("base64")}`;
junipercrew

Duplicates the integrity computation in test/helpers/fixture-registry.ts; reuse the helper.
271273 const hits: string[] = [];
272274 const server: ReturnType<typeof Bun.serve> = Bun.serve({
273275 port: 0,
@@ -284,7 +286,7 @@ describe("prefillInstallCache (#7543)", () => {
284286 "1.0.0": {
285287 name: "fixture-pkg",
286288 version: "1.0.0",
287 dist: { tarball: `http://127.0.0.1:${server.port}/fixture-pkg/-/fixture-pkg-1.0.0.tgz` },
289 dist: { tarball: `http://127.0.0.1:${server.port}/fixture-pkg/-/fixture-pkg-1.0.0.tgz`, integrity },
288290 },
289291 },
290292 });
test/helpers/fixture-registry.ts added +65
@@ -0,0 +1,65 @@
1 /**
2 * A registry on loopback serving one package, `[email protected]`, with the
3 * sha512 integrity npm serves, so a lockfile bun writes against it is one
4 * the shared cache's fill trusts (`checks/fillable.ts`). Counts its requests.
5 */
6
7 import { mkdirSync, writeFileSync } from "node:fs";
8 import { join } from "node:path";
9
10 export interface FixtureRegistry {
11 readonly url: string;
12 readonly hits: string[];
13 readonly stop: () => void;
14 }
15
16 /** Serve `[email protected]` from a tarball built under `dir`. */
17 export async function fixtureRegistry(dir: string): Promise<FixtureRegistry> {
18 mkdirSync(join(dir, "pkg", "package"), { recursive: true });
19 writeFileSync(join(dir, "pkg", "package", "package.json"), JSON.stringify({ name: "fixture-pkg", version: "1.0.0" }));
20 const made = Bun.spawnSync(["tar", "-czf", join(dir, "fixture.tgz"), "-C", join(dir, "pkg"), "package"]);
21 if (made.exitCode !== 0) throw new Error("tar is needed for the fixture registry");
22 const tarball = new Uint8Array(await Bun.file(join(dir, "fixture.tgz")).arrayBuffer());
23 const integrity = `sha512-${new Bun.CryptoHasher("sha512").update(tarball).digest("base64")}`;
24 const hits: string[] = [];
25 const server: ReturnType<typeof Bun.serve> = Bun.serve({
26 port: 0,
27 hostname: "127.0.0.1",
28 fetch(req): Response {
29 const path = new URL(req.url).pathname;
30 hits.push(path);
31 if (path === "/fixture-pkg/-/fixture-pkg-1.0.0.tgz") return new Response(tarball);
32 if (path === "/fixture-pkg")
33 return Response.json({
34 name: "fixture-pkg",
35 "dist-tags": { latest: "1.0.0" },
36 versions: {
37 "1.0.0": {
38 name: "fixture-pkg",
39 version: "1.0.0",
40 dist: { tarball: `http://127.0.0.1:${server.port}/fixture-pkg/-/fixture-pkg-1.0.0.tgz`, integrity },
41 },
42 },
43 });
44 return new Response("not found", { status: 404 });
45 },
46 });
47 return { url: `http://127.0.0.1:${server.port}/`, hits, stop: () => void server.stop(true) };
48 }
49
50 /** The `bun.lock` bun writes for a package depending on `[email protected]` from `reg`. */
51 export async function fixtureLock(dir: string, reg: string): Promise<{ packageJson: string; lock: string }> {
52 const co = join(dir, "lock-only");
53 mkdirSync(co, { recursive: true });
54 const packageJson = `${JSON.stringify({ name: "steps", version: "0.1.0", private: true, dependencies: { "fixture-pkg": "1.0.0" } })}\n`;
55 writeFileSync(join(co, "package.json"), packageJson);
56 // Async: the fixture registry answers from this process's event loop.
57 const made = Bun.spawn(["bun", "install", "--lockfile-only"], {
58 cwd: co,
59 env: { PATH: process.env["PATH"] ?? "", HOME: dir, BUN_CONFIG_REGISTRY: reg, BUN_INSTALL_CACHE_DIR: join(dir, "lock-cache") },
60 stdout: "pipe",
61 stderr: "pipe",
62 });
63 if ((await made.exited) !== 0) throw new Error(`fixture lockfile: ${await new Response(made.stderr).text()}`);
64 return { packageJson, lock: await Bun.file(join(co, "bun.lock")).text() };
65 }

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs