Public: anyone can read this; the crew can change it.
Add the release load check and have cut-release name it (part of #9405 box 1) #9515
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/9504-keep-the-first-load-baseline-in-bench-re → main 8 commits fork 3 threads · 3 unresolved
5 files
changed +105
−3
docs/features/load-test.md +4 −2
| @@ -15,11 +15,13 @@ Media (`/media/…`, the route a PDF or image is drawn from) is **not covered**: | ||
| 15 | 15 | |
| 16 | 16 | The bypasses are tested (`test/anonymous-deep.test.ts`): a signed-in reader, a repository made private, `?fragment=1` and a branch-name ref never get a page kept for a stranger. `--compare` exiting 1 is tested (`test/load-bench.test.ts`). |
| 17 | 17 | |
| 18 | **Release step:** `bun scripts/release-load.ts <version> --url <copy> --repo <owner/name>` writes `bench/results/<version>.json` and compares with the newest older kept file (`src/perf/baseline.ts`); exit 1 is the gate. With no older file it only writes the first; commit it. `scripts/cut-release.ts` prints this command as its last step; it does not run it (it needs a copy with data), and no baseline is kept yet — the first release run writes it. | |
| 19 | ||
| 18 | 20 | No baseline is kept in `bench/results/` yet: the pull's preview holds no repository to read, and the load must not be aimed at production. The first file is written by the first run against a copy with data. |
| 19 | 21 | |
| 20 | Follow-ups: profile #9503, baseline and release wiring #9504, media #9505. | |
| 22 | Follow-ups: profile #9503, baseline #9504 (the file itself, still to be written from a copy with data), media #9505. | |
| 21 | 23 | |
| 22 | Not done here: the profile of tree/blob/raw/media under production's shape, the p95 targets at 500/s, and wiring the comparison into the release check. Those need a machine and data this run does not have. | |
| 24 | Not done here: the profile of tree/blob/raw/media under production's shape, the p95 targets at 500/s. Those need a machine and data this run does not have. | |
| 23 | 25 | |
| 24 | 26 | ## The per-phase profile (#9503) |
| 25 | 27 |
scripts/cut-release.ts +4 −1
| @@ -5,10 +5,12 @@ | ||
| 5 | 5 | * reads. With `--tag` it also makes the annotated tag `v<version>` on HEAD |
| 6 | 6 | * with that record as its message; pushing the tag (`git push origin |
| 7 | 7 | * v<version>`) is the release. Nothing is committed: a release cut never |
| 8 | * moves main, so it voids no gate in flight (#6640). | |
| 8 | * moves main, so it voids no gate in flight (#6640). It ends by printing, on | |
| 9 | * stderr, the load check to run before the tag is pushed (`release-load.ts`, #9504). | |
| 9 | 10 | */ |
| 10 | 11 | |
| 11 | 12 | import { migrationFiles, NUMBERED } from "../src/core/schema.ts"; |
| 13 | import { loadCheckHint } from "../src/perf/baseline.ts"; | |
| 12 | 14 | import { compareVersions, mergeShipped, readTagged, recordFor, shippedReleases } from "../src/release/shipped.ts"; |
| 13 | 15 | |
| 14 | 16 | const VERSION = /^\d+\.\d+\.\d+$/; |
| @@ -37,3 +39,4 @@ if (flag === "--tag") { | ||
| 37 | 39 | } |
| 38 | 40 | // CLI answer for the person at the terminal. |
| 39 | 41 | console.log(record); |
| 42 | console.error(loadCheckHint(version)); |
scripts/release-load.ts added +22
| @@ -0,0 +1,22 @@ | ||
| 1 | /** | |
| 2 | * `bun scripts/release-load.ts <version> --url <copy> --repo <owner/name> [--rate N] [--seconds N]`: | |
| 3 | * the release step's load check (#9504). Runs bench/load.ts against a copy | |
| 4 | * with data, writes bench/results/<version>.json and compares with the | |
| 5 | * newest kept older result; exit 1 is a fall of more than 20%. With no older | |
| 6 | * result it only writes the first baseline. Commit the file it writes. | |
| 7 | */ | |
| 8 | ||
| 9 | import { existsSync, mkdirSync, readdirSync } from "node:fs"; | |
| 10 | import { lastBaseline, loadArgs } from "../src/perf/baseline.ts"; | |
| 11 | ||
| 12 | const [version, ...rest] = process.argv.slice(2); | |
| 13 | if (version === undefined || !/^\d+\.\d+\.\d+$/.test(version) || !rest.includes("--url") || !rest.includes("--repo")) { | |
| 14 | console.error("usage: bun scripts/release-load.ts <major.minor.patch> --url <copy> --repo <owner/name> [--rate N] [--seconds N]"); | |
| 15 | process.exit(2); | |
| 16 | } | |
| 17 | const dir = `${import.meta.dir}/../bench/results`; | |
| 18 | mkdirSync(dir, { recursive: true }); | |
| 19 | const kept = lastBaseline(existsSync(dir) ? readdirSync(dir) : [], version); | |
| 20 | const cmd = ["bun", `${import.meta.dir}/../bench/load.ts`, ...loadArgs(rest, version, dir, kept)]; | |
| 21 | console.error(kept === undefined ? `release-load: no older baseline; writing the first, ${version}.json` : `release-load: comparing with ${kept}`); | |
| 22 | process.exit(await Bun.spawn(cmd, { stdout: "inherit", stderr: "inherit" }).exited); |
src/perf/baseline.ts added +32
| @@ -0,0 +1,32 @@ | ||
| 1 | /** | |
| 2 | * Which kept load result a release is compared against (#9504). | |
| 3 | * File names in, the newest older one out; `bench/load.ts` does the judging. | |
| 4 | */ | |
| 5 | ||
| 6 | import { compareVersions } from "../release/shipped.ts"; | |
| 7 | ||
| 8 | const KEPT = /^(\d+\.\d+\.\d+)\.json$/; | |
| 9 | ||
| 10 | /** The newest kept result strictly older than `version`, as a file name; undefined when none is kept. */ | |
| 11 | export function lastBaseline(files: readonly string[], version: string): string | undefined { | |
| 12 | const older = files | |
| 13 | .map((f) => KEPT.exec(f)?.[1]) | |
| 14 | .filter((v): v is string => v !== undefined && compareVersions(v, version) > 0) | |
| 15 | .sort(compareVersions); | |
| 16 | const newest = older[0]; | |
| 17 | return newest === undefined ? undefined : `${newest}.json`; | |
| 18 | } | |
| 19 | ||
| 20 | /** The `bench/load.ts` arguments for a release: defaults filled, `--compare` added when an older result is kept. */ | |
| 21 | export function loadArgs(rest: readonly string[], version: string, dir: string, kept: string | undefined): string[] { | |
| 22 | const args = [...rest, "--version", version, "--out", `${dir}/${version}.json`]; | |
| 23 | if (!rest.includes("--rate")) args.push("--rate", "500"); | |
| 24 | if (!rest.includes("--seconds")) args.push("--seconds", "20"); | |
| 25 | if (kept !== undefined) args.push("--compare", `${dir}/${kept}`); | |
| 26 | return args; | |
| 27 | } | |
| 28 | ||
| 29 | /** The line `cut-release` ends with: the load check to run against a copy with data before pushing the tag. */ | |
| 30 | export function loadCheckHint(version: string): string { | |
| 31 | return `cut-release: before pushing the tag, run the load check against a copy with data:\n bun scripts/release-load.ts ${version} --url <copy> --repo <owner/name>\nthen commit bench/results/${version}.json`; | |
| 32 | } |
test/release-load.test.ts added +43
| @@ -0,0 +1,43 @@ | ||
| 1 | import { expect, test } from "bun:test"; | |
| 2 | import { lastBaseline, loadArgs, loadCheckHint } from "../src/perf/baseline.ts"; | |
| 3 | ||
| 4 | test("the baseline is the newest kept result older than the release", () => { | |
| 5 | const files = ["0.7.9.json", "0.7.10.json", "0.7.12.json", "notes.md", "x.json"]; | |
| 6 | expect(lastBaseline(files, "0.7.11")).toBe("0.7.10.json"); | |
| 7 | expect(lastBaseline(files, "0.7.13")).toBe("0.7.12.json"); | |
| 8 | expect(lastBaseline(files, "0.7.9")).toBeUndefined(); | |
| 9 | expect(lastBaseline([], "1.0.0")).toBeUndefined(); | |
| 10 | }); | |
| 11 | ||
| 12 | test("release-load refuses a bad version, a missing --url and a missing --repo", async () => { | |
outdated
Only a bad version is exercised. Also assert exit 2 for a valid version without --url and without --repo. |
||
| 13 | const refused = async (args: string[]) => { | |
| 14 | const p = Bun.spawn(["bun", "scripts/release-load.ts", ...args], { stdout: "pipe", stderr: "pipe" }); | |
| 15 | const [code, err] = await Promise.all([p.exited, new Response(p.stderr).text()]); | |
| 16 | expect(err).toContain("usage: bun scripts/release-load.ts"); | |
| 17 | return code; | |
| 18 | }; | |
| 19 | expect(await refused(["nope", "--url", "http://c", "--repo", "o/r"])).toBe(2); | |
| 20 | expect(await refused(["0.7.11", "--repo", "o/r"])).toBe(2); | |
| 21 | expect(await refused(["0.7.11", "--url", "http://c"])).toBe(2); | |
| 22 | }); | |
| 23 | ||
| 24 | test("the bench arguments carry the version, the output file, defaults and --compare when a baseline is kept", () => { | |
| 25 | const a = loadArgs(["--url", "http://c", "--repo", "o/r"], "0.7.11", "/r", "0.7.10.json"); | |
| 26 | expect(a).toEqual(["--url", "http://c", "--repo", "o/r", "--version", "0.7.11", "--out", "/r/0.7.11.json", "--rate", "500", "--seconds", "20", "--compare", "/r/0.7.10.json"]); | |
| 27 | const b = loadArgs(["--rate", "100", "--seconds", "5"], "1.0.0", "/r", undefined); | |
| 28 | expect(b).not.toContain("--compare"); | |
| 29 | expect(b.filter((x) => x === "--rate")).toEqual(["--rate"]); | |
outdated
This reads the source text and would stay green if the hint were removed from the output path or printed to the wrong stream. Spawn cut-release (or call an extracted printer) and assert the hint reaches stderr and stdout stays the JSON record. |
||
| 30 | expect(b).toContain("100"); | |
| 31 | }); | |
| 32 | ||
| 33 | test("cut-release names the load check on stderr and keeps stdout the JSON record alone", async () => { | |
| 34 | // A version far ahead of any release, without --tag: nothing is written. | |
| 35 | const p = Bun.spawn(["bun", "scripts/cut-release.ts", "999.0.0"], { stdout: "pipe", stderr: "pipe" }); | |
| 36 | const [code, out, err] = await Promise.all([p.exited, new Response(p.stdout).text(), new Response(p.stderr).text()]); | |
| 37 | expect(code).toBe(0); | |
| 38 | expect(err).toContain(loadCheckHint("999.0.0")); | |
| 39 | expect(err).toContain("bun scripts/release-load.ts 999.0.0 --url"); | |
| 40 | expect(out.trim().split("\n")).toHaveLength(1); | |
| 41 | expect(JSON.parse(out).version).toBe("999.0.0"); | |
| 42 | expect(out).not.toContain("release-load"); | |
| 43 | }); |
mkdirSync creates an empty bench/results that git will not keep, and the check runs even when the child fails to write. Harmless, but the file the pull is named for is still absent; see the main comment.