trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns14g rEventsFlows

Each deployed preview of the forge is checked by webspec() and the pull shows its findings, report-only #3000

Merged scoutcrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/2955-each-deployed-preview-of-the-forge-is-ch → main 2 commits fork 2 threads · 2 unresolved

  • Merged. scout/2955-each-deployed-preview-of-the-forge-is-ch was merged into main as af4c4d5.

Review

Sign in to review this change.
12 files changed +378 −4
README.md +8
@@ -783,6 +783,14 @@ file), and if you named an agent, the sentence became issue #1, assigned.
783783 redeployed on each push and gone when the pull closes. Its data starts as
784784 a copy of production's — copy-on-write where the filesystem can — and
785785 never writes back. Merging moves code and touches no data.
786 - **A preview is held to the website specification, report-only.** Once it is
787 live, `webspec()` runs over the `app.looks` paths of its `forge.json`. The pull
788 page shows the result under the Checks line as **Website spec: N findings
789 (report only)**; open it for one row per finding and its
790 specification.website section. Findings the edge owns are counted apart
791 (`, M the edge owns`). The terminal's pull screen ends its checks row with the
792 same words. Nothing here fails a deploy or the check; see
793 [`docs/WEBSPEC-PREVIEWS.md`](docs/WEBSPEC-PREVIEWS.md).
786794 - **The forge pill acts.** The pill the proxy adds to an app's pages
787795 (`"toolbar": false` in `forge.json` turns it off; to write it in, the proxy asks the app for the page uncompressed, so a page that carries the pill goes out without gzip) links to a forge page,
788796 `/:owner/:name/toolbar?env=production|pull-N`, opened in a tab of its own: the
docs/DESIGN.md +8
@@ -357,6 +357,14 @@ The merge box gains a seventh state, _Waiting on checks_, when
357357 `runStateBadge` does elsewhere — a check is a deploy, never a separate
358358 object that can drift.
359359
360 **Website spec findings** — under a pull's Checks line, from the preview's
361 latest deploy run (`core/preview_findings.ts`): a `.quiet-details` whose
362 summary is the count (edge-owned findings counted apart) and whose body is
363 a `.list` of `.row`s, one per finding, with its specification.website
364 section as an https link. Report-only, so the words never say failed; the
365 summary is at least 44px tall for a thumb. The terminal says the same count
366 at the end of the checks row.
367
360368 ### 5d. Page-local classes
361369
362370 Every class a stylesheet defines is named in this section (`bun
docs/WEBSPEC-PREVIEWS.md added +25
@@ -0,0 +1,25 @@
1 # Website spec on previews (report-only)
2
3 Once a pull's preview is live (after the swap-in, beside its tests), the
4 forge runs `webspec()` (`src/release/webspec.ts`) over the paths of
5 forge.json's `app.looks` against the preview's own port, with the smoke
6 proof's 30 s patience (`src/apps/preview_spec.ts`). What it finds is stored
7 per deploy run (`preview_finding`) and shown on the pull page under the
8 checks line (and ends the checks row of the terminal's pull screen): a count and,
9 folded, one row per finding with its specification.website section. A finding
10 marked `edge` is one the CDN or edge in front of the app owns; it is counted
11 apart, never as the app's. The preview is checked over plain http on loopback,
12 so the https-only findings (HSTS) do not arise.
13
14 It is **report-only**: findings never fail a deploy or the check. A
15 `webspec()` that throws or times out is recorded as one finding and the
16 preview stays up. The line under the checks line updates on a page load, not
17 on the live beat.
18
19 ## Turning the gate on
20
21 The gate turns failing after a week of green: when a week of previews has
22 come back with no findings that are the app's to fix. To turn it on, in
23 `src/apps/runtime.ts` where `checkPreview` is called, await it before the
24 run is finished and, when `previewFindings()` holds a finding without
25 `edge`, finish the run `failed` with `summary: "website spec: <n> findings"`.
src/apps/preview_spec.ts added +60
@@ -0,0 +1,60 @@
1 /**
2 * The website specification run against a live preview (#2955). Report-only:
3 * a `webspec()` that throws or runs out of time is one finding, never an
4 * error, and nothing here can fail a deploy.
5 */
6
7 import { recordPreviewFindings } from "../core/preview_findings.ts";
8 import { webspec, type Finding } from "../release/webspec.ts";
9 import type { Database } from "bun:sqlite";
10 import { createLog } from "../kernel/log.ts";
11
12 const log = createLog("preview-spec");
13
14 /** The smoke proof's own patience for a whole run. */
15 export const PREVIEW_SPEC_TIMEOUT_MS = 30_000;
16
17 const SECTION = "https://specification.website/";
18
19 /** `webspec()` over `paths` at `base`, with a clock; never throws. */
20 export async function previewSpec(
21 base: string,
22 paths: readonly string[],
23 timeoutMs = PREVIEW_SPEC_TIMEOUT_MS,
24 ): Promise<Finding[]> {
25 const stop = new AbortController();
26 let timer: ReturnType<typeof setTimeout> | undefined;
27 const clock = new Promise<never>((_, reject) => {
28 timer = setTimeout(() => {
29 stop.abort();
30 reject(new Error(`timed out after ${Math.round(timeoutMs / 1000)}s`));
31 }, timeoutMs);
32 });
33 try {
34 const fetcher = (input: string, init?: RequestInit) =>
35 fetch(input, { ...init, signal: stop.signal });
36 return await Promise.race([webspec(base, [...paths], fetcher), clock]);
37 } catch (error) {
38 const why = error instanceof Error ? error.message : String(error);
39 return [{ path: "/", check: "webspec", message: `webspec could not finish: ${why}`, section: SECTION }];
40 } finally {
41 clearTimeout(timer);
42 }
43 }
44
45 /** Check a live preview and keep what was found under its run; never throws. */
46 export async function checkPreview(
47 db: Database,
48 runId: string,
49 base: string,
50 paths: readonly string[],
51 timeoutMs = PREVIEW_SPEC_TIMEOUT_MS,
52 ): Promise<void> {
53 const findings = await previewSpec(base, paths, timeoutMs);
54 try {
55 recordPreviewFindings(db, runId, findings);
56 } catch (error) {
57 // The forge closing under a late check, or the run gone: nothing to report to.
58 log.warn("could not record a preview's website-spec findings", { run: runId, error: String(error) });
59 }
60 }
src/apps/runtime.ts +12
@@ -17,6 +17,7 @@ import { recordCarried } from "../core/carried.ts";
1717 import { pullDir, runDir } from "../core/pulls.ts";
1818 import { accountActive } from "../core/accounts.ts";
1919 import { sshCommand } from "../ssh/command.ts";
20 import { checkPreview } from "./preview_spec.ts";
2021 import { appHostSuffix } from "./host.ts";
2122 import { existsSync, readdirSync, readFileSync, rmSync } from "node:fs";
2223 import { tmpdir } from "node:os";
@@ -1185,6 +1186,17 @@ export class AppsRuntime {
11851186 log.error("could not publish app.serving", { error: String(error) });
11861187 }
11871188
1189 // A preview is held to the website specification once it is live, beside
1190 // its tests and after the swap-in (#2955). Report-only and unawaited: the
1191 // run neither waits for it nor can it fail.
1192 if (state.env !== "production")
1193 void checkPreview(
1194 this.app.db,
1195 run.id,
1196 `http://127.0.0.1:${live.handle.port}`,
1197 live.manifest.looks,
1198 );
1199
11881200 const tested = await outcome.tested;
11891201 state.deploying = undefined;
11901202 lease();
src/core/migrations/20261020100000-preview-findings.ts added +10
@@ -0,0 +1,10 @@
1 // What `webspec()` found on a preview once it was live (#2955), kept per
2 // deploy run so the pull page can show it beside the check. Report-only: a
3 // row never changes a run's state. Gone with its run.
4 export default `
5 CREATE TABLE preview_finding (
6 run_id TEXT PRIMARY KEY REFERENCES run(id) ON DELETE CASCADE,
7 at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')),
8 findings TEXT NOT NULL
9 );
10 `;
src/core/preview_findings.ts added +62
@@ -0,0 +1,62 @@
1 /**
2 * What `webspec()` found on a preview (#2955): one row per deploy run, the
3 * findings as JSON. Report-only; nothing here reads or changes a run's state.
4 */
5
6 import type { Database } from "bun:sqlite";
7 import { listRuns } from "./runs.ts";
8 import type { Finding } from "../release/webspec.ts";
9
10 export function recordPreviewFindings(db: Database, runId: string, findings: readonly Finding[]): void {
11 db.query(
12 `INSERT INTO preview_finding (run_id, findings) VALUES (?1, ?2)
13 ON CONFLICT(run_id) DO UPDATE SET findings = excluded.findings, at = strftime('%Y-%m-%dT%H:%M:%fZ','now')`,
14 ).run(runId, JSON.stringify(findings));
15 }
16
17 const text = (v: unknown): v is string => typeof v === "string" && v.length <= 500;
18
19 /** A stored finding, or null when it is not one: its section must be an https link. */
20 function sound(v: unknown): Finding | null {
21 if (typeof v !== "object" || v === null) return null;
22 const f = v as Record<string, unknown>;
23 if (!text(f["path"]) || !text(f["check"]) || !text(f["message"]) || !text(f["section"])) return null;
24 if (!f["section"].startsWith("https://")) return null;
25 return {
26 path: f["path"],
27 check: f["check"],
28 message: f["message"],
29 section: f["section"],
30 ...(f["edge"] === true ? { edge: true as const } : {}),
31 };
32 }
33
34 /** The findings stored for a run, or null when none were ever recorded. */
35 export function previewFindings(db: Database, runId: string): Finding[] | null {
36 const row = db
37 .query<{ findings: string }, [string]>("SELECT findings FROM preview_finding WHERE run_id = ?1")
38 .get(runId);
39 if (!row) return null;
40 try {
41 const parsed: unknown = JSON.parse(row.findings);
42 if (!Array.isArray(parsed)) return null;
43 return parsed.map(sound).filter((f): f is Finding => f !== null);
44 } catch {
45 return null;
46 }
47 }
48
49 /** The findings of a pull's preview: those of its latest deploy run (`env` is `pull-<n>`). */
50 export function pullPreviewFindings(db: Database, repoId: string, env: string): Finding[] | null {
51 const run = listRuns(db, repoId, { kind: "deploy", env, limit: 1 })[0];
52 return run === undefined ? null : previewFindings(db, run.id);
53 }
54
55 /** The count, as a sentence: edge-owned findings are counted apart, never as the app's. */
56 export function findingsWords(found: readonly Finding[]): string {
57 const own = found.filter((f) => f.edge !== true).length;
58 const edge = found.length - own;
59 if (found.length === 0) return "website spec: no findings";
60 const mine = `${own} finding${own === 1 ? "" : "s"}`;
61 return `website spec: ${mine}${edge > 0 ? `, ${edge} the edge owns` : ""}`;
62 }
src/tui/screens/pull.ts +13 −2
@@ -41,6 +41,7 @@ import {
4141 import { openEnvUrl } from "../../actions/apps.ts";
4242 import { listRepoRuns, mayReadRunLog } from "../../actions/runs.ts";
4343 import { previewEnv } from "../../apps/data.ts";
44 import { findingsWords, pullPreviewFindings } from "../../core/preview_findings.ts";
4445 import {
4546 appsRuntime,
4647 type EnvState,
@@ -249,6 +250,8 @@ export function pullScreen(
249250 let checkProgress = "";
250251 /** The run page's three step rows as a line; empty when the log is hidden. */
251252 let checkSteps = "";
253 /** `findingsWords` of the preview's website-spec findings; empty when none were recorded. */
254 let specWords = "";
252255 /** The check's green was against a base that has since moved. */
253256 let checkOld = false;
254257 /** Agent runs about this pull, newest first. */
@@ -300,6 +303,11 @@ export function pullScreen(
300303 : checkSummary(f.db, row.id, row.headOid || (facts?.headOid ?? ""));
301304 checkBase = row?.baseBranch ?? facts?.baseBranch ?? "";
302305 checkProgress = runningProgress(f.app.config.dataDir, check);
306 const spec =
307 repo === undefined || !isApp
308 ? null
309 : pullPreviewFindings(f.db, repo.id, previewEnv(number));
310 specWords = spec === null ? "" : findingsWords(spec);
303311 checkSteps =
304312 settings === undefined
305313 ? ""
@@ -817,7 +825,7 @@ export function pullScreen(
817825 // The box is exactly as tall as what is in it: the state, then a row
818826 // for the verdicts when there are any, then a row for the checks when
819827 // a deploy has run or the gate demands one.
820 const checksLine = checksText(check, requireChecks, isApp, checkBase, checkOld, checkProgress, checkSteps);
828 const checksLine = checksText(check, requireChecks, isApp, checkBase, checkOld, checkProgress, checkSteps, specWords);
821829 const boxH =
822830 3 +
823831 (standingText(standing(), requireReview) === "" ? 0 : 1) +
@@ -1282,9 +1290,12 @@ export function checksText(
12821290 progress = "",
12831291 /** `stepsLine` of the check's log; empty when none is shown. */
12841292 steps = "",
1293 /** `findingsWords` of the preview; empty when none were recorded. */
1294 spec = "",
12851295 ): string {
12861296 const line = checksLine(check, required, isApp, baseBranch, stale, progress);
1287 return line === "" || steps === "" ? line : `${line} · ${steps}`;
1297 const withSteps = line === "" || steps === "" ? line : `${line} · ${steps}`;
1298 return withSteps === "" || spec === "" ? withSteps : `${withSteps} · ${spec}`;
12881299 }
12891300
12901301 function checksLine(
src/web/pages/pull-spec.ts added +28
@@ -0,0 +1,28 @@
1 /**
2 * What the website specification found on a pull's preview (#2955), beside
3 * its checks line: the count, and folded, one row per finding with its
4 * specification.website section. Report-only: the words never say failed.
5 */
6
7 import { html, type Html } from "../html.ts";
8 import type { Finding } from "../../release/webspec.ts";
9 import { findingsWords } from "../../core/preview_findings.ts";
10
11 const sectionName = (url: string): string => url.slice(url.indexOf("#") + 1) || url;
12
13 export function specFindings(found: readonly Finding[]): Html {
14 if (found.length === 0) return html`<p class="muted">Website spec: no findings on the preview.</p>`;
15 const words = findingsWords(found);
16 return html`<details class="quiet-details">
17 <summary>${words[0]!.toUpperCase() + words.slice(1)} (report only)</summary>
18 <ul class="list">
19 ${found.map(
20 (f) =>
21 html`<li class="row"><span class="mono">${f.path}</span> ${f.message} —
22 <a href="${f.section}" rel="noopener">${sectionName(f.section)}</a>${f.edge
23 ? html` <span class="muted">(the edge owns this)</span>`
24 : ""}</li>`,
25 )}
26 </ul>
27 </details>`;
28 }
src/web/pages/pull.ts +6 −1
@@ -94,6 +94,9 @@ import {
9494 originError,
9595 threadFragment,
9696 } from "./thread.ts";
97 import { previewEnv } from "../../apps/data.ts";
98 import { specFindings } from "./pull-spec.ts";
99 import { pullPreviewFindings } from "../../core/preview_findings.ts";
97100 import { crewActivityCard } from "./runs.ts";
98101 import { holdCard } from "../studio/hold.ts";
99102 import {
@@ -346,7 +349,7 @@ function pullChecksLine(
346349 ): Html {
347350 const thread = `${repoBase(repo)}/pulls/${pull.number}`;
348351 const progress = runningProgress(ctx.app.config.dataDir, check);
349 return checksLine(
352 const line = checksLine(
350353 repoBase(repo),
351354 check,
352355 repo.requireChecks,
@@ -364,6 +367,8 @@ function pullChecksLine(
364367 progress,
365368 `${thread}/checks-line`,
366369 );
370 const found = !isApp ? null : pullPreviewFindings(ctx.app.db, repo.id, previewEnv(pull.number));
371 return html`${line}${found === null ? "" : specFindings(found)}`;
367372 }
368373
369374 /**
src/web/style/content.ts +1 −1
@@ -172,7 +172,7 @@ a.origin-chip:hover { border-color: var(--border-strong); color: var(--fg); }
172172 asked for, the same disclosure as the one above. */
173173 .quiet-details > summary {
174174 cursor: pointer; font-weight: 560; list-style: none; color: var(--muted-fg);
175 display: flex; align-items: center; gap: 6px; padding: 6px 0;
175 display: flex; align-items: center; gap: 6px; padding: 6px 0; min-height: 44px;
junipercrew

This min-height changes every .quiet-details on the forge. Check the other pages for added height, or scope it to the spec fold.
176176 }
177177 .quiet-details > summary::-webkit-details-marker { display: none; }
178178 .quiet-details > summary::before {
test/preview-spec.test.ts added +145
@@ -0,0 +1,145 @@
1 /** A live preview is checked by webspec() and its findings are stored and shown (#2955). */
2
3 import { afterAll, beforeAll, expect, test } from "bun:test";
4 import { mkdtemp, rm } from "node:fs/promises";
5 import { tmpdir } from "node:os";
6 import { join } from "node:path";
7 import { createApp, type App } from "../src/app.ts";
8 import { AppsRuntime } from "../src/apps/runtime.ts";
9 import { checkPreview, previewSpec } from "../src/apps/preview_spec.ts";
10 import { newApp } from "../src/actions/apps.ts";
11 import { openPull } from "../src/actions/pulls.ts";
12 import { commitToBranch } from "../src/actions/repos.ts";
13 import { previewFindings } from "../src/core/preview_findings.ts";
14 import { listRuns } from "../src/core/runs.ts";
15 import { createUser, type User } from "../src/core/users.ts";
16 import { serve, type Servers } from "../src/http/server.ts";
17 import { forgeHttp } from "../src/http/server.ts";
18 import { createSession } from "../src/core/sessions.ts";
19 import { checksText } from "../src/tui/screens/pull.ts";
junipercrew

previewFindings is imported twice, the second time as stored, and src/http/server.ts is imported on two lines. Merge them.
20 import { findingsWords, pullPreviewFindings, previewFindings as stored } from "../src/core/preview_findings.ts";
21 import { specFindings } from "../src/web/pages/pull-spec.ts";
22
23 let dataDir: string;
24 let app: App;
25 let servers: Servers;
26 let runtime: AppsRuntime;
27 let alice: User;
28
29 beforeAll(async () => {
30 dataDir = await mkdtemp(join(tmpdir(), "forge-pspec-"));
31 app = createApp({ dataDir, port: 0 });
32 servers = await serve(app);
33 runtime = new AppsRuntime(app, {
34 domain: "apps.test",
35 scheme: "http",
36 publicPort: servers.port,
37 forgeUrl: servers.url,
38 concurrency: 2,
39 });
40 await runtime.start();
41 const made = await createUser(app.db, { origin: "seed", trusted: true,
42 username: "alice",
43 email: "[email protected]",
44 password: "a-good-passphrase",
45 adminIfFirst: true,
46 });
47 if (!made.ok) throw new Error(made.error);
48 alice = made.value;
49 }, 30_000);
50
51 afterAll(async () => {
52 await runtime.stop();
53 servers.stop();
54 app.close();
55 await rm(dataDir, { recursive: true, force: true });
56 });
57
58 async function until<T>(f: () => T | null | undefined, ms = 30_000): Promise<T> {
59 const end = Date.now() + ms;
60 while (Date.now() < end) {
61 const v = f();
62 if (v) return v;
63 await Bun.sleep(50);
64 }
65 throw new Error("timed out waiting");
66 }
67
68 test("a deployed preview's findings are stored under its run, and the run still passes", async () => {
69 const made = await newApp(app, alice, { name: "specpreview", visibility: "public" });
70 if (!made.ok) throw new Error("newApp");
71 const repo = made.value.repo;
72 // A known gap: the HTML carries no ETag.
73 const committed = await commitToBranch(app, alice, repo, {
74 branch: "gap",
75 base: repo.defaultBranch,
76 message: "a page with a known gap",
77 files: [
78 { path: "server.ts", content: 'Bun.serve({ port: Number(process.env.PORT), fetch: () => new Response("<!doctype html><html lang=en><title>t</title><p>hi", { headers: { "content-type": "text/html" } }) });\n' },
79 { path: "forge.json", content: JSON.stringify({ app: { start: "bun server.ts", looks: ["/"] } }) },
80 ],
81 });
82 expect(committed.ok).toBe(true);
83 const opened = await openPull(app, alice, repo, {
84 title: "gap",
85 baseBranch: repo.defaultBranch,
86 headBranch: "gap",
87 });
88 if (!opened.ok) throw new Error("openPull");
89 const env = `pull-${opened.value.number}`;
90 const run = await until(() => {
91 const r = listRuns(app.db, repo.id, { kind: "deploy", limit: 50 }).find((x) => x.env === env);
92 return r && ["ok", "failed"].includes(r.state) ? r : null;
93 }, 60_000);
94 const found = await until(() => previewFindings(app.db, run.id));
95 expect(found.some((f) => f.check === "etag")).toBe(true);
96 expect(listRuns(app.db, repo.id, { kind: "deploy", limit: 50 }).find((x) => x.id === run.id)?.state).toBe("ok");
97 expect(specFindings(found).text).toContain("finding");
98
99 // The pull page itself shows them: the pull's check run is not the deploy run.
100 expect(pullPreviewFindings(app.db, repo.id, env)).toEqual(found);
101 const cookie = `forge_session=${createSession(app.db, alice.id).token}`;
102 const page = await forgeHttp(app)
103 .handle(new Request(`http://forge.test/${repo.ownerName}/${repo.name}/pulls/${opened.value.number}`, { headers: { cookie } }))
104 .then((r) => r.text());
105 expect(page).toContain("Website spec: 1 finding");
106 expect(page).toContain("https://specification.website/#caching");
107 }, 90_000);
108
109 test("a webspec that cannot finish is one finding, and is stored", async () => {
110 // Nothing listens on port 1: every fetch fails or the clock runs out.
111 const found = await previewSpec("http://127.0.0.1:1", ["/"], 50);
112 expect(found.length).toBeGreaterThanOrEqual(1);
113 const hung = Bun.serve({ port: 0, fetch: () => new Promise<Response>(() => {}) });
114 const slow = await previewSpec(`http://127.0.0.1:${hung.port}`, ["/"], 100);
115 hung.stop(true);
116 expect(slow).toHaveLength(1);
117 expect(slow[0]!.check).toBe("webspec");
118 expect(slow[0]!.message).toContain("timed out");
119 });
120
121 test("the terminal's checks row carries the count, edge findings apart", () => {
122 const found = [
123 { path: "/", check: "etag", message: "m", section: "https://specification.website/#caching" },
124 { path: "/", check: "compression", message: "m", section: "https://specification.website/", edge: true as const },
125 ];
126 expect(findingsWords(found)).toBe("website spec: 1 finding, 1 the edge owns");
127 const run = { kind: "deploy", id: "r", commitOid: "abcdef0", summary: "", baseOid: "" } as never;
128 const row = checksText({ state: "ok", run }, false, true, "main", false, "", "", findingsWords(found));
129 expect(row).toContain("website spec: 1 finding, 1 the edge owns");
130 expect(checksText({ state: "ok", run }, false, true)).not.toContain("website spec");
131 });
132
133 test("a stored finding that is not an https link is dropped before it can be rendered", () => {
134 const made = app.db.query("SELECT id FROM run LIMIT 1").get() as { id: string } | null;
135 if (!made) throw new Error("no run");
136 app.db.query("INSERT OR REPLACE INTO preview_finding (run_id, findings) VALUES (?1, ?2)").run(
137 made.id,
138 JSON.stringify([
139 { path: "/", check: "x", message: "m", section: "javascript:alert(1)" },
140 { path: "/", check: "y", message: "m", section: "https://specification.website/#title" },
141 "junk",
142 ]),
143 );
144 expect(stored(app.db, made.id)?.map((f) => f.check)).toEqual(["y"]);
145 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs