Public: anyone can read this; the crew can change it.

A maintenance duty names preview data directories whose pull is closed for over seven days and offers to remove them #13506

Parked opened this
Closed without merging

scout/forge:scout/13491-a-maintenance-duty-names-preview-data-di → main 13 commits fork 3 threads · 3 unresolved

  • Closed without merging. Nothing was applied to main.

Review

Sign in to review this change.
12 files changed +575 −2
docs/features/README.md +1
@@ -37,6 +37,7 @@ Read with the guide page [What a forge is](../guide-what-a-forge-is.md).
3737 - [Forge packs: a premade forge to deploy (`/packs`)](packs.md)
3838 - [Home paints its frame first](home-frame.md)
3939 - [Is v1.0.0 proven?](v1-proven.md)
40 - [Preview data of closed pulls, named and removed](preview-data.md)
4041 - [Looks of `/` signed out](home-looks.md)
4142 - [Milestones at the front door](front-milestones.md)
4243 - [Migration mode: a crew whose standing job is parity](migration-mode.md)
docs/features/maintenance-duties.md +9
@@ -22,6 +22,15 @@ The `merged-branches` duty finds branches of pulls that merged over seven days a
2222
2323 Tests: `test/maintain-merged-branches.test.ts`, `test/maintain-merged-web.test.ts`.
2424
25 ## Preview data (#13491)
26
27 The `preview-data` duty lists the data directories (`<data>/apps/<app>/pull-<n>`) of previews whose pull closed over seven days ago, with their size, oldest first, at most 100. A link, a directory that resolves outside `<data>/apps`, an open pull and an unknown pull are never listed.
28
29 - **On the web.** `/maintenance` folds the list under the other rows, with a required confirm box and **Remove all** (`POST /maintenance/previews/remove`, same-origin, a forge admin who is a person). A directory no longer on the list is left alone; the flash receipt names each app and pull and the bytes freed.
30 - **In the terminal.** The palette command "Preview data of pulls closed over 7 days ago" (`src/tui/screens/previewdata.ts`, admins who are people): the same rows; `d` removes all after a confirm, with the same receipt; Esc or q closes.
31
32 Tests: `test/maintain-previews.test.ts`, `test/tui-preview-data.test.ts`.
33
2534 ## Wait split (#13635)
2635
2736 The `wait-split` duty finds where a slow public page waits. Every six hours, from the forge's own host, it does three things for each of `/healthz`, `/`, `/pricing`, `/terms` and `/privacy`: it times a TCP+TLS handshake with the public origin (the edge), a GET of the public URL (to first byte and to the last byte), and a GET of the same path on `127.0.0.1:<port>`, which skips the edge and the tunnel. It also reads the `app;dur=` the forge wrote in `Server-Timing`. `src/maintain/wait_split.ts` splits the first-byte time into edge = connect, tunnel = first byte − connect − loopback, accept = loopback − app, and app. The largest share names the layer. A page under 1 s waits in `none`. It sends GET only, and its timeout is the proof's own 15 s. A slow or unanswered page is a finding, and every page is one line on `/maintenance`. Inside a cell, the bare handshake cannot pass the egress proxy: that line says so and counts the handshake as 0 ms.
docs/features/preview-data.md added +8
@@ -0,0 +1,8 @@
1 # Preview data: remove what closed pulls left behind (#13491)
2
3 A preview keeps its data under `<data>/apps/<app>/pull-<n>`. When the pull has been closed for over seven days, the `preview-data` maintenance duty lists that directory (oldest first, at most 100) and measures its size; page loads and the terminal never walk the trees, they show the size the duty last measured, or "size not measured yet". It only reads; a forge admin who is a person removes.
4
5 - **Web.** `/maintenance` has a fold "Preview data of pulls closed over 7 days ago"; tick the confirm box and press **Remove all**.
6 - **Terminal.** The palette command "Preview data of pulls closed over 7 days ago" (`src/tui/screens/previewdata.ts`): the same rows; `d` removes all after a confirm; Esc or q closes.
7
8 Both go through `removePreviewData` (`src/maintain/previews.ts`): only directories still listed at that moment are removed, never a link or anything outside `<data>/apps`, and the receipt names the first five directories (app and pull), a count of the rest, and the bytes freed. More in `maintenance-duties.md`. Tests: `test/maintain-previews.test.ts`, `test/tui-preview-data.test.ts`.
src/http/server.ts +2 −1
@@ -42,7 +42,7 @@ import { contributePage } from "../web/pages/contribute.ts";
4242 import { pipelinePage } from "../web/pages/pipeline.ts";
4343 import { adminDataPage, appDataCsv, appDataPage, dataPage } from "../web/pages/data.ts";
4444 import { fleetAct, fleetReceipt, fleetUndo } from "../web/pages/fleet-ops.ts";
45 import { maintenanceDuty, maintenancePage, maintenancePrepare, maintenanceResume, staleAction, mergedAction } from "../web/pages/maintenance.ts";
45 import { maintenanceDuty, maintenancePage, maintenancePrepare, maintenanceResume, staleAction, mergedAction, previewDataAction } from "../web/pages/maintenance.ts";
4646 import { maintenanceBudget } from "../web/pages/maintenance_budgets.ts";
4747 import { nightlyOutputPage, resiliencePage } from "../web/pages/resilience.ts";
4848 import { agentStartSet, billingLinksSet, limitsPage, limitsSet, quotasSet, signupSet, inviteMint, inviteRevoke } from "../web/pages/admin-limits.ts";
@@ -1138,6 +1138,7 @@ export function buildRouter(): Router<Handler> {
11381138 router.add("POST", "/maintenance/duties/:name", maintenanceDuty);
11391139 router.add("POST", "/maintenance/stale/:id/:what", staleAction);
11401140 router.add("POST", "/maintenance/branches/delete", mergedAction);
1141 router.add("POST", "/maintenance/previews/remove", previewDataAction);
11411142 router.add("GET", "/resilience/nightly/:job", nightlyOutputPage);
11421143 // A selection's spend and runs over time; literal, above /:owner.
11431144 router.add("GET", "/fleet", fleetPage);
src/maintain/previews.ts added +157
@@ -0,0 +1,157 @@
1 /**

Direction: a feature that changes no pipeline file, while the owner order puts #6534, #6611, #5276, #3316, #6217, #6428, #1242 and #1073 first. It waits until a person decides.

2 * Preview data (#13491): the data directories of previews whose pull closed over seven days ago,
3 * listed with their size on the maintenance board. The duty only reads; `removePreviewData` is
4 * pressed by a forge admin and removes exactly the directories the card showed that are still
5 * listed. Every path is built from a checked app id and a checked `pull-N` name, must be a real
6 * directory (never a link) and must resolve inside `<data>/apps`, or it is not listed.
7 */
8
9 import { lstatSync, readdirSync, realpathSync, rmSync } from "node:fs";
10 import { join, sep } from "node:path";
11 import type { App } from "../app.ts";
12 import { dirBytes } from "../apps/bytes.ts";
13 import { appsRoot, envDir, previewEnv } from "../apps/data.ts";
14 import type { User } from "../core/users.ts";
15 import { err, ok, type Result } from "../kernel/result.ts";
16 import { bytes as sizeWords } from "../system/format.ts";
17 import type { Duty } from "./duty.ts";
18 import { record } from "./runs.ts";
19
20 export const PREVIEW_DAYS = 7;
21 const DAY_MS = 86_400_000;
22 /** The most directories one card lists. */
23 export const PREVIEW_CAP = 100;
24 /** The receipt names this many directories, then a count. */
25 export const NAMED = 5;
26
27 export interface PreviewDir {
28 readonly appId: string;
29 readonly slug: string;
30 readonly pull: number;
31 readonly closedAt: string;
32 /** Undefined until the duty (or a removal) has measured it; a page load never walks the tree. */
33 readonly bytes: number | undefined;
34 }
35
36 /** Sizes the duty measured, by directory path; read by page loads, which never measure. */
37 const measuredBy = new WeakMap<App, Map<string, number>>();
38 const measuredOf = (app: App): Map<string, number> => {
39 let m = measuredBy.get(app);
40 if (m === undefined) measuredBy.set(app, (m = new Map()));
41 return m;
42 };
43
44 /** The size in words, or what stands in for it before the duty has measured. */
45 export const sizeText = (d: Pick<PreviewDir, "bytes">): string => (d.bytes === undefined ? "size not measured yet" : sizeWords(d.bytes));
46 /** The total of the sizes known, in words. */
47 export const totalText = (found: readonly PreviewDir[]): string =>
48 found.some((d) => d.bytes === undefined) ? "size not measured yet" : sizeWords(found.reduce((n, d) => n + (d.bytes ?? 0), 0));
49
50 /** What the form carries for one directory. */
51 export const keyOf = (d: Pick<PreviewDir, "appId" | "pull">): string => `${d.appId}/pull-${String(d.pull)}`;
52
53 interface Row { slug: string; closed_at: string | null; state: string }
54
55 const dirNames = (path: string): string[] => {
56 try { return readdirSync(path); } catch { return []; }
57 };
58
59 /** True when `path` is a real directory (not a link) that resolves inside the apps root. */
60 function inside(root: string, path: string): boolean {
61 try {
62 if (!lstatSync(path).isDirectory()) return false;
63 const real = realpathSync(path);
64 return real.startsWith(root + sep);
65 } catch {
66 return false;
67 }
68 }
69
70 /** Preview directories whose pull closed more than `days` ago, oldest first, at most `cap`. */
71 export async function previewDirs(app: App, now: Date, o: { measure?: boolean; days?: number; cap?: number } = {}): Promise<PreviewDir[]> {
72 const { measure = false, days = PREVIEW_DAYS, cap = PREVIEW_CAP } = o;
73 const data = app.config.dataDir;
74 const root = (() => { try { return realpathSync(appsRoot(data)); } catch { return undefined; } })();
75 if (root === undefined) return [];
76 const cutoff = new Date(now.getTime() - days * DAY_MS).toISOString();
77 const find = app.db.query<Row, [string, number]>(
78 `SELECT a.slug AS slug, i.closed_at AS closed_at, i.state AS state
79 FROM app a JOIN issue i ON i.repo_id = a.repo_id AND i.kind = 'pull'
80 WHERE a.id = ?1 AND i.number = ?2`,
81 );
82 const found: Omit<PreviewDir, "bytes">[] = [];
83 for (const appId of dirNames(root)) {
84 if (!/^[0-9a-z]+$/i.test(appId)) continue;
85 for (const name of dirNames(join(root, appId))) {
86 const m = /^pull-([1-9]\d{0,9})$/.exec(name);
87 if (m === null) continue;
88 const pull = Number(m[1]);
89 const row = find.get(appId, pull);
90 if (row === null || row.state !== "closed" || row.closed_at === null || row.closed_at >= cutoff) continue;
91 if (!inside(root, envDir(data, appId, previewEnv(pull)))) continue;
92 found.push({ appId, slug: row.slug, pull, closedAt: row.closed_at });
93 }
94 }
95 found.sort((a, b) => a.closedAt.localeCompare(b.closedAt) || a.slug.localeCompare(b.slug) || a.pull - b.pull);
96 const out: PreviewDir[] = [];
97 for (const f of found.slice(0, cap)) {
98 const path = envDir(data, f.appId, previewEnv(f.pull));
99 const sizes = measuredOf(app);
100 if (measure) sizes.set(path, await dirBytes(path));
101 out.push({ ...f, bytes: sizes.get(path) });
102 }
103 return out;
104 }
105
106 export type PreviewError = "forbidden" | "nothing" | "failed";
107 export const PREVIEW_WORDS: Record<PreviewError, string> = {
108 forbidden: "Only a forge admin may do this.",
109 nothing: "None of those directories is still listed; the card was out of date.",
110 failed: "The forge could not remove them.",
111 };
112
113 /**
114 * Remove the directories the card showed (`shown`: lines from `keyOf`), each only if still listed
115 * now.
116 */
117 export async function removePreviewData(app: App, admin: User | undefined, shown: readonly string[], now = new Date()): Promise<Result<{ says: string }, PreviewError>> {
118 if (admin === undefined || !admin.isAdmin || admin.kind === "agent") return err("forbidden");
119 const want = new Set(shown);
120 const gone: PreviewDir[] = [];
121 let refused = 0;
122 for (const d of await previewDirs(app, now, { measure: true })) {
123 if (!want.has(keyOf(d))) continue;
124 try {
125 rmSync(envDir(app.config.dataDir, d.appId, previewEnv(d.pull)), { recursive: true, force: true });
126 measuredOf(app).delete(envDir(app.config.dataDir, d.appId, previewEnv(d.pull)));
127 gone.push(d);
128 } catch {
129 refused++;
130 }
131 }
132 if (gone.length === 0) return err(refused > 0 ? "failed" : "nothing");
133 const total = gone.reduce((n, d) => n + (d.bytes ?? 0), 0);
134 const names = gone.slice(0, NAMED).map((d) => `${d.slug} #${d.pull}`).join(", ");
135 const says = `removed ${gone.length} preview data director${gone.length === 1 ? "y" : "ies"}, ${sizeWords(total)} (pressed by ${admin.username}): ${names}${gone.length > NAMED ? `, and ${gone.length - NAMED} more` : ""}`;
136 record(app.db, { duty: "preview-data", started: now, finished: now, findings: [], actions: [says], outcome: "ok" });
137 return ok({ says });
138 }
139
140 export const duty: Duty = {
141 name: "preview-data",
142 intervalMs: 6 * 3_600_000,
143 budget: { runsPerDay: 8, dollarsPerDay: 1, reason: "every six hours is four a day; the rest is room for a catch-up, and it spends no money" },
144 async run({ now, app }) {
145 if (app === undefined) return { findings: [], actions: [] };
146 const found = await previewDirs(app, now, { measure: true });
147 if (found.length === 0) return { findings: [], actions: [] };
148 const total = found.reduce((n, d) => n + (d.bytes ?? 0), 0);
149 return {
150 findings: [{
151 summary: `${found.length} preview data director${found.length === 1 ? "y" : "ies"} of pulls closed over ${PREVIEW_DAYS} days ago, ${sizeWords(total)}`,
152 probe: { kind: "measurement", what: "preview data directories whose pull closed over seven days ago", number: found.length, at: now.toISOString() },
153 }],
154 actions: [],
155 };
156 },
157 };
src/maintain/registry.ts +2 −1
@@ -10,12 +10,13 @@ import { duty as backupGrowth } from "./duties/backup_growth.ts";
1010 import { duty as v1Proven } from "./duties/v1_proven.ts";
1111 import { duty as backupAge } from "./duties/backup_age.ts";
1212 import { duty as mergedBranches } from "./duties/merged_branches.ts";
13 import { duty as previewData } from "./previews.ts";
1314 import { duty as diskFree } from "./duties/disk_free.ts";
1415 import { duty as staleIssues } from "./duties/stale_issues.ts";
1516 import { duty as waitSplit } from "./duties/wait_split.ts";
1617 import { dutyIdle } from "./duties/duty_idle.ts";
1718
18 const others: readonly Duty[] = [pullsWaiting, ruleWatcher, efficiencyWatcher, flowKeeper, visitorRetention, backupGrowth, backupAge, v1Proven, diskFree, mergedBranches, staleIssues, waitSplit];
19 const others: readonly Duty[] = [pullsWaiting, ruleWatcher, efficiencyWatcher, flowKeeper, visitorRetention, backupGrowth, backupAge, v1Proven, diskFree, mergedBranches, previewData, staleIssues, waitSplit];
1920
2021 /** The duties plus the one that watches them for silence. */
2122 export const duties: readonly Duty[] = [...others, dutyIdle(others)];
src/tui/screens/entries.ts +2
@@ -77,6 +77,7 @@ import { entry as maintainRuns } from "./maintainruns.ts";
7777 import { entry as dutyBudgets } from "./dutybudgets.ts";
7878 import { entry as stalePulls } from "./stalepulls.ts";
7979 import { entry as mergedBranches } from "./mergedbranches.ts";
80 import { entry as previewData } from "./previewdata.ts";
8081 import { entry as dutySwitch } from "./dutyswitch.ts";
8182 import { entry as mirrorPacksEntry } from "./mirrorpacks.ts";
8283 import { entry as packReposEntry } from "./packrepos.ts";
@@ -151,6 +152,7 @@ export const ENTRIES: readonly PaletteEntry[] = [
151152 maintenance,
152153 stalePulls,
153154 mergedBranches,
155 previewData,
154156 maintainRuns,
155157 dutySwitch,
156158 dutyBudgets,
src/tui/screens/previewdata.ts added +82
@@ -0,0 +1,82 @@
1 /**
2 * Preview data in the terminal (#13491): the /maintenance card through the same finder
3 * (`previewDirs`) and the same action (`removePreviewData`), behind the same gate, a forge admin
4 * who is a person. `d` removes every listed directory, after a confirm; the receipt is the one the
5 * page flashes. Esc or q closes it.
6 */
7
8 import { PREVIEW_DAYS, PREVIEW_WORDS, keyOf, previewDirs, removePreviewData, sizeText, type PreviewDir } from "../../maintain/previews.ts";
9 import type { Ctx, Screen } from "../app/app.ts";
10 import { binding } from "../app/keys.ts";
11 import type { Session } from "../session.ts";
12 import { style } from "../term/style.ts";
13 import { Scroll, confirmScreen, drawLines } from "./common.ts";
14 import type { PaletteEntry } from "./entries.ts";
15
16 const mayAct = (session: Session): boolean => session.user?.isAdmin === true && session.user.kind !== "agent";
17
18 /** The directories the card lists; none for anyone the gate refuses. */
19 export async function listed(session: Session): Promise<PreviewDir[]> {
20 return mayAct(session) ? previewDirs(session.app, new Date()) : [];
21 }
22
23 /** One directory a line: app and pull, size, close day; the same rows as the page. */
24 export function previewLines(found: readonly PreviewDir[]): string[] {
25 if (found.length === 0) return ["None."];
26 return found.map((d) => ` ${d.slug} #${String(d.pull)} ${sizeText(d)} closed ${d.closedAt.slice(0, 10)}`);
27 }
28
29 /** Remove what `found` lists; what it says, or nothing for anyone the gate refuses. */
30 export async function removeListed(session: Session, found: readonly PreviewDir[]): Promise<string> {
31 if (!mayAct(session) || found.length === 0) return "";
32 const done = await removePreviewData(session.app, session.user, found.map(keyOf));
33 return done.ok ? done.value.says : PREVIEW_WORDS[done.error];
34 }
35
36 export function previewDataScreen(session: Session, ctx: Ctx): Screen {
37 let found: PreviewDir[] = [];
38 let note = "";
39 const load = (): void => void listed(session).then((f) => { found = f; ctx.invalidate(); });
40 load();
41 const screen: Screen = {
42 title: `Preview data of pulls closed over ${String(PREVIEW_DAYS)} days ago`,
43 keys: [binding(["d"], "remove all"), binding(["esc", "q"], "back")],
44 render(area, s) {
45 const c = ctx.theme.c;
46 const lines = [...previewLines(found), ...(note === "" ? [] : ["", note])];
47 drawLines(s, area, c, new Scroll(), lines.map((text) => [{ text, style: style(c.fg, c.bg) }]));
48 },
49 key(e) {
50 if (e.ctrl || e.alt) return true;
51 if (e.key === "esc" || e.key === "q") ctx.pop();
52 else if (e.key === "d") {
53 if (!mayAct(session) || found.length === 0) return true;
54 const shown = found;
55 ctx.push(
56 confirmScreen(ctx, screen, {
57 title: `Remove ${String(shown.length)} preview data directories?`,
58 body: ["Their files are deleted for good; the receipt names the first few and counts the rest."],
59 actions: [{ label: "Cancel", variant: "ghost" }, { label: "Remove all", variant: "danger" }],
60 onChoose: (i) => {
61 if (i !== 1) return;
62 void removeListed(session, shown).then((says) => { note = says; load(); });
63 },
64 }),
65 );
66 return true;
67 } else return false;
68 ctx.invalidate();
69 return true;
70 },
71 };
72 return screen;
73 }
74
75 export const entry: PaletteEntry = {
76 id: "preview-data",
77 title: "Preview data of pulls closed over 7 days ago",
78 hint: "remove the closed pulls' data directories in one receipted step",
79 after: ["maintenance", "maintenance-runs", "merged-branches"],
80 offered: mayAct,
81 run: (session, ctx) => ctx.push(previewDataScreen(session, ctx)),
82 };
src/web/pages/maintenance.ts +38
@@ -18,6 +18,7 @@ import { flashTo, page } from "../layout.ts";
1818 import { badge, btn, disclosure, field } from "../ui.ts";
1919 import { STALE_DAYS, STALE_WORDS, closeStale, nudge, stalePulls } from "../../maintain/stale_pulls.ts";
2020 import { MERGED_DAYS, MERGED_WORDS, deleteMerged, lineOf, mergedBranches, type MergedBranch } from "../../maintain/duties/merged_branches.ts";
21 import { PREVIEW_DAYS, PREVIEW_WORDS, keyOf, previewDirs, removePreviewData, sizeText, totalText, type PreviewDir } from "../../maintain/previews.ts";
2122 import { writePauseFold } from "./maintenance_pause.ts";
2223 import { budgetRows } from "./maintenance_budgets.ts";
2324
@@ -56,6 +57,7 @@ export async function maintenancePage(ctx: Ctx): Promise<Response> {
5657 const ran = duties(ctx.app.config.dataDir);
5758 const states = dutyStates(ctx.app.db);
5859 const mergedPending = await mergedBranches(ctx.app, new Date());
60 const previewPending = await previewDirs(ctx.app, new Date());

Medium: sizes up to 100 directory trees on every page load. Cache it or move the sizing out of the request.

5961 const stalePending = stalePulls(ctx.app.db, new Date());
6062 const stale = stalePending.length;
6163 const overdue = ran.filter((d) => d.overdue).length;
@@ -87,6 +89,9 @@ export async function maintenancePage(ctx: Ctx): Promise<Response> {
8789 ${mergedPending.length > 0
8890 ? fold("merged-branches", `Branches of pulls merged over ${String(MERGED_DAYS)} days ago`, String(mergedPending.length), true, mergedRows(ctx.auth.user!.kind !== "agent", mergedPending))
8991 : null}
92 ${previewPending.length > 0
93 ? fold("preview-data", `Preview data of pulls closed over ${String(PREVIEW_DAYS)} days ago`, `${String(previewPending.length)} · ${totalText(previewPending)}`, true, previewRows(ctx.auth.user!.kind !== "agent", previewPending))
94 : null}
9095 ${idle
9196 ? fold("maintenance-prepare", "Prepare for a power-off", `${String(ABOUT_TO_STOP_ROWS.length)} things stop`, false,
9297 html`<p id="maintenance-standby" class="muted">${s.standby.says}</p>
@@ -217,3 +222,36 @@ export async function mergedAction(ctx: Ctx): Promise<Response> {
217222 const done = await deleteMerged(ctx.app, user, shown);
218223 return flashTo(ctx, "/maintenance", done.ok ? { message: done.value.says.slice(0, 600) } : { tone: "danger", message: MERGED_WORDS[done.error] });
219224 }
225
226 /**
227 * The preview data directories as one card: each name and size is the preview, one button removes
228 * them all.
229 */
230 function previewRows(withActions: boolean, found: readonly PreviewDir[]): Html {
231 return html`<dl class="status-rows">${found.map(
232 (d) => html`<div class="status-row" data-preview="${keyOf(d)}"><dt>${d.slug} #${String(d.pull)}</dt><dd>${sizeText(d)} <span class="muted">closed ${d.closedAt.slice(0, 10)}</span></dd></div>`,
233 )}</dl>
234 ${withActions
235 ? html`<form method="post" action="/maintenance/previews/remove" class="stack">
236 ${found.map((d) => html`<input type="hidden" name="shown" value="${keyOf(d)}">`)}
237 ${field({ name: "confirm", type: "checkbox", value: "yes", required: true, label: `Remove these ${String(found.length)} data directories` })}
238 <div class="form-actions">${btn("Remove all", { variant: "danger" })}</div>
239 </form>`
240 : null}`;
241 }
242
243 /**
244 * POST /maintenance/previews/remove: remove the directories the card showed, receipted. Same-origin
245 * by readForm.
246 */
247 export async function previewDataAction(ctx: Ctx): Promise<Response> {
248 const user = ctx.auth.user;
249 if (user === undefined || !user.isAdmin || user.kind === "agent")
250 return flashTo(ctx, "/maintenance", { tone: "danger", message: PREVIEW_WORDS.forbidden });
251 const form = await readForm(ctx, "admin");
252 if (form instanceof Response) return form;
253 if (form.get("confirm") !== "yes") return flashTo(ctx, "/maintenance", { tone: "danger", message: "Tick the box to confirm the removal." });
254 const shown = form.getAll("shown").filter((v): v is string => typeof v === "string" && v.length <= 80).slice(0, 500);
255 const done = await removePreviewData(ctx.app, user, shown);
256 return flashTo(ctx, "/maintenance", done.ok ? { message: done.value.says.slice(0, 600) } : { tone: "danger", message: PREVIEW_WORDS[done.error] });
257 }
test/maintain-previews.test.ts added +171
@@ -0,0 +1,171 @@
1 /**

Medium: add tests for duty.run (finding and total size, empty case), the 100 cap, and the exact 7-day cutoff.

2 * Preview data (#13491): the listing, the confirmed removal, and that nothing outside <data>/apps
3 * is touched.
4 */
5 import { afterAll, beforeAll, expect, test } from "bun:test";
6 import { existsSync, mkdirSync, symlinkSync, writeFileSync } from "node:fs";
7 import { mkdtemp, rm } from "node:fs/promises";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { createApp, type App } from "../src/app.ts";
11 import { createRepo } from "../src/core/repos.ts";
12 import { createUser, type User } from "../src/core/users.ts";
13 import { serve, type Servers } from "../src/http/server.ts";
14 import { id } from "../src/kernel/id.ts";
15 import { NAMED, PREVIEW_CAP, duty, previewDirs, removePreviewData } from "../src/maintain/previews.ts";
16 import { quietLogs } from "./helpers/globals.ts";
17
18 quietLogs();
19 let dir = "", outside = "", app: App, servers: Servers, admin: User, person: User, appId = "", seq = 0, cookie = "", pcookie = "";
20 const day = 86_400_000;
21 const dataOf = (n: number): string => join(dir, "apps", appId, `pull-${n}`);
22
23 async function user(name: string, isAdmin: boolean): Promise<User> {
24 const made = await createUser(app.db, { username: name, email: `${name}@forge.test`, password: "a-good-passphrase", origin: "seed", trusted: true, ...(isAdmin ? { isAdmin: true } : {}) });
25 if (!made.ok) throw new Error(name);
26 return made.value;
27 }
28 async function login(username: string): Promise<string> {
29 const res = await fetch(`${servers.url}/signin`, {
30 method: "POST", redirect: "manual",
31 headers: { "content-type": "application/x-www-form-urlencoded", origin: servers.url },
32 body: new URLSearchParams({ username, password: "a-good-passphrase" }),
33 });
34 return (res.headers.get("set-cookie") ?? "").split(";")[0] ?? "";
35 }
36 /**
37 * A pull of the app with a preview directory holding `size` bytes; closed `ago` days back, or open.
38 */
39 function seed(ago: number | "open", size = 1000): number {
40 const n = ++seq, repoId = app.db.query<{ id: string }, []>("SELECT id FROM repo LIMIT 1").get()!.id, issue = id();
41 const at = new Date(Date.now() - (ago === "open" ? 0 : ago) * day).toISOString();
42 app.db.query("INSERT INTO issue (id, repo_id, number, kind, title, body, author_id, state, created_at, updated_at, closed_at) VALUES (?1, ?2, ?3, 'pull', 'P', '', ?4, ?5, ?6, ?6, ?7)")
43 .run(issue, repoId, n, admin.id, ago === "open" ? "open" : "closed", at, ago === "open" ? null : at);
44 mkdirSync(join(dataOf(n), "data"), { recursive: true });
45 writeFileSync(join(dataOf(n), "data", "db"), "x".repeat(size));
46 return n;
47 }
48 const post = (c: string, body: string, origin = servers.url): Promise<Response> =>
49 fetch(`${servers.url}/maintenance/previews/remove`, { method: "POST", redirect: "manual", headers: { "content-type": "application/x-www-form-urlencoded", origin, cookie: c }, body });
50
51 beforeAll(async () => {
52 dir = await mkdtemp(join(tmpdir(), "forge-previews-"));
53 outside = await mkdtemp(join(tmpdir(), "forge-previews-outside-"));
54 writeFileSync(join(outside, "precious"), "keep");
55 app = createApp({ dataDir: dir, port: 0 });
56 servers = await serve(app);
57 admin = await user("boss", true);
58 person = await user("pat", false);
59 const made = await createRepo(app.db, app.reposRoot, { ownerId: admin.id, name: "r", visibility: "public", description: "r" });
60 if (!made.ok) throw new Error("repo");
61 appId = id().toLowerCase();
62 app.db.query("INSERT INTO app (id, repo_id, slug, access) VALUES (?1, ?2, 'demo', 'public')").run(appId, made.value.id);
63 cookie = await login("boss");
64 pcookie = await login(person.username);
65 });
66 afterAll(async () => {
67 servers.stop();
68 app.close();
69 await rm(dir, { recursive: true, force: true });
70 await rm(outside, { recursive: true, force: true });
71 });
72
73 test("lists only previews of pulls closed over seven days, with their size", async () => {
74 const old = seed(10, 2048), recent = seed(3), open = seed("open");
75 mkdirSync(join(dir, "apps", appId, "production", "data"), { recursive: true });
76 mkdirSync(join(dir, "apps", appId, "pull-99"), { recursive: true }); // no such pull
77 const found = await previewDirs(app, new Date());
78 expect(found.map((d) => d.pull)).toContain(old);
79 expect(found.map((d) => d.pull)).not.toContain(recent);
80 expect(found.map((d) => d.pull)).not.toContain(open);
81 expect(found.map((d) => d.pull)).not.toContain(99);
82 expect(found.find((d) => d.pull === old)!.bytes).toBeUndefined(); // a listing never walks the trees
83 expect((await previewDirs(app, new Date(), { measure: true })).find((d) => d.pull === old)!.bytes).toBe(2048);
84 expect((await previewDirs(app, new Date())).find((d) => d.pull === old)!.bytes).toBe(2048); // now remembered
85 const page = await (await fetch(`${servers.url}/maintenance`, { headers: { cookie } })).text();
86 expect(page).toContain('id="preview-data"');
87 expect(page).toContain(`data-preview="${appId}/pull-${old}"`);
88 expect(page).toContain('action="/maintenance/previews/remove"');
89 });
90
91 test("a link out of the data root is never listed and what it points at survives", async () => {
92 const n = seed(20);
93 await rm(dataOf(n), { recursive: true, force: true });
94 symlinkSync(outside, dataOf(n));
95 expect((await previewDirs(app, new Date())).map((d) => d.pull)).not.toContain(n);
96 const res = await post(cookie, new URLSearchParams({ shown: `${appId}/pull-${n}`, confirm: "yes" }).toString());
97 expect(res.status).toBeLessThan(400);
98 expect(existsSync(join(outside, "precious"))).toBe(true);
99 });
100
101 test("nothing is removed without a confirmed admin's same-origin post", async () => {
102 const n = seed(15);
103 const body = (confirm: boolean): string => new URLSearchParams({ shown: `${appId}/pull-${n}`, ...(confirm ? { confirm: "yes" } : {}) }).toString();
104 expect((await post(pcookie, body(true))).status).toBe(403);
105 expect((await post(cookie, body(true), "https://evil.example")).status).toBe(403);
106 await post(cookie, body(false));
107 expect(existsSync(dataOf(n))).toBe(true);
108 expect((await removePreviewData(app, person, [`${appId}/pull-${n}`])).ok).toBe(false);
109 expect(existsSync(dataOf(n))).toBe(true);
110 });
111
112 test("a confirmed removal deletes the shown directories only, and receipts them", async () => {
113 const a = seed(30, 500), b = seed(31, 500), kept = seed(2), unshown = seed(40);
114 const res = await removePreviewData(app, admin, [`${appId}/pull-${a}`, `${appId}/pull-${b}`, `${appId}/pull-${kept}`, `../../${outside}`, `${appId}/../x`]);
115 expect(res.ok).toBe(true);
116 if (res.ok) expect(res.value.says).toContain("removed 2 preview data directories");
117 expect(existsSync(dataOf(a))).toBe(false);
118 expect(existsSync(dataOf(b))).toBe(false);
119 expect(existsSync(dataOf(kept))).toBe(true);
120 expect(existsSync(dataOf(unshown))).toBe(true);
121 expect(existsSync(join(dir, "apps", appId, "production"))).toBe(true);
122 expect(existsSync(join(outside, "precious"))).toBe(true);
123 const again = await removePreviewData(app, admin, [`${appId}/pull-${a}`]);
124 expect(again.ok).toBe(false);
125 });
126
127 test("the seven-day boundary is exact: closed exactly seven days ago is not yet listed", async () => {
128 const now = new Date("2026-06-15T12:00:00.000Z");
129 const closeAt = (n: number, at: Date): void => void app.db.query("UPDATE issue SET closed_at = ?1 WHERE number = ?2 AND kind = 'pull'").run(at.toISOString(), n);
130 const edge = seed(1), over = seed(1);
131 closeAt(edge, new Date(now.getTime() - 7 * day));
132 closeAt(over, new Date(now.getTime() - 7 * day - 1));
133 const pulls = (await previewDirs(app, now)).map((d) => d.pull);
134 expect(pulls).not.toContain(edge);
135 expect(pulls).toContain(over);
136 });
137
138 test("the duty names the count and total size, and nothing when nothing is listed", async () => {
139 const ctx = { db: app.db, bus: app.bus, now: new Date(), app };
140 const n = seed(12, 4096);
141 const res = await duty.run(ctx);
142 expect(res.actions).toEqual([]);
143 expect(res.findings).toHaveLength(1);
144 expect(res.findings[0]!.summary).toMatch(/^\d+ preview data directories of pulls closed over 7 days ago, /);
145 expect(res.findings[0]!.probe?.number).toBe((await previewDirs(app, ctx.now)).length);
146 expect((await previewDirs(app, ctx.now)).find((d) => d.pull === n)!.bytes).toBe(4096);
147 expect((await duty.run({ ...ctx, app: undefined })).findings).toEqual([]);
148 const left = (await previewDirs(app, ctx.now)).map((d) => `${d.appId}/pull-${d.pull}`);
149 expect((await removePreviewData(app, admin, left)).ok).toBe(true);
150 expect((await duty.run(ctx)).findings).toEqual([]);
151 });
152
153 test("one card lists at most the cap, and the receipt names the first few and counts the rest", async () => {
154 for (let i = 0; i < PREVIEW_CAP + 5; i++) seed(9, 1);
155 const listed = await previewDirs(app, new Date());
156 expect(listed).toHaveLength(PREVIEW_CAP);
157 const res = await removePreviewData(app, admin, listed.map((d) => `${d.appId}/pull-${d.pull}`));
158 expect(res.ok).toBe(true);
159 if (!res.ok) return;
160 expect(res.value.says).toContain(`removed ${PREVIEW_CAP} preview data directories`);
161 expect(res.value.says).toContain(`and ${PREVIEW_CAP - NAMED} more`);
162 expect(res.value.says.length).toBeLessThan(600);
163 expect((await previewDirs(app, new Date())).length).toBe(5);
164 });
165
166 test("the maintenance page does not measure directories", async () => {
167 const n = seed(30, 123);
168 const page = await (await fetch(`${servers.url}/maintenance`, { headers: { cookie } })).text();
169 expect(page).toContain(`data-preview="${appId}/pull-${n}"`);
170 expect(page).toContain("size not measured yet");
171 });
test/palette-roster.test.ts +1
@@ -138,6 +138,7 @@ test("a site-admin person is offered the whole roster", () => {
138138 ["maintenance", "Maintenance", "prepare the forge for a power-off, check the drain, or end maintenance"],
139139 ["stale-pulls", "Pulls untouched for 14 days", "ask whether a stale pull is still wanted, or close one an agent opened"],
140140 ["merged-branches", "Branches of pulls merged over 7 days ago", "delete the merged pulls' branches in one receipted step"],
141 ["preview-data", "Preview data of pulls closed over 7 days ago", "remove the closed pulls' data directories in one receipted step"],
141142 ["maintenance-runs", "Maintenance duty runs", "what each duty found, did and probed"],
142143 ["maintenance-duties", "Pause or resume a maintenance duty", "switch a duty off, or on again after it paused itself"],
143144 ["maintenance-budgets", "Set a maintenance duty's budget", "runs per day and dollars per day a duty may use"],
test/tui-preview-data.test.ts added +102
@@ -0,0 +1,102 @@
1 /** Preview data in the terminal (#13491): the same rows and the same gate as /maintenance. */
2 import { afterAll, beforeAll, expect, test } from "bun:test";
3 import { existsSync, mkdirSync, writeFileSync } from "node:fs";
4 import { mkdtemp, rm } from "node:fs/promises";
5 import { tmpdir } from "node:os";
6 import { join } from "node:path";
7 import { createApp, type App } from "../src/app.ts";
8 import { createRepo } from "../src/core/repos.ts";
9 import { createUser, type User } from "../src/core/users.ts";
10 import { id } from "../src/kernel/id.ts";
11 import { theme } from "../src/tui/ui/theme.ts";
12 import type { Session } from "../src/tui/session.ts";
13 import { entry } from "../src/tui/screens/previewdata.ts";
14 import { listed, previewDataScreen, previewLines, removeListed } from "../src/tui/screens/previewdata.ts";
15 import { ENTRIES } from "../src/tui/screens/entries.ts";
16 import type { Ctx, Screen } from "../src/tui/app/app.ts";
17 import type { KeyEvent } from "../src/tui/term/input.ts";
18 import { quietLogs } from "./helpers/globals.ts";
19
20 quietLogs();
21 let dir = "", app: App, admin: User, person: User, appId = "";
22
23 async function user(name: string, isAdmin: boolean): Promise<User> {
24 const made = await createUser(app.db, { username: name, email: `${name}@forge.test`, password: "a-good-passphrase", origin: "seed", trusted: true, ...(isAdmin ? { isAdmin: true } : {}) });
25 if (!made.ok) throw new Error(name);
26 return made.value;
27 }
28 const as = (u: User): Session => ({ app, user: u }) as unknown as Session;
29
30 beforeAll(async () => {
31 dir = await mkdtemp(join(tmpdir(), "forge-tui-previews-"));
32 app = createApp({ dataDir: dir, port: 0 });
33 admin = await user("boss", true);
34 person = await user("pat", false);
35 const made = await createRepo(app.db, app.reposRoot, { ownerId: admin.id, name: "r", visibility: "public", description: "r" });
36 if (!made.ok) throw new Error("repo");
37 appId = id().toLowerCase();
38 app.db.query("INSERT INTO app (id, repo_id, slug, access) VALUES (?1, ?2, 'demo', 'public')").run(appId, made.value.id);
39 const at = new Date(Date.now() - 10 * 86_400_000).toISOString();
40 app.db.query("INSERT INTO issue (id, repo_id, number, kind, title, body, author_id, state, created_at, updated_at, closed_at) VALUES (?1, ?2, 5, 'pull', 'P', '', ?3, 'closed', ?4, ?4, ?4)")
41 .run(id(), made.value.id, admin.id, at);
42 mkdirSync(join(dir, "apps", appId, "pull-5", "data"), { recursive: true });
43 writeFileSync(join(dir, "apps", appId, "pull-5", "data", "db"), "x".repeat(2048));
44 });
45 afterAll(async () => {
46 app.close();
47 await rm(dir, { recursive: true, force: true });
48 });
49
50 test("the palette offers it to an admin who is a person, and it is registered", () => {
51 expect(entry.offered?.(as(admin))).toBe(true);
52 expect(entry.offered?.(as(person))).toBe(false);
53 expect(ENTRIES.some((e) => e.id === "preview-data")).toBe(true);
54 });
55
56 test("a person who is not an admin sees nothing and removes nothing", async () => {
57 expect(await listed(as(person))).toEqual([]);
58 const found = await listed(as(admin));
59 expect(await removeListed(as(person), found)).toBe("");
60 expect(existsSync(join(dir, "apps", appId, "pull-5"))).toBe(true);
61 });
62
63 test("the admin sees the row and removing it gives the receipt", async () => {
64 const found = await listed(as(admin));
65 expect(previewLines(found).join("\n")).toContain("demo #5");
66 const says = await removeListed(as(admin), found);
67 expect(says).toContain("removed 1 preview data directory");
68 expect(existsSync(join(dir, "apps", appId, "pull-5"))).toBe(false);
69 expect(previewLines(await listed(as(admin)))).toEqual(["None."]);
70 });
71
72 test("d opens a confirm and nothing is removed until it is chosen", async () => {
73 const at = new Date(Date.now() - 10 * 86_400_000).toISOString();
74 const repoId = app.db.query<{ id: string }, []>("SELECT id FROM repo LIMIT 1").get()!.id;
75 app.db.query("INSERT INTO issue (id, repo_id, number, kind, title, body, author_id, state, created_at, updated_at, closed_at) VALUES (?1, ?2, 6, 'pull', 'P', '', ?3, 'closed', ?4, ?4, ?4)")
76 .run(id(), repoId, admin.id, at);
77 const path = join(dir, "apps", appId, "pull-6");
78 mkdirSync(join(path, "data"), { recursive: true });
79 const pushed: Screen[] = [];
80 const ctx = { theme: theme("dark"), invalidate() {}, push: (s: Screen) => pushed.push(s), pop() {}, hit() {} } as unknown as Ctx;
81 const press = (s: Screen, k: string): boolean => s.key({ type: "key", key: k, ctrl: false, alt: false, shift: false } satisfies KeyEvent);
82 const screen = previewDataScreen(as(admin), ctx);
83 await Bun.sleep(50); // real-timer: the screen loads its listing off the event loop; no hook to await
84 press(screen, "d");
85 expect(pushed).toHaveLength(1);
86 expect(existsSync(path)).toBe(true); // asked, not done
87 press(pushed[0]!, "esc"); // cancel
88 await Bun.sleep(50); // real-timer: the screen loads its listing off the event loop; no hook to await
89 expect(existsSync(path)).toBe(true);
90 press(screen, "d");
91 const confirm = pushed[1]!;
92 press(confirm, "right");
93 press(confirm, "enter"); // Remove all
94 const there = (): boolean => existsSync(path);
95 const tick = (): Promise<void> => Bun.sleep(20); // real-timer: removal runs async inside the confirm; poll the disk
96 for (let i = 0; i < 100 && there(); i++) await tick();
97 expect(there()).toBe(false);
98 // a non-admin's `d` opens nothing
99 const before = pushed.length;
100 press(previewDataScreen(as(person), ctx), "d");
101 expect(pushed.length).toBe(before);
102 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs