Public: anyone can read this; the crew can change it.
A maintenance duty names preview data directories whose pull is closed for over seven days and offers to remove them #13746
Check · Review · Land · Live done — Landed and live in production.
mason/forge:mason/13491-a-maintenance-duty-names-preview-data-di → main 4 commits fork 3 threads · 3 unresolved
11 files
changed +539
−2
docs/features/README.md +1
| @@ -134,6 +134,7 @@ Read with the guide page [Tell the crew](../guide-tell-the-crew.md). | ||
| 134 | 134 | - [](people.md) |
| 135 | 135 | - [](file-holds.md) |
| 136 | 136 | - [](maintain.md) |
| 137 | - [](preview-dirs.md) | |
| 137 | 138 | - [](asks-mute-undo.md) |
| 138 | 139 | - [](next-up.md) |
| 139 | 140 | - [](next-up-why.md) |
docs/features/preview-dirs.md added +5
| @@ -0,0 +1,5 @@ | ||
| 1 | # Preview data of closed pulls | |
| 2 | ||
| 3 | `/maintenance` lists, under its other cards, each preview directory (`<data>/apps/<app>/pull-<n>`) whose pull closed over seven days ago. The list never walks a disk, so the board stays fast: a size shows once the duty (every six hours) has measured it, and "size not measured yet" before; the receipt always carries a fresh size. Open pulls, pulls the forge has no row for, links and anything not named `pull-<n>` are never listed; every path is resolved and must lie inside the apps root. | |
| 4 | ||
| 5 | One ticked box and "Remove all" (a forge admin who is a person; same-origin) removes exactly the directories the card showed that are still listed, and writes a receipt naming each with its size (`/maintenance/runs`). Nothing is removed without it. The `preview-dirs` duty only reads and names the count and bytes every six hours. The terminal's "Preview data of pulls closed over 7 days ago" screen does the same (`d`, after a confirm). Code: `src/maintain/previews.ts`. |
src/http/server.ts +2 −1
| @@ -42,7 +42,7 @@ import { contributePage } from "../web/pages/contribute.ts"; | ||
| 42 | 42 | import { pipelinePage } from "../web/pages/pipeline.ts"; |
| 43 | 43 | import { adminDataPage, appDataCsv, appDataPage, dataPage } from "../web/pages/data.ts"; |
| 44 | 44 | import { fleetAct, fleetReceipt, fleetUndo } from "../web/pages/fleet-ops.ts"; |
| 45 | import { maintenanceDuty, maintenancePage, maintenancePrepare, maintenanceResume, staleAction, mergedAction } from "../web/pages/maintenance.ts"; | |
| 45 | import { maintenanceDuty, maintenancePage, maintenancePrepare, maintenanceResume, staleAction, mergedAction, previewAction } from "../web/pages/maintenance.ts"; | |
| 46 | 46 | import { maintenanceBudget } from "../web/pages/maintenance_budgets.ts"; |
| 47 | 47 | import { nightlyOutputPage, resiliencePage } from "../web/pages/resilience.ts"; |
| 48 | 48 | import { agentStartSet, billingLinksSet, limitsPage, limitsSet, quotasSet, signupSet, inviteMint, inviteRevoke } from "../web/pages/admin-limits.ts"; |
| @@ -1138,6 +1138,7 @@ export function buildRouter(): Router<Handler> { | ||
| 1138 | 1138 | router.add("POST", "/maintenance/duties/:name", maintenanceDuty); |
| 1139 | 1139 | router.add("POST", "/maintenance/stale/:id/:what", staleAction); |
| 1140 | 1140 | router.add("POST", "/maintenance/branches/delete", mergedAction); |
| 1141 | router.add("POST", "/maintenance/previews/remove", previewAction); | |
| 1141 | 1142 | router.add("GET", "/resilience/nightly/:job", nightlyOutputPage); |
| 1142 | 1143 | // A selection's spend and runs over time; literal, above /:owner. |
| 1143 | 1144 | router.add("GET", "/fleet", fleetPage); |
src/maintain/previews.ts added +191
| @@ -0,0 +1,191 @@ | ||
| 1 | /** | |
| 2 | * Preview directories of closed pulls (#13491): the data a preview left under | |
| 3 | * `<FORGE_APP_DATA>/apps/<app id>/pull-<n>` whose pull closed over seven days ago, listed with | |
| 4 | * their size on /maintenance. The duty only reads; the removal is `removePreviewDirs`, pressed by | |
| 5 | * a forge admin who is a person and confirmed, receipted with every directory and its size. | |
| 6 | * Never listed: an open pull's directory, a pull the forge has no row for, anything that is not a | |
| 7 | * plain directory named `pull-<n>` directly under an app's directory (a link is never followed). | |
| 8 | * Every path is built by `envDir`, then resolved and checked to lie inside the apps root. | |
| 9 | */ | |
| 10 | ||
| 11 | import { lstatSync, readdirSync, realpathSync } from "node:fs"; | |
| 12 | import { join, sep } from "node:path"; | |
| 13 | import type { App } from "../app.ts"; | |
| 14 | import { appsRoot, envDir, previewEnv, removeEnv } from "../apps/data.ts"; | |
| 15 | import { dirBytes } from "../apps/bytes.ts"; | |
| 16 | import type { User } from "../core/users.ts"; | |
| 17 | import { err, ok, type Result } from "../kernel/result.ts"; | |
| 18 | import type { Duty } from "./duty.ts"; | |
| 19 | import { record } from "./runs.ts"; | |
| 20 | ||
| 21 | export const PREVIEW_DAYS = 7; | |
| 22 | const DAY_MS = 86_400_000; | |
| 23 | /** The most directories one card lists; the rest wait for the next press, once these are gone. */ | |
| 24 | export const PREVIEW_CAP = 100; | |
| 25 | ||
| 26 | export interface PreviewDir { | |
| 27 | readonly appId: string; | |
| 28 | readonly repo: string; | |
| 29 | readonly number: number; | |
| 30 | /** Undefined until a walk has measured it; the lists never walk, see `sizedPreviewDirs`. */ | |
| 31 | readonly bytes: number | undefined; | |
| 32 | readonly closedAt: string; | |
| 33 | } | |
| 34 | ||
| 35 | interface Row { id: string; repo: string; number: number; closed_at: string } | |
| 36 | ||
| 37 | /** Closed pulls of one app older than the cutoff, by number; open and unknown pulls are absent. */ | |
| 38 | const CLOSED_SQL = `SELECT app.id AS id, o.username || '/' || r.name AS repo, issue.number AS number, issue.closed_at AS closed_at | |
| 39 | FROM app JOIN repo r ON r.id = app.repo_id JOIN user o ON o.id = r.owner_id | |
| 40 | JOIN issue ON issue.repo_id = app.repo_id AND issue.kind = 'pull' AND issue.state = 'closed' | |
| 41 | WHERE app.id = ?1 AND issue.number = ?2 AND issue.closed_at IS NOT NULL AND issue.closed_at < ?3`; | |
| 42 | ||
| 43 | /** The number in a `pull-<n>` directory name; undefined for any other name. */ | |
| 44 | const pullNumber = (name: string): number | undefined => (/^pull-[1-9]\d{0,9}$/.test(name) ? Number(name.slice(5)) : undefined); | |
| 45 | ||
| 46 | /** Whether `path` is a real directory (not a link) whose resolved path lies inside `root`. */ | |
| 47 | function insideRoot(root: string, path: string): boolean { | |
| 48 | try { | |
| 49 | if (!lstatSync(path).isDirectory()) return false; | |
| 50 | const real = realpathSync(path), base = realpathSync(root); | |
| 51 | return real.startsWith(base + sep); | |
| 52 | } catch { | |
| 53 | return false; | |
| 54 | } | |
| 55 | } | |
| 56 | ||
| 57 | /** Sizes measured lately, by path: the board and the terminal read these and never walk a disk. */ | |
| 58 | const measured = new WeakMap<App, Map<string, { bytes: number; at: number }>>(); | |
| 59 | ||
| 60 | /** This forge's measured sizes. */ | |
| 61 | function sizesOf(app: App): Map<string, { bytes: number; at: number }> { | |
| 62 | let m = measured.get(app); | |
| 63 | if (m === undefined) measured.set(app, (m = new Map())); | |
| 64 | return m; | |
| 65 | } | |
| 66 | const SIZE_TTL_MS = 6 * 3_600_000; | |
| 67 | ||
| 68 | /** What a walk measured lately for `path`, if anything. */ | |
| 69 | function knownBytes(app: App, path: string, now: Date): number | undefined { | |
| 70 | const s = sizesOf(app).get(path); | |
| 71 | return s !== undefined && now.getTime() - s.at < SIZE_TTL_MS ? s.bytes : undefined; | |
| 72 | } | |
| 73 | ||
| 74 | /** Walk `path` now and remember what it holds. */ | |
| 75 | async function measure(app: App, path: string, now: Date): Promise<number> { | |
| 76 | const bytes = await dirBytes(path); | |
| 77 | sizesOf(app).set(path, { bytes, at: now.getTime() }); | |
| 78 | return bytes; | |
| 79 | } | |
| 80 | ||
| 81 | /** | |
| 82 | * Preview directories of pulls closed over `days` days ago, oldest close first, at most `cap`. | |
| 83 | * Cheap: a listing and one row each, no disk walk; `bytes` is what a walk measured lately. | |
| 84 | */ | |
| 85 | export function previewDirs(app: App, now: Date, days = PREVIEW_DAYS, cap = PREVIEW_CAP): PreviewDir[] { | |
| 86 | const cutoff = new Date(now.getTime() - days * DAY_MS).toISOString(); | |
| 87 | const root = appsRoot(app.config.dataDir); | |
| 88 | let ids: string[]; | |
| 89 | try { | |
| 90 | ids = readdirSync(root); | |
| 91 | } catch { | |
| 92 | return []; | |
| 93 | } | |
| 94 | const found: PreviewDir[] = []; | |
| 95 | for (const appId of ids) { | |
| 96 | if (!/^[0-9a-z]+$/i.test(appId) || !insideRoot(root, join(root, appId))) continue; | |
| 97 | let names: string[]; | |
| 98 | try { | |
| 99 | names = readdirSync(join(root, appId)); | |
| 100 | } catch { | |
| 101 | continue; | |
| 102 | } | |
| 103 | for (const name of names) { | |
| 104 | const number = pullNumber(name); | |
| 105 | if (number === undefined) continue; | |
| 106 | const row = app.db.query<Row, [string, number, string]>(CLOSED_SQL).get(appId, number, cutoff); | |
| 107 | if (row === null || !insideRoot(root, envDir(app.config.dataDir, appId, previewEnv(number)))) continue; | |
| 108 | found.push({ appId, repo: row.repo, number, closedAt: row.closed_at, bytes: knownBytes(app, envDir(app.config.dataDir, appId, previewEnv(number)), now) }); | |
| 109 | } | |
| 110 | } | |
| 111 | found.sort((a, b) => a.closedAt.localeCompare(b.closedAt) || a.repo.localeCompare(b.repo) || a.number - b.number); | |
| 112 | return found.slice(0, cap); | |
| 113 | } | |
| 114 | ||
| 115 | /** The same list with every size walked now (the duty and the removal use this, not the board). */ | |
| 116 | export async function sizedPreviewDirs(app: App, now: Date, days = PREVIEW_DAYS, cap = PREVIEW_CAP): Promise<PreviewDir[]> { | |
| 117 | const out: PreviewDir[] = []; | |
| 118 | for (const p of previewDirs(app, now, days, cap)) out.push({ ...p, bytes: await measure(app, envDir(app.config.dataDir, p.appId, previewEnv(p.number)), now) }); | |
| 119 | return out; | |
| 120 | } | |
| 121 | ||
| 122 | /** The sizes of `found` as words, or "size not measured yet" when any is unknown. */ | |
| 123 | export function totalWords(found: readonly PreviewDir[]): string { | |
| 124 | let n = 0; | |
| 125 | for (const p of found) { if (p.bytes === undefined) return "size not measured yet"; n += p.bytes; } | |
| 126 | return sizeWords(n); | |
| 127 | } | |
| 128 | ||
| 129 | /** One line per directory: what the form carries and the receipt names. */ | |
| 130 | export const previewLine = (p: Pick<PreviewDir, "repo" | "number">): string => `${p.repo}#${p.number}`; | |
| 131 | ||
| 132 | /** A size as a person reads it. */ | |
| 133 | export function sizeWords(bytes: number | undefined): string { | |
| 134 | if (bytes === undefined) return "size not measured yet"; | |
| 135 | if (bytes < 1024) return `${bytes} B`; | |
| 136 | const units = ["KB", "MB", "GB", "TB"]; | |
| 137 | let v = bytes / 1024, i = 0; | |
| 138 | while (v >= 1024 && i < units.length - 1) { v /= 1024; i++; } | |
| 139 | return `${v.toFixed(v < 10 ? 1 : 0)} ${units[i]}`; | |
| 140 | } | |
| 141 | ||
| 142 | export type PreviewError = "forbidden" | "nothing"; | |
| 143 | export const PREVIEW_WORDS: Record<PreviewError, string> = { | |
| 144 | forbidden: "Only a forge admin may do this.", | |
| 145 | nothing: "None of those directories is still listed; the card was out of date.", | |
| 146 | }; | |
| 147 | ||
| 148 | /** | |
| 149 | * Remove the directories the card showed (`shown`: lines from `previewLine`), each only if it is | |
| 150 | * still on the list now. The receipt names every removed directory with its size. | |
| 151 | */ | |
| 152 | export async function removePreviewDirs(app: App, admin: User | undefined, shown: readonly string[], now = new Date()): Promise<Result<{ says: string }, PreviewError>> { | |
| 153 | if (admin === undefined || !admin.isAdmin || admin.kind === "agent") return err("forbidden"); | |
| 154 | const want = new Set(shown); | |
| 155 | const gone: string[] = []; | |
| 156 | let freed = 0; | |
| 157 | for (const p of previewDirs(app, now)) { | |
| 158 | if (!want.has(previewLine(p))) continue; | |
| 159 | const path = envDir(app.config.dataDir, p.appId, previewEnv(p.number)); | |
| 160 | const bytes = await dirBytes(path); | |
| 161 | removeEnv(app.config.dataDir, p.appId, previewEnv(p.number)); | |
| 162 | sizesOf(app).delete(path); | |
| 163 | gone.push(`${previewLine(p)} (${sizeWords(bytes)})`); | |
| 164 | freed += bytes; | |
| 165 | } | |
| 166 | if (gone.length === 0) return err("nothing"); | |
| 167 | const says = `removed ${gone.length} preview director${gone.length === 1 ? "y" : "ies"}, ${sizeWords(freed)} (pressed by ${admin.username}): ${gone.join(", ")}`; | |
| 168 | record(app.db, { duty: "preview-dirs", started: now, finished: now, findings: [], actions: [says], outcome: "ok" }); | |
| 169 | return ok({ says }); | |
| 170 | } | |
| 171 | ||
| 172 | export const duty: Duty = { | |
| 173 | name: "preview-dirs", | |
| 174 | intervalMs: 6 * 3_600_000, | |
| 175 | budget: { runsPerDay: 8, dollarsPerDay: 1, reason: "every six hours is four a day; the rest is room for a catch-up, and it spends no money" }, | |
| 176 | async run({ now, app }) { | |
| 177 | if (app === undefined) return { findings: [], actions: [] }; | |
| 178 | const found = await sizedPreviewDirs(app, now); | |
| 179 | if (found.length === 0) return { findings: [], actions: [] }; | |
| 180 | const total = found.reduce((n, p) => n + (p.bytes ?? 0), 0); | |
| 181 | return { | |
| 182 | findings: [ | |
| 183 | { | |
| 184 | summary: `${found.length} preview director${found.length === 1 ? "y" : "ies"} of pulls closed over ${PREVIEW_DAYS} days ago, ${sizeWords(total)}: ${found.slice(0, 5).map(previewLine).join(", ")}${found.length > 5 ? ", …" : ""}`, | |
| 185 | probe: { kind: "measurement", what: "bytes in preview directories of pulls closed over seven days ago", number: total, at: now.toISOString() }, | |
| 186 | }, | |
| 187 | ], | |
| 188 | actions: [], | |
| 189 | }; | |
| 190 | }, | |
| 191 | }; |
src/maintain/registry.ts +2 −1
| @@ -10,12 +10,13 @@ import { duty as backupGrowth } from "./duties/backup_growth.ts"; | ||
| 10 | 10 | import { duty as v1Proven } from "./duties/v1_proven.ts"; |
| 11 | 11 | import { duty as backupAge } from "./duties/backup_age.ts"; |
| 12 | 12 | import { duty as mergedBranches } from "./duties/merged_branches.ts"; |
| 13 | import { duty as previewDirs } from "./previews.ts"; | |
| 13 | 14 | import { duty as diskFree } from "./duties/disk_free.ts"; |
| 14 | 15 | import { duty as staleIssues } from "./duties/stale_issues.ts"; |
| 15 | 16 | import { duty as waitSplit } from "./duties/wait_split.ts"; |
| 16 | 17 | import { dutyIdle } from "./duties/duty_idle.ts"; |
| 17 | 18 | |
| 18 | const others: readonly Duty[] = [pullsWaiting, ruleWatcher, efficiencyWatcher, flowKeeper, visitorRetention, backupGrowth, backupAge, v1Proven, diskFree, mergedBranches, staleIssues, waitSplit]; | |
| 19 | const others: readonly Duty[] = [pullsWaiting, ruleWatcher, efficiencyWatcher, flowKeeper, visitorRetention, backupGrowth, backupAge, v1Proven, diskFree, mergedBranches, previewDirs, staleIssues, waitSplit]; | |
| 19 | 20 | |
| 20 | 21 | /** The duties plus the one that watches them for silence. */ |
| 21 | 22 | export const duties: readonly Duty[] = [...others, dutyIdle(others)]; |
src/tui/screens/entries.ts +2
| @@ -77,6 +77,7 @@ import { entry as maintainRuns } from "./maintainruns.ts"; | ||
| 77 | 77 | import { entry as dutyBudgets } from "./dutybudgets.ts"; |
| 78 | 78 | import { entry as stalePulls } from "./stalepulls.ts"; |
| 79 | 79 | import { entry as mergedBranches } from "./mergedbranches.ts"; |
| 80 | import { entry as previewDirs } from "./previewdirs.ts"; | |
| 80 | 81 | import { entry as dutySwitch } from "./dutyswitch.ts"; |
| 81 | 82 | import { entry as mirrorPacksEntry } from "./mirrorpacks.ts"; |
| 82 | 83 | import { entry as packReposEntry } from "./packrepos.ts"; |
| @@ -151,6 +152,7 @@ export const ENTRIES: readonly PaletteEntry[] = [ | ||
| 151 | 152 | maintenance, |
| 152 | 153 | stalePulls, |
| 153 | 154 | mergedBranches, |
| 155 | previewDirs, | |
| 154 | 156 | maintainRuns, |
| 155 | 157 | dutySwitch, |
| 156 | 158 | dutyBudgets, |
src/tui/screens/previewdirs.ts added +81
| @@ -0,0 +1,81 @@ | ||
| 1 | /** | |
| 2 | * Preview data of closed pulls in the terminal (#13491): the /maintenance card through the same | |
| 3 | * finder (`previewDirs`) and the same action (`removePreviewDirs`), behind the same gate, a forge | |
| 4 | * admin who is a person. `d` removes every listed directory, after a confirm. Esc or q closes it. | |
| 5 | */ | |
| 6 | ||
| 7 | import { PREVIEW_DAYS, PREVIEW_WORDS, previewDirs, previewLine, removePreviewDirs, sizeWords, type PreviewDir } from "../../maintain/previews.ts"; | |
| 8 | import type { Ctx, Screen } from "../app/app.ts"; | |
| 9 | import { binding } from "../app/keys.ts"; | |
| 10 | import type { Session } from "../session.ts"; | |
| 11 | import { style } from "../term/style.ts"; | |
| 12 | import { Scroll, confirmScreen, drawLines } from "./common.ts"; | |
| 13 | import type { PaletteEntry } from "./entries.ts"; | |
| 14 | ||
| 15 | const mayAct = (session: Session): boolean => session.user?.isAdmin === true && session.user.kind !== "agent"; | |
| 16 | ||
| 17 | /** The directories the card lists; none for anyone the gate refuses. */ | |
| 18 | export function previews(session: Session): PreviewDir[] { | |
| 19 | return mayAct(session) ? previewDirs(session.app, new Date()) : []; | |
| 20 | } | |
| 21 | ||
| 22 | /** One directory a line: pull, size, close day; the same rows as the page. */ | |
| 23 | export function previewLines(found: readonly PreviewDir[]): string[] { | |
| 24 | if (found.length === 0) return ["None."]; | |
| 25 | return found.map((p) => ` ${previewLine(p)} ${sizeWords(p.bytes)} closed ${p.closedAt.slice(0, 10)}`); | |
| 26 | } | |
| 27 | ||
| 28 | /** Remove what `found` lists; what it says, or nothing for anyone the gate refuses. */ | |
| 29 | export async function removeListed(session: Session, found: readonly PreviewDir[]): Promise<string> { | |
| 30 | if (!mayAct(session) || found.length === 0) return ""; | |
| 31 | const done = await removePreviewDirs(session.app, session.user, found.map(previewLine)); | |
| 32 | return done.ok ? done.value.says : PREVIEW_WORDS[done.error]; | |
| 33 | } | |
| 34 | ||
| 35 | export function previewDirsScreen(session: Session, ctx: Ctx): Screen { | |
| 36 | let found: PreviewDir[] = []; | |
| 37 | let note = ""; | |
| 38 | const load = (): void => { found = previews(session); ctx.invalidate(); }; | |
| 39 | load(); | |
| 40 | const screen: Screen = { | |
| 41 | title: `Preview data of pulls closed over ${PREVIEW_DAYS} days ago`, | |
| 42 | keys: [binding(["d"], "remove all"), binding(["esc", "q"], "back")], | |
| 43 | render(area, s) { | |
| 44 | const c = ctx.theme.c; | |
| 45 | const lines = [...previewLines(found), ...(note === "" ? [] : ["", note])]; | |
| 46 | drawLines(s, area, c, new Scroll(), lines.map((text) => [{ text, style: style(c.fg, c.bg) }])); | |
| 47 | }, | |
| 48 | key(e) { | |
| 49 | if (e.ctrl || e.alt) return true; | |
| 50 | if (e.key === "esc" || e.key === "q") ctx.pop(); | |
| 51 | else if (e.key === "d") { | |
| 52 | if (!mayAct(session) || found.length === 0) return true; | |
| 53 | const shown = found; | |
| 54 | ctx.push( | |
| 55 | confirmScreen(ctx, screen, { | |
| 56 | title: `Remove ${shown.length} preview directories?`, | |
| 57 | body: ["Their data is deleted for good; each name and size goes in the receipt."], | |
| 58 | actions: [{ label: "Cancel", variant: "ghost" }, { label: "Remove all", variant: "danger" }], | |
| 59 | onChoose: (i) => { | |
| 60 | if (i !== 1) return; | |
| 61 | void removeListed(session, shown).then((says) => { note = says; load(); }); | |
| 62 | }, | |
| 63 | }), | |
| 64 | ); | |
| 65 | return true; | |
| 66 | } else return false; | |
| 67 | ctx.invalidate(); | |
| 68 | return true; | |
| 69 | }, | |
| 70 | }; | |
| 71 | return screen; | |
| 72 | } | |
| 73 | ||
| 74 | export const entry: PaletteEntry = { | |
| 75 | id: "preview-dirs", | |
| 76 | title: "Preview data of pulls closed over 7 days ago", | |
| 77 | hint: "remove the old previews' data in one receipted step", | |
| 78 | after: ["maintenance", "maintenance-runs", "merged-branches"], | |
| 79 | offered: mayAct, | |
| 80 | run: (session, ctx) => ctx.push(previewDirsScreen(session, ctx)), | |
| 81 | }; |
src/web/pages/maintenance.ts +38
| @@ -17,6 +17,7 @@ import { icon } from "../icons.ts"; | ||
| 17 | 17 | import { flashTo, page } from "../layout.ts"; |
| 18 | 18 | import { badge, btn, disclosure, field } from "../ui.ts"; |
| 19 | 19 | import { STALE_DAYS, STALE_WORDS, closeStale, nudge, stalePulls } from "../../maintain/stale_pulls.ts"; |
| 20 | import { PREVIEW_DAYS, PREVIEW_WORDS, previewDirs, previewLine, removePreviewDirs, sizeWords, totalWords, type PreviewDir } from "../../maintain/previews.ts"; | |
| 20 | 21 | import { MERGED_DAYS, MERGED_WORDS, deleteMerged, lineOf, mergedBranches, type MergedBranch } from "../../maintain/duties/merged_branches.ts"; |
| 21 | 22 | import { writePauseFold } from "./maintenance_pause.ts"; |
| 22 | 23 | import { budgetRows } from "./maintenance_budgets.ts"; |
| @@ -56,6 +57,7 @@ export async function maintenancePage(ctx: Ctx): Promise<Response> { | ||
| 56 | 57 | const ran = duties(ctx.app.config.dataDir); |
| 57 | 58 | const states = dutyStates(ctx.app.db); |
| 58 | 59 | const mergedPending = await mergedBranches(ctx.app, new Date()); |
| 60 | const previewPending = previewDirs(ctx.app, new Date()); | |
outdated
MEDIUM: previewDirs (with sizes) runs on every board load, even for a non-admin who cannot act. Make it cheap or cached (see previews.ts comment). |
||
| 59 | 61 | const stalePending = stalePulls(ctx.app.db, new Date()); |
| 60 | 62 | const stale = stalePending.length; |
| 61 | 63 | const overdue = ran.filter((d) => d.overdue).length; |
| @@ -87,6 +89,9 @@ export async function maintenancePage(ctx: Ctx): Promise<Response> { | ||
| 87 | 89 | ${mergedPending.length > 0 |
| 88 | 90 | ? fold("merged-branches", `Branches of pulls merged over ${String(MERGED_DAYS)} days ago`, String(mergedPending.length), true, mergedRows(ctx.auth.user!.kind !== "agent", mergedPending)) |
| 89 | 91 | : null} |
| 92 | ${previewPending.length > 0 | |
| 93 | ? fold("preview-dirs", `Preview data of pulls closed over ${String(PREVIEW_DAYS)} days ago`, `${String(previewPending.length)}, ${totalWords(previewPending)}`, true, previewRows(ctx.auth.user!.kind !== "agent", previewPending)) | |
| 94 | : null} | |
| 90 | 95 | ${idle |
| 91 | 96 | ? fold("maintenance-prepare", "Prepare for a power-off", `${String(ABOUT_TO_STOP_ROWS.length)} things stop`, false, |
| 92 | 97 | html`<p id="maintenance-standby" class="muted">${s.standby.says}</p> |
| @@ -217,3 +222,36 @@ export async function mergedAction(ctx: Ctx): Promise<Response> { | ||
| 217 | 222 | const done = await deleteMerged(ctx.app, user, shown); |
| 218 | 223 | return flashTo(ctx, "/maintenance", done.ok ? { message: done.value.says.slice(0, 600) } : { tone: "danger", message: MERGED_WORDS[done.error] }); |
| 219 | 224 | } |
| 225 | ||
| 226 | /** | |
| 227 | * The preview directories as one card: each with its size is the preview, one button removes them | |
| 228 | * all. | |
| 229 | */ | |
| 230 | function previewRows(withActions: boolean, found: readonly PreviewDir[]): Html { | |
| 231 | return html`<dl class="status-rows">${found.map( | |
| 232 | (p) => html`<div class="status-row" data-preview="${previewLine(p)}"><dt>${previewLine(p)}</dt><dd>${sizeWords(p.bytes)} <span class="muted">closed ${p.closedAt.slice(0, 10)}</span></dd></div>`, | |
| 233 | )}</dl> | |
| 234 | ${withActions | |
| 235 | ? html`<form method="post" action="/maintenance/previews/remove" class="stack"> | |
| 236 | ${found.map((p) => html`<input type="hidden" name="shown" value="${previewLine(p)}">`)} | |
| 237 | ${field({ name: "confirm", type: "checkbox", value: "yes", required: true, label: `Remove these ${String(found.length)} directories` })} | |
| 238 | <div class="form-actions">${btn("Remove all", { variant: "danger" })}</div> | |
| 239 | </form>` | |
| 240 | : null}`; | |
| 241 | } | |
| 242 | ||
| 243 | /** | |
| 244 | * POST /maintenance/previews/remove: remove the directories the card showed, receipted. Same-origin | |
| 245 | * by readForm. | |
| 246 | */ | |
| 247 | export async function previewAction(ctx: Ctx): Promise<Response> { | |
| 248 | const user = ctx.auth.user; | |
| 249 | if (user === undefined || !user.isAdmin || user.kind === "agent") | |
| 250 | return flashTo(ctx, "/maintenance", { tone: "danger", message: PREVIEW_WORDS.forbidden }); | |
| 251 | const form = await readForm(ctx, "admin"); | |
| 252 | if (form instanceof Response) return form; | |
| 253 | if (form.get("confirm") !== "yes") return flashTo(ctx, "/maintenance", { tone: "danger", message: "Tick the box to confirm the removal." }); | |
| 254 | const shown = form.getAll("shown").filter((v): v is string => typeof v === "string" && v.length <= 400).slice(0, 500); | |
| 255 | const done = await removePreviewDirs(ctx.app, user, shown); | |
| 256 | return flashTo(ctx, "/maintenance", done.ok ? { message: done.value.says.slice(0, 600) } : { tone: "danger", message: PREVIEW_WORDS[done.error] }); | |
| 257 | } |
test/maintain-previews-duty.test.ts added +83
| @@ -0,0 +1,83 @@ | ||
| 1 | /** | |
| 2 | * The preview-dirs duty (#13491): what it finds, the exact seven-day edge, the empty case, that it | |
| 3 | * never removes, and that the board's list walks no disk. | |
| 4 | */ | |
| 5 | import { afterAll, beforeAll, expect, test } from "bun:test"; | |
| 6 | import { existsSync, mkdirSync, writeFileSync } from "node:fs"; | |
| 7 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 8 | import { tmpdir } from "node:os"; | |
| 9 | import { join } from "node:path"; | |
| 10 | import { createApp, type App } from "../src/app.ts"; | |
| 11 | import { appsRoot } from "../src/apps/data.ts"; | |
| 12 | import { createRepo } from "../src/core/repos.ts"; | |
| 13 | import { createUser } from "../src/core/users.ts"; | |
| 14 | import { id } from "../src/kernel/id.ts"; | |
| 15 | import { duty, previewDirs } from "../src/maintain/previews.ts"; | |
| 16 | import { duties } from "../src/maintain/registry.ts"; | |
| 17 | import { quietLogs } from "./helpers/globals.ts"; | |
| 18 | ||
| 19 | quietLogs(); | |
| 20 | let dir = "", app: App, repoId = "", userId = "", appId = "", seq = 0; | |
| 21 | const day = 86_400_000; | |
| 22 | const now = new Date("2026-06-15T12:00:00.000Z"); | |
| 23 | const path = (n: number): string => join(appsRoot(dir), appId, `pull-${n}`); | |
| 24 | ||
| 25 | function pull(closedAt: Date, bytes: number): number { | |
| 26 | const n = ++seq, at = closedAt.toISOString(); | |
| 27 | app.db.query("INSERT INTO issue (id, repo_id, number, kind, title, body, author_id, state, created_at, updated_at, closed_at) VALUES (?1, ?2, ?3, 'pull', 'P', '', ?4, 'closed', ?5, ?5, ?5)").run(id(), repoId, n, userId, at); | |
| 28 | mkdirSync(join(path(n), "data"), { recursive: true }); | |
| 29 | writeFileSync(join(path(n), "data", "db"), "x".repeat(bytes)); | |
| 30 | return n; | |
| 31 | } | |
| 32 | const ctx = () => ({ db: app.db, bus: app.bus, now, app }); | |
| 33 | ||
| 34 | beforeAll(async () => { | |
| 35 | dir = await mkdtemp(join(tmpdir(), "forge-previews-duty-")); | |
| 36 | app = createApp({ dataDir: dir, port: 0 }); | |
| 37 | const u = await createUser(app.db, { username: "boss", email: "[email protected]", password: "a-good-passphrase", origin: "seed", trusted: true, isAdmin: true }); | |
| 38 | if (!u.ok) throw new Error("user"); | |
| 39 | userId = u.value.id; | |
| 40 | const r = await createRepo(app.db, app.reposRoot, { ownerId: userId, name: "r", visibility: "public", description: "r" }); | |
| 41 | if (!r.ok) throw new Error("repo"); | |
| 42 | repoId = r.value.id; | |
| 43 | appId = id().toLowerCase().replace(/[^0-9a-z]/g, ""); | |
| 44 | app.db.query("INSERT INTO app (id, repo_id, slug, access) VALUES (?1, ?2, 'r', 'public')").run(appId, repoId); | |
| 45 | }); | |
| 46 | afterAll(async () => { | |
| 47 | app.close(); | |
| 48 | await rm(dir, { recursive: true, force: true }); | |
| 49 | }); | |
| 50 | ||
| 51 | test("the duty is registered, and with nothing old it finds nothing", async () => { | |
| 52 | expect(duties.some((d) => d.name === "preview-dirs")).toBe(true); | |
| 53 | expect(await duty.run(ctx())).toEqual({ findings: [], actions: [] }); | |
| 54 | }); | |
| 55 | ||
| 56 | test("the seven-day edge: exactly seven days is not yet listed, a millisecond more is", () => { | |
| 57 | const edge = pull(new Date(now.getTime() - 7 * day), 1), past = pull(new Date(now.getTime() - 7 * day - 1), 1); | |
| 58 | const numbers = previewDirs(app, now).map((p) => p.number); | |
| 59 | expect(numbers).not.toContain(edge); | |
| 60 | expect(numbers).toContain(past); | |
| 61 | }); | |
| 62 | ||
| 63 | test("the board's list walks nothing: sizes are unknown until the duty measures them", async () => { | |
| 64 | const n = pull(new Date(now.getTime() - 20 * day), 4096); | |
| 65 | expect(previewDirs(app, now).find((p) => p.number === n)?.bytes).toBeUndefined(); | |
| 66 | await duty.run(ctx()); | |
| 67 | expect(previewDirs(app, now).find((p) => p.number === n)?.bytes).toBe(4096); | |
| 68 | }); | |
| 69 | ||
| 70 | test("a run names the count and the bytes in a finding with a probe, and removes nothing", async () => { | |
| 71 | const before = previewDirs(app, now); | |
| 72 | expect(before.length).toBeGreaterThanOrEqual(2); | |
| 73 | const res = await duty.run(ctx()); | |
| 74 | expect(res.actions).toEqual([]); | |
| 75 | expect(res.findings).toHaveLength(1); | |
| 76 | const f = res.findings[0]!; | |
| 77 | expect(f.summary).toContain(`${before.length} preview directories of pulls closed over 7 days ago`); | |
| 78 | expect(f.probe).toMatchObject({ kind: "measurement", at: now.toISOString() }); | |
| 79 | const total = previewDirs(app, now).reduce((n, p) => n + (p.bytes ?? 0), 0); | |
| 80 | expect(f.probe!.number).toBe(total); | |
| 81 | expect(total).toBeGreaterThanOrEqual(4097); | |
| 82 | for (const p of before) expect(existsSync(path(p.number))).toBe(true); | |
| 83 | }); |
test/maintain-previews.test.ts added +133
| @@ -0,0 +1,133 @@ | ||
| 1 | /** | |
MEDIUM: nothing tests the duty itself (run returns a finding naming count and bytes with no actions, returns none when empty, never removes anything), nor the exact seven-day boundary. Add tests that fail without them. |
||
| 2 | * Preview directories of closed pulls (#13491): the finder, the confirmed removal, the gates, | |
| 3 | * containment. | |
| 4 | */ | |
| 5 | import { afterAll, beforeAll, expect, test } from "bun:test"; | |
| 6 | import { existsSync, mkdirSync, symlinkSync, writeFileSync } from "node:fs"; | |
| 7 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 8 | import { tmpdir } from "node:os"; | |
| 9 | import { join } from "node:path"; | |
| 10 | import { createApp, type App } from "../src/app.ts"; | |
| 11 | import { appsRoot } from "../src/apps/data.ts"; | |
| 12 | import { createRepo, type Repo } from "../src/core/repos.ts"; | |
| 13 | import { createUser, type User } from "../src/core/users.ts"; | |
| 14 | import { serve, type Servers } from "../src/http/server.ts"; | |
| 15 | import { id } from "../src/kernel/id.ts"; | |
| 16 | import { previewDirs, removePreviewDirs, sizedPreviewDirs } from "../src/maintain/previews.ts"; | |
| 17 | import { previewLines, removeListed } from "../src/tui/screens/previewdirs.ts"; | |
| 18 | import { Session } from "../src/tui/session.ts"; | |
| 19 | import { quietLogs } from "./helpers/globals.ts"; | |
| 20 | ||
| 21 | quietLogs(); | |
| 22 | let dir = "", outside = "", app: App, servers: Servers, admin: User, person: User, repo: Repo, appId = "", seq = 0, adminCookie = "", personCookie = ""; | |
| 23 | const day = 86_400_000; | |
| 24 | ||
| 25 | async function user(name: string, isAdmin: boolean): Promise<User> { | |
| 26 | const made = await createUser(app.db, { username: name, email: `${name}@forge.test`, password: "a-good-passphrase", origin: "seed", trusted: true, ...(isAdmin ? { isAdmin: true } : {}) }); | |
| 27 | if (!made.ok) throw new Error(name); | |
| 28 | return made.value; | |
| 29 | } | |
| 30 | async function login(username: string): Promise<string> { | |
| 31 | const res = await fetch(`${servers.url}/signin`, { | |
| 32 | method: "POST", redirect: "manual", | |
| 33 | headers: { "content-type": "application/x-www-form-urlencoded", origin: servers.url }, | |
| 34 | body: new URLSearchParams({ username, password: "a-good-passphrase" }), | |
| 35 | }); | |
| 36 | return (res.headers.get("set-cookie") ?? "").split(";")[0] ?? ""; | |
| 37 | } | |
| 38 | const path = (n: number): string => join(appsRoot(dir), appId, `pull-${n}`); | |
| 39 | /** A pull in `state` that closed `ago` ms ago, with a preview directory holding `bytes` bytes. */ | |
| 40 | function pull(state: "open" | "closed", ago: number, bytes = 10): number { | |
| 41 | const n = ++seq, at = new Date(Date.now() - ago).toISOString(); | |
| 42 | const issue = id(); | |
| 43 | app.db.query("INSERT INTO issue (id, repo_id, number, kind, title, body, author_id, state, created_at, updated_at, closed_at) VALUES (?1, ?2, ?3, 'pull', 'P', '', ?4, ?5, ?6, ?6, ?7)").run(issue, repo.id, n, admin.id, state, at, state === "closed" ? at : null); | |
| 44 | mkdirSync(join(path(n), "data"), { recursive: true }); | |
| 45 | writeFileSync(join(path(n), "data", "db"), "x".repeat(bytes)); | |
| 46 | return n; | |
| 47 | } | |
| 48 | const post = (cookie: string, body: string, origin = servers.url): Promise<Response> => | |
| 49 | fetch(`${servers.url}/maintenance/previews/remove`, { method: "POST", redirect: "manual", headers: { "content-type": "application/x-www-form-urlencoded", origin, cookie }, body }); | |
| 50 | const form = (n: number, confirm: boolean): string => new URLSearchParams({ shown: `boss/r#${n}`, ...(confirm ? { confirm: "yes" } : {}) }).toString(); | |
| 51 | ||
| 52 | beforeAll(async () => { | |
| 53 | dir = await mkdtemp(join(tmpdir(), "forge-previews-")); | |
| 54 | outside = await mkdtemp(join(tmpdir(), "forge-previews-out-")); | |
| 55 | app = createApp({ dataDir: dir, port: 0 }); | |
| 56 | servers = await serve(app); | |
| 57 | admin = await user("boss", true); | |
| 58 | person = await user("pat", false); | |
| 59 | const made = await createRepo(app.db, app.reposRoot, { ownerId: admin.id, name: "r", visibility: "public", description: "r" }); | |
| 60 | if (!made.ok) throw new Error("repo"); | |
| 61 | repo = made.value; | |
| 62 | appId = id().toLowerCase().replace(/[^0-9a-z]/g, ""); | |
| 63 | app.db.query("INSERT INTO app (id, repo_id, slug, access) VALUES (?1, ?2, 'r', 'public')").run(appId, repo.id); | |
| 64 | adminCookie = await login("boss"); | |
| 65 | personCookie = await login("pat"); | |
| 66 | }); | |
| 67 | afterAll(async () => { | |
| 68 | servers.stop(); | |
| 69 | app.close(); | |
| 70 | await rm(dir, { recursive: true, force: true }); | |
| 71 | await rm(outside, { recursive: true, force: true }); | |
| 72 | }); | |
| 73 | ||
| 74 | test("only directories of pulls closed over seven days ago are listed, with their size", async () => { | |
| 75 | const old = pull("closed", 10 * day, 123), fresh = pull("closed", 2 * day), open = pull("open", 30 * day); | |
| 76 | mkdirSync(join(appsRoot(dir), appId, "pull-999"), { recursive: true }); // no pull row | |
| 77 | mkdirSync(join(appsRoot(dir), appId, "production"), { recursive: true }); | |
| 78 | const found = await sizedPreviewDirs(app, new Date()); | |
| 79 | const numbers = found.map((p) => p.number); | |
| 80 | expect(numbers).toContain(old); | |
| 81 | for (const n of [fresh, open, 999]) expect(numbers).not.toContain(n); | |
| 82 | expect(found.find((p) => p.number === old)?.bytes).toBe(123); | |
| 83 | expect(previewLines(found).join("\n")).toContain(`boss/r#${old} 123 B`); | |
| 84 | }); | |
| 85 | ||
| 86 | test("a link named like a preview is never listed and what it points at is never touched", async () => { | |
| 87 | const n = ++seq, at = new Date(Date.now() - 10 * day).toISOString(); | |
| 88 | app.db.query("INSERT INTO issue (id, repo_id, number, kind, title, body, author_id, state, created_at, updated_at, closed_at) VALUES (?1, ?2, ?3, 'pull', 'P', '', ?4, 'closed', ?5, ?5, ?5)").run(id(), repo.id, n, admin.id, at); | |
| 89 | writeFileSync(join(outside, "keep"), "keep"); | |
| 90 | symlinkSync(outside, path(n)); | |
| 91 | expect(previewDirs(app, new Date()).map((p) => p.number)).not.toContain(n); | |
| 92 | const res = await removePreviewDirs(app, admin, [`boss/r#${n}`]); | |
| 93 | expect(res.ok).toBe(false); | |
| 94 | expect(existsSync(join(outside, "keep"))).toBe(true); | |
| 95 | }); | |
| 96 | ||
| 97 | test("a non-admin and a cross-origin post are refused; an unconfirmed post removes nothing", async () => { | |
| 98 | const n = pull("closed", 10 * day); | |
| 99 | expect((await post(personCookie, form(n, true))).status).toBe(403); | |
| 100 | expect((await post(adminCookie, form(n, true), "https://evil.example")).status).toBe(403); | |
| 101 | await post(adminCookie, form(n, false)); | |
| 102 | expect(existsSync(path(n))).toBe(true); | |
| 103 | const r = await removePreviewDirs(app, person, [`boss/r#${n}`]); | |
| 104 | expect(r.ok).toBe(false); | |
| 105 | expect(existsSync(path(n))).toBe(true); | |
| 106 | }); | |
| 107 | ||
| 108 | test("the card shows the rows; a confirmed post removes only what was shown and receipts it", async () => { | |
| 109 | const doomed = pull("closed", 10 * day, 50), other = pull("closed", 11 * day, 60), open = pull("open", 30 * day); | |
| 110 | const page = await (await fetch(`${servers.url}/maintenance`, { headers: { cookie: adminCookie } })).text(); | |
| 111 | expect(page).toContain('id="preview-dirs"'); | |
| 112 | expect(page).toContain(`data-preview="boss/r#${doomed}"`); | |
| 113 | expect(page).toContain('action="/maintenance/previews/remove"'); | |
| 114 | const res = await post(adminCookie, form(doomed, true)); | |
| 115 | expect(res.status).toBe(303); | |
| 116 | expect(existsSync(path(doomed))).toBe(false); | |
| 117 | expect(existsSync(path(other))).toBe(true); | |
| 118 | expect(existsSync(path(open))).toBe(true); | |
| 119 | const cookie = (res.headers.get("set-cookie") ?? "").split(";")[0] ?? ""; | |
| 120 | const after = decodeURIComponent(await (await fetch(`${servers.url}/maintenance`, { headers: { cookie: `${adminCookie}; ${cookie}` } })).text()); | |
| 121 | expect(after).toContain(`boss/r#${doomed} (50 B)`); | |
| 122 | expect(existsSync(outside)).toBe(true); | |
| 123 | }); | |
| 124 | ||
| 125 | test("the terminal door removes for an admin who is a person and gives anyone else nothing", async () => { | |
| 126 | const n = pull("closed", 10 * day); | |
| 127 | const as = (u: User | undefined) => new Session(app, { sshUsername: "t", remoteAddress: "127.0.0.1", fingerprint: "SHA256:t", user: u }); | |
| 128 | const found = previewDirs(app, new Date()).filter((p) => p.number === n); | |
| 129 | for (const who of [as(person), as({ ...admin, kind: "agent" }), as(undefined)]) expect(await removeListed(who, found)).toBe(""); | |
| 130 | expect(existsSync(path(n))).toBe(true); | |
| 131 | expect(await removeListed(as({ ...admin, kind: "user" }), found)).toContain(`boss/r#${n}`); | |
| 132 | expect(existsSync(path(n))).toBe(false); | |
| 133 | }); |
test/palette-roster.test.ts +1
| @@ -138,6 +138,7 @@ test("a site-admin person is offered the whole roster", () => { | ||
| 138 | 138 | ["maintenance", "Maintenance", "prepare the forge for a power-off, check the drain, or end maintenance"], |
| 139 | 139 | ["stale-pulls", "Pulls untouched for 14 days", "ask whether a stale pull is still wanted, or close one an agent opened"], |
| 140 | 140 | ["merged-branches", "Branches of pulls merged over 7 days ago", "delete the merged pulls' branches in one receipted step"], |
| 141 | ["preview-dirs", "Preview data of pulls closed over 7 days ago", "remove the old previews' data in one receipted step"], | |
| 141 | 142 | ["maintenance-runs", "Maintenance duty runs", "what each duty found, did and probed"], |
| 142 | 143 | ["maintenance-duties", "Pause or resume a maintenance duty", "switch a duty off, or on again after it paused itself"], |
| 143 | 144 | ["maintenance-budgets", "Set a maintenance duty's budget", "runs per day and dollars per day a duty may use"], |
new above
offline
MEDIUM: dirBytes is awaited for up to 100 directories on every call, and the page, the terminal and the duty all call previewDirs. Each /maintenance load walks every old preview tree. Fix: list without sizes for rendering paths that do not need them, or cache sizes (keyed by dir mtime) and size only at removal/receipt time; the page must answer in milliseconds.