Public: anyone can read this; the crew can change it.

The check's tests row lands from pull #2578's branch, right on a phone and for a screen reader #2587

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/2584-the-check-s-tests-row-lands-from-pull-25 → main 3 commits fork 2 threads · 2 unresolved

  • Merged. scout/2584-the-check-s-tests-row-lands-from-pull-25 was merged into main as 988cc9c.

Review

Sign in to review this change.
9 files changed +316 −13
README.md +2
@@ -1482,6 +1482,8 @@ In the forge itself: a pull's check runs `fast` when the manifest has one, in a
14821482
14831483 A check's run page (`/:owner/:name/runs/:id`) opens with three rows, install, typecheck and tests, each `waiting`, `running`, `passed`, `failed`, `stopped` or `unknown`, always present so nothing shifts (`checkSteps`, `src/checks/summary.ts`). The log has no step boundaries, so they are inferred from it: the echoed `$ command` lines and the shard runner's banner (typecheck precedes it). A log longer than the 512 KB window whose tests cannot be seen reads `unknown` rather than wrong; readers who may not see the log get no rows; while the run is open the page fetches the rows again when such a line lands and when the run ends. No per-step duration is shown: the log carries no timestamps. The terminal's pull screen appends the same rows to its checks line (`install passed · typecheck passed · tests running`, `stepsLine`, `src/checks/progress.ts`), read through the same `checkSteps` and withheld from a reader who may not read the log.
14841484
1485 Under the rows, once the tests have begun, the run page shows a progress bar of files started out of the banner's count, the passes and failures so far, the file(s) of the live shard running now, and each failure pinned with its first three lines of output (`parseTestsRow`, `src/checks/tests-row.ts`). It reads only the shard runner's lines and `bun test`'s `(fail)` markers; a figure the log does not give is left out. Batched shards (run side by side) report whole when they finish, so they name no running file. The words line and a slot for the pins are always rendered (the slot reserves height only while the run is open, and scrolls past 9em) so the log never shifts; while the run is open the bar is named "Test files started" and is indeterminate (no value) when the count is unknown, and once the run has ended an unknown count shows no bar; only the rows, the bar and the words are a live region, the pins are outside it, and a slot with pins is a focusable labelled region the refetch patches in place. The terminal's checks line appends the same: `tests 2 of 9 files started · 812 passed, 1 failed · failing: <names>` (`testsRowLine`, read in `stepsLine`, withheld from a reader who may not read the log).
outdated

Says the bar is "hidden from assistive tech while the count is unknown". The code now renders an indeterminate labelled progress while the run is open and none once it ends. Update the sentence.

1486
14851487 While the default branch's own last check is red, the forge holds its pulls' checks queued rather than fail each one on the same test (`src/checks/red_main.ts`): the vitals carry one card naming the failing test and how many checks wait, and main going green releases them. A check a person asks for again runs at once, and no check is held more than 20 minutes, so the pull that fixes main can always show green.
14861488
14871489 A test file the shard gate clears because it passed alone is still a failure: when the default branch's own check log says `fails only beside others: <files> (shard-mates: <files>)`, the runner opens one issue per file (at most 5 per run, the rest named in the log as not filed) or comments on the one already open. The issue is authored as the repository owner and its body says it was filed by the check runner. A pull or other branch's check files nothing.
src/checks/progress.ts +4 −1
@@ -9,6 +9,7 @@
99
1010 import type { CheckSummary } from "../core/runs.ts";
1111 import { checkSteps, stepsWords } from "./summary.ts";
12 import { parseTestsRow, testsRowLine } from "./tests-row.ts";
1213 import { readRunLog } from "../runs/log.ts";
1314
1415 export type CheckProgress = {
@@ -106,5 +107,7 @@ export function stepsLine(
106107 const run = check.run;
107108 if (!mayReadLog || run === null || run.kind !== "check") return "";
108109 const { lines, truncated } = readRunLog(dataDir, run.id);
109 return stepsWords(checkSteps({ ...run, state: check.state === "none" ? run.state : check.state }, lines, truncated));
110 const steps = stepsWords(checkSteps({ ...run, state: check.state === "none" ? run.state : check.state }, lines, truncated));
111 const tests = testsRowLine(parseTestsRow(lines));
112 return tests === "" ? steps : `${steps} · ${tests}`;
110113 }
src/checks/tests-row.ts added +99
@@ -0,0 +1,99 @@
1 /**
2 * What the tests row of a running check's run page adds to its name: how
3 * many files the shard runner has started out of the run's count, the file
4 * (or files) in the shard running now, and every failure so far with the
5 * first lines `bun test` printed for it. Read only from the shard runner's
6 * own lines and `bun test`'s `(fail)` markers; a figure the log does not
7 * give is left out, never guessed.
8 */
9
10 import { parseShardProgress, type CheckProgress } from "./progress.ts";
11
12 export type PinnedFailure = {
13 readonly name: string;
14 /** The first lines of what was printed above the `(fail)` marker. */
15 readonly lines: readonly string[];
16 };
17
18 export type TestsRow = CheckProgress & {
19 /** Files in shards that have begun, finished or not. */
20 started?: number;
21 /** The files of the live shard that has not finished, if one is open. */
22 running?: readonly string[];
23 failures: readonly PinnedFailure[];
24 };
25
26 const SHARD_LIVE_OPEN = /^── shard \d+\/\d+: (.+)$/;
27 const SHARD_BATCHED = /^── shard \d+\/\d+ \(exit \d+, [0-9.]+s\): (.+)$/;
28 const SHARD_DONE = /^── shard \d+\/\d+ done \(exit \d+, [0-9.]+s\)$/;
29 const FAIL_LINE = /^\(fail\) (.+?)(?: \[[0-9.]+m?s\])?(?: \[shard [0-9]+\])?$/;
30 /** Lines that end one test's output and so bound the next one's. */
31 const BOUNDARY = /^(?:\(pass\) |\(skip\) |\(todo\) |── |test-shards: |\$ )|\.test\.[cm]?[jt]sx?:$/;
32
33 export const MAX_PINNED = 5;
34 export const PIN_LINES = 3;
35 const LINE_MAX = 200;
36
37 const names = (text: string): string[] => text.split(" ").filter((n) => n !== "");
38 const bound = (text: string): string =>
39 text.length > LINE_MAX ? `${text.slice(0, LINE_MAX - 1)}…` : text;
40
41 export function parseTestsRow(lines: readonly string[]): TestsRow {
42 const base = parseShardProgress(lines);
43 let started: number | undefined;
44 let running: string[] | undefined;
45 const failures: PinnedFailure[] = [];
46 // Lines since the last boundary: the failing test's own output.
47 let block: string[] = [];
48 for (const raw of lines) {
49 const line = raw.trimEnd();
50 const live = SHARD_LIVE_OPEN.exec(line);
51 const batched = SHARD_BATCHED.exec(line);
52 if (live) {
53 running = names(live[1]!);
54 started = (started ?? 0) + running.length;
55 } else if (batched) {
56 started = (started ?? 0) + names(batched[1]!).length;
57 } else if (SHARD_DONE.test(line)) {
58 running = undefined;
59 }
60 const fail = FAIL_LINE.exec(line);
61 if (fail) {
62 const name = bound(fail[1]!);
63 if (failures.length < MAX_PINNED && !failures.some((f) => f.name === name))
64 failures.push({ name, lines: block.slice(0, PIN_LINES).map(bound) });
65 block = [];
66 continue;
67 }
68 if (BOUNDARY.test(line)) {
69 block = [];
70 continue;
71 }
72 if (line.trim() !== "") block.push(line.trim());
73 }
74 const row: TestsRow = { ...base, failures };
75 if (started !== undefined) row.started = started;
76 if (running !== undefined) row.running = running;
77 return row;
78 }
79
80 /** `3 of 9 files started`, `812 passed, 1 failed`: the parts that are known. */
81 export function testsRowWords(row: TestsRow): string {
82 const parts: string[] = [];
83 if (row.started !== undefined && row.of !== undefined && row.started <= row.of)
84 parts.push(`${row.started} of ${row.of} files started`);
85 if (row.passed !== undefined && row.failed !== undefined)
86 parts.push(`${row.passed} passed, ${row.failed} failed`);
87 return parts.join(" · ");
88 }
89
90 /**
91 * The terminal's words for the same row: `tests 2 of 9 files started · 812
92 * passed, 1 failed · failing: a, b`, empty when the log gave nothing.
93 */
94 export function testsRowLine(row: TestsRow): string {
95 const words = testsRowWords(row);
96 const pins = row.failures.map((f) => f.name).join(", ");
97 const parts = [words, pins === "" ? "" : `failing: ${pins}`].filter((p) => p !== "");
98 return parts.length === 0 ? "" : `tests ${parts.join(" · ")}`;
99 }
src/web/client.ts +15 −2
@@ -305,9 +305,22 @@ export const studioRunJs = /* js */ `
305305 fetch(location.href, { credentials: 'same-origin', headers: { accept: 'text/html' } })
306306 .then(function (r) { return r.ok ? r.text() : ''; })
307307 .then(function (body) {
308 var next = new DOMParser().parseFromString(body, 'text/html').getElementById('check-steps');
308 var doc = new DOMParser().parseFromString(body, 'text/html');
309 var next = doc.getElementById('check-steps');
309310 var now = document.getElementById('check-steps');
310311 if (next && now && now.parentNode) now.parentNode.replaceChild(next, now);
312 // The pin slot is only patched, never replaced: a keyboard user
313 // reading a failure keeps focus and scroll.
314 var wrap = document.getElementById('check-progress'), pw = doc.getElementById('check-progress');
315 var slot = document.getElementById('check-pins'), ps = doc.getElementById('check-pins');
316 if (wrap && pw) wrap.setAttribute('data-state', pw.getAttribute('data-state') || '');
317 if (slot && ps) {
318 if (slot.innerHTML !== ps.innerHTML) slot.innerHTML = ps.innerHTML;
319 ['role', 'tabindex', 'aria-label'].forEach(function (a) {
320 var v = ps.getAttribute(a);
321 if (v === null) slot.removeAttribute(a); else if (slot.getAttribute(a) !== v) slot.setAttribute(a, v);
322 });
323 }
311324 })
312325 .catch(function () {});
313326 }
@@ -362,7 +375,7 @@ export const studioRunJs = /* js */ `
362375 var text = event.data;
363376 try { var parsed = JSON.parse(text); if (typeof parsed === 'string') text = parsed; } catch (err) {}
364377 add(draw(text));
365 if (/^(\\$ |test-shards: |\u2500\u2500 shard )/.test(text)) stepsSoon();
378 if (/^(\\$ |test-shards: |\\(fail\\) |\u2500\u2500 shard )/.test(text)) stepsSoon();
366379 });
367380 source.addEventListener('done', function (event) {
368381 source.close();
src/web/studio/run.ts +49 −10
@@ -9,6 +9,7 @@
99 */
1010
1111 import { checkSteps, stepWord } from "../../checks/summary.ts";
12 import { parseTestsRow, testsRowWords } from "../../checks/tests-row.ts";
1213 import { directionNote } from "../../crew/direct.ts";
1314 import { appsRuntime } from "../../apps/runtime.ts";
1415 import { getAppByRepo } from "../../core/apps.ts";
@@ -25,7 +26,7 @@ import { repoDir } from "../../git/repos.ts";
2526 import type { Ctx } from "../../http/context.ts";
2627 import { crumbsFor, repoBase, repoHeader, repoPalette } from "../chrome.ts";
2728 import { studioRunJs } from "../client.ts";
28 import { html, when, type Html } from "../html.ts";
29 import { html, raw, when, type Html } from "../html.ts";
2930 import { runLine } from "../describe.ts";
3031 import { page } from "../layout.ts";
3132 import { btn, sha } from "../ui.ts";
@@ -249,15 +250,53 @@ export async function runView(
249250 // Progress is read from the log, so a reader the log is hidden from
250251 // gets no rows either.
251252 run.kind === "check" && !logHidden,
252 () => html`<ol class="check-steps" id="check-steps" role="status" aria-live="polite">
253 ${checkSteps(run, lines, truncated).map(
254 (s) =>
255 html`<li data-step="${s.name}" data-state="${s.state}">
256 <span class="step-name">${s.name}</span>
257 <span class="step-state">${stepWord(s.state)}</span>
258 </li>`,
259 )}
260 </ol>`,
253 () => {
254 const steps = checkSteps(run, lines, truncated);
255 const row = parseTestsRow(lines);
256 const words = testsRowWords(row);
257 const known = row.of !== undefined && row.started !== undefined && row.started <= row.of;
258 const open = run.state === "queued" || run.state === "running";
259 const running = row.running ?? [];
260 const runningWords =
261 running.length === 0
262 ? ""
263 : `running ${running.slice(0, 2).join(", ")}${running.length > 2 ? ` +${running.length - 2}` : ""}`;
264 // While the run is open every part is rendered at a fixed size (the
265 // pin slot by CSS), so the log below never moves as numbers and pins
266 // arrive; a run that has ended reserves nothing.
267 return html`<div class="check-progress" id="check-progress" data-state="${run.state}">
268 <div id="check-steps" role="status" aria-live="polite">
269 <ol class="check-steps">
270 ${steps.map(
271 (s) =>
272 html`<li data-step="${s.name}" data-state="${s.state}">
273 <span class="step-name">${s.name}</span>
274 <span class="step-state">${stepWord(s.state)}</span>
275 </li>`,
276 )}
277 </ol>
278 ${known
279 ? html`<progress class="tests-bar" aria-label="Test files started" max="${row.of ?? 0}" value="${row.started ?? 0}"></progress>`
280 : open
281 ? html`<progress class="tests-bar" aria-label="Test files started"></progress>`
282 : ""}
283 <p class="tests-words small muted">${[words, runningWords].filter((w) => w !== "").join(" · ")}</p>
284 </div>
285 <div class="pinned-slot" id="check-pins"${row.failures.length > 0 ? raw(' role="region" tabindex="0" aria-label="Failed tests"') : ""}>
286 ${when(
287 row.failures.length > 0,
288 () => html`<ul class="pinned-failures">
289 ${row.failures.map(
290 (f) => html`<li><strong>${f.name}</strong>${when(
291 f.lines.length > 0,
292 () => html`<pre>${f.lines.join("\n")}</pre>`,
293 )}</li>`,
294 )}
295 </ul>`,
296 )}
297 </div>
298 </div>`;
299 },
261300 )}
262301 <div class="run-live" id="run-live" data-state="${run.state}">
263302 ${when(!logHidden, () => logBox({
src/web/style/content.ts +8
@@ -867,6 +867,14 @@ form.inline { display: inline; margin-left: auto; }
867867 .checks-line .now { display: inline-flex; align-items: center; gap: 4px; color: var(--attn); }
868868 .check-steps { list-style: none; margin: 12px 0 0; padding: 0; display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 8px; }
869869 .check-steps li { display: flex; justify-content: space-between; gap: 8px; padding: 8px 12px; min-height: 2.5em; border: 1px solid var(--border); border-radius: var(--radius); }
870 .tests-bar { display: block; width: 100%; height: 8px; margin-top: 8px; }
871 .tests-words { margin: 4px 0 0; min-height: 1.4em; }
872 /* Reserved only while the run is open: pins arriving never move the log. */
873 .pinned-slot { max-height: 9em; overflow-y: auto; }
874 .check-progress[data-state="queued"] .pinned-slot, .check-progress[data-state="running"] .pinned-slot { height: 9em; }
875 .pinned-failures { list-style: none; margin: 8px 0 0; padding: 0; display: grid; gap: 6px; }
876 .pinned-failures li { padding: 6px 12px; border: 1px solid var(--danger); border-radius: var(--radius); }
877 .pinned-failures pre { margin: 4px 0 0; white-space: pre-wrap; overflow-wrap: anywhere; font-size: 0.85em; }
870878 @media (max-width: 480px) { .check-steps { grid-template-columns: minmax(0, 1fr); } }
871879 .check-steps .step-state { color: var(--faint); }
872880 .check-steps [data-state="ok"] .step-state { color: var(--success); }
test/check-steps.test.ts +22
@@ -128,6 +128,28 @@ describe("the terminal's step line", () => {
128128 ).toBe("install passed · typecheck passed · tests failed");
129129 });
130130
131 test("carries the tests row and the pinned failure names, withheld with the rest", () => {
132 const d = mkdtempSync(join(tmpdir(), "forge-steps-"));
133 for (const l of [
134 INSTALL,
135 CHECK,
136 "test-shards: 9 files in 3 shards",
137 "── shard 1/3: test/a.test.ts test/b.test.ts",
138 "error: boom",
139 "(fail) b breaks",
140 " 5 pass",
141 " 1 fail",
142 "── shard 1/3 done (exit 1, 2.0s)",
143 ])
144 appendRunLog(d, "r2", l);
145 const r = { ...run, id: "r2" } as Run;
146 expect(stepsLine(d, { state: "running", run: r }, true)).toBe(
147 "install passed · typecheck passed · tests running · tests 2 of 9 files started · 5 passed, 1 failed · failing: b breaks",
148 );
149 expect(stepsLine(d, { state: "running", run: r }, false)).toBe("");
150 rmSync(d, { recursive: true, force: true });
151 });
152
131153 test("says nothing to a reader who may not read the log, nor for a deploy", () => {
132154 expect(stepsLine(dir, { state: "running", run }, false)).toBe("");
133155 expect(
test/studio-runs.test.ts +72
@@ -688,6 +688,76 @@ describe("working: a run's own page in the studio", () => {
688688 ]);
689689 });
690690
691 test("a running check's page always has the bar and words, pins the failure outside the live region, and hides all from a reader without the log", async () => {
692 const mk = (lines: string[]) => {
693 const r = createRun(app.db, {
694 repoId: repo.id,
695 kind: "check",
696 commitOid: "c740479",
697 reason: "push",
698 });
699 claimRun(app.db, r.id);
700 for (const l of lines) appendRunLog(app.config.dataDir, r.id, l);
701 return r;
702 };
703 const run = mk([
704 "$ bun install",
705 "$ gate && tsc && bun scripts/test-shards.ts",
706 "test-shards: 9 files in 3 shards",
707 "── shard 1/3: test/a.test.ts",
708 "error: <b>boom</b>",
709 "(fail) a breaks",
710 " 4 pass",
711 " 1 fail",
712 "── shard 1/3 done (exit 1, 2.0s)",
713 "── shard 2/3: test/b.test.ts",
714 ]);
715 const page = await (await get(`/alice/engine/runs/${run.id}`, writerCookie)).text();
716 expect(page).toContain('<progress class="tests-bar" aria-label="Test files started" max="9" value="2"');
717 expect(page).toContain("2 of 9 files started · 4 passed, 1 failed · running test/b.test.ts");
718 expect(page).toContain("<strong>a breaks</strong>");
719 expect(page).toContain("error: &lt;b&gt;boom&lt;/b&gt;");
720 // The pins sit after the live region closes.
721 const live = page.indexOf('role="status"');
722 expect(live).toBeGreaterThan(-1);
723 expect(page.indexOf("pinned-failures")).toBeGreaterThan(page.indexOf("tests-words"));
724 expect(page.slice(live, page.indexOf("pinned-slot"))).not.toContain("a breaks");
725 // The replaced element (#check-steps) closes before the pin slot, which is
726 // patched in place so focus and scroll survive a refetch.
727 const at = page.indexOf('id="check-steps"');
728 expect(page.slice(at, page.indexOf('id="check-pins"'))).toContain("</div>");
729 expect(page.slice(at, page.indexOf('id="check-pins"')).match(/<div/g)?.length).toBe(
730 page.slice(at, page.indexOf('id="check-pins"')).match(/<\/div>/g)?.length,
731 );
732 // Nothing logged yet: the bar and the line are still there, empty.
733 const fresh = mk([]);
734 const quiet = await (await get(`/alice/engine/runs/${fresh.id}`, writerCookie)).text();
735 expect(quiet).toContain('<progress class="tests-bar" aria-label="Test files started"></progress>');
736 const block = quiet.slice(quiet.indexOf('id="check-steps"'), quiet.indexOf('id="run-live"'));
737 expect(block).not.toContain("value=");
738 expect(block).not.toContain("tabindex");
739 expect(block).not.toContain('role="region"');
740 // With pins, the slot is focusable and labelled; the live region holds no pin.
741 expect(page).toContain('class="pinned-slot" id="check-pins" role="region" tabindex="0" aria-label="Failed tests"');
742 expect(page.slice(live, page.indexOf("</div>", live))).toContain("tests-words");
743 expect(quiet).toContain('class="tests-words');
744 // A finished run with no known count shows no bar at all.
745 const done = mk(["$ bun install"]);
746 finishRun(app.db, done.id, { state: "failed", summary: "x" });
747 const ended = await (await get(`/alice/engine/runs/${done.id}`, writerCookie)).text();
748 expect(ended).toContain("check-progress");
749 expect(ended).not.toContain("<progress");
750 // A reader the log is hidden from gets none of it.
751 const plain = await createRepo(app, alice, { name: "plainb", visibility: "public", description: "x" });
752 if (!plain.ok) throw new Error(String(plain.error));
753 const hidden = createRun(app.db, { repoId: plain.value.id, kind: "check", commitOid: "c740479", reason: "push" });
754 claimRun(app.db, hidden.id);
755 appendRunLog(app.config.dataDir, hidden.id, "(fail) secret");
756 const hid = await (await get(`/alice/plainb/runs/${hidden.id}`, readerCookie)).text();
757 expect(hid).not.toContain("tests-bar");
758 expect(hid).not.toContain("secret");
759 });
760
691761 test("the page re-reads its rows as the run goes, and a reader who cannot see the log gets none", async () => {
692762 const script = await (
693763 await get(
@@ -705,6 +775,8 @@ describe("working: a run's own page in the studio", () => {
705775 )
706776 ).text();
707777 expect(script).toContain("getElementById('check-steps')");
778 expect(script).toContain("\\(fail\\) ");
779 expect(script).toContain("['role', 'tabindex', 'aria-label']");
708780 const plain = await createRepo(app, alice, {
709781 name: "plain",
710782 visibility: "public",
test/tests-row.test.ts added +45
@@ -0,0 +1,45 @@
1 import { expect, test } from "bun:test";
2 import { parseTestsRow, testsRowWords } from "../src/checks/tests-row.ts";
3
4 const log = [
5 "test-shards: 9 files in 3 shards",
6 "",
7 "── shard 1/3: test/a.test.ts test/b.test.ts",
8 "test/a.test.ts:",
9 "(pass) a works",
10 "error: expect(received).toBe(expected)",
11 "Expected: 1",
12 "Received: 2",
13 " at <anonymous> (test/b.test.ts:4:5)",
14 "(fail) b breaks",
15 " 5 pass",
16 " 1 fail",
17 "── shard 1/3 done (exit 1, 2.0s)",
18 "(fail) b breaks [shard 1]",
19 "── shard 2/3: test/c.test.ts test/d.test.ts test/e.test.ts",
20 ];
21
22 test("two finished files and one failure give the counts and one pinned failure", () => {
23 const row = parseTestsRow(log.slice(0, 13));
24 expect(testsRowWords(row)).toBe("2 of 9 files started · 5 passed, 1 failed");
25 expect(row.failures).toEqual([
26 {
27 name: "b breaks",
28 lines: ["error: expect(received).toBe(expected)", "Expected: 1", "Received: 2"],
29 },
30 ]);
31 expect(row.running).toBeUndefined();
32 });
33
34 test("the shard running now is named, an echoed failure is not pinned twice", () => {
35 const row = parseTestsRow(log);
36 expect(row.started).toBe(5);
37 expect(row.running).toEqual(["test/c.test.ts", "test/d.test.ts", "test/e.test.ts"]);
38 expect(row.failures.length).toBe(1);
39 });
40
41 test("a log without the runner's lines gives nothing", () => {
42 const row = parseTestsRow(["tsc ok"]);
43 expect(testsRowWords(row)).toBe("");
44 expect(row.failures).toEqual([]);
45 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs