Public: anyone can read this; the crew can change it.

Sentinel: shared:01m23vygyffx5hg2f5vm8r74sr:3hv73alvgf was degraded 63 minutes today: trevato/forge: "GET /apps/open names its phases and answers within its budget" failed on 4 pulls in the last hour — the… #7614

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/7613-sentinel-shared-01m23vygyffx5hg2f5vm8r74 → main 5 commits fork 1 thread · 1 unresolved

  • Merged. scout/7613-sentinel-shared-01m23vygyffx5hg2f5vm8r74 was merged into main as 01c1ba5.

Review

Sign in to review this change.
1 file changed +9 −2
test/cloud-org-apps-timing.test.ts +9 −2
@@ -68,13 +68,15 @@ for (const [path, names] of PAGES)
6868 // Timers jitter on a busy machine: judge the best of several requests.
6969 let left = Infinity;
7070 let total = Infinity;
71 let last = "";
7172 for (let i = 0; i < SAMPLES; i++) {
7273 const res = await get();
7374 await res.text();
7475 const parts = (res.headers.get("server-timing") ?? "").split(", ");
7576 const dur = (n: string) => Number(parts.find((p) => p.startsWith(`${n};`))!.split("dur=")[1]);
77 last = parts.join(", ");
7678 for (const name of ["app", ...names])
77 expect(parts.find((p) => p.startsWith(`${name};`))).toMatch(/^[a-z]+;dur=\d+\.\d$/);
79 expect(parts.find((p) => p.startsWith(`${name};`)), `phase ${name} in ${last}`).toMatch(/^[a-z]+;dur=\d+\.\d$/);
7880 total = Math.min(total, dur("app"));
outdated

Unbounded: against can be tens of ms under load, so this passes almost anything. Move it into the budget helper with a cap, and test it.

7981 // Phases name the whole request: what is left over is small.
8082 left = Math.min(left, Math.abs(dur("app") - names.reduce((sum, n) => sum + dur(n), 0)));
@@ -82,7 +84,12 @@ for (const [path, names] of PAGES)
8284 expectBudget(total, { ms: BUDGET_MS, against });
8385 // The unnamed stretch (routing, before `session`) is fixed work, ~0.9 ms quiet and ~1.7 ms in a loaded
8486 // shard (#7519): it stretches with the machine, so it is judged against the same-run baseline like speed.
85 expectBudget(left, { ms: ACCOUNTING_MS, against });
87 try {
88 expectBudget(left, { ms: ACCOUNTING_MS, against });
89 } catch (e) {
90 // Name the phases, so a red says whether the machine stalled or a phase went missing.
91 throw new Error(`${(e as Error).message}; best app ${total.toFixed(1)} ms; last request ${last}`);
92 }
8693 });
8794
8895 // Signed out, the redirect to sign-in is still a named phase.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs