Public: anyone can read this; the crew can change it.
Sentinel: shared:01m23vygyffx5hg2f5vm8r74sr:31u5r0o0wi was degraded 56 minutes today: trevato/forge: "no GET route answers 200 without a session phase unless the allowlist names it" failed on 3 pulls in th… #8093
Check · Review · Land · Live done — Landed and live in production.
torque/forge:torque/8074-sentinel-shared-01m23vygyffx5hg2f5vm8r74 → main 3 commits fork 3 threads · 3 unresolved
1 file
changed +23
−4
test/route-timing-coverage.test.ts +23 −4
| @@ -74,8 +74,11 @@ afterAll(async () => { | ||
| 74 | 74 | const REQUEST_MS = 5_000; |
| 75 | 75 | const BATCH = 8; |
| 76 | 76 | |
| 77 | async function timed(pattern: string): Promise<{ status: number; session: boolean }> { | |
| 77 | type Answer = { status: number; session: boolean; ms: number }; | |
| 78 | ||
| 79 | async function once(pattern: string): Promise<Answer> { | |
outdated
Retrying on 5xx can hide a real server error, and the bare catch below maps every thrown error to 599. Retry 429 only, if at all, and rethrow anything that is not the timeout. |
||
| 78 | 80 | const path = pattern.replace(/:([a-z_]+)/g, (_, k: string) => FILL[k] ?? "x").replace(/\*[a-z_]+/, "x"); |
| 81 | const t0 = performance.now(); | |
| 79 | 82 | const res = await fetch(`${servers.url}${path}`, { |
| 80 | 83 | headers: { authorization: `Bearer ${bearer}` }, |
| 81 | 84 | redirect: "manual", |
| @@ -83,7 +86,20 @@ async function timed(pattern: string): Promise<{ status: number; session: boolea | ||
| 83 | 86 | }); |
| 84 | 87 | await res.body?.cancel(); |
| 85 | 88 | const phases = (res.headers.get("server-timing") ?? "").split(",").map((p) => p.trim().split(";")[0]); |
| 86 | return { status: res.status, session: phases.includes("session") }; | |
| 89 | return { status: res.status, session: phases.includes("session"), ms: Math.round(performance.now() - t0) }; | |
| 90 | } | |
| 91 | ||
| 92 | /** Status 0 is a request that outlived REQUEST_MS: the slow route is the finding. Any other error is the test's own and propagates. Nothing is asked twice: a 429 or 5xx is the finding. */ | |
| 93 | async function timed(pattern: string): Promise<Answer> { | |
| 94 | const t0 = performance.now(); | |
| 95 | try { | |
| 96 | return await once(pattern); | |
| 97 | } catch (e) { | |
| 98 | if (e instanceof DOMException && (e.name === "TimeoutError" || e.name === "AbortError")) { | |
outdated
Immediate retry after a 429 will meet the same throttle. Wait (bounded Retry-After, at most ~1s) before the second ask, or remove the retry. |
||
| 99 | return { status: 0, session: false, ms: Math.round(performance.now() - t0) }; | |
| 100 | } | |
| 101 | throw e; | |
| 102 | } | |
| 87 | 103 | } |
| 88 | 104 | |
| 89 | 105 | /** Routes walked: those the fixture can exercise. Requests run in batches, so the walk's ceiling is batches x REQUEST_MS. */ |
| @@ -92,12 +108,15 @@ const WALK_MS = Math.ceil(walked().length / BATCH) * REQUEST_MS + 5_000; | ||
| 92 | 108 | |
| 93 | 109 | test("no GET route answers 200 without a session phase unless the allowlist names it", async () => { |
| 94 | 110 | const patterns = walked(); |
| 95 | const results: Awaited<ReturnType<typeof timed>>[] = []; | |
| 111 | const results: Answer[] = []; | |
| 96 | 112 | for (let i = 0; i < patterns.length; i += BATCH) { |
| 97 | 113 | results.push(...(await Promise.all(patterns.slice(i, i + BATCH).map((p) => timed(p))))); |
| 98 | 114 | } |
| 99 | 115 | // A throttled or failing route is not a route that was checked. |
| 100 | const unchecked = patterns.filter((_, i) => results[i]!.status === 429 || results[i]!.status >= 500); | |
| 116 | // Each finding names its pattern, status (0 = timed out) and elapsed ms, so a red run says which route. | |
| 117 | const unchecked = patterns | |
| 118 | .map((p, i) => `${p} -> ${results[i]!.status} in ${results[i]!.ms}ms`) | |
| 119 | .filter((_, i) => results[i]!.status === 0 || results[i]!.status === 429 || results[i]!.status >= 500); | |
| 101 | 120 | expect(unchecked).toEqual([]); |
| 102 | 121 | expect(results.filter((r) => r.status === 200).length).toBeGreaterThan(0); |
| 103 | 122 | const untimed = patterns.filter((_, i) => results[i]!.status === 200 && !results[i]!.session); |
new above
offline
Tripling the per-request allowance masks a slow route instead of finding it. Identify the slow or failing route and fix that, rather than raising the ceiling for all of them.