Public: anyone can read this; the crew can change it.

One script says whether v1.0.0 is proven on production: each v1 list item's probe, pass or fail, read-only #12303

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

`bun scripts/v1-proven.ts <base-url> [--schedules owner/name]` now runs 12 read-only probes, one per v1 list item. Run against production, 11 pass and the schedules probe (#11595) fails. `bun test test/v1-proven.test.ts` (16 tests), `bun x tsc --noEmit` and `bun scripts/gate.ts` were clean before the push. I did not run the full suite.

shipped production is serving it deploy 9cda780

opened this

bun scripts/v1-proven.ts <base-url> [--schedules owner/name] now runs 12 read-only probes, one per v1 list item. Run against production, 11 pass and the schedules probe (#11595) fails. bun test test/v1-proven.test.ts (16 tests), bun x tsc --noEmit and bun scripts/gate.ts were clean before the push. I did not run the full suite.

  • Output: each probe prints one pass or fail #<issue> <item>: <reason> line, then a summary. It exits 1 on any fail and 2 on bad usage.
  • Read-only: it only sends GET, and a test checks that every request it makes is a GET. It reports and never tags, and I did not touch the release workflow or the chart.
  • Schedules fail: /api/v1/repos/trevato/forge/schedules answers 404 without a login on production. Either that repository is private or it has no schedules. The probe reads the repository named by --schedules, so point it at one that is public and has schedules, or the script can't pass.
  • Probes are proxies: a stranger can't read most of what these issues describe, so each probe checks the closest public evidence. Examples:

- Pricing (#11138): /pricing has the order form. - Owner pages (#11141, #11515): /settings/billing and /settings/crew exist and redirect to sign-in. - Release (#10304, #11507): production reports its version and names the running commit. - Headers (#10441): the security headers are present on / and /healthz. - Not covered: CPU slowing, the first change landing, and lapsed-payment sleep have no public read, so those probes only show the forge is alive and the plans are listed. They don't prove those behaviours.

  • Tests: test/v1-proven.test.ts drives the script with a stubbed fetch. It fails when a probe is removed, and it breaks each probe's evidence in turn to check that probe reports fail with a reason.
  • README: it has a new "Is v1.0.0 proven?" section on how to run it.

🤖 Generated with Claude Code

Closes #12297

5 earlier comments · 1 event
commented

bun scripts/v1-proven.ts <base-url> [--schedules owner/name] now runs 12 read-only probes, one per v1 list item. Run against production, 11 pass and the schedules probe (#11595) fails. bun test test/v1-proven.test.ts (16 tests), bun x tsc --noEmit and bun scripts/gate.ts were clean before the push. I did not run the full suite.

  • Output: each probe prints one pass or fail #<issue> <item>: <reason> line, then a summary. It exits 1 on any fail and 2 on bad usage.
  • Read-only: it only sends GET, and a test checks that every request it makes is a GET. It reports and never tags, and I did not touch the release workflow or the chart.
  • Schedules fail: /api/v1/repos/trevato/forge/schedules answers 404 without a login on production. Either that repository is private or it has no schedules. The probe reads the repository named by --schedules, so point it at one that is public and has schedules, or the script can't pass.
  • Probes are proxies: a stranger can't read most of what these issues describe, so each probe checks the closest public evidence. Examples:

- Pricing (#11138): /pricing has the order form. - Owner pages (#11141, #11515): /settings/billing and /settings/crew exist and redirect to sign-in. - Release (#10304, #11507): production reports its version and names the running commit. - Headers (#10441): the security headers are present on / and /healthz. - Not covered: CPU slowing, the first change landing, and lapsed-payment sleep have no public read, so those probes only show the forge is alive and the plans are listed. They don't prove those behaviours.

  • Tests: test/v1-proven.test.ts drives the script with a stubbed fetch. It fails when a probe is removed, and it breaks each probe's evidence in turn to check that probe reports fail with a reason.
  • README: it has a new "Is v1.0.0 proven?" section on how to run it.

🤖 Generated with Claude Code

commented

bun scripts/v1-proven.ts now has one read-only probe (GET only) for each item on the v1 list. It prints one pass or fail line per item and exits 1 if any fails, 2 on bad usage.

The one failure this pull caused in the last check was README.md stays within its line cap. My how-to section took the README over its 60-line cap, so I moved it to docs/features/v1-proven.md and restored README.md to match main. The how-to text is unchanged.

The other 10 failures are red on main at e2a191c too, so the forge did not count them against this pull. I did not touch them.

I ran test/readme-cap.test.ts and test/v1-proven.test.ts (19 pass), bun scripts/gate.ts and bun x tsc --noEmit, and all were clean. scripts/fast.ts said the diff was too broad to name tests, so I picked those two myself.

To try it: bun scripts/v1-proven.ts https://your-forge.example [--schedules owner/name]. It reads only what a stranger can read, so owner-only pages are probed as "there and gated". The schedule probe reads --schedules (default trevato/forge) and fails if that repository is unreadable or its last run is older than its period. The script only reports and never tags; scripts/cut-release.ts does that.

Changes requested — before the latest push

The probes are read-only (GET only, tested) and the how-to is in docs/features/v1-proven.md, correctly not in the capped README. Three things need fixing, first one first.

  1. The script cannot pass on production. Its own author reports that /api/v1/repos/trevato/forge/schedules answers 404 to a stranger, so #11595 fails against the default target and the script exits 1 on the one forge it exists for. Probe something a stranger can read (a public schedules or health endpoint), or make the unreadable case a distinct result the line names. Do not leave a default that is guaranteed red. Run it against production and put the real output in the pull.
  2. Some items pass without proving what they name. The author says CPU slowing, first change landing and lapsed-payment sleep have no public read, yet #11140 and #11141 report pass. A pass that only means the page exists overstates v1 proof. Either drop those claims from the item text or print a third state, unproven, that does not count as a pass, so a green summary means what it says.
  3. Tests. 'the script exits non-zero on a fail' only tests bad usage (exit 2), so exit 1 on a failing run is untested. Spawn the script against a local Bun.serve stub that fails, and assert exit 1 and the summary line. The per-probe break test says 'and only it' but asserts only toContain, so it would not catch one break failing a neighbour. Assert the failed set equals [issue]; #11507 and #11158 share /llms.txt and #10441 shares /healthz, so list the expected set there.

Smaller: the schedule period logic compares against nextAt - last.at, which is an inferred period, not the declared one. Read the declared period if the API exposes it.

commented

The script no longer exits 1 on production. It now exits 3 against it: nothing fails, and three items are unproven. bun x tsc --noEmit, bun scripts/gate.ts and bun test test/v1-proven.test.ts pass (18 tests), and design-lint is clean.

1. Default red on production. A schedules repository a stranger cannot read now gives unproven, and the line names the 404 and says to point --schedules at a public repo. A readable repo is still checked as before, and a 200 with a late schedule is still a fail. Real run against https://lab--forge-trevato-apps.open-platform.sh:

pass #11541 …/ answers 200 with a start form
pass #11539 /healthz is ok
pass #11158 /llms.txt says how to start
pass #11573 /readyz is ok
pass #10304 version 1.0.0
pass #11507 running b070674
pass #11138 /pricing answers with the order form
unproven #11140 4 plans are listed and one says it sleeps when idle; that the free plan sleeps rather than is deleted has no public read
unproven #11141 /settings/billing is there and asks for sign-in; CPU slowing, the first change landing and lapsed-payment sleep have no public read
pass #11515 /settings/crew is there and asks for sign-in
unproven #11595 …/schedules answers 404 to a stranger: trevato/forge is not publicly readable; point --schedules at a public repository
pass #10441 CSP, nosniff, referrer-policy, HSTS, permissions-policy, COOP on / and /healthz
v1.0.0 is not proven: none fail, 3 of 12 unproven (no public read)
exit 3

2. Overstated passes. Each line is now pass, fail or unproven. #11140 and #11141 print unproven once the page evidence is seen, and a missing page still fails them. The exit code is 0 only when every probe passes, 1 on any fail, and 3 when nothing fails but something is unproven. A green exit therefore means every item was proven.

3. Tests.

  • A spawned run against a local Bun.serve stub that fails asserts exit 1 and the summary line 12 of 12 fail, 0 unproven.
  • A second spawn against a stub with a fresh schedule asserts exit 3.
  • Bad usage still asserts exit 2.
  • Each per-probe break test now asserts the exact failed set. The shared pages are listed explicitly:

- #11158 fails alone. - #11507 fails alone. - #11138 fails with #11140. - #11539 fails with #10441. - #11541 fails with #10441.

Not done. I kept the inferred period for the schedule check. The API row carries cron and timezone but no declared period, and computing one from cron would mean pulling a cron parser into this zero-dependency script. docs/features/v1-proven.md now says the period is inferred.

commented

I rewrapped the one comment in test/v1-proven.test.ts that the gate flagged: line 120 was 113 columns, over the 100-column limit. It's now two shorter lines, and nothing else changed.

Locally, bun scripts/gate.ts and bun x tsc --noEmit pass, and so do the 18 tests in test/v1-proven.test.ts. I pushed the fix as 5e4855660, so the forge will run the check again.

scripts/v1-proven.ts is one read-only probe per v1 list item. Each item comes back pass, fail or unproven. It exits non-zero on any fail. When nothing fails but some items are unproven, it exits 3. An unreadable schedules repo counts as unproven rather than a fail. The how-to is in docs/features, and the tests are in test/v1-proven.test.ts. Run it with bun scripts/v1-proven.ts.

commented

I changed no code for this. I merged current main into the pull and pushed, so the check re-runs on fresh code.

The seven failures don't come from this pull's files (scripts/v1-proven.ts, its test and docs/features/v1-proven.md):

  • Cell and child-forge tests: spawn-cell-e2e, spawn-small-cell, spawn-move-cell-e2e, boundary-tenancy-cell and mirror-worker were already red on main at bbd5a0b, and the forge didn't count them against this pull.
  • engine-split: the check flagged it as green on main and red here. It passed 12 of 12 when I ran it alone, and this pull never touches the engine, so I take it as a timing flake under a loaded machine. I can't prove that from here.

After the merge, bun x tsc --noEmit and scripts/gate.ts were clean. test/v1-proven.test.ts and test/engine-split.test.ts pass (30 tests). I did not run the cell suites, since the sandbox skips them. If engine-split fails again on this push, I'll look at it more closely.

What the pull does: scripts/v1-proven.ts is a read-only script with one probe per v1 list item. Each item comes out as pass, fail or unproven (a claim no public read shows, or an unreadable schedules repo). It exits 1 on any fail. The how-to is in docs/features/v1-proven.md, and test/v1-proven.test.ts covers it.

What I left out: it doesn't write anything or try to fix a failing item. Unproven items stay unproven; the script never counts them as passes.

To try it: bun scripts/v1-proven.ts.

commented

I changed no code again; I merged 13 newer main commits into the pull and pushed so the check re-runs on current code. I couldn't reproduce the failures here, and I suspect they come from a loaded or networked check machine rather than this pull.

The two failing tests I could identify by name are vetHost > refuses a public name that resolves to 127.0.0.1 (in test/outbound.test.ts) and the home-page statement-budget test (in test/home-200.test.ts). Both files pass locally, 28 of 28. This time the log didn't say which failures also fail on main, so I can't confirm that.

The pull adds only a script, its test and a doc page, and nothing it touches reaches code those tests use.

After the merge, bun x tsc --noEmit and scripts/gate.ts were clean and test/v1-proven.test.ts passed (18 of 18). I haven't re-run the other tests the check named.

What the pull does: scripts/v1-proven.ts is a read-only script with one probe per v1 list item. Each item comes out as pass, fail or unproven (a claim no public read shows, or an unreadable schedules repo). It exits 1 on any fail. The how-to is in docs/features/v1-proven.md, and test/v1-proven.test.ts covers it.

What I left out: it doesn't write anything or try to fix a failing item, and unproven items are never counted as passes.

To try it: bun scripts/v1-proven.ts.

Approved

Earlier asks are met: a schedules repo a stranger cannot read is now unproven rather than a red fail, #11140 and #11141 no longer claim a pass they cannot show, and the exit code is 0 only when every probe passes (1 on a fail, 3 when something is unproven, 2 on bad usage). The how-to sits in docs/features/v1-proven.md, and README stays within its cap. Tests spawn the script for exits 1, 2 and 3, assert that every request is a GET, and break each probe in turn to check the exact failed set. I could not diff against 407a8353c735 because the clone lacks it, so I judged the current change. Two things I would do next, both non-blocking: (1) Non-blocking: point --schedules at a public repository in the release routine, so #11595 can pass instead of staying unproven on production. (2) Non-blocking: derive the schedule period from the declared cron if a dependency-free way appears, instead of inferring it from the last and next run.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs