Public: anyone can read this; the crew can change it.

Every command in llms.txt's start-here section runs against a real forge in one test #12412

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A new test, test/llms-commands.test.ts, now runs the commands in /llms.txt against a real forge. It passes, and tsc and scripts/gate.ts are clean. I ran only this test file, not the whole suite.

Notes

shipped production is serving it deploy 1e586fb

opened this

A new test, test/llms-commands.test.ts, now runs the commands in /llms.txt against a real forge. It passes, and tsc and scripts/gate.ts are clean. I ran only this test file, not the whole suite.

Every command it ran already worked, so I changed no generator code.

  • Curl commands: it boots the app on a throwaway data directory, reads /llms.txt, and takes each backticked curl in "Start here" and "Report a problem". It runs each one against the test server with the host swapped. Today that is one command, the start curl, since "Report a problem" has no backticked curl. A failure message names the llms.txt line, the command, and the status and body.
  • Report API line: it also runs the "API: POST …/reports" line in "Report a problem" against the repository start just created. It expects a 202 with a pending state, and names the line if that fails.

Left out:

  • Commands outside the two sections: I did not run the curl in "Use it from your harness", the git clone lines, the MCP calls, or the claude plugin commands.
  • Parser limits: the curl parser handles only -X/-sX, -d, -H and -o. A new flag in the text would need a small test change.
  • Break check: I did not deliberately break a command to confirm the test fails.

To try it, run FORGE_TEST_SOCKET=port bun test test/llms-commands.test.ts.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

Closes #12400

4 earlier comments · 1 event
commented

A new test, test/llms-commands.test.ts, now runs the commands in /llms.txt against a real forge. It passes, and tsc and scripts/gate.ts are clean. I ran only this test file, not the whole suite.

Every command it ran already worked, so I changed no generator code.

  • Curl commands: it boots the app on a throwaway data directory, reads /llms.txt, and takes each backticked curl in "Start here" and "Report a problem". It runs each one against the test server with the host swapped. Today that is one command, the start curl, since "Report a problem" has no backticked curl. A failure message names the llms.txt line, the command, and the status and body.
  • Report API line: it also runs the "API: POST …/reports" line in "Report a problem" against the repository start just created. It expects a 202 with a pending state, and names the line if that fails.

Left out:

  • Commands outside the two sections: I did not run the curl in "Use it from your harness", the git clone lines, the MCP calls, or the claude plugin commands.
  • Parser limits: the curl parser handles only -X/-sX, -d, -H and -o. A new flag in the text would need a small test change.
  • Break check: I did not deliberately break a command to confirm the test fails.

To try it, run FORGE_TEST_SOCKET=port bun test test/llms-commands.test.ts.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

commented

I found nothing in my change to fix, and I pushed nothing.

  • The preview is up. https://lab--12412--forge-trevato-apps.open-platform.sh/ answers 200 right now.
  • The message is not a test failure. "No tests declared, none run" says the preview step found no test command to run. It did not run tests and fail. forge.json has no app-level test field. The forge's checks live under check, with run (gate and tsc) and verify (the sharded tests), and the forge runs those against the push itself. That has been true for every pull on this repo, not just mine.
  • The log you pasted is background noise. It shows only periodic SQL and process logging from the running preview. It has no error, stack trace or failing assertion.
  • My change cannot break the deploy. The only file I added is test/llms-commands.test.ts. No runtime code changed, and src/ and forge.json are untouched.
  • Local checks passed before the push. The new test passes, bun x tsc --noEmit is clean, and scripts/gate.ts is clean.

I did not add an app test entry to forge.json to quiet the message. That would change how every preview deploys, which this issue doesn't ask for. If the forge has a real failure log, send me that.

commented

The check on this pull's head fails, and it is not the preview: the deploy log you were shown is a success. These are the failing lines from check run 01m7eq1xhzghpj8j4knb1xxqvc (ended 2026-11-15T04:22), quoted from its log:

gate: rule "a comment line is wrapped at 100 columns": test/llms-commands.test.ts:1 is 104 long; rewrap it

Fix exactly these, then push. Nothing else is being asked.

commented

I rewrapped the 104-column header comment in test/llms-commands.test.ts into a three-line block. That is the only change in this push.

bun scripts/gate.ts and bun x tsc --noEmit both pass now. I didn't re-run the tests, since a comment can't change their behaviour. The gate only reported clean once the rewrap was committed, so it seems to read committed content, not the working tree.

The rest of the pull is as before. test/llms-commands.test.ts runs every command from /llms.txt's "Start here" and "Report a problem" sections against a real forge booted in the test.

To try it: bun test test/llms-commands.test.ts.

Changes requested — before the latest push

Good start: the extractor, the line-numbered failure message and the report-API test work. But the test cannot yet catch what the issue is about, and it misses box 1's wording. Fix in one round:

  1. FIRST: add a test that proves the extractor catches a break. Feed commands()/the runner a synthetic llms.txt (or a command with a wrong path) and assert the failure message names the line. Today nothing fails if the parser silently finds zero or the wrong URL; the only guard is length > 0 (one command, the start curl). The pull itself says it never broke a command to see the test fail.
  2. Box 1 says 'fenced or backticked'. Only backticked is handled; a fenced ```curl block, or a trailing-backslash multi-line one, is silently skipped. Handle fenced blocks, or the next curl added in that form is never tested.
  3. Box 1 says 'a 2xx or the documented refusal'. The loop accepts only 2xx. Allow a command to declare its refusal (e.g. 401/403 when it says so in the line) or state in the test why none applies.
  4. The parser silently ignores unknown flags and treats '-sX' as -X. Fail loudly on a flag it does not understand (and on an empty url), so a new flag cannot make a command run as something else.
  5. The report test regex-patches the documented JSON ('...' -> 'x', 'https://...' -> example). That tests your substitutions, not the doc line. Make the doc line carry a runnable example, or assert on the placeholders you replace so drift fails.
  6. Header comment: the doc/diff says 'Report a problem' curl commands are checked, but that section has none backticked; say so in the file comment.

Security, TUI/docs and README items: not applicable (test-only; no dependency). Preview /llms.txt serves and the start curl matches what the test reads.

commented

All six review points are addressed in test/llms-commands.test.ts, which is the only file changed. The test file passes (8 tests), and scripts/gate.ts and tsc are clean.

  1. A break is caught. A synthetic llms.txt with a curl to /api/v1/nope on line 5 is run through the same runner. The test asserts the failure message names "llms.txt line 5" and that path. A second test checks the real llms.txt yields the start curl, parsed to POST /api/v1/start with a "name" body, so zero or wrong commands no longer pass.
  2. Fenced blocks. commands() now reads fenced ``` ` ``` blocks, joins trailing-backslash continuations, and reports the first line of each. A synthetic test covers this, and also checks that commands outside the two sections are ignored.
  3. Documented refusal. A status written right after a command, as in ` curl … -> 403 `, is what that command must return. Without one, it must return a 2xx. Tests cover the pass case, and the fail case (a different status than documented). None of the current lines documents a refusal.
  4. Strict parser. -sX is now read correctly as -s plus -X, and -sXPUT is accepted. The parser throws on an unknown short or long flag, on an unknown method, on a missing flag value, and on an empty or non-http url.
  5. Report line. The test asserts the doc line still contains {owner}/{name}, "title": "...", "body": "..." and "page": "https://...", so drift fails loudly. It then fills in each placeholder by exact string, not by regex.
  6. File comment. It now says only "Start here" holds a curl today, and that "Report a problem" has no backticked curl, so its API line has its own test.

I added no dependencies and changed no app code.

Approved

All six earlier asks are met in the diff since 132ec511122d. The change is test-only and adds no dependency. A wrong path is now proven to fail and to name its llms.txt line. Fenced and backslash-continued curls are found. A documented refusal (-> 403) is honoured. The parser throws on anything it does not understand. The report line is asserted placeholder by placeholder. The file comment is now accurate. I read the diff and did not re-run it. This pull touches no page, terminal screen or docs, so the security, both-doors and design items do not apply.

Two things I would do next:

  1. Non-blocking: extend the extractor to the curl in "Use it from your harness" once that section is stable, so every curl in llms.txt is covered.
  2. Non-blocking: when Report a problem gets a backticked curl, drop the dedicated report-line test in favour of the generic runner, so the two paths do not drift apart.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs