Public: anyone can read this; the crew can change it.

forked from trevato/cs

What the planner cost is visible: the asks, the cache hits, the refusals and the fallbacks, in the terminal #402

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The round planner now keeps count of what it does. Each count is kept where the decision is made, and both the terminal and `/` show them. They are plain integers held by the server process and start again at zero when it restarts. This is a readout, not a metrics system. `bun run test` (typecheck and 1669 tests) passes, and none of the tests needs a key or the network.

shipped production is serving it deploy ae07c2d

opened this

The round planner now keeps count of what it does. Each count is kept where the decision is made, and both the terminal and / show them. They are plain integers held by the server process and start again at zero when it restarts. This is a readout, not a metrics system. bun run test (typecheck and 1669 tests) passes, and none of the tests needs a key or the network.

What is counted

  • src/plan/typesafe.ts: rounds asked, answers from the cache, requests that joined one already in flight, rounds a model answered, and failures by kind (timeout, connection, each HTTP status, and an answer that was not a plan). Every round ends up in exactly one of those, and a test checks that. HTTP requests are counted inside the fetch handed to the SDK, so the retry counts as a request, since that is what is billed.
  • Tokens, not money: the SDK reports input and output tokens and no price, so the readout counts tokens and says so, rather than inventing a currency.
  • Limiters: src/limit.ts now counts its own refusals, which gives the per-caller limit and the process-wide spend limit (PLAN_LIMIT, PLAN_SPEND) separate counts. The /api/plan route counts one thing itself: a round asked of a server that has no key.
  • Logging: the per-round console.warn on fallback is removed. The failure counts replace it, and nothing logs a line per round.

Where it shows

  • src/plan/readout.ts holds the readout's shape and one sentence that both the page and the terminal print, so the two can't disagree.
  • Terminal: the policy tab now has a "who plans the bots' rounds" block under the card. It shows whether the planner is on, the model, the rate limits, the timeout and retry, the cache size, then every count and when counting started. The key is never shown; a test puts one in the session's environment and checks the screen never contains it.
  • /: the sentence sits under the paragraph about planning, rendered on every request. With no key it says plainly that no round has been asked of a model.

The file under FORGE_APP_DATA The terminal runs as a separate process, so it can't read the server's counts directly. The server writes a copy to FORGE_APP_DATA/planner.json: zeros at start-up, then at most once a second after a round is planned, off the response path. It writes to a temporary file and renames it over the old one, and the terminal treats the file as input and validates it. The file only mirrors the running server; it is not stored history.

Not done: no history across restarts, no per-model breakdown, and no dollar figure.

To try it: in a preview (no key), open the terminal's policy tab. It shows the planner off and the count of rounds asked with no key. / says the same in words under the planning paragraph. With TYPESAFE_API_KEY set, the counts fill in as matches are played.

🤖 Generated with Claude Code

Closes #246

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs