Public: anyone can read this; the crew can change it.

The model-review proposal can be applied or declined through the existing scorecard MCP tool, in the words of the page's button #15282

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The scorecard MCP tool now takes action: apply|decline with change, the id of one waiting model-review proposal. It does what the page's Move to … / Leave it buttons do and answers with the page's sentence. I did not run the full suite; the forge runs it on this push.

Notes

shipped production is serving it deploy ccd02b1

opened this

The scorecard MCP tool now takes action: apply|decline with change, the id of one waiting model-review proposal. It does what the page's Move to … / Leave it buttons do and answers with the page's sentence. I did not run the full suite; the forge runs it on this push.

  • What it answers: { change, agent, from, to, state, says }. says is "sonnet it is." on apply and "Left as it was." on decline. A change that is no longer waiting gets "That change is no longer waiting."
  • Who may: only a person who administers the forge. An agent is refused outright with a new sentence, "the model review's button is a person's who administers the forge; an agent cannot press it". Agents can't move a model, their own or anyone else's.
  • Same words as the page: the page and the tool both take their wording from CHANGE_WORDS in src/actions/model_review.ts, so they can't drift. The page's behaviour is unchanged. The tool's code path is answerChange, which calls the existing decideChange (the one the page and terminal use).
  • No-argument call: it still reads the scorecard as before. readsWithout: "action" keeps a read-only token able to make that read.
  • Size: no tool was added and no ceiling was raised. test/mcp-size.test.ts passes.
  • Docs: docs/features/model-review.md now says the button is on the tool, and doc on the tool carries it for docs/MCP.md.

Tests: test/mcp-scorecard-decide.test.ts covers apply, decline, an agent being refused with the model and change untouched, and a bad or missing argument. I also ran the model-review, scorecard, MCP size, MCP doc and MCP tools tests. All of them pass, and tsc is clean. scripts/fast.ts said the diff was too broad to name tests, so I picked these myself.

Not done: the review still doesn't look at an agent running on the runner's default model.

Push: the pre-push hook wants a "Try:" commit, so the last commit is an empty one: call scorecard with { "action": "decline", "change": "<id from modelReview.changes>" } as an admin token.

No page change, so there are no review steps or before/after looks.

Closes #15265

6 earlier comments · 1 event
commented

The scorecard MCP tool now takes action: apply|decline with change, the id of one waiting model-review proposal. It does what the page's Move to … / Leave it buttons do and answers with the page's sentence. I did not run the full suite; the forge runs it on this push.

  • What it answers: { change, agent, from, to, state, says }. says is "sonnet it is." on apply and "Left as it was." on decline. A change that is no longer waiting gets "That change is no longer waiting."
  • Who may: only a person who administers the forge. An agent is refused outright with a new sentence, "the model review's button is a person's who administers the forge; an agent cannot press it". Agents can't move a model, their own or anyone else's.
  • Same words as the page: the page and the tool both take their wording from CHANGE_WORDS in src/actions/model_review.ts, so they can't drift. The page's behaviour is unchanged. The tool's code path is answerChange, which calls the existing decideChange (the one the page and terminal use).
  • No-argument call: it still reads the scorecard as before. readsWithout: "action" keeps a read-only token able to make that read.
  • Size: no tool was added and no ceiling was raised. test/mcp-size.test.ts passes.
  • Docs: docs/features/model-review.md now says the button is on the tool, and doc on the tool carries it for docs/MCP.md.

Tests: test/mcp-scorecard-decide.test.ts covers apply, decline, an agent being refused with the model and change untouched, and a bad or missing argument. I also ran the model-review, scorecard, MCP size, MCP doc and MCP tools tests. All of them pass, and tsc is clean. scripts/fast.ts said the diff was too broad to name tests, so I picked these myself.

Not done: the review still doesn't look at an agent running on the runner's default model.

Push: the pre-push hook wants a "Try:" commit, so the last commit is an empty one: call scorecard with { "action": "decline", "change": "<id from modelReview.changes>" } as an admin token.

No page change, so there are no review steps or before/after looks.

commented

<!-- look-pages d3af888cc9cf --> Look pages for d3af888, from the gate:

  • look pages (none): this pull changes src/web/pages/crew.ts but the forge could not trace the route of /settings/security/controls (an inline or wrapped handler), so it could not infer a page; the reviewer decides. A "Looks: <path>" line naming the nearest page, or a "No look: <why>" line (10+ characters), says it
commented

The MCP scorecard tool can now apply or decline a waiting model-review proposal, using the same words as the page. The first push failed the "both doors" gate rule, and I've pushed a fix that satisfies it. I didn't re-run the gate, tests or type check here, so the forge's check on this push is the first verdict on the fix.

  • What changed: scorecard takes action: apply|decline and a waiting change's id. The decision words now live in CHANGE_WORDS in src/actions/model_review.ts. The crew page, modelReviewDecide in src/web/pages/crew.ts, reads them from there. The wording shown on the page is unchanged. docs/features/model-review.md and test/mcp-scorecard-decide.test.ts are updated.
  • The gate failure: the page edit only moves where the words come from, so the terminal has nothing new to show. An empty commit carries a No door: line saying so, plus a No look: line for the "could not trace /settings/security/controls" warning. Both are in the commit message.
  • Not done: the gate also warned that call in test/mcp-scorecard-decide.test.ts looks 0.71 similar to mcp in test/xlarge-size.test.ts. It is a note, not a failure, and I left the two helpers separate.
  • To try it: call the scorecard tool with action: "apply" or "decline" and the id of a waiting change. The crew settings page at /settings/crew shows the same message afterward.

🤖 Generated with Claude Code

commented

I made no new change for this feedback, because the look-pages comment was written against d3af888, before my last push. That push, 3f647330b, already carries the line the gate asked for: No look: the rendered words are unchanged, only their source moved to a shared constant. I haven't seen a gate or check result for that commit yet, so I can't say it clears the warning.

I did not add a Looks: <path> line, because there is no page to point at. The gate couldn't trace the route behind src/web/pages/crew.ts. Only the source of the decision words moved, to CHANGE_WORDS, and the page renders the same text. If you'd rather name /settings/crew as the nearest page, say so and I'll add the line.

What changed: the scorecard MCP tool now takes action: apply|decline plus a waiting change's id, answering in the page's own words. Those words now live in CHANGE_WORDS in src/actions/model_review.ts, and the crew page reads them from there. docs/features/model-review.md and test/mcp-scorecard-decide.test.ts are updated.

Left alone: the gate's note that call in test/mcp-scorecard-decide.test.ts resembles mcp in test/xlarge-size.test.ts. It is a note, not a failure, so I kept the two helpers separate.

To try it: call scorecard with action: "apply" or "decline" and a waiting change's id. /settings/crew shows the same message afterward.

🤖 Generated with Claude Code

commented

Two of the three failures came from scorecard becoming a write tool. I fixed both and pushed 9b30b1eaa. The third was a flake. Before pushing, the four affected test files passed and bun x tsc --noEmit and bun scripts/gate.ts --fix were clean, except for the test-similarity note below.

  • conversation-confirm.test.ts: every writes: true tool must be on the ask-first list or the NOT_ASKED list. I put scorecard on NOT_ASKED, next to the other settings an admin can put back. This puts scorecard in the same group as week and upstream, which also gained a write action behind a read-only call.
  • route-marks.test.ts: the admin sweep that calls each read-only admin tool with no arguments now also counts a tool that reads when given none, such as scorecard and forges. Without that, only 3 tools were left and the test's "more than 3" check failed.
  • sentinel-causes.test.ts ("the key is the run kind…"): it passes on its own here and doesn't touch anything this pull changed. I treated it as a flake and made no change.

The gate's similarity note between call in test/mcp-scorecard-decide.test.ts and mcp in test/xlarge-size.test.ts still shows. It is a note, not a failure, so I left it.

The commit message carries a No door: line, since only tests changed. I haven't run the full check; the forge does that on this push.

🤖 Generated with Claude Code

Changes requested — before the latest push

The change does what the issue asks and the code is tidy, but two things fail the list.

Fix first: the agent refusal is not tested on the path it was written for. In test/mcp-scorecard-decide.test.ts the agent (is_agent=1, not admin) is stopped by visible() in src/http/mcp/shared.ts with "no tool named scorecard…", before run is reached. So CHANGE_WORDS.person, the sentence the issue and docs promise, is never asserted. Make an agent that is an admin (set is_admin=1 on the bot), call apply, and assert the text contains CHANGE_WORDS.person. Also assert the sentence on the non-agent, non-admin path if it is reachable, or drop that dead branch.

Second, the issue says an agent is refused "with the page's sentence". The page has no agent sentence, so a new one was invented. Say in the description that this is the choice, or reuse an existing ADMIN_WORDS sentence.

Minor: change.length > 40 repeats the schema's maxLength, which is fine as a boundary check. The helper call resembles mcp in test/xlarge-size.test.ts; reuse the shared one if a helper exists.

commented

The agent refusal is now tested on the path it was written for, and I've pushed it as 14f0f96d6. The scorecard test file passes (5 tests) and bun x tsc --noEmit is clean. I did not re-run bun scripts/gate.ts or the other tests after this commit.

  • New test: the test sets is_admin=1 on the bot, calls apply on a waiting change, and asserts the error contains CHANGE_WORDS.person. It also asserts the change stays asked and the agent's model is untouched. The admin flag is put back afterwards.
  • Existing test, kept: the non-admin agent test now only covers the refusal from visible(), where the agent never reaches run. It still checks that nothing changes. The kind !== "user" branch can only be reached by an admin agent, so the new test is the one that covers it.
  • The sentence is a choice, not a reuse. The page has no sentence for an agent, so I wrote one: CHANGE_WORDS.person, "the model review's button is a person's who administers the forge; an agent cannot press it". I did not reuse an ADMIN_WORDS sentence because none of them says an agent can't press this button. Say if you'd rather have an existing one.
  • Test helper: I kept my own call in test/mcp-scorecard-decide.test.ts. I did not find a shared MCP call helper to reuse, so the similarity note from the gate stays.
  • maxLength: change.length > 40 stays as a boundary check that repeats the schema's maxLength.

🤖 Generated with Claude Code

commented

The three failures don't come from this change, and I changed no code to fix them. I pushed a merge of current main into the branch (39d117e9f) to get a fresh check run.

  • Failing tests: the crawler screen test (test/tui-crawlers.test.ts), the event-stream test in test/strangers.test.ts, and the blob-page switch test in test/media.test.ts.
  • Why I think they're load-sensitive: each failed in a different shard of the 83-shard check, and none touches the scorecard decision code or the crew-page message strings this pull edits. Run alone, all three files pass, three times in a row, on this branch before the merge. The check itself found test/strangers.test.ts green on main, but that doesn't show whether it passes on this branch alone.
  • If it fails again: the check's log will show whether the same three tests fail in the same shards.

After the merge, bun x tsc --noEmit is clean and these pass: test/tui-crawlers.test.ts, test/strangers.test.ts, test/mcp-scorecard-decide.test.ts, test/route-marks.test.ts. I did not re-run test/media.test.ts after the merge. I did not run the full suite, as instructed.

The gate printed two notes, both left alone. One is a comment line over 100 columns in test/route-marks.test.ts, which the nightly rewrap handles on main. The other is a duplication hint between call in test/mcp-scorecard-decide.test.ts and mcp in test/xlarge-size.test.ts, which I didn't fold together.

I had first rebased onto main, but the forge refuses a force push, so I went back to the pushed branch and merged instead.

Approved

Earlier asks are met: the agent refusal is now tested on the admin-agent path that reaches run, with the model and change left untouched, and the decision words live in one CHANGE_WORDS so the page and tool cannot drift. The tool checks who (a person who administers), caps change, validates action, and goes through the existing decideChange. The no-argument read is unchanged. The docs live in docs/features, not README. The page change only moves the words, so there is no visible change. The merge of main adds nothing to review. Non-blocking: the call test helper resembles mcp in xlarge-size.test.ts.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs