A weekly model review: the forge moves each role to the cheapest model that holds its quality, from its own numbers #7837

Once a week the forge now reviews each crew member's model from its own ledger. It proposes at most one change per agent, then reports the result a week later. tsc and gate.ts are clean. The 14 new tests pass, along with the limits, scorecard, mcp, schema-migration and cross-file tests I ran (283 of 284). The one failure was my own test's expectation, which I then fixed and re-ran green, but I did not re-run the whole batch after that. I did not run bun test on the full suite. fast.ts said the diff reaches 1347 files, too broad to name. I also didn't boot the app or open the page.

What changed

• The comparison: src/core/model_review.ts reads the last 28 days per agent and model. It compares cost per landed pull (every agent run on a pull counts, so retries cost), red-check rate, rework rounds, reverts and a hard-work share. A pull counts as hard when its run escalated by need (p0 or label:opus). The ladder is fable → opus → sonnet → haiku.
• The verdict:

- A model with reverts, a rework round per landed pull, or half its checks red asks for a step up. - A healthy model is proposed a step down, unless over half its work was hard or the cheaper model's own numbers are no better. - Under model_review.min_landed landed pulls (default 8) it says so on the page and changes nothing. - An agent already on the cheapest model, on an unlisted model, or on the runner's default is left alone. Atlas's planning lands nothing, so it stays unchanged.

• Applying:

- A step down applies itself only if the owner sets the limit model_review.auto to 1. It defaults to 0, so a person must opt in. - A step up always asks. - Asking is one button on Settings → Crew ("Move to …" or "Leave it").

• The trail: a new model_review table plus the events crew.model_asked, crew.model_changed, crew.model_declined and crew.model_result. The "before" and "after" numbers are stored on the change.
• The result: a week after a change, settleChanges compares the numbers. It undoes the change if reverts rose, the red rate rose by more than model_review.slack_pct points (default 10), or rework rose by more than half a round. It waits up to two weeks for enough pulls, then keeps the change and says too few landed. A model changed by hand in between is left alone.
• Where it shows: a "Model review" card on /settings/crew, and modelReview in GET /api/v1/crew/scorecard and the scorecard MCP tool.
• Weekly run: main.ts ticks the review hourly, and a stored marker makes it run at most once in seven days.
• Details: docs/features/model-review.md has the full rules. I added a migration and its pin, three limit keys and no dependencies.

Behaviours to know

• A healthy agent that was just stepped down and settled can be stepped down again in the same tick, as its own change. Sonnet to haiku is an example. The one-per-week rule counts from the change's creation, so one agent can move twice within eight days.
• A pending "asked" change blocks further proposals for that agent until someone decides it. It never expires.
• The step-down evidence is a heuristic. When the cheaper model has no numbers yet, the change is labelled a one-week trial, and the undo is the safeguard.

Left out

• An MCP tool for the button. The decision is on the page only. An agent can read the proposals through the scorecard tool but can't approve one.
• Splitting the verdict by issue type or pull size. Difficulty is only the p0/label:opus share.

To try it

• Open /settings/crew as an admin to see the card.
• Set model_review.auto on the limits page to let step downs apply themselves.

Closes #7815

Once a week the forge now reviews each crew member's model from its own ledger. It proposes at most one change per agent, then reports the result a week later. tsc and gate.ts are clean. The 14 new tests pass, along with the limits, scorecard, mcp, schema-migration and cross-file tests I ran (283 of 284). The one failure was my own test's expectation, which I then fixed and re-ran green, but I did not re-run the whole batch after that. I did not run bun test on the full suite. fast.ts said the diff reaches 1347 files, too broad to name. I also didn't boot the app or open the page.

What changed

• The comparison: src/core/model_review.ts reads the last 28 days per agent and model. It compares cost per landed pull (every agent run on a pull counts, so retries cost), red-check rate, rework rounds, reverts and a hard-work share. A pull counts as hard when its run escalated by need (p0 or label:opus). The ladder is fable → opus → sonnet → haiku.
• The verdict:

- A model with reverts, a rework round per landed pull, or half its checks red asks for a step up. - A healthy model is proposed a step down, unless over half its work was hard or the cheaper model's own numbers are no better. - Under model_review.min_landed landed pulls (default 8) it says so on the page and changes nothing. - An agent already on the cheapest model, on an unlisted model, or on the runner's default is left alone. Atlas's planning lands nothing, so it stays unchanged.

• Applying:

- A step down applies itself only if the owner sets the limit model_review.auto to 1. It defaults to 0, so a person must opt in. - A step up always asks. - Asking is one button on Settings → Crew ("Move to …" or "Leave it").

• The trail: a new model_review table plus the events crew.model_asked, crew.model_changed, crew.model_declined and crew.model_result. The "before" and "after" numbers are stored on the change.
• The result: a week after a change, settleChanges compares the numbers. It undoes the change if reverts rose, the red rate rose by more than model_review.slack_pct points (default 10), or rework rose by more than half a round. It waits up to two weeks for enough pulls, then keeps the change and says too few landed. A model changed by hand in between is left alone.
• Where it shows: a "Model review" card on /settings/crew, and modelReview in GET /api/v1/crew/scorecard and the scorecard MCP tool.
• Weekly run: main.ts ticks the review hourly, and a stored marker makes it run at most once in seven days.
• Details: docs/features/model-review.md has the full rules. I added a migration and its pin, three limit keys and no dependencies.

Behaviours to know

• A healthy agent that was just stepped down and settled can be stepped down again in the same tick, as its own change. Sonnet to haiku is an example. The one-per-week rule counts from the change's creation, so one agent can move twice within eight days.
• A pending "asked" change blocks further proposals for that agent until someone decides it. It never expires.
• The step-down evidence is a heuristic. When the cheaper model has no numbers yet, the change is labelled a one-week trial, and the undo is the safeguard.

Left out

• An MCP tool for the button. The decision is on the page only. An agent can read the proposals through the scorecard tool but can't approve one.
• Splitting the verdict by issue type or pull size. Difficulty is only the p0/label:opus share.

To try it

• Open /settings/crew as an admin to see the card.
• Set model_review.auto on the limits page to let step downs apply themselves.

I merged main into the branch and pushed it; the forge's check on the new push has not run yet. The earlier failure was two merge conflicts, in src/actions/scorecard.ts and src/web/pages/crew.ts. Main had added wastedDays in the same places as my modelReview lines. I kept both in each file, so the scorecard now returns modelDays, wastedDays and modelReview.

After the merge, tsc and scripts/gate.ts are clean. I ran the model-review test, the scorecard and MCP tests, the schema-migration test and the five cross files, and all 272 tests pass. I did not run the full suite.

The pull description from my earlier message is unchanged.

The check failed because the model review's events weren't classified in src/core/event_audience.ts. The commit that classifies them (d2b1e993, "Classify the model review's events for the owner") was already in my clone but not yet pushed, so the check ran without it.

I pushed it just now. Before pushing, test/event-audience-coverage.test.ts passed (5 tests) and bun x tsc --noEmit was clean. I did not run bun scripts/gate.ts or any other test files this turn. I haven't seen the forge's new check verdict yet.

The same log also shows shard 59 killed after 30 s while running test/tui-size.test.ts. That is outside this change, and the failure the check reported is the coverage one above.

I've addressed all six points from the review and pushed them to this branch. The new and neighbouring tests pass, bun x tsc --noEmit and scripts/design-lint.ts are clean, and the design tests pass. I did not run the full suite; scripts/fast.ts named 1359 of 1715 files, too broad, so I ran only the files that cover this change. scripts/gate.ts printed nothing, which I read as clean.

• Terminal door:

- Rows: "Crew scorecard per agent" (src/tui/screens/scorecard.ts) now shows a verdict row per agent and the trail below the figures. It is admin-only like the page. - Keys: on a waiting row, m moves the agent and n leaves it, each after a y to confirm. Both go through decideChange. - Docs and tests: the README and docs/features/model-review.md say how to use it. test/tui-model-review.test.ts covers move, leave, cancel, a row with nothing waiting, and a non-admin.

• Server tests:

- Admin through the server: a same-origin admin POST to take moves the model, and a POST to decline leaves it. - Refused: a cross-origin POST gets 403. An anonymous POST is not sent to /settings/crew. A non-admin's take and decline change nothing. - Lookup: the convoluted find() assertions now look the change up by id.

• Design:

- Card: the model review card is two tables (verdicts and trail), plus a ul.list of .rows with .row-actions forms for the asks. The long head paragraph moved into the body. - Lint: no new component class, and design-lint is clean.

• Lapsing asks: an asked change lapses to declined after two weeks, with an event marked expired and a result line. It no longer blocks the agent, and a test covers it.
• Atomicity:

- Transactions: the agent update, the review row and the event are written in one transaction in apply(), in decline, in expiry and in settleChanges (including the undo). A test with a failing event insert shows neither a changed model nor a half-written trail.

• Auto path: runModelReview now records the change and calls apply(), inside one transaction.
• One move a week: busy() also counts a change settled in the last week. A change that settles at day 8 no longer lets the same tick move the agent again. The "limit on" test now expects sonnet at day 8 and haiku only on the next weekly run.
• Not done: an MCP tool for the Move/Leave button is still out. The decision is on the page and in the terminal.

The full-suite proof of this head is waiting: no check slot is spare. It starts when one frees.

This push changes nothing. Sentinel approved the pull, and the only other message says the full-suite check is waiting for a free slot. The working tree is clean and the branch matches origin, so there is no change to make or push.

What the pull does

• A weekly model review moves each role to the cheapest model that holds its quality, using the forge's own numbers.
• The terminal scorecard screen shows the verdict and the trail rows. m and n with a y confirm go through decideChange, which only admins can use.
• The web card uses card, btn, ul.list and .row with POST forms, so it adds no new component class.
• The model, the trail row and the event are written in one transaction, with a test where the event write fails.
• An ask lapses to declined after two weeks, and that writes an event and has a test.
• Only one move happens a week, because busy() also counts a change that has just settled.
• Server tests cover the same-origin admin take and decline, a cross-origin POST (403), an anonymous POST and a non-admin POST.

What I left out

• There is no MCP tool for the Move/Leave decision. Sentinel marked this non-blocking.

What to look at

• The forge's full check on this head, once a slot frees. Nothing was run here this turn.