trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns12g rEventsFlows

A weekly model review: the forge moves each role to the cheapest model that holds its quality, from its own numbers #7837

Merged scoutcrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/7815-a-weekly-model-review-the-forge-moves-ea → main 4 commits fork 3 threads · 3 unresolved

  • Merged. scout/7815-a-weekly-model-review-the-forge-moves-ea was merged into main as c7df6ca.

Review

Sign in to review this change.
15 files changed +1256 −10
README.md +2
@@ -48,6 +48,8 @@ its cost over the last 5 hours, since 00:00 UTC and the last 7 days, as
4848 `/settings/spend` reads it; `—` means nothing was spent in that span (`spend —`,
4949 in all three).
5050 Runs on P0, `opus`-labelled, planner and twice-bounced work use Opus (`docs/features/model-escalation.md`); the "Per model per day" spend and landed table is on Settings → Crew, in the terminal's "Crew scorecard" list, and in `GET /api/v1/crew/scorecard`.
51
52 **Weekly model review** (`docs/features/model-review.md`): once a week the forge reads its own ledger and moves each crew member to the cheapest model that holds its quality, or asks. A forge admin reads the verdicts and the trail, and answers an ask, in Settings → Crew ("Model review": *Move to …* / *Leave it*), in the terminal (`:` → "Crew scorecard per agent": the rows below the figures; put the cursor on a `waiting` row, `m` moves the agent and `n` leaves it, each after a `y` to confirm), in `GET /api/v1/crew/scorecard` and the MCP `scorecard` tool (`modelReview`). An ask nobody answers lapses after two weeks.
5153 Settings → Crew also shows, each only when it has something, a "Handed to the
5254 planner" card (issues the crew parked twice: repo, issue link, when last
5355 parked, why) and a "Restore drill" card (the last drill's pass or fail and
docs/features/model-review.md added +12
@@ -0,0 +1,12 @@
1 # The weekly model review
2
3 Once a week (`runModelReview`, ticked hourly from `src/main.ts`, runs at most once in seven days) the forge reads its ledger and, per crew member, compares the models it ran on over the last 28 days: cost per landed pull (every agent run on a pull counts), red first-check rate, change-request rounds per landed pull, reverts, and how hard the work was (share of pulls that escalated by need: `p0`, `label:opus`). The figures are on Settings → Crew ("Model review"), in `GET /api/v1/crew/scorecard` and the `scorecard` MCP tool (`modelReview`).
4
5 - **Plain code** (`src/core/model_review.ts`): the ladder is fable → opus → sonnet → haiku. A model with reverts, a rework round per landed pull, or half its checks red is asked a step *up*. A healthy one is proposed a step *down*, unless over half its work was hard, or the cheaper model has its own numbers and they are no better.
6 - **At most one change per agent per week** (a change just settled counts), and none while one waits for a person or for its result. A change, its trail row and its event are written in one transaction.
7 - **Small samples change nothing**: under `model_review.min_landed` landed pulls (default 8) the page says so.
8 - **Applies itself or asks**: a step down applies on its own when the owner sets the limit `model_review.auto` to 1 (default 0); a step up always asks. Asking is one button on the page ("Move to …" / "Leave it") and, in the terminal's "Crew scorecard per agent", `m` / `n` on a `waiting` row, then `y` (both call `decideChange`, a forge admin's only). An ask nobody answers lapses after two weeks as declined, and frees the agent.
9 - **On the trail**: `crew.model_asked`, `crew.model_changed` (with the before numbers), `crew.model_declined`, `crew.model_result`.
10 - **A week after** a change (kept waiting up to two weeks for enough landed pulls) the result is reported; the change is undone when reverts rose, the red rate rose by more than `model_review.slack_pct` points (default 10), or rework rose by more than half a round. A model changed by hand meanwhile is left alone.
11
12 Not done: an MCP tool for the button (the decision is on the page and in the terminal), and the review does not look at an agent running on the runner's default model.
src/actions/model_review.ts added +194
@@ -0,0 +1,194 @@
1 /**
2 * The weekly model review, as actions (#7815): the tick the forge runs, the
3 * owner's one button, and the read for the scorecard's doors. Every change
4 * is an event on the outbox with its before numbers, and its result a week
5 * later; a cheaper model that did not hold is undone.
6 */
7 import type { App } from "../app.ts";
8 import { getAgent, listAgents, updateAgent as patchAgent } from "../core/agents.ts";
9 import {
10 busy,
11 fellShort,
12 getChange,
13 GIVE_UP_MS,
14 lastRun,
15 listChanges,
16 markRun,
17 modelStats,
18 recordChange,
19 review,
20 REVIEW_DAYS,
21 setState,
22 WEEK_MS,
23 type Limits,
24 type ModelChange,
25 type ModelStats,
26 type Verdict,
27 } from "../core/model_review.ts";
28 import type { User } from "../core/users.ts";
29 import { err, ok, type Result } from "../kernel/result.ts";
30 import { limitValue } from "../settings/limits.ts";
31
32 export function reviewLimits(app: App): Limits & { auto: boolean } {
33 return {
34 minLanded: limitValue(app.db, "model_review.min_landed"),
35 slackPct: limitValue(app.db, "model_review.slack_pct"),
36 auto: limitValue(app.db, "model_review.auto") === 1,
37 };
38 }
39
40 const pctOf = (n: number | null) => (n === null ? "no checks" : `${Math.round(100 * n)}% red`);
41 const usd = (n: number | null) => (n === null ? "—" : `$${n.toFixed(2)}`);
42 const figures = (s: ModelStats) =>
43 `${s.landed} landed at ${usd(s.costPerLanded)} each, ${pctOf(s.redRate)}, ${(s.reworkRounds ?? 0).toFixed(1)} rework rounds, ${s.reverts} reverted`;
44
45 const empty = (model: string): ModelStats => ({
46 model, opened: 0, landed: 0, costPerLanded: null, redRate: null, reworkRounds: null, reverts: 0, hardShare: null,
47 });
48
49 /** What the review says now for each member, without changing anything. */
50 export function readReview(app: App, now = new Date()) {
51 const limits = reviewLimits(app);
52 const stats = modelStats(app.db, new Date(now.getTime() - REVIEW_DAYS * 86_400_000));
53 return {
54 windowDays: REVIEW_DAYS,
55 limits,
56 agents: listAgents(app.db).map((agent) => {
57 const mine = stats.get(agent.user.id) ?? [];
58 const verdict: Verdict = review(agent.model, mine, limits);
59 return { username: agent.user.username, model: agent.model, models: mine, verdict };
60 }),
61 changes: listChanges(app.db),
62 };
63 }
64
65 /** The model, the review row and the event move together, or not at all. */
66 function apply(app: App, change: ModelChange, by: User | undefined, now: Date, how: string): void {
67 app.db.transaction(() => {
68 patchAgent(app.db, change.agentId, { model: change.toModel });
69 setState(app.db, change.id, { state: "applied", appliedAt: now, ...(by === undefined ? {} : { decidedBy: by.id }) });
70 app.bus.append({
71 kind: "crew.model_changed",
72 ...(by === undefined ? {} : { actorId: by.id }),
73 payload: { agent: nameOf(app, change.agentId), from: change.fromModel, to: change.toModel, how, why: change.why, before: change.before },
74 });
75 })();
76 }
77
78 const nameOf = (app: App, agentId: string) => getAgent(app.db, agentId)?.user.username ?? agentId;
79
80 /** The owner's one button on a change that asked: take it, or leave it. */
81 export function decideChange(
82 app: App,
83 actor: User | undefined,
84 changeId: string,
85 take: boolean,
86 now = new Date(),
87 ): Result<ModelChange, "not_found"> {
88 if (actor?.isAdmin !== true || actor.kind !== "user") return err("not_found");
89 const change = getChange(app.db, changeId);
90 if (change === undefined || change.state !== "asked") return err("not_found");
91 if (take) apply(app, change, actor, now, "owner");
92 else {
93 app.db.transaction(() => {
94 setState(app.db, change.id, { state: "declined", decidedBy: actor.id, settledAt: now });
95 app.bus.append({
96 kind: "crew.model_declined",
97 actorId: actor.id,
98 payload: { agent: nameOf(app, change.agentId), from: change.fromModel, to: change.toModel },
99 });
100 })();
101 }
102 return ok(getChange(app.db, changeId) ?? change);
103 }
104
105 /** A week on, say how a change went and undo it if quality fell. */
106 export function settleChanges(app: App, now = new Date()): void {
107 const limits = reviewLimits(app);
108 for (const change of listChanges(app.db, undefined, 500)) {
109 if (change.state === "asked") {
110 // One ignored ask must not block the agent for ever: after two weeks it lapses as declined.
111 if (now.getTime() - new Date(change.createdAt).getTime() < GIVE_UP_MS) continue;
112 app.db.transaction(() => {
113 setState(app.db, change.id, { state: "declined", settledAt: now, result: "nobody answered in two weeks; left as it was" });
114 app.bus.append({
115 kind: "crew.model_declined",
116 payload: { agent: nameOf(app, change.agentId), from: change.fromModel, to: change.toModel, expired: true },
117 });
118 })();
119 continue;
120 }
121 if (change.state !== "applied" || change.settledAt !== null || change.appliedAt === null) continue;
122 const since = new Date(change.appliedAt);
123 const age = now.getTime() - since.getTime();
124 if (age < WEEK_MS) continue;
125 const agent = getAgent(app.db, change.agentId);
126 if (agent === undefined) continue;
127 const after =
128 modelStats(app.db, since).get(change.agentId)?.find((s) => s.model.toLowerCase() === change.toModel.toLowerCase()) ??
129 empty(change.toModel);
130 const name = agent.user.username;
131 const finish = (state: "kept" | "undone", result: string) => {
132 app.db.transaction(() => {
133 if (state === "undone") patchAgent(app.db, change.agentId, { model: change.fromModel });
134 setState(app.db, change.id, { state, settledAt: now, after, result });
135 app.bus.append({
136 kind: "crew.model_result",
137 payload: { agent: name, from: change.fromModel, to: change.toModel, state, result, before: change.before, after },
138 });
139 })();
140 };
141 if (agent.model.toLowerCase() !== change.toModel.toLowerCase()) {
142 finish("kept", `the model was changed by hand since, so nothing was undone`);
143 } else if (after.landed < limits.minLanded) {
144 if (age >= GIVE_UP_MS) finish("kept", `only ${after.landed} landed on ${change.toModel} in two weeks, too few to judge; kept`);
145 } else if (fellShort(change.before, after, limits)) {
146 finish("undone", `quality fell on ${change.toModel} (${figures(after)}; before ${figures(change.before)}); back to ${change.fromModel}`);
147 } else {
148 finish("kept", `held on ${change.toModel} (${figures(after)}; before ${figures(change.before)})`);
149 }
150 }
151 }
152
153 /**
154 * The tick: settle what is a week old, then, once a week, review each
155 * member. Safe to call as often as you like. Returns the changes it made or
156 * asked for.
157 */
158 export function runModelReview(app: App, now = new Date()): ModelChange[] {
159 settleChanges(app, now);
160 const ran = lastRun(app.db);
161 if (ran !== undefined && now.getTime() - ran.getTime() < WEEK_MS) return [];
162 markRun(app.db, now);
163 const limits = reviewLimits(app);
sentinelcrew outdated

Duplicates apply(); call it. Wrap record+patch+event in one transaction.
164 const made: ModelChange[] = [];
165 const stats = modelStats(app.db, new Date(now.getTime() - REVIEW_DAYS * 86_400_000));
166 for (const agent of listAgents(app.db)) {
167 if (busy(app.db, agent.user.id, now)) continue;
168 const mine = stats.get(agent.user.id) ?? [];
169 const verdict = review(agent.model, mine, limits);
170 if (verdict.kind !== "propose") continue;
171 const before = mine.find((s) => s.model.toLowerCase() === agent.model.toLowerCase());
172 if (before === undefined) continue;
173 const auto = limits.auto && verdict.direction === "down";
174 const change = app.db.transaction((): ModelChange | undefined => {
175 const changeId = recordChange(app.db, {
176 agentId: agent.user.id, fromModel: agent.model, toModel: verdict.to,
177 state: "asked", why: verdict.why, before, at: now,
178 });
179 const made = getChange(app.db, changeId);
180 if (made === undefined) return undefined;
181 if (auto) apply(app, made, undefined, now, "review");
182 else {
183 app.bus.append({
184 kind: "crew.model_asked",
185 payload: { agent: agent.user.username, from: made.fromModel, to: made.toModel, why: made.why, before },
186 });
187 }
188 return getChange(app.db, changeId);
189 })();
190 if (change === undefined) continue;
191 made.push(change);
192 }
193 return made;
194 }
src/actions/scorecard.ts +8 −1
@@ -12,6 +12,7 @@ import {
1212 type SplitScorecard,
1313 } from "../core/scorecard.ts";
1414 import { modelDays, type ModelDay } from "../core/model_days.ts";
15 import { readReview } from "./model_review.ts";
1516 import { wastedDays, type WastedDay } from "../core/wasted_days.ts";
1617 import { listAgents } from "../core/agents.ts";
1718 import type { User } from "../core/users.ts";
@@ -42,7 +43,12 @@ const figures = (c: Scorecard): Scorecard => ({
4243 export function readScorecards(
4344 app: App,
4445 actor: User | undefined,
45 ): { agents: AgentScorecard[]; modelDays: ModelDay[]; wastedDays: WastedDay[] } | undefined {
46 ): {
47 agents: AgentScorecard[];
48 modelDays: ModelDay[];
49 wastedDays: WastedDay[];
50 modelReview: ReturnType<typeof readReview>;
51 } | undefined {
4652 if (actor?.isAdmin !== true) return undefined;
4753 const window = recentWindow();
4854 const all = crewScorecards(app.db);
@@ -52,6 +58,7 @@ export function readScorecards(
5258 const key = (p: SplitScorecard) => `${p.model}\u0000${p.promptVersion}`;
5359 return {
5460 modelDays: modelDays(app.db),
61 modelReview: readReview(app),
5562 wastedDays: wastedDays(app.db),
5663 agents: listAgents(app.db).map((agent) => {
5764 const id = agent.user.id;
src/core/event_audience.ts +2
@@ -157,6 +157,8 @@ export function classifiedAudience(kind: string): EventAudience | undefined {
157157 kind.startsWith("drill.") ||
158158 // A snapshot moving is the forge owner's too, and has no repository.
159159 kind.startsWith("backup.") ||
160 // The weekly model review's changes and results are the owner's (`actions/model_review.ts`).
161 kind.startsWith("crew.model_") ||
160162 // A clog is the whole forge's flow, the owner's to hear (`flow/clog_tick.ts`).
161163 kind.startsWith("clog.") ||
162164 // The forge letting go of FLOW ONLY after 24 h clear is the admins' to hear (`flow/flow_only_auto.ts`).
src/core/migration-pins/20261101140000-model-review.pin added +1
@@ -0,0 +1,1 @@
1 56d5627ddfa4c29c3a4572410500c456baeb9bf12c5e62d80e5bf05a87825ca3
src/core/migrations/20261101140000-model-review.ts added +22
@@ -0,0 +1,22 @@
1 // The weekly model review (#7815): each change of a crew member's model, with
2 // the numbers it was made on and, a week later, the numbers after it; and the
3 // time the review last ran, so it runs once a week.
4 export default `
5 CREATE TABLE model_review (
6 id TEXT PRIMARY KEY,
7 agent_id TEXT NOT NULL REFERENCES user(id) ON DELETE CASCADE,
8 from_model TEXT NOT NULL,
9 to_model TEXT NOT NULL,
10 state TEXT NOT NULL CHECK (state IN ('asked','applied','declined','kept','undone')),
11 why TEXT NOT NULL,
12 before_json TEXT NOT NULL,
13 after_json TEXT,
14 result TEXT NOT NULL DEFAULT '',
15 created_at TEXT NOT NULL,
16 decided_by TEXT,
17 applied_at TEXT,
18 settled_at TEXT
19 );
20 CREATE INDEX model_review_agent ON model_review (agent_id, created_at);
21 CREATE TABLE model_review_run (at TEXT NOT NULL);
22 `;
src/core/model_review.ts added +320
@@ -0,0 +1,320 @@
1 /**
2 * The weekly model review (#7815): from the ledger alone, how each crew
3 * member fares on each model — cost per landed pull, red first checks, rework
4 * rounds, reverts, and how hard the work was — and, from that, at most one
5 * proposed change of model per member per week. Plain code, no model.
6 *
7 * A pull belongs to the model of its author's latest agent run on it (as the
8 * scorecard's splits do). Its cost is every agent run on it, so a pull that
9 * took three tries costs three. A pull is hard when a run on it escalated by
10 * need (`p0`, `label:opus`), so a model is not judged on work it was not given.
11 *
12 * Pure data and SQL: no authz, no events (`actions/model_review.ts` has them).
13 */
14 import type { Database } from "bun:sqlite";
15 import { id as newId } from "../kernel/id.ts";
16
17 /** Dearest first. A step down is one place to the right. */
18 export const LADDER: readonly string[] = ["claude-fable-5-1", "opus", "sonnet", "haiku"];
19 /** The window a review reads, and the days between reviews and before a result. */
20 export const REVIEW_DAYS = 28;
21 export const WEEK_MS = 7 * 86_400_000;
22 /** A result still too small to judge after this long is kept as it is. */
23 export const GIVE_UP_MS = 14 * 86_400_000;
24 /** A model is unhealthy at any revert, a rework round per landed pull, or half its checks red. */
25 const BAD_REWORK = 1;
26 const BAD_RED = 0.5;
27 /** More than this share of hard work and a cheaper model is not tried. */
28 const HARD_SHARE_MAX = 0.5;
29
30 export interface ModelStats {
31 readonly model: string;
32 readonly opened: number;
33 readonly landed: number;
34 /** Agent spend on the opened pulls ÷ landed; null before one lands. */
35 readonly costPerLanded: number | null;
36 /** Failed ÷ finished check runs; null with none. */
37 readonly redRate: number | null;
38 /** Changes-requested reviews per landed pull; null before one lands. */
39 readonly reworkRounds: number | null;
40 readonly reverts: number;
41 /** Share of opened pulls that were hard; null with none opened. */
42 readonly hardShare: number | null;
43 }
44
45 export interface Limits {
46 /** Landed pulls a model needs in the window before it is judged. */
47 readonly minLanded: number;
48 /** Percentage points of red rate a cheaper model may be worse by. */
49 readonly slackPct: number;
50 }
51
52 interface PullRow {
53 author_id: string;
54 model: string;
55 landed: number;
56 rounds: number;
57 ok: number;
58 failed: number;
59 reverted: number;
60 cost: number;
61 hard: number;
62 }
63
64 /** Each author's stats per model over pulls opened in `[since, until)`. */
65 export function modelStats(
66 db: Database,
67 since: Date,
68 until: Date = new Date("9999-12-31T00:00:00Z"),
69 ): Map<string, ModelStats[]> {
70 const rows = db
71 .query<PullRow, [string, string]>(
72 `SELECT issue.author_id AS author_id,
73 COALESCE((SELECT run.model FROM run WHERE run.issue_id = issue.id
74 AND run.kind = 'agent' AND run.actor_id = issue.author_id
75 ORDER BY run.id DESC LIMIT 1), '') AS model,
76 pull.merged_at IS NOT NULL AS landed,
77 (SELECT COUNT(*) FROM review WHERE review.issue_id = issue.id
78 AND review.state = 'changes_requested') AS rounds,
79 (SELECT COUNT(*) FROM run WHERE run.issue_id = issue.id
80 AND run.kind = 'check' AND run.state = 'ok') AS ok,
81 (SELECT COUNT(*) FROM run WHERE run.issue_id = issue.id
82 AND run.kind = 'check' AND run.state = 'failed') AS failed,
83 (pull.merged_at IS NOT NULL AND EXISTS (SELECT 1 FROM pull AS back
84 WHERE back.reverts_issue_id = issue.id AND back.merged_at IS NOT NULL)) AS reverted,
85 COALESCE((SELECT SUM(run.cost_usd) FROM run WHERE run.issue_id = issue.id
86 AND run.kind = 'agent'), 0) AS cost,
87 EXISTS (SELECT 1 FROM run WHERE run.issue_id = issue.id
88 AND run.kind = 'agent' AND run.model_reason IN ('p0', 'label:opus')) AS hard
89 FROM issue
90 JOIN pull ON pull.issue_id = issue.id
91 JOIN user ON user.id = issue.author_id AND user.is_agent = 1
92 WHERE issue.created_at >= ?1 AND issue.created_at < ?2`,
93 )
94 .all(since.toISOString(), until.toISOString());
95 const groups = new Map<string, PullRow[]>();
96 for (const r of rows) {
97 const key = `${r.author_id}\0${r.model}`;
98 const list = groups.get(key);
99 if (list === undefined) groups.set(key, [r]);
100 else list.push(r);
101 }
102 const out = new Map<string, ModelStats[]>();
103 for (const list of groups.values()) {
104 const first = list[0];
105 if (first === undefined) continue;
106 const sum = (f: (r: PullRow) => number) => list.reduce((n, r) => n + f(r), 0);
107 const landed = sum((r) => r.landed);
108 const checks = sum((r) => r.ok) + sum((r) => r.failed);
109 const stats: ModelStats = {
110 model: first.model,
111 opened: list.length,
112 landed,
113 costPerLanded: landed === 0 ? null : sum((r) => r.cost) / landed,
114 redRate: checks === 0 ? null : sum((r) => r.failed) / checks,
115 reworkRounds:
116 landed === 0 ? null : sum((r) => (r.landed === 1 ? r.rounds : 0)) / landed,
117 reverts: sum((r) => r.reverted),
118 hardShare: sum((r) => r.hard) / list.length,
119 };
120 const mine = out.get(first.author_id) ?? [];
121 mine.push(stats);
122 out.set(first.author_id, mine);
123 }
124 for (const list of out.values()) list.sort((a, b) => b.opened - a.opened);
125 return out;
126 }
127
128 export type Verdict =
129 | { readonly kind: "hold"; readonly note: string }
130 | {
131 readonly kind: "propose";
132 readonly to: string;
133 readonly direction: "down" | "up";
134 readonly why: string;
135 };
136
137 const pct = (n: number | null) => (n === null ? "no checks" : `${Math.round(100 * n)}% red`);
138 const usd = (n: number | null) => (n === null ? "—" : `$${n.toFixed(2)}`);
139
140 /** Whether a model's own numbers are poor enough that a dearer one is worth asking for. */
141 export function unhealthy(s: ModelStats): boolean {
142 return s.reverts > 0 || (s.reworkRounds ?? 0) >= BAD_REWORK || (s.redRate ?? 0) >= BAD_RED;
143 }
144
145 /** Pure: one member's model and numbers in, at most one change out; a small sample changes nothing. */
146 export function review(current: string, stats: readonly ModelStats[], limits: Limits): Verdict {
147 const at = LADDER.indexOf(current.toLowerCase());
148 if (at < 0) return { kind: "hold", note: current === "" ? "runs on the runner's default; name a model to review it" : `${current} is not on the review ladder` };
149 const mine = stats.find((s) => s.model.toLowerCase() === current.toLowerCase());
150 if (mine === undefined || mine.landed < limits.minLanded)
151 return {
152 kind: "hold",
153 note: `${mine?.landed ?? 0} landed on ${current} in ${REVIEW_DAYS} days; ${limits.minLanded} are needed to judge, so nothing changes`,
154 };
155 const line = `${mine.landed} landed at ${usd(mine.costPerLanded)} each, ${pct(mine.redRate)}, ${(mine.reworkRounds ?? 0).toFixed(1)} rework rounds, ${mine.reverts} reverted`;
156 if (unhealthy(mine)) {
157 const to = LADDER[at - 1];
158 if (to === undefined) return { kind: "hold", note: `${line}: poor, and already on the top model` };
159 return { kind: "propose", to, direction: "up", why: `${current} is not holding its quality (${line})` };
160 }
161 const to = LADDER[at + 1];
162 if (to === undefined) return { kind: "hold", note: `${line}: already on the cheapest model` };
163 if ((mine.hardShare ?? 0) > HARD_SHARE_MAX)
164 return { kind: "hold", note: `${line}: over half of it was hard work, so a cheaper model is not tried` };
165 const there = stats.find((s) => s.model.toLowerCase() === to);
166 if (there !== undefined && there.landed >= limits.minLanded) {
167 const worse =
168 there.reverts > mine.reverts ||
169 (there.redRate ?? 0) > (mine.redRate ?? 0) + limits.slackPct / 100 ||
170 (there.reworkRounds ?? 0) > (mine.reworkRounds ?? 0) + 0.5;
171 const dearer = (there.costPerLanded ?? 0) >= (mine.costPerLanded ?? 0);
172 if (worse || dearer)
173 return { kind: "hold", note: `${line}; on ${to} it did ${there.landed} at ${usd(there.costPerLanded)} each, ${pct(there.redRate)}: no better, so it stays` };
174 return { kind: "propose", to, direction: "down", why: `${current} holds (${line}) and ${to} did as well on ${there.landed} landed at ${usd(there.costPerLanded)} each` };
175 }
176 return { kind: "propose", to, direction: "down", why: `${current} holds (${line}); ${to} has no numbers yet, so this is a one-week trial` };
177 }
178
179 /** Whether `after` fell short of `before` by more than the slack allows. */
180 export function fellShort(before: ModelStats, after: ModelStats, limits: Limits): boolean {
181 return (
182 after.reverts > before.reverts ||
183 (after.redRate ?? 0) > (before.redRate ?? 0) + limits.slackPct / 100 ||
184 (after.reworkRounds ?? 0) > (before.reworkRounds ?? 0) + 0.5
185 );
186 }
187
188 // ---- the trail ----
189
190 export type ReviewState = "asked" | "applied" | "declined" | "kept" | "undone";
191
192 export interface ModelChange {
193 readonly id: string;
194 readonly agentId: string;
195 readonly fromModel: string;
196 readonly toModel: string;
197 readonly state: ReviewState;
198 readonly why: string;
199 readonly before: ModelStats;
200 readonly after: ModelStats | null;
201 readonly result: string;
202 readonly createdAt: string;
203 readonly decidedBy: string | null;
204 readonly appliedAt: string | null;
205 readonly settledAt: string | null;
206 }
207
208 interface ChangeRow {
209 id: string;
210 agent_id: string;
211 from_model: string;
212 to_model: string;
213 state: ReviewState;
214 why: string;
215 before_json: string;
216 after_json: string | null;
217 result: string;
218 created_at: string;
219 decided_by: string | null;
220 applied_at: string | null;
221 settled_at: string | null;
222 }
223
224 const toChange = (r: ChangeRow): ModelChange => ({
225 id: r.id,
226 agentId: r.agent_id,
227 fromModel: r.from_model,
228 toModel: r.to_model,
229 state: r.state,
230 why: r.why,
231 before: JSON.parse(r.before_json) as ModelStats,
232 after: r.after_json === null ? null : (JSON.parse(r.after_json) as ModelStats),
233 result: r.result,
234 createdAt: r.created_at,
235 decidedBy: r.decided_by,
236 appliedAt: r.applied_at,
237 settledAt: r.settled_at,
238 });
239
240 /** Newest first; one agent's, or everyone's. */
241 export function listChanges(db: Database, agentId?: string, limit = 50): ModelChange[] {
242 return db
243 .query<ChangeRow, [string | null, number]>(
244 `SELECT * FROM model_review WHERE ?1 IS NULL OR agent_id = ?1
245 ORDER BY created_at DESC, id DESC LIMIT ?2`,
246 )
247 .all(agentId ?? null, limit)
248 .map(toChange);
249 }
250
251 export function getChange(db: Database, changeId: string): ModelChange | undefined {
252 const row = db
253 .query<ChangeRow, [string]>("SELECT * FROM model_review WHERE id = ?1")
254 .get(changeId);
255 return row === null ? undefined : toChange(row);
256 }
257
258 export function recordChange(
259 db: Database,
260 c: {
261 agentId: string;
262 fromModel: string;
263 toModel: string;
264 state: "asked" | "applied";
265 why: string;
266 before: ModelStats;
267 at: Date;
268 },
269 ): string {
270 const id = newId();
271 const at = c.at.toISOString();
272 db.query(
273 `INSERT INTO model_review (id, agent_id, from_model, to_model, state, why, before_json, created_at, applied_at)
274 VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)`,
275 ).run(id, c.agentId, c.fromModel, c.toModel, c.state, c.why, JSON.stringify(c.before), at, c.state === "applied" ? at : null);
276 return id;
277 }
278
279 export function setState(
280 db: Database,
281 changeId: string,
282 patch: { state: ReviewState; decidedBy?: string; appliedAt?: Date; settledAt?: Date; after?: ModelStats; result?: string },
283 ): void {
284 db.query(
285 `UPDATE model_review SET state = ?2, decided_by = COALESCE(?3, decided_by),
286 applied_at = COALESCE(?4, applied_at), settled_at = COALESCE(?5, settled_at),
287 after_json = COALESCE(?6, after_json), result = COALESCE(?7, result)
288 WHERE id = ?1`,
289 ).run(
290 changeId,
291 patch.state,
292 patch.decidedBy ?? null,
293 patch.appliedAt?.toISOString() ?? null,
294 patch.settledAt?.toISOString() ?? null,
295 patch.after === undefined ? null : JSON.stringify(patch.after),
296 patch.result ?? null,
297 );
298 }
299
300 /** Whether a member has a change in the last week (made or just settled), or one waiting on a person or on its result. */
301 export function busy(db: Database, agentId: string, now: Date): boolean {
302 const since = new Date(now.getTime() - WEEK_MS).toISOString();
303 return (
304 db
305 .query(
306 `SELECT 1 FROM model_review WHERE agent_id = ?1 AND state != 'declined'
307 AND (created_at >= ?2 OR settled_at >= ?2 OR state = 'asked' OR (state = 'applied' AND settled_at IS NULL))`,
308 )
309 .get(agentId, since) !== null
310 );
311 }
312
313 export function lastRun(db: Database): Date | undefined {
314 const row = db.query<{ at: string | null }, []>("SELECT MAX(at) AS at FROM model_review_run").get();
315 return row?.at == null ? undefined : new Date(row.at);
316 }
317
318 export function markRun(db: Database, at: Date): void {
319 db.query("INSERT INTO model_review_run (at) VALUES (?1)").run(at.toISOString());
320 }
src/http/server.ts +4
@@ -342,6 +342,8 @@ import {
342342 import {
343343 agentCreate,
344344 agentDelete,
345 modelReviewDecline,
346 modelReviewTake,
345347 agentPage,
346348 agentUpdate,
347349 crewPage,
@@ -979,6 +981,8 @@ export function buildRouter(): Router<Handler> {
979981 router.add("GET", "/settings/crew/:username", agentPage);
980982 router.add("POST", "/settings/crew/:username", agentUpdate);
981983 router.add("POST", "/settings/crew/:username/delete", agentDelete);
984 router.add("POST", "/settings/crew/model-review/:id/take", modelReviewTake);
985 router.add("POST", "/settings/crew/model-review/:id/decline", modelReviewDecline);
982986 // Where an OAuth authorization comes back to. Deliberately a plain route
983987 // and not an API one: it is a browser redirect, possibly from a phone, and
984988 // the only thing it carries is a `state` this forge issued minutes ago.
src/main.ts +10
@@ -4,6 +4,7 @@
44 * environment-only — a handful of variables, all with sane defaults.
55 */
66
7 import { runModelReview } from "./actions/model_review.ts";
78 import { BOOT_ACTOR, setSignup, signupFromEnv } from "./settings/signup.ts";
89 import { startMemorySampler } from "./boards/memory.ts";
910 import { startSampler } from "./boards/sampler.ts";
@@ -599,6 +600,14 @@ const holdTimer = setInterval(() => {
599600 app.log.warn("hold expiry failed", { error: String(error) });
600601 }
601602 }, 60_000);
603 // The weekly model review (#7815): settles last week's changes, then proposes this week's; it runs once a week whatever the tick.
604 const modelReviewTimer = setInterval(() => {
605 try {
606 runModelReview(app);
607 } catch (error) {
608 app.log.warn("model review failed", { error: String(error) });
609 }
610 }, 3_600_000);
602611 const processTimer = setInterval(processTick, 60_000);
603612 processTick();
604613 // A new version proves itself against the public URL; a failed one again, 5 min after and backing off to hourly (release/prove.ts).
@@ -722,6 +731,7 @@ function drain(signal: NodeJS.Signals): Promise<unknown> {
722731 stopClogTick();
723732 clearInterval(processTimer);
724733 clearInterval(holdTimer);
734 clearInterval(modelReviewTimer);
725735 clearInterval(leaseTimer);
726736 // The platform's SIGTERM: the listener stays open while routers learn
727737 // this process is going. A person's Ctrl-C does not wait for routers.
src/settings/limits.ts +28 −1
@@ -36,7 +36,10 @@ export type LimitKey =
3636 | "signin.per_forge"
3737 | "signup.per_source"
3838 | "signup.per_forge"
39 | "api.anonymous_per_source";
39 | "api.anonymous_per_source"
40 | "model_review.auto"
41 | "model_review.min_landed"
42 | "model_review.slack_pct";
4043
4144 export interface LimitDef {
4245 readonly label: string;
@@ -168,6 +171,30 @@ export const LIMITS: Record<LimitKey, LimitDef> = {
168171 max: 100_000,
169172 reason: "A token's reads are counted by the token; a stranger's have no token to count, so one source reading without one has a ceiling of its own.",
170173 },
174 "model_review.auto": {
175 label: "Weekly model review applies a cheaper model on its own",
176 unit: "0 asks first, 1 applies",
177 default: 0,
178 min: 0,
179 max: 1,
180 reason: "A step to a cheaper model is undone if quality falls, but only the owner says whether it may happen unasked; a step up always asks.",
181 },
182 "model_review.min_landed": {
183 label: "Landed pulls a model needs before the weekly review judges it",
184 unit: "pulls",
185 default: 8,
186 min: 3,
187 max: 1_000,
188 reason: "A handful of pulls is luck, not a rate, so a smaller sample changes nothing.",
189 },
190 "model_review.slack_pct": {
191 label: "Red-check rate a cheaper model may be worse by",
192 unit: "percentage points",
193 default: 10,
194 min: 0,
195 max: 100,
196 reason: "Past this the weekly review undoes its own change, so quality is the bar the saving is held to.",
197 },
171198 };
172199
173200 export const LIMIT_KEYS = Object.keys(LIMITS) as LimitKey[];
src/tui/screens/scorecard.ts +83 −8
@@ -8,6 +8,7 @@
88 */
99
1010 import { listAllAgents } from "../../actions/crew.ts";
11 import { decideChange, readReview } from "../../actions/model_review.ts";
1112 import { readSpend, type SpanName, type Spend } from "../../actions/spend.ts";
1213 import {
1314 crewScorecards,
@@ -28,7 +29,7 @@ import { style } from "../term/style.ts";
2829 import { truncate } from "../term/width.ts";
2930 import type { Session } from "../session.ts";
3031 import { fill, len, split } from "../ui/layout.ts";
31 import { ListState, drawList, type ListRow } from "./common.ts";
32 import { ListState, confirmScreen, drawList, type ListRow } from "./common.ts";
3233 import type { PaletteEntry } from "./entries.ts";
3334
3435 const landed = (s: Scorecard | undefined) =>
@@ -135,12 +136,49 @@ export function scorecardRows(
135136 return [...agentRows, ...teamRows];
136137 }
137138
139 type Review = ReturnType<typeof readReview>;
140
141 /**
142 * The weekly model review (#7815), as the page's card has it: a verdict row
143 * per agent, then every change on the trail, newest first. `changeAt[i]` is
144 * the id of the change still waiting on a person at row `i` (or undefined).
145 */
146 export function reviewRows(
147 r: Review,
148 names: ReadonlyMap<string, string>,
149 ): { rows: ListRow[]; changeAt: (string | undefined)[] } {
150 const rows: ListRow[] = [];
151 const changeAt: (string | undefined)[] = [];
152 const add = (row: ListRow, id?: string) => {
153 rows.push(row);
154 changeAt.push(id);
155 };
156 for (const a of r.agents) {
157 add({
158 title: `model review · ${a.username} on ${a.model === "" ? "default" : a.model}`,
159 secondary: a.verdict.kind === "hold" ? a.verdict.note : `would move to ${a.verdict.to}. ${a.verdict.why}`,
160 });
161 }
162 for (const c of r.changes) {
163 const who = names.get(c.agentId) ?? "an agent";
164 add(
165 {
166 title: `${c.state === "asked" ? "waiting" : "trail"} · ${who}: ${c.fromModel} → ${c.toModel} (${c.state})`,
167 secondary: c.why,
168 meta: c.state === "asked" ? "m move · n leave it" : c.result,
169 },
170 c.state === "asked" ? c.id : undefined,
171 );
172 }
173 return { rows, changeAt };
174 }
175
138176 export function scorecardScreen(f: Session, ctx: Ctx): Screen {
139177 const state = new ListState();
140178 const allowed = f.user?.isAdmin === true;
141179 // Read once when the screen opens, as the page reads it once per request.
142180 const listed = allowed ? listAllAgents(f.app, f.user) : undefined;
143 const rows =
181 const cards =
144182 listed?.ok === true
145183 ? scorecardRows(
146184 listed.value,
@@ -154,23 +192,60 @@ export function scorecardScreen(f: Session, ctx: Ctx): Screen {
154192 agentSpend(f),
155193 )
156194 : [];
157 return {
195 const names = new Map((listed?.ok === true ? listed.value : []).map((a) => [a.user.id, a.user.username]));
196 // Re-read after a decision, so the button's result shows at once.
197 let review = allowed && cards.length > 0 ? reviewRows(readReview(f.app), names) : { rows: [], changeAt: [] };
198 const rows = (): ListRow[] => [...cards, ...review.rows];
199 const waiting = (): string | undefined => review.changeAt[state.index - cards.length];
200 const decide = (take: boolean): void => {
201 const id = waiting();
202 if (id === undefined || f.user === undefined) return;
203 const change = readReview(f.app).changes.find((c) => c.id === id);
204 if (change === undefined) return;
205 const who = names.get(change.agentId) ?? "an agent";
206 ctx.push(
207 confirmScreen(ctx, self, {
208 title: take ? `Move ${who} to ${change.toModel}` : `Leave ${who} on ${change.fromModel}`,
209 body: [change.why],
210 actions: [{ label: take ? "Move" : "Leave it" }, { label: "Cancel", variant: "ghost" }],
211 yesNo: true,
212 onChoose: (choice) => {
213 if (choice !== 0) return;
214 const done = decideChange(f.app, f.user, id, take);
215 ctx.toast(
216 done.ok ? (take ? `${who} is on ${change.toModel}.` : "Left as it was.") : "That change is no longer waiting.",
217 done.ok ? "success" : "danger",
218 );
219 review = reviewRows(readReview(f.app), names);
220 ctx.invalidate();
221 },
222 }),
223 );
224 };
225 const self: Screen = {
158226 title: "Crew scorecard per agent",
159 keys: [binding(["j", "k"], "move cursor")],
227 keys: [binding(["j", "k"], "move cursor"), binding(["m"], "move a waiting agent to the proposed model"), binding(["n"], "leave it")],
160228 render(area, s) {
161229 const c = ctx.theme.c;
162230 const [head, list] = split(area, "v", [len(1), fill(1)]);
163231 const text = !allowed
164232 ? "The crew's scorecard is read by a forge admin."
165 : rows.length === 0
233 : cards.length === 0
166234 ? "No agents yet."
167235 : `Over the last ${RECENT_DAYS} days (${RECENT_DAYS}d) and all time, ` +
168 "as Settings → Crew shows it, with a pooled row per team.";
236 "as Settings → Crew shows it, with a pooled row per team, then the weekly model review.";
169237 s.text(head!.x, head!.y, truncate(text, head!.w), style(c.mutedFg, c.bg), head!.x + head!.w);
170 drawList(s, list!, c, state, rows);
238 drawList(s, list!, c, state, rows());
239 },
240 key(e) {
241 if (e.key === "m" || e.key === "n") {
242 decide(e.key === "m");
243 return true;
244 }
245 return state.key(e, rows().length);
171246 },
172 key: (e) => state.key(e, rows.length),
173247 };
248 return self;
174249 }
175250
176251 /** Settings → Crew's scorecard per agent, as the page has it: a forge admin (#2677). */
src/web/pages/crew.ts +119
@@ -35,6 +35,7 @@ import {
3535 type Scorecard,
3636 type SplitScorecard,
3737 } from "../../core/scorecard.ts";
38 import { decideChange, readReview } from "../../actions/model_review.ts";
3839 import { wastedDays, type WastedDay } from "../../core/wasted_days.ts";
3940 import { modelDays, type ModelDay } from "../../core/model_days.ts";
4041 import { notePhase, takeSessionTime } from "../../core/route_timing.ts";
@@ -455,6 +456,109 @@ function loadSpending(ctx: Ctx) {
455456 }));
456457 }
457458
459 /** The weekly model review (#7815): each member's models side by side, the verdict, the button, the trail. */
460 function modelReviewCard(ctx: Ctx): Html {
461 const r = readReview(ctx.app);
462 const names = new Map(listAllAgentsByName(ctx));
463 const cell = (n: number | null, f: (n: number) => string) =>
464 n === null ? html`<td class="num muted">—</td>` : html`<td class="num">${f(n)}</td>`;
465 const asked = r.changes.filter((c) => c.state === "asked");
466 return card(
467 html`<p class="muted small">At most one change of model per agent a week, to the cheapest that holds its quality.
468 A step up always asks; a step down ${r.limits.auto ? "applies on its own" : "asks too"} (Settings → Limits).
469 Under ${r.limits.minLanded} landed changes nothing. A week after a change the result is reported, and the
470 change undone if quality fell. An ask nobody answers lapses after two weeks.</p>
471 <div class="table-scroll">
472 <table class="table" id="model-review">
473 <thead>
474 <tr><th>Agent</th><th>Model</th><th>Landed</th><th>Per landed</th><th>Red</th><th>Rework</th><th>Reverts</th><th>Hard</th></tr>
475 </thead>
476 <tbody>
477 ${r.agents.flatMap((a) =>
478 a.models.map(
479 (m) => html`<tr>
480 <td>${a.username}</td>
481 <td class="mono">${m.model === "" ? "unrecorded" : m.model}</td>
482 <td class="num">${m.landed} <span class="muted small">of ${m.opened}</span></td>
483 ${cell(m.costPerLanded, (n) => `$${n.toFixed(2)}`)}
484 ${cell(m.redRate, (n) => `${Math.round(100 * n)}%`)}
485 ${cell(m.reworkRounds, (n) => n.toFixed(1))}
486 <td class="num">${m.reverts}</td>
487 ${cell(m.hardShare, (n) => `${Math.round(100 * n)}%`)}
488 </tr>`,
489 ),
490 )}
491 </tbody>
492 </table>
493 </div>
494 <div class="table-scroll">
495 <table class="table" id="model-review-verdicts">
sentinelcrew outdated

Hand-rolled bar form; use the named components from docs/DESIGN.md and run design-lint. Terminal parity is also missing.
496 <thead><tr><th>Agent</th><th>Model</th><th>Verdict</th></tr></thead>
497 <tbody>
498 ${r.agents.map(
499 (a) => html`<tr data-agent="${a.username}">
500 <td>${a.username}</td>
501 <td class="mono">${a.model === "" ? "default" : a.model}</td>
502 <td>${a.verdict.kind === "hold"
503 ? a.verdict.note
504 : `Would move to ${a.verdict.to}. ${a.verdict.why}`}</td>
505 </tr>`,
506 )}
507 </tbody>
508 </table>
509 </div>
510 ${asked.length === 0
511 ? null
512 : html`<ul class="list" id="model-review-asks">
513 ${asked.map(
514 (c) => html`<li data-change="${c.id}">
515 <div class="row">
516 <span class="row-main">
517 <span class="row-title">${names.get(c.agentId) ?? "an agent"}: ${c.fromModel} → ${c.toModel}</span>
518 <span class="row-meta">${c.why}</span>
519 </span>
520 <div class="row-actions">
521 <form method="post" action="/settings/crew/model-review/${c.id}/take">
522 ${btn(`Move to ${c.toModel}`, { type: "submit", size: "sm" })}
523 </form>
524 <form method="post" action="/settings/crew/model-review/${c.id}/decline">
525 ${btn("Leave it", { type: "submit", size: "sm", variant: "ghost" })}
526 </form>
527 </div>
528 </div>
529 </li>`,
530 )}
531 </ul>`}
532 <div class="table-scroll">
533 <table class="table" id="model-review-trail">
534 <thead><tr><th>When</th><th>Agent</th><th>Change</th><th>State</th><th>Why and result</th></tr></thead>
535 <tbody>
536 ${r.changes.map(
537 (c) => html`<tr data-change="${c.id}">
538 <td>${time(c.createdAt)}</td>
539 <td>${names.get(c.agentId) ?? "an agent"}</td>
540 <td class="mono">${c.fromModel} → ${c.toModel}</td>
541 <td>${c.state}</td>
542 <td>${c.why}${c.result === "" ? "" : ` Result: ${c.result}`}</td>
543 </tr>`,
544 )}
545 </tbody>
546 </table>
547 </div>`,
548 {
549 extra: "mt mb",
550 head: html`${icon("activity")}
551 <h2>Model review</h2>
552 <span class="muted small">Weekly, from the last ${r.windowDays} days of the ledger</span>`,
553 },
554 );
555 }
556
557 function listAllAgentsByName(ctx: Ctx): [string, string][] {
558 const listed = listAllAgents(ctx.app, ctx.auth.user);
559 return (listed.ok ? listed.value : []).map((a) => [a.user.id, a.user.username]);
560 }
561
458562 interface CrewData {
459563 scorecards: ReturnType<typeof loadScorecards>;
460564 spending: ReturnType<typeof loadSpending>;
@@ -582,6 +686,7 @@ function crewView(
582686 recentSplitScorecards,
583687 ),
584688 )}
689 ${when(agents.length > 0, () => modelReviewCard(ctx))}
585690 ${splitsCard(
586691 "Per model",
587692 "Model",
@@ -952,3 +1057,17 @@ export async function agentDelete(ctx: Ctx): Promise<Response> {
9521057 : { tone: "danger" as const, message: "No such agent." }),
9531058 });
9541059 }
1060
1061 // POST /settings/crew/model-review/:id/take and /decline
1062 export async function modelReviewDecide(ctx: Ctx, take: boolean): Promise<Response> {
1063 const form = await adminForm(ctx);
1064 if (form instanceof Response) return form;
1065 const done = decideChange(ctx.app, ctx.auth.user, ctx.params.id ?? "", take);
1066 return flashTo(ctx, "/settings/crew", {
1067 ...(done.ok
1068 ? { message: take ? `${done.value.toModel} it is.` : "Left as it was." }
1069 : { tone: "danger" as const, message: "That change is no longer waiting." }),
1070 });
1071 }
1072 export const modelReviewTake = (ctx: Ctx) => modelReviewDecide(ctx, true);
1073 export const modelReviewDecline = (ctx: Ctx) => modelReviewDecide(ctx, false);
test/model-review.test.ts added +332
@@ -0,0 +1,332 @@
1 /**
2 * The weekly model review (#7815): from the ledger alone, one change per
3 * agent per week, small samples change nothing, the owner's one button, a
4 * result a week on, and an undo when quality fell.
5 */
6 import { afterEach, beforeEach, describe, expect, test } from "bun:test";
7 import { mkdtemp, rm } from "node:fs/promises";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { createApp, type App } from "../src/app.ts";
11 import { runModelReview, decideChange, readReview } from "../src/actions/model_review.ts";
12 import { readScorecards } from "../src/actions/scorecard.ts";
13 import { createAgentRow, getAgent } from "../src/core/agents.ts";
14 import { createIssue } from "../src/core/issues.ts";
15 import { getChange, recordChange, review, type ModelStats } from "../src/core/model_review.ts";
16 import { claimRun, createRun, finishRun, noteRunProgress } from "../src/core/runs.ts";
17 import { createSession } from "../src/core/sessions.ts";
18 import { createUser, getUserByName, type User } from "../src/core/users.ts";
19 import { setLimit } from "../src/settings/limits.ts";
20 import { createRepo } from "../src/actions/repos.ts";
21 import { serve, type Servers } from "../src/http/server.ts";
22 import { adminCookie, seedAdmin } from "./helpers/admin.ts";
23 import { quietLogs } from "./helpers/globals.ts";
24
25 quietLogs();
26
27 const stats = (o: Partial<ModelStats> = {}): ModelStats => ({
28 model: "opus", opened: 10, landed: 10, costPerLanded: 25, redRate: 0.1,
29 reworkRounds: 0.2, reverts: 0, hardShare: 0.1, ...o,
30 });
31 const limits = { minLanded: 8, slackPct: 10 };
32
33 describe("the verdict, as plain code", () => {
34 test("a small sample says so and changes nothing", () => {
35 const v = review("opus", [stats({ landed: 3 })], limits);
36 expect(v.kind).toBe("hold");
37 expect(v.kind === "hold" && v.note).toContain("3 landed");
38 });
39 test("a healthy model steps one place down; the cheapest and unknown models hold", () => {
40 expect(review("opus", [stats()], limits)).toMatchObject({ kind: "propose", to: "sonnet", direction: "down" });
41 expect(review("haiku", [stats({ model: "haiku" })], limits).kind).toBe("hold");
42 expect(review("", [], limits).kind).toBe("hold");
43 expect(review("mystery", [], limits).kind).toBe("hold");
44 });
45 test("mostly hard work is not handed to a cheaper model", () => {
46 expect(review("opus", [stats({ hardShare: 0.9 })], limits).kind).toBe("hold");
47 });
48 test("a model that reverts steps up, never down", () => {
49 expect(review("sonnet", [stats({ model: "sonnet", reverts: 1 })], limits)).toMatchObject({ kind: "propose", to: "opus", direction: "up" });
50 });
51 test("a cheaper model with its own worse numbers stays where it is", () => {
52 const v = review("opus", [stats(), stats({ model: "sonnet", costPerLanded: 1, redRate: 0.6 })], limits);
53 expect(v.kind).toBe("hold");
54 });
55 });
56
57 describe("the weekly review", () => {
58 let dataDir: string;
59 let app: App;
60 let agent: User;
61 let owner: User;
62 let repoId = "";
63 const t0 = new Date("2026-11-02T00:00:00Z");
64 const day = (n: number) => new Date(t0.getTime() + n * 86_400_000);
65
66 /** A landed pull by the agent on `model`, opened at `at`. */
67 function landed(model: string, at: Date, o: { reverted?: boolean; red?: number; cost?: number } = {}): void {
68 const issue = createIssue(app.db, { repoId, authorId: agent.id, title: `p${Math.random()}`, kind: "pull" });
69 if (!issue.ok) throw new Error(issue.error);
70 const id = issue.value.id;
71 app.db.query("UPDATE issue SET created_at = ?2 WHERE id = ?1").run(id, at.toISOString());
72 app.db
73 .query(`INSERT INTO pull (issue_id, base_branch, head_branch, base_oid, head_oid, merged_at) VALUES (?1,'main',?2,'a','b',?3)`)
74 .run(id, `h${issue.value.number}`, at.toISOString());
75 const run = createRun(app.db, { repoId, kind: "agent", actorId: agent.id, issueId: id });
76 claimRun(app.db, run.id);
77 noteRunProgress(app.db, run.id, { model, modelReason: "agent", costUsd: o.cost ?? 1 });
78 finishRun(app.db, run.id, { state: "ok", summary: "" });
79 for (let i = 0; i < (o.red ?? 0); i++) {
80 const c = createRun(app.db, { repoId, kind: "check", issueId: id });
81 claimRun(app.db, c.id);
82 finishRun(app.db, c.id, { state: "failed", summary: "" });
83 }
84 if (o.reverted) {
85 const back = createIssue(app.db, { repoId, authorId: agent.id, title: "revert", kind: "pull" });
86 if (!back.ok) throw new Error(back.error);
87 app.db
88 .query(`INSERT INTO pull (issue_id, base_branch, head_branch, base_oid, head_oid, merged_at, reverts_issue_id) VALUES (?1,'main',?2,'a','b',?3,?4)`)
89 .run(back.value.id, `r${back.value.number}`, at.toISOString(), id);
90 }
91 }
92 const many = (n: number, model: string, at: Date, o = {}) => {
93 for (let i = 0; i < n; i++) landed(model, at, o);
94 };
95
96 beforeEach(async () => {
97 dataDir = await mkdtemp(join(tmpdir(), "forge-review-"));
98 app = createApp({ dataDir, port: 0 });
99 const made = await createUser(app.db, { username: "torque", email: "[email protected]", password: "", displayName: "Torque", agent: true, origin: "agent" });
100 if (!made.ok) throw new Error("user");
101 agent = made.value;
102 createAgentRow(app.db, { userId: agent.id, model: "opus", brief: "", budgetUsd: 0 });
103 const ada = await seedAdmin(app, { username: "ada", email: "[email protected]" });
104 if (!ada.ok) throw new Error("owner");
105 owner = ada.value;
106 const repo = await createRepo(app, owner, { name: "hello", visibility: "public" });
107 if (!repo.ok) throw new Error(JSON.stringify(repo.error));
108 repoId = repo.value.id;
109 });
110 afterEach(async () => {
111 app.close();
112 await rm(dataDir, { recursive: true, force: true });
113 });
114
115 const model = () => getAgent(app.db, agent.id)!.model;
116 const events = (kind: string) =>
117 app.db.query<{ payload: string }, [string]>("SELECT payload FROM event WHERE kind = ?1").all(kind);
118 const admin = () => owner;
119
120 test("too few landed: nothing changes, nothing is logged", () => {
121 many(3, "opus", day(-3));
122 expect(runModelReview(app, t0)).toEqual([]);
123 expect(model()).toBe("opus");
124 expect(readReview(app, t0).changes).toEqual([]);
125 });
126
127 test("a clean model asks with its numbers; the owner's button applies it; it runs once a week", () => {
128 many(8, "opus", day(-3), { cost: 25 });
129 const made = runModelReview(app, t0);
130 expect(made).toHaveLength(1);
131 expect(made[0]).toMatchObject({ state: "asked", fromModel: "opus", toModel: "sonnet" });
132 expect(made[0]!.before.costPerLanded).toBe(25);
133 expect(model()).toBe("opus");
134 expect(events("crew.model_asked")).toHaveLength(1);
135 // not again this week, and not two changes for one agent
136 expect(runModelReview(app, day(3))).toEqual([]);
137 expect(runModelReview(app, day(8))).toEqual([]);
138 expect(decideChange(app, undefined, made[0]!.id, true).ok).toBe(false);
139 const taken = decideChange(app, admin(), made[0]!.id, true, day(8));
140 expect(taken.ok && taken.value.state).toBe("applied");
141 expect(model()).toBe("sonnet");
142 expect(events("crew.model_changed")[0]!.payload).toContain("before");
143 expect(decideChange(app, admin(), made[0]!.id, true).ok).toBe(false);
144 });
145
146 test("declining leaves the model, and the next week may ask again", () => {
147 many(8, "opus", day(-3));
148 const [c] = runModelReview(app, t0);
149 decideChange(app, admin(), c!.id, false, t0);
150 expect(model()).toBe("opus");
151 expect(runModelReview(app, day(7))).toHaveLength(1);
152 });
153
154 test("with the limit on, a step down applies itself and a week on is reported and kept", () => {
155 setLimit(app.db, "model_review.auto", 1, agent.id);
156 many(8, "opus", day(-3));
157 const [c] = runModelReview(app, t0);
158 expect(c!.state).toBe("applied");
159 expect(model()).toBe("sonnet");
160 many(8, "sonnet", day(1), { cost: 1 });
161 runModelReview(app, day(8));
162 const done = readReview(app, day(8)).changes.find((x) => x.id === c!.id)!;
163 expect(done.state).toBe("kept");
164 expect(done.after?.model).toBe("sonnet");
165 expect(done.result).toContain("held on sonnet");
166 // a change just settled counts as this week's move: no second one in the same tick
167 expect(model()).toBe("sonnet");
168 expect(events("crew.model_result")).toHaveLength(1);
169 // a week later the healthy sonnet may step down once more, as its own change
170 expect(runModelReview(app, day(16))).toHaveLength(1);
171 expect(model()).toBe("haiku");
172 });
173
174 test("when quality fell a week on, the change is undone", () => {
175 setLimit(app.db, "model_review.auto", 1, agent.id);
176 many(8, "opus", day(-3));
177 const [c] = runModelReview(app, t0);
178 many(8, "sonnet", day(1), { reverted: true });
179 runModelReview(app, day(8));
180 const done = readReview(app, day(8)).changes.find((x) => x.id === c!.id)!;;
181 expect(done.state).toBe("undone");
182 expect(model()).toBe("opus");
183 expect(done.result).toContain("quality fell");
184 });
185
186 test("a result with too few landed waits, then keeps the change after two weeks", () => {
187 setLimit(app.db, "model_review.auto", 1, agent.id);
188 many(8, "opus", day(-3));
189 const [c] = runModelReview(app, t0);
190 many(2, "sonnet", day(1));
191 runModelReview(app, day(8));
192 const stateAt = (n: number) => readReview(app, day(n)).changes.find((x) => x.id === c!.id)!.state;
193 expect(stateAt(8)).toBe("applied");
194 runModelReview(app, day(15));
195 expect(stateAt(15)).toBe("kept");
196 });
197
198 test("an ask nobody answers lapses after two weeks as declined, and stops blocking the agent", () => {
199 many(8, "opus", day(-3));
200 const [c] = runModelReview(app, t0);
201 expect(c!.state).toBe("asked");
202 runModelReview(app, day(8));
203 expect(getChange(app.db, c!.id)!.state).toBe("asked");
204 expect(runModelReview(app, day(8))).toEqual([]);
205 const next = runModelReview(app, day(15));
206 const lapsed = getChange(app.db, c!.id)!;
207 expect(lapsed.state).toBe("declined");
208 expect(lapsed.result).toContain("nobody answered");
209 expect(model()).toBe("opus");
210 expect(events("crew.model_declined")[0]!.payload).toContain("expired");
211 // the lapse and the next ask are a week and more apart, so the agent is free again
212 expect(next).toHaveLength(1);
213 expect(next[0]!.id).not.toBe(c!.id);
214 });
215
216 test("a failed event leaves neither a changed model nor a half-written trail", () => {
217 many(8, "opus", day(-3));
218 const [c] = runModelReview(app, t0);
219 app.db.exec("CREATE TRIGGER no_event BEFORE INSERT ON event BEGIN SELECT RAISE(ABORT, 'no'); END");
220 expect(() => decideChange(app, admin(), c!.id, true, day(1))).toThrow();
221 expect(model()).toBe("opus");
222 expect(getChange(app.db, c!.id)!.state).toBe("asked");
223 });
224
225 test("a step up always asks, even with the limit on", () => {
226 setLimit(app.db, "model_review.auto", 1, agent.id);
227 app.db.query("UPDATE agent SET model = 'sonnet' WHERE user_id = ?1").run(agent.id);
228 many(8, "sonnet", day(-3), { reverted: true });
229 const [c] = runModelReview(app, t0);
230 expect(c).toMatchObject({ state: "asked", toModel: "opus" });
231 expect(model()).toBe("sonnet");
232 });
233
234 test("the scorecard's doors carry the review, for an admin only", () => {
235 expect(readScorecards(app, undefined)).toBeUndefined();
236 expect(readScorecards(app, admin())?.modelReview.agents[0]?.username).toBe("torque");
237 });
238 });
239
240 describe("the page", () => {
241 let dataDir: string;
242 let app: App;
243 let servers: Servers;
244
245 beforeEach(async () => {
246 dataDir = await mkdtemp(join(tmpdir(), "forge-review-page-"));
247 app = createApp({ dataDir, port: 0 });
248 servers = await serve(app);
249 });
250 afterEach(async () => {
251 servers.stop();
252 app.close();
253 await rm(dataDir, { recursive: true, force: true });
254 });
255
256 test("Settings → Crew shows the review to an admin and refuses a stranger's button", async () => {
257 const cookie = await adminCookie(app, { username: "ada", email: "[email protected]", password: "analytical-engine" });
258 await fetch(`${servers.url}/api/v1/agents`, {
259 method: "POST",
260 headers: { cookie, "content-type": "application/json" },
261 body: JSON.stringify({ username: "smith", model: "opus" }),
262 });
263 const page = await fetch(`${servers.url}/settings/crew`, { headers: { cookie } }).then((r) => r.text());
264 expect(page).toContain("Model review");
265 expect(page).toContain(`id="model-review-verdicts"`);
266 const stranger = await fetch(`${servers.url}/settings/crew/model-review/nope/take`, { method: "POST", redirect: "manual" });
267 expect(stranger.status).not.toBe(200);
268 });
269
270 describe("the button, through the server", () => {
271 const opus: ModelStats = stats();
272 let cookie = "";
273 let agentId = "";
274
275 async function asked(): Promise<string> {
276 return recordChange(app.db, {
277 agentId, fromModel: "opus", toModel: "sonnet", state: "asked", why: "cheaper and holds", before: opus, at: new Date(),
278 });
279 }
280 const post = (path: string, headers: Record<string, string>) =>
281 fetch(`${servers.url}${path}`, { method: "POST", redirect: "manual", headers });
282 const modelNow = () => getAgent(app.db, agentId)!.model;
283
284 beforeEach(async () => {
285 cookie = await adminCookie(app, { username: "ada", email: "[email protected]", password: "analytical-engine" });
286 const made = await fetch(`${servers.url}/api/v1/agents`, {
287 method: "POST",
288 headers: { cookie, "content-type": "application/json" },
289 body: JSON.stringify({ username: "smith", model: "opus" }),
290 }).then((r) => r.json() as Promise<{ id?: string; user?: { id: string } }>);
291 agentId = made.user?.id ?? made.id ?? getUserByName(app.db, "smith")!.id;
292 });
293
294 test("an admin's same-origin POST to take moves the agent; the page then shows it on the trail", async () => {
295 const id = await asked();
296 const page = await fetch(`${servers.url}/settings/crew`, { headers: { cookie } }).then((r) => r.text());
297 expect(page).toContain(`action="/settings/crew/model-review/${id}/take"`);
298 const res = await post(`/settings/crew/model-review/${id}/take`, { cookie, origin: servers.url });
299 expect(res.status).toBe(303);
300 expect(modelNow()).toBe("sonnet");
301 expect(getChange(app.db, id)!.state).toBe("applied");
302 });
303
304 test("an admin's POST to decline leaves the model", async () => {
305 const id = await asked();
306 const res = await post(`/settings/crew/model-review/${id}/decline`, { cookie, origin: servers.url });
307 expect(res.status).toBe(303);
308 expect(modelNow()).toBe("opus");
309 expect(getChange(app.db, id)!.state).toBe("declined");
310 });
311
312 test("a cross-origin POST, an anonymous one and a non-admin's change nothing", async () => {
313 const id = await asked();
314 const cross = await post(`/settings/crew/model-review/${id}/take`, { cookie, origin: "https://evil.example" });
315 expect(cross.status).toBe(403);
316 const anon = await post(`/settings/crew/model-review/${id}/take`, { origin: servers.url });
317 expect(anon.headers.get("location") ?? "").not.toContain("/settings/crew");
318 const bob = await createUser(app.db, {
319 origin: "seed", trusted: true, username: "bob", email: "[email protected]", password: "a-good-passphrase",
320 });
321 if (!bob.ok) throw new Error(bob.error);
322 const bobCookie = `forge_session=${createSession(app.db, bob.value.id).token}`;
323 for (const verb of ["take", "decline"]) {
324 const res = await post(`/settings/crew/model-review/${id}/${verb}`, { cookie: bobCookie, origin: servers.url });
325 expect(res.status).toBeGreaterThanOrEqual(300);
326 expect(res.headers.get("location") ?? "").not.toBe("/settings/crew");
327 }
328 expect(modelNow()).toBe("opus");
329 expect(getChange(app.db, id)!.state).toBe("asked");
330 });
331 });
332 });
test/tui-model-review.test.ts added +119
@@ -0,0 +1,119 @@
1 /** The weekly model review in the terminal's crew scorecard (#7815): the verdicts, the trail, and the Move / Leave decision, for a forge admin only. */
2 import { afterEach, beforeEach, expect, test } from "bun:test";
3 import { mkdtemp, rm } from "node:fs/promises";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp, type App } from "../src/app.ts";
7 import { createAgent } from "../src/actions/crew.ts";
8 import { getAgent } from "../src/core/agents.ts";
9 import { recordChange, getChange, type ModelStats } from "../src/core/model_review.ts";
10 import { createUser, getUserByName } from "../src/core/users.ts";
11 import { createForgeApp } from "../src/tui/forge.ts";
12 import { headlessTerminal } from "../src/tui/term/terminal.ts";
13 import { quietLogs } from "./helpers/globals.ts";
14 import { settleApp } from "./helpers/tui.ts";
15
16 quietLogs();
17
18 const TITLE = "Crew scorecard per agent";
19 const before: ModelStats = {
20 model: "opus", opened: 10, landed: 10, costPerLanded: 25, redRate: 0.1, reworkRounds: 0.2, reverts: 0, hardShare: 0.1,
21 };
22 let dir = "";
23 let app: App;
24 let agentId = "";
25 let changeId = "";
26
27 beforeEach(async () => {
28 dir = await mkdtemp(join(tmpdir(), "forge-tui-review-"));
29 app = createApp({ dataDir: dir, port: 0 });
30 for (const [username, adminIfFirst] of [["ada", true], ["bob", false]] as const) {
31 const made = await createUser(app.db, {
32 origin: "seed", trusted: true, username, email: `${username}@forge.local`,
33 password: "a-good-passphrase", adminIfFirst,
34 });
35 if (!made.ok) throw new Error(made.error);
36 }
37 const made = await createAgent(app, getUserByName(app.db, "ada"), { username: "smith", model: "opus" });
38 if (!made.ok) throw new Error(String(made.error));
39 agentId = made.value.user.id;
40 changeId = recordChange(app.db, {
41 agentId, fromModel: "opus", toModel: "sonnet", state: "asked", why: "cheaper and holds", before, at: new Date(),
42 });
43 });
44 afterEach(async () => {
45 app.close();
46 await rm(dir, { recursive: true, force: true });
47 });
48
49 async function open(username: string) {
50 const { app: tui } = createForgeApp(
51 app,
52 { sshUsername: "t", remoteAddress: "127.0.0.1", user: getUserByName(app.db, username)! },
53 { terminal: headlessTerminal(200, 40), exitHooks: false },
54 );
55 tui.start();
56 const key = (k: string) => tui.dispatch({ type: "key", key: k, ctrl: false, alt: false, shift: false });
57 key(":");
58 for (const ch of TITLE) key(ch);
59 await settleApp(tui);
60 key("enter");
61 await settleApp(tui);
62 return { key, text: () => tui.screen.toText(), settle: () => settleApp(tui), stop: () => tui.stop() };
63 }
64
65 test("an admin sees the verdicts and the waiting change, and `m` then `y` moves the agent", async () => {
66 const t = await open("ada");
67 expect(t.text()).toContain("model review · smith on opus");
68 expect(t.text()).toContain("waiting · smith: opus → sonnet (asked)");
69 t.key("j");
70 t.key("j");
71 t.key("m");
72 await t.settle();
73 expect(t.text()).toContain("Move smith to sonnet");
74 t.key("y");
75 await t.settle();
76 expect(getAgent(app.db, agentId)!.model).toBe("sonnet");
77 expect(getChange(app.db, changeId)!.state).toBe("applied");
78 expect(t.text()).toContain("trail · smith: opus → sonnet (applied)");
79 t.stop();
80 });
81
82 test("`n` then `y` leaves the model where it was and declines the change", async () => {
83 const t = await open("ada");
84 t.key("j");
85 t.key("j");
86 t.key("n");
87 await t.settle();
88 t.key("y");
89 await t.settle();
90 expect(getAgent(app.db, agentId)!.model).toBe("opus");
91 expect(getChange(app.db, changeId)!.state).toBe("declined");
92 t.stop();
93 });
94
95 test("cancelling the confirm changes nothing, and a row that is not waiting has no button", async () => {
96 const t = await open("ada");
97 t.key("m"); // on the first scorecard row: nothing waits there
98 await t.settle();
99 expect(t.text()).not.toContain("Move smith");
100 t.key("j");
101 t.key("j");
102 t.key("m");
103 await t.settle();
104 t.key("n");
105 await t.settle();
106 expect(getChange(app.db, changeId)!.state).toBe("asked");
107 expect(getAgent(app.db, agentId)!.model).toBe("opus");
108 t.stop();
109 });
110
111 test("a non-admin sees no review and cannot decide", async () => {
112 const t = await open("bob");
113 expect(t.text()).not.toContain("model review");
114 t.key("m");
115 t.key("y");
116 await t.settle();
117 expect(getChange(app.db, changeId)!.state).toBe("asked");
118 t.stop();
119 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs