Public: anyone can read this; the crew can change it.
Each account has its own daily and per-run model spend ceiling, checked before a run starts #2269
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/897-each-account-has-its-own-daily-and-per-r → main 7 commits fork 4 threads · 4 unresolved
README.md +2
| @@ -691,6 +691,8 @@ an organization (its page), down to one crew member: a work plan pays for | ||
| 691 | 691 | the work repositories and nothing else. Two on one grant take turns, and |
| 692 | 692 | one the provider rests hands its runs to the other until the reset. |
| 693 | 693 | **Model budget.** A new account has none: its repositories' crew does not run, spends nothing, and says "This repository has no model account". A person adds their own subscription at Settings → Model (or the terminal's **Model budget** command, `a`); it is sealed, granted to their own repositories only, and one per account. A forge admin who is a person gives an account a capped grant, in dollars a month spent on the forge's own credential, from the account's page, the terminal's **Model grant** command, `POST /api/v1/admin/accounts/:name/model-grant` (`{"usdPerMonth": 20}`, `null` takes it back) or the MCP `model_grant` tool; each change is in the audit. The cap is the UTC month's spend in the usage ledger, checked when a run starts and on every spend tick: a run past it stops, and its page says the grant is spent. Two runs that start together each begin with the whole remainder; the tick check ends both once the sum crosses the cap, so the overshoot is at most what they spend between ticks. |
| 694 | ||
| 695 | **Spend ceilings.** Apart from the grant, a forge admin who is a person may set any account's — a person's, an agent's or an organization's — daily (UTC) and per-run ceilings in dollars: on the account's page (`/settings/accounts/<name>`, an organization's too, by that address; an agent and an admin's own account have no such page), the terminal's **Model ceiling** command, `POST /api/v1/admin/accounts/:name/ceiling` (`{"dailyUsd": 20, "perRunUsd": 5}`) or the MCP `model_ceiling` tool; each change is in the audit. Both empty on the page, `none` in the terminal (its prompt shows the current ceilings, and an empty answer changes nothing), or `null` for both on the API and MCP takes them back. An organization's page is linked from its own page for an admin. An account has no ceiling until one is set, so this forge's own crew is never stopped by a default nobody chose. A run on an account with a ceiling does not start when the day's is spent or the per-run one is too small, is capped at the smaller of the per-run ceiling and what is left of the day, and stops mid-run when the day's is spent. | |
| 694 | 696 | **Sandbox.** An agent's shell is held to a policy — the network is the forge |
| 695 | 697 | and the package registry, writes stay in the clone, the forge's data and the |
| 696 | 698 | operator's credentials are unreadable — by bubblewrap and socat on Linux |
docs/APPS.md +13
| @@ -1228,3 +1228,16 @@ ro/rw access token — the same grant, not a second credential. Present that | ||
| 1228 | 1228 | access token to `/oauth/userinfo` for the claims. `/oauth/jwks` publishes the |
| 1229 | 1229 | current signing key and the previous one; keys rotate every 30 days and live |
| 1230 | 1230 | in the database, never in the repository. |
| 1231 | ||
| 1232 | ## Account spend ceilings | |
| 1233 | ||
| 1234 | An account (a repository's owner: a person, an agent or an organization) | |
| 1235 | has a daily and a per-run model spend ceiling (`core/account_ceiling.ts`), | |
| 1236 | apart from the forge-wide one. An account has none until an admin sets them, | |
| 1237 | so no default ever stops a crew nobody chose to limit: on the account's page | |
| 1238 | under Settings → Accounts (an organization's too), in the terminal's **Model | |
| 1239 | ceiling** command, `POST /api/v1/admin/accounts/:name/ceiling` | |
| 1240 | (`{"dailyUsd": 20, "perRunUsd": 5}`) or the MCP `model_ceiling` tool. A run | |
| 1241 | that would pass either does not start and its message names the ceiling; one | |
| 1242 | under way stops at the day's. Spend is the | |
| 1243 | account's own repositories' usage today (UTC); another account's never counts. |
src/actions/model_grant.ts +31
| @@ -11,6 +11,7 @@ import { | ||
| 11 | 11 | setModelGrant, |
| 12 | 12 | validGrant, |
| 13 | 13 | } from "../core/model_grant.ts"; |
| 14 | import { cents, clearAccountCeiling, setAccountCeiling, validCeiling } from "../core/account_ceiling.ts"; | |
| 14 | 15 | import { getUserByName, type User } from "../core/users.ts"; |
| 15 | 16 | import { err, ok, type Result } from "../kernel/result.ts"; |
| 16 | 17 | import { requireUser } from "./access.ts"; |
| @@ -75,3 +76,33 @@ export function addOwnModel( | ||
| 75 | 76 | }); |
| 76 | 77 | return ok(null); |
| 77 | 78 | } |
| 79 | ||
| 80 | /** An admin sets any account's — a person's, an agent's, an organization's — daily and per-run ceilings, or takes both back (null, null); the change is on the outbox with who made it. */ | |
| 81 | export function setAccountCeilings( | |
| 82 | app: App, | |
| 83 | actor: User | undefined, | |
| 84 | username: string, | |
| 85 | dailyUsd: number | null, | |
| 86 | perRunUsd: number | null, | |
| 87 | ): Result< | |
| 88 | { username: string; dailyUsd: number | null; perRunUsd: number | null }, | |
| 89 | ModelGrantError | |
| 90 | > { | |
| 91 | const admin = requireUser(actor); | |
| 92 | if (!admin.ok || !admin.value.isAdmin || admin.value.kind !== "user") | |
| 93 | return err("not_found"); | |
| 94 | const user = getUserByName(app.db, username); | |
| 95 | if (user === undefined) return err("not_found"); | |
| 96 | if ((dailyUsd === null) !== (perRunUsd === null)) return err("invalid_amount"); | |
| 97 | if (dailyUsd === null || perRunUsd === null) { | |
| 98 | clearAccountCeiling(app.db, user.id); | |
| 99 | const value = { username: user.username, dailyUsd: null, perRunUsd: null }; | |
| 100 | app.bus.append({ kind: "account.ceiling_set", actorId: admin.value.id, payload: value }); | |
| 101 | return ok(value); | |
| 102 | } | |
| 103 | if (!validCeiling(dailyUsd) || !validCeiling(perRunUsd)) return err("invalid_amount"); | |
| 104 | setAccountCeiling(app.db, user.id, dailyUsd, perRunUsd, admin.value.id, new Date()); | |
| 105 | const value = { username: user.username, dailyUsd: cents(dailyUsd), perRunUsd: cents(perRunUsd) }; | |
| 106 | app.bus.append({ kind: "account.ceiling_set", actorId: admin.value.id, payload: value }); | |
| 107 | return ok(value); | |
| 108 | } |
src/core/account_ceiling.ts added +110
| @@ -0,0 +1,110 @@ | ||
| 1 | /** | |
| 2 | * What one account's crew may spend on a model: a ceiling per UTC day and a | |
| 3 | * ceiling per run, apart from the forge-wide one. An account has none until | |
| 4 | * an admin sets them — a person's, an agent's or an organization's — so a | |
| 5 | * forge's own crew is never stopped by a default nobody chose. Spend is counted from the | |
| 6 | * usage ledger over the account's own repositories, so another account's | |
| 7 | * spend never counts here. Pure data; who may set is the action's to decide. | |
| 8 | */ | |
| 9 | ||
| 10 | import type { Database } from "bun:sqlite"; | |
| 11 | ||
| 12 | /** The most either ceiling may be, in dollars. */ | |
| 13 | export const CEILING_MAX_USD = 10_000; | |
| 14 | ||
| 15 | export interface AccountCeiling { | |
| 16 | readonly dailyUsd: number; | |
| 17 | readonly perRunUsd: number; | |
| 18 | } | |
| 19 | ||
| 20 | export const dayStart = (now: Date): string => | |
| 21 | new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate())).toISOString(); | |
| 22 | ||
| 23 | /** Whole cents, above zero and within the maximum. */ | |
| 24 | export const validCeiling = (usd: number): boolean => | |
| 25 | Number.isFinite(usd) && usd >= 0.01 && usd <= CEILING_MAX_USD; | |
| 26 | ||
| 27 | /** The ceilings an admin set, or undefined: no ceiling. */ | |
| 28 | export function getAccountCeiling(db: Database, userId: string): AccountCeiling | undefined { | |
| 29 | const row = db | |
| 30 | .query<{ daily_usd: number; per_run_usd: number }, [string]>( | |
| 31 | "SELECT daily_usd, per_run_usd FROM account_ceiling WHERE user_id = ?1", | |
| 32 | ) | |
| 33 | .get(userId); | |
| 34 | return row ? { dailyUsd: row.daily_usd, perRunUsd: row.per_run_usd } : undefined; | |
| 35 | } | |
| 36 | ||
| 37 | /** Dollars rounded to whole cents. */ | |
| 38 | export const cents = (n: number): number => Math.round(n * 100) / 100; | |
| 39 | ||
| 40 | /** Take the ceilings back: the account has none again. */ | |
| 41 | export function clearAccountCeiling(db: Database, userId: string): void { | |
| 42 | db.query("DELETE FROM account_ceiling WHERE user_id = ?1").run(userId); | |
| 43 | } | |
| 44 | ||
| 45 | /** Set both ceilings. False when either is not a valid amount. */ | |
| 46 | export function setAccountCeiling( | |
| 47 | db: Database, | |
| 48 | userId: string, | |
| 49 | dailyUsd: number, | |
| 50 | perRunUsd: number, | |
| 51 | by: string, | |
| 52 | now: Date, | |
| 53 | ): boolean { | |
| 54 | if (!validCeiling(dailyUsd) || !validCeiling(perRunUsd)) return false; | |
| 55 | db.query( | |
| 56 | `INSERT INTO account_ceiling (user_id, daily_usd, per_run_usd, set_by, set_at) | |
| 57 | VALUES (?1, ?2, ?3, ?4, ?5) | |
| 58 | ON CONFLICT(user_id) DO UPDATE SET daily_usd = ?2, per_run_usd = ?3, set_by = ?4, set_at = ?5`, | |
| 59 | ).run(userId, cents(dailyUsd), cents(perRunUsd), by, now.toISOString()); | |
| 60 | return true; | |
| 61 | } | |
| 62 | ||
| 63 | /** What the account's repositories have spent today (UTC), running runs included. */ | |
| 64 | export function daySpent(db: Database, userId: string, now: Date): number { | |
| 65 | return ( | |
| 66 | db | |
| 67 | .query<{ n: number }, [string, string]>( | |
| 68 | `SELECT COALESCE(SUM(u.cost_usd), 0) AS n | |
| 69 | FROM usage u JOIN repo r ON r.id = u.repo_id | |
| 70 | WHERE r.owner_id = ?1 AND u.at >= ?2`, | |
| 71 | ) | |
| 72 | .get(userId, dayStart(now))?.n ?? 0 | |
| 73 | ); | |
| 74 | } | |
| 75 | ||
| 76 | /** Dollars left today, or undefined when the account has no ceiling. */ | |
| 77 | export function dailyLeft(db: Database, userId: string, now: Date): number | undefined { | |
| 78 | const c = getAccountCeiling(db, userId); | |
| 79 | return c === undefined ? undefined : Math.max(0, c.dailyUsd - daySpent(db, userId, now)); | |
| 80 | } | |
| 81 | ||
| 82 | export type CeilingVerdict = | |
| 83 | | { readonly ok: true; readonly capUsd: number | undefined } | |
| 84 | | { readonly ok: false; readonly said: string }; | |
| 85 | ||
| 86 | /** | |
| 87 | * May a run start for this account, and with what cap: the smaller of its | |
| 88 | * per-run ceiling and what is left of today. A refusal names the ceiling. | |
| 89 | */ | |
| 90 | export function checkCeiling( | |
| 91 | db: Database, | |
| 92 | userId: string, | |
| 93 | minRunUsd: number, | |
| 94 | now: Date, | |
| 95 | ): CeilingVerdict { | |
| 96 | const c = getAccountCeiling(db, userId); | |
| 97 | if (c === undefined) return { ok: true, capUsd: undefined }; | |
| 98 | if (c.perRunUsd < minRunUsd) | |
| 99 | return { | |
| 100 | ok: false, | |
| 101 | said: `This account's per-run ceiling ($${c.perRunUsd.toFixed(2)}) is too small to start a run: an admin can raise it`, | |
| 102 | }; | |
| 103 | const left = dailyLeft(db, userId, now) ?? 0; | |
| 104 | if (left < minRunUsd) | |
| 105 | return { | |
| 106 | ok: false, | |
| 107 | said: `This account's daily ceiling ($${c.dailyUsd.toFixed(2)}) is spent for today: an admin can raise it`, | |
| 108 | }; | |
| 109 | return { ok: true, capUsd: Math.min(c.perRunUsd, left) }; | |
| 110 | } |
src/core/migrations/20261016180000-account-ceiling.ts added +12
| @@ -0,0 +1,12 @@ | ||
| 1 | // Each account has its own daily and per-run model spend ceiling (#897), | |
| 2 | // apart from the forge-wide one. A row only when an admin set one; | |
| 3 | // no row means no ceiling, and clearing one deletes the row. | |
| 4 | export default ` | |
| 5 | CREATE TABLE account_ceiling ( | |
| 6 | user_id TEXT PRIMARY KEY REFERENCES user(id) ON DELETE CASCADE, | |
| 7 | daily_usd REAL NOT NULL CHECK (daily_usd > 0), | |
| 8 | per_run_usd REAL NOT NULL CHECK (per_run_usd > 0), | |
| 9 | set_by TEXT, | |
| 10 | set_at TEXT NOT NULL | |
| 11 | ) WITHOUT ROWID; | |
| 12 | `; |
src/crew/runner.ts +26 −2
| @@ -112,6 +112,7 @@ import { heldNow, stuckOf } from "./held.ts"; | ||
| 112 | 112 | import { noteLimit } from "../core/standing.ts"; |
| 113 | 113 | import { pauseReason } from "./pacing.ts"; |
| 114 | 114 | import { grantLeft } from "../core/model_grant.ts"; |
| 115 | import { checkCeiling, dailyLeft } from "../core/account_ceiling.ts"; | |
| 115 | 116 | import { |
| 116 | 117 | credentialEnvOf, |
| 117 | 118 | hostOfApiBase, |
| @@ -544,6 +545,8 @@ interface Meter { | ||
| 544 | 545 | stop: () => void; |
| 545 | 546 | /** Set when the run spends an owner's capped grant: whose. */ |
| 546 | 547 | grantOwner?: string; |
| 548 | /** Set on every run: whose daily ceiling it counts against. */ | |
| 549 | ceilingOwner?: string; | |
| 547 | 550 | /** Why the run was stopped by its grant's cap, when it was. */ |
| 548 | 551 | stoppedFor?: string; |
| 549 | 552 | } |
| @@ -567,6 +570,10 @@ const LIVE_MS = 2_000; | ||
| 567 | 570 | const GRANT_SPENT = |
| 568 | 571 | "This month's model grant is spent: the run stopped at the cap; an admin can raise it"; |
| 569 | 572 | |
| 573 | /** Said on a run stopped at its account's daily ceiling. */ | |
| 574 | const DAILY_SPENT = | |
| 575 | "This account's daily ceiling is spent: the run stopped; an admin can raise it"; | |
| 576 | ||
| 570 | 577 | /** |
| 571 | 578 | * A running session's spend so far, as its `running` line in the ledger, |
| 572 | 579 | * and a `usage.delta` on the outbox so the board and the Subscriptions page |
| @@ -608,6 +615,13 @@ function meterLive( | ||
| 608 | 615 | meter.stop(); |
| 609 | 616 | } |
| 610 | 617 | } |
| 618 | if (meter.ceilingOwner !== undefined && meter.stoppedFor === undefined) { | |
| 619 | const left = dailyLeft(app.db, meter.ceilingOwner, new Date(now)); | |
| 620 | if (left !== undefined && left <= 0) { | |
| 621 | meter.stoppedFor = DAILY_SPENT; | |
| 622 | meter.stop(); | |
| 623 | } | |
| 624 | } | |
| 611 | 625 | app.bus.append({ |
| 612 | 626 | kind: "usage.delta", |
| 613 | 627 | repoId: run.repoId, |
| @@ -1011,6 +1025,14 @@ async function work( | ||
| 1011 | 1025 | write(said); |
| 1012 | 1026 | return { state: "failed", summary: said }; |
| 1013 | 1027 | } |
| 1028 | // The account's own ceilings, daily and per run, apart from the forge's: | |
| 1029 | // a run past either does not start and spends nothing. | |
| 1030 | const ceiling = checkCeiling(app.db, repo.ownerId, MIN_RUN_USD, new Date()); | |
| 1031 | if (!ceiling.ok) { | |
| 1032 | write(ceiling.said); | |
| 1033 | return { state: "failed", summary: ceiling.said }; | |
| 1034 | } | |
| 1035 | meter.ceilingOwner = repo.ownerId; | |
| 1014 | 1036 | if (picked.kind === "ready") { |
| 1015 | 1037 | write(`subscription: ${picked.subscription.name}`); |
| 1016 | 1038 | noteRunProgress(app.db, run.id, { |
| @@ -1045,12 +1067,14 @@ async function work( | ||
| 1045 | 1067 | // The budget in two parts: the work, and a reserve for landing it. An |
| 1046 | 1068 | // agent cut off mid-task otherwise leaves a tree the forge commits blind |
| 1047 | 1069 | // and a pull whose description is the error. |
| 1048 | const budget = | |
| 1070 | const budget = Math.min( | |
| 1071 | ceiling.capUsd ?? Infinity, | |
| 1049 | 1072 | picked.kind === "ready" && picked.capUsd !== undefined |
| 1050 | 1073 | ? Math.min(ownBudget, picked.capUsd) |
| 1051 | 1074 | : grantCap !== undefined |
| 1052 | 1075 | ? Math.min(ownBudget, grantCap) |
| 1053 | : ownBudget; | |
| 1076 | : ownBudget, | |
| 1077 | ); | |
| 1054 | 1078 | const reserve = wrapUpReserve(budget); |
| 1055 | 1079 | // Asked again for every invocation (`crew/sandbox.ts`): an agent made |
| 1056 | 1080 | // since the last one is denied from this one on. |
src/http/api.ts +18 −1
| @@ -8,7 +8,7 @@ | ||
| 8 | 8 | * back to continue); errors are `{ error, message }`. |
| 9 | 9 | */ |
| 10 | 10 | |
| 11 | import { setAccountGrant } from "../actions/model_grant.ts"; | |
| 11 | import { setAccountCeilings, setAccountGrant } from "../actions/model_grant.ts"; | |
| 12 | 12 | import { storageLimitMessage } from "../core/storage.ts"; |
| 13 | 13 | import { voteVariant } from "../actions/variants.ts"; |
| 14 | 14 | import { parentForgeFor } from "../core/parent_releases.ts"; |
| @@ -1604,6 +1604,23 @@ export function apiRoutes(router: Router<Handler>): void { | ||
| 1604 | 1604 | : notFound(); |
| 1605 | 1605 | }); |
| 1606 | 1606 | |
| 1607 | // An account's daily and per-run model spend ceilings, dollars (any account, orgs too). | |
| 1608 | router.add("POST", "/api/v1/admin/accounts/:username/ceiling", async (ctx) => { | |
| 1609 | const denied = accountAdmin(ctx, "admin"); | |
| 1610 | if (denied) return denied; | |
| 1611 | const input = await body(ctx.request); | |
| 1612 | const { dailyUsd, perRunUsd } = input; | |
| 1613 | const amount = (v: unknown): v is number | null => | |
| 1614 | v === null || (typeof v === "number" && Number.isFinite(v)); | |
| 1615 | if (!amount(dailyUsd) || !amount(perRunUsd)) | |
| 1616 | return apiError("invalid", 422, "dailyUsd and perRunUsd are numbers of dollars, both null to take them back"); | |
| 1617 | const done = setAccountCeilings(ctx.app, ctx.auth.user, ctx.params.username!, dailyUsd, perRunUsd); | |
| 1618 | if (done.ok) return json(done.value); | |
| 1619 | return done.error === "invalid_amount" | |
| 1620 | ? apiError("invalid", 422, "dailyUsd and perRunUsd are each from 0.01 to 10000 dollars, or both null") | |
| 1621 | : notFound(); | |
| 1622 | }); | |
| 1623 | ||
| 1607 | 1624 | router.add("POST", "/api/v1/admin/accounts/:username/unsuspend", (ctx) => { |
| 1608 | 1625 | const denied = accountAdmin(ctx, "admin"); |
| 1609 | 1626 | if (denied) return denied; |
src/http/mcp.ts +43 −1
| @@ -61,7 +61,7 @@ import { | ||
| 61 | 61 | CARD_URI, |
| 62 | 62 | cardFor, |
| 63 | 63 | } from "../web/studio/cards.ts"; |
| 64 | import { setAccountGrant } from "../actions/model_grant.ts"; | |
| 64 | import { setAccountCeilings, setAccountGrant } from "../actions/model_grant.ts"; | |
| 65 | 65 | import { keepThought } from "../actions/thought.ts"; |
| 66 | 66 | import { |
| 67 | 67 | FROM_WORDS, |
| @@ -996,6 +996,48 @@ const TOOLS: readonly Tool[] = [ | ||
| 996 | 996 | return good(set.value); |
| 997 | 997 | }, |
| 998 | 998 | }, |
| 999 | { | |
| 1000 | name: "model_ceiling", | |
| 1001 | description: | |
| 1002 | "Set an account's daily and per-run model spend ceilings, in dollars (a person's, an agent's or an organization's; an account has none until set). A run that would pass either does not start; one under way stops at the day's. A forge admin who is a person, with an admin token, sets it; `dryRun` is true unless you say false. Each change is recorded in the audit.", | |
| 1003 | inputSchema: { | |
| 1004 | type: "object", | |
| 1005 | properties: { | |
| 1006 | username: { type: "string" }, | |
| 1007 | dailyUsd: { type: ["number", "null"], description: "0.01 to 10000; null (with perRunUsd null) takes the ceilings back." }, | |
| 1008 | perRunUsd: { type: ["number", "null"], description: "0.01 to 10000; null (with dailyUsd null) takes the ceilings back." }, | |
| 1009 | dryRun: { type: "boolean", description: "Plan only (the default). Send false to set it." }, | |
| 1010 | }, | |
| 1011 | required: ["username", "dailyUsd", "perRunUsd"], | |
| 1012 | }, | |
| 1013 | writes: true, | |
| 1014 | dryByDefault: true, | |
| 1015 | run(ctx, user, args) { | |
| 1016 | const { username, dailyUsd, perRunUsd } = args; | |
| 1017 | if ( | |
| 1018 | typeof username !== "string" || | |
| 1019 | !(dailyUsd === null || (typeof dailyUsd === "number" && Number.isFinite(dailyUsd))) || | |
| 1020 | !(perRunUsd === null || (typeof perRunUsd === "number" && Number.isFinite(perRunUsd))) | |
| 1021 | ) | |
| 1022 | return bad("invalid", "send a username, dailyUsd and perRunUsd in dollars (both null takes them back)"); | |
| 1023 | if (!mayBoards(user)) return bad("forbidden", "a ceiling is a forge admin's to set"); | |
| 1024 | if (args["dryRun"] !== false) | |
| 1025 | return good({ | |
| 1026 | username, | |
| 1027 | dailyUsd, | |
| 1028 | perRunUsd, | |
| 1029 | says: "This is the plan; nothing changed. Send dryRun: false to set it.", | |
| 1030 | }); | |
| 1031 | if (!scopeAllows(ctx.auth.scope ?? "ro", "admin")) | |
| 1032 | return bad("forbidden", "a ceiling needs an admin token"); | |
| 1033 | const set = setAccountCeilings(ctx.app, user, username, dailyUsd, perRunUsd); | |
| 1034 | if (!set.ok) | |
| 1035 | return set.error === "invalid_amount" | |
| 1036 | ? bad("invalid", "dailyUsd and perRunUsd are each from 0.01 to 10000 dollars, or both null") | |
| 1037 | : bad("forbidden", "no such account"); | |
| 1038 | return good(set.value); | |
| 1039 | }, | |
| 1040 | }, | |
| 999 | 1041 | { |
| 1000 | 1042 | name: "outsiders", |
| 1001 | 1043 | description: |
src/http/server.ts +2 −1
| @@ -153,7 +153,7 @@ import { | ||
| 153 | 153 | subscriptionHeadroom, |
| 154 | 154 | subscriptionWake, |
| 155 | 155 | } from "../web/pages/subscriptions.ts"; |
| 156 | import { grantAct, modelAdd, modelPage } from "../web/pages/model.ts"; | |
| 156 | import { ceilingAct, grantAct, modelAdd, modelPage } from "../web/pages/model.ts"; | |
| 157 | 157 | import { |
| 158 | 158 | changePassword, |
| 159 | 159 | iconPng, |
| @@ -798,6 +798,7 @@ export function buildRouter(): Router<Handler> { | ||
| 798 | 798 | router.add("GET", "/settings/model", modelPage); |
| 799 | 799 | router.add("POST", "/settings/model", modelAdd); |
| 800 | 800 | router.add("POST", "/settings/accounts/:username/model", grantAct); |
| 801 | router.add("POST", "/settings/accounts/:username/ceiling", ceilingAct); | |
| 801 | 802 | router.add("GET", "/settings/accounts/:username", personSignInsPage); |
| 802 | 803 | router.add( |
| 803 | 804 | "POST", |
src/tui/forge.ts +8 −1
| @@ -47,7 +47,7 @@ import { accessScreen } from "./screens/access.ts"; | ||
| 47 | 47 | import { flagsScreen } from "./screens/flags.ts"; |
| 48 | 48 | import { cloudScreen } from "./screens/cloud.ts"; |
| 49 | 49 | import { usageScreen } from "./screens/usage.ts"; |
| 50 | import { modelGrantPrompt, modelScreen } from "./screens/model.ts"; | |
| 50 | import { modelCeilingPrompt, modelGrantPrompt, modelScreen } from "./screens/model.ts"; | |
| 51 | 51 | import { proposalTokensPrompt } from "./screens/proposal_tokens.ts"; |
| 52 | 52 | import { quotasScreen } from "./screens/quotas.ts"; |
| 53 | 53 | import { domainsScreen } from "./screens/domains.ts"; |
| @@ -387,6 +387,13 @@ export function forgeCommands(session: Session): (ctx: Ctx) => Command[] { | ||
| 387 | 387 | hint: "give an account dollars a month, or take it back", |
| 388 | 388 | run: () => ctx.push(modelGrantPrompt(session, ctx, blank("Model grant"))), |
| 389 | 389 | }); |
| 390 | if (session.user?.isAdmin === true && session.user.kind === "user") | |
| 391 | commands.push({ | |
| 392 | id: "model-ceiling", | |
| 393 | title: "Model ceiling", | |
| 394 | hint: "set an account's dollars a day and a run", | |
| 395 | run: () => ctx.push(modelCeilingPrompt(session, ctx, blank("Model ceiling"))), | |
| 396 | }); | |
| 390 | 397 | // Account quotas, as /admin/limits has them: an admin who is not an agent (#2191). |
| 391 | 398 | if (session.user?.isAdmin === true && session.user.kind !== "agent") |
| 392 | 399 | commands.push({ |
src/tui/screens/model.ts +40
| @@ -10,8 +10,11 @@ | ||
| 10 | 10 | import { |
| 11 | 11 | ADD_MESSAGES, |
| 12 | 12 | addOwnModel, |
| 13 | setAccountCeilings, | |
| 13 | 14 | setAccountGrant, |
| 14 | 15 | } from "../../actions/model_grant.ts"; |
| 16 | import { getAccountCeiling } from "../../core/account_ceiling.ts"; | |
| 17 | import { getUserByName } from "../../core/users.ts"; | |
| 15 | 18 | import { budgetLine, ownSubscription } from "../../core/model_grant.ts"; |
| 16 | 19 | import type { Ctx, Screen } from "../app/app.ts"; |
| 17 | 20 | import { binding } from "../app/keys.ts"; |
| @@ -90,3 +93,40 @@ export function modelGrantPrompt(f: Session, ctx: Ctx, parent: Screen): Screen { | ||
| 90 | 93 | ), |
| 91 | 94 | }); |
| 92 | 95 | } |
| 96 | ||
| 97 | /** What an account's ceilings are now, for the prompt's label. */ | |
| 98 | function nowCeilings(f: Session, name: string): string { | |
| 99 | const user = getUserByName(f.app.db, name); | |
| 100 | const c = user === undefined ? undefined : getAccountCeiling(f.app.db, user.id); | |
| 101 | return c === undefined ? "now none" : `now $${c.dailyUsd.toFixed(2)} a day, $${c.perRunUsd.toFixed(2)} a run`; | |
| 102 | } | |
| 103 | ||
| 104 | /** Admin: an account's name (an organization's too), then its daily and per-run ceilings in dollars. */ | |
| 105 | export function modelCeilingPrompt(f: Session, ctx: Ctx, parent: Screen): Screen { | |
outdated
A bare Enter here takes away a spend ceiling, and nothing shows what was there before. Use an explicit |
||
| 106 | return promptScreen(ctx, parent, { | |
| 107 | title: "Model ceiling", | |
| 108 | label: "account name", | |
| 109 | onSubmit: (name) => | |
| 110 | later(ctx, () => | |
| 111 | promptScreen(ctx, parent, { | |
| 112 | title: `Model ceiling for ${name.trim()}`, | |
| 113 | label: `${nowCeilings(f, name.trim())}; new: 20 5, or none`, | |
| 114 | onSubmit: (raw) => { | |
| 115 | if (raw.trim().toLowerCase() === "none") { | |
| 116 | const cleared = setAccountCeilings(f.app, f.user, name.trim(), null, null); | |
| 117 | if (!cleared.ok) return "No such account."; | |
| 118 | ctx.toast(`${name.trim()}: ceilings taken back.`); | |
| 119 | return undefined; | |
| 120 | } | |
| 121 | const [daily, perRun, ...rest] = raw.trim().split(/\s+/).map(Number); | |
| 122 | if (daily === undefined || perRun === undefined || rest.length > 0) | |
| 123 | return "Two amounts, like 20 5."; | |
| 124 | const done = setAccountCeilings(f.app, f.user, name.trim(), daily, perRun); | |
| 125 | if (!done.ok) return done.error === "invalid_amount" ? "That amount will not do." : "No such account."; | |
| 126 | ctx.toast(`${name.trim()}: $${done.value.dailyUsd?.toFixed(2)} a day, $${done.value.perRunUsd?.toFixed(2)} a run.`); | |
| 127 | return undefined; | |
| 128 | }, | |
| 129 | }), | |
| 130 | ), | |
| 131 | }); | |
| 132 | } |
src/web/pages/accounts.ts +5 −1
| @@ -42,7 +42,7 @@ import { | ||
| 42 | 42 | userLink, |
| 43 | 43 | } from "../ui.ts"; |
| 44 | 44 | import { sessionsCard } from "./account.ts"; |
| 45 | import { grantCard } from "./model.ts"; | |
| 45 | import { ceilingCard, grantCard } from "./model.ts"; | |
| 46 | 46 | import { accountSections } from "./origins.ts"; |
| 47 | 47 | import { markOf } from "../../core/marks.ts"; |
| 48 | 48 | |
| @@ -257,6 +257,10 @@ function refuseRemove(username: string): Html { | ||
| 257 | 257 | export function personSignInsPage(ctx: Ctx): Response { |
| 258 | 258 | if (!ctx.auth.user) return signInFirst(ctx); |
| 259 | 259 | if (!ctx.auth.user.isAdmin) return notFoundPage(ctx); |
| 260 | const named = getUserByName(ctx.app.db, ctx.params.username ?? ""); | |
| 261 | // An organization has no sign-ins: its page is its spend ceilings. | |
| 262 | if (named?.kind === "org" && ctx.auth.user.kind === "user") | |
| 263 | return shell(ctx, ceilingCard(ctx, named, `/settings/accounts/${encodeURIComponent(named.username)}`)); | |
Low: this page shows only the "Spend ceilings" card, so nothing on it says which organization you are setting. Add a heading or a |
||
| 260 | 264 | const seen = accountSignIns(ctx.app, ctx.auth.user, ctx.params.username ?? ""); |
| 261 | 265 | // Yourself is your own settings; another admin, an agent, nobody: no page. |
| 262 | 266 | if (!seen.ok || seen.value.user.id === ctx.auth.user.id) |
src/web/pages/model.ts +66 −2
| @@ -6,7 +6,8 @@ | ||
| 6 | 6 | * card). A token is typed once and never shown again. |
| 7 | 7 | */ |
| 8 | 8 | |
| 9 | import { ADD_MESSAGES, addOwnModel, setAccountGrant } from "../../actions/model_grant.ts"; | |
| 9 | import { ADD_MESSAGES, addOwnModel, setAccountCeilings, setAccountGrant } from "../../actions/model_grant.ts"; | |
| 10 | import { CEILING_MAX_USD, dailyLeft, getAccountCeiling } from "../../core/account_ceiling.ts"; | |
| 10 | 11 | import { |
| 11 | 12 | budgetLine, |
| 12 | 13 | getModelGrant, |
| @@ -88,10 +89,43 @@ export async function modelAdd(ctx: Ctx): Promise<Response> { | ||
| 88 | 89 | ); |
| 89 | 90 | } |
| 90 | 91 | |
| 92 | /** The admin's card for an account's spend ceilings, on a person's or an organization's page. */ | |
| 93 | export function ceilingCard(ctx: Ctx, user: User, at: string): Html { | |
| 94 | const ceiling = getAccountCeiling(ctx.app.db, user.id); | |
| 95 | const left = dailyLeft(ctx.app.db, user.id, new Date()); | |
| 96 | return card( | |
| 97 | html`<form class="stack" method="post" action="${at}/ceiling"> | |
| 98 | <div class="field"> | |
| 99 | <label for="f-daily">Daily ceiling, dollars</label> | |
| 100 | <input class="input" id="f-daily" name="daily" inputmode="decimal" | |
| 101 | autocomplete="off" value="${ceiling === undefined ? "" : ceiling.dailyUsd.toFixed(2)}" /> | |
| 102 | </div> | |
| 103 | <div class="field"> | |
| 104 | <label for="f-per-run">Per-run ceiling, dollars</label> | |
| 105 | <input class="input" id="f-per-run" name="perRun" inputmode="decimal" | |
| 106 | autocomplete="off" value="${ceiling === undefined ? "" : ceiling.perRunUsd.toFixed(2)}" /> | |
| 107 | <span class="hint">Up to $${CEILING_MAX_USD}. Both empty takes them back. A run that would pass | |
| 108 | either does not start, or stops at it; apart from the forge-wide | |
| 109 | ceiling.</span> | |
| 110 | </div> | |
| 111 | <div>${btn("Save", { variant: "primary", size: "sm" })}</div> | |
| 112 | </form>`, | |
| 113 | { | |
| 114 | head: html`<h2>Spend ceilings</h2> | |
| 115 | <p class="muted" id="ceiling-line"> | |
| 116 | ${ceiling === undefined || left === undefined | |
| 117 | ? "No ceiling set." | |
| 118 | : `$${ceiling.dailyUsd.toFixed(2)} a day, $${ceiling.perRunUsd.toFixed(2)} a run; $${left.toFixed(2)} left today.`} | |
| 119 | </p>`, | |
| 120 | }, | |
| 121 | ); | |
| 122 | } | |
| 123 | ||
| 91 | 124 | /** The admin's card on a person's page: what they have, and the grant form. */ |
| 92 | 125 | export function grantCard(ctx: Ctx, user: User, at: string): Html { |
| 93 | 126 | const grant = getModelGrant(ctx.app.db, user.id); |
| 94 | return card( | |
| 127 | const ceilings = ceilingCard(ctx, user, at); | |
| 128 | const grantForm = card( | |
| 95 | 129 | html`<form class="stack" method="post" action="${at}/model"> |
| 96 | 130 | <div class="field"> |
| 97 | 131 | <label for="f-grant">Grant, dollars a month</label> |
| @@ -108,6 +142,7 @@ export function grantCard(ctx: Ctx, user: User, at: string): Html { | ||
| 108 | 142 | <p class="muted">${budgetLine(ctx.app.db, user, new Date())}</p>`, |
| 109 | 143 | }, |
| 110 | 144 | ); |
| 145 | return html`${grantForm}${ceilings}`; | |
| 111 | 146 | } |
| 112 | 147 | |
| 113 | 148 | // POST /settings/accounts/:username/model |
| @@ -138,3 +173,32 @@ export async function grantAct(ctx: Ctx): Promise<Response> { | ||
| 138 | 173 | : `${username} has $${done.value.usdPerMonth.toFixed(2)} a month.`, |
| 139 | 174 | }); |
| 140 | 175 | } |
| 176 | ||
| 177 | // POST /settings/accounts/:username/ceiling | |
| 178 | export async function ceilingAct(ctx: Ctx): Promise<Response> { | |
| 179 | if (!ctx.auth.user) return signInFirst(ctx); | |
| 180 | if (!ctx.auth.user.isAdmin) return notFoundPage(ctx); | |
| 181 | const form = await readForm(ctx, "admin"); | |
| 182 | if (form instanceof Response) return form; | |
| 183 | const username = ctx.params.username ?? ""; | |
| 184 | // Both fields empty takes the ceilings back. | |
| 185 | const amount = (name: string): number | null => { | |
| 186 | const raw = field(form, name).trim().replace(/^\$/, ""); | |
| 187 | if (raw === "") return null; | |
| 188 | return /^\d{1,6}(\.\d{1,2})?$/.test(raw) ? Number(raw) : NaN; | |
| 189 | }; | |
| 190 | const done = setAccountCeilings(ctx.app, ctx.auth.user, username, amount("daily"), amount("perRun")); | |
| 191 | const back = `/settings/accounts/${encodeURIComponent(username)}`; | |
| 192 | if (!done.ok) { | |
| 193 | if (done.error === "not_found") return notFoundPage(ctx); | |
| 194 | return flashTo(ctx, back, { | |
| 195 | tone: "danger", | |
| 196 | message: `A ceiling is dollars, from 0.01 to ${CEILING_MAX_USD}.`, | |
| 197 | }); | |
| 198 | } | |
| 199 | if (done.value.dailyUsd === null || done.value.perRunUsd === null) | |
| 200 | return flashTo(ctx, back, { message: `${username}'s ceilings are taken back.` }); | |
| 201 | return flashTo(ctx, back, { | |
| 202 | message: `${username} has $${done.value.dailyUsd.toFixed(2)} a day and $${done.value.perRunUsd.toFixed(2)} a run.`, | |
| 203 | }); | |
| 204 | } |
src/web/pages/org.ts +7
| @@ -362,6 +362,13 @@ export async function ownerProfile(ctx: Ctx): Promise<Response> { | ||
| 362 | 362 | })} |
| 363 | 363 | </div>`, |
| 364 | 364 | )} |
| 365 | ${when( | |
| 366 | isOrg && ctx.auth.user?.isAdmin === true && ctx.auth.user.kind === "user", | |
| 367 | () => | |
| 368 | html`<p class="mt small"> | |
| 369 | <a href="/settings/accounts/${encodeURIComponent(owner.username)}">Model spend ceilings</a> | |
| 370 | </p>`, | |
| 371 | )} | |
| 365 | 372 | ${isOrg ? orgSecrets(ctx, owner.username) : null}`; |
| 366 | 373 | |
| 367 | 374 | const listed = listReposByOwner(ctx.app.db, owner.username, undefined, { |
test/account-budget.test.ts added +190
| @@ -0,0 +1,190 @@ | ||
| 1 | /** | |
| 2 | * Each account has its own daily and per-run model spend ceiling, checked | |
| 3 | * before a run starts (#897). | |
| 4 | */ | |
| 5 | ||
| 6 | import { afterAll, beforeAll, describe, expect, test } from "bun:test"; | |
| 7 | import { existsSync, writeFileSync } from "node:fs"; | |
| 8 | import { join } from "node:path"; | |
| 9 | import type { App } from "../src/app.ts"; | |
| 10 | import type { Repo } from "../src/core/repos.ts"; | |
| 11 | import type { User } from "../src/core/users.ts"; | |
| 12 | import { addToCrew } from "../src/actions/crew.ts"; | |
| 13 | import { addOwnModel, setAccountCeilings } from "../src/actions/model_grant.ts"; | |
| 14 | import { commitToBranch, createRepo } from "../src/actions/repos.ts"; | |
| 15 | import { assignIssue, openIssue } from "../src/actions/issues.ts"; | |
| 16 | import { createOrg } from "../src/core/orgs.ts"; | |
| 17 | import { createUser, getUserByName } from "../src/core/users.ts"; | |
| 18 | import { recordUsage } from "../src/core/usage.ts"; | |
| 19 | import { | |
| 20 | checkCeiling, | |
| 21 | dailyLeft, | |
| 22 | getAccountCeiling, | |
| 23 | } from "../src/core/account_ceiling.ts"; | |
| 24 | import { PLATFORM } from "../src/core/subscriptions.ts"; | |
| 25 | import { readRunLog } from "../src/runs/log.ts"; | |
| 26 | import { | |
| 27 | agentRunSettled, | |
| 28 | startStandInCrew, | |
| 29 | stopStandInCrew, | |
| 30 | type StandInCrew, | |
| 31 | } from "./helpers/crew-standin.ts"; | |
| 32 | ||
| 33 | describe("per-account spend ceilings", () => { | |
| 34 | let standIn: StandInCrew; | |
| 35 | let app: App; | |
| 36 | let admin: User; | |
| 37 | let bot: User; | |
| 38 | let n = 0; | |
| 39 | ||
| 40 | beforeAll(async () => { | |
| 41 | standIn = await startStandInCrew(); | |
| 42 | ({ app, alice: admin, bot } = standIn); | |
| 43 | }, 30_000); | |
| 44 | afterAll(() => stopStandInCrew(standIn)); | |
| 45 | ||
| 46 | async function person(name: string): Promise<{ user: User; repo: Repo }> { | |
| 47 | const made = await createUser(app.db, { | |
| 48 | trusted: true, | |
| 49 | username: name, | |
| 50 | email: `${name}@forge.local`, | |
| 51 | password: "a-good-passphrase", | |
| 52 | }); | |
| 53 | if (!made.ok) throw new Error(made.error); | |
| 54 | const user = made.value; | |
| 55 | const r = await createRepo(app, user, { name: "yard", visibility: "private" }); | |
| 56 | if (!r.ok) throw new Error(String(r.error)); | |
| 57 | const seeded = await commitToBranch(app, user, r.value, { | |
| 58 | message: "Start", | |
| 59 | files: [{ path: "README.md", content: "# yard\n" }], | |
| 60 | }); | |
| 61 | if (!seeded.ok) throw new Error(JSON.stringify(seeded.error)); | |
| 62 | expect(addToCrew(app, user, r.value, "bot").ok).toBe(true); | |
| 63 | expect(addOwnModel(app, user, { token: "sk-ant-api03-ownkey" }).ok).toBe(true); | |
| 64 | return { user, repo: r.value }; | |
| 65 | } | |
| 66 | ||
| 67 | async function runOn(user: User, repo: Repo) { | |
| 68 | const issue = openIssue(app, user, repo, { title: `Job ${++n}`, body: "Soon." }); | |
| 69 | if (!issue.ok) throw new Error(String(issue.error)); | |
| 70 | expect(assignIssue(app, user, repo, issue.value, bot).ok).toBe(true); | |
| 71 | return agentRunSettled(app, repo, (r) => r.issueId === issue.value.id); | |
| 72 | } | |
| 73 | ||
| 74 | function spend(repo: Repo, usd: number): void { | |
| 75 | recordUsage(app.db, { | |
| 76 | runId: null, | |
| 77 | repoId: repo.id, | |
| 78 | issueId: null, | |
| 79 | agentId: null, | |
| 80 | subscriptionId: PLATFORM, | |
| 81 | tokensIn: 1, | |
| 82 | tokensOut: 1, | |
| 83 | costUsd: usd, | |
| 84 | wallMs: 1, | |
| 85 | outcome: "ok", | |
| 86 | at: new Date().toISOString(), | |
| 87 | } as never); | |
| 88 | } | |
| 89 | ||
| 90 | test("an account nobody set has no ceiling, an admin's own crew included", async () => { | |
| 91 | const { user } = await person("dee"); | |
| 92 | expect(getAccountCeiling(app.db, user.id)).toBeUndefined(); | |
| 93 | expect(dailyLeft(app.db, user.id, new Date())).toBeUndefined(); | |
| 94 | expect(checkCeiling(app.db, user.id, 0.25, new Date())).toEqual({ ok: true, capUsd: undefined }); | |
| 95 | // The admin owns repositories too; a spend far past any default does not stop them. | |
| 96 | const mine = await createRepo(app, admin, { name: "crewwork", visibility: "private" }); | |
| 97 | if (!mine.ok) throw new Error(String(mine.error)); | |
| 98 | spend(mine.value, 500); | |
| 99 | expect(checkCeiling(app.db, admin.id, 0.25, new Date())).toEqual({ ok: true, capUsd: undefined }); | |
| 100 | }); | |
| 101 | ||
| 102 | test("an admin sets an organization's ceiling, and it is what the runner checks", () => { | |
| 103 | const org = createOrg(app.db, "yardorg", admin.id); | |
| 104 | if (!org.ok) throw new Error(String(org.error)); | |
| 105 | expect(setAccountCeilings(app, admin, "yardorg", 9, 2).ok).toBe(true); | |
| 106 | const id = getUserByName(app.db, "yardorg")!.id; | |
| 107 | expect(checkCeiling(app.db, id, 0.25, new Date())).toEqual({ ok: true, capUsd: 2 }); | |
| 108 | }); | |
| 109 | ||
| 110 | test("an account at its daily ceiling cannot start a run; another's spend does not count", async () => { | |
| 111 | const a = await person("ann"); | |
| 112 | const b = await person("bob"); | |
| 113 | expect(setAccountCeilings(app, admin, "ann", 3, 2).ok).toBe(true); | |
| 114 | spend(a.repo, 3); | |
| 115 | expect(dailyLeft(app.db, a.user.id, new Date())).toBe(0); | |
| 116 | const stopped = await runOn(a.user, a.repo); | |
| 117 | expect(stopped.state).toBe("failed"); | |
| 118 | expect(stopped.summary).toContain("daily ceiling"); | |
| 119 | expect(existsSync(standIn.argsFile)).toBe(false); | |
| 120 | // Bob is untouched by Ann's spend. | |
| 121 | expect(dailyLeft(app.db, b.user.id, new Date())).toBeUndefined(); | |
| 122 | const ran = await runOn(b.user, b.repo); | |
| 123 | expect(ran.summary).not.toContain("ceiling"); | |
| 124 | }, 90_000); | |
| 125 | ||
| 126 | test("a run is capped at the per-run ceiling, or refused when that is too small", async () => { | |
| 127 | const { user, repo } = await person("pru"); | |
| 128 | expect(setAccountCeilings(app, admin, "pru", 50, 1.5).ok).toBe(true); | |
| 129 | expect(checkCeiling(app.db, user.id, 0.25, new Date())).toEqual({ ok: true, capUsd: 1.5 }); | |
This tests |
||
| 130 | spend(repo, 49.5); // 0.50 left of the day: the day's rest is the cap | |
| 131 | expect(checkCeiling(app.db, user.id, 0.25, new Date())).toEqual({ ok: true, capUsd: 0.5 }); | |
| 132 | expect(setAccountCeilings(app, admin, "pru", 50, 0.1).ok).toBe(true); | |
| 133 | const v = checkCeiling(app.db, user.id, 0.25, new Date()); | |
| 134 | expect(v.ok).toBe(false); | |
| 135 | if (!v.ok) expect(v.said).toContain("per-run ceiling"); | |
| 136 | const run = await runOn(user, repo); | |
| 137 | expect(run.summary).toContain("per-run ceiling"); | |
| 138 | expect(readRunLog(standIn.dataDir, run.id).lines.join("\n")).toContain("per-run ceiling"); | |
| 139 | }, 90_000); | |
| 140 | ||
| 141 | test("the agent is handed the per-run ceiling as its budget, below the stand-in's own", async () => { | |
| 142 | const { user, repo } = await person("cap"); | |
| 143 | expect(setAccountCeilings(app, admin, "cap", 50, 0.6).ok).toBe(true); | |
| 144 | const run = await runOn(user, repo); | |
| 145 | expect(run.summary).not.toContain("ceiling"); | |
| 146 | expect(readRunLog(standIn.dataDir, run.id).lines.join("\n")).toContain("claude: budget $0.60"); | |
| 147 | }, 90_000); | |
| 148 | ||
| 149 | test("only an admin sets ceilings, amounts are validated, and who set them is recorded", async () => { | |
| 150 | const { user } = await person("val"); | |
| 151 | expect(setAccountCeilings(app, user, "val", 5, 1).ok).toBe(false); | |
| 152 | for (const bad of [0, -1, NaN, Infinity, 1e9]) { | |
| 153 | expect(setAccountCeilings(app, admin, "val", bad, 1).ok).toBe(false); | |
| 154 | expect(setAccountCeilings(app, admin, "val", 5, bad).ok).toBe(false); | |
| 155 | } | |
| 156 | expect(setAccountCeilings(app, admin, "nobody", 5, 1).ok).toBe(false); | |
| 157 | expect(getAccountCeiling(app.db, user.id)).toBeUndefined(); | |
| 158 | expect(setAccountCeilings(app, admin, "val", 7.5, 2).ok).toBe(true); | |
| 159 | const event = app.db | |
| 160 | .query<{ payload: string; actor_id: string }, []>( | |
| 161 | "SELECT payload, actor_id FROM event WHERE kind = 'account.ceiling_set' ORDER BY id DESC LIMIT 1", | |
| 162 | ) | |
| 163 | .get()!; | |
| 164 | expect(event.actor_id).toBe(admin.id); | |
| 165 | expect(JSON.parse(event.payload)).toEqual({ username: "val", dailyUsd: 7.5, perRunUsd: 2 }); | |
| 166 | const row = app.db | |
| 167 | .query<{ set_by: string }, [string]>("SELECT set_by FROM account_ceiling WHERE user_id = ?1") | |
| 168 | .get(user.id)!; | |
| 169 | expect(row.set_by).toBe(admin.id); | |
| 170 | }); | |
| 171 | ||
| 172 | test("a run under way stops when the day's ceiling is spent, and says so", async () => { | |
| 173 | const { user, repo } = await person("tick"); | |
| 174 | expect(setAccountCeilings(app, admin, "tick", 4, 4).ok).toBe(true); | |
| 175 | const issue = openIssue(app, user, repo, { title: "Slow job", body: "GRANT-TICK until told." }); | |
| 176 | if (!issue.ok) throw new Error(String(issue.error)); | |
| 177 | expect(assignIssue(app, user, repo, issue.value, bot).ok).toBe(true); | |
| 178 | const settled = agentRunSettled(app, repo, (r) => r.issueId === issue.value.id); | |
| 179 | for (let i = 0; i < 200 && !existsSync(join(standIn.dataDir, "ticked")); i++) | |
| 180 | await Bun.sleep(100); | |
| 181 | expect(existsSync(join(standIn.dataDir, "ticked"))).toBe(true); | |
| 182 | expect(dailyLeft(app.db, user.id, new Date())).toBe(4); | |
| 183 | spend(repo, 4); | |
| 184 | writeFileSync(join(standIn.dataDir, "spent"), ""); | |
| 185 | const run = await settled; | |
| 186 | expect(run.state).toBe("failed"); | |
| 187 | expect(run.summary).toContain("daily ceiling is spent"); | |
| 188 | expect(run.summary).toContain("the run stopped"); | |
| 189 | }, 90_000); | |
| 190 | }); |
test/mcp.test.ts +29
| @@ -16,6 +16,7 @@ import { createApp as insertApp } from "../src/core/apps.ts"; | ||
| 16 | 16 | import { listIssues } from "../src/core/issues.ts"; |
| 17 | 17 | import { getRepoById, updateRepoSettings, type Repo } from "../src/core/repos.ts"; |
| 18 | 18 | import { createRun, listRuns, setRunBase } from "../src/core/runs.ts"; |
| 19 | import { getAccountCeiling } from "../src/core/account_ceiling.ts"; | |
| 19 | 20 | import { createToken } from "../src/core/tokens.ts"; |
| 20 | 21 | import { createUser, type User } from "../src/core/users.ts"; |
| 21 | 22 | import { savePlacement } from "../src/core/placements.ts"; |
| @@ -178,6 +179,7 @@ describe("the forge over MCP", () => { | ||
| 178 | 179 | "health", |
| 179 | 180 | "audit", |
| 180 | 181 | "model_grant", |
| 182 | "model_ceiling", | |
| 181 | 183 | "outsiders", |
| 182 | 184 | "decisions_by", |
| 183 | 185 | "probe", |
| @@ -856,4 +858,31 @@ describe("the forge over MCP", () => { | ||
| 856 | 858 | expect(back.value).toEqual({ kinds: [], people: [] }); |
| 857 | 859 | expect(await kinds()).toContain("release"); |
| 858 | 860 | }); |
| 861 | ||
| 862 | test("model_ceiling: dry by default, an admin person with an admin token sets and clears it", async () => { | |
| 863 | const admin = createToken(app.db, alice.id, "admin", "admin").token; | |
| 864 | const readOnly = createToken(app.db, alice.id, "ro", "ro").token; | |
| 865 | const target = (await createUser(app.db, { trusted: true, username: "ceilee", email: "[email protected]", password: "a-good-passphrase" })); | |
| 866 | if (!target.ok) throw new Error(target.error); | |
| 867 | const id = target.value.id; | |
| 868 | const args = { username: "ceilee", dailyUsd: 12, perRunUsd: 3 }; | |
| 869 | // A non-admin is refused. | |
| 870 | expect((await tool(bobRw, "model_ceiling", { ...args, dryRun: false })).error).toContain("forbidden"); | |
| 871 | // A dry run changes nothing. | |
| 872 | expect((await tool(admin, "model_ceiling", args)).value.says).toContain("nothing changed"); | |
| 873 | expect(getAccountCeiling(app.db, id)).toBeUndefined(); | |
| 874 | // A read-only token is refused. | |
| 875 | expect((await tool(readOnly, "model_ceiling", { ...args, dryRun: false })).error).toContain("forbidden"); | |
| 876 | expect(getAccountCeiling(app.db, id)).toBeUndefined(); | |
| 877 | // Bad amounts are invalid; half a clear too. | |
| 878 | for (const bad of [{ dailyUsd: 0 }, { dailyUsd: 1e9 }, { perRunUsd: null }]) | |
| 879 | expect((await tool(admin, "model_ceiling", { ...args, ...bad, dryRun: false })).error).toContain("invalid"); | |
| 880 | expect(getAccountCeiling(app.db, id)).toBeUndefined(); | |
| 881 | expect((await tool(admin, "model_ceiling", { ...args, dryRun: false })).value).toEqual({ username: "ceilee", dailyUsd: 12, perRunUsd: 3 }); | |
| 882 | expect(getAccountCeiling(app.db, id)).toEqual({ dailyUsd: 12, perRunUsd: 3 }); | |
| 883 | // Both null takes them back. | |
| 884 | const cleared = await tool(admin, "model_ceiling", { username: "ceilee", dailyUsd: null, perRunUsd: null, dryRun: false }); | |
| 885 | expect(cleared.value).toEqual({ username: "ceilee", dailyUsd: null, perRunUsd: null }); | |
| 886 | expect(getAccountCeiling(app.db, id)).toBeUndefined(); | |
| 887 | }); | |
| 859 | 888 | }); |
test/model-grant-pages.test.ts +76
| @@ -11,6 +11,8 @@ import { join } from "node:path"; | ||
| 11 | 11 | import { createApp, type App } from "../src/app.ts"; |
| 12 | 12 | import { getModelGrant } from "../src/core/model_grant.ts"; |
| 13 | 13 | import { createUser, getUserByName } from "../src/core/users.ts"; |
| 14 | import { createOrg } from "../src/core/orgs.ts"; | |
| 15 | import { getAccountCeiling } from "../src/core/account_ceiling.ts"; | |
| 14 | 16 | import { createToken } from "../src/core/tokens.ts"; |
| 15 | 17 | import { serve, type Servers } from "../src/http/server.ts"; |
| 16 | 18 | import { quietLogs } from "./helpers/globals.ts"; |
| @@ -114,4 +116,78 @@ describe("a model grant, from the account's page and the API", () => { | ||
| 114 | 116 | expect((await post(tok.token, null)).status).toBe(200); |
| 115 | 117 | expect(getModelGrant(app.db, carol.id)).toBeUndefined(); |
| 116 | 118 | }); |
| 119 | ||
| 120 | test("the ceiling route: signed out, a person's 404, a bad amount's flash, saved; the card shows it", async () => { | |
| 121 | const url = `${servers.url}/settings/accounts/carol/ceiling`; | |
| 122 | const send = (cookie: string | undefined, daily: string, perRun: string) => | |
| 123 | fetch(url, { | |
| 124 | method: "POST", | |
| 125 | headers: cookie === undefined ? {} : { cookie }, | |
| 126 | body: new URLSearchParams({ daily, perRun }), | |
| 127 | redirect: "manual", | |
| 128 | }); | |
| 129 | const carol = getUserByName(app.db, "carol")!; | |
| 130 | const alice = await signIn("alice"); | |
| 131 | const bob = await signIn("bob"); | |
| 132 | expect((await send(undefined, "5", "1")).status).not.toBe(200); | |
| 133 | expect(getAccountCeiling(app.db, carol.id)).toBeUndefined(); | |
| 134 | expect((await send(bob, "5", "1")).status).toBe(404); | |
| 135 | expect(getAccountCeiling(app.db, carol.id)).toBeUndefined(); | |
| 136 | const bad = await send(alice, "abc", "1"); | |
| 137 | expect(bad.status).toBe(303); | |
| 138 | expect(getAccountCeiling(app.db, carol.id)).toBeUndefined(); | |
| 139 | const carried = (bad.headers.get("set-cookie") ?? "").split(";")[0]!; | |
| 140 | const flash = await (await fetch(`${servers.url}/settings/accounts/carol`, { | |
| 141 | headers: { cookie: `${alice}; ${carried}` }, | |
| 142 | })).text(); | |
| 143 | expect(flash).toContain("A ceiling is dollars"); | |
| 144 | expect((await send(alice, "12.5", "3")).status).toBe(303); | |
| 145 | expect(getAccountCeiling(app.db, carol.id)).toEqual({ dailyUsd: 12.5, perRunUsd: 3 }); | |
| 146 | const page = await (await fetch(`${servers.url}/settings/accounts/carol`, { | |
| 147 | headers: { cookie: alice }, | |
| 148 | })).text(); | |
| 149 | expect(page).toContain("$12.50 a day, $3.00 a run; $12.50 left today."); | |
| 150 | // Both fields empty takes them back; one empty is refused. | |
| 151 | expect((await send(alice, "5", "")).status).toBe(303); | |
| 152 | expect(getAccountCeiling(app.db, carol.id)).toEqual({ dailyUsd: 12.5, perRunUsd: 3 }); | |
| 153 | expect((await send(alice, "", "")).status).toBe(303); | |
| 154 | expect(getAccountCeiling(app.db, carol.id)).toBeUndefined(); | |
| 155 | const ev = app.db | |
| 156 | .query<{ payload: string }, []>("SELECT payload FROM event WHERE kind = 'account.ceiling_set' ORDER BY id DESC LIMIT 1") | |
| 157 | .get()!; | |
| 158 | expect(JSON.parse(ev.payload)).toEqual({ username: "carol", dailyUsd: null, perRunUsd: null }); | |
| 159 | }); | |
| 160 | ||
| 161 | test("an organization has a ceiling page and an API door; a person's token is refused", async () => { | |
| 162 | const alice = getUserByName(app.db, "alice")!; | |
| 163 | const org = createOrg(app.db, "yardco", alice.id); | |
| 164 | if (!org.ok) throw new Error(String(org.error)); | |
| 165 | const cookie = await signIn("alice"); | |
| 166 | const page = await fetch(`${servers.url}/settings/accounts/yardco`, { headers: { cookie } }); | |
| 167 | expect(page.status).toBe(200); | |
| 168 | expect(await page.text()).toContain("No ceiling set."); | |
| 169 | const bobTok = createToken(app.db, getUserByName(app.db, "bob")!.id, "t", "admin"); | |
| 170 | const tok = createToken(app.db, alice.id, "t", "admin"); | |
| 171 | const post = (token: string, body: unknown, name = "yardco") => | |
| 172 | fetch(`${servers.url}/api/v1/admin/accounts/${name}/ceiling`, { | |
| 173 | method: "POST", | |
| 174 | headers: { authorization: `Bearer ${token}`, "content-type": "application/json" }, | |
| 175 | body: JSON.stringify(body), | |
| 176 | }); | |
| 177 | expect((await post(bobTok.token, { dailyUsd: 5, perRunUsd: 1 })).status).toBe(404); | |
| 178 | expect((await post(tok.token, { dailyUsd: "5", perRunUsd: 1 })).status).toBe(422); | |
| 179 | expect((await post(tok.token, { dailyUsd: 0, perRunUsd: 1 })).status).toBe(422); | |
| 180 | expect((await post(tok.token, { dailyUsd: 5, perRunUsd: 1 }, "nobody")).status).toBe(404); | |
| 181 | expect((await post(tok.token, { dailyUsd: 30, perRunUsd: 8 })).status).toBe(200); | |
| 182 | const yardco = getUserByName(app.db, "yardco")!; | |
| 183 | expect(getAccountCeiling(app.db, yardco.id)).toEqual({ dailyUsd: 30, perRunUsd: 8 }); | |
| 184 | expect((await post(tok.token, { dailyUsd: null, perRunUsd: 8 })).status).toBe(422); | |
| 185 | expect((await post(tok.token, { dailyUsd: null, perRunUsd: null })).status).toBe(200); | |
| 186 | expect(getAccountCeiling(app.db, yardco.id)).toBeUndefined(); | |
| 187 | // The org's own page links to the ceilings; a person who is not an admin sees no link. | |
| 188 | const orgPage = await (await fetch(`${servers.url}/yardco`, { headers: { cookie } })).text(); | |
| 189 | expect(orgPage).toContain('href="/settings/accounts/yardco"'); | |
| 190 | const bobOrg = await (await fetch(`${servers.url}/yardco`, { headers: { cookie: await signIn("bob") } })).text(); | |
| 191 | expect(bobOrg).not.toContain("/settings/accounts/yardco"); | |
| 192 | }); | |
| 117 | 193 | }); |
test/tui-model.test.ts +28
| @@ -7,6 +7,7 @@ import { mkdtemp, rm } from "node:fs/promises"; | ||
| 7 | 7 | import { tmpdir } from "node:os"; |
| 8 | 8 | import { join } from "node:path"; |
| 9 | 9 | import { createApp, type App as Forge } from "../src/app.ts"; |
| 10 | import { getAccountCeiling } from "../src/core/account_ceiling.ts"; | |
| 10 | 11 | import { getModelGrant, ownSubscription } from "../src/core/model_grant.ts"; |
| 11 | 12 | import { createUser, getUserByName } from "../src/core/users.ts"; |
| 12 | 13 | import { createForgeApp } from "../src/tui/forge.ts"; |
| @@ -109,4 +110,31 @@ describe("model budget in the terminal", () => { | ||
| 109 | 110 | expect(p.text()).not.toContain("give an account dollars"); |
| 110 | 111 | p.stop(); |
| 111 | 112 | }); |
| 113 | ||
| 114 | test("an admin sets an account's ceilings in the terminal; a person has no such command", async () => { | |
| 115 | const bob = getUserByName(forge.db, "bob")!; | |
| 116 | const h = boot("alice"); | |
| 117 | await open(h, "Model ceiling"); | |
| 118 | await h.type("bob"); | |
| 119 | await h.press("enter"); | |
| 120 | await h.type("20 5"); | |
| 121 | await h.press("enter"); | |
| 122 | expect(getAccountCeiling(forge.db, bob.id)).toEqual({ dailyUsd: 20, perRunUsd: 5 }); | |
| 123 | await open(h, "Model ceiling"); | |
| 124 | await h.type("bob"); | |
| 125 | await h.press("enter"); | |
| 126 | expect(h.text()).toContain("now $20.00 a day, $5.00 a run"); | |
| 127 | await h.press("enter"); // a stray Enter takes nothing back | |
| 128 | expect(getAccountCeiling(forge.db, bob.id)).toEqual({ dailyUsd: 20, perRunUsd: 5 }); | |
| 129 | await h.type("none"); | |
| 130 | await h.press("enter"); | |
| 131 | expect(getAccountCeiling(forge.db, bob.id)).toBeUndefined(); | |
| 132 | h.stop(); | |
| 133 | ||
| 134 | const p = boot("bob"); | |
| 135 | await p.ctrl("k"); | |
| 136 | await p.type("Model ceiling"); | |
| 137 | expect(p.text()).not.toContain("set an account's dollars"); | |
| 138 | p.stop(); | |
| 139 | }); | |
| 112 | 140 | }); |
No test fails if this term is removed. Assert that the budget handed to the agent is the per-run ceiling: the
claude: budget $…log line or--max-budgetin the stand-in's args file.