trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns15g rEventsFlows

/talk says what each turn cost and how much context is left #8846

Merged scoutcrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/8822-talk-says-what-each-turn-cost-and-how-mu → main 3 commits fork 2 threads · 2 unresolved

  • Merged. scout/8822-talk-says-what-each-turn-cost-and-how-mu was merged into main as edaa486.

Review

Sign in to review this change.
16 files changed +290 −53
docs/design/areas/talk.md +2
@@ -18,3 +18,5 @@
1818 - `.talk-copy` — a quiet text button that copies a message or a code block.
1919 - `.talk-suggest` — the starter questions on an empty chat; each fills the composer.
2020 - `.talk-jump` — "Latest" button, shown when you have scrolled up from the end.
21
22 Under a reply, `.talk-cost` is one quiet line with the turn's dollars; above the log, `.talk-context` says how full the context is and, from 80 %, suggests a new conversation. Both are `small muted` text, no new colour.
docs/features/talk.md +4 −1
@@ -45,11 +45,14 @@ Prior art read: Vercel AI Elements (Apache-2.0 React on shadcn; anatomy and beha
4545 | Stick to bottom, "Latest" jump button | this pull | AI Elements `Conversation` |
4646 | Many conversations: new, list, search, rename, archive, URL each | next pull | needs a migration: today one row per person |
4747 | Edit and resend, regenerate, Stop, queue | next pulls | need turn deletion and a cancel on `crew/conversation.ts` |
48 | Attachments, cost and context left, tables, thinking, plan line | later | need storage / token data / markdown tables |
48 | Cost under each reply and how full the context is (header, and in the terminal) | this pull | the run's own usage record, never an estimate |
49 | Attachments, tables, thinking, plan line | later | need storage / token data / markdown tables |
4950 | Code highlighting and tables in replies | this pull (#8816) | a reply is stored whole and drawn whole, so no half-open fence is ever tokenised |
5051
5152 **Without the script** the page is a plain form: Enter adds a line, starters are links that prefill the box, and Copy and Latest are not drawn (the script draws them, along with the "Enter sends" hint). The log is `role="log"`; with the script, a new message while you are scrolled up is announced in the status line.
5253
54 **Cost and context.** Under a finished reply a quiet line gives the turn's dollars (`$0.12 this turn`); above the log, "Context 85% full (170k of 200k tokens)". Both are read from the run's record (`conversation_run`: `cost_usd`, and `context_tokens`/`context_window` from the last message's usage and the result's `modelUsage`), shown by `core/conversation_cost.ts`. When a run names no window, 200k is assumed and the line says so by showing it. From 80 % the line adds that a new conversation answers faster and costs less (information only: there is no way to start another yet, see "Many conversations"). A reply that streams in gets both when the stream ends; the terminal prints the last turn's cost and the share after its confirmation line.
55
5356 **The terminal** (`ssh <forge> talk <words>`, `forge talk`) sends one line and prints the answer; it has no log, so who/when, copy, tool cards, starters and the jump button do not apply to it. The conversation's whole history is read through `GET /api/v1/conversation/turns`, where every turn carries its role and time.
5457
5558 ## Code, tables and Copy in a reply (#8816)
src/core/conversation_cost.ts added +56
@@ -0,0 +1,56 @@
1 /**
2 * What /talk's turns cost and how full the context is (#8822), read from the
3 * run's own usage record (`conversation_run`), never estimated.
4 */
5
6 import type { Database } from "bun:sqlite";
7
8 /** The window assumed when a run did not report its model's. */
9 export const CONTEXT_TOKENS = 200_000;
10 /** From this share of the context a new conversation is suggested. */
11 export const FULL_AT = 0.8;
12
13 /** The dollars of the run that wrote each agent turn, by turn id: on the last agent turn inside the run's span. */
14 export function turnCosts(db: Database, conversationId: string): Map<string, number> {
15 const rows = db
16 .query<{ id: string; cost: number }, [string]>(
17 `SELECT (SELECT t.id FROM conversation_turn t
18 WHERE t.conversation_id = r.conversation_id AND t.role = 'agent'
19 AND t.created_at >= r.started_at AND t.created_at <= r.finished_at
20 ORDER BY t.rowid DESC LIMIT 1) AS id, r.cost_usd AS cost
21 FROM conversation_run r
22 WHERE r.conversation_id = ?1 AND r.finished_at IS NOT NULL AND r.cost_usd > 0`,
23 )
24 .all(conversationId);
25 const costs = new Map<string, number>();
26 for (const r of rows) if (r.id !== null) costs.set(r.id, (costs.get(r.id) ?? 0) + r.cost);
27 return costs;
28 }
29
30 export interface Context {
31 readonly tokens: number;
32 /** The model's window as its run reported it, else `CONTEXT_TOKENS`. */
33 readonly window: number;
34 readonly share: number;
35 }
36
37 /** How full the context was at the end of the latest turn, or undefined before any usage. */
38 export function contextOf(db: Database, conversationId: string): Context | undefined {
39 const row = db
40 .query<{ n: number; w: number }, [string]>(
41 "SELECT context_tokens AS n, context_window AS w FROM conversation_run WHERE conversation_id = ?1 AND context_tokens > 0 ORDER BY started_at DESC, rowid DESC LIMIT 1",
42 )
43 .get(conversationId);
44 if (row === null) return undefined;
45 const window = row.w > 0 ? row.w : CONTEXT_TOKENS;
46 return { tokens: row.n, window, share: Math.min(1, row.n / window) };
47 }
48
49 const thousands = (n: number): string => (n >= 1_000_000 ? `${+(n / 1_000_000).toFixed(1)}M` : `${Math.round(n / 1000)}k`);
50
51 /** "Context 85% full (170k of 200k tokens)", with the nudge from `FULL_AT`. */
52 export const contextText = (c: Context): string =>
53 `Context ${Math.round(c.share * 100)}% full (${thousands(c.tokens)} of ${thousands(c.window)} tokens)${c.share >= FULL_AT ? ". A new conversation answers faster and costs less" : ""}`;
54
55 /** Dollars as a quiet figure: cents below a dollar, a tenth of a cent below that. */
56 export const dollars = (usd: number): string => (usd < 0.01 ? `$${usd.toFixed(3)}` : `$${usd.toFixed(2)}`);
src/core/migration-pins/20261103120000-conversation-run-context.pin added +1
@@ -0,0 +1,1 @@
1 564debf9410c3b1bbcaf093bbcaf3fe0af0bdb6ccc14b1b6c12ff3e9a8c0ce68
src/core/migrations/20261103120000-conversation-run-context.ts added +7
@@ -0,0 +1,7 @@
1 // What a /talk turn's session had in its context at the end (#8822): the last
2 // message's input tokens and the model's window, both from the run's own usage
3 // record; 0 when none came.
4 export default `
5 ALTER TABLE conversation_run ADD COLUMN context_tokens INTEGER NOT NULL DEFAULT 0;
6 ALTER TABLE conversation_run ADD COLUMN context_window INTEGER NOT NULL DEFAULT 0;
7 `;
src/crew/claude.ts +36 −9
@@ -98,6 +98,10 @@ export interface ClaudeOutcome {
9898 */
9999 readonly tokensIn: number;
100100 readonly tokensOut: number;
101 /** What the last assistant message read: the session's context at its end; 0 when none reported. */
102 readonly contextTokens: number;
103 /** The model's context window in tokens, from the result's `modelUsage`; 0 when it named none. */
104 readonly contextWindow: number;
101105 /** `completed`, `max_budget_reached`, `timeout`, `cancelled`, `error`, `exit <n>`. */
102106 readonly reason: string;
103107 }
@@ -283,6 +287,8 @@ export async function runClaude(input: ClaudeInput): Promise<ClaudeOutcome> {
283287 let turns = 0;
284288 let tokensIn = 0;
285289 let tokensOut = 0;
290 let contextTokens = 0;
291 let contextWindow = 0;
286292 let reason = "";
287293 let ok = false;
288294 let sawResult = false;
@@ -366,19 +372,24 @@ export async function runClaude(input: ClaudeInput): Promise<ClaudeOutcome> {
366372 input.onSaid({ kind: "tool", text: `${b["name"]} ${JSON.stringify(b["input"] ?? {})}` });
367373 }
junipercrew outdated

The usage parsing is now done twice on the same message: here and in the onUsage block below. Fold them into one read.
368374 }
369 if (input.onUsage && typeof message === "object" && message !== null) {
375 if (typeof message === "object" && message !== null) {
370376 const m = message as Record<string, unknown>;
371377 if (typeof m["usage"] === "object" && m["usage"] !== null) {
372378 const key =
373379 typeof m["id"] === "string" ? m["id"] : `#${perMessage.size}`;
374 perMessage.set(key, tokensOf(m["usage"]));
375 let tokensIn = 0;
376 let tokensOut = 0;
377 for (const t of perMessage.values()) {
378 tokensIn += t.tokensIn;
379 tokensOut += t.tokensOut;
380 const used = tokensOf(m["usage"]);
381 // What the last message read is the context the session holds.
382 contextTokens = used.tokensIn;
383 perMessage.set(key, used);
384 if (input.onUsage) {
385 let tokensIn = 0;
386 let tokensOut = 0;
387 for (const t of perMessage.values()) {
388 tokensIn += t.tokensIn;
389 tokensOut += t.tokensOut;
390 }
391 input.onUsage({ tokensIn, tokensOut, costUsd: 0 });
380392 }
381 input.onUsage({ tokensIn, tokensOut, costUsd: 0 });
382393 }
383394 }
384395 return;
@@ -389,6 +400,7 @@ export async function runClaude(input: ClaudeInput): Promise<ClaudeOutcome> {
389400 result = typeof e["result"] === "string" ? e["result"] : "";
390401 costUsd =
391402 typeof e["total_cost_usd"] === "number" ? e["total_cost_usd"] : 0;
403 contextWindow = windowOf(e["modelUsage"]);
392404 turns = typeof e["num_turns"] === "number" ? e["num_turns"] : 0;
393405 ({ tokensIn, tokensOut } = tokensOf(e["usage"]));
394406 input.onUsage?.({ tokensIn, tokensOut, costUsd });
@@ -487,6 +499,8 @@ export async function runClaude(input: ClaudeInput): Promise<ClaudeOutcome> {
487499 turns,
488500 tokensIn,
489501 tokensOut,
502 contextTokens,
503 contextWindow,
490504 reason: why,
491505 };
492506 }
@@ -500,10 +514,12 @@ export async function runClaude(input: ClaudeInput): Promise<ClaudeOutcome> {
500514 turns,
501515 tokensIn,
502516 tokensOut,
517 contextTokens,
518 contextWindow,
503519 reason: `exit ${code}`,
504520 };
505521 }
506 return { ok, sessionId, result, costUsd, turns, tokensIn, tokensOut, reason };
522 return { ok, sessionId, result, costUsd, turns, tokensIn, tokensOut, contextTokens, contextWindow, reason };
507523 }
508524
509525 /**
@@ -522,6 +538,17 @@ export function reapGroup(pgid: number): void {
522538 }
523539 }
524540
541 /** The largest `contextWindow` a result's `modelUsage` names, 0 when none. */
542 export function windowOf(modelUsage: unknown): number {
543 if (typeof modelUsage !== "object" || modelUsage === null) return 0;
544 let most = 0;
545 for (const v of Object.values(modelUsage)) {
546 const w = typeof v === "object" && v !== null ? (v as Record<string, unknown>)["contextWindow"] : undefined;
547 if (typeof w === "number" && Number.isFinite(w) && w > most) most = Math.floor(w);
548 }
549 return most;
550 }
551
525552 /**
526553 * The token counts of a result's `usage` object. Input is every token the
527554 * model read: `input_tokens + cache_creation_input_tokens +
src/crew/conversation.ts +42 −38
@@ -374,47 +374,51 @@ function begin(
374374 app.log.warn("conversation turn failed", { error: String(error) });
375375 } finally {
376376 unlisten();
377 }
378 // The turn is over for /talk only once its answer and its figures are stored.
379 try {
380 const cost = outcome?.costUsd ?? 0;
381 // Stopped for a restart: the row stays `running` for the next boot to
382 // find, keeping what the session had reported (Claude Code names the
383 // cost only in its result, so a cut turn keeps its tokens and the cost
384 // it had). Its token is revoked at boot (`resumeInterrupted`).
385 if (options.signal.aborted) {
386 app.db.query("UPDATE conversation_run SET tokens_in = ?2, tokens_out = ?3, cost_usd = ?4, context_tokens = ?5, context_window = ?6 WHERE id = ?1").run(runId, outcome?.tokensIn ?? 0, outcome?.tokensOut ?? 0, cost, outcome?.contextTokens ?? 0, outcome?.contextWindow ?? 0);
387 app.db.query("UPDATE subscription SET spent_usd = spent_usd + ?2 WHERE id = ?1").run(subscriptionId, cost);
388 return;
389 }
390 app.db.query("DELETE FROM token WHERE id = ?1").run(minted.value.id);
391 if (stopper.signal.aborted) {
392 // A write held after the stop was asked is cancelled too.
393 cancelHeld(app.db, via);
394 put("agent", STOPPED_LINE);
395 app.db
396 .query("UPDATE conversation_run SET state = 'failed', tokens_in = ?2, tokens_out = ?3, cost_usd = ?4, finished_at = ?5, context_tokens = ?6, context_window = ?7 WHERE id = ?1")
397 .run(runId, outcome?.tokensIn ?? 0, outcome?.tokensOut ?? 0, cost, new Date().toISOString(), outcome?.contextTokens ?? 0, outcome?.contextWindow ?? 0);
398 app.db.query("UPDATE subscription SET spent_usd = spent_usd + ?2 WHERE id = ?1").run(subscriptionId, cost);
399 return;
400 }
401 const answer = outcome?.result ?? "";
402 if (outcome !== undefined && isRateLimited(outcome)) {
403 // The provider's words are not shown: the streamed copy of them goes.
404 const first = outcome.result.split("\n").find((l) => l.trim() !== "")?.trim() ?? "";
405 for (const turn of streamed)
406 if (turn.text === outcome.result || (first !== "" && turn.text.includes(first))) removeTurn(app.db, userId, conversation.id, turn.id);
407 put("agent", limitLine(app, picked, outcome.result));
408 }
409 else if (answer.trim() !== "" && answer !== lastText) put("agent", answer);
410 else if (answer.trim() === "" && lastText === "") put("agent", "I could not answer that just now.");
411 app.db
412 .query(
413 "UPDATE conversation_run SET state = ?2, tokens_in = ?3, tokens_out = ?4, cost_usd = ?5, finished_at = ?6, context_tokens = ?7, context_window = ?8 WHERE id = ?1",
414 )
415 .run(runId, outcome?.ok ? "ok" : "failed", outcome?.tokensIn ?? 0, outcome?.tokensOut ?? 0, cost, new Date().toISOString(), outcome?.contextTokens ?? 0, outcome?.contextWindow ?? 0);
416 // Against the account's lifetime budget, like a run's cost (`subscription.spent_usd`).
417 app.db.query("UPDATE subscription SET spent_usd = spent_usd + ?2 WHERE id = ?1").run(subscriptionId, cost);
418 } finally {
377419 if (live.get(conversation.id) === runId) live.delete(conversation.id);
378420 if (stoppers.get(conversation.id)?.runId === runId) stoppers.delete(conversation.id);
379421 }
380 const cost = outcome?.costUsd ?? 0;
381 // Stopped for a restart: the row stays `running` for the next boot to
382 // find, keeping what the session had reported (Claude Code names the
383 // cost only in its result, so a cut turn keeps its tokens and the cost
384 // it had). Its token is revoked at boot (`resumeInterrupted`).
385 if (options.signal.aborted) {
386 app.db.query("UPDATE conversation_run SET tokens_in = ?2, tokens_out = ?3, cost_usd = ?4 WHERE id = ?1").run(runId, outcome?.tokensIn ?? 0, outcome?.tokensOut ?? 0, cost);
387 app.db.query("UPDATE subscription SET spent_usd = spent_usd + ?2 WHERE id = ?1").run(subscriptionId, cost);
388 return;
389 }
390 app.db.query("DELETE FROM token WHERE id = ?1").run(minted.value.id);
391 if (stopper.signal.aborted) {
392 // A write held after the stop was asked is cancelled too.
393 cancelHeld(app.db, via);
394 put("agent", STOPPED_LINE);
395 app.db
396 .query("UPDATE conversation_run SET state = 'failed', tokens_in = ?2, tokens_out = ?3, cost_usd = ?4, finished_at = ?5 WHERE id = ?1")
397 .run(runId, outcome?.tokensIn ?? 0, outcome?.tokensOut ?? 0, cost, new Date().toISOString());
398 app.db.query("UPDATE subscription SET spent_usd = spent_usd + ?2 WHERE id = ?1").run(subscriptionId, cost);
399 return;
400 }
401 const answer = outcome?.result ?? "";
402 if (outcome !== undefined && isRateLimited(outcome)) {
403 // The provider's words are not shown: the streamed copy of them goes.
404 const first = outcome.result.split("\n").find((l) => l.trim() !== "")?.trim() ?? "";
405 for (const turn of streamed)
406 if (turn.text === outcome.result || (first !== "" && turn.text.includes(first))) removeTurn(app.db, userId, conversation.id, turn.id);
407 put("agent", limitLine(app, picked, outcome.result));
408 }
409 else if (answer.trim() !== "" && answer !== lastText) put("agent", answer);
410 else if (answer.trim() === "" && lastText === "") put("agent", "I could not answer that just now.");
411 app.db
412 .query(
413 "UPDATE conversation_run SET state = ?2, tokens_in = ?3, tokens_out = ?4, cost_usd = ?5, finished_at = ?6 WHERE id = ?1",
414 )
415 .run(runId, outcome?.ok ? "ok" : "failed", outcome?.tokensIn ?? 0, outcome?.tokensOut ?? 0, cost, new Date().toISOString());
416 // Against the account's lifetime budget, like a run's cost (`subscription.spent_usd`).
417 app.db.query("UPDATE subscription SET spent_usd = spent_usd + ?2 WHERE id = ?1").run(subscriptionId, cost);
418422 })();
419423 return ok(turn);
420424 }
src/tui/ssh-talk.ts +10
@@ -10,6 +10,8 @@
1010 */
1111
1212 import { TALK_WORDS, stopTalk, talk } from "../actions/talk.ts";
13 import { contextOf, contextText, dollars, turnCosts } from "../core/conversation_cost.ts";
14 import { openConversation } from "../core/conversations.ts";
1315 import type { App } from "../app.ts";
1416 import type { User } from "../core/users.ts";
1517 import { CONTROL_WORDS } from "./ssh-tell.ts";
@@ -50,5 +52,13 @@ export function serveTalk(app: App, session: SshSession, user: User | undefined,
5052 session.write(
5153 `${said.value.how === "steered" ? "said to the forge's agent, mid-answer" : "the forge's agent is answering"}${CRLF}`,
5254 );
55 if (user !== undefined) {
56 // The figures of the turn before this one, from its own usage record.
57 const conversation = openConversation(app.db, user.id);
58 const last = [...turnCosts(app.db, conversation.id).values()].at(-1);
59 const context = contextOf(app.db, conversation.id);
60 if (last !== undefined) session.write(`last turn ${dollars(last)}${CRLF}`);
61 if (context !== undefined) session.write(`${contextText(context)}${CRLF}`);
62 }
5363 session.exit(0);
5464 }
src/web/pages/talk.ts +18 −3
@@ -14,6 +14,7 @@
1414 import { TALK_WORDS, talk } from "../../actions/talk.ts";
1515 import { REFUSED, sayNo, sayYes, waitingConfirms } from "../../actions/conversation_confirms.ts";
1616 import { DEFAULT_PAGE, listTurns, MAX_PAGE, openConversation, turnsAfter, type Turn } from "../../core/conversations.ts";
17 import { contextOf, contextText, dollars, turnCosts, type Context } from "../../core/conversation_cost.ts";
1718 import { WORDS_MAX } from "../../core/fence.ts";
1819 import { notePhase, takeSessionTime, timePhase } from "../../core/route_timing.ts";
1920 import { ASKING } from "../../crew/conversation_confirm.ts";
@@ -76,8 +77,15 @@ const toolCard = (id: string, says: string, name: string, at: string): Html =>
7677 /** What an empty chat offers to start with; each is a link that fills the composer (`?q=`), so it works without the script. */
7778 const SUGGESTIONS = ["What needs my attention?", "What did the crew do today?", "Show me my apps"] as const;
7879
80 /** How full the context is, one quiet line above the log; always drawn so a finished run can fill it. */
81 const contextLine = (c: Context | undefined): Html =>
82 html`<p class="talk-context small muted" data-context ${c === undefined ? html`hidden` : null}>${c === undefined ? "" : contextText(c)}</p>`;
83
84 /** The cost line under a reply. */
85 const costLine = (usd: number): Html => html`<p class="talk-cost small muted">${dollars(usd)} this turn</p>`;
86
7987 /** One turn as the page and the stream both draw it; `waiting` is the confirms still open. */
80 export function turnHtml(turn: Turn, waiting: ReadonlySet<string>): Html {
88 export function turnHtml(turn: Turn, waiting: ReadonlySet<string>, cost?: number): Html {
8189 const id = `t-${turn.id}`;
8290 if (turn.role === "person")
8391 return html`<li class="talk-turn talk-person" id="${id}">
@@ -88,6 +96,7 @@ export function turnHtml(turn: Turn, waiting: ReadonlySet<string>): Html {
8896 return html`<li class="talk-turn talk-agent" id="${id}">
8997 <div class="talk-body prose">${renderMarkdown(turn.text)}</div>
9098 ${meta("Forge", turn.createdAt)}
99 ${cost === undefined ? null : costLine(cost)}
91100 </li>`;
92101 const data = parse(turn.text);
93102 if (turn.role === "confirm") {
@@ -118,11 +127,14 @@ function talkPage(ctx: Ctx, before: string | undefined, prefill: string): Respon
118127 const turns = listed.ok ? listed.value : [];
119128 const waiting = new Set(waitingConfirms(ctx, user).map((w) => w.id));
120129 const running = clock.running(ctx.app, user.id);
junipercrew outdated

The suggestion is plain text with no way to act on it, so it is a dead end for a thumb. Either add a working action (a post form button) or word it as information. The header line is also wrapped inside .page-head next to the h1, so check it at 360px.
130 const costs = turnCosts(ctx.app.db, conversation.id);
131 const context = contextOf(ctx.app.db, conversation.id);
121132 // Paging back, the log ends before the newest turn: it does not follow.
122133 const last = turns.at(-1)?.id ?? "";
123134 const body = html`<div class="page-head"><h1>Talk</h1></div>
124135 <div class="talk" data-talk data-user="${user.id}" data-paged="${before === undefined ? "0" : "1"}" data-running="${running && before === undefined ? "1" : "0"}">
125136 <div>
137 ${contextLine(context)}
126138 ${turns.length === DEFAULT_PAGE && turns[0] !== undefined
127139 ? html`<p class="talk-more small"><a href="${HERE}?before=${turns[0].id}">Earlier</a></p>`
128140 : null}
@@ -131,7 +143,7 @@ function talkPage(ctx: Ctx, before: string | undefined, prefill: string): Respon
131143 ${turns.length === 0
132144 ? html`<li class="muted talk-turn">Ask the forge anything, or tell it what to change. It acts as you, with your permissions, and asks before anything hard to undo.
133145 <span class="talk-suggest">${SUGGESTIONS.map((q) => html`<a class="btn" href="${HERE}?q=${encodeURIComponent(q)}" data-suggest="${q}">${q}</a>`)}</span></li>`
134 : turns.map((t) => turnHtml(t, waiting))}
146 : turns.map((t) => turnHtml(t, waiting, costs.get(t.id)))}
135147 </ol>
136148 <div id="end"></div>
137149 </div>
@@ -226,7 +238,10 @@ export function talkStream(ctx: Ctx): Response {
226238 }
227239 if (fresh.length >= MAX_PAGE) return;
228240 if (!running) {
229 emit("done", {});
241 // The run stored its figures before it stopped running: send them with the end.
242 const costs = [...turnCosts(app.db, conversation.id)].slice(-5).map(([id, usd]) => ({ id: `t-${id}`, html: costLine(usd).text }));
243 const context = contextOf(app.db, conversation.id);
244 emit("done", { costs, context: context === undefined ? "" : contextText(context) });
230245 close();
231246 } else if (Date.now() - started >= clock.streamMs) {
232247 emit("more", {});
src/web/pages/talk_client.ts +7 −1
@@ -27,6 +27,7 @@ export const talkCss = /* css */ `
2727 .talk-compose .bar { margin-top: 6px; gap: 8px; align-items: center; }
2828 .talk-compose .btn { min-height: 2.75rem; }
2929 .talk-status { min-height: 1.5em; margin: 0; }
30 .talk-context, .talk-cost { margin: 0; }
3031 .talk-meta { margin: 4px 0 0; display: flex; gap: 6px; align-items: center; }
3132 .talk-copy { margin-left: auto; background: none; border: 0; color: inherit; font: inherit; cursor: pointer; padding: 6px 8px; min-height: 2rem; opacity: 0.7; }
3233 .talk-copy:hover { opacity: 1; text-decoration: underline; }
@@ -136,7 +137,12 @@ if (root) {
136137 toEnd();
137138 });
138139 source.addEventListener("running", () => { mode(true); say("The forge is answering…"); });
139 source.addEventListener("done", () => { mode(false); say(""); source.close(); source = null; });
140 source.addEventListener("done", (e) => {
141 mode(false); say(""); source.close(); source = null;
142 const o = JSON.parse(e.data), c = root.querySelector("[data-context]");
143 for (const k of o.costs || []) { const li = document.getElementById(k.id); if (li && !li.querySelector(".talk-cost")) li.insertAdjacentHTML("beforeend", k.html); }
144 if (o.context && c) { c.textContent = o.context; c.hidden = false; }
145 });
140146 source.addEventListener("more", () => { source.close(); follow(); });
141147 source.addEventListener("drain", () => { source.close(); setTimeout(follow, 1000); });
142148 source.onerror = () => {
test/claude-context.test.ts added +47
@@ -0,0 +1,47 @@
1 /**
2 * What a session's own usage record says about its context (#8822): the last
3 * assistant message's input is the context held, the result's `modelUsage`
4 * names the model's window.
5 */
6 import { afterAll, expect, test } from "bun:test";
7 import { chmodSync, writeFileSync } from "node:fs";
8 import { mkdtemp, rm } from "node:fs/promises";
9 import { tmpdir } from "node:os";
10 import { join } from "node:path";
11 import { runClaude, windowOf } from "../src/crew/claude.ts";
12 import { sandboxPolicy } from "../src/crew/sandbox.ts";
13
14 const dir = await mkdtemp(join(tmpdir(), "forge-context-"));
15 afterAll(() => rm(dir, { recursive: true, force: true }));
16
17 test("a recorded usage stream gives the cost, the context held and the model's window", async () => {
18 const binary = join(dir, "recorded-claude");
19 writeFileSync(
20 binary,
21 `#!/bin/sh
22 IFS= read -r first
23 echo '{"type":"system","subtype":"init","session_id":"s1","model":"fake"}'
24 echo '{"type":"assistant","message":{"id":"m1","content":[{"type":"text","text":"one"}],"usage":{"input_tokens":10,"cache_read_input_tokens":90000,"output_tokens":5}}}'
25 echo '{"type":"assistant","message":{"id":"m2","content":[{"type":"text","text":"two"}],"usage":{"input_tokens":20,"cache_creation_input_tokens":10000,"cache_read_input_tokens":160000,"output_tokens":9}}}'
26 echo '{"type":"result","subtype":"success","is_error":false,"result":"two","total_cost_usd":0.1234,"num_turns":2,"usage":{"input_tokens":30,"cache_read_input_tokens":250000,"output_tokens":14},"modelUsage":{"fake":{"contextWindow":1000000}}}'
27 `,
28 );
29 chmodSync(binary, 0o755);
30 const spent: number[] = [];
31 const outcome = await runClaude({
32 binary, cwd: dir, prompt: "hi", appendSystemPrompt: "", budgetUsd: 1, env: {}, timeoutMs: 20_000,
33 sandbox: sandboxPolicy({ dataDir: dir, agentName: "bot", urls: [] }), log: () => {}, signal: new AbortController().signal, settleMs: 1_000,
34 onUsage: (u) => spent.push(u.tokensIn),
35 });
36 expect(outcome.contextTokens).toBe(170_020);
37 expect(outcome.contextWindow).toBe(1_000_000);
38 expect(outcome.costUsd).toBe(0.1234);
39 // The running totals still sum every message.
40 expect(spent[1]).toBe(90_010 + 170_020);
41 });
42
43 test("windowOf takes the largest window named and ignores anything else", () => {
44 expect(windowOf({ a: { contextWindow: 200000 }, b: { contextWindow: 1000000 } })).toBe(1_000_000);
45 expect(windowOf({ a: { contextWindow: "big" } })).toBe(0);
46 expect(windowOf(undefined)).toBe(0);
47 });
test/conversation-talk.test.ts +17
@@ -19,6 +19,7 @@ import { mintViaToken } from "../src/core/via_token.ts";
1919 import { TALK_PER_MINUTE } from "../src/actions/talk.ts";
2020 import { turnsLimit } from "../src/http/api/conversation_talk.ts";
2121 import { serve, type Servers } from "../src/http/server.ts";
22 import { appendTurn, openConversation } from "../src/core/conversations.ts";
2223 import { forgeSshSession } from "../src/tui/ssh.ts";
2324 import { isStop, parseTalkCommand } from "../src/tui/ssh-talk.ts";
2425 import { quietLogs } from "./helpers/globals.ts";
@@ -244,6 +245,22 @@ echo '{"type":"result","subtype":"success","is_error":false,"result":"Heard.","t
244245 expect(await persons()).toEqual(["by api", "by mcp", "by ssh $HOME"]);
245246 });
246247
248 test("ssh talk prints the last turn's cost and the context share from the run's own record", async () => {
249 const id = app.db.query<{ id: string }, []>("SELECT id FROM user WHERE username = 'alice'").get()!.id;
250 const conversation = openConversation(app.db, id);
251 const reply = appendTurn(app.db, id, conversation.id, { role: "agent", text: "answered" });
252 if (!reply.ok) throw new Error(reply.error);
253 const at = new Date(Date.now() - 1000).toISOString();
254 app.db
255 .query("INSERT INTO conversation_run (id, conversation_id, user_id, prompt, state, cost_usd, context_tokens, context_window, started_at, finished_at) VALUES ('rc1',?1,?2,'x','ok',0.5,500000,1000000,?3,?4)")
256 .run(conversation.id, id, at, new Date(Date.now() + 1000).toISOString());
257 const r = await exec("talk and the figures");
258 expect(r.out).toContain("last turn $0.50");
259 expect(r.out).toContain("Context 50% full (500k of 1M tokens)");
260 app.db.query("DELETE FROM conversation_run WHERE id = 'rc1'").run();
261 });
262
263
247264 test("ssh talk /stop stops instead of speaking; with nothing running it says so (#8818)", async () => {
248265 expect(isStop("/stop")).toBe(true);
249266 expect(isStop(" --stop ")).toBe(true);
test/crew-limits.test.ts +2
@@ -15,6 +15,8 @@ const outcome = (over: Partial<ClaudeOutcome>): ClaudeOutcome => ({
1515 turns: 1,
1616 tokensIn: 0,
1717 tokensOut: 0,
18 contextTokens: 0,
19 contextWindow: 0,
1820 reason: "error",
1921 ...over,
2022 });
test/crew-network.test.ts +2
@@ -32,6 +32,8 @@ const outcome = (over: Partial<ClaudeOutcome>): ClaudeOutcome => ({
3232 turns: 0,
3333 tokensIn: 0,
3434 tokensOut: 0,
35 contextTokens: 0,
36 contextWindow: 0,
3537 reason: "error",
3638 ...over,
3739 });
test/subscriptions.test.ts +2
@@ -269,6 +269,8 @@ describe("the provider's reset time", () => {
269269 turns: 0,
270270 tokensIn: 0,
271271 tokensOut: 0,
272 contextTokens: 0,
273 contextWindow: 0,
272274 reason: "error_during_execution",
273275 }),
274276 ).toBe(true);
test/talk-page.test.ts +37 −1
@@ -192,7 +192,7 @@ describe("the page", () => {
192192 test("the page's own bytes are held: stylesheet and script stay small", async () => {
193193 const { talkCss, talkJs } = await import("../src/web/pages/talk_client.ts");
194194 expect(Buffer.byteLength(talkCss)).toBeLessThan(3_500);
195 expect(Buffer.byteLength(talkJs)).toBeLessThan(6_500);
195 expect(Buffer.byteLength(talkJs)).toBeLessThan(7_400);
196196 say(alice, "agent", "```ts\nlet x = 1;\n```");
197197 const page = await get("/talk", alice);
198198 const size = (await page.arrayBuffer()).byteLength;
@@ -524,3 +524,39 @@ describe("a long stream", () => {
524524 }
525525 });
526526 });
527
528 describe("cost and context (#8822)", () => {
529 test("a run's stored usage gives the turn's cost under the reply and the context share in the header", async () => {
530 const erin = await createUser(app.db, { origin: "seed", trusted: true, username: "erin", email: "[email protected]", password: "a-good-passphrase" });
531 if (!erin.ok) throw new Error(erin.error);
532 promote(app.db, erin.value);
533 const conversation = openConversation(app.db, erin.value.id);
534 const before = await (await get("/talk", erin.value)).text();
535 expect(before).not.toContain("Context ");
536 say(erin.value, "person", "hello");
537 const reply = say(erin.value, "agent", "two");
538 if (!reply.ok) throw new Error(reply.error);
539 const at = new Date(Date.now() - 1000).toISOString();
540 app.db
541 .query("INSERT INTO conversation_run (id, conversation_id, user_id, prompt, state, cost_usd, tokens_in, tokens_out, context_tokens, context_window, started_at, finished_at) VALUES ('r1',?1,?2,'hello','ok',?3,?4,?5,?6,200000,?7,?8)")
542 .run(conversation.id, erin.value.id, 0.1234, 170030, 14, 170020, at, new Date(Date.now() + 1000).toISOString());
543 const html = await (await get("/talk", erin.value)).text();
544 expect(html).toContain("$0.12 this turn");
545 expect(html).toContain("Context 85% full (170k of 200k tokens)");
546 expect(html).toContain("A new conversation answers faster");
547 // A longer window is read from the run, not assumed.
548 app.db.query("UPDATE conversation_run SET context_window = 1000000").run();
549 expect(await (await get("/talk", erin.value)).text()).toContain("Context 17% full (170k of 1M tokens)");
550 app.db.query("UPDATE conversation_run SET context_window = 200000").run();
551 // The stream's end carries both, so a reply that streamed in gets them without a reload.
552 const done = (await frames("/talk/stream?after=", erin.value)).find((f) => f.event === "done");
553 const sent = JSON.parse(done?.data ?? "{}");
554 expect(sent.costs[0].html).toContain("$0.12 this turn");
555 expect(sent.context).toContain("Context 85% full");
556 // Under 80 %: no suggestion.
557 app.db.query("UPDATE conversation_run SET context_tokens = 40000").run();
558 const calm = await (await get("/talk", erin.value)).text();
559 expect(calm).toContain("Context 20% full");
560 expect(calm).not.toContain("A new conversation answers faster");
561 });
562 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs