Public: anyone can read this; the crew can change it.

A repository can carry a beat the forge owns, and a direct push is gated on its tests before the switch #13729

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/13709-a-repository-can-carry-a-beat-the-forge → main 7 commits fork 4 threads · 4 unresolved

  • Merged. scout/13709-a-repository-can-carry-a-beat-the-forge was merged into main as d883f28.

Review

Sign in to review this change.
18 files changed +688 −16
AGENTS.md +1
@@ -52,3 +52,4 @@ The check queue hands work to a runner through `src/checks/runner.ts` (#13652, R
5252
5353 An app token's requests go behind people's and repeat reads are 304s (`src/http/app_turn.ts`, `repeat_reads.ts`, `perf/asking.ts`): see `docs/features/app-asking.md`.
5454 `src/core/reports.ts`: `closedSet`, the ids of repositories whose owner closed reports from outside (#12321), read once per database and kept by `setReportsClosed`, the only writer, so every repository page's head link costs no statement; at most one id per closed repository, gone on restart (the rows are `report_closed`). `src/core/contribute_token.ts`: `openSet`, the ids of repositories that allow outside agents, read once per database and kept by `setContributeOpen`, the only writer, at most one id per allowing repository, gone on restart; a raw SQL write to `contribute_open` bypasses it. The terminal door for reports is `r` on the outside agents screen (`setOutsideReports`, admins).
55 `src/crew/beat.ts`: `lastBeat`, when each repository's beat last ran, and `declared`, the `beat` last read from each default branch with the commit it came from (#13709); at most one entry each per repository, gone on restart (a restart beats once at once; the declaration is a file in git).
docs/features/README.md +1
@@ -210,6 +210,7 @@ Read with the guide page [Ship and undo](../guide-ship-and-undo.md).
210210 - [Pulls untouched for 14 days](stale-pulls.md)
211211 - [Ranked checks](ranked-checks.md)
212212 - [A repository stuck behind its pulls](stuck-pile.md)
213 - [A beat the forge owns, and a gate on the push](beat.md)
213214 - [Release heartbeat and the main-ahead alarm](release-heartbeat.md)
214215 - [Release keeper](release-keeper.md)
215216 - [Release what the Later gate holds](release-later.md)
docs/features/beat.md added +19
@@ -0,0 +1,19 @@
1 # A beat the forge owns, and a gate on the push
2
3 ## The beat
4
5 A repository can carry a routine in its default branch's `forge.json`:

The beat is a feature added ahead of the pipeline list the owner set. Split it from the push gate, and explain how it serves the direction.

6
7 ```json
8 "beat": { "every": "1h" }
9 ```
10
11 Every `every` (`10m` to `7d`, default an hour; a misspelt key or a bad value is refused with the field named, `src/apps/beat_spec.ts`), the forge takes one turn of the repository's autonomy loop with its own agent: the next ready issue goes to the crew, or the planner is asked when the map is thin, or the feed says why not (`beat.ran`: every builder busy, the crew at its pace, autonomy paused, nothing ready). It needs nothing on anyone's machine and stops where the autonomy stops: a person turned autonomy on and set its week's share, and a spent share starts nothing.
12
13 In the autonomy tick, a repository with a beat takes its turn of the loop only when the beat is due; the passes around it (finishing, stuck pulls, requested changes) still run each minute. The feed hears when the outcome changes or something started, not every interval, and a `beat` that cannot be read is said once in the feed (`beat not kept: ...`) while the repository stays on the minute loop. A repository whose autonomy is off has no beat, and says nothing: autonomy is the switch a person owns. The last run is held in process memory (`src/crew/beat.ts`), so a restart beats once at once.
14
15 ## The gate
16
17 A commit pushed straight to the default branch of an app that declares `app.test` no longer switches before it is tested. The new tree starts beside the old one and its tests run first; green switches, red does not (the tests get a budget, `GATE_BUDGET_MS`, ten minutes; a suite that outlives it counts as red): the last green commit keeps serving, the deploy run fails with `held back: its tests failed (...); <sha> keeps serving`, the feed carries an `app.gated` post, and the owner's notification address (Account) gets the same line. The first deploy of an app and a merged pull (already checked) switch at once, as before; previews are unchanged (`src/apps/push_gate.ts`).
18
19 The app template's `AGENTS.md` states the ritual: one change, its test, its README line, its version, one commit.
src/apps/beat_spec.ts added +38
@@ -0,0 +1,38 @@
1 /**
2 * `beat` (top level of `forge.json`): a routine the forge owns for this repository.
3 *
4 * "beat": { "every": "1h" }
5 *
6 * Every `every`, the forge's own agent ships the next ready thing, or the repository's feed says
7 * why not (`crew/beat.ts`). `every` is a number and a unit, `m` or `h` or `d`, from ten minutes to
8 * a week; unset means an hour. Pure: plain data in, plain data or one sentence naming the field
9 * out. Unknown keys are refused so a misspelt `evry` fails the manifest instead of meaning an hour.
10 */
11
12 import { err, ok, type Result } from "../kernel/result.ts";
13
14 export interface BeatSpec {
15 readonly every: string;
16 readonly everyMs: number;
17 }
18
19 export const BEAT_MIN_MS = 10 * 60_000;
20 export const BEAT_MAX_MS = 7 * 86_400_000;
21 export const BEAT_DEFAULT = "1h";
22
23 const UNIT = { m: 60_000, h: 3_600_000, d: 86_400_000 } as const;
24
25 export function parseBeat(value: unknown): Result<BeatSpec | undefined, string> {
26 if (value === undefined) return ok(undefined);
27 if (typeof value !== "object" || value === null || Array.isArray(value))
28 return err('forge.json: beat must be an object like { "every": "1h" }');
29 const block = value as Record<string, unknown>;
30 const unknown = Object.keys(block).find((k) => k !== "every");
31 if (unknown !== undefined) return err(`forge.json: beat.${unknown.slice(0, 40)} is not a field; the only one is every`);
32 const every = block["every"] ?? BEAT_DEFAULT;
33 const match = typeof every === "string" ? /^([1-9]\d{0,3})([mhd])$/.exec(every) : null;
34 if (match === null) return err('forge.json: beat.every must be a number and m, h or d, like "30m" or "1h"');
35 const everyMs = Number(match[1]) * UNIT[match[2] as keyof typeof UNIT];
36 if (everyMs < BEAT_MIN_MS || everyMs > BEAT_MAX_MS) return err("forge.json: beat.every must be from 10m to 7d");
37 return ok({ every: `${match[1]}${match[2]}`, everyMs });
38 }
src/apps/gate_notice.ts added +26
@@ -0,0 +1,26 @@
1 /**
2 * Telling the owner a push did not switch (#13709): one line to the notification address they gave
3 * the forge, if any. A line that cannot be sent is dropped; it never fails the deploy.
4 */
5
6 import type { Database } from "bun:sqlite";
7 import { notifyUrlOf, post } from "../crew/notify.ts";
8 import { createLog } from "../kernel/log.ts";
9
10 const log = createLog("gate");
11
12 export async function tellGated(
13 db: Database,
14 ownerId: string,
15 line: string,
16 send: typeof post = post,
17 ): Promise<boolean> {
18 try {
19 const url = notifyUrlOf(db, ownerId);
20 if (url === undefined) return false;
21 return await send(url, line, { title: "forge" });
22 } catch (error) {
23 log.warn("could not tell the owner a push was gated", { error: String(error) });
24 return false;
25 }
26 }
src/apps/manifest.ts +9
@@ -20,6 +20,7 @@ import { isLaunch, LAUNCHES, type Launch } from "../git/builds.ts";
2020 import { parseAdapter, type AdapterSpec } from "./adapter.ts";
2121 import { parseToolchain, type ToolchainPin } from "./toolchain.ts";
2222 import { parseSchedules, type ScheduleSpec } from "./schedule_spec.ts";
23 import { parseBeat, type BeatSpec } from "./beat_spec.ts";
2324 import { parseFeature, scheduleSpecs } from "./feature.ts";
2425
2526 export interface AppManifest {
@@ -104,6 +105,11 @@ export interface AppManifest {
104105 * (`apps/schedule_spec.ts`). Unset is none.
105106 */
106107 readonly schedules?: readonly ScheduleSpec[] | undefined;
108 /**
109 * `beat` (top level of `forge.json`): the routine the forge's own agent keeps
110 * (`apps/beat_spec.ts`). Unset is none.
111 */
112 readonly beat?: BeatSpec | undefined;
107113 }
108114
109115 export type ManifestError =
@@ -222,6 +228,8 @@ export function parseManifest(
222228 return invalid(`forge.json: adapter.source.hosts names ${unopened}, which egress does not list`);
223229 const schedules = parseSchedules(parsed.value["schedules"]);
224230 if (!schedules.ok) return invalid(schedules.error);
231 const beat = parseBeat(parsed.value["beat"]);
232 if (!beat.ok) return invalid(beat.error);
225233 // A feature's scheduled tools are schedules like any other (#13533). A feature block that does
226234 // not parse is the absorb page's to report, as before: it declares none here.
227235 const feature = parseFeature(parsed.value);
@@ -267,6 +275,7 @@ export function parseManifest(
267275 ...(adapter.value === undefined ? {} : { adapter: adapter.value }),
268276 ...(toolchain.value === undefined ? {} : { toolchain: toolchain.value }),
269277 ...(allSchedules.length === 0 ? {} : { schedules: allSchedules }),
278 ...(beat.value === undefined ? {} : { beat: beat.value }),
270279 });
271280 }
272281
src/apps/push_gate.ts added +39
@@ -0,0 +1,39 @@
1 /**
2 * The gate on a direct push (#13709). A commit pushed straight to the default branch used to
3 * switch in and then be tested, so a red push served until someone noticed. Now, when production
4 * has a deployment to keep and the manifest declares `test`, the tests run on the new tree
5 * before the switch; red keeps the last green commit serving. Plain data in, plain data out.
6 */
7
8 import { during } from "../perf/jobs.ts";
9 import { err, type Result } from "../kernel/result.ts";
10
11 /** Whether this deploy waits for its tests before it switches in. */
12 export function gatesPush(env: string, reason: string, hasLive: boolean, declaresTest: boolean): boolean {
13 return env === "production" && reason === "push" && hasLive && declaresTest;
14 }
15
16 /** The run's summary and the owner's line when the gate held a push back. */
17 export function gateSummary(why: string, keptOid: string): string {
18 const first = (why.split("\n")[0] ?? "").trim().slice(0, 160);
19 return `held back: its tests failed (${first}); ${keptOid.slice(0, 7)} keeps serving`;
20 }
21
22 /** How long the gate waits for the tests: ten minutes, or `FORGE_GATE_BUDGET_MS` (1 s to 1 h). */
23 export function gateBudgetMs(raw: string | undefined = process.env["FORGE_GATE_BUDGET_MS"]): number {
24 const n = raw !== undefined && /^\d{1,7}$/.test(raw) ? Number(raw) : NaN;
25 return n >= 1_000 && n <= 3_600_000 ? n : 10 * 60_000;
26 }
27
28 /** The tests' verdict, or red when they outlive the budget: a hung suite never holds a deploy. */
29 export async function withinBudget(tested: Promise<Result<void, string>>, ms: number): Promise<Result<void, string>> {
30 let timer: ReturnType<typeof setTimeout> | undefined;
31 const late = new Promise<Result<void, string>>((resolve) => {
32 timer = setTimeout(() => during("push-gate-budget", () => resolve(err(`the tests did not finish in ${Math.round(ms / 1000)} s`))), ms);
33 });
34 try {
35 return await Promise.race([tested, late]);
36 } finally {
37 clearTimeout(timer);
38 }
39 }
src/apps/runtime.ts +35 −1
@@ -24,6 +24,8 @@ import { commitLog } from "../git/read.ts";
2424 import { noteRelease, noteReleaseTests, recordsReleases } from "./release_record.ts";
2525 import { settleStaleReleases } from "../core/app_releases.ts";
2626 import { forgetLanded, noteLanded } from "./landed.ts";
27 import { gateBudgetMs, gatesPush, gateSummary, withinBudget } from "./push_gate.ts";
28 import { tellGated } from "./gate_notice.ts";
2729 import { releaseOfCommit } from "../core/release_subject.ts";
2830 import { pullDir, runDir } from "../core/pulls.ts";
2931 import { accountActive } from "../core/accounts.ts";
@@ -1437,6 +1439,38 @@ export class AppsRuntime {
14371439 return;
14381440 }
14391441
1442 // A direct push to production is gated (#13709): the declared tests run on the new tree
1443 // before the switch, and while they fail the last green commit keeps serving. A merged
1444 // pull already passed its check, and a first deploy has nothing to keep, so both swap in
1445 // at once as before.
1446 let gated: Result<void, string> | undefined;
1447 if (gatesPush(state.env, run.reason, state.live !== undefined, outcome.value.manifest.test !== null)) {
1448 write(`gate: running the tests on ${run.commitOid.slice(0, 7)} before the switch`);
1449 gated = await withinBudget(outcome.tested, gateBudgetMs());
1450 if (!gated.ok) {
1451 const kept = state.live!;
1452 await outcome.value.handle.stop();
1453 revokeAppToken(this.app.db, outcome.value.tokenId);
1454 if (outcome.value.dir !== kept.dir) await rm(outcome.value.dir, { recursive: true, force: true });
1455 state.deploying = undefined;
1456 lease();
1457 const summary = gateSummary(gated.error, kept.commitOid);
1458 state.lastError = summary;
1459 state.lastErrorCommitOid = run.commitOid;
1460 const done = finishRun(this.app.db, run.id, { state: "failed", summary });
1461 write(`failed: ${summary}`);
1462 if (done) announceRun(this.app, done);
1463 this.app.bus.append({
1464 kind: "app.gated",
1465 repoId: repo.id,
1466 payload: { app: state.app.slug, env: state.env, commitOid: run.commitOid, keptOid: kept.commitOid, why: gated.error.slice(0, 300) },
1467 });
1468 await tellGated(this.app.db, repo.ownerId, `${state.app.slug}: ${summary}\n${this.urlFor(state.app, state.env)}`);
1469 return;
1470 }
1471 write("gate: the tests passed; switching");
1472 }
1473
14401474 // It answers: swap it in now, before its tests are done. `state.deploying`
14411475 // stays set — the run is still open, and a stop or a teardown must still
14421476 // be able to reach in and kill the test step — until they finish below.
@@ -1541,7 +1575,7 @@ export class AppsRuntime {
15411575 live.manifest.looks,
15421576 );
15431577
1544 const tested = await outcome.tested;
1578 const tested = gated ?? (await outcome.tested);
15451579 state.deploying = undefined;
15461580 lease();
15471581 noteReleaseTests(this.app.db, state.env, live.manifest, repo.id, live.commitOid, tested.ok);
src/core/event_audience.ts +2
@@ -225,6 +225,8 @@ export function classifiedAudience(kind: string): EventAudience | undefined {
225225 // A hook that keeps failing is for those who can act on it: the host and
226226 // the reason, never the address (`webhook_streak.ts`).
227227 if (kind === "webhook.failing") return "writers";
228 // What a repository's beat did says how its autonomy and week's share stand (`crew/beat.ts`).
229 if (kind === "beat.ran") return "writers";
228230 if (
229231 kind.startsWith("webhook.") ||
230232 kind.startsWith("app.env_") ||
src/crew/autonomy.ts +23 −13
@@ -115,6 +115,7 @@ import { REPEAT_NOTE } from "./repeat_note.ts";
115115 import { given, pendingReplans } from "../core/limb_replan.ts";
116116 import { handedOverPulls, SKIP_SUMMARY, skipsReplan } from "./guard.ts";
117117 import { type Held, heldBy, heldWords, replannable } from "./held.ts";
118 import { driveOrBeat } from "./beat.ts";
118119 import { forgetBacklog, noteBacklog, waitingBy, waitingWords } from "./backlog.ts";
119120 import { ceilingHold, pausedUntil, pauseSentence } from "../core/planner_pause.ts";
120121 import { CREW_OFF_WORDS, crewBuildsHere } from "../core/crew_switch.ts";
@@ -394,7 +395,9 @@ function meanRunUsd(db: App["db"], repoId: string): number {
394395 );
395396 }
396397
397 /** The issues each repository picked in this tick, by repository id: their runs may not show yet. */
398 /**
399 * The issues each repository picked in this tick, by repository id: their runs may not show yet.
400 */
398401 export type TickPicks = ReadonlyMap<string, ReadonlySet<string>>;
399402
400403 /**
@@ -649,10 +652,10 @@ export function startable(
649652
650653 /**
651654 * Give the planner the direction when the reserve is due a pass — down to
652 * half of two startable issues per builder slot (`reserve.ts`) — or when the crew has handed it an issue it parked twice,
653 * and no planning run is in flight, and the last one is old enough.
654 * Returns whether a planning run was set up. Planning runs alongside
655 * building: the builders are never waited for.
655 * half of two startable issues per builder slot (`reserve.ts`) — or when the crew has handed it an
656 * issue it parked twice, and no planning run is in flight, and the last one is old enough. Returns
657 * whether a planning run was set up. Planning runs alongside building: the builders are never
658 * waited for.
656659 *
657660 * A hand-over is owed one pass, not one a minute: it asks only while it is
658661 * newer than the last planning run. That one pass is owed even by a planner
@@ -1008,7 +1011,10 @@ export function parkForBreaker(
10081011 return true;
10091012 }
10101013
1011 /** How many times the crew has parked this issue since a person who directs it last spoke (`parks.ts`). */
1014 /**
1015 * How many times the crew has parked this issue since a person who directs it last spoke
1016 * (`parks.ts`).
1017 */
10121018 export function parkedCount(app: App, repo: Repo, issueId: string): number {
10131019 return countParks(app.db, repo, issueId);
10141020 }
@@ -1305,7 +1311,8 @@ export async function autonomyTick(
13051311 sweep: (app: App, repo: Repo) => Promise<number> = sweepGreen,
13061312 ): Promise<void> {
13071313 if (crewOffInPreview()) return;
1308 // A pull a person holds with changes requested goes back to its builder after a day of silence, in any repository.
1314 // A pull a person holds with changes requested goes back to its builder after a day of silence,
1315 // in any repository.
13091316 try {
13101317 sweepStranded(app, now);
13111318 } catch (error) {
@@ -1336,7 +1343,8 @@ export async function autonomyTick(
13361343 // the next batch is formed, not at its gate (#6611).
13371344 pruneLine(app, repo);
13381345 void startSweep(app, repo, sweep);
1339 // Each pass below is one slice of the thread: a request waiting is served between them (#8848).
1346 // Each pass below is one slice of the thread: a request waiting is served between them
1347 // (#8848).
13401348 await breathe();
13411349 // First of all: a slot freed goes to an approved pull that no longer
13421350 // merges, before a changes-requested one or a new issue (#3682).
@@ -1379,7 +1387,7 @@ export async function autonomyTick(
13791387 const requested = await moveRequested(app, current, freeSlots(app, current, now), now);
13801388 await breathe();
13811389 const unstuck = unstickNow(app, current, now);
1382 const did = await driveRepo(app, current, now, picked);
1390 const did = await driveOrBeat(app, current, now, picked, driveRepo);
13831391 await breathe();
13841392 // Rules only, no run: an issue noted already done on the base closes.
13851393 const triaged = await liveTriage(app, current, now).catch((error) => {
@@ -1432,7 +1440,8 @@ export async function autonomyTick(
14321440 // The standing reserve the planner keeps: startable/target (#3237).
14331441 reserve: reserveWords(reserveOf(held.startable.length, status.slots)),
14341442 held: heldWords(held),
1435 // What the picker passes over among the startable, so no ready issue goes unexplained (#8924).
1443 // What the picker passes over among the startable, so no ready issue goes unexplained
1444 // (#8924).
14361445 passed: passedWords(pickFilter(app, current, held.startable, now).passed),
14371446 // The files 5 or more open pulls touch, which no new build starts on (#5934).
14381447 ...(held.by.hot_file.length === 0 ? {} : { hot: hotWords(hotFiles(current.id)) }),
@@ -1448,8 +1457,8 @@ export async function autonomyTick(
14481457 idle: idleBuilders(activeNow, crewNow),
14491458 ...(governor === undefined ? {} : { governor }),
14501459 // Approved conflicted pulls: queued, in flight, and why each other is
1451 // skipped (#3682, #6237), or "0 candidates of N open" (#6263); and the lane for rebases: conflicted N,
1452 // rebasing M, room R (#6140).
1460 // skipped (#3682, #6237), or "0 candidates of N open" (#6263); and the lane for rebases:
1461 // conflicted N, rebasing M, room R (#6140).
14531462 ...finishFields(finishing.moves, readConflictLane(app.db), finishing.open),
14541463 ...(requested.length === 0
14551464 ? {}
@@ -2256,7 +2265,8 @@ async function admit(
22562265 if (!again) addComment(app, by, repo, getIssueById(app.db, pull.id) ?? pull, clashSaid(red.comment, together));
22572266 // A red that reproduces on the re-gate is the code's: only then is it the doctor's to hold.
22582267 if (again) treatRed(app, repo, by, pull, red, baseOid);
2259 // Red only on top of pulls that landed first: no single culprit yet, so the class is recorded and the re-gate alone decides.
2268 // Red only on top of pulls that landed first: no single culprit yet, so the class is recorded
2269 // and the re-gate alone decides.
22602270 else noteRed(app, repo, pull, red, baseOid, together);
22612271 },
22622272 });
src/crew/beat.ts added +111
@@ -0,0 +1,111 @@
1 /**

The off case of beatWords can never be reached, and a bad beat declaration is only logged. Make the feed say why. Also post beat.ran only on a change in outcome, not every interval.

2 * The beat a repository carries (#13709): `beat` in its `forge.json` (`apps/beat_spec.ts`) says
3 * "every hour, ship the next ready thing, or say why not". The forge keeps it with its own agent:
4 * a due beat takes one turn of the repository's autonomy loop (`driveRepo`), which starts the
5 * next ready issue on the crew, and then says in the repository's feed what came of it
6 * (`beat.ran`, with `beatWords`, when the outcome changes or something started). Spend is the
7 * autonomy's, which a person turned on: a repository whose autonomy is off has no beat, and one
8 * whose week's share is spent says so and starts nothing. A repository with a beat takes that turn
9 * only when its beat is due (`driveOrBeat`), in the tick's own pass, so the passes before it still
10 * run. Last run per repository is held in process memory (`kept`), so a restart beats once at once;
11 * the declarations are read from the default branch's `forge.json`, re-read only when that branch
12 * moves.
13 */
14
15 import type { App } from "../app.ts";
16 import { parseBeat, type BeatSpec } from "../apps/beat_spec.ts";
17 import { readFileAt } from "../apps/checkout.ts";
18 import type { Repo } from "../core/repos.ts";
19 import { resolveOid } from "../git/merge.ts";
20 import { repoDir } from "../git/repos.ts";
21 import { createLog } from "../kernel/log.ts";
22
23 const log = createLog("beat");
24
25 /** What one turn of a repository's autonomy loop did (`autonomy.ts` `Drove`). */
26 export type Drove = "off" | "paused" | "waiting" | "full" | "picked" | "planned" | "idle" | "stalled";
27 export type Drive = (app: App, repo: Repo, now: Date, picked: Map<string, Set<string>>) => Promise<Drove>;
28
29 /**
30 * Per repository: when its beat last ran, the declaration last read with the commit it came from,
31 * and what the feed was last told. Held per forge, keyed by the app.
32 */
33 type Kept = {
34 last: Map<string, number>;
35 declared: Map<string, { oid: string; beat: BeatSpec | undefined }>;
36 said: Map<string, Drove>;
37 };
38 const kept = new WeakMap<App, Kept>();
39 function keptOf(app: App): Kept {
40 let k = kept.get(app);
41 if (k === undefined) kept.set(app, (k = { last: new Map(), declared: new Map(), said: new Map() }));
42 return k;
43 }
44
45 /** Whether a beat is due: never run, or `everyMs` since the last. */
46 export const beatDue = (last: number | undefined, everyMs: number, now: number): boolean => last === undefined || now - last >= everyMs;
47
48 /** What a turn did, in a sentence a person reads in the feed. */
49 export function beatWords(did: Exclude<Drove, "off">): string {
50 switch (did) {
51 case "picked": return "started the next ready issue";
52 case "planned": return "the map was thin, so the planner was asked for more";
53 case "full": return "not started: every builder is busy";
54 case "waiting": return "not started: the crew is at its pace for now";
55 case "paused": return "not started: autonomy is paused (the week's share is spent, or nobody is set to drive)";
56 case "stalled": return "nothing started: red pulls wait for someone";
57 case "idle": return "nothing ready to ship";
58 }
59 }
60
61 /** The beat declared on a repository's default branch; re-read only when the branch moves. */
62 export async function beatOf(app: App, repo: Repo): Promise<BeatSpec | undefined> {
63 const dir = repoDir(app.reposRoot, repo.ownerName, repo.name);
64 const oid = await resolveOid(dir, repo.defaultBranch);
65 if (oid === undefined) return undefined;
66 const known = keptOf(app).declared.get(repo.id);
67 if (known?.oid === oid) return known.beat;
68 const text = await readFileAt(dir, oid, "forge.json");
69 let beat: BeatSpec | undefined;
70 if (text !== undefined) {
71 try {
72 const parsed: unknown = JSON.parse(text);
73 const read = parseBeat(typeof parsed === "object" && parsed !== null ? (parsed as Record<string, unknown>)["beat"] : undefined);
74 if (read.ok) beat = read.value;
75 else {
76 log.warn("beat not kept", { repo: `${repo.ownerName}/${repo.name}`, why: read.error });
77 // Said once, when the declaration is read (it is re-read only when the branch moves).
78 app.bus.append({ kind: "beat.ran", repoId: repo.id, payload: { every: "-", did: "invalid", says: `beat not kept: ${read.error}` } });
79 }
80 } catch {
81 beat = undefined;
82 }
83 }
84 keptOf(app).declared.set(repo.id, { oid, beat });
85 return beat;
86 }
87
88 /**
89 * One turn of a repository in the autonomy tick: without a beat it is the autonomy loop's own
90 * turn (`drive`); with one, the turn is taken only when the beat is due, and the feed hears of it
91 * when the outcome changes or something started, not every interval.
92 */
93 export async function driveOrBeat(app: App, repo: Repo, now: Date, picked: Map<string, Set<string>>, drive: Drive): Promise<Drove> {
94 let beat: BeatSpec | undefined;
95 try {
96 beat = await beatOf(app, repo);
97 } catch (error) {
98 log.warn("beat could not be read", { repo: `${repo.ownerName}/${repo.name}`, error: String(error) });
99 }
100 if (beat === undefined) return drive(app, repo, now, picked);
101 const mine = keptOf(app);
102 if (!beatDue(mine.last.get(repo.id), beat.everyMs, now.getTime())) return "waiting";
103 mine.last.set(repo.id, now.getTime());
104 const did = await drive(app, repo, now, picked);
105 if (did === "off") return did;
106 if (did === "picked" || did === "planned" || mine.said.get(repo.id) !== did) {
107 app.bus.append({ kind: "beat.ran", repoId: repo.id, payload: { every: beat.every, did, says: beatWords(did) } });
108 }
109 mine.said.set(repo.id, did);
110 return did;
111 }
src/feed/items.ts +16
@@ -780,6 +780,22 @@ function fromEvent(
780780 what: post,
781781 };
782782 }
783 // A direct push the gate held back, and a repository's beat saying what came of its turn
784 // (#13709).
785 case "app.gated": {
786 const kept = p["keptOid"];
787 return {
788 ...common,
789 kind: "release",
790 urgency: 4,
791 link: base(repo),
792 what: `a push was held back: its tests failed${typeof kept === "string" && kept !== "" ? `, ${kept.slice(0, 7)} keeps serving` : ""}`,
793 };
794 }
795 case "beat.ran": {
796 const says = p["says"];
797 return { ...common, kind: "run", link: `${base(repo)}/flow`, what: `beat: ${typeof says === "string" ? clip(says) : "ran"}` };
798 }
783799 // A tell is a post; a comment is a reply under it, in its own words.
784800 case "issue.opened": {
785801 if (number === undefined) return undefined;
src/web/studio/feed.ts +1 −1
@@ -44,7 +44,7 @@ import { appsFirst } from "../../actions/first_ask.ts";
4444
4545 /** Events that can change what the feed says. */
4646 export const FEED_LIVE =
47 "run.finished,issue.opened,issue.commented,pull.commented,pull.opened,pull.merged,pull.reviewed,pull.closed,look.taken,app.serving,issue.blocked,issue.unblocked,crew.governed,issue.closed,decision.*,thought.received,thought.placed";
47 "run.finished,issue.opened,issue.commented,pull.commented,pull.opened,pull.merged,pull.reviewed,pull.closed,look.taken,app.serving,app.gated,beat.ran,issue.blocked,issue.unblocked,crew.governed,issue.closed,decision.*,thought.received,thought.placed";
4848
4949 export const FEED_FIRST_PAGE = 20;
5050
templates/app-template.json +1 −1

Diff not shown — a line 15,877 characters wide (limit 2,000). +1 −1

View file
test/beat.test.ts added +154
@@ -0,0 +1,154 @@
1 /**
2 * The beat a repository carries (#13709): parsed at the boundary, kept on time, and heard in the
3 * feed.
4 */
5
6 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
7 import { mkdtemp, rm } from "node:fs/promises";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { autonomyTick } from "../src/crew/autonomy.ts";
11 import { createApp, type App } from "../src/app.ts";
12 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
13 import { parseBeat } from "../src/apps/beat_spec.ts";
14 import { parseManifest } from "../src/apps/manifest.ts";
15 import { beatDue, beatWords, driveOrBeat, type Drive, type Drove } from "../src/crew/beat.ts";
16 import { type Repo } from "../src/core/repos.ts";
17 import { createUser, type User } from "../src/core/users.ts";
18
19 describe("parseBeat", () => {
20 test("unset is none; every defaults to an hour", () => {
21 expect(parseBeat(undefined)).toEqual({ ok: true, value: undefined });
22 const empty = parseBeat({});
23 expect(empty.ok && empty.value?.everyMs).toBe(3_600_000);
24 });
25 test("reads numbers and units", () => {
26 const r = parseBeat({ every: "30m" });
27 expect(r.ok && r.value).toEqual({ every: "30m", everyMs: 1_800_000 });
28 const d = parseBeat({ every: "2d" });
29 expect(d.ok && d.value?.everyMs).toBe(172_800_000);
30 });
31 test("refuses what it cannot read", () => {
32 for (const bad of ["1w", "0h", "5m", "8d", "1h30m", 60, null, ["1h"], "", "-1h"]) {
33 expect(parseBeat(bad === null ? null : { every: bad }).ok).toBe(false);
34 }
35 expect(parseBeat([]).ok).toBe(false);
36 expect(parseBeat("1h").ok).toBe(false);
37 const typo = parseBeat({ evry: "1h" });
38 expect(!typo.ok && typo.error).toContain("evry");
39 });
40 test("a manifest carries it and refuses a bad one", () => {
41 const good = parseManifest(JSON.stringify({ app: { run: "bun x.ts" }, beat: { every: "1h" } }), undefined);
42 expect(good.ok && good.value.beat?.every).toBe("1h");
43 expect(parseManifest(JSON.stringify({ app: { run: "bun x.ts" }, beat: { every: "1y" } }), undefined).ok).toBe(false);
44 });
45 });
46
47 describe("beatDue and beatWords", () => {
48 test("due when never run or the interval has passed", () => {
49 expect(beatDue(undefined, 1000, 5)).toBe(true);
50 expect(beatDue(0, 1000, 999)).toBe(false);
51 expect(beatDue(0, 1000, 1000)).toBe(true);
52 });
53 test("every turn has words, and the why-not ones say not", () => {
54 expect(beatWords("picked")).toContain("started");
55 expect(beatWords("idle")).toContain("nothing ready");
56 expect(beatWords("paused")).toContain("not started");
57 });
58 });
59
60 describe("runBeats", () => {
61 let dataDir: string;
62 let app: App;
63 let alice: User;
64 let repo: Repo;
65
66 beforeAll(async () => {
67 dataDir = await mkdtemp(join(tmpdir(), "forge-beat-"));
68 app = createApp({ dataDir, port: 0 });
69 const a = await createUser(app.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", password: "a-good-passphrase", adminIfFirst: true });
70 if (!a.ok) throw new Error(a.error);
71 alice = a.value;
72 const made = await createRepo(app, alice, { name: "beating", visibility: "private" });
73 if (!made.ok) throw new Error(String(made.error));
74 repo = made.value;
75 const seeded = await commitToBranch(app, alice, repo, { message: "Start", files: [{ path: "forge.json", content: JSON.stringify({ beat: { every: "1h" } }) }] });
76 if (!seeded.ok) throw new Error(JSON.stringify(seeded.error));
77 });
78 afterAll(async () => {
79 app.close();
80 await rm(dataDir, { recursive: true, force: true });
81 });
82
83 const seen: string[] = [];
84 const turn = (did: Drove, turns: string[]): Drive => async (_a, r) => {
85 turns.push(r.id);
86 return did;
87 };
88
89 test("a due beat takes one turn, waits its hour, and tells the feed only when the outcome changes", async () => {
90 app.bus.subscribe((e) => {
91 if (e.kind === "beat.ran") seen.push(String(e.payload["says"]));
92 });
93 const turns: string[] = [];
94 const t0 = new Date("2026-11-15T10:00:00Z");
95 const at = (min: number): Date => new Date(t0.getTime() + min * 60_000);
96 const go = (min: number, did: Drove): Promise<Drove> => driveOrBeat(app, repo, at(min), new Map(), turn(did, turns));
97 expect(await go(0, "idle")).toBe("idle");
98 expect(await go(59, "idle")).toBe("waiting");
99 expect(turns).toHaveLength(1);
100 expect(await go(60, "idle")).toBe("idle");
101 expect(turns).toHaveLength(2);
102 expect(seen).toEqual(["nothing ready to ship"]); // the same outcome twice: said once
103 expect(await go(120, "full")).toBe("full");
104 expect(await go(180, "picked")).toBe("picked");
105 expect(await go(240, "picked")).toBe("picked"); // something started: said each time
106 expect(seen).toHaveLength(4);
107 expect(seen[3]).toContain("started");
108 // Autonomy off for the turn: nothing is said.
109 expect(await go(300, "off")).toBe("off");
110 expect(seen).toHaveLength(4);
111 });
112
113 test("a repository without a beat takes the autonomy loop's own turn every time", async () => {
114 const made = await createRepo(app, alice, { name: "plain", visibility: "private" });
115 if (!made.ok) throw new Error(String(made.error));
116 const seeded = await commitToBranch(app, alice, made.value, { message: "Start", files: [{ path: "README.md", content: "x\n" }] });
117 if (!seeded.ok) throw new Error(JSON.stringify(seeded.error));
118 const turns: string[] = [];
119 const t = new Date("2026-11-15T10:00:00Z");
120 await driveOrBeat(app, made.value, t, new Map(), turn("idle", turns));
121 await driveOrBeat(app, made.value, t, new Map(), turn("idle", turns));
122 expect(turns).toHaveLength(2);
123 });
124
125 test("a beat that cannot be read is said once in the feed, and the repository stays on the loop", async () => {
126 const made = await createRepo(app, alice, { name: "badbeat", visibility: "private" });
127 if (!made.ok) throw new Error(String(made.error));
128 const seeded = await commitToBranch(app, alice, made.value, { message: "Start", files: [{ path: "forge.json", content: JSON.stringify({ beat: { every: "1y" } }) }] });
129 if (!seeded.ok) throw new Error(JSON.stringify(seeded.error));
130 const before = seen.length;
131 const turns: string[] = [];
132 const t = new Date("2026-11-15T10:00:00Z");
133 await driveOrBeat(app, made.value, t, new Map(), turn("idle", turns));
134 await driveOrBeat(app, made.value, t, new Map(), turn("idle", turns));
135 expect(turns).toHaveLength(2);
136 expect(seen.slice(before)).toHaveLength(1);
137 expect(seen[before]).toContain("beat not kept");
138 });
139
140 test("the autonomy tick gives a beat-carrying repository its turn once per beat, in its own pass", async () => {
141 app.db.query("UPDATE repo SET autonomy = 'autonomous', autonomy_by = ?2 WHERE id = ?1").run(repo.id, alice.id);
142 const heard: string[] = [];
143 app.bus.subscribe((e) => {
144 if (e.kind === "beat.ran" && e.repoId === repo.id) heard.push(String(e.payload["did"]));
145 });
146 const t = new Date("2026-12-01T10:00:00Z");
147 await autonomyTick(app, t, async () => 0);
148 expect(heard).toHaveLength(1);
149 // A minute later the beat is not due: the repository's turn is "waiting", and nothing new is
150 // said.
151 await autonomyTick(app, new Date(t.getTime() + 60_000), async () => 0);
152 expect(heard).toHaveLength(1);
153 });
154 });
test/feed-reasons.test.ts +12
@@ -22,6 +22,7 @@ import { createUser, type User } from "../src/core/users.ts";
2222 import { Affinity } from "../src/feed/affinity.ts";
2323 import { plainWords } from "../src/feed/plain.ts";
2424 import { candidatesFor, eventCandidates, type Candidate, type FeedKind } from "../src/feed/items.ts";
25 import { FEED_LIVE } from "../src/web/studio/feed.ts";
2526 import { rank, REASON_MAX } from "../src/feed/rank.ts";
2627 import { quietLogs } from "./helpers/globals.ts";
2728
@@ -113,6 +114,8 @@ const events = (): Record<string, ForgeEvent> => ({
113114 tell: event("issue.opened", { number: 8 }),
114115 reply: event("issue.commented", { number: 8, body: `see ${ULID} and ${OID}` }),
115116 map: event("issue.blocked", { number: 8, blockedBy: 9 }),
117 gated: event("app.gated", { app: "site", env: "production", commitOid: OID, keptOid: "b".repeat(40), why: "1 fail" }),
118 beat: event("beat.ran", { every: "1h", did: "idle", says: "nothing ready to ship" }),
116119 syncFailed: event("adapter.sync_failed", { reason: `repo ${UUID} refused ${ULID}` }),
117120 });
118121
@@ -144,6 +147,15 @@ test("the builders emit every kind", () => {
144147 expect(kinds.has(k as FeedKind)).toBe(true);
145148 });
146149
150 test("a gated push and a beat are feed items, and the live feed hears them", () => {
151 const cs = built();
152 expect(cs.find((c) => c.what.startsWith("a push was held back"))?.what).toContain("bbbbbbb keeps serving");
153 expect(cs.find((c) => c.what === "beat: nothing ready to ship")?.kind).toBe("run");
154 const live = FEED_LIVE.split(",");
155 expect(live).toContain("app.gated");
156 expect(live).toContain("beat.ran");
157 });
158
147159 test("an id in an event's payload never reaches the clause", () => {
148160 const cs = built();
149161 const sync = cs.find((c) => c.what.startsWith("sync failed"))!;
test/job-slices.test.ts +1
@@ -278,6 +278,7 @@ const UNTIMED: Record<string, string> = {
278278 prompts: "a closure in startPromptRoutine; the fixture has no prompt routines",
279279 timers: "fires due app timers over HTTP to the apps; the fixture has no apps",
280280 "ledger-git": "commits the cost ledger chain to the ledger branch at most hourly: a git commit and signature in a data directory; the fixture has no ledger directory, and test/ledger-git.test.ts runs the tick",
281 "push-gate-budget": "a one-shot timer in apps/push_gate.ts that gives up on a push gate's tests at its budget: one resolve of a promise; push-gate tests drive the budget directly",
281282 "run-log-flush": "a one-shot timer in runs/log.ts that appends buffered lines to the run log files: I/O, no fixture runs",
282283 "spawn-restart": "a one-shot backoff timer in spawn/stopped.ts that restarts a stopped spawn; the fixture has no spawns",
283284 "origin-broker-sweep": "a one-shot timer a minute after a brokered sign-in, and once on boot: one DELETE of its spent and expired rows; the fixture brokers none",
test/push-gate.test.ts added +199
@@ -0,0 +1,199 @@
1 /**
2 * The gate on a direct push (#13709), through a live apps runtime: a push to the default branch
3 * whose declared tests fail does not switch, and the last green commit keeps serving.
4 */
5
6 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
7 import { mkdtemp, rm } from "node:fs/promises";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { createApp, type App } from "../src/app.ts";
11 import { AppsRuntime } from "../src/apps/runtime.ts";
12 import { gateBudgetMs, gatesPush, gateSummary, withinBudget } from "../src/apps/push_gate.ts";
13 import { newApp } from "../src/actions/apps.ts";
14 import { getAppByRepo } from "../src/core/apps.ts";
15 import { commitToBranch } from "../src/actions/repos.ts";
16 import type { ForgeEvent } from "../src/core/events.ts";
17 import type { Repo } from "../src/core/repos.ts";
18 import { listRuns, type Run } from "../src/core/runs.ts";
19 import { tellGated } from "../src/apps/gate_notice.ts";
20 import { setNotifyUrl } from "../src/crew/notify.ts";
21 import { ok } from "../src/kernel/result.ts";
22 import { createUser, type User } from "../src/core/users.ts";
23 import { serve, type Servers } from "../src/http/server.ts";
24
25 describe("gatesPush", () => {
26 test("only a direct push to production with something to keep and a declared test", () => {
27 expect(gatesPush("production", "push", true, true)).toBe(true);
28 expect(gatesPush("production", "push", false, true)).toBe(false);
29 expect(gatesPush("production", "push", true, false)).toBe(false);
30 expect(gatesPush("production", "merge", true, true)).toBe(false);
31 expect(gatesPush("pull-3", "push", true, true)).toBe(false);
32 });
33 test("the summary names the commit that keeps serving", () => {
34 const line = gateSummary("test failed: exit 1\nmore", "abcdef0123456789");
35 expect(line).toContain("abcdef0 keeps serving");
36 expect(line).not.toContain("more");
37 });
38 });
39
40 describe("the gate's budget", () => {
41 test("a suite that outlives it is red; one inside it keeps its verdict", async () => {
42 const hung = await withinBudget(new Promise(() => {}), 20);
43 expect(!hung.ok && hung.error).toContain("did not finish");
44 expect((await withinBudget(Promise.resolve(ok()), 1000)).ok).toBe(true);
45 });
46 test("the budget is ten minutes unless the environment sets a sane one", () => {
47 expect(gateBudgetMs(undefined)).toBe(600_000);
48 expect(gateBudgetMs("5000")).toBe(5000);
49 for (const bad of ["0", "10", "abc", "-5", "99999999", "1e3"]) expect(gateBudgetMs(bad)).toBe(600_000);
50 });
51 });
52
53 describe("tellGated", () => {
54 test("sends one line to the owner's address; a failing sender, or no address, never throws", async () => {
55 const dataDir = await mkdtemp(join(tmpdir(), "forge-gate-notice-"));
56 const app = createApp({ dataDir, port: 0 });
57 try {
58 const u = await createUser(app.db, { origin: "seed", trusted: true, username: "bob", email: "[email protected]", password: "a-good-passphrase", adminIfFirst: true });
59 if (!u.ok) throw new Error(u.error);
60 const sent: string[][] = [];
61 const send = async (url: string, line: string, h?: Record<string, string>): Promise<boolean> => {
62 sent.push([url, line, h?.["title"] ?? ""]);
63 return true;
64 };
65 expect(await tellGated(app.db, u.value.id, "held back", send)).toBe(false);
66 expect(sent).toHaveLength(0);
67 expect(setNotifyUrl(app.db, u.value.id, "https://ntfy.example.com/topic").ok).toBe(true);
68 expect(await tellGated(app.db, u.value.id, "held back", send)).toBe(true);
69 expect(sent).toEqual([["https://ntfy.example.com/topic", "held back", "forge"]]);
70 const boom = async (): Promise<boolean> => {
71 throw new Error("down");
72 };
73 expect(await tellGated(app.db, u.value.id, "held back", boom)).toBe(false);
74 } finally {
75 app.close();
76 await rm(dataDir, { recursive: true, force: true });
77 }
78 });
79 });
80
81 describe("a direct push is gated on its tests", () => {
82 let dataDir: string;
83 let app: App;
84 let servers: Servers;
85 let runtime: AppsRuntime;
86 let alice: User;
87 let repo: Repo;
88
89 const when = <T>(look: (event?: ForgeEvent) => T | undefined, what: string): Promise<T> =>
90 new Promise((resolve, reject) => {
91 const done = (value: T): void => {
92 unsubscribe();
93 clearTimeout(timer);
94 resolve(value);
95 };
96 const unsubscribe = app.bus.subscribe((event) => {
97 const value = look(event);
98 if (value !== undefined) done(value);
99 });
100 const timer = setTimeout(() => {
101 unsubscribe();
102 reject(new Error(`${what} did not happen in time`));
103 }, 25_000);
104 const now = look();
105 if (now !== undefined) done(now);
106 });
107
108 const settledAt = (oid: string): Promise<Run> =>
109 when(
110 () =>
111 listRuns(app.db, repo.id, { kind: "deploy", limit: 50 }).find(
112 (r) => r.env === "production" && r.commitOid === oid && (r.state === "ok" || r.state === "failed"),
113 ),
114 "the deploy settling",
115 );
116
117 const push = async (test: string): Promise<string> => {
118 const committed = await commitToBranch(app, alice, repo, {
119 branch: repo.defaultBranch,
120 base: repo.defaultBranch,
121 message: `test: ${test}`,
122 files: [{ path: "forge.json", content: `${JSON.stringify({ app: { run: "bun run src/server.ts", test }, note: test })}\n` }],
123 });
124 if (!committed.ok) throw new Error(JSON.stringify(committed.error));
125 return committed.value.commitOid;
126 };
127
128 beforeAll(async () => {
129 dataDir = await mkdtemp(join(tmpdir(), "forge-gate-"));
130 app = createApp({ dataDir, port: 0 });
131 servers = await serve(app);
132 runtime = new AppsRuntime(app, { domain: "apps.test", scheme: "http", publicPort: servers.port, forgeUrl: servers.url, concurrency: 2 });
133 await runtime.start();
134 const created = await createUser(app.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", password: "a-good-passphrase", adminIfFirst: true });
135 if (!created.ok) throw new Error(created.error);
136 alice = created.value;
137 const made = await newApp(app, alice, { name: "gated", visibility: "public" });
138 if (!made.ok) throw new Error(made.error);
139 repo = made.value.repo;
140 const first = await when(
141 () => listRuns(app.db, repo.id, { kind: "deploy", limit: 5 }).find((r) => r.env === "production" && r.state === "ok"),
142 "first production deploy",
143 );
144 expect(first.state).toBe("ok");
145 }, 40_000);
146
147 afterAll(async () => {
148 await runtime.stop();
149 servers.stop();
150 app.close();
151 await rm(dataDir, { recursive: true, force: true });
152 });
153
154 test("red tests keep the last green commit serving and say so; green switches", async () => {
155 const record = getAppByRepo(app.db, repo.id)!;
156 const green = await push("true");
157 expect((await settledAt(green)).state).toBe("ok");
158 const servingBefore = runtime.liveFor(record, "production")?.commitOid;
159 expect(servingBefore).toBe(green);
160
161 const gated = when(
162 (e) => (e?.kind === "app.gated" && e.repoId === repo.id ? e : undefined),
163 "the gated event",
164 );
165 const red = await push("exit 1");
166 const failed = await settledAt(red);
167 expect(failed.state).toBe("failed");
168 expect(failed.summary).toContain("held back");
169 expect(failed.summary).toContain(`${green.slice(0, 7)} keeps serving`);
170 const event = await gated;
171 expect(event.payload["keptOid"]).toBe(green);
172 expect(runtime.liveFor(record, "production")?.commitOid).toBe(green);
173
174 });
175
176 test("a suite that outlives the budget is red too, and does not hold the deploy", async () => {
177 const record = getAppByRepo(app.db, repo.id)!;
178 const green = runtime.liveFor(record, "production")?.commitOid;
179 process.env["FORGE_GATE_BUDGET_MS"] = "1000";
180 try {
181 const hung = await push("sleep 30");
182 const slow = await settledAt(hung);
183 expect(slow.state).toBe("failed");
184 expect(slow.summary).toContain("held back");
185 expect(slow.summary).toContain("did not finish");
186 expect(runtime.liveFor(record, "production")?.commitOid).toBe(green);
187 } finally {
188 delete process.env["FORGE_GATE_BUDGET_MS"];
189 }
190
191 });
192
193 test("a fixed push switches in again", async () => {
194 const record = getAppByRepo(app.db, repo.id)!;
195 const fixed = await push("true");
196 expect((await settledAt(fixed)).state).toBe("ok");
197 expect(runtime.liveFor(record, "production")?.commitOid).toBe(fixed);
198 }, 60_000);
199 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs