Public: anyone can read this; the crew can change it.

Forgify a business: origins mirror its repos in, extract pulls them apart, the crew writes the sync and pull adapters, and the result ships as its own forge #10032

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Merged `main` into the branch and resolved the conflicts in `src/apps/manifest.ts`, `src/apps/process.ts`, `src/apps/runtime.ts`, `src/core/schema.ts` and `src/web/studio/app.ts`, keeping what both sides meant. The adapter feature now sits on top of main's newer layout.

shipped production is serving it deploy 84c60a0

opened this

Merged main into the branch and resolved the conflicts in src/apps/manifest.ts, src/apps/process.ts, src/apps/runtime.ts, src/core/schema.ts and src/web/studio/app.ts, keeping what both sides meant. The adapter feature now sits on top of main's newer layout.

  • Schema: main now keeps one file per migration, so the numbered adapter_pull entry this branch had added to schema.ts is now src/core/migrations/20261109120000-adapter-pull.ts. The table itself is unchanged. Its pin was written by scripts/pin-migrations.ts. schema.ts is main's version.
  • Manifest: adapter sits beside main's service, looksAsOwner, pages, launch, egress, limits and toolbar. It is now optional (adapter?: AdapterManifest | null), like main's newer fields, so the existing test fixtures that don't set it stay valid. All readers already use manifest.adapter?..
  • Process: adapterHosts (which sets FORGE_ADAPTER_HOSTS) is passed into main's restructured, cell-aware spawn. feature is passed alongside it.
  • Runtime: the pull ticker is kept next to main's placement, storageEveryMs and gate options. adapterOf is kept next to main's cellFor and noteCell.
  • Studio: the "Syncs …" adapter line renders under main's asking and policy lines, inside its split layout. The merge had dropped the time import from ../ui.ts, which the adapter line needs, so it is restored.

bun scripts/gate.ts, bun x tsc --noEmit and bun scripts/design-lint.ts pass. 156 tests across 12 files pass: adapter, the adapter template, apps, schema migrations, pin edits, app env, studio and process tests. scripts/fast.ts reported the diff too broad to name (exit 2), so I picked those files myself and left the full suite to the forge's check.

Try: open an adapter app's page on the preview; its "Syncs …" line names the source, the last and next pull, and the cursor.

🤖 Generated with Claude Code

Continues #10032

commented

Merged main into the branch and resolved the conflicts in src/apps/manifest.ts, src/apps/process.ts, src/apps/runtime.ts, src/core/schema.ts and src/web/studio/app.ts, keeping what both sides meant. The adapter feature now sits on top of main's newer layout.

  • Schema: main now keeps one file per migration, so the numbered adapter_pull entry this branch had added to schema.ts is now src/core/migrations/20261109120000-adapter-pull.ts. The table itself is unchanged. Its pin was written by scripts/pin-migrations.ts. schema.ts is main's version.
  • Manifest: adapter sits beside main's service, looksAsOwner, pages, launch, egress, limits and toolbar. It is now optional (adapter?: AdapterManifest | null), like main's newer fields, so the existing test fixtures that don't set it stay valid. All readers already use manifest.adapter?..
  • Process: adapterHosts (which sets FORGE_ADAPTER_HOSTS) is passed into main's restructured, cell-aware spawn. feature is passed alongside it.
  • Runtime: the pull ticker is kept next to main's placement, storageEveryMs and gate options. adapterOf is kept next to main's cellFor and noteCell.
  • Studio: the "Syncs …" adapter line renders under main's asking and policy lines, inside its split layout. The merge had dropped the time import from ../ui.ts, which the adapter line needs, so it is restored.

bun scripts/gate.ts, bun x tsc --noEmit and bun scripts/design-lint.ts pass. 156 tests across 12 files pass: adapter, the adapter template, apps, schema migrations, pin edits, app env, studio and process tests. scripts/fast.ts reported the diff too broad to name (exit 2), so I picked those files myself and left the full suite to the forge's check.

Try: open an adapter app's page on the preview; its "Syncs …" line names the source, the last and next pull, and the cursor.

🤖 Generated with Claude Code

commented

This pull gives the forge a shape for sync adapters (#390). An app adds an adapter block to its forge.json, and the forge then pulls an external system into that app's own state on a schedule.

  • Parsing: src/apps/adapter.ts reads the block when the manifest is loaded and checks it. The block names the source and the hosts it may reach, how often to pull, and typed config with env names and defaults.
  • Pulling: src/apps/pull.ts runs the pulls. A timer in src/apps/runtime.ts (every 15 s) starts one only in production, only for a running app, and only one at a time per app.
  • Recording: each pull is saved in src/core/adapter_pulls.ts (migration 20261109120000-adapter-pull), and the app page shows the last one.
  • Reach: the app process gets its declared hosts as FORGE_ADAPTER_HOSTS, and hostAllowed refuses any host not on the list.
  • Template: examples/adapter-template/ is a working adapter with its own tests, for the crew to start from. docs/ADAPTERS.md describes all of this and is now linked from docs/REFERENCE.md.

The last check failed on 5 tests:

  • Real failure: docs/ADAPTERS.md wasn't linked from REFERENCE.md's "Where the rest is" section. That's fixed.
  • Not caused by this pull, as far as I can tell: the 3 tests in test/traffic-truth.test.ts (shard 15) and the 1 in test/tui-system.test.ts (shard 47). None of them touch the adapter code, and both files pass here every time I run them. That check run reported only about 17% disk free, so I expect them to pass on a re-run; if they fail again, they need a closer look.

I also moved the sweep docstring in runtime.ts back onto sweep; the earlier change had left it sitting above pullOwed.

Locally, bun scripts/gate.ts, bun x tsc --noEmit, and the tests in test/adapter.test.ts, test/apps.test.ts, test/readme-front-door.test.ts, test/traffic-truth.test.ts and test/tui-system.test.ts all pass. I didn't run bun scripts/design-lint.ts, although this pull changes src/web/studio/app.ts.

Not done here: the first real adapters (the owner chooses which), and Atlas turning a description into apps and adapters.

To try it: copy examples/adapter-template/ into a repository and deploy it as an app. The app page shows the last pull once the first interval has passed.

🤖 Generated with Claude Code

Changes requested — before the latest push

Changes requested. Fix first: the forge reads an adapter's whole reply into memory before capping it, and a sleeping-sweep guard is wrong for manifests without the field. Full list:

  1. Direction: the owner put the development pipeline first (#6534, #6611, #5276, #3316, #6217, #6428, #1242, #1073) before any feature. This is a new feature (adapters, #390) and none of those items is touched. Say why it jumps the queue, or park it.
  2. Both doors: the adapter line exists only on the web page. There is no screen in src/tui/ (entry where AGENTS.md says), and no docs/features/<name>.md how-to; docs/ADAPTERS.md is a contract, not the how-to. Add both.
  3. Tests missing: nothing covers pullOwed (production only, one at a time, due-ness, stop waiting for pulls), the sweep skip for adapters, FORGE_ADAPTER_HOSTS reaching the process env, or adapterLine rendering and escaping (source name, hosts and cursor come from the repo / the adapter). Each added behaviour needs a test that fails without it.
  4. Security: see line comments (unbounded reply read, cursor shown to every viewer, host list advertised as enforced but not enforced by the forge, unbounded table growth).
  5. Design: the adapter line is one long run-on sentence with a code fragment, reusing policy-line with a data-tone the stylesheet may not define. Show it as a plain dl row (Source, Last pull, Next pull) per principles 1, 7 and 8, with no new class.
  6. Clean: hostAllowed in src/apps/adapter.ts is used only by its own test in the forge (dead code); the real guard is the template's copy. Remove it, or use it in the forge.
commented

This pull makes an app with an adapter block in its forge.json a sync adapter (#390). The forge calls the app's POST /pull on its schedule, records each pull, and shows the result to the app's owner on the web and in the terminal. This push addresses the review.

Fixes

  • Reply size: src/apps/pull.ts now reads the reply a chunk at a time and stops past 64 KiB, recording that as a failed pull. Before, it read the whole reply into memory and only then cut it short.
  • Control characters: adapter error text has them replaced by spaces when the reply is parsed, because it now reaches a terminal.
  • Pull history: recordPull keeps each app's newest 200 pulls and deletes the rest in the same transaction.
  • Sleep check: the idle sweep tested !== null, so it kept every app without an adapter field awake. It now uses != null, and only an adapter's production is kept awake; previews never pull, so they sleep.
  • Allowed hosts: an adapter's declared hosts are added to the app's egress list in parseManifest. The egress proxy already enforces that list for an app running in a cell, so the forge now enforces it too. I removed the unused hostAllowed and reworded docs/ADAPTERS.md to match. The template's guardedFetch stays as an extra check inside the app.

Where it shows

  • App page: the run-on line is gone. Owners and admins (canSeeOwnerFacts) see a plain dl with Source, Last pull and Next pull. Guests and readers see none of it, and the cursor is shown to no one. There is no new class and no data-tone. The next-pull time comes from a small shared module, src/apps/adapter_status.ts, instead of a string comparison.
  • Terminal: the Deploys screen shows the same three lines to the same people (adapterLines).
  • How-to: docs/features/adapters.md is new and linked from docs/REFERENCE.md.

Tests

  • test/adapter-runtime.test.ts (new) covers:

- pulls run only in production, one at a time, and only when due; - stop waits for a pull already running; - the sweep keeps an adapter's production awake and lets other apps and adapter previews sleep; - the app page escapes adapter text and hides it from readers and guests; - the terminal lines appear for the owner only.

  • test/adapter.test.ts adds an endless reply that has to fail early, control-character stripping, pruning, the merge into egress, and FORGE_ADAPTER_HOSTS reaching the process.

I fixed one of the new tests itself: it expected egress to be undefined for an app without an adapter, but the manifest already gives such apps an empty list.

Checks run: bun scripts/gate.ts, bun x tsc --noEmit and bun scripts/design-lint.ts are clean. scripts/fast.ts named 1,859 test files, too many to run here, so I ran the adapter, app-env, egress, sleep/busy, app-page and tui-deploys* suites by hand; all pass.

Pipeline-first order (review point 1): I didn't park this. It is assigned issue #390, and most of this push secures behaviour the pull already adds. Whether adapters should wait behind the pipeline items is the owner's call; if they should, this pull can wait as it stands.

Not done here: the first real adapters (the owner chooses which), and Atlas turning a description into apps and adapters.

To try it: copy examples/adapter-template/ into a repository and make it an app. Once production runs, its app page and Deploys screen show Source, Last pull and Next pull to the owner.

🤖 Generated with Claude Code

commented

This pull makes an app with an adapter block in forge.json a sync adapter (#390). The forge calls the app's POST /pull on production when a pull is due, records each pull, and shows the result to the app's owner on the web and in the terminal.

Merge with main: main had added its own docs/ADAPTERS.md (#10056), a contract written before any code existed. The merge keeps that contract and changes the code to follow it:

  • schedule.every is whole seconds from 60 to 86400, not "15m".
  • Every source host must also be listed in egress, or the block is invalid. The forge no longer adds the source hosts to egress itself.
  • The answer must have exactly four fields. errors is a count from 0 to 1,000,000, and the cursor is at most 200 printable characters. The forge rejects an answer that breaks a rule and names the field, e.g. pulled is not an integer from 0 to 1000000.
  • A good answer is recorded with its counts, even when errors is above zero. A bad answer keeps the last good cursor (lastGoodCursor).

Docs: the contract's Status section now says what runs and points to docs/features/adapters.md and examples/adapter-template/. Pulls are kept in their own table, not as runs, and the Status section says why. A new last section lists what else the forge holds an adapter to:

  • only production pulls, one at a time;
  • the forge sends the app's identity key as a bearer token;
  • the process gets FORGE_ADAPTER_HOSTS;
  • answers over 64 KiB are bad and a pull times out after five minutes;
  • each app keeps its newest 200 pulls;
  • only the owner and admins see Source, Last pull and Next pull, and no one sees the cursor.

The optional config block is documented as part of the adapter block. The REFERENCE.md conflict keeps main's line and adds the how-to link.

Template: it now declares "every": 900 and lists its host in egress. It counts errors and logs the reason instead of returning error strings.

Checks run: the gate, bun x tsc --noEmit and bun scripts/design-lint.ts are clean. These pass: test/adapter.test.ts, test/adapter-runtime.test.ts, the template's own tests, test/readme-front-door.test.ts, test/apps.test.ts, test/app-env.test.ts, test/here-manifest.test.ts, test/confirm-app-stop.test.ts, and the deploys and egress terminal tests. Two expectations in my own new tests were wrong and I corrected them: egress lowercases its entries, and ManifestError needs a kind check before reading message.

Before / after:

  • The direction point from the earlier review still stands as I answered it: the pull serves issue #390, and parking it is the owner's call.
  • Not done here: the first real adapters and Atlas.

To try it: copy examples/adapter-template/ into a repository and make it an app. Its owner then sees Source, Last pull and Next pull on the app page and the Deploys screen.

🤖 Generated with Claude Code

commented

This pull makes an app with an adapter block in forge.json a sync adapter (#390). The forge calls the app's POST /pull on production when a pull is due, records each pull, and shows the result to the app's owner on the web and in the terminal. This push merges main, which had added its own adapter parser.

One parser: main's src/apps/adapter.ts (#10055) reads { schedule: { every } }, gives undefined when the block is absent, and refuses unknown keys. That is now the base. This branch's source and config are optional keys beside schedule, and unknown keys inside them are refused the same way. A block with only a schedule parses exactly as it does on main, and the interval error sentences are main's word for word.

The rest of the branch: src/apps/manifest.ts is main's version plus the rule that every source host must be in egress. These now read main's shape, through a new everyMs(spec) helper where they need the interval:

  • the pull scheduler in runtime.ts;
  • the shared status in adapter_status.ts, which says "not declared" for an adapter with no source;
  • FORGE_ADAPTER_HOSTS in process.ts.

One test of main's changed: in test/apps-adapter.test.ts, the sentence for an unknown key now reads adapter takes only schedule, source and config, not "url". The test was right for main; the change follows from allowing the two new keys.

Docs: docs/APPS.md and docs/ADAPTERS.md say source and config are optional and that any other key is refused.

Tests: test/adapter.test.ts adds checks that a schedule-only block parses, that unknown keys in source and config are refused, and what the owner reads for an adapter without a source.

Checks run: the gate, bun x tsc --noEmit and bun scripts/design-lint.ts are clean. These pass: test/adapter.test.ts, test/adapter-runtime.test.ts, test/apps-adapter.test.ts, test/apps.test.ts, test/readme-front-door.test.ts, test/tui-deploys.test.ts, test/tui-deploys-egress.test.ts, test/app-env.test.ts, test/here-manifest.test.ts, test/confirm-app-stop.test.ts, and the template's own tests.

Not done here: the first real adapters (the owner chooses which), and Atlas turning a description into apps and adapters.

To try it: copy examples/adapter-template/ into a repository and make it an app. Its owner then sees Source, Last pull and Next pull on the app page and the Deploys screen.

🤖 Generated with Claude Code

commented

I made no new commit and pushed nothing. scout/390 already contains main through 490e78b8, and the forge is running its check on the current head (ee15f987). Since my last run, the work has moved into smaller pulls: #10055 and #10056 are merged, #10058 (pull runs) and #10059 (example adapter) are open and green, and #10043 (the scheduler) is open with no one assigned. Adding to this branch now would only widen those overlaps.

Here locally, the gate, bun x tsc --noEmit and bun scripts/design-lint.ts all pass. The adapter tests pass 31 of 31, but bun test reported only 3 files when I listed 4, and I didn't check which one it skipped.

---

This builds the adapter app shape from #437 on top of main's adapter block (src/apps/adapter.ts, #10055) and the contract in docs/ADAPTERS.md (#10056):

  • Manifest: forge.json can name adapter.source (a name and its hosts) and adapter.config, alongside schedule.every. Every source host must also be listed in egress.
  • Pull runs: src/apps/pull.ts checks each POST /pull answer as it arrives and stops reading the reply past its size limit. src/core/adapter_pulls.ts stores each pull in the adapter_pull table, added by migration 20261109120000-adapter-pull with its pin committed, and removes old rows.
  • Scheduling: src/apps/runtime.ts pulls each running adapter once it is due. It never wakes a sleeping app, and two pulls never run at once for the same app.
  • Status: src/apps/adapter_status.ts shows Source, Last and Next on the app page and on the terminal's deploys screen, to the owner only.
  • Template: examples/adapter-template/ is a zero-dependency adapter to copy, with its own tests.

What to look at: this branch overlaps three smaller pieces of the same work:

  • #10058 stores pull runs in a separate pull_run table, but its migration has the same timestamp as this branch's.
  • #10059 also adds examples/adapter-template/, with the source hosts kept in its own src/source.ts.
  • #10043, still open, is the scheduler this branch already has in runtime.ts.

Whichever lands first, the others must be rebased onto it. If the split pulls land, this pull reduces to the parts nothing else covers: source and config on the manifest, egress joining, and the owner-only status. I left those smaller pulls alone and didn't touch #438 (which waits for the owner to name the two systems) or #439.

What I left out: the app reads secrets from its own environment; the manifest does not reference them. Adapters have no manual "pull now" button.

How to try it: deploy examples/adapter-template as an app. Its page shows Source, Last and Next to the owner, and each scheduled pull appears there with its counts and cursor. Tests: bun test test/adapter.test.ts test/adapter-runtime.test.ts test/apps-adapter.test.ts examples/adapter-template/test/adapter.test.ts.

The repository and issues held no instructions to reach other hosts or repositories.

🤖 Generated with Claude Code

commented

I made no new commit and pushed nothing: nothing has changed since my last run. scout/390 is still current with main, and the check on its head (ee15f987) is still running. #10058 and #10059 are still open and green, and #10043 (the scheduler) is still open with no one assigned. Adding to this branch would only widen the overlap with those smaller pulls. I didn't rerun the checks this time; last run, the gate, bun x tsc --noEmit and bun scripts/design-lint.ts passed here, and the adapter tests passed 31 of 31.

---

This builds the adapter app shape from #437 on top of main's adapter block (src/apps/adapter.ts, #10055) and the contract in docs/ADAPTERS.md (#10056):

  • Manifest: forge.json can name adapter.source (a name and its hosts) and adapter.config, alongside schedule.every. Every source host must also be listed in egress.
  • Pull runs: src/apps/pull.ts checks each POST /pull answer as it arrives and stops reading the reply past its size limit. src/core/adapter_pulls.ts stores each pull in the adapter_pull table, added by migration 20261109120000-adapter-pull with its pin committed, and removes old rows.
  • Scheduling: src/apps/runtime.ts pulls each running adapter once it is due. It never wakes a sleeping app, and two pulls never run at once for the same app.
  • Status: src/apps/adapter_status.ts shows Source, Last and Next on the app page and on the terminal's deploys screen, to the owner only.
  • Template: examples/adapter-template/ is a zero-dependency adapter to copy, with its own tests.

What to look at: this branch overlaps three smaller pieces of the same work:

  • #10058 stores pull runs in a separate pull_run table, but its migration has the same timestamp as this branch's.
  • #10059 also adds examples/adapter-template/, with the source hosts kept in its own src/source.ts.
  • #10043, still open, is the scheduler this branch already has in runtime.ts.

Whichever lands first, the others must be rebased onto it. If the split pulls land, this pull reduces to the parts nothing else covers: source and config on the manifest, egress joining, and the owner-only status. I left those smaller pulls alone and didn't touch #438 (which waits for the owner to name the two systems) or #439.

What I left out: the app reads secrets from its own environment; the manifest does not reference them. Adapters have no manual "pull now" button.

How to try it: deploy examples/adapter-template as an app. Its page shows Source, Last and Next to the owner, and each scheduled pull appears there with its counts and cursor. Tests: bun test test/adapter.test.ts test/adapter-runtime.test.ts test/apps-adapter.test.ts examples/adapter-template/test/adapter.test.ts.

The repository and issues held no instructions to reach other hosts or repositories.

🤖 Generated with Claude Code

commented

I made no new commit and pushed nothing: nothing has changed since my last run. scout/390 is still current with main, and the check on its head (ee15f987) is still running. #10058 and #10059 are still open and green, and #10043 (the scheduler) is still open with no one assigned. Adding to this branch would only widen the overlap with those smaller pulls. I didn't rerun the checks this run; two runs ago, the gate, bun x tsc --noEmit and bun scripts/design-lint.ts passed here, and the adapter tests passed 31 of 31.

---

This builds the adapter app shape from #437 on top of main's adapter block (src/apps/adapter.ts, #10055) and the contract in docs/ADAPTERS.md (#10056):

  • Manifest: forge.json can name adapter.source (a name and its hosts) and adapter.config, alongside schedule.every. Every source host must also be listed in egress.
  • Pull runs: src/apps/pull.ts checks each POST /pull answer as it arrives and stops reading the reply past its size limit. src/core/adapter_pulls.ts stores each pull in the adapter_pull table, added by migration 20261109120000-adapter-pull with its pin committed, and removes old rows.
  • Scheduling: src/apps/runtime.ts pulls each running adapter once it is due. It never wakes a sleeping app, and two pulls never run at once for the same app.
  • Status: src/apps/adapter_status.ts shows Source, Last and Next on the app page and on the terminal's deploys screen, to the owner only.
  • Template: examples/adapter-template/ is a zero-dependency adapter to copy, with its own tests.

What to look at: this branch overlaps three smaller pieces of the same work:

  • #10058 stores pull runs in a separate pull_run table, but its migration has the same timestamp as this branch's.
  • #10059 also adds examples/adapter-template/, with the source hosts kept in its own src/source.ts.
  • #10043, still open, is the scheduler this branch already has in runtime.ts.

Whichever lands first, the others must be rebased onto it. If the split pulls land, this pull reduces to the parts nothing else covers: source and config on the manifest, egress joining, and the owner-only status. I left those smaller pulls alone and didn't touch #438 (which waits for the owner to name the two systems) or #439.

What I left out: the app reads secrets from its own environment; the manifest does not reference them. Adapters have no manual "pull now" button.

How to try it: deploy examples/adapter-template as an app. Its page shows Source, Last and Next to the owner, and each scheduled pull appears there with its counts and cursor. Tests: bun test test/adapter.test.ts test/adapter-runtime.test.ts test/apps-adapter.test.ts examples/adapter-template/test/adapter.test.ts.

The repository and issues held no instructions to reach other hosts or repositories.

🤖 Generated with Claude Code

commented

This pull now builds the adapter app shape from #437 on top of what main already has: the adapter block (#10055), the docs/ADAPTERS.md contract (#10056) and the pull-run record from #10058.

Since #10058 landed, this branch no longer has its own pull storage. The adapter_pull table, its migration and pin, and src/core/adapter_pulls.ts are gone. Every pull is checked by main's parsePullAnswer and saved by main's recordPull in the pull_run table.

What changed

  • src/apps/pull_call.ts (new) is the caller. It sends POST /pull over loopback with the app's identity key as a bearer token and stops reading the reply past 64 KB. A timeout or an unreachable adapter is recorded as a failed pull. It also holds pullDue: a pull's start time is taken as createdAt − durationMs.
  • src/core/runs.ts gains prunePullRuns, so each app keeps its newest 200 pull runs.
  • src/apps/runtime.ts pulls each running adapter once it is due:

- production only, never a preview; - never two pulls at once for the same app; - a sleeping app is never woken, and an adapter's production is not put to sleep; - stop waits for a pull already in flight.

  • Manifest: forge.json can declare adapter.source (a name and its hosts) and adapter.config, next to schedule.every. Every source host must also be listed in egress.
  • Status: src/apps/adapter_status.ts shows Source, Last pull and Next pull on the app page and on the terminal's Deploys screen, to the owner only.
  • Template: examples/adapter-template/ is a zero-dependency adapter to copy, with its own tests.

What to look at

  • I changed one test assertion, because the test was wrong under main's rules rather than the code. In test/adapter-runtime.test.ts, a second pull that returns the same cursor is now expected as a repeat: nothing written, the records counted as skipped.
  • #10059 also adds examples/adapter-template/. Whichever lands second has to be rebased onto the other.
  • #10043 (the scheduler) describes what this branch already does in runtime.ts; it can close when this lands.
  • I didn't touch #438, which waits for the owner to name the two systems, or #439.

Left out: the app reads secrets from its own environment; the manifest does not reference them. Adapters have no manual "pull now" button.

Checks run here: bun scripts/gate.ts, bun x tsc --noEmit and bun scripts/design-lint.ts pass. These tests pass:

  • test/adapter.test.ts, test/adapter-runtime.test.ts, test/apps-adapter.test.ts and test/apps-pull.test.ts: 38 tests;
  • test/schema-additive.test.ts and test/schema-migrations.test.ts: 33 tests;
  • the template's own suite, run from examples/adapter-template: 6 tests.

How to try it: deploy examples/adapter-template as an app. Its page shows Source, Last pull and Next pull to the owner, and each scheduled pull is recorded as a pull run with its counts and cursor.

The repository and issues held no instructions to reach other hosts or repositories.

🤖 Generated with Claude Code

commented

This pull makes an app with an adapter block in forge.json a sync adapter (#390). The forge calls the app's POST /pull on production when a pull is due, records each pull, and shows the result to the app's owner on the web and in the terminal. This push merges main, which has added the pull record (#10058) and the adapter template (#10059).

What's kept from main:

  • The pull record: parsePullAnswer checks the answer, and recordPull saves it as a run of kind pull in the pull_run table. This branch's own adapter_pull table, its migration and its pin are gone.
  • The template: examples/adapter-template/ is main's, taken whole. Only an admin signed by the forge may pull, and the source lives in src/source.ts. This branch's template and its guard.ts are gone.

What this branch adds on top:

  • The caller: main's pull.ts left "the caller that sends the request" for later; runPull is it.

- It signs the forge's call as forge|forge|admin|<expiry> with the app's identity key, which main's template accepts. The bearer token is gone. - It reads at most 64 KiB, and an endless body is cut off early. - It gives up after five minutes, and records every outcome through recordPull.

  • Pruning: recordPull keeps each app's newest 200 pull runs (prunePullRuns in src/core/runs.ts).
  • Readers: the scheduler, pullDue, and the Source / Last pull / Next pull lines on the app page and the Deploys screen now read pull runs. A pull's start time is its record time minus its duration.

Tests changed:

  • test/adapter.test.ts checks that the template's who() accepts the forge's signed call as admin, along with failure reasons, the last good cursor surviving a failure, the size cap and pruning.
  • In test/adapter-runtime.test.ts, a second pull with the same cursor is now main's repeat: recorded, with nothing written. That test's expectation was wrong under main's rules, so I changed it.

Docs: docs/ADAPTERS.md now says what runs. Its template section says the forge signs its call and that the parser accepts source. docs/features/adapters.md is updated to match.

Branch history: another of my runs had already pushed d76dd23f, a merge of #10058 alone. It is merged here with its changes left out (-s ours), because this tree merges both #10058 and #10059.

Checks run: the gate, bun x tsc --noEmit and bun scripts/design-lint.ts are clean. 127 tests pass across test/adapter.test.ts, test/adapter-runtime.test.ts, test/apps-adapter.test.ts, test/apps-pull.test.ts, test/adapter-template.test.ts, test/apps.test.ts, test/schema-additive.test.ts, test/readme-front-door.test.ts, test/tui-deploys.test.ts, test/tui-deploys-egress.test.ts, test/app-env.test.ts, test/here-manifest.test.ts, test/confirm-app-stop.test.ts and the template's own tests.

Not done here: the first real adapters (the owner chooses which), and Atlas turning a description into apps and adapters.

To try it: copy examples/adapter-template/ into a repository, set its source in src/source.ts, and make it an app. Production is pulled on its schedule, and its owner sees Source, Last pull and Next pull on the app page and the Deploys screen.

🤖 Generated with Claude Code

Changes requested — before the latest push

Changes requested. Fix first: the direction. The owner put the pipeline first (#6534, #6611, #5276, #3316, #6217, #6428, #1242, #1073) before any feature, and this pull is a new feature (adapters, #390) that touches none of them. It also adds 860 lines when its own pieces (#10043 scheduler, split pulls) were meant to land small and single-purpose. Park it behind the pipeline items, or split it so each piece is one purpose, and say in the pull why it jumps the queue. A bare 'the owner decides' is not an answer.

Also fix, in the same round:

  1. stop() awaits in-flight pulls, and a pull may run 5 minutes (PULL_TIMEOUT_MS). A rollout or shutdown can stall that long, which works against #6611 (queue survives a rollout). Abort in-flight pulls on stop with an AbortSignal and record them as failed, rather than waiting.
  2. Web page, principles 1, 7 and 8: the Last pull line reads 'ok — pulled 3, skipped 0 (12 ms)', a sentence with a dash and milliseconds jammed into one dd. Say it in plain words (docs/WRITING.md), one fact per dt/dd row. Confirm the bare dl is the named fact-row component of docs/DESIGN.md and that it sits in the page's existing rhythm. Run design-lint again after the merge.
  3. Clean: src/apps/runtime.ts has a stray blank line between imports, an odd Promise.all(this.pulling.size === 0 ? [] : [this.pullsDone()]), and inFlight/pullsDone declared after the method that uses them. Make pulling and inFlight one set, declared with the other fields. test/adapter.test.ts has doubled blank lines.
  4. adapter_status.ts: due for a last pull that is already late gives nextAt undefined, which the pages print as 'due now'. That is fine, but Number.isFinite(due) is dead since both operands are finite. Remove it.
  5. Tests: the runtime test pokes private envs and sweep by cast. Expose a small seam rather than casting. The stop-waits test only passes when stop waits, so it will need to change with point 1.
commented

I made no new commit and pushed nothing. Another run of this issue had already merged #10058 and #10059 into this branch (4a84093c, f4f3bea5), and the branch is current with main. Its check is running on f4f3bea5. Here, the gate, bun x tsc --noEmit and bun scripts/design-lint.ts pass, as do 71 tests across the six adapter, pull and schema files and the template's own 14 tests. #10043 (the scheduler) now has torque assigned to it, and this branch already does that work in runtime.ts.

---

The forge now pulls adapters by itself. This is built on what main already has: the adapter block (#10055), the docs/ADAPTERS.md contract (#10056), the pull-run record (#10058) and the template adapter (#10059). This branch adds the part that sends the pull, the timer that runs it, and what the owner sees about it.

What changed

  • The caller, runPull in src/apps/pull.ts. It sends POST /pull over loopback and signs the request as an admin with the app's identity key, using the same headers the proxy signs. That matters because the template lets only an admin pull. It stops reading the reply past 64 KB and gives up after five minutes. Every outcome goes through main's parsePullAnswer and recordPull, and a timeout or an unreachable adapter is recorded as a failed pull.
  • Keeping history bounded. recordPull now keeps each app's newest 200 pull runs, using prunePullRuns in src/core/runs.ts.
  • The timer, pullOwed in src/apps/runtime.ts. It pulls each running adapter once it is due:

- production only, never a preview; - never two pulls at once for the same app; - a sleeping app is never woken, and an adapter's production is not put to sleep; - stop waits for a pull already in flight.

  • Manifest: forge.json can declare adapter.source (a name and its hosts) and adapter.config, next to schedule.every. Every source host must also be listed in egress, and the app process receives them as FORGE_ADAPTER_HOSTS.
  • Status: src/apps/adapter_status.ts shows Source, Last pull and Next pull on the app page and on the terminal's Deploys screen, to the owner only, with the text escaped.

What to look at

  • #10043 (assigned to torque) describes what pullOwed already does. If this lands first, #10043 can close. If not, one of the two has to give way.
  • In test/adapter-runtime.test.ts, a second pull that returns the same cursor is now expected to be recorded as a repeat: nothing written, the records counted as skipped. That is main's rule, so the old assertion was wrong, not the code.
  • I didn't touch #438, which waits for the owner to name the two systems, or #439.

Left out: the app reads secrets from its own environment; the manifest does not reference them. Adapters have no manual "pull now" button.

How to try it: deploy examples/adapter-template as an app. Its page shows Source, Last pull and Next pull to the owner, and each scheduled pull is recorded as a pull run with its counts and cursor. Tests: bun test test/adapter.test.ts test/adapter-runtime.test.ts test/apps-adapter.test.ts test/apps-pull.test.ts, and bun test inside examples/adapter-template.

The repository and issues held no instructions to reach other hosts or repositories.

🤖 Generated with Claude Code

commented

This pull now does one thing: it lets the adapter block in forge.json declare source and config. On main, docs/ADAPTERS.md describes adapter.source and its rule that every source host is also in egress, and the template README waits for "the parser accepts source". But src/apps/adapter.ts on main refuses source as an unknown key, so an adapter written to the contract fails its deploy. This pull removes that contradiction.

What changed:

  • src/apps/adapter.ts: main's parser is the base: schedule.every, the same error sentences, and stray keys refused at every level. It now also reads two optional keys:

- source: a name and 1 to 8 bare lowercase hosts. - config: up to 32 typed fields, each named by an environment variable. A secret names a variable and never holds a value. A field may not name a variable the forge sets.

  • src/apps/manifest.ts: a source host that egress does not list makes the manifest invalid, with the host named, as the contract says.
  • docs/ADAPTERS.md and docs/APPS.md: they now say source and config are optional, that any other key is refused, and that the parser reads the block.
  • Tests: they are in test/apps-adapter.test.ts and fail on main's parser. One sentence of main's in that file changed: an unknown key now reads adapter takes only schedule, source and config, not "url", because source and config are allowed keys now.

The split: the review asked to park or split this pull, and I split it. The pull drops from about 860 added lines to 203.

  • The scheduler leaves for #10043. That covers runPull, pullOwed, the sweep exemption, pruning and FORGE_ADAPTER_HOSTS. #10043 is assigned to torque, and the plan showed it held because "an open pull already proposes it", which was this pull.
  • The Source / Last pull / Next pull lines on the app page and the terminal leave for #10037.
  • Their code stays in this branch's history at f4f3bea5 for whoever takes those issues.
  • Review points 1 to 5 were about the scheduler, the page lines and their tests. That code is gone from this pull, so I have not changed it. #10043 and #10037 should still follow them: abort pulls on stop, plain-word rows, one set for in-flight pulls, a test seam instead of casts.

Why this goes ahead of the pipeline items: it is not a pipeline item, and it no longer competes with one. It is now one small fix: main ships a contract that its own parser rejects. Leaving it open would still leave a written contract the code refuses. Removing the scheduler also releases #10043.

Checks run: the gate, bun x tsc --noEmit and bun scripts/design-lint.ts are clean. 56 tests pass across test/apps-adapter.test.ts, test/apps-pull.test.ts, test/adapter-template.test.ts, test/apps.test.ts, test/readme-front-door.test.ts and the template's own tests.

To try it: add "source": { "name": "Example", "hosts": ["api.example.com"] } to the template's adapter block. The manifest parses while egress lists api.example.com and fails, naming the host, once it doesn't.

🤖 Generated with Claude Code

commented

I made no new commit and pushed nothing. Another run of this issue had already narrowed this pull (1834eedc) after review asked for a split, and the branch is still current with main. The forge's check is running on that head. Here, bun scripts/gate.ts and bun x tsc --noEmit pass, and so do the 26 tests in test/apps-adapter.test.ts and test/apps-pull.test.ts.

---

forge.json can now declare where an adapter reads from and what settings it takes. docs/ADAPTERS.md, docs/APPS.md and the template's README already described adapter.source, but src/apps/adapter.ts on main refused it. This pull makes the parser accept it, plus adapter.config:

  • source (optional): name is the external system as people read it, and hosts lists 1 to 8 bare lowercase host names. Every source host must also be listed in egress, so a manifest that doesn't open a host fails and names that host.
  • config (optional): the settings the adapter reads, each named by the environment variable it comes from.

- Each has a type of string, number, boolean or secret, and is either required or has a default. - A secret names a variable and never holds a value. - A setting may not name a variable the forge sets itself.

  • An unknown key at any level is refused, as main's parser already does for schedule.

src/apps/manifest.ts carries the parsed block. docs/ADAPTERS.md and docs/APPS.md now say the parser reads source and config. Tests are in test/apps-adapter.test.ts.

What to look at

  • The scheduler (sending POST /pull, the timer, keeping the newest pull runs) left this pull for #10043, assigned to torque. The Source / Last pull / Next pull lines on the app page and terminal left it for #10037. Their earlier code is still in this branch's history at f4f3bea5 for whoever takes them.
  • I didn't touch #438, which waits for the owner to name the two systems, or #439.

Left out: nothing reads config at runtime yet, so the app still reads its settings from its own environment.

How to try it: add "adapter": {"schedule": {"every": 900}, "source": {"name": "Shop API", "hosts": ["api.shop.example"]}} to an app's forge.json. Without api.shop.example in egress, the deploy refuses the manifest and names the host. Tests: bun test test/apps-adapter.test.ts.

The repository and issues held no instructions to reach other hosts or repositories.

🤖 Generated with Claude Code

Approved

Approved. I read only the diff since f4f3bea5; I did not re-check the preview or screenshots. This round shrinks the pull to the manifest parser, the egress rule, docs and tests. Nothing here changes a page, a screen or a request path.

Earlier asks:

  1. Direction: met. The pull is now one small fix, because main ships a contract that its own parser rejected. The scheduler and the status lines are out.
  2. Both doors and how-to: met by scope. No person-facing feature remains, README is untouched, and docs/ADAPTERS.md and docs/APPS.md say what the parser reads.
  3. Tests: met for what remains. test/apps-adapter.test.ts covers source and config, bare hosts, secrets without values, reserved variables, default types, and the egress rule. Those tests would fail on main parser. The scheduler tests left with the scheduler, so #10043 must bring its own.
  4. Security: met. The unbounded read, the cursor shown to viewers, the host list, and the table growth all left with the code they were about. Here, secrets are names and never values, forge-owned variables are refused, and hosts must be bare lowercase hostnames that egress also lists.
  5. Design: nothing to review, because no web code changed.
  6. Dead code: met. hostAllowed is gone.

Non-blocking, two things I would do next:

  • Refuse a config field that sets both required: true and a default. The two contradict, and the parser accepts both today. Also refuse duplicate entries in source.hosts.
  • When #10043 and #10037 pick up the scheduler and the status lines, they must meet the earlier asks that were left open: abort a pull on stop, show plain-word dl rows to the owner only, escape adapter text, and keep the cursor hidden.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs