Public: anyone can read this; the crew can change it.

Mirror, migrate and sync fetches count against the storage limit, so no account can mirror past it #2207

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Mirror syncs and migrate imports now mark the account's storage figure stale. A storage refusal is now `storage_limit` instead of `repository_limit`. `tsc --noEmit` is clean. `test/quotas-storage.test.ts`, `test/quotas.test.ts` and the `test/origins*` tests pass. I did not run the whole check.

shipped production is serving it deploy 4ad7a6a

opened this

Mirror syncs and migrate imports now mark the account's storage figure stale. A storage refusal is now storage_limit instead of repository_limit. tsc --noEmit is clean. test/quotas-storage.test.ts, test/quotas.test.ts and the test/origins* tests pass. I did not run the whole check.

What changed

  • Stale marking: storageStale(app, ownerId) in src/core/storage.ts only sets the flag and does not walk, so the next check does the measuring. syncMirror in src/origins/sync.ts calls it after each successful git fetch, which covers the first sync and every later one. runMigration in src/actions/migrate.ts calls it after a successful clone.
  • No mark at creation: mirrorRepo no longer marks the figure when the empty repository is made.
  • New error: createRepo, forkRepo and mirrorRepo return storage_limit when storage refuses, and migrateMessage has its own sentence for it. A new storageLimitMessage names the use, for example "1.5 MB of 1 MB".
  • Callers updated: the JSON API (create, fork, app create, migrate), the web pages (new repo, fork, new app), the TUI new and fork screens, and the mirror job error. Push-to-create over HTTP and SSH also reports the storage sentence, because a storage_limit error there would otherwise have read as "not found".

Tests

  • Two existing assertions expected repository_limit for a storage refusal. They were wrong for the new behaviour, so I changed them to storage_limit.
  • A new test mirrors a 1.5 MB repository against a 1 MB limit from the fake origin. The account starts empty, so the mirror is let in, and the next create is refused with storage_limit, including over the API (403).
  • A new test pins REWALK_MS at 5000. The disk is emptied behind the cache, and the account stays refused one second inside the window and is allowed one second after it.

Left out: no walk on sync, as the issue asked. The migrate path is covered by the same stale call but has no test of its own.

Try it: set the storage quota to 1 MB, mirror a repository larger than that from an origin, then create a repository as that account.

🤖 Generated with Claude Code

Closes #2200

Changes requested — before the latest push

The core of #2200 is right. Sync and migrate mark the figure stale without walking, the empty mirror is no longer marked at creation, storage_limit is its own error in create, fork and mirror, and both new tests pass (bun test test/quotas-storage.test.ts: 13 pass at 76c9d362). No security problems in the diff: identity still comes from ctx.auth, and no input reaches a shell or SQL.

Fix first (regression on the client-facing door, pillar 3): src/actions/make_it_yours.ts:218-241 switches on forkRepo's error and has no storage_limit case. Before this pull, a storage refusal arrived as repository_limit and got a sentence. Now it falls to default, and people see the copy could not be made (storage_limit), a raw error code. That shows on the studio page (src/web/studio/make_it_yours.ts) and through the MCP tool (src/http/mcp.ts:1399, which returns invalid instead of a storage refusal). Add { kind: "storage_limit"; message } to MakeItYoursError, map it with storageLimitMessage(app, actor.id), handle it in the studio page and in MCP (bad("storage_limit", …)), and add a test that makes a copy over the limit and asserts the kind. Before you push, grep for every other switch over CreateRepoActionError or ForkRepoError so none of them is missed.

Smaller points, in the line comments: mark the figure stale only when a fetch moved refs, and make the REWALK test stand on its own.

I could not reach the preview or the screenshots from this sandbox, so this verdict comes from the code and the tests.

commented

Make-it-yours now reports a storage refusal as storage_limit with a sentence, and both smaller points from the review are fixed. tsc --noEmit is clean, and quotas-storage (14 tests), quotas, origins-sync and the MCP tests pass. I did not run the whole check.

  • Make-it-yours regression: MakeItYoursError has a storage_limit kind carrying storageLimitMessage(app, actor.id). The studio page says "Not made: …", the JSON API returns 403 storage_limit, and the MCP tool returns bad("storage_limit", …). A new test makes a copy over the limit and asserts the kind and the "MB of 1 MB" sentence.
  • Other switches: I grepped every use of CreateRepoActionError and ForkRepoError; the only consumers are actions/apps.ts and actions/migrate.ts, which already handle it.
  • Stale only on movement: src/origins/sync.ts marks the figure stale only when the fetch moved refs, so a periodic sync that changed nothing no longer forces a walk.
  • REWALK test: it now signs up its own account (erin), pushes 1.5 MB, and waits for the measurement. It passes alone with -t REWALK.

The async on daveId went away with the helper.

Changes requested — before the latest push

Close, but one of the issue's done items has no test, and the PR body says so itself.

What holds. I checked the head (2c270e45) in a clean tree: tsc --noEmit is clean and quotas-storage plus quotas pass, 26 tests. The new mirror test does what it claims. With the storageStale call in src/origins/sync.ts removed, it fails because the create is let through. Every consumer of createRepo, forkRepo, newApp, makeItYours and migrate now maps storage_limit: the API (403), MCP, the web pages, the TUI screens, and push-to-create over HTTP and SSH. No new input reaches SQL or a shell. The messages are the actor's own figure, so nothing crosses accounts. The walk-on-sync risk is handled: sync only marks the figure stale, and only when refs moved.

Fix this first. The migrate import's stale mark (src/actions/migrate.ts:405) is one of the issue's three named writers ("a migrate import finishes writing"), and no test would fail without it. Add a test to test/quotas-storage.test.ts shaped like the mirror test:

  1. A fresh account reads its figure once, at 0.
  2. It migrates the 1.5 MB ada/bigorigin through POST /api/v1/...migrate and waits for the job.
  3. Assert that the next createRepo is { ok: false, error: "storage_limit" }.

Check that it fails when the storageStale line is deleted.

Smaller, not blocking:

  • The storage sentence on refused creates is built from ctx.auth.user.id even when owner is an org. The refusal was made on the org's figure, so the sentence falls back to the generic text. Passing the resolved owner id would name the real use.
  • walk() sets stale = false when it finishes. A sync that marks the figure stale while a walk is already running is cleared by that older walk. This is a soft quota and a rare race, but setting the flag only if no mark arrived during the walk would close it.
  • The push-to-create lines in src/http/server.ts and src/tui/ssh.ts call resolveOwner a second time inline, in a 150-character expression. Hoist it into a const.

The preview and the look PNGs were not reachable from my sandbox (the egress to the preview host and the read of /data/looks were both denied). This verdict rests on the code and the tests.

commented

The migrate import's stale mark now has a test that fails without it, and I fixed the three smaller points. tsc --noEmit is clean, and quotas-storage (15 tests), quotas and migrate pass; I did not run the whole check.

  • Migrate test: in test/quotas-storage.test.ts, a fresh account reads its figure once at 0 and migrates the 1.5 MB ada/bigorigin through POST /api/v1/migrate. It waits for the job, and the next createRepo must return { ok: false, error: "storage_limit" }. I commented out the storageStale call in src/actions/migrate.ts and the test failed. I restored the line.
  • Org figure: the API create refusal now builds its sentence from the resolved owner id, so an org's use is named.
  • Hoisted owner: the push-to-create refusals in src/http/server.ts and src/tui/ssh.ts use a const ownerId instead of the inline expression.
  • Walk race: in src/core/storage.ts each stale mark bumps a counter. A walk clears stale only if no mark arrived while it ran. A check that finds the figure stale while a walk is running still joins that older walk and judges on its result. I left that as is, since this is a soft quota.
Changes requested — before the latest push

The code change is sound. Mirror, migrate and sync fetches now mark the cached figure stale. The marks counter stops a walk from clearing a stale mark that arrived while it ran. Every door (page, API, MCP, terminal, git over HTTP and SSH) now reports storage_limit instead of borrowing repository_limit. The new tests would fail without the stale marks. Security-wise I found no problems: there is no new input, every message is built from server figures, and every page escapes them.

I could not reach the preview or read the screenshots from my sandbox, so this verdict comes from the diff alone.

Fix first — the README. The README is not touched. It still says the storage limit refuses only "a push, create or fork", and that the figure is re-measured after "a push, create, fork, mirror or delete". It does not mention mirror, migrate or the periodic sync fetch, and it does not mention the new API error. Callers that matched on repository_limit for a storage refusal now get a different code. In README.md, in the storage paragraph of the quotas section:

  • Say that create, fork, make-it-yours, mirror, migrate and new app are refused at the limit.
  • Document the API answer: 403 {"error": "storage_limit", "message"}, with the message "…storage limit: 1.9 GB of 2 GB". Pages answer "Not made: …" and the terminal shows a toast.
  • Say that a mirror's sync or a migrate import marks the figure stale, so the next check measures again and no account can mirror past the limit unseen.

Smaller — whose figure is named. createRepo checks the storage of the resolved owner, which can be an organisation. The new-repo page and the terminal's new-repo screen build their message from user.id, so an org create names the wrong account's figure, or none. The API path already resolves the owner; do the same in these two places (see line comments).

commented

Handed to a person by the crew: the reviewer asked for changes three times with no approval between, so the planner re-scopes it: split, replaced, or one more round with the reviewer's ask restated.

  1. sentinel: The core of #2200 is right. Sync and migrate mark the figure stale without walking, the empty mirror is no longer marked at creation, `stora
  2. sentinel: Close, but one of the issue's done items has no test, and the PR body says so itself.
  3. sentinel: The code change is sound. Mirror, migrate and sync fetches now mark the cached figure stale. The marks counter stops a walk from clearing
commented

One more round, for one concrete change: a storage-limit refusal names the owner that was refused, and the migrate mark has a test that fails without it. (1) In src/web/pages/site.ts near line 694, pass resolveOwner(ctx.app, ctx.auth.user, owner) ?? ctx.auth.user.id to storageLimitMessage, the way the API does. Make the same change in src/tui/screens/newrepo.ts near line 48, using the owner the screen created under rather than f.user!.id. (2) Add a migrate-over-the-limit test shaped like the mirror one: an empty account, the figure read at 0, migrate the 1.5 MB origin, and the next create is storage_limit. Deleting the mark at src/actions/migrate.ts:405 must turn it red. Add an org-owner case for (1) as well. Nothing else in the pull needs to change.

commented

Still one round open, for the same single change asked above: the storage-limit refusal names the refused owner (an org included) in src/web/pages/site.ts and src/tui/screens/newrepo.ts, and a migrate-over-the-limit test turns red if the mark at src/actions/migrate.ts:405 is removed. The head has not moved since that ask (cdba57af). The builder should push that change; the rest of the pull stands.

commented

scout only (reviewers: not for you): Steward (trevato) decision on the hand-over — one more round: the README storage paragraph exactly as Sentinel lists it (what is refused at the limit, the 403 storage_limit answer, stale marks from mirror sync and migrate), and the org-owner figure on the new-repo page and terminal screen (resolve the owner as the API path does). Push, and it goes to review once more.

commented

Mirror, migrate and sync fetches now count against the storage limit. This round names the right owner in storage refusals and updates the README. bun x tsc --noEmit printed nothing and bun test test/quotas-storage.test.ts passes (16 tests); I did not run the rest of the suite.

What changed this round

  • Owner named in the refusal: a refused create now names the figure of the owner it was refused on, an organisation included.

- The new-repo page (src/web/pages/site.ts) and the terminal's new-repo screen (src/tui/screens/newrepo.ts) both call resolveOwner(...) ?? user.id, as the API does. - The terminal's fork screen still uses the actor's own id, which is right because a fork is always created under the actor.

  • README, storage paragraph:

- Create, fork, make-it-yours, mirror, migrate and new app are refused at the limit, on the owner the repository would belong to. - Every other door answers storage_limit. The API sends a 403 {"error": "storage_limit", "message"} with "…storage limit: 1.9 GB of 2 GB", pages show "Not made: …" (a 422), and the terminal shows a toast. - A mirror's periodic sync fetch and a migrate import mark the figure stale without walking the disk. - The one fetch that crosses the limit is let in, and the next create is refused.

  • Org test: a new test in test/quotas-storage.test.ts posts the new-repo form under a full organisation whose founder holds nothing. It expects a 422 whose text carries the organisation's figure ("MB of 1 MB"). Without the fix the page names the founder's empty account instead.

Migrate test

  • The ask was for a migrate-over-the-limit test that turns red if the mark at src/actions/migrate.ts:405 goes. That test already exists in test/quotas-storage.test.ts ("a migrate import bigger than the limit is allowed in…"), so I added nothing for it.
  • I did not delete the mark to confirm it goes red, and I did not run the org test without the fix.

Left out: I did not add a terminal-screen test for the org figure. The page test covers the same resolveOwner call, and the terminal screen has no harness here.

Approved — before the latest push

Approved from the diff and tests; the preview and the look PNGs were not reachable from my sandbox (egress denied), so I could not check the page by eye.

What holds: storage refusals are now their own storage_limit through every door (API 403, MCP, pages, make-it-yours, push-to-create over HTTP and SSH, TUI new-repo and fork screens). They name the owner actually refused, an organisation included. Mirror sync and migrate only mark the figure stale and never walk the disk. The marks counter stops an older walk from clearing a newer mark. Tests cover the mirror, migrate, org and make-it-yours cases, and the README paragraph is updated. No new input reaches SQL or a shell, and identity still comes from ctx.auth.

Two things I would do next:

  1. Add a TUI test for the org-owner figure on the new-repo screen. Only the page path is tested today.
  2. The new-app web page and the TUI show a generic or raw sentence for the migrate storage_limit: "delete a repository or app data first" does not name the figure. Pass the figure through migrateMessage, as the other doors do.
commented

The branch had a README conflict with main, which had since extended the same quotas paragraph with the previews, runs and checks limits. I merged main in and pushed. bun x tsc --noEmit is clean and bun test test/quotas-storage.test.ts passes (16 tests). I did not run the rest of the suite.

In the merge I kept main's paragraph and re-applied my two storage changes to it:

  • Refusal list: create, fork, make-it-yours, mirror, migrate and new app are refused at the limit, on the owner the repository would belong to, an organisation included.
  • API answer and stale marks:

- Every other door answers storage_limit: a 403 from the API, "Not made: …" on the pages, and a toast in the terminal. - A mirror's sync fetch and a migrate import mark the figure stale without walking the disk. - The one fetch that crosses the limit is let in, and the next create is refused.

The storage-refusal code and tests are unchanged since the last push, apart from the merge.

Approved

Approved. I read the diff at c4988d3a. Storage refusals are now their own storage_limit through every door: the API (403), MCP, the new-repo, fork, new-app and make-it-yours pages, push-to-create over HTTP and SSH, and the terminal's new-repo and fork screens. The sentence comes from the cached figure and is built on the server, from the resolved owner (resolveOwner asks for org write access). It does not come from client text, so the change opens no new path for injection or for authorisation. Mirror sync marks the figure stale only when refs moved. Migrate marks it after the clone. The marks counter stops an older walk from clearing a newer mark. Removing the storageChanged call after the mirror's create is safe, because the mirror's first fetch goes through syncMirrorUnguarded, which marks the figure stale.

What I checked at the head: bun x tsc --noEmit is clean, and bun test test/quotas-storage.test.ts test/quotas.test.ts passes (28 tests). The new tests cover mirror, migrate, make-it-yours, the org figure on the page, and REWALK. The README paragraph is updated. My sandbox could not reach the preview or the look PNGs (egress and read are denied), so I judged from the code and the tests, not by eye.

One low-severity point, not blocking (see the line comment): the new-app page and the API's new-app route name the actor's figure, not the owner's, when the app is created under an organisation. Also, migrateMessage("storage_limit") does not include the figure. Both are worth a small follow-up.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs