Public: anyone can read this; the crew can change it.
Sentinel: backup-refs:mason/forge was degraded 20 minutes today: 1 branch of mason/forge cannot be backed up: their commits are missing ([redacted]… #12590
Check · Review · Land · Live done — Landed and live in production.
This pull only makes the card say how to clear it; it does not clear it. The broken branch of mason/forge is in repository data I can't read. What changed The backup-refs: card now ends with the remedy.
shipped production is serving it deploy 5228fa5
7 earlier comments · 1 event
The six check failures are in tests that don't touch this change, so I fixed nothing; I pushed an empty commit to re-run the check, which hasn't reported yet.
My only code change is one sentence in leftOutNeeds (src/snapshot/outcome.ts) and its expected string in test/snapshot-repositories.test.ts.
- Failing tests: the three snapshot-encryption tests, the budgets "home owner cold" test, and the
sealed/lane.jsontests intest/snapshot-incremental.test.ts. None of them referencebackup-refsorleftOutNeeds. - Local run: I ran the snapshot-incremental file here and its
sealed/lane.jsontests failed the same way. The snapshot-repositories tests fail here without my change too, because the sandbox blocks the loopback connection they need. - Check log: it says the
test/snappy.test.tsfailure happens only beside other tests in its shard. I read the shard-space and slow-test lines as noise, and I did not confirm any cause.
If the re-run is red again, the failures are not from this change and a person needs to look at those tests. The pull description is unchanged: the card now says how to clear it, and the broken branch itself is still unfixed.
The backup-refs:<repo> card now says how to clear it, and the check's snapshot failures are probably fixed by a test change in this push. That fix is not confirmed: I couldn't reproduce the failure with the real test order. I need the check to run again.
What changed
- In
leftOutNeeds(src/snapshot/outcome.ts), the card now ends with: "To clear this, delete the branch, push it again from a clone that has its commits, or (for a fork) detach it from its parent." test/snapshot-repositories.test.tsexpects the new sentence.test/spawn-cell.test.tsis the fix for the check failure (below).
Why the check failed
- The failing snapshot tests (
snapshot-incremental, the "encrypted before it leaves" tests) read a loopback S3 bucket. They never touch the card, which is why the last re-run changed nothing. test/spawn-cell.test.tssetHTTP_PROXYand then deleted it. Bun's fetch and S3 client keep a proxy variable they have seen, even after deletion. Later S3 calls in the same shard then went to a dead proxy and failed with "an unexpected error has occurred".- The test now puts the variables back as empty strings, as
test/outbound-gate.test.tsalready does. - I reproduced the leak in a small test: delete the variable and S3 calls fail, put it back empty and they pass. I did not reproduce it with
spawn-cellandsnapshot-incrementalrun together, and I did not confirm which test leaked in the check's shards.
Results here
tscis clean.scripts/gate.tsreports nothing.spawn-cellpasses.snapshot-incrementalpasses (22 of 22) when the sandbox's proxy variables are unset. Run normally here, 3 of its tests fail because the sandbox blocks loopback.
What I chose not to do
- I did not touch the broken ref itself: the repository data isn't readable from here, and hiding it would remove the signal the card exists to give.
- I did not add self-healing for forks.
To clear the card
Someone with access should run git for-each-ref and git fsck in the mason/forge bare repository to find the branch, then delete it or detach the fork. The card clears on the next snapshot that lands.
🤖 Generated with Claude Code
The gate failure is fixed and pushed. It flagged one comment line in test/spawn-cell.test.ts (line 251, 129 columns). I rewrapped it to two lines under 100 columns, with the wording unchanged.
After the change, bun scripts/gate.ts passes locally. I didn't run tsc or any tests, since the edit touches only a comment. The forge's check will rerun on this push.
The gate reads the committed diff, so it kept failing on the first two runs while the edit was still uncommitted. It passed once I committed.
🤖 Generated with Claude Code
The six S3 failures in snapshot-incremental, snapshot-repositories and related files were most likely a proxy leak from test/spawn-cell.test.ts. I've pushed a fix for that, but I haven't been able to confirm it clears the failures.
What I found:
- Leak: That test set
HTTPS_PROXY,HTTP_PROXYandNO_PROXYto127.0.0.1:3128on the real process, and Bun's fetch and S3 client keep a proxy variable they have seen. A shard that ran this file first would send later loopback S3 calls to that dead proxy, which gives the "unexpected error" onget sealed/lane.json. - Earlier fix: My previous attempt put the variables back as empty strings. That evidently didn't stop it.
- Not yet reproduced:
test/snapshot-incremental.test.tsfails here only because this sandbox injects its own proxy variables. It passes 22/22 with them unset. So I couldn't reproduce the forge's failures directly.
What changed:
childEnvinsrc/apps/process.tsnow takes an optional parent environment as a second argument, defaulting toprocess.env. Existing callers are unchanged.- The proxy test passes its fake proxy settings in that way, so it never touches
process.env. - The
backup-refscard's clearing text and the 100-column comment rewrap from earlier commits are unchanged.
Checks: scripts/gate.ts and tsc --noEmit pass, and test/spawn-cell.test.ts passes (40 tests, 3 skipped). The forge's check will run the shards again on this push.
🤖 Generated with Claude Code
The backup-refs:<repo> card now says how to clear it, and a test fix stops test/spawn-cell.test.ts from breaking the S3 snapshot tests that share its shard. This pull does not clear the card for mason/forge. The branch whose commits are missing is in repository data the crew can't read; the branch list for mason/forge comes back empty through the API.
What changed
src/snapshot/outcome.ts(leftOutNeeds): the sentence on the card now ends with "To clear this, delete the branch, push it again from a clone that has its commits, or (for a fork) detach it from its parent."test/snapshot-repositories.test.tsexpects the new wording.test/spawn-cell.test.ts: the test used to delete the proxy variables (HTTP_PROXYand the like) when it finished. Bun's fetch and S3 client keep using a proxy variable they have seen set, even after it is deleted. So the next test file in the same shard sent its loopback S3 calls to a proxy that no longer existed. That is what failed the sealed-lane tests intest/snapshot-incremental.test.tsand the snapshot-encryption tests intest/snapshot-repositories.test.ts. The test now sets the variables back to empty, which Bun reads as no proxy.
What I chose not to do
- I did not delete or hide the broken ref: that would remove the signal the card exists to give.
- I did not add self-repair for forks.
mason/forgeborrows its objects fromtrevato/forgethrough alternates, which is the likeliest source of the missing commits. But the forge's code never runsgcorprune, and I couldn't confirm the cause.
What I could not run here
- The sandbox blocks loopback connections, so the S3-backed snapshot tests can't run locally. The check on this push is their real test.
- The other red test, "budgets > home owner cold answers under its p95", times a page and has nothing to do with this change. I did not change anything for it.
To clear the card
Someone with access runs git for-each-ref and git fsck in the mason/forge bare repository to find the branch. Then they delete it, or detach the fork so it owns its objects. The card clears on the next snapshot that lands.
🤖 Generated with Claude Code
Three things to fix in one round.
- Split the unrelated change out. The
childEnvparent-env parameter and thetest/spawn-cell.test.tsrewrite are a CI-flake fix, not the sentinel issue. It is a production signature change made for test hygiene, and the pull itself says the flake fix is unconfirmed ('probably', 'not reproduced'). The direction asks for small single-purpose pulls and a never-flaking suite (#1073 and #1242 own flakes). Open it as its own pull where it can be judged on its own evidence, and dropsrc/apps/process.tsandtest/spawn-cell.test.tsfrom this one.
- The card does not clear the degradation the issue reports. The broken branch of mason/forge is untouched, and the pull's own text says so. A longer sentence on the card is not the fix. If the cause cannot be found from here, say so on the issue and leave it open. Do not land this as the answer to the sentinel.
- The new sentence is not something a person can act on. It sends them to 'delete the branch' and 'detach it from its parent' without saying where. The repo has detach code (
src/actions/detach_words.ts,src/actions/forks.ts), so name the page or action that does it. Also keep the sentence short: the card is a vitals line, and it now runs to three clauses. If the card links to a page, put the how-to indocs/features/and link it.
I deleted the branch that broke the backup-refs:mason/forge card. The card should clear on the next snapshot that lands, which hasn't happened yet. The card now links to the Branches page and the how-to is in docs/features/. The spawn-cell flake fix is out of this pull.
The cause. The broken ref was refs/heads/mason/7731-palette-entries-can-live-beside-their-sc on mason/forge, tip 443cf57e. That commit's tree 6d6a9f68 is missing: fetching every ref failed on the server with bad tree object 6d6a9f68…. I found the ref by fetching the fork's 1132 branches in halves. Its run had ended with nothing_to_merge and #7731 is closed as Done, so the branch held no work. I deleted it with git push origin --delete on my own fork. The commit object is still in the store, because the forge never runs gc. All 1146 refs of mason/forge now fetch with full history. I left the oids on #12581.
The card (src/snapshot/outcome.ts, src/kernel/vitals.ts)
leftOutNeedsnow returns anhreffor each card.- A
backup-refs:<repo>card links to/<repo>/branches, where a writer presses Delete. Before, it linked to the repository's front page. backup-repo:<repo>still links to/<repo>.- The added sentence is now one clause: "Delete it on its Branches page." It says "them" for more than one branch.
- Detaching a fork is no longer suggested, because it would not work: a detach repacks every object the fork borrows, and the missing object makes that repack fail.
test/snapshot-repositories.test.tschecks the card's link and the one-branch wording. I updated its expected sentence to the new text.
Docs. The how-to is in the new docs/features/backup-refs.md: what the card means, how to clear it, why detaching won't help, and how to find the ref by hand. It records this case. docs/SNAPSHOT.md links to it.
Split out. src/apps/process.ts and test/spawn-cell.test.ts are back to main. That fix should be its own pull with a reproduction, under #1073 and #1242. I haven't opened that pull.
What I did not do. I don't know why a push to mason/forge left a commit without its tree. Nothing in the code clearly causes it, and I couldn't look inside the repository on the server. This pull does not fix that, so the next case would show up the same way, on the card.
Checks. bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 1959 test files because vitals.ts is widely imported. So I ran test/snapshot-repositories.test.ts, readme-cap, features-empty, guide and the vitals suites: 83 pass, 0 fail.
To try it: on the next snapshot after this lands, backup-refs:mason/forge should no longer be in the degraded list. For a new case, the card opens /<owner>/<name>/branches.
Refs #12581. This should not close it until the sentinel sees the card stay clear for a day.
🤖 Generated with Claude Code
I deleted the branch that broke the backup-refs:mason/forge card. The card should clear on the next snapshot that lands, which hasn't happened yet. The card now links to the Branches page and the how-to is in docs/features/. The spawn-cell flake fix is out of this pull.
The cause. The broken ref was refs/heads/mason/7731-palette-entries-can-live-beside-their-sc on mason/forge, tip 443cf57e. That commit's tree 6d6a9f68 is missing: fetching every ref failed on the server with bad tree object 6d6a9f68…. I found the ref by fetching the fork's 1132 branches in halves. Its run had ended with nothing_to_merge and #7731 is closed as Done, so the branch held no work. I deleted it with git push origin --delete on my own fork. The commit object is still in the store, because the forge never runs gc. All 1146 refs of mason/forge now fetch with full history. The oids are on #12581.
The card (src/snapshot/outcome.ts, src/kernel/vitals.ts)
leftOutNeedsnow returns anhreffor each card.- A
backup-refs:<repo>card links to/<repo>/branches, where a writer presses Delete. Before, it linked to the repository's front page. backup-repo:<repo>still links to/<repo>.- The added sentence is now one clause: "Delete it on its Branches page." It says "them" for more than one branch.
- Detaching a fork is no longer suggested, because it would not work: a detach repacks every object the fork borrows, and the missing object makes that repack fail.
test/snapshot-repositories.test.tschecks the card's link and the one-branch wording. Its expected sentence is updated to the new text.
Docs. The how-to is in the new docs/features/backup-refs.md: what the card means, how to clear it, why detaching won't help, and how to find the ref by hand. It records this case. docs/SNAPSHOT.md links to it.
Pipeline-first gate. The rule landed on main after this branch's last check. It requires a Direction: line when a pull adds a docs/features/ file and no pipeline change. An empty commit carries that line, giving #12581 (a P0 Trust sentinel) and the review's request for the how-to as the reason. main's version of directionProblems reports nothing for this branch: 5 files changed.
Split out. src/apps/process.ts and test/spawn-cell.test.ts are the same as on main. That fix should be its own pull with a reproduction, under #1073 and #1242. That pull has not been opened.
Not done. It is not known why a push to mason/forge left a commit without its tree. Nothing in the code clearly causes it, and the repository on the server could not be inspected from here. This pull does not fix that, so a new case would show up the same way, on the card.
Checks. bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 1959 test files because vitals.ts is widely imported, so the covering tests were picked by hand. test/snapshot-repositories.test.ts, readme-cap, features-empty, guide and the vitals suites gave 83 pass, 0 fail. After the gate commit, test/snapshot-repositories.test.ts alone gave 10 pass, 0 fail.
To try it: on the next snapshot after this lands, backup-refs:mason/forge should no longer be in the degraded list. For a new case, the card opens /<owner>/<name>/branches.
Refs #12581. It should stay open until the sentinel sees the card stay clear for a day.
🤖 Generated with Claude Code
When a repository can't be backed up, its snapshot card now links to the repository's Branches page and says to delete the branch there. It also links to a new how-to, docs/features/backup-refs.md, which explains why a branch can't be bundled and how to clear it.
The card text is in src/snapshot/outcome.ts, with a matching line in src/kernel/vitals.ts. docs/SNAPSHOT.md points to the how-to, and test/snapshot-repositories.test.ts checks the card's link and wording.
This push merges main, which now has a rule that every feature note is listed in docs/features/README.md. The new how-to has its row there under "Backups and moving", in alphabetical order. That fixes "no feature note is an orphan" and "the index is one note to a row", the two failures the check counted against this pull.
The other failures from the last check:
test/ask-to-reach.test.ts: the check found it red onmainas well and did not count it against this pull; this pull doesn't touch it.test/etag.test.ts: passes here after the merge.
Not changed: the spawn-cell proxy fix was taken out earlier and goes in its own pull.
bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts asked for 1,967 test files, about 167 minutes, because the merge brings in main's changes. So I ran only the tests that cover this change: test/feature-orphans.test.ts, test/snapshot-repositories.test.ts, test/etag.test.ts and the snapshot-outcome and vitals tests, which all pass.
To try it, read docs/features/backup-refs.md, and check its row under "Backups and moving" in docs/features/README.md.
🤖 Generated with Claude Code
Earlier asks (link the card to where it is cleared, put the how-to in docs/features and not the README, test the link and wording) are met. The card now opens /owner/name/branches, the sentence says Delete it/them on its Branches page, the how-to is docs/features/backup-refs.md with its row in the features index, and the spawn-cell change is out of the pull. Tests cover the href and the singular wording. Next two things: (1) Non-blocking: open the separate pull for the proxy-leak flake with a reproduction under #1073/#1242. (2) Non-blocking: find why a push left a commit without its tree, since the next case will only show up as this card.
This pull only makes the card say how to clear it; it does not clear it. The broken branch of
mason/forgeis in repository data I can't read.What changed
backup-refs:<repo>card now ends with the remedy. It reads: "To clear this, delete the branch, push it again from a clone that has its commits, or (for a fork) detach it from its parent."leftOutNeeds(src/snapshot/outcome.ts). I updated the expected string intest/snapshot-repositories.test.tsto match.What I found
GET /api/v1/repos/mason/forge/branchesreturns an empty list, so the broken ref is not readable through the API.mason/forgeis a fork oftrevato/forge(forkParentIdis set). A fork borrows its objects from the parent through alternates, which is the likeliest source of the missing commits.gcorprune, so I found no code path that deletes those objects.What I chose not to do
Checks
tscis clean andscripts/gate.tsprinted nothing.test/snapshot-repositories.test.ts, 8 of 10 pass. The two that fail are the encryption tests, and they fail the same way without my change: the sandbox blocks the loopback connection they need. The test that covers this sentence passes.To clear the card Someone with access should run
git for-each-refandgit fsckin themason/forgebare repository to find the branch. Then delete it, or detach the fork. The card goes away on the next snapshot that lands.Closes #12581