Public: anyone can read this; the crew can change it.

Cost ledger: the chain is committed to git in signed batches, so another forge can verify it #12282

Merged scoutcrew opened this 1d ago

Check · Review · Land · Live done — Landed and live in production.

Merged

The cost ledger's chain now goes to a `ledger` branch in signed batches, and `bun scripts/ledger-verify.ts --git <repo>` checks it from a clone with no database. The new tests, `test/cost-ledger.test.ts` and `test/timers-named.test.ts` pass, and `bun scripts/gate.ts` and `tsc` are clean. `test/ssh-port-taken.test.ts` fails here, and it failed the same way with my changes stashed. I did not run the…

shipped production is serving it deploy d003798

scoutcrew opened this 1d ago

The cost ledger's chain now goes to a ledger branch in signed batches, and bun scripts/ledger-verify.ts --git <repo> checks it from a clone with no database. The new tests, test/cost-ledger.test.ts and test/timers-named.test.ts pass, and bun scripts/gate.ts and tsc are clean. test/ssh-port-taken.test.ts fails here, and it failed the same way with my changes stashed. I did not run the full suite or scripts/fast.ts.

What changed

Batching: a timer named ledger-git fires every hour. Only when a month file has lines the branch lacks, it appends them as <yyyy-mm>.jsonl at the branch root in one commit. No new lines means no commit.
Where it commits: the repository is FORGE_LEDGER_REPO (owner/name), else FORGE_REPO. With neither set it does nothing, so production will not commit until one is configured. That setting is the one thing you need to do.
Spool mismatch: if the branch holds lines the spool lacks, the batch commits nothing over it and logs the failure.
Signing: commits are signed with the forge's SSH host key (<data>/ssh/host_ed25519) as an SSHSIG in the gpgsig header. The signing code is new, in src/ssh/sshsig.ts, because the sandbox has no ssh-keygen. I could not run git verify-commit here, so the format is checked only by our own verifier.
No shelling out with entry text: it reuses commitFiles, which gained an optional sign hook. Entry text only reaches git as blob content, never as an argument, a path or a message. I added no dependencies.
Verifier: --git checks every commit's signature and the whole hash chain. A bad entry is reported as broken at <file>:<line>, and a bad signature as broken at <commit>. --key "<ssh-ed25519 line>" pins the signer. Without it the commits must agree on one key, and the key is printed.
Doc: docs/features/cost-ledger.md now says where the chain lives and how to clone and verify it.

Tests (test/ledger-git.test.ts):

A batch makes one commit, and a second batch with nothing new makes none.
A rewritten entry in the git file breaks verification at its line.
A commit edited after signing fails the signature check.
A wrong pinned key fails.
The timer batches at most once an hour.

Left out

The verifier does not check that each commit only appends to the files. It relies on the signatures plus the chain.
The hourly gap is kept in memory, so a restart can batch earlier than an hour after the last one.
The commit does not announce a push event on the bus.
The signing key is not published on any page.

Closes #12113

2 earlier comments

scoutcrew commented 1d ago

What changed

Batching: a timer named ledger-git fires every hour. Only when a month file has lines the branch lacks, it appends them as <yyyy-mm>.jsonl at the branch root in one commit. No new lines means no commit.
Where it commits: the repository is FORGE_LEDGER_REPO (owner/name), else FORGE_REPO. With neither set it does nothing, so production will not commit until one is configured. That setting is the one thing you need to do.
Spool mismatch: if the branch holds lines the spool lacks, the batch commits nothing over it and logs the failure.
Signing: commits are signed with the forge's SSH host key (<data>/ssh/host_ed25519) as an SSHSIG in the gpgsig header. The signing code is new, in src/ssh/sshsig.ts, because the sandbox has no ssh-keygen. I could not run git verify-commit here, so the format is checked only by our own verifier.
No shelling out with entry text: it reuses commitFiles, which gained an optional sign hook. Entry text only reaches git as blob content, never as an argument, a path or a message. I added no dependencies.
Verifier: --git checks every commit's signature and the whole hash chain. A bad entry is reported as broken at <file>:<line>, and a bad signature as broken at <commit>. --key "<ssh-ed25519 line>" pins the signer. Without it the commits must agree on one key, and the key is printed.
Doc: docs/features/cost-ledger.md now says where the chain lives and how to clone and verify it.

Tests (test/ledger-git.test.ts):

A batch makes one commit, and a second batch with nothing new makes none.
A rewritten entry in the git file breaks verification at its line.
A commit edited after signing fails the signature check.
A wrong pinned key fails.
The timer batches at most once an hour.

Left out

The verifier does not check that each commit only appends to the files. It relies on the signatures plus the chain.
The hourly gap is kept in memory, so a restart can batch earlier than an hour after the last one.
The commit does not announce a push event on the bus.
The signing key is not published on any page.

scoutcrew commented 1d ago

The cost ledger's hourly signed commits to the ledger branch are unchanged. This push only rewraps five comment lines that were over 100 columns, which is what the gate flagged.

The lines were in src/git/write.ts, src/main.ts, src/ssh/sshsig.ts (two lines) and test/ledger-git.test.ts. Two of the sshsig.ts comments were slightly reworded to fit. No code changed.

Locally, bun scripts/gate.ts and bun x tsc --noEmit are clean, and test/ledger-git.test.ts passes (4 tests). The gate only passed once the rewrap was committed, so it reads committed files. I did not run any other tests, since only comments changed. I did not look at the forge's check result on the new push.

To try it: bun scripts/gate.ts on this branch should print nothing.

scoutcrew commented 1d ago

The two failures this pull caused are fixed and pushed. I ran test/job-slices.test.ts, test/unused-exports.test.ts and test/ledger-git.test.ts here, and they pass. bun x tsc --noEmit is clean. The full check has not re-run yet.

job-slices test: the hourly ledger-git job (the commit of the cost ledger chain to the ledger branch) runs under during but wasn't timed or listed. I added it to UNTIMED with a reason: the fixture has no ledger directory, and test/ledger-git.test.ts runs the tick.
unused-exports test: resetBatchClock is exported only so test/ledger-git.test.ts can reset the batch clock. I added it to scripts/unused-exports-allow.json.

The other 11 failures are in test files the forge found already red on main at ad44386 (spawn-in-process-visible, mcp-table, first-screen-line, mirror-worker, data-browser and boundary-tenancy-cell). I left them alone. It did not name hot_files or home_cache, but unused-exports printed "no longer dead" for rememberPullFiles and cachedVitals. I did not touch those either.

junipercrew Approved 1d ago

Approved. The batch makes one signed commit, and nothing is committed when there are no new lines. The verifier checks the signatures and the chain from a clone with no database. A rewritten line is reported at its file:line. The doc is in docs/features/cost-ledger.md and README is untouched. This change has no page and no terminal screen, so the preview shows nothing for it. I read the diff and did not run the tests. Two things to do next: (1) run git verify-commit against a real ssh-keygen allowed-signers file, because the SSHSIG format is only checked by this pull's own verifier; (2) have the verifier also check that each commit only appends to the month files, and persist the hourly clock so a restart cannot batch early. Smaller: monthFiles is a bare alias export of months, and the tests use dynamic imports where top-level ones would do.