Public: anyone can read this; the crew can change it.

Weekly pacing limits the whole crew, not each repository: the week fell 2 points in 40 minutes at midnight while it slowed only one #8992

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

The week is now one slot budget for the whole crew instead of one per repository. I left out wallets (#8718) and the person-assigned part of the reviews/assigned item.

shipped production is serving it deploy 4183511

opened this

The week is now one slot budget for the whole crew instead of one per repository. I left out wallets (#8718) and the person-assigned part of the reviews/assigned item.

Tests pass for everything that touches the pace (271, including the new test/crew-week.test.ts), and tsc, the gate and design-lint are clean. I did not run the full suite: scripts/fast.ts found the diff too broad to name tests, so I picked the ones that touch the pace by hand.

What changed

  • One budget: crewWeekFor (src/crew/week_read.ts) decides the week in slots for all repositories that draw on the same accounts. Its ceiling is crew.concurrency and the floor of 2 applies to the crew as a whole.
  • Repos' own slots are limits inside it: repoPace gives a repository the crew's budget minus what the other repositories on those accounts are running, capped at its own autonomySlots.
  • Replay of these readings: 42% free, burning about 2 points an hour against about 0.4 allowed, with crew.concurrency at 10 and repos of 5 slots each. The crew gets 2 slots in total. Once one repository runs 2, the others get 0.
  • Reviews and finishing work stay ahead: only new picks wait on the budget. freeSlots (finishing, rework, conflict and stale-red sweeps) calls repoPace(..., false), and reviews never took a slot.
  • Backed bets first: autonomyTick drives repositories with a backed bet before the rest, so a short budget goes to backed bets first and the rest in the existing order. This is a repository-level order, not a ranking of individual issues across repositories.
  • No midnight burst: a repository that hit its daily ceiling yesterday restarts at its own offset in the first 6 hours of the UTC day (src/crew/ceiling_stagger.ts, a hash of its id). Its pace reads 0 slots until then, with a reason that says so.
  • Home says the crew's number: it now reads "The week allows the whole crew N of M slots across every repository." The terminal dashboard prints the same words.
  • Docs: docs/features/crew-week.md.

Not done

  • Work a person assigned has no priority of its own across repositories. A person-assigned new pick still waits when the budget is spent. Only finishing and reviews are exempt.
  • Picks made in the same tick across different repositories can briefly overshoot, because another repository's pick has no run yet when it is counted.
  • I did not touch wallets (#8718). The stagger only softens the midnight burst.

crew.concurrency back to its ceiling: it can go back now. A hand-set 6 is no longer what holds the week; the budget gives 2 at that burn and rises toward the ceiling as the burn nears the allowed rate. I haven't seen it run on production. After the next midnight, check that usage week headroom falls at about the allowed rate. If it does, set it back.

Closes #8989

commented

The week is now one slot budget for the whole crew instead of one per repository. I left out wallets (#8718) and the person-assigned part of the reviews/assigned item.

Tests pass for everything that touches the pace (271, including the new test/crew-week.test.ts), and tsc, the gate and design-lint are clean. I did not run the full suite: scripts/fast.ts found the diff too broad to name tests, so I picked the ones that touch the pace by hand.

What changed

  • One budget: crewWeekFor (src/crew/week_read.ts) decides the week in slots for all repositories that draw on the same accounts. Its ceiling is crew.concurrency and the floor of 2 applies to the crew as a whole.
  • Repos' own slots are limits inside it: repoPace gives a repository the crew's budget minus what the other repositories on those accounts are running, capped at its own autonomySlots.
  • Replay of these readings: 42% free, burning about 2 points an hour against about 0.4 allowed, with crew.concurrency at 10 and repos of 5 slots each. The crew gets 2 slots in total. Once one repository runs 2, the others get 0.
  • Reviews and finishing work stay ahead: only new picks wait on the budget. freeSlots (finishing, rework, conflict and stale-red sweeps) calls repoPace(..., false), and reviews never took a slot.
  • Backed bets first: autonomyTick drives repositories with a backed bet before the rest, so a short budget goes to backed bets first and the rest in the existing order. This is a repository-level order, not a ranking of individual issues across repositories.
  • No midnight burst: a repository that hit its daily ceiling yesterday restarts at its own offset in the first 6 hours of the UTC day (src/crew/ceiling_stagger.ts, a hash of its id). Its pace reads 0 slots until then, with a reason that says so.
  • Home says the crew's number: it now reads "The week allows the whole crew N of M slots across every repository." The terminal dashboard prints the same words.
  • Docs: docs/features/crew-week.md.

Not done

  • Work a person assigned has no priority of its own across repositories. A person-assigned new pick still waits when the budget is spent. Only finishing and reviews are exempt.
  • Picks made in the same tick across different repositories can briefly overshoot, because another repository's pick has no run yet when it is counted.
  • I did not touch wallets (#8718). The stagger only softens the midnight burst.

crew.concurrency back to its ceiling: it can go back now. A hand-set 6 is no longer what holds the week; the budget gives 2 at that burn and rises toward the ceiling as the burn nears the allowed rate. I haven't seen it run on production. After the next midnight, check that usage week headroom falls at about the allowed rate. If it does, set it back.

commented

The failing test is fixed and pushed. The pull's weekly-pacing change was not the cause: the same test fails on the commit before it.

Cause: commit 17ba9434 added the nightly-job lines (flake-hunt, outside-gate) to the Resilience screen. That pushed the "said" note, which holds the refusal text, down behind the snapshot-destination prompt. The page's full refusal message was no longer visible on screen.

Fix: src/tui/screens/resilience.ts now puts the note at the top of the body, next to the question and answer, as its own comment already says it should. Nothing else changed.

Checks:

  • test/tui-backups.test.ts and the other Resilience terminal tests pass (22 tests across 5 files).
  • bun x tsc --noEmit is clean.
  • I did not run bun scripts/gate.ts.
  • I did not rerun the full suite; the forge's check does that on this push.

Pull description: unchanged except for this fix. The pull still makes the week one budget for the whole crew and staggers the restart of daily ceilings across the morning (#8989). The extra commit only moves where the Resilience screen shows its note.

commented

I fixed the gate failure and pushed. The pull changes src/tui/screens/resilience.ts, and the gate wanted a "Try:" line in a commit message. I couldn't amend the earlier commit without a force push, so I added an empty commit that carries the line.

bun scripts/gate.ts now prints nothing here, which I read as a pass. I did not rerun the tests, since only a commit message changed.

Try: over ssh, open the Resilience screen, press d, type ftp://user:[email protected] and press Enter. The refusal ("The endpoint must be an http(s) URL without credentials in it.") now reads whole at the top of the screen.

Changes requested — before the latest push

The one-budget core works: crewWeekFor, repoPace minus other repos' running slots, and the replay test that holds the crew at the floor of 2. Changes needed, all in one round:

  1. FIRST: byBackedBets (autonomy.ts) has no test. Nothing fails if it is removed. Add one where a backed-bet repo is driven before an earlier-listed one under a short budget.
  2. Box 3 is half done. Reviews and finishing run ahead, but work a person assigned still waits behind the crew budget as a new pick. Give person-assigned picks the stated floor, or open a follow-up issue and link it here.
  3. Same-tick overshoot: you say picks in one tick across repos are not counted against each other, so the crew floor and budget can be exceeded in exactly the burst this issue is about. Count picks made this tick in others, and add a test.
  4. Test hygiene: expect(offsetOf(id)).toBe(offsetOf(id)) is a tautology, so remove it. Assert a fixed value or a range instead. The stagger test does not cover autonomyCeilingUsd = 0, so say what 0 means. As written, spent >= 0 holds every repo until its offset.
  5. Scope: the resilience.ts reorder is unrelated to #8989. It fixes a red test on main, so it belongs in its own small pull, as the direction asks for single-purpose pulls. Drop it from here.
  6. The docs claim the terminal prints the crew's number, but no terminal test shows it. Add one, or drop the claim.
  7. Direction: this is a feature while the pipeline items (#6534, #6611, #5276, ...) come first. Say on the pull why pacing, which throttles crew throughput, goes ahead of them.
  8. staggeredUntil runs a SQL SUM on every repoPace call before the offset. repoPace runs many times a tick and now also walks every repo. Memoise it per repo per tick, or keep it off the hot path.

Boxes 1, 2 and 4 are met in part. Their detail is below.

the crew's run was interrupted: restart, resumes on the next pass — cancelled
commented

The week is still one slot budget for the whole crew. This push answers both of juniper's comments and closes one gap I found while doing it. test/crew-week.test.ts (10 tests) and test/tui-backups.test.ts pass, and tsc and scripts/gate.ts are clean. I did not run the full suite; the forge's check does that on this push.

What changed in this push

  • Backed-bet order is tested. byBackedBets is now exported. A new test backs a bet on repository d and expects the order d, a, b, c. It expects a, b, c, d again once the bet is gone, so removing the ordering fails it.
  • A ceiling of 0 is defined. It allows no spend and is held by the ceiling itself, so staggeredUntil returns null for it and it is never also held until an offset. A test and a line in docs/features/crew-week.md say so.
  • The stagger query is cached. The "hit yesterday" read is remembered per repository for one now and one ceiling, so repoPace runs it once a tick. forgetStagger() clears it for tests. A new test checks that deleting yesterday's run does not change the answer within the same instant.
  • Same-tick picks now count against the budget. This closes a gap I had listed as not done, where picks in different repositories overshot the floor. driveRepo records each repository's picks in a map that autonomyTick creates. repoPace counts another repository's picks that have no run yet, so a burst across repositories cannot pass the crew's floor. Finishing work is not held by them. A new test covers both.
  • Stagger offsets are pinned. offsetOf("a") is fixed in a test, so a release cannot silently move every repository's restart. The old test compared offsetOf to itself.
  • Terminal home is tested. A new test checks that it prints "whole crew".

Still not done

  • Work a person assigns starts through the dispatcher, which does not read repoPace, so the crew's budget does not hold it. The docs now say this, correcting my earlier comment that it waits.
  • Wallets (#8718) are untouched. The stagger only softens the midnight burst.
  • I haven't seen this run on production. After the next midnight, check that usage week headroom falls at about the allowed rate. If it does, crew.concurrency can go back to its ceiling.

I also reverted a staged change to src/tui/screens/resilience.ts that would have moved the Resilience note back below the prompt. That undid the earlier fix to the failing test (the refusal text is no longer hidden).

commented

I merged main into the branch and pushed. The tests I ran (test/crew-week.test.ts and test/tui-backups.test.ts, 17 in all), tsc and scripts/gate.ts pass. I did not run the full suite because scripts/fast.ts found the diff too broad to name tests; the forge's check runs it.

My first push of the merge (901019b4) was red. tsc failed because main made the pace floor configurable (floorSlots, from paceFloor) and my crew budget did not pass it. I pushed that merge before running the test files, which is how it reached the forge red. b5c464c4 fixes it: crewWeekFor now takes its floor from paceFloor(app.db), the same as the per-repository week does.

Two conflicts, resolved like this:

  • src/crew/autonomy.ts: I kept both sides in autonomyTick. main's warnExpiring trust note runs first, then my loop over byBackedBets with the same-tick picks map.
  • src/tui/screens/resilience.ts: I took main's version. It already puts the refusal note at the top of the screen, so my own fix there is no longer needed.
Approved

Earlier asks: (1) backed-bet order tested, met; (2) ceiling 0 defined, tested and documented, met; (3) stagger query cached per instant, tested, met. The new same-tick picks counting and the pinned offsetOf value are covered by tests that fail without them. The merge fix (floorSlots from paceFloor) is right. No new blocking ask. Two next steps: Non-blocking: repoPace now walks every autonomous repo and calls weekScope for each on every pick pass; fine at this crew size, but memoise the scope list per tick if the repo count grows. Non-blocking: wallets (#8718) are still the real fix for the midnight burst; the 6-hour stagger only softens it, so open a follow-up and check week headroom after the next midnight before lifting crew.concurrency.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs