Public: anyone can read this; the crew can change it.

On a 1-CPU forge the crew run has the core: looks and checks wait for it, and one browser runs at a time #11574

Parked opened this
Closed without merging

On a forge with one CPU, looks, checks and snapshots now wait while a crew run is going, and a second browser waits for the first. Forges with two or more CPUs behave as before. `test/one-core.test.ts` passes (6 tests) and `tsc` is clean.

stopped parked by the crew deploy fdca2c0

opened this

On a forge with one CPU, looks, checks and snapshots now wait while a crew run is going, and a second browser waits for the first. Forges with two or more CPUs behave as before. test/one-core.test.ts passes (6 tests) and tsc is clean.

In test/looks.test.ts, three tests that launch a real chromium fail here. They fail the same way without my change, because the sandbox denies that connection, so I can't say whether they pass with it. The forge's own check will run them. I did not run bun scripts/gate.ts; I only confirmed it exists, so its result is unknown.

What changed

  • One new module, src/core/one_core.ts. A forge counts as one core when cpuLimit() is 1 or less, the same figure the forge already sizes its workers from. A crew run is "going" when an agent run is in the running state.
  • Checks. schedule() in checks/service.ts starts nothing while a crew run is going on a one-core forge, and re-checks every second until it ends. Queued checks then start in their usual order. This also holds the verification lane.
  • Looks. The look queue waits for the crew run to end before each capture, then takes its looks in queue order.
  • Snapshots. The snapshot waits the same way before it starts.
  • One browser. The chromium capture runs one at a time on a one-core forge.

Tests

test/one-core.test.ts covers:

  • 1 CPU: a real check is held while a fake crew run is active and starts once it ends.
  • 1 CPU: the wait that looks and snapshots use stays held until the run ends, and three browsers never overlap.
  • 2 CPUs: a check starts beside the crew run, and browsers overlap.

Left out

  • Looks and snapshots are not tested through their real services. Those tests exercise the shared wait, not a full look or snapshot start. Driving a look end to end needs a serving app, which the existing look tests already cover with a stand-in capturer.
  • The size is not read from src/spawn/sizes.ts at runtime. A forge does not know its own size, so I use its CPU limit, which for the small size is 1. The test confirms the small size has one CPU.
  • Merge-queue gates and the release/ browser tools are not held.
  • The one-second polling is a choice for simplicity. An event-driven wake-up would avoid it.

Closes #11561

commented

On a forge with one CPU, looks, checks and snapshots now wait while a crew run is going, and a second browser waits for the first. Forges with two or more CPUs behave as before. test/one-core.test.ts passes (6 tests) and tsc is clean.

In test/looks.test.ts, three tests that launch a real chromium fail here. They fail the same way without my change, because the sandbox denies that connection, so I can't say whether they pass with it. The forge's own check will run them. I did not run bun scripts/gate.ts; I only confirmed it exists, so its result is unknown.

What changed

  • One new module, src/core/one_core.ts. A forge counts as one core when cpuLimit() is 1 or less, the same figure the forge already sizes its workers from. A crew run is "going" when an agent run is in the running state.
  • Checks. schedule() in checks/service.ts starts nothing while a crew run is going on a one-core forge, and re-checks every second until it ends. Queued checks then start in their usual order. This also holds the verification lane.
  • Looks. The look queue waits for the crew run to end before each capture, then takes its looks in queue order.
  • Snapshots. The snapshot waits the same way before it starts.
  • One browser. The chromium capture runs one at a time on a one-core forge.

Tests

test/one-core.test.ts covers:

  • 1 CPU: a real check is held while a fake crew run is active and starts once it ends.
  • 1 CPU: the wait that looks and snapshots use stays held until the run ends, and three browsers never overlap.
  • 2 CPUs: a check starts beside the crew run, and browsers overlap.

Left out

  • Looks and snapshots are not tested through their real services. Those tests exercise the shared wait, not a full look or snapshot start. Driving a look end to end needs a serving app, which the existing look tests already cover with a stand-in capturer.
  • The size is not read from src/spawn/sizes.ts at runtime. A forge does not know its own size, so I use its CPU limit, which for the small size is 1. The test confirms the small size has one CPU.
  • Merge-queue gates and the release/ browser tools are not held.
  • The one-second polling is a choice for simplicity. An event-driven wake-up would avoid it.
commented

On a 1-CPU forge, the crew's run now has the core. Looks, checks and snapshots start after it ends, and only one browser runs at a time. A forge with two or more CPUs behaves as before. This push only changes test/one-core.test.ts. The only failure I could trace to this pull is the one-core test; the other three are my best reading of the log, and I couldn't confirm or fix them.

What changed

  • src/core/one_core.ts holds the rule. The crew's agent run holds the core while it is running.
  • src/checks/service.ts leaves queued checks queued and looks again every second until the crew run ends.
  • src/looks/service.ts and src/snapshot/service.ts wait on crewIdle before they start.
  • src/looks/capture.ts runs browsers one at a time through oneBrowser.

What the failing check showed

  • Four tests failed in the forge's full run: the one-core test at test/one-core.test.ts:62, plus stall-in-flight, traffic-truth and mcp-system-routes.
  • All 28 tests in those four files pass here, three runs in a row. bun x tsc --noEmit is clean. I didn't run scripts/gate.ts or scripts/design-lint.ts.
  • The one-core failure looks like a timeout. Its until() helper waited up to 8 s for a real check to leave queued, but bun's default test timeout is 5 s, so a loaded shard could trip it. I raised the helper's wait to 20 s and set a 45 s default timeout in that file.
  • The other three are timing and stall tests with no code path through this change, and none of them failed locally. I read them as load flakes on the busy check box, but I couldn't reproduce them or confirm that. If they fail again on this push, the log will say.

What I left out

  • A look or snapshot already running when a crew run starts isn't interrupted. The rule only holds back new starts.
  • The one-core tests drive the 1-CPU case with provideCpusForTest, not a real 1-CPU pod.

To try it

  • Run bun test test/one-core.test.ts.
commented

Nothing in my change fails: the last check was stopped by the pod's CPU watchdog, and no test failed. I've pushed an empty commit to re-run it, so the new check result is still pending.

What I found

  • The check was killed with "used 17.7 cores for 30s, over its limit of 16", and the log ends with no failing test.
  • The busiest process was a sh with 506 s of CPU. My change doesn't spawn a shell that could do that. test/one-core.test.ts runs only a trivial exit 0 check, and the waits in src/core/one_core.ts sleep rather than spin.
  • Locally, test/one-core.test.ts passes all 6 tests, and no stray sh is left running afterwards.

Why I didn't change code

  • The 16-core pod isn't a one-core forge, so the new hold doesn't apply there. The one-core test sets its own CPU count.
  • I found no cause to fix. If the check hits the same ceiling again, the cause is somewhere else in the 2,428-file run, and its log will name the process.

What the pull does

  • On a 1-CPU forge the crew's run has the core, and looks, checks and snapshots start after it ends.
  • Browsers run one at a time.
  • A forge with two or more CPUs behaves as before.

To try it: bun test test/one-core.test.ts.

commented

Parked by the crew: the check stayed red through 2 automatic tries

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs