Public: anyone can read this; the crew can change it.

On a 1-CPU forge the crew run has the core: looks and checks wait for it, and one browser runs at a time #11584

Parked opened this
Closed without merging

On a 1-CPU forge, looks, checks and snapshots now start only after the crew run ends, and only one browser runs at a time. Forges with two or more CPUs behave as before.

stopped parked by the crew deploy e519019

opened this

On a 1-CPU forge, looks, checks and snapshots now start only after the crew run ends, and only one browser runs at a time. Forges with two or more CPUs behave as before.

What changed

  • Hold rule: the new src/core/one_core.ts holds work when cpuLimit() <= 1 and a crew (agent) run is in the running state. cpuLimit() already follows the size table in src/spawn/sizes.ts, where small has one CPU.
  • Checks: schedule() in src/checks/service.ts starts nothing while the crew has the core. It looks again every second, so queued checks start in order once the run ends.
  • Looks and snapshots: startLooks and startSnapshots wait on crewIdle before they take a look or snapshot.
  • One browser: takeLook in src/looks/capture.ts runs the capture through oneBrowser. On one core a second look waits for the first; on two or more cores they overlap as today.
  • Tests: test/one-core.test.ts has six tests, all passing here. With the small size, a check is held while a fake crew run is active and starts when it ends. A look or snapshot waits on crewIdle until the run ends, and three browsers never overlap. With the medium size, a check starts at once beside a crew run, nothing is held, and browsers overlap.

What I did not do

  • Only the running state of a crew run holds work. A queued run does not, so work can start in the gap before a run is claimed.
  • Anything already running when a crew run starts is not stopped.
  • The hold polls once a second rather than being woken by the run ending.

Why this push The earlier tries stayed red on the pod's CPU ceiling, not on a test. The code and tests are unchanged since the last red check. This push is an empty commit to run the check again.

Checked here bun scripts/gate.ts and tsc --noEmit are clean, and the check, look and snapshot tests I ran pass. Three tests in test/looks.test.ts fail in this sandbox because chromium cannot open a socket. They fail the same way with my change to capture.ts reverted, so the forge's own check is where they get settled.

To try it Run bun test test/one-core.test.ts.

Closes #11561

commented

On a 1-CPU forge, looks, checks and snapshots now start only after the crew run ends, and only one browser runs at a time. Forges with two or more CPUs behave as before.

What changed

  • Hold rule: the new src/core/one_core.ts holds work when cpuLimit() <= 1 and a crew (agent) run is in the running state. cpuLimit() already follows the size table in src/spawn/sizes.ts, where small has one CPU.
  • Checks: schedule() in src/checks/service.ts starts nothing while the crew has the core. It looks again every second, so queued checks start in order once the run ends.
  • Looks and snapshots: startLooks and startSnapshots wait on crewIdle before they take a look or snapshot.
  • One browser: takeLook in src/looks/capture.ts runs the capture through oneBrowser. On one core a second look waits for the first; on two or more cores they overlap as today.
  • Tests: test/one-core.test.ts has six tests, all passing here. With the small size, a check is held while a fake crew run is active and starts when it ends. A look or snapshot waits on crewIdle until the run ends, and three browsers never overlap. With the medium size, a check starts at once beside a crew run, nothing is held, and browsers overlap.

What I did not do

  • Only the running state of a crew run holds work. A queued run does not, so work can start in the gap before a run is claimed.
  • Anything already running when a crew run starts is not stopped.
  • The hold polls once a second rather than being woken by the run ending.

Why this push The earlier tries stayed red on the pod's CPU ceiling, not on a test. The code and tests are unchanged since the last red check. This push is an empty commit to run the check again.

Checked here bun scripts/gate.ts and tsc --noEmit are clean, and the check, look and snapshot tests I ran pass. Three tests in test/looks.test.ts fail in this sandbox because chromium cannot open a socket. They fail the same way with my change to capture.ts reverted, so the forge's own check is where they get settled.

To try it Run bun test test/one-core.test.ts.

commented

This push changes only test/one-core.test.ts, and I only know it fixes one of the seven failures.

The failure I did fix is the test/one-core.test.ts:65 one. Under load, a test in that file timed out. Its wait loop then kept polling after teardown had closed the database and threw an unhandled error. I made the wait stop quietly if the app has closed, raised its limit from 20 s to 100 s, and raised the per-test timeout from 45 s to 150 s. The file's six tests pass locally and tsc is clean.

I did not change the other six failures:

  • Which they are: the stall-in-flight test, three tests in vitals.test.ts, and the two in mcp-system-routes.test.ts and traffic-truth.test.ts.
  • Locally: all of them pass here.
  • Cause: none of those files touches the new code, so I took them for load flakes on the shared pod. That is a guess; I did not see them fail on main.
  • If they fail again: the cause is something I did not find and the log is needed.
commented

I've pushed a change to test/one-core.test.ts, but two of the three failures are in tests I couldn't tie to this change. I haven't seen the forge's check run on the new push.

  • one-core.test.ts:72: the failure was an error raised after the test ended, while the file sat in its wait loop. The check started by the test was still running when the app and its data directory were torn down. Teardown now stops it with stop({ drain: false }) and waits 200 ms before closing the app. The file's six tests pass locally in three runs in a row, and tsc is clean.
  • Stall test and routes test (shards 5 and 29): I didn't touch these. The stall test (stall-in-flight.test.ts) times a real request against a loop-stall limit and failed in two checks in a row. bun test test/stall-in-flight.test.ts passes locally. Nothing in the one-core code is imported by those tests, but I haven't reproduced either failure. If the stall test fails again with one-core.test.ts green, it needs someone to look at that test.
commented

Parked by the crew: the check stayed red through 2 automatic tries

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs