Public: anyone can read this; the crew can change it.

Every environment is dark for a minute after a restart: a boot deploy spends 59s before the app is even started #535

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

After a restart, production now comes back before any preview, and every deploy log says where its time went. I have not found or fixed the cause of the 59-second gap, and there is no before/after from a real restart: I can't restart production from this sandbox. The full `forge.json` check passes locally (647 tests across 42 files, 0 fail).

shipped production is serving it deploy a5415ba

opened this

After a restart, production now comes back before any preview, and every deploy log says where its time went. I have not found or fixed the cause of the 59-second gap, and there is no before/after from a real restart: I can't restart production from this sandbox. The full forge.json check passes locally (647 tests across 42 files, 0 fail).

Where the time goes

  • New src/apps/timing.ts: a Stopwatch takes a lap at the end of each deploy step and prints all the laps on one timing: line.
  • runtime.ts writes that line just before the process is started. It splits the time since the deploy started into turn (the wait for a permit), manifest, checkout (marked warm when an existing tree is reused), data and setup.
  • process.ts writes a second line once the app answers: how long after it was started, and how soon it first printed anything.
  • The run log pages (web/client.ts, web/pages/runs.ts) style timing: lines like the forge's other log lines.

Production first

  • AppsRuntime.start now does every app's production first, then a second pass for previews. Before, it went app by app, production and previews together.
  • The Gate in src/git/env.ts takes a first option: a production deploy waiting for a permit goes ahead of waiting previews, and stays first-come-first-served among other productions. The lock and its limit (FORGE_APPS_CONCURRENCY) are unchanged.

Tests

  • test/apps.test.ts restarts the forge with two apps, each with an open pull. Both productions must answer before either preview, and the log must carry both timing: lines.
  • test/agents.test.ts covers the Gate ordering and the stopwatch's numbers.

What this does not do

  • I didn't change the checkout, data or spawn steps. The issue suspects parallel checkouts on one git directory, but I didn't confirm that, and changing them without a number to aim at risks the object store.
  • No test checks that N environments take less than N times one. That only matters if the gap turns out to be one environment waiting on another, and the turn lap will show that.
  • The next restart's timing: lines should be compared against the issue's two numbers (0.45s for a push, 59.5s for a boot).

Test change test/checks.test.ts was wrong, not the code: it wrote the queued checks' commits after the slot was already taken. On a loaded shard that writing could outlast the 2-second hold, so the oldest check took the free slot before the others were queued. That is the failure that closed pull #514. The commits are now written first; the assertion is unchanged.

To try it: merge, restart the forge, and open production's deploy run. Look for timing: starting the process … — turn …s · manifest …s · checkout …s · data …s · setup …s.

🤖 Generated with Claude Code

Closes #296

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs