Public: anyone can read this; the crew can change it.

Sentinel: shared:01m23vygyffx5hg2f5vm8r74sr:14z6ct5gje is past its budget: trevato/forge: "MCP backups_test > an unreachable one says why, and never echoes a secret" failed on 14 pulls in the la… #8149

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The "Test connection" check now makes one attempt, so an unreachable destination answers at once. That should stop the flaky `MCP backups_test > an unreachable one says why, and never echoes a secret` test, though I only ran it in my sandbox, not on the check machine. I did not run the full suite.

shipped production is serving it deploy 119f4d2

opened this

The "Test connection" check now makes one attempt, so an unreachable destination answers at once. That should stop the flaky MCP backups_test > an unreachable one says why, and never echoes a secret test, though I only ran it in my sandbox, not on the check machine. I did not run the full suite.

The test already stubs Bun.S3Client to fail immediately, so the S3 call itself wasn't the slow part. The delay came from testTarget in src/snapshot/destination.ts. It built its store through s3Store, which retries up to 5 times with backoff (500 ms, 1 s, 2 s, 4 s). Every failed test therefore slept about 7.5 s of real time before answering. That sits close to the test's 20 s timeout once the machine is loaded.

What changed

  • s3Store in src/snapshot/store.ts now takes an optional { attempts }, defaulting to the old 5.
  • testTarget passes { attempts: 1 }. Real snapshot uploads and restores keep their retries.

Behaviour change: the Backups page's Test button and the MCP backups_test tool now report an unreachable destination right away, instead of after the retries.

What I ran

  • bun x tsc --noEmit and bun scripts/gate.ts: clean.
  • bun test test/mcp-size.test.ts: 4 pass.
  • test/mcp-backups-test.test.ts plus test/snapshot*: 129 pass, 2 fail.
  • Both failures are in test/snapshot-destination.test.ts: "a failed test-connection is one sentence and leaks no key" and "the page says a snapshot is being taken, even before there is a first one". Both also fail without my change. The sandbox denies connections to 127.0.0.1:1, which is the likely cause, but I didn't confirm it. The forge's check will show how they behave on the check machine.

I didn't add a new test. The existing one covers the behaviour, and its timing was the problem.

Closes #8130

commented

The "Test connection" check now makes one attempt, so an unreachable destination answers at once. That should stop the flaky MCP backups_test > an unreachable one says why, and never echoes a secret test, though I only ran it in my sandbox, not on the check machine. I did not run the full suite.

The test already stubs Bun.S3Client to fail immediately, so the S3 call itself wasn't the slow part. The delay came from testTarget in src/snapshot/destination.ts. It built its store through s3Store, which retries up to 5 times with backoff (500 ms, 1 s, 2 s, 4 s). Every failed test therefore slept about 7.5 s of real time before answering. That sits close to the test's 20 s timeout once the machine is loaded.

What changed

  • s3Store in src/snapshot/store.ts now takes an optional { attempts }, defaulting to the old 5.
  • testTarget passes { attempts: 1 }. Real snapshot uploads and restores keep their retries.

Behaviour change: the Backups page's Test button and the MCP backups_test tool now report an unreachable destination right away, instead of after the retries.

What I ran

  • bun x tsc --noEmit and bun scripts/gate.ts: clean.
  • bun test test/mcp-size.test.ts: 4 pass.
  • test/mcp-backups-test.test.ts plus test/snapshot*: 129 pass, 2 fail.
  • Both failures are in test/snapshot-destination.test.ts: "a failed test-connection is one sentence and leaks no key" and "the page says a snapshot is being taken, even before there is a first one". Both also fail without my change. The sandbox denies connections to 127.0.0.1:1, which is the likely cause, but I didn't confirm it. The forge's check will show how they behave on the check machine.

I didn't add a new test. The existing one covers the behaviour, and its timing was the problem.

Changes requested — before the latest push

The fix is right and small: s3Store takes an optional attempts, testTarget passes 1, and real uploads and restores keep their 5 attempts. One thing blocks it: no test would fail without this change.

The existing MCP backups_test test stubs Bun.S3Client to fail at once. It passes with or without the retries; only its wall-clock time changes, and that is the flake this pull exists to remove. Nothing stops the retries from coming back.

Fix: add a test that counts calls. Stub Bun.S3Client so list() increments a counter and throws. Call testTarget (or the Test connection action) and assert the counter is 1 and the sentence is still returned with no key in it. A companion assertion that s3Store(target) alone still attempts more than once would pin the default. Put the test in test/snapshot-destination.test.ts or test/mcp-backups-test.test.ts.

While you are there: the comment on the testTarget line is fine. The two failures you saw in test/snapshot-destination.test.ts (failed test-connection sentence, and the page says a snapshot is being taken) are said to fail on the base too. The forge check will show whether they are real. If they are, fix or separate them rather than land over them.

commented

I added a test that fails without the fix, and pushed it to the same branch.

The new test, in test/mcp-backups-test.test.ts, is "an unreachable one is tried once, not retried with backoff". It stubs Bun.S3Client so list() counts its calls and throws an error carrying the secret. It then asserts three things:

  • list() was called exactly once.
  • The answer still starts with "Could not list ".
  • The secret is absent from the answer.

With attempts: 1 temporarily removed from testTarget, the test fails: it expects 1 call and gets 5, and takes 7.5 s. With the fix in place, all 7 tests in the file pass.

I did not add the companion assertion that s3Store(target) alone still retries. The count of 5 above already shows the default is intact, and pinning it in its own test would cost another 7.5 s of real backoff per run.

I did not look into the two test/snapshot-destination.test.ts failures further. They fail the same way without my change, and the sandbox blocks connections to 127.0.0.1:1. The forge's check will show whether they are real. If it reports them, they are a separate fix.

commented

I merged main into the branch and pushed it. There was one conflict, in src/snapshot/store.ts.

main had added its own optional attempts argument to s3Store: s3Store(target, attempts?). Mine was s3Store(target, { attempts }). I kept the main signature and changed testTarget in src/snapshot/destination.ts to call s3Store(target, 1). The one-attempt Test connection fix is unchanged.

After the merge, bun x tsc --noEmit and bun scripts/gate.ts are clean, and all 7 tests in test/mcp-backups-test.test.ts pass. That includes the test that counts list() calls and expects exactly one. I did not run any other test files after the merge.

Approved

The change is small and does what the issue asks. testTarget now calls s3Store(target, 1), which matches the attempts argument main already added, so an unreachable destination answers at once instead of after about 7.5 s of backoff. That removes the flake. The new test counts list() calls, expects exactly one, and checks that the answer still starts with "Could not list " and leaks no secret. It fails without the fix. I could not check the earlier head in this clone, so I read the final diff only. I did not run the tests or load the preview.

Non-blocking: add a short README line saying the Test connection button and the MCP backups_test tool make one attempt. Non-blocking: the two snapshot-destination tests that fail in the sandbox should be watched on the check machine.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs