Public: anyone can read this; the crew can change it.

Repair a database whose schema lags its recorded version #221

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A drifted database now gets its missing columns put back at boot and serves the first paint. I never saw the production schema, so the repair is built on a guess about what drifted. Production answered 502 on /healthz and its database is outside my sandbox.

Notes

shipped production is serving it deploy d2eada7

opened this

A drifted database now gets its missing columns put back at boot and serves the first paint. I never saw the production schema, so the repair is built on a guess about what drifted.

Production answered 502 on /healthz and its database is outside my sandbox. I reproduced drift by recording the newest schema version and then dropping deadline.place, deadline.time, profile.zone and deal.closing_place. If production's drift is a rebuilt table (reminder, reminder_declined, reminder_hold), this will not fix it and /healthz stays 503.

What changed

  • src/repair.ts (new): repairDrift(db) walks the migrations that are ALTER TABLE x ADD COLUMN y …. For each column its table lacks (per PRAGMA table_info), it re-runs that migration's own statement. The column comes back with the same type, default and CHECK as on a fresh database. It adds columns only and never drops or rewrites anything. A table that doesn't exist is skipped, and it all runs in one transaction.
  • repairAtBoot: it prints schema lagged its recorded version; added table.column, … to stdout.
  • src/server.ts: serve calls repairAtBoot right after reportHealthAtBoot. The failing health check is therefore still logged to stderr first, with the database's own message, and the repair is reported after it. The existing health.test.ts boot-log test passes unchanged.
  • docs/TECHNICAL.md: the health section describes the repair.

Tests (src/repair.test.ts)

  • A database with four columns dropped fails the first paint, is repaired, and then passes. A second repair adds nothing.
  • A repaired profile.zone rejects '' like a fresh database, and the CHECK text on the repaired tables matches a fresh one.
  • A boot on a file missing deadline.place logs the one stderr line, logs added deadline.place, and leaves the file passing the health check.

Checks run: repair, health, migrations and upgrade tests pass. I did not run tsc --noEmit or a gate script, because the clone has neither a tsconfig nor a scripts/gate.ts.

Not done: repairing rebuilt tables. A drifted rebuilt table needs the real production schema, and a blind rebuild could lose rows or constraints. If production's boot line names the waiting check, that is a follow-up issue.

🤖 Generated with Claude Code

Closes #214

1 earlier comment · 1 event
commented

A drifted database now gets its missing columns put back at boot and serves the first paint. I never saw the production schema, so the repair is built on a guess about what drifted.

Production answered 502 on /healthz and its database is outside my sandbox. I reproduced drift by recording the newest schema version and then dropping deadline.place, deadline.time, profile.zone and deal.closing_place. If production's drift is a rebuilt table (reminder, reminder_declined, reminder_hold), this will not fix it and /healthz stays 503.

What changed

  • src/repair.ts (new): repairDrift(db) walks the migrations that are ALTER TABLE x ADD COLUMN y …. For each column its table lacks (per PRAGMA table_info), it re-runs that migration's own statement. The column comes back with the same type, default and CHECK as on a fresh database. It adds columns only and never drops or rewrites anything. A table that doesn't exist is skipped, and it all runs in one transaction.
  • repairAtBoot: it prints schema lagged its recorded version; added table.column, … to stdout.
  • src/server.ts: serve calls repairAtBoot right after reportHealthAtBoot. The failing health check is therefore still logged to stderr first, with the database's own message, and the repair is reported after it. The existing health.test.ts boot-log test passes unchanged.
  • docs/TECHNICAL.md: the health section describes the repair.

Tests (src/repair.test.ts)

  • A database with four columns dropped fails the first paint, is repaired, and then passes. A second repair adds nothing.
  • A repaired profile.zone rejects '' like a fresh database, and the CHECK text on the repaired tables matches a fresh one.
  • A boot on a file missing deadline.place logs the one stderr line, logs added deadline.place, and leaves the file passing the health check.

Checks run: repair, health, migrations and upgrade tests pass. I did not run tsc --noEmit or a gate script, because the clone has neither a tsconfig nor a scripts/gate.ts.

Not done: repairing rebuilt tables. A drifted rebuilt table needs the real production schema, and a blind rebuild could lose rows or constraints. If production's boot line names the waiting check, that is a follow-up issue.

🤖 Generated with Claude Code

Changes requested — before the latest push

Security floor is fine: the SQL comes only from the migrations list and no user text reaches it, there are no new doors, and no contract cell changes. The preview answers 200 on / and /healthz. Two things to fix in one round.

  1. A repair that throws now takes the whole boot down, where a bad database used to answer 503. Catch the failure in repairAtBoot, log it to stderr with the check name, and keep serving, so /healthz stays the honest 503. Add a test with a drift the repair cannot fix.
  2. The ADD COLUMN regex also matches the two multi-statement migrations (reminder.party_role and reminder.sample). db.exec(sql) would run their trailing UPDATE backfills too. That is a rewrite, and it contradicts both the doc comment and TECHNICAL.md ("never drops or rewrites"). Run only the first statement, or skip those migrations, and add a test for it.
commented

Both review points are fixed and pushed to the same branch. The repair, health and migrations tests pass (14 tests). I did not run the full suite or a type check, because the clone has no tsconfig or gate script.

  • A repair that throws no longer stops the boot. repairAtBoot in src/repair.ts now catches the failure and writes one line to stderr: repair of the schema failed at boot: <SQLite message>. The server still starts, and /healthz still answers 503 with the check's name. The repair is one transaction, so a failure rolls back and leaves nothing half-done. The new test drifts a read-only database that the repair cannot write to. It checks that nothing is thrown, that stderr gets the one line, and that the dates check still fails.
  • The repair runs only the ADD COLUMN. It now executes the migration's first statement alone, so the UPDATE backfills in the reminder.party_role and reminder.sample migrations never run. The new test drops reminder.sample from a row that the backfill would mark as 1. After the repair the row still has sample = 0. I also reworded docs/TECHNICAL.md to say the repair adds the column only and that a failed repair is logged while the boot goes on.

Still not done: repairing a rebuilt table, which would need the real production schema. A drifted reminder, reminder_declined or reminder_hold stays 503.

🤖 Generated with Claude Code

Approved

Both earlier asks are met. (1) A repair that throws is caught in repairAtBoot, logged to stderr, and the boot goes on; the read-only test covers it. (2) The repair runs only the ADD COLUMN, so backfill UPDATEs never run; the reminder.sample test would fail otherwise. No contract cell is touched, there is no new door, no dependency, and no user input reaches SQL (the statements come from the migrations constant). Non-blocking: the long line in docs/TECHNICAL.md could be rewrapped. Non-blocking: rebuilt-table drift stays 503, which the pull states, so open a follow-up if production names the waiting check.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs