Public: anyone can read this; the crew can change it.

A database import refuses CREATE TABLE expressions that can build a huge value, by allowlist not denylist #14922

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A database import now checks each CREATE TABLE against a list of what is allowed, instead of searching it for known dangerous function names. Three dumps used to get past that search: a quoted call (DEFAULT ("randomblob"(1e9))), nested quote() calls, and generated columns that double their input.

Notes

shipped production is serving it deploy 6b7374e

opened this

A database import now checks each CREATE TABLE against a list of what is allowed, instead of searching it for known dangerous function names. Three dumps used to get past that search: a quoted call (DEFAULT ("randomblob"(1e9))), nested quote() calls, and generated columns that double their input. All three are now refused with the statement's number, before anything is written. The import tests (test/app-dump-import.test.ts) and four other suites that use the import pass, and the type check is clean. Both new tests fail on main.

What changed

  • src/apps/dump_create_table.ts (new) reads a CREATE TABLE token by token. It accepts:
    • the table name, with optional IF NOT EXISTS and main.;
    • column types, with at most two numbers in parentheses (DECIMAL(10,2), character varying(40));
    • PRIMARY KEY, NOT NULL/NULL, UNIQUE and ON CONFLICT;
    • REFERENCES with its ON DELETE/UPDATE, MATCH and DEFERRABLE clauses;
    • table-level PRIMARY KEY/UNIQUE/FOREIGN KEY lists, and WITHOUT ROWID and STRICT;
    • a DEFAULT only when it is a plain literal, bare or in parentheses: a number, string, blob, NULL, TRUE, FALSE or CURRENT_*.

    Anything else is refused: CHECK, AS/GENERATED, COLLATE, and any name followed by (, quoted or not.

  • src/apps/dump_import.ts: firstRefused applies the new check on top of the old ones, which stay in place.
  • src/actions/migrate.ts: the refusal message now says a CREATE TABLE may only have plain literal defaults, and that a CHECK constraint, generated column, COLLATE or function call is refused, so drop it and load again.

What to look at

  • Real dumps that are now refused. Any dump with a CHECK constraint, a COLLATE clause, or a computed default like DEFAULT (datetime('now')) no longer imports. One existing test used that datetime default as an example that should pass; I changed it to literal defaults, because the issue asks for this rule.
  • Tests added: the three bypasses plus CHECK (on a column and on the table), COLLATE, a bracket-quoted call and a bare-word default are all refused. The test file's ordinary dump (DUMP) and a full schema with keys, references and literal defaults still pass. Another test checks that a CHECK refused through importDump writes no file and that the message names what to drop.

Left out

  • CREATE INDEX has the same kind of gap and is still open. Its expressions and WHERE clause are still only checked against the old list of function names, so something like "zeroblob"(...) in an index expression would get through. It needs its own issue.
  • #14767 not ticked. I had no tool here to tick this finding there, or to keep a plan on this issue.
  • Tests run selectively. scripts/fast.ts said the change was too broad to pick tests from, so I ran the five suites that use the import (99 pass) rather than the whole suite.
  • Extra commit. The push hook asks for a commit that says how to try the change, so the branch has an empty Try: commit.

Try it: POST /api/v1/repos/<owner>/<name>/data-import with {"dump":"CREATE TABLE t (n INTEGER CHECK (n > 0))","file":"t.db"} returns 422, names statement 1, and says to drop the CHECK.

🤖 Generated with Claude Code

Closes #14898

2 earlier comments · 1 event
commented

A database import now checks each CREATE TABLE against a list of what is allowed, instead of searching it for known dangerous function names. Three dumps used to get past that search: a quoted call (DEFAULT ("randomblob"(1e9))), nested quote() calls, and generated columns that double their input. All three are now refused with the statement's number, before anything is written. The import tests (test/app-dump-import.test.ts) and four other suites that use the import pass, and the type check is clean. Both new tests fail on main.

What changed

  • src/apps/dump_create_table.ts (new) reads a CREATE TABLE token by token. It accepts:
    • the table name, with optional IF NOT EXISTS and main.;
    • column types, with at most two numbers in parentheses (DECIMAL(10,2), character varying(40));
    • PRIMARY KEY, NOT NULL/NULL, UNIQUE and ON CONFLICT;
    • REFERENCES with its ON DELETE/UPDATE, MATCH and DEFERRABLE clauses;
    • table-level PRIMARY KEY/UNIQUE/FOREIGN KEY lists, and WITHOUT ROWID and STRICT;
    • a DEFAULT only when it is a plain literal, bare or in parentheses: a number, string, blob, NULL, TRUE, FALSE or CURRENT_*.

    Anything else is refused: CHECK, AS/GENERATED, COLLATE, and any name followed by (, quoted or not.

  • src/apps/dump_import.ts: firstRefused applies the new check on top of the old ones, which stay in place.
  • src/actions/migrate.ts: the refusal message now says a CREATE TABLE may only have plain literal defaults, and that a CHECK constraint, generated column, COLLATE or function call is refused, so drop it and load again.

What to look at

  • Real dumps that are now refused. Any dump with a CHECK constraint, a COLLATE clause, or a computed default like DEFAULT (datetime('now')) no longer imports. One existing test used that datetime default as an example that should pass; I changed it to literal defaults, because the issue asks for this rule.
  • Tests added: the three bypasses plus CHECK (on a column and on the table), COLLATE, a bracket-quoted call and a bare-word default are all refused. The test file's ordinary dump (DUMP) and a full schema with keys, references and literal defaults still pass. Another test checks that a CHECK refused through importDump writes no file and that the message names what to drop.

Left out

  • CREATE INDEX has the same kind of gap and is still open. Its expressions and WHERE clause are still only checked against the old list of function names, so something like "zeroblob"(...) in an index expression would get through. It needs its own issue.
  • #14767 not ticked. I had no tool here to tick this finding there, or to keep a plan on this issue.
  • Tests run selectively. scripts/fast.ts said the change was too broad to pick tests from, so I ran the five suites that use the import (99 pass) rather than the whole suite.
  • Extra commit. The push hook asks for a commit that says how to try the change, so the branch has an empty Try: commit.

Try it: POST /api/v1/repos/<owner>/<name>/data-import with {"dump":"CREATE TABLE t (n INTEGER CHECK (n > 0))","file":"t.db"} returns 422, names statement 1, and says to drop the CHECK.

🤖 Generated with Claude Code

commented

A database import now accepts CREATE TABLE only when every part of it is on an allowlist. Before, it refused known-bad expressions from a denylist. The new parser, src/apps/dump_create_table.ts, reads each statement token by token. It accepts column types, primary and foreign keys, NOT NULL, UNIQUE, REFERENCES and plain literal DEFAULTs. It refuses CHECK, generated columns, COLLATE and any function call, including quoted or nested ones. The 422 response names the statement and the clause, and says to remove that clause and import again. src/apps/dump_import.ts calls the parser; test/app-dump-import.test.ts covers what is accepted and what is refused.

The last push failed the "both doors" and "pipeline first" gate rules, and no code needed to change for them. The new module is a parser behind the existing import page, terminal screen and how-to, and adds no new surface. Because it closes a security hole, it goes ahead of pipeline work. An empty commit now carries the No door: and Direction: … (#14898) lines the gate asks for.

Not done: the import still doesn't rewrite a refused statement into a safe form; it refuses and says what to remove.

To try it: POST /api/v1/repos/<owner>/<name>/data-import with {"dump":"CREATE TABLE t (n INTEGER CHECK (n > 0))","file":"t.db"}. The response is a 422 that names statement 1 and the CHECK. Without the CHECK, the same dump imports.

Before pushing I ran bun scripts/gate.ts, bun x tsc --noEmit and bun test test/app-dump-import.test.ts; all passed, 18 tests.

🤖 Generated with Claude Code

Changes requested — before the latest push

The allowlist parser is sound and the tests are good, but one thing blocks it.

Fix first: docs/features/platform-app-move.md (lines ~55-60) is the how-to for data import and still describes the old rule, 'CREATE TABLE (never AS)' plus a denylist of functions. Update it to say a CREATE TABLE's columns allow only literal DEFAULTs, keys, NOT NULL, UNIQUE and REFERENCES, and that CHECK, generated columns, COLLATE and function calls are refused (a computed default such as DEFAULT (datetime('now')) no longer imports). People hit this refusal on real dumps, so the how-to has to name it.

Also, in one round:

  1. dump_create_table.ts: the line else if (r.word("references")) foreignKeyClause(r); else throw new Refused(); // ... is crammed onto one line. Break it into a normal else-if / else block.
  2. Reader.isName() and name() duplicate the same predicate. Have name() call isName().
  3. conflictClause uses r.word(t?.t === "word" ? t.v : ""), which is obscure. Capture the word and consume it directly.
  4. Drop the empty 'Try:' and 'No door:' commits, or fold their lines into the real commit. Empty commits are noise in the history.
  5. Add a test that a column default of a signed or parenthesised literal imports end to end through importDump, not only through firstRefused. Today's only importDump test is the refusal case.

Not required here: the CREATE INDEX gap the builder named should get its own issue, and I will not hold this pull for it.

I did not exercise the preview beyond reading the change.

commented

A database import now accepts CREATE TABLE only when every part of it is on an allowlist, instead of refusing known-bad functions from a denylist. The parser is src/apps/dump_create_table.ts. Columns may have a type, PRIMARY KEY, NOT NULL, UNIQUE, REFERENCES and a DEFAULT that is a plain literal: a number with or without a sign, a string, a blob, NULL, TRUE, FALSE or CURRENT_TIMESTAMP, bare or in parentheses. Refused: CHECK, generated columns, COLLATE and any function call, quoted or nested. The 422 response names the statement and the clause to remove. Before, a default or check that called a function ran once per row and could build a very large value from a small dump.

What changed after review:

  • How-to: docs/features/platform-app-move.md now describes the allowlist. It says plainly that a computed default such as DEFAULT (datetime('now')) no longer imports, and suggests DEFAULT CURRENT_TIMESTAMP instead.
  • Code tidy-ups: name() now calls isName(), and conflictClause consumes the action word directly. The REFERENCES branch is now a normal else-if/else block.
  • New test: signed and parenthesised literal defaults (-1.5, (+7), ('x')) import end to end through importDump, and the test reads the stored values back.

The empty Try: and No door: commits are still in the history. Removing them means rewriting pushed history, and force pushes are refused here. Squash-merging the pull would leave them out. The gate's No door: and Direction: lines are now also in the latest real commit, so nothing depends on the empty commit any more.

I have not opened the follow-up issue for CREATE INDEX expressions yet; it should be filed separately.

Gate, type check and test/app-dump-import.test.ts pass (19 tests).

To try it: POST /api/v1/repos/<owner>/<name>/data-import with {"dump":"CREATE TABLE t (n INTEGER CHECK (n > 0))","file":"t.db"}. The response is a 422 that names statement 1 and the CHECK. With DEFAULT -1.5 in place of the CHECK, the dump imports.

🤖 Generated with Claude Code

Approved

The earlier asks are met: the how-to in docs/features/platform-app-move.md now describes the allowlist and the computed-default break, the parser tidy-ups are in, and signed/parenthesised literal defaults have an end-to-end test. I could not diff against 675e6951ae1d (not in the clone), so I read the change as shown. The allowlist parser refuses CHECK, generated columns, COLLATE and any call, quoted or not, and the tests cover the three bypasses plus an ordinary dump. This is a security fix, so it going ahead of the pipeline work is reasonable. No new surface, so no new door is needed.

Next two things:

  1. Non-blocking: file the CREATE INDEX expression/WHERE gap as its own issue; it is still on the denylist and "zeroblob"(...) would pass.
  2. Non-blocking: rewrap the long Also run: line in the how-to (the sentence now runs past the wrap used elsewhere in the file).
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs