Description
A SQL comment before or after a statement is a parse error. Only comments
inside a statement work.
Measured against the pinned 3.53.4 oracle (fed via stdin, because a leading
-- in argv is read as a CLI option by both binaries):
| input |
oracle |
ours |
-- c\nCREATE TABLE t(a); |
ok |
FAIL |
/* c */ CREATE TABLE t(a); |
ok |
FAIL |
CREATE TABLE t(a); -- c |
ok |
FAIL |
CREATE TABLE t(a /* col */); |
ok |
ok |
-- just a comment (nothing else) |
ok |
FAIL |
So the tokenizer handles comments once it is inside a statement, but a
statement that begins with one, or trailing text after the terminating
semicolon, is rejected.
Why it matters beyond neatness
Every hand-written schema file, migration and fixture is commented. A
consumer that reads its DDL from a .sql file — which is the normal way to
ship a schema — cannot feed it to this crate. It also makes our own corpus
fixtures awkward: tests/corpus/fixtures/consumers/sqe/catalog.sql has a
header comment explaining what it is, and that header alone is enough to make
the file unusable through sqlite-rs exec.
split_statements (src/parser/tokenizer.rs:1184) already does the right
thing — it keeps a leading comment attached to the statement that follows it,
which is what SQLite does. The failure is downstream, in the statement
parsers, which expect the first token to be a keyword.
Likely shape of the fix
Comments are trivia, not syntax. Either the tokenizer should not emit them as
tokens at statement level, or each statement parser should skip leading trivia
before dispatching on the first keyword. The second is narrower but has to be
done in every entry point; the first is one place but needs checking against
the mid-statement cases that already work, so that CREATE TABLE t(a /* col */) does not regress.
A comment-only input should succeed as a no-op, matching the oracle, rather
than erroring — worth stating explicitly because "empty program" is an easy
thing to get wrong in the other direction.
Acceptance Criteria
Complexity
Estimate: small
Reasoning: Comments are already tokenized correctly for the
mid-statement case, and split_statements already groups them correctly, so
the change is skipping leading trivia at the parser entry points. The care is
in the two directions that must not regress: comments inside string literals
must stay literal, and a comment-only input must succeed rather than produce
an error or an invalid empty program.
Found while building spec 013 Requirement 6's SQE consumer fixture family: the
fixture's own header comment made it unusable.
Refs: 002/Req-1, 013/Req-6, #695
Description
A SQL comment before or after a statement is a parse error. Only comments
inside a statement work.
Measured against the pinned 3.53.4 oracle (fed via stdin, because a leading
--inargvis read as a CLI option by both binaries):-- c\nCREATE TABLE t(a);/* c */ CREATE TABLE t(a);CREATE TABLE t(a); -- cCREATE TABLE t(a /* col */);-- just a comment(nothing else)So the tokenizer handles comments once it is inside a statement, but a
statement that begins with one, or trailing text after the terminating
semicolon, is rejected.
Why it matters beyond neatness
Every hand-written schema file, migration and fixture is commented. A
consumer that reads its DDL from a
.sqlfile — which is the normal way toship a schema — cannot feed it to this crate. It also makes our own corpus
fixtures awkward:
tests/corpus/fixtures/consumers/sqe/catalog.sqlhas aheader comment explaining what it is, and that header alone is enough to make
the file unusable through
sqlite-rs exec.split_statements(src/parser/tokenizer.rs:1184) already does the rightthing — it keeps a leading comment attached to the statement that follows it,
which is what SQLite does. The failure is downstream, in the statement
parsers, which expect the first token to be a keyword.
Likely shape of the fix
Comments are trivia, not syntax. Either the tokenizer should not emit them as
tokens at statement level, or each statement parser should skip leading trivia
before dispatching on the first keyword. The second is narrower but has to be
done in every entry point; the first is one place but needs checking against
the mid-statement cases that already work, so that
CREATE TABLE t(a /* col */)does not regress.A comment-only input should succeed as a no-op, matching the oracle, rather
than erroring — worth stating explicitly because "empty program" is an easy
thing to get wrong in the other direction.
Acceptance Criteria
tests/corpus/fixtures/consumers/sqe/catalog.sqlruns throughsqlite-rs execverbatim, comments included — this un-#[ignore]sconsumer_sqe_test::catalog_sql_file_runs_verbatimCREATE TABLE t(a /* col */),SELECT 1 /* x */ + 1)(
SELECT '-- not a comment')make lintboth clippy passesComplexity
Estimate: small
Reasoning: Comments are already tokenized correctly for the
mid-statement case, and
split_statementsalready groups them correctly, sothe change is skipping leading trivia at the parser entry points. The care is
in the two directions that must not regress: comments inside string literals
must stay literal, and a comment-only input must succeed rather than produce
an error or an invalid empty program.
Found while building spec 013 Requirement 6's SQE consumer fixture family: the
fixture's own header comment made it unusable.
Refs: 002/Req-1, 013/Req-6, #695