Skip to content

AGENTS.md

Latest commit

 

History

History
81 lines (60 loc) · 8.2 KB

AGENTS.md

File metadata and controls

81 lines (60 loc) · 8.2 KB
Raw

AGENTS.md

Guidance for AI agents and human contributors. memefish (github.com/cloudspannerecosystem/memefish) is a Cloud Spanner SQL (GoogleSQL) and GQL parser for Go: hand-written lexer + recursive descent parser, parses SQL to AST and unparses AST back to SQL.

Before You Start

  • Check whether main already handles the syntax: go run ./tools/parse 'SELECT 1' (statement mode; covers query/DDL/DML/GQL/CALL) or go run ./tools/parse -mode expr '1 + 2'.
  • Check open Issues AND open PRs (gh pr list) — many syntax gaps already have pending PRs.
  • Confirm the exact syntax in the official Cloud Spanner docs (see References). Never guess optional clauses such as IF EXISTS.

Commands

make gen            # regenerate ast/pos.go + ast/walk_internal.go (REQUIRED after ast/ast.go changes)
make lint           # golangci-lint (first run downloads bin/golangci-lint via curl)
make test           # all tests; also builds examples/ and tools/
make fmt            # format code
make update-result  # regenerate golden files — ALWAYS full, NEVER with -run:
                    #   it deletes ALL goldens first, so a filtered run silently loses the rest
make ci             # check-gen + lint + test; check-gen runs `git diff --exit-code`, so it
                    #   ALWAYS fails on uncommitted changes — use `make lint && make test` on a dirty tree

Layout

  • lexer.go, parser.go — lexer and parser (parse* methods grouped by comment banners: SELECT / Expr / DDL / DML / GQL)
  • parse_helpers.go — public API (ParseStatement, ParseQuery, ParseDDL, ParseDML, ParseExpr); split.go — SplitRawStatements
  • ast/ast.go — all AST node structs (~280 types); ast/sql.go — hand-written SQL() methods
  • ast/pos.go, ast/walk_internal.go — GENERATED by make gen; never edit by hand
  • testdata/input/{ddl,dml,query,expr,gql,gql_graph_pattern,statement}/ — one statement or standalone graph pattern per file, no trailing semicolon; !bad_ prefix marks expected-error cases. testdata/result/ — goldens, never hand-edit. Input basenames must be unique across statement categories because those categories are also re-run into the shared result/statement/; gql_graph_pattern is tested only by its standalone parser.

Each AST struct's doc comment holds a text/template syntax template and // pos = ... / // end = ... (poslang) comments; both notations are specified in the ast/ast.go package comment. The templates are documentation only — no test executes them — so update the template in the same change whenever you touch a struct's fields or its SQL().

Workflow: Adding/Modifying Syntax

Mirror an existing node with the same syntax shape (e.g. another DROP X name statement) at every step:

  1. Add the struct in ast/ast.go near its statement family, with doc template and pos/end comments; register its isStatement() / isDDL() / ... marker methods in the interface lists, keeping list order consistent with struct order.
  2. make gen.
  3. Add SQL() in ast/sql.go next to related nodes, consistent with the doc template.
  4. Add the parse* method in parser.go and wire it into the relevant dispatch switch (e.g. the DROP switch in parseDDL), mirroring a sibling.
  5. Smoke-test before writing goldens, e.g. go run ./tools/parse -mode ddl 'DROP VIEW foo' — confirm the AST shape and that the unparsed SQL matches the input.
  6. Add inputs under testdata/input/<category>/ covering: the minimal form, every optional clause, and 2+ elements for any list field (single-element tests cannot catch a wrong separator).
  7. make update-result (full run).
  8. git status + git diff --stat: only intended goldens may change or be added — unexpected diffs mean you altered unrelated unparsing. Read the new .txt files and check the AST shape and --- SQL output.
  9. make lint && make test, both exit 0.

Critical Rules

  • Keywords — check token/keywords.go first; the wrong method compiles but silently never matches. Reserved keyword (listed there): p.Token.Kind == "SELECT" / p.expect("SELECT"). Everything else is a pseudo keyword: p.Token.IsKeywordLike("SCHEMA") / p.expectKeywordLike("SCHEMA").
  • Optional token.Pos fields MUST be initialized to token.InvalidPos, never 0 — position 0 (start of file) is valid.
  • Naming: CreateTable, not CreateTableStmt. Use *ast.Ident for never-qualified names, *ast.Path for potentially qualified ones.
  • Reuse an existing node/helper when the documented syntax maps cleanly onto its fields, SQL() output, positions, and visitor semantics; add a new node when required keywords, clause order, qualification rules, or position needs differ, or reuse would make fields misleading. Don't encode every documented semantic restriction into types — follow the shape nearby parser code already uses.
  • Don't add ScalarSchemaType entries for non-reserved keywords — use the NamedType fallback. Don't store token.Pos for static keywords unless needed for Pos()/End() or optional-clause detection.
  • SQL() helpers: sqlOpt(left, node, right), sqlJoin(slice, sep), strOpt(pred, s), strIfElse(pred, a, b). Reconstruct ALL keywords/punctuation, including ones not stored in the AST.
  • Parser idioms: inline single-keyword checks for simple optional clauses; tryParse* methods for multi-token lookahead or branching; parseCommaSeparatedList(p, parseFn) over hand-rolled comma loops; peek without consuming via lexer := p.Clone() + deferred restore (lookahead* methods returning bool). Error recovery: Bad* nodes, errors accumulate in p.errors — design in docs/content/error-recover/_index.md.
  • GQL shares the lexer and reserved keywords with GoogleSQL — never add reserved keywords for GQL. GQL structural nodes keep the Statement suffix; operational nodes drop it.

Etiquette

  • One syntax feature per PR, referencing its issue (fixes #NNN); follow the shape of recent merged syntax PRs (e.g. #358, #360). Commit prefixes: feat: / fix: / doc: / chore: / ci:.
  • Pre-v1 (v0.x): breaking changes bump minor and need maintainer discussion first (see #210); everything else bumps patch. Prefer additive AST changes — downstream tools type-switch on AST nodes and construct them programmatically.

Troubleshooting

Symptom Fix
"undefined" / "missing method" after editing ast/ast.go make gen
Merge conflict in ast/pos.go never resolve by hand — make gen
Golden file mismatch make update-result (full run)
AST range error optional token.Pos left 0 — use token.InvalidPos
lint: "Can't read config ... 'Version' expected a map" stale binary — rm bin/golangci-lint && make install-dep
Sandboxed runs: Go cache/network errors use GOCACHE=/private/tmp/memefish-go-build; retry once outside the sandbox before diagnosing

References

Primary source for ALL syntax: official Cloud Spanner docs — overview (index of reference pages), DDL, DML, Query, GQL. Search within the Spanner docs for other pages (lexical rules, functions, operators, data types) rather than guessing.

When the docs are ambiguous or incomplete: emulator ddl_parser.jjt for DDL; GoogleSQL googlesql.tm for query/DML/expressions.

CRITICAL — do not mix dialects: Spanner's DDL is independent of the DDL of the GoogleSQL reference implementation (google/googlesql) — never apply its DDL grammar to Spanner DDL unless Spanner docs or the emulator confirm it. BigQuery's dialect is also called "GoogleSQL", but BigQuery's SQL/GQL — DDL included — is a subset of the GoogleSQL frontend and generally differs from Spanner GoogleSQL; never use BigQuery docs as a syntax source for memefish.