Skip to content

adding-sources.md

Latest commit

 

History

History
187 lines (129 loc) · 7.19 KB

adding-sources.md

File metadata and controls

187 lines (129 loc) · 7.19 KB
Raw

Adding Sources (Scanners + Models)

This guide shows how to add a new source framework (for example Ent or Dapper) so Valk Guard can lint its SQL and, when available, use model metadata for schema-aware rules.

Source Architecture

A source integration can include:

  1. SQL scanner that emits scanner.SQLStatement.
  2. Model extractor that emits generic schema.ModelDef.

Runtime wiring is registry-based:

  • scanner bindings: defaultScannerBindings() in internal/engine/source_bindings.go
  • model bindings: defaultModelBindings(cfg) in internal/engine/source_bindings.go

internal/engine.Run() consumes those bindings, so adding a source does not require new hardcoded switches in the CLI.

Core Contracts

SQL Scanner

Implement scanner.Scanner:

type Scanner interface {
    Scan(ctx context.Context, paths []string) iter.Seq2[SQLStatement, error]
}

Set SQLStatement.SQL, File, Line, Engine, and Disabled.

Model Extractor

Implement schema.ModelExtractor:

type ModelExtractor interface {
    ExtractModels(ctx context.Context, paths []string) ([]ModelDef, error)
}

Normalize framework metadata to:

  • ModelDef.Table, ModelDef.Source, ModelDef.Columns
  • ModelColumn.Name, ModelColumn.Type, ModelColumn.Field
  • ModelColumn.MappingKind / MappingSource for explicit vs inferred provenance
  • ModelDef.TableMappingKind / TableMappingSource for explicit vs inferred table provenance

Provenance contract:

  • Use explicit when mapping comes from declared framework metadata.
  • Use inferred when mapping comes from fallback naming.
  • Populate ...MappingSource with a stable provider token (<source>.<provider>) so rule logic can stay source-agnostic.

Integration Steps

1) Add Engine Constant

Add a new engine in internal/scanner/scanner.go, for example:

const EngineEnt Engine = "ent"

Then add it to built-in engine allowlist in internal/scanner/engines.go (knownEngines), so config engine validation accepts it.

2) Add Scanner Package

Create internal/scanner/<source>/... implementing Scan(...).

Examples:

  • internal/scanner/goqu/goqu_scanner.go (Go AST-based)
  • internal/scanner/sqlalchemy/sqlalchemy_scanner.go (Python subprocess)
  • internal/scanner/csharp/scanner.go (Go wrapper around a cached embedded Roslyn AST extractor; raw EF Core APIs plus synthetic SQL for deterministic DbSet/LINQ chains)

3) Register Scanner Binding

Add an entry in defaultScannerBindings():

{
    name: "ent",
    impl: &entscanner.Scanner{},
    extensions: []string{".go"},
}

File discovery is automatic from registered extensions via requiredExtensions(...) and collectScannerInputs(...). Top-level sources.<engine>: false config filters the binding before discovery, so disabled sources do not collect files or invoke external runtimes.

The C# wrapper materializes the embedded Roslyn extractor under os.UserCacheDir()/valk-guard/roslynextractor-<contenthash>, publishes it once as a self-contained binary for the current OS/architecture, and then executes that binary directly until the embedded extractor source changes. This trades cache size for scan-time startup cost: the published binary is intentionally not committed to the repository, is about 80 MB on linux-x64 with Roslyn included, and lets repeated scans avoid dotnet run, restore, and build work.

Synthetic SQL conventions:

  • Prefix generated statements with /* valk-guard:synthetic <source> */ so output makes the origin explicit.
  • Render non-literal bind values as PostgreSQL-style numbered placeholders ($1, $2, ...), resetting numbering per emitted statement.
  • Prefer structurally valid SQL over perfect fidelity; the parser must accept the statement and the rule engine must see the relevant clauses.

Parity checklist for ORM/query-builder scanners:

Feature Expected synthetic shape
Select star SELECT * FROM table
Missing write filter UPDATE table SET col = $1 / DELETE FROM table without WHERE
Unbounded select SELECT cols FROM table without LIMIT
LIKE / ILIKE col LIKE pattern / col ILIKE pattern
IN / NOT IN col IN ($1, $2, $3) / col NOT IN ($1, $2, $3)
Order / offset `ORDER BY col ASC
Group / having GROUP BY col HAVING predicate
Distinct SELECT DISTINCT ...
Aggregates SUM(col), MIN(col), MAX(col), AVG(col), COUNT(*)
Joins Preserve join type and emit real ON a = b when the AST exposes it; otherwise use ON 1=1
Row locks Preserve raw FOR UPDATE; only synthesize ORM lock methods that exist in the source framework

Known caveat: raw SQL format-string normalization rewrites {{0}}, {{1}}, ... to $1, $2, ... for parser/rule consistency. Literal brace placeholders in raw SQL text may be rewritten.

4) Optional: Add Model Extractor

If the source has model metadata, add internal/schema/<source>/... implementing schema.ModelExtractor.

For Go sources, model extraction mode is configurable:

go_model:
  mapping_mode: strict # strict | balanced | permissive

When writing the extractor, emit normalized mapping provenance so existing schema rules can work without source-specific branches:

  • explicit column mappings: ModelColumn.MappingKind = explicit
  • inferred column mappings: ModelColumn.MappingKind = inferred
  • explicit table mappings: ModelDef.TableMappingKind = explicit
  • inferred table mappings: ModelDef.TableMappingKind = inferred

5) Register Model Binding

Add an entry in defaultModelBindings():

{
    source:        schema.ModelSourceEnt,
    extractor:     &entmodel.Extractor{},
    extensions:    []string{".go"},
    configEngines: []scanner.Engine{scanner.EngineEnt},
    queryEngines:  []scanner.Engine{scanner.EngineEnt},
}

Binding fields control behavior:

  • configEngines: which rules.<id>.engines values enable schema rules (VG101-VG104) for this model source.
  • queryEngines: which statement engines should include this source's model snapshot for query-schema rules (VG105-VG108).

How Rule Families Use Source Bindings

  1. VG001-VG008: run on parsed statements emitted by scanners.
  2. VG101-VG104: run on migration snapshot + models filtered by configEngines.
  3. VG105-VG108: run on migration snapshot plus model snapshots mapped by queryEngines.

Required Tests

  1. Scanner package unit tests.
  2. Model extractor unit tests (if extractor added).
  3. cmd/valk-guard/main_test.go integration tests:
    • source statements are detected
    • engine scoping via rules.<id>.engines works
    • VG105/VG106 include WHERE, INNER JOIN ... ON ..., and grouping/sort coverage
    • model-aware cases when extractor is wired

Docs to Update

  1. README.md engine/rule support matrix.
  2. docs/schema-drift.md source-to-model snapshot mapping.
  3. .valk-guard.yaml.example engine examples.

Quick Checklist

  1. Engine constant added.
  2. Engine listed in internal/scanner/engines.go.
  3. Scanner implemented and bound.
  4. Optional extractor implemented and model-bound.
  5. Tests added for scanner/query-schema/schema-drift behavior.
  6. Docs updated.
  7. Source-level config docs updated if the source can be disabled.
  8. go test ./... passes.