feat(parsers): accelerate new language support using ast-grep patterns

[vitali87]

Summary

Use ast-grep's pattern syntax to simplify adding new language support to CGR, reducing per-language implementation effort by creating a generic handler that uses YAML pattern configs instead of hand-written tree-sitter traversal code.

Motivation

CGR fully supports 7 languages but has 4 more "in development" (Go, Scala, C#, PHP) and 20+ languages that ast-grep supports but CGR does not (Ruby, Kotlin, Swift, Bash, C, etc.). Currently each language needs a LanguageSpec, LanguageHandler, and FQNSpec with hundreds of lines of tree-sitter traversal code.

ast-grep patterns like def $FUNC($$$ARGS): $$$BODY can replace complex tree-sitter query + traversal logic for basic extraction. A "basic language support" tier using ast-grep patterns could cover function/class/import extraction without a full handler.

Implementation

• codebase_rag/parsers/handlers/ast_grep_handler.py (~200 lines) implementing LanguageHandler protocol using ast-grep patterns instead of tree-sitter traversal
• Language pattern configs in YAML (one per language) defining function/class/import patterns
• Register new handler as fallback in registry.py for languages without specialized handlers
• Use this approach to finish Go, C#, PHP, Scala support
• Provide a template for community contributors to add new languages easily

Acceptance Criteria

• Generic AstGrepHandler implementing LanguageHandler protocol
• YAML-based pattern config format documented and validated
• At least one language (Go or C#) fully supported via this approach
• Handler registered as fallback in parser registry
• Extracted entities (functions, classes, imports) match quality of hand-written handlers for basic use cases
• Template/guide for adding new language support via YAML config
• Unit tests comparing ast-grep handler output vs existing handlers for supported languages

Related

Part of the ast-grep integration initiative:

• feat(tools): add structural pattern search via ast-grep #411 structural search tool
• feat(tools): add AST-aware code rewriting via ast-grep #412 structural rewrite tool
• feat(graph): enrich knowledge graph with pattern/smell/vulnerability detection via ast-grep rules #413 graph enrichment via ast-grep rules
• feat(tools): full ast-grep toolkit with structural search and rewrite #415 combined search + rewrite toolkit

[vitali87]

Triage: not resolved, keep open

Confirmed unimplemented against current code:

• No codebase_rag/parsers/handlers/ast_grep_handler.py and no generic YAML-pattern-based handler.
• Language support is still per-language hand-written tree-sitter traversal: each language is a bespoke class implementing the LanguageHandler protocol, registered in parsers/handlers/registry.py:16-25 (Python, JS/TS, C++, Rust, Java, Lua, PHP).
• No ast-grep-py dependency.

This is a genuine, distinct enhancement (a new parser tier), separate from the toolkit in #415 though it shares the same ast-grep-py dependency. Keeping open.