feat: extract access modifiers and decorators via highlights.scm queries

[vitali87]

Background

Tree-sitter's highlights.scm query files tag every token in source code with a semantic category — keywords, operators, literals, and crucially, access modifiers and decorators/attributes. Every tree-sitter language package ships highlights.scm with full coverage.

Problem

Currently only Java extracts access modifiers and annotations as node properties (via custom logic in parsers/java/utils.py). For the other 11 languages:

• We can't answer "what are the public methods of this class?"
• We can't filter by visibility (pub, private, protected, static)
• We can't find decorated functions ("show all @pytest.mark.parametrize tests", "find all #[test] functions")
• LLM queries about code visibility/access patterns produce incomplete answers

The information exists in the parse tree but we don't extract it.

Approach

Vendor highlights.scm files for all 12 languages (already available in all pip packages) and extract modifier/decorator captures as node properties.

What highlights.scm captures (relevant subset)

• @keyword / @keyword.modifier — public, private, protected, static, abstract, final, async, pub, pub(crate)
• @attribute / @function.decorator — @decorator (Python), #[attr] (Rust), [[attr]] (C++), @Annotation (Java)
• @type.qualifier — const, volatile, mutable, readonly

Example patterns:

; Python
(decorator "@" @attribute)

; Rust  
(attribute_item) @attribute

; Java
(modifiers) @keyword.modifier
(marker_annotation) @attribute

; TypeScript
(accessibility_modifier) @keyword.modifier

Implementation

1. Create codebase_rag/queries/highlights/ directory with .scm files per language (or load from pip packages directly since all 12 have them)
2. During node property building (in function_ingest.py and class_ingest/mixin.py), run highlights query on the node's source range
3. Collect @keyword.modifier and @attribute captures that fall within the node's declaration
4. Store as node properties: modifiers: list[str] and decorators: list[str]
5. Extend Java's existing MethodModifiersAndAnnotations pattern to all languages
6. Add tests per language verifying modifier and decorator extraction

Property schema additions

Function/Method nodes:
  modifiers: ["public", "static", "async"]    # access and behavior modifiers
  decorators: ["@property", "#[test]"]        # decorators/attributes

Class nodes:
  modifiers: ["abstract", "final", "pub"]
  decorators: ["@dataclass", "#[derive(Debug)]"]

Languages

All 12: Python, JavaScript, TypeScript, Rust, Java, C, C++, Go, Lua, Scala, PHP, C#

References

• tree-sitter highlight queries
• tree-sitter-al v2 rewrite — named nodes enabling query visibility
• Java's existing modifier extraction: codebase_rag/parsers/java/utils.py
• Issue feat: expand tree-sitter queries to capture scope, decorator, and type information #521 — parent tracking issue

[vitali87]

PR #566 (merged) landed the core of this: a shared extract_modifiers_and_decorators in parsers/utils.py (utils.py:160), loaded via parser_loader.py, populating modifiers: list[str] and decorators: list[str] on Function/Method/Class/Module nodes from vendored highlights.scm queries. The extraction call is generic (not language-gated); real coverage depends on whether curated @keyword.modifier/@function.decorator captures exist.

Fully wired (curated .scm + tests in test_modifiers_and_decorators.py):

• Python
• JavaScript
• TypeScript (+ TSX)
• Rust
• Java
• PHP

Not curated (only incidental capture from upstream HIGHLIGHTS_QUERY, no modifier list / decorator captures):

• C++
• Scala (private/protected/override/implicit not collected)
• C (only storage-class keywords via bare @keyword)
• Go (visibility is identifier casing, no modifier keywords)
• Lua (no access modifiers / decorators)

Note on the "12 languages": the SupportedLanguage enum is python, javascript, typescript, tsx, rust, go, scala, java, c, cpp, php, lua. C# is not supported (no tree_sitter_c_sharp package, no enum member), so the 12th slot is TSX, not C#.

Keeping this open to track curating highlights.scm + tests for the languages that genuinely carry the concept (C++, Scala). C/Go/Lua noted as not-applicable; C# is out of scope until the language itself is added.