Skip to content

ExtractionMode (structural/semantic) toggle for Python AST extraction #5

Description

@pedroanisio

Context

codebase_mapper/languages/python.py::_ast_to_jsonable serializes the entire ast.AST tree unconditionally — including function bodies. For users who only need shapes (call graphs, signatures, imports) and not bodies, this produces unnecessarily large cbm:astSummary triples.

The code-mapper bundle (analyzed in this session) carries this pattern explicitly: ExtractionMode = Literal["structural", "semantic"], with _StructuralCollector and _SemanticCollector as parallel AST visitors. Structural mode drops function bodies; semantic mode keeps them.

Per README.md's ### Size cost section, AST-summary is currently ~6.6× source size for Python — bodies are the bulk.

Scope

Add a mode toggle to codebase_mapper/languages/python.py:

  1. New ExtractionMode = Literal["structural", "semantic"] (default "semantic" to preserve current behavior).
  2. In _ast_to_jsonable, when mode == "structural" and the node is ast.FunctionDef / ast.AsyncFunctionDef / ast.Lambda, omit .body (preserve args, returns, decorator_list, docstring).
  3. Wire the toggle through pipeline.pyemit_bundle.py so run_manifest.json records which mode produced the bundle.
  4. Surface via CLI: --ast-mode {structural,semantic}.

Subtasks

  • Add ExtractionMode type alias + mode: ExtractionMode = "semantic" parameter on _ast_to_jsonable
  • Pipeline / CLI plumbing (--ast-mode flag, manifest field)
  • Update regenerate.py to refuse structural-mode bundles for Python regeneration (bodies required for that path) — emit a clear error
  • Add tests/verify_ast_mode.py: same fixture, both modes, assert (a) structural is smaller, (b) structural still re-parses to valid AST (sans bodies), (c) regenerate errors cleanly on structural-mode bundles
  • Update docs/regenerate.md to document the structural/semantic distinction and the regenerate restriction
  • Update README.md ### Size cost section with measured ratio for structural mode

Acceptance

After this lands:

  • python -m codebase_mapper --repo /path --out /tmp/out --ast-mode structural produces a bundle whose ast_summary_total_bytes is materially smaller (target: <2× source).
  • regenerate over a structural bundle errors with a clear "structural-mode bundles cannot be regenerated" message.
  • Semantic mode is unchanged (backward-compatible default).

Out of scope

  • TS/JS structural mode. The leaf-text CST extractor is byte-perfect by design; trimming it would break the regenerate guarantee. Address separately if a need emerges.

Borrowed pattern, not code

We are reimplementing the structural/semantic split idiomatically in our codebase. We are not vendoring code-mapper's collectors. License-clean.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions