Skip to content

The symbol indexer ingests __corpus__ byte fixtures the differential explicitly excludes #52

Description

@nedonatelli

What

src/agent/tools/__corpus__/ holds deliberately malformed byte fixtures — BOM
variants, CR/CRLF/mixed line endings, NBSP, trailing whitespace, no-final-newline
— used to test file-handling edge cases. They are not source.

The symbol indexer indexes them anyway: 13 files contributing 21 symbols
(17 variable, 4 function) to the live graph.

src/agent/tools/__corpus__/blank-lines.ts   {'variable': 2}
src/agent/tools/__corpus__/bom-crlf.ts      {'variable': 2}
src/agent/tools/__corpus__/cr.ts            {'function': 1}
src/agent/tools/__corpus__/mixed-eol.ts     {'variable': 3}
...

Why it matters more than 21 symbols suggests

scripts/graph-differential.mjs already names this directory in
EXPECTED_MISSES:

const EXPECTED_MISSES = [
  /\/__corpus__\//,   // deliberately malformed byte fixtures — not source
  /\.(md|json|txt)$/, // not code
];

So two components disagree about what counts as source: the differential treats
__corpus__ as legitimately un-indexable, while the indexer ingests it. The
check meant to detect extraction gaps is calibrated against a different file set
than the thing it checks — small here, but it is the differential's whole job to
be the independent view.

These symbols also compete at retrieval like any other, so asking about
line-ending handling can surface a fixture named cr.ts over the code that
handles it.

What to do

Exclude __corpus__ from indexing, ideally by making the indexer and the
differential read the same list rather than maintaining the judgement in two
places — the same convergence INDEX_EXCLUDE_DIRS did for the four scanners.

Worth checking for sibling fixture directories (__fixtures__, __snapshots__,
testdata/) while in there; the reason to exclude them is identical and only
__corpus__ happens to have been noticed.

How it was found

Reconciling the live symbol graph against a direct tree-sitter run after the
tree-sitter fixes (#47/#50/#51) landed. Every per-file count matched exactly;
the only aggregate differences were these 13 files and one file over the size
cap. No defect in extraction — the file sets differed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agentAgent loop, tool use, iterationsbugSomething isn't workingcontextWorkspace index, smart context, promptsready-for-agentFully specified, ready for an AFK agent

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions