bug(auto-update): Phase 0 extension filter excludes every non-source file from incremental analysis and eviction

[conradcoffman]

What happened?

hooks/auto-update-prompt.md Phase 0 step 7 filters the changed-file set down to a hardcoded 16-extension source allowlist before any other phase runs:

7. Filter to source files only (.ts, .tsx, .js, .jsx, .py, .go, .rs, .java, .rb, .cpp, .c, .h, .cs, .swift, .kt, .php).
If no source files changed: update meta.json with the new commit hash, report "Only non-source files changed. Metadata updated." and STOP.

The full path has no equivalent restriction. scan-project.mjs assigns fileCategory from {code, config, docs, infra, data, script, markup}, and a full build produces document / config / pipeline nodes for exactly the files step 7 discards. So the two paths disagree about what belongs in the graph, and a project kept fresh by auto-update alone diverges monotonically from what /understand would produce for the same tree.

Expected: a .md / .yml / .json file added after the last full build eventually appears in the graph, and one deleted eventually leaves it.

Actual: neither happens, indefinitely. Two distinct consequences:

1. Non-source files added after the last full build never enter the graph. Markdown, YAML, JSON, shell, SQL, CSS.
2. Non-source files deleted after the last full build are never evicted. Their nodes persist indefinitely.

(2) is worth calling out separately because it is not obvious from the wording: step 7's STOP is in Phase 0, while every deletion path is downstream of it. In v2.9.0's auto-update-prompt.md:

• L117 (Phase 1) -- For deleted files (in fingerprints.json but not on disk): classify as STRUCTURAL
• L191-192 (Phase 2) -- Remove old nodes whose filePath matches ... the deleted files list
• L221 (Phase 3b) -- For deleted files: remove their IDs from layer nodeIds arrays

A commit that only deletes a .md file therefore returns "Only non-source files changed" and stops before reaching any of them. Analysis and eviction are gated on the same filter, but only analysis has a reason to be.

Observed impact. Measured across seven repositories running auto-update in CI on a weekly-to-daily cadence. Every one had drifted; details anonymized.

• Repo A (1,049 files) -- analyzed the same day it was measured, yet missing 17 files: two observability dashboard/monitor JSON definitions, two container task-definition templates, two runtime config YAMLs, three operational shell scripts, and five docs/runbooks.
• Repo B (1,671 files) -- 9 missing (seven module docs, a skill definition, a CI workflow) and 5 ghost nodes.
• Ghost nodes survived 4+ weeks. One file deleted on 2026-07-06 was still present in the committed graph on 2026-08-04.

The pattern that made this expensive to notice: the graph looks healthy. Node counts grow, source coverage is complete, and nothing errors. Only comparing the graph's filePath set against git ls-files surfaces it. This is arguably worse for infrastructure-heavy repos than for docs, since task definitions, dashboards and config YAML are exactly the files a "how does this system work" query most needs, and exactly the ones step 7 drops.

---

Proposed fix

Primary: stop maintaining a second, narrower file-type list

agents/project-scanner.md is explicit that this categorization is owned by a script and must not be duplicated:

Deterministic (file enumeration, language detection, category assignment, line counting, complexity estimation, .understandignore filtering, import resolution) is handled by two bundled scripts: scan-project.mjs and extract-import-map.mjs. Do NOT re-implement any of this logic.

and:

You do not see or maintain those tables — they live in the script.

Step 7 is that re-implementation: a prose extension list, in a different file, narrower than the canonical table, with no mechanism keeping the two in sync.

Rather than widening the list (which preserves the drift hazard, just with a longer list), the incremental path should consult the same source of truth the full path uses. The pieces for this already exist:

• scan-result.json already carries what step 7 needs. Per skills/understand/SKILL.md, reading it yields "File list with line counts and fileCategory per file (code, config, docs, infra, data, script, markup)".
• It is already preserved specifically for this purpose. SKILL.md Phase 7 keeps it on cleanup: "Preserve scan-result.json — Phase 1's deterministic file inventory. Future incremental runs (Phase 2 compute-batches.mjs --changed-files=…) need this inventory."
• compute-batches.mjs already accepts a changed-files list and reads scan-result.json, invoked as --changed-files="$UA_DIR/tmp/changed-files.txt".

So the incremental path can decide inclusion by looking a changed path up in scan-result.json and keeping anything the scanner categorized, instead of pattern-matching extensions in prose. That inherits .understandignore handling and the category table for free, and cannot drift from the full path.

Secondary, independent: decouple eviction from the analyze filter

Even if the filter stays exactly as-is, a deleted file's node should be removed regardless of its extension. "We do not want to spend tokens analyzing markdown" and "a node may point at a file that no longer exists" are different questions, and today one answer is imposed on both. Moving the deleted-file handling ahead of the step 7 STOP, or exempting deletions from the filter, fixes (2) on its own.

Relationship to #626

#626 quotes the same sentence for a different symptom: after committing generated graph artifacts, the changed set filters to empty, the prompt rewrites meta.json, and that rewrite is itself another non-source change, looping.

They are separate bugs sharing one clause, and a fix aimed at either can miss the other:

• Widening the filter without addressing bug(auto-update): committed graph metadata enters a self-update loop #626 leaves the loop, and may worsen it, since generated artifacts are .json.
• Excluding generated paths (bug(auto-update): committed graph metadata enters a self-update loop #626's fix) without addressing this issue leaves real non-source files invisible.

Anything that touches step 7 should probably be evaluated against both. One data point that may be useful for #626: adding .understand-anything/ to .understandignore suppresses the loop in our deployment, because those paths are dropped before the changed-set check rather than after.

Possibly also related, not investigated: #310 (new files not recognized) and #341 (edges to non-code nodes dropped) -- the latter at least confirms non-code nodes are a first-class concept the graph is expected to carry.

Workaround we are running

For transparency, since it may be a useful signal on the shape of the fix. In CI we clone the plugin at a pinned tag and rewrite step 7's extension list in that ephemeral clone before invoking the agent, adding .md .yml .yaml .json .xml .sql .sh .css .scss .txt .html .toml. The substitution asserts it matched exactly once, so a future release that rewords the line fails the job loudly instead of silently reverting to the old behavior.

This works but is not something we would want to maintain: it is a blunt widening that hardcodes a second list, and it pins us to a plugin version until the patch is re-verified. A fix that removes the private list entirely would let us drop it.

Minimal reproduction

No special setup; the divergence is visible in two commits.

1. Run /understand on any project containing a README.md. Confirm a node exists with filePath: "README.md" (fileCategory: docs).
2. Add docs/new-page.md. Commit.
3. Trigger the auto-update flow. It reports "Only non-source files changed. Metadata updated." and stops. docs/new-page.md has no node.
4. Delete README.md. Commit. Trigger auto-update again. Same message, same stop. The README.md node is still in the graph, now pointing at a file that does not exist.

Both states persist until something forces a full rebuild.

Plugin version

v2.9.0 (latest tag as of 2026-08-07). Also verified against main at fe8c5bc5 -- step 7 is byte-identical at both refs, so this is not already fixed on an unreleased commit.

Platform / client

Claude Code (CLI), headless: claude --print --dangerously-skip-permissions --plugin-dir <clone>, driven from GitHub Actions.

OS + Node version

Ubuntu (GitHub Actions ubuntu-latest), Node v22.

Primary language of the analyzed project

Mixed across the seven repositories measured: TypeScript/JavaScript (Next.js), Python, and C#.

Approximate file count of the analyzed project

~120 to ~1,670 files per repository.

Relevant logs

# Graph vs git ls-files, one repo, immediately after a successful auto-update run.
# "ghost" = node whose filePath no longer exists; "missing" = tracked file with no node.

  ghost (in graph, deleted from repo):
    .github/pull_request_template.md
    .github/workflows/<removed-workflow>.yml
    REVIEW.md
    src/pages/api/<removed-route>.md
  missing (in repo, never analyzed):
    docs/docs/modules/<seven module docs>.md
    .agents/skills/<skill>/SKILL.md
    .github/workflows/<new-workflow>.yml
5 9

# Same repo after forcing a full rebuild, no other change:
0 0