[Bug]: Graphify refresh never records freshness because zero-node JSON is unclassified_unexpected

[MohabMohie]

Related to #5087 (unclassified buckets), #4687 (zero-node JSON is expected data-only), #4809 (nightly refresh owner). Does not reopen those.

✅ Pre-submission Checklist

• Searched existing issues. Graphify/MemPalace corpus policy: unclassified extract skips are coverage, not a failed install #5087 shipped unclassified_unexpected as fail-closed. Restore complete Graphify extraction coverage for SQL and zero-node sources #4687 said JSON with zero nodes is expected data-only. No open ticket covers the live 2026-08-19 failure: Graphify omits those JSON files from manifest.json, so SHAFT audit treats them as unexpected and never records the freshness marker.
• This is a SHAFT maintenance-controller defect, not an upstream Graphifyy install break. Graphify extract completed and wrote graph.json (38,685 nodes). Our audit then failed closed.
• Reproduced on primary main @ 03261bfc92 after py -3 tools/repository-map/graphify_maintenance.py refresh --root .

🐛 Describe the Bug

graphify_maintenance.py refresh deletes .shaft-source-revision.json, extracts successfully, then run_audit raises Graphify coverage audit found actionable parser gaps. Cluster and --record-current never run. resolve_graph_out.py --check stays stale - cache has no indexed revision marker.

The fail-closed bucket is unclassified_unexpected. On 2026-08-19 that bucket is 115 tracked files:

Count	Suffix	Meaning
111	.json	Graphify CODE type. Zero-node JSON is already policy expected_data_only when the path is in the manifest. Live extract leaves them out of the manifest, so they fall through to unexpected.
2	.css	Javadoc vendor CSS. Noise. Should be .graphifyignore.
2	.java	Real remaining gap: shaft-intellij/src/test/java/com/shaft/intellij/build/IntellijBuildJdkContractTest.java and IntellijRunIdeCrashLogContractTest.java are tracked, not ignored, and not in the manifest.

expected_data_only is 0 on this cache. unexpected_parser_gap is 0. covered is 2159 (every manifest path has a node). Extract also printed 105 source file(s) produced zero nodes and are absent from the graph (marketplace.json, settings.json, …) and skipped credentials.json as sensitive.

So the standing rule in chaos-engine/profiles/shaft/references/graphify.md — “JSON sources with no emitted nodes remain visible expected data-only inputs; SQL or other parser gaps stop the refresh before the marker” — is inverted in code. JSON that Graphify dropped from the manifest look like a new suffix and block freshness.

This is fixable entirely in SHAFT: tools/repository-map/graphify_maintenance.py, .graphifyignore, and tests/scripts/test_graphify_maintenance.py. Do not wait on a Graphifyy pin bump (#5120 is closed/upstream).

🔁 Steps to Reproduce

1. On the primary checkout at current origin/main, run py -3 tools/repository-map/graphify_maintenance.py refresh --root .
2. Extract finishes and writes graphify-out/graph.json.
3. Audit prints a large unclassified_unexpected list dominated by .json.
4. Refresh exits 1: Graphify coverage audit found actionable parser gaps.
5. py -3 tools/repository-map/resolve_graph_out.py --check reports no indexed revision marker.
6. Re-run py -3 tools/repository-map/graphify_maintenance.py audit --root . and count suffixes (2026-08-19: 111 json / 2 css / 2 java).

✅ Expected Behavior

• Tracked .json that Graphify did not put in the manifest (zero-node or sensitive skip) classify as expected_data_only. They stay visible. They do not fail the audit.
• Javadoc .css is ignored by policy.
• Only a new unexpected family, unexpected_parser_gap, or a tracked first-class source Graphify should have ingested (the two .java files until explained) fails the audit.
• After a green audit, refresh clusters and records .shaft-source-revision.json for 03261bfc92 (or whatever HEAD was).
• test_88_json_data_files_are_visible_but_nonfatal still passes, and a new test covers JSON absent from the manifest.

❌ Actual Behavior

expected_data_only=0
unclassified_unexpected=115   # 111 .json + 2 .css + 2 .java

Refresh unlinks the marker first (refresh()), then dies in run_audit, so a successful extract still leaves the cache officially stale.

💻 Minimal Reproducible Code

Current classification hole (tools/repository-map/graphify_maintenance.py):

# Manifest JSON with zero nodes → expected_data_only (nonfatal)
elif Path(path).suffix.lower() == ".json":
    classification = "expected_data_only"

# Tracked JSON not in the manifest → unclassified_unexpected (fatal)
elif unclassified_allowlisted(path):
    result["unclassified_allowlisted"].append(path)
else:
    result["unclassified_unexpected"].append(path)

UNCLASSIFIED_ALLOWLIST_SUFFIXES has .properties / .toml / .xml / .feature / … and not .json, because JSON is supposed to be a Graphify code type.

🌍 Environment

Field	Value
SHAFT_ENGINE version	main 03261bfc92
Graphify	pinned graphifyy via uv tool run in graphify_maintenance.py
OS	Windows (primary checkout)
Command	py -3 tools/repository-map/graphify_maintenance.py refresh --root .

📋 Console Logs

Extract (2026-08-19): 244 code files AST-extracted; 219 unclassified skips (Dockerfile / .feature / … — those correctly land in unclassified_allowlisted = 251); 1 sensitive skip credentials.json; 105 zero-node sources absent from the graph.

Audit: covered=2159, expected_data_only=0, unclassified_unexpected=115.

Check: stale - cache has no indexed revision marker.

User Scenarios & Testing

User Story 1 - JSON omitted from the manifest does not block freshness (Priority: P1)

Independent Test: Fixture with a tracked data/config.json that is not a manifest key. Audit exit 0; path in expected_data_only.

Acceptance Scenarios:

1. Given a tracked .json file Graphify left out of manifest.json, When audit runs, Then it is expected_data_only and the process exits 0 if nothing else is unexpected.
2. Given the same file in the manifest with no graph node, When audit runs, Then it remains expected_data_only (existing test).

User Story 2 - Refresh records the marker on current main (Priority: P1)

Independent Test: After the classification fix (and CSS ignore), one primary refresh --root . at current origin/main. resolve_graph_out.py --check prints the HEAD SHA.

Acceptance Scenarios:

1. Given only the 111 JSON + 2 CSS leftovers from this ticket, When refresh runs, Then audit is green and .shaft-source-revision.json matches git rev-parse HEAD.
2. Given a new tracked suffix that is not allowlisted and not .json, When audit runs, Then unclassified_unexpected still fails (existing new.weirdsuffix test).

User Story 3 - The two missing Java tests stay fail-closed until explained (Priority: P2)

Independent Test: Audit still lists those two paths unless they enter the manifest or a written ignore/allowlist with a comment.

Acceptance Scenarios:

1. Given IntellijBuildJdkContractTest.java and IntellijRunIdeCrashLogContractTest.java still absent from the manifest, When JSON/CSS are reclassified, Then they remain unclassified_unexpected (or unexpected_parser_gap if they appear in the manifest with zero nodes) until a follow-up proves Graphify should skip them.
2. Given a later extract that covers them, When audit runs, Then they move to covered and no longer fail.

Edge Cases

• credentials.json is Graphify-sensitive. Classifying it as expected_data_only is correct. Do not copy its contents into the graph or into Memory.
• Hidden paths (.claude/settings.json, .github/issue-taxonomy.json) are tracked; they must use the same JSON rule.
• package-lock.json and doctor/capture fixtures are data. Do not invent a second JSON family without a test.
• Do not add *.json to .graphifyignore. That would hide product JSON we may want if Graphify later emits nodes.
• Do not treat .java as expected data-only.
• Do not record the freshness marker when unexpected_parser_gap or a true unexpected suffix remains.
• refresh() already deletes the marker at start. A failed audit must leave it absent (current). A green audit must recreate it.

Functional Requirements

• FR-001: Tracked .json not present in manifest.json classify as expected_data_only.
• FR-002: Add the two javadoc *.css paths (or **/*.css if no product CSS should be graphed) to .graphifyignore.
• FR-003: Keep fail-closed unclassified_unexpected for new suffixes and for the two IntelliJ Java tests until they are covered or explicitly justified.
• FR-004: Tests in tests/scripts/test_graphify_maintenance.py: JSON-absent-from-manifest is nonfatal; CSS ignore; weird suffix still fatal; existing 88-JSON-in-manifest test still passes.
• FR-005: One primary refresh after the fix; resolve_graph_out.py --check current. Nightly Add safe nightly Graphify and MemPalace refresh #4809 uses the same controller.
• FR-006: Keep taxonomy labels valid (one primary, one lifecycle, at least one subsystem/module).

Success Criteria

• SC-001: audit on today’s cache reports expected_data_only >= 111 and unclassified_unexpected is only the two Java files (or empty if those get a justified disposition in the same PR).
• SC-002: refresh --root . records .shaft-source-revision.json when the only leftovers are expected JSON (and ignored CSS).
• SC-003: Removing the JSON-absent-from-manifest rule makes a focused unit test fail.

Assumptions

• Graphifyy 0.9.43 still treats .json as CODE and drops zero-node JSON from the graph (and currently from the manifest). SHAFT will not wait for an upstream manifest change to record freshness.
• Nightly MemPalace mine is out of scope (Promote SHAFT properties, SPI, Docker, and plugin XML into MemPalace #5119 already shipped force-include for other families).

Out of scope

• Upstream Graphifyy CODE_EXTENSIONS (Upstream Graphifyy CODE_EXTENSIONS for SHAFT unclassified families #5120).
• Promoting .properties / SPI / Docker into the graph (allowlisted; MemPalace Promote SHAFT properties, SPI, Docker, and plugin XML into MemPalace #5119).
• Committing graphify-out/.
• Ordinary tasks refreshing the cache.

📝 Additional Context

Policy already written, not implemented for the not-in-manifest case:

JSON sources with no emitted nodes remain visible expected data-only inputs; SQL or other parser gaps stop the refresh before the marker is recorded.

tests/scripts/test_graphify_maintenance.py test_88_json_data_files_are_visible_but_nonfatal only covers JSON keys in the manifest.