fix(pipeline): make run_sift.py invocable by path; skip infra files in entity resolution - #40
Open
chrisdavidson wants to merge 3 commits into
Open
Conversation
run_sift.py:21 imports core.domain_resolver at module scope. Running a
script by path puts core/ on sys.path[0], not the project root, so the
documented invocation in commands/ingest.md failed with:
ModuleNotFoundError: No module named 'core'
The file already inserted the project root, but at line 240 inside
cmd_dashboard() for examples.workbench.* imports — far too late for a
module-level import. It only worked with an explicit PYTHONPATH.
Hoist the bootstrap above the import, matching the idiom already used in
core/build_extraction.py and core/normalize_extractions.py.
preprocess_extractions() globbed extractions/*.json unfiltered, so it also
read _normalization_report.json. The report has no entities key so nothing
was mutated, but it was parsed on every run and inflated files_processed
by one per infrastructure file.
core/normalize_extractions.py already solved this for the same directory
with _SKIP_PREFIXES = ("_",); entity_resolution.py just did not follow the
convention. Apply the same guard.
Both tests fail without the accompanying fixes and pass with them. - run_sift.py is invoked by absolute path in a subprocess with PYTHONPATH scrubbed and cwd outside the repo, so a stray '' on sys.path cannot mask the regression. - preprocess_extractions() is given one real extraction plus a _normalization_report.json and must count only the former, leaving the report byte-identical.
usathyan
approved these changes
Aug 2, 2026
usathyan
left a comment
Owner
There was a problem hiding this comment.
fix merge conflicts and merge away!
usathyan
approved these changes
Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two defects found while running
/epistract:ingestagainst an FDA product label corpus (Ozempic MAR2022 SPL). Both are invocation/hygiene bugs rather than extraction bugs.1.
run_sift.pycannot be invoked by pathcommands/ingest.mddocuments invoking this script by absolute path. That fails:run_sift.py:21importscore.domain_resolverat module scope, but running a script by path putscore/onsys.path[0]— not the project root. The file already inserts the project root, at line 240 insidecmd_dashboard()forexamples.workbench.*imports; that is far too late for an import that runs at module load. Until now it only worked with an explicitPYTHONPATH=<repo root>.Fixed by hoisting the bootstrap above the import, using the idiom already established in
core/build_extraction.py:18-22andcore/normalize_extractions.py:43-45.2.
entity_resolution.pyparses its own infrastructure filespreprocess_extractions()globsextractions/*.jsonunfiltered, so it also reads_normalization_report.json. The report has noentitieskey, so nothing is mutated — but it is parsed on every run and inflates the returnedfiles_processedcount by one per infrastructure file.core/normalize_extractions.py:63,167-170already solved this for the same directory with_SKIP_PREFIXES = ("_",).entity_resolution.pysimply did not follow the convention. Applied the same guard.Related, and deliberately not fixed here
Every graph build logs:
This originates upstream, not in epistract —
sift_kg/graph/builder.py:308-313, whereload_extractions()globs*.jsonand callsDocumentExtraction(**raw)on every match, warning on failure. It cannot be patched from this repo.The only epistract-side lever would be to stop writing the report into
extractions/, but that path is contractual — documented twice incommands/ingest.md, asserted by UT-023 (tests/test_unit.py) andtests/test_e2e.py(x2), and specified intests/TEST_REQUIREMENTS.md. That is a breaking change and out of scope here.Change 2 therefore does not silence that warning. Suggested follow-up: an upstream issue against sift-kg to apply the same underscore-skip convention in
load_extractions().Tests
Two regression tests in
tests/test_unit.py, both verified failing before the fixes and passing after:test_run_sift_importable_by_path_without_pythonpath— subprocess invocation by absolute path withPYTHONPATHscrubbed and cwd outside the repo, so a stray''onsys.pathcannot mask the regression.test_preprocess_extractions_skips_underscore_infra_files— one real extraction plus a report; asserts only the former is counted and the report is left byte-identical.Verification
The one failure,
tests/test_kg_provenance.py::TestPCCVenueProvenance::test_akka_party_referenced_in_response, fails identically onorigin/main— pre-existing and unrelated to this PR. For what it is worth, the assertion looks self-contradictory: it requires the string"akka"while its failure message reads "Response does not mention STA", and the fixture response discusses STA's obligations without ever containing "akka". Worth a separate look.Test plan
pytest tests/ -m unit— 184 passed (2 new), 1 pre-existing unrelated failureruff check— no new findings introducedpython3 core/run_sift.py --list-domainsfrom a cwd outside the repo with noPYTHONPATH