chore: clear untracked worktree scratch (#579)#608
Merged
Conversation
Resolve the ~30 untracked working-tree entries accumulated over branch
work, each by inspection rather than blanket action.
Commit (real artifacts):
- NEXT_TASKS.md, docs/MERGE_CLEANUP.md, docs/metpo/* release-diff docs
- docs/metatraits/unmapped_compounds.tsv, mappings/mediadive_unmapped_ingredients_to_curate.tsv (curation queues)
- bacdive/metabolite_mapping.json (consolidate_chemical_mappings.py input, not read by bacdive.py)
- ontologies/xrefs/mondo_xrefs.tsv (wired via constants.MONDO_XREFS_FILEPATH) + unipathways_xrefs.tsv
- .claude/skills/analyze-kg-tsne/SKILL.md
Gitignore (generated / heavy, must never be tracked):
- data/transformed/ (regenerated by `kg transform`)
- notebooks/ (8.6 GB local analysis)
- data/raw/lpsn/* (134 MB auth-derived download cache)
Delete (confirmed dead/stray/duplicate scratch, all untracked):
- utils/{transform_utils,nlp_utils,biohub_converter,diagnose_duplicates,extract_taxon_strain_nodes}.py
(0 call sites; biohub only used by dead nlp_utils; utils/transform_utils.py is a
nominal-only name clash with the transform_utils/ package, never imported)
- utils/NCBITaxon_rank.tsv (generated by tracked parse_taxon_rank.py, no readers)
- transform_utils/ontology/ (stray chebi_xrefs diff scratch; real dir is ontologies/)
- transform_utils/data/ (transform output misplaced under the package tree)
- transform_utils/bactotraits/BactoTraits.tsv (stray; transform reads data/raw/BactoTraits_databaseV2_Jun2022.csv)
- root one-offs: run.py, analyze_categories.py, download_mediadive_bulk.py
(dup of utils/mediadive_bulk_download.py), convert_merged_to_nt.yaml, load_feba.ipynb
(byte-dup of notebooks/load_feba.ipynb)
Verified: `import kg_microbe` + transform imports clean; no dangling refs to
deleted modules. Deleted files backed up to session scratchpad.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses issue #579 by turning previously-untracked scratch artifacts into tracked documentation/curation inputs, and by adding .gitignore rules to prevent large or regenerable outputs from being accidentally committed in the future.
Changes:
- Add documentation + triage artifacts (
NEXT_TASKS.md,docs/MERGE_CLEANUP.md, METPO release-diff docs) to preserve decisions and explain merge normalization behavior. - Add curation/mapping inputs used by existing tooling (
mappings/mediadive_unmapped_ingredients_to_curate.tsv,kg_microbe/transform_utils/bacdive/metabolite_mapping.json). - Expand
.gitignoreto exclude regenerable transform outputs and large local analysis directories (data/transformed/,notebooks/,data/raw/lpsn/*).
Reviewed changes
Copilot reviewed 9 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
NEXT_TASKS.md |
Snapshot/triage list for previously untracked worktree entries (needs a small status update to match PR intent). |
mappings/mediadive_unmapped_ingredients_to_curate.tsv |
MediaDive unmapped-ingredient curation queue TSV (matches the generator schema in scripts/dump_unmapped_mediadive_ingredients.py). |
kg_microbe/transform_utils/bacdive/metabolite_mapping.json |
BacDive metabolite mapping JSON referenced by consolidation tooling/constants. |
docs/metpo/metpo_2026_06_02_release_diff.md |
Narrative report comparing METPO release vs proposal TSVs. |
docs/metpo/metpo_2026_06_02_present.tsv |
TSV listing proposed IDs present in release with diffs. |
docs/metpo/metpo_2026_06_02_missing.tsv |
TSV listing proposed IDs absent from the release. |
docs/metatraits/unmapped_compounds.tsv |
Snapshot list of unmapped compound placeholders for metatraits-related curation. |
docs/MERGE_CLEANUP.md |
Documentation of post-merge TSV normalization behavior and log interpretation. |
.gitignore |
Adds ignore rules for data/transformed/, notebooks/, and data/raw/lpsn/*. |
.claude/skills/analyze-kg-tsne/SKILL.md |
Adds skill documentation for analyzing DeepWalk → tSNE/UMAP visualizations. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+8
to
+16
| > **Last reconciled: 2026-07-21.** Status: **all items below still PENDING** | ||
| > — none of these untracked files were resolved (36 untracked entries remain | ||
| > in the worktree). Tracked as issue #579. The 2026-07-20/21 work threads | ||
| > (LPSN download+transform+merge, chemical-mapping retraction, kgxval review | ||
| > integration, ontology metamodel-edge drop, GO `go.db` rebuild, `ontologies_stubs` | ||
| > regen) were all shipped in PRs #596–#603, #605 and are unrelated to this | ||
| > cleanup backlog. New deferred follow-ups from that work live as GitHub issues: | ||
| > **#599** (harden apply_retractions), **#600** (merge lpsn_api credential | ||
| > coupling), **#604** (ontology .owl/.json/.db version alignment). |
Address review findings on #608: - MEDIUM-HIGH (fixed): the blanket `notebooks/` ignore shadowed 18 already-tracked files (README, schema.*, pivot-kg-edges/, several .ipynb) and would silently swallow any future notebook. The real multi-GB scratch is notebooks/feba.db (6.7 GB, already caught by `notebooks/*.db`) and a committed notebooks/venv/ (1.6 GB) — NOT the notebooks. Replace the blanket rule with targeted ignores (venv/, .ipynb_checkpoints/) and commit the 10 real analysis notebooks that were being hidden. bacdive_reformat.ipynb (67 MB of embedded outputs) is ignored specifically with a note to nbstripout-then-commit if wanted. - MEDIUM (kept, by design): mondo_xrefs.tsv (3.8 MB) / unipathways_xrefs.tsv are transform-generated, but a sibling generated xref (mondo_gene_ids.tsv) is already tracked in the same dir, and ctd / uniprot_* transforms read all three via `.exists()` guards — committing them deliberately decouples those from a prior ontologies-transform run. Regenerable; documented here. - LOW (kept): bacdive/metabolite_mapping.json is a 6.6 KB regenerable cache but a live priority-1 input to consolidate_chemical_mappings.py — correct to track. All deletions independently re-verified safe (no tracked code references them). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves the #579 backlog: ~30 untracked working-tree entries, each triaged by inspection.
Commit (real artifacts)
NEXT_TASKS.md,docs/MERGE_CLEANUP.md,docs/metpo/*release-diff docsdocs/metatraits/unmapped_compounds.tsv,mappings/mediadive_unmapped_ingredients_to_curate.tsv(curation queues, referenced by the kg-model-review skill)bacdive/metabolite_mapping.json— input toconsolidate_chemical_mappings.py(thebacdive.py"read" was only a stale comment)ontologies/xrefs/mondo_xrefs.tsv(wired viaconstants.MONDO_XREFS_FILEPATH) +unipathways_xrefs.tsv.claude/skills/analyze-kg-tsne/SKILL.mdGitignore (generated / heavy — must never be tracked)
data/transformed/(regenerated bykg transform) — was a gitignore gapnotebooks/(8.6 GB local analysis)data/raw/lpsn/*(134 MB auth-derived download cache)Delete (confirmed dead / stray / duplicate scratch, all untracked)
utils/{transform_utils,nlp_utils,biohub_converter,diagnose_duplicates,extract_taxon_strain_nodes}.py— 0 call sites;biohub_converteronly used by deadnlp_utils;utils/transform_utils.pyis a nominal-only clash with thetransform_utils/package (kg_microbe.utils.*vskg_microbe.transform_utils— no real shadowing), never importedutils/NCBITaxon_rank.tsv— generated by trackedparse_taxon_rank.py, no readerstransform_utils/ontology/— straychebi_xrefsdiff scratch (real dir isontologies/)transform_utils/data/— transform output misplaced under the package treetransform_utils/bactotraits/BactoTraits.tsv— stray; transform readsdata/raw/BactoTraits_databaseV2_Jun2022.csvrun.py,analyze_categories.py,download_mediadive_bulk.py(dup ofutils/mediadive_bulk_download.py),convert_merged_to_nt.yaml,load_feba.ipynb(byte-dup ofnotebooks/load_feba.ipynb)Verification
import kg_microbe+ transform imports clean; no dangling refs to deleted modulesruff check kg_microbe/ tests/passes; 417 tests collect with no import errors🤖 Generated with Claude Code