chore: clear untracked worktree scratch (#579)

.claude/skills/analyze-kg-tsne/SKILL.md

@@ -0,0 +1,104 @@
+---
+name: analyze-kg-tsne
+description: Produce interpretation and parameter-optimization reports for a KG-Microbe DeepWalk -> tSNE/UMAP edge/node visualization. Use when the user invokes /analyze-kg-tsne or asks to interpret/optimize a KG tSNE figure.
+---
+
+# Analyze KG tSNE Skill
+
+Produce an interpretation report and parameter-optimization report for a
+KG-Microbe tSNE edge/node visualization (DeepWalk embedding → tSNE/UMAP).
+
+## Usage
+
+When the user invokes `/analyze-kg-tsne`, run the analysis workflow.
+
+Arguments (free text): one or more of
+- path to the tSNE PNG/SVG image
+- path to the merged KG directory (containing `merged-kg_edges.tsv`, `merged-kg_nodes.tsv`)
+- path to the DeepWalk training script (e.g., `kg_microbe_DeepWalk_*.py`)
+- path to the tSNE SLURM script (e.g., `src/tSNE_edges_only_*.sl`)
+
+If any of these are missing, ask the user for them before proceeding.
+
+## Instructions
+
+1. **Gather evidence** (run in parallel when independent):
+   - `Read` the tSNE image. Note the title, axis ranges, legend order, cluster count, and visually dominant regions.
+   - `Read` the tSNE SLURM script. Record `--method`, `--sample-size`,
+     `--edge-method`, `--perplexity`, `--n-iter`, `--learning-rate`,
+     `--max-legend-types`, and the resource request (`--mem`, `--cpus`, `--time`).
+   - `Read` the DeepWalk Python script. Record `embedding_size`, `walk_length`,
+     `iterations`, `window_size`, `epochs`, `normalize_by_degree`,
+     and the path to nodes/edges TSVs.
+   - In the KG dir, run (via `Bash` with `cut | sort | uniq -c | sort -rn`):
+     * edge counts per predicate (top 20)
+     * node counts per `category` (top 15)
+     * total edge and node counts (`wc -l`)
+   - Cross-reference predicate CURIEs against `PREDICATE_LABELS` in
+     `src/kg_microbe_pipeline_utils.py` (Grep for `PREDICATE_LABELS`).
+
+2. **Synthesize interpretation** — write
+   `<kg_dir>/tsne_interpretation_report.md` with sections:
+   - Graph composition table (nodes by category, top predicates by edge count).
+   - "What the tSNE is actually showing" — explain the edge-embedding operator
+     (Hadamard / concatenate / average / l1 / l2) and what proximity means.
+   - "Structures visible in the figure" — describe each visible cluster/lobe
+     and hypothesize which predicate(s) drive it, cross-referencing predicate
+     counts. Flag positive/negative METPO twin clusters if present.
+   - "What the plot says about the KG itself" — 5-8 numbered observations
+     about balance, curation artefacts, node-type dominance, hierarchy signal,
+     chemistry vs phenotype weight.
+   - "Limitations of this view" — predicate conflation by the chosen operator,
+     direction loss (if symmetric), 2D compression, density vs cardinality.
+   - "Quick verifications worth running next" — concrete follow-up plots.
+
+3. **Synthesize optimization recommendations** — write
+   `<kg_dir>/tsne_parameter_optimizations.md` with:
+   - Section A: DeepWalk parameters. For each suggestion include
+     current value, proposed value, expected improvement (⭐–⭐⭐⭐),
+     runtime cost, and one-sentence rationale. Known high-value knobs:
+     `window_size`, `iterations` (walks/node), `walk_length`, `embedding_size`,
+     `epochs`, `normalize_by_degree`, Node2Vec `p`/`q` biases.
+     End with a "recommended recipe" code block.
+   - Section B: tSNE parameters. Known high-value knobs:
+     `learning_rate` (use `N/12` auto rule for N>100K — frequently the single
+     biggest win), `perplexity` (try multiscale 50+500 for large N),
+     `n_iter` + early-exaggeration schedule, method swap to `cuml_tsne`/
+     `tsnecuda` for GPU, `edge-method` (hadamard vs concatenate),
+     `sample-size`, `max-legend-types`, plotting `alpha`/`point-size`,
+     coord persistence for re-plotting.
+   - Section C: priority ordering (do these N first).
+   - Section D: evaluation protocol — explicit downstream metric
+     (e.g., balanced accuracy on a held-out binary medium classifier at
+     `src/kg_microbe_train_binary_medium__pipeline.py 514 --data EC_RHEA`)
+     and a tSNE-quality score (k-NN predicate purity, 2D vs N-D).
+
+4. **Report** — summarize for the user in ≤5 lines:
+   - Path to each generated report
+   - Top 3 priority changes (usually: learning-rate auto, walks-per-node bump, GPU method swap)
+   - Any blockers noticed (e.g., script missing arg, path mismatch)
+
+## Conventions and gotchas
+
+- Always cross-reference the legend labels against `PREDICATE_LABELS` before
+  naming predicates — METPO CURIEs are remapped to readable names.
+- Hadamard is symmetric: do **not** claim direction is preserved.
+- When citing cluster sizes, prefer "edges of predicate X (count N)" over
+  "this cluster has N points" because α-overdraw distorts perceived counts.
+- Do not claim a cluster "is" a predicate from colour alone — frame as
+  "consistent with" unless a coord-level dump was inspected.
+- Report DeepWalk defaults explicitly when the script relies on them
+  (Grape `DeepWalkSkipGramEnsmallen` defaults: `walk_length=128`,
+  `iterations=10`, `window_size=5`, `epochs=30`, `learning_rate=0.01`,
+  `number_of_negative_samples=5`).
+- For openTSNE/FIt-SNE, `learning_rate="auto"` resolves to `max(200, N/12)`.
+- tSNE "sample + transform" mode smears rare predicates; point this out
+  when interpreting small peripheral clusters.
+
+## Output conventions
+
+- Both reports go in the KG merge directory (same dir as the PNG and TSVs).
+- Use absolute paths in report headers.
+- Tables for composition, bulleted ⭐ priority scores for recommendations.
+- Include one "recommended recipe" code block per script at the end of
+  each section.

.gitignore

@@ -30443,3 +30443,18 @@ notebooks/pred*_merged_output.tsv
 
 # Transform-internal tmp/ subdirectories (never authored, always cached)
 kg_microbe/transform_utils/*/tmp/
+
+# Transform outputs (regenerated by `kg transform`)
+data/transformed/
+
+# Notebook scratch not already covered above: a 1.6 GB committed virtualenv and
+# Jupyter checkpoints. (The 6.7 GB feba.db is caught by `notebooks/*.db`.) Do NOT
+# blanket-ignore `notebooks/` — 18 real files live there (README, schema.*,
+# pivot-kg-edges/, several .ipynb) and a blanket rule would shadow them and
+# silently swallow future notebook additions.
+notebooks/venv/
+notebooks/.ipynb_checkpoints/
+# 67 MB of embedded cell outputs — not source. Strip with nbstripout and commit
+# if the analysis is worth keeping; ignored as-is to avoid a permanent blob.
+notebooks/bacdive_reformat.ipynb
+data/raw/lpsn/*

NEXT_TASKS.md

@@ -0,0 +1,105 @@
+# NEXT_TASKS.md — untracked entries that need a per-file decision
+
+Snapshot of the "needs your call" pile from the branch triage on
+`chore/refresh-unified-chemical-mappings` (working-tree state as of
+2026-07-02). Each entry is either real work that should be committed
+(and to which branch), or scratch that should be gitignored / deleted.
+
+> **Last reconciled: 2026-07-21.** Status: **all items below still PENDING**
+> — none of these untracked files were resolved (36 untracked entries remain
+> in the worktree). Tracked as issue #579. The 2026-07-20/21 work threads
+> (LPSN download+transform+merge, chemical-mapping retraction, kgxval review
+> integration, ontology metamodel-edge drop, GO `go.db` rebuild, `ontologies_stubs`
+> regen) were all shipped in PRs #596–#603, #605 and are unrelated to this
+> cleanup backlog. New deferred follow-ups from that work live as GitHub issues:
+> **#599** (harden apply_retractions), **#600** (merge lpsn_api credential
+> coupling), **#604** (ontology .owl/.json/.db version alignment).
+
+Legend:
+- **commit** — keep and stage on some branch (which one to decide per row)
+- **ignore** — add to `.gitignore`, leave on disk
+- **delete** — remove from working tree
+- **?** — needs inspection before deciding
+
+## New utility modules
+
+Nothing in the branch topic (mappings refresh) references any of these,
+so they're either dead scratch or belong to a different feature branch.
+Open each and decide.
+
+- [ ] `kg_microbe/utils/biohub_converter.py` — ?
+- [ ] `kg_microbe/utils/diagnose_duplicates.py` — ?
+- [ ] `kg_microbe/utils/extract_taxon_strain_nodes.py` — ?
+- [ ] `kg_microbe/utils/nlp_utils.py` — ?
+- [ ] `kg_microbe/utils/parse_taxon_rank.py` — ? (there's also a `.py_bak` sibling → delete the bak regardless)
+- [ ] `kg_microbe/utils/transform_utils.py` — ? (name collides with the top-level `transform_utils/` package; check for shadowing bugs)
+
+## Scripts at repo root
+
+Root-level scripts are almost always one-offs. If any are permanent,
+move under `kg_microbe/` or `scripts/`.
+
+- [ ] `run.py` — ? (suspicious — `poetry run kg` is the canonical entry point)
+- [ ] `analyze_categories.py` — ?
+- [ ] `download_mediadive_bulk.py` — likely belongs alongside `download.yaml` MediaDive work; if permanent, move under `kg_microbe/`
+- [ ] `convert_merged_to_nt.yaml` — one-off config; if permanent, move under a config dir
+
+## Unknown directories
+
+Peek inside before deciding.
+
+- [ ] `kg_microbe/transform_utils/data/` — ?
+- [ ] `kg_microbe/transform_utils/ontology/` — ? (possibly a typo of the existing `ontologies/` package)
+
+## Data / mapping / xref files
+
+- [ ] `kg_microbe/transform_utils/bacdive/metabolite_mapping.json` — ?
+- [ ] `kg_microbe/transform_utils/bactotraits/BactoTraits.tsv` — per the `download.yaml` note, BactoTraits V2 is now vendored because the upstream TLS is broken. **This file probably needs to be committed** together with the `download.yaml` change on its own PR.
+- [ ] `kg_microbe/transform_utils/ontologies/xrefs/mondo_xrefs.tsv` — ?
+- [ ] `kg_microbe/transform_utils/ontologies/xrefs/unipathways_xrefs.tsv` — ?
+
+## Docs
+
+- [ ] `docs/MERGE_CLEANUP.md` — likely real; belongs on a docs / merge branch
+- [ ] `docs/metatraits/unmapped_compounds.tsv` — ?
+- [ ] `docs/metpo/` — likely belongs with the METPO commits (`b2b6c57f`, `a237751b`) that are currently misfiled on this branch
+
+## Mappings
+
+- [ ] `mappings/mediadive_unmapped_ingredients_to_curate.tsv` — probably **THIS branch** (matches mappings-refresh topic). Confirm and stage.
+
+## Notebooks
+
+Rule of thumb: notebooks with named outputs and a clear analytical
+purpose can be committed; anything with `_bak`, `_withoutput`,
+`catboost_info/`, or that's obviously scratch should be ignored or
+deleted.
+
+- [ ] `notebooks/bacdive-api-summary-table.ipynb` — ?
+- [ ] `notebooks/bacdive-process-errors-with-genus.ipynb` — ?
+- [ ] `notebooks/bacdive-summary-counts.ipynb` — ?
+- [ ] `notebooks/bacdive_mapping_resource.ipynb_bak` — **delete** (`_bak`)
+- [ ] `notebooks/bacdive_reformat.ipynb` — ?
+- [ ] `notebooks/evaluate_link_pred.ipynb` — ?
+- [ ] `notebooks/feba.ipynb_bak` — **delete** (`_bak`)
+- [ ] `notebooks/kg_bacdive.ipynb` — ?
+- [ ] `notebooks/kg_microbe_embedding.ipynb` — ?
+- [ ] `notebooks/kg_microbe_embedding.ipynb_bak` — **delete**
+- [ ] `notebooks/kg_microbe_embedding_withoutput.ipynb` — **delete** (output variant)
+- [ ] `notebooks/kg_microbe_embedding.html` — **delete** or gitignore (rendered output)
+- [ ] `notebooks/kg_microbe_train_taxa_to_media.ipynb` — ?
+- [ ] `notebooks/load_feba.ipynb` — ? (duplicate at repo root `load_feba.ipynb` — pick one location)
+- [ ] `notebooks/train_taxa_to_media.ipynb` — ?
+- [ ] `notebooks/train_taxa_to_media.ipynb_bak` — **delete**
+
+---
+
+## Questions to ask yourself for the ambiguous rows
+
+For each `?` above:
+1. Is anything in the repo already importing/referencing this file?
+   `grep -r 'basename' .` — if no hits, it's not wired in.
+2. When was it last touched? `git log --all --follow -- <path>` (returns
+   nothing if never committed anywhere).
+3. Is there a matching PR or issue you remember opening?
+4. Would losing it be a problem tomorrow? If not, delete.

docs/MERGE_CLEANUP.md

@@ -0,0 +1,129 @@
+# Post-Merge TSV Cleanup
+
+This document describes what happens when you see this line during
+`poetry run kg merge -y merge.yaml`:
+
+```
+[merge-cleanup] extracting merged-kg.tar.gz to normalize TSVs in place
+```
+
+## Where it runs
+
+`kg_microbe/merge_utils/merge_kg.py::_cleanup_merged_outputs()`, invoked from
+`load_and_merge()` immediately after `kgx.cli.cli_utils.merge` finishes. The
+cleanup is wrapped in a `try/except` so a cleanup failure never masks a
+successful merge — it prints `[merge] post-merge cleanup skipped: <error>`
+and proceeds.
+
+## Why it exists
+
+KGX's `TsvSink` produces three classes of artifacts we see in practice, and
+the cleanup is defensive normalization for each:
+
+1. **Duplicate header columns** (e.g. `provided_by` x2, `agent_type` x2) when
+   source files are headerless subsets of each other and column order is
+   reconstructed from per-record property sets.
+2. **Auxiliary columns** that leak through from obograph ingestion —
+   `subsets`, `meta`, and an edge `id` column — plus the deprecated
+   `knowledge_source` sitting alongside its biolink 3.x replacement
+   `primary_knowledge_source`.
+3. **Stray `\r` characters** emitted mid-header by `TsvSink` when a source
+   description contained an embedded CR (seen in ChEBI descriptions). This
+   corrupts CSV-reader parsing of the merged file downstream — it was the
+   exact cause of the "subject=`knowledge_level`, object=`meta`" false
+   positive that `kg-model-review` used to report when it read the stale
+   `merged-kg_default_edges.tsv`.
+
+Transform-level schema normalization (each transform writes a canonical
+header) removes most causes of (1) and (2). This post-merge step is
+idempotent and becomes a no-op when sources are already uniform. It does
+**not** fix `\r` upstream because the byte is injected by KGX's sink, not
+by the source files.
+
+## Step-by-step
+
+For each destination in `merged_graph.destination` in `merge.yaml` whose
+`format: tsv`:
+
+1. **Resolve paths** from the destination's `filename` (e.g. `merged-kg`):
+   - `nodes_file = output_dir/merged-kg_nodes.tsv`
+   - `edges_file = output_dir/merged-kg_edges.tsv`
+   - `archive   = output_dir/merged-kg.tar.gz`
+2. **Extract if compressed.** When `compression: tar.gz` is set and the
+   archive exists but the loose TSVs do not, untar `merged-kg.tar.gz` into
+   `output_dir` so the normalizer can edit files in place. This is the
+   step that prints:
+   ```
+   [merge-cleanup] extracting merged-kg.tar.gz to normalize TSVs in place
+   ```
+   A flag (`extracted_from_archive`) is remembered so the loose TSVs can be
+   deleted again after re-archiving (KGX's sink didn't leave them behind, so
+   neither do we).
+3. **Normalize nodes** (`_normalize_nodes_tsv`):
+   - Read header; strip stray `\r` from every line via `_iter_clean_lines`.
+   - Plan output columns: canonical order (`id, category, name, description,
+     xref, provided_by, synonym, deprecated, same_as`), then any unknown
+     forward-compat columns; drop `subsets, meta, iri`; dedup any repeated
+     column names by coalescing (first non-empty value wins).
+   - Log the schema diff: `dropped=[…] added=[…] deduped=[…]`, or
+     `schema already canonical (no-op)` when nothing changed.
+   - Rewrite the file atomically via a tempfile `replace`.
+4. **Normalize edges** (`_normalize_edges_tsv`):
+   - Same header cleaning and canonical reordering
+     (`subject, predicate, object, relation, primary_knowledge_source,
+     knowledge_level, agent_type`).
+   - Append the `has_percentage` extension column if present (metatraits).
+   - Drop `id, meta, knowledge_source`.
+   - Merge legacy `knowledge_source` into `primary_knowledge_source`: when
+     `primary_knowledge_source` is empty, fall back to `knowledge_source` on
+     the same row before dropping the latter.
+   - Same schema diff log line for edges.
+5. **Re-archive** (`_rewrite_tarball`). Write the normalized TSVs into a
+   temp `*.tar.gz` in the same directory and atomically `shutil.move` it
+   over `merged-kg.tar.gz`. Files are stored flat (`arcname=f.name`), so the
+   archive contains `merged-kg_nodes.tsv` and `merged-kg_edges.tsv` at its
+   root — no nested directory.
+6. **Clean up loose files.** If step 2 extracted them, delete them now.
+
+## Configuration prerequisites
+
+The cleanup only runs for destinations that are `format: tsv`. The archive
+path is `<output_directory>/<filename>.tar.gz`, where both fields come from
+`merge.yaml`:
+
+```yaml
+configuration:
+  output_directory: data/merged
+merged_graph:
+  destination:
+    merged-kg-tsv:
+      format: tsv
+      compression: tar.gz
+      filename: merged-kg
+```
+
+Given that config, the cleanup will target `data/merged/merged-kg.tar.gz`.
+
+## How to read the log output
+
+| Log line | What it means |
+|---|---|
+| `[merge-cleanup] extracting merged-kg.tar.gz to normalize TSVs in place` | Loose TSVs were missing; archive was untarred so normalizer could edit them. |
+| `[merge-cleanup] nodes merged-kg_nodes.tsv: schema already canonical (no-op)` | Transform-level normalization did its job; nothing to fix here. |
+| `[merge-cleanup] edges merged-kg_edges.tsv: dropped=['meta'] added=[] deduped=['agent_type']` | Dropped `meta` column, collapsed duplicate `agent_type` into one. |
+| `[merge] post-merge cleanup skipped: <error>` | Non-fatal: merge succeeded, cleanup raised. Investigate the error, but output is still usable (just unnormalized). |
+
+## Idempotency
+
+Running cleanup on an already-normalized archive yields two no-op log
+lines (nodes + edges) and rewrites the tarball with identical contents.
+Safe to invoke repeatedly.
+
+## Related
+
+- Source: `kg_microbe/merge_utils/merge_kg.py`
+- Constants: `CANONICAL_NODE_HEADER`, `CANONICAL_EDGE_HEADER`,
+  `EDGE_COLUMNS_TO_DROP`, `NODE_COLUMNS_TO_DROP`, `EDGE_EXTENSION_COLUMNS`
+  at the top of the same file.
+- Downstream: `.claude/skills/kg-model-review/kg_model_review.py` reads the
+  normalized `merged-kg.tar.gz` via `iter_tsv_from_tar`.