Spec reference v2: primitives, opacity, interactions; bare/spec conditions

.gitignore

@@ -28,3 +28,6 @@ runs/
 
 # Fetched Mol* bundle (run scripts/fetch_molstar.py)
 molbench/static/molstar/
+
+# mmCIF cache for task authoring (scripts/author_interaction_tasks.py)
+data/cif/

ROADMAP.md

@@ -7,6 +7,87 @@ sharpen Component 1, and validate the instrument.
 
 ---
 
+## Revised plan (2026-10-04) — read this first
+
+Three decisions, each backed by a test this round.
+
+**1. MVS stays the target. The bottleneck was our reference, not MVS.**
+The chat driver told a user "MVS cannot show hydrogen bonds". Checked against the
+Mol\* 5.9.0 source and real renders, that is wrong: `primitives` draws dashed lines
+between named atoms (`distance_measurement`, labelled with the length), and Mol\*
+computes H-bonds/metal contacts itself via `custom.molstar_show_non_covalent_interactions`
+on a component. Our reference listed only 8 node kinds (one of them wrong), so no
+model could ever draw an H-bond. A 4-prompt × 2-model A/B: old reference 0/4
+lines drawn and 2 Mol\* crashes on "50% transparent"; new reference 4/4 lines and
+2/2 transparency. Models still pick wrong chains/atoms (see grounding below).
+What MVS truly cannot do: select by distance or property, compute a
+superposition, style the computed interactions. Those stay in the API track.
+
+**2. Prompting conditions: a 2×2, no public "tuned" row.**
+
+| | no grounding | grounding pack |
+|---|---|---|
+| **bare** (output contract only) | what the model knows | knows facts, not MVS |
+| **spec** (vendored MVS reference) | **headline** | upper bound for an offline pipeline |
+
+* *Spec* = `molbench/mvs_reference.md`, a neutral, complete snapshot of MVS as
+  Mol\* 5.9.0 implements it. `--condition bare|spec` selects the row (done).
+* *Grounding pack* = facts about the task's structure prepared **offline** and
+  frozen with the task: chain ids and residue ranges, ligand codes and atom names,
+  SIFTS/UniProt mapping, ClinVar positions, the accepted entry ids. Reproducible
+  by construction; no live calls during a run. Can be ablated separately
+  (MVS-reference vs molecular-grounding).
+* The chat driver's own prompt is **tuned** on everything it sees and is owned by
+  `molstar-chat-driver`. It is not scored as a model comparison. Product claims
+  come from real user prompts that arrive *after* a prompt version is frozen
+  (graded by thumbs up/down or the C2 judge). Curated user prompts become new
+  benchmark tasks — after which they are dev data for the next prompt version.
+
+**3. External resources (PubMed, PDB, UniProt, ClinVar): closed-book core.**
+Lookups enter only as the frozen grounding pack above. Build a live tool loop
+only if grounding lifts the lookup-heavy tasks (`mvs_clinical`, `mvs_resolve`)
+by ≥10 F1 points on a few cheap models. PubMed is out of scope. This keeps
+MolBench a visualization benchmark that *measures* the value of lookups rather
+than an API-calling benchmark.
+
+### Work list
+
+| # | item | status |
+|---|---|---|
+| R1 | Spec reference v2 (primitives, opacity, interactions, colour themes, fixed rep types) | done |
+| R2 | Grader: primitives keyed on shape + atom pair (styling/direction ignored); interactions flag and colour theme graded | done |
+| R3 | `tasks/mvs_interactions/` — 6 tasks, every atom checked against the mmCIF with gemmi (`scripts/author_interaction_tasks.py`) | done |
+| R4 | `--condition bare\|spec` in the runner, recorded in run meta | done |
+| R5 | Regression run: old vs spec vs bare on 3 cheap models, all MVS tasks | done — see below |
+| R6 | Grounding packs: offline builder (gemmi + SIFTS + existing resolve/clinical provenance) and `--grounding` flag | todo |
+| R7 | Grader: resolve selections to atom sets against the mmCIF, so equivalent namings (`label_comp_id`+atom vs residue number+atom) match and empty selections score zero | todo |
+| R8 | Chat driver v2: new reference in its prompt + empty-selection lint (a primitive end that matches nothing is drawn to the origin by Mol\*) | shipped (chat-driver #18, deployed 2026-10-06) |
+| R9 | Chat driver: thumbs up/down per turn, for grading real traffic | todo |
+| R10 | Leaderboard: regimes for `resolve` and `interactions` sources (unknown sources now raise) | done |
+
+**R5 result (2026-10-06, 1 sample, 76 tasks, ~$0.85).** Mean F1:
+
+| model | old ref | spec | bare | interaction tasks old → spec |
+|---|---|---|---|---|
+| Claude Haiku 4.5 | 0.800 | 0.806 | 0.061 | 0.52 → 0.72 |
+| DeepSeek V3.2 | 0.827 | 0.814 | 0.018 | 0.58 → 0.56 |
+| Qwen3-30B-A3B | 0.576 | 0.522 | 0.000 | 0.22 → 0.18 |
+
+* **Bare ≈ 0 for every model.** MVS is not in pretraining in any usable form;
+  the bare row measures nothing but "can't". Keep it as one line in the paper,
+  not a leaderboard column.
+* **Spec costs ~0.01 on the old tasks for the strong models** (within 1-sample
+  noise: 57 tasks better, 63 worse of 228). Part of it is real: with a richer
+  reference, Haiku adds unrequested ball-and-stick on the resolve tasks
+  (same, correct entries; precision drops).
+* **The small model loses ~0.05, almost all to JSON bracket-count errors** on a
+  ~2.4× longer prompt (parse success 0.78 → 0.67). Longer documentation hurts
+  small models — a finding, and an argument for B6 (structured outputs).
+* Interaction tasks are not yet solved by anyone: line endpoints are named with
+  the wrong atoms or chains. That is the grounding gap (R6).
+
+---
+
 ## Phase A — Component 2: render + VLM judge  ← the headline
 
 Open-ended requests ("show the key hydrogen bond", "highlight the allosteric
@@ -74,7 +155,8 @@ strong judge, structured protocol, self-variance reporting); rubric quality
   ~0.5–0.67 because the `polymer cartoon` scaffold dominates F1. Add a metric that
   isolates "did you select the right residues" from scene scaffolding, and report
   it for the grounded/clinical tiers. Re-score the clinical regime.
-* **B2. Interaction/interface task type** (ChatMol-inspired). Use gemmi neighbour
+* **B2. Interaction/interface task type** (ChatMol-inspired; named-atom H-bond and
+  computed-interaction tasks STARTED, see R3). Use gemmi neighbour
   search to extract interface/contact residues (ligand–protein, protein–protein),
   giving grounded "show the binding-site residues" / "show the PPI interface"
   tasks — a realistic, high-value class we currently lack.
@@ -85,11 +167,10 @@ strong judge, structured protocol, self-variance reporting); rubric quality
   protein's UniProt accession (`accepted_ids`, folded by the grader). Next: more
   targets, and an `exists` sub-metric (did the id resolve at RCSB at all?) — a
   hallucinated id and a wrong-but-real id are different failures.
-* **B5. Colour schemes.** The chat-driver's prompt now teaches Mol*'s
-  `custom.molstar_color_theme_name` on a `color` node for rainbow / by-chain /
-  by-SS colouring (models kept writing `"spectrum"` as a colour). Porting that here
-  needs the grader to treat a theme-coloured representation as equivalent to the
-  per-chain component references it would replace; until then the two prompts differ.
+* **B5. Colour schemes (PARTLY DONE).** The Spec reference now teaches Mol*'s
+  `custom.molstar_color_theme_name` on a `color` node, and the grader scores the
+  theme instead of the placeholder colour. Still missing: treating a theme-coloured
+  representation as equivalent to the per-chain components it would replace.
 * **B6. Structured outputs.** Constrained decoding (`response_format: json_schema`,
   `strict`) with a trimmed MVS schema makes invalid trees impossible by
   construction. Measure validation failures + F1 against the prompt-only path on
@@ -147,6 +228,8 @@ The grader is a system with its own error rate; validate it against ground truth
 
 ## Suggested sequence
 
+*Superseded in part by the revised plan at the top: R5 → R6 → R7 come first.*
+
 1. **A0 renderer** (unblocks everything visual) → **A1 judge** → **A2 wire-in**
    → **A3 corpus** → **A4 validation**. This delivers a working Component 2.
 2. In parallel / between: **B1 selection-accuracy** (cheap, sharpens current

molbench/mvs.py

@@ -53,12 +53,15 @@ def _selector_signature(sel: Any) -> Any:
         comp = e.get("label_comp_id") or e.get("auth_comp_id")
         comp = comp.upper() if isinstance(comp, str) else comp
         atom = e.get("label_atom_id") or e.get("auth_atom_id")
+        element = e.get("type_symbol")
+        element = element.upper() if isinstance(element, str) else element
         sigs.append((
             ("chain", str(chain) if chain is not None else None),
             ("resnum", resnum),
             ("range", (beg, end) if (beg is not None or end is not None) else None),
             ("comp", comp),
             ("atom", atom),
+            ("element", element),
         ))
     return tuple(sorted(map(str, sigs)))
 
@@ -70,10 +73,55 @@ def _color_signature(value: Any) -> Any:
     return value
 
 
-def _param_signature(kind: str, params: dict | None) -> Any:
+_PRIMITIVE_POINTS = ("start", "end", "position", "a", "b", "c")
+_UNDIRECTED_PRIMITIVES = {"tube", "distance_measurement"}
+
+
+def _point_signature(point: Any) -> Any:
+    """A primitive position: an [x, y, z] coordinate (order matters) or an atom selection."""
+    if isinstance(point, list) and all(isinstance(v, (int, float)) for v in point):
+        return ("xyz", tuple(float(v) for v in point))
+    return _selector_signature(point)
+
+
+def _param_signature(kind: str, params: dict | None, custom: dict | None = None) -> Any:
     params = params or {}
+    custom = custom if isinstance(custom, dict) else {}
     if kind == "component" and "selector" in params:
-        return ("selector", _selector_signature(params["selector"]))
+        sig = ("selector", _selector_signature(params["selector"]))
+        # Mol* computes and draws the interactions itself when this flag is set, so
+        # it is part of what the component shows, not a styling detail.
+        if custom.get("molstar_show_non_covalent_interactions"):
+            # The radius decides which neighbours are drawn; Mol*'s default is 5 Å.
+            radius = custom.get("molstar_non_covalent_interactions_radius_ang", 5)
+            try:
+                radius = round(float(radius), 1)
+            except (TypeError, ValueError):
+                pass
+            sig += ("interactions", radius)
+        return sig
+    if kind == "primitive":
+        # Key on the shape and the atoms it joins. Radius, dash length and label text
+        # are styling. A line or distance is the same whichever end comes first; an
+        # arrow's direction and an angle's vertex are not, so those keep their roles.
+        pkind = params.get("kind")
+        if pkind in _UNDIRECTED_PRIMITIVES:
+            points = frozenset(_point_signature(params[k]) for k in _PRIMITIVE_POINTS
+                               if k in params)
+        else:
+            points = tuple((k, _point_signature(params[k])) for k in _PRIMITIVE_POINTS
+                           if k in params)
+        # A 3D label's text is its content (unlike a measurement's label template).
+        text = (" ".join(str(params.get("text", "")).split()).casefold(),) if pkind == "label" else ()
+        return ("primitive", pkind, points) + text
+    if kind == "primitives":
+        return ()  # a grouping node; its colour is styling
+    if kind == "color" and custom.get("molstar_color_theme_name"):
+        # The theme overrides the placeholder colour, so grade the theme (and where it applies).
+        sig = ("theme", custom["molstar_color_theme_name"])
+        if "selector" in params:
+            sig += (_selector_signature(params["selector"]),)
+        return sig
     if kind == "representation":
         # Key on the representation *type* only. The builder injects secondary
         # defaults (e.g. surface_type='molecular') that a model shouldn't be
@@ -124,9 +172,10 @@ def walk(node: dict, prefix: tuple) -> None:
         # "group residues in one component" and "one component each" score alike.
         if kind == "component" and isinstance(params.get("selector"), list) \
                 and len(params["selector"]) > 1:
-            variants = [("selector", _selector_signature(e)) for e in params["selector"]]
+            flag = _param_signature(kind, {"selector": "all"}, node.get("custom"))[2:]
+            variants = [("selector", _selector_signature(e)) + flag for e in params["selector"]]
         else:
-            variants = [_param_signature(kind, params)]
+            variants = [_param_signature(kind, params, node.get("custom"))]
 
         for vsig in variants:
             here = prefix + ((kind, vsig),)
@@ -263,14 +312,19 @@ def categorize(root: dict) -> list[str]:
     kinds: dict[str, int] = {}
     has_expr = False
     reps: set[str] = set()
+    interactions = False
 
     def walk(node: dict) -> None:
-        nonlocal has_expr
+        nonlocal has_expr, interactions
         k = node.get("kind")
         kinds[k] = kinds.get(k, 0) + 1
         params = node.get("params") or {}
         if k == "component" and not isinstance(params.get("selector"), str):
             has_expr = True
+        custom = node.get("custom")
+        if k == "component" and isinstance(custom, dict) \
+                and custom.get("molstar_show_non_covalent_interactions"):
+            interactions = True
         if k == "representation":
             reps.add(params.get("type"))
         for c in node.get("children") or []:
@@ -295,7 +349,13 @@ def walk(node: dict) -> None:
         cats.append("annotation")
     if reps & {"surface", "isosurface", "gaussian-surface", "molecular-surface"}:
         cats.append("surface")
-    if kinds.get("volume") or kinds.get("primitives"):
+    if kinds.get("opacity"):
+        cats.append("transparency")
+    if kinds.get("primitive"):
+        cats.append("measurement")
+    if interactions:
+        cats.append("interactions")
+    if kinds.get("volume"):
         cats.append("volume")
     return cats