composition detail=true: soft-cap co-occurrence, spill the full dump to full_result_path

architecture.md

@@ -47,7 +47,7 @@ Frontier exposes 4 tools – 3 domain tools with multiple actions, plus a skill
 | | `curate` | Pin a solution to the curated set by `solution_id` or `content_signature` (the identifier `certify`'s `per_pick` hands back, so a dominated finalist's certified replacement is pinned straight from the overlay with `source="exact"`), with custom name and notes (optional `scenario` for scenario frontiers); the response carries a `quality` gate (GOOD/WARNING/DEGENERATE — empty plan, single-option concentration, allocations pinned at bounds — surfaced, never blocking) — and manage existing pins: `remove=true` unpins, `rename="…"` renames, both by content signature (absorbs the old `uncurate` / `rename_curated`). |
 | | `curated` | List all curated solutions with `in_current_frontier` survival flag — or render the handoff export with `format="markdown"\|"csv"` (columns: name, content_signature, quality, objective values, selected_options; invalid format errors cleanly; absorbs the old `export_curated`); listings and exports carry each finalist's `quality` gate. |
 | | `marginal_analysis` | Marginal rate analysis: cost-per-unit between adjacent solutions, inflection point detection (where marginal cost jumps sharply). A per-pair `rate_guard` keeps that headline honest on both sides of the quotient. A transition whose DENOMINATOR step falls below its floor (the larger of 0.1% of that objective's range and a quarter of the frontier's typical step) is a near-tie whose rate is a division artifact — marked `degenerate`, kept in the detail rows, and excluded from the inflection and from `steepest_transitions`. A transition whose NUMERATOR step falls below its own floor bought essentially nothing, so its ~0 rate is no baseline for a jump either — marked `flat` and excluded from the inflection (as both the jump's baseline and its landing) and from `steepest_transitions` — on the duals basis a flat row's rate is the endpoint shadow price, so it can head the steepest list while having bought nothing — which keeps a marker off the end of a flat stretch (`degenerate` = the price exploded, `flat` = nothing was bought). The numerator floor is derived separately, since numerator steps are the frontier's *shape* (they swing by orders of magnitude) where denominator steps are its near-uniform *sampling*: a twentieth of the typical step, capped at half the pair's tenth-percentile step (the k-th smallest, never the smallest — one artifact must not set the cap that spares it), and carrying no range term, since a fixed slice of a fixed span measured against steps that shrink as span/n flags an ever-larger share as the frontier is sampled more finely. Both terms are scale-free in sampling density, and the cap bounds the flag to the distribution's own tail; a transition that improved BOTH objectives (the pair's conflict paid on a third) is marked `co_improvement`, since its rate is a ratio of two gains rather than a price, and a co-improving inflection's rationale says so instead of naming a price. Summary stats span every transition, so the median read is what it was before the guard. Default summary; `detail=true` for per-pair breakdown — one row per transition per pair, so above `EXPLORE_DETAIL_CAP` rows the full payload is written to `full_result_path` and the response keeps the window around each inflection. Optional `scenario` param. |
-| | `composition` | Mine the solution set (knowledge-discovery pillar): per-option `selection_count`/`selection_pct` (consensus vs distinctive – also surfaced on `tradeoffs`, and per-option on `solution` via `option_context`), `co_occurrence` lift (complements/substitutes), `design_principles` (always / never / region-bound), decision-space `clusters` (strategy families, each with defining options + objective signature – clustering in decision space, not objective space) beside a `clusters_summary` that prices the partition (`n_families`, `largest_family_share`, plus a note above 0.9 – a 104/1/1/1 split otherwise reads exactly like a meaningful 155/19), and `feedback_rules` (rules separating liked from disliked solutions when feedback exists → candidate latent constraints; every block carries `n_liked`/`n_disliked`, the rated solutions the rules were induced from, and below three per side the note calls the rule a hypothesis to confirm — separation and coverage both read 1.0 off a single liked solution by arithmetic). Operates on the active frontier, or a curated subset via `solution_ids` / `signatures`. |
+| | `composition` | Mine the solution set (knowledge-discovery pillar): per-option `selection_count`/`selection_pct` (consensus vs distinctive – also surfaced on `tradeoffs`, and per-option on `solution` via `option_context`), `co_occurrence` lift (complements/substitutes), `design_principles` (always / never / region-bound), decision-space `clusters` (strategy families, each with defining options + objective signature – clustering in decision space, not objective space) beside a `clusters_summary` that prices the partition (`n_families`, `largest_family_share`, plus a note above 0.9 – a 104/1/1/1 split otherwise reads exactly like a meaningful 155/19), and `feedback_rules` (rules separating liked from disliked solutions when feedback exists → candidate latent constraints; every block carries `n_liked`/`n_disliked`, the rated solutions the rules were induced from, and below three per side the note calls the rule a hypothesis to confirm — separation and coverage both read 1.0 off a single liked solution by arithmetic). Operates on the active frontier, or a curated subset via `solution_ids` / `signatures`. `detail=true` lifts the co-occurrence top-N to every varying pair — O(options²) — so past the shared explore detail cap the response keeps the top-ranked pairs (`truncated` echoes total vs shown) and writes the full payload to disk as `full_result_path`, the same soft-cap `solutions`/`marginal_analysis` `detail=true` carry. |
 | **get_skill** | *(single action)* | Retrieve workflow guidance by name. Without `section`: the skill's core (SKILL.md). With `section=<heading>`: exactly that section, resolved across the core **and** the skill's `references/` files — the cheap recovery path every decision response's `guidance_pointer` names. Unknown sections error with the available titles. Works with all MCP clients (unlike resources, which require client-side resource support). Available skills: `problem_framing`, `data_collection`, `optimization_strategy`, `solution_interpreter`. |
 
 **Wrong-name guards.** FastMCP silently drops unknown arguments, so the classic miscalls (`model scenarios=`, `solve scenario=`/`run_scenarios=`, `explore label=`) exist as schema-visible guard params: each carries a steering description in the advertised inputSchema (heading the miscall off at call-composition time) and, if still called, returns a redirect instead of a silent drop — except the one self-consistent miscall, `run_scenarios=true` beside `action="run_scenarios"`, which is ignored rather than bounced. Two more guards police *shape* rather than name: `explore format=` is the curated handoff export's renderer only (one word, one meaning), so `format="summary"/"compact"` on any other action returns a redirect naming `detail=` and the automatic `…_elided` blocks instead of a full payload dressed as a summary; and a single-option constraint (`force_include`/`force_exclude`/`allocation_bound`) sent with a plural `options` list is declined at parse time with the one-row-per-option shape and a pointer to `group_limit` — the type set is derived from the models, so a new single-option type inherits it (a `_ToolDecline`, so it is gated in `tests/test_server.py` with the other input-parsing behavior rather than in the schema-claim file). The schema-visible half is gated by `tests/test_tool_schema_guards.py`, which also pins the merge-vs-replace descriptions on `model`'s data params (objectives/options/constraints/reference_points replace, scores/interaction_matrices upsert, `scenario_config`'s constraint_overrides replace-or-inherit semantics, `source` load-only) against an SDK upgrade dropping `Field` metadata.

engine/explorer.py

@@ -2744,6 +2744,26 @@ def cap_marginal_detail(problem: Problem, result: dict, max_rows: int) -> dict:
     return {**result, "pairs": capped}
 
 
+def cap_composition_detail(result: dict, max_rows: int) -> dict:
+    """A token-capped view of a ``detail=true`` composition payload.
+
+    detail=true lifts the co-occurrence top-N from a summary handful to every
+    sometimes-but-not-always option pair — O(options²), which runs to thousands of rows
+    and hundreds of KB at portfolio scale. This returns a new payload keeping the top
+    ``max_rows`` pairs: the list is already ranked by departure from independence, so the
+    head IS the summary view scaled up, and no re-rendering is needed (the visualization
+    reads option_selection / principles / clusters, never the pair list). Nothing is
+    mutated: the caller keeps the full payload for the on-disk dump the capped response
+    points at.
+    """
+    pairs = result.get("co_occurrence") or []
+    if len(pairs) <= max_rows:
+        return result
+    return {**result,
+            "co_occurrence": pairs[:max_rows],
+            "truncated": {"total_pairs": len(pairs), "shown": max_rows}}
+
+
 def _rate_guard(rates: list[dict], obj_a, obj_b, floors: tuple[float, float]) -> dict:
     """What the unsigned `rate` column hides, echoed so the pair's inflection is auditable.
 

mcp_server/server.py

@@ -2113,9 +2113,10 @@ def explore(
     `model` docstring for `_skill_guidance` shape and the once-per-problem throttle).
     Structural `model/update` edits re-arm so the next solve re-injects.
 
-    full_result_path: large-result actions (solutions detail=true, marginal_analysis detail=true)
-    truncate past a cap and carry `full_result_path` — a full JSON dump on disk, also named by
-    solve/run's response. Reference the file instead of re-requesting the payload.
+    full_result_path: large-result actions (solutions detail=true, marginal_analysis detail=true,
+    composition detail=true) truncate past a cap and carry `full_result_path` — a full JSON dump
+    on disk, also named by solve/run's response. Reference the file instead of re-requesting
+    the payload.
     content_signature: stable hash of a solution's selected options (and allocations for
     proportional problems); survives re-solves, so prefer it over solution_id when referencing
     curated solutions or recording feedback across runs.
@@ -2234,6 +2235,8 @@ def explore(
                    clusters_summary reports the partition's quality (n_families,
                    largest_family_share) — a lopsided split is one strategy with corner
                    variants, not a menu of families; present it that way.
+                   Pass detail=true for the full co-occurrence pair list; past the cap it
+                   keeps the top-ranked pairs inline and carries full_result_path.
                    Optional: solution_ids, signatures, detail, source.
 
     Scenario param (optional):
@@ -2516,6 +2519,8 @@ def explore(
                     p, solution_ids=solution_ids, signatures=signatures, source=source, detail=detail)
             except ValueError as e:
                 return {"error": str(e)}
+            if detail:
+                result = _cap_composition_detail(p, result)
             return _attach_guidance_pointer(result, action)
         case _:
             return {"error": f"Unknown action: {action}."}
@@ -2545,6 +2550,29 @@ def _cap_marginal_detail(p: Problem, result: dict, scenario: str | None) -> dict
     return capped
 
 
+def _cap_composition_detail(p: Problem, result: dict) -> dict:
+    """Soft-cap `composition detail=true` the way `marginal_analysis detail=true` is capped.
+
+    detail=true lifts the co-occurrence top-N to every sometimes-but-not-always option
+    pair — O(options²), hundreds of KB on a portfolio-scale option set — so the full
+    payload goes to disk beside the run's own result file and the response keeps the
+    top-ranked pairs (the list is ranked by departure from independence, so the head is
+    the summary view scaled up). Best-effort: an unwritable data dir degrades to the full
+    inline payload rather than losing the analysis.
+    """
+    rows = len(result.get("co_occurrence") or [])
+    run_id = (result.get("frontier_source") or {}).get("run_id")
+    if rows <= EXPLORE_DETAIL_CAP or not run_id:
+        return result
+    try:
+        path = store.write_run_result(p.problem_id, f"{run_id}-composition", result)
+    except (OSError, ValueError):
+        return result
+    capped = explorer.cap_composition_detail(result, EXPLORE_DETAIL_CAP)
+    capped["full_result_path"] = str(path)
+    return capped
+
+
 def _explore_feedback(
     p: Problem,
     solution_id: int | None,

skills/solution_interpreter/references/explore-diagnostics.md

@@ -181,7 +181,7 @@ Mining the solution set — operates on the active frontier, or a curated subset
 |---|---|---|
 | `scope` | obj | `{set: "frontier"\|"curated", n_solutions, approach}` |
 | `option_selection` | list | Per option, sorted by `selection_pct` desc (also on `tradeoffs`, and per-option on `solution` via `option_context`) |
-| `co_occurrence` | list | Option pairs ranked by departure from independence (top 8; all when `detail=true`) |
+| `co_occurrence` | list | Option pairs ranked by departure from independence (top 8; all when `detail=true` — past the detail cap the top-ranked pairs stay inline, `truncated` echoes total vs shown, and the full list is on disk at `full_result_path`) |
 | `design_principles` | list | Statements that hold across the set |
 | `clusters` | list | Decision-space strategy families (omitted when too few solutions) |
 | `clusters_summary` | obj | `{n_families, largest_family_share}`, plus a `note` when the share exceeds 0.9 — rides with `clusters` |

tests/test_server.py

@@ -2950,6 +2950,34 @@ def test_composition_curated_subset(self):
         assert r["scope"]["set"] == "curated"
         assert r["scope"]["n_solutions"] == 1
 
+    def test_small_frontier_detail_stays_inline(self):
+        pid = _build_solvable_problem()
+        srv.solve(action="run", problem_id=pid)
+        r = srv.explore(action="composition", problem_id=pid, detail=True)
+        assert "full_result_path" not in r
+        assert "truncated" not in r
+
+    def test_large_frontier_detail_writes_full_result_path(self):
+        # detail=true lifts the co-occurrence top-N to every sometimes-but-not-always
+        # pair — O(options²), ~400 KB on the 300-option showcase — so detail=true dumps
+        # to disk and keeps the top-ranked pairs inline, exactly as marginal_analysis
+        # detail=true does (the claim the explore docstring makes for large-result actions).
+        import json
+        from pathlib import Path
+
+        pid = srv.model(action="load", source="capital_project_selection_300")["problem_id"]
+        r = srv.explore(action="composition", problem_id=pid, detail=True)
+        assert len(r["co_occurrence"]) == srv.EXPLORE_DETAIL_CAP
+        assert r["truncated"]["total_pairs"] > srv.EXPLORE_DETAIL_CAP
+        assert r["truncated"]["shown"] == srv.EXPLORE_DETAIL_CAP
+        # The inline response fits a tool-result budget the full dump blows past.
+        assert len(json.dumps(r)) < 100_000
+        on_disk = json.loads(Path(r["full_result_path"]).read_text())
+        assert len(on_disk["co_occurrence"]) == r["truncated"]["total_pairs"]
+        assert "truncated" not in on_disk
+        # The inline pairs are the head of the ranked full list — a window, not a re-rank.
+        assert on_disk["co_occurrence"][:srv.EXPLORE_DETAIL_CAP] == r["co_occurrence"]
+
 
 class TestRegret:
     def _add_scenarios(self, pid):