Deep live pipeline: reference-depth reports generated LIVE (fact-store + per-report fan-out) - #5
Merged
Merged
Conversation
Restructure the report suite to the gold-standard reference deliverables Akhilesh shared: each of the seven reports is now a STANDALONE document with its own branded cover, its own table of contents, and hierarchically numbered sections. Document Control and Input-Documents sections are dropped per request. Identity & components (assets.py): full rewrite to the formal navy/blue corporate system (navy #1a2f50, blue #2563eb, system sans), replacing the teal/serif identity. Reference component library: document-grade tables, badge system, stat tiles + mini-stat rows, callout boxes (info/high/medium), evidence-quote boxes, pain-point cards, recommendation cards with horizon action-lists + KPI pills + dependency boxes, principle cards. Grounded SVG infographics (render.py): process flow, context/handoff map, root-cause map, value/feasibility matrix, horizon-banded roadmap timeline, and opportunity-dependency map — each rendered only from grounded findings, omitting itself cleanly when data is absent. Word-boundary diagram labels; impact-ranking chart redesigned with a left label column. Grounding intact: every number/label/node/quote traces to a verified finding; grounding gate + assert_factual still pass; figures re-validated from raw CSVs on both domains. Suppressed client names never appear in body or chrome. make_pdf.py: per-report standalone PDFs + combined suite PDF. Both validated domains (o2c, p2p) regenerated. 100% statement+branch coverage; pyrefly clean.
…egisters
Closes the depth gap against Akhilesh's three reference reports.
1. Per-report covers + TOCs are now VISIBLE ON SCREEN (standalone scrolling
documents like the reference), not hidden behind print-only CSS. A slim top
nav-bar replaces the sidebar.
2. Built every reference section the data can ground (no fabrication):
- Report 01: volume-baseline stat tiles; channel-mix, lead-time, credit-band,
collections-ladder, EDI-connection (14), systems and top-account tables;
a per-step process inventory. 12 pages, 11 tables — reference parity.
- Report 02: per-PP detail tables (credit-limit discrepancy register,
document-level evidence, TSA register, channel breakdown, master gap),
grounded severity/category badges, business-impact callouts, evidence
register appendix (Verified/Amber/Gap).
- Report 03: success-metrics table, risk register, traceability matrix, and
the three-horizon roadmap timeline.
Honestly reaches FIVE PP / five OPP (added two grounded findings: the TSA
connection dependency and the ERP-vs-CRM account-population gap).
Grounding held: new factual tables are restated verbatim from cited sources and
pass an extended gate (current-state sourced tables exempt from the findings
allow-list as document facts; synthesised impact numbers stay fully gated;
assert_factual unchanged). Every figure re-validated from raw CSVs/SOPs.
Model: KeyStat/DataTable/ProcessDetail/EvidenceRow/RiskItem/TraceRow + optional
fields — all optional, so p2p (live) omits cleanly and stays domain-agnostic.
Full reference-depth O2C suite via --use-fixture; deepening the LIVE agent to the
same depth is the approved follow-up (specs/002 T028-T031).
184 tests; 100% coverage; pyrefly clean. Both domains regenerated (o2c 58pp PDF).
…ssumption The KG-lite that the per-report synthesis fan-out will expand from — replaces the flat ~3-finding waist with a structured, sourced collection: - QuantFact (measured number + sources + tier), DocQuote (verbatim snippet), EntityFact (account/connection/incident/… with field-level attributes), Relation (handoff/conflict/…). FactStore.numbers_allow() + slice_for(terms). - StrategyProfile (read from the manifest; shapes the strategic reports later) + PlanningAssumption (labelled forward-looking content; never fabricated as fact). - discovery/factstore.py: generic deterministic builders that harvest facts from findings' computed_values/narrative_values + the registered CSVs (rows → typed entities) — no domain constants; a thinner domain yields fewer facts. Grounding intact: numbers trace to the run; quotes are verbatim; tiers carry verified/amber/gap (challenged findings downgraded). All additive/optional, so the fixture and legacy single-emit path are unaffected. 202 tests; 100% statement+branch coverage; pyrefly clean. Per specs/003 tasks T001-T005.
discovery/fanout.py — replaces the single 16K emit_synthesis with MANY bounded per-section / per-opportunity generations: - synth_section(): one bounded, cache-keyed LLM call per report section, fed only the relevant fact-store slice (+ StrategyProfile brief for strategic reports); retries once on a grounding miss; omits the section if it can't be grounded (one bad section never aborts the suite). - validate_section(): the per-section grounding gate — same rules as the monolith (untraceable measured numbers rejected, unknown doc_key rejected, Report-01 factual-linted, sourced factual tables exempt as document facts), applied per call. - planning channel: forward-looking content (date/owner/sla/threshold/cadence/cost/ sequence) is emitted into planning_assumptions and collected as typed PlanningAssumption — NEVER number-gated as a fact. - run_synthesis_fanout(): orchestrator — fans out per report + per opportunity, merges (lists concat, scalars keep-first), returns (payload, planning). Determinism preserved (every sub-call routed through the existing LLMClient cache); no framework (plain-code, per decision.md). All driven by fake LLMs through the real ToolTurn contract — no live API, no spend. 216 tests; 100% statement+branch coverage; pyrefly clean. Per specs/003 tasks T006-T010.
…markers discovery/fanout_specs.py — the real per-report emit schemas + prompts + the top-level orchestrator (run_report_fanout): - each report emits only the SynthesisContent slice it owns (current_state with all the deep tables/process-detail for r01; pain points + detail tables + evidence register for r02; transformation + metrics + risk + traceability for r03; roadmap + posture for r05; exec summary for r00), so each call stays in its own token budget — the single-16K ceiling is gone. - one opportunity expanded per pain point (the portfolio addresses each PP), each fed its relevant fact slice but gated against the FULL run allow-list (a focused slice never starves grounding). - StrategyProfile shapes the STRATEGIC reports (03/05) only; tactical 04/06 stay direction-agnostic. - merged payload reconstructs a reference-depth SynthesisContent via the extended build._from_payload (now maps data_tables/process_detail/baseline_stats, per-PP detail_table + severity/category/consequence, evidence/risk/traceability). render: a labelled "Planning assumptions" panel (r05) + a dashed b-plan badge so forward-looking content (dates/owners/SLAs/targets/cadence) is visibly NOT a fact. Determinism preserved (per-call cache keys); no framework. Fake-LLM-driven; no spend. 222 tests; 100% statement+branch coverage; pyrefly clean. Per specs/003 T011-T014.
build.build_synthesis now defaults live runs to the deep per-report fan-out when the domain registry is available: builds the grounded fact-store, fans out per report + per opportunity, assembles a reference-depth SynthesisContent, and attaches the fact-store, StrategyProfile, and planning assumptions (the non-empty strategy fields are surfaced alongside the roadmap posture). Falls back to the legacy single-emit path when reg is absent or fanout=False — full back-compat. run.py passes the manifest-bearing registry into build_synthesis so the fan-out can read the StrategyProfile and build the fact-store. Behaviour on the o2c GOLDEN replay confirmed: discovery replays from cache, the new fan-out synthesis calls have no cached responses yet (the cache holds the old single-emit), so the run gracefully falls back to the fixture — exactly as designed. Populating the fan-out cache needs ONE live run (Phase 4); the fan-out LOGIC is fully validated offline by the fake-LLM tests. 224 tests; 100% statement+branch coverage; pyrefly clean. Per specs/003 T015-T016. (Note: lifting the hardcoded "exactly 3 findings" discovery seed is deferred to the Phase-4 live run, where re-baking the golden is expected — changing the discovery prompt would otherwise break the offline golden replay.)
…un bug) The one live o2c run surfaced a real bug the fake-LLM tests could not: the fan-out gated each section against the fact-store's narrow numbers_allow() (~15 numbers), not the run's authoritative allowed_numbers() (~2,145: tool numbers + finding values + derived ratios). Legitimately-grounded figures the live model used (e.g. row-level 16, 37) were rejected, so whole sections failed grounding and were omitted — the live suite came back empty. Fix: run_synthesis_fanout takes an `allow` set; run_report_fanout passes synthesis.allowed_numbers(raw_payload) — the same authoritative list the monolith gate uses. The fact-store slice still shapes the PROMPT; the gate uses the full allow-list. After the fix the live run produces 6 grounded pain points + 6 opportunities + 48 labelled planning assumptions + the deep current-state tables. 224 tests; 100% coverage; pyrefly clean. (Visual defects in the live SVGs/tables — matrix bubble overlap, wide-table clipping, S/4HANA node split, empty dependency section — fixed in the following commit.)
The live o2c run + visual QA surfaced 4 rendering defects the live data exposed:
1. Prioritization-rationale table clipped off the right edge → table-layout:fixed +
word-break so the three prose columns wrap within the page.
2. Value/feasibility matrix bubbles overlapped into "OPP?PP4" clusters when 5-6
opportunities shared a coordinate → spread co-located bubbles around a
crowding-scaled ring so circles and labels never overprint.
3. Data-flow map split "SAP S/4HANA" into two fake nodes → split system names only
on a spaced separator (" / ", " & ", ", "), never a bare "/".
4. Empty "How the work connects" section when opportunities had no declared
dependencies → omit the heading when the dependency map is empty.
Clean live suite re-rendered; golden re-baked so it replays offline.
224 tests; 100% coverage; pyrefly clean. Per specs/003 T017-T019.
Second live run, on the THIN p2p domain (2 sources, no fixture) — the real domain-agnostic test. The deep fan-out produced a fully grounded p2p suite (3 PP, 3 OPP, 3-horizon roadmap, 5 data tables, 9 baseline stats, registers, 35 labelled planning assumptions) — every figure from p2p's own data, ZERO o2c leakage. The thin-data risk (empty sections / broken low-node charts) did not materialise. Fixed two CSS defects visual QA surfaced (also hardens o2c): 1. Wide prose tables shredded a narrow column char-by-char (overflow-wrap:anywhere) → break-word; traceability matrix gets table-layout:fixed + 'trace' class. 2. Live model occasionally wrote a readiness reason in ALL CAPS → _deshout down-cases it (preserving acronyms); cosmetic only, never changes words/numbers. Both domains regenerated; p2p golden baked (offline replay). 225 tests; 100% coverage; pyrefly clean.
…re-bake goldens
Three live-surfaced robustness fixes + a re-bake of both goldens against the
deep fan-out path.
discovery/factstore.py
_clean_quote() strips raw tool-output field names (n_mismatch, sum_delta,
from_tool, the tool verbs) from harvested DocQuotes at the grounding boundary.
A quote that is ONLY such jargon + numbers carries no prose meaning and is
dropped; real prose is left untouched. The deterministic chokepoint: no
internal tool token can reach a per-report brief or a client report.
discovery/reportsuite/render.py
_humanize_enums() rewrites raw SHOUTY_SNAKE CSV enum values the model copied
into prose (NOT_FULFILLED -> 'not fulfilled') wherever esc() renders visible
text. Cosmetic only (casing/underscores, never numbers); single-token
acronyms (EDI, SAP, S/4HANA) are preserved. _deshout() also humanises them.
discovery/fanout.py
collect_planning() coerces a bare-string planning_assumptions item and skips
non-str/non-dict items. run_synthesis_fanout() wraps each report and each
opportunity in try/except so one malformed section is omitted, not the suite.
discovery/agent_loop.py
Loop pairs a tool_result for EVERY tool_use regardless of stop_reason (a
tool_use on a non-'tool_use' stop reason previously desynced the next request
-> 400 unpaired tool_use). Seed kept bounded; depth comes from the fact-store
+ fan-out, not finding count.
Re-baked goldens (live, Anthropic Opus):
o2c -> 6 pain points / 6 opportunities / 54 labelled planning assumptions;
no leak, no raw enums, 7-col traceability fits A4.
p2p -> 3 / 3 / 36; client 'Acme Manufacturing' shown; clean.
231 passed, 100% branch coverage, pyrefly clean.
.gitignore: v1/spike/ (LangGraph evaluation; not product code).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
Make the LIVE pipeline produce the six reports as close as possible to the reference deliverables in
expected_individual_report_formats/— not via a hand-authored fixture.Root cause it fixes
The depth ceiling was the single
emit_synthesiscall (max_tokens=16000) producing all six reportsat once — impossible for ~12-dense-pages-each, so the model thinned every field (why depth previously
had to be hand-carried in the fixture).
Architecture (plain-code, no framework — specs/003 decision.md)
StrategyProfile + PlanningAssumption. Replaces the flat ~3-finding waist.
cache-keyed LLM call with its own grounding gate. The 16K ceiling is gone.
marked assumptions — never as measured fact; the gate still rejects ungrounded measured numbers.
direction-agnostic.
Framework decision
Prototyped plain-code vs a real LangGraph StateGraph and measured it: stay plain-code (+21 deps,
identical node bodies, must bypass its model layer to keep determinism/grounding; a star fan-out uses
none of its features). Revisit only at the deferred multi-engagement orchestrator.
Validated with TWO live runs
metrics/risk/traceability/evidence registers, 48 labelled planning assumptions. Reference-grade.
roadmap, 5 tables, 9 baseline stats, registers, 35 planning assumptions — every figure from p2p's
own data, zero order-to-cash leakage, no empty/broken sections.
Both goldens baked → both replay offline. The runs surfaced (and this PR fixes) real defects the
fake-LLM tests couldn't:
run's authoritative
allowed_numbers()(the live suite came back empty before this)."SAP S/4HANA" node split, an empty dependency section, a char-by-char wrapped traceability column,
and ALL-CAPS readiness cells; all fixed.
Quality
Domain-agnostic (p2p omits what it can't ground; o2c-clean). Spec-kit followed (specs/003), local.
Follow-ups (noted, not blocking)
readiness reasons in sentence case.