feat: prompt-driven authoring (3 tools + deterministic planner), official-ecosystem alignment, sidecar port resilience, and renderable embedded-extract dashboards

[SebAustin]

Summary

This branch (14 commits ahead of main) adds prompt-driven authoring to the Tableau publish MCP server: an agent can turn a file + a business question into a published datasource and a dashboard that actually renders on Tableau Cloud. It also aligns the project to the official Tableau ecosystem (a real TWB XSD validation gate, a TSC-matching chunk boundary, repositioned docs), hardens sidecar startup against a stale-port crash, and reworks the dashboard render path to embed the .hyper extract so workbooks are self-contained. The live render criterion (E2E-3) was proven at the end of the branch — both a datasource and a rendering dashboard workbook were published to Cloud. Gate is green: 87 TS + 114 Python = 201 tests.

What's in this PR

1. Prompt-driven authoring (3 new MCP tools, 11 → 14)

Additive — no existing tool is removed or renamed (src/index.ts registers all 14 tools).

• create_datasource_from_file — multi-format ingest (csv/json/jsonl/xlsx/parquet) → .hyper/.tdsx → publish; returns {datasourceLuid, contentUrl, url} plus the real column schema read back from the .hyper (src/tools/createDatasourceFromFile.ts, sidecar/hyper_builder.py).
• design_dashboard — a deterministic, stateless BI planner (src/planner/*: field inference → marks → audience clamp → plan) with 4 modes (autonomous / interview / interview_followup / directed) and 4 audiences; side-effect-free, returns a DashboardPlan or clarifying questions (src/tools/designDashboard.ts).
• build_from_plan — turns a plan into a real <dashboard>/<zones> workbook and publishes it; returns {workbookLuid, url, datasourceLuid?} (src/tools/buildFromPlan.ts).
• Two new FastAPI sidecar routes: POST /datasource/from-file and POST /workbook/dashboard.

2. Official-ecosystem alignment (incl. the TWB XSD gate)

• Official TWB XSD fidelity gate. The genuine 7,586-line twb_2026.1.0.xsd (vendored from tableau/tableau-document-schemas at a pinned SHA, with W3C xml.xsd and PROVENANCE.md) now validates the sidecar's actual generated output (build_twb_xml, build_embedded_twb_xml). The gate was proven non-no-op: injecting a bogus element makes validate() return False. Parser is XXE/billion-laughs hardened (resolve_entities=False, no_network=True, load_dtd=False, huge_tree=False) with a behavioral regression-guard test. lxml is dev-only — never imported at runtime (sidecar/tests/test_twb_schema_validation.py, sidecar/tests/schemas/).
• Schema-valid TWB output. Fixed real structural defects so generated workbooks pass the XSD (<simple-id>, <style>, <aggregation>, populated <cards>/<viewpoints>, dashboards-before-windows ordering).
• Chunk boundary aligned to official TSC. selectPublishStrategy switched > limit → >= limit, so an exactly-64-MiB file now takes the chunked path (src/restClient.ts; boundary oracle in tests/restClient.test.ts). Chunk size was already 64 MiB on main — unchanged.
• Repositioning + 14-tool docs across README.md, docs/relationship_to_official.md, docs/tool_reference.md, docs/architecture.md, and a new docs/adr/0005-deterministic-stateless-planner.md.

3. Sidecar port resilience (f623355)

sidecarPort is now optional; when SIDECAR_PORT is unset the TS layer auto-selects a free ephemeral port (with a bounded retry on the bind race) instead of always binding 8899. This fixes a startup crash where a stale sidecar holding 8899 bricked the run ([Errno 48] address already in use). Proven live: the sidecar started while 8899 was still occupied. The token still binds 127.0.0.1 only and is passed via env, not argv (src/config.ts, src/sidecar.ts).

4. Dashboard render via embedded extract (the render keystone)

Dashboards published by build_from_plan originally returned Tableau 400011 "Dashboard references sheet 'Sheet 1' which has no visual representation." Diagnosed (against 7 real reference workbooks + an isolating worksheet-only publish) as a layered cause; each fix verified live:

1. Embed the extract. The workbook now embeds the .hyper via a federated connection (a self-contained .twbx) instead of referencing the published datasource via sqlproxy, which Tableau Cloud would not render. The governed published datasource is still created separately.
2. Per-sheet <viewpoints> (the root cause). The dashboard window emitted an empty <viewpoints/>. A "viewpoint" is Tableau's term for a sheet's visual on a dashboard, so an empty list literally means no visual representation. Now one <viewpoint name="…"><zoom type="entire-view"/></viewpoint> is emitted per sheet in both builders.
3. layout-flow zones. Worksheet zones nest in a type-v2="layout-flow" container (each with <layout-cache>), with a <style> child on <dashboard>.
4. Real-column binding + populated <cards> so worksheets bind to the datasource's actual columns.

The shipped tool (build_from_plan), not just the demo, threads the .hyper into the embedded build; the no-embeddable-extract paths (pre-published datasourceLuid only, or a SQL query) fail loudly with an actionable error.

Breaking / behavior changes

• build_from_plan requires datasourceSpec.filePath. A plan with only datasourceLuid, or a sql/query spec, is rejected at runtime — the embedded-extract render path needs a local file to embed. New-tool behavior (no prior consumer), but note a schema/runtime gap: DatasourceSpecSchema permits a sql-only spec and datasourceSpec is .optional(), so a schema-valid plan can still be rejected by the runtime guard. Also: design_dashboard does not emit datasourceSpec, so its plan must be augmented with datasourceSpec.filePath before calling build_from_plan (documented in README.md).
• Sidecar default port now auto-selected. Anyone relying on the sidecar always being on 8899 (health probes, firewall rules) must now pin SIDECAR_PORT=8899 explicitly. Documented in .env.example.
• engines.node bumped >=20 → >=22.7.5 to match the official Tableau server's Node floor. Installs on Node 20/21 will warn/fail under engine-strict. No Node-22.7.5-only API is actually required — the floor is for ecosystem alignment.
• Chunk strategy boundary > → >= (exactly 64 MiB → chunked). Only the exactly-64-MiB case changes; <64 MiB stays single, >64 MiB stays chunked.
• Dashboard workbooks now embed the data (.twbx with the full .hyper, bounded by the 500 MB / 1M-row ingest caps) rather than referencing a published datasource. Expected for standalone rendering; a data-residency consideration worth noting.
• url value semantics (shared publish path). publish() now returns the server-provided webpageUrl as url, falling back to the constructed LUID URL only when absent (previously url was always the constructed #/site/... form). Output shape is unchanged for publish_datasource/publish_workbook; only the url content may differ for consumers that pattern-matched the old format. publishWorkbook also now sends skipConnectionCheck=true, and getDatasource() now throws when no contentUrl resolves (was best-effort). These are internal/library-level behavior changes.

Testing

• Full gate green: 87 TS + 114 Python = 201 tests. No focused/skipped tests; ESLint exit 0 and ruff --select F clean; npm audit --omit=dev → 0 vulnerabilities.
• Official-XSD gate validates real production output and empirically rejects malformed XML (mutant probe confirmed).
• Planner is a real priority-ordered engine exercised by behavioral tests (tests/planner.test.ts); build_from_plan E2E threads hyperPath into the dashboard build and asserts the LUID-only and SQL branches fail loud without building.
• Embedded structure asserted at unit level (federated present, sqlproxy absent, Data/*.hyper in zip, [federated.*] refs, metadata-records, schema validity) and end-to-end via a real HTTP POST /workbook/dashboard round-trip (sidecar/tests/test_twb_embedded.py, test_server_new_routes.py).
• Secret handling covered: PAT and sidecar token never logged or passed via argv; tests/secrets.test.ts green; mcp-smoke.ts asserts the PAT never appears in tool output.
• E2E-3 proven live (2026-06-25): npm run demo:dashboard published a datasource (…/datasources/25936416) and a rendering dashboard workbook (…/workbooks/2420435) to site sebaustin.

Non-blocking follow-ups

Doc-accuracy drift (no confirmed code blocker; gate green and E2E-3 proven):

• HIGH — DEPLOYMENT.md registered-tools table lists 6 tool names that do not exist (list_datasources, list_workbooks, set_datasource_permissions, set_workbook_permissions) and omits the 4 real ones (list_content, refresh_datasource, delete_content, set_permissions). Fix the table before relying on it. (spun off as a follow-up task)
• HIGH — CODEBASE.md still describes the pre-feature 11-tool baseline (file map omits src/planner/, the 3 new tools, and the new routes; "Tech Debt" lists already-shipped items feat(interop): chain off the official server's metadata to auto-suggest workbook sheets #3/feat: prompt-driven authoring (3 tools + deterministic planner), official-ecosystem alignment, sidecar port resilience, and renderable embedded-extract dashboards #4 as open).
• HIGH — ACCEPTANCE.md E2E-1 row documents a "no-datasource-spec → 0× publishDatasource" criterion that the shipped code and current test contradict (code now requires datasourceSpec.filePath); same stale criterion in REQUIREMENTS.md / feature PLAN.md. (spun off as a follow-up task)
• MEDIUM — ACCEPTANCE.md E2E-3 criteria table still shows E2E-3 "⏳ gated / 16/17" while the appended bottom section declares it CLOSED (18/18, 201 tests). Reconcile the table.
• MEDIUM — ACCEPTANCE.md DB-1 documents an obsolete <zone type="worksheet"> structure; shipped zones are identified by name alone with no type (the deliberate 400011 fix; architecture.md is correct).
• LOW — Node-floor / test-count / FastAPI-version drift in DEPLOYMENT.md ("Node 22+", "75 TS + 71 Python", FastAPI 0.115.6) and CODEBASE.md (Node ≥20, 28/18 tests) vs the real ≥22.7.5 / 201 tests / FastAPI 0.121.0.
• LOW — twb_builder docstrings claim default output is "byte-identical to pre-feature" (it isn't — the starter worksheet/window structure was re-baselined this branch); module docstring says <active id="1"/> but code/tests use id="-1".
• LOW — TS↔server dashboards payload mismatch: the TS client sends a dashboards[].sheetTitles field the server's pydantic model drops (server always derives its own). Dead input today; harmless because every sheet title is passed anyway.
• LOW — viewpoint test coverage gap: the per-sheet <viewpoints> population that fixes 400011 is the render keystone, but the only guarding test asserts <viewpoints> is present (not populated per sheet) and the embedded path has zero viewpoint assertions — an empty-<viewpoints/> regression would pass the whole 201-test gate. Production code populates them correctly today, but nothing locks it in. Worth a "one <viewpoint> per sheet name" assertion on the embedded builder.
• INFO — repo/security hygiene clean: reference .twb/.twbx gitignored and untracked; npm pack ships only dist + 4 sidecar .py + lockfiles (no secrets/extracts/tests/XSDs); arbitrary local-file read via filePath is by-design and loopback+token-guarded; no debug/scratch leftovers.

🤖 Generated with Claude Code