diff --git a/.env.example b/.env.example index b783ff3..9bc7c3d 100644 --- a/.env.example +++ b/.env.example @@ -18,5 +18,18 @@ PAT_VALUE=replace-me-never-commit-the-real-value # Defaults to an automatically-chosen free port; set only to pin a specific port. # SIDECAR_PORT=8899 -# Optional. Target project for `npm run demo` (must exist; not the Default project). +# Optional. Localhost host for the Python authoring sidecar. Defaults to 127.0.0.1. +# SIDECAR_HOST=127.0.0.1 + +# Optional. Target project for `npm run demo` / `npm run demo:dashboard` / +# `npm run demo:superstore` (must exist; not the Default project). # DEMO_PROJECT=Sales + +# Optional. Named persona (from brand.yaml) applied by `npm run demo:superstore`. +# Equivalent to passing `--persona ` on the command line. +# PERSONA=ceo + +# NOTE on create_live_datasource (Snowflake/Presto): database credentials are NOT read from +# environment variables by this server. They are passed as tool-call parameters +# (credentials.username / credentials.password) by the calling agent, which should source them +# from its own environment — never hardcode them in a prompt, a saved plan, or this file. diff --git a/.gitignore b/.gitignore index 6b3b93d..432faa0 100644 --- a/.gitignore +++ b/.gitignore @@ -27,6 +27,9 @@ __pycache__/ tmp/ *.log +# Generated cron/launchd artifacts (contain local file paths — never commit) +scripts/cron/ + # OS / editor .DS_Store .idea/ diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 9b6974c..09ad5a4 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -320,3 +320,151 @@ Gate green: 87 TS + 114 Python = 201 tests. Also fixed this session: the stale-p Several throwaway diagnostic workbooks were published during debugging ("Diag Single Worksheet", "Diag Named Dashboard") and one datasource per demo run. The keeper artifacts are the demo datasource + workbook above; the `Diag*` workbooks can be deleted from the site. + +--- + +# E2E — Exec-Dashboards Phase 1 CLOSED: rich dashboard live on Cloud (2026-06-29) + +`npm run demo:superstore` published the full Phase-1 rich vertical slice to the dev site, +accepted by Tableau Cloud on the first attempt after the E0 fixes: + +| Artifact | URL | +|---|---| +| Datasource (68 cols, UTF-16/TSV auto-sniffed + coerced) | https://10ax.online.tableau.com/#/site/sebaustin/datasources/27108116 | +| Exec dashboard workbook (embedded extract) | https://10ax.online.tableau.com/#/site/sebaustin/workbooks/2519210 | + +**What renders:** KPI band — Sales (Δ Sales Difference, up_good), Profit, Quantity, Discount +(down_good) — above "Sales by Category" (bar, colored by Segment) and "Sales by State" +(filled US map colored by Sales), with dashboard title and the kpi_band_over_charts layout. + +**E0 fixes that landed first (offline verification caught both before any live publish):** +1. `c2e497a` — builder nested keys aligned to `model_dump()` snake_case + an integration + test over the genuine camelCase-JSON → Pydantic → builder path (TestClient 200). +2. `f17d664` — question-aware + canonical ranking (measures/dims/geo): the alphabetical + 68-column file had made the plan key on "Days to Ship" and drop Sales; now the KPI band + leads with question-mentioned measures and the map honors "by state". + +Gate at close: **241 TS + 258 Python = 499 tests.** The propose→confirm proposal shown in the +demo output is the exact contract an agent presents before build_from_plan publishes. + +--- + +# E1 — Branding system live (2026-06-29) + +`npm run demo:superstore -- --persona ceo` resolved the **ceo** persona from `brand.yaml` +(→ exec base, "My Company" brand), and published the branded dashboard: + +| Artifact | URL | +|---|---| +| Datasource | https://10ax.online.tableau.com/#/site/sebaustin/datasources/27109700 | +| Branded exec workbook | https://10ax.online.tableau.com/#/site/sebaustin/workbooks/2519210 | + +The workbook carries the brand: ``, brand-typography title/subtitle runs, and `default-format` on measures (currency/ +percent/number classified by name). The proposal states the persona + brand applied. + +Shipped in E1: `brand.yaml` kit (palette/typography/formats/rules + 5 named personas), +`src/branding` (zod loader + resolvePersona), `validate_brand` tool (15 tools), persona support +in `design_dashboard`, brand application in the builder (all XSD-valid; no-brand output +byte-identical), and the tool-vs-demo threading guard (`659181b` — build_from_plan now threads +kind/color/kpi/scatter/geo + title/layoutGrammar). Gate: **277 TS + 290 Python = 567 tests.** + +--- + +# E2 — Connectivity + automated refresh (2026-07-17) + +Shipped (23 tools, gate 433 TS + 308 Python = 741 tests): +- REST hardening: bounded idempotency-aware retry (429/5xx, Retry-After honored), typed + TableauApiError; VDS field-metadata client + `get_datasource_fields`. +- Scheduling/automation: `schedule_refresh` (Cloud per-task frequency XML + honest Bridge + caveat), `list/delete_refresh_schedule`, `create/list/delete_webhook` (HTTPS, admin errors). +- Live connections: `create_live_datasource` — Snowflake/Presto federated `.tds` (no creds in + the file), publish with `` (password never logged, + asserted); key-pair auth cleanly rejected; Presto defaults to Bridge. +- Local-file design-around: `refresh:local` + `cron:generate` (crontab + launchd templates, + generated never installed). + +**Live proof (local-file path):** `npm run refresh:local` re-ingested the Superstore file and +republished (overwrite) in 4.7s → https://10ax.online.tableau.com/#/site/sebaustin/datasources/27118502 +`cron:generate --daily 06:30` emits the install-ready template. + +**Pending live proofs (need user resources):** Snowflake scheduled server-side refresh (needs +account creds); the four VERIFY-LIVE schedule-XML details. + +--- + +# E4 — Stories live (2026-07-17) + +A generated Tableau STORY workbook published and accepted by Tableau Cloud: + +| Artifact | URL | +|---|---| +| Story workbook ("Superstore Executive Story (demo)") | https://10ax.online.tableau.com/#/site/sebaustin/workbooks/2523370 | + +6-point persona-toned arc over the exec sheets (headline KPIs → Profit → Quantity → Discount → +Sales by Category → Sales by State). Structure: `` + paired +flipboard-nav/flipboard + ``, mirroring the real reference +workbook and XSD-validated. En route, a latent XSD violation was found and fixed (one shared +`` container instead of per-call sibling wrappers). + +# E3 — Pulse (code-complete; live proof blocked on fixture) + +All 4 Pulse tools shipped (27 tools) with the exact researched payload + VDS pre-flight. Live +attempt: datasource republished with REAL date columns (VDS: `Order Date: DATETIME` — the date- +coercion fix proven live), but `POST /api/-/pulse/definitions` returns a bare 400: the +`basic_specification` internals are specified-by-example only (Tableau's own utilities repo +clones, never constructs, this block). Pending: user creates ONE metric in the Pulse UI → we GET +it, lock the fixture, correct the client, re-prove live. + +Gate at this point: **521 TS + 338 Python = 859 tests.** + +--- + +## E5 — Hardening, docs, acceptance (vibe-BI expansion close-out) + +**Date:** 2026-07-17 · **Branch:** `feat/exec-dashboards` · **Verdict: SOLID (solution-verifier, rubric 100/100)** + +### Final gate (independently re-run by the verifier) + +| Check | Result | +|---|---| +| `npm run build` (tsc) | clean | +| `npm run lint` (eslint) | clean | +| `vitest run` | **525 passed** (23 files) | +| `ruff check` / `mypy --strict` | clean | +| `pytest -q` | **338 passed** | +| **Total** | **863 / 863 · exit 0** | +| `npm audit --omit=dev` | 0 vulnerabilities | +| Focused/skipped tests | none (grep-verified) | + +### Security close-out + +- STRIDE expansion audit VB-01..VB-12 recorded in `SECURITY.md`: **0 CRITICAL · 0 HIGH · 1 MEDIUM**. +- The single MEDIUM (**VB-02**, shell injection via interpolated crontab lines) was **fixed in-branch** (`scripts/cronTemplates.ts` POSIX `shellQuote`) and locked with four adversarial tests (`$(id)`, quote-breakout, embedded single quotes, hostile repoRoot). +- Credentials discipline verified on both the success path and a 400-error path (`tests/secrets.test.ts`); live-connection `.tds` files provably never contain credentials (`test_tds_builder_live.py`). + +### Deliverables confirmed real (verifier evidence, code + tests) + +- **E1 branding**: brand.yaml → ``, branded runs, `default-format`; no-brand output byte-identical (regression-proof). Persona resolution wired into `design_dashboard`; Few rules enforced deterministically (chartDeny, KPI caps, no-pie). +- **E2 connectivity**: bounded idempotency-aware retry (chunk-append PUT never retried), Cloud extract-refresh schedules, fail-closed HTTPS webhooks, live Snowflake/Presto `.tds` with key-pair auth cleanly rejected, local-file cron design-around (VB-02-hardened). +- **E3 Pulse**: 4 tools registered, wire payload locked by deep-equal test, VDS pre-flight hard-fails on missing measure/date dimension. +- **E4 stories**: storyboard/flipboard XML XSD-validated in 6 variants, captured-sheet fail-loud at both layers, single shared `` container regression-guarded. +- **Docs**: 27 tools consistent across README / tool_reference / CODEBASE / DEPLOYMENT / architecture; ADRs 0006–0012; runbook. Spot-checked tool entries match zod schemas exactly. + +### Live proofs (Tableau Cloud dev site) — status ledger + +| Phase | Proof | Status | +|---|---|---| +| E0 | Superstore exec dashboard (embedded extract) | ✅ recorded above | +| E1 | Branded persona=ceo rebuild | ✅ recorded above | +| E2 (local path) | Local-file re-publish refresh chain | ✅ recorded above | +| E4 | Published story (storyboard) | ✅ recorded above | +| E2 (Snowflake) | Scheduled Snowflake refresh + VERIFY-LIVE schedule tokens | ⏳ **pending user**: Snowflake credentials in `.env` | +| E3 (Pulse) | Live definition + metric creation | ⏳ **pending user**: one UI-created metric on "Superstore (exec demo)" → GET fixture → correct `basic_specification` | + +No live capability is claimed beyond the ✅ rows; the two ⏳ rows are code-complete and unblocked by a ~2-minute user action each. + +### Non-blocking observations (verifier, no fix required) + +1. The E3 phase-close record above states the gate at its point-in-time count (521 TS / 859 total); the branch-final count is 525 / 863. Historical snapshot retained as-is. +2. Webhook HTTPS fail-closed is enforced by exact-prefix check; an explicit uppercase-scheme/whitespace unit test would be a nicety. diff --git a/CODEBASE.md b/CODEBASE.md index 0e35fc5..433b3d1 100644 --- a/CODEBASE.md +++ b/CODEBASE.md @@ -2,12 +2,23 @@ ## Overview -`tableau-mcp-publish` is the **write side** of Tableau MCP. It exposes 14 MCP tools over stdio that let an AI agent turn a SQL query, CSV file, or inline records into a fully governed, published Tableau datasource and starter workbook on Tableau Cloud — in a single tool call. It is architecturally complementary to `tableau/tableau-mcp`, which covers reading, querying, Desktop-local workbook editing, and admin-gated content lifecycle. +`tableau-mcp-publish` is the **write side** of Tableau MCP — **vibe-BI on Tableau Cloud**. It +exposes **27** MCP tools over stdio that let an AI agent go from a prompt to governed datasources +(files, queries, or live Snowflake/Presto connections), branded persona-aware dashboards, Tableau +Stories, Pulse metrics, and scheduled/cron-automated refresh — in a handful of tool calls. It is +architecturally complementary to `tableau/tableau-mcp`, which covers reading, querying, +Desktop-local workbook editing, and admin-gated content lifecycle. The system has two layers: -1. **TypeScript MCP server** (`src/`) — handles MCP protocol, Tableau REST API authentication, project resolution, and publish (single-request or chunked). Spawns the Python sidecar on startup. -2. **Python FastAPI sidecar** (`sidecar/`) — does all binary file authoring: Hyper extract creation (via pantab), `.tdsx` packaging (hand-built TDS XML + zip), and `.twbx` workbook XML generation. Lives at `http://127.0.0.1:8899`, bound loopback-only, guarded by a per-spawn random token. +1. **TypeScript MCP server** (`src/`) — MCP protocol, Tableau REST/VDS/Pulse API access (with + bounded retry/backoff), project resolution, publish (single-request or chunked), the + deterministic BI planner, and brand-kit/persona resolution. Spawns the Python sidecar on + startup. +2. **Python FastAPI sidecar** (`sidecar/`) — does all binary file authoring: Hyper extract creation + (via pantab), `.tdsx`/`.tds` packaging (hand-built XML + zip), and `.twbx` workbook XML + generation (including branded output and Tableau Stories). Bound loopback-only, guarded by a + per-spawn random token. --- @@ -21,6 +32,7 @@ The system has two layers: | MCP SDK | `@modelcontextprotocol/sdk` | 1.29.0 | | HTTP client | `undici` | 7.28.0 | | Schema validation | `zod` | 3.25.76 | +| YAML parsing | `yaml` | 2.9.0 — added for `brand.yaml` (Phase E1) | | Test runner | Vitest | 2.1.8 | | Linter | ESLint 9 + `typescript-eslint` | 9.17.0 / 8.18.2 | | Python sidecar | Python | 3.12.x (uv-pinned; 3.12/3.13 both tested in CI) | @@ -30,13 +42,15 @@ The system has two layers: | Hyper extract | tableauhyperapi | 0.0.21408 | | DataFrame bridge | pantab | 5.2.0 | | DataFrames | pandas | 2.2.3 | +| Excel read | openpyxl | 3.1.5 | +| Parquet read | pyarrow | (transitive via pantab) | | Python linter | ruff | 0.8.4 | | Python type check | mypy | 1.13.0 (strict) | | Python test | pytest | 8.3.4 | | Package manager (TS) | npm | — | | Package manager (Py) | uv | 0.11.18+ | -Optional Python extras (`uv sync --extra connectors`): `snowflake-connector-python`, `psycopg[binary]`, `sqlalchemy` — not required for CI or the authoring path. +Optional Python extras (`uv sync --extra connectors`): `snowflake-connector-python`, `psycopg[binary]`, `sqlalchemy` — used only by the `connection: {type: snowflake|postgres}` branch of `create_datasource_from_query`; not required for CI, the file-ingest path, or `create_live_datasource` (which builds a live `.tds` without a driver — Tableau's own connector handles the live query at render time, not this process). --- @@ -46,22 +60,31 @@ Optional Python extras (`uv sync --extra connectors`): `snowflake-connector-pyth # TypeScript npm install npm run build # tsc -> dist/ -npm run lint # eslint . -npm run typecheck # tsc --noEmit -npm test # vitest run (87 tests) +npm run lint # eslint . +npm run typecheck # tsc --noEmit +npm test # vitest run (525 tests) # Python sidecar cd sidecar -uv sync --all-extras # create .venv with dev + connectors -uv run ruff check . # lint -uv run mypy --strict . # type check -uv run pytest -q # 114 tests +uv sync --all-extras # create .venv with dev + connectors +uv run ruff check . # lint +uv run mypy --strict . # type check +uv run pytest -q # 338 tests # Full CI gate (equivalent to GitHub Actions) make ci # build + lint + test + sidecar-lint + sidecar-typecheck + sidecar-test ``` -The `make ci` target does **not** run `npm run typecheck` separately; the `build` step already runs `tsc` (which is a full type + emit check). The `lint` step does not run `eslint` with `--max-warnings 0`; it exits non-zero only on errors. +Real counts as of this branch (`feat/exec-dashboards`, re-run directly for this doc pass): +**525 TypeScript tests across 23 files** (`npm test`) + **338 Python tests across 20 files** +(`cd sidecar && uv run pytest -q`) = **863 tests total**. `ACCEPTANCE.md`'s last recorded gate +(859) predates one small follow-up security-hardening commit (VB-02, shell-quoting the cron +templates) that added 4 TypeScript tests — see `git log` for `fix(security): shell-quote cron-line +interpolations`. + +The `make ci` target does **not** run `npm run typecheck` separately; the `build` step already runs +`tsc` (which is a full type + emit check). The `lint` step does not run `eslint` with +`--max-warnings 0`; it exits non-zero only on errors. --- @@ -70,17 +93,32 @@ The `make ci` target does **not** run `npm run typecheck` separately; the `build ``` tableau-mcp-publish/ ├── src/ -│ ├── index.ts # Entry point — registers all 14 tools, signs in, spawns sidecar, starts stdio transport +│ ├── index.ts # Entry point — registers all 27 tools, signs in, spawns sidecar, starts stdio transport │ ├── config.ts # Zod schema for env-based config (SERVER, SITE_NAME, PAT_NAME, PAT_VALUE…) -│ ├── restClient.ts # TableauRestClient: signIn/signOut, publish (single + chunked), CRUD, permissions +│ ├── restClient.ts # TableauRestClient: signIn/signOut, publish (single + chunked), CRUD, permissions, +│ │ # delegates schedule/webhook/Pulse/VDS request-building to rest/ │ ├── sidecar.ts # AuthoringSidecar: spawns uv/uvicorn, health-polls, posts to /datasource/*, /workbook/* -│ ├── planner/ # Deterministic BI planner (no LLM calls) -│ │ ├── schema.ts # DashboardPlan Zod types -│ │ ├── fields.ts # Field-role inference from column hints -│ │ ├── marks.ts # Mark-type selection logic -│ │ ├── audience.ts # Audience → canvas size + sheet-count clamps -│ │ ├── questions.ts # Interview-mode clarifying-question generation -│ │ └── plan.ts # planDashboard() entry point +│ ├── rest/ # REST/VDS/Pulse module family (Phase E2/E3) — see docs/architecture.md +│ │ ├── xml.ts # xmlEscape / asArray shared helpers +│ │ ├── errors.ts # TableauApiError, parseTableauErrorBody, parseRetryAfterMs +│ │ ├── retry.ts # withRetry() — shared bounded exponential-backoff loop +│ │ ├── vds.ts # readDatasourceMetadata() — VizQL Data Service field metadata +│ │ ├── schedules.ts # extract-refresh task XML + frequency/interval validation +│ │ ├── webhooks.ts # webhook create/list/delete XML + HTTPS-only validation +│ │ ├── credentials.ts # XML fragment for live datasources +│ │ └── pulse.ts # Pulse definition/metric JSON bodies + VDS-backed pre-flight check +│ ├── branding/ # Brand-kit loading + persona resolution (Phase E1) +│ │ ├── schema.ts # Zod schema for brand.yaml (every field defaulted) +│ │ ├── load.ts # loadBrand() / resolvePersona() — the ONLY I/O in this layer +│ │ └── builderBrand.ts # Pure projection: BrandFile → sidecar's flat BuilderBrand wire shape +│ ├── planner/ # Deterministic BI planner (no LLM calls) +│ │ ├── schema.ts # DashboardPlan / ClarifyingQuestions / DashboardProposal Zod types +│ │ ├── fields.ts # Field-role inference from column hints +│ │ ├── marks.ts # Mark-type selection, KPI-strip/color/scatter/geo encoding, Few/Tufte hints +│ │ ├── audience.ts # Audience + persona-override clamp (chartDeny, kpiEmphasis, maxSheets) +│ │ ├── questions.ts # Interview-mode clarifying-question generation +│ │ ├── plan.ts # generatePlan()/generateInterview() — wires the stages, builds storyArc +│ │ └── proposal.ts # buildProposal() — DashboardPlan → human-readable DashboardProposal │ └── tools/ │ ├── context.ts # ToolContext interface + toolResult() helper │ ├── projects.ts # list_projects, create_project @@ -89,42 +127,64 @@ tableau-mcp-publish/ │ ├── createDatasourceFromQuery.ts # create_datasource_from_query │ ├── createDatasourceFromTable.ts # create_datasource_from_table │ ├── createDatasourceFromFile.ts # create_datasource_from_file (csv/json/jsonl/xlsx/parquet) +│ ├── createLiveDatasource.ts # create_live_datasource (Snowflake/Presto, no extract) │ ├── createStarterWorkbook.ts # create_starter_workbook -│ ├── designDashboard.ts # design_dashboard (autonomous/interview/interview_followup/directed) -│ ├── buildFromPlan.ts # build_from_plan (DashboardPlan → embedded .twbx → publish) +│ ├── designDashboard.ts # design_dashboard (propose→confirm; persona-aware) +│ ├── buildFromPlan.ts # build_from_plan (DashboardPlan → embedded .twbx + story → publish) +│ ├── validateBrand.ts # validate_brand +│ ├── getDatasourceFields.ts # get_datasource_fields (VDS) +│ ├── schedules.ts # schedule_refresh, list_refresh_schedules, delete_refresh_schedule +│ ├── webhooks.ts # create_webhook, list_webhooks, delete_webhook +│ ├── pulse.ts # create_pulse_definition, list_pulse_definitions, create_pulse_metric, delete_pulse_definition │ ├── publishDatasource.ts # publish_datasource (pre-built file) │ └── publishWorkbook.ts # publish_workbook (pre-built file) -├── tests/ -│ ├── restClient.test.ts # Unit: chunk math, strategy boundary, signIn, publish, resolveProjectId (11 tests) -│ ├── secrets.test.ts # PAT-never-logged assertions (3 tests) -│ ├── sidecar.test.ts # AuthoringSidecar: startup, health, build calls (7 tests) -│ ├── sidecar-columns.test.ts # Column-schema pass-through from /datasource/from-file (3 tests) -│ ├── planner.test.ts # planDashboard(): autonomous/interview/directed × audiences (29 tests) -│ └── tools.test.ts # Integration: all 14 tools via FakeServer + mock ctx (34 tests) +├── tests/ # 23 files, 525 tests (vitest) — one file per src/ module, roughly +│ ├── restClient.test.ts, restRetry.test.ts, retry.test.ts # publish strategy, retry policy +│ ├── secrets.test.ts, credentials.test.ts # PAT/password never logged +│ ├── sidecar.test.ts, sidecar-columns.test.ts # AuthoringSidecar wiring +│ ├── vds.test.ts, schedules.test.ts, webhooks.test.ts, pulse.test.ts # rest/ module tests +│ ├── branding.test.ts, builderBrand.test.ts # brand.yaml load + projection +│ ├── planner.test.ts, planner-slice4.test.ts, planner-slice5.test.ts, +│ │ planner-slice6.test.ts, planner-storyArc.test.ts # planner pipeline + proposal + storyArc +│ ├── schema-growth.test.ts # DashboardPlan schema backward-compat +│ ├── cronTemplates.test.ts, generateCron.test.ts, refreshLocalArgs.test.ts # local-file automation +│ └── tools.test.ts # integration: all 27 tools registered +├── scripts/ +│ ├── demo.ts # npm run demo — datasource + starter workbook +│ ├── demo-dashboard.ts # npm run demo:dashboard — propose→build pipeline +│ ├── demo-superstore.ts # npm run demo:superstore — rich exec dashboard + optional persona brand +│ ├── refresh-local.ts # npm run refresh:local — re-ingest + republish a local-file datasource +│ ├── generate-cron.ts # npm run cron:generate — emit crontab/launchd artifacts (never installs) +│ ├── cronTemplates.ts # pure template builders for generate-cron.ts (no I/O; shell-quoted) +│ ├── mcp-smoke.ts # npm run test:mcp-smoke — stdio handshake + tool listing +│ └── verify-setup.ts # npm run verify-setup — pre-flight config/sign-in/sidecar check ├── sidecar/ │ ├── server.py # FastAPI app — token guard, /health, /datasource/from-query, -│ │ # /datasource/from-table, /datasource/from-file, /workbook/starter, -│ │ # /workbook/dashboard -│ ├── hyper_builder.py # DataFrame/SQL/CSV/JSON/XLSX/Parquet → .hyper extract (pantab + tableauhyperapi) -│ ├── tds_builder.py # .hyper → .tdsx (hand-built TDS XML + zip) +│ │ # /datasource/from-table, /datasource/from-file, /datasource/live, +│ │ # /workbook/starter, /workbook/dashboard (7 routes) +│ ├── hyper_builder.py # DataFrame/SQL/CSV/JSON/XLSX/Parquet → .hyper extract (pantab + tableauhyperapi), +│ │ # numeric/date-string coercion +│ ├── tds_builder.py # .hyper → .tdsx (hand-built TDS XML + zip); build_live_tds() for live connections │ ├── twb_builder.py # build_twb_xml() / build_starter_twbx() / build_embedded_twb_xml() / -│ │ # build_embedded_twbx() — worksheets + optional block +│ │ # build_embedded_twbx() — worksheets + block + _build_story() │ ├── pyproject.toml # uv project config, ruff/mypy/pytest settings -│ └── tests/ -│ ├── test_hyper_builder.py # 12 tests: round-trip, column roles, CSV, max_rows, records_to_df -│ ├── test_hyper_builder_formats.py# 16 tests: csv/json/jsonl/xlsx/parquet format round-trips, byte cap -│ ├── test_tds_builder.py # 3 tests: zip structure, dbname path, column roles -│ ├── test_twb_builder.py # 13 tests: datasource ref, worksheets, mark classes, A2 structural pins -│ ├── test_twb_dashboard.py # 21 tests: zone count/names, geometry invariants, canvas size, regression -│ ├── test_twb_embedded.py # 16 tests: federated datasource, field refs, zip structure, XSD gate -│ ├── test_twb_schema_validation.py# 4 tests: XSD-valid output + XXE-safe parser guard -│ ├── test_server.py # 5 tests: health, from-table, 400 guard, workbook starter, token guard -│ └── test_server_new_routes.py # 24 tests: /datasource/from-file (5 formats), /workbook/dashboard -├── Makefile # CI gate: build lint test sidecar-lint sidecar-typecheck sidecar-test +│ └── tests/ # 20 files, 338 tests +│ ├── test_hyper_builder.py, test_hyper_builder_formats.py # extract round-trips, format coverage +│ ├── test_tds_builder.py, test_tds_builder_live.py # .tdsx / live .tds packaging +│ ├── test_twb_builder.py, test_twb_dashboard.py, test_twb_dashboard_layout.py, +│ │ test_twb_embedded.py, test_twb_encodings.py, test_twb_kpi_tile.py, +│ │ test_twb_map_filled.py, test_twb_scatter.py, test_twb_branding.py, test_twb_story.py +│ │ # worksheet/dashboard/story XML structure +│ ├── test_twb_schema_validation.py # official TWB XSD gate + XXE guard +│ ├── test_schema_growth.py # sidecar Pydantic models backward-compat +│ └── test_server.py, test_server_new_routes.py, +│ test_server_live_datasource.py, test_server_rich_dashboard.py # FastAPI route integration +├── brand.yaml # Brand kit: palette/typography/formats/rules/personas (Phase E1) +├── Makefile # CI gate: build lint test sidecar-lint sidecar-typecheck sidecar-test ├── package.json -├── tsconfig.json # strict, NodeNext, rootDir=src, outDir=dist -├── eslint.config.js # ignores: dist/, node_modules/, sidecar/, coverage/ (NOT .cursor/) -└── vitest.config.ts # tests/**/*.test.ts, extensionAlias .js->.ts +├── tsconfig.json # strict, NodeNext, rootDir=src, outDir=dist +├── eslint.config.js # ignores: dist/, node_modules/, sidecar/, coverage/ (NOT .cursor/) +└── vitest.config.ts # tests/**/*.test.ts, extensionAlias .js->.ts ``` --- @@ -137,7 +197,7 @@ AI agent (Claude / Cursor / etc.) ▼ src/index.ts ──────────── McpServer (MCP SDK) │ │ - │ registers 14 tools │ + │ registers 27 tools │ ▼ │ ToolContext { config, rest, sidecar } │ │ @@ -145,27 +205,29 @@ AI agent (Claude / Cursor / etc.) │ TableauRest │ │ AuthoringSidecar │ │ Client │ │ uv run uvicorn │ │ (undici) │ │ server:app │ - │ │ │ :8899 loopback only │ - │ Tableau │ │ │ - │ REST API │ │ FastAPI routes: │ - │ v3.28 │ │ /health │ - │ │ │ /datasource/from-query│ - └─────────────┘ │ /datasource/from-table│ - │ /datasource/from-file│ - │ /workbook/starter │ - │ /workbook/dashboard │ - │ │ - │ hyper_builder.py │ - │ pandas + pantab │ - │ -> .hyper │ + │ + rest/*.ts │ │ loopback only, │ + │ (retry- │ │ per-spawn token │ + │ hardened) │ │ │ + │ │ │ FastAPI routes: │ + │ Tableau │ │ /health │ + │ REST API │ │ /datasource/from-query│ + │ v3.28 │ │ /datasource/from-table│ + │ + VDS │ │ /datasource/from-file│ + │ + Pulse │ │ /datasource/live │ + │ │ │ /workbook/starter │ + └─────────────┘ │ /workbook/dashboard │ │ │ - │ tds_builder.py │ - │ XML + zip │ - │ -> .tdsx │ + branding/load.ts ─┐ │ hyper_builder.py │ + (reads brand.yaml,│ │ pandas + pantab │ + only I/O in the │ │ -> .hyper (+ coercion)│ + branding layer) │ │ │ + │ │ │ tds_builder.py │ + ▼ └─▶│ XML + zip │ + planner/* (pure) ────┤ -> .tdsx / .tds │ │ │ │ twb_builder.py │ │ XML + zip │ - │ -> .twbx │ + │ -> .twbx (+ story) │ └──────────────────────┘ ``` @@ -173,10 +235,13 @@ AI agent (Claude / Cursor / etc.) - The sidecar is spawned with `stdio: ['ignore','ignore','pipe']` — its stdout never reaches the MCP channel. - The sidecar binds `127.0.0.1` only. A random 24-byte hex token is generated per spawn, injected as `SIDECAR_TOKEN` env, and required as `X-Sidecar-Token` on every request (constant-time `hmac.compare_digest`). - The Tableau PAT secret is sent only in the sign-in body, never logged (asserted in `tests/secrets.test.ts`). Config validation errors print the offending field path, never the value. +- Live-connection database credentials (`create_live_datasource`) are embedded only at publish time via an in-memory `` XML fragment — never written to the `.tds` file or logged (asserted in `tests/credentials.test.ts`). - `resolveProjectId` hard-refuses the `"Default"` project by name and empty names, preventing silent publishes to ungoverned space. **Publish strategy:** `selectPublishStrategy()` in `src/restClient.ts` — files `< 64 MiB` use a single `multipart/mixed` POST; files `>= 64 MiB` (incl. exactly 64 MiB) use the `fileUploads` chunked session (TSC-aligned). Mid-stream abort does not issue a finalize POST. +**Retry policy:** `withRetry()` (`src/rest/retry.ts`), shared by `restClient.ts`, `rest/vds.ts`, and `rest/pulse.ts`. Retries only 429/502/503/504, only for calls marked idempotent by their caller, with full-jitter exponential backoff (max 3 attempts, ~8s total budget by default) honoring an upstream `Retry-After` header. See `docs/architecture.md` and `docs/adr/0008-rest-retry-hardening.md`. + --- ## Alignment with official Tableau tooling @@ -187,23 +252,25 @@ two non-adoptions, so they are not re-evaluated on every review. ### Divergences from `tableau/server-client-python` (TSC) TSC is the canonical Python REST client for Tableau Server/Cloud. Our publish path aligns with its -semantics. The chunking boundary was aligned to TSC's `>=` in A3 (exact 64 MiB now takes the chunked -path); two deliberate differences remain: +semantics. | Aspect | TSC | `tableau-mcp-publish` | Rationale | |---|---|---|---| -| Chunking boundary | `file_size >= 64 MB` (exact 64 MB → chunked) | `file_size >= 64 MB` — **aligned to TSC (A3)** | Was previously `>` (exact 64 MB single); a single multipart request at exactly 64 MB exceeds the cap once boundary overhead is added, so `>=` is correct. | +| Chunking boundary | `file_size >= 64 MB` (exact 64 MB → chunked) | `file_size >= 64 MB` — **aligned to TSC** | A single multipart request at exactly 64 MB exceeds the cap once boundary overhead is added, so `>=` is correct. | | Chunk size | 50 MB per chunk | 64 MB per chunk | Larger chunks reduce round-trips; acceptable on Cloud. | | REST API version | Auto-negotiated (latest supported by the server) | Pinned to 3.28 | Predictability over auto-negotiation; update explicitly when new endpoints are needed. | | Abort / unfinalized session | Aborts unfinalized upload sessions | Does not issue a `finalize` POST on mid-stream abort | Behaviour matches TSC: an unfinalized session is automatically discarded by the server. | +| Retry policy | TSC does not retry by default | Bounded idempotency-aware retry (429/5xx only, GET/idempotent-POST only) | Cloud rate-limits aggressively under the tool surface's higher call volume (scheduling, webhooks, Pulse, VDS); see `docs/adr/0008-rest-retry-hardening.md`. | ### Official TWB XSD (`tableau/tableau-document-schemas`) `tableau/tableau-document-schemas` publishes `schemas/2026_1/twb_2026.1.0.xsd` — a W3C XSD that describes the `.twb` XML format, maintained by the official Tableau team as a machine-validatable -fidelity gate. The sidecar test suite vendors this schema and validates `build_twb_xml()` output -against it via `lxml`. This catches the class of "parses but won't render" defects that are -invisible to structural assertions about expected elements. +fidelity gate. The sidecar test suite vendors this schema and validates every builder output +(regular dashboards, embedded extracts, and Stories) against it via `lxml`. This catches the class +of "parses but won't render" defects that are invisible to structural assertions about expected +elements — it caught a real bug during Story implementation (two sibling `` wrappers; +see `docs/adr/0012-story-shared-dashboards-container.md`). ### Non-adoption: `tableau/document-api-python` @@ -211,8 +278,8 @@ invisible to structural assertions about expected elements. states it "doesn't support creating files from scratch"; `Workbook.__init__` only opens existing files; `_prepare_dashboards()` returns names only (no zone writer); worksheets are name stubs (`# TODO: A real worksheet object`). It cannot author the XML we need to emit. We hand-roll the -TWB XML in `sidecar/twb_builder.py`. No re-evaluation is needed unless the library gains -create-from-scratch capability. +TWB/TDS XML in `sidecar/twb_builder.py`/`sidecar/tds_builder.py`. No re-evaluation is needed unless +the library gains create-from-scratch capability. ### Non-adoption: `tableau/tableau-ui` @@ -230,103 +297,108 @@ as FUTURE-ONLY. - ESM-only; `.js` import extensions pointing at `.ts` source (NodeNext resolution). - All tools follow the same pattern: a single `registerXxx(server, ctx)` function in `src/tools/`, calling `server.registerTool(name, { title, description, inputSchema, outputSchema }, handlerFn)`. Input and output schemas are Zod objects. The handler calls `ctx.sidecar.*` and/or `ctx.rest.*`, then returns `toolResult(text, structuredContent)`. - `process.stderr.write(...)` is used for structured logging — `console.*` is never used (stdout is the MCP channel). -- Errors thrown from handlers propagate as MCP error responses. +- Errors thrown from handlers propagate as MCP error responses. REST/VDS/Pulse failures throw a typed `TableauApiError` (`src/rest/errors.ts`). +- Destructive tools follow a uniform `confirm: boolean` gate pattern (`delete_content`, `delete_refresh_schedule`, `delete_webhook`, `delete_pulse_definition`); permission-elevation follows a uniform `confirmElevated` pattern (`set_permissions`). **Python:** - All modules use `from __future__ import annotations`. -- Pydantic v2 models with `model_config = ConfigDict(populate_by_name=True)` and camelCase aliases for the JSON API boundary. +- Pydantic v2 models with `model_config = ConfigDict(populate_by_name=True)` and camelCase aliases for the JSON API boundary — but `.model_dump()` always yields snake_case field names, never the aliases ("the model_dump lesson," documented at each affected model in `server.py`). - Ruff line-length 100, target py312, rules E/F/I/UP/B/SIM. - mypy `--strict`, excludes `tests/`. - Output files go to `tempfile.gettempdir()/tableau-mcp-publish/.`. -**Git/commit conventions:** Conventional commits (`feat`, `fix`, `docs`, `ci`, `chore`, `perf`). Scope tags used, e.g. `feat(demo)`, `fix(pkg)`, `docs:`. Co-authored attribution in commit footers. +**Git/commit conventions:** Conventional commits (`feat`, `fix`, `docs`, `ci`, `chore`, `perf`). Scope tags used, e.g. `feat(pulse)`, `fix(security)`, `docs(acceptance)`. --- ## Tests -### TypeScript (Vitest) — 87 tests - -| File | Count | What it tests | -|---|---|---| -| `tests/restClient.test.ts` | 11 | `splitIntoChunks` math, `selectPublishStrategy` 64 MB boundary, `signIn` parsing, single publish, chunked publish (3 chunks, 3 PUTs + 1 finalize), mid-stream abort (no finalize), `resolveProjectId` rejects empty/Default/resolves known | -| `tests/secrets.test.ts` | 3 | PAT not in sign-in output, PAT not in redacted API error, config error does not echo PAT | -| `tests/sidecar.test.ts` | 7 | AuthoringSidecar: startup health-poll, buildDatasource, buildStarterWorkbook, buildDashboardWorkbook call wiring | -| `tests/sidecar-columns.test.ts` | 3 | Column schema pass-through from `/datasource/from-file` response | -| `tests/planner.test.ts` | 29 | `planDashboard()` autonomous/interview/interview_followup/directed × exec/analyst/operational audiences; field inference; mark-type constraints; question count | -| `tests/tools.test.ts` | 34 | All 14 tools registered with description+schemas; full wiring for `create_datasource_from_query`, `create_starter_workbook`, `design_dashboard`, `build_from_plan`; guardrails: delete confirm, delete refuses Default project, `set_permissions` elevated gate, allowlist rejection, valid caps; `create_datasource_from_table` input validation; `create_datasource_from_file` unsupported extension (0 sidecar calls) | - -### Python (pytest) — 114 tests - -| File | Count | What it tests | -|---|---|---| -| `sidecar/tests/test_hyper_builder.py` | 12 | Hyper round-trip (row count + types), column roles, CSV source, max_rows cap, records_to_dataframe, read_hyper_columns | -| `sidecar/tests/test_hyper_builder_formats.py` | 16 | csv/json/jsonl/xlsx/parquet format round-trips; byte cap; `file_to_dataframe` unsupported type error; Excel sheet by index/name | -| `sidecar/tests/test_tds_builder.py` | 3 | `.tdsx` zip structure (`.tds` + `Data/*.hyper`), dbname path matches, column roles | -| `sidecar/tests/test_twb_builder.py` | 13 | Published datasource reference (sqlproxy/repository-location), one worksheet per sheet spec, mark class per type, default site path, starter `.twbx` is a valid zip; A2 structural pins (simple-id, cards, viewpoint, aggregation, style, explain-data) | -| `sidecar/tests/test_twb_dashboard.py` | 21 | Zone count/names match sheets; worksheet zones have `name` and no `type`; tiling geometry (Σ==100000, no overlap, distinct offsets); canvas size element; default-None regression (byte-identical + no ``) | -| `sidecar/tests/test_twb_embedded.py` | 16 | Federated datasource (not sqlproxy); hyper named-connection; field references use `[federated.*]`; zip contains `Data/*.hyper`; XSD gate (with and without dashboard); FileNotFoundError on missing extract | -| `sidecar/tests/test_twb_schema_validation.py` | 4 | Official TWB XSD gates sqlproxy + embedded output; XXE-safe parser guard; malformed-XML rejection | -| `sidecar/tests/test_server.py` | 5 | Health endpoint, from-table (records) returns `.tdsx`, from-table requires input (400), workbook starter returns `.twbx`, token guard blocks/passes | -| `sidecar/tests/test_server_new_routes.py` | 24 | `/datasource/from-file` for all 5 formats returns valid `.tdsx`; `/workbook/dashboard` returns valid `.twbx` with embedded extract | - -Run commands: `npm test` (TS) and `cd sidecar && uv run pytest -q` (Python). +Real counts as run for this doc pass (2026-07-17): **525 TypeScript tests** (`npm test`) across 23 +files, **338 Python tests** (`cd sidecar && uv run pytest -q`) across 20 files. See the file map +above for what each test file covers; the highlights: + +- `tests/tools.test.ts` — the single integration test asserting all **27** tools register with a + description + input/output schemas, plus full call-wiring for the multi-step tools + (`design_dashboard`, `build_from_plan`, `create_live_datasource`) and every destructive-tool + confirm-gate. +- `tests/planner*.test.ts` + `tests/planner-storyArc.test.ts` — the full deterministic planning + pipeline: field inference, mark selection, audience + persona clamps, KPI-strip/color/scatter/geo + encoding, `storyArc` generation, and `DashboardProposal` projection. +- `tests/restClient.test.ts`, `tests/restRetry.test.ts`, `tests/retry.test.ts` — chunk math, + publish-strategy boundary, sign-in parsing, and the shared retry/backoff policy (deterministic + via injected `sleep`/`jitterFn`). +- `tests/vds.test.ts`, `tests/schedules.test.ts`, `tests/webhooks.test.ts`, `tests/pulse.test.ts` — + each `rest/` module's request-building, response-parsing, and error-classification behavior. +- `tests/branding.test.ts`, `tests/builderBrand.test.ts` — `brand.yaml` load/default/validation and + the pure projection to the sidecar's wire shape. +- `tests/secrets.test.ts`, `tests/credentials.test.ts` — standing assertions that the PAT and live- + connection passwords never appear in logs, sign-in output, or redacted API errors. +- `tests/cronTemplates.test.ts`, `tests/generateCron.test.ts`, `tests/refreshLocalArgs.test.ts` — + local-file refresh automation, including adversarial shell-injection test cases (`$(id)`, quote + breakout) for the crontab-line template. +- Sidecar: `test_twb_schema_validation.py` gates every builder output path (starter, embedded, + branded, story) against the official TWB XSD; `test_twb_story.py`, `test_twb_branding.py`, + `test_twb_kpi_tile.py`, `test_twb_scatter.py`, `test_twb_map_filled.py`, + `test_twb_dashboard_layout.py` cover each Phase-1/E1/E4 encoding and layout feature end-to-end + through the actual XML output. --- ## Dependencies & Risk -**Production TypeScript deps (3):** +**Production TypeScript deps (4):** - `@modelcontextprotocol/sdk@1.29.0` — Anthropic's official MCP server SDK. -- `undici@7.28.0` — Node.js HTTP client; bumped to 7.28.0 in the most recent security fix cycle; `npm audit --omit=dev` reports 0 vulnerabilities. +- `undici@7.28.0` — Node.js HTTP client; `npm audit --omit=dev` reports 0 vulnerabilities. +- `yaml@2.9.0` — `brand.yaml` parsing (added Phase E1). - `zod@3.25.76` — schema validation. **Python deps of note:** - `tableauhyperapi@0.0.21408` — Tableau-proprietary Hyper engine; binary wheel; no Python 3.13 wheel yet (uv uses Python 3.12 inside the venv). - `pantab@5.2.0` — thin pandas/Arrow bridge over tableauhyperapi. -- Database connectors are optional extras, not in the default install. +- `starlette==0.49.3` (transitive via `fastapi==0.121.0`) — pinned explicitly to clear a set of transitive advisories flagged in an earlier ecosystem review. +- Database connectors are optional extras, not in the default install; `create_live_datasource` needs none of them (it builds connection-topology XML only — Tableau's own server-side connector executes the live query). **License:** MIT (repo). Dependencies are MIT/BSD/Apache except `tableauhyperapi` (Tableau proprietary). ---- - -## Tech Debt / Issues - -1. **Lint gate broken by `.cursor/` directory.** `eslint.config.js:7` ignores `dist/`, `node_modules/`, `sidecar/`, `coverage/` — but not `.cursor/`. The `.cursor/` directory was added to the working tree after the last green CI run. ESLint now reports ~200 errors on those CJS hook scripts, so `make ci` (`npm run lint`) exits non-zero **locally**. The upstream GitHub CI never saw `.cursor/` (it is `.gitignore`d), so CI remains green. The fix is one line: add `".cursor/**"` to the `ignores` array. This must be done before the next feature branch runs `make ci` locally. - -2. **No `typecheck` step in `make ci`.** The Makefile runs `build` (which emits JS and catches type errors), but a standalone `typecheck` (`tsc --noEmit`) step is absent from the `ci` target. In practice, `tsc` errors block `build`, so this is not a real gap, but a dedicated `typecheck` step would catch import-only type errors without producing artifacts. - -3. ~~**`twb_builder.py` emits worksheets only — no dashboard block.**~~ **Shipped.** `build_twb_xml()` / `build_embedded_twb_xml()` now emit an optional `` + `` block; `build_from_plan` drives the embedded-extract path end-to-end. - -4. ~~**File-format support is CSV-only.**~~ **Shipped.** `create_datasource_from_file` (`/datasource/from-file`) accepts csv, json, jsonl, xlsx, and parquet via `hyper_builder.file_to_dataframe()`. - -5. **`getDatasource` has a fallback list-all-datasources** when `contentUrl` is missing from the GET response (`src/restClient.ts:401–416`). This is correct but can be slow on large sites and is a fragility point if `contentUrl` is reliably missing. - -6. **Structured logging is `process.stderr.write` concatenation** — no log levels, no JSON format, no correlation IDs. Adequate for an MCP stdio server today but would need a real logger (e.g., `structlog` on the Python side is already in the deps but unused). +**npm tarball (`npm pack --dry-run`):** 130 files, ~269 kB packed / ~1.1 MB unpacked. Includes: +`dist/` (all tools + planner + branding + rest/), `brand.yaml`, `sidecar/*.py` (4 files), +`sidecar/pyproject.toml`, `sidecar/uv.lock`, `LICENSE`, `README.md`, `package.json`. Excludes: +`sidecar/tests/`, `sidecar/.venv/`, `scripts/`, `tests/`, caches, logs, `.cursor/`. --- ---- +## Tech Debt / Issues -## Regression-Guard Tests That Must Keep Passing - -All 201 tests must stay green on `make ci`: - -**TypeScript (87 tests — `npm test`):** -- `tests/restClient.test.ts`: `splitIntoChunks` math, `selectPublishStrategy` 64 MB boundary, `signIn` parsing, single-request publish, 3-chunk upload (3 PUTs + 1 finalize POST), mid-stream abort (no finalize), `resolveProjectId` rejects empty/Default/resolves known. -- `tests/secrets.test.ts`: PAT not in sign-in output, PAT not in redacted API error, config error does not echo PAT. -- `tests/sidecar.test.ts`: AuthoringSidecar startup, health-poll, build* call wiring. -- `tests/sidecar-columns.test.ts`: column schema pass-through from `/datasource/from-file`. -- `tests/planner.test.ts`: `planDashboard()` autonomous/interview/directed × all audiences; field inference; mark constraints; question count. -- `tests/tools.test.ts`: 14-tool registration count, all tool names present, `create_datasource_from_query` full wiring, `create_starter_workbook` wiring, `design_dashboard` wiring, `build_from_plan` wiring; all guardrails (delete confirm, delete Default refusal, elevated-capability gate, allowlist rejection), `create_datasource_from_table` input validation, `create_datasource_from_file` unsupported extension. - -**Python (114 tests — `cd sidecar && uv run pytest -q`):** -- `sidecar/tests/test_hyper_builder.py`: hyper round-trip row count + all column types, column role assignment, CSV source read, max_rows cap, records_to_dataframe, read_hyper_columns. -- `sidecar/tests/test_hyper_builder_formats.py`: csv/json/jsonl/xlsx/parquet round-trips; byte cap; unsupported type error; Excel sheet by index/name. -- `sidecar/tests/test_tds_builder.py`: `.tdsx` zip has exactly one `.tds` and one `Data/*.hyper`, dbname path format, column role attributes. -- `sidecar/tests/test_twb_builder.py`: sqlproxy datasource reference present, repository-location attributes, one worksheet per sheet spec with correct datasource-dependencies, mark class mapping (bar/line/text), default site path, `.twbx` is a valid zip containing a parseable ``; A2 structural pins (simple-id, cards with shelf content, viewpoint, aggregation, style, explain-data). -- `sidecar/tests/test_twb_dashboard.py`: zone count/names match sheets; worksheet zones have `name` and no `type`/`type-v2`; tiling geometry invariants (Σ==100000, no overlap, distinct offsets); canvas size element matches inputs; default-None regression (byte-identical, no ``). -- `sidecar/tests/test_twb_embedded.py`: federated datasource (not sqlproxy); hyper named-connection; `[federated.*]` field references in rows/cols; zip contains `Data/*.hyper`; XSD gate (with and without dashboard); FileNotFoundError on missing extract. -- `sidecar/tests/test_twb_schema_validation.py`: official TWB XSD gates sqlproxy and embedded output; XXE-safe parser config; malformed-XML rejected by schema. -- `sidecar/tests/test_server.py`: health returns `{"status":"ok"}`, from-table (records) produces a valid `.tdsx` zip, from-table without input returns 400, workbook starter produces `.twbx`, token guard blocks requests without header and passes with matching header. -- `sidecar/tests/test_server_new_routes.py`: `/datasource/from-file` for all 5 formats returns valid `.tdsx`; `/workbook/dashboard` returns valid `.twbx` with embedded extract. +1. **Lint gate broken by `.cursor/` directory.** `eslint.config.js` ignores `dist/`, + `node_modules/`, `sidecar/`, `coverage/` — but not `.cursor/`. If `.cursor/` is present in the + working tree (it is `.gitignore`d, so this affects local runs only, not upstream CI), ESLint + reports errors on those CJS hook scripts and `make ci` (`npm run lint`) exits non-zero locally. + The fix is one line: add `".cursor/**"` to the `ignores` array. + +2. **No standalone `typecheck` step in `make ci`.** The Makefile runs `build` (which emits JS and + catches type errors via `tsc`), but a dedicated `typecheck` (`tsc --noEmit`) step is absent from + the `ci` target. In practice `tsc` errors already block `build`, so this is not a real coverage + gap, just a missing explicit step for import-only type errors that produce no artifacts. + +3. **`getDatasource` has a fallback list-all-datasources** when `contentUrl` is missing from the + GET response (`src/restClient.ts`). Correct but can be slow on large sites and is a fragility + point if `contentUrl` is reliably missing. + +4. **Structured logging is `process.stderr.write` concatenation** — no log levels, no JSON format, + no correlation IDs, and no persistent metrics/telemetry surface. Adequate for an MCP stdio + server today (see `docs/runbook.md`'s monitoring section for the practical workaround) but would + need a real logger (e.g., `structlog` on the Python side is already in the deps but unused) for + anything beyond single-session debugging. + +5. **Pulse's create-definition payload shape is unconfirmed against a live site** (`400` with no + field-level detail on the one live attempt so far). Code-complete and honestly flagged; see + `docs/adr/0011-pulse-best-effort-payload.md` and `docs/tool_reference.md`'s Pulse section for the + concrete next step. + +6. **Live-connection XML attribute mapping is VERIFY-LIVE.** `sidecar/tds_builder.py`'s + `SNOWFLAKE_ATTRS`/`PRESTO_ATTRS` are this project's best-documented guess at the connector-class + attribute spelling and have not yet been confirmed against a Desktop-exported `.tds`. + +7. **`IncrementalRefresh`'s exact token spelling is VERIFY-LIVE** in `schedule_refresh` — reference + material is inconsistent between `IncrementalRefresh` and `IncrementalExtract`. The tool defaults + every caller to `FullRefresh` and only emits `IncrementalRefresh` on explicit request. diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index 77fdd90..9a6794f 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -37,7 +37,8 @@ fully reproducible without network access to PyPI beyond what uv already has cac ## Environment variables -All four are required at runtime. No others are read from the environment by default. +The four core variables are required at runtime — the server refuses to start without them (a +zod-validated config error, never echoing the PAT): | Variable | Description | |----------|-------------| @@ -46,7 +47,29 @@ All four are required at runtime. No others are read from the environment by def | `PAT_NAME` | Personal Access Token name | | `PAT_VALUE` | Personal Access Token secret — never log, never commit | -`DEMO_PROJECT` is also required when running the dashboard demo (see below). +Optional (`config.ts` defaults applied when absent): + +| Variable | Default | Description | +|----------|---------|-------------| +| `TABLEAU_API_VERSION` | `3.28` | Tableau REST API version. | +| `SIDECAR_HOST` | `127.0.0.1` | Loopback host for the Python sidecar. | +| `SIDECAR_PORT` | auto-selected free port | Pin only if you need a fixed port. | + +Required only for specific demos/scripts (not for the server itself to start): + +| Variable | Used by | +|----------|---------| +| `DEMO_PROJECT` | `npm run demo`, `npm run demo:dashboard`, `npm run demo:superstore` — must name an existing, non-Default project. | +| `PERSONA` | `npm run demo:superstore` — alternative to passing `--persona ` on the command line. | + +**Live-connection credentials are NOT environment variables.** `create_live_datasource` (Snowflake/ +Presto) takes `credentials.username`/`credentials.password` as **tool-call parameters**, not +server-side config — the MCP server's own environment surface is unchanged by this feature. The +tool's description recommends the *calling agent* source these from its own environment (e.g. a +convention like `SNOWFLAKE_USER`/`SNOWFLAKE_PASSWORD` in the agent's own process) and pass them as +call arguments — never hardcode them in a prompt or a saved plan. This is a documented convention +for callers, not something `config.ts` reads or `.env.example` should declare, since the server +process never consults such variables itself. See `docs/adr/0009-live-connection-credential-handling.md`. ## MCP client config @@ -69,50 +92,95 @@ All four are required at runtime. No others are read from the environment by def Runs side by side with the official `tableau` server on the same PAT. -## Registered tools (14 total) +## Registered tools (27 total) -### Data ingestion (tools 1–5) +### Ingest & datasources (6) | Tool | Description | |------|-------------| -| `create_datasource_from_query` | Run SQL against Snowflake/Postgres → Hyper extract → publish `.tdsx` | +| `create_datasource_from_query` | Run SQL against Snowflake/Postgres/csv → Hyper extract → publish `.tdsx` | | `create_datasource_from_table` | CSV or inline records → Hyper extract → publish `.tdsx` | -| `create_datasource_from_file` | **New (M2)** — local CSV, JSON, JSONL, Excel (.xlsx/.xls), or Parquet → Hyper extract → publish `.tdsx` | +| `create_datasource_from_file` | Local CSV, JSON, JSONL, Excel (.xlsx/.xls), or Parquet → Hyper extract → publish `.tdsx` (numeric/date coercion applied) | +| `create_live_datasource` | Live Snowflake/Presto connection → `.tds` (no extract) → publish with embedded credentials | | `publish_datasource` | Publish a pre-built `.tdsx` or `.hyper` file | -| `publish_workbook` | Publish a pre-built `.twbx` file | +| `publish_workbook` | Publish a pre-built `.twb`/`.twbx` file | + +### Design & build (3) + +| Tool | Description | +|------|-------------| +| `design_dashboard` | Business question + audience/persona → `DashboardProposal` (no publish); supports `autonomous`, `interview`, `interview_followup`, `directed` modes | +| `build_from_plan` | `proposal.plan` → embedded-extract `.twbx` (+ story dashboard, if any) + governed `.tdsx` → publish both | +| `create_starter_workbook` | Build and publish a `.twbx` bound to an already-published datasource | + +### Branding (1) + +| Tool | Description | +|------|-------------| +| `validate_brand` | Validate `brand.yaml`; list personas; never throws | -### Workbook authoring (tools 6–8) +### Metadata (1) | Tool | Description | |------|-------------| -| `create_starter_workbook` | Build and publish a `.twbx` bound to a published datasource | -| `design_dashboard` | **New (M5)** — generate a `DashboardPlan` JSON from a business question + audience + field hints; supports `autonomous`, `interview`, `interview_followup`, and `directed` modes | -| `build_from_plan` | **New (M6)** — build and publish a dashboard `.twbx` from a `DashboardPlan` produced by `design_dashboard` | +| `get_datasource_fields` | Real field names/types/aggregations for a published datasource, via VDS | -### Site management (tools 9–14) +### Scheduling & automation (3) + +| Tool | Description | +|------|-------------| +| `schedule_refresh` | Create a recurring Cloud extract-refresh task (honest Bridge caveat always included) | +| `list_refresh_schedules` | List every extract-refresh task on the site | +| `delete_refresh_schedule` | Delete a refresh task (`confirm: true` required) | + +### Webhooks (3) + +| Tool | Description | +|------|-------------| +| `create_webhook` | Create an HTTPS webhook on a Tableau site event (site-admin PAT required) | +| `list_webhooks` | List webhooks configured on the site | +| `delete_webhook` | Delete a webhook (`confirm: true` required) | + +### Pulse (4) + +| Tool | Description | +|------|-------------| +| `create_pulse_definition` | Create a Pulse metric definition (VDS pre-flight validated; live creation currently returns 400 pending a fixture — see `docs/adr/0011-pulse-best-effort-payload.md`) | +| `list_pulse_definitions` | List Pulse metric definitions on the site | +| `create_pulse_metric` | Create a Pulse metric instance from a definition | +| `delete_pulse_definition` | Delete a Pulse metric definition (`confirm: true` required) | + +### Site management (6) | Tool | Description | |------|-------------| | `list_projects` | List all projects on the site | | `create_project` | Create a Tableau project | | `list_content` | List published datasources or workbooks | -| `refresh_datasource` | Trigger a background refresh of a published datasource | -| `delete_content` | Delete a published datasource or workbook | -| `set_permissions` | Set project-level permissions on a content item | +| `refresh_datasource` | Trigger a single immediate extract refresh (not recurring — see `schedule_refresh`) | +| `delete_content` | Delete a published datasource or workbook (`confirm: true` required; refuses `Default` project) | +| `set_permissions` | Set permissions on a content item (allowlisted capabilities; elevated grants need `confirmElevated: true`) | + +6 + 3 + 1 + 1 + 3 + 3 + 4 + 6 = **27**. Full parameter/return/guardrail reference: +[`docs/tool_reference.md`](docs/tool_reference.md). -## Python sidecar — new deps in this release +## Python sidecar — dependency history | Package | Version | Why | |---------|---------|-----| | `openpyxl` | 3.1.5 | Excel `.xlsx`/`.xls` read support for `create_datasource_from_file` | -| `pyarrow` | 24.0.0 | Parquet read support (pulled in transitively by `pantab==5.2.0`) | -| `fastapi` | 0.121.0 | Bumped; adds `/datasource/from-file` and `/workbook/dashboard` routes | +| `pyarrow` | (transitive via pantab) | Parquet read support | +| `fastapi` | 0.121.0 | `/datasource/from-file`, `/datasource/live`, and `/workbook/dashboard` (brand + story blocks) routes | +| `yaml` (TypeScript, `yaml@2.9.0`) | 2.9.0 | `brand.yaml` parsing (`src/branding/load.ts`) | -These are all declared in `sidecar/pyproject.toml` and pinned in `sidecar/uv.lock`. -`uv sync` (any form) installs them automatically — no manual step needed. +All declared in `sidecar/pyproject.toml` / `package.json` and pinned in `sidecar/uv.lock` / +`package-lock.json`. `uv sync` / `npm install` installs them automatically — no manual step needed. ## Gated live demos +All demos are guarded and no-op with a clear message unless `SERVER`, `SITE_NAME`, `PAT_NAME`, +`PAT_VALUE`, and `DEMO_PROJECT` (an existing, non-Default project) are set. + ### Original demo — datasource + starter workbook ```bash @@ -121,10 +189,7 @@ export DEMO_PROJECT="Sales" # a project that already exists on your site npm run demo -- examples/top_customers.csv ``` -### Dashboard demo (GATED — authorized action only) - -This is the **single outward action** that writes to a real Tableau site. Run it only after -confirming the target project and credentials. +### Dashboard demo — propose→build pipeline ```bash export SERVER=... SITE_NAME=... PAT_NAME=... PAT_VALUE=... @@ -132,12 +197,44 @@ export DEMO_PROJECT="Sales" npm run demo:dashboard -- examples/top_customers.csv ``` -Publishes a Hyper datasource from the CSV, runs the autonomous planning pipeline to derive -a `DashboardPlan`, builds a `.twbx` with a dashboard tiling all sheets via the sidecar, and -publishes it to Tableau Cloud. Prints both Cloud URLs on completion. +Publishes a Hyper datasource from the CSV, runs the autonomous planning pipeline (analyst +audience) to derive a `DashboardProposal`, calls `build_from_plan` with `proposal.plan`, and prints +both published Cloud URLs. + +### Superstore demo — the richest example (KPI band, color, geo, branding, stories) + +```bash +export SERVER=... SITE_NAME=... PAT_NAME=... PAT_VALUE=... +export DEMO_PROJECT="Sales" +npm run demo:superstore # reads $HOME/Downloads/Sample - Superstore_Migrated Data.csv by default +npm run demo:superstore -- "/path/to/file.csv" # or pass an explicit path +npm run demo:superstore -- --persona ceo # also resolves a named persona and applies its brand +``` + +Ingests a UTF-16/TSV-shaped Superstore export (encoding/delimiter auto-sniffed), designs an exec +dashboard (KPI band with period deltas, a Segment-colored bar, a filled US-state map), optionally +applies a persona's brand (palette/typography/number formats), builds the self-contained embedded- +extract workbook, and publishes it. This is the one-command live proof for the branding system +(Phase E1) and the richest exec-plan path (Phase 1). -Use a free [Tableau Developer Program](https://www.tableau.com/developer) site. This is the only -step that writes to a real site; everything else is unit/headless tested. +### Local-file refresh automation (no live-site write beyond a republish) + +```bash +export SERVER=... SITE_NAME=... PAT_NAME=... PAT_VALUE=... +npm run refresh:local -- --file examples/top_customers.csv --name "Top Customers" --project "Sales" +``` + +Re-ingests the file and republishes with `overwrite=true`. Proven live: republished the Superstore +datasource in 4.7s against a real Cloud site (see `ACCEPTANCE.md`'s E2 section). + +```bash +npm run cron:generate -- --daily 06:30 --file examples/top_customers.csv --name "Top Customers" --project "Sales" +``` + +Prints (does not install) a crontab line + macOS launchd plist for the schedule above. + +Use a free [Tableau Developer Program](https://www.tableau.com/developer) site for testing. Free +port 8899 first if a stale sidecar is bound: `kill $(lsof -ti :8899)`. ## Pre-flight verification (no live credentials needed) @@ -153,59 +250,71 @@ npm run test:mcp-smoke # or: npm run test:mcp-smoke -- --publish-only ``` -Connects to the MCP server over stdio and lists all registered tools. Verifies 14 tools respond. -Requires real credentials in `.env` or the environment; exits 0 gracefully when they are absent. +Connects to the MCP server over stdio and lists all registered tools. Verifies **27** tools +respond. Requires real credentials in `.env` or the environment; exits 0 gracefully when they are +absent. -Note: if port 8899 is already bound by a running sidecar, `verify-setup` and `test:mcp-smoke` -will report a sidecar startup error. Kill the existing sidecar process first: -`kill $(lsof -ti :8899)` +Note: if the sidecar's port is already bound by a running sidecar, `verify-setup` and +`test:mcp-smoke` will report a sidecar startup error. Kill the existing sidecar process first +(default historical port 8899): `kill $(lsof -ti :8899)`. ## Rollout (gated) -1. **Private GitHub repo first.** Push the branch, let CI go green, review. *(done in this run)* -2. **Flip to public** only on explicit confirmation — once the live demo has produced real URLs and - the README/docs read well. +1. **Private GitHub repo first.** Push the branch, let CI go green, review. +2. **Flip to public** only on explicit confirmation — once the live demos have produced real URLs + and the README/docs read well. 3. **npm publish (later, gated).** When ready to ship `npx tableau-mcp-publish`: ```bash npm publish --access public ``` Requires an npm login; not run automatically. The `files` allowlist in `package.json` ensures - the tarball includes `dist/` + `sidecar/*.py` + `sidecar/pyproject.toml` + `sidecar/uv.lock` - and excludes tests, caches, logs, and `.cursor/`. + the tarball includes `dist/` + `brand.yaml` + `sidecar/*.py` + `sidecar/pyproject.toml` + + `sidecar/uv.lock` and excludes tests, caches, logs, and `.cursor/`. Verified via + `npm pack --dry-run`: **130 files, ~269 kB packed / ~1.1 MB unpacked**. ## Runbook +Day-to-day operation, monitoring, and troubleshooting live in +[`docs/runbook.md`](docs/runbook.md) — quick highlights: + - **Least privilege (important):** every tool runs with the PAT's full authority, and the SQL you pass to `create_datasource_from_query` runs with the DB credentials you supply. Use a PAT whose Tableau user has only the roles it needs, and a **read-only** Snowflake/Postgres role for queries. - See `SECURITY.md` (F-02, F-12). + See `SECURITY.md`. - **Sidecar won't start:** ensure `uv` is on PATH and `cd sidecar && uv sync` succeeded. The server surfaces the sidecar's stderr on a startup failure. -- **Port 8899 already in use:** another sidecar process is running (common after a Cursor restart - without a clean shutdown). Run `kill $(lsof -ti :8899)` to free it. +- **Port already in use:** another sidecar process is running (common after a Cursor restart + without a clean shutdown). Run `kill $(lsof -ti :8899)` (or your pinned `SIDECAR_PORT`) to free it. - **401 on sign-in:** PAT expired/typo, or `SITE_NAME` is wrong (use the site's contentUrl). - **"Project not found":** `projectName` must match an existing project exactly; create it first with `create_project`. - **Workbook opens but a field is unknown:** the sheet referenced a field that isn't in the - published datasource — field names must match the datasource columns exactly. + published datasource — call `get_datasource_fields` and use exact field names. - **Large files:** publishing automatically switches to chunked upload at 64 MB or larger. +- **Scheduled refresh created but every run fails:** the target datasource is file-based, not + cloud-reachable — see the local-file cron design-around above instead of `schedule_refresh`. +- **Pulse `create_pulse_definition` returns a bare 400:** known, honestly-flagged limitation + pending a live fixture (`docs/adr/0011-pulse-best-effort-payload.md`) — not resolvable client-side today. - **`design_dashboard` in directed mode:** `directions` is required and must be non-empty; the tool rejects calls where it is absent or blank. ## Deployment-readiness checklist - [x] Config via environment; no secrets in code; `.env` gitignored; `.env.example` provided. -- [x] CI green on both layers (87 TS + 114 Python) across the version matrix. -- [x] All 14 tools compile into `dist/` (`npm run build` clean, zero TS errors). -- [x] Headless tests for the authoring + publish paths; mocked REST (no PAT in CI). +- [x] CI green on both layers (525 TS + 338 Python) across the version matrix. +- [x] All 27 tools compile into `dist/` (`npm run build` clean, zero TS errors). +- [x] Headless tests for the authoring + publish + scheduling + webhook + Pulse paths; mocked REST (no PAT in CI). - [x] Graceful shutdown (sign out + stop sidecar) on SIGINT/SIGTERM. - [x] Sidecar bound to loopback with a per-spawn token. -- [x] New Python deps (openpyxl, pyarrow, fastapi bump) pinned in `uv.lock`; `uv sync --frozen` clean. -- [x] npm tarball verified via `npm pack --dry-run`: 75 files, 119 kB packed / 432 kB unpacked. - Includes: `dist/` (all tools + planner), `sidecar/*.py` (4 files), `sidecar/pyproject.toml`, - `sidecar/uv.lock`, `LICENSE`, `README.md`, `package.json`. Excludes: `sidecar/tests/`, - `sidecar/.venv/`, `scripts/`, `tests/`, caches, logs, `.cursor/`. +- [x] Live-connection credentials never written to disk/logs (asserted in tests). +- [x] Local-file cron template generator shell-escapes all interpolated values (VB-02 fix). +- [x] npm tarball verified via `npm pack --dry-run`: 130 files, ~269 kB packed / ~1.1 MB unpacked. - [x] Smoke test (`npm run test:mcp-smoke`) and verify-setup (`npm run verify-setup`) both exit - gracefully without creds; pass when real credentials are present (requires free port 8899). -- [ ] Live dashboard demo (`npm run demo:dashboard`) executed against a Dev site (record URLs in `ACCEPTANCE.md`). + gracefully without creds; pass when real credentials are present (requires a free sidecar port). +- [x] Live demos executed against a Dev site (datasource, dashboard, branded dashboard, story, + local-file refresh — URLs recorded in `ACCEPTANCE.md`). +- [ ] Live proof of `schedule_refresh` against a genuinely cloud-reachable (Snowflake) connection + (needs account credentials — see `ACCEPTANCE.md`'s E2 pending items). +- [ ] Pulse create-definition live fixture (needs a manually-created Pulse UI definition to GET back + and lock the payload shape — see `docs/adr/0011-pulse-best-effort-payload.md`). - [ ] npm publish (deferred until after public release). diff --git a/README.md b/README.md index 77883c6..7b3454a 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,13 @@ # tableau-mcp-publish -> The **write side** of Tableau MCP. The official `@tableau/mcp-server` lets an agent -> **read and query** Tableau (VizQL Data Service, Metadata API, Pulse). This server lets an agent -> **author and publish** — turn a SQL query or a DataFrame into a governed published datasource, -> generate a starter workbook, and publish both to Tableau Cloud. Built on the Tableau REST API, -> the Hyper API, and hand-authored Tableau document XML. Drops into the **same MCP client config** -> as the official server. +> **Vibe-BI on Tableau Cloud** — from a prompt to governed datasources, branded persona-aware +> dashboards, stories, Pulse metrics, and scheduled refresh. The official `@tableau/mcp-server` lets +> an agent **read and query** Tableau (VizQL Data Service, Metadata API, Pulse). This server lets an +> agent **author and publish** — turn a SQL query, a file, or a live warehouse connection into a +> governed datasource, design and build a branded dashboard from a business question, tell it as a +> Tableau Story, track it as a Pulse metric, and keep it fresh on a schedule. 27 tools, built on the +> Tableau REST API, the Hyper API, and hand-authored Tableau document XML. Drops into the **same MCP +> client config** as the official server. [![CI](https://github.com/SebAustin/tableau-mcp-publish/actions/workflows/ci.yml/badge.svg)](https://github.com/SebAustin/tableau-mcp-publish/actions/workflows/ci.yml) [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE) @@ -15,11 +17,13 @@ Tableau's official MCP server (`tableau/tableau-mcp` v2.18.x) covers reading, querying, and Desktop-local workbook editing — and adds admin-gated delete/refresh tools in its latest release. What it still cannot do is build a governed Hyper extract and **publish a datasource or workbook -to Tableau Cloud**, headless, from SQL or a file. +to Tableau Cloud**, headless, from SQL, a file, or a live connection — let alone design a branded, +audience-aware dashboard from a plain-language question. That's the half I work with all day as a Tableau Ambassador, so I built the companion. Run them -together and an agent goes from *"query this data"* to *"now publish a governed datasource and a -starter workbook to my Cloud site"* — without leaving the conversation. +together and an agent goes from *"query this data"* to *"design an exec dashboard for the CEO +persona, confirm it, publish it, tell it as a story, track the headline number in Pulse, and keep +it refreshed every morning"* — without leaving the conversation. ## Run it alongside the official server @@ -52,6 +56,24 @@ starter workbook to my Cloud site"* — without leaving the conversation. Same env vars. The query server and the publish server share one Personal Access Token. +## Quickstart + +```bash +git clone https://github.com/SebAustin/tableau-mcp-publish && cd tableau-mcp-publish +npm install && npm run build +cd sidecar && uv sync && cd .. +cp .env.example .env # fill SERVER, SITE_NAME, PAT_NAME, PAT_VALUE +npm run dev # starts the MCP server (stdio) + spawns the Python sidecar +``` + +Try the bundled demo (needs `DEMO_PROJECT` set to an existing, non-Default project): + +```bash +npm run demo -- examples/top_customers.csv +``` + +It publishes a datasource and a starter workbook from a bundled CSV and prints both Cloud URLs. + ## What an agent can do with it > **"Take the top 100 customers by revenue from Snowflake and publish it as a governed datasource @@ -67,132 +89,216 @@ The agent calls, in sequence: Two tool calls. A governed datasource and a workbook, live on Cloud. -## Prompt-driven authoring - -Three new tools let an agent go from a plain-language business question straight to a published -dashboard — no manual sheet-spec assembly required. - -### Datasource from a file +## Prompt-driven authoring — the propose→confirm loop -`create_datasource_from_file` reads any of CSV, JSON, JSONL, Excel (`.xlsx`/`.xls`), or Parquet -from local disk, materialises it as a Hyper extract, and publishes a `.tdsx` to Cloud. +`design_dashboard` is a **pure planner**: it turns a business question, an audience (or a named +persona), and optional field hints into a `DashboardProposal` — a human-readable summary, a KPI +strip, a list of charts, a layout description, and (when warranted) a `storyOutline`. **It never +publishes anything.** `build_from_plan` is the **only side-effecting tool**: it consumes +`proposal.plan` verbatim, builds a self-contained `.twbx` with the source data embedded (federated +`.hyper` extract), and publishes it to Cloud — plus a governed `.tdsx` datasource as an independent +artifact. ``` -create_datasource_from_file( - name="Regional Sales", - filePath="/data/sales.parquet", - projectName="Analytics" -) +design_dashboard ──▶ DashboardProposal (summary, kpiStrip, views, layoutSummary, storyOutline?) + │ │ + │ user: "change X" │ user: "confirm" + └──── directed mode ◀────┘ │ + ▼ + build_from_plan(proposal.plan) + │ + ▼ + published datasource + dashboard workbook (+ story, if any) ``` -### Dashboard from a business question — the two-tool pattern +The split is deliberate and **stateless** — the server holds no conversation state between calls; +the agent re-presents context on every call. The agent can show the proposal to the user, accept +edits via a fresh `directed`-mode call, then build — without re-running an expensive publish on +every refinement. -`design_dashboard` is a **pure planner**: it turns a business question, audience, and optional -field hints into a `DashboardPlan` JSON object. It never publishes anything. -`build_from_plan` is the **only side-effecting tool**: it consumes the plan, builds a -self-contained `.twbx` with the source data embedded (federated `.hyper` extract), worksheets, and -a tiled ``, and publishes to Cloud. It also publishes the governed `.tdsx` datasource -as a separate artifact. Both artifacts are returned in the response. +**End-to-end example — branded executive revenue dashboard:** -The split is deliberate. The agent can show the plan to the user, accept edits, then build — -without re-running an expensive publish on every refinement. +> *"From `sales.parquet`, design an exec dashboard for the 'ceo' persona answering 'How is revenue +> trending by region?', show me the proposal, then publish it once I confirm."* -**End-to-end example — executive revenue dashboard from a Parquet file:** - -> *"From `sales.parquet`, build an exec dashboard answering 'How is revenue trending by region?'"* - -The agent calls: - -1. `design_dashboard` with `mode="autonomous"`, `audience="exec"`, - `businessQuestion="How is revenue trending by region?"`, and `fieldHints` from - `@tableau/mcp-server` → returns a `DashboardPlan` (≤3 sheets, KPI lead, bar + line). Add - `datasourceSpec.filePath="sales.parquet"` to the plan before calling `build_from_plan`. -2. `build_from_plan` with the plan → builds the `.hyper` extract, publishes a governed `.tdsx` - datasource, then publishes a self-contained `.twbx` workbook with the extract embedded - (federated connection) → returns `{ workbookLuid, url, datasourceLuid }`. - -Two tool calls. A governed datasource and a dashboard workbook, both live on Cloud. The workbook -is self-contained and renders immediately — it carries its own copy of the data. +1. `design_dashboard` with `mode="autonomous"`, `persona="ceo"`, + `businessQuestion="How is revenue trending by region?"`, `fieldHints` from + `get_datasource_fields` (or `@tableau/mcp-server`), and `datasourceSpec.filePath` set on the + returned plan → returns a `DashboardProposal` (≤3 sheets, KPI lead, concise CEO tone, branded). +2. Agent relays the proposal's `summary`/`kpiStrip`/`views`/`layoutSummary` to the user. +3. User confirms → `build_from_plan(proposal.plan)` → builds the `.hyper` extract, publishes a + governed `.tdsx` datasource, applies the CEO persona's brand (palette/typography/number + formats), and publishes a self-contained, branded `.twbx` workbook → + `{ workbookLuid, url, datasourceLuid }`. ### Dashboard modes | Mode | When to use | |---|---| -| `autonomous` | Supply a business question; the planner derives sheets and layout. | +| `autonomous` | Supply a business question (+ optional `persona`); the planner derives sheets and layout. | | `interview` | The planner returns 3–7 clarifying questions; pass answers back as `interview_followup`. | -| `interview_followup` | Answers to interview questions → full `DashboardPlan`. | -| `directed` | Provide explicit sheet directions (e.g. "a bar chart of revenue by region and a trend line"). | - -The interview is a **stateless, two-call contract**: questions → plan. The server holds no -conversation state between calls; the agent supplies the full context on each call. +| `interview_followup` | Answers to interview questions → full proposal. | +| `directed` | Provide explicit sheet directions (e.g. "a bar chart of revenue by region and a trend line"), including revising a prior proposal. | ### Audience -`audience` shapes sheet count, mark types, density, and canvas size: +`audience` (or a persona's base audience) shapes sheet count, mark types, density, and canvas size: | Value | Max sheets | Canvas | Effect | |---|---|---|---| -| `exec` | 3 | 1000×800 | KPI lead, large text, minimal axes | -| `analyst` | 8 | 1200×900 | Dense, scatter/map allowed, full axes | +| `exec` | 3 | 1000×800 | KPI lead, large text, minimal axes, filled-map allowed (no bare scatter) | +| `analyst` | 8 | 1200×900 | Dense, scatter/map/filled-map allowed, full axes | | `operational` | 6 | 800×1200 | Status marks, mobile-friendly single column | | `mixed` | 6 | 1000×900 | Balanced bar/line, one summary KPI | -## Tools +## The brand.yaml kit + named personas + +[`brand.yaml`](brand.yaml) is the single source of brand truth: categorical/sequential/diverging +color palettes, semantic good/bad/neutral colors for KPI deltas, title/body/BAN (Big Number) +typography, number formats, and named **personas** that `design_dashboard` targets with +`persona: "ceo"` instead of (or alongside) a raw `audience`. Every field is optional — delete a +section, or the whole file, and built-in defaults take over (validate with `validate_brand`). + +```yaml +personas: + ceo: + base: exec # underlying audience profile + maxSheets: 3 # tighter than the exec default + kpiEmphasis: high # 4 KPI tiles up front + preferredArtifact: dashboard + tone: concise # trims the proposal summary to the essentials +``` + +Five personas ship out of the box (`ceo`, `cto`, `slt_manager`, `analyst`, `client`), each with +`maxSheets`, `kpiEmphasis`, `preferredArtifact`, `tone`, and `chartDeny` overrides — add your own +without touching code. `build_from_plan` applies the resolved brand's palette, typography, and +number formats to the generated workbook whenever the plan carries persona/brand provenance. + +## Connectivity & automation + +| Need | Tool / mechanism | Honest caveat | +|---|---|---| +| Live Snowflake/Presto connection, no extract | `create_live_datasource` | Snowflake key-pair auth is impossible over REST (throws, use Desktop); Presto/Trino is generally Tableau-Bridge-dependent on Cloud. Credentials are embedded at publish time (``) and never written to the `.tds` file or logged. | +| Recurring Cloud-side refresh | `schedule_refresh` / `list_refresh_schedules` / `delete_refresh_schedule` | Cloud can only **execute** a schedule for a cloud-reachable connection with embedded credentials (e.g. Snowflake). A file-based datasource accepts the schedule-creation call but every run fails without Tableau Bridge — the tool always surfaces this note. | +| Recurring refresh for a **local file** datasource | `npm run refresh:local` + `npm run cron:generate` | Cloud cannot refresh a local-file extract itself (no Bridge). `refresh_local.ts` re-ingests the file and republishes with `overwrite=true`; `generate-cron.ts` emits a ready-to-review crontab line + macOS launchd plist — nothing is installed automatically. | +| Event notifications | `create_webhook` / `list_webhooks` / `delete_webhook` | Requires site-administrator PAT privileges; HTTPS-only destinations enforced client-side before any REST call. | +| Single-metric subscription tracking | `create_pulse_definition` / `list_pulse_definitions` / `create_pulse_metric` / `delete_pulse_definition` | Cloud-only. Code-complete against Tableau's official reference payload shape with VDS pre-flight validation, but **live creation currently returns a 400** — the wire shape is specified-by-example only upstream. Treat as best-effort pending a locked fixture (see [`docs/tool_reference.md`](docs/tool_reference.md#pulse)). | +| Field discovery before planning | `get_datasource_fields` | Real field names/types/aggregations via VizQL Data Service — use instead of guessing column names for `fieldHints`. | + +## The design-excellence layer (Few/Tufte rules) + +Phase E1 encodes a set of deterministic rules inspired by the Stephen Few / Edward Tufte school of +analytical dashboard design. See [BI_DESIGN.md §9](docs/feature-prompt-authoring/BI_DESIGN.md) for +full source lineage and enforcement status per rule: + +- **No-pie default** — part-to-whole questions always resolve to a bar; `"pie"` isn't even a + representable mark type. +- **KPI tiles always carry context** — a comparison/delta is bound whenever the data offers one, + never a bare number with no "so what." +- **No gauges** — bullet graphs are the documented alternative (gauge is not a representable mark + type; bullet graph emission is a tracked future gap, not silently substituted). +- **Data-ink discipline** — no gridline/border/shading/drop-shadow option exists anywhere in the + plan schema, so the planner cannot emit decoration even by accident. +- **Sequential color for magnitude** — a bar or filled map colored by a quantitative field always + uses continuous color semantics, never the categorical palette. +- **Top-N discipline** — high-cardinality dimensions carry a named "Top 10 + Other" rationale note. +- **Small-multiples hint** — a color-coded bar answering a cross-dimension comparison question + gets a documented small-multiples alternative noted in its rationale. + +## Tools (27) + +Full reference with every parameter, guardrail, and example prompt in +[`docs/tool_reference.md`](docs/tool_reference.md). + +**Ingest & datasources** | Tool | What it does | |---|---| | `create_datasource_from_query` | SQL → `.hyper` → `.tdsx` → publish to Cloud | | `create_datasource_from_table` | CSV / records → `.hyper` → `.tdsx` → publish | -| `create_datasource_from_file` | CSV / JSON / JSONL / Excel / Parquet → `.tdsx` → publish | -| `create_starter_workbook` | published datasource + NL sheet specs → `.twbx` → publish | -| `design_dashboard` | business question + audience → `DashboardPlan` (no publish) | -| `build_from_plan` | `DashboardPlan` (requires `datasourceSpec.filePath`) → self-contained `.twbx` (embedded extract) + governed `.tdsx` → publish both to Cloud | -| `publish_datasource` | publish an existing `.tdsx`/`.hyper` file | -| `publish_workbook` | publish an existing `.twb`/`.twbx` file | -| `list_projects` / `create_project` | project management | -| `set_permissions` | grant/deny capabilities on published content | -| `list_content` / `refresh_datasource` / `delete_content` | content lifecycle | - -Full reference with parameters and example prompts in [`docs/tool_reference.md`](docs/tool_reference.md). +| `create_datasource_from_file` | CSV / JSON / JSONL / Excel / Parquet → `.tdsx` → publish (with numeric/date coercion) | +| `create_live_datasource` | Live Snowflake / Presto connection → `.tds` (no extract) → publish with embedded credentials | +| `publish_datasource` | Publish an existing `.tdsx`/`.hyper` file | +| `publish_workbook` | Publish an existing `.twb`/`.twbx` file | + +**Design & build** + +| Tool | What it does | +|---|---| +| `design_dashboard` | Business question + audience/persona → `DashboardProposal` (no publish) | +| `build_from_plan` | `proposal.plan` → embedded-extract `.twbx` (+ story, if any) + governed `.tdsx` → publish both | +| `create_starter_workbook` | Published datasource + NL sheet specs → `.twbx` → publish | + +**Branding** + +| Tool | What it does | +|---|---| +| `validate_brand` | Validate `brand.yaml`; list personas; never throws | + +**Metadata** + +| Tool | What it does | +|---|---| +| `get_datasource_fields` | Real field names/types/aggregations via VizQL Data Service | + +**Scheduling & automation** + +| Tool | What it does | +|---|---| +| `schedule_refresh` | Create a recurring Cloud extract-refresh task | +| `list_refresh_schedules` | List every extract-refresh task on the site | +| `delete_refresh_schedule` | Delete a refresh task (confirm-gated) | + +**Webhooks** + +| Tool | What it does | +|---|---| +| `create_webhook` | HTTPS webhook on a Tableau site event (admin-gated) | +| `list_webhooks` | List configured webhooks | +| `delete_webhook` | Delete a webhook (confirm-gated) | + +**Pulse** + +| Tool | What it does | +|---|---| +| `create_pulse_definition` | Create a Pulse metric definition (VDS pre-flight validated) | +| `list_pulse_definitions` | List Pulse metric definitions on the site | +| `create_pulse_metric` | Create a Pulse metric instance from a definition | +| `delete_pulse_definition` | Delete a Pulse metric definition (confirm-gated) | + +**Lifecycle & site management** + +| Tool | What it does | +|---|---| +| `list_projects` / `create_project` | Project management | +| `list_content` | List published datasources/workbooks | +| `refresh_datasource` | Trigger a single immediate extract refresh | +| `delete_content` | Delete published content (confirm-gated, refuses `Default` project) | +| `set_permissions` | Grant/deny capabilities (allowlisted; elevated grants confirm-gated) | ## Architecture ```mermaid flowchart LR AGENT["MCP client
Claude / Cursor"] -->|MCP tools| TS["TypeScript MCP server
(src/)"] - TS -->|"REST: signin, publish,
projects, permissions"| REST[("Tableau Cloud
REST API")] + TS -->|"REST: signin, publish,
schedules, webhooks, Pulse"| REST[("Tableau Cloud
REST API + VDS + Pulse")] TS -->|"localhost + token"| PY["Python sidecar
(Hyper API + doc XML)"] PY -->|".hyper extract"| HYPER["Hyper API"] - PY -->|".tdsx / .twbx"| DOC["document XML"] + PY -->|".tdsx / .tds / .twbx"| DOC["document XML"] PY -.->|"packaged files"| TS TS -->|"chunked publish (≥64MB)"| REST ``` -Two layers: a TypeScript MCP server (REST auth + publishing) and a Python FastAPI sidecar -(Hyper/`.tdsx`/`.twbx` authoring) the TS layer spawns over loopback. See -[`docs/architecture.md`](docs/architecture.md). - -## Setup - -```bash -git clone https://github.com/SebAustin/tableau-mcp-publish && cd tableau-mcp-publish -npm install && npm run build -cd sidecar && uv sync && cd .. -cp .env.example .env # SERVER, SITE_NAME, PAT_NAME, PAT_VALUE -npm run dev # starts the MCP server (stdio) + spawns the Python sidecar -``` - -Requirements: Node ≥ 22.7.5 (matching the official server's floor), Python 3.12–3.13, and [`uv`](https://docs.astral.sh/uv/). Use the free -[Tableau Developer Program](https://www.tableau.com/developer) for a Cloud site to test against. +Two layers: a TypeScript MCP server (REST auth, retry-hardened publishing, scheduling, webhooks, +Pulse) and a Python FastAPI sidecar (Hyper/`.tdsx`/`.tds`/`.twbx` authoring) the TS layer spawns +over loopback. See [`docs/architecture.md`](docs/architecture.md). -### Try the live demo - -```bash -# with SERVER/SITE_NAME/PAT_NAME/PAT_VALUE and DEMO_PROJECT set in your environment: -npm run demo -- examples/top_customers.csv -``` +## Prerequisites -It publishes a datasource and a starter workbook from a bundled CSV and prints both Cloud URLs. +- Node ≥ 22.7.5 (matching the official server's floor) +- Python 3.12–3.13 and [`uv`](https://docs.astral.sh/uv/) +- A Tableau Cloud site and a Personal Access Token — the free + [Tableau Developer Program](https://www.tableau.com/developer) works well for testing ## Relationship to the official server @@ -210,12 +316,17 @@ designed so this could be proposed upstream as the authoring extension if the ma make ci # build + lint + test (TS) and ruff + mypy --strict + pytest (sidecar) ``` +Operational how-to (troubleshooting, monitoring, common failures) is in +[`docs/runbook.md`](docs/runbook.md). + ## Sources 1. Tableau. *Official Tableau MCP server.* , 2026. -2. Tableau. *REST API reference — publishing datasources and workbooks.* help.tableau.com, 2026. +2. Tableau. *REST API reference — publishing datasources and workbooks, scheduling, webhooks.* help.tableau.com, 2026. 3. Tableau. *Hyper API documentation.* help.tableau.com/current/api/hyper_api, 2026. 4. Tableau. *Datasource (.tds) and workbook (.twb) XML format.* 2026. +5. Tableau. *Pulse API utilities (reference implementation).* github.com/tableau, 2026. +6. Few, S.; Tufte, E. *Analytical dashboard design principles* (source lineage documented per-rule in [BI_DESIGN.md §9](docs/feature-prompt-authoring/BI_DESIGN.md)). ## License diff --git a/SECURITY.md b/SECURITY.md index 930805b..f590640 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -117,3 +117,53 @@ items (PA-1, PA-3) and the LOW pin (PA-4) have since been **remediated** (see st > Not applicable: there is no blockchain / smart-contract / web3 component, so the > `smart-contract-audit` skill was not used. + +## Vibe-BI expansion surface (E0–E4) — added surface + +**Audited:** 2026-07-17 · branch `feat/exec-dashboards` (slice `db658b2..HEAD`) · STRIDE + +dependency/secret/input review of the branch-new surface only. Baseline (F-01..F-12) and the +prompt-authoring add-on (PA-1..PA-8) are unchanged and still hold. +**Result:** 0 CRITICAL · 0 HIGH · 1 MEDIUM · 2 LOW · 9 PASS. Shippable; VB-02 remediated in-branch. + +### New surface +- E0 Foundation: `src/rest/{errors,retry,vds}.ts`, `get_datasource_fields`. +- E1 Branding: `src/branding/{schema,load,builderBrand}.ts`, `validate_brand`, sidecar brand block. +- E2 Automation: `src/rest/{schedules,webhooks,credentials}.ts`, `create_live_datasource`, + schedule/webhook tools, sidecar `/datasource/live` + `tds_builder.build_live_tds`, and the + operator CLIs `scripts/{generate-cron,cronTemplates,refresh-local}.ts`. +- E3 Pulse: `src/rest/pulse.ts` + Pulse tools (JSON bodies, `/api/-/pulse/*`). +- E4 Stories: sidecar story models + `twb_builder` storyboard emit. +- New npm dep: `yaml@2.9.0` (only). + +### STRIDE (expansion surface) +| Threat | Result | +|---|---| +| Spoofing | Mitigated — new sidecar routes behind the same global `token_guard` (127.0.0.1 + `hmac.compare_digest`); no opt-out. | +| Tampering | Mitigated — new `tsRequest` builders `xmlEscape`/enum/regex-bound all inputs; brand/story reach `.twb` via ElementTree + hex-validated colors. | +| Repudiation | Unchanged — Cloud records the acting user. | +| Information disclosure | Mitigated — embedded DB password never logged (request body never logged), never persisted, absent from the live `.tds`; new error paths redact body, never echo the token. | +| Denial of service | Mitigated — date-coercion + CSV sniff respect the 500 MB / 1 M-row caps; sniff reads only 64 KB. | +| Elevation of privilege | Mitigated — new deletes require `confirm=true`; live publish rejects empty/`Default` project; `overwrite` defaults false. | + +### Findings & remediation status +| ID | Sev | Finding | Status | +|---|---|---|---| +| VB-01 | PASS | Embedded publish credentials never logged (incl. error + retry paths — only the response body hits stderr), `xmlEscape`d, never persisted, absent from the live `.tds`; `tests/secrets.test.ts` asserts the DB password is absent on success and on a 400. | ✅ Verified clean | +| VB-02 | MED | Shell injection in the generated crontab line: `buildCrontabLine` wrapped `file`/`name`/`project`/`persona` in double quotes only, which does not neutralize `$(…)`/backticks/`$VAR`/`"`. Operator-only CLI (not an MCP tool), generate-only, manual install required — not agent-reachable. | ✅ **Fixed in-branch** — all interpolations POSIX single-quote-escaped (`shellQuote` in `scripts/cronTemplates.ts`); adversarial-name tests added (`$(id)`, quote-breakout, embedded quotes). Launchd plist already safe (exec-array + `xmlEscape`). | +| VB-03 | LOW | DB credentials pass as `create_live_datasource` tool args, so they transit the agent/LLM context / client tool-call transcript (outside this server's control). Mitigated by env-var guidance in the tool description. | ⚠️ Accepted — inherent to the feature; consider a future env-var-name indirection. | +| VB-04 | PASS | Webhook HTTPS-only not bypassable: `startsWith("https://")` after `z.url()`; uppercase scheme and leading whitespace are rejected (fail-closed). | ✅ Verified clean | +| VB-05 | PASS | XML injection across all new `tsRequest` builders (schedules/webhooks/credentials/publish) — escaped or enum/regex-bound. Pulse/VDS use `JSON.stringify`. | ✅ Verified clean | +| VB-06 | PASS | `confirm=true` gate on `delete_refresh_schedule` / `delete_webhook` / `delete_pulse_definition`. | ✅ Verified clean | +| VB-07 | PASS | Error-body redaction + no token leakage on new REST/VDS/Pulse error paths. | ✅ Verified clean | +| VB-08 | PASS | `yaml@2.9.0` safe-by-default (no code exec); brand values hex-validated + ElementTree-escaped into XML. | ✅ Verified clean | +| VB-09 | LOW | `brandPath` arbitrary local-file read + YAML alias-expansion DoS. Read is by-design (matches F-10); alias DoS bounded by yaml's default `maxAliasCount`. | ⚠️ Accepted / documented | +| VB-10 | PASS | Ingest date coercion runs after the row clamp; CSV sniff reads only 64 KB; 500 MB pre-read cap enforced. No DoS regression vs PA-1. | ✅ Verified clean | +| VB-11 | PASS | Sidecar new routes behind the global constant-time token guard; Pydantic-validated; 400s echo only agent-supplied fields; live `.tds` carries no credentials. | ✅ Verified clean | +| VB-12 | PASS | `npm audit --omit=dev` → 0 vulnerabilities; `yaml@2.9.0`/`undici@7.28.0`/`zod@3.25.76`/SDK pinned & clean; no sidecar Python-dep changes. | ✅ Verified clean | + +### Recommended fixes for HIGH/CRITICAL +None — no CRITICAL/HIGH on the expansion surface. VB-02 (MEDIUM) was remediated in-branch +before merge. + +> Not applicable: no blockchain / smart-contract / web3 component, so the `smart-contract-audit` +> skill was not used. diff --git a/brand.yaml b/brand.yaml new file mode 100644 index 0000000..37d70dc --- /dev/null +++ b/brand.yaml @@ -0,0 +1,188 @@ +# ============================================================================= +# brand.yaml — Vibe-BI brand kit +# ============================================================================= +# This file controls how every dashboard generated by this MCP server looks +# and feels: colors, fonts, number formats, and named "personas" (CEO, CTO, +# Analyst, ...) that `design_dashboard` can target with `persona: "ceo"` +# instead of (or in addition to) a raw `audience`. +# +# Everything in this file is OPTIONAL. Delete a section — or the whole file — +# and sane built-in defaults take over automatically; you'll get a warning +# (not an error) from the `validate_brand` MCP tool if the file is missing. +# +# Run the `validate_brand` MCP tool any time after editing this file to check +# it parses and to see the list of persona names it defines. +# +# You do NOT need to touch any code to use this file — it is the single +# source of brand truth for this server. +# ============================================================================= + +# ----------------------------------------------------------------------------- +# brand — your company/team identity +# ----------------------------------------------------------------------------- +brand: + # Your company or team name. Shown in dashboard proposal summaries and + # (in a later phase) printed on generated title zones. + name: "My Company" + + # Optional: path to a logo file on disk, for a future logo-embedding + # feature. Leave blank if you don't have one yet. + logoPath: "" + + # Optional: a public URL to your logo (alternative to logoPath). + logoUrl: "" + +# ----------------------------------------------------------------------------- +# palette — the color system every chart/KPI pulls from +# ----------------------------------------------------------------------------- +palette: + # Categorical colors: used to tell DIFFERENT CATEGORIES apart (product + # lines, regions, segments, ...) in bar/line charts. List as many hex + # colors as you like — Tableau cycles through them in order. Defaults to + # the well-known "Tableau 10" palette. + categorical: + - "#4e79a7" + - "#f28e2b" + - "#e15759" + - "#76b7b2" + - "#59a14f" + - "#edc948" + - "#b07aa1" + - "#ff9da7" + - "#9c755f" + - "#bab0ac" + + # Sequential colors: a low-to-high ramp for a SINGLE continuous measure + # (e.g. sales intensity on a filled map). Give 2 or 3 stops, light to dark. + sequential: + - "#c6dbef" + - "#6baed6" + - "#08519c" + + # Diverging colors: a ramp for values that move in TWO directions around a + # midpoint (e.g. profit vs. loss, above/below target). Exactly 3 stops, in + # order: [negative end, midpoint, positive end]. + diverging: + - "#e15759" + - "#f2f2f2" + - "#59a14f" + + # Semantic colors: fixed meanings used for KPI deltas, status indicators, + # and anywhere "good vs. bad" must be unambiguous, independent of the + # categorical/sequential palette above. + semantic: + good: "#59a14f" # e.g. a positive KPI delta, "on target" status + bad: "#e15759" # e.g. a negative KPI delta, "off target" status + neutral: "#898989" # e.g. a flat/unchanged delta, secondary label text + +# ----------------------------------------------------------------------------- +# typography — fonts used across titles, body text, and KPI tiles +# ----------------------------------------------------------------------------- +# "Tableau Bold" and "Tableau Book" ship with every Tableau install, so they +# always render correctly without the viewer needing the font installed. +# Swap in a licensed corporate font here if your organization has one. +typography: + title: + font: "Tableau Bold" + size: 24 + color: "#1f1f1f" + + body: + font: "Tableau Book" + size: 11 + color: "#4d4d4d" + + # BAN = "Big Number" — the large hero figure inside a KPI tile (e.g. the + # "$1.2M" in a Total Sales tile). Deliberately large and bold so it reads + # from across a room. + ban: + font: "Tableau Bold" + size: 36 + +# ----------------------------------------------------------------------------- +# formats — number/currency/percent display formats +# ----------------------------------------------------------------------------- +# Values use Tableau's format-string syntax (e.g. "#,##0" or "$#,##0.00"). +formats: + currency: "$#,##0" + percent: "0.0%" + number: "#,##0" + # When true, large numbers are abbreviated (1.2M instead of 1,234,567) in + # KPI tiles and axis labels. + compact: true + +# ----------------------------------------------------------------------------- +# rules — free-text design do's and don'ts +# ----------------------------------------------------------------------------- +# Documentational only: these are NOT enforced by code today. They're read by +# `validate_brand` and future planner phases as house-style notes for anyone +# — human or agent — building a dashboard for your organization. +rules: + - "Do lead every executive dashboard with a KPI band of 3-4 metrics." + - "Do use direct labeling on bars instead of a legend when there are 5 or fewer categories." + - "Do use the sequential palette for magnitude (e.g. sales intensity), never the categorical palette." + - "Don't use pie charts — prefer bar charts or 100% stacked bars." + - "Don't use gauges — prefer bullet graphs (Stephen Few)." + - "Don't use 3D effects or drop shadows on marks." + +# ----------------------------------------------------------------------------- +# personas — named audiences +# ----------------------------------------------------------------------------- +# `design_dashboard` accepts `persona: "ceo"` instead of (or alongside) +# `audience`. Add, remove, or rename personas freely — no code changes +# required. Each persona is a key (the name you pass as `persona:`) mapped to: +# +# base (required) one of: exec | analyst | operational | mixed +# — the underlying audience profile this persona builds on. +# maxSheets (optional) override the max number of chart sheets. +# kpiEmphasis (optional) high | medium | low — how prominent the KPI +# band should be (documentational hint for now). +# preferredArtifact (optional) dashboard | story | pulse — the artifact +# type this persona prefers when the agent has a choice. +# tone (optional) concise | detailed — wording style for +# generated titles/summaries. +# chartDeny (optional) list of chart types this persona never wants +# to see (e.g. ["pie", "scatter"]). +# notes (optional) free-text reminder for anyone designing for +# this persona. +personas: + ceo: + base: exec + maxSheets: 3 + kpiEmphasis: high + preferredArtifact: dashboard + tone: concise + notes: "Headline numbers only. No axis labels, no jargon." + + cto: + base: exec + maxSheets: 4 + kpiEmphasis: medium + preferredArtifact: dashboard + tone: concise + chartDeny: ["pie"] + notes: "Cares about trend and reliability metrics as much as revenue." + + slt_manager: + base: mixed + maxSheets: 6 + kpiEmphasis: medium + preferredArtifact: dashboard + tone: detailed + notes: "Senior Leadership Team manager — wants a broader view than the CEO." + + analyst: + base: analyst + kpiEmphasis: low + preferredArtifact: dashboard + tone: detailed + notes: "Wants full axis labels, drill-down detail, and every relevant dimension." + + client: + base: operational + maxSheets: 4 + kpiEmphasis: high + preferredArtifact: dashboard + tone: concise + chartDeny: ["scatter", "map"] + notes: "External audience — keep it simple, on-brand, and jargon-free." diff --git a/docs/adr.md b/docs/adr.md index e76c7cc..eb8b472 100644 --- a/docs/adr.md +++ b/docs/adr.md @@ -54,5 +54,9 @@ and self-contained. a follow-up issue). Snowflake/Postgres drivers are optional `connectors` extras so CI exercises the authoring path without DB drivers; CSV always works. +> **SUPERSEDED (2026-07-17):** the deferral was lifted — `create_live_datasource` ships live +> Snowflake/Presto datasources with embedded credentials. See +> [ADR-0009](adr/0009-live-connection-credential-handling.md) for the current decision. + **Consequences.** Smaller, reliable surface for v0.1; CI stays fast and driver-free; live connections are a clean future addition. diff --git a/docs/adr/0006-brand-kit-yaml-persona-resolution.md b/docs/adr/0006-brand-kit-yaml-persona-resolution.md new file mode 100644 index 0000000..cab2f67 --- /dev/null +++ b/docs/adr/0006-brand-kit-yaml-persona-resolution.md @@ -0,0 +1,73 @@ +# ADR-0006 — `brand.yaml` as a defaulted YAML file, personas resolved only at the tool layer + +**Status:** Accepted +**Date:** 2026-06-29 +**Feature:** Branding system + design-excellence layer (Phase E1) + +--- + +## Context + +`design_dashboard` needed a way to apply organization-specific color palettes, typography, number +formats, and audience presets ("the CEO wants 3 sheets and concise wording") without hardcoding +them into the planner or requiring a code change per customer/team. Two questions needed answers: + +1. **Where does brand configuration live, and in what format?** Options: a TypeScript config + module, JSON, or YAML; embedded defaults vs. a required file. +2. **Where does persona resolution happen?** Options: inside the pure planner, or at the MCP tool + layer before the planner is invoked. + +## Decision + +**`brand.yaml` at the repo root, validated by a zod schema where every field is optional and +defaulted, loaded and resolved only by `src/tools/designDashboard.ts` (and, when branding a build, +`src/tools/buildFromPlan.ts`) — never by the planner or the sidecar directly.** + +Specifically: + +- `BrandFileSchema.parse({})` (an empty object) already produces a fully-populated, valid + `BrandFile` — "file absent" and "file present but sparse" are handled identically, both falling + back to defaults that mirror the color choices already audited in the reference `.twbx` files. +- `personas` is a free-form `Record` map, so adding a persona is a YAML + edit, not a code change. Each persona's `base` must resolve to one of the planner's existing + `AudienceEnum` values, keeping the branding layer downstream of (never mutating) the planner's + contract. +- `loadBrand()`/`resolvePersona()` are the **only** I/O in the branding layer. `design_dashboard` + reads the file once, resolves the persona (audience + `maxSheets`/`chartDeny`/`kpiEmphasis`/ + `preferredArtifact`/`tone` overrides), and passes plain, already-resolved values into + `generatePlan()`. The planner itself never touches the filesystem. + +## Alternatives considered + +**Embed brand config in TypeScript (a `.ts` module):** rejected — a non-developer editing colors +or adding a persona would need to touch source code and understand the build step, defeating the +goal of a config surface a brand/design team member can edit directly. + +**JSON instead of YAML:** rejected — YAML supports comments, which `brand.yaml` uses extensively to +document every field inline (the file is meant to be self-documenting for a non-engineer editor). +Adds one new runtime dependency (`yaml@2.9.0`). + +**Resolve personas inside the planner:** rejected — would require the planner (previously pure, +filesystem-free, and independently unit-tested per ADR-0005) to perform file I/O, breaking its +"deterministic given explicit inputs" testability and coupling it to a specific config file format. + +## Consequences + +**Positive:** + +- `validate_brand` can report `{ valid, warnings, personas, summary }` without ever throwing, + because a malformed file is a validation-layer concern (caught by `loadBrand`), not a planner + concern. +- The planner's own tests (`tests/planner*.test.ts`) remain filesystem-free and fast; brand/persona + wiring gets its own focused test files (`tests/branding.test.ts`, `tests/builderBrand.test.ts`). +- Five personas ship as defaults; adding a sixth is a YAML edit with no deploy. + +**Constraints this decision enforces:** + +- Every new persona override field must be threaded explicitly through `designDashboard.ts` into + `PlanInput` (redeclared enums, not imported, to avoid the planner depending on `branding/`) — + there is no implicit pass-through. +- `build_from_plan` re-reads `brand.yaml` independently (it does not receive the resolved brand + object from `design_dashboard`, since the two tool calls are stateless and may be minutes apart) + — a `brand.yaml` edit between `design_dashboard` and `build_from_plan` calls changes the applied + brand, which is the intended "single source of brand truth" behavior, not a bug. diff --git a/docs/adr/0007-propose-confirm-loop.md b/docs/adr/0007-propose-confirm-loop.md new file mode 100644 index 0000000..22f1ba9 --- /dev/null +++ b/docs/adr/0007-propose-confirm-loop.md @@ -0,0 +1,79 @@ +# ADR-0007 — `design_dashboard` returns a confirmable `DashboardProposal`, not a raw plan + +**Status:** Accepted +**Date:** 2026-07-16 +**Feature:** Propose→confirm loop (Slice 5, on top of ADR-0005's stateless-planner decision) + +--- + +## Context + +`design_dashboard` originally returned a `DashboardPlan` — a structural JSON object (sheet specs, +mark types, layout) meant for `build_from_plan` to consume. That shape is precise but not +human-readable: an agent relaying it to a user would either dump raw JSON or hand-write a summary +that could drift from what the plan actually contains. There was no built-in "let the user review +and revise before anything publishes" step in the tool contract itself — an agent had to invent +one, inconsistently, on top of a bare plan. + +## Decision + +**`design_dashboard` returns a `DashboardProposal` (`kind: "proposal"`) for every mode except +`interview`, derived from the `DashboardPlan` by a pure, deterministic projection +(`buildProposal()` in `src/planner/proposal.ts`). The proposal embeds the plan verbatim in its +`plan` field. `build_from_plan` still only accepts a `DashboardPlan` — the agent must extract +`proposal.plan` before calling it.** + +The proposal carries: +- `summary` — a one-paragraph natural-language description, including persona/brand provenance and + tone (`concise` trims to the first sentence + layout line; `detailed` keeps the full text). +- `kpiStrip` — one entry per KPI tile, with primary/comparison/delta measures and up/down-is-good + direction. +- `views` — one entry per chart sheet, with a human-readable encoding summary (e.g. "bar: Sales by + Category, colored by Segment"). +- `layoutSummary` — a sentence describing the dashboard's zone layout. +- `storyOutline` (optional) — ordered captions when the plan carries a `storyArc`. +- `openQuestions` (optional) — honest callouts: unresolved placeholder tokens, a persona's + `preferredArtifact` this build path doesn't produce, etc. + +The agent-facing contract is a two-step loop: present the proposal → on "confirm", call +`build_from_plan(proposal.plan)` verbatim; on "change X", re-call `design_dashboard` in `directed` +mode with updated `directions` for a fresh proposal. No state is held server-side between calls — +this is layered on top of, not a replacement for, ADR-0005's stateless-planner decision. + +## Alternatives considered + +**Return the raw `DashboardPlan` and let each calling agent format its own summary:** rejected — +produces inconsistent, potentially inaccurate summaries across agents/prompts, and duplicates +formatting logic that belongs in one deterministic, unit-tested place. + +**Make `design_dashboard` itself hold a pending-proposal state and accept a `confirm: true` flag to +publish:** rejected — reintroduces server-side session state, which ADR-0005 explicitly rejected +for the interview mode and which an MCP stdio process has no reliable mechanism for (no shared +store across restarts or concurrent agent sessions). + +**Have `build_from_plan` accept either a `DashboardPlan` or a full `DashboardProposal` (and unwrap +it internally):** rejected for this phase — keeping `build_from_plan`'s input contract to exactly +`DashboardPlan` keeps its own validation (`schemaVersion`/`kind: "plan"` guard, placeholder-token +rejection, audience re-validation) unambiguous; the agent extracting `proposal.plan` is a one-line +step already implied by "pass it verbatim." + +## Consequences + +**Positive:** + +- Every proposal is reproducible and unit-testable: `buildProposal(plan)` is pure, so + `tests/planner-slice5.test.ts` asserts exact summary/kpiStrip/views/layoutSummary shapes without + a live model or a live site. +- The confirm step is explicit and auditable: `build_from_plan` is still the only side-effecting + tool in the family, and it is always a separate call from `design_dashboard`. +- `openQuestions` gives the agent a structured place to surface caveats (placeholder fields, an + unsupported `preferredArtifact`) instead of silently proceeding or silently dropping the + preference. + +**Constraints this decision enforces:** + +- Any new plan-level field an agent should see in a human-readable form must get a corresponding + projection in `buildProposal()` — the proposal is not a passthrough of the plan. +- `DashboardProposalSchema` requires at least one `views` entry; an all-KPI-tile plan synthesizes a + minimal view from its first sheet so the schema still validates (a degenerate case documented in + `proposal.ts`, not expected in well-formed exec plans). diff --git a/docs/adr/0008-rest-retry-hardening.md b/docs/adr/0008-rest-retry-hardening.md new file mode 100644 index 0000000..f25657d --- /dev/null +++ b/docs/adr/0008-rest-retry-hardening.md @@ -0,0 +1,80 @@ +# ADR-0008 — Shared, idempotency-aware bounded retry for REST/VDS/Pulse calls + +**Status:** Accepted +**Date:** 2026-07-17 +**Feature:** REST hardening (Phase E2, Foundation slice) + +--- + +## Context + +As the tool surface grew to include scheduling, webhooks, VDS field metadata, and Pulse — all of +which are more failure-prone in practice than the original sign-in/publish path (VDS/Pulse can be +disabled per-site; Cloud rate-limits aggressively under load) — bare `undici` calls with no retry +meant a single transient `429`/`503` would fail an entire tool call outright, even though the +underlying operation was often safe to retry. + +Two constraints shaped the decision: (1) some operations are **not** safe to blindly retry (a POST +that creates a Pulse definition could double-create if the original request actually succeeded but +the response was lost), and (2) errors needed to be structured (`status`/`code`/`retriable`) so +callers and tests could branch on them instead of regexing a message string. + +## Decision + +**One shared `withRetry()` loop (`src/rest/retry.ts`), explicitly Tableau-agnostic, reused by +`restClient.ts`, `rest/vds.ts`, and `rest/pulse.ts`. Retry eligibility is a two-part gate: the +HTTP status must be in a small allowlist (429/502/503/504 — never 401, never any other 4xx), AND +the specific call must be marked `idempotent` by its caller.** + +- `TableauApiError` (`src/rest/errors.ts`) replaces bare `Error` for every REST/VDS/Pulse failure, + carrying `status`/`method`/`path`/`code`/`summary`/`detail`/`retriable`/`retryAfterMs`. The + exception message never echoes the raw response body — only status/method/path/summary — so a + logged or agent-surfaced error can never leak upstream internals. +- Backoff is "full jitter" exponential (AWS-recommended): `random() * min(base * 2^(n-1), cap)`, + honoring an upstream `Retry-After` header verbatim when present. Deterministic in tests via + injectable `sleep`/`jitterFn`/`now`. +- Idempotency is decided **per call site**, not per HTTP verb blindly: GET is always idempotent; + POST/DELETE are marked idempotent only where the operation is genuinely safe to repeat (e.g. + `read-metadata` is a POST with no side effects, so VDS marks it idempotent despite the verb). + `rest/pulse.ts` takes an even more conservative stance than the rest of the codebase — it never + retries POST or DELETE at all, because Pulse's own response shapes (especially delete) are + themselves VERIFY-LIVE unknowns, so retrying blind is a bigger risk there than elsewhere. + +## Alternatives considered + +**A generic HTTP-client-level retry library (e.g. `undici`'s built-in retry, or `p-retry`):** +rejected — none of the evaluated options exposed both (a) per-call-site idempotency control and +(b) Tableau's specific error-body shape for classification, without wrapping in custom code anyway; +a small hand-rolled module was less code than gluing a generic library to Tableau semantics. + +**Retry every POST unconditionally on 5xx (assume "the request probably failed cleanly"):** +rejected — a lost response after a successful server-side write (e.g. webhook or Pulse definition +created, but the `201` never arrived) would silently double-create the resource on retry. The +per-call idempotency flag makes this an explicit decision at each call site instead of a blanket +assumption. + +**No retry at all; let the calling agent retry the whole tool call:** rejected — pushes transient- +failure handling up to every agent integration, with no shared backoff/jitter policy, and no way +to distinguish "definitely gone, don't retry" (401, most 4xx) from "try again" (429/5xx) without +duplicating Tableau's error-body parsing at the agent layer. + +## Consequences + +**Positive:** + +- `get_datasource_fields`, `schedule_refresh`, `create_webhook`, and the Pulse tools all get + automatic resilience to Cloud rate-limiting and transient 5xx without each tool reimplementing + backoff. +- `tests/restRetry.test.ts`, `tests/retry.test.ts`, `tests/vds.test.ts`, `tests/pulse.test.ts`, and + `tests/webhooks.test.ts` all assert retry behavior deterministically (fake clock, fixed jitter) + with zero real waiting. +- Errors are typed end-to-end: `err.retriable` is a single source of truth instead of a duplicated + status-code list scattered across call sites. + +**Constraints this decision enforces:** + +- Any new REST/VDS/Pulse module must classify its own idempotency per call site — there is no + default that "does the right thing" automatically; getting this wrong for a non-idempotent + mutation risks duplicate server-side effects on retry. +- `rest/pulse.ts`'s narrower policy (GET-only retry) must be revisited once Pulse's delete/create + response shapes are confirmed against a live site — see ADR-0011. diff --git a/docs/adr/0009-live-connection-credential-handling.md b/docs/adr/0009-live-connection-credential-handling.md new file mode 100644 index 0000000..ede86f1 --- /dev/null +++ b/docs/adr/0009-live-connection-credential-handling.md @@ -0,0 +1,82 @@ +# ADR-0009 — Live connections: no credentials in the `.tds`, embedded only at publish time + +**Status:** Accepted +**Date:** 2026-07-17 +**Feature:** Live Cloud connections (Phase E2 slice C) — supersedes ADR-0004's "extract-only" scope + +--- + +## Context + +ADR-0004 deferred live-connection datasources for v0.1 in favor of extract-only publishing. Phase +E2 needed to add live Snowflake/Presto connections (no locally materialized extract) while keeping +the same security posture the rest of the project holds: credentials must never be written to +disk, logged, or embedded in a document that could be inspected/leaked independently of Tableau's +own access controls. + +Two design questions: (1) where do database credentials live between the tool call and the +publish, and (2) how does the system handle the one authentication mode (Snowflake key-pair) that +is fundamentally impossible to configure over the REST publish API. + +## Decision + +**The `.tds` file the sidecar builds carries only connection *topology* — server, warehouse, +schema, database, table. Credentials never touch it.** `create_live_datasource`'s +`credentials.username`/`credentials.password` are passed straight through the TypeScript tool call +into `TableauRestClient.publishDatasource`'s `credentials` option, which +`rest/credentials.ts`'s `buildConnectionCredentialsXml()` renders as an in-memory +`` XML fragment, nested +inside the same multipart publish request as the `.tds` bytes — and nowhere else. Every attribute +is `xmlEscape`d; `restClient.ts` only ever logs the *response* body of a failed call, never the +outgoing request body, so a failed publish can never leak the password to stderr (asserted +standingly in `tests/secrets.test.ts` and `tests/credentials.test.ts`). + +**Snowflake key-pair authentication is explicitly rejected, not silently attempted.** Tableau's +Publish Datasource REST API only supports embedding username/password or OAuth credentials — +key-pair auth requires Tableau Desktop. `create_live_datasource` and the sidecar's +`/datasource/live` route both check the requested `authentication` mode and throw/400 with an +actionable message ("configure key-pair auth in Desktop and publish from there instead") before any +network call, rather than emitting XML that would build successfully but never actually +authenticate. + +## Alternatives considered + +**Write credentials into a temp `.tds` file, then delete it after publish:** rejected — even a +short-lived credential-bearing file on disk is an unnecessary exposure window (backup tools, +crash-before-cleanup, other processes) that the in-memory-XML-fragment approach avoids entirely. + +**Accept environment-variable-only credentials (server reads `SNOWFLAKE_USER`/`SNOWFLAKE_PASSWORD` +itself):** rejected for this phase — `config.ts`'s env surface is deliberately identical to the +official `@tableau/mcp-server` (`SERVER`/`SITE_NAME`/`PAT_NAME`/`PAT_VALUE` only); adding +connector-specific env vars would mean a fixed set of env names could not represent an arbitrary +number of different live connections a single server instance might publish across a session. The +tool description explicitly recommends the *calling agent* source credentials from its own +environment and pass them as call arguments, keeping the MCP server's own env surface unchanged. + +**Silently downgrade an unsupported Snowflake auth mode to username/password:** rejected — would +either fail confusingly downstream or, worse, appear to succeed while never actually connecting. +An explicit, actionable error at the tool boundary is safer than a silent, wrong fallback. + +## Consequences + +**Positive:** + +- The security posture matches the rest of the project: no new credential-at-rest surface was + introduced by this feature. +- The key-pair rejection is a genuinely honest capability boundary, not a documented-but-broken + feature — an agent gets a clear "not possible this way" instead of a confusing runtime failure. +- Two independent checks (TypeScript tool layer, Python sidecar route) both reject key-pair auth, + so a caller bypassing the TS tool (e.g. a direct sidecar call in testing) still gets the guard. + +**Constraints this decision enforces:** + +- `LiveConnectionSpec`/`LiveDatasourceRequest` must never grow a credentials field — this is + enforced by convention (documented in both `src/sidecar.ts` and `sidecar/server.py`), not by a + runtime check, so a future edit must preserve this boundary deliberately. +- The exact XML attribute spelling for Snowflake/Presto connector classes + (`sidecar/tds_builder.py`'s `SNOWFLAKE_ATTRS`/`PRESTO_ATTRS`) remains VERIFY-LIVE — this decision + covers *where credentials live*, not *whether the connection XML is byte-perfect* — until + confirmed against a Desktop-exported `.tds`. +- Presto/Trino's `useRemoteQueryAgent` default (`true`) assumes Tableau Bridge is required unless + explicitly told otherwise — a deliberately conservative default given Cloud generally cannot + reach a private-network Presto cluster directly. diff --git a/docs/adr/0010-local-file-refresh-design-around.md b/docs/adr/0010-local-file-refresh-design-around.md new file mode 100644 index 0000000..ad07e24 --- /dev/null +++ b/docs/adr/0010-local-file-refresh-design-around.md @@ -0,0 +1,86 @@ +# ADR-0010 — Local-file datasources: cron/launchd design-around instead of a Bridge dependency + +**Status:** Accepted +**Date:** 2026-07-17 +**Feature:** Local-file refresh automation (Phase E2 slice C) + +--- + +## Context + +`schedule_refresh` lets an agent create a recurring Cloud extract-refresh task. But Tableau Cloud +can only **execute** that schedule for a connection it can reach and re-query itself — a live +Snowflake/Presto connection with embedded credentials, for example. A datasource published from a +local file (CSV/Excel/local Hyper extract, e.g. anything from `create_datasource_from_file`) has no +server-side source Cloud can go back to; refreshing it requires either **Tableau Bridge** (a +customer-installed agent that proxies Cloud's refresh requests to an on-prem/local data source) or +re-publishing the file from wherever it lives. Bridge is a separate, licensed, installed product +that this MCP server cannot configure or assume is present. + +The `schedule_refresh` REST call itself does not fail for a file-based datasource — Cloud accepts +the schedule and creates the task; only the *scheduled run* fails, and that failure is not visible +at schedule-creation time. This makes the gap easy to hit silently. + +## Decision + +**Ship a self-contained, local re-ingest-and-republish path (`scripts/refresh-local.ts`) plus a +pure template generator (`scripts/generate-cron.ts` / `cronTemplates.ts`) that emits a crontab +line and a macOS launchd plist an operator reviews and installs manually.** Nothing in this project +installs a cron job or loads a launchd agent itself. + +- `refresh-local.ts` re-runs the same file-ingest path `create_datasource_from_file` uses + (encoding/delimiter/type auto-sniffed), then republishes with `overwrite=true`. It exits 0/1 + cleanly and logs exactly one summary line per outcome, so cron/launchd's own failure-alerting + (mail, log aggregation) works without extra plumbing. +- `generate-cron.ts` only **prints** (or, with `--write`, saves under the gitignored + `scripts/cron/`) the crontab line and plist text — it never runs `crontab -e` or + `launchctl load` itself, so the operator always makes the final "yes, install this" decision. +- `schedule_refresh`'s tool description and returned `note` field always state the Bridge caveat + unconditionally (not just when a failure is detected) — the REST API gives no reliable signal + that would let the server distinguish "this will actually run" from "this will silently fail + every time," so honesty requires stating the caveat every time rather than only when detectable. + +## Alternatives considered + +**Implement or bundle a Tableau Bridge client:** rejected — Bridge is a separate, licensed Tableau +product with its own installation and site-configuration requirements; reimplementing or bundling +it is out of scope for an MCP authoring server and would require infrastructure this project has no +way to provision or test against. + +**Silently fail `schedule_refresh` for anything that isn't obviously a live connection:** rejected +— the REST API response gives no reliable signal to detect "this datasource has no cloud-reachable +source" at schedule-creation time (a datasource published from `create_live_datasource` vs. +`create_datasource_from_file` looks similar enough in the response that a heuristic would be +fragile); an honest, always-included caveat is more defensible than an unreliable auto-detection +that could produce false confidence either way. + +**Auto-install the cron/launchd job as part of `generate-cron.ts`:** rejected — modifying a user's +crontab or LaunchAgents directory without an explicit, separate confirmation step is the kind of +system-level side effect this project avoids elsewhere (compare `delete_content`'s `confirm: true` +gate); a generated, reviewable artifact respects the same principle. + +## Consequences + +**Positive:** + +- Operators get a genuinely usable path to keep a file-based datasource fresh without needing + Bridge, proven live (`npm run refresh:local` republished the Superstore datasource in 4.7s + against a real Cloud site — see `ACCEPTANCE.md`). +- The generated artifacts are pure functions of their inputs (`cronTemplates.ts` has no I/O), + making them fully unit-testable (`tests/cronTemplates.test.ts`, `tests/generateCron.test.ts`) + without touching a real crontab or LaunchAgents directory in CI. +- `schedule_refresh`'s honesty-by-default caveat means an agent presenting the tool's output to a + user always surfaces the Bridge risk, rather than only after a scheduled run has already failed + silently on the customer's site. + +**Constraints this decision enforces:** + +- `refresh-local.ts` and `generate-cron.ts` must remain scripts (`npm run refresh:local` / + `npm run cron:generate`), not MCP tools — they are operator-run automation, not agent-callable + actions, since installing a recurring local job is a machine-level decision outside a single + conversational tool-call's blast radius. +- Any datasource name, file path, or project name interpolated into the generated crontab line + must be shell-escaped (POSIX single-quoting, `cronTemplates.ts`'s `shellQuote()`) — an + agent-supplied name is untrusted input from the crontab's perspective, and double-quoting alone + does not neutralize `$(...)`/backtick/`$VAR` substitution (see `SECURITY.md`'s VB-02 finding, + remediated in the same commit line as this ADR). diff --git a/docs/adr/0011-pulse-best-effort-payload.md b/docs/adr/0011-pulse-best-effort-payload.md new file mode 100644 index 0000000..9d27198 --- /dev/null +++ b/docs/adr/0011-pulse-best-effort-payload.md @@ -0,0 +1,87 @@ +# ADR-0011 — Pulse tools shipped code-complete against a best-effort payload, honestly flagged pending a live fixture + +**Status:** Accepted (code-complete; live proof blocked) +**Date:** 2026-07-17 +**Feature:** Tableau Pulse metric definitions + metrics (Phase E3) + +--- + +## Context + +Tableau Pulse has a documented REST surface, but Tableau's own official reference implementation +(`pulse-api-utilities`) only ever **clones an existing definition it fetches**, it never +constructs a `create-definition` request body from scratch. The `basic_specification` internals +(and several enum families — granularity, range, comparison, sentiment, number format) are +therefore specified only by example in the one sample body the reference repo ships, not by a +formal schema. A first attempt to build `POST /api/-/pulse/definitions` from that one example +returned a bare `400` against a real Cloud site with no field-level detail in the error. + +The project's stated invariant (validate at the boundary, fail loud, never guess silently) made an +option of simply shipping nothing until the shape was locked untenable — the E3 milestone otherwise +had no path to make progress without user-provided Pulse-UI access to create a de-risking fixture. + +## Decision + +**Ship all four Pulse tools now, built to mirror Tableau's official example body field-for-field, +with every token beyond the few the reference material explicitly confirms marked `VERIFY-LIVE` in +both code comments and the tool descriptions surfaced to the agent — and state the live-blocked +status honestly in `ACCEPTANCE.md` and `docs/tool_reference.md` rather than implying the tools are +fully proven.** + +- `buildCreateDefinitionBody()` (`src/rest/pulse.ts`) replicates the exact nesting Tableau's + reference repo calls out as easy to get wrong: `extension_options` / `representation_options` / + `insights_options` / `comparisons` / `datasource_goals` / `related_links` / `certification` are + top-level siblings of `specification`, not nested inside it. + `create_pulse_definition` additionally runs a **VDS-backed pre-flight check** + (`validatePulsePreflight()`) that verifies the measure and time-dimension fields actually exist + on the datasource and that the time dimension's VDS `dataType` looks date-like — this catches an + entire class of caller mistakes (typo'd field name, non-date time dimension) independently of + whether the Pulse-specific payload shape itself is fully correct. +- Every enum (`PulseGranularitySchema`, `PulseRangeSchema`, `PulseComparisonSchema`, etc.) keeps + only the one confirmed default from the reference sample and documents every other member as + unconfirmed, so a caller who sticks to defaults is on the most-validated path. +- The de-risking plan is explicit and recorded, not left implicit: create one definition in the + Pulse UI, `GET` it back, and use the real response to correct `buildCreateDefinitionBody()` / + lock a fixture — tracked as the concrete next step in `ACCEPTANCE.md`'s E3 section. + +## Alternatives considered + +**Defer all Pulse tools until a live fixture is available:** rejected — would block the entire E3 +milestone indefinitely on a resource (Pulse UI access to manually create one definition) outside +this repo's automated control, with no incremental value shipped in the meantime. Shipping +code-complete, pre-flight-validated tools that are honestly flagged VERIFY-LIVE lets an operator +who *does* have Pulse UI access attempt live creation today, while making the current limitation +impossible to miss. + +**Reverse-engineer the payload shape from Tableau's web client network traffic instead of the +official reference repo:** rejected as the primary source — the official `pulse-api-utilities` +repo is Tableau's own maintained reference and the more durable source to track for future +changes; network-traffic reverse-engineering was not pursued as it was unnecessary given the +official repo's field names, and would be a weaker citation for future maintainers. + +**Suppress the VERIFY-LIVE caveats to present the tools as fully proven, matching every other +publish tool's confidence level:** rejected — directly conflicts with this project's standing +"honest caveats over false confidence" discipline (mirrored in `schedule_refresh`'s always-included +Bridge note and `create_live_datasource`'s key-pair rejection); a Pulse tool that silently fails in +production because of an unconfirmed enum token is worse than one whose docs say so upfront. + +## Consequences + +**Positive:** + +- An operator with Pulse UI access has a concrete, documented path to close the gap (create → GET → + correct the client) rather than starting from zero. +- The VDS pre-flight check provides real value today regardless of the Pulse-specific payload + shape's final correctness — bad field references are caught before ever reaching Pulse. +- `rest/pulse.ts`'s retry policy is deliberately more conservative than the rest of the codebase + (GET-only — see ADR-0008) specifically *because* of this shape uncertainty, so a retried DELETE + can't compound an already-unconfirmed failure mode. + +**Constraints this decision enforces:** + +- `docs/tool_reference.md` and `README.md` must both state the live-blocked status of Pulse + creation explicitly wherever the tools are described — silently dropping this caveat in future + doc edits would misrepresent tool maturity to an agent/user. +- Any code change to `buildCreateDefinitionBody()`/`buildCreateMetricBody()` after the fixture is + locked should update this ADR's status from "code-complete; live proof blocked" to "Accepted" and + remove the now-resolved VERIFY-LIVE markers it corrects. diff --git a/docs/adr/0012-story-shared-dashboards-container.md b/docs/adr/0012-story-shared-dashboards-container.md new file mode 100644 index 0000000..1e7d2a6 --- /dev/null +++ b/docs/adr/0012-story-shared-dashboards-container.md @@ -0,0 +1,88 @@ +# ADR-0012 — Stories are `` elements inside one shared `` container + +**Status:** Accepted +**Date:** 2026-07-17 +**Feature:** Tableau Stories (Phase E4) + +--- + +## Context + +A Tableau Story is, structurally, a special kind of dashboard (``) +that flips between "story points," each pointing at an existing worksheet or regular dashboard. +`build_from_plan` needed to add an optional story to the same `.twbx` it already builds a regular +dashboard into, driven by `plan.storyArc` (emitted deterministically by the planner when a persona +prefers `"story"` artifacts or the business question uses narrative language — see +`docs/architecture.md`'s planner section). + +While implementing this, XSD validation surfaced a real structural bug: the existing builder code +wrote each dashboard-producing call (`_build_dashboard` for the regular dashboard, then a +would-be-separate call for the story) into its **own** `` wrapper element, producing +two sibling `` elements in the `.twb`. The official TWB XSD (vendored and gated in +`test_twb_schema_validation.py` per the ecosystem-review ADR) only allows **one** `` +element per workbook. + +## Decision + +**A workbook has exactly one `` container. Every regular dashboard is appended to it +first, then every story (``) is appended after — both kinds are +siblings inside the same element, differentiated only by the `type` attribute.** + +- `_build_story()` returns a `` `ET.Element`, structurally parallel to + `_build_dashboard()`'s regular-dashboard element — same `