feat: vibe-BI platform — branding+personas, connectivity+refresh, Pulse, stories (14→27 tools) - #5
Closed
SebAustin wants to merge 26 commits into
Closed
feat: vibe-BI platform — branding+personas, connectivity+refresh, Pulse, stories (14→27 tools)#5SebAustin wants to merge 26 commits into
SebAustin wants to merge 26 commits into
Conversation
Unblocks the 'Super Sample Superstore' migrated-data file (UTF-16 LE, TAB- separated, measures stored as '$16'/'20%'/'($5)' strings): - file_to_dataframe gains optional encoding/sep (plumbed through FileRequest → FileArgs); when omitted it auto-sniffs the BOM (utf-16/utf-8-sig) and the header delimiter (tab vs comma). - _coerce_formatted_numerics converts currency/percent/accounting-negative string columns to numeric, but only when >=90% of a column parses as a number so genuine text dimensions (names, IDs, categories) are untouched. Verified on the real file: 9994x68 loads; Sales/Profit/CP/PP/Difference become numeric (Sales sums to $2,297,354), Region/State/Order ID stay text. +13 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 2 of exec-dashboards. All additions OPTIONAL/additive (schemaVersion stays
1; bump deferred to when required-by-kind validation lands):
- MarkTypeEnum += scatter, map_filled; new SheetKindEnum (chart|kpi_tile)
- SheetSpec += kind/color/kpi/scatter/geo (the rich-encoding fields)
- DashboardPlan += dashboardTitle/subtitle/textZones/layoutGrammar
- DatasourceSpec += encoding/delimiter (the sidecar already honors these)
- DashboardProposalSchema (the propose->confirm contract) + isDashboardProposal
- SheetModel/DashboardWorkbookRequest + sidecar.ts carry the new fields end-to-end
Slice 3 makes the builder EMIT them; Slice 4 makes the planner PRODUCE them.
Backward-compat verified: a plain {title,markType,rows,cols,measures} sheet and a
minimal plan still validate. Gate: 128 TS + 149 Python = 277 tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3A of exec-dashboards. twb_builder._build_worksheet now emits, mirroring wb1, only when the new optional SheetSpec fields are present: - color encoding: <encodings><color column='[ds].[:Measure Names]|instance'/> (G-01) - scatter: <mark class='Circle'/> with x-measure on cols, y-measure on rows, optional breakdown on color (G-02) - KPI tile (kind=kpi_tile): Automatic mark with primary+comparison+delta each on a <text> encoding (value + delta) Each element XSD-validated (official gate) + structure-matched to wb1, across both the sqlproxy and embedded build paths, with plain-sheet regression guards (a plain bar/line/text sheet emits no <encodings> — byte-unchanged). +48 tests. Gate: 128 TS + 197 Python = 325 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ne layout Slice 3B of exec-dashboards. _build_dashboard (mirroring wb6/wb7) now emits, when the new DashboardPlan fields are present: - <zone type-v2='text'> title/subtitle via <formatted-text><run bold fontsize> - header/footer textZones - kpi_band_over_charts: nested layout-flow (vert -> [title, horz KPI band, chart flow]); sheets routed to band (kpiTileTitles / kind=kpi_tile) vs charts Threaded through /workbook/dashboard -> build_embedded_twbx -> _build_dashboard. Default (no-title, tiled) path stays byte-identical (determinism guard); dashboard <viewpoints> still lists every worksheet (400011 guard); text-zone labels never pollute viewpoints. +25 XSD-gated tests. Gate: 128 TS + 222 Python = 350 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3C of exec-dashboards (Slice 3 complete). For a map_filled sheet with a geo spec, twb_builder now mirrors wb1's 'Sales Distribution by State' choropleth: - stamps semantic-role on the geo datasource column ([State].[Name] etc.) in both build paths (threaded geo-role map into _build_federated_datasource) - emits <mapsources>, [Latitude/Longitude (generated)] on rows/cols, a two-pane structure with <mark class='Map'> Automatic + <geometry> + colorMeasure on color - geo field + color measure added to datasource-dependencies Render confidence: HIGH on the geocoding-trigger structure (matches wb1 verbatim); live render proven in the demo slice. Superstore uses full state names → geocodes. Non-map paths byte-unchanged. +25 XSD-gated tests. Gate: 128 TS + 247 Python = 375. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…map) + interview Slice 4 of exec-dashboards. The deterministic planner now PRODUCES rich plans: - fields.ts: classify CP/PP/Difference as period_compare measures; geoRole detection (normalized for multi-word names, e.g. 'Postal Code'->zipcode, with geo taking precedence over the 'code' identifier token); suppress the ~20 Superstore helper columns (X-Axis Label, Map KPI Prefix, CLICK TO HIGHLIGHT...) - marks.ts: scatter/map_filled routing; buildKpiStrip binds X<->PP X<->X Difference by name pairing (deltaIsPositiveGood=false for Discount/Days to Ship/Returns) - audience.ts: scatter/map_filled allowed; kpi_tile cap-exempt; exec KPI lead - questions.ts: interview expanded to ~10 (q_comparison/q_geo_level/q_branding) - plan.ts: exec plan = KPI band + always-color-encoded bar + filled US map + trend/scatter; layoutGrammar=kpi_band_over_charts; derived dashboardTitle Verified deterministic; exec plan for Superstore fields = 4 KPI tiles + color bar + filled state map. Gate: 197 TS + 247 Python = 444 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…se->confirm loop) Slice 5 of exec-dashboards. The user-requested propose->confirm UX: - new proposal.ts buildProposal(plan): derives a human-readable DashboardProposal (summary, kpiStrip with up/down-good direction, per-view encodingSummary, layoutSummary, openQuestions) carrying the exact machine plan - designDashboard returns kind:'proposal' for autonomous/directed/interview_followup (interview still returns questions); output key plan->result; readable text content - tool description documents the stateless loop: design_dashboard never builds; agent presents proposal (+ optional wireframe); on confirm -> build_from_plan(proposal.plan); on change -> re-call directed mode - build_from_plan unchanged; proposal.plan passes its existing guards Sample exec Superstore proposal: 4 KPI tiles + color bar + filled state map. +39 tests. Gate: 236 TS + 247 Python = 483 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ch integration test Slice 6a of exec-dashboards. The Pydantic models model_dump() snake_case nested keys (geo_field, primary_measure, ...) but twb_builder read camelCase (geoField, primaryMeasure, ...), so POST /workbook/dashboard 500'd on every rich sheet — while the unit tests stayed green because their fixtures fed camelCase dicts the real server never sends. - twb_builder: all nested reads converted to snake_case (geo, kpi specs) across both build paths; layout_grammar keys were already snake - Slice-3 test fixtures aligned to the real (snake_case) data flow - NEW test_server_rich_dashboard.py (11 tests): exercises the genuine camelCase JSON -> DashboardWorkbookRequest -> model_dump() -> build_embedded_twbx path, asserts the rich XML markers (KPI text encodings, color, map semantic-role, title zone, kpi_band layout-flow, complete viewpoints) and a TestClient POST /workbook/dashboard -> 200 - scripts/demo-superstore.ts + npm run demo:superstore: the gated live-demo vehicle (ingest UTF-16/TSV -> exec plan -> proposal preview -> embedded build -> publish) Gate: 236 TS + 258 Python tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…and geo E0 offline verification caught this before any live publish: with the real 68-column Superstore file (alphabetical order), the exec plan keyed on 'Days to Ship' and dropped Sales entirely, and the map picked Country over State despite 'by state' in the question. - rankMeasures: question-mentioned first, then canonical business priority (sales/revenue/profit/... before operational metrics), then stable field order - rankDims: mentioned -> low-cardinality -> stable order (bar/color now honor what the user asked) - pickGeoField: mentioned geo level wins; else state > city > zipcode > country All pure/deterministic. Real-file result: KPI band = Sales/Profit/Quantity/ Discount with PP deltas; Sales by Category colored by Segment; filled State map. +5 tests (alphabetical-trap fixture). Gate: 241 TS + 258 Python. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e on Cloud Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase E1 slice A of the vibe-BI plan:
- brand.yaml: heavily-commented user-editable kit — palette (categorical/
sequential/diverging + semantic good/bad/neutral), typography (title/body/BAN,
Tableau Bold/Book defaults), formats (currency/percent/number), rules, and
named personas (ceo/cto/slt_manager/analyst/client) each = base audience +
overrides (maxSheets/kpiEmphasis/preferredArtifact/tone/chartDeny)
- src/branding: zod schema (hex validation, persona.base ∈ AudienceEnum,
defaults-when-absent) + loadBrand/resolvePersona (only I/O in the layer;
actionable errors; case-insensitive persona lookup)
- validate_brand tool (14 -> 15): never throws, reports {valid,warnings,personas}
- design_dashboard: optional persona/brandPath; tool layer resolves base
audience + maxSheets override and passes into the still-pure generatePlan;
plan carries personaName/brandName provenance; proposal summary names them
- yaml@2.9.0 dependency; brand.yaml in the npm files allowlist
+25 tests. Gate: 266 TS + 258 Python = 524 tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirroring the audited reference workbook XML:
- workbook-level <preferences><color-palette custom='true' name='{brand} Palette'>
with the categorical colors (Quota Attainment ~26-41 pattern)
- brand-driven title/subtitle runs: <run fontcolor fontname fontsize> from
typography (title/body), defaults preserved when no brand
- default-format stamped on measure columns by name classification
(currency-like -> formats.currency, ratio/percent-like -> percent, else number)
- brand block threaded end-to-end: BrandModel (Pydantic, camelCase aliases,
snake_case model_dump per the seam lesson) <- sidecar.ts WorkbookBrand <-
builderBrand.ts toBuilderBrand(BrandFile) <- buildFromPlan brandPath/persona
- demo-superstore: --persona <name> resolves brand.yaml and brands the build
Honestly skipped (not cleanly mirrorable without calculated fields): KPI delta
runs colored by sign (needs pre-split calc fields — Phase 3); mark-color on tile
panes (references don't do it). Candidate noted: zone-style background tint.
No-brand output stays byte-identical (determinism guards). +42 tests, all
branded variants XSD-valid. Gate: 276 TS + 290 Python = 566 tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ds to the sidecar The tool's sheet mapping dropped kind/color/kpi/scatter/geo and never passed dashboardTitle/subtitle/textZones/layoutGrammar — so build_from_plan (the product path) silently downgraded the confirmed plan to plain tiles while the demo script threaded everything. Same tool-vs-demo divergence class the verifier caught last feature; now locked by a threading guard test. Gate: 277 TS + 290 Python = 567 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… Cloud Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rides (E1-c) BI_DESIGN v2 gains §9 — seven deterministic Few/visionary rules with an honest ENFORCED-vs-DOCUMENTED ledger: - FEW-1 no-pie (structurally impossible; share/proportion -> bar, regression-tested) - FEW-2 KPI tiles carry comparison/delta when the data offers one (formalized) - FEW-3 no gauges (bullet-graph alternative documented; no mark class yet) - FEW-4 data-ink discipline (schema emits no gridline/border/shading chrome) - FEW-5 sequential color for measure-colored marks (structural; custom-stop binding noted as a gap) - FEW-6 top-N discipline: TOP_N_LIMIT=10 single source of truth for G-05 - FEW-7 small-multiples hint appended when 2 dims + measure + comparison intent Persona overrides now consumed end-to-end: - chartDeny -> clamp STEP 1.5 (denied marks fall back to bar + persona-named note) - kpiEmphasis -> STEP 3.5 caps the KPI band 4/3/2 (high/medium/low) - preferredArtifact story|pulse -> honest proposal openQuestion naming E4/E3 - tone concise -> one-sentence summary (persona provenance folded in) +31 tests (planner-slice6). Gate: 308 TS + 290 Python = 598 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…urce_fields (E2-a)
Phase E2 foundation slice:
- Bounded retry (max 3, 8s cap) on 429/502/503/504 only — 429 honors Retry-After,
else full-jitter exponential backoff with injectable sleep/jitter/now (no real
sleeps in tests). Idempotency-aware: GET/PUT/DELETE retried; the chunk-append
PUT explicitly NOT (mid-stream duplication risk); POST never except sign-in
and the side-effect-free VDS read (documented exceptions).
- Typed TableauApiError {status, code, summary, detail, retriable}; PAT
redaction preserved.
- src/rest/vds.ts: POST /api/v1/vizql-data-service/read-metadata -> typed field
list; actionable 404 / feature-disabled errors.
- New get_datasource_fields tool (15 -> 16): real field names + defaultAggregation
for design_dashboard fieldHints and the E3 Pulse pre-flight.
+28 mocked tests. Gate: 336 TS + 290 Python = 626 tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Six new tools (16 -> 22), all retry-aware and mocked-tested against the documented Cloud API shapes: - schedule_refresh: POST /tasks/extractRefreshes with the per-task embedded <schedule frequency><frequencyDetails start end><intervals> XML (schedule is a SIBLING of extractRefresh); zod-validated frequency spec (Hourly needs interval hours, Weekly weekDay, Monthly monthDay); always returns the honest Bridge/connectivity note (file-based extracts cannot refresh server-side) - list_refresh_schedules / delete_refresh_schedule (confirm=true gate) - create_webhook (HTTPS-only, DatasourceRefresh*/Created/Updated/Deleted + Workbook events; 403 rewrapped 'site admin required') / list / delete (gated) - update/get/runNow implemented at the client layer for later exposure - shared src/rest/xml.ts (xmlEscape/asArray) refactored out of restClient VERIFY-LIVE flags in JSDoc: IncrementalRefresh token spelling, response wrapper shape, nextRunAt attr, Hourly interval allowlist. +61 tests. Gate: 397 TS + 290 Python = 687 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ls + cron re-publish (E2-c)
Phase E2 final slice — 23 tools:
- sidecar build_live_tds + POST /datasource/live: federated .tds with
class='snowflake' (server/warehouse/dbname/schema/authentication) or
class='presto' (server/port/catalog/schema/ssl); attribute tables are
VERIFY-LIVE constants; key-pair auth rejected with a clean 400; credentials
NEVER appear in the .tds (structurally tested)
- publishDatasource gains {credentials, useRemoteQueryAgent}: emits
<connectionCredentials name password embed='true' oAuth='false'/> (escaped);
password-never-logged asserted in the secrets suite
- create_live_datasource tool: discriminated connection spec, Presto defaults
to Bridge (useRemoteQueryAgent), returns an honest schedulability note
- local-file design-around: scripts/refresh-local.ts (re-ingest + overwrite
republish) + generate-cron.ts emitting crontab + launchd templates (generated,
never installed — asserted); npm run refresh:local / cron:generate
Note: one inadvertent live sign-in/sign-out occurred during a script smoke test
(fell back to real .env; no content touched; disclosed by the build agent).
+54 tests. Gate: 433 TS + 308 Python = 741 tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Four new tools (23 -> 27), JSON-native client on /api/-/pulse/*: - create_pulse_definition: exact official payload shape (basic_specification inside specification; extension/representation/insights/comparisons TOP-LEVEL — pulse-api-utilities placement, deep-equal tested), with VDS pre-flight (measure exists + aggregation sanity warning; time dimension must be date-like; hard-fail lists available fields; VDS-unavailable degrades to warning; skipPreflight escape hatch) - list_pulse_definitions / create_pulse_metric (confirmed enum defaults; unconfirmed completions marked VERIFY-LIVE) / delete_pulse_definition (confirm=true) - Cloud-only + Pulse-disabled 404s rewrapped actionably; GETs retriable only +49 mocked tests. Gate: 482 TS + 308 Python = 790 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirrors the numeric coercion with the same >=90% guard, plus a separator/
month-name hint gate so integers and IDs are never mistaken for dates
('CA-2011-103800' carries the hint but fails to parse -> stays text). Runs
after numeric coercion (numeric columns already non-object). NULL/blank date
cells become NaT.
Real-file proof: the Superstore migrated file now ingests with Order Date and
Ship Date as real date columns — the time dimension Tableau Pulse requires.
+4 tests. Gate: 482 TS + 312 Python.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Story support mirroring the verified real-workbook structure, XSD-gated: - twb_builder _build_story: <dashboard type='storyboard'> with title zone + paired flipboard-nav/flipboard (bare type= attrs per the reference — the XSD's anyAttribute accepts them, validated) + <flipboard><story-points> with captured-sheet validation (ValueError lists valid names); story window with viewpoints over captured sheets - structural fix caught en route: _build_dashboard returned its own <dashboards> wrapper per call — multiple dashboards/stories would have emitted sibling wrappers the XSD forbids; now ONE shared container (regression-tested, no-story path byte-identical) - planner: wantsStoryArc (persona preferredArtifact=story or narrative language) -> deterministic buildStoryArc (headline + per-sheet captions, persona-toned); proposal surfaces storyOutline; buildFromPlan re-validates capturedSheet and threads stories through the camelCase->model_dump wire +65 tests. Gate: 521 TS + 338 Python = 859 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… fixture Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…udit STRIDE audit of the vibe-BI expansion surface: 0 CRITICAL, 0 HIGH, 1 MEDIUM, 2 LOW, 9 PASS — shippable. The MEDIUM (VB-02) is remediated here rather than fast-followed: buildCrontabLine wrapped agent-derived names/paths in double quotes, which do not neutralize $(...)/backticks/$VAR — a datasource named '$(rm -rf ~)' would execute when the operator installed the line. All interpolations now POSIX single-quote-escaped (shellQuote, '\'' splice), with adversarial-name tests ($(id), quote-breakout, embedded quotes) and the old double-quote assertions re-baselined. Launchd plist was already safe (exec-array + xmlEscape). Verified clean: embedded credentials never logged/persisted/in-.tds; webhook HTTPS fail-closed; XML injection escaped across all new tsRequest builders; confirm-gates on new deletes; yaml@2.9.0 safe; npm audit 0 vulns. SECURITY.md gains the full 'Vibe-BI expansion surface' section. Gate: 525 TS + 338 Python = 863 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full rewrite of tool_reference (27 tools in 8 families with honest Bridge/ Pulse-fixture caveats), README reframed as the vibe-BI front door (brand kit + personas, propose->confirm with storyOutline, connectivity/automation, Few design layer), architecture (rest/ family + retry policy, branding, storyArc, 7 sidecar routes, story XML), CODEBASE/DEPLOYMENT (27 real tool names, true counts 525 TS + 338 Python), .env.example clarified (live-connection creds are tool parameters, not env), 7 new ADRs + runbook; ADR-004 marked superseded; relationship doc count fixed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SebAustin
changed the title
SebAustin
pushed a commit
that referenced
this pull request
Jul 19, 2026
… + distribute-evenly on band container; exemplar-control proved dashboard-XML root cause, SCHEMA constraint #5 corrected (beauty-gate fix)
SebAustin
pushed a commit
that referenced
this pull request
Jul 21, 2026
… + distribute-evenly on band container; exemplar-control proved dashboard-XML root cause, SCHEMA constraint #5 corrected (beauty-gate fix)
Owner
Author
|
Superseded: the full latest work was promoted directly to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Vibe-BI platform expansion — prompt → governed, branded, automated analytics on Tableau Cloud
Grows the server from "publish a dashboard from a prompt" to a full vibe-BI pipeline: branding + personas, live/scheduled connectivity, Pulse metrics, and stories, behind the same propose→confirm contract and official-XSD gate. 14 → 27 MCP tools.
What's in
E0 — Superstore-grade exec dashboards
E1 — Branding system + design-excellence layer
brand.yamlkit (palette incl. semantic good/bad, typography, formats, named personas: ceo/cto/slt_manager/analyst/client — extensible without code).<preferences><color-palette custom='true'>, branded<run>s,default-format); no-brand output stays byte-identical.E2 — Connectivity + automated refresh + REST foundation
src/rest/split by API family; bounded idempotency-aware retry (chunk-append PUT is never retried); typed error taxonomy; VDS read-metadata client →get_datasource_fields..tds(credentials embedded only at publish, never in the file; key-pair auth cleanly rejected as REST-impossible), Cloud extract-refresh schedules, fail-closed HTTPS webhooks.E3 — Pulse metrics (code-complete)
/api/-/pulse; wire payload locked by deep-equal test; VDS pre-flight hard-fails on missing measure / non-date time dimension.basic_specificationinternals — documented fallback: GET a UI-created metric as fixture (official pulse-api-utilities clones, never constructs). Honest status recorded in ACCEPTANCE.md.E4 — Stories
<dashboard type='storyboard'>+ flipboard/story-points, XSD-validated in 6 variants; captured-sheet fail-loud at both layers; shared single<dashboards>container regression-guarded.E5 — Hardening + docs
Test plan
make cigreen: 525 TS + 338 Python = 863 tests, 0 skipped/focused.tds).envcreds)🤖 Generated with Claude Code