Get & Transform Data: queries, Power Query Editor, file/web/OData/ODBC sources, WEBSERVICE and FILTERXML by Herr-Enklo · Pull Request #1203 · storytold/gridcraft

Herr-Enklo · 2026-10-13T23:52:45Z

Closes most of gaps.md §12. Adds Excel's Get & Transform Data (Power Query) with Excel 365's Data-tab layout, plus WEBSERVICE and FILTERXML. Both functions previously returned #VALUE! as "not available".

Data tab, laid out like Excel 365

Get & Transform Data group

Get Data ▾ has Excel's full menu, in Excel's order:
- From File: Excel Workbook, Text/CSV, XML, JSON, PDF, Folder, SharePoint Folder
- From Database: SQL Server, Access, Analysis Services, SSAS (Import)
- From Azure, From Power Platform, From Online Services
- From Other Sources: Table/Range, Web, Microsoft Query, OData Feed, ODBC, OLEDB, Blank Query
- Combine Queries (Merge, Append)
- Launch Power Query Editor…, Data Source Settings…, Query Options
Beside it: From Text/CSV, From Web, From Table/Range, Recent Sources, Existing Connections.

Queries & Connections group

Refresh All split button: Refresh All, Refresh, Refresh Status, Cancel Refresh, Connection Properties….
Queries & Connections task pane (Queries / Connections tabs, row counts or errors per query).
Properties.

Sources GridCraft can't reach stay listed and disabled, with the reason on hover. The reserved key tips are now live (AFW, AFT, AFA, APNE, APNS, APNL, APNT, APNH, APND, APR, APT, ARC, ARO, ARS, AX). About 330 German labels follow German Excel.

Queries

model::query: a source plus typed applied steps, a load target (table on a new or existing sheet, or connection only), refresh options and the last status.
Sources: Text/CSV (delimiter, encoding), Excel workbook (sheet, table or name, read through gridcraft-xlsx), XML, JSON, Folder (list or combine), Web (HTML tables in a Navigator; CSV/JSON/XML by content type), OData (service root, paging), ODBC and SQL Server/Access through ODBC (Windows), Table/Range, Blank, and references to other queries.
About 35 typed steps: choose/remove columns, keep/remove rows, remove duplicates/blanks/errors, filter, sort, promote headers, change type (locale-aware), rename, replace values, split by delimiter, trim/clean/case, group by, merge (six join kinds), append, index and conditional columns, fill down/up, unpivot, transpose, duplicate column.
New queries get Power Query's Promoted Headers and Changed Type steps automatically.
Refresh is deterministic. Refresh All runs queries, then PivotTables, in one undo step. A failed refresh keeps the old table, and loads never overwrite cells that hold data.
Power Query Editor:
- queries list, preview with type badges and column menus (sort, text/number/date filters, filter by values)
- formula bar, Query Settings, Applied Steps (preview, rename, delete, delete until end, move)
- Home / Transform / Add Column / View
- read-only Advanced Editor (M-like text generated from the steps; no M interpreter)
- Close & Load / Load To
Also new: Navigator, From Web, OData, ODBC and SQL Server dialogs, Import Data, Query Properties (refresh on open / every N minutes), Data Source Settings, Query Options, Recent Sources, Existing Connections, Merge, Append.

External I/O

The engine owns a query::DataAccess trait. The engine and calc stay wasm-safe; the browser build shows disabled reasons.
A new L6 crate, gridcraft-external, implements it. The desktop app and the CLI install it.
Only http/https URLs; timeouts of 30 s (10 s for WEBSERVICE); a 64 MB download cap; at most 10 redirects; bounded HTML/JSON/XML parsing.
New dependencies (please check):
- ureq 3 with native-tls on Windows/macOS: the system TLS and certificate store, as Excel uses.
- ureq with rustls + webpki-roots on Linux/BSD.
- odbc-api on Windows only, against the driver manager Windows ships.

Security, like Excel

A workbook opened with queries or WEBSERVICE calls doesn't refresh. It shows "Security Warning: External data connections have been disabled" with Enable Content.
File › Options gets a Trust Center data-connections setting.

WEBSERVICE and FILTERXML

WEBSERVICE goes through the same data access, fetches each URL once per recalc pass, and returns #VALUE! when blocked, on error, or past 32,767 characters.
FILTERXML evaluates an XPath 1.0 subset in crates/functions/src/xpath.rs: paths, //, *, @attr, [n], [@a='x'], text(). It spills matches down a column and returns numeric text as numbers.

XLSX

Query definitions round-trip in GridCraft's own custom XML part (customXml/gridcraftQueries.xml).
Excel's own DataMashup, connections and query tables are still passed through untouched.
Loaded results are ordinary tables.

Expected overlap

The Workbook Links PR from this fork (#1200) also builds the Queries & Connections group (adding Workbook Links) and a reusable security bar (message_bar.rs). Whichever lands second needs a small reconcile; I'll rebase it.

Not included

Disabled with a reason: PDF, SharePoint, Azure, Power Platform, online services, Analysis Services, OLE DB, Microsoft Query.
ODBC on macOS/Linux (it would need unixODBC at build time).
No M interpreter: Excel's own DataMashup queries are kept on save, but not read or refreshed.
Not done: the data model and relationships, credentials and privacy levels.
Cancel Refresh is disabled: refresh runs in the foreground.
Auto-generated step names are English ("Changed Type"). German Excel names them in the interface language ("Geänderter Typ"); that needs observing each step type first.

Tests

Engine query::tests: every source (fake data access), every step, merge/append, step editing, refresh determinism, load targets, failed loads, trust prompt, WEBSERVICE/FILTERXML through a fake fetcher, hostile params.
gridcraft-external: a local HTTP server, refused schemes and timeouts, garbage responses; ODBC through the Access text driver when installed.
xpath and FILTERXML cases.
XLSX round trip beside kept custom XML.
UI kittests: queries pane, editor draft and Close & Load, security bar, German labels.
every_command_survives_empty_params covers the new commands. The command-sequence fuzz now runs data.fromTableRange, query.addStep and data.refreshAll.
cargo xtask ci: all 8 steps pass (1639 tests).

🤖 Generated with Claude Code

@A

FILTERXML evaluates an XPath 1.0 subset over its XML argument: location paths with the common axes, `*`, `@attr`, `text()`, `node()`, predicates (`[2]`, `[last()]`, `[@A='x']`, `[price>35]`, `not()`), unions, comparisons, arithmetic and the core string, number and boolean functions. Matches spill down one per row, numeric text comes back as numbers, and invalid XML or XPath, no match or a non-node-set result give #VALUE! like Excel. The XML tree and the evaluator are bounded (nodes, nesting, expression depth, work per evaluation) and pure Rust on quick-xml, so wasm-safe. WEBSERVICE asks the host through a new `Ctx::web_service` hook. The calc crate carries an optional `WebFetch` the engine injects; each URL is fetched once per recalculation pass, responses longer than a cell are #VALUE!, and without a fetcher (the browser, a workbook whose external content is disabled) WEBSERVICE gives #VALUE!. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Workbooks keep Power Query-style queries: a source (Text/CSV, Excel workbook, XML, JSON, folder, web, OData, ODBC, a table of this workbook, another query, blank), an ordered list of typed applied steps, where the result loads (a table on a new or existing sheet, or only a connection), refresh options (on open, every N minutes, background, include in Refresh All) and the status of the last refresh. Steps are a typed enum, not M. The definitions are saved as JSON in GridCraft's own custom XML part (`customXml/gridcraftQueries.xml`, namespace `urn:gridcraft:queries:1`) and read back into the model, so they survive a round trip and Excel keeps them; Excel's own DataMashup, connections and query tables stay passed through untouched. The CSV reader exposes its raw records and the XML tree parser is public for the query sources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Data › Get & Transform Data and Queries & Connections as commands: From Text/CSV, From Excel Workbook, From XML, From JSON, From Folder (file list or combined files), From Web (HTML tables in a Navigator, CSV, JSON and XML by content type), From OData Feed (service roots, paging), From ODBC, From SQL Server Database and From Microsoft Access Database (through ODBC), From Table/Range (a range becomes a table first), Blank Query, Combine Queries (Merge, Append), Recent Sources, Existing Connections, Data Source Settings, Query Options, Refresh All (queries first, then PivotTables, in one undo step), Refresh, Refresh Status, Connection Properties, Properties and the Queries & Connections pane state. Sources Excel shows but GridCraft can't reach (PDF, SharePoint, Azure, Power Platform, online services, Analysis Services, OLE DB, Microsoft Query) are registered and disabled with their reason; Cancel Refresh is disabled because refresh runs in the foreground. Creating a query reads its source once, adds Power Query's own Promoted Headers and Changed Type steps (types detected from the first 200 rows), and loads a table on a new worksheet named after the query, at a cell of an existing sheet, or only keeps the connection. Refresh rewrites the table in place, keeps the old one when the source fails, never writes over cells holding data, and is deterministic. The `query.*` primitives (set, addStep, removeStep, renameStep, moveStep, preview at any step, navigator, load, mText, rename, duplicate, reference, delete) drive the editor, the CLI and MCP. Applied steps cover Choose/Remove/Rename/ Reorder/Duplicate Columns, Keep/Remove Top/Bottom/Range of Rows, Remove Duplicates/Blank Rows/Errors, Filter Rows (value lists and text, number and date conditions), Sort, Replace Values, Split Column by Delimiter, Trim/Clean/Upper/Lower/Proper, Fill Down/Up, Group By, Unpivot, Transpose, Merge Queries (six join kinds), Append Queries, Index and Conditional columns, Change Type with locale-aware parsing, Use First Row as Headers and back. An M-like rendering is generated for the Advanced Editor. External data goes through a `DataAccess` trait the engine owns: without an injected implementation files are read with std::fs and the web and ODBC are unavailable with a reason (the browser). A workbook opened with external queries or WEBSERVICE calls shows the Security Warning until `data.enableContent` (which then recalculates WEBSERVICE and refreshes refresh-on-open queries); `file.trustCenter` sets Prompt, Enable all or Disable all data connections. URLs must be http(s), downloads and files are size-capped, and hostile HTML, JSON and XML are bounded. Key tips reserved for these commands are live now, the legacy ones point at their Get & Transform successors, and the parity checklist counts the implemented catalog entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A new L6 crate, gridcraft-external, implements the engine's DataAccess for the desktop app and the command line (both install it into their sessions; the browser build never links it): - HTTP(S) GET through ureq with rustls and Mozilla's roots, so no system TLS library is needed: http and https only, a timeout per request, bounded redirects, a cap on the body size, short error sentences. - ODBC on Windows only (odbc-api over the driver manager Windows ships), for From ODBC, From SQL Server Database and From Microsoft Access Database. Tables are listed by the driver and quoted with its quote character; values are typed by the column's SQL type. Other platforms would need unixODBC at build time, so ODBC stays disabled there. Tests run against a local HTTP server (status codes, size cap, timeout, garbage responses, refused schemes) and, where the Access text driver is installed, a CSV read through ODBC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…r tests HTTP(S) uses the system TLS stack and certificate store on Windows and macOS (SChannel, Secure Transport): no C compiler is needed there (ring needs clang to cross-compile for Windows on ARM), and roots installed by an organisation are honoured as Excel honours them. Linux and the BSDs keep rustls with Mozilla's roots, so no OpenSSL has to be installed. Checked against a live HTTPS page: its HTML tables import. Filter conditions on date columns take dates as typed (2026-01-31, 1/31/2026), in the workbook's date system. The OData source kind is spelled `odata` in query JSON. From SQL Server Database and From Microsoft Access Database share their connection-string builders with the Navigator, and a fake ODBC data access tests From ODBC, SQL Server and Access (listed tables only, typed dates, unique column names, connection strings never kept in Recent Sources). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The Data tab's Get & Transform Data group follows Excel 365: Get Data with its menu in Excel's order (From File, From Database, From Azure, From Power Platform, From Online Services, From Other Sources, Combine Queries, Launch Power Query Editor, Data Source Settings, Query Options; sources GridCraft can't reach stay listed, disabled with the reason), then From Text/CSV, From Web, From Table/Range, Recent Sources and Existing Connections. Queries & Connections has the Refresh All split button (Refresh All, Refresh, Refresh Status, Cancel Refresh, Connection Properties), Queries & Connections and Properties. The Power Query Editor edits drafts of the workbook's queries: queries list, preview grid with type badges and column menus (sort, text, number and date filters, filter by values), formula bar, Query Settings (name, properties) and Applied Steps (select to preview, rename, delete, delete until end, move), Home, Transform, Add Column and View tabs with the common transforms, an Advanced Editor view of the generated M text, and Close & Load / Close & Load To, or discard. The Navigator lists a source's tables (HTML tables, sheets and tables, JSON arrays, XML elements, entity sets, database tables) with a preview, and the delimited-text preview offers File Origin and Delimiter. From Web, From ODBC, SQL Server database, Import Data (Load To), Query Properties (refresh on open, every N minutes, background), Data Source Settings, Query Options, Recent Sources, Existing Connections, Merge and Append are dialogs; the Queries & Connections task pane lists queries with their row counts or errors and edits, refreshes, deletes and loads them. A workbook opened with external content shows the Security Warning bar with Enable Content; File › Options has the Trust Center's data connection setting. Refresh every N minutes runs from the frame loop. The desktop app picks data files and folders for the file sources. German labels throughout; the Data tab's Get Data label in every catalog. New icons drawn in code: globe, database, refresh, query. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gaps.md §12 lists what landed and what remains (the data model, Excel's own M queries, cloud sources; 30–60 h). The function, file-format, UI, localization and target-app parity docs, the roadmap and ROADMAP.md record the new sources, the editor and the working web functions (506 / 523 working, catalog 294 / 309); the architecture doc describes the DataAccess trait and the native `external` crate. cli.md documents the source commands, the query JSON and every applied step; mcp.md points agents at them and at Enable Content for opened files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A refresh whose result can't be written (it would overwrite cells that hold data, or land on a protected sheet) now leaves the workbook exactly as it was, the previous table included, and records the error on the query. Renaming a loaded table keeps it the query's table (and queries reading it follow). A second window on a workbook whose external content is disabled stays disabled. After five failed WEBSERVICE fetches in one recalculation pass the rest give #VALUE! without asking the network, so an unreachable host can't stall a recalculation once per URL. From Folder shows Power Query's Attributes record expanded (size, hidden, read-only). The Merge dialogs remember the columns they show instead of evaluating the queries every frame. The command-sequence fuzz now also creates queries from tables, adds applied steps and refreshes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

anton-belkin · 2026-10-15T07:27:25Z

Hi @Herr-Enklo, thanks for this PR: the Data tab, the Power Query Editor, the Navigator and the
source readers are a big step for Get & Transform.

I've been testing it with workbooks made in Excel, and I built a Power Query M interpreter
on top of your engine: Excel's own queries (the DataMashup part) now refresh in GridCraft. It
reuses your DataAccess, sources, type conversion and loading. On a real workbook with 19
queries and 5 loaded tables, all 19 refresh and every loaded value matches what Excel loaded
(Refresh All took ~4 s against ~30 s in Excel).

That leaves two query models side by side: your typed steps (saved in GridCraft's own JSON part,
which Excel doesn't read) and Excel's M. To converge on Excel parity, I'd like to propose that
M becomes the one model, the way Excel works:

a query is its M (let bindings = Applied Steps), stored in DataMashup so Excel can refresh
queries made in GridCraft too;
your typed steps become the editor's codec: StepOp → M (the M Excel's editor writes) and
M → StepOp (so recognised steps open your dialogs; anything else is edited as M);
one engine runs everything, and where your step code is the better implementation it becomes
the implementation of the matching Table.* function.

A first, low-risk step would be a differential test: render each query's steps as exact M and
check the interpreter gives the same table as steps::apply, across your existing step tests.

How would you like to proceed? Two options I see:

Your PR lands first (as is, or split if that helps review), and I follow with the
interpreter and the convergence PRs on top of main.
I carry your commits unchanged (your authorship kept) in one PR with the interpreter, framed
as "Excel's Power Query workbooks refresh", and Get & Transform Data: queries, Power Query Editor, file/web/OData/ODBC sources, WEBSERVICE and FILTERXML #1203 is closed in its favour.

Either way I'd suggest splitting the area: you on the editor, dialogs and sources, me on the M
language and library, with the query model (M text and the step codec) as our shared boundary.
I've written the design up as a section of docs/architecture.md ("Power Query: one model, M");
I'll push the branch with it and the interpreter and link it here. Happy to adjust it with you.

anton-belkin · 2026-10-15T07:27:59Z

As promised, the branch is up: anton-belkin/gridcraft claude/m-language. It sits on top of your commits; these are the commits it adds (the interpreter is in crates/engine/src/query/m).

The design write-up: docs/architecture.md, "Power Query: one model, M".

Herr-Enklo and others added 9 commits October 14, 2026 00:02

XLSX: the damaged query-part test uses a byte literal (clippy)

e3f9569

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

mbulloni mentioned this pull request Oct 15, 2026

Data: From Database (PostgreSQL, MySQL/MariaDB, SQLite, ODBC) #1430

Open

anton-belkin mentioned this pull request Oct 15, 2026

Power Query: Excel's queries refresh in GridCraft (M interpreter), building on #1203 #1532

Open

GabLavareda mentioned this pull request Oct 16, 2026

Add Power Query, Power Pivot and DAX Compatibility #431

Open

Herr-Enklo mentioned this pull request Oct 17, 2026

PQ19: Portuguese date names and Record.ToTable anton-belkin/gridcraft#1

Merged

Navigation Menu

Repository navigation

Get & Transform Data: queries, Power Query Editor, file/web/OData/ODBC sources, WEBSERVICE and FILTERXML - #1203

Get & Transform Data: queries, Power Query Editor, file/web/OData/ODBC sources, WEBSERVICE and FILTERXML#1203
Herr-Enklo wants to merge 9 commits into
storytold:mainfrom
Herr-Enklo:feat/get-data

Herr-Enklo commented Oct 13, 2026

Uh oh!

anton-belkin commented Oct 15, 2026

Uh oh!

anton-belkin commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Herr-Enklo commented Oct 13, 2026

Data tab, laid out like Excel 365

Queries

External I/O

Security, like Excel

WEBSERVICE and FILTERXML

XLSX

Expected overlap

Not included

Tests

Uh oh!

anton-belkin commented Oct 15, 2026

Uh oh!

anton-belkin commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants