Is there a native command to sync/refresh schema (pull new columns) via CLI?

[mrbartuss]

Hi everyone,

I have an Import mode semantic model that pulls directly from a table (loaded via dataflow in a Lakehouse). Recently, a new column was added to the upstream source table.

I understand that a standard data refresh in the Service only updates the rows, not the schema.
MS Learn

I was hoping to automate this structural update using the Tabular Editor CLI, but it seems like a standard refresh command (e.g., triggering a full refresh on the table) doesn't pull in the new column either.

Is there a simple, built-in CLI command or flag to force a "schema sync" to automatically detect and append new columns - similar to how hitting "Refresh" works natively in Power BI Desktop?

Thanks in advance for the help!

[greggyb]

Yes, refreshing is by definition only a data operation, never a metadata operation. Power BI Desktop hides this, because it uses a single GUI button, "Refresh" both to evaluate and identify metadata changes and perform a normal refresh. Power BI Desktop is special and weird in this way.

Adding fields is straightforward, but it seems like you're asking specifically about schema drift detection, which requires, very explicitly, not a regular refresh operation.

There's a schema check option for the TE2 CLI that we have not updated yet. That will compare schema only for legacy partitions: https://docs.tabulareditor.com/en/how-tos/Importing-Tables.html#cli-support

You can pretty easily use the TE cli to replicate this, though. I'm working with the SpaceParts sample dataset. You can follow along by first running the setup.te script below:

setup.te: save this as setup.te

init ./spaceparts.bim --serialization bim --name SpacePartsDemo
connect ./spaceparts.bim
add Employees -t Table --source sql --connection-string "Server=te3-training-eu.database.windows.net;Database=SpacePartsCoDW;User ID=dwreader;Password=TE3#reader!;Encrypt=true;Connect Timeout=120" --endpoint te3-training-eu.database.windows.net --source-database SpacePartsCoDW --source-table "Dimview.Employees"
rm "Employees[Employee Email]"
rename "Employees[Data Security Rule]" "Employees[Security Rule]"
set "Employees[Role]" -q dataType -i Int64
save

Then in a shell:

$ te interactive --batch --echo < setup.te # creates a new model called spaceparts.bim
$ cp spaceparts.bim spaceparts-copy.bim # just an identical copy
$ te rm Employees --force --save # remove the Employees table
$ greg@TE-GB-MacBookProM5Pro te-cli % te add Employees spaceparts-copy.bim -t Table --source sql --connection-string "Server=te3-training-eu.database.windows.net;Database=SpacePartsCoDW;User ID=dwreader;Password=TE3#reader!;Encrypt=true;Connect Timeout=120" --endpoint te3-training-eu.database.windows.net --source-database SpacePartsCoDW --source-table "Dimview.Employees" --save # add the table, using schema discovery
Discovering schema for Dimview.Employees...
Discovered 4 columns
Added: Employees
$ te diff spaceparts.bim spaceparts-copy.bim

Left:  spaceparts.bim
Right: spaceparts-copy.bim
2 added, 1 removed, 1 modified

  - Column Employees/Security Rule
  + Column Employees/Employee Email
  + Column Employees/Data Security Rule
  ~ Column/Employees/Role DataType
    - Int64
    + String

This uses a copy of the model where we remove and re-add the table that we want. The TE CLI supports schema discovery on adding a whole table. You should not use a literal password in your connection string like this, but this one is a public sample dataset.

The add+remove in a copy lets you use the diff machinery. You can also use te --output-format json diff spaceparts.bim spaceparts-copy.bim to get a structured output.

Once you've got the updated second copy, you can inspect it and add any fields you like from it to the primary.

This area of schema checking is a good one for us to explore more as we proceed through the preview.

[mrbartuss]

Thank you for such a detailed response!