Skip to content

Copy-DbaDbTableData - Map query columns onto the writable destination columns #10666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andreasjordan wants to merge 4 commits into
base: development
Choose a base branch
from
+151 −10
Open

Copy-DbaDbTableData - Map query columns onto the writable destination columns #10666

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
38d5d62
Copy-DbaDbTableData - Map query columns onto the writable destination…
andreasjordan Aug 31, 2026
58bd859
Copy-DbaDbTableData - Drop the single quotes from the skipped-column …
andreasjordan Aug 31, 2026
3a2bc61
Testing Copy-DbaDbTableData - Select the id column with the case it w…
andreasjordan Aug 31, 2026
863aaa7
Copy-DbaDbTableData - Exclude generated always columns from the writa…
andreasjordan Sep 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 26 additions & 8 deletions public/Copy-DbaDbTableData.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,14 @@ function Copy-DbaDbTableData {
Custom SQL SELECT query to use as the data source instead of copying the entire table or view. Supports 3 or 4-part object names.
Use this when you need to filter rows, join multiple tables, or transform data during the copy operation. Still requires specifying a Table or View parameter for metadata purposes.

Note: Columns are mapped by ordinal position. If the destination table has an identity column, include a placeholder value (e.g., 0) in your SELECT list at that position.
Note: Columns are mapped by position onto the writable columns of the destination table, so computed, rowversion and generated always columns of the destination do not count and must not have a counterpart in the SELECT list.
If the destination table has an identity column, include a placeholder value (e.g., 0) in your SELECT list at that position.
The placeholder will be ignored and the identity value auto-generated unless -KeepIdentity is specified.

.PARAMETER ForceExplicitMapping
When used together with Query parameter, force the use of explicit column mapping (name-based) instead of switching over to ordinal position mapping. Use with care if query contains aliases.
Default behaviour when using Query parameter is to use ordinal position mapping, due to the possibility of the query including aliases (SELECT x AS y) which could lead to column mismatching and data not copying.
The downside of it automatically switching over to ordinal mapping is that it also tries to copy over computed columns, which will cause it to fail.
Positional mapping skips the computed, rowversion and generated always columns of the destination table, so the SELECT list only has to match its writable columns.

.PARAMETER AutoCreateTable
Automatically creates the destination table if it doesn't exist, using the same structure as the source table.
Expand Down Expand Up @@ -672,11 +673,16 @@ function Copy-DbaDbTableData {
$bulkCopy.NotifyAfter = $NotifyAfter
$bulkCopy.BulkCopyTimeout = $BulkCopyTimeout

# Get list of non-computed columns from destination table to avoid insert failures
# Get list of writable columns from destination table to avoid insert failures. Computed, rowversion
# and generated always columns (temporal periods, ledger metadata) cannot be written, and they are also
# what breaks the implicit positional mapping of SqlBulkCopy: it counts a computed column (the server
# then rejects the insert) and silently drops the source column that lands on a rowversion column,
# shifting every column behind it by one (see #10661). GeneratedAlwaysType is only supported by SMO on
# SQL Server 2016 and later, so it has to be guarded by the version.
# Refresh the columns collection to ensure it's populated
$desttable.Columns.Refresh()
$destColumns = $desttable.Columns | Where-Object Computed -eq $false | Select-Object -ExpandProperty Name
Write-Message -Level Verbose -Message "Destination table has $($destColumns.Count) non-computed columns"
$destColumns = @($desttable.Columns | Where-Object { -not $PSItem.Computed -and $PSItem.DataType.SqlDataType -ne "Timestamp" -and -not ($destServer.VersionMajor -ge 13 -and $PSItem.GeneratedAlwaysType -ne "None") } | Select-Object -ExpandProperty Name)
Write-Message -Level Verbose -Message "Destination table has $($destColumns.Count) writable columns"

# The legacy bulk copy library uses a 4 byte integer to track the RowsCopied, so the only option is to use
# integer wrap so that copy operations of row counts greater than [int32]::MaxValue will report accurate numbers.
Expand Down Expand Up @@ -705,18 +711,30 @@ function Copy-DbaDbTableData {
$reader = $cmd.ExecuteReader()

# Only apply explicit column mapping for straight table copies (not custom queries)
# Custom queries may have different column names/aliases, so let SqlBulkCopy use ordinal mapping
# Custom queries may have different column names/aliases, so they are mapped by position
# Appending -ForceExplicitMapping will override this behaviour and keep explicit column mapping
if (-not (Test-Bound -ParameterName Query) -or $ForceExplicitMapping) {
# Map only columns that exist in both source and destination (excluding computed columns)
# Map only columns that exist in both source and destination (excluding computed, rowversion and generated always columns)
for ($i = 0; $i -lt $reader.FieldCount; $i++) {
$sourceColumn = $reader.GetName($i)
if ($destColumns -contains $sourceColumn) {
$null = $bulkCopy.ColumnMappings.Add($sourceColumn, $sourceColumn)
} else {
Write-Message -Level Verbose -Message "Skipping column '$sourceColumn' (not in destination or is computed)"
Write-Message -Level Verbose -Message "Skipping column $sourceColumn (not in destination or not writable)"
}
}
} else {
# Map the query columns by position onto the writable destination columns. This is what SqlBulkCopy does
# on its own, except that its list also contains the computed, rowversion and generated always columns (see above).
if ($reader.FieldCount -gt $destColumns.Count) {
$columnCountMessage = "The query returns $($reader.FieldCount) columns but $fqtndest has only $($destColumns.Count) writable columns. Computed, rowversion and generated always columns cannot be written and do not count."
$reader.Close()
throw $columnCountMessage
}
for ($i = 0; $i -lt $reader.FieldCount; $i++) {
Write-Message -Level Verbose -Message "Mapping query column $i ($($reader.GetName($i))) to destination column $($destColumns[$i])"
$null = $bulkCopy.ColumnMappings.Add($i, $destColumns[$i])
}
}

$bulkCopy.WriteToServer($reader)
Expand Down
127 changes: 125 additions & 2 deletions tests/Copy-DbaDbTableData.Tests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,12 +108,12 @@ Describe $CommandName -Tag IntegrationTests {
}

It "Copy data using a query that relies on the default source database" {
$result = Copy-DbaDbTableData -SqlInstance $TestConfig.InstanceCopy2 -Database tempdb -Table dbo.dbatoolsci_example4 -Query "SELECT TOP (1) Id FROM dbo.dbatoolsci_example4 ORDER BY Id DESC" -DestinationTable dbatoolsci_example3 -Truncate
$result = Copy-DbaDbTableData -SqlInstance $TestConfig.InstanceCopy2 -Database tempdb -Table dbo.dbatoolsci_example4 -Query "SELECT TOP (1) id FROM dbo.dbatoolsci_example4 ORDER BY id DESC" -DestinationTable dbatoolsci_example3 -Truncate
$result.RowsCopied | Should -Be 1
}

It "Copy data using a query that uses a 3 part query" {
$result = Copy-DbaDbTableData -SqlInstance $TestConfig.InstanceCopy2 -Database tempdb -Table dbo.dbatoolsci_example4 -Query "SELECT TOP (1) Id FROM tempdb.dbo.dbatoolsci_example4 ORDER BY Id DESC" -DestinationTable dbatoolsci_example3 -Truncate
$result = Copy-DbaDbTableData -SqlInstance $TestConfig.InstanceCopy2 -Database tempdb -Table dbo.dbatoolsci_example4 -Query "SELECT TOP (1) id FROM tempdb.dbo.dbatoolsci_example4 ORDER BY id DESC" -DestinationTable dbatoolsci_example3 -Truncate
$result.RowsCopied | Should -Be 1
}
}
Expand Down Expand Up @@ -216,6 +216,129 @@ Describe $CommandName -Tag IntegrationTests {
}
}

Context "When using Query without ForceExplicitMapping and the destination has unwritable columns" {
BeforeDiscovery {
# GENERATED ALWAYS columns arrived with SQL Server 2016, so the temporal scenario below cannot
# be built before that. The value decides a Skip, which Pester needs while it discovers the
# tests, so it cannot be read in BeforeAll.
$discoveryDestServer = Connect-DbaInstance -SqlInstance $TestConfig.InstanceCopy2
$destSupportsTemporal = $discoveryDestServer.VersionMajor -ge 13
}

BeforeAll {
$PSDefaultParameterValues["*-Dba*:EnableException"] = $true

$null = $sourceDb.Query("CREATE TABLE dbo.dbatoolsci_positional_source (Id INT, A INT, B INT, C INT)")
$null = $sourceDb.Query("INSERT dbo.dbatoolsci_positional_source (Id, A, B, C) VALUES (1, 11, 22, 33), (2, 111, 222, 333)")
# A computed and a rowversion column sit between the writable ones, so a positional mapping that
# counts them shifts every column behind them (#10661).
$null = $destinationDb.Query("CREATE TABLE dbo.dbatoolsci_positional_dest (Id INT, A INT, Computed AS (A * 10), RV ROWVERSION, B INT, C INT)")
$null = $destinationDb.Query("CREATE TABLE dbo.dbatoolsci_positional_identity (Id INT IDENTITY(1, 1), A INT, B INT, C INT)")
$null = $destinationDb.Query("CREATE TABLE dbo.dbatoolsci_positional_rowversion (Id INT, A INT, RV ROWVERSION, B INT, C INT)")
if ($destinationDb.Parent.VersionMajor -ge 13) {
# The period columns of a temporal table are GENERATED ALWAYS: not computed, not rowversion,
# but just as unwritable, and interleaved with the writable columns here on purpose.
$null = $destinationDb.Query("CREATE TABLE dbo.dbatoolsci_positional_temporal (Id INT PRIMARY KEY, A INT, ValidFrom DATETIME2 GENERATED ALWAYS AS ROW START NOT NULL, B INT, ValidTo DATETIME2 GENERATED ALWAYS AS ROW END NOT NULL, C INT, PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo)) WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.dbatoolsci_positional_temporal_history))")
}

$splatPositional = @{
SqlInstance = $TestConfig.InstanceCopy1
Destination = $TestConfig.InstanceCopy2
Database = "tempdb"
Table = "dbatoolsci_positional_source"
Query = "SELECT Id, A, B, C FROM dbo.dbatoolsci_positional_source"
DestinationTable = "dbatoolsci_positional_dest"
}

$PSDefaultParameterValues.Remove("*-Dba*:EnableException")
}

AfterAll {
$PSDefaultParameterValues["*-Dba*:EnableException"] = $true

$null = $sourceDb.Query("DROP TABLE IF EXISTS dbo.dbatoolsci_positional_source")
$null = $destinationDb.Query("DROP TABLE IF EXISTS dbo.dbatoolsci_positional_dest")
$null = $destinationDb.Query("DROP TABLE IF EXISTS dbo.dbatoolsci_positional_identity")
$null = $destinationDb.Query("DROP TABLE IF EXISTS dbo.dbatoolsci_positional_rowversion")
$destinationDb.Tables.Refresh()
if ($destinationDb.Tables | Where-Object Name -eq "dbatoolsci_positional_temporal") {
# System versioning has to be turned off before the temporal table can be dropped.
$null = $destinationDb.Query("ALTER TABLE dbo.dbatoolsci_positional_temporal SET (SYSTEM_VERSIONING = OFF)")
$null = $destinationDb.Query("DROP TABLE dbo.dbatoolsci_positional_temporal")
$null = $destinationDb.Query("DROP TABLE IF EXISTS dbo.dbatoolsci_positional_temporal_history")
}

$PSDefaultParameterValues.Remove("*-Dba*:EnableException")
}

It "Maps the query columns by position onto the writable destination columns only" {
$result = Copy-DbaDbTableData @splatPositional
$WarnVar | Should -BeNullOrEmpty
$result.RowsCopied | Should -Be 2

$destData = $destinationDb.Query("SELECT Id, A, Computed, B, C FROM dbo.dbatoolsci_positional_dest ORDER BY Id")
$destData.A | Should -Be @(11, 111)
$destData.Computed | Should -Be @(110, 1110)
$destData.B | Should -Be @(22, 222)
$destData.C | Should -Be @(33, 333)
}

It "Does not shift the columns behind a rowversion column" {
# This is the silent variant: SqlBulkCopy drops the source column that lands on the rowversion
# column and reports success, so the last column ends up empty and the ones before it are off by one.
$splatRowversion = $splatPositional.Clone()
$splatRowversion.DestinationTable = "dbatoolsci_positional_rowversion"
$result = Copy-DbaDbTableData @splatRowversion
$WarnVar | Should -BeNullOrEmpty
$result.RowsCopied | Should -Be 2

$destData = $destinationDb.Query("SELECT Id, A, B, C FROM dbo.dbatoolsci_positional_rowversion ORDER BY Id")
$destData.B | Should -Be @(22, 222)
$destData.C | Should -Be @(33, 333)
}

It "Does not count the generated always columns of a temporal destination" -Skip:(-not $destSupportsTemporal) {
# The period columns are GENERATED ALWAYS, so the server refuses explicit values for them.
# A positional mapping that counts them maps writable source columns onto them and fails.
$splatTemporal = $splatPositional.Clone()
$splatTemporal.DestinationTable = "dbatoolsci_positional_temporal"
$result = Copy-DbaDbTableData @splatTemporal
$WarnVar | Should -BeNullOrEmpty
$result.RowsCopied | Should -Be 2

$destData = $destinationDb.Query("SELECT Id, A, ValidFrom, B, ValidTo, C FROM dbo.dbatoolsci_positional_temporal ORDER BY Id")
$destData.A | Should -Be @(11, 111)
$destData.B | Should -Be @(22, 222)
$destData.C | Should -Be @(33, 333)
$destData.ValidFrom | Should -Not -BeNullOrEmpty
}

It "Still ignores the identity placeholder unless KeepIdentity is used" {
$splatIdentity = $splatPositional.Clone()
$splatIdentity.Query = "SELECT 0, A, B, C FROM dbo.dbatoolsci_positional_source ORDER BY Id"
$splatIdentity.DestinationTable = "dbatoolsci_positional_identity"
$result = Copy-DbaDbTableData @splatIdentity
$WarnVar | Should -BeNullOrEmpty
$result.RowsCopied | Should -Be 2

$destData = $destinationDb.Query("SELECT Id, A, B, C FROM dbo.dbatoolsci_positional_identity ORDER BY Id")
$destData.Id | Should -Be @(1, 2)
$destData.A | Should -Be @(11, 111)
$destData.C | Should -Be @(33, 333)
}

It "Refuses a query with more columns than the destination can take instead of dropping them" {
$splatTooMany = $splatPositional.Clone()
$splatTooMany.Query = "SELECT Id, A, B, C, C AS Extra FROM dbo.dbatoolsci_positional_source"
$splatTooMany.Truncate = $true
$result = Copy-DbaDbTableData @splatTooMany -WarningAction SilentlyContinue
$result | Should -BeNullOrEmpty
$WarnVar | Should -Match "5 columns"
$WarnVar | Should -Match "4 writable columns"
$destinationDb.Query("SELECT COUNT(*) AS RowCnt FROM dbo.dbatoolsci_positional_dest").RowCnt | Should -Be 0
}
}

Context "Regression tests" {
BeforeAll {
$PSDefaultParameterValues["*-Dba*:EnableException"] = $true
Expand Down