Copy-DbaDbTableData - Map query columns onto the writable destination columns

[andreasjordan]

Fixes #10661

Problem

With -Query and without -ForceExplicitMapping, a destination table with a computed or a rowversion column between its writable columns makes the copy go wrong: either the server rejects the insert, or - worse - every column behind the unwritable one shifts by one position, the last column stays empty, no warning is written and the command reports success.

Mechanism

In that mode the command added no column mappings at all and left SqlBulkCopy to map by position (Copy-DbaDbTableData.ps1:710). SqlBulkCopy's own list contains every destination column:

• a computed column stays in the list, so the server rejects the INSERT BULK with error 271 ("cannot be modified because it is either a computed column or is the result of a UNION operator") - a loud failure;
• a rowversion column is dropped from the list together with the source column that lands on it, and everything behind it moves one position to the left - the silent data corruption the issue describes.

Both reproduced in the lab (Microsoft.Data.SqlClient 6.1.5, SQL Server 2022 to 2025). The report names a computed column; on our stack that shape fails loudly and the silent shift needs a rowversion column. The reporter has since verified that their table has no rowversion column at all, so with their driver/server combination the computed column itself is dropped silently - the loud-vs-silent behaviour is version dependent. The fix covers both, since the command no longer relies on the implicit mapping in either shape.

What changed

• The destination column list the command already kept (non-computed columns) now also excludes rowversion columns (DataType.SqlDataType -eq "Timestamp"), i.e. it is the list of columns that can actually be written.
• In -Query mode without -ForceExplicitMapping the command now builds the positional mapping itself: query column n is mapped to the n-th writable destination column via ColumnMappings.Add([int], [string]). This is what SqlBulkCopy did on its own, minus the unwritable columns.
• A query that returns more columns than the destination has writable columns is refused with a message naming both counts, inside the existing try/catch so it honours -EnableException and -Continues to the next destination like every other failure there. Before, the surplus was either rejected by SqlBulkCopy with a generic error or silently lost.
• Help for -Query and -ForceExplicitMapping describes the new rule; the old text promised that ordinal mapping "will cause it to fail" on computed columns.

Identity placeholders keep working as documented: a mapping onto an identity column without -KeepIdentity is still ignored by the server, and with -KeepIdentity the value is written - verified with raw SqlBulkCopy before the change and by a test after it.

What deliberately did not change

Table mode and -ForceExplicitMapping keep their name-based mapping. The only effect the new column list has there is that a source column named like a destination rowversion column is no longer mapped - SqlBulkCopy tolerated that mapping anyway, so the result is identical.

Tests

New Context in Copy-DbaDbTableData.Tests.ps1 (COPY lane), asserting the data, not the row count:

1. computed + rowversion between the writable columns: values land in the right columns, the computed column computes;
2. rowversion only: the silent variant, values behind it are not shifted;
3. identity placeholder without -KeepIdentity: identity generated, other columns right;
4. a query with one column too many is refused with the counts in the warning and writes nothing.

Lab through the testing-dbatools harness (SQL 2022 -> SQL 2025): 18/18 green with the fix. Red on old with the command change stashed: 1 and 4 fail with the server's computed-column error, 2 fails with Expected @(22, 222) But was @(33, 333) - the reported symptom; 3 passes on both, it pins the behaviour that must not change.

Added while the PR was open

The two existing Query-mode tests ("relies on the default source database", "3 part query") selected Id from a table whose column is id. Fine on CI, which is case insensitive, but on the case sensitive SQL05 instances of the lab the server answers Invalid column name 'Id' and both tests fail with Expected 1, but got $null. Fixed in the third commit by using the column name as created; verified with the query on the CS instance and by re-running the whole file against the case sensitive COPY pair (SQL05SQL2022 -> SQL05SQL2025): 18/18 green, where the same file failed both Query-mode tests in the morning run.

The review pointed out that the writable-column filter still counted GENERATED ALWAYS columns - the period columns of a temporal table and the ledger metadata columns, which SMO reports with Computed = $false. The fourth commit excludes columns whose GeneratedAlwaysType is not None, guarded with $destServer.VersionMajor -ge 13 because SMO only supports the property on SQL Server 2016 and later, and adds a regression test with a system-versioned temporal destination whose period columns are interleaved between the writable ones. Green on both editions in the lab, red without the command change with exactly the server's 13536 error. Test file after the addition: 19/19 on both editions.

Thanks to @JankeUwe for the report and the row-level inspection that made the shift undeniable.

created by Claude and reviewed by Andreas Jordan

🤖 Generated with Claude Code

[potatoqualitee]

The positional destination mapping still treats temporal and ledger GENERATED ALWAYS columns as writable. For example, with destination columns A, ValidFrom, B, ValidTo and a query returning A, B, source B maps to ValidFrom and SqlBulkCopy fails because GENERATED ALWAYS period columns cannot accept explicit values. This materially breaks copying query results into supported temporal destinations. Exclude columns whose version-supported GeneratedAlwaysType is not None, guard the metadata lookup for older SQL Server versions, and add an interleaved temporal-column regression test.

[andreasjordan]

Ich can try to have a look at this tomorrow.

[andreasjordan]

Confirmed and fixed in 863aaa7. SMO reports the period columns of a temporal table with Computed = $false, so the writable-column filter did include them, the positional mapping put a query column onto ValidFrom, and the server rejected the insert with error 13536 ("Cannot insert an explicit value into a GENERATED ALWAYS column") - reproduced in the lab exactly as described, interleaved layout and all.

The filter now also excludes columns whose GeneratedAlwaysType is not None, which covers the temporal period columns and the ledger metadata columns in one condition. The lookup is guarded with $destServer.VersionMajor -ge 13, because SMO only supports the property on SQL Server 2016 and later - same pattern New-DbaDbMaskingConfig already uses. The name-based mapping of table mode profits as well, since it works from the same list.

New regression test: a system-versioned temporal destination with the period columns interleaved between the writable ones (Id, A, ValidFrom, B, ValidTo, C), asserting the values land in the right columns and the server fills the period. It skips below SQL Server 2016 via a discovery-time version check. Green on both editions through the testing-dbatools harness (SQL 2022 -> SQL 2025), and red without the command change, failing with exactly the 13536 error.

created by Claude and reviewed by Andreas Jordan