Copy-DbaDbTableData verbose progress counter over-reports rows copied on final batch

[JankeUwe]

Environment: dbatools 2.8.2 (also seen on 2.7.1), SQL Server

Description:
The live progress number Copy-DbaDbTableData writes to its verbose stream (e.g. "adjusted total rows copied = N") can over-report versus the destination table's real row count once the copy finishes. This looks like the command's own internal bookkeeping - built to work around SqlBulkCopy's legacy 32-bit RowsCopied counter - has a double-counting edge case on the final batch.

Repro:

• Run Copy-DbaDbTableData with -Verbose on a table large enough to span multiple batches (-BatchSize).
• Watch the verbose "adjusted total rows copied" value as the final batch completes.
• Compare against SELECT COUNT_BIG(*) on the destination table after the run finishes.
• The verbose total can be higher than the real row count.

Impact:
Cosmetic only - the extra count is not reflected in what's actually written to the destination, confirmed by a real COUNT_BIG(*) afterwards. But it's misleading for anyone live-monitoring a large transfer and expecting the reported total to match the final row count, and it can make a clean transfer look like it over-copied when it didn't.

Expected:
The verbose "rows copied" total should match the destination's actual row count once the last batch completes, or the discrepancy should be documented as a known artifact of the 32-bit counter workaround.

Workaround used:
Don't trust the live verbose counter for the final total - do a direct COUNT_BIG(*) comparison against source/destination after the transfer instead.

[niphlod]

sidenote (but don't take me for granted) the counter issue was valid with a reaally old library version. It might have been fixed on the latest (if down the line everything eventually resolves to Microsoft.Data.SqlClient in a semi-recent version)

[andreasjordan]

Analyzed on development against the current stack (Microsoft.Data.SqlClient 6.1.5, SQL Server 2022 -> 2025), with a split result.

@niphlod is right about the counter itself: SqlBulkCopy._rowsCopied is an Int64 in current Microsoft.Data.SqlClient, so the wrapping 4-byte counter that Get-AdjustedTotalRowsCopied works around is history. A multi-batch copy (tested with the batch size both aligned and not aligned to the row count) shows correct live verbose counters and a correct final total - below ~2.15 billion rows the over-report does not reproduce here.

Above [int32]::MaxValue rows, though, the bookkeeping does over-report the final total, and the mechanism sits exactly where you pointed: the helper that reads the counter by reflection still casts it to [int], above 2,147,483,647 rows that cast throws, the -1 failure sentinel is returned, and the wrap-adjustment then takes the -1 for a wrapped counter and adds ~1.3 billion phantom rows. A 3,000,000,000-row copy reports 4,294,967,295 rows copied while the destination holds the correct 3,000,000,000 - data right, reported total wrong. Fixed in #10677, together with the same pattern in Import-DbaCsv and Import-DbaParquet.

One question to close the loop: did your transfer exceed ~2.15 billion rows? If yes, your report is fully explained by the above. If it was smaller, the over-report you saw has a different cause, and the tail of the -Verbose output (the last few "The bulk copy library reported RowsCopied = ..." lines plus the final "N rows inserted" line, with the real COUNT_BIG(*)) would tell us where the numbers diverge.

created by Claude and reviewed by Andreas Jordan

[JankeUwe]

Good question - our transfer was well under that threshold: ~344.9 million rows, not billions. So this doesn't match the >2.15B mechanism you found and fixed in #10677.

Unfortunately we no longer have the original verbose log from that run (it wasn't saved at the time), so I can't hand you the exact "RowsCopied = ..." tail to pin down where the divergence happened on our end. Given the counter is now Int64 and you couldn't reproduce an over-report below 2.15B on the current stack, it's plausible our case was simply the old 32-bit wraparound bug on whatever client library version we were running at the time, as niphlod suggested - i.e. already fixed by the Int64 migration, unrelated to the >2.15B issue in #10677. Sorry I can't offer more concrete evidence than that; happy to reopen with logs if we hit it again on a current dbatools version.

[andreasjordan]

Thanks - that settles the question, with one correction to the remaining theory: at ~344.9 million rows the old 32-bit wraparound cannot explain it either. The legacy 4-byte counter only wraps above 2,147,483,647 rows, the same threshold as the mechanism fixed in #10677, so below that line both the old and the new stack count correctly. Whatever you saw at 344.9M rows was neither of the two known mechanisms.

Without the verbose tail there is nothing left to chase, so your offer is exactly the right close: if it recurs on a current dbatools version, a fresh issue with the last few "The bulk copy library reported RowsCopied = ..." lines, the final "N rows inserted" line and the real COUNT_BIG(*) will pin it down quickly. This issue will close with #10677, which hardens the counter path regardless - the reflection read now survives any row count, and a failed read can no longer masquerade as a wrapped counter.

created by Claude and reviewed by Andreas Jordan