Skip to content

[Spanner Change Stream] Make InitializeDoFn fake parent partition creation idempotent - #40808

Closed
jiangzzhu wants to merge 1 commit into
apache:masterfrom
jiangzzhu:fix-changestream-initialize-dofn-idempotent
Closed

jiangzzhu wants to merge 1 commit into
apache:masterfrom
jiangzzhu:fix-changestream-initialize-dofn-idempotent

Conversation

@jiangzzhu

Copy link
Copy Markdown
Contributor

Summary

Makes InitializeDoFn.createFakeParentPartition() idempotent under overlapping or retried InitializeDoFn executions during pipeline startup:

  • Problem: Because DoFn execution has at-least-once semantics, two workers can execute InitializeDoFn concurrently during job startup before the first CREATE TABLE IF NOT EXISTS DDL operation completes. Both executions observe partitionMetadataDao.tableExists() == false, both execute createPartitionMetadataTable() (CREATE TABLE IF NOT EXISTS), and both proceed to createFakeParentPartition(), causing the second execution to fail with SpannerException: ALREADY_EXISTS: Row [Parent0] ... already exists.
  • Fix:
    • Inside a read-write transaction (partitionMetadataDao.runInTransaction), check transaction.getPartition(...) == null before inserting each initial parent partition (consistent with ChildPartitionsRecordAction and PartitionStartRecordAction), ensuring existing Parent0 rows (and any state transitions already in progress) are never overwritten.
    • Also catch and ignore SpannerException with ErrorCode.ALREADY_EXISTS as a safeguard against concurrent insert races.

Testing

  • Updated and expanded unit tests in InitializeDoFnTest covering new insertions, pre-existing initial partitions (single and TVF list), ErrorCode.ALREADY_EXISTS handling, and non-ALREADY_EXISTS exception propagation.

@github-actions github-actions Bot added java io gcp spanner size/m M: 50-249 lines changed labels Oct 26, 2026
@jiangzzhu jiangzzhu closed this Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant