Repository navigation
[Go SDK] Make pipelinex.computeRoots and xlangx.HandlerParams ordering deterministic - #40880
Conversation
…g deterministic Fix two sources of map-iteration non-determinism in the Go SDK pipeline construction and cross-language expansion paths: 1. In pipelinex.computeRoots, sort the collected root transform IDs using sort.Sort(idSorted(roots)) before passing them to TopologicalSort. While TopologicalSort already sorts its internal frontier (ret.next) and output keys for deterministic traversal, the initial roots slice was populated by ranging over the xforms map in random order. 2. In xlangx.HandlerParams.Inputs() and Outputs(), sort the returned []PCol slice by PCol.Index (breaking ties by PCol.Local) after collecting entries from InputsMap and OutputsMap. Retaining append + slices.SortFunc (rather than indexing directly into a preallocated slice by index) also safely handles sparse index maps such as when xlang.SetOutputCoder rewrites OutputsMap to a single entry at index > 0.
There was a problem hiding this comment.
📋 Review Summary
This PR resolves non-deterministic ordering in pipelinex.computeRoots and xlangx.HandlerParams (Inputs() and Outputs()) caused by Go map iteration order. The changes are minimal, idiomatic, and well-tested.
🔍 General Feedback
- Sorting
rootswithidSortedprior toTopologicalSortincomputeRootsaligns with how transform IDs are ordered elsewhere inpipelinex. - Sorting
PColslices byIndex(withLocalas a tie-breaker) inHandlerParams.Inputs()andOutputs()cleanly guarantees deterministic ordering while preserving support for sparse index maps. - The added unit tests thoroughly cover both determinism across iterations and sparse index edge cases.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #40880 +/- ##
===========================================
+ Coverage 0 28.97% +28.97%
===========================================
Files 0 382 +382
Lines 0 71223 +71223
===========================================
+ Hits 0 20635 +20635
- Misses 0 48733 +48733
- Partials 0 1855 +1855
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Assigning reviewers: R: @shunping for label go. This pull request likely touches a core component ("core" label). Please review with scrutiny. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
When runners require deterministic pipeline graphs across restarts (such as for pipeline graph equality checks on controller failover), two map-iteration paths in the Go SDK can produce non-deterministic ordering:
pipelinex.computeRoots, root transform IDs are collected by ranging overxforms(map[string]*pipepb.PTransform) and passed directly toTopologicalSort(xforms, roots). WhileTopologicalSortalready sorts its internal frontier (ret.next) and output keys for determinism, the initialrootsslice was left in random map iteration order. Sortingrootswithsort.Sort(idSorted(roots))makes the root topological order deterministic.xlangx.HandlerParams.Inputs()andOutputs(),PColentries are collected by ranging overInputsMapandOutputsMap(map[string]int) in random order, which can cause expansion handlers to register cross-language inputs or outputs in varying orders across runs. Sorting the returned slice byPCol.Index(withPCol.Localas a tie-breaker) ensures a consistent order while still supporting sparse index maps (such as whenxlang.SetOutputCoderrewritesOutputsMapto a single entry at an index greater than 0).Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses %23123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes %23<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master)
[![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg%3Fevent=schedule