Skip to content

[Go SDK] Make pipelinex.computeRoots and xlangx.HandlerParams ordering deterministic - #40880

Merged
shunping merged 1 commit into
apache:masterfrom
malo-denielou:deterministic-pipelinex-xlangx
Oct 30, 2026
Merged

shunping merged 1 commit into
apache:masterfrom
malo-denielou:deterministic-pipelinex-xlangx

Conversation

@malo-denielou

Copy link
Copy Markdown
Contributor

When runners require deterministic pipeline graphs across restarts (such as for pipeline graph equality checks on controller failover), two map-iteration paths in the Go SDK can produce non-deterministic ordering:

  1. In pipelinex.computeRoots, root transform IDs are collected by ranging over xforms (map[string]*pipepb.PTransform) and passed directly to TopologicalSort(xforms, roots). While TopologicalSort already sorts its internal frontier (ret.next) and output keys for determinism, the initial roots slice was left in random map iteration order. Sorting roots with sort.Sort(idSorted(roots)) makes the root topological order deterministic.
  2. In xlangx.HandlerParams.Inputs() and Outputs(), PCol entries are collected by ranging over InputsMap and OutputsMap (map[string]int) in random order, which can cause expansion handlers to register cross-language inputs or outputs in varying orders across runs. Sorting the returned slice by PCol.Index (with PCol.Local as a tie-breaker) ensures a consistent order while still supporting sparse index maps (such as when xlang.SetOutputCoder rewrites OutputsMap to a single entry at an index greater than 0).

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses %23123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes %23<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master)

[![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg%3Fevent=schedule

…g deterministic

Fix two sources of map-iteration non-determinism in the Go SDK pipeline
construction and cross-language expansion paths:

1. In pipelinex.computeRoots, sort the collected root transform IDs using
   sort.Sort(idSorted(roots)) before passing them to TopologicalSort.
   While TopologicalSort already sorts its internal frontier (ret.next)
   and output keys for deterministic traversal, the initial roots slice
   was populated by ranging over the xforms map in random order.

2. In xlangx.HandlerParams.Inputs() and Outputs(), sort the returned
   []PCol slice by PCol.Index (breaking ties by PCol.Local) after
   collecting entries from InputsMap and OutputsMap. Retaining append +
   slices.SortFunc (rather than indexing directly into a preallocated
   slice by index) also safely handles sparse index maps such as when
   xlang.SetOutputCoder rewrites OutputsMap to a single entry at index > 0.
@github-actions github-actions Bot added go core size/s S: 10-49 lines changed labels Oct 29, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR resolves non-deterministic ordering in pipelinex.computeRoots and xlangx.HandlerParams (Inputs() and Outputs()) caused by Go map iteration order. The changes are minimal, idiomatic, and well-tested.

🔍 General Feedback

  • Sorting roots with idSorted prior to TopologicalSort in computeRoots aligns with how transform IDs are ordered elsewhere in pipelinex.
  • Sorting PCol slices by Index (with Local as a tie-breaker) in HandlerParams.Inputs() and Outputs() cleanly guarantees deterministic ordering while preserving support for sparse index maps.
  • The added unit tests thoroughly cover both determinism across iterations and sparse index edge cases.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@codecov

codecov Bot commented Oct 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 28.97%. Comparing base (0f1ac43) to head (e06cc8c).
⚠️ Report is 16 commits behind head on master.

Files with missing lines Patch % Lines
sdks/go/pkg/beam/core/runtime/xlangx/registry.go 83.33% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master   #40880       +/-   ##
===========================================
+ Coverage        0   28.97%   +28.97%     
===========================================
  Files           0      382      +382     
  Lines           0    71223    +71223     
===========================================
+ Hits            0    20635    +20635     
- Misses          0    48733    +48733     
- Partials        0     1855     +1855     
Flag Coverage Δ
go 28.97% <85.71%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @shunping for label go.
Backup reviewer: jrmccluskey (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

This pull request likely touches a core component ("core" label). Please review with scrutiny.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@shunping shunping left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@shunping
shunping merged commit 1c789c3 into apache:master Oct 30, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants