Skip to content

[Prism] Guard ClearOrderedListState against uninitialized state maps - #40626

Merged
jrmccluskey merged 2 commits into
apache:masterfrom
prodriguezdefino:prism/clear-ordered-list-nil-map
Oct 16, 2026
Merged

jrmccluskey merged 2 commits into
apache:masterfrom
prodriguezdefino:prism/clear-ordered-list-nil-map

Conversation

@prodriguezdefino

Copy link
Copy Markdown
Contributor

Problem:
In TentativeData.ClearOrderedListState (sdks/go/pkg/beam/runners/prism/internal/engine/data.go), Prism indexed directly into d.state[stateID][w] and wrote the cleared slice back to kMap[string(uKey)] without checking whether d.state, d.state[stateID], or winMap[w] had been initialized:

    winMap := d.state[stateID]
    w := d.toWindow(wKey)
    kMap := winMap[w]
    data := kMap[string(uKey)]
    ...
    kMap[string(uKey)] = StateData{Bag: cleared} // panic: assignment to entry in nil map

SDK harnesses (such as the Python SDK's SynchronousOrderedListRuntimeState.commit) buffer ordered-list mutations in memory during a bundle and flush pending StateRequest_Clear range requests before StateRequest_Append requests at bundle commit. When a bundle clears an ordered-list state cell (or range) in a window for which no prior bundle has persisted state on the runner, TentativeData.state has no map for (stateID, window), kMap is nil, and Prism panics on line 336.

Fix:
Use the existing d.clearState(stateID, wKey) helper in ClearOrderedListState and return early when kMap is nil, matching ClearBagState, ClearMultimapState, and ClearMultimapKeysState.

Reproduction (Python SDK on PrismRunner):

import apache_beam as beam
from apache_beam.coders import VarIntCoder
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.testing.util import assert_that, equal_to
from apache_beam.transforms.userstate import OrderedListStateSpec
from apache_beam.utils.timestamp import Timestamp

class ClearOrderedListDoFn(beam.DoFn):
    EVENTS = OrderedListStateSpec("events", VarIntCoder())

    def process(self, element, events=beam.DoFn.StateParam(EVENTS)):
        key, (ts, op) = element
        if op == "CLEAR":
            events.clear()
            yield f"cleared:{key}"
        else:
            events.add((Timestamp(micros=ts), op))
            entries = [f"{int(t.micros)}={v}" for t, v in events.read()]
            yield f"entry:{key}:{','.join(entries)}"

if __name__ == "__main__":
    options = PipelineOptions(["--runner=PrismRunner", "--environment_type=LOOPBACK"])
    with beam.Pipeline(options=options) as p:
        out = (
            p
            | beam.Create([("k", (30, 300)), ("k", (10, 100)), ("k", (0, "CLEAR"))])
            | beam.ParDo(ClearOrderedListDoFn())
        )
        assert_that(
            out,
            equal_to(["entry:k:30=300", "entry:k:10=100,30=300", "cleared:k"]),
        )

Problem:
In TentativeData.ClearOrderedListState (sdks/go/pkg/beam/runners/prism/internal/engine/data.go),
Prism indexed directly into d.state[stateID][w] and wrote the cleared slice back to
kMap[string(uKey)] without checking whether d.state, d.state[stateID], or winMap[w]
had been initialized:

    winMap := d.state[stateID]
    w := d.toWindow(wKey)
    kMap := winMap[w]
    data := kMap[string(uKey)]
    ...
    kMap[string(uKey)] = StateData{Bag: cleared} // panic: assignment to entry in nil map

SDK harnesses (such as the Python SDK's SynchronousOrderedListRuntimeState.commit and
the Rust SDK's OrderedListCell) buffer ordered-list mutations in memory during a bundle
and flush pending StateRequest_Clear range requests before StateRequest_Append requests
at bundle commit. When a bundle clears an ordered-list state cell (or range) in a
window for which no prior bundle has persisted state on the runner, TentativeData.state
has no map for (stateID, window), kMap is nil, and Prism panics on line 336.

Fix:
Use the existing d.clearState(stateID, wKey) helper in ClearOrderedListState and return
early when kMap is nil, matching ClearBagState, ClearMultimapState, and
ClearMultimapKeysState.

Reproduction (Python SDK on PrismRunner):
```python
import apache_beam as beam
from apache_beam.coders import VarIntCoder
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.testing.util import assert_that, equal_to
from apache_beam.transforms.userstate import OrderedListStateSpec
from apache_beam.utils.timestamp import Timestamp

class ClearOrderedListDoFn(beam.DoFn):
    EVENTS = OrderedListStateSpec("events", VarIntCoder())

    def process(self, element, events=beam.DoFn.StateParam(EVENTS)):
        key, (ts, op) = element
        if op == "CLEAR":
            events.clear()
            yield f"cleared:{key}"
        else:
            events.add((Timestamp(micros=ts), op))
            entries = [f"{int(t.micros)}={v}" for t, v in events.read()]
            yield f"entry:{key}:{','.join(entries)}"

if __name__ == "__main__":
    options = PipelineOptions(["--runner=PrismRunner", "--environment_type=LOOPBACK"])
    with beam.Pipeline(options=options) as p:
        out = (
            p
            | beam.Create([("k", (30, 300)), ("k", (10, 100)), ("k", (0, "CLEAR"))])
            | beam.ParDo(ClearOrderedListDoFn())
        )
        assert_that(
            out,
            equal_to(["entry:k:30=300", "entry:k:10=100,30=300", "cleared:k"]),
        )
```
@codecov

codecov Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.22%. Comparing base (76f6e6d) to head (ca7a555).
⚠️ Report is 200 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #40626      +/-   ##
============================================
+ Coverage     56.16%   56.22%   +0.05%     
============================================
  Files          1124      868     -256     
  Lines        178336   153512   -24824     
  Branches       1489        0    -1489     
============================================
- Hits         100170    86305   -13865     
+ Misses        75645    65367   -10278     
+ Partials       2521     1840     -681     
Flag Coverage Δ
go 28.64% <100.00%> (-0.50%) ⬇️
java ?
python 79.93% <ø> (+0.37%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Updated comments for clarity in data_test.go.
@prodriguezdefino
prodriguezdefino marked this pull request as ready for review October 16, 2026 07:22

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR fixes a nil map panic in Prism's TentativeData.ClearOrderedListState when clearing ordered list state for an uninitialized state ID or window. By using the existing d.clearState(stateID, wKey) helper and returning early when the key map is nil, the implementation aligns with ClearBagState, ClearMultimapState, and ClearMultimapKeysState.

🔍 General Feedback

  • The fix is clean, idiomatic, and consistent with the rest of TentativeData's state clearing methods.
  • Unit tests in data_test.go thoroughly cover the uninitialized state map cases (d.state == nil, missing stateID, and missing window entry followed by an append).

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @jrmccluskey for label go.
Backup reviewer: lostluck (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@jrmccluskey jrmccluskey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@jrmccluskey
jrmccluskey merged commit 3336b15 into apache:master Oct 16, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants