Skip to content

Fixes some bugs with GroupIntoBatches. - #40754

Open
acrites wants to merge 9 commits into
apache:masterfrom
acrites:group-into-batches-fix
Open

acrites wants to merge 9 commits into
apache:masterfrom
acrites:group-into-batches-fix

Conversation

@acrites

@acrites acrites commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Watermark Hold for Out-of-Order Elements:

  • Previously, the timer watermark hold was only set when buffering the first element (storedBatchSize.read() == 1). If subsequent elements arrived with timestamps earlier than the first element (out-of-order within the window), the watermark hold remained pinned to the first element's timestamp, risking premature watermark advancement.
  • Now, GroupIntoBatches compares currentMinBufferedTs against oldMinBufferedTs and updates the timer's output timestamp whenever an earlier timestamp arrives (num == 1 || currentMinBufferedTs < oldMinBufferedTs).

Tests:

  • Added testOutOfOrderTimestampsInBatch verifying out of order elements don't get dropped as late data.

@github-actions github-actions Bot added java core size/m M: 50-249 lines changed labels Oct 23, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR fixes several important bugs in GroupIntoBatches, including updating timer watermark holds for out-of-order elements, persisting timerTs to prevent processing-time timer starvation, and preventing timestamp overflow on window expiration calculations. It also adds display data support, improves precondition checks, and fixes Javadoc formatting.

🔍 General Feedback

  • The state management and watermark hold logic in GroupIntoBatchesDoFn are clean and properly handle out-of-order elements without resetting the processing-time deadline.
  • testOutOfOrderTimestampsInBatch could be strengthened to directly verify the updated timer watermark hold (Right now, a batch size of 2 flushes immediately inside processElement, which clears the timer before the watermark advances).

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@github-actions github-actions Bot added size/s S: 10-49 lines changed and removed size/m M: 50-249 lines changed labels Oct 26, 2026
@acrites

acrites commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

R: @arunpandianp

I made the changes more focused. There were some PreCommit failures, but I don't see anything related to GroupIntoBatches in the failing tests.

@github-actions

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

@ProcessElement
public void processElement(
@Timestamp Instant ts, OutputReceiver<Long> receiver) {
receiver.output(ts.getMillis());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the fix looks good to me. Is the test failing without the fix?

It seems like receiver.output will get fused with the GroupIntoBatches timer and i'm not sure if the elements will be dropped as late.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The elements were getting output with the wrong timestamps. Let me modify the test so that there is something like a GBK afterwards and we can directly verify that the elements get dropped due to an incorrect watermark hold. In the new example, the batch gets emitted with timestamp 2s, but the watermark hold was set to 10s.

@arunpandianp arunpandianp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this!

MoreObjects.firstNonNull(
minBufferedTs.read(), BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis());
boolean needsNewTimer = num == 1 || minBufferedTs.read() != oldOutputTs;
boolean needsNewTimer = num == 1 || currentMinBufferedTs < oldOutputTs;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: i like currentMinBufferedTs != oldOutputTs, set a timer when MinBufferedTs is modified is easier to read than when currentMinBufferedTs < oldOutputTs. they both are true when we need a new timer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core java size/s S: 10-49 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants