Repository navigation
Conversation
…h timestamp contamination, and parameter checks
There was a problem hiding this comment.
📋 Review Summary
This PR fixes several important bugs in GroupIntoBatches, including updating timer watermark holds for out-of-order elements, persisting timerTs to prevent processing-time timer starvation, and preventing timestamp overflow on window expiration calculations. It also adds display data support, improves precondition checks, and fixes Javadoc formatting.
🔍 General Feedback
- The state management and watermark hold logic in
GroupIntoBatchesDoFnare clean and properly handle out-of-order elements without resetting the processing-time deadline. testOutOfOrderTimestampsInBatchcould be strengthened to directly verify the updated timer watermark hold (Right now, a batch size of 2 flushes immediately insideprocessElement, which clears the timer before the watermark advances).
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
… hold on window expiration
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
I made the changes more focused. There were some PreCommit failures, but I don't see anything related to GroupIntoBatches in the failing tests. |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
| @ProcessElement | ||
| public void processElement( | ||
| @Timestamp Instant ts, OutputReceiver<Long> receiver) { | ||
| receiver.output(ts.getMillis()); |
There was a problem hiding this comment.
the fix looks good to me. Is the test failing without the fix?
It seems like receiver.output will get fused with the GroupIntoBatches timer and i'm not sure if the elements will be dropped as late.
There was a problem hiding this comment.
The elements were getting output with the wrong timestamps. Let me modify the test so that there is something like a GBK afterwards and we can directly verify that the elements get dropped due to an incorrect watermark hold. In the new example, the batch gets emitted with timestamp 2s, but the watermark hold was set to 10s.
…revents downstream late data drops
arunpandianp
left a comment
There was a problem hiding this comment.
Thanks for fixing this!
| MoreObjects.firstNonNull( | ||
| minBufferedTs.read(), BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis()); | ||
| boolean needsNewTimer = num == 1 || minBufferedTs.read() != oldOutputTs; | ||
| boolean needsNewTimer = num == 1 || currentMinBufferedTs < oldOutputTs; |
There was a problem hiding this comment.
nit: i like currentMinBufferedTs != oldOutputTs, set a timer when MinBufferedTs is modified is easier to read than when currentMinBufferedTs < oldOutputTs. they both are true when we need a new timer.
Watermark Hold for Out-of-Order Elements:
storedBatchSize.read() == 1). If subsequent elements arrived with timestamps earlier than the first element (out-of-order within the window), the watermark hold remained pinned to the first element's timestamp, risking premature watermark advancement.GroupIntoBatchescomparescurrentMinBufferedTsagainstoldMinBufferedTsand updates the timer's output timestamp whenever an earlier timestamp arrives (num == 1 || currentMinBufferedTs < oldMinBufferedTs).Tests:
testOutOfOrderTimestampsInBatchverifying out of order elements don't get dropped as late data.