[Bug]: SolaceIO write transform loses output elements in batch pipelines #39589

What happened?

Beam 2.75.0

SolaceIO write transform has two "writer" types - batched (sends 50 messages per batch) and streaming (sends one message at a time). With both of those writers, there is the issue when using the writer in a batch-mode Beam pipeline.

The issue is that the batch pipeline will frequently end its run before Solace publish results are output by the SolaceIO write transform.

In case of UnboundedBatchedSolaceWriter, the issue is that it relies on a timer to "finalize" outputing the publish results. In a batch pipeline, timers are not really triggering as expected, at least the pipeline will not wait the 10 seconds which is configured in the code for the timer to fire. Instead, the pipeline will just end - output is lost.

In case of UnboundedStreamingSolaceWriter, there is no timer but it calls publishResults from the @FinishBundle method which assumes that by that time there are Solace publish results to output. When there aren't, the batch pipeline again will just end, as there is nothing holding back the watermark and it does not care that there are some Solace ACK callbacks left waiting. Output is lost.

CC @stankiewicz @iht

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Prism Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner