[Task]: Investigate Failures in Go Flink Load Tests with original load #33753

Activity

1. Amar3tto

   added

   awaiting triage

   flink

   go

   task

   on Jan 24, 2025
2. github-actions

   added

   P2

   on Jan 24, 2025
3. github-actions commented on Jun 23, 2025

   

   github-actionsBot

   on Jun 23, 2025 – with GitHub Actions

   Contributor

   More actions

   This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
4. github-actions

   added

   stale

   on Jun 23, 2025
5. github-actions commented on Jul 23, 2025

   

   github-actionsBot

   on Jul 23, 2025 – with GitHub Actions

   Contributor

   More actions

   This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
6. github-actions

   closed this as not plannedon Jul 23, 2025
7. Amar3tto

   reopened this on Oct 10, 2025
8. github-actions

   removed

   stale

   on Oct 10, 2025
9. github-actions commented on Mar 9, 2026

   

   github-actionsBot

   on Mar 9, 2026 – with GitHub Actions

   Contributor

   More actions

   This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
10. github-actions

    added

    stale

    on Mar 9, 2026
11. github-actions commented on Apr 8, 2026

    

    github-actionsBot

    on Apr 8, 2026 – with GitHub Actions

    Contributor

    More actions

    This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
12. github-actions

    closed this as not plannedon Apr 8, 2026
13. Abacn

    removed

    stale

    awaiting triage

    on Aug 14, 2026
14. Abacn

    reopened this on Aug 14, 2026
15. Abacn

    added

    pinned

    on Aug 14, 2026
16. Abacn commented on Aug 14, 2026

    

    Abacn

    on Aug 14, 2026

    Contributor

    More actions

    The load has been decreased multiple times since this this issue open. Did a survey and we can revert some of the decreased load, but not all.

    In particular Go GBK test 10b and 100b would still fail on OutOfMemoryError:

    Caused by: java.lang.OutOfMemoryError: Java heap space
    	at org.apache.beam.sdk.values.KV.of(KV.java:45)
    	at org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator$SdkHarnessDoFnRunner$1.lambda$create$0(ExecutableStageDoFnOperator.java:965)
    	at org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator$SdkHarnessDoFnRunner$1$$Lambda/0x00007f97dfd5dde0.accept(Unknown Source)
    	at org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.multiplexElements(BeamFnDataInboundObserver.java:232)
    	at org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.awaitCompletion(BeamFnDataInboundObserver.java:186)
    	at org.apache.beam.runners.fnexecution.control.SdkHarnessClient$BundleProcessor$ActiveBundle.close(SdkHarnessClient.java:565)
    	at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory$1.close(DefaultJobBundleFactory.java:562)
    	at org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator$SdkHarnessDoFnRunner.finishBundle(ExecutableStageDoFnOperator.java:1067)
    	at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.finishBundle(DoFnRunnerWithMetricsUpdate.java:100)
    	at org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:133)
    	at org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.invokeFinishBundle(DoFnOperator.java:1044)
    	at org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator.flushData(ExecutableStageDoFnOperator.java:719)
    	at org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.finish(DoFnOperator.java:691)
    	at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper$$Lambda/0x00007f97dfd2e248.run(Unknown Source)
    	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
    

    Likely we need to use rockdb state backend instead of the default memory one for Flink runner. This option hasn't exposed in Go SDK. And if does, not sure it's working correctly as there isn't test coverage.