Skip to content

Remove twister2 runner support #40869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Abacn merged 1 commit into from
Oct 28, 2026
Merged

Remove twister2 runner support #40869

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .agent/skills/runners/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@ Runners execute Beam pipelines on distributed processing backends. Each runner t
| Flink | `runners/flink/` | Apache Flink |
| Spark | `runners/spark/` | Apache Spark |
| Jet | `runners/jet/` | Hazelcast Jet |
| Twister2 | `runners/twister2/` | Twister2 |

## Direct Runner
For local development and testing.
Expand Down
1 change: 0 additions & 1 deletion .github/ISSUE_TEMPLATE/bug.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,5 @@ body:
- label: "Component: Spark Runner"
- label: "Component: Flink Runner"
- label: "Component: Prism Runner"
- label: "Component: Twister2 Runner"
- label: "Component: Hazelcast Jet Runner"
- label: "Component: Google Cloud Dataflow Runner"
1 change: 0 additions & 1 deletion .github/ISSUE_TEMPLATE/failing_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,5 @@ body:
- label: "Component: Spark Runner"
- label: "Component: Flink Runner"
- label: "Component: Prism Runner"
- label: "Component: Twister2 Runner"
- label: "Component: Hazelcast Jet Runner"
- label: "Component: Google Cloud Dataflow Runner"
1 change: 0 additions & 1 deletion .github/ISSUE_TEMPLATE/feature.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,5 @@ body:
- label: "Component: Spark Runner"
- label: "Component: Flink Runner"
- label: "Component: Prism Runner"
- label: "Component: Twister2 Runner"
- label: "Component: Hazelcast Jet Runner"
- label: "Component: Google Cloud Dataflow Runner"
1 change: 0 additions & 1 deletion .github/ISSUE_TEMPLATE/task.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,5 @@ body:
- label: "Component: Spark Runner"
- label: "Component: Flink Runner"
- label: "Component: Prism Runner"
- label: "Component: Twister2 Runner"
- label: "Component: Hazelcast Jet Runner"
- label: "Component: Google Cloud Dataflow Runner"
2 changes: 1 addition & 1 deletion .github/autolabeler.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,4 +111,4 @@ io: ["sdks/go/pkg/beam/io/**/*", "sdks/java/io/**/*", "sdks/python/apache_beam/
"portability": ["runners/portability/**/*"]
"prism": ["runners/prism/**/*", "sdks/go/pkg/beam/runners/prism/**/*", "sdks/go/cmd/prism/**/*", "sdks/python/apache_beam/runners/portability/prism_runner.py","sdks/python/apache_beam/runners/portability/prism_runner_test.py"]
"spark": ["runners/spark/**/*", "sdks/go/pkg/beam/runners/spark/**/*"]
"twister2": ["runners/twister2/**/*"]

2 changes: 0 additions & 2 deletions .github/issue-rules.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,6 @@ rules:
addLabels: ['flink']
- contains: '[x] Component: Prism'
addLabels: ['prism']
- contains: '[x] Component: Twister2'
addLabels: ['twister2']
- contains: '[x] Component: Hazelcast'
addLabels: ['jet']
- contains: '[x] Component: Google Cloud Dataflow Runner'
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -357,7 +357,6 @@ PostCommit Jobs run in a schedule against master branch and generally do not get
| [ PostCommit Java ValidatesRunner Spark ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Spark.yml) | N/A |`beam_PostCommit_Java_ValidatesRunner_Spark.json`| [![.github/workflows/beam_PostCommit_Java_ValidatesRunner_Spark.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Spark.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Spark.yml?query=event%3Aschedule) |
| [ PostCommit Java ValidatesRunner SparkStructuredStreaming ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.yml) | N/A |`beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.json`| [![.github/workflows/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.yml?query=event%3Aschedule) |
| [ PostCommit Java ValidatesRunner Spark4 ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Spark4.yml) | N/A |`beam_PostCommit_Java_ValidatesRunner_Spark4.json`| [![.github/workflows/beam_PostCommit_Java_ValidatesRunner_Spark4.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Spark4.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Spark4.yml?query=event%3Aschedule) |
| [ PostCommit Java ValidatesRunner Twister2 ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Twister2.yml) | N/A |`beam_PostCommit_Java_ValidatesRunner_Twister2.json`| [![.github/workflows/beam_PostCommit_Java_ValidatesRunner_Twister2.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Twister2.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Twister2.yml?query=event%3Aschedule) |
| [ PostCommit Java ValidatesRunner ULR ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_ULR.yml) | N/A |`beam_PostCommit_Java_ValidatesRunner_ULR.json`| [![.github/workflows/beam_PostCommit_Java_ValidatesRunner_ULR.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_ULR.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_ULR.yml?query=event%3Aschedule) |
| [ PostCommit Java ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java.yml) | N/A |`beam_PostCommit_Java.json`| [![.github/workflows/beam_PostCommit_Java.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java.yml?query=event%3Aschedule) |
| [ PostCommit Javadoc ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Javadoc.yml) | N/A |`beam_PostCommit_Javadoc.json`| [![.github/workflows/beam_PostCommit_Javadoc.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Javadoc.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Javadoc.yml?query=event%3Aschedule) |
Expand Down
89 changes: 0 additions & 89 deletions .github/workflows/beam_PostCommit_Java_ValidatesRunner_Twister2.yml
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion .test-infra/metrics/sync/github/github_runs_prefetcher/code/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,6 @@ categories:
- "PreCommit Java Kudu IO Direct"
- "PostCommit Java InfluxDbIO Integration Test"
- "PostCommit Java Jpms Direct Java21"
- "PostCommit Java ValidatesRunner Twister2"
- "PreCommit Java Neo4j IO Direct"
- "PostCommit Java Jpms Direct Java11"
- "PostCommit Javadoc"
Expand Down
1 change: 1 addition & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@
* (Python) The `xgboost` extras now install `xgboost>=2.0.0,<4` and `datatable>=1.1.0,<2`, moving both the lower and upper bounds for both. The previous minimum `xgboost` and `datatable` versions (`1.6.0` and `1.0.0` respectively) are no longer allowed ([#40694](https://github.com/apache/beam/issues/40694)).
* (Java) [IcebergIO] AddFiles (`IcebergAddFiles` in YAML) now carries the destination table alongside each file through the registration chain, so that one transform can register files into many tables. The coders of its internal `PCollection`s change, which makes this an update incompatible change for streaming pipelines that use it ([#40817](https://github.com/apache/beam/pull/40817)).
* (YAML) `IcebergAddFiles` will be automatically replaced with the compatible managed transform. For older Beam versions, streaming update compatibility can be maintained by specifying the pipeline option `update_compatibility_version` ([#40827](https://github.com/apache/beam/issues/40827)).
* Twister2 runner supported is removed ([#35905](https://github.com/apache/beam/issues/35905))).

## Deprecations

Expand Down
1 change: 0 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,6 @@ Beam supports executing programs on multiple distributed processing backends thr
- The `FlinkRunner` runs the pipeline on an Apache Flink cluster. The code has been donated from [dataArtisans/flink-dataflow](https://github.com/dataArtisans/flink-dataflow) and is now part of Beam.
- The `SparkRunner` runs the pipeline on an Apache Spark cluster.
- The `JetRunner` runs the pipeline on a Hazelcast Jet cluster. The code has been donated from [hazelcast/hazelcast-jet](https://github.com/hazelcast/hazelcast-jet) and is now part of Beam.
- The `Twister2Runner` runs the pipeline on a Twister2 cluster. The code has been donated from [DSC-SPIDAL/twister2](https://github.com/DSC-SPIDAL/twister2) and is now part of Beam.

Have ideas for new Runners? See the [runner-ideas label](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Arunner-ideas).

Expand Down
1 change: 0 additions & 1 deletion build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,6 @@ tasks.register("javaPreCommit") {
dependsOn(":runners:prism:java:build")
dependsOn(":runners:spark:3:build")
dependsOn(":runners:spark:3:job-server:build")
dependsOn(":runners:twister2:build")
dependsOn(":sdks:java:build-tools:build")
dependsOn(":sdks:java:container:java11:docker")
dependsOn(":sdks:java:core:build")
Expand Down
48 changes: 0 additions & 48 deletions release/src/main/groovy/quickstart-java-twister2.groovy
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion release/src/main/scripts/script.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@ RC_VALIDATE_CONFIGS=(
java_quickstart_flink_local
java_quickstart_spark_local
java_quickstart_dataflow
java_quickstart_twister2_local
java_mobile_game_direct
java_mobile_game_dataflow
python_quickstart_mobile_game
Expand Down
112 changes: 0 additions & 112 deletions runners/twister2/build.gradle
View file
Open in desktop

This file was deleted.

Loading
Loading