diff --git a/.agent/skills/runners/SKILL.md b/.agent/skills/runners/SKILL.md index b5e25f9898ee..d7c576109f31 100644 --- a/.agent/skills/runners/SKILL.md +++ b/.agent/skills/runners/SKILL.md @@ -35,7 +35,6 @@ Runners execute Beam pipelines on distributed processing backends. Each runner t | Flink | `runners/flink/` | Apache Flink | | Spark | `runners/spark/` | Apache Spark | | Jet | `runners/jet/` | Hazelcast Jet | -| Twister2 | `runners/twister2/` | Twister2 | ## Direct Runner For local development and testing. diff --git a/.github/ISSUE_TEMPLATE/bug.yml b/.github/ISSUE_TEMPLATE/bug.yml index 594dbbc59b22..cf8587a28cf5 100644 --- a/.github/ISSUE_TEMPLATE/bug.yml +++ b/.github/ISSUE_TEMPLATE/bug.yml @@ -73,6 +73,5 @@ body: - label: "Component: Spark Runner" - label: "Component: Flink Runner" - label: "Component: Prism Runner" - - label: "Component: Twister2 Runner" - label: "Component: Hazelcast Jet Runner" - label: "Component: Google Cloud Dataflow Runner" diff --git a/.github/ISSUE_TEMPLATE/failing_test.yml b/.github/ISSUE_TEMPLATE/failing_test.yml index bc461771d4dc..3bf44a324755 100644 --- a/.github/ISSUE_TEMPLATE/failing_test.yml +++ b/.github/ISSUE_TEMPLATE/failing_test.yml @@ -79,6 +79,5 @@ body: - label: "Component: Spark Runner" - label: "Component: Flink Runner" - label: "Component: Prism Runner" - - label: "Component: Twister2 Runner" - label: "Component: Hazelcast Jet Runner" - label: "Component: Google Cloud Dataflow Runner" diff --git a/.github/ISSUE_TEMPLATE/feature.yml b/.github/ISSUE_TEMPLATE/feature.yml index abde43ea7302..f61a1db763b1 100644 --- a/.github/ISSUE_TEMPLATE/feature.yml +++ b/.github/ISSUE_TEMPLATE/feature.yml @@ -67,6 +67,5 @@ body: - label: "Component: Spark Runner" - label: "Component: Flink Runner" - label: "Component: Prism Runner" - - label: "Component: Twister2 Runner" - label: "Component: Hazelcast Jet Runner" - label: "Component: Google Cloud Dataflow Runner" diff --git a/.github/ISSUE_TEMPLATE/task.yml b/.github/ISSUE_TEMPLATE/task.yml index c02ff781ad44..a92274fbd624 100644 --- a/.github/ISSUE_TEMPLATE/task.yml +++ b/.github/ISSUE_TEMPLATE/task.yml @@ -68,6 +68,5 @@ body: - label: "Component: Spark Runner" - label: "Component: Flink Runner" - label: "Component: Prism Runner" - - label: "Component: Twister2 Runner" - label: "Component: Hazelcast Jet Runner" - label: "Component: Google Cloud Dataflow Runner" diff --git a/.github/autolabeler.yml b/.github/autolabeler.yml index b6c2ed09d98a..6b3e4b89b628 100644 --- a/.github/autolabeler.yml +++ b/.github/autolabeler.yml @@ -111,4 +111,4 @@ io: ["sdks/go/pkg/beam/io/**/*", "sdks/java/io/**/*", "sdks/python/apache_beam/ "portability": ["runners/portability/**/*"] "prism": ["runners/prism/**/*", "sdks/go/pkg/beam/runners/prism/**/*", "sdks/go/cmd/prism/**/*", "sdks/python/apache_beam/runners/portability/prism_runner.py","sdks/python/apache_beam/runners/portability/prism_runner_test.py"] "spark": ["runners/spark/**/*", "sdks/go/pkg/beam/runners/spark/**/*"] -"twister2": ["runners/twister2/**/*"] + diff --git a/.github/issue-rules.yml b/.github/issue-rules.yml index 4fafb1c87f78..534de5a81482 100644 --- a/.github/issue-rules.yml +++ b/.github/issue-rules.yml @@ -54,8 +54,6 @@ rules: addLabels: ['flink'] - contains: '[x] Component: Prism' addLabels: ['prism'] -- contains: '[x] Component: Twister2' - addLabels: ['twister2'] - contains: '[x] Component: Hazelcast' addLabels: ['jet'] - contains: '[x] Component: Google Cloud Dataflow Runner' diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 178fb6502680..682b51699c09 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -357,7 +357,6 @@ PostCommit Jobs run in a schedule against master branch and generally do not get | [ PostCommit Java ValidatesRunner Spark ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Spark.yml) | N/A |`beam_PostCommit_Java_ValidatesRunner_Spark.json`| [![.github/workflows/beam_PostCommit_Java_ValidatesRunner_Spark.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Spark.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Spark.yml?query=event%3Aschedule) | | [ PostCommit Java ValidatesRunner SparkStructuredStreaming ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.yml) | N/A |`beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.json`| [![.github/workflows/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.yml?query=event%3Aschedule) | | [ PostCommit Java ValidatesRunner Spark4 ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Spark4.yml) | N/A |`beam_PostCommit_Java_ValidatesRunner_Spark4.json`| [![.github/workflows/beam_PostCommit_Java_ValidatesRunner_Spark4.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Spark4.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Spark4.yml?query=event%3Aschedule) | -| [ PostCommit Java ValidatesRunner Twister2 ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Twister2.yml) | N/A |`beam_PostCommit_Java_ValidatesRunner_Twister2.json`| [![.github/workflows/beam_PostCommit_Java_ValidatesRunner_Twister2.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Twister2.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Twister2.yml?query=event%3Aschedule) | | [ PostCommit Java ValidatesRunner ULR ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_ULR.yml) | N/A |`beam_PostCommit_Java_ValidatesRunner_ULR.json`| [![.github/workflows/beam_PostCommit_Java_ValidatesRunner_ULR.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_ULR.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_ULR.yml?query=event%3Aschedule) | | [ PostCommit Java ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java.yml) | N/A |`beam_PostCommit_Java.json`| [![.github/workflows/beam_PostCommit_Java.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java.yml?query=event%3Aschedule) | | [ PostCommit Javadoc ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Javadoc.yml) | N/A |`beam_PostCommit_Javadoc.json`| [![.github/workflows/beam_PostCommit_Javadoc.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Javadoc.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Javadoc.yml?query=event%3Aschedule) | diff --git a/.github/workflows/beam_PostCommit_Java_ValidatesRunner_Twister2.yml b/.github/workflows/beam_PostCommit_Java_ValidatesRunner_Twister2.yml deleted file mode 100644 index a25de9168109..000000000000 --- a/.github/workflows/beam_PostCommit_Java_ValidatesRunner_Twister2.yml +++ /dev/null @@ -1,89 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -name: PostCommit Java ValidatesRunner Twister2 - -on: - schedule: - - cron: '45 4/6 * * *' - pull_request: - paths: ['release/trigger_all_tests.json', '.github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Twister2.json'] - workflow_dispatch: - -# Setting explicit permissions for the action -permissions: - actions: write - pull-requests: write - checks: write - contents: read - deployments: read - id-token: none - issues: write - discussions: read - packages: read - pages: read - repository-projects: read - security-events: read - statuses: read - -# This allows a subsequently queued workflow run to interrupt previous runs -concurrency: - group: '${{ github.workflow }} @ ${{ github.event.pull_request.number || github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.sender.login }}' - cancel-in-progress: true - -env: - DEVELOCITY_ACCESS_KEY: ${{ secrets.BEAM_DEVELOCITY_ACCESS_KEY }} - -jobs: - beam_PostCommit_Java_ValidatesRunner_Twister2: - name: ${{ matrix.job_name }} - runs-on: [self-hosted, ubuntu-24.04, main] - timeout-minutes: 120 - strategy: - matrix: - job_name: [beam_PostCommit_Java_ValidatesRunner_Twister2] - if: | - github.event_name == 'workflow_dispatch' || - github.event_name == 'pull_request' || - (github.event_name == 'schedule' && github.repository == 'apache/beam') - steps: - - uses: actions/checkout@v7 - with: - persist-credentials: false - - name: Setup repository - uses: ./.github/actions/setup-action - - name: Setup environment - uses: ./.github/actions/setup-environment-action - with: - java-version: default - - name: run validatesRunner script - uses: ./.github/actions/gradle-command-self-hosted-action - with: - gradle-command: :runners:twister2:validatesRunner - - name: Archive JUnit Test Results - uses: actions/upload-artifact@v7 - if: ${{ !success() }} - with: - name: JUnit Test Results - path: "**/build/reports/tests/" - - name: Publish JUnit Test Results - uses: EnricoMi/publish-unit-test-result-action@v2 - if: always() - with: - commit: '${{ env.GITHUB_SHA }}' - comment_mode: 'off' - check_run: ${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository }} - files: '**/build/test-results/**/*.xml' - large_files: true diff --git a/.test-infra/metrics/sync/github/github_runs_prefetcher/code/config.yaml b/.test-infra/metrics/sync/github/github_runs_prefetcher/code/config.yaml index 154bd1fe8884..a0729b9932ea 100644 --- a/.test-infra/metrics/sync/github/github_runs_prefetcher/code/config.yaml +++ b/.test-infra/metrics/sync/github/github_runs_prefetcher/code/config.yaml @@ -119,7 +119,6 @@ categories: - "PreCommit Java Kudu IO Direct" - "PostCommit Java InfluxDbIO Integration Test" - "PostCommit Java Jpms Direct Java21" - - "PostCommit Java ValidatesRunner Twister2" - "PreCommit Java Neo4j IO Direct" - "PostCommit Java Jpms Direct Java11" - "PostCommit Javadoc" diff --git a/CHANGES.md b/CHANGES.md index ca0da6986592..0d9d9e0386f5 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -103,6 +103,7 @@ * (Python) The `xgboost` extras now install `xgboost>=2.0.0,<4` and `datatable>=1.1.0,<2`, moving both the lower and upper bounds for both. The previous minimum `xgboost` and `datatable` versions (`1.6.0` and `1.0.0` respectively) are no longer allowed ([#40694](https://github.com/apache/beam/issues/40694)). * (Java) [IcebergIO] AddFiles (`IcebergAddFiles` in YAML) now carries the destination table alongside each file through the registration chain, so that one transform can register files into many tables. The coders of its internal `PCollection`s change, which makes this an update incompatible change for streaming pipelines that use it ([#40817](https://github.com/apache/beam/pull/40817)). * (YAML) `IcebergAddFiles` will be automatically replaced with the compatible managed transform. For older Beam versions, streaming update compatibility can be maintained by specifying the pipeline option `update_compatibility_version` ([#40827](https://github.com/apache/beam/issues/40827)). +* Twister2 runner supported is removed ([#35905](https://github.com/apache/beam/issues/35905))). ## Deprecations diff --git a/README.md b/README.md index 2cf529c53eb8..1cf44696db5e 100644 --- a/README.md +++ b/README.md @@ -92,7 +92,6 @@ Beam supports executing programs on multiple distributed processing backends thr - The `FlinkRunner` runs the pipeline on an Apache Flink cluster. The code has been donated from [dataArtisans/flink-dataflow](https://github.com/dataArtisans/flink-dataflow) and is now part of Beam. - The `SparkRunner` runs the pipeline on an Apache Spark cluster. - The `JetRunner` runs the pipeline on a Hazelcast Jet cluster. The code has been donated from [hazelcast/hazelcast-jet](https://github.com/hazelcast/hazelcast-jet) and is now part of Beam. -- The `Twister2Runner` runs the pipeline on a Twister2 cluster. The code has been donated from [DSC-SPIDAL/twister2](https://github.com/DSC-SPIDAL/twister2) and is now part of Beam. Have ideas for new Runners? See the [runner-ideas label](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Arunner-ideas). diff --git a/build.gradle.kts b/build.gradle.kts index d0ece7a77606..baff865c09a3 100644 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -289,7 +289,6 @@ tasks.register("javaPreCommit") { dependsOn(":runners:prism:java:build") dependsOn(":runners:spark:3:build") dependsOn(":runners:spark:3:job-server:build") - dependsOn(":runners:twister2:build") dependsOn(":sdks:java:build-tools:build") dependsOn(":sdks:java:container:java11:docker") dependsOn(":sdks:java:core:build") diff --git a/release/src/main/groovy/quickstart-java-twister2.groovy b/release/src/main/groovy/quickstart-java-twister2.groovy deleted file mode 100644 index b46538cdd54c..000000000000 --- a/release/src/main/groovy/quickstart-java-twister2.groovy +++ /dev/null @@ -1,48 +0,0 @@ -#!groovy -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -t = new TestScripts(args) - -/* - * Run the Twister2 quickstart from https://beam.apache.org/get-started/quickstart-java/ - */ - -t.describe 'Run Apache Beam Java SDK Quickstart - Twister2' - - t.intent 'Gets the WordCount Example Code' - QuickstartArchetype.generate(t) - - t.intent 'Runs the WordCount Code with Twister2 runner' - // Run the wordcount example with the Twister2 runner - t.run """mvn compile exec:java -q \ - -Dmaven.wagon.http.retryHandler.class=default \ - -Dmaven.wagon.http.retryHandler.count=5 \ - -Dmaven.wagon.http.pool=false \ - -Dmaven.wagon.httpconnectionManager.ttlSeconds=120 \ - -Dhttp.keepAlive=false \ - -Dexec.mainClass=org.apache.beam.examples.WordCount \ - -Dexec.args="--inputFile=pom.xml --output=counts \ - --runner=Twister2Runner" -Ptwister2-runner""" - - // Verify text from the pom.xml input file - String result = t.run "grep Foundation counts*" - t.see "Foundation: 1", result - - // Clean up - t.done() diff --git a/release/src/main/scripts/script.config b/release/src/main/scripts/script.config index 6ec776c1970c..a5653f82fca2 100755 --- a/release/src/main/scripts/script.config +++ b/release/src/main/scripts/script.config @@ -44,7 +44,6 @@ RC_VALIDATE_CONFIGS=( java_quickstart_flink_local java_quickstart_spark_local java_quickstart_dataflow - java_quickstart_twister2_local java_mobile_game_direct java_mobile_game_dataflow python_quickstart_mobile_game diff --git a/runners/twister2/build.gradle b/runners/twister2/build.gradle deleted file mode 100644 index 8817fe1f435f..000000000000 --- a/runners/twister2/build.gradle +++ /dev/null @@ -1,112 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * License); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an AS IS BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -import groovy.json.JsonOutput - -plugins { id 'org.apache.beam.module' } - -applyJavaNature( - automaticModuleName: 'org.apache.beam.runners.twister2', -) - -evaluationDependsOn(":sdks:java:core") -configurations { - validatesRunner -} -description = "Apache Beam :: Runners :: Twister2" -def twister2_version = '0.6.0' -dependencies { - implementation library.java.vendored_guava_32_1_2_jre - - implementation(project(path: ":runners:core-java")){ - exclude group: 'com.esotericsoftware.kryo', module: 'kryo' - } - implementation project(path: ":model:pipeline", configuration: "shadow") - implementation project(path: ":sdks:java:core", configuration: "shadow") - implementation library.java.jackson_annotations - implementation library.java.joda_time - implementation library.java.vendored_grpc_1_69_0 - implementation library.java.slf4j_api - implementation "org.mortbay.jetty:jetty-util:6.1.26" - implementation "org.twister2:comms-api-java:$twister2_version" - implementation "org.twister2:config-api-java:$twister2_version" - implementation "org.twister2:dataset-api-java:$twister2_version" - implementation "org.twister2:driver-api-java:$twister2_version" - implementation "org.twister2:exceptions-java:$twister2_version" - implementation "org.twister2:scheduler-api-java:$twister2_version" - implementation "org.twister2:task-api-java:$twister2_version" - implementation "org.twister2:api-java:$twister2_version" - implementation "org.twister2:tset-api-java:$twister2_version" - implementation "org.twister2:tset-java:$twister2_version" - implementation "org.twister2:resource-scheduler-java:$twister2_version" - implementation "org.twister2:local-runner-java:$twister2_version" - testImplementation library.java.junit - validatesRunner (project(path: ":sdks:java:core", configuration: "shadowTest")){ - exclude group: 'com.esotericsoftware.kryo', module: 'kryo' - } - validatesRunner (project(path: ":runners:core-java", configuration: "testRuntimeMigration")){ - exclude group: 'com.esotericsoftware.kryo', module: 'kryo' - } - validatesRunner (project(project.path)){ - exclude group: 'asm', module: 'asm' - } -} - -def validatesRunnerBatch = tasks.register("validatesRunnerBatch", Test) { - group = "Verification" - def pipelineOptions = JsonOutput.toJson([ - "--runner=Twister2TestRunner", - ]) - systemProperty "beamTestPipelineOptions", pipelineOptions - classpath = configurations.validatesRunner - testClassesDirs += files(project(":sdks:java:core").sourceSets.test.output.classesDirs) - testClassesDirs += files(project.sourceSets.test.output.classesDirs) - - forkEvery 1 - useJUnit { - includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner' - excludeCategories 'org.apache.beam.sdk.testing.UsesExternalService' - // Should be run only in a properly configured SDK harness environment - excludeCategories 'org.apache.beam.sdk.testing.UsesSdkHarnessEnvironment' - excludeCategories 'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders' - excludeCategories 'org.apache.beam.sdk.testing.UsesStatefulParDo' - excludeCategories 'org.apache.beam.sdk.testing.UsesTimersInParDo' - excludeCategories 'org.apache.beam.sdk.testing.UsesUnboundedSplittableParDo' - excludeCategories 'org.apache.beam.sdk.testing.UsesAttemptedMetrics' - excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics' - excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream' - excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' - excludeCategories 'org.apache.beam.sdk.testing.UsesMetricsPusher' - excludeCategories 'org.apache.beam.sdk.testing.UsesBundleFinalizer' - // Feature unsupported by this runner - excludeCategories 'org.apache.beam.sdk.testing.UsesPerKeyOrderedDelivery' - excludeCategories 'org.apache.beam.sdk.testing.UsesPerKeyOrderInBundle' - excludeCategories 'org.apache.beam.sdk.testing.UsesTriggeredSideInputs' - } - - maxHeapSize = '6g' -} - -tasks.register("validatesRunner") { - group = "Verification" - description "Validates Twister2 Runner" - dependsOn validatesRunnerBatch -} - -// Generates :runners:twister2:runQuickstartJavaTwister2 -createJavaExamplesArchetypeValidationTask(type: 'Quickstart', runner:'Twister2') diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/BeamBatchTSetEnvironment.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/BeamBatchTSetEnvironment.java deleted file mode 100644 index 370eef6bdef6..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/BeamBatchTSetEnvironment.java +++ /dev/null @@ -1,42 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2; - -import edu.iu.dsc.tws.api.compute.graph.OperationMode; -import edu.iu.dsc.tws.tset.TBaseGraph; -import edu.iu.dsc.tws.tset.env.BatchTSetEnvironment; - -/** - * This is a shell tset environment which is used on as a central driver model to fit what beam - * expects. This will be used to develop the TSet graph corresponding to the Beam Pipeline which - * will later be injected into the correct TSetEnvironment one submitted. - */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class BeamBatchTSetEnvironment extends BatchTSetEnvironment { - - public BeamBatchTSetEnvironment() { - this.settBaseGraph(new TBaseGraph(getOperationMode())); - } - - @Override - public OperationMode getOperationMode() { - return OperationMode.BATCH; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/BeamBatchWorker.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/BeamBatchWorker.java deleted file mode 100644 index 5cb49f1e9fae..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/BeamBatchWorker.java +++ /dev/null @@ -1,166 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2; - -import edu.iu.dsc.tws.api.config.Config; -import edu.iu.dsc.tws.api.tset.TBase; -import edu.iu.dsc.tws.api.tset.sets.TSet; -import edu.iu.dsc.tws.api.tset.sets.batch.BatchTSet; -import edu.iu.dsc.tws.tset.TBaseGraph; -import edu.iu.dsc.tws.tset.env.BatchTSetEnvironment; -import edu.iu.dsc.tws.tset.links.BaseTLink; -import edu.iu.dsc.tws.tset.sets.BaseTSet; -import edu.iu.dsc.tws.tset.sets.BuildableTSet; -import edu.iu.dsc.tws.tset.sets.batch.CachedTSet; -import edu.iu.dsc.tws.tset.sets.batch.ComputeTSet; -import edu.iu.dsc.tws.tset.sets.batch.SinkTSet; -import edu.iu.dsc.tws.tset.worker.BatchTSetIWorker; -import java.io.Serializable; -import java.util.ArrayDeque; -import java.util.Deque; -import java.util.HashMap; -import java.util.HashSet; -import java.util.LinkedHashMap; -import java.util.Map; -import java.util.Set; -import org.apache.beam.runners.twister2.translators.functions.DoFnFunction; -import org.apache.beam.runners.twister2.translators.functions.Twister2SinkFunction; - -/** - * The Twister2 worker that will execute the job logic once the job is submitted from the run - * method. - */ -@SuppressWarnings({ - "rawtypes", // TODO(https://github.com/apache/beam/issues/20447) - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class BeamBatchWorker implements Serializable, BatchTSetIWorker { - - private static final String SIDEINPUTS = "sideInputs"; - private static final String LEAVES = "leaves"; - private static final String GRAPH = "graph"; - private HashMap> sideInputDataSets; - private Set leaves; - - @Override - public void execute(BatchTSetEnvironment env) { - Config config = env.getConfig(); - Map sideInputIds = (LinkedHashMap) config.get(SIDEINPUTS); - Set leaveIds = (Set) config.get(LEAVES); - TBaseGraph graph = (TBaseGraph) config.get(GRAPH); - env.settBaseGraph(graph); - setupTSets(env, sideInputIds, leaveIds); - resetEnv(env, graph); - executePipeline(env); - } - - /** - * resets the proper TSetEnvironment in all the TSets in the graph that was sent over the wire. - * - * @param env the TSetEnvironment to be set - * @param graph the graph that contains all the TSets. - */ - private void resetEnv(BatchTSetEnvironment env, TBaseGraph graph) { - Set nodes = graph.getNodes(); - for (TBase node : nodes) { - if (node instanceof BaseTSet) { - ((BaseTSet) node).setTSetEnv(env); - } else if (node instanceof BaseTLink) { - ((BaseTLink) node).setTSetEnv(env); - } else { - throw new IllegalStateException("node must be either of type BaseTSet or BaseTLink"); - } - } - } - - /** - * Extract the sideInput TSets and the Leaves from the graph. - * - * @param env BatchTSetEnvironment which contains the tSet Graph - * @param sideInputIds the id's of the side Inputs - * @param leaveIds the id's of the leaf - */ - private void setupTSets( - BatchTSetEnvironment env, Map sideInputIds, Set leaveIds) { - sideInputDataSets = new LinkedHashMap<>(); - leaves = new HashSet<>(); - - // reset sources, so that the graph does not have two source objects - // created during deserialization - Set newSources = new HashSet<>(); - for (BuildableTSet source : env.getGraph().getSources()) { - newSources.add((BuildableTSet) env.getGraph().getNodeById(source.getId())); - } - env.getGraph().setSources(newSources); - - for (Map.Entry entry : sideInputIds.entrySet()) { - BatchTSet curr = (BatchTSet) env.getGraph().getNodeById(entry.getValue()); - sideInputDataSets.put(entry.getKey(), curr); - } - for (String leaveId : leaveIds) { - leaves.add((TSet) env.getGraph().getNodeById(leaveId)); - } - } - - public void executePipeline(BatchTSetEnvironment env) { - Map sideInputTSets = new HashMap<>(); - for (Map.Entry> sides : sideInputDataSets.entrySet()) { - BatchTSet sideTSet = sides.getValue(); - addInputs((BaseTSet) sideTSet, sideInputTSets); - CachedTSet tempCache = (CachedTSet) sideTSet.cache(); - sideInputTSets.put(sides.getKey(), tempCache); - } - for (TSet leaf : leaves) { - SinkTSet sinkTSet = (SinkTSet) leaf.direct().sink(new Twister2SinkFunction()); - addInputs(sinkTSet, sideInputTSets); - eval(env, sinkTSet); - } - } - - /** Adds all the side inputs into the sink test so it is available from the DoFn's. */ - private void addInputs(BaseTSet sinkTSet, Map sideInputTSets) { - if (sideInputTSets.isEmpty()) { - return; - } - - TBaseGraph graph = sinkTSet.getTBaseGraph(); - TBase currNode = null; - Deque deque = new ArrayDeque<>(); - deque.add(sinkTSet); - while (!deque.isEmpty()) { - currNode = deque.remove(); - deque.addAll(graph.getPredecessors(currNode)); - if (currNode instanceof ComputeTSet) { - if (((ComputeTSet) currNode).getComputeFunc() instanceof DoFnFunction) { - Set sideInputKeys = - ((DoFnFunction) ((ComputeTSet) currNode).getComputeFunc()).getSideInputKeys(); - for (String sideInputKey : sideInputKeys) { - if (!sideInputTSets.containsKey(sideInputKey)) { - throw new IllegalStateException("Side input not found for key " + sideInputKey); - } - ((ComputeTSet) currNode).addInput(sideInputKey, sideInputTSets.get(sideInputKey)); - } - } - } - } - } - - public void eval(BatchTSetEnvironment env, SinkTSet tSet) { - env.run(tSet); - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2BatchTranslationContext.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2BatchTranslationContext.java deleted file mode 100644 index 628425a52e45..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2BatchTranslationContext.java +++ /dev/null @@ -1,44 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2; - -import edu.iu.dsc.tws.tset.env.BatchTSetEnvironment; -import edu.iu.dsc.tws.tset.sets.batch.BatchTSetImpl; -import edu.iu.dsc.tws.tset.sets.batch.SinkTSet; -import org.apache.beam.sdk.values.PValue; -import org.apache.beam.sdk.values.WindowedValue; - -/** Twister2BatchTranslationContext. */ -public class Twister2BatchTranslationContext extends Twister2TranslationContext { - - public Twister2BatchTranslationContext(Twister2PipelineOptions options) { - super(options); - } - - @Override - public BatchTSetImpl> getInputDataSet(PValue input) { - BatchTSetImpl> baseTSet = - (BatchTSetImpl>) super.getInputDataSet(input); - return baseTSet; - } - - @Override - public void eval(SinkTSet tSet) { - ((BatchTSetEnvironment) getEnvironment()).run(tSet); - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2PipelineExecutionEnvironment.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2PipelineExecutionEnvironment.java deleted file mode 100644 index 271e9317f8a2..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2PipelineExecutionEnvironment.java +++ /dev/null @@ -1,118 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2; - -import edu.iu.dsc.tws.api.tset.sets.TSet; -import edu.iu.dsc.tws.api.tset.sets.batch.BatchTSet; -import edu.iu.dsc.tws.tset.TBaseGraph; -import java.util.Map; -import java.util.Set; -import java.util.logging.Logger; -import org.apache.beam.runners.twister2.translators.Twister2BatchPipelineTranslator; -import org.apache.beam.runners.twister2.translators.Twister2PipelineTranslator; -import org.apache.beam.runners.twister2.translators.Twister2StreamPipelineTranslator; -import org.apache.beam.sdk.Pipeline; -import org.apache.beam.sdk.runners.TransformHierarchy; -import org.apache.beam.sdk.values.PCollection; -import org.apache.beam.sdk.values.PValue; -import org.slf4j.LoggerFactory; - -/** Twister2PipelineExecutionEnvironment. */ -@SuppressWarnings({ - "rawtypes", // TODO(https://github.com/apache/beam/issues/20447) - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class Twister2PipelineExecutionEnvironment { - private static final Logger LOG = - Logger.getLogger(Twister2PipelineExecutionEnvironment.class.getName()); - - private final Twister2PipelineOptions options; - private Twister2TranslationContext twister2TranslationContext; - - public Twister2PipelineExecutionEnvironment(Twister2PipelineOptions options) { - this.options = options; - options.setTSetEnvironment(new BeamBatchTSetEnvironment()); - } - - /** translate the pipeline into Twister2 TSet graph. */ - public void translate(Pipeline pipeline) { - - TranslationModeDetector detector = new TranslationModeDetector(); - pipeline.traverseTopologically(detector); - - if (detector.isStreaming()) { - LOG.info("Found unbounded PCollection. Switching to streaming execution."); - options.setStreaming(true); - throw new UnsupportedOperationException( - "Streaming is not supported currently in the Twister2 Runner"); - } - - Twister2PipelineTranslator translator; - if (options.isStreaming()) { - twister2TranslationContext = new Twister2StreamTranslationContext(options); - translator = new Twister2StreamPipelineTranslator(); - } else { - twister2TranslationContext = new Twister2BatchTranslationContext(options); - translator = - new Twister2BatchPipelineTranslator( - options, (Twister2BatchTranslationContext) twister2TranslationContext); - } - - translator.translate(pipeline); - } - - public Map> getSideInputs() { - return twister2TranslationContext.getSideInputDataSets(); - } - - public Set getLeaves() { - return twister2TranslationContext.getLeaves(); - } - - protected TBaseGraph getTSetGraph() { - return twister2TranslationContext.getEnvironment().getGraph(); - } - - /** Traverses the Pipeline to determine if this is a streaming pipeline. */ - private static class TranslationModeDetector extends Pipeline.PipelineVisitor.Defaults { - private static final org.slf4j.Logger LOG = - LoggerFactory.getLogger(TranslationModeDetector.class); - - private boolean isStreaming; - - TranslationModeDetector() { - this.isStreaming = false; - } - - boolean isStreaming() { - return isStreaming; - } - - @Override - public void visitValue(PValue value, TransformHierarchy.Node producer) { - if (!isStreaming) { - if (value instanceof PCollection - && ((PCollection) value).isBounded() == PCollection.IsBounded.UNBOUNDED) { - LOG.info( - "Found unbounded PCollection {}. Switching to streaming execution.", value.getName()); - isStreaming = true; - } - } - } - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2PipelineOptions.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2PipelineOptions.java deleted file mode 100644 index f69c46291ace..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2PipelineOptions.java +++ /dev/null @@ -1,77 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2; - -import com.fasterxml.jackson.annotation.JsonIgnore; -import edu.iu.dsc.tws.tset.env.TSetEnvironment; -import org.apache.beam.sdk.options.Default; -import org.apache.beam.sdk.options.Description; -import org.apache.beam.sdk.options.FileStagingOptions; -import org.apache.beam.sdk.options.PipelineOptions; -import org.apache.beam.sdk.options.StreamingOptions; - -/** Twister2PipelineOptions. */ -public interface Twister2PipelineOptions - extends PipelineOptions, StreamingOptions, FileStagingOptions { - - @Description("set parallelism for Twister2 processor") - @Default.Integer(1) - int getParallelism(); - - void setParallelism(int parallelism); - - @Description("Twister2 TSetEnvironment") - @JsonIgnore - TSetEnvironment getTSetEnvironment(); - - void setTSetEnvironment(TSetEnvironment environment); - - @Description("Twister2 cluster type, supported types: standalone, nomad, kubernetes, mesos") - @Default.String("standalone") - String getClusterType(); - - void setClusterType(String name); - - @Description("Job file zip") - String getJobFileZip(); - - void setJobFileZip(String pathToZip); - - @Description("Job type, jar or java_zip") - @Default.String("java_zip") - String getJobType(); - - void setJobType(String jobType); - - @Description("Twister2 home directory") - String getTwister2Home(); - - void setTwister2Home(String twister2Home); - - @Description("CPU's per worker") - @Default.Integer(2) - int getWorkerCPUs(); - - void setWorkerCPUs(int workerCPUs); - - @Description("RAM allocated per worker") - @Default.Integer(2048) - int getRamMegaBytes(); - - void setRamMegaBytes(int ramMegaBytes); -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2PipelineResult.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2PipelineResult.java deleted file mode 100644 index 13d744fd3950..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2PipelineResult.java +++ /dev/null @@ -1,89 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2; - -import edu.iu.dsc.tws.api.scheduler.Twister2JobState; -import java.io.IOException; -import org.apache.beam.sdk.PipelineResult; -import org.apache.beam.sdk.metrics.MetricResults; -import org.joda.time.Duration; -import org.mortbay.log.Log; - -/** Represents a Twister2 pipeline execution result. */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class Twister2PipelineResult implements PipelineResult { - - PipelineResult.State state; - - public Twister2PipelineResult(Twister2JobState jobState) { - state = mapToState(jobState); - } - - @Override - public State getState() { - return state; - } - - @Override - public State cancel() throws IOException { - throw new UnsupportedOperationException("Operation not supported"); - } - - @Override - public State waitUntilFinish(Duration duration) { - Log.debug( - "Twister2 runner does not currently support wait with duration " - + "default waitUntilFinish will be executed"); - return waitUntilFinish(); - } - - @Override - public State waitUntilFinish() { - return state; - } - - @Override - public MetricResults metrics() { - throw new UnsupportedOperationException("Operation not supported"); - } - - public void setState(State state) { - this.state = state; - } - - /** - * Map Twister2 job state to Beam result State. - * - * @param jobState twister2 job state - * @return corresponding Beam result state - */ - private State mapToState(Twister2JobState jobState) { - switch (jobState.getJobstate()) { - case RUNNING: - return State.RUNNING; - case COMPLETED: - return State.DONE; - case FAILED: - return State.FAILED; - default: - return State.FAILED; - } - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2Runner.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2Runner.java deleted file mode 100644 index a6eb52a93db0..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2Runner.java +++ /dev/null @@ -1,360 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2; - -import edu.iu.dsc.tws.api.JobConfig; -import edu.iu.dsc.tws.api.Twister2Job; -import edu.iu.dsc.tws.api.config.Config; -import edu.iu.dsc.tws.api.exceptions.Twister2RuntimeException; -import edu.iu.dsc.tws.api.scheduler.Twister2JobState; -import edu.iu.dsc.tws.api.tset.sets.TSet; -import edu.iu.dsc.tws.api.tset.sets.batch.BatchTSet; -import edu.iu.dsc.tws.local.LocalSubmitter; -import edu.iu.dsc.tws.rsched.core.ResourceAllocator; -import edu.iu.dsc.tws.rsched.job.Twister2Submitter; -import java.io.File; -import java.io.FileInputStream; -import java.io.FileNotFoundException; -import java.io.FileOutputStream; -import java.io.IOException; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.HashSet; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.logging.LogManager; -import java.util.logging.Logger; -import java.util.zip.ZipEntry; -import java.util.zip.ZipOutputStream; -import org.apache.beam.sdk.Pipeline; -import org.apache.beam.sdk.PipelineResult; -import org.apache.beam.sdk.PipelineRunner; -import org.apache.beam.sdk.options.ExperimentalOptions; -import org.apache.beam.sdk.options.PipelineOptions; -import org.apache.beam.sdk.options.PipelineOptionsValidator; -import org.apache.beam.sdk.runners.PTransformOverride; -import org.apache.beam.sdk.util.construction.PTransformMatchers; -import org.apache.beam.sdk.util.construction.PTransformTranslation; -import org.apache.beam.sdk.util.construction.SplittableParDo; -import org.apache.beam.sdk.util.construction.SplittableParDoNaiveBounded; -import org.apache.beam.sdk.util.construction.resources.PipelineResources; -import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList; - -/** - * A {@link PipelineRunner} that executes the operations in the pipeline by first translating them - * to a Twister2 Plan and then executing them either locally or on a Twister2 cluster, depending on - * the configuration. - * - * @deprecated The support for twister2 is scheduled for removal in Beam 3.0. - */ -@SuppressWarnings({ - "rawtypes", // TODO(https://github.com/apache/beam/issues/20447) - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -@Deprecated -public class Twister2Runner extends PipelineRunner { - - private static final Logger LOG = Logger.getLogger(Twister2Runner.class.getName()); - private static final String SIDEINPUTS = "sideInputs"; - private static final String LEAVES = "leaves"; - private static final String GRAPH = "graph"; - - /** Provided options. */ - private final Twister2PipelineOptions options; - - protected Twister2Runner(Twister2PipelineOptions options) { - this.options = options; - } - - public static Twister2Runner fromOptions(PipelineOptions options) { - return new Twister2Runner( - PipelineOptionsValidator.validate(Twister2PipelineOptions.class, options)); - } - - @Override - public PipelineResult run(Pipeline pipeline) { - // create a worker and pass in the pipeline and then do the translation - Twister2PipelineExecutionEnvironment env = new Twister2PipelineExecutionEnvironment(options); - LOG.info("Translating pipeline to Twister2 program."); - pipeline.replaceAll(getDefaultOverrides()); - - // TODO(https://github.com/apache/beam/issues/20530): Use SDF read as default when we address - // performance issue. - if (!ExperimentalOptions.hasExperiment(pipeline.getOptions(), "beam_fn_api")) { - SplittableParDo.convertReadBasedSplittableDoFnsToPrimitiveReadsIfNecessary(pipeline); - } - - env.translate(pipeline); - setupSystem(options); - - Map configMap = new HashMap(); - JobConfig jobConfig = new JobConfig(); - if (isLocalMode(options)) { - options.setParallelism(1); - configMap.put(SIDEINPUTS, extractNames(env.getSideInputs())); - configMap.put(LEAVES, extractNames(env.getLeaves())); - configMap.put(GRAPH, env.getTSetGraph()); - configMap.put("twister2.network.buffer.size", 32000); - configMap.put("twister2.network.sendBuffer.count", options.getParallelism()); - LOG.warning("Twister2 Local Mode currently only supports single worker"); - } else { - jobConfig.put(SIDEINPUTS, extractNames(env.getSideInputs())); - jobConfig.put(LEAVES, extractNames(env.getLeaves())); - jobConfig.put(GRAPH, env.getTSetGraph()); - } - - Config config = ResourceAllocator.loadConfig(configMap); - - int workers = options.getParallelism(); - Twister2Job twister2Job = - Twister2Job.newBuilder() - .setJobName(options.getJobName()) - .setWorkerClass(BeamBatchWorker.class) - .addComputeResource(options.getWorkerCPUs(), options.getRamMegaBytes(), workers) - .setConfig(jobConfig) - .build(); - - Twister2JobState jobState; - if (isLocalMode(options)) { - jobState = LocalSubmitter.submitJob(twister2Job, config); - } else { - jobState = Twister2Submitter.submitJob(twister2Job, config); - } - - Twister2PipelineResult result = new Twister2PipelineResult(jobState); - return result; - } - - /** Check if the Runner is set to use Twister local mode or pointing to a deployment. */ - private boolean isLocalMode(Twister2PipelineOptions options) { - if (options.getTwister2Home() == null || "".equals(options.getTwister2Home())) { - return true; - } else { - return false; - } - } - - public PipelineResult runTest(Pipeline pipeline) { - // create a worker and pass in the pipeline and then do the translation - Twister2PipelineExecutionEnvironment env = new Twister2PipelineExecutionEnvironment(options); - LOG.info("Translating pipeline to Twister2 program."); - pipeline.replaceAll(getDefaultOverrides()); - - // TODO(https://github.com/apache/beam/issues/20530): Use SDF read as default when we address - // performance issue. - if (!ExperimentalOptions.hasExperiment(pipeline.getOptions(), "beam_fn_api")) { - SplittableParDo.convertReadBasedSplittableDoFnsToPrimitiveReadsIfNecessary(pipeline); - } - - env.translate(pipeline); - setupSystemTest(options); - Map configMap = new HashMap(); - configMap.put(SIDEINPUTS, extractNames(env.getSideInputs())); - configMap.put(LEAVES, extractNames(env.getLeaves())); - configMap.put(GRAPH, env.getTSetGraph()); - configMap.put("twister2.network.buffer.size", 32000); - configMap.put("twister2.network.sendBuffer.count", options.getParallelism()); - Config config = ResourceAllocator.loadConfig(configMap); - - JobConfig jobConfig = new JobConfig(); - - int workers = options.getParallelism(); - Twister2Job twister2Job = - Twister2Job.newBuilder() - .setJobName(options.getJobName()) - .setWorkerClass(BeamBatchWorker.class) - .addComputeResource(options.getWorkerCPUs(), options.getRamMegaBytes(), workers) - .setConfig(jobConfig) - .build(); - Twister2JobState jobState = LocalSubmitter.submitJob(twister2Job, config); - - Twister2PipelineResult result = new Twister2PipelineResult(jobState); - // TODO: Need to fix the check for "RUNNING" once fix for this is done on Twister2 end. - if (result.state == PipelineResult.State.FAILED) { - throw new RuntimeException("Pipeline execution failed", jobState.getCause()); - } - return result; - } - - private void setupSystem(Twister2PipelineOptions options) { - prepareFilesToStage(options); - zipFilesToStage(options); - System.setProperty("cluster_type", options.getClusterType()); - System.setProperty("job_file", options.getJobFileZip()); - System.setProperty("job_type", options.getJobType()); - if (isLocalMode(options)) { - System.setProperty("twister2_home", System.getProperty("java.io.tmpdir")); - System.setProperty("config_dir", System.getProperty("java.io.tmpdir") + "/conf/"); - } else { - System.setProperty("twister2_home", options.getTwister2Home()); - System.setProperty("config_dir", options.getTwister2Home() + "/conf/"); - - // do a simple config dir validation - File cDir = new File(System.getProperty("config_dir"), options.getClusterType()); - - String[] filesList = - new String[] { - "core.yaml", "network.yaml", "data.yaml", "resource.yaml", "task.yaml", - }; - - for (String file : filesList) { - File toCheck = new File(cDir, file); - if (!toCheck.exists()) { - throw new Twister2RuntimeException( - "Couldn't find " + file + " in config directory specified."); - } - } - - // setup logging - FileInputStream fis = null; - try { - fis = new FileInputStream(new File(cDir, "logger.properties")); - LogManager.getLogManager().readConfiguration(fis); - fis.close(); - } catch (IOException e) { - LOG.warning("Couldn't load logging configuration"); - } finally { - if (fis != null) { - try { - fis.close(); - } catch (IOException e) { - LOG.info(e.getMessage()); - } - } - } - } - } - - private void setupSystemTest(Twister2PipelineOptions options) { - prepareFilesToStage(options); - zipFilesToStage(options); - System.setProperty("cluster_type", options.getClusterType()); - System.setProperty("twister2_home", System.getProperty("java.io.tmpdir")); - System.setProperty("job_file", options.getJobFileZip()); - System.setProperty("job_type", options.getJobType()); - } - - private Set extractNames(Set leaves) { - Set results = new HashSet<>(); - for (TSet leaf : leaves) { - results.add(leaf.getId()); - } - return results; - } - - private Map extractNames(Map> sideInputs) { - Map results = new LinkedHashMap<>(); - for (Map.Entry> entry : sideInputs.entrySet()) { - results.put(entry.getKey(), entry.getValue().getId()); - } - return results; - } - - /** - * Classpath contains non jar files (eg. directories with .class files or empty directories) will - * cause exception in running log. - */ - private void prepareFilesToStage(Twister2PipelineOptions options) { - PipelineResources.prepareFilesForStaging(options); - } - - /** - * creates a single zip file from all the jar files that are listed as files to stage in options. - * - * @param options - */ - private void zipFilesToStage(Twister2PipelineOptions options) { - File zipFile = null; - Set jarSet = new HashSet<>(); - // TODO figure out if we can remove all the dependencies that come with - // the twister2 jars as well since they will be anyway provided from the - // system we do not need to pack them - List filesToStage = options.getFilesToStage(); - List trimmed = new ArrayList<>(); - // remove twister2 jars from the list - for (String file : filesToStage) { - if (!file.contains("/org/twister2")) { - trimmed.add(file); - } - } - - FileInputStream fis = null; - try { - zipFile = File.createTempFile("twister2-", ".zip"); - FileOutputStream fos = new FileOutputStream(zipFile); - ZipOutputStream zipOut = new ZipOutputStream(fos); - zipOut.putNextEntry(new ZipEntry("lib/")); - - for (String srcFile : trimmed) { - File fileToZip = new File(srcFile); - if (!jarSet.contains(fileToZip.getName())) { - jarSet.add(fileToZip.getName()); - } else { - continue; - } - - fis = new FileInputStream(fileToZip); - ZipEntry zipEntry = new ZipEntry("lib/" + fileToZip.getName()); - zipOut.putNextEntry(zipEntry); - - byte[] bytes = new byte[1024]; - int length; - while ((length = fis.read(bytes)) >= 0) { - zipOut.write(bytes, 0, length); - } - fis.close(); - } - zipOut.close(); - fos.close(); - zipFile.deleteOnExit(); - } catch (FileNotFoundException e) { - LOG.info(e.getMessage()); - } catch (IOException e) { - LOG.info(e.getMessage()); - } finally { - if (fis != null) { - try { - fis.close(); - } catch (IOException e) { - LOG.info(e.getMessage()); - } - } - } - if (zipFile != null) { - options.setJobFileZip(zipFile.getPath()); - } - } - - private static List getDefaultOverrides() { - List overrides = - ImmutableList.builder() - .add( - PTransformOverride.of( - PTransformMatchers.splittableParDo(), new SplittableParDo.OverrideFactory())) - .add( - PTransformOverride.of( - PTransformMatchers.urnEqualTo( - PTransformTranslation.SPLITTABLE_PROCESS_KEYED_URN), - new SplittableParDoNaiveBounded.OverrideFactory())) - .build(); - - return overrides; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2RunnerRegistrar.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2RunnerRegistrar.java deleted file mode 100644 index e0bb8ea94efe..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2RunnerRegistrar.java +++ /dev/null @@ -1,53 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2; - -import com.google.auto.service.AutoService; -import org.apache.beam.sdk.PipelineRunner; -import org.apache.beam.sdk.options.PipelineOptions; -import org.apache.beam.sdk.options.PipelineOptionsRegistrar; -import org.apache.beam.sdk.runners.PipelineRunnerRegistrar; -import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList; - -/** - * AutoService registrar - will register Twister2Runner and Twister2Options as possible pipeline - * runner services. - * - *

It ends up in META-INF/services and gets picked up by Beam. - */ -public class Twister2RunnerRegistrar { - private Twister2RunnerRegistrar() {} - - /** Pipeline runner registrar. */ - @AutoService(PipelineRunnerRegistrar.class) - public static class Runner implements PipelineRunnerRegistrar { - @Override - public Iterable>> getPipelineRunners() { - return ImmutableList.of(Twister2Runner.class, Twister2TestRunner.class); - } - } - - /** Pipeline options registrar. */ - @AutoService(PipelineOptionsRegistrar.class) - public static class Options implements PipelineOptionsRegistrar { - @Override - public Iterable> getPipelineOptions() { - return ImmutableList.of(Twister2PipelineOptions.class); - } - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2StreamTranslationContext.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2StreamTranslationContext.java deleted file mode 100644 index 53a25101c3d5..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2StreamTranslationContext.java +++ /dev/null @@ -1,33 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2; - -import edu.iu.dsc.tws.tset.sets.batch.SinkTSet; - -/** Twister2StreamingTranslationContext. */ -public class Twister2StreamTranslationContext extends Twister2TranslationContext { - - public Twister2StreamTranslationContext(Twister2PipelineOptions options) { - super(options); - } - - @Override - public void eval(SinkTSet tSet) { - throw new UnsupportedOperationException("Not implemented yet"); - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2TestRunner.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2TestRunner.java deleted file mode 100644 index 8cefc738731d..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2TestRunner.java +++ /dev/null @@ -1,63 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2; - -import org.apache.beam.sdk.Pipeline; -import org.apache.beam.sdk.PipelineResult; -import org.apache.beam.sdk.PipelineRunner; -import org.apache.beam.sdk.options.PipelineOptions; -import org.apache.beam.sdk.options.PipelineOptionsValidator; - -/** - * A {@link PipelineRunner} that executes the operations in the pipeline by first translating them - * to a Twister2 Plan and then executing them either locally or on a Twister2 cluster, depending on - * the configuration. This is used for testing the Twister2 runner - */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class Twister2TestRunner extends PipelineRunner { - - private Twister2Runner delegate; - - public Twister2TestRunner(Twister2PipelineOptions options) { - options.setRunner(Twister2TestRunner.class); - options.setParallelism(1); - this.delegate = Twister2Runner.fromOptions(options); - } - - public static Twister2TestRunner fromOptions(PipelineOptions options) { - Twister2PipelineOptions pipelineOptions = - PipelineOptionsValidator.validate(Twister2PipelineOptions.class, options); - return new Twister2TestRunner(pipelineOptions); - } - - @Override - public PipelineResult run(Pipeline pipeline) { - try { - return delegate.runTest(pipeline); - } catch (IllegalArgumentException ex) { - throw ex; - } catch (Throwable t) { - RuntimeException exception = - new RuntimeException(t.getCause().getMessage(), t.getCause().getCause()); - exception.setStackTrace(t.getCause().getStackTrace()); - throw exception; - } - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2TranslationContext.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2TranslationContext.java deleted file mode 100644 index 76d7a8675236..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/Twister2TranslationContext.java +++ /dev/null @@ -1,126 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2; - -import edu.iu.dsc.tws.api.tset.sets.TSet; -import edu.iu.dsc.tws.api.tset.sets.batch.BatchTSet; -import edu.iu.dsc.tws.tset.env.TSetEnvironment; -import edu.iu.dsc.tws.tset.sets.batch.SinkTSet; -import java.util.LinkedHashMap; -import java.util.LinkedHashSet; -import java.util.Map; -import java.util.Set; -import java.util.stream.Collectors; -import org.apache.beam.sdk.coders.Coder; -import org.apache.beam.sdk.runners.AppliedPTransform; -import org.apache.beam.sdk.transforms.PTransform; -import org.apache.beam.sdk.util.construction.TransformInputs; -import org.apache.beam.sdk.values.PCollection; -import org.apache.beam.sdk.values.PValue; -import org.apache.beam.sdk.values.TupleTag; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Iterables; - -/** Twister2TranslationContext. */ -@SuppressWarnings({ - "rawtypes", // TODO(https://github.com/apache/beam/issues/20447) - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public abstract class Twister2TranslationContext { - private final Twister2PipelineOptions options; - protected final Map> dataSets = new LinkedHashMap<>(); - private final Set leaves = new LinkedHashSet<>(); - private final Map> sideInputDataSets; - private AppliedPTransform currentTransform; - private final TSetEnvironment environment; - - public Twister2TranslationContext(Twister2PipelineOptions options) { - this.options = options; - this.environment = options.getTSetEnvironment(); - this.sideInputDataSets = new LinkedHashMap<>(); - } - - @SuppressWarnings("unchecked") - public T getOutput(PTransform transform) { - return (T) Iterables.getOnlyElement(currentTransform.getOutputs().values()); - } - - public Twister2PipelineOptions getOptions() { - return options; - } - - public void setOutputDataSet(PCollection output, TSet> tset) { - if (!dataSets.containsKey(output)) { - dataSets.put(output, tset); - leaves.add(tset); - } - } - - public TSet> getInputDataSet(PValue input) { - TSet> tSet = (TSet>) dataSets.get(input); - leaves.remove(tSet); - return tSet; - } - - public Map, PCollection> getInputs() { - return currentTransform.getInputs(); - } - - public T getInput(PTransform transform) { - return (T) Iterables.getOnlyElement(TransformInputs.nonAdditionalInputs(currentTransform)); - } - - public void setCurrentTransform(AppliedPTransform transform) { - this.currentTransform = transform; - } - - public AppliedPTransform getCurrentTransform() { - return currentTransform; - } - - public Map, PCollection> getOutputs() { - return getCurrentTransform().getOutputs(); - } - - public Map, Coder> getOutputCoders() { - return currentTransform.getOutputs().entrySet().stream() - .filter(e -> e.getValue() != null) - .collect(Collectors.toMap(Map.Entry::getKey, e -> ((PCollection) e.getValue()).getCoder())); - } - - public TSetEnvironment getEnvironment() { - return environment; - } - - public abstract void eval(SinkTSet tSet); - - public void setSideInputDataSet( - String value, BatchTSet> set) { - if (!sideInputDataSets.containsKey(value)) { - sideInputDataSets.put(value, set); - } - } - - public Set getLeaves() { - return leaves; - } - - public Map> getSideInputDataSets() { - return sideInputDataSets; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/package-info.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/package-info.java deleted file mode 100644 index 077521a43c33..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/package-info.java +++ /dev/null @@ -1,20 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/** Internal implementation of the Beam runner for Twister2. */ -package org.apache.beam.runners.twister2; diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translation/wrappers/Twister2BoundedSource.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translation/wrappers/Twister2BoundedSource.java deleted file mode 100644 index 1bd866d07130..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translation/wrappers/Twister2BoundedSource.java +++ /dev/null @@ -1,275 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translation.wrappers; - -import edu.iu.dsc.tws.api.tset.TSetContext; -import edu.iu.dsc.tws.api.tset.fn.BaseSourceFunc; -import java.io.IOException; -import java.io.ObjectStreamException; -import java.util.ArrayList; -import java.util.Iterator; -import java.util.List; -import java.util.NoSuchElementException; -import java.util.logging.Logger; -import org.apache.beam.model.pipeline.v1.RunnerApi; -import org.apache.beam.runners.core.construction.SerializablePipelineOptions; -import org.apache.beam.runners.twister2.Twister2TranslationContext; -import org.apache.beam.sdk.io.BoundedSource; -import org.apache.beam.sdk.io.Source; -import org.apache.beam.sdk.options.PipelineOptions; -import org.apache.beam.sdk.options.PortablePipelineOptions; -import org.apache.beam.sdk.util.SerializableUtils; -import org.apache.beam.sdk.util.construction.Environments; -import org.apache.beam.sdk.util.construction.ReadTranslation; -import org.apache.beam.sdk.util.construction.SdkComponents; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowedValues; - -/** Twister2 wrapper for Bounded Source. */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class Twister2BoundedSource extends BaseSourceFunc> { - private static final Logger LOG = Logger.getLogger(Twister2BoundedSource.class.getName()); - - private transient BoundedSource source; - private int numPartitions; - private long splitSize = 100; - private List> partitionedSources; - private Source localPartition; - private transient PipelineOptions options; - private String serializedOptions; - private transient Iterator> readerIterator; - private static final long DEFAULT_BUNDLE_SIZE = 64L * 1024L * 1024L; - - private boolean isInitialized = false; - private byte[] sourceBytes; - - private Twister2BoundedSource() { - isInitialized = false; - } - - public Twister2BoundedSource( - BoundedSource boundedSource, Twister2TranslationContext context, PipelineOptions options) { - source = boundedSource; - this.options = options; - this.serializedOptions = new SerializablePipelineOptions(options).toString(); - SdkComponents components = SdkComponents.create(); - components.registerEnvironment( - Environments.createOrGetDefaultEnvironment(options.as(PortablePipelineOptions.class))); - RunnerApi.FunctionSpec sourceProto = ReadTranslation.toProto(source); - sourceBytes = sourceProto.getPayload().toByteArray(); - } - - @Override - public void prepare(TSetContext context) { - initTransient(); - numPartitions = context.getParallelism(); - - try { - splitSize = source.getEstimatedSizeBytes(options) / numPartitions; - } catch (Exception e) { - LOG.warning( - String.format( - "Failed to get estimated bundle size for source %s, using default bundle " - + "size of %d bytes.", - source.toString(), DEFAULT_BUNDLE_SIZE)); - } - int index = context.getIndex(); - List> partitionList = new ArrayList<>(); - try { - partitionedSources = source.split(splitSize, options); - // todo make sure the partitions are balanced. - if (partitionedSources.size() == 0) { - partitionList.add(source); - } else if (numPartitions == partitionedSources.size()) { - localPartition = partitionedSources.get(index); - partitionList.add(localPartition); - } else { - // If we have more partitions than parallelism we need to allocate - // more than one partition to each process and needs to be balanced - int q = (int) Math.floor(((double) partitionedSources.size()) / numPartitions); - int r = partitionedSources.size() % numPartitions; - int assinedCount = q + ((index < r) ? 1 : 0); - int startIndex = q * index + ((index < r) ? index : r); - for (int i = startIndex; i < startIndex + assinedCount; i++) { - partitionList.add(partitionedSources.get(i)); - } - } - - final List> readers = new ArrayList<>(); - for (Source tSource : partitionList) { - BoundedSource.BoundedReader reader = createReader(tSource); - readers.add(reader); - } - readerIterator = new ReaderToIteratorAdapter(readers); - } catch (Exception e) { - throw new RuntimeException( - "Failed to create partitions for source " + source.getClass().getSimpleName(), e); - } - } - - @Override - public boolean hasNext() { - return readerIterator.hasNext(); - } - - @Override - public WindowedValue next() { - return readerIterator.next(); - } - - private BoundedSource.BoundedReader createReader(Source partition) { - try { - return ((BoundedSource) partition).createReader(options); - } catch (IOException e) { - throw new RuntimeException("Failed to create reader from a BoundedSource.", e); - } - } - - /** - * Exposes an Iterator<{@link WindowedValue}> interface on top of a {@link - * Source.Reader}. - * - *

hasNext is idempotent and returns true iff further items are - * available for reading using the underlying reader. Consequently, when the reader is closed, or - * when the reader has no further elements available (i.e, {@link Source.Reader#advance()} - * returned false), hasNext returns false. - * - *

Since this is a read-only iterator, an attempt to call remove will throw an - * UnsupportedOperationException. - */ - static class ReaderToIteratorAdapter implements Iterator> { - - private static final boolean FAILED_TO_OBTAIN_NEXT = false; - private static final boolean SUCCESSFULLY_OBTAINED_NEXT = true; - - private final List> readers; - // current reader from the list - Source.Reader reader; - private int readerIndex = 0; - - private boolean started = false; - private boolean closed = false; - private WindowedValue next = null; - - public ReaderToIteratorAdapter(List> readers) { - this.readers = readers; - this.readerIndex = 0; - this.reader = readers.get(readerIndex); - } - - private boolean tryProduceNext() { - try { - if (seekNext()) { - next = - WindowedValues.timestampedValueInGlobalWindow( - reader.getCurrent(), reader.getCurrentTimestamp()); - return SUCCESSFULLY_OBTAINED_NEXT; - } else { - return FAILED_TO_OBTAIN_NEXT; - } - - } catch (final Exception e) { - throw new RuntimeException("Failed to read data.", e); - } - } - - private void close() { - try { - reader.close(); - readerIndex++; - if (readerIndex == readers.size()) { - closed = true; - } else { - reader = readers.get(readerIndex); - started = false; - } - } catch (final IOException e) { - throw new RuntimeException(e); - } - } - - private boolean seekNext() throws IOException { - if (!started) { - started = true; - return reader.start() || advance(true); - } else { - return !closed && advance(false); - } - } - - private boolean advance(boolean calledAfterStart) throws IOException { - if (!calledAfterStart && reader.advance()) { - return true; - } else { - close(); - return seekNext(); - } - } - - private WindowedValue consumeCurrent() { - if (next == null) { - throw new NoSuchElementException(); - } else { - final WindowedValue current = next; - next = null; - return current; - } - } - - private WindowedValue consumeNext() { - if (next == null) { - tryProduceNext(); - } - return consumeCurrent(); - } - - @Override - public boolean hasNext() { - return next != null || tryProduceNext(); - } - - @Override - public WindowedValue next() { - return consumeNext(); - } - - @Override - public void remove() { - throw new UnsupportedOperationException(); - } - } - - /** - * Method used to initialize the transient variables that were sent over as byte arrays or proto - * buffers. - */ - private void initTransient() { - if (isInitialized) { - return; - } - options = new SerializablePipelineOptions(serializedOptions).get(); - source = (BoundedSource) SerializableUtils.deserializeFromByteArray(sourceBytes, "WindowFn"); - this.isInitialized = true; - } - - protected Object readResolve() throws ObjectStreamException { - return this; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translation/wrappers/Twister2EmptySource.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translation/wrappers/Twister2EmptySource.java deleted file mode 100644 index 8e875e132a1b..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translation/wrappers/Twister2EmptySource.java +++ /dev/null @@ -1,41 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translation.wrappers; - -import edu.iu.dsc.tws.api.tset.fn.BaseSourceFunc; -import org.apache.beam.sdk.values.WindowedValue; - -/** - * Empty Source wrapper. - * - * @param type of the source - */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class Twister2EmptySource extends BaseSourceFunc> { - @Override - public boolean hasNext() { - return false; - } - - @Override - public WindowedValue next() { - return null; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translation/wrappers/package-info.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translation/wrappers/package-info.java deleted file mode 100644 index 209605c0195f..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translation/wrappers/package-info.java +++ /dev/null @@ -1,20 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/** Internal implementation of the Beam runner for Twister2. */ -package org.apache.beam.runners.twister2.translation.wrappers; diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/BatchTransformTranslator.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/BatchTransformTranslator.java deleted file mode 100644 index 7197400630e2..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/BatchTransformTranslator.java +++ /dev/null @@ -1,29 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators; - -import org.apache.beam.runners.twister2.Twister2BatchTranslationContext; -import org.apache.beam.sdk.transforms.PTransform; - -/** Batch TransformTranslator interface. */ -@SuppressWarnings({ - "rawtypes" // TODO(https://github.com/apache/beam/issues/20447) -}) -public interface BatchTransformTranslator { - void translateNode(TransformT transform, Twister2BatchTranslationContext context); -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/StreamTransformTranslator.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/StreamTransformTranslator.java deleted file mode 100644 index cc666856fb28..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/StreamTransformTranslator.java +++ /dev/null @@ -1,29 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators; - -import org.apache.beam.runners.twister2.Twister2StreamTranslationContext; -import org.apache.beam.sdk.transforms.PTransform; - -/** Stream TransformTranslator interface. */ -@SuppressWarnings({ - "rawtypes" // TODO(https://github.com/apache/beam/issues/20447) -}) -public interface StreamTransformTranslator { - void translateNode(TransformT transform, Twister2StreamTranslationContext context); -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/Twister2BatchPipelineTranslator.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/Twister2BatchPipelineTranslator.java deleted file mode 100644 index ceb3b9bfcf72..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/Twister2BatchPipelineTranslator.java +++ /dev/null @@ -1,94 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators; - -import java.util.HashMap; -import java.util.Map; -import java.util.logging.Logger; -import org.apache.beam.runners.twister2.Twister2BatchTranslationContext; -import org.apache.beam.runners.twister2.Twister2PipelineOptions; -import org.apache.beam.runners.twister2.translators.batch.AssignWindowTranslatorBatch; -import org.apache.beam.runners.twister2.translators.batch.FlattenTranslatorBatch; -import org.apache.beam.runners.twister2.translators.batch.GroupByKeyTranslatorBatch; -import org.apache.beam.runners.twister2.translators.batch.ImpulseTranslatorBatch; -import org.apache.beam.runners.twister2.translators.batch.PCollectionViewTranslatorBatch; -import org.apache.beam.runners.twister2.translators.batch.ParDoMultiOutputTranslatorBatch; -import org.apache.beam.runners.twister2.translators.batch.ReadSourceTranslatorBatch; -import org.apache.beam.sdk.runners.TransformHierarchy; -import org.apache.beam.sdk.transforms.PTransform; -import org.apache.beam.sdk.util.construction.PTransformTranslation; -import org.checkerframework.checker.nullness.qual.Nullable; - -/** Twister pipeline translator for batch pipelines. */ -@SuppressWarnings({ - "rawtypes", // TODO(https://github.com/apache/beam/issues/20447) - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class Twister2BatchPipelineTranslator extends Twister2PipelineTranslator { - - private static final Logger LOG = - Logger.getLogger(Twister2BatchPipelineTranslator.class.getName()); - - /** - * A map from {@link PTransform} subclass to the corresponding {@link BatchTransformTranslator} to - * use to translate that transform. - */ - private static final Map TRANSFORM_TRANSLATORS = - new HashMap<>(); - - private final Twister2BatchTranslationContext translationContext; - - static { - TRANSFORM_TRANSLATORS.put( - PTransformTranslation.IMPULSE_TRANSFORM_URN, new ImpulseTranslatorBatch()); - TRANSFORM_TRANSLATORS.put( - PTransformTranslation.READ_TRANSFORM_URN, new ReadSourceTranslatorBatch()); - TRANSFORM_TRANSLATORS.put( - PTransformTranslation.PAR_DO_TRANSFORM_URN, new ParDoMultiOutputTranslatorBatch()); - TRANSFORM_TRANSLATORS.put( - PTransformTranslation.GROUP_BY_KEY_TRANSFORM_URN, new GroupByKeyTranslatorBatch()); - TRANSFORM_TRANSLATORS.put( - PTransformTranslation.FLATTEN_TRANSFORM_URN, new FlattenTranslatorBatch()); - TRANSFORM_TRANSLATORS.put( - PTransformTranslation.CREATE_VIEW_TRANSFORM_URN, new PCollectionViewTranslatorBatch()); - TRANSFORM_TRANSLATORS.put( - PTransformTranslation.ASSIGN_WINDOWS_TRANSFORM_URN, new AssignWindowTranslatorBatch()); - } - - public Twister2BatchPipelineTranslator( - Twister2PipelineOptions options, Twister2BatchTranslationContext twister2TranslationContext) { - this.translationContext = twister2TranslationContext; - } - - @Override - public void visitPrimitiveTransform(TransformHierarchy.Node node) { - LOG.fine(String.format("visiting transform %s", node.getTransform())); - PTransform transform = node.getTransform(); - BatchTransformTranslator translator = getTransformTranslator(transform); - if (null == translator) { - throw new IllegalStateException("no translator registered for " + transform); - } - translationContext.setCurrentTransform(node.toAppliedPTransform(getPipeline())); - translator.translateNode(transform, translationContext); - } - - private BatchTransformTranslator getTransformTranslator(PTransform transform) { - @Nullable String urn = PTransformTranslation.urnForTransformOrNull(transform); - return urn == null ? null : TRANSFORM_TRANSLATORS.get(urn); - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/Twister2PipelineTranslator.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/Twister2PipelineTranslator.java deleted file mode 100644 index 40aad1327449..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/Twister2PipelineTranslator.java +++ /dev/null @@ -1,33 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators; - -import org.apache.beam.sdk.Pipeline; - -/** Twister2PipelineTranslator, both batch and streaming translators need to extend from this. */ -public abstract class Twister2PipelineTranslator extends Pipeline.PipelineVisitor.Defaults { - - /** - * Translates the pipeline by passing this class as a visitor. - * - * @param pipeline The pipeline to be translated - */ - public void translate(Pipeline pipeline) { - pipeline.traverseTopologically(this); - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/Twister2StreamPipelineTranslator.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/Twister2StreamPipelineTranslator.java deleted file mode 100644 index 3db459bcc228..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/Twister2StreamPipelineTranslator.java +++ /dev/null @@ -1,24 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators; - -/** Twister pipeline translator for stream pipelines. */ -public class Twister2StreamPipelineTranslator extends Twister2PipelineTranslator { - - public Twister2StreamPipelineTranslator() {} -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/AssignWindowTranslatorBatch.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/AssignWindowTranslatorBatch.java deleted file mode 100644 index 39cba2ed2ddb..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/AssignWindowTranslatorBatch.java +++ /dev/null @@ -1,50 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.batch; - -import edu.iu.dsc.tws.tset.sets.batch.BatchTSetImpl; -import edu.iu.dsc.tws.tset.sets.batch.ComputeTSet; -import java.util.Iterator; -import org.apache.beam.runners.twister2.Twister2BatchTranslationContext; -import org.apache.beam.runners.twister2.translators.BatchTransformTranslator; -import org.apache.beam.runners.twister2.translators.functions.AssignWindowsFunction; -import org.apache.beam.sdk.transforms.windowing.BoundedWindow; -import org.apache.beam.sdk.transforms.windowing.Window; -import org.apache.beam.sdk.transforms.windowing.WindowFn; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowingStrategy; - -/** Assign Window translator. */ -@SuppressWarnings({ - "rawtypes" // TODO(https://github.com/apache/beam/issues/20447) -}) -public class AssignWindowTranslatorBatch implements BatchTransformTranslator> { - @Override - public void translateNode(Window.Assign transform, Twister2BatchTranslationContext context) { - BatchTSetImpl> inputTTSet = - context.getInputDataSet(context.getInput(transform)); - - final WindowingStrategy windowingStrategy = - (WindowingStrategy) context.getOutput(transform).getWindowingStrategy(); - - WindowFn windowFn = windowingStrategy.getWindowFn(); - ComputeTSet, Iterator>> outputTSet = - inputTTSet.direct().compute(new AssignWindowsFunction(windowFn, context.getOptions())); - context.setOutputDataSet(context.getOutput(transform), outputTSet); - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/FlattenTranslatorBatch.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/FlattenTranslatorBatch.java deleted file mode 100644 index 4cf5d347220b..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/FlattenTranslatorBatch.java +++ /dev/null @@ -1,70 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.batch; - -import edu.iu.dsc.tws.api.tset.sets.TSet; -import edu.iu.dsc.tws.tset.env.BatchTSetEnvironment; -import edu.iu.dsc.tws.tset.env.TSetEnvironment; -import edu.iu.dsc.tws.tset.sets.batch.BatchTSetImpl; -import java.util.ArrayList; -import java.util.Collection; -import java.util.List; -import org.apache.beam.runners.twister2.Twister2BatchTranslationContext; -import org.apache.beam.runners.twister2.translation.wrappers.Twister2EmptySource; -import org.apache.beam.runners.twister2.translators.BatchTransformTranslator; -import org.apache.beam.sdk.transforms.Flatten; -import org.apache.beam.sdk.values.PCollection; -import org.apache.beam.sdk.values.PValue; -import org.apache.beam.sdk.values.WindowedValue; - -/** Flatten translator. */ -@SuppressWarnings({ - "rawtypes" // TODO(https://github.com/apache/beam/issues/20447) -}) -public class FlattenTranslatorBatch - implements BatchTransformTranslator> { - @Override - public void translateNode( - Flatten.PCollections transform, Twister2BatchTranslationContext context) { - Collection> pcs = context.getInputs().values(); - List>> tSets = new ArrayList<>(); - BatchTSetImpl> unionTSet; - - if (pcs.isEmpty()) { - final TSetEnvironment tsetEnv = context.getEnvironment(); - unionTSet = - ((BatchTSetEnvironment) tsetEnv) - .createSource(new Twister2EmptySource(), context.getOptions().getParallelism()); - } else { - for (PValue pc : pcs) { - BatchTSetImpl> curr = context.getInputDataSet(pc); - tSets.add(curr); - } - - BatchTSetImpl> first = tSets.remove(0); - Collection>> others = new ArrayList<>(); - others.addAll(tSets); - if (tSets.size() > 0) { - unionTSet = first.union(others); - } else { - unionTSet = first; - } - } - context.setOutputDataSet(context.getOutput(transform), unionTSet); - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/GroupByKeyTranslatorBatch.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/GroupByKeyTranslatorBatch.java deleted file mode 100644 index 78c90b966ac7..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/GroupByKeyTranslatorBatch.java +++ /dev/null @@ -1,80 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.batch; - -import edu.iu.dsc.tws.api.comms.structs.Tuple; -import edu.iu.dsc.tws.tset.sets.batch.BatchTSetImpl; -import edu.iu.dsc.tws.tset.sets.batch.ComputeTSet; -import edu.iu.dsc.tws.tset.sets.batch.KeyedTSet; -import java.util.Iterator; -import org.apache.beam.runners.twister2.Twister2BatchTranslationContext; -import org.apache.beam.runners.twister2.translators.BatchTransformTranslator; -import org.apache.beam.runners.twister2.translators.functions.ByteToWindowFunction; -import org.apache.beam.runners.twister2.translators.functions.GroupByWindowFunction; -import org.apache.beam.runners.twister2.translators.functions.MapToTupleFunction; -import org.apache.beam.runners.twister2.translators.functions.internal.SystemReduceFnBuffering; -import org.apache.beam.sdk.coders.Coder; -import org.apache.beam.sdk.coders.KvCoder; -import org.apache.beam.sdk.transforms.GroupByKey; -import org.apache.beam.sdk.transforms.windowing.BoundedWindow; -import org.apache.beam.sdk.transforms.windowing.WindowFn; -import org.apache.beam.sdk.values.KV; -import org.apache.beam.sdk.values.PCollection; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowedValues; -import org.apache.beam.sdk.values.WindowingStrategy; - -/** GroupByKey translator. */ -@SuppressWarnings({ - "rawtypes" // TODO(https://github.com/apache/beam/issues/20447) -}) -public class GroupByKeyTranslatorBatch implements BatchTransformTranslator> { - - @Override - public void translateNode(GroupByKey transform, Twister2BatchTranslationContext context) { - PCollection> input = context.getInput(transform); - BatchTSetImpl>> inputTTSet = context.getInputDataSet(input); - final KvCoder coder = (KvCoder) input.getCoder(); - Coder inputKeyCoder = coder.getKeyCoder(); - WindowingStrategy windowingStrategy = input.getWindowingStrategy(); - WindowFn, BoundedWindow> windowFn = - (WindowFn, BoundedWindow>) windowingStrategy.getWindowFn(); - final WindowedValues.WindowedValueCoder wvCoder = - WindowedValues.FullWindowedValueCoder.of(coder.getValueCoder(), windowFn.windowCoder()); - KeyedTSet keyedTSet = - inputTTSet.mapToTuple(new MapToTupleFunction(inputKeyCoder, wvCoder)); - - // todo add support for a partition function to be specified, this would use - // todo keyedPartition function instead of KeyedGather - ComputeTSet>>, Iterator>>> - groupedbyKeyTset = - keyedTSet.keyedGather().map(new ByteToWindowFunction(inputKeyCoder, wvCoder)); - - // --- now group also by window. - SystemReduceFnBuffering reduceFnBuffering = new SystemReduceFnBuffering(coder.getValueCoder()); - ComputeTSet>>, Iterable>>>> - outputTset = - groupedbyKeyTset - .direct() - .>>>flatmap( - new GroupByWindowFunction( - windowingStrategy, reduceFnBuffering, context.getOptions())); - PCollection output = context.getOutput(transform); - context.setOutputDataSet(output, outputTset); - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/ImpulseTranslatorBatch.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/ImpulseTranslatorBatch.java deleted file mode 100644 index afc325e06fc4..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/ImpulseTranslatorBatch.java +++ /dev/null @@ -1,41 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.batch; - -import edu.iu.dsc.tws.tset.env.BatchTSetEnvironment; -import edu.iu.dsc.tws.tset.env.TSetEnvironment; -import edu.iu.dsc.tws.tset.sets.batch.SourceTSet; -import org.apache.beam.runners.twister2.Twister2BatchTranslationContext; -import org.apache.beam.runners.twister2.translators.BatchTransformTranslator; -import org.apache.beam.runners.twister2.translators.functions.ImpulseSource; -import org.apache.beam.sdk.transforms.Impulse; -import org.apache.beam.sdk.values.PCollection; -import org.apache.beam.sdk.values.WindowedValue; - -/** Impulse translator. */ -public class ImpulseTranslatorBatch implements BatchTransformTranslator { - - @Override - public void translateNode(Impulse transform, Twister2BatchTranslationContext context) { - final TSetEnvironment tsetEnv = context.getEnvironment(); - SourceTSet> sourceTSet = - ((BatchTSetEnvironment) tsetEnv).createSource(new ImpulseSource(), 1); - PCollection output = context.getOutput(transform); - context.setOutputDataSet(output, sourceTSet); - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/PCollectionViewTranslatorBatch.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/PCollectionViewTranslatorBatch.java deleted file mode 100644 index d6c69c8a6324..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/PCollectionViewTranslatorBatch.java +++ /dev/null @@ -1,111 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.batch; - -import edu.iu.dsc.tws.api.tset.sets.batch.BatchTSet; -import java.io.IOException; -import org.apache.beam.runners.twister2.Twister2BatchTranslationContext; -import org.apache.beam.runners.twister2.translators.BatchTransformTranslator; -import org.apache.beam.runners.twister2.translators.functions.ByteToElemFunction; -import org.apache.beam.runners.twister2.translators.functions.ByteToWindowFunctionPrimitive; -import org.apache.beam.runners.twister2.translators.functions.ElemToBytesFunction; -import org.apache.beam.runners.twister2.translators.functions.MapToTupleFunction; -import org.apache.beam.sdk.coders.Coder; -import org.apache.beam.sdk.coders.KvCoder; -import org.apache.beam.sdk.runners.AppliedPTransform; -import org.apache.beam.sdk.transforms.Materializations; -import org.apache.beam.sdk.transforms.PTransform; -import org.apache.beam.sdk.transforms.View; -import org.apache.beam.sdk.transforms.windowing.WindowFn; -import org.apache.beam.sdk.util.construction.CreatePCollectionViewTranslation; -import org.apache.beam.sdk.values.PCollection; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowedValues; -import org.apache.beam.sdk.values.WindowingStrategy; - -/** PCollectionView translator. */ -@SuppressWarnings({ - "rawtypes" // TODO(https://github.com/apache/beam/issues/20447) -}) -public class PCollectionViewTranslatorBatch - implements BatchTransformTranslator> { - @Override - public void translateNode( - View.CreatePCollectionView transform, Twister2BatchTranslationContext context) { - BatchTSet> inputDataSet = - context.getInputDataSet(context.getInput(transform)); - @SuppressWarnings("unchecked") - AppliedPTransform< - PCollection, - PCollection, - PTransform, PCollection>> - application = - (AppliedPTransform< - PCollection, - PCollection, - PTransform, PCollection>>) - context.getCurrentTransform(); - org.apache.beam.sdk.values.PCollectionView input; - PCollection inputPCol = context.getInput(transform); - final Coder coder = inputPCol.getCoder(); - WindowingStrategy windowingStrategy = inputPCol.getWindowingStrategy(); - WindowFn windowFn = windowingStrategy.getWindowFn(); - try { - input = CreatePCollectionViewTranslation.getView(application); - } catch (IOException e) { - throw new RuntimeException(e); - } - - switch (input.getViewFn().getMaterialization().getUrn()) { - case Materializations.MULTIMAP_MATERIALIZATION_URN: - KvCoder kvCoder = (KvCoder) coder; - final Coder keyCoder = kvCoder.getKeyCoder(); - final WindowedValues.WindowedValueCoder kvwvCoder = - WindowedValues.FullWindowedValueCoder.of( - kvCoder.getValueCoder(), windowFn.windowCoder()); - BatchTSet> multimapMaterialization = - inputDataSet - .direct() - .map(new MapToTupleFunction<>(keyCoder, kvwvCoder)) - .allGather() - .map(new ByteToWindowFunctionPrimitive(keyCoder, kvwvCoder)); - context.setSideInputDataSet(input.getTagInternal().getId(), multimapMaterialization); - break; - case Materializations.ITERABLE_MATERIALIZATION_URN: - final WindowedValues.WindowedValueCoder wvCoder = - WindowedValues.FullWindowedValueCoder.of(coder, windowFn.windowCoder()); - BatchTSet> iterableMaterialization = - inputDataSet - .direct() - .map(new ElemToBytesFunction<>(wvCoder)) - .allGather() - .map(new ByteToElemFunction(wvCoder)); - try { - input = CreatePCollectionViewTranslation.getView(application); - } catch (IOException e) { - throw new RuntimeException(e); - } - context.setSideInputDataSet(input.getTagInternal().getId(), iterableMaterialization); - break; - default: - throw new UnsupportedOperationException( - "Unknown side input materialization " - + input.getViewFn().getMaterialization().getUrn()); - } - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/ParDoMultiOutputTranslatorBatch.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/ParDoMultiOutputTranslatorBatch.java deleted file mode 100644 index 9230c6efd868..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/ParDoMultiOutputTranslatorBatch.java +++ /dev/null @@ -1,131 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.batch; - -import edu.iu.dsc.tws.tset.sets.batch.BatchTSetImpl; -import edu.iu.dsc.tws.tset.sets.batch.ComputeTSet; -import java.io.IOException; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.Iterator; -import java.util.List; -import java.util.Map; -import org.apache.beam.runners.twister2.Twister2BatchTranslationContext; -import org.apache.beam.runners.twister2.translators.BatchTransformTranslator; -import org.apache.beam.runners.twister2.translators.functions.DoFnFunction; -import org.apache.beam.runners.twister2.translators.functions.OutputTagFilter; -import org.apache.beam.sdk.coders.Coder; -import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFnSchemaInformation; -import org.apache.beam.sdk.transforms.ParDo; -import org.apache.beam.sdk.transforms.join.RawUnionValue; -import org.apache.beam.sdk.transforms.reflect.DoFnSignatures; -import org.apache.beam.sdk.util.construction.ParDoTranslation; -import org.apache.beam.sdk.values.PCollection; -import org.apache.beam.sdk.values.PCollectionView; -import org.apache.beam.sdk.values.TupleTag; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowingStrategy; -import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Maps; - -/** ParDo translator. */ -@SuppressWarnings({ - "rawtypes", // TODO(https://github.com/apache/beam/issues/20447) - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class ParDoMultiOutputTranslatorBatch - implements BatchTransformTranslator> { - - @Override - public void translateNode( - ParDo.MultiOutput transform, Twister2BatchTranslationContext context) { - DoFn doFn; - doFn = transform.getFn(); - if (DoFnSignatures.signatureForDoFn(doFn).processElement().isSplittable()) { - throw new UnsupportedOperationException( - String.format( - "Not expected to directly translate splittable DoFn, should have been overridden: %s", - doFn)); - } - BatchTSetImpl> inputTTSet = - context.getInputDataSet(context.getInput(transform)); - - WindowingStrategy windowingStrategy = context.getInput(transform).getWindowingStrategy(); - Coder inputCoder = (Coder) context.getInput(transform).getCoder(); - Map> sideInputMapping; - - Map, PCollection> outputs = context.getOutputs(); - Map, Coder> outputCoders = context.getOutputCoders(); - - // DoFnSignature signature = DoFnSignatures.getSignature(transform.getFn().getClass()); - DoFnSchemaInformation doFnSchemaInformation; - doFnSchemaInformation = ParDoTranslation.getSchemaInformation(context.getCurrentTransform()); - sideInputMapping = ParDoTranslation.getSideInputMapping(context.getCurrentTransform()); - - TupleTag mainOutput = transform.getMainOutputTag(); - List> additionalOutputTags = - new ArrayList<>(transform.getAdditionalOutputTags().getAll()); - Map> sideInputs = transform.getSideInputs(); - // TODO : note change from List to map in sideinputs - - // construct a map from side input to WindowingStrategy so that - // the DoFn runner can map main-input windows to side input windows - Map, WindowingStrategy> sideInputStrategies = new HashMap<>(); - for (PCollectionView sideInput : sideInputs.values()) { - sideInputStrategies.put(sideInput, sideInput.getWindowingStrategyInternal()); - } - - TupleTag mainOutputTag; - try { - mainOutputTag = ParDoTranslation.getMainOutputTag(context.getCurrentTransform()); - } catch (IOException e) { - throw new RuntimeException(e); - } - Map, Integer> outputMap = Maps.newHashMap(); - outputMap.put(mainOutputTag, 0); - int count = 1; - for (TupleTag tag : outputs.keySet()) { - if (!outputMap.containsKey(tag)) { - outputMap.put(tag, count++); - } - } - - ComputeTSet>> outputTSet = - inputTTSet - .direct() - .compute( - new DoFnFunction( - context, - doFn, - inputCoder, - outputCoders, - additionalOutputTags, - windowingStrategy, - sideInputStrategies, - mainOutput, - doFnSchemaInformation, - outputMap, - sideInputMapping)); - - for (Map.Entry, PCollection> output : outputs.entrySet()) { - ComputeTSet, Iterator> tempTSet = - outputTSet.direct().compute(new OutputTagFilter(outputMap.get(output.getKey()))); - context.setOutputDataSet((PCollection) output.getValue(), tempTSet); - } - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/ReadSourceTranslatorBatch.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/ReadSourceTranslatorBatch.java deleted file mode 100644 index ac071db64815..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/ReadSourceTranslatorBatch.java +++ /dev/null @@ -1,49 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.batch; - -import edu.iu.dsc.tws.tset.env.BatchTSetEnvironment; -import edu.iu.dsc.tws.tset.env.TSetEnvironment; -import edu.iu.dsc.tws.tset.sets.batch.SourceTSet; -import org.apache.beam.runners.twister2.Twister2BatchTranslationContext; -import org.apache.beam.runners.twister2.translation.wrappers.Twister2BoundedSource; -import org.apache.beam.runners.twister2.translators.BatchTransformTranslator; -import org.apache.beam.sdk.io.BoundedSource; -import org.apache.beam.sdk.util.construction.SplittableParDo; -import org.apache.beam.sdk.values.PCollection; -import org.apache.beam.sdk.values.WindowedValue; - -/** Source translator. */ -public class ReadSourceTranslatorBatch - implements BatchTransformTranslator> { - - @Override - public void translateNode( - SplittableParDo.PrimitiveBoundedRead transform, Twister2BatchTranslationContext context) { - BoundedSource boundedSource = transform.getSource(); - Twister2BoundedSource twister2BoundedSource = - new Twister2BoundedSource(boundedSource, context, context.getOptions()); - final TSetEnvironment tsetEnv = context.getEnvironment(); - - SourceTSet> sourceTSet = - ((BatchTSetEnvironment) tsetEnv) - .createSource(twister2BoundedSource, context.getOptions().getParallelism()); - PCollection output = context.getOutput(transform); - context.setOutputDataSet(output, sourceTSet); - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/package-info.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/package-info.java deleted file mode 100644 index b65f5798478b..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/batch/package-info.java +++ /dev/null @@ -1,20 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/** Internal implementation of the Beam runner for Twister2. */ -package org.apache.beam.runners.twister2.translators.batch; diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/AssignWindowsFunction.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/AssignWindowsFunction.java deleted file mode 100644 index 9c19868a7c5b..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/AssignWindowsFunction.java +++ /dev/null @@ -1,111 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.functions; - -import edu.iu.dsc.tws.api.tset.TSetContext; -import edu.iu.dsc.tws.api.tset.fn.ComputeCollectorFunc; -import edu.iu.dsc.tws.api.tset.fn.RecordCollector; -import java.io.ObjectStreamException; -import java.util.Collection; -import java.util.Iterator; -import java.util.logging.Logger; -import org.apache.beam.model.pipeline.v1.RunnerApi; -import org.apache.beam.runners.twister2.utils.Twister2AssignContext; -import org.apache.beam.sdk.options.PipelineOptions; -import org.apache.beam.sdk.options.PortablePipelineOptions; -import org.apache.beam.sdk.transforms.windowing.BoundedWindow; -import org.apache.beam.sdk.transforms.windowing.WindowFn; -import org.apache.beam.sdk.util.construction.Environments; -import org.apache.beam.sdk.util.construction.SdkComponents; -import org.apache.beam.sdk.util.construction.WindowingStrategyTranslation; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowedValues; -import org.apache.beam.vendor.grpc.v1p69p0.com.google.protobuf.InvalidProtocolBufferException; - -/** Assign Windows function. */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class AssignWindowsFunction - implements ComputeCollectorFunc, Iterator>> { - private static final Logger LOG = Logger.getLogger(AssignWindowsFunction.class.getName()); - - private transient boolean isInitialized = false; - private transient WindowFn windowFn; - private byte[] windowFnBytes; - - public AssignWindowsFunction(WindowFn windowFn, PipelineOptions options) { - this.windowFn = windowFn; - SdkComponents components = SdkComponents.create(); - components.registerEnvironment( - Environments.createOrGetDefaultEnvironment(options.as(PortablePipelineOptions.class))); - RunnerApi.FunctionSpec windowFnProto = - WindowingStrategyTranslation.toProto(windowFn, components); - windowFnBytes = windowFnProto.toByteArray(); - } - - @Override - public void compute(Iterator> input, RecordCollector> output) { - WindowedValue element; - try { - while (input.hasNext()) { - element = input.next(); - Collection windows = - windowFn.assignWindows(new Twister2AssignContext<>(windowFn, element)); - - for (BoundedWindow window : windows) { - output.collect( - WindowedValues.of( - element.getValue(), element.getTimestamp(), window, element.getPaneInfo())); - } - } - } catch (Exception e) { - LOG.info(e.getMessage()); - } - } - - @Override - public void prepare(TSetContext context) { - initTransient(); - } - - /** - * Method used to initialize the transient variables that were sent over as byte arrays or proto - * buffers. - */ - private void initTransient() { - if (isInitialized) { - return; - } - - try { - RunnerApi.FunctionSpec windowFnProto = RunnerApi.FunctionSpec.parseFrom(windowFnBytes); - - windowFn = - (WindowFn) - WindowingStrategyTranslation.windowFnFromProto(windowFnProto); - } catch (InvalidProtocolBufferException e) { - LOG.info(e.getMessage()); - } - this.isInitialized = true; - } - - protected Object readResolve() throws ObjectStreamException { - return this; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ByteToElemFunction.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ByteToElemFunction.java deleted file mode 100644 index ca1fe4797961..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ByteToElemFunction.java +++ /dev/null @@ -1,77 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.functions; - -import edu.iu.dsc.tws.api.tset.TSetContext; -import edu.iu.dsc.tws.api.tset.fn.MapFunc; -import java.io.ObjectStreamException; -import org.apache.beam.runners.twister2.utils.TranslationUtils; -import org.apache.beam.sdk.util.SerializableUtils; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowedValues.WindowedValueCoder; - -/** ByteToWindow function. */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class ByteToElemFunction implements MapFunc, byte[]> { - private transient WindowedValueCoder wvCoder; - - private transient boolean isInitialized = false; - private byte[] wvCoderBytes; - - public ByteToElemFunction() { - // non arg constructor needed for kryo - isInitialized = false; - } - - public ByteToElemFunction(final WindowedValueCoder wvCoder) { - this.wvCoder = wvCoder; - - wvCoderBytes = SerializableUtils.serializeToByteArray(wvCoder); - } - - @Override - public WindowedValue map(byte[] input) { - return TranslationUtils.fromByteArray(input, wvCoder); - } - - @Override - public void prepare(TSetContext context) { - initTransient(); - } - - /** - * Method used to initialize the transient variables that were sent over as byte arrays or proto - * buffers. - */ - private void initTransient() { - if (isInitialized) { - return; - } - - wvCoder = - (WindowedValueCoder) - SerializableUtils.deserializeFromByteArray(wvCoderBytes, "Custom Coder Bytes"); - this.isInitialized = true; - } - - protected Object readResolve() throws ObjectStreamException { - return this; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ByteToWindowFunction.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ByteToWindowFunction.java deleted file mode 100644 index 0a116a0d876d..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ByteToWindowFunction.java +++ /dev/null @@ -1,109 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.functions; - -import edu.iu.dsc.tws.api.comms.structs.Tuple; -import edu.iu.dsc.tws.api.tset.TSetContext; -import edu.iu.dsc.tws.api.tset.fn.MapFunc; -import java.io.ObjectStreamException; -import java.util.Iterator; -import java.util.Spliterator; -import java.util.Spliterators; -import java.util.logging.Logger; -import java.util.stream.Collectors; -import java.util.stream.StreamSupport; -import org.apache.beam.runners.twister2.utils.TranslationUtils; -import org.apache.beam.sdk.coders.Coder; -import org.apache.beam.sdk.coders.CoderException; -import org.apache.beam.sdk.util.CoderUtils; -import org.apache.beam.sdk.util.SerializableUtils; -import org.apache.beam.sdk.values.KV; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowedValues.WindowedValueCoder; - -/** ByteToWindow function. */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class ByteToWindowFunction - implements MapFunc>>, Tuple>> { - private transient Coder keyCoder; - private transient WindowedValueCoder wvCoder; - private static final Logger LOG = Logger.getLogger(ByteToWindowFunction.class.getName()); - - private transient boolean isInitialized = false; - private byte[] keyCoderBytes; - private byte[] wvCoderBytes; - - public ByteToWindowFunction() { - // non arg constructor needed for kryo - isInitialized = false; - } - - public ByteToWindowFunction(final Coder inputKeyCoder, final WindowedValueCoder wvCoder) { - this.keyCoder = inputKeyCoder; - this.wvCoder = wvCoder; - - keyCoderBytes = SerializableUtils.serializeToByteArray(keyCoder); - wvCoderBytes = SerializableUtils.serializeToByteArray(wvCoder); - } - - @Override - public KV>> map(Tuple> input) { - K key = null; - Iterable> value = null; - try { - key = CoderUtils.decodeFromByteArray(keyCoder, input.getKey()); - // TODO need to replace this with a custom iterator - value = - StreamSupport.stream( - Spliterators.spliteratorUnknownSize(input.getValue(), Spliterator.ORDERED), false) - .map(bytes -> TranslationUtils.fromByteArray(bytes, wvCoder)) - .collect(Collectors.toList()); - } catch (CoderException e) { - LOG.info(e.getMessage()); - } - return KV.of(key, value); - } - - @Override - public void prepare(TSetContext context) { - initTransient(); - } - - /** - * Method used to initialize the transient variables that were sent over as byte arrays or proto - * buffers. - */ - private void initTransient() { - if (isInitialized) { - return; - } - - keyCoder = - (Coder) SerializableUtils.deserializeFromByteArray(keyCoderBytes, "Custom Coder Bytes"); - wvCoder = - (WindowedValueCoder) - SerializableUtils.deserializeFromByteArray(wvCoderBytes, "Custom Coder Bytes"); - this.isInitialized = true; - } - - protected Object readResolve() throws ObjectStreamException { - return this; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ByteToWindowFunctionPrimitive.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ByteToWindowFunctionPrimitive.java deleted file mode 100644 index ad2f3545564a..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ByteToWindowFunctionPrimitive.java +++ /dev/null @@ -1,114 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.functions; - -import edu.iu.dsc.tws.api.comms.structs.Tuple; -import edu.iu.dsc.tws.api.tset.TSetContext; -import edu.iu.dsc.tws.api.tset.fn.MapFunc; -import java.io.ObjectStreamException; -import java.util.logging.Logger; -import org.apache.beam.runners.twister2.utils.TranslationUtils; -import org.apache.beam.sdk.coders.Coder; -import org.apache.beam.sdk.coders.CoderException; -import org.apache.beam.sdk.util.CoderUtils; -import org.apache.beam.sdk.util.SerializableUtils; -import org.apache.beam.sdk.values.KV; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowedValues; -import org.apache.beam.sdk.values.WindowedValues.WindowedValueCoder; - -/** ByteToWindow function. */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class ByteToWindowFunctionPrimitive - implements MapFunc>, Tuple> { - private transient Coder keyCoder; - private transient WindowedValueCoder wvCoder; - private static final Logger LOG = Logger.getLogger(ByteToWindowFunctionPrimitive.class.getName()); - - private transient boolean isInitialized = false; - private byte[] keyCoderBytes; - private byte[] wvCoderBytes; - - public ByteToWindowFunctionPrimitive() { - // non arg constructor needed for kryo - isInitialized = false; - } - - public ByteToWindowFunctionPrimitive( - final Coder inputKeyCoder, final WindowedValueCoder wvCoder) { - this.keyCoder = inputKeyCoder; - this.wvCoder = wvCoder; - - keyCoderBytes = SerializableUtils.serializeToByteArray(keyCoder); - wvCoderBytes = SerializableUtils.serializeToByteArray(wvCoder); - } - - @Override - public WindowedValue> map(Tuple input) { - K key = null; - WindowedValue value = null; - try { - key = CoderUtils.decodeFromByteArray(keyCoder, input.getKey()); - - value = TranslationUtils.fromByteArray(input.getValue(), wvCoder); - } catch (CoderException e) { - LOG.info(e.getMessage()); - } - WindowedValue> element; - - if (value == null) { - value = WindowedValues.valueInGlobalWindow(null); - } - element = - WindowedValues.of( - KV.of(key, value.getValue()), - value.getTimestamp(), - value.getWindows(), - value.getPaneInfo()); - - return element; - } - - @Override - public void prepare(TSetContext context) { - initTransient(); - } - - /** - * Method used to initialize the transient variables that were sent over as byte arrays or proto - * buffers. - */ - private void initTransient() { - if (isInitialized) { - return; - } - - keyCoder = - (Coder) SerializableUtils.deserializeFromByteArray(keyCoderBytes, "Custom Coder Bytes"); - wvCoder = - (WindowedValueCoder) - SerializableUtils.deserializeFromByteArray(wvCoderBytes, "Custom Coder Bytes"); - this.isInitialized = true; - } - - protected Object readResolve() throws ObjectStreamException { - return this; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/DoFnFunction.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/DoFnFunction.java deleted file mode 100644 index 4a63b717912e..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/DoFnFunction.java +++ /dev/null @@ -1,362 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.functions; - -import edu.iu.dsc.tws.api.tset.TSetContext; -import edu.iu.dsc.tws.api.tset.fn.ComputeCollectorFunc; -import edu.iu.dsc.tws.api.tset.fn.RecordCollector; -import java.io.IOException; -import java.io.ObjectStreamException; -import java.io.Serializable; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.HashSet; -import java.util.Iterator; -import java.util.List; -import java.util.Map; -import java.util.Optional; -import java.util.Set; -import java.util.logging.Logger; -import org.apache.beam.model.pipeline.v1.RunnerApi; -import org.apache.beam.runners.core.DoFnRunner; -import org.apache.beam.runners.core.DoFnRunners; -import org.apache.beam.runners.core.StepContext; -import org.apache.beam.runners.core.construction.SerializablePipelineOptions; -import org.apache.beam.runners.twister2.Twister2TranslationContext; -import org.apache.beam.runners.twister2.utils.NoOpStepContext; -import org.apache.beam.runners.twister2.utils.Twister2SideInputReader; -import org.apache.beam.sdk.coders.Coder; -import org.apache.beam.sdk.options.PipelineOptions; -import org.apache.beam.sdk.options.PortablePipelineOptions; -import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFnSchemaInformation; -import org.apache.beam.sdk.transforms.join.RawUnionValue; -import org.apache.beam.sdk.transforms.reflect.DoFnInvoker; -import org.apache.beam.sdk.transforms.reflect.DoFnInvokers; -import org.apache.beam.sdk.util.DoFnWithExecutionInformation; -import org.apache.beam.sdk.util.SerializableUtils; -import org.apache.beam.sdk.util.WindowedValueMultiReceiver; -import org.apache.beam.sdk.util.construction.Environments; -import org.apache.beam.sdk.util.construction.ParDoTranslation; -import org.apache.beam.sdk.util.construction.RehydratedComponents; -import org.apache.beam.sdk.util.construction.SdkComponents; -import org.apache.beam.sdk.util.construction.WindowingStrategyTranslation; -import org.apache.beam.sdk.values.PCollectionView; -import org.apache.beam.sdk.values.TupleTag; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowingStrategy; -import org.apache.beam.vendor.grpc.v1p69p0.com.google.protobuf.InvalidProtocolBufferException; - -/** DoFn function. */ -@SuppressWarnings({ - "rawtypes", // TODO(https://github.com/apache/beam/issues/20447) - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class DoFnFunction - implements ComputeCollectorFunc>> { - private static final Logger LOG = Logger.getLogger(DoFnFunction.class.getName()); - - private transient DoFn doFn; - private transient PipelineOptions pipelineOptions; - private static final long serialVersionUID = -5701440128544343353L; - private transient Coder inputCoder; - private transient Map, Coder> outputCoders; - private transient WindowingStrategy windowingStrategy; - private transient Map, WindowingStrategy> sideInputs; - private transient TupleTag mainOutput; - private transient Twister2SideInputReader sideInputReader; - private transient DoFnRunner doFnRunner; - private transient DoFnOutputManager outputManager; - private transient List> sideOutputs; - private StepContext stepcontext; - private transient DoFnSchemaInformation doFnSchemaInformation; - private transient Map, Integer> outputMap; - private transient Map> sideInputMapping; - private transient DoFnInvoker doFnInvoker; - - // Byte arrays needed for serialization - private transient boolean isInitialized = false; - private transient RunnerApi.FunctionSpec doFnwithEx; - private byte[] doFnwithExBytes; - private byte[] coderBytes; - private Map outputCodersBytes; - private transient RunnerApi.MessageWithComponents windowStrategyProto; - private byte[] windowBytes; - private Map sideInputBytes; - private String serializedOptions; - private List serializedSideOutputs; - private Map serializedOutputMap; - - public DoFnFunction() { - // non arg constructor needed for kryo - this.isInitialized = false; - } - - public DoFnFunction( - Twister2TranslationContext context, - DoFn doFn, - Coder inputCoder, - Map, Coder> outputCoders, - List> sideOutputs, - WindowingStrategy windowingStrategy, - Map, WindowingStrategy> sideInputs, - TupleTag mainOutput, - DoFnSchemaInformation doFnSchemaInformation, - Map, Integer> outputMap, - Map> sideInputMapping) { - this.doFn = doFn; - this.pipelineOptions = context.getOptions(); - this.serializedOptions = new SerializablePipelineOptions(pipelineOptions).toString(); - this.inputCoder = inputCoder; - this.outputCoders = outputCoders; - this.windowingStrategy = windowingStrategy; - this.sideInputs = convertToTuples(sideInputs); - this.mainOutput = mainOutput; - this.doFnSchemaInformation = doFnSchemaInformation; - this.sideOutputs = sideOutputs; - this.stepcontext = new NoOpStepContext(); - this.outputMap = outputMap; - this.sideInputMapping = sideInputMapping; - outputManager = new DoFnOutputManager(this.outputMap); - prepareSerialization(); - } - - private Map, WindowingStrategy> convertToTuples( - Map, WindowingStrategy> sideInputs) { - Map, WindowingStrategy> temp = new HashMap<>(); - for (Map.Entry, WindowingStrategy> entry : sideInputs.entrySet()) { - temp.put(entry.getKey().getTagInternal(), entry.getValue()); - } - return temp; - } - - /** - * get the tag id's of all the keys. - * - * @return A Set of String key values. - */ - public Set getSideInputKeys() { - initTransient(); - Set keys = new HashSet<>(); - for (TupleTag view : sideInputs.keySet()) { - keys.add(view.getId()); - } - return keys; - } - - @Override - public void prepare(TSetContext context) { - initTransient(); - sideInputReader = new Twister2SideInputReader(sideInputs, context); - outputManager.setup(mainOutput, sideOutputs); - doFnInvoker = DoFnInvokers.tryInvokeSetupFor(doFn, pipelineOptions); - - doFnRunner = - DoFnRunners.simpleRunner( - pipelineOptions, - doFn, - sideInputReader, - outputManager, - mainOutput, - sideOutputs, - stepcontext, - inputCoder, - outputCoders, - windowingStrategy, - doFnSchemaInformation, - sideInputMapping); - } - - @Override - public void compute( - Iterator> input, RecordCollector output) { - try { - outputManager.clear(); - doFnRunner.startBundle(); - while (input.hasNext()) { - doFnRunner.processElement(input.next()); - } - - doFnRunner.finishBundle(); - Iterator outputs = outputManager.getOutputs(); - while (outputs.hasNext()) { - - output.collect(outputs.next()); - } - } catch (final RuntimeException re) { - DoFnInvokers.invokerFor(doFn).invokeTeardown(); - throw re; - } - } - - @Override - public void close() { - Optional.ofNullable(doFnInvoker).ifPresent(DoFnInvoker::invokeTeardown); - } - - private static class DoFnOutputManager implements WindowedValueMultiReceiver, Serializable { - // todo need to figure out how this class types are handled - private static final long serialVersionUID = 4967375172737408160L; - private transient List outputs; - private transient Set> outputTags; - private Map, Integer> outputMap; - - private DoFnOutputManager() {} - - DoFnOutputManager(Map, Integer> outputMap) { - this.outputMap = outputMap; - } - - @Override - public void output(TupleTag outputTag, WindowedValue output) { - if (outputTags.contains(outputTag)) { - outputs.add(new RawUnionValue(outputMap.get(outputTag), output)); - } - } - - void setup(TupleTag mainOutput, List> sideOutputs) { - outputs = new ArrayList<>(); - outputTags = new HashSet<>(); - outputTags.add(mainOutput); - outputTags.addAll(sideOutputs); - } - - void clear() { - outputs.clear(); - } - - Iterator getOutputs() { - return outputs.iterator(); - } - } - - protected Object readResolve() throws ObjectStreamException { - return this; - } - - /** - * prepares the DoFnFunction class so it can be serialized properly. This involves using various - * protobuf's and byte arrays which are later converted back into the proper classes during - * deserialization. - */ - private void prepareSerialization() { - SdkComponents components = SdkComponents.create(); - components.registerEnvironment( - Environments.createOrGetDefaultEnvironment( - pipelineOptions.as(PortablePipelineOptions.class))); - this.serializedOptions = new SerializablePipelineOptions(pipelineOptions).toString(); - - doFnwithEx = - ParDoTranslation.translateDoFn( - this.doFn, mainOutput, sideInputMapping, doFnSchemaInformation, components); - doFnwithExBytes = doFnwithEx.getPayload().toByteArray(); - outputCodersBytes = new HashMap<>(); - try { - coderBytes = SerializableUtils.serializeToByteArray(inputCoder); - windowStrategyProto = - WindowingStrategyTranslation.toMessageProto(windowingStrategy, components); - windowBytes = windowStrategyProto.toByteArray(); - for (Map.Entry, Coder> entry : outputCoders.entrySet()) { - outputCodersBytes.put( - entry.getKey().getId(), SerializableUtils.serializeToByteArray(entry.getValue())); - } - sideInputBytes = new HashMap<>(); - for (Map.Entry, WindowingStrategy> entry : sideInputs.entrySet()) { - windowStrategyProto = - WindowingStrategyTranslation.toMessageProto(entry.getValue(), components); - sideInputBytes.put(entry.getKey().getId(), windowStrategyProto.toByteArray()); - } - - serializedSideOutputs = new ArrayList<>(); - for (TupleTag sideOutput : sideOutputs) { - serializedSideOutputs.add(sideOutput.getId()); - } - - serializedOutputMap = new HashMap<>(); - for (Map.Entry, Integer> entry : outputMap.entrySet()) { - serializedOutputMap.put(entry.getKey().getId(), entry.getValue()); - } - } catch (IOException e) { - LOG.info(e.getMessage()); - } - } - - /** - * Method used to initialize the transient variables that were sent over as byte arrays or proto - * buffers. - */ - private void initTransient() { - if (isInitialized) { - return; - } - try { - SdkComponents components = SdkComponents.create(); - - pipelineOptions = new SerializablePipelineOptions(serializedOptions).get(); - DoFnWithExecutionInformation doFnWithExecutionInformation = - (DoFnWithExecutionInformation) - SerializableUtils.deserializeFromByteArray(doFnwithExBytes, "Custom Coder Bytes"); - this.doFn = (DoFn) doFnWithExecutionInformation.getDoFn(); - this.mainOutput = (TupleTag) doFnWithExecutionInformation.getMainOutputTag(); - this.sideInputMapping = doFnWithExecutionInformation.getSideInputMapping(); - this.doFnSchemaInformation = doFnWithExecutionInformation.getSchemaInformation(); - - inputCoder = - (Coder) - SerializableUtils.deserializeFromByteArray(coderBytes, "Custom Coder Bytes"); - - windowStrategyProto = RunnerApi.MessageWithComponents.parseFrom(windowBytes); - - windowingStrategy = - (WindowingStrategy) - WindowingStrategyTranslation.fromProto( - windowStrategyProto.getWindowingStrategy(), - RehydratedComponents.forComponents(components.toComponents())); - sideInputs = new HashMap<>(); - for (Map.Entry entry : sideInputBytes.entrySet()) { - windowStrategyProto = RunnerApi.MessageWithComponents.parseFrom(entry.getValue()); - sideInputs.put( - new TupleTag<>(entry.getKey()), - WindowingStrategyTranslation.fromProto( - windowStrategyProto.getWindowingStrategy(), - RehydratedComponents.forComponents(components.toComponents()))); - } - } catch (InvalidProtocolBufferException e) { - LOG.info(e.getMessage()); - } - outputCoders = new HashMap<>(); - for (Map.Entry entry : outputCodersBytes.entrySet()) { - outputCoders.put( - new TupleTag<>(entry.getKey()), - (Coder) - SerializableUtils.deserializeFromByteArray(entry.getValue(), "Custom Coder Bytes")); - } - - sideOutputs = new ArrayList<>(); - for (String sideOutput : serializedSideOutputs) { - sideOutputs.add(new TupleTag<>(sideOutput)); - } - - outputMap = new HashMap<>(); - for (Map.Entry entry : serializedOutputMap.entrySet()) { - outputMap.put(new TupleTag<>(entry.getKey()), entry.getValue()); - } - outputManager = new DoFnOutputManager(this.outputMap); - this.isInitialized = true; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ElemToBytesFunction.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ElemToBytesFunction.java deleted file mode 100644 index 0121eb2cfcc0..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ElemToBytesFunction.java +++ /dev/null @@ -1,85 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.functions; - -import edu.iu.dsc.tws.api.tset.TSetContext; -import edu.iu.dsc.tws.api.tset.fn.MapFunc; -import java.io.ObjectStreamException; -import java.util.logging.Logger; -import org.apache.beam.sdk.coders.CoderException; -import org.apache.beam.sdk.util.CoderUtils; -import org.apache.beam.sdk.util.SerializableUtils; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowedValues; -import org.checkerframework.checker.nullness.qual.Nullable; - -/** Map to tuple function. */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class ElemToBytesFunction implements MapFunc> { - - private transient WindowedValues.WindowedValueCoder wvCoder; - private static final Logger LOG = Logger.getLogger(ElemToBytesFunction.class.getName()); - - private transient boolean isInitialized = false; - private byte[] wvCoderBytes; - - public ElemToBytesFunction() { - // non arg constructor needed for kryo - this.isInitialized = false; - } - - public ElemToBytesFunction(WindowedValues.WindowedValueCoder wvCoder) { - this.wvCoder = wvCoder; - wvCoderBytes = SerializableUtils.serializeToByteArray(wvCoder); - } - - @Override - public byte @Nullable [] map(WindowedValue input) { - try { - return CoderUtils.encodeToByteArray(wvCoder, input); - } catch (CoderException e) { - LOG.info(e.getMessage()); - } - return null; - } - - @Override - public void prepare(TSetContext context) { - initTransient(); - } - - /** - * Method used to initialize the transient variables that were sent over as byte arrays or proto - * buffers. - */ - private void initTransient() { - if (isInitialized) { - return; - } - wvCoder = - (WindowedValues.WindowedValueCoder) - SerializableUtils.deserializeFromByteArray(wvCoderBytes, "Coder"); - this.isInitialized = true; - } - - protected Object readResolve() throws ObjectStreamException { - return this; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/GroupByWindowFunction.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/GroupByWindowFunction.java deleted file mode 100644 index ba8999c20c4c..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/GroupByWindowFunction.java +++ /dev/null @@ -1,213 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.functions; - -import edu.iu.dsc.tws.api.tset.TSetContext; -import edu.iu.dsc.tws.api.tset.fn.FlatMapFunc; -import edu.iu.dsc.tws.api.tset.fn.RecordCollector; -import java.io.IOException; -import java.io.ObjectStreamException; -import java.util.ArrayList; -import java.util.Iterator; -import java.util.List; -import java.util.logging.Logger; -import org.apache.beam.model.pipeline.v1.RunnerApi; -import org.apache.beam.runners.core.InMemoryStateInternals; -import org.apache.beam.runners.core.InMemoryTimerInternals; -import org.apache.beam.runners.core.ReduceFnRunner; -import org.apache.beam.runners.core.StateInternals; -import org.apache.beam.runners.core.SystemReduceFn; -import org.apache.beam.runners.core.TimerInternals; -import org.apache.beam.runners.core.UnsupportedSideInputReader; -import org.apache.beam.runners.core.triggers.ExecutableTriggerStateMachine; -import org.apache.beam.runners.core.triggers.TriggerStateMachines; -import org.apache.beam.sdk.options.PipelineOptions; -import org.apache.beam.sdk.options.PortablePipelineOptions; -import org.apache.beam.sdk.transforms.windowing.BoundedWindow; -import org.apache.beam.sdk.util.WindowedValueReceiver; -import org.apache.beam.sdk.util.construction.Environments; -import org.apache.beam.sdk.util.construction.RehydratedComponents; -import org.apache.beam.sdk.util.construction.SdkComponents; -import org.apache.beam.sdk.util.construction.TriggerTranslation; -import org.apache.beam.sdk.util.construction.WindowingStrategyTranslation; -import org.apache.beam.sdk.values.KV; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowingStrategy; -import org.apache.beam.vendor.grpc.v1p69p0.com.google.protobuf.InvalidProtocolBufferException; -import org.joda.time.Instant; - -/** GroupBy window function. */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class GroupByWindowFunction - implements FlatMapFunc>>, KV>>> { - private static final Logger LOG = Logger.getLogger(GroupByWindowFunction.class.getName()); - private transient WindowingStrategy windowingStrategy; - private SystemReduceFn, Iterable, W> reduceFn; - // Options are kept as null this should be fixed when possible - - private transient boolean isInitialized = false; - - private transient RunnerApi.MessageWithComponents windowStrategyProto; - private byte[] windowBytes; - - public GroupByWindowFunction() { - // non arg constructor needed for kryo - this.isInitialized = false; - } - - public GroupByWindowFunction( - WindowingStrategy windowingStrategy, - SystemReduceFn, Iterable, W> reduceFn, - PipelineOptions options) { - this.windowingStrategy = windowingStrategy; - SdkComponents components = SdkComponents.create(); - components.registerEnvironment( - Environments.createOrGetDefaultEnvironment(options.as(PortablePipelineOptions.class))); - - try { - windowStrategyProto = - WindowingStrategyTranslation.toMessageProto(windowingStrategy, components); - windowBytes = windowStrategyProto.toByteArray(); - } catch (IOException e) { - LOG.info(e.getMessage()); - } - this.reduceFn = reduceFn; - } - - @Override - public void flatMap( - KV>> kIteratorKV, - RecordCollector>>> collector) { - try { - K key = kIteratorKV.getKey(); - Iterable> values = kIteratorKV.getValue(); - - InMemoryTimerInternals timerInternals = new InMemoryTimerInternals(); - timerInternals.advanceProcessingTime(Instant.now()); - timerInternals.advanceSynchronizedProcessingTime(Instant.now()); - StateInternals stateInternals = InMemoryStateInternals.forKey(key); - GABWWindowedValueReceiver outputter = new GABWWindowedValueReceiver<>(); - - ReduceFnRunner, W> reduceFnRunner = - new ReduceFnRunner<>( - key, - windowingStrategy, - ExecutableTriggerStateMachine.create( - TriggerStateMachines.stateMachineForTrigger( - TriggerTranslation.toProto(windowingStrategy.getTrigger()))), - stateInternals, - timerInternals, - outputter, - new UnsupportedSideInputReader("GroupAlsoByWindow"), - reduceFn, - null); - - // Process the grouped values. - reduceFnRunner.processElements(values); - - // Finish any pending windows by advancing the input watermark to infinity. - timerInternals.advanceInputWatermark(BoundedWindow.TIMESTAMP_MAX_VALUE); - - // Finally, advance the processing time to infinity to fire any timers. - timerInternals.advanceProcessingTime(BoundedWindow.TIMESTAMP_MAX_VALUE); - timerInternals.advanceSynchronizedProcessingTime(BoundedWindow.TIMESTAMP_MAX_VALUE); - - fireEligibleTimers(timerInternals, reduceFnRunner); - - reduceFnRunner.persist(); - Iterator>>> outputs = outputter.getOutputs().iterator(); - while (outputs.hasNext()) { - collector.collect(outputs.next()); - } - } catch (Exception e) { - LOG.info(e.getMessage()); - } - } - - private void fireEligibleTimers( - InMemoryTimerInternals timerInternals, ReduceFnRunner, W> reduceFnRunner) - throws Exception { - List timers = new ArrayList<>(); - while (true) { - TimerInternals.TimerData timer; - while ((timer = timerInternals.removeNextEventTimer()) != null) { - timers.add(timer); - } - while ((timer = timerInternals.removeNextProcessingTimer()) != null) { - timers.add(timer); - } - while ((timer = timerInternals.removeNextSynchronizedProcessingTimer()) != null) { - timers.add(timer); - } - if (timers.isEmpty()) { - break; - } - reduceFnRunner.onTimers(timers); - timers.clear(); - } - } - - @Override - public void prepare(TSetContext context) { - initTransient(); - } - - private static class GABWWindowedValueReceiver - implements WindowedValueReceiver>> { - private final List>>> outputs = new ArrayList<>(); - - @Override - public void output(WindowedValue>> output) { - outputs.add(output); - } - - Iterable>>> getOutputs() { - return outputs; - } - } - - /** - * Method used to initialize the transient variables that were sent over as byte arrays or proto - * buffers. - */ - private void initTransient() { - if (isInitialized) { - return; - } - - SdkComponents components = SdkComponents.create(); - - try { - windowStrategyProto = RunnerApi.MessageWithComponents.parseFrom(windowBytes); - windowingStrategy = - (WindowingStrategy) - WindowingStrategyTranslation.fromProto( - windowStrategyProto.getWindowingStrategy(), - RehydratedComponents.forComponents(components.toComponents())); - } catch (InvalidProtocolBufferException e) { - LOG.info(e.getMessage()); - } - this.isInitialized = true; - } - - protected Object readResolve() throws ObjectStreamException { - return this; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ImpulseSource.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ImpulseSource.java deleted file mode 100644 index 3d53fe9166c1..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/ImpulseSource.java +++ /dev/null @@ -1,44 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.functions; - -import edu.iu.dsc.tws.api.tset.fn.SourceFunc; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowedValues; - -/** A {@link SourceFunc} which executes the impulse transform contract. */ -public class ImpulseSource implements SourceFunc> { - private static final WindowedValue IMPULSE_VALUE = - WindowedValues.valueInGlobalWindow(new byte[0]); - - private boolean impulseEmitted = false; - - @Override - public boolean hasNext() { - return !impulseEmitted; - } - - @Override - public WindowedValue next() { - if (impulseEmitted) { - throw new IllegalStateException("Impulse has already emitted an element."); - } - impulseEmitted = true; - return IMPULSE_VALUE; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/MapToTupleFunction.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/MapToTupleFunction.java deleted file mode 100644 index dd1e0eb7d868..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/MapToTupleFunction.java +++ /dev/null @@ -1,110 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.functions; - -import edu.iu.dsc.tws.api.comms.structs.Tuple; -import edu.iu.dsc.tws.api.tset.TSetContext; -import edu.iu.dsc.tws.api.tset.fn.MapFunc; -import java.io.ObjectStreamException; -import java.util.logging.Logger; -import org.apache.beam.sdk.coders.Coder; -import org.apache.beam.sdk.coders.CoderException; -import org.apache.beam.sdk.util.CoderUtils; -import org.apache.beam.sdk.util.SerializableUtils; -import org.apache.beam.sdk.values.KV; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowedValues; - -/** Map to tuple function. */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class MapToTupleFunction - implements MapFunc, WindowedValue>> { - - private transient Coder keyCoder; - private transient WindowedValues.WindowedValueCoder wvCoder; - private static final Logger LOG = Logger.getLogger(MapToTupleFunction.class.getName()); - - private transient boolean isInitialized = false; - private byte[] keyCoderBytes; - private byte[] wvCoderBytes; - - public MapToTupleFunction() { - // non arg constructor needed for kryo - this.isInitialized = false; - } - - public MapToTupleFunction(Coder inputKeyCoder, WindowedValues.WindowedValueCoder wvCoder) { - this.keyCoder = inputKeyCoder; - this.wvCoder = wvCoder; - keyCoderBytes = SerializableUtils.serializeToByteArray(keyCoder); - wvCoderBytes = SerializableUtils.serializeToByteArray(wvCoder); - } - - @Override - public Tuple map(WindowedValue> input) { - Tuple element = null; - - WindowedValue>> temp = - WindowedValues.of( - KV.of( - input.getValue().getKey(), - WindowedValues.of( - input.getValue().getValue(), - input.getTimestamp(), - input.getWindows(), - input.getPaneInfo())), - input.getTimestamp(), - input.getWindows(), - input.getPaneInfo()); - try { - element = - new Tuple<>( - CoderUtils.encodeToByteArray(keyCoder, temp.getValue().getKey()), - CoderUtils.encodeToByteArray(wvCoder, temp.getValue().getValue())); - } catch (CoderException e) { - LOG.info(e.getMessage()); - } - return element; - } - - @Override - public void prepare(TSetContext context) { - initTransient(); - } - - /** - * Method used to initialize the transient variables that were sent over as byte arrays or proto - * buffers. - */ - private void initTransient() { - if (isInitialized) { - return; - } - keyCoder = (Coder) SerializableUtils.deserializeFromByteArray(keyCoderBytes, "Coder"); - wvCoder = - (WindowedValues.WindowedValueCoder) - SerializableUtils.deserializeFromByteArray(wvCoderBytes, "Coder"); - this.isInitialized = true; - } - - protected Object readResolve() throws ObjectStreamException { - return this; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/OutputTagFilter.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/OutputTagFilter.java deleted file mode 100644 index b8d00c3925ed..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/OutputTagFilter.java +++ /dev/null @@ -1,58 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.functions; - -import edu.iu.dsc.tws.api.tset.TSetContext; -import edu.iu.dsc.tws.api.tset.fn.ComputeCollectorFunc; -import edu.iu.dsc.tws.api.tset.fn.RecordCollector; -import java.util.Iterator; -import org.apache.beam.sdk.transforms.join.RawUnionValue; -import org.apache.beam.sdk.values.WindowedValue; - -/** Output tag filter. */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class OutputTagFilter - implements ComputeCollectorFunc, Iterator> { - - private int tag; - - public OutputTagFilter() { - // non arg constructor needed for kryo - } - - public OutputTagFilter(int tag) { - this.tag = tag; - } - - @Override - public void compute( - Iterator input, RecordCollector> output) { - RawUnionValue temp; - while (input.hasNext()) { - temp = input.next(); - if (temp.getUnionTag() == tag) { - output.collect((WindowedValue) temp.getValue()); - } - } - } - - @Override - public void prepare(TSetContext context) {} -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/Twister2SinkFunction.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/Twister2SinkFunction.java deleted file mode 100644 index e806ceab647d..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/Twister2SinkFunction.java +++ /dev/null @@ -1,50 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.functions; - -import edu.iu.dsc.tws.api.dataset.DataPartition; -import edu.iu.dsc.tws.api.tset.TSetContext; -import edu.iu.dsc.tws.api.tset.fn.SinkFunc; - -/** Sink Function that collects results. */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class Twister2SinkFunction implements SinkFunc { - - public Twister2SinkFunction() { - // non arg constructor needed for kryo - } - - @Override - public boolean add(T value) { - // TODO need to complete functionality if needed - return true; - } - - @Override - public void close() {} - - @Override - public DataPartition get() { - return null; - } - - @Override - public void prepare(TSetContext context) {} -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/internal/SystemReduceFnBuffering.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/internal/SystemReduceFnBuffering.java deleted file mode 100644 index f4abdfb4d938..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/internal/SystemReduceFnBuffering.java +++ /dev/null @@ -1,119 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.functions.internal; - -import java.io.ObjectStreamException; -import org.apache.beam.runners.core.MergingStateAccessor; -import org.apache.beam.runners.core.StateAccessor; -import org.apache.beam.runners.core.StateMerging; -import org.apache.beam.runners.core.StateTag; -import org.apache.beam.runners.core.StateTags; -import org.apache.beam.runners.core.SystemReduceFn; -import org.apache.beam.sdk.coders.Coder; -import org.apache.beam.sdk.state.BagState; -import org.apache.beam.sdk.state.ReadableState; -import org.apache.beam.sdk.transforms.windowing.BoundedWindow; -import org.apache.beam.sdk.util.SerializableUtils; - -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class SystemReduceFnBuffering - extends SystemReduceFn, Iterable, W> { - private static final String BUFFER_NAME = "buf"; - private transient StateTag> bufferTagLocal; - private transient Coder inputCoder; - - private transient boolean isInitialized = false; - - private byte[] coderBytes; - - public SystemReduceFnBuffering() { - super(null); - this.isInitialized = false; - } - - public SystemReduceFnBuffering(Coder valueCoder) { - super(null); - bufferTagLocal = null; - coderBytes = SerializableUtils.serializeToByteArray(valueCoder); - ; - - inputCoder = valueCoder; - } - - @Override - public void onMerge(OnMergeContext context) throws Exception { - initTransient(); - StateMerging.mergeBags(context.state(), bufferTagLocal); - } - - @Override - public void prefetchOnMerge(MergingStateAccessor state) throws Exception { - initTransient(); - StateMerging.prefetchBags(state, bufferTagLocal); - } - - @Override - public void processValue(ProcessValueContext c) throws Exception { - initTransient(); - c.state().access(bufferTagLocal).add(c.value()); - } - - @Override - public void prefetchOnTrigger(StateAccessor state) { - initTransient(); - state.access(bufferTagLocal).readLater(); - } - - @Override - public void onTrigger(OnTriggerContext c) throws Exception { - initTransient(); - c.output(c.state().access(bufferTagLocal).read()); - } - - @Override - public void clearState(Context c) throws Exception { - initTransient(); - c.state().access(bufferTagLocal).clear(); - } - - @Override - public ReadableState isEmpty(StateAccessor state) { - initTransient(); - return state.access(bufferTagLocal).isEmpty(); - } - - /** - * Method used to initialize the transient variables that were sent over as byte arrays or proto - * buffers. - */ - private void initTransient() { - if (isInitialized) { - return; - } - inputCoder = - (Coder) SerializableUtils.deserializeFromByteArray(coderBytes, "Custom Coder Bytes"); - bufferTagLocal = StateTags.makeSystemTagInternal(StateTags.bag(BUFFER_NAME, inputCoder)); - this.isInitialized = true; - } - - protected Object readResolve() throws ObjectStreamException { - return this; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/internal/package-info.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/internal/package-info.java deleted file mode 100644 index 7a775aec1034..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/internal/package-info.java +++ /dev/null @@ -1,20 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/** Internal implementation of the Beam runner for Twister2. */ -package org.apache.beam.runners.twister2.translators.functions.internal; diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/package-info.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/package-info.java deleted file mode 100644 index 061e8867d174..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/functions/package-info.java +++ /dev/null @@ -1,20 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/** Internal implementation of the Beam runner for Twister2. */ -package org.apache.beam.runners.twister2.translators.functions; diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/package-info.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/package-info.java deleted file mode 100644 index 32127c4ffa9b..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/package-info.java +++ /dev/null @@ -1,20 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/** Internal implementation of the Beam runner for Twister2. */ -package org.apache.beam.runners.twister2.translators; diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/streaming/ReadSourceTranslatorStream.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/streaming/ReadSourceTranslatorStream.java deleted file mode 100644 index fc2abb4d7ca8..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/streaming/ReadSourceTranslatorStream.java +++ /dev/null @@ -1,32 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.translators.streaming; - -import org.apache.beam.runners.twister2.Twister2StreamTranslationContext; -import org.apache.beam.runners.twister2.translators.StreamTransformTranslator; -import org.apache.beam.sdk.transforms.PTransform; -import org.apache.beam.sdk.values.PBegin; -import org.apache.beam.sdk.values.PCollection; - -/** doc. */ -public class ReadSourceTranslatorStream - implements StreamTransformTranslator>> { - @Override - public void translateNode( - PTransform> transform, Twister2StreamTranslationContext context) {} -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/streaming/package-info.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/streaming/package-info.java deleted file mode 100644 index 73b59dcf2239..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/translators/streaming/package-info.java +++ /dev/null @@ -1,20 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/** Internal implementation of the Beam runner for Twister2. */ -package org.apache.beam.runners.twister2.translators.streaming; diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/NoOpStepContext.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/NoOpStepContext.java deleted file mode 100644 index b3b19b0af0a0..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/NoOpStepContext.java +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.utils; - -import java.io.Serializable; -import org.apache.beam.runners.core.StateInternals; -import org.apache.beam.runners.core.StepContext; -import org.apache.beam.runners.core.TimerInternals; - -/** doc. */ -public class NoOpStepContext implements StepContext, Serializable { - @Override - public StateInternals stateInternals() { - throw new UnsupportedOperationException(); - } - - @Override - public TimerInternals timerInternals() { - throw new UnsupportedOperationException(); - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/TranslationUtils.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/TranslationUtils.java deleted file mode 100644 index faf1abc197de..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/TranslationUtils.java +++ /dev/null @@ -1,71 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.utils; - -import java.util.logging.Level; -import java.util.logging.Logger; -import org.apache.beam.sdk.coders.Coder; -import org.apache.beam.sdk.coders.CoderException; -import org.apache.beam.sdk.util.CoderUtils; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowedValues; - -/** doc. */ -@SuppressWarnings({ - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public final class TranslationUtils { - private static final Logger LOG = Logger.getLogger(TranslationUtils.class.getName()); - - private TranslationUtils() {} - - /** - * Utility method for deserializing a byte array using the specified coder. (From spark code) - * - * @param serialized bytearray to be deserialized. - * @param coder Coder to deserialize with. - * @param Type of object to be returned. - * @return Deserialized object. - */ - public static T fromByteArray(byte[] serialized, Coder coder) { - try { - return CoderUtils.decodeFromByteArray(coder, serialized); - } catch (CoderException e) { - LOG.log(Level.SEVERE, "Error while decoding message", e); - } - return null; - } - - /** - * Utility method for deserializing a byte array using the specified coder. (From spark code) - * - * @param Type of object to be returned. - * @param serialized bytearray to be deserialized. - * @param coder Coder to deserialize with. - * @return Deserialized object. - */ - public static WindowedValue fromByteArray( - byte[] serialized, WindowedValues.WindowedValueCoder coder) { - try { - return CoderUtils.decodeFromByteArray(coder, serialized); - } catch (CoderException e) { - LOG.log(Level.SEVERE, "Error while decoding message", e); - } - return null; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/Twister2AssignContext.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/Twister2AssignContext.java deleted file mode 100644 index 1155dded04d2..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/Twister2AssignContext.java +++ /dev/null @@ -1,52 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.utils; - -import org.apache.beam.sdk.transforms.windowing.BoundedWindow; -import org.apache.beam.sdk.transforms.windowing.WindowFn; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Iterables; -import org.joda.time.Instant; - -/** doc. */ -@SuppressWarnings({"keyfor", "nullness"}) // TODO(https://github.com/apache/beam/issues/20497) -public class Twister2AssignContext - extends WindowFn.AssignContext { - - private final WindowedValue value; - - public Twister2AssignContext(WindowFn fn, WindowedValue value) { - fn.super(); - this.value = value; - } - - @Override - public T element() { - return value.getValue(); - } - - @Override - public Instant timestamp() { - return value.getTimestamp(); - } - - @Override - public BoundedWindow window() { - return Iterables.getOnlyElement(value.getWindows()); - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/Twister2SideInputReader.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/Twister2SideInputReader.java deleted file mode 100644 index fdd36fe66979..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/Twister2SideInputReader.java +++ /dev/null @@ -1,155 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.beam.runners.twister2.utils; - -import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull; - -import edu.iu.dsc.tws.api.dataset.DataPartition; -import edu.iu.dsc.tws.api.dataset.DataPartitionConsumer; -import edu.iu.dsc.tws.api.tset.TSetContext; -import java.util.ArrayList; -import java.util.Collections; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.stream.Collectors; -import org.apache.beam.runners.core.InMemoryMultimapSideInputView; -import org.apache.beam.runners.core.SideInputReader; -import org.apache.beam.sdk.coders.Coder; -import org.apache.beam.sdk.coders.KvCoder; -import org.apache.beam.sdk.transforms.Materializations; -import org.apache.beam.sdk.transforms.Materializations.MultimapView; -import org.apache.beam.sdk.transforms.ViewFn; -import org.apache.beam.sdk.transforms.windowing.BoundedWindow; -import org.apache.beam.sdk.values.PCollectionView; -import org.apache.beam.sdk.values.TupleTag; -import org.apache.beam.sdk.values.WindowedValue; -import org.apache.beam.sdk.values.WindowingStrategy; -import org.checkerframework.checker.nullness.qual.Nullable; - -@SuppressWarnings({ - "rawtypes", // TODO(https://github.com/apache/beam/issues/20447) - "nullness" // TODO(https://github.com/apache/beam/issues/20497) -}) -public class Twister2SideInputReader implements SideInputReader { - - private final TSetContext runtimeContext; - private final Map, WindowingStrategy> sideInputs; - - public Twister2SideInputReader( - Map, WindowingStrategy> indexByView, TSetContext context) { - this.sideInputs = indexByView; - this.runtimeContext = context; - } - - @Override - public @Nullable T get(PCollectionView view, BoundedWindow window) { - checkNotNull(view, "View passed to sideInput cannot be null"); - TupleTag tag = view.getTagInternal(); - checkNotNull(sideInputs.get(tag), "Side input for %s not available.", view); - return getSideInput(view, window); - } - - @Override - public boolean contains(PCollectionView view) { - return sideInputs.containsKey(view.getTagInternal()); - } - - @Override - public boolean isEmpty() { - return false; - } - - private T getSideInput(PCollectionView view, BoundedWindow window) { - switch (view.getViewFn().getMaterialization().getUrn()) { - case Materializations.MULTIMAP_MATERIALIZATION_URN: - return getMultimapSideInput(view, window); - case Materializations.ITERABLE_MATERIALIZATION_URN: - return getIterableSideInput(view, window); - default: - throw new IllegalArgumentException( - "Unknown materialization type: " + view.getViewFn().getMaterialization().getUrn()); - } - } - - private T getMultimapSideInput(PCollectionView view, BoundedWindow window) { - Map>> partitionedElements = getPartitionedElements(view); - Map resultMap = new HashMap<>(); - - ViewFn viewFn = (ViewFn) view.getViewFn(); - for (Map.Entry>> elements : - partitionedElements.entrySet()) { - - Coder keyCoder = ((KvCoder) view.getCoderInternal()).getKeyCoder(); - resultMap.put( - elements.getKey(), - viewFn.apply( - InMemoryMultimapSideInputView.fromIterable( - keyCoder, - (Iterable) - elements.getValue().stream() - .map(WindowedValue::getValue) - .collect(Collectors.toList())))); - } - T result = resultMap.get(window); - if (result == null) { - result = viewFn.apply(InMemoryMultimapSideInputView.empty()); - } - return result; - } - - private Map>> getPartitionedElements( - PCollectionView view) { - Map>> partitionedElements = new HashMap<>(); - DataPartition sideInput = runtimeContext.getInput(view.getTagInternal().getId()); - DataPartitionConsumer dataPartitionConsumer = sideInput.getConsumer(); - while (dataPartitionConsumer.hasNext()) { - WindowedValue winValue = (WindowedValue) dataPartitionConsumer.next(); - for (BoundedWindow tbw : winValue.getWindows()) { - List> windowedValues = - partitionedElements.computeIfAbsent(tbw, k -> new ArrayList<>()); - windowedValues.add(winValue); - } - } - return partitionedElements; - } - - private T getIterableSideInput(PCollectionView view, BoundedWindow window) { - Map>> partitionedElements = getPartitionedElements(view); - - ViewFn viewFn = - (ViewFn) view.getViewFn(); - Map resultMap = new HashMap<>(); - - for (Map.Entry>> elements : - partitionedElements.entrySet()) { - resultMap.put( - elements.getKey(), - viewFn.apply( - () -> - elements.getValue().stream() - .map(WindowedValue::getValue) - .collect(Collectors.toList()))); - } - T result = resultMap.get(window); - if (result == null) { - result = viewFn.apply(() -> Collections.emptyList()); - } - return result; - } -} diff --git a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/package-info.java b/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/package-info.java deleted file mode 100644 index 794c83b87c2f..000000000000 --- a/runners/twister2/src/main/java/org/apache/beam/runners/twister2/utils/package-info.java +++ /dev/null @@ -1,20 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/** Internal implementation of the Beam runner for Twister2. */ -package org.apache.beam.runners.twister2.utils; diff --git a/scripts/beam-sql.sh b/scripts/beam-sql.sh index 3e48098fb2b9..40e6e5f66dec 100755 --- a/scripts/beam-sql.sh +++ b/scripts/beam-sql.sh @@ -233,7 +233,6 @@ function list_runners() { echo " flink - FlinkRunner (runs on Apache Flink)" echo " spark - SparkRunner (runs on Apache Spark)" echo " jet - JetRunner (runs on Hazelcast Jet)" - echo " twister2 - Twister2Runner (runs on Twister2)" echo "" echo "💡 Usage: ./beam-sql.sh --runner " echo " Default: direct" @@ -277,10 +276,6 @@ function list_runners() { echo " jet - JetRunner" echo " Runs on Hazelcast Jet." ;; - "twister2") - echo " twister2 - Twister2Runner" - echo " Runs on Twister2." - ;; "apex") echo " apex - ApexRunner" echo " Runs on Apache Apex." diff --git a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml index e734e98e7d40..ecb0853648c6 100644 --- a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml +++ b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml @@ -261,43 +261,6 @@ - - twister2-runner - - - org.apache.beam - beam-runners-twister2 - ${beam.version} - runtime - - - - - nemo-runner - - - org.apache.nemo - nemo-compiler-frontend-beam - ${nemo.version} - - - org.apache.hadoop - hadoop-common - ${hadoop.version} - - - org.slf4j - slf4j-api - - - org.slf4j - slf4j-log4j12 - - - - - - jet-runner diff --git a/sdks/java/maven-archetypes/gcp-bom-examples/src/main/resources/archetype-resources/pom.xml b/sdks/java/maven-archetypes/gcp-bom-examples/src/main/resources/archetype-resources/pom.xml index d93a6b09284f..922f48f12e94 100644 --- a/sdks/java/maven-archetypes/gcp-bom-examples/src/main/resources/archetype-resources/pom.xml +++ b/sdks/java/maven-archetypes/gcp-bom-examples/src/main/resources/archetype-resources/pom.xml @@ -253,42 +253,6 @@ - - twister2-runner - - - org.apache.beam - beam-runners-twister2 - runtime - - - - - nemo-runner - - - org.apache.nemo - nemo-compiler-frontend-beam - ${nemo.version} - - - org.apache.hadoop - hadoop-common - ${hadoop.version} - - - org.slf4j - slf4j-api - - - org.slf4j - slf4j-log4j12 - - - - - - jet-runner diff --git a/settings.gradle.kts b/settings.gradle.kts index b45c3f9ffa4c..612f99259040 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -146,7 +146,6 @@ for (version in flink_versions.split(',')) { include(":runners:flink:${version}:job-server-container") } /* End Flink Runner related settings */ -include(":runners:twister2") include(":runners:google-cloud-dataflow-java") include(":runners:google-cloud-dataflow-java:arm") include(":runners:google-cloud-dataflow-java:examples") diff --git a/website/www/site/content/en/documentation/runners/twister2.md b/website/www/site/content/en/documentation/runners/twister2.md index 193871a39195..3eed212b28a3 100644 --- a/website/www/site/content/en/documentation/runners/twister2.md +++ b/website/www/site/content/en/documentation/runners/twister2.md @@ -20,7 +20,7 @@ limitations under the License. ## Overview -**Note** Twister2 runner is deprecated and the support is planned to be removed in Beam 3.0 ([Issue](https://github.com/apache/beam/issues/35905)). +**Note** Twister2 runner support has been removed in Beam 2.78.0 ([Issue](https://github.com/apache/beam/issues/35905)). Twister2 Runner can be used to execute Apache Beam pipelines on top of a Twister2 cluster. Twister2 Runner runs Beam pipelines as Twister2 jobs, which can be executed on @@ -35,9 +35,6 @@ require high performance, and provide. Bulk Synchronous Parallel (BSP) style execution. * Native support for Beam side-inputs. -The [Beam Capability Matrix](/documentation/runners/capability-matrix/) documents the -supported capabilities of the Twister2 Runner. - ## Running WordCount with the Twister2 Runner ### Generating the Beam examples project diff --git a/website/www/site/content/en/get-started/beam-overview.md b/website/www/site/content/en/get-started/beam-overview.md index c50ebdc39b54..aefad5fa4b2c 100644 --- a/website/www/site/content/en/get-started/beam-overview.md +++ b/website/www/site/content/en/get-started/beam-overview.md @@ -49,15 +49,15 @@ Beam currently supports the following runners: - [Direct Runner](/documentation/runners/direct) - [Apache Flink Runner](/documentation/runners/flink) Apache Flink logo -- [Apache Nemo Runner](/documentation/runners/nemo) - [Apache Spark Runner](/documentation/runners/spark) Apache Spark logo - [Google Cloud Dataflow Runner](/documentation/runners/dataflow) Google Cloud Dataflow logo - [Hazelcast Jet Runner](/documentation/runners/jet) Hazelcast Jet logo -- [Twister2 Runner](/documentation/runners/twister2) Twister2 logo Runners supported in older Beam versions - [Apache Samza Runner](/documentation/runners/samza) Apache Samza logo until Beam 2.74.0. +- [Twister2 Runner](/documentation/runners/twister2) Twister2 logo +- [Apache Nemo Runner](/documentation/runners/nemo) **Note:** You can always execute your pipeline locally for testing and debugging purposes. diff --git a/website/www/site/content/en/roadmap/_index.md b/website/www/site/content/en/roadmap/_index.md index a9676da1ee5c..d59ccc60fba4 100644 --- a/website/www/site/content/en/roadmap/_index.md +++ b/website/www/site/content/en/roadmap/_index.md @@ -32,8 +32,7 @@ Below are some highlights for the project as a whole. Portability is the primary Beam vision: running pipelines authored with _any SDK_ on _any runner_. This is a cross-cutting effort across Java, Python, and Go, and every Beam runner. Portability is currently supported on the -[DataFlow](/documentation/runners/dataflow), [Flink](/documentation/runners/flink/), [Jet](/documentation/runners/jet), [Nemo](/documentation/runners/nemo), [Prism](/documentation/runners/prism/), [Spark](/documentation/runners/spark/), and [Twister2](/documentation/runners/Twister2) -runners. +[DataFlow](/documentation/runners/dataflow), [Flink](/documentation/runners/flink/), [Jet](/documentation/runners/jet), [Prism](/documentation/runners/prism/), and [Spark](/documentation/runners/spark/) runners. See the details on the [Portability Roadmap](/roadmap/portability/) diff --git a/website/www/site/layouts/partials/section-menu/en/runners.html b/website/www/site/layouts/partials/section-menu/en/runners.html index b579c9a41294..50668897af83 100644 --- a/website/www/site/layouts/partials/section-menu/en/runners.html +++ b/website/www/site/layouts/partials/section-menu/en/runners.html @@ -16,9 +16,7 @@

  • Prism Runner
  • Apache Flink
  • FlareDB Runner
    • -
  • Apache Nemo
  • Apache Spark
  • Google Cloud Dataflow
  • Hazelcast Jet
  • Kafka Streams
    • -
  • Twister2