Split CI into parallel module matrix - #5
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
Codex Review: Here are some suggestions.
Reply with @codex fix comments to fix any unresolved comments.
About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".
| build: | ||
| strategy: | ||
| matrix: | ||
| include: | ||
| - module: core | ||
| commands: > | ||
| scalafmtCheckAll scalafixAll core/test compile-fail-tests/test | ||
| paths: | | ||
| target | ||
| modules/core/target | ||
| modules/compile-fail-tests/target | ||
| - module: connectors | ||
| commands: > | ||
| connectors/test connectors-gcs/test | ||
| paths: | | ||
| target | ||
| modules/connectors/target | ||
| modules/connectors-gcs/target | ||
| - module: engines | ||
| commands: > | ||
| engines-spark/test engines-flink/test "engines-spark/testOnly *StreamingCDCSpec" "engines-flink/testOnly *EngineAbstractionSpec" | ||
| paths: | | ||
| target | ||
| modules/engines-spark/target | ||
| modules/engines-flink/target |
There was a problem hiding this comment.
[P1] Ensure all modules are built in CI matrix
The new matrix job only runs commands for core, connectors, and engines, but the repository still contains other modules such as contracts, infrastructure, quality-deequ, and the CLI subprojects. Because the workflow no longer performs a global sbt compile Test/compile, any change in those omitted modules will merge without even compiling or running their tests. Consider adding the remaining modules to the matrix or restoring a catch‑all compile step so CI continues to guard the whole build.
Useful? React with 👍 / 👎.
| ~/.ivy2/cache | ||
| ~/.cache/coursier | ||
| ~/.sbt | ||
| key: ${{ runner.os }}-sbt-${{ hashFiles('**/build.sbt', '**/project/**/*.sbt', '**/project/**/Dependencies.scala') }} | ||
| ${{ matrix.paths }} | ||
| key: ${{ runner.os }}-sbt-${{ matrix.module }}-${{ hashFiles('**/build.sbt', '**/project/**/*.sbt', format('modules/{0}/**', matrix.module)) }} | ||
| restore-keys: | | ||
| ${{ runner.os }}-sbt- | ||
|
|
||
| - name: Check formatting and linting | ||
| run: sbt scalafmtCheckAll scalafixAll | ||
|
|
||
| - name: Compile all modules | ||
| run: sbt compile Test/compile | ||
|
|
||
| - name: Run tests with coverage | ||
| run: | | ||
| sbt coverage test | ||
| sbt coverageReport | ||
| sbt coverageAggregate | ||
|
|
||
| - name: Upload coverage reports to Codecov | ||
| uses: codecov/codecov-action@v4 | ||
| with: | ||
| file: target/scala-2.13/scoverage-report/scoverage.xml | ||
| token: ${{ secrets.CODECOV_TOKEN }} | ||
| fail_ci_if_error: false | ||
| ${{ runner.os }}-sbt-${{ matrix.module }}- | ||
|
|
||
| - name: Run compile-fail tests | ||
| run: | | ||
| echo "🔥 Running compile-fail tests - proving FlowForge's core USP" | ||
| sbt "compile-fail-tests / test" | ||
| echo "✅ Compile-fail tests passed - contract drift prevention verified!" | ||
|
|
||
| - name: Spark smoke test | ||
| run: | | ||
| echo "🚀 Running Spark local smoke test" | ||
| sbt "engines-spark/testOnly *StreamingCDCSpec" | ||
| echo "✅ Spark smoke test passed!" | ||
|
|
||
| - name: Flink smoke test | ||
| run: | | ||
| echo "🌊 Running Flink smoke test" | ||
| sbt "engines-flink/testOnly *EngineAbstractionSpec" | ||
| echo "✅ Flink smoke test passed!" | ||
|
|
||
| - name: Test g8 template generation | ||
| run: | | ||
| echo "📋 Testing g8 template generation" | ||
| cd /tmp | ||
| sbt new file://${{ github.workspace }}/flowforge.g8 --name="ci-test-pipeline" --organization="com.flowforge.ci" | ||
| cd ci-test-pipeline | ||
| echo "🔨 Testing generated project compilation" | ||
| sbt compile | ||
| echo "🧪 Testing generated project tests" | ||
| sbt test | ||
| echo "✅ G8 template test passed - turnkey experience verified!" | ||
| - name: Build module | ||
| run: sbt -batch ${{ matrix.commands }} |
There was a problem hiding this comment.
[P1] Restore code coverage and Codecov upload
The previous workflow generated scoverage data (sbt coverage test followed by coverageReport/coverageAggregate) and uploaded it to Codecov. The refactored job now ends after a single sbt -batch … invocation with no coverage instrumentation or upload, so coverage metrics will stop being produced and Codecov will not receive updates. If coverage reporting is still required, reintroduce these steps—either per matrix job or in a dedicated job—before merging.
Useful? React with 👍 / 👎.
|
@codex fix comments |
Summary
Testing
|
1 participant
Summary
Testing
sbt -batch fmtsbt -batch compile Test/compilehttps://chatgpt.com/codex/tasks/task_e_68c047d3a5f88325a410d2b41327a73c