[POC] Split acceptance tests by SDK artifact dependency - #2422
Draft
corymhall wants to merge 1 commit into
Draft
Conversation
Summary: - add opt-in acceptance test suites with independent roots and shard counts - let provider-only suites start without generated SDK artifacts - fan out suites consistently across acceptance, main, nightly, and release workflows while preserving legacy output Rationale: - provider/runtime tests that do not consume generated SDKs should not wait for the full SDK build matrix - explicit suite roots keep selection based on artifact dependencies rather than language build tags Tests: - cd provider-ci && make clean && make all
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
POC: This draft is intended to validate the configuration model and
generated workflow graph. It is not ready for provider rollout as-is.
Generated acceptance tests currently use one reusable
test.ymljob. Shardedproviders therefore wait for every generated SDK build before any acceptance
test starts, even when part of the suite only needs the provider binary and
Pulumi CLI.
This POC adds an opt-in
acceptanceTestSuitesconfiguration:Each suite gets its own reusable-workflow call and shard matrix. A
provider-only suite depends on
prerequisitesandbuild_provider; anSDK-dependent suite additionally depends on
build_sdk. Sentinel and publishjobs wait for every configured suite and the SDK build.
The same fanout is generated for acceptance, main, nightly, prerelease, and
release workflows. Inside
test.yml, generated SDK downloads, Make progressrestoration, Python SDK tooling, and
make install_sdksonly run when the suitesets
requiresSDKs: true. Provider download, ESC credentials, and shared testsetup remain available to both kinds of suite.
Existing providers retain the current
shardsor language-matrix behavior whenacceptanceTestSuitesis absent. Native workflow templates reject the optionrather than silently ignoring it.
Some deliberate POC constraints:
test-folder, non-overlapping, and shell-safe;and runs package initialization and
TestMain;examples_test.gois not currently a deployable SDK-smoke root in pulumi-aws;real adoption must configure the test-bearing file or directory created for
that migration.
The main review question is whether this suite/root/
requiresSDKsmodel is theright provider-facing API before investing in a production rollout.
Validation performed:
This includes Go tests, deterministic fixture regeneration, golangci-lint, and
actionlint across all generated test providers. Only the opted-in AWS fixture
changes; legacy fixtures remain unchanged.