Skip to content

[POC] Split acceptance tests by SDK artifact dependency - #2422

Draft
corymhall wants to merge 1 commit into
masterfrom
sdk-independent-tests
Draft

[POC] Split acceptance tests by SDK artifact dependency#2422
corymhall wants to merge 1 commit into
masterfrom
sdk-independent-tests

Conversation

@corymhall

Copy link
Copy Markdown
Member

Important

POC: This draft is intended to validate the configuration model and
generated workflow graph. It is not ready for provider rollout as-is.

Generated acceptance tests currently use one reusable test.yml job. Sharded
providers therefore wait for every generated SDK build before any acceptance
test starts, even when part of the suite only needs the provider binary and
Pulumi CLI.

This POC adds an opt-in acceptanceTestSuites configuration:

acceptanceTestSuites:
  - name: yaml
    root: examples_yaml_test.go
    shards: 8
    requiresSDKs: false
  - name: sdk
    root: examples_test.go
    shards: 2
    requiresSDKs: true

Each suite gets its own reusable-workflow call and shard matrix. A
provider-only suite depends on prerequisites and build_provider; an
SDK-dependent suite additionally depends on build_sdk. Sentinel and publish
jobs wait for every configured suite and the SDK build.

The same fanout is generated for acceptance, main, nightly, prerelease, and
release workflows. Inside test.yml, generated SDK downloads, Make progress
restoration, Python SDK tooling, and make install_sdks only run when the suite
sets requiresSDKs: true. Provider download, ESC credentials, and shared test
setup remain available to both kinds of suite.

Existing providers retain the current shards or language-matrix behavior when
acceptanceTestSuites is absent. Native workflow templates reject the option
rather than silently ignoring it.

Some deliberate POC constraints:

  • suite roots are relative to test-folder, non-overlapping, and shell-safe;
  • each root must contain at least as many top-level tests as its shard count;
  • file roots select top-level tests, but Go still compiles sibling package files
    and runs package initialization and TestMain;
  • providers are responsible for arranging roots that cover every intended test;
  • the AWS test-provider configuration is illustrative only. In particular,
    examples_test.go is not currently a deployable SDK-smoke root in pulumi-aws;
    real adoption must configure the test-bearing file or directory created for
    that migration.

The main review question is whether this suite/root/requiresSDKs model is the
right provider-facing API before investing in a production rollout.

Validation performed:

cd provider-ci && make clean && make all

This includes Go tests, deterministic fixture regeneration, golangci-lint, and
actionlint across all generated test providers. Only the opted-in AWS fixture
changes; legacy fixtures remain unchanged.

Summary:
- add opt-in acceptance test suites with independent roots and shard counts
- let provider-only suites start without generated SDK artifacts
- fan out suites consistently across acceptance, main, nightly, and release
  workflows while preserving legacy output

Rationale:
- provider/runtime tests that do not consume generated SDKs should not wait
  for the full SDK build matrix
- explicit suite roots keep selection based on artifact dependencies rather
  than language build tags

Tests:
- cd provider-ci && make clean && make all
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant