feat: Add vertical autoscaling configuration for stream processors - #4665
Conversation
| db = "exampleDb" | ||
| } | ||
| autoscaling = { | ||
| enabled = true |
There was a problem hiding this comment.
One question on the Terraform shape: do we need to expose enabled in the autoscaling block? I know in the design/td, we have enabled to track if autoscaling is enabled or disabled. But given the API Contract doc scenario, when an autoscaling config is disabled, the whole object is omitted from the response and subsequent GET calls for the stream processor.
Based on the API behavior we tested in Cloud Dev, the presence of options.autoscaling enables autoscaling, and removing the block disables it—the provider sends enabled: false on the PATCH, and the backend clears the configuration. Since enabled = false cannot round-trip and we reject it today, enabled seems redundant as a Terraform argument.
Could we instead make the schema:
options = {
autoscaling = {
min_tier = "SP10"
max_tier = "SP50"
}
}Then the provider can continue translating block presence/removal to enabled: true / enabled: false in the API request. This feels more idiomatic for Terraform and avoids requiring users to set a field that only has one valid value.
There was a problem hiding this comment.
enabled seems redundant as a Terraform argument
I agree here. I am curious why we didnt apply the same judgement at the API level, why does autoScaling.enabled actually exist?
There was a problem hiding this comment.
This is for the PATCH behavior, excluding the autoscaling config in a PATCH API does not mean that it will disable it since omission means to preserve the existing config for the PATCH contract.
So we needed a way to disable the autoscaling field with the enabled field, for terraform we can simplify this with just the config.
There was a problem hiding this comment.
are we okay with this terraform behavior to exclude enabled and have the existence of the Autoscaling config be the enabling factor for terraform @AgustinBettati. If yes, I can make the change now.
EX:
removing autoscaling config from an existing stream processor will disable autoscaling.
There was a problem hiding this comment.
I did this cleanup, let me know if this should be reverted to add back enabled to the autoscaling field. I believe this change to remove the enabled field is a better user experience.
There was a problem hiding this comment.
I am aligned, lets make sure to capture this decision in associated TD for future reference.
In terms of the PATCH behaviour on the API side, one option we could have used (not sure if too late now) is that sending an explicit null (e.g. autoScaling = null) in the PATCH is the way of signalling we want to unset the property (https://mongodb.github.io/ipa/107#guidance point 5).
There was a problem hiding this comment.
I updated and tagged you in the updated TD reference to the terraform config: https://docs.google.com/document/d/1q5mmlV63bEk006CXpDEWM3nXDa41USzgvE_nvHkZGGs/edit?disco=AAACF73K4vg
There was a problem hiding this comment.
I also made a post in the thread regarding the level of effort needed to make autoscaling = null before the release this Thursday here.
There was a problem hiding this comment.
As stated in this slack thread, I updated the provider to use an explicit autoscaling: null PATCH when the autoscaling block is removed from an existing Terraform-managed processor.
I verified this in Cloud Dev with provider debug logging. The provider sent:
{
"name": "autoscaling-e2e-processor",
"options": {
"autoscaling": null
},
"pipeline": [
{
"$source": {
"connectionName": "sample_stream_solar"
}
},
{
"$emit": {
"connectionName": "__testLog"
}
}
],
"tier": "SP10"
}Cloud Dev accepted the update and cleared autoscaling.
This keeps the Terraform behavior declarative, presence of the autoscaling block enables it, while removing it disables it without exposing the API transport-level enabled field in Terraform configuration.
AgustinBettati
left a comment
AgustinBettati
left a comment
There was a problem hiding this comment.
Providing initial review
| db = "exampleDb" | ||
| } | ||
| autoscaling = { | ||
| enabled = true |
There was a problem hiding this comment.
enabled seems redundant as a Terraform argument
I agree here. I am curious why we didnt apply the same judgement at the API level, why does autoScaling.enabled actually exist?
Extend mongodbatlas_stream_processor and its data sources with the Phase 1
vertical autoscaling config: options.autoscaling { enabled, min_tier, max_tier }
and the read-only effective_tier attribute.
- Schema, model, and conversions with PATCH tri-state semantics (omit=preserve,
explicit false/block-removal=disable+clear); startWith carries autoscaling top-level.
- Plan-time validator rejecting min_tier/max_tier when enabled=false.
- Unit tests (conversions, effective_tier, create/update reqs) and acceptance
tests (create/disable, validator).
- Regenerated resource + data source docs and updated example.
WIP: the autoscaling API fields are still @hidden, so this builds against a local
SDK stub (.sdkstub/ + a go.mod replace kept local, not committed). Not releasable
until the official SDK ships the fields.
… enabled=false The backend clears a disabled autoscaling config, so a GET after disable returns no autoscaling object. An explicit enabled=false therefore cannot round-trip and produces a 'provider produced inconsistent result after apply' error. Reject enabled=false at plan time with guidance to remove the options.autoscaling block instead; the provider still sends the explicit disable to the API when the block is removed. Update schema descriptions, docs, and the validator/acceptance test accordingly.
56407e1 to
c17c42a
Compare
|
APIx bot: a message has been sent to Docs Slack channel |
🤖 Augment PR SummarySummary: Adds Phase 1 vertical autoscaling support for stream processors.
🤖 Was this summary useful? React with 👍 or 👎 |
![augmentcode[bot]](https://avatars.githubusercontent.com/in/1027498?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Adds Phase 1 vertical autoscaling support for mongodbatlas_stream_processor (resource + singular/plural data sources) by introducing an optional options.autoscaling configuration block, a computed effective_tier read-only attribute, and update semantics to explicitly clear autoscaling via PATCH when the block is removed.
Changes:
- Adds
options.autoscaling(with server-defaultedmin_tier/max_tier) and introduces computedeffective_tieracross resource and data sources. - Implements tri-state update behavior to preserve/clear autoscaling and DLQ independently, including explicit autoscaling nulling on removal.
- Updates tests, docs, examples, and changelog entries to cover and document autoscaling behavior and the new attribute.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| internal/service/streamprocessor/resource.go | Sends autoscaling at top-level for :startWith on create/update start flows; passes prior state into update request builder. |
| internal/service/streamprocessor/resource_test.go | Adds acceptance coverage for enabling autoscaling and disabling it by removing the block. |
| internal/service/streamprocessor/resource_schema.go | Adds options.autoscaling, makes options.dlq optional, adds effective_tier, and enforces non-empty options. |
| internal/service/streamprocessor/options_validator.go | Adds validator to reject explicitly empty options objects. |
| internal/service/streamprocessor/options_validator_test.go | Unit tests for the new options validator. |
| internal/service/streamprocessor/model.go | Adds autoscaling/DLQ update tri-state resolution, adds effective tier mapping, and adds autoscaling conversions. |
| internal/service/streamprocessor/model_test.go | Extends unit coverage for autoscaling conversions, effective tier, and update PATCH semantics. |
| internal/service/streamprocessor/dlq.go | Extracts DLQ plan/state resolution into helpers for update tri-state semantics. |
| internal/service/streamprocessor/autoscaling.go | Adds autoscaling TF model, conversions, plan extraction, and update tri-state resolution. |
| examples/mongodbatlas_stream_processor/main.tf | Updates example to include baseline tier and options.autoscaling. |
| docs/resources/stream_processor.md | Documents options.autoscaling, effective_tier, and the non-empty options constraint. |
| docs/data-sources/stream_processors.md | Documents options.autoscaling and effective_tier in plural data source output. |
| docs/data-sources/stream_processor.md | Documents options.autoscaling and effective_tier in singular data source output. |
| .changelog/4665.txt | Adds release-note entries for the resource and both data sources. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| func ConvertOptionsToTF(ctx context.Context, options *admin.StreamsOptions) (*types.Object, diag.Diagnostics) { | ||
| if options == nil || !options.HasDlq() { | ||
| if options == nil || (!options.HasDlq() && options.Autoscaling == nil) { | ||
| return new(types.ObjectNull(OptionsObjectType.AttributeTypes())), nil | ||
| } |
There was a problem hiding this comment.
This compiles as written. The repository uses Go 1.26.4 (go.mod), where new accepts either a type or an expression. types.ObjectNull(...) returns a types.Object, so this produces the required *types.Object.
Verified locally with:
go test ./internal/service/streamprocessor -run '^$' -count=1
make build
| - `failover_enabled` (Boolean) Indicates whether this stream processor is eligible for failover. When `true`, an operator can trigger a failover event to migrate the stream processor to a secondary region configured in the workspace's `failover_regions`. Requires an Atlas-to-Atlas or Atlas-to-Kafka pipeline with `failover_regions` configured on the workspace. | ||
| - `id` (String) Unique 24-hexadecimal character string that identifies the stream processor. | ||
| - `options` (Attributes) Optional configuration for the stream processor. (see [below for nested schema](#nestedatt--options)) | ||
| - `options` (Attributes) Optional configuration for the stream processor. Empty `options` objects are not supported. (see [below for nested schema](#nestedatt--options)) |
There was a problem hiding this comment.
@AgustinBettati
I made dlq optional so it can be configured independently from autoscaling, which matches the MMS API contract.
This exposed a removal transition that the previous schema could not represent: historically, dlq was required whenever options was present, so Terraform could never plan a processor with options but no DLQ.
For PATCH, MMS/SPM treats an omitted DLQ as “preserve the existing DLQ” and an empty DLQ object (dlq: {}) as the explicit remove signal. The provider now translates removal of a configured Terraform DLQ into that empty-object PATCH payload.
I also added validation that rejects options = {}. If removing DLQ leaves no other option configured, users should omit options entirely. This keeps the Terraform state shape aligned with the API response, which omits options when neither DLQ nor autoscaling is configured.
There was a problem hiding this comment.
Approach makes sense.
an empty DLQ object (dlq: {}) as the explicit remove signal
Following the sibling autoScaling property, I would be consistent in allowing dlq = null as the removal signal in the API side. Also aligning to https://mongodb.github.io/ipa/107#guidance. (not urgent but for considering)
There was a problem hiding this comment.
Created this ticket for DLQ unsetting behavior https://jira.mongodb.org/browse/CLOUDP-439044
| } | ||
| // Block presence is Terraform's enablement signal. The API requires this | ||
| // transport field for create/PATCH semantics, but it is not exposed in state. | ||
| req := &admin.StreamsAutoscaling{Enabled: new(true)} |
There was a problem hiding this comment.
unfortunately, enabled=true needs to be sent to the API for autoscaling to work so this logic needs to stay.
| if plan.Tier.ValueString() != "" { | ||
| startWithOptions.SetTier(plan.Tier.ValueString()) | ||
| } | ||
| // On the :startWith endpoint, `autoscaling` is TOP-LEVEL (no options wrapper). |
There was a problem hiding this comment.
this is from a convenience endpoint from the adminAPI where the terraform provider calls startsWithOptions if the state=started is used in the stream processor.
5beb161 to
12e317c
Compare
| } | ||
|
|
||
| func TestAccStreamProcessor_withAutoscaling(t *testing.T) { | ||
| func TestAccStreamProcessor_withOptionsDLQAutoscaling(t *testing.T) { |
There was a problem hiding this comment.
this now tests the options change with DLQ unsetting and setting with autoscaling setting and unsetting, also tests :startsWithOptions with autoscaling
AgustinBettati
left a comment
AgustinBettati
left a comment
There was a problem hiding this comment.
LGTM, thanks for the follow ups here
Summary
Adds Phase 1 vertical autoscaling support to
mongodbatlas_stream_processorand its singular and plural data sources:options.autoscalingblock with optional server-defaultedmin_tierandmax_tiereffective_tier, which reports the processor's current running tiertierremains the user-configured baseline tier; it does not autoscale or driftoptions.dlqandoptions.autoscalingare independently optional; an explicitly emptyoptionsobject is rejectedIncluded
options.autoscalingsends an explicitoptions.autoscaling: nullPATCH. If autoscaling is the only option, omitoptionsentirely; if a DLQ remains, remove onlyoptions.autoscaling.:startWith, where autoscaling is top-level rather than nested inoptions.effective_tier, create/update requests, optional DLQ, block-removal disablement, server-default bounds, and empty-options validation.Validation
Targeted unit tests and provider verification pass locally.
Cloud Dev E2E validation confirmed the explicit PATCH payload:
{ "options": { "autoscaling": null } }Cloud Dev accepted the update and cleared autoscaling.
Draft status
This PR remains a draft while the final review and full CI run complete.
Expected Terraform release: August 27, 2026.