[Bug] elasticstack_fleet_integration_policy fails with inconsistent result after apply

[wasfree]

Bug: elasticstack_fleet_integration_policy fails with Provider produced inconsistent result after apply for sensitive inputs

Summary

Applying an elasticstack_fleet_integration_policy for azure_blob_storage fails with:

• Provider produced inconsistent result after apply
• .inputs: inconsistent values for sensitive attribute

This appears to be a provider-side state/semantic equality issue for sensitive inputs values.

Environment

• Terraform CLI: (run locally, likely 1.8+ in this workspace)
• Provider: registry.terraform.io/elastic/elasticstack 0.16.3
• OS: macOS (darwin arm64)

From terraform providers:

• module.main -> provider[registry.terraform.io/elastic/elasticstack] 0.16.3

Failing resource address

module.main.module.fleet_integrations["azure_blob_storage"].elasticstack_fleet_integration_policy.policy["azure_blob_storage_policy"]

Error output

Error: Provider produced inconsistent result after apply

When applying changes to
module.main.module.fleet_integrations["azure_blob_storage"].elasticstack_fleet_integration_policy.policy["azure_blob_storage_policy"],
provider "module.main.provider[\"registry.terraform.io/elastic/elasticstack\"]"
produced an unexpected new value: .inputs: inconsistent values for sensitive
attribute.

Minimal repro config (extracted)

resource "elasticstack_fleet_integration_policy" "policy" {
  name                = "azure_blob_storage_policy_integration@azure_blob_storage"
  namespace           = "example.nonprod"
  description         = "Azure Blob Storage integration policy for validation testing"
  integration_name    = "azure_blob_storage"
  integration_version = "2.3.0"
  agent_policy_ids    = ["<existing-agent-policy-id>"]

  inputs = {
    "azure-blob-storage-azure-blob-storage" = {
      enabled = true
      streams = {
        "azure_blob_storage.azure-blob-storage" = {
          enabled = true
          vars = {
            oauth2                  = false
            account_name            = "stpltopsvalone"
            service_account_key     = "<sensitive>"
            "data_stream.dataset"   = "azure_blob_storage.validation"
            number_of_workers       = 3
            poll                    = true
            poll_interval           = "15s"
            containers              = "- name: test\n"
            file_selectors          = "# - regex: \"event/\"\n"
            preserve_original_event = false
            tags = ["forwarded", "azure_blob_storage-validation"]
          }
        }
      }
    }
  }
}

Reproduction steps

1. Define elasticstack_fleet_integration_policy with inputs like above (including sensitive service_account_key in stream vars).
2. Run apply (targeted or full):
   • terraform apply -auto-approve -target='module.main.module.fleet_integrations["azure_blob_storage"].elasticstack_fleet_integration_policy.policy["azure_blob_storage_policy"]'
3. Resource create starts, then provider returns inconsistent result error on .inputs.

Observed behavior

• Provider fails post-apply reconciliation with .inputs: inconsistent values for sensitive attribute.
• Terraform cannot expose value-level old/new for sensitive attributes, only path-level mismatch.

Expected behavior

• Provider should either:
  • return state exactly matching planned inputs, or
  • perform semantic equality that safely handles sensitive maps without reporting inconsistency.

Trace evidence (TF_LOG=TRACE)

From /tmp/tf-elasticstack-trace.log:

• Provider validates and manipulates nested input path:
  • inputs["azure-blob-storage-azure-blob-storage"].streams["azure_blob_storage.azure-blob-storage"].vars
• Provider applies defaults on nested key:
  • setting attribute inputs["azure-blob-storage-azure-blob-storage"].defaults to default value: <null>
• Plan modifier invoked on inputs:
  • Calling provider defined planmodifier.Map ... description="Once set, the value of this attribute in state will not change."
• Apply ultimately fails with inconsistent sensitive attribute for .inputs.

Additional notes

• Earlier payload issues were resolved (service_account_key missing, wrong input key name). Current failure remains after valid input shape and key.
• This appears specific to elasticstack_fleet_integration_policy path with sensitive inputs.

Workaround currently used

Manage azure_blob_storage via the older agent_policies.integrations path instead of fleet_integrations until provider fix is available.

[github-actions]

✅ Reproducer Factory Issue Intake completed successfully!

[github-actions]

Bug reproduction

Run at 2026-08-05T00:00:00Z — workflow run. This comment is replaced on each workflow run.

Summary

I could not build a verified reproduction of .inputs: inconsistent values for sensitive attribute for elasticstack_fleet_integration_policy this run either, for the same structural reason as the prior run plus one new confirmation:

1. No live Elastic Stack was reachable from this environment. host.docker.internal:9201/:5602 failed DNS resolution (Could not resolve host), es-proxy/kb-proxy are likewise unresolvable, and the local Docker daemon itself is unreachable (Cannot connect to the Docker daemon at (redacted) Outbound network to a small allowlist (GitHub, the Go module proxy, etc. — see /etc/hosts) does work, so this is a scoped absence of the Fleet/Kibana stack in this sandbox, not a total network outage. Without it, no acceptance test (TF_ACC=1`) can be executed to confirm or refute the failure against a real Fleet API.
2. go build ./... succeeds cleanly on main (module downloads resolved fine), so I used the available compute to go one level deeper into the framework mechanics than the previous run, detailed below. That deeper look sharpens the previous run's investigation rather than replacing it — the two open questions it raised are both still open.

I'm again not attempting to land TestAccReproduceIssue4329 this run rather than commit a test that has never executed against a real backend.

Investigation avenues

1. github.com/hashicorp/terraform-plugin-framework@v1.19.0 internal/fwserver/server_createresource.go:183-199 (pinned in go.mod; mirror: https://github.com/hashicorp/terraform-plugin-framework/blob/v1.19.0/internal/fwserver/server_createresource.go#L183-L199) — this confirms the exact mechanism the previous run's avenue [ci] add jobs to run the acceptance tests #1 could only speculate about: SchemaSemanticEquality runs inside the provider's Create/Update RPC handling, comparing req.PlannedState against the Create() function's resp.NewState using each attribute type's SemanticEquals implementation, and — if equal — silently rewrites resp.NewState to the planned value before Terraform core ever runs its own consistency check. That means policyshape.InputsValue.MapSemanticEquals (internal/fleet/policyshape/inputs_value.go:64) → InputValue.ObjectSemanticEquals (internal/fleet/policyshape/input_value.go:179) → compareStreams/stripServerManagedVarsKeys (internal/fleet/policyshape/input_value.go:398,480) are genuinely apply-time-relevant, not just plan-time cosmetics — so any gap in that semantic-equality chain is a direct, sufficient cause of "Provider produced inconsistent result after apply" on .inputs, exactly as reported.
2. internal/fleet/integration_policy/acc_test.go:85-93 (awsLogsCloudwatchVarsExpected/awsLogsStreamsJSONExpected) vs. internal/fleet/policyshape/secrets_test.go:90-135 (TestHandleRespSecrets table) — confirms the exact gap the previous run's avenue [feature] Support ILM policies #3 predicted still exists, with concrete evidence this run: the only passing acceptance test that exercises data_stream.dataset stripping (TestAccResourceIntegrationPolicySecrets/single_valued_secrets, aws_logs-aws-cloudwatch stream) keeps its secrets (access_key_id, secret_access_key, session_token) at the input-level vars_json, while data_stream.dataset lives alone, with no secret ref alongside it, in the stream-level vars (awsLogsCloudwatchVarsExpected). Likewise every case in secrets_test.go's TestHandleRespSecrets/TestHandleReqRespSecrets tables exercises exactly one key ("k") per case. Neither test constructs a single stream vars map containing both a data_stream.* key and a secret-ref-shaped key together — the reporter's azure_blob_storage stream (data_stream.dataset + service_account_key in the same streams["azure_blob_storage.azure-blob-storage"].vars) is exactly that untested shape. Reading handleVars (internal/fleet/policyshape/secrets.go:109-131 and :215-238) and stripServerManagedVarsKeys (internal/fleet/policyshape/input_value.go:480-506) key-by-key, both operate independently per map key with no apparent cross-key coupling, so this looks like a coverage gap rather than an obvious code defect from static reading alone — it needs a live-stack run (or a table-driven unit test combining both key shapes in one map, which this run's time budget did not leave room to author and validate) to settle whether the two mechanisms actually interact safely for this specific package/version.
3. CHANGELOG.md:1,20 — 0.16.3, PR #4055/commit 53d5543e — the reporter is running exactly 0.16.3, the release that shipped this fix, so they are not on a pre-fix version; if this is a real, still-open bug it is necessarily a gap in feat(fleet): fix data_stream.* vars injection and defaults inconsistency on 9.5 #4055's fix scope (the data_stream.*-plus-secret-in-one-stream combination from avenue Align "elasticsearch_connection" and "provider" blocks #2) rather than something that predates it. Worth checking azure_blob_storage package version 2.3.0's actual field definitions (e.g. via GET /api/fleet/epm/packages/azure_blob_storage/2.3.0 on a live Kibana, or the Elastic Package Registry) to confirm service_account_key is genuinely secret: true for that package/version and lives in the same stream as data_stream.dataset server-injection, since the issue report itself is a paraphrase (per the previous run's finding that the posted HCL predates the current vars-as-JSON-string schema) and may not perfectly reflect the real failing config.

References

• internal/fleet/policyshape/inputs_value.go, internal/fleet/policyshape/input_value.go, internal/fleet/policyshape/secrets.go (inspected: semantic-equality chain and secret/var substitution)
• internal/fleet/policyshape/secrets_test.go, internal/fleet/integration_policy/acc_test.go (inspected: existing coverage, confirmed no combined data_stream.*+secret-in-one-stream case)
• github.com/hashicorp/terraform-plugin-framework@v1.19.0/internal/fwserver/server_createresource.go (inspected: apply-time SchemaSemanticEquality invocation — pinned version per go.mod)
• CHANGELOG.md 0.16.3 entry, PR feat(fleet): fix data_stream.* vars injection and defaults inconsistency on 9.5 #4055 / commit 53d5543e (repo-path: CHANGELOG.md)
• Attempted connectivity check to host.docker.internal:9201/:5602, es-proxy/kb-proxy, and the local Docker daemon from the agent sandbox — all unreachable, so no live-stack verification was possible this run either
• Prior reproducer comment on this issue (run 30507688108, 2026-07-30) — carried forward and refined rather than repeated

🤖 Pipeline metadata

{
  "schema_version": "1.0",
  "outcome": "cannot-reproduce",
  "references": [
    { "type": "repo-path", "path": "internal/fleet/policyshape/inputs_value.go" },
    { "type": "repo-path", "path": "internal/fleet/policyshape/input_value.go" },
    { "type": "repo-path", "path": "internal/fleet/policyshape/secrets.go" },
    { "type": "repo-path", "path": "internal/fleet/policyshape/secrets_test.go" },
    { "type": "repo-path", "path": "internal/fleet/integration_policy/acc_test.go" },
    { "type": "repo-path", "path": "internal/debugutils/logging.go" },
    { "type": "repo-path", "path": "CHANGELOG.md" },
    { "type": "external", "url": "https://github.com/hashicorp/terraform-plugin-framework/blob/v1.19.0/internal/fwserver/server_createresource.go#L183-L199" },
    { "type": "pr", "url": "https://github.com/elastic/terraform-provider-elasticstack/pull/4055" },
    { "type": "issue", "url": "https://github.com/elastic/terraform-provider-elasticstack/issues/4329" }
  ]
}

[github-actions]

Thanks for the bug report! I've added the needs-reproduction label to this issue.

This looks like a reproducible bug — it includes a minimal Terraform configuration, the exact error message, reproduction steps, and trace log evidence — so it's been queued for the reproducer-factory pipeline. Reproducer-factory will attempt to confirm the behaviour and create a minimal reproduction case.

If I've misclassified this or you have additional context, please leave a comment — a maintainer can adjust the label manually.

[github-actions]

✅ Reproducer Factory Issue Intake completed successfully!

[tobio]

@wasfree is there a specific Stack/Kibana version you're using here?