Aspire hangs at "Starting dashboard" when DCP state was created by a newer version

[JamesNK]

Is there an existing issue for this?

• I searched existing issues and pull requests.

Related but not duplicate: #18548 covers a concurrent state-directory creation race. This issue is about an incompatible state-store schema after switching from a newer DCP to an older stable DCP.

Describe the bug

After using a newer/dogfood Aspire/DCP version, running an AppHost with stable Aspire 13.4.6 hangs at:

Connecting to AppHost...
Starting dashboard...

The dashboard never starts listening, and application resources remain in Starting or Waiting. aspire ps may show the AppHost with a null dashboard URL, while aspire describe/aspire wait cannot make progress.

The underlying DCP controller process exits immediately because the global state database was migrated by a newer DCP version:

failed to initialize state store: could not migrate state store database
'C:\Users\<user>\.dcp\state\state.sqlite3': could not apply state store migrations:
no migration found for version 2: read down for version 2 migrations: file does not exist

This error is not surfaced by aspire run. Instead, the CLI waits indefinitely at Starting dashboard....

Environment

• OS: Windows
• Aspire CLI: 13.4.6+87fe259e4fc244c599019a7b1304c85a1488f248 (stable)
• AppHost SDK: 13.4.6
• DCP bundled with stable Aspire: 0.24.3 (a2b84af5a533e915dfd805d4fe1e7f238abc143a)
• .NET SDK selected by the project: 10.0.302
• Docker Desktop: 29.6.1

The machine also uses newer/dogfood Aspire builds, which appears to be how the shared DCP database reached schema version 2.

Steps to reproduce

1. Run an AppHost with a newer Aspire/DCP version that migrates %USERPROFILE%\.dcp\state\state.sqlite3 to schema version 2.
2. Switch to a project using stable Aspire/AppHost SDK 13.4.6 and DCP 0.24.3.
3. Run:

aspire run --non-interactive

4. Observe that startup hangs at:

Starting dashboard...

To reveal the hidden cause, running the bundled controller against the generated kubeconfig reports:

dcp.exe run-controllers --kubeconfig <generated-kubeconfig> --monitor <apphost-pid> --container-runtime docker -v=debug

failed to initialize state store: could not migrate state store database
'C:\Users\<user>\.dcp\state\state.sqlite3': could not apply state store migrations:
no migration found for version 2: read down for version 2 migrations: file does not exist

DCP's API server is otherwise healthy. Resources are created in the API, but no controller reconciles them: the dashboard Executable has an empty execution ID/state/start timestamp and no process is launched.

Expected behavior

Aspire should handle switching between DCP versions safely. Possible acceptable behaviors include:

• isolate state stores by incompatible DCP/schema version;
• support the required downgrade migration;
• detect incompatible transient state and safely back it up/reset it; or
• fail immediately with the controller error and actionable recovery guidance.

At minimum, a required DCP controller startup failure must be propagated to the AppHost/CLI instead of leaving aspire run hanging at Starting dashboard....

Actual behavior

• The DCP controller exits with a state migration error.
• The failure is not shown by aspire run.
• The dashboard remains Starting with no listener.
• Dependent resources remain Starting/Waiting.
• Startup never becomes usable.

Workaround

1. Stop all AppHosts/DCP processes.
2. Back up and remove the incompatible state files:

%USERPROFILE%\.dcp\state\state.sqlite3
%USERPROFILE%\.dcp\state\state.sqlite3.migrate.lock

3. Run aspire run again.

DCP creates a compatible state store, the dashboard starts immediately, migrations complete, and the application becomes healthy.

Suggested regression coverage

• Create a state database at a newer schema version, then start an older DCP that lacks the down migration.
• Verify Aspire fails fast and includes the DCP controller's migration error.
• Verify switching between supported stable/dogfood DCP versions cannot silently make all AppHosts unusable.

[joperezr]

🤖 Aspire triage pass — 2026-07-22 14:13 UTC

Area: area-orchestrator
Kind: bug
Repro: partial — see the "Steps to reproduce" section (requires a prior run on a newer DCP that migrated ~/.dcp/state/state.sqlite3 to schema v2)
Affected version(s): 13.4.6 (DCP 0.24.3); latest: 13.4.x line
Severity: S3 — nasty failure mode (silent indefinite hang, no error surfaced) but a clear workaround exists (remove the incompatible state files) and it only triggers when mixing newer/dogfood DCP with stable DCP

Urgency: 🟡 vNext — recommend milestone 13.5
Reason: high-quality report in the core orchestrator area with concrete suggested regression coverage; robustness/forward-compat gap, not a confirmed regression introduced in 13.4 (schema v2 came from a newer DCP, and stable 13.4.6 simply lacks the down-migration), and a workaround exists — so it does not clear the 13.4.x servicing bar.

Suggested owner(s): @karolz-ms (score 30), backup @danegsta (score 20)
Evidence:

• karolz-ms closed as completed Flaky test: TestingBuilderTests.ArgsPropagateToAppHostConfiguration — DCP state store directory collision #18548, [13.3] AddKeycloak + AddFlagd with WaitFor hangs in Starting state, Container CRD never submitted to DCP (regression of #15041) #16897, Starting Dashboard failing after update to 13.3.3 #16866, Aspire no longer opening UI windows of resources #15518, [Container Tunnel] OtlpEndpointReferenceGatherer hangs forever for containers with OtlpExporterAnnotation when no containers are tunnel-dependent #15041 in area-orchestrator (last 180d) — 14 total
• danegsta closed as completed Keycloak SSL errors #18620, WithOtlpExporter() breaks ContainerLifetime.Persistent, lifecycle key mismatch on every run #17191, [13.3] AddKeycloak + AddFlagd with WaitFor hangs in Starting state, Container CRD never submitted to DCP (regression of #15041) #16897, [13.3] Container Tunnel fails to start without internet connectivity #16615, CosmosDB Preview Emulator Regression from 13.1.2 to 13.2.0 - ContainerLifetime.Persistent does not keep the same container #15626 in area-orchestrator — 9 total
• CODEOWNERS lists @JamesNK @karolz-ms @danegsta @mitchdenny @davidfowl for /src/Aspire.Hosting (DCP/orchestrator); JamesNK excluded as the issue author
• No merged PRs carried the area-orchestrator label in the last 90d, so scoring is driven by closed-completed history + CODEOWNERS

Confidence: medium
Needs human?: no

Dry-run mode — no labels or milestone applied. Reply /triage apply to promote, or edit and reapply.

[danegsta]

I've created a DCP PR (microsoft/dcp#221) intended to resolve this issue; the original intent of the schema migrations was that newer migrations should be treated as backwards compatible by default (for exactly this scenario), but that wasn't implemented properly in the DCP version that shipped with 13.4.

The new DCP PR adds a new "minor" migration version with the primary migrations table being treated as a "major" version. So bumping past version 1 would still be considered a breaking change for older DCP versions, but those migration changes will be reserved for a hypothetical intentional breaking change in the future. Instead backwards compatible changes will be added via a separate migrations index. Effectively the newer changes will be opaque in older DCP since it won't even know to check for the minor migrations version, while newer releases will explicitly treat higher minor migration versions as compatible.

I'm also adding a small compatibility fix that will convert the problematic version 2 migration we introduced to the new minor migration on version 1, meaning we'll patch the state store to a valid state for 13.4 if a daily that introduced the problematic v2 migration was run first. The one scenario we're not going to explicitly cover is if the user then runs the old daily build AGAIN after the corrective update is made. Main thing is that once this makes it into daily it'll be possible to go between daily and stable without any state store errors (and with the state store remaining on the latest migration version).