ref(migrations): establish embedded database authority - #173
Merged
Conversation
- Establish a single crate for durable schema migration ownership. - Record immutable migration history with target-scoped checksums. - Document the required migration design contract.
- Run the audit schema stream during database startup. - Remove durable audit DDL from the business store. - Verify the store receives an already-migrated database.
- Register the LLM provider schema in the config migration stream. - Upgrade legacy default flags through a transactional Rust step. - Remove provider DDL from configuration initialization.
- Move durable server tables and legacy columns into config migrations. - Keep recurring pending-import cleanup outside the migration path. - Remove server schema ownership from initialization.
- Detect pending config migrations before startup initialization. - Create a readable SQLite snapshot before upgrading existing files. - Register server schema and legacy column upgrades in the config stream.
- Let the migration crate decide whether a target has pending work. - Use one SQLite snapshot policy for config and audit databases. - Keep product startup responsible only for existing database paths.
- Register Profile authoring tables, indexes, and capability references. - Remove durable Profile DDL from initialization. - Preserve migration-time verification of the authoring surface.
- Move durable SQLite schema history into versioned migration artifacts. - Add transactional ledger, checksum, backup, and file-lock guarantees. - Restrict migration execution to physical database initialization. - Convert domain schema entrypoints into read-only contract verifiers. - Move persistence and upgrade coverage into migration integration tests. - Prepare caller fixtures through the real migration chain.
- Document the versioned migration artifact and ledger lifecycle. - Define the boundary between durable schema SQL and business queries. - Require initialization-only migration execution and read-only domain checks. - Record migration testing, backup, and fail-closed maintenance expectations.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 676f07d321
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
This PR introduces a new mcpmate-migrations crate as the single authority for durable SQLite schema evolution (config + audit), moving schema creation/upgrade logic out of domain modules and into a versioned, checksum-verified migration ledger with file-locking and pre-migration backups. It then updates runtime initialization and a large portion of the test suite to prepare/verify databases through the migration runner rather than ad-hoc DDL.
Changes:
- Add
mcpmate-migrationswith versioned migration streams, ledger/state validation, file upgrade locking, and optional backup creation for file-backed DBs. - Replace domain-module “ensure/create schema” paths with migration verification (
verify_*_database) and update initialization/open paths to callprepare_*_database. - Refactor tests to prepare in-memory/file databases via migrations, removing test-only table creation loops.
Reviewed changes
Copilot reviewed 99 out of 100 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| backend/tests/surface_runtime.rs | Prepare config schema via migrations before catalog initialization. |
| backend/tests/surface_reconciliation.rs | Prepare config schema via migrations before table initialization and tests. |
| backend/tests/surface_materialization.rs | Prepare config schema via migrations; remove manual table DDL in tests. |
| backend/tests/support/runtime_database.rs | Ensure test DB fixtures run migrations before initialization. |
| backend/tests/support/mod.rs | Export new shared database test support module. |
| backend/tests/support/database.rs | New helper to prepare config DB schema via migrations. |
| backend/tests/support/client_writeback.rs | Use migrations-based config DB preparation in fixture setup. |
| backend/tests/secrets_store_api.rs | Prepare config schema via migrations in test contexts. |
| backend/tests/profile_surface_management.rs | Prepare config schema via migrations in management DB setup. |
| backend/tests/inspector_smoke.rs | Prepare config schema via migrations prior to initialization in tests. |
| backend/tests/direct_exposure_management.rs | Prepare config schema via migrations in fixture. |
| backend/tests/configuration_mode_transition.rs | Prepare config schema via migrations in fixture. |
| backend/tests/capability_read_surface.rs | Prepare config schema via migrations before initialization in tests. |
| backend/src/test_helpers.rs | Add shared helpers for preparing config/audit DBs in unit tests. |
| backend/src/inspector/calls.rs | Update tests to prepare config schema via migrations. |
| backend/src/core/secrets.rs | Update tests to prepare config schema via migrations. |
| backend/src/core/proxy/server/resources.rs | Update tests to prepare config schema via migrations. |
| backend/src/core/proxy/server/gateway.rs | Update tests to prepare config schema via migrations. |
| backend/src/core/proxy/init.rs | Update tests to prepare config schema via migrations; remove manual DDL. |
| backend/src/core/profile/visibility.rs | Update tests to prepare config schema via migrations. |
| backend/src/core/pool/sync.rs | Update tests to prepare config schema via migrations. |
| backend/src/core/oauth/manager.rs | Update tests to prepare config schema via migrations. |
| backend/src/core/foundation/loader.rs | Update tests to prepare config schema via migrations. |
| backend/src/core/capability/runtime.rs | Update tests to prepare config schema via migrations. |
| backend/src/core/capability/resource_uri.rs | Update tests to prepare config schema via migrations. |
| backend/src/core/capability/resource_registry.rs | Update tests to prepare config schema via migrations. |
| backend/src/core/capability/read_service.rs | Update tests to prepare config schema via migrations. |
| backend/src/core/capability/connection_provider.rs | Update tests to prepare config schema via migrations. |
| backend/src/config/server/tools.rs | Update tests to prepare config schema via migrations. |
| backend/src/config/server/oauth.rs | Update tests to prepare config schema via migrations. |
| backend/src/config/server/namespace_repair.rs | Update tests to prepare config schema via migrations. |
| backend/src/config/server/meta.rs | Update tests to prepare config schema via migrations. |
| backend/src/config/server/init.rs | Remove schema-creation DDL; verify migrated schema and retain startup cleanup. |
| backend/src/config/server/import.rs | Update tests to prepare config schema via migrations. |
| backend/src/config/server/crud.rs | Update tests to prepare config schema via migrations. |
| backend/src/config/server/capabilities.rs | Update tests to prepare config schema via migrations. |
| backend/src/config/profile/server.rs | Update tests to prepare config schema via migrations. |
| backend/src/config/profile/init.rs | Remove schema-creation DDL; verify migrated profile authoring tables. |
| backend/src/config/profile/capability_ref.rs | Update tests to prepare config schema via migrations. |
| backend/src/config/llm/init.rs | Remove schema-creation DDL; verify migrated config DB. |
| backend/src/config/llm/crud.rs | Update tests to prepare config schema via migrations. |
| backend/src/config/initialization.rs | Verify migrated config DB before running domain initialization steps. |
| backend/src/config/import.rs | Update tests to prepare config schema via migrations. |
| backend/src/config/database.rs | Invoke config migrations during DB open; log created backups. |
| backend/src/config/client/init.rs | Remove schema-creation/migration logic; verify migrated client schema and keep mode helpers. |
| backend/src/config/audit_database.rs | Invoke audit migrations during audit DB open; log created backups. |
| backend/src/clients/service/state.rs | Update tests to prepare config schema via migrations. |
| backend/src/clients/service/query.rs | Update tests to prepare config schema via migrations. |
| backend/src/clients/service/list.rs | Update tests to prepare config schema via migrations. |
| backend/src/clients/service/core.rs | Update tests to prepare config schema via migrations. |
| backend/src/audit/storage.rs | Replace audit DDL with migration verification + post-check query. |
| backend/src/audit/logger.rs | Update tests to prepare audit schema via migrations. |
| backend/src/api/handlers/server/preview.rs | Update tests to prepare config schema via migrations. |
| backend/src/api/handlers/server/crud.rs | Update tests to prepare config schema via migrations. |
| backend/src/api/handlers/server/common.rs | Update tests to prepare config schema via migrations. |
| backend/src/api/handlers/server/capability.rs | Update tests to prepare config schema via migrations. |
| backend/src/api/handlers/server/basic.rs | Update tests to prepare config schema via migrations. |
| backend/src/api/handlers/secrets.rs | Update tests to prepare config schema via migrations. |
| backend/src/api/handlers/profile/token_estimate.rs | Update tests to prepare config schema via migrations. |
| backend/src/api/handlers/profile/mgmt.rs | Update tests to prepare config schema via migrations. |
| backend/src/api/handlers/profile/capabilities.rs | Update tests to prepare config schema via migrations. |
| backend/src/api/handlers/onboarding.rs | Update tests to prepare config schema via migrations. |
| backend/src/api/handlers/client/surface_reviews.rs | Update tests to prepare config schema via migrations. |
| backend/src/api/handlers/client/handlers.rs | Update tests to prepare config schema via migrations. |
| backend/src/api/handlers/audit.rs | Update tests to prepare audit schema via migrations. |
| backend/crates/mcpmate-secrets/src/store.rs | Prepare config schema in tests via migrations helper. |
| backend/crates/mcpmate-secrets/src/database.rs | Replace secrets schema DDL with migration verification boundary. |
| backend/crates/mcpmate-secrets/Cargo.toml | Add dependency on mcpmate-migrations. |
| backend/crates/mcpmate-migrations/tests/support/memory.rs | New helper for in-memory migration tests (with FK PRAGMA). |
| backend/crates/mcpmate-migrations/tests/support/file.rs | New helper for file-backed migration tests (with FK PRAGMA). |
| backend/crates/mcpmate-migrations/tests/schema_contract.rs | Add integration tests asserting schema contracts, upgrades, and checksums. |
| backend/crates/mcpmate-migrations/tests/ledger_contract.rs | Add integration tests for ledger/state behavior and verification rules. |
| backend/crates/mcpmate-migrations/tests/config_upgrade_contract.rs | Add upgrade-contract tests for preserving legacy client relationships and backups. |
| backend/crates/mcpmate-migrations/src/runner.rs | Migration runner with ledger/state validation, locking, and backup creation. |
| backend/crates/mcpmate-migrations/src/migrations/mod.rs | Migration trait + checksum machinery + SQL migration executor. |
| backend/crates/mcpmate-migrations/src/migrations/config/v0010_create_capability_catalog.rs | Rust migration for capability catalog with strict schema contract validation. |
| backend/crates/mcpmate-migrations/src/migrations/config/v0009_create_profile_authoring.sql | SQL migration for profile authoring + direct exposure schema. |
| backend/crates/mcpmate-migrations/src/migrations/config/v0008_validate_secure_store.rs | Rust migration validating/rebuilding empty legacy secure store safely. |
| backend/crates/mcpmate-migrations/src/migrations/config/v0007_create_secure_store.sql | SQL migration creating secure store tables and constraints. |
| backend/crates/mcpmate-migrations/src/migrations/config/v0006_normalize_client_configuration.rs | Rust migration normalizing legacy client schema and derived fields. |
| backend/crates/mcpmate-migrations/src/migrations/config/v0005_create_client_configuration.sql | SQL migration for current client configuration tables. |
| backend/crates/mcpmate-migrations/src/migrations/config/v0004_upgrade_server_columns.rs | Rust migration for additive server column upgrades. |
| backend/crates/mcpmate-migrations/src/migrations/config/v0003_create_server_configuration.sql | SQL migration for server configuration schema. |
| backend/crates/mcpmate-migrations/src/migrations/config/v0002_add_llm_provider_default.rs | Rust migration adding llm_provider.is_default. |
| backend/crates/mcpmate-migrations/src/migrations/config/v0001_create_llm_provider.sql | SQL migration creating initial llm_provider. |
| backend/crates/mcpmate-migrations/src/migrations/config/mod.rs | Register ordered config migration stream artifacts. |
| backend/crates/mcpmate-migrations/src/migrations/audit/v0001_create_audit_storage.sql | SQL migration for audit tables + indexes. |
| backend/crates/mcpmate-migrations/src/migrations/audit/mod.rs | Register ordered audit migration stream artifacts. |
| backend/crates/mcpmate-migrations/src/lib.rs | Public API surface for prepare/verify entrypoints and DatabaseSource. |
| backend/crates/mcpmate-migrations/README.md | Document migration ownership boundary, policies, and validation commands. |
| backend/crates/mcpmate-migrations/Cargo.toml | New crate manifest + deps (sqlx, tokio, fs2, chrono, sha2). |
| backend/crates/mcpmate-capability-store/tests/surface_store_contract.rs | Ensure capability store tests prepare config schema via migrations. |
| backend/crates/mcpmate-capability-store/tests/scale_evidence.rs | Ensure scale evidence test prepares file-backed schema via migrations. |
| backend/crates/mcpmate-capability-store/tests/reconciliation_store_contract.rs | Ensure reconciliation tests prepare config schema via migrations. |
| backend/crates/mcpmate-capability-store/tests/catalog_contract.rs | Ensure catalog tests prepare schema via migrations; remove legacy-schema rejection tests now owned by migrations crate. |
| backend/crates/mcpmate-capability-store/src/schema.rs | Replace schema creation with migration verification -> CatalogError::IncompatibleSchema on mismatch. |
| backend/crates/mcpmate-capability-store/Cargo.toml | Add dependency on mcpmate-migrations. |
| backend/Cargo.toml | Add dependency on mcpmate-migrations for backend crate. |
| backend/Cargo.lock | Lock new dependencies (mcpmate-migrations, fs2, etc.). |
| AGENTS.md | Update repo contract to declare mcpmate-migrations as sole schema owner. |
- Preserve current client state while rebuilding incomplete schemas. - Verify catalog storage and resolve SQLite paths through SQLx. - Stabilize migration checksums across line endings.
Deploying mcpmate-site with
|
| Latest commit: |
6171829
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://348a59fb.mcp-umate.pages.dev |
| Branch Preview URL: | https://feat-embedded-database-migra.mcp-umate.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Establish the embedded migration crate as the single startup authority for config, secrets, and capability catalog storage while preserving a small, auditable migration ledger.
Review follow-up
Project
Project #4: Embedded database migration foundation
Config and migration impact
Validation