From dcd254c988c4311309e8cd7f892a2987d35b1f11 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Wolf-Martell=20Montwe=CC=81?= Date: Wed, 2 Sep 2026 15:13:53 +0200 Subject: [PATCH 1/6] docs(rfc): mark global database RFC as accepted --- docs/SUMMARY.md | 2 +- docs/engineering/rfcs/0007-global-database.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 98c6dcaddc..ef3eb76133 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -39,7 +39,6 @@ generator, in this case, **mdbook**. It defines the structure and navigation of - [RFCs](engineering/rfcs/README.md) - [Template](engineering/rfcs/0000-rfc-template.md) - [Proposed]() - - [0007 - Global Database](engineering/rfcs/0007-global-database.md) - [Accepted]() - [0001 - Changelog System Replacement](engineering/rfcs/0001-changelog-system-replacement.md) - [0002 - Thundermail – Add an Account interim screen](engineering/rfcs/0002-thundermail-add-an-account-interim-screen.md) @@ -47,6 +46,7 @@ generator, in this case, **mdbook**. It defines the structure and navigation of - [0004 - Add a Declarative Feature Flag Catalog](engineering/rfcs/0004-feature-flag-new-architecture.md) - [0005: Enable IMAP IDLE ("Push") for the Inbox by Default on Newly Added IMAP Accounts](engineering/rfcs/0005-imap-idle-push-on-new-account-inboxes.md) - [0006: Remote Feature Flags](engineering/rfcs/0006-remote-feature-flags.md) + - [0007 - Global Database](engineering/rfcs/0007-global-database.md) - [Rejected]() - [Obsolete]() - [Architecture Decision Records](engineering/adr/README.md) diff --git a/docs/engineering/rfcs/0007-global-database.md b/docs/engineering/rfcs/0007-global-database.md index bd0e34eb47..72d0888a92 100644 --- a/docs/engineering/rfcs/0007-global-database.md +++ b/docs/engineering/rfcs/0007-global-database.md @@ -5,7 +5,7 @@ - Related RFC: [UUIDv7 Identifier Migration](0009-uuidv7-identifier-migration.md) - Repository pattern: [ADR 0010 proposal](https://github.com/thunderbird/thunderbird-android/pull/11452) - Portable data format: [RFC 0008: Portable Profile Data Format](0008-portable-profile-data-format.md) -- Status: **Proposed** +- Status: **Accepted** - Backend decision: **Room 3 selected by the [completed database spike](https://github.com/thunderbird/thunderbird-android/issues/11195)** From bd58206add551faa344e31555b29e9623f8b59a8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Wolf-Martell=20Montwe=CC=81?= Date: Mon, 31 Aug 2026 16:41:35 +0200 Subject: [PATCH 2/6] docs(technical-design): add global database docs and technical design --- docs/SUMMARY.md | 3 + docs/architecture/database.md | 91 ++++++++++++++++ docs/engineering/rfcs/0007-global-database.md | 2 - .../technical-designs/0003-global-database.md | 100 ++++++++++++++++++ .../legacy-table-inventory.md | 56 ++++++++++ 5 files changed, 250 insertions(+), 2 deletions(-) create mode 100644 docs/architecture/database.md create mode 100644 docs/engineering/technical-designs/0003-global-database.md create mode 100644 docs/engineering/technical-designs/0003-global-database/legacy-table-inventory.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index ef3eb76133..7c21c6cb0c 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -29,6 +29,7 @@ generator, in this case, **mdbook**. It defines the structure and navigation of - [User Flows](architecture/user-flows.md) - [Settings](architecture/settings.md) - [Feature Flags](architecture/feature-flags.md) + - [Database](architecture/database.md) - [Legacy Module Integration](architecture/legacy-module-integration.md) - [Engineering](engineering/README.md) - [Delivery Planning](engineering/delivery-planning.md) @@ -68,6 +69,8 @@ generator, in this case, **mdbook**. It defines the structure and navigation of - [Technical Designs](engineering/technical-designs/README.md) - [Template](engineering/technical-designs/0000-technical-design-template.md) - [Proposed]() + - [0003 - Global Database](engineering/technical-designs/0003-global-database.md) + - [Legacy Table Inventory](engineering/technical-designs/0003-global-database/legacy-table-inventory.md) - [Accepted]() - [0001 - Changelog System Replacement](engineering/technical-designs/0001-changelog-system-replacement.md) - [0002: Declarative Feature Flag Catalog](engineering/technical-designs/0002-feature-flag-declarative-catalog.md) diff --git a/docs/architecture/database.md b/docs/architecture/database.md new file mode 100644 index 0000000000..aa3b624367 --- /dev/null +++ b/docs/architecture/database.md @@ -0,0 +1,91 @@ +# Database Architecture + +> **Status:** Proposed direction. This describes the target architecture introduced by Global Database. It is not a +> description of the current legacy implementation. + +## Purpose + +The database architecture provides one local source of truth for mail data in the application, while keeping +mail domain code independent of storage technology and legacy storage types. +It establishes the boundaries that Global Database implements. The [RFC](../engineering/rfcs/0007-global-database.md) +and [technical design](../engineering/technical-designs/0003-global-database.md) remain the authoritative records +for its scope and implementation. + +## Current and target state + +Today, legacy mail storage uses one SQLite database and one attachment directory per account. + +Global Database introduces one Room 3-backed mail database and one file-backed attachment directory for the +application. + +## Boundaries and ownership + +```mermaid +flowchart LR + Caller[Mail domain callers] --> Contract[Focused mail repository contracts] + Contract --> Implementation[Repository implementations] + Implementation --> GlobalSource[Global local data sources] + Implementation --> LegacySource[Legacy local data sources during migration] + GlobalSource --> MailDatabase[Global mail database] + GlobalSource --> AttachmentFiles[Global attachment files] + MailDatabase --> CoreDatabase[core:database] + LegacySource --> LegacyDatabase[Per-account legacy databases] + LegacySource --> LegacyFiles[Per-account attachment files] + AppComposition[Application composition] --> Implementation +``` + +- **Mail repository contracts** are the only storage boundary visible to mail-domain callers. They follow the + [Repository pattern ADR](../engineering/adr/0010-adopt-project-wide-repository-pattern.md): they are focused, + explicitly account-scoped, and do not expose Room, SQL, cursors, files, or legacy `LocalStore` types. +- **Repository implementations and local data sources** belong to the mail feature's internal persistence + implementation. They own mappings between domain data and both global and legacy representations. +- **The global mail database** owns the legacy-compatible mail schema, migration metadata, and derived data. + It is one physical database for the application, not one database per account. +- **The attachment store** remains file-backed for content that legacy already held on disk, which is every body part + above a size threshold. Parts at or below it stay `message_parts` BLOBs, as in legacy. Database records and attachment + files are associated through internal persistence mappings. +- **`core:database`** implements Room 3 as one domain-neutral backend and provides lifecycle, transaction, migration, + and Android/JVM desktop driver support needed by the mail implementation. It composes feature schema contributions + deterministically and owns one coordinated schema and migration history. It must not own mail schema, mail entities + or DAOs, repository contracts, or mail domain types. +- **Application composition** binds repository contracts to one implementation. It is the only place that switches from + legacy to global storage after successful migration. + +## Data and migration model + +The global schema begins as a compatible representation of the legacy mail-store schema and behavior. +Account-local numeric IDs are preserved through account-qualified legacy keys. Global identifiers are added only where +repository contracts require them. +The [legacy table inventory](../engineering/technical-designs/0003-global-database/legacy-table-inventory.md) +defines the exact source-to-target mapping, including external attachment files. + +Migration has one authoritative storage representation at a time: + +| Migration phase | Authoritative storage | Normal mail access | +|------------------------------------|-------------------------------------------|-----------------------------------------------------------------------------------| +| Before cutover | Legacy per-account storage | Legacy repository implementation | +| Import and validation | Legacy per-account storage | Migration gate holds UI and background mail work. Global data remains unpublished | +| After cutover state is established | Global mail database and attachment store | Global repository implementation | + +No normal caller may read from or write to both representations. +Legacy database and attachment artifacts are deleted only after validation and the durable cutover state is established. +The legacy storage implementation stays in the codebase, unbound and unused, and a later release removes it. + +## Constraints + +- Global Database does not redesign or normalize the legacy mail schema. +- Android and JVM desktop drivers are in scope. The global attachment-store contract is KMP-safe on both targets. + Legacy import is Android-only. +- On Android, a dedicated migration screen owns the migration UI. It could reuse the existing database-migration + activity or introduce a suitable replacement. It shows non-sensitive progress and a completion or failure state with + retry and local report export. The shared migration gate blocks normal mail access and background sync until cutover + or failure. +- Database implementation types remain internal and follow the API/internal module boundary. +- Migration reports are local and user-exportable, but exclude personally identifiable data and are never uploaded. + +## Related documentation + +- [RFC 0007: Global Database](../engineering/rfcs/0007-global-database.md) +- [Technical Design 0003: Global Database](../engineering/technical-designs/0003-global-database.md) +- [Legacy Table Inventory](../engineering/technical-designs/0003-global-database/legacy-table-inventory.md) + diff --git a/docs/engineering/rfcs/0007-global-database.md b/docs/engineering/rfcs/0007-global-database.md index 72d0888a92..cb7d694bce 100644 --- a/docs/engineering/rfcs/0007-global-database.md +++ b/docs/engineering/rfcs/0007-global-database.md @@ -163,8 +163,6 @@ Validation before cutover and the archive from step 1 cover that risk instead. ## Open Questions -- Should migration require a pre-flight free-space check with a stated minimum headroom, and should a failed check block - migration or only warn? - Does declining the POP3 archive require a durable record of the user's acknowledgement? The technical design owns implementation questions, including driver configuration, the identifier mapping inventory, diff --git a/docs/engineering/technical-designs/0003-global-database.md b/docs/engineering/technical-designs/0003-global-database.md new file mode 100644 index 0000000000..3e4d135326 --- /dev/null +++ b/docs/engineering/technical-designs/0003-global-database.md @@ -0,0 +1,100 @@ +# Technical Design: Global Database + +- Issue: [#11293](https://github.com/thunderbird/thunderbird-android/issues/11293) +- RFC: [RFC 0007: Global Database](../rfcs/0007-global-database.md) +- Repository pattern: [ADR 0010 proposal](https://github.com/thunderbird/thunderbird-android/pull/11452) +- Portable data format: [RFC 0008: Portable Profile Data Format](../rfcs/0008-portable-profile-data-format.md) +- Status: **Proposed** + +## Summary + +Global Database replaces one legacy mail database per account with one Room 3-backed database for the +application. It keeps legacy mail behavior compatible, imports durable data, validates the result, and cuts over once. + +## Boundaries + +- `core:database` provides the domain-neutral Room 3 backend, lifecycle, transactions, migrations, and platform + support. It composes feature schema contributions deterministically and owns one coordinated schema and migration + history. +- The internal mail database module owns mail schema, mappings, local data sources, and repository implementations. +- Focused mail repository contracts hide both legacy and global storage from callers. +- `app-common` or an app module binds repository contracts to the active implementation. + +The boundary follows ADR 0009. Mail schema and implementation types remain internal. `core:database` does not own mail +schema, mail domain types, or mail repositories. + +## Global mail store + +The global database keeps the legacy mail-store schema and behavior as its starting point. It stores all accounts in one +database, keeps account-qualified legacy references, and adds global identifiers only where callers need them. + +The [legacy table inventory](0003-global-database/legacy-table-inventory.md) defines the exact mapping. It includes +downloaded attachments. Legacy keeps a body part on disk above a size threshold and as a `message_parts` BLOB at or +below it. The migration preserves that split rather than changing where content lives. + +## Migration + +The migration runs before normal mail access is available. A dedicated migration screen, which could reuse the existing +Android migration activity or introduce a suitable replacement, shows non-sensitive progress and a clear completion or +failure state. A migration gate holds startup, sync, and other background mail work until migration completes or fails. + +1. Create and verify the required RFC 0008 archive for POP3 accounts. The user may decline it after an explicit warning + and continue at their own risk. IMAP export is optional. +2. Check available storage against the required headroom and fail with an actionable error when it is insufficient. + Then create an unpublished global database. +3. Read legacy databases and attachment directories without modifying them. Import every durable record. +4. Copy each attachment to its target and validate it. If validation fails, record the failure in the migration result + and fail the migration. +5. Rebuild derived data, validate the imported database, and reopen it. +6. Establish the durable cutover state and switch repository bindings to the global implementation. +7. Remove confirmed legacy database and attachment artifacts. + +Before step 6, global data is not visible to normal mail code. Any failure before that step keeps legacy storage +authoritative. A later retry starts with a new unpublished import. A cleanup failure after cutover leaves global storage +authoritative and records the remaining cleanup work. + +The legacy storage implementation stays in the codebase, unbound and unused, and a later release removes it. + +## Validation + +Cutover requires all of the following: + +- every configured account was imported +- mapped data and relationships are complete +- queued operations survive restart and remain executable +- file-backed attachments, meaning parts with `data_location = 2`, are present and valid +- database integrity checks pass after reopen +- search is rebuilt and representative queries match +- the durable cutover state remains readable after reopen + +## Diagnostics + +Migration reports contain only non-sensitive progress, counts, status, and stable error codes. They do not contain mail +content, addresses, attachment names, paths, credentials, or tokens. Reports stay local and can be exported by the +user. + +## Platform support + +The global database, repository contracts, and attachment access support Android and JVM desktop. Legacy storage +reading and import are Android-only. + +## Testing + +Automated tests cover: + +- repository behavior before and after cutover +- schema creation, migration, and restart recovery +- every durable table and attachment type in the inventory +- POP3 archive gating and IMAP optional export +- validation, interruption, retry, and cleanup failure +- migration-gate behavior for startup and background work +- report redaction +- the guarantee that partial global data is never visible to normal callers + +## Open technical questions + +- Which supported legacy schema versions require dedicated fixtures? +- Which Room driver and locations apply on Android and JVM desktop? +- Which global identifier representation and mapping retention period are required? +- How should post-cutover cleanup retry a locked legacy artifact? + diff --git a/docs/engineering/technical-designs/0003-global-database/legacy-table-inventory.md b/docs/engineering/technical-designs/0003-global-database/legacy-table-inventory.md new file mode 100644 index 0000000000..b24d5b65b1 --- /dev/null +++ b/docs/engineering/technical-designs/0003-global-database/legacy-table-inventory.md @@ -0,0 +1,56 @@ +# Global Database: Legacy Table Inventory + +Supporting document for [Technical Design 0003: Global Database](../0003-global-database.md). + +## Purpose + +This inventory is the source of truth for the legacy-to-global migration mapping. +It is deliberately separate from the technical design because the table-level implementation detail will evolve while +the design's migration contract remains stable. + +Every durable legacy table is copied to a legacy-compatible target representation. +`messages_fulltext` is rebuilt because it is derived data. No other table is deferred without an explicit, tested +compatibility decision. + +| Legacy source | Classification | Global target and import rule | +|--------------------------------------------|----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `account_extra_values` | Copy | Preserve text/integer values and absence semantics, scoped to `AccountId`. | +| `folders` | Copy | Preserve every field. Map `(account_id, legacy_local_id)` and associate a global `FolderId` when required. | +| `folder_extra_values` | Copy | Preserve unknown keys and values through the account-qualified folder mapping. | +| `messages` | Copy | Preserve metadata, flags, previews, encryption, and new-message state. Map to a global `MessageId` when required. | +| `message_parts` | Copy | Preserve MIME-tree relationships, metadata, `data_location`, and every `data` BLOB. Legacy stores a body part in `data` at or below the 16 KiB threshold and on disk above it, regardless of whether the part is an attachment, so in-database attachment bytes stay BLOBs. | +| `threads` | Copy | Preserve existing message, root, and parent relationships. Do not recalculate threading in this migration. | +| `outbox_state` | Copy | Preserve send state, attempts, error timestamps, and error-state semantics. | +| `pending_commands` | Copy | Preserve command and serialized payload with the source account. Preserve account-qualified local IDs so execution resolves the mapped target records without redesigning queue behavior. | +| `messages_fulltext` | Rebuild | Rebuild from imported durable content, setting `docid` to the imported message's id so search joins resolve. Coverage equals messages whose body parts are locally present, because the legacy index was populated from text supplied at save time. Validate representative account and unified search parity against that set. | +| `notifications` | Copy | Preserve notification IDs and timestamps through the account-qualified message mapping. | +| `.db_att/` | Copy | Covers parts with `data_location = 2` (`ON_DISK`) only. Keep that content in the global file-backed directory. Do not move it into database BLOB storage, and do not promote in-database BLOBs to files. | + +For all account-local numeric IDs, target mappings preserve `(account_id, legacy_local_id)` and record a global domain +identifier only where the repository contract requires one. + +The fts4 shadow tables `messages_fulltext_content`, `_segdir`, `_segments`, `_docsize`, and `_stat` are excluded. They +are rebuilt with their virtual table and are not durable inputs. + +## Schema behavior + +The legacy schema carries six triggers whose effects are part of mail behavior, not performance tuning: + +| Trigger | Effect to preserve | +|------------------------------|---------------------------------------------------------------------------------------------------| +| `set_message_part_root` | Sets `message_parts.root = id` on insert when `root` is null. | +| `set_thread_root` | Sets `threads.root = id` on insert when `root` is null. | +| `new_message_reset` | Clears `messages.new_message` when `read` becomes 1. | +| `delete_message` | Deletes `message_parts` by `root`, `messages_fulltext` by `docid`, and `threads` by `message_id`. | +| `delete_folder` | Deletes `messages` for the folder. | +| `delete_folder_extra_values` | Deletes `folder_extra_values` for the folder. | + +Each effect is reproduced as a trigger, a foreign-key cascade, or repository logic. The choice is recorded here per +trigger and tested for parity. Indexes are reproduced for query-plan parity but carry no behavior. + +## Implementation prerequisites + +Before implementation, complete and test the column-level mapping, trigger/index compatibility checks, supported +source-schema fixtures, and representative fixtures for every source listed here. Source reading must be read-only and +must not invoke legacy repair or upgrade behavior. Queue fixtures must deserialize and execute each supported command +after restart using its account-qualified mappings. From 3750ac4ab1d8d20a893fb6f87de56fe420f9a1f2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Wolf-Martell=20Montwe=CC=81?= Date: Wed, 2 Sep 2026 15:20:22 +0200 Subject: [PATCH 3/6] docs(technical-design): expand global database design with identifier model (cherry picked from commit db3649d9706e78f6d2c982f4e63f7e6c6070a70c) --- .../technical-designs/0003-global-database.md | 73 ++++++++++++++++++- .../legacy-table-inventory.md | 34 +++++---- 2 files changed, 90 insertions(+), 17 deletions(-) diff --git a/docs/engineering/technical-designs/0003-global-database.md b/docs/engineering/technical-designs/0003-global-database.md index 3e4d135326..9289a62b52 100644 --- a/docs/engineering/technical-designs/0003-global-database.md +++ b/docs/engineering/technical-designs/0003-global-database.md @@ -32,6 +32,59 @@ The [legacy table inventory](0003-global-database/legacy-table-inventory.md) def downloaded attachments. Legacy keeps a body part on disk above a size threshold and as a `message_parts` BLOB at or below it. The migration preserves that split rather than changing where content lives. +### Identifier model + +Global identifiers are opaque, application-profile-scoped domain values. They identify local records in the global +database and are stable while that record exists, including across application restart and database-schema upgrades. +They are not protocol identifiers, portable-profile identifiers, or synchronization identifiers. In particular, an +IMAP UID, a folder server ID, and an RFC 5322 `Message-ID` header must not be used as a global database identifier. + +| Identifier | Owner | Meaning and boundary | +|----------------|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `AccountId` | `feature:account:api` | The existing UUID-backed account identifier. Global-mail records retain its existing persisted value to establish account scope. Cutover neither replaces nor regenerates it. Account settings and profile runtime state remain outside the global mail database. | +| `FolderId` | Mail domain | Identifies one local folder record across all accounts. It replaces the legacy account-local folder number at repository boundaries. | +| `MessageId` | Mail domain | Identifies one local message record across all accounts. A message copied to another folder is a separate local record and therefore has a separate `MessageId`. | +| `ThreadId` | Mail domain | Identifies one account-scoped conversation, which can contain local message records from Inbox, Sent, Archive, and other folders. It is a durable conversation aggregate, not a legacy numeric thread-root key. Thread operations use `ThreadId`. Operations on an individual local record use `MessageId`. | +| `AttachmentId` | Mail domain | An opaque attachment-access URI or equivalent reference. It resolves unambiguously after cutover and is not a raw message-part primary key. Message-part keys stay internal unless a future focused attachment contract requires one. | + +RFC 0009 owns the UUID representation and generation policy for `FolderId`, `MessageId`, and `ThreadId`. These types, +along with `AccountId` and `AttachmentId`, are the only identifiers that cross the mail repository boundary for this +design. Repository contracts never expose legacy numeric IDs or persistence keys. + +The thread builder uses the imported messages' threading headers across all folders of the same account. When a newly +observed message joins two conversations, the builder selects one existing `ThreadId` deterministically, merges the +memberships, and rewrites the other conversation's internal references in the same transaction. A `ThreadId` is not a +protocol identifier and is not used for server operations. + +Room entities may have a separate persistence-local integer surrogate key. Those keys preserve compatible relationships +and support SQLite features such as FTS, and never appear in a repository contract. In particular, FTS4 `docid` remains +an integer internal search-document key mapped to a `MessageId`. A UUID-backed `MessageId` is never stored directly as +`docid`. + +### Identifier mapping and internal keys + +For imported data, the migrator records an account-qualified source key before linking dependent rows: + +| Logical record | Source key | Target domain identifier | Required dependent rewrite | +|-----------------|-----------------------------------------------------|--------------------------|-------------------------------------------------------------------------------------------------------------------| +| Folder | `(AccountId, legacy folder id)` | `FolderId` | folder extra values, message folder references, queued-command folder references | +| Message | `(AccountId, legacy message id)` | `MessageId` | outbox state, notifications, full-text mapping, message part root, thread message reference | +| Thread | `(AccountId, legacy folder id, legacy thread root)` | `ThreadId` | rebuild cross-folder memberships from message threading headers and rewrite threaded-list and thread-cache values | +| Message part | `(AccountId, legacy message-part id)` | internal only | part root and parent references, message root-part reference, attachment-file lookup and URI resolution | +| Pending command | `(AccountId, legacy command id)` | internal only | command row identity and serialized folder references | + +The exact physical representation is internal, but it must enforce uniqueness of every source key and reject an import +that maps one source key to multiple targets. Several legacy folder-local thread roots may map to one `ThreadId` when +the imported messages form one cross-folder conversation. Account-qualified mappings remain available until validation +has completed and all durable queue, notification, attachment, and supported legacy external references have been +translated. The implementation may retain them longer for compatibility. Their retention period and removal test are +part of the migration implementation plan. + +`notifications.notification_id` is an Android application notification ID, not a mail-domain identifier. It must remain +unique across the application notification namespace. Import validates that constraint and allocates a replacement when +preserving a legacy value would collide. The notification's relationship to its message is rewritten through +`MessageId` and the internal message key. + ## Migration The migration runs before normal mail access is available. A dedicated migration screen, which could reuse the existing @@ -49,6 +102,15 @@ failure state. A migration gate holds startup, sync, and other background mail w 6. Establish the durable cutover state and switch repository bindings to the global implementation. 7. Remove confirmed legacy database and attachment artifacts. +The import writes folder and message identifier mappings before importing dependents. After all message records for an +account are available, it builds cross-folder `ThreadId` memberships from their threading headers and records the +legacy folder-local thread-root mappings before rewriting threaded-list and cache values. It then validates all +rewritten relationships before the global database is published. It translates serialized pending-command folder +references to `FolderId` or to the chosen internal mapping before the command is eligible to execute. It also accepts +supported pre-cutover message references, such as notification or activity references containing `(AccountId, +legacyFolderId, UID)`, through the compatibility mapping until that support is intentionally retired. Newly created +references use the global identifier model. + Before step 6, global data is not visible to normal mail code. Any failure before that step keeps legacy storage authoritative. A later retry starts with a new unpublished import. A cleanup failure after cutover leaves global storage authoritative and records the remaining cleanup work. @@ -65,6 +127,12 @@ Cutover requires all of the following: - file-backed attachments, meaning parts with `data_location = 2`, are present and valid - database integrity checks pass after reopen - search is rebuilt and representative queries match +- every source key maps to exactly one target, all dependent keys resolve, and no internal legacy numeric key crosses a + repository boundary +- messages from Inbox, Sent, and other folders join the same `ThreadId` when their threading headers identify one + conversation, without joining messages from different accounts +- attachment URIs resolve to the same imported part after restart +- notification IDs are unique in the application namespace after import - the durable cutover state remains readable after reopen ## Diagnostics @@ -90,11 +158,14 @@ Automated tests cover: - migration-gate behavior for startup and background work - report redaction - the guarantee that partial global data is never visible to normal callers +- identifier-mapping completeness, duplicate-key rejection, pending-command payload translation, attachment URI + resolution, and notification-ID collision handling +- cross-folder thread construction, including a conversation spanning Inbox and Sent and a merge of two existing + conversations ## Open technical questions - Which supported legacy schema versions require dedicated fixtures? - Which Room driver and locations apply on Android and JVM desktop? -- Which global identifier representation and mapping retention period are required? - How should post-cutover cleanup retry a locked legacy artifact? diff --git a/docs/engineering/technical-designs/0003-global-database/legacy-table-inventory.md b/docs/engineering/technical-designs/0003-global-database/legacy-table-inventory.md index b24d5b65b1..31038384c2 100644 --- a/docs/engineering/technical-designs/0003-global-database/legacy-table-inventory.md +++ b/docs/engineering/technical-designs/0003-global-database/legacy-table-inventory.md @@ -12,22 +12,24 @@ Every durable legacy table is copied to a legacy-compatible target representatio `messages_fulltext` is rebuilt because it is derived data. No other table is deferred without an explicit, tested compatibility decision. -| Legacy source | Classification | Global target and import rule | -|--------------------------------------------|----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `account_extra_values` | Copy | Preserve text/integer values and absence semantics, scoped to `AccountId`. | -| `folders` | Copy | Preserve every field. Map `(account_id, legacy_local_id)` and associate a global `FolderId` when required. | -| `folder_extra_values` | Copy | Preserve unknown keys and values through the account-qualified folder mapping. | -| `messages` | Copy | Preserve metadata, flags, previews, encryption, and new-message state. Map to a global `MessageId` when required. | -| `message_parts` | Copy | Preserve MIME-tree relationships, metadata, `data_location`, and every `data` BLOB. Legacy stores a body part in `data` at or below the 16 KiB threshold and on disk above it, regardless of whether the part is an attachment, so in-database attachment bytes stay BLOBs. | -| `threads` | Copy | Preserve existing message, root, and parent relationships. Do not recalculate threading in this migration. | -| `outbox_state` | Copy | Preserve send state, attempts, error timestamps, and error-state semantics. | -| `pending_commands` | Copy | Preserve command and serialized payload with the source account. Preserve account-qualified local IDs so execution resolves the mapped target records without redesigning queue behavior. | -| `messages_fulltext` | Rebuild | Rebuild from imported durable content, setting `docid` to the imported message's id so search joins resolve. Coverage equals messages whose body parts are locally present, because the legacy index was populated from text supplied at save time. Validate representative account and unified search parity against that set. | -| `notifications` | Copy | Preserve notification IDs and timestamps through the account-qualified message mapping. | -| `.db_att/` | Copy | Covers parts with `data_location = 2` (`ON_DISK`) only. Keep that content in the global file-backed directory. Do not move it into database BLOB storage, and do not promote in-database BLOBs to files. | - -For all account-local numeric IDs, target mappings preserve `(account_id, legacy_local_id)` and record a global domain -identifier only where the repository contract requires one. +| Legacy source | Classification | Global target and import rule | +|--------------------------------------------|----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `account_extra_values` | Copy | Preserve text/integer values and absence semantics, scoped to `AccountId`. | +| `folders` | Copy | Preserve every field. Map `(AccountId, legacy_local_id)` to one global `FolderId` and rewrite all folder references through that mapping. | +| `folder_extra_values` | Copy | Preserve unknown keys and values through the account-qualified folder mapping. | +| `messages` | Copy | Preserve metadata, flags, previews, encryption, and new-message state. Map `(AccountId, legacy_local_id)` to one global `MessageId` and rewrite dependent message references. | +| `message_parts` | Copy | Preserve MIME-tree relationships, metadata, `data_location`, and every `data` BLOB. Allocate an internal globally unique part key and rewrite `root`, `parent`, message root-part references, attachment-file lookup, and attachment URI resolution. Legacy stores a body part in `data` at or below the 16 KiB threshold and on disk above it, regardless of whether the part is an attachment, so in-database attachment bytes stay BLOBs. | +| `threads` | Rebuild | Use imported message threading headers to build account-scoped conversations across folders. Several legacy folder-local thread roots may become one global `ThreadId`, for example when Inbox and Sent contain one conversation. Preserve a compatibility mapping for each `(AccountId, legacy_folder_id, legacy_thread_root)` and rewrite threaded-list and thread-cache values. | +| `outbox_state` | Copy | Preserve send state, attempts, error timestamps, and error-state semantics. | +| `pending_commands` | Copy | Preserve command and serialized payload with the source account. Allocate an internal globally unique command key and translate serialized folder references before the command is executable. Do not expose its key outside persistence. | +| `messages_fulltext` | Rebuild | Rebuild from imported durable content. Keep its integer FTS `docid` as an internal search-document key mapped to `MessageId`. Do not store a UUID-backed `MessageId` directly as `docid`. Coverage equals messages whose body parts are locally present, because the legacy index was populated from text supplied at save time. Validate representative account and unified search parity against that set. | +| `notifications` | Copy | Rewrite the message relationship through the account-qualified message mapping. Preserve the timestamp and preserve `notification_id` only when it remains unique in the application notification namespace. Otherwise, allocate a replacement. | +| `.db_att/` | Copy | Covers parts with `data_location = 2` (`ON_DISK`) only. Keep that content in the global file-backed directory using the imported internal part mapping. Do not move it into database BLOB storage, and do not promote in-database BLOBs to files. | + +For all account-local numeric IDs, target mappings preserve `(AccountId, legacy_local_id)`. `FolderId`, `MessageId`, +and `ThreadId` are global domain identifiers when repository contracts require them. `ThreadId` represents an +account-scoped conversation and may contain messages from multiple folders. Message-part and pending-command keys are +internal persistence identifiers, but are globally unique and all dependent references are rewritten. The fts4 shadow tables `messages_fulltext_content`, `_segdir`, `_segments`, `_docsize`, and `_stat` are excluded. They are rebuilt with their virtual table and are not durable inputs. From 204d2823391f9d8c2f4f7729fc82028ceeae69ac Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Wolf-Martell=20Montwe=CC=81?= Date: Wed, 2 Sep 2026 15:21:43 +0200 Subject: [PATCH 4/6] docs(technical-design): add global database interruption handling, recovery, and cleanup procedures for migration (cherry picked from commit b54d7166bbcc4e63e180b75d319b3c38ece713ae) --- .../technical-designs/0003-global-database.md | 43 ++++++++++++++++--- 1 file changed, 38 insertions(+), 5 deletions(-) diff --git a/docs/engineering/technical-designs/0003-global-database.md b/docs/engineering/technical-designs/0003-global-database.md index 9289a62b52..f4521d5f8f 100644 --- a/docs/engineering/technical-designs/0003-global-database.md +++ b/docs/engineering/technical-designs/0003-global-database.md @@ -100,7 +100,7 @@ failure state. A migration gate holds startup, sync, and other background mail w and fail the migration. 5. Rebuild derived data, validate the imported database, and reopen it. 6. Establish the durable cutover state and switch repository bindings to the global implementation. -7. Remove confirmed legacy database and attachment artifacts. +7. Start post-cutover cleanup of all legacy database and attachment artifacts. The import writes folder and message identifier mappings before importing dependents. After all message records for an account are available, it builds cross-folder `ThreadId` memberships from their threading headers and records the @@ -112,10 +112,40 @@ legacyFolderId, UID)`, through the compatibility mapping until that support is i references use the global identifier model. Before step 6, global data is not visible to normal mail code. Any failure before that step keeps legacy storage -authoritative. A later retry starts with a new unpublished import. A cleanup failure after cutover leaves global storage -authoritative and records the remaining cleanup work. +authoritative. A later retry starts with a new unpublished import. -The legacy storage implementation stays in the codebase, unbound and unused, and a later release removes it. +### Interruption and recovery + +The migration must tolerate process termination at every step, including termination after the app is backgrounded. The +import database and copied attachments remain unpublished until validation and cutover complete. Each startup reads the +durable migration phase before binding mail repositories. + +If the process stopped before cutover, startup keeps legacy storage authoritative, removes the incomplete unpublished +database and copied artifacts, and starts a fresh import when the migration is retried. SQLite transactions protect +individual writes, while discarding the unpublished target prevents a partially imported database from being reused. + +Cutover commits the validated state as one durable operation. If that operation did not commit, startup uses legacy +storage. If it committed, startup uses the global store and may resume cleanup. Legacy deletion cannot begin before this +state is durable, so process termination cannot leave startup choosing between two authoritative stores. + +### Post-cutover cleanup + +Cutover atomically makes the global store authoritative and records cleanup as pending. Only then can an idempotent +cleanup job delete the legacy per-account databases and attachment directories. The job also deletes orphaned artifacts +left behind by accounts that were removed before migration. + +Cleanup is complete only after the job verifies that all known legacy artifacts are absent. If deletion fails or the app +stops during cleanup, the state remains pending and the job retries on a later startup or scheduled background run. A +retry resumes cleanup without repeating the data migration. The job never reads legacy mail into the global store and +mail repositories never bind to legacy storage after cutover. + +A pending cleanup does not block normal mail access. The migration is reported as successful, with a separate notice +that storage cleanup is incomplete and may temporarily use additional device storage. Cleanup failures and retry state +use non-sensitive error codes and do not include paths or account data. + +The app does not restore legacy storage after cutover because the global store may already contain new mail or user +actions. Switching back could discard those changes or create conflicting sources of truth. The legacy storage +implementation stays in the codebase, unbound and unused, until a later release removes it. ## Validation @@ -154,7 +184,10 @@ Automated tests cover: - schema creation, migration, and restart recovery - every durable table and attachment type in the inventory - POP3 archive gating and IMAP optional export +- process termination during every migration phase, including immediately before and after durable cutover +- rejection and removal of an incomplete unpublished database after restart - validation, interruption, retry, and cleanup failure +- cleanup of orphaned account artifacts, restart-safe retries, and completion verification - migration-gate behavior for startup and background work - report redaction - the guarantee that partial global data is never visible to normal callers @@ -167,5 +200,5 @@ Automated tests cover: - Which supported legacy schema versions require dedicated fixtures? - Which Room driver and locations apply on Android and JVM desktop? -- How should post-cutover cleanup retry a locked legacy artifact? +- How long must compatibility mappings support pre-cutover external message references after migration? From 9a4ce1ffdb0916120762d9e505ab8c84425ef008 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Wolf-Martell=20Montwe=CC=81?= Date: Wed, 2 Sep 2026 15:29:58 +0200 Subject: [PATCH 5/6] docs(technical-design): document Android foreground service for global database migration (cherry picked from commit 65b1fb8058e6c4242155d4bd50d8c41094b8a1ab) --- docs/engineering/rfcs/0007-global-database.md | 14 ++++++++----- .../technical-designs/0003-global-database.md | 21 +++++++++++++++++++ 2 files changed, 30 insertions(+), 5 deletions(-) diff --git a/docs/engineering/rfcs/0007-global-database.md b/docs/engineering/rfcs/0007-global-database.md index cb7d694bce..5945e8876e 100644 --- a/docs/engineering/rfcs/0007-global-database.md +++ b/docs/engineering/rfcs/0007-global-database.md @@ -102,12 +102,14 @@ codebase, unbound and unused, until a later release removes it. ### Migration gate and reporting -The migration surfaces progress, failure feedback, retry, and local report export in a dedicated migration screen. That -screen could reuse the existing Android database-migration activity or introduce a suitable replacement. Startup -routing and background mail work respect the migration gate. No mail work bypasses it. +The migration surfaces progress, failure feedback, retry, and local report export in a dedicated migration screen. On +Android, a user-initiated data synchronization foreground service continues the migration when the app leaves the +foreground and provides progress and completion through a persistent notification. The in-app screen and notification +show the same migration state. -Progress information, failure reports, and logs exclude personally identifiable data. They stay on the device and are -never uploaded. +Startup routing and background mail work respect the migration gate. No mail work bypasses it. Progress information, +failure reports, notifications, and logs exclude personally identifiable data. They stay on the device and are never +uploaded. ### Platform support @@ -153,6 +155,8 @@ Validation before cutover and the archive from step 1 cover that risk instead. - Migration requires time and storage, especially for downloaded attachments. - The user or operating system can terminate the app during migration, leaving an incomplete global database that must never become authoritative. The migration requires interruption-safe state tracking and restart recovery. +- Android can limit foreground service execution, and a user can explicitly stop the app. Long migrations therefore + cannot rely on the service alone and must remain safe to retry after interruption. - Rebuilding the search index during migration adds time proportional to locally stored mail. - A device without enough free space cannot migrate until space is freed. - After cutover, locally stored mail exists only in the global database and in any archive the user created. diff --git a/docs/engineering/technical-designs/0003-global-database.md b/docs/engineering/technical-designs/0003-global-database.md index f4521d5f8f..6fa90084f7 100644 --- a/docs/engineering/technical-designs/0003-global-database.md +++ b/docs/engineering/technical-designs/0003-global-database.md @@ -114,6 +114,24 @@ references use the global identifier model. Before step 6, global data is not visible to normal mail code. Any failure before that step keeps legacy storage authoritative. A later retry starts with a new unpublished import. +### Android execution and progress + +The user starts migration from the visible migration screen. Android then runs the migration orchestration in a +foreground service with the `dataSync` service type. The service owns the operation independently of the activity and +continues when the app is backgrounded or its task is dismissed. It enters the foreground immediately and keeps an +ongoing notification visible while migration is running. + +The service writes non-sensitive phase, progress, completion, and failure state to one durable migration-state source. +The migration screen observes that source and updates continuously while visible. The foreground notification reads the +same state, displays current progress, and opens the migration screen when tapped. When migration finishes, the service +stops foreground execution and posts a completion or actionable failure notification. + +The Android application declares only the foreground service permissions required for a user-initiated `dataSync` +operation. Starting the service must comply with the platform's background-start restrictions. Platform time limits, +system process termination, and an explicit user force-stop can still interrupt it. The service therefore improves +continuity but does not replace the recovery guarantees below. After a force-stop, work cannot continue until Android +allows the app to run again. + ### Interruption and recovery The migration must tolerate process termination at every step, including termination after the app is backgrounded. The @@ -184,6 +202,9 @@ Automated tests cover: - schema creation, migration, and restart recovery - every durable table and attachment type in the inventory - POP3 archive gating and IMAP optional export +- migration continuing after the activity is backgrounded or its task is dismissed +- consistency between in-app progress, foreground notification progress, and the durable migration state +- foreground service completion, failure, platform timeout, and user-stop handling - process termination during every migration phase, including immediately before and after durable cutover - rejection and removal of an incomplete unpublished database after restart - validation, interruption, retry, and cleanup failure From a78744626f8fdd88410b1a40331c82aaa448e93f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Wolf-Martell=20Montwe=CC=81?= Date: Wed, 2 Sep 2026 15:36:41 +0200 Subject: [PATCH 6/6] docs(technical-design): refine migration plan with storage pre-check and headroom validation --- .../technical-designs/0003-global-database.md | 28 ++++++++++++++++--- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/docs/engineering/technical-designs/0003-global-database.md b/docs/engineering/technical-designs/0003-global-database.md index 6fa90084f7..9094ecf6d3 100644 --- a/docs/engineering/technical-designs/0003-global-database.md +++ b/docs/engineering/technical-designs/0003-global-database.md @@ -91,10 +91,10 @@ The migration runs before normal mail access is available. A dedicated migration Android migration activity or introduce a suitable replacement, shows non-sensitive progress and a clear completion or failure state. A migration gate holds startup, sync, and other background mail work until migration completes or fails. -1. Create and verify the required RFC 0008 archive for POP3 accounts. The user may decline it after an explicit warning - and continue at their own risk. IMAP export is optional. -2. Check available storage against the required headroom and fail with an actionable error when it is insufficient. - Then create an unpublished global database. +1. Calculate the required storage headroom and compare it with available space. If space is insufficient, block the + migration and report how much additional space is needed. +2. Create and verify the required RFC 0008 archive for POP3 accounts. The user may decline it after an explicit warning + and continue at their own risk. IMAP export is optional. Then create an unpublished global database. 3. Read legacy databases and attachment directories without modifying them. Import every durable record. 4. Copy each attachment to its target and validate it. If validation fails, record the failure in the migration result and fail the migration. @@ -114,6 +114,24 @@ references use the global identifier model. Before step 6, global data is not visible to normal mail code. Any failure before that step keeps legacy storage authoritative. A later retry starts with a new unpublished import. +### Pre-flight storage check + +The pre-flight check runs as the first foreground migration phase, before creating the unpublished database. It +inventories the legacy databases and file-backed attachments, then calculates the additional space needed while legacy +and global storage coexist. Required headroom includes the estimated global database, copied attachments, rebuilt search +data, SQLite transaction and temporary-file growth, and a safety margin. If an RFC 0008 archive is written to the same +volume, its estimated size is also included. + +Migration starts only when available space on the target volume meets the calculated headroom. If it does not, the +migration screen shows the required, available, and additional space in user-readable units and provides a retry action. +A warning with an option to continue is not sufficient because running out of space makes successful migration +impossible. + +Available space can change after pre-flight. Write failures caused by exhausted storage fail the unpublished import +without changing the authoritative store. Restart recovery removes the incomplete target before another pre-flight check +and retry. The estimator and safety margin are verified against migration fixtures representing the largest supported +legacy schema and attachment layouts. + ### Android execution and progress The user starts migration from the visible migration screen. Android then runs the migration orchestration in a @@ -202,6 +220,8 @@ Automated tests cover: - schema creation, migration, and restart recovery - every durable table and attachment type in the inventory - POP3 archive gating and IMAP optional export +- headroom calculation, insufficient-space blocking, and user-visible required-space reporting +- storage exhaustion after pre-flight without publishing the incomplete target - migration continuing after the activity is backgrounded or its task is dismissed - consistency between in-app progress, foreground notification progress, and the durable migration state - foreground service completion, failure, platform timeout, and user-stop handling