[Bug] IMAP sync can silently miss individual INBOX messages while newer messages sync successfully

[handkerchief333]

Steps to reproduce

I have not yet found a deterministic way to trigger the initial message loss on demand. The issue occurred during normal use of an existing IMAP account.

The following steps reproduce and verify the faulty state once it occurs:

1. Configure an IMAP account in Nextcloud Mail and let it synchronize normally.

2. Receive multiple messages in the INBOX.

3. Observe a state where:

   • message A is synchronized to Nextcloud Mail
   • message B exists on the IMAP server but is missing in Nextcloud Mail
   • a later message C is synchronized successfully to Nextcloud Mail

4. Verify message B using another IMAP client or the provider's webmail.

   In my case the missing message was visible in:

   • Hostpoint Webmail
   • K-9 Mail

   but not in Nextcloud Mail.

5. Refresh the INBOX manually in Nextcloud Mail several times. The missing message does not appear.

6. Run a manual synchronization:

   php occ mail:account:sync -vvv <account-id>

   The sync completes successfully and reports for the INBOX:

   Performing a combined sync
   Combined sync finished
   Fetching new and changed messages after sync
   Found 0 new messages
   Found 0 changed messages
   

   The missing message still does not appear.

7. Run a forced synchronization:

   php occ mail:account:sync -vvv --force <account-id>

   The missing message still does not appear.

8. In Nextcloud Mail, use:

   INBOX -> ... -> Repair folder
   

   The missing message still does not appear.

9. On the IMAP server, create a temporary folder, for example TestMail.

10. Move the missing message from INBOX to TestMail.

11. Refresh Nextcloud Mail. The previously missing message now appears immediately in the TestMail folder.

12. Move the same message from TestMail back to INBOX.

13. Refresh Nextcloud Mail again. The message now also appears correctly in the Nextcloud INBOX.

---

Threading/conversation view was also checked and ruled out as the cause.

The important condition is that a newer message can synchronize successfully while an older individual message remains permanently absent from Nextcloud Mail.

I have not yet been able to intentionally reproduce the initial creation of this inconsistent state from a clean mailbox. If it happens again, I can preserve the affected state before applying the move workaround and collect additional UID/QRESYNC diagnostics.

Expected behavior

Nextcloud Mail should reliably synchronize every message that exists in the IMAP mailbox.

If the local synchronization state becomes inconsistent, Nextcloud Mail should detect and repair the inconsistency automatically.

In particular:

• An IMAP message must not remain permanently missing while later messages synchronize successfully.
• Manual synchronization should detect messages that exist on the IMAP server but are missing from the local Nextcloud Mail cache/database.
• A forced synchronization should be able to recover such missing messages.
• Repair folder or another repair mechanism should be able to detect server-side messages that are missing locally.
• Nextcloud Mail should not report Found 0 new messages while the local mailbox is incomplete compared with the IMAP server.
• If the synchronization state cannot be repaired automatically, the user should receive a visible warning that the mailbox may be incomplete.

For a mail client, especially when used for business email, the local mailbox should never silently differ from the authoritative IMAP mailbox.

Actual behavior

An individual message was silently missing from the Nextcloud Mail INBOX although it existed on the IMAP server.

At the same time, newer messages continued to synchronize successfully.

There was no synchronization error, warning, or other indication that the local INBOX was incomplete.

The missing message was visible in two other clients using the same mailbox:

• Hostpoint Webmail
• K-9 Mail

The following actions did not recover the message:

• manually refreshing Nextcloud Mail several times
• logging out and back into Nextcloud
• checking the complete conversation/thread
• php occ mail:account:sync -vvv <account-id>
• php occ mail:account:sync -vvv --force <account-id>
• INBOX -> Repair folder

During the manual synchronization, Nextcloud Mail reported:

Performing a combined sync
Combined sync finished
Fetching new and changed messages after sync
Found 0 new messages
Found 0 changed messages

although the IMAP server contained a message that was missing from Nextcloud Mail.

The message itself was clearly not the problem:

1. I moved the missing message from INBOX to a newly created IMAP folder called TestMail.
2. After refreshing Nextcloud Mail, the message immediately appeared in TestMail.
3. I moved exactly the same message back from TestMail to INBOX.
4. After refreshing Nextcloud Mail again, the message also appeared correctly in INBOX.

Therefore Nextcloud Mail was able to fetch, parse, store and display the message. The problem appears to have been related specifically to the synchronization state/cache of the original INBOX.

The particularly dangerous aspect is that synchronization appeared completely healthy. Newer emails continued to arrive, so the user had no reason to suspect that an individual earlier message was missing.

Mail app version

5.11.3

Nextcloud version

33.0.6

Mailserver or service

Hostpoint

Operating system

No response

PHP engine version

PHP 8.3

Nextcloud memory caching

No response

Web server

None

Database

None

Additional info

Environment

Nextcloud AIO running in Docker on Ubuntu.

IMAP provider

Hostpoint, Switzerland

Protocol: IMAP
Server: imap.mail.hostpoint.ch
Port: 993
Security: SSL

The affected account had already been in regular production use for some time and had previously synchronized normally.

IMAP diagnostics

The account connection test succeeds:

php occ mail:account:diagnose <account-id>

Result:

Account 1 uses protocol: imap
Testing IMAP connection...
Server: imaps://imap.mail.hostpoint.ch:993
Login successful
Capabilities: BINARY, CATENATE, CHILDREN, CONDSTORE, CONTEXT, ENABLE, ESEARCH, ESORT, I18NLEVEL, ID, IDLE, IMAP4REV1, LIST-EXTENDED, LIST-STATUS, LITERAL+, LOGIN-REFERRALS, MOVE, MULTIAPPEND, NAMESPACE, NOTIFY, PREVIEW, QRESYNC, QUOTA, SASL-IR, SAVEDATE, SEARCHRES, SNIPPET, SORT, SPECIAL-USE, STATUS, THREAD, UIDPLUS, UNSELECT, URL-PARTIAL, WITHIN
IMAP connection test passed

Relevant supported capabilities include:

CONDSTORE
QRESYNC
UIDPLUS
IDLE
MOVE

Relevant synchronization output

Manual synchronization:

php occ mail:account:sync -vvv <account-id>

Relevant INBOX output:

[debug] Syncing 73
[debug] Locking mailbox 73 for new messages sync
[debug] Locking mailbox 73 for changed messages sync
[debug] Locking mailbox 73 for vanished messages sync
[debug] Running partial sync for 73 with criteria 42
[debug] partial sync 1:INBOX - get all known UIDs took 0s
[debug] Performing a combined sync
[debug] Combined sync finished
[debug] Fetching new and changed messages after sync
[debug] Found 0 new messages
[debug] Found 0 changed messages
[debug] partial sync 1:INBOX - get new messages via Horde took 0s
[debug] partial sync 1:INBOX - persist new messages took 0s
[debug] Reusing cached sync response
[debug] partial sync 1:INBOX - get changed messages via Horde took 0s
[debug] partial sync 1:INBOX - persist changed messages took 0s
[debug] Reusing cached sync response
[debug] partial sync 1:INBOX - get vanished messages via Horde took 0s
[debug] partial sync 1:INBOX - delete vanished messages took 0s
[debug] partial sync 1:INBOX took 0s

No error was reported.

A later email sent to the same mailbox synchronized successfully while the affected earlier message remained missing.

Suspected UID/cache inconsistency

This is only a hypothesis and not a confirmed root cause.

The observed behavior appears consistent with a possible gap in the locally cached IMAP UID state.

For example, the server could contain:

UID 1000   existing message
UID 1001   message missed by Nextcloud
UID 1002   later synchronized message
UID 1003   later synchronized message

while Nextcloud Mail locally contains:

UID 1000
UID 1002
UID 1003

If subsequent synchronization uses the highest locally known UID as a high-water mark, the missing lower UID could potentially remain undiscovered.

This hypothesis would also be consistent with the workaround:

• moving the message to another mailbox causes it to receive a mailbox-specific UID there and Nextcloud detects it
• moving it back to INBOX causes it to be inserted into INBOX again and Nextcloud detects it there as well

The move workaround means that the original inconsistent UID state of this particular message no longer exists.

Why this is critical

This is potentially a serious reliability/data-integrity issue for business mailboxes.

A complete synchronization failure is visible to the user.

A single silently missing message is much more dangerous because:

• newer emails continue to arrive
• the mailbox looks healthy
• no synchronization error is displayed
• the user has no way to know that a message is missing

In my case the problem was only discovered because I happened to compare Nextcloud Mail with another IMAP client.

If this happens again, I can preserve the inconsistent state before moving the message and provide additional diagnostics.

[ChristophWurst]

Thanks for the report and the write-up

I have not yet been able to intentionally reproduce the initial creation of this inconsistent state from a clean mailbox. If it happens again, I can preserve the affected state before applying the move workaround and collect additional UID/QRESYNC diagnostics.

That would be very helpful!

Your observation about the behavior and missed repair is correct. It's because we assume there are no gaps. A synchronization will only look at UIDs higher than the currently highest known one.
It's possible to take all known UIDs and check if they still exist on the server of Horde vanished sync criteria but the other way around - finding what's missing in that local set - is not implemented as far as I know. But technically it would be possible.

[handkerchief333]

@ChristophWurst Could #13507 potentially turn a sync failure caused by a FETCH result without a UID into a silent UID gap?

It now drops the FETCH result without a UID while allowing the remaining messages to be processed. If later messages are persisted and highestKnownUid advances past the skipped message, could that message then become permanently invisible, given that lower UID gaps are not subsequently checked?

My original occurrence was on 5.11.3, so #13507 itself cannot have caused it. I'm wondering whether the underlying condition addressed by #13507 could be related, and whether the new behaviour could potentially create the same inconsistent state silently.