Skip to content

Add IMAP support - #445

Merged
jan-janssen merged 45 commits into
mainfrom
imap-support
Aug 24, 2026
Merged

Add IMAP support#445
jan-janssen merged 45 commits into
mainfrom
imap-support

Conversation

@jan-janssen

@jan-janssen jan-janssen commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Example

from gmailsorter import Imap

imap = Imap(
    host="imap.gmail.com",
    port=993,
    username="<your email>",
    password="<your password>",
    connection_str="sqlite:///email.db",
)
print(imap.labels)

# download messages and store them in database
imap.update_database(quick=False)

# get messages from database
imap.get_all_emails_in_database()

# train random forest model
imap.fit_machine_learning_model_to_database(
    n_estimators=100,
    max_features=400,
    random_state=42,
    bootstrap=True,
    include_deleted=False,
    max_workers=1,
)

# sort emails
imap.filter_messages_from_server(label="mailsortinbox", recommendation_ratio=0.9)

# close connection
imap.close()

jan-janssen and others added 20 commits July 25, 2026 08:37
Documents extracting the shared fetch-store-train-predict-move loop
into base/mail.py so a new imap/ package can reuse it alongside the
existing google/ backend, plus CLI and CI integration test plans.
Nine-task TDD plan: extract AbstractMailBox from GoogleMailBase, add
gmailsorter/imap/ (message, authentication, mail), wire up Imap in
local.py, add the gmailsorter-imap CLI, and add a GreenMail-backed CI
integration test plus docs.
Needed a project-local .venv to install and test gmailsorter in
isolation rather than the shared base conda environment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tartup lag

GitHub Actions services: containers are only guaranteed to be running,
not to have finished binding their ports, so a single 2s connection
attempt in _imap_server_available() could cause the integration test
to silently skip in CI while GreenMail is still starting up. Retry up
to 5 times with a 1.5s delay between attempts, keeping the worst-case
wait for a genuinely absent server well under 30s.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes the black --check CI failure on gmailsorter/google/message.py and
gmailsorter/imap/message.py, the ruff import-sort error in
gmailsorter/google/message.py, and ruff-format drift across several test
files. The mock-folder helper in tests/test_imap_integration_units.py is
restructured to an early-return default so that black and ruff format
agree on it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RFC822 is equivalent to BODY[], which implicitly sets the \Seen flag on
every fetched message (RFC 3501). update_database() fetches every message
in every folder, so it silently marked the user's entire mailbox as read.
BODY.PEEK[] returns the identical raw message without touching \Seen.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_parse_list_entry crashed with an unguarded match.group() AttributeError on
three legal LIST responses: an unquoted NIL hierarchy delimiter, a mailbox
name sent as an IMAP literal (which imaplib returns as a tuple rather than
bytes), and imaplib returning [None] for an empty mailbox list. Because
_get_label_translate_dict runs from AbstractMailBox.__init__, any of these
turned constructing Imap into an opaque failure. Unparseable entries are now
skipped instead of raising.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
update_database() scans every folder and filter_messages_from_server can
recommend moving mail into any label the model was trained on, so treating
every selectable folder as a sorting label let the classifier learn to file
mail into Trash, Spam, Sent or Drafts. Special folders are now detected via
the RFC 6154 special-use attributes and, for servers which do not advertise
them, a short list of unambiguous folder names (matched exactly and
case-insensitively, also against the leaf of a nested name).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The COPY+STORE fallback used for servers without MOVE finished with a bare
EXPUNGE, which permanently removes every \Deleted-flagged message in the
folder - including messages the user's own mail client had flagged but not
yet expunged. When the server advertises UIDPLUS the RFC 4315 UID EXPUNGE
command is now used to expunge only the message being moved.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
setUpClass skipped whenever no IMAP server answered, which is right for a
local contributor without Docker but meant the imap-integration CI job
reported green even if GreenMail never came up. With IMAP_INTEGRATION_REQUIRED
set - which the workflow now does - an unreachable server raises instead of
skipping. Without it the previous clean-skip behaviour is unchanged.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Nothing ever logged out of the IMAP connection and a dropped connection left
the Imap instance permanently unusable, which matters because IMAP servers
commonly time out idle connections and update_database() can keep one
connection busy for a long time on a large mailbox.

ImapMailBase now offers close() (tolerating an already dead connection) and
works as a context manager. The four network-calling hooks run through
_run_with_reconnect, which retries the operation exactly once after calling
the overridable _reconnect() hook; only Imap, which knows the connection
details, implements it, and the original IMAP4.abort is re-raised otherwise.
Gmail's behaviour is unchanged - none of this is on AbstractMailBox.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jan-janssen
jan-janssen marked this pull request as draft July 25, 2026 08:40
@codecov

codecov Bot commented Jul 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.75248% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 97.50%. Comparing base (a848892) to head (5fee179).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
gmailsorter/imap/__main__.py 96.77% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #445      +/-   ##
==========================================
+ Coverage   96.73%   97.50%   +0.77%     
==========================================
  Files          29       35       +6     
  Lines        1163     1484     +321     
==========================================
+ Hits         1125     1447     +322     
+ Misses         38       37       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jan-janssen and others added 6 commits July 26, 2026 17:56
MLStripper was refactored into base/message.py as private _MLStripper,
exposed via strip_html_tags(). Update the test to match.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Covers previously-missing branches: get_from/get_date with absent
headers, multipart content parsing, thread-id fallback chain, the
get_email_dict exception handler, IMAP LIST entry edge cases (str
entries, latin-1 fallback, undecodable literal names), label-dict
failure on a non-OK LIST, search/fetch/move/copy failure paths that
raise RuntimeError or return empty results, and the CLI's print_help
branch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jan-janssen
jan-janssen marked this pull request as ready for review August 24, 2026 04:41
@jan-janssen
jan-janssen merged commit aa9e6d0 into main Aug 24, 2026
14 checks passed
@jan-janssen
jan-janssen deleted the imap-support branch August 24, 2026 04:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant