Skip to content

fix(email): call the LLM to triage urgent emails instead of short-circuiting to the heuristic (#5506)#5530

Open
maxmilian wants to merge 1 commit into
odysseus-dev:devfrom
maxmilian:fix/5506-email-urgency-llm
Open

fix(email): call the LLM to triage urgent emails instead of short-circuiting to the heuristic (#5506)#5530
maxmilian wants to merge 1 commit into
odysseus-dev:devfrom
maxmilian:fix/5506-email-urgency-llm

Conversation

@maxmilian

Copy link
Copy Markdown
Contributor

Summary

The hourly Email Tags task (action_check_email_urgency) is supposed to LLM-triage new unread mail honouring the user's urgent_email_prompt, but on dev it never calls the LLM at all. The per-item loop computes a keyword heuristic verdict and then continues before the LLM block, making the entire ~120-line classification path — the only code that reads urgent_email_prompt — unreachable dead code. Every email is scored by regex and the urgency prompt has no effect. A second, compounding defect: even with the continue removed, the classify call caps max_tokens=220, so reasoning models (e.g. gemma4:31b) spend their budget "thinking" and hit finish_reason=length with an empty completion. This PR removes the short-circuit, keeps the heuristic as a graceful (uncached) fallback, and raises the token budget to 2048.

Target branch

  • This PR targets dev, not main. All PRs land in dev; main is curated by the maintainer at each release. If your PR is on main by accident, click "Edit" on this PR and change the base.

Linked Issue

Fixes #5506

Type of Change

  • Bug fix (non-breaking — fixes a confirmed issue)
  • New feature (non-breaking — adds new behaviour)
  • Breaking change (changes or removes existing behaviour)
  • Refactor / cleanup (behaviour unchanged)
  • Documentation only
  • CI / tooling / configuration

Checklist

  • I searched open issues and open PRs — this is not a duplicate. (No open PR touches check_email_urgency / email urgency / urgent_email_prompt.)
  • This PR targets dev
  • My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
  • I actually ran the app (docker compose up or uvicorn app:app) and verified the change works end-to-end. Type-checks and unit tests are not enough. — Backend scheduled-task logic, no UI surface. Seeing the fix in-app requires a live IMAP mailbox + LLM endpoint, so I verified two ways instead: (1) booted uvicorn app:app against this branch — the app starts cleanly and the builtin_actions module loads in the real process (Email built-in MCP registers, / → 302, /api/tasks → 401); (2) the regression tests below drive the real action_check_email_urgency through its full IMAP-scan → per-item → LLM-classify path with only the IMAP socket and LLM HTTP faked — i.e. the production code path minus the network.

How to Test

The change is in src/builtin_actions.py (action_check_email_urgency). Three regression tests in tests/test_check_email_urgency_llm.py exercise the real function; to confirm they genuinely catch the bug, run them against dev (red) and this branch (green):

uv venv .venv && uv pip install -r requirements.txt   # if not already set up
# On this branch — all green:
.venv/bin/python -m pytest tests/test_check_email_urgency_llm.py -q
# Prove they catch the bug — revert just the source, keep the tests:
git show dev:src/builtin_actions.py > src/builtin_actions.py
.venv/bin/python -m pytest tests/test_check_email_urgency_llm.py -q   # 3 failed
git checkout -- src/builtin_actions.py

The tests assert, over one fresh unread inbox message: (1) the LLM classifier is actually invoked (red on dev — the continue skips it); (2) the classify call requests ≥1024 max_tokens (red on dev — 220); (3) after a transient LLM failure the uid is left uncached, so the next hourly scan re-classifies instead of skipping it as already-triaged.

Live sanity check (optional): link an IMAP mailbox, set an urgent_email_prompt, configure a utility model, run the Email Tags task, and confirm data/scheduled_emails.db → email_tags.model_used now records the model (not empty) and the reason fields reflect the prompt rather than keyword matches.

Visual / UI changes — REQUIRED if you touched anything that renders

N/A — backend only. No buttons, icons, CSS, HTML, SVG, or static/js/ changes.

…cuiting to the heuristic

`action_check_email_urgency` (the hourly "Email Tags" task) computed a keyword
heuristic verdict and then `continue`d before the LLM classification block,
making the entire ~120-line LLM path — and the only use of the user's
`urgent_email_prompt` setting — dead code. Every email was scored by regex; the
urgency prompt had no effect. A second defect: even reachable, the classify call
capped `max_tokens=220`, so reasoning models spent their budget "thinking" and
hit finish_reason=length with an empty completion.

- Remove the hardcoded `continue` short-circuit so the LLM classifies each fresh
  unread email honouring `urgent_email_prompt`.
- Keep the heuristic as a graceful fallback for the current scan's aggregation,
  tagging, and notify — but do NOT cache it: only a successful LLM verdict is
  persisted, so a transient LLM failure is retried next scan instead of sticking
  the heuristic at the current TRIAGE_VERSION.
- Bump the classify budget 220 -> 2048 so reasoning models can emit their JSON.

Regression tests drive the real action end-to-end (IMAP scan -> per-item ->
classify) with only the IMAP socket and LLM HTTP faked: the LLM is invoked for a
fresh unread email, the budget is large enough, and a transient LLM failure
leaves the uid uncached for retry. All three are red on dev, green here.

Fixes odysseus-dev#5506

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the ready for review Description complete — ready for maintainer review label Jul 14, 2026
@maxmilian maxmilian marked this pull request as ready for review July 14, 2026 15:04
@swisscheez2

Copy link
Copy Markdown

I'm glad I didn't report a non issue. Thanks for fixing it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for review Description complete — ready for maintainer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Email Tags (check_email_urgency) never calls the LLM — urgent_email_prompt is ignored; classification is keyword-heuristic only

2 participants