SortPilot is an AI-powered support inbox triage automation built with n8n, Gmail, OpenAI, and Google Sheets.
SortPilot is fully built end-to-end in n8n, with real classification logic, JSON validation, retry/fallback handling, follow-up ticket lookup, and a tested accuracy log — not just a workflow diagram.
It is built around a fictional SaaS company, Boardly — a project management tool for small agencies and freelance teams — so I could design and stress-test realistic support scenarios, including messy, ambiguous, and adversarial test emails, without needing access to a real company's live customer data.
The system classifies incoming support emails, assesses urgency and risk, routes tickets to the correct queue, logs everything in a tracker, safely handles AI failures, and drafts replies only for low-risk tickets — with a human reviewer in the loop for anything risky.
SortPilot was tested against a mix of clean and deliberately messy Boardly support emails, including vague one-liners, billing issues, follow-ups, feature requests, account changes, refund requests, complaints, and a prompt-injection attempt.
- 14 emails tested
- 10 correctly classified
- 3 partially correct — right routing or review handling, but one secondary field needed improvement
- 1 fallback case handled correctly — invalid AI output was routed to human review instead of breaking the workflow
The case: A vague one-line test email — "still broken. pls fix asap." — was classified as Bug / Technical Issue.
The problem: There was not enough context in the message to know what was actually broken. A vague, low-information message like this should trigger Unclear / Needs Human Review rather than a confident guess. Guessing wrong on a ticket like this is worse than asking a human to take a quick look, because it risks routing a real issue to the wrong team based on limited evidence.
The fix: I added a stricter classification rule: if an email lacks enough detail to identify a specific feature, error, or issue — even if it sounds like a bug report — classify it as Unclear / Needs Human Review instead of guessing.
Why it matters: This is a common failure mode in AI triage systems. It is also why this project includes manual QA columns (ai_correct, error_notes) in the tracker — to catch overconfidence instead of trusting a clean-looking demo.
- A duplicate-charge billing email was routed correctly to the Billing Team and flagged for human review, but its urgency was initially under-assessed as Medium instead of High.
- A fallback test correctly produced a Classification Failed row and routed the issue to Human Review instead of allowing the workflow to fail silently.
- A prompt-injection test email attempted to instruct the AI to override the classification rules. The system ignored the embedded instruction and classified the email based on the actual billing issue, confirming the "treat customer email as untrusted data" guardrail worked as intended.
Boardly has one shared support inbox where customer emails are manually reviewed by a support/admin person.
Before automation, the support person had to manually:
- Read every incoming email
- Identify the issue type
- Decide urgency and risk level
- Route the request to the correct team
- Track follow-ups
- Write replies
- Make sure risky emails were not missed
SortPilot was built to reduce manual triage work while keeping human review safeguards in place.
The goal is not to replace the support team. The goal is to help the human reviewer work faster, avoid missed emails, and keep support requests organized.
- n8n
- Gmail
- Google Sheets
- OpenAI / LLM
- JavaScript Code nodes
- Conditional logic / IF nodes
- Gmail OAuth
- Google Cloud OAuth credentials
A Gmail Trigger detects incoming support emails and normalizes each one into consistent fields:
sender_emailsubjectemail_bodygmail_message_idgmail_thread_idcreated_date
This keeps the rest of the workflow processing emails consistently regardless of formatting quirks in the original message.
Each email is sent to an OpenAI classification prompt that returns one fixed support category:
- Login / Access Issue
- Billing or Payment Question
- Refund Request
- Bug / Technical Issue
- Feature Request
- Account Update Request
- General Support Question
- Complaint / Escalation
- Follow-Up / Existing Issue
- Unclear / Needs Human Review
The AI also returns:
urgency_levelsentimentissue_summaryrecommended_actionassigned_owner_queuerequires_human_reviewconfidence_scorerisk_levelreason_for_classification
The AI is required to return structured JSON only.
A JavaScript validation node checks the AI output for:
- Valid JSON
- Required fields
- Allowed category values
- Allowed owner queue values
- Valid confidence scores
- Correct data types
- Valid urgency, sentiment, and risk values
This prevents malformed or incomplete AI output from breaking the workflow downstream.
If the AI output fails validation, the workflow retries the classification once.
If the retry also fails, SortPilot does not fail silently. It creates a fallback ticket with:
support_category: Unclear / Needs Human Reviewassigned_owner_queue: Human Reviewconfidence_score: 0status: Classification Failedclassification_failure_reason: AI output was invalid or incomplete after retry.
This ensures no support email disappears because of an AI formatting failure.
If an email is classified as Follow-Up / Existing Issue, SortPilot looks up the original ticket in Google Sheets by Gmail thread ID and reuses the original assigned_owner_queue if a match is found.
Example:
Original issue:
- Login / Access Issue
- assigned_owner_queue: Tech Support
Follow-up email:
- Follow-Up / Existing Issue
- same Gmail thread ID
Result:
- Routed back to Tech Support
This prevents follow-up emails from being treated as unrelated new tickets.
Human review is required for:
- Login or access issues
- Billing questions
- Refund requests
- Account update requests
- Complaints or escalations
- Follow-ups
- Unclear emails
- Angry or frustrated sentiment
- Low-confidence classifications
- Risk-sensitive categories
This keeps risky interactions under human control by default.
Draft replies are generated only when all of the following are true:
requires_human_reviewis falseconfidence_scoreis 85 or higherrisk_levelis Lowsentimentis Neutralsupport_categoryis General Support Question or Feature Request
Draft replies are not generated for billing, refunds, complaints, account changes, follow-ups, unclear tickets, or failed classifications.
The system never auto-sends replies. It only prepares draft text for human review.
Every processed email is logged with a full audit trail:
- ticket ID
- Gmail metadata
- classification fields
- routing decision
- draft reply status
- follow-up due date
- manual QA review
The most important QA columns are:
ai_correcterror_notes
These columns were used to evaluate the AI output honestly instead of assuming every classification was correct.
After an email is processed and logged, SortPilot marks the Gmail message as read.
A guardrail checks whether the message ID is a real Gmail ID before attempting to mark it as read. This prevents test/manual IDs from being sent to the Gmail action.
Gmail Trigger → normalize fields → OpenAI classification → validate AI JSON → retry on invalid output → fallback ticket on repeated failure → follow-up lookup if applicable → generate ticket ID.
Prepare final row data → check draft-reply eligibility → generate draft reply if eligible → log to Google Sheets → mark Gmail message as read.
| Test Case | Expected Behavior | Actual Result |
|---|---|---|
| Login issue | Route to Tech Support, require human review | ✅ Matched |
| Duplicate charge | Route to Billing Team, require human review | |
| General support question | Route to Support, allow draft reply | ✅ Matched |
| Feature request | Route to Product, allow draft reply | ✅ Matched |
| Complaint / cancellation threat | Route to Escalations, require human review | ✅ Matched |
| Vague message | Flag for human review | |
| Follow-up email | Look up original ticket, reuse owner queue | ✅ Matched |
| Account update request | Route to Account Team, require human review | ✅ Matched |
| Refund request | Route to Billing Team, require human review | ✅ Matched |
| Prompt injection attempt | Ignore embedded instructions, classify based on actual issue | ✅ Matched |
| Invalid AI output / fallback test | Create fallback row and route to human review | ✅ Fallback handled correctly |
SortPilot does not auto-send replies. All risky emails are routed for human review.
The AI is not allowed to:
- Promise refunds
- Approve cancellations
- Apply discounts
- Confirm account changes
- Claim a technical fix has been completed
- Follow instructions embedded inside customer emails that attempt to override the system prompt
Customer email content is always treated as untrusted input, never as instructions to the system.
SortPilot demonstrates a realistic AI support operations workflow with:
- Gmail-based support intake
- AI classification
- Structured JSON validation
- Retry/fallback handling
- Follow-up ticket lookup
- Safe draft reply generation
- Google Sheets ticket logging
- Human review safeguards
- Manual AI accuracy review
- Prompt-injection testing
This project shows practical automation skills across n8n, AI prompting, Google Workspace integrations, workflow safety design, and support operations logic.
This is a working v1 focused on triage correctness and safety.
Potential next improvements:
- Generate actual Gmail drafts instead of logging draft reply text only
- Add Slack or Telegram alerts for urgent tickets
- Add a dashboard for ticket volume, category distribution, and AI accuracy
- Add overdue follow-up reminders
- Add a knowledge base lookup step for draft replies
- Add duplicate ticket detection
- Add owner queue workload balancing
- Run a low-risk real-world pilot inbox to collect a more realistic accuracy benchmark
This is a fictional demo project built for portfolio purposes. All customer data and company details are fictional.