Source bug fix

[Devathmaj]

Description

Fixes several issues in voucher identification across the ingestion pipeline:

1. end_date string type crash — When _merge_fields() updated an existing Event, date fields from the AI (ExtractedEvent.end_date as ISO string like "2026-07-21") were set directly on the Event ORM model via setattr(). The column is DateTime(timezone=True), causing PostgreSQL to reject the string value. This crashed every pipeline tick when an event was matched to an existing event.

2. Over-aggressive AI prompt causing false positives — The system prompt instructed the LLM to mark study guides, forum discussions, podcasts, and general certification info as is_voucher=true if there was "even a hint" of a discount or voucher. This caused:

   • Tech community discussion threads (e.g., AZ-400 exam discussion) marked as vouchers
   • Study guides (e.g., tutorialsdojo) marked as vouchers
   • Educational podcasts marked as promotional

3. False negatives on concrete discount offers — The Databricks "Advanced Learning Festival" post with a clear "50% discount on any Databricks Certification" was classified as is_voucher=false, confidence=0 by the AI.

4. False positives on voucher storefronts — Pearson VUE pages that simply let you "buy a voucher" at full price (standard e-commerce, not a promotion) were being classified as promotional.

5. [null] in array fields causing parse errors — The AI occasionally returned "certifications": [null] or "regions": [null] instead of null, causing Pydantic validation failures.

Type of Change

• Bug fix

Affected Components

• AI Layer (Groq / Gemini)
• Database / Migrations

Testing

• Ran pytest — all 160 tests pass
• Ran ruff check . — clean
• Ran ruff format --check . — clean
• Ran mypy voucherbot tests — no new errors

AI Layer Changes

• Prompt changes have been tested against a representative sample of real posts
• The JSON parser handles partial responses gracefully after any schema changes

Changes Made

voucherbot/services/ai/schema.py

• Added model_validator to strip None entries from certifications and regions arrays. The AI sometimes emits [null] instead of null or [], which caused Pydantic validation errors like Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]. Now [null] → None, and [null, "AZ-900", null] → ["AZ-900"].

voucherbot/services/ai/analyzer.py — System prompt rewritten:

• Top structure: Clear is_voucher=true / is_voucher=false sections with concrete examples
• False negative fix: Concrete discount examples ('50% off', 'save $100', '20% discount on certification') placed at the top so the LLM recognizes explicit offers even within learning-event context
• Storefront exclusion: Explicitly excludes pure e-commerce: "'Purchase vouchers', 'buy exam vouchers', 'order your voucher' — these are standard transactions, not promotions"
• "earn" nuance: "earn" alone → not promotional; "earn rewards", "earn discounts", "earn vouchers", "earn credits" → promotional
• "credit" nuance: "exam credit", "voucher credit" → promotional; "CPE credits", "CE credits", "college credit" → not promotional

voucherbot/services/ingestion/event_matcher.py — _merge_fields():

• Added date-string-to-datetime conversion for start_date/end_date fields before setattr(), matching the existing _parse_date() logic in _extracted_to_event_fields(). Prevents invalid input for query argument $3: '2026-07-21' (expected a datetime.date or datetime.datetime instance, got 'str').

voucherbot/providers/pearsonvue/collector.py — _PROMO_KEYWORDS:

• Replaced overly broad keywords ("voucher", "save", "sale", "credit", "promo", "deal") with more specific promotional signals. Since Pearson is a certification voucher vendor, generic terms appear on nearly every page.

Before / After

Scenario	Before	After
Databricks "50% discount" learning event	is_voucher=false, confidence=0	is_voucher=true, confidence>0.7
Pearson VUE "buy voucher" store page	is_voucher=true ("clear promotional intent")	is_voucher=false (standard transaction)
Study guide (tutorialsdojo)	is_voucher=true ("New certification voucher")	is_voucher=false (no promotional offer)
Forum exam discussion (AZ-400)	is_voucher=true ("General certification info")	is_voucher=false (no promotional offer)
Educational podcast	is_voucher=true ("promotional intent detected")	is_voucher=false (no promotional offer)
Event merge with date string	Pipeline crash (DataError)	Clean merge to datetime
AI returns [null] in arrays	Pydantic validation error	Silently cleaned to null

Additional Notes

The root cause of most false positives was the prompt instructing the LLM to treat study guides, learning paths, job postings, and news as is_voucher=true with low confidence if there was "even a hint" of a discount or voucher. The new prompt takes the opposite approach: it requires genuine promotional intent (discount amounts, free exam offers, coupon codes, giveaways, rewards programs) and explicitly lists non-promotional categories.

The end_date crash was the most severe issue — it caused every pipeline tick to fail when the event matcher attempted to merge a date string into an existing event, preventing all downstream processing.