feat: migration-led schema, weighted Groq routing, Reddit RSS-by-default

[Devathmaj]

Description

This branch consolidates the work done since main diverged, landing AI-layer routing, Reddit ingestion policy, source catalog curation, and a complete rewrite of how the database schema is managed (Alembic migrations are now authoritative and run at app startup). Together these make the app boot on a brand-new system with a brand-new DB end-to-end (schema + seed data) with no manual steps.

1. AI Layer — weighted Groq model routing (voucherbot/services/ai/analyzer.py, 92acc03)

• Replaced the round-robin batch of llama-3.1-8b-instant / gpt-oss-120b / llama-3.3-70b-versatile with weighted routing: openai/gpt-oss-20b (40%), openai/gpt-oss-120b (40%), llama-3.3-70b-versatile (20%), respecting each model's TPM/TPD/RPD quotas.
• New _pick_groq_model(exclude=...) picks a model by weight, skipping daily-exhausted ones; failures fall back to the remaining available models.
• Gemini remains the final fallback. Defaults for token-per-minute / tokens-per-day / requests-per-day updated to match the new model set.

2. Reddit ingestion policy (db1d383, 4339ca8)

• REDDIT_INGESTION_ENABLED=false (the default) now collects Reddit purely via public RSS feeds and never calls the OAuth API — enforced in RedditCollector regardless of whether OAuth credentials are configured.
• Removed the scheduler/dispatcher checks that filtered out REDDIT sources when ingestion was disabled (no longer needed — the collector itself handles it).
• Dropped author capture from Reddit posts (author=None), keeping [deleted]-style noise out of the pipeline.

3. Source catalog curation (4339ca8, docs)

• Refreshed the curated subreddit list (e.g. added O365Certification, mcsa, ccnp, linuxadmin, salesforce, vmware; removed MicrosoftLearn, LinuxCertifications, eFreebies, FREE).
• _seed_reddit_sources now disables Reddit sources no longer in the active catalog so stale subreddits stop being fetched.
• Docs/source-catalog sync updates.

4. Database schema management — Alembic migrations are now authoritative (e990211)

This is the largest change. Previously the schema was created by init_db()/create_all at startup; now alembic upgrade head runs at startup when IS_PROD=false (before seeding), and the migration chain fully reproduces the schema the running models define.

• voucherbot/main.py: non-prod startup runs alembic upgrade head (in a thread) then bootstrap_data(); prod skips DB setup entirely. New run_migrations() helper.
• Un-froze the schema-freeze migrations: g3b9 (keywords) now creates the table unconditionally; the redundant j0k1 vendor_mappings freeze was removed (its table was already created by h4d5) and the chain rewired.
• New migrations:
  • l5m6n7o8p9q1 — adds PEARSONVUE / TRAINING_PROVIDER to the sourcetype enum.
  • m6n7o8p9q0r1 — reconciles create_all-built DBs: creates the voucher_posts view and makes vendor_mappings.created_at/updated_at timezone-aware. No-op on fresh DBs.
• migrations/env.py: escapes % in the DB URL (fixes configparser interpolation crash on percent-encoded passwords), imports every model into Base.metadata, and adds an include_object filter so view-backed models are excluded from autogenerate (alembic check is now clean).
• Model alignment with migrations: vendor_mappings.url_pattern as unique index, keyword server_defaults, vendor_mapping timestamps.
• tests/test_migrations.py (new): enforces a single linear migration chain with exactly one head, resolvable down_revisions, and all model tables registered.

5. Test coverage & cleanup

• New unit tests for bootstrap Reddit source disabling, collectors, analyzer, and startup lifespan wiring (tests/test_main.py, tests/test_bootstrap.py, tests/test_collectors.py, tests/test_analyzer.py).
• Formatting (216686f) and mypy-strict fixes (6bac464).

Type of Change

• Bug fix
• New feature
• New source
• Configuration / settings change
• Database migration
• Documentation update
• Refactor (no functional change)
• Other: AI provider routing; startup lifecycle change

Affected Components

• Scheduler / Dispatcher
• HTTP Policy Layer (http_policy.py)
• RSS Collector
• Website Collector
• Reddit Integration
• AI Layer (Groq / Gemini)
• Email Notifications
• Database / Migrations
• API / Routers
• Configuration / Settings

Testing

• Ran pytest — all tests pass (355 passed, 15 skipped)
• Ran ruff check . && ruff format . — clean
• Added unit tests for new business logic
• Added integration tests using fixtures or recorded responses (no live network calls)
• Mocked Reddit API client in all new tests
• Mocked Groq / Gemini responses in all new tests

Migration

• This PR includes an Alembic migration
• The migration is safe to apply to a live database without downtime
• A sensible server-side default is provided for any new columns

Notes: On a brand-new empty DB, alembic upgrade head at startup builds the complete schema (tables, enums, indexes, voucher_posts view) automatically. For existing create_all-built DBs, run alembic stamp head (or the new m6n7o8p9q0r1 reconciliation migration) — alembic check passes cleanly after applying.

Policy Checklist

• This PR touches one or more policy-sensitive files

If checked, confirm all of the following:

• robots.txt compliance is preserved — the policy layer is not bypassed or disabled.
• No default crawl delays have been reduced below 2.0 seconds.
• Reddit rate limits (100 req/min) and the Responsible Builder Policy are respected.
• REDDIT_INGESTION_ENABLED=false still collects Reddit via RSS and makes no OAuth calls.
• No new direct httpx or aiohttp calls exist outside the policy layer.

Explanation: Reddit collection now strictly uses public RSS feeds when REDDIT_INGESTION_ENABLED=false; the OAuth path is only reachable when explicitly enabled. No policy-layer crawl delays or robots.txt handling were changed.

AI Layer Changes

• Prompt changes have been tested against a representative sample of real posts
• A before/after comparison is included below
• The JSON parser handles partial responses gracefully after any schema changes

Before / After: Model routing changed from a fixed round-robin batch to weighted 40/40/20 distribution across gpt-oss-20b / gpt-oss-120b / llama-3.3-70b-versatile, each respecting its TPM/TPD/RPD quota. No prompt or output-schema changes.

Additional Notes

No secrets are introduced or changed. alembic check verifies the migration chain matches the running models.

Summary by CodeRabbit

• New Features

  • Reddit collection now continues through public RSS feeds when OAuth ingestion is disabled or unavailable.
  • Expanded and refreshed the default Reddit source catalog.
  • Added support for multiple AI models with weighted selection and fallback handling.
  • Application startup now applies database migrations automatically outside production.

• Bug Fixes

  • Prevented disabled Reddit OAuth settings from excluding Reddit sources from scheduled collection.
  • Improved database migration handling and schema consistency.

• Documentation

  • Clarified Reddit RSS behavior, OAuth settings, setup instructions, and contributor checklists.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 33e94e60-6e3e-4113-a3c6-01f063a86d26

📥 Commits

Reviewing files that changed from the base of the PR and between 8bd0faa and 6bac464.

📒 Files selected for processing (31)

• .env.example
• .github/ISSUE_TEMPLATE/feature-request.yml
• .github/PULL_REQUEST_TEMPLATE.md
• CONTRIBUTING.md
• Sources/Subreddit.txt
• Sources/source.md
• docs/details/configuration.md
• docs/details/detailed-summary.md
• docs/setup/reddit-setup.md
• migrations/env.py
• migrations/versions/g3b9c0d1e2f3_schema_freeze_keywords.py
• migrations/versions/i9j0k1l2m3n4_add_url_pattern_to_vendor_mappings.py
• migrations/versions/j0k1l2m3n4o5_schema_freeze_vendor_mappings.py
• migrations/versions/k2l3m4n5o6p7_add_notification_outbox.py
• migrations/versions/l5m6n7o8p9q1_add_source_type_enum_values.py
• migrations/versions/m6n7o8p9q0r1_reconcile_to_head.py
• tests/test_analyzer.py
• tests/test_bootstrap.py
• tests/test_collectors.py
• tests/test_dispatcher.py
• tests/test_main.py
• tests/test_migrations.py
• voucherbot/config/settings.py
• voucherbot/database/bootstrap.py
• voucherbot/main.py
• voucherbot/models/keyword.py
• voucherbot/models/vendor_mapping.py
• voucherbot/providers/reddit/collector.py
• voucherbot/services/ai/analyzer.py
• voucherbot/services/dispatcher.py
• voucherbot/services/scheduler.py

---

📝 Walkthrough

Walkthrough

The change keeps Reddit RSS collection active when OAuth ingestion is disabled, adds Alembic migrations to startup and schema reconciliation, updates Reddit source seeding, and introduces weighted multi-model Groq routing with exhaustion handling.

Changes

Reddit ingestion behavior

Layer / File(s)	Summary
Reddit policy and source catalog .env.example, .github/..., CONTRIBUTING.md, Sources/*, docs/..., voucherbot/database/bootstrap.py	Documentation now defines RSS-only collection when OAuth ingestion is disabled. Reddit source catalogs and stale-source cleanup were updated.
Reddit collection and scheduling voucherbot/providers/reddit/collector.py, voucherbot/services/dispatcher.py, voucherbot/services/scheduler.py	Schedulers and dispatchers no longer exclude Reddit sources. The collector uses RSS when OAuth is disabled or unavailable.
Reddit behavior validation tests/test_bootstrap.py, tests/test_collectors.py, tests/test_dispatcher.py	Tests cover source seeding, RSS fallback, API collection, and Reddit source eligibility.

Migration and schema management

Layer / File(s)	Summary
Migration environment and schema revisions migrations/env.py, migrations/versions/*	Alembic now registers model metadata, excludes views, maintains migration dependencies, adds enum values, reconciles views, and updates schema columns.
Startup migration integration and model defaults voucherbot/main.py, voucherbot/config/settings.py, voucherbot/models/*	Non-production startup runs Alembic migrations before seeding. Models add server defaults, indexes, and timezone-aware timestamps.
Migration and startup tests tests/test_main.py, tests/test_migrations.py	Tests validate migration execution, startup behavior, migration graph structure, migration heads, and model table registration.

Groq model routing

Layer / File(s)	Summary
Weighted Groq selection and fallback voucherbot/services/ai/analyzer.py	Groq uses weighted selection across three models and retries available models after failures or exhaustion. Batch analysis uses the same selection logic.
Groq routing tests tests/test_analyzer.py	Tests cover weighted selection, exhausted models, quotas, retries, successful calls, and failures.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Scheduler
  participant RedditCollector
  participant RedditClient
  participant PublicRSS

  Scheduler->>RedditCollector: collect due Reddit source
  RedditCollector->>RedditClient: use OAuth API when enabled and configured
  RedditClient-->>RedditCollector: return API posts
  RedditCollector->>PublicRSS: fetch feeds when OAuth is disabled or unavailable
  PublicRSS-->>RedditCollector: return RSS posts

Loading

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch llm-reddit-db-update

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}