Curated Feeds

Curated Feeds is a FastAPI service that imports RSS/Atom feeds, deduplicates articles, extracts full text, and scores each post with an AI analysis. The goal is a curated feed: sources that consistently publish useful content remain visible, while weak sources lose score and can be removed automatically.

The technical architecture and data flow are documented in more detail in src/README.md.

Features

• Run the complete curation workflow manually through POST /jobs/workflow:
  1. apply pending reader feedback comments to the discovery profiles with a second Claude agent, so the run searches with up-to-date preferences
  2. fetch all active RSS/Atom sources
  3. discover candidate posts and sources from editable reader/topic profiles, steering away from already-known domains and resolving each new source to a verified RSS/Atom feed URL
  4. detect duplicates by URL, canonical URL, and normalized title hashes
  5. extract article full text with HTTP fetching and trafilatura
  6. summarize, keyword, and score posts with Claude/Anthropic using the reader profile and topics for relevance
  7. update source scores from post scores using a weighted moving formula; weak sources get removed, and sources stuck below the visibility threshold get auto-paused
  8. expose only posts from sufficiently well-scored sources through /posts
• Keep automatic scheduled imports independent from the manual full workflow
• Create, list, update, pause, remove, and delete feed sources through a REST API
• Import all feeds or one source manually, separate from the full workflow
• Run Claude-powered discovery as a standalone job
• Read and update discovery profile Markdown files through the API
• List visible posts and fetch post details including extracted content
• Store up/down feedback with optional comments for posts; comments are folded into the reader and topic profiles on the next workflow run
• Reset all data (sources, posts, feedback) through POST /admin/cleanup for a clean cold start

Tech Stack

• Python 3.12
• FastAPI and Uvicorn
• SQLAlchemy asyncio with SQLite
• APScheduler for periodic imports
• feedparser, httpx, and trafilatura for feed and content processing
• Anthropic SDK for structured AI analysis
• pytest for tests
• Docker Compose for local runtime

Quick Start With Docker

docker compose up --build

The service will be available at:

http://localhost:8000

Interactive API documentation is available at /docs. A minimal browser test UI for posts, sources, jobs, profiles, and cleanup is served same-origin at /ui.

Ways to use the UI:

• Same machine: open http://localhost:8000/ui/.
• Another machine / remote container: open http://<host-or-ip>:8000/ui/. The UI uses relative paths, so it talks to whatever host served it — no extra config (the container already binds 0.0.0.0:8000).
• frontend/index.html opened via file://: it cannot tell which backend to use, so point it once with a query string: file://.../frontend/index.html?api=http://<host-or-ip>:8000 (remembered via localStorage); without it, it defaults to http://localhost:8000.

For any access that is not same-origin (file://, or the UI on a different host/port than the API), the requesting origin must be allowed via ALLOWED_ORIGINS in .env. Add the specific origins (e.g. http://192.168.1.50:8000,null) or set ALLOWED_ORIGINS=* to allow all.

The Compose setup runs Claude Code inside the container. It mounts ~/.claude and ~/.claude.json only for authentication, while the project discovery skill is available inside /app/.claude/skills/curated-feed-discovery/. Alternatively, the Anthropic SDK can authenticate through the appropriate environment variables.

Local Start Without Docker

python -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"
uvicorn curated_feeds.main:app --reload

By default, the app uses this database URL:

sqlite+aiosqlite:////data/curated_feeds.db

For local development without Docker, set DATABASE_URL to a writable local file, for example:

export DATABASE_URL="sqlite+aiosqlite:///./curated_feeds.db"

Configuration

Configuration lives in src/curated_feeds/config.py and can be overridden with environment variables:

Variable	Default	Meaning
ANTHROPIC_API_KEY	None	Explicit API key for the AI analyzer; preferred over the OAuth token to avoid conflict with the claude CLI discovery subprocess
ANTHROPIC_AUTH_TOKEN	None	OAuth token for the AI analyzer; used only when no API key is set, falling back to ~/.claude/.credentials.json
ALLOWED_ORIGINS	http://localhost:8000,http://127.0.0.1:8000,null	Comma-separated CORS origins allowed to call the API; include null for the UI opened via file://, add remote hosts (e.g. http://192.168.1.50:8000), or set * to allow all
DATABASE_URL	sqlite+aiosqlite:////data/curated_feeds.db	Async SQLAlchemy database URL
IMPORT_INTERVAL_HOURS	24	Interval for automatic imports, in hours
MAX_POSTS_PER_IMPORT	20	Maximum number of new posts imported per source per import run
FEED_FETCH_TIMEOUT_SECONDS	15	HTTP timeout, in seconds, for fetching RSS/Atom feeds during import
VISIBLE_POST_MAX_AGE_HOURS	168	Maximum age for posts returned by /posts, in hours
IMPORT_MAX_AGE_HOURS	168	Maximum article age, in hours, accepted at import time. Feed entries and discovered candidates older than this are skipped before any HTTP fetch or AI scoring, so the system never spends work on posts too old to ever be visible
VISIBILITY_THRESHOLD	6.0	Minimum source score required for new posts to be visible
REMOVAL_THRESHOLD	3.0	Sources at or below this score are removed
MIN_ACTIVE_SOURCES	5	Minimum number of active sources kept, even below the removal threshold
SOURCE_SCORE_PRIOR	1.0	Pseudo-count damping how strongly each post moves an already-established source score. New sources are not seeded: the first scored post sets the score outright, then later posts move it through this damped formula
SOURCE_STALE_POST_COUNT	5	Scored-post count after which a source still below the visibility threshold is auto-paused (respecting MIN_ACTIVE_SOURCES)
PROFILE_DIR	profiles	Directory containing editable reader_profile.md and topics.md templates
SUMMARY_LANGUAGE	English	Language the AI analyzer writes post tldr summaries in
DISCOVERY_SEARCH_TIMEOUT_SECONDS	300	Timeout for Claude-powered discovery searches
DISCOVERY_MAX_ATTEMPTS	3	Maximum attempts per discovery search; transient Claude CLI failures (non-zero exit) and malformed JSON output are retried with exponential backoff.
DISCOVERY_RETRY_BACKOFF_SECONDS	5.0	Base backoff between discovery retries; delay is backoff * 2^(attempt-1).
DISCOVERY_RETRY_ON_TIMEOUT	false	Whether a discovery timeout is retried. Disabled by default because a timeout already consumed the full time budget.
DISCOVERY_EXCLUDE_KNOWN_DOMAINS	true	Inject the domains of known sources/posts into the discovery prompt and pre-skip candidates from them, so discovery finds new sources instead of rediscovering known articles.
DISCOVERY_AUDIT_LOG_RAW	false	Log raw Claude CLI stdout/stderr at debug level. Keep disabled unless debugging, because output may contain full model responses.
DISCOVERY_AUDIT_LOG_CANDIDATE_URLS	true	Include discovered candidate URLs in the normal completion log line.
DISCOVERY_AUDIT_LOG_MAX_CHARS	20000	Maximum characters per raw Claude stdout/stderr log field when raw logging is enabled.
FEED_RESOLVER_ENABLED	true	Resolve each new discovered source to a verified RSS/Atom feed URL before registering it. When disabled, the discovered URL is stored as-is and left active.
FEED_RESOLVER_AGENT_ENABLED	true	Allow the Claude agent fallback (step 4) when HTTP probing fails to find a feed.
FEED_RESOLVER_TIMEOUT_SECONDS	30	HTTP probing budget for feed resolution (already-a-feed, autodiscovery, common paths).
FEED_RESOLVER_AGENT_TIMEOUT_SECONDS	60	Timeout for the Claude feed-resolver agent subprocess.
PROFILE_TUNING_ENABLED	true	At the start of each manual workflow, hand pending feedback comments to a Claude agent that rewrites reader_profile.md and topics.md. When disabled, comments are stored but never applied.
PROFILE_TUNER_MODEL	claude-sonnet-4-6	Anthropic model used by the profile-tuning agent.
PROFILE_TUNER_MAX_COMMENTS	100	Maximum pending comments applied per workflow run.

Claude discovery writes structured lifecycle logs through the Python logger: start, timeout/failure, completion, elapsed time, result count, and candidate URLs by default. Raw Claude output is intentionally opt-in via DISCOVERY_AUDIT_LOG_RAW=true.

API Examples

Create a source:

curl -X POST http://localhost:8000/sources \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/feed.xml","name":"Example Feed"}'

Trigger an import for all active sources:

curl -X POST http://localhost:8000/jobs/import

Read or update the discovery profiles:

curl http://localhost:8000/profiles/reader-profile
curl http://localhost:8000/profiles/topics

curl -X PUT http://localhost:8000/profiles/topics \
  -H "Content-Type: application/json" \
  -d '{"content":"# Search Topics\n\n- Detailed engineering postmortems\n"}'

Run a Claude-powered discovery search:

curl -X POST http://localhost:8000/jobs/discover \
  -H "Content-Type: application/json" \
  -d '{"limit":10}'

Start the complete manual workflow:

curl -X POST http://localhost:8000/jobs/workflow \
  -H "Content-Type: application/json" \
  -d '{"discovery_limit":10}'

Check workflow status:

curl http://localhost:8000/jobs/workflow/<job_id>

List visible posts:

curl http://localhost:8000/posts

Submit feedback:

curl -X POST http://localhost:8000/posts/1/feedback \
  -H "Content-Type: application/json" \
  -d '{"vote":"up","message":"Relevant and well summarized"}'

The optional message is a free-text comment. Each unprocessed comment is read once on the next workflow run and folded into the discovery profiles, then marked processed so it is never reapplied.

Reset all data for a clean cold start (refused while a job is running):

curl -X POST http://localhost:8000/admin/cleanup
# {"sources_deleted":17,"posts_deleted":2574,"feedback_deleted":0,"jobs_cleared":0}

Tests

pytest

The existing tests cover score calculation, deduplication, and fallback behavior for AI analysis.

Project Structure

.
├── Dockerfile
├── docker-compose.yml
├── entrypoint.sh
├── frontend/
│   └── index.html
├── profiles/
│   ├── reader_profile.md
│   └── topics.md
├── pyproject.toml
├── src/
│   ├── README.md
│   └── curated_feeds/
│       ├── api/
│       ├── services/
│       ├── config.py
│       ├── database.py
│       ├── main.py
│       ├── models.py
│       └── schemas.py
└── tests/

More details about modules, the data model, and the import pipeline are available in src/README.md.