Guideline Rot

Guideline Rot: A Temporal Trust-Decay Framework for Detecting Superseded Clinical Guidance in Retrieval-Augmented Systems

Retrieval-augmented LLM systems often surface preventive-care guidance without knowing whether it has been superseded, downgraded, or left to age out. That is a documented failure mode: models answer confidently from stale corpus chunks, and users have no signal that the underlying recommendation changed years ago. This project ingests USPSTF recommendations, detects version drift with embeddings plus an LLM judge, computes an auditable Trust Decay Score per topic, and publishes the results to a public dashboard where low-trust guidance is visible before it reaches an end user.

Architecture

flowchart TB
    subgraph batch ["Scheduled batch pipeline"]
        direction TB
        USPSTF["USPSTF Prevention TaskForce API"]
        GHA["GitHub Actions - refresh_pipeline.yml"]
        ING["Ingestion - uspstf_client.py"]
        EMB["Embed and classify - embed_and_classify.py"]
        GEMINI_BATCH["Gemini API - embeddings and LLM judge"]
        SCORE["Trust scoring - trust_score.py"]
        WRITER["Firestore writer - writer.py"]
        FS[("Firestore - Spark plan")]

        GHA --> ING
        USPSTF --> ING
        ING --> EMB
        EMB --> GEMINI_BATCH
        EMB --> SCORE
        SCORE --> WRITER
        WRITER --> FS
    end

    subgraph live ["Live browser path"]
        direction TB
        HOST["Firebase Hosting - static web/"]
        DASH["Dashboard - metrics, leaderboard, digest, Q&A"]
        BROWSER["Visitor browser"]
        MATCH["Topic match - keyword overlap on loaded data"]
        GEMINI_QA["Gemini API - generativelanguage.googleapis.com"]
        QA["Q&A answer - trust-aware response"]

        FS --> DASH
        HOST --> DASH
        BROWSER --> HOST
        DASH --> MATCH
        MATCH --> GEMINI_QA
        GEMINI_QA --> QA
    end

Loading

Batch pipeline: GitHub Actions pulls USPSTF data, embeds recommendation pairs with gemini-embedding-2, classifies relationships (SUPERSEDES, REAFFIRMS, CONTRADICTS, UNRELATED) with gemini-2.5-flash-lite, applies a rule-based Trust Decay Score (age, supersession, grade volatility), and upserts recommendations, corpus_health, and changelog into Firestore.

Live browser path: Firebase Hosting serves the static dashboard. The browser loads Firestore read-only, renders trust metrics and the leaderboard, matches questions to topics locally, and optionally calls Gemini from the client for trust-aware Q&A (with an opt-in generic comparison).

Dashboard

Overview - corpus trust score hero ring, tracked/superseded metrics, trust trend chart, and spotlight on the lowest-trust stale citation.

Leaderboard - per-record trust rings and supersession status, with topic search and grade filters.

Change digest - pipeline changelog entries (score shifts, status changes, quiet-run collapse) in journal style.

Q&A demo - trust-aware Gemini answer beside an opt-in generic comparison that omits staleness auditing.

Why this is free

Every external service in this repo runs on a free tier. No credit card or billing account is required anywhere.

Service	Role	Cost
GitHub Actions	Weekly pipeline (refresh_pipeline.yml) on ubuntu-latest	Free for public repositories (minute quotas apply)
USPSTF Prevention TaskForce API	Current recommendation ingestion	Free; request an API key from uspstfpda@ahrq.gov
Google AI Studio (Gemini API)	Pipeline embeddings + classification judge; browser Q&A	Free tier via generativelanguage.googleapis.com - not Vertex AI
Firebase Spark	Firestore (read-only rules for clients) + Hosting for web/	Spark plan; no Blaze upgrade needed
Firebase Admin SDK	Server-side Firestore writes in Actions	Included with Spark Firestore

There are no Cloud Functions, Cloud Run jobs, paid VMs, or always-on servers. Compute happens in ephemeral GitHub Actions runners; the frontend is static files.

Repository layout

ingestion/              USPSTF API client + archived seed for version history
pipeline/               Embeddings, LLM pair classification, trust scoring
firestore/              Admin SDK writer (upsert by topic + changelog)
web/                    Static dashboard + browser Q&A (vanilla JS)
.github/workflows/      refresh_pipeline.yml (weekly + manual dispatch)
firestore.rules         Public read, writes denied from browsers
docs/images/            Dashboard screenshots for README

Setup

1. Clone and install

git clone https://github.com/OWNER/guideline-rot.git
cd guideline-rot
python3 -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env

Replace OWNER/guideline-rot with your published GitHub path.

2. Google AI Studio API key (pipeline)

1. Create a key at Google AI Studio.
2. Add it to .env as GEMINI_API_KEY=...
3. This key is used only by the Python pipeline (embeddings + judge). Do not reuse it for the browser Q&A demo.

3. USPSTF API key (ingestion)

Request a Prevention TaskForce API key from uspstfpda@ahrq.gov and set USPSTF_API_KEY in .env. Without it, ingestion exits non-zero. Historical pairs come from ingestion/archived_seed.json merged with live API results.

4. Firebase Spark project

1. Create a project at Firebase Console - stay on the Spark plan.
2. Enable Firestore (Native mode).
3. Register a Web app and copy the config object.
4. Create a service account key (Project settings → Service accounts → Generate new private key) for pipeline writes. Save locally; never commit it.

npm install -g firebase-tools
firebase login
firebase use --add          # updates .firebaserc
cp web/firebase-config.example.js web/firebase-config.js

Edit web/firebase-config.js:

• firebaseConfig - web app values from the console
• geminiQaConfig.apiKey - a second AI Studio key, restricted to your Hosting referrers before deploy (see Limitations)
• useLocalDemo - set to false for production

Set local pipeline credentials in .env:

FIREBASE_SERVICE_ACCOUNT_PATH=/absolute/path/to/serviceAccountKey.json
FIREBASE_PROJECT_ID=<project-id-from-firebase-console>

5. GitHub Actions secrets

In the repo: Settings → Secrets and variables → Actions → New repository secret

Secret	Purpose
GEMINI_API_KEY	Pipeline embeddings + LLM judge
FIREBASE_SERVICE_ACCOUNT	Full service account JSON (written to a temp file in the workflow)
USPSTF_API_KEY	Live USPSTF ingestion

The workflow .github/workflows/refresh_pipeline.yml runs every Monday 12:00 UTC and supports manual workflow_dispatch. Any step that fails exits non-zero so stale Firestore data is not silently overwritten by a broken run.

6. Run locally

# Full pipeline chain
python ingestion/uspstf_client.py
python pipeline/embed_and_classify.py --judge-all
python pipeline/trust_score.py
python firestore/writer.py

# Or as modules
python -m ingestion.uspstf_client
python -m pipeline.embed_and_classify --judge-all
python -m pipeline.trust_score
python -m firestore.writer

Artifacts land in ingestion/data/, pipeline/data/, then Firestore. Run tests with pytest and cd web && node --test tests/spotlight.test.js.

Preview the dashboard with bundled demo data (no Firestore required):

# web/firebase-config.js → useLocalDemo = true
python -m http.server 8080 --directory web
# open http://localhost:8080

7. Deploy

Deploy Firestore rules and Hosting:

firebase deploy

Or selectively:

firebase deploy --only firestore:rules
firebase deploy --only hosting

After deploy, run the GitHub Actions workflow once (or the local pipeline) so recommendations and corpus_health/latest exist before opening the site. Update the Live demo link at the top of this README with your Hosting URL.

Limitations and honest tradeoffs

Client-side Gemini key exposure. The Q&A demo calls generativelanguage.googleapis.com directly from the browser. The key is visible in page source and network traffic. Mitigation: use a dedicated demo key (not the pipeline key), set HTTP referrer restrictions in AI Studio to your Firebase Hosting domains (https://<project>.web.app/*, https://<project>.firebaseapp.com/*), and keep daily quotas low. Referrer restriction limits abuse to your origin; it does not make the key secret.

LLM judge in the classification step. Pair labels (SUPERSEDES vs REAFFIRMS vs UNRELATED) come from gemini-2.5-flash-lite with structured JSON output. That step is not fully deterministic - re-runs can disagree on borderline pairs. The Trust Decay Score formula itself is rule-based and auditable once labels are fixed; the uncertainty sits upstream in classification, not in the arithmetic.

USPSTF scope only. This corpus covers USPSTF preventive-care recommendations from the Prevention TaskForce API plus a small archived seed for version history. It does not include CDC, specialty society guidelines, FDA labeling, local protocols, or inpatient order sets. A low trust score here means "stale relative to this USPSTF snapshot," not "wrong for every clinical context."

Q&A retrieval is intentionally simple. The browser matches questions by keyword/topic overlap against loaded Firestore rows - not embeddings. That keeps the demo free and dependency-free but misses semantic matches a vector index would catch.

Not clinical software. Research and portfolio artifact only. No PHI. Do not use for patient care decisions.

Data provenance

Recommendation text and grades come from the U.S. Preventive Services Task Force via the Prevention TaskForce API. USPSTF is an independent panel; source material is public.

License

This project is licensed under the MIT License, see LICENSE for details. USPSTF content remains subject to its own terms and attribution requirements; this repo's license covers the code, not the underlying guideline data.