Home

gradient-server wiki

Engineering docs for Gradient — a single-user, local-first, multi-domain study engine (questions → tagging → grounded atomic facts → Notion, plus Anki linking). SPEC.md at the repo root is the forward source of truth; this wiki holds longer-form design notes that don't belong in the spec.

Stack: Python 3.12+ · FastAPI · SQLAlchemy (async) · Postgres 16 (asyncpg) + pgvector · OpenAI SDK · APScheduler · Alembic. Backend-only (V-D1): every feature is a JSON API under /api/v1/*, the root /media store, an MCP tutor seam, or a scheduled job — there is no in-repo UI.

High-level architecture

flowchart LR
    subgraph Clients
        EXT["Chrome extension<br/>(Gradient Capture)"]
        TUT["MCP tutor host"]
        ADM["Admin / curl"]
    end

    subgraph App["FastAPI app (app/main.py)"]
        API["/api/v1/* routers"]
        MED["/media/{path}"]
        SCHED["APScheduler jobs<br/>(lifespan)"]
        SVC["services/<br/>ingest · outline · kb · llm · anki · tutor"]
    end

    DB[("Postgres 16<br/>+ pgvector")]

    subgraph External
        OAI["OpenAI API"]
        ANKI["AnkiConnect"]
        NOTION["Notion API"]
    end

    EXT -->|POST /captures, /pdf/ingest| API
    TUT -->|tutor reads + persist| API
    ADM -->|admin, outline, kb reads| API
    API --> SVC
    MED --> SVC
    SVC --> DB
    SCHED --> SVC
    SVC -->|embed · transcribe · tag| OAI
    SVC -->|deck sync · allowlisted writes| ANKI
    SVC -->|one-way mirror| NOTION

Loading

Getting Started

• Getting Started — prerequisites, environment variables, uv sync, Postgres + Alembic, running the server, seeding, the X-Coach-Token header, tests, and regenerating the OpenAPI contract.

Reference

• Architecture — layered breakdown, request lifecycle, async session model, and the scheduler / background-job model.
• Data Models — every ORM table (core, Anki, KB substrate) and the request/response Pydantic schemas.
• API Reference — the full endpoint catalog grouped by feature, with auth, request shapes, and response models.
• Function Index — the most-central functions and a by-module index across the service layer.
• Glossary — domain and code vocabulary (atomic fact, node_id, discriminator, calibration, source adapter, …).

Features

• Knowledge Base — PDF ingest → embed → grounded tag → Notion mirror, plus the kb_reads read endpoints.
• Outline & Mastery — schema-as-data outline tree, >>-path resolution, subtree-as-set mastery rollups.
• Question Capture — POST /captures, the source-adapter dispatch, normalization, and course scoping.
• Anki — deck sync, tag parsing, assignment unlock, and the append-only review log.
• Tutor & MCP — the data-exposure tutor seam, cognitive-safety contract, and discriminator persistence.
• Platform & Admin — the scheduler roster, /admin routes, the status probe, manual tag overrides, and the media store.

Design Notes

• Intelligence Layer — the AI chain: PDF ingest → embed → recall → grounded tagging → calibration. Its LLM4Tag lineage, the three recall meta-paths, model selection (V-L5), and the recall-completeness fixes (V-L6).
• Core Library — the gradient-core boundary: the domain core that can be lifted out of the FastAPI app into an embeddable library, what stays transport/runtime, the runtime/config contract, and the phased migration plan.
• Deployment & storage — target architecture for the redesign: self-hosted home server (Postgres + API), Apple clients over Tailscale, API-only access, filesystem media, paired backup.
• Features — single-page catalog of what Gradient does, grouped by area, with backing code and governing SPEC invariants.

Conventions

• The spec (SPEC.md) governs; wiki pages reference its sections (§G, §C, §I, §V, §T) rather than restating them.
• This wiki is a git submodule of the main repo at docs/wiki/. Edit pages there and push the submodule to publish.

Open Questions

Staleness and ambiguities surfaced while documenting against live code. Where docs conflict with code, code wins. These are tracked here so a later pass can reconcile them.

Documentation drift (needs refresh)

• docs/BACKEND_CORE.md predates the current job set and routers. It still documents a run_categorizer job (every CATEGORIZER_INTERVAL_MINUTES) and a run_anki_topic_resolver job (every ANKI_TOPIC_RESOLVER_INTERVAL_MINUTES), and points categorization at app/services/categorizer/ (outline_lookup.py, tag_question(...), categorizer/cache.py). None of these exist. The live job roster is run_anki_sync / run_anki_assignment_unlock / run_anki_assignment_complete / run_anki_review / run_pdf_ingest / run_notion_sync / run_embed / run_grounded_tag, and categorization is app/services/llm/grounded.py + app/services/kb/persist_tags.py. It also does not mention the kb_reads router. Needs a refresh.
• docs/wiki/Features.md predates the kb_reads router. Its KB row did not list the read endpoints (/concept-edges, /atomic-facts, /notion/pages, /pdf-sources); added in this pass.
• docs/openapi.json admin coverage. The task brief claimed it omits the admin recategorize route. Verified false at the source: app/api/v1/admin.py has no recategorize route (only /questions/{id}/tags, /tags/{id}, /jobs, /jobs/{name}/trigger, /status), and BACKEND_CORE.md is the only place that still references one. So openapi.json is not missing a recategorize path — but BACKEND_CORE.md is stale on this point.
• FastAPI app title is still MCAT Coach in app/main.py despite the project being renamed to Gradient.

Live bug

• admin.py create_manual_tag route is broken. It passes topic_id / content_category_id / skill kwargs to admin_tags.create_manual_tag, whose live signature is (*, node_id, rationale=None) only — the route raises TypeError if hit. The DELETE /tags/{tag_id} route bypasses the service and re-implements source-aware delete inline.

Auth / access nuances

• Admin mutation routes are not enforced. create_manual_tag (POST /admin/questions/{id}/tags) and delete_tag (DELETE /admin/tags/{id}) have no verify_coach_token dependency and no programmatic localhost check; the module docstring calls them "localhost-only" but that is a deployment assumption, not enforced in code.
• Trigger allowlist is partial. /admin/jobs/{name}/trigger's _VALID_JOBS covers only the 4 Anki jobs; the 4 KB/LLM jobs (pdf_ingest, notion_sync, embed, grounded_tag) appear in /admin/jobs and /admin/status but cannot be hand-triggered over HTTP.

Fenced / orphaned surfaces

• Analyzer / recommendations stack was deleted (T53), not merely unmounted. No analyzer or recommendations router is mounted; recommender.py and run_feature_extraction_job are gone. Only app/schemas/analyzer.py (fenced) and the recommender_ready health flag remain. QuestionFeatures is reader-live / writer-fenced (V-RB1).
• Anki fenced routes. GET /anki/cards?topic_id=N and GET /anki/performance are commented out (FENCED, T18 / V-RB2) — not mounted. state.py and retention.py services return empties with no live route.
• Anki assignment SQL targets dropped tables. assignment.py's _CANDIDATE_SQL_TOPIC / _CANDIDATE_SQL_CC still join the legacy topics / content_categories tables and reference anki_note_tags.topic_id / content_category_id / parsed_kind IN ('aamc_topic','aamc_cc'), none of which exist in the live node_id-only schema (V-T1 / §B4), so POST /anki/assignments fails at the SQL layer until the resolver is ported. AnkiCardTagOut.topic_id is vestigial and always serializes null.

Schema / data-model notes

• pgvector deferred. ContentEmbedding.embedding is a JSONB placeholder; migration 0003 does not run CREATE EXTENSION vector / ALTER ... vector(N) (pending T25), and docker-compose.yml runs stock postgres:16, not a pgvector image.
• LlmBatchRun naming. Its anthropic_batch_id column + docstring still reference the Anthropic Message Batches API, but the project pivoted to OpenAI; the model is not exercised by the live KB jobs or read endpoints.
• Planned §I renames not yet in code. Question.qid → external_id (with UQ(source, external_id)) and Attempt.uworld_test_id → session_ref.
• .env.example documents non-Settings knobs. CATEGORIZER_MODEL, FEATURE_EXTRACTOR_MODEL, ANKI_TOPIC_RESOLVER_MODEL and gpt-4.1-mini defaults are not Settings fields and are silently dropped (extra='ignore'); live defaults are gpt-5.4-nano / gpt-5.4-mini. POSTGRES_USER/PASSWORD/DB are consumed only by docker-compose / mise run db:drop-env, not Settings.
• TaskRun.cost_usd NUMERIC(10,4) exists but is never written by any job (always NULL); token counts (incl. cached_tokens, V-L1) surface only on the synchronous PdfIngestResponse.
• Outline import payload confusion. Only aamc_outline.schema.json ({course, nodes}) is a valid outline:import body; the sibling aamc_outline.json ({version, sections}) is raw upstream source. outline:import takes a raw dict[str, Any] validated by validate_outline_schema (not a Pydantic body), so a malformed schema returns a domain 422 {errors:[...]}.

Tutor / read-shape notes

• attempts/flagged returns topics: [] by design (T14 partial port).
• Tutor question-payload tags surface a raw node_id, not a resolved >> path (unlike captures/recent and the session summary, which are label-resolved).
• Discriminator notion_block_id is always NULL today — the T32 Notion write-back is not wired, so there is no live Notion back-link despite the column and V-N1 / V-N2 intent.
• No tutor-read route declares a response_model; shapes are runtime-built dict / list[dict], not Pydantic contracts. The same holds for the outline routes and the kb_reads read routes.
• Glossary substrate caveat. atomic_fact_tags / notion_page_tags are P2 target tables that may not be fully materialized; the Notion/PDF inbox poller jobs are conditional no-ops without their credentials.