Skip to content

Feature Knowledge Base

Jump to bottom
Ramiro Cantu edited this page May 30, 2026 · 1 revision

Feature: Knowledge Base

Summary: The knowledge-base (KB / PKM) feature turns your raw study material into a tagged, connected knowledge graph and mirrors it out to Notion. You drop a lecture-notes PDF (or let the inbox poller pick one up), and the pipeline transcribes it page-by-page, extracts standalone atomic facts, embeds them, grounds each fact to an outline node via a constrained LLM categorizer, derives concept-to-concept similarity edges, and writes one Notion page per node. Five token-gated read endpoints then let the native client render the facts, the connections graph, the Notion page index, and the ingest inbox. This is the substrate every other study view reads from.

The categorization mechanics (recall → grounded pick → calibrate → persist) and the OpenAI SDK seam are owned by the Intelligence Layer; this page summarizes the flow and links there for depth. The boundary between this server and gradient-core lives in Core Library.

User flow

There are two ways material enters the KB, and one downstream pipeline both feed:

  1. Synchronous upload — POST /api/v1/pdf/ingest with a course_id and a file. The PDF runs through the vision-ingest pipeline inside the request and you get an IngestReport-shaped response back (facts created, pages, token usage).
  2. Inbox poll — drop PDF_INBOX_DIR/<course_slug>/*.pdf and the run_pdf_ingest scheduler job ingests it on its interval. The per-course slug subdirectory routes the file to its course.

Either way, atomic facts land in atomic_facts with node_id NULL. The four background jobs then take over: embed → tag → mirror.

sequenceDiagram
    actor You
    participant API as POST /pdf/ingest
    participant Ingest as kb.pdf_ingest
    participant OpenAI as OpenAI (vision + extract)
    participant DB as Postgres

    You->>API: course_id + PDF file
    API->>Ingest: ingest_pdf(course_id, path, vision_client)
    Ingest->>Ingest: SHA-256 (idempotency check)
    loop each rendered page
        Ingest->>OpenAI: transcribe_page (vision, V-KB3)
        OpenAI-->>Ingest: page text
        Ingest->>OpenAI: extract_atomic_facts (strict json_schema, V-KB4)
        OpenAI-->>Ingest: [atomic facts]
        Ingest->>DB: INSERT atomic_facts (node_id NULL, content_hash dedup)
    end
    API-->>You: PdfIngestResponse

    Note over DB: later, on scheduler intervals
    DB-->>DB: run_embed → run_grounded_tag → run_notion_sync
Loading

Endpoints

All KB endpoints require the X-Coach-Token header (401 invalid coach token on miss). The four read routes live in kb_reads.py whose whole router carries the verify_coach_token dependency; pdf/ingest declares it per-route.

Method Path Auth Body / Response Purpose
POST /api/v1/pdf/ingest token multipart: course_id (Form), file (File) → PdfIngestResponse Upload a lecture-notes PDF for an explicit course and run it synchronously through the vision-ingest pipeline.
GET /api/v1/atomic-facts token → list[dict] Grounded atomic facts, filtered by outline node_id or pdf_source_id (T46, V-KB1).
GET /api/v1/concept-edges token → list[dict] Concept-edge feed for the connections view; similarity + manual edges as-is (T45, V-E2).
GET /api/v1/notion/pages token → list[dict] Notion pointer index for the page-index view (T47, V-N1/V-N2).
GET /api/v1/pdf-sources token → list[dict] Ingested-PDF inbox with per-source facts_count rollup (T48).

Note: None of these read routes declare a response_model; each returns a plain list[dict] (shapes below). The query params have validated bounds — limit is ge=1, le=500 on /atomic-facts, /concept-edges, and /pdf-sources; out-of-range values yield a 422.

Request / response shapes

POST /pdf/ingest takes multipart/form-data — there is no Pydantic body model; FastAPI reads course_id: int = Form(...) and file: UploadFile = File(...). A missing course returns 404 course id={course_id} not found. The response is PdfIngestResponse:

PdfIngestResponse(
    pdf_source_id: int, pages: int,
    new_facts: int, dup_facts: int, reused_pdf: bool,
    extractor_version: str,
    input_tokens: int, output_tokens: int, cached_tokens: int,
)

Read-endpoint row shapes:

Endpoint Row keys
/atomic-facts id, text, node_id, page, pdf_source{id, filename}, created_at
/concept-edges id, from{node_id, name}, to{node_id, name}, kind, score, created_at
/notion/pages node_id, title, url, notion_page_id, tags, last_synced_at
/pdf-sources id, filename, sha256, status, facts_count, course_id, ingested_at, created_at

Note: On /atomic-facts, extractor_version is stamped on the tag rows (atomic_fact_tags), not on the fact row, so it is not in the fact projection. On /notion/pages, title is the outline node's name — V-N1 forbids reading Notion content back, so we surface our own pointer fields only.

The pipeline (ingest → embed → tag → mirror)

1. PDF ingest — vision transcription + grounded fact extraction (V-KB3, V-KB4)

kb/pdf_ingest.py:ingest_pdf is the core. Because lecture notes are frequently handwriting, scans, or image-only slides, the pipeline does not trust pdfplumber.extract_text — every page is rasterized (PyMuPDF, 150 DPI) and transcribed by an OpenAI vision call (transcribe_page, V-KB3). The transcription is then fed to a strict-json_schema structured-output call (extract_atomic_facts, V-KB4) that emits self-contained atomic claims.

Idempotency (V-KB1) is two-layered:

  • File level — re-ingesting a file with the same SHA-256 returns the existing pdf_sources row and writes no new facts (reused_pdf=True); no render, no API call.
  • Fact level — UQ(course_id, content_hash) on atomic_facts plus an in-run seen_hashes set drops duplicate facts (counted as dup_facts).

Every persisted fact is stamped with extractor_version (pdf-vision-v1) and page; node_id stays NULL until the categorizer runs. Token usage (prompt / output / cached) is summed across all vision + extraction calls per V-L1 — cache hits come from prompt_tokens_details.cached_tokens, never inferred.

Note: ingest_pdf runs inside the caller's transaction — it does not commit. The endpoint relies on get_session() committing on clean exit; the scheduler wrapper owns its own session + commit.

The inbox poller (kb/inbox.py:poll_inbox) wraps the same ingest_pdf over PDF_INBOX_DIR/<slug>/*.pdf. Files in the inbox root or under an unknown slug are skipped with a WARN, never an error (V41: one bad file does not abort the batch). It caches slug → Course so a many-file course resolves once.

2. Embeddings (V-E1)

kb/embeddings.py:embed_and_persist wraps the OpenAI embeddings call and writes content_embeddings rows. Each row is stamped with embedding_version (<model>-v2); a provider / dim / model change requires a version bump and a full re-embed — mixed-dim vectors in one column are forbidden (V-E1), enforced at the app layer by DimMismatchError against an EXPECTED_DIMS map. The call is idempotent on (entity_kind, entity_id, version).

kb/jobs.py:embed_pending is the batch driver: it embeds every outline_node / atomic_fact / question missing a current-version row. Outline nodes embed their full >>-delimited path (e.g. Metabolism >> Glycolysis >> Regulation), not the bare leaf name — the v1→v2 bump exists precisely because path-vectors are not comparable to old name-vectors. Outline-node vectors are the recall candidate index, so embedding gates tagging.

3. Grounded tagging — the categorizer (V-L3, V69)

This is the categorization core; the deep mechanics live in Intelligence Layer. In summary, kb/jobs.py:tag_pending runs recall → grounded pick → calibrate → persist per untagged entity:

  • Recall (kb/recall.py:retrieve_candidates) builds a constrained candidate set — V-L3 forbids free-form judgment over the full outline. Three paths merge, deduped by node: C2T (cosine vs outline-node vectors), C2C2T (similar already-tagged facts → their tags, gold/silver-weighted), and T2T (similarity-edge neighbors). It explicitly never reads Attempt.time_seconds (V-E2 negative constraint).
  • Grounded pick (llm/grounded.py:generate_grounded_tags) prompts the model to choose only from the numbered candidate list — a dual surface of a numbered NL list (reasoning) plus an int enum [1..N] (the grammar constraint, V44/V45 strict json_schema). No candidates → no LLM call.
  • Calibrate (llm/calibrator.py:calibrate_tag, V69) re-scores each picked tag via the calibrator model's Yes/No next-token logprobs (Conf = exp(L_yes) / (exp(L_yes) + exp(L_no))). The model's own optimistic self-report is no longer collected — the logprob grade is the sole confidence. <0.5 ⇒ manual_review=true (V-T3). Logprobs require reasoning_effort='none'.
  • Persist (kb/persist_tags.py:persist_grounded_tags) writes question_tags or atomic_fact_tags under the V-T2 re-run pattern (DELETE WHERE source='llm' then INSERT fresh); manual / schema_map rows are never touched. For atomic facts it also denormalizes atomic_facts.node_id to the highest-confidence non-review pick (NULL if all picks are low-confidence — the fact stays for review rather than auto-assigning a bad node).

Question scoping (V-CAP2): a question is tagged under its own course_id; an unscoped legacy question falls back to the sole course only if exactly one exists, else it is skipped as ambiguous.

4. Similarity edges (V-E2)

kb/similarity.py:derive_similarity_edges computes pairwise cosine over outline_node embeddings and persists concept_edges rows with kind='similarity' above a threshold (default 0.7). It is idempotent — an existing similarity pair is left untouched — and it never writes or inspects kind='manual' edges: similarity is derived, manual is human-verified (V-E2). The UQ(src, dst, kind) lets a manual and a similarity edge over the same node pair coexist.

5. Notion write-out (V-N1, V-N2)

kb/notion.py:sync_pending_nodes is a strict one-way mirror Postgres → Notion. It never reads Notion content back (V-N1); the only stored Notion state is the notion_pages pointer (page id, url, tags snapshot, node_id). There is one page per node (V-N2), upserted on node_id. First sync creates the page with all of the node's facts as bulleted blocks; subsequent syncs blocks.children.append only facts newer than last_synced_at — append-only, never a rewrite (V-N1/V-M3). A node is mirrorable only once tagging has set a fact's node_id, so this is empty until the categorizer runs. The same seam also mirrors discriminator factors (V-M3 idempotent on notion_block_id).

The four scheduler jobs

These run only when settings.SCHEDULER_ENABLED; each writes a TaskRun row, is guarded against concurrent runs by a module-level in-flight set + lock, and follows V41 partial-success (per-item failures are collected, the run still commits and is marked succeeded).

Job Trigger No-op condition What it does
run_pdf_ingest interval PDF_INGEST_INTERVAL_MINUTES OPENAI_API_KEY unset Poll PDF_INBOX_DIR/<slug>/*.pdf → poll_inbox → vision-ingest each.
run_embed interval EMBED_INTERVAL_MINUTES OPENAI_API_KEY unset embed_pending — embed every un-embedded node / fact / question. Gates tagging, so runs on a tighter interval.
run_grounded_tag interval GROUNDED_TAG_INTERVAL_MINUTES OPENAI_API_KEY unset The categorizer. tag_pending — recall → grounded pick → inline V69 calibrate → persist. Empty until embeddings exist.
run_notion_sync interval NOTION_SYNC_INTERVAL_MINUTES NOTION_API_TOKEN or WIKI_DB_ID unset sync_pending_nodes — mirror every node owning tagged facts to Notion.

Note: When a job's no-op condition is met it still records a succeeded TaskRun with an error_text hint (e.g. "openai unconfigured — pdf ingest skipped"). The data-flow dependency is run_pdf_ingest → run_embed → run_grounded_tag → run_notion_sync; each downstream stage is simply empty until its upstream has produced rows. Calibration runs inline inside run_grounded_tag — there is no separate calibrate job.

Functions involved

Controller → service → data-layer chains (see Function Index):

  • Upload: post_pdf_ingest → build_openai_client + ingest_pdf → (transcribe_page, extract_atomic_facts, _sha256_file) → PdfSource / AtomicFact.
  • Inbox poll: run_pdf_ingest_job → poll_inbox → _resolve_course + ingest_pdf.
  • Embed: run_embed_job → embed_pending → (_embed_outline_nodes, embed_and_persist) → ContentEmbedding.
  • Tag: run_grounded_tag_job → tag_pending → _tag_one → (load_embedding, retrieve_candidates, generate_grounded_tags → calibrate_tag, persist_grounded_tags) → AtomicFactTag / QuestionTag / AtomicFact.node_id.
  • Similarity: derive_similarity_edges → cosine → ConceptEdge.
  • Mirror: run_notion_sync_job → sync_pending_nodes → sync_node_to_notion → _ensure_node_page → NotionPage.
  • Reads: list_atomic_facts, list_concept_edges, list_notion_pages, list_pdf_sources (+ _node_names helper).

Data models touched

See Data Models:

  • PdfSource — one row per ingested PDF; sha256 is the file-level idempotency key; status walks parsing → ingested.
  • AtomicFact — one extracted claim; UQ(course_id, content_hash) fact-level dedup; node_id set by the categorizer (NULL until then); page + extractor_version stamped at ingest.
  • AtomicFactTag — calibrated source='llm' tags per fact (V-T2 re-run target); carries confidence, rationale, manual_review, extractor_version.
  • ContentEmbedding — (entity_kind, entity_id, embedding_version) vector store; one column, homogeneous dim (V-E1).
  • ConceptEdge — kind ∈ {similarity, manual} over node pairs; UQ(src, dst, kind).
  • NotionPage — pointer per node (notion_page_id, url, tags, last_synced_at); one page per node (V-N2).
  • QuestionTag — calibrated tags on captured questions (the question side of the same tagging pipeline).
  • LlmBatchRun — LLM batch-run bookkeeping for the AI chain.

Edge cases & business rules

  • Idempotency (V-KB1): same-SHA file → reused pdf_sources row, zero new facts; same (course_id, content_hash) → counted as dup_facts, not inserted.
  • No-text pages: the vision call may legitimately return empty text; extract_atomic_facts short-circuits on blank input (no LLM call, empty result) so an unreadable page contributes nothing rather than erroring.
  • Grounding (V-L3): the categorizer can only pick from the recalled candidate set — never free-form over the full outline. An empty candidate set skips the LLM call entirely (an empty strict enum would be invalid anyway).
  • Calibration gate (V-T3): a calibrated confidence <0.5 flags manual_review=true rather than dropping the tag; for atomic facts a low-confidence-only set leaves node_id NULL (the fact stays reviewable).
  • Embedding versioning (V-E1): mixed-dim vectors in content_embeddings are forbidden; a dim mismatch raises before persistence. Outline-node text changing (leaf → path) forced the v1→v2 corpus re-embed.
  • Similarity vs manual edges (V-E2): derivation only ever touches kind='similarity'; manual edges are human-verified and never recomputed, never followed by recall expansion. The connections read surfaces both as-is without re-weighting.
  • One-way Notion (V-N1/V-N2): writes only (pages.create, blocks.children.append); never pages.retrieve / databases.query. Append-only on re-sync; one page per node. The /notion/pages read exposes our pointer rows only.
  • Partial success (V41): every background job isolates per-item failures into a report and still commits — the TaskRun is succeeded (partial), and the failures land in error_text. The SHA / version filters mean the next tick resumes only the un-done work.
  • SDK boundary (V16): the OpenAI vision/extract/tagging/calibrator clients and the Notion client are all injected so tests mock at the SDK boundary; production wires them through build_openai_client (V41: max_retries ≥ 5).
  • Token accounting (V-L1): input/output/cached tokens are summed from real usage; cached tokens read from prompt_tokens_details.cached_tokens, never estimated.

See also

gradient-server

Reference

Features

Design Notes

Clone this wiki locally