aicg: learning_gap — lessons

lessons/mod-001-rag-foundations/README.md

@@ -0,0 +1,38 @@
+# mod-001 · RAG Foundations
+
+First module in the RAG & Retrieval Engineer track. Establishes the shared vocabulary the rest of the curriculum builds on: what retrieval-augmented generation is, why it exists, and the moving parts of a minimal pipeline.
+
+## Learning objectives
+
+By the end of this module a learner can:
+
+- Define retrieval-augmented generation (RAG) in the terms the original paper introduced it.
+- Distinguish parametric knowledge (weights) from non-parametric knowledge (an external corpus).
+- Name the four stages of a minimal RAG pipeline — **ingest → chunk → embed → retrieve → generate** — and explain what each stage owns.
+- Identify the failure modes that motivate later modules (hallucination on out-of-distribution queries, stale facts, unattributable answers).
+- Read a RAG system diagram and point to where each component would live.
+
+## Prerequisites
+
+See the repo-level [`PREREQUISITES.md`](../../PREREQUISITES.md). No prior retrieval or vector-store experience is assumed for this module specifically.
+
+## Contents
+
+| Path | What it is |
+|---|---|
+| [`lectures/01-what-is-rag.md`](lectures/01-what-is-rag.md) | Concept walkthrough — origin, motivation, anatomy of a pipeline. |
+| [`exercises/01-classify-retrieval-failures.md`](exercises/01-classify-retrieval-failures.md) | Short paper-and-pen exercise: given failing Q&A pairs, name the stage at fault. |
+| [`labs/01-first-rag-pipeline.md`](labs/01-first-rag-pipeline.md) | Hands-on: build a five-file naïve RAG pipeline over a small corpus. |
+| [`quizzes/01-foundations-check.md`](quizzes/01-foundations-check.md) | Ten-item self-check on the vocabulary. |
+
+## How to work through it
+
+1. Read the lecture end-to-end once. Do not stop to run code.
+2. Do the exercise on paper. It takes ~15 minutes and surfaces which stage of the pipeline you still find fuzzy.
+3. Do the lab. Expect ~60–90 minutes if you are new to embedding APIs.
+4. Take the quiz. Anything you miss, re-read the corresponding lecture section — do not memorise the answer.
+
+## Primary sources
+
+- Lewis, P., et al. (2020). *Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.* NeurIPS 2020. [arXiv:2005.11401](https://arxiv.org/abs/2005.11401)
+- Karpukhin, V., et al. (2020). *Dense Passage Retrieval for Open-Domain Question Answering.* EMNLP 2020. [arXiv:2004.04906](https://arxiv.org/abs/2004.04906)

lessons/mod-001-rag-foundations/exercises/01-classify-retrieval-failures.md

@@ -0,0 +1,41 @@
+# Exercise 1 — Classify retrieval failures
+
+**Time:** ~15 minutes. Paper and pen. No code.
+
+## What you are practising
+
+Reading a failed RAG interaction and locating which pipeline stage caused the failure. This is the diagnostic skill that shows up in every future module: *before* you tune anything, you have to name what is broken.
+
+## Setup
+
+Assume the same minimal pipeline described in the lecture:
+
+```
+Ingest → Chunk → Embed → Retrieve → Generate
+```
+
+For each case below, decide which single stage is *most likely* responsible. If more than one stage is plausibly at fault, pick the earliest one — problems propagate downstream, and the earliest broken stage is usually the one worth fixing first.
+
+## Cases
+
+For each case, write down (a) the stage, and (b) one sentence of reasoning.
+
+1. The user asks "when is our office closed for winter break?" The answer comes back "I don't have that information." The company's HR handbook has a section titled *Holiday schedule* that lists the exact dates. The section is in the corpus. The retriever returned three chunks, none of them from that section.
+
+2. Same corpus. The user asks the same question. The retriever returned the *Holiday schedule* chunk in first place. The generator answered "I don't have that information."
+
+3. Same corpus. The user asks "who approves purchase requests over $10k?" The retrieved chunk contains the sentence "Purchase requests above $10,000 require director approval." The generator answered "The CFO approves purchases over $10,000."
+
+4. The PDF export step drops all table cells and keeps only headers. The user asks about pricing tiers, which live in a table. The retriever returns the *Pricing* page but the answer omits the actual numbers.
+
+5. A wiki page was updated last week to move a policy from "quarterly" to "monthly." The retriever confidently returns the old version. Nothing in the source system points at the old version anymore.
+
+6. The user asks "how do I request time off?" The corpus uses the phrase "annual leave application" everywhere. The retriever returns unrelated chunks about *time tracking*.
+
+7. A chunk of the correct document was retrieved, but it starts mid-sentence: "…must submit the form to their manager within 48 hours." The generator, given only this fragment, answers about the wrong form.
+
+8. The user is a contractor. The retriever surfaces an internal-only compensation memo and the generator quotes from it.
+
+## Debrief
+
+Compare your answers with someone else on the track, or with your own answers after finishing the lab. There is not always a single crisp answer — case 4 and case 7, for example, are both defensible as either an ingest failure or a chunking failure depending on where you draw the line. That ambiguity is the point: production RAG bugs live at the seams between stages, and calibrating your intuition for which side of a seam a failure sits on is the actual skill.

lessons/mod-001-rag-foundations/labs/01-first-rag-pipeline.md

@@ -0,0 +1,113 @@
+# Lab 1 — Your first RAG pipeline
+
+**Time:** ~60–90 minutes if the embedding and generation APIs are new to you.
+
+## What you will build
+
+A five-file, single-process RAG pipeline over a tiny corpus you assemble yourself. No vector database, no framework, no reranker. The point is to touch every stage with your own hands so you can feel where each seam is.
+
+By the end you will have:
+
+- `corpus/` — 5–10 short plain-text documents you pick.
+- `ingest.py` — reads the corpus, produces chunks with metadata.
+- `embed.py` — turns chunks into vectors and writes them to a JSON file.
+- `query.py` — accepts a question, retrieves the top-*k* chunks, sends them to an LLM, prints the answer with citations.
+- A short `NOTES.md` where you record what surprised you.
+
+The reference solution lives in the paired [`rag-engineer-solutions`](https://github.com/ai-engineering-curriculum/rag-engineer-solutions) repo. Do not read it until you finish the "Stop and think" section at the end.
+
+## Prerequisites
+
+- Python 3.10+.
+- An API key for **one** embedding model and **one** chat model. Any of the following combinations work for this lab; pick the pair you already have access to:
+  - OpenAI `text-embedding-3-small` + `gpt-4o-mini` — see the [OpenAI API reference](https://platform.openai.com/docs/api-reference).
+  - Cohere `embed-english-v3.0` + `command-r` — see the [Cohere API reference](https://docs.cohere.com/reference/).
+  - Voyage `voyage-3` embeddings + any chat model you have — see the [Voyage AI docs](https://docs.voyageai.com/).
+  - <!-- needs-research: confirm current on-device / open-weights option that the curriculum wants to recommend as a zero-cost alternative -->
+
+You do **not** need a vector database. In-memory NumPy is fine at this scale.
+
+## Constraints for this lab
+
+To keep the seams visible, these constraints are deliberate:
+
+- **No frameworks.** Do not use LangChain, LlamaIndex, or Haystack. You will use them later, but if you skip past the primitives now you will not know what they hide.
+- **No vector database.** Store embeddings in a JSON or `.npz` file. Loop over them at query time. Yes, this is slow — but at ten documents it is *fast enough*, and you get to see the actual distance function.
+- **Chunk by fixed character count.** 800 characters with 100 characters of overlap. This is not a good policy in general; that is what a later module is for. Use it here so that the chunking policy is not what you are debugging.
+
+## Steps
+
+### 1. Assemble a corpus
+
+Pick 5–10 short factual documents. Public sources with a clear owner work best — for example:
+
+- 3–4 pages of the [Python 3 language reference](https://docs.python.org/3/reference/).
+- 2–3 pages of the [PostgreSQL documentation](https://www.postgresql.org/docs/current/).
+- One or two `README.md` files from open-source projects you know well.
+
+Save each as a plain `.txt` file in `corpus/`. Keep each under ~10 KB. Record the source URL as the first line of each file, prefixed with `SOURCE:`.
+
+### 2. Ingest
+
+Write `ingest.py` that:
+
+1. Walks `corpus/`, reads each file.
+2. For each file, produces a list of chunks. Chunk boundary rule for this lab: 800 characters, sliding window with 100 characters of overlap.
+3. Attaches `{"source": <first line after SOURCE:>, "chunk_id": <int>, "text": <string>}` to each chunk.
+4. Writes the result to `chunks.jsonl`.
+
+### 3. Embed
+
+Write `embed.py` that:
+
+1. Reads `chunks.jsonl`.
+2. Calls your chosen embedding model in batches. Check the provider's docs for the maximum batch size; do not guess.
+3. Writes `embeddings.jsonl` — same rows as `chunks.jsonl` but with an added `embedding: [float, ...]` field.
+
+### 4. Query
+
+Write `query.py` that:
+
+1. Accepts a question on the command line.
+2. Embeds the question with the *same* model you used for the corpus.
+3. Loads `embeddings.jsonl` into memory. Computes cosine similarity between the question vector and every chunk vector. Returns the top *k* = 4.
+4. Builds a prompt of the form:
+
+    ```
+    Answer the question using only the passages below. If the passages do not contain the answer, say so. Cite each claim with the source URL in square brackets.
+
+    Passages:
+    [1] {chunk 1 text}  (source: {chunk 1 source})
+    [2] {chunk 2 text}  (source: {chunk 2 source})
+    …
+
+    Question: {user question}
+    ```
+
+5. Sends the prompt to your chat model.
+6. Prints the model's answer and, underneath it, the four sources it was given.
+
+### 5. Try it
+
+Run at least six queries you invent yourself, in three deliberate categories:
+
+- **Directly answerable** — the answer is verbatim in one chunk.
+- **Requires composition** — the answer needs two chunks.
+- **Not in the corpus** — the answer is genuinely absent.
+
+Record what happens in each case in `NOTES.md`.
+
+## Stop and think
+
+Before you look at the reference solution, answer these in your notes. The lab is more useful than the answer:
+
+1. What did your pipeline do on the "not in the corpus" queries? Did it refuse, hedge, or confabulate? What in your prompt (or the retrieval) drove that behaviour?
+2. Pick one query that returned a wrong or thin answer. Which of the five stages was most at fault? What would you change in *only* that stage?
+3. Your chunks overlap by 100 characters. Did any query benefit from the overlap? How would you tell if it did?
+4. If your corpus doubled in size, which stage becomes the first bottleneck? Which becomes the first *cost* problem?
+
+Bring these notes to the next module.
+
+## Extension (optional)
+
+If you finish early: add BM25 (via [`rank_bm25`](https://pypi.org/project/rank-bm25/), a small pure-Python library) as a second retriever, take the union of its top *k* and the vector retriever's top *k*, and see whether any of your test queries improve. This is a preview of the hybrid-search module later in the track — do not tune it seriously here.

lessons/mod-001-rag-foundations/lectures/01-what-is-rag.md

@@ -0,0 +1,86 @@
+# Lecture 1 — What is RAG?
+
+## Why the technique exists
+
+Large language models store what they know in their parameters. That works for facts that are ubiquitous in the training data and stable over time, and fails in three specific ways that RAG was invented to address:
+
+1. **Fresh facts.** A model's weights are frozen at training time. Anything that happened afterward — a released product, a policy change, a new customer record — is invisible unless retrieved.
+2. **Private facts.** Anything not in the public training corpus (your company's tickets, contracts, internal wiki) cannot be recalled from weights at all.
+3. **Attributable facts.** Even when the model does recall a fact, it cannot point to the sentence it came from. That makes disagreements ("where did you get that from?") unresolvable.
+
+Retrieval-augmented generation is the pattern of answering a query by first fetching relevant text from an external corpus and then conditioning the language model's answer on that text. The corpus is swappable, updatable, and citeable; the model's weights are not.
+
+The name and the modern formulation come from Lewis et al. (2020), *Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks*, published at NeurIPS 2020 (see [arXiv:2005.11401](https://arxiv.org/abs/2005.11401)). The paper introduced two variants — RAG-Sequence and RAG-Token — that jointly train the retriever and the generator. In practice, most production systems today use a decoupled retriever plus an off-the-shelf LLM; the *architecture* has moved on, but the *idea* — "retrieve then generate" — is the one that stuck.
+
+## Parametric vs. non-parametric memory
+
+Lewis et al. frame the design as combining two kinds of memory:
+
+- **Parametric memory** — knowledge encoded in the weights of a language model. Fast to access, blends across topics, hard to update, impossible to cite.
+- **Non-parametric memory** — knowledge held in an external corpus (documents, database rows, code files). Slow to access, discrete, easy to update, trivially citeable because retrieval returns the exact passage used.
+
+A RAG system is deliberately hybrid. The retriever hands the generator a small non-parametric slice for each query; the generator uses its parametric knowledge (grammar, reasoning, prose style) to compose an answer grounded in that slice.
+
+## Anatomy of a minimal RAG pipeline
+
+Almost every RAG system, from a fifty-line prototype to a production platform, is a specialisation of five stages:
+
+```
+    ┌──────────┐   ┌────────┐   ┌───────┐   ┌────────────┐   ┌───────────┐
+    │  Ingest  │──▶│ Chunk  │──▶│ Embed │──▶│  Retrieve  │──▶│  Generate │
+    └──────────┘   └────────┘   └───────┘   └────────────┘   └───────────┘
+     (offline)     (offline)    (offline)     (online)          (online)
+```
+
+Only the last two stages run on the hot path of a user query. The first three are batch jobs over your corpus.
+
+### 1. Ingest
+
+Pull raw content out of its source (a Confluence export, a PDF drop, a Postgres table, a Git repo) and put it in a form the rest of the pipeline can iterate over. Ingest owns character-set handling, structural parsing (headings, tables, code blocks), and metadata (`source_url`, `updated_at`, `access_scope`) that later stages will attach to every chunk.
+
+### 2. Chunk
+
+Split each document into passages small enough for a retriever to score independently and for a generator to fit alongside other passages in its context window. Chunking policy — size, overlap, boundary rules — is one of the largest levers in a RAG system, and later modules devote significant space to it.
+
+### 3. Embed
+
+Convert each chunk to a vector using an embedding model. Two chunks with similar meaning should land close together in the vector space. This is what makes semantic (as opposed to lexical) retrieval possible. Vectors are written to a **vector store** — an index that supports fast approximate nearest-neighbour search.
+
+### 4. Retrieve
+
+At query time, embed the user's query with the same embedding model, ask the vector store for the *k* nearest chunks, and return them. Real systems combine this with lexical retrieval (BM25) and often a reranker; those come in later modules.
+
+### 5. Generate
+
+Assemble a prompt that contains the user's query, the retrieved chunks, and instructions to answer *from the chunks*. Send it to a language model. Return the model's answer plus references back to the retrieved chunks.
+
+## Where RAG fails, and why the rest of this curriculum exists
+
+A minimal pipeline hides a lot of ways to be wrong. Each of the following failure modes has an entire module (or several) devoted to it later in the track:
+
+- **Bad chunk boundaries.** A fact is split across two chunks and neither retrieves. The answer either invents a bridge or admits ignorance.
+- **Vocabulary mismatch.** The user asks about "PTO" and the source document says "annual leave"; a pure-semantic retriever may still miss.
+- **Relevant but not sufficient retrieval.** The right chunk was retrieved but so were four wrong ones that dominate the prompt.
+- **Unfaithful generation.** The generator produces a fluent answer that contradicts the retrieved passage. This is sometimes called *grounded hallucination*.
+- **Stale index.** The document was updated in the source; the vector store still has the old version.
+- **Access leakage.** The retriever surfaces a chunk the querying user was not allowed to see.
+
+Notice that only the last two are strictly retrieval problems. Most RAG failures are actually *seam* problems between the stages listed above. Getting good at RAG is largely getting good at spotting which seam is leaking.
+
+## Terminology check
+
+You should be able to answer the following without looking anything up before moving on:
+
+- What is the difference between parametric and non-parametric memory?
+- Which of the five pipeline stages run offline, and which run online?
+- Why is chunking not just "split every 1,000 characters"?
+- Why do we embed the query with the same model we used to embed the chunks?
+- Give one failure mode that is *not* the retriever's fault.
+
+The exercise next door works these into a short classification task.
+
+## Further reading (official / primary)
+
+- Lewis, P., et al. (2020). *Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.* NeurIPS 2020. [arXiv:2005.11401](https://arxiv.org/abs/2005.11401) — the paper that named the pattern.
+- Guu, K., et al. (2020). *REALM: Retrieval-Augmented Language Model Pre-Training.* ICML 2020. [arXiv:2002.08909](https://arxiv.org/abs/2002.08909) — an earlier, jointly-trained retrieval-augmented LM that motivates several design choices in RAG.
+- Karpukhin, V., et al. (2020). *Dense Passage Retrieval for Open-Domain Question Answering.* EMNLP 2020. [arXiv:2004.04906](https://arxiv.org/abs/2004.04906) — dense retrievers, the workhorse of the retrieval stage.