yt-chat

blog link: https://evergreen-learning-5e5.notion.site/Built-a-Youtube-Q-A-bot-that-refuses-to-guess-3accb2d4f8b4805a89e3ee559101cc11 Ask questions about a YouTube video from your terminal. You get an answer plus timestamps you can click to jump to that exact moment.

If the video doesn't cover what you asked, it says so instead of making something up.

$ yt-chat https://youtu.be/zjkBMFhNj_g

  [1hr Talk] Intro to Large Language Models
  Andrej Karpathy · 59:47 · 140 chunks

› What two files make up a large language model?

  A parameters file holding the model weights (140 GB for Llama 2 70B) and a
  run file with the code that executes them, about 500 lines of C [1][2].

  [1] 1:34   https://youtube.com/watch?v=zjkBMFhNj_g&t=94s
  [2] 2:15   https://youtube.com/watch?v=zjkBMFhNj_g&t=135s

› What is the attention mechanism?

  I cannot find this information in the video.

That last bit is the whole point. This video never mentions attention, so it refuses. Across 33 test questions the videos genuinely don't answer, it refused 32.

---

Setup

git clone https://github.com/sabertooth-123/RAG_yt_chatbot
cd RAG_yt_chatbot
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -e ".[embeddings,llm,metadata,dev]"

If you have an NVIDIA GPU, install the CUDA build of torch first, otherwise it runs on CPU and everything is slower:

pip install torch --index-url https://download.pytorch.org/whl/cu124

Grab a free API key from Groq, copy .env.example to .env, and paste it in. Or set YTCHAT_LLM_PROVIDER=ollama and run a model locally with no key at all.

---

Commands

Chat with a video

yt-chat https://youtu.be/VIDEO_ID

Opens a chat session. First run takes a minute or two while it downloads the subtitles and processes them. After that it's instant, because everything is cached.

You can also just paste the video ID:

yt-chat VIDEO_ID

Options you can add:

Option	What it does
--retriever dense	Pick the search method: dense, sparse, or hybrid
--top-k 8	How many passages to look at (default 5)
--llm groq	Which AI service to use this run
--force	Ignore the cache and reprocess the video
--quiet	Hide the progress messages

Inside the chat

Just type your question. Or use these:

Command	What it does
/ask <question>	Same as typing the question
/sources	Shows where the last answer came from, with quotes
/history	The conversation so far
/change-retriever hybrid	Switch search method without restarting
/compare <question>	Runs all three search methods side by side
/video	Video title, length, number of passages, models in use
/debug	Why the last answer came out how it did — scores, and which check refused it
/clear	Forget the conversation
/help	List these
/exit	Quit

Follow-up questions work. Ask "What is a jailbreak?" then "Why does that work?" and it figures out what "that" means before searching.

Ask one question and exit

yt-chat ask https://youtu.be/VIDEO_ID -Q "What is the LLM OS?"

Handy for scripts. Exits with code 0 if it answered, 1 if it refused.

Manage the cache

yt-chat cache stats

Shows how many videos and passages are stored locally.

yt-chat cache clear --video https://youtu.be/VIDEO_ID

Deletes everything saved for one video. It won't wipe the whole cache in one command, on purpose.

Measure how well it works

yt-chat eval calibrate benchmarks/karpathy_llm_intro.yaml -r dense

Finds the best cutoff for when to refuse. Costs nothing — no AI calls involved.

yt-chat eval run benchmarks/karpathy_llm_intro.yaml --no-judge

Compares all three search methods. --no-judge skips the AI grading, which makes it free.

yt-chat eval run benchmarks/karpathy_llm_intro.yaml -o docs/evaluation.md

The full version, including answer quality. This one uses a lot of API calls.

yt-chat eval draft https://youtu.be/VIDEO_ID -Q "your question"

Suggests time ranges for a new test question. Check them yourself before trusting them — it's only guessing.

Scripts

python scripts/run_eval_queue.py

Runs the full evaluation across all videos and search methods. Free API tiers cap you daily, so it saves progress and picks up where it left off. Run it again the next day.

python scripts/run_eval_queue.py --list

Shows what's done and what's left.

python scripts/merge_eval.py --markdown

Combines all the results into one table.

python scripts/sweep_chunk_size.py

Tests different passage sizes. Free to run.

Tests

pytest -q

195 tests, about two seconds. No internet, no API key, no model downloads needed.

---

Settings

Everything lives in .env, so you can change how it behaves without touching any code.

Setting	What it does
YTCHAT_LLM_PROVIDER	groq, gemini, openrouter, or ollama
YTCHAT_RETRIEVER	Default search method
YTCHAT_MIN_SCORE	How sure it has to be before answering. Higher = refuses more
YTCHAT_TOP_K	How many passages to send to the AI
YTCHAT_MAX_CHARS	Passage size. Changing this re-cuts the transcript
YTCHAT_EMBEDDING_MODEL	Which model converts text to numbers
YTCHAT_ENABLE_RERANK	Turn on the slower, more accurate second pass
YTCHAT_ENABLE_QUERY_REWRITING	Turn follow-up rewriting on or off

One warning: MIN_SCORE is tuned for the current passage size and embedding model. Change either of those and you should re-run eval calibrate.

---

Experiments

Every test I ran against the finished system, with the command to reproduce it.

Experiment	Command	What it does
Chunk size sweep	python scripts/sweep_chunk_size.py	Re-cuts each video at 400/700/1000/1500 characters and measures both recall and citation accuracy at each size. Free — no AI calls.
Refusal threshold sweep	yt-chat eval calibrate benchmarks/karpathy_llm_intro.yaml -r dense	Sweeps the confidence cutoff from 0.00 to 1.00 and reports correct vs. wrongly refused at each step. Free — checks retrieval confidence only, never generates an answer.
Retriever comparison (fast)	yt-chat eval run benchmarks/karpathy_llm_intro.yaml --no-judge	Compares dense, sparse, and hybrid search on one video. --no-judge skips AI grading, so it's free.
Full comparison, all videos	python scripts/run_eval_queue.py	Runs every video against every search method with full AI grading. Costs real API calls — saves progress and resumes across days on a free tier. Check status with --list.
Combine results	python scripts/merge_eval.py --markdown	Merges the queue's output into one results table.
Draft test questions	yt-chat eval draft https://youtu.be/VIDEO_ID -Q "your question"	Suggests candidate time ranges for a new benchmark question. Always check them by hand before trusting them.
Sanity check on new content	yt-chat ask https://youtu.be/VIDEO_ID -Q "your question"	How I found the threshold doesn't transfer to conversational video — ran it against a sitcom episode and every question got refused. See What doesn't work.
Regression check	pytest -q	Confirms fixed bugs — like a scoring formula that once returned 1.09 on a 0–1 scale — stay fixed.

Two more are designed but not run yet: comparing reranking on vs. off (YTCHAT_ENABLE_RERANK), and comparing the combined AI-judge call against the original four-separate-call version (combined=False in evaluation/generation_metrics.py). Both exist in the code; neither has been benchmarked, so I'm not claiming a result for either.

---

How it works

YouTube URL
    ↓
get subtitles (with timestamps)
    ↓
clean them (strip [Music], remove repeated lines)
    ↓
cut into ~700 character passages, keeping the timing
    ↓
convert each passage to numbers (embeddings) → FAISS + SQLite
    ↓
search (meaning-based, keyword, or both)
    ↓
send only the top passages to the AI with strict instructions
    ↓
check the answer's sources are real → answer + timestamps

The part I'd point at is the timing. Most projects glue the subtitles together and then guess which timestamp a passage came from. This one records which characters belong to which moment, so even a passage that starts mid-sentence gets an accurate time.

There are three separate checks that can refuse a question:

1. Nothing relevant was found — refuses before calling the AI at all
2. The AI itself says the passages don't answer it
3. The answer cites a source that doesn't exist, so it gets thrown away

No LangChain or LlamaIndex here. The keyword search, the rank fusion, and the whole evaluation setup are written directly.

For the full version — how the timestamp mapping actually works, why chunking switches strategy on punctuation, how the cache keys are layered, and the reasoning behind each design call — see docs/technical_walkthrough.md.

---

Results

Tested on three Karpathy videos totalling 6h45m, with 35 questions I wrote and checked by hand. 24 are answerable, 11 aren't.

Search method	Finds answer	Faithful	Wrongly refused	Correctly refused
dense	0.85	0.77	0.08	100%
hybrid	0.83	0.83	0.17	91%
sparse	0.60	0.86	0.38	100%

Dense is the default because it wins on nearly everything.

Sparse looks best on "faithful" only because it refuses 38% of the questions it could have answered — a refusal can't be unfaithful, so it scores full marks automatically. Worth knowing if you ever report that metric on its own.

Full breakdown, including the eight things measurement caught that I'd never have found by reading the code, is in docs/evaluation.md.

---

What doesn't work

It's tuned for lecture-style video

I pointed it at a sitcom episode to see what would happen. Every question got refused, including reasonable ones:

› who is moving out
  I cannot find this information in the video.

› what are they arguing about
  I cannot find this information in the video.

Not a bug — the confidence threshold is wrong for that content. MIN_SCORE is calibrated on technical talks, where a relevant passage scores 0.70–0.86 because the passage is genuinely about one topic. Sitcom dialogue isn't about anything in that sense. A single passage looks like this:

mocking me no I'm not mocking you or your beautiful G Moon what's up
nothing Monica and I had a...

Four people, three subjects, no topic. I probed ten questions on that episode and the entire confidence range came out 0.50–0.65 — the whole distribution sits below a threshold set at 0.65, so nothing could ever get through.

If you're using this on conversational video — interviews, podcasts, anything narrative — drop the threshold and recalibrate:

YTCHAT_MIN_SCORE=0.45

Then build a small benchmark from your own content and run yt-chat eval calibrate to find the real number. The sweep costs no API calls.

Summary questions don't work at all

"What is this video about?" has no answer sitting in any one passage, so retrieval has nothing to retrieve. Ask about specific things instead. Answering that properly needs a different technique — summarising every chunk and merging the results — which isn't built.

Everything else

• Videos with subtitles turned off. About 15% of YouTube. Would need speech recognition, which isn't built yet.
• Visual content. The transcript is dialogue only. Anything conveyed by what's on screen — physical comedy, diagrams referred to as "this bit here" — is invisible to the system.
• Non-English videos work through translated subtitles, but retrieval is noticeably worse.
• Small sample. 24 answerable questions from one speaker. Differences under about 0.15 in that table are one or two questions changing sides, so don't read too much into them.

---

MIT licensed.