Fix RAG eval, guardrail ordering, and agent indexing in v2 notebook - #6
Open
Divergent-Code wants to merge 2 commits into
Open
Fix RAG eval, guardrail ordering, and agent indexing in v2 notebook#6Divergent-Code wants to merge 2 commits into
Divergent-Code wants to merge 2 commits into
Conversation
The v2 notebook's narrative described behavior the code didn't implement. Three substantive fixes plus two smaller ones: - RAG coverage eval always reported 100%: Chroma's query() returns the nearest top_k chunks regardless of match quality, so every requirement looked "covered" and Step 4 could never surface a real gap. Add a cosine distance threshold (and set the collection to cosine space) so weak matches are dropped and gaps show up as empty evidence lists. - Guardrails were decorative: the injection screen ran in the last cell, after the JD had already been fed through the whole pipeline, and rate_limited_call/logged_call were defined but never invoked. Define the guardrails right after setup, screen the JD at intake before any prompt, route every model call through logged_call, and repurpose Step 8 as a monitoring/audit recap. - Agent didn't index its own chunks: run_resume_tailoring_agent logged "Indexed N chunks" but never added them to Chroma and retrieved against the stale global collection. Add a reusable index_chunks() tool and give the agent its own collection. - Chroma stale state: index_chunks() delete-then-recreates the collection so re-runs don't leave chunks from a previous resume behind. - v1 GitHub call crashed on a nonexistent user (API returns a dict, not a list); guard the response and degrade gracefully. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up robustness fixes in the same two notebooks: - extract_jd_requirements: the model can still emit a preamble line or a section header despite "plain list" instructions; strip bullets/numbering and drop blank, ':'-terminated, or over-long lines so they don't become fake requirements. - Guard the GitHub fetch in the linear path too (the agent already did), so a bad/rate-limited username degrades to "no repo evidence" instead of crashing the cell. - fetch_github_repos: request most-recently-updated repos with an explicit per_page, and document the unauthenticated 60/hr + one-page cap. - Add a PII caveat to the Inputs step — the resume and scraped JD are sent to the Gemini API; fitting for a notebook that teaches AI safety. - V1 notebook ended with a dangling "V1.1" header and no content, while V2 repeatedly references "Phase 1 / V1.1"; replace it with a short summary of what V1.1 added so the progression reads coherently. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The v2 notebook is a great teaching progression, but in a few places the narrative describes behavior the code doesn't actually implement — which matters in a workshop, since students learn from the mismatch. This PR closes those gaps.
Substantive fixes
1. RAG coverage eval always reported 100%.
retrieve_relevant_experiencecalls Chroma'squery(..., n_results=top_k), which always returns the nearesttop_kchunks regardless of match quality. Soevaluate_rag_coveragecould never see an empty evidence list — coverage was always 100% and the "real gaps" Steps 2 & 4 promise to surface never appeared. Added a cosine-distance threshold (and set the collection to cosine space) so weak matches are dropped and genuine gaps show up.2. Safety guardrails were decorative. The injection screen was the last cell, running after the JD had already been fed through requirement-extraction, generation, critique, and the agent;
rate_limited_call/logged_callwere defined but never called. Moved the guardrails to right after setup, screen the JD at intake before any prompt is built, routed every model call throughlogged_call, and repurposed Step 8 as a monitoring/audit recap.3. The agent didn't index its own chunks.
run_resume_tailoring_agentlogged "Indexed N chunks" but never added them to Chroma — it retrieved against the stale global collection built earlier. Added a reusableindex_chunks()tool and gave the agent its own collection, so it's self-contained.Smaller fixes
index_chunks()delete-then-recreates the collection, so re-running with a different/shorter resume doesn't leave old chunks behind.{"message": "Not Found"}), not a list — the comprehension then crashed. Guarded the response so it degrades gracefully.Notes
RELEVANCE_MAX_DISTANCE = 0.75) is deliberately loose and commented as something to tune on real data.🤖 Generated with Claude Code