feat: model-backed, evidence-cited quiz generation — no stubs left - #28
Merged
Conversation
Replaces the last stub in the codebase. The quiz engine now generates citation-carrying items and wires the full attempt loop. @study-os/quiz-engine rework: - QuizProvider contract: unit in -> evidence-cited items out, with GenerationRun provenance (provider/model/promptVersion/inputSha256/ tokens) - Quote-anchored citations: the model NEVER emits character offsets (unreliable) — it copies evidence quotes verbatim and resolveQuotesToCitations() anchors them deterministically via indexOf; a paraphrased or missing quote fails the whole batch. This makes 'citations resolve 100%' an invariant, not an aspiration. - AnthropicQuizProvider: claude-opus-4-8, adaptive thinking, structured outputs; Korean prompt treats <자료> as untrusted data and forbids ambiguous multi-answer items; evidence_sufficient=false, refusal, truncation, malformed JSON all fail closed - MockQuizProvider: deterministic, offline, all three quiz types, citations resolve by construction; default without ANTHROPIC_API_KEY - Type-specific validation: MCQ 3-5 choices with exactly 1 correct, fill-in-the-blank requires a ____ marker, difficulty 1..5 - Korean-aware grading replaces exact lowercased match: NFC normalization, whitespace collapse, trailing-punctuation strip, acceptedAnswers support; gradingMethod recorded on every Attempt Persistence + API (loop closure): - persistQuizDraft(): GenerationRun + QuizSet + items + choices + SourceSpans in ONE transaction, with unit-relative citation offsets mapped onto the immutable SourceRevision (unit.citationStart + start) so QuizItemCitation -> SourceSpan -> SourceRevision.rawText resolves - POST /api/units/:unitId/quiz: generate + persist + return cited items (422 if the unit lacks citation offsets; 502 fail-closed generation) - POST /api/quiz-items/:quizItemId/attempts: normalized grading, raw Attempt (latency/confidence/gradingMethod), and a wrong answer auto-opens an ErrorEpisode(open) — upload -> quiz -> attempt -> episode -> review queue now runs end to end over HTTP Tests: 20 quiz-engine unit tests (quote anchoring incl. paraphrase rejection, per-type contracts, determinism, NFC/NFD grading) + 6 CI integration tests walking the whole loop against real Postgres, including the revision-mapped span resolution check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 14, 2026
mikim
added a commit
that referenced
this pull request
Jul 14, 2026
#29) - PR range #18–#26 → #18–#28 - 'application wiring lands with M2' → first half of the loop is live (attempt → auto ErrorEpisode → review queue); cause suggest/confirm + interventions are the next slice - Remediation-model row updated to match - Local-development endpoint list gains the quiz endpoints Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the last stub in the codebase. The quiz engine now generates evidence-cited items (Claude-backed or deterministic mock) and the attempt loop is wired end to end: upload → quiz → graded attempt → auto-opened
ErrorEpisode→ review queue, all over HTTP.The grounding design
Models are unreliable at character offsets, so the model never emits offsets — it copies verbatim evidence quotes, and
resolveQuotesToCitations()anchors them deterministically (indexOf). A paraphrased or missing quote fails the whole batch. That turns the repo's "citations resolve 100%" gate into an invariant enforced in code, not an aspiration.Persistence completes the chain: unit-relative offsets are mapped onto the immutable
SourceRevision(unit.citationStart + offset), soQuizItemCitation → SourceSpan → SourceRevision.rawTextalways resolves — verified by integration test against real Postgres.Contents
@study-os/quiz-engine: QuizProvider contract + provenance;AnthropicQuizProvider(claude-opus-4-8, adaptive thinking, structured outputs;<자료>treated as untrusted data; ambiguous items forbidden; every failure path fail-closed) +MockQuizProvider(deterministic, offline, all 3 types); type-specific validation (MCQ 3–5 choices/exactly 1 correct,____blanks, difficulty 1–5).acceptedAnswers;gradingMethodrecorded on every rawAttempt.persistQuizDraft(one transaction) +POST /api/units/:id/quiz+POST /api/quiz-items/:id/attempts— wrong answers auto-openErrorEpisodes that appear in the existing review queue.Verified
20 new quiz-engine unit tests (quote anchoring incl. paraphrase rejection, per-type contracts, determinism, NFC/NFD grading) + 6 CI integration tests walking the full loop against real Postgres. Local: lint/typecheck/build/smoke green, 100+ tests.
🤖 Generated with Claude Code