feat: Progress Trail + Topic Appetizer for latency reduction by YishaiGlasner · Pull Request #136 · Sefaria/ai-chatbot

YishaiGlasner · 2026-05-19T18:02:47Z

Summary

Two features to reduce perceived latency in the Library Assistant, based on the team review meeting (2026-05-25):

Track A: Progress Trail

Shows all intermediate tool calls as a scrolling list instead of a single "Thinking..." bubble
States: running (spinner), complete (checkmark), error (X)
Toggle: "Show/Hide thinking (N steps)"
Collapsed in history, expanded during streaming

Track B: Topic Appetizer (parallel Haiku pipeline)

Fast parallel pipeline runs alongside the main Sonnet agent
Two-tier approach:
- Tier 1 (~300ms): Regex strips prompt wrappers → Sefaria api/name search → topic found
- Tier 2 (2-4s fallback): Haiku LLM extracts Jewish concept → retry search
Yellow box shows topic link within 5 seconds (measured: 5.1s on production)
Translation queries suppressed via deterministic router classification
Opens topic in new tab (safe for embedded chatbot)

Speed Architecture

The appetizer thread fires immediately after authentication, before session creation/summary loading/message saving. A 4KB SSE comment at stream start flushes reverse proxy buffers (nginx/gunicorn).

POST → auth → APPETIZER THREAD (parallel) → session/summary/save → SSE stream
                                                                    ↑
                                              appetizer result already in queue

Production Results

Metric	Before	After
Appetizer visible	26s	5.1s
First SSE event	21s	4.7s

Observability

Appetizer step-by-step timing attached as metadata.appetizer on Braintrust trace
SSE pipeline timing logs (generator start, proxy flush, first event)

Files Changed

server/chat/V2/views.py — appetizer thread, proxy flush, Braintrust metrics, SSE timing logs
server/chat/V2/appetizer/appetizer_service.py — two-tier service with per-step timing
server/chat/V2/appetizer/test_appetizer_service.py — 16 unit tests
server/chat/V2/agent/sefaria_client.py — search_topics() with slug fallback
server/chat/V2/agent/contracts.py — appetizer_data field
src/components/TopicAppetizer.svelte — yellow box UI
src/components/ProgressTrail.svelte — thinking trail UI
src/components/LCChatbot.svelte — wiring both components
src/i18n/locales/en.json — i18n strings

Test Plan

16 unit tests (appetizer service, search_topics, query extraction)
Frontend build passes
Playwright: appetizer appears within 6s locally (measured: 451ms)
Playwright: topic link opens in new tab
Playwright: translation suppression works
Playwright: progress trail renders with correct states
Production test: appetizer at 5.1s (down from 26s)
Production test: link opens without breaking chat

🤖 Generated with Claude Code

…uring streaming When the agent completes a source-fetching tool call (get_text, get_english_translations), a collapsible yellow box appears so the user can start reading the reference on Sefaria while Claude finishes its response. The box expands during streaming and collapses to one line once the full reply arrives. - Add tool_input to tool_end SSE progress events (tool_runtime.py) - New SourceSuggestion.svelte component (book icon, header, Sefaria link) - LCChatbot: track firstSourcePreview state, SOURCE_PROVIDING_TOOLS set, onProgress capture logic, streaming + post-response render slots - CSS via :global() rules in root component (shadow DOM requirement) - i18n: source.readWhileWaiting, source.readOnSefaria keys (en.json)

coolify-sefaria-github · 2026-05-19T18:02:54Z

The preview deployment for sefaria/ai-chatbot:server is ready. 🟢

Open app | Open Build Logs | Open Application Logs

Last updated at: 2026-05-27 15:06:23 CET

gitvelocity-reviewer · 2026-05-19T18:03:35Z

📊 Code Quality Score: 12/100

29 × 0.4 (Small ESF) = 11.6 → 12

Category	Score	Factors
🔭 Scope	8/20	4 files touched: new SourceSuggestion component, LCChatbot state additions, backend tool_runtime field, i18n strings. Single subsystem, no new API endpoints.
🏗️ Architecture	5/20	New SourceSuggestion component with clean prop interface. Minor backend AgentProgressUpdate field addition. No new service boundaries or design pattern introductions.
⚙️ Implementation	7/20	Svelte 5 runes ($state, $derived), streaming vs. static rendering modes, tool history correlation logic, URL encoding for external links, reactive state reset on send.
⚠️ Risk	4/20	Additive UI feature, easily reversible. Risk from potentially undeclared backend model field (+2). Global CSS class names risk style leakage (+1). No migrations or auth changes.
✅ Quality	4/15	No tests for new component or state logic (below 70% threshold). Good i18n usage, clear inline comments, defensive null handling fix included.
🔒 Perf / Security	1/5	rel=noopener noreferrer on external link. No benchmarks, no threat model, no rate limiting considerations.

_{Scored by GitVelocity · How are scores calculated?}

Copilot

Pull request overview

This PR surfaces a “read while you wait” source preview when the streaming agent finishes fetching a specific source, then pins a collapsible source link to the completed assistant response.

Changes:

Adds toolInput to tool_end progress events so the client can identify the fetched reference.
Introduces SourceSuggestion and client state to display the first successful source-providing tool result during and after streaming.
Adds English i18n strings and parent-level styles for the source suggestion panel.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
`server/chat/V2/agent/tool_runtime.py`	Includes tool input on streamed `tool_end` progress updates.
`src/components/LCChatbot.svelte`	Tracks the first fetched source and renders the live/collapsed source suggestion UI.
`src/components/SourceSuggestion.svelte`	Adds the stateless source suggestion panel component.
`src/i18n/locales/en.json`	Adds labels used by the source suggestion UI.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Adds SefariaClient.search_topics() which calls api/name/{query}?type=topic and returns [{title, slug}, ...] filtered to Topic completions only. Creates server/chat/V2/appetizer/ package with three passing unit tests. Also fixes pre-existing B905 ruff warning (zip without strict=). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…nd tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Renders a growing list of tool-call/status progress entries with collapse/expand toggle (collapsed=true shows "Show thinking (N steps)", collapsed=false streams all entries live). SVG icons reuse paths from LCChatbot.svelte for visual consistency. NOTE: requires two i18n keys in en.json (a later task): "progress.showThinking": "Show thinking ({count} steps)" "progress.hideThinking": "Hide thinking ({count} steps)" Until added, svelte-i18n will fall back to the raw key strings. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace single thinking bubble with scrolling progress trail showing all tool calls and status events. Trail collapses to toggle after answer arrives. Adds i18n keys and shadow DOM CSS. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…er event Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Show a Sefaria topic link in a yellow box within 5 seconds via the parallel Haiku appetizer pipeline. Includes collapse/expand, fade-in animation, and appetizer_click custom event for analytics. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

1. Fix executor leak: shut down appetizer_executor in finally block 2. Fix tool_end matching: use findLastIndex by toolName instead of array position — prevents stuck spinners when status events interleave with tool calls 3. Fix stuck status entries: mark all running trail entries as complete before persisting to assistantMessage 4. Fix empty state: show "Thinking..." fallback before first SSE progress event arrives 5. Fix NONE check: strip trailing punctuation before comparing Haiku's concept extraction response 6. Fix multi-instance: use $host() instead of document.querySelector for appetizer click tracking 7. Remove dead currentProgress variable (no longer read in template) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Tier 1: regex strips prompt wrappers, sends keywords to Sefaria name API (<500ms). Covers most prompts. Tier 2: Haiku extracts concept for abstract prompts (2-4s). Both run inside 5-second timeout. No Haiku cost for common queries.

…eline

The Sefaria api/name endpoint returns the Shabbat tractate (type=ref) before the Shabbat topic. Now search_topics tries a direct slug lookup via api/v2/topics/{slug} when the name API returns zero Topic-type results. Also adds debug logging to the appetizer thread. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The chat.appetizer logger was not registered in Django's LOGGING config, causing appetizer logs to be silently dropped and making it difficult to debug appetizer functionality. Added chat.appetizer logger entry with handlers=['chat_console'], level='INFO', propagate=False to match other chat module loggers.

Test results from May 25, 2026 after fixing chat.appetizer logging: - TA-1: PASS — appetizer appeared at ~12.8s for 'find me sources about Shabbat' - TA-2: PASS — link href=https://www.sefaria.org/topics/shabbat, text='Shabbat →' - TA-3: PASS — appetizer fully collapsible: clicking header toggles aria-expanded and removes/adds .appetizer-body - TA-4: PASS — translation prompt ('translate Genesis 1:1') does not generate appetizer, only discovery prompts do - TA-5: PASS — appetizer persists across multiple messages, remains collapsed initially Root cause of TA-1 regression was missing 'chat.appetizer' logger config in Django LOGGING dict.

Move appetizer thread start from inside generate_sse() to right after authentication — before session creation, summary loading, and message saving. This eliminates the 2-5s setup overhead and delivers the appetizer event within ~450ms of the request. Fix TopicAppetizer link click by using window.open() explicitly instead of relying on shadow DOM <a> default navigation which was unreliable. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Record timing for each appetizer pipeline step (classification, tier-1 search, tier-2 Haiku extraction, tier-2 search) and attach as metadata.appetizer on the Braintrust trace span. Non-intrusive — metrics are collected passively and attached after the agent completes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

window.location.href navigates away from the page and destroys the chatbot. Revert to window.open(_blank) which safely opens the topic in a new tab. Same-page SPA navigation requires Sefaria host-side changes to intercept the appetizer_click custom event. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…very Many reverse proxies (nginx, gunicorn, cloud LBs) buffer the first 4-8KB before forwarding to the client. This caused SSE events (including the appetizer) to be delayed ~25 seconds on production. Send a 4KB SSE comment at the start of the stream to flush these buffers. The comment is ignored by EventSource but forces the proxy to start forwarding immediately. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Log timestamps at each stage: appetizer thread start, SSE generator start, proxy flush, first event yield. All relative to request start time for easy correlation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remove target="_blank", window.open, and e.preventDefault. Use a plain <a href> so the browser's native click event propagates up through the shadow DOM. Sefaria's React app intercepts same-origin clicks and does client-side routing — the chatbot stays alive and the topic page loads in the same tab. Verified on production: clicking the topic link navigates to the topic page while the chatbot remains visible with conversation intact. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…andle /topics/ Sefaria's sefaria:bootstrap-url event handles text references (/Genesis.1.1) but silently ignores topic URLs (/topics/shabbat). Detect /topics/ paths in handleMessageLinkClick and fall back to window.open so topic links actually navigate. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…action The old tier-1 used English-only regexes + direct name API lookup. Hebrew prompts like 'תן לי מקורות על סיוון כ'' sent the raw Hebrew to the name API, where 'תן' matched 'תניא' (Tanya) — wrong topic. Now Haiku is the single entry point: it extracts the core Jewish concept from any prompt language, then the name API resolves the slug. Expected latency: ~1.5–2.5s (well under the 5s timeout). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Two bugs: 1. Haiku prompt too narrow: 'if not about Jewish texts, return NONE' filtered out Sefaria topics like parenting, money, relationships. New prompt scopes to 'topics that could appear in the Sefaria library' which covers universal topics through a Jewish-text lens. 2. Appetizer never dismissed: once shown it persisted forever, even after the answer arrived — cluttering the loading experience. Now cleared immediately after the answer message is committed to state (the topic is still saved on the message object for history). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…pic card Previous commit incorrectly cleared appetizerData (the topic card). The intended behavior: clear firstSourcePreview so the source mention disappears once the answer is shown, reducing loading-experience clutter. The TopicAppetizer persists as intended. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Deleted SourceSuggestion.svelte and all references: - import, SOURCE_PROVIDING_TOOLS constant, firstSourcePreview + sourcePreviewMessageId state, tool_end population logic, both render sites (streaming + history), and all CSS. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… in trail Appetizer improvements: - Haiku now returns up to 3 ranked candidates (comma-separated) instead of a single concept. Tries each until one matches — fixes cases like 'Herod the Great' where the first candidate fails but 'Herod' hits. Multiple candidates also add natural think-time (~2-4s total). - Canonical short names prompted: 'Herod' not 'Herod the Great'. - search_topics now accepts PersonTopic and AuthorTopic in addition to Topic — 'Herod' was filtered out because it's type PersonTopic. ProgressTrail: - Tool entry descriptions with single-quoted refs (e.g. 'Pesachim 119b') are now rendered as clickable sefaria.org links. SourceSuggestion: - File restored (kept for future re-enablement, currently unused). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…e double quotes) Refs in tool descriptions use double quotes e.g. "Mishnah Shabbat 7:2". Previous regex only matched single quotes. Updated to use backreference so both 'Pesachim 119b' and "Mishnah Shabbat 7:2" are linkified. Non-ref quoted strings ("both", "en") still pass through unchanged since refToUrl returns null for them. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Upgrade appetizer model from Haiku to Sonnet for smarter topic extraction - Return up to 3 matched topics (exhaustive search, deduplicated by slug) - Bump timeout from 5s to 8s for Sonnet inference + sequential API calls - SSE payload now sends topics array instead of single flat object - Frontend renders comma-separated topic links (no arrow, no hint) - New header: "Discover sources connected to related topics..." - Same-page navigation via sefaria:bootstrap-url event (no new tab) - Migration guard for old flat appetizerData format in localStorage - ProgressTrail refs styled bold with external-link icon - Refined prompt: prefer specific topics, distinct candidates Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Sonnet generates full markdown responses with search XML tags instead of comma-separated topic names. Haiku follows the system prompt correctly and returns quality multi-topic results in ~1.5-2.5s. Keep 8s timeout for headroom. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ction Sonnet ignores system prompts and enters "assistant mode" with free-form text. Tool-forcing via tool_choice + extract_topics tool constrains output to structured data — physically impossible to produce markdown. Returns 3 quality topics in ~2-3.5s. Based on CandleKeep research: Anthropic Prompting Best Practices recommends structured output tools with tool_choice as the strongest format enforcement for Claude 4.x models (prefilling is deprecated/errors on 4.x). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ation <a> tags with href always navigate even with e.preventDefault() in shadow DOM context. Switched to <button> elements styled as links — clicking dispatches sefaria:bootstrap-url for in-page navigation without any default browser navigation to intercept. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…lsewhere Svelte 5 delegated onclick doesn't reliably preventDefault on <a> tags in shadow DOM. Fix: use Svelte action (use:attachClickHandler) with direct addEventListener for reliable preventDefault. On sefaria.org: dispatches sefaria:bootstrap-url for in-page navigation. Off sefaria.org: window.open as fallback (opens topic in new tab). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Header was truncating in narrow panel. Shortened to "Discover sources on related topics". Added margin-right: 4px on comma separators so topic names don't run together. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…page navigation Previously, message-body links to /topics/* were opening in a new tab. This removes the early return so they dispatch the sefaria:bootstrap-url event, enabling in-page navigation on Sefaria just like ref links. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Filter external-domain links: open in new tab instead of dispatching sefaria:bootstrap-url with a foreign path - Filter off-Sefaria embeds: open topic/ref on sefaria.org in new tab when the host page isn't sefaria.org (mirrors handleAppetizerClick) This makes handleMessageLinkClick consistent with handleAppetizerClick, which already has the onSefaria hostname guard. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

YishaiGlasner requested review from Copilot, yitzhakc and yodem May 19, 2026 18:02

Copilot started reviewing on behalf of YishaiGlasner May 19, 2026 18:03 View session

Copilot AI reviewed May 19, 2026

View reviewed changes

yodem and others added 20 commits May 25, 2026 16:11

Merge branch 'main' (early part) into waiting-source

3d8b23f

feat(appetizer): add AppetizerService with Haiku concept extraction a…

0dbad0a

…nd tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat(appetizer): run parallel Haiku topic lookup and emit SSE appetiz…

ab9d930

…er event Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix(sefaria): restore limit param in search_topics without type filter

995a7a3

test(appetizer): comprehensive tests for two-tier keyword + Haiku pip…

efd062e

…eline

fix(appetizer): navigate topic link in same page instead of new tab

32bd109

yodem changed the title ~~Surface found source as "read while waiting" panel during streaming~~ feat: Progress Trail + Topic Appetizer for latency reduction May 26, 2026

yodem and others added 16 commits May 26, 2026 14:18

chore(appetizer): log Haiku raw response and decision for every call

0c61148

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

yodem mentioned this pull request Jun 10, 2026

feat: appetizer latency metric + new LA click tracking #150

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Progress Trail + Topic Appetizer for latency reduction#136

feat: Progress Trail + Topic Appetizer for latency reduction#136
YishaiGlasner wants to merge 38 commits into
mainfrom
waiting-source

YishaiGlasner commented May 19, 2026 •

edited by yodem

Loading

Uh oh!

coolify-sefaria-github Bot commented May 19, 2026 •

edited

Loading

Uh oh!

gitvelocity-reviewer Bot commented May 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

YishaiGlasner commented May 19, 2026 • edited by yodem Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Track A: Progress Trail

Track B: Topic Appetizer (parallel Haiku pipeline)

Speed Architecture

Production Results

Observability

Files Changed

Test Plan

Uh oh!

coolify-sefaria-github Bot commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gitvelocity-reviewer Bot commented May 19, 2026

📊 Code Quality Score: 12/100

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

YishaiGlasner commented May 19, 2026 •

edited by yodem

Loading

coolify-sefaria-github Bot commented May 19, 2026 •

edited

Loading