Add keyframe animation pipeline and translation quality improvements - #2
Merged
Conversation
Animation engine (signs_library.js, avatar.js): - Add signWithFrames() factory, prebakeFrameQuats(), findFrame(), and slerpBetweenFrames() to support real motion-capture keyframe sequences - TransitionEngine.tick() now has a dual path: plays through keyframe curves when .frames is present, falls back to SLERP for existing signs - avatar.js respects per-sign .duration and wires sign-level NMM data (browLift, headNod, etc.) through existing setNMMs() automatically - All 284 hand-crafted signs are fully backward-compatible Data pipeline (convert_signs.py, scripts/): - convert_signs.py v2.0 extracts full frame sequences (not just peak frame) using angular-displacement keyframe selection (30fps → 8–12 keyframes) - scripts/record_signs.py: new MediaPipe Holistic webcam recorder that outputs keyframes in the same format as the converter - scripts/merge_sign_data.py: merge tool that generates signs_library_generated.js from real data without touching the source Translation (sasl_transformer/, backend/): - grammar_rules.py: 5 new SASL rules (classifiers, SASSes, topicalization, plurality via MANY, spatial loci) + uncertain token flag - models.py: GlossToken.uncertain field; TranslationResponse gains sign_coverage and fingerspelled_words - transformer.py: coverage scoring + conservative LLM retry when <70% of tokens have known signs - backend/main.py and WebSocket broadcast now surface sign_coverage and fingerspelled_words to the frontend Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds support for real keyframed sign animation (motion-capture/recorded sequences) and improves SASL translation quality/metadata surfaced to clients.
Changes:
- Added keyframe-capable sign factory/utilities and extended
TransitionEngineto play sign-internal frame sequences when available. - Introduced data tooling: upgraded
convert_signs.pyto output keyframes + duration, plus newrecord_signs.py(MediaPipe recorder) andmerge_sign_data.py(override generator). - Improved translation robustness: added new grammar rules, introduced
uncertaintokens, computedsign_coverage, and exposedfingerspelled_wordsvia REST/WebSocket.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
src/windows/deaf/avatar.js |
Applies sign-level NMM markers and uses sign-provided duration for hold time. |
signs_library.js |
Adds keyframe interpolation helpers and keyframe playback mode in TransitionEngine. |
scripts/record_signs.py |
New webcam recorder producing keyframe JSON compatible with the keyframe sign format. |
scripts/merge_sign_data.py |
New tool to generate a JS overrides layer for real-data frames/durations. |
sasl_transformer/transformer.py |
Adds coverage-based conservative retry + uncertain handling and coverage/fingerspell metadata. |
sasl_transformer/models.py |
Extends response/token models with uncertain, sign_coverage, fingerspelled_words. |
sasl_transformer/grammar_rules.py |
Adds new SASL grammar guidance and JSON output schema updates (uncertain flag). |
convert_signs.py |
Upgrades converter to extract all frames and downsample to keyframes with durations. |
backend/requirements.txt |
Adds MediaPipe/OpenCV deps for recording script usage. |
backend/main.py |
Surfaces sign_coverage and fingerspelled_words in API and WebSocket payloads. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Five jobs:
- python-lint: ruff on backend/, sasl_transformer/, scripts/ and convert_signs.py
- python-tests: pytest tests/ (rule-based tests, no API key required)
- python-imports: smoke-import of all new modules (sasl_transformer, convert_signs)
- js-syntax: node --check on signs_library.js, avatar.js, src/main.js
- signs-library-check: Node script that verifies all required exports exist,
sign count ≥ 200, TransitionEngine methods, and findFrame() correctness
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rors Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Ignore E402/F601/W292 — pre-existing issues in backend files not
introduced by this PR
- Write check_lib.js to GITHUB_WORKSPACE instead of /tmp so
require('./signs_library.js') resolves correctly
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- convert_signs.py: remove unused 'os' import, fix bare f-strings (F541) - merge_sign_data.py: fix bare f-strings (F541) - record_signs.py: remove unused 'os'/'time' imports (F401), remove assigned-but-never-used body_front variable (F841) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Merge resolution: - Take main's modular backend (routers/, services/, ws/) as base of truth - Port sign_coverage + fingerspelled_words into sasl_pipeline, ws handler, and speech router (previously in the old monolithic backend/main.py) - Rewrite conservative coverage retry to use Ollama (main removed Gemini; the old settings.gemini_api_key access would have crashed translate()) - Enrich rule-based translations via translate_with_rules() so coverage metadata is accurate on the fallback path - backend/requirements.txt deleted (main moved deps to root); mediapipe + opencv-python appended to root requirements.txt for the sign recorder Copilot review fixes: - TransitionEngine: transition INTO the first keyframe before playing frames, then mark done (was: frames first, then SLERP backwards) - TransitionEngine._interpolate: target first-keyframe handshape when entering a keyframed sign (was lerping toward the last frame's hand) - findFrame(): clamp t beyond first/last frame; clamp localT to [0,1] - signWithFrames(): validate durationMs > 0, frame shape; sort by t - merge_sign_data: generated overrides now retarget R/L and _Rq/_Lq start/end from first/last keyframes and sort frames before prebake - merge_sign_data: fix invalid '}},' emitted per sign (broken JS output) - convert_signs: propagate .pkl basename into output 'source' metadata - write_generated_library docstring matches actual behaviour - print_coverage_report uses all_sign_names as fallback name set - tests/test_coverage_enrichment.py: 7 tests for uncertain-token fingerspelling and sign_coverage ratios Windows + CI robustness: - Scripts reconfigure stdout to UTF-8 on cp1252 consoles (crash fix) - test_e2e_pipeline skips with clear message when a stale backend without /auth/session-secret occupies port 8000 - CI: sign-count threshold 200 -> 100 (main consolidated the library), lint tests/ too, read scripts with explicit utf-8 - Remove dead imports / bare f-strings across backend and scripts (ruff) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
signs_library.jsandavatar.jsnow support real motion-capture keyframe sequences viasignWithFrames(). TheTransitionEngineplays through multi-frame curves when data is available, falling back to the existing SLERP path for all 284 hand-crafted signs (fully backward-compatible).convert_signs.pyupgraded to extract full frame sequences using angular-displacement keyframe selection (30fps → 8–12 keyframes). Two new scripts:scripts/record_signs.py(MediaPipe webcam recorder) andscripts/merge_sign_data.py(merge tool that generatessigns_library_generated.jswithout touching the source library).sign_coverageandfingerspelled_wordsare now surfaced in API responses and WebSocket broadcasts.Test plan
signWithFrames()with 3 keyframes and verify the avatar traces through the intermediate posepython convert_signs.py --inspect <pkl>against a SignAvatars .pkl and confirm output hasframesarray with 8–12 entriespython scripts/record_signs.py, record a sign, confirm output JSON matches the converter formatpython scripts/merge_sign_data.py --data poses.json --output signs_library_generated.js --reportand verify the coverage tablesign_coverage≥ 0.7 in the response (no retry logged)fingerspelled_wordsis populated🤖 Generated with Claude Code