fix: landmark dim, preprocessing consistency, training script - #1
Open
esmasila wants to merge 3 commits into
Open
fix: landmark dim, preprocessing consistency, training script#1esmasila wants to merge 3 commits into
esmasila wants to merge 3 commits into
Conversation
…cript - preprocess.py: Extract all 468 face landmarks (was 33) to match MultimodalDigitsCNN's expected landmark_dim=1629 (543*3). This eliminates 1305 wasted zero-padded features and provides full facial expression data. - service.py: Remove adaptive thresholding from /predict endpoint that was destroying color information needed by the CNN image branch. Unify landmark padding logic with a named constant for clarity. - config.py: Fix SEQUENCE_MODEL_CONFIG num_classes from 100→10 to match CLASS_TO_TURKISH which only defines 10 digit classes. - requirements.txt: Add missing ultralytics>=8.0.0 (required for YOLO imports in service.py and train_yolo.py). - signbridge/training/train_multimodal.py: New training script for MultimodalDigitsCNN. Handles landmark caching, data augmentation, train/val/test split, early stopping, LR scheduling, and saves checkpoints/multimodal_digits_best.pt. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add autsl_transformer/ module (inference.py + trained model files): - SignTransformerPro architecture (6 layers, 384 d_model, 12 heads) - 226 Turkish sign language words (AUTSL dataset) - Val acc: 83.9%, Test acc: 81.2% - 718 features: raw(225) + velocity + acceleration + distances - Refactor service.py to use AUTSLPredictor as primary model: - POST /predict/frame — single frame, server-side rolling buffer (30 frames) - POST /predict/sequence — placeholder for batch sequence input - POST /reset/buffer — clear the rolling buffer - GET /model/info — shows AUTSL + YOLO model status - Remove unused multimodal/CNN code paths - Update .gitignore: exclude autsl_transformer/old_model/ and large .npz Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
preprocess.py'de_combine_landmarks()yalnızca 468 yüz landmark'ının ilk 33'ünü alıyordu. Bu durum toplam feature boyutunu 1629 yerine 324 yapıyordu.MultimodalDigitsCNN'in beklediğilandmark_dim=1629ile uyumsuzdu. Artık tüm 468 yüz landmark'ı kullanılıyor./predictendpoint'i yalnızca orada uygulanan adaptive thresholding ile CNN'nin renk bilgisini yok ediyordu. Kaldırıldı. Her iki endpoint de artık aynı ham görüntüyü işliyor.ultralyticseklendi:requirements.txt'eultralytics>=8.0.0eklendi;service.pyvetrain_yolo.pybu paketi import ediyor ama eksikti.SEQUENCE_MODEL_CONFIG.num_classesdüzeltildi: 100 → 10 (yalnızca rakam sınıfları tanımlı).signbridge/training/train_multimodal.py— landmark önbellekleme, veri artırma, train/val/test bölünmesi, early stopping, LR scheduling vecheckpoints/multimodal_digits_best.ptkaydını içeriyor.Modeli eğitmek için
Test plan
python -m signbridge.training.train_multimodalbaşarıyla çalışır vecheckpoints/multimodal_digits_best.ptoluştururpython -m signbridge.api.servicemodel checkpoint ile başarıyla başlar/predictve/predict/imageendpoint'leri aynı sonuçları döndürürfrom ultralytics import YOLOhata vermez🤖 Generated with Claude Code