repo: Behind the Barks
Know how your dog feels, live, from any phone.
Why · Demo · Photos · New capability · How it works · Run it · Team
Dogs can't tell us how they feel, and we usually find out too late. They spend hours home alone. Worry shows up in their ears, tail, posture and whines, and nobody is there to see it. By the time we notice a chewed door or a skipped meal, the bad day already happened.
WagWatch puts an old phone next to the food bowl. It watches from a distance, with no collar or gadget on the dog. When the dog's mood really changes, the owner gets a short note saying what changed and why.
"Why no wearables? We should adapt to them, rather than they adapting to us!" our handwritten takeaway, photographed below
The problem: a dog's mood is written in its body and voice, but nobody is watching.
16-second loop from the live run. Click for the full 90-second video.
What just happened
- Input: an iPhone at the bowl opens a web link and streams camera and mic. No app to install.
- What the models did: they found the dog, tracked 12 body points and 46 face points, listened for barks and whines, then scored all 8 moods.
- Result: the dashboard went No dog in view → Excited → Happy as Bruno arrived and settled. It also queued an owner alert.
Finished build: phone at the bowl, dashboard live |
 Close-up, UI: Relaxed at 56%, alert toast |
 Close-up, model view: boxes, pose and live scores |
 Close-up: a second dog, another breed, same pipeline |
 Handwritten note: our takeaway |
 Behind the scenes: the first signal sketch |
 Behind the scenes: demo shot list |
More: screen recording and UI sketches
- Full 90-second screen recording:
assets/WagWatch_Showcase_90s.mp4 - All UI screens, designed before any code was written (desktop, mobile, camera states, alerts):
ui/screens/ - Planning sticky notes: why/how/what, trigger → computer → actor
Long-horizon agentic building: two people shipped a 54-commit, 381-test, multimodal system in two days. Claude Opus in Claude Code did the building, working from a shared spec.
Each person drove their own Claude session through a numbered plan: PROMPTS-DATA.md for vision and audio, PROMPTS-WEB.md for the backend and dashboard, 27 steps in total. CLAUDE.md holds the JSON contracts both sessions obeyed. That is why the two halves plugged together on the first evening, with no rewrite.
| What it took | How we know it held |
|---|---|
| Keeping one spec across 27 steps and two parallel sessions | Both sides built to backend/contracts.py. The generated TS types are checked by tests/web/test_ts_types.py |
| Wiring 4 pretrained models it had never seen together (YOLO, SuperAnimal pose, DogFLW face, YAMNet) | End to end in backend/pipeline.py and backend/vision/, covered by tests/data/ |
| Writing the tests alongside the code, not after | 381 tests across tests/data and tests/web, all passing |
Where the model's work shows most:
- Contract-first split:
CLAUDE.md› JSON contracts andbackend/contracts.py - The step-by-step prompts both of us ran:
PROMPTS-DATA.md,PROMPTS-WEB.md - Mood scoring from pose, face and sound:
backend/fusion/rules.py, weights inconfig.yaml
Note
An optional runtime LLM interpreter is wired up: backend/fusion/prompts.py#L13 and llm_interpreter.py#L56. It reads the frame and signals through Groq or OpenRouter. It was not on during the live demo: every mood you see came from the rules engine (the "Rules only" chip in the photos).
A phone opens our /camera page and streams video frames and audio to the backend. Pretrained models find the dog and read its body, face and sounds. A rules engine turns those signals into one of eight moods, which must hold for 3 seconds before it counts. The dashboard updates live, and the owner is notified on real changes only, with a 60-second cooldown per mood.
flowchart LR
P["📱 Phone at the bowl<br/>/camera page"] -- "JPEG + PCM over WebSocket" --> B["FastAPI backend<br/>/ingest"]
B --> Y["YOLO11<br/>find the dog"]
Y --> K["SuperAnimal-Quadruped<br/>12 body points"]
Y --> F["DogFLW<br/>46 face points"]
B --> A["YAMNet<br/>bark · whine · growl"]
K & F & A --> R["Rules engine<br/>scores 8 moods"]
R -. optional .-> L["LLM interpreter<br/>Groq / OpenRouter"]
R & L --> S["State machine<br/>3 s hold · cooldowns"]
S --> D["🖥️ Live dashboard"]
S --> T["🔔 Owner alert<br/>Telegram or in-app"]
| Layer | Tool | Why |
|---|---|---|
| Camera | Phone browser, getUserMedia |
Any phone works from a link, with no app |
| Detection | Ultralytics YOLO11 | Pretrained COCO "dog" class, fast on a laptop |
| Body pose | DeepLabCut SuperAnimal-Quadruped | Pretrained on many breeds; gives tail, back and legs |
| Face | DogFLW 46-point model | Ears, eyes and mouth, the main mood cues |
| Sound | YAMNet (AudioSet) | Tells barks, yips, whimpers and growls apart out of the box |
| Mood | Rules in config.yaml |
Readable, tunable, and works offline |
| Backend | Python 3.11, FastAPI, WebSocket | One process for ingest, fan-out and MJPEG video |
| Dashboard | Next.js 16, React, Tailwind v4 | Canvas overlay, timeline and alerts; works on mobile |
| Alerts | Telegram Bot API, or in-dashboard | No native app needed |
Works live today
- ✅ Any phone as the camera and mic, over an HTTPS tunnel
- ✅ Dog detection, 12-point pose and 46-point face tracking in real time (~6–8 fps)
- ✅ Bark, yip, whimper, growl and howl detection
- ✅ All 8 moods scored, with a persistence check and cooldowns
- ✅ Dashboard: live overlay, mood card, signals, session timeline, "treat dropped" button
- ✅ Owner alerts in the dashboard ("would send to owner")
- ✅ Offline demo mode that replays 7 pre-recorded clips with zero network
Mocked or not yet proven
⚠️ LLM interpreter: code and tests exist, but it was off in the live demo⚠️ Telegram alerts: adapter plusscripts/telegram_test.py; the demo ran in dashboard-only mode⚠️ Mood accuracy: on the 7 fallback clips, the dominant mood matched our hand label on 4 of 7 (manifest.json)⚠️ Fallback clips are licensed stock footage, not our own dog (SOURCES.md)
Fastest path: offline, no camera, no keys
git clone https://github.com/ComputerTech99/Behind_The_Barks && cd Behind_The_Barks
python3.11 -m venv .venv && source .venv/bin/activate
pip install -r requirements-web.txt
cd frontend && npm install && cd ..
# config.yaml: set web.pipeline: mock (the committed value, real, needs the ML stack)
make demo # terminal 1: DEMO_MODE=1 backend on :8000
make dev-frontend # terminal 2 → http://localhost:3000, then flip to Demo| Variable | What it does | Needed? |
|---|---|---|
LLM_PROVIDER |
groq or openrouter |
Optional |
LLM_API_KEY |
Key for that provider | Optional |
LLM_BASE_URL |
OpenAI-compatible endpoint | Optional |
LLM_MODEL |
Any model name from the provider | Optional |
LLM_VISION |
1 sends the frame, 0 sends text only |
Optional |
TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID |
Owner alerts on Telegram | Optional |
NOTIFY_MODE |
telegram or dashboard_only |
Optional |
DEMO_MODE |
1 replays cached clips offline |
Optional |
Copy .env.example to .env. Anything left blank degrades gracefully to rules-only moods and dashboard-only alerts.
Live phone camera with the real vision pipeline
pip install -r requirements-data.txt # heavy ML stack, 5–10 min
python scripts/check_env.py # torch/MPS, tensorflow, ultralytics import check
python scripts/fetch_face_model.py # downloads the DogFLW model once
# config.yaml: data.source.type: browser, web.pipeline: real
make dev-backend && make dev-frontend
cloudflared tunnel --url http://localhost:8000 # backend → https://X.trycloudflare.com
cloudflared tunnel --url http://localhost:3000 # frontend → https://Y.trycloudflare.comOn the phone, open https://Y.trycloudflare.com/camera?backend=https://X.trycloudflare.com. Allow camera and mic, then point it at the bowl. The full runbook and failure drills are in docs/DEMO_RUNBOOK.md.
Python 3.11 or 3.12 only (TensorFlow has no 3.13+ wheels yet). Don't leave web.pipeline: real as your default: DEMO_MODE does not override it.
Tests
pytest # 381 passed in ~86 s (needs both requirement sets installed)
make test-frontend # vitest + typecheck + lintCredits and licence
Pretrained work: Ultralytics YOLO, DeepLabCut SuperAnimal-Quadruped, hugocornellier/dog-face-landmarks (DogFLW, CC BY-NC 4.0, non-commercial), and Google YAMNet / AudioSet. Fallback clips come from Pexels, Pixabay, ESC-50 and AudioSet, with the source and licence of each file in data/fallback/SOURCES.md.
No licence has been chosen for this repo yet, so all rights are reserved.
Built at Claude Community Bangalore · Opus Build Day, Creature Commons, 26–27 September 2026.
 Ojash Gupta Data: vision, audio, rules, fallback clips |
Amit Web: backend, dashboard, phone camera, alerts |
| Criterion | Evidence | Proof |
|---|---|---|
| New Capability | Claude Opus built a 54-commit, 381-test multimodal system in 2 days, across two parallel sessions tied together by one spec | New capability |
| It Works | Ran live on a real dog from an iPhone; video, photos and an honest works/mocked list | See it work · Photos · Run it |
| Keep or Share | Any dog owner who leaves the house can reuse an old phone, with nothing on the dog | Why |
| Clarity of Demo | One-line problem, a 16 s GIF on the first screen, and "No dog → Excited → Happy" in 3 steps | See it work |