Try it live · How it works · Results · Where it fails · Run it yourself

• Live app: https://d235zashbh751z.cloudfront.net (AWS Graviton4 with COOL). No webcam? Settings → Demo and judging → Replay a recorded session.
• Five-minute guide: docs/JUDGE_QUICKSTART.md
• Technical report: docs/TECHNICAL_REPORT.md, also as PDF
• Diagrams: docs/ARCHITECTURE.md · Evaluation: evaluation/README.md

Praxis helps people do the shoulder, arm and hand exercises a therapist prescribes. There are three: Arm out to the side, Waiter – cup (moving a cup across five printed dots) and the Nine-Hole Peg Test (a weekly timed test). OpenCV 5 checks each rep from the laptop camera. Hard-coded safety rules check every event first. Then a coach gives a short cue that stays inside the prescription.

Praxis is an adherence and coaching aid. It is not a medical device. The therapist prescribes, the coach suggests changes inside that prescription, and the safety rules in code always win. Video is processed in memory and never stored.

Why it matters

• More than 70 % of people who have a stroke have arm problems, and in one study of 102 stroke patients with a weak arm, only 11.6 % got full arm function back at six months.
• Recovery needs many repetitions. In 312 observed therapy sessions, arm practice happened in half of them, about 32 reps each time. The rest is meant to happen at home, where up to 70 % of people don't keep up their exercises.
• At home nobody checks how well a rep was done or how much to do on a bad day, and the therapist only hears about it at the next visit.

Sources: report §1.

How to use it

1. Print a sheet Lay it flat in front of the laptop and tilt the screen until the four corners show.	2. Say how you feel Your pain sets today's dose. 7 or more: rest is advised and your therapist is told.	3. Do the exercise OpenCV checks each rep. Anything it could not see is counted as unverified, never as a fail.

How it works

Sheet → millimetres Four printed markers give the homography from camera pixels to millimetres on the sheet. Nothing is counted until all four are seen. cv2.aruco · findHomography calibration.py	Pegs in holes The board is warped flat, holes found once, and each hole read as empty or occupied from peg colour and change from the empty board. A timer runs from first lift to ninth peg. warpPerspective · HoughCircles · inRange · findContours · matchTemplate board.py · peg_fsm.py	Cup on the dots The cup base is what changed on the empty sheet. A rep counts when it lands on the next dot; the reach time is measured per dot. warpPerspective · findContours · convexHull · moments cup.py
Arm angle The OpenCV Zoo pose model gives the shoulder angle (smoothed; the target is capped at 90°) and flags leaning or a hiked shoulder when the hips are in view. The mannequin only shows the move; it never scores. cv2.dnn pose.py · arm_fsm.py	Safety rules first, then the coach Pain, repeated leaning and repeated missed or unverified reps pause or stop the session and flag the therapist; losing you on camera pauses until you're back. All of this runs before any coach sees the event. Every cue traces back to an OpenCV event. stop_rules.py · coach/rules.py	Our agent (optional) Claude Sonnet 5.5 runs at checkpoints and calls OpenCV tools over MCP. It gets numbers and short text, never frames. Every dose change is a prompt the patient accepts. agent/tools.py · AGENT.md

_{The top three pictures are 3-D Blender renders run through the real OpenCV code (make_pipeline_figs.py), not a real camera.}

Prizes we're entering

Prize	What it asks for	Where we show it
Overall Awards	OpenCV 5 doing real image analysis, and a meaningful part running on AWS	The whole entry, live on AWS
Best Use of COOL · $1,000	COOL running the core workload on AWS Graviton	The session service on EC2 c8g.large (Graviton4) from the COOL AMI. Its /health says opencv_build.cool: true. Benchmark
Agentic Vision · $1,000	An agent using OpenCV tools, where what OpenCV sees changes its next action	Our agent's loop below, and a real trace in docs/AGENT.md

Architecture 1: the system on AWS with COOL

Architecture 2: our agent's loop

The detailed diagrams (OpenCV steps per exercise, one full agent run) are in docs/ARCHITECTURE.md.

Results

What	Result	Evidence
Arm angle on real video (REHAB24-6, a public lab dataset: 8 healthy adults, 88 reps) vs motion capture	angle error (MAE) 3.8°, rep F1 0.96	rehab24_v0.md
Same videos cropped to laptop framing (hips at or below the bottom edge)	rep F1 0.91–0.93 (34–35 of 35 reps found), angle error 5.0–7.5°. Before our 25 Oct fix: F1 0.28–0.76	rehab24_laptop_framing_v0.md
Our agent (Sonnet 5.5) on real video (REHAB24-6 through the product, 40 sessions, 12 Oct run)	Named a fault that motion capture confirmed, and gave a correction, in 41 of 44 set reviews (patient taps simulated). The rule coach alone, cueing during the rep, also addressed 22 of 22, but fewer of its claims held up (36 of 79, against 67 of 83 for our agent). 0 guardrail violations; first action 3.7 s median; about $0.05 a session	agent_video_v0.md
Our agent on 15 synthetic test scenarios	15 of 15 pass. One newer, harder case (mixed signals) fails in all 3 runs; it is still open	AGENT.md
Nine-Hole Peg Test and Waiter – cup, live on real webcams (our team, laptop camera, 13–24 Oct)	Found real bugs the renders missed, fixed and re-tested on renders, not yet re-tested live: plain wooden pegs and an ordinary cup not seen, a hand over the board counted as pegs (1 real peg read as 9), cup counts that arrived seconds late. Speed in 4 real cup sessions on our Railway test build (not the AWS server): camera to result in 132–147 ms (median), network included. No stopwatch-checked accuracy yet	report §8.1.1 · live_sessions_2026-10-24.md
Nine-Hole Peg Test and Waiter – cup, measured on 3-D Blender renders	Computer-made 3-D videos, not real people. Peg test: F1 0.97, total time off by 1.2 s on average; most seats (90 of 96) are counted but marked unverified (grasp not seen). Cup: F1 0.79, reach time off by 0.7 s	synthetic3d_v0.md
COOL vs stock OpenCV, same c8g.large (measured 11 Oct)	1.2–1.6× faster on colour threshold, contours, Hough and template matching; 10–16 % slower on JPEG decode, ArUco, warp and the hand model; end to end, a peg-test frame is 3 % faster and a cup frame 6 % slower	cool_vs_stock_c8g.md

Where it fails

• Measured real-video accuracy exists only for Arm out to the side, and only on healthy adults in a lab. We store no video, so our own live sessions have no labelled ground truth.
• The peg test and cup have no stopwatch-checked real number yet. On the 3-D renders the cup still misses about a third of reaches, almost all in a dim room and at a very flat 20° screen tilt; the failures are listed in synthetic3d_v0.md.
• With the hips out of view, Arm out to the side still counts reps, but it cannot check shoulder hiking or leaning, and the angle is less accurate.
• Angles are relative to the camera, not clinical goniometry. The printed peg sheet is a timing aid, not the clinical instrument.
• No usability (SUS) study and no patients. Our own testing was 17 saved live sessions by 3 of us (13–24 Oct; a count from the hosted store, 13 of them in the committed snapshots) plus 7 automated runs; the 21 Oct snapshot is in team_test_2026-10-21.md.
• No therapist has used or reviewed Praxis yet. The cup task comes from the published GRASP programme (UBC), the peg test is a standard clinical test, and Arm out to the side is a standard shoulder exercise. The prescription in the app is a sample we wrote, and the therapist view shows sample data.
• On AWS we use EC2 (COOL AMI) and CloudFront. S3 storage and a one-command CDK deploy are not built, and the hosted app has no sign-in.

Full list: report §10. Failure cases with evidence, one file each: evaluation/failure_cases/.

Run it yourself

cd services/session && uv sync && uv run python tools/fetch_models.py   # Python 3.12 + uv, OpenCV Zoo models
uv run uvicorn praxis_session.main:app --port 8000                       # → http://127.0.0.1:8000/health
# in a second terminal, from the repo root
cd web && pnpm install && pnpm dev                                        # → http://localhost:5173

Print the peg sheet or the cup sheet at 100 %. No webcam: Settings → Demo and judging → Replay a recorded session. Tests: uv run pytest -q (service) and pnpm test (web). Agent, evaluation, Docker and deploy commands: docs/DEVELOPING.md.

Repo map

services/session/   FastAPI + OpenCV 5 + safety rules + coach + agent tools (one process)
web/                React + TypeScript web app, Three.js mannequin
evaluation/         datasets, scripts that regenerate every number, results
benchmark/          COOL vs stock OpenCV
infra/ec2/          AWS deploy script and CloudWatch setup (CloudWatch not yet run on AWS)
docs/               report (+ PDF), quickstart, diagrams, agent, accessibility, licences, sheets, wireframes
traces/             sample coach traces: OpenCV event → decision → action

Team

Team Praxis: Rinky · Shruthi · Sathvik

License

Praxis code is MIT (LICENSE). Models, datasets, fonts and GRASP material keep their own terms: docs/LICENSES.md. GRASP ideas from UBC / Heart & Stroke are paraphrased with our own illustrations.