Arduino Physical AI Challenge India 2026 · Category: Gaming, Robotics & Interactive AI Built by Aman Maheshwari · Upside Down Labs · New Delhi, India
It transforms muscle activity and hand motion into game controls using the Arduino UNO Q and the NPG Armband (built on Upside Down Labs' NPG Lite Armband device). EMG signals are combined with accelerometer data to recognize both static hand gestures and dynamic motion gestures, enabling more accurate and natural interaction. An AI-based gesture recognition model classifies user actions in real time to control a game, providing an intuitive, touch-free gaming experience.
This README assumes you've never done anything like this before. Every step is spelled out - if you get stuck, re-read the relevant section slowly rather than skipping ahead.
- What you need
- Flash the NPG Armband's firmware (do this first)
- Set up the software (one-time)
- Run it
- Using the app - every tab and field explained
- About the game (NEUROSTRIKE)
- Playing on a separate laptop (or any other computer) over Wi-Fi
- Running individual scripts instead of the GUI (advanced, optional)
- Troubleshooting
- Repository structure
- An Arduino UNO Q, set up with its Debian Linux desktop, connected to a monitor with a mouse and keyboard (or accessed however you normally use it).
- Two NPG Armbands (required) — Upside Down Labs' NPG Lite Armband device, worn on your forearm, charged and within Bluetooth range. This project needs both — one for muscle (EMG) gestures and one for motion — not one or the other.
- (Optional) An Edge Impulse account - free, sign up at edgeimpulse.com if you don't have one. Only needed if you want to record your own gestures and train your own model — skip this if you're starting with the pretrained model below.
- (Optional) A second computer (Windows, macOS, or Linux) if you want gestures to control a game on a different machine over Wi-Fi instead of playing directly on the UNO Q.
Already have a trained model? This repo includes a ready-to-use pretrained model in the model/ folder, so you can connect your armbands and start playing straight away without recording anything or training your own model first — just point Settings (Tab 5) at the .eim file inside model/. You can always record and train your own gestures later (Tabs 3 and 4) and swap it in whenever you want.
You do not need to know how to code. Every command below is copy-paste-able exactly as written.
Before anything else, both NPG Armbands need their firmware loaded - this is a one-time step per device, done from any computer with a Chrome or Edge browser (no software install needed for this part).
- Watch this short video walkthrough first: youtu.be/sKYY074Yg3A - it shows exactly what to click.
- Open the NPG Lite Web Flasher in Chrome or Edge: upsidedownlabs.github.io/NPG-Lite-Flasher-Web
- Connect your NPG Armband to your computer with a USB cable.
- On the web flasher page, choose the firmware file
firmware/npg-lis3dh_ino.binfrom this repository, and follow the on-screen steps to flash it. - Once flashing finishes, unplug the USB cable - the armband is now ready to connect over Bluetooth.
Repeat this for both of your two armbands, one at a time — each one needs the firmware flashed separately.
Open a terminal on the UNO Q's desktop (right-click the desktop or find "Terminal" in the applications menu) and run:
git clone <this-repo-url>
cd <this-repo-folder>
chmod +x run_linux.sh
./run_linux.shThat's it - run_linux.sh automatically installs Python, the GUI toolkit, and everything else this project needs, then starts the app for you. The first run takes a few minutes (downloading packages); every run after that is fast.
If something in that script fails partway through (rare, but possible depending on your exact UNO Q image), see Troubleshooting below, or set it up by hand:
sudo apt update
sudo apt install -y python3 python3-pip python3-tk python3-venv
python3 -m venv env
source env/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install --break-system-packages -r requirements.txt
python3 gui_app.pyDouble-click run_windows.bat. It finds or installs Python automatically, installs everything needed, and starts the app.
python3 -m venv env
source env/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt
python3 gui_app.py(On Linux, also run sudo apt install python3-tk - or your distro's equivalent - first if it's not already installed.)
Every time after the first setup, you only need two commands:
cd <this-repo-folder>
source env/bin/activate # Windows: env\Scripts\activate
python3 gui_app.py # Windows: python gui_app.pyThis one app is everything you need. You never have to run any of the other .py files in this repo directly - the GUI does it all: connecting to your armband, calibrating, recording gestures, training your model, and playing. (If you're curious or want more control, Section 8 shows how to run pieces individually - but you don't need to for normal use.)
A window titled "Gesture Control Lab" opens, with a sidebar of six numbered steps. Read the next section before clicking anything.
The sidebar has 6 tabs, numbered in the order you'll normally use them. Every tab in the real app also has small (i) icons you can hover over for a reminder of exactly this same information - this section just has it all in one place.
A pale yellow banner appears on some tabs telling you when you can skip that whole tab - read those, they'll save you time.
Do this first, every time you open the app. Every other tab needs a connected armband to work.
| Field / Button | What it means |
|---|---|
| Scan for devices | Looks for nearby NPG Armbands that are turned on and in range. If nothing shows up, check the armband is powered on and its light is blinking. |
| Discovered devices list | Shows every NPG Armband found nearby. |
| Add to connect list -> | Click a device in the list above first, then click this to add it to your connect list on the right. You'll be asked how many channels it has and where it's worn (e.g. "forearm"). |
| Connect list table | The order here matters if you have more than one armband - the first one becomes "dev1", the second "dev2". Use Move up/Move down to change which is which. The Status column shows live connection state per device. |
| Remove / Move up / Move down | Edit your connect list before connecting. |
| Connect && Start Streaming | Connects to every armband in your connect list and starts streaming data. Wait for the status bar at the bottom to say "Connected" before moving on. |
| Disconnect | Cleanly disconnects all armbands. If one loses connection on its own (e.g. out of range), the app notices automatically and tells you - you don't need to click this yourself in that case. |
Skip this whole tab if you're not using arm-motion (tilt) gestures - just leave "Use accelerometer" unchecked in Settings (Tab 5).
| Field / Button | What it means |
|---|---|
| Accel device | If you have more than one armband connected, pick the one that's actually going to sense motion (has an accelerometer chip). |
| Movement that triggers accelA/accelB | Choose the direction you'll actually move your arm in (e.g. left/right or up/down). Only this one axis gets used - accelA fires past one threshold, accelB past the other, and "rest" is in between. |
| Start calibration | Walks you through 3 short holds: resting still, moving to accelA, and moving to accelB. You can click this again anytime to redo calibration - it always re-measures from scratch, even on an armband you already calibrated before. |
| Captured values table | Shows the raw sensor numbers recorded for rest/accelA/accelB, for reference. |
Skip this whole tab (and Tab 4) if you already have a trained model file (.eim) - just load it in Settings (Tab 5) and go straight to Classify (Tab 6).
This is where you teach the system your gestures by recording examples of yourself doing them.
| Field / Button | What it means |
|---|---|
| Subject name | Whoever's wearing the armband right now. Used to organize your recordings - use the same name every time for the same person. |
| Gesture name | The label the model will learn to predict. Pick from the list (fist, pinch, extension, flexion) or type your own - just spell it exactly the same way every time you record it. "rest" records one long relaxed hold; anything else records repeated go/stop reps. |
| Also upload each clean segment to Edge Impulse (live) | Edge Impulse is the free website that trains your gesture model in the cloud. Checking this sends recordings there automatically as you record. |
| Edge Impulse ingestion API key | A password-like code proving these recordings belong to your Edge Impulse project. Find it by logging into Edge Impulse Studio, opening your project, then Dashboard > Keys. Shown as dots since it's secret - never share it. |
| Test split (0-1) | What fraction of each upload goes into Edge Impulse's "test" bucket instead of "training". 0.2 (default) means roughly 1 in 5 recordings is held back to check how well the model does on data it's never seen. |
| Start Recording / Stop | Starts the recording sequence: a countdown, then a single beep = start the gesture, double beep = stop/relax (a big on-screen GO/STOP banner shows the same cue visually too, in case you can't hear the beep). |
Skip this whole tab if you already have a trained model file (.eim).
| Field / Button | What it means |
|---|---|
| Project ID | The number in your Edge Impulse project's URL or Dashboard. |
| Studio API key | A different key from the Record & Send one - this one can trigger training jobs. Also found in Dashboard > Keys. |
| Skip DSP feature regeneration | DSP = "digital signal processing", the step that turns raw sensor readings into the numbers the model learns from. Only skip this if you haven't changed your recordings since the last time you trained. |
| Also build on-device model after training | Automatically does the Download Model step below right after training finishes. |
| Training cycles / Learning rate | Advanced, optional. Leave both blank to use Edge Impulse's sensible defaults. |
| Trigger Retrain | Starts training in the cloud using everything you've recorded so far. Watch the Log box for progress - can take under a minute to several minutes. |
| Deploy type | What kind of file to download. Keep runner-linux-aarch64 for the UNO Q (that's its CPU type). |
| Engine | Which format the model's math runs in - tflite is the normal, safe default. |
| Model type | Leave as "(project default)" unless you specifically need int8 (smaller/faster) or float32 (slightly more accurate). |
| Save to folder | Where the downloaded .eim model file goes on this computer. Remember this - you'll pick it from here in Settings. |
| Download Model | Builds and downloads the project's current trained model as a ready-to-run .eim file. |
Always check this tab before going to Classify.
| Field / Button | What it means |
|---|---|
| Model file (.eim) | The trained gesture-recognition file. This repo ships with a ready-to-use pretrained model in the model/ folder — click Browse and pick that .eim file to get started immediately, or use one you got from Train & Deploy (Tab 4) after recording and training your own gestures. |
| Stride (ms) | How often (in milliseconds) the model re-checks the sensor and makes a new guess. Smaller = reacts faster but uses more CPU. 100–200ms is a good starting point. |
| Confidence threshold (0-1) | How sure the model must be before it "counts" a gesture. 0.99 = very strict (fewer mistakes, might miss some real gestures). Lower it if it feels too fussy; raise it if it guesses wrong too often. |
| Send to IP or .local name | Only needed if you want gestures sent over Wi-Fi to another computer instead of (or as well as) this one. Type that computer's IP address, or its network name ending in .local - it's looked up automatically. Click Find name? if you don't know that computer's name, and Test / Resolve to check it works before starting. |
| Saved nicknames | Save a short nickname (like "myLaptop") for a long device name you use often, so you can just type the short one next time. |
| Send port | The network "door number" gestures go out through - 5115 is a fine default. |
| Use accelerometer | Turns on arm-motion gestures alongside muscle gestures. Leave unchecked (and skip Tab 2) if your project only needs muscle gestures. |
| Send over network | Sends each detected gesture to the IP/port above - for controlling a game on a different computer. |
| Send as local keystroke on this system | Makes your gestures act like keyboard key-presses right here on this computer - e.g. a fist could press the space bar. Both this and "Send over network" can be on, off, or both at once independently. |
| Configure EMG gesture -> key mapping... | Pick which keyboard key each gesture (like "fist" or "pinch") should press. |
| Configure accelerometer -> key mapping... | Same, but for accelA/accelB (your calibrated tilt directions). |
The "play" tab.
| Field / Button | What it means |
|---|---|
| Start / Stop | Start turns on live gesture detection using the model and settings from Tab 5. Stop turns it off - your armband stays connected, so you can Start again anytime. |
| EMG GESTURE indicator | What the muscle-sensor model currently thinks you're doing. Shows "-" when nothing confident is detected. |
| ACCELEROMETER indicator | What direction the NPG Armband's motion sensor currently thinks your arm is moving. Stays "-" the whole time if "Use accelerometer" is unchecked in Settings - that's normal. |
| Log box | Shows what's happening under the hood, useful for troubleshooting. |
Typical first-time order: Tab 1 (Connect) -> Tab 2 (Calibrate, only if using motion) -> Tab 3 (Record & Send) -> Tab 4 (Train & Deploy) -> Tab 5 (Settings) -> Tab 6 (Classify) -> play. If you already have a trained model: Tab 1 -> Tab 5 -> Tab 6.
This repo includes a bundled demo game, NEUROSTRIKE - a small graveyard-defense shooter, built deliberately lightweight (plain 2D, no 3D engine, no GPU needed) so it runs smoothly on the UNO Q's own hardware, right alongside the gesture classification pipeline, on the same board, at the same time.
It's a completely ordinary keyboard-controlled game - it has no idea whether the arrow-key presses it receives came from a real keyboard, from gui_app.py's Classify tab, or from gesture_key_mapper_gui.py on another computer. That's the whole point: any keyboard-controlled game or app can be gesture-controlled this way, not just this one - NEUROSTRIKE is just the one that's included so you have something to try immediately.
Controls (map these to your gestures in Settings, Tab 5):
| Key | Action |
|---|---|
| Left / Right | Move |
| Up | Fire |
| Down | Heavy power shot (costs more ammo, more damage) |
| C | Cycle gun (Pistol / Shotgun / Rifle) |
| R | Reload |
| Esc | Quit |
To play it:
gui_app.py and neurostrike.py are two separate programs - you run the GUI in one terminal and the game in a second, separate terminal, at the same time, on the same computer. The game doesn't need to be started from inside the GUI; it just needs to be the window that's focused (on top) when you're making gestures, since that's what actually receives the keystrokes.
- Make sure
gui_app.pyis already running (however you normally start it -./run_linux.sh, orsource env/bin/activatethenpython3 gui_app.py), and you're connected, calibrated, and have your gestures mapped to keys in Settings (Tab 5) with "Send as local keystroke on this system" checked. - Open a new, separate terminal window - don't close or reuse the one running the GUI. On the UNO Q's desktop, this is usually right-click > "Open Terminal", or find "Terminal" in the applications menu.
- In that new terminal, go to the same repo folder and activate the same virtual environment you set up in Section 3 - this is required, since that's where
pygame(which the game needs) was installed:(Windows:cd <this-repo-folder> source env/bin/activate
cd <this-repo-folder>thenenv\Scripts\activate) - Now start the game itself:
(Windows:
python3 neurostrike.py
python neurostrike.py) - Click on the NEUROSTRIKE game window so it's focused, then go back to the GUI and hit Start in Classify (Tab 6). Your gestures now control the game.
You'll end up with two windows open side by side (or one on top of the other) - the GUI, and the game - both running from the same computer, same virtual environment, at the same time. That's expected.
- See the next section.
By default, everything runs and plays on the UNO Q itself. But you can just as easily have the UNO Q do the gesture sensing, while a different computer - a laptop, say - receives those gestures over Wi-Fi and turns them into keystrokes there, so you can play NEUROSTRIKE (or any other game) on that computer's screen instead.
Both computers must be connected to the same Wi-Fi network (or the same local network some other way). Gesture data travels as small network packets on your local network - it does not go over the internet, and it cannot reach a computer on a different network. If the UNO Q is on your home Wi-Fi and your laptop is on mobile data or a different Wi-Fi network, this will not work - connect them to the exact same network first.
You need to tell the UNO Q where to send gestures. Do this on the receiving laptop:
Windows:
- Open Command Prompt (search "cmd" in the Start menu).
- Type
ipconfigand press Enter. - Look for "IPv4 Address" under your active Wi-Fi adapter - that's a number like
192.168.1.42. - Alternatively, type
hostnameto get your computer's name, then use<that name>.local(e.g.DESKTOP-ABC123.local) - this works if your network supports it, and the app can auto-look this up for you (see "Test / Resolve" in Settings, Tab 5).
macOS:
- Go to System Settings > General > Sharing. Your computer's
.localname is shown right there (e.g.Johns-MacBook.local) - use this directly. - Or, for a plain IP address: open Terminal and type
ipconfig getifaddr en0(oren1if you're on a different network adapter).
Linux:
- Open a terminal and type
hostname -I- the first number shown is your IP address. - Or use your computer's name with
.localappended: typehostnameto see the name, then use<that name>.local.
In gui_app.py's Settings tab (Tab 5):
- Type the laptop's IP address or
.localname into "Send to IP or .local name". - Click "Test / Resolve" to confirm it can actually find your laptop before you rely on it. If you're not sure of the name, click "Find name?" for a walkthrough.
- (Optional) Click "Save current as nickname..." to save it under a short name (like
laptop) so you don't have to type the full address again next time. - Check "Send over network".
- Set your gesture-to-key mappings as normal - these are the keys that will be pressed on the receiving computer, not the UNO Q.
Everything needed is in the receiver/ folder of this repo:
cd receiver
python3 -m venv env # Windows: py -3 -m venv env
source env/bin/activate # Windows: env\Scripts\activate
pip install -r requirements-receiver.txt
python gesture_key_mapper_gui.pyThis opens a small window where you build the gesture-to-key mapping by physically pressing the key you want, instead of typing its name. It listens on port 5115 by default (matching the UNO Q's default Settings port), and includes a safety feature that automatically releases any key that gets stuck held down.
Useful options:
python gesture_key_mapper_gui.py --port 5115
python gesture_key_mapper_gui.py --config my_mapping.json
python gesture_key_mapper_gui.py --config my_mapping.json --no-inject # edit the mapping only, don't press keysWith both sides running (Classify started on the UNO Q, gesture_key_mapper_gui.py running on the laptop), open NEUROSTRIKE - or any other keyboard-controlled game - on the laptop, and your gestures will control it there.
You do not need this section for normal use - it's here for anyone who wants finer control, or to understand what the GUI is doing under the hood. Every one of these is already handled automatically by gui_app.py.
Run live classification from the command line, instead of the GUI's Classify tab:
python classify_local.py --model modelfile.eim
python classify_local.py --model modelfile.eim --stride_ms 200 --threshold 0.95
python classify_local.py --model modelfile.eim --send-to 192.168.1.42 # stream to a laptop instead
python classify_local.py --model modelfile.eim --send-to 192.168.1.42 --accel # also use arm-motion gesturesTrigger a retrain from the command line, instead of the GUI's Train & Deploy tab:
export EI_CLASS_KEY=ei_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
python train_model_trigger.py --project_id <your_project_id>
python train_model_trigger.py --project_id <your_project_id> --skip-features # data hasn't changed
python train_model_trigger.py --project_id <your_project_id> --deploy # also build a downloadable model afterDownload a trained model from the command line, instead of clicking "Download Model" in the GUI:
export EI_CLASS_KEY=ei_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
python download_model.py --project_id <your_project_id>
# add --type runner-linux-x86_64 etc. for a non-UNO-Q targetPlay the bundled game directly, without going through the GUI's Classify tab:
python neurostrike.py(It's just a normal keyboard-controlled game - it doesn't know or care whether the keys it receives came from gui_app.py, gesture_key_mapper_gui.py, or your actual keyboard.)
record_gesture.py is a library module imported by both gui_app.py and classify_local.py - it's not meant to be run directly.
- A chain of
ModuleNotFoundErrormessages, one after another (edge_impulse_linux, thennumpy, thenpyaudio, then an OpenCV message) -edge_impulse_linux's own code unconditionally imports audio/camera submodules this project never uses. A freshpip install -r requirements.txtinstalls the full chain up front, so this shouldn't happen if you followed Section 3 - if it does, run that command again. - No sound on the go/stop recording beep - the audio backend tries a couple of options and silently falls back to a muted terminal bell if none work. Try
pip install sounddevice(already inrequirements.txt) orsudo apt install alsa-utils. Either way, the Record & Send tab also shows a large on-screen visual GO/STOP cue, so recording still works even without sound. - One of two connected armbands drops mid-session - the app keeps the still-connected one usable instead of freezing everything. Go back to Connect and click "Connect && Start Streaming" again to reconnect just the dropped one.
pip: command not foundinside an activated virtual environment - usually meanspython3-venv/pipweren't available at the OS level when the environment was created. Runsudo apt install python3-pip python3-venv, then delete theenvfolder and recreate it (Section 3).- A gesture or a key seems "stuck" - click Stop in Classify, then Start again.
gesture_key_mapper_gui.py(receiver side) also auto-releases stuck keys after a few seconds by design.
.
├── README.md <- you are here
├── requirements.txt <- Python packages needed on the UNO Q
├── gui_app.py <- the app you actually run (see Section 4)
├── record_gesture.py <- armband BLE + recording logic (used by gui_app.py)
├── classify_local.py <- live on-device classification + output channels (used by gui_app.py; also runnable standalone - Section 8)
├── train_model_trigger.py <- Edge Impulse training trigger (used by gui_app.py; also runnable standalone - Section 8)
├── download_model.py <- downloads a trained model (used by gui_app.py, or standalone - Section 8)
├── neurostrike.py <- the bundled demo game
├── model/
│ └── *.eim <- pretrained model, ready to use - point Settings (Tab 5) at this to skip Record & Send / Train & Deploy entirely
├── run_linux.sh <- one-click setup + launch (Linux / UNO Q)
├── run_windows.bat <- one-click setup + launch (Windows)
├── firmware/
│ └── npg-lis3dh_ino.bin <- NPG Armband firmware - flash this first, see Section 2
└── receiver/
├── gesture_key_mapper_gui.py <- runs on a SEPARATE laptop, for network gesture mode (Section 7)
└── requirements-receiver.txt
Built for the Arduino Physical AI Challenge India 2026, organised by Robu.in in partnership with Arduino.