Infant Monitoring System

A graduation project that monitors an infant's state from sound and vision, combining an embedded camera/microphone device, two on-device ML models, and a mobile app.

An Arduino Nicla Vision board captures camera snapshots and audio and serves them over WiFi (HTTP). A Flutter app fetches that data and runs two machine-learning models on the phone:

• an audio model that classifies a baby's cry (hungry, tired, belly pain, …), and
• a vision model that classifies the baby's visible state (crying, laughing, normal, sleeping).

Versions

The project is developed in three independent versions, each kept on its own git branch. Use the GitHub branch dropdown to browse them, or git checkout <branch> locally.

Version	Branch	Includes	Description
V1	v1-audio-only	nicla_vision	Audio model runs on the Nicla Vision board only — no mobile app, no vision model. Minimal embedded build.
V2	v2-local (= main)	nicla_vision · audio_model · vision_model · flutter_app	The full system running locally/on-device: device streams over WiFi, the Flutter app runs both models on the phone.
V3	v3-cloud	V2 + cloud/	Adds a cloud backend that runs larger AI models off-device.

git checkout v1-audio-only   # view/run Version 1
git checkout v2-local        # view/run Version 2 (same as main)
git checkout v3-cloud        # view/run Version 3
git checkout main            # back to the default branch

Tagged snapshots of each version are published under Releases (v1.0, v2.0, v3.0). Branches v1-audio-only and v3-cloud appear once those versions are built.

Repository structure

This is a monorepo containing the four components of the system:

Folder	Stack	What it is
audio_model/	Python · scikit-learn · librosa	Trains the two-stage baby-cry classifier and exports the model bundle the app runs.
vision_model/	Python · TensorFlow/Keras	Trains the baby-state image classifier and exports a TensorFlow Lite model for the app.
nicla_vision/	C++ · PlatformIO (Arduino)	Firmware for the Nicla Vision board; serves camera/audio over HTTP.
flutter_app/	Dart · Flutter	Mobile app: connects to the device, runs both models on-device, shows results.

How the pieces connect

 Nicla Vision (camera + mic)
        │  WiFi / HTTP  (/snapshot, /audio, /status, /events, /ping)
        ▼
   Flutter app  ──►  audio model  (cry classification)
                ──►  vision model (baby-state classification)

Components

audio_model — baby-cry classification

A two-stage LinearSVC pipeline over librosa-derived audio features:

• Stage A routes a clip into belly_pain, burping, laugh, silence, or others.
• Stage B splits others into hungry, cold_hot, discomfort, tired.

train_inapp_model.py retrains the model using a feature extractor that is reproducible in Dart (no librosa on the phone) and exports inapp_audio_model.json, which is bundled into the Flutter app.

vision_model — baby-state classification

A small CNN trained on labeled images (crying, laughing, normal, sleeping) at 64×64, simulating the Nicla camera pipeline. step2_train_model.py trains/exports and step3_deploy.py copies baby_classifier.tflite into flutter_app/assets/models/.

nicla_vision — device firmware

PlatformIO project for the Nicla Vision board. Connects to WiFi and exposes HTTP endpoints (/snapshot, /audio, /status, /events, /ping) that the app consumes. Build/upload with PlatformIO (upload.ps1 helper included for Windows).

flutter_app — mobile application

Cross-platform Flutter app. Discovers/connects to the Nicla device over HTTP, pulls snapshots and audio, and runs the audio and vision models on-device. Deployed model assets live in flutter_app/assets/models/.

Not included in this repo

To keep the repository small, large and regenerable artifacts are excluded via .gitignore and are not on GitHub:

• Raw datasets (audio_model/SourceData, MergedData*, baby_crying_sound, vision_model/camSet, VisionDataSet.zip, …)
• Virtual environments (audio_model/.venv, vision_model/baby_classifier_env)
• Generated training outputs (TwoStage_Aug_Output, vision_model/models, exported_models, large feature CSVs)
• Flutter build output (flutter_app/build/)

The trained model artifacts the app needs (baby_classifier.tflite, inapp_audio_model.json) are committed under flutter_app/assets/models/, so the app builds without retraining.

Getting started

Each component has its own dependencies. In brief:

# Audio model (Python 3.9)
cd audio_model
python -m venv .venv; .\.venv\Scripts\Activate.ps1
pip install -r requirements.txt

# Vision model (Python 3.12)
cd ..\vision_model
python -m venv baby_classifier_env; .\baby_classifier_env\Scripts\Activate.ps1
pip install -r requirements.txt

# Firmware (PlatformIO)
cd ..\nicla_vision
pio run                 # build;  see upload.ps1 to flash the board

# Mobile app (Flutter)
cd ..\flutter_app
flutter pub get
flutter run

Training scripts reference local dataset folders (excluded above). To reproduce training you'll need to supply the datasets in the expected folders, then run train_inapp_model.py (audio) or step2_train_model.py (vision).