🔌 Happify-IOT — Voice Companion Firmware

Detect Early. Support Meaningfully. Grow for Life.

The companion-device firmware of Happify — an ESP32-C3 sketch that always listens for speech, records it, sends it to Happify Backend for AI processing, and speaks the reply back through a physical speaker, with a hard safety rule: distressing responses never autoplay.

🌐 Frontend · ☁️ Backend · 🧠 AI · 📱 Mobile · 🔌 IoT

📌 Overview

Happify-IOT is the physical presence of the Happify ecosystem — a small ESP32-C3 device that turns a microphone and a speaker into a voice companion. It has no screen, no app, and no direct line to the AI: it records what it hears, uploads the WAV to Happify Backend, and plays back whatever MP3 the backend sends in return.

Happify-IOT's role in the ecosystem: "A voice in the room — always ready to listen, never the one who decides what's safe to say back."

The firmware calls Happify Backend only, never Happify-AI directly. The backend authenticates the device, resolves which user it's paired to, forwards the audio through Happify-AI, and returns a structured response plus protected response audio.


MCU	ESP32-C3
Toolchain	Arduino IDE + ESP32 board support
Microphone	INMP441 (I2S digital MIC)
Amplifier	MAX98357A (I2S DAC + Class-D amp)
MP3 Decoding	`minimp3.h` (single-header, embedded locally)
Auth	Device-scoped runtime token (`Authorization: Device <token>`), issued after pairing
Default Voice Language	Indonesian (`id`)

🌐 Happify Ecosystem

Happify-IOT is paired to a single Happify user account through Happify-Mobile or Happify-Frontend, then talks exclusively to Happify-BE's device runtime endpoints — it never sees a Firebase token, only its own device credential.

Repository	Role	Link
🌐 Happify-FE	Web dashboard — mood, journal, community, care	Happiffy/Happify-FE
☁️ Happify-BE	Node.js API, device runtime auth, PostgreSQL, WebSocket hub	Happiffy/Happify-BE
🧠 Happify-AI	FastAPI service — speech-to-text, response generation, risk detection, TTS	Happiffy/Happify-AI
📱 Happify-Mobile	Flutter app — pairs and manages this device	Happiffy/Happify-Mobile
🔌 Happify-IOT	ESP32-C3 Voice Companion firmware (this repo)	Happiffy/Happify-IOT

System architecture:

graph TD
    DEV["🔌 Happify-IOT device (this repo)"] -->|"Authorization: Device <token>"| BE["☁️ Happify-Backend"]
    MB["📱 Happify-Mobile"] -->|pairing session| BE
    FE["🌐 Happify-Frontend"] -->|"user's Firebase token"| BE
    BE -->|voice turn| AI["🧠 Happify-AI"]
    AI -->|"transcript, response, risk, TTS"| BE
    BE -->|"protected MP3"| DEV
    BE --> PG[("🗄️ PostgreSQL")]

    style DEV fill:#FF9600,color:#fff,stroke:#D97A00
    style BE fill:#1CB0F6,color:#fff,stroke:#168CC7
    style AI fill:#CE82FF,color:#fff,stroke:#A760D2

Principles this firmware is built around:

Safety over autoplay — HIGH/CRISIS risk responses never play automatically; the device enters a silent safety state instead.
Least privilege — The device only ever holds a scoped runtime token, never a user's Firebase credential or database ID.
Consent-gated — Every voice turn requires an active Voice Processing consent on the paired account; the device checks this at boot via runtime context.
No silent logging — Transcripts, AI response text, tokens, and Wi-Fi passwords are never written to production serial logs.
Resilient by design — Wi-Fi reconnects automatically, upstream failures are retried with idempotency keys, and playback interruption is always possible.

🔧 Hardware

Part	Role
ESP32-C3	Main controller — Wi-Fi, HTTP client, I2S peripheral, state machine
INMP441	I2S digital MEMS microphone — captures 16kHz mono PCM
MAX98357A	I2S DAC + Class-D amplifier — drives the speaker
Speaker	Plays the decoded MP3 voice response

Wiring:

Component	Pin	ESP32-C3
INMP441 VDD	Power	3.3V
INMP441 GND	Ground	GND
INMP441 SD	Data out	GPIO 6
INMP441 WS	Word select	GPIO 4
INMP441 SCK	Bit clock	GPIO 5
INMP441 L/R	Channel select	GND
MAX98357A VIN	Power	5V
MAX98357A GND	Ground	GND
MAX98357A DIN	Data in	GPIO 7
MAX98357A BCLK	Bit clock	GPIO 5
MAX98357A LRC	Word select	GPIO 4
MAX98357A SD	Enable	3.3V

The mic and speaker share the WS/BCLK clock lines (GPIO 4 / GPIO 5) — the firmware switches the I2S peripheral between mic and speaker mode rather than running both simultaneously, which is also why recording pauses during playback.

✨ Features

🎙️ Always-Listening Voice Activity Detection (VAD) — Continuously samples the mic and starts recording only once a peak-amplitude threshold is crossed.
📼 WAV Recording — Captures 16kHz mono 16-bit PCM directly from the INMP441 with an in-memory WAV header.
🔐 Device-Authenticated Requests — Every backend call uses a device-scoped runtime token (Authorization: Device <token>), issued only after the device is paired to a user.
💾 Session Persistence — activeSessionId and sessionCounter persist across reboots in ESP32 NVS (non-volatile storage) via Preferences.
🇮🇩 Indonesian by Default — Voice requests default to id, configurable per runtime context.
🔊 Automatic Response Playback — Downloads the protected MP3 reply and decodes/plays it through I2S via minimp3.h, for LOW/MEDIUM risk turns only.
🛡️ Safety-Gated Autoplay — HIGH/CRISIS risk responses are intentionally not played automatically; the device instead enters STATE_SAFETY_ALERT.
📶 Wi-Fi Resilience — Detects disconnection in the main loop and reconnects automatically without a manual reset.
🔁 Idempotent Uploads — Each voice turn carries an Idempotency-Key header so a retried upload after a network blip doesn't create a duplicate turn.
🙈 No Sensitive Serial Logging — Transcript text, AI responses, device tokens, and Wi-Fi credentials are excluded from production serial output.

🛠️ Tech Stack

Layer	Technology	Purpose
MCU / SoC	ESP32-C3 (`ESP32C3 Dev Module` board)	Wi-Fi, HTTP client, I2S audio, main firmware loop
Framework	Arduino core for ESP32	`setup()`/`loop()` firmware structure
Audio I/O	ESP32 I2S driver	Reads mic PCM samples, writes decoded PCM to the amplifier
MP3 Decoding	`minimp3.h` (lieff/minimp3)	Single-header decoder for the backend's MP3 voice responses
JSON	ArduinoJson (Benoit Blanchon)	Parses runtime context and voice-turn API responses
Storage	ESP32 `Preferences` (NVS)	Persists `activeSessionId` and `sessionCounter` across reboots
Networking	ESP32 `WiFi` + `HTTPClient`	Wi-Fi connection and HTTPS calls to Happify-BE

📁 Repository Layout

Happify-IOT/
├── mic_test.ino     # ESP32-C3 Voice Companion firmware — I2S, VAD, state machine,
│                     # backend HTTP calls, MP3 playback (~700 lines)
├── minimp3.h         # MP3 decoder header, vendored locally (~1,900 lines)
└── README.md         # Setup, wiring, and API contract (this file)

device-secrets.h (Wi-Fi credentials, base URL, and device runtime token) is not committed — see Private Configuration below.

<<<<<<< HEAD

⚙️ How the Firmware Works

Device state machine (DeviceState enum in mic_test.ino):

stateDiagram-v2
    [*] --> STATE_BOOT
    STATE_BOOT --> STATE_WIFI_CONNECTING
    STATE_WIFI_CONNECTING --> STATE_CONTEXT_LOADING: Wi-Fi connected
    STATE_WIFI_CONNECTING --> STATE_WIFI_ERROR: connection failed
    STATE_CONTEXT_LOADING --> STATE_READY: GET /devices/runtime/context OK
    STATE_CONTEXT_LOADING --> STATE_TOKEN_EXPIRED: 401 DEVICE_UNAUTHENTICATED
    STATE_CONTEXT_LOADING --> STATE_CONSENT_REQUIRED: 403 AI_CONSENT_REQUIRED

    STATE_READY --> STATE_RECORDING: VAD peak > threshold
    STATE_RECORDING --> STATE_RECORDING: still speaking
    STATE_RECORDING --> STATE_READY: too short, discard
    STATE_RECORDING --> STATE_UPLOADING: silence timeout / buffer full

    STATE_UPLOADING --> STATE_WAITING_FOR_AI: POST /devices/runtime/voice/turns
    STATE_WAITING_FOR_AI --> STATE_PLAYING_RESPONSE: LOW / MEDIUM risk
    STATE_WAITING_FOR_AI --> STATE_SAFETY_ALERT: HIGH / CRISIS risk (no autoplay)
    STATE_PLAYING_RESPONSE --> STATE_READY: playback finished
    STATE_SAFETY_ALERT --> STATE_READY: after safety delay

    STATE_WIFI_ERROR --> STATE_WIFI_CONNECTING: retry
    STATE_TOKEN_EXPIRED --> [*]: needs new device credential
    STATE_CONSENT_REQUIRED --> [*]: needs consent enabled in-app

A full voice turn, end to end:

Copy device-secrets.example.h to device-secrets.h beside the sketch, then configure Wi-Fi, the issued device token, and the trusted CA PEM for api.happify.my.id.

cp device-secrets.example.h device-secrets.h

device-secrets.h is ignored by Git. Firmware uses WiFiClientSecure with the bundled Let’s Encrypt YE2 intermediate CA and will not use insecure TLS. Firmware synchronizes NTP time before TLS validation; do not use setInsecure().

The token is issued after the device is paired to a Happify user:

07d8d7b18fe82d4719dd42c6ce7629805f721f85

sequenceDiagram
    participant D as Happify-IOT device
    participant BE as Happify-BE
    participant AI as Happify-AI

    D->>D: VAD detects speech, records WAV (16kHz mono PCM)
    D->>D: Silence timeout reached -> finishRecording()
    D->>BE: POST /devices/runtime/voice/turns (WAV, Device token, Idempotency-Key)
    BE->>AI: Forward audio for STT + response + risk analysis
    AI-->>BE: transcript, responseText, riskLevel, audioUrl
    BE-->>D: 200 { turn: { responseText, riskLevel, audioUrl } }
    alt riskLevel is LOW or MEDIUM
        D->>BE: GET /devices/runtime/voice/turns/{turnId}/audio
        BE-->>D: Protected MP3 binary
        D->>D: Decode with minimp3 + play via I2S (MAX98357A)
    else riskLevel is HIGH or CRISIS
        D->>D: Enter STATE_SAFETY_ALERT — do NOT autoplay
    end
    D->>D: Return to STATE_READY, resume listening

Voice activity detection loop (inside loop()): while STATE_READY, the firmware samples the mic every iteration and compares peak amplitude against VAD_THRESHOLD (5000). Once speech is detected it moves to STATE_RECORDING, keeps buffering until SILENCE_TIMEOUT_MS (1500ms) of quiet, then either discards clips shorter than MIN_SPEECH_MS (300ms) or finalizes and uploads the recording (capped at MAX_RECORD_SECS = 3 seconds).

🔌 Backend Contract

The firmware speaks to Happify-BE's device runtime API only — see Happify-BE's /devices/runtime/* routes.

Load runtime context (on boot):

GET /devices/runtime/context
Authorization: Device <DEVICE_RUNTIME_TOKEN>

Tells the firmware whether Voice Processing consent is active on the paired account and returns the runtime voice endpoints.

Send a voice turn:

POST /devices/runtime/voice/turns
Authorization: Device <DEVICE_RUNTIME_TOKEN>
Content-Type: audio/wav
X-Voice-Language: id
X-Session-Id: nanda-session-1
Idempotency-Key: iot-voice-1

<raw WAV bytes>

{
  "status": "success",
  "data": {
    "turn": {
      "id": "voice-turn-id",
      "sessionId": "nanda-session-1",
      "responseText": "Terima kasih sudah berbagi.",
      "riskLevel": "LOW",
      "audioUrl": "/devices/runtime/voice/turns/voice-turn-id/audio"
    }
  }
}

Download response audio:

GET /devices/runtime/voice/turns/{turnId}/audio
Authorization: Device <DEVICE_RUNTIME_TOKEN>
Accept: audio/mpeg

Returns raw MP3 binary. The firmware always stops current playback before starting the next response.

Runtime error handling:

API result	Firmware action
`401 DEVICE_UNAUTHENTICATED`	Device token expired/revoked — a new credential must be issued after re-pairing
`403 AI_CONSENT_REQUIRED`	Prompt the user to enable Voice Processing consent in the Happify app
`400 INVALID_AUDIO`	Ask the user to record again
`413 AUDIO_TOO_LARGE`	Shorten the recording duration
`415 UNSUPPORTED_AUDIO_TYPE`	Ensure `Content-Type: audio/wav` is sent
`502` / `503` / `504`	Retry the same WAV with the same `Idempotency-Key` after backoff

🔐 Private Configuration

Wi-Fi credentials and the device runtime token must never be written into mic_test.ino. Create a device-secrets.h file beside the sketch:

<<<<<<< HEAD

#pragma once

#define WIFI_SSID "YOUR_WIFI_SSID"
#define WIFI_PASSWORD "YOUR_WIFI_PASSWORD"
#define HAPPIFY_API_BASE_URL "https://api.happify.my.id"
#define HAPPIFY_DEVICE_RUNTIME_TOKEN "ISSUED_DEVICE_RUNTIME_TOKEN"
=======
```text
mic_test.ino     ESP32-C3 Voice Companion firmware
minimp3.h                     MP3 decoder header
README.md                      Setup and API contract
device-secrets.example.h       Private configuration template
device-secrets.h               Ignored real Wi-Fi/token/CA configuration
>>>>>>> 07d8d7b18fe82d4719dd42c6ce7629805f721f85

The runtime token is issued after the device has been paired to a Happify user (via Happify-Mobile or Happify-Frontend):

POST /devices/{deviceId}/credentials
Authorization: Bearer <paired-user-firebase-id-token>

Use the returned data.credential.token in device-secrets.h only — never commit it, and never reuse a user's Firebase Bearer token for device runtime calls (those use Authorization: Device <token>).

🚀 Getting Started

Software Requirements

Arduino IDE with ESP32 board support installed
Board: ESP32C3 Dev Module
Library: ArduinoJson (by Benoit Blanchon)
minimp3.h from lieff/minimp3 — already vendored in this repo, keep it in the sketch directory

Recommended board setting:

USB CDC On Boot: Disabled

Setup

git clone https://github.com/Happiffy/Happify-IOT.git
cd Happify-IOT

Wire the INMP441 and MAX98357A per the Hardware table above.
Create device-secrets.h with your Wi-Fi credentials and a paired device runtime token (see Private Configuration).
Open mic_test.ino in the Arduino IDE, select board ESP32C3 Dev Module, and set USB CDC On Boot: Disabled.
Flash the sketch and open the Serial Monitor to watch boot, Wi-Fi, and VAD logs.

First Boot Checklist

Device connects to Wi-Fi (STATE_WIFI_CONNECTING → STATE_READY).
GET /devices/runtime/context succeeds — confirms the device token is valid and Voice Processing consent is enabled on the paired account.
Speak near the mic — Serial should print [VAD] Speech detected.
After you stop speaking, the device uploads and (for non-crisis responses) plays back the AI's reply.

🎓 Project Context

Built for

Garuda Hacks 7.0 — International Hackathon Competition

AI-Powered Mental Wellness Platform

Happify-IOT is the physical companion device of Happify, a privacy-aware digital wellbeing ecosystem built around early detection, meaningful support, and lifelong emotional growth:

Layer	Component	Role
🌐 Web	Happify-FE	Mood tracking, journaling, anonymous community, care workflows
☁️ Backend	Happify-BE	API, device runtime auth, PostgreSQL, WebSocket hub
🧠 AI	Happify-AI	Speech-to-text, response generation, risk detection, TTS
📱 Mobile	Happify-Mobile	Flutter app — pairs and manages this device
🔌 IoT	Happify-IOT (this repo)	ESP32-C3 Voice Companion firmware

👥 Team

Outstanding BINUSIAN Team — Garuda Hacks 7.0

Name	Role
Andrian Pratama	Full-stack Developer
Khalisa Amanda Sifa Ghaizani	IoT Engineer
Michella Arlene Wijaya Radika	Product Developer
Stanley Nathanael Wijaya	Product Developer

📄 License

This project is licensed under the MIT License — free to use, modify, and distribute.

MIT License

Copyright (c) 2026 Happify — Garuda Hacks 7.0

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software.

"Detect early. Support meaningfully. Grow for life."

Made with 🌱 for Garuda Hacks 7.0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

🔌 Happify-IOT — Voice Companion Firmware

Detect Early. Support Meaningfully. Grow for Life.

📌 Overview

🌐 Happify Ecosystem

🔧 Hardware

✨ Features

🛠️ Tech Stack

📁 Repository Layout

⚙️ How the Firmware Works

A full voice turn, end to end:

🔌 Backend Contract

🔐 Private Configuration

🚀 Getting Started

Software Requirements

Setup

First Boot Checklist

🎓 Project Context

Garuda Hacks 7.0 — International Hackathon Competition

👥 Team

📄 License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 7 Commits
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
device-secrets.example.h		device-secrets.example.h
mic_test.ino		mic_test.ino
minimp3.h		minimp3.h

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

🔌 Happify-IOT — Voice Companion Firmware

Detect Early. Support Meaningfully. Grow for Life.

📌 Overview

🌐 Happify Ecosystem

🔧 Hardware

✨ Features

🛠️ Tech Stack

📁 Repository Layout

⚙️ How the Firmware Works

A full voice turn, end to end:

🔌 Backend Contract

🔐 Private Configuration

🚀 Getting Started

Software Requirements

Setup

First Boot Checklist

🎓 Project Context

Garuda Hacks 7.0 — International Hackathon Competition

👥 Team

📄 License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages