feat(iot): v1.2.2 zero-trust usb serial fallback

.github/workflows/iot-ci.yml

@@ -58,6 +58,23 @@ jobs:
         /tmp/test_ringbuffer
         g++ -std=c++11 -I src test/test_detection.cpp -lm -o /tmp/test_detection
         /tmp/test_detection
+        g++ -std=c++11 -I src test/test_serial_fallback.cpp -o /tmp/test_serial_fallback
+        /tmp/test_serial_fallback
+
+    - name: Serial Bridge Parser Smoke Test
+      run: |
+        cd firmware/tools
+        python - <<'EOF'
+        import serial_bridge
+        line = '[QG:FB]{"value":250,"sensor_id":42,"device_timestamp":1720000000,"signature_hex":"ab"}'
+        p = serial_bridge.parse_frame(line)
+        assert p == {"value":250,"sensor_id":42,"device_timestamp":1720000000,"signature_hex":"ab"}, p
+        assert serial_bridge.parse_frame("[BOOT] some log line") is None
+        assert serial_bridge.parse_frame("[QG:FB]{not json") is None
+        assert serial_bridge.parse_frame("[QG:FB]{'bad':1}") is None
+        assert serial_bridge.parse_frame("[QG:FB]{}") is None
+        print("Serial bridge parser tests PASSED")
+        EOF
 
     - name: Build Host SIL CLI (same core as firmware)
       run: |

README.md

@@ -15,7 +15,7 @@
 ![Docker](https://img.shields.io/badge/Docker-Containerization-2496ED?style=for-the-badge&logo=docker&logoColor=white)
 ![Local AI](https://img.shields.io/badge/Local_AI-Ollama_%7C_Llama_3.2-000000?style=for-the-badge&logo=meta&logoColor=white)
 ![HiveMQ](https://img.shields.io/badge/HiveMQ-Cloud_MQTT-FFC107?style=for-the-badge&logo=mqtt&logoColor=black)
-![ngrok](https://img.shields.io/badge/ngrok-HTTPS_Tunnel-1F1E37?style=for-the-badge&logo=ngrok&logoColor=white)
+![Cloudflare](https://img.shields.io/badge/Cloudflare-HTTP2_Tunnel-F38020?style=for-the-badge&logo=cloudflare&logoColor=white)
 
 ![CI Backend](https://github.com/GiZano/QuakeGuard/actions/workflows/backend-ci.yml/badge.svg)
 ![CI Frontend](https://github.com/GiZano/QuakeGuard/actions/workflows/frontend-ci.yml/badge.svg)

ROADMAP.md

@@ -19,7 +19,7 @@ Edge seismic detection on ESP32 with local alerts.
 Data Plane migration to MQTT Cloud (HiveMQ), REST Control Plane (HTTPS) and TLS security.
 
 - Data Plane: ESP32 → HiveMQ Cloud (port 8883, TLS + username/password)
-- Control Plane: ngrok HTTPS tunnel for REST provisioning
+- Control Plane: HTTPS tunnel for REST provisioning — *Cloudflare quick tunnel* in dev (the ngrok free-tier edge is unusable by ESP32 IoT clients: it terminates ESP-IDF/mbedTLS TLS handshakes via JA3 fingerprinting)
 - `setInsecure()` for TLS handshake on ESP32
 - MQTT Bridge (Python Paho) with TLS
 - Working mobile dashboard with live data
@@ -59,9 +59,12 @@ Geographic zone division designed so the system is ready for the GNSS upgrade (v
 
 Signed telemetry over a serial link (USB CDC) when MQTT/WiFi connectivity is lost, so the host still receives data during offline simulations.
 
-- Second consumer of the existing event queue: signed telemetry over USB CDC when MQTT is unreachable
-- ECDSA-signed payloads, identical signing to the MQTT data plane
-- Host-side bridge collecting serial output and forwarding to the ingestion pipeline
+- ✅ **Second consumer of the existing event queue** — `networkTask` dispatches each event to the first available path: MQTT publish (unchanged data plane), USB serial fallback, or in-memory retention ring when no path exists
+- ✅ **ECDSA-signed payloads, identical signing to the MQTT data plane** — `SerialFallback.h` builds `[QG:FB]{...}` frames with the exact MQTT JSON; retained events are re-signed with the current wall time at drain so the backend ±300 s replay window accepts them
+- ✅ **USB-host-aware retention** — `Serial.isConnected()` (HWCDC) distinguishes a real host from a power-only USB charger: with no host, events are retained in the ring instead of being written to a dead port; drained FIFO when a path becomes available
+- ✅ **Offline wall clock** — software clock anchored at the first NTP sync (`epoch_at_sync` + `millis()`), so timestamps stay valid even after WiFi drops; no frames emitted before time is valid
+- ✅ **Host-side bridge collecting serial output and forwarding to the ingestion pipeline** — `firmware/tools/serial_bridge.py` reads `/dev/ttyACM0`, filters `[QG:FB]` frames, and POSTs them to `/readings/` with `X-API-Key` (same forwarding as the MQTT bridge)
+- ✅ **Automatic first-boot provisioning** — compile-time `SENSOR_ID` shortcut removed; the node POSTs `/devices/register` (public key + MAC + enrollment token + GNSS-ready coords) and the backend assigns the `sensor_id` and zone. Verified live on hardware. Backend accepts NULL geometry when a node has no GNSS fix yet
 
 ---
 

backend/src/main.py

@@ -363,7 +363,11 @@ def locate_zone(latitude: float, longitude: float, db: Session = Depends(get_db)
 def _create_sensor(db: Session, active: bool, zone_id: int | None, latitude: float | None, longitude: float | None, public_key_hex: str, mac_address: str | None = None) -> models.Sensor:
     """Create and persist a Sensor with spatial zone auto-assignment."""
     assigned_zone_id = zone_id or resolve_zone(db, latitude, longitude)
-    point = WKTElement(f"POINT({longitude} {latitude})", srid=4326)
+    # GNSS-ready: coordinates are optional at first boot. Store NULL geometry
+    # when unknown instead of building an invalid "POINT(None None)".
+    point = None
+    if latitude is not None and longitude is not None:
+        point = WKTElement(f"POINT({longitude} {latitude})", srid=4326)
     db_sensor = models.Sensor(
         active=active,
         zone_id=assigned_zone_id,

backend/tests/integration/test_api.py

@@ -166,3 +166,24 @@ def test_register_device_valid(self, client, override_db):
             assert resp.status_code == 201
             data = resp.json()
             assert "sensor_id" in data
+
+    def test_register_device_without_coordinates(self, client, override_db):
+        """GNSS-ready: a node with no fix yet must still register.
+        Regression: _create_sensor built 'POINT(None None)' for NULL coords,
+        which PostGIS rejects with a 500 on the INSERT."""
+        with patch("src.main.ENROLLMENT_TOKEN", "valid_token"):
+            override_db.query.return_value.filter.return_value.first.return_value = None
+            resp = client.post(
+                "/devices/register",
+                json={
+                    "public_key_hex": "beef" * 32,
+                    "mac_address": "AA:BB:CC:DD:EE:01",
+                    "enrollment_token": "valid_token",
+                },
+            )
+            assert resp.status_code == 201
+            assert "sensor_id" in resp.json()
+            added = override_db.add.call_args[0][0]
+            assert added.location is None
+            assert added.latitude is None
+            assert added.longitude is None

docs/whitepaper/01-architecture.typ

@@ -15,7 +15,7 @@ The infrastructure is decoupled into three primary tiers:
 Following the v1.1.0 cloud migration, the architecture strictly separates the data and control pipelines:
 
 - *Data Plane (Telemetry):* Flows exclusively through a HiveMQ Cloud Serverless broker on port 8883[cite: 1]. Communication is fully authenticated and TLS-encrypted[cite: 1]. A Python-based MQTT bridge (`mqtt_subscriber.py`) subscribes to the `quakeguard/telemetry` topic and forwards payloads to the internal FastAPI ingestion pipeline via HTTP POST[cite: 1].
-- *Control Plane (Provisioning & Management):* Device onboarding, cryptographic handshakes, and REST retrieval operations are routed through an ngrok HTTPS tunnel, directly exposing the FastAPI endpoints (e.g., `/devices/register`)[cite: 1].
+- *Control Plane (Provisioning & Management):* Device onboarding, cryptographic handshakes, and REST retrieval operations are routed through an HTTPS tunnel to the FastAPI endpoints (e.g., `/devices/register`)[cite: 1]. In development the tunnel is a *Cloudflare quick tunnel* (`cloudflared tunnel --url http://localhost:8000`); production should use a real HTTPS domain. The ngrok free-tier edge is not used because its bot-protection terminates ESP-IDF (mbedTLS) TLS handshakes via JA3 fingerprinting *before* any HTTP header can be read, so IoT clients never reach the backend.
 
 == Key Design Principles
 

docs/whitepaper/07-deployment.typ

@@ -53,7 +53,7 @@ The firmware strictly requires compile-time secret injection to operate.
   ```bash
   cp esp32_config.env.example esp32_config.env
   ```
-+ Edit `esp32_config.env` to include your local Wi-Fi credentials, the ngrok tunnel URL (or local IP) for the `SERVER_HOST`, and the `ENROLLMENT_TOKEN`.
++ Edit `esp32_config.env` to include your local Wi-Fi credentials, the HTTPS tunnel URL (a *Cloudflare quick tunnel*, `https://<random>.trycloudflare.com`, or a real domain) for the `SERVER_HOST`, and the `ENROLLMENT_TOKEN`. Note: an ngrok *free-tier* tunnel does not work for the node — its edge terminates ESP-IDF (mbedTLS) TLS handshakes via JA3 fingerprinting before any HTTP header can be read.
 + Connect the ESP32-C3 via USB and trigger the PlatformIO upload sequence.
 + Open the Serial Monitor at `115200` baud. On its first boot, the device will generate its ECDSA keys, connect to the network, and automatically register with the backend.
 

firmware/README.md

@@ -35,6 +35,23 @@ Due to the specific layout of the ESP32-C3 SuperMini, the I2C bus is forced via
 * **Integrity:** Every payload is hashed (SHA-256) and signed. The server can verify the origin using the device's Public Key.
 * **Replay Protection:** Timestamps are synchronized via NTP (`pool.ntp.org`) to prevent replay attacks.
 
+### USB Serial Fallback (v1.2.2)
+When the MQTT data plane is unreachable, the node re-certifies each event and emits it over the **USB CDC** port as a machine-readable frame so a co-located host still receives data during offline simulations:
+
+```
+[QG:FB]{"value":250,"sensor_id":42,"device_timestamp":1720000000,"signature_hex":"..."}
+```
+
+* **Identical signing** to the MQTT data plane — the backend ECDSA + replay-window checks apply unchanged.
+* **USB-host aware:** frames are only written while a real USB host is attached (`Serial.isConnected()`, HWCDC). Plugged into a power-only charger, events are **retained in an in-memory ring** (last 100) instead of being sent to a dead port, and are drained FIFO when a path returns.
+* **Offline wall clock:** timestamps come from a software clock anchored at the first NTP sync, so they stay valid after WiFi drops; retained events are re-signed with the current time at drain.
+* **Host bridge:** `tools/serial_bridge.py` reads the CDC device and forwards each frame to the ingestion pipeline:
+  ```bash
+  pip install pyserial requests
+  python tools/serial_bridge.py --port /dev/ttyACM0 --api-key "$IOT_API_KEY"
+  ```
+* **Toggle:** set `SERIAL_FALLBACK_ENABLED=0` in `esp32_config.env` for MQTT-only behaviour.
+
 ## 4. Configuration
 
 Before compiling, ensure the network and server credentials in `src/main.cpp` are updated:
@@ -49,6 +66,15 @@ Before compiling, ensure the network and server credentials in `src/main.cpp` ar
 #endif
 ```
 
+### HTTPS Tunnel Compatibility (IoT TLS clients)
+The device registers over plain HTTPS using the ESP-IDF **mbedTLS** stack. The ngrok **free-tier** edge terminates TLS handshakes from such clients via **JA3 fingerprinting** (bot-protection) *before* any HTTP header — including `ngrok-skip-browser-warning` — can be read, so the request never reaches the backend (`HTTP Code: -1`, `SSL - The connection indicated an EOF`). For the dev tunnel we use a **Cloudflare quick tunnel** (`cloudflared tunnel --url http://localhost:8000`), whose edge does not fingerprint IoT TLS clients, or a real HTTPS domain in production. Set `SERVER_HOST` (no scheme) and `SERVER_PROTOCOL` accordingly.
+
+> **Dev tunnel lifecycle (ephemeral).** The quick tunnel is account-less and has no uptime guarantee: it dies with the `cloudflared` process and the URL changes at **every restart**. The node only calls the HTTP endpoint at **first boot** (provisioning); after that it publishes over MQTT. If you erase the node's NVS (re-provisioning) or build fresh, restart the tunnel and update `SERVER_HOST` (firmware) and `EXPO_PUBLIC_API_BASE_URL` (mobile) with the new URL:
+> ```bash
+> ~/bin/cloudflared tunnel --url http://localhost:8000 --no-autoupdate --protocol http2
+> ```
+> `--protocol http2` is required on networks that block the default QUIC edge connection. For a stable URL, use a named Cloudflare tunnel with your own domain.
+
 ### Enabling the Optional GNSS Module (Experimental)
 The firmware ships with a **GNSS heartbeats module**, disabled by default. When enabled, the device reads a connected UART GNSS receiver, computes a geohash from the current fix, and includes it in every heartbeat sent to the server (used by the geo-spatial zone-alerting feature on the backend).
 
@@ -62,31 +88,30 @@ The firmware ships with a **GNSS heartbeats module**, disabled by default. When
 ### Step 1: Upload Firmware
 Connect the ESP32-C3 via USB and upload the firmware using PlatformIO or Arduino IDE.
 
-### Step 2: Key Extraction (Crucial)
-On the **first boot**, the device will generate a new cryptographic key pair. You must capture the **Public Key** from the Serial Monitor to register the device on the server.
+### Step 2: Automatic Registration (no manual step)
+On the **first boot** the device performs the automated handshake:
+
+1. Generates a fresh **ECDSA key pair** and seals the private key in NVS.
+2. Opens the WiFi captive portal (`QuakeGuard-Setup`) so you can connect it to your network.
+3. POSTs `/devices/register` with its `public_key_hex`, `mac_address`, `enrollment_token` and (GNSS-ready) coordinates.
+4. Receives its `sensor_id` back from the backend and persists it in NVS.
 
-1.  Open the Serial Monitor (Baud Rate: **115200**).
-2.  Reset the board.
-3.  Look for the security header:
+No per-device configuration is needed for distribution — the backend assigns the ID
+and the zone (via PostGIS) at registration time. The serial output shows:
 
 ```text
-[BOOT] QuakeGuard Security System First...
-[SEC] Generating New ECDSA Key Pair...
-[SEC] Keys Generated and Saved to NVS.
-[SEC] DEVICE PUBLIC KEY (HEX): 04a3b2c1... <COPY THIS STRING>
+[PROV] SUCCESS! Assigned Sensor ID: 7
+[PROV] Public key: 3059301306072a8648ce3d0201...
 ```
 
-4.  **Copy the HEX string.** You have a 10-second window before the sensor initialization begins.
-5.  Register this key in your backend database associated with `SENSOR_ID 101`.
-
-**Note:** If the server does not have this key, it will reject data with `403 Forbidden`.
-
 ## 6. LED / Serial Status Codes
 
 * `[SYS] Sensor OK`: Hardware initialization successful.
 * `[SENSOR] Stabilizing...`: Calibrating the accelerometer baseline (do not move the device).
 * `[SENSOR] EARTHQUAKE DETECTED!`: The STA/LTA ratio exceeded **1.8** and intensity exceeded **0.04G**.
 * `[NET] Transmission Successful`: JSON payload accepted by the server.
+* `[NET] MQTT Publish OK.` / `[NET] Serial Fallback Publish OK.`: event dispatched over the active path.
+* `[NET] No delivery path: event retained in ring.`: MQTT down and no USB host — the event is buffered for later drain.
 
 ## 7. Troubleshooting