OpenFace-Adaptive

Robust Multimodal Emotion Recognition via Reliability-Aware Gating and Heterogeneous Graph Attention

Accepted at the IEEE 5th World Conference on Applied Intelligence and Computing (AIC 2026)

Overview

OpenFace-Adaptive addresses the critical challenge of sensor failure in real-world multimodal emotion recognition. Unlike existing fusion methods that assume all modalities are equally reliable, our framework dynamically assesses modality quality and suppresses corrupted signals before fusion.

Key contributions:

1. Self-Supervised Reliability Gating — Per-modality trust scores computed via reconstruction autoencoders, enabling zero-shot corruption detection without explicit noise labels.
2. Heterogeneous Graph Attention Network (HGAT) — 2-layer message passing with 9 distinct edge-type projection matrices (3×3 source-target combinations) for modality-specific cross-modal fusion.
3. Multi-Objective Training — Focal Loss for class imbalance + reconstruction loss for gate supervision + InfoNCE contrastive loss for cross-modal alignment.
4. Edge-Deployable — 1.94M parameters, 1.9 MB after INT8 quantization, <10 ms CPU inference latency.

Architecture

Results

Evaluated on CMU-MOSEI (23,000+ segments, 7-class sentiment intensity) with 5-seed averaging:

Metric	Value
Acc-7 (7-class)	39.4% ± 0.7
Acc-2 (binary)	81.5% ± 0.6
Robustness gain (audio noise)	+2.0 pp vs. ungated
Model size (quantized)	1.9 MB

Getting Started

Prerequisites

• Python 3.9+
• PyTorch 2.0+
• CUDA (optional, for GPU training)

Installation

git clone https://github.com/ram-cs7/OpenFace-Adaptive.git
cd OpenFace-Adaptive

conda env create -f environment.yaml
conda activate openface_adaptive

Data Preparation

# Preprocess CMU-MOSEI raw data (requires cmu_mosei_raw/ directory)
python preprocess_mosei.py

Training

# Run the full 5-seed experimental suite with ablation study
python experiment_runner.py

This trains all configurations (full model, no-gate ablation, no-graph ablation, no-InfoNCE, no-Mixup) across 5 seeds and reports mean ± std for all metrics.

Evaluation and Visualization

# Generate 7-class confusion matrix
python generate_confusion_matrix.py

# Verify binary accuracy (positive vs. negative)
python verify_binary.py

# Visualize per-modality trust scores under corruption
python plot_trust.py

# Quantize model for edge deployment (dynamic INT8)
python quantize.py

Live Demo

# Real-time emotion recognition with webcam and microphone
# Requires: OpenFace 2.0, webcam, microphone
python live_demo.py

Project Structure

OpenFace-Adaptive/
├── model.py                     # HGAT + Reliability Gate architecture
├── data_loader.py               # CMU-MOSEI dataset loader
├── preprocess_mosei.py          # Raw MOSEI → processed pickle
├── experiment_runner.py         # 5-seed training + ablation pipeline
├── generate_confusion_matrix.py # 7-class confusion matrix
├── verify_binary.py             # Binary accuracy computation
├── plot_trust.py                # Trust score visualization
├── quantize.py                  # Dynamic INT8 quantization
├── live_demo.py                 # Real-time webcam inference
├── extract_modern_features.py   # Optional: RoBERTa/Wav2Vec2 features
└── environment.yaml             # Conda environment specification

Citation

If you use this code in your research, please cite:

@inproceedings{chennaka2026openface,
  title     = {OpenFace-Adaptive: Robust Multimodal Emotion Recognition via Reliability-Aware Gating and Heterogeneous Graph Attention},
  author    = {Chennaka, Sairam and Srinivasa Rao, P. C.},
  booktitle = {Proc. IEEE 5th World Conference on Applied Intelligence and Computing (AIC)},
  year      = {2026}
}

License

This project is licensed under the MIT License. See LICENSE for details.

Contact

• Sairam Chennaka — sairam.chennaka@gmail.com
• P. C. Srinivasa Rao — vasu.piit@gmail.com