Tone AI

About

ToneAI is a machine learning model framework capable of identifying human emotions in speech/noises. The sociological model used for mapping tone to emotion is Plutchik's Emotion–Intensity Graph. It is a well documented, widely agreed upon approach at understanding human emotions. Using this we aim to:

1. Research how ML can undersand audible human emotions.
2. Supply via other ML models or services a Plutchik human emotion for its use cases.
3. Research which deep learning models work best for mapping emotion-inetensities to Plutchik emotion.s

The underlying premise for this project is purely focused on research, development, and academia.

How to Setup

If you intend on creating your own ToneAI models, or replicating the training found in the research paper, below will explain all the steps needed to do so.

1. Setting up Dataset(s)

This repository will not supply any of the datasets used in training. Rather we will list the datasets used and where you might acquire them. The Tone AI authors do not claim ownership of these datasets, but encourage those who wish to replicate the tests to use them.

Datasets Used

• CREMA-D Dataset
• EmoGator Dataset
• JL Corpus Dataset
• RAVDESS Dataset
• SAVEE Dataset
• TESS Dataset
• MELD Dataset
• VIVAE Dataset

How to Add Datasets for Training

For existing datasets listed, the loader, parsing, and excel mapping functions are already present in ./datasets/scrpits. But to add actual files, create a directory called ./datasets/files, ensure all dataset files are in a directory of the following format ./datasets/files/<DATASET NAME>/*.wav. All dataset files should be .wav to match with torchaudio's best understanding. Since not all datasets will use the standard .wav format, you can use other software, like ffmpeg to easily convert them all. Once finished the dataset/files directory should look like this:

datasets/files/
├── CREMA-D
├── EmoGator
├── JL Corpus
├── MELD
├── RAVDESS
├── SAVEE
├── TESS
└── VIVAE

The only one that does not have all *.wav in its root dataset directory is MELD, due to its unique dataset configuration. MELD specifically should look like this:

MELD/
├── dev
├── test
└── train

2. Python Libraries

Python libraries used are as follows:

torch==2.10.0+cu130
torchaudio==2.10.0+cu130
torchmetrics==1.8.2
torchsummary==1.5.1
optuna==4.7.0
numpy==2.3.5
pandas==3.0.0
scikit-learn==1.8.0
seaborn==0.13.2
matplotlib==3.10.8
tqdm==4.67.2

3. Initializing Testing Setup

All files are in excel format. To load all dataset loader files, run one of the following for your desired intention:

• load.sh: For loading all the emotions under ./datasets/categrories/. Emotion range is the full primary Plutchik's Emotion–Intensity Graph range.
• load_dataset_permutations.py: For loading files of all dataset permutations, will be under ./datasets/categrories/dataset_combinations.
• load_emotion_permutations.py: For loading files of all subset permutations of primary Plutchik's Emotion–Intensity Graph emotions, will be under ./datasets/categrories/emotion_combinations/raw/.
• load_emotion_permutations_normalized.py: For loading files of all subset permutations of primary Plutchik's Emotion–Intensity Graph emotions in n1 or low normalized format, will be under ./datasets/categrories/emotion_combinations/norm/.

4. Running Tests

To run tests use the directory called ./labs/. Here, make a new folder of your lab test name, and create a file with the following template (.py or .ipynb):

import sys 
from pathlib import Path
ROOT_DIR = Path.cwd().resolve().parent.parent
sys.path.insert(0, str(ROOT_DIR))

import shutil
from itertools import combinations
from tqdm.notebook import tqdm

from toneai.pytorch.loader import load_data, print_dataset_stats
from toneai.pytorch.testing import ToneAIOptunaTest
from toneai.pytorch.models import <MODEL USED>

DATASET_PATH = ROOT_DIR / "datasets" / "categories" / <DATASET EXCEL FILE PATH USED>
pdata = load_data(DATASET_PATH)

OPTUNA_DB_NAME = "optuna_<emotion/intensity>"
EMOTION_STUDY_NAME = "optuna_<emotion/intensity>_<n1/n2/n3>_<all/neu/joy/.../ang/ant/...>"
DB_PATH = Path.cwd().resolve() / "stats"
OPTUNA_DB_USED_PATH = DB_PATH / OPTUNA_DB_NAME
emotest = ToneAIOptunaTest(OPTUNA_DB_USED_PATH, EMOTION_STUDY_NAME, pdata, <MODEL USED>)

emotest.set_epoch_n(<ADD EPOCH NUMBER>)
emotest.set_patience_n(<ADD PATIENCE LEVEL>)
emotest.start_emo_testing(trial_n = <ADD TRIAL NUMBER>)

Report statistic file can be used to determine the best trial from a model optuna databse file:

import sys
from pathlib import Path

ROOT_DIR = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(ROOT_DIR))

from toneai.pytorch.testing import OptunaStats, LabResult


OPTUNA_DB_DIR = Path(__file__).resolve().parent / "stats"
class_test_stats = OptunaStats(OPTUNA_DB_DIR , [
    LabResult(db_name="", 
              study_name="", 
              print_name=""),
    ... <Add more databases here!>
])

class_test_stats.print_all(title="", max=True) # For max scores
class_test_stats.print_all(title="",) # For ALL scores

5. Using a Model

ToneAI includes a real-time demo that runs a trained emotion and intensity model against live microphone input and displays predictions in a browser.

Demo Requirements

The demo requires the following additional libraries (install into your virtual environment):

pip install fastapi "uvicorn[standard]" sounddevice

Microphone access must also be granted to your terminal application via System Settings → Privacy & Security → Microphone.

Demo Structure

The demo is located in ./demo/ and is split into three layers:

demo/
├── input/            # Mic capture and audio preprocessing (MelSpectrogram)
├── model/            # Model wrappers and predictor
│   ├── cnn_emotion.py       # CNN emotion classifier (9 Plutchik classes)
│   ├── lstm_intensity.py    # Bidirectional LSTM intensity classifier (4 levels)
│   └── predictor.py         # Combines both models into one prediction
├── application/      # FastAPI server + browser frontend
│   └── static/       # HTML/CSS frontend (WebSocket-driven)
└── Model_files/      # Trained model weights (.pt)
    ├── emo-cnnnetwork-b50-no_neutral.pt
    └── inten-lstmmodel-b1.pt

Running the Demo

From the root of the repository, run:

.venv/bin/python -m demo

Then open http://127.0.0.1:8000 in a browser. The page will display:

• The predicted emotion (e.g. Joy, Anger, Sadness) with a Plutchik color background
• The predicted intensity level (Neutral, Low, Medium, High)
• A live microphone level bar showing audio input activity
• Per-class confidence scores for all 9 emotion classes

To run on a different port:

.venv/bin/python -m demo --port 8080

To run with a mock model (no trained weights required, useful for UI testing):

.venv/bin/python -m demo --mock

Swapping in a New Model

The demo is model-agnostic. To use a different trained model, subclass EmotionModel from demo/model/base.py, implement the forward() method, and pass an instance to the Predictor in demo/__main__.py:

from demo.model.base import EmotionModel
import torch

class MyModel(EmotionModel):
    def forward(self, waveform: torch.Tensor) -> torch.Tensor:
        # waveform shape: (1, n_mels=64, time_frames)
        ...

predictor = Predictor(model=MyModel(), processor=processor, intensity_model=intensity_model)

(Optional) Modifications

Config

Feel free to change any of the values here for modifying structure of all tests if that is the desired intention. This is located in the ./toneai_config.py file.

Adding Additional Datasets

Feel free to add more datasets or remove existing ones if that is your desired intention for expanding upon the Tone AI testing framework. You can do so by modifying the following files

• ./datsets/scripts/load.sh
• ./datsets/scripts/cli_interpreter.py
• ./datsets/scripts/dataset_manager.py As well as by adding your dataset directory to ./datasets/files/. Just be sure to make sure all files are .wav format!

Adding Models

To add additional models add them here ./tonai/pytorch/models.py.

Adding Dataset Class Types

For our project we are Mel Spectrograms as our input format. If you would like to change this, you can add your classes here ./tonai/pytorch/dataset.py.