Environment Preparations

cd UltraEval-Audio
conda create -n aduioeval python=3.10 -y
conda activate aduioeval
pip install -r requirements.txt

Run Example

export PYTHONPATH=$PWD:$PYTHONPATH

# qwen2-audio-offline test
pip install -r requirements-offline-model.txt
CUDA_VISIBLE_DEVICES=0 python audio_evals/main.py --dataset sample --model qwen2-audio-chat



Result files after test:

```txt
- res
    |-- $model-name
        |-- $dataset
            |-- $time.jsonl
            |-- $time-overview.jsonl

Test Commands:

python audio_evals/main.py --dataset <dataset_name> --model <model_name> --config_path <config_path>
python audio_evals/main.py --dataset <dataset_name> --model <model_name> --config_path <config_path> --debug_mode <debug_mode> --limit <#sample_to_test>

Use detailed path for--config_path, e.g., audio_eval/external_code/scripts/DyCoke/qwen2_temporal_prune_50.yml, For quick test evaluation, tedlium-release1 for ASR/STT task, mmau-test-mini for audio understanding QA task.

Dataset/Benchmark Choices

--dataset to specify datasets, supported datasets:

• speech-chatbot-alpaca-eval
• llama-questions
• speech-web-questions
• speech-triviaqa
• tedlium-release1
• tedlium-release2
• tedlium-release3
• catdog
• audiocaps
• covost2-en-ar
• covost2-en-ca
• covost2-en-cy
• covost2-en-de
• covost2-en-et
• covost2-en-fa
• covost2-en-id
• covost2-en-ja
• covost2-en-lv
• covost2-en-mn
• covost2-en-sl
• covost2-en-sv
• covost2-en-ta
• covost2-en-tr
• covost2-en-zh
• covost2-zh-en
• covost2-it-en
• covost2-fr-en
• covost2-es-en
• covost2-de-en
• GTZAN
• TESS
• nsynth
• meld-emo
• meld-sentiment
• clotho-aqa
• ravdess-emo
• ravdess-gender
• COVID-recognizer
• respiratory-crackles
• respiratory-wheezes
• KeSpeech
• audio-MNIST
• librispeech-test-clean
• librispeech-dev-clean
• librispeech-test-other
• librispeech-dev-other
• mls_dutch
• mls_french
• mls_german
• mls_italian
• mls_polish
• mls_portuguese
• mls_spanish
• heartbeat_sound
• vocalsound
• fleurs-zh
• voxceleb1
• voxceleb2
• chord-recognition
• wavcaps-audioset
• wavcaps-freesound
• wavcaps-soundbible
• air-foundation
• air-chat
• desed
• peoples-speech
• WenetSpeech-test-meeting
• WenetSpeech-test-net
• gigaspeech
• aishell-1
• cv-15-en
• cv-15-zh
• cv-15-fr
• cv-15-yue

Datasets Descriptions

<dataset_name>	name	task	domain	metric
speech-chatbot-alpaca-eval	speech-chatbot-alpaca-eval	SpeechQA	speech2speech	GPT-score
llama-questions	llama-questions	SpeechQA	speech2speech	acc
speech-web-questions	speech-web-questions	SpeechQA	speech2speech	acc
speech-triviaqa	speech-triviaqa	SpeechQA	speech2speech	acc
tedlium-*	tedlium	ASR(Automatic Speech Recognition)	speech	wer
clotho-aqa	ClothoAQA	AQA(AudioQA)	sound	acc
catdog	catdog	AQA	sound	acc
mls-*	multilingual_librispeech	ASR	speech	wer
KeSpeech	KeSpeech	ASR	speech	cer
librispeech-*	librispeech	ASR	speech	wer
fleurs-*	FLEURS	ASR	speech	wer
aishell-1	AISHELL-1	ASR	speech	wer
WenetSpeech-*	WenetSpeech	ASR	speech	wer
covost2-*	covost2	STT(Speech Text Translation)	speech	BLEU
GTZAN	GTZAN	MQA(MusicQA)	music	acc
TESS	TESS	EMO(emotional recognition)	speech	acc
nsynth	nsynth	MQA	music	acc
meld-emo	meld	EMO	speech	acc
meld-sentiment	meld	SEN(sentiment recognition)	speech	acc
ravdess-emo	ravdess	EMO	speech	acc
ravdess-gender	ravdess	GEND(gender recognition)	speech	acc
COVID-recognizer	COVID	MedicineCls	medicine	acc
respiratory-*	respiratory	MedicineCls	medicine	acc
audio-MNIST	audio-MNIST	AQA	speech	acc
heartbeat_sound	heartbeat	MedicineCls	medicine	acc
vocalsound	vocalsound	MedicineCls	medicine	acc
voxceleb*	voxceleb	GEND	speech	acc
chord-recognition	chord	MQA	music	acc
wavcaps-*	wavcaps	AC(AudioCaption)	sound	acc
air-foundation	AIR-BENCH	AC,GEND,MQA,EMO	sound,music,speech	acc
air-chat	AIR-BENCH	AC,GEND,MQA,EMO	sound,music,speech	GPT4-score
desed	desed	AQA	sound	acc
peoples-speech	peoples-speech	ASR	speech	wer
gigaspeech	gigaspeech	ASR	speech	wer
cv-15-*	common voice 15	ASR	speech	wer

Build up your own dataset: docs/how add a dataset.md