Skip to content

LATENCY.md

Latest commit

 

History

History
119 lines (87 loc) · 5.12 KB

LATENCY.md

File metadata and controls

119 lines (87 loc) · 5.12 KB
Raw

Android synthesis latency

Measured on-device with SynthesisBenchmark (logcat tag SynthBench). Production stays on the ORT CPU execution provider until a candidate beats it on warm service-path TTFA without audio regressions.

What already shipped

Lever Status
Sentence / punctuation chunking + streaming to SynthesisCallback production
FIRST_CHUNK_LIMIT (96) after full text normalization production
Shared EngineRepository for activity + service production
One-shot language warm-up; ORT RunOptions.setTerminate on stop production
Fused edge-fade / clip / PCM16 with reused scratch buffer production
Preview AudioTrack streaming in MainActivity production
In-app synthesis benchmark (Quick / Compare backends) production
Experimental · NNAPI / XNNPACK / thread overrides opt-in in settings UI (falls back to CPU)
Chunking profiles (first / subsequent): Fast 64/160, Balanced 96/280, Continuous 280/560 opt-in in settings UI

Smart sentence/punctuation splitting stays the same; profiles only change the character budgets. Smaller first limits lower TTFA; larger subsequent limits keep longer phrases in one decode (fewer mid-sentence breaths). Total decode work for the same text is roughly unchanged.

Benchmark entry points

In the app

Open TensorSpeak → Run benchmark for on-device timing with your current engine settings (Quick) or a Compare backends sweep (CPU / XNNPACK / NNAPI). Results can be copied or shared; the same lines are logged under tag SynthBench.

The full variant × provider matrix and Nano CPU threading sweep remain instrumented tests (multi-hour). The benchmark screen shows the adb one-liner; logcat workflow below is unchanged.

adb (full suite)

cd android
./gradlew :app:installDebug :app:installDebugAndroidTest

# Providers (CPU / XNNPACK / NNAPI) + stage timings + service-path TTFA
adb shell am instrument -w \
  -e class com.github.aljge.tensorspeak.SynthesisBenchmark#benchmarkVariantsAndProviders \
  com.github.aljge.tensorspeak.test/androidx.test.runner.AndroidJUnitRunner

# FP32 assets vs files/experimental-ort/<variant>/ (INT8 / offline ORT candidates)
adb shell am instrument -w \
  -e class com.github.aljge.tensorspeak.SynthesisBenchmark#benchmarkExperimentalGraphs \
  com.github.aljge.tensorspeak.test/androidx.test.runner.AndroidJUnitRunner

# Intra-op thread / spin / global-pool sweep (Nano)
adb shell am instrument -w \
  -e class com.github.aljge.tensorspeak.SynthesisBenchmark#benchmarkCpuThreading \
  com.github.aljge.tensorspeak.test/androidx.test.runner.AndroidJUnitRunner

Look for stages … lines (normalize / phoneme / duration / decode / post), svc50 (PCM path), dNative, and thermal=.

Optional ORT chrome-trace: construct with RuntimeConfig(enableProfiling = true).

Experimental track (not in the APK)

Offline optimized ONNX / ORT format

python scripts/optimize_onnx_assets.py --model all
adb push out/experimental-ort/nano \
  /sdcard/Android/data/com.github.aljge.tensorspeak/files/experimental-ort/nano

benchmarkVariantsAndProviders probes experimental-ort/<variant>/ and logs load/TTFA. Gate: colder session construction and no worse warm decode vs online ALL_OPT.

NNAPI

Enabled as OnnxTts.Provider.NNAPI in the provider benchmark. Expect fragmentation on 1-D conv / ConvTranspose; keep only if partitions are large and TTFA wins.

Selective INT8 decode

python scripts/quantize_decode_experiment.py --model all
# optional quality/speed presets: --profile conv_only|per_tensor|uint8_act|reduce_range
# optional A/B WAVs under out/experimental-int8/<variant>/listen/
python scripts/quantize_decode_experiment.py --model nano --profile conv_only --emit-wavs

# On WSL2 with a USB device owned by Windows adb, use Windows adb.exe for push/install.
adb push out/experimental-int8/nano/. \
  /sdcard/Android/data/com.github.aljge.tensorspeak/files/experimental-ort/nano/
# Shell-created dirs need world traverse so the app uid can read them:
adb shell chmod -R a+rx /sdcard/Android/data/com.github.aljge.tensorspeak/files/experimental-ort

Writes decode.onnx (static QDQ INT8) plus float duration.onnx and REPORT.json under out/experimental-int8/<variant>/ (not decode.int8.onnx). The on-device probe loads those exact names from files/experimental-ort/<variant>/.

Focused instrumented compare (FP32 assets vs pushed graphs):

adb shell am instrument -w \
  -e class com.github.aljge.tensorspeak.SynthesisBenchmark#benchmarkExperimentalGraphs \
  com.github.aljge.tensorspeak.test/androidx.test.runner.AndroidJUnitRunner

Gate: device TTFA/decodeMs, duration stability, numerical sanity, blinded listening. Do not ship FP16 on CPU (ORT upcasts). QNN is Snapdragon-only and needs a custom ORT build.

Research notes and Pixel 9a numbers: docs/INT8_DECODE.md.

Acceptance

Primary: lower warm p50/p95 service-path TTFA for Micro and Nano, especially normalization-expanding lines, with frontend chunk/phoneme/token parity intact.

Secondary: no worse underruns, thermal throttling, peak memory, or total RTF.