# What benchmarks are available for comparing different TTS engines?

Source: https://zilliz.com/ai-faq/what-benchmarks-are-available-for-comparing-different-tts-engines
Author: zilliz

To compare text-to-speech (TTS) engines, benchmarks typically fall into three categories: **objective metrics**, **subjective evaluations**, and **task-specific benchmarks**. These help assess quality, performance, and suitability for specific use cases. Below is a breakdown of common benchmarks and their applications.

**1. Objective Metrics** 
Objective benchmarks use quantitative measurements to evaluate technical aspects of TTS output. Examples include: 
- **Mel-Cepstral Distortion (MCD)**: Measures spectral accuracy by comparing synthesized speech to ground-truth recordings. Lower values indicate better quality. 
- **Real-Time Factor (RTF)**: Calculates processing speed (e.g., RTF of 0.5 means generating 1 second of audio takes 0.5 seconds). Critical for real-time applications. 
- **Word Error Rate (WER)**: Uses automatic speech recognition (ASR) to transcribe TTS output and checks if words match the input text. High WER suggests pronunciation issues. 
- **Latency**: Time taken to generate audio from text input, important for interactive applications. 

These metrics are reproducible and scalable but may not fully capture human-perceived quality. For example, a low MCD doesn’t guarantee natural-sounding prosody.

**2. Subjective Evaluations and Task-Specific Benchmarks** 
Human evaluations address nuances objective metrics miss. Common methods include: 
- **Mean Opinion Score (MOS)**: Listeners rate speech quality on a scale (e.g., 1–5) for naturalness, clarity, and intonation. 
- **Comparative MOS (CMOS)**: Listeners compare two TTS systems directly. 
Task-specific benchmarks focus on specialized use cases. For example: 
- **Blizzard Challenge**: Evaluates general-purpose TTS systems using standardized datasets. 
- **VCTK Corpus**: Tests multi-speaker synthesis with diverse accents. 
- **Emotional Speech Datasets**: Assesses models’ ability to convey emotions like happiness or anger. 

These benchmarks require careful design to avoid bias but provide insights into real-world usability.

**3. Practical Benchmarks for Developers** 
Developers often prioritize integration and efficiency. Key benchmarks include: 
- **Inference Speed**: How quickly the engine runs on target hardware (e.g., CPUs vs. GPUs). 
- **Memory Footprint**: RAM/VRAM usage, especially for edge devices. 
- **Framework Compatibility**: Ease of deployment in production environments (e.g., TensorFlow, PyTorch, ONNX). 
- **Customization Support**: Ability to fine-tune voices or adapt to new languages with minimal data. 

For example, a TTS engine might excel in MOS but fail on embedded devices due to high memory use. Developers should balance quality with operational constraints like latency or hardware limitations when choosing a system.