Leaderboard
Average = unweighted mean over the sets shown (Elm's convention). “Δ vs Whisper v3” = paired bootstrap difference in WER on shared utterances, Elm normalizer; negative is better than the baseline. Numeral probe: share of references containing a number where the hypothesis has the same digits. Entity probe: share of reference names/places/orgs found in the hypothesis. Every submitted system appears in every view; a row marked “common slice only” was evaluated on the ~1,000-utterance-per-set slice (the ZeroGPU submission path) and its numbers are shown in place of full-set numbers. Contamination markers: D the model card declares training on the same corpus as that test set (“in-domain”; excluded from the average by default); A the cell is far below what the system’s own level and the set’s difficulty predict (median-polish residual ≤ −0.30, robust z ≤ −2.5), the pattern a test-set leak produces; F the system’s hypotheses reproduce that corpus’s transcription conventions (hamza, ta-marbuta, punctuation, digits…) far more closely on that set than it tracks any other set’s conventions, the signature of training on its transcripts; G the system is far better on the test split than on the same corpus’s validation split (dev-vs-test gap ≤ −0.25 in log WER); P its advantage on that set collapses under word-preserving audio perturbation (speed ±6%, 20 dB pink noise, telephone band) much more than elsewhere; L for decoders we can score, the reference transcript is more probable than the model’s own output on an abnormal share of utterances. Hover a marker for the numbers; every signal is defined with its threshold in the methodology. Click a row for the model card and the per-utterance outputs.
Test sets
Production profile measured on one H20 (left) · declared from the model card (right)
Measured: fixed timing slice (~60 utterances, 3 to 15 s), warm, batch 1, concurrency 1, median of 3 passes. RTFx = audio seconds per wall second. Latency = seconds per file (offline), p50 / p95. VRAM = peak allocated. “shared GPU” means other jobs were on the GPU during the measurement. Declared: taken from the model card, not verified by us. * the card is silent, so the value is the default for the model's architecture (for example a Whisper fine-tune inherits Whisper's 30 s window and word timestamps). Streaming: native = trained and served for streaming; chunked = offline model usable with chunked pseudo-streaming; hover a value for the card's own wording. Sorted by RTFx.
By dialect WER, pooled across sets that carry dialect labels (SADA, Casablanca)
Submit a model
Any public Hugging Face model with a Transformers-native ASR architecture: Whisper, wav2vec2 and HuBERT CTC fine-tunes, Qwen3-ASR, Cohere Transcribe, Voxtral. Repositories that need trust_remote_code, NeMo checkpoints, and weights above 24 GB wait for a maintainer. The submission is evaluated on the common slice (about 1,000 utterances per set) on Hugging Face compute, usually within the hour, and the result is posted in its request thread.
The only permission requested is to open a discussion in the request queue on your behalf.
Queue every submission and its status
| Model | Status | Submitted | By | Note |
|---|
Method
Every system is run zero-shot, as released, with the decode settings recorded in its run.json. Hypotheses and references go through the same published normalizer. WER and CER are corpus-level. Every number carries a 95% utterance-level percentile bootstrap interval (2,000 resamples). Test sets are the six public ones used by Elm's Open Universal Arabic ASR Leaderboard, reconstructed utterance for utterance from Elm's manifests; audio stays on our machine, only outputs are published. The common slice is a fixed, seeded, dialect-stratified sample of about 1,000 utterances per set.
Normalizers. elm is a line-for-line port of Elm's, tested for string equality against the original, so that column is comparable to Elm's board. tolerant adds NFC, tatweel, alef/alef-maksura/ta-marbuta/hamza folding and a published spelling-variant table.
Probes. Numeral: references with digits or an unambiguous spoken number word, scored on digit-string equality with number words mapped on both sides. Entity: reference PERS/LOC/ORG spans from CAMeL-Lab's NER tagger (score ≥ 0.8, published side file), scored by containment after tolerant normalization.
Scope. Open-weight models and global vendors only. Vendors domiciled in the GCC appear only after written opt-in. This page reports measured values and makes no other claims. Full details, code, manifests and every per-utterance output: github.com/h9-tec/tahqiq and tahqiq-results.