ArTST: Arabic Text and Speech Transformer
Abstract: We present ArTST, a pre-trained Arabic text and speech transformer for supporting open-source speech technologies for the Arabic language. The model architecture follows the unified-modal framework, SpeechT5, that was recently released for English, and is focused on Modern Standard Arabic (MSA), with plans to extend the model for dialectal and code-switched Arabic in future editions. We pre-trained the model from scratch on MSA speech and text data, and fine-tuned it for the following tasks: Automatic Speech Recognition (ASR), Text-To-Speech synthesis (TTS), and spoken dialect identification. In our experiments comparing ArTST with SpeechT5, as well as with previously reported results in these tasks, ArTST performs on a par with or exceeding the current state-of-the-art in all three tasks. Moreover, we find that our pre-training is conducive for generalization, which is particularly evident in the low-resource TTS task. The pre-trained model as well as the fine-tuned ASR and TTS models are released for research use.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
ArTST: Arabic Text and Speech Transformer — A simple explanation
Overview
This paper introduces ArTST, a powerful computer model that understands and works with both Arabic speech (audio) and Arabic text. It’s designed especially for Modern Standard Arabic (MSA). The model can:
- turn speech into text (ASR: Automatic Speech Recognition),
- turn text into speech (TTS: Text-to-Speech),
- and recognize which Arabic dialect is being spoken (Dialect Identification).
The big idea is that focusing on one language (Arabic) and using one smart model for both text and speech can beat general, multilingual models at Arabic tasks.
What questions did the researchers ask?
The researchers wanted to know:
- Can we build one Arabic-focused model that handles both speech and text well?
- If we train it from scratch on Arabic, will it outperform large multilingual models (like Whisper and MMS) on Arabic tasks?
- Can this model learn to speak Arabic naturally even without adding diacritics (the small marks that show short vowels) to the input text?
- Will the model generalize (adapt) to different datasets and styles, including some dialects?
How did they do it? (Methods explained simply)
ArTST is built using a transformer, a kind of AI that is very good at spotting patterns and understanding context. Think of it as a very attentive reader and listener that can remember the big picture.
- One shared brain, two “ears and mouths”:
- The core of ArTST is a shared encoder–decoder “brain” that works for both speech and text.
- It has special “front ends” and “back ends” for speech and for text, so it can handle audio features and characters properly.
- Learning without answers first (pre-training):
- The model first learns by “self-study” on lots of Arabic speech and text without needing labels. This is like practicing puzzles:
- Masked speech prediction: hide chunks of the sound and ask the model to guess what’s missing. To help, they first group similar sounds into a set of “sound labels,” like clustering tones into 500 categories.
- Speech denoising: add noise or remove parts and train the model to rebuild clean speech features (like fixing a scratched audio clip).
- Text denoising: scramble or mask parts of text and train the model to reconstruct the original sentence (like fixing a sentence with missing words).
- Cross-modal linking: make speech and text share a common “codebook” (a small shared dictionary of patterns), so the model learns how sounds and letters relate.
- Then learning with answers (fine-tuning):
- After pre-training, the model is given specific tasks (ASR, TTS, dialect ID) with the correct answers, and it adjusts to do each task really well.
- Data used:
- About 1,000 hours of Arabic news/broadcast speech (MGB2) for training.
- Two clean Arabic TTS datasets (ASC and ClArTTS) for speaking naturally.
- A multi-dialect dataset (QASR) for testing how well it generalizes.
- Text was cleaned (e.g., removed diacritics) and audio standardized to 16 kHz.
What did they find? (Main results)
ArTST did very well—often better than other models—on three tasks.
- Automatic Speech Recognition (speech to text):
- On the standard MGB2 test, ArTST had about 13% word error rate (lower is better), and about 12.8% with a small LLM added. This matches or beats previous strong systems.
- It also beat big multilingual models like Whisper and MMS on all the Arabic test sets they tried, even though those models are larger.
- Text-To-Speech (text to speech):
- When trained on Arabic, ArTST produced clear, natural-sounding Arabic speech.
- It worked even without diacritics in the input text, which is usually very hard for Arabic. Listeners rated its speech quality highly (around 4.1–4.3 out of 5).
- Starting TTS training with extra speech from the ASR dataset (even though it’s not “studio clean”) improved generalization further.
- Dialect Identification:
- ArTST correctly identified Arabic dialects with about 94% accuracy on a standard benchmark, outperforming previous single-model systems and coming close to the best multi-system fusion.
Why this matters:
- It shows that a model trained specifically for Arabic can beat larger multilingual systems on Arabic tasks.
- It can synthesize speech without needing a separate diacritization tool, making it easier and more robust in real use.
Why is this important?
- Better tools for Arabic: ArTST is open-source and can help students, developers, and researchers build better Arabic speech apps: dictation, audiobooks, voice assistants, and more.
- Fewer add-ons: Because it doesn’t need diacritics to speak well, it simplifies pipelines and avoids errors from separate diacritization models.
- Future-proof design: One unified model for text and speech means it can be extended to more tasks (like speech-to-speech, or text generation) and to more Arabic varieties.
Limitations and what’s next
- The current version focuses on Modern Standard Arabic trained mostly on one big dataset (MGB2). Adding more diverse data could help.
- Dialects and code-switching (mixing Arabic with English or French) are planned for future versions.
- They used an English-trained helper model (for sound labels) in pre-training; an Arabic version might improve performance even more.
- They plan to explore the model’s “inner workings” to guide architectural improvements.
In short: ArTST is a specialized, all-in-one Arabic model for both text and speech. By training it from scratch on Arabic, the researchers achieved strong, sometimes state-of-the-art results for recognizing speech, generating speech, and spotting dialects—often beating much larger multilingual models.
Collections
Sign up for free to add this paper to one or more collections.