Mova0.5 pre-release⌕
☰
home / docs / papers / Semitic languages / Beyond Orthography: Automatic Recovery of Short Vowels and Dialectal Sounds in Arabic

Beyond Orthography: Automatic Recovery of Short Vowels and Dialectal Sounds in Arabic

What it is about

Arabic dialects contain sounds that Modern Standard Arabic spelling does not show (such as the g of Gulf and Egyptian loanwords, or p and v), and the written text often lacks short vowels. The framework takes frame-level features from XLS-R, quantises them with k-means into a small codebook (k = 128, 256 or 512), and feeds the codes to a small transformer that predicts the characters, including the dialect and borrowed sounds and the restored short vowels. With about 1.5 hours of training data, the paper reports a character error rate about 7% lower than the frozen-feature baseline on the new ArabVoice15 test set (5 hours, 15 countries). The paper also gives a rule-based letter-to-sound guide and an annotation protocol for verbatim dialectal transcription, in which ambiguous words are transcribed twice and files below 90% sample accuracy go back for full correction. The speech parts (the codebook and the perception tests) do not apply to Mova, which is text-only.

Ideas

For Mova

I1 is a plan for a future Arabic chapter, not a current file. I2 goes into tools/hbo, for its gold set. Nothing in the speech pipeline applies to the text parser.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.711

@inproceedings{kheir2024beyond,
  title = {{Beyond Orthography: Automatic Recovery of Short Vowels and Dialectal Sounds in Arabic}},
  author = {Kheir, Yassine El and Mubarak, Hamdy and Ali, Ahmed and Chowdhury, Shammur},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.711},
  url = {https://doi.org/10.18653/v1/2024.acl-long.711}
}

References

53 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16. find our card
  2. Ahmed Abdelali, Nadir Durrani, Cenk Demiroglu, Fahim Dalvi, Hamdy Mubarak, and Kareem Darwish. 2022. Natiq: An end-to-end text-to-speech system for arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop, pages 394–398, Abu Dhabi, UAE. Association for Computational Linguistics. find
  3. Ahmed Abdelali, Hamdy Mubarak, Shammur Absar Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Yousseif Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2024. LAraBench: Benchmarking Arabic AI with Large Language Models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, Malta. Association for Computational Linguistics. find
  4. Mansour M Al-Ghamdi, Husni Al-Muhtasib, and Moustafa Elshafei. 2004. Phonetic rules in arabic script. Journal of King Saud University-Computer and Information Sciences, 16:85–115. find
  5. Tuka Al Hanai and James R Glass. 2014. Lexical modeling for arabic asr: a systematic approach. In IN-TERSPEECH, pages 2605–2609. Speech Recognition and Understanding Workshop (ASRU), pages 1026–1033. IEEE. find
  6. Ahmed Ali, Yifan Zhang, Patrick Cardinal, Najim Da-hak, Stephan Vogel, and James Glass. 2014. A complete kaldi recipe for building arabic speech recognition systems. In 2014 IEEE spoken language technology workshop (SLT), pages 525–529. IEEE. find
  7. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670. ↗
  8. Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. 2021a. Unsupervised speech recognition. Advances in Neural Information Processing Systems, 34:27826–27839. find
  9. Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. 2021b. Unsupervised speech recognition. In NeurIPS. find
  10. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460. find
  11. Saurabhchand Bhati, Jesús Villalba, Piotr Żelasko, Laureano Moro-Velazquez, and Najim Dehak. 2022. Unsupervised speech segmentation and variable rate representation learning using segmental contrastive predictive coding. IEEE/ACM Transactions on Audio, Speech, and Language Processing. find
  12. Fadi Biadsy, Julia Bell Hirschberg, and Nizar Y Habash. 2009. Spoken arabic dialect identification using phonotactic modeling. find
  13. Hanan Aldarmaki and Ahmad Ghannam. 2023. Diacritic recognition performance in arabic asr. arXiv preprint arXiv:2302.14022. ↗
  14. Xuankai Chang, Brian Yan, Yuya Fujita, Takashi Maekaku, and Shinji Watanabe. 2023. Exploration of efficient end-to-end asr using discretized input from self-supervised learning. arXiv preprint arXiv:2305.18108. ↗
  15. Ahmed Ali, Peter Bell, James Glass, Yacine Messaoui, Hamdy Mubarak, Steve Renals, and Yifan Zhang. 2016. The MGB-2 challenge: Arabic multi-dialect broadcast media recognition. In SLT. find
  16. Kuan-Yu Chen et al. 2019. Completely unsupervised phoneme recognition by a generative adversarial network harmonized with iteratively refined hidden markov models. In Interspeech. find
  17. Ahmed Ali, Shammur Chowdhury, Mohamed Afify, Wassim El-Hajj, Hazem Hajj, Mourad Abbas, Amir Hussein, Nada Ghneim, Mohammad Abushariah, and Assal Alqudah. 2021. Connecting arabs: Bridging the gap in dialectal speech recognition. Communications of the ACM, 64(4):124–129. find
  18. Ahmed Ali, Suwon Shon, Younes Samih, Hamdy Mubarak, Ahmed Abdelali, James Glass, Steve Renals, and Khalid Choukri. 2019. The mgb-5 challenge: Recognition and dialect identification of dialectal arabic speech. In 2019 IEEE Automatic Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518. find
  19. Shammur A Chowdhury, Ahmed Ali, Suwon Shon, and James R Glass. 2020a. What does an end-to-end dialect identification model learn about non-dialectal information? In INTERSPEECH, pages 462–466. find
  20. Shammur A Chowdhury, Younes Samih, Mohamed Eldesouki, and Ahmed Ali. 2020b. Effects of dialectal code-switching on speech modules: A study using egyptian arabic broadcast speech. find
  21. Shammur Absar Chowdhury, Nadir Durrani, and Ahmed Ali. 2023. What do end-to-end speech models learn about speaker, language and channel information? a layer-wise and neuron-level analysis. Computer Speech & Language, 83:101539. find
  22. Hung yi Lee Da-Rong Liu, Kuan-Yu Chen and Lin shan Lee. 2018. Completely unsupervised phoneme recognition by adversarially learning mapping relationships from audio embeddings. In Interspeech. find
  23. Yi-chen Chen Sung-feng Huang Shun-po Chuang Dayi Wu Da-rong Liu, Po-chun Hsu and Hung yi Lee. 2022. Learning phone recognition from unpaired audio and phone sequences based on generative adversarial network. IEEE/ACM Transactions on Audio, Speech, and Language Processing. find
  24. Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali, Majd Hawasly, Nadir Durrani, and Firoj Alam. 2024. LLMeBench: A flexible framework for accelerating llms benchmarking. find
  25. David L Davies and Donald W Bouldin. 1979. A cluster separation measure. IEEE transactions on pattern analysis and machine intelligence, (2):224–227. find
  26. Ewan Dunbar et al. 2017. The zero resource speech challenge 2017. In ASRU. find
  27. Ryan Eloff, André Nortje, Benjamin van Niekerk, Avashna Govender, Leanne Nortje, Arnu Pretorius, Elan Van Biljon, Ewald van der Westhuizen, Lisa van Staden, and Herman Kamper. 2019. Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks. arXiv preprint arXiv:1904.07556. ↗
  28. Abhinav Garg, Jiyeon Kim, Sushil Khyalia, Chanwoo Kim, and Dhananjaya Gowda. 2024. Data-driven grapheme-to-phoneme representations for a lexicon-free text-to-speech. arXiv preprint arXiv:2401.10465. ↗
  29. John S Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, and David S Pallett. 1993. Darpa timit acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1. NASA STI/Recon technical report n, 93:27403. find
  30. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211. ↗
  31. Nawar Halabi and Mike Wald. 2016. Phonetic inventory for an arabic speech corpus. find
  32. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021a. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing. find
  33. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021b. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460. find
  34. Yassine El Kheir, Ahmed Ali, and Shammur Absar Chowdhury. 2024. Speech representation analysis based on inter- and intra-model similarities. In Explainable Machine Learning for Speech and Audio Workshop, ICASSP. find
  35. Felix Kreuk, Joseph Keshet, and Yossi Adi. 2020. Self-supervised contrastive learning for unsupervised phoneme segmentation. In Interspeech. find
  36. Xu Li, Shaoguang Mao, Xixin Wu, Kun Li, Xunying Liu, and Helen Meng. 2018. Unsupervised discovery of non-native phonetic patterns in l2 english speech for mispronunciation detection and diagnosis. In INTERSPEECH, pages 2554–2558. find
  37. John Makhoul, Salim Roucos, and Herbert Gish. 1985. Vector quantization in speech coding. Proceedings of the IEEE, 73(11):1551–1588. find
  38. Shaoguang Mao, Xu Li, Kun Li, Zhiyong Wu, Xunying Liu, and Helen Meng. 2018. Unsupervised discovery of an extended phoneme set in l2 english speech for mispronunciation detection and diagnosis. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6244–6248. IEEE. find
  39. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association. find
  40. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. find
  41. Alex S. Park and James R. Glass. 2008. Unsupervised pattern discovery in speech. IEEE Transactions on Audio, Speech, and Language Processing. find
  42. Ankita Pasad, Ju-Chieh Chou, and Karen Livescu. 2021. Layer-wise analysis of a self-supervised speech representation model. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 914–921. IEEE. find
  43. Ankita Pasad, Bowen Shi, and Karen Livescu. 2023. Comparative layer-wise analysis of self-supervised speech models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  44. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR. find
  45. Sara Shatnawi, Sawsan Alqahtani, and Hanan Aldarmaki. 2023. Automatic restoration of diacritics for speech data sets. arXiv preprint arXiv:2311.10771. ↗
  46. Amitay Sicherman and Yossi Adi. 2023. Analysing discrete self supervised speech representation for spoken language modeling. In ICASSP. find
  47. Vrunda Sukhadia and Shammur Absar Chowdhury. 2024. Children’s speech recognition through discrete token enhancement. In INTERSPEECH 2024. find
  48. Benjamin Van Niekerk, Leanne Nortje, and Herman Kamper. 2020. Vector-quantized neural networks for acoustic unit discovery in the zerospeech 2020 challenge. arXiv preprint arXiv:2005.09409. ↗
  49. Benjamin van Niekerk, Leanne Nortje, and Herman Kamper. 2020. Vector-quantized neural networks for acoustic unit discovery in the zerospeech 2020 challenge. In Interspeech 2020, pages 4836–4840. find
  50. Dimitra Vergyri and Katrin Kirchhoff. 2004. Automatic diacritization of arabic for acoustic modeling in speech recognition. In Proceedings of the workshop on computational approaches to Arabic script-based languages, pages 66–73. find
  51. Maarten Versteegh et al. 2015. The zero resource speech challenge 2015. In Interspeech. find
  52. Dan Wells, Hao Tang, and Korin Richmond. 2022. Phonetic analysis of self-supervised representations of english speech. In Interspeech. find
  53. Mingzhi Yu, Hieu Duy Nguyen, Alex Sokolov, Jack Lepird, Kanthashree Mysore Sathyendra, Samridhi Choudhary, Athanasios Mouchtaris, and Siegfried Kunzmann. 2020. Multilingual grapheme-to-phoneme conversion with byte representation. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8234–8238. IEEE. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/acl-2024-kheir-beyond-orthography-automatic-recovery-short/