How Much Does Lookahead Matter for Disambiguation? Partial Arabic Diacritization Case Study

What it is about

The authors ask which short vowels a reader of unpointed Arabic really needs, and they print only those. Two neural diacritizers are trained on the Tashkeela corpus: a unidirectional LSTM that sees only the words already read, and a transformer that sees the whole sentence. Where the two disagree, the whole-sentence reading is kept and its diacritic is printed; everywhere else the letter stays bare. A new set of 75 MSA tweets, fully and partially vowelized by native readers (who kept 19–26% of the diacritics), supports the evaluation. The partial diacritizer improves downstream translation over no diacritics and over random selection. The full-sentence model reaches 3.57 DER / 8.52 WER with case endings on the Tashkeela test set, including NO DIACRITIC.

Ideas

• I1. Lookahead as disagreement: run a left-context-only analysis and a full-sentence analysis over the same text, and mark each word where they differ as "needs lookahead". This is a deterministic diff of two passes, and it names the words a left-to-right reader cannot settle.
• I2. Measure the lookahead set on unpointed Biblical Hebrew: how many MACULA tokens change segmentation or lemma when the right context is removed. It gives a number for how much unpointed text really depends on what follows.
• I3. Data rule for training gold: drop lines whose share of marked letters falls under a threshold (the paper uses 50% for MSA against 80% in earlier work). Keep the threshold as a recorded parameter, so every run can state it.

For Mova

I1 and I2 go into tools/hbo (unpointed mode), as a second pass next to the segmenter; the lookahead set is also a check in 13. Biblical Hebrew in Mova: the tools. I2 uses treebanks/hbo/ptnk as its gold. I3 is a rule for any pointed training text built for vowel restoration in tools/hbo.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00456

@article{esmail2022how,
  title = {{How Much Does Lookahead Matter for Disambiguation? Partial Arabic Diacritization Case Study}},
  author = {Esmail, Saeed and Bar, Kfir and Dershowitz, Nachum},
  journal = {Computational Linguistics},
  year = {2022},
  doi = {10.1162/coli\_a\_00456},
  url = {https://doi.org/10.1162/coli_a_00456}
}

References

56 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abandah, Gheith A., Alex Graves, Balkees Al-Shagoor, Alaa Arabiyat, Fuad Jamour, and Majid Al-Taee. 2015. Automatic diacritization of Arabic text using recurrent neural networks. International Journal on Document Analysis and Recognition (IJDAR), 18(2):183–197. https://doi.org/10.1007/s10032-015 -0242-2 ↗
2. Abbad, Hamza and Shengwu Xiong. 2020. Multi-components system for automatic Arabic diacritization. In European Conference on Information Retrieval, pages 341–355. https://doi.org/10 .1007/978-3-030-45439-5 23 ↗
3. Abu-Hamour, Bashir, Hanan Al-Hmouz, and Mohammed Kenana. 2013. The effect of short vowelization on curriculum-based measurement of reading fluency and comprehension in Arabic. Australian Journal of Learning Difficulties, 18(2):181–197. https://doi.org/10 .1080/19404158.2013.852980 ↗
4. Abu-Rabia, Salim. 1995. Learning to read in Arabic: Reading, syntactic, orthographic and working memory skills in normally achieving and poor Arabic readers. Reading Psychology: An International Quarterly, 16(4):351–394. https://doi.org/10.1080/0270271950160401 ↗
5. Abu-Rabia, Salim. 1996. The role of vowels and context in the reading of highly skilled native Arabic readers. Journal of Psycholinguistic Research, 25(6):629–641. https://doi.org/10.1007/BF01712413 ↗
6. Abu-Rabia, Salim. 1997a. The need for cross–cultural considerations in reading theory: The effects of Arabic sentence context in skilled and poor readers. Journal of Research in Reading, 20(2):137–147. https://doi.org/10.1111/1467-9817 .00026 ↗
7. Abu-Rabia, Salim. 1997b. Reading in Arabic orthography: The effect of vowels and context on reading accuracy of poor and skilled native Arabic readers in reading paragraphs, sentences, and isolated words. Journal of Psycholinguistic Research, 26(4):465–482. find
8. Abu-Rabia, Salim. 1998a. Attitudes and culture in second language learning among Israeli-Arab students. Curriculum and Teaching, 13(1):13–30. https://doi.org/10.7459/ct/13.1.03 ↗
9. Abu-Rabia, Salim. 1998b. Reading Arabic texts: Effects of text type, reader type and vowelization. Reading and Writing, 10(2):105–119. https://doi.org/10 .1023/A:1007906222227 ↗
10. Abu-Rabia, Salim. 1999. The effect of Arabic vowels on the reading comprehension of second-and sixth-grade native Arab children. Journal of Psycholinguistic Research, 28(1):93–101. https://doi.org/10.1023/A:1023291620997, PubMed: 9949716 ↗
11. Abu-Rabia, Salim. 2001. The role of vowels in reading Semitic scripts: Data from Arabic and Hebrew. Reading and Writing, 14(1):39–59. https://doi.org/10.1023/A:1008147606320 ↗
12. Abu-Rabia, Salim. 2019. The role of short vowels in reading Arabic: A critical literature review. Journal of Psycholinguistic Research, 48(4):785–795. https://doi.org/10.1007/s10936-019-09631-4, PubMed: 30719613 ↗
13. AlKhamissi, Badr, Muhammad ElNokrashy, and Mohamed Gabr. 2020. Deep diacritization: Efficient hierarchical recurrence for improved Arabic diacritization. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 38–48. find
14. Alnefaie, Rehab and Aqil M. Azmi. 2017. Automatic minimal diacritization of Arabic texts. Procedia Computer Science, 117:169–174. https://doi.org/10.1016/j.procs.2017.10.106 ↗
15. Alqahtani, Sawsan, Hanan Aldarmaki, and Mona Diab. 2019. Homograph disambiguation through selective diacritic restoration. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 49–59. https://doi.org/10.18653/v1/W19-4606 ↗
16. Alqahtani, Sawsan, Ajay Mishra, and Mona Diab. 2019. Efficient convolutional neural networks for diacritic restoration. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1442–1448. https://doi.org/10.18653/v1/D19 -1151 ↗
17. Alqahtani, Sawsan, Ajay Mishra, and Mona Diab. 2020. A multitask learning approach for diacritic restoration. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8238–8247. https://doi.org/10.18653/v1/2020 .acl-main.732 ↗ our card
18. Asadi, Ibrahim A. 2017. Reading Arabic with the diacritics for short vowels: Vowelised but not necessarily easy to read. Writing Systems Research, 9(2):137–147. https://doi.org/10.1080/17586801 .2017.1400493 ↗
19. Belinkov, Yonatan and James Glass. 2015. Arabic diacritization with recurrent neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2281–2285. https://doi.org/10.18653/v1/D15-1274 ↗ our card
20. Bouamor, Houda, Wajdi Zaghouani, Mona Diab, Ossama Obeid, Kemal Oflazer, Mahmoud Ghoneim, and Abdelati Hawwari. 2015. A pilot study on Arabic multi-genre corpus diacritization. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 80–88. https://doi.org/10.18653/v1/W15 -3209 ↗
21. Dang, Trung Duc Anh and Thi Thu Trang Nguyen. 2020. TDP–A hybrid diacritic restoration with transformer decoder. In Proceedings of the 34th Pacific Asia Conference on Language, Information and Computation, pages 76–83. find
22. Daniels, Peter T. and William Bright. 1996. The World’s Writing Systems. Oxford University Press on Demand. find
23. Darwish, Kareem, Hamdy Mubarak, and Ahmed Abdelali. 2017. Arabic diacritization: Stats, rules, and hacks. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 9–17. https://doi.org/10.18653/v1/W17 -1302 ↗ our card
24. Diab, Mona, Mahmoud Ghoneim, and Nizar Habash. 2007. Arabic diacritization in the context of statistical machine translation. Proceedings of MT Summit, 7 pages. https://aclanthology.org/2007 .mtsummit-papers.20.pdf Fadel, Ali, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019a. Neural Arabic text diacritization: State of the art results and a novel approach for machine translation. In Proceedings of the 6th Workshop on Asian Translation, WAT@EMNLP-IJCNLP 2019, pages 215–225. https://doi.org/10 .18653/v1/D19-5229 ↗ our card
25. Fadel, Ali, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019b. Arabic text diacritization using deep neural networks. In Proceedings of the 2nd International Conference on Computer Applications & Information Security (ICCAIS), pages 1–7. https://doi.org/10.1109/CAIS.2019.8769512 ↗
26. Frost, Ram, Leonard Katz, and Shlomo Bentin. 1987. Strategies for visual word recognition and orthographical depth: A multilingual comparison. Journal of Experimental Psychology: Human Perception and Performance, 13(1):104. https://doi.org/10.1037/0096-1523 .13.1.104 ↗
27. Habash, Nizar and Owen Rambow. 2007. Arabic diacritization through full morphological tagging. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pages 53–56. https://doi.org/10.3115/1614108.1614122 ↗ our card
28. Habash, Nizar, Anas Shahrour, and Muhamed Al-Khalil. 2016. Exploiting Arabic diacritization for high quality automatic annotation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4298–4304. find our card
29. Habash, Nizar Y. 2010. Introduction to Arabic Natural Language Processing, volume 10 of Synthesis Lectures on Human Language Technologies (Graeme Hirst, ed.). Morgan & Claypool Publishers. https://doi.org/10.1007/978-3-031-02139-8 ↗
30. Hochreiter, Sepp and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735, PubMed: 9377276 ↗
31. Hucko, A. and P. Lacko. 2018. Diacritics restoration using deep neural networks. In 2018 World Symposium on Digital Intelligence for Systems and Machines (DISA), pages 195–200. https://doi.org/10.1109/DISA.2018.8490624 ↗
32. Hung, Bui T. 2018. Vietnamese diacritics restoration using deep learning approach. In 2018 10th International Conference on Knowledge and Systems Engineering (KSE), pages 347–351. https://doi.org/10 .1109/KSE.2018.8573427 ↗
33. Ibrahim, Raphiq. 2013. Reading in Arabic: New evidence for the role of vowel signs. Creative Education, 4(4):248–253. https://doi.org/10.4236/ce.2013.44036 ↗
34. Katz, Leonard and Ram Frost. 1992. The reading process is different for different orthographies: The orthographic depth hypothesis. In Advances in Psychology, volume 94. Elsevier, pages 67–84. https://doi.org/10.1016/S0166 -4115(08)62789-2 ↗
35. Laki, László János and Zijian Gyozo Yang. 2020. Automatic diacritic restoration with transformer model based neural machine translation for east-central European languages. In Proceedings of the 11th International Conference on Applied Informatics (ICAI), pages 190–202. find
36. Liberman, Isabelle Y., Alvin M. Liberman, Ignatius Mattingly, and Donald Shankweiler. 1980. Orthography and the beginning reader. In J. F. Kavanagh and R. L. Vinezky, editors, Orthography, Reading, and Dyslexia. University Park Press, Baltimore, MD, chapter 10, pages 137–153. find
37. Madhfar, Mokthar Ali Hasan and Ali Mustafa Qamar. 2020. Effective deep learning models for automatic diacritization of Arabic text. IEEE Access, 9:273–288. https://doi.org/10.1109/ACCESS.2020.3041676 ↗
38. Marcus, Mitchell Philip. 1978. A Theory of Syntactic Recognition for Natural Language. Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, MA. find
39. Mijlad, Ali and Yacine El Younoussi. 2019. Arabic text diacritization: Overview and solution. In Proceedings of the 4th International Conference on Smart City Applications, SCA ’19, pages 1–7. https://doi.org/10.1145/3368756.3369088 ↗
40. Mubarak, Hamdy, Ahmed Abdelali, Hassan Sajjad, Younes Samih, and Kareem Darwish. 2019. Highly effective Arabic diacritization using sequence to sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2390–2395. https://doi.org/10.18653/v1/N19-1248 ↗
41. Náplava, Jakub, Milan Straka, Pavel Straňák, and Jan Hajič. 2018. Diacritics restoration using neural networks. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 1566–1573. find
42. Nga, Cao Hong, Nguyen Khai Thinh, Pao-Chi Chang, and Jia-Ching Wang. 2019. Deep learning based Vietnamese diacritics restoration. In 2019 IEEE International Symposium on Multimedia (ISM), pages 331–334. https://doi.org/10 .1109/ISM46123.2019.00074 ↗
43. Nguyen, Kiet, Vu Nguyen, Anh Nguyen, and Ngan Nguyen. 2020. A Vietnamese dataset for evaluating machine reading comprehension. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2595–2605. https://doi.org/10.18653/v1/2020 .coling-main.233 ↗
44. Nuţu, Maria, Beata Lőrincz, and Adriana Stan. 2019. Deep learning for automatic diacritics restoration in Romanian. In 2019 IEEE 15th International Conference on Intelligent Computer Communication and Processing (ICCP), pages 235–240. https://doi.org/10.1109/ICCP48234 .2019.8959557 ↗
45. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. B LEU : A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. https://doi.org/10.3115/1073083 .1073135 ↗
46. Shmidman, Avi, Shaltiel Shmidman, Moshe Koppel, and Yoav Goldberg. 2020. Nakdan: Professional Hebrew diacritizer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 197–203. https://doi.org/10.18653/v1/2020 .acl-demos.23 ↗ our card
47. Stankevičius, Lukas, Mantas Lukoševičius, Jurgita Kapočiūtė-Dzikienė, Monika Briedienė, and Tomas Krilavičius. 2022. Correcting diacritics and typos with a ByT5 transformer model. Applied Sciences, 12(5):2636. https://doi.org/10.3390/app12052636 ↗
48. Taha, Haitham. 2016. Deep and shallow in Arabic orthography: New evidence from reading performance of elementary school native Arab readers. Writing Systems Research, 8(2):133–142. https://doi.org/10.1080/17586801.2015.1114910 ↗
49. Uzun, Aysenur. 2018. Diacritic restoration using neural network. Technical report, Computer Engineering, Istanbul Technical University. find
50. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. find
51. Wang, Yuxuan, R. J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Z. Chen, Samy Bengio, Quoc V. Le, Yannis Agiomyrgiannakis, Robert A. J. Clark, and Rif A. Saurous. 2017. Tacotron: Towards end-to-end speech synthesis. In Proceedings of INTERSPEECH, pages 4006–4010. https://doi.org/10 .21437/Interspeech.2017-1452 ↗
52. Xue, Linting, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. https://doi.org/10.1162/tacl a 00461 ↗
53. Zalmout, Nasser and Nizar Habash. 2020. Joint diacritization, lemmatization, normalization, and fine-grained morphological tagging. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8297–8307. https://doi.org/10.18653/v1/2020 .acl-main.736 ↗ our card
54. Zerrouki, Taha and Amar Balla. 2017. Tashkeela: Novel corpus of Arabic vocalized texts, data for auto-diacritization systems. Data in Brief, 11:147–151. https://doi.org/10.1016/j.dib.2017 .01.011, PubMed: 28224131 ↗
55. Zitouni, Imed and Ruhi Sarikaya. 2009. Arabic diacritic restoration approach based on maximum entropy models. Computer Speech & Language, 23(3):257–276. https://doi.org/10.1016/j.csl.2008 .06.001 ↗
56. Zitouni, Imed, Jeffrey S. Sorensen, and Ruhi Sarikaya. 2006. Maximum entropy based restoration of Arabic diacritics. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 577–584. https://doi.org/10.3115/1220175 .1220248 ↗ our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/cl-2022-esmail-how-much-does-lookahead-matter/