Mova0.5 pre-release⌕
☰
home / docs / papers / Modern Hebrew / When Similar Means Different: Evaluating LLMs on Arabic–Hebrew Cognates

When Similar Means Different: Evaluating LLMs on Arabic–Hebrew Cognates

What it is about

Arabic and Hebrew share many words with similar spellings. Some are true cognates with the same meaning, some are false friends with different meanings, and some are loanwords from a third language. The authors build SemCog Bench: 1,858 validated word pairs (858 true cognates, 364 false friends, 636 loanwords), checked by two annotators with a Gwet AC1 of 0.80 on the class-validated subset. Each pair gets generated sentences that were reviewed by hand. The benchmark has two tasks, cognate identification and semantic disambiguation, both multiple choice. Several LLMs are tested on four input forms: undiacritized, diacritized, IPA and Romanized. Models lean on surface similarity, so false friends are the weakest class, and loanwords vary widely. Original-script input does best. The best cognate identification score is 82.1% (Qwen3.6-Plus), and the best open model reaches 78.5% (Gemma-4-31B-IT). Context and scale help only some models.

Ideas

For Mova

I1 goes into The lexicon of Biblical Aramaic and 8. Lexicon and register, as a root-level table. I2 goes into 8. Lexicon and register, for loanwords such as the ones listed in Modern Hebrew. I3 is a measurement for tools/hbo. The Arabic side is not yet a Mova target, so it is noted only as a future treebanks/ar check.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2606.13218

@misc{liang2026when,
  title = {{When Similar Means Different: Evaluating LLMs on Arabic–Hebrew Cognates}},
  author = {Liang, Junhong and Mokh, Noor Abo and Alhafni, Bashar},
  year = {2026},
  eprint = {2606.13218},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.13218}
}

References

63 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eihab Abu-Rabiah. 2025. Semantic and phonological effects on l2 to l1 translation: The case of cognates, false cognates, and non-cognate translation equivalents. Electronic Journal of Foreign Language Teaching, 22(1). find
  2. V.S.D.S.Mahesh Akavarapu and Arnab Bhattacharya. 2023. Cognate transformer for automated phonological reconstruction and cognate reflex prediction. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6852–6862, Singapore. Association for Computational Linguistics. find
  3. V.S.D.S.Mahesh Akavarapu and Arnab Bhattacharya. 2024. Automated cognate detection as a supervised link prediction task with cognate transformer. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 965–975, St. Julian’s, Malta. Association for Computational Linguistics. find
  4. Mohamed Anwar, Abed Alhakim Freihat, George Ibrahim, Mostafa Awad, Abdelrahman Sadallah, Gurpreet Gosal, Gokulakrishnan Ramakrishnan, Sarath Chandran, Biswajit Mishra, Rituraj Joshi, Ahmed Frikha, Etienne Goffinet, Abhishek Maiti, Ali El Filali, Sarah AlBarri, Samujjwal Ghosh, Rahul Pal, Parvez Mullah, Awantika Shukla, and 41 others. 2026. Jais 2: A family of arabic-centric open large language models. Preprint, arXiv:2608.13580. ↗
  5. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzieh Fadaee, and 2 others. 2024. Aya 23: Open weight releases to further multilingual progress. Preprint, arXiv:2405.15032. ↗
  6. Niyati Bafna, Josef van Genabith, Cristina España-Bonet, and Zdeněk Žabokrtský. 2022. Combining noisy semantic signals with orthographic cues: Cognate induction for the Indic dialect continuum. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 110–131, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  7. Khuyagbaatar Batsuren, Gabor Bella, and Fausto Giunchiglia. 2019. CogNet: A large-scale cognate database. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3136–3145, Florence, Italy. Association for Computational Linguistics. find
  8. Khuyagbaatar Batsuren, Gábor Bella, and Fausto Giunchiglia. 2022. A large and evolving cognate database. Language Resources and Evaluation, 56(1):165–189. find
  9. Yonatan Belinkov and James Glass. 2016. Large-scale machine translation between arabic and hebrew: Available corpora and initial results. Preprint, arXiv:1609.07701. ↗ our card
  10. Samuel Cahyawijaya, Ruochen Zhang, Holy Lovenia, Jan Christian Blaise Cruz, Elisa Gilbert, Hiroki Nomoto, and Alham Fikri Aji. 2025. Thank you, stingray: Multilingual large language models can not (yet) disambiguate cross-lingual word senses. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3228–3250, Mexico City, Mexico. Association for Computational Linguistics. find
  11. Mauro Cettolo. 2016. An Arabic-Hebrew parallel corpus of TED talks. Preprint, arXiv:1610.00572. ↗ our card
  12. Pedro J. Chamizo Domínguez and Brigitte Nerlich. 2002. False friends: their origin and semantics in some selected languages. Journal of Pragmatics, 34(12):1833–1849. find
  13. Alina Maria Cristea, Liviu P. Dinu, Simona Georgescu, Mihnea-Lucian Mihai, and Ana Sabina Uban. 2021. Automatic discrimination between inherited and borrowed Latin words in Romance languages. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2845–2855, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  14. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. ↗
  15. Liviu Dinu, Ana Uban, Alina Cristea, Anca Dinu, Ioan-Bogdan Iordache, Simona Georgescu, and Laurentiu Zoicas. 2023. RoBoCoP: A comprehensive ROmance BOrrowing COgnate package and benchmark for multilingual cognate identification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7610–7629, Singapore. Association for Computational Linguistics. find
  16. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne, Australia. Association for Computational Linguistics. find
  17. Alvan R. Feinstein and Domenic V. Cicchetti. 1990. High agreement but low kappa: I. the problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549. find
  18. Stephen Gadd. 2026. Symphonym: Universal phonetic embeddings for cross-script name matching. Preprint, arXiv:2601.06932. ↗ our card
  19. Bar Gazit, Shaltiel Shmidman, Avi Shmidman, and Yuval Pinter. 2025. Splintering nonconcatenative languages for better tokenization. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22405–22417, Vienna, Austria. Association for Computational Linguistics. find
  20. Team GLM, :, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, and 40 others. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. Preprint, arXiv:2406.12793. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  21. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  22. Nizar Habash. 2002. The Palisra Project: A vision for a one-state solution. https://www.palisra.com/mission. Accessed: 2026-06-06. ↗
  23. Nizar Habash. 2023. An introduction to Palisri: One language for Israel–Palestine. https://www.palisra.com/. Accessed: 2026-06-06. ↗
  24. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods, pages 15–22. Springer, Netherlands. find
  25. Nizar Y Habash. 2010. Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers. find
  26. Bradley Hauer and Grzegorz Kondrak. 2015. Automatic cognate identification with gap-weighted string subsequence kernels. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1227–1231, Denver, Colorado. Association for Computational Linguistics. find
  27. Ulf Hermjakob, Jonathan May, and Kevin Knight. 2018. Out-of-the-box universal Romanization tool uroman. In Proceedings of ACL 2018, System Demonstrations, pages 13–18, Melbourne, Australia. Association for Computational Linguistics. find
  28. Diptesh Kanojia, Raj Dabre, Shubham Dewangan, Pushpak Bhattacharyya, Gholamreza Haffari, and Malhar Kulkarni. 2020a. Harnessing cross-lingual features to improve cognate detection for low-resource languages. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1384–1395, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  29. Diptesh Kanojia, Sumegh Munukutla, Sharmila Ghodekar, Pushpak Bhattacharyya, and Manish Kulkarni. 2020b. Keep your dimensions on a leash: True cognate detection using siamese deep neural networks. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, pages 324–325, Hyderabad, India. Association for Computing Machinery. find
  30. Diptesh Kanojia, Prashant Sharma, Sayali Ghodekar, Pushpak Bhattacharyya, Gholamreza Haffari, and Malhar Kulkarni. 2021. Cognition-aware cognate detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3281–3292, Online. Association for Computational Linguistics. find
  31. Salam Khalifa, Abdelrahim Qaddoumi, Jordan Kodner, and Owen Rambow. 2025. Learning cross-dialectal morphophonology with syllable structure constraints. In Proceedings of the 12th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 157–167, Abu Dhabi, UAE. Association for Computational Linguistics. find
  32. Yakov Kolani, Maxim Melichov, Cobi Calev, and Morris Alper. 2025. Phonikud: Hebrew grapheme-to-phoneme conversion for real-time text-to-speech. Preprint, arXiv:2506.12311. ↗ our card
  33. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626. find
  34. Agnieszka Lijewska. 2020. Cognate processing effects in bilingual lexical access. Bilingual lexical ambiguity resolution, pages 71–95. find
  35. Johann-Mattis List. 2012a. Lexstat: Automatic detection of cognates in multilingual wordlists. Proceedings of the EACL 2012 Joint Workshop of LINGVIS & UNCLH, pages 117–125. Tutorial and method description for the LingPy LexStat algorithm. find
  36. Johann-Mattis List. 2012b. Sca: Phonetic alignment based on sound classes. In New Directions in Logic, Language and Computation, pages 32–51, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  37. Johann-Mattis List and Steven Moran. 2013. An open source toolkit for quantitative historical linguistics. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 13–18, Sofia, Bulgaria. Association for Computational Linguistics. find
  38. Johann-Mattis List, Ekaterina Vylomova, Robert Forkel, Nathan Hill, and Ryan Cotterell. 2022. find
  39. The SIGTYP 2022 shared task on the prediction of cognate reflexes. In Proceedings of the 4th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 52–62, Seattle, Washington. Association for Computational Linguistics. find
  40. Roddy MacSween and Andrew Caines. 2020. An expectation maximisation algorithm for automated cognate detection. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 476–485, Online. Association for Computational Linguistics. find
  41. Duaa Abu-Elhij’a Mahajna. 2019. A study of loanwords and code switching in spoken and online written Arabic by Palestinian Israelis. Ph.D. thesis, Indiana University. find
  42. Juan Moreno Gonzalez, Bashar Alhafni, and Nizar Habash. 2026. A tale of two scripts: Transliteration and post-correction for Judeo-Arabic. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2100–2113, Rabat, Morocco. Association for Computational Linguistics. find
  43. Abhijnan Nath, Rahul Ghosh, and Nikhil Krishnaswamy. 2022. Phonetic, semantic, and articulatory features in Assamese-Bengali cognate detection. In Proceedings of the Ninth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 41–53, Gyeongju, Republic of Korea. Association for Computational Linguistics. find
  44. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association. find
  45. Tetsuji Ohyama. 2021. Statistical inference of gwet’s ac1 coefficient for multiple raters and binary outcomes. Communications in Statistics - Theory and Methods, 50(15):3564–3572. find
  46. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. GPT-OSS-120B and GPT-OSS-20b model card. Preprint, arXiv:2508.10925. ↗
  47. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774. ↗
  48. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. ↗
  49. Taraka Rama. 2018. Similarity dependent Chinese restaurant process for cognate identification in multilingual wordlists. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 271–281, Brussels, Belgium. Association for Computational Linguistics. find
  50. Taraka Rama and Johann-Mattis List. 2019. An automated framework for fast cognate detection and Bayesian phylogenetic inference in computational historical linguistics. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6225–6235, Florence, Italy. Association for Computational Linguistics. find
  51. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. find
  52. Mohammed J. Saeed, Tommi Vehvilainen, Evgeny Fedoseev, Sevil Caliskan, and Tatiana Vodolazova. 2025. Impact: Inflectional morphology probes across complex typologies. Preprint, arXiv:2506.23929. ↗
  53. Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Galletebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yannick Versley, and 4 others. 2013. Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 146–182, Seattle, Washington, USA. Association for Computational Linguistics. find
  54. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, and 13 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. Preprint, arXiv:2308.16149. ↗
  55. Reshef Shilon, Nizar Habash, Alon Lavie, and Shuly Wintner. 2010. Machine translation between Hebrew and Arabic: Needs, challenges and preliminary solutions. In Proceedings of the 9th Conference of the Association for Machine Translation in the Americas: Student Research Workshop, Denver, Colorado, USA. Association for Machine Translation in the Americas. find our card
  56. Shaltiel Shmidman, Avi Shmidman, Amir DN Cohen, and Moshe Koppel. 2024. Adapting llms to hebrew: Unveiling dictalm 2.0 with enhanced vocabulary and instruction capabilities. Preprint, arXiv:2407.07080. ↗ our card
  57. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker-Whitcomb, Alex Beutel, Alex Karpenko, and 467 others. 2026. Openai gpt-5 system card. Preprint, arXiv:2601.03267. ↗
  58. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, and 1 others. 2025. Gemini: A family of highly capable multimodal models. Preprint, arXiv:2312.11805. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179 others. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. ↗
  59. Reut Tsarfaty, Dan Bareket, Stav Klein, and Amit Seker. 2020. From SPMRL to NMRL: What did we learn (and unlearn) in a decade of parsing morphologically-rich languages (MRLs)? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7396–7408, Online. Association for Computational Linguistics. find
  60. Ana Sabina Uban, Liviu P Dinu, Ioan-Bogdan Iordache, Simona Georgescu, and Claudia Vlad. 2025. Friend or foe? a computational investigation of semantic false Friends across Romance languages. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15298–15312, Suzhou, China. Association for Computational Linguistics. find
  61. Nahathai Wongpakaran, Tinakon Wongpakaran, Danny Wedding, and Kilem L. Gwet. 2013. A comparison of Cohen’s Kappa and Gwet’s Ac1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Medical Research Methodology, 13(1):61. find
  62. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗
  63. Wajdi Zaghouani, Mustafa Jarrar, Nizar Habash, Houda Bouamor, Imed Zitouni, Mona Diab, Samhaa El-Beltagy, and Muhammed AbuOdeh. 2024. The FIGNEWS shared task on news media narratives. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 530–547, Bangkok, Thailand. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2026-liang-similar-means-different-evaluating-llms/