Issues in Arabic Orthography and Morphology Analysis
What it is about
A case report from the LDC, where the Buckwalter Arabic Morphological Analyzer was used on about 542,000 word tokens of three Arabic newswire corpora (2002–2004). The author lists the orthographic anomalies that the annotators met. In one agency's text, hamza is dropped on alif in stem-medial and stem-final positions. In another, word-final alif maqsura is spelled with two dots. Some "run-on" words join a preceding function word to the next word with no space, such as la- or ma- plus a verb. The analyzer was changed for some cases (both ya and alif maqsura are now tried) and kept strict for others (the "qala 'anna" construction is accepted as grammatical). The paper is not a measurement study. Its argument is that each normalisation carries assumptions that the output should record, and that run-on words make the definition of a "word token" itself doubtful.
Ideas
- I1. Record the normalisation in every analysis: each analysis carries a flag for the method used (exact match, or a rewrite such as alif maqsura for ya, or a hamza change), as the paper argues. In
tools/hbo(unpointed mode) a segmentation should say whether it was read as written or after a rewrite, so that later scores can tell the two apart. - I2. A test set of run-on forms: collect cases where a clitic or a short function word has lost its space before the next word (the paper's la + verb, ma + verb), and check that the segmenter proposes both one-token and two-token readings. This is a measure for
tools/hbo, and any Semitic segmenter Mova adds later. - I3. Keep "possible tokens" separate from "word tokens": an input string may be one token or several, and each reading should be scored on its own. This matches the way
tools/hboalready runs segmentation before parsing.
For Mova
I1 and I3 go into the design of tools/hbo (the segmenter stage, unpointed mode), and the orthographic variation discussed in the paper matches what 1. Script and pointing and 2. Script and spelling have to describe. I2 is a test set for tools/hbo. Arabic is not a Mova language yet, so nothing here goes into a grammar file.
Cite
@inproceedings{buckwalter2004issues,
title = {{Issues in Arabic Orthography and Morphology Analysis}},
author = {Buckwalter, Tim},
booktitle = {Proceedings of the Workshop on Computational Approaches to Arabic Script-based Languages},
year = {2004},
doi = {10.3115/1621804.1621813},
url = {https://doi.org/10.3115/1621804.1621813}
}References
- Elsaid Badawi, M.G. Carter, and Adrian Wallace. 2004. Modern Written Arabic: A Comprehensive Grammar. Routledge, London. find
- Tim Buckwalter. 1992. “Orthographic Variation in Arabic and its Relevance to Automatic Spell-Checking,” in Proceedings of the Third International Conference on Multilingual Computing (Arabic and Roman Script), University of Durham, U.K., December 10-12, 1992. find
- Tim Buckwalter. 2002. Buckwalter Arabic Morphological Analyzer Version 1.0. Linguistic Data Consortium, catalog number LDC2002L49 and ISBN 1-58563-257-0. < http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId= LDC2002L49 > Mohamed Maamouri, et al. 2003. Arabic Treebank: Part 1 v 2.0. Linguistic Data Consortium, catalog number LDC2003T06 and ISBN: 1-58563-261-9. < http://www.ldc.upenn.-edu/Catalog/CatalogEntry.jsp?catalogId=LDC2003T06 > Mohamed Maamouri, et al. 2004. Arabic Treebank: Part 2 v 2.0. Linguistic Data Consortium, catalog number LDC2004T02 and ISBN: 1-58563-282-1. < http://www.ldc.upenn.-edu/Catalog/CatalogEntry.jsp?catalogId=LDC2004T02 > Mohamed Maamouri, et al. 2004. Arabic Treebank: Part 3 v 1.0. Linguistic Data Consortium, catalog number LDC2004T11 and ISBN: 1-58563-298-8. < http://www.ldc.upenn.-edu/Catalog/CatalogEntry.jsp?catalogId=LDC2004T11 > ↗
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2004-buckwalter-issues-arabic-orthography-morphology-analysis/