Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop

What it is about

Tokenization, POS tagging and morphological disambiguation are handled as one step for undiacritized Arabic. A morphological analyzer (ALMORGEANA, built on the Buckwalter lexicon) first lists every analysis of each word, about 2 on average in the Penn Arabic Treebank. Then one SVM classifier (Yamcha) is trained for each of ten morphological features, and a choice is made among the analyses. Five ways of choosing are compared: majority vote, confidence-weighted vote, two mixtures of the two, and a rule learner (Ripper) over the classifier outputs. On the first test set the majority vote scores 96.6% per token against a unigram baseline of 92.1%. On the second set the rule learner is best (94.8%). Word-level tokenization reaches 99.3% on the first set. Reduced-tagset POS accuracy is 98.1%, and 97.6% on the Penn tagset. The authors report these as the best published figures for tokenization and tagging of natural undiacritized input, and note that unanalysed words (mostly names) are a small problem.

Ideas

• I1. Choose among analyses with a readable scorer instead of a black box: for each candidate analysis of a word, count how many feature decisions agree with it (majority combiner), and break ties by a fixed rule. The count is deterministic and can be printed as the reason for the choice. This suits the "say why" rule in tools/hbo after its segmenter has produced candidates.
• I2. A rule learner over agreement and confidence (the Ripper variant) as a second, separately measured chooser. It learns lower bounds from data, and its rules can be read. Test it on a held-out treebank split, and retrain only the chooser when the corpus changes, as the paper does.
• I3. Measure tokenization by its own unit: a word counts as right only when all its clitic fields and its stem match. Report word accuracy and token P/R/F separately, as the paper does. Use this as the gold test for tools/hbo against the MACULA-derived Hebrew trees, and report the share of words with no analysis separately (0.61% in the paper's test set).

For Mova

I1 and I3 go into tools/hbo, in the step after the candidate segmentations. I2 is a second chooser to measure against I1 on the same split. The per-word tokenization metric (I3) is also a test for Pronouns, particles and prepositions, which must say how clitics such as the conjunction and the pronominal suffixes are separated. Arabic itself is not a Mova language yet; the transferable part is the method, not the Arabic features.

Topics: Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1219840.1219911

@inproceedings{habash2005arabic,
  title = {{Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop}},
  author = {Habash, Nizar and Rambow, Owen},
  booktitle = {Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05)},
  year = {2005},
  doi = {10.3115/1219840.1219911},
  url = {https://doi.org/10.3115/1219840.1219911}
}

References

11 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Imad A. Al-Sughaiyer and Ibrahim A. Al-Kharashi. 2004. Arabic morphological analysis techniques: A comprehensive survey. Journal of the American Society for Information Science and Technology, 55(3):189–213. find
2. Tim Buckwalter. 2002. Buckwalter Arabic Morphological Analyzer Version 1.0. Linguistic Data Consortium, University of Pennsylvania, 2002. LDC Catalog No.: LDC2002L49. find
3. William Cohen. 1996. Learning trees and rules with set-valued features. In Fourteenth Conference of the American Association of Artificial Intelligence. AAAI. find
4. Kareem Darwish. 2003. Building a shallow Arabic morphological analyser in one day. In ACL02 Workshop on Computational Approaches to Semitic Languages, Philadelpia, PA. Association for Computational Linguistics. find our card
5. Mona Diab, Kadri Hacioglu, and Daniel Jurafsky. 2004. Automatic tagging of arabic text: From raw text to base phrase chunks. In 5th Meeting of the North American Chapter of the Association for Computational Linguistics/Human Language Technologies Conference (HLT-NAACL04), Boston, MA. find
6. Nizar Habash. 2005. Arabic morphological representations for machine translation. In Abdelhadi Soudi, Antal van den Bosch, and Guenter Neumann, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods, Text, Speech, and Language Technology. Kluwer/Springer. in press. find
7. Jan Hajič. 2000. Morphological tagging: Data vs. dictionaries. In 1st Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL’00), Seattle, WA. find
8. Taku Kudo and Yuji Matsumoto. 2003. Fast methods for kernel-based text analysis. In 41st Meeting of the Association for Computational Linguistics (ACL’03), Sapporo, Japan. find
9. Young-Suk Lee, Kishore Papineni, Salim Roukos, Ossama Emam, and Hany Hassan. 2003. Language model based Arabic word segmentation. In 41st Meeting of the Association for Computational Linguistics (ACL’03), pages 399–406, Sapporo, Japan. find
10. Mohamed Maamouri, Ann Bies, and Tim Buckwalter. 2004. The penn arabic treebank : Building a large-scale annotated arabic corpus. In NEMLAR Conference on Arabic Language Resources and Tools, Cairo, Egypt. find
11. Monica Rogati, J. Scott McCarley, and Yiming Yang. 2003. Unsupervised learning of arabic stemming using a parallel corpus. In 41st Meeting of the Association for Computational Linguistics (ACL’03), pages 391–398, Sapporo, Japan. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2005-habash-arabic-tokenization-part-speech-tagging/