Simultaneous Tokenization and Part-Of-Speech Tagging for Arabic without a Morphological Analyzer

What it is about

Kulick tokenizes and tags Arabic in one step, without a full morphological analyzer. He splits the words into closed-class and open-class items. Closed-class words are matched by regular expressions taken from the ATB morphological guidelines. Open-class words are matched by generic affix templates, and a CRF (Mallet) classifies only the possible stem, over 24 labels that combine stem names with reduced core tags. Each source token must match exactly one expression once the tag is taken into account. On ATB3-v3.2 (402,291 tree tokens, 40 reduced core tags) a simple baseline reaches 96.0% tokenization and 87.4% POS accuracy, and the second run reaches 95.1% POS accuracy overall. Words not seen in training are much weaker than seen words, and the paper is clear that this is where the difficulty lies.

Ideas

• I1. Closed-class items as named regular expressions with a tag set per group, and a single-match constraint after the tag check. The constraint is a test: each token must match one expression.
• I2. Let the classifier predict only the stem frame (which affixes apply), and look up the affix tags from counts of (affix, tag) pairs. This keeps the label set small, and the counts can come from the MACULA trees.
• I3. Report seen and unseen tokens separately in every segmenter result. The gap between them is the real test of generalization.

For Mova

I1 goes into tools/hbo (the segmenter's pre and lex tables) and into the closed-class list in Pronouns, particles and prepositions. I2 is a design change for the unpointed mode of tools/hbo. I3 is a reporting rule for every tools/hbo evaluation.

Topics: Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{kulick2010simultaneous,
  title = {{Simultaneous Tokenization and Part-Of-Speech Tagging for Arabic without a Morphological Analyzer}},
  author = {Kulick, Seth},
  booktitle = {Proceedings of the ACL 2010 Conference Short Papers},
  year = {2010},
  url = {https://aclanthology.org/P10-2063/}
}

References

10 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tim Buckwalter. 2004. Buckwalter Arabic morphological analyzer version 2.0. Linguistic Data Consortium LDC2004L02. find
2. Mona Diab, Kadri Hacioglu, and Daniel Jurafsky. 2007. Automatic processing of Modern Standard Arabic text. In Abdelhadi Soudi, Antal van den Bosch, and Gunter Neumann, editors, Arabic Computational Morphology, pages 159–179. Springer. find
3. Mona Diab. 2009. Second generation tools (AMIRA 2.0): Fast and robust tokenization, pos tagging, and base phrase chunking. In Proceedings of 2nd International Conference on Arabic Language Resources and Tools (MEDAR), Cairo, Egypt, April. find
4. Nizar Habash and Owen Rambow. 2005. Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573– 580, Ann Arbor, Michigan, June. Association for Computational Linguistics. find our card
5. Seth Kulick, Ann Bies, and Mohamed Maamouri. 2010. Consistent and flexible integration of morphological annotation in the Arabic Treebank. In Language Resources and Evaluation (LREC). find
6. Mohamed Maamouri, Ann Bies, Sondos Krouna, Fatma Gaddeche, Basma Bouziri, Seth Kulick, Wig-dane Mekki, and Tim Buckwalter. 2009a. Arabic Treebank Morphological and Syntactic guidelines, July. http://projects.ldc.upenn.edu/ArabicTreebank. ↗
7. Mohamed Maamouri, Ann Bies, Seth Kulick, Sondos Krouna, Fatma Gaddeche, and Wajdi Zaghouani. 2009b. Arabic treebank part 3 - v3.2. Linguistic Data Consortium LDC2010T08, April. find
8. Mohammed Maamouri, Basma Bouziri, Sondos Krouna, David Graff, Seth Kulick, and Tim Buckwalter. 2009c. Standard Arabic morphological analyzer (SAMA) version 3.1. Linguistic Data Consortium LDC2009E73. find
9. Andrew McCallum. 2002. Mallet: A machine learning for language toolkit. http://mallet.cs.umass.edu. ↗
10. Ryan Roth, Owen Rambow, Nizar Habash, Mona Diab, and Cynthia Rudin. 2008. Arabic morphological tagging, diacritization, and lemmatization using lexeme models and feature ranking. In Proceedings of ACL-08: HLT, Short Papers, pages 117–120, Columbus, Ohio, June. Association for Computational Linguistics. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2010-kulick-simultaneous-tokenization-part-speech-tagging/