Arabic Diacritization through Full Morphological Tagging

What it is about

The authors restore diacritics by choosing a whole diacritized analysis for each white-space token, not by labelling characters. The candidate analyses come from the Buckwalter analyzer; a set of taggers, one per morphological feature (including case and mood), score them; the best-scoring analysis is kept, and remaining ties are broken by n-gram models over fully diacritized word forms or over lexeme citation forms (trigrams over lemmas act as a rough word-sense step). The training data is ATB3-Train (288,000 words) with the devtest of 52,000 words used as the only held-out set, to compare with Zitouni et al. (2006) on their metric. The best system (tagger plus trigram lexeme model) gets a WER of 14.9% and a DER of 4.8% over all diacritics, and a WER of 5.5% and DER of 2.2% when the final letter is ignored. The lexeme model alone is weak; it helps only together with the tagger. The authors report that their system beats the Zitouni et al. figures in every column.

Ideas

• I1. Choose among whole candidate pointings, not among single marks: for an unpointed Hebrew token, the candidates produced by tools/hbo (segmenter and parser output) are scored and the best one kept, so that one wrong vowel does not split the word into many errors. The paper's WER-against-DER gap shows why this matters.
• I2. Report the last letter apart: the paper's second column drops the final letter, where case and mood sit, and the gap is large (WER 14.9% against 5.5%). For Hebrew, add the same column for the final vowel and the matres, so that the finals of verbs and nouns are measured on their own in tools/hbo.
• I3. A deterministic tie-break by counting: the paper's first method counts how many predicted feature values each candidate matches; this is a rule with no learning and it is a fair baseline. For Mova it is the first version of the choice step in tools/hbo before any learned scorer, and its numbers are the reference the SLM has to beat.

For Mova

I1 and I3 are design points for the unpointed mode of tools/hbo. I2 is a measure for the same tool, reported next to WER and DER. The lexeme trigram of the paper is a candidate for the lemma tables of treebanks/hbo/ptnk, to be tested on dev only; the tie-break idea also fits 1. Script and pointing, where the rules for pointing are written.

Topics: Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1614108.1614122

@inproceedings{habash2007arabic,
  title = {{Arabic Diacritization through Full Morphological Tagging}},
  author = {Habash, Nizar and Rambow, Owen},
  booktitle = {Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers},
  year = {2007},
  doi = {10.3115/1614108.1614122},
  url = {https://doi.org/10.3115/1614108.1614122}
}

References

14 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Ananthakrishnan, S. Narayanan, and S. Bangalore. 2005. Automatic diacritization of arabic transcripts for asr. In Proceedings of ICON-05, Kanpur, India. find
2. T. Buckwalter. 2004. Buckwalter Arabic morphological analyzer version 2.0. find
3. J. Giménez and L. Màrquez. 2004. Svmtool: A general pos tagger generator based on support vector machines. In Proceedings of LREC’04. find
4. N. Habash and O. Rambow. 2005. Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop. In Proceedings of (ACL’05). find our card
5. N. Habash and O. Rambow. 2007. Arabic Diacritization through Full Morphological Tagging: A Detailed Discussion. Techncial Report, Center for Computational Learning Systems, Columbia University. find our card
6. Jan Hajič, Otakar Smrž, Tim Buckwalter, and Hubert Jin. 2005. Feature-based tagger of approximations of functional Arabic morphology. In Proceedings of the Workshop on Treebanks and Linguistic Theories (TLT), Barcelona, Spain. find
7. Jan Hajič. 2000. Morphological tagging: Data vs. dictionaries. In Proceedings of (NAACL’00). find
8. Taku Kudo and Yuji Matsumoto. 2003. Fast methods for kernel-based text analysis. In Proceedings of (ACL’03). find
9. Mohamed Maamouri, Ann Bies, and Tim Buckwalter. 2004. The Penn Arabic Treebank: Building a large-scale annotated arabic corpus. In NEMLAR Conference on Arabic Language Resources and Tools, Cairo, Egypt. find
10. Mohamed Maamouri, Ann Bies, and Seth Kulick. 2006. Diacritization: A challenge to arabic treebank annotation and parsing. In Proceedings of the Conference of the Machine Translation SIG of the British Computer Society. find
11. Noah A. Smith, David A. Smith, and Roy W. Tromble. 2005. Context-based morphological disambiguation with random fields. In Proceedings of (EMNLP05). find
12. Andreas Stolcke. 2002. SRILM - an Extensible Language Modeling Toolkit. In Proceedings of the International Conference on Spoken Language Processing (ICSLP). find
13. Dimitra Vergyri and Katrin Kirchhoff. 2004. Automatic diacritization of arabic for acoustic modeling in speech recognition. In COLING 2004 Workshop on Computational Approaches to Arabic Script-based Languages, Geneva, Switzerland. find
14. Imed Zitouni, Jeffrey S. Sorensen, and Ruhi Sarikaya. 2006. Maximum entropy based restoration of arabic diacritics. In Proceedings of ACL’06. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2007-habash-arabic-diacritization-through-full-morphological/