Arabic Diacritization Using Weighted Finite-State Transducers

What it is about

The paper restores short vowels and other diacritics (shadda, sukun, tanween, hamza forms) to undiacritized Modern Standard Arabic with a cascade of weighted finite-state transducers, modelling the loss of diacritics generatively and decoding with Viterbi. The cascade is: a word trigram language model with Katz backoff over diacritized words; a spelling transducer from words to letters; a diacritic-drop transducer, which also lists the alternative written forms of hamza without rules for when each occurs; a clitic-concatenation transducer that can join a fixed list of prefixes (b, l, k, w, f, Al, s) and pronoun suffixes to a word, with no knowledge of legal orders; and an unknown-word transducer. Unknown words are then diacritized by a 4-gram letter model composed with the drop transducer. Training and testing use the Arabic Treebank (Al-Hayat, 90/10 random split, about 14,000 test words). Without case endings, the unigram baseline gives WER 15.48% and DER 17.33%; the trigram model alone helps little (14.64/16.9); adding the clitic model gives 8.49/9.32; adding the letter model gives 7.33/6.35. With case endings the best is WER 23.61%, DER 12.79%. The text extraction of this PDF failed (font-encoded Arabic), so the card was written from the page images.

Ideas

• I1. The whole method is deterministic once trained and needs very little morphology: count-based word n-grams, a clitic list with no ordering rules, and a letter n-gram model only for unknown words. For unpointed Hebrew this is a direct baseline for vowel restoration in tools/hbo: word trigram over pointed forms, a prefix/suffix transducer from the segmenter's tables, and a letter model as fallback — measured with the same two numbers (WER, DER).
• I2. The ablation shape is the lesson: the clitic model gives the largest single gain (WER 14.64 → 8.49), larger than the language model itself, because a clitic-split corpus has far fewer types. Measure the same step for Hebrew (pointing with and without the prefix split) before adding anything else.
• I3. Use the letter model selectively, only on words unknown to the word model; the authors found that better than running it on everything. A two-tier rule (lexicon first, letter model second, with the tier recorded) keeps each pointing explainable.
• I4. Report results with and without the final (case/mood) vowel, since the gap is large; for Hebrew the analogue is the word-final vowel and the paragogic endings.

For Mova

I1–I3 apply to the unpointed mode of tools/hbo and to 1. Script and pointing; I4 is a reporting rule for the same measurement. Gal's HMM (card aclbulk-2002-gal-hmm-approach-vowel-restoration-arabic) is the nearest relative and the baseline to compare against.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1621787.1621802

@inproceedings{nelken2005arabic,
  title = {{Arabic Diacritization Using Weighted Finite-State Transducers}},
  author = {Nelken, Rani and Shieber, Stuart M.},
  booktitle = {Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages},
  year = {2005},
  doi = {10.3115/1621787.1621802},
  url = {https://doi.org/10.3115/1621787.1621802}
}

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2005-nelken-arabic-diacritization-weighted-finite-state/