An HMM Approach to Vowel Restoration in Arabic and Hebrew

What it is about

Hebrew and Arabic are written without most vowels, so many words have several readings, and a native reader uses context to choose. Gal restores vowels with a bigram hidden Markov model: each hidden state is a fully vowelled word that emits its unvowelled form, transitions are bigram counts with Katz back-off and Good–Turing discounting, and Viterbi finds the best path. Training and testing use a Hebrew Bible text (about 300,000 words) and a transliterated Qur'an (about 90,000 words), with 10% held out. Word accuracy is 81% for Hebrew (87% for the phonetic group of each vowel) and 86% for Arabic, against unigram baselines of 68% and 74%. Most word errors come from words absent from the training data (11% in Hebrew, 8% in Arabic).

Ideas

• I1. Use the unigram baseline as the floor for tools/hbo in unpointed mode: any claim there must beat 68% on the same kind of held-out Hebrew split. Report the split and the word accuracy beside it.
• I2. Katz back-off for unseen bigrams: when a pair of segmented words is unseen in the counts, back off to the unigram estimate with a discounted mass, rather than scoring zero. This is a small deterministic scorer, with the counts in BTreeMaps.
• I3. An "unseen word" state that emits the bare word with a fixed mass, rather than a guessed vowel pattern. Gal names morphological analysis as the next step for unseen words; macula dict has no roots yet, so it is the place where that analysis would come from.

For Mova

I1 and I2 go into tools/hbo (unpointed mode and its evaluation). I3 goes into The lexicon, linked to macula dict.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1118637.1118641

@inproceedings{gal2002hmm,
  title = {{An HMM Approach to Vowel Restoration in Arabic and Hebrew}},
  author = {Gal, Ya'akov},
  booktitle = {Proceedings of the ACL-02 Workshop on Computational Approaches to Semitic Languages},
  year = {2002},
  doi = {10.3115/1118637.1118641},
  url = {https://doi.org/10.3115/1118637.1118641}
}

References

8 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Academy of the Hebrew Language 1957. “The rules for Hebrew-Latin transcription, ” In Memiors of the Academy of the Hebrew Language, pages 5-8 (in Hebrew) find
2. Beesley, K. 1998. "Arabic finite-state morphological analysis and generation," in COLING-96 Proceedings 1 : 89-94, Copenhagen. find
3. Charniak, E. 1995. Statistical Language Learning, MIT Press. find
4. Choueka, Y. and Neeman, Y. 1995. “Nakdan-Text, (an In-Context Text-Vocalizer for Modern Hebrew),” BISFAI-95, The Fifth Bar Ilan Symposium for Artificial Intelligence Dagan, D., Pereira P., and Lee L. 1994. “Similarity-based estimation of word cooccurrence probabilities, ” In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics. find
5. Gale, W. A. and Sampson, G. 1995. “Good-Turing Frequency Estimation Without Tears,” Journal of Quantitative Linguistics 2, 217-237. Katz, S., “Estimation of probabilities from sparse data for the language model component of a speech recognizer, ” In IEEE Transactions on Acoustics, Speech, and Signal Processing. find
6. Kontorovich L.and Lee D. 2001. “Problems in Semitic NLP,” NIPS Workshop on Machine Learning Methods for Text and Images Levinger, M., Ornan U., Itai A. 1995. “Learning Morpho-Lexical Probabilities from an Untagged Corpus with an Application to Hebrew, ” Computational Linguistics, 21(3): 383-404 find our card
7. Segel, A. 1997. “A probabilistic Morphological Analyzer for Hebrew undotted text,” MSc thesis, Israeli Institute of Technology, Technion. (in Hebrew) find
8. Westminster Theological Seminar 1998. "The Hebrew Morphological Database", Philadelphia, PA find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2002-gal-hmm-approach-vowel-restoration-arabic/