Learning Morpho-Lexical Probabilities from an Untagged Corpus with an Application to Hebrew

What it is about

Unpointed Hebrew writing leaves most vowels out, so many word tokens have several morphological analyses; in a sample of press text about 55% of tokens were ambiguous. The authors' point is that the forms of one lexical entry help each other: some forms are unambiguous, so their analyses can be counted in an untagged corpus. For each ambiguous word they estimate the probability of each analysis, P(analysis | word), from how often that analysis is the right one, using the evidence of sibling forms of the same lexeme, with a morphological analyzer supplying the candidates and no hand tagging. They then use these probabilities to resolve ambiguity. On 21 test articles (3,400 tokens, about half of them ambiguous) they report results in the seventies to nineties, with separate figures for accuracy and applicability. Their error analysis finds 5 wrong out of 53 words in the test groups (9.5%), and a coverage failure, where the corpus lacks enough data, in only about 3% of ambiguous words.

Ideas

• I1. Sibling-form voting for the unpointed mode of tools/hbo: when a token has several analyses, count how often each candidate lemma appears with unambiguous forms in the corpus, choose the analysis that its siblings support, and leave the token ambiguous otherwise. The lemma list can come from macula dict.
• I2. Report applicability and accuracy as two numbers, as the paper does: how many ambiguous tokens are fully resolved, and how many of those are right. Mova's tools/hbo and tools/macula compare reports should carry both.
• I3. A coverage fallback: when a lemma has no unambiguous sibling in the corpus, fall back to a cross-lexeme prior and log that the fallback was used. The paper finds this case rare, and the log would show how rare it is for Ancient Hebrew.

For Mova

I1 is a rule for the unpointed mode of tools/hbo and for its segmenter tables. I2 is a measure for tools/hbo and tools/macula compare. I3 goes into 13. Biblical Hebrew in Mova: the tools, with the sibling-form evidence noted in The lexicon. The paper's Hebrew is Modern, so the numbers do not carry over; only the method does.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@misc{levinger1995learning,
  title = {{Learning Morpho-Lexical Probabilities from an Untagged Corpus with an Application to Hebrew}},
  author = {Levinger, Moshe and Ornan, Uzzi and Itai, Alon},
  year = {1995},
  url = {https://aclanthology.org/J95-3004/}
}

References

22 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Academy of The Hebrew Language (1957). "The rules for Hebrew-Latin transcription." Memoirs of the Academy of the Hebrew Language, 5--8. find
2. Agresti, Alan (1990). Categorical Data Analysis. John Wiley and Sons. find
3. Albeck, Orly (1992). "Formal analysis in a register of Israeli Hebrew using a predicative grammar." In Hebrew Computational Linguistics, edited by U. Ornan, G. Arieli, and I. Doron, 88-102. Israel Ministry of Science and Technology. find
4. Bentur, Esther; Angel, Aviela; and Segev, D. (1992). "Computerized analysis of Hebrew words." In Hebrew Computerized Linguistics. Israel Ministry of Science and Technology. find
5. Choueka, Yaacov (1980). "Full-text systems and research in the humanities." Computers and the Humanities, 14, 153-169. find
6. Choueka, Yaacov, and Lusignan, S. (1985). "Disambiguation by short contexts." Computers and the Humanities, 19, 147-157. find
7. Church, Kenneth W. (1988). "A stochastic parts program and noun phrase parser for unrestricted text." In Proceedings, ACL Conference on Applied Natural Language Processing, 136-143. find
8. Church, Kenneth W. (1992). "Current practice in part of speech tagging and suggestions for the future." In For Henry Ku~era, edited by A. W. Mackie, T. K. McAuley, and C. Simmons, 13-48. Michigan Slavic Publications, University of Michigan. find
9. Cutting, Doug; Kupiec, Julian; Pedersen, Jan; and Sibun, Penelope (1992). "A practical part-of-speech tagger." In Proceedings, ACL Conference on Applied Natural Language Processing, 133-140. find
10. Dagan, Ido, and Itai, Alon (1994). "Word sense disambiguation using a second language monolingual corpus." Computational Linguistics, 20(4), 563-596. find
11. Dagan, Ido; Itai, Alon; and Schwall, Ulrike (1991). "Two languages are more informative than one." In Proceedings, Annual Meeting of the ACL, 130-137. find
12. DeRose, Steven J. (1988). "Grammatical category disambiguation by statistical optimization." Computational Linguistics, 14(1), 31-39. find
13. ISO Conference (1994). Conversion of Hebrew characters into Latin characters, Part 3: Phonemic Conversion (ISO-259-3). Stockholm: Ornan, Uzzi. find
14. Jelinek, Frederick; Mercer, Robert L.; and Roukos, Salim (1992). "Principles of lexical language modeling for speech recognition." In Advances in Speech Signal Processing, edited by Sadaoki Furui and M. Mohan Sondhi, 651-699. Marcel Dekker, Inc. find
15. Karlsson, Fred (1990). "Constraint grammar as a framework for parsing running text." In Proceedings, 13 th International Conference on Computational Linguistics (COLING-90). 3:168-179. find
16. Kupiec, Julian (1992). "Robust part-of-speech tagging using a hidden Markov model." Computer Speech and Language, 6, 225-242. find
17. Levinger, Moshe (1992). Morphological disambiguation. Master's thesis, Computer Science Department, Technion--Israel Institute of Technology. find
18. Ornan, Uzzi (1986). "Phonemic script: A central vehicle for processing NL--the case of Hebrew." Technical Report 88-181, IBM Scientific Center, Haifa. find
19. Ornan, Uzzi (1987). "Computerized index to decisions of supreme court by phonemic script." Mishpatim, 17, 15-24. find
20. Ornan, Uzzi (1991). "Theoretical gemination in Israeli Hebrew." In Semitic Studies in Honor of Wolf Leslau, edited by Alan S. Kaye, Otto Harrassowitz. 1158-1168. find
21. Pines, Ronny (1975). "Ambiguity on the syntactic level." In Rosen Memorial Volume, edited by Ben-Zion Fischler and Uzzi Ornan, 74-85. Council of the Teaching of Hebrew. find
22. Yarowsky, David (1992). "Word sense disambiguation using statistical models of Roget's categories trained on large corpora." In Proceedings, 14th International Conference on Computational Linguistics (COLING-92). 454-460. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/cl-1995-levinger-learning-morpho-lexical-probabilities-untagged/