A corpus based morphological analyzer for unvocalized modern Hebrew

What it is about

Most words in unvocalised Modern Hebrew are morphologically ambiguous, and the authors choose one analysis per token. The system has three stages. The word stage gives each analysis a probability from tag and lemma frequencies in a 10-million-word untagged newspaper corpus, with the tags assumed independent. The pair stage applies Brill-style transformation rules, learned from a 4,900-word tagged corpus, that adjust the scores of neighbouring analyses rather than overwrite them. The sentence stage uses a hand-written grammar of about 150 rules and a dynamic-programming cover of the sentence. On a 468-token test set the error falls from 36.0% (all analyses equiprobable) to 14.0% with the word stage alone, 7.0% with word and pair, 5.3% with word and sentence, and 3.8% with all three, about 96% accuracy. The authors name unknown words, which the system treats as proper nouns, and the lack of idiom handling as the main error sources.

Ideas

I1. Word stage as a baseline for the tools/hbo segmenter: rank each split candidate (from the pre and lex tables) by the product of independently estimated tag and lemma probabilities from a large unannotated corpus, and keep the top one. This is a non-neural tie-breaker for unpointed Hebrew.
I2. Keep scores, not decisions: the pair-stage rules add a fixed increment to an analysis score and renormalise, and the choice is made later. For tools/hbo, pass the segmenter's scored alternatives on to gparse instead of one split.
I3. A clause-cover check: pick the fewest clauses covering the sentence, with a fixed cost per clause, solved as a shortest path. Use it as a sanity measure of clause boundaries over treebanks/hbo/ptnk, for Syntax: the clause and the sentence.
I4. Report error by stage and the learning curve (rules learned against training size), on a held-out token set. Use the paper's table layout as the template for the unpointed-mode measurement in tools/hbo.

For Mova

I1 and I2 go into tools/hbo (segmenter ranking and score hand-off to gparse). I3 goes into Syntax: the clause and the sentence, with a check over treebanks/hbo/ptnk. I4 is the measurement format for tools/hbo unpointed mode. The unknown-words-as-proper-nouns error is a warning for the tools/hbo lexicon fallback.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{itai2003corpus,
  title = {{A corpus based morphological analyzer for unvocalized modern Hebrew}},
  author = {Itai, Alon and Segal, Erel},
  booktitle = {Workshop on Machine Translation for Semitic languages: issues and approaches},
  year = {2003},
  url = {https://aclanthology.org/2003.mtsummit-semit.9/}
}

References

13 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Steven Abney. 1996. Partial Parsing via Finite-State Cascades. In J. of Natural Language Engineering, 2(4), pp. 337-344. find
Orly Albeck. 1992. Formal analysis by a restriction grammar on one of the stages of Modern Hebrew. Computerized analysis of Hebrew words. In Israel Ministry of Science and Technology Symposium. (In Hebrew.) find
Alon Altman. 2002. Private communication. find
Esther Ben-Tur, Aviela Angel, Danit Ben-Ari, and Alon Lavie. 1992. Computerized analysis of Hebrew words. In Israel Ministry of Science and Technology Symposium. (In Hebrew.) find
Eric Brill. 1995. Transformation-based error-driven learning and natural language processing: a case study in part-of-speech tagging. In Computational Linguistics, 21, pp. 543-565. find
David Carmel and Yoëlle Maarek. 1999. Morphological disambiguation for Hebrew search systems. In Next Generation Information Technologies and Systems, NGITS ’99, Springer LNCS 1649, pp. 312-325. find
Yaacov Choueka and S. Lusignan. 1985. Disambiguation by short context. In Computers and the Humanities, 19(3). find
Yaacov Choueka. 2002. Rav Millim. Technical report, The Center for Educational Technology in Israel. (http://www.cet.ac.il/rav-milim/) ↗
Kenneth W. Church. 1988. A stochastic parts program and noun phrase parser for unrestricted text. In ANLP, 2, pp. 136-143. find
Steven J. DeRose. 1988. Grammatical category disambiguation by statistical optimization, In Computational Linguistics, 14, pp. 31-39. find
ISO. 1999. “Information and documentation – Conversion of Hebrew characters into Latin characters – Part 3: Phonemic Conversion, ISO/FDIS 259-3: (E). André Kempe Probabilistic parsing with feature structures. find
Moshe Levinger, Uzzi Ornan and Alon Itai. 1995. Morphological disambiguation in Hebrew using a priori probabilities, Computational Linguistics 21, pp. 383-404. find
Uzzi Ornan and Michael Katz. 1994. A new program for Hebrew index based on the phonemic script, Technical Report #LCL 94-7, find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2003-itai-corpus-based-morphological-analyzer-unvocalized/