Resources for processing Israeli Hebrew

What it is about

Wintner and Yona, at Haifa, describe work in progress on a set of Hebrew resources: newspaper corpora (about 1.3 million tokens from 2,500 articles, and a larger Arutz 7 set of 55,310 articles and about 6.35 million tokens), a cleaning and sentence-splitting pipeline, a tokeniser that treats any non-alphabetic character as a boundary, an XML lexicon, a finite-state morphological analyser built with XFST that produces all analyses of a word, and a plan for a Hebrew WordNet within MultiWordNet. At the time the lexicon held about 350 nouns and 50 verbs, and the verb weak paradigms were not yet tested. The planned next steps are learned lexicon extension from unseen forms, and a cascade of finite-state transducers for numerals, dates, names and phrase boundaries. The paper is a status report and gives no accuracy figures.

Ideas

I1. A finite-state cascade before the parser: small transducers for numerals, dates, addresses, names and phrase boundaries, run in order. For tools/hbo, run the same kind of cascade over unpointed text before gparse.
I2. Exceptions in the lexicon, not the rules: store only the idiosyncratic plural endings (the default is im for masculine and wt for feminine) and the secondary verb bases that are hard to generate. Check the Biblical noun and verb chapters against this split.
I3. Cleaning policy: prefer accuracy over coverage, and skip a span when unsure. Tokenise by alphabetic runs and keep the known failures (for example a multi-word place name split into two tokens) as a test set for tools/langseg and tools/ocrfix.
I4. Undotted spelling variants: one word appears in several spellings, sometimes within one document. The analyser should map the variants to one lexeme. This is a rule for 2. Script and spelling.

For Mova

I1 goes into tools/hbo pre-processing. I2 goes into The Noun, the Adjective and the Numeral and The verb. I3 is a test for tools/langseg and tools/ocrfix. I4 goes into 2. Script and spelling.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{wintner2003resources,
  title = {{Resources for processing Israeli Hebrew}},
  author = {Wintner, Shuly and Yona, Shlomo},
  booktitle = {Workshop on Machine Translation for Semitic languages: issues and approaches},
  year = {2003},
  url = {https://aclanthology.org/2003.mtsummit-semit.12/}
}

References

11 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Abney, Steven. 1996. Partial parsing via finite-state cascades. In Workshop on Robust Parsing, 8th European Summer School in Logic, Language and Information, pages 8–15, Prague, Czech Republic. find
Beesley, Kenneth R. and Lauri Karttunen. 2003. Finite-State Morphology: Xerox Tools and Techniques. CSLI, Stanford. find
Bentivogli, Luisa, Emanuele Pianta, and Christian Girardi. 2002. MultiWordNet: developing an aligned multilingual database. In Proceedings of the First International Conference on Global WordNet, Mysore, India, January. find
Bentur, Esther, Aviella Angel, and Danit Segev. 1992. Computerized analysis of Hebrew words. Hebrew Linguistics, 36:33–38, December. In Hebrew. find
Choueka, Yaacov. 1980. Computerized full-text retrieval systems and research in the humanities: The Responsa project. Computers and the Humanities, 14:153–169. find
Choueka, Yaacov. 1993. Response to “Computerized analysis of Hebrew words”. Hebrew Linguistics, 37:87, December. In Hebrew. Choueka, Yaacov and Yoni Ne’eman. “Nakdan-T”, a text vocalizer for modern Hebrew. In Proceedings of the Fourth Bar-Ilan Symposium on Foundations of Artificial Intelligence, June. find
Dahan, Hiya. 1997. Hebrew–English English– Hebrew Dictionary. Academon, Jerusalem. find
Fellbaum, Christiane, editor. 1998. WordNet: An Electronic Lexical Database. Language, Speech and Communication. MIT Press. find
Horvath, Julia and Paul Wexler. 1997. Relexification in Creole and Non-Creole Languages – With Special Attention to Haitian Creole, Modern Hebrew, Romani, and Rumanian, volume xiii of Mediterranean Language and Culture Monograph Series. Harrassowitz, Wiesbaden. find
Izre’el, Shlomo, Benjamin Hary, and Giora Rahav. 2002. Designing CoSIH: The corpus of Spoken Israeli Hebrew. International Journal of Corpus Linguistics, 6(2):171–197. find
Kaplan, Ronald M. and Martin Kay. 1994. Regular models of phonological rule systems. Computational Linguistics, 20(3):331–378, September. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2003-wintner-resources-processing-israeli-hebrew/