A Characterwise Windowed Approach to Hebrew Morphological Segmentation
What it is about
Zeldes treats Hebrew segmentation as pure splitting: for every character of a space-delimited token he decides whether a boundary follows, with no morphological analysis. Each decision uses a window of two characters on either side, the first and last letters of the neighbouring tokens, the position and length of each unit, a flag for letters that may be vowel carriers (w, y, h, aleph), and lexicon-lookup features (the coarse POS tags attested for each substring around the target). Extra lexicon entries for foreign names (about 2% of the lexicon) were added, and a boundary is forbidden where no prefix or suffix of that shape occurs in training. Among several learners, Extra Trees gave the best result on the SPMRL benchmark (about 98.2% of tokens perfectly segmented, against 94.3% for yap) and on a new hand-segmented Wikipedia test (about 97.6%, against 92.7%). Its main gain is precision: fewer spurious splits.
Ideas
- I1. Write
tools/hbo's boundary decisions as a character-wise classifier over the existinglexandpretables: for each position, the features are the window letters, the table tags of each substring, and the lengths. The features are rules a reader can check one by one; a learner is only added when a rule-set falls short. - I2. Use the vowel-carrier flag (w, y, h, aleph) as an explicit feature in unpointed mode, since the letters that mark vowels are the same letters that mark consonants. The paper shows this flag helps in its final model.
- I3. A guard, cheap and deterministic: no boundary between letters and digits, and no prefix or suffix that is not attested in the training data. Measure it with and without the guard, and add a second out-of-domain test set for the Hebrew segmenter, scored on the share of perfectly segmented tokens and on boundary precision.
For Mova
I1 and I3 are design for tools/hbo (segmenter tables lex, pre). I2 belongs in 1. Script and pointing (matres lectionis) and in 2. Script and spelling. The modern-reader plan in 12. Mova and Modern Hebrew: where it stands, what it would need can cite the paper's error classes (morphologically ambiguous known words; out-of-vocabulary names) as its test categories.
Cite
@inproceedings{zeldes2018characterwise,
title = {{A Characterwise Windowed Approach to Hebrew Morphological Segmentation}},
author = {Zeldes, Amir},
booktitle = {Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology},
year = {2018},
doi = {10.18653/v1/w18-5811},
url = {https://doi.org/10.18653/v1/w18-5811}
}References
- Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, pages 265–283, Savannah, GA. find
- Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the NAACL 2016: Demonstrations, pages 11–16, San Diego, CA. find our card
- Meni Adler and Michael Elhadad. 2006. An unsupervised morpheme-based HMM for Hebrew morphological disambiguation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 665–672, Sydney. find our card
- Rami Al-Rfou, Vivek Kulkarni, Bryan Perozzi, and Steven Skiena. 2015. Polyglot-NER: Massive multilingual named entity recognition. In Proceedings of the 2015 SIAM International Conference on Data Mining, Vancouver, Canada. find
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2018-zeldes-characterwise-windowed-approach-hebrew-morphological/