Unsupervised Concept Discovery In Hebrew Using Simple Unsupervised Word Prefix Segmentation for Hebrew and Arabic
What it is about
Hebrew and Arabic attach function words (and, in, the, that) to the next word, so word-based pattern methods for concept discovery miss the patterns that use "and" or "or". The authors split off the closed set of prefixes without any annotation. A word is split only when a rank, which compares the frequency of the prefixed form with the frequency of the stem and a short-word factor, is below a threshold; a trigram model over morphemes, found by dynamic programming, picks the best split of each sentence. On a 24,000-word hand-segmented Hebrew sample the method reaches F 0.851 and accuracy 0.893, against 0.523 for a baseline and 0.658 for Morfessor. On a 130 MB corpus, segmentation raises the number of Hebrew concepts from 36 to 148 and their average size from 2.9 to 4.1, with one junk word; without it, about ten times as much data is needed for similar results.
Ideas
- I1. A rank-and-threshold split for unseen prefixed forms in
tools/hbo: keep the closed prefix list the segmenter already uses, and score each candidate split by the frequency ratio of the prefixed form to its stem, with a length factor for short words. Test it on the unseen-word path against a hand-segmented sample, and report F and accuracy. - I2. Use the prefix "w" (and) as a coordination signal: a pattern "x w y" with the content words on both sides is a symmetric relation. Test whether the coordination reading helps the event chain or the coordinated-NP attachment in Syntax: the clause and the sentence.
- I3. Measure how much corpus Mova's lexicon needs with and without prefix splitting, on the same corpus size, by counting distinct lexemes. This is a cheap check of the segmenter's value for
world.
For Mova
I1 is a fallback for unseen forms in tools/hbo (the segmenter; see 13. Biblical Hebrew in Mova: the tools). I2 belongs to Syntax: the clause and the sentence (coordination) and Prepositions, prefixes, conjunctions and particles (the prefix "w"). I3 is a measurement for 12. Mova and Modern Hebrew: where it stands, what it would need, run on the corpus outside the repo. The method is unsupervised and needs only the prefix list, which fits Mova's rules. The paper notes that the "bbyt" form has two readings; keep it as a test case for tools/hbo.
Cite
@inproceedings{dinur2009unsupervised,
title = {{Unsupervised Concept Discovery In Hebrew Using Simple Unsupervised Word Prefix Segmentation for Hebrew and Arabic}},
author = {Dinur, Elad and Davidov, Dmitry and Rappoport, Ari},
booktitle = {Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages},
year = {2009},
doi = {10.3115/1621774.1621782},
url = {https://doi.org/10.3115/1621774.1621782}
}References
- Meni Adler, Michael Elhadad, 2006. An Unsupervised Morpheme-Based HMM for Hebrew Morphological Disambiguation. ACL ’06. find our card
- Roy Bar-Haim, Khalil Simaan, Yoad Winter, 2005. Choosing an Optimal Architecture for Segmentation and POS-Tagging of Modern Hebrew. ACL Workshop on Computational Approaches to Semitic Languages ’05. find our card
- Mathias Creutz, 2003. Unsupervised Segmentation of Words Using Prior Distributions of Morph Length and Frequency. ACL ’03. find
- Mathias Creutz and Krista Lagus, 2005. Unsupervised Morpheme Segmentation and Morphology Induction from Text Corpora Using Morfessor 1.0. In Computer and Information Science, Report A81, Helsinki University of Technology. find
- Sajib Dasgupta and Vincent Ng, 2007. High Performance, Language-Independent Morphological Segmentation. NAACL/HLT ’07. find
- Dmitry Davidov, Ari Rappoport, 2006. Efficient Unsupervised Discovery of Word Categories Using Symmetric Patterns and High Frequency Words. ACL ’06. find
- Dmitry Davidov, Ari Rappoport, Moshe Koppel, 2007. Fully Unsupervised Discovery of Concept-Specific Relationships by Web Mining. ACL ’07. find
- Dmitry Davidov, Ari Rappoport, 2008a. Classification of Semantic Relationships between Nominals Using Pattern Clusters. ACL ’08. find
- Dmitry Davidov, Ari Rappoport, 2008b. Unsupervised Discovery of Generic Relationships Using Pattern Clusters and its Evaluation by Automatically Generated SAT Analogy Questions. ACL ’08. find
- Vera Demberg, 2007. A Language-Independent Unsupervised Model for Morphological Segmentation. ACL ’07. find
- Beate Dorow, Dominic Widdows, Katarina Ling, Jean-Pierre Eckmann, Danilo Sergi, Elisha Moses, 2005. Using Curvature and Markov Clustering in Graphs for Lexical Acquisition and Word Sense Discrimination. MEANING ’05. find
- Samarth Keshava, Emily Pitler, 2006. A Simpler, Intuitive Approach to Morpheme Induction. In Proceedings of 2nd Pascal Challenges Workshop, Venice, Italy. find
- Young-Suk Lee, Kishore Papineni, Salim Roukos, Ossama Emam, Hany Hassan, 2003. Language Model Based Arabic Word Segmentation. ACL ’03. find
- Moshe Levinger, Uzzi Ornan, Alon Itai, 1995. Learning Morpho-Lexical Probabilities from an Untagged Corpus with an Application to Hebrew. Comput. Linguistics, 21:383:404. find our card
- Dekang Lin, 1998. Automatic Retrieval and Clustering of Similar Words. COLING ’98. find
- Uzzi Ornan, 2005. Hebrew in Latin Script. Lesonenu, LXIV:137:151 (in Hebrew). find
- Patrick Pantel, Dekang Lin, 2002. Discovering Word Senses from Text. SIGKDD ’02. find
- Patrick Pantel, Deepak Ravichandran, Eduard Hovy, 2004. Towards Terascale Knowledge Acquisition. COLING ’04. find
- Fernando Pereira, Naftali Tishby, Lillian Lee, 1993. Distributional Clustering of English Words. ACL ’93. find
- Benjamin Snyder, Regina Bazilay, 2008. Unsupervised Multilingual Learning for Morphological Segmentation. ACL/HLT ’08. find
- Andreas Stolcke, 2002. SRILM – an Extensible Language Modeling Toolkit. ICSLP, pages 901-904, Denver, Colorado. find
- Dominic Widdows, Beate Dorow, 2002. A Graph Model for Unsupervised Lexical Acquisition. COL-ING ’02. find
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2009-dinur-unsupervised-concept-discovery-hebrew-simple/