Constrained Sequence-to-sequence Semitic Root Extraction for Enriching Word Embeddings

What it is about

The authors extract the root of a Semitic word with a sequence-to-sequence model (a BiGRU encoder and a GRU decoder over characters). The novelty is constrained beam search: a trie built from the dictionary of valid roots prunes every decoding step, so the output is always a real root. The model is compared with multiclass classifiers, unconstrained seq2seq models and rule-based stemmers. Five-fold accuracy is 98.8% on a Hebrew word-root list of about 11,500 words (roughly 500 roots, Wiktionary-derived) and 83.2% on an Arabic list of 140K words (11K roots). On rare Arabic roots (three or fewer training occurrences) and on unseen roots the constrained model still does best. The root is then used to split words into prefix, stem and suffix slots, and these slots enrich FastText-style embeddings, which helps on morphemic analogies and word similarity.

Ideas

• I1. Trie-constrained output as a plain lookup: a candidate root is accepted only if it is in the root table, and otherwise the analyser keeps all candidates and reports the word as unknown. For Hebrew, with a few hundred roots, this is a finite-state check that costs nothing and can be explained.
• I2. Stem and slot rule: the stem is the shortest contiguous span that contains the root in order; letters outside it are prefixes and suffixes, and letters that fall inside the gaps of the root are infixes. For Hebrew, write the analogous templates from the grammar (the binyanim and the weak-verb patterns), not by copying the Arabic slots.
• I3. A test set for root extraction: split Hebrew roots by frequency (rare, unseen) and report accuracy on each group, as the paper does, before adding any learned component to tools/hbo.

For Mova

I1 is a rule for The verb and The Noun, the Adjective and the Numeral (root check); I2 belongs in The weak verbs and The Noun, the Adjective and the Numeral; I3 is a measure for tools/hbo. The Wiktionary-derived root list is CC BY-SA data: it may be tested against and learned from, but it is not carried into a release.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4610

@inproceedings{elkishky2019constrained,
  title = {{Constrained Sequence-to-sequence Semitic Root Extraction for Enriching Word Embeddings}},
  author = {El-Kishky, Ahmed and Fu, Xingyu and Addawood, Aseel and Sobh, Nahil and Voss, Clare and Han, Jiawei},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Workshop},
  year = {2019},
  doi = {10.18653/v1/w19-4610},
  url = {https://doi.org/10.18653/v1/w19-4610}
}

References

30 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yasir Alhanini and Mohd Juzaiddin Ab Aziz. 2011. The enhancement of arabic stemming by using light stemming and dictionary-based stemming. JSEA. find
2. Mohammed Aljlayl and Ophir Frieder. 2002. On arabic search: improving the retrieval effectiveness via a light stemming approach. In CIKM. find
3. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided open vocabulary image captioning with constrained beam search. In EMNLP. find
4. Andras Kornai. 2007. Mathematical linguistics. Springer Science & Business Media. find
5. Abderrahim Boudlal, Mohamed Ould Abdallahi Ould Bebah, Abdelhak Lakhouaja, Azzeddine Mazroui, and Abdelouafi Meziane. 2011. A markovian approach for arabic root extraction. Int. Arab J. Inf. Technol. find
6. Leah S Larkey, Lisa Ballesteros, and Margaret E Connell. 2002. Improving stemming for arabic information retrieval: light stemming and co-occurrence analysis. In ACM SIGIR. find
7. Yaacov Choueka. 1990. Mlim-a system for full, exact, on-line grammatical analysis of modern hebrew. In ICCE. find
8. Yoong Keok Lee, Aria Haghighi, and Regina Barzilay. 2011. Modeling syntactic context improves morphological segmentation. In CoNLL. find
9. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. ↗
10. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025. ↗
11. Kareem Darwish. 2002. Building a shallow arabic morphological analyzer in one day. In ACL-02 workshop on Computational approaches to semitic languages. find
12. Mohanned Momani and Jamil Faraj. 2007. A novel algorithm to extract tri-literal arabic roots. In Computer Systems and Applications, 2007. AICCSA’07. find
13. Ezra Daya, Dan Roth, and Shuly Wintner. 2004. Learning hebrew roots: Machine learning with linguistic constraints. In EMNLP. find our card
14. Armand Joulin Piotr Bojanowski, Edouard Grave and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL. find
15. Ezra Daya, Dan Roth, and Shuly Wintner. 2008. Identifying semitic roots: Machine learning with linguistic constraints. Computational Linguistics. find our card
16. Paul Rodrigues and Damir Cavar. 2007. Learning arabic morphology using statistical constraint-satisfaction models. Amsterdam studies in the theory and history of linguistic science. Series 4. find
17. Ahmed El-Kishky, Frank Xu, Aston Zhang, Stephen Macke, and Jiawei Han. 2018. Entropy-based subword mining with an application to word embeddings. In SCLeM 2018. find
18. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS. find
19. Khaled Elghamry. 2005. A constraint-based algorithm for the identification of arabic roots. In Proceedings of the Midwest Computational Linguistics Colloquium. Indiana University. Bloomington, IN. find
20. Kazem Taghva, Rania Elkhoury, and Jeffrey Coombs. 2005. Arabic stemming without a root dictionary. In Information Technology: Coding and Computing. find
21. Qussai Yaseen and Ismail Hmeidi. 2014. Extracting the roots of arabic words without removing affixes. Journal of Information Science. find
22. Mohammed Elrazzaz, Shady Elbassuoni, Khaled Shaban, and Chadi Helwe. 2017. Methodical evaluation of arabic word embeddings. In ACL. find
23. Muhammad Murtada al Zabidi and Sayyid Muhammad Murthada. 1886. Taj al-‘arus min jawahir al-qamus. Dar al-Hidayah. find
24. André Freitas, Siamak Barzegar, Juliano Efson Sales, Siegfried Handschuh, and Brian Davis. 2016. Semantic Relatedness for All (Languages): A Comparative Analysis of Multilingual Semantic Relatedness Using Machine Translation. find
25. Taha Zerrouki. 2010. Tashaphyne, arabic light stemmer/segment. find
26. Yaoyuan Zhang, Zhenxu Ye, Yansong Feng, Dongyan Zhao, and Rui Yan. 2017. A constrained sequence-to-sequence neural model for sentence simplification. arXiv preprint arXiv:1704.02312. ↗
27. Daniela Gerz, Ivan Vulić, Edoardo Ponti, Jason Naradowsky, Roi Reichart, and Anna Korhonen. 2018. Language modeling for morphologically rich languages: Character-aware modeling for word-level prediction. TACL. find
28. Nizar Y Habash. 2010. Introduction to arabic natural language processing. Synthesis Lectures on Human Language Technologies. find
29. Shereen Khoja and Roger Garside. 1999. Stemming arabic text. Lancaster, UK, Computing Department, Lancaster University. find
30. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2019-kishky-constrained-sequence-sequence-semitic-root/