Morphological Word Embeddings for Arabic Neural Machine Translation in Low-Resource Settings

What it is about

Standard word embeddings are poor for rare words, and Arabic inflects heavily. The authors change the word2vec skip-gram objective so that the context is predicted from the word vector concatenated with a weighted average of lemma vectors, with lemmas from MADAMIRA. They test the embeddings on an Arabic word similarity set (a WordSim353 variant) and as the initial source-side embeddings of a small Arabic-to-English NMT system, trained on about 2.9 million source tokens of TED subtitles. The word+lemma embeddings beat word2vec and fastText on similarity when the lemma part is used. In NMT, averaged over three runs, fastText (unfixed) reaches 29.10 BLEU, the morphological embeddings (unfixed) 28.76, word2vec (unfixed) 28.38, and BPE 27.80, against 26.55 for random initialization. The morphological embeddings are not the best in that test.

Ideas

I1. Lemma back-off for unseen forms: when a surface form is not in the world, look up its lemma's entry instead. The paper's out-of-vocabulary handling rests on the lemma part of the vector. For Mova this is a deterministic table: form to lemma, from macula dict or the segmenter, and a fallback to the lemma's entry in world.
I2. Keep both fields. The paper finds that predicting from the lemma alone does worse than the word and lemma together. So the tree and the world should keep the surface form and the lemma as two fields, and each rule should state which field it reads.
I3. A coverage report for every test text: the paper gives the share of types and tokens unseen in training (9% of types, 3% of tokens for its test). Mova should report the same two shares for each hbo or arc test text, so that a lemma back-off effect is measured where it matters, in The lexicon.

For Mova

I1 goes into world and the lemma fallback of tools/macula dict, with its test in The lexicon. I2 is a rule for the tree and world readers (prag, world). I3 is a coverage table for every test text. The embeddings are not reused: Mova works from rules and tables, and the lemma idea carries over as a table.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-1201

@inproceedings{shapiro2018morphological,
  title = {{Morphological Word Embeddings for Arabic Neural Machine Translation in Low-Resource Settings}},
  author = {Shapiro, Pamela and Duh, Kevin},
  booktitle = {Proceedings of the Second Workshop on Subword/Character LEvel Models},
  year = {2018},
  doi = {10.18653/v1/w18-1201},
  url = {https://doi.org/10.18653/v1/w18-1201}
}

References

48 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Amjad Almahairi, Kyunghyun Cho, Nizar Habash, and Aaron Courville. 2016. First result on arabic neural machine translation. arXiv preprint arXiv:1606.02680 . ↗
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . ↗
Ryan Cotterell and Kevin Duh. 2017. Low-resource named entity recognition with cross-lingual, character-level neural conditional random fields. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers). Asian Federation of Natural Language Processing, Taipei, Taiwan, pages 91– 96. http://www.aclweb.org/anthology/ I17-2016. ↗
Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based parsing by modeling characters instead of words with lstms. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 349–359. http://aclweb.org/ anthology/D15-1041. ↗
Ryan Cotterell and Hinrich Schütze. 2015. Morphological word-embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pages 1287–1292. https://doi.org/10.3115/v1/N15-1140. ↗
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 861–872. https://doi.org/10.18653/v1/P17-1080. ↗
Ryan Cotterell, Hinrich Schütze, and Jason Eisner. 2016. Morphological smoothing and extrapolation of word embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 1651–1660. https://doi.org/10.18653/ v1/P16-1156. ↗
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association of Computational Linguistics 5:135–146. http://www.aclweb.org/ anthology/Q17-1010. ↗
Mathias Creutz and Krista Lagus. 2007. Unsupervised models for morpheme segmentation and morphology learning. ACM Transactions on Speech and Language Processing (TSLP) 4(1):3. find
Jan Botha and Phil Blunsom. 2014. Compositional morphology for word representations and language modeling. In Proceedings of the International Conference on Machine Learning (ICML). find
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, and Stephan Vogel. 2017. Understanding and improving morphological learning in the neural machine translation decoder. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers). volume 1, pages 142–151. find
Tim Buckwalter. 2002. Arabic transliteration. URL http://www.qamus. org/transliteration. htm . ↗
Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. Wit 3 : Web inventory of transcribed and translated talks. In Proceedings of the 16 th Conference of the European Association for Machine Translation (EAMT). Trento, Italy, pages 261–268. find
Mattia A Di Gangi and Federico Marcello. 2017. Can monolingual embeddings improve neural machine translation? . find
Xinxiong Chen, Lei Xu, Zhiyuan Liu, Maosong Sun, and Huanbo Luan. 2015. Joint learning of character and word embeddings. https://www.aaai.org/ocs/index. php/IJCAI/IJCAI15/paper/view/11000. ↗
Cicero dos Santos and Victor Guimarães. 2015. Boosting named entity recognition with neural character Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions. Association for Computational Linguistics, pages 177–180. http://www.aclweb.org/anthology/P07-2045. embeddings. In Proceedings of the Fifth Named Entity Workshop. Association for Computational Linguistics, Beijing, China, pages 25–33. http:// www.aclweb.org/anthology/W15-3904. ↗
Cicero Nogueira dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In ICML. find
Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, and Chris Dyer. 2016. Problems with evaluation of word embeddings using word similarity tasks. https://doi.org/10.18653/v1/ W16-2506. ↗
Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation. Association for Computational Linguistics, Vancouver, pages 28–39. http://www.aclweb.org/anthology/W17-3204. ↗
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the 10th international conference on World Wide Web. ACM, pages 406–414. find
Angeliki Lazaridou, Marco Marelli, Roberto Zamparelli, and Marco Baroni. 2013. Compositional-ly derived representations of morphologically complex words in distributional semantics. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Sofia, Bulgaria, pages 1517–1526. http://www.aclweb.org/anthology/P13-1149. ↗
Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pages 866–875. https://doi.org/10.18653/v1/ N16-1101. ↗
Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics 5:365–378. https://transacl.org/ojs/ index.php/tacl/article/view/1051. ↗
Samer Hassan and Rada Mihalcea. 2009. Cross-lingual semantic relatedness using encyclopedic knowledge. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 1192–1201. http://www.aclweb.org/ anthology/D09-1124. ↗
Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015a. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 1520–1530. http://aclweb.org/ anthology/D15-1176. ↗
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Advances in Neural Information Processing Systems. pages 820–828. find
Aizhan Imankulova, Takayuki Sato, and Mamoru Komachi. 2017. Improving low-resource neural machine translation with filtered pseudo-parallel corpus. In Proceedings of the 4th Workshop on Asian Translation (WAT2017). Asian Federation of Natural Language Processing, Taipei, Taiwan, pages 70– 78. http://www.aclweb.org/anthology/ W17-5704. ↗
Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. 2015b. Character-based neural machine translation. arXiv preprint arXiv:1511.04586 . ↗
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 1412–1421. https:// doi.org/10.18653/v1/D15-1166. ↗
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In AAAI. pages 2741–2749. find
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations. Association for Computational Linguistics, pages 67– 72. http://www.aclweb.org/anthology/ P17-4012. ↗
Thang Luong, Richard Socher, and Christopher Manning. 2013. Better word representations with recursive neural networks for morphology. http:// www.aclweb.org/anthology/W13-3512. ↗
Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The penn arabic treebank: Building a large-scale annotated arabic corpus. In NEMLAR conference on Arabic language resources and tools. volume 27, pages 466–467. the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, pages 601–607. https://doi.org/10.18653/v1/ P17-2095. ↗ our card
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 . ↗
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 86–96. https://doi.org/10.18653/v1/ P16-1009. ↗
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. pages 3111–3119. find
Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of the 24th international conference on Machine learning. ACM, pages 641–648. find
Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource, related languages for neural machine translation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers). Asian Federation of Natural Language Processing, Taipei, Taiwan, pages 296–301. http://www.aclweb.org/anthology/I17-2050. ↗
Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). European Language Resources Association (ELRA). http://www.aclweb.org/ anthology/L14-1479. ↗ our card
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 1715–1725. https://doi.org/10.18653/ v1/P16-1162. ↗
Henning Sperr, Jan Niehues, and Alex Waibel. 2013. Letter n-gram-based input encoding for continuous space language models. In Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality. Association for Computational Linguistics, Sofia, Bulgaria, pages 30– 39. http://www.aclweb.org/anthology/ W13-3204. ↗
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. pages 3104–3112. find
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Charagram: Embedding words and sentences via character n-grams. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 1504– 1515. https://aclweb.org/anthology/ D16-1157. ↗
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pages 1532–1543. https://doi.org/ 10.3115/v1/D14-1162. ↗
Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 . ↗
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 1568–1575. https:// doi.org/10.18653/v1/D16-1163. ↗
Siyu Qiu, Qing Cui, Jiang Bian, Bin Gao, and Tie-Yan Liu. 2014. Co-learning of word representations and morpheme representations. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. Dublin City University and Association for Computational Linguistics, Dublin, Ireland, pages 141–150. http://www.aclweb.org/ anthology/C14-1015. ↗
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, Ahmed Abdelali, Yonatan Belinkov, and Stephan Vogel. 2017. Challenging language-dependent segmentation for arabic: An application to machine translation and part-of-speech tagging. In Proceedings of find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2018-shapiro-morphological-word-embeddings-arabic-neural/