Evaluating Word Embeddings on Low-Resource Languages

What it is about

The analogy task (Mikolov et al.) is the usual way to tune word-embedding hyperparameters, but it needs very large training corpora and presumes analogies exist in the language. The authors propose two set-based tests, OddOneOut and Topk, which only ask whether a word lies near a category, and they generate the category test sets automatically from Wikidata in 581 languages. On English Wikipedia subsets, the analogy score stays at zero until the corpus is large, while Topk and OddOneOut improve earlier. For emoji embeddings they reproduce the choice of learning rate made by a bespoke emoji metric. They then train embeddings for 16 extinct languages, the largest being Ancient Hebrew (41 million tokens) and the smallest Old Gujarati (1,813 tokens). OddOneOut is estimated by sampling 1,000 (set, word) pairs, and the recommended k is 3.

Ideas

• I1. A Topk gate for lexicon-level embeddings: for a set of Hebrew categories (biblical figures, plants, body parts) take k=3 nearest neighbours of each word and score how many fall in the same category; use the score to choose window and dimension for any embedding the SLM builds from a small corpus.
• I2. Sample OddOneOut instead of enumerating it. Use a fixed seed and BTreeMap ordering of the categories, so the sampled pairs and the score are identical on every run, as the determinism rule asks.
• I3. Build the category sets from the Wikidata data we already hold (outside the repo, in $MOVA_DATA) for the Biblical and Mishnaic lexicon, and check them against macula dict lemmas before use.

For Mova

I1 and I2 are a measurement for the word-similarity side of tools/hbo and world; they belong in a test, not in the parser. I3 feeds the lexicon chapter The lexicon. The paper's main point, that small corpora need set-based evaluation, matches our rule of measuring before claiming a gain.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.eval4nlp-1.17

@inproceedings{stringham2020evaluating,
  title = {{Evaluating Word Embeddings on Low-Resource Languages}},
  author = {Stringham, Nathan and Izbicki, Mike},
  booktitle = {Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems},
  year = {2020},
  doi = {10.18653/v1/2020.eval4nlp-1.17},
  url = {https://doi.org/10.18653/v1/2020.eval4nlp-1.17}
}

References

33 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Oliver Adams, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. 2017. Cross-lingual word embeddings for low-resource language modeling. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 937–947, Valencia, Spain. Association for Computational Linguistics. find
2. Wei Ai, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei. 2017. Untangling emoji popularity through semantic embeddings. In Eleventh International AAAI Conference on Web and Social Media. find
3. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. arXiv preprint arXiv:1708.00524. ↗
4. Ziad Al-Halah, Andrew Aitken, Wenzhe Shi, and Jose Caballero. 2019. Smile, be happy:) emoji embedding for visual sentiment analysis. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0–0. find
5. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the 10th international conference on World Wide Web, pages 406– 414. find
6. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual nlp. arXiv preprint arXiv:1307.1662. ↗
7. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. arXiv preprint arXiv:1802.06893. ↗
8. Hosein Azarbonyad, Mostafa Dehghani, Kaspar Beelen, Alexandra Arkut, Maarten Marx, and Jaap Kamps. 2017. Words are malleable: Computing semantic shifts in political and media discourse. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 1509–1518. find
9. Vishwani Gupta, Sven Giesselbach, Stefan Rüping, and Christian Bauckhage. 2019. Improving word embeddings using kernel PCA. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 200–208, Florence, Italy. Association for Computational Linguistics. find
10. Francesco Barbieri, Miguel Ballesteros, and Horacio Saggion. 2017. Are emojis predictable? arXiv preprint arXiv:1702.07285. ↗
11. James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. The Journal of Machine Learning Research, 13(1):281–305. find
12. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 136–145. find
13. Christian Buck, Kenneth Heafield, and Bas Van Ooyen. 2014. N-gram counts and language models from the common crawl. In LREC, volume 2, page 4. Citeseer. find
14. Jana Chamonikolasová. 2014. Middle english. A concise history of English, pages 46–61. find
15. Baitong Chen, Satoshi Tsutsui, Ying Ding, and Feicheng Ma. 2017. Understanding the topic evolution in a scientific domain: An exploratory study for the field of information retrieval. Journal of Informetrics, 11(4):1175–1189. find
16. William L Hamilton, Jure Leskovec, and Dan Jurafsky. 2016a. Cultural shift or linguistic drift? comparing two computational measures of semantic change. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2016, page 2116. NIH Public Access. find
17. William L Hamilton, Jure Leskovec, and Dan Jurafsky. 2016b. Diachronic word embeddings reveal statistical laws of semantic change. arXiv preprint arXiv:1605.09096. ↗
18. Chao Jiang, Hsiang-Fu Yu, Cho-Jui Hsieh, and Kai-Wei Chang. 2018. Learningword embeddings for low-resource languages by pu learning. arXiv preprint arXiv:1805.03366. ↗
19. Kyle P. Johnson. 2014. CLTK: The Classical Language Toolkit. https://github.com/cltk/cltk . ↗
20. Jakob Jungmaier, Nora Kassner, and Benjamin Roth. 2020. Dirichlet-smoothed word embeddings for low-resource settings. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 3560–3565, Marseille, France. European Language Resources Association. find
21. Haim Dubossarsky, Daphna Weinshall, and Eitan Grossman. 2017. Outta control: Laws of semantic change and inherent biases in word representation models. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 1136–1145. find
22. Katharina Kann, Kyunghyun Cho, and Samuel R. Bowman. 2019. Towards realistic practices in low-resource natural language processing: The development set. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3342–3349, Hong Kong, China. Association for Computational Linguistics. find
23. Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko Bošnjak, and Sebastian Riedel. 2016. emoji2vec: Learning emoji representations from their description. arXiv preprint arXiv:1609.08359. ↗
24. Austin C Kozlowski, Matt Taddy, and James A Evans. 2019. The geometry of culture: Analyzing the meanings of class through word embeddings. American Sociological Review, 84(5):905–949. find
25. Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detection of linguistic change. In Proceedings of the 24th International Conference on World Wide Web, pages 625–635. find
26. Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal. 2018. Diachronic word embeddings and semantic shifts: a survey. arXiv preprint arXiv:1806.03537. ↗
27. Shangsong Liang, Xiangliang Zhang, Zhaochun Ren, and Evangelos Kanoulas. 2018. Dynamic embeddings for user profiling in twitter. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1764– 1773. find
28. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
29. Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. 2011. A word at a time: computing word relatedness using temporal semantic analysis. In Proceedings of the 20th international conference on World wide web, pages 337–346. find
30. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45– 50, Valletta, Malta. ELRA. http://is.muni.cz/ publication/884893/en . ↗
31. Terrence Szymanski. 2017. Temporal word analogies: Identifying lexical replacement with diachronic word embeddings. In Proceedings of the 55th annual meeting of the association for computational linguistics (volume 2: short papers), pages 448–453. find
32. Xuri Tang. 2018. A state-of-the-art of semantic change computation. arXiv preprint arXiv:1801.09872. ↗
33. Sanjaya Wijeratne, Lakshika Balasuriya, Amit Sheth, and Derek Doran. 2017. A semantics-based measure of emoji similarity. In Proceedings of the International Conference on Web Intelligence, pages 646–653. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2020-stringham-evaluating-word-embeddings-low-resource/