Mova0.5 pre-release⌕
☰
home / docs / papers / Ancient Hebrew / Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All

Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All

What it is about

The authors train AlephBERTGimmel, a Modern Hebrew BERT with a 128K word-piece vocabulary (earlier models had 2.4K–52K), on the same ~2 billion words as AlephBERT. They also merge the UD, SPMRL and NEMO annotations of the same Hebrew texts into one consistent benchmark (UD-NEMO). The large-vocabulary model reaches the best results on segmentation, POS, morphological features, named entities and sentiment. Their analysis shows that the more pieces a word is split into, the worse every model does on it, so a vocabulary that keeps more words whole helps most, especially on semantic tasks.

Ideas

For Mova

The model is a neural Modern Hebrew model and cannot be part of Mova at answer time; it could at most serve as a teacher or a baseline. I1 bears on the word segmentation described in 13. Biblical Hebrew in Mova: the tools §4 (tokens) and on the Modern Hebrew manual (languages/he/); I2 on the measurements in 12-biblical-mishnaic-modern.md ("why a Modern Hebrew parser misreads the Bible").

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2211.15199

@misc{gueta2022large,
  title = {{Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All}},
  author = {Gueta, Eylon and Shmidman, Avi and Shmidman, Shaltiel and Shmidman, Cheyn Shmuel and Guedalia, Joshua and Koppel, Moshe and Bareket, Dan and Seker, Amit and Tsarfaty, Reut},
  year = {2022},
  eprint = {2211.15199},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.15199}
}

References

12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Antoine Nzeyimana and Andre Niyongabo Rubungo. 2022. KinyaBERT: a morphology-aware Kinyarwanda language model. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5347 5363, Dublin, Ireland. Association for Compu tational Linguistics. find
  2. Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from modern hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242 2252. find our card
  3. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. find
  4. Dan Bareket and Reut Tsarfaty. 2021. Neural modeling for named entities and morphology (nemo^2). Transactions of the Association for Computational Linguistics, 9:909–928. find
  5. Shoval Sade, Amit Seker, and Reut Tsarfaty. 2018. The hebrew universal dependency treebank: Past present and future. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 133 143. find our card
  6. Idan Brusilovsky and Reut Tsarfaty. 2022. Neural token segmentation for high token-internal complexity. arXiv preprint arXiv:2203.10845. ↗
  7. Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Galletebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yannick Versley, Veronika Vincze, Marcin Woliński, Alina Wróblewska, and Eric Villemonte de la Clergerie. 2013. Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 146–182, Seattle, Washington, USA. Association for Computa tional Linguistics. find
  8. Avihay Chriqui and Inbal Yahav. 2021. Hebert & hebemo: a hebrew bert model and a tool for polarity analysis and emotion recognition. find our card
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  10. Omri Keren, Tal Avinari, Reut Tsarfaty, and Omer Levy. 2022. Breaking character: Are subwords good enough for mrls after all? find
  11. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert: A hebrew large pre-trained language model to start-off your hebrew NLP application with. CoRR, abs/2104.04052. ↗ our card
  12. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State of the-art natural language processing. CoRR, abs/1910.03771. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/hbo-2022-gueta-large-pre-trained-models-extra/