AlephBERT: A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application With

What it is about

The authors build AlephBERT, a BERT-base style language model for Modern Hebrew, pre-trained on about 1.9 billion words: deduplicated OSCAR, a large set of Hebrew tweets (72% of sentences), and Hebrew Wikipedia. The paper argues that Hebrew is starved of text and of shared benchmarks, so it evaluates the model across the whole Hebrew pipeline: sentiment, token-level NER, and morpheme-level tasks (segmentation, POS and morphological features) on the SPMRL and UD Hebrew treebanks. Morpheme-level outputs come from a char-based seq2seq segmenter on top of the model. AlephBERT-base beats the previous state of the art on NER (NEMO token F1 84.91 against 77.75; BMC 91.12 against 85.22) and on SPMRL morphological features (91.41 against 85.98). Segmentation F1 is already near 98 for all BERT variants, so the gains are small there.

Ideas

• I1. Make the pipeline gate explicit: segmentation first, then tagging and NER. The paper shows that pipeline NER quality tracks segmentation quality. Mova can report segmentation F1 on a held-out set as the gate for any tagger built on tools/hbo or a Modern Hebrew segmenter, before any tagging figure is claimed.
• I2. Keep entity spans at morpheme level. The paper shows that token-level NER misses entities whose boundary falls inside a token (a preposition or conjunction glued to the name). A Modern Hebrew NER output in Mova should carry the span on the segment, not the token.
• I3. Use the paper's split of the three benchmarks (SPMRL, UD Hebrew, NEMO/BMC) as the source of a held-out Modern Hebrew test set, tuned on train only and run once per release.

For Mova

I1 goes into 11. Modern Hebrew in Universal Dependencies and treebanks/he/htb (gate before tagging). I2 goes into Prepositions, prefixes, conjunctions and particles (clitics glued to names) and the plan in 12. Mova and Modern Hebrew: where it stands, what it would need. I3 is a measurement set for 12. Mova and Modern Hebrew: where it stands, what it would need; the model itself is neural and is not a Mova component.

Topics: Named entity recognition

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2104.04052

@misc{seker2021alephbert,
  title = {{AlephBERT: A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application With}},
  author = {Seker, Amit and Bandel, Elron and Bareket, Dan and Brusilovsky, Idan and Greenfeld, Refael Shaked and Tsarfaty, Reut},
  year = {2021},
  eprint = {2104.04052},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.04052}
}

References

18 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adam Amram, Anat Ben-David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from modern hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 2242–2252. find our card
2. Giusepppe Attardi. 2015. Wikiextractor. https:// github.com/attardi/wikiextractor . ↗
3. Dan Bareket and Reut Tsarfaty. 2020. Neural modeling for named entities and morphology (nemoˆ2). CoRR, abs/2007.15620. ↗
4. Naama Ben Mordecai and Michael Elhadad. 2005. Hebrew named entity recognition. find
5. Avihay Chriqui and Inbal Yahav. 2021. Hebert —& hebemo: a hebrew bert model and a tool for polarity analysis and emotion recognition. find our card
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
7. Stav Klein and Reut Tsarfaty. 2020. Getting the find
8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. find
9. Amir Pouran Ben Veyseh Minh Van Nguyen, Viet Lai and Thien Huu Nguyen. 2021. Trankit: A lightweight transformer-based toolkit for multilingual natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. find
10. Amir More, Amit Seker, Victoria Basmova, and Reut Tsarfaty. 2019. Joint transition-based models for morpho-syntactic parsing: Parsing strategies for mrls and a case study from modern hebrew. Trans. Assoc. Comput. Linguistics, 7:33–48. find our card
11. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoı̂t Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Online. Association for Computational Linguistics. find
12. Shoval Sadde, Amit Seker, and Reut Tsarfaty. 2018. The hebrew universal dependency treebank: Past present and future. In Proceedings of the Second Workshop on Universal Dependencies, UDW@EMNLP 2018, Brussels, Belgium, November 1, 2018, pages 133–143. find our card
13. Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Galletebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yannick Versley, Veronika Vincze, Marcin Wolinski, Alina Wróblewska, and Éric Villemonte de la Clergerie. 2013. Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, SPMRL@EMNLP 2013, Seattle, Washington, USA, October 18, 2013, pages 146–182. find
14. Amit Seker and Reut Tsarfaty. 2020. A pointer network architecture for joint morphological segmentation and tagging. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4368–4378, Online. Association for Computational Linguistics. find our card
15. Reut Tsarfaty, Dan Bareket, Stav Klein, and Amit Seker. 2020. From SPMRL to NMRL: what did we learn (and unlearn) in a decade of parsing morphologically-rich languages (mrls)? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7396–7408. find
16. Reut Tsarfaty, Shoval Sadde, Stav Klein, and Amit Seker. 2019. What’s wrong with hebrew nlp? and how to make it right. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 - System Demonstrations, pages 259– 264. find our card
17. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find
18. Daniel Zeman, Jan Hajič, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. CoNLL 2018 shared task: Multilingual parsing from raw text to Universal Dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–21, Brussels, Belgium. Association for Computational Linguistics. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2021-seker-alephbert-hebrew-large-pre-trained/