HeRo: RoBERTa and Longformer Hebrew Language Models

What it is about

The authors build a Hebrew pre-training corpus, HeDC4, from mC4 and OSCAR 22.01. They deduplicate it with MinHash, clean it, and reach about 2.5 times the size of the corpus used for AlephBERT. On it they train HeRo, a RoBERTa-base model for inputs up to 512 tokens, and LongHeRo, a Longformer started from a mid-training HeRo checkpoint for long documents. HeRo beats AlephBERT on sentiment (SMCD accuracy 89.56 against 89.02), on NER (BMC F1 93.56) and on ParaShoot QA (F1 58.2 against 49.6). LongHeRo reaches 89.1 F1 on a machine-translated Hyperpartisan set, close to the English Longformer (90.1).

Ideas

• I1. Near-duplicate removal as a step before any Hebrew corpus is used: MinHash over shingles, applied within each source and across sources. Measure how many documents go and check the removed pairs by eye, as the paper shows one near-duplicate example. Targets the corpus pipeline of tools/ocrfix output and any web text Mova collects.
• I2. Report every Hebrew test in two views, as the paper does: morph-based (gold segmentation given) and token-based (raw tokens). Run tools/hbo on the same MACULA-derived gold both ways, so the segmenter's loss is separated from the parser's loss.
• I3. Translated test sets need a manual check on a sample. The authors machine-translated Hyperpartisan and checked only a subset by hand. Any Hebrew test Mova builds by translation should be sampled and checked the same way, and kept as a test, never as training data.

For Mova

I2 goes into tools/hbo (the evaluation harness) and the segmentation chapter 1. Script and pointing. I1 is a data step for the Hebrew corpora. I3 is a rule for any translated test set in languages/he/.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2304.11077

@misc{shalumov2023hero,
  title = {{HeRo: RoBERTa and Longformer Hebrew Language Models}},
  author = {Shalumov, Vitaly and Haskey, Harel},
  year = {2023},
  eprint = {2304.11077},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.11077}
}

References

27 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards a cleaner document-oriented multilingual crawled corpus. arXiv preprint arXiv:2201.06642. ↗
2. Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from modern hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242– 2252. find our card
3. Dan Bareket and Reut Tsarfaty. 2021. Neural modeling for named entities and morphology (nemoˆ2). Transactions of the Association for Computational Linguistics, 9:909– 928. find
4. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. ↗
5. Andrei Z Broder. 1997. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SE-QUENCES 1997 (Cat. No. 97TB100171), pages 21–29. IEEE. find
6. Avihay Chriqui and Inbal Yahav. 2021. Hebert & hebemo: a hebrew bert model and a tool for polarity analysis and emotion recognition. arXiv preprint arXiv:2102.01909. ↗ our card
7. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. ↗
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
9. Eylon Guetta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2022. Large pre-trained models with extra-large vocabularies: A contrastive analysis of hebrew bert models and a new one to outperform them all. arXiv preprint arXiv:2211.15199. ↗ our card
10. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International Conference on Machine Learning, pages 3929–3938. PMLR. find
11. Omri Keren and Omer Levy. 2021. Parashoot: A hebrew question answering dataset. arXiv preprint arXiv:2109.11314. ↗ our card
12. Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. Semeval-2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 829–839. find
13. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499. ↗
14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
15. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. ↗
16. Naama Ben Mordecai and Michael Elhadad. 2005. Hebrew named entity recognition. find
17. Gabriele Sarti and Malvina Nissim. 2022. It5: Large-scale text-to-text pretraining for italian language understanding and generation. arXiv preprint arXiv:2203.03759. ↗
18. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert: A hebrew large pre-trained language model to start-off your hebrew nlp application with. arXiv preprint arXiv:2104.04052. ↗ our card
19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. ↗
20. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. arXiv preprint arXiv:2006.06202. ↗
21. Jörg Tiedemann and Santhosh Thottingal. 2020. Opus-mt–building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. European Association for Machine Translation. find
22. Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847. ↗
23. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45. find
24. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934. ↗
25. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33:17283–17297. find
26. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. Advances in neural information processing systems, 32. find
27. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2023-shalumov-hero-roberta-longformer-hebrew-language/