TafBERTa: Learning Grammatical Rules from Small-Scale Language Acquisition Data in Hebrew

What it is about

The paper builds TafBERTa, a small RoBERTa-style masked model (3.3M parameters, 7,317 BPE items) trained from scratch on 233K words of Hebrew child-directed speech from the CHILDES Berman longitudinal corpus, rewritten in Hebrew script and merged with its %mor and %gra annotations (HTBerman: 53K sentences). The new test set HeCLiMP is built from minimal pairs on determiner–noun agreement, in number (e.g. ha-kova ha-ze vs ha-kovaim ha-ele) and gender, using templates such as "Look at this ..." filled with nouns from the corpus. Scoring is holistic (summed cross-entropy). TafBERTa reaches 69.4 overall against HeRo's 73.5, which is trained on 4.7B words, but it is better on number (80.5 vs 69.1) and worse on gender (58.2 vs 77.9). A RoBERTa trained on the same child data gets 65.6, and DictaLM 2.0 gets 31.4 on number. The paper shows that data-small models can handle number agreement, while gender agreement needs more data or a different objective.

Ideas

• I1. A minimal-pair test for Hebrew determiner–noun agreement, built from templates and a noun table with gender and number, scored by the parser: the test passes when the parse links the determiner to a noun whose features agree, and fails otherwise. Keep number and gender as two separate sets, since the paper shows they behave differently.
• I2. A child-directed mode: the %gra tier (head–dependent relations, one per word, aligned with %mor) gives a small gold dependency set for short, simple utterances. It is a candidate test for the early-school grammar, once its licence is checked.
• I3. Score each HeCLiMP pair as the sum of the two sentences' errors, so that a rule-based scorer uses the same holistic rule as the neural baseline and can be compared with it on equal terms.

For Mova

I1 goes into Nouns, adjectives, pronouns and numerals (agreement) and 05. The verb: system, binyanim, tense and use for the verb analogue, as a generated test for the parser output. I2 is a candidate set for 7. Syntax and for the plan in 12. Mova and Modern Hebrew: where it stands, what it would need; check the corpus licence before use. I3 is a method note for tools/hbo-style evaluation scripts, and a measure to run before any claim.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.babylm-main.6

@inproceedings{gelboim2025tafberta,
  title = {{TafBERTa: Learning Grammatical Rules from Small-Scale Language Acquisition Data in Hebrew}},
  author = {Gelboim, Anita and Sulem, Elior},
  booktitle = {Proceedings of the First BabyLM Workshop},
  year = {2025},
  doi = {10.18653/v1/2025.babylm-main.6},
  url = {https://doi.org/10.18653/v1/2025.babylm-main.6}
}

References

33 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In The 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2623–2631. find
2. Eylon Gueta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2023. Large pre-trained models with extra-large vocabularies: A contrastive analysis of hebrew bert models and a new one to outperform them all. ArXiv 2211.15199 [cs.CL]. ↗ our card
3. Aviad Albert, Brian MacWhinney, Bracha Nir, and Shuly Wintner. 2012. A morphologically annotated hebrew childes corpus. In Proc. of the Workshop on Comp. Models of Language Acquisition and Loss, pages 20–22. find our card
4. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proc. of NAACL’18, pages 1195–1205. find
5. Sharon Armon-Lotem. 1996. The minimalist child: Parameters and functional heads in the acquisition of Hebrew. Tel Aviv University. find
6. Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. BabyBERTa: Learning more grammar with small-scale child-directed language. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 624–646, Online. Association for Computational Linguistics. find
7. Bastian Bunzeck, Daniel Duran, and Sina Zarrieß. 2025. Do construction distributions shape formal language learning in German BabyLMs? In Proceedings of the 29th Conference on Computational Natural Language Learning, pages 169–186, Vienna, Austria. Association for Computational Linguistics. ↗
8. Philip A. Huebner and Jon A. Willits. 2021. Chapter eight - using lexical context to discover the noun category: Younger children have it easier. In Kara D. Federmeier and Lili Sahakyan, editors, The Context of Cognition: Emerging Perspectives, volume 75, pages 279–331. Academic Press. find
9. Luca Capone, Alice Suozzi, Gianluca Lebani, and Alessandro Lenci. 2024. Babies: A benchmark for the linguistic evaluation of italian baby language models. In Italian Conf. on Comp. Ling. find
10. Filippo Chiarello, Vito Giordano, Irene Spada, Simone Barandoni, and Gualtiero Fantoni. 2024. Future applications of generative large language models: A data-driven case study on chatgpt. Technovation, 133:103002. find
11. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361. https://childes.talkbank.org/ ↗
12. Marvin Lavechin, Yaya Sy, Hadrien Titeux, María Andrea Cruz Blandón, Okko Räsänen, Hervé Bredin, Emmanuel Dupoux, and Alejandrina Cristia. 2023. Babyslm: language-acquisition-friendly benchmark of self-supervised spoken language models. In IN-TERSPEECH 2023. ISCA. find
13. Suchir Salhan, Richard Diehl Martinez, Zébulon Goriely, and Paula Buttery. 2024. Less is more: Pre-training cross-lingual small-scale language models with cognitively-plausible curriculum learning strategies. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pages 174–188, Miami, FL, USA. Association for Computational Linguistics. find
14. Jackson L. Lee, Ross Burkholder, Gallagher B. Flinn, and Emily R. Coppess. 2016. Working with chat transcripts in python. Technical report, Department of Computer Science, University of Chicago. find
15. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. Preprint, arXiv:1910.01108. ↗
16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
17. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Greenfeld, and Reut Tsarfaty. 2022. AlephBERT: Language model pre-training and evaluation from sub-word to sentence level. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 46–56, Dublin, Ireland. Association for Computational Linguistics. find our card
18. Brian MacWhinney. 2000. The CHILDES Project: Tools for analyzing talk, Third Edition. Mahwah, NJ: Lawrence Erlbaum Associates. find
19. Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Comput. Surv. find
20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics. find
21. Amir More, Amit Seker, Victoria Basmova, and Reut Tsarfaty. 2019. Joint transition-based models for morpho-syntactic parsing: Parsing strategies for MRLs and a case study from Modern Hebrew. Transactions of the Association for Computational Linguistics, 7:33–48. find our card
22. Vitaly Shalumov and Harel Haskey. 2023. Hero: Roberta and longformer hebrew language models. arXiv preprint arXiv:2304.11077. ↗ our card
23. Zhewen Shen, Aditya Joshi, and Ruey-Cheng Chen. 2024. BAMBINO-LM: (bilingual-)human-inspired continual pre-training of BabyLM. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 1–7, Bangkok, Thailand. Association for Computational Linguistics. find
24. Aaron Mueller and Tal Linzen. 2023. How to plant trees in language models: Data and architectural effects on the emergence of syntactic inductive biases. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11237–11252, Toronto, Canada. Association for Computational Linguistics. find
25. Shaltiel Shmidman, Avi Shmidman, Amir DN Cohen, and Moshe Koppel. 2024. Adapting llms to hebrew: Unveiling dictalm 2.0 with enhanced vocabulary and instruction capabilities. Preprint, arXiv:2407.07080. ↗ our card
26. Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, and Tal Linzen. 2020. Cross-linguistic syntactic evaluation of word prediction models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5523–5539, Online. Association for Computational Linguistics. find
27. Shaltiel Shmidman, Avi Shmidman, and Moshe Koppel. 2023. Dictabert: A state-of-the-art bert suite for modern hebrew. ArXiv 2308.16687 [cs.CL]. ↗ our card
28. Taiga Someya and Yohei Oseki. 2023. JBLiMP: Japanese benchmark of linguistic minimal pairs. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1581–1594, Dubrovnik, Croatia. Association for Computational Linguistics. find
29. Miyu Oba, Tatsuki Kuribayashi, Hiroki Ouchi, and Taro Watanabe. 2023. Second language acquisition of neural language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13557–13572, Toronto, Canada. Association for Computational Linguistics. find
30. Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast WordPiece tokenization. In Proc. of EMNLP’21, pages 2089–2103. find
31. Riccardo Orlando, Luca Moroni, Pere-Lluís Huguet Cabot, Simone Conia, Edoardo Barba, Sergio Orlandini, Giuseppe Fiameni, and Roberto Navigli. 2024. Minerva llms: The first family of large language models trained from scratch on italian data. In Italian Conf. on Comp. Ling. find
32. Reut Tsarfaty, Shoval Sadde, Stav Klein, and Amit Seker. 2019. What‘s wrong with Hebrew NLP? and how to make it right. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International A.1 Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 259–264, Hong Kong, China. Association for Computational Linguistics. In Hebrew corpus files, there is incompatibility in files’ order with English Berman longitudinal dataset. In order to overcome this problem, we made manual changes to the Hebrew corpus, including removing blank files and reordering according to Berman longitudinal dataset files’ order. find our card
33. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mos-quera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 1–34, Singapore. Association for Computational Linguistics. A.2 Utterance-level matching find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/babylm-2025-gelboim-tafberta-learning-grammatical-rules-small/