Mova0.5 pre-release⌕
☰
home / docs / papers / Probing the Dynamics of Syntactic Ability Acquisition Throughout LLM Pretraining

Probing the Dynamics of Syntactic Ability Acquisition Throughout LLM Pretraining

Summary

The authors study when syntactic skills appear during LLM pretraining and which ones stay weak. Their method, LoRA probing, keeps the base model frozen and trains only low-rank adapters to produce step-by-step UD parses in a simple tabular format. They apply it to the public intermediate checkpoints of OLMo-2-7B, training adapters at 24 pretraining stages and testing on UD treebanks in 33 languages. To fit OLMo-2's short context window they design a compact two-step template that predicts tags first and then heads and labels together without repeating word forms; it roughly matches the three-step template in accuracy while halving the context and greatly increasing throughput. The output format is learned very early in pretraining. Although the model was pretrained on almost exclusively English text, it reaches LAS above 80 in most of the languages tested, but relations such as iobj and csubj are acquired late and remain unstable across many languages.

How Mova uses it

Effectiveness in Mova

Not measured separately. The paper explains why the rules target these relations; the rules themselves are checked against gold treebanks (for example, in EWT 2.18 iobj with a case dependent occurs 0 times out of 795, and nsubj on a VERB occurs 4 times), but those are rule-validation counts, not a measured contribution of the paper.

👨‍🔬💥

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/2n8gkp49p3wg

@inproceedings{matsuda2026probing,
  title = {{Probing the Dynamics of Syntactic Ability Acquisition Throughout LLM Pretraining}},
  author = {Matsuda, Hiroshi and Asahara, Masayuki},
  booktitle = {Proceedings of the Ninth Workshop on Universal Dependencies (UDW 2026)},
  year = {2026},
  doi = {10.63317/2n8gkp49p3wg},
  url = {https://doi.org/10.63317/2n8gkp49p3wg}
}

References

41 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Laura Pérez-Mayos, Miguel Ballesteros, and Leo Wanner. 2021. How much pretraining data do language models need to learn syntax? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1571–1582, Online and Punta Cana, Dominican Republic. Association for Compu… find
  2. BigScience Workshop, :, Teven Le Scao, et al. 2023. Bloom: A 176b-parameter open-access multilingual language model. find
  3. Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2022. Analyzing the mono- and cross-lingual pretraining dynamics of multilingual language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3575–3590, Abu Dhabi, United Arab Emirates. Association… find
  4. Qwen, :, An Yang, et al. 2025. Qwen2.5 technical report. find
  5. Team OLMo, Pete Walsh, et al. 2025. 2 olmo 2 furious. find
  6. Yuanhe Tian, Fei Xia, and Yan Song. 2024. Large language models are no longer shallow parsers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7131–7142, Bangkok, Thailand. Association for Computational Linguistics. find
  7. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online (v2020.4). Zenodo. find
  8. Gemma Team, Morgane Riviere, et al. 2024. Gemma 2: Improving open language models at a practical size. find
  9. Aaron Grattafiori et al. 2024. The llama 3 herd of models. find
  10. Hetong Wang, Pasquale Minervini, and Edoardo Ponti. 2024. Probing the emergence of cross-lingual alignment during LLM training. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12159–12173, Bangkok, Thailand. Association for Computational Linguistics. find
  11. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. find
  12. Jeffrey Li et al. 2025. Datacomp-lm: In search of the next generation of training sets for language models. find
  13. Language Resource References Zeyu Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. 2021. Probing across time: What does RoBERTa know and Agić, Željko and Ljubešić, Nikola. 2015. Universal Dependencies for Croatian (that work for Serbian, too). INCOMA Ltd. Shoumen, BULGARIA. find
  14. Asahara, Masayuki and Kanayama, Hiroshi and Tanaka, Takaaki and Miyao, Yusuke and Uematsu, Sumire and Mori, Shinsuke and Matsumoto, Yuji and Omura, Mai and Murawaki, Yugo. 2018. Universal Dependencies Version 2 for Japanese. European Language Resources Association (ELRA). find
  15. Hajič, Jan and others. 2024. Prague Dependency Treebank - Consolidated 2.0 (PDT-C 2.0). LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL). find
  16. Anders Johannsen and Héctor Martínez Alonso and Barbara Plank. 2015. Universal Dependencies for Danish. find
  17. Batanović, Vuk and Ljubešić, Nikola and Samardžić, Tanja and Erjavec, Tomaž. 2018. Training corpus SETimes.SR 1.0. Slovenian language resource repository CLARIN.SI. find
  18. Lynn, Teresa and Foster, Jennifer. 2016. Universal Dependencies for Irish. Association pour le Traitement Automatique des Langues. Dépendances universelles de l’irlandais. find
  19. Agnė Bielinskienė and Loïc Boizou and Jolanta Kovalevskaitė and Erika Rimkutė. 2016. Lithuanian Dependency Treebank ALKSNIS. In: I. Skadin , a and R. Rozis (Eds.): Human Language Technologies – The Baltic Perspective. Amsterdam: IOS Press. find
  20. Marşan, Büşra and Akkurt, Salih Furkan and Şen, Muhammet and Gürbüz, Merve and Güngör, Onur and Özateş, Şaziye Betül and Üsküdarlı, Suzan and Özgür, Arzucan and Güngör, Tunga and Öztürk, Balkız. 2022. Enhancements to the BOUN Treebank Reflecting the Agglutinative Nature of Turkish. find
  21. Bosco, Cristina and Montemagni, Simonetta and Simi, Maria. 2013. Converting Italian Treebanks: Towards an Italian Stanford Dependency Treebank. Association for Computational Linguistics. find
  22. McDonald, Ryan and Nivre, Joakim and Quirmbach-Brundage, Yvonne and Goldberg, Yoav and Das, Dipanjan and Ganchev, Kuzman and Hall, Keith and Petrov, Slav and Zhang, Hao and Täckström, Oscar and Bedini, Claudia and Bertomeu Castelló, Núria and Lee, Jungmee. 2013. Universal Dependency Annotation for M… find
  23. Bouma, Gosse and van Noord, Gertjan. 2017. Increasing Return on Annotation Investment: The Automatic Construction of a Universal Dependency Treebank for Dutch. Association for Computational Linguistics. find
  24. Chun, Jayeol and Han, Na-Rae and Hwang, Jena D. and Choi, Jinho D. 2018. Building Universal Dependency Treebanks in Korean. European Language Resources Association (ELRA). find
  25. Verginica Barbu Mititelu. 2018. Modern Syntactic Analysis of Romanian. Publishing House of "Alexandru Ioan Cuza" University. find
  26. Dobrovoljc, Kaja and Erjavec, Tomaž and Krek, Simon. 2017. The Universal Dependencies Treebank for Slovenian. Association for Computational Linguistics. find
  27. Muischnek, Kadri and Müürisep, Kaili and Puolakainen, Tiina. 2016. Estonian Dependency Treebank: from Constraint Grammar tagset to Universal Dependencies. European Language Resources Association (ELRA). find
  28. Kira Droganova and Olga Lyashevskaya and Daniel Zeman. 2018. Data Conversion and Consistency of Monolingual Corpora: Russian UD Treebanks. Linköping University Electronic Press. find
  29. Nivre, Joakim and Megyesi, Beáta. 2007. Bootstrapping a Swedish Treebank Using Cross-Corpus Harmonization and Annotation Projection. find
  30. Guillaume, Bruno and de Marneffe, Marie-Catherine and Perrier, Guy. 2019. Conversion et améliorations de corpus du français annotés en Universal Dependencies [Conversion and Improvement of Universal Dependencies French corpora]. ATALA (Association pour le Traitement Automatique des Langues). find
  31. Petya Osenova and Kiril Simov. 2004. BTB-TR05: BulTreeBank Stylebook ą 05. find
  32. Hajič, Jan and Smrž, Otakar and Zemánek, Petr and Pajas, Petr and Šnaidauf, Jan and Beška, Emanuel and Kracmar, Jakub and Hassanová, Kamila. 2009. Prague Arabic Dependency Treebank 1.0. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL). find
  33. Prokopidis, Prokopis and Papageorgiou, Haris. 2017. Universal Dependencies for Greek. Association for Computational Linguistics. find
  34. Pretkalniundefineda, Lauma and Rituma, Laura and Saulundefinedte, Baiba. 2018. Deriving Enhanced Universal Dependencies from a Hybrid Dependency-Constituency Treebank. Springer-Verlag. find
  35. Shishkina, Yana and Lyashevskaya, Olga. 2021. Sculpting Enhanced Dependencies for Belarusian. Springer-Verlag. find
  36. Natalia Silveira and Timothy Dozat and Marie-Catherine de Marneffe and Samuel Bowman and Miriam Connor and John Bauer and Christopher D. Manning. 2014. A Gold Standard Dependency Corpus for English. find
  37. A.2. The specifications of LLMs used in the experiments are shown in Table 6. B. find
  38. Taulé, Mariona and Martí, M. Antònia and Recasens, Marta. 2008. AnCora: Multilevel Annotated Corpora for Catalan and Spanish. European Language Resources Association (ELRA). find
  39. Zeman, Daniel. 2017. Slovak Dependency Treebank in Universal Dependencies. find
  40. A.1. Settings Environment and Hyper-parameters The environment and the hyper-parameters used in the experiments are shown in Table 4 and 5, respectively. Hardware GPU CPU RAM SSD Software OS CUDA Python PyTorch Transformers TRL PEFT vLLM Comparison between 3-step and 2-step-no-form prompts The detai… find
  41. Wróblewska, Alina. 2018. Extended and Enhanced Polish Dependency Bank in Universal Dependencies Format. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/matsuda-2026-probing-dynamics-syntactic-ability-acquisition/