Step-by-step Instructions and a Simple Tabular Output Format Improve the Dependency Parsing Accuracy of LLMs
Summary
The paper asks how to get structurally valid, accurate Universal Dependencies parses out of large language models. Instead of bracketed trees, the model emits a reduced CoNLL-U-like table with only the ID, FORM, UPOS, HEAD and DEPREL columns, which makes non-projective trees easy to express and malformed output easy to repair. The instruction splits the task into ordered steps in a chain-of-thought style: part-of-speech tags first, then heads, then relation labels. Models are fine-tuned with LoRA and evaluated on 17 UD r2.15 treebanks, comparing several proprietary and open models. Splitting the task into steps gives a clear gain over a single-step prompt, and a mid-sized open model with a small number of trainable parameters beats UDPipe 2.0 and Hexatagger in LAS on all 17 treebanks. Structural errors such as cycles or multiple roots are almost absent, and a single multilingual model performs about as well as monolingual ones and transfers reasonably to unseen languages.
How Mova uses it
- Typical failures of LLM annotators for UD — cited as evidence in the catalogue of typical LLM annotation failures: formal errors (cycles are rare) and what helps (the UPOS → head → relation step order and tabular output).
- The same seed motivates which checks Mova's rule gates run on LLM-produced UD trees: heads and relations are checked first, because LLMs are better at tags than at trees.
- The general lesson (decompose the parse into ordered steps, constrain the output format) is background for the annotation workflow in
ai/annot, where the deterministic Rust parser produces a draft and the LLM only corrects it.
Effectiveness in Mova
Not measured separately. The paper serves as background evidence for one knowledge seed and for the design of the annotation gates; no ablation of the step order or the tabular format has been run in Mova.
👨🔬💥
Cite
@inproceedings{matsuda2025step,
title = {{Step-by-step Instructions and a Simple Tabular Output Format Improve the Dependency Parsing Accuracy of LLMs}},
author = {Matsuda, Hiroshi and Ma, Chunpeng and Asahara, Masayuki},
booktitle = {Proceedings of the 18th International Conference on Parsing Technologies (IWPT, SyntaxFest 2025)},
year = {2025},
eprint = {2506.09983},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2506.09983}
}References
- Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Eval… find
- Afra Amini, Tianyu Liu, and Ryan Cotterell. 2023. Hexatagging: Projective dependency parsing as tagging. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1453–1464, Toronto, Canada. Association for Computational Linguistics. find our card
- Masaki Asada and Makoto Miwa. 2025. Improving relation extraction by sequence-to-sequence-based dependency parsing pre-training. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7099–7105, Abu Dhabi, UAE. Association for Computational Linguistics. find
- Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108. Associat… find our card
- Xuefeng Bai, Jialong Wu, Yulong Chen, Zhongqing Wang, and Yue Zhang. 2023. Constituency parsing using llms. Preprint, arXiv:2310.19462. find
- Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE. find
- Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. 2025. Security and privacy challenges of large language models: A survey. ACM Computing Surveys, 57(6):1–39. find
- Milan Straka. 2018. UDPipe 2.0 prototype at CoNLL 2018 UD shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 197–207, Brussels, Belgium. Association for Computational Linguistics. find our card
- Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural… find our card
- Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Advances in Neural Information Processing Systems, volume 28, page 2773–2781. Curran Associates, Inc. find
- Carlos Gómez-Rodríguez and David Vilares. 2018. Constituent parsing as sequence labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1314–1324, Brussels, Belgium. Association for Computational Linguistics. find
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22. find
- Claudiu Daniel Hromei, Danilo Croce, and Roberto Basili. 2024. U-deppllama: Universal dependency parsing via auto-regressive large language models. Italian Journal of Computational Linguistics, 10. find
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/matsuda-2025-step-step-instructions-simple-tabular/