LLM Dependency Parsing with In-Context Rules

Summary

The paper tests whether explicit symbolic rules in the prompt help large language models do dependency parsing for low-resource languages, where there is too little annotated data to train a classic neural parser. It uses UD treebanks of eight typologically diverse languages (including Bambara, Cantonese, Erzya, K'iche' and Yoruba), with non-projective sentences removed. Four kinds of prompt additions are compared: only the list of allowed relation labels; rules that the LLM first writes itself from five annotated examples; word contexts drawn from training data (head plus relation pairs); and the official UD annotation guidelines. Each is tried with 0, 3 and 5 in-context examples chosen by chrF++ similarity. Symbolic knowledge clearly helps when there are no examples, but the advantage almost vanishes at five examples. Word contexts work best and are also cheapest, since they need no extra LLM call, while the human-written guidelines help least. On test data no LLM beat mBERT/XLM-R-based parsers or UDPipe, which makes this a useful negative result. It also suggests that compact rules can replace long example lists and shorten prompts.

How Mova uses it

• Typical failures of LLM annotators for UD: one of the sources for the catalogue of typical errors LLM annotators make in UD. The paper backs item 6: LLM-generated rules tend to be too narrow (for example treating a verb as AUX) and ignore word order, and UD guidelines in the prompt help only when there are no examples.
• Each failure mode in that catalogue is paired with a deterministic check in the English error-rule set. Mova's draft mode is designed around this picture: the Rust model annotates first and the large model only edits, rather than the large model parsing from scratch with guidelines in the prompt.

Effectiveness in Mova

Background reading. It supports one point in a design note and does not drive any rule or module directly. Not measured separately.

👨‍🔬💥

Topics: Dependency parsing

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.xllm-1.17

@inproceedings{ginn2025llm,
  title = {{LLM Dependency Parsing with In-Context Rules}},
  author = {Ginn, Michael and Palmer, Alexis},
  booktitle = {Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025)},
  year = {2025},
  doi = {10.18653/v1/2025.xllm-1.17},
  url = {https://doi.org/10.18653/v1/2025.xllm-1.17}
}

References

28 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wasi Ahmad, Zhisong Zhang, Xuezhe Ma, Eduard Hovy, Kai-Wei Chang, and Nanyun Peng. 2019. On difficulties of cross-lingual transfer with order differences: A case study on dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Ling… find
2. Seth Aycock, David Stap, Di Wu, Christof Monz, and Khalil Sima’an. 2025. Can LLMs really learn to translate a low-resource language from one grammar book? In The Thirteenth International Conference on Learning Representations. find
3. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308. find our card
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
5. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In International Conference on Learning Representations. find our card
6. Xuefeng Bai, Jialong Wu, Yulong Chen, Zhongqing Wang, and Yue Zhang. 2023. Constituency parsing using llms. Preprint, arXiv:2310.19462. find
7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and alia. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783. find
8. Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2023. Prompting language models for linguistic structure. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6649–6663, Toronto, Canada. Association for Computational Linguistics. find
9. Ana Ezquerro, Carlos Gómez-Rodríguez, and David Vilares. 2025. Better benchmarking llms for zero-shot dependency parsing. arXiv preprint arXiv:2502.20866. find
10. Samuel Cahyawijaya, Holy Lovenia, and Pascale Fung. 2024. LLMs are few-shot in-context low-resource language learners. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 405–… find
11. Michael Ginn, Mans Hulden, and Alexis Palmer. 2024. Can we teach language models to gloss endangered languages? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5861– 5876, Miami, Florida, USA. Association for Computational Linguistics. find
12. Junjie Chen, Xiangheng He, and Yusuke Miyao. 2024. Language model based unsupervised dependency parsing with conditional mutual information and grammatical constraints. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag… find
13. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
14. Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2016. A representation learning framework for multi-source transfer parsing. In AAAI Conference on Artificial Intelligence. find
15. Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert. 2023. MLX: Efficient and flexible machine learning on apple silicon. find
16. Claudiu Daniel Hromei, Danilo Croce, and Roberto Basili. 2024. U-deppllama: Universal dependency parsing via auto-regressive large language models. IJCoL. Italian Journal of Computational Linguistics, 10(10, 1). find
17. Daniel Jurafsky and James H. Martin. 2025. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models, 3rd edition. Online manuscript released January 12, 2025. find
18. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. Preprint, arXiv:2409.04109. find
19. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395, Barcelona, Spain. Association for Computational Linguistics. find
20. Jianling Li, Meishan Zhang, Peiming Guo, Min Zhang, and Yue Zhang. 2023. LLM-enhanced self-training for cross-domain constituency parsing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8174–8185, Singapore. Association for Computational Linguistics. find
21. Zuchao Li, Jiaxun Cai, Shexia He, and Hai Zhao. 2018. Seq2seq dependency parsing. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3203–3214, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
22. Boda Lin, Zijun Yao, Jiaxin Shi, Shulin Cao, Binghao Tang, Si Li, Yong Luo, Juanzi Li, and Lei Hou. 2022a. Dependency parsing via sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7339–7353, Abu Dhabi, United Arab Emirates. Association for Computati… find
23. Boda Lin, Xinyi Zhou, Binghao Tang, Xiaocheng Gong, and Si Li. 2023. Chatgpt is a potential zero-shot dependency parser. Preprint, arXiv:2310.16654. — [arXiv:2310.16654](https://arxiv.org/abs/2310.16654) find
24. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano… find
25. Milan Straka and Jana Straková. 2017. Tokenizing, POS tagging, lemmatizing and parsing UD 2.0 with UDPipe. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 88–99, Vancouver, Canada. Association for Computational Linguistics. find our card
26. Yuanhe Tian, Fei Xia, and Yan Song. 2024. Large language models are no longer shallow parsers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7131–7142, Bangkok, Thailand. Association for Computational Linguistics. find
27. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems. find
28. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ginn-2025-llm-dependency-parsing-context-rules/