Parsing the Switch: LLM-Based UD Annotation for Complex Code-Switched and Low-Resource Languages

Summary

The paper tackles syntactic annotation of code-switched text, where monolingual parsers generalise poorly and annotated data hardly exists. It introduces the BiLingua Pipeline: an LLM (GPT-4.1) is prompted with few-shot examples and a set of linguistic rules to produce UD part-of-speech tags, heads and relations, after which experts and native speakers review and correct the output. The pipeline is applied to Spanish-English conversational data from the Miami Corpus and to Spanish-Guaraní social-media and news text, yielding two released datasets, including the first UD corpus for Spanish-Guaraní. After expert revision the annotation reaches up to 95.29% LAS, well above earlier baselines and multilingual parsers. The authors also propose a more lenient evaluation that does not penalise linguistically acceptable alternatives, and analyse at which syntactic positions speakers switch languages. The overall message is that a carefully guided LLM plus human review can bootstrap syntactic resources for low-resource, mixed-language settings.

How Mova uses it

• dep — only as a last resort — cited as a source of the rule en.errors.dep-last-resort: LLM annotators fall back to dep on repetitions, ellipsis and contractions (the paper's prompt even allowed dep for ellipsis), so every dep is flagged for review in EWT style.
• Typical failures of LLM annotators for UD — failure mode 5 ("inconsistency"): human correction raised LAS from about 76 to 95, which Mova reads as a warning that the yardstick must be an independent gold standard, not corrected LLM output.
• VP ellipsis: the auxiliary becomes the head — background for the rule en.errors.aux-before-verb (LLMs are inconsistent on ellipsis).

Effectiveness in Mova

The paper informs one warning rule and the catalogue of LLM-annotator failure modes used when Mova's deterministic annotator is checked against LLM drafts. Not measured separately.

👨‍🔬💥

Topics: Universal Dependencies

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-emnlp.863 · arXiv 2506.07274

@inproceedings{kellert2025parsing,
  title = {{Parsing the Switch: LLM-Based UD Annotation for Complex Code-Switched and Low-Resource Languages}},
  author = {Kellert, Olga and Tyagi, Nemika and Imran, Muhammad and Licona-Guevara, Nelvin and Gómez-Rodríguez, Carlos},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.863},
  eprint = {2506.07274},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2025.findings-emnlp.863}
}

References

18 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. 2020. LinCE: A centralized benchmark for linguistic code-switching evaluation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1803–1813, Marseille, France. European Language Resources Association. find
2. Irshad Bhat, Riyaz A. Bhat, Manish Shrivastava, and Dipti Sharma. 2018. Universal Dependency parsing for Hindi-English code-switching. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pap… find
3. Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X), pages Carol Myers-Scotton. 2002. Contact Linguistics: Bilingual Encounters and Grammatical Outcomes. Oxford Univ… find our card
4. Maria Carmen Parafita Couto, Marianne Gullberg, and Pieter Muysken. 2015. Subject positioning in spanish–english code-switching. Linguistic Approaches to Bilingualism, 5(3):277–300. find
5. Shana Poplack. 1980. Sometimes i’ll start a sentence in spanish y termino en español: Toward a typology of code-switching. Linguistics, 18(7-8):581–618. find
6. Shruti Rijhwani and Thamar Solorio. 2016. Estimating code-switching on twitter with a novel generalized word-level classification model. In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 33–42, Austin, Texas. Association for Computational Linguistics. find
7. Shruti Rijhwani, Lawrence Wolf-Sonkin, Victor Kuperman, Timothy Baldwin, and Thamar Solorio. 2017. Analyzing code-switched social media text. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. find
8. Diego Roca, David Vilares, and Carlos Gómez-Rodríguez. 2023. A system for constituent and dependency tree linearization. Kalpa Publications in Computing. find
9. Şazi Murat Özateş, Özlem Çetinoğlu, Reut Tsarfaty, Dilek Küçük, and Olcay Taner Yıldız. 2022. Improving code-switching dependency parsing with semi-supervised auxiliary tasks. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1126–1141, Seattle, United States. Associ… find
10. Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Christopher D. Manning. 2014. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). find
11. Thamar Solorio, Emily Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Di Lin, et al. 2014. Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Workshop on Computational… find
12. Thamar Solorio and Yang Liu. 2008. Part-of-speech tagging for english-spanish code-switched text. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 1051– 1060, Honolulu, Hawaii. Association for Computational Linguistics. find
13. Michalina Strzyz, David Vilares, and Carlos Gómez-Rodríguez. 2019. Viable dependency parsing as sequence labeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), page… find
14. Hakyung Sung and Gyu-Ho Shin. 2025. Second language korean universal dependency treebank v1. 2: Focus on data augmentation and annotation scheme refinement. arXiv preprint arXiv:2503.14718. find
15. Mariona Taulé, Maria Antònia Martí, and Marta Recasens. 2008. Ancora: Multilevel annotated corpora for catalan and spanish. In Lrec, volume 2008, pages 96–101. find
16. Almeida Jacqueline Toribio. 2001. Accessing bilingual code-switching competence. International Journal of Bilingualism, 5(4):403–436. find
17. Daniel Zeman, Jan Hajič, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. find
18. CoNLL 2018 shared task: Multilingual parsing from raw text to Universal Dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–21, Brussels, Belgium. Association for Computational Linguistics. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/kellert-2025-parsing-switch-llm-based-ud/