Pre-annotation Matters: A Comparative Study on POS and Dependency Annotation for an Alsatian Dialect

Summary

The study asks whether automatic pre-annotation helps human annotators build a UD treebank (POS tags and dependencies) for an Alsatian Alemannic dialect that has no annotated data, no tools and no standard spelling. Three near-zero-shot pre-annotators are compared: German UDPipe 2 models applied after a light normalisation of the dialect towards German, the generative LLM Mistral Large driven by prompts, and an ArboratorGrew parser trained on a mix of closely related languages and dialects. Human correctors then fixed the output batch by batch. Good final quality was reachable with all set-ups, and annotators adjusted their correction effort to how good the pre-annotation looked rather than accepting it blindly. The parser trained on related varieties gave the best overall results on both tasks. The LLM tagged parts of speech better than one of the UDPipe baselines but built noticeably worse trees, and its output often needed repairs to be valid CoNLL-U. The paper is practical guidance for small teams building treebanks for low-resource varieties.

How Mova uses it

• en/seeds/errors/llm-annotator-failure-modes.md: a seed card listing typical failure modes of LLM-based UD annotators; this paper supplies the first two — "tags are better than trees" (LLM tagging accuracy high, attachment much lower) and "short arcs" (LLM output has a shorter mean dependency distance than the gold standard, i.e. a bias towards the nearest head).
• The short-arc finding motivates several deterministic checks in the expert-system rules (engine en/src/expert.rs): advcl under a noun (en/seeds/errors/advcl-on-noun.md), obl under a nominal (obl-on-nominal.md) and an auxiliary after its head in VP ellipsis.
• The observation that all tools confused the perfect with the copula/passive is cited in copula-be-only.md and passive-structure.md (rules: cop only on be; nsubj:pass/aux:pass only with a VBN head).
• "Dependencies worse than tags" backs function-words-leaves.md (auxiliaries and copulas must be leaves) and nominal-rels-on-verb.md (nominal relations must not hang on verbs).
• More generally it supports Mova's "draft" mode in the annotator: the Rust parser annotates first and the large model only edits, rather than letting an LLM produce trees from scratch.

Effectiveness in Mova

Not measured separately as an idea. The rules it informs are validated on gold treebanks (UD English EWT 2.18), for example: advcl under a noun fired 201 times with 15 violations (fragments and headlines); cop not on be — 0 of 5,939; nsubj:pass with a non-VBN head — 0 of 1,445; auxiliaries/copulas with core dependents — 0 of 15,648. These checks show the rules are consistent with the gold standard; the paper's role is as a source of which errors to look for.

👨‍🔬💥

Topics: Treebanks · Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.law-1.14

@inproceedings{bernhard2025pre,
  title = {{Pre-annotation Matters: A Comparative Study on POS and Dependency Annotation for an Alsatian Dialect}},
  author = {Bernhard, Delphine and Beiner, Nathanaël and Hoff, Barbara},
  booktitle = {Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025)},
  year = {2025},
  doi = {10.18653/v1/2025.law-1.14},
  url = {https://doi.org/10.18653/v1/2025.law-1.14}
}

References

44 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Noëmi Aepli. 2018. Parsing Approaches for Swiss German. Master’s thesis, University of Zurich. find
2. Noëmi Aepli and Rico Sennrich. 2022. Improving Zero-Shot Cross-lingual Transfer Between Closely Related Languages by Injecting Character-Level Noise. In Findings of the Association for Computational Linguistics: ACL 2022, pages 4074–4083, Dublin, Ireland. Association for Computational Linguistics. find
3. Delphine Bernhard. 2023. Transfert zero-shot pour l’étiquetage morphosyntaxique : analyse de l’impact de la transformation des données à étiqueter pour les dialectes alsaciens. In Actes des 5èmes journées du Groupement de Recherche CNRS “ Linguistique Informatique, Formelle et de Terrain ”, pages 30–38, Nancy, France. find
4. Delphine Bernhard, Pascale Erhart, Dominique Huck, and Lucie Steiblé. 2018. Part-of-speech annotation guidelines for the alsatian dialects. Zenodo: 10.5281/zenodo.1171925. ↗
5. Delphine Bernhard and Anne-Laure Ligozat. 2013. Hassle-free POS-Tagging for the Alsatian Dialects. In Marcos Zampieri and Sascha Diwersy, editors, Non-Standard Data Sources in Corpus Based-Research, ZSM Studien, pages 85–92. Shaker. find
6. Yevgeni Berzak, Yan Huang, Andrei Barbu, Anna Korhonen, and Boris Katz. 2016. Anchoring and Agreement in Syntactic Annotations. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2215–2224, Austin, Texas. Association for Computational Linguistics. ↗
7. Verena Blaschke, Barbara Kovačić, Siyao Peng, Hinrich Schütze, and Barbara Plank. 2024. MaiBaam: A Multi-Dialectal Bavarian Universal Dependency Treebank. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). find
8. Verena Blaschke, Hinrich Schütze, and Barbara Plank. 2023. Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languages. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (Var-Dial 2023), pages 40–54, Dubrovnik, Croatia. Association for Computational Linguistics. find
9. Emanuel Borges Völker, Maximilian Wendt, Felix Hennig, and Arne Köhn. 2019. HDT-UD: A very large Universal Dependencies Treebank for German. In Proceedings of the Third Workshop on Universal Dependencies (UDW, SyntaxFest 2019), pages 46–57, Paris, France. Association for Computational Linguistics. find
10. Philippe Boula de Mareüil, Frédéric Vernier, and Albert Rilliard. 2018. A Speaking Atlas of the Regional Languages of France. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), page 6, Miyazaki, Japan. find
11. Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X Shared Task on Multilingual Dependency Parsing. In Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X), pages 149–164, New York City. Association for Computational Linguistics. find our card
12. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. ↗
13. Marie-Catherine De Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal dependencies. Computational linguistics, 47(2):255– 308. find our card
14. Wietse de Vries, Martijn Wieling, and Malvina Nissim. 2022. Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7676–7685, Dublin, Ireland. Association for Computational Linguistics. find
15. Stefanie Dipper, Cora Haiber, Anna Maria Schröter, Alexandra Wiemann, and Maike Brinkschulte. 2024. Universal Dependencies: Extensions for Modern and Historical German. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 17101–17111, Torino, Italia. ELRA and ICCL. find
16. Timothy Dozat and Christopher D Manning. 2017. Deep Biaffine Attention for Neural Dependency Parsing. In Proceedings of ICLR 2017. find our card
17. Hanne Martine Eckhoff and Aleksandrs Berdičevskis. 2016. Automatic parsing as an efficient pre-annotation tool for historical texts. In Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH), pages 62–70, Osaka, Japan. find
18. The COLING 2016 Organizing Committee. find
19. Jason M. Eisner. 1996. Three New Probabilistic Models for Dependency Parsing: An Exploration. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics. find our card
20. Ana Ezquerro, Carlos Gómez-Rodríguez, and David Vilares. 2025. Better benchmarking LLMs for zero-shot dependency parsing. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pages 121–135, Tallinn, Estonia. University of Tartu Library. find
21. Karën Fort and Benoît Sagot. 2010. Influence of pre-annotation on POS-tagged corpus development. In Proceedings of the Fourth ACL Linguistic Annotation Workshop, pages 56–63. find
22. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences of the United States of America, 120(30):1–3. find
23. Gaël Guibon, Marine Courtin, Kim Gerdes, and Bruno Guillaume. 2020. When collaborative treebank curation meets graph grammars. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5293–5302, Marseille, France. European Language Resources Association. find
24. Kirian Guiller. 2020. Analyse syntaxique automatique du pidgin-créole du Nigeria à l’aide d’un transformer (BERT) : Méthodes et Résultats. Master’s thesis, Sorbonne Nouvelle - Paris 3. find
25. Harald Hammarström, Robert Forkel, Martin Haspelmath, and Sebastian Bank. 2024. Glottolog 5.1. find
26. Jirka Hana, Anna Feldman, and Katsiaryna Aharodnik. 2011. A Low-budget Tagger for Old Czech. In Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities (LaTeCH ’11), pages 10–18. find
27. John D. Hunter. 2007. Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9(3):90–95. find
28. Klaus Krippendorff. 1970. Estimating the reliability, systematic error and random error of interval data. Educational and psychological measurement, 30(1):61–70. find
29. Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483–4499, Online. Association for Computational Linguistics. find
30. Haitao Liu, Chunshan Xu, and Junying Liang. 2017. Dependency distance: A new perspective on syntactic patterns in natural languages. Physics of Life Reviews, 21:171–193. ↗
31. Andy Lücking, Giuseppe Abrami, Leon Hammerla, Marc Rahn, Daniel Baumartz, Steffen Eger, and Alexander Mehler. 2024. Dependencies over Times and Tools (DoTT). In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4641–4653, Torino, Italia. ELRA and ICCL. find
32. Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50–60. find
33. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational linguistics, 19(2):313–330. find our card
34. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, and 1 others. 2013. Universal dependency annotation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 92–97. find
35. Marie Mikulová, Milan Straka, Jan Štěpánek, Barbora Štěpánková, and Jan Hajic. 2022. Quality and Efficiency of Manual Annotation: Pre-annotation Bias. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2909–2918, Marseille, France. European Language Resources Association. find
36. Joakim Nivre, Johan Hall, and Jens Nilsson. 2004. Memory-Based Dependency Parsing. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004, pages 49–56, Boston, Massachusetts, USA. Association for Computational Linguistics. find
37. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12(null):2825–2830. find
38. Ziqian Peng, Kim Gerdes, and Kirian Guiller. 2022. Pull your treebank up by its own bootstraps. In Actes Des Journées Jointes Des Groupements de Recherche Linguistique Informatique, Formelle et de Terrain (LIFT) et Traitement Automatique Des Langues (TAL)., pages 139–153, Marseille, France. find
39. Fred Philippy, Siwen Guo, and Shohreh Haddadan. 2023. Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5877–5891, Toronto, Canada. Association for Computational Linguistics. find
40. Alistair Plum, Caroline Döhmer, Emilia Milano, Anne-Marie Lutgen, and Christoph Purschke. 2024. LuxBank: The First Universal Dependency Treebank for Luxembourgish. In Proceedings of the 22nd Workshop on Treebanks and Linguistic Theories (TLT 2024), pages 30–39, Hamburg,Germany. Association for Computational Linguistics. find
41. Arne Skjærholt. 2014. A chance-corrected measure of inter-annotator agreement for syntax. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 934–944, Baltimore, Maryland. Association for Computational Linguistics. find
42. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Ev-geni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, and 16 others. 2020. SciPy 1.0: Fundamental algorithms for scientific computing in python. Nature Methods, 17:261–272. find
43. Xinyi Wang, Sebastian Ruder, and Graham Neubig. 2022. Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 863–877, Dublin, Ireland. Association for Computational Linguistics. find
44. Michael L. Waskom. 2021. Seaborn: Statistical data visualization. Journal of Open Source Software, 6(60):3021. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/bernhard-2025-pre-annotation-matters-comparative-study/