Building UD Cairo for Old English in the Classroom

What it is about

The paper builds an Old English treebank of the 20 UD Cairo sentences, which illustrate the constructions of the world's languages, as a classroom exercise in historical linguistics. The sentences were first drafted with an LLM, and the drafts needed correction: the model added anachronisms and gave wrong gender agreement on articles. Students then annotated the sentences, several per sentence, and their results were compared and adjudicated. Agreement was measured with accuracy, Cohen's kappa and F1, and the students came close to the experienced group. Parsers trained on Modern English GUM with mBERT reach UAS 85.38 and LAS 78.36 on the Old English sentences with hyperlemma and gloss features. The authors' lesson is that LLM output needs a quality-assurance step before it becomes data.

Ideas

• I1. Cairo check for English: run the en parser on the English UD Cairo sentences (small, test only) and report errors per construction at each release, as a regression check rather than training data.
• I2. Two-teacher agreement in annot: when two teachers (Haiku and Opus) annotate the same sentence, report accuracy and Cohen's kappa per field (UPOS, FEATS, deprel), as the paper does, and use the disagreements as the review queue.
• I3. Gate for LLM drafts: a Rust check in annot that rejects any silver tree breaking the registry or the tools/treeagree rules before it reaches corpus/; measure the rejection rate on 300 EWT sentences and compare it with the paper's experience of manual correction.

For Mova

The annotation pipeline is the annot crate with the Rust gates, the agreement checks belong there, and the parser test is en parse-eval. The chapter is 09. Early Modern and archaic forms, which covers older English. Measure first the rejection rate of the gate on 300 EWT sentences, because it shows whether a gate adds anything on top of the teacher.

Topics: Universal Dependencies

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2504.18718

@inproceedings{levine2025building,
  title = {{Building UD Cairo for Old English in the Classroom}},
  author = {Levine, Lauren and Min, Junghyun and Zeldes, Amir},
  booktitle = {Proceedings of the Eighth Workshop on Universal Dependencies (UDW, SyntaxFest 2025)},
  year = {2025},
  eprint = {2504.18718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.18718}
}

References

20 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David Bamman and Gregory Crane. 2007. The Latin dependency treebank in a cultural heritage digital library. In Proceedings of the Workshop on Language Technology for Cultural Heritage Data (LaTeCH 2007)., pages 33–40, Prague, Czech Republic. Association for Computational Linguistics. find
2. Yevgeni Berzak, Yan Huang, Andrei Barbu, Anna Korhonen, and Boris Katz. 2016. Anchoring and agreement in syntactic annotations. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2215–2224, Austin, Texas. Association for Computational Linguistics. ↗
3. Christian Chiarcos, Benjamin Kosmehl, Christian Fäth, and Maria Sukhareva. 2018. Analyzing middle high german syntax with rdf and sparql. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). ↗
4. Kevin Clark, Thang Luong, Quoc V. Le, and Christopher Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In ICLR. find
5. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. ↗
6. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308. find our card
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
8. Stefanie Dipper, Lukas Faulstich, Ulf Leser, and Anke Lüdeling. 2004. Challenges in modelling a richly annotated diachronic corpus of german. In Workshop on XML-based richly annotated corpora, pages 21– 29. find
9. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In International Conference on Learning Representations. find our card
10. Gaël Guibon, Marine Courtin, Kim Gerdes, and Bruno Guillaume. 2020. When collaborative treebank curation meets graph grammars. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5291–5300, Marseille, France. European Language Resources Association. find
11. Bruno Guillaume. 2021. Graph matching and graph rewriting: GREW tools for corpus exploration, maintenance and conversion. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 168–175, Online. Association for Computational Linguistics. ↗
12. Jan Hajič, Otakar Smrž, Petr Zemánek, Petr Pajas, Jan Šnaidauf, Emanuel Beška, Jakub Kracmar, and Kamila Hassanová. 2009. Prague arabic dependency treebank 1.0. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
13. Mayank Jobanputra, Maitrey Mehta, and Çağrı Çöltekin. 2024. A Universal Dependencies treebank for Gujarati. In Proceedings of the Joint Workshop on Multiword Expressions and Universal Dependencies (MWE-UD) @ LREC-COLING 2024, pages 56–62, Torino, Italia. ELRA and ICCL. ↗
14. Alek Keersmaekers, Wouter Mercelis, Colin Swaelens, and Toon Van Hal. 2019. Creating, enriching and valorizing treebanks of Ancient Greek. In Proceedings of the 18th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2019), pages 109–117, Paris, France. Association for Computational Linguistics. ↗
15. Anthony Kroch. 2020. Penn parsed corpora of historical english. find
16. Anthony Kroch, Beatrice Santorini, and Lauren Delfs. 2004. Penn-helsinki parsed corpus of early modern english. find
17. Seth Kulick, Neville Ryant, and Beatrice Santorini. 2023. Parsing “early English books online” for linguistic search. In Proceedings of the Society for Computation in Linguistics 2023, pages 222–242, Amherst, MA. Association for Computational Linguistics. find
18. Karel Kučera and Martin Stluka. 2014. Data processing and lemmatization in digitized 19th-century czech texts. In Proceedings of the First International Conference on Digital Access to Textual Cultural Heritage, DATeCH ’14, page 193–196, New York, NY, USA. Association for Computing Machinery. find
19. Stefano Lusito and Jean Maillard. 2021. A Universal Dependencies corpus for Ligurian. In Proceedings of the Fifth Workshop on Universal Dependencies (UDW, SyntaxFest 2021), pages 121–128, Sofia, Bulgaria. Association for Computational Linguistics. find
20. Francesco Mambrini. 2016. The Ancient Greek Dependency Treebank: Linguistic annotation in a teaching environment. In M. Romanello and G. Bodard, editors, Digital Classics Outside the Echo-Chamber. Ubiquity Press, London. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ud-2025-levine-building-ud-cairo-old-english/