Evaluating Inter-Annotator Agreement on Historical Spelling Normalization

What it is about

Normalisation labels are open and sparse, and most tokens are close to their surface form, so the usual agreement measures do not fit the task well. The paper proposes a way to use chance-corrected agreement (Krippendorff's alpha) with a character-level edit distance, so that near-identical normalisations count as partial agreement. Five annotators labelled a German dataset. Word-based percentage agreement over all tokens is 92.62%. The character-based alpha is higher than the word-based one, because partial matches count. The paper points out that the word-based figure of 96.9% reported for another historical German corpus ignores partial agreement and chance.

Ideas

• I1. en morph-eval: add an edit-distance agreement score next to exact accuracy, so that a one-letter miss is shown apart from a wrong lemma. Measure both on the lemma set, where accuracy is 99.3–99.5%, and report the split.
• I2. Distance-aware error report: in en/src/morph.rs, group the errors by edit distance to the gold form, and give each group its count per lesson.
• I3. Double annotation for archaic tokens: two annotators label about 200 archaic tokens from the lessons, and the chance-corrected agreement is computed before any lemma accuracy for them is claimed. This is a test set only.

For Mova

The chapter is 06. Spelling rules: inflection, derivation, compounds and national standards, with 09. Early Modern and archaic forms for the forms. The code is en/src/morph.rs and en morph-eval. The paper is about German. Its method is for evaluation only, and it adds no rule to the build.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-1711

@inproceedings{bollmann2016evaluating,
  title = {{Evaluating Inter-Annotator Agreement on Historical Spelling Normalization}},
  author = {Bollmann, Marcel and Dipper, Stefanie and Petran, Florian},
  booktitle = {Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)},
  year = {2016},
  doi = {10.18653/v1/w16-1711},
  url = {https://doi.org/10.18653/v1/w16-1711}
}

References

12 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. find
2. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. ↗
3. Stefanie Dipper and Simone Schultz-Balluff. 2013. The Anselm corpus: Methods and perspectives of a parallel aligned corpus. In Proceedings of the NODALIDA Workshop on Computational Historical Linguistics, Oslo, Norway. find
4. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. find
5. Julia Krasselt, Marcel Bollmann, Stefanie Dipper, and Florian Petran. 2015. Guidelines for normalizing historical German texts. Bochumer Linguistische Arbeitsberichte, 15. find
6. Klaus Krippendorff. 1980. Content Analysis: An Introduction to Its Methodology. SAGE, Beverly Hills, CA. find
7. Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707–710. find
8. Johann-Mattis List and Robert Forkel. 2016. LingPy. A Python library for historical linguistics. Version 2.4. http://lingpy.org. With collaborations by Steven Moran, Peter Bouda, Johannes Dellert, Taraka Rama, Frank Nagel, and Simon Greenhill: Max Planck Institute for the Science of Human History. ↗
9. Saul B. Needleman and Christian D. Wunsch. 1970. A gene method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology, 48:443–453. find
10. Rebecca J. Passonneau. 2004. Computing reliability for coreference annotation. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC), volume 4, pages 1503–1506, Lisbon, Portugal. find
11. Silke Scheible, Richard J. Whitt, Martin Durrell, and Paul Bennett. 2011. A gold standard corpus of Early Modern German. In Proceedings of the ACL-HLT 2011 Linguistic Annotation Workshop (LAW V), pages 124–128, Portland, Oregon, USA. find our card
12. William A. Scott. 1955. Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3):321–325. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2016-bollmann-evaluating-inter-annotator-agreement-historical/