Evaluating Historical Text Normalization Systems: How Well Do They Generalize?
What it is about
The paper argues that normalization systems are judged badly. Most reports give one accuracy over all tokens, but in historical text most test tokens were seen in training, and for those a memorization baseline (map each form to its most frequent modern form in training) is very hard to beat. The authors compare two neural encoder-decoder models, one with soft attention and one with hard monotonic attention, against that baseline and against an earlier SMT result, on five languages. Accuracy is reported separately for seen and unseen tokens and for several training sizes. The hard-attention model is best on unseen tokens in every language. On seen tokens the baseline wins, so learned models should be applied only to unseen forms. A worked example shows how a system can lose to the baseline overall while winning on unseen tokens. Finally, a downstream check: the normalized text is POS-tagged with a modern tagger. Normalizing helps over raw text, but neither neural model clearly beats the baseline there.
Ideas
- I1. Report seen and unseen tokens separately for every author delta and for the standard. A delta that is better overall can still lose on seen tokens, and the split shows which.
- I2. Treat the standard encoder with no delta as the memorization baseline. A delta should win on unseen concepts and must not lose on seen ones. This is the test the translator should apply before adding any override.
- I3. Add a downstream check. Run the output through the roundtrip parser and compare tag or parse accuracy with and without the author delta. A text can match the target author and still parse worse, which is the paper's extrinsic warning.
- I4. Test a monotonic constraint in the Hebrew word alignment, the counterpart of the monotonic attention used here. Measure the alignment error on the MACULA glosses before and after.
For Mova
The code is tools/translate (profile.rs for the baseline split, hebrew.rs for the alignment test, measure.rs for the seen/unseen report). The neural models are not for Mova. Their evaluation protocol is, and it addresses open problems 2 and 6. The downstream check uses tools/roundtrip.
Cite
@inproceedings{robertson2018evaluating,
title = {{Evaluating Historical Text Normalization Systems: How Well Do They Generalize?}},
author = {Robertson, Alexander and Goldwater, Sharon},
booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
year = {2018},
doi = {10.18653/v1/n18-2113},
url = {https://doi.org/10.18653/v1/n18-2113}
}References
- Eva Pettersson, Beáta Megyesi, and Joakim Nivre. 2013a. Normalisation of historical text using context-sensitive weighted Levenshtein distance and compound splitting. In Proceedings of the 19th Nordic Conference of Computational Linguistics (NoDaLiDa 2013); Oslo University; Norway. NEALT Proceedings Series 16. Linköping University Electronic Press, 085, pages 163–179. find our card
- Petar Mitankin, Stefan Gerdjikov, and Stoyan Mihov. 2014. An approach to unsupervised historical text normalisation. In Proceedings of the First International Conference on Digital Access to Textual Cultural Heritage. ACM, New York, NY, USA, DATeCH ’14, pages 29–34. find
- Roee Aharoni, Yoav Goldberg, and Israel Ramat-Gan. 2017. Morphological inflection generation with hard monotonic attention. In Proceedings of ACL. find
- Alistair Baron and Paul Rayson. 2008. VARD2: A tool for dealing with spelling variation in historical corpora. In Postgraduate Conference in Corpus Linguistics. find
- Marcel Bollmann. 2012. Automatic normalization of historical texts using distance measures and the Norma tool. In Proceedings of the Second Workshop on Annotation of Corpora for Research in the Humanities (ACRH-2), Lisbon, Portugal. find
- Marcel Bollmann, Joachim Bingel, and Anders Søgaard. 2017. Learning attention for historical text normalization by learning to pronounce. In Proceedings of ACL 2017. find our card
- Eva Pettersson, Beáta Megyesi, and Joakim Nivre. 2014. A multilingual evaluation of three spelling normalisation methods for historical text. In LaTeCH@ EACL. pages 32–41. find
- Eva Pettersson, Beáta Megyesi, and Jörg Tiedemann. 2013b. An SMT approach to automatic annotation of historical text. In Proceedings of the Workshop on Computational Historical Linguistics at NoDaLiDa 2013; Oslo; Norway. NEALT Proceedings Series 18. Linköping University Electronic Press, 087, pages 54–69. find
- Eiríkur Rögnvaldsson, Anton Karl Ingason, Einar Freyr Sigurðsson, and Joel Wallenberg. 2012. The Icelandic parsed historical corpus (IcePaHC). In LREC. pages 1977–1984. find our card
- Silke Scheible, Richard J Whitt, Martin Durrell, and Paul Bennett. 2011. A gold standard corpus of Early Modern German. In Proceedings of the 5th linguistic annotation workshop. Association for Computational Linguistics, pages 124–128. find our card
- Yves Scherrer and Tomaž Erjavec. 2013. Modernizing historical Slovene words with character-based SMT. In Proceedings of the 4th Biennial International Workshop on Balto-Slavic Natural Language Processing. Association for Computational Linguistics, Sofia, Bulgaria, pages 58–62. find
- Eszter Simon. 2014. Corpus building from Old Hungarian codices . find
- Ann Taylor, Arja Nurmi, Anthony Warner, Susan Pintzuk, and Terttu Nevalainen. 2006. The York-Helsinki Parsed Corpus of Early English Correspondence (PCEEC). Department of Linguistics, University of York. find
- Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning. pages 2048–2057. find
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/robertson-2018-evaluating-historical-text-normalization-systems/