Semi-supervised Contextual Historical Text Normalization

What it is about

The paper normalises historical text with a simple generative model over characters, combined with a target-side language model that sees the context of each word. The model is trained with expectation-maximisation on unlabelled historical text. Across eight datasets, the error reduction against a character-level SMT baseline is about 19% with 500 labelled tokens and about 3% with 5,000. With no labelled training data, and only a 500-token development set, the best setting reaches 88.4% of the best fully supervised scores reported for the same datasets. Reranking adds about 5% of error reduction and bootstrapping about 14% in total. The English dataset has the largest share of identical spellings, at 75%.

Ideas

• I1. en/src/ctx.rs and en/src/morph.rs: a target-side model made of counts. Count modern-word bigrams from the EWT training split in a BTreeMap, and use them to rerank the archaic candidates for tools/gutenberg lessons. Measure accuracy on a 500-token dev set with and without the reranking.
• I2. Unlabelled data for the lookup: the in-code dictionary in en/src/lexgen.rs gives modern forms without labels. Run a simple EM over archaic-to-modern pairs from the lessons, and measure accuracy at 500, 1,000 and 5,000 tokens against the paper's curve.
• I3. Negative control for context: shuffle the context words in the reranker and expect the gain to disappear. If the gain stays, the reranker is not using context, and the gate should show red.

For Mova

The chapter is 09. Early Modern and archaic forms. The code is en/src/ctx.rs, en/src/morph.rs and en/src/lexgen.rs. The paper's English data is Middle English and its licence is not in our records, so check it before any use in a build. The neural parts are not for Mova; the count-based reranker is.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.650

@inproceedings{makarov2020semi,
  title = {{Semi-supervised Contextual Historical Text Normalization}},
  author = {Makarov, Peter and Clematide, Simon},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.650},
  url = {https://doi.org/10.18653/v1/2020.acl-main.650}
}

References

54 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alistair Baron and Paul Rayson. 2008. VARD2: A tool for dealing with spelling variation in historical corpora. In Postgraduate conference in corpus linguistics. 2 https://github.com/ZurichNLP/ acl2020-historical-text-normalization Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of machine learning research. ↗
2. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H Clark, and Philipp Koehn. 2013. Scalable modified kneser-ney language model estimation. In ACL. find
3. Taylor Berg-Kirkpatrick, Alexandre Bouchard-Côté, John DeNero, and Dan Klein. 2010. Painless unsupervised learning with features. In NAACL-HLT. find
4. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8). find
5. Taylor Berg-Kirkpatrick, Greg Durrett, and Dan Klein. 2013. Unsupervised transcription of historical documents. In ACL. find
6. Marcel Bollmann. 2012. (semi-)automatic normalization of historical texts using distance measures and the Norma tool. In Second Workshop on Annotation of Corpora for Research in theHumanities (ACRH-2). find
7. Marcel Bollmann. 2018. Normalization of historical texts with neural network models. Sprachwissenschaftliches Institut, Ruhr-Universität. find
8. Marcel Bollmann. 2019. A large-scale comparison of historical text normalization systems. In NAACL-HLT. find our card
9. Marcel Bollmann, Joachim Bingel, and Anders Søgaard. 2017. Learning attention for historical text normalization by learning to pronounce. In ACL. find our card
10. Marcel Bollmann, Florian Petran, and Stefanie Dipper. 2011. Rule-based normalization of historical texts. In Proceedings of the Workshop on Language Technologies for Digital Humanities and Cultural Heritage (LaTeCH). find our card
11. Marcel Bollmann, Anders Søgaard, and Joachim Bingel. 2018. Multi-task learning for historical text normalization: Size matters. In Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP. find our card
12. Peter F. Brown, Stephen A. Della-Pietra, Vincent J. Della-Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation. Computational Linguistics. find
13. Tomaz Erjavec. 2012. The goo300k corpus of historical Slovene. In LREC. find
14. Izaskun Etxeberria, Iñaki Alegria, Larraitz Uria, and Mans Hulden. 2016. Evaluating the noisy channel model for the normalization of historical texts: Basque, Spanish and Slovene. In LREC. find
15. Rosemarie Fiebranz, Erik Lindberg, Jonas Lindström, and Maria Ågren. 2011. Making verbs count: the research project ‘Gender and Work’ and its methodology. Scandinavian Economic History Review, 59(3). find
16. Dan Garrette, Jason Mielens, and Jason Baldridge. 2013. Real-world semi-supervised learning of pos-taggers for low-resource languages. In ACL. find
17. Bo Han and Timothy Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a #twitter. In ACL. find
18. Mark Hopkins and Jonathan May. 2011. Tuning as ranking. In EMNLP. find
19. Mans Hulden. 2009. Fast aproximate string matching with finite automata. Procesamiento del lenguaje natural, 43. find
20. Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. 1999. An introduction to variational methods for graphical models. Machine learning. find
21. Bryan Jurish. 2010. More than words: Using token context to improve canonicalization of historical german. Journal For Language Technology And Computational Linguistics. find
22. Katharina Kann, Kyunghyun Cho, and Samuel R Bowman. 2019. Towards realistic practices in low-resource natural language processing: The development set. In EMNLP. find
23. Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2019. Learning to discover, ground and use words with segmental neural language models. In ACL. find
24. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In ACL. find
25. Lingpeng Kong, Chris Dyer, and Noah A Smith. 2016. Segmental recurrent neural networks. In ICLR. find
26. Natalia Korchagina. 2017. Normalizing medieval german texts: from rules to deep learning. In Proceedings of the NoDaLiDa 2017 Workshop on Processing Historical Language. find
27. Julia Krasselt, Marcel Bollmann, Stefanie Dipper, and Florian Petran. 2015. Guidelines für die Nor-malisierung historischer deutscher Texte / Guidelines for Normalizing Historical German Texts. Bochumer Linguistische Arbeitsberichte: 15. find
28. Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics doklady, 10(8). find
29. Nikola Ljubešić, Katja Zupan, Darja Fišer, and Tomaž Erjavec. 2016. Normalising Slovene data: historical texts vs. user-generated content. In KONVENS. find
30. Massimo Lusetti, Tatyana Ruzsics, Anne Göhring, Tanja Samardžić, and Elisabeth Stark. 2018. Encoder-decoder methods for text normalization. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
31. Peter Makarov and Simon Clematide. 2018. Neural transition-based string transduction for limited-resource setting in morphology. In COLING. In Language and Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics. find
32. Manfred Markus. 1999. Manual of ICAMET (Innsbruck Computer Archive of Machine-Readable English Texts). Leopold-Franzens-Universitat Innsbruck. find
33. Rajhans Samdani, Ming-Wei Chang, and Dan Roth. 2012. Unified expectation maximization. In NAACL-HLT. find
34. Tomáš Mikolov. 2012. Statistical language models based on neural networks. Ph.D. thesis, Brno University of Technology, Czech Republic. find
35. Yves Scherrer and Nikola Ljubešić. 2016. Automatic normalisation of the Swiss German ArchiMob corpus using character-level machine translation. In KONVENS. find
36. Mehryar Mohri. 1997. Finite-state transducers in language and speech processing. Computational linguistics. find
37. Claude Elwood Shannon. 1948. A mathematical theory of communication. Bell system technical journal. find
38. Radford M Neal and Geoffrey E Hinton. 1998. A view of the em algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models. Springer. find
39. Eszter Simon. 2014. Corpus building from Old Hungarian codices. In The evolution of functional left peripheries in Hungarian syntax. find
40. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In ACL. find
41. Carolin Odebrecht, Malte Belz, Amir Zeldes, Anke Lüdeling, and Thomas Krause. 2017. RIDGES Her-bology: designing a diachronic multi-layer corpus. Language Resources and Evaluation, 51(3). find
42. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. Journal of machine learning research. find
43. Eva Pettersson. 2016. Spelling normalisation and linguistic analysis of historical text for information extraction. Ph.D. thesis, Acta Universitatis Upsaliensis. find
44. Eva Pettersson, Beáta Megyesi, and Joakim Nivre. 2013a. Normalisation of historical text using context-sensitive weighted Levenshtein distance and compound splitting. In NODALIDA. find our card
45. Eva Pettersson, Beáta Megyesi, and Jörg Tiedemann. 2013b. An SMT approach to automatic annotation of historical text. In Proceedings of the Nodalida Workshop on Computational Historical Linguistics. find
46. Sujith Ravi and Kevin Knight. 2011. Deciphering foreign language. In ACL-HLT. find
47. Alexander Robertson and Sharon Goldwater. 2018. Evaluating historical text normalization systems: How well do they generalize? In NAACL. find our card
48. Eirı́kur Rögnvaldsson, Anton Karl Ingason, Einar Freyr Sigurdhsson, and Joel Wallenberg. 2012. The icelandic parsed historical corpus (icepahc). In LREC. find our card
49. Tanja Samardžić, Yves Scherrer, and Elvira Glaser. 2015. Normalising orthographic and dialectal variants for the automatic processing of swiss german. find
50. Valentin I Spitkovsky, Hiyan Alshawi, Daniel Jurafsky, and Christopher D Manning. 2010. Viterbi training improves unsupervised dependency parsing. In CoNLL. find
51. Gongbo Tang, Fabienne Cap, Eva Pettersson, and Joakim Nivre. 2018. An evaluation of neural machine translation models on historical spelling normalization. In COLING. find
52. Gael Vaamonde. 2017. Userguide for digital edition of texts in PS Post Scriptum. find
53. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329. ↗
54. Matthew D Zeiler. 2012. ADADELTA: an adaptive learning rate method. arXiv:1212.5701. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/acl-2020-makarov-semi-supervised-contextual-historical-text/