Few-Shot and Zero-Shot Learning for Historical Text Normalization

What it is about

The paper evaluates 63 multi-task configurations of an encoder-decoder normaliser on ten datasets in eight languages. The auxiliary tasks are autoencoding, grapheme-to-phoneme mapping and lemmatisation. The configurations differ in which encoder and decoder parts are shared. Multi-task learning gives consistent gains when the target data is small and little or none when it is large. On the English set with 1,000 training tokens, the best configuration reaches 75.9% dev accuracy, the worst 58.6%, and 49 configurations beat the single-task model. Sharing more components is generally better. Zero-shot learning beats the identity baseline, in which a word is left unchanged.

Ideas

• I1. Sharing as a rule: in en/src/morph.rs, let the candidate ranking for archaic forms share one lookup table for the form-to-lemma and the lemma-to-form steps. Measure accuracy on 1,000 tokens of tools/gutenberg lessons against separate tables.
• I2. Lemmatisation as an auxiliary signal: use the lemma accuracy of en morph-eval (99.3–99.5% on modern words) as a check for the archaic lookup. Measure archaic-form accuracy on the lessons with and without the check.
• I3. Baseline rows: every historical test table should start with the identity baseline and a dictionary-only lookup, as the paper's zero-shot comparison does. Report both with the date of the run.

For Mova

The chapter is 09. Early Modern and archaic forms. The code is en/src/morph.rs and en morph-eval. The paper's English set has no licence in our records, so check it before any use in a build. The neural configurations are not for Mova; the sharing idea and the baseline rows are.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-6112

@inproceedings{bollmann2019few,
  title = {{Few-Shot and Zero-Shot Learning for Historical Text Normalization}},
  author = {Bollmann, Marcel and Korchagina, Natalia and Søgaard, Anders},
  booktitle = {Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)},
  year = {2019},
  doi = {10.18653/v1/d19-6112},
  url = {https://doi.org/10.18653/v1/d19-6112}
}

References

38 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Linguistics: Technical Papers, pages 1903–1913, Osaka, Japan. find
2. The COLING 2016 Organizing Committee. find
3. Antonios Anastasopoulos and David Chiang. 2018. Tied multitask learning for neural speech translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 82–91, New Orleans, Louisiana. Association for Computational Linguistics. find
4. Rich Caruana. 1993. Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the 10th International Conference on Machine Learning (ICML), pages 41–48. find
5. Aliya Deri and Kevin Knight. 2016. Grapheme-to-phoneme models for (almost) any language. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 399–408, Berlin, Germany. Association for Computational Linguistics. find
6. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. ↗
7. Miguel Domingo and Francisco Casacuberta. 2018. Spelling normalization of historical documents by using a machine translation approach. In Proceedings of the 21st Annual Conference of the European Association for Machine Translation, pages 129– 137. find our card
8. Alistair Baron and Paul Rayson. 2008. VARD 2: A tool for dealing with spelling variation in historical corpora. In Proceedings of the Postgraduate Conference in Corpus Linguistics. find our card
9. Joachim Bingel and Anders Søgaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 164–169, Valencia, Spain. Association for Computational Linguistics. find
10. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1723–1732, Beijing, China. Association for Computational Linguistics. find
11. Marcel Bollmann. 2012. (Semi-)automatic normalization of historical texts using distance measures and the Norma tool. In Proceedings of the Second Workshop on Annotation of Corpora for Research in the Humanities (ACRH-2), Lisbon, Portugal. find
12. Izaskun Etxeberria, Iñaki Alegria, Larraitz Uria, and Mans Hulden. 2016. Evaluating the noisy channel model for the normalization of historical texts: Basque, Spanish and Slovene. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 1064– 1069, Portorož, Slovenia. European Language Resources Association (ELRA). find
13. Marcel Bollmann. 2018. Normalization of historical texts with neural network models. Bochumer Linguistische Arbeitsberichte, 22. find
14. Marcel Bollmann. 2019. A large-scale comparison of historical text normalization systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3885–3898. Association for Computational Linguistics. find our card
15. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. CoRR, abs/1601.01073. ↗
16. Mika Hämäläinen, Tanja Säily, Jack Rueter, Jörg Tiedemann, and Eetu Mäkelä. 2018. Normalizing early english letters to present-day english spelling. In Proceedings of the Second Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 87–96. Association for Computational Linguistics. find our card
17. Marcel Bollmann, Joachim Bingel, and Anders Søgaard. 2017. Learning attention for historical text normalization by learning to pronounce. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 332–344, Vancouver, Canada. Association for Computational Linguistics. find our card
18. Marcel Bollmann, Anders Søgaard, and Joachim Bingel. 2018. Multi-task learning for historical text normalization: Size matters. In Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP, pages 19–24. Association for Computational Linguistics. find our card
19. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s multilingual neural machine translation system: Enabling zero-shot translation. CoRR, abs/1611.04558. ↗
20. Chloé Braud, Barbara Plank, and Anders Søgaard. 2016. Multi-view and multi-task training of RST discourse parsers. In Proceedings of COLING 2016, the 26th International Conference on Computational Bryan Jurish. 2010. More than words: using token context to improve canonicalization of historical German. Journal for Language Technology and Computational Linguistics, 25(1):23–39. NLP Systems, pages 19–26, Copenhagen, Denmark. Association for Computational Linguistics. find
21. Katharina Kann, Ryan Cotterell, and Hinrich Schütze. 2017. One-shot neural cross-lingual transfer for paradigm completion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1993–2003, Vancouver, Canada. Association for Computational Linguistics. find
22. Eva Pettersson, Beáta Megyesi, and Joakim Nivre. 2013a. Normalisation of historical text using context-sensitive weighted Levenshtein distance and compound splitting. In Proceedings of the 19th Nordic Conference of Computational Linguistics (NODALIDA 2013), pages 163–179, Oslo, Norway. Linköping University Electronic Press. find our card
23. Akisato Kimura, Zoubin Ghahramani, Koh Takeuchi, Tomoharu Iwata, and Naonori Ueda. 2018. Few-shot learning of neural networks from scratch by pseudo example optimization. arXiv e-prints, abs/1802.03039. ↗
24. Eva Pettersson, Beáta Megyesi, and Jörg Tiedemann. 2013b. An SMT approach to automatic annotation of historical text. In Proceedings of the Workshop on Computational Historical Linguistics at NODAL-IDA 2013, NEALT Proceedings Series 18, pages 54– 69. Linköping University Electronic Press. find
25. Eliyahu Kiperwasser and Miguel Ballesteros. 2018. Scheduled multi-task learning: From syntax to translation. Transactions of the Association for Computational Linguistics, 6:225–240. find
26. Michael Piotrowski. 2012. Natural Language Processing for Historical Texts. Number 17 in Synthesis Lectures on Human Language Technologies. Morgan & Claypool. find
27. Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Graldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Arya D. McCarthy, Sandra Kbler, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. UniMorph 2.0: Universal morphology. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 1868–1873. European Language Resources Association (ELRA). find
28. Jordi Porta, José-Luis Sancho, and Javier Gómez. 2013. Edit transducers for spelling variation in Old Spanish. In Proceedings of the Workshop on Computational Historical Linguistics at NODAL-IDA 2013, NEALT Proceedings Series 18, pages 70– 79. Linköping University Electronic Press. find
29. Natalia Korchagina. 2017. Normalizing medieval German texts: from rules to deep learning. In Proceedings of the NoDaLiDa 2017 Workshop on Processing Historical Language, pages 12–17, Gothenburg. Linköping University Electronic Press. find
30. Alexander Robertson and Sharon Goldwater. 2018. Evaluating historical text normalization systems: How well do they generalize? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 720–725. Association for Computational Linguistics. find our card
31. Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015. Multi-task sequence to sequence learning. CoRR, abs/1511.06114. ↗
32. Yves Scherrer and Tomaž Erjavec. 2016. Modernising historical Slovene words. Natural Language Engineering, 22(6):881–905. find
33. Benjamin Milde, Christoph Schmidt, and Joachim Khler. 2017. Multitask sequence-to-sequence models for grapheme-to-phoneme conversion. In Proceedings of Interspeech 2017, pages 2536–2540. find
34. Gongbo Tang, Fabienne Cap, Eva Pettersson, and Joakim Nivre. 2018. An evaluation of neural machine translation models on historical spelling normalization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1320–1331, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
35. Graham Neubig, Matthias Sperber, Xinyi Wang, Matthieu Felix, Austin Matthews, Sarguna Padmanabhan, Ye Qi, Devendra Singh Sachan, Philip Arthur, Pierre Godard, John Hewitt, Rachid Riad, and Liming Wang. 2018. XNMT: The extensible neural machine translation toolkit. In Conference of the Association for Machine Translation in the Americas (AMTA) Open Source Software Showcase, Boston. find
36. Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. 2016. Multi-task cross-lingual sequence tagging from scratch. CoRR, abs/1603.06270. ↗
37. Nanyun Peng and Mark Dredze. 2017. Multi-task domain adaptation for sequence tagging. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 91–100, Vancouver, Canada. Association for Computational Linguistics. find
38. Ben Peters, Jon Dehdari, and Josef van Genabith. 2017. Massively multilingual neural grapheme-to-phoneme conversion. In Proceedings of the First Workshop on Building Linguistically Generalizable find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2019-bollmann-few-shot-zero-shot-learning/