Spelling Normalization of Historical Documents by Using a Machine Translation Approach

What it is about

The paper treats spelling normalization as translation from historical spelling to modern spelling. It tests three approaches on four historical corpora from three periods, in two languages and three alphabets: phrase-based statistical MT, neural MT, and character-based MT, where each word is split into characters. It adds a very simple baseline, a statistical dictionary built from how often each word was changed in training, applied word by word and left alone if unknown. Character-based statistical MT is best on most corpora. Neural MT with word or subword units learns poorly on the small corpora, and the character-based neural model is the worst on the smallest. The dictionary baseline comes close to the learned systems. Word-level scores such as BLEU and TER understate the gains, since spelling changes touch only a few letters per word; character error rate shows them better.

Ideas

• I1. Keep the statistical dictionary as the control. A table of how often a word is changed in the training pairs, applied only when seen, is the standard encoder with its lookup. Report its numbers before any learned delta is credited.
• I2. Use character-level chunks for unseen words. Character-based statistical MT is the best of these systems, so for the morph fallback of an unseen concept a character n-gram count model, built from the target author's spellings, is the count-based analogue.
• I3. Add a character-level measure next to chrF in normal mode. Word-level scores penalize small spelling changes heavily and make a correct but respelled word look wrong.
• I4. Avoid the small-corpus regime for neural models. The paper's neural results on the smallest corpora are poor, which argues for keeping the translator count-based until a corpus is large.

For Mova

The code is tools/translate (profile.rs for the dictionary control and the unseen-word fallback, measure.rs for the character-level measure). The paper's MT systems are not for Mova. Its dictionary baseline and its metric warning are, and they support the control protocol of open problems 2 and 6.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{domingo2018spelling,
  title = {{Spelling Normalization of Historical Documents by Using a Machine Translation Approach}},
  author = {Domingo, Miguel and Casacuberta, Francisco},
  booktitle = {Proceedings of the 21st Annual Conference of the European Association for Machine Translation},
  year = {2018},
  url = {https://aclanthology.org/2018.eamt-main.13/}
}

References

41 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bahdanau, D., Cho, K., and Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations (arXiv:1409.0473). ↗
2. Baron, A. and Rayson, P. (2008). VARD2: A tool for dealing with spelling variation in historical corpora. Postgraduate conference in corpus linguistics. find
3. Bollmann, M. and Søgaard, A. (2016). Improving historical spelling normalization with bidirectional lstms and multi-task learning. In Proceedings of the International Conference on the Computational Linguistics, pages 131–139. find
4. Britz, D., Goldie, A., Luong, T., and Le, Q. (2017). Massive exploration of neural machine translation architectures. arXiv preprint arXiv:1703.03906. ↗
5. Brown, P. F., Pietra, V. J. D., Pietra, S. A. D., and Mercer, R. L. (1993). The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. find
6. Chen, S. F. and Goodman, J. (1996). An empirical study of smoothing techniques for language modeling. In Proceedings of the Annual Meeting on Association for Computational Linguistics, pages 310–318. find
7. Chung, J., Cho, K., and Bengio, Y. (2016). A character-level decoder without explicit segmentation for neural machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1693–1703. find
8. Costa-Jussà, M. R., Aldón, D., and Fonollosa, J. A. (2017). Chinese–spanish neural machine translation enhanced with character and word bitmap fonts. Machine Translation, 31:35–47. find
9. Costa-Jussà, M. R. and Fonollosa, J. A. (2016). Character-based neural machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 357–361. find
10. Domingo, M., Chinea-Rios, M., and Casacuberta, F. (2017). Historical documents modernization. The Prague Bulletin of Mathematical Linguistics, 108:295–306. find
11. F. Jehle, F. (2001). Works of Miguel de Cervantes in Old- and Modern-spelling. Indiana University Purdue University Fort Wayne. find
12. Gao, Q. and Vogel, S. (2008). Parallel implementations of word alignment tool. In Proceedings of the Association for Computational Linguistics Software Engineering, Testing, and Quality Assurance Workshop, pages 49–57. find
13. Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8):1735–1780. find
14. Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
15. Klein, G., Kim, Y., Deng, Y., Senellart, J., and Rush, A. M. (2017). OpenNMT: Open-Source Toolkit for Neural Machine Translation. arXiv preprint arXiv:1701.02810. ↗
16. Koehn, P. (2004). Statistical significance tests for machine translation evaluation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 388–395. find
17. Koehn, P. (2010). Statistical Machine Translation. Cambridge University Press. find
18. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 177–180. find
19. Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase-based translation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, pages 48–54. find
20. Korchagina, N. (2017). Normalizing medieval german texts: from rules to deep learning. In Proceedings of the Nordic Conference on Computational Linguistics Workshop on Processing Historical Language, pages 12–17. find
21. Ling, W., Trancoso, I., Dyer, C., and Black, A. W. (2015). Character-based neural machine translation. arXiv preprint arXiv:1511.04586. ↗
22. Ljubešić, N., Zupan, K., Fišer, D., and Erjavec, T. (2016). Dataset of normalised slovene text KonvNormSl 1.0. Slovenian language resource repository CLARIN.SI. http://hdl.handle.net/11356/1068. ↗
23. Ljubešic, N., Zupan, K., Fišer, D., and Erjavec, T. (2016). Normalising slovene data: historical texts vs. user-generated content. In Proceedings of the Conference on Natural Language Processing, pages 146–155. find
24. Nakov, P. and Tiedemann, J. (2012). Combining word-level and character-level models for machine translation between closely-related languages. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 301–305. find
25. Och, F. J. (2003). Minimum error rate training in statistical machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 160–167. find
26. Och, F. J. and Ney, H. (2002). Discriminative training and maximum entropy models for statistical machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 295–302. find
27. Och, F. J. and Ney, H. (2003). A systematic comparison of various statistical alignment models. Computational Linguistic, 29(1):19–51. find
28. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a method for automatic evaluation of machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 311–318. find our card
29. Pascanu, R., Gulcehre, C., Cho, K., and Bengio, Y. (2013). How to construct deep recurrent neural networks. arXiv preprint arXiv:1312.6026. ↗
30. Piotrowski, M. (2012). Natural Language Processing for Historical Texts. Number 17 in Synthesis Lectures on Human Language Technologies. Morgan & Claypool. find
31. Porta, J., Sancho, J.-L., and Gómez, J. (2013). Edit transducers for spelling variation in old spanish. In Proceedings of the workshop on computational historical linguistics, pages 70–79. find
32. Scherrer, Y. and Erjavec, T. (2013). Modernizing historical slovene words with character-based smt. In Proceedings of the Biennial International Workshop on Balto-Slavic Natural Language Processing, pages 58–62. find
33. Schuster, M. and Paliwal, K. K. (1997). Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673–2681. find
34. Sennrich, R., Haddow, B., and Birch, A. (2015). Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. ↗
35. Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1715–1725. find
36. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In Proceedings of the Association for Machine Translation in the Americas, pages 223–231. find our card
37. Stolcke, A. (2002). SRILM - an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing, pages 257–286. find
38. Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to sequence learning with neural networks. find
39. Tiedemann, J. (2009). Character-based PSMT for closely related languages. In Proceedings of the Annual Conference of the European Association for Machine Translation, pages 12–19. find
40. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ↗
41. Zens, R., Och, F. J., and Ney, H. (2002). Phrase-based statistical machine translation. In Proceedings of the Annual German Conference on Advances in Artificial Intelligence, volume 2479, pages 18–32. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/domingo-2018-spelling-normalization-historical-documents-machine/