Multi-task learning for historical text normalization: Size matters

What it is about

The paper asks when multi-task learning helps historical normalisation. It tests an encoder-decoder model with and without auxiliary tasks on 10 datasets from 8 languages, from German and English to Hungarian, Icelandic, Slovene, Spanish, Portuguese and Swedish. Most datasets either always gain or always lose from the auxiliary task. The four that consistently gain (German RIDGES, Icelandic, Slovene Bohorič and Swedish) are the smallest, with about 50,000 tokens or less. In the sparse setting, with 5,000 training tokens, multi-task learning almost always helps. On the sparse English set, a Twitter normalisation dataset helps more than the best historical auxiliary set.

Ideas

• I1. Size gate: before adding any auxiliary source to a Mova normaliser, check the size of the target lessons. Test an auxiliary source only on sparse targets of about 5,000 tokens, and report the gain with its date.
• I2. In-language auxiliary data: the paper finds a modern in-language dataset helps the sparse English target. Test the lemma list in en/src/lexgen.rs (110,308 lemmas) as the auxiliary source for the archaic-form rules on 5,000 tokens of tools/gutenberg text.
• I3. Decide by two-fold check: a gain on one split is not a decision. Use en folds on the sparse setting, with a fixed seed, and keep the held-out part for the release test only.

For Mova

The chapter is 09. Early Modern and archaic forms. The code is en/src/morph.rs, en/src/lexgen.rs and en folds. Check the licence of the historical English dataset before any use in a build; it is not in our records. The neural model is not for Mova; the size rule and the two-fold check are.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-3403

@inproceedings{bollmann2018multi,
  title = {{Multi-task learning for historical text normalization: Size matters}},
  author = {Bollmann, Marcel and Søgaard, Anders and Bingel, Joachim},
  booktitle = {Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP},
  year = {2018},
  doi = {10.18653/v1/w18-3403},
  url = {https://doi.org/10.18653/v1/w18-3403}
}

References

1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joachim Bingel and Anders Søgaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 164–169. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2018-bollmann-multi-task-learning-historical-text/