Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer

What it is about

The paper builds the first large parallel corpus for formality transfer. It takes about 53,000 informal sentences each from two Yahoo Answers domains, filtered for length and form, and has crowd workers write one formal rewrite per sentence. Tune and test sets get four expert rewrites each, to reduce the noise of a single reference. Three systems are built and compared with the same rules: a rule-based editor (capitalize, expand contractions, normalize punctuation), a phrase-based MT system, and a neural encoder-decoder with copying, plus combined variants. Human judges rate formality, fluency and meaning. The rule-based editor keeps meaning best; the neural systems change the formality most, and the combined neural model is the most fluent. The paper also checks each automatic metric against human ranks. Formality and fluency metrics correlate moderately, while the meaning metric correlates poorly.

Ideas

• I1. Make register edits a rule layer on the surface. Capitalization, contraction expansion and punctuation are deterministic and keep meaning, so they can sit after en/src/lin.rs as an author-style layer. Check meaning preservation for each rule on held-out sentences.
• I2. Vet every style measure before reporting it. The paper correlates each automatic metric with human ranks and reports the result. Mova's style scores (see the Xu 2012 card) should go through the same check.
• I3. Use several references where the corpus allows. The Bible verses that appear in many versions can act as multi-reference sets for a style measure, which reduces the noise of a single reference.
• I4. The formality classifier can filter candidate outputs. The count form is the register lexicon score, a sum of per-word values over the chunk.

For Mova

The code is en/src/lin.rs (the rule layer) and tools/translate (measure.rs for the vetting step). The neural systems are not for Mova. The rule-based editor and the metric-vetting protocol are, and they address open problem 8.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1012

@inproceedings{rao2018dear,
  title = {{Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer}},
  author = {Rao, Sudha and Tetreault, Joel},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1012},
  url = {https://doi.org/10.18653/v1/n18-1012}
}

References

41 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michael Heilman, Aoife Cahill, Nitin Madnani, Melissa Lopez, Matthew Mulholland, and Joel Tetreault. 2014. Predicting grammaticality on an ordinal scale. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Baltimore, Maryland, pages 174–180. http://www.aclweb.org/ anthology/P14-2029. ↗
2. Eneko Agirre, Carmen Banea, Daniel M Cer, Mona T Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval@ NAACL-HLT. pages 497–511. find
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . ↗
4. Francis Heylighen and Jean-Marc Dewaele. 1999. Formality of language: definition, measurement and behavioral determinants. Interner Bericht, Center Leo Apostel, Vrije Universiteit Brüssel . find
5. Julian Brooke and Graeme Hirst. 2014. Supervised ranking of co-occurrence profiles for acquisition of continuous lexical attributes. In COLING. pages 2172–2183. find
6. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780. find
7. Julian Brooke, Tong Wang, and Graeme Hirst. 2010. Automatic acquisition of lexical formality. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters. Association for Computational Linguistics, pages 90–98. find
8. Eduard Hovy. 1987. Generating natural language under pragmatic constraints. Journal of Pragmatics 11(6):689–719. find
9. Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In International Conference on Machine Learning. pages 1587–1596. find
10. Chris Callison-Burch. 2008. Syntactic constraints on paraphrases extracted from parallel corpora. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Honolulu, Hawaii, pages 196–205. http://www.aclweb.org/ anthology/D08-1021. ↗
11. Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Eric Nyberg. 2017. Shakespearizing modern language using copy-enriched sequence-to-sequence models. Proceedings of the Workshop on Stylistic Variation, EMNLP 2017 pages 10–19. find our card
12. David L Chen and William B Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, pages 190–200. find
13. Tomoyuki Kajiwara and Mamoru Komachi. 2016. Building a monolingual parallel corpus for text simplification using sentence similarity based on alignment between word embeddings. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. pages 1147–1158. find
14. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder–decoder approaches. Syntax, Semantics and Structure in Statistical Translation page 103. find
15. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions. Association for Computational Linguistics, pages 177– 180. find
16. Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. Proceedings of the Workshop on Stylistic Variation, EMNLP 2017 . find
17. Hua He, Kevin Gimpel, and Jimmy J Lin. 2015. Multi-perspective sentence similarity modeling with convolutional neural networks. In EMNLP. pages 1576–1586. find
18. Shibamouli Lahiri, Prasenjit Mitra, and Xiaofei Lu. 2011. Informality judgment at sentence level and experiments with formality score. In International Conference on Intelligent Text Processing and Computational Linguistics. Springer, pages 446–457. find
19. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. Sofia, Bulgaria, pages 690–696. https://kheafield.com/papers/ edinburgh/estimate_paper.pdf. ↗
20. Robert C Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the ACL 2010 conference short papers. Association for Computational Linguistics, pages 220–224. find
21. Alejandro Mosquera and Paloma Moreda. 2012. Smile: An informality classification tool for helping to assess quality and credibility in web 2.0 texts. In Proceedings of the ICWSM workshop: Real-Time Analysis and Mining of Social Streams (RAMSS). 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 86–96. https://doi.org/10.18653/v1/ P16-1009. ↗
22. Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2016. There’s no comparison: Reference-less evaluation metrics in grammatical error correction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 2109–2115. https://aclweb.org/anthology/D16-1228. ↗
23. Fadi Abu Sheikha and Diana Inkpen. 2010. Automatic classification of documents by formality. In Natural Language Processing and Knowledge Engineering (NLP-KE), 2010 International Conference on. IEEE, pages 1–5. find
24. Fadi Abu Sheikha and Diana Inkpen. 2011. Generation of formal and informal sentences. In Proceedings of the 13th European Workshop on Natural Language Generation. Association for Computational Linguistics, pages 187–193. find
25. Xing Niu, Marianna Martindale, and Marine Carpuat. 2017. A study of style in machine translation: Controlling the formality of machine translation output. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pages 2804–2809. find
26. Matthew Snover, Nitin Madnani, Bonnie J Dorr, and Richard Schwartz. 2009. Fluency, adequacy, or hter?: exploring different human judgments with a tunable mt metric. In Proceedings of the Fourth Workshop on Statistical Machine Translation. Association for Computational Linguistics, pages 259– 268. find
27. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pages 311–318. find our card
28. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. pages 3104–3112. find
29. Ellie Pavlick and Ani Nenkova. 2015. Inducing lexical style properties for paraphrase and genre differentiation. In HLT-NAACL. pages 218–224. find
30. Nicola Ueffing. 2006. Self-training for machine translation. In NIPS workshop on Machine Learning for Multilingual Information Access. find
31. Ellie Pavlick and Joel Tetreault. 2016. An empirical analysis of formality in online communication. Transactions of the Association for Computational Linguistics 4:61–74. find our card
32. Tong Wang, Ping Chen, John Rochford, and Jipeng Qiang. 2016. Text simplification using neural machine translation. In AAAI. pages 4270–4271. find
33. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pages 1532–1543. find
34. Sander Wubben, Antal Van Den Bosch, and Emiel Krahmer. 2012a. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1. Association for Computational Linguistics, pages 1015– 1024. find our card
35. Kelly Peterson, Matt Hohensee, and Fei Xia. 2011. Email formality in the workplace: A case study on the enron corpus. In Proceedings of the Workshop on Languages in Social Media. Association for Computational Linguistics, pages 86–95. find
36. Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2012b. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Jeju Island, Korea, pages 1015–1024. http://www.aclweb.org/anthology/P12-1107. ↗ our card
37. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Controlling politeness in neural machine translation via side constraints. In HLT-NAACL. pages 35–40. find
38. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Edinburgh neural machine translation systems for wmt 16. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 371–376. http://www.aclweb.org/ anthology/W16-2323. ↗
39. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics 4:401–415. find our card
40. Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for style. Proceedings of COLING 2012 pages 2899–2914. find our card
41. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016c. Improving neural machine translation models with monolingual data. In Proceedings of the Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of the 23rd international conference on computational linguistics. Association for Computational Linguistics, pages 1353–1361. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/rao-2018-dear-sir-or-madam-may/