Shakespearizing Modern Language Using Copy-Enriched Sequence to Sequence Models

What it is about

The paper turns modern English into Shakespearean English with a sentence-level encoder-decoder that has a pointer (copy) component, so it can copy words from the input. The parallel data are line-by-line modern paraphrases of sixteen Shakespeare plays, with about 18,000 training sentences and one play held out as test. Because the data are small, the authors pre-train word embeddings in two ways: with extra English text, and with a dictionary of Shakespearean-to-modern word pairs, using a retrofitting step that pulls linked words toward each other while the rest stay fixed. The best model reaches a BLEU score above 31, about six points over the strongest statistical baseline. Copying matters because many words are shared between the two styles, and the dictionary covers the archaic forms that the corpus rarely shows.

Ideas

• I1. Use the dictionary as a prior on lexicon merges. A pair of forms linked by a dictionary gets a bonus for merging into one concept in meta.rs, and a merge is kept only if reader and encoder bits go down (open problem 2). Run it twice on dev and compare the merge set.
• I2. Count, per concept, how often the standard encoder must output the source word itself. Concepts with high copy rates are the ones a delta does not need to touch. This gives a copy profile per author pair and names the words that get rewritten.
• I3. Check for gaps in the dictionary before counting merges. The paper adds the second-person forms by hand because they are frequent but missing from the dictionary. The same audit for thou, thee, ye, hath and doth would show which author features the lexicon cannot yet reach.
• I4. The neural retrofitting is not for Mova; the count form is a co-occurrence table where dictionary links act as hard constraints on merges.

More ideas (a second reading)

• I5. en/src/morph.rs: measure how many of the 1,524 original-to-modern pairs the archaic function already maps to the modern lemma, and which pairs it misses. The dictionary is a crawled list with no licence in our records, so use it for this test only.
• I6. en/src/lin.rs: an archaic surface generator as an option for en lin-eval. From a modern tree, produce thou, thee and -est forms from the lemma, and score the output against the Romeo and Juliet test set by exact match. This is a test, not a model.
• I7. Copy rule: in en/src/lin.rs, copy unchanged any word that has no dictionary entry. Measure the exact-match rate with and without the rule, on the same test set.

For Mova

The code is tools/translate (meta.rs, profile.rs). The paper is a neural system, so only its data design carries over: a small dictionary of author-to-standard pairs as a constraint on merges, and a copy profile as a measure. Open problems 2 and 6 are the targets. The reported BLEU belongs to the paper's own split and is not comparable to Mova's numbers.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-4902

@inproceedings{jhamtani2017shakespearizing,
  title = {{Shakespearizing Modern Language Using Copy-Enriched Sequence to Sequence Models}},
  author = {Jhamtani, Harsh and Gangal, Varun and Hovy, Eduard and Nyberg, Eric},
  booktitle = {Proceedings of the Workshop on Stylistic Variation},
  year = {2017},
  doi = {10.18653/v1/w17-4902},
  url = {https://doi.org/10.18653/v1/w17-4902}
}

References

27 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473. ↗
2. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pages 160–167. Association for Computational Linguistics. find
3. Roger Brown, Albert Gilman, et al. 1960. The pronouns of power and solidarity. Article. find
4. David L Chen and William B Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 190–200. Association for Computational Linguistics. find
5. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
6. Manaal Faruqui, Jesse Dodge, Sujay K Jauhar, Chris Dyer, Eduard Hovy, and Noah A Smith. 2014. Retrofitting word vectors to semantic lexicons. arXiv preprint arXiv:1411.4166. ↗
7. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global Vectors for Word Representation. In EMNLP, volume 14, pages 1532–1543. find
8. Varun Gangal, Harsh Jhamtani, Graham Neubig, Eduard Hovy, and Eric Nyberg. 2017. Charmanteau: Character embedding models for portmanteau creation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Copenhagen, Denmark. find
9. Paul Rayson, Dawn Archer, Alistair Baron, Jonathan Culpeper, and Nicholas Smith. 2007. Tagging the Bard: Evaluating the accuracy of a modern POS tagger on Early Modern English corpora. Article. find
10. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685. ↗
11. Gaurush Hiranandani, Pranav Maneriker, and Harsh Jhamtani. 2017. Generating appealing brand names. arXiv preprint arXiv:1706.09335. ↗
12. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
13. Rishiraj Saha Roy, Aishwarya Padmakumar, Guna Prasaad Jeganathan, and Ponnurangam Kumaraguru. 2015. Automated Linguistic Personalization of Targeted Marketing Messages Mining User-Generated Text on Social Media. In 16th International Conference on Intelligent Text Processing and Computational Linguistics 2015 (CICLing ’15), pages 203–224. Springer International Publishing. find
14. Eliza Kitis. 1997. Adspart of our lives: linguistic awareness of powerful advertising. Word & Image, 13(3):304–313. find
15. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions, pages 177–180. Association for Computational Linguistics. find
16. H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Lukasz Dziurzynski, Stephanie M Ramones, Megha Agrawal, Achal Shah, Michal Kosinski, David Stillwell, Martin EP Seligman, et al. 2013. Personality, gender, and age in the language of social media: The open-vocabulary approach. PloS one, 8(9):e73791. find
17. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. arXiv preprint arXiv:1704.04368. ↗
18. Pranava Swaroop Madhyastha, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Mapping unseen words to task-trained embedding spaces. arXiv preprint arXiv:1510.02387. ↗
19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Controlling politeness in neural machine translation via side constraints. In Proceedings of NAACL-HLT, pages 35–40. find
20. Balaji Vasan Srinivasan, Rishiraj Saha Roy, Harsh Jhamtani, Natwar Modani, and Niyati Chhaya. 2017. Corpus-based automatic text expansion. In CICLING. find
21. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Neural information processing systems, pages 3104–3112. find
22. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692–2700. find
23. Shuohang Wang and Jing Jiang. 2016. Machine comprehension using match-lstm and answer pointer. arXiv preprint arXiv:1608.07905. ↗
24. Wei Xu. 2014. Data-driven approaches for paraphrasing across language variations. Ph.D. thesis, New York University. find
25. Wei Xu, Alan Ritter, William B Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for style. In 24th International Conference on Computational Linguistics, COLING 2012. find our card
26. Wenyuan Zeng, Wenjie Luo, Sanja Fidler, and Raquel Urtasun. 2016. Efficient Summarization with Read-Again and Copy Mechanism. arXiv preprint arXiv:1611.03382. ↗
27. Barret Zoph and Kevin Knight. 2016. source neural translation. arXiv arXiv:1601.00710. Multi-preprint ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/jhamtani-2017-shakespearizing-modern-language-copy-enriched/