Low-Resource Authorship Style Transfer: Can Non-Famous Authors Be Imitated?

What it is about

The paper defines low-resource authorship style transfer: rewriting text in the style of a target author who has only about 16 posts, roughly 500 tokens. Source and target authors are Reddit users. The STRAP paraphrase method, which works well when the target has a book's worth of text, degrades sharply as target text shrinks. The paper's own baseline, STYLL, is an in-context few-shot prompt built on a large language model; it is the strongest baseline tested. Style accuracy is measured with an authorship-representation model: the output should be attributed to the target author. Meaning is scored by a similarity measure, and a joint geometric score combines the two. Human judges agree with the representation model at a similar rate (63.4% against 62.2%). Used as an adversary to an identification model over about 111,000 authors, STYLL pushes the target author into the top eight in 12 to 13% of cases in the single-topic setting, and the source author almost never. The paper concludes that the task is feasible but not solved.

Ideas

• I1. Learning curve per author. Measure the translator's gain from a delta as a function of the number of verses per author, and check where the ≥3-token rule starts to fail on small authors. The paper's sharp drop below a few hundred tokens suggests the test should be done on subsets.
• I2. Nearest-author identification as a style metric. For a delta-transferred verse, rank the target author among all authors in the corpus by a count-based profile (concept-tag frequencies). The rank of the target is a style-accuracy measure that needs no learned embedding.
• I3. Adversarial probe. Ask whether an output still identifies the source author; a delta that keeps the source identity in the top ranks is a weak delta.

For Mova

The authorship-identification idea maps onto tools/translate (deltas and measure.rs) as a count-based rank test, which is cheap and easy to run twice for determinism. The paper's large-model baselines are not for Mova; only the evaluation design and the learning-curve question carry over.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2212.08986

@misc{patel2022low,
  title = {{Low-Resource Authorship Style Transfer: Can Non-Famous Authors Be Imitated?}},
  author = {Patel, Ajay and Andrews, Nicholas and Callison-Burch, Chris},
  year = {2022},
  eprint = {2212.08986},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.08986}
}

References

33 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Babakov, N.; Dale, D.; Logacheva, V.; and Panchenko, A. 2022. A large-scale computational study of content preservation measures for text style transfer and paraphrase generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, 300–321. Dublin, Ireland: Association for Computational Linguistics. find
2. BigScience. 2022. BigScience Language Open-science Open-access Multilingual (BLOOM) Language Model. https://huggingface.co/bigscience/bloom-7b1. ↗
3. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020. Language Models are Few-Shot Learners. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M. F.; and Lin, H., eds., Advances in Neural Information Processing Systems, volume 33, 1877– 1901. Curran Associates, Inc. find
4. Callison-Burch, C.; Osborne, M.; and Koehn, P. 2006. Re-evaluating the role of BLEU in machine translation research. In 11th conference of the european chapter of the association for computational linguistics, 249–256. find
5. Carlson, K.; Riddell, A.; and Rockmore, D. 2018. Evaluating prose style transfer with the Bible. Royal Society open science, 5(10): 171920. find our card
6. Cer, D.; Yang, Y.; Kong, S.-y.; Hua, N.; Limtiaco, N.; John, R. S.; Constant, N.; Guajardo-Cespedes, M.; Yuan, S.; Tar, C.; et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175. ↗
7. Chung, H. W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y.; Fedus, W.; Li, E.; Wang, X.; Dehghani, M.; Brahma, S.; et al. 2022. Scaling Instruction-Finetuned Language Models. arXiv preprint arXiv:2210.11416. ↗
8. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
9. Haddow, B.; Bawden, R.; Miceli Barone, A. V.; Helcl, J.; and Birch, A. 2022. Survey of Low-Resource Machine Translation. Computational Linguistics, 1–60. find
10. Iglesias-Flores, R.; Mishra, M.; Patel, A.; Malhotra, A.; Kriz, R.; Palmer, M.; and Callison-Burch, C. 2021. Top-GuNN: Fast NLP Training Data Augmentation using Large Corpora. In Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances, 86– 101. Online: Association for Computational Linguistics. find
11. Jin, D.; Jin, Z.; Hu, Z.; Vechtomova, O.; and Mihalcea, R. 2022. Deep Learning for Text Style Transfer: A Survey. Computational Linguistics, 48(1): 155–205. find
12. Johnson, J.; Douze, M.; and Jégou, H. 2021. Billion-Scale Similarity Search with GPUs. IEEE Transactions on Big Data, 7: 535–547. find
13. Khan, A.; Fleming, E.; Schofield, N.; Bishop, M.; and Andrews, N. 2021. A deep metric learning approach to account linking. arXiv preprint arXiv:2105.07263. ↗
14. Krishna, K.; Wieting, J.; and Iyyer, M. 2020. Reformulating Unsupervised Style Transfer as Paraphrase Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 737–762. find our card
15. Lample, G.; Subramanian, S.; Smith, E.; Denoyer, L.; Ranzato, M.; and Boureau, Y.-L. 2018. Multiple-attribute text rewriting. In International Conference on Learning Representations. find
16. Lample, G.; Subramanian, S.; Smith, E. M.; Denoyer, L.; Ranzato, M.; and Boureau, Y.-L. 2019. Multiple-Attribute Text Rewriting. In ICLR. find
17. Landis, J. R.; and Koch, G. G. 1977. The measurement of observer agreement for categorical data. Biometrics, 33 1: 159–74. find
18. Li, J.; Jia, R.; He, H.; and Liang, P. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. arXiv preprint arXiv:1804.06437. ↗ our card
19. Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
20. Patel, A.; Li, B.; Rasooli, M. S.; Constant, N.; Raffel, C.; and Callison-Burch, C. 2022. Bidirectional Language Models Are Also Few-shot Learners. find
21. Press, O.; Zhang, M.; Min, S.; Schmidt, L.; Smith, N. A.; and Lewis, M. 2022. Measuring and Narrowing the Compositionality Gap in Language Models. arXiv preprint arXiv:2210.03350. ↗
22. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019. Language Models are Unsupervised Multitask Learners. OpenAI. find
23. Reif, E.; Ippolito, D.; Yuan, A.; Coenen, A.; Callison-Burch, C.; and Wei, J. 2022. A Recipe for Arbitrary Text Style Transfer with Large Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 837–848. Dublin, Ireland: Association for Computational Linguistics. find
24. Riley, P.; Constant, N.; Guo, M.; Kumar, G.; Uthus, D. C.; and Parekh, Z. 2021. TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 3786–3800. find
25. Rivera-Soto, R. A.; Miano, O. E.; Ordonez, J.; Chen, B. Y.; Khan, A.; Bishop, M.; and Andrews, N. 2021. Learning Universal Authorship Representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 913–919. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics. find
26. Sudhakar, A.; Upadhyay, B.; and Maheswaran, A. 2019. “Transforming” Delete, Retrieve, Generate Approach for Controlled Text Style Transfer. In EMNLP. find
27. Wang, B.; and Komatsuzaki, A. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax. ↗
28. Warstadt, A.; Singh, A.; and Bowman, S. R. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7: 625–641. find
29. Wegmann, A.; Schraagen, M.; and Nguyen, D. 2022. Same Author or Just Same Topic? Towards Content-Independent Style Representations. In Proceedings of the 7th Workshop on Representation Learning for NLP, 249–268. find
30. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Chi, E.; Le, Q.; and Zhou, D. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. ↗
31. Xu, P.; Cheung, J. C. K.; and Cao, Y. 2020. On Variational Learning of Controllable Representations for Text without Supervision. In III, H. D.; and Singh, A., eds., Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, 10534–10543. PMLR. find
32. Xu, W.; Ritter, A.; Dolan, W. B.; Grishman, R.; and Cherry, C. 2012. Paraphrasing for style. In Proceedings of COLING 2012, 2899–2914. find our card
33. Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/patel-2022-low-resource-authorship-style-transfer/