BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language model

What it is about

Personal pronouns are usually unstressed, so a stressed pronoun («HE loves her») signals contrastive focus that needs discourse knowledge to predict. From audiobooks of public-domain novels (five for training, ~41.6K utterances; one for testing, each read by three speakers) the authors derive three-level acoustic prominence labels and hand-check 493 contrastively stressed pronouns (393 stressed by at least two of three readers; rater kappa 0.85–0.90). A fine-tuned BERT predicting prominence per word is best overall but barely beats a per-word majority baseline, and on contrastive pronouns its recall is only about 0.24; giving it the previous one or two sentences helps little. A FastSpeech 2 model conditioned on the labels can then render the prominence, as a listening test confirms.

Ideas

• I1. Mova reads text, but the tales mark emphasis in writing: italics rendered as _he_, small capitals or CAPITALS in Gutenberg texts. A pronoun marked this way is contrastively focused, and a contrastive pronoun takes the antecedent the unmarked one would not (the parallel-role preference reverses: «John hit Bill and then he hit him» → he = Bill). Add an «emphasised pronoun» flag in the frame reader's pronoun resolution (prag/src/frames.rs, Ctx) that skips the top salience candidate in favour of the next agreeing one, with the reason printed. First count emphasised pronouns in the tale corpus; if there are few, record the rule in the chapter only.
• I2. Keep the emphasis marks through tokenisation into a token flag rather than stripping them, so that focus-sensitive readings (contrast, only) can use them; check where the text pipeline currently drops underscores.

For Mova

29. Ellipsis, word order and information structure (focus and contrast), 35-reference-and-coreference.md §2 (salience), 05-writing-system.md/07-punctuation-capitals-numbers.md (emphasis in writing). Code: prag/src/frames.rs (pronoun resolution), the en tokenizer. Measure: number of emphasised pronouns in the tales and frame bench coref on sentences that contain them.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2207.01718

@misc{stephenson2022bert,
  title = {{BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language model}},
  author = {Stephenson, Brooke and Besacier, Laurent and Girin, Laurent and Hueber, Thomas},
  year = {2022},
  eprint = {2207.01718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.01718}
}

References

28 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Rooth, “A theory of focus interpretation,” Natural Language Semantics, vol. 1, no. 1, pp. 75–116, 1992. find
2. T. Kenter, M. Sharma, and R. Clark, “Improving the prosody of RNN-based english text-to-speech synthesis by incorporating a BERT model,” in Proc. of Interspeech, Shanghai, China (virtual conference), 2020. find
3. Y. Zou, S. Liu, X. Yin, H. Lin, C. Wang, H. Zhang, and Z. Ma, “Fine-grained prosody modeling in neural speech synthesis using ToBI representation,” in Proc. of Interspeech, Brno, Czech Republik, 2021. find
4. A. Talman, A. Suni, H. Celikkanat, S. Kakouros, J. Tiedemann, and M. Vainio, “Predicting prosodic prominence from text with pre-trained contextualized word representations,” in Proc. of Nordic Conference on Computational Linguistics (NoDaLiDa), Turku, Finland, 2019. find
5. Y. Xiao, L. He, H. Ming, and F. K. Soong, “Improving prosody with linguistic and BERT derived features in multi-speaker based Mandarin Chinese neural TTS,” in Proc. of IEEE ICASSP, Barcelona, Spain (virtual conference), 2020. find
6. Z. Hodari, A. Moinet, S. Karlapati, J. Lorenzo-Trueba, T. Merritt, A. Joly, A. Abbas, P. Karanasou, and T. Drugman, “CAMP: a two-stage approach to modelling prosody in context,” in Proc. of IEEE ICASSP, Toronto, Canada, 2021. find
7. D. Stanton, Y. Wang, and R. Skerry-Ryan, “Predicting expressive speaking style from text in end-to-end speech synthesis,” in Proc. of IEEE Spoken Language Tech. Workshop (SLT), Athens, Greece, 2018. find
8. T. Hayashi, S. Watanabe, T. Toda, K. Takeda, S. Toshniwal, and K. Livescu, “Pre-trained text embeddings for enhanced text-to-speech synthesis,” in Proc. of Interspeech, Graz, Austria, 2019. find
9. A. Nenkova, J. Brenier, A. Kothari, S. Calhoun, L. Whitton, D. Beaver, and D. Jurafsky, “To memorize or to predict: Prominence labeling in conversational speech,” in Proc. of the Conf. of the North Am. Chapter of the Assoc. for Computational Linguistics, Rochester, New York, 2007. find
10. J. Hirschberg, “Pitch accent in context: Predicting intonational prominence from text,” Artificial Intelligence, vol. 63, no. 1–2, pp. 305–340, 1993. find
11. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. of ACL, Florence, Italy, 2019. find
12. A. Suni, J. Simko, D. Aalto, and M. Vainio, “Hierarchical representation and estimation of prosody using continuous wavelet transform,” Computer Speech and Language, vol. 45, pp. 123– 136, 2017. find
13. T. Raitio, R. Rasipuram, and D. Castellani, “Controllable neural text-to-speech synthesis using intuitive prosodic features,” in Proc. of Interspeech, Shanghai, China (virtual conference), 2020. find
14. Y. Wang, D. Stanton, Y. Zhang, R.-S. Ryan, E. Battenberg, J. Shor, Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous, “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,” in Int. Conf. on Machine Learning. PMLR, 2018. find
15. L. Badino, J. S. Andersson, J. Yamagishi, and R. A. Clark, “Identification of contrast and its emphatic realization in HMM based speech synthesis,” in Proc. of Interspeech, Brighton, UK, 2009. find
16. A. Suni, S. Kakouros, M. Vainio, and J. Šimko, “Prosodic prominence and boundaries in sequence-to-sequence speech synthesis,” in Proc. of ISCA Int. Conf. on Speech Prosody, Tokyo, Japan, 2020. find
17. S. Latif, I. Kim, I. Calapodescu, and L. Besacier, “Controlling prosody in end-to-end TTS: A case study on contrastive focus generation,” in Proc. of the Conf. on Computational Natural Language Learning (CoNLL), Punta Cana, Dominican Republic, 2021. find
18. J. Reeve, “Chapterize,” chapterize, 2016. https://github.com/JonathanReeve/ ↗
19. M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd, “spaCy: Industrial-strength Natural Language Processing in Python,” https://spacy.io/, 2020. ↗
20. M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal Forced Aligner: Trainable text-speech alignment using Kaldi,” in Proc. of Interspeech, Stockholm, Sweden, 2017. find
21. J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, 1960. find
22. Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947. find
23. Y. Ren, C. Hu, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fast-Speech 2: Fast and high-quality end-to-end text-to-speech,” arXiv preprint arXiv:2006.04558, 2020. ↗
24. C. Wang, W.-N. Hsu, Y. Adi, A. Polyak, A. Lee, P.-J. Chen, J. Gu, and J. Pino, “FAIRSEQ S 2 : A scalable and integrable speech synthesis toolkit,” arXiv preprint arXiv:2109.06912, 2021. ↗
25. R. Yamamoto, E. Song, and J.-M. Kim, “Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,” in Proc. of ICASSP, Barcelona, Spain (virtual conference), 2020. find
26. B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” in Proc. of Interspeech, Shanghai, China (virtual conference), 2020. find
27. M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio, “SpeechBrain: A general-purpose speech toolkit,” 2021, arXiv:2106.04624. ↗
28. N. Jillings, D. Moffat, B. De Man, and J. D. Reiss, “Web Audio Evaluation Tool: A browser-based listening test environment,” in Proc. of the Sound and Music Computing Conf., Maynooth, Ireland, 2015. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/stephenson-2022-bert-he-predict-contrastive-focus/