Enhancing TTS Stability in Hebrew using Discrete Semantic Units

What it is about

The paper presents LoTHM, a text-to-speech language model for unvocalised Hebrew. It predicts HuBERT semantic units rather than acoustic codec tokens, so no separate diacritisation step is needed; a vocoder with a speaker embedding then turns the units into audio. The training data is about 4,500 hours of Hebrew podcast speech (HebDB and ivrit.ai), plus English data for a second model. The argument is about stability: acoustic-token models need best-of-three sampling to get the words right. On 100 SASPEECH prompts, the single-sample WER falls from 0.34 (HebTTS) to 0.17, and best-of-three gives 0.10. A 10-item human study with native raters covers homographs whose pronunciation depends on context.

Ideas

• I1. Do not commit early to one reading. The method drops the diacritisation step that picks a single reading before synthesis. For tools/hbo in unpointed mode, carry the candidate splits of an unpointed word (from the prefix and lexicon tables) as a small lattice to the parser, instead of one split, and measure against a gold set whether this lowers errors.
• I2. Measure stability as spread: run a component several ways and report the disagreement, not one run only. Mova's parser is deterministic, so the analogue is the number of ambiguous words per sentence, reported next to accuracy.
• I3. A small homograph test set, each item a sentence where the same spelling has different readings or senses depending on context (for example Hebrew ספר, "book" or "counted"). Mova can hold such items as tests with gold readings.

For Mova

I1 is a design test for tools/hbo (unpointed mode). I2 is a reporting rule for the same tool and for 11. Modern Hebrew in Universal Dependencies. I3 is a test set for 2. Script and spelling, since it checks how spelling maps to meaning. The speech system itself is not for Mova.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2410.21502

@misc{zeldes2024enhancing,
  title = {{Enhancing TTS Stability in Hebrew using Discrete Semantic Units}},
  author = {Zeldes, Ella and Tal, Or and Adi, Yossi},
  year = {2024},
  eprint = {2410.21502},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.21502}
}

References

37 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, pages 1298–1312. PMLR, 2022. find
2. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460, 2020. find
3. Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers. arXiv preprint arXiv:2406.05370, 2024. ↗
4. Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022. ↗
5. Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification. arXiv preprint arXiv:2005.07143, 2020. ↗
6. Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEEE Transactions on acoustics, speech, and signal processing, 32(2):236–243, 1984. find
7. Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, et al. Textually pretrained speech language models. Advances in Neural Information Processing Systems, 36, 2024. find
8. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460, 2021. find
9. Eugene Kharitonov, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Paden Tomasello, Ann Lee, Ali Elkahky, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, et al. textless-lib: A library for textless spoken language processing. arXiv preprint arXiv:2202.07359, 2022. ↗
10. Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 11:1703–1718, 2023. find
11. Minchan Kim, Myeonghun Jeong, Byoung Jin Choi, Dongjune Lee, and Nam Soo Kim. Transduce and speak: Neural transducer for text-to-speech with semantic token prediction. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–7. IEEE, 2023. find
12. Seung-Bin Kim, Sang-Hoon Lee, and Seong-Whan Lee. Transentence: speech-to-speech translation via language-agnostic sentence-level speech encoding without language-parallel data. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12722–12726. IEEE, 2024. find
13. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:17022–17033, 2020. find
14. Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. Textless speech emotion conversion using discrete and decomposed representations. arXiv preprint arXiv:2111.07402, 2021. ↗
15. Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336–1354, 2021. find
16. Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, et al. Direct speech-to-speech translation with discrete units. arXiv preprint arXiv:2107.05604, 2021. ↗
17. Yanir Marmor, Kinneret Misgav, and Yair Lifshitz. ivrit. ai: A comprehensive dataset of hebrew speech for ai research and development. arXiv preprint arXiv:2307.08720, 2023. ↗ our card
18. Pooneh Mousavi, Luca Della Libera, Jarod Duret, Artem Ploujnikov, Cem Subakan, and Mirco Ravanelli. Dasb–discrete audio and speech benchmark. arXiv preprint arXiv:2406.14294, 2024. ↗
19. Tu Anh Nguyen, Wei-Ning Hsu, Antony d’Avirro, Bowen Shi, Itai Gat, Maryam Fazel-Zarani, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, et al. Expresso: A benchmark and analysis of discrete expressive speech resynthesis. arXiv preprint arXiv:2308.05725, 2023. ↗
20. Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R Costa-Jussa, Maha Elbayad, Sravya Popuri, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, et al. Spirit-lm: Interleaved spoken and written language model. arXiv preprint arXiv:2402.05755, 2024. ↗
21. Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.00355, 2021. ↗
22. Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(97):1–52, 2024. find
23. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR, 2023. find
24. Amit Roth, Arnon Turetzky, and Yossi Adi. A language modeling approach to diacritic-free hebrew tts. arXiv preprint arXiv:2407.12206, 2024. ↗ our card
25. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. Alephbert: A hebrew large pre-trained language model to start-off your hebrew nlp application with. arXiv preprint arXiv:2104.04052, 2021. ↗ our card
26. Orian Sharoni, Roee Shenberg, and Erica Cooper. Saspeech: A hebrew single speaker dataset for text to speech and voice conversion. In Proc. Interspeech, 2023. find
27. Avi Shmidman, Shaltiel Shmidman, Moshe Koppel, and Yoav Goldberg. Nakdan: Professional hebrew diacritizer. arXiv preprint arXiv:2005.03312, 2020. ↗ our card
28. Amitay Sicherman and Yossi Adi. Analysing discrete self supervised speech representation for spoken language modeling. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. find
29. David Snyder, Daniel Garcia-Romero, Alan McCree, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. Spoken language recognition using x-vectors. In Odyssey, volume 2018, pages 105–111, 2018. find
30. Or Tal, Moshe Mandel, Felix Kreuk, and Yossi Adi. A systematic comparison of phonetic aware techniques for speech enhancement. arXiv preprint arXiv:2206.11000, 2022. ↗
31. Arnon Turetzky, Or Tal, Yael Segal-Feldman, Yehoshua Dissen, Ella Zeldes, Amit Roth, Eyal Cohen, Yosi Shrem, Bronya R Chernyak, Olga Seleznova, et al. Hebdb: a weakly supervised dataset for hebrew speech processing. arXiv preprint arXiv:2407.07566, 2024. ↗ our card
32. Christophe Veaux, Junichi Yamagishi, and Simon King. The voice bank corpus: Design, collection and data analysis of a large regional accent speech database. In 2013 international conference oriental COCOSDA held jointly with 2013 conference on Asian spoken language research and evaluation (O-COCOSDA/CASLRE), pages 1–4. IEEE, 2013. find
33. Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023. ↗
34. Zengwei Yao, Liyong Guo, Xiaoyu Yang, Wei Kang, Fangjun Kuang, Yifan Yang, Zengrui Jin, Long Lin, and Daniel Povey. Zipformer: A faster and better encoder for automatic speech recognition. In The Twelfth International Conference on Learning Representations, 2023. find
35. Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, et al. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. arXiv preprint arXiv:2408.17175, 2024. ↗
36. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021. find
37. Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882, 2019. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2024-zeldes-enhancing-tts-stability-hebrew-using/