A Language Modeling Approach to Diacritic-Free Hebrew TTS

What it is about

The authors build a text-to-speech system for undotted (undiacritized) Hebrew. An autoregressive language model over discrete EnCodec tokens is conditioned on a word-piece encoding of the plain text, with a 3-second acoustic prompt. The model is trained on about 4,500 hours of weakly transcribed podcast speech (ivrit.ai and HebDB), with Whisper transcripts. Compared with MMS and Overflow, which predict diacritics first, it gets the best human content and naturalness scores; its WER and CER are similar to the baselines. Word-piece input beats character input on human content scores (3.8 against 2.6). The paper names two limits: slow autoregressive inference, and skipped or invented words.

Ideas

I1. A reading-ambiguity test set: collect undotted Hebrew forms whose vocalisation depends on context (the paper's example is one written form that reads as "gift" or "conditioning"). Take the gold reading from a parse of the context in tools/hbo unpointed mode, and count how often the segmenter and parser pick the right one.
I2. A segmentation baseline check: the paper finds word-piece input better than characters. For tools/hbo, compare the table-driven segmenter (lex, pre) against a plain character baseline on the same unpointed text, and report the gap per token.
I3. An output check for generation: a generated or rewritten Hebrew sentence should keep the same word count and the same content words, as the paper's skip and invention check. A cheap deterministic version can be written with the parser output.

For Mova

I1 goes into Phonology of Biblical Hebrew and 3. Phonology as a reading-ambiguity list, and into tools/hbo as its test. I2 is a test for tools/hbo. I3 is for any future Hebrew generation step, not the current parser.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2407.12206

@misc{roth2024language,
  title = {{A Language Modeling Approach to Diacritic-Free Hebrew TTS}},
  author = {Roth, Amit and Turetzky, Arnon and Adi, Yossi},
  year = {2024},
  eprint = {2407.12206},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.12206}
}

References

35 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

L. Campbell, “Ethnologue: Languages of the world,” 2008. find
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” 2022. find
V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi, A. Baevski, Y. Adi, X. Zhang, W.-N. Hsu, A. Conneau, and M. Auli, “Scaling speech technology to 1,000+ languages,” 2023. find
S. Izre’el, B. Hary, and G. Rahav, “Designing cosih: the corpus of spoken israeli hebrew,” International Journal of Corpus Linguistics, vol. 6, no. 2, pp. 171–197, 2001. find
J. Azogui, A. Lerner, and V. Silber-Varod, “The open university of israel map task corpus (matacop),” 2016. find
M. Marmorstein and N. Matalon, “The huji corpus of spoken hebrew: An interaction-oriented design of a corpus,” 2022. find
O. Sharoni, R. Shenberg, and E. Cooper, “Saspeech: A hebrew single speaker dataset for text to speech and voice conversion,” in Proc. Interspeech, 2023. find
C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li et al., “Neural codec language models are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023. ↗
Y. Marmor, K. Misgav, and Y. Lifshitz, “ivrit. ai: A comprehensive dataset of hebrew speech for ai research and development,” arXiv preprint arXiv:2307.08720, 2023. ↗ our card
A. Turetzky, O. Tal, Y. Segal-Feldman, Y. Dissen, E. Zeldes, A. Roth, E. Cohen, Y. Shrem, B. R. Chernyak, O. Seleznova et al., “Hebdb: a weakly supervised dataset for hebrew speech processing,” arXiv preprint arXiv:2407.07566, 2024. ↗ our card
S. Mehta, A. Kirkland, H. Lameris, J. Beskow, E. Szekely, and G. E. Henter, “Overflow: Putting flows on top of neural transducers for better tts,” Aug. 2023. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2023-1996 ↗
J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” Advances in neural information processing systems, vol. 33, pp. 17 022– 17 033, 2020. find
A. Défossez, J. Copet, G. Synnaeve, and Y. Adi, “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438, 2022. ↗
N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 495–507, 2021. find
J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Défossez, “Simple and controllable music generation,” Advances in Neural Information Processing Systems, vol. 36, 2024. find
F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. Défossez, J. Copet, D. Parikh, Y. Taigman, and Y. Adi, “Audiogen: Textually guided audio generation,” arXiv preprint arXiv:2209.15352, 2022. ↗
Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi et al., “Audiolm: a language modeling approach to audio generation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023. find
R. Sheffer and Y. Adi, “I hear your true colors: Image guided audio generation,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5. find
E. Kharitonov, D. Vincent, Z. Borsos, R. Marinier, S. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour, “Speak, read and prompt: High-fidelity text-to-speech with minimal supervision,” Transactions of the Association for Computational Linguistics, vol. 11, pp. 1703–1718, 2023. find
D. Lyth and S. King, “Natural language guidance of high-fidelity text-to-speech with synthetic annotations,” arXiv preprint arXiv:2402.01912, 2024. ↗
X. Tan, T. Qin, F. Soong, and T.-Y. Liu, “A survey on neural speech synthesis,” arXiv preprint arXiv:2106.15561, 2021. ↗
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019. find
A. Seker, E. Bandel, D. Bareket, I. Brusilovsky, R. S. Greenfeld, and R. Tsarfaty, “Alephbert: A hebrew large pre-trained language model to start-off your hebrew nlp application with,” arXiv preprint arXiv:2104.04052, 2021. ↗ our card
Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey et al., “Google’s neural machine translation system: Bridging the gap between human and machine translation,” arXiv preprint arXiv:1609.08144, 2016. ↗
M. Schuster and K. Nakajima, “Japanese and korean voice search,” in 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2012, pp. 5149– 5152. find
P. Gage, “A new algorithm for data compression,” The C Users Journal archive, vol. 12, pp. 23–38, 1994. [Online]. Available: https://api.semanticscholar.org/CorpusID:59804030 ↗
E. Kharitonov, A. Lee, A. Polyak, Y. Adi, J. Copet, K. Lakhotia, T.-A. Nguyen, M. Rivière, A. Mohamed, E. Dupoux et al., “Text-free prosody-aware generative spoken language modeling,” in ACL 2022-Association for Computational Linguistics, vol. 1. MIT Press, 2022, pp. 8666–8681. find
S. Team, “Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier,” 2021. find
Z. Yao, L. Guo, X. Yang, W. Kang, F. Kuang, Y. Yang, Z. Jin, L. Lin, and D. Povey, “Zipformer: A faster and better encoder for automatic speech recognition,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=9WD9KwssyT ↗
R. San Roman, Y. Adi, A. Deleforge, R. Serizel, G. Synnaeve, and A. Défossez, “From discrete tokens to high-fidelity audio using multi-band diffusion,” arXiv preprint arXiv:, 2023. find
S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, p. 1505–1518, Oct. 2022. [Online]. Available: http://dx.doi.org/10.1109/JSTSP.2022.3188113 ↗
A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux, “Speech resynthesis from discrete disentangled self-supervised representations,” arXiv preprint arXiv:2104.00355, 2021. ↗
C. Wang, W.-N. Hsu, Y. Adi, A. Polyak, A. Lee, P.-J. Chen, J. Gu, and J. Pino, “fairseq sˆ 2: A scalable and integrable speech synthesis toolkit,” arXiv preprint arXiv:2109.06912, 2021. ↗
A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020. find
S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao, “Large language models: A survey,” arXiv preprint arXiv:2402.06196, 2024. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2024-roth-language-modeling-approach-diacritic-free/