HebDB: a Weakly Supervised Dataset for Hebrew Speech Processing

What it is about

HebDB is a speech corpus of roughly 2,500 hours of spontaneous Hebrew: podcasts on many topics plus testimonies of World War II survivors, released both raw and pre-processed. The pipeline resamples to 16 kHz mono, cuts speech with a voice activity detector, and transcribes each segment with Whisper large-v2; text with non-Hebrew characters is dropped, numbers and dates are spelled out, and final letters are folded. A forced aligner (MMS, on romanised and diacritised text) gives each utterance a confidence score; a threshold of 0.3 keeps about 1,470 of 1,690 hours. Two baselines are trained on it, HuBERT (95M) and a Conformer (110M). On the FLEURS Hebrew test set, the filtered HuBERT with a language model reaches 33.6 WER, the Conformer 41.9, Whisper 1.55B without a language model 27.1, and MMS (1,107 languages) with one 40.0.

Ideas

• I1. Keep a confidence score per item and train only on items above a threshold. Any weakly labelled Hebrew source that Mova might use (automatic transcripts, auto-parsed text) can be gated the same way, and the gate is tested on a gold subset. The gain here is small (HuBERT 33.9 to 33.6, Conformer 43.0 to 41.9 with LM), so the gate must be measured, not assumed.
• I2. Deterministic normalisation before the labels are used: spell out numbers and dates, drop items with off-script characters (about 1% of samples in this paper), and fold final letters to their regular forms. The paper found the final-letter folding helpful. Each rule should be a counted test: how many items it changes on a sample, and whether the change helps.
• I3. Report a yield table per source (hours kept against hours recorded). In this corpus one source fell from 492 h to 67.4 h after processing, which shows that source quality varies a lot. Mova should log the same yield for every new data source, so that a bad source is visible before it reaches training.

For Mova

I1 is a rule for the weak-data side of 11. Modern Hebrew in Universal Dependencies (how automatically produced trees are trusted). I2 goes into 2. Script and spelling (final-letter folding as a test) and into the script filter of tools/langseg. I3 is a reporting habit for every corpus we build. The paper's speech models are outside Mova's scope, which parses text only.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2407.07566

@misc{turetzky2024hebdb,
  title = {{HebDB: a Weakly Supervised Dataset for Hebrew Speech Processing}},
  author = {Turetzky, Arnon and Tal, Or and Segal-Feldman, Yael and Dissen, Yehoshua and Zeldes, Ella and Roth, Amit and Cohen, Eyal and Shrem, Yosi and Chernyak, Bronya R. and Seleznova, Olga and Keshet, Joseph and Adi, Yossi},
  year = {2024},
  eprint = {2407.07566},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.07566}
}

References

36 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020. find
2. P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. d. C. Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov et al., “Audiopalm: A large language model that can speak and listen,” arXiv preprint arXiv:2306.12925, 2023. ↗
3. C. Wang, Y. Wu, Y. Qian, K. Kumatani, S. Liu, F. Wei, M. Zeng, and X. Huang, “Unispeech: Unified speech representation learning with labeled and unlabeled data,” in International Conference on Machine Learning. PMLR, 2021, pp. 10 937–10 947. find
4. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210. find
5. J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen et al., “Libri-light: A benchmark for asr with limited or no supervision,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7669–7673. find
6. C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, “Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,” arXiv preprint arXiv:2101.00390, 2021. ↗
7. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International Conference on Machine Learning. PMLR, 2023, pp. 28 492–28 518. find
8. V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi et al., “Scaling speech technology to 1,000+ languages,” arXiv preprint arXiv:2305.13516, 2023. ↗
9. Y. Zhang, W. Han, J. Qin, Y. Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V. Axelrod, G. Wang et al., “Google usm: Scaling automatic speech recognition beyond 100 languages,” arXiv preprint arXiv:2303.01037, 2023. ↗
10. C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li et al., “Neural codec language models are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023. ↗
11. M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar et al., “Voicebox: Text-guided multilingual universal speech generation at scale,” Advances in neural information processing systems, vol. 36, 2024. find
12. C. K. Reddy, H. Dubey, V. Gopal, R. Cutler, S. Braun, H. Gamper, R. Aichner, and S. Srinivasan, “Icassp 2021 deep noise suppression challenge,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6623–6627. find
13. A. Defossez, G. Synnaeve, and Y. Adi, “Real time speech enhancement in the waveform domain,” arXiv preprint arXiv:2006.12847, 2020. ↗
14. Y. Dissen, F. Kreuk, and J. Keshet, “Self-supervised Speaker Diarization,” in Proc. Interspeech 2022, 2022, pp. 4013–4017. find
15. H. Yadav and S. Sitaram, “A survey of multilingual models for automatic speech recognition,” arXiv preprint arXiv:2202.12576, 2022. ↗
16. T. Likhomanenko, Q. Xu, V. Pratap, P. Tomasello, J. Kahn, G. Avidov, R. Collobert, and G. Synnaeve, “Rethinking evaluation in asr: Are our models robust enough?” arXiv preprint arXiv:2010.11745, 2020. ↗
17. L. Campbell, “Ethnologue: Languages of the world,” 2008. find
18. A. Petrov, E. La Malfa, P. Torr, and A. Bibi, “Language model tokenizers introduce unfairness between languages,” Advances in Neural Information Processing Systems, vol. 36, 2024. find
19. A. W. Black, “Cmu wilderness multilingual speech dataset,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 5971– 5975. find
20. V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “Mls: A large-scale multilingual dataset for speech research,” arXiv preprint arXiv:2012.03411, 2020. ↗
21. A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 798–805. find
22. S. Izre’el, B. Hary, and G. Rahav, “Designing cosih: the corpus of spoken israeli hebrew,” International Journal of Corpus Linguistics, vol. 6, no. 2, pp. 171–197, 2001. find
23. J. Azogui, A. Lerner, and V. Silber-Varod, “The open university of israel map task corpus (matacop),” 2016. find
24. M. Marmorstein and N. Matalon, “The huji corpus of spoken hebrew: An interaction-oriented design of a corpus,” 2022. find
25. O. Sharoni, R. Shenberg, and E. Cooper, “Saspeech: A hebrew single speaker dataset for text to speech and voice conversion,” in Proc. Interspeech, 2023. find
26. Y. Marmor, K. Misgav, and Y. Lifshitz, “ivrit. ai: A comprehensive dataset of hebrew speech for ai research and development,” arXiv preprint arXiv:2307.08720, 2023. ↗ our card
28. S. Team, “Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier,” 2021. find
29. J. Sohn, N. S. Kim, and W. Sung, “A statistical model-based voice activity detection,” IEEE signal processing letters, vol. 6, no. 1, pp. 1–3, 1999. find
30. V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi et al., “Scaling speech technology to 1,000+ languages,” arXiv preprint arXiv:2305.13516, 2023. ↗
31. W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021. find
32. A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu et al., “Conformer: Convolution-augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100, 2020. ↗
33. M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, “fairseq: A fast, extensible toolkit for sequence modeling,” arXiv preprint arXiv:1904.01038, 2019. ↗
34. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd international conference on Machine learning, 2006, pp. 369–376. find
35. A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented Transformer for Speech Recognition,” in Proc. Interspeech 2020, 2020, pp. 5036–5040. find
36. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. find
37. C. Kim, D. Gowda, D. Lee, J. Kim, A. Kumar, S. Kim, A. Garg, and C. Han, “A review of on-device fully neural end-to-end automatic speech recognition algorithms,” in 2020 54th Asilomar Conference on Signals, Systems, and Computers. IEEE, 2020, pp. 277–283. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2024-turetzky-hebdb-weakly-supervised-dataset-hebrew/