The eBible Corpus: Data and Model Benchmarks for Bible Translation for Low-Resource Languages

What it is about

The paper presents the eBible corpus: 1,009 Bible translations in 833 languages from 75 language families, gathered from eBible.org and aligned verse by verse. The authors normalise versification and encoding, which differ between translations, since they block simple use of the data. They benchmark neural translation (NLLB-based models) on paired-verse data with three protocols: a standard train, test and validation split, a randomised cross-validation over translation pairs (because the sets are small, no 80/10/10 split is used), and tasks that follow a realistic translation order, such as Gospels first, then epistles, then Pentateuch, and separate New and Old Testament tasks. Scores are BLEU, spBLEU and chrF3. For one Bible translation task the Austronesian and Trans-New Guinea families reach 35.1 and 31.6 BLEU. The paper also notes that the corpus files have mixed licences: per the paper, some are CC BY-SA, two are CC BY-NC, and most are permissive.

Ideas

• I1. Check the translator's loaders against the paper's main data problem: one verse in one translation can span two in another. A verse-key file (versification) that all translations are mapped onto is the fix; test whether tools/translate handles the merged-verse cases before measuring anything.
• I2. Adopt the randomised cross-validation over translation pairs as the protocol for the author-delta decisions, which matches the cv rule of the project.
• I3. Test the delta learner on a growing set of books in the paper's translation order, to measure how much of an author's delta is learned from a partial translation. Measure, do not assume.

For Mova

The relevant parts are the verse alignment and the benchmark protocols for tools/translate and measure.rs. The NMT models are not for Mova. Because the corpus mixes licences file by file, the data stays outside the build and only the tools and protocols are taken.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2304.09919

@misc{akerman2023ebible,
  title = {{The eBible Corpus: Data and Model Benchmarks for Bible Translation for Low-Resource Languages}},
  author = {Akerman, Vesa and Baines, David and Daspit, Damien and Hermjakob, Ulf and Jang, Taeho and Leong, Colin and Martin, Michael and Mathew, Joel and Robie, Jonathan and Schwarting, Marcus},
  year = {2023},
  eprint = {2304.09919},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.09919}
}

References

25 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Orevaoghene Ahia, Julia Kreutzer, and Sara Hooker. 2021. The low-resource double bind: An empirical study of pruning for low-resource machine translation. arXiv preprint arXiv:2110.03036. ↗
2. Benjamin Akera, Jonathan Mukiibi, Lydia Sanyu Naggayi, Claire Babirye, Isaac Owomugisha, Solomon Nsumba, Joyce Nakatumba-Nabende, Engineer Bainomugisha, Ernest Mwebaze, and John Quinn. 2022. Machine translation for African languages: Community creation of datasets and models in Uganda. find
3. David Baines, Joel Mathew, and Michael Martin. 2023a. The eBible corpus. https://github.com/ BibleNLP/ebible. ↗
4. David Baines, Joel Mathew, and Michael Martin. 2023b. eBible corpus NMT experiments. https://github.com/BibleNLP/ebible-experiments. ↗
5. Christos Christodouloupoulos and Mark Steedman. 2015. A massively parallel corpus: the Bible in 100 languages. Language resources and evaluation, 49:375–395. find
6. Richard Oliver Collin. 2010. Ethnologue. Ethnopoli-tics, 9(3-4):425–432. find
7. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. ↗
8. Damien Daspit. 2023. SIL-NLP. https://github.com/sillsdev/silnlp. ↗
9. Yonggang Deng and William Byrne. 2008. HMM word and phrase alignment for statistical machine translation. IEEE Transactions on Audio, Speech, and Language Processing, 16(3):494–507. find
10. Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648. find
11. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538. find
12. Ulf Hermjakob. 2023. Wildebeest. https://github.com/uhermjakob/wildebeest. ↗
13. Katharina Kann, Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, John E Ortega, Annette Rios, Angela Fan, Ximena Gutierrez-Vasques, Luis Chiruzzo, Gustavo A Giménez-Lugo, et al. 2022. Americasnli: Machine translation and natural language inference systems for Indigenous languages of the Americas. Frontiers in Artificial Intelligence, 5:266. find
14. Jeff Klassen. 2022. Unified standard format markers. https://github.com/ubsicap/usfm. ↗
15. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810. ↗
16. Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press. find
17. Sami Liedes. 2018. Machine translating the Bible into new languages. https://github.com/sliedes/ fairseq-py. ↗
18. Arya D McCarthy, Rachel Wicks, Dylan Lewis, Aaron Mueller, Winston Wu, Oliver Adams, Garrett Nicolai, Matt Post, and David Yarowsky. 2020. The Johns Hopkins University Bible corpus: 1600+ tongues for typological exploration. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2884–2892. find
19. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038. ↗
20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. find our card
21. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterhub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 46–54. find
22. Maja Popović. 2015. chrf: character n-gram F-score for automatic MT evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392–395. find our card
23. Philip Resnik, Mari Broman Olsen, and Mona Diab. 1999. The Bible as a parallel corpus: Annotating the ‘Book of 2000 Tongues’. Computers and the Humanities, 33:129–153. find
24. Winston Wu, Nidhi Vyas, and David Yarowsky. 2018. Creating a translation matrix of the Bible’s names across 591 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find our card
25. Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2020. Incorporating bert into neural machine translation. arXiv preprint arXiv:2002.06823. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/akerman-2023-ebible-corpus-data-model-benchmarks/