An Arabic-Hebrew parallel corpus of TED talks

What it is about

The author builds an Arabic-Hebrew benchmark of about 2,000 TED talks from the WIT3 subtitle inventory, with 1,908 talks aligned, 225k sentences and about 3.5M tokens per language. The raw subtitles are split by captions, and the Hebrew side often lacks the English punctuation, so sentences cannot be rebuilt from the target side. The fix is to pivot through English: each language is aligned to English, the two English sides are re-aligned, and sentences are rebuilt wherever the English has strong punctuation. About 0.4% of the English pairs differ after the two alignments, and 9% of talks have translator resegmentation. Sentence-length statistics show that pivot-rebuilt sentences keep the variance of the original captions, and have four to five times fewer sentences over 100 tokens than punctuation-based rebuilding. Baseline BLEU scores are close to those of Google Translate, and the paper says the gaps are small.

Ideas

• I1. Pivot projection for Hebrew gold trees: align Hebrew and English subtitles through the English side, then project English UD trees to the Hebrew side. A sample checked by hand gives a Hebrew treebank seed without a Hebrew parser as the starting point, which fits Mova's English-first design.
• I2. The pivot rule as a segmenter: a caption boundary becomes a sentence boundary only when the English side has strong punctuation there. The same rule can cut unpunctuated Hebrew text in tools/ocrfix output, using the aligned English as the cue.
• I3. Length statistics as an intake check: report the mean, spread and share of very long sentences for any corpus Mova imports, and compare them with the source captions, as the paper does.

For Mova

I1 belongs in treebanks/he/, with the projection checked on a gold sample there. I2 is a rule for tools/ocrfix output and for the sentence view in 2. Script and spelling. I3 is an intake check for every corpus under treebanks/he/.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 1610.00572

@misc{cettolo2016arabic,
  title = {{An Arabic-Hebrew parallel corpus of TED talks}},
  author = {Cettolo, Mauro},
  year = {2016},
  eprint = {1610.00572},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1610.00572}
}

References

10 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yonatan Belinkov and James Glass. 2016. Large-scale Machine Translation between Arabic and Hebrew: Available Corpora and Initial Results. In Proc. of SeMaT, Austin, US-TX. find our card
2. Fabienne Braune and Alexander Fraser. 2010. Improved Unsupervised Sentence Alignment for Symmetrical and Asymmetrical Parallel Corpora. In Proc. of Coling 2010: Posters, pp. 81– 89, Beijing, China. find
3. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. WIT 3 : Web Inventory of Transcribed and Translated Talks. In Proc. of EAMT, pp. 261–268, Trento, Italy. find
4. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proc. of NAACL, pp. 644–648, Atlanta, US-GA. find
5. Ahmed El Kholy and Nizar Habash. 2015. Morphological Constraints for Phrase Pivot Statistical Machine Translation. In Proc. of MT Summit XV, vol.1: MT Researchers’ Track, pp. 104– 116, Miami, US-FL. find
6. Marcello Federico, Nicola Bertoldi, and Mauro Cettolo. 2008. IRSTLM: an Open Source Toolkit for Handling Large Scale Language Models. In Proc. of Interspeech, pp. 1618–1621, Brisbane, Australia. find
7. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proc. of ACL: Demo and Poster Sessions, pp. 177–180, Prague, Czech Republic. find
8. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Proc. of MT Summit X, pp. 79–86, Phuket, Thailand. find
9. Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proc. of LREC, pp. 923–929, Portorož, Slovenia. find
10. Reshef Shilon, Nizar Habash, Alon Lavie, and Shuly Wintner. 2012. Machine Translation between Hebrew and Arabic. Machine Translation, 26(1-2):177–195, March. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2016-cettolo-arabic-hebrew-parallel-corpus-ted/