Fixing Translation Divergences in Parallel Corpora for Neural MT

What it is about

Parallel corpora are noisy: some sentence pairs have extra segments, unrelated content or inaccurate translations. The authors train a neural network on the corpus itself, without labels, to score the cross-lingual similarity of words. The network yields a sentence similarity score and a word-level divergence prediction. Pairs below a score threshold are filtered out, and in some pairs the divergent segments are removed while the rest is kept, which the paper calls fixing. For English–French and English–German neural translation, training on the filtered corpus improves BLEU by +0.94 and +2.25 over the full corpus. On OpenSubtitles, fixing 2.5 million sentences adds +2.01 BLEU, while the same sentences added unchanged added nothing. The method needs no supervision, and the code is public.

Ideas

• I1. A count-based divergence score for verse pairs: for each word pair take the alignment probability from the IBM Model 1 table in ibm.rs, and mark a verse divergent when its aligned mass is low in both directions. This replaces the neural similarity with a deterministic rule that can be explained word by word.
• I2. Filter, then measure: remove the divergent verses before the standard encoder is learned, and compare exact-match and residual bits. Run each twice to check determinism. A gain would come from a cleaner standard, not from an author delta, so report them separately.
• I3. Partial fix for a verse with one divergent clause: keep the clause whose meta tokens agree with the other translations of the same verse, and count how many verses this rescues.

For Mova

The code is tools/translate/ibm.rs and measure.rs. The divergence filter touches open problem 6 (weight the corpora one vote each), because the Bible pairs are the main data for the standard. The neural parts of the paper are not for Mova; I1 and I2 are the count-based forms of the same idea.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1328

@inproceedings{pham2018fixing,
  title = {{Fixing Translation Divergences in Parallel Corpora for Neural MT}},
  author = {Pham, MinhQuang and Crego, Josep and Senellart, Jean and Yvon, François},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1328},
  url = {https://doi.org/10.18653/v1/d18-1328}
}

References

29 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre, Carmen Banea, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval@NAACL-HLT, pages 497–511. The Association for Computational Linguistics. find
2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In Proceedings of the Sixth International Conference on Learning Representations, Vancouver, Canada. find
3. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 Conference on Machine Translation (WMT17). In Proceedings of the Second Conference on Machine Translation, pages 169–214, Copenhagen, Denmark. Association for Computational Linguistics. find
4. Marine Carpuat, Yogarshi Vyas, and Xing Niu. 2017. Detecting cross-lingual semantic divergence for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 69–79, Vancouver. Association for Computational Linguistics. find
5. Boxing Chen, Roland Kuhn, George Foster, Colin Cherry, and Fei Huang. 2016. Bilingual methods for adaptive training data selection for machine translation. In Proceedings of the 12th biennial conference of the Association for the Machine Translation in Americas (AMTA2016), Austin, TX. find
6. Mara Chinea-Rios, Álvaro Peris, and Francisco Casacuberta. 2017. Adapting neural machine translation with parallel synthetic data. In Proceedings of the Second Conference on Machine Translation, pages 138–147, Copenhagen, Denmark. Association for Computational Linguistics. find
7. Christian Federmann and Will Lewis. 2016. Microsoft speech language translation (MSLT) corpus: The IWSLT 2016 release for English, French and German. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT2016), Seattle, WA. find
8. Pascale Fung and Percy Cheung. 2004. Mining very-non-parallel corpora: Parallel sentence and lexicon extraction via bootstrapping and EM. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, EMNLP. find
9. Cyril Goutte, marine carpuat, and Georges Foster. 2012. The impact of sentence alignment errors on phrase-based machine translation performance. In The Tenth Biennial Conference of the Association for Machine Translation in the Americas, San Diego, California. find
10. Francis Grégoire and Philippe Langlais. 2018. Extracting parallel sentences with bidirectional recurrent neural networks to improve machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 1442–1453. find
11. Jeenu Grover and Pabitra Mitra. 2017. Bilingual word embeddings with bucketed CNN for parallel sentence extraction. In Proceedings of ACL 2017, Student Research Workshop, pages 11–16, Vancouver, Canada. Association for Computational Linguistics. find
12. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving human parity on automatic Chinese to English news translation. CoRR, abs/1803.05567. ↗
13. Hua He and Jimmy Lin. 2016. Pairwise word interaction modeling with deep neural networks for semantic similarity measurement. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 937– 948, San Diego, California. Association for Computational Linguistics. find
14. Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In International Conference on Learning Representations, Long Beach, CA. find
15. Joël Legrand, Michael Auli, and Ronan Collobert. 2016. Neural network-based word alignment through score aggregation. In Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers, pages 66–73, Berlin, Germany. Association for Computational Linguistics. find
16. Junyi Jessy Li, Marine Carpuat, and Ani Nenkova. 2014. Cross-lingual discourse relation analysis: A corpus study and a semi-supervised classification system. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 577–587, Dublin, Ireland. Dublin City University and Association for Computational Linguistics. find
17. Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In 10th International Conference on Language Resources and Evaluation (LREC 2016), Portoroz, Slovenia. European Language Resources Association (ELRA). find
18. Jonas Mueller and Aditya Thyagarajan. 2016. Siamese recurrent architectures for learning sentence similarity. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, pages 2786–2792, Phoenix, Arizona. AAAI Press. find
19. Dragos S. Munteanu and Daniel Marcu. 2005. Improving machine translation performance by exploiting non-parallel corpora. Computational Linguistics, 31(4):477–504. find
20. Resources and Evaluation (LREC 2016), Portoroz, Slovenia. European Language Resources Association (ELRA). find
21. Bing Zhao and Stephan Vogel. 2002. Adaptive parallel sentences mining from web bilingual news collection. In Proceedings of the 2002 IEEE International Conference on Data Mining, ICDM ’02, Washington, DC, USA. IEEE Computer Society. find
22. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28, ICML’13, pages III– 1310–III–1318, Atlanta, GA, USA. JMLR.org. find
23. Holger Schwenk. 2018. Filtering and mining parallel data in a joint multilingual space. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 228–234, Melbourne, Australia. Association for Computational Linguistics. find
24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational Linguistics. find
25. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Linguistics. find
26. Jörg Tiedemann. 2011. Bitext Alignment. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. find
27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010, Long Beach, CA. find
28. Yogarshi Vyas, Xing Niu, and Marine Carpuat. 2018. Identifying semantic divergences in parallel text without annotations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1503–1515, New Orleans, Louisiana. Association for Computational Linguistics. find
29. Yong Xu and François Yvon. 2016. Novel elicitation and annotation schemes for sentential and sub-sentential alignments of bitexts. In Proceedings of the Tenth International Conference on Language find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/pham-2018-fixing-translation-divergences-parallel-corpora/