LLM-Assisted Rule Based Machine Translation for Low/No-Resource Languages

What it is about

The paper builds two teaching translators for Owens Valley Paiute, a critically endangered language with no public corpus. The language-specific part is hand-written: a sentence builder that offers only the choices that keep a sentence valid, given the choices already made, and a deterministic table of suffixes and verbs. A large language model handles the parts the authors could not hand-write: it turns a free English sentence into simple subject-verb or subject-verb-object clauses, and it turns structured records into fluent English. For the Paiute-to-English direction, 98 of 100 random sentences from the builder were translated correctly. For English-to-Paiute, 125 sentences were scored by embedding similarity against the input, the simplified clauses, the comparator (with unknown words removed), and the back-translation. The method needs no parallel data, but it depends on the model for the English side and gives only partial meaning when the vocabulary is missing.

Ideas

• I1. Make generation constraint-driven. The builder enumerates only the continuations that are licensed by the choices already made. Mova's linearizer in en/src/lin.rs can do the same: a table of valid next slots for each frame, so an invalid order is never produced.
• I2. Use the back-translation as a reference-free check. In normal mode, translate the output back with the same stack and compare it with the source meaning. This is a proxy for the missing reference and it should be reported beside exact match and chrF.
• I3. Calibrate the similarity against a null distribution. The paper sets its threshold at the mean plus three standard deviations of unrelated pairs. For Mova, which is count-based, the null comes from the shuffled-lexicon control: the score of a translation under a shuffled lexicon gives the floor a result must clear.
• I4. Replace unknown words by a typed placeholder in a comparator run. This separates errors caused by a missing lexicon entry from errors of structure. Mova's residual can report unknown-concept tokens as their own class.

For Mova

The code is tools/translate (the encoder and the linearizer hand-off) and en/src/lin.rs. The language-model parts are not for Mova: at test time the SLM does not call any large model, and a large model is only a teacher. What transfers is the rule-based builder, the constraint-driven generation, and the separation of lexicon gaps from structure errors, which map to the residual classes in measure.rs.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.americasnlp-1.9

@inproceedings{coleman2024llm,
  title = {{LLM-Assisted Rule Based Machine Translation for Low/No-Resource Languages}},
  author = {Coleman, Jared and Krishnamachari, Bhaskar and Rosales, Ruben and Iskarous, Khalil},
  booktitle = {Proceedings of the 4th Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP 2024)},
  year = {2024},
  doi = {10.18653/v1/2024.americasnlp-1.9},
  url = {https://doi.org/10.18653/v1/2024.americasnlp-1.9}
}

References

25 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. May 2-6, 2023, pages 2006–2029. Association for Computational Linguistics. find
2. Jessie Little Doe Baird. 2016. Wopanaak language reclamation program: bringing the language home. Journal of Global Indigeneity, 2(2). find
3. OpenAI. 2023. GPT-4 Technical Report. find
4. OpenAI. 2024a. New and Improved Embedding Model. https://openai.com/blog/ new-and-improved-embedding-model. Accessed: 3 Mar 2024. ↗
5. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4. find
6. OpenAI. 2024b. New embedding models and API updates. https://openai.com/blog/ new-embedding-models-and-api-updates. Accessed: 3 Mar 2024. ↗
7. Jialun Cao, Meiziniu Li, Yeting Li, Ming Wen, Shing-Chi Cheung, and Haiming Chen. 2022. Semmt: A semantic-based testing approach for machine translation systems. ACM Trans. Softw. Eng. Methodol., 31(2). find
8. Tommi A Pirinen. 2019. Workflows for kickstarting RBMT in virtually no-resource situation. In Proceedings of the 2nd Workshop on Technologies for MT of Low Resource Languages, pages 11–16, Dublin, Ireland. European Association for Machine Translation. find
9. Aakanksha Chowdhery et al. 2022. PaLM: Scaling Language Modeling with Pathways. find
10. Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. 2023. Neural machine translation for low-resource languages: A survey. ACM Comput. Surv., 55(11):229:1–229:37. find
11. Serafín M. Coronel-Molina and Teresa L. McCarty. 2016. Indigenous language revitalization in the americas. find
12. Explosion. 2024. Industrial-Strength Natural Language Processing. https://spacy.io/. Accessed: 3 Mar 2024. ↗
13. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
14. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. find
15. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
16. Tanmai Khanna, Jonathan N. Washington, Francis M. Tyers, Sevilay Bayatlı, Daniel G. Swanson, Tommi A. Pirinen, Irene Tang, and Hèctor Alòs i Font. 2021. Recent advances in Apertium, a free/open-source rule-based machine translation platform for low-resource languages. Machine Translation, 35(4):475– 502. find
17. Nathaniel R. Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023. ChatGPT MT: Competitive for High- (but not Low-) Resource Languages. find
18. Séamus Lankford, Haithem Afli, and Andy Way. 2023. adaptmllm: Fine-tuning multilingual language models on low-resource languages with integrated LLM playgrounds. Inf., 14(12):638. find
19. SIL International. 2024. 639 Identifier Documentation: mnr. Accessed: 11 Mar 2024. find
20. Yurun Song, Junchen Zhao, and Lucia Specia. 2021. Sentsim: Crosslingual semantic evaluation of machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 3143–3156. Association for Computational Linguistics. find
21. Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. find
22. Joshua Taylor and Timothy Kochem. 2022. Access and empowerment in digital language learning, maintenance, and revival: a critical literature review. Diaspora, Indigenous, and Minority Education, 16(4):234–245. find
23. Christopher Moseley. 2010. Atlas of the World’s Languages in Danger. Unesco. find
24. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. MTEB: massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, Daniel Torregrosa, Nivranshu Pasricha, Maraim Masoud, Bharathi Raja Chakravarthi, Juan A. Alonso, Noe Casas, and Mihael Arcan. 2019. Leveraging rule-based machine translation knowledge for under-resourced neural machine translation models. In Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks, MT-Summit 2019, Dublin, Ireland, August 19-23, 2019, pages 125–133. European Association for Machine Translation. find our card
25. William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Trans. Inf. Syst., 28(4):20:1–20:38. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/coleman-2024-llm-assisted-rule-based-machine/