Mova0.5 pre-release⌕
☰
home / docs / papers / How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark

How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark

Summary

The paper measures how much large language models are thrown off in grade-school math reasoning when a problem contains irrelevant information. Earlier benchmarks such as GSM-IC added a single unrelated sentence without control over its structure. The authors build GSM-DC, where each problem is a directed acyclic graph of dependencies between quantities: the solution path is found by topological sorting, and nodes outside that path serve as controlled distractors before the graph is rendered into text with templates. This lets reasoning depth and amount of noise be varied independently and every reasoning step be checked automatically. Accuracy drops steadily as distractors increase, affecting both the choice of reasoning path and the arithmetic. Models trained on strong distractors were the most robust, and a stepwise tree search guided by a process reward model added further robustness out of distribution.

How Mova uses it

Effectiveness in Mova

Measured and harmful: SVAMP 57.3% → 52.7% (same result in two runs). The artificial distractor always sits in the same place, so the model learns the position rather than the principle; in the papers distractors are tailored to the problem (similar object, same topic). The idea is not used by default.

👨‍🔬💥

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2505.18761

@misc{yang2025how,
  title = {{How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark}},
  author = {Yang, Minglai and Huang, Ethan and Zhang, Liang and Surdeanu, Mihai and Wang, William and Pan, Liangming},
  year = {2025},
  eprint = {2505.18761},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.18761}
}

References

51 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zeyuan Allen-Zhu and Yuanzhi Li. 2024a. Physics of language models: Part 1, learning hierarchical language structures. Preprint, arXiv:2305.13673. find
  2. Zeyuan Allen-Zhu and Yuanzhi Li. 2024b. Physics of language models: Part 3.2, knowledge manipulation. Preprint, arXiv:2309.14402. find
  3. Zeyuan Allen-Zhu and Yuanzhi Li. 2024c. Physics of language models: Part 3.3, knowledge capacity scaling laws. Preprint, arXiv:2404.05405. — [arXiv:2404.05405](https://arxiv.org/abs/2404.05405) find
  4. Ujjwala Anantheswaran, Himanshu Gupta, Kevin Scaria, Shreyas Verma, Chitta Baral, and Swaroop Mishra. 2024. Cutting through the noise: Boosting llm performance on math word problems. arXiv preprint arXiv:2406.15444. find our card
  5. Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2024. The reversal curse: Llms trained on "a is b" fail to learn "b is a". Preprint, arXiv:2309.12288. find
  6. Leonardo Bertolazzi, Albert Gatt, and Raffaella Bernardi. 2024. A systematic analysis of large language models as soft reasoners: The case of syllogistic inferences. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16,… find
  7. Michael Boratko, Xiang Li, Tim O’Gorman, Rajarshi Das, Dan Le, and Andrew McCallum. 2020. ProtoQA: A question answering dataset for prototypical common-sense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1122–1136, Online. Associa… find
  8. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024. Longlora: Efficient fine-tuning of long-context large language models. Preprint, arXiv:2309.12307. find
  9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. find
  10. Kuicai Dong, Yujing Chang, Xin Deik Goh, Dexun Li, Ruiming Tang, and Yong Liu. 2025a. Mmdocir: Benchmarking multi-modal retrieval for long documents. Preprint, arXiv:2501.08828. — [arXiv:2501.08828](https://arxiv.org/abs/2501.08828) find
  11. Kuicai Dong, Yujing Chang, Shijie Huang, Yasheng Wang, Ruiming Tang, and Yong Liu. 2025b. Benchmarking retrieval-augmented multimomal generation for document question answering. Preprint, arXiv:2505.16470. — [arXiv:2505.16470](https://arxiv.org/abs/2505.16470) find
  12. Kuicai Dong, Derrick Goh Xin Deik, Yi Quan Lee, Hao Zhang, Xiangyang Li, Cong Zhang, and Yong Liu. 2024. MC-indexing: Effective long document retrieval via multi-view content-aware indexing. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2673–2691, Miami, Florida, US… find
  13. Barbara A. Eriksen and Charles W. Eriksen. 1974. Effects of noise letters upon the identification of a target letter in a nonsearch task. Perception & Psychophysics, 16:143–149. find
  14. Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, Zhen Wang, and Zhiting Hu. 2024. LLM reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models. In First Conference on Language Model… find
  15. Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron C. Courville, Alessandro Sordoni, and Rishabh Agarwal. 2024. V-star: Training verifiers for self-taught reasoners. CoRR, abs/2402.06457. find
  16. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large language models cannot self-correct reasoning yet. Preprint, arXiv:2310.01798. find
  17. Yining Huang, Bin Li, Keke Tang, and Meilian Chen. 2025. Lora-par: A flexible dual-system lora partitioning approach to efficient llm fine-tuning. Preprint, arXiv:2507.20999. find
  18. Ming Jiang, Tingting Huang, Biao Guo, Yao Lu, and Feng Zhang. 2024a. Enhancing robustness in large language models: Prompting for mitigating the impact of irrelevant information. CoRR, abs/2408.10615. — [arXiv:2408.10615](https://arxiv.org/abs/2408.10615) find
  19. Ziyan Jiang, Xueguang Ma, and Wenhu Chen. 2024b. Longrag: Enhancing retrieval-augmented generation with long-context llms. Preprint, arXiv:2406.15319. — [arXiv:2406.15319](https://arxiv.org/abs/2406.15319) find
  20. Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D. Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M. Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal M. P. Behbahani, and Aleksandra Faust. 2024. Training lang… find
  21. Aitor Lewkowycz, Eric Zelikman, Ran Tao, Tatsunori Hashimoto, John Schulman, Xinyun Chen, C Lawrence Zitnick, Christopher D Manning, Daniel Saunders, Adam Santoro, et al. 2022. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858. find
  22. Zhiming Li, Yushi Cao, Xiufeng Xu, Junzhe Jiang, Xu Liu, Yon Shin Teo, Shang-Wei Lin, and Yang Liu. 2024. Llms for relational reasoning: How far are we? In LLM4CODE@ICSE, pages 119–126. find
  23. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open-Rev… find
  24. Wanlong Liu, Junying Chen, Ke Ji, Li Zhou, Wenyu Chen, and Benyou Wang. 2024. Rag-instruct: Boosting llms with diverse retrieval-augmented instructions. Preprint, arXiv:2501.00353. find
  25. Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multi-step reasoning and tool-use for large language models. Preprint, arXiv:2303.09014. find
  26. Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In Thirty-seventh Conference on Neural Information Processing Systems. find
  27. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. 2023a. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 20… find
  28. Xiaoming Shi, Siqiao Xue, Kangrui Wang, Fan Zhou, James Y. Zhang, Jun Zhou, Chenhao Tan, and Hongyuan Mei. 2023b. Language models can improve event prediction by few-shot abductive reasoning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing… find
  29. Seok Hwan Song and Wallapak Tavanapong. 2024. How much do prompting methods help llms on quantitative reasoning with irrelevant information? In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM 2024, Boise, ID, USA, October 21-25, 2024, pages 2128–213… find
  30. Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. 2021. Proofwriter: Generating implications, proofs, and abductive statements over natural language. Preprint, arXiv:2012.13048. find
  31. Shi-Yu Tian, Zhi Zhou, Kun-Yang Yu, Ming Yang, Lin-Han Jia, Lan-Zhe Guo, and Yu-Feng Li. 2025. Vc search: Bridging the gap between well-defined and ill-defined problems in mathematical reasoning. Preprint, arXiv:2406.05055. find
  32. Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Y. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process- and outcome-based feedback. CoRR, abs/2211.14275. find
  33. Xiaonan Wang, Bo Shao, and Hansaem Kim. 2025a. Toward reliable vlm: A fine-grained benchmark and framework for exposure, bias, and inference in korean street views. Preprint, arXiv:2506.03371. — [arXiv:2506.03371](https://arxiv.org/abs/2506.03371) find
  34. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,… find
  35. Yifei Wang, Feng Xiong, Yong Wang, Linjing Li, Xiangxiang Chu, and Daniel Dajun Zeng. 2025b. Position bias mitigates position bias:mitigate position bias through inter-position knowledge distillation. Preprint, arXiv:2508.15709. find
  36. Jiyao Wei, Saiping Guan, Da Li, Xiaolong Jin, Jiafeng Guo, and Xueqi Cheng. 2025. A survey of link prediction in n-ary knowledge graphs. Preprint, arXiv:2506.08970. find
  37. Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. 2024. How easily do irrelevant inputs skew the responses of large language models? CoRR, abs/2404.03302. find
  38. Zikai Xiao, Ziyang Wang, Wen Ma, Yan Zhang, Wei Shen, WangYan WangYan, Luqi Gong, and Zuozhu Liu. 2025. Mitigating posterior salience attenuation in long-context LLMs with positional contrastive decoding. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol… find
  39. Feng Xiong, Hongling Xu, Yifei Wang, Runxi Cheng, Yong Wang, and Xiangxiang Chu. 2025. Hs-star: Hierarchical sampling for self-taught reasoners via difficulty estimation and budget reallocation. Preprint, arXiv:2505.19866. find
  40. Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang. 2024. Pride and prejudice: Llm amplifies self-bias in self-refinement. Preprint, arXiv:2402.11436. find
  41. Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. 2023. Qa-lora: Quantization-aware low-rank adaptation of large language models. Preprint, arXiv:2309.14717. — [arXiv:2309.14717](https://arxiv.org/abs/2309.14717) find
  42. Bo Yang, Qingping Yang, Yingwei Ma, and Runtao Liu. 2025. Utmath: Math evaluation with unit test via reasoning-to-coding thoughts. Preprint, arXiv:2411.07240. — [arXiv:2411.07240](https://arxiv.org/abs/2411.07240) find
  43. Shinn Yao, Jeffrey Zhao, Dian Yu, Izhak Zhao, Yujia Yu, Eric Gao, Maarten Bosma, Yi Wang, and Denny Zhou. 2023a. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601. find
  44. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023b. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023,… find
  45. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023c. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. find
  46. Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. 2025. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. In The Thirteenth International Conference on Learning Representations. find
  47. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. Star: Bootstrapping reasoning with reasoning. Preprint, arXiv:2203.14465. find our card
  48. Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2024. Processbench: Identifying process errors in mathematical reasoning. CoRR, abs/2412.06559. find
  49. Xuezhi Zhou, Nathanael Schärli, Luheng Hou, Jason Wei, Swaroop Mishra, Xinyun Huang, Quoc Le, Hyung Won Chung, Hieu Pham, Barret Zoph, et al. 2022. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625. find
  50. Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. 2025. Gsm-infinite: How do your llms behave over infinitely increasing context length and reasoning complexity? CoRR, abs/2502.05252. find
  51. Content of Appendix A Dataset Samples A.1 Training Dataset with Different Level of IC for Finetuned Model A.2 Prompts for Closed-Sourced Models B Empirical Noise Stratification B Clarifying the Role of Synthetic Problem Generation D Operation-Range Bias and Our Train/Test Protocol E Process Reward M… find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/yang-2025-llm-reasoning-distracted-irrelevant-context/