PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering

What it is about

PreWoMe is a three-step prompting method for GPT-3.5 and GPT-4 at temperature 0, with no parameter updates. First the model lists the presuppositions of a question, and a fixed presupposition "there is a clear and single answer" is added. Second it writes feedback, a judgement on whether a presupposition is false, which also classifies the question as normal, ambiguous or false-presupposition, and an action. Third it answers with the question, the feedback and the action as guidance. On ASQA (ambiguous), the false-presupposition subset of (QA)2 and BioASQ (normal), GPT-4 with PreWoMe beats the plain prompt on all three and beats chain-of-thought on the misleading sets. Manual checks show that 391 of 395 extracted presuppositions (about 99%) were indeed assumed. The feedback classifies normal questions correctly in 74.9% of cases and false-presupposition questions in 74.4%, but ambiguous questions only in 35.9%. Wrong feedback hurts: GPT-3.5 copies a bad action and answers from it.

Ideas

• I1. Presupposition triggers as rules: a lexicon of trigger patterns (definite descriptions such as "the only", ordinals such as "the first", factive verbs) in 33. Implicature and presupposition, emitting presupposition frames in prag. Check each frame against the world and measure the precision on the school questions.
• I2. The "single clear answer" presupposition becomes a count check: if the world holds two or more answers for the question slot, flag ambiguity. The paper's weak point is exactly this class, so measure the recall of the flag separately.
• I3. Make the feedback a rule-based label with a confusion matrix like the paper's Table 3, so that every misclassification can be explained by the rule that fired.
• I4. A quality gate before answering: do not pass feedback to the answerer unless it passes a check against the world. Negative control: random feedback must score worse than no feedback.

For Mova

The chapters are 33. Implicature and presupposition and 26. Questions. The code is in the prag crate and world/src/qa.rs. The measures are the precision of the presupposition rules on a held-out set, the recall of the ambiguity flag, and the accuracy of the answer with and without the gate. Run each twice.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.517 · arXiv 2310.16147

@inproceedings{han2023prewome,
  title = {{PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering}},
  author = {Han, Wookje and Park, Jinsol and Lee, Kyungjae},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.517},
  eprint = {2310.16147},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.517}
}

References

31 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Reinald Kim Amplayo, Kellie Webster, Michael Collins, Dipanjan Das, and Shashi Narayan. 2022. Query refinement prompts for closed-book long-form question answering. arXiv preprint arXiv:2210.17525. ↗
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
3. Marie Duží and Martina Číhalová. 2015. Questions, answers, and presuppositions. Computación y Sistemas, 19(4):647–659. find
4. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. Eli5: Long form question answering. arXiv preprint arXiv:1907.09190. ↗
5. Liping Ge. 2011. Pragmatic functions of presupposition in advertising english. Asian Culture and History, 3(1):153. find
6. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406. ↗
7. Najoung Kim, Phu Mon Htut, Samuel R Bowman, and Jackson Petty. 2022. Θ2: Question answering with questionable assumptions. arXiv preprint arXiv:2212.10003. ↗
8. Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which linguist invented the lightbulb? presupposition verification for question-answering. arXiv preprint arXiv:2101.00391. ↗ our card
9. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213. find
10. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. arXiv preprint arXiv:2103.06332. ↗
11. Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. Bioasq-qa: A manually curated corpus for biomedical question answering. Scientific Data, 10(1):170. find
12. Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2022. Large language models with controllable working memory. arXiv preprint arXiv:2211.05110. ↗
13. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81. find
14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
15. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651. ↗
16. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. find
17. George A Miller, Galanter Eugene, and Karl H Pri-bram. 2017. Plans and the structure of behaviour. In Systems Research for Behavioral Sciencesystems Research, pages 369–382. Routledge. find
18. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. Ambigqa: Answering ambiguous open-domain questions. arXiv preprint arXiv:2004.10645. ↗
19. OpenAI. 2023. Gpt-4 technical report. find
20. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350. ↗
21. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822. ↗
22. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802. ↗
23. Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of ACL. find
24. Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366, 14. ↗
25. Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. Asqa: Factoid questions meet long-form answers. arXiv preprint arXiv:2204.06092. ↗
26. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. ↗
27. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. 2023. Large language models are better reasoners with self-verification. CoRR, abs/2212.09561. ↗
28. Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A critical evaluation of evaluations for long-form question answering. arXiv preprint arXiv:2305.18201. ↗
29. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. ↗
30. Xinyan Velocity Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. Crepe: Open-domain question answering with false presuppositions. arXiv preprint arXiv:2211.17257. ↗ our card
31. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023. How language model hallucinations can snowball. arXiv preprint arXiv:2305.13534. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/han-2023-prewome-exploiting-presuppositions-as-working/