"I'd rather just go to bed": Understanding Indirect Answers

What it is about

The paper builds Circa, 34,268 pairs of a polar question and an indirect answer (no yes/no words), in ten everyday scenarios, written and labelled by crowd workers. Each pair has five judgements of how the asker would read it: yes, probably yes, yes under a condition, no, probably no, in the middle, unsure, other (relaxed: yes 48.5%, no 37.5%, conditional yes 7.5%, middle 2.8%; Fleiss kappa 0.61 strict, 0.76 relaxed). A mined sample of movie dialogues shows only 53% of answers to yes/no questions are direct. BERT models reach 88% (relaxed, 4 classes) and 85% (strict, 6) when the scenario was seen in training and 6–10 points less on an unseen one; the answer alone already gives 82%, the question alone 56%. An off-the-shelf NLI model, used by turning the question into a declarative and reading entailment as yes and contradiction as no, is good on no and poor on yes.

Ideas

• I1. Read indirect answers in Mova's frames with the paper's question-to-declarative trick, which Mova already does (statement_order): put the yes/no question in statement order, read both into frames, then decide deterministically. The answer frame matches the question frame → yes; same frame negated → no; the same role filled by a different filler («Do you like pizza?» – «I prefer pasta») → no; an if/when/«depends» clause → conditional yes; a hedge → probably. The decision prints the matched or contrasting role. Use it in world/src/talk.rs and for quoted replies in tales; measure on a hand-built dev set from tales before any outside data.
• I2. The answer-only result (82%) means a small cue list carries most of the signal: «I would love to», «Let's do it», «I wish!», «I don't drink», «If I can afford it», «It depends», «I'm not sure yet». Add them as reply cues (yes, no, conditional, unsure) to prag/data/cues.tsv with our own phrasing and count how many agree/disagree lines in the silver they cover.
• I3. Add a conditional reading to replies, which the schema lacks: in Circa conditional yes is the third class (7.5%). In tales «If you give me the cheese, I will…» answers a request with a condition. Either a value of a reply qualifier or a flag on the Say frame (cond already exists on frames); measure how often the silver's means for such lines states a condition.
• I4. The paper's discourse trick (insert because or but between the declarative question and the answer) matches Mova's cause and contrast links: an answer that reads as a reason for the declarative is a yes, one that needs but is a no. A rule over the existing link detector, checked on the same dev set as I1.

For Mova

38. Dialogue, discourse markers and reported speech §9 (questions and answers in dialogue), 33-implicature-and-presupposition.md §5 (Relevance inferences), 31-speech-acts.md (agree/disagree). Code: prag/src/frames.rs (statement_order, frame matching), world/src/talk.rs, prag/data/cues.tsv. Measures: accuracy on a tale dev set of question–reply pairs (relaxed four classes), agree/disagree F1 in prag eval. Circa itself only as a test set, and only if its licence allows.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.601 · arXiv 2010.03450

@inproceedings{louis2020i,
  title = {{"I'd rather just go to bed": Understanding Indirect Answers}},
  author = {Louis, Annie and Roth, Dan and Radlinski, Filip},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.601},
  eprint = {2010.03450},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.601}
}

References

20 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a Human-Like Open-Domain Chatbot. arXiv preprint arXiv:2001.09977. ↗
2. Penelope Brown and Stephen C Levinson. 1978. Universals in Language Usage: Politeness Phenomena. In E.N. Goody, editor, Questions and Politeness: Strategies in Social Interaction, pages 56–311. Cambridge University Press. find
3. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question Answering in Context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184. find
4. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936. find
5. Cristian Danescu-Niculescu-Mizil and Lillian Lee. 2011. Chameleons in Imagined Conversations: A New Approach to Understanding Coordination of Linguistic Style in Dialogs. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, ACL 2011. find
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. find
7. Nancy Green and Sandra Carberry. 1999. Interpreting and Generating Indirect Answers. Computational Linguistics, 25(3):389–435. find
8. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation Artifacts in Natural Language Inference Data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112. find
9. Julia B. Hirschberg. 1985. A Theory of Scalar Implicature. Ph.D. thesis, University of Pennsylvania. find
10. Beth Ann Hockey, Deborah Rossen-Knill, Beverly Spejewski, Matthew Stone, and Stephen Isard. 1997. Can you Predict Responses to Yes/No Questions? Yes, No, and Stuff. In Fifth European Conference on Speech Communication and Technology. find
11. Marie-Catherine de Marneffe, Scott Grimm, and Christopher Potts. 2009. Not a Simple Yes or No: Uncertainty in Indirect Answers. In Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 136–143. find
12. Marie-Catherine de Marneffe, Christopher D Manning, and Christopher Potts. 2010. Was it Good? It was Provocative. Learning the Meaning of Scalar Adjectives. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 167–176. find our card
13. Allen Nie, Erin Bennett, and Noah D. Goodman. 2019. DisSent: Learning Sentence Representations from Explicit Discourse relations. In Proceedings of the 57th Conference of the Association for Computational Linguistics, Volume 1: Long Papers, pages 4497–4510. find
14. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis-Only Baselines in Natural Language Inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191. find
15. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. CoQA: A Conversational Question Answering Challenge. Transactions of the Association for Computational Linguistics, 7:249–266. find
16. Deborah Rossen-Knill, Beverly Spejewski, Beth Ann Hockey, Stephen Isard, and Matthew Stone. 1997. Yes/No Questions and Answers in the Map Task Corpus. Technical report, University of Pennsylvania. find
17. Anna-Brita Stenström. 1984. Questions and Responses in English Conversation: Lund Studies in English. Claes Schaar and Jan Svartvik, editors, Lund Studies in English 68, CWK Gleerup, Malmo Sweden. find
18. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. the Proceedings of ICLR. find
19. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. find
20. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation. arXiv preprint arXiv:1911.00536. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/louis-2020-i-d-rather-just-go/