Disentangling Indirect Answers to Yes-No Questions in Real Conversations

What it is about

The authors take every yes/no question in the Switchboard Dialog Act corpus whose next turn is not a direct answer: 2,544 questions, which is 43.5% of all yes/no questions there (77% of conversations have one); 41% of the questions are themselves indirect, without a question mark («I don't know if you're familiar with that issue»). Annotators chose yes / probably yes / middle / probably no / no twice: once from the question and the next turn only, once with three turns before and after (weighted kappa 0.81 and 0.78). Context changes the label a lot: half of the «middle» labels become yes or probably yes, and some answers flip, because the decisive turn often comes later (the next turn helps 96% of the time, the asker's follow-up 21%). A question with negation reverses the labels («You didn't move to Alaska, right?» – «I've been in Alaska 13 years» = no). RoBERTa reaches only 0.46 F1 (majority 0.25); MNLI, BoolQ and the crowd-written Circa barely transfer to real talk, and feeding the whole context hurts unless only the annotator-chosen turns are given (0.49).

Ideas

• I1. Read declarative questions as questions: «I don't know if…», «I wonder whether…», «I'm not sure if…» + clause, and a declarative with a tag («…, right?»). Add them as ask cues with indirect=1 in prag/data/cues.tsv; in the frame reader treat the embedded clause as the questioned frame (Ask::Frame). Measure how many silver lines with these openings are labelled ask and how the perceptron does on them.
• I2. When a yes/no question is negated, flip the polarity of the answer before matching frames, as the annotation guideline does. This is a one-line rule wherever Mova answers yes/no questions from frames (course questions «Did the fox not see…?») and in the indirect-answer reader of louis-2020-i-d-rather-just-go I1; check it with a minimal pair test (positive vs negated question, same text).
• I3. Search for the answer beyond the next turn, ranked as the annotators' data suggests: the reply right after, then the asker's follow-up and the reply after it, rarely the turns before. For tales this is a window of about three quoted lines after the question for the functional-dependence link of bunt-2020-iso-standard-dialogue-act-annotation I1. Measure how often the linked reply is right on a hand-checked set of tale questions.
• I4. A filter of direct-answer words decides which replies are indirect (yes, yeah, no way, nope, never, sure, right, you bet, of course, certainly, maybe, definitely, uh huh). Reuse it as a closed list for the agree/disagree cue slots and for counting what share of replies to questions in the tales are indirect, which tells how much the frame-based reading of answers is needed.

For Mova

38. Dialogue, discourse markers and reported speech §9 (questions and answers), 26-questions.md (declarative and tag questions), 27-negation.md (negative questions). Code: prag/data/cues.tsv, prag/src/frames.rs (Ask::Frame, negated questions, answer link), world/src/talk.rs. Measures: per-cue precision on the silver, minimal-pair test for negated questions, accuracy of answer links on tale dialogues.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.naacl-main.345

@inproceedings{sanagavarapu2022disentangling,
  title = {{Disentangling Indirect Answers to Yes-No Questions in Real Conversations}},
  author = {Sanagavarapu, Krishna and Singaraju, Jathin and Kakileti, Anusha and Kaza, Anirudh and Mathews, Aaron and Li, Helen and Brito, Nathan and Blanco, Eduardo},
  booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.345},
  url = {https://doi.org/10.18653/v1/2022.naacl-main.345}
}

References

33 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Penelope Brown and Stephen C Levinson. 1978. Universals in language usage: Politeness phenomena. In Questions and politeness: Strategies in social interaction, pages 56–311. Cambridge University Press. find
2. Jonathan P Chang, Caleb Chiam, Liye Fu, Andrew Z Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020. Convokit: A toolkit for the analysis of conversations. arXiv preprint arXiv:2005.04246. ↗
3. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184, Brussels, Belgium. Association for Computational Linguistics. find
4. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics. find
5. Marie-Catherine de Marneffe, Scott Grimm, and Christopher Potts. 2009. Not a simple yes or no: Uncertainty in indirect answers. In Proceedings of the SIGDIAL 2009 Conference, pages 136–143, London, UK. Association for Computational Linguistics. find
6. Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2010. “was it good? it was provocative.” learning the meaning of scalar adjectives. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 167–176, Uppsala, Sweden. Association for Computational Linguistics. find our card
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
8. Sarah E. Finch, James D. Finch, Daniil Huryn, William M. Hutsell, Xiaoyuan S. Huang, Han He, and Jinho D. Choi. 2021. An Approach to Inference-Driven Dialogue Management within a Social Chatbot. In Proceedings of the 4th Alexa Prize Socialbot Grand Challenge, AlexaPrize’21. find
9. Nancy Green and Sandra Carberry. 1999. Interpreting and generating indirect answers. Computational Linguistics, 25(3):389–435. find
10. Julia Bell Hirschberg. 1985. A theory of scalar implicature (natural languages, pragmatics, inference). Ph.D. thesis, University of Pennsylvania. find
11. Beth Ann Hockey, Deborah Rossen-Knill, Beverly Spejewski, Matthew Stone, and Stephen Isard. 1997. Can you predict responses to yes/no questions? yes, no, and stuff. In Fifth european conference on speech communication and technology. Citeseer. find
12. Daniel Jurafsky, Elizabeth Shriberg, and Debra Bi-asca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual, draft 13. Technical Report 97-02, University of Colorado, Boulder Institute of Cognitive Science, Boulder, CO. find
13. Sungdong Kim, Sohee Yang, Gyuwan Kim, and Sang-Woo Lee. 2020. Efficient dialogue state tracking by selectively overwriting memory. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 567–582, Online. Association for Computational Linguistics. find
14. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. find
15. Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443–2453, Copenhagen, Denmark. Association for Computational Linguistics. find
16. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Nose-worthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122–2132, Austin, Texas. Association for Computational Linguistics. find
17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. find
18. Annie Louis, Dan Roth, and Filip Radlinski. 2020. “I’d rather just go to bed”: Understanding indirect answers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7411–7425, Online. Association for Computational Linguistics. find our card
19. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. find
20. Ian Plewis and Thomas Coram Research Unit. 1982. Statistical methods for rates and proportions. find
21. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266. find
22. Deborah Rossen-Knill, Beverly Spejewski, Beth Ann Hockey, Stephen Isard, and Matthew Stone. 1997. Yes/no questions and answers in the map task corpus. find
23. Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004, pages 97–100, Cambridge, Massachusetts, USA. Association for Computational Linguistics. find
24. Anna-Brita Stenström. 1984. Questions and responses in English conversation. Krieger Pub Co. find
25. Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric Xing, and Zhiting Hu. 2019. Target-guided open-domain conversation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5624–5634, Florence, Italy. Association for Computational Linguistics. find
26. Marta Tatu. 2005. Automatic discovery of intentions in text and its application to question answering. In Proceedings of the ACL Student Research Workshop, pages 31–36, Ann Arbor, Michigan. Association for Computational Linguistics. find
27. Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Persuasion for good: Towards a personalized persuasive dialogue system for social good. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5635–5649, Florence, Italy. Association for Computational Linguistics. find
28. Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731–3741, Florence, Italy. Association for Computational Linguistics. find
29. Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 438–449, Valencia, Spain. Association for Computational Linguistics. find
30. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. find
31. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find
32. Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 917–929, Online. Association for Computational Linguistics. find
33. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204–2213, Melbourne, Australia. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/sanagavarapu-2022-disentangling-indirect-answers-yes-no/