"I'll be there for you": The One with Understanding Indirect Answers

What it is about

Instead of crowd-writing questions and answers as Circa did, the authors mined 5,930 polar question–indirect answer pairs from the transcripts of a sitcom (228 episodes), removed bare yes/no-type words from the answers, sometimes added context to the question, and had three in-house annotators label them with Circa's relaxed classes (yes, no, conditional yes, neither). Agreement is high (Fleiss kappa 0.88; all three agree on 75%). The data is harder than Circa: longer, more varied answers, irony, mind changes within an answer. CNN classifiers reach 55% accuracy / 33 macro-F1 with GloVe and 64 / 49 with BERT features (majority 49 / 16); the answer alone again carries most of the signal. Training on each annotator's raw labels through a "crowd layer" raises macro-F1 (to 55 on dev), mostly on the rare conditional class, while adding Circa to training does not help.

Ideas

• I1. Mine the same kind of test set from Mova's own tales: a quoted yes/no question followed by a quoted reply of another character, bare yes/no stripped, labelled by the teacher with the four relaxed classes. It is in domain and licence-clean, unlike either sitcom or crowd data, and is the dev set for the frame-based reading of indirect answers proposed under louis-2020-i-d-rather-just-go I1.
• I2. Use the paper's two lists of answer cues as a starting cue set with strength: the removed direct mappers (yes, yeah, yep, no, nope, nah) as strong agree/disagree cues, and the kept ones (of course (not), sure, I guess (not), no way, all right, fine, not really) as weaker or hedged ones. Put them into prag/data/cues.tsv (our own list) and check how many silver agree/disagree lines each cue covers and with what precision.
• I3. For the hardest prag fields (indirect, polarity), keep more than one teacher judgement where they differ (two prompts or two teacher runs) and train the perceptron on soft or weighted targets instead of one label. The paper's gain from raw annotations is on the rare class, which is where Mova's act and indirect fields are weakest. Measure macro-F1 for rare classes in prag eval; control: the same with the labels of the two runs shuffled between sentences.
• I4. The failed Circa+Friends union is a warning for Mova: data from another genre may not help tales. Any outside data for prag gets a dev check per genre (tales, Tatoeba, lessons) before it is kept.

For Mova

38. Dialogue, discourse markers and reported speech §9 (questions and answers in dialogue), 31-speech-acts.md (agree/disagree). Code: prag/data/cues.tsv, prag/src/model.rs (weighted targets), prag/src/silver.rs (second teacher judgement). Measures: accuracy and macro-F1 on a tale question–reply set; rare-class macro-F1 in prag eval.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.codi-main.1

@inproceedings{damgaard2021i,
  title = {{"I'll be there for you": The One with Understanding Indirect Answers}},
  author = {Damgaard, Cathrine and Toborek, Paulina and Eriksen, Trine and Plank, Barbara},
  booktitle = {Proceedings of the 2nd Workshop on Computational Approaches to Discourse},
  year = {2021},
  doi = {10.18653/v1/2021.codi-main.1},
  url = {https://doi.org/10.18653/v1/2021.codi-main.1}
}

References

21 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24. find
2. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
3. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics. find
4. Marie-Catherine de Marneffe, Scott Grimm, and Christopher Potts. 2009. Not a simple yes or no: Uncertainty in indirect answers. In Proceedings of the SIGDIAL 2009 Conference, pages 136–143, London, UK. Association for Computational Linguistics. find
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
6. Nancy Green and Sandra Carberry. 1992. Conversational implicatures in indirect replies. In 30th Annual Meeting of the Association for Computational Linguistics, pages 64–71, Newark, Delaware, USA. Association for Computational Linguistics. find
7. Nancy Green and Sandra Carberry. 1999. Interpreting and generating indirect answers. Computational Linguistics, 25(3):389–435. find
8. Beth Ann Hockey, Deborah Rossen-Knill, Beverly Spe-jewski, Matthew Stone, and Stephen Isard. 1997. Can you predict responses to yes/no questions? yes, no, and stuff. In Fifth European Conference on Speech Communication and Technology. find
9. and Cathy Sotillo. 1993. The HCRC map task corpus: natural dialogue for speech recognition. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993. find
10. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar. Association for Computational Linguistics. find
11. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. The Journal of Artificial Intelligence Research, Forthcoming. find
12. Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy. 2014. Learning from multiple annotators with varying expertise. Machine Learning, page 291–327. find
13. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. volume abs/1412.6980. ↗
14. Annie Louis, Dan Roth, and Filip Radlinski. 2020. “I’d rather just go to bed”: Understanding indirect answers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7411–7425, Online. Association for Computational Linguistics. find our card
15. Jennimaria Palomaki, Olivia Rhinehart, and Michael Tseng. 2018. A case for a range of acceptable annotations. In SAD/CrowdBias@ HCOMP, pages 19– 31. find
16. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. find our card
17. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), page 1532–1543, Doha, Qatar. Association for Computational Linguistics. find
18. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Learning part-of-speech taggers with inter-annotator agreement loss. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–751, Gothenburg, Sweden. Association for Computational Linguistics. find
19. Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. 2013. Learning from multiple annotators: Distinguishing good from random labelers. Pattern Recognition Letters, 34(12):1428–1436. find
20. Filipe Rodrigues and Francisco C. Pereira. 2018. Deep learning from crowds. In Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18). Association for the Advancement of Artificial Intelligence. find
21. Anna-Brita Stenström. 1984. Questions and responses in English conversation. Krieger Pub Co. Henry S. Thompson, Anne H. Anderson, Ellen Gurman Bard, Gwyneth Doherty-sneddon, Alison Newlands, find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/damgaard-2021-i-ll-be-there-you/