Developing a Corpus of Indirect Speech Act Schemas

What it is about

The paper borrows the design of Winograd schemas for indirect speech acts. An ISA schema is an utterance, its literal meaning, several contexts and the intended meaning in each; the utterance is indirect in a context when intended ≠ literal, and a schema must have at least one direct and one indirect context, so a system that ignores context cannot get both halves right. Contexts are sets of feature values: location, task, speaker role, hearer role, co-present item, interaction history. "Contextual difficulty" is the number of features that differ between the two contexts (one differing feature = hardest). The authors took 215 «Can you …?» utterances from DailyDialog, had crowd workers write contexts (92 schemas), and validated them: lay validators agreed with the authors only 37% and 44% of the time, and after expert tie-breaking 30 schemas (60 test items) remained. Utterances like «Can you tell me about it?» were dropped as truly ambiguous: they ask for both the answer and the act.

Ideas

I1. Build a small minimal-pair test for Mova's indirect field from the tales: the same «Can you / Will you / Could you …?» line in two paragraphs where it is once a question and once a request. Score it by pairs, as Winograd schemas are scored: a pair counts only if both halves are right. Today's features are sentence-local apart from prev, so pair accuracy should be near zero. That makes it a gate that can go red, and a yardstick for any context feature added to prag/src/feats.rs.
I2. Derive the paper's context features from what Mova already reads, instead of from crowd workers: co-present item = the object of the request appears among the entities of the earlier frames of the paragraph; interaction history = an earlier frame of the hearer doing or being able to do the requested verb; roles = a vocative or title of the addressee (Mother, King). Add them as prag feature slots and measure indirect macro-F1 (59.9 now) and request recall.
I3. Record the contextual difficulty (number of differing context features) for each test pair and report accuracy per difficulty level, so a gain can be traced to easy or hard pairs.
I4. Give «Can you tell me / show me …?» a label convention of its own instead of forcing ask vs request: ISO's conditional request covers it. In the silver prompt (prag/src/silver.rs, RULES) and the agreer (prag/src/agree.rs) it should be request with indirect=1. Count how often the teacher and the SLM disagree on this form before and after.

For Mova

31. Speech acts §5 (indirect speech acts, §5.4 Mova's indirect requests) and 32-politeness-and-hedging.md §3 (requests). Code: prag/src/feats.rs (context slots), prag/src/silver.rs and prag/src/agree.rs (label convention), world/src/talk.rs (the precondition reasoning after Sarathy et al. 2020 already uses the same context idea). Measures: pair accuracy on the minimal pairs, indirect macro-F1 in prag eval.

Topics: Speech act theory

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{roque2020developing,
  title = {{Developing a Corpus of Indirect Speech Act Schemas}},
  author = {Roque, Antonio and Tsuetaki, Alexander and Sarathy, Vasanth and Scheutz, Matthias},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.28/}
}

References

22 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Amsili, P. and Seminck, O. (2017). A Google-proof collection of French Winograd Schemas. In Proceedings of the 2nd Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2017), pages 24–29. find
Bender, D. (2015). Establishing a human baseline for the Winograd Schema Challenge. In MAICS, pages 39–45. find
Briggs, G. M. and Scheutz, M. (2013). A hybrid architectural approach to understanding and appropriately generating indirect speech acts. In Twenty-Seventh AAAI Conference on Artificial Intelligence. find
Eric, M. and Manning, C. D. (2017). Key-value retrieval networks for task-oriented dialogue. In Proceedings of the SIGDIAL 2017 Conference. find
Gibbs Jr, R. W. (1979). Contextual effects in understanding indirect requests. Discourse Processes, 2(1):1–10. find
Levesque, H., Davis, E., and Morgenstern, L. (2012). The Winograd Schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
Li, Y., Su, H., Shen, X., Li, W., Cao, Z., and Niu, S. (2017). DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986–995, Taipei, Taiwan, November. Asian Federation of Natural Language Processing. find
McDonnell, J., Martin, J., Markant, D., Coenen, A., Rich, A., and Gureckis, T. (2012). psiTurk (version 1.02)[software]. New York, NY: New York University. find
Morgenstern, L., Davis, E., and Ortiz, C. L. (2016). Planning, executing, and evaluating the Winograd Schema Challenge. AI Magazine, 37(1):50–54. find
OED Online. (2019). schema, n. https://www.oed.com/view/Entry/172307 (accessed December 02, 2019). ↗
Perrault, C. R. and Allen, J. F. (1980). A plan-based analysis of indirect speech acts. Computational Linguistics, 6(3-4):167–182. find
Rahman, A. and Ng, V. (2012). Resolving complex cases of definite pronouns: the Winograd Schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 777–789. Association for Computational Linguistics. find
Roemmele, M., Bejan, C. A., and Gordon, A. S. (2011). Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. find
Rus, V., McCarthy, P. M., McNamara, D. S., and Graesser, A. C. (2008). A study of textual entailment. International Journal on Artificial Intelligence Tools, 17(04):659–685. find
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. (2020). WINOGRANDE: An adversarial Winograd Schema Challenge at scale. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20). find
Searle, J. R. (1975). Indirect speech acts. Syntax & Semantics, 3: Speech Act, pages 59–82. find
Storks, S., Gao, Q., and Chai, J. Y. (2019). Commonsense reasoning for natural language understanding: A survey of benchmarks, resources, and approaches. arXiv preprint arXiv:1904.01172. ↗
Sun, K., Yu, D., Chen, J., Yu, D., Choi, Y., and Cardie, C. (2019). DREAM: A challenge dataset and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics. find
Trichelair, P., Emami, A., Trischler, A., Suleman, K., and Cheung, J. C. K. (2019). How reasonable are common-sense reasoning tasks: A case-study on the Winograd schema challenge and SWAG. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3382–3387, Hong Kong, China, November. Association for Computational Linguistics. find
Williams, T., Thames, D., Novakoff, J., and Scheutz, M. (2018). Thank you for sharing that interesting fact!: Effects of capability and context on indirect speech act use in task-based human-robot dialogue. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, pages 298–306. ACM. find
Wilske, S. and Kruijff, G.-J. (2006). Service robots dealing with indirect speech acts. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4698–4703. IEEE. find
Zellers, R., Bisk, Y., Schwartz, R., and Choi, Y. (2018). SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP). find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/roque-2020-developing-corpus-indirect-speech-act/