Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue

What it is about

The authors build a sarcasm corpus from online debate forums (Internet Argument Corpus 2.0) with three balanced parts: generic sarcasm, rhetorical questions and hyperbole. To make annotation affordable they first learn high-precision patterns of not-sarcastic posts with the weakly supervised pattern learner AutoSlog-TS (80–90% precision) and filter out posts containing them, which leaves a pool richer in sarcasm; posts are annotated in quote–response pairs by selected crowd workers (89% agreement with the majority for the core annotators). Rhetorical questions are found by the «question in the middle of a post answered by the same speaker» rule (75% truly rhetorical in a check) and hyperbole by regex cues; the share of sarcastic posts per cue was measured (for example oh wait 87%, wow 44%). Simple n-gram and Word2Vec classifiers reach precision up to 0.73 and F up to 0.74; the learned patterns show what separates sarcastic from sincere uses when the surface cue is held constant.

Ideas

• I1. Replace the flat +1 weights of pragmatic cues in prag/src/irony.rs («oh great», «yeah right», «!!», …) by weights from measured precision, as the paper's cue table does: for each cue, the share of silver sentences with it whose means marks irony. Keep the weight as a small integer in the cue list so the explanation still shows it. Measure irony-candidate precision on the silver before and after.
• I2. Build a «not ironic» filter of high-precision sincere patterns and apply it before the irony rule: sentences with a sincere-gratitude or sincere-praise frame whose target is a narrated good event («I love my mother», «Thank you for saving me» after a rescue). The paper found negative patterns easier to make precise than positive ones; for Mova this lowers false candidates. Negative controls already exist in irony.rs tests; add the filter's own.
• I3. Interpret a rhetorical question as the assertion of the opposite polarity («Can you read?» → you cannot read; «Who would do that?» → nobody would), the default the paper adopts after Frank (1990). Add this as the means of rhetorical-question candidates (detected as in oraby-2017-are-you-serious-rhetorical-questions I1).

For Mova

34. Irony and figurative language §4 (cues of irony), §5 (hyperbole), §6 (rhetorical questions), §10 (Mova's irony detector). Code: prag/src/irony.rs, prag/data/cues.tsv. Measures: irony-candidate precision and recall on silver lines whose means names irony or sarcasm.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-3604 · arXiv 1709.05404

@inproceedings{oraby2016creating,
  title = {{Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue}},
  author = {Oraby, Shereen and Harrison, Vrindavan and Reed, Lena and Hernandez, Ernesto and Riloff, Ellen and Walker, Marilyn},
  booktitle = {Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2016},
  doi = {10.18653/v1/w16-3604},
  eprint = {1709.05404},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w16-3604}
}

References

37 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, volume 2, pages 757–762. find
2. Robert Abbott, Brian Ecker, Pranav Anand, and Marilyn Walker. 2016. Internet argument corpus 2.0: An sql schema for dialogic social media and the corpora to go with it. In Language Resources and Evaluation Conference, LREC2016. find
3. Raquel Justo, Thomas Corcoran, Stephanie M Lukin, Marilyn Walker, and M Inés Torres. 2014. Extracting relevant knowledge for the detection of sarcasm and nastiness in the social web. Knowledge-Based Systems. find
4. Dario Bertero and Pascale Fung. 2016. A long short-term memory framework for detecting humor in dialogues. In North American Association of Computational Linguistics Conference, NAACL-16. find
5. Roger J. Kreuz and Richard M. Roberts. 1995. Two cues for verbal irony: Hyperbole and the ironic tone of voice. Metaphor and Symbolic Activity, 10(1):21–31. find
6. Shohini Bhattasali, Jeremy Cytryn, Elana Feldman, and Joonsuk Park. 2015. Automatic identification of rhetorical questions. In Proc. of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Short Papers). find
7. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in nlp. In North American Association of Computational Linguistics Conference, NAACL-16. find
8. Laura Cano Mora. 2009. All or nothing: a semantic analysis of hyperbole. Revista de Lingüı́stica y Lenguas Aplicadas, pages 25–35. find
9. Stephanie Lukin and Marilyn Walker. 2013. Really? well. apparently bootstrapping improves the performance of sarcasm and nastiness classifiers for online dialogue. NAACL 2013, page 30. find
10. Herbert L. Colston and Shauna B. Keller. 1998. You’ll never believe this: Irony and hyperbole in expressing surprise. Journal of psycholinguistic research, 27(4):499–513. find
11. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL (System Demonstrations), pages 55–60. find
12. Herbert L. Colston and Jennifer O’Brien. 2000. Contrast and pragmatics in figurative language: Anything understatement can do, irony can do better. Journal of Pragmatics, 32(11):1557 – 1583. find
13. Michael McCarthy and Ronald Carter. 2004. ”there’s millions of them”: hyperbole in everyday conversation. Journal of Pragmatics, 36(2):149–184. find
14. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Language Resources and Evaluation Conference, LREC2012. find
15. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
16. Jane Frank. 1990. You call that a rhetorical question?: Forms and functions of rhetorical questions in conversation. Journal of Pragmatics, 14(5):723–738. find
17. Shereen Oraby, Lena Reed, Ryan Compton, Ellen Riloff, Marilyn Walker, and Steve Whittaker. 2015. And thats a fact: Distinguishing factual and emotional argumentation in online dialogue. 2nd Workshop on Argument Mining, NAACL HLT 2015, page 116. find
18. Raymond W. Gibbs. 2000. Irony in talk among friends. Metaphor and Symbol, 15(1):5–27. find
19. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: a closer look. In Proc. of the 49th Annual Meeting of the Association for Computational Linguistics, volume 2, pages 581–586. find
20. Alan Partington. 2007. Irony and reversal of evaluation. Journal of Pragmatics, 39(9):1547–1569. find
21. Chung-hye Han. 1997. Deriving the interpretation of rhetorical questions. In The Proc. of the Sixteenth West Coast Conference on Formal Linguistics, WC-CFL16. find
22. Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, Edouard Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. find
23. Julia Hirschberg. 1985. A Theory of Scalar Implicature. Ph.D. thesis, University of Pennsylvania, Computer and Information Science. find
24. Cornelia Ilie. 1994. What else can I tell you?: a pragmatic study of English rhetorical questions as discursive and argumentative acts. Acta Universitatis Stockholmiensis: Stockholm studies in English. Almqvist & Wiksell International. find
25. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proc. of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45–50. In Proc. of the Association for Computational Linguistics, pages 512–516. find
26. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in twitter. Data Knowl. Eng., 47(1):239–268, March. find
27. Antonio Reyes, Paolo Rosso, and Davide Buscaldi. 2012. From humor recognition to irony detection: The figurative language of social media. Data Knowl. Eng., 74:1–12. find
28. Ellen Riloff and Janyce Wiebe. 2003. Learning extraction patterns for subjective expressions. In Proc. of the 2003 conference on Empirical methods in natural language processing-Volume 10, pages 105–112. Association for Computational Linguistics. find
29. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proc. of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP 2013). find our card
30. Ellen Riloff. 1996. Automatically generating extraction patterns from untagged text. In Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI-96), pages 1044–1049. find
31. Deborah Schaffer. —2005—. Can rhetorical questions function as retorts? : Is the pope catholic? Journal of Pragmatics, 37:433–600. find
32. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642. find
33. Reid Swanson, Stephanie Lukin, Luke Eisenberg, Thomas Chase Corcoran, and Marilyn A Walker. 2014. Getting reliable annotations for sarcasm in online dialogues. In Language Resources and Evaluation Conference, LREC 2014. find
34. Michael Thelen and Ellen Riloff. 2002. A bootstrapping method for learning semantic lexicons using extraction pattern contexts. In Proc. of the ACL-02 Conference on Empirical Methods In Natural Language Processing, pages 214–221. find
35. Marilyn Walker, Pranav Anand, Robert Abbott, and Jean E. Fox Tree. 2012a. A corpus for research on deliberation and debate. In Language Resources and Evaluation Conference, LREC2012, pages 812– 817. find
36. Marilyn A. Walker, Pranav Anand, Rob Abbott, Jean E Fox Tree, Craig Martell, and Joseph King. 2012b. That’s your evidence?: Classifying stance in online political debate. Decision Support Systems, 53(4):719–729. find
37. Byron C. Wallace, Do Kook Choe, Laura Kertz, and Eugene Charniak. 2014. Humans require context to infer ironic intent (so computers probably do, too). find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/oraby-2016-creating-characterizing-diverse-corpus-sarcasm/