iSarcasm: A Dataset of Intended Sarcasm

What it is about

Sarcasm datasets label either by hashtags (#sarcasm, /s) or by annotators; the first mislabels tweets that merely mention sarcasm and misses untagged ones, the second captures how readers perceive a text, not what its author meant. iSarcasm asks Twitter users themselves for one sarcastic and three non-sarcastic tweets of their own, with an explanation of why the sarcastic one is sarcastic and a sincere rephrasing; a linguist then sorts the sarcastic tweets into sarcasm, irony, satire, understatement, overstatement and rhetorical question, discarding unclear ones (777 sarcastic, 3,707 non-sarcastic tweets kept). Models that reach F 0.87 on earlier datasets score at most 0.36 on iSarcasm, and third-party annotators also miss much of the intended sarcasm, which suggests earlier datasets were biased towards obvious cases.

Ideas

• I1. Tales offer intended labels for free: the narrator often states a character's intent («said the fox slyly», «mocked the hare», «he answered in jest», «she said, laughing at him»). Collect the quoted lines with such narrator markers as an «intended irony» test set for prag/src/irony.rs, and the same speakers' plainly introduced lines as negatives. Evaluate the detector with the marker masked (see hee-2016-monday-mornings-are-my-fave I2).
• I2. Treat the Opus silver as perceived irony and measure its agreement with the narrator-intended set from I1. If the teacher misses intended irony as the paper's annotators did, irony labels should not be trusted from silver alone.
• I3. Give irony candidates a subtype from the paper's six categories, each with its own deterministic cue: sarcasm (critical and aimed at the addressee: second-person target), overstatement (extreme degree words or numbers), understatement (litotes «not bad», «a bit» for a large thing), rhetorical question (see oraby-2017-are-you-serious-rhetorical-questions), irony (contradicts the situation without a target), satire. The subtype goes into the explanation and can be scored per class.

For Mova

34. Irony and figurative language §2 (verbal irony and sarcasm), §5 (hyperbole and understatement), §10 (Mova's detector), §12 (figurative language in the tales). Code: prag/src/irony.rs, prag/src/frames.rs (narrator markers). Measures: precision/recall on narrator-intended lines; teacher-vs-intended agreement.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.118 · arXiv 1911.03123

@inproceedings{oprea2020isarcasm,
  title = {{iSarcasm: A Dataset of Intended Sarcasm}},
  author = {Oprea, Silviu and Magdy, Walid},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.118},
  eprint = {1911.03123},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.118}
}

References

49 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Previous models that achieved high performance in detecting sarcasm on datasets sampling perceived sarcasm (third-party labels) or hash-tagged sarcasm (distant supervision) have failed dramatically to detect sarcasm as meant by its author. This motivates the need to develop more effective methods for detecting intended sarcasm. Potentially, building models that account for sociocultural traits of the authors (available on, or inferred from, their Twitter profiles), or consider other contextual elements to judge the sarcasm in our dataset (Rockwell and Theriot, 2001). Previous research has considered certain contextual elements (Bamman and Smith, 2015b; Amir et al., 2016; Hazarika et al., 2018; Oprea and Magdy, 2019), but only on sarcasm captured by previous labelling methods. We believe the iSarcasm dataset, with its novel method of sampling sarcasm as intended by its Gavin Abercrombie and Dirk Hovy. 2016. Putting Sarcasm Detection into Context: The Effects of Class Imbalance and Manual Labelling on Supervised Machine Classification of Twitter Conversations. In Proceedings of the ACL 2016 Student Research Workshop, pages 107–113. ACL. find
2. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mario J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In CoNLL, pages 167–177. ACL. find
3. Salvatore Attardo. 2002. Talk is cheap: Sarcasm, alienation, and the evolution of language. Journal of Pragmatics, 34. find
4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR. find
5. David Bamman and Noah A. Smith. 2015a. Contextualized sarcasm detection on twitter. In ICWSM, pages 574–577. AAAI Press. find
6. David Bamman and Noah A. Smith. 2015b. Contextualized sarcasm detection on twitter. In ICWSM, pages 574–577. AAAI Press. find
7. Megan L. Dress, Roger J. Kreuz, Kristen E. Link, and Gina M. Caucci. 2008. Regional variation in the use of sarcasm. JLS, 27(1):71–85. find
8. Francesco Barbieri, Francesco Ronzano, and Horacio Saggion. 2014a. Italian irony detection in twitter: a first approach. In CLiC-it, page 28. AILC. find
9. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In LREC. ELRA. find
10. Francesco Barbieri, Horacio Saggion, and Francesco Ronzano. 2014b. Modelling sarcasm in twitter, a novel approach. In WASSA, pages 50–58. ACL. find
11. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: A closer look. In HLT, pages 581–586. ACL. find
12. Farah Benamara, Cyril Grouin, Jihen Karoui, Véronique Moriceau, and Isabelle Robba. 2017. Analyse d’opinion et langage figuratif dans des tweets: présentation et résultats du défi fouille de textes deft2017. find
13. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. Cascade: Contextual sarcasm detection in online discussion forums. In COLING, pages 1837–1848. ACL. find
14. S. K. Bharti, K. S. Babu, and S. K. Jena. 2015. Parsing-based sarcasm sentiment recognition in twitter data. In ASONAM, pages 1373–1380. ACM. find
15. Delia Irazú Hernández Farı́as, Emilio Sulis, Viviana Patti, Giancarlo Ruffo, and Cristina Bosco. 2015. Valento: Sentiment analysis of figurative language tweets with irony and sarcasm. In SemEval, pages 694–698. ACL. find
16. S. K. Bharti, K. S. Babu, and S. K. Jena. 2015. Parsing-based sarcasm sentiment recognition in twitter data. In ASONAM, pages 1373–1380. ACM. find
17. Mondher Bouazizi and Tomoaki Ohtsuki. 2015. Opinion mining in twitter how to make use of sarcasm to enhance sentiment analysis. In ASONAM, pages 1594–1597. ACM. find
18. F. Bravo-Marquez, E. Frank, S. M. Mohammad, and B. Pfahringer. 2016. Determining word-emotion associations from tweets by multi-label classification. In WI, pages 536–539. IEEE. find
19. Felipe Bravo-Marquez, Marcelo Mendoza, and Barbara Poblete. 2014. Meta-level sentiment models for big social data analysis. Knowledge-Based Systems, 69:86 – 99. find
20. Reynier Ortega Bueno, Francisco M. Rangel Pardo, Delia Irazú Hernández Farı́as, Paolo Rosso, Manuel Montes y Gómez, and José Medina-Pagola. 2019. Overview of the task on irony detection in spanish variants. In IberLEF@SEPLN. find
21. John D Campbell and Albert N Katz. 2012. Are there necessary conditions for inducing a sense of sarcastic irony? Discourse Processes, 49(6):459–480. find
22. Sepp Hochreiter and Jrgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
23. Julia Jorgensen, George A Miller, and Dan Sperber. 1984. Test of the mention theory of irony. Journal of Experimental Psychology: General, 113(1):112– 120. find
24. Aditya Joshi, Pushpak Bhattacharyya, Mark Carman, Jaya Saraswati, and Rajita Shukla. 2016a. How do cultural differences impact the quality of sarcasm annotation?: A case study of indian annotators and american text. In LaTeCH, pages 95–99. ACL. find
25. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In IJCNLP, pages 757–762. ACL. find
26. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016b. Are word embedding-based features useful for sarcasm detection? In EMNLP, pages 1006–1011. ACL. find
27. David S. Kaufer. 1981. Understanding ironic communication. Journal of Pragmatics, 5(6):495 – 510. find
28. Alessandra Teresa Cignarella, Simona Frenda, Valerio Basile, Cristina Bosco, Viviana Patti, Paolo Rosso, et al. 2018. Overview of the EVALITA 2018 task on irony detection in Italian tweets (IronITA). In Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian (EVALITA 2018), volume 2263, pages 1–6. CEUR-WS. find
29. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2018. A large self-annotated corpus for sarcasm. In LREC. ELRA. find
30. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. ↗
31. John S Leggitt and Raymond W Gibbs Jr. 2000. Emotional reactions to verbal irony. find
32. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcasm in twitter and amazon. In CoNLL, pages 107–116. ACL. find
33. Quoc Le and Tomas Mikolov. 2014. Distributed Representations of Sentences and Documents. In ICML, pages 1188–1196. PMLR. find
34. Christine Liebrecht, Florian Kunneman, and Antal Van den Bosch. 2013. The perfect solution for detecting sarcasm in tweets #not. In WASSA, pages 29–37. ACL. find
35. Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In LREC. ELRA. find
36. Saif Mohammad and Felipe Bravo-Marquez. 2017. WASSA-2017 shared task on emotion intensity. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 34–49. Association for Computational Linguistics. find
37. Silviu Oprea and Walid Magdy. 2019. Exploring author context for detecting intended vs perceived sarcasm. In ACL, pages 2854–2859. ACL. detection with densely connected LSTM and multi-task learning. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 51– 56, New Orleans, Louisiana. Association for Computational Linguistics. find
38. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480–1489, San Diego, California. Association for Computational Linguistics. find
39. Silviu Oprea and Walid Magdy. 2020. The effect of sociocultural variables on sarcasm communication online. Proceedings of The 23rd ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW). find
40. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In COLING, pages 213–223. ACL. find
41. Antonio Reyes and Paolo Rosso. 2012. Making objective decisions from subjective data: Detecting irony in customer reviews. Decision Support Systems, 53(4):754–760. find
42. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In EMNLP, pages 704–714. ACL. find our card
43. Patricia Rockwell and Evelyn M. Theriot. 2001. Culture, gender, and gender mix in encoders of sarcasm: A self-assessment analysis. Communication Research Reports, 18(1):44–52. find
44. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018. Disan: Directional self-attention network for rnn/cnn-free language understanding. In AAAI. AAAI Press. find
45. Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. In ACL, pages 1010–1020. ACL. find
46. Mike Thelwall, Kevan Buckley, and Georgios Pal-toglou. 2012. Sentiment strength detection for the social web. J. Am. Soc. Inf. Sci. Technol., 63(1):163– 173. find
47. Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2018. SemEval-2018 task 3: Irony detection in English tweets. In SemEval, pages 39–50. ACL. find
48. Deirdre Wilson. 2006. The pragmatics of verbal irony: Echo or pretence? Lingua, 116(10):1722–1743. find
49. Chuhan Wu, Fangzhao Wu, Sixing Wu, Junxin Liu, Zhigang Yuan, and Yongfeng Huang. 2018. THU NGN at SemEval-2018 task 3: Tweet irony find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/oprea-2020-isarcasm-dataset-intended-sarcasm/