Sarc7: Evaluating Sarcasm Detection and Generation with Seven Types and Emotion-Informed Techniques

What it is about

The 690 sarcastic utterances of the MUStARD sitcom dataset are relabelled with seven sarcasm types taken from a classroom-discourse taxonomy: self-deprecating, brooding, deadpan, polite, obnoxious, raging and manic (four annotators, a fifth for agreement: kappa 0.67; brooding, deadpan and polite are hardest even for people). Large models (GPT-4o, Claude 3.5 Sonnet, Gemini 2.5, Qwen 2.5, Llama 4) are prompted to classify the type: chain-of-thought gives the best accuracy (57%), while an «emotion-based» prompt (classify the emotion of the context, then of the utterance, then judge the incongruity) gives the best macro-F1 (0.37) and helps rare types, but makes models «trigger-happy», cutting accuracy on non-sarcastic inputs by about a quarter. For generation, fixing incongruity, shock value, context dependence and emotion per type improves type accuracy by 38% over zero-shot.

Ideas

• I1. The «trigger-happy» effect is a warning for Mova's teacher prompts: asking the teacher explicitly about emotions or incongruity raises false irony labels. Keep the irony question out of the general silver prompt (prag/src/silver.rs), or, if irony is asked for, measure the teacher's false-positive rate on a set of known sincere lines (tale lines with sincere narrator framing) before trusting it.
• I2. The context-vs-utterance emotion comparison is the same signal as the emotional-shift check proposed in ducret-2020-linguistic-features-automatic-sarcasm-detection I1; keep it as one deterministic rule in prag/src/irony.rs with a negative control (a joyful reply to good news must not fire). No need for the seven types: with human kappa 0.67 and models at macro-F1 0.37, they are not a target for the SLM.

For Mova

34. Irony and figurative language §2, §9–10. Code: prag/src/silver.rs (teacher prompt), prag/src/irony.rs. Measures: teacher false-positive rate on sincere lines; precision of the emotional-shift rule with its negative control.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.winlp-main.25

@inproceedings{xiong2025sarc7,
  title = {{Sarc7: Evaluating Sarcasm Detection and Generation with Seven Types and Emotion-Informed Techniques}},
  author = {Xiong, Lang and Gao, Raina and Jeong, Alyssa},
  booktitle = {Proceedings of the 9th Widening NLP Workshop},
  year = {2025},
  doi = {10.18653/v1/2025.winlp-main.25},
  url = {https://doi.org/10.18653/v1/2025.winlp-main.25}
}

References

19 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Stephen Skalicky and Scott Crossley. 2018. Linguistic features of sarcasm and metaphor production quality. Proceedings of the Workshop on Figurative Language Processing. find
2. Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Anthropic Report. find
3. Prasanna Biswas, Anupama Ray, and Pushpak Bhattacharyya. 2019. Computational model for understanding emotions in sarcasm: A survey. CFILT Technical Report, Indian Institute of Technology Bombay. find
4. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an _Obviously_ perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4619–4629, Florence, Italy. Association for Computational Linguistics. find
5. Google DeepMind, Rohan Anil, Stefano Arolfo, Igor Babuschkin, Lucas Beyer, Maarten Bosma, and ... 2023. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗
6. Paul Ekman. 1992. Are there basic emotions? Psychological Review, 99(3). find
7. Montgomery Gole, Williams-Paul Nwadiugwu, and Andriy Miranskyy. 2024. On sarcasm detection with openai gpt-based models. In 2024 34th International Conference on Collaborative Advances in Software and COmputiNg (CASCON), pages 1–6. IEEE. find
8. Nivin A Helal, Ahmed Hassan, Nagwa L Badr, and Yasmine M Afify. 2024. A contextual-based approach for sarcasm detection. Scientific Reports, 14(1):15415. find
9. Joshua Lee, Wyatt Fong, Alexander Le, Sur Shah, Kevin Han, and Kevin Zhu. 2024. Pragmatic metacognitive prompting improves llm performance on sarcasm detection. arXiv preprint arXiv:2412.04509. ↗
10. John S Leggitt and Raymond W Gibbs. 2000. Emotional reactions to verbal irony. Discourse processes, 29(1):1–24. find
11. Meta AI. 2024. Llama-4-maverick-17b-128e-original. Hugging Face Model Hub: https://huggingface.co/meta-llama/ Llama-4-Maverick-17B-128E-Original. Accessed: 2025-06-27. ↗
12. OpenAI. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. ↗
13. Sawsan Abdul-Muneim Qasim. 2021. A critical pragmatic study of sarcasm in american and british social interviews. Journal of Strategic Research in Social Science. find
14. Ellen Riloff, Aditya Qadir, Prajakta Surve, Lakshika De Silva, Nisheeth Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714. ACL. find our card
15. Qwen Team. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. ↗
16. Ben Yao, Yazhou Zhang, Qiuchi Li, and Jing Qin. 2024. Is sarcasm detection a step-by-step reasoning process in large language models? arXiv preprint arXiv:2407.12725. ↗
17. Yazhou Zhang, Chunwang Zou, Zheng Lian, Prayag Tiwari, and Jing Qin. 2024. Sarcasmbench: Towards evaluating large language models on sarcasm understanding. arXiv preprint arXiv:2408.11319. ↗
18. Xingjie Zhuang, Fengling Zhou, and Zhixin Li. 2025. Multi-modal sarcasm detection via knowledge-aware focused graph convolutional networks. ACM Transactions on Multimedia Computing, Communications and Applications. find
19. Ari Tantra Zuhri and Rakhmat Wahyudin Sagala. 2022. Irony and sarcasm detection on public figure speech. Journal of Elementary School Education, 1(1):41–45. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/xiong-2025-sarc7-evaluating-sarcasm-detection-generation/