Irony Detection in Hebrew Documents: A Novel Dataset and an Evaluation of Neural Classification Methods

What it is about

Hebrew has no capital letters, so a single word in quotation marks can mean three different things: a name (naming, e.g. a prayer or a party), a lexical peculiarity (slang, a live metaphor, a register shift), or ironic criticism. The authors annotate 4,190 such cases from Israeli op-eds of 2019–2020, with three pragmatics experts: 45.1% naming, 23.4% lexical peculiarity, 31.5% irony. They pass each sentence through DictaBERT, take the contextual vector of the quoted word, and train a small MLP with 10-fold cross-validation. Irony F1 is 0.780 with the target word masked, 0.818 unmasked, and 0.825 when the sentence CLS vector is added. Naming reaches 0.908; lexical peculiarity stays at about 0.5–0.6. Extra co-text (up to five sentences) and redundancy features (quote counts, paragraph length, recurrence) do not improve irony.

Ideas

• I1. A pragmatic cue flag: a single word in quotation marks that the sentence treats as non-literal is a cue for irony, not a verdict. The masked-target result (F1 0.78 from the surrounding words alone) suggests that a rule over the frame of the quoted word, with the criticism context, could give a usable flag.
• I2. Names in quotes as a lookup: the naming class is the easiest one (F1 0.91), because its members are titles, prayers, parties and institutions. A gazetteer of such names can catch them deterministically, before any pragmatic reading.
• I3. Report per-class precision and recall. The lexical-peculiarity class is the weak point in every setting, so any quote rule must be scored on that class on its own. The text does not state the dataset licence; check it before any test.

For Mova

I1 is a pragmatic flag for prag, planned in 9. Pragmatics and politeness and applied once Hebrew input reaches prag. I2 goes into 8. Lexicon and register for names, slang and register shift. I3 is the scoring template for any quote rule.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.nlp4dh-1.9

@inproceedings{shmidman2025irony,
  title = {{Irony Detection in Hebrew Documents: A Novel Dataset and an Evaluation of Neural Classification Methods}},
  author = {Shmidman, Avi and Weizman, Elda and Gerczuk, Avishay},
  booktitle = {Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities},
  year = {2025},
  doi = {10.18653/v1/2025.nlp4dh-1.9},
  url = {https://doi.org/10.18653/v1/2025.nlp4dh-1.9}
}

References

47 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Salvatore Attardo, Jodi Eisterhol, Jennifer Hay, and Isabella Poggi. 2003. Multimodal markers of irony and sarcasm. Humor, 16(2):243–260. find
2. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010b. Semi-supervised recognition of sarcasm in Twitter and Amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 107–116, Uppsala, Sweden. Association for Computational Linguistics. find
3. Christian Burgers, Margot van Mulken, and Peter Jan Schellens. 2012. Type of evaluation and marking of irony: The role of perceived complexity and comprehension. Journal of Pragmatics, 44(3):231–242. find
4. Raymond W. Gibbs and Herbert L. Colston. 2023. Irony and Thought: The State of the Art, pages 3–14. Cambridge Handbooks in Psychology. Cambridge University Press. find
5. Christian Burgers, Margot van Mulken, and Peter Jan Schellens. 2013. The use of co-textual irony markers in written discourse. Humor, 26(1):45–68. find
6. Preni Golazizian, Behnam Sabeti, Seyed Arad Ashrafi Asli, Zahra Majdabadi, Omid Momenzadeh, and Reza Fahmi. 2020. Irony detection in Persian language: A transfer learning approach using emoji prediction. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2839– 2845, Marseille, France. European Language Resources Association. find
7. Salvatore Attardo. 2000. Irony as relevant inappropriateness. Journal of Pragmatics, 32(6):793–826. find
8. Konstantin Buschmeier, Philipp Cimiano, and Roman Klinger. 2014. An impact analysis of features in a classification approach to irony detection in product reviews. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 42–49. find
9. Herbert Paul Grice. 1968. Utterer’s Meaning, Sentence-Meaning, and Word-Meaning, page 49–66. Springer Netherlands. find
10. Paula Carvalho, Luis Sarmento, Mário J. Silva, and Eu-génio de Oliveira. 2009. Clues for detecting irony in user-generated contents: oh...!! it’s "so easy" ;-). In Proceedings of the 1st International CIKM Workshop on Topic-Sentiment Analysis for Mass Opinion, pages 53–56. find
11. Herbert Paul Grice. 1975. Logic and conversation. Syntax and Semantics: Speech Acts, III:41–58. find
12. Herbert Paul Grice. 1978. Further notes on logic and conversation. volume 9, pages 113–127. Pragmatics. Academic Press. find
13. Paula Carvalho, Luís Sarmento, Jorge Teixeira, and Mário J. Silva. 2011. Liars and saviors in a sentiment annotated corpus of comments to political debates. In Proceedings of the 49th Annual Meeting of the Helmut Gruber. 1993. Evaluation devices in newspaper reports. Journal of Pragmatics, 19(5):469–486. find
14. Helmut Gruber. 2015a. Intertextual references in Austrian parliamentary debates: Between evaluation and argumentation, page 25–56. John Benjamins Publishing Company. find
15. Helmut Gruber. 2015b. Policy-oriented argumentation or ironic evaluation: A study of verbal quoting and positioning in austrian politicians’ parliamentary debate contributions. Discourse Studies, 17(6):682–702. find
16. Helmut Gruber. 2017. Quoting and retweeting as communicative practices in computer mediated discourse. Discourse, Context amp; Media, 20:1–9. find
17. Galia Hirsch. 2011. Redundancy, irony and humor. Language Sciences, 33(2):316–329. find
18. Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys, 50(5):1–22. find
19. Jihen Karoui, Farah Benamara, Véronique Moriceau, Viviana Patti, Cristina Bosco, and Nathalie Aussenac-Gilles. 2017. Exploring the impact of pragmatic phenomena on irony detection in tweets: A multilingual corpus study. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 262–272, Valencia, Spain. Association for Computational Linguistics. find
20. Jihen Karoui, Farah Benamara Zitoune, Véronique Moriceau, Nathalie Aussenac-Gilles, and Lamia Hadrich Belguith. 2015. Towards a contextual pragmatic model to detect irony in tweets. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 644–650, Beijing, China. Association for Computational Linguistics. find
21. Zohar Livnat. 2011. Quantity, truthfulness and ironic effect. Language Sciences, 33(2):305–315. find
22. Shaltiel Shmidman, Avi Shmidman, and Moshe Koppel. 2023. Dictabert: A state-of-the-art bert suite for modern hebrew. Preprint, arXiv:2308.16687. ↗ our card
23. Dan Sperber and Deirdre Wilson. 1981. Irony and the use-mention distinction. In P. Cole, editor, Radical pragmatics, pages 295–318. Academic Press, New York. find
24. Dan Sperber and Deirdre Wilson. 1986. Relevance: Communication and Cognition. Blackwell, Oxford. find
25. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. We usually don’t like going to the dentist: Using common sense to detect irony on twitter. Computational Linguistics, 44(4):793–832. find our card
26. Tony Veale. 2021. Your Wit Is My Command: Building AIs with a Sense of Humor. The MIT Press. find
27. Tony Veale. 2023. Great Expectations and EPIC Fails: A Computational Perspective on Irony and Sarcasm, page 216–234. Cambridge Handbooks in Psychology. Cambridge University Press. find
28. Byron C. Wallace. 2013. Computational irony: A survey and new perspectives. Artificial Intelligence Review, 43(4):467–483. find
29. Byron C. Wallace, Do Kook Choe, and Eugene Charniak. 2015. Sparse, contextually informed models for irony detection: Exploiting user communities, entities and sentiment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1035–1044, Beijing, China. Association for Computational Linguistics. find
30. Elda Weizman. 1984. Some register characteristics of journalistic language: Are they universals? Applied Linguistics, 5(1):39–50. find
31. Alan Partington. 2007. Irony and reversal of evaluation. Journal of Pragmatics, 39(9):1547–1569. find
32. Elda Weizman. 2001. Addresser, addressee and target. In Negotiation and Power in Dialogic Interaction, pages 125–137. John Benjamins Publishing Company, Amsterdam. find
33. Alan Partington. 2011. Phrasal irony: Its form, function and exploitation. Journal of Pragmatics, 43(6):1786–1800. find
34. Elda Weizman. 2008. Positioning in Media Dialogue. Dialogue Studies. John Benjamins Publishing, Amsterdam, Netherlands. find
35. Alan Partington, Alison Duguid, and Charlotte Taylor. 2013. Patterns and Meanings in Discourse: Theory and practice in corpus-assisted discourse studies (CADS). John Benjamins Publishing Company. find
36. Elda Weizman. 2011. Conveying indirect reservations through discursive redundancy. Language Sciences, 33(2):295–304. find
37. Stefano Predelli. 2003. Scare quotes and their relation to other semantic issues. Linguistics and Philosophy, 26(1):1–28. find
38. Elda Weizman. 2020. The discursive pattern ‘claim+ indirect quotation in quotation marks’: Strategic uses in french and hebrew online journalism. Journal of Pragmatics, 157:131–141. find
39. Marcel Schlechtweg and Holden Härtl. 2023. Quotation marks and the processing of irony in english: evidence from a reading time study. Linguistics, 61(2):355–390. find
40. Elda Weizman. 2022. Explicitating irony in a cross-cultural perspective: Discursive practices in online op-eds in french and in hebrew. Contrastive Pragmatics, 4(3):437–465. find
41. Elda Weizman and Marcelo Dascal. 1991. On clues and cues: Strategies of text-understanding. Journal of Literary Semantics, 20(1):18–30. find
42. Deirdre Wilson. 2012. Metarepresentation in linguistic communication, page 230–258. Cambridge University Press. find
43. Deirdre Wilson and Dan Sperber. 1992. On verbal irony. Lingua, 87(1–2):53–76. find
44. Deirdre Wilson and Dan Sperber. 2012. Meaning and Relevance. Cambridge University Press. find
45. Jan Wiślicki. 2023. Scare quotes as deontic modals. Linguistics, 61(2):417–457. find
46. Francisco Yus. 2023. Inferring irony online. In The Cambridge Handbook of Irony and Thought, pages 160–180. Cambridge University Press. find
47. Michele Zappavigna. 2022. Social media quotation practices and ambient affiliation: Weaponising ironic quotation for humorous ridicule in political discourse. Journal of Pragmatics, 191:98–112. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/prag-2025-shmidman-irony-detection-hebrew-documents-novel/