Annotation Matters: Resolving Cross-Corpus Performance Drops in Hebrew Offensive Language Detection

What it is about

Hebrew offensive-language detectors trained on one forum corpus lose much of their accuracy on two external corpora, a Facebook set and a Twitter set. The authors ask whether this is domain shift or annotation shift. They relabel both external sets with the same taxonomy-based prompt (intent, target, tone, impact) and keep only items on which two LLMs agree, dropping implicit cases. The labels change a lot: 18.6% flip on the Facebook set, mostly from non-offensive to offensive, and 56% on the Twitter set, where 60% of flips go from non-offensive to offensive. The flips cluster around political humiliation and reporting of violence, and a manual audit mostly sides with the LLMs. With the same model weights, accuracy rises from 78.4% to 91.1% on Facebook and from 45.4% to 85.2% on Twitter once the new labels are used. The authors conclude that most cross-dataset failure is annotation mismatch.

Ideas

• I1. Operationalise offensiveness in Mova as four checkable features from the tree: intent (a verb or predicate aimed at a person), target (a person or group referred to by a pronoun or noun), tone (a lexicon hit), impact (a second rule). Each rule records the reason it fired, so a decision can be explained.
• I2. Before any external Hebrew offence set is used as a test in Mova, measure the flip rate between its labels and a second labelling, by category (direct profanity, political humiliation, violence reporting). This is a measurement only; the paper's labels are non-commercial and are never built into anything.
• I3. Keep a small hard-case file of sarcasm and ironic emoji, which the paper shows both LLMs and humans disagree on, as a known-failure test for the rules, and leave implicit cases out of the first gold set as the paper does.

For Mova

I1 is a rule layer in prag, with its frames explained per decision. I2 is a measurement in the prag tests, never built into the release. I3 is a test file for 9. Pragmatics and politeness. Licence note: the source is NC, so only this summary is kept in the repository.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/3fss6oc5mono

@inproceedings{hefetz2026annotation,
  title = {{Annotation Matters: Resolving Cross-Corpus Performance Drops in Hebrew Offensive Language Detection}},
  author = {Hefetz, Gili Berger and Shrem, Yossef Haim and Vanetik, Natalia and Liebeskind, Chaya},
  booktitle = {Proceedings of Computational Affective Science (CAS) @ LREC 2026},
  year = {2026},
  doi = {10.63317/3fss6oc5mono},
  url = {https://doi.org/10.63317/3fss6oc5mono}
}

References

14 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gili Berger Hefetz, Yossef Haim Shrem, and Chaya Liebeskind. 2026. Hebrew offensive language detection via prompt-guided llm annotation and transformer fine-tuning. In Proceedings of the IEEE Conference on Computer Systems and Applications, Jerusalem, Israel. To appear. find
2. Avihay Chriqui and Inbal Yahav. 2022. Hebert and hebemo: A hebrew bert model and a tool for polarity analysis and emotion recognition. IN-FORMS Journal on Data Science, 1(1):81–95. find
3. Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. Acm Computing Surveys (Csur), 51(4):1–30. find
4. Eylon Gueta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2022. Large pre-trained models with extra-large vocabularies: A contrastive analysis of hebrew bert models and a new one to outperform them all. arXiv preprint arXiv:2211.15199. ↗ our card
5. Nagham Hamad, Mustafa Jarrar, Mohammad Khalilia, and Nadim Nashif. 2023. Offensive hebrew corpus and detection using bert. In 2023 20th ACS/IEEE International Conference on Computer Systems and Applications (AICCSA), pages 1–8. IEEE. find our card
6. Chaya Liebeskind, Ali Afawi, Marina Litvak, and Natalia Vanetik. 2024. Classifying offensive language in arabic: a novel taxonomy and dataset. Lodz Papers in Pragmatics, 20(2):433–462. find
7. Chaya Liebeskind and Shmuel Liebeskind. 2018. Identifying abusive comments in hebrew facebook. In 2018 IEEE International conference on the science of electrical engineering in Israel (ICSEE), pages 1–5. IEEE. find
8. Chaya Liebeskind, Natalia Vanetik, and Marina Litvak. 2023. Hebrew offensive language taxonomy and dataset. Lodz Papers in Pragmatics, 19(2):325–351. find
9. Marina Litvak, Natalia Vanetik, Chaya Liebeskind, Omar Hmdia, and Rizek Abu Madeghem. 2022. Offensive language detection in Hebrew: can other languages help? In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3715–3723, Marseille, France. European Language Resources Association. find our card
10. Marina Litvak, Natalia Vanetik, Yaser Nimer, Abdulrhman Skout, and Israel Beer-Sheba. 2021. Offensive language detection in semitic languages. In Multimodal hate speech workshop, volume 2020, pages 7–12. find
11. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert: A hebrew large pre-trained language model to start-off your hebrew nlp application with. arXiv preprint arXiv:2104.04052. ↗ our card
12. Vitaly Shalumov and Harel Haskey. 2023. Hero: Roberta and longformer hebrew language models. arXiv preprint arXiv:2304.11077. ↗ our card
13. Shaltiel Shmidman, Avi Shmidman, and Moshe Koppel. 2023. Dictabert: A state-of-the-art bert suite for modern hebrew. arXiv preprint arXiv:2308.16687. ↗ our card
14. Zeerak Talat. 2016. Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In Proceedings of the first workshop on NLP and computational social science, pages 138–142. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2026-hefetz-annotation-matters-resolving-cross-corpus/