The Truth, The Whole Truth, and Nothing but the Truth: A New Benchmark Dataset for Hebrew Text Credibility Assessment

What it is about

The paper introduces HeTrue, 1,021 statements by Israeli politicians and public figures, collected and graded by the journalists of an IFCN fact-checking outlet. Grades run over a five-step truth scale, plus "Unverifiable" and "Other". Each statement is stored with its surrounding context (51 tokens on average, against 25 for the statement) and with metadata such as party, media source and field. The authors compare hand-crafted linguistic features, an RNN with fastText, and AlephBERT models, including a context model and an attention model over search snippets as evidence. Under nested five-fold cross-validation with bootstrap, the best model (statement, context and evidence) reaches macro-F1 48.3 on the full five-class set and 72.8 on the True/False subset, against a majority baseline of 11.0 and 43.0. Context helps in every setting, metadata alone reaches 21.1, and written statements are easier to classify than transcribed speech.

Ideas

• I1. Turn the Hebrew deception cues from the earlier literature into explicit counts over the UD trees: third-person against first-person verbs (person feature), past-tense verbs (Tense=Past), share of dependent clauses (advcl, ccomp, acl), negation (Polarity=Neg), and generalising words. Test each count alone against the labels under nested cross-validation before combining them.
• I2. Context as a separate input: test whether a rule-based classifier improves when the previous sentence's frame in world is added, as the paper shows for the neural models.
• I3. Reuse the design: a written-only cut to separate transcription artefacts from content, a majority baseline, and bootstrap intervals. The authors also warn against using claimant profiles as features; Mova should keep speaker identity out of any prag judgement.

For Mova

I1's verb counts go into 05. The verb: system, binyanim, tense and use and its clause counts into 7. Syntax. I2 is a test for world plus prag. I3 is a measurement protocol, not a product: credibility is not a truth detector in Mova.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.251

@inproceedings{hagag2023truth,
  title = {{The Truth, The Whole Truth, and Nothing but the Truth: A New Benchmark Dataset for Hebrew Text Credibility Assessment}},
  author = {Hagag, Ben and Tsarfaty, Reut},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.251},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.251}
}

References

51 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sadia Afroz, Michael Brennan, and Rachel Greenstadt. 2012. Detecting hoaxes, frauds, and deception in writing style online. In 2012 IEEE Symposium on Security and Privacy, pages 461–475. IEEE. find
2. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623–2631. find
3. Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. Multifc: A real-world multi-domain dataset for evidence-based fact checking of claims. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4685–4697. find
4. Ramy Baly, Mitra Mohtarami, and James Glass. 2018. Integrating stance detection and fact checking in a unified corpus. In Proceedings of NAACL-HLT, pages 21–27. find
5. James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. Advances in neural information processing systems, 24. find
6. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5:135–146. ↗
7. Jianlin Cheng, Zheng Wang, and Gianluca Pollastri. 2008. A neural network approach to ordinal regression. In 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence), pages 1279–1284. IEEE. find
8. Nadia K Conroy, Victoria L Rubin, and Yimin Chen. 2015. Automatic deception detection: Methods for finding fake news. Proceedings of the association for information science and technology, 52(1):1–4. find
9. João Victor de Souza, Jorão Gomes Jr, Fernando Marques de Souza Filho, Alessandreia Marta de Oliveira Julio, Jairo Francisco de Souza, et al. 2020. A systematic mapping on automatic classification of fake news in social media. Social Network Analysis and Mining, 10(1):1–21. find
10. R Dilmon. 2004. Linguistic differences between lie and truth in spoken hebrew. Doctoral diss., Bar Ilan University, Ramat Gan, Israel.[in Hebrew]. find
11. Rakefet Dilmon. 2007. Fiction or fact? comparing true and untrue anecdotes. Balšanwt iybriyt, 59. find
12. Rakefet Dilmon. 2013. False speech, linguistic aspects in hebrew. The Encyclopedia of Hebrew Language and Linguistics, Leiden, Brill, Boston and Tokyo, pages 542–546. find
13. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 1383–1392. find
14. Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion, 6(3-4):169–200. find our card
15. JR Flesch. 1965. Flesch-kincaid readability formula. find
16. Bruce Fraser. 1991. Questions of witness credibility. Harvard Law School. find
17. Bilal Ghanem, Simone Paolo Ponzetto, Paolo Rosso, and Francisco Rangel. 2021. Fakeflow: Fake news detection by modeling the flow of affective information. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 679–689. find
18. Anastasia Giachanou, Paolo Rosso, and Fabio Crestani. 2019. Leveraging emotional signals for credibility detection. In Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pages 877–880. find
19. Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10:178–206. find
20. Ashim Gupta and Vivek Srikumar. 2021. X-fact: A new benchmark dataset for multilingual fact checking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 675–682. find
21. Yaakov HaCohen-Kerner, Rakefet Dilmon, Shimon Friedlich, and Daniel Nisim Cohen. 2015. Distinguishing between true and false stories using various linguistic features. In Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation: Posters, pages 176–186. find
22. Jeffrey T Hancock, Lauren E Curry, Saurabh Goorha, and Michael Woodworth. 2007. On lying and being lied to: A linguistic analysis of deception in computer-mediated communication. Discourse Processes, 45(1):1–23. find
23. P Sol Hart and Erik C Nisbet. 2012. Boomerang effects in science communication: How motivated reasoning and identity cues amplify opinion polarization about climate mitigation policies. Communication research, 39(6):701–723. find
24. Naeemul Hassan, Bill Adair, James T Hamilton, Chengkai Li, Mark Tremayne, Jun Yang, and Cong Yu. 2015. The quest to automate fact-checking. In Proceedings of the 2015 Computation+ Journalism Symposium. find
25. Naeemul Hassan, Gensheng Zhang, Fatma Arslan, Josue Caraballo, Damian Jimenez, Siddhant Gawsane, Shohedul Hasan, Minumol Joseph, Aaditya Kulkarni, Anil Kumar Nayak, et al. 2017. Claimbuster: the first-ever end-to-end fact-checking system. Proceedings of the VLDB Endowment, 10(12):1945–1948. of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421. find
26. Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. find
27. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
28. Tanushree Mitra and Eric Gilbert. 2015. Credbank: A large-scale social media corpus with associated credibility annotations. In Ninth international AAAI conference on web and social media. find
29. Amir More, Amit Seker, Victoria Basmova, and Reut Tsarfaty. 2019. Joint transition-based models for morpho-syntactic parsing: Parsing strategies for MRLs and a case study from modern Hebrew. Transactions of the Association for Computational Linguistics, 7:33–48. find our card
30. Jeff Z Pan, Siyana Pavlova, Chenxi Li, Ningxi Li, Yangmei Li, and Jinshuo Liu. 2018. Content based fake news detection using knowledge graphs. In International semantic web conference, pages 669–683. Springer. find
31. James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates, 71(2001):2001. find
32. Kashyap Popat, Subhabrata Mukherjee, Andrew Yates, and Gerhard Weikum. 2018. Declare: Debunking fake news and false claims using evidence-aware deep learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 22–32. find
33. Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. 2011. Sequential model-based optimization for general algorithm configuration. In International conference on learning and intelligent optimization, pages 507–523. Springer. find
34. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 2931–2937. find
35. Stephan Lewandowsky, Werner GK Stritzke, Klaus Oberauer, and Michael Morales. 2005. Memory for fact, fiction, and misinformation: The iraq war 2003. Psychological Science, 16(3):190–195. find
36. Julio CS Reis, André Correia, Fabrício Murai, Adriano Veloso, and Fabrício Benevenuto. 2019. Supervised learning for fake news detection. IEEE Intelligent Systems, 34(2):76–81. find
37. Yunfei Long, Qin Lu, Rong Xiang, Minglei Li, and Chu-Ren Huang. 2017. Fake news detection through multi-perspective speaker profiles. In Proceedings of the eighth international joint conference on natural language processing (volume 2: Short papers), pages 252–256. find
38. Arjun Roy, Kingshuk Basak, Asif Ekbal, and Pushpak Bhattacharyya. 2018. A deep ensemble framework for fake news detection and classification. arXiv preprint arXiv:1811.04670. ↗
39. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings Natali Ruchansky, Sungyong Seo, and Yan Liu. 2017. Csi: A hybrid deep model for fake news detection. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 797– 806. find
40. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert: A hebrew large pre-trained language model to start-off your hebrew nlp application with. arXiv preprint arXiv:2104.04052. ↗ our card
41. LLOYD SHAPLEY. 1953. A value for n-person games. contributions to the theory of games ii, kuhn, h., tucker, a. find
42. James Thorne and Andreas Vlachos. 2018. Automated fact checking: Task formulations, methods and future directions. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3346–3359. find
43. Reut Tsarfaty, Shoval Sadde, Stav Klein, and Amit Seker. 2019. What’s wrong with hebrew nlp? and how to make it right. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 259–264. find our card
44. UBOHO Victor. 2020. Robust semi-supervised learning for fake news detection. Ph.D. thesis, Prairie View A&M University Prairie View, TX, USA. find
45. Marco Viviani and Gabriella Pasi. 2017. Credibility in social media: opinions, news, and health information—a survey. Wiley interdisciplinary reviews: Data mining and knowledge discovery, 7(5):e1209. find
46. Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 workshop on language technologies and computational social science, pages 18–22. find
47. Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science, 359(6380):1146–1151. find
48. William Yang Wang. 2017. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426. find
49. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45. find
50. Lina Zhou, Judee K Burgoon, Jay F Nunamaker, and Doug Twitchell. 2004. Automating linguistics-based cues for detecting deception in text-based asynchronous computer-mediated communications. Group decision and negotiation, 13(1):81–106. find
51. Xinyi Zhou and Reza Zafarani. 2018. Fake news: A survey of research, detection methods, and opportunities. arXiv preprint arXiv:1812.00315. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/findings-2023-hagag-truth-whole-truth-nothing-but/