Mova0.5 pre-release⌕
☰
home / docs / papers / Modern Hebrew / HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition

HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition

What it is about

The authors pre-train HeBERT, a BERT model for modern Hebrew, on Hebrew Wikipedia and OSCAR, and test it on fill-in-the-blank, out-of-vocabulary, NER, POS and polarity tasks. They find that larger models do better on tasks about single terms, while a smaller model captures the sentiment of a whole sentence better. HebEMO is a document-level tool built on HeBERT for polarity and emotion. It is trained on Israeli news comments about Covid-19 from 2020, labelled with an active-learning loop. It reaches weighted F1 0.96 for polarity and about 0.8 to 0.97 for most emotions, but only 0.41 for surprise.

Ideas

For Mova

I1 and I3 apply to the Hebrew pragmatics plan in 9. Pragmatics and politeness and to the rules in prag. I2 is a design rule for 12. Mova and Modern Hebrew: where it stands, what it would need. The model itself is neural, so only the task lists and measures are reused.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1287/ijds.2022.0016 · arXiv 2102.01909

@misc{chriqui2021hebert,
  title = {{HeBERT \& HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition}},
  author = {Chriqui, Avihay and Yahav, Inbal},
  year = {2021},
  doi = {10.1287/ijds.2022.0016},
  eprint = {2102.01909},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1287/ijds.2022.0016}
}

References

74 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Acheampong FA, Wenyu C, Nunoo-Mensah H (2020) Text-based emotion detection: Advances, challenges, and opportunities. Engineering Reports e12189. find
  2. Adamopoulos P, Ghose A, Todri V (2018) The impact of user personality traits on word of mouth: Text-mining social media platforms. Information Systems Research 29(3):612–640. find
  3. Ahmad Z, Jindal R, Ekbal A, Bhattachharyya P (2020) Borrow from rich cousin: transfer learning for emotion detection using cross lingual embedding. Expert Systems with Applications 139:112851. find
  4. Ahorsu DK, Lin CY, Imani V, Saffari M, Griffiths MD, Pakpour AH (2020) The fear of covid-19 scale: development and initial validation. International journal of mental health and addiction . find
  5. Amram A, David AB, Tsarfaty R (2018) Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from modern hebrew. Proceedings of the 27th International Conference on Computational Linguistics, 2242–2252. find our card
  6. Antoun W, Baly F, Hajj H (2020) Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104 . ↗
  7. Argaman O (2010) Linguistic markers and emotional intensity. Journal of psycholinguistic research 39(2):89– 99. find
  8. Bareket D, Tsarfaty R (2020) Neural modeling for named entities and morphology (nemoˆ 2). arXiv preprint arXiv:2007.15620 . ↗
  9. Belinkov Y, Durrani N, Dalvi F, Sajjad H, Glass J (2017) What do neural machine translation models learn about morphology? arXiv preprint arXiv:1704.03471 . ↗
  10. Bellstam G, Bhagat S, Cookson JA (2020) A text-based analysis of corporate innovation. Management Science . find
  11. Bojanowski P, Joulin A, Mikolov T (2015) Alternative structures for character-level rnns. arXiv preprint arXiv:1511.06303 . ↗
  12. Chatterjee A, Narahari KN, Joshi M, Agrawal P (2019) Semeval-2019 task 3: Emocontext contextual emotion detection in text. Proceedings of the 13th International Workshop on Semantic Evaluation, 39–48. find
  13. Chitturi R, Raghunathan R, Mahajan V (2007) Form versus function: How the intensities of specific emotions evoked in functional versus hedonic trade-offs mediate product preferences. Journal of marketing research 44(4):702–714. find
  14. Desmet B, Hoste V (2013) Emotion detection in suicide notes. Expert Systems with Applications 40(16):6351– 6358. find
  15. Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 . ↗
  16. Dong L, Wei F, Tan C, Tang D, Zhou M, Xu K (2014) Adaptive recursive neural network for target-dependent twitter sentiment classification. Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: Short papers), 49–54. find
  17. Ekman P (1999) Basic emotions. Handbook of cognition and emotion 98(45-60):16. find
  18. Fattah K, Fierke KM (2009) A clash of emotions: The politics of humiliation and political violence in the middle east. European journal of international relations 15(1):67–93. find
  19. Fedus W, Goodfellow I, Dai AM (2018) Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736 . ↗
  20. Ghanbari-Adivi F, Mosleh M (2019) Text emotion detection in social networks using a novel ensemble classifier based on parzen tree estimator (tpe). Neural Computing and Applications 31(12):8971–8983. find
  21. Hemmatian F, Sohrabi MK (2019) A survey on classification techniques for opinion mining and sentiment analysis. Artificial Intelligence Review 1–51. find
  22. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural computation 9(8):1735–1780. find
  23. Jawahar G, Sagot B, Seddah D (2019) What does bert learn about the structure of language? find
  24. Joulin A, Grave E, Bojanowski P, Douze M, Jégou H, Mikolov T (2016) Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651 . ↗
  25. Kim Y (2014) Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 . ↗
  26. Kim-Prieto C, Diener E (2009) Religion as a source of variation in the experience of positive and negative emotions. The Journal of Positive Psychology 4(6):447–460. find
  27. Klein S, Tsarfaty R (2020) Getting the## life out of living: How adequate are word-pieces for modelling complex morphology? Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, 204–209. find
  28. Kövecses Z (2003) Metaphor and emotion: Language, culture, and body in human feeling (Cambridge University Press). find
  29. Kratzwald B, Ilić S, Kraus M, Feuerriegel S, Prendinger H (2018) Deep learning for affective computing: Text-based emotion recognition in decision support. Decision Support Systems 115:24–35. find
  30. Krippendorff K (1970) Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement 30(1):61–70. find
  31. Levine Y, Lenz B, Lieber O, Abend O, Leyton-Brown K, Tennenholtz M, Shoham Y (2020) Pmi-masking: Principled masking of correlated spans. arXiv preprint arXiv:2010.01825 . ↗
  32. Li S, Ju S, Zhou G, Lin X (2012) Active learning for imbalanced sentiment classification. Proceedings of the 2012 Joint conference on empirical methods in natural language processing and computational natural language learning, 139–148. find
  33. Liu B (2012) Sentiment analysis and opinion mining. Synthesis lectures on human language technologies 5(1):1–167. find
  34. Liu B, Zhang L (2012) A survey of opinion mining and sentiment analysis. Mining text data, 415–463 (Springer). find
  35. Liu R, Shi Y, Ji C, Jia M (2019) A survey of sentiment analysis based on transfer learning. IEEE Access 7:85401–85412. find
  36. Medhat W, Hassan A, Korashy H (2014) Sentiment analysis algorithms and applications: A survey. Ain Shams engineering journal 5(4):1093–1113. find
  37. Mohammad S, Bravo-Marquez F, Salameh M, Kiritchenko S (2018) Semeval-2018 task 1: Affect in tweets. Proceedings of the 12th international workshop on semantic evaluation, 1–17. find
  38. Mohammad SM, Turney PD (2013) Crowdsourcing a word-emotion association lexicon 29(3):436–465. find
  39. Mordecai NB, Elhadad M (2005) Hebrew named entity recognition. MONEY 81(83.93):82–49. find
  40. More A, Seker A, Basmova V, Tsarfaty R (2019) Joint transition-based models for morpho-syntactic parsing: Parsing strategies for mrls and a case study from modern hebrew. Transactions of the Association for Computational Linguistics 7:33–48. find our card
  41. Ortiz Suárez PJ, Romary L, Sagot B (2020) A monolingual approach to contextualized word embeddings for mid-resource languages. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 1703–1714 (Online: Association for Computational Linguistics), URL https://www.aclweb.org/anthology/2020.acl-main.156. ↗
  42. Ortony A, Clore GL, Foss MA (1987) The referential structure of the affective lexicon. Cognitive science 11(3):341–364. find
  43. Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22(10):1345–1359. find
  44. Patwa P, Aguilar G, Kar S, Pandey S, PYKL S, Gambäck B, Chakraborty T, Solorio T, Das A (2020) Semeval-2020 task 9: Overview of sentiment analysis of code-mixed tweets. arXiv e-prints arXiv–2008. find
  45. Pedrosa AL, Bitencourt L, Fróes ACF, Cazumbá MLB, Campos RGB, de Brito SBCS, e Silva ACS (2020) Emotional, behavioral, and psychological impact of the covid-19 pandemic. Frontiers in psychology 11. find
  46. Pennebaker JW, Francis ME, Booth RJ (2001) Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates 71(2001):2001. find
  47. Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. arXiv preprint arXiv:1802.05365 . ↗
  48. Pfefferbaum B, North CS (2020) Mental health and the covid-19 pandemic. New England Journal of Medicine . find
  49. Plutchik R (1980) A general psychoevolutionary theory of emotion. Theories of emotion, 3–33 (Elsevier). find
  50. Pota M, Marulli F, Esposito M, De Pietro G, Fujita H (2019) Multilingual pos tagging by a composite deep architecture based on character-level features and on-the-fly enriched word embeddings. Knowledge-Based Systems 164:309–323. find
  51. Radford A, Narasimhan K, Salimans T, Sutskever I (2018) Improving language understanding by generative pre-training. find
  52. Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9. find
  53. Rosaldo MZ, Shweder RA, LeVine RA (1984) Culture theory: essays on mind, self, and emotion. find
  54. Schuster M, Nakajima K (2012) Japanese and korean voice search. 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5149–5152 (IEEE). find
  55. Shapira N, Lazarus G, Goldberg Y, Gilboa-Schechtman E, Tuval-Mashiach R, Juravski D, Atzil-Slonim D (2020) Using computerized text analysis to examine associations between linguistic features and clients’ distress during psychotherapy. Journal of counseling psychology . find
  56. Sima’an K, Itai A, Winter Y, Altman A, Nativ N (2001) Building a tree-bank of modern hebrew text. Traitement Automatique des Langues 42(2):247–380. find
  57. Steiner T (2016) President rivlin’s” four tribes” initiative: The foreign policy implications of a democratic & inclusive process to address israel’s socio-demographic transformation . find
  58. Straka M, Hajic J, Straková J (2016) Udpipe: trainable pipeline for processing conll-u files performing tokenization, morphological analysis, pos tagging and parsing. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), 4290–4297. find our card
  59. Tay Y, Dehghani M, Bahri D, Metzler D (2020) Efficient transformers: A survey. arXiv preprint arXiv:2009.06732 . ↗
  60. Tsarfaty R, Bareket D, Klein S, Seker A (2020) From spmrl to nmrl: What did we learn (and unlearn) in a decade of parsing morphologically-rich languages (mrls)? arXiv preprint arXiv:2005.01330 . ↗
  61. Tsarfaty R, Seddah D, Goldberg Y, Kübler S, Versley Y, Candito M, Foster J, Rehbein I, Tounsi L (2010) Statistical parsing of morphologically rich languages (spmrl) what, how and whither. Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages, 1–12. find
  62. Ullah R, Amblee N, Kim W, Lee H (2016) From valence to emotions: Exploring the distribution of emotions in online product reviews. Decision Support Systems 81:41–53. find
  63. Vania C, Grivas A, Lopez A (2018) What do character-level models learn about morphology? the case of dependency parsing. arXiv preprint arXiv:1808.09180 . ↗
  64. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. Advances in neural information processing systems, 5998–6008. find
  65. Wierzbicka A (1994) Emotion, language, and cultural scripts. . find
  66. Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, Davison J, Shleifer S, von Platen P, Ma C, Jernite Y, Plu J, Xu C, Scao TL, Gugger S, Drame M, Lhoest Q, Rush AM (2020) Transformers: State-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 38– 45 (Online: Association for Computational Linguistics), URL https://www.aclweb.org/anthology/2020.emnlp-demos.6. ↗
  67. Wu Y, Schuster M, Chen Z, Le QV, Norouzi M, Macherey W, Krikun M, Cao Y, Gao Q, Macherey K, et al. (2016) Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 . ↗
  68. Yadav A, Vishwakarma DK (2020) Sentiment analysis using deep learning architectures: a review. Artificial Intelligence Review 53(6):4335–4385. find
  69. Yin W, Kann K, Yu M, Schütze H (2017) Comparative study of cnn and rnn for natural language processing. arXiv preprint arXiv:1702.01923 . ↗
  70. Yue L, Chen W, Li X, Zuo W, Yin M (2019) A survey of sentiment analysis in social media. Knowledge and Information Systems 1–47. find
  71. Zampieri M, Malmasi S, Nakov P, Rosenthal S, Farra N, Kumar R (2019) Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). arXiv preprint arXiv:1903.08983 . ↗
  72. Zhang L, Wang S, Liu B (2018) Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8(4):e1253. find
  73. Zhong P, Wang D, Miao C (2019) Knowledge-enriched transformer for emotion detection in textual conversations. arXiv preprint arXiv:1909.10681 . ↗
  74. Zhou D, Wu S, Wang Q, Xie J, Tu Z, Li M (2020) Emotion classification by jointly learning to lexiconize and classify. Proceedings of the 28th International Conference on Computational Linguistics, 3235–3245. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2021-chriqui-hebert-hebemo-hebrew-bert-model/