Mova0.5 pre-release⌕
☰
home / docs / papers / COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs

COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs

Summary

The paper argues that hand-built commonsense knowledge graphs can never cover everything an NLP system will meet. It therefore proposes judging a knowledge graph by how well neural "knowledge models" trained on it can generate new commonsense facts. With this goal it introduces ATOMIC 2020, a general-purpose commonsense graph of if-then knowledge about social interactions, physical objects and events, built to contain knowledge that pretrained language models do not readily have. The authors compare it with other major commonsense graphs in a large pairwise study of coverage and accuracy. They show that knowledge models (COMET) trained on ATOMIC 2020 generate more accurate knowledge for unseen entities and events than models trained on other resources. In human evaluation, a BART-based COMET model trained on ATOMIC 2020 beats few-shot GPT-3 by about 12 absolute points while being over 430 times smaller.

How Mova uses it

Effectiveness in Mova

Measured (architecture status) on FairytaleQA feeling questions, comparing the story reader before and after adding the ATOMIC xReact table: test 0.130 → 0.158, validation 0.105 → 0.118 (inferred answers 28 → 71). Overall reader score 0.212 → 0.215, identical on a repeat run (deterministic), with no change on fables. Adding oReact: validation 0.118 → 0.127, test unchanged at 0.158. The ObjectUse/CapableOf tables pass their negative-control test but have no downstream measurement yet.

👨‍🔬💥

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2010.05953

@misc{hwang2020comet,
  title = {{COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs}},
  author = {Hwang, Jena D. and Bhagavatula, Chandra and Bras, Ronan Le and Da, Jeff and Sakaguchi, Keisuke and Bosselut, Antoine and Choi, Yejin},
  year = {2020},
  eprint = {2010.05953},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.05953}
}

References

42 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ammanabrolu, P.; Cheung, W.; Broniec, W.; and Riedl, M. 2020. Automated Storytelling via Causal, Commonsense Plot Ordering. ArXiv abs/2009.00829. — [arXiv:2009.00829](https://arxiv.org/abs/2009.00829) find
  2. Auer, S.; Bizer, C.; Kobilarov, G.; Lehmann, J.; Cyganiak, R.; and Ives, Z. G. 2007. DBpedia: A Nucleus for a Web of Open Data. In ISWC/ASWC. find
  3. Bisk, Y.; Zellers, R.; Le Bras, R.; Gao, J.; and Choi, Y. 2020. PIQA: Reasoning about Physical Commonsense in Natural Language. In AAAI. find
  4. Bosselut, A.; Rashkin, H.; Sap, M.; Malaviya, C.; Çelikyilmaz, A.; and Choi, Y. 2019. COMET: Commonsense Transformers for Automatic Knowledge Graph Construction. In ACL. find
  5. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krüger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B… find
  6. Chakrabarty, T.; Ghosh, D.; Muresan, S.; and Peng, N. 2020. Rˆ3: Reverse, Retrieve, and Rank for Sarcasm Generation with Commonsense Knowledge. In ACL. find
  7. Davis, E.; and Marcus, G. 2015. Commonsense Reasoning and Commonsense Knowledge in Artificial Intelligence. Commun. ACM 58(9): 92–103. find
  8. Davison, J.; Feldman, J.; and Rush, A. 2019. Commonsense Knowledge Mining from Pretrained Models. In EMNLP-IJCNLP, 1173–1178. Hong Kong, China. find
  9. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT. find
  10. Feng, Y.; Chen, X.; Lin, B. Y.; Wang, P.; Yan, J.; and Ren, X. 2020. Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering. ArXiv abs/2005.00646. find
  11. Fleiss, J. L. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76(5): 378. find
  12. Gordon, J.; and Van Durme, B. 2013. Reporting bias and knowledge acquisition. In AKBC ’13. ACM. find
  13. Herskovits, A. 2009. Language and Spatial Cognition: An Interdisciplinary Study of the Prepositions in English. find
  14. Kearns, W. R.; Kaura, N.; Divina, M.; Vo, C. V.; Si, D.; Ward, T. M.; and Yuwen, W. 2020. A Wizard-of-Oz Interface and Persona-based Methodology for Collecting Health Counseling Dialog. Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems . find
  15. Landau, B.; and Jackendoff, R. 1991. Spatial language and spatial cognition. Bridges Between Psychology and Linguistics: A Swarthmore Festschrift for Lila Gleitman 145. find
  16. Lascarides, A.; and Asher, N. 1991. Discourse relations and defeasible knowledge. In ACL. find
  17. Lenat, D. B. 1995. CYC: A large-scale investment in knowledge infrastructure. Communications of the ACM . find
  18. Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL. find
  19. Li, X.; Taheri, A.; Tu, L.; and Gimpel, K. 2016. Commonsense Knowledge Base Completion. In ACL. find
  20. Lin, B. Y.; Chen, X.; Chen, J.; and Ren, X. 2019. KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning. In EMNLP/IJCNLP. find
  21. Lin, C.-Y. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. find
  22. Liu, Y.; Yang, T.; You, Z.; Fan, W.; and Yu, P. S. 2020. Commonsense Evidence Generation and Injection in Reading Comprehension. In SIGDIAL. find
  23. Miller, G. A. 1995. WordNet: A Lexical Database for English. Communications of the ACM . find our card
  24. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. BLEU: a method for automatic evaluation of machine translation. In ACL. find our card
  25. Petroni, F.; Rocktäschel, T.; Lewis, P.; Bakhtin, A.; Wu, Y.; Miller, A. H.; and Riedel, S. 2019. Language Models as Knowledge Bases? In EMNLP. find
  26. Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018. Improving language understanding by generative pre-training URL https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language understanding paper.pdf. ↗
  27. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019. Language models are unsupervised multitask learners. OpenAI blog 1(8): 9. find
  28. Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2019. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. ArXiv abs/1910.10683. find
  29. Roberts, A.; Raffel, C.; and Shazeer, N. 2020. How Much Knowledge Can You Pack Into the Parameters of a Language Model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 5418–5426. find
  30. Romero, J.; Razniewski, S.; Pal, K.; Z. Pan, J.; Sakhadeo, A.; and Weikum, G. 2019. Commonsense properties from query logs and question answering forums. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 1411–1420. URL https://arxiv.org/pdf/1905.10989.p… ↗
  31. Sap, M.; Le Bras, R.; Allaway, E.; Bhagavatula, C.; Lourie, N.; Rashkin, H.; Roof, B.; Smith, N. A.; and Choi, Y. 2019. ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning. In AAAI. find
  32. Shwartz, V.; West, P.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2020. Unsupervised Commonsense Question Answering with Self-Talk. ArXiv abs/2004.05483. find
  33. Speer, R.; Chin, J.; and Havasi, C. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI. find
  34. Talmy, L. 1988. Force Dynamics in Language and Cognition. Cognitive Science 12: 49–100. find
  35. Tamborrino, A.; Pellicanò, N.; Pannier, B.; Voitot, P.; and Naudin, L. 2020. Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning. In ACL. find
  36. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. find
  37. Vedantam, R.; Lawrence Zitnick, C.; and Parikh, D. 2015. Cider: Consensus-based image description evaluation. In CVPR, 4566–4575. find
  38. Wang, A.; Pruksachatkun, Y.; Nangia, N.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2019. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. ArXiv abs/1905.00537. find
  39. Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Scao, T. L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. M. 2019. HuggingFace’s Transformers: State… find
  40. Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? In ACL. find
  41. Zhang, H.; Khashabi, D.; Song, Y.; and Roth, D. 2020a. TransOMCS: From Linguistic Graphs to Commonsense Knowledge. In IJCAI. find
  42. Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K.; and Artzi, Y. 2020b. BERTScore: Evaluating Text Generation with BERT. ArXiv abs/1904.09675. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/hwang-2020-comet-atomic-2020-symbolic-neural/