GLUCOSE: GeneraLized and COntextualized Story Explanations

What it is about

The question is which implicit causal inferences a reader makes about a story sentence, and whether a model can produce them. The data are about 670K crowdsourced entries on 4,881 five-sentence children's stories (from ROCStories). Each sentence is explained along ten dimensions: causes and enablers before it (events, emotions and drives, location and possession states, other attributes) and effects after it. Each entry is a story-specific statement and a general rule with indexed variables, such as "Someone A" or "Something A", written as antecedent, connective, consequent from a few typed slots. The method fine-tunes a T5 encoder-decoder (770M) and compares it with GPT-2 prompting, one-sided and full-rule language models, and a nearest-neighbour baseline; the scores are human ratings from 0 to 3 on a 500-item test set, and BLEU, which the paper reports correlates with the human scores (Spearman 0.89). Untuned GPT-2 is weak, and the fine-tuned encoder-decoder is competitive with humans on many dimensions, though weaker on general rules than on specific statements. Overlap with ConceptNet and ATOMIC is small: at most 1.9% and 7.8% for any dimension, by fuzzy verb matching.

Ideas

• I1. A causal schema for the tale readers. Store the rules of each story sentence in world, with the ten dimensions and the connectives (Causes, Enables, Motivates, Results in) as Rust enums rather than strings. Use them for the causal and prediction questions in world/src/qa.rs (FairytaleQA types), and make each answer cite the rule it used. Measure the share of causal questions answered with a rule trace on held-out tale sentences, and the count of questions where no rule fires.
• I2. Typed variables, with a negative control. Generalise a story entity to a typed variable through onto (a named child to "Someone A", a dish to "Something A (that is food)"), keeping the slot constraints of the paper's table: feels, wants or likes for motivations, and locative prepositions for location states. A generalised rule is kept only if it fires on at least one other story and does not fire on a random unrelated story. The second check is the red case.
• I3. An agreement filter for the rules. The paper keeps a test item only when three annotators agree (round-robin BLEU above 0.75). For Mova's own tale rules, keep a rule only when two independent sources (the teacher, or a rule built in onto) give the same consequent above a similarity threshold. Report the kept fraction and the error on held-out tales, and set the threshold by two-fold cv, not on the test.
• I4. A novelty measure. The paper's overlap test (a fuzzy match on the main verbs, then more than 50% lexical overlap) can count how many of Mova's causal rules are already covered by ConceptNet or ATOMIC. Use it only as a measurement. ConceptNet is share-alike, so nothing from it enters a release.

For Mova

The chapters are 36. Discourse relations and coherence (causal relations) and 35. Reference and coreference (the variables). The code is world/src/qa.rs, world/src/story.rs, world/src/knowledge.rs, the rules in onto/, and the questions in qagen/. Measures: rule-trace share and no-rule count on held-out tales (I1), the negative-control rule (I2), and the kept fraction with error (I3), each run twice before a claim. Licence: the paper is CC BY 4.0, but it does not state the licence of the GLUCOSE data beyond a download link, so check the dataset's own page before any use. The ROCStories source has its own terms. Until both are checked, the paper's test items are for measurement only and no GLUCOSE entry goes into a release.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.370 · arXiv 2009.07758

@inproceedings{mostafazadeh2020glucose,
  title = {{GLUCOSE: GeneraLized and COntextualized Story Explanations}},
  author = {Mostafazadeh, Nasrin and Kalyanpur, Aditya and Moon, Lori and Buchanan, David and Berkowitz, Lauren and Biran, Or and Chu-Carroll, Jennifer},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.370},
  eprint = {2009.07758},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.370}
}

References

39 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Niranjan Balasubramanian, Stephen Soderland, Mausam, and Oren Etzioni. 2013. Generating coherent event schemas at scale. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1721–1731, Seattle, Washington, USA. Association for Computational Linguistics. find
2. Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. 2019. On adversarial removal of hypothesis-only bias in natural language inference. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 256–262, Minneapolis, Minnesota. Association for Computational Linguistics. find
3. Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762–4779, Florence, Italy. Association for Computational Linguistics. find
4. Justin TA Busch, Aiyana K Willard, and Cristine H Legare. 2018. Explanation scaffolds causal learning and problem solving in childhood. In Active Learning from Infancy to Childhood, pages 113–127. Springer. find
5. Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of ACL-08: HLT, pages 789–797, Columbus, Ohio. Association for Computational Linguistics. find
6. Nathanael Chambers and Dan Jurafsky. 2009. Unsupervised learning of narrative schemas and their participants. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 602–610, Suntec, Singapore. Association for Computational Linguistics. find
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
8. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics. find
9. Albert Gatt and Emiel Krahmer. 2018. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. J. Artif. Int. Res., 61(1):65–170. find
10. Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, AKBC ’13, pages 25–30, New York, NY, USA. ACM. find
11. Ilaria Grazzani, Veronica Ornaghi, Elisabetta Conte, Alessandro Pepe, and Claudia Caprin. 2018. The relation between emotion understanding and theory of mind in children aged 3 to 8: The key role of language. Frontiers in Psychology, 9:724. find
12. Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1689–1701, Minneapolis, Minnesota. Association for Computational Linguistics. find
13. Walter Kintsch and Teun A Van Dijk. 1978. Toward a model of text comprehension and production. Psychological review, 85(5):363. find
14. Victor Kuperman, Hans Stadthagen-Gonzalez, and Marc Brysbaert. 2012. Age-of-acquisition ratings for 30,000 english words. Behavior Research Methods, 44(4):978–990. find
15. Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 605–612, Barcelona, Spain. find
16. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122–2132, Austin, Texas. Association for Computational Linguistics. find
17. Tania Lombrozo. 2006. The structure and function of explanations. Trends in Cognitive Sciences, 10(10):464–470. find
18. P. Mergenthaler, U. Lindauer, G. A. Dienel, and A. Meisel. 2013. Sugar for the brain: the role of glucose in physiological and pathological brain function. Trends in neurosciences. find
19. Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267:1–38. find
20. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849, San Diego, California. Association for Computational Linguistics. find
21. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics. find
22. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics. find our card
23. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
24. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. find
25. Hannah Rashkin, Antoine Bosselut, Maarten Sap, Kevin Knight, and Yejin Choi. 2018a. Modeling naive psychology of characters in simple commonsense stories. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2289– 2299, Melbourne, Australia. Association for Computational Linguistics. find
26. Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. 2018b. Event2Mind: Commonsense inference on events, intents, and reactions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 463–473, Melbourne, Australia. Association for Computational Linguistics. find
27. Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah Smith, and Yejin Choi. 2019. ATOMIC: An atlas of machine commonsense for if-then reasoning. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence. find
28. Roger C. Schank and Robert P. Abelson. 1977. Scripts, Plans, Goals and Understanding: an Inquiry into Human Knowledge Structures. L. Erlbaum, Hillsdale, NJ. find
29. Lei Sha, Sujian Li, Baobao Chang, and Zhifang Sui. 2016. Joint learning templates and slots for event schema induction. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 428–434, San Diego, California. Association for Computational Linguistics. find
30. Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. 2018. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 752–757, Melbourne, Australia. Association for Computational Linguistics. find
31. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. find
32. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics. find
33. Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. 2019. WIQA: A dataset for “what if...” reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6076–6085, Hong Kong, China. Association for Computational Linguistics. find
34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. find
35. Shaohua Yang, Qiaozi Gao, Sari Sadiya, and Joyce Chai. 2018. Commonsense justification for action explanation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2627–2637, Brussels, Belgium. Association for Computational Linguistics. find
36. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93– 104, Brussels, Belgium. Association for Computational Linguistics. find
37. Yilun Zhou, Steven Schockaert, and Julie Shah. 2019. Predicting conceptnet path quality using crowdsourced assessments of naturalness. In The World Wide Web Conference, WWW ’19, pages 2460– 2471, New York, NY, USA. ACM. find
38. Rolf A. Zwaan, Mark C. Langston, and Arthur C. Graesser. 1995. The construction of situation models in narrative comprehension: An event-indexing model. Psychological Science, 6(5):292–297. find
39. Rolf A Zwaan and Gabriel A Radvansky. 1998. Situation models in language comprehension and memory. Psychological bulletin, 123(2):162. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/commonsense-2020-mostafazadeh-glucose-generalized-contextualized-story-explanations/