Mova0.5 pre-release⌕
☰
home / docs / papers / English / Temporal Reasoning in Natural Language Inference

Temporal Reasoning in Natural Language Inference

What it is about

The paper turns four temporal datasets (TempEval3, TimeBank-Dense, RED and UDS-Time) into NLI pairs about two kinds of temporal reasoning: event order and event duration. Order is asked through 8 templates about the start and end points of two events, mapped from Allen-style relations. Duration is asked through 18 templates ("X did last longer than a minute"), built from the 11 duration labels of UDS-Time. The recast sets have more than a million pairs in total (504,136 duration pairs and 562,944 order pairs from UDS-Time, and 27,240, 11,910 and 5,578 from the other three). A validation study with 48 crowd workers finds that entailed pairs are rated as more likely true (χ²(1) = 226.1, p < 0.001). Models trained on MNLI do poorly on these sets, often below the majority baseline. Duration is partly solvable by templates alone: a template-based majority reaches 80.2% on the duration test set, and the neural models gain about 10 points over it.

Ideas

For Mova

The code is world/src/events.rs and world/src/narr.rs. The chapters are 36. Discourse relations and coherence and 13. Verbs for tense and aspect. Measure order and duration accuracy on school stories, with the two-fold cv split, run twice.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.363

@inproceedings{vashishtha2020temporal,
  title = {{Temporal Reasoning in Natural Language Inference}},
  author = {Vashishtha, Siddharth and Poliak, Adam and Lal, Yash Kumar and Durme, Benjamin Van and White, Aaron Steven},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.363},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.363}
}

References

59 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James F Allen. 1983. Maintaining knowledge about temporal intervals. Communications of the ACM, 26(11):832–843. find
  2. James F Allen. 1984. Towards a general theory of action and time. Artificial intelligence, 23(2):123– 154. find
  3. Yonatan Belinkov, Adam Poliak, Stuart M Shieber, Benjamin Van Durme, and Alexander M Rush. 2019. Don’t take the premise for granted: Mitigating artifacts in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 877–891. find
  4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. find
  5. Bertram C Bruce. 1972. A model for temporal references and its appliicationin a question answering program. Artificial intelligence. find
  6. Roberto Casati and Achille C. Varzi. 1996. Events. Dartmouth. find
  7. Nathanael Chambers, Taylor Cassidy, Bill McDowell, and Steven Bethard. 2014. Dense event ordering with a multi-pass architecture. Transactions of the Association for Computational Linguistics, 2:273– 284. find
  8. Yao Cheng, Peter Anick, Pengyu Hong, and Nianwen Xue. 2013. Temporal relation discovery between events and temporal expressions identified in clinical narrative. Journal of biomedical informatics, 46:S48–S53. find
  9. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics. find
  10. Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al. 1996. Using the framework. Technical report. find
  11. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177– 190. Springer. find
  12. David Graff. The aquaint corpus of English news text:[content copyright] Portions c 1998-2000 New York Times, Inc., c 1998-2000 Associated Press, Inc., c 1996-2000 Xinhua News Service. Linguistic Data Consortium. find
  13. Rujun Han, I-Hung Hsu, Mu Yang, Aram Galstyan, Ralph Weischedel, and Nanyun Peng. 2019. Deep structured neural network for event temporal relation extraction. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 666–106, Hong Kong, China. Association for Computational Linguistics. find
  14. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1681–1691. find
  15. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models imppressive? learning implicature and presupposition. arXiv preprint arXiv:2004.03066. ↗ our card
  16. Hyuckchul Jung, James Allen, Nate Blaylock, William de Beaumont, Lucian Galescu, and Mary Swift. 2011. Building timelines from narrative clinical records: initial results based-on deep natural language understanding. In Proceedings of BioNLP 2011 workshop, pages 146–154. find
  17. Kenneth Kahn and G Anthony Gorry. 1977. Mechanizing temporal knowledge. Artificial intelligence, 9(1):87–108. find
  18. Daniel Khashabi. 2019. Reasoning-Driven Question-Answering for Natural Language Understanding. Ph.D. thesis, University of Pennsylvania. find
  19. Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sabrina J Mielke, Arya D McCarthy, Sandra Kübler, et al. 2018. Unimorph 2.0: universal morphology. arXiv preprint arXiv:1810.11101. ↗
  20. Robert Kowalski and Marek Sergot. 1989. A logic-based calculus of events. In Foundations of knowledge base management, pages 23–55. Springer. find
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  22. Hector Llorens, Nathanael Chambers, Naushad Uz-Zaman, Nasrin Mostafazadeh, James Allen, and James Pustejovsky. 2015. Semeval-2015 task 5: Qa tempeval-evaluating temporal information understanding with question answering. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 792–800. find
  23. John McCarthy and Patrick J Hayes. 1987. Some philosophical problems from the standpoint of artificial intelligence. In Readings in nonmonotonic reasoning, pages 26–45. find
  24. Drew McDermott. 1982. A temporal logic for reasoning about processes and plans. Cognitive science, 6(2):101–155. find
  25. Yuanliang Meng and Anna Rumshisky. 2018. Context-aware neural model for temporal information extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 527–536, Melbourne, Australia. Association for Computational Linguistics. find
  26. Nasrin Mostafazadeh, Alyson Grealish, Nathanael Chambers, James Allen, and Lucy Vanderwende. 2016. CaTeRS: Causal and temporal relation scheme for semantic annotation of event structures. In Proceedings of the Fourth Workshop on Events, pages 51–61, San Diego, California. Association for Computational Linguistics. find
  27. Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. Natural language inference by tree-based convolution and heuristic matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 130–136, Berlin, Germany. Association for Computational Linguistics. find
  28. Aakanksha Naik, Luke Breitfeller, and Carolyn Rose. 2019. TDDiscourse: A dataset for discourse-level temporal ordering of events. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 239–249, Stockholm, Sweden. Association for Computational Linguistics. find
  29. Alexander Nakhimovsky. 1987. Temporal reasoning in natural language understanding: The temporal structure of the narrative. In Proceedings of the Third Conference on European Chapter of the Association for Computational Linguistics, EACL ’87, page 262–269, USA. Association for Computational Linguistics. find
  30. Qiang Ning, Zhili Feng, and Dan Roth. 2017. A structured learning approach to temporal relation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1027–1037. find
  31. Qiang Ning, Zhili Feng, Hao Wu, and Dan Roth. 2018a. Joint reasoning for temporal and causal relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 2278–2288. find
  32. Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. Torque: A reading comprehension dataset of temporal ordering questions. arXiv preprint arXiv:2005.00242. ↗ our card
  33. Qiang Ning, Hao Wu, and Dan Roth. 2018b. A multi-axis annotation scheme for event temporal relations. arXiv preprint arXiv:1804.07828. ↗
  34. Tim O’Gorman, Kristin Wright-Bettner, and Martha Palmer. 2016. Richer event description: Integrating event coreference with temporal, causal and bridging annotation. In Proceedings of the 2nd Workshop on Computing News Storylines (CNS 2016), pages 47–56. find
  35. AK Pani and GP Bhattacharjee. 2001. Temporal representation and reasoning in artificial intelligence: A review. Mathematical and Computer Modelling, 34(1-2):55–80. find
  36. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  37. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 67–81, Brussels, Belgium. Association for Computational Linguistics. find
  38. James Pustejovsky, Patrick Hanks, Roser Sauri, Andrew See, Robert Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, et al. 2003. The timebank corpus. In Corpus linguistics, volume 2003, page 40. Lancaster, UK. find
  39. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. arXiv preprint arXiv:2003.07082. ↗ our card
  40. Gabriel A Radvansky and Jeffrey M Zacks. 2014. Event cognition. Oxford University Press. find
  41. Nils Reimers, Nazanin Dehghani, and Iryna Gurevych. 2016. Temporal anchoring of events for the timebank corpus. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2195– 2204. find
  42. Alexis Ross and Ellie Pavlick. 2019. How well do NLI models capture verb veridicality? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2230–2240, Hong Kong, China. Association for Computational Linguistics. find our card
  43. Lenhart Schubert. 2002. Can we derive general world knowledge from texts? In Proceedings of the second international conference on Human Language Technology Research, pages 94–97. Morgan Kaufmann Publishers Inc. find
  44. John Sylak-Glassman, Christo Kirov, David Yarowsky, and Roger Que. 2015. A language-independent feature schema for inflectional morphology. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 674–680. find
  45. Niket Tandon, Bhavana Dalvi Mishra, Joel Grus, Wen-tau Yih, Antoine Bosselut, and Peter Clark. 2018. Reasoning about actions and state changes by injecting commonsense knowledge. arXiv preprint arXiv:1808.10012. ↗
  46. Julien Tourille, Olivier Ferret, Aurélie Névéol, and Xavier Tannier. 2017. Neural architecture for temporal relation extraction: A bi-LSTM approach for detecting narrative containers. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 224–230, Vancouver, Canada. Association for Computational Linguistics. find
  47. Naushad UzZaman, Hector Llorens, James Allen, Leon Derczynski, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 Task 1: TempEval-3: Evaluating Time Expressions, Events, and Temporal Relations. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 1–9, Atlanta, Georgia. Association for Computational Linguistics. find
  48. Benjamin D Van Durme. 2011. Extracting implicit knowledge from text, proquest. find
  49. Siddharth Vashishtha, Benjamin Van Durme, and Aaron Steven White. 2019. Fine-grained temporal relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2906–2919, Florence, Italy. Association for Computational Linguistics. find
  50. Chengyu Wang, Xiaofeng He, and Aoying Zhou. 2018. Event phase oriented news summarization. World Wide Web, 21(4):1069–1092. find
  51. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. find
  52. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗
  53. Jeffrey M Zacks and Barbara Tversky. 2001. Event structure in perception and conception. Psychological bulletin, 127(1):3. find
  54. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017a. Ordinal common-sense inference. Transactions of the Association of Computational Linguistics, 5(1):379–395. find
  55. Sheng Zhang, Rachel Rudinger, and Benjamin Van Durme. 2017b. An evaluation of predpatt and open ie via stage 1 semantic role labeling. In IWCS 2017—12th International Conference on Computational Semantics (Short papers). find
  56. Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3363–3369, Hong Kong, China. Association for Computational Linguistics. find
  57. Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020. Temporal common sense acquisition with minimal supervision. arXiv preprint arXiv:2005.04304. ↗
  58. Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. Inference is everything: Recasting semantic resources into a unified evaluation framework. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 996–1005, Taipei, Taiwan. Asian Federation of Natural Language Processing. find our card
  59. Aaron Steven White, Drew Reisinger, Keisuke Sakaguchi, Tim Vieira, Sheng Zhang, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2016. Universal decompositional semantics on universal dependencies. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1713–1723, Austin, TX. Association for Computational Linguistics. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/vashishtha-2020-temporal-reasoning-natural-language-inference/