Mova0.5 pre-release⌕
☰
home / docs / papers / English / Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework

Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework

What it is about

The paper turns existing semantic annotations into textual-inference (RTE) pairs with templates. Three resources are recast: semantic proto-roles (SPR, about 155k pairs), FrameNet paraphrases (FN+, about 155k) and definite pronoun resolution (DPR, about 3.7k). Each pair is labelled entailed or not from the original human judgments. A manual check of 100 pairs per set gives label accuracy of 85% (FN+), 94% (SPR), 98% (DPR) and 91% (SNLI). A neural RTE model trained and tested on the same set scores above 80% for FN+, SPR and SNLI, but a model trained on all four sets is at chance on DPR. Complex anaphora is the hardest phenomenon and paraphrase the easiest. Error analysis on SPR shows that change-of-state entailments cause most errors.

Ideas

For Mova

Chapters: 35. Reference and coreference (the DPR-type pairs) and 36. Discourse relations and coherence. Code: coref/src/sieve.rs for I3, prag/src/roles.rs for I1, world/src/qa.rs for I2. Measure: per-phenomenon accuracy on the generated set, run twice with identical output, plus the shuffled-antecedent control.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{white2017inference,
  title = {{Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework}},
  author = {White, Aaron Steven and Rastogi, Pushpendre and Duh, Kevin and Durme, Benjamin Van},
  booktitle = {Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2017},
  url = {https://aclanthology.org/I17-1100/}
}

References

53 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James Allen, Mehdi Manshadi, Myroslava Dzikovska, and Mary Swift. 2007. Deep linguistic processing for spoken dialogue systems. In Workshop on Deep Linguistic Processing. ACL, pages 49–56. find
  2. Laura Banarescu et al. 2013. Abstract meaning representation for sembanking. In 7th Linguistic Annotation Workshop and Interoperability with Discourse. ACL, Sofia, Bulgaria, pages 178–186. http://www.aclweb.org/anthology/W13-2322. ↗ our card
  3. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge. In Second PASCAL challenges workshop on RTE. volume 6, pages 6–4. find
  4. Luisa Bentivogli, Peter Clark, Ido Dagan, Hoa T. Dang, and Danilo Giampiccolo. 2010. The sixth PASCAL recognizing textual entailment challenge. In TAC. find
  5. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2011. The seventh PASCAL recognizing textual entailment challenge. In TAC. find
  6. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge. In TAC. volume 9, pages 14–24. find
  7. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Synthesis Lectures on HLT 6(4):1–220. find
  8. David Dowty. 1991. Thematic proto-roles and argument selection. language pages 547–619. find
  9. Charles Fillmore, Christopher Johnson, and Miriam R.L. Petruck. 2003. Background to framenet. International Journal to Lexicography 16(3). find
  10. Charles J. Fillmore and Collin F. Baker. 2001. Frame semantics for text understanding. In WordNet and Other Lexical Resources Workshop. NAACL. find
  11. LiMin Fu. 1994. Rule generation from neural networks. IEEE Transactions on Systems, Man, and Cybernetics 24(8):1114–1124. https://doi.org/10.1109/21.299696. ↗
  12. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. Ppdb: The paraphrase database. In NAACL-HLT. find our card
  13. Danilo Giampiccolo, Hoa Trang Dang, Bernardo Magnini, Ido Dagan, Elena Cabrio, and Bill Dolan. 2009. The fourth PASCAL recognizing textual entailment challenge. In TAC. find
  14. Guillaume Bouchard, Sameer Singh, and Théo Trouillon. 2015. On approximate reasoning capabilities of low-rank vector spaces. In 2015 AAAI Spring Symposium Series. find
  15. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In ACL-PASCAL workshop on textual entailment and paraphrasing. ACL, pages 1–9. find
  16. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326 . ↗
  17. Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational linguistics 28(3):245–288. find our card
  18. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Proceedings of the ACL. ACL. find
  19. Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading http://arxiv.org/abs/1601.06733v6. ↗
  20. Robin Cooper et al. 1996. Using the framework FRACAS a framework for computational semantics. Technical report, The FRACAS consortium. find
  21. Ann Copestake, Dan Flickinger, Carl Pollard, and Ivan A. Sag. 2005. Minimal recursion semantics: An introduction. Research on Language and Computation 3(2-3):281–332. find
  22. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine learning challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising textual Entailment, Springer, pages 177–190. find
  23. Levesque Hector, Davis Ernest, and Morgenstern Leora. 2012. The winograd schema challenge. In Knowledge Representation and Reasoning Conference. find
  24. Jerry R. Hobbs. 1985. Ontological promiscuity. In Proceedings of the ACL. ACL, pages 60–69. find
  25. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735– 1780. find
  26. Hans Kamp, Josef Van Genabith, and Uwe Reyle. 2011. Discourse representation theory. In Handbook of philosophical logic, Springer, pages 125–394. find
  27. Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078 . ↗
  28. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the conll-2011 shared task. In Proceedings of the fifteenth conference on computational natural language learning: Shared task. Association for Computational Linguistics, pages 28–34. find our card
  29. Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang. 2016. Learning natural language inference using bidirectional lstm model and inner-attention http://arxiv.org/abs/1605.09090v1. ↗
  30. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A sick cure for the evaluation of compositional distributional semantic models. In LREC. pages 216–223. find
  31. Tomas Mikolov, Wen Tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In NAACL-HLT. pages 746– 751. find
  32. Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2015. Natural language inference by tree-based convolution and heuristic matching http://arxiv.org/abs/1512.08422v3. ↗
  33. Tsendsuren Munkhdalai and Hong Yu. 2016. Neural tree indexers for text understanding http://arxiv.org/abs/1607.04492v1. ↗
  34. Lenhart K. Schubert and Chung Hee Hwang. 2000. Episodic logic meets little red riding hood: A comprehensive, natural representation for language understanding. NLP And Knowledge Representation pages 111–174. find
  35. Wang Shuohang and Jiang Jing. 2015. Learning natural language inference with lstm http://arxiv.org/abs/1512.08849v1. ↗
  36. Mark Steedman. 2000. ume 24. MIT Press. The Syntactic Process, vol-Ilya Sutskever, Oriol Vinyals, and Quoc Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems. pages 3104–3112. find
  37. Adam R Teichert, Adam Poliak, Benjamin Van Durme, and Matthew R Gormley. 2017. Semantic proto-role labeling. In AAAI. pages 4459–4466. find
  38. Sebastian B. Thrun. 1993. Extracting provably correct rules from artificial neural networks. Technical report, University of Bonn. find
  39. Martha Palmer, Dan Gildea, and Paul Kingsbury. 2005. The proposition bank: A corpus annotated with semantic roles. Computational Linguistics 31:1. find
  40. Sebastian B. Thrun. 1995. Extracting rules from artificial neural networks with distributed representations. In Advances in Neural Information Processing Systems, pages 505–512. http://papers.nips.cc/paper/3869-conditional-neural-fields.pdf. ↗
  41. Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference http://arxiv.org/abs/1606.01933v1. ↗
  42. Geoffrey G. Towell and Jude W. Shavlik. 1993. Extracting refined rules from knowledge-based neural networks. Machine Learning 13(1):71–101. https://doi.org/10.1007/BF00993103. ↗
  43. Ellie Pavlick, Travis Wolfe, Pushpendre Rastogi, Chris Callison-Burch, Mark Drezde, and Benjamin Van Durme. 2015. Framenet+: Fast paraphrastic tripling of framenet. In ACL. ACL, Beijing, China. find
  44. Jason Weston, Antoine Bordes, Sumit Chopra, Sasha Rush, Bart van Merrienboer, Armand Joulin, and Tomas Mikolov. 2016. Towards ai-complete question answering: A set of prerequisite toy tasks. In ICLR. find our card
  45. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. ACL. find
  46. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: The winograd schema challenge. In Joint Conference on EMNLP and CoNLL. ACL, pages 777–789. find
  47. Pushpendre Rastogi and Benjamin Van Durme. 2014. Augmenting framenet via PPDB. In Second Workshop on EVENTS: Definition, Detection, Coreference, and Representation. ACL. http://www.aclweb.org/anthology/W14-2901. ↗
  48. Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015. Semantic proto-roles. Transactions of the ACL 3:475–488. find
  49. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664 . ↗
  50. Aaron Steven White, Drew Reisinger, Keisuke Sakaguchi, Tim Vieira, Sheng Zhang, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2016. Universal decompositional semantics on universal dependencies. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, TX, pages 1713–1723. find our card
  51. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2:67–78. find
  52. Matthew Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In ECCV. Springer, pages 818–833. find
  53. Matthew D. Zeiler. 2012. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701 . ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/white-2017-inference-is-everything-recasting-semantic/