Mova0.5 pre-release⌕
☰
home / docs / papers / English / Inherent Disagreements in Human Textual Inferences

Inherent Disagreements in Human Textual Inferences

What it is about

The paper asks whether disagreement among annotators of NLI pairs is noise that aggregation removes, or a stable signal. Fifty crowd raters labelled each pair on a sliding scale from −50 to +50; 100 pairs were drawn from each of five datasets (RTE2, SNLI, MNLI, JOCI, DNC). After filtering, 496 pairs remain with about 39 ratings each. A single Gaussian explains most pairs; a Gaussian mixture with a second component fits held-out ratings better for about 20% of pairs, and the extra component is not an overfitting artefact. In a context study with 300 word pairs, variance rises from 0.34 for single words to 0.41 for sentences and 0.56 for paragraphs. A fine-tuned BERT gives softmax distributions that fit the human labels poorly: the average cross-entropy is 0.37 against 0.03 for random draws from the model, and the log-probability of the human labels is −21.5 against an expected −1.6. The authors argue that NLI should be scored against full distributions, not majority labels.

Ideas

For Mova

Chapters: 33. Implicature and presupposition and 30. Pragmatics and discourse: overview. Code: prag/src/infer.rs for the vote spread, prag/src/rsa.rs for the pragmatic listener. Measure: cross-entropy against the held-out distributions, run twice, against the random-draw baseline.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00293

@article{pavlick2019inherent,
  title = {{Inherent Disagreements in Human Textual Inferences}},
  author = {Pavlick, Ellie and Kwiatkowski, Tom},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.1162/tacl\_a\_00293},
  url = {https://doi.org/10.1162/tacl_a_00293}
}

References

60 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lora Aroyo, Anca Dumitrache, Praveen Paritosh, Alex Quinn, and Chris Welty, editors. 2018. Proc. Subjectivity, Ambiguity and Disagreement in Crowdsourcing (SAD), volume 1 of 1. HCOMP, Zurich, Switzerland. find
  2. Islam Beltagy, Katrin Erk, and Raymond Mooney. 2014. Probabilistic soft logic for semantic textual similarity. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1210–1219, Baltimore, MD. Association for Computational Linguistics. find
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. find
  4. Razvan Bunescu. 2008. Learning with probabilistic features for improved pipeline models. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 670–679, Honolulu, HI. Association for Computational Linguistics. find
  5. Chris Callison-Burch and Mark Dredze. 2010. Creating speech and language data with Amazon’s mechanical turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 1–12, Los Angeles, CA. Association for Computational Linguistics. find
  6. Stergios Chatzikyriakidis, Robin Cooper, Simon Dobnik, and Staffan Larsson. 2017. An overview of natural language inference data collection: The way forward? In Proceedings of the Computing Natural Language Inference Workshop. find
  7. Timothy Chklovski and Patrick Pantel. 2004. VerbOcean: Mining the Web for fine-grained semantic verb relations. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 33–40, Barcelona, Spain. Association for Computational Linguistics. 2009 Conference on Empirical Methods in Natural Language Processing, pages 440–449, Singapore. Association for Computational Linguistics. find
  8. Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, and Steve Pullman. 1996, Using the framework. Technical report, Technical Report LRE 62-051 D-16, The FraCaS Consortium. find
  9. Allyson Ettinger, Ahmed Elgohary, Colin Phillips, and Philip Resnik. 2018. Assessing composition in sentence vector representations. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1790–1801, Santa Fe, NM. Association for Computational Linguistics. find
  10. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006, The PASCAL recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Textual Entailment, pages 177–190. find
  11. Springer Christiane Fellbaum. 1998. WordNet, Wiley Online Library. find
  12. Jenny Rose Finkel, Christopher D. Manning, and Andrew Y. Ng. 2006. Solving the problem of cascading errors: Approximate Bayesian inference for linguistic annotation pipelines. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 618–626, Sydney, Australia. Association for Computational Linguistics. find
  13. Ishita Dasgupta, Demi Guo, Andreas Stuhlmüller, Samuel J. Gershman, and Noah D. Goodman. 2018. Evaluating compositionality in sentence embeddings. arXiv preprint arXiv:1802.04302. ↗
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, MN. Association for Computational Linguistics. find
  15. Nir Friedman, Lise Getoor, Daphne Koller, and Avi Pfeffer. 1999. Learning probabilistic relational models. In IJCAI, 99: 1300–1309. find
  16. Konstantina Garoufi. 2007. Towards a Better Understanding of Applied Textual Entailment. Ph.D. thesis, Citeseer. find
  17. Michael L. Geis and Arnold M. Zwicky. 1971. On invited inferences. Linguistic inquiry, 2(4):561–566. find
  18. Anca Dumitrache, Lora Aroyo, and Chris Welty. 2019. A crowdsourced frame disambiguation corpus with ambiguity. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2164– 2170, Minneapolis, MN. Association for Computational Linguistics. find
  19. Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics. find
  20. Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. ArXiv, abs/1602.02410. ↗
  21. Anca Dumitrache, Lora Aroyo, Christopher A. Welty, Robert-Jan Sips, and Anthony Levas. 2013. Dr. Detective: Combining gamification techniques and crowdsourcing to create a gold standard for the medical domain. In CrowdSem Workshop at the International Semantic Web Conference. find
  22. David Jurgens. 2013. Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 556–562, Atlanta, GA. Association for Computational Linguistics. find
  23. Katrin Erk and Diana McCarthy. 2009. Graded word sense assignment. In Proceedings of the Lauri Karttunen, Stanley Peters, Annie Zaenen, and Cleo Condoravdi. 2014. The chameleon-like nature of evaluative adjectives. Empirical Issues in Syntax and Semantics, 10:233–250. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. find
  24. Angelika Kimmig, Stephen Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. 2012. A short introduction to probabilistic soft logic. In Proceedings of the NIPS Workshop on Probabilistic Programming: Foundations and Applications, pages 1–4. find
  25. Richard Montague. 1970. Universal grammar. Theoria, 36(3):373–398. find
  26. Jennimaria Palomaki, Olivia Rhinehart, and Michael Tseng. 2018. A case for a range of acceptable annotations. In Proceedings of Workshop on Subjectivity, Ambiguity, and Disagreement (SAD). HCOMP. find
  27. Volodymyr Kuleshov and Percy S. Liang. 2015. Calibrated structured prediction. In Advances in Neural Information Processing Systems, pages 3474–3482. find
  28. Christopher D. Manning. 2006. Local textual inference: Its hard to circumscribe, but you know it when you see it–and NLP needs it. https://nlp.stanford.edu/manning/papers/Textual ↗
  29. Inference.pdf Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2018. Factivity in doubt: Clause-embedding predicates in naturally occurring discourse. Sinn und Bedeutung 23 (Poster). find
  30. Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2012. Did it happen? The pragmatic complexity of veridicality assessment. Computational Linguistics, 38(2):301–333. find
  31. Rebecca J. Passonneau, Vikas Bhardwaj, Ansaf Salleb-Aouissi, and Nancy Ide. 2012. Multiplicity and word sense: Evaluating and learning from multiply labeled word sense annotations. Language Resources and Evaluation, 46(2):219–252. find
  32. Ellie Pavlick and Chris Callison-Burch. 2016a. Most ‘‘babies’’ are ‘‘little’’ and most ‘‘problems’’ are ‘‘huge’’: Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2164–2173, Berlin, Germany. Association for Computational Linguistics. find
  33. Ellie Pavlick and Chris Callison-Burch. 2016b. So-called non-subsective adjectives. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 114–119, Berlin, Germany. Association for Computational Linguistics. find
  34. Héctor Martı́nez Alonso, Anders Johannsen, and Barbara Plank. 2016. Supersense tagging with inter-annotator disagreement. In Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016), pages 43–48, Berlin, Germany. Association for Computational Linguistics. find
  35. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Learning part-of-speech taggers with inter-annotator agreement loss. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–751, Gothenburg, Sweden. Association for Computational Linguistics. find
  36. Héctor Martı́nez Alonso, Barbara Plank, Arne Skjærholt, and Anders Søgaard. 2015. Learning to parse with IAA-weighted loss. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1357–1361, Denver, CO. Association for Computational Linguistics. find
  37. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. find our card
  38. Massimo Poesio and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky, pages 76–83, Ann Arbor, MI. Association for Computational Linguistics. find
  39. Massimo Poesio, Jon Chamberlain, Silviu Paun, Juntao Yu, Alexandra Uma, and Udo Kruschwitz. 2019. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1778–1789, Minneapolis, MN. Association for Computational Linguistics. why. Semantics and Linguistic Theory, 20: 309–327. find
  40. Nathaniel J. Smith, Noah Goodman, and Michael Frank. 2013. Learning and using language via recursive pragmatic reasoning about other agents. In Advances in Neural Information Processing Systems, pages 3039–3047. find
  41. Adam Poliak, Yonatan Belinkov, James Glass, and Benjamin Van Durme. 2018a. On the evaluation of semantic phenomena in neural machine translation using natural language inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 513–523, New Orleans, LA. Association for Computational Linguistics. find
  42. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018b. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 67–81, Brussels, Belgium. Association for Computational Linguistics. find
  43. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast—but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 254–263, Honolulu, HI. Association for Computational Linguistics. find
  44. M. Tanenhaus, G. Carlson, and M. S. Seidenberg. 1985. Do listeners compute linguistic representations? D. Dowty, L. Karttunen, and A. Zwicky, editors, Natural Language Parsing. Cambridge University Press. find
  45. Judith Tonhauser, David I. Beaver, and Judith Degen. 2018. How Projective is Projective Content? Gradience in Projectivity and At-issueness. Journal of Semantics, 35(3):495–542. find
  46. Y. Versley. 2008. Vagueness and referential ambiguity in a large-scale annotated corpus. Re-searchon Language and Computation, 6:333–353. find
  47. Marta Recasens, Eduard Hovy, and M. Antònia Martı́. 2011. Identity, non-identity, and near-identity: Addressing the complexity of coreference. Lingua, 121(6):1138–1152. find
  48. Matthijs Westera and Gemma Boleda. 2019. Don’t blame distributional semantics if it can’t do entailment. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 120–133, Gothenburg, Sweden. Association for Computational Linguistics. find
  49. Dennis Reidsma and Rieks op den Akker. 2008. Exploiting ‘subjective’ annotations. In Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics, pages 8–16, Manchester, UK. Coling 2008 Organizing Committee. find
  50. Matthew Richardson and Pedro Domingos. 2006. Markov logic networks. Machine Learning, 62(1–2):107–136. find
  51. Aaron S. White, Valentine Hacquard, and Jeffrey Lidz. 2018. Semantic information and the syntax of propositional attitude verbs. Cognitive Science, 42(2):416–456. find
  52. Mike Schaekermann, Edith Law, Alex C. Williams, and William Callaghan. 2016. Resolvable vs. irresolvable ambiguity: A new hybrid framework for dealing with uncertain ground truth. In 1st Workshop on Human-Centered Machine Learning at SIGCHI. find
  53. Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. Inference is everything: Recasting semantic resources into a unified evaluation framework. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 996–1005, Mandy Simons, Judith Tonhauser, David Beaver, and Craige Roberts. 2010. What projects and Taipei, Taiwan. Asian Federation of Natural Language Processing. find our card
  54. Lotfi A. Zadeh. 1996. Fuzzy logic = computing with words. IEEE Transactions on Fuzzy Systems, 4(2):103–111. find
  55. Aaron Steven White and Kyle Rawlins. 2017. The role of veridicality and factivity in clause selection. 48th Annual Meeting of the North East Linguistic Society, Reykjavı́k. http:// iceland2017.nelsconference.org/wp-content/ uploads/2017/08/White-Rawlins.pdf. ↗
  56. Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 694–699. ACM. find
  57. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, LA. Association for Computational Linguistics. find
  58. Annie Zaenen, Lauri Karttunen, and Richard Crouch. 2005. Local textual inference: Can it be defined or circumscribed? In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, pages 31–36, Ann Arbor, MI. Association for Computational Linguistics. find
  59. Lotfi A. Zadeh. 1994. Fuzzy logic, neural networks, and soft computing. Communications of the ACM, 37(3):77–84. find
  60. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 1996. Ordinal Common-sense Inference. Transactions of the Association for Computational Linguistics, 5:379–395. https://www.aclweb.org/anthology/ Q17-1027. doi:10.1162/tac a 00068. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/pavlick-2019-inherent-disagreements-human-textual-inferences/