``Was It Good? It Was Provocative.'' Learning the Meaning of Scalar Adjectives

What it is about

Indirect answers to yes/no questions often do not say "yes" or "no" outright. "Was it good? It was provocative" leaves the answer open, while "It was great" does not. The authors mine two kinds of evidence. First, scale order between gradable adjectives comes from online IMDB reviews: a word's expected star rating (from the rating distribution of its reviews) puts "good" below "excellent" on a common scale. A decision rule then maps the question and answer predications to yes, no or uncertain, including negation. Second, for numerical answers ("about 350 people", "seven years old") the authors fit a logistic model of the probability that a number counts as the adjective, using hit counts from Web queries with and without negation. The test set is 224 question-answer pairs from CNN interview transcripts and the Switchboard dialogue corpus, each labelled by 30 Mechanical Turk workers. The system matches the dominant worker answer in 71% of pairs (159 of 224). On the 66 pairs where workers fully agreed the rate is 89%. For "other adjective" answers the method reaches 60%, against 56% for a WordNet-based sentiment method. It correctly marks 18 of 31 oppositional pairs, against 5 for a hand-built antonym list.

Ideas

• I1. prag/src/infer.rs: a yes/no decision table over scale order, as in the paper's rule set (different signs mean "no", a larger magnitude means "yes", negation flips yes and no). Write it as a short rule list with a BTreeMap of scale positions. Measure against the paper's 224 pairs when they can be obtained under a licence that allows it, or against hand-made pairs.
• I2. Agreement as a confidence signal: the paper's logistic fit shows accuracy falls as worker entropy rises. Mova can output "uncertain" when its own rule has no scale position for either predication, and check the rate against a held-out set.
• I3. For number-plus-adjective answers, a rule-based version: a table of typical units and thresholds per adjective (age for "little kids", degrees for "warm weather"), not a Web-count model. The paper's eight well-fitting items are a small test.
• I4. The oppositional-term test (31 items, 18 correct) is a useful negative control: any new antonym table must beat a plain antonym list on these items.

For Mova

The chapters are 15. Adjectives and adverbs (scales and antonyms) and 36. Discourse relations and coherence (indirect answers). The code is prag/src/infer.rs for the decision rule and en/src/dict.rs for the scale table. Measure: accuracy against the dominant answer on a held-out set of Mova's own dialogues, with a shuffled-scale control that must score lower. Note: the paper's IMDB and CNN text is not covered by this card; its licence was not checked.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{marneffe2010was,
  title = {{Was It Good? It Was Provocative.'' Learning the Meaning of Scalar Adjectives}},
  author = {Marneffe, Marie-Catherine de and Manning, Christopher D. and Potts, Christopher},
  booktitle = {Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
  url = {https://aclanthology.org/P10-1018/}
}

References

26 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. James F. Allen and C. Raymond Perrault. 1980. Analyzing intention in utterances. Artificial Intelligence, 15:143–178. find
2. Nicholas Asher and Alex Lascarides. 2003. Logics of Conversation. Cambridge University Press, Cambridge. find
3. Sasha Blair-Goldensohn, Kerry Hannan, Ryan McDonald, Tyler Neylon, George A. Reis, and Jeff Reynar. 2008. Building a sentiment summarizer for local service reviews. In WWW Workshop on NLP in the Information Explosion Era (NLPIX). find
4. Judith A. Chevalier and Dina Mayzlin. 2006. The effect of word of mouth on sales: Online book reviews. Journal of Marketing Research, 43(3):345– 354. find
5. Herbert H. Clark. 1979. Responding to indirect speech acts. Cognitive Psychology, 11:430–477. find
6. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC-2006). find our card
7. Marie-Catherine de Marneffe, Scott Grimm, and Christopher Potts. 2009. Not a simple ‘yes’ or ‘no’: Uncertainty in indirect answers. In Proceedings of the 10th Annual SIGDIAL Meeting on Discourse and Dialogue. find
8. Gilles Fauconnier. 1975. Pragmatic scales and logical structure. Linguistic Inquiry, 6(3):353–375. find
9. Nancy Green and Sandra Carberry. 1994. A hybrid reasoning model for indirect answers. In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics, pages 58–65. find
10. Nancy Green and Sandra Carberry. 1999. Interpreting and generating indirect answers. Computational Linguistics, 25(3):389–435. find
11. Beth Ann Hockey, Deborah Rossen-Knill, Beverly Spejewski, Matthew Stone, and Stephen Isard. 1997. Can you predict answers to Y/N questions? Yes, No and Stuff. In Proceedings of Eurospeech 1997, pages 2267–2270. find
12. Laurence R Horn. 1972. On the Semantic Properties of Logical Operators in English. Ph.D. thesis, UCLA, Los Angeles. find
13. Nan Hu, Paul A. Pavlou, and Jennifer Zhang. 2006. Can online reviews reveal a product’s true quality?: Empirical findings and analytical modeling of online word-of-mouth communication. In Proceedings of Electronic Commerce (EC), pages 324–330. find
14. Daniel Jurafsky, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual, draft 13. Technical Report 97-02, University of Colorado, Boulder Institute of Cognitive Science. find
15. Hans Kamp and Barbara H. Partee. 1995. Prototype theory and compositionality. Cognition, 57(2):129– 191. find
16. Christopher Kennedy and Louise McNally. 2005. Scale structure and the semantic typology of gradable predicates. Language, 81(2):345–381. find
17. Christopher Kennedy. 2007. Vagueness and grammar: The semantics of relative and absolute gradable adjectives. Linguistics and Philosophy, 30(1):1–45. find
18. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Meeting of the Association of Computational Linguistics. find our card
19. Stephen C. Levinson. 2000. Presumptive Meanings: The Theory of Generalized Conversational Implicature. MIT Press, Cambridge, MA. find
20. Saif Mohammad, Bonnie Dorr, and Graeme Hirst. 2008. Computing word-pair antonymy. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-2008). find
21. Robert Munro, Steven Bethard, Victor Kuperman, Vicky Tzuyin Lai, Robin Melnick, Christopher Potts, Tyler Schnoebelen, and Harry Tily. 2010. Crowdsourcing and language studies: The new generation of linguistic data. In NAACL 2010 Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk. find
22. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1):1–135. find
23. C. Raymond Perrault and James F. Allen. 1980. A plan-based analysis of indirect speech acts. American Journal of Computational Linguistics, 6(3-4):167–182. find
24. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of KDD-2008. find
25. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-2008). find
26. Henk Zeevat. 1994. Questions and exhaustivity in update semantics. In Harry Bunt, Reinhard Muskens, and Gerrit Rentier, editors, Proceedings of the International Workshop on Computational Semantics, pages 211–221. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/marneffe-2010-was-it-good-it-was/