Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders

What it is about

Intent embedding models (for chatbots that map «I want to order a pizza» to order_food) are tested on two things standard benchmarks miss: negation («I don't want to order pizza», a different intent with high word overlap) and implicature («I'm really hungry», the same intent with little overlap). Test utterances for three intent benchmarks (BANKING77, HWU64, CLINC150) are generated with GPT models and checked by people (faithful in most cases; 62% of implicature utterances judged not explicit). In the hard triplet test (is the paraphrase or implicature closer to the original than its negation?) all embedding models succeed less than 25% of the time; implicature utterances also lower classification and clustering scores. Fine-tuning with a contrastive loss on LLM-generated positives and negatives improves the triplet task but trades off against other tasks.

Ideas

• I1. Build the paper's triplet test for Mova's student and frame matcher, which is a word-overlap trap of the same kind: for a lesson fact, a negated variant (high overlap, opposite meaning) and an implied variant (low overlap, same meaning, e.g. «The crow could not keep the cheese» vs «The cheese fell from the crow's beak»). The matching score of the student (world/src/knowledge.rs) must rank the implied variant above the negated one. Generate the variants once with the teacher, check them by hand, keep them as a fixed test with the success rate reported; it is the gate the Torah negation finding in Watching the SLM think: the diagnosis of its answers lacks.
• I2. In the dialogue layer (world/src/talk.rs), check commands and requests for negation before matching them to an action («don't tell me again» must not trigger tell); add the negated forms of each command to its test list as negative controls.

For Mova

27. Negation, 33-implicature-and-presupposition.md §5, 31-speech-acts.md (requests). Code: world/src/knowledge.rs (matching score), world/src/talk.rs (commands). Measures: triplet success rate; command tests with negated controls.

Topics: Implicature

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.33 · arXiv 2403.04314

@inproceedings{zhang2024can,
  title = {{Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders}},
  author = {Zhang, Yuwei and Singh, Siffi and Sengupta, Sailik and Shalyminov, Igor and Su, Hang and Song, Hwanjun and Mansour, Saab},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.33},
  eprint = {2403.04314},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.acl-long.33}
}

References

27 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. We would like to thank the AWS AI Team in general and James Gung, Hossein Aboutalebi, and Dennis Ulmer in particular for insightful discussions and their contribution to our pilot study. In addition, we also would like to thank the reviewers in two iterations of ARR, for the February 2024 and the April 2024 cycle, whose comments and suggestions have helped us improve the work. find
2. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al-shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Hes-low, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of-the-art performance. find
3. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics. find our card
4. Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45, Online. Association for Computational Linguistics. find
5. Xilun Chen, Asish Ghoshal, Yashar Mehdad, Luke Zettlemoyer, and Sonal Gupta. 2020. Low-resource domain adaptation for compositional task-oriented semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5090–5100, Online. Association for Computational Linguistics. find
6. Pierre Colombo, Nathan Noiry, Ekhine Irurozki, and Stephan Clemencon. 2022. What are the best systems? new perspectives on nlp benchmarking. arXiv preprint arXiv:2202.03799. ↗
7. Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
8. Asa Cooper Stickland, Sailik Sengupta, Jason Krone, Saab Mansour, and He He. 2023. Robustification of multilingual language models to real-world noise in crosslingual zero-shot settings with robust contrastive pretraining. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1375–1391, Dubrovnik, Croatia. Association for Computational Linguistics. find
9. Thomas Dopierre, Christophe Gravier, and Wilfried Logerais. 2021. PROTAUGMENT: Unsupervised diverse short-texts paraphrasing for intent detection meta-learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2454–2466, Online. Association for Computational Linguistics. find
10. Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226–231. find
11. Pablo A Estévez, Michel Tesmer, Claudio A Perez, and Jacek M Zurada. 2009. Normalized mutual information feature selection. IEEE Transactions on neural networks, 20(2):189–201. find
12. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. find
13. James Gung, Raphael Shu, Emily Moeng, Wesley Rose, Salvatore Romeo, Yassine Benajiba, Arshit Gupta, Saab Mansour, and Yi Zhang. 2023. Intent induction from conversations for task-oriented dialogue track at dstc 11. find
14. Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2787–2792, Brussels, Belgium. Association for Computational Linguistics. find
15. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. find our card
16. Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1311–1316, Hong Kong, China. Association for Computational Linguistics. find
17. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
18. Xiaoming Liu, Hongyu Lin, Xianpei Han, and Le Sun. 2023. Sentbench: Comprehensive evaluation of self-supervised sentence representation with benchmark construction. In Chinese Computational Linguistics, pages 449–463, Singapore. Springer Nature Singapore. find
19. Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2021. Benchmarking Natural Language Understanding Services for Building Conversational Agents, pages 165–183. Springer Singapore, Singapore. find
20. Tingting Ma, Qianhui Wu, Zhiwei Yu, Tiejun Zhao, and Chin-Yew Lin. 2022. On the effectiveness of sentence encoding for intent detection meta-learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3806–3818, Seattle, United States. Association for Computational Linguistics. find
21. James MacQueen et al. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pages 281–297. Oakland, CA, USA. find
22. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 216–223, Reykjavik, Iceland. European Language Resources Association (ELRA). find
23. Aaron Mueller, Jason Krone, Salvatore Romeo, Saab Mansour, Elman Mansimov, Yi Zhang, and Dan Roth. 2022. Label semantic aware pre-training for few-shot text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8318– 8334, Dublin, Ireland. Association for Computational Linguistics. find
24. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics. find
25. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268. ↗
26. Frank Nielsen and Frank Nielsen. 2016. Hierarchical clustering. Introduction to HPC with MPI for Data Science, pages 195–211. find
27. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/zhang-2024-can-your-model-tell-negation/