Training LLMs to Recognize Hedges in Dialogues about Roadrunner Cartoons

What it is about

The authors annotate hedge tokens in 63 spoken retellings of a Roadrunner cartoon (5,508 lines, about a quarter with a hedge; inter-coder kappa 0.985). Each speaker retold the story three times: to a new listener, to the same listener again, and to a second new listener. Their coding scheme covers hedging like, you know and just; approximators (kind of, about); proxies for details the speaker cannot recall (thing, whatever, or something, and stuff); vague suffixes (circley); expressions of doubt (I don't know, maybe, what's it called?); and tag questions. Disfluencies (uh, um) are deliberately not hedges. Hedges are most frequent in the first telling and least frequent when the story is retold to the same listener: they track shared knowledge. A fine-tuned BERT BIO tagger reaches F1 0.91; few-shot GPT-4o gets 0.71 and over-predicts, often calling fillers hedges. A third of BERT's "errors" turned out to be errors in the gold annotation, which was then corrected.

Ideas

• I1. Add the proxy and vagueness class to prag/data/cues.tsv as hedges: or something, or whatever, and stuff, and everything, whatchamacallit, thingy, and what's it called. These are common in retold tales and dialogue. Measure the hedge recall gain on the silver in prag eval.
• I2. Disambiguate like with the tree before counting it: case/mark like with a nominal or clause («it's like an open elevator») is a comparison; like as discourse or as an adverb before a verb or number («he like went over») is a hedge. The same applies to just («only» vs softener). Use rules in prag/src/feats.rs, and keep the ambiguous just out of the cue list until a per-cue precision check passes.
• I3. Keep fillers (uh, um) out of hedge and under intj/disfluency. The paper shows that a model conflating them loses precision.
• I4. Use SLM disagreement to audit the silver labels, as BERT exposed gold errors here. The prag "agreement audit" already plans this: sample hedge sentences where perceptron and IGTree disagree with the silver, and count how many silver labels are wrong.
• I5. The new-vs-old-listener effect gives a testable discourse prediction for chapter 38: in tales, a repeated description of the same referent should carry fewer hedges than the first one. Measure it with coref chains on the silver.

For Mova

32. Politeness and hedging §7.2 (hedging devices), 38. Dialogue, discourse markers and reported speech §5 (fillers vs hedges); prag/data/cues.tsv, prag/src/feats.rs; measure: hedge F1 in prag eval and the per-cue precision table.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.sigdial-1.18

@inproceedings{paige2024training,
  title = {{Training LLMs to Recognize Hedges in Dialogues about Roadrunner Cartoons}},
  author = {Paige, Amie and Soubki, Adil and Murzaku, John and Rambow, Owen and Brennan, Susan E.},
  booktitle = {Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2024},
  doi = {10.18653/v1/2024.sigdial-1.18},
  url = {https://doi.org/10.18653/v1/2024.sigdial-1.18}
}

References

38 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alafate Abulimiti, Chloé Clavel, and Justine Cassell. 2023a. When to generate hedges in peer-tutoring interactions. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 572–583, Prague, Czechia. Association for Computational Linguistics. find our card
2. Herbert H. Clark and Susan E. Brennan. 1991. Grounding in communication. In L. B. Resnick, J. Levine, and S. D. Teasley, editors, Perspectives on Socially Shared Cognition, pages 127–149. American Psychological Association. find
3. Alafate Abulimiti, Chloé Clavel, and Justine Cassell. 2023b. How about kind of generating hedges Herbert H. Clark and Richard J. Gerrig. 1990. Quotations as demonstrations. Language, 66:764–805. find
4. Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22(1):1– 39. find
5. Delphine Dahan. 2023. Collaboration under uncertainty in unscripted conversations: The role of hedges. Journal of Experimental Psychology: Learning, Memory, and Cognition, 49:320–335. find
6. Shih-Chieh Dai, Aiping Xiong, and Lun-Wei Ku. 2023. LLM-in-the-loop: Leveraging large language model for thematic analysis. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9993–10001, Singapore. Association for Computational Linguistics. find
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
9. Mona Diab, Lori Levin, Teruko Mitamura, Owen Rambow, Vinodkumar Prabhakaran, and Weiwei Guo. 2009. Committed belief annotation and tagging. In Proceedings of the Third Linguistic Annotation Workshop (LAW III), pages 68–73, Suntec, Singapore. Association for Computational Linguistics. find
10. Bruce Fraser. 2010. Pragmatic competence: The case of hedging. New Approaches to Hedging, 9:15–34. find
11. Alexia Galati and Susan E. Brennan. 2010. Attenuating information in spoken communication: For the speaker, or for the addressee? Journal of Memory and Language, 62:35–51. find
12. Alexia Galati and Susan E. Brennan. 2014. Speakers adapt gestures to addressees’ knowledge: implications for models of co-speech gesture. Language, Cognition and Neuroscience, 29:435 – 451. find
13. Alexia Galati and Susan E. Brennan. 2021. What is retained about common ground? Distinct effects of linguistic and visual co-presence. Cognition, 215. find
14. H Paul Grice. 1957. Meaning. The philosophical review, 66(3):377–388. find
15. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543. ↗
16. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. find
17. Nanjiang Jiang and Marie-Catherine de Marneffe. 2021. He thinks he knows better than the doctors: BERT for event factuality fails on pragmatics. Transactions of the Association for Computational Linguistics, 9:1081–1097. find our card
18. George Lakoff. 1975. Hedges: A study in meaning citeria and the logic of fuzzy concepts. Journal of Philosophical Logic, pages 458–508. find
19. Robin Lakoff. 1973. Language and woman’s place. Language in Society, 2(1):45–79. find
20. Kenton Lee, Yoav Artzi, Yejin Choi, and Luke Zettlemoyer. 2015. Event detection and factuality assessment with non-expert supervision. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1643–1648, Lisbon, Portugal. Association for Computational Linguistics. find
21. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mo-hta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965. find
22. Kris Liu and Jean Fox Tree. 2012. Hedges enhance memory but inhibit retelling. Psychonomic bulletin & review, 19:892–8. find
23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
24. Michael A. Madaio, Justine Cassell, and Amy E. Ogan. 2017. “i think you just got mixed up”: confident peer tutors hedge to support partners’ face needs. International Journal of Computer-Supported Collaborative Learning, 12:401–421. find
25. John Murzaku, Tyler Osborne, Amittai Aviram, and Owen Rambow. 2023. Towards generative event factuality prediction. In Findings of the Association for Computational Linguistics: ACL 2023, pages 701– 715, Toronto, Canada. Association for Computational Linguistics. find
26. John Murzaku, Adil Soubki, and Owen Rambow. 2024. Multimodal belief prediction. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH 2024. International Speech Communication Association. find
27. John Murzaku, Peter Zeng, Magdalena Markowska, and Owen Rambow. 2022. Re-examining FactBank: Predicting the author’s presentation of factuality. In Proceedings of the 29th International Conference on Computational Linguistics, pages 786–796, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. find
28. OpenAI. 2024. Gpt-4o. index/hello-gpt-4o/. https://openai.com/ ↗
29. Amanda Pogue and Michael K. Tanenhaus. 2018. Learning from uncertainty: exploring and manipulating the role of uncertainty on expression production and interpretation. In Proceedings of the 40th Annual Conference of the Cognitive Science Society. find
30. Amir Pouran Ben Veyseh, Thien Huu Nguyen, and Dejing Dou. 2019. Graph based neural networks for event factuality prediction using syntactic and semantic structures. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4393–4399, Florence, Italy. Association for Computational Linguistics. find our card
31. Vinodkumar Prabhakaran, Owen Rambow, and Mona Diab. 2010. Automatic committed belief tagging. In Coling 2010: Posters, pages 1014–1022, Beijing, China. Coling 2010 Organizing Committee. find
32. E. F. Prince, J. Frader, and C. Bosk. 1982. On hedging in physician-physician discourse. In Robert Di Prieto, editor, Linguistics and the Professions, pages 83–97. Albex. find
33. Anna Prokofieva and Julia Hirschberg. 2014. Hedging and speaker commitment. In 5th International Workshop on Emotion, Social Signals, Sentiment & Linked Open Data. find
34. Yann Raphalen, Chloé Clavel, and Justine Cassell. 2022. “You might think about slightly revising the title”: Identifying hedges in peer-tutoring interactions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2160–2174, Dublin, Ireland. Association for Computational Linguistics. find our card
35. Gabriel Stanovsky, Judith Eckle-Kohler, Yevgeniy Puzikov, Ido Dagan, and Iryna Gurevych. 2017. Integrating deep linguistic features in factuality prediction over unified datasets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 352–357, Vancouver, Canada. Association for Computational Linguistics. find
36. Morgan Ulinski, Seth Benjamin, and Julia Hirschberg. 2018. Using hedge detection to improve committed belief tagging. In Proceedings of the Workshop on Computational Semantics beyond Events and Roles, pages 1–5, New Orleans, Louisiana. Association for Computational Linguistics. find our card
37. Morgan Ulinski and Julia Hirschberg. 2019. Crowdsourced hedge term disambiguation. In LAW@ACL. find our card
38. Jiarui Yao, Haoling Qiu, Jin Zhao, Bonan Min, and Nianwen Xue. 2021. Factuality assessment as modal dependency parsing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1540–1550, Online. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/paige-2024-training-llms-recognize-hedges-dialogues/