Do LLMs Understand Dialogues? A Case Study on Dialogue Acts

What it is about

On fine-grained dialogue act classification (MRDA meetings, 50 classes; Switchboard, 43) prompted LLMs do far worse than small fine-tuned models: LLaMA-3.1-70B few-shot reaches 19 macro-F1 / 31% accuracy on MRDA against 40 / 66% for a RoBERTa model with speaker-turn modelling, and GPT-4 with audio only 28 / 42%. The authors split the task into three «pre-tasks» derived from existing labels: turn management (new floor, floor continuation, no floor), communicative function (forward- vs backward-looking), and dialogue structure (which earlier utterances form an adjacency pair with this one). On each, the LLM does no better or worse than a trivial rule: a speaker-change rule plus two backchannel words beats it on turns, a short-utterance list beats it on function, and «link to the previous utterance» nearly matches it on structure. The LLM over-predicts backward-looking functions and prefers near utterances. Errors on the pre-tasks are significantly associated with act errors, and untrained humans agree well on the pre-tasks (kappa 0.65–0.67) where the LLMs reach 0.36–0.50.

Ideas

• I1. This bears on Mova's teacher: Opus labels act in the silver, and these results say a large model is weakest exactly on context-dependent acts (agree, disagree, accept, backchannel-like replies). Hand-label a small gold sample of tale dialogue lines (a few hundred, rare acts over-sampled) and measure the teacher's per-act agreement with it before trusting silver for rare acts. Report it next to the SLM figures in prag — Mova pragmatics: LLM silver and a transparent SLM.
• I2. Add the pre-tasks as cheap rule features in prag/src/feats.rs: speaker changed (the quote's speaker differs from the previous quote's speaker; today quote=cont only says the speech continues) and backward-looking cue (the utterance starts with a reply word: yes, no, well, oh, indeed, so, a repeated word from the previous line). Measure act macro-F1 and agree/disagree recall.
• I3. Use the paper's rule baselines as negative-control-style yardsticks in prag eval: a forward/backward rule and a «link to previous line» rule. Any learned model for these must beat them; a model that does not is not learning context.

For Mova

31. Speech acts §9 (how Mova labels acts; teacher changes), 38-dialogue-and-discourse-markers.md §1, §5, §10 (turns, backchannels, dialogue acts). Code: prag/src/feats.rs, prag/src/eval.rs, prag/src/silver.rs. Measures: teacher-vs-gold per-act agreement; act macro-F1 with the new slots; rule baselines.

Topics: Speech act theory

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1271

@inproceedings{qamar2025do,
  title = {{Do LLMs Understand Dialogues? A Case Study on Dialogue Acts}},
  author = {Qamar, Ayesha and Tong, Jonathan and Huang, Ruihong},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1271},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1271}
}

References

45 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ali Ahmadvand, Jason Ingyu Choi, and Eugene Agichtein. 2019. Contextual dialogue act classification for open-domain conversational agents. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR’19, page 1273–1276, New York, NY, USA. Association for Computing Machinery. find
2. James Allen and Mark Core. 1997. Draft of damsl: Dialog act markup in several layers. find
3. Gaurav Arora, Shreya Jain, and Srujana Merugu. 2024. Intent detection in the age of LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1559–1570, Miami, Florida, US. Association for Computational Linguistics. find
4. Gemma Bel-Enguix and M Dolores Jiménez-López. 2006. Ambiguous turn-taking games in conversations. In Actes de la 13ème conférence sur le Traitement Automatique des Langues Naturelles. Posters, pages 398–406. find
5. Kristy Boyer, Robert Phillips, Eun Young Ha, Michael Wallis, Mladen Vouk, and James Lester. 2009. Modeling dialogue structure with adjacency pair analysis and hidden markov models. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers, pages 49–52. find
6. David Brazil. 1997. The communicative value of intonation in English book. Cambridge University Press. find
7. H Bunt, J Alexandersson, J-W CHOE, FANG Chengyu, K Hasida, V Petukhova, A Popescu-Belis, and D Traum. 2012. Iso 24617-2: 2012 language resource management–semantic annotation framework (semaf)–part 2: Dialogue acts: 2012 language resource management–semantic annotation framework (semaf)–part 2: Dialogue acts. find
8. Harry Bunt. 1994. Context and dialogue control. Think Quarterly, 3(1):19–31. find
9. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. ↗
10. Mark G Core and James Allen. 1997. Coding dialogs with the damsl annotation scheme. In AAAI fall symposium on communicative action in humans and machines, volume 56, pages 28–35. Boston, MA. find
11. Rajdip Dhillon, Sonali Bhagat, Hannah Carvey, and Elizabeth Shriberg. 2004. Meeting recorder project: Dialog act labeling guide. Technical report, Citeseer. find
12. Margherita Grandini, Enrico Bagli, and Giorgio Visani. 2020. Metrics for multi-class classification: an overview. arXiv preprint arXiv:2008.05756. ↗
13. Zihao He, Leili Tavabi, Kristina Lerman, and Mohammad Soleymani. 2021. Speaker turn modeling for dialogue act classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2150–2157, Punta Cana, Dominican Republic. Association for Computational Linguistics. find our card
14. Michael Heck, Nurul Lubis, Benjamin Ruppik, Renato Vukovic, Shutong Feng, Christian Geishauser, Hsien-chin Lin, Carel van Niekerk, and Milica Gasic. 2023. ChatGPT for zero-shot dialogue state tracking: A solution or an opportunity? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 936–950, Toronto, Canada. Association for Computational Linguistics. find
15. Thomas M Holtgraves. 2013. Language as social action: Social psychology and language use. Psychology Press. find
16. Dan Jurafsky. 1997. Switchboard swbd-damsl shallow-discourse-function annotation coders manual. Institute of Cognitive Science Technical Report. find
17. Dongjin Kang, Sunghwan Kim, Taeyoon Kwon, Seungjun Moon, Hyunsouk Cho, Youngjae Yu, Dongha Lee, and Jinyoung Yeo. 2024. Can large language models be good emotional supporter? mitigating preference bias on emotional support conversation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15232–15261, Bangkok, Thailand. Association for Computational Linguistics. find
18. Taewoon Kim and Piek Vossen. 2021. Emoberta: Speaker-aware emotion recognition in conversation with roberta. arXiv preprint arXiv:2108.12009. ↗
19. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213. find
20. Harshit Kumar, Arvind Agarwal, Riddhiman Dasgupta, and Sachindra Joshi. 2018a. Dialogue act sequence labeling using hierarchical encoder with crf. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18. AAAI Press. find
21. Harshit Kumar, Arvind Agarwal, Riddhiman Dasgupta, and Sachindra Joshi. 2018b. Dialogue act sequence labeling using hierarchical encoder with crf. In Proceedings of the aaai conference on artificial intelligence, volume 32. find
22. Md Tahmid Rahman Laskar, Xue-Yong Fu, Cheng Chen, and Shashi Bhushan TN. 2023. Building real-world meeting summarization systems using large language models: A practical perspective. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 343–352, Singapore. Association for Computational Linguistics. find
23. Stephen C Levinson. 1983. Pragmatics. Cambridge UP. find
24. Guan-Ting Lin and Hung-yi Lee. 2024. Can LLMs understand the implication of emphasized sentences in dialogue? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13391– 13401, Miami, Florida, USA. Association for Computational Linguistics. find
25. Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021. Towards emotional support dialog systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3469–3483, Online. Association for Computational Linguistics. find
26. Yang Liu, Kun Han, Zhao Tan, and Yun Lei. 2017. Using context information for dialog act classification in DNN framework. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2170–2178, Copenhagen, Denmark. Association for Computational Linguistics. find
27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
28. Xiang Luo, Zhiwen Tang, Jin Wang, and Xuejie Zhang. 2024. Zero-shot cross-domain dialogue state tracking via dual low-rank adaptation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5746–5765, Bangkok, Thailand. Association for Computational Linguistics. find
29. Md Messal Monem Miah, Adarsh Pyarelal, and Ruihong Huang. 2023. Hierarchical fusion for online multimodal dialog act classification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7532–7545, Singapore. Association for Computational Linguistics. find
30. Wenbo Pan, Qiguang Chen, Xiao Xu, Wanxiang Che, and Libo Qin. 2023. A preliminary evaluation of chatgpt for zero-shot dialogue understanding. arXiv preprint arXiv:2304.04256. ↗
31. Andrei Popescu-Belis. 2005. Dialogue acts: One or more dimensions. ISSCO WorkingPaper, 62:1–46. find
32. Ayesha Qamar, Adarsh Pyarelal, and Ruihong Huang. 2023. Who is speaking? speaker-aware multiparty dialogue act classification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10122–10135, Singapore. Association for Computational Linguistics. find
33. Vipul Raheja and Joel Tetreault. 2019. Dialogue Act Classification with Context-Aware Self-Attention. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3727–3733, Minneapolis, Minnesota. Association for Computational Linguistics. find
34. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics. find
35. Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. 1974. A simplest systematics for the organization of turn-taking for conversation. language, 50(4):696– 735. find
36. John R Searle. 1969. Speech acts: An essay in the philosophy of language. Cambridge University. find
37. Guokan Shang, Antoine Tixier, Michalis Vazirgiannis, and Jean-Pierre Lorré. 2020. Speaker-change aware CRF for dialogue act classification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 450–464, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
38. Zhouxing Shi and Minlie Huang. 2019. A deep sequential model for discourse parsing on multi-party dialogues. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7007–7014. find
39. Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The icsi meeting recorder dialog act (mrda) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004, pages 97–100. find
40. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373. ↗ our card
41. Zhiqiang Wang, Yiran Pang, and Yanbin Lin. 2023. Large language models are zero-shot text classifiers. arXiv preprint arXiv:2312.01044. ↗
42. Jiabao Xu, Peijie Huang, Youming Peng, Jiande Ding, Boxi Huang, and Simin Huang. 2022. Adjacency pairs-aware hierarchical attention networks for dialogue intent classification. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7622–7626. IEEE. find
43. Piotr Żelasko, Raghavendra Pappagari, and Najim De-hak. 2021. What helps transformers recognize conversational structure? importance of context, punctuation, and labels in dialog act recognition. Transactions of the Association for Computational Linguistics, 9:1163–1179. find
44. Tenggan Zhang, Xinjie Zhang, Jinming Zhao, Li Zhou, and Qin Jin. 2024. ESCoT: Towards interpretable emotional support dialogue systems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13395–13412, Bangkok, Thailand. Association for Computational Linguistics. find
45. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/qamar-2025-do-llms-understand-dialogues-case/