MIDAS: A Dialog Act Annotation Scheme for Open Domain Human-Machine Spoken Conversations

What it is about

MIDAS is a dialogue act scheme for people talking to a social chatbot, built because a SWBD-DAMSL tagger reached only 47% on such talk (commands are 9% of user turns there, 0.4% in Switchboard). It is a tree with 23 leaf tags in two branches. The semantic branch splits initiative acts (yes/no question, open factual question, open opinion question, command, task command, invalid command) from responsive ones (statement, general opinion, appreciation, complaint, comment, positive/negative/other answer). The functional branch holds incomplete turns (abandoned, nonsense) and social conventions (opening, closing, thanks, apology, back-channel, hold). Tags may co-occur, except opinion with statement and question with answer, and a priority order (question > command > answer > opinion > statement) picks the one the system acts on. Two annotators reached kappa 0.94 on 24K segments; a BERT multi-label tagger with the previous turn as context reaches F1 0.79 (0.71 without context), and the previous act label as context gives high precision but lower recall than the previous text.

Ideas

• I1. Use the MIDAS priority order to pick Mova's single act when cues of several acts fire in one sentence («Thank you, but will you come tomorrow?» → ask over thank), and keep the others as secondary acts when a second slot is added (see bunt-2017-revisiting-iso-standard-dialogue-act I3). A rule in prag/src/agree.rs; measure act accuracy on silver sentences with two or more act cues.
• I2. Split Mova's evaluate the way MIDAS splits opinion: a comment on the previous speaker's claim with nothing new («She's probably right», «How true!») is closer to agree/disagree, while a general opinion brings new content. Silver shows evaluate→assert as the main confusion (139); check how many of those are comments on the previous line by another speaker, using the other-speaker-previous-act slot.
• I3. Test answers whose polarity is not their surface: «Why not?» as a yes, «I'd love to» as yes, «I don't think so» to «Do you want to…?» as a no that also refuses. Add them as minimal-pair tests for the reply reading and the polarity field (grammatical negation is not neg, as the schema already says).
• I4. Feed both the previous act and the previous words as context, not one of them: in MIDAS act-only context traded recall for precision. In prag/src/feats.rs add the previous sentence's first word and end punctuation next to prev; measure act macro-F1 with and without, on the cross-validated silver.
• I5. Separate factual from opinion questions in what the SLM student is asked: an opinion question («What do you think of the fox?») cannot be answered from remembered frames and should be routed or answered with a hedge in world/src/talk.rs. Count how many course questions are of this kind.

For Mova

31. Speech acts §7 (dialogue act schemes; MIDAS can sit next to DAMSL and ISO in the mapping table) and 38-dialogue-and-discourse-markers.md §5, §10. Code: prag/src/agree.rs (priority), prag/src/feats.rs (context slots), prag/src/schema.rs, world/src/talk.rs. Measures: act accuracy on multi-cue sentences, evaluate/assert confusion, act macro-F1 in prag eval.

Topics: Speech act theory

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.eacl-main.94

@inproceedings{yu2021midas,
  title = {{MIDAS: A Dialog Act Annotation Scheme for Open Domain Human-Machine Spoken Conversations}},
  author = {Yu, Dian and Yu, Zhou},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.94},
  url = {https://doi.org/10.18653/v1/2021.eacl-main.94}
}

References

35 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Harry Bunt. 1997. Dynamic interpretation and dialogue theory. find
2. Harry Bunt. 2009. The dit++ taxonomy for functional dialogue markup. In AAMAS 2009 Workshop, Towards a Standard Markup Language for Embodied Dialogue Acts, pages 13–24. find
3. Harry Bunt, Jan Alexandersson, Jean Carletta, Jae-Woong Choe, Alex Chengyu Fang, Koiti Hasida, Kiyong Lee, Volha Petukhova, Andrei Popescu-Belis, Laurent Romary, Claudia Soria, and David Traum. 2010. Towards an ISO standard for dialogue act annotation. In Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC’10), Valletta, Malta. European Languages Resources Association (ELRA). find
4. Harry Bunt, Volha Petukhova, and Alex Fang. 2017. Revisiting the iso standard for dialogue act annotation. In Proceedings 13th Joint ISO - ACL Workshop on Interoperable Semantic Annotation (isa-11), pages 37–50. Research Unit(s) information for this publication is provided by the author(s) concerned. find our card
5. Harry Bunt, Volha Petukhova, Andrei Malchanau, Kars Wijnhoven, and Alex Fang. 2016. The DialogBank. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 3151–3158, Portorož, Slovenia. European Language Resources Association (ELRA). find
6. Chun-Yen Chen, Dian Yu, Weiming Wen, Yi Mang Yang, Jiaping Zhang, Mingyang Zhou, Kevin Jesse, Austin Chau, Antara Bhowmick, Shreenath Iyer, Giritheja Sreenivasulu, Runxiang Cheng, Ashwin Bhandare, and Zhou Yu. 2018. Gunrock: Building a human-like social bot by leveraging large scale real user data. In 2nd Proceedings of Alexa Prize. find
7. Eunah Cho, Kevin Kilgour, Jan Niehues, and Alexander H. Waibel. 2015. Combination of nn and crf models for joint detection of punctuation and disfluencies. In INTERSPEECH. find
8. Shammur Absar Chowdhury, Evgeny Stepanov, and Giuseppe Riccardi. 2016. Transfer of corpus-specific dialogue act annotation to ISO standard: Is it worth it? In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 132–135, Portorož, Slovenia. European Language Resources Association (ELRA). find
9. Mark G. Core and James F. Allen. 1997. Coding dialogs with the damsl annotation scheme. In Working Notes of the AAAI Fall Symposium on Communicative Action in Humans and Machines, pages 28–35. find
10. Cristian Danescu-Niculescu-Mizil and Lillian Lee. 2011. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, ACL 2011. find
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
12. B. Favre, D. Hakkani-Tur, S. Petrov, and D. Klein. 2008. Efficient sentence segmentation using syntactic features. In 2008 IEEE Spoken Language Technology Workshop, pages 77–80. find
13. John J. Godfrey, Edward C. Holliman, and Jane Mc-Daniel. 1992. Switchboard: telephone speech corpus for research and development. In [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 517–520 vol.1. find
14. Jennifer Hill, W. Randolph Ford, and Ingrid G. Farreras. 2015. Real conversations with artificial intelligence: A comparison between human–human online conversations and human–chatbot conversations. Computers in Human Behavior, 49:245 – 250. find
15. Dan Jurafsky, Liz Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual. Technical Report Draft 13, University of Colorado, Institute of Cognitive Science. find
16. Ioannis Katakis, Grigorios Tsoumakas, and Ioannis Vlahavas. 2008. Multilabel text classification for automated tag suggestion. In Proceedings of the ECML/PKDD 2008 Discovery Challenge. find
17. C. Khatri, R. Goel, B. Hedayatnia, A. Metanillou, A. Venkatesh, R. Gabriel, and A. Mandal. 2018. Contextual topic modeling for dialog systems. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 892–899. find
18. Diederik P. Kingma and Jimmy Ba. 2014. A method for stochastic optimization. abs/1412.6980. Adam: CoRR, O. Klejch, P. Bell, and S. Renals. 2017. Sequence-to-sequence models for punctuated transcription combining lexical and acoustic features. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5700–5704. ↗
19. Tina Klüwer. 2011. “i like your shirt” - dialogue acts for enabling social talk in conversational agents. In Intelligent Virtual Agents, pages 14–27, Berlin, Heidelberg. Springer Berlin Heidelberg. find
20. Yang Liu, Kun Han, Zhao Tan, and Yun Lei. 2017. Using context information for dialog act classification in dnn framework. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2170–2178. Association for Computational Linguistics. find
21. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal. Association for Computational Linguistics. find
22. Stefano Mezza, Alessandra Cervone, Evgeny Stepanov, Giuliano Tortoreto, and Giuseppe Riccardi. 2018. Iso-standard domain-independent dialogue act tagging for conversational agents. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3539–3551. Association for Computational Linguistics. find our card
23. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). find
24. Stephan Peitz, Markus Freitag, Arne Mauser, and Hermann Ney. 2011. Modeling punctuation prediction as machine translation. In IWSLT. find
25. Eugénio Ribeiro, Ricardo Ribeiro, and David Martins de Matos. 2015. The influence of context on dialogue act recognition. CoRR, abs/1506.00839. ↗
26. Lina M. Rojas-Barahona, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, Stefan Ultes, Tsung-Hsien Wen, and Steve Young. 2016. Exploiting sentence and context representations in deep neural models for spoken language understanding. In Proceedings of COL-ING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 258–267, Osaka, Japan. find
27. The COLING 2016 Organizing Committee. find
28. Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004, pages 97–100, Cambridge, Massachusetts, USA. Association for Computational Linguistics. find
29. Aditya Siddhant, Anuj Goyal, and Angeliki Metallinou. 2019. Unsupervised transfer learning for spoken language understanding in intelligent agents. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):4959–4966. find
30. Andreas Stolcke, Noah Coccaro, Rebecca Bates, Paul Taylor, Carol Van Ess-Dykema, Klaus Ries, Elizabeth Shriberg, Daniel Jurafsky, Rachel Martin, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Comput. Linguist., 26(3):339–373. find our card
31. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104–3112. Curran Associates, Inc. find
32. Henry S. Thompson, Anne Anderson, Ellen Gurman Bard, Gwyneth Doherty-Sneddon, Alison Newlands, and Cathy Sotillo. 1993. The hcrc map task corpus: Natural dialogue for speech recognition. In Proceedings of the Workshop on Human Language Technology, HLT ’93, pages 25–30, Stroudsburg, PA, USA. Association for Computational Linguistics. find
33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. Curran Associates, Inc. find
34. Marilyn Walker and Rebecca Passonneau. 2001. DATE: A dialogue act tagging scheme for evaluation of spoken dialogue systems. In Proceedings of the First International Conference on Human Language Technology Research. find
35. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/yu-2021-midas-dialog-act-annotation-scheme/