Towards Universal Dialogue Act Tagging for Task-Oriented Dialogues

What it is about

Task-oriented dialogue corpora (Google simulated dialogues for movies and restaurants, DSTC2, MultiWOZ 2.0) each use their own act names, and even shared names mean different things. The authors align them into one «universal» schema of 20 system and user acts, first by hand, then by merging or splitting acts that confused the tagger but can be restored from other information (offer/select merged, user/system request merged, «affirm with a value» split into affirm + inform, «anything else?» variants merged). Each step is measured: cross-dataset F1 rises from 0.44 to 0.56 while within-dataset F1 stays about 0.90. A hierarchical LSTM tagger trained on the aligned human-machine data reaches 0.54 F1 on human-human MultiWOZ turns; self-training on unlabelled MultiWOZ raises it to 0.58, which would otherwise take about 1,700 hand-labelled turns. Keyword rules (bye, thank you) were needed to repair MultiWOZ's own inconsistent labels.

Ideas

• I1. Merge in the model, split by rule: when two labels differ by something a deterministic rule can restore, train the classifier on the merged label and restore the split afterwards. For Mova the clearest case is narrate/assert: the teacher changed 13,316 labels to narrate and 23,371 to assert, and the border is largely voice, tense and genre, which prag already sees. Train on a merged «inform» and restore with a rule (narrator voice + past event frame → narrate); compare act macro-F1 and agreement with the teacher against the current 16-way model.
• I2. Measure each label merge or split the way the paper does, one step at a time, with cross-genre and within-genre scores side by side (tales vs Tatoeba vs lessons in prag eval), so a change that helps one genre and hurts another is seen.
• I3. Self-training on unlabelled in-domain sentences, but only those where the perceptron, IGTree and k-NN agree (84.5% accurate on act), as cheap extra training data for a new genre (lessons). Measure on the held-out silver of that genre, with the agreement audit from prag — Mova pragmatics: LLM silver and a transparent SLM (a random sample of agreed sentences to the teacher) as the guard against learning shared errors.

For Mova

31. Speech acts §7.3 (Mova's acts against the schemes) and §9 (how Mova labels acts; the assert/narrate border). Code: prag/src/schema.rs, prag/src/agree.rs (restoring rules), prag/src/model.rs, prag/src/eval.rs. Measures: act macro-F1 per genre, agreement with teacher labels, gain from self-training on a held-out genre.

Topics: Speech act theory

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 1907.03020

@misc{paul2019universal,
  title = {{Towards Universal Dialogue Act Tagging for Task-Oriented Dialogues}},
  author = {Paul, Shachi and Goel, Rahul and Hakkani-Tur, Dilek},
  year = {2019},
  eprint = {1907.03020},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.03020}
}

References

25 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. L. Austin, How to do things with words. press, 1975. Oxford university find
2. A. Stolcke, K. Ries, N. Coccaro, E. Shriberg, R. Bates, D. Jurafsky, P. Taylor, R. Martin, C. V. Ess-Dykema, and M. Meteer, “Dialogue act modeling for automatic tagging and recognition of conversational speech,” Computational linguistics, vol. 26, no. 3, pp. 339–373, 2000. find our card
3. A. H. Anderson, M. Bader, E. G. Bard, E. Boyle, G. Doherty, S. Garrod, S. Isard, J. Kowtko, J. McAllister, J. Miller et al., “The hcrc map task corpus,” Language and speech, vol. 34, no. 4, pp. 351–366, 1991. find
4. M. G. Core and J. Allen, “Coding dialogs with the damsl annotation scheme,” in AAAI fall symposium on communicative action in humans and machines, vol. 56. Boston, MA, 1997. find
5. H. Bunt, “The dit++ taxonomy for functional dialogue markup,” in AAMAS 2009 Workshop, Towards a Standard Markup Language for Embodied Dialogue Acts, 2009, pp. 13–24. find
6. S. Mezza, A. Cervone, G. Tortoreto, E. A. Stepanov, and G. Riccardi, “Iso-standard domain-independent dialogue act tagging for conversational agents,” arXiv preprint arXiv:1806.04327, 2018. ↗ our card
7. S. Young, “Cued standard dialogue acts,” Report, Cambridge University Engineering Department, 14th October, vol. 2007, 2007. find
8. H. Bunt, J. Alexandersson, J. Carletta, J.-W. Choe, A. C. Fang, K. Hasida, K. Lee, V. Petukhova, A. Popescu-Belis, L. Romary et al., “Towards an iso standard for dialogue act annotation,” in Seventh conference on International Language Resources and Evaluation (LREC’10), 2010. find
9. P. Shah, D. Hakkani-Tür, B. Liu, and G. Tur, “Bootstrapping a neural conversational agent with dialogue self-play, crowdsourcing and on-line reinforcement learning,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL/HLT), vol. 3, 2018, pp. 41–51. find
10. M. Gašić, F. Lefevre, F. Jurčı́ček, S. Keizer, F. Mairesse, B. Thomson, K. Yu, and S. Young, “Back-off action selection in summary space-based pomdp dialogue systems,” in IEEE Workshop on Automatic Speech Recognition & Understanding. IEEE, 2009, pp. 456–461. find
11. J. D. Williams, K. Asadi, and G. Zweig, “Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning,” in 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2017. find
12. B. Liu, G. Tur, D. Hakkani-Tür, P. Shah, and L. Heck, “Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL/HLT), 2018. find
13. P. Budzianowski, T.-H. Wen, B.-H. Tseng, I. Casanueva, S. Ultes, . O. Ramadan, and M. Gašić, “Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling,” arXiv preprint arXiv:1810.00278, 2018. ↗
14. B. Krause, M. Damonte, M. Dobre, D. Duma, J. Fainberg, F. Fancellu, E. Kahembwe, J. Cheng, and B. Webber, “Edina: Building an open domain socialbot with self-dialogues,” Alexa Prize Proceedings, 2017. find
15. J. J. Godfrey, E. C. Holliman, and J. McDaniel, “Switchboard: Telephone speech corpus for research and development,” in icassp. IEEE, 1992, pp. 517–520. find
16. M. Henderson, B. Thomson, and J. D. Williams, “The second dialog state tracking challenge,” in Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), 2014, pp. 263–272. find
17. H. Bunt, “The semantics of dialogue acts,” in Proceedings of the Ninth International Conference on Computational Semantics. Association for Computational Linguistics, 2011, pp. 1–13. find
18. J. Ang, Y. Liu, and E. Shriberg, “Automatic dialog act segmentation and classification in multiparty meetings,” in Proceedings.(ICASSP’05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005., vol. 1. IEEE, 2005, pp. I–1061. find
19. M. Zimmermann, “Joint segmentation and classification of dialog acts using conditional random fields,” in Tenth Annual Conference of the International Speech Communication Association, 2009. find
20. G. Ji and J. Bilmes, “Dialog act tagging using graphical models,” in Proceedings.(ICASSP’05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005., vol. 1. IEEE, 2005, pp. I–33. find
21. J. Y. Lee and F. Dernoncourt, “Sequential short-text classification with recurrent and convolutional neural networks,” arXiv preprint arXiv:1603.03827, 2016. ↗
22. Y. Liu, K. Han, Z. Tan, and Y. Lei, “Using context information for dialog act classification in dnn framework,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 2170–2178. find
23. E. Shriberg, R. Dhillon, S. Bhagat, J. Ang, and H. Carvey, “The icsi meeting recorder dialog act (mrda) corpus,” in Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004, 2004. find
24. A. Margolis, K. Livescu, and M. Ostendorf, “Domain adaptation with unlabeled data for dialog act tagging,” in Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing. Association for Computational Linguistics, 2010, pp. 45–52. find
25. P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,” Transactions of the Association for Computational Linguistics, vol. 5, pp. 135–146, 2017. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/paul-2019-towards-universal-dialogue-act-tagging/