Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech

What it is about

The paper labels each utterance of spontaneous telephone speech with one of 42 dialogue acts (statement, opinion, yes-no question, backchannel, agreement, answer and others), and asks how much the order of the acts helps. The data are 1,155 Switchboard conversations, hand-labelled by eight graduate students (about 205,000 utterances, kappa 0.80), with 1,115 conversations for training and 19 for test. The model is a hidden Markov model: the acts are the hidden states, the words (true or recognised) and the prosody give the observations, an act n-gram gives the transitions, and forward-backward gives each utterance's posterior. With true words, accuracy rises from 54.3% with no act grammar to 70.6% with a bigram and 71.0% with a trigram, against 35% chance and 84% human agreement; with recognised words the best is 64.8%. Prosody alone reaches 49.7% with a bigram, and adding it to the recognised words gains only 0.7 points. Act-specific language models in the recogniser cut word error only slightly, and even perfect act labels give 2.2% relative.

Ideas

I1. Test whether an act grammar adds anything beyond the previous-act feature. The SLM's act labeller (prag/src/feats.rs, prag/src/agree.rs) already sees the act of the previous sentence. On school dialogues with gold acts, add an act-bigram prior with forward-backward over each conversation, and compare it with the current labeller on held-out conversations by two-fold cv. The paper's gain over no grammar is large, but the trigram adds only 0.4 points, so a higher order stays out unless cv asks for it.
I2. Shallow trees for the confusable pairs, as the paper does for backchannel against agreement (its Figure 3). The labeller's largest confusions (assert against narrate, from the teacher-change counts in 31. Speech acts §9) can get a depth-3 tree over the existing text features (length, the previous act, the question mark, the first word). Each label is printed with the path that produced it. Keep the tree only if it beats the current labeller on those pairs by cv; run twice and compare the outputs.
I3. A negative control for the act sequence: shuffle the acts within each conversation and rerun the bigram. If accuracy does not fall back to the no-grammar level, the gain comes from something other than order, and the claim is not made.
I4. Adjacency checks in the dialogue layer (world/src/talk.rs). The paper's bigram counts give expected next acts: about 30% of yes-no questions are followed by a yes answer and 14% by a no answer, commands are followed by agreement in 23% of cases, and statements draw backchannels in 26%. Log each turn that breaks a high-probability pair together with the rule it broke. This is a transparent check, not a learned answer.

For Mova

The chapters are 31. Speech acts (§7 schemes, §9 how Mova labels acts, §10 prosody absent from text) and 38. Dialogue, discourse markers and reported speech. The code is prag/src/feats.rs, prag/src/agree.rs, prag/src/schema.rs (the act enum) and world/src/talk.rs. Measures: two-fold cv on school dialogues, two identical runs, and the shuffled-order control in I3. Licence: the paper is CC BY-NC-SA, so this card is a summary only. The Switchboard transcripts are distributed by the LDC, and the paper does not state their licence, so any Switchboard data is for tests only. The 42-label set is a design reference; Mova's own act set is what goes into a release. The prosodic ideas do not apply to text, so I2 uses text features only.

Topics: Speech act theory

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv cs/0006023

@misc{stolcke2000dialogue,
  title = {{Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech}},
  author = {Stolcke, Andreas and Ries, Klaus and Coccaro, Noah and Shriberg, Elizabeth and Bates, Rebecca and Jurafsky, Daniel and Taylor, Paul and Martin, Rachel and Ess-Dykema, Carol Van and Meteer, Marie},
  year = {2000},
  eprint = {cs/0006023},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/cs/0006023}
}

References

81 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Alexandersson, Jan and Norbert Reithinger. 1997. Learning dialogue structures from a corpus. In G. Kokkinakis, N. Fakotakis, and E. Dermatas, editors, Proceedings of the 5th European Conference on Speech Communication and Technology, volume 4, pages 2,231-2,234. Rhodes, Greece, September. find
Anderson, Anne H., Miles Bader, Ellen G. Bard, Elizabeth H. Boyle, Gwyneth M. Doherty, Simon C. Garrod, Stephen D. Isard, Jacqueline C. Kowtko, Jan M. McAllister, Jim Miller, Catherine F. Sotillo, Henry S. Thompson, and Regina Weinert. 1991. The HCRC Map Task corpus. Language and Speech, 34(4):351-366. find
Austin, J. L. 1962. How to do Things with Words. Clarendon Press, Oxford. find
Bahl, Lalit R., Frederick Jelinek, and Robert L. Mercer. 1983. A maximum likelihood approach to continuous speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 5(2):179-190, March. find
Bard, Ellen G., Catherine Sotillo, Anne H. Anderson, and M. M. Taylor. 1995. The DCIEM Map Task corpus: Spontaneous dialogues under sleep deprivation and drug treatment. In Isabel Trancoso and Roger Moore, editors, Proceedings of the ESCA-NATO Tutorial and Workshop on Speech under Stress, pages 25-28, Lisbon, September. find
Baum, Leonard E., Ted Petrie, George Soules, and Norman Weiss. 1970. A maximization technique occurring in the statistical analysis of probabilistic functions in Markov chains. The Annals of Mathematical Statistics, 41(1):164-171. find
Berger, Adam L., Stephen A. Della Pietra, and Vincent J. Della Pietra. 1996. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39-71. find
Bourlard, Herv6 and Nelson Morgan. 1993. Connectionist Speech Recognition. A Hybrid Approach. Kluwer Academic Publishers, Boston, MA. find
Breiman, L., J. H. Friedman, R. A. Olshen, and C. J. Stone. 1984. Classification and Regression Trees. Wadsworth and Brooks, Pacific Grove, CA. find
Bridle, J. S. 1990. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In F. Fogleman Soulie and J. Herault, editors, Neurocomputing: Algorithms, Architectures and Applications. Springer, Berlin, pages 227-236. find
Brill, Eric. 1993. Automatic grammar induction and parsing free text: A transformation-based approach. In Proceedings of the ARPA Workshop on Human Language Technology, Plainsboro, NJ, March. find
Carletta, Jean. 1996. Assessing agreement on classification tasks: The Kappa statistic. Computational Linguistics, 22(2):249-254. find
Carlson, Lari. 1983. Dialogue Games: An Approach to Discourse Analysis. D. Reidel. find
Chu-Carroll, Jennifer. 1998. A statistical model for discourse act recognition in dialogue interactions. In Jennifer Chu-Carroll and Nancy Green, editors, Applying Machine Learning to Discourse Processing. Papers from the 1998 AAAI Spring Symposium. Technical Report SS-98-01, pages 12-17. AAAI Press, Menlo Park, CA. find
Church, Kenneth Ward. 1988. A stochastic parts program and noun phrase parser for unrestricted text. In Second Conference on Applied Natural Language Processing, pages 136-143, Austin, TX. find
Core, Mark and James Allen. 1997. Coding dialogs with the DAMSL annotation scheme. In Working Notes of the AAAI Fall Symposium on Communicative Action in Humans and Machines, pages 28-35, Cambridge, MA, November. find
Dermatas, Evangelos and George Kokkinakis. 1995. Automatic stochastic tagging of natural language texts. Computational Linguistics, 21(2):137-163. find
Eckert, Wieland, Florian Gallwitz, and Heinrich Niemann. 1996. Combining stochastic and linguistic language models for recognition of spontaneous speech. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 423-426, Atlanta, GA, May. find
Finke, Michael, Maria Lapata, Alon Lavie, Lori Levin, Laura Mayfield Tomokiyo, Thomas Polzin, Klaus Ries, Alex Waibel, and Klaus Zechner. 1998. Clarity: Inferring discourse structure from speech. In Jennifer Chu-Carroll and Nancy Green, editors, Applying Machine Learning to Discourse Processing. Papers from the 1998 AAAI Spring Symposium. Technical Report SS-98-01, pages 25-32. AAAI Press, Menlo Park, CA. find
Fowler, Carol A. and Jonathan Housum. 1987. Talkers' signaling of "new" and "old" words in speech and listeners' perception and use of the distinction. Journal of Memory and Language, 26:489-504. find
Fukada, Toshiaki, Detlef Koll, Alex Waibel, and Kouichi Tanigaki. 1998. Probabilistic dialogue act extraction for concept based multilingual translation systems. In Robert H. Mannell and Jordi Robert-Ribes, editors, Proceedings of the International Conference on Spoken Language Processing, volume 6, pages 2,771-2,774, Sydney, December. Australian Speech Science and Technology Association. find
Godfrey, J. J., E. C. Holliman, and J. McDaniel. 1992. SWITCHBOARD: Telephone speech corpus for research and development. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 517-520, San Francisco, CA, March. find
Grosz, Barbara J. and Candace L. Sidner. 1986. Attention, intention, and the structure of discourse. Computational Linguistics, 12(3):175-204. find
Hirschberg, Julia B. and Diane J. Litman. 1993. Empirical studies on the disambiguation of cue phrases. Computational Linguistics, 19(3):501-530. find
Iyer, Rukmini, Mari Ostendorf, and J. Robin Rohlicek. 1994. Language modeling with sentence-level mixtures. In Proceedings of the ARPA Workshop on Human Language Technology, pages 82-86, Plainsboro, NJ, March. find
Jefferson, Gail. 1984. Notes on a systematic deployment of the acknowledgement tokens 'yeah' and ' m m hm'. Papers in Linguistics, 17:197-216. find
Jekat, Susanne, Alexandra Klein, Elisabeth Maier, Ilona Maleck, Marion Mast, and Joachim Quantz. 1995. Dialogue acts in VERBMOBIL. Verbmobil-Report 65, Universit~it Hamburg, DFKI GmbH, Universit~it Erlangen, and TU Berlin, April. find
Jurafsky, Dan, Rebecca Bates, Noah Coccaro, Rachel Martin, Marie Meteer, Klaus Ries, Elizabeth Shriberg, Andreas Stolcke, Paul Taylor, and Carol Van Ess-Dykema. 1997. Automatic detection of discourse structure for speech recognition and understanding. In Proceedings of the IEEE Workshop on Speech Recognition and Understanding, pages 88-95, Santa Barbara, CA, December. find
Jurafsky, Daniel, Rebecca Bates, Noah Coccaro, Rachel Martin, Marie Meteer, Klaus Ries, Elizabeth Shriberg, Andreas Stolcke, Paul Taylor, and Carol Van Ess-Dykema. 1998. Switchboard discourse language modeling project final report. Research Note 30, Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, January. find
Jurafsky, Daniel, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard-DAMSL Labeling Project Coder's Manual. Technical Report 97-02, University of Colorado, Institute of Cognitive Science, Boulder, CO. http://www.colorado.edu/ling/jurafsky/manual.augustl.html. ↗
Jurafsky, Daniel, Elizabeth E. Shriberg, Barbara Fox, and Traci Curl. 1998. Lexical, prosodic, and syntactic cues for dialog acts. In Proceedings of ACL/COLING-98 Workshop on Discourse Relations and Discourse Markers, pages 114-120. Association for Computational Linguistics. find
Katz, Slava M. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, 35(3):400-401, March. find
Kita, Kenji, Yoshikazu Fukui, Masaaki Nagata, and Tsuyoshi Morimoto. 1996. Automatic acquisition of probabilistic dialogue models. In H. Timothy Bunnell and William Idsardi, editors, Proceedings of the International Conference on Spoken Language Processing, volume 1, pages 196-199, Philadelphia, PA, October. find
Kompe, Ralf. 1997. Prosody in speech understanding systems. Springer, Berlin. find
Kowtko, Jacqueline C. 1996. The Function of Intonation in Task Oriented Dialogue. Ph.D. thesis, University of Edinburgh, Edinburgh. find
Kuhn, Roland and Renato de Mori. 1990. A cache-base natural language model for speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(6):570-583, June. find
Levin, Joan A. and Johanna A. Moore. 1977. Dialogue games: Metacommunication structures for natural language interaction. Cognitive Science, 1(4):395-420. find
Levin, Lori, Klaus Ries, Ann Thym~-Gobbel, and Alon Lavie. 1999. Tagging of speech acts and dialogue games in Spanish CallHome. In Towards Standards and Tools for Discourse Tagging (Proceedings of the ACL Workshop at ACL'99), pages 42-47, College Park, MD, June. find
Linell, Per. 1990. The power of dialogue dynamics. In Ivana Markov~ and Klaus Foppa, editors, The Dynamics of Dialogue. Harvester, Wheatsheaf, New York, London, pages 147-177. find
Mast, M., R. Kompe, S. Harbeck, A. Kiel~ling, H. Niemann, E. NOth, E. G. Schukat-Talamazzini, and V. Warnke. 1996. Dialog act classification with the help of prosody. In H. Timothy Bunnell and William Idsardi, editors, Proceedings of the International Conference on Spoken Language Processing, volume 3, pages 1,732-1,735, Philadelphia, PA, October. find
Menn, Lise and Suzanne E. Boyce. 1982. Fundamental frequency and discourse structure. Language and Speech, 25:341-383. find
Meteer, Marie, Ann Taylor, Robert MacIntyre, and Rukmini Iyer. 1995. Dysfluency annotation stylebook for the Switchboard corpus. Distributed by LDC, ftp://ftp.cis.upenn.edu/pub/treebank/ swbd/doc/DFL-book.ps, February. find
Revised June 1995 by Ann Taylor. find
Morgan, Nelson, Eric Fosler, and Nikki Mirghafori. 1997. Speech recognition using on-line estimation of speaking rate. In G. Kokkinakis, N. Fakotakis, and E. Dermatas, editors, Proceedings of the 5th European Conference on Speech Communication and Technology, volume 4, pages 2,079-2,082, Rhodes, Greece, September. find
Munk, Marcus. 1999. Shallow Statistical Parsing for Machine Translation. Diploma thesis, Carnegie Mellon University. find
Nagata, Masaaki. 1992. Using pragmatics to rule out recognition errors in cooperative task-oriented dialogues. In John J. Ohala, Terrance M. Nearey, Bruce L. Derwing, Megan M. Hodge, and Grace E. Wiebe, editors, Proceedings of the International Conference on Spoken Language Processing, volume 1, pages 647-650, Banff, Canada, October. find
Nagata, Masaaki and Tsuyoshi Morimoto. 1993. An experimental statistical dialogue model to predict the speech act type of the next utterance. In Katsuhiko Shirai, Tetsunori Kobayashi, and Yasunari Harada, editors, Proceedings of the International Symposium on Spoken Dialogue, pages 83-86, Tokyo, November. find
Nagata, Masaaki and Tsuyoshi Morimoto. 1994. First steps toward statistical modeling of dialogue to predict the speech act type of the next utterance. Speech Communication, 15:193-203. find
Ohler, Uwe, Stefan Harbeck, and Heinrich Niemann. 1999. Discriminative training of language model classifiers. In Proceedings of the 6th European Conference on Speech Communication and Technology, volume 4, pages 1607-1610, Budapest, September. find
Pearl, Judea. 1988. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Mateo, CA. find
Power, Richard J. D. 1979. The organization of purposeful dialogues. Linguistics, 17:107-152. find
Rabiner, L. R. and B. H. Juang. 1986. An introduction to hidden Markov models. IEEE ASSP Magazine, 3(1):4-16, January. find
Reithinger, Norbert, Ralf Engel, Michael Kipp, and Martin Klesen. 1996. Predicting dialogue acts for a speech-to-speech translation system. In H. Timothy Bunnell and William Idsardi, editors, Proceedings of the International Conference on Spoken Language Processing, volume 2, pages 654-657, Philadelphia, PA, October. find
Reithinger, Norbert and Martin Klesen. 1997. Dialogue act classification using language models. In G. Kokkinakis, N. Fakotakis, and E. Dermatas, editors, Proceedings of the 5th European Conference on Speech Communication and Technology, volume 4, pages 2,235-2,238, Rhodes, Greece, September. find
Ries, Klaus. 1999a. HMM and neural network based speech act classification. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 497-500, Phoenix, AZ, March. find
Ries, Klaus. 1999b. Towards the detection and description of textual meaning indicators in spontaneous conversations. In Proceedings of the 6th European Conference on Speech Communication and Technology, volume 3, pages 1,415--1,418, Budapest, September. find
Rose, R. C., E. I. Chang, and R. P. Lippmann. 1991. Techniques for information retrieval from voice messages. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 317-320, Toronto, May. find
Sacks, H., E. A. Schegloff, and G. Jefferson. 1974. A simplest semantics for the organization of turn-taking in conversation. Language, 50(4):696-735. find
Samuel, Ken, Sandra Carberry, and K. Vijay-Shanker. 1998. Dialogue act tagging with transformation-based learning. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, volume 2, pages 1,150-1,156, Montreal. find
Schegloff, Emanuel A. 1968. Sequencing in conversational openings. American Anthropologist, 70:1,075-1,095. find
Schegloff, Emanuel A. 1982. Discourse as an interactional achievement: Some uses of 'uh huh' and other things that come between sentences. In Deborah Tannen, editor, Analyzing Discourse: Text and Talk. Georgetown University Press, Washington, D.C., pages 71-93. find
Searle, J. R. 1969. Speech Acts. Cambridge University Press, London-New York. find our card
Shriberg, Elizabeth, Rebecca Bates, Andreas Stolcke, Paul Taylor, Daniel Jurafsky, Klaus Ries, Noah Coccaro, Rachel Martin, Marie Meteer, and Carol Van Ess-Dykema. 1998. Can prosody aid the automatic classification of dialog acts in conversational speech? Language and Speech, 41(3-4):439--487. find
Shriberg, Elizabeth, Andreas Stolcke, Dilek Hakkani-Ti~r, and GOkhan Tiir. 2000. Prosody-based automatic segmentation of speech into sentences and topics. Speech Communication, 32(1-2). Special Issue on Accessing Information in Spoken Audio. To appear. find
Siegel, Sidney and N. John Castellan, Jr. 1988. Nonparametric Statistics for the Behavioral Sciences. Second edition. McGraw-Hill, New York. find
Stolcke, Andreas and Elizabeth Shriberg. 1996. Automatic linguistic segmentation of conversational speech. In H. Timothy Bunnell and William Idsardi, editors, Proceedings of the International Conference on Spoken Language Processing, volume 2, pages 1,005-1,008, Philadelphia, PA, October. find
Suhm, B. and A. Waibel. 1994. Toward better language models for spontaneous speech. In Proceedings of the International Conference on Spoken Language Processing, volume 2, pages 831-834, Yokohama, September. find
Taylor, Paul A. 2000. Analysis and synthesis of intonation using the tilt model. Journal of the Acoustical Society of America, 107(3):1,697-1,714. find
Taylor, Paul A., Simon King, Stephen Isard, and Helen Wright. 1998. Intonation and dialog context as constraints for speech recognition. Language and Speech, 41(3-4):489-508. find
Taylor, Paul A., Simon King, Stephen Isard, Helen Wright, and Jacqueline Kowtko. 1997. Using intonation to constrain language models in speech recognition. In G. Kokkinakis, N. Fakotakis, and E. Dermatas, editors, Proceedings of the 5th European Conference on Speech Communication and Technology, volume 5, pages 2,763-2,766, Rhodes, Greece, September. find
Van Ess-Dykema, Carol and Klaus Ries. 1998. Linguistically engineered tools for speech recognition error analysis. In Robert H. Mannell and Jordi Robert-Ribes, editors, Proceedings of the International Conference on Spoken Language Processing, volume 5, pages 2,091-2,094, Sydney, December. Australian Speech Science and Technology Association. find
Viterbi, A. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 13:260-269. find
Warnke, Volker, Stefan Harbeck, Elmar N0th, Heinrich Niemann, and Michael Levit. 1999. Discriminative estimation of interpolation parameters for language model classifiers. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 525-528, Phoenix, AZ, March. find
Warnke, Volker, R. Kompe, Heinrich Niemann, and Elmar NOth. 1997. Integrated dialog act segmentation and classification using prosodic features and language models. In G. Kokkinakis, N. Fakotakis, and E. Dermatas, editors, Proceedings of the 5th European Conference on Speech Communication and Technology, volume 1, pages 207-210, Rhodes, Greece, September. find
Weber, Elizabeth G. 1993. Varieties of Questions in English Conversation. John Benjamins, Amsterdam. find
Witten, Ian H. and Timothy C. Bell. 1991. The zero-frequency problem: Estimating the probabilities of novel events in adaptive text compression. IEEE Transations on Information Theory, 37(4):1,085-1,094, July. find
Woszczyna, M. and A. Waibel. 1994. Inferring linguistic structure in spoken language. In Proceedings of the International Conference on Spoken Language Processing, volume 2, pages 847-850, Yokohama, September. find
Wright, Helen. 1998. Automatic utterance type detection using suprasegmental features. In Robert H. Mannell and Jordi Robert-Ribes, editors, Proceedings of the International Conference on Spoken Language Processing, volume 4, pages 1,403-1,406, Sydney, December. Australian Speech Science and Technology Association. find
Wright, Helen and Paul A. Taylor. 1997. Modelling intonational structure using hidden Markov models. In Intonation: Theory, Models and Applications. Proceedings of an ESCA Workshop, pages 333-336, Athens, September. find
Yngve, Victor H. 1970. On getting a word in edgewise. In Papers from the Sixth Regional Meeting of the Chicago Linguistic Society, pages 567-577, Chicago, April. University of Chicago. find
Yoshimura, Takashi, Satoru Hayamizu, Hiroshi Ohmura, and Kazuyo Tanaka. 1996. Pitch pattern clustering of user utterances in human-machine dialogue. In H. Timothy Bunnell and William Idsardi, editors, Proceedings of the International Conference on Spoken Language Processing, volume 2, pages 837-840, Philadelphia, PA, October. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/prag-2000-stolcke-dialogue-act-modeling-automatic-tagging/