Large-scale text pre-training helps with dialogue act recognition, but not without fine-tuning

What it is about

A BERT utterance encoder with a small RNN over the dialogue is tested on dialogue act recognition in Switchboard (42 tags) and the AMI meetings (16 tags), comparing a frozen pre-trained encoder, a fine-tuned one, and a randomly initialised one trained on the task. Fine-tuning is essential: the frozen encoder reaches 55.6% accuracy / 7.8 macro-F1 on Switchboard and predicts one of the two commonest tags 86% of the time (they make up 51% of the data), while the fine-tuned one reaches 76.6% / 36.8 and the randomly initialised one 73.8% / 32.2. Further pre-training on dialogue (in-domain, cross-domain, or 350M tokens of subtitles) adds a little accuracy and more macro-F1 (up to 47.8 on Switchboard). Pre-training helps mostly on rare tags: where it wins, the rarest word of the utterance is about twice as rare as usual.

Ideas

• I1. The rare-word finding transfers to Mova without any neural model: rare acts are carried by rare words. Back off rare words to a class slot in prag/src/feats.rs (the VerbNet class of the root verb from prag/data/verbnet.tsv, or the cue class from prag/data/cues.tsv) so that «I beseech you», «I vow» or «I entreat you» share evidence with their common synonyms. Measure macro-F1 on the rare acts (promise, warn, offer, suggest) in prag eval.
• I2. Keep reporting macro-F1 next to accuracy, and add the paper's bias check: the share of predictions falling on the two commonest acts against their share in the silver. A model that over-predicts assert/narrate shows it here even when accuracy looks fine.

For Mova

31. Speech acts §9 (how Mova labels acts). Code: prag/src/feats.rs, prag/data/verbnet.tsv, prag/data/cues.tsv, prag/src/eval.rs. Measures: rare-act macro-F1 and the majority-bias share in prag eval.

Topics: Speech act theory

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{noble2021large,
  title = {{Large-scale text pre-training helps with dialogue act recognition, but not without fine-tuning}},
  author = {Noble, Bill and Maraev, Vladislav},
  booktitle = {Proceedings of the 14th International Conference on Computational Semantics (IWCS)},
  year = {2021},
  url = {https://aclanthology.org/2021.iwcs-1.16/}
}

References

25 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. John L. Austin and James O. Urmson. 2009. How to Do Things with Words: The William James Lectures Delivered at Harvard University in 1955, 2. ed., [repr.] edition. Harvard Univ. Press, Cambridge, Mass. OCLC: 935786421. find
2. Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2020. PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 85–96, Online. Association for Computational Linguistics. find
3. Chandrakant Bothe, Sven Magg, Cornelius Weber, and Stefan Wermter. 2018a. Conversational analysis using utterance-level attention-based bidirectional recurrent neural networks. Proc. Interspeech 2018, pages 996–1000. find
4. Chandrakant Bothe, Cornelius Weber, Sven Magg, and Stefan Wermter. 2018b. A Context-based Approach for Dialogue Act Recognition using Simple Recurrent Neural Networks. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
5. Jean Carletta. 2007. Unleashing the killer corpus: Experiences in creating the multi-everything AMI Meeting Corpus. Language Resources and Evaluation, 41(2):181–190. for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
6. Daniel Jurafsky, Liz Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL Shallow-Discourse-Function Annotation Coders Manual. find
7. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Convolutional Neural Networks for Discourse Compositionality. In Proceedings of the Workshop on Continuous Vector Space Models and Their Compositionality, pages 119–126. find
8. Zornitsa Kozareva and Sujith Ravi. 2019. ProSeqo: Projection Sequence Networks for On-Device Text Classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3894–3903, Hong Kong, China. Association for Computational Linguistics. find
9. Pierre Lison and Jorg Tiedemann. 2016. OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation, page 7. find
10. Christophe Cerisara, Pavel Král, and Ladislav Lenc. 2017. On the effects of using word2vec representations in neural networks for dialogue act recognition. Computer Speech & Language, 47:175–193. ↗
11. Shikib Mehri, Evgeniia Razumovskaia, Tiancheng Zhao, and Maxine Eskenazi. 2019. Pretraining Methods for Dialog Context Representation Learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3836–3845, Florence, Italy. Association for Computational Linguistics. find
12. Saurabh Chakravarty, Raja Venkata Satya Phanindra Chava, and Edward A Fox. 2019. Dialog acts classification for question-answer corpora. In ASAIL@ ICAIL. find
13. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. NIPS Proceedings, page 9. ↗
14. Wenhu Chen, Jianshu Chen, Pengda Qin, Xifeng Yan, and William Yang Wang. 2019. Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3696–3709, Florence, Italy. Association for Computational Linguistics. find
15. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗
16. Zheqian Chen, Rongqin Yang, Zhou Zhao, Deng Cai, and Xiaofei He. 2018. Dialogue Act Recognition via CRF-Attentive Structured Network. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, pages 225–234, New York, NY, USA. Association for Computing Machinery. find
17. Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 7–14, Florence, Italy. Association for Computational Linguistics. find
18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association Eugénio Ribeiro, Ricardo Ribeiro, and David Martins de Matos. 2019. Deep dialog act recognition using multiple token, segment, and context information representations. Journal of Artificial Intelligence Research, 66:861–899. find
19. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech. Computational Linguistics, 26(3):339–373. ↗ our card
20. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to Fine-Tune BERT for Text Classification? In Chinese Computational Linguistics, Lecture Notes in Computer Science, pages 194–206, Cham. Springer International Publishing. ↗
21. Quan Hung Tran, Ingrid Zukerman, and Gholamreza Haffari. 2017a. A Hierarchical Neural Model for Learning Sequences of Dialogue Acts. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 428–437, Valencia, Spain. Association for Computational Linguistics. find
22. Quan Hung Tran, Ingrid Zukerman, and Gholamreza Haffari. 2017b. Preserving Distributional Information in Dialogue Act Classification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2151–2156, Copenhagen, Denmark. Association for Computational Linguistics. find
23. Jesse Vig and Kalai Ramea. 2019. Comparison of Transfer-Learning Approaches for Response Selection in Multi-Turn Conversations. In Proceedings of the Workshop on Dialog System Technology Challenges, page 7, Honolulu, Hawaii. find
24. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find
25. Tianyu Zhao and Tatsuya Kawahara. 2018. A unified neural architecture for joint dialog act segmentation and recognition in spoken dialog system. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 201–208. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/noble-2021-large-scale-text-pre-training/