Mova0.5 pre-release⌕
☰
home / docs / papers / English / COMICORDA: Dialogue Act Recognition in Comic Books

COMICORDA: Dialogue Act Recognition in Comic Books

What it is about

The authors build COMICORDA, over 1,400 comic panels (from the public COMICS collection and another source) with speech and narration balloons, corrected text, faces linked to balloons, and a dialogue act for each utterance, annotated by three people after the VERBMOBIL guide with additions: a separate Order class (commands are frequent), questions split into yes/no, wh- and or-questions, and Not_Classifiable for fragments. More than half of the utterances are Inform, because comic speech is short and pushes the plot. The pipeline detects balloons (YOLOv8 is best, mAP 0.90 at IoU 0.5), reads them with OCR (10.3% character error rate), separates narration from speech (BERT 93%, SVM 87%) and tags acts: BERT pre-trained on Switchboard reaches 78.5% accuracy / 52 macro-F1 on clean text and 70% on OCR text (SVM baseline 67%). The authors suggest acts as a cue for ordering balloons when layout alone is ambiguous.

Ideas

For Mova

38. Dialogue, discourse markers and reported speech §8 (dialogue in tales) and §11 (Say frames, measurements), 31-speech-acts.md §8 (speech acts inside narrative). Code: prag/src/frames.rs (quote speakers), prag/src/eval.rs. Measures: frame bench speaker F1; act distribution comparison on character lines.

Topics: Speech act theory

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{martinek2024comicorda,
  title = {{COMICORDA: Dialogue Act Recognition in Comic Books}},
  author = {Martinek, Jiri and Kral, Pavel and Lenc, Ladislav and Baloun, Josef},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  url = {https://aclanthology.org/2024.lrec-main.316/}
}

References

83 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ehsan Ahmadi, Zohreh Azimifar, Maryam Shams, Mahmoud Famouri, and Mohammad Javad Shafiee. 2015. Document image binarization using a discriminative structural classifier. Pattern recognition letters, 63:36–42. find
  2. Jan Alexandersson, Bianka Buschbeck-Wolf, Tsutomu Fujinami, Michael Kipp, Stephan Koch, Elisabeth Maier, Norbert Reithinger, Birte Schmitz, and Melanie Siegel. 1998. Dialogue acts in Verbmobil 2. DFKI Saarbrücken. find
  3. Jan Alexandersson, Bianka Buschbeck-Wolf, Tsutomu Fujinami, Elisabeth Maier, Norbert Reithinger, Birte Schmitz, and Melanie Siegel. 1997. Dialogue acts in verbmobil-2. find
  4. Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817–1853. find
  5. Galen Andrew and Jianfeng Gao. 2007. Scalable training of L 1 -regularized log-linear models. In Proceedings of the 24th International Conference on Machine Learning, pages 33–40. find
  6. Nicolas Audebert, Catherine Herold, Kuider Slimani, and Cédric Vidal. 2019. Multimodal deep networks for text and image-based document classification. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 427–443. Springer. find
  7. José-Miguel Benedı, Eduardo Lleida, Amparo Varona, Marıa-José Castro, Isabel Galiano, Raquel Justo, I López, and Antonio Miguel. 2006. Design and acquisition of a telephone spontaneous speech dialogue corpus in spanish: Dihana. In Fifth International Conference on Language Resources and Evaluation (LREC), pages 1636–1639. find
  8. Taylor Berg-Kirkpatrick, Greg Durrett, and Dan Klein. 2013. Unsupervised transcription of historical documents. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 207–217. find
  9. Harry Bunt. 1994. Context and dialogue control. Think Quarterly, 3(1):19–31. find
  10. S.V. Burtsev and Ye.P. Kuzmin. 1993. An efficient flood-filling algorithm. Computers & Graphics, 17(5):549 – 561. find
  11. C. Cerisara, P. Král, and L. Lenc. 2018a. On the effects of using word2vec representations in neural networks for dialogue act recognition. Computer Speech and Language, 47:175–193. find
  12. Christophe Cerisara, Somayeh Jafaritazehjani, Adedayo Oluokun, and Hoa Le. 2018b. Multi-task dialog act and sentiment recognition on mastodon. arXiv preprint arXiv:1807.05013. ↗
  13. Ilias Chalkidis, Xiang Dai, Manos Fergadiotis, Prodromos Malakasiotis, and Desmond Elliott. 2022. An exploration of hierarchical attention transformers for efficient long document classification. arXiv preprint arXiv:2210.05529. ↗
  14. Emile Chapuis, Pierre Colombo, Matteo Manica, Matthieu Labeau, and Chloe Clavel. 2020. Hierarchical pre-training for sequence labelling in spoken dialog. arXiv preprint arXiv:2009.11152. ↗
  15. Zheqian Chen, Rongqin Yang, Zhou Zhao, Deng Cai, and Xiaofei He. 2018. Dialogue act recognition via crf-attentive structured network. In The 41st international acm sigir conference on research & development in information retrieval, pages 225–234. find
  16. Pierre Colombo, Emile Chapuis, Matteo Manica, Emmanuel Vignon, Giovanna Varni, and Chloe Clavel. 2020. Guiding attention in sequence-to-sequence models for dialogue act prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7594–7601. find
  17. James W. Cooley and John W. Tukey. 1965. An algorithm for the machine calculation of complex Fourier series. Mathematics of Computation, 19(90):297–301. find
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  19. David Dubray and Jochen Laubrock. 2019. Deep cnn-based speech balloon detection and segmentation for comic books. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1237–1243. IEEE. find
  20. Arpita Dutta, Samit Biswas, and Amit Kumar Das. 2022. Bcbid: first bangla comic dataset and its applications. International Journal on Document Analysis and Recognition (IJDAR), pages 1–15. find
  21. Joe Frankel and Simon King. 2001. Asr-articulatory speech recognition. In Seventh European Conference on Speech Communication and Technology. find
  22. Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2015. Region-based convolutional networks for accurate object detection and segmentation. IEEE transactions on pattern analysis and machine intelligence, 38(1):142–158. find
  23. John J. Godfrey, Edward C. Holliman, and Jane Mc-Daniel. 1992. Switchboard: Telephone speech corpus for research and development. In Proceedings of the 1992 IEEE International Conference on Acoustics, Speech and Signal Processing - Volume 1, ICASSP’92, page 517âĂŞ520, USA. IEEE Computer Society. find
  24. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369–376. ACM. ↗
  25. Clément Guérin, Christophe Rigaud, Antoine Mercier, Farid Ammar-Boudjelal, Karell Bertet, Alain Bouju, Jean-Christophe Burie, Georges Louis, Jean-Marc Ogier, and Arnaud Revel. 2013. ebdtheque: a representative database of comics. In 2013 12th International Conference on Document Analysis and Recognition, pages 1145– 1149. IEEE. find
  26. Dan Gusfield. 1997. Algorithms on Strings, Trees and Sequences. Cambridge University Press, Cambridge, UK. find
  27. Lichy Han and Maulik R Kamdar. 2017. Mri to mgmt: predicting methylation status in glioblastoma patients using convolutional recurrent neural networks. find
  28. Rita Hartel and Alexander Dunst. 2021a. An ocr pipeline and semantic text analysis for comics. In International Conference on Pattern Recognition, pages 213–222. Springer. find
  29. Rita Hartel and Alexander Dunst. 2021b. An ocr pipeline and semantic text analysis for comics. In Pattern Recognition. ICPR International Workshops and Challenges, pages 213–222, Cham. Springer International Publishing. find
  30. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969. find
  31. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  32. Mohit Iyyer, Varun Manjunatha, Anupam Guha, Yogarshi Vyas, Jordan Boyd-Graber, Hal Daume, and Larry S Davis. 2017a. The amazing mysteries of the gutter: Drawing inferences between panels in comic book narratives. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pages 7186–7195. find
  33. Mohit Iyyer, Varun Manjunatha, Anupam Guha, Yogarshi Vyas, Jordan Boyd-Graber, Hal Daumé III, and Larry Davis. 2017b. The amazing mysteries of the gutter: Drawing inferences between panels in comic book narratives. In IEEE Conference on Computer Vision and Pattern Recognition. find
  34. Rajiv Jain and Curtis Wigington. 2019. Multimodal document image classification. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 71–77. IEEE. find
  35. Susanne Jekat, Alexandra Klein, Elisabeth Maier, Ilona Maleck, Marion Mast, and J Joachim Quantz. 1995. Dialogue acts in verbmobil. find
  36. S. Jekat et al. 1995. Dialogue Acts in VERBMOBIL. In Verbmobil Report 65. find
  37. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651. ↗
  38. Daniel Jurafsky, Rebecca Bates, Noah Coccaro, Rachel Martin, Marie Meteer, Klaus Ries, Elizabeth Shriberg, Andreas Stolcke, Paul Taylor, and Carol Van Ess-Dykem̃a. 1997. Automatic Detection of Discourse Structure for Speech Recognition and Understanding. In IEEE Workshop on Speech Recognition and Understanding, pages 88–95, Santa Barbara. find
  39. Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. 2019. Supervised multimodal bitransformers for classifying images and text. arXiv preprint arXiv:1909.02950. ↗
  40. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  41. Jingye Li, Hao Fei, and Donghong Ji. 2020. Modeling local contexts for joint dialogue act recognition and sentiment classification with bi-channel dynamic convolutions. In Proceedings of the 28th International Conference on Computational Linguistics, pages 616–626. find
  42. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986–995. find
  43. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988. find
  44. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  45. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265. ↗
  46. Jiří Martínek, Pavel Král, and Ladislav Lenc. 2021. Dialogue act recognition using visual information. In International Conference on Document Analysis and Recognition, pages 793–807. Springer. find
  47. Jiří Martínek, Pavel Král, Ladislav Lenc, and Christophe Cerisara. 2019a. Multi-Lingual Dialogue Act Recognition with Deep Learning Methods. In Proc. Interspeech 2019, pages 1463– 1467. find
  48. Jiří Martínek, Ladislav Lenc, Pavel Král, Anguelos Nicolaou, and Vincent Christlein. 2019b. Hybrid training data for historical text ocr. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 565–570. IEEE. find
  49. Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2017. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications, 76(20):21811–21838. find
  50. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
  51. Nhu-Van Nguyen, Christophe Rigaud, and Jean-Christophe Burie. 2017. Comic characters detection using deep learning. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 3, pages 41– 46. IEEE. find
  52. Nhu-Van Nguyen, Christophe Rigaud, and Jean-Christophe Burie. 2019. Multi-task model for comic book image analysis. In International Conference on Multimedia Modeling, pages 637–649. Springer. find
  53. Nhu-Van Nguyen, Christophe Rigaud, Arnaud Revel, and Jean-Christophe Burie. 2020. A learning approach with incomplete pixel-level labels for deep neural networks. Neural Networks, 130:111–125. find
  54. Nhu-Van Nguyen, Xuan-Son Vu, Christophe Rigaud, Lili Jiang, and Jean-Christophe Burie. 2021. Icdar 2021 competition on multimodal emotion recognition on comics scenes. In International Conference on Document Analysis and Recognition, pages 767–782. Springer. find
  55. Nobuyuki Otsu. 1979. A threshold selection method from gray-level histograms. IEEE transactions on systems, man, and cybernetics, 9(1):62–66. find
  56. Christophe Ponsard, Ravi Ramdoyal, and Daniel Dziamski. 2012. An ocr-enabled digital comic books viewer. In International Conference on Computers for Handicapped Persons, pages 471–478. Springer. find
  57. Xiaoran Qin, Yafeng Zhou, Zheqi He, Yongtao Wang, and Zhi Tang. 2017. A faster r-cnn based method for comic characters face detection. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 1, pages 1074–1080. IEEE. find
  58. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  59. Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015. Yara parser: A fast and accurate dependency parser. Computing Research Repository, arXiv:1503.06733. Version 2. ↗
  60. Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  61. Dillon Reis, Jordan Kupec, Jacqueline Hong, and Ahmad Daoudi. 2023. Real-time flying object detection with yolov8. arXiv preprint arXiv:2305.09972. ↗
  62. Norbert Reithinger and Martin Klesen. 1997. Dialogue act classification using language models. In Fifth European Conference on Speech Communication and Technology. find
  63. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28. find
  64. Christophe Rigaud, Jean-Christophe Burie, and Jean-Marc Ogier. 2017. Text-independent speech balloon segmentation for comics and manga. In International Workshop on Graphics Recognition, pages 133–147. Springer. find
  65. Christophe Rigaud, Nam Le Thanh, J-C Burie, J-M Ogier, Motoi Iwata, Eiki Imazu, and Koichi Kise. 2015. Speech balloon and speaker association for comics and manga understanding. In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pages 351–355. IEEE. find
  66. Christophe Rigaud, Srikanta Pal, Jean-Christophe Burie, and Jean-Marc Ogier. 2016. Toward speech text recognition for comic books. In Proceedings of the 1st International Workshop on CoMics ANalysis, Processing and Understanding, MANPU ’16, New York, NY, USA. Association for Computing Machinery. find
  67. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pages 234–241, Cham. Springer International Publishing. find
  68. Ken Samuel, Sandra Carberry, and K Vijay-Shanker. 1998. Dialogue act tagging with transformation-based learning. arXiv preprint cmp-lg/9806006. find
  69. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520. find
  70. Jaakko Sauvola and Matti Pietikäinen. 2000. Adaptive document image binarization. Pattern recognition, 33(2):225–236. find
  71. Guokan Shang, Antoine Jean-Pierre Tixier, Michalis Vazirgiannis, and Jean-Pierre Lorré. 2020. Speaker-change aware crf for dialogue act classification. arXiv preprint arXiv:2004.02913. ↗
  72. Baoguang Shi, Xiang Bai, and Cong Yao. 2017. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence, 39(11):2298–2304. find
  73. Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The icsi meeting recorder dialog act (mrda) corpus. Technical report, International Computer Science Inst Berkely CA. find
  74. Ray Smith. 2007. An overview of the tesseract ocr engine. In Ninth international conference on document analysis and recognition (ICDAR 2007), volume 2, pages 629–633. IEEE. find
  75. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000a. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373. find our card
  76. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Met̃eer. 2000b. Dialog Act Modeling for Automatic Tagging and Recognition of Conversational Speech. In Computational Linguistics, volume 26, pages 339–373. find
  77. Weihan Sun, Jean-Christophe Burie, Jean-Marc Ogier, and Koichi Kise. 2013. Specific comic character detection using local feature matching. In 2013 12th International Conference on Document Analysis and Recognition, pages 275–279. IEEE. find
  78. Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. 2022. Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv preprint arXiv:2207.02696. ↗
  79. Jerod Weinman, Ziwen Chen, Ben Gafford, Nathan Gifford, Abyaya Lamsal, and Liam Niehus-Staab. 2019. Deep neural networks for text detection and recognition in historical maps. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 902–909. IEEE. find
  80. Christoph Wick, Christian Reul, and Frank Puppe. 2020. Calamari - A High-Performance Tensorflow-based Deep Learning Package for Optical Character Recognition. Digital Humanities Quarterly, 14(1). find
  81. Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named entity recognition in tweets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. find
  82. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536. ↗
  83. Yuan Zhuang and Ellen Riloff. 2020. Exploring the role of context to distinguish rhetorical and information-seeking questions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 306–312. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/martinek-2024-comicorda-dialogue-act-recognition-comic/