The Aligned Multimodal Movie Treebank: An audio, video, dependency-parse treebank

What it is about

The Aligned Multimodal Movie Treebank (AMMT) takes dialogue from Hollywood films, with word-level alignment to the audio and video, POS tags and UD dependency parses. It has 31,264 sentences and 218,090 words. The authors also describe a tool for fast audio alignment during annotation. The parses were first made by Stanza and then corrected by annotators. Accuracy is lower on very short sentences (under 3 words) and on long ones (over 20 words) than on average, and the authors note a bias in favour of Stanza. The treebank is the first in UD that combines speech, video and parse.

Ideas

• I1. en parse-eval with a length split (under 3, 3 to 20, over 20 words) in en/src/parse.rs, reported next to the mean, on AMMT and EWT (test only for AMMT).
• I2. en on movie dialogue: LAS of the perceptron parser on AMMT against EWT, to measure the dialogue gap for Mova's spoken data (test only).
• I3. A word-alignment check in the annot review loop, for spoken source material, measured by the share of tokens whose timing disagrees with a forced alignment (no new model).

For Mova

The paper gives a spoken, scripted English test set with a length profile, which is useful for the en parser. The audio side is out of scope for Mova. Relevant chapters are languages/en/38-dialogue-and-discourse-markers and 37-register-and-variation. Measure first the LAS of en on AMMT split by sentence length.

Topics: Dependency parsing · Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.emnlp-main.648

@inproceedings{yaari2022aligned,
  title = {{The Aligned Multimodal Movie Treebank: An audio, video, dependency-parse treebank}},
  author = {Yaari, Adam and DeWitt, Jan and Hu, Henry and Stankovits, Bennett and Felshin, Sue and Berzak, Yevgeni and Aparicio, Helena and Katz, Boris and Cases, Ignacio and Barbu, Andrei},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.648},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.648}
}

References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lars Ahrenberg. 2007. Lines: An English-Swedish Parallel Treebank. In Proceedings of the 16th Nordic Conference of Computational Linguistics (NODAL-IDA 2007). find our card
2. Yevgeni Berzak, Yan Huang, Andrei Barbu, Anna Korhonen, and Boris Katz. 2016. Anchoring and agreement in syntactic annotations. Conference on Empirical Methods in Natural Language Processing (EMNLP) 2016. find
3. Paul Boersma. 2001. Praat, a system for doing phonetics by computer. Glot. Int., 5(9):341–345. find
4. Andrew Caines, Michael McCarthy, and Paula Buttery. 2017. Parsing transcripts of speech. Conference on Empirical Methods in Natural Language Processing (EMNLP) 2017. find
5. John J Godfrey, Edward C Holliman, and Jane Mc-Daniel. 1992. SWITCHBOARD: Telephone Speech Corpus for Research and Development. In IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1. find
6. Google. 2020. Speech-to-text: Automatic speech recognition — Google Cloud. find
7. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision. find
8. Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. find our card
9. Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. 2015. librosa: Audio and Music Signal Analysis in Python. In Proceedings of the 14th Python in Science Conference, volume 8. find
10. Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal Dependencies v1: A Multilingual Treebank Collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). find our card
11. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection. In Proceedings of the 12th Language Resources and Evaluation Conference. find our card
12. Joakim Nivre, Jens Nilsson, and Johan Hall. 2006. Talbanken05: A Swedish Treebank with Phrase Structure and Dependency Annotation. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06). find
13. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. ↗ our card
14. Natalia Silveira, Timothy Dozat, Marie-Catherine De Marneffe, Samuel R Bowman, Miriam Connor, John Bauer, and Christopher D Manning. 2014. A Gold Standard Dependency Corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14). find
15. Milan Straka. 2018. UDPipe 2.0 prototype at CoNLL 2018 UD shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. *Figure 5: Distribution of sentence lengths in AMMT. Most sentences are quite short. The mean sentence length is 6.97 words long. Compare to standard corpora derived from written sources like the English Web Treebank (15.33 words/sentence) long and the Penn Treebank (23.73 words/sentence in the test set).* *Table 3: The distribution of POS tags (left), and the most common dependencies (right). There is a long tail of dependencies.* 25050 14003 12825 12325 11274 9286 7830 6653 5693 4958 4306 3996 3912 3682 3335 3322 3134 Count nsubj advmod obj det case aux cop obl mark amod xcomp nmod:poss discourse cc compound conj vocative Dependencies 9829 12464 13688 18965 3746 12984 6275 25457 1835 7202 36370 8679 30301 2140 10 28139 6 Count ADJ ADP ADV AUX CCONJ DET INTJ NOUN NUM PART PRON PROPN PUNCT SCONJ SYM VERB X POS find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ud-2022-yaari-aligned-multimodal-movie-treebank-audio/