The CODI-CRAC 2022 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue

What it is about

The second edition of a shared task on anaphora in dialogue, with three tasks: identity anaphora (including split antecedents and singletons), bridging reference, and discourse deixis (a pronoun such as that or this referring to an event, fact or utterance). Data are four dialogue corpora annotated in the ARRAU scheme — LIGHT (fantasy text-adventure dialogues), AMI meetings, PERSUASION and Switchboard — with ARRAU's TRAINS dialogues for training: about 215K tokens and 61K markables, scored with the Universal Anaphora scorer. Three teams submitted 230 runs. On identity anaphora the best system beats a SpanBERT baseline by more than 25 points and passes 80 CoNLL F1 on LIGHT and PERSUASION, but AMI's long meetings stay about 20 points lower because antecedents are far away. Bridging with predicted mentions is very hard (baseline about 5 F1, best about four times that), and gold anaphors help far more than gold mentions. The discourse deixis baseline is a two-line heuristic: take demonstrative pronouns as anaphors and link each to the immediately preceding clause or utterance.

Ideas

• I1. Implement the discourse-deixis baseline as a sieve in coref/src/sieve.rs (the disabled demonstratives slot is the place): this/that (and it as subject of happened, was true, means) not followed by a noun → the preceding clause, as an event mention with a VERB head (coref/src/mention.rs). It directly targets the README's first error type («it» referring to an event or statement). Measure on GUM dev the link=disc cases (113) and CoNLL F1, and the precision of «it» from the per-pronoun table.
• I2. In the frame reader the same rule gives a pronoun an event antecedent: «There was nothing so remarkable in that» → the frame of the rabbit running past. prag/src/frames.rs should be allowed to resolve that/this to a frame index rather than an entity, so that «Why was Alice surprised?» can reach the event. Measure frame bench coref F1 and diagnosis on why-questions.
• I3. LIGHT (fantasy role-play dialogue) is the subcorpus closest to Mova's tales; if its licence allows test use, it is a ready external check for the speaker sieve (I/you across turns) and for anaphora in character dialogue. Check the licence before downloading anything.
• I4. The AMI gap is a warning that distance kills: report Mova's pronoun precision by sentence distance to the antecedent (0, 1, 2–3, more) in coref eval, so a window change or salience change (Lappin–Leass) can be judged where it matters.

For Mova

35. Reference and coreference §3.4 (abstract anaphora), §6 (reference in quoted speech and dialogue), §9 (the coref crate). Code: coref/src/sieve.rs, coref/src/mention.rs, coref/src/eval.rs, prag/src/frames.rs. Measures: CoNLL F1 on GUM dev with gold and SLM trees, «it» precision, disc-link recall, frame bench coref F1.

Topics: Discourse analysis · Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{yu2022codi,
  title = {{The CODI-CRAC 2022 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue}},
  author = {Yu, Juntao and Khosla, Sopan and Manuvinakurike, Ramesh and Levin, Lori and Ng, Vincent and Poesio, Massimo and Strube, Michael and Rosé, Carolyn},
  booktitle = {Proceedings of the CODI-CRAC 2022 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue},
  year = {2022},
  url = {https://aclanthology.org/2022.codi-crac.1/}
}

References

57 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. John A. Hawkins. 1978. Definiteness and Indefiniteness. Croom Helm, London. find
2. Tatiana Anikina, Cennet Oguz, Natalia Skachkova, Siyu Tao, Sharmila Upadhyaya, and Ivana Kruijff-Korbayova. 2021. Anaphora resolution in dialogue: Description of the DFKI-TalkingRobots system for the CODI-CRAC 2021 shared-task. In Proceedings of the CODI-CRAC 2021 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue, pages 32–42, Punta Cana, Dominican Republic. Association for Computational Linguistics. find our card
3. Peter A. Heeman and Graeme Hirst. 1995. Collaborating on referring expressions. Computational Linguistics, 21(3):351–382. find
4. Yufang Hou. 2020. Bridging anaphora resolution as question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1428–1438, Online. Association for Computational Linguistics. ↗ our card
5. David Bamman, Olivia Lewke, and Anya Mansoor. 2020. An annotated dataset of coreference in english literature. In Proc. of LREC. European Language Resources Association (ELRA), Association for Computational Linguistics (ACL). find our card
6. Yufang Hou, Katja Markert, and Michael Strube. 2018. Unrestricted bridging resolution. Computational Linguistics, 44(2):237–284. find our card
7. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. ation for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–692. find
8. Quan Liu, Hui Jiang, Andrew Evdokimov, Zhen-Hua Ling, Xiaodan Zhu, Si Wei, and Yu Hu. 2017. Cause-effect knowledge acquisition and neural association model for solving a set of winograd schema problems. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 2344–2350. find
9. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803–5808, Hong Kong, China. Association for Computational Linguistics. find
10. Pengcheng Lu and Massimo Poesio. 2021. Coreference resolution for the biomedical domain: A survey. In Proc. of the CRAC Workshop. find
11. Hans Kamp and Uwe Reyle. 1993. From Discourse to Logic. D. Reidel, Dordrecht. find
12. Ana Marasović, Leo Born, Juri Opitz, and Anette Frank. 2017. A mention-ranking model for abstract anaphora resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 221–232, Copenhagen, Denmark. Association for Computational Linguistics. find our card
13. Ben Kantor and Amir Globerson. 2019. Coreference resolution with entity equalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 673–677, Florence, Italy. Association for Computational Linguistics. find
14. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: the Penn Treebank. Computational Linguistics, 19(2):313–330. ↗ our card
15. Sopan Khosla, Juntao Yu, Ramesh Manuvinakurike, Vincent Ng, Massimo Poesio, Michael Strube, and Carolyn Rosé. 2021. find
16. The CODI-CRAC 2021 shared task on anaphora, bridging, and discourse deixis in dialogue. In Proceedings of the CODI-CRAC 2021 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue, pages 1–15, Punta Cana, Dominican Republic. Association for Computational Linguistics. find our card
17. Katja Markert, Yufang Hou, and Michael Strube. 2012. Collective classification for fine-grained information status. In Proc. of the ACL, Juju island, Korea. find
18. Mark-Christoph Müller. 2008. Fully Automatic Resolution of It, This And That in Unrestricted Multy-Party Dialog. Ph.D. thesis, Universität Tübingen. find
19. Hideo Kobayashi, Shengjie Li, and Vincent Ng. 2021. Neural anaphora resolution in dialogue. In Proceedings of the CODI-CRAC 2021 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue, pages 16–31, Punta Cana, Dominican Republic. Association for Computational Linguistics. find our card
20. Mark-Christoph Müller and Michael Strube. 2006. Multi-level annotation of linguistic data with mmax2. In S. Braun, K. Kohn, and J. Mukherjee, editors, Corpus Technology and Language Pedagogy. New Resources, New Tools, New Methods, volume 3 of English Corpus Linguistics, pages 197–214. Peter Lang. find
21. Hideo Kobayashi and Vincent Ng. 2021. Bridging resolution: Making sense of the state of the art. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1652–1659, Online. Association for Computational Linguistics. find
22. Judith Muzerelle, Anaïs Lefeuvre, Emmanuel Schang, Jean-Yves Antoine, Aurore Pelletier, Denis Maurel, Iris Eshkol, and Jeanne Villaneau. 2014. Ancor_centre, a large free spoken french coreference corpus. In Proc. of LREC. find
23. Varada Kolhatkar, Adam Roussel, Stefanie Dipper, and Heike Zinsmeister. 2018. Anaphora with non-nominal antecedents in computational linguistics: a Survey. Computational Linguistics, 44(3):547–612. find our card
24. Anna Nedoluzhko. 2013. Generic noun phrases and annotation of coreference and bridging relations in the prague dependency treebank. In Proc. of LAW, pages 103–111. find
25. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. ↗ our card
26. Anna Nedoluzhko, Michal Novák, Martin Popel, Zdenĕk Z̆abokrtský, and Daniel Zeman. 2021. Coreference meets universal dependencies – a pilot experiment on harmonizing coreference datasets for 11 languages. ÚFAL Technical Report TR-2021-66, Charles University, Prague. find our card
27. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Associ-Malvina Nissim, Shipra Dingare, Jean Carletta, and Mark Steedman. 2004. An annotation scheme for information status in dialogue. In Proc. of LREC. find
28. Silviu Paun, Juntao Yu, Nafise Sadat Moosavi, and Massimo Poesio. 2022. Scoring coreference chains with split-antecedent anaphors. find our card
29. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34:8732–8740. find
30. Massimo Poesio. 2004. Discourse annotation and semantic annotation in the GNOME corpus. In Proc. of the ACL Workshop on Discourse Annotation, pages 72–79, Barcelona. find
31. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van-Ess-Dykema, and Marie Meteer. 1997. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339–371. find our card
32. Massimo Poesio and Ron Artstein. 2008. Anaphoric annotation in the ARRAU corpus. In Proc. of LREC, Marrakesh. find
33. Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, Samuel Humeau, and Jason Weston. 2019. Learning to speak and act in a fantasy text adventure game. ArXiv preprint arXiv:1903.03094. ↗
34. Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Silviu Paun, Alexandra Uma, and Juntao Yu. 2019. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proc. of NAACL, page 1778–1789, Minneapolis. Association for Computational Linguistics (ACL). find
35. Olga Uryupina, Ron Artstein, Antonella Bristot, Federica Cavicchio, Francesca Delogu, Kepa J. Rodriguez, and Massimo Poesio. 2020. Annotating a broad range of anaphoric phenomena, in a variety of genres: the ARRAU corpus. Journal of Natural Language Engineering. find
36. Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolution with the ARRAU corpus. In Proceedings of the First Workshop on Computational Models of Reference, Anaphora and Coreference, pages 11–22, New Orleans, Louisiana. Association for Computational Linguistics. find
37. Hardik Vala, Andrew Piper, and Derek Ruths. 2016. The more antecedents, the merrier: Resolving multi-antecedent anaphors. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2287– 2296, Berlin, Germany. Association for Computational Linguistics. find
38. Massimo Poesio, Sameer Pradhan, Marta Recasens, Kepa Rodriguez, and Yannick Versley. 2016. Annotated corpora and annotation tools. In M. Poesio, R. Stuckardt, and Y. Versley, editors, Anaphora Resolution: Algorithms, Resources and Applications, chapter 4. Springer. find
39. Yannick Versley. 2008. Vagueness and referential ambiguity in a large-scale annotated corpus. Research on Language and Computation, 6:333–353. find
40. Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Persuasion for good: Towards a personalized persuasive dialogue system for social good. In Proc. of ACL. find
41. Massimo Poesio and Hannes Rieser. 2010. Completions, coordination, and alignment in dialogue. Dialogue and Discourse, 1(1):1–89. find
42. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea. find our card
43. Bonnie L. Webber. 1991. Structure and ostension in the interpretation of discourse deixis. Language and Cognitive Processes, 6(2):107–135. find
44. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617. find our card
45. Ellen F. Prince. 1981. Toward a taxonomy of given-new information. In P. Cole, editor, Radical Pragmatics, pages 223–256. Academic Press, New York. find
46. Sam Wiseman, Alexander M. Rush, Stuart Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1416– 1426, Beijing, China. Association for Computational Linguistics. find
47. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: The Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 777–789, Jeju Island, Korea. Association for Computational Linguistics. find
48. Marta Recasens and M. Antònia Martí. 2010. AnCora-CO: Coreferentially annotated corpora for Spanish and Catalan. Language Resources and Evaluation, 44(4):315–345. ↗
49. Liyan Xu and Jinho D Choi. 2020. Revealing the myth of higher-order inference in coreference resolution. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8527–8533. find
50. Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020a. Neural mention detection. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020). find
51. Juntao Yu, Sopan Khosla, Nafise Sadat Moosavi, Silviu Paun, Sameer Pradhan, and Massimo Poesio. 2022. The universal anaphora scorer. In Proceedings of the Language Resources and Evaluation Conference, pages 4873–4883, Marseille, France. European Language Resources Association. find
52. Juntao Yu, Nafise Sadat Moosavi, Silviu Paun, and Massimo Poesio. 2020b. Free the plural: Unrestricted split-antecedent anaphora resolution. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6113–6125, Barcelona, Spain (Online). International Committee on Computational Linguistics. find our card
53. Juntao Yu, Nafise Sadat Moosavi, Silviu Paun, and Massimo Poesio. 2021. Stay together: A system for single and split-antecedent anaphora resolution. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. find
54. Juntao Yu and Massimo Poesio. 2020. Multitask learning based neural bridging reference resolution. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3534–3546, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
55. Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. ↗
56. Ethan Zhou and Jinho D. Choi. 2018. They exist! introducing plural mentions to coreference resolution and entity linking. In Proceedings of the 27th International Conference on Computational Linguistics, pages 24–34, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
57. Yilun Zhu, Sameer Pradhan, and Amir Zeldes. 2021. OntoGUM: Evaluating contextualized SOTA coreference resolution on 12 more genres. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 461–467, Online. Association for Computational Linguistics. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/yu-2022-codi-crac-2022-shared-task/