Opinion Piece: Can we Fix the Scope for Coreference? Problems and Solutions for Benchmarks beyond OntoNotes

What it is about

An argument, with examples mostly from GUM, that the OntoNotes coreference guidelines on which most systems are trained leave out cases users would expect, because they were chosen for high agreement. OntoNotes treats all indefinite and bare NPs as generic and does not link them, so a specific wax figure mentioned twice with a is two entities; generic pronouns are left out although in context they form distinct chains (in a vlog, one you is the person considering pregnancy, another you in «you know» is a discourse marker to the audience); compound modifiers are excluded although they can be specific and referred back to; a year inside a date cannot corefer with the same year mentioned alone; nested mentions are only annotated if mentioned again; and copular predication is excluded altogether. The author proposes three principles: decide coreference by semantics (same entity), not by form; include singletons and indefinites; and handle the real problem behind predication — changes of time or modal scope («a fresh major in the Swedish army… the now general in the Russian army») — with explicit scope information rather than by omission.

Ideas

• I1. Keep Mova's pred sieve (subject = predicative noun, as GUM annotates) but stamp the link with its scope: tense/time of the copula frame and any modal or cond context. In the frame reader the predicate becomes an attribute of the entity valid at that time, so the school student does not answer «Was the buried man a Swedish major?» from an identity link. Check on GUM dev that pred precision (58%) does not change, and count course questions about «what was X» at different times.
• I2. Split generic you by function, as the paper's vlog example does: you know, you see, mind you, you bet are discourse markers and form their own chain (or are flagged), and must not be merged by the speaker sieve with a referential you addressed to someone. Add the closed list to coref/src/sieve.rs with the reason printed; measure CoNLL F1 on the GUM dev conversation, interview and vlog documents.
• I3. Indefinite anaphors: GUM links repeated specific indefinites («a wax figure… a wax figure of herself»), the current sieves link indefinites only on exact match. Count on GUM dev how many gold links have an indefinite second mention and what they share (same head plus a specific modifier), before deciding whether a narrow rule is worth it.
• I4. A year or day inside a date is a referent: let the frame reader keep when: values comparable across frames (same year in «May 7, 2005» and «in 2005»), so time questions can match them, even though numbers and times are never pronoun antecedents.

For Mova

35. Reference and coreference §4 (non-referential and generic uses), §7 (annotating coreference), §9. Code: coref/src/sieve.rs (pred, speaker), prag/src/frames.rs (predicate as time-stamped attribute, dates). Measures: CoNLL F1 on GUM dev (per genre), pred precision; course questions about states at different times.

Topics: Discourse analysis · Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.5210/dad.2022.102

@inproceedings{zeldes2022opinion,
  title = {{Opinion Piece: Can we Fix the Scope for Coreference? Problems and Solutions for Benchmarks beyond OntoNotes}},
  author = {Zeldes, Amir},
  booktitle = {Dialogue \& Discourse},
  year = {2022},
  doi = {10.5210/dad.2022.102},
  url = {https://doi.org/10.5210/dad.2022.102}
}

References

58 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. Abstract Meaning Representation for Sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186, Sofia, Bulgaria, 2013. find our card
2. BBN Technologies. OntoNotes English co-reference guidelines version 7.0. Technical report, BBN Technologies, 2007. find
3. BBN Technologies. OntoNotes Chinese co-reference guidelines version 4.0. Technical report, BBN Technologies, 2008. find
4. Archna Bhatia, Chu-Cheng Lin, Nathan Schneider, Yulia Tsvetkov, Fatima Talib Al-Raisi, Laleh Roostapour, Jordan Bender, Abhimanu Kumar, Lori Levin, Mandy Simons, and Chris Dyer. Automatic classification of communicative functions of definiteness. In Proceedings of COLING 2014, pages 1059–1070, Dublin, Ireland, 2014. find
5. Kevin Clark and Christopher D. Manning. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of EMNLP 2016, pages 2256–2262, Austin, TX, 2016. find
6. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. Universal Dependencies. Computational Linguistics, 47(2):255–308, 2021. find our card
7. George R. Doddington, Alexis Mitchell, Mark A. Przybocki, Lance A. Ramshaw, Stephanie Strassel, and Ralph M. Weischedel. The Automatic Content Extraction (ACE) program. tasks, data, and evaluation. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC-2004), pages 837–840, 2004. find
8. John W. Du Bois, Wallace L. Chafe, Charles Meyer, Sandra A. Thompson, Robert Englebretson, and Nii Martey. Santa barbara corpus of spoken american english, parts 1-4, 2000-2005. find
9. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. AllenNLP: A deep semantic natural language processing platform. In Proceedings of ACL 2018, pages 1–6, Melbourne, 2018. find
10. Iris Hendrickx, Amália Mendes, and Silvia Mencarelli. Modality in text: a proposal for corpus annotation. In Proceedings of LREC 2012, pages 1805–1812, Istanbul, Turkey, 2012. find
11. Lynette Hirschman and Nancy Chinchor. Appendix F: MUC-7 coreference task definition (version 3.0). In Seventh Message Understanding Conference (MUC-7), 1998. find
12. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. OntoNotes: The 90% solution. In Proceedings of NAACL 2006: Short Papers, pages 57–60, New York, 2006. find
13. Lili Jiang, Yafang Wang, Johannes Hoffart, and Gerhard Weikum. Crowdsourced entity markup. In Proceedings of Crowdsourcing the Semantic Web, pages 59–68, Sydney, 2013. find
14. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. BERT for coreference resolution: Baselines and analysis. In Proceedings of EMNLP-IJCNLP 2019, pages 5803–5808, Hong Kong, China, 2019. find
15. Manfred Krifka. Basic notions of information structure. Acta Linguistica Hungarica, 55:243–276, 2008. find
16. Kerstin Kunz. Variation in English and German Nominal Coreferece. A Study of Political Essays. Peter Lang, Frankfurt, 2009. find
17. Fred Landman. Indefinites and the Type of Sets. Blackwell, Malden, MA, 2004. find
18. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885–916, 2013. find our card
19. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. End-to-end neural coreference resolution. In Proceedings of EMNLP 2017, pages 188–197, Copenhagen, 2017. find our card
20. Jessica Lin and Amir Zeldes. WikiGUM: Exhaustive entity linking for Wikification in 12 genres. In Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 170–175, Punta Cana, Dominican Republic, 2021. find
21. Nafise Sadat Moosavi and Michael Strube. Which coreference evaluation metric do you trust? A proposal for a link-based entity aware metric. In Proceedings of ACL 2016, pages 632–642, Berlin, 2016. find our card
22. Anna Nedoluzhko, Jiří Mírovský, Radek Ocelák, and Jiří Pergler. Extended coreferential relations and bridging anaphora in the Prague Dependency Treebank. In Proceedings of the 7th Discourse Anaphora and Anaphor Resolution Colloquium (DAARC 2009), pages 1–16, Goa, India, 2009. find
23. Anna Nedoluzhko, Michal Novák, Martin Popel, Zdeněk Žabokrtský, Amir Zeldes, and Daniel Zeman. CorefUD 1.0: Coreference meets Universal Dependencies. In Proceedings of LREC 2022, Marseille, France, 2022. find our card
24. Azadeh Nikfarjam, Ehsan Emadzadeh, and Graciela Gonzalez. Towards generating a patient’s timeline: Extracting temporal relationships from clinical notes. Biomedical Informatics, 46:40–47, 2013. find
25. Malvina Nissim, Paola Pietrandrea, Andrea Sansò, and Caterina Mauri. Cross-linguistic annotation of modality: a data-driven hierarchical model. In Proceedings of the 9th Joint ISO - ACL SIGSEM Workshop on Interoperable Semantic Annotation, pages 7–14, Potsdam, Germany, 2013. find
26. Michal Novák and Anna Nedoluzhko. Correspondences between Czech and English coreferential expressions. Discours, 16:3–41, 2015. find
27. Jan-Ola Östman. You Know: A discourse-functional approach. Pragmatics & Beyond, II:7. John Benjamins, Amsterdam, 1981. find
28. Tim O’Gorman, Kristin Wright-Bettner, and Martha Palmer. Richer event description: Integrating event coreference with temporal, causal and bridging annotation. In Proceedings of the 2nd Workshop on Computing News Storylines (CNS 2016), pages 47–56, Austin, TX, 2016. find
29. Martha Palmer, Daniel Gildea, and Paul Kingsbury. The Proposition Bank: A corpus annotated with semantic roles. Computational Linguistics, 31(1):71–106, 2005. find
30. Barbara H. Partee. Noun phrase interpretation and type-shifting principles. In Jeroen Groenendijk, Martin Stokhof, and Dick de Jongh, editors, Studies in Discourse Representation Theory and the Theory of Generalized Quantifiers, pages 115–143. Foris, Dordrecht, 1986. find
31. Massimo Poesio and Ron Artstein. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of the Workshop on Frontiers in Corpus Annotation II: Pie in the Sky, pages 76–83, Stroudsburg, PA, 2005. ACL. find
32. Massimo Poesio and Ron Artstein. Anaphoric annotation in the ARRAU corpus. In Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, and Daniel Tapias, editors, Proceedings of LREC 2008, pages 1170–1174, Marrakesh, Morocco, 2008. find
33. Massimo Poesio, Sameer Pradhan, Marta Recasens, Kepa Rodriguez, and Yannick Versley. Annotated corpora and annotation tools. In Massimo Poesio, Roland Stuckardt, and Yannick Versley, editors, Anaphora Resolution: Algorithms, Resources, and Applications, Theory and Applications of Natural Language Processing, pages 97–140. Springer, Heidelberg, 2016. find
34. Paul M. Postal. Anaphoric islands. In Robert I. Binnick, editor, 5th regional meeting of the Chicago Linguistic Society, volume 5, pages 205–239. Chicago Linguistic Society, Chicago, 1969. find
35. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of the Conference on Computational Natural Language Learning: Shared Task, pages 1–27, Portland, OR, 2011. find our card
36. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 30–35, Baltimore, MD, 2014. find our card
37. James Pustejovsky and Amber Stubbs. Increasing informativeness in temporal annotation. In 5th Linguistic Annotation Workshop (LAW V), pages 152–160, Portland, OR, 2011. find
38. James Pustejovsky, Ken Lai, and Nianwen Xue. Modeling quantification and scope in abstract meaning representations. In Proceedings of the First International Workshop on Designing Meaning Representations (DMR), pages 28–33, Florence, Italy, 2019. find
39. Swayambhu Nath Ray, Shib Sankar Dasgupta, and Partha Talukdar. AD3: Attentive deep document dater. In Proceedings of EMNLP 2018, pages 1871–1880, Brussels, Belgium, 2018. find
40. Marta Recasens, M. Antònia Martí, and Mariona Taulé. Where anaphora and coreference meet. annotation in the Spanish CESS-ECE corpus. In Proc. RANLP 2007, Borovets, Bulgaria, 2007. find
41. Marta Recasens, M. Antònia Martí, and Constantin Orasan. Annotating near-identity from coreference disagreements. In Proceedings of LREC 2012, pages 165–172, Istanbul, Turkey, 2012. find
42. Tanya Reinhart. Pragmatics and linguistics. An analysis of sentence topics. Philosophica, 27:53–94, 1982. find
43. Aynat Rubinstein, Hillary Harner, Elizabeth Krawczyk, Daniel Simonson, Graham Katz, and Paul Portner. Toward fine-grained annotation of modality in text. In Proceedings of the Workshop on Annotation of Modal Meanings in Natural Language, pages 38–46, Potsdam, Germany, 2013. find
44. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL/IJCNLP 2009, pages 656–664, Suntec, Singapore, 2009. find our card
45. William F. Styler IV, Steven Bethard, Sean Finan, Martha Palmer, Sameer Pradhan, Piet C de Groen, Brad Erickson, Timothy Miller, Chen Lin, Guergana Savova, and James Pustejovsky. Temporal annotation in the clinical domain. TACL, 2:143–154, 2014. find
46. Kees van Deemter and Rodger Kibble. On coreferring: Coreference in MUC and related annotation schemes. Computational Linguistics, 26(4):629–637, 2000. find our card
47. Gregory Ward, Richard Sproat, and Gail McKoon. A pragmatic analysis of so-called anaphoric islands. Language, 63(3):439–474, 1993. find
48. Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Nianwen Xue, Martha Palmer, Jena D. Hwang, Claire Bonial, Jinho Choi, Aous Mansouri, Maha Foster, Abdel aati Hawwary, Mitchell Marcus, Ann Taylor, Craig Greenberg, Eduard Hovy, Robert Belvin, and Ann Houston. Ontonotes release 5.0. Technical report, Linguistic Data Consortium, Philadelphia, 2012. find
49. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6953–6963, Online, 2020. find
50. Annie Zaenen. Mark-up barking up the wrong tree. Computational Linguistics, 32(4):577–580, 2006. find
51. Amir Zeldes. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612, 2017. find
52. Amir Zeldes. Multilayer Corpus Studies. Routledge Advances in Corpus Linguistics 22. Routledge, London, 2018. find
53. Amir Zeldes and Shuo Zhang. When annotation schemes change rules help: A configurable approach to coreference resolution beyond OntoNotes. In Proceedings of the Workshop on Coreference Resolution Beyond OntoNotes (CORBON), pages 92–101, San Diego, CA, 2016. find
54. Amir Zeldes, Debopam Das, Erick Galani Maziero, Juliano Desiderato Antonio, and Mikel Iruskieta. The DISRPT 2019 Shared Task on elementary discourse unit segmentation and connective detection. In Proceedings of Discourse Relation Treebanking and Parsing (DISRPT 2019), pages 97–104, Minneapolis, MN, 2019. find our card
55. Amir Zeldes, Yang Janet Liu, Mikel Iruskieta, Philippe Muller, Chloé Braud, and Sonia Badene, editors. Proceedings of the 2nd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2021), Punta Cana, Dominican Republic, 2021. find
56. Rui Zhang, Cícero Nogueira dos Santos, Michihiro Yasunaga, Bing Xiang, and Dragomir Radev. Neural coreference resolution with deep biaffine attention by joint mention detection and mention clustering. In Proceedings of ACL 2018, pages 102–107, Melbourne, Australia, 2018. find
57. Shuo Zhang and Amir Zeldes. GitDOX: A linked version controlled online XML editor for manuscript transcription. In Proceedings of FLAIRS-30, pages 619–623, Marco Island, FL, 2017. find
58. Yilun Zhu, Sameer Pradhan, and Amir Zeldes. Anatomy of OntoGUM—Adapting GUM to the OntoNotes scheme to evaluate robustness of SOTA coreference algorithms. In Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC 2021), pages 141–149, Punta Cana, Dominican Republic, 2021. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/zeldes-2022-opinion-piece-can-we-fix/