Subjectivity in the Annotation of Bridging Anaphora

What it is about

The paper studies where human judgment enters bridging annotation, and why agreement is low. It is a pilot on the test set of the GUM corpus: 26 documents, about 26k tokens, in 16 genres, each annotated by two people. The authors separate three decisions: recognising the bridging anaphor, choosing its antecedent, and choosing the subtype. The pilot's F1 is 0.38 for anaphor recognition and 0.28 for anaphor and antecedent together. The antecedent is correct in 72% of 133 cases where both annotators agreed on the anaphor. The subtype kappa is 0.58 on 96 agreed pairs. Of 172 instances that only one annotator marked, 64 (37%) could reasonably count as bridging after review. The authors propose 11 subtypes under COMPARISON, ENTITY, SET and OTHER, and conclude that recognising the anaphor is the most subjective step.

Ideas

• I1. Split any Mova bridging detector into the same three decisions, and report each one's precision and recall separately, as the paper does. The anaphor step should be checked first.
• I2. Add a fixed priority order for subtypes that apply together, and a rule that picks the antecedent nearer to the anaphor when two are equally plausible. Count how often each rule decides.
• I3. Tie the accessibility test for "world knowledge" to a fixed list, for example the knowledge base in world/src/knowledge.rs, so that an entity is bridging-accessible only if the list does not already give it. This removes the annotator's own knowledge from the decision.
• I4. Keep the table of disagreement types (matching, different subtype, different antecedent, one annotator only) as an error log for any Mova output checked by hand.

For Mova

The chapter is 35. Reference and coreference. The code is coref/src/mention.rs and the knowledge base in world. The measure is per-step precision and recall, and the kappa on subtypes against a hand-checked sample. The paper is CC BY, but GUM itself is CC BY-NC-SA, so its documents are for evaluation only.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.law-1.4 · arXiv 2506.07297

@inproceedings{levine2025subjectivity,
  title = {{Subjectivity in the Annotation of Bridging Anaphora}},
  author = {Levine, Lauren and Zeldes, Amir},
  booktitle = {Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025)},
  year = {2025},
  doi = {10.18653/v1/2025.law-1.4},
  eprint = {2506.07297},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2025.law-1.4}
}

References

26 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tatsuya Aoyama, Shabnam Behzad, Luke Gessler, Lauren Levine, Jessica Lin, Yang Janet Liu, Siyao Peng, Yilun Zhu, and Amir Zeldes. 2023. GENTLE: A genre-diverse multilayer challenge set for English NLP and linguistic evaluation. In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pages 166–178, Toronto, Canada. Association for Computational Linguistics. ↗ our card
2. Herbert H. Clark. 1975. Bridging. In Theoretical Issues in Natural Language Processing. find
3. Kerstin Eckart, Arndt Riester, and Katrin Schweitzer. 2012. A Discourse Information Radio News Database for Linguistic Analysis, pages 65–76. Springer Berlin Heidelberg, Berlin, Heidelberg. find
4. Biaoyan Fang, Timothy Baldwin, and Karin Verspoor. 2022. What does it take to bake a cake? the RecipeRef corpus and anaphora resolution in procedural text. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3481–3495, Dublin, Ireland. Association for Computational Linguistics. ↗
5. Yulia Grishina. 2016. Experiments on bridging across languages and genres. In Proceedings of the Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2016), pages 7–15, San Diego, California. Association for Computational Linguistics. ↗
6. Kian Kenyon-Dean, Eisha Ahmed, Scott Fujimoto, Jeremy Georges-Filteau, Christopher Glasz, Barleen Kaur, Auguste Lalande, Shruti Bhanderi, Robert Belfer, Nirmal Kanagasabai, Roman Sarrazingen-dron, Rohit Verma, and Derek Ruths. 2018. Sentiment analysis: It‘s complicated! In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1886–1895, New Orleans, Louisiana. Association for Computational Linguistics. find
7. Maciej Ogrodniczuk and Magdalena Zawisławska. 2016. Bridging relations in Polish: Adaptation of existing typologies. In Proceedings of the Workshop on Coreference Resolution Beyond OntoNotes (COR-BON 2016), pages 16–22, San Diego, California. Association for Computational Linguistics. ↗
8. Hideo Kobayashi, Yufang Hou, and Vincent Ng. 2023. PairSpanBERT: An enhanced language model for bridging resolution. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6931– 6946, Toronto, Canada. Association for Computational Linguistics. ↗
9. Cecilia Ovesdotter Alm. 2011. Subjective natural language problems: Motivations, applications, characterizations, and implications. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 107–112, Portland, Oregon, USA. Association for Computational Linguistics. find
10. Hideo Kobayashi and Vincent Ng. 2020. Bridging resolution: A survey of the state of the art. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3708–3721, Barcelona, Spain (Online). International Committee on Computational Linguistics. ↗
11. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 507–511, Baltimore, Maryland. Association for Computational Linguistics. ↗
12. Lauren Levine and Amir Zeldes. 2024. Unifying the scope of bridging anaphora types in English: Bridging annotations in ARRAU and GUM. In Proceedings of The Seventh Workshop on Computational Models of Reference, Anaphora and Coreference, pages 41–51, Miami. Association for Computational Linguistics. ↗ our card
13. Massimo Poesio. 2004. Discourse annotation and semantic annotation in the GNOME corpus. In Proceedings of the Workshop on Discourse Annotation, pages 72–79, Barcelona, Spain. Association for Computational Linguistics. ↗
14. Jessica Lin and Amir Zeldes. 2021. WikiGUM: Exhaustive entity linking for wikification in 12 genres. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 170–175, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
15. Massimo Poesio and Ron Artstein. 2008. Anaphoric annotation in the ARRAU corpus. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC‘08), Marrakech, Morocco. European Language Resources Association (ELRA). ↗
16. Ellen F. Prince. 1981. Toward a taxonomy of given-new information. Radical pragmatics, pages 223–255. find
17. Katja Markert, Yufang Hou, and Michael Strube. 2012. Collective classification for fine-grained information status. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 795–804, Jeju Island, Korea. Association for Computational Linguistics. find
18. Marta Recasens, Eduard Hovy, and M. Antònia Martí. 2010. A typology of near-identity relations for coreference (NIDENT). In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA). ↗
19. Natalia N Modjeska. 2004. Resolving other-anaphora. find
20. Anna Nedoluzhko, Jiří Mírovský, and Petr Pajas. 2009. The coding scheme for annotating extended nominal coreference and bridging anaphora in the Prague dependency treebank. In Proceedings of the Third Linguistic Annotation Workshop (LAW III), pages 108–111, Suntec, Singapore. Association for Computational Linguistics. ↗
21. Ina Roesiger. 2016. SciCorp: A corpus of English scientific articles annotated for information status analysis. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1743–1749, Portorož, Slovenia. European Language Resources Association (ELRA). ↗ our card
22. Ina Rösiger. 2018. BASHI: A corpus of Wall Street Journal articles annotated with bridging links. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). ↗
23. Malvina Nissim, Shipra Dingare, Jean Carletta, and Mark Steedman. 2004. An annotation scheme for information status in dialogue. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC‘04), Lisbon, Portugal. European Language Resources Association (ELRA). ↗
24. Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175–190, Seattle, United States. Association for Computational Linguistics. ↗
25. Tim O’Gorman, Kristin Wright-Bettner, and Martha Palmer. 2016. Richer Event Description: Integrating event coreference with temporal, causal and bridging annotation. In Proceedings of the 2nd Workshop on Computing News Storylines (CNS 2016), pages 47– 56, Austin, Texas. Association for Computational Linguistics. ↗
26. C OMPARISON Katrin Schweitzer, Kerstin Eckart, Markus Gärtner, Agnieszka Falenska, Arndt Riester, Ina Rösiger, Antje Schweitzer, Sabrina Stehwien, and Jonas Kuhn. 2018. German radio interviews: The GRAIN release of the SFB732 silver standard collection. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/levine-2025-subjectivity-annotation-bridging-anaphora/