PCR4ALL: A Comprehensive Evaluation Benchmark for Pronoun Coreference Resolution in English

What it is about

PCR4ALL converts eight pronoun resolution datasets (CoNLL-2012/OntoNotes, WSC, DPR, WinoGrande, KnowRef, the i2b2 medical records, CIC multi-party TV dialogue, WinoGender) into one format — a text with a target pronoun and two candidate noun phrases, pick the right one — about 130K questions split 8:1:1. Beyond overall error rate it reports by perspective: linguistic knowledge (only one candidate agrees with the pronoun in gender/number, decided by a rule system) vs world knowledge (both agree), formal vs casual language, medical domain, training size, word frequency, lexical relevance to training data (BM25), candidate «polarity» (memorised good candidates), and gender consistency (swap he/she, same prediction?). Off-the-shelf resolvers (CoreNLP, spaCy neuralcoref, AllenNLP) and language models fine-tuned or not show large differences across these perspectives; none is reliable on all.

Ideas

• I1. Split Mova's pronoun precision table (coref — SLM coreference: a sieve of rules with a reason for every link, by form) by the paper's main perspective: cases where agreement alone leaves one candidate in the window vs cases where two or more agreeing candidates remain. Computed deterministically from Mova's own features in coref/src/eval.rs on GUM dev. If errors concentrate in the first group, the fault is in features (gender, animacy, number); if in the second, in ranking (Hobbs order, salience). It decides between the two «What next» directions: an animacy lexicon or Lappin–Leass weights.
• I2. Add the binary-choice format as a small unit test set for the frame reader's resolver: a tale sentence, a pronoun, two candidates, the right one. It isolates resolution from mention detection and is cheap to grow from the harvested cases of emami-2019-knowref-coreference-corpus-removing-gender I1. Report error rate by perspective (agreement decides / does not).
• I3. Run the gender-consistency check on Mova's resolvers: replace he with she where no gendered noun or earlier pronoun fixes gender; the choice should not change when nothing in the text supports gender. A gate that shows red if a hidden gender guess sneaks in.

For Mova

35. Reference and coreference §8–10. Code: coref/src/eval.rs (perspective split), coref/src/sieve.rs (pronoun, compatible), prag/src/frames.rs (Ctx). Measures: pronoun precision by perspective on GUM dev, error rate on a tale binary-choice set, the he/she consistency gate. The component datasets carry their own licences (OntoNotes LDC, WinoGrande, i2b2 restricted) — tests only.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{zhao2022pcr4all,
  title = {{PCR4ALL: A Comprehensive Evaluation Benchmark for Pronoun Coreference Resolution in English}},
  author = {Zhao, Xinran and Zhang, Hongming and Song, Yangqiu},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  url = {https://aclanthology.org/2022.lrec-1.641/}
}

References

43 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amati, G., (2009). Encyclopedia of Database Systems: BM25, pages 257–260. Springer US, Boston, MA. find
2. Bengtson, E. and Roth, D. (2008). Understanding the value of features for coreference resolution. In Proceedings of EMNLP 2008, pages 294–303. find
3. Chang, K., Samdani, R., and Roth, D. (2013). A constrained latent variable model for coreference resolution. In Proceedings of EMNLP 2013, pages 601– 612. find
4. Chen, Y. and Choi, J. D. (2016). Character identification on multiparty conversation: Identifying mentions of characters in TV shows. In Proceedings of SIGDIAL 2016, pages 90–100. find
5. Chinchor, N. A. (1998). Overview of MUC-7/MET-2. In the Seventh Message Understanding Conference(MUC7). find
6. Clark, K. and Manning, C. D. (2015). Entity-centric coreference resolution with model stacking. In Proceedings of ACL 2015, pages 1405–1415. find
7. Clark, K. and Manning, C. D. (2016a). Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256–2262, Austin, Texas, November. Association for Computational Linguistics. find
8. Clark, K. and Manning, C. D. (2016b). Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653, Berlin, Germany, August. Association for Computational Linguistics. find
9. Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019). BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019, pages 4171–4186. find
10. Doddington, G., Mitchell, A., Przybocki, M., Ramshaw, L., Strassel, S., and Weischedel, R. (2004). The automatic content extraction (ACE) program – tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), pages 837–840, Lisbon, Portugal, May. European Language Resources Association (ELRA). find
11. Edens, R. J., Gaylard, H. L., Jones, G. J. F., and Lam-Adesina, A. M. (2003). An investigation of broad coverage automatic pronoun resolution for information retrieval. In Proceedings of SIGIR 2003, pages 381–382. find
12. Emami, A., Trichelair, P., Trischler, A., Suleman, K., Schulz, H., and Cheung, J. C. K. (2019). The knowref coreference corpus: Removing gender and number cues for difficult pronominal anaphora resolution. In Proceedings of ACL 2019, pages 3952– 3961. find our card
13. Emami, A., Suleman, K., Trischler, A., and Cheung, J. C. K. (2020). An analysis of dataset overlap on Winograd-style tasks. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5855–5865, Barcelona, Spain (Online), December. International Committee on Computational Linguistics. find
14. Grishman, R. and Sundheim, B. (1996). Message understanding conference- 6: A brief history. In Proceedings of COLING 1996, pages 466–471. find
15. Hobbs, J. R. (1978). Resolving pronoun references. Lingua, 44(4):311–338. find our card
16. Honnibal, M., Montani, I., Van Landeghem, S., and Boyd, A. (2020). spaCy: Industrial-strength Natural Language Processing in Python. find
17. Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., and Levy, O. (2020). Spanbert: Improving pre-training by representing and predicting spans. Trans. Assoc. Comput. Linguistics, 8:64–77. find
18. Kantor, B. and Globerson, A. (2019). Coreference resolution with entity equalization. In Proceedings of ACL 2019, pages 673–677. find
19. Katz, J. J. and Fodor, J. A. (1963). The structure of a semantic theory. Language, 39(2):170–210. find
20. Kocijan, V., Cretu, A., Camburu, O., Yordanov, Y., and Lukasiewicz, T. (2019). A surprisingly robust trick for the winograd schema challenge. In Proceedings of ACL 2019, pages 4837–4842. find
21. Lee, K., He, L., Lewis, M., and Zettlemoyer, L. (2017). End-to-end neural coreference resolution. In Proceedings of EMNLP 2017, pages 188–197. find our card
22. Lee, K., He, L., and Zettlemoyer, L. (2018). Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of NAACL-HLT 2018, pages 687–692. find
23. Levesque, H., Davis, E., and Morgenstern, L. (2012). The Winograd Schema Challenge. In Proceeedings of KRR 2012. find
24. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
25. Mitkov et al., R. (1995). Anaphora resolution in machine translation. In TMMT. find
26. Ng, V. (2005). Supervised ranking for pronoun resolution: Some recent improvements. In Proceedings of AAAI 2005, pages 1081–1086. find
27. Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P. S. H., Bakhtin, A., Wu, Y., and Miller, A. H. (2019). Language models as knowledge bases? In Kentaro Inui, et al., editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2463–2473. Association for Computational Linguistics. find
28. Pradhan, S., Ramshaw, L. A., Marcus, M. P., Palmer, M., Weischedel, R. M., and Xue, N. (2011). Conll-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of CoNLL 2011, pages 1–27. find our card
29. Pradhan, S., Moschitti, A., Xue, N., Uryupina, O., and Zhang, Y. (2012). Conll-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of CoNLL 2012, pages 1–40. find our card
30. Raganato, A., Camacho-Collados, J., and Navigli, R. (2017). Word sense disambiguation: A unified evaluation framework and empirical comparison. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 99–110, Valencia, Spain, April. Association for Computational Linguistics. find our card
31. Raghunathan, K., Lee, H., Rangarajan, S., Chambers, N., Surdeanu, M., Jurafsky, D., and Manning, C. (2010). A multi-pass sieve for coreference resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 492–501, Cambridge, MA, October. Association for Computational Linguistics. find our card
32. Rahman, A. and Ng, V. (2012). Resolving complex cases of definite pronouns: The winograd schema challenge. In Proceedings of EMNLP-CoNLL 2012, pages 777–789. find
33. Ribeiro, M. T., Wu, T., Guestrin, C., and Singh, S. (2020). Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912, Online, July. Association for Computational Linguistics. find
34. Rudinger, R., Naradowsky, J., Leonard, B., and Durme, B. V. (2018). Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pages 8–14. find
35. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. (2020). WinoGrande: an adversarial winograd schema challenge at scale. In Proceedings of AAAI 2020, pages 99–106. find
36. Steinberger, J., Poesio, M., Kabadjov, M. A., and Jeek, K. (2007). Two uses of anaphora resolution in summarization. Inf. Process. Manage., 43(6):1663–1680, nov. find
37. Strube, M. and Müller, C. (2003). A machine learning approach to pronoun resolution in spoken dialogue. In Proceedings of ACL 2003, pages 168–175. find
38. Uzuner, O., Bodnari, A., Shen, S., Forbush, T., Pestian, J., and South, B. R. (2012). Evaluating the state of the art in coreference resolution for electronic medical records. J. Am. Medical Informatics Assoc., 19(5):786–791. find
39. Wu, W., Wang, F., Yuan, A., Wu, F., and Li, J. (2020). Corefqa: Coreference resolution as query-based span prediction. In Dan Jurafsky, et al., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 6953–6963. Association for Computational Linguistics. find
40. Zhang, H., Song, Y., and Song, Y. (2019a). Incorporating context and external knowledge for pronoun coreference resolution. In Proceedings of NAACL-HLT 2019, pages 872–881. find
41. Zhang, H., Song, Y., Song, Y., and Yu, D. (2019b). Knowledge-aware pronoun coreference resolution. In Proceedings of ACL 2019, pages 867–876. find
42. Zhang, H., Zhao, X., and Song, Y. (2020). A brief survey and comparative study of recent development of pronoun coreference resolution. CoRR, abs/2009.12721. ↗
43. Zhao, J., Wang, T., Yatskar, M., Ordonez, V., and Chang, K. (2018). Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of NAACL-HLT 2018, pages 15–20. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/zhao-2022-pcr4all-comprehensive-evaluation-benchmark-pronoun/