GUM-SAGE: A Novel Dataset and Approach for Graded Entity Salience Prediction

What it is about

Salience is defined operationally: an entity is as salient as the number of one-sentence summaries of the document (out of five) that mention it, giving a 0–5 score for every named and unnamed entity of GUM's 213 documents in 12 genres. Training documents get one expert summary and four LLM summaries; dev and test get human summaries and manual alignment. Mentions are aligned to summaries by an ensemble of string match, a coreference model run over document plus summary, and GPT-4o judgements, combined by logistic regression with entity type, genre and position. On average 13.8% of entities are salient at all, 2.1% score 5. Predicting salience from the document alone, the summary-based approach correlates best with the human scores (Spearman 0.54); GPT-4o prompted directly reaches only 0.23–0.25 and over-predicts mid scores, early entities and persons/places, and does worst on abstract, event and time entities. A position baseline (earlier first mention = more salient) is included.

Ideas

• I1. Compute a document-level salience score from Mova's own chains, with no learning: chain length, position of the first mention, share of mentions in subject position, presence in the first sentence or title. Score it against GUM-SAGE on GUM dev (Spearman, top-1/top-3 F1) next to the paper's position baseline, using coref output on gold and SLM trees. A measure the coref work currently lacks: which chains matter, not only whether links are right.
• I2. Use that global salience in pronoun resolution as a tie-breaker after agreement and Hobbs order: the frame reader's Ctx already uses local salience (best role per sentence, halved per sentence); a small bonus for the document's top entity would help he/she in long stretches about the protagonist. Measure he/she/his precision on GUM dev and frame bench coref F1.
• I3. In the student, weight remembered frames by the salience of their entities so that options about the protagonist outrank options about a passing character when both are supported (step «right option outscored» in Watching the SLM think: the diagnosis of its answers, 21% on the Torah course). Measure the course test counts with the weight on and off.
• I4. For lessons the teacher can write a one-sentence summary once per lesson; entities present in it become the salient set, exactly the paper's definition. A cheap silver signal for I3 without per-entity labelling; check its agreement with I1 on a few lessons first.

For Mova

35. Reference and coreference §2.4 (salience in practice), §10. Code: coref/src/eval.rs (salience score and its evaluation), prag/src/frames.rs (Ctx salience), world/src/knowledge.rs (weights). Measures: Spearman with GUM-SAGE on dev, pronoun precision on GUM dev, frame bench coref F1, course test counts.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-acl.24 · arXiv 2504.10792

@inproceedings{lin2025gum,
  title = {{GUM-SAGE: A Novel Dataset and Approach for Graded Entity Salience Prediction}},
  author = {Lin, Jessica and Zeldes, Amir},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-acl.24},
  eprint = {2504.10792},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2025.findings-acl.24}
}

References

32 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. 2024. The Claude 3 model family: Opus, Sonnet, Haiku. https://assets.anthropic.com/m/61e7d27f8c8f5919/ original/Claude-3-Model-Card.pdf. ↗
2. Tatsuya Aoyama, Shabnam Behzad, Luke Gessler, Lauren Levine, Jessica Lin, Yang Janet Liu, Siyao Peng, Yilun Zhu, and Amir Zeldes. 2023. GENTLE: A genre-diverse multilayer challenge set for English NLP and linguistic evaluation. In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pages 166–178, Toronto, Canada. Association for Computational Linguistics. ↗ our card
3. Eliyar Asgarieh, Kapil Thadani, and Neil O’Hare. 2024. Scalable detection of salient entities in news articles. ↗
4. Benjamin Bullough, Harrison Lundberg, Chen Hu, and Weihang Xiao. 2024. Predicting entity salience in extremely short documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 50–64, Miami, Florida, US. Association for Computational Linguistics. ↗
5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
6. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308. find our card
7. Milan Dojchinovski, Dinesh Reddy, Tomáš Kliegr, Tomáš Vitvar, and Harald Sack. 2016. Crowdsourced corpus with entity salience annotations. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3307–3311. ↗
8. Jesse Dunietz and Dan Gillick. 2014. A new entity salience task with millions of training examples. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 205–209. ↗
9. Michael Gamon, Tae Yano, Xinying Song, Johnson Apacible, and Patrick Pantel. 2013. Identifying salient entities in web pages. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pages 2375–2380. ↗
10. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. find
11. Nghia T Le and Alan Ritter. 2023. Are large language models robust coreference resolvers? arXiv preprint arXiv:2305.14489. ↗
12. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
13. Jessica Lin and Amir Zeldes. 2021. WikiGUM: Exhaustive entity linking for wikification in 12 genres. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 170–175, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
14. Jessica Lin and Amir Zeldes. 2024. GUMsley: Evaluating entity salience in summarization for 12 English genres. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2575–2588, St. Julian’s, Malta. Association for Computational Linguistics. ↗
15. Houjun Liu, John Bauer, Karel D’Oosterlinck, Christopher Potts, and Christopher D. Manning. 2024. MSCAW-coref: Multilingual, singleton and conjunction-aware word-level coreference resolution. In Proceedings of The Seventh Workshop on Computational Models of Reference, Anaphora and Coreference, pages 33–40, Miami. Association for Computational Linguistics. ↗
16. Janet Yang Liu and Amir Zeldes. 2023a. GUMSum: Multi-genre data and evaluation for English abstractive summarization. In Findings of ACL 2023, Toronto. ↗
17. Yang Janet Liu and Amir Zeldes. 2023b. Why can‘t discourse parsing generalize? A thorough investigation of the impact of data diversity. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3112–3130, Dubrovnik, Croatia. Association for Computational Linguistics. find
18. Zhengzhong Liu, Chenyan Xiong, Teruko Mitamura, and Eduard Hovy. 2018. Automatic event salience identification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1226–1236, Brussels, Belgium. Association for Computational Linguistics. ↗
19. Mounica Maddela, Mayank Kulkarni, and Daniel Preotiuc-Pietro. 2022. EntSUM: A data set for entity-centric extractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3355–3366, Dublin, Ireland. Association for Computational Linguistics. ↗
20. Meta AI. 2024. Introducing meta llama 3: The most capable openly available llm to date. Accessed: October 22, 2024. find
21. Anna Nedoluzhko, Michal Novák, Martin Popel, Zdeněk Žabokrtský, Amir Zeldes, and Daniel Zeman. 2022. CorefUD 1.0: Coreference meets Universal Dependencies. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4859–4872, Marseille, France. European Language Resources Association. ↗ our card
22. OpenAI. 2024. Gpt-4 Technical Report. https:// openai.com/index/gpt-4-research/. ↗
23. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
24. Ivan Sekulic, Krisztian Balog, and Fabio Crestani. 2024. Towards self-contained answers: Entity-based answer rewriting in conversational search. In Proceedings of the 2024 Conference on Human Information Interaction and Retrieval, pages 209–218. ↗
25. Kawshik Sundar, Shubham Toshniwal, Makarand Tapaswi, and Vineet Gandhi. 2024. Major entity identification: A generalizable alternative to coreference resolution. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11679–11695. ↗
26. Salvatore Trani, Claudio Lucchese, R. Perego, David E. Losada, Diego Ceccarelli, and Salvatore Orlando. 2018. SEL: A unified algorithm for salient entity linking. Computational Intelligence, 34:2 – 29. ↗
27. Chuan Wu, Evangelos Kanoulas, Maarten de Rijke, and Wei Lu. 2020. Wn-salience: A corpus of news articles with entity salience annotations. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2095–2102. find
28. Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. ↗
29. Amir Zeldes. 2022. Can we fix the scope for coreference? Problems and solutions for benchmarks beyond OntoNotes. Dialogue & Discourse, 13(1):41– 62. ↗ our card
30. Shuo Zhang and Amir Zeldes. 2017. GitDOX: A linked version controlled online XML editor for manuscript transcription. In Proceedings of the Thirtieth International Florida Artificial Intelligence Research Society Conference (FLAIRS 2017), pages 619–623. find
31. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 1097–1100. ↗
32. Yilun Zhu, Sameer Pradhan, and Amir Zeldes. 2021. OntoGUM: Evaluating contextualized SOTA coreference resolution on 12 more genres. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 461–467, Online. Association for Computational Linguistics. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lin-2025-gum-sage-novel-dataset-approach/