QUDeval: The Evaluation of Questions Under Discussion Discourse Parsing

What it is about

A QUD dependency parse links each answer sentence to an earlier anchor sentence by an implicit question; evaluating such free-form questions has been manual. QUDeval turns the theoretical constraints into four criteria per edge: language quality; answer compatibility (graded: the focus of the answer sentence answers the question, it answers only in its background, or not at all); givenness (the question may use only concepts that are discourse-old or inferable — «mediated» — from the text up to the anchor; a concept that first appears in the answer sentence is answer leakage, one that appears nowhere is hallucination); and anchor relevance. Three linguistics students annotated 2,190 questions over 51 news articles from four parsers (a fine-tuned pipeline, ChatGPT, GPT-4, Alpaca) and the crowd-written DCQA questions, with moderate agreement. All systems write fluent questions (>92%), but many violate the other criteria; the fine-tuned parser picks the best anchors but hallucinates most, GPT-4 is best at compatibility. As automatic metrics, reference-based similarity scores are barely above random, a simple rule (content-word overlap with the anchor) and GPT scoring do somewhat better, and GPT judges rate ChatGPT's own questions too high.

Ideas

• I1. Extend the givenness rule already in world/src/qud.rs (a lemma new in the answer is reported as leakage) with the paper's «mediated» class: a question lemma counts as given when it is a part, attribute or typical participant of a given entity (the door of a mentioned house, the beak of the crow). Take the part-whole and attribute links from Mova's ontology tables; report the share of leakage reports that disappear and check a sample by hand.
• I2. Grade answer compatibility in qud.rs as the paper does: a sentence answers by its focus (the asked slot is filled in the main clause) or only in the background (the slot is filled in a relative clause, apposition or subordinate clause). A background answer should not pop the question. Add the distinction and a test pair.
• I3. Record an anchor for every accommodated implicit question: the earliest sentence after which all the question's lemmas are given. This gives Mova's QUD output the same (question, anchor, answer) triple as the paper and makes the anchor-relevance check automatic.
• I4. When the teacher is asked to judge questions or labels that it, or a sibling model, produced, expect self-preference (the paper's GPT judges favoured ChatGPT's own questions). Mova's checks of teacher output should be rule-based or by people, not by the same teacher.

For Mova

36. Discourse relations and coherence §9 (questions under discussion), §11.5 (the QUD stack), 35-reference-and-coreference.md §3.2 (bridging, for mediated givenness), 29-ellipsis-word-order-information-structure.md (focus). Code: world/src/qud.rs, the ontology tables used by world. Measures: QUD tests with negative controls; leakage reports before and after the mediated rule.

Topics: Discourse analysis

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.325 · arXiv 2310.14520

@inproceedings{wu2023qudeval,
  title = {{QUDeval: The Evaluation of Questions Under Discussion Discourse Parsing}},
  author = {Wu, Yating and Mangla, Ritika and Durrett, Greg and Li, Junyi Jessy},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.325},
  eprint = {2310.14520},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.325}
}

References

46 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacopo Amidei, Paul Piwek, and Alistair Willis. 2018. Evaluation methodologies in automatic question generation 2013-2018. In Proceedings of the 11th International Conference on Natural Language Generation, pages 307–317. find
2. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational linguistics, 34(4):555–596. find
3. Nicholas Asher, Nicholas Michael Asher, and Alex Lascarides. 2003. Logics of conversation. Cambridge University Press. find
4. David I Beaver and Brady Z Clark. 2009. Sense and sensitivity: How focus determines meaning. John Wiley & Sons. find
5. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. ↗
6. Anton Benz and Katja Jasinskaja. 2017. Questions under discussion: From sentence to discourse. Discourse Processes, 54(3):177–186. find
7. Daniel Buring. 2008. What’s new (and what’s given) in the theory of focus? In Annual Meeting of the Berkeley Linguistics Society, volume 34, pages 403– 423. find
8. Shuyang Cao and Lu Wang. 2021. Controllable open-ended question generation with a new question type ontology. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6424–6439. find
9. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.14799. ↗
10. Tuhin Chakrabarty, Justin Lewis, and Smaranda Muresan. 2022. CONSISTENT: Open-ended question generation from news articles. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6954–6968, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. https://tacc.utexas.edu ↗
11. Kordula De Kuthy, Nils Reiter, and Arndt Riester. 2018. QUD-based annotation of discourse structure and information structure: Tool and evaluation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find our card
12. Xuan Long Do, Bowei Zou, Liangming Pan, Nancy Chen, Shafiq Joty, and Aiti Aw. 2022. Cohs-cqg: Context and history selection for conversational question generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 580–591. find
13. Sujatha Das Gollapalli and See-Kiong Ng. 2022. QSTS: A question-sensitive text similarity measure for question generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3835–3846. find
14. Jing Gu, Mostafa Mirshekari, Zhou Yu, and Aaron Sisto. 2021. ChainCQG: Flow-Aware Conversational Question Generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2061–2070. find
15. Yufang Hou. 2021. End-to-end neural information status classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1377–1388. find
16. Andrew Kehler and Hannah Rohde. 2017. Evaluating an expectation-driven question-under-discussion model of discourse interpretation. Discourse Processes, 54(3):219–238. ↗
17. Gangwoo Kim, Sungdong Kim, Kang Min Yoo, and Jaewoo Kang. 2022. Generating information-seeking conversations from unlabeled documents. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2362– 2378. find
18. Wei-Jen Ko, Te-yuan Chen, Yiyan Huang, Greg Durrett, and Junyi Jessy Li. 2020. Inquisitive question generation for high level text comprehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6544–6555. find our card
19. Wei-Jen Ko, Cutter Dalton, Mark Simmons, Eliza Fisher, Greg Durrett, and Junyi Jessy Li. 2022. Discourse comprehension: A question answering framework to represent sentence connections. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11752–11764. find
20. Wei-Jen Ko, Yating Wu, Cutter Dalton, Dananjay Srinivas, Greg Durrett, and Junyi Jessy Li. 2023. Discourse analysis via questions and answers: Parsing dependency structures of questions under discussion. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11181–11195. find our card
21. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. pages 193–203. find
22. Klaus Krippendorff. 2011. Computing Krippendorff’s alpha-reliability. find
23. Kalpesh Krishna and Mohit Iyyer. 2019. Generating question-answer hierarchies. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2321–2334. find
24. Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228–231. find
25. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
26. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. arXiv preprint arXiv:2303.16634. ↗
27. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. ChatGPT as a factual inconsistency evaluator for abstractive text summarization. arXiv preprint arXiv:2303.15621. ↗
28. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-interdisciplinary Journal for the Study of Discourse, 8(3):243–281. find our card
29. Katja Markert, Yufang Hou, and Michael Strube. 2012. Collective classification for fine-grained information status. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 795–804. find
30. Mao Nakanishi, Tetsunori Kobayashi, and Yoshihiko Hayashi. 2019. Towards answer-unaware conversational question generation. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 63–71. find
31. Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, Kuzman Ganchev, Annie Louis, Fantine Huot, Anders Sandholm, Dipanjan Das, and Mirella Lapata. 2023. Conditional generation with a question-answering blueprint. Transactions of the Association for Computational Linguistics, 11:974–996. ↗
32. Preksha Nema and Mitesh M Khapra. 2018. Towards a better metric for evaluating question generation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3950–3959. find
33. Benjamin Newman, Luca Soldaini, Raymond Fok, Arman Cohan, and Kyle Lo. 2023. A controllable QA-based framework for decontextualization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. find
34. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. find our card
35. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The Penn Discourse TreeBank 2.0. In Language Resources and Evaluation Conference. find our card
36. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Technical Report. find
37. Sudha Rao and Hal Daumé III. 2019. Answer-based adversarial training for generating clarification questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 143–155. find
38. Arndt Riester, Lisa Brunetti, and Kordula De Kuthy. 2018. Annotation guidelines for questions under discussion and information structure. Information structure in lesser-described languages. Studies in prosody and syntax, pages 403–443. ↗ our card
39. Craige Roberts. 2012. Information structure: Towards an integrated formal theory of pragmatics. Semantics and pragmatics, 5:6–1. ↗ our card
40. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca. ↗
41. Jan Van Kuppevelt. 1995. Discourse structure, topicality and questioning. Journal of linguistics, 31(1):109– 147. ↗
42. Manya Wadhwa, Jifan Chen, Junyi Jessy Li, and Greg Durrett. 2023. Using natural language explanations to rescale human judgments. arXiv preprint arXiv:2305.14770. ↗
43. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is ChatGPT a good NLG evaluator? A preliminary study. arXiv preprint arXiv:2303.04048. ↗
44. Matthijs Westera, Laia Mayol, and Hannah Rohde. 2020. TED-Q: TED talks and the questions they evoke. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1118–1127. find
45. Yating Wu, William Sheffield, Kyle Mahowald, and Junyi Jessy Li. 2023. Elaborative simplification as implicit questions under discussion. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. find our card
46. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/wu-2023-qudeval-evaluation-questions-under-discussion/