STARC: Structured Annotations for Reading Comprehension

What it is about

STARC is a scheme for four-option reading questions in which the options have fixed roles tied to marked text spans: A is right and rests on a «critical span»; B mentions the critical span but misunderstands it; C refers to a separate «distractor span»; D has no support in the text. Questions avoid arithmetic, outside knowledge and speculation, and options are written so that elimination tricks do not work. OneStopQA applies it to 30 Guardian articles in three difficulty versions (162 paragraphs, 486 questions, 1,458 question–paragraph pairs). Ablations show the value: without the passage RoBERTa guesses 37.2% on OneStopQA but 47.1% on RACE; removing the critical span drops accuracy towards chance, as it should. Crowd judgements find 18.3% of RACE questions without a single right answer against 3.4% for OneStopQA, giving approximate ceilings of 88.8% and 97.9%; the structured options also show what kind of misreading each wrong choice is.

Ideas

• I1. Classify the student's wrong answers with the STARC types, by rule from the trace: the chosen option matched the frame that answers the question but with a wrong role or polarity (B — misread the right place), matched another frame of the lesson (C — wrong place), or matched nothing (D — guess). Add the split to world knowledge diagnose; B-type errors point at role and negation handling, C-type at retrieval and salience.
• I2. Right for the right reason: when lessons and questions are made (the teacher writes them, docs/school-making.md), also store the critical sentence for each question. Then check that the frame the student used comes from that sentence; an answer right but resting on another sentence is counted apart. This is a cheap guard against lucky matches in the course numbers.
• I3. Critical-span ablation as a validity check of course questions: remove the critical sentence; the student should fall to guessing on that question. Questions still answered right are guessable from elsewhere or from the options and should be fixed or marked.
• I4. Ask the teacher to write options in the A/B/C/D structure for new courses, so B-type options test reading of the right sentence directly; check on a pilot that the student's score on B-type distractors differs from C and D.

For Mova

30. Pragmatics and discourse: overview §9 (measurements). Code: world/src/knowledge.rs (diagnosis, evidence check), world/src/school.rs, world/src/lessons.rs; docs/school-making.md for the teacher's question format. Measures: error-type shares B/C/D per course, share of right answers resting on the critical sentence, accuracy with the critical sentence removed.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.507

@inproceedings{berzak2020starc,
  title = {{STARC: Structured Annotations for Reading Comprehension}},
  author = {Berzak, Yevgeni and Malmaud, Jonathan and Levy, Roger},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.507},
  url = {https://doi.org/10.18653/v1/2020.acl-main.507}
}

References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016. Analyzing the behavior of visual question answering models. In EMNLP, pages 1955–1960. find
2. Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2016. Embracing data abundance: Booktest dataset for reading comprehension. arXiv preprint arXiv:1610.00956. — [arXiv:1610.00956](https://arxiv.org/abs/1610.00956) find
3. Wei-Lun Chao, Hexiang Hu, and Fei Sha. 2018. Being negative but constructively: Lessons learnt from creating better visual question answering datasets. In NAACL-HLT, pages 431–441. find
4. Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the CNN/Daily mail reading comprehension task. In ACL, pages 2358–2367. find
5. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In CVPR. find
6. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NeurIPS, pages 1693–1701. find
7. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The Goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301. find
8. Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. 2016. Revisiting visual question answering baselines. In ECCV, pages 727–739. Springer. find
9. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551. find
10. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In NAACL, pages 252–262. find our card
11. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (Automated Readability Index, Fog count and Flesch reading ease formula) for navy enlisted personnel. find
12. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… find
13. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale reading comprehension dataset from examinations. In EMNLP, pages 785–794. find our card
14. G. Harry MC Laughlin. 1969. SMOG grading-a new readability formula. Journal of Reading, 12(8):639– 646. find
15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
16. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human-generated machine reading comprehension dataset. find
17. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, pages 2383–2392. find
18. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP, pages 193–203. find our card
19. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. find
20. Sowmya Vajjala and Ivana Lučić. 2018. OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification. Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 297–304. find
21. Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, Dong Yu, David McAllester, and Dan Roth. 2019. Evidence sentence extraction for machine reading comprehension. In CoNLL, pages 696–707. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/berzak-2020-starc-structured-annotations-reading-comprehension/