MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

What it is about

The paper presents MCTest, a set of short fictional stories written for children, with four multiple-choice questions per story. Workers on Amazon Mechanical Turk wrote the stories and questions. Quality was held by a grammar pre-test (80% pass rate), by a vocabulary check against about 8,000 words a seven-year-old is likely to know, by rules that require one answer to be unique and most questions to need two sentences, and by a second crowd round that rated each story. The corpus has 500 stories (MC500) and 160 curated stories (MC160). Two lexical baselines (a sliding window over words, and the same with a distance penalty) reach about 57% on the MC500 test set and 66% on MC160. A textual-entailment system scores 53.5% on the MC160 test set (below the baseline) and 59.8% on the MC500 test set (above the baseline's 56.7%); the paper's table combines the two and reaches 60.8% on MC500. The crowd judges score near 100%. The task is to show that reading comprehension of open-domain text is still hard for simple systems.

Ideas

• I1. A multiple-choice reader for tales that explains each choice. world/src/ans.rs and world/src/ftqa.rs answer from the world state. Run them on the MC500 test set as a measurement (not as training) and report the share of answers with a stated reason, plus the accuracy against the two lexical baselines. Then build the same kind of item for open tales (Gutenberg) with the teacher, and compare the reader with the lexical baseline on held-out items by two-fold cv.
• I2. Item gates in qagen, as the paper does. Four checks can be coded in Rust: each answer is unique; the correct answer is absent from the story, or at least two distractors are in it; at least half of the non-stopword terms of an answer must appear in the story when it is said to appear; and a vocabulary check against a child's word list. The school vocabulary in tools/school-sents can serve as the list. Negative control: insert one distractor that is an exact sentence from the story and check that the gate rejects it.
• I3. Single or multi-sentence labels for questions. The paper asks writers to mark whether a question needs one or several sentences, and reports accuracy for each group (Tables 4 and 5). Mova can get the same label from the world: the number of events the answer proof uses. Report accuracy per group, so a gain on multi-sentence questions is visible and not averaged away.
• I4. Guesser baselines. Before any reader is claimed, measure a guesser that picks the longest option, and one that picks the first option, on the Mova items. The grammar courses already report such guessers in their course records (see the commit history); the same floors should be reported for the tale items, so that a reader is compared with a floor and not with chance.

For Mova

Code: qagen (training questions for tale paragraphs), world/src/ans.rs, world/src/ftqa.rs, world/src/babi.rs (the level check), tools/school-sents, tools/gutenberg (the open tales that Mova may use for items). Chapter: 19. The lexicon: vocabulary, multiword expressions and register for the word-list gate. Licence note: the MCTest stories are CC BY-NC-SA in our library, so they are for measurement and tests only, never in what is built or released. Do not copy the stories, the questions or the answers into qagen's training data. Items that Mova generates from open tales with its own teacher are its own, and they go into releases only after the gates pass.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{richardson2013mctest,
  title = {{MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text}},
  author = {Richardson, Matthew and Burges, Christopher J.C. and Renshaw, Erin},
  booktitle = {Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  url = {https://aclanthology.org/D13-1020/}
}

References

24 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Agarwal and P. Mannem. 2011. Automatic Gap-fill Question Generation from Text Books. In Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications, 56–64. find
2. E. Breck, M. Light, G.S.Mann, E. Riloff, B. Brown, P. Anand, M. Rooth M. Thelen. 2001. Looking under the hood: Tools for diagnosing your question answering engine. In Proceedings of the workshop on Open-domain question answering, 12, 1-8. find
3. E. Charniak. 1972. Toward a Model of Children’s Story Comprehension. Technical Report, 266, MIT Artificial Intelligence Laboratory, Cambridge, MA. find
4. P. Clark, P. Harrison, and X. Yao. An Entailment-Based Approach to the QA4MRE Challenge. 2012. In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF) 2012. find
5. S. Cucerzan and E. Agichtein. 2005. Factoid Question Answering over Unstructured and Structured Content on the Web. In Proceedings of the Fourteenth Text Retrieval Conference (TREC). find
6. I. Dagan, O. Glickman, and B. Magnini. 2006. The PASCAL Recognising Textual Entailment Challenge. In J. Quiñonero-Candela, I. Dagan, B. Magnini, F. d'Alché-Buc (Eds.), Machine Learning Challenges. Lecture Notes in Computer Science, Vol. 3944, pp. 177-190, Springer. find
7. D. Goldwasser, R. Reichart, J. Clarke, D. Roth. 2011. Confidence Driven Unsupervised Semantic Parsing. In Proceedings of the 49 th Annual Meeting of the Association for Computational Linguistics, 1486-1495. find
8. E. Grois and D.C. Wilkins. 2005. Learning Strategies for Story Comprehension: A Reinforcement Learning Approach. In Proceedings of the Twenty Second International Conference on Machine Learning, 257-264. find
9. S.M. Harabagiu, S.J. Maiorano, and M.A. Pasca. 2003. Open-Domain Textual Question Answering Techniques. Natural Language Engineering, 9(3):1-38. Cambridge University Press, Cambridge, UK. find
10. L. Hirschman, M. Light, E. Breck, and J.D. Burger. 1999. Deep Read: A Reading Comprehension System. In Proceedings of the 37 th Annual Meeting of the Association for Computational Linguistics (ACL), 325-332. find
11. J. Horton and L. Chilton. 2010. The labor economics of paid crowdsourcing. In Proceedings of the 11 th ACM Conference on Electronic Commerce, 209-218. find
12. V. Kuperman, H. Stadthagen-Gonzalez, M. Brysbaert. 2012. Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods, 44(4):978-990. find
13. J.L. Leidner, T. Dalmas, B. Webber, J. Bos, C. Grover. 2003. Automatic Multi-Layer Corpus Annotation for Evaluating Question Answering Methods: CBC4Kids. In Proceedings of the 3rd International Workshop on Linguistically Interpreted Corpora. find
14. E.T. Mueller. 2010. Story Understanding Resources. http://xenia.media.mit.edu/~mueller/storyund/storyre s.html. ↗
15. G. Paolacci, J. Chandler, and P. Iperirotis. 2010. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making. 5(5):411-419. find
16. E. Stamatatos. 2009. A survey of modern authorship attribution methods. J. Am. Soc. Inf. Sci., 60:538– 556. find
17. A. Stern and I. Dagan. 2011. A Confidence Model for Syntactically-Motivated Entailment Proofs. In Proceedings of Recent Advances in Natural Language Processing (RANLP). find
18. L.R. Tang and R.J. Mooney. 2001. Using Multiple Clause Constructors in Inductive Logic Programming for Semantic Parsing. In Proceedings of the 12 th European Conference on Machine Learning (ECML), 466-477. find
19. G. Tur, D. Hakkani-Tur, and L.Heck. 2010. What is left to be understood in ATIS? Spoken Language Technology Workshop, 19-24. find
20. E.M. Voorhees and D.M. Tice. 1999. The TREC-8 Question Answering Track Evaluation. In Proceedings of the Eighth Text Retrieval Conference (TREC-8). find
21. B. Wellner, L. Ferro, W. Greiff, and L. Hirschman. 2005. Reading comprehension tests for computer-based understand evaluation. Natural Language Engineering, 12(4):305-334. Cambridge University Press, Cambridge, UK. find
22. J.M. Zelle and R.J. Mooney. 1996. Learning to Parse Database Queries using Inductive Logic Programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI), 1050-1055. find
23. L.S. Zettlemoyer and M. Collins. 2009. Learning Context-Dependent Mappings from Sentences to Logical Form. In Proceedings of the 47 th Annual Meeting of the Association for Computation Linguistics (ACL), 976-984. find
24. G. Zweig and C.J.C. Burges. 2012. A Challenge Set for Advancing Language Modeling. In Proceedings of the Workshop on the Future of Language Modeling for HLT, NAACL-HLT. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/missing-2013-richardson-mctest-challenge-dataset-open-domain/