An Evaluation of Parser Robustness for Ungrammatical Sentences

What it is about

The question is whether a dependency parser that is accurate on clean text also keeps its structure when the input is ungrammatical. The authors compare eight parsers, trained on the Penn Treebank and on Tweebank, on 10,000 learner sentences (FCE) and 10,000 machine-translation outputs, each paired with its corrected version. Robustness is scored without a gold tree: the parse of the corrected sentence is the reference, and dependencies that touch an extra or missing word are ignored. The best ESL robustness F1 rises from 93.72 (Turbo, PTB) to 94.36 (Malt, Tweebank), and robustness falls faster with the number of errors on MT than on ESL. Missing-word errors are the hardest type, replacement errors the easiest, and errors on closed-class words (prepositions, determiners) hurt more than errors on open-class words in most settings.

Ideas

• I1. en parse-eval --robust: parse the ungrammatical sentence and its corrected version with en, then score shared dependencies over the unmatched words as the paper defines; measure the robustness F1 on learner text that we write or license openly, and on FCE as test only.
• I2. en/seeds/errors/: check the missing-word rules (missing determiner, auxiliary, preposition) against the robustness analysis; measure the flag recall for missing versus unnecessary words on the same learner sample.
• I3. en/src/ptag.rs and en/src/parse.rs: a closed-class lexicon feature, so that a wrong preposition or determiner is not allowed to cascade into the head choice; measure robustness F1 on closed-class errors before and after.
• I4. en beam-curve extended: plot robustness F1 against the number of errors per sentence for the greedy and the beam-8 parser; measure the slope for each, as the paper does for its eight parsers.

For Mova

Touches languages/en/12-pronouns-determiners-numerals, 23-complementation and 27-negation, the en crate (parse.rs, ptag.rs), en/seeds/errors/ and the en parse-eval command. FCE is research-only: test only. Measure first the robustness F1 of the current en parser on a small set of sentences we can release.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1182

@inproceedings{hashemi2016evaluation,
  title = {{An Evaluation of Parser Robustness for Ungrammatical Sentences}},
  author = {Hashemi, Homa B. and Hwa, Rebecca},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1182},
  url = {https://doi.org/10.18653/v1/d16-1182}
}

References

36 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. arXiv preprint arXiv:1603.06042. ↗
2. Johnny Bigert, Jonas Sjöbergh, Ola Knutsson, and Magnus Sahlgren. 2005. Unsupervised evaluation of parser robustness. In Computational Linguistics and Intelligent Text Processing, pages 142–154. find
3. E. Black, S. Abney, S. Flickenger, C. Gdaniec, C. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini, and T. Strzalkowski. 1991. A procedure for quantitatively comparing the syntactic coverage of English grammars. In Proceedings of the DARPA Speech and Natural Language Workshop, pages 306– 311. find
4. Bernd Bohnet. 2010. Very high accuracy and fast dependency parsing is not a contradiction. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 89–97. find
5. Aoife Cahill. 2015. Parsing learner text: to shoehorn or not to shoehorn. In Proceedings of LAW IX - The 9th Linguistic Annotation Workshop, page 144. find
6. Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750. find our card
7. Jinho D Choi, Joel Tetreault, and Amanda Stent. 2015. It depends: Dependency parser comparison using a web-based evaluation tool. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pages 26–31. find our card
8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493– 2537. find
9. Joachim Daiber and Rob van der Goot. 2016. The denoised web treebank: Evaluating dependency parsing under noisy input conditions. In LREC. find our card
10. Marie-Catherine De Marneffe, Bill MacCartney, Christopher D Manning, et al. 2006. Generating typed dependency parses from phrase structure parses. In LREC, number 2006, pages 449–454. find our card
11. Jennifer Foster, Özlem Çetinoglu, Joachim Wagner, Joseph Le Roux, Stephen Hogan, Joakim Nivre, Deirdre Hogan, Josef Van Genabith, et al. 2011. # hardtoparse: POS tagging and parsing the twitterverse. In proceedings of the Workshop On Analyzing Microtext (AAAI 2011), pages 20–25. find
12. Jennifer Foster. 2004. Parsing ungrammatical input: an evaluation procedure. In LREC. find our card
13. Jennifer Foster. 2007. Treebanks gone bad. International Journal of Document Analysis and Recognition, 10(3-4):129–145. find
14. Jennifer Foster. 2010. “cba to check the spelling” investigating parser performance on discussion forum posts. In The Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 381–384. find our card
15. Jeroen Geertzen, Theodora Alexopoulou, and Anna Korhonen. 2013. Automatic linguistic annotation of large scale l2 databases: the EF-Cambridge open language database (EFCamDat). In Proceedings of the 31st Second Language Research Forum. find
16. Daniel Gildea. 2001. Corpus variation and parser performance. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 167–202. find our card
17. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2011. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In ACL-HLT, pages 42–47. find
18. Tuomo Kakkonen. 2007. Robustness evaluation of two CCG, a PCFG and a link grammar parsers. Proceedings of the 3rd Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics. find
19. Lingpeng Kong and Noah A Smith. 2014. An empirical comparison of parsing methods for stanford dependencies. arXiv preprint arXiv:1404.4314. ↗ our card
20. Lingpeng Kong, Nathan Schneider, Swabha Swayamdipta, Archna Bhatia, Chris Dyer, and Noah A Smith. 2014. A dependency parser for tweets. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
21. Jonathan K Kummerfeld, David Hall, James R Curran, and Dan Klein. 2012. Parser showdown at the wall street corral: An empirical investigation of error types in parser output. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1048–1059. find our card
22. André FT Martins, Miguel Almeida, and Noah A Smith. 2013. Turning on the turbo: Fast third-order non-projective turbo parsers. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 617–622. find our card
23. David McClosky, Eugene Charniak, and Mark Johnson. 2010. Automatic domain adaptation for parsing. In The Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 28–36. find
24. Ryan McDonald and Joakim Nivre. 2011. Analyzing and integrating dependency parsers. Computational Linguistics, 37(1):197–230. find
25. Ryan T McDonald and Fernando CN Pereira. 2006. Online learning of approximate dependency parsing algorithms. In EACL. find our card
26. Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev, Gülsen Eryigit, Sandra Kübler, Svetoslav Marinov, and Erwin Marsi. 2007. Maltparser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(02):95–135. find
27. Niels Ott and Ramon Ziai. 2010. Evaluating dependency parsing performance on german learner language. Proceedings of the Ninth Workshop on Treebanks and Linguistic Theories (TLT-9), 9:175–186. find
28. Slav Petrov, Pi-Chuan Chang, Michael Ringgaard, and Hiyan Alshawi. 2010. Uptraining for accurate deterministic question parsing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 705–713. find
29. Marion Potet, Emmanuelle Esperança-Rodier, Laurent Besacier, and Hervé Blanchon. 2012. Collection of a large database of French-English SMT output corrections. In LREC, pages 4043–4048. find
30. Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2008. The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics, 34(2):257–287. find
31. Marwa Ragheb and Markus Dickinson. 2012. Defining syntax for learner language annotation. In COLING (Posters), pages 965–974. find
32. Mohammad Sadegh Rasooli and Joel Tetreault. 2015. Yara parser: A fast and accurate dependency parser. arXiv preprint arXiv:1503.06733. ↗
33. Brian Roark, Mary Harper, Eugene Charniak, Bonnie Dorr, Mark Johnson, Jeremy G Kahn, Yang Liu, Mari Ostendorf, John Hale, Anna Krasnyanskaya, et al. 2006. Sparseval: Evaluation metrics for parsing speech. In LREC. find
34. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In NAACL, pages 173–180. find
35. Huichao Xue and Rebecca Hwa. 2014. Redundancy detection in esl writings. In EACL, pages 683–691. find
36. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 180–189. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2016-hashemi-evaluation-parser-robustness-ungrammatical-sentences/