Corpus Annotation for Parser Evaluation

What it is about

Bracket-based parser scores (PARSEVAL) are hard to compare across grammars, because the bracketing of a treebank and the output of a parser differ in ways that are not errors. The authors propose an annotation scheme that records grammatical relations between heads and dependents (subject, object, modifier and so on), organised in a hierarchy so that a shallow parser may leave a label underspecified. They mark up a corpus of 500 sentences (10K words) taken from SUSANNE, with 9.72 relations per sentence and around 95% inter-annotator agreement. A robust lexicalised parser scores an F-score of 75.1 on the general dependent relation, 86.5 on non-clausal subjects and 50.7 on indirect objects.

Ideas

• I1. en parse-eval: add a per-label table (precision, recall, F) for the UD labels on EWT test, in the paper's style, with the general dep score first. Measure which labels lower LAS most, and start with the lowest-F label.
• I2. Hierarchical diagnostic: count a predicted label as a near miss when it shares a parent with the gold label (nsubj and csubj, obl and nmod). Report this beside LAS as a diagnostic only, never as a tuning target. Measure: the gap between LAS and the near-miss score on EWT test.
• I3. Control in en/src/expert.rs: a rule that adds a second, enhanced edge when one nominal is the subject of two verbs (the paper's example of control). Trees stay single-headed; the extra edge is reported only. Measure: nsubj F on EWT test with and without the enhanced edges, and the number of such cases.

For Mova

The chapters are 20. Syntax: the Universal Dependencies frame and 23-complementation.md (control). The code is en/src/parse.rs with the en parse-eval command, and en/src/expert.rs for the control rule. The paper's corpus is not in Mova's data, and its numbers belong to a 1999 parser, so they are not comparable. Every figure must come from EWT test, which is measured once per release.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv cs/9907013

@inproceedings{carroll1999corpus,
  title = {{Corpus Annotation for Parser Evaluation}},
  author = {Carroll, John and Minnen, Guido and Briscoe, Ted},
  booktitle = {Proceedings of the EACL99 workshop on Linguistically Interpreted Corpora (LINC), Bergen, Norway, June 12},
  year = {1999},
  eprint = {cs/9907013},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/cs/9907013}
}

References

24 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Atwell, E. (1996) Comparative evaluation of grammatical annotation models. In R. Sutcliffe, H. Koch & A. McElligott (Eds.), Industrial Parsing of Software Manuals, 25–46. Amsterdam: Rodopi. find
2. Bies, A., Ferguson, M., Katz, K., MacIntyre, R., Tredinnick, V., Kim, G., Marcinkiewicz, M., Schasberger, B. (1995) Bracketing guidelines for Treebank II style Penn Treebank Project. Technical Report, CIS, University of Pennsylvania, Philadelphia, PA. find
3. Carpenter, B. & Manning, C. (1997) Probabilistic parsing using left corner language models. In Proceedings of the 5th ACL/SIGPARSE International Workshop on Parsing Technologies. MIT, Cambridge, MA. find
4. Carroll, J., Briscoe E. & Sanfilippo, A. (1998) Parser evaluation: a survey and a new proposal. In Proceedings of the International Conference on Language Resources and Evaluation, 447–454. Granada, Spain. Available online at <ftp://ftp.cogs.susx.ac.uk/pub/users/johnca/ lre98-final.ps>. find
5. Carroll, J., Minnen, G. & Briscoe E. (1998) Can subcategorisation probabilities help a statistical parser?. In Proceedings of the 6th ACL/SIGDAT Workshop on Very Large Corpora. Montreal, Canada. Available online at <http://xxx.lanl.gov/abs/cmp-lg/9806013>. ↗
6. Charniak, E. (1996) Tree-bank grammars. In Proceedings of the 13th National Conference on Artificial Intelligence, AAAI-96, 1031–1036. . find
7. Collins, M. (1996) A new statistical parser based on bigram lexical dependencies. In Proceedings of the 34th Meeting of the Association for Computational Linguistics, 184–191. Santa Cruz, CA. find
8. Gaizauskas, R. (1998) Special issue on evaluation. Computer Speech & Language. find
9. Gaizauskas, R., Hepple M. & Huyck, C. (1998) Modifying existing annotated corpora for general comparative evaluation of parsing. In Proceedings of the LRE Workshop on Evaluation of Parsing Systems. Granada, Spain. find
10. Grishman, R., Macleod, C. & Sterling, J. (1992) Evaluating parsing strategies using standardized parse files. In Proceedings of the 3rd ACL Conference on Applied Natural Language Processing, 156–161. Trento, Italy. find
11. Harrison, P., Abney, S., Black, E., Flickinger, D., Gdaniec, C., Grishman, R., Hindle, D., Ingria, B., Marcus, M., Santorini, B. & Strzalkowski, T. (1991) Evaluating syntax performance of parser/grammars of English. In Proceedings of the Workshop on Evaluating Natural Language Processing Systems. ACL. find
12. Leech, G. & Garside, R. (1991) Running a grammar factory: the production of syntactically analysed corpora or ‘treebanks’. In S. Johansson & A. Stenstrom (Eds.), English Computer Corpora: Selected Papers and Bibliography, Mouton de Gruyter, Berlin. find
13. Lehmann, S., Oepen, S., Regnier-Prost, S., Netter, K., Lux, V., Klein, J., Falkedal, K., Fouvry, F., Estival, D., Dauphin, E., Compagnion, H., Baur, J., Balkan, L. & Arnold, D. (1996) tsnlp — test suites for natural language processing. In Proceedings of the International Conference on Computational Linguistics, COLING-96, 711–716. Copenhagen, Denmark. find
14. Lin, D. (1995) A dependency-based method for evaluating broad-coverage parsers. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, 1420–1425. Montreal, Canada. find
15. Lin, D. (1996) Dependency-based parser evaluation: a study with a software manual corpus. In R. Sutcliffe, H-D. Koch & A. McElligott (Eds.), Industrial Parsing of Software Manuals, 13–24. Amsterdam, The Netherlands: Rodopi. find
16. Marcus, M., Santorini, B. & Marcinkiewicz (1993) Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2), 313–330. find our card
17. Roland, D. & Jurafsky, D. (1998) How verb subcategorization frequencies are affected by corpus choice. In Proceedings of the 17th International Conference on Computational Linguistics (COLING-ACL’98), 1122–1128. Montreal, Canada. find
18. Rubio, A. (Ed.) (1998) International Conference on Language Resources and Evaluation. Granada, Spain. find
19. Sampson, G. (1995) English for the Computer. Oxford, UK: Oxford University Press. find
20. Sanfilippo, A., Barnett, R., Calzolari, N., Flores, S., Hellwig, P., Leech, P., Melero, M., Montemagni, S., Odijk, J., Pirrelli, V., Teufel, S., Villegas M. & Zaysser, L. (1996) Subcategorization Standards. Report of the EAGLES Lexicon/Syntax Interest Group. Available through eagles@ilc.pi.cnr.it. find
21. Sekine, S. (1997) The domain dependence of parsing. In Proceedings of the 5th ACL Conference on Applied Natural Language Processing, 96–102. Washington, DC. find
22. Srinivas, B., Doran, C., Hockey B. & Joshi A. (1996) An approach to robust partial parsing and evaluation metrics. In Proceedings of the ESSLLI’96 Workshop on Robust Parsing. Prague, Czech Republic. find
23. Srinivas, B., Doran, C. & Kulick, S. (1995) Heuristics and parse ranking. In Proceedings of the 4th ACL/SIGPARSE International Workshop on Parsing Technologies. Prague, Czech Republic. find
24. van Rijsbergen, C. (1979) Information Retrieval. Butterworth, London. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/enud-1999-carroll-corpus-annotation-parser-evaluation/