Different Flavors of GUM: Evaluating Genre and Sentence Type Effects on Multilayer Corpus Annotation Quality

What it is about

Tagging and parsing accuracy is usually reported on newswire, and genre is the usual explanation when it falls elsewhere. The authors ask whether sentence type (declarative, imperative, question, fragment, and so on) explains annotation quality better than genre. Using the GUM corpus (54 documents, about 44,000 tokens, four genres: news, interviews, travel guides, how-to guides), they compare uncorrected student annotations, the corrected gold, and three automatic taggers and one parser, with mixed-effects models that include document as a random effect and sentence length as a control. Sentence type predicts errors across the three tasks (POS tagging, dependency correction, coreference), and genre mostly disappears once document is in the model. Fragments and non-declarative types are the hardest for the taggers and the parser, and imperatives, which the EWT paper singles out, are not a strong problem.

Ideas

• I1. en tag-eval and en ud-eval broken down by sentence type (decl, imp, q, wh, frag, sub, inf) and by genre, with a mixed-effects check that sentence type stays significant when document is a random effect; measure UPOS and LAS per sentence type on GUM test (test only).
• I2. en/src/ptag.rs: a sentence-type feature from the first word and the presence of a finite verb, to help the tagger on fragments and headings; measure UPOS on fragments and gerund headings in GUM test before and after.
• I3. annot routing: send fragment and other-type sentences to a second teacher pass with a stricter gate, since the paper shows they carry most errors; measure gate violations per 1k silver sentences by sentence type with treeagree check.
• I4. en/src/ctx.rs: document-level features (genre tag or a heading flag) only where they add to the sentence-type model; measure with the mixed-effects check, not by accuracy alone, since the paper finds genre adds nothing once sentence type is in.

For Mova

Touches languages/en/37-register-and-variation, 10-word-classes and 38-dialogue-and-discourse-markers, and the en crate (ptag.rs, ctx.rs, parse.rs), annot and tools/treeagree. GUM is non-commercial: test only, never in a shipped model. Measure first the per-sentence-type breakdown of the current en scores on GUM test, which costs nothing and shows whether fragments are the weak spot.

Topics: Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-1709

@inproceedings{zeldes2016different,
  title = {{Different Flavors of GUM: Evaluating Genre and Sentence Type Effects on Multilayer Corpus Annotation Quality}},
  author = {Zeldes, Amir and Simonson, Dan},
  booktitle = {Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)},
  year = {2016},
  doi = {10.18653/v1/w16-1709},
  url = {https://doi.org/10.18653/v1/w16-1709}
}

References

3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Emilia Apostolova, Noriko Tomuro, Pattanasak Mongkolwat and Dina Demner-Fushman. 2012. Domain Adaptation of Coreference Resolution for Radiology Reports. In Proceedings of the 2012 Workshop on Biomedical Natural Language Processing (BioNLP 2012). Montreal, 118–121. R. Harald. find
2. Baayen. 2008. Analyzing Linguistic Data. A Practical Introduction to Statistics using R. Cambridge: Cambridge University Press. find
3. Douglas Biber. 2009. Multi-Dimensional Approaches. In Anke Lüdeling & Merja Kytö (eds.), Corpus find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2016-zeldes-different-flavors-gum-evaluating-genre/