"cba to check the spelling": Investigating Parser Performance on Discussion Forum Posts

What it is about

The question is how a WSJ-trained parser copes with forum posts. The author evaluates the Berkeley parser (five split-merge cycles, trained on WSJ sections 2-21) on posts from a BBC Sport football forum, under three conditions: the parser's own tokenisation, gold tokens, and gold tokens with corrected spelling. The development set has 42 posts (185 sentences) and gives a Parseval F-score of 77.56, against 89.17 on WSJ section 23. Two kinds of change lift it to 80.27 on development and to 79.93 on the test set of 40 posts (170 sentences): input rewriting (all-caps sentences, "cos" to "because", removing "lol") and treebank rewriting. Spelling errors matter less than errors in function words, and the paper lists coordination, idioms and informal capitalisation as the main failures.

Ideas

• I1. Informal-form table in en/src/tok.rs: map closed forms such as "cos", "didnt" and "im" to standard tokens, with one explanation line per change in en explain. Measure en tok-eval and LAS on EWT test sentences that contain these forms.
• I2. All-caps normalisation before tagging in en/src/ptag.rs: when a whole sentence is uppercase, lowercase it and restore names from the lexicon. Measure tag accuracy and LAS on an uppercased copy of EWT test that we make ourselves.
• I3. Negative control for spelling correction in en/src/morph.rs: correct only out-of-lexicon tokens with a one-edit closed-class candidate, and log each change. Measure LAS on EWT test with misspelled function words added by script, and check that clean text does not change at all.

For Mova

The chapters are 06. Spelling rules: inflection, derivation, compounds and national standards, 05-writing-system.md and 37-register-and-variation.md. The code is en/src/tok.rs, en/src/ptag.rs, en/src/morph.rs and the en tok-eval and en parse-eval commands. The forum data is not in Mova's data. The paper's figures are Parseval scores for a constituency parser, so Mova's numbers must come from EWT test, held out until the release check.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{foster2010cba,
  title = {{"cba to check the spelling": Investigating Parser Performance on Discussion Forum Posts}},
  author = {Foster, Jennifer},
  booktitle = {Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
  url = {https://aclanthology.org/N10-1060/}
}

References

10 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AiTi Aw, Min Zhang, Juan Xiao, and Jian Su. 2006. A phrase-based statistical model for SMS text normalisation. In Proceedings of the 21st COLING/44th ACL. find
2. Ann Bies, Mark Ferguson, Karen Katz, and Robert Mac-Intyre. 1995. Bracketing guidelines for Treebank II style, Penn Treebank Project. Technical Report Tech Report MS-CIS-95-06, University of Pennsylvania. find
3. Daniel Bikel. 2004. Intricacies of Collins Parsing Model. Computational Linguistics, 30(4):479–511. find
4. Eugene Charniak and Mark Johnson. 2005. Course-to-fine n-best-parsing and maxent discriminative reranking. In Proceedings of the 43rd ACL. find
5. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Reranking and self-training for parser adaptation. In Proceedings of the 21st COLING/44th ACL. find our card
6. Slav Petrov and Dan Klein. 2007. Improved inference for unlexicalized parsing. In Proceedings of HLT NAACL 2007. find
7. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact and interpretable tree annotation. In Proceedings of the 21st COLING and the 44th ACL. find our card
8. Brian Roark, Mary Harper, Eugene Charniak, Bonnie Dorr, Mark Johnson, Jeremy G. Kahn, Yang Liu, Mari Ostendorf, John Hale, Anna Krasnyanskaya, Matthew Lease, Izhak Shafran, Matthew Snover, Robin Stewart, and Lisa Yung. 2006. SParseval: Evaluation metrics for parsing speech. In Proceedings of LREC. find
9. Sali A. Tagliamonte and Derek Dennis. 2008. Linguistic ruin? LOL! Instant messaging and teen language. American Speech, 83(1). find
10. Lonneke van der Plas, James Henderson, and Paola Merlo. 2009. Domain adaptation with artificial data for semantic parsing of speech. In Proceedings of HLT NAACL 2009, Companion Volume: Short Papers. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2010-foster-cba-check-spelling-investigating-parser/