The Denoised Web Treebank: Evaluating Dependency Parsing under Noisy Input Conditions

What it is about

The authors build a 500-tweet English treebank (250 development, 250 test) with a normalization layer: each tweet keeps its original tokens, and a normalized sentence with word alignments is parsed. Insertions, deletions and zero copulas are allowed in the normalized side, so standard UAS and LAS no longer apply; the paper defines an aligned precision, recall and F1 that reduce to UAS and LAS when tokens match one to one. On the development set, a graph-based MST parser with coarse Twitter tags plus a fine tagger reaches 72.41 unlabeled F1 without normalization, 76.17 with Twitter-specific syntax rules, and 77.08 with lexical normalization combined with machine translation. Gold normalization and gold tags give 79.28, so the normalization gain is close to the gold upper bound.

Ideas

• I1. A deterministic normalization layer in Rust before en/src/tok.rs (lexical rules and a short-form list, in the spirit of the paper's unsupervised lexical model); measure the aligned F1 on the Denoised Web Treebank development set, test only, against the unnormalized en parse.
• I2. en/src/ud.rs and tools/treeagree: Twitter-specific rules (leading usernames, RT markers, hashtags and final URLs attached to the main verb as an unclassified relation); measure how many dep labels and gate violations appear on web text before and after the rules.
• I3. Aligned evaluation in en parse-eval: report the aligned F1 when the tokens of a system differ from gold, so that normalization experiments are scored on the original text; measure the score agreement with UAS and LAS on EWT, where they must match.
• I4. en/src/ptag.rs: combine a fine tagger with a coarse, domain-specific one by n-best voting, as the paper does; measure UPOS on web text (test only) with the vote and without it.

For Mova

Touches languages/en/19-lexicon-mwe-register, 37-register-and-variation and 38-dialogue-and-discourse-markers, and the en crate (tok.rs, ud.rs, ptag.rs) and tools/treeagree. The treebank is NC: use it for tests and measurements only, never in a shipped model. Measure first the unnormalized en aligned F1 on the test half, run twice to confirm determinism.

Topics: Dependency parsing · Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/2w9ohr48zi4a

@inproceedings{daiber2016denoised,
  title = {{The Denoised Web Treebank: Evaluating Dependency Parsing under Noisy Input Conditions}},
  author = {Daiber, Joachim and Goot, Rob van der},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/2w9ohr48zi4a},
  url = {https://doi.org/10.63317/2w9ohr48zi4a}
}

References

24 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aw, A., Zhang, M., Xiao, J., and Su, J. (2006). A phrase-based statistical model for SMS text normalization. In Proceedings of the COLING/ACL on Main conference poster sessions, pages 33–40. find
2. Bies, A., Ferguson, M., Katz, K., MacIntyre, R., Tredinnick, V., Kim, G., Marcinkiewicz, M. A., and Schasberger, B. (1995). Bracketing guidelines for Treebank II style Penn Treebank project. Technical report, University of Pennsylvania. find
3. Bies, A., Mott, J., Warner, C., and Kulick, S. (2012). Bracketing webtext: An addendum to Penn Treebank II guidelines. Technical report, Linguistic Data Consortium. find
4. Brants, T. (2000). TnT: a statistical part-of-speech tagger. In Proceedings of the Sixth Conference on Applied Natural Language Processing, ANLC ’00, pages 224–231, Stroudsburg, PA, USA. find our card
5. Buchholz, S. and Marsi, E. (2006). CoNLL-X shared task on multilingual dependency parsing. In Proceedings of the Tenth Conference on Computational Natural Language Learning, pages 149–164. find our card
6. De Marneffe, M.-C., MacCartney, B., Manning, C. D., et al. (2006). Generating typed dependency parses from phrase structure parses. In Proceedings of LREC, volume 6, pages 449–454. find our card
7. Efron, B. and Tibshirani, R. (1993). An Introduction to the Bootstrap. Chapman & Hall, New York. find
8. Federico, M. and Cettolo, M. (2007). Efficient handling of n-gram language models for statistical machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 88–95. find
9. Foster, J., Çetinoglu, Ö., Wagner, J., Le Roux, J., Nivre, J., Hogan, D., and van Genabith, J. (2011a). From news to comment: Resources and benchmarks for parsing the language of web 2.0. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 893–901, Chiang Mai, Thailand, November. Asian Federation of Natural Language Processing. find
10. Foster, J., Çetinoglu, Ö., Wagner, J., Le Roux, J., Hogan, S., Nivre, J., Hogan, D., and Van Genabith, J. (2011b). #hardtoparse: POS Tagging and Parsing the Twitterverse. In AAAI 2011 Workshop On Analyzing Microtext, pages 20–25, United States. find
11. Gimpel, K., Schneider, N., O’Connor, B., Das, D., Mills, D., Eisenstein, J., Heilman, M., Yogatama, D., Flanigan, J., and Smith, N. A. (2011). Part-of-speech tagging for Twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 42–47. find
12. Han, B. and Baldwin, T. (2011). Lexical normalisation of short text messages: Makn sens a #twitter. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 368–378. find
13. Jurafsky, D. and Martin, J. H. (2000). Speech and Language Processing An Introduction to Natural Language Processing, Computational Linguistics, and Speech. Pearson Education. find
14. Kaljahi, R., Foster, J., Roturier, J., Ribeyre, C., Lynn, T., and Le Roux, J. (2015). Foreebank: Syntactic analysis of customer support forums. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1341–1347, Lisbon, Portugal, September. find
15. Khan, M., Dickinson, M., and Kübler, S. (2013). Towards domain adaptation for parsing web data. In Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013, pages 357–364, Hissar, Bulgaria, September. INCOMA Ltd. Shoumen, Bulgaria. find
16. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., et al. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pages 177–180. find
17. Kong, L., Schneider, N., Swayamdipta, S., Bhatia, A., Dyer, C., and Smith, N. A. (2014). A dependency parser for Tweets. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1001–1012, Doha, Qatar, October. find our card
18. Liu, X., Zhou, M., Wei, F., Fu, Z., and Zhou, X. (2012). Joint inference of named entity recognition and normalization for tweets. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 526–535. find
19. McDonald, R. T., Pereira, F., Ribarov, K., and Hajic, J. (2005). Non-projective dependency parsing using spanning tree algorithms. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing. find our card
20. Och, F. J. (2003). Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, volume 1, pages 160–167. find
21. Petrov, S. and McDonald, R. (2012). Overview of the 2012 shared task on parsing the web. In Notes of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL), volume 59. find
22. Prins, R. P. (2005). Finite-state pre-processing for natural language analysis. Ph.D. thesis, University of Groningen. find
23. Raghunathan, K. and Krawczyk, S. (2009). Investigating SMS text normalization using statistical machine translation. Technical report. find
24. Silveira, N., Dozat, T., de Marneffe, M.-C., Bowman, S., Connor, M., Bauer, J., and Manning, C. (2014). A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 2897–2904, Reykjavik, Iceland, May. European Language Resources Association (ELRA). find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2016-daiber-denoised-web-treebank-evaluating-dependency/