Mova0.5 pre-release⌕
☰
home / docs / papers / English / Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future

Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future

What it is about

The authors survey methods for finding annotation errors, reimplement 18 of them, and test them on 9 English datasets for text classification, token labelling and span labelling. They define one evaluation protocol, including the task formalisation and how to score detection. The main questions are which methods work across tasks, whether calibrating model probabilities helps, and how model quality relates to detection quality. Label aggregation, Retag, Diverse Ensemble and Borda Count perform best across tasks and datasets. Retag and classification uncertainty are the methods used most in practice and they also do well. Calibration helps some model-based methods and not others. Detection works better when the underlying model is trained by cross-validation.

Ideas

For Mova

Mova's tagger already has a suspect rule, and this paper gives a benchmark for comparing it with the others. The rule that the flagger must be trained in cross-validation matches Mova's held-out discipline. No English chapter covers annotation quality directly; the closest is languages/en/20-syntax-overview. Measure first: precision of the current suspect rule on injected noise against Retag on the same data, and run twice to check determinism.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00464

@article{klie2023annotation,
  title = {{Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future}},
  author = {Klie, Jan-Christoph and Webber, Bonnie and Gurevych, Iryna},
  journal = {Computational Linguistics},
  year = {2023},
  doi = {10.1162/coli\_a\_00464},
  url = {https://doi.org/10.1162/coli_a_00464}
}

References

99 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahrens, Sönke. 2017. How to Take Smart Notes: One Simple Technique to Boost Writing, Learning and Thinking: For Students, Academics and Nonfiction Book Writers. CreateSpace, North Charleston, SC. find
  2. Akbik, Alan, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy-to-use framework for state-of-the-art NLP. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 54–59. find
  3. Alt, Christoph, Aleksandra Gabryszak, and Leonhard Hennig. 2020. TACRED revisited: A thorough evaluation of the TACRED relation extraction task. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1558–1569. https://doi.org/10 .18653/v1/2020.acl-main.142 ↗
  4. Ambati, Bharat Ram, Rahul Agarwal, Mridul Gupta, Samar Husain, and Dipti Misra Sharma. 2011. Error detection for treebank validation. In Proceedings of the 9th Workshop on Asian Language Resources, pages 23–30. find
  5. Amiri, Hadi, Timothy Miller, and Guergana Savova. 2018. Spotting spurious data with neural networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2006–2016. https://doi.org/10.18653/v1/N18-1182 ↗
  6. Angle, Sachi, Pruthwik Mishra, and Dipti Mishra Sharma. 2018. Automated error correction and validation for POS tagging of Hindi. In Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation, pages 11–18. find
  7. Aroyo, Lora and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24. https://doi.org/10.1609/aimag.v36i1.2564 ↗
  8. Barnes, Jeremy, Lilja Øvrelid, and Erik Velldal. 2019. Sentiment analysis is not solved! Assessing and probing sentiment classification. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 12–23. https://doi.org/10 .18653/v1/W19-4802 ↗
  9. Basile, Valerio, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. We need to consider disagreement in evaluation. In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15–21. https://doi.org/10.18653/v1/2021.bppf-1.3 ↗
  10. Behrens, Heike, editor. 2008. Corpora in Language Acquisition Research: History, Methods, Perspectives, volume 6 of Trends in Language Acquisition Research. John Benjamins Publishing Company, Amsterdam. https://doi.org/10.1075/tilar.6.03beh Boyd, Adriane, Markus Dickinson, and W. Detmar Meurers. 2008. On detecting errors in dependency treebanks. Research on Language and Computation, 6(2):113–137. https://doi.org/10.1007/s11168-008 -9051-9 ↗
  11. Breunig, Markus M., Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander. 2000. LOF: Identifying density-based local outliers. ACM SIGMOD Record, 29(2):93–104. https://doi.org/10.1145/335191.335388 ↗
  12. Burkard, Rainer, Mauro Dell’Amico, and Silvano Martello. 2012. Assignment Problems: Revised Reprint. Society for Industrial and Applied Mathematics. https://doi.org/10.1137/1 .9781611972238 ↗
  13. Cer, Daniel, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169–174. https://doi.org/10.18653/v1/D18-2029 ↗
  14. Chinchor, Nancy and Beth Sundheim. 1993. MUC-5 evaluation metrics. In Proceedings of the Fifth Message Understanding Conference (MUC-5), pages 69–78. https://doi.org/10.3115/1072017 .1072026 ↗
  15. Cui Zhu, H. Kitagawa, and C. Faloutsos. 2005. Example-based robust outlier detection in high dimensional datasets. In Fifth IEEE International Conference on Data Mining (ICDM’05), pages 829–832. find
  16. Davis, Jesse and Mark Goadrich. 2006. The relationship between precision-recall and ROC curves. In Proceedings of the 23rd International Conference on Machine Learning - ICML ’06, pages 233–240. find
  17. Dawid, A. P. and A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28(1):20–28. https://doi.org/10.2307/2346806 ↗
  18. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. find
  19. Dickinson, Markus. 2006. From detecting errors to automatically correcting them. In 11th Conference of the European Chapter of the Association for Computational Linguistics, pages 265–272. find
  20. Dickinson, Markus. 2015. Detection of annotation errors in Corpora. Language and Linguistics Compass, 9(3):119–138. https://doi.org/10.1111/lnc3.12129 ↗
  21. Dickinson, Markus and Chong Min Lee. 2008. Detecting errors in semantic annotation. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), pages 605–610. find
  22. Dickinson, Markus and W. Detmar Meurers. 2003a. Detecting errors in part-of-speech annotation. In Proceedings of the Tenth Conference on European Chapter of the Association for Computational Linguistics - Volume 1, EACL ’03, pages 107–114. https://doi.org/10.3115/1067807 .1067823 ↗
  23. Dickinson, Markus and W. Detmar Meurers. 2003b. Detecting inconsistencies in treebanks. In Proceedings of the Second Workshop on Treebanks and Linguistic Theories, pages 1–12. find
  24. Dickinson, Markus and W. Detmar Meurers. 2005. Detecting errors in discontinuous structural annotation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics - ACL ’05, pages 322–329. https://doi.org/10 .3115/1219840.1219880 ↗
  25. Dligach, Dmitriy and Martha Palmer. 2011. Reducing the need for double annotation. In Proceedings of the 5th Linguistic Annotation Workshop, pages 65–73. find
  26. Dwork, Cynthia, Ravi Kumar, Moni Naor, and D. Sivakumar. 2001. Rank aggregation methods for the Web. In Proceedings of the Tenth International Conference on World Wide Web - WWW ’01, pages 613–622. find
  27. Fornaciari, Tommaso, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2591–2597. https://doi.org/10.18653/v1/2021 .naacl-main.204 ↗
  28. Gal, Yarin and Zoubin Ghahramani. 2016. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning, pages 1050–1059. ↗
  29. Gimpel, Kevin, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 42–47. https://doi.org/10.21236/ADA547371 ↗
  30. Grivas, Andreas, Beatrice Alex, Claire Grover, Richard Tobin, and William Whiteley. 2020. Not a cute stroke: Analysis of Rule- and Neural Network-based Information Extraction Systems for Brain Radiology Reports. In Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis, pages 24–37. https://doi.org/10.18653/v1/2020.louhi-1.4 ↗
  31. Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1321–1330. ↗
  32. Gururangan, Suchin, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360. https://doi.org/10 .18653/v1/2020.acl-main.740 ↗
  33. Haselbach, Boris, Kerstin Eckart, Wolfgang Seeker, Kurt Eberle, and Ulrich Heid. 2012. Approximating theoretical linguistics classification in real data: The case of German “nach” particle verbs. In Proceedings of COLING 2012, pages 1113–1128. find
  34. Hedderich, Michael A., Dawei Zhu, and Dietrich Klakow. 2021. Analysing the noise model error for realistic noisy label data. Proceedings of the AAAI Conference on Artificial Intelligence, 35(9):7675–7684. https://doi.org/10.1609/aaai .v35i9.16938 ↗
  35. Heinzerling, Benjamin and Michael Strube. 2018. BPEmb: Tokenization-free pre-trained subword embeddings in 275 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 2989–2993. find
  36. Hemphill, Charles T., John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Proceedings of the Workshop on Speech and Natural Language, pages 96–101. https://doi.org/10.3115/116580.116613 ↗
  37. Hendrycks, Dan and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of International Conference on Learning Representations, pages 1–12. ↗
  38. Hollenstein, Nora, Nathan Schneider, and Bonnie Webber. 2016. Inconsistency detection in semantic annotation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3986–3990. find
  39. Hovy, Dirk, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130. find
  40. Jamison, Emily and Iryna Gurevych. 2015. Noise or additional information? find
  41. Leveraging crowdsource annotation item agreement for natural language tasks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 291–297. https://doi.org/10.18653/v1/D15-1035 ↗
  42. Joulin, Armand, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431. https://doi.org/10.18653/v1/E17-2068 ↗
  43. Ke, Guolin, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A highly efficient gradient boosting decision tree. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 1–9. find
  44. Kehler, A., L. Kertz, H. Rohde, and J. L. Elman. 2007. Coherence and coreference revisited. Journal of Semantics, 25(1):1–44. https://doi.org/10.1093/jos/ffm018, PubMed: 22923856 ↗
  45. Kendall, M. G. 1938. A new measure of rank correlation. Biometrika, 30(1–2):81–93. https://doi.org/10.1093/biomet/30 .1-2.81 ↗
  46. Khandelwal, Urvashi, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations (ICLR), pages 1–13. ↗
  47. Küppers, Fabian, Jan Kronenberger, Amirhossein Shantia, and Anselm Haselhoff. 2020. Multivariate confidence calibration for object detection. In 2nd Workshop on Safe Artificial Intelligence for Automated Driving (SAIAD), pages 1–9. find
  48. Kvĕtoň, Pavel and Karel Oliva. 2002. (Semi-)automatic detection of errors in PoS-tagged corpora. In COLING 2002: The 19th International Conference on Computational Linguistics, pages 1–7. https://doi.org/10.3115/1072228 .1072249 ↗
  49. Larson, Stefan, Adrian Cheung, Anish Mahendran, Kevin Leach, and Jonathan K. Kummerfeld. 2020. Inconsistencies in crowdsourced slot-filling annotations: A typology and identification methods. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5035–5046. https://doi.org/10 .18653/v1/2020.coling-main.442 ↗
  50. Larson, Stefan, Anish Mahendran, Andrew Lee, Jonathan K. Kummerfeld, Parker Hill, Michael A. Laurenzano, Johann Hauswald, Lingjia Tang, and Jason Mars. 2019. Outlier detection for improved data quality and diversity in dialog systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 517–527. https://doi.org/10.18653/v1/N19-1051 ↗
  51. Leitner, Sebastian. 1974. So Lernt Man Leben [How to Learn to Live]. Droemer-Knaur, Munich. find
  52. Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arxiv preprints 11692. find
  53. Loftsson, Hrafn. 2009. Correcting a POS-Tagged corpus using three complementary methods. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 523–531. https://doi.org/10.3115/1609067 .1609125 ↗
  54. Lord, F. M., M. R. Novick, and Allan Birnbaum. 1968. Statistical Theories of Mental Test Scores. Addison-Wesley, Oxford, England. find
  55. Manning, Christopher D. 2011. Part-of-speech tagging from 97% to 100%: Is it time for some linguistics? In Computational Linguistics and Intelligent Text Processing, volume 6608, pages 171–189. https://doi.org/10.1007/978-3-642 -19400-9_14 ↗ our card
  56. Manning, Christopher D., Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press, New York. ↗
  57. Ménard, Pierre André and Antoine Mougeot. 2019. Turning silver into gold: Error-focused corpus reannotation with active learning. In Proceedings - Natural Language Processing in a Deep Learning World, pages 758–767. https://doi.org/10.26615/978-954-452-056-4_088 ↗
  58. Naeini, Mahdi Pakdaman, Gregory F. Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 2901–2907. find
  59. Nivre, Joakim, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4034–4043. ↗ our card
  60. Northcutt, Curtis, Lu Jiang, and Isaac Chuang. 2021. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373–1411. https://doi.org/10 .1613/jair.1.12125 ↗
  61. Northcutt, Curtis G., Anish Athalye, and Jonas Mueller. 2021. Pervasive label errors in test sets destabilize machine learning benchmarks. In 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track, pages 1–13. ↗
  62. Paun, Silviu, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018. Comparing Bayesian models of annotation. Transactions of the Association for Computational Linguistics, 6(0):571–585. https://doi.org/10.1162/tacl_a_00040 ↗
  63. Pavlick, Ellie and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. ↗ our card
  64. Pennington, Jeffrey, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. https://doi.org/10.3115/v1/D14-1162 ↗
  65. Peters, Matthew E., Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? Adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 7–14. https://doi.org/10.18653/v1/W19-4302 ↗
  66. Plank, Barbara, Dirk Hovy, and Anders Søgaard. 2014a. Learning part-of-speech taggers with inter-annotator agreement loss. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–751. find
  67. Plank, Barbara, Dirk Hovy, and Anders Søgaard. 2014b. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 507–511. https://doi.org/10.3115/v1/P14-2083 ↗
  68. Platt, John C. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10(3):1–9. find
  69. Pustejovsky, J. and Amber Stubbs. 2013. Natural Language Annotation for Machine Learning. O’Reilly Media, Sebastopol, CA. find
  70. Qian, Kun, Ahmad Beirami, Zhouhan Lin, Ankita De, Alborz Geramifard, Zhou Yu, and Chinnadhurai Sankar. 2021. Annotation inconsistency and entity bias in MultiWOZ. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 326–337. find
  71. Rehbein, Ines. 2014. POS error detection in automatically annotated corpora. In Proceedings of LAW VIII - the 8th Linguistic Annotation Workshop, pages 20–28. find
  72. Reimers, Nils and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3980–3990. https://doi.org/10.18653/v1/D19-1410 ↗
  73. Reiss, Frederick, Hong Xu, Bryan Cutler, Karthik Muthuraman, and Zachary Eichenberger. 2020. Identifying incorrect labels in the CoNLL-2003 corpus. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 215–226. https://doi.org/10 .18653/v1/2020.conll-1.16 ↗
  74. Rodrigues, Filipe and Francisco Pereira. 2018. Deep learning from crowds. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 1611–1618. ↗
  75. Rodriguez, Pedro, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, and Jordan Boyd-Graber. 2021. Evaluation examples are not equally informative: How should that change NLP leaderboards? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4486–4503. https://doi.org/10 .18653/v1/2021.acl-long.346 ↗
  76. Saito, Takaya and Marc Rehmsmeier. 2015. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3):1–21. https://doi.org/10.1371/journal.pone.0118432, PubMed: 25738806 ↗
  77. Sanguinetti, Manuela and Cristina Bosco. 2015. PartTUT: The Turin University Parallel Treebank. In Basili, Roberto, Cristina Bosco, Rodolfo Delmonte, Alessandro Moschitti, and Maria Simi, editors, Harmonization and Development of Resources and Tools for Italian Natural Language Processing within the PARLI Project, volume 589, pages 51–69. https://doi.org/10.1007/978-3-319-14206-7_3 ↗
  78. Sanh, Victor, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing, pages 1–5. find
  79. Schreibman, Susan, Ray Siemens, and John Unsworth, editors. 2004. A Companion to Digital Humanities. Blackwell Publishing Ltd, Malden, MA, USA. find
  80. Shelmanov, Artem, Evgenii Tsymbalov, Dmitri Puzyrev, Kirill Fedyanin, Alexander Panchenko, and Maxim Panov. 2021. How certain is your transformer? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1833–1840. https://doi.org/10 .18653/v1/2021.eacl-main.157 ↗
  81. Socher, Richard, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642. find
  82. Song, Hwanjun, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. 2020. Learning from noisy labels with deep neural networks: A survey. arxiv preprint, 2007.8199. https://doi.org/10.1109/TNNLS.2022.3152527, PubMed: 35254993 ↗
  83. Stoica, George, Emmanouil Antonios Platanios, and Barnabas Poczos. 2021. Re-TACRED: Addressing shortcomings of the TACRED dataset. In Proceedings of the 35th AAAI Conference on Artificial Intelligence 2021, pages 13843–13850. https://doi.org/10.1609/aaai .v35i15.17631 ↗
  84. Sun, Chen, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. 2017. Revisiting unreasonable effectiveness of data in deep learning era. In IEEE International Conference on Computer Vision (ICCV), pages 1–13. find
  85. Swayamdipta, Swabha, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275–9293. https://doi.org/10 .18653/v1/2020.emnlp-main.746 ↗
  86. Szpiro, George. 2010. Numbers Rule: The Vexing Mathematics of Democracy, from Plato to the Present. Princeton University Press. https://doi.org/10.1515/9781400834440 ↗
  87. Tjong Kim Sang, Erik F. and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. https://doi.org/10.3115/1119176 .1119195 ↗
  88. Ustalov, Dmitry, Nikita Pavlichenko, Vladimir Losev, Iulian Giliazev, and Evgeny Tulin. 2021. A general-purpose crowdsourcing computational quality control toolkit for Python. In the Ninth AAAI Conference on Human Computation and Crowdsourcing: Works-in-Progress and Demonstration Track, pages 1–4. find
  89. van Halteren, Hans. 2000. The detection of inconsistency in manually tagged text. In Proceedings of the COLING-2000 Workshop on Linguistically Interpreted Corpora, pages 48–55. find
  90. Vlachos, Andreas. 2006. Active annotation. In Proceedings of the Workshop on Adaptive Text Extraction and Mining (ATEM 2006), pages 64–71. find
  91. Wang, Zihan, Jingbo Shang, Liyuan Liu, Lihao Lu, Jiacheng Liu, and Jiawei Han. 2019. CrossWeigh: Training named entity tagger from imperfect annotations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5153–5162. https://doi.org/10.18653/v1/D19 -1519, PubMed: 31303768 ↗
  92. Wilcoxon, Frank. 1945. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80. https://doi.org/10.2307/3001968 ↗
  93. Wisniewski, Guillaume. 2018. Errator: A tool to help detect annotation errors in the Universal Dependencies project. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 4489–4493. find our card
  94. Yaghoub-Zadeh-Fard, Mohammad Ali, Boualem Benatallah, Moshe Chai Barukh, and Shayan Zamanirad. 2019. A study of incorrect paraphrases in crowdsourced user utterances. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 295–306. https://doi.org/10.18653/v1/N19-1026 ↗
  95. Zadrozny, Bianca and Charles Elkan. 2001. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, pages 609–616. find
  96. Zadrozny, Bianca and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 694–700. find
  97. Zeldes, Amir. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. https://doi.org/10.1007/s10579-016-9343-x Zhang, Xiang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, pages 649–657. ↗
  98. Zhang, Xin, Guangwei Xu, Yueheng Sun, Meishan Zhang, and Pengjun Xie. 2021. Crowdsourcing learning as domain adaptation: A case study on named entity recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5558–5570. https://doi.org/10 .18653/v1/2021.acl-long.432 ↗
  99. Zheng, Guoqing, Ahmed Hassan Awadallah, and Susan Dumais. 2021. Meta label correction for noisy label learning. In Proceedings of the Thirty-fifth AAAI Conference on Artificial Intelligence 2021, pages 11053–11061. https://doi.org/10 .1609/aaai.v35i12.17319 ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/cl-2023-klie-annotation-error-detection-analyzing-past/