Committed Belief Tagging on the Factbank and LU Corpora: A Comparative Study

What it is about

Committed belief (CB) is how strongly a writer commits to a proposition. Two corpora label it: the LU corpus (16 documents, four genres, 3-way labels) and FactBank (newswire, about four times larger, factuality labels). The authors map both to a common 4-way scheme: committed belief, non-committed belief, non-attributable, and reported belief. They split the old non-committed class into the writer's own uncertainty and belief reported from someone else. They then compare five learners (CRF, linear and quadratic SVM, naive Bayes, logistic regression) on lexical and dependency features from CoreNLP, with sentence-level splits (every sixth sentence for test). Best results: F1 75.7 on FactBank (logistic regression, FB-4 and LU combinations) and 72.9 on LU-4 (linear SVM). Reported-belief tags were 100 in LU-4 training against 1800 in FB-4, and F1 for reported belief was much higher on FactBank. Training on more but differently-genred data did not help the smaller corpus.

Ideas

• I1. prag/src/roles.rs: add a "reported" tag for an event under a say-type verb whose subject is not the writer, as a rule over the UD tree (ccomp under a reporting verb). Count how many events the rule adds and check them by hand on a sample.
• I2. Keep genre-matched evaluation: the paper finds homogeneity matters more than size. Report Mova's accuracy per genre (school text, news, dialogue) and not only overall.
• I3. Report per-class F1 for each committed-belief class, as the paper does, so that the rare reported-belief class is not hidden by the majority class.

For Mova

The chapter is 30. Pragmatics and discourse: overview and 31. Speech acts. The code is prag/src/roles.rs (who said what) and prag/src/frames.rs. Measures: per-class F1 on a hand-checked sample from school texts, split by genre, run twice for identical output. The paper uses a machine learner; for Mova use the rule-based reported-belief tag and compare it with the learner's labels as a baseline, not as a replacement.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/v1/w15-1304

@inproceedings{werner2015committed,
  title = {{Committed Belief Tagging on the Factbank and LU Corpora: A Comparative Study}},
  author = {Werner, Gregory and Prabhakaran, Vinodkumar and Diab, Mona and Rambow, Owen},
  booktitle = {Proceedings of the Second Workshop on Extra-Propositional Aspects of Meaning in Computational Semantics (ExProM 2015)},
  year = {2015},
  doi = {10.3115/v1/w15-1304},
  url = {https://doi.org/10.3115/v1/w15-1304}
}

References

12 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Steven Bethard, Philip Ogren, and Lee Becker. 2014. ClearTK 2.0: Design Patterns for Machine Learning in UIMA. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 3289–3293, Reykjavik, Iceland, 5. European Language Resources Association (ELRA). find
2. Mona Diab, Lori Levin, Teruko Mitamura, Owen Rambow, Vinodkumar Prabhakaran, and Weiwei Guo. 2009. Committed Belief Annotation and Tagging. In Proceedings of the Third Linguistic Annotation Workshop, pages 68–73, Suntec, Singapore, August. find
3. Thorsten Joachims. 1999. Making Large-Scale SVM Learning Practical. In Bernhard Schölkopf, Christopher J.C. Burges, and A. Smola, editors, Advances in Kernel Methods - Support Vector Learning, Cambridge, MA, USA. MIT Press. find
4. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Maryland, June. find
5. Andrew Kachites McCallum. 2002. MALLET: A Machine Learning for Language Toolkit. http://www.cs.umass.edu/ mccallum/mallet. ↗
6. Philip Ogren and Steven Bethard. 2009. Building Test Suites for UIMA Components. In Proceedings of the Workshop on Software Engineering, Testing, and Quality Assurance for Natural Language Processing (SETQA-NLP 2009), pages 1–4, Boulder, Colorado, June. find
7. Vinodkumar Prabhakaran, Owen Rambow, and Mona Diab. 2010. Automatic Committed Belief Tagging. In Coling 2010: Posters, pages 1014–1022, Beijing, China, August. Coling 2010 Organizing Committee. find
8. Vinodkumar Prabhakaran, Tomas By, Julia Hirschberg, Owen Rambow, Samira Shaikh, Tomek Strzalkowski, Jennifer Tracey, Michael Arrigo, Rupayan Basu, Micah Clark, Adam Dalton, Mona Diab, Louise Guthrie, Anna Prokofieva, Stephanie Strassel, Gregory Werner, Janyce Wiebe, and Yorick Wilks. 2015. A New Dataset and Evaluation for Belief/Factuality. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics (*SEM 2015), Denver, USA, June. find
9. Randolph Quirk, Sidney Greenbaum, Geoffrey Leech, and Jan Svartvik. 1985. A Comprehensive Grammar of the English Language. Longman, London. find
10. Sara Rosenthal and Kathleen McKeown. 2012. Detecting Opinionated Claims in Online Discussions. In Semantic Computing (ICSC), 2012 IEEE Sixth International Conference on, pages 30–37. IEEE. find
11. Roser Saurı́ and James Pustejovsky. 2009. Fact-Bank: A Corpus Annotated with Event Factuality. Language Resources and Evaluation, 43:227–268. 10.1007/s10579-009-9089-9. ↗
12. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating Expressions of Opinions and Emotions in Language. Language Resources and Evaluation, 39(2-3):165–210. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/werner-2015-committed-belief-tagging-factbank-lu/