Categories of coherence relations in discourse annotation

What it is about

The paper tests whether untrained annotators can label coherence relations when the decision is split into the four cognitive primitives of Sanders et al. (CCR): polarity (positive, or negative when one segment is the negation of the expected consequence, as with but/although), basic operation (causal, conditional, additive, temporal; non-causal for negatives), source of coherence (objective, between described situations, vs subjective, involving the speaker's reasoning or a speech act) and order (cause/argument first = basic, or after = non-basic). Forty Dutch humanities students annotated 36 relations from the DiscAn corpus with a flowchart, half with plain instructions and half with substitution tests («Can you use but / because / if / then / and here?») and paraphrase tests. Agreement with the expert labels: polarity kappa 0.86, order 0.61, basic operation 0.49, source 0.31; the tests helped polarity, operation and order but not source. Temporal was most confused with additive (temporal F 0.34–0.51).

Ideas

• I1. Describe each Mova discourse link by the four primitives instead of by a name: cause = causal/positive, next = temporal/positive, cond = conditional/positive. The missing concession of although/but (chapter 36 §11.1 example, §11.6) then is simply causal/negative: «Although the crow was angry, she said nothing» links the two frames with the expectation denied. A field in the link struct of prag/src/frames.rs costs little and lets why-questions skip negative links. Measure on the frame bench and on course questions with although/but.
• I2. Use the substitution tests as the operational definition of next in the frame bench gold and in the teacher prompt: a pair gets next only if then can join the two clauses without changing the meaning; plain and that only passes the and test is additive. The paper's main confusion (temporal vs additive) is exactly Mova's weakest role (next dev F1 .23, gold narrower than and). Re-check a sample of bench gold with the test and measure how many next disagreements it settles.
• I3. Mark the source of coherence where English shows it cheaply (a cause clause attached to a Say frame, to an evaluation, or to must/surely = subjective), as in bunt-2020-iso-standard-dialogue-act-annotation I3; the paper's low agreement on source (0.31) argues for only marking clear cases and leaving the rest unspecified.

For Mova

36. Discourse relations and coherence §2 (the families), §11.1, §11.6. Code: prag/src/frames.rs (link primitives), the frame bench gold (world frames bench-make), prag/src/infer.rs. Measures: frame bench next/cause F1, course questions with although/but.

Topics: Discourse analysis

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.5087/dad.2016.201

@inproceedings{scholman2016categories,
  title = {{Categories of coherence relations in discourse annotation}},
  author = {Scholman, Merel C.J. and Evers-Vermeul, Jacqueline and Sanders, Ted J.M.},
  booktitle = {Dialogue \& Discourse},
  year = {2016},
  doi = {10.5087/dad.2016.201},
  url = {https://doi.org/10.5087/dad.2016.201}
}

References

57 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Omar Alonso and Stefano Mizzaro (2012). Using crowdsourcing for TREC relevance assessment. Information Processing and Management, 48(6): 1053-1066. find
2. Amal Al-Saif and Katja Markert (2010). The Leeds Arabic discourse treebank: Annotating discourse connectives for Arabic. In N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, M. Rosner & D. Tapias (eds.), Proceedings of 7th International Conference on Language Resources and Evaluation (LREC 2010): 2046-2053, Malta.Ron Artstein and Massimo Poesio (2008). Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4): 555-596. find
3. Nicholas Asher and Alex Lascarides (2003). Logics of Conversation. Cambridge: Cambridge University Press. find
4. Lois Bloom, Margaret Lahey, Lois Hood, Karin Lifter and Kathleen Fiess (1980). Complex sentences: Acquisition of syntactic connectives and the semantic relations they encode. Journal of Child Language, 7(2): 235-261. find
5. Thorsten Brants (2000). Inter-annotator agreement for a German newspaper corpus. Proceedings of the Sixth Conference on Applied Natural Language Processing (LREC). Seattle, WA. find
6. Anneloes R. Canestrelli, Willem M. Mak and Ted J.M. Sanders (2013). Causal connectives in discourse processing: How differences in subjectivity are reflected in eye-movements. Language and Cognitive Processes, 28(9): 1394-1413. find
7. Jean Carletta (1996). Assessing agreement on classification tasks: the kappa statistic. Computational Linguistics, 22(2): 249-254. find
8. Lynn Carlson and Daniel Marcu (2001). Discourse Tagging Reference Manual. Available online via http://www.isi.edu/~marcu/discourse/tagging-ref-manual.pdf. ↗
9. Lynn Carlson, Daniel Marcu and Mary E. Okurowski (2003). Building a Discourse-Tagged Corpus in the Framework of Rhetorical Structure Theory. In J. van Kuppevelt and R. Smith (eds.), Current Directions in Discourse and Dialogue: 85-112. Dordrecht: Kluwer Academic Publishers. find our card
10. Susan Conrad (2002). Corpus linguistic approaches for discourse analysis. Annual Review of Applied Linguistics, 22: 75-95. find
11. Liesbeth Degand and Henk Pander Maat (2003). A contrastive study of Dutch and French causal connectives on the Speaker Involvement Scale. In A. Verhagen and J. van de Weijer (eds.), Usage-based Approaches to Dutch: 175-199. Utrecht: LOT. find
12. Liesbeth Degand (2001). Form and Function of Causation: A Theoretical and Empirical Investigation of Causal Constructions in Dutch. Leuven: Peeters. find
13. Oswald Ducrot (1980). Essai d’application: MAIS - les allusions à l’énonciation – délocutifs, performatifs, discours indirect. In: H. Parret (ed.), Le langage en context: Etudes philosophiques et linguistiques de pragmatique : 487-575. Amsterdam: John Benjamins. find
14. Jacqueline Evers-Vermeul (2005). The development of Dutch connectives; change and acquisition as windows on form-function relations. Ph.D. dissertation. Utrecht: LOT. Available online via http://www.lotpublications.nl/Documents/110_fulltext.pdf. ↗
15. Jacqueline Evers-Vermeul and Ted J.M. Sanders (2009). The emergence of Dutch connectives: how cumulative cognitive complexity explains the order of acquisition. Journal of Child Language, 36(4): 829-854. find
16. Barbara J. Grosz and Candace L. Sidner (1986). Attention, intentions and the structure of discourse. Computational Linguistics, 12(3): 175-204. find
17. Michael A.K. Halliday and Ruqaiya Hasan (1976). Cohesion in English. London: Longman. find
18. Jerry R. Hobbs (1979). Coherence and coreference. Cognitive Science, 3(1): 67-90. find
19. Jerry R. Hobbs (1985). On the Coherence and Structure of Discourse. CSLI Center for the Study of Language and Information, Stanford University. find
20. Andrew Kehler (2002). Coherence, Reference, and the Theory of Grammar. Stanford, CA: CSLI Publications. find
21. Alistair Knott and Robert Dale (1994). Using linguistic phenomena to motivate a set of coherence relations. Discourse Processes, 18: 35-62. find
22. Alistair Knott and Ted J.M. Sanders (1998). The classification of coherence relations and their linguistic markers: An exploration of two languages. Journal of Pragmatics, 30: 135-175. find
23. Klaus Krippendorff (1980). Content Analysis: An Introduction to Its Methodology. Beverly Hills, CA: Sage. find
24. Ewald Lang (1984). The semantics of coordination. Amsterdam: John Benjamins. find
25. Fang Li, Jacqueline Evers-Vermeul and Ted J.M. Sanders (2013). Subjectivity and result marking in Mandarin: A corpus-based investigation. Chinese Language and Discourse, 4(1): 74-119. find
26. William C. Mann and Sandra A. Thompson (1988). Rhetorical structure theory: Towards a functional theory of text organization. Text, 8(3): 243-281. find
27. Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi and Bonnie Webber (2004). Annotating discourse connectives and their arguments. Proceedings of the Frontiers in Corpus Annotation 2004 NAACL/HLT Conference Workshop, Boston. find
28. Megan Moser and Johanna D. Moore (1996). On the Correlation of Cues with Discourse Structure: Results from a Corpus Study. University of Pittsburgh, Learning Research and Development Center. Available online via homepages.inf.ed.ac.uk/jmoore/papers/rda.ps. find
29. Megan Moser, Johanna D. Moore and Erin Glendening (1996). Instructions for Coding Explanations: Identifying Segments, Relations and Minimal Units. University of Pittsburgh, Learning Research and Development Center. Available online via http://homepages.inf.ed.ac.uk/jmoore/papers/rda-instr.ps. ↗
30. Leo G.M. Noordman and Femke de Blijzer (2000). On the processing of causal relations. In: E. Couper Kuhlen & B. Kortmann (eds.), Cause, Condition, Concession, Contrast: Cognitive and Discourse Perspectives. Berlin, New York: Mouton de Gruyter. find
31. Leo G.M. Noordman and Wietske Vonk (1998): Memory-‐based processing in understanding causal information, Discourse Processes, 26(2-3): 191-212. find
32. Stefanie Nowak and Stefan Rüger (2010). How reliable are annotations via crowdsourcing: A study about inter-annotator agreement for multi-label image annotation. Proceedings of the International Conference on Multimedia Information Retrieval (MIR), Philadelphia, USA. find
33. Henk Pander Maat and Ted J.M. Sanders (2000). Domains of use or subjectivity? The distribution of three Dutch causal connectives explained. Topics in English Linguistics, 33: 57-82. find
34. PDTB Research Group (2007). The Penn Discourse Treebank 2.0 Annotation Manual. Available online via http://www.seas.upenn.edu/~pdtb/PDTBAPI/pdtb-annotation-manual.pdf. ↗ our card
35. Mirna Pit (2007). Cross-linguistic analyses of backward causal connectives in Dutch, German and French. Languages in Contrast, 7(1): 53-82. find
36. Emily Pitler and Ani Nenkova (2009). Using syntax to disambiguate explicit discourse connectives in text. Proceedings of the ACL-IJCNLP 2009 Conference: 13-16, Singapore. find our card
37. Massimo Poesio and Ron Artstein (2005). The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. Proceedings of the Workshop on Frontiers in Corpus Annotations ii: Pie in the sky: 76-83.Rashmi Prasad and Harry Bunt (2015). Semantic relations in discourse: The current state of ISO 24617-8. In H. Bunt (ed.), Proceedings of the 11th Joint ACL - ISO Workshop on Interoperable Semantic Annotation (isa-11): 80-91. Tilburg: TiCC, Tilburg University. find
38. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi and Bonnie Webber (2008). The Penn Discourse Treebank 2.0. Proceedings of the 6 th International Conference of Language Resources and Evaluation (LREC 2008), Marrakech. find our card
39. Ted J.M. Sanders (1997). Semantic and pragmatic sources of coherence: On the categorization of coherence relations in context. Discourse Processes, 24: 119-147. find
40. Ted J.M. Sanders and Leo G.M. Noordman (2000). The role of coherence relations and their linguistic markers in text processing. Discourse Processes, 29: 37-60. find
41. Ted J.M. Sanders and Wilbert P.M.S. Spooren (2009). Causal categories in discourse: Converging evidence from language use. In: T.J.M. Sanders and E. Sweetser (eds.), Causal categories in discourse and cognition. Berlin: Walter de Gruyter. find
42. Ted J.M. Sanders and Wilbert P.M.S. Spooren (2015). Causality and subjectivity in discourse: The meaning and use of causal connectives in spontaneous conversation, chat interactions and written text. Linguistics, 53(1): 53-92. find
43. Ted J.M. Sanders, Wilbert P.M.S. Spooren and Leo G.M. Noordman (1992). Toward a taxonomy of coherence relations. Discourse Processes, 15: 1-35. find
44. Ted J.M. Sanders, Wilbert P.M.S. Spooren and Leo G.M. Noordman (1993). Coherence relations in a cognitive theory of discourse representation. Cognitive Linguistics, 4(2): 93-133. find
45. Ted J.M. Sanders, Kirsten Vis and Daan Broeder (2012). Project notes of CLARIN project DiscAn: Towards a Discourse Annotation system for Dutch language corpora. Project notes. Utrecht: Utrecht University. find
46. William A. Scott (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3): 321-325. find
47. Wilbert P.M.S. Spooren and Liesbeth Degand (2010). Coding coherence relations: Reliability and validity. Corpus Linguistics and Linguistic Theory, 6(2): 241-266. find
48. Wilbert P.M.S. Spooren and Ted J.M. Sanders (2008). The acquisition order of coherence relations: On cognitive complexity in discourse. Journal of Pragmatics, 40: 2003-2026. find
49. Manfred Stede (2004). The Potsdam Commentary Corpus. Proceedings ACL Workshop on Discourse Annotation. Pennsylvania: ACL. find
50. Ninke M. Stukker and Ted J.M. Sanders (2012). Subjectivity and prototype structure in causal connectives. A cross-linguistic perspective. Journal of Pragmatics, 44(2): 169-190. find
51. Ninke Stukker, Ted J.M. Sanders and Arie Verhagen (2008). Causality in verbs and in discourse connectives: Converging evidence of cross-level parallels in Dutch linguistic categorization. Journal of Pragmatics, 40:1296-1322. find
52. Kai-Ming Ting (2010). Precision and Recall. In: C. Sammut and G.I. Webb (eds.), Encyclopedia of Machine Learning. New York: Springer US. find
53. Matthew J. Traxler, Michael D. Bybee and Martin J. Pickering (1997). Influences of connectives on language comprehension: Eye-tracking evidence for incremental interpretation. Quarterly Journal of Experimental Psychology, 50(3): 481-497. find
54. Matthew J. Traxler, Anthony J. Sanford, Joy P. Aked and Linda M. Moxey (1997). Processing causal and diagnostic statements in discourse. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23(1): 88-101. find
55. Nynke van der Vliet, Ildikó Berzlanovich, Gosse Bouma, Markus Egg and Gisela Redeker (2011). Building a discourse-annotated Dutch text corpus. In: S. Dipper and H. Zinsmeister (eds.), Proceedings Beyond Semantics (DGfS workshop). Bochumer Linguistische Arbeitsberichte 3: 157–171. find
56. Yannick Versley and Anna Gastel (2012). Linguistic tests for discourse relations in the TüBa-D/Z corpus of written German. Dialogue and Discourse, 4(2): 142-173. find
57. Sandrine Zufferey (2012). “Car, parce que, puisque” revisited: Three empirical studies on French causal connectives. Journal of Pragmatics, 44(2): 138-153. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/scholman-2016-categories-coherence-relations-discourse-annotation/