How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT

What it is about

Eight papers from 2013 to 2017 reported large gains in RST discourse parsing, with a claimed relative error reduction of 24 to 51%. The gains were attributed to distributed (neural) representations of discourse units. The authors asked the authors of nine parsers for their test-set predictions on the RST-DT, which has 385 articles (347 for training, 38 for test), and re-scored them. Their main finding is that most of the reported gains come from evaluation differences. The neural parsers were scored with macro-averaged F1 across documents, while the older parsers used micro-averaged F1. Under the same micro-averaged procedure the neural group loses its lead on the span and nuclearity scores. Its best gain on relations and full labels is 0.9 and 1.4 points, a relative error reduction of 11% and 16%. A second check used the standard Parseval procedure on right-heavy binarised trees instead of RST-Parseval. Under it the best full-label F1 is 46.3, and the gains are at most 16% relative error reduction.

Ideas

• I1. coref/src/eval.rs and any future discourse scorer: always report micro and macro averages side by side, computed from BTreeMap-ordered per-document counts, and refuse a claim of gain unless both are stated. The paper's whole finding depends on this.
• I2. A replication step before any claim of progress: keep the previous run's predictions and re-score them with the new scorer. A gain that appears only with the new scorer is an evaluation artefact, the paper's main lesson.
• I3. Add a standard Parseval mode for Mova's own discourse trees: right-heavy binarisation of the reference, then labelled attachment decisions. Compare it with the native scorer on the same output to see how much the scorer matters.
• I4. Keep a simple feature baseline (unit length, distance between units, word and POS n-grams, as in the paper's first group) in every discourse measurement. The paper shows such a baseline is competitive, so a new rule system must beat it on the same scorer.

For Mova

The chapters are 36. Discourse relations and coherence (relations and structure) and 38. Dialogue, discourse markers and reported speech (markers). The code is coref/src/eval.rs as the model for scorer conventions, and prag/src/frames.rs for the discourse layer. Mova has no RST parser in the repo; if one is built, the measure is the paper's table layout (S, N, R, F) with both averages, run twice, plus the replication check against the old predictions. The RST-DT is evaluation-only data.

Topics: Discourse analysis

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1136

@inproceedings{morey2017how,
  title = {{How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT}},
  author = {Morey, Mathieu and Muller, Philippe and Asher, Nicholas},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1136},
  url = {https://doi.org/10.18653/v1/d17-1136}
}

References

17 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. E. Black, S. Abney, D. Flickinger, C. Gdaniec, R. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini, and T. Strzalkowski. 1991. Procedure for quantitatively comparing the syntactic coverage of english grammars. In Proceedings of the Workshop on Speech and Natural Language, HLT ’91, pages 306–311. Association for Computational Linguistics. find
2. Chloé Braud, Maximin Coavoux, and Anders Søgaard. 2017. Cross-lingual rst discourse parsing. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 292–304, Valencia, Spain. Association for Computational Linguistics. find
3. Chloé Braud, Barbara Plank, and Anders Søgaard. 2016. Multi-view and multi-task training of rst discourse parsers. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1903–1913, Osaka, Japan. find
4. The COLING 2016 Organizing Committee. find
5. Kenji Sagae. 2009. Analysis of discourse structure with syntactic dependencies and data-driven shift-reduce parsing. In Proceedings of the 11th International Conference on Parsing Technologies, IWPT ’09, pages 81–84, Stroudsburg, PA, USA. Association for Computational Linguistics. find
6. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Jan van Kuppevelt and Ronnie Smith, editors, Current Directions in Discourse and Dialogue, pages 85–112. Kluwer Academic Publishers. find our card
7. Mihai Surdeanu, Thomas Hicks, and Marco A Valenzuela-Escárcega. 2015. Two practical rhetorical structure theory parsers. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 1–5. find
8. Vanessa Wei Feng and Graeme Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 511– 521, Baltimore, Maryland. Association for Computational Linguistics. find
9. Katsuhiko Hayashi, Tsutomu Hirao, and Masaaki Nagata. 2016. Empirical comparison of dependency conversions for rst discourse trees. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 128–136, Los Angeles. Association for Computational Linguistics. find
10. Michael Heilman and Kenji Sagae. 2015. Fast Rhetorical Structure Theory Discourse Parsing. arXiv preprint, arXiv:1505.02425. ↗
11. Hugo Hernault, Helmut Prendinger, David A. duVerle, and Mitsuru Ishizuka. 2010. HILDA: A Discourse Parser Using Support Vector Machine Classification. Dialogue and Discourse, 1(3):1–33. find
12. Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13–24, Baltimore, Maryland. Association for Computational Linguistics. find
13. Shafiq Joty, Giuseppe Carenini, and Raymond T Ng. 2015. Codra: A novel discriminative framework for rhetorical analysis. Computational Linguistics, 41(3):385–435. find
14. Shafiq R Joty, Giuseppe Carenini, Raymond T Ng, and Yashar Mehdad. 2013. Combining intra-and multi-sentential rhetorical parsing for document-level discourse analysis. In ACL (1), pages 486–496. find
15. Qi Li, Tianshi Li, and Baobao Chang. 2016. Discourse parsing with attention-based hierarchical neural networks. In EMNLP, pages 362–371. find
16. William C. Mann and Sandra A. Thompson. 1988. Rhetorical Structure Theory: Towards a Functional Theory of Text Organization. Text, 8(3):243–281. find
17. Daniel Marcu. 2000. The theory and practice of discourse parsing and summarization. MIT press. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/morey-2017-how-much-progress-have-we/