CoNLL 2016 Shared Task on Multilingual Shallow Discourse Parsing

What it is about

The shared task asks systems to find shallow discourse relations in news text. Each relation has a connective (explicit, such as "because", or implicit, inserted between adjacent sentences), two argument spans (Arg1 and Arg2, defined semantically for Chinese) and a sense (for example Cause, Condition or Contrast). The second edition covers English (PDTB 2.0 data, with the 2015 test sets) and Chinese (CDTB 0.5 with a blind Wikinews test set of 29,892 words). Twenty-three teams took part. The main score is end-to-end F1, where a relation counts only if both argument spans and the sense match exactly. A partial score accepts arguments with an average F1 above 0.7. The English winner, Oslo-Potsdam-Teesside, scored 27.77 F1 on the blind test, 3.77 points above the previous year's winner. The Chinese winner, cip2016 from the Institute of Automation, CAS, scored 26.90, 0.3 points above the second team. Explicit senses were predicted with high accuracy; the main challenge was implicit senses, which rely on the arguments. Partial scores were much higher than strict ones, showing that errors pass down a long pipeline. Two new practices were introduced: a blind test on TIRA, where systems run without seeing the test set, and a sense-only subtask with gold arguments.

Ideas

• I1. Connective table in Rust: for each explicit connective, a sense entry in a BTreeMap, marked unambiguous or ambiguous. Ambiguous ones (such as "since" or "while") get a rule that reads the frame, so that the sense comes from the structure, as the paper's remark on connectives suggests.
• I2. Report component scores beside the end-to-end score, as the scorer does: connective identification, Arg1 and Arg2 identification, and sense, each with precision, recall and F1, under strict and partial matching. A gain in the end score that does not show in a component is suspect.
• I3. Replicability: store each discourse run with a version tag and re-score the stored output, so a claimed gain can be checked without rerunning the pipeline. This is the TIRA idea in a local form.
• I4. Paired connectives ("not only ... but also", "either ... or"): the Chinese task shows that connective parts can be discontinuous. A rule over the tree that matches the two parts and their argument roles, rather than a contiguous string, is a small test case for en/src/parse.rs.

For Mova

The chapters are 36. Discourse relations and coherence (senses and connectives), 24. Coordination and subordination (clause linking) and 38. Dialogue, discourse markers and reported speech. The code is prag/src/frames.rs for sense labels, en/src/tok.rs and en/src/parse.rs for spans, and a new connective table if none exists yet (grep first). The measure: component F1 under strict and partial matching on any English discourse data the licence allows, run twice, with the sense-only subtask as the isolated test.

Topics: Discourse analysis

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/k16-2001

@inproceedings{xue2016conll,
  title = {{CoNLL 2016 Shared Task on Multilingual Shallow Discourse Parsing}},
  author = {Xue, Nianwen and Ng, Hwee Tou and Pradhan, Sameer and Rutherford, Attapol and Webber, Bonnie and Wang, Chuan and Wang, Hongmin},
  booktitle = {Proceedings of the CoNLL-16 shared task},
  year = {2016},
  doi = {10.18653/v1/k16-2001},
  url = {https://doi.org/10.18653/v1/k16-2001}
}

References

39 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nicholas Asher and Alex Lascarides. 2003. Logics of conversation. Cambridge University Press. find
2. Yusuke Kido and Akiko Aizawa. 2016. Discourse relation sense classification with two-step classifiers. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
3. Or Biran and Kathleen McKeown. 2013. Aggregated word pair features for implicit discourse relation disambiguation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. find
4. Prashant Chandrasekar, Xuan Zhang, Saurabh Chakravarty, Arijit Ray, John Krulick, and Alla Rozovskaya. 2016. The virginia tech system at conll-2016 shared task on shallow discourse parsing. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
5. Fang Kong, Sheng Li, and Guodong Zhou. 2015. The SoNLP-DP system in the CoNLL-2015 shared task. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning: Shared Task. find
6. Fang Kong, Sheng Li, Junhui Li, Muhua Zhu, and Guodong Zhou. 2016. Sonlp-dp system for conll-2016 english shallow discourse parsing. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
7. Ziwei Fan, Zhenghua Li, and Min Zhang. 2016. Finding arguments as sequence labeling in discourse parsing. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
8. Majid Laali, Andre Cianflone, and Leila Kosseim. 2016. The clac discourse parser at conll-2016. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
9. Tim Gollub, Benno Stein, and Steven Burrows. 2012. Ousting Ivory Tower Research: Towards a Web Framework for Providing Experiments as a Service. In Bill Hersh, Jamie Callan, Yoelle Maarek, and Mark Sanderson, editors, 35th International ACM Conference on Research and Development in Information Retrieval (SIGIR 12), pages 1125–1126. ACM, August. find
10. Junhui Li, Fang Kong, Sheng Li, Muhua Zhu, and Guodong Zhou. 2016a. Sonlp-dp system for conll-2016 chinese shallow discourse parsing. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
11. Devanshu Jain and Prasenjit Majumder. 2016. Da-iict submission for pdtb-styled discourse parser. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
12. Zhongyi Li, Hai Zhao, Chenxi Pang, Lili Wang, and Huan Wang. 2016b. A constituent syntactic parse tree based discourse parser. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
13. Ping Jian, Xiaohan She, Chenwei Zhang, Pengcheng Zhang, and Jian Feng. 2016. Discourse relation sense classification systems for conll-2016 shared task. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Percy Liang. 2005. Semi-supervised learning for natural language. Ph.D. thesis, Massachusetts Institute of Technology. American Chapter of the Association for Computational Linguistics. find
14. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. find our card
15. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2014. A PDTB-styled end-to-end discourse parser. Natural Language Engineering, 20(2):151–184. find
16. Jin Kiat Low, Hwee Tou Ng, and Wenyuan Guo. 2005. A maximum entropy approach to Chinese word segmentation. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, pages 161–164. find
17. Martin Potthast, Tim Gollub, Francisco Rangel, Paolo Rosso, Efstathios Stamatatos, and Benno Stein. 2014. Improving the Reproducibility of PAN’s Shared Tasks: Plagiarism Detection, Author Identification, and Author Profiling. In Evangelos Kanoulas, Mihai Lupu, Paul Clough, Mark Sanderson, Mark Hall, Allan Hanbury, and Elaine Toms, editors, Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. 5th International Conference of the CLEF Initiative (CLEF 14), pages 268– 299, Berlin Heidelberg New York, September. Springer. find
18. William C Mann and Sandra A Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text, 8(3):243–281. ↗ our card
19. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. find
20. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse Treebank 2.0. In Proceedings of the 6th International Conference on Language Resources and Evaluation. find our card
21. Todor Mihaylov and Annette Frank. 2016. Discourse relation sense classification using cross-argument semantic similarity based on word embeddings. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
22. Rashmi Prasad, Bonnie Webber, and Aravind Joshi. 2014. Reflections on the Penn Discourse Treebank, comparable corpora, and complementary annotation. Computational Linguistics, 40(4):921–950. find our card
23. Minh Nguyen. 2016. Sdp-jaist: A shallow discourse parsing system conll 2016 shared task. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
24. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016. Shallow discourse parsing using convolutional neural network. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
25. Stephan Oepen, Jonathon Read, Tatjana Scheffler, Uladzimir Sidarenka, Manfred Stede, Eric Velldal, and Lilja Ovrelid. 2016. Opt: Oslo– potsdam–teesside. pipelining rules, rankers, and classifier ensembles for shallow discourse parsing. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
26. Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through Brown cluster pair representation and coreference patterns. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. find
27. Maria Leonor Pacheco, I-Ta Lee Lee, Xiao Zhang, Abdullah Khan Zehady, Pranjal Daga, Di Jin, Ayush Parolia, and Dan Goldwasser. 2016. Adapting event embedding for implicit discourse relation recognition. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
28. Attapol T. Rutherford and Nianwen Xue. 2016. Robust non-explicit neural discourse parser in english and chinese. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
29. Niko Schenk, Christian Chiarcos, Kathrin Donandt, Samuel Samuel Rönnqvist, Evgeny Stepanov, and Giuseppe Riccardi. 2016. Do we really need all those rich linguistic features? find
30. Slav Petrov and Dan Klein. 2007. Improved inferencing for unlexicalized parsing. In Proceedings of the Human Language Technology/North a neural network-based approach to implicit sense labeling. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
31. Evgeny Stepanov and Giuseppe Riccardi. 2016. Unitn end-to-end discourse parser for conll 2016 shared task. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
32. Jianxiang Wang and Man Lan. 2015. A refined end-to-end discourse parser. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning: Shared Task. find
33. Jianxiang Wang and Man Lan. 2016. Two end-to-end shallow discourse parsers for english and chinese in conll-2016 shared task. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
34. Zhiguo Wang and Nianwen Xue. 2014. Joint pos tagging and transition-based constituent parsing in chinese with non-local features. In ACL (1), pages 733–742. find
35. Gregor Weiss and Marco Bajec. 2016. Discourse sense classification from scratch using focused rnns. In Proceedings of the Twentieth Conference on Computational Natural Language Learning: Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find
36. Naiwen Xue, Fei Xia, Fu-Dong Chiou, and Marta Palmer. 2005. The penn chinese treebank: Phrase structure annotation of a large corpus. Natural language engineering, 11(02):207–238. find
37. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The CoNLL-2015 shared task on shallow discourse parsing. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning: Shared Task. find
38. Yuping Zhou and Nianwen Xue. 2012. PDTB-style annotation of Chinese text. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. find
39. Yuping Zhou and Nianwen Xue. 2015. The chinese discourse treebank: a chinese corpus annotated with discourse relations. Language Resources and Evaluation, 49(2):397–431. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/xue-2016-conll-2016-shared-task-multilingual/