Sentence Boundary Detection on Line Breaks in Japanese

What it is about

In Japanese text, line breaks are not always sentence boundaries, and many of them do not follow a sentence delimiter. The authors show this on a balanced corpus, where 33.4% of line breaks are not followed by a delimiter. They build annotated corpora of line breaks, and they train sentence boundary detectors by fine-tuning BERT. The model trained on the balanced corpus gets an F1 of 94.8, and the model trained on the Jalan corpus gets 92.8 on the paper's evaluation data. The authors then test whether pseudo-annotated corpora can replace hand annotation, and find that much annotated data helps even when the writing style differs. The study is on Japanese only, and it uses a neural model.

Ideas

• I1. en/src/tok.rs: a line break in the input counts as a boundary only when the token before it has no final punctuation and a learned count says so. Measure F1 over boundary points on the noisy-text treebank treebanks/en/gumreddit used as a test set only, with the rule switched on and off.
• I2. Add a line-break feature to the period classifier of en/src/tok.rs, and check on the same test set whether the errors fall for the tokens that end a line.
• I3. The paper's test of pseudo-annotated data: generate line-broken English from EWT test sentences, using only the rule of this card, and measure whether the rule gives the same boundaries as the original text. This tests the rule without using any other data.

For Mova

The code is en/src/tok.rs, and the metric is en tok-eval. The chapter is 07. Punctuation, capital letters, abbreviations and numbers. The paper is about Japanese and uses BERT, so only the idea of line breaks as a cue carries over. GUMReddit is a test set only, never as training data. Measure first on EWT test with the line-break rule off, then on, twice.

Topics: Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.wnut-1.10

@inproceedings{hayashibe2020sentence,
  title = {{Sentence Boundary Detection on Line Breaks in Japanese}},
  author = {Hayashibe, Yuta and Mitsuzawa, Kensuke},
  booktitle = {Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)},
  year = {2020},
  doi = {10.18653/v1/2020.wnut-1.10},
  url = {https://doi.org/10.18653/v1/2020.wnut-1.10}
}

References

11 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Devlin, Ming-Wei Chang, et al. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 4171– 4186. find
2. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise Phoneme Classification with Bidirectional LSTM and Other Neural Network Architectures. Neural Networks, 18(5):602–610. find
3. Hen-Hsen Huang and Hsin-Hsi Chen. 2011. Pause and Stop Labeling for Chinese Sentence Boundary Detection. In Proceedings of the International Conference Recent Advances in Natural Language Processing 2011, pages 146–153. find
4. Hikari Konishi, Takenori Nakamura, et al. 2015. Correction of Sentence Boundaries in the Balanced Corpus of Contemporary Written Japanese DVD Version 1.0. NINJAL Research Papers, (9):81–100. find
5. John Lafferty, Andrew McCallum, et al. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of the 18th International Conference on Machine Learning, pages 282–289. find
6. Kikuo Maekawa. 2008. Balanced Corpus of Contemporary Written Japanese. In Proceedings of the 6th Workshop on Asian Language Resources. find
7. Yukiko Nishimura. 2003. Linguistic Innovations and Interactional Features of Casual Online Communication in Japanese. Journal of Computer-Mediated Communication, 9(1). find
8. Peng Qi, Yuhao Zhang, et al. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108. find our card
9. Noboru Sakai. 2013. The role of sentence closing as an emotional marker: A case of Japanese mobile phone e-mail. Discourse, Context & Media, 2(3):149–155. find
10. Akihiro Shinmori, Manabu Okumura, et al. 2003. Patent Claim Processing for Readability: Structure Analysis and Term Explanation. In Proceedings of the ACL-2003 Workshop on Patent Corpus Processing, pages 56–65. find
11. Conghui Zhu, Jie Tang, et al. 2007. A Unified Tagging Approach to Text Normalization. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 688–695. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2020-hayashibe-sentence-boundary-detection-line-breaks/