Unsupervised Multilingual Sentence Boundary Detection

What it is about

The authors build a sentence boundary detector that needs no annotated data and no language-specific resources. Most ambiguous periods come from abbreviations, so the system first finds abbreviations from an unannotated corpus. Three criteria decide this, and all three use only the candidate type, not its context: a truncated word with a final period is a tight collocation, abbreviations are usually short, and some contain internal periods. The same collocation evidence also helps with initials and ordinal numbers. The system was developed on English and tested on eleven languages with newspaper corpora. Without any change to the system, the average error rate for sentence boundaries is 1.26%, ranging from 0.35% on German to 2.12% on Estonian, and the average error rate for abbreviation detection is 0.80%.

Ideas

• I1. en/src/tok.rs: an abbreviation detector that scores each token ending in a period by its collocation with the period, its length and internal periods, learned from an unannotated English corpus, with a fixed threshold. Measure the sentence-boundary error rate on EWT test against the current rules, and the abbreviation error rate on the same tokens.
• I2. Use the paper's separate measures: report the error of abbreviation detection and of boundary detection, so a bad abbreviation list is visible in the number, not only in the final score.
• I3. Learn the initials and ordinal-number decisions with the same collocation counts, and check on EWT test whether the count of "A." and "1st." cases goes down, since the paper shows the gain only for these subtasks.

For Mova

The code is en/src/tok.rs, with the metric from en tok-eval. The chapter is 07. Punctuation, capital letters, abbreviations and numbers. The collocation statistics come from a corpus, and the paper's own corpora are news in other languages, so the numbers are not Mova's. Use the English part only as a check. Measure first on EWT test, with a two-fold check (en folds).

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/coli.2006.32.4.485

@article{kiss2006unsupervised,
  title = {{Unsupervised Multilingual Sentence Boundary Detection}},
  author = {Kiss, Tibor and Strunk, Jan},
  journal = {Computational Linguistics},
  year = {2006},
  doi = {10.1162/coli.2006.32.4.485},
  url = {https://doi.org/10.1162/coli.2006.32.4.485}
}

References

26 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aluisio, Sandra M., Gisele M. Pinheiro, Marcelo Finger, Maria das Graças V. Nunes, and Stella E. O. Tagnin. 2003. The Lacio-Web Project: Overview and issues in Brazilian Portuguese corpora creation. In Proceedings of Corpus Linguistics, pages 14–21, Lancaster, UK. find
2. Brill, Eric. 1995. Transformation-based error-driven learning and natural language processing: A case study in part-of-speech tagging. Computational Linguistics, 21(4):543–565. find
3. Dudenredaktion, editor. 2004. Duden. Die deutsche Rechtschreibung (23rd ed.), volume 1 of Der Duden in zwölf Bänden. Dudenverlag, Mannheim. find
4. Dunning, Ted. 1993. Accurate methods for the statistics of surprise and coincidence. Computational Linguistics, 19(1):61–74. find
5. Evert, Stefan and Brigitte Krenn. 2001. Methods for the qualitative evaluation of lexical association measures. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 188–195, Toulouse, France. find
6. Firth, John R. 1957. A synopsis of linguistic theory 1930-55. In Studies in Linguistic Analysis (special volume of the Philological Society). The Philological Society, Oxford, pages 1–32. find
7. Francis, Nelson W. and Henry Kucera. 1982. Frequency Analysis of English Usage: Lexicon and Grammar. Houghton Mifflin, New York. find
8. Grefenstette, Gregory. 1999. Tokenization. In Hans van Halteren, editor, Syntactic Wordclass Tagging. Kluwer Academic Publishers, Dordrecht, pages 117–133. find
9. Grefenstette, Gregory and Pasi Tapanainen. 1994. What is a word, what is a sentence? Problems of tokenization. In Proceedings of the 3rd International Conference on Computational Lexicography, pages 79–87, Budapest, Hungary. find
10. Kiss, Tibor and Jan Strunk. 2002a. Scaled log likelihood ratios for the detection of abbreviations in text corpora. In Proceedings of COLING 2002, pages 1228–1232, Taipei, Taiwan. find
11. Kiss, Tibor and Jan Strunk. 2002b. Viewing sentence boundary detection as collocation identification. In Proceedings of KONVENS 2002, pages 75–82, Saarbrücken, Germany. find
12. Manning, Christopher D. and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. The MIT Press, Cambridge, MA. find
13. Marcus, Mitchell P., Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
14. Mikheev, Andrei. 2000. Tagging sentence boundaries. In Proceedings of ANLP-NAACL 2000, pages 264–271, Seattle, Washington. find
15. Mikheev, Andrei. 2002. Periods, capitalized words, etc. Computational Linguistics, 28(3):289–318. find
16. Müller, Hans, V. Amerl, and G. Natalis. 1980. Worterkennungsverfahren als Grundlage einer Universalmethode zur automatischen Segmentierung von Texten in Sätze. Ein Verfahren zur maschinellen Satzgrenzenbestimmung im Englischen. Sprache und Datenverarbeitung, 4(1):46–64. find
17. Nunberg, Geoffrey. 1990. The Linguistics of Punctuation, volume 18 of CSLI Lecture Notes. CSLI Publications, Stanford, CA. find
18. Palmer, David D. and Marti A. Hearst. 1997. Adaptive multilingual sentence boundary disambiguation. Computational Linguistics, 23(2):241–267. find
19. Reynar, Jeffrey C. and Adwait Ratnaparkhi. 1997. A maximum entropy approach to identifying sentence boundaries. In Proceedings of the Fifth ACL Conference on Applied Natural Language Processing, pages 16–19, Washington, DC. find
20. Riley, Michael D. 1989. Some applications of tree-based modeling to speech and language indexing. In Proceedings of the DARPA Speech and Natural Language Workshop, pages 339–352, Cape Cod, MA. find
21. Silla Jr., Carlos N. and Celso A. A. Kaestner. 2004. An analysis of sentence boundary detection systems for English and Portuguese documents. In Proceedings of CICLing 2004, pages 135–141, Seoul, Korea. find
22. Silla Jr., Carlos N., Jaime Dalla Valle Jr., and Celso A. A. Kaestner. 2003. Detecção automática de sentenças com o uso de expressões regulares. In Proceedings of CBComp 2003, pages 548–560, Itajaı́, Brazil. find
23. Stamatatos, Efstathios, Nikos Fakotakis, and George K. Kokkinakis. 1999. Automatic extraction of rules for sentence boundary disambiguation. In Proceedings of the Workshop on Machine Learning in Human Language Technology, pages 88–92, Chania, Greece. find
24. Strunk, Jan, Carlos N. Silla Jr., and Celso A. A. Kaestner. 2006. A comparative evaluation of a new unsupervised sentence boundary detection approach on documents in English and Portuguese. In Proceedings of CICLing 2006, pages 132–143, Mexico City, Mexico. van Rijsbergen, Cornelis J. 1979. Informat… find
25. Walker, Daniel J., David E. Clements, Maki Darwin, and Jan W. Amtrup. 2001. Sentence boundary detection: A comparison of paradigms for improving MT quality. In Proceedings of the MT Summit VIII, Santiago de Compostela, Spain. find
26. Willmann, Helmut and Heinz Messinger, editors. 1996. Langenscheidts Großwörterbuch Englisch. Der Kleine Muret-Sanders (7th ed.), volume I: English– German. Langenscheidt, Berlin/Munich. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/cl-2006-kiss-unsupervised-multilingual-sentence-boundary-detection/