UD_Japanese-BCCWJ
Treebank card from UD release 2.18: metadata and texts come from UD_Japanese-BCCWJ/README.txt; size is counted by streaming the .conllu files and checked against the release stats.xml. Generated by tools/treebanks build; do not edit by hand. Papers: Papers: UD_Japanese-BCCWJ; seeds: seeds/; converter: convert.md.
| Field | Value |
|---|---|
| UD name | UD_Japanese-BCCWJ |
| Language | Japanese (ja) |
| Family | Japanese |
| Treebank code | ja_bccwj |
| UD page | universaldependencies.org/treebanks/ja bccwj/index.html |
| License | CC BY-NC-SA 4.0 |
| Genres | news, nonfiction, fiction, blog, web |
| Text in data | no |
| Parallel | no |
| In UD since | UD v2.2 |
| Authors | Mai Omura; Masayuki Asahara; Yusuke Miyao; Takaaki Tanaka; Hiroshi Kanayama; Yuji Matsumoto; Shinsuke Mori; Sumire Uematsu; Yugo Murawaki |
| Contact | masayu-a@ninjal.ac.jp |
| Development | outside the UD repo (elsewhere): fixes go to the source or the converter |
Annotation layers
| Layer | Origin |
|---|---|
| Lemmas | not available |
| UPOS | converted from manual |
| XPOS | not available |
| Features | not available |
| Relations | converted from manual |
UD scale: manual native — annotated by hand directly in UD; converted from manual — converted from manual annotation in another scheme; converted with corrections — converted and partly corrected by hand; automatic with corrections — automatic, with partial manual corrections; automatic — automatic; not available — the layer is absent.
Size
| Split | Sentences | Tokens | Words | Multiword tokens | Empty nodes |
|---|---|---|---|---|---|
| train | 40 801 | 908 738 | 908 738 | 0 | 0 |
| dev | 8 427 | 178 306 | 178 306 | 0 | 0 |
| test | 7 881 | 166 859 | 166 859 | 0 | 0 |
| total | 57 109 | 1 253 903 | 1 253 903 | 0 | 0 |
Tokens are surface tokens (Spanish vámonos is one), words are syntactic words (vamos + nos); multiword tokens are those split into several words; empty nodes are the N.M nodes of the enhanced graph.
Summary
This Universal Dependencies (UD) Japanese treebank is based on the definition of UD Japanese convention described in the UD documentation. The original sentences are from `Balanced Corpus of Contemporary Written Japanese'(BCCWJ).
Introduction
The Japanese UD treebank contains the sentences from BCCWJ [1] clrd.ninjal.ac.jp/bccwj/en with BCCWJ-DepPara[2] annotation.
The word units is based on Short Unit Word in BCCWJ [1]. We prepared conversion rules from BCCWJ-DepPara to UD_Japanese v2.1 guidelines [3][4].
Recovering data
The data is provided in the CoNLL format, but original texts are stripped off due to the license issue.
The corpus is obtained by running the following command:
./merge/merge.sh -c BCCWJ_CORE_FILEwhere
BCCWJ_CORE_FILEdenotes the file of BCCWJ core file (core_SUW.txt) the script requires Python (3.x)or
The BCCWJ DVD edition purchaser can download the data with the original text from bccwj-data.ninjal.ac.jp
Spliting
Each data set contains UD annotations for the following parts in BCCWJ
training: annotation B, C, D, E subsets development: annotation A subset test: annotation A subset
See also github.com/masayu-a/BCCWJ-ANNOTATION-ORDER
Citation
You are encouraged to cite the following paper when you refer to the Universal Dependencies Japanese Treebank.
Asahara, M., Kanayama, H., Tanaka, T., Miyao, Y., Uematsu, S., Mori, S., Matsumoto, Y., Omura, M., & Murawaki, Y. (2018). Universal Dependencies Version 2 for Japanese. In LREC-2018.
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/treebanks/ja/bccwj/