UD_Japanese-BCCWJLUW
Treebank card from UD release 2.18: metadata and texts come from UD_Japanese-BCCWJLUW/README.md; size is counted by streaming the .conllu files and checked against the release stats.xml. Generated by tools/treebanks build; do not edit by hand. Papers: Papers: UD_Japanese-BCCWJLUW; seeds: seeds/; converter: convert.md.
| Field | Value |
|---|---|
| UD name | UD_Japanese-BCCWJLUW |
| Language | Japanese (ja) |
| Family | Japanese |
| Treebank code | ja_bccwjluw |
| UD page | universaldependencies.org/treebanks/ja bccwjluw/index.html |
| License | CC BY-NC-SA 4.0 |
| Genres | news, nonfiction, fiction, blog, web |
| Text in data | no |
| Parallel | no |
| In UD since | UD v2.9 |
| Authors | Mai Omura; Masayuki Asahara; Yusuke Miyao; Takaaki Tanaka; Hiroshi Kanayama; Yuji Matsumoto; Shinsuke Mori; Sumire Uematsu; Yugo Murawaki |
| Contact | masayu-a@ninjal.ac.jp |
| Development | outside the UD repo (elsewhere): fixes go to the source or the converter |
Annotation layers
| Layer | Origin |
|---|---|
| Lemmas | not available |
| UPOS | converted from manual |
| XPOS | not available |
| Features | not available |
| Relations | converted from manual |
UD scale: manual native — annotated by hand directly in UD; converted from manual — converted from manual annotation in another scheme; converted with corrections — converted and partly corrected by hand; automatic with corrections — automatic, with partial manual corrections; automatic — automatic; not available — the layer is absent.
Size
| Split | Sentences | Tokens | Words | Multiword tokens | Empty nodes |
|---|---|---|---|---|---|
| train | 40 801 | 715 759 | 715 759 | 0 | 0 |
| dev | 8 427 | 145 398 | 145 398 | 0 | 0 |
| test | 7 881 | 134 475 | 134 475 | 0 | 0 |
| total | 57 109 | 995 632 | 995 632 | 0 | 0 |
Tokens are surface tokens (Spanish vámonos is one), words are syntactic words (vamos + nos); multiword tokens are those split into several words; empty nodes are the N.M nodes of the enhanced graph.
Summary
This Universal Dependencies (UD) Japanese treebank is based on the definition of UD Japanese convention described in the UD documentation. The original sentences are from `Balanced Corpus of Contemporary Written Japanese'(BCCWJ).
UD-Japanese-BCCWJLUW is the other word segmentation version of UD-Japanese-BCCWJ. We use Long Unit Word (LUW) as their syntactic word in UD definition.
Introduction
The Japanese UD treebank contains the sentences from BCCWJ [1] clrd.ninjal.ac.jp/bccwj/en with BCCWJ-DepPara[2] annotation.
The word units is based on Long Unit Word in BCCWJ [1]. We prepared conversion rules from BCCWJ-DepPara to UD_Japanese v2.1 guidelines [3][4].
Recovering data
The data is provided in the CoNLL format, but original texts are stripped off due to the license issue.
The corpus is obtained by running the following command:
./merge/merge.sh -c BCCWJ_CORE_FILEwhere
BCCWJ_CORE_FILEdenotes the file of BCCWJ core file (core_LUW.txt, OT) the script requires Python (3.x)or
The BCCWJ DVD edition purchaser can download the data with the original text from bccwj-data.ninjal.ac.jp
Spliting
Each data set contains UD annotations for the following parts in BCCWJ
training: annotation B, C, D, E subsets development: annotation A subset test: annotation A subset
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/treebanks/ja/bccwjluw/