UD_Japanese-GSD
Treebank card from UD release 2.18: metadata and texts come from UD_Japanese-GSD/README.md; size is counted by streaming the .conllu files and checked against the release stats.xml. Generated by tools/treebanks build; do not edit by hand. Papers: Papers: UD_Japanese-GSD; seeds: seeds/; converter: convert.md.
| Field | Value |
|---|---|
| UD name | UD_Japanese-GSD |
| Language | Japanese (ja) |
| Family | Japanese |
| Treebank code | ja_gsd |
| UD page | universaldependencies.org/treebanks/ja gsd/index.html |
| License | CC BY-SA 4.0 |
| Genres | news, blog |
| Text in data | yes |
| Parallel | jagsd |
| In UD since | UD v1.4 |
| Authors | Mai Omura; Yusuke Miyao; Hiroshi Kanayama; Hiroshi Matsuda; Aya Wakasa; Kayo Yamashita; Masayuki Asahara; Takaaki Tanaka; Yugo Murawaki; Yuji Matsumoto; Shinsuke Mori; Sumire Uematsu; Ryan McDonald; Joakim Nivre; Daniel Zeman |
| Contact | hkana@jp.ibm.com |
| Development | in the UD repo (here) |
Annotation layers
| Layer | Origin |
|---|---|
| Lemmas | converted from manual |
| UPOS | converted from manual |
| XPOS | manual native |
| Features | not available |
| Relations | converted from manual |
UD scale: manual native — annotated by hand directly in UD; converted from manual — converted from manual annotation in another scheme; converted with corrections — converted and partly corrected by hand; automatic with corrections — automatic, with partial manual corrections; automatic — automatic; not available — the layer is absent.
Size
| Split | Sentences | Tokens | Words | Multiword tokens | Empty nodes |
|---|---|---|---|---|---|
| train | 7 050 | 168 333 | 168 333 | 0 | 0 |
| dev | 507 | 12 287 | 12 287 | 0 | 0 |
| test | 543 | 13 034 | 13 034 | 0 | 0 |
| total | 8 100 | 193 654 | 193 654 | 0 | 0 |
Tokens are surface tokens (Spanish vámonos is one), words are syntactic words (vamos + nos); multiword tokens are those split into several words; empty nodes are the N.M nodes of the enhanced graph.
Summary
This Universal Dependencies (UD) Japanese treebank is based on the definition of UD Japanese convention described in the UD documentation. The original sentences are from Google UDT 2.0.
Introduction
The Japanese UD treebank contains the sentences from Google Universal Dependency Treebanks v2.0 (legacy): github.com/ryanmcd/uni-dep-tb. First, Google UDT v2.0 was converted to UD-style with bunsetsu-based word units (say "master" corpus).
The word units in "master" is significantly different from the definition of the documents based on Short Unit Word (SUW) [1], then the sentences are automatically re-processed by Hiroshi Kanayama in Feb 2017. It is the Japanese_UD v2.0 and used in the CoNLL 2017 shared task. In November 2017, UD_Japanese v2.0 is merged with the "master" data so that the manual annotations for dependencies can be reflected to the corpus. It reduced the errors in the dependency structures and relation labels.
Still there are slight differences in the word unit between UD_Japanese v2.1 and UD_Japanese-KTC 1.3.
In May 2020, we introduce UD_Japanese BCCWJ[3] like coversion method for UD_Japanese GSD v2.6.
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/treebanks/ja/gsd/