UD_Persian-PerDT
Treebank card from UD release 2.18: metadata and texts come from UD_Persian-PerDT/README.md; size is counted by streaming the .conllu files and checked against the release stats.xml. Generated by tools/treebanks build; do not edit by hand. Papers: Papers: UD_Persian-PerDT; seeds: seeds/; converter: convert.md.
| Field | Value |
|---|---|
| UD name | UD_Persian-PerDT |
| Language | Persian (fa) |
| Family | IE |
| Treebank code | fa_perdt |
| UD page | universaldependencies.org/treebanks/fa perdt/index.html |
| License | CC BY-SA 4.0 |
| Genres | news, fiction, nonfiction, academic, web, blog |
| Text in data | yes |
| Parallel | no |
| In UD since | UD v2.7 |
| Authors | Mohammad Sadegh Rasooli; Pegah Safari; Amirsaeid Moloodi; Alireza Nourian |
| Contact | rasooli@seas.upenn.edu, pegh.safari@gmail.com |
| Development | here source |
Annotation layers
| Layer | Origin |
|---|---|
| Lemmas | converted with corrections |
| UPOS | converted from manual |
| XPOS | manual native |
| Features | converted with corrections |
| Relations | converted with corrections |
UD scale: manual native — annotated by hand directly in UD; converted from manual — converted from manual annotation in another scheme; converted with corrections — converted and partly corrected by hand; automatic with corrections — automatic, with partial manual corrections; automatic — automatic; not available — the layer is absent.
Size
| Split | Sentences | Tokens | Words | Multiword tokens | Empty nodes |
|---|---|---|---|---|---|
| train | 26 196 | 445 587 | 452 496 | 6 508 | 0 |
| dev | 1 456 | 24 751 | 25 147 | 369 | 0 |
| test | 1 455 | 23 825 | 24 133 | 296 | 0 |
| total | 29 107 | 494 163 | 501 776 | 7 173 | 0 |
Tokens are surface tokens (Spanish vámonos is one), words are syntactic words (vamos + nos); multiword tokens are those split into several words; empty nodes are the N.M nodes of the enhanced graph.
Summary
The Persian Universal Dependency Treebank (PerUDT) is the result of automatic coversion of Persian Dependency Treebank (PerDT) with extensive manual corrections. Please refer to the follwoing work, if you use this data:
- Mohammad Sadegh Rasooli, Pegah Safari, Amirsaeid Moloodi, and Alireza Nourian. "The Persian Dependency Treebank Made Universal". 2020 (to appear).
Introduction
The Persian Universal Dependency Treebank (PerUDT) is based on <a href="https://www.aclweb.org/anthology/N13-1031v1.pdf"> Persian Dependency Treebank (PerDT) </a>(Rasooli et al.,2013). The original Treebank consists of 29K sentences sampled from contemporary Persian text in different genres including: news, academic papers, magazine articles and fictions.
This treebank was annotated based on a language-specific schema and its automatic conversion involved three main steps: revising tokenization, POS mapping and dependency mapping.
In tokenization step, in order to separate multiword inflections of simple verbs grouped as one token in PerDT, we followed the guidelines in (Rasooli et al., 2013, Table 3) to automatically find the main verbs. Also we automatically separated pronominal clitics.
In POS conversion step, we used the state of the art <a href="https://arxiv.org/abs/2003.08875"> BERT-based Persian NER tagger</a> (Taher et al.,2020) with manual corrections to extend recall. Through seven different entities detected by tagger, we used Person and Location to mark PROPN tags.
PerDT contains 43 syntactic relations with no straightforward mapping for most of them, conjunctions arranged from the beginning of the sentence to the end and more importantly, prepositions regarded as the head of prepositional phrases and auxiliary verbs as the head of sentences. So we rearranged the order of conjunctions from end to the beginning through a script and tailored rules to convert each kind of relation to its UD version properly. Through the whole process and at the end of each step, we investigated the results and applied manual corrections if it was needed.
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/treebanks/fa/perdt/