Mova0.6 pre-release

the flag of Japan 9. Word Classes and Segmentation

Old Japanese word classes are identified by where forms occur, what they combine with, and which inflections they take. The written record can place several grammatical units in one uninterrupted string, and a single graph can represent material of more than one kind. This chapter describes the grammatical distinctions and proposes how to represent them as tokens, Universal Dependencies part-of-speech tags, and features.

Traditional categories and distributional diagnostics

Traditional analysis distinguishes inflecting predicates, such as verbs and adjectives, from uninflected expressions, such as nouns and particles. It also distinguishes a predicate that can head a clause from an auxiliary that requires a host. These distinctions remain useful even when the modern labels applied to individual forms differ. Motoori’s treatment of particles and predicate endings illustrates why the analysis must extend across a clause. A particle next to a nominal can help account for the inflection of a predicate several words away. (Motoori 1785, § on koso and predicate endings.)

Distribution to checkWhat it helps identifyQualification
The form heads an argument phrase and can be followed by a case or discourse particle.Noun, proper name, pronoun, or numeral in nominal useAn adnominal predicate or a larger expression can itself occur in a nominal position.
The form has a stem and participates in a series of predicate endings.Verb, adjective, or auxiliarySome endings have the same surface form. Syntactic position and particle constructions help identify the form in use.
The form takes a predicate in a particular stem form, adds a grammatical meaning, and can itself carry an ending.Inflecting auxiliaryA lexical verb can also follow another predicate. Check whether it still contributes its own lexical predication.
The form follows a nominal, predicate, or clause and supplies a grammatical relation or discourse function.ParticleThe host can be a whole phrase or clause. Similar-looking particles can have different functions with different hosts.
The form modifies a following noun or qualifies a predicate without taking a predicate ending of its own.Adnominal expression or adverbA noun with a particle, an inflected adjective, and a subordinate clause can all serve as modifiers. Their function alone does not determine their word class.
The form links units of equal or unequal status.Coordinator or subordinatorCheck whether the linking function belongs to a free word, a particle, or a construction involving an inflected predicate.
The form can stand as an utterance or be set apart from the surrounding clause.InterjectionA clause-final particle still has a host even when the surrounding utterance is brief.

Use the diagnostics together. For example, a verb in its adnominal form can modify a noun directly, and an adjective can head a complete predicate. A particle after either form does not by itself establish that the preceding form is a noun. Conversely, a nominal expression with a predicative interpretation can be part of a copular construction.

The examples below use readable Japanese spellings and normalized romanization. The Japanese spellings do not reproduce the graphs of a particular witness. The romanization omits vowel distinctions that do not affect the analysis at hand. Spaces in the romanization aid reading, and hyphens in the gloss show morphological analysis. The token boundaries proposed for annotation are given separately where they matter.

Nominals, predicates and modifiers

Nominal expressions

Common nouns name entities, places, times, and other concepts. Proper names identify particular people, places, or other named entities. Both can head phrases with postposed particles and can occur with modifiers before them. Ordinary nouns have no obligatory singular–plural contrast in their endings. A bare noun can receive a singular, plural, or generic interpretation from the construction and context. A numeral, a collective expression, or an overt plural element supplies additional evidence when present.

Pronouns include personal forms such as ware and forms with demonstrative or interrogative uses. An independent form that fills a nominal position is distinct from an adnominal expression that accompanies a following noun. Where the latter is internally analyzable, retain that analysis. For example, the sequence ko no can be analyzed as the demonstrative ko followed by the adnominal particle no. An English translation with a single determiner does not settle the Old Japanese segmentation.

FormRomanizationGlossTranslation and context
我こそware koso1SG FOC“I myself,” part of a clause in which the speaker asserts his presence. (Keichū 1690, § on Man’yōshū 1.1.)
この丘にko no woka niPROX ADN hill LOC“On this hill.” (Keichū 1690, § on Man’yōshū 1.1.)
出雲IdumoIzumo“Izumo,” the place name in the serpent episode’s song. (Chamberlain 1882, § on the eight-forked serpent; Motoori 1822, § on the same episode.)

Numerals can count entities, occur with a counting element, or enter a compound. Determine whether the numeral is an independent syntactic unit before assigning it a separate token. The element ya ‘eight’ occurs inside ya-pe-gaki ‘eightfold fence’. Its meaning is recoverable inside that expression, which functions as a compound nominal. This example gives no basis for taking every occurrence of ya to be a numeral. The stem of a numeral, an interrogative or exclamatory particle of the same spelling, and an independent numeral are distinguished by their distributions. (Chamberlain 1882, § on the eight-forked serpent; Motoori 1822, § on the same episode.)

Verbs and adjectives

A lexical verb heads a predication and can license its own arguments. Its stem and inflections furnish forms used in environments such as auxiliary attachment, clause linkage, clause ending, modification of a noun, and commands. The following forms of yuku ‘go’ show how position helps recover the analysis when forms coincide. The traditional names designate slots in the paradigm.

Traditional slotFormRomanizationStem and endingDiagnostic environment
Irrealis行かyukayuk-aBefore compatible auxiliaries and in the hypothetical construction with ba.
Continuative行きyukiyuk-iBefore compatible auxiliaries and linkers such as te.
Conclusive行くyukuyuk-uAt the end of a clause whose construction calls for the conclusive.
Adnominal行くyukuyuk-uBefore a modified noun or in another construction that selects the adnominal.
Realis行けyukeyuk-eBefore ba in the construction with an established or contextually given circumstance. It is also selected by certain focus constructions.
Imperative行けyukeyuk-eIn a direct command.

The realis is a morphological slot. Its function in a construction can be more specific than the name suggests. In particular, the realis selected in a koso construction does not require a conditional interpretation. In ware koso wore ‘I myself am here’, the focused pronoun takes koso and the predicate is in the realis. (Keichū 1690, § on Man’yōshū 1.1; Motoori 1785, § on koso and predicate endings.)

Inflecting adjectives predicate properties, modify nouns in the adnominal form, and qualify predicates in an adverbial form. The distribution of the last form does not change the lexeme into a lexical adverb. The table shows the contrast among three environments in two adjective patterns. It does not enumerate all environments or adjective forms.

Pattern and meaningConclusiveAdnominalAdverbial
taka- ‘high’高し takasi高き takaki高く takaku
kana- ‘dear’愛し kanasi愛しき kanasiki愛しく kanasiku
FormRomanizationGlossTranslation
山高しyama takasimountain high-CONCL“The mountain is high.” (constructed)
高き山takaki yamahigh-ADN mountain“A high mountain.” (constructed)
高く飛ぶtakaku tobuhigh-ADV.FORM fly.CONCL“It flies high.” (constructed)
行く人yuku pitogo.ADN person“A person who goes.” (constructed)

Other modifiers

A lexical adverb modifies a predicate, another modifier, or a clause without participating in a verbal or adjectival inflectional paradigm. An adverbial position can also be filled by a time noun, a case phrase, an inflected adjective, or a clause. Thus ni on a place expression can give the phrase a locative use while the place word remains a noun. Similarly, in adnominal position, a noun followed by no, an adjective in its adnominal form, and a relative clause have different internal structures.

Words that express recurrence, sequence, or manner should be classified by their scope and role in the passage. For instance, mata in the meaning ‘again’ can qualify the event expressed by a predicate. When an expression occurs between sentences, its position alone does not establish that it coordinates them.

Particles, auxiliaries, conjunctions and interjections

Particles and their hosts

Particles are generally uninflected. Some follow nominal expressions and mark relations such as object, location, or attribution. Others follow predicates or clauses, or add focus, topic, interrogation, or another discourse function. A particle can follow a phrase containing another particle. Take each particle’s host and function from the construction.

UseExamples of forms and hostsDiagnostic
Case and nominal linkagewo after an object, ni with a location or goal, no and ga in adnominal linkageThe particle relates a nominal expression to another constituent. No and ga can also mark a subject in certain constructions, including adnominal clauses.
Topic and focuspa with a topic, mo with an additive or scalar reading, koso with focusThe particle ranges over a constituent. Its effect may extend to the form of the clause predicate.
Interrogationka or ya in interrogative usesPosition, the scope of the question, and the predicate construction help determine the analysis.
Clause linkagete after a continuative, ba after an irrealis or realisThe host is a predicate form. In the ba construction, the host form helps distinguish hypothetical conditions from established circumstances.
Quotationto after quoted materialThe preceding expression gives what is said, thought, or otherwise presented as a quotation. A nominal comitative to has a different host and relation.
Utterance markingFinal particles with a clause or utterance as hostTheir scope is the utterance or predicate. A form shared with a particle in another position is classified in context.
FormRomanizationGlossTranslation
山を見yama wo mimountain ACC see.CONT“Seeing the mountain,” part of a larger utterance. (constructed)
行かばyuka bago.IRR COND“If [someone] goes.” (constructed)
行けばyuke bago.REAL SUB“When [someone] goes,” or “since [someone] goes,” as the context permits. (constructed)
行くと言ふyuku to ipugo.CONCL QUOT say.CONCL“They say they are going.” (constructed)

Auxiliary predicates and auxiliary chains

An inflecting auxiliary selects a compatible form of the preceding predicate and contributes a grammatical value, for instance past, aspect, modality, or voice. If another element follows, that element can select a form of the auxiliary. The final item in such a chain can therefore bear the visible ending required by clause type or adnominal use. The lexical predicate’s own form can be a continuative or another selected stem even when the clause as a whole is finite.

Selection is specific to the auxiliary. For example, the past auxiliary ki attaches to a continuative in yuki ki. The lexical verb supplies the motion meaning, and the auxiliary carries the past. The perfective auxiliary nu and the adnominal negative form -nu must be distinguished by the host and the rest of the construction. A preceding continuative and a preceding irrealis provide different evidence. This matters especially where an annotation presents the two strings in the same romanization.

FormRomanizationGlossTranslation
行ききyuki kigo.CONT PST.CONCL“[Someone] went.” (constructed)
行かぬ人yukanu pitogo.IRR-NEG.ADN person“A person who does not go.” (constructed)

A following verb retains lexical status when it still introduces a distinct predication. Motion, existence, and speech verbs need particular attention because their lexical uses can resemble grammatical uses in a chain. The relevant evidence is the meaning of the second item, its argument structure, its selection of the preceding form, and its distribution elsewhere. Inflectability alone is shared by lexical verbs and inflecting auxiliaries.

Linkers and interjections

Coordination relates units of equal syntactic status. Subordination gives one unit a role within another, as in a condition or a quotation. Old Japanese often links clauses through the form of the preceding predicate, alone or together with a following particle. A continuative in a sequence of clauses remains a form of its verb. The tokenization of a linker depends on whether the expression contains a separate connecting word or particle.

An interjection can voice a reaction, call, or response independently of the surrounding clause. A vocative noun, by comparison, remains a nominal even when spoken in isolation. Likewise, an expressive final particle belongs with the host whose utterance force it marks.

FormRomanizationGlossTranslation
あな、高しana, takasiEXCL high-CONCL“Oh, it is high!” (constructed)

Bound morphemes and proposed token boundaries

A written graph, a morpheme, and a token are different units of analysis. An inflected predicate can contain several morphemes within one token. At the other extreme, a case or focus particle can have a separate token even when the script and pronunciation attach it closely to its host. Apply the following conventions to a grammatical reading of the passage.

  1. Keep a lexical base with its inflection. An ending that supplies a form of a verb or adjective remains inside its token. Segment yuku as one token in both the conclusive and the adnominal. Productive derivational affixes and reduplicative material within a single lexical word also remain inside that word.
  2. Give syntactically active particles separate tokens. Split them from nouns, pronouns, verbs, and clauses. Where particles stack, split each independently identifiable particle. Thus ware koso has two tokens. The same rule gives ye wo mo three tokens when wo marks the object and mo adds focus or inclusion. (Keichū 1690, § on Man’yōshū 1.1.)
  3. Split an auxiliary with a recoverable surface form and a distinct grammatical role. Keep the ending of the auxiliary inside its own token. In yuki ki, yuki and ki are separate tokens even though the auxiliary selects the continuative and depends on it. For a fused form without recoverable surface units, keep the attested form as one token. Analyze its grammatical contribution on that token only where the reading supports it.
  4. Keep the bound negative series with its predicate for this annotation. In yukazu and yukanu, the negative exponent and the predicate belong to one token. This convention represents bound negation within the verbal paradigm. It is compatible with describing the series as an auxiliary in traditional morphology. A separate auxiliary nu with perfective use follows the auxiliary rule when its boundary is identifiable.
  5. Keep lexical compounds together. A compound can contain recoverable roots, including a numeral, without a token boundary at each root. Split a sequence when the parts retain independent syntactic behavior, as when a demonstrative takes a separate adnominal particle or a noun phrase bears an external case particle. Read names as proper-name expressions and resolve internal boundaries from their construction, rather than from the number of graphs used to write them.
  6. Resolve ambiguity with the surrounding construction. Check the host form, the scope of the particle, the presence of other predicates, and the distribution of comparable forms. An editor’s space, a supplied reading, and a graph with more than one value are evidence about the text. They do not independently require a token boundary. If a reading remains uncertain, avoid adding analyses of internal material that the reading cannot establish.

The vertical bar in the table marks a proposed boundary. The gloss can show internal morphemes in either a single token or a longer sequence.

FormRomanization and token boundariesGlossTranslation
家をもye | wo | mohome ACC ADD“The home too.” (Keichū 1690, § on Man’yōshū 1.1.)
この丘にko | no | woka | niPROX ADN hill LOC“On this hill.” (Keichū 1690, § on Man’yōshū 1.1.)
八重垣yapegakieight-fold-fence“An eightfold fence.” (Chamberlain 1882, § on the eight-forked serpent; Motoori 1822, § on the same episode.)
行ききyuki | kigo.CONT PST.CONCL“[Someone] went.” (constructed)
行かずyukazugo.IRR-NEG“[Someone] does not go,” with a clause-ending reading. (constructed)
行くと言ふyuku | to | ipugo.CONCL QUOT say.CONCL“They say they are going.” (constructed)

Proposed Universal Dependencies tags and features

Part-of-speech tags

UPOS records the category of a token in context. A traditional category can correspond to more than one UPOS tag, especially where the traditional analysis groups particles by attachment or treats multiple types of linkage together. The table gives the conditions for using each UPOS label. It also covers tags whose application depends on the edition or on the analysis of a particular expression.

UPOSProposed use
NOUNCommon nominal heads, including ordinary time and place nouns used adverbially. Use it for lexical nominal compounds, including compounds with an internal numeral.
PROPNLexical tokens used as names of particular individuals, places, or other named entities. Analyze attached particles separately.
PRONIndependent personal, demonstrative, and interrogative forms that fill nominal positions. Use it for an independent demonstrative serving as the host of adnominal no.
DETAn independent token with an adnominal determiner distribution. Apply it only where the reading supports that unit and function. A sequence analyzed as PRON plus an adnominal particle receives those two tags.
NUMAn independent numeral used to count or order. For an ordinal, use this tag where the token retains numeral distribution. Tag a compound with a numeral inside it by the category of the whole compound.
VERBA lexical verbal predicate in any inflectional form, including continuatives, adnominals, and forms with integrated negation. Keep this tag for verbs with distinct lexical predication in a sequence.
ADJAn inflecting property word, whether used predicatively, adnominally, or in its adverbial form.
ADVA lexical adverb modifying a predicate, modifier, or clause. Use it for mata where the meaning is iterative ‘again’. A noun in a time expression retains NOUN.
AUXA separate grammatical predicate in an auxiliary chain, or a separately tokenized copular element in copular use. Grammatical function in context licenses the tag.
ADPA postposed case or relation marker with a nominal host, such as object wo, locative ni, adnominal no, or comitative to. This includes a marker on a nominal subject in an environment that licenses it.
PARTA separate topic, focus, additive, interrogative, or final particle in that function. Use it for koso with a focused constituent.
CCONJA separate word or particle whose function is to coordinate units. Use it where the analysis identifies coordination carried by the token itself.
SCONJA separate clause-linking particle in subordinating use, including conditional ba and to introducing a clausal quotation. Use it for te when it marks a dependent clause. Where a construction with te is analyzed as coordination and the token itself bears the coordinating function, use CCONJ.
INTJAn independent exclamation, call, or response of interjectional type.
PUNCTA punctuation mark retained as a token under the edition’s policy. The absence of a punctuation token does not determine a grammatical boundary.
SYMA retained token that functions as a nonlinguistic symbol. A graph representing a Japanese word receives the tag of that word.
XA retained linguistic token whose category cannot be assigned from the reading or analysis. Where the category is clear and some other detail is unresolved, assign the category and omit the uncertain detail.

Quote marking requires attention to the scope of the quoted material. In yuku to ipu, the quotation has a predicate, and to is SCONJ. A nominal with comitative to takes ADP for the particle. An interrogative particle in a reported question retains its interrogative function within the question, even when another token connects the quotation to the reporting predicate.

Morphological features

Put each feature on the token that expresses it. An auxiliary that independently expresses past takes the tense feature. A particle gets its own tag, while the noun next to it receives only features supported by the noun form or by the proposed treatment of its morphology. A clause can have tense, polarity, or discourse focus as an interpretation even when only one of its tokens carries an overt morphological exponent.

FeatureWhen to use it
PronTypeUse Prs, Dem, or Int for an identifiable personal, demonstrative, or interrogative pronominal form. Apply Dem or Int to a determiner if the determiner analysis is independently supported.
PersonAssign a value where the personal form grammatically encodes it. A discourse-supplied subject does not supply a person feature on an inflected verb.
NumberAssign a value where the form encodes number. Leave it unset for an ordinary number-neutral noun, including one whose plurality is recoverable from a separate numeral or context.
NumTypeUse Card for an independent cardinal numeral and Ord for an ordinal analyzed as a numeral. A compound noun retains its nominal tag and lacks a numeral feature based solely on an internal root.
Tense and AspectAdd a value where the construction and form support it. For example, use Tense=Past on past ki. Keep the analysis of perfective or resultative auxiliaries tied to their construction and to the meaning of the selected UD value.
MoodUse Mood=Imp for an imperative. In a construction with ba, the conditional relation itself is expressed through the linker and its selected host form. The name irrealis alone does not establish a UD mood value.
PolarityUse Polarity=Neg on a predicate with integrated negative morphology, including the bound forms in yukazu and yukanu.
VoiceAssign an identifiable voice value to the unit that carries it, for example Voice=Cau for a causative or Voice=Pass for a passive. Resolve multifunctional forms in context before selecting a value.
VerbFormUse a supported UD value where the analysis establishes the relevant distinction, for example VerbForm=Fin on a conclusive predicate. A traditional stem slot is recorded separately from this feature.
CaseUse a morphological case feature where case is expressed in the nominal form itself. With an independent case particle, represent the overt marker as its own token.
PoliteUse a value such as Elev or Humb where the token encodes the corresponding honorific distinction. Social information inferred from the passage does not by itself license a feature.

An Old Japanese XPOS scheme can preserve the traditional slot names and lexical inflection classes. The finite and adnominal uses of yuku can then receive different analyses despite their identical spelling. Likewise, the yuke selected by ba can be distinguished from an imperative yuke. Record an ambiguous slot only to the precision allowed by the construction. The following table shows how morphological features distribute over tokens. It gives selected features relevant to each contrast.

ExampleTokenUPOSSelected FEATS or other analysis
yuki ki ‘[Someone] went’yukiVERBContinuative slot.
yuki kikiAUXTense=Past, VerbForm=Fin.
yukanu pito ‘a person who does not go’yukanuVERBPolarity=Neg. The negative series supplies the adnominal form.
yukanu pitopitoNOUNOrdinary number-neutral noun.
yuke ba ‘when [someone] goes’yukeVERBRealis slot selected by ba.
yuke babaSCONJSubordinating particle.
行け yuke, go.IMP, ‘Go!’ (constructed)yukeVERBMood=Imp.

How Mova handles segmentation and tagging without an Old Japanese model

At level 1, tools/mova-parse uses tools/langseg to assign a language to sentences and spans. tools/langseg distinguishes English and French in Latin script, Hebrew-script varieties, and Hebrew or Aramaic in scholarly transliteration. Other scripts are reported as other. The Old Japanese writing in this chapter therefore falls under other, and the supported Latin-script distinctions provide no Old Japanese identification for its romanized examples.

At level 2, tools/mova-parse uses a model for the language of each span. There is no level-2 model for ojp. When a language model is absent, the # parse comment reports that condition. The output tokens receive _ in LEMMA, UPOS, XPOS, FEATS, HEAD, and DEPREL, with Parse=None in MISC. These rows provide no Old Japanese tokenization analysis, lemmatization, part-of-speech tagging, or dependency parse.

The en crate provides the training and evaluation workflow. After en xpos-fill, en train trains a parser and tagger on a language’s UD training split, and en parse-eval measures UAS and LAS on the development split. Mova has no annotated Old Japanese treebank, so there is no measured baseline for ojp. The English-tree consumers prag and world have no Old Japanese parse to read.

Squid reads the manual’s grammatical points as claims. Formalizable claims enter its queue and await implementation. The proposed particle and auxiliary boundaries, UPOS assignments, and morphological features in this chapter are all unimplemented. Those decisions supply the kinds of rules a tokenizer, tagger, and UD treebank baseline would need first.

Sources for this chapter

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/languages/ja/ojp/09-word-classes-and-segmentation/