2. Spelling, print conventions and tokens

EModE had no fixed spelling until well into the 17th century. A writer could spell the same word three ways on one page, and printers padded or trimmed words to justify a line. The texts Mova reads today are almost all respelled: the 1769 KJV, JPS 1917 and the Gutenberg tales use modern spelling with a thin layer of old forms. This chapter lists both: the real EModE variation that a future corpus (Shakespeare quartos, EEBO) would bring, and the residue that is already in Mova's data, with what the tokenizer and tagger do with it.

2.1 Letters

Sources: Hejná & Walkden 2022, ch. 4 (orthography and the printing press); Yang & Eisenstein 2016, Fig. 1 («and drewe vnto hym all ryottours & wylde dysposed persones» → «and drew unto him all ryottours & wild disposed persons», a VARD normalization). Yang & Eisenstein report that VARD at a 50 % threshold rewrites 12 % of the PPCEME tokens (236 298 of 1 961 157) and that tagging the PPCEME with a Penn Treebank-trained tagger reaches only 74.2 % without normalization and 76.7 % with it (their Table 4); combining normalization and domain adaptation gives 79.1 %.

2.2 Spellings and word splits that are still in JPS 1917

The LORD lemma split matters for the world of the text: one entity, three lemmas. The usual UD practice keeps the case of a proper noun's lemma as written (LORD). Chapter 13 proposes one lemma for all 1 811 tokens.

2.3 Quotation, punctuation and verse marks

• Quotation. JPS 1917 introduces speech with a colon and single quotes: «And God said: 'Let there be light.'» (jps:GEN:2, GEN 1.3). 855 sentences contain : '. Nested speech uses no second level of quotes: «God hath said: Ye shall not eat of it» (jps:GEN:63, GEN 3.3). The KJV has no quotation marks at all: «And God said, Let there be light» (KJV Gen 1:3). A parser must find the reported clause from the colon and the verb of saying, not from quote marks; in the db the quoted clause attaches as ccomp of said (jps:GEN:69).
• Colon and semicolon carry clause boundaries that modern English marks with full stops; sentences are long (183 660 tokens in 6 065 sentences = 30.3 tokens per sentence in the db).
• Italics and brackets. The KJV prints words added by the translators in italics; the eBible text marks them with brackets, «darkness [was] upon the face of the deep» (KJV Gen 1:2). They must be removed before parsing (done for the counts in chapter 1). JPS 1917 has no such marks.
• Paragraph marks. The KJV ¶ (pilcrow) opens a paragraph (KJV Gen 1:6). Remove it.
• Verse numbers inside sentences. A sentence that spans a different verse division in Hebrew and English may carry both numbers: «(32-33) Therefore the children of Israel eat not the sinew…» (db text of jps:GEN:1032).

2.4 Elision and contraction

EModE speech forms that appear in plays and in the tales (none in JPS 1917):

Abbott notes that metre drives many of these contractions in the plays (Abbott 1884, §459 on spelling, §460 on dropped prefixes, §461 «Other Contractions», §§462–466 on contraction that the spelling does not show). They are a tokenizer matter more than a grammar matter: a split that follows the modern contraction rules ('s, 'll, 're are separate tokens in UD English) keeps the EModE forms parseable.

2.5 Spelling variation in the tales

The 0.3 tales are 19th-century translations and retellings. They use modern spelling, but the dialogue keeps EModE and dialect forms on purpose:

• «"Then if ye did it your own self," cried the elf-mother shrilly, "what's the use o' making all this fash about it?"» (book:pg14241:294, More English Fairy Tales) — ye as a plain plural or polite you, o' for of, Scots fash (bother).
• «"Hasn't thou naught for supper, dame?" said he to his wife.» (book:pg14241:1137) — the northern dialect agreement thou has, contracted with n't, a mix that no standard EModE grammar lists.
• «"Well," quoth the old woman, "yon's for you."» (book:pg14241:304) — quoth (past of quethe, say) and yon (that, yonder).

2.6 What Mova's tokenizer and tagger do with spelling

Normalization or not? Yang & Eisenstein (2016) show that spelling normalization alone helps a modern tagger by about 2.5 points on PPCEME, and that the combination with domain adaptation is best. For the texts Mova reads now, normalization is not the issue: their spelling is already modern. What breaks the parser is morphology and closed-class words (chapters 3–5), which normalization tools like VARD leave alone because thou, hath and shalt are correct words, not misspellings. If Mova ever reads unnormalized EModE (EEBO-TCP, the First Folio), a respelling layer before the tokenizer — a deterministic table plus the positional u/v and i/j rules — is the cheap first step; the lexicon of this manual is the second.

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/languages/en-x-emode/02-spelling-and-tokens/