Human-AI Annotation Error Auditing for Hebrew Diacritization with Frontier LLMs

What it is about

The authors audit the labels of the EACL 2023 Hebrew Homograph Challenge Set (about 150,000 sentences, niqqud on homographs in Modern Hebrew) for annotation errors. Frontier LLMs are asked to flag likely niqqud errors, with a recall-first prompt, and their flags are checked by a human adjudicator. On a focused set of 7,241 sentences, the gold standard has 271 errors. Gemini 3 Pro reaches 83.6% recall at batch size 50 and 99.1% precision, against 62.4% and 42.8% for the two human experts, and 73.4% for the union of their findings. Recall stays near 83% from batch 50 to 200 and falls to 71.4% at batch 500. Morphology and dictionary context did not help consistently. Errors missed by all models were construct-versus-adjective cases and dormant number mismatches that need collocational knowledge. The corrected set and difficulty labels (Easy, Medium, Hard, Undetected) are released.

Ideas

• I1. An audit step for treebank labels: a teacher (a large model, offline) flags suspicious labels in batches of about 200 items, and a deterministic rule set decides which flags to keep. The large model never answers at test time; it only proposes what to check, which matches Mova's teacher role.
• I2. Stratify every test by difficulty, using the cross-system detection rate of each error as a label. A Mova rule that finds only the easy errors shows its ceiling in the numbers.
• I3. A checklist of the two error types the paper finds hard: the construct-versus-attributive ambiguity and the dormant singular-versus-plural cases. Each can be written as a feature check over the trees (noun in construct with an adjective-like neighbour; number mismatch with a collocate).

For Mova

I1 and I2 fit treebanks/hbo/ptnk and tools/macula compare: an audit pass that flags disagreement between Mova's parse and the gold labels, with the batch and difficulty stratification in the report. I3 is a set of checks for The Noun, the Adjective and the Numeral (construct state) and for Pronouns, particles and prepositions where number mismatch is discussed. The paper is on Modern Hebrew; its batch sizes and recall are not Mova's numbers and should be measured again on Biblical data.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.law-main.4

@inproceedings{gershuni2026human,
  title = {{Human-AI Annotation Error Auditing for Hebrew Diacritization with Frontier LLMs}},
  author = {Gershuni, Hillel and Shmidman, Avi},
  booktitle = {Proceedings of the 20th Linguistic Annotation Workshop (LAW XX)},
  year = {2026},
  doi = {10.18653/v1/2026.law-main.4},
  url = {https://doi.org/10.18653/v1/2026.law-main.4}
}

References

3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yaacov Choueka and Yoni Neeman. 1995. Nakdan-text,(an in-context text-vocalizer for modern hebrew). In BISFAI-95, The Fifth Bar Ilan Symposium for Artificial Intelligence. find
2. Domenic V. Cicchetti and Alvan R. Feinstein. 1990. High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6):551–558. find
3. Alvan R. Feinstein and Domenic V. Cicchetti. 1990. High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2026-gershuni-human-ai-annotation-error-auditing/