A Massive Scale Semantic Similarity Dataset of Historical English

What it is about

The paper builds HEADLINES, about 393.6 million positive pairs of headlines from off-copyright U.S. local newspapers, covering 1920 to 1989. Local papers reprinted wire stories but wrote their own headlines, so the headlines of two reprints of the same story form a positive pair. The pipeline digitises front pages, OCRs the content regions, links headlines, bylines and article text with a layout-plus-language model, and then finds articles from the same source despite noise and abridgement with a neural bi-encoder. Content falls sharply in the late 1970s after a copyright change in 1978. Benchmarked on the clustering task of the Massive Text Embedding Benchmark, the best average v-measure is 43.69 (MPNet). The association model is trained on 12,769 positive pairs.

Ideas

• I1. Take the pair design, not the neural model. A school question is a short "headline" and the lesson sentences are the "articles". A count-based similarity over lemmas (TF-IDF, lemmas in BTreeMap order) gives a deterministic retrieval, and its explanation is the list of shared lemmas.
• I2. Measure: for each school question, is the right lesson sentence ranked first? The negative control is the same retrieval with the lesson sentences shuffled, which must fall to chance. Run it twice.
• I3. If the licence check passes, use a sample of the dataset as an evaluation set for short-text similarity only, never in a release. Its decade statistics (words per headline, line counts) also suggest a drift check for vocabulary between early-modern and modern samples in 10. Vocabulary: archaic words and false friends.

For Mova

The relevant code is world/src/qa.rs (answering), world/src/reader.rs and world/src/knowledge.rs (lesson sentences). The relevant chapter is 19. The lexicon: vocabulary, multiword expressions and register. The measures: top-1 retrieval accuracy on school questions, and the negative control with shuffled lessons.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2306.17810

@misc{silcock2023massive,
  title = {{A Massive Scale Semantic Similarity Dataset of Historical English}},
  author = {Silcock, Emily and Dell, Melissa},
  year = {2023},
  eprint = {2306.17810},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.17810}
}

References

27 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A MBA H OMBAIAH , S., C HEN , T., Z HANG , M., B ENDERSKY , M., AND N AJORK , M. Dynamic language models for continuously evolving content. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (2021), pp. 2514–2524. find
2. BERT, S. stackexchange, 2021. find
3. C HEN , X., F ANG , H., L IN , T.-Y., V EDANTAM , R., G UPTA , S., D OLLÁR , P., AND Z IT -NICK , C. L. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015). ↗
4. E HRMANN , M., H AMDI , A., P ONTES , E. L., R OMANELLO , M., AND D OUCET , A. Named entity recognition and classification in historical documents: A survey. ACM Computing Surveys (2021). find
5. E THAYARAJH , K. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. arXiv preprint arXiv:1909.00512 (2019). ↗
6. F ADER , A., Z ETTLEMOYER , L., AND E TZIONI , O. Open question answering over curated and extracted knowledge bases. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (2014), pp. 1156–1165. find
7. F ALKNER , S., K LEIN , A., AND H UTTER , F. BOHB: Robust and efficient hyperparameter optimization at scale. In Proceedings of the 35th International Conference on Machine Learning (10–15 Jul 2018), J. Dy and A. Krause, Eds., vol. 80 of Proceedings of Machine Learning Research, PMLR, pp. 1437–1446. find
8. G EBRU , T., M ORGENSTERN , J., V ECCHIONE , B., V AUGHAN , J. W., W ALLACH , H., AU 2, H. D. I., AND C RAWFORD , K. Datasheets for datasets, 2021. find
9. G UARNERI , J. Newsprint Metropolis. University of Chicago Press, 2017. find
10. H ADSELL , R., C HOPRA , S., AND L E C UN , Y. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06) (2006), vol. 2, IEEE, pp. 1735–1742. find
11. H E , K., G KIOXARI , G., D OLLÁR , P., AND G IRSHICK , R. Mask r-cnn. Proceedings of the IEEE international conference on computer vision (2017), 2961–2969. find
12. H ENDERSON , M., B UDZIANOWSKI , P., C ASANUEVA , I., C OOPE , S., G ERZ , D., K UMAR , G., M RKŠI Ć , N., S PITHOURAKIS , G., S U , P.-H., V ULI Ć , I., ET AL . A repository of conversational datasets. In Proceedings of the First Workshop on NLP for Conversational AI (2019), pp. 1–10. find
13. I YER , S., D ANDEKAR , N., AND C SERNAI , K. First quora dataset release: Question pairs, 2017. find
14. J OHNSON , J., D OUZE , M., AND J ÉGOU , H. Billion-scale similarity search with gpus. IEEE Transactions on Big Data 7, 3 (2019), 535–547. find
15. L IU , Y., O TT , M., G OYAL , N., D U , J., J OSHI , M., C HEN , D., L EVY , O., L EWIS , M., Z ETTLEMOYER , L., AND S TOYANOV , V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). ↗
16. L OUREIRO , D., B ARBIERI , F., N EVES , L., A NKE , L. E., AND C AMACHO -C OLLADOS , J. Timelms: Diachronic language models from twitter. arXiv preprint arXiv:2202.03829 (2022). ↗
17. M ANJAVACAS , E., AND F ONTEYN , L. Macberth: Development and evaluation of a historically pre-trained language model for english (1450-1950). In Proceedings of the Workshop on Natural Language Processing for Digital Humanities (2021), pp. 23–36. find our card
18. M ANJAVACAS , E., AND F ONTEYN , L. Adapting vs. pre-training language models for historical languages. Journal of Data Mining & Digital Humanities, Digital humanities in languages (2022). find
19. M UENNIGHOFF , N., T AZI , N., M AGNE , L., AND R EIMERS , N. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316 (2022). ↗
20. O CKERBLOOM , J. M. Everybody’s library questions: Newspaper copyrights, notices, and renewals, 2019. find
21. R ASTAS , I., R YAN , Y. C., T IIHONEN , I., Q ARAEI , M., R EPO , L., B ABBAR , R., M ÄKELÄ , E., T OLONEN , M., AND G INTER , F. Explainable publication year prediction of eighteenth century texts with the bert model. In Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change (2022), pp. 68–77. find
22. R EIMERS , N., AND G UREVYCH , I. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019). ↗
23. R OSENBERG , A., AND H IRSCHBERG , J. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL) (2007), pp. 410–420. find
24. S ILCOCK , E., D’A MICO -W ONG , L., Y ANG , J., AND D ELL , M. Noise-robust de-duplication at scale. International Conference on Learning Representations (2023). find
25. S ONG , K., T AN , X., Q IN , T., L U , J., AND L IU , T.-Y. Mpnet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems 33 (2020), 16857–16867. find
26. W ANG , T., AND I SOLA , P. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. International Conference on Machine Learning 119 (2020), 9929–9939. find
27. Y OUNG , P., L AI , A., H ODOSH , M., AND H OCKENMAIER , J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2 (2014), 67–78. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/silcock-2023-massive-scale-semantic-similarity-dataset/