Mova0.5 pre-release⌕
☰
home / docs / papers / Evaluating large language models for the tasks of PoS tagging within the Universal Dependency framework

Evaluating large language models for the tasks of PoS tagging within the Universal Dependency framework

Summary

The paper tests how well large language models, without fine-tuning, assign the 17 UD universal POS tags to Brazilian Portuguese text. Evaluation uses the journalistic part of the Porttinari-base treebank, released after the models were trained, which lowers the risk of test contamination. Three systems are compared: GPT-3 via API, a locally run LLaMA-7B, and the Portuguese commercial service Maritaca. Each model sees ten tagged example sentences in a Portuguese prompt and must tag the next sentence; sampling temperature is tuned per model on a small sample. GPT-3 is clearly best at about 90% accuracy, LLaMA trails by roughly 20 points, and Maritaca is weakest, partly because it invents labels outside the UD tagset. The authors conclude that LLMs can provide a first-pass UPOS annotation for under-resourced settings if a human checks the result.

How Mova uses it

Effectiveness in Mova

Background evidence for one item of the LLM-annotator failure catalogue. Not measured separately.

👨‍🔬💥

Topics: Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{machado2024evaluating,
  title = {{Evaluating large language models for the tasks of PoS tagging within the Universal Dependency framework}},
  author = {Machado, Mateus and Ruiz, Evandro},
  booktitle = {Proceedings of the 16th International Conference on Computational Processing of Portuguese - Vol. 1},
  year = {2024},
  url = {https://aclanthology.org/2024.propor-1.46/}
}

References

27 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2022. Prompting language models for linguistic structure. arXiv preprint arXiv:2211.07830. find
  2. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109. find
  3. Alebachew Chiche and Betselot Yitagesu. 2022. Part of speech tagging: a systematic review of deep learning and machine learning approaches. Journal of Big Data, 9(1):1–25. find
  4. Marie-Catherine De Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308. find our card
  5. Magali Duran, Lucelene Lopes, Maria das Graças Nunes, and Thiago Pardo. 2023. The Dawn of the Porttinari Multigenre Treebank: Introducing its Journalistic Portion. In Anais do XIV Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana, pages 115–124, Porto Alegre, RS, Brasil. SBC. find
  6. Lane Green. 2017. Technology Quarterly: Finding a Voice. The Economist. find
  7. Go Inoue, Hiroyuki Shindo, and Yuji Matsumoto. 2017. Joint prediction of morphosyntactic categories for fine-grained Arabic part-of-speech tagging exploiting tag dictionary information. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 421–431. find our card
  8. Taja Kuzman, Igor Mozetič, and Nikola Ljubešić. 2023. ChatGpt: Beginning of an end of manual linguistic data annotation? Use case of automatic genre identification. ArXiv, abs/2303.03953. find
  9. Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning. arXiv preprint arXiv:2304.05613. find
  10. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Mi… find
  11. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic Knowledge and Transferability of Contextual Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… find
  12. Lucelene Lopes, Magali Duran, and Thiago Alexandre Pardo. 2023. Verifica-UD: a Verifier for Universal Dependencies Annotation for Portuguese. In Anais do XIV Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana, pages 443–452, Porto Alegre, RS, Brasil. SBC. find
  13. Louis Martin, Benjamin Muller, Pedro Ortiz Suarez, Yoann Dupont, Laurent Romary, Éric Villemonte De La Clergerie, Djamé Seddah, and Benoît Sagot. 2020. Camembert: a tasty french language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203– 72… find
  14. Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56(2):1–40. find
  15. OpenAI. 2020. GPT-3: Language Models for Few-Shot Learning. OpenAI. find
  16. Jianquan Ouyang and Mengen Fu. 2022. Improving machine reading comprehension with multi-task learning and self-training. Mathematics, 10(3):310. find
  17. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True Few-Shot Learning with Language Models. In Advances in Neural Information Processing Systems, volume 34, pages 11054–11070. Curran Associates, Inc. find
  18. Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira. 2023. Sabiá: Portuguese large language models. In Intelligent Systems, pages 226–240, Cham. Springer Nature Switzerland. find
  19. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476. find
  20. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  21. Pradeep Kumar Roy, Sunil Saumya, Jyoti Prakash Singh, Snehasish Banerjee, and Adnan Gutub. 2023. Analysis of community question-answering issues via machine learning and deep learning: State-of-the-art review. CAAI Transactions on Intelligence Technology, 8(1):95–117. find
  22. Devansh Shah, Arun Singh, and Sudha Shanker Prasad. 2022. find
  23. Sentimental Analysis Using Supervised Learning Algorithms. In 2022 3rd International Conference on Computation, Automation and Knowledge Management (ICCAKM), pages 1–6. find
  24. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics. find
  25. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMa: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. find
  26. Xiangpeng Wei, Heng Yu, Yue Hu, Rongxiang Weng, Weihua Luo, Jun Xie, and Rong Jin. 2022. Learning to generalize to more: Continuous semantic augmentation for neural machine translation. arXiv preprint arXiv:2204.06812. find
  27. Leonardo Zilio, Rodrigo Wilkens, and Cédrick Fairon. 2018. PassPort: a dependency parsing model for portuguese. In International Conference on Computational Processing of the Portuguese Language, pages 479–489. Springer. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/machado-2024-evaluating-large-language-models-tasks/