Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning
Meysam Alizadeh University of Zurich Zurich, Switzerland、Maël Kubli University of Zurich Zurich, Switzerland、Zeynab Samei Institute for Fundamental Research Tehran, Iran、Shirin Dehghani Allameh Tabataba’i University Tehran, Iran、Mohammadmasiha Zahedivafa Iran University of Science and Technology Tehran, Iran、Juan D. Bermeo University of Zurich Zurich, Switzerland、Maria Korobeynikova University of Zurich Zurich, Switzerland、Fabrizio Gilardi University of Zurich Zurich, Switzerland Corresponding author
Abstract
This paper studies the performance of open-source Large Language Models (LLMs) in text classification tasks typical for political science research. By examining tasks like stance, topic, and relevance classification, we aim to guide scholars in making informed decisions about their use of LLMs for text analysis. Specifically, we conduct an assessment of both zero-shot and fine-tuned LLMs across a range of text annotation tasks using news articles and tweets datasets. Our analysis shows that fine-tuning improves the performance of open-source LLMs, allowing them to match or even surpass zero-shot GPT-3.5 and GPT-4, though still lagging behind fine-tuned GPT-3.5. We further establish that fine-tuning is preferable to few-shot training with a relatively modest quantity of annotated text. Our findings show that fine-tuned open-source LLMs can be effectively deployed in a broad spectrum of text annotation applications. We provide a Python notebook facilitating the application of LLMs in text annotation for other researchers.
中文速览
政治学研究里大量依赖人工标注文本，既费钱又费时，研究者越来越想用大型语言模型（LLM）来替代，但到底该选哪种模型、要不要微调（fine-tuning）、开源模型够不够用，一直缺乏系统性的实证比较。这项研究用涵盖立场识别、话题检测、相关性判断等多个政治学典型任务的推文和新闻数据集，系统对比了GPT-3.5、GPT-4与LLaMA、FLAN等开源模型在零样本、少样本和微调三种设置下的表现。结果发现，经过微调的开源模型可以追平甚至超越零样本的GPT-3.5和GPT-4，而且相比少样本学习，用相对少量的标注数据进行微调就能带来更显著的性能提升，尽管与微调后的GPT-3.5相比仍有一定差距。这一发现对学术界意义重大——研究者无需依赖昂贵的商业闭源模型，借助开源LLM加上适量标注数据即可完成高质量的文本标注，同时还能兼顾数据安全、透明度和研究可重复性。
原文 arXiv:2307.02179;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2307.02179v2