Fine-Tuned ‘Small’ LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification
Martin Juan José Bucher Stanford University Marco Martini University of Zurich
Abstract
Generative AI offers a simple, prompt-based alternative to fine-tuning smaller BERT-style LLMs for text classification tasks. This promises to eliminate the need for manually labeled training data and task-specific model training. However, it remains an open question whether tools like ChatGPT can deliver on this promise. In this paper, we show that smaller, fine-tuned LLMs (still) consistently and significantly outperform larger, zero-shot prompted models in text classification. We compare three major generative AI models (ChatGPT with GPT-3.5/GPT-4 and Claude Opus) with several fine-tuned LLMs across a diverse set of classification tasks (sentiment, approval/disapproval, emotions, party positions) and text categories (news, tweets, speeches). We find that fine-tuning with application-specific training data achieves superior performance in all cases. To make this approach more accessible to a broader audience, we provide an easy-to-use toolkit alongside this paper. Our toolkit, accompanied by non-technical step-by-step guidance, enables users to select and fine-tune BERT-like LLMs for any classification task with minimal technical and computational effort.
中文速览
用ChatGPT这类生成式AI直接"提问得答案"来做文本分类，听起来省时省力，但它真的能替代专门训练的小模型吗？研究者用情感分析、立场识别、情绪检测、政党位置判断等四类任务，横跨新闻、推文、政治演讲等多种文本，系统比较了ChatGPT（GPT-3.5/GPT-4）和Claude Opus的零样本表现，以及经过标注数据微调（fine-tuning）的BERT、RoBERTa等小型模型的表现。结果发现，在所有测试场景中，经过微调的小模型表现始终显著优于体量更大的生成式AI，差距在任务越专业、越偏门时越明显。这说明模型越大并不等于越好用，专属训练数据仍是文本分类的核心竞争力；为此，研究者还同步发布了一套基于Jupyter Notebook的易用工具包，让没有深度学习背景的用户也能低门槛地完成模型微调，对社会科学等领域需要大规模文本分析的研究者极具实用价值。
原文 arXiv:2406.08660;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2406.08660v2