Abstract

We review the cost of training large-scale language models, and the drivers of these costs. The intended audience includes engineers and scientists budgeting their model-training experiments, as well as non-practitioners trying to make sense of the economics of modern-day Natural Language Processing (NLP).11 1 We thank Barak Lenz, Shai Shalev-Shwartz and other members of AI21 Labs, as well as Jack Clark, Jeff Dean, Deep Ganguli, Chris Re, Sebastian Ruder and Lior Wolf, who generously commented on previous drafts. Further comments on the document are welcome, and the document will be updated as appropriate. Note: While the comments of our colleagues from other organizations greatly improved the document, they were not representing their organizations, did not share any proprietary information, and may not necessarily agree with everything written here.