Automated Unit Test Improvement using Large Language Models at Meta
Nadia Alshahwan https://orcid.org/0009-0009-4763-0396 , Jubin Chheda https://orcid.org/0009-0005-0311-7890 , Anastasia Finegenova https://orcid.org/0009-0005-5824-5179 , Beliz Gokkaya https://orcid.org/0009-0003-0197-6806 , Mark Harman https://orcid.org/0000-0002-5864-4488 , Inna Harper https://orcid.org/0009-0008-9359-0949 , Alexandru Marginean https://orcid.org/0009-0001-5311-762X , Shubho Sengupta https://orcid.org/0009-0007-4204-5185 and Eddy Wang https://orcid.org/0009-0009-8825-6986 Meta Platforms Inc.,1 Hacker WayMenlo ParkCaliforniaUSA
Abstract
This paper describes Meta’s TestGen-LLM tool, which uses LLMs to automatically improve existing human-written tests. TestGen-LLM verifies that its generated test classes successfully clear a set of filters that assure measurable improvement over the original test suite, thereby eliminating problems due to LLM hallucination. We describe the deployment of TestGen-LLM at Meta test-a-thons for the Instagram and Facebook platforms. In an evaluation on Reels and Stories products for Instagram, 75% of TestGen-LLM’s test cases built correctly, 57% passed reliably, and 25% increased coverage. During Meta’s Instagram and Facebook test-a-thons, it improved 11.5% of all classes to which it was applied, with 73% of its recommendations being accepted for production deployment by Meta software engineers. We believe this is the first report on industrial scale deployment of LLM-generated code backed by such assurances of code improvement.
中文速览
Meta开发的TestGen-LLM工具专门解决大规模工业软件中测试用例不够全面的问题，它利用大型语言模型（LLM）自动扩充工程师已有的Kotlin测试类，补充被遗漏的边界场景。为了防止LLM"幻觉"生成无效或错误的代码，系统设计了三道过滤器：依次检查生成的测试能否正常编译、能否稳定通过（排除"脆弱测试"flaky test）、以及是否真正提升了代码覆盖率，只有三关全过的测试才会被推荐给工程师。在Instagram的Reels和Stories产品评估中，75%的生成测试可成功编译，57%能稳定通过，25%实现了覆盖率提升；在Instagram和Facebook的"测试马拉松"（test-a-thon）活动中，11.5%的被处理测试类得到改进，73%的改进建议被工程师接受并合并进生产代码库。这项工作的意义在于，它首次证明了LLM生成的代码可以在有可验证质量保障的前提下大规模落地工业生产系统，为将AI辅助编程真正用于严肃软件工程提供了可信赖的范式。
原文 arXiv:2402.09171;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2402.09171v1