Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow
Wenqi Zhang1, Yongliang Shen1, Zeqi Tan1, Guiyang Hou1, Weiming Lu1, Yueting Zhuang1 1College of Computer Science and Technology, Zhejiang University {zhangwenqi, Project Page: https://github.com/zwq2018/Data-Copilot
Abstract
Industries such as finance, meteorology, and energy generate vast amounts of heterogeneous data daily. Efficiently managing, processing, and visualizing such data is labor-intensive and frequently necessitates specialized expertise. Leveraging large language models (LLMs) to develop an automated workflow presents a highly promising solution. However, LLMs are not adept at handling complex numerical computations and table manipulations, and they are further constrained by a limited length context. To bridge this, we propose Data-Copilot, a data analysis agent that autonomously performs data querying, processing, and visualization tailored to diverse human requests. The advancements are twofold: First, it is a code-centric agent that leverages code as an intermediary to process and visualize massive data based on human requests, achieving automated large-scale data analysis. Second, Data-Copilot involves a data exploration phase in advance, which autonomously explores how to design universal and error-free interfaces from data, reducing the error rate in real-time responses. Specifically, It imitates common requests from data sources, abstracts them into universal interfaces (code mo
中文速览
金融、气象、能源等行业每天产生海量异构数据，人工分析既费时又需要专业知识，而直接让大语言模型（LLM）读取这些数据又面临上下文长度有限、数值计算能力弱、无法生成图表等瓶颈。为此，作者提出了 Data-Copilot——一个以代码为核心的数据分析智能体，它先进入"自探索"阶段：自动模拟用户请求、将常见需求抽象成经过测试和优化的通用接口模块（代码函数库），再在实际响应用户时直接调用这些预验证的接口来完成数据查询、处理与可视化，从而大幅降低实时生成代码的错误率并减少推理开销。作者以中国股票、基金、财经新闻等大规模金融数据为基础进行了开源验证，定量实验表明该"探索-部署"策略在请求成功率、token 消耗和可解释性上均优于直接让 LLM 现场生成代码的方案。这项工作的价值在于为各类数据密集型行业提供了一套低错误、高效率的自动化数据分析范式，让普通用户无需编程专业知识也能便捷地从海量数据中获取洞见。
原文 arXiv:2306.07209;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2306.07209v8