Publishing Identifiable Experiment Code And Configuration Is Important, Good and Easy
Richard T. Vaughan Jens Wawerla Simon Fraser University
Abstract
A few months ago, a graduate student in another country called me (Vaughan) to ask for the source code of one of my multi-robot simulation experiments. The student had an idea for a modification that she thought would improve the system’s performance. By the standards of scientific practice this was a perfectly reasonable request and I felt obliged to give it to her. With our original code, the student could (i) re-run our experiments to verify that we reported the results correctly; (ii) inspect the code to make sure that it actually implements the algorithm described in our paper; (iii) change parameters and initial conditions to make sure our results were not a fluke of the particular experimental setting; (iv) modify the robot controllers and quantitatively compare her new method with our originals. It would cost me nothing to make her a copy of our code, and her methodology would be impeccable. Why then do we read so few papers using this methodology?
中文速览
机器人学领域长期存在一个尴尬现实：论文声称的实验结果往往无法被独立核实，因为作者几乎从不公开支撑实验的源代码。作者以自身经历为切入点，提出一套简单可行的解决方案——将完整的实验代码打包，用SHA1加密哈希值标记版本，连同下载链接一并发表在论文中，读者下载后可自行验证代码与论文描述完全一致。这一做法不仅能让他人复现、质疑和改进实验，还能倒逼研究者写出更规范、更少漏洞的代码，从而整体提升研究质量与效率。作者呼吁学界将代码发布纳入发表规范，并建议期刊、会议及资助机构通过激励与要求并举的方式推动这一文化转变，因为公开可复现的实验成果才是真正对社区和纳税人负责的科学贡献。
原文 arXiv:1204.2235;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1204.2235v1