Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions
Ayaan Haque UC Berkeley Matthew Tancik UC Berkeley Alexei A. Efros UC Berkeley Aleksander Holynski UC Berkeley Angjoo Kanazawa UC Berkeley
Abstract
We propose a method for editing NeRF scenes with text-instructions. Given a NeRF of a scene and the collection of images used to reconstruct it, our method uses an image-conditioned diffusion model (InstructPix2Pix) to iteratively edit the input images while optimizing the underlying scene, resulting in an optimized 3D scene that respects the edit instruction. We demonstrate that our proposed method is able to edit large-scale, real-world scenes, and is able to accomplish more realistic, targeted edits than prior work. Result videos can be found on the project website: https://instruct-nerf2nerf.github.io.
中文速览
用文字指令直接编辑真实三维场景，一直是个难题——你没法像修图那样直接"刷新"一个NeRF。这项工作提出了Instruct-NeRF2NeRF方法：给定一个已重建好的NeRF场景和它对应的照片集，只需输入一句自然语言指令（比如"给他戴顶牛仔帽"），系统就能自动完成三维场景的编辑。核心思路是"迭代数据集更新"——交替用图像条件扩散模型InstructPix2Pix逐张修改数据集图片，再用修改后的图片重新训练NeRF，如此循环，让二维编辑结果逐步"渗透"并收敛成三维一致的新场景。实验表明，该方法在人物、物体和大规模室外场景上都能实现真实感强、细节可控的编辑效果，明显优于直接用SDS损失或CLIP引导的基线方案。这项工作首次将自然语言指令引入三维NeRF编辑，让没有专业3D软件技能的普通用户也能轻松、直观地修改真实捕获的三维场景。
原文 arXiv:2303.12789;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.12789v2