StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis
Jiatao Gu†, Lingjie Liu‡ , Peng Wang⋄, Christian Theobalt‡ †Facebook AI ‡Max Planck Institute for Informatics ⋄The University of Hong Kong corresponding author.
Abstract
We propose StyleNeRF, a 3D-aware generative model for photo-realistic high-resolution image synthesis with high multi-view consistency, which can be trained on unstructured 2D images. Existing approaches either cannot synthesize high-resolution images with fine details or yield noticeable 3D-inconsistent artifacts. In addition, many of them lack control over style attributes and explicit 3D camera poses. StyleNeRF integrates the neural radiance field (NeRF) into a style-based generator to tackle the aforementioned challenges, i.e., improving rendering efficiency and 3D consistency for high-resolution image generation. We perform volume rendering only to produce a low-resolution feature map and progressively apply upsampling in 2D to address the first issue. To mitigate the inconsistencies caused by 2D upsampling, we propose multiple designs, including a better upsampler and a new regularization loss. With these designs, StyleNeRF can synthesize high-resolution images at interactive rates while preserving 3D consistency at high quality. StyleNeRF also enables control of camera poses and different levels of styles, which can generalize to unseen views. It also supports challenging ta
中文速览
想让神经网络既能生成超高清、细节丰富的人脸或场景图片，又能在换个拍摄角度后保持三维一致性，一直是个两难困境。StyleNeRF 把风格生成网络（StyleGAN）和神经辐射场（NeRF）结合起来：先用 NeRF 在低分辨率下做体渲染、得到特征图，再通过专门设计的 2D 上采样模块逐步放大到高分辨率，同时引入"NeRF 路径正则化"损失和改进的上采样器来压制多视角不一致的伪影。实验表明，StyleNeRF 能以交互帧率合成 1024×1024 的逼真图像，同时在多视角一致性上显著优于 GIRAFFE 等现有方法，还支持相机视角控制、风格混合、语义编辑等丰富应用。这项工作的意义在于，它首次在单张无标注 2D 图像训练条件下，同时实现了高分辨率、高一致性和实时渲染，为可控三维感知生成模型打开了新的实用化窗口。
原文 arXiv:2110.08985;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.08985v1