Bayes' Rays: Uncertainty Quantification for Neural Radiance Fields (original) (raw)

NeRFs rely on training images, taken as 2D projections of a 3D scene, to reconstruct the scene geometry and appearance. Inherently, the projection causes loss of information and leads to ambiguity in the reconstruction problem, making multiple solutions valid for the reconstruction. This is even more prominent in real scenarios where limited training views of the scene is available. Below is a toy 2D example of a NeRF, where single ray training cameras are observing the scene and the ray color is calculated through volume rendering. All camera rays either start from the top side and end at the bottom or vice versa. Training multiple NeRFs on this training data, leads to different reconstructions, all valid and minimizing the training loss. Analyzing the geometry of this problem, we can see there exist a null space for each of the blue and red segments. One can perturb the learned lines, inside this space freely without hurting the reconstruction problem.