NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

The problem it addresses

Producing an image of a scene from a viewpoint where no camera ever stood is a long-standing task in graphics and vision, usually called novel view synthesis. The usual route is to reconstruct geometry first — points, meshes, voxels — and then render it. The NeRF paper takes a different route: it reports state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function, using only a sparse set of input views^[1].

A scene as a continuous function

The representation is a fully-connected, non-convolutional deep network. Its input is a single continuous 5D coordinate, and its output is the volume density and the view-dependent emitted radiance at that location^[1].

Unpacking the coordinate: three of the five numbers are a spatial location, and the remaining two are a viewing direction. The network is therefore asked, for any point in space and any direction of looking, two questions — how much material sits at that point (density), and what light leaves that point towards that direction (radiance). Because the direction is part of the input, the colour the network returns may change with the angle from which the point is seen, so appearance that depends on viewpoint can be expressed within the same function.

Turning the function into an image

To make a picture, the method queries 5D coordinates along camera rays and uses classic volume rendering techniques to project the output colours and densities into an image^[1]. For each pixel, a ray is cast through the scene; the network is queried at points along that ray; each queried point contributes its colour, weighted by its density and by how much of the ray survived to reach it. Summing those contributions along the ray gives the pixel's colour. Rendering is thus a fixed, well-understood procedure — the learned part is only the function being queried.

How it is trained

Volume rendering is naturally differentiable, so the only input required to optimize the representation is a set of images with known camera poses^[1]. In practice this means: take a photograph and its camera pose, render the corresponding pixels through the network, compare the rendered colours with the actual photograph, and propagate the difference back through the rendering step into the network's weights. Nothing else is supplied — no depth maps, no segmentation, no explicit 3D model. The scene geometry and appearance are whatever the network settles on in order to reproduce the photographs from their recorded viewpoints.

Results

The paper demonstrates results that outperform prior work on neural rendering and view synthesis^[1]. It was written by Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi and Ren Ng, and submitted to arXiv on March 19, 2020^[1].

Why it matters

The notable move is what gets stored. In earlier pipelines a scene is a collection of explicit primitives — triangles, voxels, textures — and the work goes into building and updating them. Here the scene is a function, and the thing being optimized is the function's parameters. Two consequences follow from the design. First, the supervision signal is ordinary photographs plus camera poses, which is exactly what a photographer already produces. Second, because the input coordinate is continuous, the function can be queried anywhere, not only at the positions of a fixed grid.

What the design gives up

Two constraints are visible in the formulation itself. The method assumes the camera poses are known^[1], so it is not by itself a structure-from-motion system; the poses must come from somewhere. And rendering a single pixel requires querying the network at many points along its ray, so the cost of producing one image grows with the number of samples taken per ray. The paper's contribution is the framing — view synthesis posed as the optimization of a differentiable rendering pipeline whose only learnable component is a continuous scene function — and that framing is what the name neural radiance field describes.