Abstract

Neural radiance fields enable state-of-the-art photorealistic view synthesis. However, existing radiance field representations are either too compute-intensive for real-time rendering or require too much memory to scale to large scenes. We present a Memory-Efficient Radiance Field (MERF) representation that achieves real-time rendering of large-scale scenes in a browser. MERF reduces the memory consumption of prior sparse volumetric radiance fields using a combination of a sparse feature grid and high-resolution 2D feature planes. To support large-scale unbounded scenes, we introduce a novel contraction function that maps scene coordinates into a bounded volume while still allowing for efficient ray-box intersection. We design a lossless procedure for baking the parameterization used during training into a model that achieves real-time rendering while still preserving the photorealistic view synthesis quality of a volumetric radiance field.

Video

Real-Time Interactive Viewer Demos

Real Captured Scenes

Representation

For a location \(\mathbf{x}\) along a ray: (1) We query its eight neighbors on a low-resolution 3D grid; and we project it onto each of the three axis-aligned planes, and then query each projection’s four neighbors on a high-resolution 2D grid. (2) The eight low-resolution 3D neighbors are evaluated and trilinearly interpolated while the three sets of four high-resolution 2D neighbors are evaluated and bilinearly interpolated, and the resulting features are summed into a single feature vector \(\mathbf{t}\). (3) The feature vector is split and nonlinearly mapped into three components: density \(\tau\) , RGB color \(\mathbf{c}_d\), and a feature vector \(\mathbf{f}\) encoding view dependence effects.

Piecewise-projective contraction

To model unbounded scenes we employ a contraction function. Existing works use spherical contraction, which maps straight lines to curves (left). This makes computing intersections between rays and axis-aligned bounding boxes intractable, which is required for empty space skipping. We propose a novel contraction function (right) that maps a line to a small number of segments. Intersections can be computed efficiently and thereby our contraction function is more suitable for real-time rendering.

Acknowledgements

The website template was borrowed from Michaël Gharbi. Image sliders are based on dics.