Latent Radiance Fields with 3D-aware 2D Representations

TL;DR: Create 3D latent representation and render high-quality images.

Method overview

A diagram explaining the method in broad strokes, like explained in the caption.

In this work, we propose a method to achieve 3D-aware 2D representations and enable 3D reconstruction in the latent space. To enhance the 3D awareness of both encoder and decoder of the VAE, we present a three-stage pipeline. The first stage focuses on improving the 3D awaresness of the VAE's encoder through a novel correspondence-aware constraint on the latent space, making the 2D representations follow the geometry consistency; The second stage builds a latent radiance field (LRF) to represent 3D scenes from the 3D-aware 2D representations; The third stage further introduces a VAE-Radiance Field (VAE-RF) alignment method to boost the reconstruction performance. In together, our LRF enables 3D reconstruction on the 2D latent space instead of the image space. It can render high-quality and photorealistic novel views, even for the unbounded scenes.

Comparisons to other methods

Compare the renders our method LRF (right) with baseline methods (left). Try selecting different methods and scenes!

Baseline

Ours

BibTeX

@inproceedings{zhou2025lrf,
  title={Latent Radiance Fields with 3D-Aware 2D Representations},
  author={Zhou, Chaoyi and Liu, Xi and Luo, Feng and Huang, Siyu},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2025}
}