Institution Logo

Depth Priors for Monocular Dynamic 3D Reconstruction

Master Thesis

Eberhard Karls Universität Tübingen
Faculty of Science · Department of Computer Science · Autonomous Vision Group

Jonas Klötzl
Supervisor: Stefano Esposito
Examiners: Andreas Geiger, Gerard Pons-Moll

Abstract


Recent progress in dynamic scene reconstruction has largely been driven by extending existing 3D representations, such as NeRF and, more recently, 3D Gaussian Splatting, into the dynamic domain. While these approaches have demonstrated impressive results, many current dynamic Gaussian Splatting methods rely solely on RGB supervision, leaving the inherently ill-posed nature of the task largely unresolved. Incorporating depth priors might therefore reduce depth ambiguities and improve reconstruction quality, particularly for explicit representations such as Gaussian Splatting. However, the practical benefits within existing monocular dynamic reconstruction pipelines remain largely unexplored. Leveraging the standardized Monocular Dynamic Gaussian Splatting evaluation framework, this thesis examines how depth priors can enhance state-of-the-art monocular dynamic 3D reconstruction methods through three major contributions: First, an extensive evaluation of modern monocular depth prediction models exposes the most suitable for reconstruction. Then, depth supervision is integrated into the reconstruction pipeline through a differentiable inverse-depth rendering mechanism and a specifically designed loss function for depth maps using an adaptive weighting scheme. Lastly, a depth guided initialization procedure is presented for the dynamic foreground area, enabling a more informed placement of Gaussian primitives at the start of training. Our experiments show that, for two out of the three investigated methods, depth-based supervision and initialization consistently enhances reconstruction quality, with improvements in PSNR of around 0.3 dB to nearly 1 dB. However, the effectiveness is highly dependent on the underlying motion model, dataset characteristics and depth prediction quality, indicating that depth integration requires careful and model-specific assessment considering both the prior quality and the model’s representation of motion.


Qualitative Results



Depth-Based Supervision


Nerfies


broom curls tail toby-sit

Click to select different scenes.

iPhone


apple backpack block creeper handwavy haru-sit mochi-high-five paper-windmill pillow spin sriracha-tree teddy

Click to select different scenes.


Depth-Based Gaussian Initialization


Nerfies


broom curls tail toby-sit

Click to select different scenes.

iPhone


apple backpack block creeper handwavy haru-sit mochi-high-five paper-windmill pillow spin sriracha-tree teddy

Click to select different scenes.