Self-Calibrating, Fully Differentiable NLOS Inverse Rendering
DescriptionExisting time-resolved non-line-of-sight (NLOS) imaging methods reconstruct hidden scenes by inverting the optical paths of indirect illumination measured at visible relay surfaces. These methods are prone to reconstruction artifacts due to inversion ambiguities and capture noise, which are typically mitigated through the manual selection of filtering functions and parameters. We introduce a fully-differentiable end-to-end NLOS inverse rendering pipeline that self-calibrates the imaging parameters during the reconstruction of hidden scenes, using as input only the measured illumination while working both in the time and frequency domains. Our pipeline extracts a geometric representation of the hidden scene from NLOS volumetric intensities and estimates the time-resolved illumination at the relay wall produced by such geometric information using differentiable transient rendering. We then use gradient descent to optimize imaging parameters by minimizing the error between our simulated time-resolved illumination and the measured illumination. To make our pipeline efficient and differentiable, we combine diffraction-based imaging with path-space light transport and a simple ray marching technique for surface extraction. Unlike the majority of previous works, our method extracts detailed, dense sets of surface points and normals of the hidden scene. Our results demonstrate the robustness of our method to consistently reconstruct geometry and albedo, even with significant noise interference.
Authors
Event Type
Technical Papers
TimeTuesday, 12 December 20239:30am - 12:45pm
LocationDarling Harbour Theatre, Level 2 (Convention Centre)
