Learning to Enhance Aperture Phasor Field for Non-Line-of-Sight Imaging

ECCV 2024

In Cho, Hyunbo Shim, Seon Joo Kim
Yonsei University
Paper arXiv Code

Abstract

This paper aims to facilitate more practical NLOS imaging by reducing the number of samplings and scan areas. To this end, we introduce a phasor-based enhancement network that is capable of predicting clean and full measurements from noisy partial observations. We leverage a denoising autoencoder scheme to acquire rich and noise-robust representations in the measurement space. Through this pipeline, our enhancement network is trained to accurately reconstruct complete measurements from their corrupted and partial counterparts. However, we observe that the \naive application of denoising often yields degraded and over-smoothed results, caused by unnecessary and spurious frequency signals present in measurements. To address this issue, we introduce a phasor-based pipeline designed to limit the spectrum of our network to the frequency range of interests, where the majority of informative signals are detected. The phasor wavefronts at the aperture, which are band-limited signals, are employed as inputs and outputs of the network, guiding our network to learn from the frequency range of interests and discard unnecessary information. The experimental results in more practical acquisition scenarios demonstrate that we can look around the corners with 16x or 64x fewer samplings and 4x smaller apertures.

Overview

(a) A typical Non-line-of-sight (NLOS) imaging system. NLOS imaging aims to reconstruct scenes that are hidden in direct line-of-sight systems, with a laser illuminating a relay wall, and a time-resolved detector recording the returning photons.

(b) While previous methods require sufficient sampling points, acquisition time, and scanning areas for high-quality results, we extend our focus to more practical acquisition scenarios relevant to real-world applications. These practical acquisition scenarios of NLOS imaging are sparse samplings and scanning with smaller apertures.

(c) Results on confocal 16 × 16 measurements of Bike. Our method exhibits high-quality results with 16× fewer sampling points and a shorter acquisition time, whereas previous signal recovery network (SSN) and simple addition of the denoising criterion to SSN (SSN+) fail to correctly reconstruct the hidden objects.

Method

We propose the phasor-based neural network, coined as Learning to Enhance Aperture Phasor field (LEAP), which can predict clean and full measurements from noisy partial observations. We begin by sampling partial inputs from full measurements H and corrupting them with Poisson noise. Then the enhancement network takes these noisy partial inputs and predicts the optimal phasor field at the aperture, containing full scans and clean signals, in the frequency domain. We train our network by minimizing the L1 distance between the predicted and the optimal phasor field at the aperture. After training, hidden scenes are reconstructed by propagating the predicted phasor field using the RSD algorithm.

Results

Confocal results

As shown in the results, our method clearly outperforms all baselines in both sparse sampling and smaller aperture scenarios. Evaluation scenarios involve 16 × 16 and 8 × 8 sparse samplings with 2 m × 2 m apertures, and the 1 m × 1 m smaller aperture with 16 × 16 samplings.

Results with varying exposures

The denoising criterion becomes evidently effective as the exposure time is reduced, but it fails to reveal some details of the object. Our method consistently presents high-quality results, exhibiting the noise robustness of the proposed phasor-based scheme.

BibTeX

@article{cho2024leap,
  author    = {Cho, In and Shim, Hyunbo and Kim, Seon Joo},
  title     = {Learning to Enhance Aperture Phasor Field for Non-Line-of-Sight Imaging},
  journal   = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2024},
}