CADS: Passive Snapshot Coded Aperture Dual-Pixel RGB-D Imaging

CVPR 2024

Bhargav Ghanekar1, Salman Siddique Khan1, Pranav Sharma2, Shreyas Singh2, Vivek Boominathan1, Kaushik Mitra2 Ashok Veeraraghavan1
Rice University, Houston TX USA1 IIT Madras, Chennai, India2
Paper arXiv Code Supp
Rice IITM

Coded Aperture Dual-Pixel Sensing (CADS) is a novel imaging principle that can allow passive snapshot RGB-D imaging. The novelty lies in combining an optimal (end-to-end learnt) coded aperture alongwith a dual-pixel sensor, enabling a better depth vs. deblurring trade-off.


Abstract

Passive, compact, single-shot 3D sensing is useful in many application areas such as microscopy, medical imaging, surgical navigation, and autonomous driving where form factor, time, and power constraints can exist. Obtaining RGB-D scene information over a short imaging distance, in an ultra-compact form factor, and in a passive, snapshot manner is challenging. Dual-pixel (DP) sensors are a potential solution to achieve the same. DP sensors collect light rays from two different halves of the lens in two interleaved pixel arrays, thus capturing two slightly different views of the scene, like a stereo camera system. However, imaging with a DP sensor implies that the defocus blur size is directly proportional to the disparity seen between the views. This creates a trade-off between disparity estimation vs. deblurring accuracy. To improve this trade-off effect, we propose CADS (Coded Aperture Dual-Pixel Sensing), in which we use a coded aperture in the imaging lens along with a DP sensor. In our approach, we jointly learn an optimal coded pattern and the reconstruction algorithm in an end-to-end optimization setting. Our resulting CADS imaging system demonstrates improvement of >1.5dB PSNR in all-in-focus (AIF) estimates and 5-6% in depth estimation quality over naive DP sensing for a wide range of aperture settings. Furthermore, we build the proposed CADS prototypes for DSLR photography settings and in an endoscope and a dermoscope form factor. Our novel coded dual-pixel sensing approach demonstrates accurate RGB-D reconstruction results in simulations and real-world experiments in a passive, snapshot, and compact manner.

Video

Principle

The CADS pipeline consists of a coded aperture in the imaging lens along with a dual-pixel sensor.

Principle

Pipeline

We formulate a differentiable image formation model to render out images captured by a coded aperture dual-pixel sensor system. We then jointly learn an optimal coded aperture pattern and the reconstruction algorithm in an end-to-end optimization setting.

Pipeline

The coded aperture is optimized to improve the depth vs. deblurring trade-off for dual-pixel image captures. The CADS system is compact and passive, and can be used in a wide range of applications.

Results

Naive DP vs CADS (AIF predictions)

Naive DP CADS
Naive StdPix Naive DP CADS

Naive DP vs CADS (Depth predictions)

Naive StdPix Naive DP CADS
Naive StdPix Naive DP CADS

Results from prototype CADS Endoscope and CADS Dermoscope

CADS Endoscope
CADS Dermoscope
CADS Left CADS Right
AIF Pred Depth Pred
CADS Left CADS Right
AIF Pred Depth Pred
CADS Left CADS Right
AIF Pred Depth Pred
CADS Left CADS Right
AIF Pred Depth Pred

BibTeX

        
          @inproceedings{ghanekar2024passive,
            title={Passive Snapshot Coded Aperture Dual-Pixel RGB-D Imaging},
            author={Ghanekar, Bhargav and Khan, Salman Siddique and Sharma, Pranav and Singh, Shreyas and Boominathan, Vivek and Mitra, Kaushik and Veeraraghavan, Ashok},
            booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
            pages={25348--25357},
            year={2024}
          }

Acknowledgement

This work was supported by NSF Award IIS-1730574, NIH DeepDOF-RO1 Award R01DE032051-01. Kaushik Mitra acknowledges NSF, IITM Pravartak Technologies Foundation and funding from the Department of Science and Technology, India.