EXTREME LOW-LIGHT IMAGING OF DYNAMIC HDR SCENES USING DEEP LEARNING METHODS

Imaging in low light is difficult because few photons can arrive at the sensor in a particular time interval. Increasing the exposure time is not always an option, as images will be blurry if the scenes are dynamic. If scenes or objects are moving, one can capture multiple frames with short exposure time and fuse them using carefully designed algorithms; however, aligning the pixels in adjacent frames is challenging due to the high photon shot noise and sensor read noise at low light. If the dynamic range of the scene is high, one needs to further blend multiple exposures from the frames. This blending requires removal of spatially varying noise at various lighting conditions while todays high dynamic range (HDR) fusion algorithms usually assume well illuminated scenes. Therefore, this low-light HDR imaging problem remains unsolved.

To address these dynamic low-light imaging problems, researches in this dissertation explore both conventional CMOS image sensors and a new type of image sensor, named quanta image sensor (QIS), develop models of the imaging conditions of interest, and propose new image reconstruction algorithms based on deep neural networks together with new training protocols to assist the learning. Researches in this dissertation target to reconstruct dynamic HDR scenes at a light level of 1 photon per pixel (ppp) or less than 1 lux illuminance.