ENHANCING PRIVACY AND IMMERSIVENESS IN VIDEO CONFERENCING: PORTRAIT VIDEO SEGMENTATION AND SPATIAL VIEW INTERPOLATION

In recent years, video conferencing technology has gained widespread use, driving the demand for enhanced virtual communication experiences. However, survey shows that users are highly concerned about information leaks from their video backgrounds and often re- port difficulties maintaining attention during long meetings. This dissertation focuses on improving privacy and immersion in video conferencing. To improve segmentation quality and inter-frame segmentation consistency, sources of temporal guidance are benchmarked. Furthermore, we develop a lightweight deep neural network with efficient temporal guid- ance for real-time portrait video segmentation. The method achieves good balance between processing time and segmentation quality, making it ideal for real-time applications such as background blurring and replacement. To promote immersiveness in video conferencing, we propose a cost-effective telepresence system that delivers more immersive viewing ex- periences. The system integrates multi-view capture, spatial view interpolation, and view rendering. Leveraging a fully synthetic multi-view portrait dataset, the quality of the spatial view interpolation method is significantly improved. Additionally, we introduce an effective multi-stage network which significantly reduces the computation cost in generating multiple interpolated views at finer scales without sacrificing image quality. Furthermore, a simplified system with only two camera inputs is explored, which utilizes pose information to assist spatial view interpolation. The proposed telepresence system offers an immersive multi-angle viewing experience.