IMMERSIVE TELECONFERENCING EXPERIENCE VIA VIDEO SEGMENTATION AND NOVEL VIEW SYNTHESIS

As international collaboration deepens and the demand for seamless virtual interactions grows, there is an escalating need to enhance the quality and realism of teleconferencing systems. Central to this effort is the pursuit of immersive experiences that not only improve communication efficiency but also foster a sense of proximity among participants, thereby bridging the gap between physical and virtual interactions. This dissertation addresses the challenge of enhancing immersion in video conferencing through innovative algorithmic advancements in two key areas: video segmentation and novel view synthesis. The research begins by focusing on the development of advanced segmentation algorithms for RGB-D video streams, which enable the creation of videos with naturally blurred backgrounds. This is achieved through a rapid bokeh effect rendering method that significantly improves visual quality and contributes to a more lifelike experience. Furthermore, an end-to-end segmentation algorithm incorporating background fusion has been developed to address critical issues such as boundary segmentation accuracy and temporal consistency, thereby ensuring smoother transitions and more realistic renderings. A significant contribution of this work lies in the unification of human segmentation algorithms for both ego-view and webcam-view scenarios. This breakthrough allows a single algorithm to be effectively deployed across traditional teleconferencing devices and virtual reality (VR) headsets, thus simplifying implementation and enhancing cross-platform compatibility. The dissertation also presents the design of a novel teleconferencing system that leverages multiple cameras to integrate traditional video conferencing with immersive VR experiences. A frame interpolation algorithm has been developed as part of this system to enable high-quality novel view synthesis, further advancing the state-of-the-art in virtual collaboration. Through these contributions, this research successfully elevates the immersiveness of teleconferencing systems, paving the way for more engaging virtual interactions. The proposed algorithms not only address existing limitations but also open up new possibilities for future advancements in video communication technology, which align with the evolving needs of an increasingly globalized world