TOWARDS THE AUTHORING OF MIXED REALITY EXPERIENCES BASED ON HUMAN-AI INTERACTION

<pre>The current research landscape on Mixed Reality (MR) is significantly empowered by the rapid development in Artificial Intelligence (AI). AI has proven its capability to capture, understand, and generate spatial-temporal information in various MR contexts and has become a powerful tool to author MR applications.<br>On the other hand, often deployed in scenarios with rich spatial-temporal information and natural user interactions, MR applications heavily rely on intuitive and seamless interactions between humans and AI to enhance user engagement and immersion.<br>There remains a significant gap in the research on the design of user interfaces and interaction modalities to optimize AI behavior to enhance user experience and system performance.<br>This thesis aims to investigate the design space of human-AI interaction in general and provide solutions for designing human-AI interactions to enhance the authoring process of MR real-world applications.<br><br>To ground the proposed research on prior efforts in the domain, we first collect and analyze existing Human-Computer Interaction (HCI) literature on the interactions between humans and generative AI across various platforms beyond MR.<br>Based on our analysis, we derive a novel taxonomy of the design space of human-AI interaction to further locate the key considerations and research challenges for AI-empowered MR applications.<br><br>Inspired by our findings, we depict the topic of authoring MR in three aspects: interaction, which refers to how users physically or virtually act upon digital content — using their hands, gestures, or other inputs to control or explore MR environments; embodiment in MR, which means how users are visually and functionally represented in MR — often through avatars — allowing them to see themselves, express actions, and be perceived by others in the space; and reality in MR,<br>which encompasses the spatial backdrop of MR — including both the real physical world and digitally generated elements — which together form the canvas for immersive experience and content creation.<br><br>In this thesis, we cover the three aspects with three research projects, respectively.<br>First, we introduce Ubi-TOUCH, a system to author AR interactions with ubiquitous tangible objects, focusing on consistent hand-object interaction.<br>In this project, we discuss how users' interaction with the environment can be remapped and rendered into real-time visual feedback.<br>Then, we introduce our research on authoring the embodiment of MR.<br>We develop an AR system called CARING-AI, which allows users to create humanoid avatar animation as the representation of their physical selves.<br>CARING-AI leverages the state-of-the-art generative AI algorithm and allows users to author their embodiment with merely natural language instructions and positional inputs.<br>Eventually, we discuss the authoring of the reality in MR. We present a dataset for image-to-3D editing tasks, with which we develop and evaluate an algorithm for 3D scene editing via generative AI.<br><br>The collective work outlined in this dissertation proposes the key HCI considerations of designing the AI-empowered MR authoring process.<br>Specifically, deriving from an extensive literature review, this work introduces the concept of three distinctive aspects of human-AI interaction in authoring MR experiences.<br>The proposed designs of human-AI interaction in MR are then demonstrated and evaluated through three real-world MR applications.<br>With implications for various application scenarios, the research underscores the significance of designing human-AI interaction patterns for MR experiences.<br><br><br></pre><p></p>