Supporting the Design and Authoring of Pervasive Smart Environments

The accelerated development of mobile computational platforms and artificial intelligence (AI) has led to increase in interconnected products with sensors that are creating smart environments. The smart environment expands the interactive spaces from limited digital screens, such as desktops and phones, to a much broader category that includes everyday objects, smart things, surrounding contexts, robots, and humans. The improvement of personal computing devices including smartphones, watches, and AR glasses further broadens the communication bandwidth between us and the ambient intelligence from the surrounding environment. Additionally in this smart environment people want to pursue personalization and are motivated to design and build their own smart environments and author customized functions.

My work in this thesis focuses on investigating workflows and methods to support end-users to create personalized interactive experiences within smart environments. In particular, I designed the authoring systems by inspecting different interaction modalities, namely the direct input, spatial movement, in-situ activity and embodied interactions between users and everyday objects, smart things, robots and virtual mid-air contents. To this end, we explored 1) the software tools, hardware modules, and machines that support users to augment non-smart environments with digital interfaces and functions, and 2) the intelligence and context-awareness powered by the smart environments that deliver automatic and assistance during living and entertaining experiences. In this thesis, I mainly studied the following authoring workflows and systems: 1) customizing interactive interfaces on ordinary objects by surface painted circuits, 2) constructing a spatially aware environment for service robots with IoT modules, 3) authoring robot and IoT applications that can be driven by body actions and daily activities and 4) creating interactive and responsive augmented reality applications and games that can be played through natural input modalities.

Takeaways from the main body of the research indicate that the authoring systems greatly lower the barrier for end-users to understand, control, and modify the smart environments. We conclude that seamless, fluent, and intuitive authoring experiences are crucial for building harmonious human-AI symbiotic living environments.