LEVERAGING MULTIMODAL SENSING FOR ENHANCING THE SECURITY AND PRIVACY OF MOBILE SYSTEMS

Mobile systems, such as smartphones, wearables (e.g., smartwatches, AR/VR headsets),
and IoT devices, have come a long way from being just a method of communication to
sophisticated sensing devices that monitor and control several aspects of our lives. These
devices have enabled several useful applications in a wide range of domains ranging from
healthcare and finance to energy and agriculture industries. While such advancement has
enabled applications in several aspects of human life, it has also made these devices an
interesting target for adversaries.
In this dissertation, I specifically focus on how the various sensors on mobile devices can
be exploited by adversaries to violate users’ privacy and present methods to use sensors
to improve the security of these devices. My thesis posits that multi-modal sensing can be
leveraged to enhance the security and privacy of mobile systems.
In this, first, I describe my work that demonstrates that human interaction with mobile de-
vices and their accessories (e.g., stylus pencils) generates identifiable patterns in permissionless
mobile sensors’ data, which reveal sensitive information about users. Specifically, I developed
S3 to show how embedded magnets in stylus pencils impact the mobile magnetometer sensor
and can be exploited to infer a users incredibly private handwriting. Then, I designed LocIn
to infer a users indoor semantic location from 3D spatial data collected by mixed reality
devices through LiDAR and depth sensors. These works highlight new privacy issues due to
advanced sensors on emerging commodity devices.
Second, I present my work that characterizes the threats against smartphone authentication
and IoT device pairing and proposes usable and secure methods to protect against these threats.
I developed two systems, FaceRevelio and IoTCupid, to enable reliable and secure user and
device authentication, respectively, to protect users’ private information (e.g., contacts,
messages, credit card details) on commodity mobile and allow secure communication between
IoT devices. These works enable usable authentication on diverse mobile and IoT devices
and eliminate the dependency on sophisticated hardware for user-friendly authentication.