Publications

2025

1. 

   Beyond the Phone: Exploring Phone-XR Integration through Multi-View Transitions for Real-World Applications

   Fengyuan Zhu, Xun Qian , Daniel Kalmar , Mahdi Tayarani , Eric J. Gonzalez , Mar Gonzalez-Franco , David Kim , and Ruofei Du

   In 2025 IEEE Conference Virtual Reality and 3D User Interfaces (VR) , Saint Malo, France, 2025

   Abs PDF Video

   Despite the growing prevalence of Extended Reality (XR) headsets, their integration with mobile phones remains limited. Existing approaches primarily replicate the phone’s interface in XR or use the phone solely as a 6DOF controller. This paper introduces a novel framework for seamless transitions among mirrored, magnified, and augmented views, dynamically adapts the interface with the content and state of mobile applications. To achieve this, we establish a design space through literature reviews and expert workshops, outline user journeys with common real-world applications, and develop a prototype system that automatically analyzes UI layouts to provide enhanced controls and spatial augmentation. We validate our prototype system with a user study to assess its adaptability to a broad spectrum of applications at runtime, reported its strengths and weaknesses, and suggest directions to advance the future adaption in Phone-XR integration.

2024

1. 

   PhoneInVR: An Evaluation of Spatial Anchoring and Interaction Techniques for Smartphone Usage in Virtual Reality

   Fengyuan Zhu, Mauricio Sousa , Ludwig Sidenmark , and Tovi Grossman

   In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , Honolulu, HI, USA, 2024

   Abs PDF Video

   When users wear a virtual reality (VR) headset, they lose access to their smartphone and accompanying apps. Past work has proposed smartphones as enhanced VR controllers, but little work has explored using existing smartphone apps and performing traditional smartphone interactions while in VR. In this paper, we consider three potential spatial anchorings for rendering smartphones in VR: On top of a tracked physical smartphone which the user holds (Phone-locked), on top of the users empty hand, as if holding a virtual smartphone (Hand-locked), or in a static position in front of the user (World-locked). We conducted a comparative study of target acquisition, swiping, and scrolling tasks across these anchorings using direct Touch or above-the-surface Pinch. Our findings indicate that physically holding a smartphone with Touch improves accuracy and speed for all tasks, and Pinch performed better with virtual smartphones. These findings provide a valuable foundation to enable smartphones in VR.

2023

1. 

   PinchLens: Applying Spatial Magnification and Adaptive Control-Display Gain for Precise Selection in Virtual Reality

   Fengyuan Zhu, Ludwig Sidenmark , Mauricio Sousa , and Tovi Grossman

   In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) , Sydney, New South Wales, Australia, 2023

   Abs PDF Video

   We present PinchLens, a new free-hand target selection technique for acquiring small and dense targets in Virtual Reality. Traditional pinch-based selection does not allow people to precisely manipulate small and dense objects effectively due to tracking and perceptual inaccuracies. Our approach combines spatial magnification, an adaptive control-display gain, and visual feedback to improve selection accuracy. When a user starts the pinching selection process, a magnifying bubble expands the scale of nearby targets, an adaptive control-to-display ratio is applied to the user‘s hand for precision, and a cursor is displayed at the estimated pinch point for enhanced visual feedback. We performed a user study to compare our technique to traditional pinch selection and several variations to isolate the impact of each of the technique’s features. The results showed that PinchLens significantly outperformed traditional pinch selection, reducing error rates from 18.9% to 1.9%. Furthermore, we found that magnification was the dominant feature to produce this improvement, while the adaptive control-display gain and visual cursor of pinch were also helpful in several conditions.

2022

1. 

   Touching The Droid: Understanding and Improving Touch Precision With Mobile Devices in Virtual Reality

   Fengyuan Zhu, Zhuoyue Lyu , Mauricio Sousa , and Tovi Grossman

   In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) , Singapore, Singapore, 2022

   Abs PDF Video

   Touch interaction with physical smartphones and tablets in Virtual Reality offers interesting opportunities for cross-device input. Unfortunately, any imprecision in the alignment of the visual representation of either the hand or device can impact the precision of touch and the realism of the experience. We first study a user‘s ability to rely solely on preoperative feedback to perform touch interaction in VR, where no rendering of the hand is provided. Results indicate that touch in VR is possible without a visual representation of the hand, but accuracy is influenced by how the device is held and the distance traveled to the target. We then introduce a dynamic calibration algorithm to minimize the offset between the physical hand and its virtual representation. In a second study, we show that this algorithm can increase touch accuracy by 43%, and minimize depth-based “screen penetration” or “floating touch” errors.

2020

1. 

   BISHARE: Exploring Bidirectional Interactions Between Smartphones and Head-Mounted Augmented Reality

   Fengyuan Zhu, and Tovi Grossman

   In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , Honolulu, HI, USA, 2020

   Abs PDF Video

   In pursuit of a future where HMD devices can be used in tandem with smartphones and other smart devices, we present BISHARE, a design space of cross-device interactions between smartphones and ARHMDs. Our design space is unique in that it is bidirectional in nature, as it examines how both the HMD can be used to enhance smartphone tasks, and how the smartphone can be used to enhance HMD tasks. We then present an interactive prototype that enables cross-device interactions across the proposed design space. A 12-participant user study demonstrates the promise of the design space and provides insights, observations, and guidance for the future.

2017

1. 

   Monocular Visual-Inertial State Estimation for Mobile Augmented Reality

   Peiliang Li , Tong Qin , Botao Hu , Fengyuan Zhu, and Shaojie Shen

   In 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) , 2017

   Abs PDF

   Mobile phones equipped with a monocular camera and an inertial measurement unit (IMU) are ideal platforms for augmented reality (AR) applications, but the lack of direct metric distance measurement and the existence of aggressive motions pose significant challenges on the localization of the AR device. In this work, we propose a tightly-coupled, optimization-based, monocular visual-inertial state estimation for robust camera localization in complex indoor and outdoor environments. Our approach does not require any artificial markers, and is able to recover the metric scale using the monocular camera setup. The whole system is capable of online initialization without relying on any assumptions about the environment. Our tightly-coupled formulation makes it naturally robust to aggressive motions. We develop a lightweight loop closure module that is tightly integrated with the state estimator to eliminate drift. The performance of our proposed method is demonstrated via comparison against state-of-the-art visual-inertial state estimators on public datasets and real-time AR applications on mobile devices. We release our implementation on mobile devices as open source software1.

2. 

   PhyShare: Sharing Physical Interaction in Virtual Reality

   Zhenyi He , Fengyuan Zhu, and Ken Perlin

   In Adjunct Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology , Québec City, QC, Canada, 2017

   Abs PDF Video

   We present PhyShare, a new haptic user interface based on actuated robots. Virtual reality has recently been gaining wide adoption, and an effective haptic feedback in these scenarios can strongly support user’s sensory in bridging virtual and physical world. Since participants do not directly observe these robotic proxies, we investigate the multiple mappings between physical robots and virtual proxies that can utilize the resources needed to provide a well rounded VR experience. PhyShare bots can act either as directly touchable objects or invisible carriers of physical objects, depending on different scenarios. They also support distributed collaboration, allowing remotely located VR collaborators to share the same physical feedback.

2016

1. 

   From Breakage to Icebreaker: Inspiration for Designing Technological Support for Human-Human Interaction

   Xiaojuan Ma , Ke Fang , and Fengyuan Zhu

   In Proceedings of the 2016 ACM Conference on Designing Interactive Systems , Brisbane, QLD, Australia, 2016

   Abs PDF

   This paper explores why and how accidental breakage of technologies can promote humans to interact and ultimately lead to positive behavioral, emotional, and relational change. Through a set of research activities, including meta-synthesis of daily anecdotes, design workshops, and a case study, we gain insights into what may hinder or trigger human-human communication, and propose the conceptual and actionable process of Breakage-to-Icebreaker (B2I) design. Instead of intentionally breaking a technology, B2I design embeds mechanisms into existing products and services, creating opportunities for users to interpersonally interact online and/or offline while enjoying the original features and functionalities. Finally, we envision a broader and extended use of B2I thinking in everyday design research and practices.