Channel Title: Future Interfaces Group Channel Description: The Future Interfaces Group (FIG) is an interdisciplinary research laboratory within the Human-Computer Interaction Institute at Carnegie Mellon University. We create new sensing and interface technologies that aim to make interactions between humans and computers more fluid, intuitive, and powerful. These efforts often lie in emerging use modalities, such as wearable computing, touch interfaces and gestural interaction. <video> Pantœnna (ACM UIST 2023 Talk) Description: Demo video: https://youtu.be/ya_KWEJTKsU Abstract: Methods for faithfully capturing a user's holistic pose have immediate uses in AR/VR, ranging from multimodal input to expressive avatars. Although body-tracking has received the most attention, the mouth is also of particular importance, given that it is the channel for both speech and facial expression. In this work, we describe a new RF-based approach for capturing mouth pose using an antenna integrated into the underside of a VR/AR headset. Our approach side-steps privacy issues inherent in camera-based methods, while simultaneously supporting silent facial expressions that audio-based methods cannot. Further, compared to bio-sensing methods such as EMG and EIT, our method requires no contact with the wearer's body and can be fully self-contained in the headset, offering a high degree of physical robustness and user practicality. We detail our implementation along with results from two user studies, which show a mean 3D error of 2.6 mm for 11 mouth keypoints across worn sessions without re-calibration. Citation: Daehwa Kim and Chris Harrison. 2023. Pantœnna: Mouth pose estimation for ar/vr headsets using low-profile antenna and impedance characteristic sensing. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23). Association for Computing Machinery, New York, NY, USA, Article 83, 1–12. https://doi.org/10.1145/3586183.3606805 </video> <video> Power-over-Skin: Full-Body Wearables Powered By Intra-Body RF Energy Description: Research Presentation Video: https://youtu.be/gi56CEKdBYY https://www.figlab.com/research/2024/poweroverskin https://andykong.org Powerful computing devices are now small enough to be easily worn on the body. However, batteries pose a major design and user experience obstacle, adding weight and volume, and generally requiring periodic device removal and recharging. In response, we developed Power-over-Skin, an approach using the human body itself to deliver power to many distributed, battery-free, worn devices. We demonstrate power delivery from on-body distances as far as from head-to-toe, with sufficient energy to power microcontrollers capable of sensing and wireless communication. We share results from a study campaign that informed our implementation, as well as experiments that validate our final system. We conclude with several demonstration devices, ranging from input controllers to longitudinal bio-sensors, which highlight the efficacy and potential of our approach. Kong, A., Kim, D. and Harrison, C. 2024. Power-over-Skin: Full-Body Wearables Powered by Intra-Body RF Energy. To appear in Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (October 13 –16, 2024). UIST '23. ACM, New York, NY. </video> <video> SmartPoser: Arm Pose Estimation with a Smartphone and Smartwatch Using UWB and IMU Data Description: The ability to track a user’s arm pose could be valuable in a wide range of applications, including fitness, rehabilitation, augmented reality input, life logging, and context-aware assistants. Unfortunately, this capability is not readily available to consumers. Systems either require cameras, which carry privacy issues, or utilize multiple worn IMUs or markers. In this work, we describe how an off-the-shelf smartphone and smartwatch can work together to accurately estimate arm pose. Moving beyond prior work, we take advantage of more recent ultra-wideband (UWB) functionality on these devices to capture absolute distance between the two devices. This measurement is the perfect complement to inertial data, which is relative and suffers from drift. We quantify the performance of our software-only approach using off-the-shelf devices, showing it can estimate the wrist and elbow joints without the user having to provide training data. Citation: Nathan DeVrio*, Vimal Mollyn*, and Chris Harrison. 2023. SmartPoser: Arm Pose Estimation with a Smartphone and Smartwatch Using UWB and IMU Data. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23). Association for Computing Machinery, New York, NY, USA, Article 79, 1–11. https://doi.org/10.1145/3586183.3606821 *Equal contribution </video> <video> Power-over-Skin: Full Body Wearables Powered By Intra-Body RF Energy (Presentation @ ACM UIST 2024) Description: Research video: https://youtu.be/5PEN04-jyCU Paper PDF: https://www.figlab.com/research/2024/poweroverskin More Info: https://andykong.org Powerful computing devices are now small enough to be easily worn on the body. However, batteries pose a major design and user experience obstacle, adding weight and volume, and generally requiring periodic device removal and recharging. In response, we developed Power-over-Skin, an approach using the human body itself to deliver power to many distributed, battery-free, worn devices. We demonstrate power delivery from on-body distances as far as from head-to-toe, with sufficient energy to power microcontrollers capable of sensing and wireless communication. We share results from a study campaign that informed our implementation, as well as experiments that validate our final system. We conclude with several demonstration devices, ranging from input controllers to longitudinal bio-sensors, which highlight the efficacy and potential of our approach. Kong, A., Kim, D. and Harrison, C. 2024. Power-over-Skin: Full-Body Wearables Powered by Intra-Body RF Energy. To appear in Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (October 13 –16, 2024). UIST '23. ACM, New York, NY. </video> <video> IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and Earbuds Description: Tracking body pose on-the-go could have powerful uses in fitness, mobile gaming, context-aware virtual assistants, and rehabilitation. However, users are unlikely to buy and wear special suits or sensor arrays to achieve this end. Instead, in this work, we explore the feasibility of estimating body pose using IMUs already in devices that many users own – namely smartphones, smartwatches, and earbuds. This approach has several challenges, including noisy data from low-cost commodity IMUs, and the fact that the number of instrumentation points on a user's body is both sparse and in flux. Our pipeline receives whatever subset of IMU data is available, potentially from just a single device, and produces a best-guess pose. To evaluate our model, we created the IMUPoser Dataset, collected from 10 participants wearing or holding off-the-shelf consumer devices and across a variety of activity contexts. We provide a comprehensive evaluation of our system, benchmarking it on both our own and existing IMU datasets. Citation: Mollyn, V., Arakawa, R., Goel, M., Harrison, C. and Ahuja, K. 2023. IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and Earbuds. To appear in Proceedings of the 41st Annual SIGCHI Conference on Human Factors in Computing Systems (April 23 – 30, 2023). CHI '23. ACM, New York, NY. </video> <video> Flat Panel Haptics: Embedded Electroosmotic Pumps for Scalable Shape Displays Description: We present a new, miniaturizable type of shape-changing display using embedded electroosmotic pumps (EEOPs). Our pumps, controlled and powered directly by applied voltage, are 1.5mm in thickness, and allow complete stackups under 5mm. Nonetheless, they can move their entire volume's worth of fluid in 1 second, and generate pressures of +/-50kPa, enough to create dynamic, millimeter-scale tactile features on a surface that can withstand typical interaction forces. These are the requisite technical ingredients to enable, for example, a pop-up keyboard on a flat smartphone. We experimentally quantify the mechanical and psychophysical performance of our displays and conclude with a set of example interfaces. Citation: Shultz, Craig and Harrison, Chris. 2023. Flat Panel Haptics: Embedded Electroosmotic Pumps for Scalable Shape Displays. To appear in Proceedings of the 41st Annual SIGCHI Conference on Human Factors in Computing Systems (April 23 – 30, 2023). CHI '23. ACM, New York, NY. </video> <video> SmartPoser (ACM UIST 2023 Talk) Description: Demo Video: https://youtu.be/AHh2vYQVb_8 Abstract: The ability to track a user’s arm pose could be valuable in a wide range of applications, including fitness, rehabilitation, augmented reality input, life logging, and context-aware assistants. Unfortunately, this capability is not readily available to consumers. Systems either require cameras, which carry privacy issues, or utilize multiple worn IMUs or markers. In this work, we describe how an off-the-shelf smartphone and smartwatch can work together to accurately estimate arm pose. Moving beyond prior work, we take advantage of more recent ultra-wideband (UWB) functionality on these devices to capture absolute distance between the two devices. This measurement is the perfect complement to inertial data, which is relative and suffers from drift. We quantify the performance of our software-only approach using off-the-shelf devices, showing it can estimate the wrist and elbow joints without the user having to provide training data. Citation: Nathan DeVrio*, Vimal Mollyn*, and Chris Harrison. 2023. SmartPoser: Arm Pose Estimation with a Smartphone and Smartwatch Using UWB and IMU Data. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23). Association for Computing Machinery, New York, NY, USA, Article 79, 1–11. https://doi.org/10.1145/3586183.3606821 *Equal contribution </video> <video> TeslaTouch: Electrovibration for Touch Surfaces (ACM UIST 2010) Description: Bau, O., Poupyrev, I., Israr, A., and Harrison, C. 2010. TeslaTouch: Electrovibration for Touch Surfaces. In Proceedings of the 23rd Annual ACM Symposium on User interface Software and Technology (New York, New York, October 3 - 6, 2010). UIST '10. ACM, New York, NY. 283-292. TeslaTouch infuses finger-driven interfaces with physical feedback. The technology is based on the electrovibration principle, which can programmatically vary the electrostatic friction between fingers and a touch panel. Importantly, there are no moving parts, unlike most tactile feedback technologies, which typically employ mechanical actuators. This allows for different fingers to feel different sensations. When combined with an interactive graphical display, TeslaTouch enables the design of a wide variety of interfaces that allow the user to feel virtual elements through touch. For example, when dragging a file, the level of friction could convey the file size. Objects could "snap" into place when designing a presentation. Or perhaps with a quick "rub" of your email application's icon, you could sense how many emails are unread. Finally, imagine a (flat) touch keyboard where the virtual keys can be felt. Tactile feedback based on electrovibration has several compelling properties. It is fast, low-powered, dynamic, and can be used in a wide range of interaction scenarios and applications, including multitouch interfaces. Our system demonstrates an exceptionally broad bandwidth and uniformity of response across a wide range of frequencies and amplitudes. Furthermore, the technology is highly scalable and can be used efficiently on touch surfaces of any size, shape and configuration, including large interactive tables, hand-held mobile devices, as well as curved, flexible and irregular touch surfaces. Lastly, because our design does not have any moving parts, it can be easily added to existing devices with minimal physical modification. </video> <video> DynaButtons: Fast Interactive Soft Buttons with Analog Control (IEEE HAPTICS 2024) Description: More Research: Fluid Reality Gloves: https://youtu.be/UJXLBqG9E_s Flat Panel Haptics: https://youtu.be/j_rErbhxNFM Research Team: Tucker Rae-Grant, Chris Harrison, and Craig Shultz Carnegie Mellon University and University of Illinois Urbana-Champaign While mechanical buttons are ubiquitous, their haptic response is fxed, reducing interface flexibility and precluding an avenue for rich feedback. Here, we describe a new type of dynamic button, or DynaButton, which can vary its visual and haptic response via rapid shape change. To achieve this, we made several advances in the performance of embedded electroosmotic pumps: we increased core pump speed more than 300% over prior work, demonstrated closed-loop control, and investigated analog output that varies in response to pressure and force inputs. We validated our system with a physical characterization and performed a series of stimuli recognition studies to validate the expressivity and discriminability of three regimes of button interaction. </video> <video> Expressive, Scalable, Mid-Air Haptics with Synthetic Jets Description: Non-contact, mid-air haptic devices have been utilized for a wide variety of experiences, including those in extended reality, public displays, medical, and automotive domains. In this work, we explore the use of synthetic jets as a promising and under-explored mid-air haptic feedback method. We show how synthetic jets can scale from compact, low-powered devices, all the way to large, long-range, and steerable devices. We built seven functional prototypes targeting different application domains, in order to illustrate the broad applicability of our approach. These example devices are capable of rendering complex haptic effects, varying in both time and space. We quantify the physical performance of our designs using spatial pressure and wind flow measurements, and validate their compelling effect on users with stimuli recognition and qualitative studies. Vivian Shen, Chris Harrison, and Craig Shultz. 2023. Expressive, Scalable, Mid-Air Haptics with Synthetic Jets. ACM Trans. Comput.-Hum. Interact. https://doi.org/10.1145/3635150 </video> <video> Fluid Reality: High-Resolution, Untethered Haptic Gloves Using Electroosmotic Pump Arrays Description: Same tech under touchscreen: https://youtu.be/j_rErbhxNFM (keyboard you can feel!) Virtual and augmented reality headsets are making significant progress in audio-visual immersion and consumer adoption. However, their haptic immersion remains low, due in part to the limitations of vibrotactile actuators which dominate the AR/VR market. In this work, we present a new approach to create high-resolution shape-changing fingerpad arrays with 20 haptic pixels per square cm. Unlike prior pneumatic approaches, our actuators are low-profile (5mm thick), low-power (approximately 10mW/pixel), and entirely self-contained, with no tubing or wires running to external infrastructure. We show how multiple actuator arrays can be built into a five-finger, 160-actuator haptic glove that is untethered, lightweight (207g, including all drive electronics and battery), and has the potential to reach consumer price points at volume production. We describe the results from a technical performance evaluation and a suite of eight user studies, quantifying the diverse capabilities of our system. This includes recognition of object properties such as complex contact geometry, texture, and compliance, as well as expressive spatiotemporal effects. Vivian Shen, Tucker Rae-Grant, Joe Mullenbach, Chris Harrison, and Craig Shultz. 2023. Fluid Reality: High-Resolution, Untethered Haptic Gloves using Electroosmotic Pump Arrays. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23). Association for Computing Machinery, New York, NY, USA, Article 8, 1–20. https://doi-org.cmu.idm.oclc.org/10.1145/3586183.3606771 </video> <video> Fluid Reality (ACM UIST 2023 Talk) Description: Academic talk at ACM UIST 2023. More details: https://www.figlab.com/research/2023/FluidReality </video> <video> WorldPoint: Finger Pointing as a Rapid and Natural Trigger for In-the-Wild Mobile Interactions Description: Pointing with one's finger is a natural and rapid way to denote an area or object of interest. It is routinely used in human-human interaction to increase both the speed and accuracy of communication, but it is rarely utilized in human-computer interactions. In this work, we use the recent inclusion of wide-angle, rear-facing smartphone cameras, along with hardware-accelerated machine learning, to enable real-time, infrastructure-free, finger-pointing interactions on today's mobile phones. We envision users raising their hands to point in front of their phones as a "wake gesture". This can then be coupled with a voice command to trigger advanced functionality. For example, while composing an email, a user can point at a document on a table and say "attach". Our interaction technique requires no navigation away from the current app and is both faster and more privacy-preserving than the current method of taking a photo. Citation: Daehwa Kim, Vimal Mollyn, and Chris Harrison. 2023. WorldPoint: Finger Pointing as a Rapid and Natural Trigger for In-the-Wild Mobile Interactions. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (ISS '23). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3626478 </video> <video> Pantœnna: Mouth Pose Estimation for VR/AR Headsets Using Low-Profile Antenna Description: Methods for faithfully capturing a user's holistic pose have immediate uses in AR/VR, ranging from multimodal input to expressive avatars. Although body-tracking has received the most attention, the mouth is also of particular importance, given that it is the channel for both speech and facial expression. In this work, we describe a new RF-based approach for capturing mouth pose using an antenna integrated into the underside of a VR/AR headset. Our approach side-steps privacy issues inherent in camera-based methods, while simultaneously supporting silent facial expressions that audio-based methods cannot. Further, compared to bio-sensing methods such as EMG and EIT, our method requires no contact with the wearer's body and can be fully self-contained in the headset, offering a high degree of physical robustness and user practicality. We detail our implementation along with results from two user studies, which show a mean 3D error of 2.6 mm for 11 mouth keypoints across worn sessions without re-calibration. Citation: Daehwa Kim and Chris Harrison. 2023. Pantœnna: Mouth pose estimation for ar/vr headsets using low-profile antenna and impedance characteristic sensing. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23). Association for Computing Machinery, New York, NY, USA, Article 83, 1–12. https://doi.org/10.1145/3586183.3606805 </video> <video> ISS 2024 Lasting Impact Talk - Estimating 3D Finger Angle on Commodity Touchscreens (ITS 2015) Description: A retrospective look at our work on "Estimating 3D Finger Angle on Commodity Touchscreens" (ITS 2015) and subsequent research utilizing touchscreen capacitive images. Original Paper: Xiao, R. Schwarz, J. and Harrison, C. 2015. Estimating 3D Finger Angle on Commodity Touchscreens. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (Madeira, Portugal, November 15 - 18, 2015). ITS ‘15. ACM, New York, NY. 47-50. https://www.figlab.com/research/2015/fingerpose </video>

Contacts and Links

Description

The Future Interfaces Group (FIG) is an interdisciplinary research laboratory within the Human-Computer Interaction Institute at Carnegie Mellon University. We create new sensing and interface technologies that aim to make interactions between humans and computers more fluid, intuitive, and powerful. These efforts often lie in emerging use modalities, such as wearable computing, touch interfaces and gestural interaction.

Videos from channel