Not Seeing the Whole Picture: Challenges and Opportunities in Using AI for Co-Making Physical, DIY-AT for People with Visual ImpairmentsExisting assistive technologies (AT) often adopt a one-size-fits-all approach, overlooking the diverse needs of people with visual impairments (PVI). Do-it-yourself AT (DIY-AT) toolkits offer one path toward customization, but most remain limited—targeting co-design with engineers or requiring programming expertise. Non-professionals with disabilities, including PVI, also face barriers such as inaccessible tools, lack of confidence, and insufficient technical knowledge. These gaps highlight the need for prototyping technologies that enable PVI to directly make their own AT. Building on emerging evidence that large language models (LLMs) can serve not only as visual aids but also as co-design partners, we present an exploratory study of how LLM-based AI can support PVI in the tangible DIY-AT co-making process. Our findings surface key challenges and design opportunities: the need for greater spatial and visual support, strategies for mitigating novel AI errors, and implications for designing more accessible AI-assisted prototypes.2026BKBen Kosa et al.University of Wisconsin--MadisonElectrical Muscle Stimulation (EMS)Generative AI (Text, Image, Music, Video)Explainable AI (XAI)CHI
Moisture Transfer: A Perceptual Wetness Illusion Through Thermal and Wet IntegrationSimulating wetness in interactive systems is challenging due to the lack of dedicated hygroreceptors in human skin and the complexity of delivering physical moisture. We introduce Moisture Transfer, a perceptual wetness illusion in which users feel moisture at a dry site when cold and wet stimuli are applied nearby. This illusion arises from the brain’s spatial integration of thermal and tactile cues, offering a new pathway to render wetness without direct contact. We investigate this illusion by establishing it with a single finger and show that thermal congruence enhances perceived wetness. We then explored its spatial extent across five fingers, revealing lateral transfer of wetness. Finally, we applied these findings to create a proof-of-concept VR interface that evokes full-hand wetness using minimal actuation. We conclude with design implications for XR and wearable systems and outline future work exploring body-wide wetness illusions and multisensory integration.2026YSYatharth Singhal et al.University of Texas at DallasMultisensory Fusion ExperienceThermal & Temperature InteractionImmersion & Presence ResearchCHI
AgentCoach: LLM-Based Adaptive Coaching Feedback for Motor Skill LearningWe present AgentCoach, an LLM-powered system that provides adaptive feedback for motor skill learning from tutorial videos. The system works by extracting key coaching points (CPs) and compiling CP-specific evaluators that map each cue to measurable kinematic parameters. This process allows AgentCoach to connect high-level semantic meaning with low-level postural estimation for accurate, context-aware evaluation. During practice, learners receive concise visual diagnostics of their mistakes paired with prescriptive verbal feedback that adapts based on their performance history. We technically validate the CP extraction and evaluator compilation across a wide range of common sports and exercise videos. A user study confirms the system's usability and shows the system's potential effectiveness of its adaptive feedback across multiple skills.2026DMDizhi Ma et al.Purdue UniversityHuman Pose & Activity RecognitionFitness Tracking & Physical Activity MonitoringBehavior Change & Reflection TechnologyCHI
Touch with Meaning: A Contextual Analysis of Social TouchSocial touch is a rich channel of human communication, conveying emotion, intent, and meaning embedded in context. Yet most HCI studies treat touch in isolation, overlooking the layered subtleties that shape interpretation. We present a contextual analysis of 5,016 social touch events, grounded in a large collection of annotated scenes from films, dramas, and documentaries. Using a computer vision pipeline, we segmented touch events from video and annotated them across dimensions, including who is involved, how the gesture is performed, where on the body it occurs, and the cultural backdrop. Our analysis shows that identical gestures can convey distinct meanings depending on body location, relationship type, and context. Similar intentions—like comfort, encouragement, or dominance—may be expressed through different gestures or locations, shaped by relational dynamics, cultural norms, and public or private settings. These insights inform the design of socially aware touch technologies, including avatars, social agents, and mediated communication systems.2026ABAyush Bhardwaj et al.The University of Texas at DallasVibrotactile Feedback & Skin StimulationEmpathy & Emotional DesignSocial Robot InteractionCHI
FretFlow: Adaptive Haptics for Rhythm and Articulation in Guitar LearningRhythm and articulation are essential for expressive guitar performance. Existing tools provide basic beat cues, whereas beginners often struggle to align with these cues when playing complex techniques, such as strumming and muting. Informed by a formative study with five instructors and grounded in embodied learning theories, we present FretFlow, a haptic vest-based tool that simulates common instructional practices to guide learners through physical interactions like tapping. The key to FretFlow is its design space that maps rhythmic and articulation patterns in various playing techniques to distinct haptic patterns, enabling authoring of haptic scores. FretFlow further dynamically adapts haptic intensity based on learners' real-time performance accuracy, accompanied by multimodal guidance across haptic, visual, and audio channels. We iteratively refined haptic designs across two rounds with 46 participants, followed by a two-week user study with 20 beginners. Results show that FretFlow improves learners’ rhythmic accuracy and expressive performance.2026XSXin Shu et al.Newcastle UniversityHaptic WearablesBehavior Change & Reflection TechnologyFull-Body Interaction & Embodied InputCHI
Thermal Masking Across the Human Body: Patterns, Pathways, and Perceptual BoundariesThermal masking, a vibration-induced illusion in which concurrent tactile input induces a vivid thermal sensation at the tactile site, is a promising mechanism for wearable interfaces and extended reality because it can deliver rich thermal feedback with minimal hardware. While prior work has examined this phenomenon on limited body parts, its expression across the full body remains under exploration. We present four studies mapping thermal masking across eight regions: head, face, neck, arms, hands, torso, legs, and feet. Results show that masking strength is location-dependent, producing perceptual patterns that align primarily with somatosensory pathways rather than proximity. On smaller regions such as the fingers, masking was localized, while on larger areas such as the torso and neck, it extended more broadly. Dorsal–ventral and inter-body tests revealed viable pairings and perceptual boundaries. These findings provide the first comprehensive atlas of body-wide thermal masking, advancing understanding and guiding efficient thermal–tactile interface design.2026HWHaokun Wang et al.University of Texas at DallasThermal & Temperature InteractionHaptic WearablesEmotion-Sensing WearablesCHI
SensoryBlox: Plug-and-Feel Modular Multi-Sensory User Interface for Immersive Cardboard VRWe present SensoryBlox, a modular, multi-sensory user interface designed for integration with cardboard-based virtual reality (VR) head-mounted displays (HMDs). SensoryBlox features interchangeable sensory modules—vibration, temperature, wind, and olfactory—that enable users to assemble customized multi-sensory configurations tailored to diverse VR contexts. The system includes in-VR interfaces for module scanning, spatial tracking, and real-time customization of feedback patterns. To inform SensoryBlox design, we conducted three user studies. The initial study explored application scenarios and associated sensory modalities to identify design requirements for a modular multi-sensory VR system. Based on these findings, we developed the hardware modules and in-VR software interfaces. In the second study, we evaluated the usability and interaction experience of SensoryBlox across all functionalities. Finally, a comparison study examined the impact of multi-sensory feedback on user experience. Our findings demonstrate the potential of a modular multi-sensory system in enriching immersion and engaging interactions within low-cost VR environments.2026HGHyunjae Gil et al.The University of Texas at DallasMultisensory Fusion ExperienceImmersion & Presence ResearchPhysical-Digital Hybrid InteractionCHI
UltraEdit: In-Situ Design Environment for Ultrasound HaptizationWe present UltraEdit, an in-situ design environment that enables users to directly edit ultrasound haptic sensations in VR using barehand interactions. UltraEdit represents haptic sensations as tangible objects, called blobs, which users can modify or apply to 3D objects through intuitive hand gestures. Users can create, edit, copy, and apply blobs, allowing them to work with multiple haptic sensations to design virtual objects with diverse tactile feedback. Additionally, UltraEdit allows users to create spatial and temporal haptic patterns using time blobs and spatial drawing features. Users can draw custom-shaped haptic patterns directly on their hands, adapting to comprehensive design scenarios. To evaluate UltraEdit, we conducted an exploratory user study assessing its usability, effectiveness, and ease of learning. We also compared its performance to an existing desktop-based haptic editing tool. Participants found UltraEdit intuitive to learn, enjoyable to use, and effective for adding haptic feedback to virtual objects.2025RNRichard Huynh Noeske et al.Mid-Air Haptics (Ultrasonic)Shape-Changing Interfaces & Soft Robotic MaterialsHand Gesture RecognitionUIST
HeatFlow: A Thermal-Tactile Display for Dynamic 2D Thermal MovementsWe introduce HeatFlow, a thermal display capable of generating dynamic thermal movements on a 2D surface by integrating thermal and tactile sensations. Leveraging a perception-driven approach, we combine vibration-induced thermal referral and apparent tactile motion to create the illusion of moving thermal flows. The system features a 2D array of nine tactile actuators paired with a single nearby thermal actuator. To validate our approach, we conducted three user studies. The first study determined the optimal parameters for perceivable 2D thermal flows by examining different array sizes and motion speeds under varying temperature conditions. The second study focused on optimizing our algorithm to produce smooth and continuous curved thermal flows. Comparative evaluations against two prior tactile motion techniques demonstrated that our algorithm outperforms existing methods in generating realistic thermal motion. Finally, we integrated HeatFlow into virtual reality (VR) environments, assessing its feasibility in six interactive scenarios across three body sites—the arm, palm, and face. Our findings highlight the potential of HeatFlow to enhance immersive VR experiences by providing realistic thermal feedback across multiple body locations. Additionally, we present HeatFlow as a design tool—an application that enables users to create custom thermal flows by adjusting key parameters such as motion speed, intensity, and temperature. This work lays the foundation for more advanced thermal interfaces in interactive systems.2025YSYatharth Singhal et al.Mid-Air Haptics (Ultrasonic)Shape-Changing Interfaces & Soft Robotic MaterialsFull-Body Interaction & Embodied InputUIST
AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multimodal Information Between Reality and VideosVideos offer rich audiovisual information that can support people in performing activities of daily living (ADLs), but they remain largely inaccessible to blind or low-vision (BLV) individuals. In cooking, BLV people often rely on non-visual cues---such as touch, taste, and smell---to navigate their environment, making it difficult to follow the predominantly audiovisual instructions found in video recipes. To address this problem, we introduce AROMA, an AI system that provides timely responses to the user based on real-time, context-aware assistance by integrating non-visual cues perceived by the user, a wearable camera feed, and video recipe content. AROMA uses a mixed-initiative approach: it responds to user requests while also proactively monitoring the video stream to offer timely alerts and guidance. This collaborative design leverages the complementary strengths of the user and AI system to align the physical environment with the video recipe, helping the user interpret their current cooking state and make sense of the steps. We evaluated AROMA through a study with eight BLV participants and offered insights for designing interactive AI systems to support BLV individuals in performing ADLs.2025ZNZheng Ning et al.Conversational ChatbotsVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Context-Aware ComputingUIST
VRSight: An AI-Driven Scene Description System to Improve Virtual Reality Accessibility for Blind PeopleVirtual Reality (VR) is inaccessible to blind people. While research has investigated many techniques to enhance VR accessibility, they require additional developer effort to integrate. As such, most mainstream VR apps remain inaccessible as the industry de-prioritizes accessibility. We present VRSight, an end-to-end system that recognizes VR scenes post hoc through a set of AI models (e.g., object detection, depth estimation, LLM-based atmosphere interpretation) and generates tone-based, spatial audio feedback, empowering blind users to interact in VR without developer intervention. To enable virtual element detection, we further contribute DISCOVR, a VR dataset consisting of 30 virtual object classes from 17 social VR apps, substituting real-world datasets that remain not applicable to VR contexts. Nine participants used VRSight to explore an off-the-shelf VR app (Rec Room), demonstrating its effectiveness in facilitating social tasks like avatar awareness and available seat identification.2025DKDaniel Killough et al.Social & Collaborative VRExplainable AI (XAI)Visual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)UIST
PropType: Everyday Props as Typing Surfaces in Augmented RealityWe introduce PropType, an interactive interface that transforms everyday objects into typing surfaces within an Augmented Reality (AR) environment. Users can interact with nearby props, such as cups, water bottles, boxes, and various other objects, utilizing them as on-the-go keyboards. To develop PropType, we conducted three studies. The first study involved observing users to understand how they naturally engage with prop surfaces for typing. The second study assessed the reachability and efficiency of touch input across four props with different sizes and shapes. Based on these insights, we designed customized keyboard layouts for each prop. In the third study, we evaluated typing performance using PropType, achieving an average typing speed of up to 26.1 words per minute (WPM) with 2.2% corrected error rate (CER) and 1.1% uncorrected error rate (UER). Finally, we present a PropType editing tool that allows users to customize keyboard layouts and visual effects for prop-based typing.2025HGHyunjae Gil et al.The University of Texas at Dallas, Department of Computer ScienceHand Gesture RecognitionAR Navigation & Context AwarenessCHI
Let It Snow: Designing Snowfall Experience in VRWang等人设计并评估VR中的雪景体验，通过多感官反馈技术创造沉浸式自然现象模拟，提升用户真实感。2024HWHaokun Wang et al.Immersion & Presence ResearchUbiComp
Thermal In Motion: Designing Thermal Flow Illusions with Tactile and Thermal InteractionThis study presents a novel method for creating moving thermal sensations by integrating the thermal referral illusion with tactile motion. Conducted through three experiments on human forearms, the first experiment examined the impact of temperature and thermal actuator placement on perceived thermal motion, finding the clearest perception with a centrally positioned actuator under both hot and cold conditions. The second experiment identified the speed thresholds of perceived thermal motion, revealing a wider detectable range in hot conditions (1.8 cm/s to 9.5cm/s) compared to cold conditions (2.4cm/s to 5.0cm/s). Finally, we integrated our approach into virtual reality (VR) to assess its feasibility through two interaction scenarios. Our results shed light on the comprehension of thermal perception and its integration with tactile cues, promising significant advancements in incorporating thermal motion into diverse thermal interfaces for immersive VR experiences.2024YSYatharth Singhal et al.Vibrotactile Feedback & Skin StimulationImmersion & Presence ResearchUIST
Fiery Hands: Designing Thermal Glove through Thermal and Tactile Integration for Virtual Object ManipulationWe present a novel approach to render thermal and tactile feedback to the palm and fingertips through thermal and tactile integration. Our approach minimizes the obstruction of the palm and inner side of the fingers and enables virtual object manipulation while providing localized and global thermal feedback. By leveraging thermal actuators positioned strategically on the outer palm and back of the fingers in interplay with tactile actuators, our approach exploits thermal referral and tactile masking phenomena. Through a series of user studies, we validate the perception of localized thermal sensations across the palm and fingers, showcasing the ability to generate diverse thermal patterns. Furthermore, we demonstrate the efficacy of our approach in VR applications, replicating diverse thermal interactions with virtual objects. This work represents significant progress in thermal interactions within VR, offering enhanced sensory immersion at an optimal energy cost.2024HWHaokun Wang et al.Vibrotactile Feedback & Skin StimulationEye Tracking & Gaze InteractionImmersion & Presence ResearchUIST
CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low VisionCooking is a central activity of daily living, supporting independence as well as mental and physical health. However, prior work has highlighted key barriers for people with low vision (LV) to cook, particularly around safely interacting with tools, such as sharp knives or hot pans. Drawing on recent advancements in computer vision (CV), we present CookAR, a head-mounted AR system with real-time object affordance augmentations to support safe and efficient interactions with kitchen tools. To design and implement CookAR, we collected and annotated the first egocentric dataset of kitchen tool affordances, fine-tuned an affordance segmentation model, and developed an AR system with a stereo camera to generate visual augmentations. To validate CookAR, we conducted a technical evaluation of our fine-tuned model as well as a qualitative lab study with 10 LV participants for suitable augmentation design. Our technical evaluation demonstrates that our model outperforms the baseline on our tool affordance dataset, while our user study indicates a preference for affordance augmentations over the traditional whole object augmentations.2024JLJaewook Lee et al.AR Navigation & Context AwarenessVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Deaf & Hard-of-Hearing Support (Captions, Sign Language, Vibration)UIST
MIMOSA: Human-AI Co-Creation of Computational Spatial Audio Effects on VideosSpatial audio offers more immersive video consumption experiences to viewers; however, creating and editing spatial audio often expensive and requires specialized equipment and skills, posing a high barrier for amateur video creators. We present MIMOSA, a human-AI co-creation tool that enables amateur users to computationally generate and manipulate spatial audio effects. For a video with only monaural or stereo audio, MIMOSA automatically grounds each sound source to the corresponding sounding object in the visual scene and enables users to further validate and fix the errors in the locations of sounding objects. Users can also augment the spatial audio effect by flexibly manipulating the sounding source positions and creatively customizing the audio effect. The design of MIMOSA exemplifies a human-AI collaboration approach that, instead of utilizing state-of-art end-to-end "black-box" ML models, uses a multistep pipeline that aligns its interpretable intermediate results with the user’s workflow. A lab user study with 15 participants demonstrates MIMOSA’s usability, usefulness, expressiveness, and capability in creating immersive spatial audio effects in collaboration with users.2024ZNZheng Ning et al.Generative AI (Text, Image, Music, Video)Music Composition & Sound Design ToolsCreative Collaboration & Feedback SystemsC&C
SPICA: Interactive Video Content Exploration through Augmented Audio Descriptions for Blind or Low-Vision ViewersBlind or Low-Vision (BLV) users often rely on audio descriptions (AD) to access video content. However, conventional static ADs can leave out detailed information in videos, impose a high mental load, neglect the diverse needs and preferences of BLV users, and lack immersion. To tackle these challenges, we introduce SPICA, an AI-powered system that enables BLV users to interactively explore video content. Informed by prior empirical studies on BLV video consumption, SPICA offers novel interactive mechanisms for supporting temporal navigation of frame captions and spatial exploration of objects within key frames. Leveraging an audio-visual machine learning pipeline, SPICA augments existing ADs by adding interactivity, spatial sound effects, and individual object descriptions without requiring additional human annotation. Through a user study with 14 BLV participants, we evaluated the usability and usefulness of SPICA and explored user behaviors, preferences, and mental models when interacting with augmented ADs.2024ZNZheng Ning et al.University of Notre DameVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)CHI
Thermal Masking: When the Illusion Takes Over the RealThis paper reports on a thermal illusion called thermal masking. Thermal masking is a phenomenon induced by thermal referral to completely mask the original thermal sensation, providing thermal sensation only at the tactile site. Three experiments are conducted using thermal and vibrotactile actuators to investigate the nature of thermal masking on human arms. The first experiment investigates the effects of different temperatures on masking. The results show a higher percentage of thermal masking occurs in warm than hot or cold conditions. The second experiment examines how far the thermal masking can be perceived. The results show that masking can reach up to 24 cm from the thermal site. The third experiment explores the interaction space by placing the tactile actuators on the opposite side of the thermal actuator. The results confirm that thermal masking can reach the other side of the arm, and the performance was higher in warm conditions.2024HWHaokun Wang et al.University of Texas at DallasVibrotactile Feedback & Skin StimulationForce Feedback & Pseudo-Haptic WeightCHI
PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual DataAudio-visual learning seeks to enhance the computer’s multi-modal perception leveraging the correlation between the auditory and visual modalities. Despite their many useful downstream tasks, such as video retrieval, AR/VR, and accessibility, the performance and adoption of existing audio-visual models have been impeded by the availability of high quality datasets. Annotating audio-visual datasets is laborious, expensive, and time consuming. To address this challenge, we designed and developed an efficient audio visual annotation tool called Peanut. Peanut’s human-AI collaborative pipeline separates the multi-modal task into two single-modal tasks, and utilizes state-of-the-art object detection and sound-tagging models to reduce the annotators’ effort to process each frame and the number of manually-annotated frames needed. A within-subject user study with 20 participants found that Peanut can significantly accelerate the audio-visual data annotation process while maintaining high annotation accuracy.2023ZZZheng Zhang et al.Conversational ChatbotsHuman-LLM CollaborationRecommender System UXUIST