AcouLetter: Enabling Fine-Grained Mid-Air Alphabetic Thumb Gesture Recognition on Smartphones with Acoustic and Inertial SensingMobile interaction is fundamentally tied to the screen, requiring users to look at and physically touch their devices by tapping on-screen controls. We introduce AcouLetter, a system for recognizing fine-grained, mid-air alphabetic thumb gestures above the screen that enables one-handed and non-contact input. Our method fuses acoustic signals with inertial sensor data to discern the subtle characteristics of each letter gesture, capturing the dynamics of the hand's motion. We employ a multi-stage architecture that integrates a convolutional network for acoustic features and a transformer-based network for inertial data. AcouLetter first detects the initiation of the gesture using inertial measurements, determines which hand holds the phone, and then classifies the specific letter among 27 classes, including the 26 letters of the alphabet and a null gesture. The system achieves high accuracy and demonstrates robust performance even under ambient noise while a user is standing or walking. Since letter-shaped gestures are meaningful and readily recognizable, they support an intuitive system of expressive shortcuts in which letters act as mnemonics for complex functions. A user study further confirmed the practicality of this approach for one-handed, contact-free interaction. Our results demonstrate that this expressive alphabetic gestural interface is a feasible new modality for commodity smartphones.2026SSSu Yeon Shim et al.POSTECHHand Gesture RecognitionFoot & Wrist InteractionMobile App User ExperienceIUI
Effects of Haptic Feedback on Gaming Experiences: A Case Study Comparing Players and Spectators in FPS GamesHaptic feedback has become a common feature in game experiences, yet little is known about how its effects differ between active players and passive spectators. This study investigated how haptic feedback influences user experience and technology acceptance in the context of first-person shooter (FPS) games, particularly by comparing its effects on active players and passive spectators. An experiment with 60 participants tested four conditions defined by two factors: haptic feedback (present vs. absent) and user role (player vs. spectator). The results showed that haptic feedback enhanced the intention to use games in both roles, with a stronger effect among spectators. Players’ intention was primarily driven by perceived enjoyment through a hedonic pathway, whereas spectators responded to both perceived enjoyment and perceived usefulness through both hedonic and eudaimonic pathways. These findings highlight the need for role-sensitive haptic design and validate two pathways of the Haptic-Augmented Game Technology Acceptance Model (HAG-TAM) for gaming experiences.2026HSHeeji Sohn et al.Pohang University of Science and Technology (POSTECH)Vibrotactile Feedback & Skin StimulationGame UX & Player BehaviorLive Streaming & Spectating ExperienceCHI
"Here, Let Me Help": An Empirical Study of User Interventions in Human–Web Agent CollaborationWeb agents aim to execute complex online tasks from high-level instructions, yet fully autonomous execution remains challenging in practice. We present an empirical study of user interventions in human–web agent collaboration, moving beyond outcome-based metrics to examine how interventions unfold during execution. We conducted a controlled in-lab study with 30 participants whose interactions reflected early-stage web agent adoption across 12 structured tasks in shopping, travel, and information-seeking domains using live websites. Analyzing interaction logs, user inputs, and screen recordings, we identify diverse behaviors and propose a taxonomy capturing both the reasons for intervention and the forms they take. We distinguish explicit interventions, where users halt or override actions, from implicit interventions, where users guide or prepare the environment without stopping execution. Our findings reveal how task structure and execution breakdowns shape intervention behaviors to provide process-level evidence for designing web agents that better support users as active collaborators.2026JKJoohee Kim et al.Ulsan National Institute of Science and TechnologyHuman-LLM CollaborationAI-Assisted Decision-Making & AutomationUser Research Methods (Interviews, Surveys, Observation)CHI
Touch with Meaning: A Contextual Analysis of Social TouchSocial touch is a rich channel of human communication, conveying emotion, intent, and meaning embedded in context. Yet most HCI studies treat touch in isolation, overlooking the layered subtleties that shape interpretation. We present a contextual analysis of 5,016 social touch events, grounded in a large collection of annotated scenes from films, dramas, and documentaries. Using a computer vision pipeline, we segmented touch events from video and annotated them across dimensions, including who is involved, how the gesture is performed, where on the body it occurs, and the cultural backdrop. Our analysis shows that identical gestures can convey distinct meanings depending on body location, relationship type, and context. Similar intentions—like comfort, encouragement, or dominance—may be expressed through different gestures or locations, shaped by relational dynamics, cultural norms, and public or private settings. These insights inform the design of socially aware touch technologies, including avatars, social agents, and mediated communication systems.2026ABAyush Bhardwaj et al.The University of Texas at DallasVibrotactile Feedback & Skin StimulationEmpathy & Emotional DesignSocial Robot InteractionCHI
HaRing: A Haptic Ring Interface for One-Handed Interaction with High-Dimensional Spatial InformationRing interfaces have gained attention in wearable technology for their lightweight and hands-free design. However, their compact form factor limits them to conveying simple information, such as direction or notification, through vibration, electrotactile, or force feedback. In this paper, we introduce HaRing, a novel haptic ring interface equipped with a 4 × 6 pin array display. This dynamic display delivers rich spatial patterns that simple vibration cannot express, effectively conveying high-dimensional information such as directions, semantic symbols, and letters. Its design enables one-handed, eyes-free interaction that does not interfere with visual tasks. We conducted a series of perceptual and user studies to demonstrate its effectiveness, showing a high recognition accuracy of over 94% for complex letters after a brief training period. We anticipate that HaRing can serve as an innovative haptic-only interface for multitasking in real-world or VR environments with high visual load.2026SNSuheon Nam et al.POSTECHMid-Air Haptics (Ultrasonic)Haptic WearablesImmersion & Presence ResearchCHI
I Feel We Are Together: How People Perceive Personalized Face-Swapped GIFs in Text-Based CommunicationNonverbal cues in text-based computer-mediated communication (CMC), initially introduced to compensate for the lack of social and emotional cues, have evolved beyond their original purpose to express user identity. In particular, embodied identity cues—such as a user’s real face—remain relatively underexplored in text-based CMC despite their potential as richer cues. Recent advances in generative AI have lowered the barrier to AI-mediated self-presentation, yet empirical research is still needed to understand how these cues operate in real interactions and how users experience and accept them. To address this gap, we investigate the social and emotional effects of face-swapped GIFs (FSGIFs) created via generative AI. In a two-phase within-subjects experiment with 32 participants (16 dyads of close acquaintances), we find that FSGIFs significantly enhance relational benefits, including greater co-presence and intimacy compared to generic GIFs. Based on these findings and insights from interviews, we discuss design implications for AI-mediated self-presentation in text-based CMC.2026DJDaeun Jeong et al.Pohang University of Science and TechnologyGenerative AI (Text, Image, Music, Video)Affective Human-Computer DialogueAgent Personality & AnthropomorphismCHI
Automatic Tuning of Haptic Motion Effects to Evoke Specific Feelings in Multisensory ContentAutomating the authoring of haptic motion effects, while enabling designers to carefully consider user feelings to provide high-quality user experiences, is crucial for effective multisensory content. We present a motion effect-tuning method that elicits desired perceptual or affective attributes from users watching a video. To this end, we test three modulation methods: (1) Altering the extent of low-frequency motion fluctuations, (2) Changing the motion amplitude in a high-frequency band, and (3) Sampling and interpolating significant motion peaks. Our tuning method transforms an input draft waveform using the modulation techniques to obtain an output motion effect that elicits the goal adjective scores. This method requires two regression models accounting for the effects of motion modulation and audiovisual stimuli, respectively, and we obtain them by conducting perceptual experiments. Lastly, we confirm the method's effectiveness through another user study and explore potential users' feedback and suggestions for future applications through open-ended survey questions.2025JLJiwan Lee et al.Pohang University of Science and Technology (POSTECH), Computer Science and EngineeringVibrotactile Feedback & Skin StimulationMusic Composition & Sound Design ToolsVideo Production & EditingCHI
Real-time Semantic Full-Body Haptic Feedback Converted from Sound for Virtual Reality GameplayWe present a multisensory virtual reality (VR) system that enables users to experience concurrent visual, auditory, and haptic feedback, featuring semantic classification of events from sound, sound-to-haptic conversion, and full-body haptic effects. This concept is applied to enhance the user experience of virtual reality (VR) gameplay. The system utilizes a Long-Short-Term Memory (LSTM) model to classify game sounds and detect key events such as gunfire, explosions, and hits. These events are translated into full-body haptic patterns through a haptic suit, providing users with realistic and immersive haptic experiences. The system operates with low latency, ensuring the seamless synchrony between sound and haptic feedback. Evaluations through user studies demonstrate significant improvements in user experience compared to traditional sound-to-haptic methods, emphasizing the importance of accurate sound classification and well-designed haptic effects.2025GYGyeore Yun et al.Kyungpook National University, Computer Science & Engineering / College of IT Engineering /Full-Body Interaction & Embodied InputGamification DesignCHI
SkinHaptics: Exploring Skin Softness Perception and Virtual Body Embodiment Techniques to Enhance Self-Haptic InteractionsProviding haptic feedback for soft, deformable objects is challenging, requiring complex mechanical hardware combined with modeling and rendering software. As an alternative, we advance the concept of self-haptics, where the user's own body delivers physical feedback, to convey dynamically varying softness in VR. Skin can exhibit different levels of contact softness by altering the biomechanical state of the body. We propose SkinHaptics, a device-free approach that changes the states of musculoskeletal structures and virtual hand-object representations. In this study, we conduct three experiments to demonstrate SkinHaptics. Using the same scale, we measure skin softness across various hand poses and contact points and evaluate the just noticeable difference in skin softness. We investigate the effect of hand-object representations on self-haptic interactions. Our findings indicate that the visual representations have a significant influence on the embodiment of a self-haptic hand, and the degree of the hand embodiment strongly affects the haptic experience.2025JLJungeun Lee et al.Pohang University of Science and Technology (POSTECH), Convergence IT Engineering / Interaction LaboratoryHaptic WearablesEye Tracking & Gaze InteractionCHI
Toward Affective Empathy via Personalized Analogy Generation: A Case Study on MicroaggressionThe importance of empathy cannot be overstated in modern societies where people of diverse backgrounds increasingly interact together. The HCI community has strived to foster affective empathy through immersive technologies. Many previous techniques are built upon a premise that presenting the same experience as-is may help evoke the same emotion, which however faces limitations in matters where the emotional responses largely differ across individuals. In this paper, we present a novel concept of generating a personalized experience based on a large language model (LLM) to facilitate affective empathy between individuals despite their differences. As a case study to showcase its effectiveness, we developed EmoSync, an LLM-based agent that generates personalized analogical microaggression situations, facilitating users to personally resonate with a specific microaggression situation of another person. EmoSync is designed and evaluated along a 3-phased user study with 100+ participants. We comprehensively discuss implications, limitations, and possible applications.2025HJHyojin Ju et al.POSTECH, Department of Computer Science and EngineeringHuman-LLM CollaborationEmpowerment of Marginalized GroupsCHI
Modulating Heart Activity and Task Performance using Haptic Heartbeat Feedback: A Study Across Four Body PlacementsThis paper explores the impact of vibrotactile haptic feedback on heart activity when the feedback is provided at four different body locations (chest, wrist, neck, and ankle) and with two feedback rates (50 bpm and 110 bpm). A user study found that the neck placement resulted in higher heart rates and lower heart rate variability, and higher frequencies correlated with increased heart rates and decreased heart rate variability. The chest was preferred in self-reported metrics, and neck placement was perceived as less satisfying, harmonious, and immersive. This research contributes to understanding the interplay between psychological experiences and physiological responses when using haptic biofeedback resembling real body signals.2024AVAndreia Valente et al.Vibrotactile Feedback & Skin StimulationUIST
Augmenting Perceived Length of Handheld Controllers: Effects of Object Handle PropertiesIn the realm of virtual reality (VR), shape-changing controllers have emerged as a means to enhance visuo-haptic congruence during user interactions. The major emphasis has been placed on manipulating the inertia tensor of a shape-changing controller to control the perceived shape. This paper delves deeper by exploring how the material properties of the controller's handle, distinct from the inertial information, affect the perceived shape, focusing on the perceived length. We conducted three perceptual experiments to examine the effects of the handle's softness, thermal conductivity, and texture, respectively. Results demonstrated that a softer handle increases the perceived length, whereas a handle with higher thermal conductivity reduces it. Texture, in the form of varying bumps, also alters the length perception. These results provide more comprehensive knowledge of the intricate relationship between perceived length and controller handle properties, expanding the design alternatives for shape-changing controllers for immersive VR experiences.2024CPChaeyong Park et al.Pohang University of Science and Technology (POSTECH)Force Feedback & Pseudo-Haptic WeightShape-Changing Interfaces & Soft Robotic MaterialsCHI
Open Sesame? Open Salami! Personalizing Vocabulary Assessment-Intervention for Children via Pervasive Profiling and Bespoke Storybook GenerationChildren acquire language by interacting with their surroundings. Due to the different language environments each child is exposed to, the words they encounter and need in their life vary. Despite the standard tools for assessment and intervention as per predefined vocabulary sets, speech-language pathologists and parents struggle with the absence of systematic tools for child-specific custom vocabulary, i.e., out-of-standard but personally more important. We propose "Open Sesame? Open Salami! (OSOS)", a personalized vocabulary assessment and intervention system with pervasive language profiling and targeted storybook generation, collaboratively developed with speech-language pathologists. Melded into a child's daily life and powered by large language models (LLM), OSOS profiles the child's language environment, extracts priority words therein, and generates bespoke storybooks naturally incorporating those words. We evaluated OSOS through 4-week-long deployments to 9 families. We report their experiences with OSOS, and its implications in supporting personalization outside standards.2024JLJungeun Lee et al.POSTECHMultilingual & Cross-Cultural Voice InteractionGenerative AI (Text, Image, Music, Video)Special Education TechnologyCHI
ProxiFit: Proximity Magnetic Sensing Using a Single Commodity Mobile toward Holistic Weight Exercise Monitoring"Although many works bring exercise monitoring to smartphone and smartwatch, inertial sensors used in such systems require device to be in motion to detect exercises. We introduce ProxiFit, a highly practical on-device exercise monitoring system capable of classifying and counting exercises even if the device stays still. Utilizing novel proximity sensing of natural magnetism in exercise equipment, ProxiFit brings (1) a new category of exercise not involving device motion such as lower-body machine exercise, and (2) a new off-body exercise monitoring mode where a smartphone can be conveniently viewed in front of the user during workouts. ProxiFit addresses common issues of faint magnetic sensing by choosing appropriate preprocessing, negating adversarial motion artifacts, and designing a lightweight yet noise-tolerant classifier. Also, application-specific challenges such as a wide variety of equipment and the impracticality of obtaining large datasets are overcome by devising a unique yet challenging training policy. We evaluate ProxiFit on up to 10 weight machines (5 lower- and 5 upper-body) and 4 free-weight exercises, on both wearable and signage mode, with 19 users, at 3 gyms, over 14 months, and verify robustness against user and weather variations, spatial and rotational device location deviations, and neighboring machine interference." https://doi.org/10.1145/36109202023JKJiha Kim et al.Fitness Tracking & Physical Activity MonitoringUbiComp
AI-to-Human Actuation: Boosting Unmodified AI’s Robustness by Proactively Inducing Favorable Human Sensing Conditions"Imagine a near-future smart home. Home-embedded visual AI sensors continuously monitor the resident, inferring her activities and internal states that enable higher-level services. Here, as home-embedded sensors passively monitor a free person, good inferences happen randomly. The inferences' confidence highly depends on how congruent her momentary conditions are to the conditions favored by the AI models, e.g., front-facing or unobstructed. We envision new strategies of AI-to-Human Actuation (AHA) that empower the sensory AIs with proactive actuation so that they induce the person's conditions to be more favorable to the AIs. In this light, we explore the initial feasibility and efficacy of AHA in the context of home-embedded visual AIs. We build a taxonomy of actuations that could be issued to home residents to benefit visual AIs. We deploy AHA in an actual home rich in sensors and interactive devices. With 20 participants, we comprehensively study their experiences with proactive actuation blended with their usual home routines. We also demonstrate the substantially improved inferences of the actuation-empowered AIs over the passive sensing baseline. This paper sets forth an initial step towards interweaving human-targeted AIs and proactive actuation to yield more chances for high-confidence inferences without sophisticating the model, in order to improve robustness against unfavorable conditions. https://dl.acm.org/doi/10.1145/3580812"2023SCSungjae Cho et al.Brain-Computer Interface (BCI) & NeurofeedbackContext-Aware ComputingSmart Home Interaction DesignUbiComp
Generating Real-Time, Selective, and Multimodal Haptic Effects from Sound for Gaming Experience EnhancementWe propose an algorithm that generates a vibration, an impact, or a vibration+impact haptic effect by processing a sound signal in real time. Our algorithm is selective in that it matches the most appropriate type of haptic effects to the sound using a machinelearning classifier (random forest) that is built on expert-labeled datasets. Our algorithm is tailored to enhance user experiences for video game play, and we present two examples for the RPG (roleplaying game) and FPS (first-person shooter) genres. We demonstrate the effectiveness of our algorithm by a user study in comparison to other state-of-the-art (SOTA) methods for the same crossmodal conversion. Our system elicits better multisensory user experiences than the SOTA algorithms for both game genres.2023GYGyeore Yun et al.POSTECHVibrotactile Feedback & Skin StimulationGame UX & Player BehaviorCHI
Generating Haptic Motion Effects for Multiple Articulated Bodies for Improved 4D Experiences: A Camera Space ApproachMotion effects are indispensable for improving 4D experiences in highly interactive applications, such as amusement parks, 4D theaters, and virtual reality games. Their recent emergence calls for effective algorithms generating motion effects synchronized with audiovisual content. This paper presents an automatic algorithm for synthesizing the object-based motion effects that express the movements of multiple articulated bodies inclusively {\hb when the objects' motion trajectories are available in the 3D camera space.} By taking the visual velocities and sizes of all object parts, our method computes a \textit{motion proxy} that represents the objects' movements by one point and converts the motion proxy to a motion command through a motion cueing algorithm. The motion proxy is determined by linearly combining the velocities, and its best combination was selected from several candidates by user studies. The results of user studies indicate that our algorithm can produce compelling object-based motion effects that enhance the multisensory experience.2023SHSangyoon Han et al.Pohang University of Science and Technology (POSTECH)Shape-Changing Interfaces & Soft Robotic MaterialsShape-Changing Materials & 4D PrintingCHI
Visuo-haptic Crossmodal Shape Perception Model for Shape-Changing Handheld Controllers Bridged by Inertial TensorWe present a visuo-haptic crossmodal model of shape perception designed for shape-changing handheld controllers. The model uses the inertia tensor of an object to bridge the two senses. The model was constructed from the results of three perceptual experiments. In the first two experiments, we validate that the primary moment and product of inertia (MOI and POI) in the inertia tensor have critical effects on the haptic perception of object length and asymmetry. Then, we estimate a haptic-to-visual shape matching model using MOI and POI as two link variables from the results of the third experiment for crossmodal magnitude production. Finally, we validate in a summative user study that the inverse of the shape matching model is effective for pairing a perceptually-congruent haptic object from a virtual object-the functionality we need for shape-changing handheld interfaces to afford perceptually-fulfilling sensory experiences in virtual reality.2023CPChaeyong Park et al.Pohang University of Science and TechnologyHaptic WearablesShape-Changing Interfaces & Soft Robotic MaterialsCHI
SleepGuru: Personalized Sleep Planning System for Real-life Actionability and NegotiabilityWidely-accepted sleep guidelines advise regular bedtimes and sleep hygiene. An individual's adherence is often viewed as a matter of self-regulation and anti-procrastination. We pose a question from a different perspective: What if it comes to a matter of one's social or professional duty that mandates irregular daily life, making it incompatible with the premise of standard guidelines? We propose SleepGuru, an individually actionable sleep planning system featuring one's real-life compatibility and extended forecast. Adopting theories on sleep physiology, SleepGuru builds a personalized predictor on the progression of the user's sleep pressure over a course of upcoming schedules and past activities sourced from her online calendar and wearable fitness tracker. Then, SleepGuru service provides individually actionable multi-day sleep schedules which respect the user's inevitable real-life irregularities while regulating her week-long sleep pressure. We elaborate on the underlying physiological principles and mathematical models, followed by 3-stage study and deployment. We develop a mobile user interface providing individual predictions and adjustability backed by cloud-side optimization. We deploy SleepGuru in-the-wild to 20 users for 8 weeks, where we found positive effects of SleepGuru in sleep quality, compliance rate, sleep efficiency, alertness, long-term followability, and so on.2022JLJungeun Lee et al.Sleep & Stress MonitoringElectronic Textiles (E-textiles)UIST
Vibration-Augmented Buttons: Information Transmission Capacity and Application to Interaction DesignOne can embed a vibration actuator to a physical button and augment the physical button's original kinesthetic response with a programmable vibration generated by the actuator. Such vibration-augmented buttons inherit the advantages of both physical and virtual buttons. This paper reports the information transmission capacity of vibration-augmented buttons. It was obtained by conducting a series of absolute identification experiments while increasing the number of augmented buttons. The information transmission capacity found was 2.6 bits, and vibration-augmented and physical buttons showed similar abilities in rendering easily recognizable haptic responses. In addition, we showcase a VR text entry application that utilizes vibration-augmented buttons. Our method provides several error messages to the user during text entry using a VR controller that includes an augmented button. We validate that the variable haptic feedback improves task performance, cognitive workload, and user experience for a transcription task.2022CPChaeyong Park et al.Pohang University of Science and TechnologyIn-Vehicle Haptic, Audio & Multimodal FeedbackVibrotactile Feedback & Skin StimulationCHI