'Show It, Don't Just Say It': The Complementary Effects of Instruction Multimodality for Software GuidanceDesigning adaptive tutoring systems for software learning presents challenges in determining appropriate instructional modalities. To inform the design of such systems, we conducted an observational study of ten human teacher-student pairs (N=20), where experienced design software users taught novices two new graphic design software features through multi-step procedures. These lessons were limited to three communication channels (speech, visual annotations, and remote screen control) to mimic possible AI tutor modalities. We found that annotations complement speech with spatial precision and remote control complements it with spatial and temporal precision but both of them cause intrusion to learner agency. Teachers adaptively select modalities to balance the need for instruction progress with students' cognitive engagement and sense of digital territory ownership. Our results provide further support to the contiguity principles and the value of agency in learning, while suggesting precision-agency trade-off and digital territoriality as new design constraints for adaptive software guidance.2026EPEmran Poh et al.Singapore Management UniversityIntelligent Tutoring Systems & Learning AnalyticsUser Research Methods (Interviews, Surveys, Observation)Prototyping & User TestingCHI
Interaction-Augmented Instruction: Modeling the Synergy of Prompts and Interactions in Human-GenAI CollaborationText prompt is the most common way for human-generative AI (GenAI) communication. Though convenient, it is challenging to convey fine-grained and referential intent. One promising solution is to combine text prompts with precise GUI interactions, like brushing and clicking. However, there lacks a formal model to capture synergistic designs between prompts and interactions, hindering their comparison and innovation. To fill this gap, via an iterative and deductive process, we develop the Interaction-Augmented Instruction (IAI) model, a compact entity–relation graph formalizing how the combination of interactions and text prompts enhances human-GenAI communication. With the model, we distill twelve recurring and composable atomic interaction paradigms from prior tools, verifying our model’s capability to facilitate systematic design characterization and comparison. Four usage scenarios further demonstrate the model’s utility in applying, refining, and innovating these paradigms. These results illustrate the IAI model’s descriptive, discriminative, and generative power for shaping future GenAI systems.2026LSLeixian Shen et al.The Hong Kong University of Science and TechnologyGenerative AI (Text, Image, Music, Video)Human-LLM CollaborationPrototyping & User TestingCHI
Challenges in Synchronous & Remote Collaboration Around VisualizationWe characterize 16 challenges faced by those investigating and developing remote and synchronous collaborative experiences around visualization. Our work reflects the perspectives and prior research efforts of an international group of 29 experts from across human-computer interaction and visualization sub-communities. The challenges are anchored around five collaborative activities that exhibit a centrality of visualization and multimodal communication. These activities include exploratory data analysis, creative ideation, visualization-rich presentations, joint decision making grounded in data, and real-time data monitoring. The challenges also reflect the changing dynamics of these activities in the face of recent advances in extended reality (XR) and artificial intelligence (AI). As an organizing scheme for future research at the intersection of visualization and computer-supported cooperative work, we align the challenges with a sequence of four sets of research and development activities: technological choices, social factors, AI assistance, and evaluation.2026MBMatthew Brehmer et al.University of WaterlooInteractive Data VisualizationRemote Work Tools & ExperienceMulti-User Large Display CollaborationCHI
BAIT: Visual-illusion-inspired Privacy Preservation for Mobile Data VisualizationWith the prevalence of mobile data visualizations, there have been growing concerns about their privacy risks, especially shoulder surfing attacks. Inspired by prior research on visual illusion, we propose BAIT, a novel approach to automatically generate privacy-preserving visualizations by stacking a decoy visualization over a given visualization. It allows visualization owners at proximity to clearly discern the original visualization and makes shoulder surfers at a distance be misled by the decoy visualization, by adjusting different visual channels of a decoy visualization (e.g., shape, position, tilt, size, color and spatial frequency). We explicitly model human perception effect at different viewing distances to optimize the decoy visualization design. Privacy-preserving examples and two in-depth user studies demonstrate the effectiveness of BAIT in both controlled lab study and real-world scenarios.2026SCSizhe Cheng et al.Nanyang Technological UniversityUncertainty VisualizationPrivacy by Design & User ControlPrivacy Perception & Decision-MakingCHI
Understanding the Feasibility of Auditory Hand-Steering Guidance for Blind and Low-Vision PeopleEveryday tasks like hand-washing and tea-making require people to steer their hands to use tools, navigating their hands to reach targets while avoiding hazards. Hand-steering becomes challenging when one cannot visually recognize if their hand is approaching the target and is away from hazards. Currently, no practical technological solutions support blind and low-vision (BLV) individuals' hand-steering. We designed and developed two auditory hand steering guidance methods: VERBAL and Follow-Your-Finger (FYF). VERBAL uses spoken directional instructions, while FYF uses sonification to guide hand-steering. We conducted a user study with 12 BLV participants to evaluate the feasibility of the methods in supporting hand-steering. VERBAL lacked precision, 24.6% error rate for one of the easiest conditions, but FYF showed promise, achieving 4.17% error rate for the same condition. Among the six participants who preferred FYF, the error rate was 1.39%. The results demonstrate the feasibility of auditory hand steering guidance for BLV individuals.2026YAYuki Abe et al.Hokkaido UniversityVibrotactile Feedback & Skin StimulationAudio Accessibility (Captions, Sign Language, Vibration)Motor Impairment Assistive Input TechnologiesCHI
Motor-Mediated Creativity: Bridging Embodied Skill Training and Digital ExpressionExpressive digital drawing requires nuanced motor control, subtle variations in pressure, velocity, and rhythm that convey affect and style. While experts develop this embodied fluency through years of practice, novices struggle to produce marks that match their intentions, creating a gap between vision and execution. We propose motor-mediated creativity: treating motor training as integral to digital expression. Our system, {\system}, instantiates this through structured practice of expressive primitives, expert-referenced feedback, and ideation prompts that encourage exploration. We report a two-stage investigation. A formative study characterized: (a) novice challenges in motor fluency, (b) examined how different feedback types, including corrective feedback, helped participants understand their mistakes, (c) how prompts, generic or embodied, support engagement with abstract expressive content. A controlled evaluation then linked fluency gains to subjective and expert ratings of expressiveness. Together, our findings show that scaffolding motor skills is a viable strategy for enhancing expressive agency in digital drawing.2026PBPasindu Bolonghege et al.University of MoratuwaHaptic WearablesShape-Changing Interfaces & Soft Robotic MaterialsPrototyping & User TestingCHI
Who You Explain To Matters: Learning by Explaining to Conversational Agents with Different Pedagogical RolesConversational agents are increasingly used in education for learning support. An application is "learning by explaining", where learners explain their understanding to an agent. However, existing research focuses on single roles, leaving it unclear how different pedagogical roles influence learners' interaction patterns, learning outcomes and experiences. We conducted a between-subjects study (N=96) comparing agents with three pedagogical roles (Tutee, Peer, Challenger) and a control condition while learning an economics concept. We found that different pedagogical roles shaped learning dynamics, including interaction patterns and experiences. Specifically, the Tutee agent elicited the most cognitive investment but led to high pressure. The Peer agent fostered high absorption and interest through collaborative dialogue. The Challenger agent promoted cognitive and metacognitive acts, enhancing critical thinking with moderate pressure. The findings highlight how agent roles shape different learning dynamics, guiding the design of educational agents tailored to specific pedagogical goals and learning phases.2026ZXZhengtao Xu et al.National University of SingaporeIntelligent Tutoring Systems & Learning AnalyticsHuman-LLM CollaborationConversational ChatbotsCHI
Rethinking Teaching Evaluation Reports: Designing AI-transformed Student Feedback for Instructor EngagementStudent evaluations of teaching (SETs) represent a valuable yet often underutilized resource, as many instructors struggle with the substantial time, cognitive, and emotional demands of processing this feedback effectively. While these evaluations contain crucial insights into students' learning experiences that could enhance instruction, their potential remains largely untapped. Our work explores how to redesign SET reports using language models (LMs) to distill, highlight, and present student feedback in more engaging and actionable ways. We systematically explored a $4 \times 4$ strategy-presentation design space, creating six representative mock-ups that integrate different analytical strategies with various presentation formats. Through interviews with 16 post-secondary instructors, we learned how and when they engage with current SETs, and how they would perceive and use the LM-augmented redesigned SET mock-ups. Our findings revealed that instructors' preferences for different redesigns aligned with distinct goals: whether improving their teaching practices, gaining quick insights into their teaching effectiveness, or preparing summative teaching performance reports. These findings shed light on new opportunities for designing dynamic SET systems where AI can adaptively process and present feedback based on instructors' specific needs and contexts.2025RSRuoxi Shang et al.Technology Use in Higher EducationCSCW
EmoShortcuts: Emotionally Expressive Body Augmentation for Social Mixed Reality AvatarsWe present EmoShortcuts, a novel social Mixed Reality (MR) framework that enhances emotional expression by dynamically augmenting avatar body gestures to reflect users’ emotional states. While social MR enables immersive remote interactions through avatars, conveying emotions remains challenging due to limitations in head-mounted display (HMD) tracking (e.g., missing lower-body movements like stomping or defensive postures) and users' tendency to deprioritize nonverbal expression during multitasking. EmoShortcuts addresses these challenges by introducing an augmentation framework that generates expressive body gestures even when users' physical movements are restricted. We conducted a formative study with 12 participants to identify key challenges in emotional expression and explore user preferences for AI-assisted gesture augmentation. Based on these insights, we designed an interface that enables adaptive gesture augmentation and allows for both pre-set and real-time user control. Through an extensive user study (n = 16), our findings show that EmoShortcuts significantly improves emotion expression accuracy and reduces cognitive workload, demonstrating its potential for more immersive and emotionally rich virtual communication.2025HSHyunA Seo et al.Mixed Reality WorkspacesIdentity & Avatars in XRUIST
Cracking Aegis: An Adversarial LLM-based Game for Raising Awareness of Vulnerabilities in Privacy ProtectionTraditional methods for raising awareness of privacy protection often fail to engage users or provide hands-on insights into how privacy vulnerabilities are exploited. To address this, we incorporate an adversarial mechanic in the design of the dialogue-based serious game Cracking Aegis. Leveraging LLMs to simulate natural interactions, the game challenges players to impersonate characters and extract sensitive information from an AI agent, Aegis. A user study (n=22) revealed that players employed diverse deceptive linguistic strategies, including storytelling and emotional rapport, to manipulate Aegis. After playing, players reported connecting in-game scenarios with real-world privacy vulnerabilities, such as phishing and impersonation, and expressed intentions to strengthen privacy control, such as avoiding oversharing personal information with AI systems. This work highlights the potential of LLMs to simulate complex relational interactions in serious games, while demonstrating how an adversarial strategy provides a unique perspective in designing for social good, particularly in privacy protection.2025JFJiaying Fu et al.Serious & Functional GamesPrivacy Perception & Decision-MakingDark Patterns RecognitionDIS
Enhancing Deliberativeness: Evaluating the Impact of Multimodal Reflection NudgesNudging participants with text-based reflective nudges enhances deliberation quality on online deliberation platforms. The effectiveness of multimodal reflective nudges, however, remains largely unexplored. Given the multi-sensory nature of human perception, incorporating diverse modalities into self-reflection mechanisms has the potential to better support various reflective styles. This paper explores how presenting reflective nudges of different types (direct: persona and indirect: storytelling) in different modalities (text, image, video and audio) affects deliberation quality. We conducted two user studies with 20 and 200 participants respectively. The first study identifies the preferred modality for each type of reflective nudges, revealing that text is most preferred for persona and video is most preferred for storytelling. The second study assesses the impact of these modalities on deliberation quality. Our findings reveal distinct effects associated with each modality, providing valuable insights for developing more inclusive and effective online deliberation platforms.2025SYShunYi Yeo et al.Singapore University of Technology and DesignParticipatory DesignInteractive Narrative & Immersive StorytellingCHI
Prompting an Embodied AI Agent: How Embodiment and Multimodal Signaling Affects Prompting BehaviourCurrent voice agents wait for a user to complete their verbal instruction before responding; yet, this is misaligned with how humans engage in everyday conversational interaction, where interlocutors use multimodal signaling (e.g. nodding, grunting, or looking at referred to objects) to ensure conversational grounding. We designed an embodied VR agent that exhibits multimodal signaling behaviors in response to situated prompts, by turning its head, or by visually highlighting objects being discussed or referred to. We explore how people prompt this agent to design and manipulate the objects in a VR scene. Through a Wizard of Oz study, we found that participants interacting with an agent that indicated its understanding of spatial and action references were able to prevent errors 30% of the time, and were more satisfied and confident in the agent's abilities. These findings underscore the importance of designing multimodal signalling communication techniques for future embodied agents.2025TZTianyi Zhang et al.Singapore Management UniversityFull-Body Interaction & Embodied InputVoice User Interface (VUI) DesignSocial & Collaborative VRCHI
"Ronaldo's a poser!": How the Use of Generative AI Shapes Debates in Online ForumsOnline debates can enhance critical thinking but may escalate into hostile attacks. As humans are increasingly reliant on Generative AI (GenAI) in writing tasks, we need to understand how people utilize GenAI in online debates. To examine the patterns of writing behavior while making arguments with GenAI, we created an online forum for soccer fans to engage in turn-based and free debates in a post format with the assistance of ChatGPT, arguing on the topic of "Messi vs Ronaldo". After 13 sessions of two-part study and semi-structured interviews with 39 participants, we conducted content and thematic analyses to integrate insights from interview transcripts, ChatGPT records, and forum posts. We found that participants prompted ChatGPT for aggressive responses, created posts with similar content and logical fallacies, and sacrificed the use of ChatGPT for better human-human communication. This work uncovers how polarized forum members work with GenAI to engage in debates online.2025YZYuhan Zeng et al.City University of Hong Kong, Department of Computer ScienceGenerative AI (Text, Image, Music, Video)Social Platform Design & User BehaviorMisinformation & Fact-CheckingCHI
“I can run at night!": Using Augmented Reality to Support Nighttime Guided Running for Low-vision RunnersDark environment challenges low-vision (LV) individuals to engage in running by following sighted guide—a Caller-style guided running—due to insufficient illumination, because it prevents them from using their residual vision to follow the guide and be aware about their environment. We design, develop, and evaluate RunSight, an augmented reality (AR)-based assistive tool to support LV individuals to run at night. RunSight combines see-through HMD and image processing to enhance one's visual awareness of the surrounding environment (e.g., potential hazard) and visualize the guide's position with AR-based visualization. To demonstrate RunSight's efficacy, we conducted a user study with 8 LV runners. The results showed that all participants could run at least 1km (mean = 3.44 km) using RunSight, while none could engage in Caller-style guided running without it. Our participants could run safely because they effectively synthesized RunSight-provided cues and information gained from runner-guide communication.2025YAYuki Abe et al.Hokkaido University, Human-Computer Interaction LabAR Navigation & Context AwarenessVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)CHI
Cyberoception: Finding A Painlessly-Measurable New Sense In The Cyberworld Towards Emotion-awareness In ComputingIn Affective computing, recognizing users' emotions accurately is the basis of affective human–computer interaction. Understanding users' interoception contributes to a better understanding of individually different emotional abilities, which is essential for achieving inter-individually accurate emotion estimation. However, existing interoception measurement methods, such as the heart rate discrimination task, have several limitations, including their dependence on a well-controlled laboratory environment and precision apparatus, making monitoring users' interoception challenging. This study aims to determine other forms of data that can explain users' interoceptive or similar states in their real-world lives and propose a novel hypothetical concept "cyberoception," a new sense (1) which has properties similar to interoception in terms of the correlation with other emotion-related abilities, and (2) which can be measured only by the sensors embedded inside commodity smartphone devices in users' daily lives. Results from a 10-day-long in-lab/in-the-wild hybrid experiment reveal a specific cyberoception type "Turn On." (users' subjective sensory perception about the frequency of turning-on behavior on their smartphones)2025TOTadashi Okoshi et al.Keio University, Faculty of Environment and Information StudiesBrain-Computer Interface (BCI) & NeurofeedbackBiosensors & Physiological MonitoringCHI
SmarTeeth: Augmenting Manual Toothbrushing with In-ear MicrophonesImproper toothbrushing practices persist as a primary cause of oral health issues such as tooth decay and gum disease. Despite the availability of high-end electric toothbrushes that offer some guidance, manual toothbrushes remain widely used due to their simplicity and convenience. We present SmarTeeth, an earable-based toothbrushing monitoring system designed to augment manual toothbrushing with functionalities typically offered only by high-end electric toothbrushes, such as brushing surface tracking. The underlying idea of SmarTeeth is to leverage in-ear microphones on earphones to capture toothbrushing sounds transmitted through the oral cavity to ear canals through facial bones and tissues. The distinct propagation paths of brushing sounds from various dental locations to each ear canal provide the foundational basis for our methods to accurately identify different brushing locations. By extracting customized features from these sounds, we can detect brushing locations using a deep-learning model. With only one registration session (~2 mins) for a new user, the average accuracy is 92.7% for detecting six regions and 75.6% for sixteen tooth surfaces. With three registration sessions (~6 mins), the performance can be boosted to 98.8% and 90.3% for six-region and sixteen-surface tracking, respectively. A key advantage of using earphones for monitoring is that they provide natural auditory feedback to alert users when they are overbrushing or underbrushing. Comprehensive evaluation validates the effectiveness of SmarTeeth under various conditions (different users, brushes, orders, noise, etc.), and the feedback from the user study (N=13) indicates that users found the system highly useful (6.0/7.0) and reported a low workload (2.5/7.0) while using it. Our findings suggest that SmarTeeth could offer a scalable and effective solution to improve oral health globally by providing manual toothbrush users with advanced brushing monitoring capabilities.2025QYQiang Yang et al.University of Cambridge, Department of Computer Science and TechnologyBiosensors & Physiological MonitoringElectronic Textiles (E-textiles)Context-Aware ComputingCHI
W4-Groups: Modeling the Who, What, When and Where of Group Behavior via Mobility SensingHuman social interactions occur in group settings of varying sizes and locations, depending on the type of social activity. The ability to distinguish group formations based on their purposes transforms how group detection mechanisms function. Not only should such tools support the effective detection of serendipitous encounters, but they can derive categories of relation types among users. Determining who is involved, what activity is performed, and when and where the activity occurs are critical to understanding group processes in greater depth, including supporting goal-oriented applications (e.g., performance, productivity, and mental health) that require sensing social factors. In this work, we propose W4-Groups that captures the functional perspective of variability and repeatability when automatically constructing short-term and long-term groups via multiple data sources (e.g., WiFi and location check-in data). We design and implement W4-Groups to detect and extract all four group features who-what-when-where from the user’s daily mobility patterns. We empirically evaluate the framework using two real-world WiFi datasets and a location check-in dataset, yielding an average of 92% overall accuracy, 96% precision, and 94% recall. Further, we supplement two case studies to demonstrate the application of W4-Groups for next-group activity prediction and analyzing changes in group behavior at a longitudinal scale, exemplifying short-term and long-term occurrences.2024AAAkanksha Atrey et al.Session 3f: Enhancing Virtual Presence and InteractionCSCW
ClearSpeech: Improving Voice Quality of Earbuds Using Both In-Ear and Out-Ear MicrophonesMa 等人提出 ClearSpeech 系统，结合耳塞的内耳和外耳双麦克风进行语音增强，在嘈杂环境中可将语音质量提升 40%，显著改善真无线耳塞的通话体验。2024DMDong Ma et al.Haptic WearablesVoice User Interface (VUI) DesignUbiComp
BreathPro: Monitoring Breathing Mode during Running with EarablesHu 等人开发 BreathPro 系统，利用耳穿戴设备传感器实时监测跑步时的呼吸模式，为运动健康监测提供新方案。2024CHChangshuo Hu et al.Fitness Tracking & Physical Activity MonitoringBiosensors & Physiological MonitoringUbiComp
Conversational Localization: Indoor Human Localization through Intelligent ConversationSheshadri 等人提出 Conversational Localization 方法，通过智能对话交互实现室内人体定位。2024SSSmitha Sheshadri et al.Multilingual & Cross-Cultural Voice InteractionAR Navigation & Context AwarenessUbiComp