From Text to Movement: LLM-driven Swarm User Interfaces for Embodied and Interactive StorytellingThis paper introduces PuppetLine, an interactive storytelling system that translates natural language narratives into coordinated performances using tabletop robots. The system combines large language models with a constrained set of action and emotion primitives to generate physically executable and interpretable multi-robot motions. We describe the system design and report findings from a designer-centered user study examining how people interpret and reflect on robot enactments. The results provide formative insights into mapping narrative intent to embodied interaction and inform the design of narrative Swarm User Interfaces.2026RWRuhan Wang et al.Tsinghua University3D Modeling & AnimationInteractive Narrative & Immersive StorytellingHuman-Robot Collaboration (HRC)IUI
DuoMorph: Synergistic Integration of FDM Printing and Pneumatic Actuation for Shape-Changing InterfacesWe introduce DuoMorph, a design and fabrication method that synergistically integrates Fused Deposition Modeling(FDM) printing and pneumatic actuation to create novel shape-changing interfaces. In DuoMorph, the printed structures and heat-sealed pneumatic elements are mutually designed to actuate and constrain each other, enabling functions that are difficult for either component to achieve in isolation. Moreover, the entire hybrid structure can be fabricated through a single, seamless process using only a standard FDM printer—including both heat-sealing and 3D/4D printing. In this paper, we define a design space including four primitive categories that capture the fundamental ways in which printed and pneumatic components can interact. To support this process, we present a fabrication method and an accompanying design tool. Finally, we demonstrate the potential of DuoMorph through example applications and performance demonstrations.2026XLXueqing Li et al.Tsinghua universityShape-Changing Interfaces & Soft Robotic MaterialsShape-Changing Materials & 4D PrintingCHI
AI-generated AR Reassembly Guidance from Disassembly Videos to Scaffold Everyday RepairRepair is a valuable yet challenging activity, especially when product manuals are missing or outdated. Augmented Reality (AR) has been widely explored for repair tasks, but most systems rely on CAD models or pre-constructed assets, which escalate authoring costs and constrain scalability. We introduce RePairAR, a system that leverages multimodal large language models (MLLMs) to generate interactive AR reassembly guidance derived directly from user-recorded egocentric disassembly videos. RePairAR deduces step-part-relation structures, reverses these for reassembly planning, and delivers the guidance through mixed-media AR visualizations. In a user study with repair novices, RePairAR significantly reduced perceived temporal demand compared to traditional how-to videos. Both media improved self-efficacy, with RePairAR providing greater gains. Follow-up interviews revealed the mechanisms behind these effects. We contribute a validated MLLM-driven pipeline and highlight design implications for scalable, situated support in everyday repair practices.2026WDWenjing Deng et al.Tsinghua UniversityAR Navigation & Context AwarenessMixed Reality WorkspacesHuman-LLM CollaborationCHI
GestuProp: 3D Virtual Reality Prop Generation with Co-Speech GesturesVirtual Reality (VR) has been widely adopted in domains such as gaming, education, and healthcare, where 3D props play a central role in enabling immersive interaction. With the advancement of generative AI, 3D props can now be created rapidly; however, little research has explored how gestures and speech can be integrated to support prop generation. To address this gap, we introduce GestuProp, a VR prop generation system driven by co-speech gestures. Building on a formative study with 30 participants, we proposed a gesture design space and developed the VR system GestuProp. We then conducted a user study with 14 participants, which showed that GestuProp demonstrates good usability and favorable user experiences, while also revealing how object categories influence gesture use and interaction. These findings highlight the potential of gesture–speech synergy to advance prop generation in VR.2026ZYZhihao Yao et al.Tsinghua UniversitySocial & Collaborative VR3D Modeling & AnimationHand Gesture RecognitionCHI
Designing for Long-Term Emotion Regulation: A Breathing Biofeedback Game for Women in Compulsory Isolation Drug Rehabilitation CentersIn the rehabilitation process of women with substance use disorders, difficulties in emotion regulation represent a major risk factor for relapse, while shame, low self-esteem, and cognitive impairments further undermine their ability to learn and sustain emotion regulation skills. To address the lack of research on long-term skill transfer among marginalized populations in closed environments, we designed and evaluated a seven-day phased breathing biofeedback game based on low-cost audio input. The game was intended to support skill acquisition and transfer, strengthen self-efficacy and self-acceptance, and sustain attention and long-term engagement. In a six-week controlled study involving 60 participants, the results demonstrated that the game not only maintained high levels of engagement but also effectively improved breathing skills, facilitated their transfer into daily life, and alleviated negative emotions. Finally, we discussed and reflected on five design implications that emerged from these findings.2026QCQi Chen et al.Wuhan Textile UniversityAffective Feedback & Emotion Regulation InterfacesHealth Self-TrackingBehavior Change & Reflection TechnologyCHI
MorphingSkin: A Skin-like Platform that Integrates Multimodal Hydraulic Actuators Based on Flexible Electroosmotic PumpsInteractive surfaces have garnered significant attention in Human-Computer Interaction, with fluid-driven actuators being a promising actuation technology due to their flexible form factors and multimodal interactivities. However, traditional fluid-driven systems typically rely on bulky and noisy electromechanical hardware, limiting their portability and practicality. While recent work has introduced compact hydraulic actuators like electroosmotic pumps (EOPs) in haptic devices, their potential for building multifunctional interactive surfaces remains largely unexplored. In this work, we present MorphingSkin, a skin-like platform that integrates multiple, multimodal hydraulic actuators using flexible EOPs as lightweight and self-contained fluidic actuators. We introduce the architecture of MorphingSkin and its versatile design space for multimodal actuation in force, shape, optical, and weight domains. We demonstrate interactive and robotic applications that integrate multiple actuators working collectively within a single MorphingSkin device. Through this work, we envision the future of using MorphingSkin technology for interactive surfaces that integrate flexible form factors and multimodal actuation capabilities.2025TYTianyu Yu et al.Haptic WearablesShape-Changing Interfaces & Soft Robotic MaterialsUIST
Outline and Detail: A Semantic-Driven Framework for Layered 2D Character Generation2D cartoon-style digital characters represent an important art form in games, animation, and virtual live streaming. However, traditional 2D character creation workflows involve tedious manual layering, complex skeleton rigging, and professional animation skills, posing challenges for independent studios and non-professional users. While existing AI generation technologies can quickly create visual content, they typically produce non-layered, difficult-to-edit composite images that cannot be integrated into current workflows. This paper presents Spiritus, a semantic-driven 2D character generation framework. Unlike existing text-based AI animation workflows, Spiritus integrates mixed text and sketch inputs, achieving character image generation and automatic component segmentation through an open mask library and semantic matching. We validated the system's effectiveness in character generation freedom, character animation quality, and technical barrier reduction through comparative evaluation of user experiment results. Finally, we explored the possibilities of applying generated characters to various workflows and scenarios, including game development, animation production, and interactive illustrations.2025QSQirui Sun et al.Music Composition & Sound Design ToolsVideo Production & Editing3D Modeling & AnimationUIST
A Card-based Co-Design Toolkit for Exploring Smart Material Applications with Multiple Stakeholders: A Case Study on Automotive Interior DesignSmart materials have garnered significant attention in both academia and industry, yet identifying pragmatically impactful applications still requires contributions from multiple stakeholders, including researchers, designers, and industry professionals. Although previous research has explored novel technical approaches or user-centered applications of smart materials, this study focuses on how to stimulate effective dialogue among stakeholders to explore impactful smart material applications. To this end, we provided a card-based co-design toolkit, tailored to automotive interior design as a case study. We conducted a multi-stakeholder co-design workshop to examine the performance and reveal the benefits of employing the toolkit in the co-design process. The workshops resulted in 16 concept designs. Qualitative interviews further revealed that, using our toolkit, the co-design process effectively fostered mutual understanding among stakeholders, enhanced both creativity and depth throughout the design process, and provided practical insights for each stakeholder's future work.2025TYTianyu Yu et al.Shape-Changing Interfaces & Soft Robotic MaterialsDIS
Exploring the Design of LLM-based Agent in Enhancing Self-disclosure Among the Older AdultsSocial difficulties have become an increasingly serious issue among older adults. For older adults, regular self-disclosure is essential for maintaining mental health and building close relationships. Leveraging conversational agents to encourage self-disclosure in older adults has shown increasing potential. Understanding how LLM-based agents can influence and stimulate self-disclosure across different topics is crucial for designing future agents tailored to older users. This study introduces Disclosure-Agent, an LLM-based conversational agent, and examines its impact on self-disclosure in older adults through a user study involving 20 participants, 8 topics, and two interactive interfaces equipped with Disclosure-Agent. The findings provide valuable insights into how LLM-based agents can promote self-disclosure in older adults and offer design recommendations for future elderly-oriented conversational agents.2025YGYijie Guo et al.Tsinghua University, Academy of Arts and Design; Tsinghua University, The Future LaboratoryAgent Personality & AnthropomorphismHuman-LLM CollaborationCHI
Mul-O: Encouraging Olfactory Innovation in Various Scenarios Through a Task-Oriented Development PlatformOlfactory interfaces are pivotal in HCI, yet their development is hindered by limited application scenarios, stifling the discovery of new research opportunities. This challenge primarily stems from existing design tools focusing predominantly on odor display devices and the creation of standalone olfactory experiences, rather than enabling rapid adaptation to various contexts and tasks. Addressing this, we introduce Mul-O, a novel task-oriented development platform crafted to aid semi-professionals in navigating the diverse requirements of potential application scenarios and effectively prototyping ideas. Mul-O facilitates the swift association and integration of olfactory experiences into functional designs, system integrations, and concept validations. Comprising a web UI for task-oriented development, an API server for seamless third-party integration, and wireless olfactory display hardware, Mul-O significantly enhances the ideation and prototyping process in multisensory tasks. This was verified by a 15-day workshop attended by 30 participants. The workshop produced seven innovative projects, underscoring Mul-O's efficacy in fostering olfactory innovation.2024PGPeizhong Gao et al.Haptic WearablesShape-Changing Interfaces & Soft Robotic MaterialsContext-Aware ComputingUIST
Lumina: A Software Tool for Fostering Creativity in Designing Chinese Shadow PuppetsShadow puppetry, a culturally rich storytelling art, faces challenges transitioning to the digital realm. Creators in the early design phase struggle with crafting intricate patterns, textures, and basic animations while adhering to stylistic conventions - hindering creativity, especially for novices. This paper presents Lumina, a tool to facilitate the early Chinese shadow puppet design stage. Lumina provides contour templates, animations, scene editing tools, and machine-generated traditional puppet patterns. These features liberate creators from tedious tasks, allowing focus on the creative process. Developed based on a formative study with puppet creators, the web-based Lumina enables wide dissemination. An evaluation with 18 participants demonstrated Lumina's effectiveness and ease of use, with participants successfully creating designs spanning traditional themes to contemporary and science-fiction concepts.2024ZYZhihao Yao et al.Museum & Cultural Heritage DigitizationInteractive Narrative & Immersive StorytellingUIST
InkBrush: A Sketching Tool for 3D Ink PaintingInkBrush is a new sketch-based 3D drawing tool for creating 3D ink paintings using free-form 3D ink strokes. It offers a digital calligraphy brush and various editing tools to generate realistic ink-like brush strokes with attributes like hairy edges, ink drips, and scattered dots. Users can adjust parameters such as moisture, color, darkness, dryness, and stroke style to customize the appearance of the brush strokes. The development of InkBrush was guided by a design study involving artists and designers. It was developed as a plugin for Blender, a popular 3D modeling tool, and its effectiveness and usability were evaluated through a user study involving 75 participants. Preliminary feedback from the participants was overwhelmingly positive, indicating that InkBrush was intuitive and easy to use. Following this, we also sought in-depth assessments from experts in ink painting and 3D design. Their evaluations further demonstrated the effectiveness of InkBrush.2024ZYZhihao Yao et al.Tsinghua University3D Modeling & AnimationCustomizable & Personalized ObjectsCHI
Co-designing the Collaborative Digital Musical Instruments for Group Music TherapyDigital Musical Instruments (DMIs) have been integrated into group music therapy, providing therapists with alternative ways to engage in musical dialogues with their clients. However, existing DMIs used in group settings are primarily designed for individual use and often overlook the social dynamics inherent in group therapy. Recognizing the crucial role of social interaction in the effectiveness of group therapy, we argue that Collaborative Digital Musical Instruments (CDMIs), seamlessly integrating social interaction with musical expression, hold significant potential to enhance group music therapy. To better tailor CDMIs for group music therapy, we engaged in a co-design process with music therapists, designing and practicing group therapy sessions involving the prototype ComString. In the end, we reflected on the co-design case to suggest future directions for designing CDMIs in group music therapy.2024YFYuan-Ling Feng et al.Tsinghua UniversityVibrotactile Feedback & Skin StimulationConversational ChatbotsMusic Composition & Sound Design ToolsCHI
Sustainflatable: Harvesting, Storing and Utilizing Ambient Energy for Pneumatic Morphing InterfacesWhile the majority of pneumatic interfaces are powered and controlled by traditional electric pumps and valves, alternative sustainable energy-harnessing technology has been attracting attention. This paper presents a novel solution to this challenge with the development of the Sustainflatable system, a self-sustaining pneumatic system that can harvest renewable energy sources such as wind, water flow, moisture, and sunlight, convert the energy into compressed air, and store it for later use in a programmable and intelligent way. The system is completely electronic-free, incorporating customized energy harvesting pumps, storage units with variable volume-pressure characteristics, and tailored valves that operate autonomously. Additionally, the paper provides a design tool to guide the development of the system and includes several environmental applications to showcase its capabilities.2023QLQiuyu Lu et al.Shape-Changing Interfaces & Soft Robotic MaterialsEcological Design & Green ComputingUIST
NaCanva: Exploring and Enabling the Nature-Inspired Creativity for ChildrenNature has been a bountiful source of materials, replenishment, inspiration, and creativity. Nature collage, as a crafting technique, offers children a fun and educational way to explore nature and express their creativity. However, the collection of raw material has been limited to static objects like leaves, ignoring inspiration from nature’s sounds and dynamic elements such as babbling creeks. To address this limitation, we have developed a mobile application with the aim of encouraging children’s creativity through renewed material collection and careful observation in nature. To explore the possibility of this approach, we conducted a formative study with children (N=20) and a design workshop with experts (N=6). With the results of these studies, we formulate NaCanva, an AI-assisted multi-modal collage creation system for children. Drawing upon the interactive relationship between children and nature, NaCanva facillitates a multi-modal material collection, including images, sound, and videos, which differs our system from traditional collages. We validated this system with a between-subject user study (N =30), and the results indicated that NaCanva enhances children’s multidimensional observation and engagement with nature, thereby unleashing their creativity in the creation of nature collages.2023ZYZihan Yan et al.Generative AI (Text, Image, Music, Video)Digital Art Installations & Interactive PerformanceFood Culture & Food InteractionMobileHCI
Thermotion: Design and fabrication of thermofluidic composites for animation effects on object surfacesWe introduce Thermotion, a novel method using thermofluidic composites to design and display thermochromic animation effects on object surfaces. With fluidic channels embedded under the object surfaces, the composites utilize thermofluidic flows to dynamically control the surface temperature as an actuator for thermochromic paints, which enables researchers and designers for the first time to create animations not only on two and three-dimensional surfaces but also on the surface made of a few flexible everyday materials. We report the design space with six animation primitives and two modification effects, and we demonstrate the design and fabrication workflow with a customized software platform for design and simulation. A range of applications is shown leveraging the objects' dynamic displays both visually and thermally, including dynamic artifacts, teaching aids, and ambient displays. We envision an opportunity to extend thermofluidic composites to other heat-related practices for further dynamic and programmable interactions with temperature.2023TYTianyu Yu et al.Tsinghua UniversityShape-Changing Interfaces & Soft Robotic MaterialsShape-Changing Materials & 4D PrintingCHI
Digital Making for Inheritance and Enlivening Intangible Cultural Heritage: A Case of Hairy Monkey HandicraftsDigital technologies can conduct an important role in preserving intangible cultural heritage (ICH). Nonetheless, existing work tends to be limited to digital storage, presentation, dissemination, and education, with comparatively little concentrating on production and reproduction of the craft, the key to revitalizing ICH. In this paper, we explore digital making as an approach for both the inheritance and innovation of ICH handicrafts. Taking Hairy Monkey craftsmanship as an instance, we conducted a workshop with 15 groups of makers, teaching them the traditional Hairy Monkey craft and subsequently enabling them to create innovative works with digital technologies in their own time. As revealed by our interviews with these participants, ICH brings cultural inspiration to digital making, and digital making rejuvenates ICH through innovative art forms and its positive influence on participants. As demonstrated in this paper, digital technologies can be deeply integrated with ICH through making to revitalize ICH from the core through living transmission.2023GLGuanhong Liu et al.Tsinghua UniversityDesktop 3D Printing & Personal FabricationMuseum & Cultural Heritage DigitizationCHI
LaserShoes: Low-Cost Ground Surface Detection Using Laser Speckle ImagingGround surfaces are often carefully designed and engineered with various textures to fit the functionalities of human environments and thus could contain rich context information for smart wearables. Ground surface detection could power a wide array of applications including activity recognition, mobile health, and context-aware computing, and potentially provide an additional channel of information for many existing kinesiology approaches such as gait analysis. To facilitate the detection of ground surfaces, we present LaserShoes, a texture-sensing-enabled system using laser speckle imaging that can be retrofitted to shoes. Our system captures videos of speckle patterns induced on ground surfaces and uses pre-processing to identify ideal images with clear speckle patterns collected when users' feet are in contact with ground surfaces. We demonstrated our technique with a ResNet-18 model and achieved real-time inference. We conducted an evaluation in different conditions and demonstrated results that verified the feasibility.2023ZYZihan Yan et al.MIT Media LabFull-Body Interaction & Embodied InputBiosensors & Physiological MonitoringContext-Aware ComputingCHI
Reviewing and Reflecting Smart Home Research from the Human-Centered PerspectiveWhile there has been rapid growth in smart home research from a technical perspective – focusing on home automation, devices, software, and protocols – few review papers examine the human-centered perspective. A human-centered focus is crucial for achieving the goals of providing natural, convenient, comfortable, friendly, and safe user experiences in the smart home. To understand key innovations in human-centered smart home research, we analyzed keyword changes over time via 19,091 papers from 2000 to 2022, then selected 55 papers from high-impact venues in the last five years, and summarized them through a combination of qualitative and quantitative methods. Our analysis revealed five research trends with unique characteristics and interdependence. Drawing on this review, we elaborate on the future of smart home design research with respect to multidisciplinary development, stakeholder involvement, and the shift of design implications.2023YYYuan Yao et al.School of Architecture and Design, Academy of Arts & DesignUniversal & Inclusive DesignSmart Home Interaction DesignCHI
Establishing Design Consensus toward Next-Generation Retail: Data-Enabled Design Exploration and Participatory AnalysisIntegration of online and offline retail faces challenges in technology adoption, interaction style evolution and customer behavior shifts, while also being complicated by diverse perspectives from different stakeholders of consumers, retail staff, and retail business unit people. To explore how we can tackle the aforementioned challenges, this work applied the data-enabled design method and participatory data analysis to a case study, where 400 student consumers' shopping behavior data was collected, cross-analyzed, and visualized in a campus chain store. We then invited 13 stakeholders to join a co-creation workshop for a further participatory data analysis. In the workshop, the different stakeholders came to a design consensuses which we summarized into a series of practical design recommendations for improving the current store. Finally, we generalized the case study process as a contextual-informed-aware model, which can contribute to professional design practice for the retail industry.2022YYYuan Yao et al.Tsinghua UniversityInteractive Data VisualizationParticipatory DesignPrototyping & User TestingCHI