Towards Unobtrusive Physical AI: Augmenting Everyday Objects with Intelligence and Robotic Movement for Proactive AssistanceUsers constantly interact with physical, most often passive, objects. Consider if familiar objects instead proactively assisted users, e.g., a stapler moving across the table to help users organize documents, or a knife moving away to prevent injury as the user is inattentively about to lean against the countertop. In this paper, we build on the qualities of tangible interaction and focus on recognizing user needs in everyday tasks to enable ubiquitous yet unobtrusive tangible interaction. To achieve this, we introduce an architecture that leverages Large Language Models~(LLMs) to perceive users’ environment and activities, perform spatial-temporal reasoning, and generate object actions aligned with inferred user intentions and object properties. We demonstrate the system’s utility providing proactive assistance with multiple objects and in various daily scenarios. To evaluate our system components, we compare our system-generated output for user goal estimation and object action recommendation with human-annotated baselines, with results indicating good agreement.2025VHViolet Yinuo Han et al.Ubiquitous ComputingCommunity Engagement & Civic TechnologyUIST
CompAct: Designing Interconnected Compliant Mechanisms with Targeted Actuation TransmissionsCompliant mechanisms enable the creation of compact and easy-to-fabricate devices for tangible interaction. This work explores interconnected compliant mechanisms consisting of multiple joints and rigid bodies to transmit and process displacements as signals that result from physical interactions. As these devices are difficult to design due to their vast and complex design space, we developed a graph-based design algorithm and computational tool to help users program and customize such computational functions and procedurally model physical designs. When combined with active materials with actuation and sensing capabilities, these devices can also render and detect haptic interaction. Our design examples demonstrate the tool’s capability to respond to relevant HCI concepts, including building modular physical interface toolkits, encrypting tangible interactions, and customizing user augmentation for accessibility. We believe the tool will facilitate the generation of new interfaces with enriched affordance.2025HYHumphrey Yang et al.Carnegie Mellon University, Human-Computer Interaction InstituteShape-Changing Interfaces & Soft Robotic MaterialsCustomizable & Personalized ObjectsCHI
Dancing Delicacies: Designing Computational Food for Dynamic Dining Trajectories Contemporary human-food interaction design is often a technology-driven endeavor in which food’s materiality has been largely underexplored. Building on the concept of “computational food”, this paper explores the design of food as a material realization of computation through a material-centered approach. We engaged with a “Research through Design” exploration by designing a computational food system called “Dancing Delicacies”, which enables food items to be “programmed” and “reconfigured” within dynamic trajectories. Our practice led to a design framework resulting in four original dish designs. Our dishes aim to illustrate the richness of this new design space for computational food. Furthermore, through engaging with expert practitioners from the hospitality industry, we provide a first account of understanding the design of computational food for dynamic dining trajectories and its speculative use contexts in the industry. With this work, we hope to inspire researchers and designers to envision a new future of human-food interaction.2023JDJialin Deng et al.Laser Cutting & Digital FabricationFood Culture & Food InteractionDIS
FlexTure: Designing Dynamic and Configurable Surface FeaturesWe present FlexTure, a method for creating pop-up kirigami structures with a selectively bonded bilayer. These surfaces enable a new design space for accessible and rapid prototyping of dynamic surfaces. Using a flexible material selectively attached to a stretched substrate, we can create metamaterial surfaces that change texture. The tactile and aesthetic effects of these surfaces can be tuned through the configuration of cuts in the top layer of material, as well as the selection of the layers themselves. We provide a design workflow and accessible methods to achieve target effects and experimentally measure some mechanical properties of the surfaces. Several application concepts are offered along with a computational design tool.2023TJTate Johnson et al.Shape-Changing Interfaces & Soft Robotic MaterialsShape-Changing Materials & 4D PrintingDIS
EpoMemory: Multi-state Shape Memory for Programmable Morphing InterfacesSmart shape-changing materials can be adapted to different usages, which have been leveraged for dynamic affordances and on-demand haptic feedback in HCI. However, the applicability of these materials is often bottlenecked by their complex fabrication and the challenge of programming localized and individually addressable responses. In this work, we propose a toolkit for designing and fabricating programmable morphing objects using off-the-shelf epoxies. Our method involves varying the crosslinker to epoxy resin ratio to control morphing temperatures from 40 ℃ to 90 ℃, either across different regions of a shape memory device or across devices. Functional components (e.g., conductive fabric, magnetic particles) are also incorporated with the epoxy for sensing and active reconfiguration. A toolbox of fabrication methods and a primitive design library are introduced to support design ideation and programmable morphing. Finally, we demonstrate application examples, including morphing toys, a shape-changing input device, and an active window shutter.2023KZKe Zhong et al.Carnegie Mellon UniversityShape-Changing Interfaces & Soft Robotic MaterialsShape-Changing Materials & 4D PrintingCHI
Exploring Challenges and Opportunities to Support Designers in Learning to Co-create with AI-based Manufacturing Design ToolsAI-based design tools are proliferating in professional software to assist engineering and industrial designers in complex manufacturing and design tasks. These tools take on more agentic roles than traditional computer-aided design tools and are often portrayed as “co-creators.” Yet, working effectively with such systems requires different skills than working with complex CAD tools alone. To date, we know little about how engineering designers learn to work with AI-based design tools. In this study, we observed trained designers as they learned to work with two AI-based tools on a realistic design task. We find that designers face many challenges in learning to effectively co-create with current systems, including challenges in understanding and adjusting AI outputs and in communicating their design goals. Based on our findings, we highlight several design opportunities to better support designer-AI co-creation.2023FGFrederic Gmeiner et al.Carnegie Mellon UniversityGenerative AI (Text, Image, Music, Video)Human-LLM CollaborationCHI
ReCompFig: Designing Dynamically Reconfigurable Kinematic Devices Using Compliant Mechanisms and Tensioning CablesFrom creating input devices to rendering tangible information, the field of HCI is interested in using kinematic mechanisms to create human-computer interfaces. Yet, due to fabrication and design challenges, it is often difficult to create kinematic devices that are compact and have multiple reconfigurable motional degrees of freedom (DOFs) depending on the interaction scenarios. In this work, we combine compliant mechanisms (CMs) with tensioning cables to create dynamically reconfigurable kinematic mechanisms. The devices’ kinematics (DOFs) is enabled and determined by the layout of bendable rods. The additional cables function as on-demand motion constraints that can dynamically lock or unlock the mechanism’s DOFs as they are tightened or loosened. We provide algorithms and a design tool prototype to help users design such kinematic devices. We also demonstrate various HCI use cases including a kinematic haptic display, a haptic proxy, and a multimodal input device.2022HYHumphrey Yang et al.Carnegie Mellon UniversityShape-Changing Interfaces & Soft Robotic MaterialsCHI
Practice-Based Teacher Questioning Strategy Training with ELK: A Role-Playing Simulation for Eliciting Learner KnowledgePractice is essential for learning. However, for many interpersonal skills, there often are not enough opportunities and venues for novices to repeatedly practice. Role-playing simulations offer a promising framework to advance practice-based professional training for complex communication skills, in fields such as teaching. In this work, we introduce ELK (Eliciting Learner Knowledge), a role-playing simulation system that helps K-12 teachers develop effective questioning strategies to elicit learners' prior knowledge. We evaluate ELK with 75 pre-service teachers through a mixed-method study. We find that teachers demonstrate a modest increase in effective questioning strategies and develop sympathy towards students after using ELK for 3 rounds. We implement a supplementary activity in ELK in which users evaluate transcripts generated from past role-play sessions. We demonstrate that evaluating conversation moves is as effective for learning as role-playing, while without requiring the presence of a partner. We contribute design implications for role-play systems for communication strategy training.2021XWXu Wang et al.Learning and MentoringCSCW
Freeform Fabrication of Fluidic Edible MaterialsFrom providing nutrition to being social platforms, food plays an essential role in our daily lives and cultures. In HCI, we are interested in using food as an interaction medium and a context of personal fabrication. Yet, the design space of available food printing methods is limited to shapes with minimal overhangs and materials that have a paste-like consistency. In this work, we seek to expand this design space by adapting support bath-assisted printing to the food context. The bath scaffolds the embedded materials and preserves shapes during the printing processes, enabling us to create freeform food with fluid-like materials. We provide users guidelines for choosing the appropriate support bath type and processing methods depending on the printing material’s properties. A design tool suite and application examples, including confectionery arts, 4D printed food, and edible displays are also offered to demonstrate the enabled interaction design space.2021HYHumphrey Yang et al.Carnegie Mellon UniversityShape-Changing Materials & 4D PrintingFood Culture & Food InteractionCHI
SimuLearn: Fast and Accurate Simulator to Support Morphing Materials Design and WorkflowsMorphing materials allow us to create new modalities of interaction and fabrication by leveraging the materials’ dynamic behaviors. Yet, despite the ongoing rapid growth of computational tools within this realm, current developments are bottlenecked by the lack of an effective simulation method. As a result, existing design tools must trade-off between speed and accuracy to support a real-time interactive design scenario. In response, we introduce SimuLearn, a data-driven method that combines finite element analysis and machine learning to create real-time (0.61 seconds) and truthful (97% accuracy) morphing material simulators. We use mesh-like 4D printed structures to contextualize this method and prototype design tools to exemplify the design workflows and spaces enabled by a fast and accurate simulation method. Situating this work among existing literature, we believe SimuLearn is a timely addition to the HCI CAD toolbox that can enable the proliferation of morphing materials.2020HYHumphrey Yang et al.Shape-Changing Interfaces & Soft Robotic MaterialsShape-Changing Materials & 4D PrintingComputational Methods in HCIUIST
Morphlour: Personalized Flour-based Morphing Food Induced by Dehydration or Hydration MethodIn this paper, we explore personalized morphing food that enhances traditional food with new HCI capabilities, rather than replacing the chef and authentic ingredients (e.g., flour) with an autonomous machine and heterogeneous mixtures (e.g., gel). Thus, we contribute a unique transformation mechanism of kneaded and sheeted flour-based dough, with an integrated design strategy for morphing food during two general cooking methods: dehydration (e.g., baking) or hydration (e.g., water boiling). We also enrich the design space of morphing food by demonstrating several application cases. We end by discussing hybrid cooking between human and design tool to ensure accuracy while preserving customizability for morphing food.2019YTYe Tao et al.Human-Nature Relationships (More-than-Human Design)Food Culture & Food InteractionUIST
A-line: 4D Printing Morphing Linear Composite StructuresThis paper presents A-line, a 4D printing system for designing and fabricating morphing three-dimensional shapes out of simple linear elements. In addition to the commonly known benefit of 4D printing to save printing time, printing materials, and packaging space, A-line also takes advantage of the unique properties of thin lines, including their suitability for compliant mechanisms and ability to travel through narrow spaces and self-deploy or self-lock on site. A-line integrates a method of bending angle control in up to eight directions for one printed line segment, using a single type of thermoplastic material. A software platform to support the design, simulation and tool path generation is developed to support the design and manufacturing of various A-line structures. Finally, the design space of A-line is explored through four application areas, including line sculpting, compliant mechanisms, self-deploying, and self-locking structures.2019GWGuanyun Wang et al.Carnegie Mellon UniversityShape-Changing Interfaces & Soft Robotic MaterialsShape-Changing Materials & 4D PrintingCHI
Printed Paper Actuator: A Low-cost Reversible Actuation and Sensing Method for Shape Changing InterfacesWe present a printed paper actuator as a low cost, reversible and electrical actuation and sensing method. This is a novel but easily accessible enabling technology that expands upon the library of actuation-sensing materials in HCI. By integrating three physical phenomena, including the bilayer bending actuation, the shape memory effect of the thermoplastic and the current-driven joule heating via conductive printing filament, we developed the actuator by simply printing a single layer conductive Polylactide (PLA) on a piece of copy paper via a desktop fused deposition modeling (FDM) 3D printer. This paper describes the fabrication process, the material mechanism, and the transformation primitives, followed by the electronic sensing and control methods. A software tool that assists the design, simulation and printing toolpath generation is introduced. Finally, we explored applications under four contexts: robotics, interactive art, entertainment and home environment.2018GWGuanyun Wang et al.Carnegie Mellon UniversityShape-Changing Interfaces & Soft Robotic MaterialsDesktop 3D Printing & Personal FabricationShape-Changing Materials & 4D PrintingCHI
4DMesh: 4D Printing Morphing Non-Developable Mesh SurfacesWe present 4DMesh, a method of combining shrinking and bending thermoplastic actuators with customized geometric algorithms to 4D print and morph centimeter- to meter-sized functional non-developable surfaces. We will share two end-to-end inverse design algorithms. With our tools, users can input CAD models of target surfaces and produce respective printable files. The flat sheet printed can morph into target surfaces when triggered by heat. This system saves shipping and packaging costs, in addition to enabling customizability for the design of relatively large non-developable structures. We designed a few functional artifacts to leverage the advantage of non-developable surfaces for their unique functionalities in aesthetics, mechanical strength, geometric ergonomics and other functionalities. In addition, we demonstrated how this technique can potentially be adapted to customize molds for industrial parts (e.g., car, boat, etc.) in the future.2018GWGuanyun Wang et al.Shape-Changing Interfaces & Soft Robotic MaterialsShape-Changing Materials & 4D PrintingUIST