AnnotateGPT: Designing Human–AI Collaboration in Pen-Based Document AnnotationProviding high-quality feedback on writing is cognitively demanding, requiring reviewers to identify issues, suggest fixes, and ensure consistency. We introduce AnnotateGPT, a system that uses pen-based annotations as an input modality for AI agents to assist with essay feedback. AnnotateGPT enhances feedback by interpreting handwritten annotations and extending them throughout the document. One AI agent classifies the purpose of each annotation, which is confirmed or corrected by the user. A second AI agent uses the confirmed purpose to generate contextually relevant feedback for other parts of the essay. In a study with 12 novice teachers annotating essays, we compared AnnotateGPT with a baseline pen-based tool without AI support. Our findings demonstrate how reviewers used annotations to regulate AI feedback generation, refine AI suggestions, and incorporate AI-generated feedback into their review process. We highlight design implications for AI-augmented feedback systems, including balanced human-AI collaboration and using pen annotations as subtle interaction.2026BLBenedict Leung et al.Ontario Tech UniversityAI-Assisted Writing & Text GenerationNote-Taking & Sketching ToolsGenerative AI (Text, Image, Music, Video)CHI
SwipeSense: Exploring the Feasibility of Back-of-Device Swipe Interaction Using Built-In IMU SensorsThe growing dimensions of smartphones have intensified the challenges associated with screen reachability. Back-of-device (BoD) interaction expands the range of reachability and offers a promising solution to mitigate screen occlusion while enhancing one-handed interactions. However, much of the existing research relies on incorporating additional hardware components. In this paper, we present SwipeSense a technique for exploring the feasibility of directional swipe interactions on the back of devices, utilizing built-in inertial measurement unit (IMU) sensors and machine learning models. We conducted a user study with 12 participants who performed 9600 BoD swipes in 8 distinct directions while holding the device naturally. The results of our machine learning models indicate that various directional swipes on the back of the device can be accurately distinguished using only the built-in IMU sensors of the phone, achieving a range of model accuracy between 72% and 95%. Furthermore, we showcase potential applications for these gestures.2025NSNeel Shah et al.Hand Gesture RecognitionContext-Aware ComputingUbiquitous ComputingMobileHCI
Exploring Interactive Color Palettes for Abstraction-Driven Exploratory Image ColorizationColor design is essential in areas such as product, graphic, and fashion design. However, current tools like Photoshop, with their concrete-driven color manipulation approach, often stumble during early ideation, favoring polished end results over initial exploration. We introduced Mondrian as a test-bed for abstraction-driven approach using interactive color palettes for image colorization. Through a formative study with six design experts, we selected three design options for visual abstractions in color design and developed Mondrian where humans work with abstractions and AI manages the concrete aspects. We carried out a user study to understand the benefits and challenges of each abstraction format and compare the Mondrian with Photoshop. A survey involving 100 participants further examined the influence of each abstraction format on color composition perceptions. Findings suggest that interactive visual abstractions encourage a non-linear exploration workflow and an open mindset during ideation, thus providing better creative affordance.2024XSXinyu Shi et al.University of WaterlooGenerative AI (Text, Image, Music, Video)Graphic Design & Typography ToolsCreative Collaboration & Feedback SystemsCHI
Escapement: A Tool for Interactive Prototyping with Video via Sensor-Mediated Abstraction of Time We present Escapement, a video prototyping tool that introduces a powerful new concept for prototyping screen-based interfaces by flexibly mapping sensor values to dynamic playback control of videos. This recasts the time dimension of video mock-ups as sensor-mediated interaction. This abstraction of time as interaction, which we dub video-escapement prototyping, empowers designers to rapidly explore and viscerally experience direct touch or sensor-mediated interactions across one or more device displays. Our system affords cross-device and bidirectional remote (tele-present) experiences via cloud-based state sharing across multiple devices. This makes Escapement especially potent for exploring multi-device, dual-screen, or remote-work interactions for screen-based applications. We introduce the core concept of sensor-mediated abstraction of time for quickly generating video-based interactive prototypes of screen-based applications, share the results of observations of long-term usage of video-escapement techniques with experienced interaction designers, and articulate design choices for supporting a reflective, iterative, and open-ended creative design process.2023MNMolly Jane Nicholas et al.UC BerkeleyTeleoperation & TelepresencePrototyping & User TestingCHI
TiiS: Learn, Generate, Rank, Explain: A Case Study of Visual Explanation by Generative Machine LearningKim 等人提出了一个整合学习、生成、排序与解释功能的视觉解释系统，通过生成式机器学习为用户提供可解释的视觉内容分析。2022CKChris Kim et al.Explainable AI (XAI)AI-Assisted Creative WritingIUI
Supporting Serendipitous Discovery and Balanced Analysis of Online Product Reviews with Interaction-Driven Metrics and Bias-Mitigating SuggestionsIn this study, we investigate how supporting serendipitous discovery and analysis of online product reviews can encourage readers to explore reviews more comprehensively prior to making purchase decisions. We propose two interventions --- Exploration Metrics that can help readers understand and track their exploration patterns through visual indicators and a Bias Mitigation Model that intends to maximize knowledge discovery by suggesting sentiment and semantically diverse reviews. We designed, developed, and evaluated a text analytics system called Serendyze, where we integrated these interventions. We asked 100 crowd workers to use Serendyze to make purchase decisions based on product reviews. Our evaluation suggests that exploration metrics enabled readers to efficiently cover more reviews in a balanced way, and suggestions from the bias mitigation model influenced readers to make confident data-driven decisions. We discuss the role of user agency and trust in text-level analysis systems and their applicability in domains beyond review exploration.2022MJMahmood Jasim et al.University of Massachusetts Amherst, University of Massachusetts AmherstExplainable AI (XAI)Algorithmic Transparency & AuditabilityRecommender System UXCHI
Lexichrome: Text Construction and Lexical Discovery with Word-Color Associations Using Interactive VisualizationBased on word-color associations from a comprehensive, crowdsourced lexicon, we present Lexichrome: a web application that explores the popular perception of relationships between English words and eleven basic color terms using interactive visualization. Lexichrome provides three complementary visualizations: "Palette" presents the diversity of word-color associations across the color palette; "Words" reveals the color associations of individual words using a dictionary-like interface; "Roget's Thesaurus" uncovers color association patterns in different semantic categories found in the thesaurus. Finally, our text editor allows users to compose their own texts and examine the resultant chromatic fingerprints throughout the process. We studied the utility of Lexichrome in a two-part qualitative user study with nine participants from various writing-intensive professions. We find that the presence of word-color associations promotes awareness surrounding word choice, editorial decision, and audience reception, and introduce a variety of use cases, features, and opportunities applicable to creative writing, corporate communication, and journalism.2020CKChris Kim et al.Interactive Data VisualizationData StorytellingDIS
A Postmortem on Playtesting: Exploring the Impact of Playtesting on the Critical Reception of Video GamesGame studios aim to develop titles that deliver a fun and engaging experience for players. Playtesting promises to help identify opportunities to improve player experience and assist developers in achieving their design intent. However, a lack of research on the added value of playtesting means that many studios are still uncertain about its commercial viability and impact on product success. This gap in understanding is further complicated by the vague definition of "success" afforded by sales figures and review scores. In this paper, we assess reported feature quality of three commercial titles by analyzing playtesting reports and game reviews. By comparing themes and design issues expressed in game reviews to the results of pre-release playtesting for each game, we aim to highlight the value of playtesting and propose a set of guidelines for selecting playtest methods based on the needs of a given game evaluation. Through the real-world case studies presented, this paper contributes to the growing domain of games user research and highlights the value of playtesting in game development.2020PMPejman Mirza-Babaei et al.Ontario Tech UniversityGame UX & Player BehaviorPrototyping & User TestingCHI
Aggregated Visualization of Playtesting DataPlaytesting is a key component in the game development process aimed at improving the quality of games through the collection of gameplay data and identification of design issues. Visualization techniques are currently being employed to help integrate quantitative and qualitative data. Despite that, two existing challenges are to determine the level of detail to be presented to developers based on their needs and to effectively communicate the collected data so that informed design changes can be reached. In this paper, we first propose an aggregated visualization technique that makes use of clustering, territory tessellation, and trajectory aggregation to simultaneously display mixed playtesting data. Secondly, to assess the usefulness of our technique we evaluate it through interviews with professional game developers and compare it to a non-aggregated visualization. The results of this study also provide an important contribution towards identifying areas of improvement in the portrayal of gameplay data.2019GWGünter Wallner et al.Eindhoven University of TechnologyInteractive Data VisualizationGame UX & Player BehaviorCHI
Let's Play Together: Adaptation Guidelines of Board Games for Players with Visual ImpairmentBoard games present accessibility barriers for players with visual impairment since they often employ visuals alone to communicate gameplay information. Our research focuses on board game accessibility for those with visual impairment. This paper describes a three-phase study conducted to develop board game accessibility adaptation guidelines. These guidelines were developed through a user-centered design approach that included in-depth interviews and a series of user studies using two adapted board games. Our findings indicate that participants with and without visual impairment were able to play the adapted games, exhibiting a balanced experience whereby participants had complete autonomy and were provided with equal chances of victory. Our paper also contributes to the game and accessibility communities through the development of adaptation guidelines that allow board games to become inclusive irrespective of a player's visual impairment.2019FFFrederico da Rocha Tomé Filho et al.University of Ontario Institute of TechnologyUniversal & Inclusive DesignGame AccessibilityCHI
Saliency Deficit and Motion Outlier Detection in Animated ScatterplotsWe report the results of a crowdsourced experiment that measured the accuracy of motion outlier detection in multivariate, animated scatterplots. The targets were outliers either in speed or direction of motion, and were presented with varying levels of saliency in dimensions that are irrelevant to the task of motion outlier detection (e.g., color, size, position). We found that participants had trouble finding the outlier when it lacked irrelevant salient features and that visual channels contribute unevenly to the odds of an outlier being correctly detected. Direction of motion contributes the most to accurate detection of speed outliers, and position contributes the most to accurate detection of direction outliers. We introduce the concept of saliency deficit in which item importance in the data space is not reflected in the visualization due to a lack of saliency. We conclude that motion outlier detection is not well supported in multivariate animated scatterplots.2019RVRafael Veras et al.University of Ontario Institute of TechnologyInteractive Data VisualizationVisualization Perception & CognitionCHI
Games and Play SIG: Engaging Small Developer CommunitiesThe Games-and-Play community has thrived at ACM SIGCHI with a consistent increase in games- and play-related submissions across research papers, workshops, posters, demos, and competitions. The community has attracted a significant number of academic researchers, students, and practitioners to CHI conferences in recent years. CHI 2018 is being held in Montréal, a major game development hub. Montréal is not only a home for major game studios but also more than 100 smaller game studios. In line with the “Engage With CHI” spirit of CHI 2018, this SIG aims to engage the Games and Play community in a discussion about the directions that we can take to advance towards demographics that will benefit from HCI games research but are currently underrepresented: small, independent developers, non-profit organizations, and academics that create mobile games, games for health, games for change, and/or educational games.2018LNLennart E. Nacke et al.University of WaterlooGame UX & Player BehaviorSerious & Functional GamesCollaborative Learning & Peer TeachingCHI
Metatation: Annotation as Implicit Interaction to Bridge Close and Distant ReadingIn the domain of literary criticism, many critics practice close reading, annotating by hand while performing a detailed analysis of a single text. Often this process employs the use of external resources to aid analysis. In this article, we present a study and subsequent tool design focused on leveraging a critic’s annotations as implicit interactions for initiating context-specific computational support that automatically searches external resources. We observed 14 poetry critics performing a close reading, revealing a set of cognitive practices supported through free-form annotation that have not previously been discussed in this context. We used guidelines derived from our study to design a tool, Metatation, which uses a pen-and-paper system with a peripheral display to utilize reader annotations as underspecified interactions to augment close reading. By turning paper based annotations into implicit queries, Metatation provides relevant supplemental information in a just-in-time manner and acts as a bridge between close and distant reading.2018HMHrim R Mehta et al.University of Ontario Institute of TechnologyUser Research Methods (Interviews, Surveys, Observation)Prototyping & User TestingCHI