Leveraging Passive User Context For Human-AI Collaboration

Abstract

The rapid advancement of Artificial Intelligence (AI) powered tools, particularly Large Language Models (LLMs), understanding user intent has emerged as a fundamental challenge for creating effective, user-centric tools. Although users can articulate their goals explicitly, traditional approaches to eliciting intent—such as detailed prompts, additional examples, or formal specifications—often impose a high cognitive burden and fail to fully capture the subtleties of users’ evolving needs. This dissertation argues that augmenting context awareness, particularly the passive capture of environmental and interaction cues, can reduce ambiguity in user intent, leading to more intuitive and efficient human-AI collaboration. We demonstrate the value of passive context awareness across three domains. In Chapter 1, we apply pragmatic reasoning in regular expression synthesis, reducing the need for exhaustive user examples by reasoning over examples not provided by the user as contextual cues to synthesize regular expressions. Chapter 2 introduces DynaVis, a dynamic interface for visualization editing that combines natural language input with dynamically generated UI widgets, showcasing how local workflow and task context can streamline iterative edits. Chapter 3 discusses MagicCopy, an AI-driven copy-and-paste tool that infers user intent by analyzing source and target applications alongside user instructions to automate cross-application data transformations. Finally, we envision the future of AI design workflows, emphasizing the importance of two-way grounding where systems not only interpret user context but also reveal their reasoning and capabilities. Together, these contributions highlight the potential of passive context-awareness to transform interactive AI by delivering more seamless, contextually informed assistance.