Person:

Ying, Lance

Human beings are social creatures. We routinely reason about other agents, and a crucial component of this social reasoning is inferring people’s goals as we learn about their actions. In many settings, we can perform intuitive but reliable goal inference from language descriptions of agents, actions, and the background environments. In this paper, we study this process of language driving and influencing social reasoning in a probabilistic goal inference domain. We propose a neurosymbolic model that carries out goal inference from linguistic inputs of agent scenarios. The “neuro” part is a large language model (LLM) that translates language descriptions to code representations, and the “symbolic” part is a Bayesian inverse planning engine. To test our model, we design and run a human experiment on a linguistic goal inference task. Our model closely matches human response patterns and better predicts human judgements than using an LLM alone.

In Human-AI collaboration, human agents often have a clear goal in mind and the AI Assistant tries to help users achieve their goals more efficiently. However, inferring users’ goals is non-trivial based on noisy user behavior and there is often mismatch in agent’s belief about each other’s knowledge of the ground-truth goal, leading to coordination failure. In this study, we propose that building common goal knowledge through communication improves human user’s mental model of the AI Assistant and leads to more efficient and effective human-AI collaboration. To test this hypothesis, we design an experiment where an AI assistant helps a human user shop for recipes on a grocery platform. We compare the user behavior and team performance under three experimental conditions: AI providing no information over its knowledge over human goals, AI expressing its belief over human’s goal through verbal communication(“Show”) and AI indicating its confidence in its belief over human’s goal (“Tell”). We find that communicating goal knowledge (in “Show” and “Tell”) increases user’s tendency to use AI when AI is indeed correct and improves user’s subjective ratings of the AI assistant.

Spoken language instructions are ubiquitous in agent collaboration. However, in real-world human-robot collaboration, following human spoken instructions can be challenging due to various speaker and environmental factors, such as background noise or mispronunciation. When faced with noisy auditory inputs, humans can leverage the collaborative context in the embodied environment to interpret noisy spoken instructions and take pragmatic assistive actions. In this paper, we present a cognitively inspired neurosymbolic model, Spoken Instruction Following through Theory of Mind (SIFToM), which leverages a Vision-Language Model with model-based mental inference to enable robots to pragmatically follow human instructions under diverse speech conditions. We test SIFToM in both simulated environments (VirtualHome) and real-world human-robot collaborative settings with human evaluations. Results show that SIFToM can significantly improve the performance of a lightweight base VLM (Gemini 2.5 Flash), outperforming state-of-the-art VLMs (Gemini 2.5 Pro) and approaching human-level accuracy on challenging spoken instruction following tasks.