Conversational AI as Information Mediator: From Perception to Presentation

Abstract

Humans have long designed representational tools---charts, diagrams, maps---to offload cognitive work and make complex information tractable. Conversational AI is becoming a powerful new information intermediary, one that increasingly determines how people access and interpret data. How AI systems process information, what they attend to, and how they treat the people querying them all shape the data understanding processes central to modern life---yet these interfaces are poorly understood. This thesis examines AI as an information intermediary across three studies, each probing a different stage in the pipeline from data to human insight. The first study examines what goes in: how input representation shapes what AI can extract from data. Across three representative analysis tasks, the two systems describe synthetic datasets more precisely and accurately when raw data is accompanied by a scatterplot, especially as datasets grow in complexity. Comparison with two baselines---providing a blank chart and a chart with mismatched data---shows that the improved performance is due to the content of the charts. Our results are initial evidence that AI systems, like humans, can benefit from visualization. The second study examines what happens inside: whether AI models' internal computations during chart question-answering are similar to human visual processing. These models process images through ``attention''---an internal mechanism that assigns more weight to particular regions of the input. Individual AI attention components on inputs charts correlate with human eye-tracking data. Also, a linear combination of attention components predicts human gaze on par with than prior methods, and generalizes better out-of-domain. Ablation studies indicate the attention components most associated with human fixation patterns are implicated in AI chart understanding. The third study examines what comes out: specifically, what information reaches the user at all. Guardrail behavior---whether an AI model refuses or complies with a sensitive query---underscores that AI systems model the people querying them. Refusal rates vary systematically with declared user demographics, political affiliation, and even NFL team fandom, as models infer ideological identity even from incidental contextual details. Together, these findings show that representational format, internal structure, and inferred user identity each shape what people can ultimately learn from AI-mediated information systems, with implications for the design of AI systems that are accurate, explainable, and equitable.