Evaluating The Effectiveness of Different AI-Driven Virtual Patient Modules in Enhancing Student Clinical Communication Skills

Abstract

Background: Effective clinical communication is central to patient-centered care and is associated with improved patient understanding, trust, adherence, and health outcomes. Despite its importance, traditional communication-skill training that relies on standardized patients and faculty-facilitated simulation is resource-intensive and difficult to scale. Artificial intelligence (AI)-driven virtual patient simulations represent a promising, scalable alternative by offering standardized, repeatable, and accessible training environments. However, the educational value of embodied AI-avatar simulations versus AI-chatbot simulations remains insufficiently understood. Methods: We conducted a randomized comparative study evaluating AI-avatar versus AI-chatbot virtual patient modules among first-year medical students at Harvard Medical School. Outcomes were assessed using a multi-modal evaluation framework, including a retrospective pre-post self-assessment survey measuring confidence and comfort in clinical communication, performance in an Objective Structured Clinical Examination (OSCE), standardized-patient perception ratings scored using the adapted Kalamazoo Essential Elements Communication Checklist (KEECC-A), and a brief post-OSCE usability survey. Quantitative analyses compared self-reported confidence and comfort, observed communication performance, and leaner-perceived usability between the two conditions. Results: No statistically significant differences were observed between the AI-avatar and AI-chatbot groups across primary outcomes, including rapport building, learner comfort, perceived usability, standardized-patient ratings, and overall OSCE performance. These findings may reflect limited statistical power due to sample size, and potential moderating factors such as prior experience with AI technology, which significant correlated with perceived engagement and usability. Despite comparable performance outcomes, students in the AI-avatar group reported greater emotional engagement and preferred a combined approach of AI-based and traditional in-person simulation (64%). In contrast, students in the AI-chatbot group demonstrated a clear preference for traditional in-person simulation alone (75%). Conclusions: These findings suggest that the added value of AI-avatar simulation may lie less in immediate measurable performance gains and more in its capacity to enhance relational and experiential dimensions of communication training, highlighting the importance of affective and experiential factors in educational design. A head-to-head comparison of avatar- and chatbot-based virtual patients may clarify whether visual embodiment and non-verbal affordances produce meaningful educational gains beyond lower-barrier conversational interfaces. This study contributes to the evidence base for integrating AI into medical education and offers guidance for developing scalable, psychologically safe, and pedagogically grounded clinical communication training.