Assessing the Impact of the Student-Generated Reasoning Tool Through a Two-Phase Study: Harvard Medical School and Université de Montréal Medical School

Abstract

Various approaches to fostering medical students’ critical thinking skills exist, from unstructured (e.g., open-ended) to structured (e.g., multiple-choice) methods. However, metacognitive strategies as a teaching modality remain infrequent. We developed and implemented the Student-Generated Reasoning Tool (SGRT) to combine structured and unstructured methods, emphasize student creativity, and use retrieval practice and feedback in collaborative learning environments. We conducted a two-phase mixed-methods study in 2019 on preclinical medical students. Students filled out SGRT by generating hypotheses, creating findings, and predicting the effect of these findings on suggested hypotheses. Students worked individually, then in teams, and faculty provided feedback. Students practiced with collected SGRTs by predicting their peer-generated findings on hypotheses. In Phase 1, at Harvard Medical School (n=171), we aimed to determine the feasibility and impact of SGRT. We collected and analyzed quiz scores, comparing them to historical controls. We conducted a Phase 2 randomized controlled trial, at Université de Montréal (n=12), comparing students using the SGRT in two experimental conditions (creative vs. non-creative) and one control condition (no SGRT), to isolate the effects of creation. We compared average exam scores. In both phases, we obtained data through surveys and interviews, which we analyzed using statistical and thematic analyses, respectively. In Phase 1, SGRT-users were significantly more likely to get quiz questions correct vs. non-users (OR=4.9; p<.001). In Phase 2, average exam scores were 33.5% higher in the creative, experimental condition vs. non-creative (mean=75.0% vs. 50.0%; respectively). Student-created findings were more diverse when using SGRT in teams vs. individually. Survey ratings on overall usefulness of SGRT were higher by 20% in Phase 2 vs. Phase 1 (100% vs. 80%, respectively). Qualitative analysis in Phase 1 revealed suggestions for improvement that we successfully implemented in Phase 2. The marked increase in average quiz performance suggests gains in learning that were likely attributable to the SGRT. Phase 1 recommendations led to Phase 2 modifications that may help explain the significant rise in students’ perception. The results of Phase 2 highlight the importance of creativity and teamwork. Our preliminary results with SGRT uncover an innovative approach to teaching critical thinking to medical students.