Implementing Milestones: A Qualitative Analysis of Impact

Abstract

Background: Changes to programs of assessment following the shift to milestones-based assessment in the Next Accreditation System have not been fully characterized. Better understanding is critical to helping programs across all specialties capitalize on emerging best practices and overcome shared challenges to meaningful milestones implementation. Methods: Semi structured interviews were conducted with 15 residency program directors or associate program directors in 6 medical specialties at 8 academic medical centers. Questions explored four areas: 1) reviews of pre-milestones assessment programs to prepare for milestones-based assessment and reporting; 2) what programs retained from previous assessment efforts; 3) what programs added or changed from previous assessment efforts; and 4) what was the impact of the shift to milestones-based assessment on the program. Thematic analysis was used to identify codes, sub-themes, and themes. Results: Four themes were defined in the data: 1) challenges to effective implementation; 2) adaptability and making milestones work; 3) value and utility of assessment changes; and 4) accountability and transparency to society, program, and residents. Conclusions: Three of the four themes identified elaborated the impact of the shift to milestones, such as identification of gaps in curricula or the ability to obtain previously lacking objective performance data. Program leaders desired more guidance and support in redesigning programs of assessment but are resourceful in adapting milestones to their context. This underscores the importance of supporting programs with practical strategies and resources to support success in milestones-based assessment.