A Multidimensional Proficiency Index for Assessing Crisis Management in Anesthesiology: Development and Validation

Abstract

Assessment of anesthesiology crisis management remains challenging due to the multidimensional and context-dependent nature of clinical performance. Competency-based medical education (CBME) requires robust, interpretable measures of competence that extend beyond knowledge or isolated skill domains. We developed a multidimensional Proficiency Index (PI) integrating four domains: workflow, crisis resource management (CRM), clinical skills, and knowledge to provide a comprehensive assessment of crisis management performance. Methods: The PI was developed using a modified Delphi process to establish content validity and define domain structure and weighting. It was subsequently evaluated in a prospective cross-sectional study of anesthesiology residents (CA1-CA3) completing two standardized high-fidelity simulation scenarios (intraoperative anaphylaxis and unexpected difficult airway) and a 45-item knowledge assessment. Each simulation performance was independently rated by two faculty assessors. Descriptive statistics were calculated at the participant level. Inter-rater reliability was assessed using intraclass correlation coefficients (ICC[2,1]) and Bland–Altman analysis. Measurement precision was evaluated using the standard error of measurement (SEM) and minimal detectable change (MDC95). Differences across training levels were analyzed using ANOVA and mixed-effects models with Tukey-adjusted pairwise comparisons. Effect sizes were calculated using Hedges’ g. Results: Forty-eight residents participated (22 CA1, 14 CA2, 12 CA3). Mean PI scores increased progressively across training levels (CA1: 39.7±5.0; CA2: 59.4±3.4; CA3: 71.8±5.2; p.001). Performance domains (workflow, CRM, clinical skills) demonstrated consistent separation across training levels, whereas knowledge demonstrated minimal difference between CA2 and CA3. Inter-rater reliability was excellent for the PI (ICC 0.96, 95% CI 0.92–0.98). Measurement precision was acceptable (SEM 2.9; MDC95 8.0). Mixed-effects models confirmed training level as a significant predictor of PI and performance domains, but not knowledge at higher levels. Effect sizes were large to very large for the PI and performance domains. Conclusions: The Proficiency Index provides a reliable, multidimensional measure of anesthesiology crisis management performance that discriminates across training levels. Divergent trajectories across domains highlight the importance of integrated assessment frameworks in CBME and support the use of domain-specific metrics to guide individualized training and evaluation and inform competency-based progression decisions.