OpenXAI: A Comprehensive Framework for Transparent Evaluation and Benchmarking of Explanation Methods for Machine Learning Models

Abstract

Machine learning (ML) models are being used increasingly in critical, life-changing decisions in various fields, such as medicine, finance, law, and science. To ensure that the decision-making processes and final predictions of these models are understandable, post hoc explanation methods have been developed to provide explanations for the output of a machine learning model after it has been trained and deployed. However, there is a need to verify that the explanations generated by these methods are reliable, accurate, and unbiased, especially in cases where the stakes are high, such as in healthcare. In this thesis, we introduce OpenXAI, a comprehensive and extensible open-source framework for transparent and reproducible evaluations of explanation methods. OpenXAI includes seven state-of-the-art feature attribution methods and twenty-two quantitative metrics to evaluate the faithfulness, stability, and fairness of these methods. It also includes a comprehensive collection of seven real-world datasets spanning diverse domains and sixteen different pre-trained models. One of the unique features of OpenXAI is its novel synthetic data generator, which can synthesize datasets of varying sizes, complexity, and dimensionality. This feature allows researchers to construct reliable ground truth explanations for evaluating explanation methods. To promote transparency, we have developed the first-ever public XAI leaderboards, which provide a platform for evaluating different explanation methods across a wide variety of synthetic and real-world datasets, evaluation metrics, and predictive models. Using OpenXAI, we perform a rigorous empirical benchmark of the state-of-the-art feature attribution methods to determine which methods are effective with respect to what notions of reliability across a wide variety of datasets and predictive models. Our results demonstrate the effectiveness of certain explanation methods in various domains and highlight the need for standardized evaluations of post hoc explanation methods. The lack of trust in post hoc explanation methods contributes to the slow adoption of machine learning in high stakes domains such as healthcare. OpenXAI addresses this critical problem by enabling researchers to select the most appropriate explainable ML tools, leading to more trustworthy and reliable machine learning models for decision-making.