Generalizable and Explainable Deep Learning in Medical Imaging with Small Data

Abstract

Deep learning algorithms, such as those used for image recognition, holds promise for automated medical diagnosis and in guiding clinical decision-making. At the same time, there remain several important challenges to the development and clinical translation of medical deep learning systems. First, it is costly to develop large and well-annotated datasets. Second, it is necessary for medical image interpretation to identify subtle key features for lesions despite the wide range in physiologic appearance across the population. Third, it is challenging to transfer the performance of deep learning algorithms from one setting to another because of domain shift problems. Fourth, the outputs of deep learning systems need to be explainable in order to make the systems understandable to clinicians. This dissertation investigates how to address these challenges, building generalizable and explainable deep learning models from small datasets. The thesis studies the impact on model performance of transferring prior knowledge learned from a non-medical source — ImageNet — to medical applications, especially when the dataset size is not sufficient. Instead of direct transfer learning from ImageNet, GrayNet is proposed as a bridge dataset to create a pre-trained model enriched with medical image representations on top of the generic image features learned from ImageNet. Benefits of GrayNet are analyzed with regard to overall performance and generalization across different imaging scanners, in comparison with training from scratch with small data and transfer learning from ImageNet. Domain-specific techniques including window setting optimization and slice interpolation, inspired by how radiologists interpret images for a diagnosis, are also introduced and shown to further enhance model performance. A new visualization module is introduced, able to generate an atlas of images during training, and display this as the basis of model predictions made during testing, in order to justify model predictions and make them more understandable for clinicians. This thesis demonstrates the potential of deep learning for medical image interpretation through three different applications, including AI-assisted bone age assessment to improve human’s accuracy and variability, finding previously unrecognized patterns to perform bone sex classification in hand radiographs, and processing raw computed tomography data without image reconstruction. The contributions of this thesis are expected to facilitate the development of generalizable and explainable deep learning algorithms for a variety of medical applications and consequently accelerate the adoption of AI systems into clinical practice.