Tumor Profiling from Pathology Images using Deep Learning

Abstract

Pathologists diagnose tumor biopsies under the microscope, navigating different levels of magnification to survey cellular composition, local glands, and global tissue landscapes across a slide. Recently, digital whole slide images (WSI) were approved by the FDA for use in clinical diagnosis by pathologists. Deep learning can be used to classify histology tissue images based on pathological representations to augment in clinical diagnoses and potentially prognoses. Digital scans of hematoxylin and eosin (H&E) biopsies are used in pathology image analysis, inclusive of cell and tissue segmentation, and extraction of expertly annotated features for classification with machine learning. Deep learning encodes several features of these images by using artificial neurons constructed in multiple layers, without the need for handcrafting features. There is an opportunity to leverage deep learning to characterize the tumor immune microenvironments (TIME) and to stratify the diagnoses of actionable biomarkers from biopsy images. The current implementation of deep learning models largely based on training select patches of whole slide images (WSI) is often at odds with pathology criteria of evaluating the TIME across the whole biopsy, particularly in precision medicine of drugs that target specific tumor mutations. The impact of tumor heterogeneity, stromal activity, immune infiltration cells, and other cells in the tumor microenvironment requires deep learning surveillance across wholes slides rather than just patches. In this research, we integrate multiple magnification levels of the H&E biopsy slides to encode as much TIME information as possible in the deep neural network layers to improve the diagnostic task of predicting and classification of tumor proliferation. In a novel approach that we have named the Digital Biopsy, we could traverse both the epithelial level and the cellular level, spanning the whole tissue landscape. Using breast cancer biopsy images from the TCGA database, as curated by the TUPAC16 Challenge, we found that convolutional neural network (CNN) classifiers trained on Digital Biopsies generalized better with a micro-average AUROC = 0.89 and quadratic weighted Cohen Kappa score of k = 0.774 compared to the benchmark of k = 0.567 that utilized individual level patches in the task of classifying mitotic score counts from pathology images. Our results open possibilities to maximize biopsy information from pathology images, accessing cell type changes and tumor immune responses influenced by genomic profiles. We marry the concept of the physical biopsy under the microscope into a digital framework that can thrive in AI and TIME studies, impacting clinical immune therapy treatments. The feasibility of our workflow mitigates the challenge of requiring a large amount of data for training deep learning models for every whole slide sample. As the notorious black boxes of deep learning are further opened, Digital Biopsies can help with interpretable AI diagnostics and generalizability across datasets.