Multimodal and Context-Aware Computational Pathology

Abstract

Cancers are defined by hallmark histopathological, genomic, and transcriptomic heterogeneity in the tumor and tissue microenvironment that contributes towards variability in treatment response rates and patient outcomes. The current clinical paradigm for cancer classification and prognostication is the manual assessment of histopathologic features such as tumor invasion, immune-infiltrates, and necrosis, which has been demonstrated to suffer from large inter- and intra-observer variability. Current advances in artificial intelligence (AI) for computational pathology (CPath) have demonstrated clinical-grade performance of AI workflows that exceed human pathologist performance in the objective characterization of histopathologic biomarkers in whole-slide images (WSIs) for diagnostic tasks. However, in prognostic tasks such as survival outcome prediction and response-to-treatment assessment, such tasks entail capturing contextual and multimodal information in the tumor microenvironment, which present challenging barriers for CPath workflows seeking clinical translation. In this dissertation, we expand the prognostic capabilities of current techniques, ranging from: (1) combining histology and genomics via multimodal deep learning, to (2) modeling contextual relationships based on permutation-equivariant aggregation techniques in set-based deep learning. For multimodal integration, we explored multimodal fusion architectures for resolving the data heterogeneity gap between WSIs and bulk molecular features, developing a computational technique that spatially deconvolves molecular pathway features onto a WSI. These data highlight not only superior prognostic performance over unimodal approaches, but also capabilities in finding image-omic biomarkers and visualizing feature interactions between morphology and molecular pathways. To model context, we explored set-based architectures for learning relationships between histology features, developing a computational technique based on Transformer attention for learning hierarchical cell-tissue relationships in the tumor microenvironment. These data highlight not only improved prognostic performance over traditional CPath techniques across diverse tumor types, but also clinical interpretability for characterizing contextual relationships in pathology. Together, these studies study two important gaps for cancer prognosis in CPath workflows: (1) multimodal integration and (2) modeling context, serving as a basis for harnessing pathology for prognosis and biomarker discovery.