A Multi-resolution Hard Attention Model to Select Regions of Interest on Whole Pathology Slide Images

Abstract

With drastic improvements in the performance of neural networks and computer vision algorithms, deep learning-based imaging analysis models have been applied to a wide variety of fields. Along with this expansion, many applications increasingly involve very large inputs, which directly affects memory usage and the scaling of model architectures. Two of such challenges in the field include: 1) efficiently processing very high resolution images, and 2) detecting and classifying tiny objects relative to the size of the image. Both of these challenges are especially salient in the field of computational pathology, where most of the workflow involves processing high resolution images of scanned whole pathology slides on the scale of gigapixels. While rapid progress has been made in the past few years in cancer diagnosis, subtyping, and survival prediction using whole pathology slide images (WSIs), these methods involve patching and processing the entire WSI or at least the segmented tissue region at the highest resolution. However, many problems in computational pathology only require making a decision based on identifying small regions of interest (ROI) that make up a tiny proportion of the WSI, such as identifying cancer metastasis, identifying diseased glomeruli, etc. Additionally, WSIs come in the format of a multi-resolution image pyramid, yet most current methods only examine the slide at a fixed (usually a very high) resolution, without taking advantage of the multi-resolution data. To address these challenges, we propose a hard-attention method trained with reinforcement learning that identifies ROIs by selectively processing the WSI in a sequential top-down approach: examining the slide at lower resolutions and selectively zooming into patches that are likely to contain ROIs. We apply our method to the task of identifying glomeruli in kidney biopsies and show that it significantly lowers the proportion of the WSI sampled at high resolutions while maintaining high coverage of the structures of interest. Our method has the potential to reduce the time and cost in computational settings as well as to integrate into the traditional pathology workflow, with a higher expected impact in more resource-constrained settings.