Building a Generalized Decoder for Immune Profiling in Pathology

Abstract

Accurate immune profiling of the tumor microenvironment (TME) requires integrating nuclei, tissue, and spatial-level information across sequential reasoning steps. Recent advances in vision language models (VLMs) have enabled natural language interaction with histopathology images, yet existing pathology VLMs are predominantly limited to single-turn question-answering and lack the structured, multi-step reasoning required for clinical immune profiling. We present a dataset construction, fine-tuning, and evaluation framework for multi-turn visual question answering (VQA), applied to computational immune profiling of the TME. Our multi-turn conversational pipeline decomposes immune profiling into sequential, clinically-grounded subtasks: nucleus detection and classification, cell-type quantification, tissue region analysis, and TME summarization. We introduce a structured approach to generate multi-turn VQA training data from annotated histopathology images. Our dataset comprises 238,747 patches spanning 22 organ types from 7 datasets, including hematoxylin and eosin (H&E) and immunohistochemistry (IHC) slides, from which we construct 515,382 multi-turn conversation samples across 13 turn types that span pixel-level and patch-level spatial scales. We further introduce a modular, turn-aware evaluation framework that assesses turn-level accuracy, cross-turn consistency, and conversational coherence. To our knowledge, this is the first such framework for multi-turn histopathology VQA. On an external test set, our fine-tuned model achieves a median nuclei detection F1 of 0.667, outperforming HoverNet on nuclei detection and segmentation. On a held-out test set, it surpasses all four VLM baselines across multi-turn evaluation metrics. This work establishes the feasibility of fine-tuning a general-purpose VLM for structured immune profiling. We envision this framework as a generalizable basis for developing and benchmarking conversational VLMs across immune profiling tasks and broader digital pathology applications.