Population Coding for Multiple Features in the Human Brain and Convolutional Neural Networks

Abstract

Entities in the world comprise multiple features: how might an intelligent system, whether biological or synthetic, encode those feature combinations and put them to use in task-relevant processing? While this is a perennial and ubiquitous problem in the cognitive sciences, several recent developments make it timely to examine it anew: the recent characterization of color- and shape-sensitive regions in the primate ventral visual pathway opens a promising window towards examining how color and form are jointly represented in visual processing, the explosive success of artificial neural networks raises the question of how they encode feature combinations to solve the visual tasks at which they excel, and recent developments in computational neuroscience have highlighted the importance of neurons that exhibit nonlinear interaction effects to combinations of stimulus and task variables. In this dissertation, I draw on these recent developments in three projects examining how color and form are encoded in the human ventral visual pathway and convolutional neural networks, and how task and stimulus information are encoded throughout the human visual system. I develop several methods that can characterize the joint coding structure of multiple features. I find that the human ventral visual pathway largely encodes color and form information in an anatomically intermingled but representationally independent manner, that convolutional neural networks code color and form in an increasingly interactive manner throughout processing, and that the superior intraparietal sulcus, but not early visual cortex, encodes stimulus and task information in a nonlinear manner when attentional demands are carefully controlled.