Role of Recurrent Computations in Object Completion

Abstract

Existing models of visual object recognition posit that recognition is orchestrated by a hierarchy of processing layers. In these models, neural computation proceeds in a largely feed-forward path up this hierarchy, without substantial feedback or recurrent processing. These feed-forward models provide a parsimonious account of experimental data, and have given rise to deep convolutional networks in computer vision that outperform previous approaches to object recognition. In this dissertation, we challenge these feed-forward theories by considering the problem of occlusion.

In natural vision, objects are often partially visible, either due to occlusion, limited viewing angles, or poor illumination. The vast majority of previous neurophysiological studies focus on the completion of simple contours, geometric shapes, or line drawings. These studies typically contrast neural responses to occluded objects against responses to unrecognizable scrambled counterparts, thus confounding object completion mechanisms with neural activity linked to perceptual awareness. We set out to provide conceptual advances by using naturalistic objects and measuring the selectivity and tolerance of neural responses when objects are only partially visible.

While we know that feedback and recurrent connections are prevalent throughout visual cortex, their underlying roles are unclear. We present three lines of evidence for the role of recurrence in recognition of occluded objects. We first recorded intracranial field potentials from electrodes surgically implanted in epilepsy patients and measured neural responses to whole and occluded objects. Responses along the ventral visual stream remained selective despite heavy occlusion. However, these visually selective signals emerged ~100 ms later than responses to whole objects. The processing delays were particularly pronounced in higher visual areas within the ventral stream, suggesting the involvement of additional recurrent processing. Second, we conducted psychophysical experiments to demonstrate that disrupting this recurrence with backward masking after ~75 ms significantly impaired recognition of occluded, but not whole, objects. Lastly, we augmented a computational model with recurrence that significantly outperformed existing feed-forward models and matched human performance.