Multisensory Integration of Spatio-Temporal Information in Attention and Working Memory

Abstract

Real-world perception is fundamentally multifaceted and multisensory. It requires us to attend and remember temporal and spatial features of multiple objects in different sensory modalities, and to selectively integrate cross-modal information emanating from a common source. While previous studies with human subjects have examined each of these processes in isolation, the overarching goal of this dissertation is to examine them together, in complex sensory settings inspired by the challenges we encounter in real-world environments. In the first aim, spatial and temporal processing of unisensory stimuli are examined in a dual-task paradigm involving attentional and working memory demands. Behavioral, autonomic, and electrophysiological signatures of dual-task interference reflected the strain on neural resources when the tasks matched in sensory modality and information domain (temporal versus spatial). The remainder of the dissertation is focused on the role that cross-modal spatial alignment plays in facilitating audio-visual (AV) integration. While temporal synchrony across modalities is a known driver of multisensory integration, the practical importance of spatial alignment has been relatively understudied. In Aim 2, this was investigated using a multisensory search program in which the auditory and visual components of a multisensory target could be aligned or misaligned. Results demonstrated that when only one pair of temporally correlated auditory and visual streams was embedded amid visual distractors, the AV target was integrated regardless of cross-modal spatial alignment. However, when a second AV distractor object was added, search benefits resulting from AV integration depended on spatial alignment of the temporally correlated AV streams. The large-N online study presented in Aim 3 extends these findings to AV speech-in-noise perception. In a two-talker speech attention paradigm, aligning corresponding auditory and visual information in the same hemifield facilitated comprehension of the target talker’s speech. Taken together, these results characterize dual-task interference patterns in spatial and temporal processing and indicate that cross-modal spatial alignment is an important factor in AV integration when the sensory environment is complex or attentionally demanding.