Connecting Language Representations in Humans and Machines

Abstract

In the past decade, biologically-inspired neural network models have surprisingly uncovered ways of building upon our understanding of the human brain, especially in the field of computer vision. Of particular interest, more recently, however, are the mechanisms by which language is processed in the brain. State-of-art natural language processing models have gained greater traction because of their exceptional performance on linguistic tasks; however, there is still a significant gap in understanding how these models learn language and what parallels, if any, can be drawn between the human brain and these models. Parts I-III of this thesis introduces the language regions of interest, describes deep neural network model architectures, and outlines the theory and metrics of evaluations behind correlation methods. Parts IV and V demonstrate our exploratory analysis and experimental findings specific to data collected from natural reading, describing patterns found spatially in the brain and layer-wise in deep neural network models. We find more discriminatory metrics when decoding from model-to-brain (the traditional method of decoding in neuroscience literature) compared to brain-to-model. We also observe that pretrained word embeddings perform similarly to contextualized word embeddings when decoding from model-to-brain, and we further identify the left middle posterior temporal cortex and the left posterior temporal cortex as language regions of most interest. No conclusive evidence was found to support a relationship between language models layer-wise to brain regions spatially. Given the exploratory nature of this work, our framework holds promise in understanding how artificial neural networks represent language and lays the groundwork for further investigation in connecting brain data with model embeddings.