Finding Structure in Human Cognition: From Temporal Order Codes in Memory to Behavioral Signatures in Vision and Language

Abstract

As humans, we carry out many cognitive feats effortlessly; for example, we parse complex scenes, hold conversations, and remember experiences that unfold over time. Yet the computations that make these abilities possible are far from trivial. While neuroscience seeks to uncover the mechanisms that generate human and non-human cognition, much of artificial intelligence and computational neuroscience aims to reproduce cognition at the behavioral and circuit levels through artificial neural networks. This thesis uses both natural and artificial approaches to study elements of emergent cognitive structure – within the brain and in behavior. A central and fascinating challenge for both brains and machines is episodic memory, or how we transform lived experiences into personal memories we can later recall. These memories shape how we perceive the world, how we learn, and constitute a fundamental fabric of our identity. As our lives unfold over time, our memories must preserve not just what happened, but the order in which events unfolded. Yet, the mechanisms through which the brain encodes this ordinal structure, and threads it back together during recollection, remain poorly understood. In the first part of this thesis, we recorded the electrophysiological activity of nearly a thousand individual neurons in the human brain while participants viewed sequences of naturalistic events. Surprisingly, we found neurons whose activity was selective for an event’s ordinal position – that is, whether the event occurred first, second or third within the sequence, revealing a neural code for event order. At the mesoscale, we analyzed local field potentials, focusing on the theta-band rhythms, a low-frequency brain oscillation thought to play a crucial role in memory. We found that theta-band activity varied systematically across the event sequence, and that the relationship between order selective neurons’ spike timing and the underlying theta rhythm during encoding predicts subsequent memory for order. Together, these findings point to a combined rate- and oscillation-based code that supports the organization and retrieval of temporally structured experience. These results, on the one hand, improve our understanding of episodic memory, and on the other, they shed light on mechanisms underlying physiological memory formation, which may go awry in several memory-related pathologies. The second part of this thesis turns to tasks where machines have made remarkable strides, asking just how human-like they have become. Artificial models have become increasingly successful at emulating aspects of human behavior. With advances in computer vision and large language models, artificial systems can now perform tasks once thought uniquely human – but how close are they to appearing human? This question captivated Alan Turing, who proposed an “imitation game” in which a judge tries to determine, from a written conversation, whether they are engaging with a human or a machine. Building on this idea, we developed Turing-like tests in the domains of vision and language. Humans and state-of-the-art models completed six tasks (three vision tasks and three language tasks), and separate judges – either human participants or simple machine classifiers – were asked to infer whether each response was produced by a person or by a model. We found that current models are approaching human-level imitability across several vision and language tasks, and that simple machine judges substantially outperform human judges at correctly detecting AI-generated outputs. Strikingly, imitability is largely independent of standard task-performance metrics, highlighting human-likeness as a distinct axis for model evaluation with practical implications for how we assess, trust, and attribute human versus machine-generated content. These two bodies of work illustrate two complementary windows into human cognition: one tracing the neural mechanisms that structure how we experience and remember sequences of events, and one mapping the behavioral landscape that current artificial systems must navigate to pass as human. More broadly, this thesis contributes to a growing synergy between neuroscience and artificial intelligence — one that may ultimately bring us closer to understanding, and replicating, the full richness of human cognition.