More Than Meets the Eye: Is It Feasible To Uncover Intent From Saccade Sequences?

Abstract

In 1967, Alfred Yarbus demonstrated that human eye movements are influenced by cognitive tasks, suggesting they can reveal an observer’s intent. While subsequent research produced mixed results regarding this claim, advancements in machine learning provide new ways to explore the predictability of intent from eye movement data. This thesis revisits this topic, focusing on whether saccade sequences can predict intent beyond random chance and what patterns models prioritize in these predictions. Our findings confirm that intent can be reliably predicted above chance when models are trained on targeted data, such as distinguishing between tasks where a target is present (TP) or absent (TA). Performance improvements were noted when the models specialized in either condition, although cross-scope generalization remained challenging. Temporal dependencies emerged as crucial, demonstrated by a significant accuracy drop when saccade sequences were reversed, highlighting the role of sequence order. Additionally, spatial occlusion experiments showed that specific image regions contribute differently to prediction confidence, with context-based models leveraging broader scenes and detail-based models focusing on identifiable objects. These results support Yarbus' hypothesis of purpose-driven eye movements, underscoring the intricate interplay of spatial and temporal elements in intent prediction.