Cognitive Brain Research 17 (2003) 406–418 www.elsevier.com / locate / cogbrainres

Research report

Brain networks for analyzing eye gaze
Christine I. Hooker a , * ,1 , Ken A. Paller a , Darren R. Gitelman b , Todd B. Parrish c , M.-Marsel Mesulam a , Paul J. Reber a
a b

Department of Psychology, Northwestern University, 2029 Sheridan Road, Evanston, IL 60208 -2710, USA Department of Neurology, Northwestern University, 2029 Sheridan Road, Evanston, IL 60208 -2710, USA c Department of Radiology, Northwestern University, 2029 Sheridan Road, Evanston, IL 60208 -2710, USA Accepted 31 March 2003

Abstract The eyes convey a wealth of information in social interactions. This information is analyzed by multiple brain networks, which we identified using functional magnetic resonance imaging (MRI). Subjects attempted to detect a particular directional cue provided either by gaze changes on an image of a face or by an arrow presented alone or by an arrow superimposed on the face. Another control condition was included in which the eyes moved without providing meaningful directional information. Activation of the superior temporal sulcus accompanied extracting directional information from gaze relative to directional information from an arrow and relative to eye motion without relevant directional information. Such selectivity for gaze processing was not observed in face-responsive fusiform regions. Brain activations were also investigated while subjects viewed the same face but attempted to detect when the eyes gazed directly at them. Most notably, amygdala activation was greater during periods when direct gaze never occurred than during periods when direct gaze occurred on 40% of the trials. In summary, our results suggest that increases in neural processing in the amygdala facilitate the analysis of gaze cues when a person is actively monitoring for emotional gaze events, whereas increases in neural processing in the superior temporal sulcus support the analysis of gaze cues that provide socially meaningful spatial information.  2003 Elsevier B.V. All rights reserved.
Theme: Neural basis of behavior Topic: Cognition Keywords: Superior temporal sulcus (STS); Amygdala; fMRI; Social cognition; Emotion; Face perception; Joint attention; Theory of mind; Autism; Schizophrenia

1. Introduction The eyes move not only in the service of visual perception but also to support communication by indicating direction of attention, intention, or emotion [7]. Infants stare longer at the eyes than other facial features [39] and they spontaneously follow someone else’s gaze as early as 10 weeks of age [28]. Likewise, adults tend to automatically shift their attention in the direction of another

*Corresponding author. Tel.: 11-510-642-8463; fax: 11-510-6439334. E-mail address: chooker@socrates.berkeley.edu (C.I. Hooker). 1 Current address: Department of Psychology and Helen Wills Neuroscience Institute, University of California, Berkeley, 4143 Tolman Hall, Berkeley, CA 94720-5050, USA. 0926-6410 / 03 / $ – see front matter  2003 Elsevier B.V. All rights reserved. doi:10.1016 / S0926-6410(03)00143-5

person’s gaze [20], and when this happens, the outcome is that both people are attending to the same thing. This phenomenon of joint attention has been shown to facilitate the development of language and social cognition in children, and to particularly facilitate theory of mind skills—the understanding of another person’s mental state [5,38,41]. In addition, developmental delays in gaze-following have been shown to predict a later diagnosis of autism [4], so the infant’s behavioral response to gaze cues has become an important developmental marker. Direct eye contact—when two people gaze directly at one another—is also an important aspect of gaze behavior [35]. Perceiving the direct gaze of another person, unlike averted gaze, indicates that the direction of attention is focused on the viewer. Perceiving eye contact directs and fixes attention on the observed face [23] and in visual

C.I. Hooker et al. / Cognitive Brain Research 17 (2003) 406–418

407

search paradigms, it is detected faster than averted gaze [52]. Eye contact has been shown to increase physiological response in social interactions [43], and the amount and quality of eye contact are considered important indicators of social and emotional functioning [35]. Poor eye contact is a specific diagnostic feature of autism [3] and a key component of the negative symptom syndrome in schizophrenia [2]. Thus, investigating neural mechanisms of gaze may provide key insights for understanding neurobiological factors that mediate social development, social interactions, and, ultimately, how dysfunctions in these mechanisms might be related to symptoms observed in disorders such as autism and schizophrenia. There is mounting evidence to suggest that specific regions of the temporal lobe, such as the fusiform gyrus, superior temporal sulcus (STS), and the amygdala, are involved in gaze processing [1,26,32,33,46,47,54]. Gaze is usually perceived in the context of a face, and faces are known to activate both the fusiform gyrus and the STS [1,32,46]. However, the fusiform gyrus responds more to whole faces and the STS responds more to facial features, particularly the eyes. Face identity judgments produce relatively more fusiform gyrus activity whereas gaze direction judgments of the same visual stimuli produce relatively more STS activity [26,27]. Furthermore, lesions in the region of the fusiform gyrus can produce prosopagnosia, the inability to recognize familiar faces [16,19,31,57]. Deficits in gaze direction discrimination are generally not found after fusiform damage but are found after STS damage [11]. Despite this evidence implicating the STS region in gaze perception, the exact nature of its contribution remains unresolved. Given that the STS responds to various kinds of biological motion, Haxby and colleagues [26] proposed that STS activity to gaze reflects processing of eyes as one of several movable facial features that is useful in social communication; in contrast, fusiform activity reflects processing of invariant facial features that are most useful for discerning personal identity. STS activation to eye and mouth movements [47] is consistent with this hypothesis, as is STS activation to passive viewing of averted and direct gaze [54] given that movable facial features have implied motion even when viewed as static pictures [37]. Another hypothesis about the STS, derived from neuronal recordings of STS activity in monkeys, emphasizes that this region processes cues about the direction of attention of others [44,45]. STS cells show varying activity to pictures of different head orientations and gaze directions but show maximum firing when head and gaze are oriented in the same direction. Perrett and colleagues interpreted these data as evidence that the cells respond to the direction of attention of the observed individual [44,45]. This idea has been especially influential because it relates to joint attention and the associated deficits in autism. However, people automatically shift their own attention in response to the directional information in gaze [20], so it

is difficult to separate perception of direction of attention of another from perception of the directional information inherent in that stimulus. Thus an alternative hypothesis is that the STS responds to any type of directional cue. This notion is supported by research showing activity in the STS and adjacent areas to directional attention cues that are not biological [34]. In addition, lesions in the posterior portion of the STS, e.g. the temporal parietal junction (TPJ), can compromise spatial attention skills [40]. Furthermore, during passive viewing of averted gaze [23], activity in the STS region correlates with activity in the intraparietal sulcus (IPS)—a brain area that has been consistently implicated in neural networks of spatial attention [14]. To obtain information about specific visual analyses taking place in STS, we designed an experiment to determine whether differential STS activity would be elicited by repetitive eye motion vs. eye motion providing relevant directional information vs. directional information from a nonfacial source. Similar STS responses to all types of eye motion would implicate a basic visual motion function, whereas preferential response to cues to direction of attention would implicate analyses more closely tied to the attentional relevance of the stimuli. The amygdala is also centrally involved in gaze processing. Patients with bilateral amygdala damage experience difficulty identifying gaze direction [56]. Amygdala activation measured with positron emission tomography (PET) has been reported to passive viewing of both direct and averted gaze [54] and to active detection of eye contact (bilateral amygdala) and averted gaze (left amygdala) [33]. These results illustrate that the amygdala is involved in monitoring gaze and suggest that the right amygdala is instrumental in the perception of direct gaze. However, it is still unclear from these studies whether the aymgdala is responding to the presence of direct gaze or the process of monitoring for its appearance. Prior fMRI gaze studies were not able to contribute to these ideas because the region of the amygdala was not scanned [27,47]. We investigated brain activations associated with gaze processing in two experiments. Although emotional facial expression was not of primary interest, we used happy and angry expressions in different blocks in both experiments. This design feature allowed us to minimize betweensubject variability in spontaneous judgments of facial expression that tends to occur with neutral faces [18], and also allowed us to investigate effects of emotional expression on gaze processing. In the first experiment we simulated the use of gaze as a cue to direction of attention. We directly tested whether regions such as STS would exhibit differential activation to gaze cues indicating direction of attention compared to nongaze cues providing directional information or to eye motion not providing directional information. In the primary condition, subjects viewed a face while the eyes of the face shifted in a fashion that implied eye motion, as if

408

C.I. Hooker et al. / Cognitive Brain Research 17 (2003) 406–418

the individual was looking sequentially at different spatial locations. This Gaze task required that subjects discriminate whether or not the eyes gazed at a particular target location. Whereas prior fMRI experiments typically included only left and right gaze, we included ten different gaze positions such that fine-grained discriminations were required, thus approximating a more demanding and ecologically valid perceptual analysis of gaze cues. In control tasks an arrow, isolated or superimposed on a face,

provided directional information instead of the eyes, or the eyes moved without providing relevant directional information (see Fig. 1 for examples of stimuli). In a second experiment, we sought to determine whether amygdala activation was specifically associated with viewing direct gaze or with the act of monitoring gaze. Subjects identified direct gaze among direct and averted gaze trials, and trials were blocked so that direct gaze occurred on either 40% or 0% of the trials.

Fig. 1. Average signal change across participants in Experiment 1, displayed on sagittal slices of a single subject’s structural MRI scan (x5644). Only activations above threshold are shown, as indicated by the legend at the upper left. (a) Activations across all runs, collapsing across the different control conditions. (b) Activations for analyzing directional information from gaze versus directional information from an arrow (Gaze vs. Arrow). (c) Activations for analyzing directional information from gaze versus directional information from an arrow on a face (Gaze vs. Face-Arrow). (d) Activations for eye motion that provides directional information versus eye motion that does not (Gaze vs. Eye-Motion).

C.I. Hooker et al. / Cognitive Brain Research 17 (2003) 406–418

409

2. Materials and methods

2.1. Tasks
In the Gaze task, a face remained continuously on the screen and the eyes of the face looked to a certain spatial location for 300 ms and looked back to the viewer for 900 ms. The subject pressed one button when the eyes shifted to a target location and another button when the eyes shifted towards any nontarget location. All gaze cues were based on a clock template such that the target location assigned on each run corresponded to one quadrant of the clock (i.e. 1 o’clock and 2 o’clock for upper right quadrant, 4 o’clock and 5 o’clock for lower right quadrant, 7 o’clock and 8 o’clock for lower left quadrant, 10 o’clock and 11 o’clock for upper left quadrant). Eye positions for 3 o’clock and 9 o’clock always served as nontargets. Eye positions for 6 o’clock and 12 o’clock were not presented. Each eye position occurred equally often, but in a randomized order, such that target probability was 20%. In the Arrow task, a dot remained centrally on the screen and an arrow jutted out from the dot to indicate a spatial location on each trial. In the Face Arrow task, the arrow indicated spatial locations in the same manner but was superimposed in between the eyes of the face, which remained fixated forward. In the Eye Motion task, the eyes of the face moved inward toward the nose in a cross-eyed fashion on every trial. The target in this condition occurred when the eyes simultaneously changed to a slightly lighter shade of gray. The presentation rate (1 trial every 1200 ms) and the 1:4 ratio of targets to nontargets was equivalent in all tasks. In Experiment 2, subjects were instructed to detect when the eyes of the stimulus face were looking directly at them. The stimuli in this task thus closely resembled those in the Gaze task, except that direct gaze was not used between gaze cues. After the eyes looked at a spatial location for 300 ms, the eyes closed for 900 ms. Also, trials with direct gaze were included along with the 10 averted eye positions corresponding to the clock face. Trials were blocked so that direct gaze occurred on either 0% or 40% of the trials. The face was displayed with a static happy expression in half of the blocks and with a static angry expression in the other half of the blocks. The intensity of the two emotional expressions was roughly equivalent, based on ratings made by each subject using a 5-point scale. The emotional expression remained constant throughout each block of trials. The different eye positions of each trial were created by ‘cutting and pasting’ the eyes from another picture into the stimulus face. The glasses on the model ensured that the eyes were always placed in the same location. Subjects practiced each task before entering the scanner. Practice stimuli for the Gaze task, Arrow task, and Face Arrow task were shown surrounded by a clock outline in order to define the spatial location of the targets. Clock numbers which were on the practice stimuli were not

present in the main experiments when subjects performed the tasks in the scanner. Subjects were instructed to keep their eyes fixated during the experiment. In Experiment 1, Gaze was paired with each control condition in a different run (Gaze and Arrow; Gaze and Face Arrow; Gaze and Eye Motion), and there were two runs of each task pair. Experiment 2 consisted of two runs with alternating blocks of 0% and 40% direct gaze. For both experiments, each run consisted of eight alternating blocks of trials, and each block consisted of 30 trials and lasted 36 s. Run and task order were counterbalanced across subjects within each experiment. Experiment 1 preceded Experiment 2 for all subjects. The opposite order would have likely given the direct gaze cue undue relevance in Experiment 1, because direct gaze was meant to be a ‘neutral’ stimulus in Experiment 1, but was a ‘salient’ stimulus in Experiment 2. The target detection tasks were designed to ensure that subjects would keep attention focused on relevant aspects of the stimuli. Behavioral results indicated that subjects attended to the stimuli in all conditions [mean percent accuracy (min–max): Gaze564% (40–85), Face Arrow5 78% (50–96), Arrow583% (53–95), Eye Motion564% (35–92), Direct Gaze566% (22–93)]. Debriefing suggested that most of the errors were not due to failures to accurately discriminate target and nontarget events. Rather, many responses were made too late. In particular, the high proportion of nontargets encouraged a repetitive motor response that became habitual, such that extra time was needed to disengage from this habitual response. Many correct responses did not occur prior to the beginning of the next trial. Unfortunately, late responses were not registered and so those trials were scored as incorrect. Most importantly for present purposes, however, participants’ reports suggest that their attention was actively engaged in the tasks even if they were making errors, and that they were differentially attending to different stimulus dimensions according to the task requirements.

2.2. Imaging
We scanned 10 healthy, right-handed volunteers (six females and four males with a mean age of 24 years, S.D.53) on a Siemens 1.5 Tesla Vision Scanner. The institutional review board of Northwestern University approved the protocol. Each subject gave informed consent and received monetary compensation for their participation. A vacuum bag assisted in keeping the subject’s head stationary throughout the scan. Subjects registered their task responses using a button box held in their right hand. Images were projected onto a custom-designed, nonmagnetic rear projection screen. Subjects viewed the screen, located approximately 54 in. away, via a mirror placed above their eyes. Functional T2* weighted images were acquired using echo-planar imaging with a 3000 ms TR, 40 ms TE, 908 flip angle, 240 mm FOV, a 64364 pixel

410

C.I. Hooker et al. / Cognitive Brain Research 17 (2003) 406–418

matrix for 24 contiguous 6-mm thick axial slices with a resulting voxel size of 3.7533.7536 mm. Each run lasted 303 s including 9 s of initial fixation (with a static stimulus face on the screen), eight alternating 36-s task blocks and 6 s of an ending fixation point. In all functional runs, the MR signal was allowed to achieve equilibrium over four initial scans that were excluded from the analysis. Thus data from each scanning run consisted of 97 images. Anatomic images were acquired using a T1-weighted 3D FLASH sequence with a 22 ms TR, 5.6 ms TE, 258 flip angle, 240 mm FOV, and a 2563256 pixel matrix, with 1-mm thick axial slices.

2.3. Data analysis
MRI data were analyzed using AFNI software [15]. Images were co-registered through time using a threedimensional registration algorithm. Within each run, voxels containing a signal change of greater than 10% in one repetition (3 s) were assumed to be contaminated by motion or other artifact and those voxels were eliminated from further analysis. Linear drift over the time course of each run was removed. Each slice was spatially smoothed using a Gaussian filter (full-width half-maximum (FWHM)57.5 mm). Data from the 10 subjects were normalized using the Montreal Neurological Institute Autoreg program [13]. Functional runs were visually checked for artifact and motion contamination. The last run for two subjects was dropped from analyses due to extensive signal loss from motion contamination, such that only one run was entered in the analysis for Experiment 2. Brain areas exhibiting task-specific activity were identified by correlating the observed time course of each voxel against an idealized reference function derived from the alternating task blocks and adjusted to reflect the lag between neural activity and hemodynamic response. The signal change for each subject in each run was averaged in a random effects analysis to identify areas of significant activation across subjects for each task contrast. This group analysis was then thresholded to identify voxels that reached the minimum statistical requirement of t(9)53.25, P,0.01 (uncorrected) occurring within a cluster of adjacent voxels at least 650 mm 3 in volume (.41 voxels in the normalized data or approximately 8 voxels in the original anatomical space). All task comparisons and stated activations used this statistical threshold.

3. Results

3.1. Experiment 1
A comparison between the Gaze task and all three control conditions combined gave an overview of neural regions responsive to gaze cues that indicate the direction of another person’s attention. This combined contrast

controls for visual processing of faces per se, for nonmeaningful eye motion, and for generic cognitive demands of extracting directional information from a stimulus. The neural network for gaze processing isolated by this analysis included three key regions: the posterior portion of the STS bilaterally, corresponding to Brodmann’s areas 22 and 39 (BA 22,39), the same area previously identified in gaze studies [27,47]; a right prefrontal region centered in the frontal eye fields (BA 8,9); and a ventral prefrontal region centered in the inferior frontal gyrus (BA 44,45). These activations are shown in Fig. 1 and listed in Table 1. However, it is essential to also evaluate contrasts with each control condition individually. When the Gaze condition was compared separately to each control condition, STS activation was also evident (Fig. 1 and Table 2). STS was more active on the right with the control condition involving analysis of direction from an arrow (Gaze–Arrow). STS was more active bilaterally with the control condition in which the arrow was superimposed on the face (Gaze–Face Arrow). And STS was more active on the left with the control condition involving eye motion that did not provide directional information (Gaze–Eye Motion). Right fusiform gyrus and multiple areas of bilateral prefrontal cortex (PFC) were activated in the Gaze task compared to both arrow tasks. However, this pattern was reversed for the Eye Motion task such that both the fusiform gyrus and the dorsolateral prefrontal cortex (primarily BA 46) were more active in the Eye Motion task than in the Gaze task. To further quantify STS, fusiform gyrus, and PFC activations, we performed a region-of-interest (ROI) analysis for these three regions. ROIs were defined by creating a union of all three contrasts. This procedure identified brain regions activated by either the Gaze task or the control condition in each contrast. The average signal change across participants was computed bilaterally for STS, fusiform gyrus, and PFC (BA46 and posterior BA10, extending in the left hemisphere to superior BA44, 45). Fig. 2 shows the amount of signal change for each region in each contrast. This single analysis confirms the pattern of activity seen in the whole-brain contrasts. A repeatedmeasures ANOVA showed a significant Region by Task interaction [F(4,36)522, P,0.001], substantiating the different patterns. To further elucidate this interaction, we performed one group t-tests for each task contrast in each region of interest. These effects all indicated greater responses in the Gaze condition than in the control condition, with the exception of the activations in the contrast with the Eye Motion condition, where the fusiform and PFC responses were larger in the Eye Motion condition. Specifically, significant activations for the Gaze task in the Gaze–Arrow contrast were shown in bilateral fusiform and right PFC, in the Gaze–Face Arrow contrast in bilateral STS, right fusiform, and bilateral PFC, and for the Gaze–Eye Motion contrast in left STS. There was significant activation for the Eye Motion condition in the

C.I. Hooker et al. / Cognitive Brain Research 17 (2003) 406–418 Table 1 Significant activations for Gaze vs. combined control conditions, using all runs in Experiment 1 Contrast Brain regions Brodmann’s Area Talairach coordinates (mm) x 247 46 4 37 y 218 21 23 18 z 60 52 44 4

411

Volume (mm 3 )

Gaze.Control

Frontal Left Precentral Gyrus Right Precentral Sulcus / Precentral Gyrus Bilateral Superior Frontal Gyrus Right Inferior Frontal Gyrus Temporal Bilateral Superior Temporal Sulcus Right Left Parietal Left Middle Cingulate Occipital Bilateral Occipital Cortex Right Left Frontal Left Superior Frontal Gyrus Temporal /Occipital Bilateral Occipital Temporal Gyrus Right Left Parietal Bilateral Inferior Parietal Gyrus Right Left

4 4, 6 8 44, 45 22, 39

1297 3734 2172 4016

50 249 24 17, 18 21 221 9 36 28 230 19 32 243 215 219

245 261 19

16 16 25

6469 1188 703

297 294 55

3 26 30

7906 12016 906

Control.Gaze

243 248

217 217

1219 922

285 287

22 20

984 1062

Gaze–Eye Motion contrast in the bilateral fusiform, and right PFC (see Fig. 2 for response magnitude). In addition, we examined the relationship of the hemodynamic response patterns in the six regions of interest. For this analysis, we correlated the time series for each region of interest within each subject, and then identified the strength of the correlations across all subjects. This showed a significant correlation of brain activity between the specified regions. All regions of interest (STS, FFA and PFC bilaterally) were significantly correlated (t(9). 2.82, P,0.05), with the exception of the left FFA and the right PFC. Whole brain results were also analyzed as a function of whether a happy or angry face appeared in the Gaze task. Angry faces produced more activation than happy faces in gaze processing regions, including the STS and prefrontal cortex, primarily in the right hemisphere (Table 3). The effect of expression was strongest in the Gaze task, but angry faces also elicited more activity in the control conditions (Face Arrow and Eye Motion).

nent (Fig. 3). Additional activation in the left temporal lobe was lateral to the amygdala, centered in the inferior temporal gyrus. Significant activations for direct gaze blocks were concentrated in frontal cortex (Table 4).

4. Discussion Patterns of activation revealed with fMRI in these two experiments showed that distinct contributions arise from brain networks in four regions thought to be involved in gaze processing. Face and gaze perception is accomplished via contributions from the STS, amygdala, fusiform gyrus, and PFC. Several insights into how gaze information is extracted can be gained by considering the distinct ways in which facial input is analyzed in these regions.

4.1. Gaze as a directional cue 4.1.1. STS One key portion of the distributed networks for gaze processing is centered in the posterior STS region of the temporal lobe. The STS was more active when subjects analyzed direction from gaze than when they analyzed comparable directional information from an arrow, even when that arrow was superimposed on a face with the eyes clearly visible. Furthermore, the STS was more active to

3.2. Experiment 2
In the comparison between blocks in which the direct gaze target occurred on 40% vs. 0% of the trials, the right amygdala was more active during the 0% condition, when the unfulfilled anticipation of direct gaze was most promi-

412

C.I. Hooker et al. / Cognitive Brain Research 17 (2003) 406–418

Table 2 Significant activations for Gaze vs. each control condition separately in Experiment 1 Contrast Gaze.Arrow Brain regions Frontal Right Precentral Gyrus Left Middle Frontal Gyrus Right Inferior Frontal Gyrus Temporal Right Fusiform Gyrus Right Superior Temporal Sulcus Occipital Bilateral Occipital Cortex Right Left Arrow.Gaze Frontal Left Superior Frontal Gyrus (Anterior) Bilateral Superior Frontal Gyrus Temporal Right Middle Temporal Gyrus (Anterior) Parietal Left Inferior Parietal Gyrus Occipital Bilateral Precuneous Left Cerebellum Gaze.Face Arrow Frontal Right Precentral Gyrus Bilateral Superior Frontal Gyrus Right Middle Frontal Gyrus Left Middle Frontal Sulcus Right Inferior Frontal Gyrus Operculum Triangular Anterior portion Right Inferior Frontal Sulcus Temporal Bilateral Superior Temporal Sulcus Right Left Right Middle Temporal Gyrus Right Fusiform Gyrus Parietal Right Inferior Parietal Lobule Face Arrow.Gaze Parietal Left Intraparietal Sulcus Occipital Bilateral Lateral Occipital Cortex Right Left Gaze.Eye Motion Frontal Right Precentral Gyrus Left Post Central Sulcus Left Inferior Frontal Sulcus Temporal Left Superior Temporal Sulcus Left Parahippocampal Gyrus Right Parahippocampal Region Parietal Left Posterior Cingulate Occipital Right Mid / Superior Occipital Cortex Brodmann’s Area 6 10 46 20, 37 22, 39 17, 18, 19 25 223 294 294 1 25 20312 26156 Talairach coordinates (mm) x 49 241 44 46 45 y 3 50 20 255 244 z 44 19 16 219 11 Volume (mm 3 ) 2156 1219 578 a 1188 766

9 8 21 39, 19 30, 7, 19

211 0 58 246 23 234

51 65 212 283 271 285

35 14 220 27 10 237

2984 1234 3547 984 19984 656

4, 6 6, 8 46 46 44, 45 10 45, 46 44, 46 22 22 20, 37 7

49 1 48 253 49 45 30 42

4 21 33 16 11 43 21 13

46 46 17 15 0 25 23 23

2766 1000 2062 6547 2766 2453 1188 969

54 253 50 45 47

251 255 225 257 257

11 15 24 220 50

16422 5203 969 2984 750

7 39, 19

222

268

53

3141

36 238

289 291

18 19

1078 1312

6 7 44 39 28 28 23 7

22 237 234 247 213 19 224 11

28 252 210 267 233 222 247 270

51 66 28 17 217 222 17 54

828 781 703 4875 1625 1156 969 1766

C.I. Hooker et al. / Cognitive Brain Research 17 (2003) 406–418 Table 2. Continued Contrast Eye Motion.Gaze Brain regions Frontal Bilateral Middle Frontal Gyrus Right Left Temporal /Occipital Bilat. Occipital Cortex–Fusiform Gyrus Right Left
a

413

Brodmann’s Area 46

Talairach coordinates (mm) x y z

Volume (mm 3 )

47 247 17, 20, 37 36 238

32 29

11 9

3953 531 a

273 277

22 24

8281 12125

This cluster is below volume threshold and is listed because of theoretical importance.

eye motion that indicated direction of attention than eye motion that did not. The most likely functional role for STS activity is therefore in the analysis of meaningful eye motion that can indicate information of social importance such as the intentions of another person or their direction of attention. Our results address three hypotheses concerning the function of the STS, namely, that the STS is specifically responsive to moving or movable facial features [26], to the direction of attention of another person [44,45], or to any type of directional cue. Our findings support the hypothesis that STS responds to gaze cues that indicate the direction of attention of another individual. The finding that eye motion activated STS more when it provided directional information than when it did not implies that this region is sensitive not simply to motion cues but that it preferentially responds to certain types of eye motion. Eye

motion that provided task-relevant information about spatial location was more meaningful. Subjects attended to moving eyes in both the Gaze and Eye Motion tasks. Therefore, STS activity cannot be described simply as a response to this movable facial feature. A satisfactory explanation for increased STS activity in the Gaze task relative to the Eye Motion task must go beyond hypotheses emphasizing either movable facial features or generic biological motion. The hypothesis that STS is central to the analysis of gaze indicators of direction of attention does not rule out a broader role in analyzing various biological signals important for human social communication [9]. STS activity has been observed in response to several types of human movements, including hand and whole-body movements [8,24,25,48]. STS activity in these contexts may reflect the analysis of biological cues that provide meaningful social

Fig. 2. Percent signal change in functionally defined regions of interest for each contrast in Experiment 1. Activations greater than zero at P,0.05 are denoted by *.

414

C.I. Hooker et al. / Cognitive Brain Research 17 (2003) 406–418

Table 3 Significant activations between happy and angry faces Condition and contrast GAZE: Angry.Happy Brain regions Frontal Bilateral Middle Frontal Gyrus Right Anterior Portion Right Middle Portion Left Anterior Portion Left Middle Portion Left Middle Frontal Sulcus Anterior Portion Middle / Inferior Portion Temporal Bilateral Superior Temporal Sulcus Right Left Bilateral Middle Temporal Sulcus Right Left Occipital Bilateral Precuneous Bilateral Precuneous (Inferior) FACE ARROW: Angry.Happy Frontal Left Post Central Gyrus and Sulcus Temporal Left Posterior Lateral Sulcus Occipital Right Occipital Lobe, Cuneous FACE ARROW: Happy.Angry EYE MOTION: Angry.Happy Right Medial Cerebellum Frontal Right Middle Frontal Sulcus Right Middle Frontal Gyrus Left Middle Frontal Sulcus and Gyrus Frontal Right Middle Cingulate Frontal Right Anterior Cingulate Brodmann’s Area Talairach coordinates (mm) x y z Volume (mm 3 )

46 8, 9 8, 9 9 10, 46 44 22, 39

48 31 247 249 247 240

27 26 15 23 39 5

21 45 38 24 2 26

3266 3172 1266 688 5578 1984 35641 a

42 235 21 54 260 7 30 1 9

257 267 252 254 267 260

26 31 25 25 42 16 1578 703 35641 1969

4, 6 40 18, 19

234 238 15 11

243 236 290 255

60 16 32 237

859 719 812 859

44, 45, 46 46, 9 10

34 45 244

40 32 51

8 27 2

4047 969 1375

DIRECT GAZE: Angry.Happy

23,24

5

214

24

2562

DIRECT GAZE: Happy.Angry
a

12,32

6

47

27

3188

This cluster extended from the STS up through the IPS bilaterally.

Fig. 3. Average signal change across participants for Experiment 2, showing the contrast between 0% and 40% Direct Gaze conditions, displayed on a coronal ( y54) and a sagittal (x526) slice from a single subject’s structural MRI scan.

C.I. Hooker et al. / Cognitive Brain Research 17 (2003) 406–418 Table 4 Significant activations in Experiment 2 Contrast Brain regions Frontal Left Superior Frontal Gyrus Temporal Right Amygdala Left Inferior Temporal Gyrus Direct.Anticipation Frontal Right Precentral Sulcus Left Post Central Gyrus Bilateral Superior Frontal Gyrus Parietal Right Intraparietal Sulcus Left Cerebellum Brodmann’s Area Talairach coordinates (mm) x 219 28 248 y 43 3 27 z 49 227 231 Volume (mm 3 )

415

Anticipation.Direct

9

1891 672 672

20, 21

4, 6 2, 3, 5 6 7

39 250 1 36 240

24 233 8 260 253

52 55 49 46 237

2344 1828 1391 734 953

signals. In other words, STS may analyze meaningful biological motion, where ‘meaning’ depends on the social situation within which the movement occurs. This idea is supported by several studies comparing meaningful and non-meaningful human movements. When videos of American Sign Language were viewed, the STS was more active for viewers who understood the signals than for those who didn’t [42]. STS activation has also been reported in response to possible versus impossible human movements [50] as well as meaningful vs. non-meaningful hand motions [17]. The extant data thus indicate that STS processing is concerned with more than just perceptual aspects of moving or movable body parts. Rather, networks in this brain region may analyze gaze and other movements to the extent that these cues meaningfully contribute to social communication. Our findings suggest that achieving joint attention, a pivotal skill in social cognition, is facilitated by the analysis of sensory cues in the STS. Furthermore, given that gaze provides a highly informative window into mental state [7,51], the STS could be part of a larger neural network mediating theory of mind [10].

4.1.2. Fusiform gyrus and dorsolateral PFC Our findings demonstrated a different functional role for the fusiform gyrus. The pattern of fusiform activations across the three contrasts illustrates that this region, while engaged in face and gaze processing, is not specifically responsive to directional gaze information. Both fusiform gyrus and dorsolateral portion of the PFC (BA 46) were more active in the Eye Motion task than in the Gaze task (Fig. 1d). Given that both tasks required attention to the eyes, fusiform activity cannot simply be explained as a response to invariant aspects of a face. There are several reasons why the Eye Motion task may have provoked such strong fusiform activity.

One possibility is that fusiform and dorsolateral PFC are engaged in analyzing perceptual properties of faces. In the Eye Motion task, subjects attended to moving eyes while vigilant for a gray-scale color change of the pupil. Perceptual processing of these facial features could have provoked fusiform activity. Previous fMRI studies indicate that color perception provokes activity in inferior temporal regions that are adjacent to and partially overlapping with face sensitive areas [12]. Because subjects here were attending to color within the face, inferior temporal regions sensitive to both faces and color may have been active. The dorsolateral PFC has been shown to modulate activity in visual cortex [6], so the dorsolateral PFC activity in this task could reflect the monitoring and identification of perceptual changes in the face stimuli. Another possibility is that fusiform and dorsolateral PFC responses may reflect stimulus novelty. Given that people seldom view eyes moving repetitively in a cross-eyed fashion, the novelty of this perceptual stimulus may have increased activity in the fusiform region and visual cortex generally. The PFC region activated in this task (BA 46) is likewise active in response to novel stimuli [36]. However a novelty hypothesis cannot explain the prominent activity of both fusiform and this PFC region for Gaze as compared to the Arrow and Face Arrow conditions. The finding that the fusiform gyrus was more active when subjects were analyzing direction from gaze than when analyzing direction from an arrow, by itself or superimposed on a face, is consonant with other findings that the fusiform face responsive area is more active during selective attention to faces than selective attention to objects [26,31,32,46]. Our study adds to these data by showing that selective attention to a specific facial feature can also produce differential fusiform activation. Clearly the fusiform response to a face is not uniform and automatic but rather is sensitive to attention [55]. The

416

C.I. Hooker et al. / Cognitive Brain Research 17 (2003) 406–418

present data also support prior suggestions that this fusiform area is relatively less important for gaze perception than for facial perception in general.

4.1.3. Medial and ventral PFC Medial and lateral portions of the superior frontal gyrus (BA 8,9), precentral gyrus (BA 4,6) and inferior frontal gyrus (BA 44, 45) were active to meaningful gaze cues while controlling for eye motion and nonbiological directional information (Table 1; Fig. 1a). These areas may facilitate understanding another person’s mental state. The superior frontal gyrus (BA 8,9) has been consistently activated in a variety of theory of mind tasks [22,49]. In addition, the precentral gyrus and inferior frontal gyrus have been identified as ‘mirror system’ areas [29]—brain regions that are active to both the observation and execution of an action. This neural ‘mirroring’ may facilitate our understanding of another person’s behavior by simulating the observed action [48]. Because we did not specifically manipulate theory of mind processes in this experiment, it is unclear whether these regions are engaged in analyzing mental state. However, future studies aimed specifically at identifying mental state from gaze cues may be helpful in further identifying the contribution of medial and ventral PFC to gaze processing.

idea that the amygdala is involved in monitoring gaze [33]. Moreover, amygdala activity may be heightened when a person is particularly vigilant for direct gaze. Specifically, the intriguing pattern of amygdala activity revealed in the present design suggests that the right amygdala responds less to the experience of direct gaze per se than to circumstances in which one is awaiting such social contact to happen momentarily. Sensory processing of gaze information is thus not the key factor eliciting amygdala activity. This view of the amygdala as important for sensory monitoring provides a reinterpretation of previous data showing amygdala activity to various kinds of gaze cues [54] and gaze tasks [33]. Our results support the general notion that the amygdala is part of a vigilance system that can facilitate the analysis of sensory input for emotionally or socially salient information [53].

Acknowledgements This research was supported by an NIMH Predoctoral Fellowship F31-MH12982 (CIH). The authors would like to thank Rick Zinbarg, William Revelle, and Marcia Grabowecky for helpful suggestions on experimental design, Katherine Byrne for assistance with stimuli production, and Joseph Coulson for assistance with manuscript preparation.

4.2. Emotion effects
Angry faces in the Gaze task elicited more activity in the STS region than did happy faces, and this activation extended from the STS dorsally through the intraparietal sulcus (IPS). Although this finding was not predicted, this influence of facial expression on STS activity adds support to our interpretation that the STS is responsive to gaze cues that are meaningful to social interaction. Gaze that provides information about the direction of attention of an angry person may, perhaps automatically, be more meaningful than that same information from a happy person [21,30]. Understanding where an angry person is attending and what they might be angry about has immediate relevance to one’s own ability to successfully navigate a potentially threatening or aversive environment. In support of this idea, the IPS activity may reflect recruitment of neural networks supporting spatial attention. References
[1] T. Allison, A. Puce, G. McCarthy, Social perception from visual cues: role of the STS region, Trends Cognit. Sci. 4 (2000) 267–278. [2] N.C. Andreasen, Negative symptoms in schizophrenia. Definition and reliability, Arch. Gen. Psychiatry 39 (1982) 784–788. [3] American Psychiatric Association (APA), Diagnostic and Statistical Manual of Mental Disorders, 4th Edition, American Psychiatric Association, Washington, DC, 1994. [4] G. Baird, T. Charman, S. Baron-Cohen, A. Cox, J. Swettenham, S. Wheelwright, A. Drew, A screening instrument for autism at 18 months of age: a 6-year follow-up study, J. Am. Acad. Child Adolescent Psychiatry 39 (2000) 694–702. [5] D. Baldwin, Understanding the link between joint attention and language, in: C. Moore, P.J. Dunham (Eds.), Joint Attention: Its Origins and Role in Development, Lawrence Erlbaum Associates, Hillsdale, NJ, 1995. [6] F. Barcelo, S. Suwazono, R.T. Knight, Prefrontal modulation of visual processing in humans, Nat. Neurosci. 3 (2000) 399–403. [7] S. Baron-Cohen, Mindblindness: An Essay On Autism and Theory of Mind, MIT Press, Cambridge, MA, 1995. [8] E. Bonda, M. Petrides, D. Ostry, A. Evans, Specific involvement of human parietal systems and the amygdala in the perception of biological motion, J. Neurosci. 16 (1996) 3737–3744. [9] L. Brothers, The social brain: a project for integrating primate behaviour and neurophysiology in a new domain, Concep. Neurosci. 1 (1990) 27–51. [10] A.J. Calder, A.D. Lawrence, J. Keane, S.K. Scott, A.M. Owen, I. Christoffels, A.W. Young, Reading the mind from eye gaze, Neuropsychologia 40 (2002) 1129–1138.

4.3. Direct gaze
The aim of Experiment 2 was to disentangle monitoring for the appearance of direct gaze from actual observation of direct gaze. During the direct gaze detection task, the amygdala was more active during blocks when direct gaze never occurred than it was during blocks when it occurred on 40% of the trials. These results are consistent with the

C.I. Hooker et al. / Cognitive Brain Research 17 (2003) 406–418 [11] R. Campbell, C. Heywood, A. Cowey, M. Regard, T. Landis, Sensitivity to eye gaze in prosopagnosia patients and monkeys with superior temporal sulcus ablation, Neuropsychologia 28 (1990) 1123–1142. [12] V.P. Clark, R. Parasuraman, K. Keil, R. Kulansky, S. Fannon, J.M. Maisog, L.G. Ungerleider, J.V. Haxby, Selective attention to face identity and color studied with fMRI, Hum. Brain Mapping 5 (1997) 293–297. [13] D.L. Collins, P. Neelin, T.M. Peters, A.C. Evans, Automatic 3D inter-subject registration of MR volumetric data in standardized Talairach space, J. Comput. Assisted Tomogr. 18 (1994) 192–205. [14] M. Corbetta, Frontoparietal cortical networks for directing attention and the eye to visual locations: identical, independent, or overlapping neural systems?, Proc. Natl. Acad. Sci. 95 (1998) 831–838. [15] R.W. Cox, AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages, Comput. Biomed. Res. 29 (1996) 162–173. [16] E. De Renzi, P. Faglioni, D. Grossi, P. Nichelli, Apperceptive and associative forms of prosopagnosia, Cortex 27 (1991) 213–221. [17] J. Decety, J. Grezes, N. Costes, D. Perani, M. Jeannerod, E. Procyk, F. Grassi, F. Fazio, Brain activity during observation of actions. Influence of action content and subject’s strategy, Brain 120 (1997) 1763–1777. [18] P. Ekman, W.V. Friesen, Pictures of Facial Affect, Consulting Psychologists Press, Palo Alto, CA, 1976. [19] M.J. Farah, K.D. Wilson, H.M. Drain, J.R. Tanaka, The inverted face inversion effect in prosopagnosia: evidence for mandatory, face-specific perceptual mechanisms, Vis. Res. 35 (1995) 2089– 2093. [20] C.K. Friesen, A. Kingstone, The eyes have it! Reflexive orienting is triggered by nonpredictive gaze, Psychonom. Bull. Rev. 5 (1998) 490–495. [21] L.M. Frodi, M.E. Lamb, L.A. Leavitt, W.L. Donovan, Fathers’ and mothers’ responses to infants’ smiles and cries, Infant Behav. Dev. 1 (1978) 187–198. [22] H.L. Gallagher, C.D. Frith, Functional imaging of ‘theory of mind’, Trends Cognit. Sci. 7 (2003) 77–83. [23] N. George, J. Driver, R. Dolan, Seen gaze-direction modulates fusiform activity and its coupling with other brain areas during face processing, Neuroimage 13 (2001) 1102–1112. [24] J. Grezes, N. Costes, J. Decety, The effects of learning and intention on the neural network involved in the perception of meaningless actions, Brain 122 (1999) 1875–1887. [25] E. Grossman, M. Donnelley, R. Price, D. Pickens, V. Morgan, G. Neighbor, R. Blake, Brain areas involved in the perception of biological motion, J. Cognit. Neurosci. 12 (2000) 711–720. [26] J.V. Haxby, E.A. Hoffman, I.M. Gobbini, The distributed human neural system for face perception, Trends Cognit. Sci. 4 (2000) 223–233. [27] E.A. Hoffman, J.V. Haxby, Distinct representations of eye gaze and identity in the distributed human neural system for face perception, Nat. Neurosci. 3 (2000) 80–84. [28] B.M. Hood, J.D. Willen, J. Driver, Adult’s eyes trigger shifts of visual attention in human infants, Psychol. Sci. 9 (1998) 131–134. [29] M. Iacoboni, R.P. Woods, M. Brass, H. Bekkering, J.C. Mazziotta, G. Rizzolatti, Cortical mechanisms of human imitation, Science 286 (1999) 2526–2528. [30] D. Kahneman, A. Tversky, Choices, values, and frames, Am. Psychol. 39 (1984) 341–350. [31] N. Kanwisher, Domain specificity in face perception, Nat. Neurosci. 3 (2000) 759–763. [32] N. Kanwisher, J. McDermott, M.M. Chun, The fusiform face area: a module in human extrastriate cortex specialized for face perception, J. Neurosci. 17 (1997) 4302–4311. [33] R. Kawashima, M. Sugiura, T. Kato, A. Nakamura, K. Hatano, K. Ito, H. Fukuda, S. Kojima, K. Nakamura, The human amygdala plays an important role in gaze monitoring. A PET study, Brain 122 (1999) 779–783.

417

[34] Y.H. Kim, D.R. Gitelman, A.C. Nobre, T.B. Parrish, K.S. LaBar, M.M. Mesulam, The large-scale neural network for spatial attention displays multifunctional overlap but differential asymmetry, Neuroimage 9 (1999) 269–277. [35] C.L. Kleinke, Gaze and eye contact: a research review, Psychol. Bull. 100 (1986) 78–100. [36] R.T. Knight, Decreased response to novel stimuli after prefrontal lesions in man, Electroencephalogr. Clin. Neurophysiol. 59 (1984) 9–20. [37] Z. Kourtzi, N. Kanwisher, Activation in human MT / MST by static images with implied motion, J. Cognit. Neurosci. 12 (2000) 48–55. [38] K. Loveland, S. Landry, Joint attention and language in autism and developmental language delays, J. Autism Dev. Disord. 16 (1986) 335–349. [39] D. Maurer, Infants perception of facedness, in: T. Field, N. Fox (Eds.), Social Perception of Infants, Ablex, Norwood, NJ, 1985. [40] M.M. Mesulam, Spatial attention and neglect: parietal, frontal and cingulate contributions to the mental representation and attentional targeting of salient extrapersonal events [published erratum appears in Philos Trans R Soc Lond B Biol Sci 1999 Dec 29;354(1352):2083], Philos. Trans. R. Soc. Lond. B: Biol. Sci. 354 (1999) 1325–1346. [41] P. Mundy, M. Crowson, Joint attention an early social communication: Implications for research on intervention with autism, J. Autism Dev. Disord. 27 (1997) 653–676. [42] H.J. Neville, D. Bavelier, D. Corina, J. Rauschecker, A. Karni, A. Lalwani, A. Braun, V. Clark, P. Jezzard, R. Turner, Cerebral organization for language in deaf and hearing subjects: biological constraints and effects of experience, Proc. Natl. Acad. Sci. USA 95 (1998) 922–929. [43] K. Nichols, B. Champness, Eye Gaze and GSR, J. Exp. Soc. Psychol. 7 (1971) 623–626. [44] D.I. Perrett, J.K. Hietanen, M.W. Oram, P.J. Benson, Organization and functions of cells responsive to faces in the temporal cortex, Philos. Trans. R. Soc. Lond. B: Biol. Sci. 335 (1992) 23–30. [45] D.I. Perrett, P.A. Smith, D.D. Potter, A.J. Mistlin, A.S. Head, A.D. Milner, M.A. Jeeves, Visual cells in the temporal cortex sensitive to face view and gaze direction, Proc. R. Soc. Lond. B: Biol. Sci. 223 (1985) 293–317. [46] A. Puce, T. Allison, M. Asgari, J.C. Gore, G. McCarthy, Differential sensitivity of human visual cortex to faces, letterstrings, and textures: a functional magnetic resonance imaging study, J. Neurosci. 16 (1996) 5205–5215. [47] A. Puce, T. Allison, S. Bentin, J.C. Gore, G. McCarthy, Temporal cortex activation in humans viewing eye and mouth movements, J. Neurosci. 18 (1998) 2188–2199. [48] G. Rizzolatti, L. Fadiga, M. Matelli, V. Bettinardi, E. Paulesu, D. Perani, F. Fazio, Localization of grasp representations in humans by PET: 1. Observation versus execution, Exp. Brain Res. 111 (1996) 246–252. [49] M. Siegal, R. Varley, Neural systems involved in ‘theory of mind’, Nat. Rev. Neurosci. 3 (2002) 463–471. [50] J.A. Stevens, P. Fonlupt, M. Shiffrar, J. Decety, New aspects of motion perception: selective neural encoding of apparent human movements, Neuroreport 11 (2000) 109–115. [51] M. Tomasello, Joint attention as social cognition, in: C. Moore, P.J. Dunham (Eds.), Joint Attention: Its Origins and Role in Development, Lawrence Erlbaum Associates, Hillsdale, NJ, 1995, pp. 103– 130. [52] M. Von Grunau, C. Anston, The detection of gaze direction: A stare in the crowd effect, Perception 24 (1995) 1297–1313. [53] P. Whalen, Fear, vigilance and ambiguity: Initial neuroimaging studies of the human amygdala, Curr. Direct. Psychol. Sci. 7 (1998) 177–188. [54] B. Wicker, F. Michel, M.A. Henaff, J. Decety, Brain regions involved in the perception of gaze: a PET study, Neuroimage 8 (1998) 221–227.

418

C.I. Hooker et al. / Cognitive Brain Research 17 (2003) 406–418 [57] A.W. Young, F. Newcombe, E.H. de Haan, M. Small, D.C. Hay, Face perception after brain injury. Selective impairments affecting identity and expression, Brain 116 (1993) 941–959.

[55] E. Wojciulik, N. Kanwisher, J. Driver, Covert visual attention modulates face-specific activity in the human fusiform gyrus: fMRI study, J. Neurophysiol. 79 (1998) 1574–1578. [56] A.W. Young, J.P. Aggleton, D.J. Hellawell, M. Johnson, P. Broks, R. Hanley, Face processing impairments after amygdalotomy, Brain 118 (1995) 15–24.