Humans but Not Chimpanzees Vary Face-Scanning Patterns Depending on Contexts during Action Observation

Masako Myowa-Yamakoshi; Chisato Yoshida; Satoshi Hirata

doi:10.1371/journal.pone.0139989

Abstract

Human and nonhuman primates comprehend the actions of other individuals by detecting social cues, including others’ goal-directed motor actions and faces. However, little is known about how this information is integrated with action understanding. Here, we present the ontogenetic and evolutionary foundations of this capacity by comparing face-scanning patterns of chimpanzees and humans as they viewed goal-directed human actions within contexts that differ in whether or not the predicted goal is achieved. Human adults and children attend to the actor’s face during action sequences, and this tendency is particularly pronounced in adults when observing that the predicted goal is not achieved. Chimpanzees rarely attend to the actor’s face during the goal-directed action, regardless of whether the predicted action goal is achieved or not. These results suggest that in humans, but not chimpanzees, attention to actor’s faces conveying referential information toward the target object indicates the process of observers making inferences about the intentionality of an action. Furthermore, this remarkable predisposition to observe others’ actions by integrating the prediction of action goals and the actor’s intention is developmentally acquired.

Citation: Myowa-Yamakoshi M, Yoshida C, Hirata S (2015) Humans but Not Chimpanzees Vary Face-Scanning Patterns Depending on Contexts during Action Observation. PLoS ONE 10(11): e0139989. https://doi.org/10.1371/journal.pone.0139989

Editor: Rachel L. Kendal, Centre for Coevolution of Biology & Culture, University of Durham, UNITED KINGDOM

Received: May 15, 2015; Accepted: September 21, 2015; Published: November 4, 2015

Copyright: © 2015 Myowa-Yamakoshi et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited

Data Availability: All relevant data are within the paper and its Supporting Information files.

Funding: This research was supported by Grants-in-Aid for Scientific Research from the Japan Society for the Promotion of Science; the Ministry of Education, Culture, Sports, Science and Technology (24000001, 24300103, 24119005, and 26245069), https://www.jsps.go.jp/english/e-grants/index.html; ERATO, the Okanoya Emotional Information Project, Japan Science and Technology Agency, http://www.jst.go.jp/erato/project/ojj_P/ojj_P.html.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Human and nonhuman primates are social beings who live in highly cohesive groups. This primate group living may shape sensitivity to subtle social cues. For example, understanding other individuals’ actions is crucial for anticipating their future actions and appropriately adjusting one’s own behaviour. Both monkeys and humans have common neural mechanisms underlying the capacity to predict and understand the motor goals of others’ actions, which are called mirror neurons (e.g., [1–7]). It has been argued that the mirror neuron system automatically and pre-reflexively matches action perception and action execution [3,4] (e.g., the neurons that fire when I grasp a cup of water are similar to those that fire when I watch another person grasp a cup). However, several researchers have emphasized the differences in mirror system function between monkeys and humans. For example, the vast majority of monkey mirror neurons are responsive to actions-on-objects, whereas human mirror activation occurs in response to both intransitive meaningless gestures and actions-on-objects [8–10] (regarding a cross-species comparison of mirror system circuity in monkeys, chimpanzees, and humans, see [11]).

Faces also provide crucial information about individuals’ social lives. By detecting information expressed in the faces of other individuals, such as gaze direction and emotional facial expressions, individuals are able to rapidly identify and respond to dynamic changes in their environment [12]. Whilst research investigating the roles of motor actions and facial information have proved valuable for increasing our knowledge about action understanding, to date the majority of these studies have investigated characteristics of motor actions and facial information independently. For example, the ability to understand others’ actions has been primarily studied by only focusing on transitive or goal-related motor acts, such as reaching, grasping, and manipulating an object (e.g., [13–16]). It is therefore currently unclear how the information regarding both goal-directed motor acts and actors' faces is integrated when understanding actions, despite the fact that both motor acts and others’ faces are observed as daily events unfold [17]. Here, using eye-tracking technology, we investigated time-series changes in scanning patterns while observing object-related actions, with a particular focus on the actor’s face.

Mirror neurons, which are responsive to actions-on-objects, may underlie action recognition in both monkeys and humans. However, humans also appear to have a predisposition to observe goal-directed actions by integrating information from the actor’s face. Previous studies have demonstrated that both humans and nonhuman primates rationally predict and understand the object-related actions of other individuals [18–20]. However, our own previous study demonstrated distinct differences between the scanning patterns of humans and chimpanzees when viewing goal-directed actions ²⁰. More specifically, captive chimpanzees who were familiar with humans primarily attended to information regarding motor acts, including hand movements and manipulated objects; however, they rarely looked at the actor’s face, regardless of whether the actor was a chimpanzee or a human. By contrast, humans (both adults and infants) attended to others’ faces to a greater extent than chimpanzees.

It is likely that these two species differ with respect to when and why they refer to faces when coding an actor’s goal-directed actions. We assume that, by scanning the actor’s face, humans make ‘explicit’ or active inferences concerning whether an action is likely to have been performed intentionally. For example, facial expressions of frustration or disappointment might convey that an agent's goal had been unsuccessful or that the outcome was accidental. Therefore, when predicted motor goals are not completed, humans in particular devote substantially greater attention to the actor’s face to seek additional referential information.

Our first aim was to investigate the timing of facial references along with the process of encoding the goal-directedness of an action in humans and chimpanzees. We compared the two species’ time-series scanning patterns of actions in the following two contexts: when a possible goal was achieved and when it was not (Experiments 1 and 2). We predicted that when the possible goal was not achieved, humans would search for additional referential information from the actor. By contrast, assuming that chimpanzees understand goal-directed actions primarily based on the physical characteristics of the environment (i.e., manipulated objects), no clear differences may be evident in chimpanzees’ facial scanning patterns between the two action contexts. We show that humans do increase their attention to the actor’s face when they observe that the predicted goal is not completed whereas chimpanzees do not attend to faces even when the actor’s predicted action goal is not completed.

The second aim of the current study was to investigate the human ontogeny of the ability to use referential information for understanding others’ actions. When do humans first present evidence of adult-like social inferences? How does this ability develop? We examined the developmental changes in referential face-scanning patterns for faces when coding an actor’s goal-directed actions by comparing 12-month-old infants who do not yet use explicit language and 3.5-year-old children who use explicit language. (Experiment 3).

Experiment 1: Facial Scanning Patterns for Object-Related Actions in Humans and Chimpanzees

Materials and Methods

Participants.

A total of 14 adults (seven males, mean age = 23.8 years, s. d. = 2.9 years) and six captive chimpanzees (Pan troglodytes: two males, 6–16 years) participated in Experiment 1. The two males (both 16 years old) and four females (15, 15, 12 and 6 years old) lived as a group. The chimpanzees were cared for at the Great Ape Research Institute, Hayashibara Biomedical Laboratories, Inc. All the chimpanzees were familiar with humans and had previously participated in several behavioural cognitive tasks, including tool use procedures, sequential learning paradigms using touch screens, and eye-tracking measures [20, 21]. The chimpanzees spent several hours each day indoors interacting with humans for study or husbandry purposes, which included play activities using materials available in human environments such as hats, glasses, gloves, socks, clothes, watches, pens, books, papers, various types of containers (cups, bottles, buckets, etc.) and so forth.

Ethics statement.

All the experiments in this study were approved by the Ethical Committee of Kyoto University, Kokoro-Unit. This study was conducted in accordance with the standards specified in the 1964 Declaration of Helsinki. The care and use of chimpanzees adhered to the guidelines established by the Primate Society of Japan. This study was approved by the Animal Welfare and Animal Care Committee of the Hayashibara Biochemical Laboratories. We have obtained consent from the model actor to her image being used in the figures and video files (in Supporting Information) of the present paper.

Apparatus and stimuli.

A Tobii (Stockholm, Sweden) T60 Eye Tracker integrated with a 17-inch TFT monitor was used to present the stimuli and record eye movements using image-processing algorithms (60 Hz; Tobii Studio 2.1.12, Tobii Technology). This eye tracker could record eye gazes from both humans and chimpanzees without using body mounting devices such as glasses. The participants were seated approximately 60-cm from the monitor and no barrier was involved. The stimulus presentation and recording were controlled via a computer (Dell T7500 for human adults, Dell M4400 for chimpanzees) with Tobii Studio software. The entire video subtended 21.6°×16.2° of the visual angle. Prior to the video presentation, small animation videos were shown to the participants to direct their attention to the monitor.

Test stimuli were videos depicting an unfamiliar female human adult actor seated in front of a table in a well-lit room, who either poured some juice from a bottle into a clear glass cup (Congruent action) or poured some juice onto the table top (Incongruent action). Both videos lasted 14.0 s, and were approximately matched in relation to temporal features (i.e., the velocity of the actor’s hand movements and the timing when the actor made the pouring motion; specific details of timing can be found in the Methods and the velocity profiles for the actor’s right hand during these videos are shown in S2 Fig). Both the human and chimpanzee participants were familiar with this type of goal-directed action and were capable of performing the action by themselves. For example, the chimpanzees received juice as a part of their meal in the morning (juice was on their breakfast menu on average 3–4 times per week) and occasionally in the daytime. The chimpanzees watched familiar human staff pour the juice in front of them from a bottle into a cup, and they drank from the cup. In addition, various types of containers were provided in their enclosures for enrichment purposes, and all the chimpanzees in the present study had been observed, multiple times, engaging in pouring liquid materials from one container to another (e.g., pouring pure water from a bottle to a cup) during their free activities by using these enrichment materials.

For chimpanzees, the calibration errors were estimated prior to testing, and the average rate of error across the participants was 0.40° (s.d. = 0.38°) of the chimpanzees’ visual angle [21]. We did not precisely measure the calibration errors for the human groups given the accumulated knowledge regarding the validity of data collection using the device we employed [13, 14, 20, 22]. However, the errors can be estimated to be within the range of 1° of the visual angle of most of our participants, based on their fixation data from the stimulus used for attention-getting. One degree of visual angle is larger than the difference between the outline of each feature (i.e., face, cup or container, and trajectory of the manipulated object) and that of the respective area of interest AOI (see Data Analysis). Therefore, it is unlikely that calibration errors affected our analyses of gaze behaviour.

Procedure.

Upon arrival at the laboratory, human participants were brought into the study room which was softly illuminated to make the monitor screen the most salient feature of the room. An initial calibration procedure was conducted which was completed when measures from five calibration points were obtained. This procedure was repeated until the calibration criterion was met. The participants were instructed to simply watch the video until it ended. Human participants did not receive an external reward (food or social) for their participation. Chimpanzees were brought from their enclosures into a study room which was connected to their enclosures via a passageway by calling their names. They could receive a part of their daily meals in this room, which was a daily routine procedure for study and husbandry purposes. Familiar human experimenters remained in the study room during testing. One experimenter sat beside the chimpanzee and lightly held their chins or heads during the recording. This was to ensure that the distance between their eyes and the monitor remained appropriate for testing. The calibration for each chimpanzee was achieved at the beginning of the session by presenting a brief video clip at two calibration points. This relatively small number of reference points (compared to humans) was adopted for chimpanzees because they tend to view these reference points only briefly; if a large number of calibration points are presented they tend not to look at all of them, which would result in calibration failure [15, 21]. Despite this methodological difference between humans and chimpanzees our validation session with respect to the latter’s calibration data confirmed a comparable accuracy between species. All participants were then shown the videos involving the actor. The experimenter gave food to the chimpanzees before and after the completion of the video but not while they were viewing the video. Before the video presentation, food was given if the chimpanzees sat in front of the monitor. After the video presentation, they could receive a food reward regardless of their behaviour while watching the video. The experiment relied on voluntary participation by the chimpanzees, and they were not deprived of food or water for testing. That is, they could receive their daily meals regardless of whether they participated in the study. During testing they presented no negative emotional expressions such as screaming or grimacing.

The human and chimpanzee participants were shown two repetitions of the congruent action video followed by two repetitions of the incongruent version (a session), which were separated by an interval of approximately 5 to 7 s (Congruent―interval―Congruent―interval―Incongruent―interval―Incongruent). The order of the C-C-I-I session was fixed to minimize the expectation violation effect. During the interval, an animation or video clip was presented.

Data analysis.

Fixations were scored using a Tobii fixation filter with a threshold radius of 35 pixels. The statistical tests were calculated using SPSS 22.0 (IBM Corp.). We applied parametric tests after examining the normality of our data sample using graphical inspection of a Q–Q plot for normality and conducting a Shapiro–Wilk test. The looking time data for the first trials of the congruent and incongruent actions were used for subsequent analysis. If the total looking time on the first trial in each action condition was less than 25% of a video’s duration, the data from the second trial was used. These data were analysed using angular transformations.

For the analysis, we defined the three AOIs for the two actions as follows: the majority of the trajectory of the moving object (Trajectory AOI), the target object (Goal AOI), and the actor’s face during manipulation (Face AOI). The initiation of goal achievement was defined as the onset of pouring juice into a cup (Congruent action) or onto the table top (Incongruent action; Fig 1A). The video stimuli were further divided into three phases as follows: before-, during-, and after-goal achievement phases (Fig 2A). The before-goal phase began with the frame at which the manipulation of the object began and concluded with the frame depicting the onset of pouring juice (3.6 s for the congruent action and 3.4 s for the incongruent action). The during-goal phase began with the frame depicting the onset of pouring and concluded with the frame depicting the end of the pouring action (7.7 s for the congruent action and 7.9 s for the incongruent action). The after-goal phase began with the frame depicting the end of pouring and concluded with the frame depicting that the actor’s hand had returned to the location at which the manipulation of the object began (4.8 s for the congruent action and 4.6 s for the incongruent action). Two-tailed paired-sample t-tests using the Bonferroni correction were used for the pairwise comparisons.

Download:

Fig 1. A selected scene from the video stimulus used in Experiments 1, 2, and 3 and areas of interest (AOIs) for the analyses.

(a) Experiments 1 and 3: A female human adult actor sits in front of a table and pours some juice from a bottle into a clear glass cup (congruent action, upper) or pours some juice onto the floor (incongruent action, below). The videos of both the congruent and incongruent actions lasted 14.0 s. (b) Experiment 2: A female human adult actor sits at a table and places balls into a container (congruent action, upper) or places balls on the floor (incongruent action, below). The video of the congruent action lasted 13.5 s, and the video of the incongruent action lasted 13.9 s.

https://doi.org/10.1371/journal.pone.0139989.g001

Download:

Fig 2. (a) Experiments 1 and 3: A female human adult actor sits in front of a table and pours juice from a bottle into a clear glass cup (congruent action, upper) or pours juice onto the table top (incongruent action, below). (b) Experiment 2: A female human adult actor sits at a table and places balls into a container (congruent action, upper) or places balls on the table top (incongruent action, below).

These video stimuli were divided into the three phases of before, during, and after goal achievement. The before-goal phase began with the frame in which the manipulation of the object began and concluded with the frame depicting the onset of pouring juice or transporting the first grasped ball. The during-goal phase began with the frame depicting the onset of pouring or transporting and concluded with the frame depicting the end of the pouring or transporting action. The after-goal phase began with the frame depicting the end of the pouring or transporting action and concluded with the frame showing that the actor’s hand had returned to the location where the manipulation of the object began.

https://doi.org/10.1371/journal.pone.0139989.g002

Results

Preliminary analysis.

Both humans and chimpanzees anticipated the goal of the action prior to the onset of pouring the juice into the cup (see S1 Text and S1A Fig). The mean durations of time human adults spent looking toward the stimuli were 8.1 s (s.d. = 0.9) for the congruent action and 8.3 s (s.d. = 1.3) for the incongruent action, whilst chimpanzee spent 5.9 s (s.d. = 2.4) for the congruent action and 4.6 s (s.d. = 2.1) for the incongruent action. The total fixation durations revealed a significant effect of group (F_1,18 = 25.58, P < 0.001, η² = 0.35). Next, we compared the fixation durations for the face and object areas (i.e., Goal plus Trajectory AOIs) between humans and chimpanzees. Previous studies that presented still photographs as the test stimuli demonstrated that chimpanzees generally move their eyes more rapidly than humans [23, 24]. However, the average fixation durations did not differ between the two species (285 ms for chimpanzees, 344 ms for humans; F_1,18 = 1.02, P = 0.33).

Attentional allocation to an actor’s face.

We assessed when and how chimpanzees and humans referred to the actor’s face (ratio of looking time across the three AOIs combined) while viewing the goal-directed actions which consisted of the following three phases: before, during, and after goal achievement (S2 Fig).

We analysed the ratio of the fixations on the actor’s face area to the total time looking toward the three areas combined. A 3 (phase: before-, during-, after-goal) × 2 (action type: congruent, incongruent) × 2 (group: humans, chimpanzees) mixed ANOVA revealed significant main effects for group (F_1,18 = 15.29, P < 0.01, partial η² = 0.46) and phase (F_2,36 = 7.43, P < 0.01, partial η² = 0.29). A 3 (phase) × 2 (action type) repeated-measures ANOVA for each group revealed a significant main effect of phase in humans (F_2,26 = 11.24, P < 0.01, partial η² = 0.46) but not in chimpanzees (F_2,10 = 1.30, P = 0.32, partial η² = 0.21). Post hoc testing (Bonferroni adjusted) for the human group revealed that the ratio of looking toward the face area in the before-goal phase was significantly higher than in the during-goal phase in both the congruent and incongruent conditions (Ps < 0.01). Importantly, only in the incongruent condition was the ratio of looking time toward the face area higher in the after-goal phase than in the during-goal phase (P < 0.01, Fig 3A; also see S1 and S2 Movies). The present findings suggest that in implausible contexts, humans recruit active social inferences.

Download:

Fig 3. (a) Ratios of looking times towards the face area to total looking times towards the three areas combined across the three phases in Experiment 1; human adults (left) and chimpanzees (right). (b) Ratios of looking times towards the face area to total looking times towards the three areas combined across the three phases in Experiment 2; human adults (left) and chimpanzees (right). (c) Ratios of looking times towards the face area to total looking times towards the three areas combined across the three phases in Experiment 3; 12-month-old human infants (left) and 3.5-year-old human children (right).

The error bars represent the standard error of the mean (SEM).

https://doi.org/10.1371/journal.pone.0139989.g003

In Experiment 1, two factors influenced the design of the test stimuli. First, the actions that were depicted are present in the daily experiences of both the humans and the chimpanzees participating in this study. Second, most object-related actions that have been observed in wild chimpanzees (e.g., tool-using behaviours) are aimed at obtaining food [25]. However, the possibility remains that the species differences that we observed here may be a result of the chimpanzees simply attending to the food present in the videos. To eliminate this possibility, we conducted another experiment (Experiment 2).