Skip to main content

Impact of mask use on face recognition: an eye-tracking study

Abstract

We examined how mask use affects performance and eye movements in face recognition and whether strategy change reflected in eye movements is associated with performance change. Eighty-eight participants performed face recognition with masked faces either during learning only, during recognition only, or during both learning and recognition. As compared with the baseline condition where faces were unmasked during both learning and recognition, participants had impaired performance in all three scenarios, with larger impairment when mask conditions during learning and recognition did not match. When recognizing unmasked faces, whether the faces were learned with or without a mask on did not change eye movement behavior. Nevertheless, when recognizing unmasked faces that were learned with a mask on, participants who adopted more eyes-focused patterns had less performance impairment as compared with the baseline condition. When recognizing masked faces, participants had more eyes-focused patterns and more consistent gaze transition behavior than recognizing unmasked faces regardless of whether the faces were learned with or without a mask on. Nevertheless, when recognizing masked faces that were learned without a mask, participants whose gaze transition behavior was more consistent had less performance impairment as compared with the baseline condition. Thus, although eye movements during recognition were mainly driven by the mask condition during recognition but not that during learning, those who adjusted their strategy according to the mask condition difference between learning and recognition had better performance. This finding has important implications for identifying populations vulnerable to the impact of mask use and potential remedial strategies.

Significance statement

The abrupt change in social life due to preventive mask use during the COVID-19 pandemic calls for urgent research on its impact on social cognition, in particular the recognition of faces, as well as identification of vulnerable populations and possible remedial strategies. While individual differences in face recognition have been much studied in recent years, there has not been any research on how mask use influences individuals’ face recognition memory, and whether some people are more vulnerable to the impact than others. Here, we show that the most challenging scenario of mask use on face recognition is when the mask conditions during face learning and recognition do not match, including recognizing unmasked faces that were masked during learning, or recognizing masked faces that were unmasked during learning. In these scenarios, individuals with better abilities to adjust their eye movement strategy according to the mask condition difference are affected less by mask use in recognition performance. These findings suggest that individuals with poor cognitive flexibility to switch strategies during face learning and recognition, or poor problem-solving skills to develop a new visual routine for recognizing masked faces, can become vulnerable to the impact of mask use on face recognition ability during daily life. Potential vulnerable populations include children, older adults, and individuals with autism spectrum disorders. Explicit instructions on better eye movement strategies to tackle these scenarios may be beneficial to them. After the world has recovered from the pandemic, these strategies can continue being used under face covering scenarios.

Introduction

The COVID-19 pandemic has completely changed our social life: Social distancing has become the new normal, and protective mask use has become prevalent. This situation is likely to last for a long time. Nevertheless, there is virtually no research on how widespread mask use in a society impacts how people remember and recognize faces, and whether a particular information processing strategy can facilitate face recognition when masks are used. Face recognition is an essential skill in social life. Misidentification of faces may interfere with normal social functioning and enhance social anxiety (e.g., Davis et al., 2011). Thus, it may affect mental health at the individual level. Also, face recognition ability is shown to be associated with social biases in the society (e.g., Morgan & Hills, 2019; Young et al., 2011). Thus, its impact on human life is far reaching and there is an urgency to fill this research gap.

Facial features such as the eyes, the nose, and the mouth convey important information for face recognition. When people wear masks, some facial features, i.e., the nose and the mouth, are covered and not available for memory encoding (face learning) or retrieval (face recognition). Mask wearing is shown to impair holistic face processing (Freud et al., 2020), an important perceptual skill supporting face perception and recognition (e.g., Tanaka & Simonyi, 2016; Wang et al., 2012). It is also shown to impair face matching performance (Carragher & Hancock, 2020; Dhamecha et al., 2014), even for super-recognizers (Noyes et al., 2021). In face recognition memory, mask wearing can happen during face learning, during face recognition, or during both face learning and recognition. When the information available for memory encoding during face learning and memory retrieval during face recognition does not match, such as recognizing an unmasked face that was learned with a mask on, or recognizing a masked face that was learned without a mask, recognition can become particularly challenging. Indeed, a mismatch between information used for memory encoding and retrieval has been consistently reported to impair memory recall performance (e.g., Uner & Roediger, 2018; Unsworth et al., 2020). Thus, we may need to change our information extraction strategy in response to the mask conditions during face learning and recognition in order to reduce the impact of mask use on our recognition performance. Nevertheless, it remains unclear how we change our information extraction strategy when we remember or recognize masked faces, and whether a particular strategy can facilitate face recognition when masks are used. Here, we aimed to fill this research gap through examining how mask use under different scenarios affects recognition performance and eye movement behavior during face recognition. We also aimed to examine whether information extraction strategy change in response to mask use, as reflected in eye movement behavior, is associated with recognition performance change.

Recent research has reported substantial individual differences in eye movement patterns during face recognition that can indicate differences in recognition performance and cognitive abilities (e.g., Chan et al., 2018; Chuk et al., 2017b; Hsiao et al., 2021a; Peterson & Eckstein, 2013; Peterson et al., 2016). To take individual differences in both temporal and spatial dimensions of eye movements into account in data analysis, Chuk et al. (2014) developed a machine learning-based approach, eye movement analysis with hidden Markov models (EMHMM; hidden Markov model, or HMM, is a type of time-series statistical model in machine learning), which provides quantitative measures of eye movement pattern and consistency (Chan et al., 2018; Hsiao et al., 2021b). In this approach, an individual’s eye movement pattern in a visual task is first summarized using an HMM, including person-specific regions of interest (ROIs) and transition probabilities among the ROIs. Individual HMMs can be clustered according to similarities (Coviello et al., 2014) to discover representative eye movement patterns in the population, and similarities among individual eye movement patterns can be quantified using their data likelihood given the models of the representative patterns. In addition, consistency of an individual’s eye movements across task trials can be directly measured as the entropy of the HMM (entropy is a measure of predictability; higher entropy indicates lower consistency; Cover & Thomas, 2006).

Using EMHMM on eye movement data during face recognition, two representative eye movement patterns, i.e., eyes-focused and nose-focused patterns, have been consistently reported in adult face recognition, and the eyes-focused pattern is shown to be associated with better recognition performance (An & Hsiao, 2021; Chuk et al., 2017a, 2017b; Hsiao et al., 2021a). These eye movement patterns for face recognition observed in adults have been shown to be consistent over time within an individual and impervious to the influence of transitory mood changes (An & Hsiao, 2021; Hsiao et al., 2021a; Peterson & Eckstein, 2013; Peterson et al., 2016). This phenomenon may be related to adults’ abundant experience in face recognition, which leads to a well-developed, consistent visual routine for recognizing faces that is inflexible to change. Indeed, using EMHMM, Hsiao et al. (2020) showed that adults had more consistent (lower entropy) eye movement patterns than children in face recognition. This finding is consistent with the observation in the literature that adult face recognition performance has limited plasticity for improvement through training, which may also be related to adults’ abundant face recognition experience that has made their recognition performance reach a capacity limit (Tree et al., 2017).

The findings summarized above suggest that when encountering faces with a mask, adults’ information extraction strategy for either face learning or recognition may be affected by their well-developed visual routines for recognizing unmasked faces, and individuals who have a better ability to develop a new routine according to the current mask condition of the faces without being affected by the old routine may perform better in face recognition when masks are used. For example, when recognizing an unmasked face that was learned with a mask on, only information not covered by the mask, including the eye and forehead region, was available for memory encoding. Thus, during recognition, adopting a more eyes- or forehead-focused eye movement pattern than one’s original visual routine may be beneficial. Similarly, when recognizing a masked face, only information in the eye and forehead region is available for retrieval. Hence, it may be beneficial to focus on developing new strategies based on the information available without being affected by one’s original routine for recognizing unmasked faces, especially when the face was learned without a mask on and thus not all information available during memory encoding is available for recognition. Note however that some previous studies have suggested a dissociation between eye movement patterns during face learning and face recognition (Chuk et al., 2017a; Henderson et al., 2005). These findings suggested that our original, well-developed visual routines during face recognition may still play an important role for successful recognition when encountering masked faces. Thus, it remains unclear what eye movement strategies are beneficial for learning and recognizing masked faces.

Accordingly, here we examined three scenarios where face recognition performance could be impaired by mask use: when masks were used during face learning only, during recognition only, or during both face learning and recognition. In the baseline condition, unmasked faces were used during both face learning and recognition. We hypothesized that as compared with the baseline condition, mask use impaired face recognition performance in all three scenarios, and larger impairment may be observed when there was a mismatch in mask condition between face learning and face recognition. We also hypothesized that when recognizing an unmasked face that was learned with a mask on, individuals may differ in how well they could adjust their eye movement strategy to focus on the available information during learning, i.e., the eye and forehead region. And those who had a larger change in eye movement pattern toward the eye and forehead region may have better performance. In the scenario of recognizing a masked face, those who were able to develop a new, consistent visual routine unaffected by the old routine for unmasked faces, especially when the face was learned without a mask on, may have better recognition performance. To test these hypotheses, we conducted an eye tracking study of face recognition under these scenarios and use the EMHMM method to quantify participants’ eye movement pattern and consistency in different scenarios. Recent research has suggested that general intelligence is a poor predictor for face recognition performance (e.g., Wilmer, 2017). However, eye movement behavior in face recognition has been reported to be associated with some cognitive abilities, particularly those related to executive function, visual attention, and working memory (e.g., Chan et al., 2018; Hsiao et al., 2020). To understand whether the hypothesized associations between eye movement behavior change and performance impairment due to mask use could still be observed after partialling out these general intelligence and cognitive ability factors, here we also measured participants’ general intelligence and cognitive abilities.

Method

Design

The face recognition task consisted of two phases: a learning phase and a recognition phase. For the learning phase data, we focused our analysis on difference in eye movement behavior between viewing unmasked and masked faces. The design consisted of a within-participant variable mask condition (unmasked vs. masked), and the dependent variables were eye movement measures. For the recognition phase data, the design consisted of two within-participant variables: mask condition during learning (unmasked vs. masked) and mask condition during recognition (unmasked vs. masked). The dependent variables were recognition performance and eye movement measures (Table 1). Recognition performance was measured in discrimination sensitivity d′ and RT. The eye movement measures of interest included eye movement pattern, marginal entropy of the first fixation, conditional entropy of the second fixation, and conditional entropy of the third fixation, as measured using eye movement analysis with hidden Markov models (EMHMM; see the “Eye movement data analysis” section). We then used paired t test to examine the impact of mask use in the following three scenarios using the scenario with unmasked faces at both phases as the baseline: (1) Effect of mask use during learning: masked faces at learning and unmasked faces at recognition versus baseline; (2) effect of mask use during recognition: unmasked faces at learning and masked faces at recognition versus baseline; and (3) effect of mask use in the face recognition task: masked faces at both phases versus baseline. Correlation analysis was used to examine whether participants’ recognition performance impairment due to mask use in the above three scenarios was associated with the corresponding change in eye movement behavior. In addition, partial correlation was used to examine whether these associations could be still observed after participants’ general intelligence and cognitive ability measures (including working memory, executive planning, and selective attention) were partialled out.

Table 1 Four mask conditions used in the design

Participant

Participants were 88 university students or graduates (51 females). Their age ranged from 17 to 30 (M = 21.1, SD = 2.36). All participants had normal vision or corrected to normal vision by contact lenses or glasses. According to a power analysis, a sample size of 85 was needed to acquire a medium effect size (\({f}^{2}\) = .15) in a linear multiple regression with 4 tested predictors (β = .2; α = .05).

Materials

Face stimuli

The face stimuli consisted of 256 colored frontal-view Asian face images from face databases developed in Professor William Hayward’s lab and the Attention Brain and Cognition Lab, respectively, at the University of Hong Kong (Chan et al., 2018; Chuk et al., 2014; Hsiao et al., 2021a). Half of them were young adult faces, one-fourth were old adult faces, and one-fourth were child faces. Within each face age group, half of the images were females. All faces had a neutral expression and were unfamiliar to the participants. The face images were cropped along the face shape so that only the inner features were available for recognition. Each face subtended a horizontal visual angle of 6° under the viewing distance of 55 cm, equivalent to the size of a real face under a functional distance for face identification (~ 2 m; McKone, 2009). All face images were scaled and aligned according to the distance between the centers of the two eyes.

The face images were randomly divided into two groups to be used as the target and foil images in the recognition task, with the number of faces in each gender by age combination matched between the two groups. For each face identity, images with different mask conditions (i.e., masked and unmasked), lighting conditions (i.e., yellow light and white light), and mask colors (blue mask and white mask) were created using Adobe Photoshop. During the face recognition task, images used during the learning and the recognition phases differed in both lighting condition and mask color for masked faces.

Raven’s standard progressive matrices (RSPM)

The RSPM was used to assess participants’ general intelligence (Raven, 2000). It is a multiple-choice test for assessing abstract thinking and reasoning. Participants were required to select a missing piece to complete a matrix-like pattern. The nine-item version, which is derived from the full set of RSPM and has obtained good accuracy in predicting the total score of the full set (Bilker et al., 2012), was used in this study.

Two-back task

Two-back tasks were used to assess participants’ working memory ability, i.e., to encode, maintain and update incoming information (Jaeggi et al., 2010). In the verbal two-back task, in each trial participants were presented with a single-digit number at the center of the screen for 1000 ms, followed by a blank screen for 2500 ms. They were asked to judge whether the presented number was the same as the one presented two trials back. In the spatial two-back task, in each trial participants were presented with a symbol appearing at one of 12 possible locations on the screen for 1000 ms, followed by a 2500 ms blank screen. They were asked to judge whether the presented symbol location was the same as the one presented two trials back (Lau et al., 2015). In both tasks, participants finished two blocks of 26 trials each. Accuracy and response times (RTs) of correct trials were measured.

Tower of London (TOL) task

The TOL task was used to assess participants’ executive planning ability (Berg & Byrd, 2002). Participants were presented with three color beads randomly placed on three pegs as a starting position together with a goal position. They were asked to move one bead at a time to reach the goal position with the least number of moves. In total, there were 12 trials. We measured number of correct trials, preplanning time (time before the first move) and execution times (time after the first move).

Flanker task

The flanker task was used to assess selective attention and response inhibition (Eriksen & Eriksen, 1974; Ridderinkhof et al., 1999). In each trial, participants judged the direction of an arrow flanked by four other arrows. In congruent trials, the flanking arrows pointed in the same direction as the target arrow, whereas in incongruent trials, they pointed in the opposite direction. The stimulus was presented for 500 ms, followed by a blank screen until response. Under a viewing distance of 50 cm, the flanking arrows subtended a horizontal and vertical visual angle of 1.03° × 1.03°, and the target arrow subtended a horizontal and vertical visual angle of .83° × .83°. In total there were 120 trials. We measured the flanker effect as \(\frac{\left(I - C\right)}{\left(I + C\right)}\), where I and C stand for the performance in the incongruent and congruent trials, respectively. Accuracy and correct RTs were measured.

Apparatus

Participants’ eye movements during the face recognition task were recorded using an EyeLink 1000 eye tracker. A chinrest was used to minimize head movements. Pupil and corneal reflection tracking mode was used with a sampling rate of 1000 Hz. EyeLink default settings for cognitive research were used for data acquisition, i.e., saccade motion threshold of .1 degree of visual angle, saccade acceleration threshold of 8000 degree/square second, and saccade velocity threshold of 30 degrees.

Procedure

In the learning phase of the face recognition task, participants were presented with 16 face images, one at a time, and asked to remember the faces. In the recognition phase, participants were presented with the 16 target (old) faces together with 16 foil (new) faces one at a time in a random order and asked to judge whether they had seen the face during the learning phase. The face image stayed on the screen until response. Participants performed eight blocks of the task. To examine the effect of mask use on recognition performance, in each block, among the 16 target faces, there were four stimuli in each of the following conditions: unmasked at both phases, masked at both phases, unmasked at learning and masked at recognition, and masked at learning and unmasked at recognition (Table 1). The 16 foil faces used in the recognition phase matched the conditions of the target faces. The faces used in the four mask conditions were counterbalanced across participants using a Latin square design. Among the eight blocks, participants performed four blocks with faces under white light with blue masks during learning and under yellow light with white masks during recognition; in the other four blocks, the light and mask colors in the two phases were swapped. The light and mask colors used for each face stimulus were counterbalanced across participants. For each participant, the block order was randomized.

Before each learning and recognition phase, a nine-point calibration procedure was performed. Each trial started with a solid circle at the screen center for drift correction. Re-calibration took place whenever drift correction error exceeded 1° of visual angle. During the learning phase, in each trial participants were presented with a face at one of the four quadrants of the screen for 5 s and were asked to remember the face. During the recognition phase, in each trial participants were presented with a face at one of the four quadrants of the screen and were asked to judge whether the face was an old face presented during the learning phase by pressing the “f” key for “yes” responses and “j” key for “no” responses using the left and right index fingers, respectively. The face remained on the screen until they responded. Participants took a short break between blocks. After they finished the face recognition task, they completed the verbal two-back task, the spatial two-back task, the Flanker task, and the TOL task.

Eye movement data analysis

The eye movement analysis with hidden Markov models (EMHMM) method was used to analyze eye movement data (Chuk et al., 2014). For eye movement data during the learning phase, for each participant, we used one hidden Markov model (HMM) to summarize the participant’s eye movement pattern when viewing unmasked faces, and another HMM to summarize the participant’s eye movement pattern when viewing masked faces. In other words, each participant had two HMMs, with each corresponding to eye movement pattern for unmasked and masked faces, respectively. The variational Bayesian expectation–maximization (VBEM) algorithm (Bishop, 2006) was used to train individual HMMs, and the optimal number of ROIs for each HMM was determined by VBEM from a preset range 1 to 10. (In previous EMHMM studies of face recognition, the median number of ROIs discovered in individual models was typically between 2 and 4 ROIs; e.g., An & Hsiao, 2021; Chan et al., 2018; Chuk et al., 2017a, 2017b; Hsiao et al., 2021a.) Following previous EMHMM studies on face processing, we clustered all individual HMMs to discover two representative eye movement patterns for face learning among the participants, Pattern A and Pattern B, using the variational hierarchical expectation–maximization (VHEM) algorithm (Coviello et al., 2014). The median of the number of ROIs among the individual HMMs was used to generate the HMMs of the two representative patterns. For each participant, we quantified the eye movement pattern during face learning, for viewing unmasked and masked faces separately, along the dimension contrasting Pattern A and Pattern B using AB scale, which is defined as:

$${(}A - B{)}/(\left| A \right| + |B|)$$

where A refers to the log-likelihood of the participant’s eye movement data being generated by the HMM of Pattern A, and B refers to the log-likelihood of the participant’s data being generated by the HMM of pattern B (see, e.g., An & Hsiao, 2021; Chan et al., 2018, 2020a, 2020b, 2020c; Chan et al., 2022a, b; Chuk et al., 2020; Hsiao et al., 2021a, 2021c; Lee et al., 2021; Zhang et al., 2019; Zheng et al., 2022). A more positive AB scale indicates greater similarity to Pattern A, whereas a more negative AB scale indicates greater similarity to Pattern B.

For eye movement data during the recognition phase, each participant had four HMMs, with each summarizing the eye movement pattern in one of the four mask conditions in Table 1. Similar to the analysis for face learning, we clustered all individual models to discover two representative patterns, Pattern A and Pattern B and then quantified each participant’s eye movement pattern in each of the four mask conditions using AB scale.

In addition to AB scale, we examined participants’ eye movement consistency across trials in different mask conditions using the entropy of the HMMs. Entropy is a measure of predictability; higher entropy indicates lower consistency (Cover & Thomas, 2006). In EMHMM, entropy of HMMs has been used to quantify participants’ eye movement consistency during visual tasks (Hsiao et al., 2021b). Previous studies on face recognition have suggested that the first 2–3 fixations in a trial play a more important role in accounting for recognition performance than later fixations (Chuk et al., 2017b; Hsiao & Cottrell, 2008). To better understand the temporal dynamics of eye movement consistency in different mask conditions, we measured marginal entropy of the first fixation, conditional entropy of the second fixation given the first fixation, and conditional entropy of the third fixation given the second fixation, to quantify consistency of the first fixation location, consistency of the transition from the first fixation to the second fixation, and consistency of the transition from the second fixation to the third fixation, respectively (Hsiao et al., 2021b). These three types of entropy were measured separately for each mask condition outlined in Table 1.

Results

Face recognition performance

Participants’ face recognition performance as measured in d′ and RT is shown in Fig. 1. The 2 × 2 ANOVA showed a main effect of mask condition during learning, F(1,87) = 40.110, p < .001, \({\eta }_{p}^{2}\) = .316, 90% CI = [.1848, .4268]Footnote 1: Participants had lower d′ in the recognition of faces learned with than without a mask on; a main effect of mask condition during recognition, F(1,87) = 22.340, p < .001, \({\eta }_{p}^{2}\) = .204, 90% CI = [.0900, .3193]: Participants had lower d′ when recognizing masked faces than unmasked faces. There was also an interaction effect between mask condition during learning and mask condition during recognition, F(1,87) = 63.266, p < .001, \({\eta }_{p}^{2}\) = .421, 90% CI = [.2890, .5218]: After learning unmasked faces, participants had lower d′ when recognizing them as masked faces than as unmasked faces, t(171) =  − 9.103, p < .001, d =  − .686, 95% CI [− .8504, − .5220]; in contrast, after learning masked faces, participants had lower d′ when recognizing them as unmasked faces than as masked, t(171) = 2.895, p = .022, d = .218, 95% CI [.0687, .3677]. These results showed that participants had lower d′ when mask conditions during the learning and recognition phases did not match.

Fig. 1
figure 1

a Participants’ d′ in different mask conditions (unmasked–unmasked: Faces were unmasked during both learning and recognition. Unmasked–masked: Faces were unmasked during learning and masked during recognition. Masked–unmasked: Faces were masked during learning and unmasked during recognition. Masked–masked: Faces were masked during both learning and recognition). b Participants’ RT in ms in different mask conditions (*p < .05, ** p < .01, ***p < .001, paired t test)

In RT, the 2 × 2 ANOVA showed a main effect of mask condition during recognition, F(1,87) = 5.343, p = .023, \({\eta }_{p}^{2}\) = .058, 90% CI = [.0042, .1508]: Participants had longer RT when recognizing masked faces than unmasked faces; it interacted with mask condition during learning, F(1,87) = 26.534, p < .001, \({\eta }_{p}^{2}\) = .234, 90% CI = [.1133, .3487]: After learning unmasked faces, participants had longer RT when recognizing them as masked faces than as unmasked faces, t(172) = 5.39, p < .001, d = .406, 95% CI [.2526, .5600]; in contrast, after learning masked faces, participants did not have significantly different RT when recognizing them as unmasked faces or as masked faces, t(172) =  − 2.32, p = .098.

We then examined the changes in performance due to mask in the three planned comparisons separately. On the effect of mask use during learning (masked–unmasked vs. unmasked–unmasked), participants had lower d′, t(87) =  − 8.880, p < .001, d =  − .947, 95% CI [− 1.1980, − .6952], and longer RT, t(87) = 4.347, p < .001, d = .463, 95% CI [.2435, .6833], when recognizing an unmasked face learned with than without a mask on.

On the effect of mask use during recognition (unmasked–masked vs. unmasked–unmasked), participants had lower d′, t(87) =  − 8.370, p < .001, d =  − .892, 95% CI [− 1.1393, − .6452], and longer RT, t(87) = 5.091, p < .001, d = .543, 95% CI [.3189, .7665], when recognizing masked faces than unmasked faces that were learned without a mask on.

On the effect of mask use in the face recognition task (masked–masked vs. unmasked–unmasked), participants had lower d′, t(87) =  − 7.539, p < .001, d =  − .804, 95% CI [− 1.0440, − .5633], when performing the task with masked faces than with unmasked faces. However, their RT did not differ significantly between the two conditions, t(87) = 1.897, p = .061.

Eye movement behavior during face learning

The two representative eye movement patterns during face learning discovered through clustering using EMHMM are shown in Fig. 2. In Pattern A, a scan path always started with a fixation at a broad region centered at the mid-point between the two eyes, covering both the eye region and the nose region (red, 100%). Afterward, it either stayed exploring in the broad region (Red to Magenta, 36%), switched to the eye region (red to green, 38%), or the forehead region (Red to Blue, 20%), and then most likely remained in the same region. In contrast, in Pattern B, a scan path typically started with a fixation at a broad region centered at the mid-point between the two eyes (red, 95%), with a small probability to start with a fixation at the forehead region (magenta, 5%). After a fixation at the broad region (red), it most likely switched to the eye region (red to green, 55%), or the nose region (red to blue, 33%), and occasionally to the forehead region (red to magenta, 11%). As compared with Pattern A, Pattern B also had more transitions between the eyes (green) and the nose (blue) regions, and a larger and higher forehead ROI (magenta). The two patterns significantly differed according to KL divergence estimates (Chuk et al., 2014): Data from participants adopting Pattern A were more likely to be generated by Pattern A HMM than Pattern B HMM, and vice versa for data from participants adopting Pattern B, F(1, 174) = 316.99, p < .001, \({\eta }_{p}^{2}\) = .646, 90% CI [.5776, .6957].

Fig. 2
figure 2

Eye movement data during face learning. a The two representative eye movement patterns discovered using EMHMM during face learning: In each pattern, ellipses show ROIs as 2-D Gaussian emissions. Priors in the table show the probabilities that a fixation sequence starts from the ellipse. The table also shows transition probabilities among the ROIs. The smaller image on the top-right shows the assignment of actual fixations to different ROIs. The assignment of fixations to the ROIs was based on the ROI sequence with the largest posterior probability given the fixation sequence. The smaller image on the bottom-right shows the corresponding heatmap. b Eye movement pattern measured in AB scale in the unmasked and masked conditions during face learning

We then examined whether participants’ eye movement behavior, including eye movement pattern and consistency, differed when viewing masked versus unmasked face during face learning. Participants’ eye movement pattern was quantified using AB scale according to the two representative patterns, and eye movement consistency was assessed using entropy of the HMMs. The results showed that there was no significant difference in eye movement behavior between viewing masked and unmasked faces during face learning as measured in AB scale, t(87) =  − .4245, p = .672, marginal entropy of the first fixation, t(87) = .1793, p = .858, conditional entropy of the second fixation given the first fixation, t(87) = .7774, p = .439, or conditional entropy of the third fixation given the second fixation, t(87) = .0997, p = .921. In other words, participants’ eye movement behavior did not differ significantly when viewing masked versus unmasked faces during face learning.

Eye movement patterns during face recognition

The two representative eye movement patterns during face recognition discovered through clustering using EMHMM are shown in Fig. 3a. In Pattern A, participants fixated across a broad region centered at the bridge of the nose between the two eyes, covering both the eye region and the nose region. In contrast, in Pattern B, a scan path typically started with a fixation at a broad region centered at the mid-point between the two eyes covering both the eye and the nose regions (red, 96%). Then, it most likely switched to the eye region (red to green, 97%), and then remained in the same region (green to green, 96%). Occasionally, it started (magenta, 3%) and stayed at the forehead region (blue and magenta). The two patterns significantly differed, as data from participants adopting Pattern A were more likely to be generated by Pattern A HMM than Pattern B HMM, and vice versa for data from participants adopting Pattern B (following Chuk et al., 2014), F(1, 350) = 25.8, p < .001, \({\eta }_{p}^{2}\) = .069, 90% CI [.0318, .1146].

Fig. 3
figure 3

Eye movement data during the recognition phase. a The two representative eye movement patterns discovered using EMHMM during face recognition: In each pattern, the ellipses show ROIs as 2-D Gaussian emissions. Priors in the table show the probabilities that a fixation sequence starts from the ellipse. The table also shows transition probabilities among the ROIs. The smaller image on the top-right shows the assignment of actual fixations to different ROIs. The assignment of fixations to the ROIs was based on the ROI sequence with the largest posterior probability given the fixation sequence. The smaller image on the bottom-right shows the corresponding heatmap. Note that ROI 5 (Cyan) in both patterns captures outlier fixations that do not belong to other ROIs. b Eye movement pattern measured in AB scale in different mask conditions. c Eye gaze transition consistency from the first fixation to the second fixation as measured in conditional entropy. d Eye gaze transition consistency from the second fixation to the third fixation as measured in conditional entropy (*p < .05, ** p < .01, ***p < .001, paired t test)

We then examined whether participants’ eye movement behavior, including eye movement pattern and consistency, differed among different mask conditions during face learning and recognition (Table 1). The 2 (mask condition during learning) × 2 (mask condition during recognition) ANOVA on eye movement pattern as measured in AB scale showed a main effect of mask condition during recognition (Fig. 3b), F(1, 87) = 208.24, p < .001, \({\eta }_{p}^{2}\) = .705, 90% CI [.6167, .7611]: Participants showed an eye movement pattern more similar to Pattern A when recognizing unmasked faces than masked faces. A main effect of mask condition during recognition was also observed in conditional entropy of the second fixation given the first fixation (Fig. 3c), F(1, 87) = 68.714, p < .001, \({\eta }_{p}^{2}\) = .441, 90% CI [.3103, .5395]: Participants showed more consistent second fixation given the first fixation when recognizing masked faces than unmasked faces. Similarly, a main effect of mask condition during recognition was observed in conditional entropy of the third fixation given the second fixation (Fig. 3d), F(1, 87) = 122.552, p < .001, \({\eta }_{p}^{2}\) = .585, 90% CI [.4709, .6616]: Participants showed more consistent third fixation given the second fixation when recognizing masked faces than unmasked faces. No main effect or interaction was observed in marginal entropy of the first fixation. Together these results showed that participants eye movements were more eyes-focused (Pattern B) with more consistent gaze transition patterns when recognizing masked faces than unmasked faces, regardless of whether the faces were learned with or without a mask on during learning.

We then examined eye movement behavior change due to mask use in the three planned comparisons separately. On the effect of mask use during learning (masked–unmasked vs. unmasked–unmasked), no significant effect was observed in A–B scale, t(87) =  − 1.815, p = .073, marginal entropy of the first fixation, t(87) =  − 1.420, p = .159, conditional entropy of the second fixation given the first fixation, t(87) =  − 1.108, p = .271, or conditional entropy of the third fixation given the second fixation between the two conditions, t(87) =  − .715, p = .476. These results suggested that eye movement behavior during recognition was mainly driven by information available at the recognition phase but not that presented at the learning phase.

On the effect of mask use during recognition (unmasked–masked vs. unmasked–unmasked), participants had lower AB scale, t(87) =  − 14.203, p < .001, d =  − 1.514, 95% CI [− 1.8201, − 1.2080], lower conditional entropy of the second fixation given the first fixation, t(87) =  − 6.587, p < .001, d =  − .702, 95% CI [− .9355, − .4689], and lower conditional entropy of the third fixation given the second fixation, t(87) =  − 7.984, p < .001, d =  − .851, 95% CI [− 1.0950, − .6073], when recognizing masked faces than unmasked faces. No significant difference was observed in marginal entropy of the first fixation, t(87) =  − 1.338, p = .185. Thus, they had more eye-focused eye movement pattern and more consistent gaze transition behavior when recognizing masked faces than unmasked faces.

On the effect of mask use in the face recognition task (masked–masked vs. unmasked–unmasked), participants had lower AB scale, t(87) =  − 13.405, p < .001, d =  − 1.429, 95% CI [− 1.7260, − 1.1320], lower conditional entropy of the second fixation given the third fixation, t(87) =  − 6.733, p < .001, d =  − .718, 95% CI [− .9520, − .4834], and lower conditional entropy of the third fixation given the second fixation, t(87) =  − 10.075, p < .001, d =  − 1.074, 95% CI [− 1.3364, − .8117], when performed the face recognition task with masked faces than with unmasked faces. No significant different was found in marginal entropy of the first fixation, t(87) =  − .752, p = .454. Thus, they had more eye-focused eye movement pattern and more consistent gaze transition behavior in the recognition task with masked faces than unmasked faces.

Relationship between eye movement behavior change and performance change due to mask use during face recognition

We then examined whether eye movement behavior change was associated with participants’ recognition performance change due to mask use during face recognition. On the change of recognition performance due to mask use during learning (unmasked–unmasked condition minus masked–unmasked condition), larger performance impairment in d′ was correlated with the smaller change toward Pattern B with mask use, r(86) =  − .240, p = .024 (Fig. 4). This correlation was still significant when we partialled out general intelligence as measured in RSPM, r(85) =  − .249, p = .020, or when we partialled out both general intelligence and cognitive ability measures, r(75) =  − .269, p = .018, using partial correlation analysis. These results suggested that individuals who adjusted their eye movement patterns to be more eyes-focused (Pattern B) when recognizing an unmasked face that was learned with a mask on during learning had less recognition performance impairment.

Fig. 4
figure 4

Correlation between the change in AB scale due to mask use during learning (unmasked–unmasked condition minus masked–unmasked condition; more positive change indicates larger change toward the more eyes-focused Pattern B with mask use) and the corresponding performance impairment in d′: The more change toward the more eyes-focused Pattern B, the less the performance impairment

On the change of recognition performance due to mask use during recognition (unmasked–unmasked condition minus unmasked–masked condition), smaller performance impairment in d′ was correlated with larger change toward low conditional entropy of the third fixation given the second fixation with mask use, r(86) =  − .217, p = .043 (Fig. 5). This correlation was still significant when we partialled out general intelligence as measured in RSPM, r(85) =  − .235, p = .028, and it became marginal when we partialled out both general intelligence and cognitive ability measures, r(75) =  − .217, p = .058. In an explorative analysis examining which cognitive abilities were correlated with change in conditional entropy of the third fixation given the second fixation, we found that it was correlated with general intelligence (RSPM), r(86) =  − .276, p = .009, and TOL execution time, r(85) = .214, p = .045, suggesting that higher general intelligence and shorter execution time in TOL were associated with larger change toward low conditional entropy of the third fixation given the second fixation. These results suggested that individuals who had increased eye gaze transition consistency when recognizing a masked face that was learned without a mask on had less recognition performance impairment due to the mask use.

Fig. 5
figure 5

Correlation between the change in conditional entropy of the third fixation given the second fixation due to mask use during recognition (unmasked–unmasked condition minus unmasked–masked condition; more positive change indicates larger change toward lower entropy and thus more consistent transition with mask use) and the corresponding performance impairment in d′: The more change toward low-entropy/more consistent gaze transition behavior, the less the performance impairment

On the change in performance due to mask use in the face recognition task (unmasked–unmasked condition minus masked–masked condition), the change in face recognition performance was not associated with the change in any of the eye movement behavior measures.

Discussion

Here, we examined how mask use affects performance and eye movement behavior in face recognition under three scenarios: seeing masked faces during face learning only, during face recognition only, and during both face learning and recognition. We tested the hypothesis that mask use impaired face recognition performance in all three scenarios, and the largest impairment may be found when the mask conditions between face learning and face recognition did not match. In these scenarios, individuals who could adjust their eye movement behavior according to the mask condition difference between face learning and recognition may have better performance. Consistent with our hypothesis, we found that as compared with the baseline condition where regular, unmasked faces were presented during both face learning and recognition, participants’ recognition performance was impaired in all three scenarios, with the largest impairment observed when there was a mismatch in mask condition between face learning and recognition. This finding is consistent with the encoding specificity effect reported in the literature (Uner & Roediger, 2018; Unsworth et al., 2020): A mismatch between information used for memory encoding and retrieval impairs memory recall performance. This is also consistent with the part-whole effect reported in the face perception literature, which is argued to be an indication of holistic face processing: People have better identification performance when a facial part (e.g., the eyes) is presented in the context of the whole face than when it is presented in isolation (after face learning with the whole face; Seitz, 2001; Tanaka & Farah, 1993).

In addition, we found that when recognizing a regular, unmasked face, although participants performed worse when the face was learned with a mask on than without, whether the face was learned with or without a mask on did not significantly change participants’ eye movement pattern or consistency. This result suggested that eye movements during recognition are mainly driven by the information available during recognition but not that during learning (Chuk et al., 2017a). While this result also suggested that recognition performance impairment due to mask use during face learning could not be accounted for by change in eye movement pattern or consistency during recognition at the group level, an interesting individual difference effect was found: As compared with the baseline scenario where unmasked faces were used during both face learning and recognition, the recognition performance impairment due to mask use during face learning could be predicted by the amount of eye movement change toward a more eyes-focused pattern during recognition (Patten B in Fig. 3a): The larger the change in eye movement pattern toward a more eyes-focused pattern, the less the recognition performance impairment due to mask use during face learning (Fig. 4). This association remained significant after general intelligence and cognitive ability factors were accounted for, suggesting that the contribution from eye movement pattern change was unlikely to be confounded with general intelligence or eye movement pattern.

In the literature on visual cognition, it has been proposed that the mental representation of a visual object or pattern includes the perceptuomotor cycle involved during memory encoding (i.e., the scan path theory; Noton & Stark, 1971a, 1971b). Thus, the eye movement pattern elicited during memory encoding may become part of the mental representation, which may be reactivated and affect the viewer’s eye movement pattern and performance during memory retrieval (e.g., Blais et al., 2008; Laeng & Teodorescu, 2002). For highly learned skills such as face recognition, this perceptuomotor memory during face learning may interact with the visual routine memory developed through years of experience to affect eye movement planning behavior during recognition. Chuk et al. (2017a) showed that in face recognition, the match between eye movement pattern adopted during face learning and recognition did not predict recognition performance. In contrast, a more eyes-focused eye movement pattern during recognition predicted better recognition performance. This result suggests that retrieving diagnostic features plays a more important role in successful recognition than reactivating perceptuomotor memory during the recognition of unmasked faces that were also learned without a mask on. In contrast to this finding, here we showed that when there is a mismatch between the information available for memory encoding and retrieval due to mask use, as in the case of recognizing an unmasked face that was learned with a mask on, the reactivation of perceptuomotor memory may become beneficial, as it helps the viewer to attend to the features that were available during memory encoding when these features are presented in a different context during retrieval.

We also found that when recognizing masked faces, participants had more eyes-focused eye movement pattern (Pattern B in Fig. 3a) and more consistent gaze transition behavior (as reflected in the conditional entropy of the early fixations in a trial) than recognizing unmasked faces, regardless of whether the faces were learned with or without a mask on. The task demands of recognizing a masked face differ from our usual experience of recognizing unmasked faces. Thus, new visual routines have to be developed to solve the new recognition problem. Our results suggested that adopting a more eyes-focused eye movement pattern and engaging in more consistent gaze transition behavior when recognizing masked faces as compared with recognizing unmasked faces (regardless of whether faces were learned with or without a mask on) were a general strategy adopted by the participants for solving this new recognition problem. A more eyes-focused pattern facilitates information extraction from the face region not covered by the mask, whereas more consistent gaze transition behavior may be related to a more restricted region (i.e., only the forehead and eye regions are available for recognition) available for developing a visual routine, leading to more predictable gaze transition behavior.

In addition, we observed an interesting individual difference effect: As compared with the baseline condition where there was no mask during both face learning and recognition, the recognition performance impairment due to mask use during recognition could be predicted by the amount of change toward more consistent gaze transition behavior (conditional entropy of the third fixation given the second fixation; Fig. 5): The larger the change in eye movement consistency toward more consistent transition behavior, the less the recognition performance impairment due to mask use during face recognition. This correlation remained significant after the contribution from general intelligence was partialled out and became marginal when both general intelligence and cognitive ability factors were partialled out. This result suggested that this association was unlikely to be confounded with general intelligence, although it may be related to some cognitive abilities. Indeed, in an explorative analysis, we found that this change in consistency of gaze transition behavior was significantly correlated with participants’ general intelligence and TOL performance. Individual differences in gaze transition consistency when solving a recognition problem typically reflect differences in the ability to achieve efficient and successful task processing: Those who are able to discover diagnostic information and learn to identify it efficiently and effectively would develop a more consistent visual routine that leads to better performance. For example, young children who have more consistent eye movement patterns across trials when learning to recognize faces are shown to have better recognition performance than those with less consistent patterns (Hsiao et al., 2020). Thus, change in gaze transition consistency may reflect the ability to discover and identify diagnostic features for recognizing masked faces. Note, however, that this relationship between change in gaze transition consistency and change in recognition performance was only observed when the mask conditions during face learning and recognition did not match, i.e., recognizing masked faces that were learned without a mask on, but not when recognizing masked faces that were also learned with a mask on. Thus, this gaze transition consistency effect may be related to the ability to develop a new visual routine when only part of the information learned during memory encoding (face learning) is available for retrieval during recognition.

Together the findings in the current study suggest that when the mask condition differs between face learning and recognition, it presents a particularly challenging scenario where face recognition performance can be significantly affected. In this scenario, individuals who have better ability to adjust their eye movement strategy according to the mask condition difference are affected less by mask use and thus may have a better ability to adapt to life in a society with widespread mask use brought about by the COVID-19 pandemic. This finding has important implications for identifying individuals vulnerable to the impact of mask use and potential remedial strategies. More specifically, the ability to adjust eye movement strategies according to mask conditions during face recognition may require good cognitive flexibility to switch between perceptuomotor and visual routine memories, and good problem-solving skills to develop new visual routines when there is a mismatch between information available for memory encoding and retrieval. Thus, it may be particularly challenging to young children, whose cognitive flexibility and problem-solving skills are still developing (e.g., Peng et al., 2018); older adults, whose cognitive abilities may have declined (e.g., Giller & Beste, 2019); and individuals with autism spectrum disorders (ASD), who are typically characterized by low cognitive flexibility (e.g., English et al., 2017; Fujino et al., 2019; Soriano et al., 2018). Explicit instructions on strategies that may benefit recognition performance, such as adjusting to a more eyes-focused eye movement pattern when recognizing an unmasked face that was learned with a mask on, may help these vulnerable individuals adapt better. Future work will examine these possibilities.

Conclusions

In conclusion, here we showed that while mask use impairs face recognition performance regardless of whether masks are used during face learning only, during face recognition only, or during both face learning and recognition, the largest impairment happens when there is a mismatch between the mask conditions during face learning and recognition. When recognizing regular, unmasked faces that are learned with a mask on, those who adjust their eye movement strategy to be more eyes-focused such that it is more consistent with their perceptuomotor memory during face learning are less affected by mask use in recognition performance. When recognizing masked faces that are learned without a mask on, those who demonstrated more consistent gaze transition behavior, an indication of better visual routine development ability, are less affected by mask use in recognition performance. These results suggest that when there is a mismatch between the mask conditions during face learning and recognition, those who have a better ability to adjust their information extraction strategy according to the mask condition difference are affected less by mask use. This finding has important implications for identifying individuals who may have poor cognitive flexibility or problem-solving skills to adjust eye movement strategy and thus may be vulnerable to the impact of mask use on face recognition performance, including children, older adults, and individuals with ASD, and the possibility to provide in-time intervention for them.

Availability of data and materials

The datasets generated and/or analyzed during the current study are available in the OSF data repository, https://osf.io/g6es5/?view_only=5904749c5e124e0b81cde7feb071bd00.

Notes

  1. Note that in reporting confidence interval information, as suggested by Lakens (2013), a 90% CI was used for ANOVA results and a 95% CI was used for t test results due to the fact that the F test in ANOVA is usually a one-sided test, and adopting a 95% CI may include 0 in the interval when the test is statistically significant.

Abbreviations

ASD:

Autism spectrum disorder

EMHMM:

Eye movement analysis with hidden Markov models

RT:

Reaction time

RSPM:

Raven’s Standard Progressive Matrices

TOL:

Tower of London

WM:

Working memory

ROIs:

Regions of interest

VBEM:

Variational Bayesian expectation–maximization

VHEM:

Variational hierarchical expectation–maximization

References

  • An, J., & Hsiao, J. H. (2021). Modulation of mood on eye movement and face recognition performance. Emotion, 21(3), 617–630.

    PubMed  Article  PubMed Central  Google Scholar 

  • Berg, W. K., & Byrd, D. L. (2002). The Tower of London spatial problem-solving task: Enhancing clinical and research implementation. Journal of Clinical and Experimental Neuropsychology, 24(5), 586–604.

    Article  Google Scholar 

  • Bilker, W. B., Hansen, J. A., Brensinger, C. M., Richard, J., Gur, R. E., & Gur, R. C. (2012). Development of abbreviated nine-item forms of the Raven’s Standard Progressive Matrices Test. Assessment, 19(3), 354–369.

    PubMed  PubMed Central  Article  Google Scholar 

  • Bishop, C. M. (2006). Pattern recognition and machine learning. Springer.

    Google Scholar 

  • Blais, C., Jack, R. E., Scheepers, C., Fiset, D., & Caldara, R. (2008). Culture shapes how we look at faces. PLoS ONE, 3(8), e3022.

    PubMed  PubMed Central  Article  Google Scholar 

  • Carragher, D. J., & Hancock, P. J. B. (2020). Surgical face masks impair human face matching performance for familiar and unfamiliar faces. CRPI, 5(1), 1–15.

    Google Scholar 

  • Chan, C. Y. H., Chan, A. B., Lee, T. M. C., & Hsiao, J. H. (2018). Eye-movement patterns in face recognition are associated with cognitive decline in older adults. Psychonomic Bulletin & Review, 25(6), 2200–2207.

    Article  Google Scholar 

  • Chan, F. H. F., Barry, T. J., Chan, A. B., & Hsiao, J. H. (2020a). Understanding visual attention to face emotions in social anxiety using hidden Markov models. Cognition and Emotion, 34(8), 1704–1710.

    PubMed  Article  PubMed Central  Google Scholar 

  • Chan, F. H. F., Jackson, T., Hsiao, J. H., Chan, A. B., & Barry, T. J. (2020b). The interrelation between interpretation biases, threat expectancies and pain-related attentional processing. European Journal of Pain, 24(10), 1956–1967.

  • Chan, F. H. F., Suen, H., Hsiao, J. H., Chan, A. B., & Barry, T. J. (2020c). Interpretation biases and visual attention in the processing of ambiguous information in chronic pain. European Journal of Pain, 24(7), 1242–1256.

    PubMed  Article  PubMed Central  Google Scholar 

  • Chan, S. K. W., Hsiao, J. H., Wong, A. O. Y., Liao, Y., Suen, Y., Yan, E. W. C., Poon, L.-T., Siu, M. W., Hui, C. L. M., Chang, W. C., Lee, E. H. M., & Chen, E. Y. H. (2022a). Explicit and implicit mentalization of patients with first-episode schizophrenia: A study of self-referential gaze perception and eye movement analysis using hidden Markov models. European Archives of Psychiatry and Clinical Neuroscience, 1-11.

  • Chan, F. H. F., Suen, H., Chan, A. B., Hsiao, J. H., & Barry, T. J. (2022b). The effects of attentional and interpretation biases on later pain outcomes among younger and older adults: a prospective study. European Journal of Pain, 26(1), 181–196.

  • Chuk, T., Chan, A. B., & Hsiao, J. H. (2014). Understanding eye movements in face recognition using hidden Markov models. Journal of Vision, 14(11), 8.

    PubMed  Article  Google Scholar 

  • Chuk, T., Chan, A. B., & Hsiao, J. H. (2017a). Is having similar eye movement patterns during face learning and recognition beneficial for recognition performance? Evidence from hidden Markov modeling. Vision Research, 141, 204–216.

    PubMed  Article  Google Scholar 

  • Chuk, T., Chan, A. B., Shimojo, S., & Hsiao, J. H. (2020). Eye movement analysis with switching hidden Markov models. Behavior Research Methods, 52(3), 1026–1043.

  • Chuk, T., Crookes, K., Hayward, W. G., Chan, A. B., & Hsiao, J. H. (2017b). Hidden Markov model analysis reveals the advantage of analytic eye movement patterns in face recognition across cultures. Cognition, 169, 102–117.

    PubMed  Article  Google Scholar 

  • Cover, T. M., & Thomas, J. A. (2006). Elements of information theory. Wiley.

    Google Scholar 

  • Coviello, E., Chan, A. B., & Lanckriet, G. R. G. (2014). Clustering hidden Markov models with variational HEM. Journal of Machine Learning Research, 15(1), 697–747.

    Google Scholar 

  • Davis, J. M., McKone, E., Dennett, H., O’Connor, K. B., O’Kearney, R., & Palermo, R. (2011). Individual differences in the ability to recognise facial identity are associated with social anxiety. PLoS ONE, 6(12), e28800.

    PubMed  PubMed Central  Article  Google Scholar 

  • Dhamecha, T. I., Singh, R., Vatas, M., & Kumar, A. (2014). Recognizing disguised faces: Human and machine evaluation. PLoS ONE, 9(7), e99212.

    PubMed  PubMed Central  Article  Google Scholar 

  • English, M. C. W., Maybery, M. T., & Visser, T. A. W. (2017). Modulation of global and local processing biases in adults with autistic-like traits. Journal of Autism and Developmental Disorders, 47(9), 2757–2769.

    PubMed  Article  Google Scholar 

  • Eriksen, B. A., & Eriksen, C. W. (1974). Effects of noise letters upon the identification of a target letter in a nonsearch task. Perception & Psychophysics, 16(1), 143–149.

    Article  Google Scholar 

  • Freud, E., Stajduhar, A., Rosenbaum, R. S., Avidan, G., & Ganel, T. (2020). The COVID-19 pandemic masks the way people perceive faces. Scientific Reports-UK, 10(1), 1–8.

    Article  Google Scholar 

  • Fujino, J., Tei, S., Itahashi, T., Aoki, Y., Ohta, H., Kubota, M., Isobe, M., Hashimoto, R. I., Nakamura, M., Kato, N., & Takahashi, H. (2019). Need for closure and cognitive flexibility in individuals with autism spectrum disorder: A preliminary study. Psychiatry Research, 271, 247–252. https://doi.org/10.1016/j.psychres.2018.11.057

    Article  PubMed  Google Scholar 

  • Giller, F., & Beste, C. (2019). Effects of aging on sequential cognitive flexibility are associated with fronto-parietal processing deficits. Brain Structure & Function, 224(7), 2343–2355.

    Article  Google Scholar 

  • Henderson, J. M., Williams, C. C., & Falk, R. J. (2005). Eye movements are functional during face learning. Memory & Cognition, 33(1), 98–106.

    Article  Google Scholar 

  • Hsiao, J. H., An, J., Zheng, Y., & Chan, A. B. (2021a). Do portrait artists have enhanced face processing abilities? Evidence from hidden Markov modeling of eye movements. Cognition, 211, 104616.

    PubMed  Article  Google Scholar 

  • Hsiao, J. H., An, J. H., & Chan, A. B. (2020). The role of eye movement consistency in learning to recognise faces: Computational and experimental examinations. In Proceedings of the 42nd annual conference of the cognitive science society (pp. 1072–1078). Cognitive Science Society.

  • Hsiao, J. H, Lan, H., Zheng, Y., & Chan, A. B. (2021c). Eye Movement analysis with Hidden Markov Models (EMHMM) with co-clustering. Behavior Research Methods, 53, 2473–2486.

  • Hsiao, J. H., Chan, A. B., An, J., Yeh, S.-L., & Jingling, L. (2021b). Understanding the collinear masking effect in visual search through eye tracking. Psychonomic Bulletin & Review, 28(6), 1933–1943.

    Article  Google Scholar 

  • Hsiao, J. H., & Cottrell, G. W. (2008). Two fixations suffice in face recognition. Psychological Science, 9(10), 998–1006.

    Article  Google Scholar 

  • Jaeggi, S. M., Buschkuehl, M., Perrig, W. J., & Meier, B. (2010). The concurrent validity of the N-back task as a working memory measure. Memory, 18(4), 394–412.

    PubMed  Article  Google Scholar 

  • Laeng, B., & Teodorescu, D. S. (2002). Eye scanpaths during visual imagery reenact those of perception of the same visual scene. Cognitive Science, 26(2), 207–231. https://doi.org/10.1016/S0364-0213(01)00065-9

    Article  Google Scholar 

  • Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front. Psychol., 4, 1–12.

  • Lau, E. Y. Y., Choi, E. W., Lai, E. S., Lau, K. N., Au, C., Yung, W., & Li, A. M. (2015). Working memory impairment and its associated sleep-related respiratory parameters in children with obstructive sleep apnea. Sleep Medicine, 16(9), 1109–1115.

    PubMed  Article  PubMed Central  Google Scholar 

  • Lee, H. H., Chen, Z. L., Yeh, S. L., Hsiao, J. H., & Wu, A. Y. (2021). Mind-wandering as revealed by eye movement analysis with hidden Markov models. Sensors, 21(22), 7569.

    PubMed  PubMed Central  Article  Google Scholar 

  • McKone, E. (2009). Holistic processing for faces operates over a wide range of sizes but is strongest at identification rather than conversational distances. Vision Research, 49(2), 268–283.

    PubMed  Article  PubMed Central  Google Scholar 

  • McPartland, J., Dawson, G., Webb, S. J., Panagiotides, H., & Carver, L. J. (2004). Event-related brain potentials reveal anomalies in temporal processing of faces in autism spectrum disorder. Journal of Child Psychology and Psychiatry, 45(7), 1235–1245.

    PubMed  Article  PubMed Central  Google Scholar 

  • Miellet, S., Caldara, R., & Schyns, P. G. (2011). Local Jekyll and global Hyde: The dual identity of face identification. Psychological Science, 22(12), 1518–1526.

    PubMed  Article  PubMed Central  Google Scholar 

  • Morgan, M., & Hills, P. J. (2019). Correlations between holistic processing, autism quotient, extraversion, and experience and the own-gender bias in face recognition. PLoS ONE, 14(7), e0209530.

    PubMed  PubMed Central  Article  Google Scholar 

  • Noton, D., & Stark, L. (1971a). Scan paths in eye movements during pattern perception. Science, 171, 308–311.

    PubMed  Article  PubMed Central  Google Scholar 

  • Noton, D., & Stark, L. (1971b). Scan paths in saccadic eye movements while viewing and recognizing patterns. Vision Research, 11, 929–942.

    PubMed  Article  PubMed Central  Google Scholar 

  • Noyes, E., Davis, J. P., Petrov, N., Gray, K. L. H., & Ritchie, K. L. (2021). The effect of face masks and sunglasses on identity and expression recognition with super-recognizers and typical observers. Royal Society Open Science, 8(3), 201169.

    PubMed  PubMed Central  Article  Google Scholar 

  • Peng, A., Kirkham, N. Z., & Mareschal, D. (2018). Task switching costs in preschool children and adults. Journal of Experimental Child Psychology, 172, 59–72. https://doi.org/10.1016/j.jecp.2018.01.019

    Article  PubMed  PubMed Central  Google Scholar 

  • Peterson, M. F., & Eckstein, M. P. (2013). Individual differences in eye movements during face identification reflect observer specific optimal points of fixation. Psychological Science, 24(7), 1216–1225.

    PubMed  Article  Google Scholar 

  • Peterson, M. F., Lin, J., Zaun, I., & Kanwisher, N. (2016). Individual differences in face-looking behavior generalize from the lab to the world. Journal of Vision, 16, 12.

    PubMed  Article  Google Scholar 

  • Raven, J. (2000). The Raven’s progressive matrices: Change and stability over culture and time. Cognitive Psychology, 41, 1–48.

    PubMed  Article  Google Scholar 

  • Ridderinkhof, K. R., Band, G., & Logan, G. (1999). A study of adaptive behavior: Effects of age and irrelevant information on the ability to inhibit one’s actions. Acta Psychologica, 101(1999), 315–337.

    Article  Google Scholar 

  • Seitz, K. (2001). Parts and wholes in person recognition: Developmental trends. Journal of Experimental Child Psychology, 82(4), 367–381.

    Article  Google Scholar 

  • Soriano, M. F., Ibáñez-Molina, A. J., Paredes, N., & Macizo, P. (2018). Autism: Hard to switch from details to the whole. Journal of Abnormal Child Psychology, 46(6), 1359–1371. https://doi.org/10.1007/s10802-017-0384-4

    Article  PubMed  Google Scholar 

  • Tanaka, J. W., & Farah, M. J. (1993). Parts and wholes in face recognition. The Quarterly Journal of Experimental Psychology Section A, 46(2), 225–245.

    Article  Google Scholar 

  • Tanaka, J. W., & Simonyi, D. (2016). The “parts and wholes” of face recognition: A review of the literature. The Quarterly Journal of Experimental Psychology, 69(10), 1876–1889.

    PubMed  Article  Google Scholar 

  • Tree, J. J., Horry, R., Riley, H., & Wilmer, J. (2017). Are portrait artists superior face recognizers? Limited impact of adult experience on face recognition ability. Journal of Experimental Psychology—Human Perception and Performance, 43(4), 667–676.

    PubMed  Article  PubMed Central  Google Scholar 

  • Uner, O., & Roediger, H. L. (2018). Are encoding/retrieval interactions in recall driven by remembering, knowing, or both? Journal of Memory and Language, 103, 44–57.

    Article  Google Scholar 

  • Unsworth, N., Brewer, G. A., & Spillers, G. J. (2020). Variation in working memory capacity and episodic memory: Examining the importance of encoding specificity. Psychonomic Bulletin & Review, 18(6), 1113–1118.

    Article  Google Scholar 

  • Wang, R. S., Li, J. G., Fang, H. Z., Tian, M. Q., & Liu, J. (2012). Individual differences in holistic processing predict face recognition ability. Psychological Science, 23(2), 169–177.

    PubMed  Article  Google Scholar 

  • Wilmer, J. B. (2017). Individual differences in face recognition: A decade of discovery. Current Directions in Psychological Science, 26(3), 225–230.

    Article  Google Scholar 

  • Young, S. G., Hugenberg, K., Bernstein, M. J., & Sacco, D. F. (2011). Perception and motivation in face recognition: A critical review of theories of the cross-race effect. Personality & Social Psychology Review, 16(2), 116–142.

    Article  Google Scholar 

  • Zhang, J., Chan, A. B., Lau, E. Y. Y., & Hsiao, J. H. (2019). Individuals with insomnia misrecognize angry faces as fearful faces while missing the eyes: An eye-tracking study. Sleep, 42(2), zsy220.

    Article  Google Scholar 

  • Zheng, Y., Ye, X., & Hsiao, J. H. (2022). Does adding video and subtitles to an audio lesson facilitate its comprehension? Learning & Instruction, 77, 101542.

    Article  Google Scholar 

Download references

Acknowledgements

We are grateful to Jacky Wong, who helped build the program used for data collection and collected part of the data. We are also grateful to Karen Ng, who helped collect part of the data.

Funding

This study is funded by the Research Grant Council of Hong Kong (Collaborative Research Fund #C7129-20G).

Author information

Affiliations

Authors

Contributions

JH conceptualized the study, designed the experiment, interpreted the data and wrote the manuscript. WL collected and analyzed the data, drafted the “Method” and the “Results” sections, and contributed to the writing of the manuscript. RT designed the experiment and contributed to the data interpretation and writing of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Janet Hui-wen Hsiao.

Ethics declarations

Ethics approval and consent to participate

The experiment was approved by the Human Research Ethics Committee of the University of Hong Kong (HREC reference number: EA200090).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hsiao, J.Hw., Liao, W. & Tso, R.V.Y. Impact of mask use on face recognition: an eye-tracking study. Cogn. Research 7, 32 (2022). https://doi.org/10.1186/s41235-022-00382-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s41235-022-00382-w

Keywords

  • Face recognition
  • Face mask use
  • Eye movement
  • EMHMM