Face recognition performance
Participants’ face recognition performance as measured in d′ and RT is shown in Fig. 1. The 2 × 2 ANOVA showed a main effect of mask condition during learning, F(1,87) = 40.110, p < .001, \({\eta }_{p}^{2}\) = .316, 90% CI = [.1848, .4268]Footnote 1: Participants had lower d′ in the recognition of faces learned with than without a mask on; a main effect of mask condition during recognition, F(1,87) = 22.340, p < .001, \({\eta }_{p}^{2}\) = .204, 90% CI = [.0900, .3193]: Participants had lower d′ when recognizing masked faces than unmasked faces. There was also an interaction effect between mask condition during learning and mask condition during recognition, F(1,87) = 63.266, p < .001, \({\eta }_{p}^{2}\) = .421, 90% CI = [.2890, .5218]: After learning unmasked faces, participants had lower d′ when recognizing them as masked faces than as unmasked faces, t(171) = − 9.103, p < .001, d = − .686, 95% CI [− .8504, − .5220]; in contrast, after learning masked faces, participants had lower d′ when recognizing them as unmasked faces than as masked, t(171) = 2.895, p = .022, d = .218, 95% CI [.0687, .3677]. These results showed that participants had lower d′ when mask conditions during the learning and recognition phases did not match.
In RT, the 2 × 2 ANOVA showed a main effect of mask condition during recognition, F(1,87) = 5.343, p = .023, \({\eta }_{p}^{2}\) = .058, 90% CI = [.0042, .1508]: Participants had longer RT when recognizing masked faces than unmasked faces; it interacted with mask condition during learning, F(1,87) = 26.534, p < .001, \({\eta }_{p}^{2}\) = .234, 90% CI = [.1133, .3487]: After learning unmasked faces, participants had longer RT when recognizing them as masked faces than as unmasked faces, t(172) = 5.39, p < .001, d = .406, 95% CI [.2526, .5600]; in contrast, after learning masked faces, participants did not have significantly different RT when recognizing them as unmasked faces or as masked faces, t(172) = − 2.32, p = .098.
We then examined the changes in performance due to mask in the three planned comparisons separately. On the effect of mask use during learning (masked–unmasked vs. unmasked–unmasked), participants had lower d′, t(87) = − 8.880, p < .001, d = − .947, 95% CI [− 1.1980, − .6952], and longer RT, t(87) = 4.347, p < .001, d = .463, 95% CI [.2435, .6833], when recognizing an unmasked face learned with than without a mask on.
On the effect of mask use during recognition (unmasked–masked vs. unmasked–unmasked), participants had lower d′, t(87) = − 8.370, p < .001, d = − .892, 95% CI [− 1.1393, − .6452], and longer RT, t(87) = 5.091, p < .001, d = .543, 95% CI [.3189, .7665], when recognizing masked faces than unmasked faces that were learned without a mask on.
On the effect of mask use in the face recognition task (masked–masked vs. unmasked–unmasked), participants had lower d′, t(87) = − 7.539, p < .001, d = − .804, 95% CI [− 1.0440, − .5633], when performing the task with masked faces than with unmasked faces. However, their RT did not differ significantly between the two conditions, t(87) = 1.897, p = .061.
Eye movement behavior during face learning
The two representative eye movement patterns during face learning discovered through clustering using EMHMM are shown in Fig. 2. In Pattern A, a scan path always started with a fixation at a broad region centered at the mid-point between the two eyes, covering both the eye region and the nose region (red, 100%). Afterward, it either stayed exploring in the broad region (Red to Magenta, 36%), switched to the eye region (red to green, 38%), or the forehead region (Red to Blue, 20%), and then most likely remained in the same region. In contrast, in Pattern B, a scan path typically started with a fixation at a broad region centered at the mid-point between the two eyes (red, 95%), with a small probability to start with a fixation at the forehead region (magenta, 5%). After a fixation at the broad region (red), it most likely switched to the eye region (red to green, 55%), or the nose region (red to blue, 33%), and occasionally to the forehead region (red to magenta, 11%). As compared with Pattern A, Pattern B also had more transitions between the eyes (green) and the nose (blue) regions, and a larger and higher forehead ROI (magenta). The two patterns significantly differed according to KL divergence estimates (Chuk et al., 2014): Data from participants adopting Pattern A were more likely to be generated by Pattern A HMM than Pattern B HMM, and vice versa for data from participants adopting Pattern B, F(1, 174) = 316.99, p < .001, \({\eta }_{p}^{2}\) = .646, 90% CI [.5776, .6957].
We then examined whether participants’ eye movement behavior, including eye movement pattern and consistency, differed when viewing masked versus unmasked face during face learning. Participants’ eye movement pattern was quantified using A–B scale according to the two representative patterns, and eye movement consistency was assessed using entropy of the HMMs. The results showed that there was no significant difference in eye movement behavior between viewing masked and unmasked faces during face learning as measured in A–B scale, t(87) = − .4245, p = .672, marginal entropy of the first fixation, t(87) = .1793, p = .858, conditional entropy of the second fixation given the first fixation, t(87) = .7774, p = .439, or conditional entropy of the third fixation given the second fixation, t(87) = .0997, p = .921. In other words, participants’ eye movement behavior did not differ significantly when viewing masked versus unmasked faces during face learning.
Eye movement patterns during face recognition
The two representative eye movement patterns during face recognition discovered through clustering using EMHMM are shown in Fig. 3a. In Pattern A, participants fixated across a broad region centered at the bridge of the nose between the two eyes, covering both the eye region and the nose region. In contrast, in Pattern B, a scan path typically started with a fixation at a broad region centered at the mid-point between the two eyes covering both the eye and the nose regions (red, 96%). Then, it most likely switched to the eye region (red to green, 97%), and then remained in the same region (green to green, 96%). Occasionally, it started (magenta, 3%) and stayed at the forehead region (blue and magenta). The two patterns significantly differed, as data from participants adopting Pattern A were more likely to be generated by Pattern A HMM than Pattern B HMM, and vice versa for data from participants adopting Pattern B (following Chuk et al., 2014), F(1, 350) = 25.8, p < .001, \({\eta }_{p}^{2}\) = .069, 90% CI [.0318, .1146].
We then examined whether participants’ eye movement behavior, including eye movement pattern and consistency, differed among different mask conditions during face learning and recognition (Table 1). The 2 (mask condition during learning) × 2 (mask condition during recognition) ANOVA on eye movement pattern as measured in A–B scale showed a main effect of mask condition during recognition (Fig. 3b), F(1, 87) = 208.24, p < .001, \({\eta }_{p}^{2}\) = .705, 90% CI [.6167, .7611]: Participants showed an eye movement pattern more similar to Pattern A when recognizing unmasked faces than masked faces. A main effect of mask condition during recognition was also observed in conditional entropy of the second fixation given the first fixation (Fig. 3c), F(1, 87) = 68.714, p < .001, \({\eta }_{p}^{2}\) = .441, 90% CI [.3103, .5395]: Participants showed more consistent second fixation given the first fixation when recognizing masked faces than unmasked faces. Similarly, a main effect of mask condition during recognition was observed in conditional entropy of the third fixation given the second fixation (Fig. 3d), F(1, 87) = 122.552, p < .001, \({\eta }_{p}^{2}\) = .585, 90% CI [.4709, .6616]: Participants showed more consistent third fixation given the second fixation when recognizing masked faces than unmasked faces. No main effect or interaction was observed in marginal entropy of the first fixation. Together these results showed that participants eye movements were more eyes-focused (Pattern B) with more consistent gaze transition patterns when recognizing masked faces than unmasked faces, regardless of whether the faces were learned with or without a mask on during learning.
We then examined eye movement behavior change due to mask use in the three planned comparisons separately. On the effect of mask use during learning (masked–unmasked vs. unmasked–unmasked), no significant effect was observed in A–B scale, t(87) = − 1.815, p = .073, marginal entropy of the first fixation, t(87) = − 1.420, p = .159, conditional entropy of the second fixation given the first fixation, t(87) = − 1.108, p = .271, or conditional entropy of the third fixation given the second fixation between the two conditions, t(87) = − .715, p = .476. These results suggested that eye movement behavior during recognition was mainly driven by information available at the recognition phase but not that presented at the learning phase.
On the effect of mask use during recognition (unmasked–masked vs. unmasked–unmasked), participants had lower A–B scale, t(87) = − 14.203, p < .001, d = − 1.514, 95% CI [− 1.8201, − 1.2080], lower conditional entropy of the second fixation given the first fixation, t(87) = − 6.587, p < .001, d = − .702, 95% CI [− .9355, − .4689], and lower conditional entropy of the third fixation given the second fixation, t(87) = − 7.984, p < .001, d = − .851, 95% CI [− 1.0950, − .6073], when recognizing masked faces than unmasked faces. No significant difference was observed in marginal entropy of the first fixation, t(87) = − 1.338, p = .185. Thus, they had more eye-focused eye movement pattern and more consistent gaze transition behavior when recognizing masked faces than unmasked faces.
On the effect of mask use in the face recognition task (masked–masked vs. unmasked–unmasked), participants had lower A–B scale, t(87) = − 13.405, p < .001, d = − 1.429, 95% CI [− 1.7260, − 1.1320], lower conditional entropy of the second fixation given the third fixation, t(87) = − 6.733, p < .001, d = − .718, 95% CI [− .9520, − .4834], and lower conditional entropy of the third fixation given the second fixation, t(87) = − 10.075, p < .001, d = − 1.074, 95% CI [− 1.3364, − .8117], when performed the face recognition task with masked faces than with unmasked faces. No significant different was found in marginal entropy of the first fixation, t(87) = − .752, p = .454. Thus, they had more eye-focused eye movement pattern and more consistent gaze transition behavior in the recognition task with masked faces than unmasked faces.
Relationship between eye movement behavior change and performance change due to mask use during face recognition
We then examined whether eye movement behavior change was associated with participants’ recognition performance change due to mask use during face recognition. On the change of recognition performance due to mask use during learning (unmasked–unmasked condition minus masked–unmasked condition), larger performance impairment in d′ was correlated with the smaller change toward Pattern B with mask use, r(86) = − .240, p = .024 (Fig. 4). This correlation was still significant when we partialled out general intelligence as measured in RSPM, r(85) = − .249, p = .020, or when we partialled out both general intelligence and cognitive ability measures, r(75) = − .269, p = .018, using partial correlation analysis. These results suggested that individuals who adjusted their eye movement patterns to be more eyes-focused (Pattern B) when recognizing an unmasked face that was learned with a mask on during learning had less recognition performance impairment.
On the change of recognition performance due to mask use during recognition (unmasked–unmasked condition minus unmasked–masked condition), smaller performance impairment in d′ was correlated with larger change toward low conditional entropy of the third fixation given the second fixation with mask use, r(86) = − .217, p = .043 (Fig. 5). This correlation was still significant when we partialled out general intelligence as measured in RSPM, r(85) = − .235, p = .028, and it became marginal when we partialled out both general intelligence and cognitive ability measures, r(75) = − .217, p = .058. In an explorative analysis examining which cognitive abilities were correlated with change in conditional entropy of the third fixation given the second fixation, we found that it was correlated with general intelligence (RSPM), r(86) = − .276, p = .009, and TOL execution time, r(85) = .214, p = .045, suggesting that higher general intelligence and shorter execution time in TOL were associated with larger change toward low conditional entropy of the third fixation given the second fixation. These results suggested that individuals who had increased eye gaze transition consistency when recognizing a masked face that was learned without a mask on had less recognition performance impairment due to the mask use.
On the change in performance due to mask use in the face recognition task (unmasked–unmasked condition minus masked–masked condition), the change in face recognition performance was not associated with the change in any of the eye movement behavior measures.