Eye movement feedback fails to improve visual search performance

In real-world search tasks, such as baggage screening, radiology, and surveillance, the absence of a potential threat (e.g., a gun, cancer, or enemy base) is never certain. This leaves observers lacking knowledge regarding their performance because they cannot know when or how frequently they have missed a target. Yet, in laboratory-based studies of visual search, feedback that a target was missed is a critical cue to adapt search strategies (Chun & Wolfe, 1996). Because this cue is unavailable in real-world scenarios, we attempted to develop a new form of feedback, eye movement feedback (EMF), to help adapt search strategies. EMF uses eye tracking to visually mark areas that the observer has already searched for a potential target. By marking previously inspected areas, EMF’s goal is to make the observer aware of the unsearched areas of an image, thus guiding their attention to new areas for which a target may be present. If effective, EMF could reduce selection errors or failures to inspect a target before deciding a target is absent, which are the most common form of misses (Peltier & Becker, 2016a). We evaluated the effectiveness of EMF through four experiments where we compared rare target detection rates between EMF and control groups. Despite using different forms of EMF and search stimuli, we found that EMF rarely increased accuracy and in some cases actually reduced it. We conclude that EMF is unlikely to be an effective tool to improve visual search performance.

Radiologists miss as many as 30% of cancers in their examinations (Berlin, 1994; Bird, Wallace, & Yankaskas, 1992). Eye-tracking studies indicate that more than one-third of these misses are the result of an incomplete search in which the radiologist fails to fixate the area around the cancer (Bird et al., 1992; Krupinski, 1995). One possible reason why misses are so high is because cancers are quite rare in scans.

Research suggests that miss rates are far higher for low- than for high-prevalence search targets (Ishibashi, Kita, & Wolfe, 2012; Mitroff & Biggs, 2014; Rich et al., 2008; Schwark, Sandry, Macdonald, & Dolgov, 2012; Wolfe et al., 2007; Wolfe, Horowitz, & Kenner, 2005). Similarly to what occurs in radiology, many of these misses are selection errors (Peltier & Becker, 2016a) whereby the observer fails to inspect the target before responding target absent. To explain this pattern, researchers have posited that rare targets result in low quitting thresholds (Wolfe et al., 2005; Wolfe & Van Wert, 2010). With low quitting thresholds, observers inspect less of the display before responding that the target is absent, resulting in faster reaction times but many misses (Gur et al., 2003; Rich et al., 2008; Schwark et al., 2012; Schwark, Macdonald, Sandry, & Dolgov, 2013; Schwark, Sandry, & Dolgov, 2013; Van Wert, Horowitz, & Wolfe, 2009; Wolfe et al., 2005; Wolfe et al., 2007).

Because most errors are caused by an incomplete search (Peltier & Becker, 2016a), a manipulation that delays search termination might entice observers to perform a more thorough search, thereby increasing the hit rate. On the basis of this logic, Wolfe et al. (2007) attempted to alleviate the low prevalence effect (LPE) by giving “speeding tickets” when responses were made too quickly. This manipulation successfully increased latency to absent responses. Unexpectedly, there was no change in accuracy, criterion, or sensitivity, despite the increased response time. One possible explanation for this pattern of results is that observers made an internal target-absent decision and ceased active search but delayed making a target-absent response to avoid a speeding ticket. These results suggest that simple delay manipulations are particularly ineffective; searches take longer but are no more accurate.

Wolfe et al. (2007) conducted seven experiments and found only one technique that was successful in increasing hit rates in low-prevalence searches. This technique involved interspersing low-prevalence search blocks without feedback with blocks of high-prevalence trials with trial-by-trial performance-based feedback (information about correct or incorrect responses). Although this approach increased target detections in the low-prevalence search blocks, sensitivity did not increase. Instead, the increased hit rate was accompanied by increased false alarm rates, a signature of a shift toward a more liberal decision criterion or an informed increase in target-present guessing as prevalence increases (Peltier & Becker, 2017a, 2017b). This type of criterion shift may not be desirable, particularly because there are many opportunities for false alarms in a low-prevalence search. If this method were applied to a real-world situation, observers’ workloads would increase to allow for “dummy blocks” of high-prevalence searches, and the manipulation would result in a higher proportion of false alarms. Ideally, a manipulation that reduces the LPE would increase hits without an increase in false alarms.

Although Wolfe et al.’s (2007) interspersed high-prevalence block experiment did not achieve this ideal, it did demonstrate the important role that feedback can play in setting quitting thresholds. Indeed, according to an influential model of quitting thresholds, feedback about misses is critical to adaptably setting quitting thresholds (Chun & Wolfe, 1996; Danielmeier & Ullsperger, 2011). Under this theory, feedback about a missed target provides information that the previous search was not adequate, thereby increasing quitting thresholds. In low-prevalence and real-world search scenarios, there may be insufficient feedback about misses to set optimal quitting thresholds. In low-prevalence search tasks, the sparse targets result in few opportunities for feedback about misses. In real-world search tasks, the ability to give accurate feedback about one’s performance is often unavailable because the ground truth is unknown.

Given the important role that feedback can play in setting quitting thresholds, as well as the problems associated with providing performance-based feedback (feedback that a target was missed) in low-prevalence and real-world search scenarios, here we attempt to improve search by providing a different form of feedback, namely “eye movement feedback (EMF),” that provides real-time feedback about an observer’s scanning pattern, allowing an observer to see what aspects of the scene they have and have not searched. The ability to provide this type of EMF requires no foreknowledge of a target’s presence or absence and is not impacted by target prevalence rates. Even so, it may act as a substitute for performance-based feedback and may influence quitting thresholds to make searches more complete.

To investigate this possibility, across four experiments, we provided observers with EMF while they searched for rare targets. Three of these experiments used an eye tracker to provide real-time feedback about the portions of the scene that had been inspected during the trial. A fourth automatically revealed sections of the display one at a time, thereby providing observers information about the amount of the scene that had not yet been searched. Although our initial experiment showed promise for the EMF method, across the experiments and manipulations, it became clear that the EMF approach is not a panacea that mitigates the high miss rates that occur when search targets are rare.

What conclusions can we draw from this mixed bag of results? First, we can say with some confidence that EMF is not a panacea that eliminates the high miss rates associated with a low-prevalence search. Although we found one case (Experiment 1) where it appeared to help, that result seems to be the exception rather than the rule. Making slight modifications to the type of feedback (Experiments 2 and 3) or the type of stimuli (Experiment 4) showed no benefit of EMF and in some cases showed a cost of doing so. In short, our results suggest that providing EMF is not likely to improve rare target detection.

It is worth noting that, at the outset, we truly thought that this was a promising approach, and we wanted to find evidence that EMF could help. Drew and Williams (in press) also believed this method might be effective. However, across a number of attempts, they also found little evidence that EMF improved performance (also see Dickinson & Zelinsky, 2005).

Across the present experiments, as well as those of Drew and Williams, a wide variety of feedback approaches have been tried; stimuli and search difficulty have varied widely; and the consensus of the results as a whole is that this type of EMF is an ineffective way of reducing the high miss rates in low-prevalence searches.

As such, the work we present here, as well as that of Drew and Williams, adds to a growing body of work that highlights how robust and stubborn the low-prevalence effect is (Kunar, Rich, & Wolfe, 2010; Schwark, Macdonald, et al., 2013; Wolfe et al., 2007). Although there were solid theoretical bases to believe that EMF may improve search for rare search targets (Hout, Walenchok, Goldinger, & Wolfe, 2015; Peltier & Becker, 2016a), visual search performance seems unlikely to be improved by these manipulations. At the theoretical level, this suggests that the quitting thresholds and decision criteria that have been deemed responsible for the low-prevalence effect (Hout et al., 2015; Peltier & Becker, 2016a; Wolfe & Van Wert, 2010), although sensitive to target prevalence, may be insensitive to other types of manipulations.

At a practical level, these results offer two conclusions. First, we would not recommend that researchers attempt to implement this type of feedback in hopes of improving rare target detections. On the whole, the evidence suggests that doing so is unlikely to improve performance, and depending on how the manipulation is performed, it may even hinder performance. Second, we would advise that researchers who are interested in discovering ways of improving visual search performance for rare targets explore other techniques. Across both our sets of experiments and those of Drew and Williams, we believe the space of possible EMF interventions has been well probed. Indeed, in our laboratory, we have moved on, and rather than finding ways to modify the search tasks to improve low-prevalence search performance, we have had success (Peltier & Becker, 2016b, 2017a) investigating individual differences and developing a screener that could be used to identify those individuals who are likely to have high target detection rates, even in the face of low target prevalence. That approach has been more successful in both our laboratory (Peltier & Becker, 2016b, 2017a, 2017b) and another (Schwark, Sandry, et al., 2013).

Finally, it is worth discussing the fact that some of our manipulations appear to reduce target detection rates. Our belief is that some of these manipulations (those in Experiments 2 and 3) may have encouraged people to move rapidly from section to section without revisiting previously inspected segments. Given that low prevalence shifts the decision criterion for evaluating an object as a target higher (Peltier & Becker, 2016a; Wolfe & Van Wert, 2010), the reduced time spent inspecting a segment may lead to more identification errors (inspecting an object but not identifying it as a target). Consistent with this speculation, we have previously reported that successful target detection requires longer fixation on the target when targets are rare (Peltier & Becker, 2016a, 2016b). If an EMF method reduces the time spent viewing each item, it may increase these types of identification errors.

Despite our best efforts to develop a method of providing real-time feedback about the search process (i.e., which areas have and have not been inspected) to improve rare target search, we conclude that these EMF manipulations, as we have implemented them here, are unlikely to improve rare target search performance and may even hinder it. Taken together with the independent results of Drew and Williams, we believe we have relatively compelling evidence that attempting to improve rare target search with EMF is an unfruitful approach.