How did I miss that? Developing mixed hybrid visual search as a ‘model system’ for incidental finding errors in radiology

Incidental findings are a significant issue in clinical radiology. They are difficult to study in a systematic fashion because both the stimulus material and the observer population are hard to assemble. Here we propose a “model system”, in the form of a ‘mixed hybrid search’ task. It can be used to investigate the fundamental cognitive processes that lie behind incidental finding errors. Once those principles are identified, tractable experiments can be designed for clinical settings with expert observers.

A radiologist is asked to read a chest X-ray to determine if a patient has pneumonia. After assessing the exam, he decides that she does not. He is correct; she does not have pneumonia, but she does have clear signs of lung cancer that the radiologist fails to report. The radiologist has missed an “incidental finding” (Beigelman-Aubry, Hill, & Grenier, 2007).

Incidental findings are items of potential clinical significance that may not have been the primary object of the search of the image. In one review, incidental findings appeared on 24% of a mixed collection of radiologic cases (Lumbreras, Donat, & Hernández-Aguado, 2010). Not all such findings turn out to be important. The vigor with which incidental findings should be reported and followed up is debatable (Berlin, 2016; Pandharipande et al., 2016a, 2016b). A recent study of head computed tomography (CT) from 5800 patients described possible incidental findings in about 10% of cases, followed up on about 3% and found that most of those were “without direct clinical consequences” (Bos et al., 2016). Nevertheless, there are cases where the missed finding is clinically significant and where failure to report the finding can have adverse consequences for the patient as well as for the clinician, in the form of a malpractice suit. Radiologists know that the search for these incidental targets is part of the task and should be considered whenever they look at an image.

Very similar problems occur outside of the medical field as well. If you are about to cross the street and you look both ways for cars only to be very nearly knocked down by a bicycle, you have, arguably, committed the same type of error. You knew that your task was to look for anything that might have direct consequence on your ability to safely cross the street, yet you failed to respond to the bicycle (which, for purposes of argument, we will assume was clearly visible). How can we better understand the processes behind this common problem and how should we try to ameliorate it? This class of real-world problems is difficult to study in the real world. If we stay with the radiology example, it is possible to retrospectively study the issue; for example, by doing a second reading of a set of cases, specifically looking for incidental findings. However, it is neither practical nor ethical to manipulate variables in the clinic simply on the hunch that they might alter the rate of incidental findings found. Rather, like other problems in medicine, we need a model system that can be studied extensively in the lab before proposing more limited, evidence-based hypotheses that can be tested in the clinic (or the bike lane). The purpose of this paper is to propose one such model system, “mixed hybrid search,” in which observers search a visual display for some specific targets (e.g., “this rabbit”) and some general targets (e.g., “any vehicle”). As we will describe, the chance of missing a general target can be markedly elevated in mixed hybrid tasks, perhaps in a manner similar to the elevated rate with which incidental findings are missed.

Before describing the mixed hybrid model system, it is worth discussing two other possible models that have been extensively studied in recent years: inattentional blindness (Mack & Rock, 1998) and satisfaction of search (Berbaum et al., 1990; Tuddenham, 1962). Perhaps the most famous example of inattentional blindness is the Simons and Chabris (1999) gorilla experiment. In that experiment, observers are asked to count the number of times the white-shirted team touches the ball in a ball-passing game. About half of those observers fail to report an actor in a gorilla suit walk through the middle of the game. The phenomenon has been extensively researched (Cohen, Cavanagh, Chun, & Nakayama, 2012) with extensions to other senses (audition: Dalton & Fraenkel, 2012) and to various real-world settings (Castel, Vendetti, & Holyoak, 2012; Chabris & Simons, 2011). Our lab explicitly connected the phenomenon to the incidental finding problem by placing an image of a gorilla in a lung CT and showing that expertise did not immunize radiologists against inattentional blindness. Twenty of 24 radiologists failed to report the gorilla (Drew, Vo, & Wolfe, 2013).

The difficulty with inattentional blindness as a model of incidental findings is that no radiologist is looking for a gorilla in the lung, even incidentally. Nor were Simons and Chabris (1999) observers looking for a gorilla. Indeed, the effect goes away if observers are told to count ball passes and to keep an eye open for the occasional gorilla. Incidental findings, in contrast, are targets that plausibly could be present and that should be kept in mind, but are often missed, nevertheless.

The phenomenon of satisfaction of search involves missing targets that the observer is, in fact, looking for. The problem was originally described in radiology when it was discovered that finding one target (e.g., a fracture) made it less likely that a second target in the same image would be found (Berbaum et al., 2001). The problem was dubbed “satisfaction of search” by Tuddenham (1962), based on the hypothesis that observers were “satisfied” by finding the first target and abandoned the search too quickly thereafter. Berbaum et al. (1991) subsequently showed that the rapid quitting idea was not correct. Nevertheless, the term persists though Adamo, Cain, and Mitroff (2013) have proposed “Subsequent Search Misses (SSM)” as a theory-neutral term. A significant body of work exists in both medical image perception (reviewed in Berbaum, Franken, Caldwell, & Shartz, 2010) and in the basic visual cognition literature (Cain, Adamo, & Mitroff, 2013). Nevertheless, like inattentional blindness, satisfaction of search is not quite the right model system for incidental findings. There are two problems. First, the missed, second target, is typically of the same type as the first target: two fractures, two lung nodules, etc. Incidental findings are typically of a different type than the primary target of search: look for pneumonia, miss the cancer. Second, by definition, the error in satisfaction of search is the missing of a second target in an image. An incidental finding can be the only clinically significant finding in the image, but missed nevertheless.

Our goal is to create a model system for studying incidental findings in which observers know what they are looking for but, nevertheless, show elevated error rates for one class of stimuli that serve as our stand-in for incidental findings. To do this, we had observers search for a mixture of specific and categorical target types. The logic of this mixture approach is that the observer will know the nature of the targets (no surprise gorillas). We know that attention can be guided to categorical targets (Nako, Wu, Smith, & Eimer, 2014; Yang & Zelinsky, 2009), but we would expect observers to be less precise in their search for these less precise, categorical target (Maxfield & Zelinsky, 2012). We would expect them to miss more categorical than specific items. This elevated error rate can, then, stand in for the incidental finding errors we are trying to model.

Search tasks that have observers searching for multiple types of targets in a visual search display are known as “hybrid search” tasks (Schneider & Shiffrin, 1977; Wolfe, 2012a): “Hybrid” because they combine visual search with memory search in the same task. Using photographic images of specific objects, Wolfe (2012a, 2012b) found that hybrid search was characterized by reaction times (RTs) that were a linear function of the visual set size – the number of items in the visual display. The RTs were a linear function of the log of the memory set size – the number of target types held in memory. In these experiments, observers typically learn a memory set of target items and then search for members of that set in a block of several hundred trials. Different patterns of results are found if observers learn new targets on each trial (Nosofsky, Cao, Cox, & Shiffrin, 2014; Nosofsky, Cox, Cao, & Shiffrin, 2013). The logarithmic relationship between memory set size and hybrid search RT holds for large memory set sizes of 100 (Wolfe, 2012a) or even 500 specific items (Wolfe, Boettcher, Josephs, Cunningham, & Drew, 2015) and appears to be based on the recognition of items as targets rather than a more basic feeling of ‘familiarity’ (Wolfe, Boettcher, Josephs, Cunningham, & Drew, 2015). A similar pattern of results is seen with other types of targets such as words (Boettcher & Wolfe, 2015).

Importantly for present purposes, the same pattern of results is seen when broad categories are used as stimuli (Cunningham & Wolfe, 2014). Observers cannot easily memorize 100 categories in the way that they can memorize 100 specific objects. However, in Cunningham and Wolfe (2014), observers could easily memorize 1–8 categories like “plants, furniture, animals, weapons, picture frames, signs, flags, and cars.” Results again showed a linear relationship of RT to the visual set size and to the log of the memory set size. Searching for categorical targets is markedly more difficult that searching for specific targets. This is illustrated in Fig. 1.

The figure shows average RTs for target-present trials with a memory set of eight target types and visual set sizes of four, eight, and 16 items. These are extracted from larger data sets for illustrative purposes from Wolfe (2012a, 2012b) in the case of specific target types and Cunningham and Wolfe (2014) in the case of categorical target types. Error rates are low (<10%) in both conditions.

Clearly, the categorical task is slower than the specific task. Suppose that we mixed target types. That is, on any given trial, the target could be one of four specific objects or a member of one of four categories. It could be that the internal process of testing if the current visual item belongs to this memory set of eight items becomes as slow as the search for eight categorical target types. It could be that, as in Fig. 1, the specific targets are identified quickly and the categorical targets are found slowly. If that is the case, what happens when no target is found? Search termination seems to involve setting an internal quitting threshold based on experience with finding targets (Chun & Wolfe, 1996; Moran, Zehetleitner, Liesefeld, Müller, & Usher, 2015; Schwarz & Miller, 2016; Wolfe, 2012a, 2012b). With two different types of targets, the quitting threshold could reflect the time to find the harder targets. However, if the quitting time was substantially influenced by a contribution from the easier targets, observers might quit relatively quickly and, as a consequence, they might miss a relatively high proportion of the more difficult, categorical targets. This is, in fact, what the data show.

In the real incidental finding situation, the items that are likely to be missed are not only a different type of target, they also appear less frequently in the population. Thus, if you are checking images of the lung for pneumonia, you should report lung cancer if it is present, but, not only is it not your specific target, it is also probably less likely to appear in a collection of patients suspected of having pneumonia. In Experiment 2, we repeat the mixed hybrid search experiment with a systematic variation in the prevalence of the specific and categorical targets.

Results and Discussion

Trials with RTs < 200 msec or greater than 10,000 msec were eliminated. This removed just 0.4% of trials.

Figure 4 shows the RT × set size functions for each of the conditions with RTs for specific and categorical targets shown separately. An ANOVA with condition (specific target, categorical target, and absent) and visual set size as factors shows large effects of condition (F(4,44) = 30.8, p < 0.0001_, ges = 0.42), a strong effect of visual set size (F(2,22) = 177.0, p < 0.0001_, ges = 0.60), and a strong interaction (F(8,88) = 15.5, p < 0.0001_, ges = 0.09).

Again, it is the miss errors that are of prime interest and the critical condition is the condition in which the categorical targets are rare: the 20/80 condition. In that condition, observers missed just 5% of the specific targets and 37.5% of the categorical targets (t(11) = 20.6, p < 0.0001). The 50/50 case replicates the equivalent condition of the previous experiment: Specific errors constituted 9% of errors, categorical errors 23% (t(11) = 7.5, p < 0.0001). Only when categorical targets are four times more common than specific targets does the difference go away: Categorical errors drop to 14% compared to 17% specific errors (11) = 1.4, p < 0.18).

The results are clear and in line with previous work on target prevalence in search (Baddeley & Colquhoun, 1969; Wolfe, 2012b; Wolfe & VanWert, 2010): as the probability of a target decreases, the chance that such targets will be missed increases. Rare categorical targets are missed more than common categorical targets and rare specific targets are missed more than common specific targets. The combination of this prevalence effect with the effects of mixing specific and categorical targets produces dramatic differences in error rates in the critical case of rare, categorical targets. Categorical targets are more than seven times more likely to be missed in the 20/80 condition, shown in Fig. 5.

In Experiments 1 and 2, the presence of specific targets is inversely related to the presence of categorical targets because a given trial has either one or the other type of target. It might have no target. It cannot have both. As a model of incidental findings, this is flawed since the presence of a broken rib does not rule out the possibility of pneumonia, for example. Accordingly, in Experiment 3, the presences of specific and categorical targets on a trial were independent of each other. This raises the possibility of multiple targets on each trial.

Results

Given the need to locate up to two targets and to make an explicit search termination response, trials were removed from analysis if the first or second RT was greater than 7000 msec or if the total RT for the trial was less than 200 or greater than 20,000 msec in length. This removed 0.5% of trials.

The pattern of RTs is shown in Fig. 6.

It shows the same pattern of findings as the earlier experiments. It is somewhat harder to find categorical targets than to find specific targets. Target 2 (T2) RTs are measured from the time of the T1 response. Obviously, they are very similar to the T1 responses with, perhaps, a small increase in overall RT. Absent trials, as is typical in search experiments, are slower and increase more steeply with visual set size.

As in Experiments 1 and 2, the important data in Experiment 3 are the error rates. Figure 7 shows error rates for trials with either one or no target.

Obviously, replicating Experiments 1 and 2, categorical targets are missed at a much higher rate (36.6%) than specific targets (3.6%), (t(11) = 9.1, p < 0.0001). Figure 8 shows the results when two categorical or two specific targets were present. Here we show the chance of missing one and the chance of missing both.

As above, many more categorical targets are missed (Miss one: t(11) = 9.4, p < 0.0001; Miss both: t(11) = 7.3, p < 0.0001). We can use the results in Fig. 7 to predict those in Fig. 8. Specifically, if we know the probability of finding one target, the probability of finding two, independent instances should be P(find 1) * P(find 1). For the specific targets, this works as expected. The average of the square of the chance of finding one target is 93.0% and the probability of finding two specific targets when two are present is 92.4%. The difference is not significant (t(11) = 0.9, p = 0.39). Interestingly, this is not true for categorical targets. The average of the square of the chance of finding one categorical target is 42.3%, but the probability of actually finding two categorical targets when two are present is 57%, (t(11) = 5.0, p = 0.0004). This may be an example of a reverse satisfaction of search effect (Berbaum et al., 1990; Berbaum et al., 2015). Normally, finding one target makes it less likely that a second will be found. Here, finding one, relatively rare categorical item may be reminding the observer that it might be worth look for a second.

The most interesting case in Experiment 3 is the case where there are both categorical and specific targets in the same display. Figure 9 shows the pattern of errors in these trials.

Observers missed both targets 2.1% of the time. They missed only the specific target on another 2.1% of trials and missed only the categorical target on 34.8%. The difference is highly significant (t(11) = 8.5, p = 0.0001). On the 61% of trials where both items were found, the specific item was found first on 2/3rds of the cases (t(11) = 5.5, p = 0.0002). The probability of finding both items can be predicted from the probabilities of finding the single items: P(find 1 specific target) = 96.4%, P(categorical) = 63.4%; 96.4%*63.4% = 61.4%, actual P(find both) = 61.6%; t(11) = 0.8, p = 0.47). Thus, unlike the case of two categorical targets, finding a specific target does not appear to remind observers to look for a categorical target. The chance of finding both is equal to the product of finding each alone.

The mixed hybrid search paradigm, introduced in this paper, appears to have promise as a ‘model system’ in which to study the problem of missed incidental findings in medicine. We can produce large and reliable rates of false negative errors in the search for targets that are known to the observer and searched for by the observer. This is an advance over, for example, our finding that radiologists miss a gorilla inserted into the lung (Drew et al., 2013). While it is interesting that most radiologists missed a gorilla, it must be conceded that they really were not looking for gorillas. In the present experiments, observers missed a third of items like “masks”, “signs”, or “furniture”, even though they knew perfectly well that they were looking for members of those categories. The errors in our experiments are not examples of what would typically be called “inattentional blindness” (Mack & Rock, 1998; Ward & Scholl, 2015). These are simply search errors; though these results and the analogy with incidental finding errors make it clear that simple search errors and inattentional blindness may be more closely related than we tend to think.

Is mixed hybrid search actually a good model of incidental finding errors? An answer to that question is beyond the scope of this paper, but the results of these experiments do provide a roadmap for future investigation into their relationship to the clinical situation we are trying to model. First, we did not find evidence of a satisfaction of search (or SSM) effect on trials where there was one specific and one categorical target. If mixed hybrid search is a model of incidental findings, then that should be true in the clinical setting as well. That is, the probability of reporting an incidental finding should be similar on cases that are positive or negative for the primary reason for the exam. This could be assessed though a review of existing cases.

Second, recall that observers did better than expected when there were two categorical targets. This suggests that the discovery of the first categorical target ‘primed’ the detection of the second (Kristjansson, Saevarsson, & Driver, 2013; Maljkovic & Nakayama, 1994). Perhaps we can reduce incidental finding errors by reminding observers to look for those loosely defined targets. Some priming effects can be quite long lasting (Kruijne & Meeter, 2016), so it is possible that showing observers the equivalent of these categorical targets at the start of a reading session might cut down on errors. This could be tested using our model system and then, if successful, tried in a clinical setting. This points to the basic motivation for this work. If mixed hybrid search is a good model for incidental finding errors, then interventions that bring down that >30% error rate in the lab might bring down the error rate in the clinic as well. Then, not only will we know more about the fundamental processes of visual search, we will be able to use that knowledge to improve medical care. One final note about reducing the incidental finding error rate: Ideally, we would want to reduce those errors without increasing the rates of false positive errors. We tend to focus on miss errors and, to be sure, missing a clinically significant finding is important. However, false positive errors carry their own costs. Moreover, in cases where the pathology is rare (e.g., cancer screening), a change in the false positive rate will affect many more people than a change of the same percentage in the miss error rate. Even if a single miss error is more consequential than a single false positive, a mass of false positives could, in principle, outweigh the benefits of fewer misses. In signal detection terms, our first goal should be to improve d’. If we are merely shifting criterion, we need to weigh the costs of one type of error against another.

To briefly summarize, this paper introduces the “mixed hybrid search” paradigm, in which observers look through visual displays for any instance of several specific targets (“this boot”, “this screwdriver”, etc.) and several categorical targets (“any vegetable”, “any toy”, etc.). The key finding is that observers miss many more of the categorical targets, especially when the categorical targets are relatively rare. We propose this as a ‘model system’ for the problem of incidental findings in radiology. If we can manipulate the errors in the lab, we may be able to do the same in the clinic.