A radiologist is asked to read a chest X-ray to determine if a patient has pneumonia. After assessing the exam, he decides that she does not. He is correct; she does not have pneumonia, but she does have clear signs of lung cancer that the radiologist fails to report. The radiologist has missed an “incidental finding” (Beigelman-Aubry, Hill, & Grenier, 2007).
Incidental findings are items of potential clinical significance that may not have been the primary object of the search of the image. In one review, incidental findings appeared on 24% of a mixed collection of radiologic cases (Lumbreras, Donat, & Hernández-Aguado, 2010). Not all such findings turn out to be important. The vigor with which incidental findings should be reported and followed up is debatable (Berlin, 2016; Pandharipande et al., 2016a, 2016b). A recent study of head computed tomography (CT) from 5800 patients described possible incidental findings in about 10% of cases, followed up on about 3% and found that most of those were “without direct clinical consequences” (Bos et al., 2016). Nevertheless, there are cases where the missed finding is clinically significant and where failure to report the finding can have adverse consequences for the patient as well as for the clinician, in the form of a malpractice suit. Radiologists know that the search for these incidental targets is part of the task and should be considered whenever they look at an image.
Very similar problems occur outside of the medical field as well. If you are about to cross the street and you look both ways for cars only to be very nearly knocked down by a bicycle, you have, arguably, committed the same type of error. You knew that your task was to look for anything that might have direct consequence on your ability to safely cross the street, yet you failed to respond to the bicycle (which, for purposes of argument, we will assume was clearly visible). How can we better understand the processes behind this common problem and how should we try to ameliorate it? This class of real-world problems is difficult to study in the real world. If we stay with the radiology example, it is possible to retrospectively study the issue; for example, by doing a second reading of a set of cases, specifically looking for incidental findings. However, it is neither practical nor ethical to manipulate variables in the clinic simply on the hunch that they might alter the rate of incidental findings found. Rather, like other problems in medicine, we need a model system that can be studied extensively in the lab before proposing more limited, evidence-based hypotheses that can be tested in the clinic (or the bike lane). The purpose of this paper is to propose one such model system, “mixed hybrid search,” in which observers search a visual display for some specific targets (e.g., “this rabbit”) and some general targets (e.g., “any vehicle”). As we will describe, the chance of missing a general target can be markedly elevated in mixed hybrid tasks, perhaps in a manner similar to the elevated rate with which incidental findings are missed.
Before describing the mixed hybrid model system, it is worth discussing two other possible models that have been extensively studied in recent years: inattentional blindness (Mack & Rock, 1998) and satisfaction of search (Berbaum et al., 1990; Tuddenham, 1962). Perhaps the most famous example of inattentional blindness is the Simons and Chabris (1999) gorilla experiment. In that experiment, observers are asked to count the number of times the white-shirted team touches the ball in a ball-passing game. About half of those observers fail to report an actor in a gorilla suit walk through the middle of the game. The phenomenon has been extensively researched (Cohen, Cavanagh, Chun, & Nakayama, 2012) with extensions to other senses (audition: Dalton & Fraenkel, 2012) and to various real-world settings (Castel, Vendetti, & Holyoak, 2012; Chabris & Simons, 2011). Our lab explicitly connected the phenomenon to the incidental finding problem by placing an image of a gorilla in a lung CT and showing that expertise did not immunize radiologists against inattentional blindness. Twenty of 24 radiologists failed to report the gorilla (Drew, Vo, & Wolfe, 2013).
The difficulty with inattentional blindness as a model of incidental findings is that no radiologist is looking for a gorilla in the lung, even incidentally. Nor were Simons and Chabris (1999) observers looking for a gorilla. Indeed, the effect goes away if observers are told to count ball passes and to keep an eye open for the occasional gorilla. Incidental findings, in contrast, are targets that plausibly could be present and that should be kept in mind, but are often missed, nevertheless.
The phenomenon of satisfaction of search involves missing targets that the observer is, in fact, looking for. The problem was originally described in radiology when it was discovered that finding one target (e.g., a fracture) made it less likely that a second target in the same image would be found (Berbaum et al., 2001). The problem was dubbed “satisfaction of search” by Tuddenham (1962), based on the hypothesis that observers were “satisfied” by finding the first target and abandoned the search too quickly thereafter. Berbaum et al. (1991) subsequently showed that the rapid quitting idea was not correct. Nevertheless, the term persists though Adamo, Cain, and Mitroff (2013) have proposed “Subsequent Search Misses (SSM)” as a theory-neutral term. A significant body of work exists in both medical image perception (reviewed in Berbaum, Franken, Caldwell, & Shartz, 2010) and in the basic visual cognition literature (Cain, Adamo, & Mitroff, 2013). Nevertheless, like inattentional blindness, satisfaction of search is not quite the right model system for incidental findings. There are two problems. First, the missed, second target, is typically of the same type as the first target: two fractures, two lung nodules, etc. Incidental findings are typically of a different type than the primary target of search: look for pneumonia, miss the cancer. Second, by definition, the error in satisfaction of search is the missing of a second target in an image. An incidental finding can be the only clinically significant finding in the image, but missed nevertheless.
Our goal is to create a model system for studying incidental findings in which observers know what they are looking for but, nevertheless, show elevated error rates for one class of stimuli that serve as our stand-in for incidental findings. To do this, we had observers search for a mixture of specific and categorical target types. The logic of this mixture approach is that the observer will know the nature of the targets (no surprise gorillas). We know that attention can be guided to categorical targets (Nako, Wu, Smith, & Eimer, 2014; Yang & Zelinsky, 2009), but we would expect observers to be less precise in their search for these less precise, categorical target (Maxfield & Zelinsky, 2012). We would expect them to miss more categorical than specific items. This elevated error rate can, then, stand in for the incidental finding errors we are trying to model.
Search tasks that have observers searching for multiple types of targets in a visual search display are known as “hybrid search” tasks (Schneider & Shiffrin, 1977; Wolfe, 2012a): “Hybrid” because they combine visual search with memory search in the same task. Using photographic images of specific objects, Wolfe (2012a, 2012b) found that hybrid search was characterized by reaction times (RTs) that were a linear function of the visual set size – the number of items in the visual display. The RTs were a linear function of the log of the memory set size – the number of target types held in memory. In these experiments, observers typically learn a memory set of target items and then search for members of that set in a block of several hundred trials. Different patterns of results are found if observers learn new targets on each trial (Nosofsky, Cao, Cox, & Shiffrin, 2014; Nosofsky, Cox, Cao, & Shiffrin, 2013). The logarithmic relationship between memory set size and hybrid search RT holds for large memory set sizes of 100 (Wolfe, 2012a) or even 500 specific items (Wolfe, Boettcher, Josephs, Cunningham, & Drew, 2015) and appears to be based on the recognition of items as targets rather than a more basic feeling of ‘familiarity’ (Wolfe, Boettcher, Josephs, Cunningham, & Drew, 2015). A similar pattern of results is seen with other types of targets such as words (Boettcher & Wolfe, 2015).
Importantly for present purposes, the same pattern of results is seen when broad categories are used as stimuli (Cunningham & Wolfe, 2014). Observers cannot easily memorize 100 categories in the way that they can memorize 100 specific objects. However, in Cunningham and Wolfe (2014), observers could easily memorize 1–8 categories like “plants, furniture, animals, weapons, picture frames, signs, flags, and cars.” Results again showed a linear relationship of RT to the visual set size and to the log of the memory set size. Searching for categorical targets is markedly more difficult that searching for specific targets. This is illustrated in Fig. 1.
The figure shows average RTs for target-present trials with a memory set of eight target types and visual set sizes of four, eight, and 16 items. These are extracted from larger data sets for illustrative purposes from Wolfe (2012a, 2012b) in the case of specific target types and Cunningham and Wolfe (2014) in the case of categorical target types. Error rates are low (<10%) in both conditions.
Clearly, the categorical task is slower than the specific task. Suppose that we mixed target types. That is, on any given trial, the target could be one of four specific objects or a member of one of four categories. It could be that the internal process of testing if the current visual item belongs to this memory set of eight items becomes as slow as the search for eight categorical target types. It could be that, as in Fig. 1, the specific targets are identified quickly and the categorical targets are found slowly. If that is the case, what happens when no target is found? Search termination seems to involve setting an internal quitting threshold based on experience with finding targets (Chun & Wolfe, 1996; Moran, Zehetleitner, Liesefeld, Müller, & Usher, 2015; Schwarz & Miller, 2016; Wolfe, 2012a, 2012b). With two different types of targets, the quitting threshold could reflect the time to find the harder targets. However, if the quitting time was substantially influenced by a contribution from the easier targets, observers might quit relatively quickly and, as a consequence, they might miss a relatively high proportion of the more difficult, categorical targets. This is, in fact, what the data show.