Supporting dynamic change detection: using the right tool for the task
© The Author(s) 2016
Received: 23 June 2016
Accepted: 17 November 2016
Published: 19 December 2016
Detecting task-relevant changes in a visual scene is necessary for successfully monitoring and managing dynamic command and control situations. Change blindness—the failure to notice visual changes—is an important source of human error. Change History EXplicit (CHEX) is a tool developed to aid change detection and maintain situation awareness; and in the current study we test the generality of its ability to facilitate the detection of changes when this subtask is embedded within a broader dynamic decision-making task. A multitasking air-warfare simulation required participants to perform radar-based subtasks, for which change detection was a necessary aspect of the higher-order goal of protecting one’s own ship. In this task, however, CHEX rendered the operator even more vulnerable to attentional failures in change detection and increased perceived workload. Such support was only effective when participants performed a change detection task without concurrent subtasks. Results are interpreted in terms of the NSEEV model of attention behavior (Steelman, McCarley, & Wickens, Hum. Factors 53:142–153, 2011; J. Exp. Psychol. Appl. 19:403–419, 2013), and suggest that decision aids for use in multitasking contexts must be designed to fit within the available workload capacity of the user so that they may truly augment cognition.
KeywordsChange blindness Dynamic decision-making Change History EXplicit (CHEX) Eye tracking Decision support system
Detecting changes in our visual environment is essential to several everyday activities (e.g. driving a car), as well as in safety-critical work settings such as air traffic control and security surveillance. Change blindness—the incapacity to notice visual changes—can negatively impact decision-making and lead to important consequences such as human errors, critical incidents, and, in worse cases, loss of human lives. This current study intends to broaden our understanding of the sources of change blindness and finds that while some changes may go unnoticed because they are never looked at, participants within a complex environment may also be “blind” to changes/events upon which they are fixating. In a multitasking context, attention is stretched and necessarily divided among several subtasks, leaving fewer attentional resources to ensure that a change—although fixated—actually reaches conscious awareness. While change detection support tools are available, their efficacy is dependent upon the available attentional capacity of the user. We demonstrate that one tool (a change history table) designed to improve detection performance within an air traffic control simulated environment, actually worsens detection performance—perhaps due to an increase in workload and subsequently stretched attentional resources that render the changes less likely to be consciously detected. This indicates the importance of testing potential support tools within a dynamic and demanding environment to determine the extent to which they are actually able to improve performance.
The ability to discern pertinent objects and events in our environment is essential to a multitude of everyday activities, such as driving and air traffic control (St. John, Smallman, & Manes, 2005). A driver, for example, must constantly monitor the unfolding dynamic visual scene for incoming traffic and changing road signs in order to drive safely and avoid hazards (Horswill & McKenna, 2004). The failure to detect task-relevant changes in a visual scene, often referred to as change blindness (CB; Rensink, Regan, & Clark, 1997), is the source of many human errors; in complex and dynamic situations, in particular, the high volume of information to monitor can overload the operator leaving critical incidents unnoticed and in turn reduce decision-making quality (Durlach, 2004; Varakin, Levin, & Fidler, 2004). Graphical user interface add-ons and tools, such as the Change History EXplicit (CHEX; see Smallman & St. John, 2003), have been designed to support change detection, situation awareness and decision-making in such demanding situations. In the current paper, we test whether the efficacy of CHEX, as designed by Smallman and St. John (2003), can generalize to a dynamic and multitasking context, aiming to extend its use beyond the change-detection-only task environment in which it was originally developed. Using a cognitive and holistic approach will enable us to assess the impact of this type of tool on all the aspects of the wider task environment (including change detection performance) and to further investigate the sources of attentional failures in dynamic situations.
The ability to detect a change depends on attentional and memory processes and the efficiency of the visual system to capture visual transients associated with the transition between the pre-change and the post-change states of an object. When the visual system fails to localize such transients, CB is likely to occur (e.g. Simons, Franconeri, & Reimer, 2000). In static visual scenes, techniques such as the flicker, the mud-splash, and the saccade-contingent paradigms have been used to prevent the visual system from capturing these transient signals and thus induce CB (e.g. Henderson & Hollingworth, 2003; O’Regan, Rensink, & Clark, 1999; Rensink et al., 1997; Simons & Ambinder, 2005). CB has also been studied using dynamic visual scenes in more realistic contexts, such as simulated command and control (C2), security surveillance environments, and monitoring geospatial displays (e.g. DiVita, Obermayer, Nugent, & Linville, 2004; Durlach & Chen, 2003; Durlach, Kring, & Bowens, 2008; Stelzer & Wickens, 2006). In complex and dynamic situations, multiple independent objects in the scene are changing frequently, unpredictably, and sometimes simultaneously, making important visual changes difficult to process and detect (Durlach et al., 2008). For example, in one simulated environment, participants were asked to monitor the geospatial display of an airspace and to detect and identify significant changes (e.g. new aircraft appearing on the display). They missed about 13% of all critical changes (St. John, Smallman, Manes, Feher, & Morrison, 2005). In such dynamic visual scenes, transient signals are naturally less detectable than in static scenes given that numerous features in the scene are moving (hence constantly changing) and can act as distractors (Boot, Kramer, Becic, Wiegmann, & Kubose, 2006). Competition between task-irrelevant transients made by distractors and those produced by goal-relevant changes reduce the detectability of critical changes (Vachon, Vallières, Jones, & Tremblay, 2012).
Given that change detection relies on a comparison between the operator’s perception of the current situation and their representation of the past situation, support tools to supplement the operator’s limited memory and attentional capacity should improve the ability to notice these changes. One such tool is CHEX (Smallman & St. John, 2003), a tool developed within the context of an air-warfare task to aid participants in discriminating significant changes from irrelevant changes that occur in an airspace. CHEX automatically detects and logs all changes in an interactive table positioned next to the radar display, thus providing an external aid and a “second chance” to notice an important change that may have previously gone unnoticed. Each row of the table displays the time the change occurred, the identity of the aircraft concerned, and a brief description of the change, while each column can be filtered by the user as needed. Selecting a row highlights that row, circles the location of the related aircraft on the geospatial display, and highlights other changes in the table made by the same aircraft, and vice versa (dynamic visual linking; see St. John & Smallman, 2008). The CHEX permanent repository is an uninterrupted display of situational changes intended to help operators in recovering from situation awareness breakdowns caused by interruptions and multitasking. Situation awareness is the perception of events, the comprehension of their meaning, and the anticipation of their status with respect to time and space (Endsley, 1995). The CHEX display has been applied to complex dynamic situations known to require multitasking and high attentional load (e.g. unmanned vehicle control; St. John & Smallman, 2008; Parasuraman, Cosenzo, & De Visser, 2009; Vachon et al., 2012).
St. John et al. (2005) compared CHEX with four other tools designed principally to attract attention towards significant events in the dynamic visual scene: a static table that chronologically listed the time and nature of each significant change; two types of instant replay; and explicit markers with “pop” sounds to accompany changes. The participants’ only task was to monitor the airspace and to explicitly declare (i.e. detect and identify) all relevant changes as quickly as possible. Results demonstrated that participants using CHEX were faster and more accurate to detect significant changes and made fewer omissions or errors than when using any other tool. The authors concluded that CHEX was superior to other tools as it supported both the detection and the identification components of the participants’ task. Indeed, one utility of this type of CHEX over the other types of tools is that, in addition to signaling or showing what were the changes in the situation, it gives information about what has changed. For example, explicit markers only aided the detection component and participants had to rely on their memory to infer what had happened based upon contextual cues, whereas changes were unobtrusively notified and described in the CHEX table. Indeed, the table format of CHEX appears to provide a good trade-off between information accessibility and distraction by presenting all changes textually in a permanent log, but by not presenting additional change-related information on the already cluttered geo-spatial plot. CHEX thus seems a very useful support tool for detecting changes within a dynamically changing visual scene and could potentially be used to reduce CB in complex C2 environments.
The current paper aims to extend the use of CHEX beyond the change-detection-only task environment in which it was originally developed (see Smallman & St. John, 2003; St. John et al., 2005), to a more ecologically valid context that mimics the multitasking nature of real C2 operations (see Vachon et al., 2012). The objective is to test the general efficacy of the CHEX situation awareness tool to work settings that are the most likely to require such support systems. In this study, the CHEX serves as a change-detection external aid. Although change detection is a central part of many C2 tasks, it is generally only one of several concurrent subtasks. As such, limited cognitive resources needed for noticing these dynamic changes are even more stretched in these circumstances, as various other aspects of the work are also attention demanding (Parasuraman et al., 2009). Another difference between lab studies of CB, as well as the studies of St. John, et al. (2005; Smallman & St. John, 2003), is that C2 operators rarely need to declare changes explicitly on their displays for the purpose of their mission; change detection is assumed when, after a task-relevant change, an appropriate action is taken (Vachon et al., 2012). Little research has been conducted regarding the extent and prevention of CB in multitasking C2 environments whereby change detection is embedded within the higher-order task of the operator (DiVita et al., 2004; Liebhaber & Feher, 2002). In the present study, we use a simulated naval air-warfare task, whereby participants play the role of a radar operator who must detect (and act upon) critical situational changes while also performing a threat-evaluation and weapon-assignment (TEWA) task. Change detection is thus an important subcomponent of the mission’s goals, but it is embedded within a broader dynamic decision-making task. CHEX has proven useful in a number of studies in which change detection was the only task to perform, but for it to be considered a beneficial change detection tool for C2, it must first be validated within the kind of high tempo, multitasking environment that would likely require such a support system.
In our assessment of the impact of decision support on change detection, we seek to discriminate between two different sources of CB: changes missed because they were never attended and changes that were processed to a certain extent but go unnoticed because of a failure of attentional processes (Vachon et al., 2012). More precisely, the pre-attentive source of CB refers to the idea that unattended changes—i.e. those that are never looked at—are less likely to be consciously perceived, hence more likely to remain undetected; i.e. the change was not perceived (e.g. Rensink et al., 1997; Simons & Ambinder, 2005). The attentional source of CB arises when a change is attended at a time when attentional processes are overloaded with many other sources of information to process, leaving insufficient resources for the change to reach consciousness. This failure to detect corresponds to looking at the change without seeing it (e.g. Caplovitz, Fendrich, & Hughes, 2008; Drew, Võ, & Wolfe, 2013; O’Regan, Deubel, Clark, & Rensink, 2000), although in the absence of conscious awareness, this change may be registered implicitly (Beck, Peterson, & Angeline, 2007; Fernandez-Duque & Thornton, 2003). Vachon et al. (2012) provided psychophysiological evidence that fixated changes that remained undetected were nevertheless processed up to a certain level, probably reflecting a (unanswered) call for attention generated by the automatic detection of attended visual transients. Here we broaden the definition of the attentional source of CB to include the memory processes, comprehension (sense-making), and decision-making necessary for change detection. Attention serves as a supervisory control for different cognitive functions (including those involved in change detection; see Baddeley, 1993), by overseeing comparisons between the pre-change and post-change representations stored in memory, the comprehension of the change (awareness of the evolution of the situation), and the decision-making needed to act upon the change (see Lamme, 2003). A change that is not consciously detected may be due to a breakdown at any of these stages, i.e. a lack of attentional resources to either (and among others) enable the encoding of fixated elements, make comparisons in memory, or understand the change that occurred in the situation (Caplovitz et al., 2008).
Considering the distinction between the two sources of CB, we suggest that CHEX has the potential to be a double-edged sword. On the one hand, if participants miss a change because they were attending elsewhere and did not fixate on it (the pre-attentive source of CB), CHEX provides another means to detect that change as information relating to it is displayed in a permanent and dynamically linked table. On the other hand, providing an additional source of visual information to monitor could increase cognitive load detrimentally, particularly if participants’ attentional resources are already close to capacity given the nature of the tasks. As such, this could increase the risk of attentional breakdown and exacerbate the second source of CB.
The present study
The task used was the Simulated Combat Control System microworld (S-CCS; Lafond et al., 2009; Vachon et al., 2012), a simulation of the essential subset of cognitive activities performed by tactical coordinators aboard Canadian navy frigates. The use of a microworld represents the best compromise between internal and ecological validities; it provides a high degree of experimental control while allowing the re-creation of displays and scenarios that are functionally comparable to those encountered in real C2 operations (Brehmer & Dörner, 1993; Elliot et al., 2004). For Experiment 1, participants were required to perform TEWA processes while also being alert and responding to unexpected critical changes (i.e. an aircraft changing from a non-threatening to a threatening status) which could potentially compromise the safety of the own ship. The TEWA task involved categorizing aircraft appearing on a radar display according to their threat level (threat evaluation) and scheduling retaliatory actions against threatening aircraft (weapon assignment). Participants were required to detect the critical changes as part of the higher-order goal of ensuring that aircraft were categorized appropriately and defending the own ship.
In Experiment 1, change detection was embedded within the TEWA task; participants were not required to explicitly report the changes occurring, but detection of these changes would enable performance of a timely action to protect the own ship. Smallman and St. John’s (2003) CHEX tool was reproduced and integrated within the S-CCS interface and evaluated according to a holistic approach: the efficiency of this decision support system (DSS) included not only its effects on the function it was designed to support—here, change detection—but also on other cognitive functions such as categorization and scheduling (see Lafond, Vachon, Rousseau, & Tremblay, 2010; Vachon, Lafond, Vallieres, Rousseau, & Tremblay, 2011). In this manner, the efficiency of CHEX was determined by its level of CB prevention, as well as by its impact—positive or negative—on threat evaluation performance, defensive effectiveness, and perceived workload. Experiment 2 again used the S-CCS microworld, but this time participants performed an explicit change detection task (akin to that of Smallman & St. John, 2003; see also St. John et al., 2005b), rather than the TEWA task in which change detection is implicitly required. The same version of the CHEX used in Experiment 1 was implemented within the S-CCS interface for Experiment 2. This second experiment served to replicate the conditions in which the CHEX had previously proven effective, but within the same air-warfare simulation used in Experiment 1.
Along with determining the efficacy of CHEX to improve overall change-detection performance, we measured the contribution of the DSS to prevent or reduce the two sources of CB by further subdividing the percentage of undetected changes according to whether they were fixated or not (see Vachon et al., 2012). Non-fixated (i.e. unattended) undetected changes are indicative of the pre-attentive CB—the failure to direct attention towards the change—whereas fixated (i.e. attended) undetected changes would reveal the occurrence of the second source of CB—insufficient amount of attentional resources to consciously detect a change.
Finally, we assessed participants’ subjective level of mental workload according to CHEX and control conditions using the NASA-TLX questionnaire (Hart & Staveland, 1988), a multidimensional rating procedure that produces an overall workload score based on a weighted average of ratings on six subscales. For the purpose of the present study, we focused exclusively on the mental demands and temporal demands subscales, examining the perceived mental load and time pressure in relation to change detection and TEWA performance.
Experiment 1 tested the efficacy of the CHEX tool, as it was designed by Smallman and St. John (2003), when the task of change detection was embedded within the broader context of a TEWA task. Based on the method of Vachon et al. (2012), we created a between-subjects experimental design in which we compared the results on change detection, subjective mental workload, and TEWA performance of two groups of participants: a group who performed the task with CHEX and another group with no DSS.
Fifty-one students at Université Laval (26 men; mean age, 23.04 years), were randomly assigned to either the CHEX (25 participants) or control condition (26 participants). All reported normal or corrected-to-normal vision and received $20 compensation for their participation in a single 2-h experimental session. This study was approved by the Université Laval ethics committee and consent to publish has been obtained from participants.
The S-CCS simulation (Lafond et al., 2009; Vachon et al., 2012) is dynamic insofar as it evolves according to different scenarios in interaction with the participants’ actions. Scenarios involve multiple aircraft moving in the vicinity of the ship and some of them may represent a potential threat to the frigate. As a tactical coordinator, participants must perform categorization and prioritization of threats, plan and schedule the application of combat power through retaliatory missile firing, and be alert to any changes made by aircraft in the operational space.
The visual display of S-CCS in the control condition can be divided into three sections (see Fig. 1): the radar screen (Area 1), the parameter list (Area 2), and the action buttons (Area 3). The interface is 1024 pixels wide by 768 pixels high which corresponds to the computer monitor resolution. The radar screen visually represents aircraft that move in real time in the ship’s immediate environment with various speeds and directions. Each unselected aircraft is represented as a white dot surrounded by a green square. Participants left-click with the mouse on its icon to select an aircraft. As an aircraft selection visual feedback, the surrounding square turns red to indicate that aircraft is now selected. The parameters list displays information related to the selected aircraft. The action buttons allow participants to assign threat level and immediacy and to neutralize a target aircraft.
The participants’ task included four subtasks to be performed concurrently throughout the entire scenario: (1) threat-level assessment, (2) threat-immediacy assessment, (3) neutralization of hostile aircraft, and (4) critical change detection. First, participants were asked to classify all aircraft as they progressively appeared on the radar screen according to three threat levels (non-hostile, uncertain, and hostile). To do so, they had to follow a pre-set classification rule by taking into account five equally weighted parameters displayed in the list: Origin, Altitude, Identification Friend-or-Foe, Military Electronic Emissions, and Detection of Weapons. As each of these parameters could take either a threatening or a non-threatening value, the number of threatening cues established the threat level of an aircraft. An aircraft was (1) non-hostile when 0–1 of his critical parameters were threatening, (2) uncertain when it showed 2–3 threatening cues, or (3) hostile when 4–5 of its critical parameters were threatening. Participants clicked on the corresponding classification button to register their decision, changing the aircraft’s white dot to one of three colors: red for hostile, yellow for uncertain, or green for non-hostile. The threat level of an aircraft could change at any time during a scenario and so participants were instructed to regularly recheck the parameters of already-classified aircraft in case their threat level needed reassessment. The color of the dot representing it only changed in accordance with the classification assigned by the participant and thus provided no cue regarding the occurrence of critical changes.
Second, for all hostile aircraft, participants were instructed to determine the level of threat immediacy based on their temporal proximity from the ship. Participants added the parameter values for the time to closest point of approach and the closest point of approach in units of time to prioritize their actions. Threat immediacy could be high (<15 s from hitting the ship), medium (15–30 s), or low (>30 s). As they made their decision, participants had to click on the corresponding immediacy button (1 to 3, respectively).
Third, because all hostile aircraft were programmed to hit the ship, participants were instructed to defend the ship against hostile tracks by launching an anti-aircraft missile. In real life, of course, there would be many safeguards and intermediate steps before an aircraft could be engaged; however, this was substantially simplified for the purpose of our experiment and required participants only to select the aircraft they wanted to neutralize and then click on the “Engage” button to launch the missile. The weapon was represented by a small white dot on the radar screen. Only one anti-aircraft missile projectile could be airborne at one time.
Finally, participants needed to be sensitive to changes occurring in the operational space in addition to performing the other three subtasks. Each aircraft, when appearing on the radar screen, had a status that could be either non-hostile or uncertain. Over the course of a scenario, some aircraft could change their status from non-hostile or uncertain to become hostile if they displayed 4–5 threatening cues. Such changes were considered critical and required detection in order to avoid the hostile aircraft attacking the frigate. All critical changes were accompanied by a visual transient on the radar screen that is a change in the aircraft speed and/or direction. Selection could be made using the radar, or CHEX if available. If no actions were made on a hostile aircraft within 15 s (the minimal time delay between two critical changes), the change was considered undetected. Since the occurrence of a ship hit would not necessarily indicate that a critical change had been missed (other factors may contribute to ship hits), this was not used as a criterion for change detection. There were also non-critical changes that could be either: (1) a non-hostile or an uncertain aircraft changing speed and/or direction; (2) a non-hostile aircraft becoming uncertain; and (3) a non-hostile aircraft becoming an uncertain and changing speed and/or direction.
In the CHEX condition, the tool was positioned to the right of the radar screen (Fig. 2, Area 4). As in Smallman and St. John (2003), the role of CHEX is to automatically detect and permanently store every change that occurs in the airspace. The apparition of change-related information in the table was not accompanied by an auditory alert. The table is separated into three sortable columns: time, ID, and changing parameter(s). Each time a change occurred (critical or not), CHEX logged all the modified parameters, one below the other, with the corresponding time, aircraft ID, and description. The next change was logged at the top of the table and previous changes moved towards the bottom of the table. Given the large number of changes accumulated in the table, CHEX had a scroll bar to allow participants to consult changes that had occurred at the beginning of a scenario and were therefore listed at the bottom of the table. The CHEX automation is fixed as its functionalities cannot be turned off by the operator and its table cannot be hidden or minimized on the display. As all changes are logged (critical and otherwise) and there is no further filtering of this information, the role of CHEX is to aid information acquisition rather than to provide any deeper analysis of the change. Participants can refer to CHEX at any time to determine whether a particular aircraft has undergone a certain type of change and when this change happened. A dynamic visual linking interconnects CHEX and the radar screen enabling the selection of an aircraft on either display, which highlights information relative to the selected aircraft in both displays.
Eye movements were recorded with a Tobii T1750 eye tracker at a sampling rate of 50 Hz. Participants were seated in front of a computer monitor at a distance of 60 cm. Infrared eye tracking cameras are integrated into this monitor allowing participants to freely move their head. Each fixation had to last at least 100 ms to be recorded by the eye tracker and the fixation field corresponds to a 50-pixel radius circle. The functional field of view of the participant thus corresponds to a visual angle of approximately 2°. As well as analyzing fixations in the AOIs illustrated in Figs. 1 and 2, we also assessed fixations on specific aircraft during the 15 s following a change; that is, a fixation on a critically changed aircraft during that time window would be classified as a fixated rather than a non-fixated change. A critical change was considered fixated if the aircraft which underwent the change (or the associated change entry in the tool table when available) fell within 50 pixels of the center point of a fixation at least once during the 15-s post-change period and if this fixation lasted at least 100 ms.
At the beginning of the experiment, participants were given a short tutorial to familiarize themselves with the context of the simulation and their tasks. This tutorial also informed participants assigned to the CHEX condition about the utility of the tool. After the instructions, they were required to perform the threat evaluation and immediacy judgments from nine static screenshots to confirm their understanding of the tasks. They then familiarized themselves with the microworld’s dynamic environment through two training sessions which comprised four 3-min scenarios. After calibrating the eye-tracking system, participants performed four randomly presented experimental blocks separated by 5-min rest periods. Each block comprised four 4-min scenarios of similar difficulty, presented randomly, for a total of 64 min of experimental testing. Each test scenario included a set of 27 aircraft (11 non-hostile, 8 uncertain, and 8 hostile) varying in speed and trajectory on the radar. A total of 33 changes occurred during a test scenario: 25 were non-critical and eight were critical. A maximum of 10 aircraft could appear on the radar screen at any one time. Overall, a participant had to detect 128 critical changes. After each scenario, participants answered two questions aloud regarding the mental and temporal demand indices from the NASA-TLX subjective workload questionnaire: (1) “How much mental and perceptual activity was required (e.g., thinking, deciding, calculating, looking, searching, etc.)? Was the task easy or demanding, simple or complex?” and (2) “How much time pressure did you feel due to the rate or pace at which the task occurred?” Participants indicated their perceived level of mental load and time pressure on a Likert-type scale of 1–10.
Results and discussion
CHEX and control groups were compared according to three aspects of operator activities: (1) change-detection performance; (2) subjective mental workload; (3) usage of CHEX; and (4) TEWA performance. The alpha level was set at .05.
Overall change detection
Fixated versus non-fixated undetected changes
Dwell time (mean + SD) on changing aircraft using the radar and/or the CHEX during 15-s post-change interval in Experiment 1
Using the radar (in ms)
Using the tool (in ms)
Measures of subjective workload were taken using the two subscales of the NASA-TLX. An independent-samples t test revealed that the CHEX condition was associated with a higher level of perceived mental load (M: 7.17, SD: 1.29) when compared to that obtained in the control condition (M: 6.29, SD: 1.30), t(49) = −2.405, p = 0.020, d = −0.697. However, there was no significant difference between the average level of time pressure reported by participants assigned to the CHEX condition (M: 6.83, SD: 1.69) and that perceived by participants in the control condition (M: 6.43, SD: 1.31), t(49) < 1. The CHEX had a detrimental effect on subjective workload as participants judged that their tasks were more demanding and complex and required more mental and perceptual activities in the presence of the tool than in its absence.
This suggests that participants placed in a multitasking dynamic situation may not have been able to make full sense of the change-related information contained in the CHEX table due to limited cognitive resources, making it less effective than in studies using a single-task environment. If, as suggested by Vachon et al. (2012), the extraction of visual information can be momentarily degraded by the high cognitive demands of the situation, it is likely that the extraction of information logged in CHEX suffers the same complications.
TEWA + CHEX
Change detection only + CHEX
Mean number of aircraft selections
On the tool
On the radar
Selections on the tool (%)
Eye movements on the tool
Overall dwell time (%)
Because data were averaged across scenarios, it is possible that the “overall” sporadic use of CHEX may be due to participants abandoning the tool during the course of the experiment. To test this possibility, we contrasted the three “usage” metrics across the four blocks of test scenarios using repeated-measures ANOVAs. A significant difference was observed across test blocks in terms of the percentage of selections made using CHEX, F(3, 72) = 2.897, p = 0.041, η2 p = 0.108, as selections seemed to increase between test 1 (M: 12.7%, SD: 12.27%) and test 2 (M: 14.2%, SD: 12.71%) and seemed to decrease and to remain the same in the last two tests (M: 11.93, SD: 1123%; M: 11.26%, SD: 11.75%, respectively). However, none of the multiple comparison tests performed were significant. The results were more clear cut for the other CHEX usage metrics: the percentage of fixations on the CHEX table did not change over time, F(3, 72) = 1.47, p = 0.230, η2 p = 0.058 (overall mean: 3.98%), nor the dwell time on the tool, F(3, 72) < 1 (overall mean: 23.19 s). These results indicate that CHEX was used (sporadically) throughout the entire experiment.
The increase in subjective workload in the CHEX condition would seem to provide some explanation for the associated decrease in TEWA performance in the presence of the tool. Such a result suggests that the presence of the DSS on the display provided an additional source of information to monitor that could have increased the apparent cognitive load of the task (Perry, Wiggins, Childs, & Fogarty, 2013). Indeed, introducing a table listing all the changes (critical and non-critical) occurring in the complex dynamic situation increased the amount of visual information to process before making a decision. It has also been shown that the mere presence of a tool or modifying a part of an interface can change the way a task is performed when compared to that observed beforehand (McCrickard, Catrambone, Chewar, & Stasko, 2003; Rousseau, Tremblay, Lafond, Vachon, & Breton, 2007). Therefore, this extra burden and pressure that the addition of CHEX put on participants may have affected their efficacy in making precise and timely decisions with regard to weapon assignment, thus leaving the own ship more exposed to attack (ship hits).
The finding that CHEX did not improve change detection in our task would seem to be at odds with the work of St. John et al. (2005) which demonstrated a clear reduction in CB with CHEX. Although both our study and that of St. John, Smallman, and Manes used dynamic, complex, and fairly realistic air-warfare microworlds, we suggest that the discrepancy may lie with the general task demands placed upon participants in each context; that is, whether detecting changes was the only task to perform or whether it formed part of a more complex, higher-order task. In Experiment 2, to test this hypothesis, we modified the S-CCS interface to remove elements associated with the TEWA task and participants’ only requirement was to identify critical changes that were associated with a classification change in the threat level of an aircraft. If CHEX improved change detection in this modified S-CCS context, thus replicating the results of St. John, Smallman, and Manes, it would suggest that the ability of CHEX to reduce CB is dependent upon the wider task requirements. The CHEX external aid was the same in Experiment 2 as that in Experiment 1, and participants used the information provided by the tool in the same manner in both experimental settings, for detecting and identifying critical changes.
Thirty-nine students from Université Laval took part: 20 were assigned to the control condition (10 men; mean age, 24.5 years) and the others were assigned to the CHEX condition (9 men; mean age, 26.3 years). None had taken part in Experiment 1.
In order to discourage participants from systematically clicking on the “Detection” button as soon as they saw a “hostile” status in the parameter list and thus to focus only on changing—newly hostile—aircraft, two hostile aircraft acting as false alarms were present on the radar screen at the beginning of each experimental scenario. The dot representing these hostile aircraft was white as for all other aircraft appearing on the radar. Participants were aware of these “non-changing” hostile aircraft and therefore knew that the presence of a hostile aircraft did not automatically imply a critical change to the situation. Participants were asked not to detect and report these non-changing hostiles and to focus exclusively on new threats. When participants noticed a change in direction and/or speed, they had to select the associated aircraft and verify the threat parameter to determine whether it constituted a critical change (i.e. a change from a non-threatening to a hostile status) or not (i.e. the aircraft was not hostile or had been hostile since the beginning of the scenario).
Results and discussion
Overall change detection
Fixated versus non-fixated undetected changes
Dwell time (mean + SD) on changing aircraft using the radar and/or the CHEX during 15-s post-change interval in Experiment 2
Using the radar (in ms)
Using the tool (in ms)
With CHEX support, participants reported significantly lower levels of mental load (M: 6.02, SD: 2.13), t(36) = 2.038, p = 0.049, d = 0.661, and time pressure (M: 5.07, SD: 1.75), t(36) = 2.079, p = 0.045, d = 0.675, than without such support (M: 7.21, SD 1.41 and M: 6.20, SD: 1.59, respectively). It thus appears that, contrary to Experiment 1, CHEX can help to reduce subjective workload but only when this is the only task to perform. In such cases, the operator has available attentional capacity to be able to use the tool to their advantage; with the knowledge that changes are automatically logged in a table, perceptions of pressure and mental load are decreased because participants know they can always refer to the permanent record of CHEX to supplement what they see on the radar.
As shown in Table 2, when change detection was the sole task, participants made significantly more aircraft selections using CHEX than when change detection was embedded within the TEWA task (Experiment 1), t(42) = −3.493, p = 0.001, d = −0.994. In Experiment 2, participants spent 11.5% of the experiment duration fixating CHEX, which is significantly higher than that in the CHEX condition of Experiment 1 (2.42%), t(42) = −3.563, p = 0.002, d = −1.149.
We performed repeated-measures ANOVAs on all “usage” metrics and found the percentage of selections using CHEX varied over time, F(3, 54) = 5.486, p = 0.022, ηp 2 = 0.234. The average percentage of selections using the CHEX was 41.59% (SD: 42.38%) for test 1, 46.62% (SD: 46.71%) for test 2, 47.29% (47.31%) for test 3, and 47.84% (SD: 47.88%) for test 4. Even though none of the multiple comparisons tests performed were significant, a trend analysis shows that the percentage of selections using CHEX increased linearly across the four test scenarios, F(1, 18) = 6.286, p = 0.022, ηp 2 = 0.259, suggesting that participants began to see the benefit in using the tool the more they became familiar with it. Dwell time on CHEX, F(3, 54) = 1.477, p = 0.231, ηp 2 = 0.076 (overall mean: 117.71 s), and percentage of fixations on the tool (F < 1; (overall mean: 27.35%) remained the same over time.
The objective of the present study was to test whether the efficacy of CHEX in supporting situation awareness maintenance could generalize to a context in which situational change detection served the higher goals of a primary task that comprised three other subtasks. However, far from supporting participants, results from Experiment 1 showed that CHEX increased the attentional source of CB, reduced defensive effectiveness, and led to higher levels of perceived mental workload. The success of a CHEX-like tool in augmenting change detection thus seems to depend very much upon the task in which it is used. Indeed, in line with the findings of St. John et al. (2005), CHEX was able to reduce CB within our naval air-warfare simulation when the detection of critical changes was the only task to perform (Experiment 2). In these circumstances CHEX can provide a useful log of changes to augment cognition and facilitate cross-referencing with the radar display.
We acknowledge that, with the introduction of a change-detection tool to the interface, the radar size varied across the two conditions of both experiments and may have impacted performance. However, even though in the CHEX condition the spacing between radar objects was slightly smaller than that observed in the control condition, the aircraft icon size remained the same and critical changes were always accompanied by the same visible change of speed and/or direction on the radar. These radar changes were obvious and unequivocal and aircraft were never occluded by another object. Therefore, we are confident that any impact of variations in radar size would have been negligible.
One recent model of attention behavior, NSEEV (noticing – salience, effort, expectancy, value; Steelman, McCarley, & Wickens, 2011; Wickens et al., 2009), can provide a useful theoretical basis for the current work on change detection. This model predicts scanning and noticing behaviors in a dynamic visual workspace by combining bottom-up factors of attentional control (e.g. the salience of a target) and top-down factors (e.g. the expectancy of an event occurring in a particular location, and the value or criticality of that information) and the effort needed to shift attention towards that item (eccentricity from current point of gaze). More recent work on the NSEEV emphasized on the interactions between salience, expectancy, and eccentricity in alert detection. Eccentricity includes factors pertaining to the target location within the visual scene, such as the acuity loss in the peripheral vision and the effort needed to direct attention towards the different regions of the scene, to explain increasing reaction times and error rates as a function of the distance between the target location and the gaze location of the participant. Steelman, McCarley, and Wickens (2013) showed that visual alerts appearing in low-probability locations not only were associated with poorer detection performance, but exacerbated the negative impacts of low-salience and eccentric targets. We consider how features of this model might explain noticing behavior in the current task and thus the extent to which CHEX hindered change detection rates under multitasking conditions.
CHEX and sources of CB
The pre-attentive source of CB occurs when the participant never looks at the change (failure to direct attention towards a change; e.g. Henderson & Hollingworth, 2003; O’Regan et al., 2000; Rensink et al., 1997; Simons & Ambinder, 2005). On a large display featuring numerous aircraft on the radar, as well as a list of associated parameters, shifting attention between all areas of the screen can be effortful and items or events that are further away from the point of gaze are more likely to be missed (one factor of the eccentricity parameter of NSEEV; Steelman et al., 2011, 2013). Despite having an additional area of interest on the display in the case of the CHEX condition—which could have increased the chance that participants were looking in the wrong place at the wrong time—fixations on the tool were minimal and there appeared to be no difference between conditions in terms of changes missed because attention was directed elsewhere.
Directed attention is required for change detection, but it is not always sufficient (see Simons & Rensink, 2005; Beck et al., 2007); changes may also be missed due to a failure of attentional processes (Privitera, Renninger, Carney, Klein, & Aguilar, 2010; Vachon et al., 2012). According to this second source of CB, missing changes can result from a failure in selection the objects to encode in memory, making sense of the changes, or in making the appropriate action after the changes. Registering a change requires a mental comparison with that object’s previous state and in the current task, recollection and comparison of subtle changes in a dynamic rather than a static attribute may be particularly demanding (e.g. a change in speed of an already moving object). In the NSEEV model, the salience of the target is influenced by both static features (contrast of the item of interest and the environment; see Itti & Koch, 2000) and dynamic or motion features (moment-to-moment changes of static salience; see Loy, Xiang, & Gong, 2012; Yantis & Jonides, 1990). Indeed, the current task may be particularly vulnerable to CB in general, as there are no salient static or dynamic features that distinguish a critically changed aircraft from any other on the radar screen in any of the two conditions. This concurs with the higher dwell time on the changing objects during the post-change interval for unnoticed changes in the CHEX condition than in the control condition of Experiment 1. Participants devoted a greater amount of attentional resources in the attempt to extract, compare, and comprehend the change-related contextual cues displayed by the radar and the tool, but failed to consciously detect the attended changes.
The NSEEV model also suggests that the size of the functional field of view narrows under conditions of high workload, meaning that the current task—likely to require a broad attentional breadth in order to register the necessary dynamic changes, as well as perform the TEWA task consecutively—may be particularly vulnerable to increases in workload. The CHEX condition introduces a higher visual load, resulting in impaired detection of these dynamic changes. The CHEX table keeps a permanent list of every change (critical and non-critical) that has occurred as well as numerous attributes associated with it. This serves to increase the amount of information on the screen, but fails to improve the salience of the most important information as critical and non-critical changes are not distinguished from each other. Although use of the tool was sporadic, occasional monitoring of the changes list may have created a more demanding divided attention condition whereby participants intermittently shifted attention towards the cluttered table to check if important information might be gleaned from it (Rosenholtz, Li, Mansfield, & Jin, 2006). Moreover, the mere presence of CHEX on the screen may have increased workload through an overhead cognitive cost in deciding when—or indeed whether—to shift attention to the automated tool. This high workload combination (CHEX plus TEWA task) means that attentional focus is narrowed and reduces the probability that sufficient attention can be directed at a change in order to detect it (Perry et al., 2013).
In Experiment 2, the replica of the original CHEX then proved a useful support tool in reducing both types of CB. In a complex dynamic task situation, attention can often be directed towards other areas of the interface leading critical changes to sometimes go unnoticed as the more eccentric the changes occur from the participant’s point of gaze, the more the peripheral vision is degraded. CHEX permanent repository of changes can help to compensate for those momentary lapses of attention. When operator capacity allows, participants can then rely on CHEX to re-deploy attention to the information they need in order to report an initially undetected change. Importantly in this single-task rather than multitask environment, CHEX was also able to reduce the attentional source of CB relative to No-DSS: since the tool allowed for offloading—rather than detrimentally overloading—cognitive processes needed for extracting, comparing, and understanding the evolution of the situation, participants were better able to answer the calls for attention demanded by the critical changes. The high-load context of Experiment 1 likely narrowed attentional breadth (narrowing of functional field of view) making the detection of fixated dynamic changes more difficult thus compromising the ability to register, extract, and comprehend dynamic change information. In Experiment 2 however, CHEX actually reduced workload and so a broader attentional breadth may have better allowed for the processing of change-related information when a critically changed aircraft was fixated. Future work should apply the computational capacity of the NSEEV model to predict the detection rates in both with and without CHEX conditions of this study in order to confirm the influence of the bottom-up and top-down factors in the inefficacy of CHEX.
TEWA task performance
The mere presence of CHEX on the S-CCS interface was enough to disrupt TEWA performance. It is possible that while cognitive resources were devoted to carrying out unnecessary actions such as crosschecking the veracity of a change notified by CHEX, an insufficient amount of resources were allocated to the other TEWA subtasks which might have resulted in scheduling impairment (Dixon & Wickens, 2006). As shown by several studies, the simple modification of a part of an interface can change information seeking and eye movement behaviors, as well as decision-making processes, and therefore can modulate how a task is performed (McCrickard et al., 2003; Rousseau et al., 2007). In accordance with this finding, modifying the S-CCS interface by introducing CHEX could have been enough to change the way participants conducted TEWA, hence leading to poorer defensive effectiveness.
Sporadic use of CHEX
Because usage of CHEX did not vary over time, its sporadic use cannot be attributed to a lack of training with the tool—whereby one would have expected usage to increase with familiarity across the 88 min experiment—or to participants’ mistrust in CHEX automation (Parasuraman & Riley, 1997), whereby usage might have decreased given the number of non-critical or ‘false alarm’ changes logged in the table creating a sense of unreliability (a cry-wolf effect; Brenitz 1983; Wickens et al., 2009b). More likely is that given the high cognitive workload imposed by the concurrent TEWA subtasks (Experiment 1) and the fact that making sense of the table required quite an investment in time and resources, participants lacked the necessary cognitive resources to devote to the tool consistently over the whole course of the experiment (Wickens, 2002). There may have been a feeling that by devoting too much time to the table, they would miss key information displayed on other critical areas of the interface (e.g. radar) that was necessary to perform the TEWA activities. On the other hand, in a single-task context (Experiment 2), usage of CHEX was greater as more cognitive resources were available for processing and understanding critical change-information from the CHEX table.
The NSEEV model parameters could also shed light on the disuse of the tools, in the way that participants do not want to invest the effort (eccentricity) in shifting attention to the right-hand side of the screen, when the information they are using to perform the TEWA task can be found centrally or on the left-hand side (radar screen and parameters list). The NSEEV model discourages large attentional shifts by including a spatial filter that reduces the salience of peripheral stimuli and therefore mimics the inhibitory properties of effort and poor acuity of long visual saccades. While the radar and parameters lists are essential for the focal task of TEWA, CHEX is perhaps seen as an optional addition that is too effortful to process under these high-load conditions and so participants rarely shift their attention towards it. Moreover, participants might have privileged the radar screen when searching for a critical change as they were expecting to notice it more easily in this part of the interface rather than in the CHEX table. The combination of the high eccentricity and the low expectancy associated with the CHEX tool might explain the sporadic use of the CHEX. Indeed, in Experiment 2 when change detection was the only task to perform, there was less emphasis on the parameters list on the left-hand side of the screen, so participants could devote more time to the tool on the right-hand side without requiring long effortful attentional reallocation between the two. The longer dwell times over the CHEX table show that participants expected to find critical changes using the tool.
NSEEV also emphasizes the importance of value (or task-relevancy and priority) of the information displayed in the different portions of an interface in determining the likelihood of change detection and participants may consider certain areas of the screen to be more useful for change detection than others (Horrey, Wickens, & Consalus, 2006). It is possible that the value of the CHEX table was lower when it was used within the multitasking context as it was relevant and directly useful to only one subtask. Participants might have prioritized the use of the radar and the parameters list as they were relevant for all their subtasks. Their propensity to “neglect” CHEX may have also been amplified by their subjective tendency to rely on tools that maintain the high-fidelity spatio-temporal realism of the visual scene (naïve realism; Smallman & St. John, 2005), preventing them from considering the true utility of CHEX to help perform their task. Indeed, the transposition of the events occurring in the situation into verbal entries displayed in the cluttered CHEX table may have been less appealing or less valuable for change detection than presenting them over a geospatial display. Participants may have believed that any critical changes would be more easily apparent and detected on the radar that displays speed/direction information dynamically, than by sorting through a list of written information. In Experiment 2, given that the focus was solely on change detection and not on the performance of other subtasks, participants may have placed a higher level of value on the change detection tool provided for them specifically to facilitate the task and thus usage was higher. Participants’ judgment of where the most useful information could be gleaned from would have influenced their gaze behavior.
Solutions for preventing CB
The NSEEV model can guide us towards a number of ways to prevent CB. Increasing the salience of critical changes is a fundamental step (see Loy et al., 2012; Zhang, Yuan, Zheng, Sheng, & Liu, 2010): given that CHEX stores change-related information for all objects in a dynamic situation, the table can rapidly become cluttered (Rosenholtz et al., 2006). The design could perhaps benefit from the implementation of an algorithm that would prioritize key information (see St. John et al., 2005). Instead of the default chronological ordering, an algorithm could process visual objects in such a way that the most important changes would be displayed at the top and those less important would appear at the bottom of the table (see Yeh & Wickens, 2001) or the level of importance could be varied through color intensity (see St. John et al., 2005b) to further increase the static salience of critical changes. A prioritization algorithm could reduce the negative impact of this clutter by enabling more effective filtering of critical from non-critical information (see Carver & Turoff, 2007; Hegarty, 2011), decreasing the saliency of entries less likely to be critical (see Wickens, Ambinder, & Alexander, 2004) and thus increasing the value of the information held in the table. However, any form of automatic prioritization could also generate errors. Current technology is not able to achieve a perfect hit rate (e.g. all critical changes are detected and correctly ordered) and false positives could not be completely avoided (Thomas & Rantanen, 2006). Automated prioritization systems, optimal or not, are also prone to the well-known risks of automation such as operator mistrust and overreliance (see Parasuraman, Sheridan, & Wickens, 2008; Parasuraman & Riley, 1997).
NSEEV predicts that detection of an event is related to the eccentricity of the target location in the visual scene. One potential solution then would be to integrate the real-time change tracking functionality within the radar view rather than adding a separate tool on the display (see Loy et al., 2012; Zhang et al., 2010). This would reduce the need for shifting gaze across a wide area and for moving the focus of attention across very different display formats (see Athènes, Chatty, & Bustico, 2000). Given the inefficacy of an external aid in the current multitasking context, radar-integrated notification systems should be tested in future studies. Indeed, such systems may seem intuitively more appealing and valuable to operators as they believe they are more likely to notice changes when presented with a dynamic spatial display than a written list. Having this naïve realism would perhaps mean the operator assigning more value to the radar-integrated change-tracking tool than a list of changes and so events may be more easily detected. In air-traffic control, visual notifications to aircraft that require an immediate response from the controller can emerge from the radar display (e.g. Imbert et al., 2014). These notifications are visual alerts to single critical events whereas the CHEX tool stores and displays a list of changes to all objects that may require a response. Whether an alert-like notification or a change history table is best suited depends on the nature of the task, its tempo (frequency of non-critical and critical events), and also on the level of discriminability and reliability of the filtering algorithm-based system.
The current study suggests that decision aids for use in multitasking contexts must be designed to fit within the available workload capacity of the user and the timeliness of the situation so that they may truly augment cognition. Here we provide a demonstration that sometimes a system designed to help a specific users’ cognitive function can also backfire and have unintended consequences such as decision-making errors (Trafton & Ratwani, 2014). By providing a comprehensive cognitive assessment of the support system, we clearly show that changing the nature and the demands of a dynamic decision-making task made a tool—proven effective to prevent CB within a change-detection-only task environment—ineffective once used in a multitasking context. The efficacy of CHEX did not generalize to an environment that replicates the multitasking conditions and the attentional demands of real C2 operations often observed in air traffic control, surface warfare and security surveillance. In relation to the CB phenomenon, it is clear that its rate of occurrence is likely to be very high in dynamic real-life monitoring tasks and that cognitive support to prevent this problem still remains to be developed and validated.
Moreover, the current study provides further empirical evidence of the existence of two sources of CB as previously shown by Vachon et al. (2012). Therefore, the evaluation of a change-detection support system could involve a test of its efficacy to prevent both sources of CB by employing eye tracking. An efficient change-detection tool must not only be able to help an operator detect changes, but also identify and understand them, while limiting the cost in attentional resources needed to process the information it provides, especially in such contexts that operators’ cognitive resources are already overburdened.
Although participants in Experiment 2 had only to perform change detection, the percentage of undetected changes observed in the control condition was nevertheless much higher than in the control condition of Experiment 1. Such a counterintuitive finding may have arisen from an underestimation of the percentage of undetected changes in the previous experiment. Indeed, when change detection was embedded within the TEWA task, correct detections were indexed by the timely actions made on the changed aircraft. It is thus possible in such a context that some changes never perceived by the participants were nonetheless recorded as detected because participants selected within the 15-s post-change period the changed aircraft on the basis of the requirements of the other subtasks they had to perform (and not because they had truly noticed the change). Such an underestimation is impossible when change detection is the sole task to perform.
Analysis of variance
Area of interest
Command and control
Change History Explicit
Decision support system
NASA task load index
Noticing Salience Effort Expectancy Value
Simulated Combat Control System
Threat Evaluation and Weapon Assignment
We are thankful to Julie Champagne, Sergei Smolov, Thierry Moisan, and Laurence Dumont for their assistance in programming, data collection, and data analysis.
This work was supported by R&D partnership grants from the National Sciences and Engineering Research Council of Canada (NSERC) and PROMPT Quebec, with Defence R&D Canada – Valcartier, and Thales Canada, Defense and Security Division, awarded to Sébastien Tremblay. Also, Benoît R. Vallières receives scholarship supports from NSERC and the Fonds de Recherche du Québec – Nature et Technologies (FRQ-NT).
Availability of data and materials
All datasets on which the conclusions of the manuscript rely and additional supporting files were deposited in a publicly accessible Dropbox folder https://www.dropbox.com/sh/pfei0cpsg6y5am3/AAA5G2BXUL8sf_NOFEk75Jm_a?dl=0.
BR-V is the primary author as this study formed part of his Ph.D. thesis. He was central to the experimental design, data collection, data analysis, and interpretation of results. Dr. HMH contributed to the theoretical development and manuscript redaction. Prof. FV’s expertise on change blindness and theories of attention were also invaluable during of the manuscript redaction process. Prof. ST is the director of Co-DOT Laboratory and was the supervisor of Dr. BRV’s thesis. He oversaw the development of the microworld and the experimental design of the study and was also involved the manuscript redaction. All authors read and approved the final manuscript.
Benoît R. Vallières is a Postdoctoral fellow in applied cognitive psychology at Université Laval. He received his Ph.D. in cognitive psychology from Université Laval in 2015. He also completed a research internship at Advanced Distributed Learning Co-Laboratories in Orlando in 2013. His research interests include change blindness, decision support system, and cognitive training.
Helen M. Hodgetts is a Lecturer at the School of Applied Psychology, Cardiff Metropolitan University, UK. She received her Ph.D in cognitive psychology from Cardiff University in 2005 and was a postdoctoral researcher at Cardiff University (2005–2009; 2011–2012) and Université Laval (2009–2010).
François Vachon is an Associate Professor at the School of Psychology, Université Laval Québec, Canada. His main research interests include the basic and applied cognitive psychology of attention and human performance. He received his Ph.D. in cognitive psychology from Université Laval in 2007. He then completed postdoctoral fellowships in cognitive psychology at Cardiff University (2007), in cognitive neuroscience at Université de Montréal, Canada (2008–2009), and in applied cognitive psychology at Université Laval (2009–2011).
Sébastien Tremblay is a Professor at the School of Psychology, Université Laval and honorary Professor of Cardiff University (UK). He is also lead researcher at the Co-DOT lab and Director of the GR3C, a Group of Researchers interested in the study of Cognition in Command and Control. He has expertise in a wide range of cognitive human factors issues. He holds a Ph.D. in Psychology (1999, Cardiff University). His program of research focuses on issues of human behavior and adaptive technologies related to maximizing performance and wellbeing in a wide range of domains such as public safety, transportation, and entertainment.
The authors declare that they have no competing interests.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
- Athènes, S., Chatty, S., & Bustico, A. (2000). Proceeding of the 3rd Air Traffic Management Seminar. Naples, Italy: ATM Seminar. Human factors in ATC alarms and notifications design: An experimental evaluation. http://www.atmseminar.org/seminar.cfm?seminar_id=3
- Baddeley, A. D. (1993). Working memory or working attention? In A. Baddeley & L. Weiskrantz (Eds.), Attention: selection, awareness, and control: A tribute to Donald Broadbent (pp. 152–170). Oxford, UK: Clarendon.Google Scholar
- Beck, M. R., Peterson, M. S., & Angeline, B. L. (2007). The roles of encoding, retrieval, and awareness in change detection. Memory & Cognition, 35, 610–620. doi:10.3758/BF03193299.View ArticleGoogle Scholar
- Boot, W. R., Kramer, A. F., Becic, E., Wiegmann, D. A., & Kubose, T. (2006). Detecting transient changes in dynamic displays: The more you look, the less you see. Human Factors, 48, 759–773. doi:10.1518/001872006779166424.View ArticlePubMedGoogle Scholar
- Brehmer, B., & Dörner, D. (1993). Experiments with computer-simulated microworlds: Escaping both the narrow straits of the laboratory and the deep blue sea of the field study. Computers in Human Behavior, 9, 171–184. doi:10.1016/0747-5632(93)90005-D.View ArticleGoogle Scholar
- Brenitz, S. (1983). Cry-Wolf: The Psychology of False Alarms. Hillsdale: Lawrence. Erlbaum.
- Caplovitz, G. P., Fendrich, R., & Hughes, H. C. (2008). Failures to see: attentive blank stares revealed by change blindness. Consciousness and Cognition, 17, 877–886. doi:10.1016/j.concog.2007.08.006.View ArticlePubMedGoogle Scholar
- Carver, L., & Turoff, M. (2007). Human-computer interaction: The human and computer as a team in emergency management information systems. Communications of the Association for Computing Machinery, 50, 33–38. doi:10.1145/1226736.1226761.View ArticleGoogle Scholar
- DiVita, J., Obermayer, R., Nugent, W., & Linville, J. M. (2004). Verification of the change blindness phenomenon while managing critical events on a combat information display. Human Factors, 46, 205–218. doi:10.1518/hfes.18.104.22.168340.View ArticlePubMedGoogle Scholar
- Dixon, S. R., & Wickens, C. D. (2006). Automation reliability in unmanned aerial vehicle flight control: A reliance-compliance model of automation dependence in high workload. Human Factors, 48, 474–486. doi:10.1518/001872006778606822.View ArticlePubMedGoogle Scholar
- Drew, T., Võ, M. L.-H., & Wolfe, J. M. (2013). The invisible gorilla strikes again: Sustained inattentional blindness in expert observers. Psychological Science, 24, 1848–1853. doi:10.1177/0956797613479386.View ArticlePubMedPubMed CentralGoogle Scholar
- Durlach, P. J. (2004). Change blindness and its implications for complex monitoring and control systems design and operator training. Human-Computer Interaction, 19, 423–451. doi:10.1207/s15327051hci1904_10.View ArticleGoogle Scholar
- Durlach, P. J., & Chen, J. Y. C. (2003). Proceedings of the Interservice/Industry Training, Simulation, and Education Conference. Orlando, FL: IITSEC. Visual change detection in digital military displays.Google Scholar
- Durlach, P. J., Kring, J., & Bowens, L. D. (2008). Detection of icon appearance and disappearance on a digital situation awareness display. Military Psychology, 20, 81–94. doi:10.1080/08995600701869502.View ArticleGoogle Scholar
- Elliott, L. R., Dalrymple, M. A., Schiflett, S. G., & Miller, J. C. (2004). Scaling Scenarios. Development and application to C4ISR sustained operations research. In S. G. Schiflett, L. R. Elliott, E. Salas, & M. D. Coovert (Eds.), Scaled Worlds: Development, Validation, and Applications (pp. 119–134). Hampshire: Ashgate.
- Endsley, M. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32–64.View ArticleGoogle Scholar
- Fernandez-Duque, D., & Thornton, I. M. (2003). Explicit mechanisms do not account for implicit localization and identification of change: An empirical reply to Mitroff et al. (2002). Journal of Experimental Psychology: Human Perception and Performance, 29, 846–858. doi:10.1037/0096-1522.214.171.1246.PubMedGoogle Scholar
- Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (task load index). Results of empirical and theoretical research. Advances in Psychology, 52, 139–183. doi:10.1016/S0166-4115(08)62386–9.
- Hauland, G. (2003). Measuring team situation awareness by means of eye movement data. In D. Harris, V. Duffy, M. Smith, & C. Stephanidis (Eds.), Proceedings of the Human-Computer Interaction International Conference (pp. 230–234). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.Google Scholar
- Hegarty, M. (2011). The cognitive science of visual-spatial displays: Implications for design. Topics in Cognitive Science, 3, 446–474. doi:10.1111/j.1756-8765.2011.01150.x.View ArticlePubMedGoogle Scholar
- Henderson, J., & Hollingworth, A. (2003). Global transsaccadic change blindness during scene perception. Psychological Science, 14, 493–497. doi:10.1111/1467-9280.02459.View ArticlePubMedGoogle Scholar
- Horrey, W. J., Wickens, C. D., & Consalus, K. P. (2006). Modeling drivers’ visual attention allocation while interacting with in-vehicle technologies. Journal of Experimental Psychology: Applied, 12, 67–78. doi:10.1037/1076-898X.12.2.67.PubMedGoogle Scholar
- Horswill, M. S., & McKenna, F. P. (2004). Drivers’ hazard perception ability: Situation awareness on the road. In S. P. Banbury & S. Tremblay (Eds.), A cognitive approach to situation awareness: Theory and application (pp. 155–175). Aldershot, UK: Ashgate.Google Scholar
- Imbert, J.-P., Hodgetts, H. M., Parise, P. R., Vachon, F., Dehais, F., & Tremblay, S. (2014). Inattentional costs and failures in air traffic control notifications. Ergonomics, 57(12), 1817–1832.View ArticlePubMedGoogle Scholar
- Itti, L., & Koch, C. (2000). A saliency-based search mechanism for overt and covert shifts of visual attention. Visual Research, 40, 1489–1506. doi:10.1016/S0042-6989(99)00163-7.Google Scholar
- Lafond, D., Champagne, J., Hervet, G., Gagnon, J.-F., Tremblay, S., & Rousseau, R. (2009). Decision analysis using policy capturing and process tracing techniques in a simulated naval air-defense task. Proceedings of the 53th Human Factors and Ergonomics Society Annual Meeting (pp. 1220–1224). Santa Monica, CA: Human Factors and Ergonomics Society. doi:10.1177/154193120905301812.
- Lafond, D., Vachon, F., Rousseau, R., & Tremblay, S. (2010). A cognitive and holistic approach to developing metrics for decision support in command and control. In D. B. Kaber & G. Boy (Eds.), Advances in Cognitive Ergonomics (pp. 65–75). Danvers, MA: CRC Press.Google Scholar
- Lamme, V. A. F. (2003). Why visual attention and awareness are different. Trends in Cognitive Sciences, 7, 12–18.View ArticlePubMedGoogle Scholar
- Liebhaber, M. J., & Feher, B. A. (2002). Surface warfare threat assessment: Requirements definition. San Diego, CA: Pacific Science. http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA406337.Google Scholar
- Loy, C. C., Xiang, T., & Gong, S. (2012). Salient motion detection in crowded scenes. Proceedings of the 5th International Symposium on Communications Control and Signal (pp. 1–4). IEEE. doi:10.1109/ISCCSP.2012.6217836.
- Masson, M. E. J., & Loftus, G. R. (2003). Using confidence intervals for graphically based data interpretation. Canadian Journal of Experimental Psychology, 57, 203–220.View ArticlePubMedGoogle Scholar
- McCarley, J. S., & Kramer, A. F. (2008). Eye movements as a window on perception and cognition. In R. Parasuraman & M. Rizzo (Eds.), Neuroergonomics: The brain at work (pp. 95–112). New York, NY: Oxford University Press, Inc.Google Scholar
- McCrickard, D. S., Catrambone, R., Chewar, C. M., & Stasko, J. T. (2003). Establishing tradeoffs that leverage attention for utility: Empirically evaluating information display in notification systems. International Journal of Human-Computer Studies, 58, 547–582. doi:10.1016/S1071-5819(03)00022-3.View ArticleGoogle Scholar
- Morrison, J. G., Marshall, S. P., Kelly, R. T., & Moore, R. A. (1997). Eye tracking in tactical decision making environments : implications for decision support evaluation. Proceedings of the 3th International Command and Control Research and Technology Symposium. Fort McNair, DC
- O’Regan, J. K., Deubel, H., Clark, J. J., & Rensink, R. A. (2000). Picture changes during blinks: Looking without seeing and seeing without looking. Visual Cognition, 7, 191–211. doi:10.1080/135062800394766.View ArticleGoogle Scholar
- O’Regan, J. K., Rensink, R. A., & Clark, J. J. (1999). Change blindness as a result of “mudsplashes.”. Nature, 398, 34. doi:10.1038/17953.View ArticlePubMedGoogle Scholar
- Parasuraman, R., Cosenzo, K. A., & De Visser, E. (2009). Adaptive automation for human supervision of multiple uninhabited vehicles: effects on change detection, situation awareness, and mental workload. Military Psychology, 21, 270–297. doi:10.1080/08995600902768800.View ArticleGoogle Scholar
- Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39, 230–253. doi:10.1518/001872097778543886.View ArticleGoogle Scholar
- Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2008). Situation awareness, mental workload, and trust in automation: Viable, empirically supported cognitive engineering constructs. Journal of Cognitive Engineering and Decision Making, 2, 140–160. doi:10.1518/155534308X284417.View ArticleGoogle Scholar
- Perry, N. C., Wiggins, M. W., Childs, M., & Fogarty, G. (2013). The application of reduced-processing decision support systems to facilitate the acquisition of decision-making skills. Human Factors, 55, 535–544. doi:10.1177/0018720812467367.View ArticlePubMedGoogle Scholar
- Poole, A., & Ball, L. J. (2006). Eye tracking in human-computer interaction and usability research. In C. Ghaoui (Ed.), Encyclopedia of human computer interaction (pp. 211–219). Hershey, PA: Idea Group Reference.View ArticleGoogle Scholar
- Privitera, C. M., Renninger, L. W., Carney, T., Klein, S., & Aguilar, M. (2010). Pupil dilation during visual target detection. Journal of Vision, 10, 1–14. doi:10.1167/10.10.3.View ArticleGoogle Scholar
- Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology, 62, 1457–1506. doi:10.1080/17470210902816461.View ArticleGoogle Scholar
- Rensink, R. A., Regan, J. K. O., & Clark, J. J. (1997). To see or not to see: The need for attention to perceive changes in scenes. Psychological Science, 8, 1–6. http://www.cs.ubc.ca/~rensink/publications/download/PsychSci97-RR.pdf.View ArticleGoogle Scholar
- Rosenholtz, R., Li, Y., Mansfield, J., & Jin, Z. (2006). Feature congestion: A measure of display clutter. Journal of Vision, 6, 827. doi:10.1167/6.6.827.View ArticleGoogle Scholar
- Rousseau, R., Tremblay, S., Lafond, D., Vachon, F., & Breton, R. (2007). Assessing temporal support for dynamic decision making in C2. Proceedings of the 51th Annual Meeting of Human Factors and Ergonomics Society (pp. 1259–1262). Santa Monica, CA: Human Factors and Ergonomics Society. doi:10.1177/154193120705101844.
- Simons, D. J., & Ambinder, M. S. (2005). Theory and consequences. Psychological Science, 14, 44–48. doi:10.1111/j.0963-7214.2005.00332.x.Google Scholar
- Simons, D. J., Franconeri, S. L., & Reimer, R. L. (2000). Change blindness in the absence of visual disruption. Perception, 29, 1143–1154. http://viscog.psych.northwestern.edu/publications/Simons_etal_gradual.pdf.
- Simons, D. J., & Rensink, R. A. (2005). Change blindness: past, present, and future. Trends in Cognitive Sciences, 9, 16–20. doi:10.1016/j.tics.2004.11.006.
- Smallman, H. S., & St. John, M. (2005). Naive realism: Misplaced faith in realistic displays. Ergonomics in Design: The Quarterly of Human Factors Applications, 13, 6–13. doi:10.1177/106480460501300303.View ArticleGoogle Scholar
- Smallman, H. S., & St. John, M. (2003). CHEX (Change History EXplicit): New HCI concepts for change awareness. Proceedings of the 47th Annual Meeting of Human Factors and Ergonomics Society (pp. 528–532). Santa Monica, CA: Human Factors and Ergonomics Society
- St. John, M., Smallman, H. S., & Manes, D. I. (2005). Recovery from interruptions to a dynamic monitoring task: The beguiling utility of instant replay. Proceedings from the 49th Annual Meeting of Human Factors and Ergonomics Society (pp. 473–477). Santa Monica, CA: Human Factors and Ergonomics Society. doi:10.1518/001872005774860014
- St. John, M., & Smallman, H. S. (2008). Staying up to speed: Four design principles for maintaining and recovering situation awareness. Human Factors, 2, 118–139. doi:10.1518/155534308X284381.
- St. John, M., Smallman, H. S., Manes, D. I., Feher, B. A., & Morrison, J. G. (2005). Heuristic automation for decluttering tactical displays. Human Factors, 47, 509–525. doi:10.1518/001872005774860014.
- Steelman, K. S., McCarley, J. S., & Wickens, C. D. (2011). Modeling the control of attention in visual workspaces. Human Factors, 53, 142–153. doi:10.1177/0018720811404026.View ArticlePubMedGoogle Scholar
- Steelman, K. S., McCarley, J. S., & Wickens, C. D. (2013). Great expectations: Top-down attention modulates the costs of clutter and eccentricity. Journal of Experimental Psychology: Applied, 19, 403–419. doi:10.1037/a0034546.PubMedGoogle Scholar
- Stelzer, E. M., & Wickens, C. D. (2006). Pilots strategically compensate for display enlargements in surveillance and flight control tasks. Human Factors, 48, 166–181. doi:10.1518/001872006776412225.View ArticlePubMedGoogle Scholar
- Thomas, L. C., & Rantanen, E. M. (2006). Human factors issues in implementation of advanced aviation technologies: A case of false alerts and cockpit displays of traffic information. Theoretical Issues in Ergonomics Science, 7, 501–523. doi:10.1080/14639220500090083.View ArticleGoogle Scholar
- Trafton, G. J., & Ratwani, R. M. (2014). The law of unintended consequences: The case of external subgoal support. Proceedings of the 32rd SIGCHI Conference on Human Factors in Computing Systems (pp., 1767–1776). New York, NY: ACM Press
- Vachon, F., Lafond, D., Vallieres, B. R., Rousseau, R., & Tremblay, S. (2011). Supporting situation awareness: A tradeoff between benefits and overhead. Proceeding of the 1st IEEE International Conference on Cognitive Methods in Situation Awareness and Decision Support (pp. 282–289). Miami Beach, FL: IEEE. doi:10.1109/COGSIMA.2011.5753460
- Vachon, F., Vallières, B. R., Jones, D. M., & Tremblay, S. (2012). Nonexplicit change detection in complex dynamic settings: what eye movements reveal. Human Factors, 54, 996–1007. doi:10.1177/0018720812443066.View ArticlePubMedGoogle Scholar
- Varakin, D. A., Levin, D. T., & Fidler, R. (2004). Unseen and unaware: Implications of recent research on failures of visual awareness for human-computer interface design. Human-Computer Interaction, 19, 389–422. doi:10.1207/s15327051hci1904_9.View ArticleGoogle Scholar
- Wickens, C. D. (2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3, 158–177. doi:10.1080/14639220210123806.View ArticleGoogle Scholar
- Wickens, C. D., Ambinder, M. S., & Alexander, A. L. (2004). The role of highlighting in visual search through maps. Proceedings of the 48th Human Factors and Ergonomics Society Meeting (pp. 1895–1899). Santa Monica, CA: Human Factors and Ergonomics Society. doi:10.1177/154193120404801622
- Wickens, C. D., Hooey, B. L., Gore, B. F., Sebok, A., & Koenicke, C. S. (2009). Identifying black swans in NextGen: predicting human performance in off-nominal conditions. Human Factors, 51, 638–651.View ArticlePubMedGoogle Scholar
- Wickens, C. D., Rice, S., Keller, D., Hutchins, S., Hughes, J., & Clayton, K. (2009). False alerts in air traffic control conflict alerting system: Is there a “cry wolf” effect? Human Factors, 51, 446–462. doi:10.1177/0018720809344720.View ArticlePubMedGoogle Scholar
- Yantis, S., & Jonides, J. (1990). Abrupt visual onsets and selective attention: Voluntary versus automatic allocation. Journal of Experimental Psychology: Human Perception and Performance, 16, 121–134. doi:10.1037/0096-15126.96.36.199.PubMedGoogle Scholar
- Yeh, M. Y., & Wickens, C. D. (2001). Attentional filtering in the design of electronic map displays: a comparison of color coding, intensity coding, and decluttering techniques. Human Factors, 43, 543–562. doi:10.1518/001872001775870359.View ArticlePubMedGoogle Scholar
- Zhang, G., Yuan, Z., Zheng, N., Sheng, X., & Liu, T. (2010). Visual saliency based object trackin. In H. Zha, R.-I. Taniguchi, & S. Mayband (Eds.), Computer Vision (pp. 193–203). ACCV: Springer.Google Scholar