Through the Google Glass: The impact of heads-up displays on visual attention

In five experiments, we break new empirical ground by characterizing dual-task impairments associated with secondary information presented on a heads-up display (HUD) (i.e., Google Glass) during a primary visual search task. By combining two classical cognitive psychology paradigms (visual search and recognition memory), our studies dissociate impairments to selective and preattentive mechanisms while quantifying the extent to which secondary information is processed in a context increasingly encountered in the real world (e.g., information projected on a windshield when driving). Our results indicate that secondary HUD-based information is ubiquitously disruptive to attentional mechanisms, independent of user-centric prioritization and the time course of secondary information.

Our lives are being continuously and increasingly intermingled with technology (e.g., smartphones, wearable HUDs). While creating informationally rich environments might lead to productivity benefits in some contexts and convenience in others, designers, scientists, and users need to understand how technological integration might also be harmful. We investigate this latter context in our present research, which contains a unique blend of theoretically relevant and practically applicable data that should be of interest to a wide audience, including psychologists, engineers, designers, policy makers, and the general public.

Mobile technology has become essential and pervasive in the everyday lives of many people. Understanding the extent to which increasingly integrated information systems, such as cell phones (Drews, Yazdani, Godfrey, Cooper, & Strayer, 2009; Strayer, Drews, & Johnston, 2003) and other user interfaces, impact human performance on a range of common tasks and cognitive processes is of critical importance. Specifically, how does the adoption of various technologies remove a user from the present moment or task at hand, and at what cost (Starner, 2002)? Mobile technologies, for instance, have progressed from cell phones to wearable interfaces, leaving users in constant contact with their devices, regardless of whether they explicitly choose to engage with that device.

It has been well established that engaging in multitasking induces costs to performance (Allport, 1980; Horrey & Wickens, 2006; Neider, McCarley, Crowell, Kaczmarski, & Kramer, 2010; Strayer et al., 2003). In the practical domain, much of this research is focused on cell phone engagement in the context of driving or walking (Horrey & Wickens, 2006; Kramer, Hahn, Irwin, & Theeuwes, 1999; Neider et al., 2010). For example, using a cell phone or text-to-speech interface while driving significantly increases cognitive load and crash risk (Drews et al., 2009; Strayer et al., 2013), and it impairs memory for visual information (Strayer et al., 2003). While a focus on cell phone-related distraction has made practical sense, given the approximately 7.1 billion mobile subscriptions internationally (International Telecommunication Union ITU, 2015), emergent technologies are moving toward a user-integrated approach favoring HUDs. HUDs have long been used in aviation cockpits and are now being employed in everyday environments, such as automobiles (e.g. Cadillac and Mercedes vehicles), or integrated directly with the user, such as with Google Glass (GG) and Oculus Rift (Ceurstemont, 2014). Unlike cell phones, HUDs typically present users with a persistent stream of visual information (though systems such as GG can provide auditory information as well), increasing the likelihood of interference with other concurrent visual tasks (Wickens, 2002, 2008). Although prior work in the multitasking domain is largely ubiquitous in demonstrating performance impairments under such conditions across a variety of contexts, novel reappropriations of existing technologies can carry with them some implicit expectation that they might immunize against such impairments. HUDs, which make use of transparent displays, have been used with great success in the aviation domain; however, the information-processing needs and priorities of a pilot at 30,000 feet are likely to be very different from those of a driver on the ground who might have only seconds to respond to a potential hazard. Consequently, as HUDs become increasingly used in less specialized contexts, it becomes imperative to understand how they might impact overall behavior when set against attentional limitations. To date, the literature relating HUD-based technology to attention and performance costs in everyday contexts has been minimal (Starner, 2002; Wolffsohn, McBrien, Edgar, & Stout, 1998).

Our goals in the present experiments were twofold. First, we wanted to characterize the extent to which visual information presented on a user-worn HUD (e.g., GG) impacts performance on a primary visual task, and how such effects might be modulated by the relevance and temporal presentation (i.e., onset prior to, concurrently, or following onset of primary task) of the HUD-based information. Second, we wanted to shed light on possible attentional mechanisms underlying performance costs arising from information presented on HUDs while engaged in a concurrent primary task (analogous to conversing on a cell phone while driving). To do so, we employed a visual search paradigm as our primary task, allowing us to isolate impairments to both parallel and serial attention mechanisms. Whereas efficient search for singleton targets is thought to involve parallel, preattentive processes (and less so selective attention), searches that are inefficient are thought to require serial attention processes that rely heavily on selective attention (Wolfe, 1998). Critically, if performance impairments occurred only during inefficient search, it would suggest that secondary task information presented on the GG is largely detrimental to selective attentional processes, perhaps those related to efficiently guiding attention toward the target. Alternatively, if secondary information presented on the GG induces performance costs during singleton search, it would suggest impairment to preattentive processes as well (though it would not rule out some impairment to selective attention mechanisms), and more generally to broader visual processing. An additional benefit of using a search task is that search is a vital operation for everyday function; humans must constantly locate task-relevant information (such as a pedestrian about to run into a roadway) in the environment. Thus, visual search is both a theoretically useful and practically relevant paradigm to assess HUD-based dual-task effects.

In all experiments, the participant’s primary task was to locate a T target among L distractors displayed on a computer screen. In some conditions, the secondary information, in the form of a single word, was concurrently presented on a GG that was worn during a portion of the experiment. In Experiment 1, we characterized primary task performance costs associated with the presentation of secondary information on the GG while also manipulating the perceived relevance of the secondary information (through instructions) to the participant. We predicted response time (RT) costs to the visual search task in the presence of a secondary information stream, as well as an added cost when participants were told the information was useful. The extent to which secondary task information was processed was assessed through a surprise recognition memory task administered after all search trials were completed. In Experiment 2, we manipulated the context of the secondary information presented on the GG by informing participants of the recognition memory task. We expected secondary information to be more disruptive to the primary task when participants were aware that they would be tested on it. In Experiment 3, we explored the degree to which variation in the time course of the onset of secondary information impacted primary task performance (prior to, concurrently, or following the primary task), and the extent to which this might interact with the perceived relevance of that information. We expected concurrent presentation to produce larger costs to primary task performance, with this cost increasing when the secondary task was perceived as more relevant. In Experiment 4, we manipulated the saliency of the target T to elicit singleton search behavior to evaluate whether performance costs are exclusive to selective attention mechanisms or exist for preattentative processes as well. In the final experiment, we masked the onset and offset of the secondary task information to guard against the possibility that our effects might be more closely related to some reflexive reorienting of attentional processes toward an abrupt stimulus onset, as opposed to informational processing impairments associated with managing dual-task demands.

Generally speaking, presenting information on the GG concurrently with the search task induced costs to RT performance. However, participants were unaware that they would be tested on the secondary information presented on the GG, which may have disproportionately biased them toward discounting that information. To address this possibility, in Experiment 2 we informed participants of the memory recognition test.

Our data derived from Experiments 1 and 2 clearly suggest that secondary information presented on a HUD elicits RT costs to concurrent tasks involving visual attention; however, the data are limited to cases where the task information is time-locked to onset concurrently. In the real world, information ebbs and flows. Distracting information often is received when an observer is already engaged in another task (e.g., text messages received while driving a vehicle). As such, in Experiment 3, we manipulated the timing of the onset of the secondary information.

Results

We removed seven participants from the analyses because of accuracy or RT values more than 2 SD from the mean. We found no differences for accuracy across conditions (see Table 1). Because there was no secondary information to manipulate in the control conditions, we conducted two separate analyses of variance: one to evaluate whether there was a cost of secondary information to the primary visual search task compared with the control conditions that collapsed across the timing manipulation (including set size and GG condition), and a second that omitted the control and set size conditions but included the timing manipulation (dual-task GG conditions and temporal onset). For the former, the overall RT data were similar to those derived from Experiments 1 and 2: RT costs were observed when secondary information was presented on the GG (F[3, 61] = 3.95, p = .013, η² = .160) (see Fig. 2c). Furthermore, planned contrasts indicated that all GG conditions were different from the control conditions (p _s < .05), but not from each other (p = .486). We also found an effect of set size (F[1, 61] = 71.46, p < .001, η² = .539), but no interaction between set size and GG condition (F[3, 61] = 0.89, p = .450, η² = .042). RT × set size function slopes averaged 23.25 milliseconds per item. The latter analysis revealed no effect of temporal onset of the secondary information (F[4, 120] = 1.24, p = .259, η² = .043) or any interaction of secondary information temporal onset with GG condition (F[4, 120] = 2.38, p = .055, η² = .073) (see Fig. 4). It is perhaps worth noting that visual inspection of Fig. 4 suggests that when the onset of both the primary search task and secondary GG information was concurrent, information that was communicated as irrelevant did not induce a performance cost. This insignificant trend is an outlier to all of our data thus far, that such secondary information produces robust interference with the primary task, and may be related to some change in strategy with regard to attentional deployment associated with the varied-onset timing of the secondary information. Given that the pattern is insignificant and contrary to the first three experiments, as well as being represented by fewer trials than in the previous experiments owing to the additional factor of timing, any conjecture related to it must be made with caution.

Combined, these data suggest that, generally speaking, secondary information induced a cost to the primary visual search task regardless of when it appeared, and they underline how generally distracting HUD-based information may be during multitasking. Again, recognition memory performance was above chance (p _s < .05), regardless of dual-task condition (F[1, 30] = 0.04, p = .844, η² = .001).

In our previous experiments, we used a visual search paradigm where the target was difficult to discern from the distractors. In Experiment 4, we altered our primary search task by making the target object red, effectively creating a singleton search task. Importantly, singleton search relies on parallel preattentive mechanisms, as opposed to selective attention (Treisman & Gelade, 1980; Wolfe, 2010). Our goal in Experiment 4 was to evaluate whether the costs associated with irrelevant information presented on the GG are exclusive to tasks in which selective attention mechanisms are required.

In Experiments 1–4, the screen on the GG remained blank until a word was presented. As a result, word presentations on the GG could be characterized as abrupt onsets. A large body of literature has shown that such onsets are particularly effective at capturing attentional processes and might be reflexive in nature (Chua, 2013; Folk & Remington, 2015; Theeuwes, Kramer, Hahn, Irwin, & Zelinsky, 1999; Yantis & Jonides, 1984). Given these findings, it is possible that the dual-task costs observed up until this point may not be associated with some limitation in multitasking ability, but rather arose solely from the sudden onset of the secondary stimulus. To test this possibility, in Experiment 5 we presented a persistent visual mask on the GG that was replaced by a word at the onset of the primary visual search task. Finding a pattern of data consistent with Experiments 1–4 would support the assertion that dual-task performance costs associated with HUD-based information are best characterized within the context of basic attentional limitations.

Results

Three participants were removed from the analyses because of accuracy or RT values more than 2 SD from the mean. We found no differences for accuracy across conditions (see Table 1). Patterns of RT costs were also similar to those in our previous experiments (see Fig. 5a). There were significant main effects of GG condition (F[2, 51] = 4.89, p = .011, η² = .161) and set size (F[1, 51] = 75.23, p < .001, η² = .596), but no interaction between condition and set size (F[2, 51] = 1.37, p = .263, η² = .051). We again found that RTs in the dual-task conditions were longer than in the control condition (p < .05), regardless of the instructed relevance of the secondary information (p = .781). The search slope was 24.86 milliseconds per item. Performance in the recognition task remained above chance (p _s < .05) and did not differ across GG conditions (F[1, 34] = 0.05, p = .834, η² = .001) (see Fig. 5b).

Overall, our data show that there is a cost associated with wearable technology in dual-task contexts that approximate situations often encountered in the real world. What’s more, this effect is robust and, at the very least, difficult to mitigate. In Experiment 1, we found evidence of a dual-task cost when wearing the GG and that that cost was not offset by relevance instructions pertaining to the secondary information; costs persisted even when participants were instructed to ignore the secondary information. In Experiment 2, we found that when participants were informed that they would be tested on the secondary information, performance costs were even more robust. In Experiment 3, we showed that RT costs associated with the HUD-based secondary information were largely orthogonal to the temporal onset of that information in relation to the primary task; secondary information was nearly always disruptive to visual search, regardless of time of onset. Experiment 4 indicated that the costs of secondary HUD-based information are not only incurred to selective attention mechanisms, but are in fact present at early processing stages thought to be associated with preattentive mechanisms. Finally, in Experiment 5, we tested the possibility that the patterns of data observed in Experiments 1–4 may have been associated with the abrupt onset of the HUD-based information and found that the pattern persisted when the abrupt onsets were eliminated.

Our results provide robust evidence that primary task performance is impaired by secondary information presented on a wearable HUD and is relatively independent of task relevance. Although there was some evidence that participants weighed secondary information portrayed as relevant to the primary task in Experiment 2 more heavily than information portrayed as irrelevant, and in turn had larger overall performance costs, this finding was not replicated in all experiments. Generally speaking, information pertinence may not matter when set against broader distraction, as previous researchers have found that items relevant to safety were not recognized any more often than irrelevant items in either single- or dual-task scenarios (Strayer & Drews, 2007). That these costs exist in a simplified environment is particularly worrisome when speculating about how they might generalize to more realistic multitasking situations (Horrey & Wickens, 2006). Even under relatively simple task conditions, performance decrements were substantial, at ranges of 450–600 milliseconds compared with control conditions. Given the practicality and growing practice of implementing HUDs for a wider variety of users (beyond those in aviation), these costs should give researchers and practitioners pause (Crawford & Neal, 2006; Liu et al., 2009). It is not unreasonable to speculate that these costs might be more severe under increasingly complex, realistic task conditions (e.g., when driving) (Strayer et al., 2003). In simulated environments, GG has produced impairments similar to those present when using a cellular device; however, the performance decrements are less severe (He, Choi, McCarley, Chaparro, & Wang, 2015; Sawyer, Finomore, Calvo, & Hancock, 2014).

Importantly, in our studies, performance impairments were present regardless of whether the primary task depended on preattentive parallel processes or serial attention, suggesting that costs under real-world conditions are likely to occur across a broad array of tasks and conditions. Whereas previous findings have demonstrated impairments in perceptual memory under dual-task conditions (Strayer et al., 2003), our data suggest broad-spectrum impairments to attentional processes as well. Our findings are consistent with those derived from previous theoretical models suggesting that cross-task interference is likely to be high when competing information is presented within the same perceptual modality (Wickens, 2008). However, the cognitive mechanisms underlying the interference are often left unspecified. Strayer and Drews (2007) proposed that the underlying interference accompanying technology-based distraction is likely associated with inattentional blindness; secondary information impedes the encoding of primary task information. Our finding that secondary information can impede processes associated with both inefficient and efficient search suggests that dual-task performance impairments may actually arise quite early in the information-processing chain and impact the selection of which low-level information in the environment is passed on to higher-order processes for scrutiny. This explanation is not inconsistent with the proposal of Strayer and Drews. Rather, it provides some broader perspective on where their inattentional blindness findings may emerge from: broad impairments to the deployment of attentional processes. Still, it is worth noting that while we found evidence for impairments to both parallel and serial attentional mechanisms through our experimental manipulations, within our studies there was no interaction of GG condition with set size in the primary visual search task, which one might expect to observe in the presence of selective attention impairments (though this might also be reflective of some sort of decision-making process as opposed to selective attention alone). This might suggest that the locus of dual-task impairments, at least as it pertains to our particular set of tasks, is more complex than can be described by attentional impairments alone. In future work, researchers should continue to explore the phenomena at the mechanistic level.

Overall, performance costs in our studies occurred regardless of perceived importance of secondary information (participants were unable to ignore secondary information even when instructed to do so) and time course of information presentation. Combined, our data strongly suggest that caution should be exercised when deploying HUD-based informational displays in circumstances where the primary user task is visual in nature. Just because we can, does not mean we should.