Change deafness for real spatialized environmental scenes

Our laboratory, in addition to basic applied research, works closely with the test and evaluation community to quantify the safety and effectiveness of communication devices and hearing protection systems and other forms of personal protective equipment. An important part of this interaction is supporting the development of test requirements and methodologies that enable systematic evaluations of systems under conditions that mirror those of the real world. However, the reality of tests and evaluations is that they are based on simple, well-understood behaviors that are limited in scope. This is generally advantageous because the goal of such tests is to produce reliable and valid results that are highly reproducible across a variety of laboratories. For example, typical measures include speech intelligibility with high noise, single-source sound localization, and auditory detection. These measures represent worst-case conditions for simple perceptual behaviors, but fail to account for the kind of challenges faced during perception under real world complexity and variability. The current study presents research that was inspired by our interactions with the test and evaluation community and the realization that there is a significant gap in the understanding of how human performance will be affected in real-world complex environments. The change deafness phenomenon provides a balance between the traditional psychophysical approach and real-world complexity while enabling systematic characterizations of performance. The research presented here can inform the development of future test and evaluation requirements and procedures.

An ‘auditory scene’ is broadly defined as an array of concurrent sound sources (Gygi & Shafiro, 2010). Auditory scenes can be as simple as a set of pure tones, one or more chords, or a complex array of environmental sounds, like that encountered on a busy city street corner, in a restaurant kitchen during the lunch rush, or in a stadium filled for a rock concert. Increasing the complexity of an auditory scene increases the perceptual and cognitive demands for processing and can lead to misperceptions. Such misperceptions can manifest as inaccurate or inaccessible perceptual representations (Darwin et al. 1972; Näätänen & Winkler, 1999), which can be influenced by top-down factors such as attention, attributions of relevance, and interactions between short- and long-term memory processes (e.g., Zimmermann et al., 2016; Kidd et al., 2008; Gregg & Samuel, 2008; Cowan, 2001). Informational factors, such as stimulus similarity and uncertainty, also contribute to inaccurate or inaccessible perceptual experiences (Dickerson & Gaston, 2014 for review).

An example of the pervasiveness of perceptual errors is evident in the ‘change deafness’ phenomenon, which describes the failure of listeners to notice changes when they are embedded within complex auditory scenes. Change deafness, like a similar finding in the visual literature, change blindness, demonstrates that, despite a subjective impression of coherence and completeness, perceptual experience is incomplete and can be inaccurate. Change deafness has been demonstrated across several auditory domains to include speech (Sinnett et al. 2006; Vitevitch, 2003), environmental sounds (Gregg & Samuel, 2008, 2009; Eramudugolla et al. 2005; Gregg & Snyder, 2012), and music (Agres & Krumhansl, 2008). Several authors have also demonstrated the difficulty of auditory change perception tasks using artificial (synthesized) scenes such as pure tones shaped into scenes via amplitude modulation and shaped noise arrays (e.g., Constantino et al., 2012). A review of the recent change deafness literature (Dickerson & Gaston, 2014) noted that changes are missed 20–50% of the time depending on various perceptual and cognitive factors. For example, similarity between the sound that is changed and the other sounds in the scene influences change perception accuracy, with changes that are acoustically and semantically dissimilar from background sounds producing fewer errors (Gregg & Samuel, 2009), as would be expected from a signal-noise ratio perspective. Gregg et al. (2014) demonstrated that familiarity is also an important factor in driving change perception errors by showing that temporally scrambled and unrecognizable sounds produced significantly more errors than unscrambled and recognizable sounds. The manner in which a change occurs also appears to influence performance. Constantino et al. (2012) found that listeners performed better when a new sound was added to a scene than when a sound was deleted. Finally, change deafness seems to be influenced by attention, namely cueing or directing attention to the spatial location of a changed sound source can reduce the frequency of change perception errors (Eramudugolla et al., 2005; Backer & Alain, 2012).

The reportedly positive effect of providing a cue to the location of a change suggests that spatial position and spatial separation may be useful for perceptually segmenting a scene, which in turn may reduce change perception errors. Studies addressing the relationship between spatial separation and change perception, however, are few, limited to virtual audio manipulations, and are generally not in agreement. Gregg and Samuel (2008) found no segregation advantage for spatially separated sources in a virtual array, whereas Eramudugolla et al. (2005) found that spatial separation resulted in significantly fewer change perception errors. More generally, spatial position can be a cue to successful perceptual segregation (e.g., Bregman, 1993; Yost, 1993, 1997) and a number of psychophysical studies have shown (auditory) spatial cues to provide beneficial effects for perceptual performance (e.g., Broadbent, 1954; Best et al., 2006; Jones & Litovsky, 2011), including auditory search (Eramudugolla et al., 2008). Specifically, spatial separation has been shown to provide a reduction or elimination of informational masking effects (e.g., Ihlefeld & Shinn-Cunningham, 2008; Kidd et al., 1994), a phenomenon that may share common perceptual mechanisms with patterns in reports of change deafness (Dickerson & Gaston, 2014). There is clearly reason to expect that spatial separation may reduce change perception errors by reducing perceptual ambiguity, but as was previously mentioned, the literature on this topic as it relates to change deafness is sparse and conflicting. The present study uses a physical multi-speaker array and compares scenes with spatially separated versus spatially co-located sounds to systematically evaluate the role of spatial separation in modulating change errors. The use of sounds presented over speakers in the free-field, rather than the use of a virtual spatial manipulation, is an important methodological change from previous studies, as it is often the case that virtual spatial arrays are more often lateralized than truly localized (Yost, 1993) and artifacts associated with headphone lateralization, or the use of a generic head-related transfer function could, in part, explain the mixed results of previous studies investigating the role of spatial cues in change deafness.

In addition to the spatial manipulation, we follow the path of others in this area by examining how the type of change influences perceptual errors. In the current study, we instantiate changes via source additions and source removals. Change deafness studies in the past have manifested changes via source removals (e.g., Eramudugolla et al., 2005), as in ‘token and type changes’ (Gregg and Samuel, 2009), where a source in the scene is replaced with a signal that is either semantically and acoustically dissimilar (token change) or only acoustically dissimilar (type change), a ‘switch’ in which a sound is replaced with a different sound (Gregg et al., 2014), or a position ‘swap’, where two sounds change spatial position (Backer & Alain, 2012). Only Constantino et al. (2012) appear to have looked at both the addition and the deletion of a source within a single study context. Constantino et al. (2012) found that the addition of a source was easier to detect because the new source ‘pops-out’ from the background, compared to a deletion, in which the information in each frequency band must be iteratively compared. The present study examines performance for both additions and removals, as there is some suggestion (from Constantino et al., 2012, and others) that fewer errors should occur for source additions, as these changes will be perceived as ‘onset events’ and may pop out. Onset events are likely to elicit an automatic allocation of attention (e.g., Samuel & Weiner, 2001), which is known to facilitate change perception in both vision (Miller, 1989) and audition (Sussman et al., 2003). Thus, the addition of a new sound to the scene should be especially salient, causing participants to make fewer errors in the addition than in the removal condition. Although source additions may be more salient events, all of the scenes in the present study commence with the same number of sounds. Thus, an addition will result in a scene that has two more sources than in the sound removal condition. If there is some limit on the number of stimuli that can be represented in memory, then there should be more errors in the source addition condition with a trend toward reductions in errors as the size of scene two decreases (from 5 or 4, to 3 in the Addition, No-change, and Removal conditions, respectively).

This idea that the scene size, and therefore memory load, plays a role in change perception errors has been examined previously. Gregg and Samuel (2008) found evidence for change deafness (high errors) despite generally accurate performance on a cued recall task. In vision, Mitroff et al. (2004) report a similar finding; however, their results are less clear. Mitroff et al. (2004) in fact reported that memory for pre- and post-change scenes was preserved even when participants reported no awareness of a change, but in follow-up experiments they found that the stored representations are fragile; simply reversing the question order from cued-recall first to cued-recall last lead to significant decrements in recall. To further explore the interaction between recall accuracy and change perception errors and assess memory for scene elements, the present study presents participants with a cued recall task following each change perception trial.

Finally, change deafness is usually characterized in terms of hits, or accuracy, in indicating that a change has occurred. Change deafness, the failure to notice a change that has occurred, would be most directly measured by looking at hits or misses (e.g., Gregg & Samuel, 2008). However, restricting analyses to accuracy is a potential limitation because hit rates can be substantially influenced by listener response biases. In Signal Detection Theory (SDT) (Macmillan & Creelman, 2005), changes in response bias correspond to changes in decision criteria that ultimately result in systematic changes in hit and false alarm rates. These systematic changes can be modeled in receiver operator characteristic space and show that, across changes in criteria, sensitivity remains the same. In the change blindness literature, the influence of response bias is recognized and thus analyses typically report performance based on SDT measures of sensitivity in addition to accuracy measures (e.g., Mitroff et al., 2004). Although SDT approaches have not been broadly applied in the change deafness literature, there are notable examples (e.g., Eramudugolla et al., 2005; Gregg & Samuel, 2008; McAnally et al., 2010; Puschmann et al., 2013a, 2013b) that report evidence of change deafness despite using a bias-free measure of sensitivity. Here, we report measures of accuracy and SDT measures to examine patterns of hits and false alarms as well as a bias-free measure of sensitivity (d’) using an AX (same-different) task. We refer to the phenomenon of ‘change deafness’, but will also use the term ‘change discrimination’ where appropriate to denote the experimental procedure underlying measurement of the phenomenon.

To summarize, the present study fills a gap in the emerging change deafness literature by manipulating several common factors thought to influence change perception performance. We address the mixed results over the role of spatial cues in change deafness by comparing performance for spatially distributed or spatially co-located scenes using real spatial sources over a loudspeaker array. We examine two common change implementation strategies to investigate both the possibility that pop out could occur for additions and the secondary goal of evaluating set size effects. Finally, we present accuracy and SDT analyses together to eliminate the possibility that change discrimination errors are not simply an artifact of listener bias.¹

Table 2 ² shows hits and false alarm rates, and overall d’ values averaged across participants calculated using a differencing model, which is appropriate for same/different comparisons (Macmillan & Creelman, 2005) for the Spatially Co-located and Spatially Separated conditions, as a function of change type. For change trials, hit rates varied widely across conditions (from 0.37 to 0.68), but on average were consistent with previous change deafness reports that used 1 s sound samples. For example, the average hit rate of 0.47 across conditions here is equivalent to the 0.47 hit rate reported by Gregg and Samuel (2008) in their Experiment 1. The average false alarm rate here was somewhat lower than Gregg and Samuel (2008) reported (0.04 vs. 0.10), which resulted in a somewhat higher estimate of average d’ in the present study than the previous study (2.80 vs. 2.22, respectively). Consistent with others (e.g., Puschmann et al., 2013a, 2013b; McAnally et al., 2010), average d’ values across conditions were fairly high, indicating that listeners were quite sensitive to changes in the sound scenes.¹

For the cued recall task, hit rates were high (0.65–0.77); however, false alarm rates were much greater than found in the change discrimination task (Table 3). Estimates of sensitivity during cued recall were calculated using a yes/no model (Macmillan & Creelman, 2005) and resulted in d’ values ranging from 1.02 to 1.55. These d’ values correspond to [p(c) _unb ] values ranging from 0.70 to 0.77 (where [p(c) _unb ] estimates performance of an optimal unbiased observer), which are very similar to the observed hit rates reported in previous studies (Gregg & Samuel, 2008).

Change discrimination results

An ANOVA with the within subjects factors of Trial Block Order (1st, 2nd, 3rd), Change Type (Addition, Removal), and Change Level (–3, 0, +3 dB), and the between subjects factor of Spatial Separation (Co-located and Spatially Separated), was performed on the calculated d’ values. Post hoc testing was performed using Tukey’s Honestly Significant Difference test. First, the main effect of change type was not significant (F < 1). The main effect of Block Order was only marginally significant (F (2, 48) = 3.05, P = 0.057, _p η ²  = 0.11). There was a small gradual decline in sensitivity across blocks; however, none of the post-hoc pairwise differences were significant (M _diffs  < 0.2, P > 0.05).

The main effect of Change Level (–3, 0, +3 dB) was significant with a large effect size (F(2, 48) = 107.05, P < .0001, _p η ²  = 0.82); errors systematically increased as relative level decreased, with each level significantly different from the lower one (M _diff  > 0.27, P < 0.05; Fig. 3). This random application of a level difference irrespective of change type was introduced to avoid participants basing their responses simply on possible perceived changes in loudness due to the addition or subtraction of a sound source. The observed monotonic relationship between change level and change discrimination performance was not surprising, given the differences in target-background saliency that resulted from the combination of change type and change level.

Including all three target-background levels, the main effect of the between-subjects manipulation, Spatial Separation, was not significant (F (1, 24) = 3.23, P = 0.085, _p η ²  = 0.12). However, there was a significant interaction and large effect size between Change Level and Spatial Separation (F (2, 48) = 112.14, P < 0.0001, _p η ²  = 0.82). Figure 3 clearly shows that the interaction is due to an essentially flat function of performance and relative target-background level in the Spatially Co-Located scenes condition, while change discrimination is a strong monotonic function of target-background level in the Spatially Separated scenes condition. Post-hoc tests revealed that, in the Spatially Separated scenes condition, the –3 dB relative target level produced the worst performance, which is somewhat expected given the low target-to-background signal-to-noise ratio. This condition (–3 dB, with spatial separation) was the primary contributor in washing out the effect of spatial separation in the overall ANOVA. By excluding the –3 dB level data in the Spatially Separated sources condition, a convincing pattern emerges; an advantage of spatial separation on change discrimination, with sensitivity in both the 0 and +3 dB levels significantly greater than any of the three relative target levels in the Spatially Co-located condition (M _diff  > 0.37, P < 0.05). In terms of accuracy (also excluding data from the –3 dB level), the smallest difference in hit rate between equivalent target background levels was more than 0.07 for the 0 dB level and more than 0.17 for the +3 dB level.

A secondary question was whether the relative spatial region of the sound scenes affected change discrimination. In localization, typically, spatial precision is best directly in front of the listener and worse at lateral positions (e.g., Middlebrooks & Green, 1991). To address this question, a follow-up repeated-measures ANOVA was performed on the data from the Spatially Separated sources, and the results showed that the effect of spatial region was not significant (F (2, 24) = 2.01, P = 0.15, _p η ²  = 0.15). The lack of significance for spatial region may be due to the spacing of the loudspeakers at intervals of 22.5°, which is much greater than localization precision for broadband sounds presented in isolation (Middlebrooks & Green, 1991).

Memory recall results

A repeated-measures ANOVA with the within subjects variables of Trial Block Order (1st, 2nd, 3rd), Change Type (Add, Remove, and None) and the between subjects variable of Spatial Separation (Co-Located and Spatially Separated) was performed on the calculated d’ values. The main effect of trial block order was not significant (F < 1) and neither was the main effect of Spatial Separation (F < 1). Change type, however, was significant (F (2, 48) = 10.79, P < 0.0001, η ²  = 0.31), with Remove trials sensitivity (d’ = 1.49, SE = 0.07) higher than that of No-change trials (d’ = 1.29, SE = 0.08) and Add trials (d’ = 1.13, SE = 0.06). The two-way interaction between change type and spatial condition was not significant (F (2, 48) = 2.58, P = 0.09, _p η ²  = 0.10) and neither were the interactions between block and change type, or block and spatial condition (F < 1). However, the three-way interaction between change type, block order, and spatial separation was significant (F (4, 96) = 3.01, P < 0.02, _p η ²  = 0.11).

These results, which are shown in Fig. 4, indicate that listener sensitivity systematically decreased as the number of sounds in the second scene increased. When a source was added in the change deafness task, the number of sounds increased from 4 to 5. For No-change trials, the number of sounds remained constant at 4, and in Remove trials, the number of sounds went from 4 to 3. This pattern was also reflected in corresponding hit rates with Remove trials producing the highest hit rates ([p(c)] = 0.76, SE = 0.01), followed by No-change trials ([p(c)] = 0.68, SE = 0.02) and Add trials ([p(c)] = 0.67, SE = 0.02). This result makes sense from the perspective that an increase in the number of items should increase memory demands and, more specifically, should affect the ability to recall all the sounds presented within the span of such a short (1 s) interval.

The present study integrates many of the previous change deafness manipulations and contributes meaningfully to a small but growing number of voices suggesting that change deafness is a distinct perceptual phenomenon represented by a particular pattern of performance, namely low hits and low false alarms. However, despite similar nomenclature, the temptation to draw links between change deafness and its visual counterpart (change blindness) should be avoided. There are methodological and physiological differences between vision and audition that would have to be ignored to make direct comparisons between the two effects tenable. For example, the AX paradigm and accuracy measures that are standard in the change deafness literature are essentially a single ‘flicker’ and thus are only roughly analogous to single flicker or ‘one-shot’ change discrimination paradigms in vision (e.g., Mitroff et al., 2004). In standard flicker paradigms, multiple flickers can occur, and the primary dependent measure is reaction time to identify the change. When the change is found, the dependent measure of accuracy is almost always asymptotic. In contrast, the majority of change deafness studies are based on only one ‘flicker’ and thus the dependent measure is restricted to accuracy following a restricted exposure window that limits the effective time to encode scene elements. The two methods are fundamentally different, and thus difficult to compare. One possible solution is to provide analogous situations by designing auditory flicker paradigms where the goal is instead to identify the auditory change, and the dependent measure is response time to identify the change (e.g., Hall et al., 2015).