The perceptual processing of fused multi-spectral imagery

Fusion assessment

Image fusion is mostly studied within the field of computer vision, hence the vast majority of the metrics of fusion quality are based on computational principles. One of the more common measures is of the preservation of edge information, at the individual pixel level (Xydeas & Petrović, 2000); the local, 8×8 pixel grid level (Piella & Heijmans, 2003); or the global image level (Petrović & Xydeas, 2004; Qu, Zhang, & Yan, 2002). These image-level metrics are valuable in that they provide an objective assessment of the amount and quality of information from each single sensor that is represented in the composite image for minimal cost. Two major deficits of limiting assessment to image quality metrics is that they do not account for task-relevant information and are not always predictive of human performance (Smeelen, Schwering, Toet, & Loog, 2014).

To address the shortcomings of computer-based image quality metrics, subjective user experience questionnaires (asking for example, overall reported image preference, comfort, etc.) are used (Krishnamoorthy & Soman, 2010; Petrović, 2007). This approach offers a partial solution, but subjective quality assessments can also fail to predict variation in performance. Furthermore, when they are used, user experience assessments are only used for outcome assessment and not to inform directly the design process (Toet et al., 2010). Hence, while the subjective quality of a display yields some benefits, to gain understanding of what design aspects lead to better decision-making and human performance and inform the design of new fusion approaches, it is important to measure directly human performance on a specific task (cf. Blum, 2006; Dixon et al., 2006; Dong, Zhuang, Huang, & Fu, 2009).

Despite being a relatively limited literature, human performance with fused imagery has been used with a range of basic visual tasks including detection (Krebs et al., 1999), discrimination (e.g., whether a global scene is upright or vertically inverted; Krebs & Sinai, 2002, Toet et al., 1997), recognition (Ryan & Tinkler, 1995; Sinai, McCarley, & Krebs, 1999; Toet & Franken, 2003), and visual search (Neriani, Pinkus, & Dommett, 2008). This research has been conducted in contexts including aviation (Ryan & Tinkler, 1995; Steele & Perconti, 1997) and surveillance (Neriani et al., 2008; Toet & Franken, 2003; Toet et al., 1997). Among these applications, there is a wide range of reported results and overall conclusions. Such discrepancies are potentially due to methodological variation (Ahumada & Krebs, 2000; Essock, Sinai, McCarley, Krebs, & DeFord, 1999; Steele & Perconti, 1997), differences in task descriptions (Krebs & Sinai, 2002; McCarley & Krebs, 2000), and variation in fusion algorithms or sensor combinations (McCarley & Krebs, 2000; Neriani et al., 2008). Additional manipulations often cited in the literature are task type and difficulty, image scene, sensors, and fusion algorithms (Krebs & Sinai, 2002; McCarley & Krebs, 2000). Thus far there is no standard way to compare across manipulations that controls for the amount and type of information provided by each component image.

In many of these studies, performance with composite images was compared to performance with an individual sensor (e.g., LWIR plus visible compared to visible alone). Unfortunately, this comparison confounds whether image fusion enhances performance because of the fusion method implemented or simply because it supplies more information to the observer. We are concerned with answering the question of whether the observer is processing each sensor image as efficiently in a multi-sensor context as when presented in isolation. To answer this question effectively, we must compare performance with multiple sensors to predict how well they should perform given their performance with each individual sensor image.

When an observer is provided with two sensor images, regardless of the display type, they have redundant information to inform them of the correct decision, thereby suggesting an overall faster response. Although it may seem intuitive to equate a performance gain with redundant signals with facilitatory processing, parallel processes with no facilitation can predict significant redundancy gains (Duncan, 1980; Kahneman, 1973; Miller, 1982; Raab, 1962; Townsend & Wenger, 2004). Furthermore, performance decrements may still be observed relative to single-source imagery due to our perceptual system dealing with multiple pieces of information (cf. Townsend & Ashby, 1983; Townsend & Wenger, 2004). Thus, it is important to use an appropriate baseline for assessing the gain (or loss) due to an added signal. The capacity coefficient, a measure from SFT that we describe in detail in the next section, addresses this issue because it uses individual source performance to predict what performance would be in a multi-signal context under a baseline model assumption.

By using SFT, we go beyond the simple better/worse distinctions that are possible with the previously applied metrics. SFT allows us to examine the reason for observed performance differences including the differential effects of increasing the amount of available information (i.e., processing efficiency), facilitation or inhibition between the perception of each source of information, whether processing one image source is sufficient or both sources must be processed, and the temporal organization of the perception (i.e., serial versus parallel).

Systems factorial technology

To examine the basic perceptual processing of cognitively and algorithmically fused imagery, we applied SFT. The SFT framework supplies information about important cognitive properties including workload capacity, independence, architecture, and the stopping rule. Workload capacity refers to the change in the processing rate of information of an individual sensor when going from single- to multi-sensor presentation. Independence is the degree to which the processing of each type of sensor information influences the processing of the other. Architecture refers to whether processing is simultaneous (parallel processing), sequential (serial processing), or information is pooled (coactive processing). The stopping rule refers to whether one or both sensors must be finished processing when a response is made (e.g., OR or AND).

These SFT constructs are measured using two statistics. The capacity coefficient is used to examine workload capacity and independence. Thus, it is useful for examining how the cognitive processes involved for each source of information (e.g., each sensor image) speed up or slow down as more sources are simultaneously presented (e.g., multiple sensors). The survivor interaction contrast (SIC) is used to examine architecture and the stopping rule, i.e., the SIC is useful for examining the temporal organization of information and the extent to which one or both sensors are processed to completion.

Survivor interaction contrast

The SIC is used to examine whether multiple sources of information are processed serially, in parallel, or information is pooled together (coactive) and if one (OR processing) or both (AND processing) sensors are processed to their entirety. Inference based on SICs is done by examining the interaction between slowing down and speeding up cognitive processing of each individual source. We use S(t) for the survivor function (i.e., the probability that a participant has not responded by a given time) and indicate the level of the salience manipulation by the subscript of S(t). High salience conditions are denoted H and low salience conditions are denoted L. Throughout this paper, the first subscript indicates the level of the LWIR signal and the second subscript indicates the level of the visible sensor. For example, the survivor function of the RTs when LWIR is high salience and visible is low salience is denoted S _HL(t). Using this notation, the SIC is defined as:

$$ \text{SIC}(t)= \left[S_{\text{LL}}(t)-S_{\text{LH}}(t)\right]-\left[S_{\text{HL}}(t)-S_{\text{HH}}(t)\right]. $$

(2)

The manipulations that speed up or slow down processing, known as the salience manipulations, must affect only the speed of processing for the respective source of information, a property known as selective influence (Ashby & Townsend, 1980; Dzhafarov, 2003). If the manipulation is effective and selective influence holds, the fastest responses are made when both sources have high salience and slowest when both sources have low salience. If affective selective influence manipulations are used, each of the five classes of models predicts a unique SIC shape (see Fig. 1; Dzhafarov, Schweickert, & Sung, 2004; Houpt & Townsend, 2011; Townsend & Nozawa, 1995; Zhang & Dzhafarov, 2015).

Positive and negative SIC deviations from zero are tested using the Houpt–Townsend statistic (Houpt & Townsend, 2010) and are used to reject candidate processing models. Specifically, the statistic tests for significant deviations from zero of both the largest positive (D ⁺) and largest negative (D ⁻) value of the SIC curve. If the cognitive process follows a serial-OR rule, the predicted SIC is flat and hence neither D ⁺ nor D ⁻ should be significant. A parallel-AND model implies an all negative SIC, which should lead to a significant D ⁻ but non-significant D ⁺. A parallel-OR implies an all positive SIC, hence a significant D ⁺ but non-significant D ⁻. Both a serial-AND and coactive model result in an SIC that is first negative then positive so both D ⁺ and D ⁻ should be significant.

Rather than using the traditional conservative cutoff for statistical significance (α=0.05), we use α=0.33 for our applications of the Houpt–Townsend statistic. Typically, α is set to be biased towards indicating a non-significant effect to limit Type I errors. The null hypothesis for the Houpt–Townsend statistic is SIC(t)=0 for all t and hence conservative α levels bias the tests toward indicating a serial-OR signature (flat SIC). While this approach has worked well for model recovery in simulated data (Houpt, 2014), we also applied a recently developed hierarchical Bayesian analysis to the mean interaction contrast (MIC), which we introduce next, to corroborate conclusions from the Houpt–Townsend statistics.

Hypotheses

The use of SFT allows us to examine the underlying processes to help explain why we may see performance benefits of a particular operator display. Each variation in processing structure may inform the cause for a particular pattern of performance. If participants are presented with task-relevant yet redundant information across sensors, they may adopt a processing strategy in which information from only one sensor is used to make the decision (i.e., OR processing or first-terminating). OR processing may combine with either a parallel- or serial-processing structure: either information from both sensors is processed simultaneously but only the fastest to finish is used to make the discrimination (parallel-OR) or information from one sensor is processed and is used for the decision while the alternative sensor is not processed (serial-OR). Alternatively, individual sensor images may each contribute unique complementary information forcing participants to process both sensors entirely to make a correct decision (AND processing). AND processing may also combine with either a parallel- or serial-processing structure: both sensors are processed simultaneously and the slowest to finish is used to make the discrimination (parallel-AND) or both sensors are fully processed, first one, then the other (serial-AND). Fusion also allows for a single percept in which all information is processed in parallel and is pooled to make a decision (coactive processing).

Here we discuss what particular processing mechanisms suggest on a more conceptual level about visual cognition for each presentation type: algorithmic and cognitive fusion.

For algorithmically fused images, standard serial and parallel architectures may be possible, although are a priori unlikely. An interpretation of such a finding would be that participants can selectively attend to one particular spatial frequency information based on the distinctive features to complete the task (Morrison & Schyns, 2001). Alternatively, if observers are unable to extract information selectively from each perceptual dimension, as indicated by McCarley & Krebs (2006), then a coactive or interactive parallel process is more likely (cf. Eidels, Houpt, Pei, Altieri, & Townsend, 2011). For algorithm-fused imagery, we hypothesize: (1) individuals’ efficiency will be at least as high as respective UCIP predictions (i.e., unlimited capacity) across all discrimination stimuli and (2) individuals will use a highly interactive parallel mechanism for processing the multi-sensor information.

When images are presented beside one another (i.e., cognitive fusion), people may process each sensor image in series or in parallel. If processing both images requires visual attention shifts between the two images, then it may be more likely that the images are processed in series. This mechanism limits performance by the constraints of mental integration across several samples of information (Irwin, 1991; Rayner, McConkie, & Zola, 1980). However, serial processes can lead to efficient processing if information from only one image is sufficient for adequate judgments and the additional image is redundant and potentially unnecessary (Neriani et al., 2008).

The cognitive processes involved with utilizing information from multiple sensors may vary from the processing of one sensor image. A cognitively motivated baseline model can encode a specific set of processes so that systematic deviations from the baseline will give evidence for how the processes have changed. Furthermore, using a standardized method to assess deviations of actual performance from predicted performance given the individual parts yields a flexible approach to make comparisons of human processes across several experimental manipulations such as alternative sensors, stimuli, and fusion methods.

Double factorial paradigm

The trials for the SIC were collected in a separate block from those blocks that were included for estimating the capacity coefficient. This allowed us to balance the number of trials in such a way as not to bias responses to one source based on the other source (conditioned on the stimulus) in accordance with the constraints outlined in Houpt et al. (2012) following Mordkoff & Yantis (1991).

To estimate the capacity coefficient, we need RTs from trials in which participants can respond to both visible and LWIR images (i.e., either algorithmically or cognitively fused imagery) as well as trials in which they are focused only on a single source (i.e., visible only or LWIR only). To get the best estimate of what UCIP performance would be, trial type was blocked. Hence, each participant had a block that was entirely dedicated to visible imagery, a separate block dedicated to LWIR imagery, and a block dedicated to fused imagery.

For capacity analyses, we used the imagery without any added noise, which corresponded to the high salience (H) conditions in the SIC analysis (outlined in the “Survivor interaction contrast” section above). Recall, the order of the elements in the subscript indicates the source of information, with the first subscript indicating the LWIR information and the second indicating the visible information. Hence, we denote the visible-only trials with the subscript ∅H, the LWIR trials with H ∅, and the fusion trials with HH.

To estimate the SIC, we need RTs from each factorial combination of source image salience (i.e., with or without added noise). To interpret the SIC appropriately, the salience manipulations must satisfy the assumption of selective influence: the presence or absence of noise added to a source image (e.g., LWIR) should affect the perception of that source but not the other source (e.g., visible).

Participants

All participants self-reported right-handedness, normal or corrected to normal visual acuity, normal color vision, and no difficulties reading English.

Materials

Stimuli were presented using PsychoPy (Peirce, 2009) on a 20-inch Sony Trinitron monitor. Participants sat at a table 75 cm from the monitor. Responses were made using a right or left click on a two-button mouse.

Fusion

We used the Laplacian pyramid transform (LPT; Burt & Adelson, 1983) to combine the visible and LWIR information into one composite image. Subjective and image quality assessments support the use of LPT (Petrović, 2007). LPT is a pixel-level pyramid-based algorithm utilizing six band filters to pass across both sensor images resulting in a series of image components at different resolution qualities. The component images were averaged together across sensors at each band-pass level and combined using a Laplacian transform. The result was a single composite image containing information from both individual sensors (see Fig. 2 for an example).

Note that, as evident in Figs. 2 and 3, combining a LWIR image and a visible image using the algorithm does not necessarily enhance the image and may actually degrade the quality of the composite representation. Often, added image enhancement techniques are used to provide benefits above raw algorithmic fusion. In our study, we use only the existing algorithm supported in the literature to simulate a more real-world environment where the particular task information, and how to enhance this information further, is unknown before displaying the composite algorithmic image.

Procedure

Each experiment consisted of 10 days of 1-hour sessions. All participants were compensated $8 per session with a $2 per session completion bonus: $8 + $2 bonus × 10 days = $100 in total for each experiment.

The algorithmically fused images were always presented in the center of the screen within 2.86° of visual angle. For cognitive fusion, both single-sensor images were simultaneously presented 0.67° apart (inner-edge to inner-edge) within 6.39° of visual angle on the screen and directly to the left and right of center screen (cf. Fig. 4).

At the beginning of each trial, either a single localization box was shown in the center (algorithmic fusion blocks) or two boxes were presented side by side (cognitive fusion blocks). Localization boxes were always presented for a random interval of time between 400 and 500 msec followed by the stimulus. In the algorithmic fusion blocks, one image was randomly selected and always presented in the middle of the screen. In cognitive fusion blocks, the single-sensor trials required one image that was displayed either to the left or right of the center. In each cognitive fusion trial, the stimuli were displayed with minimal visual angle to allow participants to keep their eyes fixated in the center of the screen without having to saccade for perceptual processing of all the information. Following the stimulus, a blank screen was presented for a response. No trial-by-trial feedback was given.

Analysis

To analyze differences in operator performance when presented with cognitive or algorithmic fusion, we first applied a traditional analysis of mean correct RTs and accuracy, followed by an SFT analysis. For the SFT analysis, we estimated the capacity coefficient for each individual in each condition. We analyzed the SIC and MIC only for individuals for whom their data did not indicate a violation of selective influence. In the “Results” section, we note whether a participant passed or failed selective influence. To pass selective influence, we used paired Kolmogorov–Smirnov tests of RT survivor distributions to test that, for all t: S _HH(t)<S _HL(t) and \(S_{\text {HH}} \ngtr S_{\text {HL}}\), S _HH(t)<S _LH(t) and \(S_{\text {HH}} \ngtr S_{\text {LH}}\), S _LL(t)>S _HL(t) and \(S_{\text {LL}} \nless S_{\text {HL}}\), and S _LL(t)<S _LH(t) and \(S_{\text {HH}} \ngtr S_{\text {LH}}\).

Method

Results

In summary, responses were faster and more accurate with visible imagery than LWIR imagery for the pointing discrimination stimuli but the reverse is shown with similar facing discrimination stimuli. Participants had limited capacity with both fusion types, more so for algorithmic than cognitive fusion.

Accuracy and mean correct RT analysis

Because the number of sensors could not be fully crossed with fusion type (fused imagery, whether cognitive or algorithmic, included more than one sensor by definition) or with sensor type (when two sensor types were present, then both infrared and visible were necessarily displayed), we computed three separate repeated-measures ANOVA to examine, respectively, single- to multi-sensor comparisons, within multi-sensor comparisons, and within single-sensor comparisons for the mean RT and accuracy.

Table 1 gives the results of a 2×2 repeated-measures ANOVA to assess the effects of the number of sensors presented (single or multiple) and the stimuli (pointing or facing) for both correct RTs and accuracy. For both correct RTs and accuracy, there was a significant interaction between number of sensors and stimuli with the main effects of the number of sensors presented and stimuli type (Table 1). Figure 5 indicates slower, less accurate performance with the facing discrimination stimuli. Across facing and pointing stimuli, performance with multi-sensor imagery suffers more than performance with single-sensor imagery.

	Correct response time			Accuracy
Condition	F	df	\({\eta _{G}^{2}}\)	F	df	\({\eta _{G}^{2}}\)
Number of sensors × stimuli	12.45**	1, 9	0.01	60.53***	1, 9	0.19
Number of sensors	5.28*	1, 9	0.00	9.54*	1, 9	0.05
Stimuli	11.57**	1, 9	0.28	17.19**	1, 9	0.36

\({\eta _{G}^{2}}\) generalized eta-squared

* p<0.05; ** p<0.01; *** p<0.001

LWIR long-wave infrared

Table 2 gives the results of an additional 2×2 repeated-measures ANOVA to assess the effects of the multi-sensor fusion method (algorithmic or cognitive) and the stimuli (pointing or facing) for correct RTs and accuracy. For correct RTs, we found a significant interaction between fusion method and stimuli type with a significant main effect of stimuli. However, we did not find a significant main effect of fusion method (likely due to the cross-over interaction). An analysis of accuracy (Table 2) indicated a significant interaction of fusion type and stimuli with significant main effects of both fusion type and stimuli. Figure 5 indicates algorithmic fusion is faster and slightly less accurate in the pointing discrimination, but is slower and less accurate in the facing discrimination.

	Correct response time			Accuracy
Condition	F	df	\({\eta _{G}^{2}}\)	F	df	\({\eta _{G}^{2}}\)
Fusion technique × stimuli	6.73*	1, 9	0.09	56.84***	1, 9	0.43
Fusion technique	0.07	1, 9	0.00	76.17***	1, 9	0.52
Stimuli	13.87**	1, 9	0.27	37.62***	1, 9	0.51

\({\eta _{G}^{2}}\) generalized eta-squared

* p<0.05; ** p<0.01; *** p<0.001

LWIR long-wave infrared

Lastly, Table 3 gives the results of a 2×2×2 repeated-measures ANOVA to assess the effects of single-image presentation type (left/right of center or center), sensor (visible or LWIR), and stimuli (pointing or facing) to predict correct RTs and accuracy. For both correct RTs and accuracy, the three-way interaction of presentation type, sensor, and stimuli and two-way interaction between presentation type and sensor were not significant. For both correct RTs and accuracy, there was a significant interaction of presentation type and stimuli and a significant interaction of sensor and stimuli with main effects of presentation type and sensor. There was a significant main effect of stimuli for correct RTs but not for accuracy.

	Response time			Accuracy
Condition	F	df	\({\eta _{G}^{2}}\)	F	df	\({\eta _{G}^{2}}\)
Display method × sensor × stimuli	5.11	1, 9	0.00	2.77	1, 9	0.01
Display method × sensor	2.04	1, 9	0.00	0.11	1, 9	0.00
Display method × stimuli	6.92*	1, 9	0.04	7.29*	1, 9	0.07
Sensor × stimuli	32.58***	1, 9	0.05	32.56***	1, 9	0.28
Display method	92.53***	1, 9	0.26	6.46*	1, 9	0.08
Sensor	6.15*	1, 9	0.00	40.46***	1, 9	0.13
Stimuli	8.93*	1, 9	0.22	2.11	1, 9	.04

\({\eta _{G}^{2}}\) generalized eta-squared

* p<0.05; ** p<0.01; *** p<0.001

LWIR long-wave infrared

Recall that for algorithmic fusion blocks, the single-sensor image was always presented in the middle of the screen. In cognitive fusion blocks, the single-sensor trials required one image that was displayed either to the left or right of the center. Figure 5 indicates both LWIR and visible single-sensor trials were faster and more accurate when visual attention was anticipating stimuli on a smaller visual area (algorithm-fused block of trials) than a larger visual area (cognitive-fused block of trials) even though the same single-sensor image was presented in both conditions.

Discussion

Both cognitive and algorithmic fusion hindered processing of the individual source images relative to independent parallel processing. Because information was redundant across the two images, participants should be faster with two images than with a single image, even with independent parallel processing of each image (cf. Raab, 1962). Subjects were slightly faster with the side-by-side images than the single-source images; however, the capacity results indicate that the speed-up was not as much as would be observed from independent parallel processing. Performance was even worse with the algorithmically fused images: RTs were slower with algorithmically fused images than with either of the single-sensor images. Hence, capacity coefficient values were quite low for algorithmic fusion, much lower than cognitive fusion.

Low capacity coefficient values can result from a number of different violations of the baseline UCIP model predictions. All other factors being equal, serial processing systems have more limited capacity than parallel, while coactive processing systems have higher capacity than standard parallel (Townsend & Nozawa, 1995; Townsend & Wenger, 2004).¹ Unfortunately, our results from the SIC analysis did not lead to clear results regarding processing architecture. All participants’ data indicated violations of selective influence for the algorithmically fused images. Most participants indicated a violation of selective influence with cognitive fusion. Of those participants that did not violate the distribution ordering implied by selective influence, null-hypothesis testing indicated a variety or processing strategies: parallel-OR process and parallel-AND with the pointing stimuli and serial-OR with the facing stimuli. The Bayesian analysis of the MIC indicated that there is very slight evidence in favor of a zero MIC at the group level (MIC=0) and similarly minimal evidence for any MIC category (positive, negative, or zero) at the individual level for both stimuli types.

Among those participants wo may be using a parallel-OR processing strategy, capacity coefficients were still quite limited indicating that there may be other deficits relative to the UCIP model. Given the short presentation time and that at least one of the images was extrafoveal, a violation of the unlimited capacity assumption is a likely cause. With a single image, participants can fixate on the most informative region of that image to get the most out of the image. When there are two images, at most one can be fixated so information uptake is almost certainly not the same with two images relative to one. Limitations of visual short-term memory may degrade the ability to integrate information from multiple sensors or potentially facilitate the strategy to process only a single informative sensor image (Irwin, 1991; Rayner et al., 1980).

With algorithmic fusion, only a single image is presented, so participants can fixate the most informative region. Hence, the limitations on visual attention that may explain low capacity values for cognitive fusion are not sufficient for algorithmic fusion. Although we were not able to draw direct inferences from the SIC, we can make some inferences about the processing. Independent serial or parallel processing are unlikely candidates, as they should have led to effective selective influence and hence ordered distributions (Dzhafarov, 2003; Houpt & Townsend, 2010; Houpt, Blaha, McIntire, Havig, & Townsend, 2014). A priori, it is difficult to imagine how (or why) the visual system would separate the information from each source before processing. Indeed, previous research using sophisticated accuracy-based methodologies found that individual sensor information was perceptually nonseparable in an algorithmically combined image (McCarley & Krebs, 2006). Because the combined algorithmic image is processed as a single unit of information that integrates information from both sensors, the visual processing decision is like a coactive process. However, unlike most coactive processes, the capacity values are much lower than independent parallel, not higher. This suggests that useful information is lost in the fusion process, perhaps more akin to an inhibitory parallel process (cf. Eidels et al., 2011). The potential information loss is evident in Figs. 2 and 3, in which the person looks more clearly differentiated from the background in the single-sensor images than in the algorithmically fused image.

Based on McCarley & Krebs (2000) and Krebs & Sinai (2002), we had assumed that a more difficult stimulus set (i.e., degraded quality of image and type of psychophysical task) would lead to higher capacity coefficient values for the algorithmically fused imagery. The more difficult stimuli in our experiment were when the facing stimuli were not directly centered (since the pointing stimuli were centered) and there were fewer physical cues to aid in decision-making. Capacity was higher at the group level with the pointing stimuli than with the facing stimuli when using algorithmic fusion (as well as cognitive fusion), although it was not enough of an increase to reach the capacity values from cognitive fusion, let alone the predicted UCIP baseline.

There was some evidence of a differential speed–accuracy trade-off between the algorithmically fused imagery and the cognitively fused images. Algorithmic fusion led to faster and slightly less accurate performance than cognitive fusion in the pointing stimuli. However, algorithmic fusion led to both slower and much less accurate performance than cognitive fusion in the facing stimuli. This may suggest that different fusion approaches may be more appropriate for situations in which accuracy or speed are more important, at least for more simple discriminations, but more exploration is necessary.

Differences in speed–accuracy focus can be problematic for capacity coefficients. Assessment functions (Donkin, Little, & Houpt, 2014; Townsend & Altieri, 2012) are a variation on the capacity coefficient that can ameliorate this problem; however, no inferential statistics are available for the assessment function so we only reported capacity coefficients. We did calculate assessment functions and in all cases, the visual patterns matched our conclusions drawn from the capacity coefficients. These data indicate no significant speed–accuracy impact on processing efficiencies for either algorithmic or cognitive fusion.

Method

Procedures

Experiment 2 instructions were the same as those used with the facing stimuli in Experiment 1. Participants indicated whether a person was facing to to the left or the right side of the screen (see Fig. 3) using the corresponding mouse button. Participants were told to perform the task as quickly and accurately as possible. At the end of each session, participants were informed of their accuracy in each fusion condition. This feedback was provided to keep participants motivated to improve in performance over the course of the training sessions.

Each participant completed 10 days of 1-hour sessions. The first eight sessions contained trials to compute the capacity coefficient for both cognitive and algorithmic fusion. As with Experiment 1, there were 120 trials per distribution (LWIR-alone, visible-alone, LWIR and visible together) for a total of 720 trials for capacity analysis.

The remaining two sessions required first the estimates of each sensor’s psychophysical thresholds at 82% accuracy by manipulating the amount of pink noise added to the image (120 trials for each sensor for each day) followed by trials required to estimate the SIC (2160 trials in total). The SIC trials consisted of factorial combinations of high (no noise) and low (individualized amount of pink noise) of both the LWIR and visible images. LWIR was always presented on the left and visible on the right. For trials with only one sensor present (e.g., LWIR with high salience, visible is absent), the localization box would appear in place of the image (example shown in Fig. 7).

In cognitive fusion blocks, we fixed the location of where the LWIR and visible images are presented across all trials (LWIR: left of center; visible: right of center) instead of randomly displaying each on the left or right for every trial (as in Experiment 1). This gave operators the opportunity to anticipate where each type of information was going to be presented.

The stimulus presentation duration was extended to 2 seconds across all conditions (algorithmic and cognitive, single- and multi-sensor) to allow the operator to sample all the information from each image and allow strategies of processing the information potentially to improve with time. The LWIR image was always displayed on the left and the visible image was always displayed on the right. Following the stimulus, a blank screen was presented for 500 msec, allowing the participant a total of 2.5 seconds to respond starting from stimulus onset.

Results

RTs and accuracy with fused imagery were worse than with single-sensor images. Performance on both single- and multi-sensor imagery improved with training; however, the capacity coefficient consistently indicated inefficient performance with both algorithmic and cognitive fusion, lower capacity results for algorithmic fusion than cognitive fusion, and no efficiency improvements with training. Nonetheless, we found strong evidence for parallel and coactive processing strategies with cognitive fusion, both of which are normally associated with efficient processing.

SFT analysis

Table 9 gives the results of a 2×8 repeated-measures ANOVA to analyze the effects of training on the efficiency of processing multi-sensor information to predict capacity z scores. Participants 12 and 17 were excluded from efficiency comparisons across training sessions because of low accuracy in the early training sessions. Figure 8 illustrates that individual capacity z scores with cognitive fusion were less limited than z scores with algorithmic fusion [ t(143)=−12.19, p<.05, d=1.45]. Capacity z scores become significantly more limited from the first day (Day 1) to the last day (Day 8) of training for both algorithmic fusion [ t(17)=3.03, p<.05, d=1.02] and cognitive fusion [ t(17)=2.99, p<.05, d=0.49].

	z score
Condition	F	df	\({\eta _{G}^{2}}\)
Number of training sessions × fusion technique	0.03	1, 16	0.00
Number of training sessions	10.29 **	1, 16	0.05
Fusion technique	21.12***	1, 16	0.53

\({\eta _{G}^{2}}\) generalized eta-squared

* p<0.05; ** p<0.01; *** p<0.001

Table 10 indicates the participants whose data passed the selective influence test, the participants’ Houpt–Townsend SIC statistic for both positive and negative deviations from zero, the MIC statistic, and the processing model that predicts their pattern of results. Distributional orderings did not indicate violations of selective influence for 11 participants. Ten of those participants had a significantly positive SIC (p<.33). Four of those participants had a significantly positive MIC and significantly negative SIC. One participant had a significantly positive and negative SIC with a non-significant MIC. One participant had a significantly negative SIC. Participant 9 had SIC/MIC results that are not predicted by any of the independent serial/parallel/coactive AND/OR models.

Subject	Pass/fail	D +	D −	MIC	Architecture
3	Pass	0.349***	0.241**	60.101***	Coactive
9	Pass	0.160 +	0.182*	-4.752 +	Ambiguous
10	Pass	0.190 +	0.071	48.077 +	Parallel-OR
11	Pass	0.257**	0.125 +	103.470*	Coactive
13	Pass	0.429***	0.071	152.638***	Parallel-OR
14	Pass	0.109 +	0.151	16.710	Ambiguous
15	Pass	0.263**	0.225*	51.046*	Coactive
16	Pass	0.230*	0.154 +	51.050 +	Coactive
17	Pass	0.198*	0.048	62.970***	Parallel-OR
19	Pass	0.142 +	0.258**	32.772	Serial-AND
20	Pass	0.041	0.165 +	−42.617	Ambiguous

MIC mean interaction contrast Houpt–Townsend statistic (D ⁺, D ⁻): ⁺ p<0.33; * p<0.05; ** p<0.01; *** p<0.001

MIC: ⁺ p<0.33; * p<0.05; ** p<0.01; *** p<0.001

Bold D ⁺ and D ⁻ statistics indicate a significant Houpt–Townsend statistic at p<0.33

With the hierarchical Bayesian MIC analysis, we found good evidence for a positive MIC at the group level (\(\hat {p}_{\text {posterior}}^{+}=0.73\)). The remaining models were equally unlikely (\(\hat {p}_{\text {posterior}}^{-}=0.15\); \(\hat {p}_{\text {posterior}}^{0}=0.12\)). At the individual level, the posterior probabilities supported the conclusions drawn from the Houpt–Townsend statistic of positive and negative deviations of the SIC (Table 10). The most likely model for nine participants had MIC>0 with MIC=0 as the second most likely. Among those participants, the ratio of posterior odds ranged from 4.5 to 70.0, indicating strong to decisive evidence for each individual. Participant 9 had the most likely positive MIC with MIC<0 second most likely. Participant 20 had a most likely negative MIC with MIC>0 second most likely. Both Participants 9 and 20 had minimal evidence in favor of the most likely model, with a ratio of posterior odds of 1.9 and 2.0 over the next best model, respectively.

Discussion

In Experiment 2, our aim was to produce consistency within an individual and across people in the processes involved with multi-sensor information. We found nearly identical capacity results with those of Experiment 1 despite the several experimental changes: (1) increased experience with multi-sensor imagery, (2) realistic degradation of image quality with pink noise, (3) longer stimulus presentation time, and (4) fixing LWIR to the left-hand side of the screen and visible to the right-hand side. Even with many experimental changes, we consistently found limited workload capacity with both algorithmic and cognitive fusion. Similarly, the discrepancy between single- and multi-sensor performance with algorithmic fusion was much larger than cognitive fusion. Likewise, we found lower capacity results for algorithmic fusion than cognitive fusion.

When participants had undergone training, there were clear results indicating processing architecture from SIC analyses. We found group-level evidence of parallel-OR or coactive processing (the MIC cannot distinguish between these processing strategies). The ability to process both images in parallel leaves opportunity for facilitation in performance from the redundancy speed-ups across the two images (Kahneman, 1973; Pollatsek et al., 1984).

Over the course of training, performance improved for all single- and multi-sensor conditions. These raw RT results cannot discriminate whether the multi-sensor performance improvement was due to better use of single-sensor images or improvements in the integration of the sensor images. By applying the capacity coefficient, it was clear that integration of multi-sensor imagery did not improve with training, and in fact may have degraded.

Despite limited capacity results, we still find evidence for efficient processing strategies. SIC and MIC results from the cognitive fusion conditions indicate clear evidence against serial processes, in favor of parallel-OR or even coactive processing. Although we could not draw conclusions from the algorithm-fused imagery, we assumed serial processing of each source was highly improbable, and the process is more likely a type of coactivation. Thus, the limited capacity results are not due to inefficient serial processing of information. For cognitively fused imagery, the available processing capacity could be divided between the two sources of information and in turn slow down the processing of the individual sensors or the information provided from each sensor inhibits processing of the alternative. For algorithm-fused imagery, limited capacity results may result from inhibition that degrades sensor integration in the overall composite image.

Algorithmic fusion

Based on the existing research with algorithmic image fusion, we expected fusion would provide, at a minimum, equally efficient processing as an UCIP model. However, our results indicate just the opposite in that it has been an assumption that multi-spectral fusion can enhance both speed and accuracy performance compared to individual sensor images. This discrepancy is partially due to alternative methods of analysis. For some conditions, the traditional analyses of RTs would indicate a benefit in performance with cognitive fusion compared to either single sensor alone (Fig. 5). While it seems as if performance is enhanced with the side-by-side presentation, these RT speed-ups are not faster than what can be attributed to what is expected when completing a task that only demands one source and the fastest of the two can be sampled on each given trial (i.e., statistical facilitation; Raab, 1962).

Some previous research based on traditional analyses has suggested that algorithmic fusion, at best, performed just as well as individual sensor performance and potentially hinders performance or situational awareness (Krebs & Sinai, 2002; Steele & Perconti, 1997; Krebs, Scribner, Miller, Ogawa, & Schuler 1998). In those studies and our current work, it is possible that the quality of information in the algorithmically fused image was degraded compared to the individual sensor images. Even if the fused image were of equal quality to one or other of the original images, it would not be sufficient to achieve unlimited capacity performance because there would be no opportunity for redundancy gain. The algorithmically fused image would need to have better information quality than either single-source image.

The potential reduction in image quality may be because no consideration of the task or stimuli was used in choosing the particular algorithm. If task-specific image enhancement techniques are not utilized, task-relevant information may be filtered out in the fusion (Dixon et al., 2006; Toet & Hogervorst, 2012). Ideally, the choice of algorithm should attempt to adjust to particular task demands and environmental constraints to obtain optimal scene information (e.g., Yong, Weiqi, & Rui, 2010); however, when systems are designed for general use, the task many not be known in advance.

Cognitive fusion

For cognitive fusion, we found RT speed-ups for some conditions when comparing an individual sensor image to the presentation of both images side-by-side. However, those speed-ups were not significantly faster than our predicted model baseline. Limited capacity may result from any violation of the baseline assumptions: unlimited capacity, independence, or parallel processing. By using careful experimental control in Experiment 2, we saw strong evidence for parallel (even coactive) processing, leaving two potential explanations for limited capacity with cognitive fusion. Although the capacity coefficient cannot directly distinguish between violations of independence and workload, we can speculate about the potential underlying mechanisms using previous research in conjunction with our findings: (1) There could be a limitation of workload capacity or (2) there could be dependencies between processing of the two sources of information (Eidels et al., 2011). Although the first is possible, there would have to be an extreme workload capacity limitation to overcome the benefits of coactivation (cf. Townsend & Wenger, 2004). In favor of the latter, McCarley & Krebs (2006) used general recognition theory (Ashby & Townsend, 1986) and found the perceptual dimensions of algorithmically combined imagery are nonseparable. For future research, we are interested in investigating cognitive fusion with general recognition theory as well.