Color inference in visual communication: the meaning of colors in recycling

We generated predictions for the local and global assignment hypotheses using the color-object associations data from the pilot experiment described in Additional file 1 and shown in Fig. 4. To solve an assignment problem under the local assignment hypothesis, we simply match each object with its highest rated color. Under the global assignment hypothesis, we consider both objects and both colors together and pick the pairings that yields the largest total association rating.

One approach for generating these predictions might be to solve assignment problems under each hypothesis by calculating merit scores using the mean color-object association ratings presented in Fig. 4. This approach would be problematic because solving an assignment problem is a deterministic and absolute procedure. This means that if the outcome (e.g. whether “Trash” gets assigned to dark-yellow or saturated-red) hinges on whether one merit score is greater than another, the result will be the same regardless of whether the difference between these merit scores is large or small. However, we want predictions that reflect the sensitivity of outcomes to small changes in the merit scores.

To produce predictions with this sensitivity property, we used a sampling approach. We describe the sampling procedure for generating predictions in detail in Additional file 1. Roughly, we added small random perturbations to the association ratings in Fig. 4, solved the assignment problem using the perturbed values, repeated a large number of times, and then averaged all of the outcomes. This procedure has the desired effect because when association ratings are very different, adding small perturbations has little influence on the outcome of the assignment problem. However, when two association ratings are similar, perturbing them will sometimes cause a reversal in which association rating is largest and yield a different solution to the assignment problem. Repeating many times produces a distribution of outcomes that reflects the magnitude of the differences between merit scores. This approach should approximate the uncertainty in human judgments when different objects have similar color-object associations.

Figure 6 shows the predictions of the local assignment hypothesis (Fig. 6a) and the global assignment hypothesis (Fig. 6b). It also shows the mean proportion of trials (out of 12) that participants chose each color within each color set for each object (Fig. 6c). The pattern of average responses across all objects and color sets was correlated with the predictions of the local assignment hypothesis (r(22) = 0.668, p < 0.001) and the global assignment hypothesis (r(22) = 0.991, p < 0.001), but the correlation with the global assignment hypothesis was significantly stronger (z = 6.13, p < 0.001). There are cases where the global and local assignment hypotheses make similar predictions, but where their predictions diverge, participants’ judgments appear to be more consistent with global assignment.

We also compared the two hypotheses at the individual participant level by conducting logistic regressions⁴ to predict each participant’s response on each trial using the predictions under each hypothesis and comparing the model fits across participants. We coded responses as 1 = left color chosen and 0 = right color chosen (12 trials in color set per object per participant).⁵ We coded predictors for the local and global assignment hypotheses as the probability of choosing the left color, using the values from Fig. 6a and b, respectively. For 18 out of 24 participants, the beta weights for the global assignment model were higher than the beta weights for the local assignment model. A sign test indicated that this distribution is significantly different from chance (p = 0.023), indicating that the global assignment hypothesis better predicted participant responses.

The results of this experiment demonstrate that people’s interpretations of the meanings of colors are highly context-dependent. They change from trial to trial, depending on the other colors in the scene. For example, participants almost always responded that red was correct for paper when paired with dark-yellow, almost never responded that red was correct for paper when paired with white, and responded near chance (0.5) when saturated-red was paired with saturated-purple (Fig. 6c).

Further, the results demonstrate that people can accurately interpret encoded assignments when none of the colors are strongly associated with the target object. They can do so as long as one of the colors is associated with the non-target object (at least when there are only two objects). For example, saturated-purple and dark-yellow are similarly weakly associated with paper (see Fig. 4 and Additional file 1: Table S2), yet participants systematically reported that saturated purple was the color for paper in SP/DY trials. They can do so because dark-yellow is strongly associated with trash (the non-target object for that trial), and solving the assignment problem tells them if dark-yellow is for trash, then saturated-purple must be the color for paper.

Although people can solve decoding assignment problems when both colors are weakly associated with the target, the response time (RT) data suggest doing so is more difficult compared to cases when one of the colors is strongly associated with the target (Fig. 6d). We analyzed the RT data by first calculating the median RT across all 12 trials within each condition. Figure 6d shows the mean of these median RTs across subjects for each condition, categorized as “Target strong” (one of the colors was strongly associated with the target) or “Target weak” (both colors were weakly associated with the target). RTs were significantly faster for the Target strong conditions (t(23) = 5.26, p < 0.001, d _z  = 1.07), suggesting there is indeed a processing cost to solving the assignment problem when all of the colors are weakly associated with the target.

In summary, Experiment 1 provided evidence that participants approached interpreting the color-coding system in our task as a global assignment problem; they determined which assignments between colors and objects optimized the color-object associations of the entire set. This process sometimes resulted in observers interpreting that objects were intended to be assigned to colors that were their weakest associates, even when there was a stronger associate on the screen. However, there was a processing cost when the target did not have a strongly associated color in the color set.

Figure 10a shows the mean proportion of correct trials each Color Set Type (optimized, baseline) for participants in each Optimized Color Set Group (isolated, balanced) in each Task (select, match). Recall that each Optimized Color Set group saw different optimized color sets determined by the isolated and balanced merit functions but saw the same baseline color set (Fig. 7). The horizontal gray line represents chance (1/6; given six color options per trial). As expected, responses in the baseline condition did not significantly differ from chance (select task – isolated group: t(23) = –1.66, p = 0.111; select task – balanced group: t(23) = 0.24, p = 0.810; match task – isolated group: t(23) = 0.54, p = 0.592; match task – balanced group: t(23) = –0.55, p = 0.590).

We analyzed the accuracy data in Fig. 10a using mixed-effects logistic regression. Each trial was coded such that correct = 1 and incorrect = 0 (see Fig. 7 for specification of correct color-object pairings). The fixed effects were: Color Set Type (optimized vs baseline; within-subjects), Optimized Color Set Group (isolated vs balanced; between subjects), and Task (select vs match; between subjects), and their interactions. The random effects were random intercepts for subjects and random slopes for subjects for Color Set Type.

Overall, there were significant effects of Color Set Type (β = 1.99, z = 12.76, p < 0.001), Optimized Color Set Group (β = 0.52, z = 3.29, p < 0.001), and Task (β = 0.46, z = 2.88, p = 0.004). Accuracy was greater for the optimized color sets than the for baseline color set, for participants in the balanced group than in the isolate group, and for participants in the match task than in the select task. However, Color Set Type interacted with Optimized Color Set Group (β = 0.94, z = 3.02, p = 0.003) and with Task (β = 0.79, z = 2.54, p = 0.011), and there was a three-way interaction (β = 1.50, z = 2.44, p = 0.015).

To understand these interactions, we conducted a similar analysis, but separated the data for each task. As shown in Fig. 10a, responses were more accurate for the optimized color set than for the baseline color set within both the select task (β = 1.58, z = 8.55, p < 0.001) and match task (β = 2.46, z = 9.32, p < 0.001). Within the match task, Color Set Type interacted with Optimized Color Set Group (β = 1.79, z = 3.40, p < 0.001), revealing a benefit of optimizing using balanced merit function compared to the isolated function. There was no such interaction for the select task (β = 0.18, z = 0.50, p = 0.62), which suggests balanced optimization may not benefit performance when participants are only considering one object at a time.

Figure 10b shows mean RTs for each condition, regardless of whether responses were accurate. As in Experiment 1, we first calculated the median RT across the replications for each participant within each condition and conducted the analyses on those medians. We conducted separate analyses for the select and match tasks because they were not directly comparable—the select task involved discarding one object per trial, whereas the match task required assigning each of six objects to a unique color on each trial. Within each task, we conducted a 2 Color Set Types (optimized, baseline; within-subject) × 2 Optimized Color Set Group (isolated, balanced; between-subject) mixed-design ANOVA. Overall, RTs were faster for the optimized color sets than the baseline color sets within the select task (F(1,46) = 22.93, p < 0.001, \( {\eta}_p^2 \) = 0.33) and match task (F(1,46) = 8.23, p = 0.006, \( {\eta}_p^2 \) = 0.15). There was no effect of Optimized Color Set Group for (Fs < 1) and no interaction (Fs < 1) for either task. These results suggest there were greater processing costs for trying to interpret the encoded color-object assignments for the baseline color sets than for the optimized color sets, regardless of the merit function used to optimize the color sets or the task.

Thus far we have focused on the responses averaged over objects. Now we examine the data separated by object. Figure 11 shows predicted responses and actual responses for each of the six objects for each optimized color set and task. We generated predictions based on the global assignment hypothesis for all six objects with each color set using the same procedure described in Experiment 1 and the Additional file 1.⁷ Figure 11a shows the predicted proportion of times each color should be chosen for each object, within the isolated and balanced color sets. Figure 11b and c show the mean proportion of times each color was chosen for each object within each color set, for the select and match tasks, respectively. The corresponding predictions and data for the baseline color set are in Additional file 1: Figure S3. As can be observed in Fig. 11, the model predictions were strongly correlated with the pattern of responses for both color sets within each task: select – isolated (r(34) = 0.80, p < 0.001), select – balanced (r(34) = 0.91, p < 0.001), match – isolated (r(34) = 0.91, p < 0.001), and match – balanced (r(34) = 0.96, p < 0.001). The strong correspondence between the predicted and actual responses suggests participants could successfully interpret the encoded color-object assignments.

We also compared the pattern of responses for the select task (Fig. 11b) and match task (Fig. 11c) within each color set and found they were strongly correlated (isolated color set: r(34) = 0.90, p < 0.001; balanced color set: r(34) = 0.93, p < 0.001). This suggests that participants generally approached the select task as though they had to find the best one-to-one correspondences between colors and objects (as participants in the match task had to do), even though participants in the select task could have put multiple object types in the same colored bin if desired. One notable exception is responses for plastic in the balanced color set (Fig. 11b). The encoding assignment problem assigned plastic to saturated-red, even though plastic is weakly associated with saturated-red, because it is more strongly associated with that color than any of the other objects are (Fig. 8b). The results suggest that people could not infer that plastic should be assigned to saturated-red due to this weak association if they judged one object at a time in the select task (Fig. 11b), but they could form that inference when they completed one-to-one assignments in the match task by a process of elimination (Fig. 11c). It is noteworthy that in the select task, participants did not appear to discard plastic in the white bin even though plastic is strongly associated with white (Fig. 8b), presumably because they were trying to solve an assignment problem with global scope and white is reserved for paper.

The next analysis aimed at understanding how the pattern of results related to color-object associations. As shown in Fig. 12, accuracy significantly increased as the color-object association strength increased. This was true for both the isolated and balanced color sets within the select and match tasks. Figure 12 also shows that RTs for each object decreased as the association strength between the object and the correct color increased in the select task. We could not test for a similar trend in the match task because there were no individual RTs for each object in the match task.

Although there is a positive correlation between association strength and accuracy for both color sets, this does not imply that one should maximize association strength when selecting colors (e.g. isolated merit function). As seen in Figs. 10a and 12, the balanced color set achieves a higher accuracy on average than the isolated color set even though it results in a larger spread of association strengths.

In summary, Experiment 2 showed that participants accurately decoded color-object assignments, even though there were extensive one-to-many and many-to one mappings. Participants were better at doing so when the assignments maximized association strength between assigned color-object pairings and minimized association strength between non-assigned color-object pairings, even though that required pairing objects with more weakly associated colors. However, responses were faster and more accurate when target objects were more strongly associated with their assigned color.

In this study, we investigated how people interpret color-coding systems used for visual communication. We approached this question by considering two types of assignment problems. There is an encoding assignment problem, in which a designer selects which color to pair with each concept in the color coding system, and a decoding assignment problem, in which the observer infers which color the designer paired with each concept in the color-coding system.

In Experiment 1, we used a recycling paradigm to test two competing hypotheses for how people approach the decoding assignment problem: the global assignment hypothesis and the local assignment hypothesis. Evidence supported the global assignment hypothesis. When deciding which colored bin to use for discarding paper or trash, people not only considered the association strength between the target object and candidate bin colors, but also accounted for the association strength between all other objects and colors within the scope of the color-coding system. This often resulted in discarding objects into colored bins that were weakly associated, if it resulted in better overall pairings for all objects considered.

In Experiment 2, we used a similar recycling paradigm with a larger set of objects and colors to test how people’s ability to decode color-coding systems is influenced by the merit functions used to generate the color sets in the encoding assignment problem. We tested two merit functions that produced two optimized color sets. The isolated merit function maximized the association strength between assigned color object parings without concern for associations between unassigned pairings. The balanced merit function simultaneously maximized the association strength between assigned color object parings while minimizing associations between unassigned pairings. Participants could reliably interpret both color sets, but they were more accurate for the balanced color set, primarily in the match task. These results suggest that it is easier to decode color-coding systems when they are designed to avoid strong associations between unassigned color-object pairs, even if that comes at a cost of reducing the overall color-object association strength for assigned color-object pairs.

We note that although we propose that people interpret color-coding systems by solving decoding assignment problems, we do not claim that there is some part of the brain that performs the same computations as in Matlab’s linprog function using the merit scores defined with Eq. 2. Instead, we argue that our results specify constraints on the underlying mechanisms that the brain implements when determining how colors correspond to objects in color-coding systems.

We also note that there is a conceptual distinction between color-object associations, such as rating how strongly white is associated with paper, and color inferences, such as inferring which colored bin is designated for paper. The color-object associations we present here are direct correspondences between colors and objects, given that participants in the pilot study were asked to rate how much they associated each color with each object. At the outset of this study, it was unknown whether participants would approach the recycling tasks by making inferences based on these color-object associations or if they would use some other approach. For example, although paper is strongly associated with white, that does not mean that people would infer that white was the correct bin for discarding paper. Participants could have associated white with the concept of paper, yet green with the concept of recycling paper, in which case they would have inferred the green bin was correct for discarding paper. Yet, we found that people inferred that white bins were for paper and green bins were for compost (Fig. 11), supporting the notion that assignment inference operates on color-object associations.

The present study focused on color-coding in recycling, but the results should generalize to any domain in which there are systematic associations between concepts and the colors that serve as their referents. The limitation, therefore, should depend on the degree to which there are systematic color-concept associations. Previous work on assigning colors to concepts for use in data visualization suggested such limitations might arise for concepts that are not highly “colorable” (e.g. types of fruit), where “colorabilty” was defined with respect to how systematically colors co-occurred with concepts in Google image searches (Lin et al., 2013). Subsequent work on color-concept assignments focused only on concepts that were highly colorable, where colorability was as defined with respect to co-occurrences between concepts and basic color terms (Berlin & Kay, 1969) in the Google n-grams corpus (Setlur & Stone, 2016). However, it is possible that a different method for quantifying color-concept associations—such as direct human judgments used here, or inferring color-word topics from human-designed media (Jahanian, Keshvari, Vishwanathan, & Allebach, 2017)—will reveal systematic associations useful for assigning more abstract concepts to colors for visual communication.