People can interpret complex messages encoded in visual features. They know red splotches on a weather map signal impending storms, red traffic lights signal stop, and red milk cartons signal that the container holds whole milk. Given this ability, people use colors to communicate important and time-sensitive information. For example, a recent surgical protocol for separating conjoined twins used green and purple tape to signal which monitors and equipment were dedicated to each twin (Associated Press, 2017), presumably so they did not get mixed up during surgery.
Color is one of many visual features that can be used to communicate abstract information, with others including size, texture, orientation, and shape (Bertin, 1983; Ware, 2012). However, color is especially useful for signaling because it can be observed quickly from a distance and it provides meaningful information that is independent from spatial structure. In nature, changes in face color can signal changes in emotional state independent of facial features and changes in fruit color signal ripeness independent from changes in shape (Lafer-Sousa, Conway, & Kanwisher, 2016; Thorstenson, Elliot, Pazda, Perrett, & Xiao, 2017). In human-made artifacts, differences in font color can signal different meanings in signs and maps without affecting legibility of the text. People even make inferences about student ability and teacher competence based on the ink color used to provide feedback on essays (Richards & Fink, 2017). Most relevant to the present study, differences in surface colors can signal different kinds of recycling bins without interfering with the ability to insert objects into the bins.
Yet, interpreting colors is complicated because there is no one-to-one correspondence between colors and concepts (Fig. 1a) in nature or the human-made world (Elliot & Maier, 2012; Humphrey, 1976; Lin, Fortuna, Kulkarni, Stone, & Heer, 2013; Setlur & Stone, 2016). There are one-to-many mappings (Fig. 1b), in which the same color is associated with multiple concepts (e.g. red associated with ripe apples, strawberries, fire, the US Republican Party, and the University of Wisconsin–Madison) and many-to-one mappings (Fig. 1c), in which many colors are associated with the same concept (shades of reds, yellows, and greens associated with ripe apples) (Schloss & Heck, 2017). How, then, do observers interpret reliable and meaningful signals from colors?
We addressed this question by investigating how observers interpret colors in color-coding systems designed for visual communication. When people attempt to communicate through visual media (e.g. graphs, maps, signs, and artifacts), two distinct tasks emerge. There is an encoding task, in which designersFootnote 1 select perceptual features to signify concepts for a design, and a decoding task, in which observers interpret how perceptual features map onto concepts in the design (Cleveland & McGill, 1984; Wood, 1968). Ideally, observers will be able to decode the same message that was encoded by the designer.
This decoding ability depends on the degree to which encodings match people’s predicted mappings, or expectations (Norman, 1988, 2013; Tversky, 2011; Tversky, Morrison, & Betrancourt, 2002; Zacks & Tversky, 1999). For example, observers are faster at interpreting bar graphs depicting fruit sales when the bar colors match the colors of the fruit they represent (e.g. banana – yellow, blueberry – blue) than when they mismatch (e.g. banana – orange, blueberry – green) (Lin et al., 2013). One might argue that if color-coding systems are clearly labeled, then interpreting those systems is trivial—you just look up the answer. However, Lin et al. (2013) demonstrated that there is a processing cost to interpreting color-coding systems (even with clear labels) if they do not correspond to people’s predictions for how colors should map onto concepts. The question is, what determines people’s predicted mappings for color-coding systems?
We approach this question by considering visual communication as a set of assignment problems. In optimization and operations research, assignment problems (also known as maximum-weight matching problems) are mathematical models that describe how to pair items from two different categories (Kuhn, 1955; Munkres, 1957). Examples include optimally assigning employees to jobs in a company, machines to tasks in a factory, and trucks to routes in a shipping network (Williams, 2013; Winston & Goldberg, 2004).
Here, we consider two types of assignment problems for generating and interpreting color-coding systems, which correspond to the encoding and decoding tasks described above: encoding assignment problems and decoding assignment problems. Although we focus on color-coding systems here, the principles can generalize to any coding system in which concepts map onto perceptual features.
Encoding assignment problem
Designers can use encoding assignment problems to generate color-coding systems by determining optimal assignments between colors and concepts. Figure 2a illustrates an encoding assignment problem as a bipartite graph. There are 37 colors (denoted using index \( i\in \left\{1,\dots, 37\right\} \)) and six objects (denoted using index \( j\in \left\{1,\dots, 6\right\} \)). Here, and henceforth, we typically refer to "objects" instead of "concepts" because the focus of this paper is on color-coding systems for objects to be discarded in trash and recycling bins. The choice of numerical labels is arbitrary and only serves to simplify the explanation. Each potential pairing \( \left(i,j\right) \) has a corresponding merit score, \( {m}_{ij} \), which quantifies the desirability of pairing \( i \) with \( j \), computed using a merit function. Merit scores can be thought of as weights on each of the edges of the graph in Fig. 2a.
Solving an encoding assignment problem means to select a subset of the edges such that each object is assigned to exactly one color and the sum of merit scores along selected edges is maximized. In Fig. 2a, each assigned color-object pair is represented by a black edge and each unassigned color-object pair is represented by a gray edge. The optimal assignment will depend on the particular choice of merit scores.Footnote 2
Lin et al. (2013) used this kind of approach to study how the association strength between colors and concepts influenced people’s ability to interpret color-coding systems in bar graphs. To encode the color-concept pairings for their test stimuli (e.g. graphs of fruit sales), they first obtained frequency distributions of colors in Google Image Search for a series of concepts (e.g. fruits) and then interpreted the resulting color-concept histograms as probability distributions. The authors then computed a merit function, called an affinity, by weighting each probability of a color occurring for a given concept by the inverse of the entropy of that color’s probability distribution across all concepts. This approach rewards strong color-object associations for the intended pairing while penalizing the associations of unintended pairings. Another way to achieve this qualitative property is to use the pointwise mutual information, another information-theoretic quantity, as a merit function (Setlur & Stone, 2016).
Decoding assignment problem
We propose that when people interpret color-coding systems, they solve a decoding assignment problem. To do so, they make inferences about how the designer had mapped colors onto concepts while generating the color-coding system. In the decoding assignment problem in Fig. 2b, there are six colors and six objects that have been selected by the designer in the encoding assignment problem (Fig. 2a). The observer’s task is to infer the encoded assignments (dashed black lines), but how might they go about doing so?
Color Inference Framework
The Color Inference Framework (Schloss, in press) proposes that people make inferences about colors (color inferences) based on an internal representation of color-concept associations that is stored in their minds. There are different kinds of color inference processes that operate on the same internal representation, which are modulated by perceptual context (e.g. colors in a color-coding system) and conceptual context (e.g. concepts in a color-coding system). Here, we aim to understand the assignment inference process for interpreting mappings between colors and concepts, which we believe enables people to solve decoding assignment problems.
We studied assignment color inference in the domain of recycling, where the color-coding system mapped different colored bins to different kinds of objects to be discarded. As described below, our approach was to manipulate input into the color inference system (i.e. the colors people saw in the experiments), measure the output of the system (i.e. people’s interpretations of how colors mapped onto objects to be discarded in our recycling task), and use those measures to evaluate hypotheses about how people make assignment color inferences.
Input for assignment inference
We selected the colors for each experiment based on color-object association ratings obtained from 49 participants in a pilot experiment (see Additional file 1 for methodological details). In short, participants rated how strongly they associated each of the Berkeley Color Project 37 (BCP-37) colors (Palmer & Schloss, 2010; Schloss, Strauss, & Palmer, 2013) with each of six objects related to recycling: paper, plastic, glass, metal, compost, and trash. Approximations of the colors are shown in Fig. 3 and the CIE 1931 xyY coordinates are in Additional file 1: Table S1). The mean association ratings are displayed in Fig. 4, with the colors sorted from least associated to most associated with each object. We describe the details of how we selected the colors for Experiment 1 and Experiment 2 within the sections on each experiment below.
Output for assignment inference
To assess participants’ interpretations of how colors mapped onto objects to be discarded, we devised a recycling classification task. Participants saw images of unlabeled colored bins along with the name of object to discard (e.g. paper or trash) and they reported which colored bin was the correct one for discarding the object. It has previously been established that asking participants to interpret messages encoded in unlabeled perceptual features reveals how observers extract meaning from visual media (Zacks & Tversky, 1999).
Hypotheses about assignment color inference
We proposed and evaluated two hypotheses about how people perform assignment color inference. When people are given a single object and are asked to map it onto one color from a given set of colors, the local assignment hypothesis predicts that they simply match the object with its most strongly associated color. This means that two different objects could be mapped to the same color if that color is the strongest associate for both objects. In contrast, the global assignment hypothesis predicts that people not only consider the association strength between the object and candidate colors, but also account for the association strengths between all other objects and colors within the scope of the color-coding system. This can result in pairing colors with objects that are weakly associated if it results in better overall pairings for all objects considered.
In this study, we investigated assignment color inference in two experiments. Experiment 1 tested whether people perform local or global assignment in a simple scenario with two objects (paper and trash) and sets of two colors. We chose paper and trash because the colors most associated with these objects are distinct (see Fig. 4 and Additional file 1: Table S2). This avoids conflicts that arise from one-to-many and many-to-one mappings and therefore makes the task relatively easy, at least when one of the colors is strongly associated with paper and the other is strongly associated with trash. Experiment 2 tested whether people can still perform assignment inference with a larger set of six objects and six colors that contain conflicts due to one-to-many and many-to-one mappings. These conflicts make the task of selecting which six colors to use a nontrivial one. We determined which colors to use by designing different merit functions and solving the corresponding encoding assignment problems.