Visual routines are associated with specific graph interpretations

Graphs are often considered intuitive because they display data visuospatially; yet, students still struggle. Because data comparisons are critical for many science, technology, engineering, and mathematics (STEM) tasks, we must learn the underlying reasons why graph comprehension can fail and how it might be improved. Here we explore a core visual mechanism for extracting relations from values in graphs. We argue that when people extract some types of relationships, as simple as, “Is X larger than Y?,” they must use a “visual routine” to extract it. Because data points can be compared in multiple ways (e.g., based on size, contrast, or spatial location), part of the difficulty of between-value comparisons may be that users must focus this routine on relevant comparisons. Here we show that, when there are multiple potential relations to extract from an otherwise extremely simple dataset, visual routines are associated with specific relational comparisons. If the location and order of attention through a graph are associated with the relations that are extracted, then there may be a role for teaching the “right” ordering to students.

Graphical literacy – the ability to read, construct, and interpret visual displays of information – is a critical skill for all citizens, but particularly for those in STEM fields. Everyday STEM tasks require people to judge whether information in a graph supports a claim, identify the cause of a problem on the basis of anomalous data, and extrapolate trends to predict future performance. Although graphs are often perceived as intuitive because they group relevant information together in a visuospatial format (e.g., Larkin & Simon, 1987), graph interpretation requires analysis of the parts of a graph over time, more akin to slowly reading a paragraph than to glancing at a picture (e.g., Carpenter & Shah, 1998). Because of visual capacity limitations, the relations that people extract from graphs may be tightly constrained by the order in which they attend to graphical elements.

Specifically, we propose that extracting a between-value graph comparison (e.g., “Are there more X than Y?”) elicits a serial operation that we refer to as a visual routine (Cavanagh, 2004; Ullman, 1984) in which attention shifts to at least one of the objects in the relation to guide the comparison process. For instance, when judging whether a two-bar graph depicts a specific relation configuration (i.e., [short tall] or [tall short]?), most people either systematically attend to the left bar first or the taller bar first (Michal, Parrot & Franconeri: Three modes for seeing relations between objects, in preparation). The requirement of using a visual routine to extract such relations is not unique to graphs – people also judge color configurations of objects (e.g., “green circle left of red circle?”) by shifting attention to one of the colored objects (Franconeri, Scimeca, Roth, Helseth, & Kahn, 2012; Holcombe, Linares, & Vaziri-Pashkam, 2011; Yuan, Uttal, & Franconeri, 2016).

These visual routines may be needed because of exceptionally tight capacity limits on these types of visual relation extraction (Logan, 1994, 1995; Wolfe, 1998), with one proposal requiring strictly serial processing of objects within a relation (Franconeri et al., 2012). In order to judge a particular spatial relation between two objects (e.g., “Is the larger object on the left?”), one object has to be designated as the target (e.g., the larger object) and one object has to be designated as the referent (e.g., the smaller object). Although the individual features of multiple objects may initially be available at a global, scene-statistic level, such as knowing there are two sizes, two contrast values, and two locations in a display (e.g., Alvarez, 2011), the visual system must have a mechanism for assigning features belonging to the target object and referent object, and one likely mechanism is strict spatial isolation of attention across temporal intervals (Franconeri et al., 2012, Treisman & Gelade, 1980; but see Hummel & Biederman, 1992). Thus, when judging whether the left of two objects is the larger object, a visual routine would isolate the two features “larger” and “left” to a single object, making the location of the larger object explicit.

Past evidence shows that visual routines occur during graph comprehension (e.g., Michal, Uttal, Shah, & Franconeri, 2016), but converging evidence is needed to show that these routines are instrumental for extracting specific relations. For example, when judging the size configuration of the bars in the top graph of Fig. 1, we claim that looking toward the taller bar first allows the viewer to extract the relation “the taller bar is on the right,” rather than “the shorter bar is on the left.” That claim requires evidence that the type of relation judged affects the way that attention shifts (as reflected in eye movements).

In support of this idea, we recently found that interpretations of magnitude relations depended on which item people attended to first (Michal et al., 2016). Participants were asked to verify whether a two-bar graph matched a statement such as, “Are there more blueberries than oranges?,” and visual routines that mimicked the linguistic order within the question led to faster responses. By attending first to the relational target (e.g., the blueberry bar), people were more likely to interpret the graph in a way that was aligned with the question (i.e., they interpreted the graph as “more blueberries than oranges” rather than “fewer oranges than blueberries”), which led to quicker responses.

Because Michal et al. (2016) asked participants to verify a specific framing of a visual relation (e.g., blue bar larger than orange bar?), participants may have been primed to attend to the graph using a sentence-order visual routine. We must therefore additionally show that people naturally interpret graphs in a sentence-like way even when there is no verbal statement to compare the graph to, and when they are free to attend to the graphed values in any temporal order they choose.

One way to test whether visual routines are associated with specific relational comparisons is to ask people to judge relations for which there are multiple ways to compare the two objects – and therefore multiple potential routines. For example, although the bottom graph in Fig. 1 (under the words until response) has only two bars, one could compare the bars’ locations, sizes, or contrast values. Each comparison can also be described on the basis of different dimensions, such as “the taller bar is on the right” or “the darker bar is on the left.” Finally, each comparison can be framed differently; for instance, judging that “the taller bar is on the right” is distinct from “the shorter bar is on the left” (e.g., Clark & Chase, 1972). Thus, when there are several possible ways of comparing data points in a graph, it is critical to be able to make relevant comparisons. For instance, if a person wanted to verify whether the right bar was larger than the left bar in the bottom graph of Fig. 1, they would need to compare the sizes of the bars and not the contrasts or spatial locations.

We asked participants to judge configurations of a two-bar graph on the basis of size (i.e., “[short tall] or [tall short]?”) and contrast values (i.e., “[light dark] or [dark light]?”). By measuring which bar participants attended to first over a range of display types, we could infer the features or “anchor points” (Couclelis, Golledge, Gale, & Tobler, 1987) that guided their comparisons. If people are systematic about the feature that they attend to first, then it is likely that their interpretation of the graph is driven by that guiding feature. For example, a typical observer judging the top graph in Fig. 1 might use the taller bar as an anchor point when comparing the sizes of the bars, whereas she might use the darker bar as an anchor point when comparing the contrasts (middle graph of Fig. 1). But do these visual routines happen simply because of the relative bottom-up salience of those feature values, or are the saccades guided by the relational decision in a top-down manner? The example in the lowest graph of Fig. 1 pits the two strategies of attending to the taller or darker bar first against each other because the graph varies in both size and contrast, but only one dimension is task-relevant. Thus, if visual routines are associated with specific graph interpretations, then the routine used to compare two data points should be based on whichever dimension is currently relevant for the comparison. If a viewer prefers dark and tall, then in this display they should attend left when judging contrast versus right when judging size, even in otherwise identical displays.

By measuring the target object of first saccades over a range of configurations, we first identified anchor points for each participant while they judged graph configurations on the basis of size (with task-irrelevant contrast kept constant) and contrast (with task-irrelevant size kept constant). After completing these single-dimension tasks, participants completed an orthogonal task in which graphs varied in both size and contrast configurations. Half of the participants judged size configurations while task-irrelevant contrast values varied independently (Experiment 1a), and the other half judged contrast configurations while task-irrelevant size values varied independently (Experiment 1b). If visual routines are guided by the task-relevant dimension (size or contrast) during graph comprehension, then people should show the same anchor point biases for the orthogonal task as for the corresponding single-dimension task. Furthermore, if visual routines are implemented in a task-relevant way, then the order of this visual routine might affect how people interpret relations in graphs; if so, then exploring the “right” order for a given problem could have pedagogical implications.

People tend to process graph relations using systematic visual routines, particularly when coordinating data with text (e.g., Michal et al., 2016). Here we show that people exhibited idiosyncratic but highly consistent feature preferences (“anchor points”) to guide visual routines when judging graph relations, even when there was no verbal component to the task. Most participants judged size configurations by attending to the taller bar first, whereas most participants judged contrast configurations by attending to the darker bar first. Importantly, these anchor points persisted in a task-dependent manner: When participants judged graphs that varied in both size and contrast (orthogonal task), preference ratios defined by the task-relevant dimension were significantly stronger than preference ratios defined by the task-irrelevant dimension. Thus, when a graph can be interpreted in multiple ways, people attend to a single dimension (e.g., size) in a specific order (e.g., taller bar first), which may generate a specific interpretation of the graph (e.g., “the taller bar is on the left”). These results are consistent with the proposal that visual routines do not merely facilitate graph comparisons by deploying attention to data points over time; rather, specific visual routines are associated with specific graph interpretations.

Although visual routines were implemented in a task-dependent manner, we cannot determine on the basis of our data whether multiple relations were processed in serial or parallel. However, we speculate that relational comparisons may be computed in serial for two reasons. First, serial processing is a parsimonious solution to the problem of binding separately processed features to single objects (Treisman & Gelade, 1980). If that solution were implemented during graph processing, then focused attention would be required to compute relations over those bound features (e.g., the user only explicitly knows that the left object is the taller one by shifting attention to a single object). Judging a spatial relation that is based on multiple dimensions would then require localization of multiple features that may or may not appear in the same location (e.g., “Where is the taller bar?” plus “Where is the darker bar?”); thus, we think it is unlikely that multiple relations could be judged in a single shift of attention. Second, in a change detection study, when we asked participants to detect changes to two relations instead of only one relation, performance was substantially impaired (Michal & Franconeri: Higher-order relational comparisons are limited to a single dimension at a time, in preparation). Although the present findings cannot confirm whether multiple relational comparisons are processed serially, our results are consistent with a serial mechanism.

Although eye movements were guided primarily by anchor points from the task-relevant dimension in the orthogonal task, variations in the task-irrelevant dimension did affect behavioral responses: Participants responded more quickly, on average, when their task-irrelevant anchor point appeared on the same object (congruent) than when it appeared on a different object (incongruent) as their task-relevant anchor point. Participants may have responded more slowly on incongruent trials if they were tempted to judge the task-irrelevant relation or momentarily forgot which dimension was task-relevant. Thus, despite the fact that visual routines were based on the task-relevant dimension, the task-irrelevant dimension competed for attention during the orthogonal task.

Given that most participants preferred the darker and taller objects, it is difficult to disentangle whether our RT findings reflect interference from the task-irrelevant relation or the relatively greater bottom-up salience of displays in which one object was both darker and taller. However, even if the interference effects were driven by low-level salience, it would still suggest that top-down control is necessary for extracting task-relevant comparisons. In other words, there would still be more competition for attention when salient features appeared on different objects than on the same object; thus, in either case, people need to be able to focus on the task-relevant feature to make a specific type of graph comparison. Importantly, this suggests that people may need to exert top-down control in order to focus on relevant comparisons in graphs.

Viewing patterns interact with cognitive performance in a range of tasks. For example, encouraging relevant eye movement patterns can prime successful problem solving (Grant & Spivey, 2003). In these studies, the claim was that an embodied signal from the eye movements themselves facilitated cognitive processing. In the present study, it is possible that people judged configurations in part by noting the direction of their own eye movements (Franconeri et al., 2012). For instance, when judging the size configuration of two bars, we posit that the size features are processed independently from their locations initially without linking which size appears at which location; however, the act of moving the eyes toward the taller bar would then allow one to infer the bar’s relative location (e.g., left) from the direction of the eye movement (e.g., leftward).

Overt eye movements would not be necessary to harness this embodied signal; the visual system could rely instead on the covert attentional shifts that typically accompany eye movements but are also made in the absence of eye movements. For instance, although we excluded data from participants who did not make a sufficient number of eye movements toward centrally presented bar stimuli, it is possible that these participants were still shifting attention covertly to individual bars when judging their relations. Our results are consistent with the idea that visual routines are embodied within the attentional shifts that typically—but do not always—accompany eye movements. Relations between objects are coded as the direction of motor movements that we would make to fixate on them.

If graph interpretations are driven by embodied visual routines, then there are several direct implications for how to best teach students to extract relevant information in graphs. First, students should focus on whichever comparison is currently task-relevant, which is a skill in itself that students can be taught (e.g., Richland, Morrison, & Holyoak, 2006). Second, as a general strategy for interpreting graphs in an open-ended way (i.e., they are not being asked about a specific directional relationship), students should choose one task-relevant anchor point to attend to first (e.g., the larger data point when comparing sizes) rather than attending to data points in an arbitrary order. By attending to graphs using a task-relevant visual routine and implementing that routine systematically, students may be able to improve their ability to focus on relevant graph comparisons.

In conclusion, although individuals vary extensively in how they implement them, visual routines are strongly associated with specific graph interpretations. However, relations defined by other dimensions can interfere with the deployment of spatial attention. Thus, top-down control may be necessary to inhibit task-irrelevant comparisons; for instance, a person who can successfully ignore the contrast relation in the bottom graph of Fig. 1 may find it easier to judge the size configuration than a person who cannot inhibit the contrast relation. Further work is necessary to test whether selectively attending to relational structures is critical for both first-order (e.g., comparing one pair of bars) and higher-order comparisons in graphs (e.g., comparing two pairs of bars), and whether increased top-down control can improve conceptual reasoning about graphs.