Mastering algebra retrains the visual system to perceive hierarchical structure in equations

Object-based attention in vision and reasoning

Our perceptual systems constantly construct and impose structure upon the observed environment. The visual system, for instance, imposes structure on sensory input by organizing the visual world into discrete objects (Wagemans et al., 2012). One facet of this structured, hierarchical visual processing is object-based attention, in which the visual world is organized into discrete objects, and attending to one part or feature of an object facilitates attention to the rest of the object (Kahneman & Henik, 1981; Kimchi, Yeshurun, & Cohen-Savransky, 2007; Vecera & Farah, 1994). Object-based attention is typically detected in experimental paradigms involving visual property verification. People are better at comparing visual properties (e.g., color) when elements are within a single visual object rather than distributed between objects (Duncan, 1984; Fig. 1a).

The construction of visual objects does not depend exclusively on sensory cues but is shaped also by experience-dependent expectations. For instance, Zemel, Behrmann, Mozer, and Bavelier (2002) had participants make a comparative judgment about features of objects in a visual scene, but added an occluding object to make it ambiguous whether the features belonged to a single, unusually shaped object or two separate objects. When participants had not observed the unusual shape previously, their responses were consistent with comparing features that belonged unambiguously to separate objects. This suggests that, in line with Gestalt principles, they had interpreted the two parts as belonging to distinct objects. But when participants had previous experience with objects with that unusual shape, they were faster to make the perceptual judgment, as if now they interpreted the two features as belonging to the same oddly shaped object. Thus, the visual system constructs objects based not only on sensory cues but also past perceptual experience.

Could mathematical expertise involve adapting object-based attention to perform algebraic reasoning? The rules of algebra—such as the rules governing operator precedence—impose a hierarchical structure that combines simple elements into more complex expressions (Fig. 1b). For instance, when constants and variables are multiplied together, they act as a unified grouping within the larger expression, known as a “term.” Individual terms are then added, subtracted, or combined in other ways to create even more complex expressions, much like words and phrases are combined to form complex sentences. Recognizing this hierarchical structure is critical to algebraic reasoning. When a complex algebraic expression is manipulated, valid manipulations maintain algebraic sub-expressions or act on them in systematic ways; invalid manipulations violate or ignore sub-expressions (Fig. 1c). For instance, given the expression ‘a • b + x • y,’ the rules of algebra license swapping the two algebraic sub-expressions, ‘x • y’ and ‘a • b,’ to get the new expression ‘x • y + a • b.’ By contrast, one cannot swap the two adjacent variables ‘b’ and ‘x’ to get ‘a • x + b • y.’ This manipulation violates the precedence rules for arithmetic operations. But to detect this violation, it suffices to notice that it breaks apart the two algebraic sub-expressions. Thus, if our visual system were retrained so that—in addition to constructing objects on the basis of sensory cues or experience-based expectations—it also imposed visual objects that were consistent with the requirements of formal mathematics, then attending to these algebraic sub-expressions would be one way for our perceptual systems to accomplish aspects of algebraic reasoning without recourse to abstract, symbolic mental representations. By perceiving algebraic elements that are closer together in a hierarchical structure as a unified, perceptual object, one could transform the conceptual task of verifying algebraic validity into the perceptual task of checking that transformations do not violate algebraic objects.

Current study

To investigate whether people competent in algebra impose perceptual objects on algebraic expressions, we adapted the property verification paradigm used previously to demonstrate object-based attention (e.g., Baylis & Driver, 1992; Zemel et al., 2002). Participants were first evaluated for mastery of the basic rules that govern the hierarchical structure of algebra (i.e., order of operations). They were then tested for object-based attention within algebraic expressions (e.g., w + a × c + f). On each trial, two adjacent variables changed color, from black to either blue or red, and participants had to determine whether these variables had the same color or different color. If visual objects are constructed based on the expression’s hierarchical structure, then color verification should be facilitated when performed within an algebraic sub-expression (i.e., variables separated by multiplication), compared to when performed between sub-expressions (i.e., variables separated by addition). Moreover, this within-object advantage should occur only among those participants who have mastered the rules that generate the hierarchical structure of algebra. To investigate whether retraining the visual system modulates algebraic performance, we also tested participants on a purely mathematical task: evaluating the algebraic equivalence of two expressions. If, after participants master the syntax of algebra, their visual system is retrained to play a functional role in algebraic reasoning, then object-based attention for algebraic sub-expressions should improve performance in algebraic reasoning.

Participants

Volunteer adults (N = 150, M _age = 20 years; 73 men, 71 women, 6 other gender) participated online in return for partial course credit. Sample size was determined in advance based on a pilot study (n = 30), using the same procedure as in the current study, which found object-based attention within algebraic sub-expressions (p <  .05), with evidence for this effect only among participants who had mastered the syntax of algebra (Bayes Factor BF₁₀ < 1 for participants who had not mastered algebraic syntax). Based on the effect size of the interaction in this pilot (η_p ² = .02), a sample size of n = 135 would have a power of .95 to detect the interaction between Algebraic Term and Syntax Knowledge (Faul, Erdfelder, Lang, & Buchner, 2007).

Materials

Expressions were displayed on a computer monitor in a monospaced, sans-serif, black font. They consisted of four variables separated by arithmetic operations, either multiplication or addition (Fig. 1c). The symbol for multiplication was created by rotating the addition symbol by 45°. On each trial, variables were represented by a random selection of unique letters from the Roman alphabet—excluding three letters that resemble numerals (i, l, and o) and one that resembled the multiplication symbol (x). Expressions had two possible formats: one with multiplication in the center and additions on the outside, and the other with addition in the center and multiplications on the outside. This assured that both arithmetic operations appeared equally in every position within the expressions.

On Algebraic Equivalence trials, initial expressions were joined by a rearranged version, which appeared to the right of an equals sign (Fig. 1c; see Procedure, below). Following Landy and Goldstone (2007a), this second expression was created by applying one of eight possible permutations to the first expression. Half of these permutations produced expressions that were equivalent algebraically to the original; the rest produced expressions that were not equivalent (Fig. 1c).

Procedure

Participants were first evaluated for their knowledge of the order of precedence for arithmetic operations (“Syntax Knowledge”). Two arithmetic problems with both addition and multiplication (e.g., 4 + 3 × 2 + 1) were followed by four alternatives. One alternative was the correct solution, obtained by performing multiplication before addition (e.g., 11). Other alternatives included the solution obtained if addition were performed before multiplication (e.g., 21) and the solution obtained if operations were completed from left to right (e.g., 15). Participants were considered “Syntax Knowers” if they answered both questions correctly, and “Non-Knowers” otherwise.

Participants then completed the main experimental trials, which involved one of two tasks, assigned randomly on each trial: Color Verification or Algebraic Equivalence. All trials began with the presentation of an algebraic expression, just left of the display’s midline. The Color Verification task was modeled after the paradigm used by Zemel et al. (2002) to study object-based attention in a purely visual context. On Color Verification trials, 3000 ms after the initial appearance of the algebraic expression, two adjacent variables changed color from black to blue or red (Fig. 1c). This cued participants to determine whether the colored variables were the same color (e.g., both red) or different colors (e.g., one red, one blue). Color Verification trials were used to measure object-based attention. On Algebraic Equivalence trials, the presentation of the initial algebraic expression was followed after 3000 ms by the appearance of a second expression, separated from the first expression by an equals sign (Fig. 1c). This cued participants to determine the algebraic equivalence of the left- and right-side expressions. On both Algebraic Equivalence and Color Verification trials, participants responded by pressing the ‘p’ (same/equivalent) or ‘q’ (different/non-equivalent) keys. Participants were instructed to respond as quickly and accurately as possible; they had up to 10 s to respond, and received immediate feedback after incorrect responses. They completed 336 trials ordered randomly over four blocks, each consisting of 12 Algebraic Equivalence and 72 Color Verification trials.

After completing the main experimental trials, participants reported their age and gender, and responded to a series of questions about their mathematical abilities: mathematics anxiety (from 1 to 10); whether they had completed a college course on finite mathematics (e.g., combinatorics); and their score on the quantitative section of the SAT (which very few participants remembered). No other measures were collected.

Analysis

For Color Verification trials, Signal Detection Theory (Green & Swets, 1966) was used to analyze perceptual sensitivity while controlling for potential response biases. Pilot results indicated that the effect of object-based attention, in this paradigm, was most pronounced in perceptual sensitivity rather than reaction time.² After removing trials where participants did not respond (<1%), discriminability (d’) was calculated for each participant, Algebraic Term (within vs. between algebraic sub-expressions), and Expression Format (either “v ₁  + v ₂  × v ₃  + v ₄ ” or “v ₁  × v ₂  + v ₃  × v ₄ ”). Since many participants had perfect discrimination in at least one condition (n = 93), 0.25 was added to all cells of the signal detection matrix to correct for infinite estimates of discriminability (Brown & White, 2005). The main results were confirmed by analyses of accuracy (see Appendix).

Analyses were conducted in the R software package (Core Team, 2015). Hierarchical (i.e., mixed-effects) models were fit with the lme4 package (Bates, Maechler, Bolker, & Walker, 2015). All predictors were centered. P-values for fixed effects were calculated using Satterthwaite approximations (Kuznetsova, Brockhoff, & Christensen, 2015). Participants were removed for below-chance performance on Algebraic Equivalence trials (n = 16) and for poor accuracy (<75%) on Color Verification trials (n = 10). Including all participants did not change the pattern or statistical significance of the main results.

Relations to algebra performance

We next investigated whether object-based attention during Color Verification trials predicted algebraic performance. If mathematically competent undergraduates rely on retrained object-based attention to parse algebraic expressions, then participants who exhibited a greater within-term advantage on Color Verification trials should be better at determining algebraic equivalence. We thus calculated, for each participant, a measure of object-based attention on Color Verification trials, by subtracting mean d’ on between-term comparisons, from mean d’ on within-term comparisons. This measure is more positive when discriminability is better for comparisons performed within (vs. between) algebraic term.

First, we verified that Syntax Knowledge facilitated performance on the Algebraic Equivalence task. As expected, participants who had mastered the rules governing order of operations (i.e., Syntax Knowers) were better at evaluating algebraic equivalence (M = 83.2% vs. 76.6%), t ₁₂₂ = –2.5, p = .01, Cohen’s d = 0.47. To confirm that Syntax Knowledge made a unique contribution to algebra performance, we analyzed trial-by-trial accuracy with a mixed-logit model that included additional fixed effects for available control measures: standardized mathematics anxiety; whether participants had completed college finite mathematics; and, to control for overall engagement, standardized mean accuracy on the Color Verification trials.³ The random effects structure was the maximal converging structure motivated by the design, with random effects of Subject and Equation Format, random intercepts, and all random slopes (Barr et al., 2013). Both anxiety (β = –0.22 ± 0.09 SEM, p = .016) and performance on Color Verification trials (β = 0.42 ± 0.09 SEM, p < .001) were significant predictors of accuracy on Algebraic Equivalence trials. Even after controlling for these factors, however, Syntax Knowledge still predicted algebra performance (b = 0.41 ± 0.20 SEM, p = .036).

We next investigated whether object-based attention also facilitated judgments of algebraic equivalence. To the full mixed-logit model of algebra accuracy, we added the measure of participants’ object-based attention, its interaction with Syntax Knowledge, and all associated random slopes. Once again, both anxiety and performance on Color Verification trials predicted algebra performance (both p < .01), as did Syntax Knowledge (p = .045). The only other significant predictor was the interaction between Syntax Knowledge and object-based attention (b = 1.2 ± 0.42 SEM, p = .006; Fig. 3). Follow-up subset analyses found that, while Syntax Non-Knowers were overall worse than Knowers at judging algebraic equivalence, their performance was unrelated to their object-based attention (p = .17). For the higher-performing Syntax Knowers, by contrast, object-based attention was a highly significant predictor of success in judging algebraic equivalence (b = 0.68 ± 0.26 SEM, p < .01). Thus, there was evidence that judging algebraic equivalence—a purely mathematical task—was supported by object-based visual attention, but only among those participants who had mastered the basic hierarchical structure of algebra (i.e., Syntax Knowers). Indeed, among Syntax Knowers, our measure of object-based attention accounted for nearly 10% of the variance in participants’ mean accuracy, even after controlling for mathematics education, mathematics anxiety, and overall task engagement.

The nature of mathematical expertise

The current results suggest that relying on visual processing might be a boon, not a barrier, to mathematical reasoning. This might come as a surprise. Confronted with evidence of students’ reliance on misleading, superficial visual strategies in algebra, some have argued that mathematical training should avoid and even suppress perceptual strategies (e.g., Kirshner, 1989; Kirshner & Awtry, 2004). For example, when asked to solve 4 + 4/2 + 2, some students might be led to answer “2,” incorrectly, because of the superficially tempting, perceptually strong 4 + 4 and 2 + 2 groups. Indeed, we sometimes found evidence for perceptual grouping around addition, rather than multiplication, particularly among participants who had yet to master the hierarchical syntax of algebra. But the fact that novices use perceptual strategies to arrive at incorrect answers does not imply that experts abandon such strategies entirely. Instead, experts may refine those perceptual strategies so that they become reliable, robust, and rapid routes to correct solutions (Goldstone et al., 2010; cf., Hutchins, 1995, and Rumelhart et al., 1986). In line with this, participants who had mastered the hierarchical syntax of algebra also exhibited object-based attention for algebraic sub-expressions. Mathematical expertise, therefore, might be better thought of as the skillful deployment of perception.

Thus, the mathematical expert is made more expert, on the one hand, by mastering clever notations in which conceptual relations are presented perceptually and, on the other, by retraining their visual system to perform some aspects of algebraic reasoning. Both this perspective on mathematical practice and its resistance have a long heritage. To quote Whitehead (1911, p. 61) yet again: “It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them.” The resistance continues to this day. New Mathematics was a relatively recent, and particularly controversial, movement in education that attempted to foreground the “important operations” of mathematics, at the expense of procedural mastery (Adler, 1972). But one implication of our perspective is that mathematical training might be better spent encouraging students to adapt—not abandon—their perceptual grouping processes. Instead of minimizing students’ reliance on perceptual strategies (Kirshner, 1989; Kirshner & Awtry, 2004), education should aim to refine students’ use of perception and action, so that they rig up their perception and action systems like mathematical experts. This could take the form of explicit instruction on how the visuospatial layout of algebraic equations contains hints to the hierarchical relations that they represent. Additionally, future curricula or tools could intervene in targeted ways on the embodied routines that contribute to mathematical expertise, taking advantage of decades of research on perceptual and motor learning (Ottmar & Landy, in press).

Regardless of what we do as teachers, children pick up on the perceptual regularities of their environments, implicitly developing perceptual associations and routines. These can become obstacles, such as when children interpret the visual form of the equals sign as a cue to calculate, hindering learning in early algebra (McNeil, 2008). But they can also offer long-term benefits, such as the perceptual strategy documented in the current study. We imagine a future where computer-based tools will systematically manipulate the visual and interactive features of mathematical representations so that children pick up on the perceptual regularities that help, rather than hurt (e.g., Weitnauer, Landy, & Ottmar, 2016).

Of course, perception alone is insufficient to account for all of mathematical reasoning. However, we suspect it is a critical part of the larger, distributed system that accomplishes mathematics, a system in which resources within the skull are brought into coordination with resources outside (e.g., gestures, inscriptions), skillfully soft-assembled to respond to the situated demands of the task (Clark, 2008). These sundry resources are often “embodied,” from neural circuits that evolved for perceiving and acting, to the fleshy hands that do the literal “manual labor” of mathematics (Marghetis, Edwards, & Núñez, 2014). For example, brain circuits that evolved for perceiving motion or shifting attention are redeployed to support mathematical skills like symbolic arithmetic, where attention is shifted along a simulated number-line (Knops, Thirion, Hubbard, Michel, & Dehaene, 2009; Marghetis, Núñez, & Bergen, 2014; McCrink, Dehaene, & Dehaene-Lambertz, 2007), or solving equations, where terms are imagined to move across the equals sign (Goldstone et al., 2010). Our bodies, too, are disciplined by mathematical training. When a mathematical expression is examined, eye movements respect the expression’s hierarchical structure, starting with the highest-precedence operation and moving sequentially to gradually lower-precedence operations (Landy, Jones, & Goldstone, 2008; Schneider, Maruyama, Dehaene, & Sigman, 2012). And while gestures can shape children’s early mathematical knowledge (Goldin-Meadow, Cook, & Mitchell, 2009), even experts gesture spontaneously to express their mathematical understanding (Marghetis & Núñez, 2013). A complete understanding of mathematical cognition requires that we study mathematics as it is actually accomplished, as an embodied practice: eyes darting across the blackboard, hands scribbling away.

The widespread role of regimented perception

While the current study has focused on retraining our perceptual apparatus to perform algebraic reasoning, mathematics is full of other practices that also likely depend on the regimentation of perception. Visual proofs in Euclidean geometry are unreliable when treated naively as exact depictions, but the expert geometer learns to ignore those diagrammatic features that could lead to invalid conclusions (e.g., exact length) while perceiving those features that can make valid contributions to a proof (e.g., containment; Manders, 2008). And this is not restricted to high school mathematics. Category Theory, a branch of modern mathematics, relies on a proof technique known as “diagram chasing” that relies entirely on the creation and interpretation of diagrams in which spatial locations indicate mathematical relations. Indeed, visuospatial ability is significantly greater among professional mathematicians compared to non-mathematicians, and it completely mediates the relation between basic numerical abilities and the attainment of advanced mathematical expertise (Sella, Sader, Lolliot, & Cohen Kadosh, 2016). Thus, while algebra has been our case study, we propose that mathematics more generally depends for its accomplishment on the cultural regimentation of our perceptual apparatus.

And this may be an even more general phenomenon, with regimented perception playing a role in the reproduction of many, if not most, sociocultural systems. Biases in face perception, for instance, may contribute to the reproduction of structural racism: implicit racial biases, which shape the perception of facial emotions (Hugenberg & Bodenhausen, 2003), can influence split-second decisions by law enforcement about whether or not to shoot a suspect (Correll, Park, Judd, & Wittenbrink, 2007), thus reproducing structural inequalities in safety and policing. Marx even argued that a similar process of regimented perception contributes to the reproduction of capitalist society as a whole, such that we learn to see the world in terms of objects to be owned (Marx, 2012). Thus, the cultivation of highly disciplined ways of seeing and acting may be a critical mechanism by which we reproduce immense sociocultural systems (Bourdieu, 1977), from structural inequality to the inferential structure of mathematics.