Many children fail to achieve proficiency in math. In the United States, only 40% of students in grade four, 34% of students in grade eight, and 25% of students in grade 12 were above proficient in mathematics at both public and private schools in 2017 (National Assessment of Educational Progress: National Center for Education Statistics (NCES), 2017). Several factors likely contribute to students’ difficulties in acquiring expected levels of math proficiency; here we focus on two factors that have previously been shown to influence math learning: instruction that includes hand gesture and visuospatial, verbal, and kinesthetic WMC in the learners.
Working memory and math learning
Working memory processes are known to support math learning (e.g. Alloway & Alloway, 2010). Baddeley developed a modality-specific model of working memory, with two modality specific subsystems, a phonological loop, which stores linguistic information, and a visuospatial sketchpad, which stores visual and spatial information (Baddeley & Hitch, 1974; Baddeley, 2003; although see Engle, 2002 and Cowan, 1999 for alternative accounts of working memory).
Modality-specific working memory capacities are related to math learning. The capacity of verbal working memory predicts various types of mathematical success, across time and across development (Alloway & Alloway, 2010; Gathercole, Pickering, Knight, & Stegmann, 2004; Noël, Seron, & Trovarelli, 2004; Passolunghi, Vercelloni, & Schadee, 2007). For example, children’s verbal working memory skills at age five are correlated with their academic achievement in mathematics 6 years later (Alloway & Alloway, 2010). Visuospatial working memory also predicts mathematical success (Kyttälä, 2008; van der Ven, van der Maas, Straatemeier, & Jansen, 2013). Further, some studies have measured both verbal and visuospatial WMC and find that both predict success in mathematics (Jarvis & Gathercole, 2003; Swanson & Beebe-Frankenberger, 2004), although this pattern is not always seen (St. Clair-Thompson & Gathercole, 2006).
Spatial ability is also a significant predictor of mathematical ability in both children and adults (Cheng & Mix, 2014; Lubinski, 2010; Wai, Lubinski, & Benbow, 2009). Measures of spatial ability most often encompass spatial visualization and spatial reasoning, and it is possible that visuospatial working memory and spatial abilities overlap as cognitive constructs (Miyake, Friedman, Rettinger, Shah, & Hegarty, 2001).
Gesture and math learning
Although WMC is considered to be a fixed capacity, characteristics of instruction can influence how working memory resources are used during learning (Paas, Renkl, & Sweller, 2004). According to cognitive load theory, instruction that increases the capacity of learners to hold relevant information in working memory should increase learning (Sweller, 2010). One hypothesis is that multimodal instruction facilitates learning by allowing learners to efficiently use available working memory resources (Mayer, 2005; Mayer & Moreno, 2003; Mousavi, Low, & Sweller, 1995).
Indeed, multimodal instruction that includes visual hand gestures along with auditory speech has been shown to improve mathematical learning. Gestures are hand movements that spontaneously accompany speech, that are related to speech both semantically and temporally, and that do not serve any other known function. There are several different types of hand gestures that emerge across development and that are tightly coupled with language (see Capone & McGregor, 2004 for a review). Here we focus mostly on deictic gestures, which index specific objects or items in the environment and are often used to direct attention to the item being referenced (Bangerter & Louwerse, 2005). For example, in the current study, a math instructor simultaneously points at and audibly names each element of a mathematical equation as she explains a procedure for solving the problem. These sorts of gestures are frequent in math instruction (Alibali et al., 2014; Flevares & Perry, 2001).
Observing gesture during instruction enhances math learning, for adults as well as for children (e.g. Cook et al., 2013; Cook et al., 2016; Hendrix, Fenn, & Cook, 2018; Ping & Goldin-Meadow, 2008; Valenzeno, Alibali, & Klatzky, 2003). Furthermore, the beneficial effect of the observation of gesture on performance is pronounced for more challenging problem-solving tasks (Hou & So, 2017).
The mechanism by which gesture increases learning is not known. From the perspective of cognitive load theory, there are two possibilities for how multimodal instruction might reduce demand on working memory. One possibility is that multimodal instruction, because it involves separate systems for each modality, allows learners to capitalize on working memory resources that would otherwise not be recruited. On this account, gesture might provide a mechanism for involving visuospatial and kinesthetic working memory in learning (Wu & Coulson, 2007, 2011) that would otherwise be subserved only by verbal working memory. An alternative is that multimodal instruction might improve the efficiency with which information is encoded without necessarily recruiting additional resources. Gestures allow instructors to direct attention to the features of the problem. By providing a tool for signaling relevant information during instruction (Mayer, 2002), gestures may enhance encoding of relevant information into working memory.
These two accounts make different predictions about which learners should benefit most from instruction that includes gesture. If gestures recruit additional resources, perhaps visuospatial resources, then gesture should support learners with greater visuospatial WMC by allowing these learners to capitalize on their greater resources. Alternatively, if gestures facilitate encoding, gesture should enhance learning for learners with lower relevant WMC, by allowing these learners to make better use of their limited resources.
Understanding the working memory resources that support learning from gesture has implications for identifying the mechanisms underlying gesture processing more generally. Although gestures co-occur with language, they represent information visually and spatially and so they may engage spatial processing during linguistic communication. Indeed, some theories of gesture suggest that they are particularly important during spatial communication (Hostetter & Alibali, 2007) while others emphasize the relationship between gesture and language (McNeill, 1992; Rowe & Goldin-Meadow, 2009). If gestures are coded spatially, we would expect gesture processing to be related to spatial WMC (. Alternatively, if gestures function to influence the processing of concurrent linguistic information, we would expect gesture processing to be related to verbal WMC.
The available evidence suggests that sensitivity to gestures may depend on available visuospatial and/or kinesthetic working memory resources. For example, in a priming task that included both speech and gesture, individuals with higher spatial WMC showed more sensitivity to information in gesture, while those with higher verbal WMC showed more sensitivity to information in speech (Özer & Göksun, 2019). Similarly, Wu and Coulson (2014b) examined the role of verbal and visuospatial WMC in gesture comprehension and found that individuals with high visuospatial WMC were more sensitive to gesture than those with low visuospatial WMC. In a separate study, these researchers examined the relationship between performance on a gesture comprehension task and kinesthetic working memory, a novel subsystem of working memory which is postulated to be responsible for the storage and manipulation of bodily movements (Wu & Coulson, 2015). Individuals with high kinesthetic WMC were more sensitive to gestures and were better able to inhibit information from irrelevant gestures (Wu & Coulson, 2015). Together, these findings suggest that, in perception, sensitivity to gesture may depend on available visuospatial and/or kinesthetic resources, benefitting those with high capacity in relevant modalities.
However, the resources involved in learning from gesture may be distinct from those involved in understanding through gesture. In this article, we examine the relationships between math learning with gesture present or absent at instruction, visuospatial WMC, verbal WMC, and kinesthetic WMC. In Study 1, we examined performance on an abstract mathematical task with gesture present at instruction. We used only a single instructional condition in order to increase power to detect relationships between WMC and learning. After finding such relationships in Study 1, in Study 2, we examined performance on an abstract mathematical task with gesture absent at instruction using a new sample of participants. We then combined findings from Study 1 and Study 2 to compare patterns across instructional conditions.
Predictions
The findings from Wu and Coulson (2014b, 2015) and from Özer and Göksun (2019) reveal that gestures load on visuospatial and kinesthetic WMC. However, these studies investigated action words and discourse processing, not learning. If these findings generalize to instructional contexts, we would expect the availability of gesture during instruction might allow learners to use visuospatial and kinesthetic WMC that would otherwise not be engaged during instruction. If so, then adding gesture to instruction should improve learning for individuals with high visuospatial and kinesthetic WMC.
To test these predictions, Study 1 examined the relationship between individual differences in verbal, visuospatial, and kinesthetic WMC and learning when instruction includes gesture. Participants watched video instruction on a new mathematical system where the instructor used both speech and gesture. Following instruction, participants completed a posttest and a transfer test to assess learning. They then completed a visuospatial working memory task, a verbal working memory task, and a kinesthetic working memory task. Finally, participants completed an abbreviated mathematical anxiety rating scale and gesture attitudes questionnaire. Our goal in Study 1 was to assess how learning a novel mathematical concept with both speech and gesture at instruction relate to visuospatial, verbal, and kinesthetic WMC.
Study 1
The objective of Study 1 was to examine the relationship between individual differences in visuospatial, verbal, and kinesthetic WMCs and learning mathematical equivalence with gesture. Approval was obtained from the Institutional Review Board prior to data collection.
Participants
Seventy-five University of Iowa undergraduates participated in the study. Eleven participants were excluded from the final analyses. Participants were excluded for being non-native English speakers (n = 2), technical errors (n = 3), for not performing above chance in the learning task (n = 2), or because they did not have available ACT scores (n = 4). Thus, only 64 native English speakers were included in the analyses (35 male, 29 female). We determined this sample size a priori, based on simulations of our design and assumptions about effect sizes derived from prior research (Marstaller & Burianová, 2013; Wu & Coulson, 2015). We also preregistered our analytic approach with the Open Science Framework based on our expectations. Participants received course credit for participation.
Materials
Abstract Mathematical Equivalence Task
To assess learning with gesture, we used an abstract Mathematical Equivalence Task modified from Hendrix et al., 2018 (originally adapted from Kaminski, Sloutsky, & Heckler, 2008). This is a completely novel mathematical task, created for studying math learning in the laboratory. The task follows a system of modular arithmetic and requires students to learn to solve problems in a commutative group of order three, a mathematical system operating over shapes (diamond, circle, and squiggle) (see Fig. 1). We did not include a pretest because participants had no prior experience with the stimuli and the rules for combining them in our abstract math system. As such, they had no knowledge of the meaning of the symbols or the ways in which to combine them.
Participants first learned six rules for combining the three shapes in this novel mathematical system. Rules were presented one at a time in written format on a computer screen. All six rules are displayed in Fig. 1a. Participants had an unlimited amount of time to read each rule and chose when to proceed. After each rule was presented, participants answered one or two practice problems before seeing the next rule. These practice problems all tested simple equations that required participants to combine two shapes and calculate the result. These problems tested participants’ understanding of the preceding rule. Participants received feedback if they selected an incorrect response and were required to repeat the question until they selected the correct response. After viewing the rules and solving the practice problems, participants were shown all six rules together (Fig. 1a) and had as much time as needed to read and reflect on the six rules before proceeding to the instructional videos.
Participants then viewed six video-recorded explanations that described how to solve more complex problems in the symbol system. The problems used during instruction were based on math problems used to study the concept of equivalence in younger children and required participants to apply the rules to problems with five symbols; there were three symbols on the left side of the problem and one symbol and one blank space (a missing symbol) on the right sides of the problem. The instructor explained how to solve the problem to find the symbol that belonged in the blank space. The videos included both speech and gesture and ranged from 13 to 33 seconds. All gestures used in the instructional videos were deictic gestures; the instructor in the video pointed to the shapes in the equation as she elaborated mathematical equivalence problem-solving strategies (see Fig. 1b). In each video, the instructor points to each shape and explains how the shapes combine with each other as well as what shape each side of the question reduces to. She then gives the answer for which shape belongs in the blank space (access https://osf.io/wh92e/ for instructional videos).
After each instructional video, participants solved a practice problem, similar in form to the problem presented in the video. Participants could not move forward until they selected the correct response. Following instruction, participants were given a posttest (27 questions) and a transfer test (12 questions) to assess learning and generalization. All of the problems on the posttest and transfer test were novel; these problems did not appear during training. All posttest and transfer test problems were scored as correct or incorrect, or 1 or 0, respectively. The transfer task followed a similar mathematical structure as the abstract math task, but it used different objects, requiring participants to generalize their knowledge beyond the learning context (see Fig. 1d). Unlike in the posttest, however, during the transfer test, the rules for combining symbols in this new system were visible during problem solving on each trial, so participants did not need to keep this information in memory. The top portion of the image in Fig. 1d appeared on each transfer test problem. Cronbach’s alpha for the posttest was 0.86 and for the transfer test was 0.74. Thus, both tests demonstrated sufficient internal consistency to serve as individual measures of learning.
Visual Patterns Task
To assess visuospatial WMC, we used an adaptation of the Visual Patterns Task (Chu & Kita, 2011; adapted from Della Sala, Gray, Baddeley, & Wilson, 1997). Test-retest reliability of the Visual Patterns Task is 0.75 (Della Sala et al., 1997).
Participants viewed patterns of white and black blocks, presented for 3 seconds each. Immediately after the presentation of the pattern, the patterns of blocks were replaced with letters in every block, and participants were prompted to verbally recall the letters corresponding to the black blocks that were previously shown. Participants did not need to remember these letters, as the letters were visible throughout recall, serving to provide an efficient way of referencing the spatial locations in the grid. Spans ranged from seven to eleven black blocks with five trials at each level and an equal number of white and black blocks at each level. Responses were video recorded and scored online and offline.
Sentence Span Task
To assess verbal WMC, we used an adaptation of the Reading Span Task (Waters, Caplan, & Hildebrandt, 1987). The Reading Span Task is considered the standard task for assessing verbal WMC. Test-retest reliability composite Z-score measures were calculated separately for cleft subject sentence (rz = 0.75) and for subject-object sentences (rz = 0.83), demonstrating high test-retest reliability (Waters & Caplan, 1996).
Participants viewed a series of sentences and made judgments about each sentence. At the end of each trial (two to eight sentences), participants were prompted to verbally recall the last word of each sentence in the order that they had been shown. Spans ranged from two to eight sentences with five trials at each level. Responses were video recorded and scored online and offline.
Movement Span Task
To assess kinesthetic working memory, we used an adaptation of the Movement Span Task that has previously been related to sensitivity to information from gesture (Wu & Coulson, 2014a; Wu & Coulson, 2015). Analyses demonstrate a high reliability estimate of α = 0 .95 (Wu & Coulson, 2014a).
Participants viewed a series of hand and arm movements presented on video. At the end of each trial, participants were prompted to replicate the movements with as much detail as possible. Spans ranged from one to five movements with three trials at each level. Responses were video recorded, and movements were coded and scored offline. Single points were given for each movement that was replicated correctly, half points were given for each movement that reflected the target movement, but had slight deviations, and no points were given to movements that did not reflect any movements within a span. Points did not depend on the order in which participants recalled each movement within a span. Because we implemented a specific coding system for the movements, we assessed reliability; the intercoder agreement for scoring kinesthetic working memory span was 99%.
Composite ACT Score
Composite ACT score was obtained from university records and was used to control for cognitive ability. Performance on the ACT correlates highly with independent measures of general intelligence (Koenig, Frey, & Detterman, 2008). The median reliability estimate for the composite ACT score is 0.97. We expected that composite ACT score would positively predict learning in our novel math learning task.
Abbreviated Math Anxiety Rating Scale (A-MARS)
The A-MARS was included at the end of the study as an exploratory measure of mathematical anxiety (Alexander & Martray, 1989). The A-MARS is a 25-item questionnaire with a 5-point Likert scale ranging from “Not At All” to “Very Much”; participants were asked to indicate their level of anxiety in mathematical-relevant scenarios. Analyses demonstrate a high reliability coefficient of α = 0 .98.
Gesture Attitudes Questionnaire
A Gesture Attitudes Questionnaire was included at the end of the study as an exploratory measure (Nathan, Yeo, Boncoddo, Hostetter, & Alibali, in press). The Gesture Attitudes Questionnaire is a 16-item questionnaire with a 5-point Likert scale ranging from “Strongly Disagree” to “Strongly Agree”; participants were asked to indicate their level of agreement to specific statements about the function of gesture during communication.
Procedure
Participants were run individually, and the experimenter was in the testing room throughout the session. The study was conducted in a fixed order; all participants completed each task in the order in which they were described previously and completed a short participant information questionnaire at the end of the study.