The relationship between confidence and accuracy with verbal and verbal + numeric confidence scales

The relationship between confidence and accuracy with verbal and numeric confidence scales

The relationship between confidence and accuracy has long been debated within eyewitness research, but recently a resolution to the debate has occurred. Studies from the 1980s until relatively recently, using a point-biserial correlation technique (a correlation between the dichotomous accuracy variable and the corresponding confidence ratings), led to the conclusion that there was little to no relationship between confidence and accuracy (Kassin, Ellsworth, & Smith, 1989; Wells & Murray, 1984). However, Juslin, Olsson, and Winman (1996) argued that even when the point-biserial correlation was small, a strong confidence–accuracy (CA) relationship could still exist in the data. They computed a calibration plot using the formula C = # correct ID/(# correct ID + # incorrect ID) that includes all filler IDs from target-absent (TA) and target-present (TP) lineups and showed a strong CA relation even when the point-biserial correlation was low. Because fillers in TP lineups are known to be innocent, another way to compute calibration plots is to exclude these filler IDs. This approach, now called confidence–accuracy calibration characteristic plots (CAC; Mickes, 2015), provide measures of confidence (in bins) on the abscissa and measures of accuracy from low to high on the ordinate. Typically, these plots show that high confidence is associated with quite high accuracy (Mickes, 2015; Palmer, Brewer, Weber, & Nagesh, 2013; Sauer, Brewer, Zweck, & Weber, 2010; Wixted, Mickes, Clark, Gronlund, & Roediger, 2015; for a review, see Wixted & Wells, 2017). This outcome seems true of initial witness reports. If a witness has been repeatedly tested and has moved from a low confidence initial identification to a high confidence courtroom identification, then an error is likely. High confidence in the courtroom usually should not be weighed heavily (if at all) if the witness’s confidence at the first lineup was low.

Studies using CAC analysis have often employed numeric confidence scales (e.g. 20-point scales or 100-point scales), which are not reflective of eyewitness identification as conducted by police departments. Unlike laboratory studies, police departments usually have eyewitnesses verbally express their confidence or use a small range of confidence scales (perhaps highly confident or somewhat confident) instead of 100-point scales. Behrman and Richards (2005) reported that eyewitnesses typically used phrases such as “He resembles the guy” or “I think he did it” to indicate their certainty (or lack thereof). Wells (2014) reported that the Houston Police Department used a three-level verbal scale (positive, strong tentative, or weak tentative) for eyewitness identifications. Together, these police procedures raise an important question: Are such verbal confidence scales with few levels of confidence as predictive of eyewitness accuracy as the more fine-grained numeric scales used in laboratory studies? This is the issue we addressed in our paper, but we first review related findings.

Dodson and Dobolyi (2015a) showed that providing verbal or numeric labels and varying the number of confidence points on a 100-point scale (6 points: 0, 20, 40, 60, 80, 100; or 11 points: 0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100) did not change the CA relationship for eyewitness identification. Nonetheless, they employed a 100-point scale, which is unlikely to be used by police departments. Recently, Tekin and Roediger (2017) compared narrow ranges (4- and 5-point scales) to wider ranges (20- and 100-point scales) and concluded that the scale range did not affect the CA relationship with numeric scales. However, they used unrelated words and faces as materials, not lineups.

In the present study, we directly compared two- and four-level scales using lineups, because the four-level scale can be directly compared to the two-level scale by combining levels. The four-level scale may also be applicable in some police departments. Although previous research from our lab revealed that the 4-point and wider (e.g. 20-point, 100-point) scales did not differ from one another in the CA relation (Tekin & Roediger, 2017), no one has examined the issue with smaller scales (e.g. two- and four-level scales). Importantly, we used verbal confidence statements, as often used by police departments, and we also examined whether providing numerical values for the verbal confidence statements provided any benefits to the CA relationship compared to only verbal statements. In a quest for external validity, we did not compare verbal scales to just numeric scales because eyewitnesses are unlikely to give a numeric confidence without a verbal statement; thus, we added numbers to the verbal confidence scales to make them comparable. In addition, we employed two different material sets to establish some generalizability. Interestingly, one lineup turned out to be much more difficult for individuals than the other lineup and thus we can examine the effect of lineup difficulty on the CA relationship.

The current experiment addressed three primary questions. First, do small scales produce similar CA relationships (e.g. 2 values of confidence compared to 4 values), as is true for larger numeric scales (e.g. 20 points compared to 100 points)? Second, does adding numbers to purely verbal scales affect the CA relationship compared to using only verbal scales? Third, do the results replicate across two different sets of material (crime scenes and associated lineups)? To answer these questions, individuals viewed two videos, made identifications for possible suspects in each video with TP and TA lineups, and then indicated their confidence on either: (1) a verbal-only two-level scale; (2) a verbal + numeric 2-point scale; (3) a verbal-only four-level scale; or (4) a verbal + numeric 4-point scale.

Participants

To detect a small effect using chi-square tests, an a priori power analysis was conducted using a small effect size (φ = 0.1), an alpha of 0.05, df of 1, and a power of 0.80. Based on this, a sufficient number of observations was 785. We recruited 833 Amazon’s Mechanical Turk (MTurk) workers (mean age = 34.25, SD = 10.04) who were located in the United States and had high completion and performance rates (> 90%) in prior studies. Seven individuals were eliminated based on the questionnaire at the end of the experiment (four did not view the videos, three did not follow the instructions) and one due to experiment error. Of the remaining 825 individuals, 22 of them provided one observation and the rest provided two observations (one per lineup), totaling up to 1628 data points.¹

Design and material

The experiment employed a 2 × 2 × 2 mixed design. The scale range (two levels vs four levels) and the scale type (verbal-only vs verbal + numeric) were manipulated between-subjects, whereas the lineup type (TP vs TA) was manipulated within-subjects. Individuals viewed two videos and were later tested with a TP lineup for one of the videos and a TA lineup for the other. The order of the videos and the lineups were counterbalanced. Thus, every 16 participants constituted a complete experimental counterbalancing.

Two silent mock crime videos (Set A and Set B), in which a young, white male stole a laptop, were used as materials (Mickes, Flowe, & Wixted, 2012; L. Mickes, personal communication, September 12, 2017). Both videos were approximately 30 s long and the suspects showed their faces transiently to the camera for about 3–4 s in both videos.

TP and TA lineups for each video consisted of six black and white headshots and they were presented simultaneously in a matrix of two rows with three faces in each row. The positions of the headshots were randomized across subjects. The TP lineups included five fillers and the suspect whereas the TA lineups were composed of six fillers (see Fig. 1). The fillers were selected based on their resemblance (e.g. hair color, age) to the suspects. The TA lineup only included fillers.

The distractor tasks between the video and test phases of the experiment were questionnaires for openness to experience and conscientiousness that consisted of 20 short statements each (Costa & McCrae, 1992). The order of the questionnaires was counterbalanced across individuals.

Procedure

Participants were randomly assigned to one of the four between-subject conditions and were instructed that they would see videos and then later complete memory tests about the videos. After viewing the first video, they completed one of the personality questionnaires. Then they were given instructions about the lineup they were about to see and were asked to pick a suspect from the lineup, if he were there. Participants were informed that the suspect may or may not be present in the lineup. If the suspect was not in the lineup, they were told to select the “Not Present” option. They were then presented with either a TP or a TA lineup and were asked to make an identification decision. Immediately following their decision, they were asked to indicate their level of confidence on a confidence scale in accord with their assigned condition. Individuals who were assigned to the verbal-only two-level scale were presented with options of “not sure at all” and “absolutely sure” following their identification decision, whereas those who were assigned to the verbal-only four-level scale were presented with options of “not sure at all,” “somewhat sure,” “very sure,” and “absolutely sure.” For participants who were in the verbal + numeric condition, the labels also had a corresponding number next to them (e.g. “3-very sure”). Following their confidence judgments, individuals were asked to recall U.S. states for 1 min before proceeding to the second video, just to provide mental separation between the two parts of the experiment.

The second half of the experiment was identical to the first half except for the difference in the material, questionnaire, and lineup. Individuals viewed the second mock crime video and took the alternate personality questionnaire. Finally, they were given the same lineup instructions and were presented with either a TP or a TA lineup that was the same as the first type of lineup. They rated their confidence on the same scale as used for the first lineup, but if they had initially seen a TP lineup, they received a TA lineup the second time (and vice versa). The experiment lasted approximately 8 min.

CAC analyses

Figures 2 and 3 show CAC plots for scale ranges and scale types, respectively, for Set A and Set B sets separately. For CAC plots, we binned the lowest two ratings of the 4-point scale to compare them to the lowest rating of the 2-point scale (low confidence) and the highest two ratings of the 4-point scale to compare them to the highest rating of the 2-point scale (high confidence). Table 2 provides the frequencies of responses at each confidence level. For low and high confidence responses, accuracy (A) was computed using the formula A = # correct suspect IDs/(# correct suspect IDs + # incorrect suspect IDs/6), as recommended by Mickes (2015). First, replicating prior work, when people make a high confident decision (even on a 2-point scale), they are highly accurate; they were 94% correct for the Set A lineup and over 87% correct for Set B lineup.

Identification response: TP lineup
	Confidence level
4 points	1	2	3	4
Suspect ID	10	100	80	21
Filler ID	12	51	12	3
Non-ID	12	48	42	18
Identification response: TA lineup
	Confidence level
	1	2	3	4
Filler ID	22	110	37	8
Non-ID	16	100	78	38
Identification response: TP lineup
	Confidence level
2 points		1	2
Suspect ID		73	122
Filler ID		70	34
Non-ID		37	68
Identification response: TA lineup
	Confidence level
		1	2
Filler ID		123	73
Non-ID		68	142

Material set

We also tested whether the difficulty of material sets affected accuracy at different confidence levels. Figure 4 shows the CAC plots for the two material sets. For both low and high confidence, Set A led to higher accuracy levels than Set B (a 7% difference at the high level of confidence, with non-overlapping error bars). Thus, unlike scale range and type, material difficulty did affect accuracy levels even for high confidence responses. Still, even for the quite difficult Set B materials, high confidence indicated relatively high accuracy (87%).