This experiment was pre-registered (https://aspredicted.org/bs5sh.pdf).
Method
Subjects
To boost precision, we aimed to recruit 300 Amazon Mechanical Turk workers who live in the USA. A total of 325 people participated, but 32 failed to complete the experiment. The resulting dataset comprises 293 people (103 women, 190 men; Mage = 36 years, age range: 21–69 years). According to a sensitivity analysis, this sample size gives us adequate power to detect a small interaction effect by conventional standards (f = 0.05).
Design
We again manipulated News Source within subjects and assigned subjects to one of three Political Identification groups based on responses to a political identification question.
Materials and procedure
The materials and procedure were identical to Experiments 1 and 2, except as follows.
We asked subjects to rate how familiar they were with each news source—presented in alphabetical order—on a scale from 1 (Not at all familiar) to 5 (Very familiar). This rating task followed immediately after the phase in which subjects provided real news, fake news, and propaganda ratings for the news sources.Footnote 5
Subjects then completed a Cognitive Reflection Test (CRT) as a measure of analytic thinking (Frederick 2005). This test comprises 3 questions that tend to elicit different answers when thinking relatively effortlessly versus effortfully. For example, one question asks: “A bat and a ball cost $1.10. The bat costs $1.00 more than the ball. How much does the ball cost?” The intuitive answer is 10 cents, but a more analytic approach reveals the correct answer of 5 cents. For each of the three CRT questions, subjects provided a numeric response.
Results and discussion
Beliefs about news sources
As in Experiments 1 and 2, we first examined the extent to which the three rating types subjects made were related to each other, collapsed across news sources. Table 2 presents the Pearson correlations for these three measures and their associated 95% CIs. As the table shows, real news ratings were positively associated with fake news and propaganda ratings, and fake news ratings were positively associated with propaganda ratings. In general, two of these three associations are markedly different from those reported in Experiments 1 and 2 and could reflect a shift over time in people’s beliefs about the type of information reported by our list of news sources.
We next classified subjects into three political groups (Left: n = 33; Center: n = 78; Right: n = 182). Before turning to our primary questions, we wondered how people’s ratings varied according to political identification, irrespective of news source. We display the three ratings—split by political identification—in the bottom panel of Fig. 2. As the figure shows, the results are both similar and different to our earlier samples. One-way ANOVAs were statistically significant for all three averaged ratings: Real news F(2, 292) = 11.19, p < 0.001, η2 = 0.07; Fake news F(2, 292) = 15.18, p < 0.001, η2 = 0.09; Propaganda F(2, 292) = 25.25, p < 0.001, η2 = 0.15. Follow-up Tukey comparisons showed that conservatives gave higher real news ratings than liberals or centrists (Right-Left Mdiff = 0.53, 95% CI [0.24, 0.81], t(213) = 4.37, p < 0.001, d = 0.715; Right-Center Mdiff = 0.23, 95% CI [0.03, 0.43], t(258) = 2.70, p = 0.020, d = 0.315); higher fake news ratings than liberals or centrists (Right-Left Mdiff = 0.65, 95% CI [0.27, 1.04], t(213) = 4.01, p < 0.001, d = 0.886; Right-Center Mdiff = 0.53, 95% CI [0.26, 0.80], t(258) = 4.55, p < 0.001, d = 0.719); and higher propaganda ratings than liberals or centrists (Right-Left Mdiff = 0.82, 95% CI [0.49, 1.15], t(213) = 5.87, p < 0.001, d = 1.110; Right-Center Mdiff = 0.52, 95% CI [0.28, 0.75], t(258) = 5.17, p < 0.001, d = 0.700). Together, these results are consistent with Experiments 1 and 2 in suggesting that—compared to their liberal and centrist counterparts—conservatives generally believe that the sources used in this study provide more fake news and more propaganda. What appears to have changed over time is that conservatives now also believe that these sources report more real news.
We now turn to our primary questions. First, to what extent does political affiliation affect which news sources people consider real news, fake news, or propaganda? To answer that question, we ran two-way ANOVAs on each of the three rating types, treating Political Identification as a between-subjects factor with three levels (Left, Center, Right) and News Source as a within-subject factor with 42 levels (i.e., Table 1). These analyses showed that the influence of political identification on subjects’ ratings differed across the sources—but only for real news ratings: F(2, 82) = 1.57, p < 0.001, η2 = 0.01, all other interaction effect p values > 0.280.
We again adopted the approach from Experiments 1 and 2 to investigate this interaction, displaying the largest 5 differences in Table 3. The table shows a partisan divide, with conservatives rating these news sources more highly as sources of real news than liberals. In addition, these differences are close to or greater than a value of 1, representing an entire category shift up or down the rating scale. Perhaps of note is that in comparison with the 2017 and 2018 data, none of these news sources are traditional, mainstream agencies.
Beliefs about “fake news”
Recall again our second primary question: To what extent does political identification affect how people interpret the term “fake news”? To answer that question, we again analyzed the responses subjects gave when asked what fake news and propaganda mean. We analyzed only those responses in which subjects offered a definition for either term (55%, n = 162). Note that the proportion of subjects who provided such definitions was lower than in Experiments 1 (95%) and 2 (88%). Upon closer examination, we found that several subjects had likely pasted definitions from an Internet search. In an exploratory analysis, we found a statistically significant difference in the likelihood that participants provided a pasted definition, based on Political Identification, χ2 (2, N = 162) = 7.66, p = 0.022. Specifically, conservatives (23%) were more likely than centrists (6%) to provide a pasted definition, χ2 (1, N = 138) = 7.29, p = 0.007, OR = 4.57, 95% CI [1.29, 16.20], all other p values > 0.256. Liberals fell between these extremes, with 13% providing a pasted definition. Because we were interested in subjects’ own definitions, we excluded these suspicious responses from analysis (n = 27).
We then followed the same analytic procedure as in Experiments 1 and 2. Table 4 displays these data. As the table shows, the proportions of subjects whose responses included the characteristics described in Experiment 1 were similar across political identification. Specifically, we did not replicate the finding from Experiment 1, wherein people who identified left were more likely to provide separate definitions for the terms than people who identified right, χ2 (1, N = 90) = 1.42, p = 0.233, all other p values > 0.063.
Additional exploratory analyses
We now turn to our additional exploratory analyses specific to this experiment. First, we examine the extent to which people’s reported familiarity with our news sources varies according to their political identification. Liberals and conservatives may be familiar with different sources, and we know that familiarity can act as a guide in determining what is true (Alter and Oppenheimer 2009). To examine this idea, we ran a two-way ANOVA on familiarity, treating Political Identification as a between-subjects factor with three levels (Left, Center, Right) and News Source as a within-subject factor with 42 levels (i.e., Table 1). This analysis showed that the influence of political identification on subjects’ familiarity ratings differed across the sources: F(2, 82) = 2.11, p < 0.001, η2 = 0.01. Closer inspection revealed that conservatives reported higher familiarity than liberals for most news sources, with centrists falling in-between (Fs range 6.62—23.27, MRight-Left range 0.62—1.39, all p values < 0.002). The exceptions—that is, where familiarity ratings were not meaningfully different across political identification—were the media giants: The BBC, CNN, Fox News, Google News, The Guardian, The New York Post, The New York Times, The Wall Street Journal, The Washington Post, Yahoo News, and CBS News.
We also predicted that familiarity with our news sources would be positively associated with real news ratings and negatively associated with fake news ratings. To test this idea, we calculated—for each news source—correlations between familiarity and real news ratings, and familiarity and fake news ratings. In line with our prediction, we found that familiarity was positively associated with real news ratings across all news sources: maximum rReal(292) = 0.48, 95% CI [0.39, 0.57]; minimum rReal(292) = 0.15, 95% CI [0.04, 0.26]. But in contrast with what we predicted, we found that familiarity was also positively associated with fake news ratings, for two out of every three news sources: maximum rFake(292) = 0.34, 95% CI [0.23, 0.44]; minimum rFake(292) = 0.12, 95% CI [0.01, 0.23]. Only one of the remaining 14 sources—CNN—was negatively correlated, rFake(292) = -0.15, 95% CI [-0.26, -0.03]; all other CIs crossed zero. Taken together, these exploratory results, while tentative, might suggest that familiarity with a news source leads to a bias in which people agree with any claim about that source.
Next, we examined how the tendency to think analytically influences people’s interpretations of news sources. We know from related work that people who think more analytically—regardless of political affiliation—are better able to discern real news headlines from fake news headlines (Pennycook and Rand 2019). We might therefore expect that some of our observed differences relate to the ability to think analytically. We calculated a CRT performance score for each subject ranging from 0 to 3, according to whether each subject gave correct (+ 1) or incorrect (+ 0) answers to the three CRT questions. Most of the sample answered zero questions correctly (67%, n = 196), 18% answered one correctly (n = 53), 11% answered two correctly (n = 31), and the remaining 4% answered all questions correctly (n = 13). We then compared CRT scores across political identification and found that liberals scored higher than centrists and conservatives, F(2, 292) = 4.52, p = 0.012, η2 = 0.03; Left-Center MDiff = 0.49, 95% CI [0.08, 0.90], p = 0.015, d = 0.58; Left–Right MDiff = 0.46, 95% CI [0.08, 0.83], p = 0.013, d = 0.54.
Next, we explored how the tendency to think analytically affected real news, fake news, and propaganda ratings of the various news sources. Specifically, we ran repeated-measures analyses of covariance (RM-ANCOVAs) on each rating type, treating news source as a within-subject factor and CRT score as a continuous covariate. For real and fake news ratings, we found that the influence of analytic thinking interacted with news sources: FReal(41, 251) = 2.60, p < 0.001, η2 = 0.01; FFake(41, 251) = 1.81, p = 0.003, η2 = 0.003. Closer inspection showed that higher scores on the CRT led to lower real news ratings for less reputable news sources, such as Infowars and Occupy Democrats: the 14 statistically significant Bs ranged from -0.29 to -0.14. Higher CRT scores also led to lower fake news ratings for highly reputable news sources, such as Reuters and the Associated Press: the 12 statistically significant Bs ranged from -0.28 to -0.16.Footnote 6 For propaganda ratings, however, we found only a main effect of the tendency to think analytically: FPropaganda(1, 292) = 9.80, p = 0.002, η2 = 0.03, B = -0.17. Together, these patterns of results suggest that the tendency to engage in critical thinking helps people differentiate between high- and low-quality news sources. Given the exploratory nature of these analyses, the skew of the CRT scores, and the relatively small pool of subjects who identified “Left,” we encourage cautious interpretation of these findings.