Standard experimental paradigm designs and data exclusion practices in cognitive psychology can inadvertently introduce systematic “shadow” biases in participant samples

Cognitive Research: Principles and Implications

Table 1 Data from a subset of surveys completed by a large cohort of participants (see Appendix of Grady et al., 2022 for a full list of surveys) as part of a broader study (e.g., Grady et al., 2022; Silverman et al., 2022)

	Self-report survey	Excluded (N = 33) Mean (SD)	Included (N = 919) Mean (SD)	Mann–Whitney U value	p value	Rank biserial correlation (effect size)
Predict Excluded> included (1-tailed)	ADHD symptoms	55.33 (24.13)	44.12 (22.87)	19,147	0.005	0.263
	Autism symptoms	21.88 (4.96)	19.04 (5.77)	19,695.5	0.002	0.299
	Impulsivity	69.85 (9.82)	62.29 (10.72)	21,296	< 0.001	0.404
	Trait anxiety	2.48 (0.30)	2.38 (0.27)	18,374.5	0.019	0.212
Predict excluded< included (1-tailed)	Conscientiousness	3.34 (0.51)	3.49 (0.64)	12,460	0.041	− 0.178
	Maximization	4.53 (0.82)	4.61 (0.99)	13,885	0.205	− 0.084
	Responsibility	94.94 (23.78)	92.60 (22.00)	16,781	0.851	0.107
	Self-control	111.00 (14.95)	115.94 (19.15)	12,355	0.035	− 0.185
No prediction (2-tailed)	CCD symptoms	24.58 (12.94)	16.62 (11.54)	20,866.5	< 0.001	0.376

Data are separated into those who would be excluded in subsequent analyses based upon failing an “attention check” criterion (“Excluded”) and those that would have passed all inclusion criteria (“Included”). Reported statistics are from Mann–Whitney U tests, and all p values are 1-tailed except for OCD, as there was no a priori directional prediction for that survey
Bolded text represents significant values of p < 0.05