Seeing What They Want to See

Sometimes it takes a fresh eye (or a bleary-eyed grad student) to catch what others have missed. Jonah Lehrer, Mind Hacks, and Neurocritic all blog about a paper by Edward Vul and others revealing some troubling exaggerations of correlations in social neuroscience imaging. Hopefully this paper will get some traction and cause the methodology in question to be reexamined–that’s how science progresses, after all.

The paper is available online (PDF) and the abstract is here:

The newly emerging field of Social Neuroscience has drawn much attention in recent years, with high-profile studies frequently reporting extremely high (e.g., >.8)
correlations between behavioral and self-report measures of personality or emotion and
measures of brain activation obtained using fMRI. We show that these correlations often
exceed what is statistically possible assuming the (evidently rather limited) reliability of
both fMRI and personality/emotion measures. The implausibly high correlations are all
the more puzzling because social-neuroscience method sections rarely contain sufficient
detail to ascertain how these correlations were obtained. We surveyed authors of 54
articles that reported findings of this kind to determine the details of their analyses. More
than half acknowledged using a strategy that computes separate correlations for
individual voxels, and reports means of just the subset of voxels exceeding chosen
thresholds. We show how this non-independent analysis grossly inflates correlations,
while yielding reassuring-looking scattergrams. This analysis technique was used to
obtain the vast majority of the implausibly high correlations in our survey sample. In
addition, we argue that other analysis problems likely created entirely spurious correlations in some cases. We outline how the data from these studies could be
reanalyzed with unbiased methods to provide the field with accurate estimates of the
correlations in question. We urge authors to perform such reanalyses and to correct the
scientific record.


2 Responses to Seeing What They Want to See

  1. It turns out that the article claiming to identify “voodoo” correlations itself had deep flaws in its statistical reasoning and sampling methodology. See the rebuttal (that will be published alongside the voodoo paper) here:

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: