When That Failed Replication Bling

 

One of my favorite uses of the Drake meme has disappeared from the internet, or at least is beyond my ability to find it. It went like this: next to the top image, in which Drake looks disgusted, it said "Polls are predictions." Next to the bottom image, in which Drake is endorsing, it said "Polls are snapshots."

It's a good way for people to reframe their thinking on opinion polls and to stop blaming Nate Silver for "being wrong." Polls just tell us how people generally felt at the time they were being asked the questions. By the time the poll is released, people may have already changed their mind. (It makes you wonder how many elections would be different if they had been delayed or moved ahead a week since people vote based on how they are feeling at the moment.)

I think about this meme when I think about the replication crisis. You know the story, some scientists tested subjects, made a big discovery, it gets cited thousands of times, someone tries to replicate the study and they find no correlation. The most famous example is probably the marshmallow test.

There is always a 20/20 hindsight perspective telling us about problems with the sample size, publication bias, or whatever. But sometimes there is nothing fundamentally wrong with the study.

Reading Robert Putnam's work has made me more aware of how much generations differ from one another. If people's values and mindsets are different depending on when they were born, is it too much to assume that those values will influence how they respond to their environment? And if this is possible, then maybe these studies that fail to replicate aren't bad, they're just a reflection of the subjects response at the time of the study.

In other words, maybe psychological experiments aren't explanations of universal human behavior. They are snapshots of human behavior at a particular point in time.