Thursday, December 26, 2013

Great discussion of framing effects replication

Joseph Simmons and Leif Nelson recently wrote up the results of their replication attempt of a framing effects experiment. The experiment was done by David Mandel, and included an attempted replication of Tversky and Kahneman's "Asian Disease Problem" framing effect experiment. Mandel changed the wording of the original slightly to test its robustness. He added the word "exactly" so as to rule out misinterpretation of the original wording, which could potentially be read as meaning "at least." Simmons and Nelson attempted to replicate Mandel's results, and found a notably different outcome than Mandel.

What I want to focus on is not the details of the discussion, as interesting as they are. For the details, I'd recommend reading Mandel's original paper, the Simmons/Nelson replication and Mandel's reply. Instead, I want to comment on some great features of the exchange.

First, the discussion was remarkably rapid. Mandel's replication study was published in August 2013. Simmons and Nelson responded with their own replication, posted on their blog, in December. Mandel responded in detail to their replication on his own blog within a few days of receiving the drafted post from Simmons/Nelson. It's great to be able to read not just what Simmons/Nelson found, but also Mandel's take on it, without long delays.

Second, not only was the discussion rapid, but it's also high quality. Both Simmons/Nelson and Mandel seemed to engage really closely with what their experiment results showed, and also made the discussion accessible to readers.

Third, the tone is great. Simmons/Nelson point out why Mandel's replication is important and worth replicating:
The original finding is foundational and the criticism is both novel and fundamental. We read Mandel’s paper because we care about the topic, and we replicated it because we cared about the outcome.
Then, Mandel makes it clear right away that he is taking their replication in a collegial way. He titles it "AT LEAST I REPLIED: Reply to Simmons & Nelson's Data Colada blog post." Making a joke in his title is a nice signal of the spirit in which he takes their replication; he then adds:
First, let me say that when Joe contacted me, he noted that his investigation with Leif was conducted entirely in the scientific spirit and not meant to prove me wrong. He said they planned to publish their results regardless of the outcome. I accept that that’s so, and my own comments are likewise intended in the spirit of scientific debate.
A barrier to doing and discussing replications is that current academic incentives can make the practice awkward and professionally unrewarding. Researchers who replicate might not be able to publish their work since it's often not seen as original enough, and the authors whose work is replicated may not welcome the efforts. While the former issue isn't something that this discussion bears on, since the posts appear on blogs, it clearly is relevant to the latter.

Having examples of thoughtful exchanges like this one is a nice demonstration of what replication can be. At its best, it is detailed and thoughtful work that helps us sort through which effects we can more confidently rely on.


  1. Interesting post Stephanie!

    For another good example of replication work that was constructive and remained civil between replicators and original authors, see

  2. Thanks, Etienne! That's definitely another great example.