We have submitted the study described below to a replication attempt, and we invite you to read the description and then to predict whether the main finding replicated. We consider a finding to have replicated if the original result was statistically significant and our result was statistically significant and in the same direction, OR if the original result was statistically insignificant and our result was also statistically insignificant.

Replication of Study 2A from “Knowledge about others reduces one’s own sense of anonymity” in Nature

What is this Replication Project?

The project involves replications of randomly-selected, newly-published psychology papers in prestigious journals, with the overall aim to reward best practices and to shift incentives in social science toward more replicable science.

Note that we also rate papers on their transparency and on how unlikely they are to be misinterpreted by readers, but the focus here is only to describe the study in enough detail to allow people to predict the outcome.

How often have social science studies tended to replicate in the past?

In one historical project that attempted to replicate 100 experimental and correlation studies from 2008 in three important psychology journals, analysis indicated that they successfully replicated 40%, failed to replicate 30%, and the remaining 30% were inconclusive. (To put it another way, of the replications that were not inconclusive, 57% were successful replications.) 

In another project, researchers attempted to replicate all experimental social science science papers (that met basic inclusion criteria) published in Nature or Science (the two most prestigious general science journals) between 2010 and 2015. They found a statistically significant effect in the same direction as the original study for 62% (i.e., 13 out of 21) studies, and the effect sizes of the replications were, on average, about 50% of the original effect sizes. Replicability varied between 57% and 67% depending on the replicability indicator used.

Summary of this study

We subjected Study 2A from Knowledge about others reduces one’s own sense of anonymity, published in Nature, to a replication attempt. Our replication study (N = 475) examined whether people assigned a higher probability to another person detecting their lie if they were given information about that other person than if they were not. 

In the replication experiment, like in the original study:

1. Participants were asked to write 5 statements about themselves: 4 truths and 1 lie. They were told those statements would be shared with another person, who would then guess which one was the lie. 

2. Participants were either given 4 true statements about their ‘partner’ (Information Condition), or they were given no information about their ‘partner’ (No Information Condition). 

3. Participants were asked to assign a percentage chance describing how likely their ‘partner’ would be to detect their lie.*

*In our replication, prior to asking people to give their estimated percentage, we reminded participants that their 'partner' was shown their 5 statements and not told which were true. This was done in case participants had forgotten the conditions of the experiment. In the original study, this reminder had not been provided.

We collected data from 481 participants. We excluded 4 participants who were missing demographic data. We also excluded 2 participants who submitted nonsensical single word answers to the four truths and a lie prompt. Participants could not proceed in the experiment if they left any of those statements blank, but there was no automated check on the content of what was submitted. The authors of the original study did not remove any subjects from their analysis, but they recommended that we do this quality check in our replication.

In the original experiment and in our replication, participants were not actually connected to a ‘partner.’ They were informed about this fact after all participants had completed the experiment.

To test the main hypothesis, we used two-tailed independent samples t-tests. The main analysis asked whether participants in the Information Condition assigned a different probability to the chance of their ‘partner’ detecting their lie than participants in the No Information Condition. 

Summary of this study - in flowchart form

Our replication study is summarized in the diagram below. (Here’s a link to a higher-resolution version.)

If you'd like to see the above study description in the form of a Google Document instead, here's the link: https://docs.google.com/document/d/1wmXfOu17WZ0FnBGlCZay5zx9nVZ1yQaaIBpbbGPek70/edit?usp=sharing

Close date updated to 2022-08-02 11:59 pm

Close date updated to 2022-08-07 11:59 pm

Close date updated to 2022-08-14 11:59 pm

Close date updated to 2022-08-26 11:59 pm

Post-resolution update - here is a link to our report: https://replications.clearerthinking.org/replication-2022nature603/