This question will resolve as Yes if the main study result is statistically significant in our replication (at significance level of p < 0.05, with the effect in the same direction as the original results). Otherwise, it will resolve as No.

Original Study Results

In Study 2 of the original research paper, participants who were “endowed” with a bundle of 3 facts to learn (as shown in the Study Summary) were more likely to choose to learn that 3-fact bundle instead of learning a 4-fact bundle presented as an alternative option; in contrast, participants who weren’t “endowed” with either bundle and who could freely choose between them were more likely to choose to learn the 4-fact bundle.

Click here for a higher resolution version of the study image

The original study randomized participants into one of two conditions: endowed and nonendowed. In the endowed condition, participants were told that they were on course to learn a specific bundle of three facts and were then offered the option to learn a separate bundle of four facts instead. In the nonendowed condition, participants were simply offered a choice between learning a bundle of three or a separate bundle of four facts, with the bundles shown in randomized order.

Results of a chi-square goodness-of-fit test indicated that participants in the endowed condition were more likely to express a preference for learning three (versus four) facts than participants in the nonendowed condition. This supported the original researchers’ hypothesis that individuals exhibit the endowment effect for non-instrumental information.

In our study, we followed the same procedure, except that we distinguished between the half of the nonendowed condition where the 3-fact bundle was displayed on top and the half where the 4-fact bundle was displayed on top (these were represented by Conditions 2 and 3 respectively in the diagram above), so that we could conduct an additional analysis in case order effects had been contributing to the original study results (explained below). For the purposes of assessing whether the original study replicated in our dataset, however, Conditions 2 and 3 were pooled into one combined nonendowed condition. This means that our pooled Conditions 2 and 3 are together equivalent to (a larger version of) the original study’s nonendowed condition. (For details, see The Details section.)

None of the facts presented were of objectively greater utility or interest than any of the others. Facts related to, for example, the behavior of a particular animal, or the fact that the unicorn is the national animal of a country. Furthermore, each time we ran the experiment, we randomized which facts appeared in which order across both bundles. The subjective utility of a given fact would not be expected to affect experimental results due to this randomization process.

The original study included 146 adult participants from Prolific. Our replication included 631 adult participants (not counting exclusions) from MTurk via Positly.com.

The Details

In the original experiment, two variables had varied across conditions - both endowment and the order of presentation of the two bundles had varied. Option order had been randomized within the endowed condition such that a 3-fact bundle was shown on top half the time while a 4-fact bundle was shown on top the other half of the time within that condition. On the other hand, option order was not randomized in the endowed condition: the 3-fact bundle was always shown on top within that condition. To control for ordering effects, we increased sample size to 1.5 times our original planned size and split the nonendowed condition (now double the size it would otherwise have been) into two separate conditions: Conditions 2 and 3. Our participants were randomized into one of three conditions, as described below:

• Condition 1: Endowed - Participants were told that they were on course to learn a specific bundle of three facts and were then offered the option to learn four different facts instead.

• Condition 2: Nonendowed with 3-fact bundle displayed on top - Participants were offered a choice between learning three facts or four facts, with the bundle of 3 facts appearing as the top option.

• Condition 3: Nonendowed with 4-fact bundle displayed on top - Participants were offered a choice between learning three facts or four facts, with the bundle of 4 facts appearing as the top option

Analysis

To evaluate the replicability of the original study, we ran a chi-square goodness-of-fit test to evaluate differences in preference for learning three facts between participants in the endowed versus the pooled nonendowed conditions. As stated in our pre-registration, our policy was to consider the study to have replicated if this test yielded a statistically significant result, with the difference in the same direction as the original finding (i.e., with a higher proportion of participants selecting the 3-fact bundle in the endowed compared to the pooled nonendowed conditions).

The Transparent Replications Project by Clearer Thinking aims to replicate studies from randomly-selected, newly-published papers in prestigious psychology journals, as well as any psychology papers recently published in Nature or Science involving online participants.

Context: How often have social science studies tended to replicate in the past?

In one historical project that attempted to replicate 100 experimental and correlation studies from 2008 in three important psychology journals, analysis indicated that they successfully replicated 40%, failed to replicate 30%, and the remaining 30% were inconclusive. (To put it another way, of the replications that were not inconclusive, 57% were successful replications.)

In another project, researchers attempted to replicate all experimental social science science papers (that met basic inclusion criteria) published in Nature or Science (the two most prestigious general science journals) between 2010 and 2015. They found a statistically significant effect in the same direction as the original study for 62% (i.e., 13 out of 21) studies, and the effect sizes of the replications were, on average, about 50% of the original effect sizes. Replicability varied between 57% and 67% depending on the replicability indicator used.

The replication described here was run as part of the Transparent Replications project, which has not run enough replications yet for us to give any base replication rates. Having said that, if you’re interested in reading more about the project, you can read more here. And here is where you can find write-ups for the eight previous replications we’ve completed.