We have submitted the study described below to a replication attempt. We aimed to replicate the two main results figures from the original study (shown in the diagram below) using multicategorical conditional process analysis. In the original diagrams, there are a total of 13 significant results along the “a” and “b” paths combined (the c’ path is not the focus here), plus four significant results relating to the effects of gender. This gives a total of 17 significant results in the parts of the diagrams that are most relevant to the authors’ hypotheses. We invite you to guess how many of those 17 significant results replicated - i.e., how many of them were also significant (and with effects in the same direction) in our study.

Replication of Study 3 from "Counterfeit Diversity: How Strategically Misrepresenting Gender Diversity Dampens Organizations’ Perceived Sincerity and Elevates Women’s Identity Threat Concerns" in JPSP

What is the Transparent Replications Project?

The project involves replications of randomly-selected, newly-published psychology papers in prestigious journals, with the overall aim to reward best practices and to shift incentives in social science toward more replicable science.

Note that we also rate papers on their transparency and on how unlikely they are to be misinterpreted by readers, but the focus of the text below is only to describe the study in enough detail to allow people to predict the outcome.

How often have social science studies tended to replicate in the past?

In one historical project that attempted to replicate 100 experimental and correlation studies from 2008 in three important psychology journals, analysis indicated that they successfully replicated 40%, failed to replicate 30%, and the remaining 30% of the findings were inconclusive. (To put it another way, of the replications that were not inconclusive, 57% were successful replications.) 

In another project, researchers attempted to replicate all experimental social science science papers (that met basic inclusion criteria) published in Nature or Science (the two most prestigious general science journals) between 2010 and 2015. They found a statistically significant effect in the same direction as the original study for 62% (i.e., 13 out of 21) studies, and the effect sizes of the replications were, on average, about 50% of the original effect sizes. Replicability varied between 57% and 67% depending on the replicability indicator used.

This replication study

The second paper that we submitted to our ratings and replication attempt procedures was randomly selected from issue 3 of volume 122 of JPSP. We randomly selected "Counterfeit Diversity: How Strategically Misrepresenting Gender Diversity Dampens Organizations’ Perceived Sincerity and Elevates Women’s Identity Threat Concerns". Within the set of studies in that paper, we selected Study 3 to be subjected to our replication attempt.

• The study assessed men’s and women’s beliefs about working for tech companies with different ratios of men and women (either 3:1 or 1:1) among their staff. Participants reacted to a hypothetical scenario in which they considered applying for, obtaining, then commencing a project management position in the tech industry.  

• The main claim from the original study was as follows: when a company has a male:female staff ratio of 3:1 (even if its promotional materials display an equal balance of men and women), it is perceived as not being sincerely interested in increasing gender diversity, and women (but not men) are more likely to have identity threat concerns about working there (e.g., concerns about being left out or stereotyped, or not having their contributions valued due to their gender); also, both men and women (but especially women) tend to be less interested in working for that organization. These effects are mediated by the perception that the company is not sincerely interested in increasing gender diversity.

• The authors focused on conditional process modeling in their discussion of the results, so we also focused on their conditional process modeling results. In the original paper, the authors created two figures to summarize those results, which were labeled as Figures 8 and 9. We aimed to reproduce Figures 8 and 9 from the original study using the data collected in our replication. 

Summary of this study - in flowchart form

Our replication study is summarized in the diagram below. (Here’s a link to a higher-resolution version.)

A quick summary of the original study

This study assessed men’s and women’s interest in and hypothetical reactions to working for tech companies with different male:female staff ratios (either 3:1 or 1:1). Participants were asked to imagine applying for, obtaining, then commencing a project management position in the tech industry. At the application stage, they were shown recruitment materials that contained images of male and female staff in either a 3:1 or a 1:1 ratio (depending on which condition they had been randomized to). 

Later, when participants imagined starting the project management role, they were told that the on-site (also known as “on-the-ground”) staff ratio that they witnessed on their first day at work revealed either a 3:1 or a 1:1 male:female staff ratio (again depending on which condition they had been randomized to). 

The researchers assessed the perceived sincerity of the organization by asking participants two questions about the perceived sincerity of the company’s interest in improving gender diversity. They assessed identity threat by averaging the responses from six questions that asked participants the degree to which they would be concerned about being left out or stereotyped, not respected, or not having their opinion or contributions valued due to their gender.

The researchers then used multicategorical conditional process modelling (explained below) to show that:

1. The perceived sincerity (of a company’s interest in increasing gender diversity) mediates the relationship between on-the-ground gender diversity and identify threat concerns - and this mediation relationship is moderated by participant gender; and

2. The perceived sincerity (of a company’s interest in increasing gender diversity) also mediates the relationship between on-the-ground diversity and company interest post-measurements - and this mediation relationship is also moderated by participant gender. 

What participation involved

To see what the study involved, you can preview it. In summary, once a given participant provided informed consent:

1. They were randomized into one of four different conditions. The four different conditions are listed in the next section. 

2. They were shown three company site images about a project manager position in the technology industry. The content of the images depended on the condition to which they were assigned. Some participants saw a company that looks “gender diverse,” with a 50:50 gender split. Others see a company that appears to have a 3:1 male:female staff ratio.

3. They were asked their level of interest in the project manager position at the company and were asked a series of questions about the images they reviewed. 

4. They were asked to imagine obtaining and starting the project manager role at the technology company. They were told about the ratio of men to women observed during their first day on the job. Depending on the condition to which they have been randomized, some participants were told the actual ratio of men to women observed on their first day was 1:1, while others were instead told the ratio of men to women was 3:1. 

5. They were again asked their level of interest in the project manager position at the company and were asked a series of questions about the gender ratio that they had just been told about.

6. Participants were also asked how “sincerely interested” in gender diversity the company seemed to be. They were then presented with a series of identity threat questions, an attention check, and a question about their gender.

Experimental conditions

As in the original experiment, participants were randomly assigned to one of four conditions, listed below (with a probability of 0.25 of going to any one of the four conditions). 

Condition 0 = Authentically Diverse: participants in this condition were…

1. Shown company site images with a 50:50 gender split (i.e., they see an equal number of men and women featured on the Harrison Technologies website)

2. Told that, on their first day on the job, there was again a 50:50 gender split  

Condition 1 = Aspirational Diversity: participants in this condition were…

1. Shown company site images with a 3:1 male:female gender ratio 

2. Told that the gender split on the ground on their first day was 3:1 men:women

3. Given a statement from top company executives stating that the company isn’t yet where it wants to be in terms of gender diversity, but that they’re working toward increasing gender diversity in the future

Condition 2 = Authentic NonDiversity: participants in this condition were…

1. Shown company site images with a 3:1 male:female gender ratio 

2. Told that the gender split on the ground on their first day was 3:1 men:women
   

Condition 3 = Counterfeit Diversity: participants in this condition were…

1. Shown company site images with a 50:50 gender split

2. Told that the gender split on the ground on their first day was 3:1 men:women 

Perceived sincerity

The authors defined perceived sincerity as the average of the responses to the following two questions:

• To what extent do you think Harrison Technologies is sincerely interested in increasing gender diversity in their workforce? (Rated from "Not at all sincere" - which was scored as 1, to "Extremely sincere" - which was scored as 5)

• How believable is Harrison Technologies' interest in increasing gender diversity in their workforce? (Rated from "Not at all believable" - which was scored as 1, to "Extremely believable" - which was scored as 5)

You can read further details about the study and the analysis methods here.

Close date updated to 2022-08-26 11:59 pm

Post-resolution update - here is a link to our report: https://replications.clearerthinking.org/replication-2022jpsp122-3/