At Transparent Replications, we conduct replications of recent human psychology and human behavior papers published in prominent journals.

How we choose studies to replicate

The diagram shows how we determine which papers are candidates for replication by our team.

Once we determine that a paper is a candidate for replication, we use the following method to determine which replications to conduct.

• First, if there is a candidate paper from Nature or Science, we prioritize that paper for a replication study. Our aim is to replicate every new paper from Nature and Science that meets our constraints.
• Once all of those are underway or completed, we alternate between the other three journals – choosing a paper at random from the pool of recent candidate papers from that journal.
• For each journal, we draw one paper at random out of the six most recent eligible papers from that journal. Once a batch of six has been used for a drawing, the papers in that set are not eligible to be drawn from again. This process is designed to result in an approximately equal chance of each paper being selected for replication. 
• If there are multiple studies in a selected paper that meet our criteria for replicability, we select a study from within a paper by prioritizing studies that are:
  • More directly connected to testing a central hypotheses of the paper
  • More developed or confirmatory, as opposed to preliminary or exploratory
  • More exactly reproducible by our team
  • More unexpected in their findings (rather than findings that would match a typical person’s common sense predictions)
• Unlike our criteria for putting papers into our replication pool or selecting which papers to replicate from the pool, the choice of an individual study from a selected paper is a judgment call. Our overall aim is to select the study (within a given paper) where we think our replication effort can add the most value. 

As we proceed with this project, we expect that we’ll learn things that may cause us to modify our process, and when that happens, we plan to document those changes and update this article to reflect those changes.

How we conduct replications

Once we have selected a study to replicate, we develop the materials for running the study, and share them with the original paper authors to get their feedback on how closely our materials match their original study.

It is extremely important to us that our replications accurately reflect the original study. Hence, we revise our materials using feedback from the original authors. In the event that the original research team does not reply to our requests for them to give feedback on the replication materials, we will, after giving them a reasonable time to respond, proceed with the replication study regardless. 

Before launching the study, we draft a pre-registration of the study and analysis plan, and the study materials and plan are reviewed by the independent Ethics Evaluator. After the study passes ethics review, we file the finalized pre-registration of our data collection and analysis plan.

Next, we collect the data, analyze the data, and write a draft of our replication report.

We share our draft report with the original research team to give them the opportunity to raise any issues that they see (e.g., to tell us if they think there are any mistakes in our analysis), and to write a response that we will then publish along with our report.

Finally, we publish our report to the Transparent Replications website. 

We welcome feedback on our published reports. We may share particularly valuable feedback on the report page (with credit and permission), or may amend a report on the basis of feedback we receive. As we continue with this project, we expect that our processes will evolve based on our growing experience and feedback from the community, and that the procedures outlined in this article will change accordingly.

Learn more about us by reading about our mission, checking out the frequently asked questions or meeting the team.