At Transparent Replications, we conduct replications of recent human psychology and human behavior papers published in prominent journals.

How we choose studies to replicate

The diagram shows how we determine which papers are candidates for replication by our team.

When we are ready to begin a new replication, we use the following method to determine which replications to conduct (note: this process was updated in August, 2025)¹.

• First, if there are candidate papers from Nature or Science published in the past 4 months, we prioritize whichever paper was published most recently for a replication study. Our aim is to replicate as many papers meeting our constraints from Nature and Science as possible.
• When there is no candidate paper from Nature or Science published within the past 4 months, we rotate through the other three journals – choosing the most recently published candidate paper from whichever journal is up next in our rotation. If there are multiple feasible papers published on the same date, we randomly select among those papers.
• Note that some papers are published “online first,” meaning they are published on the journal’s website before being included in an official volume and issue of that journal. Because our aim is to conduct replications soon after papers are published, we include “online first” papers during our selection process, and consider the date a paper was released as “online first” to be its publication date.
• If there are multiple studies in a selected paper that meet our criteria for replicability, we select a study from within a paper by prioritizing studies that are:
  • More directly connected to testing a central hypothesis of the paper
  • More developed or confirmatory, as opposed to preliminary or exploratory
  • More exactly reproducible by our team
  • More unexpected in their findings (rather than findings that would match a typical person’s common sense predictions)
• Ultimately, the choice of an individual study from a selected paper is a judgment call. Our overall aim is to select the study (within a given paper) where we think our replication effort can add the most value. 

We update these processes occasionally in order to make improvements. When we do, we update this article to reflect our new procedure, and we document the prior procedure in footnotes.

How we conduct replications

Once we have selected a study to replicate, we attempt to reproduce the study’s main results using the original data and analysis code (when available). We then develop the materials for running the study, and share them with the original paper authors to get their feedback on how closely our materials match their original study.

It is extremely important to us that our replications accurately reflect the original study. Hence, we revise our materials using feedback from the original authors. In the event that the original research team does not reply to our requests for them to give feedback on the replication materials, we will, after giving them a reasonable time to respond, proceed with the replication study regardless. 

Before launching the study, we draft a pre-registration of the study and analysis plan, and the study materials and plan are reviewed by the independent Ethics Evaluator. After the study passes ethics review, we file the finalized pre-registration of our data collection and analysis plan.

Next, we collect the data, analyze the data, and write a draft of our replication report.

We share our draft report with the original research team to give them the opportunity to raise any issues that they see (e.g., to tell us if they think there are any mistakes in our analysis), and to write a response that we will then publish along with our report.

Finally, we publish our report to the Transparent Replications website. 

Through the process of trying to reproduce the study’s results and developing materials for replication, we sometimes identify a serious methodological issue in the study design that precludes a valid test of the paper’s stated hypothesis. In such cases, if we determine that collecting new data with a similar study design would be uninformative or that a replication result would risk being misleading, we may choose not to replicate the study. Instead, we write a report that evaluates the paper’s transparency and clarity and explains the design issues that led us to forgo collecting replication data. 

We welcome feedback on our published reports. We may share particularly valuable feedback on the report page (with credit and permission), or may amend a report on the basis of feedback we receive. As we continue with this project, we expect that our processes will evolve based on our growing experience and feedback from the community, and that the procedures outlined in this article will change accordingly.

Learn more about us by reading about our mission, checking out the frequently asked questions or meeting the team.

¹Prior to our updated process for finding new candidate papers to replicate, we aimed to replicate every candidate paper published in Nature and Science. For the other three journals, we kept a running list of all candidate papers and we drew one paper at random out of the six most recent candidate papers from that journal. The papers for reports 1-13 and 15 were selected using the original selection process. We found that this approach caused too long of a gap between when the paper was published and when our replication report was posted. Our new approach is designed to reduce that gap as much as possible, while still ensuring an unbiased paper selection process.