What if there was a simple practice that, if it were used consistently, would improve the reproducibility and rigor of psychology research?

We developed the Simplest Valid Analysis to do exactly that by addressing the problems of Importance Hacking and p-hacking (discussed in Part 3 of this series). The basic idea is that the simplest analysis that provides a valid test of a hypothesis should always be run and reported for any study, even when a more complex analysis has been chosen as the main test of the hypothesis.

In our replication efforts, we’ve found the Simplest Valid Analysis to be a useful technique for determining when claims in a paper may not match the provided evidence, whether due to Importance Hacking, p-hacking, or errors in statistical analyses.

We wanted to find out what academic psychologists thought about the potential value of this technique, so we introduced it in our recent survey of psychology experts, and asked what participants thought about whether the Simplest Valid Analysis should be required for published papers.

We emailed the survey to more than 2,500 academic psychologists, and promoted the survey on relevant listservs and social media. We received 87 fully completed surveys, and another 110 that answered at least some of the substantive questions we asked. These 210 respondents indicated that they were all either experts or experts-in-training in psychology or a related field. There were additional participants who did not meet our screening criteria because they are not experts or experts in training in relevant fields, so their data were excluded from all analyses. For more information about the participants and to access the anonymized data from the study, see the survey demographics appended to Part 1 of this series.

For a technique we were introducing to people for the first time, we were surprised to see how much support there was among experts for making the Simplest Valid Analysis a requirement!

What is the Simplest Valid Analysis?

We introduced the concept to academic psychologists in our survey using the following definition:

The Simplest Valid Analysis is the simplest way to analyze the data from a study that provides a valid test of the study’s main claim.

Studies sometimes include complex analyses of their data without reporting the result of the Simplest Valid Analysis.

For instance, studies may use a fancy statistical technique (and not report a simple t-test) when a t-test would have been a valid way to test the study’s main claim, or they may apply a complex machine learning algorithm without reporting the results of ordinary linear regression in a situation where linear regression would be a valid analysis of the hypothesis.

It’s important to be clear that we’re not advocating for using the Simplest Valid Analysis instead of more complex analyses. There are often good reasons for running a complex analysis to account for various features of the data and research question. We propose that, in those cases, researchers run and report the results of the Simplest Valid Analysis alongside their more complex analysis, and explain why they believe the more complex analysis is a superior test of their hypothesis. In cases where the simple and complex analyses agree, this process serves as evidence of the robustness of the hypothesis. In cases where they don’t agree, reporting both provides useful context for interpreting the results in a way that reduces the chances of overclaiming or misinterpretation.

We are also aware that sometimes there isn’t an obvious Simplest Valid Analysis. In cases where it’s not clear what the Simplest Valid Analysis would be, we wouldn’t expect that one be included. This is meant for cases where there is an obvious simple test that could be conducted, which we think is fairly common.

Why is the Simplest Valid Analysis Useful?

We developed the Simplest Valid Analysis to solve a problem we were having when running replications. We saw a number of papers that used complex analyses, and we couldn’t tell why they had chosen that analysis instead of a simpler analysis. When we ran the simple analysis we found that, in some cases, the results weren’t consistent with the results reported from the complex analysis. This discrepancy helped us uncover serious problems in how the complex analysis was being interpreted or implemented.

We believe there are three main reasons why publications are more reliable when researchers include the Simplest Valid Analysis:

• Increasing Ease of Interpretation – It is much easier for reviewers and readers to misinterpret the meaning of a result from a complex analysis than from a simple one. If the result of the Simplest Valid Analysis is reported alongside the complex model, and the results of the two analyses are consistent, that increases confidence that the complex analysis is performing as expected and being interpreted correctly.

• Reducing researcher degrees of freedom and p-hacking – If a study isn’t pre-registered, and only a complex analysis is reported, it raises a question about whether a simpler analysis was tried, but not reported because it didn’t generate a significant result. The Simplest Valid Analysis reduces researcher degrees of freedom, and reduces opportunities for p-hacking (conscious or unconscious). 

• Reducing Importance Hacking – If a simple analysis doesn’t lead to a significant result, but a complex analysis does, reporting only the complex analysis can leave readers with the impression that the evidence supporting the substantive claims in the paper is stronger and less equivocal than it really is. It’s possible that a complex analysis is evidence for a more specific and narrow claim than the simple analysis tests, but by leaving out the simple analysis it can be made to seem that the broader claim is also supported by the published results.

We have seen studies that use mixed effects models where the complex analysis doesn’t make it easy for readers to get a baseline understanding of the data. Although the results of the complex analysis may be interpreted correctly by the authors, they can still leave readers with impressions that aren’t supported when a simple analysis is also reported. For example, simple correlations may be lower than readers assume if they only see an analysis that shows that one interaction between two variables is stronger than the other possible interactions it is compared to.

For these reasons we recommend conducting and reporting the Simplest Valid Analysis to researchers and replicators. And we recommend that editors and reviewers ask to see the results of the Simplest Valid Analysis when reviewing papers for publication.

For more examples see Transparent Replications’ post Always Conduct the ‘Simplest Valid Analysis’.

Since we’ve found the Simplest Valid Analysis to be a useful tool, we wanted to find out what academic psychologists thought of it. Did they believe it should be included in published research?

Academic psychologists believe that the “Simplest Valid Analysis” should be included in publications

After providing the definition we gave earlier in this article, we asked participants the following question:

In cases where it’s clear what the Simplest Valid Analysis would be (e.g., there is a standard simple statistical analysis that would be a valid way to test the hypothesis), what is your view on whether peer reviewers should ask submitting authors to run and report the “Simplest Valid Analysis” when they haven’t done so?

In the chart below, you can see the results:

Two-thirds of participants believed that reviewers should ask for the Simplest Valid Analysis to be reported generally, and another 27% thought reviewers should ask for it if the study is not pre-registered.  Only 7% thought there generally wasn’t a need for it.

Key Takeaways

There is a very strong consensus in favor of including the Simplest Valid Analysis – 83% of the academic psychologists we surveyed thought that it should be required (at least in the case of studies that are not-pregistered). Given that we were introducing the Simplest Valid Analysis phrase to researchers for the first time, we were surprised to see such a strong endorsement.

Given the strong support for this practice that already exists, and the benefits we have found using it in replication studies, we would encourage the following:

1. Reviewers should ask researchers to provide results for the Simplest Valid Analysis when it is clear what that analysis would be.
2. Journal editors should include reporting the results of the Simplest Valid Analysis in their guidelines for authors and reviewers.
3. Replication efforts should run the Simplest Valid Analysis as part of running a replication, if it is clear what it would be, whether it is included in the original study or not.
4. Researchers should make use of the Simplest Valid Analysis proactively to diagnose potential limitations or complications of more complex methods they are using.

Including the Simplest Valid Analysis in publications has the potential to substantially improve the reliability and robustness of psychology research by reducing error, preventing p-hacking, and reducing latitude for Importance Hacking. It also already has strong support from experts in the field, as our survey shows. For these reasons, we believe it would make a valuable addition to the open science best practices toolkit.

This article is the fourth in a four-part series. For more of what we learned, check out Part 1 on the Replication Crisis, Part 2 on our first dozen replication attempts, and Part 3 on Importance Hacking and p-hacking. Demographic information on the survey sample is in the appendix.