We wanted to find out if academic psychologists were seeing similar patterns in the field to what we had noticed. Do they perceive p-hacking to still be a common practice in top journals? Were they concerned about Importance Hacking (once we explain clearly what we mean by that phrase), or did they not see it as a serious issue?

To find out, we emailed a survey to more than 2,500 academic psychologists, and promoted the survey on relevant listservs and social media. We received 87 fully completed surveys, and an additional 123 that answered at least some of the substantive questions we asked. These 210 respondents indicated that they were all either experts or experts-in-training in psychology or a related field. There were additional participants who did not meet our screening criteria because they are not experts or experts in training in relevant fields, so their data were excluded from all analyses. For more information about the participants and to access the anonymized data from the study, see the survey demographics appended to Part 1 of this series.

What are Importance Hacking and p-hacking?

Before asking academic psychologists about p-hacking and Importance Hacking, we wanted to be clear about how we were defining both terms. We provided study participants with these two definitions:

p-hacking is taking advantage of choices available to researchers in data collection or data analysis to generate or selectively report results that meet the statistical significance threshold (e.g., p<0.05), when a result wouldn’t otherwise have been statistically significant. p-hacking can be done consciously or unconsciously, but as defined here it is a separate category from fraud (by which we mean falsifying or making up some or all of the data).

Importance Hacking is obscuring or exaggerating the meaning of results to make them appear to have more value so as to get them published, when in reality the results are not worthy of publication. A variety of issues could contribute to Importance Hacking including overclaiming, hype, lack of generalizability, or tiny effect sizes that lack real world significance. Importance Hacking can be done consciously or unconsciously, but as defined here it is a separate category from fraud. Unlike with p-hacking, results that are Importance Hacked do replicate, they just don’t have the meaning that is claimed about them.

These definitions highlight that p-hacked results are unlikely to replicate, while Importance-Hacked results (while not meaning what they are described as meaning) do still replicate. Hence, false positives (including p-hacked results) and Importance Hacked results (as defined here) are mutually exclusive categories.

Importance Hacking and p-hacking as strategies for publishing

As we conducted our replications, we realized that there are basically four strategies for getting research published. Producing good research is the most obvious one, but it is also the most difficult to do. The easiest and quickest one is committing fraud (for example by making up data and results), but this is obviously immoral, there are strong norms against it, and serious penalties if one is caught. A common “solution” to the difficulty of producing publishable research without resorting to fraud used to be p-hacking, but as more attention has been paid to the problem of p-hacking, that approach appears to be less commonly used because it is considered less acceptable than in the past. With p-hacking in decline, and researchers still under equally intense pressure to publish, it seems likely that other methods for making work appear more valuable to reviewers would increase. That led us to consider whether other methods exist, at which point we identified Importance Hacking as the missing fourth strategy for publication.

We shared the diagram below with psychology experts in our survey to illustrate these four different mutually exclusive, collectively exhaustive types of published studies:

While 10 of the 12 studies we randomly selected replicated and we didn’t find evidence of p-hacking in any of the 12, we saw many studies that were being described (whether consciously or subconsciously) in ways that made the results seem like they meant something different than they actually did. Often, we wouldn’t even realize this until we had carefully rebuilt the study from scratch (as part of our replication efforts). While some attention has been paid to specific problems that fall within this broader category, like overgeneralizing or validity issues, there wasn’t an overall term that encompassed the use of these and related techniques as a strategy to publish papers that would otherwise not be seen as worthy of publication. Much like p-hacking encompasses several more specific techniques, including selectively reporting results after running multiple tests, Importance Hacking is a term for a set of research practices that inflates the value of a research finding by making it seem more novel, clean, or impactful than it really is. For more about the types of Importance Hacking see Spencer Greenberg’s Clearer Thinking article.

Did the experts that we surveyed also see Importance Hacking as a common publication strategy? We were surprised to find out that they did, even though this survey was the first time the term was introduced to them.

Finding 1: Importance Hacking was perceived to be at least as common as p-hacking

We wanted to assess how academic psychologists perceived the literature in the field, so after presenting them with the figure above, we asked them to estimate what percentage of published papers fell into each of the 4 mutually exclusive and collectively exhaustive categories, including papers that are importance-hacked, and papers that are non-replicating for reasons including p-hacking. Here is the question we asked:

Considering only empirical studies that were published in the last 12 months in what you consider to be the top 5 psychology journals: what’s your best guess as to what percentage of them fall into each category? 

Please make sure your answers sum to 100%. 

1. Real Findings that Merit Publication– studies that report real findings that make a sufficiently valuable contribution to merit publication

2. Real Findings that Lack Merit for Publication due to Importance Hacking– studies that report findings that *would* replicate, but the paper obscures the fact that the findings reported are not worthy of publication  

3. Results that would Not Replicate– non-fraudulent studies that report results that wouldn’t replicate (e.g., false positives due to p-hacking, honest mistakes, or bad luck)

4. Fraud– studies that report fraudulent results (e.g., some or all of the data is purposely manipulated or faked)

Participants divided published studies into these four categories, and here is the average percentage assigned to each category:

Academic psychologists believe Importance Hacked papers make up more than a quarter of the papers published in the top 5 journals in the last 12 months. They rated Importance Hacking to be as common as non-replicability. The non-replicating category includes p-hacking and other causes of non-replication, meaning that the predicted rate of p-hacking is less than 26% of studies. This suggests that people may believe that Importance Hacking is more common than p-hacking.

It is worth noting that the percentage of papers that participants believed would replicate in these questions (the Importance Hacked and Real Contribution categories combined), was about 68%, which is 13 percentage points higher than the 55% participants said would replicate in response to the earlier question in this survey asking what percentage of studies published in the top 5 journals in the last 12 months would replicate. It seems possible that this question prompted more thorough reflection because participants’ answers needed to sum to 100%, and the category of Importance Hacking for studies that would replicate but still didn’t make a real contribution was explicitly introduced, and that difference in context may explain the discrepancy.¹

Fraud was suspected to account for almost 6% of articles, which is a small but concerning amount. It’s worth noting that suspecting that nearly 6% of articles are fraudulent is not the same as suspecting that percentage of researchers commit fraud. Since creating fraudulent data is a lot less work than collecting real data, researchers who commit fraud may submit articles more frequently, and also can make more novel claims because the fraudulent data can be used to support whatever conclusions they wish to advance. 

After considering the percentage of papers that were problematic in one of these 3 ways, psychologists judged only 41% of papers published in top journals in the last 12 months to be real contributions to the field.

We also asked directly about how serious of a problem psychologists perceived p-hacking and Importance hacking to be in the field.

Finding 2: Importance Hacking was seen as a more severe problem by academic psychologists than p-hacking

We asked two questions about severity, one about p-hacking and one about Importance Hacking: 

“How severe of a problem do you think that [p-hacking / Importance Hacking] is in papers published in the last 12 months in what you consider to be the top 5 psychology journals?”

The response scale for these questions ranged from “Not at all” (coded as 0) to “Extremely severe” (coded as 4). The chart below compares the mean response for the two questions:

For recently-published studies in top journals, Importance Hacking was seen by academic psychologists as a more serious problem than p-hacking.

This further underscores that the next frontier in improving psychology research may be Importance Hacking.

Finding 3: Exaggerated claims are a top reason psychology experts say they would reject papers 

We also asked psychologists to check boxes next to possible reasons that they would reject papers if they were a reviewer. Of the 7 reasons for rejecting a paper that participants could check, most people checked 3 (35%) or 4 (32%) of them.

The table below shows which reasons were checked most to least frequently by participants:

Exaggerated claims (which may be indicative of Importance Hacking) was tied for the most commonly selected reason to reject a paper, with 86% of participants saying that “the paper makes exaggerated claims that go beyond what was demonstrated by the actual findings” was a reason that they would reject a paper as a reviewer.

This shows just how strong a consensus there is that exaggerated claims can be grounds for rejecting a paper, further demonstrating that academic psychologists see Importance Hacking (which is closely linked to exaggerated claims) as a serious issue that needs to be addressed.

It’s a little less clear how this pattern of responses could relate to rates of p-hacking. Of the options in the question, rejecting papers that didn’t follow their pre-registration, rejecting papers with small sample sizes, or rejecting papers with p=0.04 could all potentially be reviewer-driven reasons why p-hacking would be on the decline. Given that respondents say they largely don’t reject papers with p=0.04 on the main result, that is unlikely to be part of the explanation. It’s possible that rejecting papers with small sample sizes, which reduces the potential impacts of removing outliers or other methods for fiddling with the dataset, may be a reviewer-driven reason that fewer papers with p-hacking are being published. Although a large number of respondents said they would reject a paper with undisclosed deviations from its preregistration, it’s not clear how often this problem comes to the attention of reviewers for most journals, since comparing the paper and the preregistration is not a standard part of a reviewer’s workflow. Psychological Science added checking for deviations from preregistrations to their editorial process at the beginning of 2025, but this practice isn’t in place at many other journals to our knowledge.

Key Takeaways

Importance Hacking is seen as a common, and serious problem in the psychology literature. Academic psychologists view it as a more severe problem than p-hacking in top journals currently. This perception is consistent with what we found in our first dozen replications, where we noticed Importance Hacking frequently, but found no instances of suspected p-hacking.

Although we don’t know what the rates of p-hacking were in the past, the lower replication rate found in major replication projects looking at prominent earlier findings suggests that it may have been much more common than it is now. It seems plausible that both the widespread awareness of why p-hacking is a problem, and the adoption of pre-registration as a technique for reducing it, may have led the practice to drop dramatically in recent publications in top journals. This could be driven by researchers improving their own practices, by reviewers being more attuned to p-hacking, or a combination of both. This is a reason for optimism that open science reforms are changing research practices, leading to a more robust and rigorous published literature.

If p-hacking is on the decline, and researchers are still held to the same standard for number of publications in top journals, there may be increased incentives to engage in Importance Hacking. It is difficult to do high quality research, so if one of the easy shortcuts to publication is eliminated, people may be pushed to use another workaround. You can think of the challenge of achieving high research quality as analogous to a pipe with multiple leaks. When you patch one hole (e.g. reducing the amount of p-hacking), more water will spray out of the remaining holes (e.g. importance hacking). To really solve the problem, you need to address all of the holes in the pipe. We believe the next biggest hole is Importance Hacking.

Our next article in this series is about a technique that we developed that can be used to help address Importance Hacking called the Simplest Valid Analysis. Watch for Part 4 in our series to learn what academic psychologists think about the Simplest Valid Analysis, and how it can be used to improve published research.

This article is the third in a four-part series. For more of what we learned, check out Part 1 on the Replication Crisis and Part 2 on Three Suprises from our Replications. Demographic information and anonymized data is available in the Appendix to Part 1.

1. 144 participants answered the earlier question, while 103 participants answered this question much later in the questionnaire. We don’t have reason to think the respondents who continued with the survey would have systematically different responses from those who didn’t, but wanted to note the difference in participants between the two questions. ︎

Transparency	Replicability	Clarity

We ran a replication of Study 4 from this paper, which found that the order in which partial vote counts are reported during an election can affect people’s perceptions of election fraud. 

During an election, partial vote totals are often reported as votes are counted across different municipalities. Inevitably, some municipalities are faster or slower to report electoral results given factors such as poll closing times or the volume of votes to be counted. This can cause candidates who ultimately lose the race to appear ahead at certain periods as vote counts are continuously updated. 

The authors of this study hypothesized that people would be more likely to believe election fraud occurred when the electoral candidate who ultimately won took a late lead, rather than an early lead, during the continuous reporting of partial election returns. The authors hypothesized this given research on the Cumulative Redundancy Bias (CRB)—the finding that people harbor better impressions of competitors who were leading during a competition, regardless of the competition’s final outcome (see Summary of the methods for a more detailed definition). 

The results from the original study confirmed the authors’ predictions: When the winning candidate took a late lead versus an early lead in the partial reported vote counts, participants were more likely to think that election fraud had occurred and that the wrong candidate had won. 

Our replication found the same results.  

​​The study received a transparency rating of 5 stars because its materials, cleaned data, and analysis code were publicly available, and it adhered to its preregistration. The paper received a replicability rating of 5 stars because all of its primary findings replicated. The study received a clarity rating of 5 stars because the claims were well-calibrated to the study design and statistical results.

Full Report

Study Diagram

Replication Conducted

We ran a replication of Study 4 from: Vaz, A., Ingendahl, M., Mata, A., & Alves, H. (2025). “Stop the Count!”—How Reporting Partial Election Results Fuels Beliefs in Election Fraud. Psychological Science, 36(8), 676-688. https://doi.org/10.1177/09567976251355594

How to cite this replication report: Transparent Replications by Clearer Thinking (2025). Report #14: Replication of a study from “‘Stop the Count!’—How Reporting Partial Election Results Fuels Beliefs in Election Fraud” (Psychological Science | Vaz et al 2025) https://replications.clearerthinking.org/2025psci36-8

Key Links

• Our Research Box for this replication report includes the pre-registration, study materials, de-identified data, and analysis files.

Overall Ratings

To what degree was the original study transparent, replicable, and clear?

Transparency:  how transparent was the original study?	Materials, analysis code, and cleaned data were publicly available, and raw data was provided upon request. The study was pre-registered and the preregistration was adhered to.
Replicability: to what extent were we able to replicate the findings of the original study?	All primary findings from the original study replicated.
Clarity: how unlikely is it that the study will be misinterpreted?	This study is explained accurately, the statistics used for the main analyses are straightforward and interpreted correctly, and the claims were well-calibrated to the study design and statistical results.

Detailed Transparency Ratings

Overall Transparency Rating:
1. Methods Transparency:	The materials were publicly available and were complete.
2. Analysis Transparency:	The analysis code was publicly available and complete.
3. Data availability:	The cleaned data were publicly available and complete. The raw data were provided upon request.
4. Preregistration:	The study was preregistered and the preregistration was adhered to.

Summary of Study and Results

Summary of the methods

During an election, partial vote counts are often reported as votes are being tallied up across different municipalities. If municipalities report their votes in different orders, the trajectory of vote counts could look radically different, even though the final tally is the same.

The original study (N=195) and our replication (N=364) examined whether the order in which partial vote counts are reported affect people’s perceptions of election fraud. 

The original authors hypothesized that people would be more likely to attribute election fraud when the candidate who ultimately wins gains the lead towards the end of the vote count reporting period. For example, a candidate who was trailing for most of the night as partial vote counts rolled in, but then ended up gaining a late lead and winning the election, might be more likely to be accused of election fraud than a winning candidate who led the vote count the whole night. 

The authors hypothesized this given their previous research on the Cumulative Redundancy Bias. The Cumulative Redundancy Bias is a cognitive bias that occurs when people observe the progression of a competition in a cumulative format (e.g., votes added to a running total). In such a cumulative format, any new observation already contains the data from previous standings, such that a rational individual should ignore previous standings and only rely on the end result in their judgment. However, people are nevertheless influenced by interim standings and judge a “leading” competitor more positively, even if the end result is the same. As the authors state, “The repeated observation of a competitor being ahead seems to leave a lasting impression on observers that is not entirely erased by the final result” (Vaz et al., 2025, p. 677).      

To test their hypothesis about attributions of election fraud, the authors showed participants the progression of partial vote counts of an alleged election in Eastern Europe. Participants saw the accumulated vote counts for two different candidates across 10 timepoints, where each timepoint corresponded to roughly 10% more of the vote coming in. For example, below are screenshots of the final three timepoints that some participants saw.

Importantly, however, there were two different patterns of results participants might see. 

Participants in the Late Lead Condition saw accumulated vote counts such that the candidate who ultimately won was trailing at each of the first 9 timepoints before gaining the lead at the 10th and final timepoint. 

Participants in the Early Lead Condition saw accumulated vote counts such that the candidate who ultimately won was leading during all 10 of the timepoints. 

The final timepoint that participants saw, which presented 100% of the vote count, was the exact same in both conditions. The key difference between the conditions was that the candidate who ultimately won appeared to be losing up until the final timepoint in the Late Lead condition, but appeared to be winning the whole time in the Early Lead Condition.  

Even though participants thought they were seeing vote counts from an Eastern European election, the vote accumulations in both conditions were derived from the real election data for the state of Georgia during the 2020 U.S. presidential election. The accumulated vote counts shown in the Late Lead Condition reflected the actual order in which the vote count was reported during the election coverage. The accumulated vote counts shown in the Early Lead Condition presented roughly what the partial vote counts would have been if precincts had been reported in reverse order.

These two different vote count trajectories are displayed in a figure from the original paper, copied below:

After participants saw the vote counts at all 10 timepoints, they were told: 

Shortly after the vote count was finished, rumours emerged that the vote count may have been rigged and that the wrong candidate won as a result. The people responsible for the vote count, however, denied the allegation.

Participants then answered the two primary questions of interest:

How likely do you think it is that the vote count was manipulated, on a scale from 1 (very unlikely) to 10 (very likely)?

How likely do you think it is that the wrong candidate won, on a scale from 1 (very unlikely) to 10 (very likely)?

On the next page, participants were asked to recall which of the candidates won the election:

You’re almost done.

Before you finish, we want to check if you remember who won the election.

* Lukas P.
* Miroslav K.
* Don’t Remember

This question served as an attention check; any participants who failed it were dropped from our analyses.

In the original study, participants then provided their age and gender to complete the study. In our replication, before participants provided their age and gender, we asked them one additional question:

Here’s how you answered two of the questions you were asked earlier:

Question: “How likely do you think it is that the vote count was manipulated, on a scale from 1 (very unlikely) to 10 (very likely)?”
Your response: {this displayed the participant’s response}

Question: “How likely do you think it is that the wrong candidate won, on a scale from 1 (very unlikely) to 10 (very likely)?”
Your response: {this displayed the participant’s response}

Can you briefly explain your reasoning? 

We included this open-ended question so that we could assess whether the rationales participants provided were in line with the hypotheses put forth by the original paper. 

Summary of the results

In the original study, participants in the Late Lead Condition thought it was more likely that the vote count was manipulated (p = .002) and more likely that the wrong candidate won (p < .001), on average. 

Our replication found the same general pattern:

Dependent variable	Mean difference between Late Lead and Early Lead conditions		Finding replicated?
	Findings from original study	Findings from our replication
“How likely do you think it is that the vote count was manipulated, on a scale from 1 (very unlikely) to 10 (very likely)?”	Mean diff: 1.04 t(170.66) = -3.14, p = .002	Mean diff: 1.14 t(351.29) = -4.78, p < .001
“How likely do you think it is that the wrong candidate won, on a scale from 1 (very unlikely) to 10 (very likely)?”	Mean diff: 1.31 t(160.49) = -4.06, p < .001	Mean diff: 1.41 t(335.45) = -6.05, p < .001

Study and Results in Detail

This section provides details about participant recruitment, minor study design differences between the original study and the replication, differences in response distributions between the original study and the replication, and results from the open-ended question.

Additional details about the methods

Participant recruitment

We aimed to have a final sample size of 366, which would provide 90% power to detect an effect that was 75% of the size of the smaller of the two effect sizes reported in the original paper. We anticipated an exclusion rate of 3% based on that observed in the original study. As a result, we recruited 377 participants on Positly.com to complete the study. 

In total, 379 participants completed the study (occasionally, a few more participants complete a study than the number recruited on the platform). After excluding any participants who failed the attention check (i.e., those who couldn’t recall which candidate won the election), we were left with a final sample size of N=364 (the sample size of the original study was N=195).

Minor design differences between the original study and the replication

We kept the study design nearly identical to that of the original study, but we made three tiny alterations. We verified our study plan, including our planned alterations, with the original authors before collecting data.

Alteration 1

In the original study, the order in which the candidate names were listed was randomized and which candidate was the winner was randomized, but the candidate who ended up winning was always listed on the top row of the table that displayed the partial election counts. 

We felt that it would be better to counterbalance the order of the winner, rather than the order in which the names were listed. So we tweaked the display order such that the candidates names were always displayed in alphabetical order (Lukas first, Miroslav second), but which candidate wins—and thus whether the winner is listed first or second—was randomized. 

Alteration 2

In the original study, all three response options for the attention check (Lukas P.; Miroslav K.; Don’t remember) were displayed in a random order. We thought it looked strange when it was randomized such that “Don’t remember” was the middle option. We changed the randomization such that “Don’t remember” was always the third option and “Lukas P.” and “Miroslav K.” were randomly assigned to be the first and second option. 

Alteration 3

As mentioned earlier, we thought it would be helpful to have participants explain why they answered the dependent variables as they did, so we included a question asking them to explain their reasoning. This question came after the two dependent variables and the attention check, such that it would not affect the results of the study in any way. 

We thought this question could provide more insight into participants’ thinking process. However, we stated in our preregistration that participants’ responses to this question would not impact the replicability score the paper received.

Additional details about the results

Differences in distributions of participant responses

Even though the results from the statistical tests run on the original data and our replication data returned very similar results (see Table 1), some of the distributions of participants’ responses looked different between the two datasets. 

As you can see in Figure 3, copied below, the distribution of participants’ responses in the Early Lead Condition look pretty similar between the original data and the replication data. However, the distributions of responses in the Late Lead Condition differ. In the original data, participants’ responses in the Late Lead Condition appear bimodal, such that there are two common responses (values in the lower-middle part of the scale, and values in the upper-middle part of the scale). In the replication data, participants’ responses appear unimodal, such that there is a single most common response (values in the middle of the scale).  

It is not clear what to attribute this difference to, but future research should focus on this discrepancy because different distributions of responses tell different stories. 

The original data seem to suggest that, in the Late Lead Condition, a sizable group of people think it’s likely that election fraud occurred and another sizable group of people think it’s unlikely that election fraud occurred. These data could mean that there’s a sizable group of people that are quite sensitive to whether a candidate takes a late lead, and there’s another sizable group that are insensitive to late-lead situations.

On the other hand, our replication data seems to suggest that, in the Late Lead Condition, the most common response is that people are unsure whether election fraud occurred. These data could mean that most people are slightly sensitive to whether a candidate takes a late lead. 

However, because this study used a between-subjects design, it is difficult to precisely assess how many people were affected by the experimental manipulation and in what way they were affected. A within-subjects design would be needed to assess how each individual participant’s ratings differed in the Late Lead Condition compared to the Early Lead Condition. This would allow us to more accurately estimate what percentage of people are likely to be unaffected, mildly affected, and strongly affected by a candidate taking a lead late rather than an early lead. 

Future work that employs a within-subjects design could provide more information about individuals’ sensitivity to late-lead dynamics, which could help contextualize the distributions observed in the Late Lead Condition. 

Results from open-ended question

As described earlier, we asked participants to explain why they answered the two key questions as they did. Here’s the exact question we asked them: 

Here’s how you answered two of the questions you were asked earlier:

Question: “How likely do you think it is that the vote count was manipulated, on a scale from 1 (very unlikely) to 10 (very likely)?”
Your response: {this displayed the participant’s response}

Question: “How likely do you think it is that the wrong candidate won, on a scale from 1 (very unlikely) to 10 (very likely)?”
Your response: {this displayed the participant’s response}

Can you briefly explain your reasoning?

We included this open-ended question so that we could assess whether the rationales participants provided were in line with the hypotheses put forth by the original paper. We preregistered that, since this question was not included in the original study, we would not factor participants’ responses to this question into our replicability rating. 

To get a high-level sense of how participants answered this question, we ran their responses through a large language model (LLM)—specifically, ChatGPT-5.2-thinking. 

It’s important to say upfront that the results from the LLM we present below should not be interpreted as rigorous, well-vetted findings. Rather, it’s akin to having a human read through all of the responses and providing their own idiosyncratic grouping of people’s explanations. That said, we still think it’s a helpful way to get an approximation of what participants were thinking. Anyone interested in doing more formal analyses on this free response data is welcome (and encouraged!) to do so. This data is publicly available here. Additionally, we’ve included the exact prompts we used and links to the full responses from the model in the Appendix (see the section titled “Additional Information about the Results”). 

We asked the LLM to summarize the different types of explanations participants provided. We fed the model the responses from participants in the Early Lead Condition and the Late Lead Condition separately. In other words, we gave the model the same prompt twice and in one instance it had access to the Early Lead Condition data and in the other instance it had access to the Late Lead Condition data. 

Here’s how the model summarized the responses from participants in the Early Lead Condition:

Here’s how the model summarized the responses from participants in the Late Lead Condition:

These summaries raise a few interesting points. 

First, some of the common explanations in both conditions support the hypothesis from the original paper. According to the LLM, a common explanation in the Late Lead Condition was that it was suspicious that the winner only gained the lead towards the end of the vote count reporting cycle. Moreover, a common explanation in the Early Lead Condition was that it was unlikely that manipulation occurred given that the eventual winner was in the lead throughout the vote count reporting. Both of these explanations are consistent with the original paper’s cumulative-redundancy-bias account for why people would think manipulation is more likely in the Late Lead Condition. 

Second, some participants seemed savvy to more benign reasons a candidate might gain a late lead in the vote count reporting cycle. Although these explanations were less common than those discussed in the paragraph above, some participants noted that the order in which geographic areas report vote counts can cause late-lead dynamics. 

Finally, a lot of participants seemed to justify their responses by simply appealing to their prior beliefs about how common or uncommon election manipulation is. 

Overall, the fact that some of the most common explanations participants provided were directly aligned with the authors’ cumulative-redundancy-bias account corroborates the authors’ hypothesis, especially for the subset of participants who provided that explanation explicitly. 

It is also possible that the original paper’s account accurately explains the judgments of participants who provided explanations that do not align with the Cumulative Redundancy Bias. After all, it’s possible that some participants were influenced by the Cumulative Redundancy Bias, but didn’t realize it. Interestingly, in Study 6 of the paper (which we did not attempt to replicate), participants were provided a benign explanation for swings in the vote count totals—that votes were counted first in the rural areas where the losing candidate was more popular. The study found that, even with this explanation in hand, participants in the Late Lead Condition still thought election manipulation was more likely. 

As such, it’s difficult to estimate from these explanations alone how many people’s judgments were influenced by the Cumulative Redundancy Bias. Even with this explanation data, we don’t know whether the observed differences between conditions were driven by a subset of participants responding strongly, or by a large proportion of participants responding mildly. As mentioned in the previous subsection, a within-subjects design would provide more insight into how many participants showed the hypothesized effect and to what degree. Coupled with participants’ explanations for their ratings, a within-subjects design would allow researchers to examine questions like: how many participants who were sensitive to the winning candidate taking a late lead provided an explanation consistent with this experimental manipulation? Ultimately, the fact that so many participants conjured explanations that directly align with the Cumulative Redundancy Bias supports the main hypothesis in the paper. 

Interpreting the Results

All of the results from the original study replicated when analyzed on the data we collected. In addition to finding the same general patterns, the mean values and differences between conditions we observed were very similar to the original study (see Figure 3). 

When assessing this study in isolation, one might wonder if the observed effects are really attributable to whether the candidate gained a late lead versus an early lead. After all, if you inspect the sequence of partial vote counts participants saw in both conditions (see Figure 2), there were other differences beyond the early-lead/late-lead factor. For example, in the Late Lead condition, the race appears fairly close at every timepoint, whereas, in the Early Lead condition, the race only becomes close towards the end. This represents a difference between the conditions that was not experimentally controlled for. From this study alone, we can’t know whether such differences mattered. However, the original paper has a total of seven studies, many of which rule out alternative explanations. For example, Study 5 uses almost exactly the same sequence of partial vote counts in both conditions, and still finds a similar effect to that observed in Study 4. We recommend reviewing the other studies in the original paper for those interested in alternative explanations for these results. 

Another thing to note is that because this study used a between-subjects design, we are not able to directly assess what percentage of the participants were influenced by the order in which partial vote counts were reported. For example, it could be the case that the differences in average ratings between the conditions were due to a small number of participants being strongly affected by the partial vote count order or due to a large number of participants being mildly affected by the partial vote count order. This could be useful for future research to untangle. 

Finally, it is worth mentioning that Psychological Science, the academic journal that published the original paper, has recently instituted a series of transparency requirements. They now require authors to publicly share data, study materials, analysis code, and deviations from preregistrations. They also ensure that the results in a paper are computationally reproducible and emphasize the importance of not overclaiming. These are not simply stated policies of the journal, but are elements actively assessed by specific editors at the journal whose role is to evaluate the transparency of submitted articles. (You can read about these policies in more detail, and the rationale behind them, in this editorial.) 

This was the first Psychological Science paper we replicated that was published after these policies were implemented. Many of the issues we’ve run into when replicating previous studies—e.g., overclaiming, coding errors, deviations from preregistrations that went unacknowledged—were not present in this paper.  We cannot, of course, attribute the quality of this paper to the new policies at Psychological Science since the authors might have taken the same actions regardless of the journal’s policies. Nevertheless, Psychological Science’s policies seem likely to substantially improve the transparency and clarity of articles published in this journal.¹  

¹It is important to acknowledge that I (the author of this report) volunteer as a reproducibility checker for Psychological Science, which could be biasing my view on the positive potential of these policies instituted at Psychological Science. 

Conclusion

Overall, we successfully replicated the two primary findings from the original study. Both the original study and our replication found that participants were more likely to think election fraud had occurred and that the wrong candidate had won when the winning candidate took a late lead, rather than an early lead.   

​​The study received a transparency rating of 5 stars, a replicability rating of 5 stars, and a clarity rating of 5 stars. 

It is important to note that the study we replicated was one of seven studies reported in the original paper. As such, our replication only directly assesses a small proportion of the findings reported in the paper.

Acknowledgements

We want to thank the authors of the original paper for making their data and materials publicly available, and for their quick and helpful correspondence throughout the replication process. Any errors or issues that may remain in this replication effort are the responsibility of the Transparent Replications team.

We also owe a big thank you to our 364 research participants who made this study possible.

Finally, we are extremely grateful to the rest of the Transparent Replications team for their advice and guidance throughout the project.

Purpose of Transparent Replications by Clearer Thinking

Transparent Replications conducts replications and evaluates the transparency of randomly-selected, recently-published psychology papers in prestigious journals, with the overall aim of rewarding best practices and shifting incentives in social science toward more replicable research.

We welcome reader feedback on this report, and input on this project overall.

Appendices

Additional Information about the Methods

The table below displays the full set of partial vote counts participants witnessed, depending on the experimental condition they were assigned to. 

The “% of votes reported” column displays the percentage of votes that participants learned had been counted at that point.

 The “Vote count” column displays the running total of the votes each candidate had received. 

In this table, we have denoted the candidates as “Winner” or “Loser” which refers to whether the candidate ended up winning or losing once the final votes were in (participants did not see these terms; instead they saw the names of the candidates). 

See Figure 1 in the Summary of the methods section to see how these numbers were presented to participants at each timepoint. 

Additional Information about the Results

Below is the prompt we fed to ChatGPT-5.2-thinking (on December 22, 2025) in order to have it summarize the explanations participants provided in the study. As described in the section titled “Results from open-ended question,” we fed the model the responses from participants in the Early Lead Condition and the Late Lead Condition separately.

I have attached data from a study about election fraud. In this study, participants saw the progression of partial vote counts of an election in Eastern Europe. Participants saw the accumulated vote counts for two different candidates across 10 timepoints, where each timepoint corresponded to roughly 10% more of the vote coming in. The final timepoint showed the official, final vote count.

After participants saw the vote counts at all 10 timepoints, they were told: 

“Shortly after the vote count was finished, rumours emerged that the vote count may have been rigged and that the wrong candidate won as a result. The people responsible for the vote count, however, denied the allegation.”

Participants then answered the two primary questions of interest:

“How likely do you think it is that the vote count was manipulated, on a scale from 1 (very unlikely) to 10 (very likely)?” (in this dataset, this is the column “likely_manipulated”)

“How likely do you think it is that the wrong candidate won, on a scale from 1 (very unlikely) to 10 (very likely)?” (in this dataset, this is the column “likely_wrong_candida”)

Participants were then asked to explain their reasoning for their responses to these questions (in this dataset, these explanations are shown in the column “explanation”).

Can you please summarize the different types of explanations participants provided? 

You can view the full response from ChatGPT-5.2-thinking for the Late Lead Condition here (https://chatgpt.com/share/694981bd-8460-8006-ad49-a9030c424918) and the Early Lead Condition here (https://chatgpt.com/share/69504a28-95cc-8006-a90a-8c314ddd4b53). 

References

Faul, F., Erdfelder, E., Buchner, A., & Lang, A.G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149-1160. https://doi.org/10.3758/BRM.41.4.1149

Vaz, A., Ingendahl, M., Mata, A., & Alves, H. (2025). “Stop the Count!”—How Reporting Partial Election Results Fuels Beliefs in Election Fraud. Psychological Science, 36(8), 676-688. https://doi.org/10.1177/09567976251355594

Are we still experiencing a “replication crisis,” where only 40-60% of results replicate when the study is conducted again?

Psychology experts who we surveyed predicted that 55% of recently published studies published in top journals would replicate, suggesting that they think the field is still experiencing a serious replication crisis, although they also believe that substantial progress has been made, as we discussed in Part 1. Is their assessment accurate?

We completed our first dozen replication attempts on recent papers selected randomly¹ from top journals, and what we found really surprised us! As we’ll explore in the rest of this article, while the research looked much better than we expected on one metric, results on another metric (that’s rarely discussed) are more discouraging.

Unlike other replication projects, which have focused on prominent older findings or have been limited to a single journal, our project focuses on recent papers, randomly selected from the top journals in the field. By selecting papers randomly and focusing on recent publications at the top of the field, we can use these replication results to reflect on the state of the field right now.

In addition to using a different selection process for papers, at Transparent Replications we don’t look at replicability in isolation. We rate studies on three criteria:

1. The Transparency rating assesses the availability of study materials, data, and analysis code; as well as whether study was pre-registered and how well the pre-registration was followed. 
2. The Replicability rating reports how many of the main findings reported in the study replicated when we conducted the study again with new data. 
3. The Clarity rating evaluates how likely we believe a reader is to come away with an accurate impression of the study and results from reading the paper.

We rate studies on these three categories because replicability alone doesn’t tell the whole story of what makes papers useful and reliable.Transparency makes it possible to understand a result, and is often necessary for a faithful replication or reproduction. Clarity, which is a novel rating that we developed, allows us to assess factors that could be a problem even in papers that replicate – for example, overclaiming, validity issues, or other errors in the paper that would lead readers to misunderstand the implications of a result.

The table below shows the distribution of ratings on Transparency, Replicability, and Clarity for the first dozen reports that we conducted. Ratings under 4 stars are in bold. The ratings are on a scale of 0 to 5 stars.

We found our results on all three of these ratings to be somewhat unexpected, but the replication rate is especially at odds with psychology experts’ perceptions about the field.

Surprise 1: Replication rates are higher than experts predicted and p-hacking is much less common than we expected!

One of the most surprising things to us is how well the studies replicated. We’ve completed 12 reports (with a number of others currently in progress). In the replication studies that we conducted, 10 of them completely or mostly replicated, and only 2 had primary findings that mostly did not replicate.² This is a rate of 83%, compared to the experts prediction of 55%. Of course 12 is a small number, so these should be considered preliminary findings until we have completed more replication reports.

The replicability rating score is the percent of study’s main findings that replicated, converted into a 0 to 5 star range. A study that received a rating of 4 had four-fifths (or 80%) of its main findings replicate, while a study with a rating of 2 only had two-fifths (or 40%) of its main findings replicate. Many studies only had one main finding, which means they could only receive a score of 5 (100%) if the finding replicated, or a score of 0 (0%) if the finding did not replicate.

Here’s a summary of the replicability scores:

In addition to the high replicability rate overall, it’s informative to look into the reasons why the 2 studies that largely failed to replicate didn’t replicate.

In one case we believe that the lack of replication was due to the statistical power issues.³ For that reason, we don’t take it as meaningful evidence that we should reduce our confidence in the original paper’s findings. That report was instructive for demonstrating how much impact subtle experimental design decisions can have on statistical power, especially in more complex statistical models.

In the other case of replication failure, we think the study’s main finding didn’t replicate because the original sample had a peculiar characteristic that the authors diagnosed and acknowledged, but that influenced the results in an unanticipated way. In this case we do think the lack of replication should reduce confidence that the claimed effect in the paper is real, but we don’t see any evidence of p-hacking in this paper. This finding not replicating demonstrates the value of replicating research findings even when no p-hacking is suspected – spurious results can occur even when researchers do their work carefully, and replication is how those results are detected.

That means that in our first 12 completed replications, we did not find a single case where we believe substantial p-hacking meaningfully impacted the results! (As a reminder, p-hacking is consciously or unconsciously taking advantage of choices available to researchers in data collection or data analysis to generate or selectively report results that meet the statistical significance threshold (e.g., p<0.05), when a result wouldn’t otherwise have been statistically significant.)

The lack of evidence of p-hacking is shocking when you compare it to large replication studies, like the Open Science Collaboration’s replication of 100 studies from the 2008 issues of three prominent journals, the replication of 21 papers published from 2000-2015 in Nature and Science, or the Many Labs project’s multiple replications of prominent findings that were originally reported from 1936 to 2014. In these replication projects, covering papers from ten or more years ago, roughly 40%-60% of papers failed to replicate, with many (and perhaps the vast majority) of those failures seemingly due to p-hacking.

While 12 is obviously a small number (and we’ll have more data over time), if we assume that rates of substantial p-hacking for main findings is 40% – which we believe is a reasonable estimate of what they were 15 years ago based on data from large-scale replication studies, then there would only be about a half of a percent chance that we would find no cases of substantial p-hacking out of 12 replications conducted! Even if we are mistaken and 1 of the studies we replicated had substantial p-hacking influencing the finding, that would still indicate less than a 3% chance of having that few (or fewer) such cases out of 12 if the base rate was 40%! (Supporting calculations for this paragraph are in the Appendix.)

This suggests to us that p-hacking may now be substantially less common than it used to be. Increasing transparency, preregistration, and awareness of the problem may have influenced reviewer comments, and editor decisions. Additionally, as p-hacking has come to be considered less acceptable and the problems with it more widely understood, researchers may simply be holding themselves to a higher standard in their own research.

Surprise 2: Public availability of data and materials is widespread, yet deviations from pre-registration are commonly not acknowledged

In addition to higher than expected replication rates, we were pleasantly surprised by how strong transparency practices are in recent papers in top journals, although more work needs to be done to ensure that deviations from pre-registration are acknowledged in published papers.

Looking at the chart below, you can see that the lowest transparency rating so far has been a 3 out of 5. The average transparency rating of our reports overall is 4.1. At least from this limited dataset, what this tells us is that, in top journals in the field, data, analysis code, and experimental materials are usually publicly shared. This may be due to top journals expecting that these materials are shared. Preregistration is fairly common, but far from universal.

Our Transparency rating includes 4 sub-ratings. The first three assess the availability and completeness of  study materials (1), analysis code (2), and data (3). The fourth is about pre-registration, including whether the study is pre-registered, how well the pre-registration is followed, and whether deviations from the pre-registration are acknowledged in the paper. In practice, a study receiving a 3 for Transparency may have study materials and data publicly available, but not have analysis code available, and have major undisclosed deviations from the preregistration. A study receiving a 4 likely has materials, data, and code that are available, but the study wasn’t pre-registered. A study receiving a 5 follows its pre-registration (or acknowledges and explains any deviations), and has study materials, analysis code, and data that are complete and publicly available. A full explanation of our Transparency rating system is available here.

This level of transparency is a serious improvement over past practices, and makes it much more possible for replication and reproduction of studies to be conducted. Open science norms about transparency appear to be much more widespread than they used to be.

The most serious transparency issue that we ran into is that a study may be pre-registered, but deviate from the pre-registered analysis plan without acknowledging the changes that were made. In the first dozen reports, seven of the studies were preregistered; however, of those seven studies, only two followed their preregistration without any unacknowledged deviations. One had minor deviations in exclusion criteria that weren’t disclosed, two more had moderate unacknowledged deviations from their preregistration, and two had major unacknowledged deviations.

Sometimes it is appropriate to deviate from a preregistration, but when that happens, the paper should acknowledge the changes and explain why they were made. Preregistration can only do its job of reducing researcher degrees of freedom and preventing questionable research practices like p-hacking and unreported instances of HARKing (Hypothesizing After the Results are Known) if the preregistration is followed.

When journals evaluate submitted papers, it should be standard practice to compare the preregistration to the paper to see if they are consistent, and if there are inconsistencies ensure that they are disclosed and explained. It’s excellent to see that Psychological Science has started doing exactly that with all published papers starting at the beginning of this year. We hope to see more journals implement that best practice. 

Since top journals are starting to require submissions to meet many of these transparency benchmarks, we think it is likely that we’ll continue to see high transparency ratings for papers as we conduct more replications. Hopefully other journals will follow the lead of Psychological Science and check for deviations from pre-registration and include a report of those deviations with the published paper. That would go a long way to improving the main transparency issue that we found in our first twelve replication reports.

Surprise 3: Importance Hacking and/or errors affect most papers, and appear to be much bigger issues than p-hacking! 

The rating area where we see the most need for improvement is Clarity. From looking at the chart you can see that Clarity ratings vary much more widely than Transparency ratings. The Clarity rating averaged over our first dozen reports was 3.1, an entire point lower than the averaged Transparency rating.

The clarity rating addresses how likely we believe a reader is to come away with an accurate impression of the study and results from reading the paper. Low clarity suggests that a reader may be likely to misunderstand key aspects of the research or its implications.

There are two main classes of issues that reduce the clarity of a paper. Only two of the twelve papers we evaluated had neither type of clarity issue.

Clarity Issue 1: Errors (and imprecision)

The first clarity issue we look for are errors or imprecision in the study materials, analyses, and paper. We also evaluate the severity of errors and impressions. For example, an error that is minor or that doesn’t impact the main takeaway of a study impacts the clarity rating much less than an error that changes the study’s takeaway, and much less than if the study involved a long list of small errors. 

A total of eight of the twelve papers we evaluated had issues of this type, three of which we would consider to have major issues.

For example, across different studies that we investigated, we found composite variables that were miscalculated, incorrect statistical tests being used, and experimental materials which included mistakes in key questions. 

We also evaluated studies where key features of the study that would be important to the reader understanding and interpreting the results were not clearly described in the paper. We found five papers with minor issues in communicating information the reader would need to understand the study and results. These issues included inaccuracies in descriptions of study procedures, incorrect numbers in results tables, and omissions of important information about key variables.

We consider these issues of error and imprecision together because the distinction can be difficult to make in practice. For example, if a variable is calculated differently in an analysis than how it seems to be described in the paper, it’s possible that the calculation was done that way incorrectly (in error), or that the explanation in the paper is an unclear (or imprecise) description of what was done. Ultimately, whether such an occurrence is an error (they did a calculation that was different than they intended) or an example of imprecision (they did what they intended but misexplained it to the reader) comes down to the intention of the researchers, which usually can’t be evaluated from the paper and its materials.

We were surprised by the amount of error that we encountered in published papers (which, recall, were all published in top peer-reviewed journals). This suggests that improvements need to be made to editorial processes so that these issues are detected and addressed prior to publication. 

Along these lines we are pleased to see that, in addition to reporting on deviations from preregistration, Psychological Science has started reproducing statistical results prior to publication for many of their papers. While some of the errors that we encountered would have required a more in-depth investigation to detect, we suspect that at least two of the three cases of serious errors we found would have been detected had they been subjected to this review process. 

If other journals implement more rigorous pre-publication checks, that would go a long way to addressing the more severe cases of this issue. If the analysis code doesn’t run properly, the analysis has issues (like the model failing to converge), the paper mislabels the statistical tests used, or there are discrepancies in the reported results, this kind of check would have a good chance of detecting it.

Clarity Issue 2: Importance Hacking

The second issue that reduces the clarity of a paper is what we call, “Importance Hacking.” Oddly, we do not believe this concept had a standard name before we gave it one, despite it being commonplace. We think it’s critical to have a name for this phenomenon, because we believe it is not only common, but important to address for making further improvements in how science is practiced.

Seven of the twelve studies had at least a minor Importance Hacking issue in our analysis, three of which were more severe. 

Importance Hacking (which can be done consciously or subconsciously) is obscuring or exaggerating the meaning of results to make them appear to have more value or interest than they really have so as to get them published, when in reality (if reviewers understood the true meaning of results) they would be unlikely to recommend the paper for publication. A variety of issues can contribute to Importance Hacking including overclaiming, hype, lack of generalizability, claims that don’t actually follow from the statistical results, and/or tiny effect sizes that lack real world significance. For more about the types of Importance Hacking see Spencer Greenberg’s Clearer Thinking article.

We found lack of generalizability and insufficient engagement with plausible alternative explanations were the most common Importance Hacking issues in the first dozen papers. In addition to those more common issues, we found a study that used a complex analysis implying a result that wasn’t supported if a simple (but still valid) analysis was done. Another study made central claims that did not match the evidence provided.

Although there have been some calls for attention to issues of generalizability, ecological validity of experiments, and small effect sizes; addressing Importance Hacking hasn’t yet gotten the attention that tackling p-hacking and other questionable research practices has received. Our preliminary findings suggest this is the next major frontier for improving research.

To tackle Importance Hacking we need to change norms and develop new techniques. For example, requiring papers to include the Simplest Valid Analysis addresses some types of Importance Hacking as well as p-hacking. Studies being presented in a consistent way using a Study Diagram may address another kind of Importance Hacking by making the critical aspects of a study clearer at a glance, which makes overgeneralizing and making unjustified claims more difficult to do without it being noticed.

Conclusions

This chart combines the Transparency, Replicability, and Clarity ratings charts from above, showing the number of studies with each rating on each of the three criteria.

Our overall take on these first dozen reports is that there is a lot of reason for optimism about psychology as a field, and yet major hurdles remain.

The Replication rate in these randomly-selected recent papers from top journals (83%) appears to be substantially higher than in papers from fifteen or more years ago (40-60%), and substantially higher than what the experts we surveyed predicted (55%). Although this is a small sample of papers, the fact that they were selected through a randomized process and that we didn’t find a single clear-cut instance of p-hacking means we can take this as preliminary evidence suggesting meaningful improvement.

The widespread adoption of transparency practices for papers in top journals is another reason for optimism. Additionally, at least one journal, Psychological Science, is addressing unacknowledged deviations from the study’s preregistration. By checking for deviations from the preregistration, reporting whether any were found at the end of the paper, and including a table in the supplemental materials listing them, Psychological Science is likely to be able to largely prevent the most serious issue we found in transparency – a paper claiming that a study is preregistered, but our review revealing substantial undisclosed discrepancies between the paper and preregistration. Sometimes deviations from the preregistration are done for good reason, but they should always be disclosed, and this approach ensures that.

As Replicability and Transparency seem to have improved, there is more need to focus on the Clarity issues that continue to plague published research. Addressing the error portion of Clarity should be relatively straightforward – making it standard practice for journals to check results prior to publication, as Psychological Science does, would take a big step towards solving this problem.

We believe the next frontier in improving psychology research is tackling Importance Hacking, which will require changing norms and developing techniques to tackle problems with validity, generalizability, overclaiming, small effect sizes, and other ways that a study can be made to seem more valuable than it truly is.

The problem of Importance Hacking also struck the experts we surveyed as a serious issue meriting greater attention from the field. We will address what psychology experts think about the severity of the problem of Importance Hacking compared to the problem of p-hacking in the field today in Part 3 of this series.

This article is the second in a four-part series. For more of what we learned, check out Part 1 on the Replication Crisis.

Appendix

Chi-Squared Goodness of Fit Test Results for P-Hacking Estimates

For 0 suspected p-hacking instances out of 12 observations, when the expected rate is 40%, the p-value of the Chi-Squared Test of Goodness of Fit is .00468, or a 0.468% chance of achieving a result that extreme or more extreme by chance if the true rate of p-hacking is 40%.

If we missed one that we should have suspected, and there was 1 suspected p-hacking instance out of 12 observations, the p-value of the Chi-Squared Goodness of Fit test is .02514, or a 2.514% chance of achieving a result that extreme or more extreme by chance if the true rate of p-hacking is 40%.

Calculations completed using the Social Science Statistics Chi-Square Goodness of Fit calculator.

Psychological Science Transparency and Reproducibility Policy Changes

See Simine Vazire’s editorial “The Next Chapter at Psychological Science,” and Tom Hardwicke and Simine Vazire’s editorial “Transparency is Now the Default at Psychological Science,” for more information about the changes to their transparency and reproducibility policies.

Additional information about Psychological Science’s STAR editors’ responsibilities from their Contributor FAQ:

1. For more on this see “What We Do” on our website, which explains our selection process and the constraints on which papers we consider eligible, which take into account ethical, logistical, and cost considerations. ︎
2. Note that the chart contains one report (#12) for which we did not attempt a replication due to methodological issues with the original study. The chart also contains one report (#4) for which we selected 2 studies, both of which replicated. That is why we report 10 out of 12 studies mostly replicating, despite the chart only showing replication ratings for 11 of the 12 reports. ︎
3. See the replication report for details. ︎

Transparency	Replicability	Clarity
	The main finding was not replicated, but there was a trend toward significance and the effect was in the same direction in our replication dataset as in the original study.

We replicated an experiment from this PNAS paper. Adults participated in a computerized (simulated) apple-picking (foraging) task and completed an adverse childhood experiences (ACEs) questionnaire. The original study found those with high levels of adverse childhood experiences (ACEs) tended to spend longer picking apples before moving to a new tree in the simulated foraging task compared to adults with fewer ACEs. From this, it was inferred that people with high ACEs tend to explore less than those with low ACEs. The main finding didn’t replicate in our study, although there was a trend in the same direction as the original experiment. 

The paper received a moderate transparency rating. Experimental materials and scripts were shared transparently, although the public materials were missing important data cleaning steps, and the experiment and analysis scripts required substantial editing in order to run properly. The primary weakness in transparency was that the paper described the study as pre-registered, but there were major deviations from the pre-registration which were not acknowledged in the paper or supplementary materials. 

Several factors limited the paper’s clarity. Firstly, we think the findings have more limited generalizability than the paper suggested. The paper could also have discussed several alternative explanations for the findings. Finally, certain terms were used in ways that were counterintuitive and also inconsistent with the paper from which the terms were derived.

Full Report

Study Diagram

Replication Conducted

We ran a replication of the main study from: 

Lloyd, A., McKay, R. T., & Furl, N. (2022). Individuals with adverse childhood experiences explore less and underweight reward feedback. Proceedings of the National Academy of Sciences, 119(4), e2109373119.

How to cite this replication report: Transparent Replications, Harris C.D., & Svoboda, J. (2025). Report #13: Replication of “Individuals with adverse childhood experiences explore less and underweight reward feedback” (PNAS | Lloyd, McKay & Furl, 2022) https://replications.clearerthinking.org/replication-2022pnas119-4

Key Links

• Our Research Box for this replication report includes the pre-registration, de-identified data, and analysis files. 

Overall Ratings

To what degree was the original study transparent, replicable, and clear?

Transparency: how transparent was the original study?	The data, code, and materials were publicly shared; however, the shared materials were missing important data cleaning steps, and the analysis scripts required substantial editing in order to reproduce the original results. The experiment also had a bug which we fixed. The primary weakness in transparency was that the study was described as pre-registered, but there were major deviations from the pre-registration, and these were not acknowledged in the paper or supplementary materials. We outline the discrepancies in a table.
Replicability: to what extent were we able to replicate the findings of the original study?	The main finding did not replicate, but there was a trend toward significance and the effect was in the same direction in our replication dataset as in the original study.
Clarity: how unlikely is it that the study will be misinterpreted?	We think the findings have more limited generalizability than the paper suggested, and several alternative explanations for the findings could have been discussed. Finally, certain terms were used in ways that were counterintuitive and also inconsistent with the paper from which the terms were derived.

Detailed Transparency Ratings

Overall Transparency Rating:
1. Methods Transparency:	There were some missing elements, but we were provided with further materials on request, and the missing information did not prevent us from replicating the study. The authors hadn’t provided the original Gorilla code on their online repository, or the specific wording of the ACE questionnaire, but they provided these promptly following specific requests. The Gorilla code still required several changes before it was able to function as expected. The authors also hadn’t included their participant-facing description of the study (including how the bonus system was described to participants), but they provided details about how they told participants about the bonus on request.
2. Analysis Transparency:	We outline ways in which analysis transparency could have been improved in a table in the appendix.
3. Data availability:	A cleaned version of the dataset was publicly available and we were able to reproduce the original results using it.A de-identified version of the raw dataset, however, was not publicly available, so we confirmed that the data cleaning steps worked by doing them on freshly generated data.
4. Preregistration:	The original paper stated that the study was pre-registered, but there were major deviations from the pre-registration, none of which were acknowledged in the paper or supplementary materials. We outline the discrepancies in a table in the appendices of this report.

Summary of Study and Results   

The study we replicated sought to investigate how decision-making in adults is impacted by childhood experiences of adversity. The scientific aim of the original paper was to “inform our understanding of the computational mechanisms underlying different decision-making strategies associated with early adversity and their relationship with risk-taking behaviors.” The authors also recognized the ethical implications of this research, highlighting “the need for children to be protected from these experiences.”

The titular result of the original paper (which we refer to hereafter as the “headline result” or “main result”) was that adults “with adverse childhood experiences explore less and underweight reward feedback.” In their concluding paragraph, the authors summarize their study as having “demonstrated that ACEs are associated with reduced exploration and with underweighting positive-reward feedback in a patch-foraging paradigm.” 

The authors reached this conclusion by recruiting people from both trauma support groups and the general population, then administering a widely-used computerized apple-picking task intended to track differences in individuals’ tendencies to explore (versus exploit) environments with different distributions of rewards (different distributions of apples per “harvest”). They also administered an ACEs questionnaire to all participants.

To put the original paper’s main result more precisely: participants with scores of four or above in the study’s ACEs questionnaire (i.e., those classified as being in the “high ACE” group) explored less in the experimental task than individuals in the “low ACE” group. The authors made that claim based on a mixed analysis of variance (ANOVA), with the richness of the digital foraging environment as the factor that varied within all participants (i.e., the within-subjects factor) and ACE group as the factor that distinguished one group from another (i.e., the between-subjects factor). 

The analysis evaluated the average number of apples left at the time a participant switched from one tree to the next, for the last two trees of that environment; the authors refer to this as the “leaving threshold”. In the original dataset, there was a statistically significant main effect of group (having high vs. low ACEs): the high ACE group had fewer apples left in each tree on average (i.e., staying longer) before they switched to new trees (i.e., they tended to “exploit” for longer and didn’t tend to “explore” the next tree until later). 

We did the same analysis in our replication dataset. In our replication, the main effect of group was not significant, although our dataset did show a trend toward significance [F(1,144) = 2.975, p = 0.087, η² = 0.019], with effects in the same direction as in the original study. 

The original pre-registration did not include a plan to run an ANOVA but instead had said that t-tests would be the analyses of interest. So, although t-tests were not reported in the original paper or supplementary materials, we did run t-tests on both datasets, comparing leaving thresholds between ACE groups (within both environments separately and also on average overall). The tables below show the results. 

As seen in the tables above, the original study had different degrees of freedom for the t-test comparison within the rich environment compared to the poor environment. That was because there were six participants who had missing leaving thresholds in the rich environment.¹

In the original study, participants also completed an additional questionnaire after the experimental task, but as described in the next section, that questionnaire wasn’t administered in the replication study, since the analyses involving that questionnaire produced null results in the original study. 

In the in-depth sections that follow, we explain aspects of the original study design that meant that some alternative explanations for the headline results couldn’t be tested until we amended certain aspects of the study design. We also discuss the t-test results in more depth and suggest reasons that the generalizability of the original study findings may be limited.

Study and Results in Detail

This section explains the recruitment procedures, study tasks, exclusion criteria, data cleaning, and data analysis steps, as well as going into more detail about the results presented above.

Recruitment methods

The original study used two complementary recruitment methods:  

“To selectively recruit participants who had been exposed to ACEs, we advertised among four international charities and support groups for adult survivors of childhood trauma. These were the following: Survivors South West Yorkshire, the National Association for People Abused in Childhood (NAPAC), The Survivor’s Trust, and one anonymous support group. Control participants were recruited from a recruitment platform (Sona Systems; https://www.sona-systems.com/) hosted by a United Kingdom–based university and through Prolific (https://www.prolific.co/). The Prolific sample was recruited from the same regions that the charities were based in the United Kingdom and Europe”

We clarified this with the authorship team, and they informed us that most participants in the high ACE group were recruited from the charities/support groups, while a small proportion of them were recruited through Sona/Prolific. 

In our study, all participants were recruited via Positly. We also used complementary recruitment strategies, including some specifically targeting those who’d be expected to have higher ACE scores. More specifically:

• For some of our experiment “runs” on Positly, the only participants who were shown the study or able to participate were those who had previously completed an ACE questionnaire in a past study on the same platform and who had a score of at least 4 recorded on our system from that past study. 
• The other “runs” on Positly allowed any participants to participate. 
• All participants were recruited via Positly, and at the end of our study, all participants still completed the same ACE questionnaire version as the one in the original study.

Consent and study description

In both the original study and our replication, participants were told (as part of the consent form) that they would complete some computerized tasks and that they would answer questions about their childhood, which might be stressful. The original authors rightly highlighted the importance of considering the ethical implications of a study like this. As part of the consent form, participants were informed that they could withdraw from the study at any time without penalty. A copy of the consent form is in the appendices.

Those who consented were then given instructions on the simulated apple foraging task. They completed a practice run followed by the actual task. At the completion of the task, participants were directed to questionnaires. (The differences between their questionnaires and ours, which did not affect the replicability rating, are explained in a later section.)

Overview of the task

The task was administered in Gorilla^TM Experiment Builder. We requested and obtained the original code from the original authors. Screenshots of the experiment are included below as well as in the original paper.

The foraging task instructed participants that they would be presented with a “tree” of “apples” and that they’d have to choose whether to “stay” at the tree (by pressing “S”) to collect apples or to “leave” the tree (by pressing “L”) to move to another tree. 

If a participant chose to “stay” at a given decision point, they were next presented with a picture of a set of apples (representing the number that they had “picked” due to having selected to “stay”) and were simultaneously shown a number on the screen (representing an overall cumulative score so far). 

If they chose to “leave,” participants were presented with a stationary cartoon of a person walking before being presented with the next tree and the next decision point (to stay or leave). If they didn’t make a decision within 3 seconds, this was treated as a timeout, and they were presented with the still image of the walking person to symbolize moving to the next tree. 

Each tree had progressively fewer apples to “pick” each time the participant “stayed” at a given tree, and the rate at which apples were depleting depended on which “environment” the participant was in for that part of the experiment (explained below). 

The number of apples left at the point where a participant moved to the next tree was called the “leaving threshold.” The “average” leaving threshold for a given environment was the average number of apples “from the last two harvests,” which is also what Constantino & Daw (2015) did in their stochastic depletion experiments.

S keypress (“stay” decision)

S keypress (“stay” decision)

L keypress (“leave” decision)

Instructions given to participants

After consenting, participants were presented with the following instructions:

“Your aim is to collect as many apples as possible within the time limit. The more apples you collect, the larger your score at the end of this experiment and the bigger your prize will be.

• You can either stay to continue picking apples from the current tree or leave and find a new tree. If you leave and travel to a new tree, you have to wait a fixed amount of time. This time is fixed and has nothing to do with your internet connection or page loading.
• You will only need your keyboard for this task. You can either press ‘S’ to stay with the tree or ‘L’ to leave the tree and find a new one.
• You will not know how many apples are on a new tree until you stay and pick them, so it is a good idea to stay with each tree at least once before moving on.
• The number of apples left on a tree will decrease with time, meaning there will be fewer apples left on the tree to collect the longer you stay there. Apples do not grow back on each tree, so your job is to decide how long you want to spend at each tree.

After seven minutes you will move into a completely new environment (think of it as a new orchard). This environment may be richer or poorer than the others. In some environments it may be better to stay with a tree for longer and in others it may be better to stay with a tree for less time.

After completing the study, you will earn an additional bonus payment of up to $3, based on your score!

You will now begin a quick practice run of the study. Your practice score will not count towards your score in the main task. The task should take 16 minutes altogether (including the practice).”

“Rich” and “Poor” Environments

In both the original study and our replication, each participant was exposed to two different environments. In both the original and in our replication, the order of the two environments was counterbalanced across participants.  

In the original study, the environments were distinguished by the following features: 

• Rich environment: apples reduced more gradually; travel time between trees was 6 seconds
• Poor environment: apples reduced more rapidly; travel time between trees was 12 seconds

In our replication, the environments had the following characteristics: 

• Rich environment: apples reduced more gradually (consistent with original study); travel time between trees was 6 seconds (consistent with original study) 
• Poor environment: apples reduced more rapidly (consistent with original study); travel time between trees was 6 seconds (shorter than original study to maintain consistency in travel time between environments) 

The reason we kept travel time the same between the environments (6 seconds between trees for both environments) was so that the only characteristic that differed between the environments was the rewardingness of the trees, not the costs of switching trees. We thought it was important, like the original paper mentioned in its introduction, to just vary one of those things at a time.³ 

This point seems especially important given that the original study demonstrated that people in the high ACE group tended to stay longer (compared to those in the low ACE group) when harvesting from trees in the poor environment, but the same did not apply in the rich environment. That finding could have been consistent with either those with high ACEs having higher sensitivity to costs, or reduced reward sensitivity, or a combination of both, but because both costs and rewards were being varied concurrently, we can’t disentangle the effects based on the original study. The replication study only varied rewards between environments, to simplify the interpretation of results.⁴ 

Questions after the experimental task

Following the task, participants in the original experiment did both an ACE questionnaire and the Domain-Specific Risk-Taking scale (DOSPERT) survey (Blais & Weber, 2006). We did not administer the DOSPERT survey because the analyses of data from that survey yielded null results in the original study. As we discuss in the Clarity section, instead of the hypothesis that involved the DOSPERT-related analyses, we are instead only focusing on the headline result from Hypothesis 1a for the replicability rating of this study.

Instead of the DOSPERT, we administered a cognitive task. We administered the cognitive task prior to the ACE questionnaire to avoid potentially negatively impacting participants’ performance by reminding them of adverse childhood experiences (where applicable).⁵

Cognitive Task

The original study collected information pertaining to educational attainment among the participants but did not investigate the possibility that cognitive abilities differed between the two groups. Since we hypothesized that this could have been one of the potential explanations behind their findings, we included a three-minute-long cognitive task based on a study of over 3,000 people by Clearer Thinking. An intelligence quotient (IQ) was predicted for each participant based on their performance in those tasks.

Data exclusion criteria 

For our main analysis, we used the same rules for excluding observations as the original study. We note that the original study’s pre-registration ( https://osf.io/8znyx/registrations ) stated that “During the task, trials where participants timeout (i.e. do not provide a response in the allocated time) will be excluded, as this does not provide information about participants’ leaving values and is therefore uninformative in the analysis.” However, following further correspondence with the original authors, it became clear that their final exclusion criteria for timeouts differed from their pre-registration. Timeouts in the original study resulted in the exclusion of both the individual timed out trial and all preceding trials for that particular foraging patch (i.e., a given apple tree). In their words:

“It is important that patches where participants timed out are not included in the leaving threshold analysis, as they do not tell us anything about when participants chose to leave their current patch and explore a new one. Therefore, it is important to exclude data from patches (not just trials) where participants timed out. This process can be quite laborious. However, it is a necessary step.”

In accordance with what the original study did and the authors’ recommendations, our replication excluded all trials from any foraging patch (tree) where a participant timed out.

After a set number of forages on a given tree (33 in the poor environment and 60 in the rich environment), participants were forced to advance. These thresholds represent the number of forages required to receive zero apples for more than one screen for any possible tree for that environment. These were treated as timeouts in our analysis. For our main analysis, even if a participant stayed in one or more of their runs until there were zero apples left, we still included their other data from other runs. For our supplementary analysis, we excluded participants if they attempted to continue to select “stay” after zero apples had already displayed for more than one trial, since that suggested they were not being adequately attentive.

Lastly, anyone who didn’t complete the whole study was excluded. As a result of the exclusion criteria above, two people were excluded from the dataset. These two participants never chose to pick apples and always timed out.

Data cleaning and analysis

The data cleaning steps were not shared on the original OSF site with the study materials. Fortunately, the original authorship team readily shared them with us upon request. Data cleaning instructions were shared as a word document with written instructions for spreadsheet manipulation to prepare data for analysis. The manual and non-standard data cleaning materials introduced unnecessary labor and opportunities for human error, reducing the original study’s transparency rating. The rating could have been improved by sharing a more standardized, accessible version of data cleaning materials, such as an R script.

The analysis code was provided on the OSF site with study materials, but was not sufficient on its own to reproduce their results – significant additional editing was required to be able to reproduce their results. 

The study was pre-registered, and that fact was mentioned in the paper. However, the specific statistical tests reported in the paper had no overlap with the tests listed in pre-registration; this change in analysis methods was not acknowledged in the paper or supplementary materials. 

Results in detail

The main finding did not replicate in our dataset, although there was a trend in the same direction as the original results. 

The mixed ANOVA described earlier was run on our replication dataset (once again comparing the average number of apples left before a participant switched to another tree, with foraging environment as the within-subjects factor and ACE group as the between-subjects factor). Consistent with the original study, we did find a significant main effect of environment [F(1,144) = 28.576, p < .001, η² = 0.011]; participants switched to the next tree when there were more apples left (on average) in the rich environment compared to the poor environment. The headline result, though – the main effect of group – was not significant in our replication, although our dataset did show a trend toward significance [F(1,144) = 2.975, p = 0.087, η² = 0.019], with effects in the same direction – i.e., participants classified into the high-ACE group had a trend towards leaving when there were fewer apples left (i.e., they took longer to switch trees – they “explored” less) than those in the low-ACE group. As in the original study, there was again no significant interaction between environment type and ACE exposure [F(1,144) = 0.071, p = 0.791, η² < 0.001].

Here is another copy of the results table from the summary section (included again here to spare our readers from scrolling).

Simplest Valid Analysis: t-test results

We also ran t-tests on both the original and replication datasets, comparing leaving thresholds between ACE groups (within both environments separately and also on average overall). The tables below show the results (shown here again for convenience). 

As mentioned, the original study had different degrees of freedom for the t-test comparison within the rich environment compared to the poor environment because there were six participants who had missing leaving thresholds in the rich environment.

Interpretation of the results 

The original paper made it very clear in their abstract and in the body of the paper that some of their hypotheses had not been supported by their experimental findings. They also clearly communicated what the experiment involved and what the effect sizes were; the graphs were particularly clear in that they displayed key variables using violin plots and included outliers (instead of displaying box plots alone, for example). 

However, the paper’s clarity suffered with respect to its discussion of the model they used and the implications of their findings. The implications of the findings deserve special mention, so they are discussed in detail below.

Generalizability and validity concerns

The paper did not go into much detail discussing the limits of the study’s face validity and ecological validity, or the related topic of its generalizability. With respect to the findings contained in the original paper, there was already some tension between the headline result and the lack of significant results for hypothesis 2, which is arguably the most directly connected with real-world outcomes (since the survey asks about participants’ real-world behaviors). In the original paper, hypothesis 2 was “that ACE-related decision strategies would lead to real-world problematic outcomes in the form of a positive relationship between ACEs and self-reported risk-taking.” 

The original paper described a series of regressions – each one focused on a different subscale of the DOSPERT (which assesses risk-taking in financial, health/safety, recreational, ethical, and social domains). Each DOSPERT subscale was entered as the outcome variable of a regression, with ACE score, gender, and age as predictors in each case. None of these regressions yielded significant results. These null findings were only briefly explored in the results and discussion, without the further implications of this being elaborated upon. 

The lack of significant findings in those regressions casts doubt on the degree to which the results in favor of hypothesis 1a can be taken to represent something with real-life implications. This point might have been interesting to explore further, especially given that reward networks appear to be involved in risk-taking behaviors in other studies (e.g., Wang et al., 2022). Even if reward pathways were different between the two groups (which was the implied explanation for hypothesis 1a), it seems they weren’t different enough to result in significant findings in the regressions testing hypothesis 2.

This tension might point to a broader problem with the paradigm used in this experiment. We would argue that it has limited face and ecological validity for testing individual differences in adults’ general tendencies towards exploration in everyday life. If so, this would substantially limit the generalizability of the original paper’s results, even if those results had replicated. To be clear, this problem does not uniquely apply to this paper, but to many that use this paradigm.⁷ We are also far from the first to write about this. For example, Hall-McMaster & Luyckx (2019) pointed out that “current task designs involving random encounters with choice items do not reflect situations in which animals can make use of their knowledge in the environment to encounter items strategically.” Real-life choices between exploration and exploitation involve leveraging experience and expectations about unexplored environments. Decision-makers also understand that these environments are dynamic, potentially offering varying rewards over time and/or in relation to other variables. It may be that simplified tasks (such as the current apple foraging task) are too far removed from practical decision-making to be a representative measure of exploration.

Even setting aside real-world generalizability, it also remains to be seen whether findings from the apple foraging task would consistently generalize to other experimental explore-vs-exploit tasks. Some evidence suggests that task-specific factors can prevent generalizability between different paradigms. An example is given in von Helversen et al. (2018). In that study, 261 participants completed three different paradigms, each designed to study “exploration–exploitation trade-offs.” None of those tasks used the apple-picking paradigm, but all of them were designed to study participants’ tendencies towards exploration. Structural equation modelling suggested that there “was no single, general factor underlying exploration behavior in all tasks, even though individual differences in exploration were stable across the two versions of the same task.” This study is only indirectly suggestive, but it at least raises questions about the degree to which psychological explore-exploit paradigms can specifically isolate and measure tendencies towards exploration, as opposed to also eliciting individual differences related to other (non-exploration-related) tendencies (which could interact with task-specific factors). 

Robustness concerns 

The original paper could have been clearer if they had also explained the pre-registration deviations and their implications – including the paper’s lower generalizability in the context of the non-robust significant finding. The paper does not discuss the fact that the originally-planned t-tests yield a null result within the rich environment. If this had been discussed, the potential non-robustness of the main results could have been clearer to readers.

Lack of clear explanations for missing data

There were six participants in the original study whose leave thresholds in the rich environment were missing and who had “NA” recorded there instead in the file on OSF. These missing rich environment leave thresholds were not explained in the paper or supplementary materials as far as we could see. An understanding of why that data were missing may have helped with interpreting the original study results.

Comments on model specification

The paper states that the study’s findings “demonstrate the negative impacts on reward-processing that are associated with adversity in childhood.” It also states: “Using computational modeling, we identify that reduced exploration is associated with ACE-exposed individuals underweighting reward feedback, which highlights a cognitive mechanism that may link childhood trauma to the onset and maintenance of psychopathology.” Some readers might interpret these statements as if the authors had ruled out more hypotheses than they actually had.The paper implies that the explanation for the study findings was that those in the high ACEs group were underweighting reward feedback. However, there were other possible explanations.

The original paper’s explanation that they were employing one specific model of learning in this task – and that other models could also have been used – was made quite clear in their methods section. As the original paper notes, the Marginal Value Theorem employed to describe learning in an apple foraging task first introduced by Constantino & Daw (2015) – is “a prominent” one. This, of course, does not imply that the model employed in their analyses was the only model that could have been used. For example, they also noted (in the methods section) that they “compared this model, which uses only a single learning rate for all outcomes, to a model in which the learning rate was split for better-than-expected and poorer-than- expected outcome.” Although that wasn’t covering all the comparisons they said they would make in their pre-registration, they avoided leaving the reader with the false impression that the model tested was the only thing that could have explained the data collected. 

Notwithstanding that, the potential for other explanations of participants’ foraging behavior (aside from MVT) was not explained as much as it could have been in the current study. Only one other potential model was mentioned (and the calculations for it were only shown in the supplementary materials). The authors only briefly discussed the possibility of other models explaining the ways in which participants may have been engaging with the task, but this was part of the description of the methods and was not discussed in more detail elsewhere. That is despite the fact that a lot of the later discussion depended on the model on which they chose to focus.

Alternative hypotheses

Beyond the choice of computational model, there are other  explanations for the original headline result that can’t be ruled out based on the original study design. Below, we list some examples that could have at least partly contributed to the differences between groups in the original study. Some of these possible explanations were discussed in the original paper, which is noted where applicable.

• Cognitive differences – The study looked at educational attainment as a possible variable that could explain performance differences between the groups, but did not look at cognitive differences. A recent meta-analysis showed small-to-medium negative associations between ACEs and overall cognitive control (g = −0.32), as well as between ACEs and each of the following domains of cognitive control: working memory (g = −0.28), cognitive flexibility (g = −0.28), and inhibitory control (g = −0.32) (Rahapsari et al., 2025).To briefly explore this possible explanation, we included a three-minute cognitive task after the foraging task (and before the ACE questionnaire). We did not find significant differences in the cognitive performance of the two groups in our dataset. This is discussed further in the appendix. 
• Cost Sensitivity – The groups could have differed in their sensitivity to costs, in addition to or instead of rewards. The paper acknowledges that the original study varied two things (both rewards and costs) between the two environments, but in its overall conclusions does not address the possibility of costs having contributed. It seems to us that the original study results could have been related to either different reward processing, or different processing of costs, or both, or neither.
• Travel time – We made the travel time between trees consistent between the two environments in our replication in order to remove the potential confounding effect of different costs across environments. But the fact that there was more than one thing varying between environments in the original experiment introduced additional explanations for the original study results which were inadequately explored in the original discussion, and that reduced the clarity of the paper.
• Stress levels – The original paper discusses differences in stress level between the two groups as a possible alternative explanation that could not be ruled out by the study design. To quote from the paper: “We did not control for rates of stress, which mediate the association between ACEs and adult psychopathology (49). State and trait stress have been associated with decreased exploration in a foraging paradigm.” The clarity of the paper is improved by acknowledging this potential confound. 
• Undetected confounds – Differences in recruitment methods between the two groups could have potentially introduced other confounding variables we haven’t considered (e.g., different levels of access to online spaces, different socioeconomic backgrounds, and so on). The original study did conduct a comparison between the groups checking for differences in age or educational levels, which showed no significant difference between the groups). This suggests the authors were aware of potential confounds, and took reasonable steps to evaluate whether differences (other than ACE group membership) could have influenced the findings. We include the risk of undetected confounds here because that is always a possibility with study designs of this type. 

Other issues relating to clarity 

In addition to the generalizability and validity, robustness, and potential alternative explanations, there were a few more minor issues relating to how easily a reader may understand and interpret the paper. The most important of these is the mis-labeling of a key variable used in the paper’s computational model.

As mentioned above, the original paper based their conceptualization of the task and the application of the marginal value theorem (MVT) to it on Constantino & Daw (2015). In that paper, Constantino & Daw introduce a depletion parameter (κ). In the current paper, that parameter is instead called the depletion rate, even though the depletion rate is actually inversely proportional to the depletion parameter. This would introduce unnecessary confusion to readers, since they would wonder why an environment labeled as having a higher depletion “rate” has, in fact, a lower depletion rate. Using the same term as Constantino and Daw (i.e., depletion parameter) could have avoided some of that confusion.

In other words, the paper uses the term “depletion rate” when referring to what the original study called a “depletion parameter,” which had a different meaning (i.e., it is not a synonym for “depletion rate”). When apples depleted faster, the current paper labeled this as having a lower “depletion rate,” which is the direct opposite to what readers would intuit. Instead, the paper could have stayed with the original term of “depletion parameter” and could have thereby avoided that confusion.

The other smaller clarity issues that we identified are discussed in the appendix.

Conclusion

We attempted to replicate a study that had shown that adults with high levels of adverse childhood experiences (ACEs) tend to spend longer picking apples before moving to a new tree in a computerized foraging task, compared to adults with fewer ACEs. This finding was used to infer that people with high ACEs tend to explore less than those with low ACEs. The main finding didn’t replicate in our study, although there was a trend in the same direction as the original experiment. 

The paper received a moderate rating for transparency because the  experimental materials, analysis code, and data for the study were publicly shared; however, major pre-registration deviations were not disclosed. We also found that several factors limited the paper’s clarity. We concluded that the findings have more limited generalizability than the paper suggests. The paper could also have benefited from more discussion of possible alternative explanations for the findings. Finally, there were some more minor clarity issues, such as mis-labeling a key term.

Acknowledgements

We thank the participants for their valuable time. We also thank the original authorship team, who were responsive, helpful, polite, and always ready to review our replication materials and report when we asked. Many thanks go to Amanda Metskas and Spencer Greenberg for leading Transparent Replications and providing invaluable guidance and feedback throughout this replication.

Authors’ Response

Thank you to the Transparent Replications Team for their important work, though we are of course disappointed that our findings did not replicate in this sample. We will take their feedback on board to improve our Open Science practices in the future.

One potential reason for the lack of replication may be due to the differences in recruitment methods between our original study and the replication. The sample from our 2022 paper were recruited through charities that support individuals for specific traumatic events (e.g., physical or sexual abuse), meaning our sample of individuals with 4 or more ACEs would have had high rates of threatening experiences (as defined in dimensional models of childhood adversity; McLaughlin et al., 2016, Current Directions in Psychological Science). It may be that cases of threatening experiences are lower in the community sample recruited for the replication project and that their high-ACE sample comprised participants reporting more experiences of neglect or family disruption (e.g., parental divorce, having an incarcerated parent). These potential differences between the samples are important to consider as it has been proposed that specific forms of childhood trauma may differentially impact processes relevant to these studies (McLaughlin et al., 2016). However, we recognise this explanation would need empirical testing in future research.

We think the alternative explanations proposed for our findings would be interesting to consider in future longitudinal work on this topic and whether processes such as cognitive differences or cost sensitivity may mediate the association between ACEs and explore/exploit choices in adulthood.

Once again, we thank the Transparent Replications Team for their careful work.

Signed, 

Alex Lloyd on behalf of the authors

Purpose of Transparent Replications by Clearer Thinking

Transparent Replications conducts replications and evaluates the transparency of randomly-selected, recently-published psychology papers in prestigious journals, with the overall aim of rewarding best practices and shifting incentives in social science toward more replicable research.
We welcome reader feedback on this report, and input on this project overall.

Appendices

Additional information about transparency ratings

Analysis transparency

Pre-registration deviations in the original study

The following table details the main ways in which the paper deviated from the pre-registration document for the original study.

Unacknowledged deviations

Other notes

Additional Information about the Methods

Preview of the study

You can preview the study at this link.

The consent form

Below is a copy of the consent form:

Informed Consent Statement

Please read this consent statement carefully before deciding whether to participate.

About this study

Participation will involve playing a virtual apple foraging game where your goal is to explore different foraging patches and gather as many apples as you can. You will have an opportunity to gain additional compensation based on your score in the foraging game.

Participating in this study will take no longer than 25 minutes.

Risks in participating:

The principal disadvantage of participating in this study is the time it will take you to participate in the testing session. After the computerized tasks, you will be asked some questions about your childhood, which might be stressful to read. If these questions cause you distress, you may withdraw from the study at any point.

Benefits of participating:

This research is not intended to benefit you personally. The main benefit of participation is the monetary compensation you’ll receive for participation. (In addition to an initial sum of $4.50 for taking part, you could make up to an additional $3, dependent on your score on the study task. Higher scores on the task will mean you get a bigger payment at the end of the study.)

Confidentiality:

Any information you provide will not be personally linked back to you. Any personally identifying information will be removed and not published. By participating in this study, you are agreeing to have your anonymized responses and data used for research purposes, as well as potentially used in write-ups and/or publications.

Participation and Withdrawal:

Your participation in this study is completely voluntary, and you have the right to withdraw at any time without penalty, though you will not be paid if you do so. To withdraw, simply close this browser tab at any time.

Contact Information:

For general questions about the study or what it involves, or if you have any technical problems relating to completing the questions, please contact us at: replications@clearerthinking.org

If you have questions about your rights as a research participant, contact the Human Research Ethics Committee, HKU (+852 2241-5267). Approval number: EA240437

Additional methodological differences between the original study and the replication

Travel Time

We kept the travel time the same (6 seconds) across the rich and poor environments so that only the reward rate (rather than also the time cost involved) was changing across environments. We did this because, as the authors pointed out in their introduction, it seems useful to vary the reward independent of cost in order to isolate the effects of differences in rewards. We note the original authorship team’s comment on this change: “With regards to the changes you have made to the experiment, I don’t have any concerns about the proposed changes – thank you for checking. However, I would flag that by only changing the reward rate (and not the travel time as well), there is likely to be a smaller effect size of the environment quality (rich versus poor) on participants’ leaving thresholds. Therefore, you may need a larger sample size to detect this effect. However, as this is not the primary aim of the replication then it may not be a concern and your exploratory analysis would be sufficient to examine the difference in leaving thresholds between those with 4 or more ACEs, and those with fewer/no ACEs.”

A note about the number of apples per row

For the visual depiction of how many apples had just been “picked” every time a participant chose to “stay,” we displayed eight apples per row instead of six. Everything else about the visuals was kept the same as the original. We did this just in case having six apples per row had been giving participants a visual cue which encouraged them to prefer a five or six apple leaving threshold. 

The original experiment had up to six apples displayed per row (when apples were displayed at each “foraging” turn), which was close to the optimal number of apples after which participants “should” switch trees, on average (for an optimal number of points, according to the equations that the authors theorized that participants’ decisions might be well-modeled by, the optimal threshold was 7.04 in rich environments and 5.07 in poor environments). Our experiment included eight apples per row and yet still showed similar leaving thresholds to the original experiment. This strengthens the original authors’ claim that their decision-making equation can be used to model participants’ decisions, since it appeared to predict behavior even when the maximum number of apples per row was disentangled from the optimal leaving threshold.

The score bug

In the course of setting up the study, we uncovered a bug affecting how the original scores were displayed beyond a certain number of trials. We found that the score displayed to participants became inaccurate after 32 trials (at a given tree) in the rich environment. This seems related to a bug in the task script which caused later trials to refer to incorrect cells in the spreadsheet used to populate score values and stimuli.

For example, choosing to “stay” for the 35th time might display zero apples, but the total score still increased from trial to trial (as if the participant had moved on to a new tree). In light of that bug, we let the original authorship team know and then altered the task script to accurately display the total score in the rich environment regardless of the number of “stay” decisions. 

In the course of replicating this study, we discovered a bug affecting how some scores were displayed in the original experiment. In response to the email where we told the original authorship team about this, they said: 

“I must admit I haven’t seen the bug affecting the scores in my data, so thank you for bringing this to my attention. My suggestion for fixing this bug would be to i) add additional columns to the spreadsheet to calculate the depletion from scores after 32. To do this, you will need to add three columns for each new score drawn.”

Extending the maximum number of forages

This was done in the process of fixing the score bug that we found in the original experiment. The original experiment forced participants to advance after fewer “stay” decisions. We ensured that all possible trees would reach a forced advance only after every possible tree had passed more than one forage producing zero apples.

Early task completion bug

Some participants experienced a bug which would interrupt the foraging task and direct participants to a screen erroneously displaying a “Task Complete” message, and interfering with redirection toward the cognitive task. This was fixed by including timed screens in the Gorilla builder. As a result, participants occasionally spent a few extra seconds in each environment as Gorilla processed the screen timer transitions. As opposed to an abrupt cut-off exactly seven minutes into a foraging environment, participants who reached the seven-minute mark were allowed to complete the current tree before being advanced to the next environment. Participants who encountered the bug were not included in our data set. It is not clear to us whether this bug did or did not affect any participants in the original study.

Notes on sample size

Regarding sample sizes: as per our pre-registration, using GPower, we calculated the effect size in the result in the original study to be 0.1758631 (for an eta squared of 0.03), and 75% of that effect size is 0.132 (rounded to 3 decimal places).

Putting 0.132 into GPower, along with 2 groups (high and low ACEs) and 2 measures (average patch residency in rich and poor environments respectively) with a correlation between the average patch residency in those 2 environments being 0.70 (rounded to 2 decimal places, based on the data from their original study), we found a total required sample size minimum of 94. This would correspond to 47 people per group. In this project, in cases where our sample size calculations are lower than the original experiment sample size, we tend to default to the higher of the two values (i.e., in this case, the original experiment sample size) as the minimum sample unless there are good reasons not to do so. In other words, we defaulted to being well above the required sample size to have adequate power to find an effect that could have even been smaller than the original study.

The original authors had 47 people in the high ACE group and 98 in the low ACE group for a total of 145 across the two groups. For our replication, we decided to collect data from at least 47 eligible people per group and 145 in total across the two groups.

We stopped data collection after a data check had shown that each group had exceeded the minimum number of eligible participants, and the total number of participants had exceeded 145. Then we conducted the analyses as pre-registered.

Additional Analyses

Exploratory results

Results relatively unchanged with different exclusion criteria

In our pre-registration, we said: “For our supplementary analysis, we may exclude people if they attempted to continue to select “stay” after zero apples had already displayed for at least one trial.” Though this was an optional point, we did check to see how many would be excluded if they attempted to continue to select “stay” after zero apples had already been displayed for at least one trial. It turned out that only two participants were excluded following that criterion, and when we reran the main ANOVA with those two participants removed, we still obtained a non-significant result but with the effect still trending in the expected direction. The main effect of ACE group was F(1,142) = 1.473, p = 0.227. (So our replicability rating would have been the same whether we used our main or supplementary analyses.)

No significant differences in learning rates or cognitive performance between the groups

For all of the following other exploratory findings, Shapiro-Wilk tests showed significant deviations from normality (p < 0.001) so we performed Mann-Whitney U tests. 

Learning rates

We checked for differences in alpha values derived from the MVT model described in the paper, for which lower values imply higher learning rates (as explained in the methods section of the original paper). We did not find statistically significant differences. The mean alpha parameter in environment 1 (referred to here as sym_alpha_en1 as in the original study) among those in the high ACE group was 0.389, and in the low ACE group it was 0.345; U = 2126, n₁ = 91, n₂ = 55, p = 0.129. The mean alpha parameter in environment 2 (sym_alpha_en2 for those in the high ACE group was 0.400, and for those in the low ACE group it was 0.363; U = 2215, n₁ = 91, n₂ = 55, p = 0.246).

Cognitive task results

In our pre-registration, we wrote: “We will also assess the estimated IQ (based on the scoring methods associated with the screening test we are using from Clearer Thinking) in both the high and low ACE group, and we will conduct an independent t-test comparing IQ estimates between the two groups. If there is a difference or a trend towards a difference between the groups, we may conduct follow-up analyses examining whether IQ correlates with task performance and/or implied learning rate.” 

If there had been intelligence differences between groups in the original study (which we cannot rule in or out based on our results), it would have been an open question as to the degree to which factors contributed to the results in that study (e.g., underweighting reward feedback, general cognitive differences, or other factors). 

In our replication dataset, we did not achieve the original headline result, so there is less to explain in the first place. However, we still went forward with our comparison, as per our pre-registration. We compared the estimated intelligence quotient (IQ) between the two groups in our dataset. There was no significant difference in the mean normalized scores on the cognitive tasks between those in the high ACE group (M = 105.160, SD = 2.080) and those in the low ACE group (M = 105.860; SD = 2.213); U = 2962, p = 0.064.

Additional notes about clarity 

Overall, we gave this paper a Clarity rating of 2.25 stars. The main reasons for this have been outlined in the body of this report. There were some additional (more minor) points that informed this study’s Clarity rating. These are explained below.

• Visual clarity issues: As shown in the images of the task earlier, if participants had not read the experimental instructions carefully, they may have been confused about some aspects of the experiment. For example, participants may have focused on either the number of apples, or the number shown as the cumulative score, or both. Whether they focused on one, the other, or both could have affected their behavior in the experiment.
• Additional terminology issue: The body of this piece explained how one of the parameters was mis-labeled in the paper. There was an additional, less important terminology issue as well: the original paper referred to higher and lower “leave thresholds” in a way that was technically correct but which could be misinterpreted by some readers. In the paper, a “high” leaving threshold referred to the tendency to leave sooner while there were still more apples left, i.e., leave more readily. A “low” leaving threshold referred to the tendency to leave later, when there were fewer apples left, i.e., leave less readily. Some readers might think that a “low” leaving threshold would instead refer to leaving more readily. It may have been preferable for the authors to instead refer to the “average remaining apples per trial” or something else that more literally communicated what the variable in question represented.
• Supplementary table error: Supplementary Table 1 appears to have swapped the labels for the rows for the “dual learning rate” and the “single learning rate” models. The text implies that lower values were obtained for the single learning rate model, which would be consistent with the paper, which also states this. If that’s the case, though, then the table rows that have been mis-labeled.

References

Blais, A. R., & Weber, E. U. (2006). A Domain-Specific Risk-Taking (DOSPERT) scale for adult populations. Judgment and Decision making, 1(1), 33-47.

Constantino, S. M., & Daw, N. D. (2015). Learning the opportunity cost of time in a patch-foraging task. Cognitive, Affective, & Behavioral Neuroscience, 15, 837-853.

Faul, F., Erdfelder, E., Buchner, A., & Lang, A. G. (2009). Statistical power analyses using G* Power 3.1: Tests for correlation and regression analyses. Behavior research methods, 41(4), 1149-1160.

Hall-McMaster, S., & Luyckx, F. (2019). Revisiting foraging approaches in neuroscience. Cognitive, Affective, & Behavioral Neuroscience, 19(2), 225-230.

von Helversen, B., Mata, R., Samanez-Larkin, G. R., & Wilke, A. (2018). Foraging, exploration, or search? On the (lack of) convergent validity between three behavioral paradigms. Evolutionary Behavioral Sciences, 12(3), 152.

Lloyd, A., McKay, R. T., & Furl, N. (2022). Individuals with adverse childhood experiences explore less and underweight reward feedback. Proceedings of the National Academy of Sciences, 119(4), e2109373119.

Rahapsari, S., & Levita, L. (2025). The impact of adverse childhood experiences on cognitive control across the lifespan: A systematic review and meta-analysis of prospective studies. Trauma, Violence, & Abuse, 26(4), 712-733.Wang, M., Zhang, S., Suo, T., Mao, T., Wang, F., Deng, Y., Eickhoff, S., Pan, Y., Jiang, C. & Rao, H. (2022). Risk‐taking in the human brain: An activation likelihood estimation meta‐analysis of the balloon analog risk task (BART). Human brain mapping, 43(18), 5643-5657.

1. We asked the authorship team about this. The lead author responded: “Thank you for bringing this to my attention. I do not have a record of reasons for these missing data, so have no additional context to provide.” ︎
2. The results shown in the table are the ones derived after excluding the six participants in the dataset who had missing leaving thresholds in the rich environment. If those people were instead included, the t-test results would have been different: t(143) = 1.825, p = 0.070. ︎
3. The original authors already pointed out that varying both depletion rates and travel time represented an issue with the original study. To quote the original paper: “Future research could address this limitation by comparing environments with long and short travel times, while independently manipulating fast and slow depletion rates (e.g., ref. 24). Administering environments more than once (e.g., ref. 50) might further enhance the effect of environment quality on foraging behavior that we observed in the current study.” ︎
4. As we found out later, in the replication, in which the costs of switching trees were consistently lower across environments, there was no statistically significant difference between the leaving thresholds between those in the high versus low ACE groups (whether we look at rich or poor environments separately or look at the results overall). This could be consistent with sensitivity to costs driving the original results, but since we did not test that hypothesis directly in the replication, this is only speculation at this point. ︎
5. As per our pre-registration, the analyses involving the cognitive tasks were only relevant to the Clarity rating of this paper, and did not affect the replicability rating at all. ︎
6. The results shown in the table are the ones derived after excluding the six participants in the dataset who had missing leaving thresholds in the rich environment. If those people were instead included, the t-test results would have been different: t(143) = 1.825, p = 0.070. ︎
7. Please note that the paradigm has been widely used and cited. The paper introducing it has been cited 279 times since its publication in 2015, according to Google scholar. ︎

After emailing the survey to more than 2,500 academic psychologists and promoting it on relevant listservs, our newsletters, and social media, we received 87 fully completed surveys and another 123 that answered at least some of the substantive questions we asked. These 210 respondents indicated that they were all either experts or experts-in-training in psychology or a related field. There were additional participants who did not meet our screening criteria because they are not experts or experts in training in relevant fields, so their data were excluded from all analyses.

While we attempted to reach a wide range of academic psychologists without biasing the sample in any particular direction, our sample of respondents is, of course, not going to be perfectly representative of the field. For example, we can’t rule out the possibility that those who chose to respond were more likely to have certain opinions than a perfectly random sample of academic psychologists. For more information about the participants and to access the anonymized data from the study, see the appendix.

Here’s what we learned about how psychologists think the field is doing:

1. They believe the replication crisis is real and still happening, but that meaningful progress has been made

Nearly two-thirds of the participants in our study believed that the replication crisis was a real, serious issue, but that substantial progress has been made to address it. Nearly one-third believed it was a serious issue, but that little progress has been made. Only 5% of participants believed it was either never an issue, or that it was an issue that had been completely solved.

It’s striking to see how strong a majority there is for the belief that there still is a replication crisis ongoing, though most with that view also believe that substantial progress has been made.

2. Psychologists predict that 55% of new studies in top journals would replicate (median estimate). The median prediction increases to 75% if the field were healthy.

In order to get a more quantitative assessment of how academic psychologists believed the field was doing with respect to replicability, we asked them to predict how likely a study in a top journal would be to replicate under different conditions. The question we asked was:

What percent of studies published in the last 12 months in what you consider to be one of the top 5 psychology journals do you think would replicate in a high powered (i.e., 99% power) replication that is completely faithful to the original study design?

The first version of the question established participants’ baseline prediction when considering “all such studies.” We then modified the question by asking about more specific circumstances to see how that changed psychologists’ predictions about replicability (with the conditions of the baseline question still applying, such as just considering recent published papers in the top 5 psychology journals). The list of circumstances we asked about in order was:

• All such studies [baseline question]
• All such pre-registered studies
• All such studies with a main finding with p<0.001
• All such studies with a main finding with p between 0.001 and 0.01
• All such studies with a main finding with p between 0.01 and 0.04
• All such studies with a main finding with p between 0.04 and 0.05
• All such studies if the field were in a healthy state

The overall question and each of the 7 circumstances were displayed on screen at the same time, and participants responded to each one using a slider ranging from 0% to 100% in 1% increments. The chart below shows the median percentage of studies that academic psychologists predicted would replicate under each of the circumstances, with the red bar showing the median replication prediction for the baseline circumstance “All such studies.”

The median estimated replication rate for the studies “if the field were in a healthy state” was 75%, 20 percentage points higher than the 55% median for the current replicability rate. Studies with p < 0.01 and those that are pre-registered were predicted to be more likely to replicate overall (than the baseline), while those with p-values between 0.01 and 0.05 being estimated to be less likely to replicate. Interestingly, respondents see pre-registered studies and also studies where a main finding has p<0.001 as being fairly close to as likely to replicate as if the field were in a healthy state (70% and 70% compared to 75%).

This suggests that academic psychologists believe that the field has some work left to do, but that they believe that pre-registration is an effective tool for improving replicability, and that very small p-values are a meaningful indicator of replicability.

It is worth noting that the median values for the predicted replication rate were a little bit higher than the mean values for the top three questions. The means and medians were consistent with each other for the other four questions. From the distributions of responses it appeared that there were low outliers that were skewing the mean values for the top three questions. For that reason, we decided to focus on the median values in this discussion. The chart below displays the mean and median values for comparison.

Interestingly, in an analysis we conducted on an unrelated data set, where we examined 325 studies that had undergone replication, when the original study’s p-value was less than or equal to 0.01, about 72% of the papers replicated (very slightly higher than the 60.5% to 70% range of estimates in this study). But when p was larger than 0.01, only 48% replicated (within the 40% to 50% range of estimates in this study). However, exact numbers are likely to vary depending on the data set used, as it likely varies with field and topic.

We were somewhat surprised that the median replicability given “if the field were in a healthy state” is only about 75% – our team anticipated people would say that a healthy replicability would be more like 80%-90%. To determine these numbers, consider our chart below. If studies are designed to have a reasonable level of statistical power (e.g., 80%), and there is a 50% prior chance of a hypothesis that’s studied being true, and p<0.05 is achieved on the first try, 94% of such results should be replicable.

3. Researchers report that they have changed their own research practices, and indicate that they estimate 5 to 7 percentage points higher replicability for their future papers compared to their past papers.

Consistent with the belief that the field is improving, the vast majority of academic psychologists reported that they had made changes to their research practices in the last 10 years to improve the quality, rigor, or robustness of their research. Of the 139 people who responded to this question, 115 reported that they had conducted research in the last 10 years. In addition to these participants, there were 24 people who said they had not conducted research in the last 10 years, who aren’t included in the chart below. Among respondents who had conducted research in the last 10 years, 88% said they had made changes to improve their research during that time.

We asked an open-ended follow-up question to those who reported that they had made changes to their research, asking, “What are the most important changes you have made over the last 10 years to improve the quality, rigor, or robustness of your research?” The most common change researchers reported making was pre-registering their studies. Participants also mentioned publicly sharing data, materials, and analysis code; as well as increasing sample sizes and conducting power analyses. 

We asked a checkbox follow-up question to those who indicated that they had done research in the past 10 years, but had not made changes, about why they hadn’t made changes. The most common response was that they were already using best practices (9 out of 14 people).

In addition to asking psychologists about changes to their research practices, we also asked them to assess the replicability of their own past and planned future work. Below are the mean, median and modal percentages of their own work that academic psychologists expect would replicate in a high-powered replication.

Comparing the mean, median, and modal responses, psychologists assigned a higher likelihood of replication to their planned future work than to their past work by 5 to 7 percentage points. 

A paired-samples t-test shows that the higher mean predicted replicability for psychologists’ planned future studies compared to their predictions about their past studies is modest, but statistically significant (p < 0.001), suggesting that academic psychologists are slightly more optimistic about the replicability of their future work than their past work.

Participants, on average, predicted that their own past work would replicate at a higher rate than their predicted replication rate for top journals in the field overall, and they predicted that their future work’s replicability rate would exceed the replicability rate for top journals if the field were in a healthy state. Perhaps it’s not surprising that people perceive their own work to be above average, but it does suggest that the “healthy state” prediction participants made may be a little low or their assessment of their own future work may be excessively high. 

Since people are likely to be overly positive in their assessments of their own work, we don’t think the baseline replicability assessments people provide for their own work are especially useful for understanding the state of the field; however, we do think comparisons between participants’ assessments of their own past work and their own future work may provide useful insights. For example, if participants thought their future work would replicate at the same rate as their past work, it would suggest that they planned to use the same research practices in future work as they used in the past. Researchers saying that they expect their future work would be more likely to replicate than their past work suggests that they are changing their research practices in ways they believe will improve the replicability of their future work compared to their past work.

4. Academic psychologists believed that, if there were a substantial likelihood of a visible replication soon after publication, that would change colleagues’ research practices

We asked participants to consider how their colleagues’ research practices might change if there was a substantial chance of a highly visible replication of their paper being performed shortly after publication. The bulk of respondents chose either “moderately likely” (40%) or “highly likely (29%).

When given a list of 12 possible research practices that people might change, with the ability to check any number of them that they thought their colleagues might be more likely to do if replication was more common, the most popular answers were larger sample sizes, power analysis, not submitting findings researchers lacked confidence in, and pre-registration. The full list is in the table below:

Key Takeaways

The academic psychologists who responded to our survey still see the replication crisis as an ongoing, serious problem; however, they also see improvement over the last decade. This is most clearly reflected in nearly two-thirds of psychologists selecting the response, when asked about the replication crisis in psychology, “There currently still is one, but substantial progress has been made toward improving the situation during the last ten years, so it’s not as bad as it used to be.”

We also see this belief about the state of the field reflected in the psychologists’ answers to other questions in our survey. Academic psychologists predicted that, at present, only 55% of studies published in top five psychology journals would replicate, whereas the median prediction if the field were in a healthy state was that 75% would replicate. There is a 20 percentage point gap between where psychologists believe the field is today, and where they believe it should be in terms of replicability, which serves as additional evidence that academic psychologists see the replication crisis as an ongoing issue.

There is also further evidence in this survey that academic psychologists believe that progress has been made in addressing the replication crisis. The vast majority (88%) of participants who conducted research over the last decade reported that they have made quality, rigor, or robustness improvements to their own research practices. Experts also predicted a 5 to 7 percentage point improvement in the replicability rate of their own planned future studies compared to their own previously published studies, suggesting that participants believe that their future research practices will be more robust than those used in some of their previously published work.

Additionally, more than three-quarters of academic psychologists surveyed reported that they believe their colleagues would be at least moderately likely to make changes to their research practices if there was a substantial chance of a highly visible replication attempt shortly after publication. This suggests that academic psychologists believe that their colleagues respond to incentives when making research decisions, and that sufficiently large changes in the incentives around the use of best practices may have a good chance of increasing adoption of these practices.

How do psychologists’ perceptions of the field compare to how the field is actually doing? We ran replications on 12 randomly-selected, recently published, studies from top journals, and what we found diverges in a few unexpected ways from the predictions of experts in the field. Part two of this series explores those results.

This article is the first in a four-part series. For more of what we learned, check out Part 2 on our first dozen replication attempts.

Appendix: Demographics of Survey Participants and Anonymized Data

Demographics

Education in Psychology or a related field

Of the 210 participants who considered themselves experts or experts in training, participants listed the following education levels:

Note that a few of these participants may not seem to qualify as experts or experts in training on the basis of their answer to this question. We used participants’ self-identification for the main data analysis. The main data analysis excluded 63 people who participated in the survey, but indicated that they were not experts or experts in training in psychology or a related field.

Subfield

We asked a few more basic demographic questions at the end of the survey, so the responses below only include participants (N = 87) who made it all the way to the end of the study.

Age

Note that one participant’s age was excluded because it was reported as 10, outside of the reasonable range for the study. 

Gender

Ethnicity

Participants were asked “Which of these categories describe you? (Select all that apply).” 

Anonymized Dataset and Open Ended Responses

Anonymized .csv dataset

This .csv file includes anonymized closed-ended survey responses for the 210 participants included in the main data analysis. Note that most demographic information is not included in the anonymized dataset file to prevent the identification of individual participants.

Anonymized open ended responses

This .pdf includes the open-ended responses to the survey from the 210 participants included in the main dataset. The order of these responses has been randomized so they do not correspond to the order of the anonymized dataset. These responses have been lightly redacted to remove specific examples and other comments that may have allowed the identification of participants.

Transparency	Replicability	Clarity
	N/A

Study 2 from this paper tested whether prompting people to rate the accuracy of news headlines would affect how likely people were to share false headlines versus true headlines—especially among people who habitually share news on Facebook.

While trying to reproduce the results from the study using the original paper’s data and analysis code, we found many different issues. These issues included claims that did not match the provided evidence, statistical results that could be unreliable, group comparisons based on values beyond a scale’s possible range, inflated effect sizes due to a statistical artifact, and numbers we could not reproduce with the original data and code. Given these issues, we decided not to replicate the study. To be clear, the findings of Study 2 could be true, but we consider the evidence presented inadequate for supporting the claims made in the paper.

Generally, when the Transparent Replications team conducts replications, we focus on only one study within a paper (you can learn more about our process here). This can be thought of as similar to a spot-checking approach. If the work in a scientific paper is reliable, it shouldn’t matter which study we choose. That said, it’s of course possible that, by chance, we choose a study that is far less reliable than the other studies in the paper. In this case, we briefly reviewed the other three studies in the original paper to see if they had the same issues as Study 2. Each of the other studies contained some, but not all, of the same issues as Study 2. These are described at the end of the report. Only study 2 was reviewed thoroughly, and it is the study on which the ratings are based.

This study received a transparency rating of 3.75 stars because the materials, data, and code were publicly available and the study was preregistered, but the preregistration was not followed and these deviations were not acknowledged in the paper. The paper did not receive a replicability rating because we did not attempt to replicate it. The paper received a clarity rating of 0 stars because the quantity and severity of the issues we encountered would almost certainly cause readers to misinterpret the results of the study.  

Finally, it is important to highlight that the authors of the original paper disagree about the severity of many of the issues we describe. You can see their full response here, and we have included each point they make at the end of every corresponding section in our report.

Full Report

Study Diagram

Evaluation Conducted

We evaluated study 2 from: Ceylan, G., Anderson, I.A., & Wood, W. (2023). Sharing of misinformation is habitual, not just lazy or biased. Proceedings of the National Academy of Sciences, 120(4). https://doi.org/10.1073/pnas.2216614120

How to cite this report: Transparent Replications, Handley-Miner, I. & Metskas, A. (2025). Evaluation of a study from “Sharing of misinformation is habitual, not just lazy or biased” (PNAS | Ceylan, Anderson, & Wood 2023). Clearer Thinking. https://replications.clearerthinking.org/replication-2023pnas120-30
(Report DOI: https://doi.org/10.5281/zenodo.17705056)

Key Links

• Our OSF repository includes the supplemental materials for this report.
• You can read the original paper here.
• You can review the supporting materials for the original paper including the preregistration, data, analysis code,  and experimental stimuli.

Overall Ratings

To what degree was the original study transparent, replicable, and clear?

Transparency:  how transparent was the original study?	The study materials, analysis code, and data were publicly available. The study was preregistered, but the preregistered analyses were not followed. Deviations from the preregistration were not acknowledged (the paper states, “We preregistered all hypotheses, primary analyses, and sample sizes.”)
Replicability: to what extent were we able to replicate the findings of the original study?	N/A
Clarity: how unlikely is it that the study will be misinterpreted?	We uncovered many significant issues with how the study was conducted, analyzed, and reported. These issues include claims that don’t match the provided evidence, statistical results that could be unreliable, group comparisons based on values beyond a scale’s possible range, inflated effect sizes due to a statistical artifact, and numbers we cannot reproduce with the original data and code. We think that these issues, collectively, will almost certainly cause readers to misinterpret the results of the study.

Detailed Transparency Ratings

Overall Transparency Rating:
1. Methods Transparency:	The materials were publicly available and were complete
2. Analysis Transparency:	The analysis code was publicly available and complete, but could not successfully run on the provided data
3. Data availability:	The data were publicly available and almost complete, and authors gave remaining data on request
4. Preregistration:	The study was preregistered, but the preregistration was not followed, and the fact that the preregistration was not followed was not acknowledged

Summary of Study and Results

Summary of methods and results

In Study 2, 839 participants recruited from Amazon Mechanical Turk saw 16 news headlines (8 false headlines and 8 true headlines). Participants were asked to imagine that they were seeing these news headlines while scrolling their Facebook newsfeed. 

Participants completed two tasks for each headline: 

• The Share Task asked participants, “If you were to see the article on Facebook, would you share it?” 
• The Accuracy Task asked participants, “To the best of your knowledge, is the claim in the above headline accurate?”

Half of the participants completed the share task first for all of the headlines (control condition) and the other half of participants completed the accuracy task first for all of the headlines (treatment condition).  

Study 2 had two stated outcomes of interest: spread of misinformation and discernment. Spread of misinformation was operationalized as the number of false headlines that participants chose to share. Discernment was operationalized as the relationship between whether a news headline was shared and whether it was true. Participants who shared more true headlines and fewer false headlines were considered more discerning.   

After completing both tasks, participants completed several measures. Of those measures, the only important one for understanding Study 2 was the news sharing habit measure. This measure asked participants to answer four questions on a 7-point scale (1 = disagree; 7 = agree):

• Sharing news on Facebook is something I do without thinking 
• Sharing news on Facebook is something I do automatically
• Sometimes I start sharing news on Facebook before I realize I’m doing it  
• Sharing news on Facebook is something I do without having to consciously remember  

(Note: as described later, there was an error in the wording of one of these questions; the wording above is the wording the measure was reported in the paper to have had.)

The primary analysis conducted on the data was a generalized linear mixed effects model that predicted whether participants chose to share or not share each headline by the three-way interaction between the following variables:

• Veracity: whether the news headline was true or false
• Condition: whether the participant completed the share task (control condition) or the accuracy task (treatment condition) first
• News Sharing Habit: the participant’s score on the news sharing habit measure 

This analysis approach meant that discernment was assessed via the relationship between veracity and sharing. The more likely participants were to share true articles and not share false articles, the greater their discernment.   

Study 2 reports the following results: 

• no significant three-way interaction between veracity, condition, and news sharing habit
• a significant two-way interaction between veracity and news sharing habit, such that discernment (the relationship between veracity and sharing) was lower among participants with higher news sharing habit scores
• a significant two-way interaction between veracity and condition, such that discernment (the relationship between veracity and sharing) was higher when participants rated accuracy first
• a significant effect of condition, such that participants shared fewer headlines when they completed the accuracy task first

Study 2 also reports several direct comparisons of the amount of true and false news headlines shared between participants who had a news sharing habit score of at least one standard deviation below the average (“weak habitual sharers”) and participants one standard deviation above the average (“strong habitual sharers”). 

Study 2 concludes by stating, “Thus, highlighting accuracy proved useful in reducing the spread of misinformation but not among the most habitual users. Echoing the first study, 15% of the strongest habit participants were responsible for sharing a disproportionate amount of misinformation—39% across all experimental conditions (habit estimated from SRBAI, 30% with habit estimated from past frequency).”

Our overall assessment of Study 2 

The role Study 2 plays in the narrative of the paper is to claim that prompting social media users to consider accuracy does not cause habitual sharers to share less misinformation. In fact, the study’s title is “Considering Accuracy Does Not Deter Habitual Sharing: Study 2.” Here are other examples of this claim from the paper:

“Thus, highlighting accuracy proved useful in reducing the spread of misinformation but not among the most habitual users.”

“Once sharing habits have formed, they are relatively insensitive to changing goals through accuracy primes (4) and the display of metrics such as how many people scrolled over a post (29). Thus, existing individual-frame interventions remain relatively ineffective for the habitual sharers who are most responsible for misinformation spread on these platforms.”  

Our conclusion is that this general claim is based on an incorrect interpretation of a nonsignificant 3-way interaction in a statistical model that had the following issues: 

• It used a measure of news sharing habit that had an error in the question wording
• It was run on Amazon Mechanical Turk data that had no quality checks 
• It did not converge 
• It was underpowered 
• It was not an analysis included in the preregistration (but was claimed to be)

Additionally, we found many issues with the reported results, including:

• Almost all of the numbers reported were model predictions, but were not stated to be so
• The reported results (which turned out to be model predictions) for weak habitual sharers are misleading because it was not possible for a participant to have a low enough sharing score to be considered a weak habitual sharer
• The way discernment was calculated may have inflated the differences between stronger and weaker habitual sharers
• Figure 3 is reported as being Study 1 data, but it appears to be created using Study 2 data
• There were many numbers we could not reproduce or that were calculated incorrectly

Many of these issues cannot be fully corrected without the collection of new data (e.g., the lack of data quality checks, the low statistical power, the lack of preregistration for the primary model). Although, in principle, a replication could solve these issues, we feel that there is insufficient conceptual clarity regarding the hypotheses and claims of the study to know what a replication should test. Additionally, given the number of errors we found, it is unclear what should count as successfully replicating the findings. Given these issues, we do not plan to conduct a replication of the study. 

It is also important to note that after encountering the issues described above in Study 2, we did a quick review of the other three studies in the paper to see if they contained any of the most significant issues we detected in Study 2. We detected many of the same Study 2 issues in Studies 1 and 3, but fewer issues in Study 4. Nevertheless, Study 4 still contained several minor instances of some of the Study 2 issues and also had some numerical errors. 

The next section provides specific details for all of the issues we encountered in the paper. 

Study and Results in Detail

Description of issues with Study 2

After reading through the paper and trying to reproduce the results from Study 2 using the original data and analysis code, we encountered many issues with the implementation, analysis, reporting, and interpretation of the study. The following issues are described in detail in the subsections below: 

• The primary claims don’t match the provided evidence
• The primary claims are based on statistical results that could be unreliable
  • There was an error in one of the key measures
  • Participants were not evaluated with quality checks
  • The statistical model failed to converge
  • Central claims rely on null results, but the study is likely underpowered
  • The preregistration was not followed, but was claimed to have been followed
• Most numbers supplied in the paper are model predictions, not direct descriptions of the data
• The way discernment was calculated may have inflated the difference between strong and weak habitual sharers
• The study contains errors and numbers we cannot reproduce
  • Figure 3 reports the wrong study’s data and contains errors
  • There were many numbers we could not reproduce
  • There were many issues in the analysis code

We consider the fact that the primary claims don’t match the provided evidence to be the most significant individual issue. 

We think the next most significant issues are that (a) Amazon Mechanical Turk participants were not evaluated with quality checks, (b) the study regularly discusses “low habitual sharers,” who are a group that cannot exist because they are defined as having a News Sharing Habit score that is below the lowest possible score (explained in the section titled “Most numbers supplied in the paper are model predictions, not direct descriptions of the data”), and (c) the way discernment was calculated may have inflated the difference between strong and weak habitual sharers. 

While the remaining issues are more minor, we think that the quantity and diversity of issues is perhaps the biggest concern for this study because it is difficult to tell how they, collectively, impacted the study results. 

That said, it is important to point out that the authors of the original paper disagree about the severity of many of the issues we describe. You can see their full response here, and we have included each point they make at the end of every corresponding section in our report.

The primary claims don’t match the provided evidence

The main claim put forward in Study 2 is stated in the study’s title: “Considering Accuracy Does Not Deter Habitual Sharing: Study 2.” The paper makes several similar claims when discussing the results from Study 2:

• “Thus, highlighting accuracy proved useful in reducing the spread of misinformation but not among the most habitual users.”
• “Priming accuracy concerns prior to sharing had only a modest impact on the discernment of everyone and did not ameliorate high habitual sharing of misinformation (Study 2).”
• “Once sharing habits have formed, they are relatively insensitive to changing goals through accuracy primes” 

As a reminder, the claims are based on a statistical model (a generalized linear mixed effects model) that predicted whether participants shared a given news headline by the three-way interaction between headline veracity (i.e., whether the news headline was true or false), experimental condition (i.e., share task first or accuracy task first), and the participant’s score on the news sharing habit measure. The following results from this model were reported in the paper: 

• no significant three-way interaction between veracity, condition, and news sharing habit
• a significant two-way interaction between veracity and news sharing habit, such that discernment (the relationship between veracity and sharing) was lower among participants with higher news sharing habit scores
• a significant two-way interaction between veracity and condition, such that discernment (the relationship between veracity and sharing) was higher when participants rated accuracy first
• a significant effect of condition, such that participants shared fewer headlines when they completed the accuracy task first

The full fixed-effects results of the model are shown in the table below:

Taken at face value, the non-significant three-way interaction means that the study did not find evidence that the experimental condition affected the relationship between participants’ news sharing habit scores and their sharing discernment. Here’s another way of thinking about this: According to the results from the statistical model, people with higher news sharing habit scores showed worse discernment than people with lower habitual sharing scores and the model did not find strong evidence that asking people to rate accuracy first changed this difference in discernment between higher and lower habitual sharers. 

However, the non-significant three-way interaction does not mean that “considering accuracy does not deter habitual sharing.” For example, imagine that rating accuracy first caused a large and equal increase in discernment for every single participant. This would not produce a significant three-way interaction because higher habitual sharers would still show relatively worse discernment than lower habitual sharers. But, it would have still improved discernment for higher habitual sharers—it just would have also improved discernment for low habitual sharers.     

The study does, however, come close to interpreting the results correctly elsewhere: “In general, rating accuracy first did not increase the discernment of strongly habitual users any more than less habitual ones.” This interpretation is not perfectly correct because it interprets the null result as evidence of no difference (i.e., “did not increase the discernment”). This is a very common error in scientific papers, encapsulated by the adage, “absence of evidence is not evidence of absence” (Aczel et al., 2018). That aside, the paper’s interpretation does, in this instance, correctly note that the three-way interaction speaks to whether the experimental condition had different effects on participants’ sharing discernment depending on their Sharing Habit Scores.

In sum, the provided evidence does not support the titular claim, “Considering Accuracy Does Not Deter Habitual Sharing: Study 2.” Similar claims should be revised to reflect the finding that the experimental condition caused participants to be more discerning (on average) and that the study did not find evidence that this differed by participants’ news sharing habit scores. 

In communication with the authors after drafting this report, they noted:

We interpreted the lack of three-way interaction based on the data pattern I shared with you in the reactions document.

You are making a thought experiment but frankly, you can just examine the pattern of the data.

The data is showing us that everybody (both high and low habitual users) their sharing slightly, supporting the lack of three-way interaction.

(Note that the referenced “reactions document” refers to the document linked to in the “Response from the Original Authors” section.)

The primary claims are based on statistical results that could be unreliable 

In addition to the primary claims not matching the provided evidence, we identified several methodological and statistical issues that suggest that the results from the main analysis may not be a reliable test of the primary questions of interest. 

As a reminder, the statistical model used in Study 2 is a generalized linear mixed effects model and one of the central claims in the study is based on a null result for the three-way interaction between headline veracity, experimental condition, and news sharing habit score.  

The five issues we identified—an error in the news sharing habit measure, an Amazon Mechanical Turk sample that wasn’t vetted for response quality, a generalized linear mixed effects model that didn’t converge, low statistical power, and a preregistration that wasn’t followed—suggest that interpretation of the null result on the three-way interaction that the primary claims in Study 2 rely on should be done cautiously, if at all.  

The subsections below explain each of these five issues in detail. 

There was an error in the news sharing habit measure

The primary claims in this study concern how people’s decisions about whether to share news headlines differ depending on their scores on the news sharing habit measure. This measure is described in the paper as follows:

“In all studies, habit strength was measured with four 7-point scales (1 = never to 7 = always, adapted from the self-report habit index, (36): “Sometimes I start sharing news on Facebook before I realize I’m doing it,” “Sharing news on Facebook is something I do without thinking,” “Sharing news on Facebook is something I do automatically,” and “Sharing news on Facebook is something I do without having to consciously remember.” The items were averaged into a composite measure of habit strength (ɑ = 0.89).” 

As can be seen in the publicly available materials for Study 2, one of the four sharing habit questions mistakenly says “reading” instead of “sharing” in the survey:

This error was mentioned by at least three participants in comments they provided at the end of the survey. For example, one participant said:

“There is one matrix that asks about your news sharing, specifically how automatic sharing/reading news is. The first battery was all about reading news,. The second matrix was mostly about sharing news, but there was one that was about reading news, that I think may have been an error (not changed when the question was copied/pasted)”

Given this error, one of the four questions that comprised a key independent variable for Study 2 measured a different psychological construct. In theory, this should make the news sharing habit measure a worse measure of participants’ news sharing habits.   

In communication with the authors after drafting this report, they noted:

As you pointed out, there is a typo in one of the measures. However, all 4 items in the scale are highly correlated. If you drop this item, you will see that the results still hold. We also used other habits measures (reading habits and frequency of sharing). While reading habits are a weaker predictor compared to sharing habits but the results hold using any of these scales. 

Amazon Mechanical Turk participants were not evaluated with quality checks

Study 2 recruited 839 participants from Amazon Mechanical Turk, and used the responses from all 839 participants in the analyses. 

Amazon Mechanical Turk is an online crowdsourcing platform that is well-known for having high proportions of inattentive and/or fraudulent participants (i.e., bots or people who click through surveys at random in order to receive study payments) (see Chmielewski & Kucker, 2020; Stagnaro et al., 2024; Webb & Tangney, 2022). It is still a useful platform for collecting data, but careful quality checks are needed to weed out unreliable participants (Cuskley & Sulik, 2022). The Transparent Replications team has a lot of experience running studies on similar web-based platforms, and we find that it is common to have high rates of spam if no quality checks are implemented. This study did not use any quality checks, so a portion of the data could be noise from people or bots randomly clicking through the study. All else equal, adding random noise to a dataset will make null results more likely (e.g., for the three-way interaction tested in the statistical model). 

Moreover, Study 2 finds that participants with higher news sharing habit scores tend to be less discerning—i.e., they are worse at distinguishing between true news and false news—compared to those with lower news sharing habit scores. If a group of inattentive participants took this survey completely at random, one would expect them to have an average news sharing habit score of about 4 (the midpoint of the scale) and to not discern between true and false news headlines. The distribution of scores on the news sharing habit measure is highly skewed such that most participants have very low scores:

So, scores around the midpoint of the scale are “high” relative to most participants. As such, having a group of inattentive participants might make it more likely that you would find the effect that participants with higher news sharing habit scores tend to be less discerning. 

To be clear, this does not mean that the effect observed in Study 2 was generated by inattentive participants—we cannot know without knowing which participants were inattentive. However, it demonstrates one of the issues with not using data quality checks. 

In communication with the authors after drafting this report, they noted:

We have replicated these results many, many times, and it is implausible that the result is due to noise instead of habit strength.  We even built habits in Study 4 to demonstrate causality of the effect. We have included quality checks such as attention checks and elimination of duplicate ip addresses in subsequent research, and we have obtained comparable results to those in the set of studies published in PNAS.

The statistical model failed to converge

Generalized linear mixed effects models estimate many parameters, and thus will sometimes fail to converge on a solution. When a model fails to converge, the results can be inaccurate. So, it’s good practice to resolve convergence errors to ensure the model results are reliable (Seedorff et al., 2019). The version of the model run in Study 2 failed to converge and, judging by the code shared by the authors, there were no attempts to find a version of the model that successfully converged to ensure the results were accurate.  

In communication with the authors after drafting this report, they noted:

We computed many models including participants and headline fixed effects and decided to report the most comprehensive and conservative model. We also computed a model without random effects, and with an optimizer (control = glmerControl(optimizer = “bobyqa”). In all these cases, models converged and results remained virtually identical. The consistent results despite different models attest to the robustness of the effect. We did not include these  in the web appendix because our focus was on reporting the other models including the various covariates requested by reviewers.

Central claims rely on null results, but the study is likely underpowered 

When describing the statistical power for Study 2, the paper states, 

“For Studies 2 and 3, with multiple experimental conditions, we increased the sample size to 400 per condition. With 16 stimuli and a sample size of 400 per condition, these studies have a power of at least 0.75 to detect an effect (d) of approximately 0.45.” 

To put the effect size of d = 0.45 into context, we can examine Figure 3 in the paper, which shows the size of the relationship between the amount of false news participants shared and various measures used throughout this paper:

The two largest effect sizes come from the news sharing habit measure (referred to as “Social media habits (SRBAI scale)” in the figure) and Past Sharing Frequency. Because the amount of false news shared is measured in the study as the number of false headlines shared—not as the amount of false news relative to true news shared—one would expect the news sharing habit measure and Past Sharing Frequency to have a strong relationship with the amount of false news participants shared. After all, both measures tap into how readily participants share news. If people share more news, they will likely share more false news articles, in total. The effect size for both of these measures was less than d = 0.45. 

(Note: The structure of the paper implies that the data in Figure 3 come from Study 1. However, we are confident that this data is in fact from Study 2, which we explain in detail later. As such, the news sharing habit measure has the wording error discussed earlier. This could have affected the effect size presented in Figure 3.)

The main claims in Study 2 rest on a nonsignificant three-way interaction between the news sharing habit measure, the experimental condition, and the veracity of the news headline. It seems unlikely that a three-way interaction between these variables would exceed the effect size of the simple relationship between the news sharing habit measure and the amount of false news one shares. As such, we think this statistical model was likely underpowered to detect a significant effect for the three-way interaction. 

Improving the statistical power of this study would likely require including more stimuli. As Westfall et al. (2014) demonstrate, statistical power for study designs and statistical models like the ones used in this paper depend equally on how many participants and how many stimuli are used in the study. In other words, having low numbers of participants or stimuli can put an upper bound on the achievable statistical power. As shown in Figure 2 of Westfall et al. (2014), if a study has 16 stimuli, increasing the number of participants beyond 200 does very little to increase the statistical power.

In communication with the authors after drafting this report, they noted:

The power analysis we reported is for the focal effect, which is the interaction between habits scale and headline veracity. We are able to detect this effect even with 200 participants. Since we added a between-subjects variable (question order), we increased the sample size 4 times, which is in line with standard practices in the field. New approaches to power analysis with mixed effects offer various recommendations on how to calculate power. Even a recent paper suggests that power analysis does not lead to reliable results especially for mixed effect models (Pek, Pitt, and Wegener 2024).

Pek, J., Pitt, M. A., & Wegener, D. T. (2024). Uncertainty limits the use of power analysis. Journal of Experimental Psychology: General, 153(4), 1139.

The preregistration was not followed, but was claimed to have been

The paper says, “We preregistered all hypotheses, primary analyses, and sample sizes (except Study 1).”  Study 2 was indeed preregistered, but the preregistered hypotheses and analyses were not what was reported in the paper.  

Here is the hypothesis section of the preregistration:

According to this section, Study 2 was concerned only with the relationships between habitual sharing and false news sharing and between habitual sharing and “truth discernment.” (Note: we assume “truth discernment” was meant to be “sharing discernment” given the context of the preceding sentence in the preregistration. “Truth discernment” generally refers to correctly identifying headlines as true or false, whereas “sharing discernment” generally refers to preferentially sharing true headlines over false headlines.) 

There were no hypotheses related to the experimental manipulation of prompting participants to share first or rate accuracy first. Yet, the paper frames the experimental manipulation as central to the goals of Study 2. Study 2 opens with:

“One potential explanation for habitual sharing is that people share indiscriminately when they are not able or motivated to assess the accuracy of information. In this account, habitual sharers spread misinformation just because strong habits limit attention to accuracy. To test this possibility, we examined whether highlighting accuracy prior to sharing would reduce the habitual spread of misinformation and increase sharing discernment (4).”  

And here is the analysis section of the preregistration:

The primary preregistered statistical model predicts participants’ sharing behaviors by the news sharing habit measure, headline veracity, and the interaction between the two. However, the primary statistical model reported in the paper predicts participants’ sharing behaviors by the news sharing habit measure, headline veracity, experimental condition, and all two-way and three-way interactions between those variables. 

We think that the statistical model reported in the paper is a more parsimonious way to test the various relationships the authors appear interested in testing given the other preregistered analyses. It is possible the authors came to the same conclusion after preregistering this study. However, the primary purpose of preregistering analyses in advance is to limit analytical flexibility. Thus, if deviations from the preregistration are made, the original preregistered analyses should still be reported and analyses that were not preregistered need to be labeled as such. Instead, this paper claimed, “We preregistered all hypotheses, primary analyses, and sample sizes (except Study 1).”

Since the primary analysis was not preregistered and there were no preregistered hypotheses related to the experimental condition, we think the results from this model are less likely to be reliable. 

In communication with the authors after drafting this report, they noted:

This is an interesting claim. Our central prediction was for a two-way interaction. We did not expect that this effect would be modified by question order, and thus we did not specify the three-way interaction in the preregistration. Instead, we outlined the core, central effect we expected to be significant. We are unaware of any guidelines specifying that nonsignificant effects need to be preregistered. 

Summary of reasons why the statistical results might be unreliable

Taken together, these five issues—an error in a key measure, an Amazon Mechanical Turk sample that wasn’t vetted for response quality, a model that didn’t converge, low statistical power to detect effects under d = 0.45, and a preregistration that wasn’t followed—cast doubt on whether the null result on a three-way interaction that the primary claims in Study 2 rely on should be interpreted as evidence for the study’s claims.  

Most reported numbers were model predictions, but were not stated as such

The paper frequently reports numbers for Study 2 that sound like descriptive statistics. For example:

• “Habitual participants shared 42% of the true headlines and 26% of the false headlines”
• “weak habit participants were 1.9 times more discerning than strong habit ones”
• “rating accuracy first reduced participants’ sharing of false headlines (M_{accuracy first} = 9%; M_{sharing first} = 13%)”
• “15% of the strongest habit participants were responsible for sharing a disproportionate amount of misinformation—39% across all experimental conditions”

Additionally, the figure used to represent the results from Study 2 (Figure 4), depicts the probability of sharing true and false headlines among “strong” and “weak” habitual sharers in the two experimental conditions:

The most straightforward interpretation of statements like, “Habitual participants shared 42% of the true headlines and 26% of the false headlines” is that if you tallied up the number of true and false headlines that habitual participants shared, you would find that they shared 42% of the true headlines and 26% of the false headlines. 

However, as far as we can tell from the results generated by the authors’ analysis code, almost every number reported in Study 2 comes from the fixed effects results from the generalized linear mixed effects model they ran or estimated marginal means calculations for specific values of the dependent variables. In other words, almost all of the numbers provided in Study 2 are model predictions. 

One of the biggest issues this presents for Study 2 is that many of the statistics reported are for a group of participants that cannot exist. 

The paper often contrasts “strong habitual sharers” and “weak habitual sharers.” For example, Figure 4 (shown above) directly compares strong habitual sharers (green bars) to weak habitual sharers (blue bars). To define these groups, the study considers all participants whose score on the habitual sharing measure is at least one standard deviation above the mean as strong habitual sharers and all those whose score is at least one standard deviation below the mean as weak habitual sharers. 

However, in this sample of participants, the news sharing habit measure, which ranged from 1-7, had a mean of 2.26 and a standard deviation of 1.40. So, one standard deviation below the mean was a score of 0.87. However, the lowest possible score on the scale was 1. So, no participant could have had a score of at least 1 standard deviation below the mean. The figure below shows the distribution of scores on the sharing habit measure, with the cut-offs for the strong and weak habitual sharers. 

So, statements like “Less habitual participants shared 13% of the true headlines and 3% of the false headlines” are misleading, because it was impossible for a participant to meet the criterion for being a “less habitual participant.”

More generally, if you calculate statistics directly from the data, instead of using model predictions, many of the numbers are quite different. For example, the figure below compares the numbers provided in the study’s main figure for strong habitual sharers (in yellow-green), side-by-side with the actual data for strong habitual sharers (in gray). (Note: we did not plot bars for weak habitual sharers because there are no weak habitual sharers in the actual data.)

As shown in the figure, for each of the estimates for strong habitual sharers, the model prediction data was between 15 and 24 percentage points different from the actual data.

Although there is nothing wrong, in principle, with only reporting model predictions, we think the way this study reported model predictions led to three major issues. 

First, the study provides statistics about a group of participants that do not exist (“weak habitual sharers”). 

Second, the study does not specify that the reported numbers are model predictions. It only became apparent to us that the reported numbers were model predictions after trying to reproduce all of the numbers in the paper. We think readers would assume that descriptions about how often participants shared news headlines would be numbers calculated directly from the study data. 

Third, using model predictions instead of actual descriptive statistics is a less direct approach to measuring what the study purports to measure. For example, the model predictions about how often participants shared false news were reported instead of how often participants actually shared false news.

In communication with the authors after drafting this report, they noted:

Since we used the prediction model, we can still report 1 SD deviation below the mean. Technically, this is an appropriate way to analyze the data. More importantly, the results are nearly identical if we compare the predicted probabilities at sharing habit 1 and sharing habit 0.87 (-1SD below the mean). An alternative way to analyze the data would be determining Johnson Neyman points. But this approach would not have changed any of our conclusions, as shown by the results below 

We reported predicted probabilities, and these are clearly marked in our graphs. However, I plotted predicted and actual sharing at every habit bin. As you can see, on average, they are aligned, which simply means that our model successfully recovered the data. They are aligned across the different question order conditions and for real and fake headlines. If anything, in the accuracy first condition (cond_r = 1), actual sharing seems slightly ahead of predicted sharing especially for fake headlines at high levels of habits. In general, the area under the curve is pretty similar for predicted and actual values.

The way discernment was calculated may have inflated the difference between strong and weak habitual sharers

The primary outcome of interest in the paper is participants’ sharing discernment. We believe the way it was calculated may have inflated the difference between strong and weak habitual sharers.

Conceptually, in the context of misinformation, sharing discernment can be thought of as the tendency to share news that is true and not share news that is false. Given this, there are at least two reasonable approaches to calculating discernment. 

The first approach is to assess how the veracity of a news headline affects each decision to share or not share the headline. Higher discernment, in this case, would mean that headlines are more likely to be shared when they are true (and less likely to be shared when they are false). This is the general approach used by this paper (and many other misinformation papers). 

The second approach is to simply calculate, for each participant, what proportion of their decisions were the “discerning decision” (i.e., sharing a news headline when it is true or not sharing a news headline when it is false). Scores closer to 1 would signify participants who are more discerning (i.e., made the discerning decision a higher proportion of the time) and scores closer to 0 would signify participants who are less discerning (i.e., made the discerning decision a lower proportion of the time). 

In many cases, it shouldn’t matter which approach is used. Both approaches conceptualize discernment similarly—sharing a true headline and not sharing a false headline are discerning decisions, and sharing a false headline and not sharing a true headline are non-discerning decisions. However, we believe that the first approach presents an issue in this particular study.

In brief, the issue stems from two features: 

1. This study tests whether participants who have different news sharing habits have different levels of discernment 
2. Participants who have higher news sharing habits tend to share a medium amount of news headlines, while participants who have lower news sharing habits tend to share a small amount of news headlines.    

These two features, when combined with the underlying assumptions of the statistical test, can make it appear that the group who shares a small amount of news headlines is more discerning than the group who shares a medium amount of articles—even if both groups make the exact same proportion of discerning decisions. If you want to read about the statistical reasons for this, see the appendix section titled “Additional information about the issue with how discernment was calculated.” 

To illustrate this issue, we simulated data to mirror the structure of the data in Study 2. We simulated 839 participants who decided whether to share or not share 8 true headlines and 8 false headlines (same as the original study). We gave each participant a 50-50 chance of being a strong habitual sharer or a weak habitual sharer.

We then created three different scenarios:

Scenario 1: 

• every strong habitual sharer decided to:
  • share 5 true headlines and not share 3 true headlines 
  • share 4 false headlines and not share 4 false headlines
• every weak habitual sharer decided to:
  • share 2 true headlines and not share 6 true headlines 
  • share 1 false headline and not share 7 false headlines

Scenario 2: 

• every strong habitual sharer decided to:
  • share 7 true headlines and not share 1 true headline
  • share 6 false headlines and not share 2 false headlines
• every weak habitual sharer decided to:
  • share 2 true headlines and not share 6 true headlines 
  • share 1 false headline and not share 7 false headlines

Scenario 3: 

• every strong habitual sharer decided to:
  • share 7 true headlines and not share 1 true headline 
  • share 6 false headlines and not share 2 false headlines
• every weak habitual sharer decided to:
  • share 5 true headlines and not share 3 true headlines
  • share 4 false headlines and not share 4 false headlines

Discerning decisions are denoted by green text and non-discerning decisions are denoted by red text.

Note, that this means that every single participant, across all three scenarios, made the exact same number of discerning decisions (9) and non-discerning decisions (7). Additionally, every participant shared the same net number of true headlines—the number of true headlines shared was one more than the number of false headlines shared. 

The only difference between strong and weak habitual sharers in each scenario was that strong habitual sharers always shared more headlines in total than the weak habitual sharers did.  

We then ran a generalized linear mixed effects model (the same type of statistical model used in the paper) that predicted whether participants shared the headline by the interaction between the veracity of the headline (true or false) and whether participants were in the strong or weak habitual sharer group. This analysis, like the one used in the paper, uses the first approach to calculating discernment described above.

The table below shows the odds ratios and associated p-values for the effect of this interaction in each of the three scenarios.

What these simulations show is that the sheer number of headlines the different groups shared mattered for the model’s evaluation of which group had better discernment (so much so that the results flipped direction between Scenarios #1 and #3). This happened even though every single participant made the same proportion of discerning decisions.  

In Scenario 1, where weak habitual sharers shared very few headlines (3 of 16) and strong habitual sharers shared a medium amount of headlines (9 of 16), the results of the model suggest that weak habitual sharers have better discernment. This scenario is similar to the pattern of sharing observed in the real data.

But in Scenario 3, where weak habitual sharers shared a medium amount of headlines (9 of 16) and strong habitual sharers shared a high amount of headlines (13 of 16), the results flipped, suggesting that weak habitual sharers have worse discernment. 

Whereas in Scenario 2, when weak and strong habitual sharers were on perfectly opposite sides of the sharing distribution (sharing 3 of 16 versus 13 of 16), there was an odds ratio of exactly 1, suggesting no difference between weak and strong habitual sharers. 

So, it’s not the case that comparing high versus low habitual sharers has to lead to weak habitual sharers appearing to have higher discernment—it depends on the amount of total sharing each of these groups does. 

If discernment had instead been calculated as the proportion of times participants made the discerning decision (the second approach discussed early on in this section), there would be exactly zero difference in discernment between the strong habitual sharers and weak habitual sharers in all three of these scenarios. 

Of course, the simulation we ran was different from the actual study in several meaningful ways: 

• The statistical model in Study 2 treated news sharing habits as a continuous variable, not a binary variable 
• The statistical model in Study 2 also included the effect of experimental condition 
• In our simulation, participants labeled as strong habitual sharers always shared more news headlines than participants labeled as weak habitual sharers. In the actual Study 2 data, because the news sharing habit measure was self-report, it was possible for participants to score highly on the measure but share very few headlines in the actual study (and vice versa) 

As such, using the real Study 2 data, we calculated a discernment score for each participant as the proportion of times they made the discerning decision (i.e., the second approach to calculating discernment discussed above). We then ran a linear regression predicting these discernment scores by experimental condition, news sharing habit scores, and the interaction between the two. (We structured this model to be as conceptually close to the original model as possible.) This allowed us to see whether the effect size found using the original analytical approach might be inflated, as our simulations suggest.

We found results that were directionally similar to those of the original model—people in the accuracy condition tended to show greater discernment, people who had higher news sharing habit scores tended to show lower discernment, and no significant interaction between condition and news sharing habit scores on discernment: 

However, the effects observed in this new model were, arguably, much weaker than the effects in the original model. 

It is important to note that it is difficult to precisely compare the size of the relationship between discernment and news sharing habit scores observed in the original model and our model. Primarily, this is because the models are estimating different quantities—log-odds of sharing an article in the original model and proportion of discerning decisions made in the new model. 

However, we can look at a couple different indicators to get a rough sense of how they compare. First, the p-value for this relationship in the original model is <0.001, whereas it is 0.035 in the new model. Second, if we standardize the model coefficient for the relationship between discernment and news sharing habit score (the coefficient that has a value of -0.006 in the model results above), the standardized coefficient has a value of -0.41 in the original model, but -0.07 in the new model.  

Perhaps the best way to get an intuition for the effect size found in the new model is to see the relationship between participants’ news sharing habit scores and their discernment plotted:

According to the results of the new model we ran, the expected difference in discernment score between someone with the lowest news sharing habit score (score of 1) and someone with the highest news sharing habit score (score of 7) is 0.036 (on average, all else equal). In other words, the strongest possible habitual sharers would be expected to make a discerning decision 3.6% less often than the weakest possible habitual sharers. This effect size feels at odds with descriptions in the paper, such as “As predicted, and replicating Study 1, strongly habitual participants continued to share with limited sensitivity to the veracity of headlines.”  

In sum, it appears that the analytical approach used throughout the paper to assess discernment may have caused inflated effect sizes because of the fact that stronger habitual sharers tended to share a medium amount of the headlines while weaker habitual sharers tended to share very few of the headlines. That said, the general finding that higher habitual sharers had worse discernment held even when calculating discernment differently. The effect was just weaker. 

The study contains errors and numbers we cannot reproduce 

Throughout Study 2, there were several instances where the numbers reported in the paper differed from the numbers we reproduced with the authors’ data and code. Sometimes, these deviations were caused by identifiable errors, but other times we could not identify what led to the differences. The deviations were usually small and inconsequential, but the frequency of numbers we could not reproduce was concerning and could suggest that there are other errors we did not detect. This section will not provide an exhaustive list of these issues, but will highlight a handful. 

Figure 3 reports the wrong study’s data and contains errors

Figure 3 in the paper (see figure below) is reported as displaying results from Study 1. However, the publicly available code for Study 2 has a section at the end that creates Figure 3. If you run that code using the data from Study 2, it perfectly recreates this figure. Thus, it appears that Figure 3 displays results from Study 2, despite the paper suggesting that these results are from Study 1. 

In addition to using data from the wrong study, there are a few errors in Figure 3. 

First, the magnitude of the effect size value for “Critical Reflection (Need for Cognition)” is miscalculated (as described in detail later in this section). 

Second, the effect size for “Critical Reflection (Need for Cognition)” should be in the opposite direction from the other other measures (there is a negative relationship between sharing false news and Critical Reflection, whereas the other variables have a positive relationship with sharing false news). It would be reasonable to present these effect sizes as absolute values, but it was not specified in the paper that these were absolute values. 

Third, the 95% confidence interval for the effect of ‘Critical Reflection (Need for Cognition)’ appears to be extremely narrow. The generalized linear mixed effects model used to estimate this effect did not converge, which likely caused the implausibly narrow confidence interval. The model output shows a confidence interval width of zero, and the z-value for Critical Reflection is -834.71, suggesting that the effect estimate is 834 standard deviations away from zero (see the first screenshot below).

Such an extreme z-value is implausible and suggests that the model’s fit is problematic. This is the type of issue that can arise with mixed effects models that don’t converge, which is concerning since most of the mixed effects models run in Study 2 (including those reported in the supplementary information) do not converge. The 95% confidence interval for the effect of ‘Critical Reflection (Need for Cognition)’ should almost certainly have a width greater than 0. 

There were many numbers we could not reproduce 

There are many numbers reported in Study 2 that we cannot reproduce. This report does not document every such instance both for the sake of time and because, even if the numbers were reproducible, they would still suffer from all of the issues noted above.

But, as an example of numbers we could not reproduce, take the numbers reported in the main figure for Study 2:

These numbers appear to be estimated marginal means that were calculated to compare the average probability of sharing in each experimental condition, broken down by the four possible combinations of weak habitual sharers versus strong habitual sharers and true headlines versus false headlines. Here is a  screenshot of the results from the authors’ code that we believe was used to calculate these values:

Here’s how to read these results: 

• “Sharehabit = 0.9” refers to weak habitual sharers 
• “Sharehabit = 3.7” refers to strong habitual sharers 
• “iv_credf = real” refers to true headlines 
• “iv_credf = fake” refers to false headlines 
• “cond_r = 1” refers to the judge accuracy first condition (treatment condition) 
• “cond_r = -1” refers to the share first condition (control condition) 

So, for example, the bottom value in the screenshot of 0.1931 should be the value shown in the Figure for strong habitual sharers when evaluating false headlines in the judge accuracy first condition. The value in the figure is 22%, but these results suggest it should be 19%. There are a couple of other values in the figure that are off by a few percentage points, but several of the other values in the figure align with the results in the screenshot. 

The numbers used to create the Figure 4 plot were hard-coded in the authors’ code, so we cannot be certain where they came from. Our best guess is that these discrepancies were either a transcription error or that a slightly different version of the model was run and used to calculate the numbers for the plot, and then the model was updated at a later point but these numbers were not. 

It is also worth discussing the numbers mentioned in the Figure 4 caption. The caption states, “In the sharing first condition (4A), weak habit participants were 2.2 times more discerning than strong habit ones. In the judge accuracy first condition (4B), this difference reduced slightly to 1.7 times.” We did not find these numbers calculated in the analysis code, and at first we were not able to reproduce them because we assumed they were based on the numbers in the figure. Eventually, however, we were able to arrive at the stated numbers if we did the following: First, calculate the marginal effect of headline veracity on the likelihood of sharing for both weak and strong habitual sharers in both the share first condition and the judge accuracy first condition. Second, convert the odds ratios from this analysis into d values. Third, compare the d values for weak and strong habitual sharers in the share first condition to arrive at the number 2.2, and compare the d values for weak and strong habitual sharers in the judge accuracy first condition to arrive at the number 1.7. If that is how these numbers were calculated, it seems unclear to describe them as how many “times more discerning” some participants are than others. This is also an example of why it can be difficult to reproduce specific numbers if they are not explicitly calculated in the analysis code.    

It is important to note that failure to reproduce specific numbers is not necessarily due to errors in the paper. It is possible that we made mistakes when attempting to reproduce the numbers or that numbers were calculated through a complex procedure that was not documented, as illustrated by the example above. There are also more benign reasons for irreproducible numbers, such as packages used in the analysis code that changed between the time the analyses were originally run and the time we ran them. 

There were many issues in the analysis code 

Even though our inability to reproduce some numbers does not necessarily reflect issues with the paper, the analysis code had several issues that suggest that some of the difficulty reproducing numbers could be because of the code. 

First, the code did not allow for seamless reproduction of the analyses. It seemed to have been written to run on a different version of the data because it called column numbers that do not exist in the publicly available version of the data. (However, the authors very promptly shared the raw data file with us when we asked, which allowed us to identify the columns in question.) There were also typos in model names and variable names, and variables called that weren’t created until later in the script. This suggests that the script was never run all the way through, which can be a cause of irreproducible numbers. 

Second, there were calculation errors in the code that did or could have led to reporting incorrect numbers. For example, the analysis for the Critical Reflection effect in Figure 3 discussed above yields an odds ratio of 0.80 (suggesting a negative effect). The analysis code specifies that this was converted to an odds ratio for a positive effect of the same magnitude by using the formula “(1-0.80+1 = 1.20)” (see screenshot below). This is not the correct way to convert odds ratios. The correct conversion would be 1 / .80 = 1.25. This incorrect odds ratio was then converted to a d effect size and reported in Figure 3.    

Third, there were some results reported for Study 2 in the supplement for which there was no code in the analysis script (specifically, the section titled “Supporting Text S2: Study 2 Predicting Truth Judgment from Three-Way Interaction Models”).  

Collectively, these issues suggest that some of the irreproducible numbers reported in the paper could have come from errors in the code.   

Summary of issues discovered in Study 2 

In sum, through the process of trying to reproduce the results from Study 2, we encountered many issues with the implementation, analysis, reporting, and interpretation of the study, including:

• The primary claims don’t match the provided evidence
• The primary claims are based on statistical results that could be unreliable
  • There was an error in one of the key measures
  • Participants were not evaluated with quality checks
  • The statistical model failed to converge
  • Central claims rely on null results, but the study is likely underpowered
  • The preregistration was not followed, but was claimed to have been followed
• Most numbers supplied in the paper are model predictions, not direct descriptions of the data, which causes comparisons with an “impossible” group of participants
• The way discernment was calculated may have inflated the difference between strong and weak habitual sharers
• The study contains errors and numbers we cannot reproduce

(Later in the report, we detail some similar issues that we encountered in the other studies in the paper.)

We consider the most significant individual issue in Study 2 to be that the primary claims don’t match the provided evidence. Even if there were no other issues in the paper, incorrect claims cause the paper (and thus readers) to draw incorrect conclusions from the provided evidence. 

We think the following three issues are the next most significant. 

(a) The fact that Amazon Mechanical Turk participants were not evaluated with quality checks seems particularly problematic for this study because, as explained in the section “Participants were not evaluated with quality checks,” inattentive/spam participants could make it more likely to find a null result for the three-way interaction, while also making it more likely to find a relationship between participants’ News Sharing Habits and their discernment.  

(b) The study regularly discusses “low habitual sharers,” who are a group that cannot exist because they are defined as having a News Sharing Habit score that is below the lowest possible score (as explained in the section titled “Most numbers supplied in the paper are model predictions, not direct descriptions of the data”). This causes the results presented in the study to be, in our view, misleading because high habitual sharers (a group that exists in the study) are regularly compared against low habitual sharers (a group that does not exist in the study)—for example, Study 2 states, “weak habit participants were 1.9 times more discerning than strong habit ones.”  

(c) As argued in the section “The way discernment was calculated may have inflated the difference between strong and weak habitual sharers,” the observed relationship between News Sharing Habit and discernment is much smaller when calculated using an approach that doesn’t suffer from the statistical artifact we identified. Although the general relationship described in the paper is still present when correcting for this, we think the smaller effect size makes it unclear if the effect is practically meaningful. We say “unclear” because we acknowledge that assessing whether an effect size is “meaningful” is challenging and, ultimately, a subjective judgment. 

While the remaining issues are more minor, and may not individually affect the general pattern of results, we think that the overall quantity and diversity of issues is perhaps the biggest concern for this study because it is difficult to tell how they, collectively, impacted the study results.

Although, in theory, a replication could address some of these issues with Study 2, we have decided not to replicate the study. This decision is explained in the next subsection. 

Why we are not replicating Study 2

The primary reason we are choosing not to replicate Study 2 is that it is unclear precisely what Study 2 was trying to test.  

Broadly, Study 2 examines the relationships between whether people share a news headline, the veracity of that headline, whether people are prompted to judge the accuracy of that headline before deciding whether to share it, and how people score on the news sharing habit measure. 

The paper motivates Study 2 by stating:

“One potential explanation for habitual sharing is that people share indiscriminately when they are not able or motivated to assess the accuracy of information. In this account, habitual sharers spread misinformation just because strong habits limit attention to accuracy. To test this possibility, we examined whether highlighting accuracy prior to sharing would reduce the habitual spread of misinformation and increase sharing discernment.” 

This suggests that the primary tests of interest should be whether the experimental manipulation—prompting participants to judge accuracy first or decide what to share first—causes habitual sharers to share less misinformation and improve their sharing discernment. 

Yet, the three-way interaction tested in the primary statistical model assesses whether the experimental condition influenced sharing discernment differently for people depending on how habitual of a sharer they were. Based on the results, the paper states, “In general, rating accuracy first did not increase the discernment of strongly habitual users any more than less habitual ones.” Contrary to the stated motivation for Study 2, this analysis suggests that Study 2 aims to test whether rating accuracy first is more effective for habitual sharers than for non-habitual sharers. 

The paper notes that the two-way interaction between headline veracity and the experimental manipulation is significant such that the accuracy intervention appeared to improve sharing discernment among the whole participant sample, on average. But it then concludes by stating, “Thus, highlighting accuracy proved useful in reducing the spread of misinformation but not among the most habitual users.” One way to interpret this statement is that the study is primarily focused on how the intervention affects the most habitual users, consistent with the stated motivation for the study. However, the paper never reports a statistical test that directly assesses whether the experimental manipulation improved the spread of misinformation among habitual users. So, our best guess is that this conclusion was an (incorrect) callback to the results of the three-way interaction. 

The closest the study comes to assessing whether the experimental manipulation improved the spread of misinformation among habitual users is a plot (Figure 4) that purportedly shows the probability of participants sharing a headline, broken down by the headline’s veracity, the experimental condition, and whether the participant is a strong or weak habitual sharer:  

On its face, this plot seems to show that the experimental manipulation led the strong habitual sharers to improve their discernment and share less false news—the probability of sharing was 42% for true news and 30% for false news in the Share First condition versus 42% for true news and 22% for false news in the Judge Accuracy First condition. (We do not recommend taking this plot at face value as described earlier in the report, but we highlight it because it was used as evidence in the paper for the study’s claims.) So, this figure seems to go against the conclusion from Study 2, “Thus, highlighting accuracy proved useful in reducing the spread of misinformation but not among the most habitual users.” 

Between the stated goals of the study, the analyses conducted, the results presented, and the interpretations of the results, it is unclear to us what the study is trying to test. This is further complicated by the fact that the preregistration does not mention the experimental condition in its hypotheses. The hypotheses section centers on testing for differences in sharing behavior between habitual sharers and non-habitual sharers:

The preregistration suggests that there was no a priori motivation for including the experimental manipulation (although it could have been mistakenly left out of the preregistration). 

Two smaller, but still important, challenges to the conceptual clarity of this study are caused by the variables used in the analyses. 

First, throughout the paper, participants are discussed as being strong habitual sharers or weak habitual sharers. The framing around strong habitual sharers suggests that strong habitual users are a distinct type of person, whose sharing discernment needs to be improved (e.g., “habitual users were responsible for sharing a disproportionate amount of false information”). Study 2 seems particularly concerned with improving outcomes for strong habitual sharers. 

Yet, judging from the distribution of scores on the news sharing habit measure (see figure below) as well as the relationship between the proportion of discerning decisions made and news sharing habit scores (see figure below), the data does not seem to motivate a special focus on those with scores +1 SD above the mean on the news sharing habit measure. Perhaps if there was a bimodal distribution of news sharing habit scores or if those with medium-to-high news sharing habit scores were far less discerning, then it would be straightforward to dichotomize participants into “strong” and “weak” sharers.

Although, the study splits participants into “strong” and “weak” habitual sharers and compares model predictions about headline sharing rates among these groups, the primary analysis in Study 2 treats habitual sharing as a continuous measure, and the study does not report any direct tests of the effect of the experimental manipulation on strong habitual sharers (as mentioned earlier). This makes it challenging to know whether the goal of this study is to assess the effects on strong habitual sharers, or whether the study is interested, more generally, in the relationship between sharing habits and sharing outcomes. 

A second challenge to the conceptual clarity of the study is that sometimes the focus of a particular statistical test is sharing discernment, while other times it is the amount of false news shared. (While these are related concepts, they are not the same: one could reduce the amount of false news they share, but if they also proportionally reduce the amount of true news they share, they would have the same sharing discernment.) Because results for both are not always tested and/or reported, it is unclear whether Study 2 is always interested in outcomes for both or whether it depends on the test in question.

In sum, it is difficult to pin down what analysis one should run if they wanted to test the central hypotheses and verify the claims made in this study. Moreover, it is unclear what it would mean to “replicate” the results from the original study given that many of the reported results had errors or other issues. As such, we have decided not to replicate the study. 

There are similar issues in the other studies in the paper

This paper has 4 studies in total. After encountering the issues described above in Study 2, we did a quick review of the other studies to see if some of the most significant issues we detected in Study 2 were also present in the other studies. 

The table below summarizes what we found. The first column lists each issue we assessed. The other columns indicate whether that issue was present in Studies 1, 3, and 4.

In the subsections below, we provide details for each of the issues indicated by this table. To fully understand how these issues may have impacted the results in Studies 1, 3, and 4, we recommend reading through these studies in the original paper to understand the study designs and reported results. 

Studies 1 & 3

Studies 1 & 3 were quite similar to Study 2 and had many of the same issues as those identified in Study 2. The table below explains the issues in detail. 

Study 4

Study 4 used a substantively different study design from Studies 1-3 and did not seem to have the same quantity or severity of issues as those identified in Study 2. The table below explains the issues in detail. 

We did not try to fully reproduce all of the results for Studies 1, 3, & 4 as we did for Study 2. However, in the course of checking whether Study 4 contained any of the same major issues as Study 2, we came across a number of minor errors, as well. Although we think these errors do not affect the high-level pattern of results, they do add to our general concern about the reliability of the findings reported in the paper (to see examples of these errors, see the Appendix section titled “Additional information about errors detected in Study 4”).

Summary of issues in other studies

In sum, from a quick review of Studies 1, 3, and 4, we detected many of the same Study 2 issues in Studies 1 & 3, but fewer issues in Study 4. Nevertheless, Study 4 still contained several minor instances of some of the Study 2 issues and also had a handful of numerical errors. 

Conclusion

Because of the issues we encountered when trying to reproduce the results reported in Study 2 (described above), we chose not to replicate the study. Based on a quick review, the other studies in the paper have some, but not all, of the same issues as Study 2. Study 2 did not receive a replicability rating since we did not attempt to replicate it. 

The materials, pre-processed data, and analysis code were publicly available for Study 2. The analysis code did not successfully run on the provided pre-processed data, but the authors readily provided the raw data upon request. The study was preregistered, but the preregistered analyses were not followed and these deviations were not mentioned in the paper. The study received 3.75 stars for transparency. 

The issues with Study 2 documented above led to unreliable results and/or incorrect claims. We think these issues will cause readers to come away with an inaccurate impression of what the study shows. The study received 0 stars for clarity. 

Purpose of Transparent Replications by Clearer Thinking

Transparent Replications conducts replications and evaluates the transparency of randomly-selected, recently-published psychology papers in prestigious journals, with the overall aim of rewarding best practices and shifting incentives in social science toward more replicable research.

We welcome reader feedback on this report, and input on this project overall.

Author Response

Our feedback process with the original authors proceeded in a few rounds of communication. They are presented in order below.

First Round

The lead author submitted the following response on behalf of the authorship team. The numbered points correspond to the section in the report being addressed. 

We appreciate you taking the time to analyze our paper.

However, we respectfully but firmly disagree with the majority of your critiques.

As detailed in our attached response, we have extensively replicated these findings across multiple studies using various measures and methods. While you correctly identified a typo in one habits measure, this does not impact our results’ robustness, as demonstrated through multiple replications and alternative measures.

Your analysis examines a single study in isolation, overlooking the paper’s comprehensive evidence across multiple studies that collectively support our conclusions. This approach does not align with standard scientific practice of evaluating evidence holistically.

We have provided detailed responses to each point raised and consider this matter closed. We respect your right to publish your analysis, though we believe doing so would not serve the scientific discourse productively.

Best,

Gizem (on behalf of the authors)

1. The study used a measure of news sharing habit that had an error in the question wording

As you pointed out, there is a typo in one of the measures. However, all 4 items in the scale are highly correlated. If you drop this item, you will see that the results still hold. We also used other habits measures (reading habits and frequency of sharing). While reading habits are a weaker predictor compared to sharing habits but the results hold using any of these scales.

2. The study data were collected on Amazon Mechanical Turk with no quality checks

We have replicated these results many, many times, and it is implausible that the result is due to noise instead of habit strength.  We even built habits in Study 4 to demonstrate causality of the effect. We have included quality checks such as attention checks and elimination of duplicate ip addresses in subsequent research, and we have obtained comparable results to those in the set of studies published in PNAS. 

3. The model did not converge 

We computed many models including participants and headline fixed effects and decided to report the most comprehensive and conservative model. We also computed a model without random effects, and with an optimizer (control = glmerControl(optimizer = “bobyqa”). In all these cases, models converged and results remained virtually identical. The consistent results despite different models attest to the robustness of the effect. We did not include these  in the web appendix because our focus was on reporting the other models including the various covariates requested by reviewers.

4. The study was underpowered 

The power analysis we reported is for the focal effect, which is the interaction between habits scale and headline veracity. We are able to detect this effect even with 200 participants. Since we added a between-subjects variable (question order), we increased the sample size 4 times, which is in line with standard practices in the field. New approaches to power analysis with mixed effects offer various recommendations on how to calculate power. Even a recent paper suggests that power analysis does not lead to reliable results especially for mixed effect models (Pek, Pitt, and Wegener 2024).

Pek, J., Pitt, M. A., & Wegener, D. T. (2024). Uncertainty limits the use of power analysis. Journal of Experimental Psychology: General, 153(4), 1139.

5. Our central analysis was not included in the preregistration (but was claimed to be)

This is an interesting claim. Our central prediction was for a two-way interaction. We did not expect that this effect would be modified by question order, and thus we did not specify the three-way interaction in the preregistration. Instead, we outlined the core, central effect we expected to be significant. We are unaware of any guidelines specifying that nonsignificant effects need to be preregistered. 

6. Predicted probabilities 1SD below the mean

Since we used the prediction model, we can still report 1 SD deviation below the mean. Technically, this is an appropriate way to analyze the data. More importantly, the results are nearly identical if we compare the predicted probabilities at sharing habit 1 and sharing habit 0.87 (-1SD below the mean). An alternative way to analyze the data would be determining Johnson Neyman points. But this approach would not have changed any of our conclusions, as shown by the results below 

7. Predicted probabilities vs. actual sharing

We reported predicted probabilities, and these are clearly marked in our graphs. However, I plotted predicted and actual sharing at every habit bin. As you can see, on average, they are aligned, which simply means that our model successfully recovered the data. They are aligned across the different question order conditions and for real and fake headlines. If anything, in the accuracy first condition (cond_r = 1), actual sharing seems slightly ahead of predicted sharing especially for fake headlines at high levels of habits. In general, the area under the curve is pretty similar for predicted and actual values. 

(Ceylan, email published with permission, 1/26/2025)

In addition to this response, the authors also provided a follow up email response to our section in the report titled “the primary claims don’t match the provided evidence.” That response is below:

We interpreted the lack of three-way interaction based on the data pattern I shared with you in the reactions document.

You are making a thought experiment but frankly, you can just examine the pattern of the data.

The data is showing us that everybody (both high and low habitual users) their sharing slightly, supporting the lack of three-way interaction.

(Ceylan, email published with permission, 1/30/2025)

Transparent Replications Team Response to First Round

After receiving this response the Transparent Replications team included those points in the relevant sections of the report as well as at the end, and made changes to the executive summary clarifying the scope of our claims to make more clear that the Clarity rating is our attempt to evaluate the presentation of the evidence in the paper itself, and how likely a reader may be to misinterpret what is presented in the paper.

We communicated to the authorship team that we included their responses in the relevant sections of the report in order to allow their critique to speak for itself, without a reader having to get to the end of the report to see it.

Second Round

The original authorship team was not satisfied with the The Transparent Replications team’s level of engagement with their critique, and asked that we include the following statement:

“As the original authors, we are deeply disappointed by the level of engagement Clearer Thinking has demonstrated in this critique. Despite providing detailed clarifications and transparent analyses, the report persists in promoting conclusions that appear driven more by prior beliefs than a fair reading of the evidence. We view this as a significant misrepresentation of our work and not in line with the standards of rigorous scientific evaluation. We urge greater care and intellectual honesty in future engagements with scholarly work.”

(Ceylan, email published with permission, 4/1/2025)

Transparent Replications Team Response to Second Round

In response to this concern, our team drafted brief responses to each of the points raised by the original authorship team in the first round of feedback. Our response is below:

We appreciate you granting permission to share your responses, and will add the statement you request to the report. I wanted to address your concern about us not engaging enough with your comments and feedback.

We initially opted to simply share your comments as you wrote them in the report, rather than providing a response to them because we felt like it was most fair to offer readers your perspective alongside our perspective. After reading your message, I can see how that came across as us not engaging with your feedback, and I apologize for that. I wanted to offer a brief response to your feedback to give you more of a sense of why it didn’t result in more of a change to our report.

I think there may be a bit of a misunderstanding about our process and how we evaluate papers. Our Clarity evaluation of the study is focused fundamentally on how likely we believe readers are to come away with accurate impressions of the evidence presented in the paper from reading the paper itself. Several of your responses point to additional tests run by the authorship team that were not included in the paper, and other studies the team conducted on this topic. That is certainly valuable information for someone evaluating the claims about habits and misinformation made in the paper, and we’re happy to be able to include that information in our report. At the same time, that information doesn’t change our clarity rating which is about evaluating the evidence as it was presented in the paper itself, rather than how likely it is that the underlying hypotheses are true. Our goal in the revisions we made to the Executive Summary after your initial feedback was to make sure the scope of our claims is clearly defined.

With that said, the attached document contains our brief responses to your specific points. Again, apologies for not offering these before. Our intention is to include these responses following your responses at the very end of our report.

(Email from Metskas on behalf of TR team, 8/11/2025)

Appendices

Additional analyses we preregistered and ran on the original data

After we had first attempted to reproduce the analyses reported in Study 2, we thought that the primary analysis conducted in the paper—a generalized linear mixed effects model predicting headline sharing by the three-way interaction between veracity, condition, and news sharing habit—was an overly complex analysis given the stated goals of Study 2. So we decided to run what we considered to be the simplest valid analyses on the data.

After running these analyses, we uncovered other issues with the study (e.g., an error in the news sharing habit measure, no participant quality checks, low statistical power, mixed messages about what exactly the study is supposed to test) that led us to believe that the simplest valid analyses we ran were also probably unreliable and/or uninformative. 

As such, we think these analyses should be taken with a grain of salt, but we include them here for transparency. 

Description of the simplest valid analyses

The original paper stated the goal of Study 2 as follows:

“One potential explanation for habitual sharing is that people share indiscriminately when they are not able or motivated to assess the accuracy of information. In this account, habitual sharers spread misinformation just because strong habits limit attention to accuracy. To test this possibility, we examined whether highlighting accuracy prior to sharing would reduce the habitual spread of misinformation and increase sharing discernment.”

We thought that the most direct way to test the questions of interest in this study were to assess whether highlighting accuracy prior to sharing would:

1. reduce the spread of misinformation among habitual sharers
2. increase sharing discernment of habitual sharers

Because we interpreted this study as being primarily concerned with the effect of the accuracy intervention on the behavior of habitual sharers, we thought the simplest valid analyses should test only habitual sharers (those who had a news sharing habit score of at least one standard deviation above the mean). Since we were focusing only on habitual sharers, we did not use news sharing habit as an independent variable, which simplified the analyses.  

Another way to simplify the analyses used in the paper was to create two scores for each participant: 

1. Spread of misinformation score: how much misinformation each participant shared, which we calculated as the proportion of the false headlines they shared
2. Sharing discernment score: how discerning each participant was, which we calculated as the proportion of discerning sharing decisions they made—in other words, the proportion of true headlines a participant shared and the proportion of false headlines a participant did not share

Creating a single score for each participant meant that, instead of using generalized linear mixed effects models, we could run independent samples t-tests—a simpler statistical model. (The original paper’s approach treated each decision to share or not share as a single data point. This meant that each participant had multiple data points, which required a model that accounted for the clustering of observations within participants; e.g., a mixed effects model.)

Additionally, as discussed in the section titled “The study contains errors and numbers we cannot reproduce,” creating a single discernment score for each participant meant that we no longer needed to include veracity as an independent variable in the model. 

In sum, this approach allowed us to run two simple analyses addressing what we interpreted as the questions of interest:

1. Did highlighting accuracy prior to sharing reduce the spread of misinformation among habitual sharers? – tested via an independent samples t-test assessing whether habitual sharers’ spread of misinformation scores differed between experimental conditions
2. Did highlighting accuracy prior to sharing increase sharing discernment of habitual sharers? – tested via an independent samples t-test assessing whether habitual sharers’ sharing discernment scores differ between experimental conditions

For comparison, we also ran versions of the original generalized linear mixed effects model used in the paper to test the two questions of interest. Here’s a description of these models:

1. Did highlighting accuracy prior to sharing reduce the spread of misinformation among habitual sharers? – a generalized linear mixed effects model, run on only habitual sharers, predicting sharing as a function of condition, among only the 8 trials in which participants evaluated false headlines
2. Did highlighting accuracy prior to sharing increase sharing discernment of habitual sharers? – a generalized linear mixed effects model, run on only habitual sharers, predicting sharing as a function of the interaction between headline veracity and condition 

We preregistered these four analyses here (see preregistration for greater detail about the analyses, including random effects structure, variable coding, etc).

In brief, all four analyses found null results. Below are more detailed results. 

Did highlighting accuracy prior to sharing reduce the spread of misinformation among habitual sharers?

Simplest valid analysis 

The results of the independent samples t-test assessing whether habitual sharers’ spread of misinformation scores differed between experimental conditions indicated that there was no significant difference between the two groups, t(131.99) = 0.84, p = .400. 

Generalized linear mixed effects model for comparison

The results of the generalized linear mixed-effects model (run on only habitual sharers) predicting sharing as a function of condition among only the 8 trials in which participants evaluated false headlines indicated that condition was not a significant predictor of sharing, b = -0.12, SE = 0.14, z = -0.82, p = .414. 

Did highlighting accuracy prior to sharing increase sharing discernment of habitual sharers?

Simplest valid analysis 

The results of the independent samples t-test assessing whether habitual sharers’ sharing discernment scores differ between experimental conditions indicated that there was no significant difference between the two groups, t(125.11) = -1.48, p = .141.

Generalized linear mixed effects model for comparison

The results of the generalized linear mixed-effects model (run on only habitual sharers) predicting sharing as a function of the interaction between headline veracity and condition indicated that the interaction between headline veracity and condition was not significant, b = 0.16, SE = 0.12, z = 1.34, p = .180.

Low statistical power

It is difficult to interpret the null results for each of these analyses because the analyses were likely underpowered. 

Because only 134 of the 839 participants met the criteria for being a habitual sharer, according to sensitivity power analyses conducted with G*Power 3.1 (Faul et al., 2009), these t-tests only had 95% power to detect an effect size of at least d = .63 and 80% power to detect an effect size of at least d = .49. As discussed in the section titled “Central claims rely on null results, but the study is likely underpowered,” these effect sizes are quite large given other documented effects in the study.    

Moreover, as discussed in the section titled “Central claims rely on null results, but the study is likely underpowered,” the primary model in the original study was probably underpowered, so the two generalized linear mixed effects models we ran that resembled the primary analyses, but only tested habitual sharers, were probably at least as underpowered.   

Summary

We ran two preregistered analyses that met our criteria for simplest valid analyses and two preregistered comparison analyses designed to mirror the original analytical approach. These analyses found null effects. However, we do not think these results should be interpreted strongly given that they were underpowered and given the other issues we identified with Study 2 after completing these analyses (e.g., an error in the news sharing habit measure, no participant quality checks, mixed messages about what exactly the study is supposed to test). Nevertheless, we have reported the analyses here for full transparency. 

Additional information about the issue with how discernment was calculated 

Throughout the paper, discernment was calculated as the relationship between the decision to share a headline and the headline’s veracity. A stronger relationship between a headline being true and it being shared means greater discernment.  

In order to assess the relationship between participants’ news sharing habits and discernment, the paper ran statistical models with the decision to share a headline as the dependent variable and the veracity of the headline, participants’ news sharing habit scores, and the interaction between the two as the independent variables. 

In this model, a significant  interaction between veracity and news sharing habit scores would suggest that people with different news sharing habit scores have different discernment levels. 

Because the dependent variable in this statistical model is binary and because there are multiple trials per participant, the study uses a generalized linear mixed effects model. This class of model—generalized linear models—fit S-shaped curves (a sigmoid function) to the data when there are binary outcomes. These curves represent the predicted probabilities of a binary outcome.

In principle, this statistical approach makes sense for this data. The problem, however, is that participants who score low on the news sharing habit measure tend to share very few of the 16 headlines they see in the study, while participants who score high on the news sharing habit measure tend to share closer to half of the 16 headlines. 

Under the hood, these types of statistical models do not estimate effects on the probability scale directly. Instead, they use a linear predictor (a weighted sum of predictors) that is then mapped onto probabilities using a sigmoid (S-shaped) function. This mapping is nonlinear: equal shifts in the linear predictor translate into different changes in probability, depending on whether the baseline probability is near 0%, near 50%, or near 100% (notice how the sigmoid curve is steeper in the middle and flatter at either end).

For participants who share very few articles, their baseline probability of sharing (regardless of veracity) may be close to the lower end of the S-curve. In this region, even a small linear increase in the underlying predictor—reflecting differences between true and false headlines—can translate into a relatively large proportional change in the odds of sharing because going from a very low probability to a slightly higher one represents a big relative jump. In contrast, participants who share about half of the articles start near the midpoint of the S-curve. Here, the same increase in the probability of sharing true headlines vs. false headlines represents a smaller proportional change in the odds of sharing.

To make this concrete, here’s an example. Suppose we have two groups, both making the same absolute difference in “discerning” behavior. 

For simplicity, we’ll use one of the examples from the simulations we ran (discussed in the section titled “The way discernment was calculated may have inflated the difference between strong and weak habitual sharers”):

High Sharer Group:

• True headlines shared: 5 out of 8 = 0.625 probability

• False headlines shared: 4 out of 8 = 0.5 probability

• Absolute difference: 0.625 – 0.5 = 0.125

Low Sharer Group:

• True headlines shared: 2 out of 8 = 0.25 probability

• False headlines shared: 1 out of 8 = 0.125 probability

• Absolute difference: 0.25 – 0.125 = 0.125

On the raw probability scale, both have the same 0.125 difference in discerning behavior. Yet the model concludes that the low sharers are more “discerning.” Why? Because logistic regression doesn’t directly model probability differences. It models differences in log-odds:

logit(p) = log(p/(1-p))

If we compute the log-odds for these probabilities, we get:

High Sharer Group:

• For True: logit(0.625) = log(0.625/0.375) = 0.51

• For False: logit(0.5) = log(0.5/0.5) = log(1) = 0

• Difference in log-odds: 0.51 – 0 = 0.51

Low Sharer Group:

• For True: logit(0.25) = log(0.25/0.75) = -1.10

• For False: logit(0.125) = log(0.125/0.875) = -1.95

• Difference in log-odds: (-1.10) – (-1.95) = 0.85

So, even though both groups have the same 0.125 absolute difference in probability, the difference in log-odds is much larger for the low sharers (0.85) than for the high sharers (0.51).

This happens because, at extreme ends of the sigmoid function, a small absolute increase represents a much bigger relative change in odds. 

In other words, the statistical model “sees” that for the low sharers, the difference between how they respond to true vs. false headlines is, in terms of log-odds, more pronounced. Even though the raw probability difference is the same, the low sharers look like they’re making a bigger relative change in their sharing behavior. This gets captured as a stronger effect of discernment in the model output.

So, in the example we provided, it’s not that one group truly has better or worse discernment. Rather, the nonlinear transformation imposed by the logistic model combined with different baseline sharing tendencies can cause identical levels of discernment to appear different once expressed as odds ratios. This phenomenon is a statistical artifact reflecting how the logistic curve and the odds ratio metric interact with varying baseline probabilities.

Additional information about errors detected in Study 4

As mentioned earlier in the report, we did not try to fully reproduce all of the results for Studies 1, 3, & 4 as we did for Study 2. However, in the course of checking whether Study 4 contained any of the same major issues as Study 2, we came across a number of minor errors, as well. Although we think these errors do not affect the high-level pattern of results, they do add to our general concern about the reliability of the findings reported in the paper. Below are three examples. 

Minor error – example 1

The primary discernment effect sizes that are reported in Study 4 appear to be incorrect based on the results in the analysis script. Here is Figure 7 from Study 4, which reports the effect sizes of the probability of sharing false versus true headlines in each of the conditions:

These d effect sizes are calculated in the code by converting odds ratios to d values. These specific d values are arrived at by converting odds ratio values that are hard-coded in the analysis script: 

However, there is no code in the analysis script that calculates those specific values of 12.64, 4.00, and 3.48. Moreover, the code immediately preceding this effect size conversion is code that appears to calculate the odds ratios for the comparisons of interest:

So, as far as we can tell, the odds ratios calculated in the first part of this output (3.54; 9.74; 3.15) should be the odds ratio values to convert, instead of the hard-coded values (12.64; 4.00; 3.48).  

Minor error – example 2

The results for Study 4 state: “Also as expected, discernment was lower in the control and misinformation training conditions, although participants still shared more true headlines (control: 40%; misinformation: 54%)…” 

From re-running the analysis code, it appears that the correct number for the control condition is 43%, not 40%. This is also apparent from examining the gray bar for the control condition in Figure 7: it is clearly above 40% (.40). 

Minor error – example 3

The error bars in Figure 6 are stated to be 95% confidence intervals (see figure caption): 

However, it is clear from the analysis script that the error bars represent standard errors instead (the highlighted line shows the error bars being calculated by adding and subtracting the standard error):

Minor errors – conclusion

Again, it is important to emphasize that these errors do not meaningfully change the interpretation of the results. But, they do represent reproducibility issues and some of these errors could, in theory, affect future meta-analyses that use the results reported in the paper. 

References

Aczel, B., Palfi, B., Szollosi, A., Kovacs, M., Szaszi, B., Szecsi, P., … & Wagenmakers, E. J. (2018). Quantifying support for the null hypothesis in psychology: An empirical investigation. Advances in Methods and Practices in Psychological Science, 1(3), 357-366. https://doi.org/10.1177/2515245918773742

Chmielewski, M., & Kucker, S. C. (2020). An MTurk crisis? Shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11(4), 464-473. https://doi.org/10.1177/1948550619875149

Cuskley, C., & Sulik, J. (2022). The burden for high-quality online data collection lies with researchers, not recruitment platforms. Perspectives on Psychological Science, 17456916241242734. https://doi.org/10.1177/17456916241242734

Faul, F., Erdfelder, E., Buchner, A., & Lang, A.G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149-1160. https://doi.org/10.3758/BRM.41.4.1149

Seedorff, M., Oleson, J., & McMurray, B. (2019). Maybe maximal: Good enough mixed models optimize power while controlling Type I error. PsyArXiv. https://doi.org/10.31234/osf.io/xmhfr

Stagnaro, M. N., Druckman, J., Berinsky, A. J., Arechar, A. A., Willer, R., & Rand, D. (2024). Representativeness versus attentiveness: A comparison across nine online survey samples. PsyArXiv. https://doi.org/10.31234/osf.io/h9j2d

Webb, M. A., & Tangney, J. P. (2022). Too good to be true: Bots and bad data from Mechanical Turk. Perspectives on Psychological Science, 17456916221120027. https://doi.org/10.1177/17456916221120027

Westfall, J., Kenny, D. A., & Judd, C. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143(5), 2020-2045. https://doi.org/10.1037/xge0000014

Why create a Study Diagram?

We create a Study Diagram for each of our reports because we believe that readers should be presented with the key points of the hypotheses, methods, and results in a consistent format that can be understood at a glance. We do this because clear communication is essential to the scientific process functioning well.

Too often in the research literature key pieces of information are spread throughout the text of a paper, making it more time-consuming and difficult to get a clear overall picture of a study. Sometimes the paper itself doesn’t include all of the necessary information, and readers have to refer to supplemental materials to understand what was actually done. This makes it harder for people to find relevant studies, evaluate their claims, and put the information in them to use.

In contrast, imagine a world in which all published empirical research had a Study Diagram. Understanding the gist of a paper would be faster because the Study Diagram takes much less time to review than the whole paper, while also being more standardized and informative than a typical abstract. The Study Diagram would improve the clarity of published research, making it easier to evaluate how well the claims made in the paper correspond to what is being done in the study itself. This would make it easier to identify possible overclaiming or validity issues that can be signs of Importance Hacking. Finally, it would become much easier to sort through literature to find studies that are relevant to your question. At a glance you would be able to compare key features of studies, like their sample size, exclusion criteria, and whether participants were randomly assigned to conditions.

Our goal at Transparent Replications is to incentivize practices that improve quality and robustness of psychology research. We see Study Diagrams as one of those best practices, and would like to see them become widely adopted in the field.

If you’d like to include a Study Diagram in your research, the sections below walk you through how to create one.

How to diagram a study

Here’s an example study diagram from Report #7:

The Study Diagram is in three parts:

• Hypotheses – a few sentences in plain language explaining the main hypotheses being tested in the study.
• Flowchart of the study – the core of the diagram including information about participants, conditions, study tasks, and exclusions.
• Table of findings – a list of results for the key findings.

Making the flowchart of the study

Participants

The first box includes the number of participants, type of participants, and how they participated.

Although this is typically straightforward, here are two things to pay attention to when reporting on participants:

• Sample criteria filtering and stratifying – If a sample is limited by certain characteristics, this box is where that information belongs. If an eligibility filter is being used to only collect data from certain subgroups of the population, or to collect a certain number of participants in certain categories, that information also belongs here.

• Completed vs. started – Depending on the task and the method being used for data collection it may make sense to report only the number of participants who completed the task, or all of the participants who started the task whether they completed it or not. Either option can be reasonable, but make sure to pay attention here so that the number you are reporting is accurate.

Here’s an example from Report #10 for a study with only one experimental condition, but with more complex requirements for participants:

Study tasks

The next section outlines the tasks that participants did in the study. This section might be one box or a few boxes depending on the complexity of the experimental design. The example diagrams above are for a study with simple randomization to two experimental conditions, and a study with a single condition. The example below is from Report #6 for a study with more complex randomization to multiple conditions:

This section always starts with any initial parts of the experiment that all participants see or complete. Then it goes into the main task which, for studies with multiple conditions, is represented by boxes side-by-side showing what participants in each condition see and do. Finally, if there are parts of the experiment that all participants see or do after the main task, those are presented.

Exclusions

This is the final box of the flowchart, and it reports the number of participants whose data were included in the analysis. It also indicates why other participants were excluded. If participants who completed the study are reported in the first box, then the only exclusions reported here are people who completed the study whose data was not used for some other reason, such as not meeting eligibility criteria. If all participants who started the study are reported in the first box, the number of people who started the study but didn’t complete it would also be reported here.

Making the table of findings

The final section of the Study Diagram is the table of findings. The purpose of this table is to allow the reader to see at a glance what the study tested, and whether the results matched those expectations or not.

Determining what to include in this table can be a bit nuanced. Often there are more results calculated and reported in a paper than would be considered main findings, and including those additional results in this table can make it more difficult for readers to get the high-level overview that the Study Diagram is meant to provide. For example, results related to a manipulation check probably shouldn’t be included in this table. Additionally, if there are multiple statistical tests that pertain to the same claim, reporting those as part of a single row might make sense. 

This first column lists each main claim that was tested, and the later columns present the findings in a simplified way. Typically those findings should be represented with a single word (like “more,” “less,” or “equivalent”) or with a single symbol such as +, -, or 0 to indicate a positive, negative, or null result. With our replication studies, we focus on whether the result from the original study replicated, so the table is designed to make it easy to see if the first column and the second column match. In the case of a study that isn’t a replication, but has pre-registered hypotheses, the table would have a column for the prediction that was made before data collection, and a column for the result. If there were no predictions made in advance, the table would just report the main findings.

What isn’t included in the diagram

You may have noticed that the Study Diagram doesn’t include information about how the statistical tests were conducted. The diagram also doesn’t include actual numerical findings. When we were developing this tool, we determined that it was simply not feasible to include that information while keeping the diagram manageable and understandable. The Study Diagram is not meant to be a replacement for the entire paper.

The Study Diagram gives the reader a quick overview of what participants did and what claims were tested on that basis. The body of the paper is a better place for the level of detail required to explain the statistical methods used, and provide the detailed numerical results.

This means that the Study Diagram is a good starting point for evaluating a study, but determining whether one should have confidence in the reported findings will, of course, continue to require going beyond this tool.

Transparency	Replicability	Clarity
	0 of 1 findings replicated*

We ran a replication of the experiment from this paper which found that as women were exposed to more images of thin bodies, they were more likely to consider ambiguous bodies to be overweight. The finding was not replicated in our study, but this isn’t necessarily evidence against the hypothesis itself.

The study asked participants to make many rapid judgments of pictures of bodies. The bodies varied in body mass index (BMI) with a range from emaciated to very overweight. Each body was judged by participants as either “overweight” or “not overweight”. Participants were randomized into two conditions: “increasing prevalence” and “stable prevalence”. The increasing prevalence condition saw more and more thin bodies as the experiment progressed. Meanwhile, stable prevalence participants saw a consistent mixture of thin and overweight bodies throughout the experiment. The original study found support for their first hypothesis; compared to participants in the stable prevalence condition, participants in the increasing prevalence condition became more likely to judge ambiguous bodies as “overweight” as the experiment continued. The original paper also examined two additional hypotheses about body self-image judgements, but did not find support for them – we did not include these in our replication.

The original study received a high transparency rating due to being pre-registered and having publicly available data, experimental materials, and analysis code, but could have benefitted from more robust documentation of its exclusion criteria. The primary result from the original study failed to replicate; however, this failure to replicate is likely due to an experimental design decision that made the study less sensitive to detecting the effect than anticipated. The images with BMIs in the range where the effect was most likely to occur were shown very infrequently in the increasing prevalence condition. As such, it may not constitute substantial evidence against the hypothesis itself. The clarity rating could have been improved by discussing the implications of hypotheses 2 and 3 having non-significant results for the paper’s overall claims. Clarity could also have been improved by giving the reader more information about the BMIs of the body images shown to participants and the implications of that for the experiment.

Full Report

Study Diagram

Replication Conducted

We ran a replication of the main study from: Devine, S., Germain, N., Ehrlich, S., & Eppinger, B. (2022). Changes in the prevalence of thin bodies bias young women’s judgments about body size. Psychological Science, 33(8), 1212-1225. https://doi.org/10.1177/09567976221082941

How to cite this replication report: Transparent Replications, Svoboda J., & Metskas, A. (2024). Report #11: Replication of a study from “Changes in the prevalence of thin bodies bias young women’s judgments about body size” (Psychological Science | Devine et al., 2022). Clearer Thinking. https://replications.clearerthinking.org/replication-2022psci33-8
(Report DOI: https://doi.org/10.5281/zenodo.17705407)

Key Links

• Our Research Box for this replication report includes the pre-registration, de-identified data, and analysis files.
• Our GitLab repository for this replication report includes the code for running the experiment.

Overall Ratings

To what degree was the original study transparent, replicable, and clear?

Transparency:  how transparent was the original study?	All materials were publicly available or provided upon request, but some exclusion criteria deviated between pre-registration and publication.
Replicability: to what extent were we able to replicate the findings of the original study?	The original finding did not replicate. Our analysis found that the key three-way interaction between condition, trial number, and size was not statistically significant. In this case, lack of replication is likely due to an experimental design decision that made the study less sensitive to detecting the effect than was anticipated, rather than evidence against the hypothesis itself. This means that the original simulated power analysis underestimated the sample size needed to detect the effect with this testing procedure.
Clarity: how unlikely is it that the study will be misinterpreted?	The discussion accurately summarizes the findings related to hypothesis 1 but does not discuss potential implications of lack of support for hypotheses 2 and 3.It is easy to misinterpret the presentation of the spectrum of stimuli used in the original experiment as they relate to the relative body mass indexes of the images shown to participants. Graphical representations of the original data only include labels for the minimum and maximum model sizes, making it difficult to interpret the relationship between judgements and stimuli. The difficulty readers would have determining the thin/overweight cutoff value, and the range of results for which judgements were ambiguous from the information presented in the paper could leave readers with misunderstandings about the study’s methods and results.

Detailed Transparency Ratings

Overall Transparency Rating:
1. Methods Transparency:	All materials are publicly available. There were some inconsistencies between the exclusion criteria reported in the paper, supplemental materials, and analysis code provided. We were able to determine the exact methods and rationale for the exclusion criteria through communication with the original authors.
2. Analysis Transparency:	Analysis code is publicly available.
3. Data availability:	The data are publicly available.
4. Preregistration:	We noted two minor deviations from pre-registered exclusion criteria: The preregistration indicated that participants would be excluded if they record 5 or more trial responses where the time between the stimulus display and participant response input is greater than 7 seconds. This criteria diverges slightly from both the final supplemental exclusion report and the exclusions as executed in the analysis script. Additionally, The preregistration indicated that participants with greater than 90% similar judgements across their trials would be excluded. One participant who met this criteria was included in the final analysis. Overall, these inconsistencies are minor and likely had no bearing on the results of the original study.

Summary of Study and Results

Both the original study (n = 419) and our replication (n = 201) tested for evidence of the cognitive mechanism prevalence-induced concept change as an explanation for shifting body type ideals in the context of women’s body image judgments. 

The original paper tested 3 main hypotheses, but only found support for the first hypothesis. Since the original study didn’t find support for hypotheses 2 or 3, our replication focused on testing hypothesis 1: “…if the prevalence of thin bodies in the environment increases, women will be more likely to judge other women’s bodies as overweight than if this shift did not occur.” Our pre-registration of our analysis plan is available here. 

Prevalence-induced concept change happens when a person starts seeing more and more cases of a specific conceptual category. For example, we can consider hair color. Red hair and brown hair are two different conceptual categories of hair color. Some people have hair that is obviously red or obviously brown, but there are many cases where it could go either way. We might call these in-between cases “auburn” or “reddish-brown” or even “brownish-red”. If a person starts seeing many many other people with obviously red hair, then they might start thinking of auburn hair as obviously brown. Their conceptual class of “red hair” has shrunk to exclude the ambiguous cases. 

To test prevalence induced concept change in women’s body image, both the original study and our replication showed participants computer-generated images of women’s bodies and asked participants to judge whether they thought any given body was “overweight” or “not overweight”. The image library included 61 images, ranging from a BMI minimum of 13.19 and a maximum BMI of 120.29. Each participant was randomly assigned to one of two conditions: stable-prevalence or increasing-prevalence. Stable-prevalence participants saw an equal 50/50 split of images of bodies with BMIs above 19.79 (the “overweight” category)¹ and images of bodies with BMIs below 19.79 (the “thin” category). Increasing-prevalence participants saw a greater and greater proportion of bodies with BMIs below 19.79 as the experiment proceeded. If participants in the increasing-prevalence condition became more likely to judge thin or ambiguous bodies as overweight in the later trials of the experiment, compared to participants in the stable-prevalence condition, that would be evidence supporting the hypothesis of prevalence-induced concept change affecting body image judgements.

Overview of Main Task

During the task, participants were shown an image of a human body (all images can be found here). The body image stimulus displays on screen for 500 milliseconds (half of a second), followed by 500 milliseconds of a blank screen and finally a question mark, indicating to participants that it is time to input a response. Participants then recorded a binary judgment by pressing the “L” key on their keyboard to indicate “overweight” or by pressing the “A” key to indicate “not overweight”. Judgements were meant to be made quickly, between 100 and 7000 milliseconds, for each trial. This process was repeated for 16 blocks of 50 iterations each, meaning that each participant recorded 800 responses. 

Additionally, participants completed a self-assessment once before and once after the main task. For this assessment, participants chose a body image from the stimulus set which most closely resembled their own body. Participants were asked to judge the self-representative body from their first self-assessment as “overweight” or “not overweight” before completing their second and final self-assessment. These self-assessments were used for testing hypothesis 2, hypothesis 3, and the exploratory analyses in the original paper. We focused on hypothesis 1 so did not include self-assessment data in our analysis.

Figure 1: Example frames from the task

A (Introduction)

B (Example instruction frame)

C (Block 1 start)

D (Stimulus display [500ms])

E (Prompt to respond)

Results

The original study found a significant three-way interaction between condition, trial, and size (β = 3.85, SE = 0.38, p = 1.09 × 10⁻²³), meaning that participants were more likely to judge ambiguous bodies as “overweight” as they were exposed to more thin bodies over the course of their trials. Our replication, however, did not find this interaction to be significant (β = 0.53, SE = 1.81, p = 0.26). Although it was not significant, the effect was in the correct direction, and the lack of significance may be due to an experimental design decision resulting in lower-than-estimated statistical power. 

Study and Results in Detail

Main Task in Detail

Both the original study and our replication began with a demographic questionnaire. In our replication, the demographic questionnaire from the original study was pared down to only include questions relevant for exclusion criterion and a potential supplemental analysis regarding the original hypothesis 3. The maintained questions are listed below.

• What is your gender?
  • Options: Female, Male, Transgender, Non-Binary
• What is your age in years?
• For statistical purposes, what is your weight?
• For statistical purposes, what is your height?
• Please enter your date of birth.
• Please enter your (first) native language.

We included an additional screening question to ensure recruited participants were able to complete the task.

• Are you currently using a device with a full keyboard?
  • Options: “Yes, I am using a full keyboard”, “No”

The exact proportion of bodies under 19.79 BMI presented out of the total bodies per block for each condition are detailed in Figure 2. Condition manipulations relative to stimuli BMI can be seen in Figure 3. 

Figure 2: Stimuli Prevalence by Condition and Block, Table

Figure 3: Estimated BMI of Stimuli and World Health Organization Categories

Stimulus	Categorization for Study Conditions	BMI	WHO Classification**
T300	Thin	13.19	Thin
T290	Thin	13.38	Thin
T280	Thin	13.47	Thin
T270	Thin	13.77	Thin
T260	Thin	13.86	Thin
T250	Thin	14.10	Thin
T240	Thin	14.28	Thin
T230	Thin	14.46	Thin
T220	Thin	14.65	Thin
T210	Thin	14.87	Thin
T200	Thin	15.06	Thin
T190	Thin	15.24	Thin
T180	Thin	15.49	Thin
T170	Thin	15.67	Thin
T160	Thin	15.74	Thin
T150	Thin	16.12	Thin
T140	Thin	16.40	Thin
T130	Thin	16.64	Thin
T120	Thin	16.81	Thin
T110	Thin	17.08	Thin
T100	Thin	17.28	Thin
T090	Thin	17.56	Thin
T080	Thin	17.77	Thin
T070	Thin	18.01	Thin
T060	Thin	18.26	Thin
T050	Thin	18.50	Normal Range
T040	Thin	18.77	Normal Range
T030	Thin	19.1	Normal Range
T020	Thin	19.3	Normal Range
T010	Thin	19.61	Normal Range
N000	NA*	19.79	Normal Range
H010	Overweight	21.55	Normal Range
H020	Overweight	23.35	Normal Range
H030	Overweight	25.37	Overweight
H040	Overweight	27.37	Overweight
H050	Overweight	29.57	Overweight
H060	Overweight	31.84	Overweight
H070	Overweight	34.13	Overweight
H080	Overweight	36.58	Overweight
H090	Overweight	39.10	Overweight
H100	Overweight	41.76	Overweight
H110	Overweight	44.55	Overweight
H120	Overweight	47.37	Overweight
H130	Overweight	50.23	Overweight
H140	Overweight	53.21	Overweight
H150	Overweight	56.26	Overweight
H160	Overweight	59.31	Overweight
H170	Overweight	62.64	Overweight
H180	Overweight	66.04	Overweight
H190	Overweight	69.56	Overweight
H200	Overweight	73.30	Overweight
H210	Overweight	76.95	Overweight
H220	Overweight	80.98	Overweight
H230	Overweight	85.49	Overweight
H240	Overweight	89.89	Overweight
H250	Overweight	94.40	Overweight
H260	Overweight	99.27	Overweight
H270	Overweight	104.4	Overweight
H280	Overweight	109.45	Overweight
H290	Overweight	114.82	Overweight
H300	Overweight	120.29	Overweight

Data Collection

Data were collected using the Positly recruitment platform and the Pavlovia experiment hosting platform. Data collection began on the 15th of May, 2024 and ended on the 5th of August, 2024.

In consultation with the original authors we determined that a sample size of 200 participants after exclusions would provide adequate statistical power for this replication effort. In the simulations for the original study the authors determined that 140 participants would provide 89% power to detect their expected effect size for hypothesis 1. Typically for replications we aim for a 90% chance to detect an effect that is 75% of the size of the original effect size. To emulate that standard for this study we decided on a sample of 200 participants. It is important to note that the original study had 419 participants after exclusions. This final sample size for the original study was set by simulation-based power analyses for hypotheses 2 and 3 requiring a sample size of ~400 participants for adequate statistical power. Because our replication study did not test hypotheses 2 and 3–since they weren’t supported in the original analysis–we did not need to power the study based on those hypotheses.

While a sample size of 200 subjects was justified at the time, we later learned that the original simulation-based power analysis relied on faulty assumptions, which could only be determined from the empirical data in the original sample. The sample size needed to provide adequate statistical power for hypothesis 1 was underestimated. Because the original study used a larger sample size to power hypotheses 2 and 3, the underestimate of the sample size needed for hypothesis 1 wasn’t detected. As a result, our replication study may have been underpowered.

Excluding Participants and/or Observations

For participants to be eligible to take part in the study, they had to be:

1. Female
2. Aged 18-30
3. English speaking

After data collection, participants were excluded from the analysis under the following circumstances:

1. Participants who took longer than 7 seconds to respond in more than 10 trials.
2. Participants who demonstrated obviously erratic behavior e.g. repeated similar responses across long stretches of trials despite variation in stimuli (see Additional Information about the Exclusion Criteria appendix section).
3. Participants who did not complete the full 800 trials.
4. Participants who do not meet the eligibility criteria.

Additionally, at the suggestion of the original authors, we excluded any observations in which the response was given more than 7 seconds after the display of the stimulus.

249 participants completed the main task. 8 participants did not have their data written due to technical malfunctions (these participants were still compensated as usual). 8 participants were excluded for reporting anything other than “Female” for their gender on the questionnaire. 23 participants were excluded for being over 30 years old. 6 participants were excluded for taking longer than 7 seconds to respond on more than 10 trials. 4 participants were excluded for obviously erratic behavior. Note that some participants fall into two or more of these exclusion categories, so the sum of exclusions listed above is greater than the total number of excluded participants.

Analysis

Both the original paper and our replication utilized a logistic multilevel model to assess the data:

Y_ij = 𝛽_0j + 𝛽_1jTrial_ij + 𝛽_2jSize_ij + 𝛽_3j(Trial_ij x Size_ij)

𝛽_0j = Ɣ₀₀ +  Ɣ₀₁ Condition_j + U_oj

𝛽_1j =  Ɣ₁₀ + Ɣ₁₁ Condition_j + U_1j

𝛽_2j = Ɣ₂₀ + Ɣ₂₁Condition_j

𝛽_3j = Ɣ₃₀ + Ɣ₃₁Condition_j

Where Size is the ordinal relative BMI of computerized model images. That is, the degree to which each body image stimulus is thin or overweight. 

Y_ij represents the log odds of participant j making an “overweight” judgment for trial i. 

U_oj are random intercepts per participant. U_1j are random slopes per participant. Ɣ_xx represents fixed effects. 

Results in Detail

The original study found a significant three-way interaction between condition, trial number, and size (β = 3.85, SE = 0.38, p = 1.09 × 10⁻²³), indicating that as the prevalence of thin bodies in the environment increased, participants were more likely to judge ambiguous bodies (not obviously overweight and not obviously underweight) as overweight. The authors note that this effect is restricted to judgements of “thin and average-size stimuli” due to the increasing-prevalence condition requiring a low frequency of “overweight” stimuli. 

Figure 4: Original Results Table

Figure 5: Original Results Data Representations

From “Changes in the prevalence of thin bodies bias young women’s judgments about body size,” by Devine, S., Germain, N., Ehrlich, S., & Eppinger, B., 2022, Psychological Science, 33(8), 1212-1225.

Figure 6: Replication Results Table

Figure 7: Replication Results Data Representation

Interpreting the Results

The failure of this result to replicate is likely to be due to characteristics of the study design that made the experiment a less sensitive test of the hypothesis. For that reason the failure of this study to replicate should not be taken as strong evidence against the original hypothesis that prevalence induced concept change occurs for body images.

The main study design issue that could possibly account for the non-replication of the results is the categorization of “thin” and “overweight” images for the condition manipulation: “thin” images were 19.61 BMI and below, and “overweight” images were 21.55 BMI and above. This low threshold means that participants in the increasing prevalence condition would have seen a very small number of images of bodies that were in the ambiguous or normal range of BMI in which prevalence induced concept change is most likely to occur. Unfortunately, we did not notice this issue with the BMI cutoff between the thin and overweight groups until after we had collected the replication data. This means that our replication, while having the benefit of being faithful to the original study, has the drawback of being affected by the same study design issue.

We presented this issue to the authors after determining that it may explain the lack of replication. The authors explained their rationale for setting the image cutoff at the baseline image:

“In designing the study, we anticipated the most “ambiguous” stimuli to be those near the reference image (BMI of 19.79; the base model). This was based on two factors. First, WHO guidelines suggest that a “normal” BMI lies between 18.5 and 24.9—hence a BMI of 19.79 fell nicely within this range and, as mentioned, allowed for a clean division of the stimulus set into two equal categories. Second, irrespective of the objective BMI, we anticipated participants would judge the reference image as maximally ambiguous in the context of the stimulus set, owing to the range available to participants’ judgements when completing the experiment. Accordingly, the power analysis we conducted was based on this assumption that responses most sensitive to PICC would be those to images near in size to the reference image. But this turned out not to be the case when we acquired the data from our sample. As you point out, increased sensitivity to PICC was at a slightly higher (and evidently under-sampled) range of size (BMI 23.35 – 31.84). As such, the sample size required to detect effects in these ranges with sufficient power may be higher than previously thought.” (Devine, email communication 9/11/24) 

Understanding the Categorization Used

It took us some time to recognize this issue because the original paper does not clearly explain how the “thin” and “overweight” image categories correspond to BMI values of the images, and none of the figures in the original paper show BMI values along the axes representing image sizes. From the paper alone it is not possible for a reader to determine what BMI values the stimuli presented correspond to, with the exception of the endpoints. The paper says: 

Specifically, the proportion of thin to overweight bodies had the following order across each block in the increasing-prevalence condition: .50, .50, .50, .50, .60, .72, .86, .94, .94, .94, .94, .94, .94, .94, .94, .94. In the stable-prevalence condition, the proportion of overweight and thin bodies in the environment did not change; it was always .50 (see Fig. 1b). Bodies were categorized as objectively thin or overweight by Moussally et al. (2017) according to World Health Organization (1995) guidelines. Body mass index (BMI) across all bodies ranged from 13.19 (severely underweight) to 120.29 (very severely obese). (Devine et al, 2022) [Bold italics added for emphasis]

From the information provided in the paper, a reader would be likely to assume that the images in the “overweight” category had BMIs of greater than 25, because a BMI of 25 is the dividing line between “healthy/normal” and “overweight” according to the WHO. Another possible interpretation of this text in the paper would suggest that the bodies that were categorized as thin and/or median in the Moussally et al. (2017) stimulus validation paper were the ones increasing in prevalence in that condition, and those categorized as overweight in the validation study were diminishing in prevalence. Either of these likely reader assumptions would also be supported by the presentation of the results in the original paper:

Most importantly, we found a three-way interaction between condition, trial, and size (β = 3.85, SE = 0.38, p = 1.09 × 10−23). As seen in Figure 2a, this result shows that when the prevalence of thin bodies in the environment increased over the course of the task, participants judged more ambiguous bodies (average bodies) as overweight than when the prevalence remained fixed. We emphasize here that this effect is restricted to judgments of thin and average-size stimuli because the nature of our manipulation reduced the number of overweight stimuli seen by participants in the increasing-prevalence condition (as reflected by larger error bars for larger body sizes in Fig. 2a).  (Devine et al, 2022) [Bold italics added for emphasis]

Moussally et al. developed the stimuli that were used in this study by using 3D modeling software. They started with a default female model (corresponding to 19.79 BMI according to their analysis), scaling down from that default model in 30 increments of the modeling software’s “thin/heavy” dimension to get lower BMIs (down to a low of 13.9), and then scaling up from that default model by 30 increments to get higher BMIs (up to a high of 120.29). They then validated the image set by asking participants to rate images on a 9 point Likert scale where 1 = “fat” and 9 = “thin”. Based on those ratings they established three categories for body shape: “thin, median, or fat.”

Figure 8: Ratings of Body Shape for all Stimuli from Moussally et al. (2017)

Stimulus	BMI	Mean Rating (Likert 1-9)	Validation Study Classification
T300	13.19	8.94	Thin
T290	13.38	8.95	Thin
T280	13.47	8.97	Thin
T270	13.77	8.88	Thin
T260	13.86	8.91	Thin
T250	14.10	8.86	Thin
T240	14.28	8.77	Thin
T230	14.46	8.70	Thin
T220	14.65	8.63	Thin
T210	14.87	8.67	Thin
T200	15.06	8.59	Thin
T190	15.24	8.56	Thin
T180	15.49	8.37	Thin
T170	15.67	8.18	Thin
T160	15.74	8.22	Thin
T150	16.12	8.11	Thin
T140	16.40	8.12	Thin
T130	16.64	8.05	Thin
T120	16.81	7.95	Thin
T110	17.08	7.90	Thin
T100	17.28	7.79	Thin
T090	17.56	7.90	Thin
T080	17.77	7.79	Thin
T070	18.01	7.88	Thin
T060	18.26	7.74	Thin
T050	18.50	7.84	Thin
T040	18.77	7.76	Thin
T030	19.1	7.74	Thin
T020	19.3	7.78	Thin
T010	19.61	7.50	Thin
N000	19.79	7.63	Thin
H010	21.55	7.28	Thin
H020	23.35	6.21	Median
H030	25.37	5.65	Median
H040	27.37	5.26	Median
H050	29.57	4.85	Median
H060	31.84	4.28	Median
H070	34.13	3.63	Fat
H080	36.58	3.62	Fat
H090	39.10	3.10	Fat
H100	41.76	2.78	Fat
H110	44.55	2.65	Fat
H120	47.37	2.45	Fat
H130	50.23	2.32	Fat
H140	53.21	2.02	Fat
H150	56.26	1.95	Fat
H160	59.31	1.68	Fat
H170	62.64	1.56	Fat
H180	66.04	1.59	Fat
H190	69.56	1.44	Fat
H200	73.30	1.45	Fat
H210	76.95	1.30	Fat
H220	80.98	1.23	Fat
H230	85.49	1.17	Fat
H240	89.89	1.16	Fat
H250	94.40	1.11	Fat
H260	99.27	1.06	Fat
H270	104.4	1.09	Fat
H280	109.45	1.10	Fat
H290	114.82	1.06	Fat
H300	120.29	1.05	Fat

The “median” images according to the judgements reported in Moussally et. al. (2017) ranged from a BMI of 23.35 to 31.84; however, neither of those cutoffs nor the commonly used WHO BMI guideline of 25 and above as “overweight” were used to set the cutoff between the groups of “thin” and “overweight” images in the experiment we replicated. From looking at the study code itself, this study used the 30 images scaled down from the baseline image of 19.79 BMI as the “thin” group and the 30 images scaled up from the baseline as the “overweight” group. The 19.79 BMI image was not included in either group, so it was not presented to participants in the experiment. That means that the “thin” images that were increasing in prevalence ranged from a BMI of 13.19 to 19.61, and the “overweight” images that were decreasing in prevalence ranged from a BMI of 21.55 to 120.29. The 21.55 BMI image was categorized as “thin” in the Moussally et al. (2017) validation study, and is well within the normal/healthy weight range according to the WHO, and yet it was categorized with the “overweight” images in this study. This 21.55 BMI image was judged as “not overweight” for 96% of trials in the original dataset for the present study, further suggesting that the experiment’s cutoff between “thin” and “overweight” was placed at too low of a BMI to adequately capture data for ambiguous body images.

Implications of the Categorization

Figure 2b in the original paper presents the results for a BMI of 23.35, which is within the “normal/healthy” range according to the WHO, and is the lowest BMI “median” image according to the validation study. This is clearly meant to be one of the normal or ambiguous body sizes for which prevalence induced concept change would be most expected. The inclusion of this image in the “overweight” grouping for which the prevalence was decreasing means this image would not have been shown to participants very often. The caveat in the results section that “this effect is restricted to judgments of thin and average-size stimuli because the nature of our manipulation reduced the number of overweight stimuli seen by participants in the increasing-prevalence condition,” applies to the 23.35 BMI image that is presented in the paper as a demonstration of the effect.

In the last 200 trials in the increasing prevalence condition only 6% of the images presented would have been from the set of 30 “overweight” images. That means that each participant only saw 12 presentations of “overweight” images in the last 200 trials. Each individual subject in the increasing prevalence condition would only have had an approximately 33% chance of seeing the BMI 23.35 image at least once during the last 200 trials. Ideally, this image–and others in the ambiguous range–should be shown much more frequently in order to capture possible effects of prevalence induced concept change.

In the original study, looking at the last 200 trials, the 23.35 BMI image was presented only 80 times out of 42,600 image presentations to the increasing prevalence condition. In the replication study, looking at the last 200 trials, that image was only presented 51 times out of 20,000 image presentations to the increasing prevalence condition.

Figure 9 below shows how many times stimuli of BMI values 18.77 – 31.84 were presented and what percentage of them were judged as “overweight”  in the last 200 trials in each condition across all subjects for the original dataset and the replication dataset. The rows that are color coded by condition and have BMI values in bold are from the “overweight” group.

Figure 9: Data Presentation Frequency and % “Overweight” judgements in last 200 trials

A (Original Data)

B (Replication Data)

From looking at these tables, it’s easy to see that in both conditions only a small percentage of the stimuli from 18.77 to 19.61 BMI were judged to be overweight. There is much more variation in judgment in the 21.55 to 31.84 BMI images, but the number of times those were presented in the increasing prevalence condition was very small. The fact that the most important stimuli for demonstrating the proposed effect were presented extremely infrequently in the study likely undermines the reliability of this test of the prevalence induced concept change hypothesis by making it much less sensitive to detecting whether the effect is present.

Implications of Nonreplication for the Prevalence Induced Concept Change Hypothesis

If we look more closely at the results for the range of BMI values for which there is ambiguity in both the original data and the replication data we can see that the pattern of results for those values looks similar.

Figure 10: Data for the last 200 trials

A (Original Data)

B (Replication Data)

Figure 10 above shows that only one datapoint in the replication data has results that are clearly outside of the margin of error (BMI = 29.57), but the pattern looks similar to what we see in the original data. This suggests that despite the issues with the experimental design, the original study may have been able to detect an effect because it was much more highly powered than should have been necessary to test this hypothesis due to the need for a higher statistical power for hypotheses 2 and 3 in the original paper. In the replication study, which was powered appropriately according to the original study’s simulation analysis, the effective power was lower than what was simulated due to the miscategorization of the ambiguous images into the overweight group.

Proposed Experimental Design Changes

In our view, a better threshold between the “thin” and “overweight” images for testing this hypothesis would be 31.84 (the high end of the “median” range reported in the Moussally et. al. (2017) paper). This threshold would ensure that participants are presented with many opportunities to judge the images that are in the ambiguous range where prevalence induced concept change is most likely to be observed. Shifting to this threshold would make this experiment better suited to detecting the hypothesized effect.

Additionally, this experiment would benefit from having more stimuli that are in the ambiguous range of values – i.e. more stimuli with BMIs between 23.35 and 31.84. In this study only 5 of the images (23.35, 25.37, 27.37, 29.57, 31.84) are in the range Moussally et al. determine to be “median.” A larger set of stimuli in the ambiguous range would provide more data points in the relevant range for testing the hypothesis. We recognize that this change would require developing and validating additional stimuli, which would be labor-intensive.

Comparing the stimuli used in this study to those used in the Levari et al. (2018) experiment–on which this study is based–provides an illustration that helps explain why this would be important for testing this hypothesis. Levari et al. tested prevalence induced concept change using images of 100 dots that ranged in color from purple to blue. When they decreased the prevalence of blue dots, they found that people were more likely to consider ambiguous dots to be blue. Stimuli from Levari et al.’s paper can be seen in Figure 11c, where there are 18-19 stimuli at color values in between each of the dots shown. From looking at these representative stimuli it’s clear that there were many examples of different stimuli in the range of values that were ambiguous.

Figure 11: Levari et al. (2018) Colored Dots Study 1

A-B (Results visualization)

C (Color spectrum stimuli examples)

Prevalence-induced concept change should be observable mainly in ambiguous stimuli. We expect this effect to be non-existent for the extreme exemplars of the relevant conceptual category. That is, the bluest dots will always be identified as blue, but judgements of ambiguous dots should be susceptible to the effect. Looking at Figures 11a-11b, a substantial fraction of the 100 different dot images were ambiguous (identified as blue some of the time, rather than 100% or 0% of the time). A wide range of ambiguous stimuli make this effect easier to capture. Additionally, these ambiguous dots were clustered on the purple half of the color spectrum. This is important because Levari et al.’s manipulation increased the frequency of the purple-spectrum dots. So, their data contained many observations of ambiguous dots despite the condition manipulation decreasing the frequency of blue-spectrum dots. Compare the above Figures 11a-11b from Levari et al. to the below Figure 12 generated from the original body image study data:

Figure 12: PICCBI Original Results Visualization

It’s not possible to see the curve shift in the increasing prevalence condition here (Figure 12), despite the model having a significant result. This is likely because there are many fewer observations in the ambiguous range of stimuli. This makes the model more sensitive to noise at the extreme values. Looking at these figures for the replication data in Figure 13, we see that noise in the infrequently presented larger BMI images shapes the divergence between the curves in a way that’s not consistent with the hypothesis:

Figure 13: PICCBI Replication Results Visualization

Taking more measurements in the ambiguous range by having more stimulus images with BMI values in that range would improve the ability of this experiment to reliably detect whether prevalence induced concept change occurs for body images.

It’s also worth noting that this issue with the study design was somewhat obscured by the design of the figures presenting the data in the paper. Instead of using the curves above like the Levari et al. (2018) paper used, the data for this study was presented by showing the percentage of overweight ratings for the first 200 trials subtracted from the last 200 trials, as seen in Figure 5. This method highlights the relevant change from the early trials to the later trials, but has the downside of not clearly presenting the actual values. Many of those values didn’t change from the early to the late trials because they were near the ceiling or the floor (almost all judgements were one-way). It was not possible to tell what the actual percentages of overweight judgements were from the information presented in the paper, which meant it was not clear which stimuli had overweight judgements near the ceiling or floor and which were ambiguous. Being able to tell where the ambiguous values were would have been useful to readers attempting to interpret the results of this study.

By incorporating these changes, a new version of this study would shed a lot of light on the question of whether prevalence induced concept change can be reliably detected for body images.

Conclusion

The results of the original paper failed to replicate, which we suspect was due to the experiment being less sensitive to the effect than anticipated. For this reason we emphasize that our findings do not provide strong evidence against the original hypothesis. Prevalence-induced concept change may affect women’s body image judgements, but the present experiment was not as sensitive to detect this effect with the current sample size as previously believed. The design could be improved by raising the BMI cutoff between “thin” and “overweight” images for the prevalence manipulation and/or including additional stimuli within the range of ambiguous body sizes (BMI 23.35 – 31.84) to increase the frequency of ambiguous stimuli, which are the most important for demonstrating a change in concept.

The clarity rating of 2.5 stars was due to two factors. The original discussion section did not address the potential implications of the lack of support for hypotheses 2 and 3. Since hypotheses 2 and 3 related to people applying these changes in the concept of thinness to their own bodies, the lack of support for those hypotheses may limit the claims that should be made about potential real world effects of prevalence induced concept change for body image. Additionally, the difficulty of determining the stimulus BMI values, the thin/overweight cutoff value, and the range of results for which judgements were ambiguous from the information presented in the paper could leave readers with misunderstandings about the study’s methods and results. 

The study had a high transparency rating of 4.5 stars because all of the original data, experiment/analysis code, and pre-registration materials were publicly available. There were minor discrepancies in exclusion criteria based on reaction times between the pre-registration and the analysis, and some documentation for exclusion criteria and code for evaluating participant quality wasn’t publicly posted. However, the undocumented code was provided upon request, and the inconsistency in exclusion criteria was subtle and likely had no bearing on the results.

Author Acknowledgements

We would like to thank the authors of “Changes in the Prevalence of Thin Bodies Bias Young Women’s Judgements About Body Size”: Sean Devine, Nathalie Germain, Stefan Ehrlich, and Ben Eppinger for everything they’ve done to make this replication report possible. We thank them for their original research and for making their data, experiment code, analysis, and other materials publicly available. The original authors provided feedback with expedient, supportive correspondence and this report was greatly improved by their input. 

Thank you to Isaac Handley-Miner for your consultation on multilevel modeling for our analysis. Your expertise was invaluable.

Thank you to Soundar and Nathan from the Positly team for your technical support with the data collection. 

Thank you to Spencer Greenberg for your guidance and feedback throughout the project.

Last, but certainly not least, thank you to all 249 individuals who participated in the replication experiment.

Response from the Original Authors

The original paper’s authorship team offers this response (PDF) to our report. We are grateful for their thoughtful engagement with our report.

Purpose of Transparent Replications by Clearer Thinking

Transparent Replications conducts replications and evaluates the transparency of randomly-selected, recently-published psychology papers in prestigious journals, with the overall aim of rewarding best practices and shifting incentives in social science toward more replicable research.
We welcome reader feedback on this report, and input on this project overall.

Appendices

Additional Information about the Exclusion Criteria

249 participants completed the main task

• 8 participants were excluded due to technical malfunctions.
  • 5 of these participants did not have their data written due to terminating their connection to Pavlovia before the data saving operations could complete. These participants were compensated for completion of the full task.
  • 3 of these participants were excluded for incomplete data sets.These 3 exclusions stand out as unexplained data writing malfunctions. These participants were compensated for completion of the full task, despite the partial datasets. 
• 8 participants were excluded for reporting anything other than “Female” for their gender on the questionnaire. 
• 23 participants were excluded for being over 30 years old. 
• 6 participants were excluded for taking longer than 7 seconds to respond on more than 10 trials. 
• 4 participants were excluded for obviously erratic behavior. 

The “erratic behavior” exclusions were determined by generating graphical representations of individual subject judgements over time and manually reviewing them for signs of unreasonable behavior. The code for generating these individual subject graphs was provided by the original authors and we consulted with the original authors on their assessment of the graphs. The generation code and a complete set of graphics can be found in our gitlab repository. Figure 14a is an example of expected behavior from a participant. They tended to judge very thin stimuli as “not overweight” and very overweight stimuli as “overweight” with some variance, especially around ambiguous stimuli closer to the middle of the spectrum. Figures 14b-14e are the subjects we excluded based on their curves. 14b made judgments exactly opposite the expected behavior for their first 200 trials which indicates that this participant was confused about which key on their keyboard related to which judgment. In 14c, we see that this participant’s judgements in the last 200 trials were completely random. They likely stopped paying attention at some point during the task and assigned judgements randomly. Because this criterion is somewhat subjective, only the most obviously invalid data were excluded. Any participants with questionable but ambiguous curves had their data included to avoid the possibility of biased exclusions.

Figure 14: Individual Subject Curves

A  (Good Subject Curve)

B

C

D

E

References

Devine, S., Germain, N., Ehrlich, S., & Eppinger, B. (2022). Changes in the prevalence of thin bodies bias young women’s judgments about body size. Psychological Science, 33(8), 1212-1225. https://doi.org/10.1177/09567976221082941

Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149-1160. Download PDF

Levari, D. E., Gilbert, D. T., Wilson, T. D., Sievers, B., Amodio, D. M., & Wheatley, T. (2018). Prevalence-induced concept change in human judgment. Science, 360(6396), 1465-1467. https://doi.org/10.1016/j.cognition.2022.105196

Moussally, J. M., Rochat, L., Posada, A., & Van der Linden, M. (2017). A database of body-only computer-generated pictures of women for body-image studies: Development and preliminary validation. Behavior research methods, 49(1), 172-183. https://doi.org/10.3758/s13428-016-0703-7

World Health Organization. (1995). Physical status: The use of and interpretation of anthropometry. Report of a WHO expert committee. https://apps.who.int/iris/handle/10665/37003

1. We are using the category labels “thin” and “overweight” because they were used in the original paper. These labels do not necessarily correspond to what they would mean in everyday usage, and should not be taken as objective measures of health, perception, nor the opinions of the researchers. More information on the decisions behind the categorizations can be found in the Understanding the Categorizations Used section. ︎

A significant and pretty common problem I see when reading papers in social science (and psychology in particular) is that they present a fancy analysis but don’t show the results of what we have named the “Simplest Valid Analysis” – which is the simplest possible way of analyzing the data that is still a valid test of the hypothesis in question.

This creates two potentially serious problems that make me less confident in the reported results:

1. Fancy analyses impress people (including reviewers), but they are often harder to interpret than simple analyses. And it’s much less likely the reader really understands the fancy analysis, including its limitations, assumptions, and gotchas. So, the fancy analysis can easily be misinterpreted, and is sometimes even invalid for subtle reasons that reviewers, readers (and perhaps the researchers themselves) don’t realize. As a mathematician, I am deeply unimpressed when someone shows me a complex mathematical method when a simple one would have sufficed, but a lot of people fear or are impressed by fancy math, so complex analyses can be a shield that people hide behind.

2. Fancy analyses typically have more “researcher degrees of freedom.” This means that there is more wiggle room for researchers to choose an analysis that makes the results look the way the researcher would prefer they turn out. These choices can be all too easy to justify for many reasons including confirmation bias, wishful thinking, and a “publish or perish” mentality. In contrast, the Simplest Valid Analysis is often very constrained, with few (if any) choices left to the researcher. This makes it less prone to both unconscious and conscious biases.

When a paper doesn’t include the Simplest Valid Analysis, I think it is wise to downgrade your trust in the result at least a little bit. It doesn’t mean the results are wrong, but it does mean that they are harder to interpret.

I also think it’s fine and even good for researchers to include more sophisticated (valid) analyses and to explain why they believe those are better than the Simplest Valid Analysis, as long as the Simplest Valid Analysis is also included. Fancy methods sometimes are indeed better than simpler ones, but that’s not a good reason to exclude the simpler analysis.

Here are some real-world examples where I’ve seen a fancier analysis used while failing to report the Simplest Valid Analysis:

• Running a linear regression with lots of control variables when there is no need to control for all of these variables (or no need to control for more than one or two of the variables)
• Use of ANOVA with lots of variables when really the hypothesis only requires a simple comparison of two means
• Use of a custom statistical algorithm when a very simple standard algorithm can also test the hypothesis
• Use of fancy machine learning when simple regression algorithms may perform just as well
• Combining lots of tests into one using fancy methods rather than performing each test one at a time in a simple way

The problems that can occur when the results of Simplest Valid Analysis aren’t reported was one of the reasons that we decided to include a Clarity Criterion in our evaluation of studies for Transparent Replications. As part of evaluating a study’s Clarity, if it does not present the results of the Simplest Valid Analysis, we determine what that analysis would be, and pre-register and conduct the Simplest Valid Analysis on both the original data and the new data we collect for the replication. Usually it is fairly easy to determine what the Simplest Valid Analysis would be for a research question, but not always. When there are multiple analyses that could be used as the Simplest Valid Analysis, we select the one that we believe is most likely to be informative, and we select that analysis prior to running analyses on the original data and prior to collecting the replication data.

In my view, while it is very important that a study replicates, replication alone does not guarantee that a study’s results reflect something real in the world. For that to be the case, we also have to be confident that the results obtained are from valid tests of the hypotheses. One way to increase the likelihood of that being the case is to report the results from the Simplest Valid Analysis.

My advice is that, when you’re reading scientific results, look for the Simplest Valid Analysis, and if it’s not there, downgrade your trust in the results at least a little bit. If you’re a researcher, remember to report the Simplest Valid Analysis to help your work be trusted and to help avoid mistakes (I aspire always to do so, though there have likely been times I have forgotten to). And if you’re a peer reviewer or journal editor, ask authors to report the Simplest Valid Analysis in their papers in order to reduce the risk that the results have been misinterpreted.

Transparency	Replicability	Clarity
	4 of 4 findings replicated

We ran a replication of Study 5b from this paper. This study tested whether people believe that morality is declining over time. 

The paper noted that people encounter disproportionately negative information about current-day people (e.g., via the media) and people often have weaker emotional responses to negative events from the past. As such, the authors hypothesized that participants would think people are less moral today than people used to be, but that this perception of moral decline would diminish when comparing timepoints before participants were born.    

To test these hypotheses, the study asked each participant to rate how “kind, honest, nice, and good” they thought people are today and were at four previous timepoints corresponding, approximately, to when participants were 20 years old, when they were born, 20 years before they were born, and 40 years before they were born.

The results from the original study confirmed the authors’ predictions: Participants perceived moral decline during their lifetime, but there was no evidence of perceived moral decline for the time periods before participants were born. 

Our replication found the same pattern of results.

​​The study received a transparency rating of 4.25 stars because its materials, data, and code were publicly available, but it was not pre-registered. The paper received a replicability rating of 5 stars because all of its primary findings replicated. The study received a clarity rating of 5 stars because the claims were well-calibrated to the study design and statistical results.

Full Report

Study Diagram

Replication Conducted

We ran a replication of Study 5b from: Mastrioanni, A.M., & Gilbert, D.T. (2023). The illusion of moral decline. Nature, 618, 782–789. https://doi.org/10.1038/s41586-023-06137-x 

How to cite this replication report: Transparent Replications by Clearer Thinking. (2024). Report #10: Replication of a study from “The illusion of moral decline” (Nature | Mastroianni & Gilbert 2023) https://replications.clearerthinking.org/replication-2023nature618
(Preprint DOI: https://doi.org/10.31234/osf.io/32xvw)

Key Links

• Our Research Box for this replication report includes the pre-registration, study materials, de-identified data, and analysis files.

Overall Ratings

To what degree was the original study transparent, replicable, and clear?

Transparency: how transparent was the original study?	All materials, analysis code, and data were publicly available. The study was not pre-registered.
Replicability: to what extent were we able to replicate the findings of the original study?	All primary findings from the original study replicated.
Clarity: how unlikely is it that the study will be misinterpreted?	This study is explained clearly, the statistics used for the main analyses are straightforward and interpreted correctly, and the claims were well-calibrated to the study design and statistical results.

Detailed Transparency Ratings

Overall Transparency Rating:
1. Methods Transparency:	The materials were publicly available and complete.
2. Analysis Transparency:	The analysis code was publicly available and complete. We successfully reproduced the results in the original paper from the publicly available code and data.
3. Data availability:	The raw data were publicly available and complete.
4. Preregistration:	The study was not pre-registered.

Summary of Study and Results

Summary of the hypotheses

The original study made two key predictions: 

1. For time periods during study participants’ lifetimes, participants would perceive moral decline. In other words, they would believe people are morally worse today than people were in the past.
2. For time periods before participants were born, participants’ perceptions of moral decline would diminish, disappear, or reverse (relative to the time periods during their lifetimes). 

The original paper argues that these results are predicted by the two features that the authors hypothesize produce perceptions of moral decline: (a) a biased exposure effect whereby people see more negative information than positive information about current-day people (e.g., via the media); (b) a biased memory effect whereby people are less likely to have strong negative emotional responses to negative events from the past.

Summary of the methods

The original study (N=387) and our replication (N=533) examined participants’ perceptions of how moral other people were at different points in time. 

Participants from the following age groups were recruited to participate in the study:

• 18–24
• 25–29
• 30–34
• 35–39
• 40–44
• 45–49
• 50–54
• 55–59
• 60–64
• 65–69

After answering a few pre-study questions (see “Study and Results in Detail” section), participants were told, “In this study, we’ll ask you how kind, honest, nice, and good people were at various points in time. If you’re not sure or you weren’t alive at that time, that’s okay, just give your best guess.”

Participants then completed the five primary questions of interest for this study, reporting how “kind, honest, nice, and good” people were at five different timepoints:

• today (“today”)
• around the year the participant turned 20 (“20 years after birth”) 
• around the year the participant was born (“birth year”)
• around 20 years before the participant was born (“20 years before birth”)
• around 40 years before the participant was born (“40 years before birth”)

Going forward, we will use the terms in parentheses as shorthand for each of these timepoints. But please note that the timepoints asked about were approximate—for example, “birth year” is not the exact year each participant was born, but it is within a 5-year range of each participant’s birth year.

Figure 1 shows the versions of the primary questions that a 50-54 year-old participant would receive. Each question was asked on a separate survey page. Participants in other age groups saw the same general questions, but the number of “years ago” in questions 2-5 was adjusted to their age group. Participants aged 18-24 did not receive the second question because today and 20 years after birth were the same period of time for participants in this age group.

After completing the primary questions of interest, participants completed a consistency-check question, attention-check question, and demographic questionnaire (see “Study and Results in Detail” section).

Summary of the primary results

The original paper compared participants’ average ratings of how “kind, honest, nice, and good” people were between each adjacent timepoint. They found that:

• Participants rated people as less kind, honest, nice, and good today vs 20 years after birth.
• Participants rated people as less kind, honest, nice, and good 20 years after birth vs birth year.
• Participants rated people as equivalently kind, honest, nice, and good at birth year vs 20 years before birth.
• There was no statistically significant evidence of either a difference or equivalence between participants’ ratings of how kind, honest, nice, and good people were 20 years before birth vs 40 years before birth. (However, if anything, participants’ ratings were lower at 40 years before birth, which was consistent with the original paper’s hypotheses.)

See “Study and Results in Detail” section for details on the statistical analyses and model results.

When the original authors reviewed our pre-registration prior to replication data being collected, Dr. Mastroianni offered insights about what results they would be more or less surprised by if we found them in our replication data. Because his comments are from prior to the collection of new data, we and the original authors both thought they added useful context to our report:

As for what constitutes a replication, it’s an interesting question. We ran our studies to answer a question rather than to prove a point, so the way I think about this is, “what kinds of results would make me believe the answer to the question is different from the one I believe now?”

• If Contrast 1 was not significant, this would be very surprising, as it would contradict basically every other study in the paper, as well as the hundreds of surveys we review in Study 1.
• If Contrast 2 was not significant, this would be mildly surprising. Contrast 2 is a direct replication of a significant contrast we also saw in Study 2c (as is Contrast 1, for that matter). But this difference was fairly small both times, so it wouldn’t be completely crazy if it didn’t show up sometimes.
• Contrasts 3 and 4 were pretty flat in the original paper. It would be very surprising if those were large effects in the replication. If they’re significant but very small in either direction, it wouldn’t be that surprising.

Basically, it would be very surprising if people perceive moral decline at both points before their birth, but they perceive moral improvement at both points after their birth. That would really make us scratch our heads. It would be surprising in general if there was more decline in Contrasts 3 & 4 than in 1 & 2.

Dr. Adam Mastroianni in email to Transparent Replications team, 2/29/2024.

Summary of replication results

When we analyzed our data, the results of our replication aligned extremely closely with the results of the original study (compare Figure 2 below to Figure 4 in the original paper).

The only minor difference in the statistical results between the original study and our replication was that our replication found statistically significant evidence of equivalence between participants’ ratings of how kind, honest, nice, and good people were at 20 years before birth versus 40 years before birth. As specified in our preregistration, we still consider this a replication of the original results because it is consistent with the paper’s hypothesis (and subsequent claims) that perceptions of moral decline diminish, disappear, or reverse if people rate time periods before they were born.

Here is a summary of the findings in the original study compared to the replication study:

Morality ratings in original study	Morality ratings in replication study	Replicated?
today < 20 years after birth	today < 20 years after birth
20 years after birth < birth year	20 years after birth < birth year
birth year = 20 years before birth	birth year = 20 years before birth
20 years before birth ? 40 years before birth	20 years before birth = 40 years before birth

Study and Results in Detail

Methods in detail

Preliminary survey questions

Before completing the primary questions of interest in the survey, participants indicated which of the following age groups they belonged to:

• 18–24
• 25–29
• 30–34
• 35–39
• 40–44
• 45–49
• 50–54
• 55–59
• 60–64
• 65–69
• 70+

Participants who selected 70+ were screened out from completing the full survey. The original study recruited nearly equal numbers of participants for each of the other 10 age groups. Our replication attempted to do the same, but did not perfectly recruit equal numbers from each age group (see Appendix for more information). 

Participants also completed three questions that, according to the original paper, were designed to test “English proficiency and knowledge of US American culture”:

Which of the following are not a type of footwear?

• Sneakers 
• Slippers
• Flip-flops 
• High heels 
• Bell bottoms 

Which of the following would be most likely to require an RSVP?

• A wedding invitation 
• A restaurant bill 
• A diploma
• A thank-you note 
• A diary 

Which of the following would be most likely to have a sign that says “out of order”?

• An elevator 
• A person 
• A pizza 
• A book
• An umbrella

Consistency check

After completing the five primary questions of interest described in the “Summary of Study and Results” section above, participants answered the following consistency check question: 

Please choose the option below that best represents your opinion: 

• People are MORE kind, honest, nice, and good today compared to about [X] years ago
• People are LESS kind, honest, nice, and good today compared to about [X] years ago
• People are equally kind, honest, nice, and good today compared to about [X] years ago

“[X]” took on the same value as the final timepoint—around 40 years before the participant was born. This question was designed to ensure that participants were providing consistent ratings in the survey. 

Demographics and attention check

After completing the consistency check question, participants reported their age, gender, race/ethnicity, household income, educational attainment, parental status, and political ideology. 

Embedded among these demographic questions was the following attention-check question:

Some people are extroverted, and some people are introverted. Please select the option “other” and type in the word “apple”.

• Extroverted
• Introverted
• Neither extroverted nor introverted
• Other _______ 

Exclusion criteria

Participants’ responses were excluded from the data if any of the following applied:

• They did not complete the study
• They reported being in the 70+ age group
• They failed any of the three English proficiency questions
• They failed the attention check question
• Their answer to the consistency check question was inconsistent with their ratings for today and 40 years before birth
• Their reported age in the demographics section was inconsistent with the age group they selected at the beginning of the study

Of the 721 participants who took the survey, 533 passed all exclusion criteria and were thus included in our analyses. 

Primary analyses: detailed results

As pre-registered, we ran the same statistical analyses as the original paper. 

To analyze the primary questions of interest, we ran a linear mixed effects model, with random intercepts for participants, testing whether participants’ morality ratings differed by timepoint (using the lmer package in R).

We then tested four specific contrasts between the five timepoints using a Holm-Bonferroni correction for multiple comparisons (using the emmeans package in R):

• today vs 20 years after birth
• 20 years after birth vs birth year
• birth year vs 20 years before birth
• 20 years before birth vs 40 years before birth 

Here are the results of these contrasts:

There were statistically significant differences between today and 20 years after birth and between 20 years after birth and birth year, but not between birth year and 20 years before birth or between 20 years before birth and 40 years before birth—the same pattern as the original study results. 

Next, we conducted equivalence tests (using the parameters package in R), for the two comparisons that were not statistically significant. Here are the results:

These tests found that, for both contrasts, 100% of the confidence interval range was inside the region of practical equivalence (ROPE). (See the Appendix for a brief discussion on how the ROPE was determined.) Thus, there was statistically significant evidence that birth year and 20 years before birth were equivalent and that 20 years before birth and 40 years before birth were equivalent. (You can read about how to interpret equivalence test results from the parameters package here.) 

In the original study, birth year and 20 years before birth were found to be equivalent, but there was not statistically significant evidence for equivalence between 20 years before birth and 40 years before birth. As mentioned earlier, we consider equivalence between 20 years before birth and 40 years before birth to be a successful replication of the original study’s findings because it is in line with the claims in the paper that perceptions of moral decline diminish, disappear, or reverse when people are asked about time periods before they were born.

Secondary analyses

As in the original paper, we also tested for relationships between participants’ morality ratings and various demographic variables. Since this analysis was not central to the paper’s claims, we preregistered that these results would not count towards the replicability rating for this paper. 

Following the analytical approach in the original paper, we ran a linear regression predicting the difference in participants’ morality ratings between today and birth year by all of the following demographic variables:

• Age 
• Political ideology
• Parental status
• Gender
• Race/ethnicity 
• Educational attainment

Here are the statistical results from this analysis:

We explain the results for each demographic variable below: 

Age

The original study found a statistically significant effect of age such that older people perceived more moral decline (i.e., a larger negative difference between today and birth year morality ratings). However, the original paper argued that this was because the number of years between today and birth year was larger for older participants. 

Our replication did not find a statistically significant effect of age. 

Political ideology

Participants could choose any of the following options for political ideology:

• Very liberal
• Somewhat liberal
• Neither liberal nor conservative
• Somewhat conservative
• Very conservative

We converted this to a numeric variable ranging from -2 (very liberal) to 2 (very conservative). 

The original study found a statistically significant effect of political ideology such that more conservative participants perceived more moral decline. Our replication found the same result. 

Following the original study, we ran a one-sample t-test to determine whether participants who identified as “very liberal” or “somewhat liberal” still perceived moral decline, on average. These participants had an average score of less than zero (mean difference = -0.76, t(295) = -9.6252, p < 2.2e-16), meaning that they did, on average, perceive moral decline. 

Parental status

Participants reported how many children they had. We converted this into a binary variable representing whether or not each participant is a parent. 

The original study did not find a statistically significant effect of parental status. However, our replication found a significant effect such that parents perceived more moral decline than non-parents.

Gender

Participants could choose any of the following options for gender:

• Male
• Female
• Other

The original study did not find a statistically significant effect of gender. Our replication, on the other hand, found a significant effect of gender such that participants who selected “Other” did not perceive moral decline, on average. However, we do not recommend giving much credence to this statistical difference because only 3 out of the 533 participants selected “Other.” We think conclusions should not be drawn in either direction with such a small sample size for that category.

Race/ethnicity

Participants could choose any of the following options for race/ethnicity:

• American Indian or Alaska Native
• Asian
• Black or African-American
• Hispanic or Latino Origin
• Hawaiian or Pacific Islander
• White
• Other
• More than 1 of the above

Neither the original study nor our replication found a statistically significant effect of race/ethnicity when the variable is dummy coded with White as the comparison group.  

Education

Participants could choose any of the following options for education:

• Did not complete high school
• High school diploma
• Some college
• Associate’s degree
• Four-year college degree
• Some graduate school
• Graduate school

We converted this to a numeric variable ranging from 0 (did not complete high school) to 6 (graduate school).

Neither the original study nor our replication found a statistically significant effect of education.

Interpreting the Results

All of the primary original-study results replicated in the data we collected, according to the replication criteria we pre-registered. 

It is worth highlighting that there was one minor statistical discrepancy between the primary results for the two datasets. The original study did not find statistical evidence for either a difference or equivalence between 20 years before birth and 40 years before birth. Our replication also found no statistical evidence for a difference between these timepoints, but it did find evidence for equivalence between the timepoints. We specified in advance that this pattern of results would qualify as a successful replication because it supports the original paper’s hypothesis that perceptions of moral decline diminish, disappear, or reverse when people are asked about time periods before they were born.

Among the secondary analyses, which tested the relationship between perceptions of moral decline and various demographic factors, our replication results differed from the original study results for a few variables. The original study found that only political ideology and age were statistically significant predictors of participants’ perceptions of moral decline. Our replication found similar results for political ideology, but it did not find age to be a significant predictor. Additionally, our replication found parental status and gender to be significant predictors. However, we strongly caution against interpreting the gender result strongly. This result was driven by the fact that the gender response option “Other” had a substantially different average moral decline rating from the response options “Male” and “Female,” but only 3 out of 533 participants comprised the “Other” category (see Figure 5). We consider this too small of a subgroup sample size to draw meaningful conclusions from. As we pre-registered, the secondary analyses were not considered in our replication ratings because they were not central to the paper’s hypotheses and the authors did not strongly interpret or theorize about the demographic-level findings.  

Finally, the paper was careful to note that its findings are not direct evidence for the biased exposure and biased memory effects that it postulates as causes of the perception of moral decline:

“The illusion of moral decline is a robust phenomenon that surely has several causes, and no one can say which of them produced the illusion that our studies have documented. Studies 5a and 5b do not directly implicate the BEAM mechanism in that production but they do make it a viable candidate for future research.” (p. 787)

We would like to reiterate this interpretation: the observed result is what one would expect if the biased exposure effect and biased memory effect gave rise to perceptions of moral decline, but this study does not provide causal evidence for either of these mechanisms.  

Conclusion

Overall, we successfully replicated all of the primary findings from the original study. Collectively, these findings suggest that people in the U.S. (aged 18-69), on average, perceive moral decline for time periods during their lifetimes, but not for time periods before they were born. The study received 5 stars for replicability.  

All of the study’s data, materials, and analysis code were publicly available and well-documented, which made this replication straightforward to conduct. We also successfully reproduced the results in the original paper using the provided data and analysis code. The one area for improvement on the transparency front is preregistration: this study was not pre-registered, even though it was very similar to a previous study in this paper (Study 2c). The study received 4.25 stars for transparency.

Generally, the study’s analyses were appropriate and its claims were well-calibrated to its study design and results. The study received 5 stars for clarity.

Acknowledgements

We want to thank the authors of the original paper for making their data, analysis code, and materials publicly available, and for their quick and helpful correspondence throughout the replication process. Any errors or issues that may remain in this replication effort are the responsibility of the Transparent Replications team.

We also owe a big thank you to our 533 research participants who made this study possible.

Finally, I am extremely grateful to Amanda Metskas and the rest of the Transparent Replications team for their advice and guidance throughout the project. 

Author Response

The authors of the original study shared the following response to this report:

“We are pleased to see these effects replicate, and we are grateful to the Transparent Replications team for their work.”

Dr. Adam Mastroianni via email 7/5/2024

Purpose of Transparent Replications by Clearer Thinking

Transparent Replications conducts replications and evaluates the transparency of randomly-selected, recently-published psychology papers in prestigious journals, with the overall aim of rewarding best practices and shifting incentives in social science toward more replicable research.

We welcome reader feedback on this report, and input on this project overall.

Appendices

Additional Information about the Methods

Recruitment

Both the original study and our replication recruited a sample of participants stratified by age. However, the original study and our replication used slightly different methods for doing so, which resulted in small differences in age-group proportions between the two studies. 

In the original study, participants were first asked to report their age. A quota system was set up inside the survey software such that, in theory, only 50 participants from each of the following age group should be allowed to participate: 18–24, 25–29, 30–34, 35–39, 40–44, 45–49, 50–54, 55–59, 60–64, 65–69. If participants indicated that they were 70 or older or if they were not among the first 50 participants from a given age group to take the study, they were not allowed to participate in the study (the original study did not have a perfect split by age, but it was quite close to 50 per group; see the table below). After completing the age question, participants completed the three English proficiency and knowledge of US American culture questions. If they failed any of the proficiency questions, they were not allowed to participate in the study.

In order to ensure that all participants were paid for the time they spent on the study, we did not use the same pre-screening process used in the original study. In the original study, if the age quota for a participant’s age group was already reached, or if a participant didn’t pass the screening questions, they were not paid for the initial screening questions they completed. In order to avoid asking participants to answer questions for which they wouldn’t be paid, we used age quotas within Positly to recruit participants in approximately equal proportions for each age group. Participants still indicated their age in the first part of the survey, but they were no longer screened out by a built-in age quota. This process did not achieve perfectly equal recruitment numbers by age group. We expect that this is because some participants reported an age in our experiment that differed from their listed age in the recruitment platform’s records. This could be for a variety of reasons including that some members of a household might share an account. 

Although our recruitment strategy did not achieve perfect stratification by age group, the two studies had relatively similar age-group breakdowns. The table below shows the pre-exclusion and post-exclusion stratification by age group for both studies.

We also want to note a minor deviation from our pre-registered recruitment strategy. In our pre-registration we said: 

“We will have 600 participants complete the study. If we do not have 520 or more participants remaining after we apply the exclusion criteria, then we will collect additional participants in batches of 20 until we reach 520 post-exclusion participants. We will not conduct any analyses until data collection is complete. When collecting data, we will apply age-group quotas by collecting 60 participants from each of the following ten age groups: 18–24, 25–29, 30–34, 35–39, 40–44, 45–49, 50–54, 55–59, 60–64, 65–69. If we need to recruit additional participants, we will apply the age-group quotas in such a way as to seek balance between the age groups.”

Because recruiting participants from the youngest age group (18-24) and the oldest age group (65-69) turned out to be extremely slow, we decided not to “apply the age-group quotas in such a way as to seek balance between the age groups” when we recruited participants beyond the original 600. (Note: We did not look at the dependent variables in the data until we had fully finished data collection, so this small deviation from the preregistration was not influenced by the data itself.) 

It’s also worth noting that the total number of participants we recruited was not a multiple of 20 despite our stated recruitment approach. This was because each time one collects data from an online crowdsourcing platform like Positly it’s possible that a few additional participants will complete the study than the original recruitment target. For example, sometimes participants complete the study in the survey software but do not indicate to the crowdsourcing platform that they completed the study. Because we had many rounds of recruitment for this study, each round had the opportunity to collect slightly more participants than the targeted number. 

We also want to note one change we made in how subjects were recruited during our data collection. In the early portion of our data collection the recruited subjects first completed a pre-screener that asked the three English proficiency and knowledge of US American culture questions and confirmed that they were within the eligible age range for the study. All participants were paid for the pre-screener, and those who passed it were invited to continue on to take the main study. 146 participants passed the pre-screener and went on to take the main study.

We found that the pre-screening process was slowing down recruitment, so we incorporated the screening questions into the main study and allowed recruited participants to complete and be paid for the study even if they failed the screening. We excluded participants who failed the screening from our data analysis. 575 participants took the study after this modification was made.

Finally, it’s important to note that our pre-exclusion sample size of n=721 is the number of participants who provided consent to participate in our study; the number of participants in our replication who passed the screening criteria of being between ages 18-69 and correctly answering the three English proficiency and knowledge of US American culture questions was n=703. 

Additional Information about the Results

Corrections for multiple comparisons

For the primary analysis in which participants’ morality ratings are compared between timepoints, we followed the analytical approach used in the original paper and used a Holm-Bonferroni correction for multiple comparisons for the four contrasts that were tested. However, we think it is unnecessary to correct for multiple comparisons in this situation. As argued by Rubin (2024), multiple comparisons would only be necessary in this context if the authors would have considered their hypothesis confirmed if at least one of the contrasts returned the hypothesized result. Rather, the authors needed each of the four contrasts to match their expected pattern in order to confirm their hypothesis. As such, we argue that correcting for multiple comparisons is overly conservative in this study. However, not correcting for multiple comparisons on our replication data does not change the statistical significance of any of the findings.

Region of practical equivalence (ROPE) for equivalence tests 

It’s important to note that when conducting equivalence tests, evidence for equivalence depends on what one sets as the region of practical equivalence (ROPE). The original authors chose to use the default calculation of ROPE in the parameters package in R (see here for more information). Given that the original study was not pre-registered, we think this is a reasonable decision; after knowing the study results, it could be difficult to justify a particular ROPE without being biased by how this would affect the findings. To make our results comparable to the original study, we also used the default calculation of ROPE. However, we want to note that this is not a theoretical justification for the specific ROPE used in this study; other researchers might reasonably argue for a wider or narrower ROPE. 

References

Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149-1160. Download PDF

Mastrioanni, A. M., & Gilbert, D. T. (2023). The illusion of moral decline. Nature, 618, 782–789. https://doi.org/10.1038/s41586-023-06137-x 

Rubin, M. (2024). Inconsistent multiple testing corrections: The fallacy of using family-based error rates to make inferences about individual hypotheses. Methods in Psychology, 10, 100140. https://doi.org/10.1016/j.metip.2024.100140