🔎 How Publication Decisions Were Modeled and Tested

A formal model of the publication process is developed that captures the interaction between an initial study and a subsequent replication. The model identifies how editorial and authorial choices about what to submit and what to accept shape published evidence across these two-stage research sequences.

📋 Three Biases the Model Highlights

• File-drawer bias: statistically significant original results are favored over nonsignificant ones.
• Repeat-study bias: a reluctance to publish replications relative to original studies.
• Gotcha bias: replications that contradict earlier positive findings are more likely to be published than confirming replications.

🔬 How These Biases Were Measured

• Parameters of the model were estimated using a vignette experiment administered to political science professors teaching at Ph.D.-granting institutions in the United States.
• The vignette experiment elicited judgments about publication likelihood for combinations of initial and replication study outcomes, allowing estimation of bias magnitudes in a controlled yet realistic setting.

📌 Key Findings

• Evidence was found for all three types of publication bias identified by the model.
• Biases that specifically involve replication studies (repeat-study bias and gotcha bias) are present but notably smaller in magnitude than the classic file-drawer bias.
• When all biases operate together, their aggregation increases the prevalence of false positive findings in the published literature.

⚠️ Why This Matters

• The presence of replication-specific biases, even if smaller, means the growing replication movement may still face distortions in what gets published.
• The combined effect of multiple biases undermines the reliability of published results and raises the probability that false positives persist in scholarly debates.

➡️ Next Steps Suggested

• Further empirical work to quantify these biases across disciplines and publication stages.
• Investigation of editorial and incentive reforms to reduce both classic file-drawer effects and replication-related biases.