📊 What Was Reanalyzed
A large replication and reanalysis of 49 articles from leading journals that used two-way fixed effects (TWFE) models on observational panel data with binary treatments. The project responds to recent methodological concerns about heterogeneous treatment effects (HTE) and violations of the parallel trends (PT) assumption, alongside a proliferation of new estimators and procedures that have confused best practices.
🔍 How the Reanalysis Was Done
The reanalysis combined replication with a battery of HTE-aware approaches and diagnostics:
- Applied six HTE-robust estimators to each original study's data and design
- Ran diagnostic tests aimed at detecting PT violations
- Conducted sensitivity analyses to assess robustness of causal claims
📈 Key Findings
- HTE-robust estimators typically produce qualitatively similar conclusions to original TWFE results, but estimates vary substantially across estimators, sometimes changing apparent magnitudes and precision.
- A minority of reanalyzed studies show clear signs of PT violations; however, many studies do not present evidence that the PT assumption holds.
- When HTE and potential PT violations are accounted for, a substantial number of studies appear underpowered to deliver reliable causal inferences.
⚖️ What This Means for Research Practice
These results underscore the need for stronger research designs and routine validation of identifying assumptions. Specifically, researchers should:
- Pre-register and justify the choice of estimator given plausible HTE
- Report diagnostic checks for parallel trends and sensitivity to alternative estimators
- Assess statistical power under HTE-aware models before making strong causal claims
💡 Why It Matters
TWFE remains widely used, but reliance on it without HTE-robust checks, PT diagnostics, and power assessments can produce unstable or weakly supported causal conclusions. Rigorous validation and more careful design choices will improve the credibility of panel-based causal inference.
