Student Samples Vary Demographically but Produce Similar Experimental Effects

🔍 What Was Compared

Three student subject pools at distinct universities completed the same survey experiment about crisis bargaining between states to test whether convenience samples of college students are comparable across institutions.

🛠️ How the Study Was Run

The experimental protocol was replicated across the three sites and analyzed for three sets of outcomes:

• demographic composition of each subject pool (drawing attention to selection through university matriculation and volunteer recruitment into experiments),
• baseline attitudes and thinking about international politics, and
• responses to experimental treatments, with estimation of conditional average treatment effects across subgroups.

📊 Key Findings

• Substantial demographic differences exist across the three student subject pools, consistent with selection effects tied to who attends which university and who volunteers for lab or survey experiments.
• Despite these demographic differences, subjects from different institutions responded similarly to most experimental treatments.
• A clear exception occurred when treatments provided information about a state’s regime type: responses diverged across subject pools in this case.
• There is little evidence that the observed demographic differences meaningfully change conditional average treatment effects across subgroups.

🤔 Why This Matters

These results indicate that while college subject pools differ demographically across institutions, such differences do not generally undermine the external validity of treatment effects in this crisis-bargaining survey experiment. Caution is warranted for treatments that cue regime type, and these findings carry important implications for the use of student samples in political science—especially within international relations—and for discussions about replication across sites.