🔎 Problem Observed:
There is growing interest in experiments where outcomes are measured by surveys while treatments are delivered separately (e.g., mailers, door‑to‑door canvasses, phone calls, or online ads). Common designs for these studies are often prohibitively expensive, vulnerable to bias, and raise ethical concerns.
⚙️ Four Design Practices That Work Together:
- Online surveys recruited from a defined sampling frame
- At least one baseline wave of surveying prior to treatment
- Multiple survey items combined into an index to measure outcomes
- A placebo control when feasible
These four practices have previously undocumented complementarities: when at least two are used in combination, they can dramatically relax cost, bias, and ethical constraints.
🧠A Practical Framework for Design Choices:
A general and extensible framework is provided to help researchers determine the most efficient mix of these practices for diverse applications. The framework makes tradeoffs explicit and identifies when combinations of practices yield the biggest gains in precision, robustness, and ethical soundness.
đź“Š Empirical Tests of the Practices:
- Representativeness Check: Online panel respondents recruited from a defined sampling frame were compared to phone panel respondents; representativeness of the online panel compares favorably to the phone panel.
- Field Implementation: An original door‑to‑door canvassing experiment successfully implemented all four practices, demonstrating feasibility in a real‑world application.
đź“Ś Key Findings:
- Combining at least two of the recommended practices produces large improvements in efficiency, bias reduction, and ethical acceptability.
- Defined sampling‑frame online panels can be as representative as phone panels for survey outcomes.
- The full four‑practice design is implementable in a field canvassing context.
đź“Ł Why It Matters:
This work offers concrete, actionable design guidance for survey‑outcome field experiments that improves cost‑effectiveness, statistical robustness, and ethical standards. The framework also points to potential extensions for other treatment delivery modes and outcome measures.
