Robustness and Replicability in Economics (R2E)
In economics, most empirical work is based on non-laboratory experimental and quasi-experimental designs. While in other more standardized disciplines, replicability mostly is a matter of generalizability across contexts, in economics it also hinges on the robustness of findings across different specifications. This project will provide replicability rates in economics by conducting computational and robustness replications of 30 non-laboratory studies in economics using the original data set. These studies cover different empirical methods such as primary data-based Randomized Controlled Trials (RCTs) and secondary data-based quasi-experimental methods. Complementing these robustness replications, we will also run surveys with experts in the field to elicit how they assess replication success or failure and to obtain assessments of generalizability across contexts – a frequently discussed limitation of RCTs in particular.
The main objectives are, first, to define replication success for robustness replications. We will also develop standardized protocols on how to conduct robustness replications and standardized forms for reporting the results. Second, we will compare replicability rates across methods and evaluate differences. The prior of experts usually is that secondary data-based studies are more prone to p-hacking, HARKing, and publication bias than RCTs. Third, we will develop an expert interview-based toolkit to assess robustness replicability and generalizability across contexts.