The Role of Measurement in the Replicability of Empirical Findings
The goal of this project is to investigate the role of measurement in the replicability of empirical findings. We will focus on two widespread, potentially problematic measurement practices: 1) the use of ad hoc scales and 2) the use of modified scales. Ad hoc scales are scales that authors constructed for a particular study and that have not (or only very superficially) been validated. The term “modified scales” refers to deviations from preexisting, validated scales for example in terms of the number of items, the items’ wording, or the response format. We will investigate how these practices influence replication rates and the heterogeneity of effect sizes using three complementary methodological approaches: by reanalyzing existing empirical data, running an experiment, and conducting a simulation study.
In the first study, we will reanalyze data from replication projects that applied item-based measures such as the Many Labs projects. In the second study, we will conduct a multisample replication project on multiple original effects from different fields that is combined with an experiment on the influence of measurement on replicability and effect size heterogeneity. The third study will consist of a simulation study in which the “original” scale will be compared to ad hoc scales and different types of modifications of the original scale in terms of the recovery of the true effect sizes. Our application context is multidisciplinary and includes several areas within psychology (social psychology, health psychology) as well as the political sciences and economics, allowing us to make comparisons across fields. In sum, this project will contribute to the META-REP priority program and the metascientific literature by investigating measurement as a factor that can explain the replicability of empirical findings.