META-REP: A Meta-scientific Programme to Analyse and Optimise Replicability in the Behavioural, Social, and Cognitive Sciences (SPP 2317)

Links and Functions

Breadcrumb Navigation


Replicating Failed Replications: A Validity-based Framework for Predicting Replication Success

The inconsistency of a statistically significant result in an original study and a non-significant result in its replication study can be due to problems with statistical conclusion validity, construct validity, internal validity, and external validity in the original and replication study. While problems with statistical conclusion validity in original studies have been the main focus in the replication debate (e.g., inflated alpha-errors due to questionable research practices, inflated effect sizes due to under-powered studies in combination with publication bias), problems concerning the other types of validity have been largely neglected until recently.

In the project, we will develop a model from machine learning to predict replication success based on potentially non-linear relations of properties of original and replication studies as input variables. The first class of input variables consists of indicators of all types of validity coded for original and replication studies in an extensive literature review. The second class consists of assessments from the experts of those studies: We will ask authors of original and replication studies to rate to which extent their studies achieved the different types of validities. In addition, we will ask authors with a “failed” replication to indicate reasons for non-replication in an open question format. We will use the model to identify replication studies that “failed” although the predicted replication success is high. As a concept-of-proof for the machine learning model and to demonstrate our model-driven approach, we will select and replicate a small sample of those failed replications using a validity-optimized study design. The coding of original and replication studies as well as the assessments from the authors of these studies will be made available in a standardized database for researchers interested in meta-science on the (non-)replicability of research results in psychology.

Further Information

Our Team