The reproducibility and robustness of secondary analyses in educational research: The role of publication bias and researcher degress of freedom
The subproject at the IQB's Research Data Centre (FDZ, Forschungsdatenzentrum), led by Dr. Malte Jansen and Dr. Aleksander Kocaj, addresses questions concerning the reproducibility and robustness of research results based on secondary analyses in empirical educational research (META-REP-IQB; Funding period: 02/2022-01/2025, total funding: 231.578€). The project is based on applications for secondary data analyses at the FDZ at IQB. In these applications, researchers describe their central questions, hypotheses and their planned analytic approach. We will systematically compare these applications with publications resulting from the applications. In addition, we aim to reproduce the results for selected publications.
The project at the FDZ of the IQB has four goals. First, we examine whether data applications that result in significant or hypothesis-confirming results are also more likely to be submitted for publication and published (publication bias). Second, we plan to develop an index that quantifies the similarity between data use applications and resulting publications to provide evidence of selective reporting. Third, we will test whether the results of selected applications that resulted from data usage applications be reproduced using the information provided in the manuscripts. This will take advantage of the fact that the underlying data are available at the FDZ of the IQB. Based on this, we will investigate how robust selected research results are with regard to alternative plausible analytic approaches (multiverse analysis) in the fourth step. Overall, the project aims at identifying sources of heterogeneity that may influence the replicability and reproducibility of research findings in empirical educational research.