METEOR – MastEring ThE OppRessive number of forking paths unfolded by noisy and complex neural data
There is a replication crisis and a “real-world or the lab” dilemma in psychology and cognitive neuroscience. Solving the dilemma and overcoming the crisis at the same time is arguably a serious challenge. One of the main aims in cognitive neuroscience is to discover brain-cognition associations which are replicable across laboratories. A precondition for replicability of individual differences findings in terms of brain-cognition associations obtained inside or outside of the laboratory is rank order stability of neural parameters derived from noisy and complex signal recordings. However, to date we do not know well enough how much hitherto unsuccessful replications are due to the oppressive number of methodological decisions researchers have to make á priori to testing a brain-cognition association. Moreover, we do not yet have standards with respect to the unit of analysis at which replications should be considered successful. We also lack a knowledge app containing a systematic and exhaustive overview of potential methodological choices that are defensible in a typical individual differences analysis workflow for mobile EEG or fMRI, as well as multivariate behavioral data. Thus, laboratories still stick to their customized choices which are sometimes passed over through many generations of young scientist. But – as described in the very recent literature – variability of workflows and of associated substantial findings is huge across laboratories. Finally, hitherto proposed statistical approaches for analyzing the multiverse of potentially constructed datasets for noisy and highly complex multidimensional neural data need extensions through tools available for big data analysis. Such approaches would allow learning about influential decisions and would predict potential heterogeneity of future findings.
To take a large step toward filling these gaps, METEOR aims to bring together a larger group of scientists with different and complementary expertise (cognitive neuroscientists using mobile EEG methodology, network neuroscientists working with fMRI data and statisticians experienced with big data analyses tools). By joining forces and a fruitful environment of a collaborative research programme, METEOR will provide standards on a replication success definition for cognitive neuroscience applicable across neuroimaging modalities. Furthermore, it will deliver systematized knowledge and analytic solutions for the multiverse in two neuroimaging modalities – mobile EEG and resting state fMRI – applied to the realm of assessiong individual differences and brain cognition associations. Proposed solutions will be discussed with respect to their applicability to further research questions in the future.