A large-scale collaborative assessment of the replicability and robustness of EEG research
EEG is widely used to investigate human cognition and other psychological phenomena. Yet, despite its popularity, the credibility of several EEG findings has recently been debated. This new skepticism is based on the observation that novel hypotheses are oftentimes tested only in small samples, while replication studies are usually deemed unattractive. Moreover, there is a great deal of flexibility in any EEG analysis, such that analysis pipelines are highly variable across studies. We contend that without assessing the replicability and robustness of EEG research by assessing its results using new data and alternative analyses, we are potentially building a house of cards. Inspired by the lessons emerging from the replication crisis in the psychological sciences, we have a unique opportunity to create a stronger foundation for EEG research. In this proposal, we present two large-scale, international collaborative projects addressing the replicability and robustness of EEG research, respectively,
The #EEGManyLabs project is mobilizing an international network of researchers to replicate the most influential published EEG studies, representing the largest neuroscience replication project undertaken to date. This project will provide an enormous body of data that will be made publicly available and will establish a library of effect sizes for the most commonly reported EEG phenomena. Thus, this project will help assessing the replicability of previous studies and designing future studies.
The #EEGManyPipelines project is an international many-analyst project, in which all participating researchers are provided with the same dataset and are instructed to analyze the data with an analysis pipeline they deem sensible and representative of their own research. Analysts will then report their results and a detailed description of the analysis pipeline, allowing us to analyze the diversity of analysis pipelines and their effects on results. Thus, this project will help assessing the robustness of EEG findings across alternative analyses, identifying (sub)optimal analysis pipelines, and informing guidelines for reporting EEG analyses in publications.
We expect that this project will help improving the credibility of EEG findings and the quality of analyses, and will inspire new standards for conducting and reporting EEG studies.