The Role of Theory in Understanding and Resolving the Reliability Crisis
The reliability of a given empirical test is a product of the quality of the theory used to design that test. Too many unknowns or competing causal pathways renders test results unreliable or even uninterpretable. If hypothesis tests are repeated under theoretical ambiguity, an entire area of study may be unreliable. Such a scenario could explain a lack of consensus and failures to reproduce findings in that area. In the behavioral and social sciences especially, theoretical ambiguity is acute due to the complexities of human interaction and societal organization. Theoretical inquiry to improve a single hypothesis test is time consuming, and alone will not improve reliability across an entire area of study. Therefore, I propose an approach to maximize theory while minimizing the time investments of scholars working in an area.
This project will test the extent of theoretical ambiguity in one area of study, as a cause of empirical unreliability in that area. Then it will check if computer-assisted comparison of causal models can efficiently identify theoretical ambiguities in that area. Next, it tests if crowdsourcing theoretical claims combined with computer-assisted causal model comparison can improve reliability and the replicability of findings in that area. Finally, if initial results are positive, then it will develop computer systems to improve and economize the process for deployment across all areas of cognitive, behavioral and social sciences. It should ideally contribute to knowledge on why replication and reliability varies across studies and hypotheses, and how to improve them via technology and meta-theoretical work.