Maximizing the Power of Reaction Time Research: A Congruency Sequence Effect Simulation Study

Abstract

Cognitive control is critical for goal-directed behavior, in contrast to otherwise automatic actions. The Stroop task, where participants identify font color while ignoring the word meaning, exemplifies this by inducing cognitive conflict and creating the Stroop effect. The Congruency Sequence Effect (CSE) demonstrates cognitive control by showing smaller Stroop effects following incongruent trials compared to congruent ones. Despite the consensus on CSE’s existence, its effect size is small, necessitating high statistical power for detection. However, inconsistencies in data analysis practices across studies complicate replicability. This study examines how arbitrary analytical decisions impact CSE detection. We reviewed prevalent preprocessing and processing steps in CSE studies, distinguishing between theory-backed and arbitrary decisions. We then employed a multiverse analysis on simulated datasets, reflecting different sample sizes, effect sizes, outlier exclusion criteria, and statistical methods, to assess their influence on CSE detection. We have found that different preprocessing and analytical decisions result in different true positive rates on the same sample and effect sizes, thus careful consideration and transparent reporting of analytical decisions is key to report reliable data. The simulation pipeline reported in this study provides an easy-to-adapt method to pre-analyze statistical power of specific reaction time effects.