expected to be close to 5%. Note that the standard error of the ML estimator of the p-value in the simulations is calculated as sqrt(p * (1 − p)/N), where p denotes the percentage of significant tests observed in the simulations (nominal p-value) given a chosen α, and N the total number of simulations. The 95% confidence interval for a correct nominal p-value of .05 (given α = .05) and N = 2000 thus corresponds to CI-95 = (p − 1.96 * SE, p + 1.96 * SE), and thus equals .04–.06. This implies that, given α = .05, any observed nominal p-value outside the .04–.06 range should be considered incorrect: p-values <.04 suggest that the model is too conservative, while p-values >.06 suggest that the model is too liberal.
