Requiring that replication studies use the same phenotype definition used in the initial study also helps avoid false positives due to “data dredging,” the temptation to generate small p-values by testing many different traits (different case subtypes, continuous traits dichotomize using different, arbitrary cut points, etc.). [4] When many phenotypes or phenotype definitions and analyses are used, there should be a penalty for multiple testing. Applying this penalty is not always straightforward, given that most of the phenotypes and analyses are usually correlated or even highly correlated. However, the danger exists for an association to be claimed replicated, after searching through repeated modifications of the phenotypes and analyses thereof. A p-value that has been obtained through such an iterative searching path is not the same as one that was obtained from a single main analysis of a single phenotype.
