Figure 4 shows the results of an uncorrected chi-squared test and Fisher’s exact test, both two-sided and one-sided, for 10,000 independent comparisons of two samples of size 10 with a binary outcome variable with probability 0.5 of being 0 and probability 0.5 of being 1. The uncorrected chi-squared test and Fisher’s exact tests were carried out. For small samples like two groups of size 10 a chi-squared test would not be valid, because most such tables will have an expected frequency less than 5.0. Fisher’s exact test might be preferable. However, both clearly show a distribution very far from Uniform, with only a few P-values being represented.
