There are several tests and metrics of between-dataset heterogeneity, borrowed from applications of meta-analysis in other fields. The most popular are Cochran’s Q test of homogeneity, [49] the I2 metric (obtained by (Q-degrees of freedom)/Q), and the between-study variance estimator τ2. [50] There are shortcomings to all of them. [51] The Q test is underpowered in the common situation where there are few datasets and may be overpowered when there are many, large datasets. There are now readily-available approaches that can be used to compute the power of the Q test to detect a given tau-squared. [52] When the Q test is underpowered, the I2 metric has large uncertainty and this can be readily visualized by computing its 95% confidence intervals. [53] Similarly, estimates of τ2 may have large uncertainty. One potentially useful approach may be to estimate the magnitude of between-study variability compared with the observed effect size θ, i.e. h=τ/θ. For a small effect size, even small τ2 may question the generalizability of the conclusion that there is an association across all datasets. This conclusion would not be as easily challenged in the presence of a large effect size.
