In addition to this, all statistical tests and variable selection schemes based on the original permutation importance, such as those suggested by Diaz-Uriarte and Alvarez de Andrés (2006) and Rodenburg, Heidema, Boer, Bovee-Oudenhoven, Feskens, Mariman, and Keijer (2008), suffer from another artifact, that is induced by the way the permutation importance is constructed: the artificial preference of correlated predictor variables. In a permutation test framework Strobl et al. (2008) show that only a conditional permutation scheme reflects the desired null hypothesis, and the resulting conditional importance describes the actual effect of a variable in the presence of correlations more reliably.
