Bernau, ChristophRiester, MarkusBoulesteix, Anne-LaureParmigiani, GiovanniHuttenhower, CurtisWaldron, LeviTrippa, Lorenzo2014-07-072014Bernau, Christoph, Markus Riester, Anne-Laure Boulesteix, Giovanni Parmigiani, Curtis Huttenhower, Levi Waldron, and Lorenzo Trippa. 2014. “Cross-study validation for the assessment of prediction algorithms.” Bioinformatics 30 (12): i105-i112. doi:10.1093/bioinformatics/btu279. http://dx.doi.org/10.1093/bioinformatics/btu279.1367-4803http://nrs.harvard.edu/urn-3:HUL.InstRepos:12406610Motivation: Numerous competing algorithms for prediction in high-dimensional settings have been developed in the statistical and machine-learning literature. Learning algorithms and the prediction models they generate are typically evaluated on the basis of cross-validation error estimates in a few exemplary datasets. However, in most applications, the ultimate goal of prediction modeling is to provide accurate predictions for independent samples obtained in different settings. Cross-validation within exemplary datasets may not adequately reflect performance in the broader application context. Methods: We develop and implement a systematic approach to ‘cross-study validation’, to replace or supplement conventional cross-validation when evaluating high-dimensional prediction models in independent datasets. We illustrate it via simulations and in a collection of eight estrogen-receptor positive breast cancer microarray gene-expression datasets, where the objective is predicting distant metastasis-free survival (DMFS). We computed the C-index for all pairwise combinations of training and validation datasets. We evaluate several alternatives for summarizing the pairwise validation statistics, and compare these to conventional cross-validation. Results: Our data-driven simulations and our application to survival prediction with eight breast cancer microarray datasets, suggest that standard cross-validation produces inflated discrimination accuracy for all algorithms considered, when compared to cross-study validation. Furthermore, the ranking of learning algorithms differs, suggesting that algorithms performing best in cross-validation may be suboptimal when evaluated through independent validation. Availability: The survHD: Survival in High Dimensions package (http://www.bitbucket.org/lwaldron/survhd) will be made available through Bioconductor. Contact: levi.waldron@hunter.cuny.edu Supplementary information: Supplementary data are available at Bioinformatics online.en-USJournal Article2014-07-0710.1093/bioinformatics/btu279