Advertisement

When Optimism Hurts: Inflated Predictions in Psychiatric Neuroimaging

      The ability to predict outcomes from neuroimaging data has the potential to answer important clinical questions such as which depressed patients will respond to treatment, which abstinent drug users will relapse, or which patients will convert to dementia. However, many prediction analyses require methods and techniques, not typically required in neuroimaging, to accurately assess a model’s predictive ability. Regression models will tend to fit to the idiosyncratic characteristics of a particular sample and consequently will perform worse on unseen data. Failure to account for this inherent optimism is especially pernicious when the number of possible predictors is high relative to the number of participants, a common scenario in psychiatric neuroimaging. We show via simulated data that models can appear predictive even when data and outcomes are random, and we note examples of optimistic prediction in the literature. We provide some recommendations for assessment of model performance.

      Key Words

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Biological Psychiatry
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Ambroise C.
        • McLachlan G.
        Selection bias in gene extraction on the basis of microarray gene-expression data.
        Proc Natl Acad Sci U S A. 2002; 99: 6562-6566
        • Goddard M.E.
        • Wray N.R.
        • Verbyla K.
        • Visscher P.M.
        Estimating effects and making predictions from genome-wide marker data.
        Stat Sci. 2009; 24: 517-529
        • Evans D.
        • Visscher P.
        • Wray N.
        Harnessing the information contained within genome-wide association studies to improve individual prediction of complex disease risk.
        Hum Mol Genet. 2009; 18: 3525-3531
        • Powell J.
        • Zietsch B.
        Predicting sensation seeking from dopamine genes: Use and misuse of genetic prediction.
        Psychol Sci. 2011; 22: 413-415
        • Stanislaw H.
        • Todorov N.
        Calculation of signal detection theory measures.
        Behav Res Methods Instruments Comput. 1999; 31: 137-149
        • Lee P.M.
        Bayesian Statistics: An Introduction.
        Wiley, London2012
        • Luo X.
        • Zhang S.
        • Hu S.
        • Bednarski S.
        • Erdman E.
        • Farr O.
        • et al.
        Error processing and gender shared and specific neural predictors of relapse in cocaine dependence.
        Brain. 2013; 1356: 1231-1244
        • Lavretsky H.
        • Zheng L.
        • Weiner M.
        • Mungas D.
        • Reed B.
        • Kramer J.
        • et al.
        Association of depressed mood and mortality in older adults with and without cognitive impairment in a prospective naturalistic study.
        Am J Psychiatry. 2010; 167: 589-597
        • Garner B.
        • Pariante C.
        • Wood S.
        • Velakoulis D.
        • Phillips L.
        • Soulsby B.
        • et al.
        Pituitary volume predicts future transition to psychosis in individuals at ultra-high risk of developing psychosis.
        Biol Psychiatry. 2005; 58: 417-423
        • Rando K.
        • Hong K.I.
        • Bhagwagar Z.
        • Li C.S.
        • Bergquist K.
        • Guarnaccia J.
        • et al.
        Association of frontal and posterior cortical gray matter volume with time to alcohol relapse: A prospective study.
        Am J Psychiatry. 2011; 168: 183-192
        • Walterfang M.
        • Yung A.
        • Wood A.
        • Reutens D.
        • Phillips L.
        • Wood S.
        • et al.
        Corpus callosum shape alterations in individuals prior to the onset of psychosis.
        Schizophr Res. 2008; 103: 1-10
        • Devanand D.
        • Bansal R.
        • Liu J.
        • Hao X.
        • Pradhaban G.
        • Peterson B.
        MRI hippocampal and entorhinal cortex mapping in predicting conversion to Alzheimer’s disease.
        Neuroimage. 2012; 60: 1622-1629
        • Tupler L.
        • Krishnan K.R.
        • Greenberg D.
        • Marcovina S.
        • Payne M.
        • MacFall J.
        • et al.
        Predicting memory decline in normal elderly: Genetics, MRI, and cognitive reserve.
        Neurobiol Aging. 2007; 28: 1644-1656
        • Zipoli V.
        • Goretti B.
        • Hakiki B.
        • Siracusa G.
        • Sorbi S.
        • Portaccio E.
        • et al.
        Cognitive impairment predicts conversion to multiple sclerosis in clinically isolated syndromes.
        Mult Scler. 2009; 16: 62-67
        • Braverman E.R.
        • Blum K.
        • Damle U.J.
        • Kerner M.
        • Dushaj K.
        • Oscar-Berman M.
        • et al.
        Evoked potentials and neuropsychological tests validate positron emission topography (PET) brain metabolism in cognitively impaired patients.
        PloS One. 2013; 8: e55398
        • Lin Y.-T.
        • Liu C.-M.
        • Chiu M.-J.
        • Liu C.-C.
        • Chien Y.-L.
        • Hwang T.-J.
        • et al.
        Differentiation of schizophrenia patients from healthy subjects by mismatch negativity and neuropsychological tests.
        PloS One. 2012; 7: e34454
        • Prichep L.
        • John E.
        • Ferris S.
        • Rausch L.
        • Fang Z.
        • Cancro R.
        • et al.
        Prediction of longitudinal cognitive decline in normal elderly with subjective complaints using electrophysiological imaging.
        Neurobiol Aging. 2006; 27: 471-481
        • Peduzzi P.
        • Concato J.
        • Kemper E.
        • Holford T.
        • Feinstein A.
        A simulation study of the number of events per variable in logistic regression analysis.
        J Clin Epidemiol. 1996; 49: 1373-1379
        • Vittinghoff E.
        • McCulloch C.
        Relaxing the rule of ten events per variable in logistic and Cox regression.
        Am J Epidemiol. 2007; 165: 710-718
        • Zou H.
        • Hastie T.
        Regularization and variable selection via the elastic net.
        J R Stat Soc B Stat Methodol. 2005; 67: 301-320
        • Tibshirani R.
        Regression shrinkage and selection via the lasso.
        J R Stat Soc B Stat Methodol. 1996; 58: 267-288
        • Moons K.
        • Donders A.
        • Steyerberg E.
        • Harrell F.
        Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: A clinical example.
        J Clin Epidemiol. 2004; 57: 1262-1270
        • Bramer M.
        Using J-pruning to reduce overfitting in classification trees.
        Knowledge Based Syst. 2002; 15: 301-308
        • Efron B.
        • Tibshirani R.J.
        An Introduction to the Bootstrap.
        Vol. 57. Chapman & Hall, New York1993
        • Efron B.
        • Tibshirani R.
        Improvements on cross-validation: The 632+ bootstrap method.
        J Am Stat Assoc. 1997; 92: 548-560
        • Magdon‐Ismail M.
        • Mertsalov K.
        A permutation approach to validation.
        Stat Analysis Data Mining. 2010; 3: 361-380
        • Kohavi R.
        A study of cross-validation and bootstrap for accuracy estimation and model selection.
        Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence. 1995; 2: 1137-1143
      1. Ng AY. Preventing overfitting of cross-validation data. Presented at the 14th International Conference on Machine Learning (ICML), 1997. Available at: http://robotics.stanford.edu/~ang/papers/cv-final.pdf. Accessed May 28, 2013.

        • Hoerl A.E.
        • Kennard R.W.
        Ridge regression: Biased estimation for nonorthogonal problems.
        Technometrics. 1970; 12: 55-67
      2. IBM Corp. IBM SPSS Statistics for Windows. Armonk, NY: IBM Corp.

        • Gomar J.
        • Bobes-Bascaran M.
        • Conejero-Goldberg C.
        • Davies P.
        • Goldberg T.
        Utility of combinations of biomarkers, cognitive markers, and risk factors to predict conversion from mild cognitive impairment to Alzheimer disease in patients in the Alzheimer’s disease neuroimaging initiative.
        Arch Gen Psychiatry. 2011; 68: 961-969
        • Clark V.
        • Beatty G.
        • Anderson R.
        • Kodituwakku P.
        • Phillips J.
        • Lane T.
        • et al.
        Reduced fMRI activity predicts relapse in patients recovering from stimulant dependence [published online ahead of print September 27].
        Hum Brain Mapp. 2012; https://doi.org/10.1002/hbm.22184
        • Devanand D.
        • Liu X.
        • Tabert M.
        • Pradhaban G.
        • Cuasay K.
        • Bell K.
        • et al.
        Combining early markers strongly predicts conversion from mild cognitive impairment to Alzheimer’s disease.
        Biol Psychiatry. 2008; 64: 871-879
        • Mechelli A.
        • Riecher-Rössler A.
        • Meisenzahl E.
        • Tognin S.
        • Wood S.
        • Borgwardt S.
        • et al.
        Neuroanatomical abnormalities that predate the onset of psychosis: A multicenter study.
        Arch Gen Psychiatry. 2011; 68: 489-495
        • Poline J.-B.
        • Brett M.
        The general linear model and fMRI: Does love last forever?.
        Neuroimage. 2012; 62: 871-880