Advertisement

Methodological Considerations for Brain-Based Predictive Modelling in Psychiatry

  • Elvisha Dhamala
    Correspondence
    Corresponding authors: Elvisha Dhamala
    Affiliations
    Department of Psychology, Yale University, New Haven, USA

    Kavli Institute for Neuroscience, Yale University, New Haven, USA
    Search for articles by this author
  • B.T. Thomas Yeo
    Affiliations
    Centre for Sleep & Cognition & Centre for Translational Magnetic Resonance Research, Yong Loo Lin School of Medicine, Singapore, National University of Singapore, Singapore, Singapore

    Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore

    N.1 Institute for Health & Institute for Digital Medicine, National University of Singapore, Singapore, Singapore

    Integrative Sciences and Engineering Programme (ISEP), National University of Singapore, Singapore, Singapore

    Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, MA, USA
    Search for articles by this author
  • Avram J. Holmes
    Correspondence
    Corresponding authors: Avram J. Holmes
    Affiliations
    Department of Psychology, Yale University, New Haven, USA

    Kavli Institute for Neuroscience, Yale University, New Haven, USA

    Department of Psychiatry, Yale University, New Haven, USA

    Wu Tsai Institute, Yale University, New Haven, USA
    Search for articles by this author
Open AccessPublished:September 28, 2022DOI:https://doi.org/10.1016/j.biopsych.2022.09.024

      Abstract

      Psychiatric illnesses are heterogeneous in nature. No illness manifests in the same way across individuals, and no two patients with a given diagnosis will exhibit identical symptom profiles. Over the last several decades, group-level analyses of in vivo neuroimaging data have led to fundamental advances in our understanding of the neurobiology of psychiatric illnesses. More recently, access to computational resources and large publicly available datasets alongside the rise of predictive modelling and precision medicine approaches have facilitated the study of psychiatric illnesses at an individual-level. Data-driven machine learning analyses can be applied to identify disease-relevant biological subtypes, predict individual symptom profiles, and recommend personalized therapeutic interventions. However, when developing these predictive models, methodological choices must be carefully considered to ensure accurate, robust, and interpretable results. Choices pertaining to algorithms, neuroimaging modalities and states, data transformation, phenotypes, parcellations, sample sizes, and populations we are specifically studying can influence model performance. Here, we review applications of neuroimaging-based machine learning models to study psychiatric illnesses and discuss the effects of different methodological choices on model performance. An understanding of these effects is crucial for the proper implementation of predictive models in psychiatry and will facilitate more accurate diagnoses, prognoses, and therapeutics.

      Keywords

      Introduction

      Progress across the brain sciences has provided unprecedented opportunities for advancing our understanding of the etiology and pathogenesis of psychiatric illness. In the last several decades, new in vivo imaging techniques, such as magnetic resonance imaging, have facilitated the successful identification of neural correlates of human behavior. Traditionally, patient populations were treated as distinct entities, and corresponding group-level analyses shed light on the neurobiological changes that characterize these nominally distinct diagnostic constructs. However, the heterogeneity in psychiatric illnesses and murky boundaries between health and disease make it difficult to establish the extent to which brain-behavior relationships are consistently or uniquely disrupted. To address this heterogeneous nature and circumvent diagnostic biases, precision medicine has emerged as a powerful tool to understand clinical presentation at an individual-level by considering a range of biological and environmental factors(

      Council NR (2011): Toward precision medicine: building a knowledge network for biomedical research and a new taxonomy of disease.

      ). We are currently at an exciting time where we can harness the joint powers of precision medicine, machine learning, and neuroimaging to enhance our understanding of the neurobiology of psychiatric illnesses and associated symptom profiles. Increased access to computational resources and large publicly available datasets are supporting a field-wide shift to data-driven discovery science, allowing descriptive models of psychiatric illness, predictions of clinically-relevant behaviors, and personalized prescriptions for therapeutic interventions ushering in a new era for psychiatry.
      In psychiatry, precision medicine and predictive modeling have two primary goals: to discover how the brain gives rise to behavior and to identify practically useful markers of behavior and clinical outcome. Precision medicine transitions away from the one-size-fits-all model of traditional medicine and seeks to identify the underlying causes of disease and appropriate therapeutic interventions at a patient-specific level. This individualized approach has led to significant advances across the health sciences, for instance in oncology where personalized treatments are now available for a range of cancers(
      • Dugger S.A.
      • Platt A.
      • Goldstein D.B.
      Drug development in the era of precision medicine.
      ). The application of predictive models in psychiatry, termed precision psychiatry(

      Vieta E (2015): Personalised medicine applied to mental health: Precision psychiatry. pp 117-118.

      ), incorporates insights from molecular biology, cognitive neuroscience, neurophysiology, and behavioral and clinical psychology to capture clinically-relevant biomarkers. If successful, precision psychiatry will transform how we diagnose and treat psychiatric illnesses.
      There is growing interest in the development and application of novel tools to predict illness risk, disease progression, and identify shared and unique properties of diagnostic groups and associated symptom profiles. Psychiatric diagnoses categorize clusters of co-occurring symptoms that often lack clear discernible borders between health and disease(
      • Hyman S.E.
      The diagnosis of mental disorders: the problem of reification.
      ). While the presence of specific symptom profiles can distinguish diagnoses, phenotypic presentations can overlap across patient groups making it difficult to identify specific markers of illness. In this regard, discrete and dimensional models of psychiatric illness provide complementary information. Discrete medical models provide diagnostic labels based on clinical descriptions of signs and symptoms, while dimensional models describe broad hierarchical features of psychopathology. Together, different models of psychiatric illness can facilitate improved understanding of psychopathology at an individual-level.
      Traditional analyses in psychiatry have largely sought to uncover group-level patterns characterizing distinct illnesses. While these analyses have provided foundational insights in our understanding of psychiatric illness, they fail to adequately capture individual variability in symptom presentations. Machine learning provides a rich analytic framework to reveal the neurobiological correlates of discrete and dimensional measures of psychopathology at an individual-level. By leveraging our understanding of brain-behavior relationships and considering a specific patient’s personalized profile, machine learning can be used to generate individual-level behavioral predictions. Machine learning analyses can be characterized into three groups: descriptive, predictive, and prescriptive, each with distinct utility for psychiatry researchers (Figure 1). These analyses rely on supervised, semi-supervised, and unsupervised algorithms. As an exhaustive overview of different analyses and algorithms is not possible here, we provide key terms and examples in Table 1 (for additional discussion see(
      • Yip S.W.
      • Kiluk B.
      • Scheinost D.
      Toward addiction prediction: an overview of cross-validated predictive modeling findings and considerations for future neuroimaging research.
      )).
      Figure thumbnail gr1
      Figure 1Predictive models based on neuroimaging data can be applied in psychiatry to perform descriptive, predictive, and prescriptive analyses. Information obtained from structural, functional, and diffusion scans at a population-level can be leveraged to gain insights at an individual-level. Descriptive analyses detect underlying patterns in the neurobiological data at a population-level and describe how those patterns manifest at an individual-level. These approaches rely largely on unsupervised algorithms and are technically postdictive, in that they retrospectively detect trends in existing data, rather than prospective (i.e., future) predictions. Predictive analyses capture brain-behavior relationships at a population-level and use those to predict psychiatric diagnosis, dimensions, cognition, and personality at an individual. These predictive models rely on supervised algorithms and have the potential to identify future risks for development or deterioration of illness in a personalized manner. Prescriptive analyses establish neurobiological predictors of treatment response trends at a population-level and provide recommendations on interventions at an individual-level. These approaches rely on supervised algorithms to identify anomalies and suggest personalized interventions that can most effectively alleviate individual specific patterns of concerns and/or other relevant factors. Together, descriptive, predictive, and prescriptive analyses can yield clinically meaningful insights that can facilitate the development of personalized diagnostic tools and therapeutic interventions.
      Table 1Definitions of Key Terms
      CategoryTermDefinition/DescriptionExamples and Additional Resources
      AnalysesDescriptiveRely on existing observations to identify underlying patterns that can describe and summarize the data in an interpretable mannerIdentification of distinct subtypes of autism (
      • Tang S.
      • Sun N.
      • Floris D.L.
      • Zhang X.
      • Di Martino A.
      • Yeo B.T.T.
      Reconciling Dimensional and Categorical Models of Autism Heterogeneity: A Brain Connectomics and Behavioral Study.
      ,
      • Hong S.J.
      • Valk S.L.
      • Di Martino A.
      • Milham M.P.
      • Bernhardt B.C.
      Multidimensional Neuroanatomical Subtyping of Autism Spectrum Disorder.
      ), depression (
      • Drysdale A.T.
      • Grosenick L.
      • Downar J.
      • Dunlop K.
      • Mansouri F.
      • Meng Y.
      • et al.
      Resting-state connectivity biomarkers define neurophysiological subtypes of depression.
      ,
      • Wang Y.
      • Tang S.
      • Zhang L.
      • Bu X.
      • Lu L.
      • Li H.
      • et al.
      Data-driven clustering differentiates subtypes of major depressive disorder with distinct brain connectivity and symptom features.
      ), psychosis (
      • Clementz B.A.
      • Sweeney J.A.
      • Hamm J.P.
      • Ivleva E.I.
      • Ethridge L.E.
      • Pearlson G.D.
      • et al.
      Identification of distinct psychosis biotypes using brain-based biomarkers.
      ), schizophrenia (
      • Talpalaru A.
      • Bhagwat N.
      • Devenyi G.A.
      • Lepage M.
      • Chakravarty M.M.
      Identifying schizophrenia subgroups using clustering and supervised learning.
      ,
      • Chen J.
      • Patil K.R.
      • Weis S.
      • Sim K.
      • Nickl-Jockschat T.
      • Zhou J.
      • et al.
      Neurobiological Divergence of the Positive and Negative Schizophrenia Subtypes Identified on a New Factor Structure of Psychopathology Using Non-negative Factorization: An International Machine Learning Study.
      ), and bipolar and schizoaffective disorders (

      Yan W, Zhao M, Fu Z, Pearlson GD, Sui J, Calhoun VD (2021): Mapping relationships among schizophrenia, bipolar and schizoaffective disorders: A deep classification and clustering framework using fMRI time series. Schizophrenia Research.

      )
      PredictiveEstablish associations between independent and dependent variables in existing data, uses learned associations to predict the trends and properties in unobserved samplesPrediction of clinical, cognitive, and personality traits across different psychiatric illnesses (
      • Mellem M.S.
      • Liu Y.
      • Gonzalez H.
      • Kollada M.
      • Martin W.J.
      • Ahammad P.
      Machine learning models identify multimodal measurements highly predictive of transdiagnostic symptom severity for mood, anhedonia, and anxiety.
      ,
      • Chen J.
      • Muller V.I.
      • Dukart J.
      • Hoffstaedter F.
      • Baker J.T.
      • Holmes A.J.
      • et al.
      Intrinsic Connectivity Patterns of Task-Defined Brain Networks Allow Individual Prediction of Cognitive Symptom Dimension of Schizophrenia and Are Linked to Molecular Architecture.
      ,
      • Wang D.
      • Li M.
      • Wang M.
      • Schoeppe F.
      • Ren J.
      • Chen H.
      • et al.
      Individual-specific functional connectivity markers track dimensional and categorical features of psychotic illness.
      ,
      • Kebets V.
      • Holmes A.J.
      • Orban C.
      • Tang S.
      • Li J.
      • Sun N.
      • et al.
      Somatosensory-Motor Dysconnectivity Spans Multiple Transdiagnostic Dimensions of Psychopathology.
      ,
      • Xia C.H.
      • Ma Z.
      • Ciric R.
      • Gu S.
      • Betzel R.F.
      • Kaczkurkin A.N.
      • et al.
      Linked dimensions of psychopathology and connectivity in functional brain networks.
      )
      PrescriptiveIdentify patterns and trends in the underlying data and recommend personalized interventions that can most effectively alleviate individual specific patterns of concernPrediction of clinical responses to medications (
      • Ju Y.
      • Horien C.
      • Chen W.
      • Guo W.
      • Lu X.
      • Sun J.
      • et al.
      Connectome-based models can predict early symptom improvement in major depressive disorder.
      ,
      • Sarpal D.K.
      • Argyelan M.
      • Robinson D.G.
      • Szeszko P.R.
      • Karlsgodt K.H.
      • John M.
      • et al.
      Baseline striatal functional connectivity as a predictor of response to antipsychotic drug treatment.
      ) and cognitive behavioral therapy (
      • Whitfield-Gabrieli S.
      • Ghosh S.
      • Nieto-Castanon A.
      • Saygin Z.
      • Doehrmann O.
      • Chai X.
      • et al.
      Brain connectomics predict response to treatment in social anxiety disorder.
      ,
      • Doehrmann O.
      • Ghosh S.S.
      • Polli F.E.
      • Reynolds G.O.
      • Horn F.
      • Keshavan A.
      • et al.
      Predicting treatment response in social anxiety disorder from functional magnetic resonance imaging.
      ,
      • Hahn T.
      • Kircher T.
      • Straube B.
      • Wittchen H.-U.
      • Konrad C.
      • Ströhle A.
      • et al.
      Predicting treatment response to cognitive behavioral therapy in panic disorder with agoraphobia by integrating local neural information.
      )
      AlgorithmsSupervisedRely on pre-observed ground truth labels to learn associations between the input and output dataPrediction of treatment response in social anxiety disorder (
      • Doehrmann O.
      • Ghosh S.S.
      • Polli F.E.
      • Reynolds G.O.
      • Horn F.
      • Keshavan A.
      • et al.
      Predicting treatment response in social anxiety disorder from functional magnetic resonance imaging.
      )
      Semi-SupervisedCombine a small amount of labeled data and a large amount of unlabeled data to identify associations between the input and output dataIdentification of subtypes of internalizing disorders in youth (
      • Kaczkurkin A.N.
      • Sotiras A.
      • Baller E.B.
      • Barzilay R.
      • Calkins M.E.
      • Chand G.B.
      • et al.
      Neurostructural heterogeneity in youths with internalizing symptoms.
      )
      UnsupervisedGroup and interpret patterns in unlabeled data to characterize dataInvestigation of overlapping and specific correlates of common psychiatric illnesses (
      • Zhao X.
      • Rangaprakash D.
      • Yuan B.
      • Denney Jr., T.S.
      • Katz J.S.
      • Dretsch M.N.
      • et al.
      Investigating the Correspondence of Clinical Diagnostic Grouping With Underlying Neurobiological and Phenotypic Clusters Using Unsupervised Machine Learning.
      )
      Supervised AlgorithmsClassificationEstablish relationships between the input variables and discrete binary or multi-class categorical labelsDistinguishing healthy and clinical populations

      See (
      • Nielsen A.N.
      • Barch D.M.
      • Petersen S.E.
      • Schlaggar B.L.
      • Greene D.J.
      Machine learning with neuroimaging: Evaluating its applications in psychiatry.
      ) for a review on the application of clinically informative classifiers
      RegressionPredict continuous variables based on the input dataPrediction of disease severity

      See (
      • Sui J.
      • Jiang R.
      • Bustillo J.
      • Calhoun V.
      Neuroimaging-based individualized prediction of cognition and behavior for mental disorders and health: methods and promises.
      ) for a review on the use of regression algorithms to elucidate brain-behavior relationships
      Unsupervised AlgorithmsClusteringEnable the characterization of data into distinct groups (clusters) based on shared underlying patternsBiologically driven disease subtype identification
      ModelsClassical Machine LearningRelatively ‘simple’ models that rely on linear or non-linear and sparse or non-sparse approaches to derive associations between the input and output dataLinear regression, logistic regression, support vector machines
      Deep LearningSubset of machine learning models that rely on one or more neural networks (e.g., perceptrons) to progressively extract higher-level features from the input data to make predictions about the outputFeedforward neural networks, convolutional neural networks
      VariablesBinaryDiscrete data that can fall in one of two categoriesPresence or absence of an illness
      CategoricalDiscrete data that can fall in one of several categoriesSubtypes of an illness
      ContinuousData can take on any value within a continuous rangeDimensional measures of impairment
      RegularizationL1 (Sparse)Uses the absolute value of magnitude of the parameters as the penalty term, thus encouraging sparsity by shrinking least important coefficients to zero
      L2 (Non-Sparse)Uses the squared magnitude of the parameters as the penalty term, thus reducing the magnitudes of parameters
      Model PropertiesComplexityA more complex model has the capacity for more complicated relationship among the features and target variables
      TransparencyAbility to understand and interpret the brain-behavior relationships captured by the model
      Model PerformanceAccuracyCorrelation (or correspondence) between true and predicted values when evaluating a model on a (held out) test set that is from the same population (or dataset) as the training set
      GeneralizabilityCorrelation (or correspondence) between true and predicted values when evaluating a model on a population (or dataset) that is unique from the population in which it was trained
      Guidelines for implementing predictive models to study behavioral variability based on neuroimaging data have been proposed(
      • Yip S.W.
      • Kiluk B.
      • Scheinost D.
      Toward addiction prediction: an overview of cross-validated predictive modeling findings and considerations for future neuroimaging research.
      ,
      • Scheinost D.
      • Noble S.
      • Horien C.
      • Greene A.S.
      • Lake E.M.
      • Salehi M.
      • et al.
      Ten simple rules for predictive modeling of individual differences in neuroimaging.
      ,
      • Rosenberg M.D.
      • Casey B.
      • Holmes A.J.
      Prediction complements explanation in understanding the developing brain.
      ,
      • Varoquaux G.
      • Raamana P.R.
      • Engemann D.A.
      • Hoyos-Idrobo A.
      • Schwartz Y.
      • Thirion B.
      Assessing and tuning brain decoders: cross-validation, caveats, and guidelines.
      ,
      • Woo C.-W.
      • Chang L.J.
      • Lindquist M.A.
      • Wager T.D.
      Building better biomarkers: brain models in translational neuroimaging.
      ,
      • Arbabshirani M.R.
      • Plis S.
      • Sui J.
      • Calhoun V.D.
      Single subject prediction of brain disorders in neuroimaging: Promises and pitfalls.
      ,

      Khosla M, Jamison K, Ngo GH, Kuceyeski A, Sabuncu MR (2019): Machine learning in resting-state fMRI analysis. Magn Reson Imaging.

      ), but there has not yet been a review of how different methodological considerations can influence the models. Here, we review the effects of algorithm selection, neuroimaging modality, data transformation, sample sizes, brain parcellation, phenotype selection, and population-specific approaches on predictive models and contextualize how insights gained from those analyses can facilitate advances in psychiatry (Figure 2).
      Figure thumbnail gr2
      Figure 2Methodological choices for machine learning models based on neuroimaging can bias the performance of predictive models. Sample size: the number of individuals and the heterogeneity within the population. Neuroimaging modality: the choice of structural, functional, or diffusion imaging scans. State: functional scans may be acquired to measure activation patterns corresponding to intrinsic (resting-state), task-evoked, or naturalistic states (e.g., movie watching). Parcellations: distinct parcellations derived using structural, functional, clustering, gradient, or other information may be applied to the neuroimaging data. Data transformation: neuroimaging data must be preprocessed and transformed before it can be input into a predictive algorithm. Population-specific vs. general models: the entire sample may be considered as a single population or separated into distinct sub-samples based on different factors such as demographics or diagnosis. Algorithm: predictive algorithms include linear or non-linear traditional machine learning and deep learning approaches. Phenotype: the specific variable to be predicted may be a categorical diagnosis, dimensional measures of psychopathology, cognitive abilities, or personality traits. Predictive modelling in psychiatry must be implemented must be designed with careful consideration of how these factors may potentially influence model accuracy and interpretability.

      Algorithm Selection

      The structure of neuroimaging data presents a unique set of analytic challenges in terms of dimensionality, collinearity, and sparsity. To address these challenges, different algorithms have been designed including the connectome-based predictive model(
      • Shen X.
      • Finn E.S.
      • Scheinost D.
      • Rosenberg M.D.
      • Chun M.M.
      • Papademetris X.
      • et al.
      Using connectome-based predictive modeling to predict individual behavior from brain connectivity.
      ) and BrainNet convolutional neural network(
      • Kawahara J.
      • Brown C.J.
      • Miller S.P.
      • Booth B.G.
      • Chau V.
      • Grunau R.E.
      • et al.
      BrainNetCNN: Convolutional neural networks for brain networks; towards predicting neurodevelopment.
      ). The benefits of these cutting-edge approaches include: (

      Council NR (2011): Toward precision medicine: building a knowledge network for biomedical research and a new taxonomy of disease.

      ) data-driven feature selection algorithms that do not rely on brain regions or connections selected a priori, but rather account for the fact that behavior and cognition may be best characterized as properties of covariation across large-scale networks; (
      • Dugger S.A.
      • Platt A.
      • Goldstein D.B.
      Drug development in the era of precision medicine.
      ) built-in cross-validation, which ensures models are generalizable and replicable; and (

      Vieta E (2015): Personalised medicine applied to mental health: Precision psychiatry. pp 117-118.

      ) the potential for continuous predictions that, in contrast to categorical or diagnostic predictions, provide more dimensional measures of psychopathology. Connectome-based predictive modelling is a data-driven protocol to predict behavior from connectivity data. Here, researchers determine which connectivity features are most strongly correlated with a given phenotype, and then use those features to build a linear model to predict behavior. This approach has identified functional connectivity correlates of attention(
      • Yoo K.
      • Rosenberg M.D.
      • Hsu W.-T.
      • Zhang S.
      • Li C.-S.R.
      • Scheinost D.
      • et al.
      Connectome-based predictive modeling of attention: Comparing different functional connectivity features and prediction methods across datasets.
      ,
      • Rosenberg M.D.
      • Hsu W.-T.
      • Scheinost D.
      • Todd Constable R.
      • Chun M.M.
      Connectome-based models predict separable components of attention in novel individuals.
      ), memory(
      • Barron D.S.
      • Gao S.
      • Dadashkarimi J.
      • Greene A.S.
      • Spann M.N.
      • Noble S.
      • et al.
      Transdiagnostic, connectome-based prediction of memory constructs across psychiatric disorders.
      ), and anxiety(
      • Wang Z.
      • Goerlich K.S.
      • Ai H.
      • Aleman A.
      • Luo Y-j
      • Xu P.
      Connectome-based predictive modeling of individual anxiety.
      ) across distinct populations, and predicted responses to antidepressants(
      • Ju Y.
      • Horien C.
      • Chen W.
      • Guo W.
      • Lu X.
      • Sun J.
      • et al.
      Connectome-based models can predict early symptom improvement in major depressive disorder.
      ) and treatment for cocaine and opioid use disorders(
      • Yip S.W.
      • Scheinost D.
      • Potenza M.N.
      • Carroll K.M.
      Connectome-based prediction of cocaine abstinence.
      ,
      • Lichenstein S.D.
      • Scheinost D.
      • Potenza M.N.
      • Carroll K.M.
      • Yip S.W.
      Dissociable neural substrates of opioid and cocaine use identified via connectome-based modelling.
      ). BrainNet relies on a convolutional neural network framework that maintains the topological complexity of brain networks. This framework has been applied to predict future cognitive and motor development in preterm infants based on white matter connectivity(
      • Kawahara J.
      • Brown C.J.
      • Miller S.P.
      • Booth B.G.
      • Chau V.
      • Grunau R.E.
      • et al.
      BrainNetCNN: Convolutional neural networks for brain networks; towards predicting neurodevelopment.
      ).
      While both connectome-based predictive models and BrainNet convolutional networks support the analysis of connectomic data, other classical machine learning and deep learning algorithms can also generate meaningful predictions. Amidst ongoing debates on which algorithms yield the most accurate predictions, head-to-head comparisons suggest a suite of approaches may be clinically useful. In one recent comparison, kernel ridge regression, a generic feedforward network, a BrainNet convolutional network, and a graph convolutional network generated comparable cognitive predictions based on functional connectivity(
      • He T.
      • Kong R.
      • Holmes A.J.
      • Nguyen M.
      • Sabuncu M.R.
      • Eickhoff S.B.
      • et al.
      Deep neural networks and kernel regression achieve comparable accuracies for functional connectivity prediction of behavior and demographics.
      ). A separate analysis also reported that classical approaches, including elastic net (which optimizes between L1 and L2 regularization), achieve comparable or better predictive accuracies than deep learning methods to capture associations between functional connectivity and cognition(
      • Pervaiz U.
      • Vidaurre D.
      • Woolrich M.W.
      • Smith S.M.
      Optimising network modelling methods for fMRI.
      ). However, findings have been mixed, and a separate comparison revealed deep learning models produced the best cognitive predictions based on whole brain gray matter volume in healthy and clinical (mild cognitive impairment, Alzheimer’s disease) populations(
      • Abrol A.
      • Fu Z.
      • Salman M.
      • Silva R.
      • Du Y.
      • Plis S.
      • et al.
      Deep learning encodes robust discriminative neuroimaging representations to outperform standard machine learning.
      ).
      Comparisons of classical and deep learning approaches for classification problems have also produced mixed findings. One study found deep learning classifiers were more accurate when relying on anatomical or functional data (or both) to distinguish individuals with autism from healthy controls(
      • Mellema C.J.
      • Nguyen K.P.
      • Treacher A.
      • Montillo A.
      Reproducible neuroimaging features for diagnosis of autism spectrum disorder with machine learning.
      ). On the contrary, an international challenge to predict autism spectrum diagnosis from anatomical and functional data revealed deep learning models were more likely to overfit such that they failed to generalize to different datasets whereas simpler models achieved more robust performance(

      Traut N, Heuer K, Lemaître G, Beggiato A, Germanaud D, Elmaleh M, et al. (2022): Insights from an autism imaging biomarker challenge: promises and threats to biomarker discovery. NeuroImage.119171.

      ). Relatedly, across different classical algorithms, non-sparse linear methods more accurately distinguish different psychiatric diagnostic groups from healthy controls than sparse linear or non-linear methods(
      • Dadi K.
      • Rahim M.
      • Abraham A.
      • Chyzhyk D.
      • Milham M.
      • Thirion B.
      • et al.
      Benchmarking functional connectome-based predictive models for resting-state fMRI.
      ).
      Hence, different machine learning algorithms are suitable for distinct predictions, and algorithm selection should consider whether accuracy, interpretability, or both should be prioritized and be primarily driven by the goals of the study.

      Neuroimaging Modality

      Different neuroimaging modalities capture distinct information about the underlying neurobiology. Structural MRI provides information about the shape, size, and integrity of gray and white matter brain structures, and can be used to study physical damage and lesions. Functional MRI indirectly measures brain activation by detecting changes in blood oxygenation, facilitating the study of functional activation patterns and correlated signals across functional systems. Finally, diffusion MRI measures diffusion of water molecules in biological tissues and enables the quantification of white matter tracts throughout the brain. These modalities yield distinct biologically relevant features for the study of psychiatric illnesses and prediction of clinically-relevant phenotypes.
      Converging evidence suggests that individual differences in cognition, personality, and emotional traits, as well as psychiatric diagnoses, are most accurately predicted by functional connectivity(
      • Mellema C.J.
      • Nguyen K.P.
      • Treacher A.
      • Montillo A.
      Reproducible neuroimaging features for diagnosis of autism spectrum disorder with machine learning.
      ,

      Traut N, Heuer K, Lemaître G, Beggiato A, Germanaud D, Elmaleh M, et al. (2022): Insights from an autism imaging biomarker challenge: promises and threats to biomarker discovery. NeuroImage.119171.

      ,
      • Dhamala E.
      • Jamison K.W.
      • Jaywant A.
      • Dennis S.
      • Kuceyeski A.
      Distinct functional and structural connections predict crystallised and fluid cognition in healthy adults.
      ,
      • Mansour S.L.
      • Tian Y.
      • Yeo B.T.
      • Cropley V.
      • Zalesky A.
      High-resolution connectomic fingerprints: Mapping neural identity and behavior.
      ,
      • Ooi L.Q.R.
      • Chen J.
      • Zhang S.
      • Kong R.
      • Li J.
      • Dhamala E.
      • et al.
      Comparison of individualized behavioral predictions across anatomical, diffusion and functional connectivity MRI.
      ,
      • Mellem M.S.
      • Liu Y.
      • Gonzalez H.
      • Kollada M.
      • Martin W.J.
      • Ahammad P.
      Machine learning models identify multimodal measurements highly predictive of transdiagnostic symptom severity for mood, anhedonia, and anxiety.
      ,
      • Seguin C.
      • Tian Y.
      • Zalesky A.
      Network communication models improve the behavioral and functional predictive utility of the human structural connectome.
      ). These results, reported across healthy and clinical populations, suggest that functional MRI data may be the best modality for predictive modelling. However, there is evidence to suggest that other modalities may also produce clinically informative predictions, especially in psychiatric illnesses associated with structural abnormalities in the brain.
      Cortical gray matter volume and thickness successfully predicts diagnosis of Alzheimer’s, autism, and attention deficit disorders(
      • Sabuncu M.R.
      • Konukoglu E.
      • AsDN Initiative
      Clinical prediction from structural brain MRI scans: a large-scale empirical study.
      ) and symptom burden in schizophrenia(
      • Talpalaru A.
      • Bhagwat N.
      • Devenyi G.A.
      • Lepage M.
      • Chakravarty M.M.
      Identifying schizophrenia subgroups using clustering and supervised learning.
      ). Gray matter density reliably estimates response to repetitive transcranial magnetic stimulation in patients with schizophrenia(
      • Koutsouleris N.
      • Wobrock T.
      • Guse B.
      • Langguth B.
      • Landgrebe M.
      • Eichhammer P.
      • et al.
      Predicting response to repetitive transcranial magnetic stimulation in patients with schizophrenia using structural magnetic resonance imaging: a multisite machine learning analysis.
      ). Finally, voxel-level structural scans distinguish whether a patient with depression may be treatment-resistant(
      • Johnston B.A.
      • Steele J.D.
      • Tolomeo S.
      • Christmas D.
      • Matthews K.
      Structural MRI-based predictions in patients with treatment-refractory depression (TRD).
      ).
      Meanwhile, diffusion-derived measures of fractional anisotropy along with radial, mean, and axial diffusivity predict response to cognitive behavioral therapy in social anxiety disorder(
      • Whitfield-Gabrieli S.
      • Ghosh S.
      • Nieto-Castanon A.
      • Saygin Z.
      • Doehrmann O.
      • Chai X.
      • et al.
      Brain connectomics predict response to treatment in social anxiety disorder.
      ). White matter connectivity characterizes tic improvement following deep brain stimulation for Tourette syndrome(
      • Johnson K.A.
      • Duffley G.
      • Anderson D.N.
      • Ostrem J.L.
      • Welter M.-L.
      • Baldermann J.C.
      • et al.
      Structural connectivity predicts clinical outcomes of deep brain stimulation for Tourette syndrome.
      ) and distinguishes healthy controls from patients with mild cognitive impairments and patients with dementia(
      • Shao J.
      • Myers N.
      • Yang Q.
      • Feng J.
      • Plant C.
      • Böhm C.
      • et al.
      Prediction of Alzheimer's disease using individual structural connectivity networks.
      ).
      Intrinsic vs. Task-Evoked Measures of Brain Functioning
      Intrinsic (or resting state) functional connectivity captures the extent to which neural activation patterns between a pair of brain regions covaries over time, providing insight into general functional network organization in the brain. Task-evoked functional connectivity characterizes covariations in neural activation patterns while performing specific tasks and facilitates the understanding of the functional networks that underlie different behaviors. While both intrinsic and task-evoked measures of functional network connectivity can generate insightful behavioral predictions, models based on task-evoked connectivity more successfully predict individual cognitive traits(
      • Greene A.S.
      • Gao S.
      • Scheinost D.
      • Constable R.T.
      Task-induced brain state manipulation improves prediction of individual traits.
      ,
      • Chen J.
      • Tam A.
      • Kebets V.
      • Orban C.
      • Ooi L.Q.R.
      • Asplund C.L.
      • et al.
      Shared and unique brain network features predict cognitive, personality, and mental health scores in the ABCD study.
      ). Moreover, stacked models incorporating multimodal data yield the greatest accuracy(
      • Ooi L.Q.R.
      • Chen J.
      • Zhang S.
      • Kong R.
      • Li J.
      • Dhamala E.
      • et al.
      Comparison of individualized behavioral predictions across anatomical, diffusion and functional connectivity MRI.
      ). Intrinsic striatal connectivity predicts responses to antipsychotic drug treatment in patients with schizophrenia and acute psychosis(
      • Sarpal D.K.
      • Argyelan M.
      • Robinson D.G.
      • Szeszko P.R.
      • Karlsgodt K.H.
      • John M.
      • et al.
      Baseline striatal functional connectivity as a predictor of response to antipsychotic drug treatment.
      ), while whole-brain intrinsic functional connectivity distinguishes bipolar and major depressive disorder(
      • Du Y.
      • Fu Z.
      • Sui J.
      • Gao S.
      • Xing Y.
      • Lin D.
      • et al.
      NeuroMark: An automated and adaptive ICA based pipeline to identify reproducible fMRI markers of brain disorders.
      ), and estimates risk of developing anxiety disorders(
      • Wang Z.
      • Goerlich K.S.
      • Ai H.
      • Aleman A.
      • Luo Y-j
      • Xu P.
      Connectome-based predictive modeling of individual anxiety.
      ). Meanwhile, connections evoked during emotional face perception tasks meaningfully distinguish individuals with social anxiety disorder, panic disorder, and healthy controls(
      • Pantazatos S.P.
      • Talati A.
      • Schneier F.R.
      • Hirsch J.
      Reduced anterior temporal and hippocampal functional connectivity during face processing discriminates individuals with social anxiety disorder from healthy controls and panic disorder, and increases following treatment.
      ). Task-evoked connectivity using memory and general executive function tasks also transdiagnostically predicts memory deficits across schizophrenic, bipolar, and attention deficit hyperactivity disorders, and these models generalize to healthy participants(
      • Barron D.S.
      • Gao S.
      • Dadashkarimi J.
      • Greene A.S.
      • Spann M.N.
      • Noble S.
      • et al.
      Transdiagnostic, connectome-based prediction of memory constructs across psychiatric disorders.
      ). Alternatively, task-evoked measures of brain activation are also clinically meaningful. Activation patterns discriminate between individuals with schizophrenia and bipolar disorder(
      • Calhoun V.D.
      • Maciejewski P.K.
      • Pearlson G.D.
      • Kiehl K.A.
      Temporal lobe and “default” hemodynamic brain modes discriminate between schizophrenia and bipolar disorder.
      ), distinguish between major depression with and without mania from bipolar depression(
      • Cha J.
      • Speaker S.
      • Hu B.
      • Altinay M.
      • Koirala P.
      • Karne H.
      • et al.
      Neuroimaging correlates of emotional response-inhibition discriminate between young depressed adults with and without sub-threshold bipolar symptoms (Emotional Response-inhibition in Young Depressed Adults).
      ), and predict clinical responses to cognitive behavioral therapy in social anxiety disorder(
      • Doehrmann O.
      • Ghosh S.S.
      • Polli F.E.
      • Reynolds G.O.
      • Horn F.
      • Keshavan A.
      • et al.
      Predicting treatment response in social anxiety disorder from functional magnetic resonance imaging.
      ) and in panic disorder(
      • Hahn T.
      • Kircher T.
      • Straube B.
      • Wittchen H.-U.
      • Konrad C.
      • Ströhle A.
      • et al.
      Predicting treatment response to cognitive behavioral therapy in panic disorder with agoraphobia by integrating local neural information.
      ).
      The acquisition of high-quality functional MRI scans sensitive to motion may be difficult to achieve in psychiatric populations. Patient wellbeing can be prioritized while addressing this to acquire high-quality data using different strategies including familiarizing subjects with the scanner environment, using physical cushioning/restraints, and implementing motion correction strategies(
      • Greene D.J.
      • Black K.J.
      • Schlaggar B.L.
      Considerations for MRI study design and implementation in pediatric and clinical populations.
      ,
      • Power J.D.
      • Barnes K.A.
      • Snyder A.Z.
      • Schlaggar B.L.
      • Petersen S.E.
      Spurious but systematic correlations in functional connectivity MRI networks arise from subject motion.
      ,
      • Power J.D.
      • Mitra A.
      • Laumann T.O.
      • Snyder A.Z.
      • Schlaggar B.L.
      • Petersen S.E.
      Methods to detect, characterize, and remove motion artifact in resting state fMRI.
      ,
      • Power J.D.
      • Schlaggar B.L.
      • Petersen S.E.
      Recent progress and outstanding issues in motion correction in resting state fMRI.
      ). Moreover, while task scans predict behavioral traits more accurately in healthy populations, they can be more cognitively demanding on participants, require additional scan time, and present language barriers, making them less feasible in clinical samples. To circumvent this issue, resting-state data can be used to synthesize task data(
      • Tavor I.
      • Jones O.P.
      • Mars R.B.
      • Smith S.
      • Behrens T.
      • Jbabdi S.
      Task-free MRI predicts individual differences in brain activity during task performance.
      ) and naturalistic stimuli (movie-watching) data(
      • Gruskin D.C.
      • Patel G.H.
      Brain connectivity at rest predicts individual differences in normative activity during movie watching.
      ). Those synthesized data can predict individual differences in brain activity during task performance(
      • Tavor I.
      • Jones O.P.
      • Mars R.B.
      • Smith S.
      • Behrens T.
      • Jbabdi S.
      Task-free MRI predicts individual differences in brain activity during task performance.
      ), naturalistic viewing(
      • Gruskin D.C.
      • Patel G.H.
      Brain connectivity at rest predicts individual differences in normative activity during movie watching.
      ), and behavioral traits(

      Gal S, Tik N, Bernstein-Eliav M, Tavor I (2022): Predicting Individual Traits from Unperformed Tasks. NeuroImage.118920.

      ) in healthy populations. In a clinical context, this approach can facilitate more accurate behavioral predictions in a psychiatric population, reducing (and even eliminating) the monetary and cognitive resources associated with the acquisition of task and naturalistic scans.

      Data Transformation

      To transform noisy estimates of brain structure and function into reliable predictors of human behavior, neuroimaging data must be carefully preprocessed and cleaned. Preprocessing pipelines and analytical methods can introduce variability into neuroimaging studies and subsequently bias findings(
      • Botvinik-Nezer R.
      • Holzmeister F.
      • Camerer C.F.
      • Dreber A.
      • Huber J.
      • Johannesson M.
      • et al.
      Variability in the analysis of a single neuroimaging dataset by many teams.
      ,
      • Bhagwat N.
      • Barry A.
      • Dickie E.W.
      • Brown S.T.
      • Devenyi G.A.
      • Hatano K.
      • et al.
      Understanding the impact of preprocessing pipelines on neuroimaging cortical surface analyses.
      ,
      • Bowring A.
      • Nichols T.E.
      • Maumet C.
      Isolating the sources of pipeline‐variability in group‐level task‐fMRI results.
      ,
      • Bowring A.
      • Maumet C.
      • Nichols T.E.
      Exploring the impact of analysis software on task fMRI results.
      ).
      Before any processing, analyses, or modelling can be performed, neuroimaging data must undergo quality control to ensure images are free of artifacts and have a high signal-to-noise ratio. Unfortunately, stringent quality control is likely to result in samples that are less representative of the population in terms of age, sex, race, and socioeconomic status and remove meaningful variance in the data(
      • Chen J.
      • Tam A.
      • Kebets V.
      • Orban C.
      • Ooi L.Q.R.
      • Asplund C.L.
      • et al.
      Shared and unique brain network features predict cognitive, personality, and mental health scores in the ABCD study.
      ). Lower data quality may also be indicative of and associated with psychiatric illness. Hence, careful consideration must be given to arbitrary quantitative and qualitative data quality thresholds that may bias subsequent predictive modelling analyses.
      Relatedly, missing imaging and behavioral data is also a particular point of concern in clinical populations. Missing data may be a direct result of data entry errors, participant difficulty in completing data collection, attrition at follow-up, or an unintended consequence of stringent quality control(

      Thomas RM, Bruin W, Zhutovsky P, van Wingen G (2020): Dealing with missing data, small sample sizes, and heterogeneity in machine learning studies of brain disorders. Mach Learn: Elsevier, pp 249-266.

      ). Similar to quality control, missing data is more likely to be present in clinical populations, especially those suffering from severe psychiatric illness, and individuals from specific demographic groups that are likely already underrepresented. Although potential solutions for missing data imputation have been proposed, the field has not yet reached a consensus about best practices. Future research should incorporate lessons learned from decades of clinically-oriented research seeking to capture complex clinical outcomes using incomplete data.
      Global signal regression is a highly debated method used to remove the effects of global variations in activity from the functional time series of each voxel(
      • Murphy K.
      • Fox M.D.
      Towards a consensus regarding global signal regression for resting state functional connectivity MRI.
      ,
      • Liu T.T.
      • Nalci A.
      • Falahpour M.
      The global signal in fMRI: Nuisance or Information?.
      ). Most, although not all, behavioral predictions across cognitive, personality, and emotional domains based on intrinsic and task-based functional connectivity benefit from global signal regression(
      • Greene A.S.
      • Gao S.
      • Scheinost D.
      • Constable R.T.
      Task-induced brain state manipulation improves prediction of individual traits.
      ,
      • Li J.W.
      • Kong R.
      • Liegeois R.
      • Orban C.
      • Tan Y.R.
      • Sun N.B.
      • et al.
      Global signal regression strengthens association between resting-state functional connectivity and behavior.
      ).
      Another critical analytical choice for functional MRI addresses the parametrizations of the functional connectome. Full correlation is a standardized form of covariance. Partial correlation is derived using the inverse of the precision matrix and represents the correlation between a pair of brain parcels after adjusting for the time series of all other parcels. Furthermore, not every matrix is a valid correlation (or covariance) matrix. Tangent space representations allow the manipulation of the functional connectivity data consistent with the mathematical properties of correlation (or covariance) matrices(
      • Pervaiz U.
      • Vidaurre D.
      • Woolrich M.W.
      • Smith S.M.
      Optimising network modelling methods for fMRI.
      ). Some studies have suggested that compared with full and partial correlations, tangent space representations yield more accurate and reliable diagnostic classifications across distinct psychiatric illnesses(
      • Dadi K.
      • Rahim M.
      • Abraham A.
      • Chyzhyk D.
      • Milham M.
      • Thirion B.
      • et al.
      Benchmarking functional connectome-based predictive models for resting-state fMRI.
      ).
      Relatedly, white matter connectivity is defined using deterministic and probabilistic tractography algorithms(
      • Descoteaux M.
      • Deriche R.
      • Knosche T.R.
      • Anwander A.
      Deterministic and probabilistic tractography based on complex fibre orientation distributions.
      ,
      • Sarwar T.
      • Ramamohanarao K.
      • Zalesky A.
      Mapping connectomes with diffusion MRI: deterministic or probabilistic tractography?.
      ). Deterministic tractography relies on fixed orientations to direct streamlines while probabilistic algorithms estimate a distribution of fiber orientations and randomly draw a sample from this distribution to direct streamline propagation(
      • Sarwar T.
      • Ramamohanarao K.
      • Zalesky A.
      Mapping connectomes with diffusion MRI: deterministic or probabilistic tractography?.
      ). White matter connectivity derived using probabilistic tractography more accurately and reliably predict cognition than deterministic tractography(
      • Dhamala E.
      • Jamison K.W.
      • Jaywant A.
      • Dennis S.
      • Kuceyeski A.
      Distinct functional and structural connections predict crystallised and fluid cognition in healthy adults.
      ). Taken together, these results suggest that connectomic definitions can also bias behavioral predictions.
      Additionally, the use of anatomical features for predictive modelling involves crucial data transformation steps that can bias behavioral predictions. Individual differences in intracranial volume are typically considered to be confounds(
      • Buckner R.L.
      • Head D.
      • Parker J.
      • Fotenos A.F.
      • Marcus D.
      • Morris J.C.
      • et al.
      A unified approach for morphometric and functional data analysis in young, old, and demented adults using automated atlas-based head size normalization: reliability and validation against manual measurement of total intracranial volume.
      ) but proportional correction of those individual variations can differentially bias cognitive predictions based on cortical surface area, gray matter volume, and thickness(
      • Dhamala E.
      • Ooi L.Q.R.
      • Chen J.
      • Kong R.
      • Anderson K.M.
      • Chin R.
      • et al.
      Proportional intracranial volume correction differentially biases behavioral predictions across neuroanatomical features and populations.
      ). To better capture clinical deviations in brain structure and function, normative modelling can be used to model expected variations in a given neuroimaging feature(
      • Marquand A.F.
      • Rezek I.
      • Buitelaar J.
      • Beckmann C.F.
      Understanding heterogeneity in clinical cohorts using normative models: beyond case-control studies.
      ). Deviations from normative models of regional cortical volumes more accurately predict overall and dimension-specific psychopathology compared to raw measures of cortical volume(
      • Parkes L.
      • Moore T.M.
      • Calkins M.E.
      • Cook P.A.
      • Cieslak M.
      • Roalf D.R.
      • et al.
      Transdiagnostic dimensions of psychopathology explain individuals’ unique deviations from normative neurodevelopment in brain structure.
      ).
      Critically, global brain activation(
      • Yang G.J.
      • Murray J.D.
      • Repovs G.
      • Cole M.W.
      • Savic A.
      • Glasser M.F.
      • et al.
      Altered global brain signal in schizophrenia.
      ), intracranial volume(
      • Klein M.
      • Walters R.K.
      • Demontis D.
      • Stein J.L.
      • Hibar D.P.
      • Adams H.H.
      • et al.
      Genetic markers of ADHD-related variations in intracranial volume.
      ,
      • Ohi K.
      • Shimada T.
      • Kataoka Y.
      • Yasuyama T.
      • Kawasaki Y.
      • Shioiri T.
      • et al.
      Genetic correlations between subcortical brain volumes and psychiatric disorders.
      ), regional cortical volume(
      • Churchwell J.C.
      • Lopez-Larson M.
      • Yurgelun-Todd D.A.
      Altered frontal cortical volume and decision making in adolescent cannabis users.
      ,
      • Wang X.
      • Xie H.
      • Chen T.
      • Cotton A.S.
      • Salminen L.E.
      • Logue M.W.
      • et al.
      Cortical volume abnormalities in posttraumatic stress disorder: an ENIGMA-psychiatric genomics consortium PTSD workgroup mega-analysis.
      ,
      • Kuperberg G.R.
      • Broome M.R.
      • McGuire P.K.
      • David A.S.
      • Eddy M.
      • Ozawa F.
      • et al.
      Regionally localized thinning of the cerebral cortex in schizophrenia.
      ,
      • Makris N.
      • Goldstein J.M.
      • Kennedy D.
      • Hodge S.M.
      • Caviness V.S.
      • Faraone S.V.
      • et al.
      Decreased volume of left and total anterior insular lobule in schizophrenia.
      ) have been implicated in and may be indicative of psychiatric illness. Hence, it is crucial to account for the underlying, clinically-relevant signals that may be captured in distinct properties and consider how they should be handled when developing predictive models to study psychiatric populations.

      Phenotype Selection

      An unresolved debate pertains to the structure of psychiatric illnesses and whether researchers should focus on diagnoses, symptoms, or dimensional profiles(
      • Shorter E.
      • van Praag H.M.
      Disease versus dimension in diagnosis.
      ,
      • Waszczuk M.A.
      • Zimmerman M.
      • Ruggero C.
      • Li K.
      • MacNamara A.
      • Weinberg A.
      • et al.
      What do clinicians treat: Diagnoses or symptoms? The incremental validity of a symptom-based, dimensional characterization of emotional disorders in predicting medication prescription patterns.
      ,
      • Clark L.A.
      • Cuthbert B.
      • Lewis-Fernández R.
      • Narrow W.E.
      • Reed G.M.
      Three approaches to understanding and classifying mental disorder: ICD-11, DSM-5, and the National Institute of Mental Health’s Research Domain Criteria (RDoC).
      ). Psychiatric diagnoses rely on a clinician’s description of the syndrome’s nature, timespan, severity, and pre-existing risk factors to establish a categorical classification(
      • Heugten-Van Der Kloet V.
      • van Heugten T.
      The classification of psychiatric disorders according to DSM-5 deserves an internationally standardized psychological test battery on symptom level.
      ). These descriptions may be biased and the resulting diagnoses unreliable. In fact, two of the most common psychiatric illnesses, major depressive disorder and generalized anxiety disorder, have inter-rater reliabilities of 0.28 and 0.21, respectively(

      Freedman R, Lewis DA, Michels R, Pine DS, Schultz SK, Tamminga CA, et al. (2013): The initial field trials of DSM-5: new blooms and old thorns. Am Psychiatric Assoc, pp 1-5.

      ). As a result, clinically distinct phenotypic profiles can be assigned the same diagnosis. Remarkably, there are 636,120 distinct clinical presentations of post-traumatic stress disorder(
      • Galatzer-Levy I.R.
      • Bryant R.A.
      636,120 ways to have posttraumatic stress disorder.
      ). It is naïve to believe the same neurobiological properties will underlie the unique clinical presentations of a given diagnosis(
      • Saggar M.
      • Uddin L.Q.
      Pushing the boundaries of psychiatric neuroimaging to ground diagnosis in biology.
      ). Further impacting the clinical utility of brain-behavior models, clinical scales can vary substantially in terms of the symptoms and dimensions they capture, and single symptomatic or dimensional measures are insufficient to capture the complex nature and presentation of a psychiatric illness in an individual. These data raise many questions about which phenotypes or diagnoses we should aim to predict and whether those predictions will be biologically meaningful. Data scientists are familiar with the idea of “garbage in, garbage out”: if relying on poor phenotypic data and/or labels to train predictive models, the predictions and underlying associations captured are also likely of poor quality. Therefore, to maximize the accuracy and interpretability of prediction models in psychiatry, the variables being predicted must be reliable, validated, and representative of the overarching psychiatric illness or phenotype being studied in the specific population of interest.
      Directly addressing this pitfall, researchers have begun to develop frameworks to reconceptualize diagnostic criterion of psychiatric illnesses into quantifiable behavioral measures, such as the Extended Strengths and Weaknesses Assessment of Normal Behavior(

      Alexander LM, Salum GA, Swanson JM, Milham MP (2017): Balancing Strengths and Weaknesses in Dimensional Psychiatry. bioRxiv.207019.

      ). Moreover, the use passive digital data collection (i.e., from smartphones and wearable technology) along with better-distributed self-report measures is likely to yield more representative and robust quantifications of clinical behaviors. Overall, an improvement on the quantification of complex behaviors is crucial for the successful implementation of predictive models to capture relationships between those behaviors and neurobiological markers.
      Ongoing research also suggests certain illnesses, behaviors, or phenotypes may be more predictable than others. Intrinsic and task-based functional connectivity, and structural properties yield higher accuracies when predicting cognitive traits than personality or mental health traits(
      • Chen J.
      • Tam A.
      • Kebets V.
      • Orban C.
      • Ooi L.Q.R.
      • Asplund C.L.
      • et al.
      Shared and unique brain network features predict cognitive, personality, and mental health scores in the ABCD study.
      ,
      • Marek S.
      • Tervo-Clemmens B.
      • Calabro F.
      • Nichols T.
      • Dosenbach N.
      Reproducible brain-wide association studies require thousands of individuals.
      ). Within the cognitive domain, composite scores summarizing broad cognitive domains (
      • Carlozzi N.E.
      • Tulsky D.S.
      • Wolf T.J.
      • Goodnight S.
      • Heaton R.K.
      • Casaletto K.B.
      • et al.
      Construct validity of the NIH Toolbox Cognition Battery in individuals with stroke.
      ,
      • Heaton R.K.
      • Akshoomoff N.
      • Tulsky D.
      • Mungas D.
      • Weintraub S.
      • Dikmen S.
      • et al.
      Reliability and validity of composite scores from the NIH Toolbox Cognition Battery in adults.
      ,
      • Weintraub S.
      • Dikmen S.S.
      • Heaton R.K.
      • Tulsky D.S.
      • Zelazo P.D.
      • Slotkin J.
      • et al.
      The cognition battery of the NIH toolbox for assessment of neurological and behavioral function: validation in an adult sample.
      ), are more accurately predicted than individual task scores(
      • Dhamala E.
      • Jamison K.W.
      • Jaywant A.
      • Kuceyeski A.
      Shared functional connections within and between cortical networks predict cognitive abilities in adult males and females.
      ). Therefore, in addition to a variable being valid, reliable, and of clinical interest, its associations with neurobiological properties (and the underlying signal to noise ratio) must also be strong enough to be captured using multivariate approaches.
      Relatedly, evaluation of model generalizability across datasets can be tricky as they may collect different clinical scales and psychometric measures. Fortunately, recent attempts to characterize patient profiles using symptomatology and behavioral measures revealed that specific connectivity pattern underlie multiple phenotypic expressions(
      • Goldstein-Piekarski A.N.
      • Ball T.M.
      • Samara Z.
      • Staveland B.R.
      • Keller A.S.
      • Fleming S.L.
      • et al.
      Mapping neural circuit biotypes to symptoms and behavioral dimensions of depression and anxiety.
      ). While further research is needed in this area, these findings suggest that, in some instances, a single model may be generalizable across datasets or populations to predict different independent, but phenotypically correlated, measures.

      Brain Parcellations

      Machine learning analyses using neuroimaging data are prone to the curse of dimensionality. Imaging data at a voxel-level contains hundreds of thousands of features. Brain parcellations reduce dimensionality by segmenting the brain into tens or hundreds of regions or networks(
      • Eickhoff S.B.
      • Yeo B.
      • Genon S.
      Imaging-based parcellations of the human brain.
      ,

      Uddin LQ, Betzel RF, Cohen JR, Damoiseaux JS, De Brigard F, Eickhoff SB, et al. (2022): Controversies and current progress on large-scale brain network nomenclature from OHBM WHATNET: Workgroup for HArmonized Taxonomy of NETworks.

      ). These parcellations may be defined based on cytoarchitectural organization(
      • Amunts K.
      • Mohlberg H.
      • Bludau S.
      • Zilles K.
      Julich-Brain: A 3D probabilistic atlas of the human brain’s cytoarchitecture.
      ,
      • Eickhoff S.B.
      • Rottschy C.
      • Kujovic M.
      • Palomero-Gallagher N.
      • Zilles K.
      Organizational principles of human visual cortex revealed by receptor mapping.
      ,
      • Eickhoff S.B.
      • Stephan K.E.
      • Mohlberg H.
      • Grefkes C.
      • Fink G.R.
      • Amunts K.
      • et al.
      A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data.
      ), sulco-gyral localization(
      • Destrieux C.
      • Fischl B.
      • Dale A.
      • Halgren E.
      Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature.
      ,

      Talairach J (1988): Co-planar stereotaxic atlas of the human brain-3-dimensional proportional system. An approach to cerebral imaging.

      ,
      • Desikan R.S.
      • Segonne F.
      • Fischl B.
      • Quinn B.T.
      • Dickerson B.C.
      • Blacker D.
      • et al.
      An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest.
      ), data-driven modelling of intrinsic(
      • Schaefer A.
      • Kong R.
      • Gordon E.M.
      • Laumann T.O.
      • Zuo X.-N.
      • Holmes A.J.
      • et al.
      Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI.
      ,
      • Yeo B.T.
      • Krienen F.M.
      • Sepulcre J.
      • Sabuncu M.R.
      • Lashkari D.
      • Hollinshead M.
      • et al.
      The organization of the human cerebral cortex estimated by intrinsic functional connectivity.
      ,
      • Craddock C.R.
      • James G.A.
      • Holtzheimer 3rd, P.E.
      • Hu X.P.
      • Mayberg H.S.
      A whole brain fMRI atlas generated via spatially constrained spectral clustering.
      ,
      • Power J.D.
      • Cohen A.L.
      • Nelson S.M.
      • Wig G.S.
      • Barnes K.A.
      • Church J.A.
      • et al.
      Functional network organization of the human brain.
      ,
      • Dosenbach N.U.
      • Nardos B.
      • Cohen A.L.
      • Fair D.A.
      • Power J.D.
      • Church J.A.
      • et al.
      Prediction of individual brain maturity using fMRI.
      ,
      • Bellec P.
      • Rosa-Neto P.
      • Lyttelton O.C.
      • Benali H.
      • Evans A.C.
      Multi-level bootstrap analysis of stable clusters in resting-state fMRI.
      ) or task-evoked(
      • Smith S.M.
      • Fox P.T.
      • Miller K.L.
      • Glahn D.C.
      • Fox P.M.
      • Mackay C.E.
      • et al.
      Correspondence of the brain's functional architecture during activation and rest.
      ) functional activation signals, or independent component analyses(
      • Wu L.
      • Calhoun V.D.
      • Jung R.E.
      • Caprihan A.
      Connectivity‐based whole brain dual parcellation by group ICA reveals tract structures and decreased connectivity in schizophrenia.
      ,

      Ji Y, Hervé P-Y, Aickelin U, Pitiot A (2009): Parcellation of fmri datasets with ica and pls-a data driven approach. International Conference on Medical Image Computing and Computer-Assisted Intervention: Springer, pp 984-991.

      ). Dimensionality reduction through parcellation reduces noise, decreases computational complexity, and enhances interpretability(
      • Kong R.
      • Li J.
      • Orban C.
      • Sabuncu M.R.
      • Liu H.
      • Schaefer A.
      • et al.
      Spatial Topography of Individual-Specific Cortical Networks Predicts Human Cognition, Personality, and Emotion.
      ), but can also result in loss of biologically meaningful signals captured at a voxel-level and obscure valuable individual-level variations in brain organization(
      • Kong R.
      • Li J.
      • Orban C.
      • Sabuncu M.R.
      • Liu H.
      • Schaefer A.
      • et al.
      Spatial Topography of Individual-Specific Cortical Networks Predicts Human Cognition, Personality, and Emotion.
      ,
      • Thirion B.
      • Varoquaux G.
      • Dohmatob E.
      • Poline J.-B.
      Which fMRI clustering gives good brain parcellations?.
      ,
      • Kong R.
      • Yang Q.
      • Gordon E.
      • Xue A.
      • Yan X.
      • Orban C.
      • et al.
      Individual-Specific Areal-Level Parcellations Improve Functional Connectivity Prediction of Behavior.
      ). While some analyses have shown evidence that parcellation resolution and definition can affect predictive performance(
      • Dadi K.
      • Rahim M.
      • Abraham A.
      • Chyzhyk D.
      • Milham M.
      • Thirion B.
      • et al.
      Benchmarking functional connectome-based predictive models for resting-state fMRI.
      ,
      • Dhamala E.
      • Jamison K.W.
      • Jaywant A.
      • Dennis S.
      • Kuceyeski A.
      Distinct functional and structural connections predict crystallised and fluid cognition in healthy adults.
      ,

      Dhamala E, Jamison KW, Sabuncu MR, Kuceyeski A (2020): Sex classification using long-range temporal dependence of resting-state functional MRI time series. Human Brain Mapping. n/a.

      ,
      • Zeighami Y.
      • Evans A.C.
      Association vs. prediction: The impact of cortical surface smoothing and parcellation on brain age.
      ), others have reported generally consistent results across parcellations(
      • Mellema C.J.
      • Nguyen K.P.
      • Treacher A.
      • Montillo A.
      Reproducible neuroimaging features for diagnosis of autism spectrum disorder with machine learning.
      ).
      To eliminate parcellation biases, ensemble approaches integrating information across parcellations have been proposed(
      • Khosla M.
      • Jamison K.
      • Kuceyeski A.
      • Sabuncu M.R.
      Ensemble learning with 3D convolutional neural networks for functional connectome-based prediction.
      ,
      • Kalmady S.V.
      • Greiner R.
      • Agrawal R.
      • Shivakumar V.
      • Narayanaswamy J.C.
      • Brown M.R.
      • et al.
      Towards artificial intelligence in mental health by improving schizophrenia prediction with multiple brain parcellation ensemble-learning.
      ). Ensembling across pre-defined parcellations improved the classification of individuals with schizophrenia versus healthy controls based on intrinsic functional connectivity and regional activation properties compared to predictions based on a single parcellation(
      • Kalmady S.V.
      • Greiner R.
      • Agrawal R.
      • Shivakumar V.
      • Narayanaswamy J.C.
      • Brown M.R.
      • et al.
      Towards artificial intelligence in mental health by improving schizophrenia prediction with multiple brain parcellation ensemble-learning.
      ). Similarly, ensembling across pre-defined or stochastic parcellations yielded more accurate and robust classification of individuals with autism versus healthy controls based on intrinsic functional connectivity than predictions based on a single pre-defined parcellation(
      • Khosla M.
      • Jamison K.
      • Kuceyeski A.
      • Sabuncu M.R.
      Ensemble learning with 3D convolutional neural networks for functional connectome-based prediction.
      ).
      A second alternative involves individual-specific parcellations. These parcellations consider individual differences in spatial arrangement of the brain and are likely to be critical in psychiatric populations with known differences in network topography(
      • Kong R.
      • Li J.
      • Orban C.
      • Sabuncu M.R.
      • Liu H.
      • Schaefer A.
      • et al.
      Spatial Topography of Individual-Specific Cortical Networks Predicts Human Cognition, Personality, and Emotion.
      ,
      • Kong R.
      • Yang Q.
      • Gordon E.
      • Xue A.
      • Yan X.
      • Orban C.
      • et al.
      Individual-Specific Areal-Level Parcellations Improve Functional Connectivity Prediction of Behavior.
      ,
      • Gordon E.M.
      • Laumann T.O.
      • Gilmore A.W.
      • Newbold D.J.
      • Greene D.J.
      • Berg J.J.
      • et al.
      Precision functional mapping of individual human brains.
      ,
      • Gratton C.
      • Laumann T.O.
      • Nielsen A.N.
      • Greene D.J.
      • Gordon E.M.
      • Gilmore A.W.
      • et al.
      Functional brain networks are dominated by stable group and individual factors, not cognitive or daily variation.
      ). These individual-specific network topography patterns have demonstrated improved accuracy for behavioral predictions compared to population-level parcellations(
      • Kong R.
      • Li J.
      • Orban C.
      • Sabuncu M.R.
      • Liu H.
      • Schaefer A.
      • et al.
      Spatial Topography of Individual-Specific Cortical Networks Predicts Human Cognition, Personality, and Emotion.
      ).

      Sample Sizes and Transfer Learning

      Historically, psychiatric neuroimaging studies relied on as few as tens of subjects to characterize interactions between neurobiology and diagnoses/phenotypic presentations(
      • Szucs D.
      • Ioannidis J.P.
      Sample size evolution in neuroimaging research: An evaluation of highly-cited studies (1990–2012) and of latest practices (2017–2018) in high-impact journals.
      ). A core issue when applying machine learning to neuroimaging is sample sizes. Generally, machine learning algorithms require sample sizes larger than the dimensionality of the feature set to prevent overfitting(
      • Janssen R.J.
      • Mourão-Miranda J.
      • Schnack H.G.
      Making individual prognoses in psychiatry using neuroimaging and machine learning.
      ). Over the last decade, sample sizes have increased to hundreds and even thousands. Limited sample sizes result in decreased reliability of predictions regardless of algorithm, feature type, and phenotype(
      • Marek S.
      • Tervo-Clemmens B.
      • Calabro F.
      • Nichols T.
      • Dosenbach N.
      Reproducible brain-wide association studies require thousands of individuals.
      ,
      • Cui Z.
      • Gong G.
      The effect of machine learning regression algorithms and sample size on individualized behavioral prediction with functional connectivity features.
      ,
      • Jollans L.
      • Boyle R.
      • Artiges E.
      • Banaschewski T.
      • Desrivieres S.
      • Grigis A.
      • et al.
      Quantifying performance of machine learning methods for neuroimaging data.
      ,
      • Varoquaux G.
      Cross-validation failure: Small sample sizes lead to large error bars.
      ), and produce overly optimistic estimates of prediction accuracy(
      • Janssen R.J.
      • Mourão-Miranda J.
      • Schnack H.G.
      Making individual prognoses in psychiatry using neuroimaging and machine learning.
      ,
      • Santos M.S.
      • Soares J.P.
      • Abreu P.H.
      • Araujo H.
      • Santos J.
      Cross-Validation for Imbalanced Datasets: Avoiding Overoptimistic and Overfitting Approaches.
      ). These issues are of particular concern in heterogenous populations, such as individuals with psychiatric illnesses, where small datasets will likely produce models that cannot generalize to independent test samples and thus lack clinical utility. In fact, there appears to be a strong negative relationship between accuracy and sample size for diagnostic and behavioral predictions in Alzheimer’s, depression, schizophrenia, psychosis, and autism(
      • Janssen R.J.
      • Mourão-Miranda J.
      • Schnack H.G.
      Making individual prognoses in psychiatry using neuroimaging and machine learning.
      ,
      • Varoquaux G.
      Cross-validation failure: Small sample sizes lead to large error bars.
      ). However, large clinical samples can be difficult to acquire.
      Fortunately, there is evidence to suggest that brain-behavior predictive models developed in the general population can boost predictions of new phenotypes in smaller independent datasets. A “meta-matching” framework developed to translate predictive models based on neuroimaging features from large samples into non-brain imaging features in smaller samples has shown promising results when generalizing from aging to young adult populations(
      • He T.
      • An L.
      • Chen P.
      • Chen J.
      • Feng J.
      • Bzdok D.
      • et al.
      Meta-matching as a simple framework to translate phenotypic predictive models from big to small data.
      ). Connectome-based predictive models of attention have also demonstrated the capacity to generalize across healthy and clinical populations(
      • Yoo K.
      • Rosenberg M.D.
      • Hsu W.-T.
      • Zhang S.
      • Li C.-S.R.
      • Scheinost D.
      • et al.
      Connectome-based predictive modeling of attention: Comparing different functional connectivity features and prediction methods across datasets.
      ,
      • Rosenberg M.D.
      • Hsu W.-T.
      • Scheinost D.
      • Todd Constable R.
      • Chun M.M.
      Connectome-based models predict separable components of attention in novel individuals.
      ,
      • Rosenberg M.D.
      • Finn E.S.
      • Scheinost D.
      • Papademetris X.
      • Shen X.
      • Constable R.T.
      • et al.
      A neuromarker of sustained attention from whole-brain functional connectivity.
      ), while models of memory have successfully generalized across psychiatric illnesses(
      • Barron D.S.
      • Gao S.
      • Dadashkarimi J.
      • Greene A.S.
      • Spann M.N.
      • Noble S.
      • et al.
      Transdiagnostic, connectome-based prediction of memory constructs across psychiatric disorders.
      ). Translational applications of these predictive frameworks provide the opportunity to leverage information about brain-behavior relationships in the general population or within a specific clinical population to better understand psychiatric presentation. This can drastically reduce the clinical sample size needed to yield accurate and reliable predictions.

      Population-Specific Models

      The rise of machine learning applications in neuroscience and psychiatry has begun to expose how (lack of) demographic representation in datasets and algorithmic biases can inadvertently introduce biases in behavioral predictions for specific populations. These discrepancies cannot be entirely attributed to under-representation in datasets or unequal sample sizes and may also be tied to differences in how accurately and reliably neuroimaging and behavioral measures are able to capture phenotypes across populations(

      Ricard JA, Parker TC, Dhamala E, Kwasa J, Allsop AS, Holmes AJ (2022): Confronting racist and exclusionary practices in the acquisition and analyses of neuroimaging data. Submitted.

      ). Evidence suggests that behavioral prediction models trained on datasets dominated by White Americans may fail to generalize in African American populations and the underlying brain-behavior associations captured by the models are specifically representative of White Americans(
      • Li J.
      • Bzdok D.
      • Chen J.
      • Tam A.
      • Ooi L.Q.R.
      • Holmes A.J.
      • et al.
      Cross-ethnicity/race generalization failure of behavioral prediction from resting-state functional connectivity.
      ). Similarly, predictions of cognitive and personality traits can fail to generalize across sexes(
      • Dhamala E.
      • Jamison K.W.
      • Jaywant A.
      • Kuceyeski A.
      Shared functional connections within and between cortical networks predict cognitive abilities in adult males and females.
      ,
      • Jiang R.
      • Calhoun V.D.
      • Cui Y.
      • Qi S.
      • Zhuo C.
      • Li J.
      • et al.
      Multimodal data revealed different neurobiological correlates of intelligence between males and females.
      ,
      • Jiang R.
      • Calhoun V.D.
      • Fan L.
      • Zuo N.
      • Jung R.
      • Qi S.
      • et al.
      Gender differences in connectome-based predictions of individualized intelligence quotient and sub-domain scores.
      ,
      • Nostro A.D.
      • Müller V.I.
      • Varikuti D.P.
      • Pläschke R.N.
      • Hoffstaedter F.
      • Langner R.
      • et al.
      Predicting personality from network-based resting-state functional connectivity.
      ). Distinct demographic groups (i.e., age groups, sexes, races/ethnicities) often exhibit differences in psychiatric presentations including prevalence, symptom profile, and treatment response. Traditional neuroimaging analyses relied on covariate regression to minimize effects of sex, age, and/or other demographic or data collection dependent variables (i.e., site). However, common methods to remove covariate information may not sufficiently account for their effects(
      • Dhamala E.
      • Ooi L.Q.R.
      • Chen J.
      • Kong R.
      • Anderson K.M.
      • Chin R.
      • et al.
      Proportional intracranial volume correction differentially biases behavioral predictions across neuroanatomical features and populations.
      ,
      • Sanchis-Segura C.
      • Ibañez-Gual M.V.
      • Aguirre N.
      • Forn C.
      Cruz-Gómez ÁJ
      Effects of different intracranial volume correction methods on univariate sex differences in grey matter volume and multivariate sex prediction.
      ) and are likely to result in a loss of meaningful variance in the data. Covariate regression, if not performed properly within nested designs, can also lead to information leakage, and artifactually inflate model performance and effect sizes. Alternatively, the use of population-specific prediction models can provide insight into how brain-behavior relationships vary across different populations.
      While general predictions can capture shared neural correlates of different clinically-relevant behaviors, population-specific models can provide complementary information about brain-behavior relationships specific to different groups. Recent work has shown that general prediction models often fail in individuals who do not fit a sample population’s stereotypical profile for brain-behavior relationships(
      • Greene A.S.
      • Shen X.
      • Noble S.
      • Horien C.
      • Hahn C.A.
      • Arora J.
      • et al.
      Brain–phenotype models fail for individuals who defy sample stereotypes.
      ). Demonstrating the value of population-specific predictive modelling, several groups have shown that sex-specific predictions of cognition and personality are more accurate and insightful than sex-independent ones(
      • Greene A.S.
      • Gao S.
      • Scheinost D.
      • Constable R.T.
      Task-induced brain state manipulation improves prediction of individual traits.
      ,
      • Jiang R.
      • Calhoun V.D.
      • Cui Y.
      • Qi S.
      • Zhuo C.
      • Li J.
      • et al.
      Multimodal data revealed different neurobiological correlates of intelligence between males and females.
      ,
      • Jiang R.
      • Calhoun V.D.
      • Fan L.
      • Zuo N.
      • Jung R.
      • Qi S.
      • et al.
      Gender differences in connectome-based predictions of individualized intelligence quotient and sub-domain scores.
      ,
      • Nostro A.D.
      • Müller V.I.
      • Varikuti D.P.
      • Pläschke R.N.
      • Hoffstaedter F.
      • Langner R.
      • et al.
      Predicting personality from network-based resting-state functional connectivity.
      ). Specifically, these models reveal the shared and unique correlates of behaviors in males and females. Together, these data emphasize the clinical utility of population-specific predictive models. Given the well-established sex differences in prevalence, symptomatology, and clinical trajectory in psychiatric illnesses, sex-specific predictive models will be especially informative in distinct clinical populations. Similarly, precision psychiatry will likely also benefit from models specific to other demographic and clinical variables that can drive differences in disease presentation and treatment seeking behaviors.
      A crucial factor to acknowledge when developing population-specific models is intersectionality. The overlapping and interdependent nature of how distinct demographic and social identities (i.e., age, sex, race, ethnicity, sexuality, physical health, psychiatric diagnoses, socioeconomic status, religion) may intersect is extremely complex and individuals suffering from psychiatric illnesses may be members of several different marginalized groups(
      • Oexle N.
      • Corrigan P.W.
      Understanding mental illness stigma toward persons with multiple stigmatized conditions: Implications of intersectionality theory.
      ). Other variables pertaining to data collection (i.e., data quality, site) are likely to also be closely intertwined with participants’ intersectional identities. Hence, when developing population-specific models, researchers must carefully consider how models may generalize across intersectional populations. Models specific to one population may not comparably generalize across other demographic and social groups, but they may still reveal brain-behavior associations specific to that group that can aid in clinical decision-making. Moreover, ensembling predictions across models developed across distinct populations that an individual belongs to may also provide clinically-relevant insights.

      Clinical Utility and Future Directions

      An ongoing debate in the field seeks to establish whether predictive models should prioritize accuracy, generalizability, or interpretability. Existing literature suggests a wide range of modelling choices can be leveraged to yield clinically-informative predictions. Models with high accuracy and interpretability but low generalizability might offer important insights into brain-behavior relationships in clinical populations but lack clinical utility. Conversely, models with high accuracy and generalizability but low interpretability may have high clinical utility but are unable to provide clear explanations that can aid clinical decision-making. Finally, while models with high interpretability and generalizability but low accuracy are extremely unlikely by definition, they can produce mechanistic insights into the neurobiological features underlying a phenotype. Moving forward, the field of precision psychiatry must incorporate interdisciplinary researchers and carefully consider how methodological choices can be used to develop machine learning models that are in line with their broader analytical goals.
      Brain-based predictive modelling in psychiatry and neuroscience has led to monumental advances in our understanding of the neurobiological correlates of psychiatric illness. In the next decade, we expect continued growth in this area of research. Two particular areas of focus crucial for significant advances include scale development and clinical evaluation. First, the development and implementation of behavioral scales that accurately, reliably, and robustly capture individual differences across distinct clinically-relevant behaviors is fundamental. Brain-based behavioral prediction models can only be successful if the behaviors they are optimized to capture are correctly measured. Second, predictive and prescriptive analyses must be prospectively clinically evaluated in appropriately pre-registered randomized controlled clinical trials. A clinical evaluation of these approaches is necessary before we can actualize and implement predictive models in psychiatry.

      Conclusion

      Historical advances in our understanding of neurobiological correlates of psychiatry have paralleled technological advances enabling the visualization and quantification of human brain functions. Future advances can exploit multivariate analytical tools to model accurate and interpretable relationships between brain and behavior at an individual-level. Machine learning with neuroimaging data provides a framework that can support innovations in precision neuropsychiatry. However, the development of predictive models to facilitate accurate diagnosis, informative prognosis, and impactful treatment must consider methodological choices that can influence model performance. Here, we review how different factors can affect both accuracy and interpretability of these models. An understanding of these effects is crucial for the mindful implementation of choices that are most optimal for a specific research goal. Proper development and deployment of these approaches will yield significant improvements in our understanding of psychiatric illness and improve the lives of millions of individuals suffering from them.

      Acknowledgements

      This work was supported by the National Institute of Mental Health (R01MH120080 and R01MH123245 to A.J.H.) and the Kavli Institute for Neuroscience at Yale University (Postdoctoral Fellowship for Academic Diversity to E.D.). This work was also supported by the following awards to B.T.T.Y.: the Singapore National Research Foundation (NRF) Fellowship (Class of 2017), the NUS Yong Loo Lin School of Medicine (NUHSRO/2020/124/TMR/LOA), the Singapore National Medical Research Council (NMRC) LCG (OFLCG19May-0035), and the NMRC STaR (STaR20nov-0003). Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the Singapore NRF or the Singapore NMRC.
      Financial Disclosures
      Drs. Dhamala, Yeo, and Holmes reported no biomedical financial interests or potential conflicts of interest.

      References

      1. Council NR (2011): Toward precision medicine: building a knowledge network for biomedical research and a new taxonomy of disease.

        • Dugger S.A.
        • Platt A.
        • Goldstein D.B.
        Drug development in the era of precision medicine.
        Nature reviews Drug discovery. 2018; 17: 183-196
      2. Vieta E (2015): Personalised medicine applied to mental health: Precision psychiatry. pp 117-118.

        • Hyman S.E.
        The diagnosis of mental disorders: the problem of reification.
        Annual review of clinical psychology. 2010; 6: 155-179
        • Yip S.W.
        • Kiluk B.
        • Scheinost D.
        Toward addiction prediction: an overview of cross-validated predictive modeling findings and considerations for future neuroimaging research.
        Biological Psychiatry: Cognitive Neuroscience and Neuroimaging. 2020; 5: 748-758
        • Scheinost D.
        • Noble S.
        • Horien C.
        • Greene A.S.
        • Lake E.M.
        • Salehi M.
        • et al.
        Ten simple rules for predictive modeling of individual differences in neuroimaging.
        NeuroImage. 2019; 193: 35-45
        • Rosenberg M.D.
        • Casey B.
        • Holmes A.J.
        Prediction complements explanation in understanding the developing brain.
        Nature communications. 2018; 9: 1-13
        • Varoquaux G.
        • Raamana P.R.
        • Engemann D.A.
        • Hoyos-Idrobo A.
        • Schwartz Y.
        • Thirion B.
        Assessing and tuning brain decoders: cross-validation, caveats, and guidelines.
        NeuroImage. 2017; 145: 166-179
        • Woo C.-W.
        • Chang L.J.
        • Lindquist M.A.
        • Wager T.D.
        Building better biomarkers: brain models in translational neuroimaging.
        Nature neuroscience. 2017; 20: 365-377
        • Arbabshirani M.R.
        • Plis S.
        • Sui J.
        • Calhoun V.D.
        Single subject prediction of brain disorders in neuroimaging: Promises and pitfalls.
        Neuroimage. 2017; 145: 137-165
      3. Khosla M, Jamison K, Ngo GH, Kuceyeski A, Sabuncu MR (2019): Machine learning in resting-state fMRI analysis. Magn Reson Imaging.

        • Shen X.
        • Finn E.S.
        • Scheinost D.
        • Rosenberg M.D.
        • Chun M.M.
        • Papademetris X.
        • et al.
        Using connectome-based predictive modeling to predict individual behavior from brain connectivity.
        nature protocols. 2017; 12: 506-518
        • Kawahara J.
        • Brown C.J.
        • Miller S.P.
        • Booth B.G.
        • Chau V.
        • Grunau R.E.
        • et al.
        BrainNetCNN: Convolutional neural networks for brain networks; towards predicting neurodevelopment.
        Neuroimage. 2017; 146: 1038-1049
        • Yoo K.
        • Rosenberg M.D.
        • Hsu W.-T.
        • Zhang S.
        • Li C.-S.R.
        • Scheinost D.
        • et al.
        Connectome-based predictive modeling of attention: Comparing different functional connectivity features and prediction methods across datasets.
        Neuroimage. 2018; 167: 11-22
        • Rosenberg M.D.
        • Hsu W.-T.
        • Scheinost D.
        • Todd Constable R.
        • Chun M.M.
        Connectome-based models predict separable components of attention in novel individuals.
        Journal of Cognitive Neuroscience. 2018; 30: 160-173
        • Barron D.S.
        • Gao S.
        • Dadashkarimi J.
        • Greene A.S.
        • Spann M.N.
        • Noble S.
        • et al.
        Transdiagnostic, connectome-based prediction of memory constructs across psychiatric disorders.
        Cerebral Cortex. 2021; 31: 2523-2533
        • Wang Z.
        • Goerlich K.S.
        • Ai H.
        • Aleman A.
        • Luo Y-j
        • Xu P.
        Connectome-based predictive modeling of individual anxiety.
        Cerebral Cortex. 2021; 31: 3006-3020
        • Ju Y.
        • Horien C.
        • Chen W.
        • Guo W.
        • Lu X.
        • Sun J.
        • et al.
        Connectome-based models can predict early symptom improvement in major depressive disorder.
        Journal of affective disorders. 2020; 273: 442-452
        • Yip S.W.
        • Scheinost D.
        • Potenza M.N.
        • Carroll K.M.
        Connectome-based prediction of cocaine abstinence.
        American Journal of Psychiatry. 2019; 176: 156-164
        • Lichenstein S.D.
        • Scheinost D.
        • Potenza M.N.
        • Carroll K.M.
        • Yip S.W.
        Dissociable neural substrates of opioid and cocaine use identified via connectome-based modelling.
        Molecular psychiatry. 2021; 26: 4383-4393
        • He T.
        • Kong R.
        • Holmes A.J.
        • Nguyen M.
        • Sabuncu M.R.
        • Eickhoff S.B.
        • et al.
        Deep neural networks and kernel regression achieve comparable accuracies for functional connectivity prediction of behavior and demographics.
        NeuroImage. 2020; 206116276
        • Pervaiz U.
        • Vidaurre D.
        • Woolrich M.W.
        • Smith S.M.
        Optimising network modelling methods for fMRI.
        Neuroimage. 2020; 211116604
        • Abrol A.
        • Fu Z.
        • Salman M.
        • Silva R.
        • Du Y.
        • Plis S.
        • et al.
        Deep learning encodes robust discriminative neuroimaging representations to outperform standard machine learning.
        Nature communications. 2021; 12: 1-17
        • Mellema C.J.
        • Nguyen K.P.
        • Treacher A.
        • Montillo A.
        Reproducible neuroimaging features for diagnosis of autism spectrum disorder with machine learning.
        Scientific Reports. 2022; 12: 1-13
      4. Traut N, Heuer K, Lemaître G, Beggiato A, Germanaud D, Elmaleh M, et al. (2022): Insights from an autism imaging biomarker challenge: promises and threats to biomarker discovery. NeuroImage.119171.

        • Dadi K.
        • Rahim M.
        • Abraham A.
        • Chyzhyk D.
        • Milham M.
        • Thirion B.
        • et al.
        Benchmarking functional connectome-based predictive models for resting-state fMRI.
        NeuroImage. 2019; 192: 115-134
        • Dhamala E.
        • Jamison K.W.
        • Jaywant A.
        • Dennis S.
        • Kuceyeski A.
        Distinct functional and structural connections predict crystallised and fluid cognition in healthy adults.
        Hum Brain Mapp. 2021; 42: 3102-3118
        • Mansour S.L.
        • Tian Y.
        • Yeo B.T.
        • Cropley V.
        • Zalesky A.
        High-resolution connectomic fingerprints: Mapping neural identity and behavior.
        NeuroImage. 2021; 229117695
        • Ooi L.Q.R.
        • Chen J.
        • Zhang S.
        • Kong R.
        • Li J.
        • Dhamala E.
        • et al.
        Comparison of individualized behavioral predictions across anatomical, diffusion and functional connectivity MRI.
        BioRxiv. 2022;
        • Mellem M.S.
        • Liu Y.
        • Gonzalez H.
        • Kollada M.
        • Martin W.J.
        • Ahammad P.
        Machine learning models identify multimodal measurements highly predictive of transdiagnostic symptom severity for mood, anhedonia, and anxiety.
        Biological Psychiatry: Cognitive Neuroscience and Neuroimaging. 2020; 5: 56-67
        • Seguin C.
        • Tian Y.
        • Zalesky A.
        Network communication models improve the behavioral and functional predictive utility of the human structural connectome.
        Network Neuroscience. 2020; 4: 980-1006
        • Sabuncu M.R.
        • Konukoglu E.
        • AsDN Initiative
        Clinical prediction from structural brain MRI scans: a large-scale empirical study.
        Neuroinformatics. 2015; 13: 31-46
        • Talpalaru A.
        • Bhagwat N.
        • Devenyi G.A.
        • Lepage M.
        • Chakravarty M.M.
        Identifying schizophrenia subgroups using clustering and supervised learning.
        Schizophr Res. 2019; 214: 51-59
        • Koutsouleris N.
        • Wobrock T.
        • Guse B.
        • Langguth B.
        • Landgrebe M.
        • Eichhammer P.
        • et al.
        Predicting response to repetitive transcranial magnetic stimulation in patients with schizophrenia using structural magnetic resonance imaging: a multisite machine learning analysis.
        Schizophrenia Bulletin. 2018; 44: 1021-1034
        • Johnston B.A.
        • Steele J.D.
        • Tolomeo S.
        • Christmas D.
        • Matthews K.
        Structural MRI-based predictions in patients with treatment-refractory depression (TRD).
        PloS one. 2015; 10e0132958
        • Whitfield-Gabrieli S.
        • Ghosh S.
        • Nieto-Castanon A.
        • Saygin Z.
        • Doehrmann O.
        • Chai X.
        • et al.
        Brain connectomics predict response to treatment in social anxiety disorder.
        Molecular psychiatry. 2016; 21: 680-685
        • Johnson K.A.
        • Duffley G.
        • Anderson D.N.
        • Ostrem J.L.
        • Welter M.-L.
        • Baldermann J.C.
        • et al.
        Structural connectivity predicts clinical outcomes of deep brain stimulation for Tourette syndrome.
        Brain. 2020; 143: 2607-2623
        • Shao J.
        • Myers N.
        • Yang Q.
        • Feng J.
        • Plant C.
        • Böhm C.
        • et al.
        Prediction of Alzheimer's disease using individual structural connectivity networks.
        Neurobiology of aging. 2012; 33: 2756-2765
        • Greene A.S.
        • Gao S.
        • Scheinost D.
        • Constable R.T.
        Task-induced brain state manipulation improves prediction of individual traits.
        Nature communications. 2018; 9: 1-13
        • Chen J.
        • Tam A.
        • Kebets V.
        • Orban C.
        • Ooi L.Q.R.
        • Asplund C.L.
        • et al.
        Shared and unique brain network features predict cognitive, personality, and mental health scores in the ABCD study.
        Nat Commun. 2022; 13: 2217
        • Sarpal D.K.
        • Argyelan M.
        • Robinson D.G.
        • Szeszko P.R.
        • Karlsgodt K.H.
        • John M.
        • et al.
        Baseline striatal functional connectivity as a predictor of response to antipsychotic drug treatment.
        American Journal of Psychiatry. 2016; 173: 69-77
        • Du Y.
        • Fu Z.
        • Sui J.
        • Gao S.
        • Xing Y.
        • Lin D.
        • et al.
        NeuroMark: An automated and adaptive ICA based pipeline to identify reproducible fMRI markers of brain disorders.
        NeuroImage: Clinical. 2020; 28102375
        • Pantazatos S.P.
        • Talati A.
        • Schneier F.R.
        • Hirsch J.
        Reduced anterior temporal and hippocampal functional connectivity during face processing discriminates individuals with social anxiety disorder from healthy controls and panic disorder, and increases following treatment.
        Neuropsychopharmacology. 2014; 39: 425-434
        • Calhoun V.D.
        • Maciejewski P.K.
        • Pearlson G.D.
        • Kiehl K.A.
        Temporal lobe and “default” hemodynamic brain modes discriminate between schizophrenia and bipolar disorder.
        Human brain mapping. 2008; 29: 1265-1275
        • Cha J.
        • Speaker S.
        • Hu B.
        • Altinay M.
        • Koirala P.
        • Karne H.
        • et al.
        Neuroimaging correlates of emotional response-inhibition discriminate between young depressed adults with and without sub-threshold bipolar symptoms (Emotional Response-inhibition in Young Depressed Adults).
        Journal of affective disorders. 2021; 281: 303-311
        • Doehrmann O.
        • Ghosh S.S.
        • Polli F.E.
        • Reynolds G.O.
        • Horn F.
        • Keshavan A.
        • et al.
        Predicting treatment response in social anxiety disorder from functional magnetic resonance imaging.
        JAMA psychiatry. 2013; 70: 87-97
        • Hahn T.
        • Kircher T.
        • Straube B.
        • Wittchen H.-U.
        • Konrad C.
        • Ströhle A.
        • et al.
        Predicting treatment response to cognitive behavioral therapy in panic disorder with agoraphobia by integrating local neural information.
        JAMA psychiatry. 2015; 72: 68-74
        • Greene D.J.
        • Black K.J.
        • Schlaggar B.L.
        Considerations for MRI study design and implementation in pediatric and clinical populations.
        Developmental cognitive neuroscience. 2016; 18: 101-112
        • Power J.D.
        • Barnes K.A.
        • Snyder A.Z.
        • Schlaggar B.L.
        • Petersen S.E.
        Spurious but systematic correlations in functional connectivity MRI networks arise from subject motion.
        Neuroimage. 2012; 59: 2142-2154
        • Power J.D.
        • Mitra A.
        • Laumann T.O.
        • Snyder A.Z.
        • Schlaggar B.L.
        • Petersen S.E.
        Methods to detect, characterize, and remove motion artifact in resting state fMRI.
        Neuroimage. 2014; 84: 320-341
        • Power J.D.
        • Schlaggar B.L.
        • Petersen S.E.
        Recent progress and outstanding issues in motion correction in resting state fMRI.
        Neuroimage. 2015; 105: 536-551
        • Tavor I.
        • Jones O.P.
        • Mars R.B.
        • Smith S.
        • Behrens T.
        • Jbabdi S.
        Task-free MRI predicts individual differences in brain activity during task performance.
        Science. 2016; 352: 216-220
        • Gruskin D.C.
        • Patel G.H.
        Brain connectivity at rest predicts individual differences in normative activity during movie watching.
        NeuroImage. 2022; 253119100
      5. Gal S, Tik N, Bernstein-Eliav M, Tavor I (2022): Predicting Individual Traits from Unperformed Tasks. NeuroImage.118920.

        • Botvinik-Nezer R.
        • Holzmeister F.
        • Camerer C.F.
        • Dreber A.
        • Huber J.
        • Johannesson M.
        • et al.
        Variability in the analysis of a single neuroimaging dataset by many teams.
        Nature. 2020; 582: 84-88
        • Bhagwat N.
        • Barry A.
        • Dickie E.W.
        • Brown S.T.
        • Devenyi G.A.
        • Hatano K.
        • et al.
        Understanding the impact of preprocessing pipelines on neuroimaging cortical surface analyses.
        GigaScience. 2021; 10 (giaa155)
        • Bowring A.
        • Nichols T.E.
        • Maumet C.
        Isolating the sources of pipeline‐variability in group‐level task‐fMRI results.
        Human brain mapping. 2022; 43: 1112-1128
        • Bowring A.
        • Maumet C.
        • Nichols T.E.
        Exploring the impact of analysis software on task fMRI results.
        Human brain mapping. 2019; 40: 3362-3384
      6. Thomas RM, Bruin W, Zhutovsky P, van Wingen G (2020): Dealing with missing data, small sample sizes, and heterogeneity in machine learning studies of brain disorders. Mach Learn: Elsevier, pp 249-266.

        • Murphy K.
        • Fox M.D.
        Towards a consensus regarding global signal regression for resting state functional connectivity MRI.
        Neuroimage. 2017; 154: 169-173
        • Liu T.T.
        • Nalci A.
        • Falahpour M.
        The global signal in fMRI: Nuisance or Information?.
        Neuroimage. 2017; 150: 213-229
        • Li J.W.
        • Kong R.
        • Liegeois R.
        • Orban C.
        • Tan Y.R.
        • Sun N.B.
        • et al.
        Global signal regression strengthens association between resting-state functional connectivity and behavior.
        Neuroimage. 2019; 196: 126-141
        • Descoteaux M.
        • Deriche R.
        • Knosche T.R.
        • Anwander A.
        Deterministic and probabilistic tractography based on complex fibre orientation distributions.
        IEEE transactions on medical imaging. 2008; 28: 269-286
        • Sarwar T.
        • Ramamohanarao K.
        • Zalesky A.
        Mapping connectomes with diffusion MRI: deterministic or probabilistic tractography?.
        Magnetic resonance in medicine. 2019; 81: 1368-1384
        • Buckner R.L.
        • Head D.
        • Parker J.
        • Fotenos A.F.
        • Marcus D.
        • Morris J.C.
        • et al.
        A unified approach for morphometric and functional data analysis in young, old, and demented adults using automated atlas-based head size normalization: reliability and validation against manual measurement of total intracranial volume.
        Neuroimage. 2004; 23: 724-738
        • Dhamala E.
        • Ooi L.Q.R.
        • Chen J.
        • Kong R.
        • Anderson K.M.
        • Chin R.
        • et al.
        Proportional intracranial volume correction differentially biases behavioral predictions across neuroanatomical features and populations.
        NeuroImage. 2022;
        • Marquand A.F.
        • Rezek I.
        • Buitelaar J.
        • Beckmann C.F.
        Understanding heterogeneity in clinical cohorts using normative models: beyond case-control studies.
        Biol Psychiat. 2016; 80: 552-561
        • Parkes L.
        • Moore T.M.
        • Calkins M.E.
        • Cook P.A.
        • Cieslak M.
        • Roalf D.R.
        • et al.
        Transdiagnostic dimensions of psychopathology explain individuals’ unique deviations from normative neurodevelopment in brain structure.
        Translational psychiatry. 2021; 11: 1-13
        • Yang G.J.
        • Murray J.D.
        • Repovs G.
        • Cole M.W.
        • Savic A.
        • Glasser M.F.
        • et al.
        Altered global brain signal in schizophrenia.
        Proceedings of the National Academy of Sciences. 2014; 111: 7438-7443
        • Klein M.
        • Walters R.K.
        • Demontis D.
        • Stein J.L.
        • Hibar D.P.
        • Adams H.H.
        • et al.
        Genetic markers of ADHD-related variations in intracranial volume.
        American Journal of Psychiatry. 2019; 176: 228-238
        • Ohi K.
        • Shimada T.
        • Kataoka Y.
        • Yasuyama T.
        • Kawasaki Y.
        • Shioiri T.
        • et al.
        Genetic correlations between subcortical brain volumes and psychiatric disorders.
        The British Journal of Psychiatry. 2020; 216: 280-283
        • Churchwell J.C.
        • Lopez-Larson M.
        • Yurgelun-Todd D.A.
        Altered frontal cortical volume and decision making in adolescent cannabis users.
        Frontiers in Psychology. 2010; 1: 225
        • Wang X.
        • Xie H.
        • Chen T.
        • Cotton A.S.
        • Salminen L.E.
        • Logue M.W.
        • et al.
        Cortical volume abnormalities in posttraumatic stress disorder: an ENIGMA-psychiatric genomics consortium PTSD workgroup mega-analysis.
        Molecular psychiatry. 2021; 26: 4331-4343
        • Kuperberg G.R.
        • Broome M.R.
        • McGuire P.K.
        • David A.S.
        • Eddy M.
        • Ozawa F.
        • et al.
        Regionally localized thinning of the cerebral cortex in schizophrenia.
        Arch Gen Psychiatry. 2003; 60: 878-888
        • Makris N.
        • Goldstein J.M.
        • Kennedy D.
        • Hodge S.M.
        • Caviness V.S.
        • Faraone S.V.
        • et al.
        Decreased volume of left and total anterior insular lobule in schizophrenia.
        Schizophr Res. 2006; 83: 155-171
        • Shorter E.
        • van Praag H.M.
        Disease versus dimension in diagnosis.
        The Canadian Journal of Psychiatry. 2010; 55: 59-64
        • Waszczuk M.A.
        • Zimmerman M.
        • Ruggero C.
        • Li K.
        • MacNamara A.
        • Weinberg A.
        • et al.
        What do clinicians treat: Diagnoses or symptoms? The incremental validity of a symptom-based, dimensional characterization of emotional disorders in predicting medication prescription patterns.
        Comprehensive Psychiatry. 2017; 79: 80-88
        • Clark L.A.
        • Cuthbert B.
        • Lewis-Fernández R.
        • Narrow W.E.
        • Reed G.M.
        Three approaches to understanding and classifying mental disorder: ICD-11, DSM-5, and the National Institute of Mental Health’s Research Domain Criteria (RDoC).
        Psychological Science in the Public Interest. 2017; 18: 72-145
        • Heugten-Van Der Kloet V.
        • van Heugten T.
        The classification of psychiatric disorders according to DSM-5 deserves an internationally standardized psychological test battery on symptom level.
        Frontiers in psychology. 2015; 6: 1108
      7. Freedman R, Lewis DA, Michels R, Pine DS, Schultz SK, Tamminga CA, et al. (2013): The initial field trials of DSM-5: new blooms and old thorns. Am Psychiatric Assoc, pp 1-5.

        • Galatzer-Levy I.R.
        • Bryant R.A.
        636,120 ways to have posttraumatic stress disorder.
        Perspectives on psychological science. 2013; 8: 651-662
        • Saggar M.
        • Uddin L.Q.
        Pushing the boundaries of psychiatric neuroimaging to ground diagnosis in biology.
        ENeuro. 2019; 6
      8. Alexander LM, Salum GA, Swanson JM, Milham MP (2017): Balancing Strengths and Weaknesses in Dimensional Psychiatry. bioRxiv.207019.

        • Marek S.
        • Tervo-Clemmens B.
        • Calabro F.
        • Nichols T.
        • Dosenbach N.
        Reproducible brain-wide association studies require thousands of individuals.
        Nature. 2022;
        • Carlozzi N.E.
        • Tulsky D.S.
        • Wolf T.J.
        • Goodnight S.
        • Heaton R.K.
        • Casaletto K.B.
        • et al.
        Construct validity of the NIH Toolbox Cognition Battery in individuals with stroke.
        Rehabil Psychol. 2017; 62: 443-454
        • Heaton R.K.
        • Akshoomoff N.
        • Tulsky D.
        • Mungas D.
        • Weintraub S.
        • Dikmen S.
        • et al.
        Reliability and validity of composite scores from the NIH Toolbox Cognition Battery in adults.
        J Int Neuropsychol Soc. 2014; 20: 588-598
        • Weintraub S.
        • Dikmen S.S.
        • Heaton R.K.
        • Tulsky D.S.
        • Zelazo P.D.
        • Slotkin J.
        • et al.
        The cognition battery of the NIH toolbox for assessment of neurological and behavioral function: validation in an adult sample.
        J Int Neuropsychol Soc. 2014; 20: 567-578
        • Dhamala E.
        • Jamison K.W.
        • Jaywant A.
        • Kuceyeski A.
        Shared functional connections within and between cortical networks predict cognitive abilities in adult males and females.
        Hum Brain Mapp. 2022;
        • Goldstein-Piekarski A.N.
        • Ball T.M.
        • Samara Z.
        • Staveland B.R.
        • Keller A.S.
        • Fleming S.L.
        • et al.
        Mapping neural circuit biotypes to symptoms and behavioral dimensions of depression and anxiety.
        Biol Psychiat. 2022; 91: 561-571
        • Eickhoff S.B.
        • Yeo B.
        • Genon S.
        Imaging-based parcellations of the human brain.
        Nature Reviews Neuroscience. 2018; 19: 672-686
      9. Uddin LQ, Betzel RF, Cohen JR, Damoiseaux JS, De Brigard F, Eickhoff SB, et al. (2022): Controversies and current progress on large-scale brain network nomenclature from OHBM WHATNET: Workgroup for HArmonized Taxonomy of NETworks.

        • Amunts K.
        • Mohlberg H.
        • Bludau S.
        • Zilles K.
        Julich-Brain: A 3D probabilistic atlas of the human brain’s cytoarchitecture.
        Science. 2020; 369: 988-992
        • Eickhoff S.B.
        • Rottschy C.
        • Kujovic M.
        • Palomero-Gallagher N.
        • Zilles K.
        Organizational principles of human visual cortex revealed by receptor mapping.
        Cerebral cortex. 2008; 18: 2637-2645
        • Eickhoff S.B.
        • Stephan K.E.
        • Mohlberg H.
        • Grefkes C.
        • Fink G.R.
        • Amunts K.
        • et al.
        A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data.
        Neuroimage. 2005; 25: 1325-1335
        • Destrieux C.
        • Fischl B.
        • Dale A.
        • Halgren E.
        Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature.
        Neuroimage. 2010; 53: 1-15
      10. Talairach J (1988): Co-planar stereotaxic atlas of the human brain-3-dimensional proportional system. An approach to cerebral imaging.

        • Desikan R.S.
        • Segonne F.
        • Fischl B.
        • Quinn B.T.
        • Dickerson B.C.
        • Blacker D.
        • et al.
        An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest.
        Neuroimage. 2006; 31: 968-980
        • Schaefer A.
        • Kong R.
        • Gordon E.M.
        • Laumann T.O.
        • Zuo X.-N.
        • Holmes A.J.
        • et al.
        Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI.
        Cerebral cortex. 2018; 28: 3095-3114
        • Yeo B.T.
        • Krienen F.M.
        • Sepulcre J.
        • Sabuncu M.R.
        • Lashkari D.
        • Hollinshead M.
        • et al.
        The organization of the human cerebral cortex estimated by intrinsic functional connectivity.
        Journal of neurophysiology. 2011; 106: 1125-1165
        • Craddock C.R.
        • James G.A.
        • Holtzheimer 3rd, P.E.
        • Hu X.P.
        • Mayberg H.S.
        A whole brain fMRI atlas generated via spatially constrained spectral clustering.
        Hum Brain Mapp. 2012; 33: 1914-1928
        • Power J.D.
        • Cohen A.L.
        • Nelson S.M.
        • Wig G.S.
        • Barnes K.A.
        • Church J.A.
        • et al.
        Functional network organization of the human brain.
        Neuron. 2011; 72: 665-678
        • Dosenbach N.U.
        • Nardos B.
        • Cohen A.L.
        • Fair D.A.
        • Power J.D.
        • Church J.A.
        • et al.
        Prediction of individual brain maturity using fMRI.
        Science. 2010; 329: 1358-1361
        • Bellec P.
        • Rosa-Neto P.
        • Lyttelton O.C.
        • Benali H.
        • Evans A.C.
        Multi-level bootstrap analysis of stable clusters in resting-state fMRI.
        Neuroimage. 2010; 51: 1126-1139
        • Smith S.M.
        • Fox P.T.
        • Miller K.L.
        • Glahn D.C.
        • Fox P.M.
        • Mackay C.E.
        • et al.
        Correspondence of the brain's functional architecture during activation and rest.
        Proceedings of the national academy of sciences. 2009; 106: 13040-13045
        • Wu L.
        • Calhoun V.D.
        • Jung R.E.
        • Caprihan A.
        Connectivity‐based whole brain dual parcellation by group ICA reveals tract structures and decreased connectivity in schizophrenia.
        Human brain mapping. 2015; 36: 4681-4701
      11. Ji Y, Hervé P-Y, Aickelin U, Pitiot A (2009): Parcellation of fmri datasets with ica and pls-a data driven approach. International Conference on Medical Image Computing and Computer-Assisted Intervention: Springer, pp 984-991.

        • Kong R.
        • Li J.
        • Orban C.
        • Sabuncu M.R.
        • Liu H.
        • Schaefer A.
        • et al.
        Spatial Topography of Individual-Specific Cortical Networks Predicts Human Cognition, Personality, and Emotion.
        Cerebral Cortex. 2018; 29: 2533-2551
        • Thirion B.
        • Varoquaux G.
        • Dohmatob E.
        • Poline J.-B.
        Which fMRI clustering gives good brain parcellations?.
        Frontiers in neuroscience. 2014; 8: 167
        • Kong R.
        • Yang Q.
        • Gordon E.
        • Xue A.
        • Yan X.
        • Orban C.
        • et al.
        Individual-Specific Areal-Level Parcellations Improve Functional Connectivity Prediction of Behavior.
        Cerebral Cortex. 2021;
      12. Dhamala E, Jamison KW, Sabuncu MR, Kuceyeski A (2020): Sex classification using long-range temporal dependence of resting-state functional MRI time series. Human Brain Mapping. n/a.

        • Zeighami Y.
        • Evans A.C.
        Association vs. prediction: The impact of cortical surface smoothing and parcellation on brain age.
        Frontiers in big Data. 2021; 4: 15
        • Khosla M.
        • Jamison K.
        • Kuceyeski A.
        • Sabuncu M.R.
        Ensemble learning with 3D convolutional neural networks for functional connectome-based prediction.
        Neuroimage. 2019; 199: 651-662
        • Kalmady S.V.
        • Greiner R.
        • Agrawal R.
        • Shivakumar V.
        • Narayanaswamy J.C.
        • Brown M.R.
        • et al.
        Towards artificial intelligence in mental health by improving schizophrenia prediction with multiple brain parcellation ensemble-learning.
        npj Schizophrenia. 2019; 5: 1-11
        • Gordon E.M.
        • Laumann T.O.
        • Gilmore A.W.
        • Newbold D.J.
        • Greene D.J.
        • Berg J.J.
        • et al.
        Precision functional mapping of individual human brains.
        Neuron. 2017; 95 (e797): 791-807
        • Gratton C.
        • Laumann T.O.
        • Nielsen A.N.
        • Greene D.J.
        • Gordon E.M.
        • Gilmore A.W.
        • et al.
        Functional brain networks are dominated by stable group and individual factors, not cognitive or daily variation.
        Neuron. 2018; 98 (e435): 439-452
        • Szucs D.
        • Ioannidis J.P.
        Sample size evolution in neuroimaging research: An evaluation of highly-cited studies (1990–2012) and of latest practices (2017–2018) in high-impact journals.
        NeuroImage. 2020; 221117164
        • Janssen R.J.
        • Mourão-Miranda J.
        • Schnack H.G.
        Making individual prognoses in psychiatry using neuroimaging and machine learning.
        Biological Psychiatry: Cognitive Neuroscience and Neuroimaging. 2018; 3: 798-808
        • Cui Z.
        • Gong G.
        The effect of machine learning regression algorithms and sample size on individualized behavioral prediction with functional connectivity features.
        Neuroimage. 2018; 178: 622-637
        • Jollans L.
        • Boyle R.
        • Artiges E.
        • Banaschewski T.
        • Desrivieres S.
        • Grigis A.
        • et al.
        Quantifying performance of machine learning methods for neuroimaging data.
        Neuroimage. 2019; 199: 351-365
        • Varoquaux G.
        Cross-validation failure: Small sample sizes lead to large error bars.
        Neuroimage. 2018; 180: 68-77
        • Santos M.S.
        • Soares J.P.
        • Abreu P.H.
        • Araujo H.
        • Santos J.
        Cross-Validation for Imbalanced Datasets: Avoiding Overoptimistic and Overfitting Approaches.
        Ieee Comput Intell M. 2018; 13: 59-76
        • He T.
        • An L.
        • Chen P.
        • Chen J.
        • Feng J.
        • Bzdok D.
        • et al.
        Meta-matching as a simple framework to translate phenotypic predictive models from big to small data.
        Nature Neuroscience. 2022; : 1-10
        • Rosenberg M.D.
        • Finn E.S.
        • Scheinost D.
        • Papademetris X.
        • Shen X.
        • Constable R.T.
        • et al.
        A neuromarker of sustained attention from whole-brain functional connectivity.
        Nature neuroscience. 2016; 19: 165-171
      13. Ricard JA, Parker TC, Dhamala E, Kwasa J, Allsop AS, Holmes AJ (2022): Confronting racist and exclusionary practices in the acquisition and analyses of neuroimaging data. Submitted.

        • Li J.
        • Bzdok D.
        • Chen J.
        • Tam A.
        • Ooi L.Q.R.
        • Holmes A.J.
        • et al.
        Cross-ethnicity/race generalization failure of behavioral prediction from resting-state functional connectivity.
        Science Advances. 2022; 8eabj1812
        • Jiang R.
        • Calhoun V.D.
        • Cui Y.
        • Qi S.
        • Zhuo C.
        • Li J.
        • et al.
        Multimodal data revealed different neurobiological correlates of intelligence between males and females.
        Brain imaging and behavior. 2020; 14: 1979-1993
        • Jiang R.
        • Calhoun V.D.
        • Fan L.
        • Zuo N.
        • Jung R.
        • Qi S.
        • et al.
        Gender differences in connectome-based predictions of individualized intelligence quotient and sub-domain scores.
        Cerebral Cortex. 2020; 30: 888-900
        • Nostro A.D.
        • Müller V.I.
        • Varikuti D.P.
        • Pläschke R.N.
        • Hoffstaedter F.
        • Langner R.
        • et al.
        Predicting personality from network-based resting-state functional connectivity.
        Brain Structure and Function. 2018; 223: 2699-2719
        • Sanchis-Segura C.
        • Ibañez-Gual M.V.
        • Aguirre N.
        • Forn C.
        • Cruz-Gómez ÁJ
        Effects of different intracranial volume correction methods on univariate sex differences in grey matter volume and multivariate sex prediction.
        Scientific Reports. 2020; 10: 1-15
        • Greene A.S.
        • Shen X.
        • Noble S.
        • Horien C.
        • Hahn C.A.
        • Arora J.
        • et al.
        Brain–phenotype models fail for individuals who defy sample stereotypes.
        Nature. 2022; : 1-10
        • Oexle N.
        • Corrigan P.W.
        Understanding mental illness stigma toward persons with multiple stigmatized conditions: Implications of intersectionality theory.
        Psychiatric Services. 2018; 69: 587-589
        • Tang S.
        • Sun N.
        • Floris D.L.
        • Zhang X.
        • Di Martino A.
        • Yeo B.T.T.
        Reconciling Dimensional and Categorical Models of Autism Heterogeneity: A Brain Connectomics and Behavioral Study.
        Biol Psychiatry. 2020; 87: 1071-1082
        • Hong S.J.
        • Valk S.L.
        • Di Martino A.
        • Milham M.P.
        • Bernhardt B.C.
        Multidimensional Neuroanatomical Subtyping of Autism Spectrum Disorder.
        Cereb Cortex. 2018; 28: 3578-3588
        • Drysdale A.T.
        • Grosenick L.
        • Downar J.
        • Dunlop K.
        • Mansouri F.
        • Meng Y.
        • et al.
        Resting-state connectivity biomarkers define neurophysiological subtypes of depression.
        Nature medicine. 2017; 23: 28-38
        • Wang Y.
        • Tang S.
        • Zhang L.
        • Bu X.
        • Lu L.
        • Li H.
        • et al.
        Data-driven clustering differentiates subtypes of major depressive disorder with distinct brain connectivity and symptom features.
        The British Journal of Psychiatry. 2021; 219: 606-613
        • Clementz B.A.
        • Sweeney J.A.
        • Hamm J.P.
        • Ivleva E.I.
        • Ethridge L.E.
        • Pearlson G.D.
        • et al.
        Identification of distinct psychosis biotypes using brain-based biomarkers.
        American Journal of Psychiatry. 2016; 173: 373-384
        • Chen J.
        • Patil K.R.
        • Weis S.
        • Sim K.
        • Nickl-Jockschat T.
        • Zhou J.
        • et al.
        Neurobiological Divergence of the Positive and Negative Schizophrenia Subtypes Identified on a New Factor Structure of Psychopathology Using Non-negative Factorization: An International Machine Learning Study.
        Biol Psychiatry. 2020; 87: 282-293
      14. Yan W, Zhao M, Fu Z, Pearlson GD, Sui J, Calhoun VD (2021): Mapping relationships among schizophrenia, bipolar and schizoaffective disorders: A deep classification and clustering framework using fMRI time series. Schizophrenia Research.

        • Chen J.
        • Muller V.I.
        • Dukart J.
        • Hoffstaedter F.
        • Baker J.T.
        • Holmes A.J.
        • et al.
        Intrinsic Connectivity Patterns of Task-Defined Brain Networks Allow Individual Prediction of Cognitive Symptom Dimension of Schizophrenia and Are Linked to Molecular Architecture.
        Biol Psychiatry. 2021; 89: 308-319
        • Wang D.
        • Li M.
        • Wang M.
        • Schoeppe F.
        • Ren J.
        • Chen H.
        • et al.
        Individual-specific functional connectivity markers track dimensional and categorical features of psychotic illness.
        Mol Psychiatry. 2020; 25: 2119-2129
        • Kebets V.
        • Holmes A.J.
        • Orban C.
        • Tang S.
        • Li J.
        • Sun N.
        • et al.
        Somatosensory-Motor Dysconnectivity Spans Multiple Transdiagnostic Dimensions of Psychopathology.
        Biol Psychiatry. 2019; 86: 779-791
        • Xia C.H.
        • Ma Z.
        • Ciric R.
        • Gu S.
        • Betzel R.F.
        • Kaczkurkin A.N.
        • et al.
        Linked dimensions of psychopathology and connectivity in functional brain networks.
        Nat Commun. 2018; 9: 3003
        • Kaczkurkin A.N.
        • Sotiras A.
        • Baller E.B.
        • Barzilay R.
        • Calkins M.E.
        • Chand G.B.
        • et al.
        Neurostructural heterogeneity in youths with internalizing symptoms.
        Biol Psychiat. 2020; 87: 473-482
        • Zhao X.
        • Rangaprakash D.
        • Yuan B.
        • Denney Jr., T.S.
        • Katz J.S.
        • Dretsch M.N.
        • et al.
        Investigating the Correspondence of Clinical Diagnostic Grouping With Underlying Neurobiological and Phenotypic Clusters Using Unsupervised Machine Learning.
        Front Appl Math Stat. 2018; 4
        • Nielsen A.N.
        • Barch D.M.
        • Petersen S.E.
        • Schlaggar B.L.
        • Greene D.J.
        Machine learning with neuroimaging: Evaluating its applications in psychiatry.
        Biological Psychiatry: Cognitive Neuroscience and Neuroimaging. 2020; 5: 791-798
        • Sui J.
        • Jiang R.
        • Bustillo J.
        • Calhoun V.
        Neuroimaging-based individualized prediction of cognition and behavior for mental disorders and health: methods and promises.
        Biol Psychiat. 2020; 88: 818-828