Abstract
We use parallel interacting goal-directed and habitual strategies to make our daily
decisions. The arbitration between these strategies is relevant to inflexible repetitive
behaviors in psychiatric disorders. Goal-directed control, also known as model-based
control, is based on an affective outcome relying on a learned internal model to prospectively
make decisions. In contrast, habit control, also known as model-free control, is based
on an integration of previous reinforced learning autonomous of the current outcome
value and is implicit and more efficient but at the cost of greater inflexibility.
The concept of model-based control can be further extended into pavlovian processes.
Here we describe and compare tasks that tap into these constructs and emphasize the
clinical relevance and translation of these tasks in psychiatric disorders. Together,
these findings highlight a role for model-based control as a transdiagnostic impairment
underlying compulsive behaviors and representing a promising therapeutic target.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Biological PsychiatryAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Model-based influences on humans’ choices and striatal prediction errors.Neuron. 2011; 69: 1204-1215
- States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning.Neuron. 2010; 66: 585-595
- The impact of orbitofrontal dysfunction on cocaine addiction.Nat Neurosci. 2012; 15: 358-366
- Instrumental responding following reinforcer devaluation.Q J Exp Psychol. 1981; 33: 109-122
- The role of learning in the operation of motivational systems.in: Gallister C.R. Stevens’ Handbook of Experimental Psychology: Learning, Motivation and Emotion, 3rd ed. John Wiley, New York2002: 497-534
- Differential engagement of the ventromedial prefrontal cortex by goal-directed and habitual behavior toward food pictures in humans.J Neurosci. 2009; 29: 11330-11338
- Goals and habits in the brain.Neuron. 2013; 80: 312-325
- The role of the dorsomedial striatum in instrumental conditioning.Eur J Neurosci. 2005; 22: 513-523
- Goal-directed instrumental action: Contingency and incentive learning and their cortical substrates.Neuropharmacology. 1998; 37: 407-419
- Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning.Eur J Neurosci. 2004; 19: 181-189
- Coordination of actions and habits in the medial prefrontal cortex of rats.Cereb Cortex. 2003; 13: 400-408
- A specific role for posterior dorsolateral striatum in human habit learning.Eur J Neurosci. 2009; 29: 2225-2232
- Changes in corticostriatal connectivity during reinforcement learning in humans.Hum Brain Mapp. 2015; 36: 793-803
- Corticostriatal connectivity underlies individual differences in the balance between habitual and goal-directed action control.J Neurosci. 2012; 32: 12066-12075
- Disruption of dorsolateral prefrontal cortex decreases model-based in favor of model-free control in humans.Neuron. 2013; 80: 914-919
- Mapping value based planning and extensively trained choice in the human brain.Nat Neurosci. 2012; 15: 786-791
- Disorders of compulsivity: A common bias towards learning habits.Mol Psychiatry. 2015; 20: 345-352
- Fronto-striatal organization: Defining functional and microstructural substrates of behavioural flexibility.Cortex. 2016; 74: 118-133
- Determining the neural substrates of goal-directed learning in the human brain.J Neurosci. 2007; 27: 4019-4026
- Simple plans or sophisticated habits? State, transition and learning interactions in the two-step task.PLoS Comput Biol. 2015; 11: e1004648
Miller K, Akrami A, Botvinick M, Brody C (2015): The role of the orbitofrontal cortex in model-based planning in the rat. Presented at the annual meeting of the Society for Neuroscience, October 17–21, Chicago.
Groman SM, Chen L, Smith NJ, Lee D, Taylor JR (2014): Dorsomedial striatum lesions disrupt the balance between model-free and model-based learning in a multi-stage decision-making task. Presented at the annual meeting of the Society for Neuroscience, November 15–19, Washington, DC.
Miranda B, Malalasekera N, Dayan P, Kennerley S (2014): Evidence of model-based and model-free reinforcement learning in prefrontal cortex and striatal neurons. Presented at the annual meeting of the Society for Neuroscience, November 15–19, Washington, DC.
- Neural systems of reinforcement for drug addiction: From actions to habits to compulsion.Nat Neurosci. 2005; 8: 1481-1489
- Dopamine, reinforcement learning, and addiction.Pharmacopsychiatry. 2009; 42: S56-S65
- Alcohol seeking by rats: Action or habit?.Q J Exp Psychol B. 2002; 55: 331-348
- Repeated episodes of chronic intermittent ethanol promote insensitivity to devaluation of the reinforcing effect of ethanol.Alcohol. 2014; 48: 639-645
- Habitual alcohol seeking: Time course and the contribution of subregions of the dorsal striatum.Biol Psychiatry. 2012; 72: 389-395
- Shift from goal-directed to habitual cocaine seeking after prolonged experience in rats.J Neurosci. 2010; 30: 15457-15463
- Post-training cocaine administration facilitates habit learning and requires the infralimbic cortex and dorsolateral striatum.Neurobiol Learn Mem. 2015; 118: 105-112
- Amphetamine exposure enhances habit formation.J Neurosci. 2006; 26: 3805-3812
- No association of goal-directed and habitual control with alcohol consumption in young adults.Addict Biol. 2017; ([published online ahead of print Jan 23])
- Characterizing a psychiatric symptom dimension related to deficits in goal-directed control.eLife. 2016; 5: e11305
- The involuntary nature of binge drinking: Goal directedness and awareness of intention.Addict Biol. 2017; ([published online ahead of print Apr 16])
- Behavioral and neuroimaging evidence for overreliance on habit learning in alcohol-dependent patients.Transl Psychiatry. 2013; 3: e337
- Model-based and model-free decisions in alcohol dependence.Neuropsychobiology. 2014; 70: 122-131
- When habits are dangerous: Alcohol expectancies and habitual decision making predict relapse in alcohol dependence [published online ahead of print May 22].Biol Psychiatry. 2017;
- Addiction history associates with the propensity to form habits.J Cogn Neurosci. 2016; 28: 1024-1038
- Carrots and sticks fail to change behavior in cocaine addiction.Science. 2016; 352: 1468-1471
- Binge-like consumption of a palatable food accelerates habitual control of behavior and is dependent on activation of the dorsolateral striatum.J Neurosci. 2014; 34: 5012-5022
- Slave to habit? Obesity is associated with decreased behavioural sensitivity to reward devaluation.Appetite. 2015; 87: 175-183
- Disruption in the balance between goal-directed behavior and habit learning in obsessive-compulsive disorder.Am J Psychiatry. 2011; 168: 718-726
- Motivation and value influences in the relative balance of goal-directed and habitual behaviours in obsessive-compulsive disorder.Transl Psychiatry. 2015; 5: e670
- Enhanced avoidance habits in obsessive-compulsive disorder.Biol Psychiatry. 2014; 75: 631-638
- Orbitofrontal activation restores insight lost after cocaine use.Nat Neurosci. 2014; 17: 1092-1099
- Smokers’ brains compute, but ignore, a fictive error signal in a sequential investment task.Nat Neurosci. 2008; 11: 514-520
- Behavioral and neural signatures of reduced updating of alternative options in alcohol-dependent patients during flexible decision-making.J Neurosci. 2016; 36: 10935-10948
- Impaired flexible reward-based decision-making in binge eating disorder: Evidence from computational modeling and functional neuroimaging.Neuropsychopharmacology. 2017; 42: 628-637
- Counterfactual processing of economic action–outcome alternatives in obsessive-compulsive disorder: Further evidence of impaired goal-directed behavior.Biol Psychiatry. 2014; 75: 639-646
- Thinking outside the box: Orbitofrontal cortex, imagination, and how we can treat addiction.Neuropsychopharmacology. 2016; 41: 2966-2976
- Impairments in goal-directed actions predict treatment response to cognitive-behavioral therapy in social anxiety disorder.PLoS One. 2014; 9: e94778
- Reduced model-based decision-making in schizophrenia.J Abnorm Psychol. 2016; 125: 777-787
- Accelerated habit formation following amphetamine exposure is reversed by D1, but enhanced by D2, receptor antagonists.Front Neurosci. 2013; 7: 76
- Lesion to the nigrostriatal dopamine system disrupts stimulus–response habit formation.J Neurosci. 2005; 25: 2771-2780
- Reliance on habits at the expense of goal-directed control following dopamine precursor depletion.Psychopharmacology (Berl). 2012; 219: 621-631
- Habitual versus goal-directed action control in Parkinson’s disease.J Cogn Neurosci. 2011; 23: 1218-1229
- Dopamine selectively remediates “model-based” reward learning: A computational approach.Brain. 2016; 139: 355-364
- Dopamine enhances model-based over model-free choice behavior.Neuron. 2012; 75: 418-424
- Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making.Proc Natl Acad Sci U S A. 2015; 112: 1595-1600
- Reduced forebrain serotonin transmission is causally involved in the development of compulsive cocaine seeking in rats.Neuropsychopharmacology. 2012; 37: 2505-2514
- Increased expression of 5-HT6 receptors in dorsolateral striatum decreases habitual lever pressing, but does not affect learning acquisition of simple operant tasks in rats.Eur J Neurosci. 2011; 34: 343-351
- Tryptophan depletion promotes habitual over goal-directed control of appetitive responding in humans.Int J Neuropsychopharmacol. 2015; 18: pyv013
- Opponent interactions between serotonin and dopamine.Neural Netw. 2002; 15: 603-616
- Tonic dopamine: Opportunity costs and the control of response vigor.Psychopharmacology (Berl). 2007; 191: 507-520
- Speed/accuracy trade-off between the habitual and the goal-directed processes.PLoS Comput Biol. 2011; 7: e1002055
- Devaluation and sequential decisions: Linking goal-directed and model-based behavior.Front Hum Neurosci. 2014; 8: 587
- Working-memory capacity protects model-based learning from stress.Proc Natl Acad Sci U S A. 2013; 110: 20941-20946
- The interaction of acute and chronic stress impairs model-based behavioral control.Psychoneuroendocrinology. 2015; 53: 268-280
- The curse of planning: Dissecting multiple reinforcement-learning systems by taxing the central executive.Psychol Sci. 2013; 24: 751-761
- Processing speed enhances model-based over model-free reinforcement learning in the presence of high working memory functioning.Front Psychol. 2014; 5: 1450
- Model-based learning protects against forming habits.Cogn Affect Behav Neurosci. 2015; 15: 523-536
- From creatures of habit to goal-directed learners: Tracking the developmental emergence of model-based reinforcement learning.Psychol Sci. 2016; 27: 848-858
- When does model-based control pay off?.PLoS Comput Biol. 2016; 12: e1005090
- Model-based learning and the contribution of the orbitofrontal cortex to the model-free world.Eur J Neurosci. 2012; 35: 991-996
- Orbitofrontal cortex supports behavior and learning using inferred but not cached values.Science. 2012; 338: 953-956
- Disruption of model-based behavior and learning by cocaine self-administration in rats.Psychopharmacology (Berl). 2013; 229: 493-501
- Reflection impulsivity in binge drinking: Behavioural and volumetric correlates.Addict Biol. 2016; 21: 504-515
- Food-pics: An image database for experimental research on eating and appetite.Front Psychol. 2014; 5: 617
- Modelling individual differences in the form of Pavlovian conditioned approach responses: A dual learning systems approach with factored representations.PLoS Comput Biol. 2014; 10: e1003466
Article Info
Publication History
Published online: April 23, 2017
Accepted:
April 12,
2017
Received in revised form:
April 11,
2017
Received:
December 14,
2016
Identification
Copyright
© 2017 Society of Biological Psychiatry.
