Advertisement

A Formal Valuation Framework for Emotions and Their Control

  • Quentin J.M. Huys
    Correspondence
    Address correspondence to Quentin J.M. Huys, Translational Neuromodeling Unit, Wilfriedstrasse 6, 8032 Zürich, Switzerland.
    Affiliations
    Translational Neuromodeling Unit, Institute for Biomedical Engineering, University of Zurich and Swiss Federal Institute of Technology (ETH Zurich), Zürich, Switzerland

    Centre for Addictive Disorders, Department of Psychiatry, Psychotherapy and Psychosomatics, Hospital of Psychiatry, University of Zurich, Zürich, Switzerland
    Search for articles by this author
  • Daniel Renz
    Affiliations
    Translational Neuromodeling Unit, Institute for Biomedical Engineering, University of Zurich and Swiss Federal Institute of Technology (ETH Zurich), Zürich, Switzerland
    Search for articles by this author
Open AccessPublished:July 11, 2017DOI:https://doi.org/10.1016/j.biopsych.2017.07.003

      Abstract

      Computational psychiatry aims to apply mathematical and computational techniques to help improve psychiatric care. To achieve this, the phenomena under scrutiny should be within the scope of formal methods. As emotions play an important role across many psychiatric disorders, such computational methods must encompass emotions. Here, we consider formal valuation accounts of emotions. We focus on the fact that the flexibility of emotional responses and the nature of appraisals suggest the need for a model-based valuation framework for emotions. However, resource limitations make plain model-based valuation impossible and require metareasoning strategies to apportion cognitive resources adaptively. We argue that emotions may implement such metareasoning approximations by restricting the range of behaviors and states considered. We consider the processes that guide the deployment of the approximations, discerning between innate, model-free, heuristic, and model-based controllers. A formal valuation and metareasoning framework may thus provide a principled approach to examining emotions.

      Keywords

      Computational psychiatry is a young field hoping to leverage advances in computational techniques to understand and improve mental health (
      • Huys Q.J.M.
      • Moutoussis M.
      • Williams J.
      Are computational models of any use to psychiatry?.
      ,
      • Maia T.V.
      • Frank M.J.
      From reinforcement learning models to psychiatric and neurological disorders.
      ,
      • Montague P.R.
      • Dolan R.J.
      • Friston K.J.
      • Dayan P.
      Computational psychiatry.
      ,
      • Stephan K.E.
      • Mathys C.
      Computational approaches to psychiatry.
      ,
      • Huys Q.J.M.
      • Maia T.V.
      • Frank M.J.
      Computational psychiatry as a bridge from neuroscience to clinical applications.
      ). It is motivated on the one hand by the necessity to bring novel statistical and machine-learning techniques to bear on the rapidly expanding complexity of novel datasets relevant to mental health, and on the other hand by the complexity of the problem itself as mental health relates to the most difficult tasks performed by the most complex of organs.
      Emotions are central to mental health, and emotional disorders contribute substantially to the burden of mental illnesses (
      • Whiteford H.A.
      • Degenhardt L.
      • Rehm J.
      • Baxter A.J.
      • Ferrari A.J.
      • Erskine H.E.
      • et al.
      Global burden of disease attributable to mental and substance use disorders: Findings from the global burden of disease study 2010.
      ). The traditional dichotomization of emotion and reason might question the feasibility of applying computational techniques to the core issues of emotion. It is therefore imperative for computational psychiatry that we consider the ability of a computational and mathematical framework to address core emotional phenomena. Here, we argue that approaching emotion computationally requires the introduction of model-based valuation and metareasoning. Metareasoning considers optimal valuation in the face of resource constraints (
      • Simon H.A.
      Rational choice and the structure of the environment.
      ,
      • Russell S.
      • Wefald E.
      Principles of metareasoning.
      ,
      • Payne J.W.
      • Bettman J.R.
      • Johnson E.J.
      The Adaptive Decision Maker.
      ). The proposal is that human emotions involve strategies to deal with the complexity of model-based or goal-directed decision making by focusing on particular aspects of the problem at hand.
      Research on human emotions is complicated as questions about their nature continue to divide the scientific community (
      • Barrett L.F.
      Are emotions natural kinds?.
      ,
      • Panksepp J.
      Neurologizing the psychology of affects: How appraisal-based constructivism and basic emotion theory can coexist.
      ). Nevertheless, there is consensus on a number of key components that characterize emotions, and this review attempts to view them in a computational light. We first provide a description of important features of emotions, then introduce valuation and the metareasoning problem, then relate approximate metareasoning strategies to features of emotions, and finally describe the control of approximate metareasoning strategies.

      Ingredients of a Computational Approach to Emotions

      Key features of human emotions that require accounting for and that are emphasized to various degrees in different conceptualizations are 1) correlated physiological, psychological and behavioral processes shaped by evolutionarily predefined neural circuitry; 2) interpretations or appraisals; and 3) conscious verbal self-report about emotions. Key problems in contemporary research on human emotions include to what extent the three feature domains are related (e.g., how conscious emotions in humans relate to evolutionarily predefined circuitry) and to what extent emotions are discrete entities.
      Basic emotion theories suggest that there are a limited, relatively fixed, number of universal, evolutionarily shaped, culture-independent, and neurobiologically hard-coded emotional categories including happiness, surprise, sadness, disgust, anger, and fear (
      • Panksepp J.
      Neurologizing the psychology of affects: How appraisal-based constructivism and basic emotion theory can coexist.
      ,
      • Ekman P.
      An argument for basic emotions.
      ,
      • Izard C.E.
      Basic emotions, natural kinds, emotion schemas, and a new paradigm.
      ). For the present purpose, what is important is that these represent a set of innately interlinked physiological, behavioral, and psychological processes that are triggered in an inflexible manner by species-specific salient stimuli, akin to unconditioned responses. Animal research, in which specific responses to species-relevant stimuli are observable and readily quantifiable, has contributed to this view. However, behavioral responses in animals cannot be directly translated to emotional experiences in humans. Amygdalar and hippocampal damage, for instance, abolish physiological and autobiographical signatures of aversive conditioning, respectively, while leaving the other intact (
      • LeDoux J.E.
      Coming to terms with fear.
      ). Furthermore, aversive conditioning can be performed subliminally and can evoke amygdala activity and physiological response, but can fail to result in any emotion of fear (
      • Raio C.M.
      • Carmel D.
      • Carrasco M.
      • Phelps E.A.
      Nonconscious fear is quickly acquired but swiftly forgotten.
      ,
      • Knight D.C.
      • Waters N.S.
      • Bandettini P.A.
      Neural substrates of explicit and implicit fear memory.
      ), while amygdalar lesions can leave human fear unaffected (
      • Feinstein J.S.
      • Buzza C.
      • Hurlemann R.
      • Follmer R.L.
      • Dahdaleh N.S.
      • Coryell W.H.
      • et al.
      Fear and panic in humans with bilateral amygdala damage.
      ,
      • Khalsa S.S.
      • Feinstein J.S.
      • Li W.
      • Feusner J.D.
      • Adolphs R.
      • Hurlemann R.
      Panic anxiety in humans with bilateral amygdala lesions: Pharmacological induction via cardiorespiratory interoceptive pathways.
      ).
      Human emotional responses to stimuli are characterized by substantial within- and between-subject variability. Appraisal theory locates one explanation for this variability in the interpretation (be it conscious or unconscious) of a particular situation or stimulus as being relevant to the individual’s goals (
      • Roseman I.J.
      • Smith C.A.
      Appraisal Theory.
      ). This interpretation depends on the goal and the individual’s beliefs in addition to the stimulus. A stimulus or situation being interpreted as increasing the changes of reaching one’s goals would, for instance, result in the emotion of joy or happiness (
      • Frijda N.H.
      • Kuipers P.
      • Ter Schure E.
      Relations among emotion, appraisal, and emotional action readiness.
      ,

      Lazarus RS: Stress and Emotion: A New Synthesis. New York: Springer.

      ,
      • Ellsworth P.C.
      Appraisal theory: Old and new questions.
      ). However, just like basic emotion theories, appraisal theories often view the expressed emotion itself as a “definable pattern of outputs that preexist within the individual” (
      • Barrett L.F.
      Are emotions natural kinds?.
      ). For instance, Scherer (
      • Scherer K.R.
      What are emotions? And how can they be measured?.
      ) defined them as “episode[s] of interrelated, synchronized changes in the states of all or most […] organismic subsystems in response to the evaluation of an external or internal stimulus event as relevant to major concerns of the organism.”
      The evidence for discrete emotions is controversial. Autonomic responses, electroencephalographic features, and facial expressions do not permit simple categorization and show little evidence of the predicted correlations (
      • Barrett L.F.
      Are emotions natural kinds?.
      ,
      • Cacioppo J.T.
      • Berntson G.G.
      • Larsen J.T.
      • Poehlmann K.M.
      • Ito T.A.
      • et al.
      The psychophysiology of emotion.
      ,
      • Jack R.E.
      • Garrod O.G.B.
      • Yu H.
      • Caldara R.
      • Schyns P.G.
      Facial expressions of emotion are not culturally universal.
      ), though newer machine learning approaches have shown that categorical information can be extracted from physiological (
      • Stephens C.L.
      • Christie I.C.
      • Friedman B.H.
      Autonomic specificity of basic emotions: Evidence from pattern classification and cluster analysis.
      ) and neural (
      • Kragel P.A.
      • LaBar K.S.
      Multivariate neural biomarkers of emotional states are categorically distinct.
      ,
      • Saarimki H.
      • Gotsopoulos A.
      • Jskelinen I.P.
      • Lampinen J.
      • Vuilleumier P.
      • Hari R.
      • et al.
      Discrete neural signatures of basic emotions.
      ) data. The latter analyses have, however, clarified that there is no single underlying substrate for particular emotions. Rather, each emotional category depends on a distributed network of limbic but also cortical components that reflect the particular neurocognitive processes involved (
      • Wager T.D.
      • Kang J.
      • Johnson T.D.
      • Nichols T.E.
      • Satpute A.B.
      • Barrett L.F.
      A Bayesian model of category-specific emotional brain responses.
      ).
      An alternative view is that the discreteness of emotions arises from the categorical labeling of internal events for the purpose of intra- or intersubject communication. Neuroimaging has provided some support for such a model, arguing that the ventrolateral prefrontal cortex is involved in categorical labeling of emotional states (
      • Barrett L.F.
      Solving the emotion paradox: Categorization and the experience of emotion.
      ,
      • Lieberman M.D.
      • Eisenberger N.I.
      • Crockett M.J.
      • Tom S.M.
      • Pfeifer J.H.
      • Way B.M.
      Putting feelings into words: Affect labeling disrupts amygdala activity in response to affective stimuli.
      ,
      • Satpute A.B.
      • Shu J.
      • Weber J.
      • Roy M.
      • Ochsner K.N.
      The functional neural architecture of self-reports of affective experience.
      ) evolving along the two major axes of valence (from good to bad) and arousal (from high to low). Indeed, factor analyses of a variety of measures of emotions including similarity ratings among words, facial expression, and autonomic measures reliably identify these two separate dimensions (
      • Russell J.A.
      Core affect and the psychological construction of emotion.
      ). Neuroimaging has also been used to argue that while the amygdala tracks arousal, the orbitofrontal cortex tracks valence across emotions (
      • Wilson-Mendenhall C.D.
      • Barrett L.F.
      • Barsalou L.W.
      Neural evidence that human emotions share core affective properties.
      ).

      Valuation and Emotion

      Basic and animal emotion research, with its grounding in evolutionarily shaped responses, emphasizes the importance of emotions in guiding behavior adaptively. A focus on adaptive responding is also present in appraisal theories, which suggest that emotions arise when events are judged to be relevant to the individual’s “needs, attachments, values, current goals and beliefs” (
      • Moors A.
      • Ellsworth P.C.
      • Scherer K.R.
      • Frijda N.H.
      Appraisal theories of emotion: State of the art and future development.
      ). Computationally, inferring adaptive choices involves integrating not only immediate rewards, but also longer-term rewards, and for that reason requires consideration of the future course of events. This evaluation of the future is where the problem lies, as the further into the future one looks, the broader the range of potential events. Specifically, valuation involves summing over an exponentially expanding decision tree of future possibilities. Optimal valuation would search the entire tree, which is rarely feasible. Reinforcement learning is a thriving subfield of machine learning concerned with algorithmic solutions to this problem.

      Model-Free Accounts of Emotional Expression

      A substantial body of work has related one such algorithmic solution to how emotional expressions change over time (
      • Bach D.R.
      • Dayan P.
      Algorithms for survival: A comparative perspective on emotions.
      ). In so-called model-free reinforcement learning, the stability of the world is exploited to replace integration over the future with actual past experiences. Clever bookkeeping allows the use of prediction errors to update values that, in the limit of extensive experience, are guaranteed to yield the true long-term values of states and behaviors (
      • Sutton R.S.
      • Barto A.G.
      Reinforcement Learning: An Introduction.
      ). Here, emotional responses are viewed as a type of high-level action, involving multiple biological and neural subsystems. One example of such an “action” is a freezing response, which has behavioral, attentional, and physiological components. These high-level actions are thought to be emitted either in an innate fashion (
      • Hirsch S.
      • Bolles R.
      On the ability of prey to recognize predators.
      ) in response to the appropriate species-specific unconditional stimulus (
      • Bolles R.C.
      Species-specific defense reactions and avoidance learning.
      ,
      • Seligman M.E.
      On the generality of the laws of learning.
      ,
      • Timberlake W.
      • Wahl G.
      • King D.
      Stimulus and response contingencies in the misbehavior of rats.
      ), or after learning in response to a conditioned stimulus. In the latter case, the expression of the action is proportional to the value attached to the conditioned stimulus, which in turn is a scalar measure of the average expected unconditional stimulus strength (
      • Huys Q.J.M.
      • Cools R.
      • Gölzer M.
      • Friedel E.
      • Heinz A.
      • Dolan R.J.
      • et al.
      Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding.
      ,
      • Dayan P.
      • Niv Y.
      • Seymour B.
      • Daw N.D.
      The misbehavior of value and the discipline of the will.
      ,
      • Huys Q.J.M.
      • Eshel N.
      • O’Nions E.
      • Sheridan L.
      • Dayan P.
      • Roiser J.P.
      Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees.
      ). This has been applied to a wide variety of affective responses, including heart rate changes (
      • Seymour B.
      • O’Doherty J.P.
      • Koltzenburg M.
      • Wiech K.
      • Frackowiak R.
      • Friston K.
      • et al.
      Opponent appetitive-aversive neural processes underlie predictive learning of pain relief.
      ), approach (
      • Lesaint F.
      • Sigaud O.
      • Flagel S.B.
      • Robinson T.E.
      • Khamassi M.
      Modelling individual differences in the form of pavlovian conditioned approach responses: A dual learning systems approach with factored representations.
      ), avoidance (
      • Moutoussis M.
      • Bentall R.P.
      • Williams J.
      • Dayan P.
      A temporal difference account of avoidance learning.
      ,
      • Maia T.V.
      Two-factor theory, the actor-critic model, and conditioned avoidance.
      ), extinction (
      • Gershman S.J.
      • Blei D.M.
      • Niv Y.
      Context, learning, and extinction.
      ), vigor (
      • Niv Y.
      • Daw N.D.
      • Joel D.
      • Dayan P.
      Tonic dopamine: Opportunity costs and the control of response vigor.
      ,
      • Dayan P.
      Instrumental vigour in punishment and reward.
      ), and others. Perhaps the most striking success of these models is their ability to capture how pavlovian affective responses can lead to maladaptive choices (
      • Dayan P.
      • Niv Y.
      • Seymour B.
      • Daw N.D.
      The misbehavior of value and the discipline of the will.
      ,
      • Breland K.
      • Breland M.
      The misbehavior of organisms.
      ).
      Model-free approaches are very valuable to understand how the expression of affect transfers between situations with experience. Although mostly restricted to individual laboratory sessions, the underlying model likely plays an important role in explaining how individual differences in the expression of (affective) behaviors emerge over (life)time, and potentially in response to behavioral psychotherapeutic interventions. Furthermore, a hierarchical version of model-free reinforcement learning has the capacity to explain how complex high-level actions consisting of multiple correlated processes might emerge (
      • Breland K.
      • Breland M.
      The misbehavior of organisms.
      ,
      • Sutton R.S.
      • Precup D.
      • Singh S.
      Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning.
      ,
      • Botvinick M.M.
      • Niv Y.
      • Barto A.C.
      Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective.
      ), though this awaits application to the correlations among physiological, psychological, and behavioral aspects of emotions.

      Appraisals Require Model-Based Inference

      Pure model-free accounts, however, fail to explain context effects on conditioning. For instance, the physiological response to a threat differs depending on whether the animal is restrained or freely moving (
      • Iwata J.
      • LeDoux J.E.
      Dissociation of associative and nonassociative concomitants of classical fear conditioning in the freely behaving rat.
      ) as well as whether a refuge or obstacle is present and at what distance (
      • Domenici P.
      • Blagburn J.M.
      • Bacon J.P.
      Animal escapology II: Escape trajectory case studies.
      ,
      • Domenici P.
      • Blagburn J.M.
      • Bacon J.P.
      Animal escapology I: Theoretical issues and emerging trends in escape trajectories.
      ,
      • Zani P.
      • Jones T.
      • Neuhaus R.
      • Milgrom J.
      Effect of refuge distance on escape behavior of side-blotched lizards (uta stansburiana).
      ). In humans, framing the same movement as approach or withdrawal alters whether a pavlovian conditioned stimulus promotes or inhibits it (
      • Huys Q.J.M.
      • Cools R.
      • Gölzer M.
      • Friedel E.
      • Heinz A.
      • Dolan R.J.
      • et al.
      Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding.
      ). Even startle reflexes are potentiated by fear induction (
      • Grillon C.
      • Davis M.
      Fear-potentiated startle conditioning in humans: explicit and contextual cue conditioning following paired versus unpaired training.
      ), enhanced by upregulating negative emotions (
      • Jackson D.C.
      • Malmstadt J.R.
      • Larson C.L.
      • Davidson R.J.
      Suppression and enhancement of emotional responses to unpleasant pictures.
      ), and reduced by positive emotions (
      • Lang P.J.
      • Bradley M.M.
      • Cuthbert B.N.
      Emotion, attention, and the startle reflex.
      ). More fundamentally, context determines what affective behavior is emitted (

      Lazarus RS: Stress and Emotion: A New Synthesis. New York: Springer.

      ,
      • Lazarus R.S.
      • Alfert E.
      Short-circuiting of threat by experimentally altering cognitive appraisal.
      ): the same emotion of anger may motivate harm not only through physical means in a boxing ring, but also through financial transactions in a boardroom setting.
      Similarly, scalar summaries of past experience cannot account for the impact of appraisals on human emotions. The appraisal of an event involves an assessment of what the event “means” (
      • Moors A.
      • Ellsworth P.C.
      • Scherer K.R.
      • Frijda N.H.
      Appraisal theories of emotion: State of the art and future development.
      ). This interpretative process determines whether and which emotion results by inference of latent causes: a smile is pleasant if interpreted as emanating from kindness, but aversive if viewed as an expression of condescension. The fact that an event is meaningful can be inferred from changes in model-free values because these capture the expectation of future well-being in a relatively stable environment, and sudden changes in model-free expected values therefore indicate a meaningful event. This is reminiscent of the argument that changes in core affect invoke appraisals (
      • Barrett L.F.
      Are emotions natural kinds?.
      ,
      • Russell J.A.
      Core affect and the psychological construction of emotion.
      ). Certain aspects of meaning may also be precomputed and result in automatic appraisals (
      • Moors A.
      Automatic constructive appraisal as a candidate cause of emotion.
      ,
      • Daw N.D.
      • Dayan P.
      The algorithmic anatomy of model-based evaluation.
      ), but what an event means for well-being cannot be derived from model-free values. This assessment involves a series of variables such as goal congruence, controllability, and agency (
      • Moors A.
      • Ellsworth P.C.
      • Scherer K.R.
      • Frijda N.H.
      Appraisal theories of emotion: State of the art and future development.
      ,
      • Lazarus R.S.
      Emotion and Adaptation.
      ,
      • Scherer K.R.
      On the nature and function of emotion: A component process approach.
      ) that capture how the changed contingencies induced by the event and the behavior influence the controllable achievability of the goal (
      • Huys Q.J.M.
      • Dayan P.
      A Bayesian formulation of behavioral control.
      ). Goal congruence, for instance, measures how events influence the ability and cost of achieving current goals and as such involves replanning a new path toward the goal and comparing the cost of this path to that of the previous plan.
      The computation of the meaning part of the appraisal requires the integration of a model capturing an individual’s beliefs about the consequences of choices, what reinforcements will be obtained in which states, how observations relate to hidden states, and how different states relate to each other (
      • Sutton R.S.
      • Barto A.G.
      Reinforcement Learning: An Introduction.
      ,
      • Daw N.D.
      • Dayan P.
      The algorithmic anatomy of model-based evaluation.
      ,
      • Daw N.D.
      • Niv Y.
      • Dayan P.
      Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control.
      ). Inferring such values requires a model to be inverted or simulated. Mirroring the notion that some appraisals rely on rule-based processes, it suggests a role for nonautomatic components; it captures that appraisals generalize and change over time as new information is progressively integrated; it suggests how maladaptive beliefs influence emotions; and it suggests how a new understanding can alter emotions in explicit reappraisal (
      • Moors A.
      • Ellsworth P.C.
      • Scherer K.R.
      • Frijda N.H.
      Appraisal theories of emotion: State of the art and future development.
      ,
      • Lazarus R.S.
      • Alfert E.
      Short-circuiting of threat by experimentally altering cognitive appraisal.
      ,
      • Lazarus R.S.
      Emotion and Adaptation.
      ,
      • Beck A.T.
      Depression: Clinical, Experimental and Theoretical Aspects.
      ,
      • Bohus M.
      • Wolf-Arehult M.
      Interactive Skill Training for Borderline Patients [German].
      ).
      However, a measured and “rational” consideration of all possible outcomes is hardly a sufficient model of emotions (
      • Zajonc R.B.
      Feeling and thinking: Preferences need no inferences.
      ). In fact, reasoning itself is profoundly affected by emotions (
      • Beck A.T.
      Depression: Clinical, Experimental and Theoretical Aspects.
      ), as are perception, learning, and memory.

      Metareasoning

      One factor that may be useful to consider is that model-based inference is mostly impossible due to the sheer size of most relevant model-based valuation problems. Figure 1A shows a simple planning task in a maze the solution of which corresponds to a binary decision tree (Figure 1B). The best action at the root of the tree is the one that leads to the path with the best total outcome, and this may not be the action with the best immediate reward. As in reality, there is usually more than one way to the goal, but the different paths have different intermediate outcomes rendering some better than others. Despite its simplicity, humans have difficulty solving the task fully, and employ strategies to avoid evaluating the entire tree even when there are only three or four choices to go (Figure 1C). Unless they are highly constrained, such as in feedforward motor control (
      • Kawato M.
      Internal models for motor control and trajectory planning.
      ), optimal decisions in realistic situations are computationally extremely demanding.
      Figure thumbnail gr1
      Figure 1Model-based inference problems rapidly become too hard to fully solve. Consider a task (A) where subjects see six rectangles arranged hexagonally and are taught to navigate according to the underlying transition matrix indicated by the arrows, so that they can transition from each state to two successor states depending on which of two buttons they press. Every transition yields a reward or loss. Finding a path of a given length that maximizes total earnings corresponds to a tree-shaped decision problem, for instance the one in panel (B) for three transitions starting from state 3. Participants typically choose not to expend cognitive effort evaluating subtrees below a salient loss (here below transitions with –70 points), resulting in a cutting of or “pruning” of decision trees (B). (C) This in turn results in worse performance when the optimal path requires transitioning through a salient loss (red) than when it does not (orange). Black dots and gray lines show the effect in individual participants. (A) and (B) are modified with permission from Huys et al.
      (
      • Huys Q.J.M.
      • Eshel N.
      • O’Nions E.
      • Sheridan L.
      • Dayan P.
      • Roiser J.P.
      Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees.
      )
      . (C) shows data replotted from Huys et al.
      (
      • Huys Q.J.M.
      • Lally N.
      • Faulkner P.
      • Eshel N.
      • Seifritz E.
      • Gershman S.J.
      • et al.
      Interplay of approximate planning strategies.
      )
      .
      The limited resources lead to the metareasoning problem, which concerns the optimal deployment of the available computational power (
      • Simon H.A.
      Rational choice and the structure of the environment.
      ,
      • Russell S.
      • Wefald E.
      Principles of metareasoning.
      ,
      • Payne J.W.
      • Bettman J.R.
      • Luce M.F.
      When time is money: Decision behavior under opportunity-cost time pressure.
      ,
      • Lieder F.
      • Griffiths T.L.
      • Huys Q.J.
      • Goodman N.D.
      The anchoring bias reflects rational use of cognitive resources [published online ahead of print May 8].
      ). It is a decision problem about which of the various options to evaluate internally (Figure 2). Formally, the estimated value of performing a computation is the difference in expected utility between taking a choice without the additional computation, and taking a new alternative choice after having invested in the computation (
      • Russell S.
      • Wefald E.
      Principles of metareasoning.
      ,
      • Keramati M.
      • Dezfouli A.
      • Piray P.
      Speed/accuracy trade-off between the habitual and the goal-directed processes.
      ). Although this decision problem is mathematically similar to the original problem, it is different from the original problem because simulations do not actually incur the costs of the real problem, and while taking real poor actions should be avoided to avoid incurring their loss, internally simulating poor actions can be useful (
      • Bubeck S.
      • Munos R.
      • Stoltz G.
      Pure exploration in finitely-armed and continuous-armed bandits.
      ,
      • Hay N.
      • Russell S.J.
      Metareasoning for Monte Carlo tree search. Technical Report, Electrical Engineering and Computer Sciences.
      ). Thus, the states in this metareasoning problem are all possible partial trees of the original tree, which is a far larger state space than in the original problem.
      Figure thumbnail gr2
      Figure 2Metareasoning. Given a state and a sequence of possible decisions, optimal action choice involves evaluating a decision tree (top left). An agent with constrained resources faces the challenge of deciding what to simulate, i.e., what to “think about.” For instance, it could choose to first simulate the action going left, and then continue down this branch (leftmost set of trees). Alternatively, it could superficially consider the right action, and then start examining the left action. The metareasoning problem hence is a decision problem where the states are knowledge states about the decision tree, and the task is to choose to think about the components of the problem at hand in the way that is most likely to yield a good final choice.

      Emotions Implement Approximate Metareasoning Strategies

      Model-based reasoning is hence faced with two profound challenges: the size of the problem and the even harder task of apportioning limited resources in an adaptive manner. These are fundamental problems and a strategy to deal with them is mandatory. The proposal here is that emotions can implement approximate solutions to these challenges. In particular, emotional states 1) come with a strong focus on particular behaviors and 2) induce a strong perceptual and processing focus such that evaluation is concentrated on a narrow set of states. Emotions thereby effectively function as approximate metareasoning strategies that prescribe how computational resources are allocated. To what extent these approximations are adaptive depends on how they are invoked. We first provide an outline for the implementation in terms of action tendencies and state observations, and then turn to the control issues.

      Action Tendencies

      One of the features of emotions about which there is more agreement is that they prioritize certain actions (
      • Izard C.E.
      Basic emotions, natural kinds, emotion schemas, and a new paradigm.
      ,
      • Moors A.
      • Ellsworth P.C.
      • Scherer K.R.
      • Frijda N.H.
      Appraisal theories of emotion: State of the art and future development.
      ,
      • James W.
      The Principles of Psychology.
      ,
      • Posner J.
      • Russell J.A.
      • Peterson B.S.
      The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology.
      ,
      • Lang P.J.
      • Bradley M.M.
      Appetitive and defensive motivation: Goal-directed or goal-determined?.
      ). Constraining the action space can substantially simplify the valuation problem because the computational cost is exponential in the size of the action space.
      At an abstract level, emotional states are accompanied by distinct and richly experienced urges toward particular classes of actions. Frijda et al. (
      • Frijda N.H.
      • Kuipers P.
      • Ter Schure E.
      Relations among emotion, appraisal, and emotional action readiness.
      ) asked people to remember events of particular emotions and then to rate a list or 26 items about the kinds of behaviors they wanted to engage in, such as “I gave up,” “I wanted to protect myself from someone or something,” or “I wanted to help someone, to take care of someone.” From the ratings of these statements, the emotion characterizing the episode could be reliably recovered. Though very abstract, such rich descriptions are also important in psychotherapeutic settings. In dialectic behavior therapy, individuals are initially taught to recognize emotions by the action tendencies they feel (
      • Bohus M.
      • Wolf-Arehult M.
      Interactive Skill Training for Borderline Patients [German].
      ).
      Emotions also induce physiological and vegetative changes. However, physiological signatures of emotions do not appear to readily differentiate between categorically defined emotions, but rather provide a few classes of general action preparations [(
      • James W.
      The Principles of Psychology.
      ,
      • Barrett L.F.
      • Ochsner K.N.
      • Gross J.J.
      On the automaticity of emotion.
      ,
      • Friedman B.H.
      Feelings and the body: The Jamesian perspective on autonomic specificity of emotion.
      ,
      • Lang P.J.
      • Bradley M.M.
      Emotion and the motivational brain.
      ), though see (
      • Stephens C.L.
      • Christie I.C.
      • Friedman B.H.
      Autonomic specificity of basic emotions: Evidence from pattern classification and cluster analysis.
      )]. A preparatory increase in heart rate to compensate for the anticipated drop in peripheral resistance upon supplying blood to large muscle groups is required when running, be it for escape or fun. As such, these can be seen as a preparation toward a class of behaviors that share physiological requirements.

      State Observation

      The complexity of model-based evaluation is also exponential in the range of states considered. There is ample evidence for emotion- or mood-congruent processing biases (
      • MacLeod C.
      • Mathews A.
      • Tata P.
      Attentional bias in emotional disorders.
      ,
      • Mathews A.
      • MacLeod C.
      Cognitive vulnerability to emotional disorders.
      ). For instance, Bradley et al. (
      • Bradley B.P.
      • Mogg K.
      • Lee S.C.
      Attentional biases for negative information in induced and naturally occurring dysphoria.
      ) showed that exposure to sad music and recollection of sad memories produces an attentional bias toward sad words, and such biases arise from the emotional state rather than purely from the exposure to the emotional word (
      • Gilboa-Schechtman E.
      • Revelle W.
      • Gotlib I.H.
      Stroop interference following mood induction: Emotionality, mood congruence, and concern relevance.
      ). By restricting attention to particular states and disregarding others, the problem could again be reduced in size (
      • Dean T.
      • Kaelbling L.P.
      • Kirman J.
      • Nicholson A.
      Planning under time constraints in stochastic domains.
      ), for instance by pruning (
      • Huys Q.J.M.
      • Eshel N.
      • O’Nions E.
      • Sheridan L.
      • Dayan P.
      • Roiser J.P.
      Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees.
      ) searches along branches of the decision tree that result in states outside the attentional focus.
      A further aspect is that there is usually uncertainty about the state. This profoundly complicates the computational task of valuation because policies for the various possible states have to be computed (
      • Kaelbling L.P.
      • Littman M.L.
      • Cassandra A.R.
      Planning and acting in partially observable stochastic domains.
      ). By ascertaining the state, this complexity can be reduced. Introspection about the state of the body likely plays a particularly important role: the impact of a muscle’s activation depends on the joint position and the chances of success in a fight are reduced when already wounded.

      Controlling Metareasoning Strategies

      If there are multiple approximate metareasoning strategies, then there must be some control over which is deployed when. The first source of control is likely evolutionary, where species-specific responses provide a (potentially very strong) bias toward evaluating particular actions, rather than toward emitting the action entirely inflexibly. This allows for the kinds of context effects on even innate responses mentioned above.
      The second source of control could be, confusingly, model-free. Performers may learn from experience that a certain amount of catastrophizing improves their performance (
      • Wolfe M.L.
      Correlates of adaptive and maladaptive musical performance anxiety.
      ), without an understanding of why that is. Etkin et al. (
      • Etkin A.
      • Büchel C.
      • Gross J.J.
      The neural bases of emotion regulation.
      ) have recently argued for a model-free component in serial adaptations in the emotional conflict task, where individuals have to indicate the facial expression (fear or happy) of a face with either the matching or conflicting word superimposed over it (
      • Etkin A.
      • Egner T.
      • Peraza D.M.
      • Kandel E.R.
      • Hirsch J.
      Resolving emotional conflict: A role for the rostral anterior cingulate cortex in modulating activity in the amygdala.
      ). Model-free learning has been argued to account for learning in strategy selection (
      • Marewski J.N.
      • Link D.
      Strategy selection: An introduction to the modeling challenge.
      ): with repeated experience, individuals can slowly increase their frequency of using adaptive strategies for solving problems (
      • Rieskamp J.
      • Otto P.E.
      SSL: A theory of how people learn to select strategies.
      ). We have recently shown how the results of costly model-based evaluations are memorized and simply replayed upon repeated encounter of the same problem in a process called memoization (
      • Huys Q.J.M.
      • Lally N.
      • Faulkner P.
      • Eshel N.
      • Seifritz E.
      • Gershman S.J.
      • et al.
      Interplay of approximate planning strategies.
      ) that gives rise to decision-making biases that are characteristic of the individual but highly variable across the population.
      The third evaluative process for emotions allows for knowledge to be incorporated in the form of heuristics. Research on decisions about options with many attributes [for instance cars, with price, speed, size, brand, etc. (
      • Gigerenzer G.
      • Goldstein D.G.
      Reasoning the fast and frugal way: Models of bounded rationality.
      )] have identified a host of different strategies. “Take the best” is appropriate in noncompensatory environments where one feature is most informative and can be used alone to rank options. In compensatory environments, humans spend more time and cognitive effort on examining multiple features and integrating the information, but only if they are not under time pressure (
      • Newell B.R.
      • Shanks D.R.
      Take the best or look at the rest? Factors influencing “one-reason” decision making.
      ). This suggests that individuals can access approximate measures of how adaptive a particular cognitive strategy is, and use this to guide their choice (
      • Lieder F.
      • Plunkett D.
      • Hamrick J.B.
      • Russell S.J.
      • Hay N.
      • Griffiths T.
      Algorithm selection by rational metareasoning as a model of human strategy selection.
      ). In the affective domain, misguided beliefs or schemas (
      • Beck A.T.
      Cognitive models of depression.
      ) about the adaptiveness of strategies relate to a number of pathological emotion regulation phenomena. Pathological worry in generalized anxiety disorders (
      • Wells A.
      A metacognitive model and therapy for generalized anxiety disorder.
      ,
      • Borkovec T.
      • Ray W.J.
      • Stober J.
      Worry: A cognitive phenomenon intimately linked to affective, physiological, and interpersonal behavioral processes.
      ) and rumination in depressive disorders (
      • Treynor W.
      • Gonzalez R.
      • Nolen-Hoeksema S.
      Rumination reconsidered: A psychometric analysis.
      ) are maintained by explicit beliefs about the usefulness of worry and rumination, respectively. People who dislike emotion regulation are more likely to respond with anger to provocation (
      • Mauss I.B.
      • Evers C.
      • Wilhelm F.H.
      • Gross J.J.
      How to bite your tongue without blowing your top: Implicit evaluation of emotion regulation predicts affective responding to anger provocation.
      ). Depressed persons are not impaired at emotion regulation strategies such as positive imagery to improve their mood, but they have a reduced tendency to employ them (
      • Ehring T.
      • Tuschen-Caffier B.
      • Schnlle J.
      • Fischer S.
      • Gross J.J.
      Emotion regulation and vulnerability to depression: Spontaneous versus instructed use of emotion suppression and reappraisal.
      ).
      The fourth evaluative process, again confusingly, could be model-based, where the precise consequences of particular emotions are examined and evaluated. This is rarely feasible and probably only commonly done in situational analyses in psychotherapy, where emotions, thoughts, behavior, and consequences are explicitly discussed (
      • Bohus M.
      • Wolf-Arehult M.
      Interactive Skill Training for Borderline Patients [German].
      ,
      • Beck A.T.
      Cognitive models of depression.
      ,
      • Gross J.J.
      Antecedent- and response-focused emotion regulation: Divergent consequences for experience, expression, and physiology.
      ). This allows patients to learn to consciously and explicitly assess whether a particular emotion is appropriate and helpful in a given situation, and to adapt it by using reappraisal and other emotion regulation strategies if necessary.

      Discussion

      We have attempted to sketch out a valuation framework for emotions. We have argued why appraisals point toward a model-based framework; how emotions may have a potentially important role in facilitating model-based decisions by functioning as internal strategies to allocate computational resources; how emotions’ adaptive nature depends on their deployment; and how a variety of different processes can lead to adaptive or maladaptive deployment of emotion strategies.
      Three desiderata for a computational framework of emotions were put forth. The first was the at best partially correlated nature of physiological, psychological, and behavioral features. The flexibility the proposed framework allows for contrasts with the view of basic emotions as relatively fixed behavioral and physiological action packages. As such, it reflects the lack of identifiably discrete physiological or behavioral patterns or single neurobiological cause (
      • Barrett L.F.
      Are emotions natural kinds?.
      ,
      • Cacioppo J.T.
      • Berntson G.G.
      • Larsen J.T.
      • Poehlmann K.M.
      • Ito T.A.
      • et al.
      The psychophysiology of emotion.
      ,
      • Wager T.D.
      • Kang J.
      • Johnson T.D.
      • Nichols T.E.
      • Satpute A.B.
      • Barrett L.F.
      A Bayesian model of category-specific emotional brain responses.
      ). Similar to other proposals, it emphasizes the importance of emotional processes in more complex decision-making settings (
      • Forgas J.P.
      Mood and judgment: The affect infusion model (aim).
      ). Space constraints have prevented us from exploring the distinction between valence and arousal important to circumplex and core affect models, but this might naturally emerge from valuation in continuous-time settings, where the rate at which actions are emitted depends on the average reward rate in the environment, albeit in sometimes complex ways (
      • Bach D.R.
      • Dayan P.
      Algorithms for survival: A comparative perspective on emotions.
      ,
      • Niv Y.
      • Daw N.D.
      • Joel D.
      • Dayan P.
      Tonic dopamine: Opportunity costs and the control of response vigor.
      ,
      • Dayan P.
      Instrumental vigour in punishment and reward.
      ,
      • Constantino S.M.
      • Daw N.D.
      Learning the opportunity cost of time in a patch-foraging task.
      ). Notably, the current proposal allows for mixed emotions through a combination of metareasoning policies.
      The second desideratum was the ability to account for appraisal and contextual effects. The complexity of the model-based valuation required for this led to the notion of approximate metareasoning strategies. These approximate strategies are necessarily often suboptimal and may capture the prototypical adverse influences of emotion on cognition (
      • Loewenstein G.
      Out of control: Visceral influences on behavior.
      ). The focus on valuation is compatible with models emphasizing prediction (
      • Barrett L.F.
      The theory of constructed emotion: An active inference account of interoception and categorization.
      ), but distinct in that it suggests that the relevant predictions must be about long-term utility, and that emotions play a key role in facilitating such predictions, albeit approximately.
      The third desideratum concerned the nature of conscious qualia of emotions. Doing so fully awaits a theory of consciousness. However, two aspects are interesting to consider. First, Dehaene and Naccache’s (
      • Dehaene S.
      • Naccache L.
      Toward a cognitive neuroscience of consciousness: Basic evidence and a workspace framework.
      ) notion of a global workspace sits naturally with the notion of metareasoning. Situations with a high estimated value of computation should recruit neural resources more extensively, and hence be more likely to involve the brain-wide states postulated as representing the global workspace. Second, it has been suggested that the component processes in verbal self-report involve an interoceptive component followed by a classification process (
      • Barrett L.F.
      Solving the emotion paradox: Categorization and the experience of emotion.
      ). In our proposal, the metareasoning strategies would profoundly influence what information was processed, and as such may strongly determine future classification. Interestingly, the classification process has been suggested to involve the ventrolateral prefrontal cortex (
      • Lieberman M.D.
      • Eisenberger N.I.
      • Crockett M.J.
      • Tom S.M.
      • Pfeifer J.H.
      • Way B.M.
      Putting feelings into words: Affect labeling disrupts amygdala activity in response to affective stimuli.
      ,
      • Satpute A.B.
      • Shu J.
      • Weber J.
      • Roy M.
      • Ochsner K.N.
      The functional neural architecture of self-reports of affective experience.
      ), which is also known to mediate arbitration between valuation strategies (
      • Lee S.W.
      • Shimojo S.
      • O’Doherty J.P.
      Neural computations underlying arbitration between model-based and model-free learning.
      ).
      The framework laid out here makes a number of testable predictions. First, the argument that appraisal involves model-based reasoning means that it should be influenced by cognitive, endocrine, and neuromodulatory variables known to influence model-based reasoning (
      • Otto A.R.
      • Gershman S.J.
      • Markman A.B.
      • Daw N.D.
      The curse of planning: Dissecting multiple reinforcement-learning systems by taxing the central executive.
      ,
      • Otto A.R.
      • Raio C.M.
      • Chiang A.
      • Phelps E.A.
      • Daw N.D.
      Working-memory capacity protects model-based learning from stress.
      ,
      • Deserno L.
      • Huys Q.J.M.
      • Boehme R.
      • Buchert R.
      • Heinze H.J.
      • Grace A.A.
      • et al.
      Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making.
      ,
      • Wunderlich K.
      • Smittenaar P.
      • Dolan R.J.
      Dopamine enhances model-based over model-free choice behavior.
      ). Second, we emphasized the importance of the control of metareasoning strategies and suggested that it may be subject to substantial malleability. This in turn predicts that by training particular metareasoning strategies it should be possible to selectively facilitate certain emotions over others. Third, it suggests that emotions do not represent valuations themselves, but rather that they determine the process by which valuation occurs. This predicts an influence of emotion on search in a relevant model-based task rather than on values directly. The main challenge for the framework is that it points to the critical importance of understanding both the internal models of individuals and their strategies in searching them. Measuring these is a difficult scientific problem. Though this is as yet not feasible, recent advances including whole-brain mapping of semantic representations (
      • Huth A.G.
      • de Heer W.A.
      • Griffiths T.L.
      • Theunissen F.E.
      • Gallant J.L.
      Natural speech reveals the semantic maps that tile human cerebral cortex.
      ,
      • Huth A.G.
      • Lee T.
      • Nishimoto S.
      • Bilenko N.Y.
      • Vu A.T.
      • Gallant J.L.
      Decoding the semantic content of natural movies from human brain activity.
      ) combined with active and passive sensing using mobile devices (
      • Torous J.
      • Kiang M.V.
      • Lorme J.
      • Onnela J.P.
      New tools for new research in psychiatry: A scalable and customizable platform to empower data driven smartphone research.
      ,
      • Baker J.
      • Mueller N.
      • Onnela J.P.
      • Ongur D.
      • Buckner R.
      368-deep dynamic phenotyping: Neural changes underlying fluctuations in bipolar disorder over one year.
      ) should open promising avenues.
      The emphasis on model-based processes was partially motivated by the finding that model-free measures of reward processing and learning are unimpaired in depression (
      • Huys Q.J.M.
      • Dayan P.
      • Daw
      Depression: A decision-theoretic account.
      ,
      • Rutledge R.B.
      • Moutoussis M.
      • Smittenaar P.
      • Zeidman P.
      • Taylor T.
      • Hrynkiewicz L.
      • et al.
      Association of neural and emotional impacts of reward prediction errors with major depression.
      ), and as such this work is an effort to start integrating cognitive phenomena of clinical importance like dysfunctional attitudes (

      Weissman A, Beck A (1978): Development and validation of the Dysfunctional Attitude Scale. Presented at the Annual Meeting of the Association for the Advancement of Behavior Therapy, Chicago.

      ), helplessness (
      • Huys Q.J.M.
      • Dayan P.
      A Bayesian formulation of behavioral control.
      ), attributional/cognitive styles (
      • Peterson C.
      • Semmel A.
      • von Baeyer C.
      • Abramson L.
      • Metalsky G.
      • Seligman M.
      The attributional Style Questionnaire.
      ,
      • Abramson L.Y.
      • Alloy L.B.
      • Hogan M.E.
      • Whitehouse W.G.
      • Cornette M.
      • Akhavan S.
      • et al.
      Suicidality and cognitive vulnerability to depression among college students: A prospective study.
      ), and appraisals (
      • Moors A.
      • Ellsworth P.C.
      • Scherer K.R.
      • Frijda N.H.
      Appraisal theories of emotion: State of the art and future development.
      ) into a valuation framework.

      Acknowledgments and Disclosures

      This work was supported by Swiss National Science Foundation Grant No. 320030L_153449/1 (to QJMH).
      We thank Peter Dayan for comments on an earlier version of this manuscript.
      The authors report no biomedical financial interests or potential conflicts of interest.

      References

        • Huys Q.J.M.
        • Moutoussis M.
        • Williams J.
        Are computational models of any use to psychiatry?.
        Neural Netw. 2011; 24: 544-551
        • Maia T.V.
        • Frank M.J.
        From reinforcement learning models to psychiatric and neurological disorders.
        Nat Neurosci. 2011; 14: 154-162
        • Montague P.R.
        • Dolan R.J.
        • Friston K.J.
        • Dayan P.
        Computational psychiatry.
        Trends Cogn Sci. 2012; 16: 72-80
        • Stephan K.E.
        • Mathys C.
        Computational approaches to psychiatry.
        Curr Opin Neurobiol. 2014; 25: 85-92
        • Huys Q.J.M.
        • Maia T.V.
        • Frank M.J.
        Computational psychiatry as a bridge from neuroscience to clinical applications.
        Nat Neurosci. 2016; 19: 404-413
        • Whiteford H.A.
        • Degenhardt L.
        • Rehm J.
        • Baxter A.J.
        • Ferrari A.J.
        • Erskine H.E.
        • et al.
        Global burden of disease attributable to mental and substance use disorders: Findings from the global burden of disease study 2010.
        Lancet. 2013; 382: 1575-1586
        • Simon H.A.
        Rational choice and the structure of the environment.
        Psychol Rev. 1956; 63: 129-138
        • Russell S.
        • Wefald E.
        Principles of metareasoning.
        Artif Intell. 1991; 49: 361-395
        • Payne J.W.
        • Bettman J.R.
        • Johnson E.J.
        The Adaptive Decision Maker.
        Cambridge University Press, New York1993
        • Barrett L.F.
        Are emotions natural kinds?.
        Perspect Psychol Sci. 2006; 1: 28-58
        • Panksepp J.
        Neurologizing the psychology of affects: How appraisal-based constructivism and basic emotion theory can coexist.
        Perspect Psychol Sci. 2007; 2: 281-296
        • Ekman P.
        An argument for basic emotions.
        Cogn Emot. 1992; 6: 169-200
        • Izard C.E.
        Basic emotions, natural kinds, emotion schemas, and a new paradigm.
        Perspect Psychol Sci. 2007; 2: 260-280
        • LeDoux J.E.
        Coming to terms with fear.
        Proc Natl Acad Sci U S A. 2014; 111: 2871-2878
        • Raio C.M.
        • Carmel D.
        • Carrasco M.
        • Phelps E.A.
        Nonconscious fear is quickly acquired but swiftly forgotten.
        Curr Biol. 2012; 22: R477-R479
        • Knight D.C.
        • Waters N.S.
        • Bandettini P.A.
        Neural substrates of explicit and implicit fear memory.
        Neuroimage. 2009; 45: 208-214
        • Feinstein J.S.
        • Buzza C.
        • Hurlemann R.
        • Follmer R.L.
        • Dahdaleh N.S.
        • Coryell W.H.
        • et al.
        Fear and panic in humans with bilateral amygdala damage.
        Nat Neurosci. 2013; 16: 270-272
        • Khalsa S.S.
        • Feinstein J.S.
        • Li W.
        • Feusner J.D.
        • Adolphs R.
        • Hurlemann R.
        Panic anxiety in humans with bilateral amygdala lesions: Pharmacological induction via cardiorespiratory interoceptive pathways.
        J Neurosci. 2016; 36: 3559-3566
        • Roseman I.J.
        • Smith C.A.
        Appraisal Theory.
        Oxford University Press, Oxford, UK2001
        • Frijda N.H.
        • Kuipers P.
        • Ter Schure E.
        Relations among emotion, appraisal, and emotional action readiness.
        J Pers Soc Psychol. 1989; 57: 212-228
      1. Lazarus RS: Stress and Emotion: A New Synthesis. New York: Springer.

        • Ellsworth P.C.
        Appraisal theory: Old and new questions.
        Emot Rev. 2013; 5: 125-131
        • Scherer K.R.
        What are emotions? And how can they be measured?.
        Soc Sci Information. 2005; 44: 695-729
        • Cacioppo J.T.
        • Berntson G.G.
        • Larsen J.T.
        • Poehlmann K.M.
        • Ito T.A.
        • et al.
        The psychophysiology of emotion.
        in: Lewis R. Haviland-Jones J.M. Handbook of Emotions, 2nd ed. Guilford Press, New York2000: 173-191
        • Jack R.E.
        • Garrod O.G.B.
        • Yu H.
        • Caldara R.
        • Schyns P.G.
        Facial expressions of emotion are not culturally universal.
        Proc Natl Acad Sci U S A. 2012; 109: 7241-7244
        • Stephens C.L.
        • Christie I.C.
        • Friedman B.H.
        Autonomic specificity of basic emotions: Evidence from pattern classification and cluster analysis.
        Biol Psychol. 2010; 84: 463-473
        • Kragel P.A.
        • LaBar K.S.
        Multivariate neural biomarkers of emotional states are categorically distinct.
        Soc Cogn Affect Neurosci. 2015; 10: 1437-1448
        • Saarimki H.
        • Gotsopoulos A.
        • Jskelinen I.P.
        • Lampinen J.
        • Vuilleumier P.
        • Hari R.
        • et al.
        Discrete neural signatures of basic emotions.
        Cereb Cortex. 2016; 26: 2563-2573
        • Wager T.D.
        • Kang J.
        • Johnson T.D.
        • Nichols T.E.
        • Satpute A.B.
        • Barrett L.F.
        A Bayesian model of category-specific emotional brain responses.
        PLoS Comput Biol. 2015; 11: e1004066
        • Barrett L.F.
        Solving the emotion paradox: Categorization and the experience of emotion.
        Pers Soc Psychol Rev. 2006; 10: 20-46
        • Lieberman M.D.
        • Eisenberger N.I.
        • Crockett M.J.
        • Tom S.M.
        • Pfeifer J.H.
        • Way B.M.
        Putting feelings into words: Affect labeling disrupts amygdala activity in response to affective stimuli.
        Psychol Sci. 2007; 18: 421-428
        • Satpute A.B.
        • Shu J.
        • Weber J.
        • Roy M.
        • Ochsner K.N.
        The functional neural architecture of self-reports of affective experience.
        Biol Psychiatry. 2013; 73: 631-638
        • Russell J.A.
        Core affect and the psychological construction of emotion.
        Psychol Rev. 2003; 110: 145-172
        • Wilson-Mendenhall C.D.
        • Barrett L.F.
        • Barsalou L.W.
        Neural evidence that human emotions share core affective properties.
        Psychol Sci. 2013; 24: 947-956
        • Moors A.
        • Ellsworth P.C.
        • Scherer K.R.
        • Frijda N.H.
        Appraisal theories of emotion: State of the art and future development.
        Emot Rev. 2013; 5: 119-124
        • Bach D.R.
        • Dayan P.
        Algorithms for survival: A comparative perspective on emotions.
        Nat Rev Neurosci. 2017; 18: 311-319
        • Sutton R.S.
        • Barto A.G.
        Reinforcement Learning: An Introduction.
        MIT Press, Cambridge, MA1998
        • Hirsch S.
        • Bolles R.
        On the ability of prey to recognize predators.
        Z Tierpsychol. 1980; 54: 71-84
        • Bolles R.C.
        Species-specific defense reactions and avoidance learning.
        Psychol Rev. 1970; 77: 32-48
        • Seligman M.E.
        On the generality of the laws of learning.
        Psychol Rev. 1970; 77: 406-418
        • Timberlake W.
        • Wahl G.
        • King D.
        Stimulus and response contingencies in the misbehavior of rats.
        J Exp Psychol Anim Behav Process. 1982; 8: 62-85
        • Huys Q.J.M.
        • Cools R.
        • Gölzer M.
        • Friedel E.
        • Heinz A.
        • Dolan R.J.
        • et al.
        Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding.
        PLoS Comput Biol. 2011; 7: e1002028
        • Dayan P.
        • Niv Y.
        • Seymour B.
        • Daw N.D.
        The misbehavior of value and the discipline of the will.
        Neural Netw. 2006; 19: 1153-1160
        • Huys Q.J.M.
        • Eshel N.
        • O’Nions E.
        • Sheridan L.
        • Dayan P.
        • Roiser J.P.
        Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees.
        PLoS Comput Biol. 2012; 8: e1002410
        • Seymour B.
        • O’Doherty J.P.
        • Koltzenburg M.
        • Wiech K.
        • Frackowiak R.
        • Friston K.
        • et al.
        Opponent appetitive-aversive neural processes underlie predictive learning of pain relief.
        Nat Neurosci. 2005; 8: 1234-1240
        • Lesaint F.
        • Sigaud O.
        • Flagel S.B.
        • Robinson T.E.
        • Khamassi M.
        Modelling individual differences in the form of pavlovian conditioned approach responses: A dual learning systems approach with factored representations.
        PLoS Comput Biol. 2014; 10: e1003466
        • Moutoussis M.
        • Bentall R.P.
        • Williams J.
        • Dayan P.
        A temporal difference account of avoidance learning.
        Network. 2008; 19: 137-160
        • Maia T.V.
        Two-factor theory, the actor-critic model, and conditioned avoidance.
        Learn Behav. 2010; 38: 50-67
        • Gershman S.J.
        • Blei D.M.
        • Niv Y.
        Context, learning, and extinction.
        Psychol Rev. 2010; 117: 197-209
        • Niv Y.
        • Daw N.D.
        • Joel D.
        • Dayan P.
        Tonic dopamine: Opportunity costs and the control of response vigor.
        Psychopharmacology (Berl). 2007; 191: 507-520
        • Dayan P.
        Instrumental vigour in punishment and reward.
        Eur J Neurosci. 2012; 35: 1152-1168
        • Breland K.
        • Breland M.
        The misbehavior of organisms.
        Am Psychol. 1961; 16: 681-684
        • Sutton R.S.
        • Precup D.
        • Singh S.
        Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning.
        Artif Intell. 1999; 112: 181-211
        • Botvinick M.M.
        • Niv Y.
        • Barto A.C.
        Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective.
        Cognition. 2009; 113: 262-280
        • Huys Q.J.M.
        • Lally N.
        • Faulkner P.
        • Eshel N.
        • Seifritz E.
        • Gershman S.J.
        • et al.
        Interplay of approximate planning strategies.
        Proc Natl Acad Sci U S A. 2015; 112: 3098-3103
        • Iwata J.
        • LeDoux J.E.
        Dissociation of associative and nonassociative concomitants of classical fear conditioning in the freely behaving rat.
        Behav Neurosci. 1988; 102: 66-76
        • Domenici P.
        • Blagburn J.M.
        • Bacon J.P.
        Animal escapology II: Escape trajectory case studies.
        J Exp Biol. 2011; 214: 2474-2494
        • Domenici P.
        • Blagburn J.M.
        • Bacon J.P.
        Animal escapology I: Theoretical issues and emerging trends in escape trajectories.
        J Exp Biol. 2011; 214: 2463-2473
        • Zani P.
        • Jones T.
        • Neuhaus R.
        • Milgrom J.
        Effect of refuge distance on escape behavior of side-blotched lizards (uta stansburiana).
        Can J Zoology. 2009; 87: 407-414
        • Grillon C.
        • Davis M.
        Fear-potentiated startle conditioning in humans: explicit and contextual cue conditioning following paired versus unpaired training.
        Psychophysiology. 1997; 34: 451-458
        • Jackson D.C.
        • Malmstadt J.R.
        • Larson C.L.
        • Davidson R.J.
        Suppression and enhancement of emotional responses to unpleasant pictures.
        Psychophysiology. 2000; 37: 515-522
        • Lang P.J.
        • Bradley M.M.
        • Cuthbert B.N.
        Emotion, attention, and the startle reflex.
        Psychol Rev. 1990; 97: 377-395
        • Lazarus R.S.
        • Alfert E.
        Short-circuiting of threat by experimentally altering cognitive appraisal.
        J Abnorm Psychol. 1964; 69: 195-205
        • Moors A.
        Automatic constructive appraisal as a candidate cause of emotion.
        Emot Rev. 2010; 2: 139-156
        • Daw N.D.
        • Dayan P.
        The algorithmic anatomy of model-based evaluation.
        Philos Trans R Soc Lond B Biol Sci. 2014; 369: 20130478
        • Lazarus R.S.
        Emotion and Adaptation.
        Oxford University Press, Oxford, UK1994
        • Scherer K.R.
        On the nature and function of emotion: A component process approach.
        in: Scherer K.R. Ekman P. Approaches to Emotion. Erlbaum, Hillsdale, NJ1984: 293-317
        • Huys Q.J.M.
        • Dayan P.
        A Bayesian formulation of behavioral control.
        Cognition. 2009; 113: 314-328
        • Daw N.D.
        • Niv Y.
        • Dayan P.
        Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control.
        Nat Neurosci. 2005; 8: 1704-1711
        • Beck A.T.
        Depression: Clinical, Experimental and Theoretical Aspects.
        Harper & Row, New York1967
        • Bohus M.
        • Wolf-Arehult M.
        Interactive Skill Training for Borderline Patients [German].
        Schattauer Verlag, Stuttgart, Germany2013
        • Zajonc R.B.
        Feeling and thinking: Preferences need no inferences.
        Am Psychol. 1980; 35: 151
        • Kawato M.
        Internal models for motor control and trajectory planning.
        Curr Opin Neurobiol. 1999; 9: 718-727
        • Payne J.W.
        • Bettman J.R.
        • Luce M.F.
        When time is money: Decision behavior under opportunity-cost time pressure.
        Organ Behav Hum Dec Process. 1996; 66: 131-152
        • Lieder F.
        • Griffiths T.L.
        • Huys Q.J.
        • Goodman N.D.
        The anchoring bias reflects rational use of cognitive resources [published online ahead of print May 8].
        Psychon Bull Rev. 2017;
        • Keramati M.
        • Dezfouli A.
        • Piray P.
        Speed/accuracy trade-off between the habitual and the goal-directed processes.
        PLoS Comput Biol. 2011; 7: e1002055
        • Bubeck S.
        • Munos R.
        • Stoltz G.
        Pure exploration in finitely-armed and continuous-armed bandits.
        Theor Comput Sci. 2011; 412: 1832-1852
        • Hay N.
        • Russell S.J.
        Metareasoning for Monte Carlo tree search. Technical Report, Electrical Engineering and Computer Sciences.
        University of California at Berkeley, Berkeley, CA2011
        • James W.
        The Principles of Psychology.
        Dover, New York1980
        • Posner J.
        • Russell J.A.
        • Peterson B.S.
        The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology.
        Dev Psychopathol. 2005; 17: 715-734
        • Lang P.J.
        • Bradley M.M.
        Appetitive and defensive motivation: Goal-directed or goal-determined?.
        Emot Rev. 2013; 5: 230-234
        • Barrett L.F.
        • Ochsner K.N.
        • Gross J.J.
        On the automaticity of emotion.
        in: Bargh J.A. Social Psychology and the Unconscious: The Automaticity of Higher Mental Processes. Psychology Press, Suffolk, UK2007: 173-217
        • Friedman B.H.
        Feelings and the body: The Jamesian perspective on autonomic specificity of emotion.
        Biol Psychol. 2010; 84: 383-393
        • Lang P.J.
        • Bradley M.M.
        Emotion and the motivational brain.
        Biol Psychol. 2010; 84: 437-450
        • MacLeod C.
        • Mathews A.
        • Tata P.
        Attentional bias in emotional disorders.
        J Abnorm Psychol. 1986; 95: 15-20
        • Mathews A.
        • MacLeod C.
        Cognitive vulnerability to emotional disorders.
        Annu Rev Clin Psychol. 2005; 1: 167-195
        • Bradley B.P.
        • Mogg K.
        • Lee S.C.
        Attentional biases for negative information in induced and naturally occurring dysphoria.
        Behav Res Ther. 1997; 35: 911-927
        • Gilboa-Schechtman E.
        • Revelle W.
        • Gotlib I.H.
        Stroop interference following mood induction: Emotionality, mood congruence, and concern relevance.
        Cogn Therapy Res. 2000; 24: 491-502
        • Dean T.
        • Kaelbling L.P.
        • Kirman J.
        • Nicholson A.
        Planning under time constraints in stochastic domains.
        Artif Intell. 1995; 76: 35-74
        • Kaelbling L.P.
        • Littman M.L.
        • Cassandra A.R.
        Planning and acting in partially observable stochastic domains.
        Artif Intell. 1998; 101: 99-134
        • Wolfe M.L.
        Correlates of adaptive and maladaptive musical performance anxiety.
        Med Prob Perform Artists. 1989; 4: 49-56
        • Etkin A.
        • Büchel C.
        • Gross J.J.
        The neural bases of emotion regulation.
        Nat Rev Neurosci. 2015; 16: 693-700
        • Etkin A.
        • Egner T.
        • Peraza D.M.
        • Kandel E.R.
        • Hirsch J.
        Resolving emotional conflict: A role for the rostral anterior cingulate cortex in modulating activity in the amygdala.
        Neuron. 2006; 51: 871-882
        • Marewski J.N.
        • Link D.
        Strategy selection: An introduction to the modeling challenge.
        Wiley Interdiscip Rev Cogn Sci. 2014; 5: 39-59
        • Rieskamp J.
        • Otto P.E.
        SSL: A theory of how people learn to select strategies.
        J Exp Psychol Gen. 2006; 135: 207-236
        • Gigerenzer G.
        • Goldstein D.G.
        Reasoning the fast and frugal way: Models of bounded rationality.
        Psychol Rev. 1996; 103: 650-669
        • Newell B.R.
        • Shanks D.R.
        Take the best or look at the rest? Factors influencing “one-reason” decision making.
        J Exp Psychol Learn Mem Cogn. 2003; 29: 53-65
        • Lieder F.
        • Plunkett D.
        • Hamrick J.B.
        • Russell S.J.
        • Hay N.
        • Griffiths T.
        Algorithm selection by rational metareasoning as a model of human strategy selection.
        in: Ghahramani Z. Welling M. Cortes C. Lawrence N.D. Weinberger K.Q. Advances in Neural Information Processing Systems 27: 28th Annual Conference on Neural Information Processing Systems. Curran Associates, Red Hook, NY2014: 2870-2878
        • Beck A.T.
        Cognitive models of depression.
        J Cog Psychotherapy Int Quart. 1987; 1: 5-37
        • Wells A.
        A metacognitive model and therapy for generalized anxiety disorder.
        Clin Psychol Psychother. 1999; 6: 86-95
        • Borkovec T.
        • Ray W.J.
        • Stober J.
        Worry: A cognitive phenomenon intimately linked to affective, physiological, and interpersonal behavioral processes.
        Cogn Therapy Res. 1998; 22: 561-576
        • Treynor W.
        • Gonzalez R.
        • Nolen-Hoeksema S.
        Rumination reconsidered: A psychometric analysis.
        Cogn Therapy Res. 2003; 27: 247-259
        • Mauss I.B.
        • Evers C.
        • Wilhelm F.H.
        • Gross J.J.
        How to bite your tongue without blowing your top: Implicit evaluation of emotion regulation predicts affective responding to anger provocation.
        Pers Soc Psychol Bull. 2006; 32: 589-602
        • Ehring T.
        • Tuschen-Caffier B.
        • Schnlle J.
        • Fischer S.
        • Gross J.J.
        Emotion regulation and vulnerability to depression: Spontaneous versus instructed use of emotion suppression and reappraisal.
        Emotion. 2010; 10: 563-572
        • Gross J.J.
        Antecedent- and response-focused emotion regulation: Divergent consequences for experience, expression, and physiology.
        J Pers Soc Psychol. 1998; 74: 224-237
        • Forgas J.P.
        Mood and judgment: The affect infusion model (aim).
        Psychol Bull. 1995; 117: 39-66
        • Constantino S.M.
        • Daw N.D.
        Learning the opportunity cost of time in a patch-foraging task.
        Cogn Affect Behav Neurosci. 2015; 15: 837-853
        • Loewenstein G.
        Out of control: Visceral influences on behavior.
        Org Behav Hum Decis Process. 1996; 65: 272-292
        • Barrett L.F.
        The theory of constructed emotion: An active inference account of interoception and categorization.
        Soc Cogn Affect Neurosci. 2017; 12: 1-23
        • Dehaene S.
        • Naccache L.
        Toward a cognitive neuroscience of consciousness: Basic evidence and a workspace framework.
        Cognition. 2001; 79: 1-37
        • Lee S.W.
        • Shimojo S.
        • O’Doherty J.P.
        Neural computations underlying arbitration between model-based and model-free learning.
        Neuron. 2014; 81: 687-699
        • Otto A.R.
        • Gershman S.J.
        • Markman A.B.
        • Daw N.D.
        The curse of planning: Dissecting multiple reinforcement-learning systems by taxing the central executive.
        Psychol Sci. 2013; 24: 751-761
        • Otto A.R.
        • Raio C.M.
        • Chiang A.
        • Phelps E.A.
        • Daw N.D.
        Working-memory capacity protects model-based learning from stress.
        Proc Natl Acad Sci U S A. 2013; 110: 20941-20946
        • Deserno L.
        • Huys Q.J.M.
        • Boehme R.
        • Buchert R.
        • Heinze H.J.
        • Grace A.A.
        • et al.
        Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making.
        Proc Natl Acad Sci U S A. 2015; 112: 1595-1600
        • Wunderlich K.
        • Smittenaar P.
        • Dolan R.J.
        Dopamine enhances model-based over model-free choice behavior.
        Neuron. 2012; 75: 418-424
        • Huth A.G.
        • de Heer W.A.
        • Griffiths T.L.
        • Theunissen F.E.
        • Gallant J.L.
        Natural speech reveals the semantic maps that tile human cerebral cortex.
        Nature. 2016; 532: 453-458
        • Huth A.G.
        • Lee T.
        • Nishimoto S.
        • Bilenko N.Y.
        • Vu A.T.
        • Gallant J.L.
        Decoding the semantic content of natural movies from human brain activity.
        Front Syst Neurosci. 2016; 10: 81
        • Torous J.
        • Kiang M.V.
        • Lorme J.
        • Onnela J.P.
        New tools for new research in psychiatry: A scalable and customizable platform to empower data driven smartphone research.
        JMIR Ment Health. 2016; 3: e16
        • Baker J.
        • Mueller N.
        • Onnela J.P.
        • Ongur D.
        • Buckner R.
        368-deep dynamic phenotyping: Neural changes underlying fluctuations in bipolar disorder over one year.
        Biol Psychiatry. 2017; 81: S150-S151
        • Huys Q.J.M.
        • Dayan P.
        • Daw
        Depression: A decision-theoretic account.
        Annu Rev Neurosci. 2015; 38: 1-23
        • Rutledge R.B.
        • Moutoussis M.
        • Smittenaar P.
        • Zeidman P.
        • Taylor T.
        • Hrynkiewicz L.
        • et al.
        Association of neural and emotional impacts of reward prediction errors with major depression.
        JAMA Psychiatry. 2017; 74: 790-797
      2. Weissman A, Beck A (1978): Development and validation of the Dysfunctional Attitude Scale. Presented at the Annual Meeting of the Association for the Advancement of Behavior Therapy, Chicago.

        • Peterson C.
        • Semmel A.
        • von Baeyer C.
        • Abramson L.
        • Metalsky G.
        • Seligman M.
        The attributional Style Questionnaire.
        Cogn Therapy Res. 1982; 6: 287-299
        • Abramson L.Y.
        • Alloy L.B.
        • Hogan M.E.
        • Whitehouse W.G.
        • Cornette M.
        • Akhavan S.
        • et al.
        Suicidality and cognitive vulnerability to depression among college students: A prospective study.
        J Adolesc. 1998; 21: 473-487