If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Translational Neuromodeling Unit, Institute for Biomedical Engineering, University of Zurich and Swiss Federal Institute of Technology (ETH Zurich), Zürich, SwitzerlandCentre for Addictive Disorders, Department of Psychiatry, Psychotherapy and Psychosomatics, Hospital of Psychiatry, University of Zurich, Zürich, Switzerland
Translational Neuromodeling Unit, Institute for Biomedical Engineering, University of Zurich and Swiss Federal Institute of Technology (ETH Zurich), Zürich, Switzerland
Computational psychiatry aims to apply mathematical and computational techniques to help improve psychiatric care. To achieve this, the phenomena under scrutiny should be within the scope of formal methods. As emotions play an important role across many psychiatric disorders, such computational methods must encompass emotions. Here, we consider formal valuation accounts of emotions. We focus on the fact that the flexibility of emotional responses and the nature of appraisals suggest the need for a model-based valuation framework for emotions. However, resource limitations make plain model-based valuation impossible and require metareasoning strategies to apportion cognitive resources adaptively. We argue that emotions may implement such metareasoning approximations by restricting the range of behaviors and states considered. We consider the processes that guide the deployment of the approximations, discerning between innate, model-free, heuristic, and model-based controllers. A formal valuation and metareasoning framework may thus provide a principled approach to examining emotions.
). It is motivated on the one hand by the necessity to bring novel statistical and machine-learning techniques to bear on the rapidly expanding complexity of novel datasets relevant to mental health, and on the other hand by the complexity of the problem itself as mental health relates to the most difficult tasks performed by the most complex of organs.
Emotions are central to mental health, and emotional disorders contribute substantially to the burden of mental illnesses (
). The traditional dichotomization of emotion and reason might question the feasibility of applying computational techniques to the core issues of emotion. It is therefore imperative for computational psychiatry that we consider the ability of a computational and mathematical framework to address core emotional phenomena. Here, we argue that approaching emotion computationally requires the introduction of model-based valuation and metareasoning. Metareasoning considers optimal valuation in the face of resource constraints (
). The proposal is that human emotions involve strategies to deal with the complexity of model-based or goal-directed decision making by focusing on particular aspects of the problem at hand.
Research on human emotions is complicated as questions about their nature continue to divide the scientific community (
). Nevertheless, there is consensus on a number of key components that characterize emotions, and this review attempts to view them in a computational light. We first provide a description of important features of emotions, then introduce valuation and the metareasoning problem, then relate approximate metareasoning strategies to features of emotions, and finally describe the control of approximate metareasoning strategies.
Ingredients of a Computational Approach to Emotions
Key features of human emotions that require accounting for and that are emphasized to various degrees in different conceptualizations are 1) correlated physiological, psychological and behavioral processes shaped by evolutionarily predefined neural circuitry; 2) interpretations or appraisals; and 3) conscious verbal self-report about emotions. Key problems in contemporary research on human emotions include to what extent the three feature domains are related (e.g., how conscious emotions in humans relate to evolutionarily predefined circuitry) and to what extent emotions are discrete entities.
Basic emotion theories suggest that there are a limited, relatively fixed, number of universal, evolutionarily shaped, culture-independent, and neurobiologically hard-coded emotional categories including happiness, surprise, sadness, disgust, anger, and fear (
). For the present purpose, what is important is that these represent a set of innately interlinked physiological, behavioral, and psychological processes that are triggered in an inflexible manner by species-specific salient stimuli, akin to unconditioned responses. Animal research, in which specific responses to species-relevant stimuli are observable and readily quantifiable, has contributed to this view. However, behavioral responses in animals cannot be directly translated to emotional experiences in humans. Amygdalar and hippocampal damage, for instance, abolish physiological and autobiographical signatures of aversive conditioning, respectively, while leaving the other intact (
). Furthermore, aversive conditioning can be performed subliminally and can evoke amygdala activity and physiological response, but can fail to result in any emotion of fear (
Human emotional responses to stimuli are characterized by substantial within- and between-subject variability. Appraisal theory locates one explanation for this variability in the interpretation (be it conscious or unconscious) of a particular situation or stimulus as being relevant to the individual’s goals (
). This interpretation depends on the goal and the individual’s beliefs in addition to the stimulus. A stimulus or situation being interpreted as increasing the changes of reaching one’s goals would, for instance, result in the emotion of joy or happiness (
). However, just like basic emotion theories, appraisal theories often view the expressed emotion itself as a “definable pattern of outputs that preexist within the individual” (
) defined them as “episode[s] of interrelated, synchronized changes in the states of all or most […] organismic subsystems in response to the evaluation of an external or internal stimulus event as relevant to major concerns of the organism.”
The evidence for discrete emotions is controversial. Autonomic responses, electroencephalographic features, and facial expressions do not permit simple categorization and show little evidence of the predicted correlations (
) data. The latter analyses have, however, clarified that there is no single underlying substrate for particular emotions. Rather, each emotional category depends on a distributed network of limbic but also cortical components that reflect the particular neurocognitive processes involved (
An alternative view is that the discreteness of emotions arises from the categorical labeling of internal events for the purpose of intra- or intersubject communication. Neuroimaging has provided some support for such a model, arguing that the ventrolateral prefrontal cortex is involved in categorical labeling of emotional states (
) evolving along the two major axes of valence (from good to bad) and arousal (from high to low). Indeed, factor analyses of a variety of measures of emotions including similarity ratings among words, facial expression, and autonomic measures reliably identify these two separate dimensions (
Basic and animal emotion research, with its grounding in evolutionarily shaped responses, emphasizes the importance of emotions in guiding behavior adaptively. A focus on adaptive responding is also present in appraisal theories, which suggest that emotions arise when events are judged to be relevant to the individual’s “needs, attachments, values, current goals and beliefs” (
). Computationally, inferring adaptive choices involves integrating not only immediate rewards, but also longer-term rewards, and for that reason requires consideration of the future course of events. This evaluation of the future is where the problem lies, as the further into the future one looks, the broader the range of potential events. Specifically, valuation involves summing over an exponentially expanding decision tree of future possibilities. Optimal valuation would search the entire tree, which is rarely feasible. Reinforcement learning is a thriving subfield of machine learning concerned with algorithmic solutions to this problem.
Model-Free Accounts of Emotional Expression
A substantial body of work has related one such algorithmic solution to how emotional expressions change over time (
). In so-called model-free reinforcement learning, the stability of the world is exploited to replace integration over the future with actual past experiences. Clever bookkeeping allows the use of prediction errors to update values that, in the limit of extensive experience, are guaranteed to yield the true long-term values of states and behaviors (
). Here, emotional responses are viewed as a type of high-level action, involving multiple biological and neural subsystems. One example of such an “action” is a freezing response, which has behavioral, attentional, and physiological components. These high-level actions are thought to be emitted either in an innate fashion (
), or after learning in response to a conditioned stimulus. In the latter case, the expression of the action is proportional to the value attached to the conditioned stimulus, which in turn is a scalar measure of the average expected unconditional stimulus strength (
Modelling individual differences in the form of pavlovian conditioned approach responses: A dual learning systems approach with factored representations.
), and others. Perhaps the most striking success of these models is their ability to capture how pavlovian affective responses can lead to maladaptive choices (
Model-free approaches are very valuable to understand how the expression of affect transfers between situations with experience. Although mostly restricted to individual laboratory sessions, the underlying model likely plays an important role in explaining how individual differences in the expression of (affective) behaviors emerge over (life)time, and potentially in response to behavioral psychotherapeutic interventions. Furthermore, a hierarchical version of model-free reinforcement learning has the capacity to explain how complex high-level actions consisting of multiple correlated processes might emerge (
), though this awaits application to the correlations among physiological, psychological, and behavioral aspects of emotions.
Appraisals Require Model-Based Inference
Pure model-free accounts, however, fail to explain context effects on conditioning. For instance, the physiological response to a threat differs depending on whether the animal is restrained or freely moving (
): the same emotion of anger may motivate harm not only through physical means in a boxing ring, but also through financial transactions in a boardroom setting.
Similarly, scalar summaries of past experience cannot account for the impact of appraisals on human emotions. The appraisal of an event involves an assessment of what the event “means” (
). This interpretative process determines whether and which emotion results by inference of latent causes: a smile is pleasant if interpreted as emanating from kindness, but aversive if viewed as an expression of condescension. The fact that an event is meaningful can be inferred from changes in model-free values because these capture the expectation of future well-being in a relatively stable environment, and sudden changes in model-free expected values therefore indicate a meaningful event. This is reminiscent of the argument that changes in core affect invoke appraisals (
), but what an event means for well-being cannot be derived from model-free values. This assessment involves a series of variables such as goal congruence, controllability, and agency (
). Goal congruence, for instance, measures how events influence the ability and cost of achieving current goals and as such involves replanning a new path toward the goal and comparing the cost of this path to that of the previous plan.
The computation of the meaning part of the appraisal requires the integration of a model capturing an individual’s beliefs about the consequences of choices, what reinforcements will be obtained in which states, how observations relate to hidden states, and how different states relate to each other (
). Inferring such values requires a model to be inverted or simulated. Mirroring the notion that some appraisals rely on rule-based processes, it suggests a role for nonautomatic components; it captures that appraisals generalize and change over time as new information is progressively integrated; it suggests how maladaptive beliefs influence emotions; and it suggests how a new understanding can alter emotions in explicit reappraisal (
One factor that may be useful to consider is that model-based inference is mostly impossible due to the sheer size of most relevant model-based valuation problems. Figure 1A shows a simple planning task in a maze the solution of which corresponds to a binary decision tree (Figure 1B). The best action at the root of the tree is the one that leads to the path with the best total outcome, and this may not be the action with the best immediate reward. As in reality, there is usually more than one way to the goal, but the different paths have different intermediate outcomes rendering some better than others. Despite its simplicity, humans have difficulty solving the task fully, and employ strategies to avoid evaluating the entire tree even when there are only three or four choices to go (Figure 1C). Unless they are highly constrained, such as in feedforward motor control (
), optimal decisions in realistic situations are computationally extremely demanding.
Figure 1Model-based inference problems rapidly become too hard to fully solve. Consider a task (A) where subjects see six rectangles arranged hexagonally and are taught to navigate according to the underlying transition matrix indicated by the arrows, so that they can transition from each state to two successor states depending on which of two buttons they press. Every transition yields a reward or loss. Finding a path of a given length that maximizes total earnings corresponds to a tree-shaped decision problem, for instance the one in panel (B) for three transitions starting from state 3. Participants typically choose not to expend cognitive effort evaluating subtrees below a salient loss (here below transitions with –70 points), resulting in a cutting of or “pruning” of decision trees (B). (C) This in turn results in worse performance when the optimal path requires transitioning through a salient loss (red) than when it does not (orange). Black dots and gray lines show the effect in individual participants. (A) and (B) are modified with permission from Huys et al.
). It is a decision problem about which of the various options to evaluate internally (Figure 2). Formally, the estimated value of performing a computation is the difference in expected utility between taking a choice without the additional computation, and taking a new alternative choice after having invested in the computation (
). Although this decision problem is mathematically similar to the original problem, it is different from the original problem because simulations do not actually incur the costs of the real problem, and while taking real poor actions should be avoided to avoid incurring their loss, internally simulating poor actions can be useful (
). Thus, the states in this metareasoning problem are all possible partial trees of the original tree, which is a far larger state space than in the original problem.
Figure 2Metareasoning. Given a state and a sequence of possible decisions, optimal action choice involves evaluating a decision tree (top left). An agent with constrained resources faces the challenge of deciding what to simulate, i.e., what to “think about.” For instance, it could choose to first simulate the action going left, and then continue down this branch (leftmost set of trees). Alternatively, it could superficially consider the right action, and then start examining the left action. The metareasoning problem hence is a decision problem where the states are knowledge states about the decision tree, and the task is to choose to think about the components of the problem at hand in the way that is most likely to yield a good final choice.
Model-based reasoning is hence faced with two profound challenges: the size of the problem and the even harder task of apportioning limited resources in an adaptive manner. These are fundamental problems and a strategy to deal with them is mandatory. The proposal here is that emotions can implement approximate solutions to these challenges. In particular, emotional states 1) come with a strong focus on particular behaviors and 2) induce a strong perceptual and processing focus such that evaluation is concentrated on a narrow set of states. Emotions thereby effectively function as approximate metareasoning strategies that prescribe how computational resources are allocated. To what extent these approximations are adaptive depends on how they are invoked. We first provide an outline for the implementation in terms of action tendencies and state observations, and then turn to the control issues.
Action Tendencies
One of the features of emotions about which there is more agreement is that they prioritize certain actions (
). Constraining the action space can substantially simplify the valuation problem because the computational cost is exponential in the size of the action space.
At an abstract level, emotional states are accompanied by distinct and richly experienced urges toward particular classes of actions. Frijda et al. (
) asked people to remember events of particular emotions and then to rate a list or 26 items about the kinds of behaviors they wanted to engage in, such as “I gave up,” “I wanted to protect myself from someone or something,” or “I wanted to help someone, to take care of someone.” From the ratings of these statements, the emotion characterizing the episode could be reliably recovered. Though very abstract, such rich descriptions are also important in psychotherapeutic settings. In dialectic behavior therapy, individuals are initially taught to recognize emotions by the action tendencies they feel (
Emotions also induce physiological and vegetative changes. However, physiological signatures of emotions do not appear to readily differentiate between categorically defined emotions, but rather provide a few classes of general action preparations [(
)]. A preparatory increase in heart rate to compensate for the anticipated drop in peripheral resistance upon supplying blood to large muscle groups is required when running, be it for escape or fun. As such, these can be seen as a preparation toward a class of behaviors that share physiological requirements.
State Observation
The complexity of model-based evaluation is also exponential in the range of states considered. There is ample evidence for emotion- or mood-congruent processing biases (
) showed that exposure to sad music and recollection of sad memories produces an attentional bias toward sad words, and such biases arise from the emotional state rather than purely from the exposure to the emotional word (
) searches along branches of the decision tree that result in states outside the attentional focus.
A further aspect is that there is usually uncertainty about the state. This profoundly complicates the computational task of valuation because policies for the various possible states have to be computed (
). By ascertaining the state, this complexity can be reduced. Introspection about the state of the body likely plays a particularly important role: the impact of a muscle’s activation depends on the joint position and the chances of success in a fight are reduced when already wounded.
Controlling Metareasoning Strategies
If there are multiple approximate metareasoning strategies, then there must be some control over which is deployed when. The first source of control is likely evolutionary, where species-specific responses provide a (potentially very strong) bias toward evaluating particular actions, rather than toward emitting the action entirely inflexibly. This allows for the kinds of context effects on even innate responses mentioned above.
The second source of control could be, confusingly, model-free. Performers may learn from experience that a certain amount of catastrophizing improves their performance (
) have recently argued for a model-free component in serial adaptations in the emotional conflict task, where individuals have to indicate the facial expression (fear or happy) of a face with either the matching or conflicting word superimposed over it (
). We have recently shown how the results of costly model-based evaluations are memorized and simply replayed upon repeated encounter of the same problem in a process called memoization (
) that gives rise to decision-making biases that are characteristic of the individual but highly variable across the population.
The third evaluative process for emotions allows for knowledge to be incorporated in the form of heuristics. Research on decisions about options with many attributes [for instance cars, with price, speed, size, brand, etc. (
)] have identified a host of different strategies. “Take the best” is appropriate in noncompensatory environments where one feature is most informative and can be used alone to rank options. In compensatory environments, humans spend more time and cognitive effort on examining multiple features and integrating the information, but only if they are not under time pressure (
). This suggests that individuals can access approximate measures of how adaptive a particular cognitive strategy is, and use this to guide their choice (
Algorithm selection by rational metareasoning as a model of human strategy selection.
in: Ghahramani Z. Welling M. Cortes C. Lawrence N.D. Weinberger K.Q. Advances in Neural Information Processing Systems 27: 28th Annual Conference on Neural Information Processing Systems. Curran Associates,
Red Hook, NY2014: 2870-2878
) about the adaptiveness of strategies relate to a number of pathological emotion regulation phenomena. Pathological worry in generalized anxiety disorders (
) are maintained by explicit beliefs about the usefulness of worry and rumination, respectively. People who dislike emotion regulation are more likely to respond with anger to provocation (
). Depressed persons are not impaired at emotion regulation strategies such as positive imagery to improve their mood, but they have a reduced tendency to employ them (
The fourth evaluative process, again confusingly, could be model-based, where the precise consequences of particular emotions are examined and evaluated. This is rarely feasible and probably only commonly done in situational analyses in psychotherapy, where emotions, thoughts, behavior, and consequences are explicitly discussed (
). This allows patients to learn to consciously and explicitly assess whether a particular emotion is appropriate and helpful in a given situation, and to adapt it by using reappraisal and other emotion regulation strategies if necessary.
Discussion
We have attempted to sketch out a valuation framework for emotions. We have argued why appraisals point toward a model-based framework; how emotions may have a potentially important role in facilitating model-based decisions by functioning as internal strategies to allocate computational resources; how emotions’ adaptive nature depends on their deployment; and how a variety of different processes can lead to adaptive or maladaptive deployment of emotion strategies.
Three desiderata for a computational framework of emotions were put forth. The first was the at best partially correlated nature of physiological, psychological, and behavioral features. The flexibility the proposed framework allows for contrasts with the view of basic emotions as relatively fixed behavioral and physiological action packages. As such, it reflects the lack of identifiably discrete physiological or behavioral patterns or single neurobiological cause (
). Space constraints have prevented us from exploring the distinction between valence and arousal important to circumplex and core affect models, but this might naturally emerge from valuation in continuous-time settings, where the rate at which actions are emitted depends on the average reward rate in the environment, albeit in sometimes complex ways (
). Notably, the current proposal allows for mixed emotions through a combination of metareasoning policies.
The second desideratum was the ability to account for appraisal and contextual effects. The complexity of the model-based valuation required for this led to the notion of approximate metareasoning strategies. These approximate strategies are necessarily often suboptimal and may capture the prototypical adverse influences of emotion on cognition (
), but distinct in that it suggests that the relevant predictions must be about long-term utility, and that emotions play a key role in facilitating such predictions, albeit approximately.
The third desideratum concerned the nature of conscious qualia of emotions. Doing so fully awaits a theory of consciousness. However, two aspects are interesting to consider. First, Dehaene and Naccache’s (
) notion of a global workspace sits naturally with the notion of metareasoning. Situations with a high estimated value of computation should recruit neural resources more extensively, and hence be more likely to involve the brain-wide states postulated as representing the global workspace. Second, it has been suggested that the component processes in verbal self-report involve an interoceptive component followed by a classification process (
). In our proposal, the metareasoning strategies would profoundly influence what information was processed, and as such may strongly determine future classification. Interestingly, the classification process has been suggested to involve the ventrolateral prefrontal cortex (
The framework laid out here makes a number of testable predictions. First, the argument that appraisal involves model-based reasoning means that it should be influenced by cognitive, endocrine, and neuromodulatory variables known to influence model-based reasoning (
). Second, we emphasized the importance of the control of metareasoning strategies and suggested that it may be subject to substantial malleability. This in turn predicts that by training particular metareasoning strategies it should be possible to selectively facilitate certain emotions over others. Third, it suggests that emotions do not represent valuations themselves, but rather that they determine the process by which valuation occurs. This predicts an influence of emotion on search in a relevant model-based task rather than on values directly. The main challenge for the framework is that it points to the critical importance of understanding both the internal models of individuals and their strategies in searching them. Measuring these is a difficult scientific problem. Though this is as yet not feasible, recent advances including whole-brain mapping of semantic representations (
The emphasis on model-based processes was partially motivated by the finding that model-free measures of reward processing and learning are unimpaired in depression (
Weissman A, Beck A (1978): Development and validation of the Dysfunctional Attitude Scale. Presented at the Annual Meeting of the Association for the Advancement of Behavior Therapy, Chicago.
Modelling individual differences in the form of pavlovian conditioned approach responses: A dual learning systems approach with factored representations.
Algorithm selection by rational metareasoning as a model of human strategy selection.
in: Ghahramani Z. Welling M. Cortes C. Lawrence N.D. Weinberger K.Q. Advances in Neural Information Processing Systems 27: 28th Annual Conference on Neural Information Processing Systems. Curran Associates,
Red Hook, NY2014: 2870-2878
Weissman A, Beck A (1978): Development and validation of the Dysfunctional Attitude Scale. Presented at the Annual Meeting of the Association for the Advancement of Behavior Therapy, Chicago.
31773-0&id=gr1.jpg){kind=link}
31773-0&id=gr2.jpg){kind=link}