[1] 12w What Are Memories For? The Hippocampus Bridges Past Experience with Future Decisions

[1] 555w Many decisions require flexible reasoning that depends on inference, generalization, and deliberation. Here, we review emerging findings indicating that the hippocampus, known for its role in long-term memory, contributes to these flexible aspects of value-based decision-making. This work offers new insights into the role of memory in decision-making and suggests that memory may shape decisions even in situations that do not appear, at first glance, to depend on memory at all. Uncovering the pervasive role of memory in decision-making challenges the way we define what memory is and what it does, suggesting that memory's primary purpose may be to guide future behavior and that storing a record of the past is just one way to do so. What Are Memories For?Learning is essential for adaptive behavior, allowing past experiences to improve the decisions we make in the future. Much research has focused on how repeated experience with outcomes enables the dynamic updating of predictions that guide later decisions [1]. However, we often have to make decisions without having had any prior experience with that exact same decision before. Such decisions must involve reasoning about new options, a process that involves generalization (see Glossary) [2], inference [3], and deliberation [4]. We discuss how these sorts of decisions depend on the flexible use of memory, supported by the hippocampus.The importance of memory for decision-making is beautifully exemplified in The Giver, by Lois Lowry. The novel describes a dystopian world without decisions and without agency, a world completely governed by habits. The Giver focuses on one member of society who must keep all memories of the old society in mind and it describes the process by which these memories are passed on from the current memory-keeper (The Giver) to a young boy who comes to replace him (The Receiver). Through this process, the dystopian story reflects on the content and purpose of memories. At a critical point in the story, the young boy asks The Giver what exactly memories are for. The Giver replies that memories are kept so that they can be consulted by the elders in case new situations are confronted. With this response, the novel makes the point that memory is necessary when things are uncertain, changing, and dynamic. In a world full of uncertainty and change, how does memory help us make decisions?Here, we present emerging findings showing that the hippocampus, known for its role in longterm episodic memory (for review, see [5,6]), also contributes to multiple aspects of flexible decision-making. This work offers new insights into the role of hippocampal memory processes in guiding inference and deliberation in service of value-based decisions. Together, these new findings raise the possibility that the fundamental role of the hippocampus may be to guide future behavior and that remembering the past is just one way to do so. HighlightsMemory plays a pervasive role in flexible decision-making that depends on inference, generalization, and deliberation. This function of memory in decisionmaking is supported by the hippocampus, suggesting that the role of the hippocampus may be to create a record of the past in the service of future behavior. This view reconciles findings from the fields of memory and decision-making. It offers new insight into why some memories are prioritized over others, why memory loss sometimes leads to impaired decision-making, and why decisions are shaped by regret and counterfactual thinking.

[1] 225w In the past decade, substantial progress has been made in understanding the brain mechanisms by which we learn from experience to improve our decisions (for a review, see [7]). The brief summary of this work is that repeated experience with decisions and their outcomes allows learning from the past to guide predictions about the future. For example, when contemplating what to order from a menu, we may choose falafel because when we have done so in the past, we were usually rewarded (Figure 1A, Key Figure). In other words, past experience helps us develop a tendency (a habit) of repeating choices that were rewarded and avoiding choices that were not. This process of trial-and-error learning from rewards depends on the striatum and its dopaminergic inputs, as revealed by converging evidence from single cell recordings [8][9][10], optogenetics [11][12][13], computational models [14][15][16], studies of patients with disrupted striatal activity (e.g., due to Parkinson's disease [17,18]), and studies with fMRI in healthy humans [19,20]. The common computational framework to account for this process is model-free reinforcement learning. Model-free reinforcement learning provides a framework for learning-based decisions in which a prediction about value for a given option is generated based on past experience with choosing this option. This account focuses on the idea that what is learned from reinforcement is the average value of candidate cues or actions [14].

[2] 96w But many decisions cannot depend on the learned average value from experiencing the same decision in the past (Figure 1). For example, we often face decisions for which we have very little direct past experience (maybe we have only tried this dish once before), while other decisions involve a choice between completely new choice options (every time we try a new cuisine). Moreover, even when we have extensive experience, surely we also have specific memories of particular moments (that time we had falafel in that small market) that play prominently in shaping our decisions every time.

[3] 189w All these cases depend on memory, but not the kind of overlearned memory that underlies habits. Computationally, it has been suggested that another class of algorithms, model-based reinforcement learning, may help support decisions that involve richer cognitive representations than model-free learning [21]. Model-based reinforcement learning contrasts with modelfree reinforcement in the assumptions it makes about what is being learned. In particular, the model-based algorithm assumes that a learner forms associative knowledge of the structure of the experience (such as the prediction of intermediary events) and not just the choices associated with valuable outcomes [14,22]. Model-based learning thus involves richer forms of memory, including relations between neutral events, that allow the sort of predictive inferences that are essential for making flexible decisions [23]. Indeed, cognitive theories of heuristics and biases in judgment and decision-making have long emphasized the central role of memory-dependent associations in shaping decisions [24][25][26][27]. These studies all point to potentially important links between memory and decision-making, raising questions about the mechanisms by which memory guides flexible decisions, how it contributes to the construction of a model of the world, and how this information enters the decision-making process.

[4] 67w The Hippocampus: Building Memories for Future Decisions Decades of research have advanced our understanding of the neural mechanisms that contribute to long-term memory for events or episodes, referred to as episodic memory. Episodic memories are formed rapidly (after even a single experience) and are rich in contextual details. Extensive converging evidence indicates that episodic memory depends on the hippocampus and surrounding medial temporal lobe (MTL) cortices [5,28].

[5] 64w feature of hippocampal memories that refers to their ability to be retrieved and used in new ways and new contexts. This form of flexibility is precisely what is missing from theories of model-free reinforcement learning and provides a mechanism for understanding how memories for past events can be integrated to form a flexible 'cognitive map' [46], or world model [47], to support new decisions.

[6] 134w The basic idea is that the hippocampus encodes the relations between different features of an experience and when elements of a current experience overlap with those of a previously encoded experience, the hippocampus reactivates the encoded memory and completes the pattern, integrating the new memory with an old one [48][49][50]. Once this relational map is formed, inference about elements within the map that were never directly experienced together can take place [51][52][53][54][55]. As described later, this basic mechanism could support flexibility and inferences in value-based decisions in a number of different ways: by allowing value learned in one context to become associated with otherwise neutral items (Figure 1B); by allowing novel recombinations of value to support decisions about new options altogether (Figure 1C); and by supporting comparison of options across different dimensions (Figure 1D).

[7] 208w The generalization mechanisms reviewed earlier can explain how we make novel decisions between choice options that were already somewhat familiar on their own. But what about decisions between completely novel options, which were never experienced in the past? Consider, for example, someone who encounters the decision of whether to eat falafel for the very first time (Figure 1C). Because they never experienced this specific dish before, they did not yet integrate this information with other relevant memories or stored associations of value. In this scenario, memory can serve as a good heuristic to solve the decision in a couple of different ways. One approach is to assess the overall similarity of the dish to other familiar foods and predict falafel's value based on the value of similar foods (e.g., it is a Middle Eastern dish and therefore in the same category as shawarma). Another approach is to mentally decompose the dish into its familiar components and integrate the value of all the separate components (e.g., it is made of chickpeas and served with hummus and pita bread, etc.) Both of these mechanisms rely on the ability to flexibly use and integrate previous experiences to compute a decision variable and there is some evidence that both involve the hippocampus.

[8] 129w Some generalizations are based on the perceptual similarity of a new option to others that have already been experienced. Early studies tested perceptual generalization in animals using conditioning to train the animals to associate an outcome with one exemplar (e.g., a specific tone) and then testing generalization of the conditioned response to a range of new tones that vary in similarity to the conditioned tone. Lesions to the hippocampus were found to change perceptual generalization [74][75][76]. More recently, fMRI studies have linked activity in human hippocampus to perceptual generalization in the context of value-based decisions [77]. For example, two stimuli that vary along a single perceptual dimension (e.g., two lines with different orientations) were either paired with reward, or not. At test, participants were presented with novel stimuli (that

[9] 42w varied in orientation) and had to decide which is more valuable. Participants chose stimuli that were perceptually similar to the rewarded stimulus (e.g., with similar orientation), a behavioral tendency that was related to functional connectivity between the hippocampus and the striatum [77,78].

[10] 115w Other studies have addressed the question of how we make decisions about the value of novel options by examining whether individuals decompose choice options into familiar components. When asked to evaluate novel food items (e.g., tea-jelly), participants in one study accessed memories for each of the separate components (e.g., tea and jelly, Figure 2, center [79]). The medial prefrontal cortex and hippocampus contributed to this process: both regions displayed reductions in activity (repetition suppression) when the novel item was preceded by one of its components (e.g., tea), suggesting that the old components were evoked to make the new decisions. This change in blood-oxygenation-level-dependent (BOLD) activity was correlated with the value assigned to the novel items.

[11] 131w Together, these studies highlight the constructive nature of value-based decisions. In particular, they suggest that when faced with a decision that we have not experienced before, one for which we do not have a precomputed predicted value, rather than just guessing, we construct a prediction based on the most relevant past experiences we have access to. Put simply, we rely on what we know to predict outcomes in new situations (as described in The Giver). We use memory as evidence in this constructive process and rely on relational mechanisms to engage in the necessary construction, comparison, and generalization of value to guide decisions [31]. In the next section, we discuss how this constructive memory-based process contributes broadly to deliberation, even when making decisions between items that are not new at all.

[1] 55w Far from being a separate cognitive process from value and decisions, memory plays a pervasive role in shaping value-based decisions. Interactions between memory and decision-making play out in different ways, at different time points (Figure 1). The initial encoding of individual episodes allows reward-relevant information to affect the prioritization of some memories over others [88][89][90][91][92].

[2] 73w Across episodes, interactions between encoding and retrieval allow the integration of separate events into an associative world model that is well-suited for guiding later generalization [2,52,53]. When faced with a decision, memory-dependent processes of inference and prospection can enter the deliberation process by allowing memory to provide new evidence bearing on the decision at hand [79,111], thereby resolving difficult decisions between samevalue options and providing opportunities for value updating and changes of mind.

[3] 152w These different kinds of decisions all depend on the hippocampus (see Box 1). Traditionally, the hippocampus was viewed as a 'cold' memory system, specialized for building long-term explicit memories of neutral events, as distinct from a 'hot' reward-guided system for decisions and actions, supported by the striatum. The work reviewed here suggests a different view: that the hippocampus supports an adaptive function in the service of value-based decisions, which is reflected both in how memories are encoded and in how they are used. This view offers a new perspective on the role of memory in a variety of decision-making phenomena, including generalization and prospection (for review, see [31]), as well as deliberation, and the role of regret and anticipated regret in value-based decisions (Box 2). It also suggests that a consideration of the principles of episodic memory offers a framework for both understanding and generating predictions about the factors that influence decision-making.

[4] 22w Importantly, hippocampal contributions to decision-making can happen outside of conscious awareness, allowing for automatic and implicit effects on behavior [112,113]. Moreover, although

[5] 20w What are the unifying computational principles that link hippocampal function at the physiological level to decision-making at the behavioral level?

[6] 17w How is value updated by the reshaping of mnemonic networks, for example, following consolidation or memory loss?

[7] 39w What role does the hippocampus play in the experience and anticipation of regret? Does it support the construction of an alternative reality? Does it facilitate the reactivation of a prior decision? How do amnesic patients experience or anticipate regret?

[8] 41w What are the consequences of deliberation? When deliberating about options, we often engage in mental time-travel involving the different alternatives. Do we encode this mental time-travel? And if so, do these new memories affect the way we perceive our own choices?

[9] 91w Trends in Cognitive Sciences not the focus of this paper, there is extensive evidence for important interactions between the hippocampus and the striatum in value-based decisions, as well as between the hippocampus and other decision-related regions in the brain such as the OFC and parietal cortex [53,60,61,87,111,114]. Recent studies are beginning to uncover the nature of these interactions, although many open questions remain about the circuit-level mechanisms by which information from memory can come to bear on a decision. This will be an important topic for future research (see Outstanding Questions).

[10] 182w Recognizing that memory plays an essential role in a wide range of value-based decisions offers important insight into how decisions are made. But it also suggests a shift in how we think about memory itself. This work suggests not only that memory can be used for guiding decisions, but that this may be, fundamentally, what memory is for. This idea is supported by a consideration of common errors in memory. Indeed, memories are notoriously vulnerable, prone not only to forgetting but also to suggestibility and to false insertions [115]. To some extent, these errors are only errors when one considers the job of memory to be the accurate recording of past experiences. However, through the lens of decision-making, each of these errors can be viewed as markers of a flexible system that serves up relevant past experience to guide behavior in a changing and uncertain world. Indeed, memory errors can be attributed to a fundamental adaptive function: the integration of experiences, separated in time, into a complex and interconnected model of our experiences, tying together the past, the present, and the future.

[1] 17w Anterograde amnesia: loss of the ability to create new memories after the event that caused the amnesia.

[2] 127w Cognitive map: a mental representation that organizes knowledge systematically across multiple domains. Deliberation: consideration of evidence in order to make a decision. Episodic memory: memory for a specific event that is associated with a particular time and space. Generalization: using what is learned in one context to guide behavior in a different context. Habit: a tendency to take the same action or make the same decision in a particular situation that is encountered repeatedly. Inference: a conclusion or a decision that is based on evidence and reasoning. Model-based reinforcement learning: a class of algorithms that learn an internal model of the environment that includes actionstate associations and their expected value. Model-free reinforcement learning: a class of algorithms that learn the expected value of actions through direct experience.

[3] 78w Place cell: a neuron in the hippocampus that fires when an animal is in a particular spatial location. Reinforcement learning: a class of algorithms that learn the values of different actions to guide rewardmaximizing behavior. Relational memory: the ability to remember arbitrary associations between elements of an event or several events, even if those events do not happen concurrently. Representational flexibility: the ability to use memory flexibly to guide performance in new contexts other than the learning event.

[4] 49w Sensory preconditioning: a form of classical conditioning in which two neutral stimuli A and B are paired. Subsequently, stimulus A is paired with an unconditioned stimulus (e.g., reward). If stimulus B (which is never paired with the unconditioned stimulus) elicits a response (e.g., approach behavior), then sensory preconditioning has

[5] 309w In addition to its role in building long-term episodic memories, the hippocampus is also critical for planning for the future [29][30][31]. Patients with bilateral damage to the hippocampus are famously impaired at forming new memories for events and episodes they experience, a condition known ). In a model-based approach, the decider has access to an associative memory network that represents related experiences and the relations between them, allowing for the generalization of value. Here, the decider first observed a falafel in a specific restaurant without tasting it (no value information), associating the falafel with the restaurant. Later the decider had an amazing salad in the same restaurant, creating an associative link between the high value of the salad and the (never tasted) falafel through the common associations with the restaurant. Finally, when encountering the option of falafel, the falafel was already associated with high value, leading the decider to choose it. (C) Recombination of experienced components (when faced with a decision). A model-based approach can also apply to novel situations, in which the integration of value happens at the time of the decision. In the case of a decider who has never before had falafel, the decision can be guided by separating the choice option into its different components and using a model-based calculation to recombine them and infer the integrated value (e.g., the decider might consider how much she likes chickpeas, pita, and hummus to infer how much she would like falafel). (D) Memory-guided deliberation (when faced with a decision). In this scenario, the decider has already experienced both falafel and shawarma multiple times in the past and likes them both equally. Memory mechanisms can support deliberation by allowing a comparison of the options, the retrieval of relevant memories, and the extraction of specific features of each option that can break the tie by constructing value de novo.

[6] 130w as anterograde amnesia [32]. These same patients, however, are also impaired at imagining future hypothetical events [33]. The first report of a possible connection between memory and imagination came from the case study of patient K.C. [34]. Patient K.C. displayed typical and devastating anterograde amnesia as a result of bilateral damage to the hippocampus and MTL. When K.C. was asked what he would be doing the next day, he was unable to provide details. This observation was later systematically confirmed in studies with other patients with bilateral hippocampal damage [33,35,36] (but see [37]). When asked to imagine a birthday party in a year or a vacation next month, patients with hippocampal damage provide fewer details compared with healthy participants, suggesting that their ability to prospect about the future is impoverished.

[7] 177w Converging evidence for the role of the hippocampus in prospecting about future events was obtained from fMRI studies in healthy individuals. Comparing brain activity related to retrieving a memory of a past event (e.g., think of your birthday last year) with imagining a future event (e.g., imagine a beach vacation next summer) revealed overlap in the network of brain regions that were active during memory retrieval and during the imagination of future events, including the hippocampus [38]. Additional evidence supporting the view that the hippocampus contributes to planning future behavior comes from studies of navigation. In rodents, hippocampal activity is famously related to spatial location [39]. Hippocampal 'place cells' consistently fire when a rodent visits a specific spatial location [40,41], but also when the rodent is navigating a maze and planning where to go next [42,43]. Similar findings have been reported in humans engaged in a spatial navigation task using virtual reality simulations [44]. Thus, whether imagining a future event or planning where to go, hippocampal activity enables the use of past experience to guide future behavior.

[8] 89w Collectively, the work reviewed in this section points to a common role for the hippocampus across cognitive processes that seem, at first glance, to be quite different: encoding of memories for past events, simulating future events, and planning of spatial navigation. A compelling theory suggests that these functions may all be accounted for under a single mechanism by considering the role of the hippocampus in 'relational memory', a term coined by Howard Eichenbaum and Neal Cohen to capture the common role of the hippocampus across seemingly different behaviors [28].

[9] 27w According to this view, the hippocampus binds the separate elements of an experience in memory, encoding events as relational maps of items within spatial and temporal contexts.

[10] 137w Encoding relations between multiple elements of an event allows episodic memories to be flexibly deployed and generalized across changing circumstances. As detailed later, drawing on this perspective offers new predictions and ways of thinking about how, why, and when the hippocampus contributes to value-based decisions. For instance, it offers a useful framework for understanding why damage to the hippocampus leads to decision-making impairments even in tasks that do not appear to involve episodic memory (see Box 1; also [31]). It also helps explain why decision-making phenomena, such as the experience of regret and the representation of counterfactual information, are related to hippocampal activity (Box 2). Moreover, as we describe next, it suggests that the hippocampus is well-suited to contribute to the sort of cognitive processes that are necessary for flexible decisions that involve generalization, inference, and deliberation.

[11] 151w The idea that the hippocampus supports relational memory offers novel predictions about the role of the hippocampus in making decisions that are not strictly based on direct prior experience. In particular, it offers a framework for considering 'representational flexibility' [45], a hallmark The first demonstration of the causal role of the hippocampus in episodic memory came from the famous case of patient H.M., who had bilateral damage to the hippocampus and surrounding medial temporal lobe (MTL) and suffered severe anterograde amnesia as a result [116]. Extensive subsequent work showed that people with damage to the hippocampus exhibit an overwhelming and highly selective anterograde amnesia, characterized by an inability to create new episodic memories, while virtually all other cognitive functions remain intact. If memory is important for decision-making, however, amnesic patients should also exhibit changes in how they make decisions. Table I summarizes available studies that assessed value-based decision-making in amnesic patients.

[12] 97w The studies can be organized into two groups. The first group of tasks (the first three in Table I) involve incremental learning about value without requiring explicit recollection of a specific past event. Nonetheless, amnesics show impairments on these tasks. For example, in the Iowa gambling task [117,118], amnesics were unable to integrate information across trials to choose advantageous decks (despite showing sensitivity to value information). In the two-step task [119], patients were much less likely to engage in model-based decision-making that requires learning the structure of events across trials (despite gaining overall similar levels of reward).

[13] 180w The tasks in the second group (the last three in Table I) are simple decision tasks that do not involve learning and merely require a choice between two options, for example, between delayed and immediate rewards, or between two food items. On the surface, patients' behavior appears intact: they exhibit delay discounting, the tendency to devalue larger and delayed rewards over smaller and immediate ones [120,121], and their food choices are consistent with their initial preferences [87]. And yet, a more subtle analysis of both behaviors reveals a different story: when asked to imagine oneself using the larger reward in the future, healthy controls attenuate the discounting of future rewards, but amnesics do not [35] (but see [122]). Also, when making simple value-based choices, amnesic patients take much longer [87]. Altogether, these findings suggest that, even when patients seem to reach the same decision as healthy controls, the mechanism by which they do so may be different. Future work would benefit from examining not just the outcomes of the decisions that patients make, but also how they reached that decision.

[14] 26w Table I. Studies with Amnesic Patients That Highlight the Role of the Hippocampus in Decision-Making a Decision-making task Spared versus impaired behavior Iowa gambling task [123].

[15] 40w Participants decide between four decks with varying outcomes: two advantageous decks with small rewards and smaller punishments, and two disadvantageous decks with large rewards and larger punishments. The task requires learning and updating the contingencies of multiple decks across time.

[16] 54w Spared: patients and healthy controls showed increased skin conductance responses following high punishments, suggesting spared sensitivity to value information [117]. Impaired: patients did not develop a preference for advantageous decks [117,118], whether the outcomes were delayed or immediate [118] (but see [124]). They also showed anticipatory skin conductance responses prior to disadvantageous choices [117].

[17] 4w Probabilistic learning task [125].

[18] 31w Participants are asked to predict the outcome of a cue or a configuration of cues and are given feedback. The relationship between cues and outcomes is probabilistic and changes across trials.

[19] 68w Spared: performance was intact when amnesic patients had to learn to associate a single cue with immediate probabilistic feedback [126] and in early trials of learning about configurations of cues [17,125] (but see [127] for impairment even in early training). Impaired: patients' learning was impaired with configural cue structure [17,125] and with single probabilistic cues with delayed feedback, suggesting impairment in relating cue-outcome events across temporal delays [126].

[20] 56w Participants first make one choice between two stimuli, A and B. Stimulus A commonly transitions to one pair of stimuli and rarely to a different pair of stimuli, whereas for stimulus B, the transition probabilities are reversed. Next, the participant makes another decision between two second-stage stimuli, each associated with a slowly changing probability of reward.

[21] 60w Spared: patients gained similar rewards as healthy controls [119]. Impaired: patients showed altered strategies: compared with controls, they relied more on the model-free learning (repeating choices that led to reward in the previous trial) and less on the model-based learning (considering the transition frequency, suggesting learning of task structure). This effect was driven mostly by patients with right-lateralized lobectomy [119].

[22] 3w Intertemporal choice [93,128,129].

[23] 34w Participants are asked to choose between an immediate reward and a larger reward later. In a subsequent version of the task, participants were cued to imagine a future event prior to making the decision.

[24] 54w Spared: patients showed delay discounting (the tendency to devalue the delayed reward), similar to healthy controls [120,121]. Impaired: when cued to imagine spending the delayed money in the future, healthy controls exhibited attenuation of delay discounting, but patients did not [35] (but see [122] for evidence of attenuation in several patients using personalized cues).

[25] 33w Participants are asked to rate items, decide between them, and rate again. The task measures choice-induced preference change: the tendency to rate chosen items higher and unchosen items lower, compared with initial ratings.

[26] 38w Spared: patients exhibited preference change, suggesting intact value updating [131], yet this could relate to a revealed preference artifact, rather than preference change [132][133][134]. Impaired: patients' memory of their choice was worse than that of healthy controls [131].

[27] 9w Value-based (food choice) and perceptual (color discrimination) decisions [87].

[28] 53w In the value-based task, participants are shown two familiar food items and are asked to choose which one they prefer. In the perceptual task, participants view a cloud of flickering dots that are either yellow or blue, with varying proportions. Participants are asked to indicate whether there are more blue or yellow dots.

[29] 47w Spared: patients showed comparable performance with controls in the perceptual task. For the value-based task, patients' choices were consistent with initial ratings of the food items [87]. Impaired: for value-based decisions, patients responded more slowly and more stochastically than healthy controls, suggesting differences in valuation mechanisms [87].

[30] 40w a Amnesic patients were tested on a representative set of decision-making tasks that do not overtly rely on declarative memory. Patients show a pattern of spared and impaired behavior that implicates the hippocampus in generalization, inference, and deliberation during decision-making.

[31] 86w To test whether the hippocampus supports generalization in decision-making, studies have used tasks that involve the encoding of specific experiences or events, followed by an opportunity to use what was learned to generalize and make flexible inferences about new combinations of familiar options. Consider the phenomenon of 'sensory preconditioning' (Figure 2, left). Participants first learn a series of associations through direct experience (e.g., that A is associated with B and that C is associated with D), then learn that a subset of these cues are associated

[32] 60w Regret is the emotion we experience when we realize or imagine that our current state could have been better had we made a different choice. While regret is rooted in a past decision, it can also guide and affect future decisions [135]. As such, regret offers an interesting lens through which to consider the influence of memory on decision-making [136].

[33] 212w Regret is typically triggered when we learn that an unchosen option turned out to be more valuable than a chosen one [137,138]. Consider a scenario in which you expected to receive a $10 reward but ended up receiving $50 instead. The surprising $40 gain is rewarding. But what would your experience be if it turned out that the alternative choice would have led to a $200 win? And what if it would have led to a $5 win? Clearly these two alternative outcomes lead to different emotional responses as well as to different subjective valuations of the outcome received [138][139][140][141][142]. Consequently, regret is not only an aversive emotion, but it can also serve as a learning marker that drives value updating and behavioral change [140,143,144]. Indeed, studies have found that both humans [142,145,146] and animals [147][148][149][150] represent the potential outcomes of counterfactual options and use this information in subsequent choices (for a functional account of counterfactual thinking, see [151]). Regret also extends beyond the moment of learning about potential outcomes to futureoriented deliberation that can often consider future anticipated regret [140,144,152]. When deliberating between options, people often try to predict not only how they will feel with the chosen option, but also how they will feel about forfeiting the unchosen option [153][154][155].

[34] 159w The retrospective and prospective nature of regret has interesting parallels to the function of memory in decision-making. The experience of regret involves mental travel to a past decision, imagining what would happen had the choice been different. The anticipation of regret also requires mental travel, into a possible future in which the previously unchosen option out-values the chosen one. In either case, the mental construction of an alternative reality involves flexible recombination of past experiences. This process is adaptive, in that it can inform future decisions, and it may involve relational memory mechanisms supported by the hippocampus. Indeed, in recent years there is mounting evidence for hippocampal involvement in counterfactual thinking and regret. For example, activity in the hippocampus has been shown to track the value of counterfactual options in a decision-making task [150]. Furthermore, episodic counterfactual thinking recruits the same core network of brain regions that are activated when remembering the past and thinking about the future [156][157][158].

[35] 46w Finally, the subjective experience of regret is associated with activity in the hippocampus (as well as in the OFC and anterior cingulate cortex) [140]. The links between regret, memory, and value-based decisions are just beginning to be explored and are an important area for future research.

[36] 213w with reward (e.g., that B is associated with reward). Participants are later probed to make a choice about a novel combination of cues (e.g., between stimulus A and C). Formally speaking, A and C are both neutral cues and neither has been directly paired with reward. Yet, empirical findings across species indicate that animals and humans tend to choose stimulus A. Both fMRI and magnetoencephalography studies have shown that this tendency, taken as the behavioral marker of value spreading across separate memories, is related to hippocampal activity [52][53][54]56]. Furthermore, lesions to the hippocampus impair this behavior [57] (as do lesions to the perirhinal cortex [58] and inactivation of the orbitofrontal cortex (OFC) [59]). They then learned that one stimulus (e.g., the blue fractal) led to reward and the other (e.g., the red fractal) did not. In a final choice phase, participants chose between two scenes, neither of which had been directly associated with reward. Participants often preferred the scene that was associated with the rewarded fractal, suggesting that they had integrated the reward value associated with the blue fractal to the scene through memory associations. This choice behavior is correlated with blood-oxygenation-level-dependent (BOLD) activity in the hippocampus [MNI coordinates of peak activation (26, -34, -12) in green]. In the center, Barron et al.

[37] 212w [79] asked participants to evaluate different foods (e.g., avocado, raspberries, tea, jelly, etc.) and then to choose between two novel food combinations (e.g., avocado and raspberry smoothie or tea-jelly). fMRI analysis revealed repetition suppression in the hippocampus [MNI coordinates of peak suppression (30, -10, -20) in teal] when a novel food was preceded by a component food. Furthermore, the level of suppression was correlated with the value of the novel good. These findings suggest that the hippocampus facilitates the construction of value for novel goods by flexibly combining previously experienced options. On the right, Bakkour et al. [87] used a standard food choice task to investigate the role of the hippocampus in familiar choices about familiar options. Participants first provided a measure of subjective value for familiar food items in a valuation phase. They then chose between foods in a separate choice phase. The time it took to make these food choices (i.e., reaction time) correlated with BOLD activity in the hippocampus [MNI coordinates of peak activation (28, -10, -28) in blue], and patients with amnesia took longer, suggesting that hippocampal memory mechanisms were recruited during deliberation about which food to choose. Together, these studies highlight that the role of the hippocampus in value-based decisions may be more ubiquitous than previously thought.

[38] 143w One interesting aspect of sensory preconditioning is that the integration across separate events can occur during learning, before a decision is ever faced (e.g., [52][53][54]). This has to do with the mechanisms by which the hippocampus encodes and retrieves memories: when encountering the B-reward association, presentation of the familiar B is thought to evoke retrieval of the previously encoded A-B association and this retrieval, in turn, leads to integrated encoding of A with reward in memory, despite the fact that A was never directly paired with reward. Later, when faced with a decision, stimulus A has already acquired reward value. Evidence for such a mechanism was adduced with fMRI studies that show that hippocampal activity during the reward phase predicts later generalization [52] and that during the reward phase there is evidence for reactivation of the A stimulus when B is presented [53].

[39] 79w This is not to say that such pre-encoding is the only way the hippocampus can support the integration of value across separate events (for review, see [31]). For example, sensory preconditioning and other forms of integration can also occur after learning, by retrieving associated items and integrating them at the time a decision is faced. This form of integration 'on the fly' has been shown to involve the OFC and interactions with the hippocampus [59,60] (for review, see [3,61]).

[40] 80w Indeed, the dynamic integration of events across time and space is evident across many different kinds of behaviors, from shortcut learning in spatial navigation [62], to acquired equivalence [52,63], transitive inference [64][65][66], learning of latent structure [57,67], odor sequence learning [68,69], reward revaluation [70,71], and rule abstraction [72,73]. Although the specific mechanism may vary across these wide-ranging behaviors, the hippocampus seems to play a common role: integrating related but separate experiences into a rich associative structure that supports flexible behavior.

[41] 76w Integration of memories and construction of value are necessary when we make decisions about new options. However, we may engage in construction of value even when we make choices between highly familiar items for which value is well-known [4]. In this section, we examine the potentially pervasive role of the hippocampus in decision-making by focusing on the interesting problem of resolving approach-approach decisions in which we decide between two choice options that are of similar value.

[42] 110w In cases where the cached value of choice options is the same, this cached value cannot resolve the choice, even if we had extensive prior experience with both options before. Indeed, such decisions are notoriously difficult and are known to take more time and effort. This observation has puzzled economists and philosophers for decades and is encapsulated in what is known as 'Buridan's ass' paradox [80]: a donkey that is equally thirsty and hungry and placed exactly at equal distance from water and food should not be able to decide whether it should go drink first, or eat first. Indeed, when both options are equally appealing, how do we choose?

[43] 90w One suggestion has been that perhaps such decisions are arbitrary: that they are made by sampling noisy value estimates. Dynamic sampling of noisy evidence, even when the mean values of the options are equal, will eventually lead to one choice over the other, avoiding deadlock (for review, see [81]). On this view, decisions between options of equal value take more time because of the dynamic sampling process itself, not because of any consideration of the content of the evidence or its relation to the specific decision that is being considered.

[44] 167w A recent alternative view suggests that the evidence may not emanate from stored noisy value representations, but instead comes from memories related to the options at stake. According to this view, value-based decisions may leverage hippocampal-dependent memory mechanisms to weigh the options, compare them, and decompose them [4]. Such processes would contribute to the accumulation of new evidence about the options to help guide a decision. The idea that decision-making involves a process of evidence accumulation has deep roots in models of perceptual decision-making (for review, see [82]). In tasks of perceptual decision-making (such as deciding whether dots on a screen are moving to the left or the right), choices and reaction times adhere to principles of evidence accumulation. These models assume that statistically independent samples in favor of a choice option are accumulated over time and when enough evidence is accumulated, a commitment to a choice is made [83,84]. This bounded accumulator model also explains simple value-based decisions, such as the choice between two foods [85,86].

[45] 61w Whereas the source of evidence in a display of dynamically moving dots is clear (it comes from the external stimulus) the source of internal evidence in value-based decisions is ambiguous. It has been proposed that relational processing in the hippocampus may supply samples of internal evidence in the service of value-based decisions, even when these decisions involve familiar choice options [4].

[46] 153w In support of this hypothesis, a recent fMRI study has shown that the amount of time it takes to choose between two snacks is correlated with BOLD activity in the hippocampus. This correlation was specific to value-based decisions and was found within hippocampal subregions that contributed to memory retrieval. Furthermore, amnesic patients with bilateral hippocampal damage made perceptual decisions that were intact, but their value-based decisions were impaired [87]. This pattern of findings suggests that individuals with amnesia were able to sample and use external information for perceptual decisions and recognize individual items, but they had difficulty comparing the items and took much longer to resolve pairwise value-based decisions than did healthy controls (see Box 1). Together, these findings demonstrate that the hippocampus supports the sampling of internal information during deliberation about value-based decisions, even between highly familiar items, decisions that do not appear, at face value, to depend on memory at all.

[47] 58w So far, we have discussed how memories are used to make decisions. But the interplay between memory and decision-making also has implications for understanding how memories are formed. Emerging work suggests that even the initial encoding of a memory can be prioritized based on its later potential value, allowing reward-relevant information to prioritize some memories over others [88][89][90][91][92].

[48] 102w Value-related prioritization of memory can be due to the anticipation of value, before an event occurs, or due to retroactive consolidation of memory for an event, based on its outcomes. A seminal study found that a priori information about the potential value of remembering an image was associated with enhanced hippocampal-midbrain coupling, and better memory, for high-value images [92]. In addition to such anticipatory effects, retroactive effects of reward have also been observed. For example, reward affects prioritization of memory for events that were neutral at the time they were experienced, but were later revealed to be predictive of a reward [90].

[49] 63w These findings suggest that value is embedded in hippocampal memory representations, whether this value information is provided prospectively [42,[93][94][95] or retroactively [96,97]. The encoding of value information appears to follow the same general principles of contextual encoding of other behaviorally relevant information, such as space or time [61]. Indeed, recording studies in animals have shown that reward information is encoded in hippocampal neurons

[50] 21w Trends in Cognitive Sciences [73,98] and is organized by reward context contingencies, much like the representation of space and actions [98].

[51] 132w Advances in understanding interactions between memory and decision-making have led to new computational efforts to bring memory mechanisms into decision models. One direction has been the development of models in which value-based decisions are based on retrieval of a singular event in the past, rather than the aggregate value of multiple past events [99][100][101][102][103][104]. These models are often referred to as 'episodic reinforcement learning' models. Another recent model has focused more specifically on the link between decisions for reward and hippocampal place cell activity [105]. This model aims to explain which spatial memories are prioritized and accessed when an animal makes decisions about navigation for reward. By considering how useful a memory is for predicting the location of reward, the model offers a framework that links together value with memory-guided spatial decisions.

[52] 98w A critical challenge moving forward is to develop unifying computational models that connect known computational principles of the hippocampus to the sort of value-based decisions reviewed here. The field of memory has benefited from modeling efforts that address hippocampal mechanisms at multiple levels of analysis, from neural plasticity and circuit mechanisms, to spatial navigation and memory at the behavioral level [2,[106][107][108][109][110]. However, so far there has been a lack of models linking hippocampal physiology to value-based decisions specifically. Such models will be particularly important for generating mathematically precise and testable predictions about how the hippocampus supports value-based decisions.