Intertemporal Choice - Toward an Integrative Framework

Economic, psychological and neuroscientific perspectives

Until recently, the main contribution of economics to the study of intertemporal decisions was modeling. For nearly 80 years, economists have analyzed intertemporal decisions using the discounted utility (DU) model, which assumes that people evaluate the pleasures and pains resulting from a decision in much the same way that financial markets evaluate losses and gains, exponentially 'discounting' the value of outcomes according to how delayed they are in time. DU has been used to describe how people actually make intertemporal choices and it has been used as a tool for public policy. Policy decisions about how much to spend on research and development, health and education all depend on the discount rate used to analyze the decision. Indeed, recently the discount rate has proven to be a key parameter in the policy debate about global warming [ The main contribution of psychology has been to identify, through empirical research, psychological mechanisms underlying intertemporal choice. For example, George Ainslie's research on the structure of time discounting posed the first serious challenge to the DU model -specifically to the assumption that people discount the future exponentially [2,3]. The concept of 'hyperbolic time discounting' (explained below) can be considered the first observed pattern of behavior that is inconsistent with DU -a DU 'anomaly'. Subsequent research by both psychologists and economists has identified a wide range of additional anomalies [4][5][6][7][8][9][10][11][12].
Economists have responded to these findings by constructing new models of intertemporal choice, which incorporate psychological insights, to explain otherwise anomalous patterns of economic behavior [13].

96 97
Neuroscience is the most recent entrant into what was already a rich interdisciplinary mix of research. Although still in its infancy, neuroscience research on intertemporal choice has led to an enhanced understanding of how intertemporal choices might be implemented in the brain [14][15][16][17], and, as we document, has already begun to inform economic modeling and to provide new clues about productive empirical and theoretical avenues for future research.

Time discounting
The great strengths of the DU model are its simplicity and generality. DU is easy to apply mathematically to any kind of intertemporal choice. According to DU, intertemporal choices are no different from any other type of choices except that some consequences are delayed, and hence must be anticipated and discounted (i.e. reweighted to take account of delay). Much of the research on intertemporal choice has, therefore, focused on the degree to which people anticipate and discount future events.
Numerous experiments in animals, notably rats and pigeons, have shown that under operant conditioning paradigms, the effectiveness of a reinforcer diminishes the further in time it is delayed [18]. In pigeons, for instance, the reinforcement value of three units of reward available in 11 s is approximately equal to the reinforcement value of eight units of reward available after 20 s [19]. The traditional model of intertemporal choice uses 'exponential discounting', in which a reward of magnitude x occurring at some time t in the future is worth δ t x, where δ ≤ 1 is a fixed constant (the discount factor). In other words, the value of the reward decays by the same proportion for each minute that its occurrence is delayed. Figure 1 plots three different discount functions, including an exponential function with δ = 0.95 .   61  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  However, the bulk of the evidence (primarily from rats and pigeons) suggests that animals discount the future in a non-exponential manner. The most commonly described discounting behavior is hyperbolic, which means that delayed rewards are discounted by functions that are inversely proportional to delay -for example, 1/t or generalizations thereof [18][19][20][21]. Hyperboloid discount functions decay at a more rapid rate in the short run than in the long run, so a hyperbolic discounter is more impatient when making short-run tradeoffs than when making long-run tradeoffs. Figure 1 also  Some researchers have speculated that the difference between humans and other animals lies in our ability to form a mental image of, and care about, delayed outcomes [24], and there is widespread agreement that the prefrontal cortex, which is disproportionately large in humans relative to other species, has an important role in this capability. The first clues about the function of the prefrontal cortex came from people who experienced damage to it, either through accident, stroke or frontal lobotomy [24][25][26]. Studies have traced the development of self-control capabilities in children to the maturation of prefrontal areas [27], and still other studies hav connected criminality and violent out-of-control behavior to childhood injury to prefrontal regions [28,29]. Human undoubtedly share with other animals the mechanism that produce rapid hyperbolic time discounting, but w also have the capacity, seemingly enabled by th prefrontal cortex, to make decisions that take account of much longer span of time. All of these pieces of evidence, as well as the commo observance in humans of extremes in apparent regard (o disregard) for the future, have led to a perspective that i both new and old. According to this perspective, tim discounting in humans results from the interaction of tw systems, one which is capable of anticipating and carin about the distant future, and the other which is muc more oriented toward the present. Empirical support fo such a perspective comes from a recent study in whic subjects' brains were scanned while they made choice between smaller money amounts that could be receive earlier and large amounts that could be received later [14 Some of the choices were between an immediate and a delayed payment, and others were between delayed and even more delayed payments. The researchers found that prefrontal regions were involved in all intertemporal choices (relative to rest) but that the mesolimbic dopamine system and associated regions were involved only in choices with an immediate outcome. Moreover, when immediate payment was one of the options, the relative activation of the two regions (prefrontal or dopamine) was a significant predictor of choice. This research lends support to the idea that hyperbolic time discounting results from the splicing of two systems with different perspectives toward the future, and that the prefrontal cortex has an especially important role in implementing more patient preferences. However, it does not provide definitive evidence of causal relationships, because the data are purely correlational.

Other dimensions of intertemporal choice
Time discounting might be the most frequently studied aspect of intertemporal choice, but it is only one of several dimensions that come into play. In this section, we discuss three other mechanisms that, prior research suggests, have an especially important role in intertemporal choice: 'anticipation', 'self-control' and 'representation'. Anticipation refers to an individual's propensity to imagine, and experience pleasure and pain in anticipation of, a future event. Self-control refers to the tensions that people experience when they attempt to implement a farsighted decision in the presence of immediate temptation. Representation refers to the way that the brain interprets or frames a set of choices. Representation often happens first in a decision time-line, but we discuss representation last because less is known about this component of intertemporal decision making. Although these mechanisms, in some situations, come into competition with time discounting, in other situations they contribute to it. Indeed, as touched upon above, there is some question of whether these are the mechanisms underlying time discounting.

Anticipation
The classical economic model of intertemporal choice assumes that choices have no utility consequences other than the consumption events that result from those choices. For example, the pleasure of a decadent meal is assumed to arise from the meal itself and not the awareness, before the event, that it will take place. In practice, however, when a plan is made in advance -for instance a dinner reservation -there is a waiting period during which the future outcome is anticipated. Moreover, this period of anticipation might have its own affective consequences for the actor. The period between decision and outcome has received relatively little consideration from economic researchers because economic models typically do not treat purely mental events as intrinsic sources of utility [30].
From a behavioral perspective, however, both animals and humans experience subjective changes in mental state associated with this continuous period of anticipation. When rats are conditioned to associate a neutral stimulus with a noxious outcome (a loud noise), they enter a state of physiological arousal between the stimulus and outcome. The degree of arousal is associated with their tendency to 'startle' in response to the noise. Hence, the startle response serves as a measure of the degree of learning that has occurred [   1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39   The anticipation of an outcome can lead to physiological arousal, but does this state of anticipation enter into the decision-making process? Under certain circumstances it does. Consideration of the anticipation of a particularly pleasurable event, such as the promise of a kiss from a movie star, or the dread of something painful, such as an electric shock, often enters into the decisions that people make; for example, causing them to get unpleasant outcomes over with quickly to eliminate what otherwise would be an aversive period of waiting [36,37], behavior that is contrary to the most basic prediction of the DU model, assuming that people discount the future. A concise explanation of this phenomenon is that anticipation can confer utility (or disutility) in, and of, itself. Human neuroimaging data demonstrate that activity in regions associated with the experience of pain increases in anticipation of delayed painful stimuli [38][39][40][41][42][43][44] Anticipatory responses to appetitive stimuli are also common in neural systems, although these tend to be in different regions than for aversive stimuli. Anticipatory activity in the ventral striatum and orbitofrontal cortex has been associated with the prospect of receiving a financial windfall [45][46][47] It is often difficult to wait for a delayed reward when a immediately gratifying alternative is available. Fo instance, quitting smoking is difficult because cigarette are available at every news-stand and drug store. Situations such as this can lead to 'preference reversals', wherein people initially decide to take a far-sighted cours of action -quitting smoking -but subsequently succum to temptation [20]. Preference reversals are observabl phenomena that point to the weaknesses of standard DU theory, and they occur in a wide variety of circumstances Although it is possible, as we shall see, to modify th discount function in a way that explains preferenc reversals, the core mechanism might be generated b phenomena other than the discount function. Successful implementation of a far-sighted plan of behavior, such as ending a bad habit, thus involves at least two distinct components. First, the individual needs to make an initial far-sighted decision, which is likely to depend on the ability to anticipate future consequences. Second, she needs to resist short-run temptations, which will undermine her ability to implement that decision. Any successful model of intertemporal choice should incorporate features that accurately describe the tug of war between long-run ('virtuous') intentions and short-run temptations.
As a benchmark, the DU model fails this descriptive challenge. As Samuelson [52] noted, the DU model (with exponential discounting) implies that resolutions once made are never broken. Economists refer to this property as dynamic consistency. Anyone who follows the exponential discounting model will be dynamically consistent -they will never change their state-contingent preferences. Plans or preferences made for the future will be the same as decisions executed at the moment of action. In this framework, resolutions to quit smoking or stick to a diet are always carried out (unless new decision-relevant information arrives).
Real people don't have such exquisite self-command [20,53]. Most people experience preference reversals: plans made at one date are broken at some later date. For instance, estimates of relapse rates exceed 50% during the first year after quitting smoking. Many other types of behavior illustrate this tendency to backslide, including credit card spending, exercise and nutrition [54][55][56]. Beginning with the groundbreaking work of Ainslie [2,20], these types of effects have been integrated into models of time discounting. The exponential discounting model counterfactually rules out preference reversals. However, any other discounting behavior has the potential to generate preference reversals, which economists refer to as dynamic inconsistency. This potential was first discussed by Samuelson [52] and then developed by others [22,57]. Most research has focused on the class of hyperbolic [2,7] and quasi-hyperbolic discount functions [13], which predict that agents will make patient plans and then break them at the moment of execution (Box 1).

Representation
Economic analysis assumes that how a choice is represented is an objective matter. But, in fact, it is possible to mentally represent the same situation in a variety of different ways [82]. People use a wide range of choice heuristics to make the decisions they face and which heuristics come into play depends crucially on how they construe these decisions [83,84]. As a result, differences in context or in the way that a decision is 'framed' or cognitively construed can have an impact on the intertemporal tradeoffs that people make.
A child's ability to delay gratification depends on the manner in which the child is instructed to mentally represent a reward [9,85]. When given a choice between an immediate single pretzel or two delayed pretzels, children were more likely to wait if instructed to represent the pretzel in pallid or unappealing terms -for instance, as 'little brown logs' -than if they were to represent the pretzel in consumatory terms -'yummy, tasty'. In research with adults, Wilson  A variety of studies have shown that framing an intertemporal choice in a fashion that draws more attention to the need to wait during the delay interval tends to produce steeper time discounting -less willingness to delay. For example, subjects are much less willing to delay gratification when they made a choice that was expressed in terms of delay than when the same choice was expressed in terms of speed-up or simply as a choice between outcomes at two different points in time [37]. More recently, several studies have shown that people tend to display flatter time discounting when the delay interval of an intertemporal choice is presented in terms of dates -for example, x today or y on a particular date -than when expressed in terms of a delay intervalfor example, x today or y after a wait of z days (where the interval in the two choices is equal) [87].  One important choice heuristic that people seem to employ is to choose sequences of outcomes that improv over time -a pattern of choice that effectively results i 'negative time preference': subjects prefer to have th smaller rewards early and the larger rewards later contrary to what the DU model would predict. However whether a particular intertemporal choice is represente as a sequence, and hence whether this heuristic is applied, can depend on relatively subtle factors. In the firs demonstration of this point, Prelec and Loewenstein [88 asked some subjects to hypothetically choose whether to consume a fancy French dinner on the following weeken or on a weekend one month later. Most subjects chose t have the French dinner on the earlier date. However when the decision was represented as a sequence of tw events on fixed dates, where subjects could choose to eat at home on one weekend and eat the fancy dinner on th other, a majority of subjects now chose to delay the fanc French dinner to the later date. Later research found tha the more coherent a sequence was made to seem, the mor probable subjects were to opt for improving sequences [89

Conclusion
The research reviewed above identifies three operation that affect intertemporal choice. Anticipation produce immediate hedonic consequences, even when the anticipated consumption event is delayed in time. Selfcontrol is used to resist temptations to reverse patient plans. Representations evoke specific choice heuristics that increase or decrease the salience of delayed rewards and make waiting more or less aversive. Any comprehensive account of intertemporal choice should incorporate all of these mechanisms. At the moment, we know little about how these mechanisms interact, which should be a priority for future research. At the most general level, it is important to determine whether the brain has one all-purpose time discounting mechanism or whether the brain draws upon different systems, each with its own occasionally competing time perspective.

Box 1. Modeling preference reversals
Standard economic theory assumes that individuals (agents) have preferences that are stable through time. In this context a preference refers to a rank ordering of outcomes, or choices, that an individual makes. For example, a person might be said to prefer tea over coffee. However, actions speak louder than words and simply professing such a preference is no guarantee that, given a choice, such an individual would actually choose tea. Because of the hidden nature of preferences, eliciting choices (e.g. through forced-choice or willingness-to-pay) is the only reliable way to measure preferences. Even so, individuals often exhibit reversals in their apparent preferences when it comes to delayed outcomes. Dieting, for example, often falls into this trap of preference reversals. An individual makes a New Year's resolution to lose weight (a temporally remote outcome), but when confronted with the deliciousness of food, changes his mind (a temporally immediate outcome). Such preference reversals can be modeled in terms of a non-exponential discount function. Assume that an economic agent has a quasi-hyperbolic discount function: 1, β·δ, β·δ 2 , β·δ 3 , ….. (Figure 1). In general, this discount function is parameterized with 0 < β < 1 and 0 < δ < 1, but to simplify the illustrative example, set β = 1/2 and δ = 1, so the discount function takes the form 1, 1/2, 1/2, 1/2, ….,. Immediate payoffs have a weight of one and all future payoffs have a weight of 1/2. Assume that an investment activity has an immediate cost of four and a delayed benefit of six.
If agents anticipate such preference reversals [57], they might find ways to commit themselves in advance -for instance, scheduling an appointment to exercise with a trainer or putting their saving into illiquid accounts [13]. If agents fail to anticipate their preference reversals, they might engage in patently self-defeating behaviors, such as perpetually paying monthly dues at a gym that they never attend [54] or, more generally, procrastinating [58,59].
The predictions of the basic hyperbolic discounting model have been experimentally and empirically validated [20,60]. But the basic hyperbolic discount function provides only a partial account of intertemporal preferences [6]. Most importantly, temporal immediacy of rewards is only one of many factors that seem to produce impulsivity. Other factors include sensory proximity -the sight, sound, smell or touch of a desired reward -and the activation of drive states, such as hunger, thirst or sexual arousal. Thus, for example, mild opioid deprivation in a population of heroin-addicted outpatients produces greater discounting of monetary rewards [61]. Likewise, nicotine deprivation among smokers also produces greater monetary discounting [62,63]. People often lose control in the 'heat of the moment' or when willpower is depleted [64].
Although preference reversals are often attributed to hyperbolic time discounting, they can also result from other mechanisms (which themselves, in some cases, can help to explain hyperbolic time discounting). Three (overlapping) categories of mechanisms are visceral influences, cue-contingent influences and temptation preferences.
Visceral influences are associated with emotion and affect, and are directly related to changes in drive state. Visceral preferences are generated by immediate biological imperatives -for instance, thirst, hunger, sexual arousal, exhaustion, pain, the need to physically dominate an opponent, or fear for physical safety. Loewenstein has argued that visceral needs often overwhelm other goals and produce short-sighted behavior [65]. This assumption has also been adopted in a two-state decision-making model [66]. In the cold state, the decision-maker is guided by forward-looking rational deliberations. In the hot state, the decision-maker is completely controlled by her myopic visceral needs. Hence, highly impatient behavior would be associated with time periods in which the visceral preferences are dominant, explaining many addictive behaviors, including excess use of an addictive substance and relapse after detoxification.
Cue-contingent preferences have been studied since Pavlov's feeding experiments [67]. Cue-contingent preferences are formed when a neutral stimulus is repeatedly paired with a non-neutral stimulus, such as a consumption event. The end result is a change in drive state, even though the eliciting stimulus was, at one point, neutral. For instance, a heroin user might come to associate the visual stimuli of a certain  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24   25  26  27  28  29  30  31  32  33  34  35  36 environment with ingestion of heroin. Such pairings might be strong enough to elicit cue-contingent drug cravings and cue-contingent tolerance, so that the user's desire to take heroin becomes much stronger when the cues are present [68]. Cue-contingent cravings might produce preference reversals, transitory efforts to achieve immediate gratification, and forward-looking efforts to modify cue exposure [65,66,69]. Indeed, several brain-imaging experiments have demonstrated the powerful effect of showing pictures of drug-related paraphernalia to people who are addicted to these substances [70][71][72][73][74][75]. Although craving, in and of itself, does not represent a breakdown in self-control, it does represent an emotional state that places the individual at risk for a preference reversal. The biological substrates of craving, however, are complex and recruit a wide range of circuits in the brain that include memory regions such as the hippocampus, executive control regions in the prefrontal cortex, and visceral regions such as the insula. However, no single brain region has been demonstrated to be singularly responsible for self-control. Instead, multiple systems process different psychological dimensions of competing preferences.
Temptation preferences arise in two-system models and are another way of describing the temporal immediacy effect of rewards by invoking the cost of self-control [66, [76][77][78]. Rather than postulating a non-exponential discount function, temptation preferences are typically modeled as a drive for immediate gratification, which can be cognitively overridden with some utility cost generated by mental effort (selfcontrol). In the models cited here, the cost is associated with the degree to which the impatient preference is violated. The end result, however, is the same as a non-exponential discount function. For example, imagine that an agent has a craving to eat a (full) bowl of ice cream sitting in front of him, but allows himself to eat only some fraction of that bowl. Temptation models assume that the cost of temptation is falling in the amount that the agent eats. If the agent eats nothing, then temptation costs are maximal. If the agent eats the whole bowl, then temptation costs are zero. Temptation preferences are one way of formally modeling the interaction between the patient (cortical) system and the impatient (mesolimbic dopamine reward related) system. Little is known about the nature of the interaction of these two putative systems, but one brain region, the anterior cingulate cortex (ACC), is thought to have a role in mediating the conflict between competing actions [79,80]. The exertion of self-control requires the suppression of either cravings or temptations, which are the types of competing responses that the ACC modulates. Another region, the inferior prefrontal cortex, seems to be involved in achieving self-control by inhibiting one of these responses [81]. Importantly, how the ACC processes these conflicts and how the inferior prefrontal cortex inhibits one or another depends on the context in which these temptations occur, which leads to the third aspect of intertemporal choice: representation.

Box 2. Directions for future research
How can neurobiological data be used to develop and test models of intertemporal choice? In the past, the tautology of choice and preference has excluded analysis of neurobiological mechanisms. In recent years, a growing body of data based on brain imaging is enabling researchers to link intertemporal decisions to neural activation patterns, producing both new empirical regularities and new controversies [14,90,91]. The challenge will be to marry neurobiological descriptions with theoretical ones. Exponential discounting assumes a constant rate of discounting, e.g. δ t where δ is the discount rate (here, δ = 0.95). Hyperbolic discounting is generally greater for short time periods than long periods, and can be described by a function of the form 1 / (K * t + 1). Here, K = 0.1. Quasi-hyperbolic discounting is a piecewise function that follows a form similar to exponential discounting after the first discount period (i.e. the first year): 1, β·δ, β·δ 2 , …, β·δ t . Here, β = 0.792 and δ = 0.96).