Storyboard

Decision making under uncertainty is the process of selecting an action when the outcome is not completely predictable. Expected utility theory (Von Neumann-Morgenstern) provides the normative framework: optimal action maximizes expected utility EU = P(o|a)·U(o). However, real organisms systematically deviate from this optimum in telling ways.

The drift diffusion model (DDM) is the most successful mechanistic model of the decision process: evidence is accumulated with drift (proportional to SNR) and noise until a threshold is reached. The threshold controls the speed-accuracy trade-off: higher slower but more accurate. Neurons in LIP and area MT show ramping activity consistent with this model.

Hyperbolic temporal discounting V = R/(1+k·t) describes how organisms devalue future rewards: unlike exponential discounting (V = R·e^(-k·t)), hyperbolic produces temporally inconsistent preferences (preferences reverse as time passes). This 'irrational' behavior is observed in humans, rats, pigeons and non-human primates.

The reward prediction error signal (_t = r_t + ·V(s_{t+1}) - V(s_t)) is the core of temporal reinforcement learning (TD-learning). Neurobiologically, midbrain dopaminergic neurons respond exactly with _t: they fire when an unexpected reward arrives ( > 0), they inhibit when an expected reward does not arrive ( < 0), and they do not respond to predicted rewards ( = 0).

Risk aversion ( < 1 in the utility function) explains why animals prefer safe options over equal rewards: marginal utility decreases with magnitude. However, in the loss domain, organisms are risk seekers (convex function), which is described by the prospect theory of Kahneman and Tversky. This asymmetry has evolutionary value: loss aversion is adaptive in environments with variable resources.

>Model

ID:('ky', 590)

Description

ID:(0, 590)