Glossary Entry

Temporal-Difference Learning

Value-learning methods that update predictions from the gap between successive estimates, r + gamma V(s') - V(s), rather than waiting for the final outcome, trading a little bias for much lower variance.

RL Decision Making

Also called: TD learning, TD error, temporal difference

Seed source: Sutton & Barto, Reinforcement Learning - An Introduction

Temporal-difference learning updates a value estimate as soon as one step of experience arrives: compare what you predicted before the step with the reward you got plus what you now predict from the next state, and move your prediction toward that target. That gap is the TD error, and learning from it means you never have to wait for an episode to finish.

The catch is that the target contains your own current estimate of the next state, so while that estimate is wrong the updates are biased; in exchange, the variance is far lower than Monte-Carlo methods that use full returns. In actor-critic methods the TD error plays a second role: it is an estimate of the advantage of the action just taken, and so also serves as the actor’s learning signal.