Glossary Entry

Markov Decision Process

The standard formalism for sequential decision problems, defined by states, actions, transition dynamics, a reward function, and a discount factor, where the next state depends only on the current state and action.

RL Decision Making

Also called: MDP, MDPs, Markov decision processes

Seed source: Sutton & Barto, Reinforcement Learning An Introduction

A Markov decision process packages a sequential problem into five ingredients: the states the world can be in, the actions available, the transition probabilities that say where an action leads, the reward received along the way, and a discount factor that trades near-term against long-term reward. The “Markov” part is the key simplifying assumption: the future depends only on the present state and action, not on the full history.

Almost every reinforcement learning method starts by casting its problem as an MDP, because once a problem is in this form, value functions, policies, and the Bellman equation all apply. Real systems like robots usually violate the assumptions a little (the true state is only partially observed through sensors), which is why policies in practice often consume histories of observations rather than a single snapshot.