Glossary Entry

Actor-Critic

A reinforcement learning architecture that pairs a policy (the actor) that chooses actions with a learned value function (the critic) that evaluates them, using the critic's feedback to reduce the variance of policy updates.

RL Decision Making

Also called: actor critic, actor-critic methods

Seed source: Sutton & Barto, Reinforcement Learning - An Introduction

An actor-critic agent learns two functions at once: the actor is the policy that decides what to do in each state, and the critic is a value function that predicts how well things usually go from that state. After every step, the gap between what the critic predicted and what actually happened (the TD error) tells the actor whether its action was a positive or negative surprise, and both networks update from that one signal.

The construction exists to fix the high variance of pure policy-gradient methods: the critic is a learned, state-dependent baseline that keeps the expected update the same while shrinking its noise, at the cost of some bias while the critic is still inaccurate. The pattern is the skeleton of most well-known deep RL algorithms, including A2C, A3C, DDPG, TD3, SAC and PPO.