Policy-gradient methods skip learning a value function and optimise the policy itself: sample trajectories under the current policy, then adjust the parameters so actions that led to high reward become more likely. The key identity is that the gradient of expected return equals the expectation of the gradient of the log-policy weighted by reward, which can be estimated purely from sampled experience.
The simplest instance, REINFORCE, weights each action by the whole trajectory’s return, which is unbiased but very noisy. Most practical descendants are about taming that variance: subtracting baselines, weighting by advantages instead of raw returns, or constraining update sizes, which leads naturally to actor-critic methods and PPO.