The advantage function rephrases “was this a good action?” as “was this action better than what the policy usually achieves from here?”. A positive advantage means the action beat expectations and should become more likely; a negative one means it underperformed and should be suppressed; an action that merely met expectations contributes nothing to the update.
Centring updates this way is what makes advantage-weighted policy gradients so much less noisy than reward-weighted ones. Because the true advantage is unknown, algorithms estimate it: one-step TD errors, blends of them like Generalized Advantage Estimation, or, in critic-free methods like GRPO, the reward of a sample measured against the average of its peers.