Reinforcement learning objective and reward maximization

Summary

In reinforcement learning, the objective is to learn a policy that maximizes an accumulated reward signal over time. The agent interacts with an environment modeled as a Markov decision process: at each step it observes a state, selects an action according to a policy, receives an immediate reward from the transition, and the environment moves to a new state. Because immediate rewards may be negative, the agent must reason about long-term consequences, creating a long-term vs. short-term reward trade-off. This reward maximization is formalized using value functions. The state-value function Vπ(s) estimates the expected discounted return starting from state s and following policy π, where future rewards are weighted by a discount factor γ. More generally, the optimality criterion uses the maximum expected discounted return across policies, leading to optimal value functions V*(s) and action-value functions Q*(s,a). Knowing the optimal action-value function is sufficient to act optimally by choosing, in each state, the action with the highest Q-value. Value-function methods and direct policy search are two broad approaches to optimize these objectives.

Key Points

• 1

  RL maximizes expected cumulative (discounted) reward by learning a policy from interaction with the environment.

• 2

  Value functions (Vπ and Qπ) quantify how good states or state-action pairs are under a policy, enabling reward maximization.

• 3

  Optimality is defined via maximum expected discounted return (V*(s), Q*(s,a)), and acting optimally corresponds to choosing the highest Q-value action in each state.