← Notes

Deep Reinforcement Learning

Before diving into deep reinforcement learning, let's first motivate the key idea behind it.

Motivation

The world evolves over time, often in ways that are uncertain or difficult to predict. Rather than describe every quantity that could possibly characterize the world, we represent it using a collection of variables relevant to the problem at hand. We call this representation the state \(s_t\).

If the evolution of this state is uncertain, we can describe it probabilistically:

\[ s_{t+1} \sim P(s_{t+1} \mid s_t) \]

Now suppose an agent is embedded within this world and can take actions. The next state is no longer determined only by the current state; it also depends on what the agent chooses to do. We therefore extend the model to include the agent's action \(a_t\):

\[ s_{t+1} \sim P(s_{t+1} \mid s_t, a_t) \]

This tells us how the world may evolve in response to the agent's actions, but it does not yet tell us which outcomes the agent should prefer. To turn this into a decision-making problem, we need some way to express what constitutes a desirable or undesirable transition.

We do this with a reward function. For each transition, the reward assigns a scalar value that indicates how desirable that outcome is:

\[ r_t = R(s_t, a_t, s_{t+1}) \]

The transition model \(P\) therefore describes what can happen, while the reward function \(R\) describes what we want to happen.

We now have the basic ingredients required to describe sequential decision making in a stochastic environment. This framework is known as a Markov Decision Process (MDP):

\[ \mathcal{M} = (\mathcal{S}, \mathcal{A}, P, R) \]

where:

Markov Decision Process

Reinforcement Learning

The MDP gives us a framework for describing how an agent interacts with a stochastic environment, but we still need to define what it means for the agent to behave well.

Reinforcement learning makes this precise through the reward hypothesis: the agent's goal can be expressed as maximizing cumulative reward over time.

\[ J(\pi) = \mathbb{E}_{\tau \sim \pi} \left[ \sum_{t=0}^{T} r_t \right] \]

Rather than judging each action only by its immediate reward, we evaluate a policy according to the sequence of rewards it produces.

More generally, future rewards can be discounted:

\[ J(\pi) = \mathbb{E}_{\tau \sim \pi} \left[ \sum_{t=0}^{T} \gamma^t r_t \right] \]

where \(\gamma \in [0,1]\) is the discount factor. It determines how strongly rewards farther into the future contribute to the objective.

The expectation appears because the trajectory produced by a policy is generally not deterministic. The policy itself may be stochastic, the environment may be stochastic, or both. As a result, executing the same policy can produce different trajectories and therefore different returns.

We therefore do not optimize the return of a single trajectory. Instead, we maximize the expected return: the average return over the distribution of trajectories that the policy may produce.

Reinforcement learning then asks how an agent can learn a policy that maximizes this objective through interaction with the environment.

At each timestep, the agent observes the current state \(s_t\) and selects an action according to its policy:

\[ a_t \sim \pi(a_t \mid s_t) \]

The environment then responds to that action by producing a reward \(r_t\) and transitioning to a new state \(s_{t+1}\). The agent observes this new state, chooses another action, and the process repeats:

\[ s_t \;\rightarrow\; a_t \;\rightarrow\; r_t,\; s_{t+1} \;\rightarrow\; a_{t+1} \;\rightarrow\; \cdots \]
The agent-environment interaction loop in reinforcement learning

Diagram source: Gymnasium [1]

Through repeated interaction, the agent gathers experience about how its actions influence both immediate and future rewards. Reinforcement learning uses this experience to improve the policy so that the trajectories it produces achieve greater expected cumulative return.

Deep Reinforcement Learning

Deep reinforcement learning extends reinforcement learning by using deep neural networks to represent the functions that the agent must learn.

For example, we can represent the policy as a neural network parameterized by \(\theta\):

\[ \pi_\theta(a \mid s) \]

where:

The reinforcement learning objective itself does not change. We still want to find a policy that maximizes expected cumulative reward. The difference is that the policy is now represented by a high-capacity function approximator, and learning the policy becomes a problem of finding neural network parameters \(\theta\) that maximize the same objective:

\[ J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ R(\tau) \right] \tag{1} \label{eq:policy-objective} \]

The learning problem can therefore be written as:

\[ \theta^* = \arg\max_\theta J(\theta) \]

The remaining question is how to adjust \(\theta\) so that \(J(\theta)\) increases.

Find out here.

References

  1. Gymnasium: A standard API for reinforcement learning and a diverse set of reference environments. https://gymnasium.farama.org/.
  2. OpenAI. ChatGPT. https://chatgpt.com/.