In deep learning and reinforcement learning (RL), optimization is at the heart of progress. While many of us simply call optimizer.step() in our training scripts, there lies a rich world of ideas and techniques that can improve both the stability and performance of our models. In this article, we dive into the concept of trust regions in optimization, examine how they helped overcome issues in RL, discuss their applications in other areas, and speculate on why renewed interest in these methods might be on the horizon with the rise in available compute.

1. Introduction to Optimization

In deep learning, optimization is indispensable it kept Walter White out of jail — it is the very core of training any neural network. In this article we will consider a simple classifier: we design a network that transforms raw inputs into feature representations, and the final layer is optimized to create a space where classes become linearly separable. While early layers perform feature extraction, it is the final output layer - the REPRESENTATION - (and its loss function) that drives the overall performance. In this way, the optimizer is the guiding light, adjusting every parameter via backpropagation of an error (that is a loss function).

Quick Overview of Optimizers in Deep Learning

The network defines a mapping from input data to a feature space, and the optimizer guides the transformation of these features so that they are easily separable by a linear classifier (such as logistic regression). The standard practice is to use first-order (gradient-based) methods, striking a sweet trade-off between speed and accuracy. However, not many of us (not you optimization folks) really try to go on different sides of this spectre. In this article, we will cover the side of accuracy of the spectre. This is where second-order methods come in. They provide a more nuanced view of the optimization landscape by considering curvature information. But one that we will eventually talk about is trust region.

What Is a Trust Region and Why Do We Care?

Trust regions is a region around the point in the optimization space where the approximation is being “trusted” to be correct enough to objective function. That sounds quite vague. Lets build a more intuitive way of thinking of it (mathematicians close your eyes here and skip). When we optimize some function (e.g. loss w.r.t parameters of neural network), we can express all the parameters as some point in optimization space. Within this space we opt to find a point where our parameters minimize our loss function well and to do so, we refer to optimization (take gradient, subtract and etc). However, this all becomes meaningless once we have either too big of a learning rate (too big fluctuations so that we cant settle) or too small (wait infinite time for convergence). Hence, we need a sweet spot where the learning rate as big as possible for faster convergence, but bounded by the fact that we really would not like to fly over the optimal point. This is rather a challenging task and often it is one of the most (if not the most important) hyperparameter. But what if I tell you that here trust regions come into play? What if I tell you that we can deduce a certain radius around a point where we can estimate that we shall not mess up hard. A lawyer region you can trust to be safe to be optimized within. It is the intuitive explanation of what trust region represent in optimization.

2. Why Trust Regions Began to Shine in Reinforcement Learning

The Issue of a Goldfish Memory Agent

One of the early algorithms in RL is the REINFORCE (or Vanilla Policy Gradient (VPG)) method introduced by Williams (1992). Its update rule is elegantly simple and intuitive. After an episode, the gradient of the log-probability of the taken actions is scaled by the return, (intuitively it translates to if this action led to a higher return do a bigger step here) leading to the following update equation:

$$ \theta_{t+1} = \theta_t + \alpha , \nabla_\theta \log \pi_\theta(a|s) , R $$

While REINFORCE is conceptually straightforward, it suffers from high variance and instability in practice. For example, an agent might learn to pass the ball one episode, only to forget it a few episodes later when it learns how to shoot over-adjustsing to new, potentially suboptimal behaviors — a thing that remind me of “goldfish memory.”

TRPO: Fixing the Goldfish Memory Problem

To mitigate this instability, Trust Region Policy Optimization (TRPO) was introduced by Schulman et al. (2015). The central idea of TRPO is to restrict the update step to a trust region in the space of policies so that the new policy does not deviate too much from the old one. This constraint helps preserve previously learned behaviors while still enabling improvement. Intuitively, it translates to “do not jump to regions where you may forget how to pass a ball before you learn to shoot”

TRPO formalizes its update as follows:

$$ \max_{\theta} \mathbb{E}_{s,a \sim \pi_{\theta_{\text{old}}}} \left[ \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)} A^{\theta_{\text{old}}}(s,a) \right] $$ $$ \text{subject to} \quad \mathbb{E}_{s \sim \pi_{\theta_{\text{old}}}} \left[ D_{\text{KL}}\Big(\pi_{\theta_{\text{old}}}(\cdot|s) \,\|\, \pi_\theta(\cdot|s)\Big) \right] \leq \delta $$

where:

  • $( \pi_\theta(a|s) )$ is the policy parameterized by $( \theta )$,
  • $( A^{\theta_{\text{old}}}(s,a) )$ is the advantage function computed under the old policy,
  • $( D_{\text{KL}}(\cdot|\cdot) )$ denotes the Kullback–Leibler divergence,
  • $( \delta )$ is a predefined threshold.

By enforcing the KL-divergence constraint, TRPO ensures that each update remains within a “trusted” region of the parameter space, thereby reducing the risk of catastrophic updates (that would make agent less goldfish memorish and more EMERGENT in behaviour)

So, problem solved?

I guess, by now, you are wondering for a question, why would I even bother with all that and talk about it if it was never in deep learning tutorials and not like millions of engineers talk about it as they talk about RLHF or any other thing. The main issue that is concealed here is how to actually respect that constraint. Yes, we have a formal update rule, but lets dive deeper into understanding: from the definition we know that trust region is some form of constraint, so we have to solve our primary objective (loss that we optimize) along a constraint (the KL divergence). That makes a problem a quadratic optimization problem that would require us to use a second order optimization method (yes, that I mentioned earlier). Aaaaand yes, compute Hessian. That is the main issue that lies here - computation. We will cover it a little bit later, but now lets dive deeper into applications of it.

3. Applying Trust Regions Beyond RL

Although trust regions were popularized in the context of policy optimization, their core idea has far-reaching applications in other areas of machine deep learning and optimization.

Implicit Trust Regions in Supervised Learning and Data Augmentation

In supervised learning, many modern techniques implicitly use trust region ideas. For instance:

  • Data Augmentation: By enriching the training dataset with perturbed samples, data augmentation methods define a region in the input space where the network’s predictions are expected to remain invariant. This can be seen as an implicit trust region on the input distribution. Intuitively, we say these images are still ok lil bro
  • Regularization Techniques: Methods like dropout or weight decay restrict the model’s capacity to change drastically, effectively constraining the optimization steps to remain in a “safe” region. Intuitively, we just try to

Adaptive Optimizers

Adaptive optimizers such as AdaGrad, RMSprop, and Adam (sorry I got tired of citing, but all creds to authors) adjust the learning rate on a per-parameter basis. While not explicitly formulated as trust region methods, these optimizers adjust the update step size based on local curvature information and historical gradient statistics. In doing so, they implicitly control the “trust” in the current gradient direction, ensuring that updates do not stray too far from reliable estimates. They rely on the same intuition on making a region of optimization “safe” to jump further

4. Simplifications to Trust Regions: The Case of PPO

While TRPO provides a solid theoretical foundation for stable policy updates, it is computationally expensive due to the constrained optimization problem it poses. This complexity spurred the development of simpler methods that approximate the benefits of trust regions without the heavy computational cost.

Proximal Policy Optimization (PPO)

Proximal Policy Optimization (PPO) by my beloved king Schulman et al is one such method that simplifies TRPO by implementing a clipped surrogate objective that “mimics” the trust region, without explicitly doing it.

$$ L^{\text{CLIP}}(\theta) = \mathbb{E}_t \left[ \min\Big( r_t(\theta) A_t, ; \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t \Big) \right], $$

where:

  • $( r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)} )$ is the probability ratio,
  • $( A_t )$ is the advantage estimate,
  • $( \epsilon )$ is a hyperparameter that determines the clipping range.

PPO continued

To be fair, they also presented a version where they just subtracted KL divergence, but it is of less interest as clipped surrogate objective goes brr

5. Why Growing Compute May Spark Interest in Trust Regions Again

Trust region methods, despite their appealing theoretical guarantees, have historically been sidelined in favor of more computationally efficient first-order methods (say TRPO « PPO). The constrained optimization and the need to compute (or approximate) second-order information have been seen as a major drawback. However, there is one precise trend that drives a perspective change:

  • COMPUTE: Modern hardware (GPUs, TPUs, and specialized accelerators (Hi DeepSeek)) now makes it feasible to incorporate more computationally intensive methods into training routines (and memory becomes cheaper, Jensen, please)

The trade-off between computational cost and optimization stability is becoming less severe as compute resources grow, possibly leading to a renaissance in trust region methods across various domains. So, who knows maybe we will see another trust regions reign soon.

Conclusion

From the early days of REINFORCE to the sophisticated formulations of TRPO and the simplified yet effective PPO, trust regions have played an important part in learning to shoot while being able to passstabilizing and improving the optimization process in reinforcement learning. Their underlying principles are not confined to RL: they can be seen across many areas in deep learning — from data augmentation to adaptive optimizers that guard against bad updates.

As compute power continues to increase and the demands on model robustness intensify, revisiting trust region methods may offer new understanding and intuitino for stable and efficient learning. Whether you are fine tuning BERTs or develop AGI, understanding and leveraging trust regions can provide that extra layer of reliability and intuition in your optimization process.