I want to highlight this paper (from Sept 29, 2025) of an alternative to RL (for fine-tuning pre-trained LLMs) which:

Performs better
Requires less data
Consistent across seeds
Robust (ie don't need to do a grid search on your hyperparameters)
Less "Reward Hacking" (ie when optimizing for conciseness, it naturally stays close to the original model ie low KL-Divergence)

They claim the magic sauce behind all this is the evolutionary strategy optimizing over distributions of model parameters. Surprisingly, they've scaled this to optimize over billion-parameter models.

Let's get into their method.

Evolutionary Strategy (ES) Algorithm

They start w/ a "Basic ES Algorithm" which is:

In other words, we're gonna sample noise around the original model's weights N times (ie we're going to explore around the model weights where the variance I is the identity covariance).