Both algorithms are state of the art – but they solve different problems optimally. This closing article compares PPO and SAC along all relevant axes (on-/off-policy, sample efficiency, stability, action space, parallelisability), provides a decision Mermaid, maps typical application areas (robotics, trading, RLHF, games) and bridges to the self-learning-agents series.
1. The Central Question: Which Problem Are You Solving?
A single method for all nails is even rarer in RL than elsewhere. The choice between PPO and SAC is not "better or worse" but the answer to two questions:
- How expensive is a world interaction? (real-world seconds vs. CPU seconds)
- What does your action space look like? (discrete vs. continuous vs. high-dimensional)
"Sample efficiency is a property of the environment, not of the algorithm. The right algorithm is the one whose data appetite matches your environment's data supply." – paraphrased from Levine, CS285 Lectures.
2. Direct Comparison: All Axes
| Axis | PPO | SAC |
|---|---|---|
| Policy type | On-policy | Off-policy |
| Data reuse | $K$ epochs per batch | Arbitrarily many updates from replay buffer |
| Action space | Discrete or continuous | Primarily continuous (discrete SAC variants exist) |
| Exploration | Entropy bonus (small, optional) | Maximum entropy native in objective |
| Sample efficiency | Medium | High (5–10× better on MuJoCo) |
| Stability | Very high (clipping) | High (twin-Q + target nets) |
| Implementation effort | Low (200 lines are enough) | Medium (replay, target nets, auto-α) |
| Parallelisation | Excellent (many parallel envs collect rollouts) | Limited benefit (bottleneck at the gradient update) |
| Hyperparameter robustness | High | Very high (auto-α) |
| Wall-clock time | Low when sim is cheap | Lower when sim is expensive |
Quick check: Which algorithm is more sample-efficient and why?
3. The Decision Path
In words:
- Discrete actions (e.g. Atari, strategy-game moves) → PPO is the standard pick.
- Continuous actions + expensive world (real robotics, wet lab) → SAC, clearly.
- Continuous actions + cheap, parallelisable sim (MuJoCo, Unity ML-Agents) → PPO wins on wall-clock time because hundreds of envs collect in parallel.
- Multi-modal solution desired (e.g. trading with two legitimate strategies) → SAC via maximum entropy.
4. Application Fields in Detail
Robotics & Sim-to-Real
SAC dominates. Every physical interaction costs seconds to minutes and mechanical wear. Sample efficiency is decisive.
| Aspect | Why SAC fits |
|---|---|
| Few real rollouts | Replay buffer recycles every trajectory |
| Continuous torques | Native Gaussian policy with tanh squash |
| Robust sim-to-real transfer | Maximum-entropy policies generalise better |
Games & Large Sim Clusters
PPO dominates. OpenAI Five (Dota 2), the AlphaStar predecessors and the Unity ML-Agents default stack are all PPO-based. The reason: thousands of cheap envs in parallel, gathered with minimal synchronisation. PPO's $K$-epoch update fits perfectly into a synchronous train step.
RLHF for LLMs
PPO is the current standard (InstructGPT line, Llama-Chat, Claude family). The task looks unusual at first glance – discrete tokens, very long trajectories, dense reward model – but PPO's stability and easy scaling have prevailed. Important adaptations:
- Reference-KL penalty in addition to clipping, to prevent drifting too far from the pretrained model.
- Reward model instead of a direct world reward.
- Token-level advantage with GAE.
Off-policy alternatives like DPO (Direct Preference Optimization) and GRPO have been gaining ground since 2024 – a trend also tracked in self-learning-agents/02-rl-fuer-code-agenten.
Trading & Finance Applications
Treacherous terrain. SAC is theoretically attractive (continuous volumes, native exploration), but exploration on a live market is genuinely expensive. In practice: SAC for backtests and simulation, followed by a conservative offline-RL method for production.
Quick check: Which algorithm currently dominates RLHF pipelines for LLMs?
5. Hybrid and Off-Policy Variants
The strict "on vs. off" split is being eroded by research:
- MPO (Maximum a Posteriori Policy Optimization) – DeepMind line, combines SAC-style stochasticity with PPO-style trust region.
- DPO / IPO / KTO – direct preference optimisation for LLMs, skips the RL step entirely.
- GRPO (Group Relative Policy Optimization) – DeepSeek variant, on-policy but critic-free via group baselines.
- AWR / CRR / IQL – pure offline RL from logs, no world interaction.
The trend: fewer separate phases, more blending. But PPO and SAC remain the building blocks you need to understand in order to classify the hybrids.
6. Bridge to the self-learning-agents Series
In self-learning-agents/02-rl-fuer-code-agenten we see how RL concepts feed into modern agent pipelines:
- Tests as a reward signal corresponds to a dense, sparse-reward MDP (article 01).
- Memory & replay for code agents is conceptually SAC's replay buffer (article 05).
- RLHF for code generation is PPO's home field (article 04).
- Exploration via stochastic tool selection is maximum entropy in action (article 05).
Whoever has understood PPO and SAC can analyse any modern agent-training pipeline (RLHF, agent-coder RL, AutoRL) along these axes.
7. Wrap-up Quiz
You have a 6-DoF robot arm and only a few thousand real trajectories. What is the best choice?
You train an agent in a Unity sim where 256 envs deliver rollouts in parallel very fast. What is the best choice?
Which of the following properties holds for PPO but *not* for SAC?
What does the statement 'PPO is semi-on-policy' mean?
Which statement about trends in modern LLM RL pipelines is correct?
8. Summary of the Whole Series
| Article | Topic | Core idea |
|---|---|---|
| 01 | MDP fundamentals | $(S, A, P, R, \gamma)$, Bellman, random policy as baseline |
| 02 | Policy gradients | Likelihood-ratio trick, REINFORCE, high variance |
| 03 | Actor-critic | Baseline → advantage → critic; bias-variance trade-off |
| 04 | PPO | Trust region via clipping; semi-on-policy; RLHF standard |
| 05 | SAC | Maximum entropy + off-policy + twin Q + reparameterisation |
| 06 | Comparison | Choice by sample cost, action space, parallelisability |
The toolbox is complete: from the MDP vocabulary over policy gradients to the two dominant deep-RL algorithms of the late 2010s and 2020s.
9. Further Reading
- Cross reference: self-learning-agents – RL as a building block of modern agent pipelines.
- Cross reference: categories/reinforcement-learning.
- Knowledge base:
masteringpytorch-secondedition.md– chapter on modern deep RL.
Sources
- Schulman, J. et al. (2017) – Proximal Policy Optimization Algorithms.
- Haarnoja, T. et al. (2018) – Soft Actor-Critic Algorithms and Applications.
- Ouyang, L. et al. (2022) – Training language models to follow instructions with human feedback (InstructGPT / PPO for RLHF).
- Rafailov, R. et al. (2023) – Direct Preference Optimization (DPO as a PPO alternative for LLMs).
- DeepSeek-AI (2024) – DeepSeekMath (GRPO as a critic-free PPO variant).