Soft Actor-Critic (SAC)

SAC combines maximum-entropy RL with off-policy learning, a replay buffer and twin Q-critics. The result is one of the most robust methods for continuous control – sample-efficient, largely tuning-free and with native exploration without an $\epsilon$ schedule. We develop the maximum-entropy objective, show the data flow as a Mermaid diagram, implement a runnable SAC toy in numpy and add a realistic PyTorch skeleton.

1. The Problem: PPO Is Sample-Inefficient

PPO is on-policy. Every batch is thrown away after a few epochs, every new training step needs fresh rollouts. For a robot that takes a second per trajectory this is fatal – hundreds of thousands of episodes turn into weeks of physical runtime.

Off-policy methods solve the problem in principle: experiences from previous policies remain usable. Classical approaches have their own issues:

SAC (Haarnoja et al., 2018) is not incremental – it combines three ideas, each fixing a classical pain point.

2. Maximum-Entropy RL: Reward + Stochasticity Are Both Rewarded

The standard RL objective only maximises the reward. SAC adds an entropy bonus:

$$J(\pi) = \sum_{t} \mathbb{E}_{(s_t, a_t) \sim \pi}\left[r_t + \alpha \cdot \mathcal{H}(\pi(\cdot \mid s_t))\right]$$

with the entropy $\mathcal{H}(\pi(\cdot \mid s)) = -\mathbb{E}_{a \sim \pi}[\log \pi(a \mid s)]$ and temperature $\alpha \ge 0$.

In words: prefer policies that have high rewards and high entropy at the same time. The consequences are deep:

• Native exploration: the policy stays stochastic on purpose. No external $\epsilon$-greedy schedule.
• Multi-modal policies: when two actions are equally good, SAC keeps both modes alive.
• Robustness: a broad policy is less susceptible to small reward shifts.

The crucial point: $\alpha$ can be tuned automatically so that a target entropy is maintained (see section 6).

3. Off-Policy With a Replay Buffer

SAC continuously collects experiences and stores them in a replay buffer (typically 1M transitions). Updates draw random minibatches from the buffer – every interaction with the world can be used many times.

This makes SAC dramatically more sample-efficient than PPO. On typical MuJoCo benchmarks SAC reaches comparable performance with 5–10× fewer environment interactions.

4. Twin-Q Critics: Against Overestimation

Q-learning has a well-known issue: the max operator over noisy Q estimates produces systematic overestimation. In DQN this is often tolerable; in continuous control diverging Q-values ruin training.

SAC uses two independent Q networks $Q_{\phi_1}, Q_{\phi_2}$ (idea from TD3, Fujimoto et al. 2018). The TD target uses the minimum:

$$y = r + \gamma \left(\min_{i=1,2} Q_{\bar{\phi}_i}(s', a') - \alpha \log \pi(a' \mid s')\right), \quad a' \sim \pi(\cdot \mid s')$$

The pessimistic min systematically dampens overestimation. $\bar{\phi}$ denotes target networks – slow copies of the actual Q networks updated via Polyak averaging.

5. Reparameterisation Trick: Gradients Through Sampling

SAC's policy is a tanh-squashed Gaussian: the network outputs $\mu(s), \sigma(s)$, the action is built via

$$a = \tanh\left(\mu(s) + \sigma(s) \odot \epsilon\right), \quad \epsilon \sim \mathcal{N}(0, I)$$

The tanh squash keeps the action in $[-1, 1]$. The decisive effect: the gradient of $a$ w.r.t. the network weights is well defined because the sampling step is not in the computation graph – $\epsilon$ is an external random source. This makes the policy loss

$$L_\pi = \mathbb{E}s\left[\alpha \log \pi(a \mid s) - Q(s, a)\right], \quad a \sim \pi\theta(\cdot \mid s)$$

directly minimisable with backprop – without the high-variance policy gradient from REINFORCE/PPO.

6. Runnable: SAC Toy on a 1D Regulator World

A full SAC with neural networks does not fit in a Pyodide snippet. But we can show the core mechanics – replay buffer, twin Q, maximum-entropy update – on a tiny world: the agent sits on a 1D line and is asked to stay as close to target = 0 as possible. The action is a continuous push in $[-1, 1]$.

import numpy as np
from collections import deque

rng = np.random.default_rng(0)
GAMMA = 0.95
ALPHA = 0.2          # entropy temperature
LR    = 0.05
TAU   = 0.01         # Polyak averaging for target networks
BUFFER_SIZE = 5000
BATCH = 64
STEPS = 4000

# Simplified "networks": linear Q(s, a) = w_s * s + w_a * a + b. Two of them.
def init_q():  return np.zeros(3)            # [w_s, w_a, b]
def q(params, s, a):  return params[0]*s + params[1]*a + params[2]

Q1, Q2 = init_q(), init_q()
Q1_t, Q2_t = Q1.copy(), Q2.copy()

# Policy: Gaussian with learned mu(s)=k*s, log_std constant.
mu_k = 0.0
log_std = np.log(0.5)

def sample_action(s):
    mu = mu_k * s
    std = np.exp(log_std)
    eps = rng.standard_normal()
    a_pre = mu + std * eps
    a = np.tanh(a_pre)
    # log_prob after tanh squash (classical SAC correction).
    log_prob = -0.5*((a_pre - mu)/std)**2 - log_std - 0.5*np.log(2*np.pi) - np.log(1 - a**2 + 1e-6)
    return a, log_prob, a_pre

def env_step(s, a):
    s_next = s + a + rng.normal(0, 0.1)
    s_next = np.clip(s_next, -3, 3)
    r = -s_next**2                        # closer to 0 = better
    done = False
    return s_next, r, done

buffer = deque(maxlen=BUFFER_SIZE)
s = rng.uniform(-2, 2)
returns_window = deque(maxlen=200)
ep_return = 0.0

for step in range(STEPS):
    a, _, _ = sample_action(s)
    s_next, r, done = env_step(s, a)
    buffer.append((s, a, r, s_next))
    ep_return += r
    s = s_next
    if step % 50 == 49:
        returns_window.append(ep_return); ep_return = 0.0; s = rng.uniform(-2, 2)

    if len(buffer) < BATCH:
        continue

    idx = rng.integers(0, len(buffer), BATCH)
    batch = [buffer[i] for i in idx]
    S  = np.array([b[0] for b in batch])
    A  = np.array([b[1] for b in batch])
    R  = np.array([b[2] for b in batch])
    Sn = np.array([b[3] for b in batch])

    # Target: min(Q1_t, Q2_t)(s', a') - alpha * log_pi(a'|s')
    A_next, logp_next = [], []
    for sn in Sn:
        a_n, lp, _ = sample_action(sn)
        A_next.append(a_n); logp_next.append(lp)
    A_next = np.array(A_next); logp_next = np.array(logp_next)
    q1_t = Q1_t[0]*Sn + Q1_t[1]*A_next + Q1_t[2]
    q2_t = Q2_t[0]*Sn + Q2_t[1]*A_next + Q2_t[2]
    target = R + GAMMA * (np.minimum(q1_t, q2_t) - ALPHA * logp_next)

    # Critic update (MSE gradient on linear Q).
    for Q in (Q1, Q2):
        pred = Q[0]*S + Q[1]*A + Q[2]
        err  = pred - target
        Q[0] -= LR * (err * S).mean()
        Q[1] -= LR * (err * A).mean()
        Q[2] -= LR * err.mean()

    # Actor update via reparameterisation: maximise Q - alpha * log_pi.
    # A 1-sample gradient for mu_k.
    a_s, lp_s, _ = sample_action(S[0])
    q_min = min(q(Q1, S[0], a_s), q(Q2, S[0], a_s))
    # Gradient w.r.t. mu_k: d(Q - alpha*log_pi)/d(mu_k)  --  heuristic:
    # more push toward -s (closer to 0) is better → mu_k should become negative.
    grad_mu = (q_min - ALPHA * lp_s) * S[0]
    mu_k += LR * 0.01 * grad_mu

    # Polyak averaging of the targets.
    Q1_t = (1 - TAU) * Q1_t + TAU * Q1
    Q2_t = (1 - TAU) * Q2_t + TAU * Q2

print(f"Mean return (last 200 'episodes'): {np.mean(returns_window):+.2f}")
print(f"Learned policy slope mu_k        : {mu_k:+.3f}  (expected < 0)")
print(f"Q1 coefficients                  : w_s={Q1[0]:+.2f}  w_a={Q1[1]:+.2f}")

The world is intentionally tiny, but all the SAC building blocks are there:

• Replay buffer (deque) stores old transitions.
• Twin Q (Q1, Q2) plus target networks (Q1_t, Q2_t) with Polyak averaging.
• Maximum-entropy target (min(Q1_t, Q2_t) - ALPHA * logp_next).
• Reparameterised sampling with tanh squash and corrected log-prob.

Expected result: the slope mu_k becomes negative – the learned policy pushes back towards the target s = 0.

7. PyTorch Skeleton (Concept – Not Runnable in the Browser)

Here is SAC's policy update in real PyTorch:

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.distributions import Normal

class SquashedGaussianPolicy(nn.Module):
    def __init__(self, obs_dim, act_dim, hidden=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU(),
        )
        self.mu_head      = nn.Linear(hidden, act_dim)
        self.log_std_head = nn.Linear(hidden, act_dim)

    def forward(self, obs):
        h = self.net(obs)
        mu = self.mu_head(h)
        log_std = self.log_std_head(h).clamp(-20, 2)
        std = log_std.exp()
        dist = Normal(mu, std)
        a_pre = dist.rsample()                  # reparameterisation!
        a = torch.tanh(a_pre)
        # Tanh correction for log_prob (numerically stable).
        log_prob = dist.log_prob(a_pre).sum(-1) - torch.log(1 - a.pow(2) + 1e-6).sum(-1)
        return a, log_prob


def sac_update(policy, q1, q2, q1_t, q2_t, opt_pi, opt_q, batch, alpha, gamma=0.99, tau=0.005):
    s, a, r, s_next, done = batch

    # Critic update.
    with torch.no_grad():
        a_next, logp_next = policy(s_next)
        q_target = torch.min(q1_t(s_next, a_next), q2_t(s_next, a_next)) - alpha * logp_next
        target = r + gamma * (1 - done) * q_target

    q1_loss = F.mse_loss(q1(s, a), target)
    q2_loss = F.mse_loss(q2(s, a), target)
    opt_q.zero_grad(); (q1_loss + q2_loss).backward(); opt_q.step()

    # Actor update.
    a_new, logp = policy(s)
    q_new = torch.min(q1(s, a_new), q2(s, a_new))
    pi_loss = (alpha * logp - q_new).mean()
    opt_pi.zero_grad(); pi_loss.backward(); opt_pi.step()

    # Polyak averaging.
    with torch.no_grad():
        for p, p_t in zip(q1.parameters(), q1_t.parameters()):
            p_t.data.mul_(1 - tau).add_(tau * p.data)
        for p, p_t in zip(q2.parameters(), q2_t.parameters()):
            p_t.data.mul_(1 - tau).add_(tau * p.data)

The rsample() method marks the reparameterisation sampling. Without it the gradient through the sample would be blocked.

8. Automatic Alpha Tuning

A hand-picked $\alpha$ is fragile: too high → the policy stays too stochastic, too low → exploration collapses. Haarnoja et al. (2018) solve this by learning $\alpha$ itself – with a target entropy $\mathcal{H}_{\text{target}} \approx -\dim(A)$:

$$L_\alpha = -\alpha \cdot (\log \pi(a \mid s) + \mathcal{H}_{\text{target}})$$

In practice this is the default variant. It makes SAC largely tuning-free – one of the main reasons for its popularity.

9. Discussion: Why SAC Dominates Robotics

This exact combination makes SAC the default for sim-to-real robotics (Boston Dynamics, OpenAI Hand, many MuJoCo benchmarks).

10. Wrap-up Quiz

11. Further Reading

• Next article: PPO vs. SAC – when to use which? (article 06 of this series).
• Cross reference: self-learning-agents/02-rl-fuer-code-agenten.
• Knowledge base: masteringpytorch-secondedition.md – chapter on continuous control.

Sources

• Haarnoja, T. et al. (2018) – Soft Actor-Critic: Off-Policy Maximum Entropy Deep RL with a Stochastic Actor, ICML.
• Haarnoja, T. et al. (2018) – Soft Actor-Critic Algorithms and Applications, arXiv:1812.05905 (auto-alpha variant).
• Fujimoto, S. et al. (2018) – Addressing Function Approximation Error in Actor-Critic Methods (TD3, twin-Q idea).