A/B Testing & Uplift Modelling

Statistical tests answer "Is the difference real?", uplift models answer "On whom does the campaign actually work?". Two related but fundamentally different questions – with two different method families and a from-scratch two-model example including a Qini curve.

1. Two Different Questions, Two Different Tools

Anyone who only predicts conversions optimises the wrong quantity: a conversion model sends coupons to people who would have bought anyway. The clean separation looks like this:

Both methods share the same data type: a randomised sample with treatment indicator T ∈ {0,1} and outcome Y ∈ {0,1}.

2. A/B Tests: What Is Actually Significant

❌ Pitfall: Peeking

Repeatedly evaluating during the runtime inflates the false-positive rate. With 10 interim checks at α=5 % the probability of a false hit quickly rises above 30 %.

✅ Clean practice

• Determine sample size upfront from a power calculation.
• Evaluate only at the pre-specified endpoint.
• Bootstrap confidence intervals as a robust alternative to t-tests (no normality assumption).

3. Uplift Modelling: The Four Segments

Every individual can conceptually be assigned to one of four segments:

The core problem: for each individual we observe only one of the two worlds (Y(T=0) or Y(T=1)), never both. Uplift models estimate the individual treatment effect from randomised data.

4. Two-Model Approach: From Scratch with Qini Curve

The simplest uplift approach trains two separate models (one on the treatment group, one on the control group) and takes the difference of the probabilities as the uplift estimate. For evaluation the Qini curve is used: it plots the cumulative incremental conversion against the share of the ranked population.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(2)
n = 6000
X = rng.normal(size=(n, 4))
T = rng.integers(0, 2, n)                                      # randomised treatment

# Heterogeneous treatment effect: effective only when X[:,1] > 0
p_ctrl = 1 / (1 + np.exp(-(0.5 * X[:, 0] - 1.0)))             # base rate ~ 25 %
uplift_true = 0.25 * (X[:, 1] > 0)                            # true CATE
p = np.clip(p_ctrl + T * uplift_true, 0, 1)
y = (rng.random(n) < p).astype(int)

X_tr, X_te, T_tr, T_te, y_tr, y_te = train_test_split(
    X, T, y, test_size=0.4, random_state=0,
)

# Two-model approach: separate models for T=0 and T=1
m_t = LogisticRegression(max_iter=1000).fit(X_tr[T_tr == 1], y_tr[T_tr == 1])
m_c = LogisticRegression(max_iter=1000).fit(X_tr[T_tr == 0], y_tr[T_tr == 0])
uplift_hat = m_t.predict_proba(X_te)[:, 1] - m_c.predict_proba(X_te)[:, 1]

print(f"Estimated mean uplift: {uplift_hat.mean():+.3f}")
print(f"True mean uplift in test: {(0.25 * (X_te[:, 1] > 0)).mean():+.3f}")

# Qini curve: sort population by uplift_hat (descending),
# plot cumulative incremental conversions.
order = np.argsort(-uplift_hat)
y_sorted = y_te[order]
T_sorted = T_te[order]

# Cumulative conversions in treatment and control, scaled to the same size
cum_t = np.cumsum(y_sorted * (T_sorted == 1))
cum_c = np.cumsum(y_sorted * (T_sorted == 0))
n_t = np.cumsum(T_sorted == 1).clip(min=1)
n_c = np.cumsum(T_sorted == 0).clip(min=1)
# Increment = T rate − C rate, scaled to the population
incremental = cum_t - cum_c * (n_t / n_c)
share = np.arange(1, len(y_te) + 1) / len(y_te)

# Random baseline (straight line to the endpoint)
random_line = share * incremental[-1]

fig, ax = plt.subplots(figsize=(7, 5))
ax.plot(share, incremental, label="Two-model uplift")
ax.plot(share, random_line, "k--", label="Random targeting")
ax.set_xlabel("Share of addressed population (sorted by uplift score)")
ax.set_ylabel("Cumulative incremental conversions")
ax.set_title("Qini curve")
ax.legend()
ax.grid(alpha=0.3)
plt.tight_layout()
plt.show()

# Qini score: area between the model curve and the random line
qini_score = np.trapz(incremental - random_line, share)
print(f"Qini score (higher = better): {qini_score:.3f}")

For a useful uplift model the Qini curve lies above the random line. The larger the area between the curves (Qini score), the better the model separates Persuadables from the other segments. The bend at the end is normal: once you also address the bottom 30–40 % of the population, the incremental effect drops – that is where Sure Things, Lost Causes and Do-Not-Disturb live.

5. Discussion: When Two-Model and When Something Better?

Two-model is the simplest but not the most efficient method:

• Pro: any sklearn-compatible estimator can be used; transparent and debuggable.
• Con: doubly estimated noise – both models have their own errors that don't cancel in the difference.
• Better alternatives for production: class transformation (one model, transformed label), uplift trees (splits maximise uplift instead of purity), X-learner / R-learner (two-stage meta-learners).

Practical heuristic: start your first uplift project with two-model, validate the Qini curve against random. Only invest in more complex estimators once ROI justifies it.

6. Wrap-up Quiz

7. Further Reading

• Next article: Sentiment & content insights with small LLMs (article 05 of this series).
• Libraries: scikit-uplift, causalml, econml (Microsoft).
• Künzel et al. (2019), Metalearners for estimating heterogeneous treatment effects using machine learning.

Sources

• Machine Learning with PyTorch and Scikit-Learn (Sebastian Raschka et al.) – chapter on model evaluation.
• Radcliffe & Surry (2011), Real-World Uplift Modelling with Significance-Based Uplift Trees.