Click-Through-Rate (CTR) Prediction

CTR prediction is the textbook example of a heavily imbalanced classification problem. Class weights, calibration and feature hashing for high-cardinality ad data – and why accuracy is misleading here and a calibration plot reveals more than any AUC.

1. The Problem: When 1 % Performance Decides Over Millions

In display advertising real click-through rates often lie between 0.5 % and 3 %. A "model" that always predicts "no click" reaches over 97 % accuracy – and is still useless. Three consequences follow:

1. Accuracy is unsuitable as a metric. PR-AUC, log loss and calibration are the more honest measures.
2. Calibrated probabilities are mandatory, not optional. Bidding systems multiply P(click) × value(click). A probability off by a factor of 2 directly burns budget.
3. High-cardinality features (user-ID, ad-ID, domain) require special handling – OneHot encoding would create millions of columns.

2. Class Imbalance: What Actually Helps

❌ Naive solution: more data, same loss

More training data alone does not solve the problem. With 99 % negatives the model keeps learning primarily the negative pattern.

✅ Three effective levers

3. Feature Hashing for High-Cardinality Categories

User-ID, ad-ID, publisher domain – each of these fields has potentially millions of distinct values. OneHotEncoder is not practical. Two established alternatives:

• FeatureHasher / HashingVectorizer: hash each category onto a fixed vector index. Collisions are unavoidable but harmless in practice if the dimensionality (n_features) is chosen large enough (typically 2¹⁸–2²⁰).
• Target / frequency encoding: replaces the category with the mean click rate of that category. Caveat: prone to target leakage, so always with out-of-fold computation.

4. End-to-End: Imbalanced CTR with Calibration

The block below trains a LogisticRegression with class weighting and compares it with the calibrated variant. For diagnosis a reliability diagram is drawn: it plots the predicted probability against the actually observed frequency. A perfectly calibrated model lies on the diagonal.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.metrics import average_precision_score, log_loss
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(1)
n = 10000
X = rng.normal(size=(n, 6))
# CTR ~ 2 % – heavily imbalanced
logit = X[:, 0] + 0.5 * X[:, 1] - 4.0
p_true = 1 / (1 + np.exp(-logit))
y = (rng.random(n) < p_true).astype(int)
print(f"Positive rate: {y.mean():.3f}")

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)

# Variant A: LogReg with class_weight="balanced" (distorted probabilities)
base = LogisticRegression(class_weight="balanced", max_iter=1000).fit(X_tr, y_tr)
p_base = base.predict_proba(X_te)[:, 1]

# Variant B: same LogReg, but calibrated
cal = CalibratedClassifierCV(
    LogisticRegression(class_weight="balanced", max_iter=1000),
    method="isotonic", cv=5,
).fit(X_tr, y_tr)
p_cal = cal.predict_proba(X_te)[:, 1]

print(f"PR-AUC  base={average_precision_score(y_te, p_base):.3f}  cal={average_precision_score(y_te, p_cal):.3f}")
print(f"LogLoss base={log_loss(y_te, p_base):.4f}  cal={log_loss(y_te, p_cal):.4f}")

# Reliability diagram
frac_base, mean_base = calibration_curve(y_te, p_base, n_bins=10, strategy="quantile")
frac_cal,  mean_cal  = calibration_curve(y_te, p_cal,  n_bins=10, strategy="quantile")

fig, ax = plt.subplots(figsize=(6, 6))
ax.plot([0, 1], [0, 1], "k--", label="Perfectly calibrated")
ax.plot(mean_base, frac_base, "o-", label="LogReg balanced (uncalibrated)")
ax.plot(mean_cal,  frac_cal,  "s-", label="+ Isotonic calibration")
ax.set_xlabel("Predicted probability")
ax.set_ylabel("Observed frequency")
ax.set_title("Reliability diagram (CTR model)")
ax.legend()
ax.grid(alpha=0.3)
plt.tight_layout()
plt.show()

The uncalibrated variant with class_weight="balanced" often delivers the same ranking quality (PR-AUC) as the calibrated one, but deviates clearly from the diagonal in the reliability diagram. For pure sorting tasks (top-N ranking) that is enough. As soon as probabilities enter a downstream decision as numbers – real-time bidding, expected conversion, budget allocation – they must be calibrated.

5. Offline vs. Online Evaluation

Offline metrics (PR-AUC, log loss, reliability) are necessary but not sufficient. Three extra steps are standard in production CTR systems:

• Time-based split instead of random split: models on ad marketplaces experience permanent drift; a chronological split is more honest.
• Lift curve: how much better is my model than the baseline CTR on the top-x % of the ranked impressions?
• Online A/B test: the final verdict; see article 04.

6. Wrap-up Quiz

7. Further Reading

• Next article: A/B testing & uplift modelling (article 04 of this series).
• Knowledge base: machinelearningwithpytorchandscikit-learn.md – section on probability calibration.
• Niculescu-Mizil & Caruana (2005), Predicting Good Probabilities With Supervised Learning.

Sources

• Machine Learning with PyTorch and Scikit-Learn (Sebastian Raschka et al.) – chapter on model evaluation.
• Display Advertising with Real-Time Bidding (Wang, Yuan, Wang) – background on calibration requirements.