Will this post perform? A compact scikit-learn pipeline that predicts engagement from post metadata (time of day, hashtags, length, media type) – with logistic regression and gradient boosting as fair baselines, evaluated via ROC-AUC and PR-AUC.
1. Framing the Problem as Classification
"Will this post perform?" is the most common question in content marketing. Before training a model, it has to be translated into a well-defined learning problem:
| Element | Concrete form |
|---|---|
| Input | Post metadata at publication time |
| Output | high_engagement ∈ {0,1} (top 20 % of the engagement distribution) |
| Estimator | Calibrated probability P(high_engagement = 1 | features) |
| Success metric | ROC-AUC + PR-AUC + calibration |
A binary threshold (top 20 %) is more robust than a regression on raw like counts: the latter is dominated by viral outliers.
2. Feature Engineering from Post Metadata
Four feature families cover most of the explainable signal:
- Time features: hour, weekday, holiday flag, "prime time" indicator.
- Content features: text length, number of hashtags, number of mentions, emoji ratio, language.
- Media features: has_image, has_video, has_link, image count.
- Historical features: rolling-mean engagement of the account (last 30 days).
Quick check: Which feature is most likely a 'data leak' for the engagement model?
❌ Imperative: Scattered Feature Construction
# Anti-pattern: features are built ad hoc in notebooks,
# train and test logic drift apart.
df["hour"] = df["timestamp"].dt.hour
df["prime_time"] = ((df["hour"] >= 18) & (df["hour"] <= 22)).astype(int)
df_train["text_len"] = df_train["text"].str.len()
df_test["text_len"] = df_test["text"].str.len()
# ... 50 more lines, each with potential inconsistency
✅ Functional: Pipeline as Composed Transformations
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler
# A single, reusable definition – no drift between train and test.
pipeline = Pipeline([
("preprocess", ColumnTransformer([
("num", StandardScaler(), ["hour", "text_len", "n_hashtags"]),
# Categorical features would get OneHot/target encoding here.
])),
("clf", LogisticRegression(max_iter=1000)),
])
3. End-to-End Pipeline: Data → Model → Evaluation
The runnable block below trains both models (LogReg, GradientBoosting) on a synthetic dataset and evaluates them with both AUC variants plus calibration.
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, average_precision_score, brier_score_loss
rng = np.random.default_rng(42)
n = 3000
# Synthetic post dataset with realistic drivers.
hour = rng.integers(0, 24, n)
has_image = rng.integers(0, 2, n)
has_video = rng.integers(0, 2, n)
text_len = rng.integers(20, 280, n)
n_hashtags = rng.integers(0, 8, n)
account_avg = rng.normal(50, 20, n).clip(5, 200)
prime_time = ((hour >= 18) & (hour <= 22)).astype(int)
logit = (
-1.5
+ 0.9 * has_image
+ 0.6 * has_video
+ 0.7 * prime_time
+ 0.01 * (account_avg - 50)
- 0.002 * (text_len - 150)
+ 0.05 * n_hashtags
+ rng.normal(0, 0.5, n)
)
p = 1 / (1 + np.exp(-logit))
y = (rng.random(n) < p).astype(int)
X = pd.DataFrame({
"hour": hour, "has_image": has_image, "has_video": has_video,
"text_len": text_len, "n_hashtags": n_hashtags, "account_avg": account_avg,
"prime_time": prime_time,
})
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
logreg = Pipeline([("sc", StandardScaler()), ("clf", LogisticRegression(max_iter=1000))])
gbm = GradientBoostingClassifier(random_state=0, n_estimators=100, max_depth=3)
def evaluate(name, model):
model.fit(X_tr, y_tr)
p_hat = model.predict_proba(X_te)[:, 1]
print(f"{name:8s} ROC-AUC={roc_auc_score(y_te, p_hat):.3f} "
f"PR-AUC={average_precision_score(y_te, p_hat):.3f} "
f"Brier={brier_score_loss(y_te, p_hat):.3f}")
print(f"Positive rate in test set: {y_te.mean():.3f}")
evaluate("LogReg", logreg)
evaluate("GBM", gbm)
Three metrics tell three different stories:
- ROC-AUC measures the global ranking quality (higher = better, 0.5 = random).
- PR-AUC is more sensitive to the positive class – more relevant when the class is rare.
- Brier score measures calibration (lower = better): are the probabilities realistic?
Quick check: What does the AUC value (ROC-AUC) measure?
4. Feature Importance and Explainability
LogReg yields directly interpretable coefficients (after standardisation). GBM needs feature_importances_ or, better, SHAP values. The snippet below shows both side by side:
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
n = 3000
X = pd.DataFrame({
"hour": rng.integers(0, 24, n),
"has_image": rng.integers(0, 2, n),
"has_video": rng.integers(0, 2, n),
"text_len": rng.integers(20, 280, n),
"n_hashtags": rng.integers(0, 8, n),
})
y = ((0.9 * X["has_image"] + 0.6 * X["has_video"] + rng.normal(0, 0.5, n)) > 0.3).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
logreg = Pipeline([("sc", StandardScaler()), ("clf", LogisticRegression(max_iter=1000))]).fit(X_tr, y_tr)
gbm = GradientBoostingClassifier(random_state=0).fit(X_tr, y_tr)
print("LogReg coefficients (standardised):")
for name, coef in zip(X.columns, logreg.named_steps["clf"].coef_[0]):
print(f" {name:12s} {coef:+.3f}")
print("\nGBM feature importances:")
for name, imp in zip(X.columns, gbm.feature_importances_):
print(f" {name:12s} {imp:.3f}")
5. Discussion: Trade-offs Between Model Classes
| Aspect | Logistic Regression | Gradient Boosting |
|---|---|---|
| Performance on typical marketing data | Solid baseline | Often 1–3 percentage points AUC better |
| Training time | Seconds | Minutes to hours |
| Explainability | Direct coefficients | SHAP, permutation importance |
| Robustness against drift | High | Medium |
| Out-of-the-box calibration | Good | Rather poor → CalibratedClassifierCV |
In practice the honest question is rarely "Which model is better?" but "Is the performance gain worth the extra complexity?". For an engagement-prediction use case with 50,000 posts per day and a business leverage in the single-digit percentage range, the answer is almost always: a calibrated LogReg wins once you factor in maintenance.
6. Wrap-up Quiz
Which statement about LogReg vs. Gradient Boosting makes the most sense in the marketing context?
Why is a `Pipeline` (sklearn) better than manual feature engineering in a notebook?
In which scenario is PR-AUC more informative than ROC-AUC?
7. Further Reading
- Next article: Click-Through-Rate (CTR) Prediction (article 03 of this series).
- Knowledge base:
machinelearningwithpytorchandscikit-learn.md– chapter on pipelines and model selection. - Cross reference: recommendation-systems.
Sources
- Machine Learning with PyTorch and Scikit-Learn (Sebastian Raschka et al.) – chapters 6 + 7.
- Hands-On Machine Learning (Aurélien Géron) – chapter on ensemble methods.