Pseudo Labeling

Overview

Pseudo labeling leverages unlabeled test data by treating high-confidence model predictions as ground truth. After a base model is trained with KFold CV, test samples where the model is very confident (near 0 or 1) are added to training with their predicted labels. The model is retrained and the result is kept only if OOF AUC improves.

Quick Start

import xgboost as xgb
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import roc_auc_score
import numpy as np

# 1. Train base model with KFold, collect OOF and test predictions
oof_preds, test_preds = train_kfold(X_train, y_train, X_test, n_splits=5)
base_auc = roc_auc_score(y_train, oof_preds)

# 2. Select high-confidence test samples
THRESHOLD = 0.995
mask_pos = test_preds > THRESHOLD
mask_neg = test_preds < (1 - THRESHOLD)
pseudo_mask = mask_pos | mask_neg

X_pseudo = X_test[pseudo_mask]
y_pseudo = (test_preds[pseudo_mask] > 0.5).astype(int)

# 3. Augment training set
X_aug = np.concatenate([X_train, X_pseudo])
y_aug = np.concatenate([y_train, y_pseudo])

# 4. Retrain and compare
oof_preds_aug, _ = train_kfold(X_aug, y_aug, X_test, n_splits=5)
new_auc = roc_auc_score(y_aug[:len(y_train)], oof_preds_aug[:len(y_train)])

# 5. Keep only if improved
if new_auc > base_auc:
    print(f"Pseudo labeling improved AUC: {base_auc:.5f} -> {new_auc:.5f}")

Workflow

1. Train base model with StratifiedKFold CV (typically 5-10 folds).
2. Collect OOF predictions on train and mean predictions on test.
3. Filter test samples by confidence threshold (e.g., >0.995 or <0.005).
4. Assign pseudo labels (round predictions to 0 or 1).
5. Concatenate pseudo-labeled samples with original training data.
6. Retrain with the same KFold CV setup on the augmented dataset.
7. Compare OOF AUC on the original training labels only. Keep if improved.

Key Decisions

Decision	Guidance
Confidence threshold	Start high (0.995). Lower thresholds add more data but risk label noise.
How many rounds	One round is usually enough. Iterating risks confirmation bias.
Validation	Always compare AUC on the original train labels, not on pseudo labels.
Class balance	Check that pseudo labels don't skew class ratio excessively.
Model choice	Works best with gradient boosting (XGBoost, LightGBM, CatBoost).
When to skip	If base model AUC is low (<0.80), pseudo labels are too noisy to help.

References

• Kaggle: "S6E3 Detail EDA + Baseline XGB" (playground-series-s6e3)
• Lee, D.H. "Pseudo-label: The simple and efficient semi-supervised learning method" (ICML 2013 Workshop)