swe-rm-verifier
Improve SWE agent RL via execution-free reward models optimized for three metrics beyond TTS: discriminative ability (AUC) and calibration (ECE). Shows TTS alone insufficient—models must distinguish c
SKILL.mdskills/skillxiv-v0.0.2-claude-opus-4.6/swe-rm-verifier/SKILL.md
GitHubOverview
SWE-RM shows that reward model evaluation requires three complementary metrics.
Core Technique
Three-Metric Evaluation:
# Not just TTS (top-1 ranking)
tts_score = rank_best_solution_first(predictions)
# Also discriminative ability
auc_score = compute_auc(correct_vs_incorrect)
# And calibration
ece_score = expected_calibration_error(confidence, accuracy)
When to Use
Use when: SWE agent training, RL reward modeling, importance of calibration.
References
- Discriminative ability vs ranking metrics
- Calibration error measurement
- Multi-objective reward model design