Validation and governance

Discrimination: ROC-AUC, Gini, and KS

Measure ranking ability while keeping class balance, horizon, and threshold context visible.

A visual overview of Measure ranking ability while keeping class balance, horizon, and threshold context visible. See the guide for assumptions and implementation details.

Purpose

Define common discrimination measures and explain what they do not tell a reviewer.

Key terms

Hover over or focus a term for a quick definition, or open its glossary entry.

Key definitions

ROC-AUC measures ranking across thresholds for a binary outcome.
Gini is commonly expressed as 2×AUC−1 under a stated convention; KS compares cumulative score distributions.

Formula

Gini = 2×ROC-AUC−1; KS = maxₜ |F_bad(score≤t) − F_good(score≤t)| under the chosen score orientation.

Worked example

An AUC of 0.70 corresponds to Gini 0.40 under the common transformation; it does not mean 70% of individual decisions are correct.

Detailed explanation

Discrimination metrics ask whether higher-risk cases tend to rank ahead of lower-risk cases. ROC-AUC integrates true-positive and false-positive rates across thresholds. Gini is a linear transformation commonly used in credit score reporting. KS measures the maximum separation between cumulative score distributions, with direction depending on score orientation.

Report event and non-event counts, confidence intervals where useful, horizon, population, and segmentation. Compare over time using consistent definitions and note that AUC is insensitive to some forms of prevalence change. Pair ranking metrics with calibration, threshold performance, stability, and operational outcomes. No one metric or cutoff universally validates a credit model.

Try a reproducible Python example

These examples use synthetic inputs and run locally in a browser worker. Edit the code, run it, and compare its output with the published example.

Runnable Python · local in your browser

Monitor observed-to-expected and Brier score

Compare a small set of matured binary outcomes against predicted PD values.

Editable Python example

Output

Run the code to see the output.

Runtime loads only after you press Run. Code runs in a separate browser worker with a 10-second limit and a 20,000-character output cap. Use synthetic examples only; browser execution is not a secure environment for secrets or sensitive records.

Runnable Python · local in your browser

Calculate ROC-AUC, Gini, KS, precision, and recall

Use synthetic out-of-sample scores and matured labels. Higher scores indicate higher risk; the example computes pairwise ROC-AUC and a threshold-specific operating point.

Editable Python example

scores = [0.92, 0.81, 0.65, 0.58, 0.42, 0.20]
observed_default = [1, 1, 1, 0, 0, 0]
positives = [score for score, y in zip(scores, observed_default) if y == 1]
negatives = [score for score, y in zip(scores, observed_default) if y == 0]
pairs = [(bad, good) for bad in positives for good in negatives]
auc = sum(1 if bad > good else 0.5 if bad == good else 0 for bad, good in pairs) / len(pairs)
gini = 2 * auc - 1
threshold = 0.60
tp = sum(y == 1 and score >= threshold for score, y in zip(scores, observed_default))
fp = sum(y == 0 and score >= threshold for score, y in zip(scores, observed_default))
fn = sum(y == 1 and score < threshold for score, y in zip(scores, observed_default))
precision = tp / (tp + fp) if tp + fp else 0
recall = tp / (tp + fn) if tp + fn else 0
ks = max(abs(sum(y == 1 and score >= cut for score, y in zip(scores, observed_default)) / len(positives) - sum(y == 0 and score >= cut for score, y in zip(scores, observed_default)) / len(negatives)) for cut in scores)
print(f"ROC-AUC={auc:.3f}; Gini={gini:.3f}; KS={ks:.3f}")
print(f"At threshold {threshold:.2f}: precision={precision:.3f}; recall={recall:.3f}")

Output

Run the code to see the output.

Assumptions

The validation sample has defined positive class, observation unit, target horizon, and sufficient events.

Limitations

Discrimination does not establish calibration or acceptable customer outcomes.
AUC can look stable while default rates or segment performance change.

References

Metrics and scoring: quantifying the quality of predictions — scikit-learn documentation
Guidelines on PD estimation, LGD estimation and treatment of defaulted assets — European Banking Authority

Educational material only. Methods and examples are simplified; state and validate the assumptions required for your use case.