Methods · validation

Model validation metrics

Discrimination asks whether risk is ranked meaningfully. Calibration asks whether forecasts match outcomes for a defined population and horizon. Report both with matured, out-of-sample evidence.

Discrimination and calibration

discrimination

ROC-AUC

Formula / definition

Area under the receiver operating characteristic curve: TPR against FPR over score cut-offs.

Interpretation

Measures ranking: the probability a randomly selected default receives a worse risk rank than a randomly selected non-default, under the stated score direction.

Population

Matured labeled observations with both outcome classes; use out-of-sample scores.

Horizon

The same outcome horizon and observation date for all records.

Assumptions

Scores are not used to fit and evaluate on the same observations.
Default definition and event maturity are consistent.

Useful segmentation

Time, product, origination vintage, and material risk groups.

Limitations

Does not assess probability calibration or economic value.
Can be high when operational cut-offs still perform poorly.

Threshold caveat

Set investigation limits using portfolio history, uncertainty, model purpose, and sample size; no AUC boundary applies universally.

discrimination

Gini coefficient

Formula / definition

Gini = 2 × ROC-AUC − 1, using a convention that maps random ranking to zero.

Interpretation

A linear rescaling of ROC-AUC; confirm whether a source uses a different sign or scale convention.

Population

The same matured labeled sample used for ROC-AUC.

Horizon

The same fixed outcome horizon used for ROC-AUC.

Assumptions

ROC-AUC uses consistent positive-class and score orientation.

Useful segmentation

Product, time, score band, and validation sample.

Limitations

It is not a separate calibration or stability test.

Threshold caveat

Interpret changes with sampling uncertainty and portfolio mix; do not apply universal pass/fail bands.

discrimination

Kolmogorov–Smirnov (KS)

Formula / definition

KS = maxₜ |F₁(t) − F₀(t)|, the largest gap between cumulative score distributions for defaults and non-defaults.

Interpretation

Shows the greatest separation at one threshold; report score orientation and the threshold where the maximum occurs.

Population

Matured defaults and non-defaults in a defined validation sample.

Horizon

A common binary-outcome horizon.

Assumptions

Class labels are mature and score distributions use a consistent orientation.

Useful segmentation

Compare by period, product, and relevant risk segment.

Limitations

Selecting the maximum creates optimism on small samples.
It does not measure calibration.

Threshold caveat

Set portfolio-specific bands and uncertainty procedures; a generic KS rule is not a universal model standard.

discrimination

Precision (positive predictive value)

Formula / definition

Precision = TP / (TP + FP) at a declared decision threshold.

Interpretation

Among accounts flagged positive at that threshold, the share that are positive; state what a positive alert means.

Population

Predictions and matured outcomes for the same eligible population.

Horizon

The same decision and outcome horizon, with the threshold stated.

Assumptions

Positive-class definition, threshold, and confusion-matrix counts are disclosed.

Useful segmentation

Product, channel, score band, and decision threshold.

Limitations

Changes with outcome prevalence and threshold.
Undefined when no positives are predicted.

Threshold caveat

Choose operating points from risk appetite, customer impact, capacity, and costs, not a universal precision threshold.

discrimination

Recall (sensitivity)

Formula / definition

Recall = TP / (TP + FN) at a declared decision threshold.

Interpretation

Among actual positives, the share captured by the rule at that threshold.

Population

Matured positive cases and all eligible observations in the validation population.

Horizon

The same event horizon and threshold as the associated decision.

Assumptions

All outcomes have matured and false negatives are identified consistently.

Useful segmentation

Time, product, risk band, and threshold.

Limitations

Does not capture false-positive cost; can be raised by flagging more accounts.

Threshold caveat

Set acceptable trade-offs from business impact and capacity; thresholds must be calibrated for the portfolio.

discrimination

Precision–recall AUC

Formula / definition

Area under the precision–recall curve as the decision threshold varies.

Interpretation

Summarizes positive-class precision and recall; report the positive prevalence as a reference because the baseline depends on it.

Population

Out-of-sample scores and matured outcomes with a clearly defined positive class.

Horizon

One fixed outcome horizon.

Assumptions

Positive class, ranking direction, and interpolation convention are documented.

Useful segmentation

Compare products and time periods with prevalence shown.

Limitations

Sensitive to prevalence and calculation convention; not directly comparable across differently sampled populations.

Threshold caveat

Use portfolio-specific review criteria and a prevalence-aware baseline; there is no universal PR-AUC cutoff.

calibration

Observed-to-expected (O/E) defaults

Formula / definition

O/E = observed defaults / Σᵢ predicted PDᵢ for an aligned sample.

Interpretation

Around one indicates aggregate observed counts near predicted counts in that sample; show numerator, denominator, and uncertainty.

Population

Eligible observations with matured default labels and matching PD forecasts.

Horizon

Forecast horizon equals label horizon; observations are not counted repeatedly without a weighting rule.

Assumptions

Forecasts are out of sample.
Exposure, censoring, weights, and definition of default are aligned.

Useful segmentation

Time, product, rating band, geography, and material portfolio groups.

Limitations

Aggregate agreement can hide offsetting segment errors.
Small default counts create wide uncertainty.

Threshold caveat

Use confidence intervals, materiality, and portfolio context; a fixed O/E band is not a universal trigger.

calibration

Calibration curve

Formula / definition

For bins b, compare mean predicted PDᵦ with observed event rateᵦ; include bin counts and uncertainty.

Interpretation

A curve near the identity line suggests average calibration within those groups, but results depend on grouping and maturity.

Population

Matured out-of-sample probability forecasts from the intended population.

Horizon

A declared PD horizon matching observed outcomes.

Assumptions

Bins and missing-value handling are disclosed.
Sufficient observations/events support each bin.

Useful segmentation

Time, product, predicted-risk band, and segment.

Limitations

Binning can conceal local errors; sparse bins are noisy.

Threshold caveat

Choose bin-level review criteria with uncertainty and sample size; no one calibration curve boundary fits every portfolio.

calibration

Calibration intercept and slope

Formula / definition

Fit logit(P(Y=1)) = α + β × logit(predicted PD); ideal reference values are α=0, β=1 under this setup.

Interpretation

Intercept reflects average level shift and slope reflects over/under-extremity; define any recalibration on separate data.

Population

Matured out-of-sample forecasts and outcomes for one intended population.

Horizon

Forecast and outcome horizons match.

Assumptions

Probabilities are finite and strictly between zero and one or clipped under a disclosed rule.
Fit is identifiable with enough events.

Useful segmentation

Time, product, score range, and risk segments.

Limitations

Estimates can be unstable with few events or narrow risk ranges.

Threshold caveat

Use uncertainty intervals, sample size, and portfolio impact analysis; coefficient tolerances should reflect the intended use.

calibration

Brier score

Formula / definition

Mean squared probability error: (1/n) Σᵢ(pᵢ − yᵢ)² for binary y.

Interpretation

Lower is better for the evaluated sample, but the score combines calibration and discrimination and depends on prevalence.

Population

Matured binary outcomes and paired forecast probabilities.

Horizon

Forecast horizon equals binary-label horizon.

Assumptions

Probabilities and outcomes are aligned at the observation level.
Sample weighting is documented.

Useful segmentation

Compare time, product, risk band, and a stated reference forecast.

Limitations

A single average can mask segment errors; not a direct calibration-only statistic.

Threshold caveat

Compare against appropriate baselines and portfolio costs; do not use a universal Brier threshold.

calibration

Log loss

Formula / definition

−(1/n) Σᵢ[yᵢ ln(pᵢ) + (1−yᵢ) ln(1−pᵢ)], with documented probability clipping.

Interpretation

Penalizes confident wrong probabilities more sharply; lower indicates better probabilistic prediction on the evaluated sample.

Population

Matured outcomes paired with out-of-sample probabilities.

Horizon

Common binary forecast horizon.

Assumptions

Probability clipping and sample weights are reported.
The outcome labels are complete and aligned.

Useful segmentation

Time, product, score band, and model challenger.

Limitations

Can be dominated by a few extreme wrong forecasts; changes with prevalence.

Threshold caveat

Use a relevant comparator and portfolio-specific materiality rather than a generic pass/fail value.

Runnable validation examples

Run the transparent Python calculations on synthetic scores and outcomes. Change the inputs to explore how discrimination and calibration measures respond.

Runnable Python · local in your browser

Calculate ROC-AUC, Gini, KS, precision, and recall

Use synthetic out-of-sample scores and matured labels. Higher scores indicate higher risk; the example computes pairwise ROC-AUC and a threshold-specific operating point.

Editable Python example

scores = [0.92, 0.81, 0.65, 0.58, 0.42, 0.20]
observed_default = [1, 1, 1, 0, 0, 0]
positives = [score for score, y in zip(scores, observed_default) if y == 1]
negatives = [score for score, y in zip(scores, observed_default) if y == 0]
pairs = [(bad, good) for bad in positives for good in negatives]
auc = sum(1 if bad > good else 0.5 if bad == good else 0 for bad, good in pairs) / len(pairs)
gini = 2 * auc - 1
threshold = 0.60
tp = sum(y == 1 and score >= threshold for score, y in zip(scores, observed_default))
fp = sum(y == 0 and score >= threshold for score, y in zip(scores, observed_default))
fn = sum(y == 1 and score < threshold for score, y in zip(scores, observed_default))
precision = tp / (tp + fp) if tp + fp else 0
recall = tp / (tp + fn) if tp + fn else 0
ks = max(abs(sum(y == 1 and score >= cut for score, y in zip(scores, observed_default)) / len(positives) - sum(y == 0 and score >= cut for score, y in zip(scores, observed_default)) / len(negatives)) for cut in scores)
print(f"ROC-AUC={auc:.3f}; Gini={gini:.3f}; KS={ks:.3f}")
print(f"At threshold {threshold:.2f}: precision={precision:.3f}; recall={recall:.3f}")

Output

Run the code to see the output.

Runtime loads only after you press Run. Code runs in a separate browser worker with a 10-second limit and a 20,000-character output cap. Use synthetic examples only; browser execution is not a secure environment for secrets or sensitive records.

Runnable Python · local in your browser

Compare expected defaults, Brier score, and log loss

Review several probability metrics on the same synthetic, matured population. Small samples are noisy; this output is not an alert rule.

Editable Python example

Output

Run the code to see the output.

A high ranking statistic cannot repair a mismatched target, biased sample, unstable feature pipeline, or uncalibrated probability. Use independent review and portfolio-specific standards.