Validation and governance

Portfolio outcomes and model governance

Bring migration, vintage, recovery, EAD, and ECL outcomes into a controlled model lifecycle.

A visual overview of Bring migration, vintage, recovery, EAD, and ECL outcomes into a controlled model lifecycle. See the guide for assumptions and implementation details.

Purpose

Connect account outcomes and model monitoring to ownership, independent review, escalation, and change control.

Key terms

Hover over or focus a term for a quick definition, or open its glossary entry.

Key definitions

Vintage analysis follows cohorts originated or activated in a common period.
Model governance assigns responsibilities for inventory, review, approval, monitoring, and retirement.

Formula

Vintage cumulative default rate at age t = cohort defaults observed by age t ÷ cohort eligible units, with censoring treatment stated.

Worked example

Compare 12-month default outcomes for equally mature origination quarters rather than comparing a mature quarter with a newly booked one.

Detailed explanation

A broad monitoring view combines bad/default rates, rating transitions, roll rates, vintages, arrears, cures, prepayments, recoveries, LGD, CCF/EAD, and observed-versus-expected ECL. Slice by time, product, score band, and risk segment when definitions and sample sizes support it.

Governance gives the metrics an owner and an action. Maintain a model inventory, intended use, assumptions, limitations, data lineage, validation findings, approved changes, monitoring standards, and contingency plans. Independent review should challenge conceptual soundness and performance, while the owner responds to findings. Monitoring thresholds are institution-specific; no statistic alone declares a model compliant or fit for purpose.

Try a reproducible Python example

These examples use synthetic inputs and run locally in a browser worker. Edit the code, run it, and compare its output with the published example.

Runnable Python · local in your browser

Monitor observed-to-expected and Brier score

Compare a small set of matured binary outcomes against predicted PD values.

Editable Python example

Output

Run the code to see the output.

Runtime loads only after you press Run. Code runs in a separate browser worker with a 10-second limit and a 20,000-character output cap. Use synthetic examples only; browser execution is not a secure environment for secrets or sensitive records.

Runnable Python · local in your browser

Calculate ROC-AUC, Gini, KS, precision, and recall

Use synthetic out-of-sample scores and matured labels. Higher scores indicate higher risk; the example computes pairwise ROC-AUC and a threshold-specific operating point.

Editable Python example

scores = [0.92, 0.81, 0.65, 0.58, 0.42, 0.20]
observed_default = [1, 1, 1, 0, 0, 0]
positives = [score for score, y in zip(scores, observed_default) if y == 1]
negatives = [score for score, y in zip(scores, observed_default) if y == 0]
pairs = [(bad, good) for bad in positives for good in negatives]
auc = sum(1 if bad > good else 0.5 if bad == good else 0 for bad, good in pairs) / len(pairs)
gini = 2 * auc - 1
threshold = 0.60
tp = sum(y == 1 and score >= threshold for score, y in zip(scores, observed_default))
fp = sum(y == 0 and score >= threshold for score, y in zip(scores, observed_default))
fn = sum(y == 1 and score < threshold for score, y in zip(scores, observed_default))
precision = tp / (tp + fp) if tp + fp else 0
recall = tp / (tp + fn) if tp + fn else 0
ks = max(abs(sum(y == 1 and score >= cut for score, y in zip(scores, observed_default)) / len(positives) - sum(y == 0 and score >= cut for score, y in zip(scores, observed_default)) / len(negatives)) for cut in scores)
print(f"ROC-AUC={auc:.3f}; Gini={gini:.3f}; KS={ks:.3f}")
print(f"At threshold {threshold:.2f}: precision={precision:.3f}; recall={recall:.3f}")

Output

Run the code to see the output.

Assumptions

Cohorts have comparable definitions and monitoring metrics have named owners and actions.

Limitations

Different product mix or collection policy can confound cohort comparisons.
Monitoring does not replace independent validation or accounting controls.

References

Guideline E-23: Model Risk Management — Office of the Superintendent of Financial Institutions Canada
Guidance on credit risk and accounting for expected credit losses — Basel Committee on Banking Supervision

Educational material only. Methods and examples are simplified; state and validate the assumptions required for your use case.