Credit-risk data engineering

Joins, missingness, duplicates, and outliers

Treat data quality as a measurable property of the model population, not a one-time cleanup.

A visual overview of Treat data quality as a measurable property of the model population, not a one-time cleanup. See the guide for assumptions and implementation details.

Purpose

Describe practical controls for common credit-risk data defects and transformations.

Key terms

Hover over or focus a term for a quick definition, or open its glossary entry.

Data quality

Key definitions

Missingness is the absence of an expected value under a defined field and population.
An outlier is an observation unusual under a stated data or business rule, not automatically an error.

Formula

Missing rate = missing eligible records ÷ eligible records; duplicate-key rate = repeated analytical keys ÷ expected keys.

Worked example

If 80 of 2,000 eligible records lack a required income value, the missing rate is 4% for that field and population.

Detailed explanation

Start by profiling row counts, key uniqueness, missingness by period and segment, category frequencies, ranges, and date coverage. Distinguish structural missingness from collection failure and invalid values. Reconcile duplicate keys before deduplication; do not keep an arbitrary first row.

Keep raw fields alongside normalized values and record transformations. Outlier rules should be tied to source constraints or a modeling rationale, and their effect on risk segments should be measured. Monitor the same checks after deployment. Changes in missingness or field distributions can signal data pipeline problems even before a model performance metric moves.

Try a reproducible Python example

These examples use synthetic inputs and run locally in a browser worker. Edit the code, run it, and compare its output with the published example.

Runnable Python · local in your browser

Detect a point-in-time feature leak

A miniature data preparation check compares each feature's observation date with the prediction date.

Editable Python example

Output

Run the code to see the output.

Runtime loads only after you press Run. Code runs in a separate browser worker with a 10-second limit and a 20,000-character output cap. Use synthetic examples only; browser execution is not a secure environment for secrets or sensitive records.

Runnable Python · local in your browser

Engineer numeric credit-risk features

Use a training-only median, explicit missingness flags, a log transform, a payment-to-income ratio, and a delinquency indicator. Amounts are synthetic CAD; transforms are illustrative and should be validated for the intended population.

Editable Python example

from math import log1p

training = [
    {"income": 48000, "monthly_payment": 420, "utilization": 0.35, "dpd": 0},
    {"income": 72000, "monthly_payment": 900, "utilization": 0.72, "dpd": 12},
    {"income": 56000, "monthly_payment": 650, "utilization": 0.51, "dpd": 0},
]
scoring = [
    {"account": "A", "income": 60000, "monthly_payment": 750, "utilization": 0.60, "dpd": 0},
    {"account": "B", "income": None, "monthly_payment": 500, "utilization": None, "dpd": 35},
]

# Fit the imputation value on training data only; reuse it unchanged at scoring time.
income_values = sorted(row["income"] for row in training if row["income"] is not None)
training_income_median = income_values[len(income_values) // 2]
utilization_values = sorted(row["utilization"] for row in training if row["utilization"] is not None)
training_utilization_median = utilization_values[len(utilization_values) // 2]

for row in scoring:
    income_missing = row["income"] is None
    utilization_missing = row["utilization"] is None
    income = row["income"] if not income_missing else training_income_median
    utilization = row["utilization"] if not utilization_missing else training_utilization_median
    features = {
        "log_income": round(log1p(income), 4),
        "income_missing": int(income_missing),
        "payment_to_monthly_income": round(row["monthly_payment"] / (income / 12), 4),
        "utilization": utilization,
        "utilization_missing": int(utilization_missing),
        "dpd_30_plus": int(row["dpd"] >= 30),
    }
    print("{}: {}".format(row["account"], features))

Output

Run the code to see the output.

Assumptions

Rules preserve raw values and track corrections, imputation, exclusion, and reason codes.

Limitations

Imputation can hide operational or segment-specific data problems.
Winsorizing or removing extreme values can discard real high-risk behavior.

References

pandas user guide — pandas documentation
Guideline E-23: Model Risk Management — Office of the Superintendent of Financial Institutions Canada

Educational material only. Methods and examples are simplified; state and validate the assumptions required for your use case.