Error Metrics

Model Error Metrics

These are the standard model development metrics. Referring back to the retinal images classification example, we may define the performance metrics criteria as follows: ¹

EXAMPLE

Based on the (a) error matrix measures, i. e., True Positive, False Negative, False Positive, True Negative, and (b) a threshold defined as the intersection of precision & sensitivity, the expected criteria are:

Overarching Criteria

Precision > 0.9, Sensitivity (True Positive Rate) > 0.9, Specificity (True Negative Rate) > 0.9, Youden's J Statistic > 0.9, and False Positive Rate < 0.1.

Disaggregated Criteria

For each class, and by the same overarching threshold value, ensure that each criterion metric value is within its defined range; as outlined above.

Model Card

This is for auditing purposes. A model's model card should summarise the artefacts of the released, in-use, in-production, model. A model card example:

• Model Cards for Model Reporting
• Model Cards for Model Reporting: Prototypes

---

1. N: # of negative cases, P: # of positive cases. | TN: True Negative, FP: False Positive, FN: False Negative, TP: True Positive | TNR = TN/N: True Negative Rate (specificity) | FPR = FP/N: False Positive Rate (fall out) | FNR = FN/P: False Negative Rate (miss rate) | TPR = TP/P: True Positive Rate (hit rate, sensitivity, recall) ↩