Understanding Classification Models and Metrics

One crucial function of medical AI models is predicting whether a patient has a specific condition. By understanding classification metrics, you can better assess AI model predictions.

Let’s consider a matrix that represents the intersection of AI predictions and the actual presence or absence of the condition. This matrix is called a confusion matrix. Picture a graph where one axis specifies the actual or true diagnoses. The other axis specifies the classification system’s prediction.

Each axis can then be divided into positive and negative zones. If the model predicts the presence of a condition and the condition is a true diagnosis, the box where those two zones intersect can be labeled a True Positive or TP for short.

Conversely, if the model predicts a patient is negative for a condition and the patient, in fact, does not have the condition, this zone can be labeled as True Negative.

The other two zones are then made up of combinations of a negative prediction, but an actual positive case, or a positive prediction, but an actual or true negative case.

Now that we’ve covered the confusion matrix, let’s conceptualize the key metrics of sensitivity, specificity and positive predictive value.

First, sensitivity or recall measures how well the model spots the target clinical diagnosis. Sensitivity tells us out of all actual cases, how many did we catch. It compares the number of true positives to the number of false negatives.

Next, specificity focuses on patients without the diagnosis. It tells us out of all negative cases, how many did we correctly rule out? It compares the number of true negatives to false positives. Specificity is the proportion of actual negatives correctly identified as negative by the model.

Positive Predictive Value, or PPV, is sometimes called precision. PPV tells us how reliable a positive result is. It’s the proportion of predicted cases that are actually correct. In other words, it compares the true positives to the false positives.

Finally, we need a way to determine the performance of our model. One way is to compare the true positive rate against the false positive rate. The AUC or Area Under the ROC curve measures exactly this. The AUC is the trade off between true positive rate and false positive rate at different classification thresholds.

While the calculation for AUC is complicated, your AI classification program should provide the AUC for you. Just remember that the closer to 1, the better.

These metrics, sensitivity, specificity, positive predictive value, and AUC empower you to interpret AI classification models confidently.

How can classification metrics help you interpret an AI classification model’s performance? Five key metrics help you understand and better use AI models.

Let’s consider these classification metrics through an example. This example, respiratory failure prediction model analyzes historical EHR data to identify potential respiratory failure cases by recognizing critical symptoms, such as low oxygen levels, and rapid breathing as key markers.

As with all classification models, this one considers the actual diagnosis of a condition and the AI model’s predictions. We will use synthetic data related to respiratory failure for the various categories of prediction and actual diagnosis to illustrate the key classification metrics of sensitivity, specificity, flag rate, and positive predictive value.

First, sensitivity or recall measures how well the model spots the target clinical diagnosis. In our case, sensitivity tells how well the model identifies respiratory failure cases. Sensitivity tells us, out of all actual respiratory failure cases, how many did we catch?

In our model, with a sensitivity of 0.68, we’re identifying 68% of true respiratory failure patients. This is critical for early intervention to improve outcomes.

Next, specificity focuses on patients without respiratory failure. It is the model’s ability to designate an individual who does not have a condition as negative. Specificity tells us, out of all nonrespiratory failure cases, how many did we correctly rule out?

With a specificity of 0.97, our model avoids flagging 97% of healthy patients. This reduces unnecessary treatment.

A win for efficiency and patient comfort.

Flag rate is the number of positive predictions divided by the total number of cases. This determines how many cases the model predicted as positive. The total number of positives divided by the total number of patients is our flag rate.

In this case, the flag rate is 0.21, or 21%. A flag rate of 21% means the model flags 21% of all patients as potentially having respiratory failure, helping clinicians prioritize follow-up while minimizing unnecessary interventions.

Positive Predictive Value, or PPV, tells us how reliable a positive result is. It’s the proportion of predicted respiratory failure cases that are actually correct.

With a precision of 0.88, 88% of our respiratory failure predictions are spot on. This increases trust in the model when acting on its predictions.

Finally, we need a way to determine the performance of our model. One way is to compare the true positive rate against the false positive rate. The AUC, or Area Under the ROC Curve, measures exactly this.

An AUC of 0.93 means our model is excellent at distinguishing respiratory failure from nonrespiratory failure.

For AUC calculations, remember that the closer to one, the better.

In summary, while the program or software that runs your classification model will most likely provide these numbers to you, understanding how they are calculated will help you confidently analyze a classification model’s results.