Judging a model through the TP, TN, FP, and FN values

Question

I am evaluating a model that predicts the existence or not existence of a "characteristic" (for example, "there is a dog in this image") using several datasets. The system outputs me for every dataset their TP, TN, FP, FN.

I would like a metric(s) to judge how good is the model doing its work but I realize that I can not plot for example just the TP because for example the first dataset has 20 instances where there is the characteristic (there is a dog) and the second dataset has say only 10. Even if the model is perfect the second dataset would have only 10 TP.

I am thinking of calculating accuracy, precision and recall for each dataset and for all datasets.

I have also run the model three times per each dataset, with small variations

I am investigating also precision-recall curves but it seem that these are for different threshold values and obviously I have only one set of precision , recall per dataset

Are there any good way to judge if a model is "good"? Due to my inexperience I can not come with a good judge criteria

At first I thought plotting the distribution of each (TP etc) for all datasets Then I thought of plotting a confusion matrix combining all datasets Any advice will be greatly appreciated

---

As a simple fictitious example I thought of

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score

# Example fictitious data
datasets = {
    'datasetA': {'TP': 150, 'TN': 200, 'FP': 50, 'FN': 100, 'no_GT': 34},
    'datasetB': {'TP': 180, 'TN': 220, 'FP': 40, 'FN': 81, 'no_GT': 20},
    'datasetC': {'TP': 160, 'TN': 240, 'FP': 70, 'FN': 110, 'no_GT': 30},
    'datasetD': {'TP': 190, 'TN': 250, 'FP': 60, 'FN': 90, 'no_GT': 42},
}

def calculate_metrics(TP, TN, FP, FN):
    accuracy = (TP + TN) / (TP + TN + FP + FN)
    precision = TP / (TP + FP) if (TP + FP) > 0 else 0
    recall = TP / (TP + FN) if (TP + FN) > 0 else 0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
    
    return {
        'Accuracy': accuracy,
        'Precision': precision,
        'Recall': recall,
        'F1 Score': f1
    }

# Aggregate counts
total_TP = sum(data['TP'] for data in datasets.values())
total_TN = sum(data['TN'] for data in datasets.values())
total_FP = sum(data['FP'] for data in datasets.values())
total_FN = sum(data['FN'] for data in datasets.values())

# Calculate overall metrics
overall_metrics = calculate_metrics(total_TP, total_TN, total_FP, total_FN)

# Calculate metrics for each dataset
metrics_df = pd.DataFrame({dataset: calculate_metrics(data['TP'], data['TN'], data['FP'], data['FN']) for dataset, data in datasets.items()})

# Add overall metrics
metrics_df['Overall'] = overall_metrics

print(metrics_df)

# Visualization
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
axes = axes.flatten()

for i, (dataset, data) in enumerate(datasets.items()):
    cm = confusion_matrix([1] * data['TP'] + [0] * data['TN'] + [1] * data['FN'] + [0] * data['FP'],
                          [1] * (data['TP'] + data['FP']) + [0] * (data['TN'] + data['FN']))
    sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', ax=axes[i])
    axes[i].set_title(f'Confusion Matrix - {dataset}')
    axes[i].set_xlabel('Predicted')
    axes[i].set_ylabel('True')

plt.tight_layout()
plt.show()

and I get

           datasetA  datasetB  datasetC  datasetD   Overall
Accuracy   0.700000  0.767754  0.689655  0.745763  0.725696
Precision  0.750000  0.818182  0.695652  0.760000  0.755556
Recall     0.600000  0.689655  0.592593  0.678571  0.640905
F1 Score   0.666667  0.748441  0.640000  0.716981  0.693524

and

Answer 1

Do not use any of accuracy, precision, recall, or the F1 score. They all suffer from the same issues, especially - but not only - for "unbalanced" data: Why is accuracy not the best measure for assessing classification models?

Instead, use probabilistic predictions and assess these using proper scoring rules.

Answer 2

What scoring rule you should use depends on the particular situation and how good the TP and TN are and how bad FP and FN are.

These vary hugely by situation. For instance, if the event is "the Challenger shuttle will explode" then even a very small false negative is really bad. OTOH, for some high risk surgeries, even a small false positive may be bad.

As so often happens, you need subject matter expertise to solve the problem.

Answer 3

I will provide an answer, coming from a different prespective. My background is from the diagnostic tests (think Covid, pregnancy, etc.), not ML. But maybe that different angle can give you some insights.
The performance of your classifiers (dog/no dog) is indeed fully specified by the 4 metrics you get (TP, TN, FP, FN). Every other statistic based on these 4 original metrics is just re-arranging them in various ways.
In the diagnostics domain, these 4 basic metrics are re-arranged into 4 different other metrics:

• Sensitivity ($\frac {TP} {(TP+FN)}$: how good is your model at finding positives) aka Recall, Hit Rate, TPR (true positive rate), with 1-Sensitivity=Miss rate or False negative rate (FNR).
• Specificity ($\frac {TN} {(TN+FP)}$: how good is your mode at finding negatives) aka TNR (true negative rate) wit 1-Specifiity=Fallout or False positive rate (FPR).
• Positive Predictive Value (PPV) ($\frac {TP} {(TP+FP)}$: if your model gives you a positive result, how likely is it to be truly positive) aka Precision (with 1-PPV=FDR false discovery rate).
• Negative Predictive Value (NPV) ($\frac {TN} {(TN+FN)}$: if your model gives you a negative result, how likely is it to be truly negative) aka ????(seems other fields do not have a synonim) with 1-NPV=FOR false omission rate).
  These 4 metrics are basically all you need (because you can regenerate the TP,TN,FP,FN from them). Any other combination is then redundant; for example Accuracy, Negative likelihood ratio (LR-), Positive likelihood ratio (LR+), F1 etc. are redundant (they do not add new information. Some however may prefer one over another).
  As you can see from the various acronyms, synonims, etc. different fields have different terms, which only adds to the confusion, w/o adding much value. But that is the current state. Now you could pick another set of 4 to summarize/represent TP,TN,FP,FN. I presonally like this particular set of 4 (and you can call them by whatever your domain terminology is) because of their nice symmetry: sensitivity/specificity are ratios to the column totals (of a confusion matrix), PPV/NPV are rations to the row totals, sensitivity/specificity do not depend on prevalence, while PPV/NPV do, Sensitivity/PPV tell you about "positives (subjects or results), specificity/NPV tell you about negatives. Sensitivity/Specificity tell you how good a job the classifier does at finding positives/negatives, while PPV/NPV tell you how much you can believe the positive/negative results.
  Now, all these metrics are just observed values. So you should compute CI's for all 4 of them. A Sensitivity or PPV value, w/o a CI is not of much use.

Now, if you have results where you "tweaked" the models differently, you can indeed create a ROC plot. And you should also add the CI bounds on the plot. The "best" version of your model is the one closest to the top-left corner.