Skip to content

Precision and Recall are calculated in opposite ways? #160

Description

@khyeongkyun

In the line:479-481 at pangaea.engine.evaluator, Precision and Recall are assigned in opposite ways.

I think confusion_matrix are set with pred class in rows and target class in columns. Therefore, .sum(dim=0) is for Recall and .sum(dim=1) is for Precision.

Here reproduce the issue with re-use the part of the current version pangaea.engine.evaluator.SegEvaluator

import torch

num_classes = 2
target = torch.tensor([
    [0, 0, 0, 0],
    [0, 1, 1, 0],
    [0, 1, 1, 0],
    [0, 0, 0, 0]
]).flatten()
pred = torch.tensor([
    [0, 0, 1, 0],  
    [0, 1, 1, 0],  
    [0, 1, 1, 1],  
    [0, 0, 1, 0]   
]).flatten()
device = 'cpu'

# line: 430 ---------------------------------------
confusion_matrix = torch.zeros(
    (num_classes, num_classes), device=device
)
#  ---------------------------------------

# line: 454  ------------------------------
count = torch.bincount(
    (pred * num_classes + target), minlength=num_classes ** 2
)
confusion_matrix += count.view(num_classes, num_classes)
#  ---------------------------------------

# line: 474 ------------------------------
# Calculate IoU for each class
intersection = torch.diag(confusion_matrix)
union = confusion_matrix.sum(dim=1) + confusion_matrix.sum(dim=0) - intersection
iou = (intersection / (union + 1e-6)) * 100

# Calculate precision and recall for each class
precision = intersection / (confusion_matrix.sum(dim=0) + 1e-6) * 100
recall = intersection / (confusion_matrix.sum(dim=1) + 1e-6) * 100
#  ---------------------------------------

fix_precision = intersection / (confusion_matrix.sum(dim=1) + 1e-6) * 100
fix_recall = intersection / (confusion_matrix.sum(dim=0) + 1e-6) * 100

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions