Skip to content

Linear Evaluation Result Incomplete Problem #320

@ywyz233

Description

@ywyz233

Hi, I found a small problem with the field main_metric_name in dataclass Evaluator in the file dinov3/eval/linear.py. When I tried to test multiple metrics for the same testing dataset, the main_metric_name for different metrics are same then only one metric's result can be saved to disk. I don't know if there is any specific purpose of designing in this way or is there any other method to save result for all metrics. If not, I think it would be better to add metric name into main_metric_name. Thanks for your time and for this great project and the link to the file and code snippet are attached below.
link: https://github.com/facebookresearch/dinov3/blob/main/dinov3/eval/linear.py#L292

@dataclass
class Evaluator:
    batch_size: int
    num_workers: int
    transform_config: TransformConfig
    dataset_str: str
    metric_type: ClassificationMetricType
    metrics_file_path: str
    training_num_classes: int
    save_results_func: Optional[Callable]

    def __post_init__(self):
        self.data_loader, self.class_mapping = make_eval_data_loader(
            test_dataset_str=self.dataset_str,
            batch_size=self.batch_size,
            num_workers=self.num_workers,
            transform_config=self.transform_config,
            metric_type=self.metric_type,
        )
        self.main_metric_name = f"{self.dataset_str}_accuracy"

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions