Skip to content

[Bug] Score discrepancy and data contamination in Bioavailability_Ma leaderboard #394

Description

@Taekya

Describe the bug
There appears to be a score discrepancy and potential data contamination on the ADMET Leaderboard for the Bioavailability_Ma task. It seems that the submission values/scores for Pgp_Broccatelli were accidentally applied to Bioavailability_Ma, as the scores for several overlapping models across both leaderboards are identical.

To Reproduce
Steps to reproduce the behavior:

  1. Go to the TDC ADMET Leaderboard.
  2. Compare the model scores/rankings for the Bioavailability_Ma task.
  3. Compare them with the scores for the Pgp_Broccatelli task.
  4. See the identical scores across multiple overlapping models.

Expected behavior
The Bioavailability_Ma leaderboard should display the correct, independent evaluation scores for the submitted models based on the Bioavailability dataset, rather than duplicating the Pgp_Broccatelli results.

Screenshots

Image

Environment:

  • OS: N/A (Web platform issue)
  • TDC version: Leaderboard Website

Additional context
First of all, thank you for maintaining the TDC Leaderboard; it has been incredibly useful for my research.

I noticed this issue while analyzing the benchmarks. Since the scores of several models perfectly match between the two completely different tasks (Bioavailability_Ma and Pgp_Broccatelli), it strongly suggests a data mapping or logging error during the leaderboard update.

Best regards,
Taekyeong An

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions