[Story] Review the organization contribution types

[tschaffter]

What product(s) is this story for?

OpenChallenges

As a user, I want

No response

Description

Important

This initial goal of this ticket was to clarify one type of organization contribution (sponsor) to address a feedback received from DREAM. The goal of this ticket is now extend to the review of all the org contribution types.

We received this feedback from DREAM:

Also, the use of the word 'sponsor' on the new site is confusing. In the grant world the 'sponsor' is the organization paying for the project. We don't pay for any challenges. We receive funding from sponsors for the challenges. I am just not clear how 'sponsor' is being used in the context of that site although maybe it means platform being used/organization hosting?

What about using "benchmarking community" for organizations like DREAM, Kaggle, MICCAI, etc.?

Acceptance criteria

• A clear definition is defined for each contribution role
• A contribution role is defined for DREAM and similar organization
• The OC Data have been updated
• If needed, the backend and frontend have been updated

Tasks

No response

Anything else?

No response

Have you linked this story to a GitHub Project?

• I have linked this story to a GitHub Project and set its metadata.

[gaiaandreoletti]

I agree using "benchmarking community" instead of sponsors or possible even "crowdsourcing community"

[gaiaandreoletti]

added it ina separate tab and ready for discussion

[tschaffter]

@gaiaandreoletti Can you please add an update to this ticket? We usually do it like this:

## Update yyyy-mm-dd

Description of where this ticket stands, whether there are blockers, expected completion date.

[gaiaandreoletti]

Updated 2023-11-20

added it in a separate tab and ready for discussion with the main group. I would consider the ticket closed.

[tschaffter]

@gaiaandreoletti See the section "Acceptance criteria" of this ticket that lists the requirements that must be met before we can close this ticket.

• A clear definition is defined for each contribution role => Are these contribution roles defined somewhere?
• A contribution role is defined for DREAM and similar organizations => What are the organizations that have been assigned the new contribution role (should depend on the approved definition of Benchmarking community)?
• If needed, the backend and frontend have been updated => Neither Rong nor I have done this. The above steps must be completed first before Rong or I can start this step.

Cc @vpchung

[gaiaandreoletti]

@tschaffter I was hoping we would have discussed this during the OC meeting. A tab now called "Benchmarking Community" in the OpenChallenges Data lists all Benchmarking communities.
These include:

• DREAM
• BioCreative
• CAFA
• CAGI
• CAMDA
• CAMEO
• CAMI
• CAPRI
• CASP
• GAW
• iDash
• MICCAI
• precisionFDA
• sbv IMPROVER
• Kaggle
• CACHE
• OpenEBench
• CodaLab
• Codabench
• OpenML
• PapersWithCode
• Grand Challenge
• Health Data Hub
• PhysioNet
• HeroX
• BioHackathons

Definition:
Benchmarking Community: Benchmarking communities in biological data involves evaluating and comparing their performance, characteristics, and attributes to establish standards and identify patterns. This practice provides a framework for understanding their structure, composition, and dynamics. Communities engaged in benchmarking often seek funding from sponsors to support their activities. The financial support obtained from sponsors plays a crucial role in the success and sustainability of benchmarking efforts.

[tschaffter]

Looking good in general.

Benchmarking communities in biological data involves evaluating and comparing their performance [...]

What do you mean by "the performance of a benchmarking community"? Maybe mention tools and (AI/ML) models whose performance is evaluated.

Another elements that could be added to the description is that a benchmarking community may focus on very specific tasks (e.g. CAGI, CAMI) or benchmark a broad range of biomedical tools (e.g. DREAM that benchmarks various tasks like gene network reconstruction, breast cancer detection from imaging data, etc.).

Please add the updated description to a Google doc so that we can find it easily when new contributions are added to the DB as well as for enable Jake and other to contribute to the description until it's approved.

[tschaffter]

Added to Sprint 23.12

[tschaffter]

Added to Sprint 24.1

[tschaffter]

@gaiaandreoletti See "Important" note added to the first comment of this ticket.