Skip to content

[Task/Feature/Issue] Add HF models to the data catalog #220

@jolson-ibm

Description

@jolson-ibm

Is there an existing issue for this?

  • I have searched the existing issues

Task/Feature/Issue Details

Following the established pattern of using the HF API to gather stats on HF datasets, create a job to collect model level stat information.

Only problem here is that we won't be able to collect stats on all the models, as there are 2,000,000 of them, and the get_models API call is rate limited, with no way to ask for the oldest models, we can only sort by last changed, or do a search.

Additional Context

No response

Metadata

Metadata

Assignees

Labels

data setsAcquiring new data sets to catalog or create.

Projects

Status

No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions