What are the dataset limits?

[dcaribou]

• What are the dataset limits?
• Can we add more seasons to it?
• Can we add more clubs, players, leagues...?
• Can we add more attributes?

[dcaribou]

Limits on history

Although in theory it would be possible to extract the data for all historical seasons (as long as it is accessible in Transfermakt), the scraper scope is limited to a few recent seasons (see #12). The two main reasons are

• Scraping a full new season is a significant effort, particularly the data validation step. Historical data tends to be less accurate, and it's harder to keep the dataset consistent as we add in older seasons
• Most of the value of the data tends to be on the most recent seasons anyways, so it's also more "economic" to focus on those

Also, although less importantly, data.world does not support datasets larger than 100MB in its free tier, and with the current set of seasons that limit is already exceeded.

Limits on scope

The scraping scope (and hence the dataset scope) is defined by the list of competitions that is provided in the data/competitions.json file, which currently contains around 40 of them. From there, the scraper recurses into leagues, clubs, players, etc. With the current scope, the scraping step takes close to 5h to run to collect / update the data.

There is no particular reason why the competitions set couldn't be a bit more or a bit less than the current 40, but with the acquiring step taking already close to 5h there is no room for a lot more.

Limits on attributes

Refer to #177

So is that it?

I am currently happy with the dataset coverage as it is and I do not have plans to add meaningful extensions to its scope and history myself, however I am not against the idea of accepting incremental additions to it made by contributors. If you feel like doing it, get in touch and I can help you with the setup.