Skip to content

Improve first-time observation load for large datasets #455

@JWCook

Description

@JWCook

For a large number of observations (say, >10K records), a number of problems can come up during the first-time data load. Some general tasks, roughly in order of complexity:

  • Ensure app is responsive before initial observation load finishes
  • Make incremental batches of observations available for tagging as they become available
  • Add option to only download n most recent observations (or from a given start date)
    • May break some assumptions elsewhere that all user observations are in local db
  • Fix progress bar (somewhat complex issue with parallel batch loader reporting to main UI thread)
  • Bootstrap observation data load with GBIF research-grade export and/or iNat CSV export
    • Show a notification when attempting to download >10K records and recommend starting with a data export

User with >100K records to stress test with: https://www.inaturalist.org/observations?user_id=martin_der_prinz
ref: https://forum.inaturalist.org/t/naturtag-organize-your-photo-collection-with-inat-metadata/33959/22

Will split this into separate issues after more research/testing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    dataChanges related to image metadata or app dataenhancementNew feature or requestperformanceImprovements to application performance

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions