ENH: Add Canadian Invertebrates Dataset - #61
Merged
Conversation
scottclowe
changed the title
Created CanadianInvertebrates.py with essential classes and functions adapted from bioscan5m.py. - Removed unused image-related functions and parameters. - Copied necessary components from BIOSCAN5M, preserving argument names, method signatures, and docstrings where applicable. - Formatted code to follow PEP8, PEP257, and NumPy Style Guide.
- Adapted dataset loading pipeline to support the Canadian Invertebrates 1.5M metadata. - Used download_url from torchvision.datasets.utils to download CSV. - Added conversion of object-type columns to categorical dtype for label indexing. Note: Still need to add MD5 hash for file integrity check.
scottclowe
force-pushed
the
Adding-Canadian-Invertebrates-dataset
branch
2 times, most recently
from
August 5, 2025 22:52
2e9c7b4 to
1824414
Compare
- Added Canadian Invertebrates 1.5M dataset to the bioscan-dataset API documentation. - Updated `__init__.py` in bioscan_dataset to expose the new dataset class and loader.
- Add column aliases to ensure compatibility with the previous metadata structure
Ensured metadata and files are saved in the correct root directory Added Hugging Face download URL
CanadianInvertebrates.py -> canadian_invertebrates.py
load_CanadianInvertebrates_metadata -> load_canadian_invertebrates_metadata
We need to use the huggingface version since the uwaterloo fileshare will go down in a few months. There are some differences between the two, so the md5 hash is different.
scottclowe
force-pushed
the
Adding-Canadian-Invertebrates-dataset
branch
5 times, most recently
from
August 7, 2025 17:57
76d2e0b to
54c5eef
Compare
scottclowe
force-pushed
the
Adding-Canadian-Invertebrates-dataset
branch
3 times, most recently
from
August 8, 2025 02:16
c992330 to
a2cbf5c
Compare
scottclowe
marked this pull request as ready for review
August 8, 2025 16:08
scottclowe
changed the title
- Change introductory text so details about CanInv are not in the middle of description of BIOSCAN-1M/5M features. Highlight the differences between the datasets that are relevant to ML users. - Fix typo in sample code block CanadianInvertebrate -> CanadianInvertebrates - Re-order content so differences between BIOSCAN-1M and 5M are described immediately after BIOSCAN-1M. - Add missing word "paper" when referring to Canadian Invertebrates paper. - Don't have spaces in between links and commas; use Oxford commas. - Re-order bibtex list so CLIBD for alternative BIOSCAN-1M splits is described immediately after BIOSCAN-1M - Fix malformed bibtex (missing comma on penultimate line) - Improve bibtex: add DOI, change handle to be dataset name to match other paper handles, add month and day etc. from official bibtex, list all author names instead of truncating (let the .bst file choose whether to truncate instead).
There are no columns of interest in the dataset other than taxonomic labels, and these should be accessed using the target_type argument to ensure _index versions are loaded in dict mode.
…he baseline dataset
Since our cleaned version of the dataset doesn't have 1.5M samples and we offer no way of loading the original 1.5M samples.
scottclowe
force-pushed
the
Adding-Canadian-Invertebrates-dataset
branch
from
August 9, 2025 18:24
a2cbf5c to
afb012c
Compare
Member
|
Closes #60 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Created CanadianInvertebrate1-5m.py that includes essential classes and functions adapted from bioscan5m.py.