Improve Flickr scraper compatibility and CI

[glenn-jocher]

Summary

• avoid the deprecated flickrapi.walk() path by using JSON photos.search, fixing Python 3.9+ getchildren() failures
• allow Flickr credentials through --key/--secret or FLICKR_API_KEY/FLICKR_API_SECRET while keeping source constants as a fallback
• add --page, clearer Flickr API error reporting, HTTP download status checks, README updates, tests, and a uv-based CI workflow

Issues

Fixes #4
Fixes #5
Fixes #6
Fixes #8
Fixes #9
Helps #25

Tests

• python -m compileall -q .
• uv run --with-requirements requirements.txt --with pytest pytest -q
• git diff --check
• uv run --with-requirements requirements.txt python flickr_scraper.py --search 'honeybees on flowers' --n 1 (expected credential error path)\n\nNote: a full-tree Ruff check still reports a pre-existing docstring issue in utils/clean_images.py; touched files pass the repo Ruff settings.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

🚀 This PR makes the Flickr scraper more reliable and easier to use by improving credential handling, switching to a more compatible Flickr API flow, adding pagination support, and introducing automated tests and CI.

📊 Key Changes

• 🔐 Improved API credential handling

  • Added support for Flickr credentials via FLICKR_API_KEY and FLICKR_API_SECRET environment variables.
  • Added --key and --secret CLI options as overrides.
  • Added clear error messages when credentials are missing.

• 🔄 Reworked Flickr search logic

  • Replaced the older flickrapi.walk() approach with direct photos.search JSON responses.
  • Added a helper to build image URLs, with fallback logic when original image URLs are unavailable.
  • Added better handling for Flickr API failures and request exceptions.

• 📄 Added pagination support

  • New --page argument lets users continue scraping from later Flickr result pages.
  • URL de-duplication now happens within a run to reduce repeated downloads.

• 📥 Cleaner download behavior

  • Download requests now explicitly check for HTTP errors before saving files.
  • Existing filename sanitization behavior is covered by tests.

• ✅ Added automated testing and CI

  • New GitHub Actions CI workflow compiles the code and runs pytest on Python 3.11.
  • Added unit tests for URL building, credential resolution, API behavior, error handling, and downloads.

• 📝 README improvements

  • Updated minimum Python version from 3.7 to 3.8.
  • Documented environment variable setup, CLI credential options, pagination, duplicate-result behavior, and compatibility notes.
  • Corrected progress output examples to use 1/10 through 10/10.

• 📦 Dependency update

  • Added requests to requirements.txt.

🎯 Purpose & Impact

• 👍 Easier for users to get started

  • Environment variable and CLI-based credential setup is safer and more flexible than editing the script directly.

• 🛠️ Better compatibility and stability

  • Moving away from walk() helps avoid known compatibility issues on newer Python versions.

• 🎯 More predictable scraping for dataset collection

  • Pagination support makes it easier to gather more images beyond the first batch and reduce repeated results.

• 🚨 Clearer failure reporting

  • Users now get actionable error messages for invalid credentials, Flickr-side errors, and request failures.

• 🔍 Higher code quality and maintainability

  • Automated tests and CI help catch regressions early and make future updates safer.

• 📚 Improved documentation

  • The updated README better explains real-world usage, especially for repeated scraping and troubleshooting.

[UltralyticsAssistant]

👋 Hello @glenn-jocher, thank you for submitting a ultralytics/flickr_scraper 🚀 PR! This is an automated message to help with review readiness—an engineer will assist you shortly. Please review the checklist below to help ensure a smooth merge ✨

• ✅ Define a Purpose: Clearly explain the purpose of your fix or feature in your PR description, and link to any relevant issues. Ensure your commit messages are clear, concise, and adhere to the project's conventions.
• ✅ Synchronize with Source: Confirm your PR is synchronized with the ultralytics/flickr_scraper main branch. If it's behind, update it by clicking the 'Update branch' button or by running git pull and git merge main locally.
• ✅ Ensure CI Checks Pass: Verify all Ultralytics Continuous Integration (CI) checks are passing. If any checks fail, please address the issues.
• ✅ Update Documentation: Update the relevant documentation for any new or modified features.
• ✅ Add Tests: If applicable, include or update tests to cover your changes, and confirm that all tests are passing.
• ✅ Sign the CLA: Please ensure you have signed our Contributor License Agreement if this is your first Ultralytics PR by writing "I have read the CLA Document and I sign the CLA" in a new message.
• ✅ Minimize Changes: Limit your changes to the minimum necessary for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." — Bruce Lee

For more guidance, please refer to our Contributing Guide. Don't hesitate to leave a comment if you have any questions. Thank you for contributing to Ultralytics! 🚀

[UltralyticsAssistant]

🔍 PR Review

_{Made with ❤️ by Ultralytics Actions}

The JSON search refactor, credential resolution, and test coverage look good overall. The main issue is pagination: changing per_page between requests breaks page continuity and can miss results. Also, download failures currently bypass the new CLI error handling and can abort a run with an uncaught exception.

💬 Posted 2 inline comments

[UltralyticsAssistant]

⚠️ HIGH: per_page needs to stay constant across paginated requests. Because it currently shrinks as len(urls) grows, page=2 is no longer the continuation of page=1 whenever a page yields duplicates, so later requests can overlap earlier results and skip large parts of the search result set.

Suggested change:

Suggested change

	per_page=min(500, n - len(urls)), # 1-500
	per_page=min(500, n), # 1-500

[UltralyticsAssistant]

💡 MEDIUM: download_uri() now raises on HTTP failures, but this call isn't handled here. With --download, a single 404/timeout will escape as an uncaught requests exception because the CLI only catches RuntimeError/ValueError, so one bad image can terminate the whole run with a traceback instead of surfacing a clean scraper error or skipping that photo.

[UltralyticsAssistant]

Merged! 🎉 Huge thanks to @glenn-jocher for this thoughtful upgrade to the Flickr scraper.

As Thomas Edison said, “There’s a way to do it better—find it.” This PR reflects that perfectly: better credential handling, a more robust Flickr API flow, pagination support, clearer error reporting, and the addition of tests and CI all make the scraper more reliable, maintainable, and user-friendly.

Really appreciate the care put into both the implementation and documentation here—this is a meaningful improvement for everyone using the project.