Improve Flickr scraper compatibility and CI (#45)

Co-authored-by: UltralyticsAssistant <web@ultralytics.com>

Author: glenn-jocher

.github/workflows/ci.yml

@@ -0,0 +1,38 @@
+# Ultralytics 🚀 AGPL-3.0 License - https://ultralytics.com/license
+
+name: CI
+
+on:
+  pull_request:
+    branches: [main]
+  push:
+    branches: [main]
+  schedule:
+    - cron: "0 8 * * *"
+
+permissions:
+  contents: read
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@v5
+
+      - name: Install dependencies
+        run: uv pip install --system -r requirements.txt pytest ultralytics
+
+      - name: Compile Python files
+        run: python -m compileall -q .
+
+      - name: Run tests
+        run: pytest -q

README.md

@@ -17,7 +17,7 @@ The Flickr Scraper is a [Python](https://www.python.org/) tool designed to help
 
 ## 🔧 Requirements
 
-Ensure you have Python 3.7 or later installed. The necessary dependencies can be installed using [pip](https://pip.pypa.io/en/stable/):
+Ensure you have Python 3.8 or later installed. The necessary dependencies can be installed using [pip](https://pip.pypa.io/en/stable/):
 
 ```bash
 pip install -U -r requirements.txt
@@ -47,7 +47,14 @@ pip install -U -r requirements.txt
 Before you begin scraping images:
 
 1.  **Get a Flickr API Key**: Obtain your unique API key and secret by applying [here](https://www.flickr.com/services/apps/create/apply).
-2.  **Configure API Credentials**: Insert your API key and secret into the `flickr_scraper.py` script:
+2.  **Configure API Credentials**: Set your Flickr API key and secret as environment variables:
+
+```bash
+export FLICKR_API_KEY="YOUR_API_KEY"
+export FLICKR_API_SECRET="YOUR_API_SECRET"
+```
+
+You can also pass credentials directly with `--key` and `--secret`, or insert them into `flickr_scraper.py` if you prefer a local-only script:
 
 ```python
 # flickr_scraper.py
@@ -69,9 +76,9 @@ python3 flickr_scraper.py --search 'honeybees on flowers' --n 10 --download
 You should see output indicating the download progress:
 
 ```plaintext
-0/10 https://live.staticflickr.com/21/38596887_40df118fd9_o.jpg
+1/10 https://live.staticflickr.com/21/38596887_40df118fd9_o.jpg
 ...
-9/10 https://live.staticflickr.com/1770/43276172331_e779b8c161_o.jpg
+10/10 https://live.staticflickr.com/1770/43276172331_e779b8c161_o.jpg
 Done. (4.1s)
 All images saved to /Users/glennjocher/PycharmProjects/flickr_scraper/images/honeybees_on_flowers/
 ```
@@ -80,6 +87,22 @@ The downloaded images will be available in the specified folder (e.g., `images/h
 
 <img src="https://user-images.githubusercontent.com/26833433/75074332-4792c600-54b0-11ea-8c98-22acf58ba8e7.jpg" width="600" alt="Example scraped image of a honeybee on a flower">
 
+## 🔁 Downloading More Results
+
+Flickr search order is controlled by the Flickr API and may change over time. If you run the same command again with the same search term, Flickr can return many of the same results. To continue from a later result page, use `--page`:
+
+```bash
+python3 flickr_scraper.py --search 'honeybees on flowers' --n 50 --download --page 2
+```
+
+The script de-duplicates URLs within each run, but it does not keep a persistent database of previously downloaded Flickr photo IDs. For repeat dataset collection, keep each search in its own folder and use `--page` to request later Flickr pages.
+
+## 🧩 Compatibility Notes
+
+This scraper uses Flickr's JSON `photos.search` API response directly. It does not call `flickrapi.walk()`, avoiding the older `getchildren()` compatibility error reported with `flickrapi==2.4.0` on Python 3.9 and newer.
+
+If Flickr returns a `500` or another API error, the script reports the Flickr error code and message. These errors usually come from Flickr or invalid credentials; retry the command after checking your API key and secret.
+
 ## 📜 Citation
 
 If the Flickr Scraper tool helps your research or work, please consider citing it using the following DOI:

flickr_scraper.py

@@ -3,6 +3,7 @@
 # Generated by Glenn Jocher (glenn.jocher@ultralytics.com) for https://github.com/ultralytics
 
 import argparse
+import os
 import time
 from pathlib import Path
 
@@ -14,54 +15,109 @@
 secret = ""
 
 
-def get_urls(search="honeybees on flowers", n=10, download=False):
-    """Fetch Flickr URLs for `search` term images, optionally downloading them; supports up to `n` images."""
-    t = time.time()
-    flickr = FlickrAPI(key, secret)
-    license = ()  # https://www.flickr.com/services/api/explore/?method=flickr.photos.licenses.getInfo
-    photos = flickr.walk(
-        text=search,  # http://www.flickr.com/services/api/flickr.photos.search.html
-        extras="url_o",
-        per_page=500,  # 1-500
-        license=license,
-        sort="relevance",
+def build_photo_url(photo):
+    """Return the best available static image URL from a Flickr photo response."""
+    url = photo.get("url_o")  # original size
+    return url or (
+        f"https://farm{photo.get('farm')}.staticflickr.com/"
+        f"{photo.get('server')}/{photo.get('id')}_{photo.get('secret')}_b.jpg"
     )
 
+
+def resolve_credentials(api_key=None, api_secret=None):
+    """Resolve Flickr API credentials from arguments, environment variables, or source constants."""
+    api_key = api_key or os.getenv("FLICKR_API_KEY") or key
+    api_secret = api_secret or os.getenv("FLICKR_API_SECRET") or secret
+    if not api_key or not api_secret:
+        help_url = "https://www.flickr.com/services/apps/create/apply"
+        raise ValueError(
+            "Flickr API key and secret are required. Pass --key/--secret, set "
+            f"FLICKR_API_KEY/FLICKR_API_SECRET, or edit flickr_scraper.py. To apply visit {help_url}"
+        )
+    return api_key, api_secret
+
+
+def get_urls(search="honeybees on flowers", n=10, download=False, api_key=None, api_secret=None, page=1):
+    """Fetch Flickr URLs for a search term, optionally downloading up to n images."""
+    if n < 1:
+        return []
+
+    t = time.time()
+    api_key, api_secret = resolve_credentials(api_key, api_secret)
+    flickr = FlickrAPI(api_key, api_secret, format="parsed-json")
+    licenses = ""  # https://www.flickr.com/services/api/explore/?method=flickr.photos.licenses.getInfo
+    page = max(page, 1)
+
     if download:
         dir_path = Path.cwd() / "images" / search.replace(" ", "_")
         dir_path.mkdir(parents=True, exist_ok=True)
 
     urls = []
-    for i, photo in enumerate(photos):
-        if i <= n:
-            try:
-                url = photo.get("url_o")  # original size
-                if url is None:
-                    url = f"https://farm{photo.get('farm')}.staticflickr.com/{photo.get('server')}/{photo.get('id')}_{photo.get('secret')}_b.jpg"
-
-                if download:
-                    download_uri(url, dir_path)
-
-                urls.append(url)
-                print(f"{i}/{n} {url}")
-            except Exception:
-                print(f"{i}/{n} error...")
-
-        else:
-            print(f"Done. ({time.time() - t:.1f}s)" + (f"\nAll images saved to {dir_path}" if download else ""))
+    while len(urls) < n:
+        try:
+            response = flickr.photos.search(
+                text=search,  # https://www.flickr.com/services/api/flickr.photos.search.html
+                extras="url_o",
+                per_page=min(500, n - len(urls)),  # 1-500
+                page=page,
+                license=licenses,
+                media="photos",
+                sort="relevance",
+            )
+        except Exception as e:
+            raise RuntimeError(f"Flickr API request failed: {e}") from e
+
+        if response.get("stat") == "fail":
+            code = response.get("code", "unknown")
+            message = response.get("message", "unknown Flickr API error")
+            raise RuntimeError(f"Flickr API error {code}: {message}")
+
+        photos = response.get("photos", {})
+        photo_list = photos.get("photo", [])
+        if not photo_list:
             break
 
+        for photo in photo_list:
+            url = build_photo_url(photo)
+            if url in urls:
+                continue
+
+            if download:
+                download_uri(url, dir_path)
+
+            urls.append(url)
+            print(f"{len(urls)}/{n} {url}")
+
+            if len(urls) >= n:
+                break
+
+        if page >= int(photos.get("pages", page)):
+            break
+        page += 1
+
+    print(f"Done. ({time.time() - t:.1f}s)" + (f"\nAll images saved to {dir_path}" if download else ""))
+    return urls
+
 
 if __name__ == "__main__":
     parser = argparse.ArgumentParser()
     parser.add_argument("--search", nargs="+", default=["honeybees on flowers"], help="flickr search term")
     parser.add_argument("--n", type=int, default=10, help="number of images")
     parser.add_argument("--download", action="store_true", help="download images")
+    parser.add_argument("--page", type=int, default=1, help="Flickr results page to start from")
+    parser.add_argument("--key", default=None, help="Flickr API key, overrides FLICKR_API_KEY")
+    parser.add_argument("--secret", default=None, help="Flickr API secret, overrides FLICKR_API_SECRET")
     opt = parser.parse_args()
 
-    print(f"nargs {opt.search}")
-    help_url = "https://www.flickr.com/services/apps/create/apply"
-    assert key and secret, f"Flickr API key required in flickr_scraper.py L11-12. To apply visit {help_url}"
-
-    for search in opt.search:
-        get_urls(search=search, n=opt.n, download=opt.download)
+    try:
+        for search in opt.search:
+            get_urls(
+                search=search,
+                n=opt.n,
+                download=opt.download,
+                api_key=opt.key,
+                api_secret=opt.secret,
+                page=opt.page,
+            )
+    except (RuntimeError, ValueError) as e:
+        parser.exit(1, f"{e}\n")

requirements.txt

@@ -3,4 +3,5 @@
 flickrapi
 numpy
 pillow
+requests
 tqdm

tests/conftest.py

@@ -0,0 +1,8 @@
+# Ultralytics 🚀 AGPL-3.0 License - https://ultralytics.com/license
+
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))