Use git scraping in refurbished products

[zmoog]

https://simonwillison.net/2020/Oct/9/git-scraping/

[zmoog]

The first step is creating a new repository to host the data: https://github.com/zmoog/refurbished-history

[zmoog]

I am adding the label git-scraping topic as other people are doing.

[zmoog]

Try to scrape some data:

asdf local python 3.10.8

python -m venv venv
. venv/bin/activate

pip install refurbished
pip install --upgrade pip

rfrb it ipads --format json | jq > it-ipads.json

[zmoog]

We should cache Python packages to use few resources as possible. Here is some article to start from:

• https://github.blog/changelog/2021-11-23-github-actions-setup-python-now-supports-dependency-caching/

[zmoog]

Added a simple workflow to start scraping:

name: Scrape latest data

on:
  push:
  workflow_dispatch:
  # schedule:
  #   - cron:  '6,26,46 * * * *'

jobs:
  scheduled:
    runs-on: ubuntu-latest
    steps:
    - name: Check out this repo
      uses: actions/checkout@v2
    - uses: actions/setup-python@v2
      with:
        python-version: '3.10'
        cache: 'pip'
    - run: pip install refurbished
    - name: Fetch latest data
      run: |-
        # curl https://www.fire.ca.gov/umbraco/Api/IncidentApi/GetIncidents | jq . > incidents.json
        for STORE in "it" "uk" "us" "cn" "ie" "fr" "ca"
        do
            PRODUCTS="stores/$STORE"
            mkdir -p ${PRODUCTS}

            for PRODUCT in "ipads" "iphones" "macs"
            do
                echo "Scraping $STORE $PRODUCT"
            rfrb ${STORE} ${PRODUCT} --format json 2> /dev/null > ${PRODUCTS}/${PRODUCT}.json
            done
        done        
    - name: Commit and push if it changed
      run: |-
        git config user.name "Automated"
        git config user.email "actions@users.noreply.github.com"
        git add -A
        timestamp=$(date -u)
        git commit -m "Latest data: ${timestamp}" || exit 0
        git push

I am disabling the schedule until I tested it using workflow_dispatch

[zmoog]

Weird, the setup-python@v2 step fails with this error:

Error: No file in /home/runner/work/refurbished-history/refurbished-history matched to [**/requirements.txt], make sure you have checked out the target repository

It looks like this steps looks for a requirements.txt file in the repo.

[zmoog]

Oh, I see it uses the hash of requirements.txt to cache dependencies:

The action defaults to searching for a dependency file (requirements.txt for pip, Pipfile.lock for pipenv or poetry.lock for poetry) in the repository, and uses its hash as a part of the cache key.

See Caching packages dependencies for more details.

[zmoog]

So I will:

• upgrade setup-python to v4, since there are several warning from the action.
• embrace requirements.txt

[zmoog]

The next step is schedule the scrape. I believe we can start with once a day at 9:00 am (CET).

[zmoog]

GitHub Actions schedule trigger uses UTC.

Let's schedule the scraping job at 9:00 UTC:

on:
  push:
  workflow_dispatch:
  schedule:
    - cron:  '00 09 * * *'