Migrate to poetry instead of conda (#55)

Author: dcaribou

.github/workflows/scrapy-checks.yml

@@ -21,10 +21,14 @@ jobs:
 
     steps:
     - uses: actions/checkout@v2
-    - uses: conda-incubator/setup-miniconda@v2
+    - name: Install Poetry
+      uses: snok/install-poetry@v1
       with:
-        environment-file: environment.yml
-        activate-environment: transfermarkt-scraper
+        virtualenvs-create: true
+        virtualenvs-in-project: true
+        installer-parallel: true
+    - name: Install library
+      run: poetry install --no-interaction
     - name: Check scrapy contracts
       run: |
-        scrapy check -s USER_AGENT='transfermarkt scrapy contracts'
+        poetry run scrapy check -s USER_AGENT='transfermarkt scrapy contracts'

Dockerfile

@@ -1,14 +1,16 @@
-FROM continuumio/miniconda3
+FROM python:3.8
 
 WORKDIR /app
 
-# The code to run when container is started:
-COPY tfmkt tfmkt
-COPY scrapy.cfg .
+RUN apt-get update && \
+    apt-get -y install gcc python3-dev
 
-# Create the environment:
-COPY environment.yml .
-RUN conda env create -f environment.yml
+COPY pyproject.toml /app
 
-ENV PATH /opt/conda/envs/transfermarkt-scraper/bin:$PATH
-RUN /bin/bash -c "source activate transfermarkt-scraper"
+ENV PYTHONPATH=${PYTHONPATH}:${PWD}
+
+RUN pip3 install poetry
+RUN poetry config virtualenvs.create false
+RUN poetry install --no-dev
+
+RUN /bin/bash -c "poetry shell"

README.md

@@ -18,22 +18,20 @@ Each one of these entities can be discovered and refreshed separately by invokin
 ## Installation
 
 This is a [scrapy](https://scrapy.org/) project, so it needs to be run with the
-`scrapy` command line util. A conda `environment.yml` file is provided with a definition
-for the necessary environment to run the scraper.
+`scrapy` command line util. This and all other required dependencies can be installed using [poetry](https://python-poetry.org/docs/).
 
 ```console
-# create and activate conda environment
-conda env create -f environment.yml
-conda activate transfermarkt-scraper
+cd transfermarkt-datasets
+poetry install
+poetry shell
 ```
-> :information_source: On Apple silicon chips fallback to rosetta to avoid well-known [pyopenssl issues](https://github.com/pyca/pyopenssl/issues/873) by creating your conda environment as `CONDA_SUBDIR=osx-64 conda env create -f environment.yml`
+
+## Usage
 
 > :warning: This project will not run without a user agent string being set. This can be done one of two ways:
 > - add `ROBOTSTXT_USER_AGENT = <your user agent>` to your tfmkt/settings.py file, or
 > - specify the user agent token in the command line request (for example, `scrapy crawl players -s USER_AGENT=<your user agent> `)
 
-## Usage
-
 These are some usage examples for how the scraper may be run.
 
 ```console