AI Documentation Scraper

A tool for scraping and generating Q&A pairs for RNN training using GPT-4.

Prerequisites

Install Poetry (if not already installed):

curl -sSL https://install.python-poetry.org | python3 -

Install dependencies:
```
poetry install
```
Set up environment variables:
- Copy .env.example to .env
- Add your OpenAI API Key to the .env file

Start the scraper in the Poetry environment:

poetry run python scrape.\*.py

Generated Q&A pairs will be saved to `data/*.jsonl

Name		Name	Last commit message	Last commit date
Latest commit History 24 Commits
.vscode		.vscode
batch00		batch00
batch01		batch01
batch02		batch02
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
check.api.key.py		check.api.key.py
combine_jsonl.py		combine_jsonl.py
credits-left.py		credits-left.py
example.py		example.py
poetry.lock		poetry.lock
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
scrape-generic-docs.ps1		scrape-generic-docs.ps1
scrape.dockerhub.docs.py		scrape.dockerhub.docs.py
scrape.generic.docs.py		scrape.generic.docs.py
scrape.github.actions.docs.py		scrape.github.actions.docs.py
scrape.github.docs.py		scrape.github.docs.py
scrape.github.packages.docs.py		scrape.github.packages.docs.py
scrape.npm.docs.py		scrape.npm.docs.py
stats.json		stats.json
to-training-data-only.py		to-training-data-only.py
training-data.jsonl		training-data.jsonl