Skip to content

README.md

Latest commit

 

History

History
177 lines (124 loc) · 4.85 KB

README.md

File metadata and controls

177 lines (124 loc) · 4.85 KB

Data Scraping API

A lightweight Python project for scraping product details from any API and saving the results in json format.

This repo contains two implementations:

  • api_call_async.py — asynchronous scraping using aiohttp (recommended for high concurrency)
  • api_call_multi_thread.py — multi-threaded scraping using requests and ThreadPoolExecutor

✅ What this project does

  • Reads a list of product IDs from an Excel file
  • Fetches product details from a configurable API endpoint in batches
  • Writes:
    • successful records to a fetched_data folder
    • failed/errored records to a faulty_data folder
  • Uses a rotating User-Agent pool to reduce blocking
  • Implements retries + exponential backoff for transient failures (e.g., 429 Too Many Requests)

✅ Features

  • Batch-driven scraping with configurable batch size
  • Rotating User-Agent pool to reduce blocking
  • Retry + exponential backoff for transient failures (e.g., 429 Too Many Requests)
  • Output results to JSON files (fetched / faulty)
  • Optional async mode for better throughput

🧩 Project Structure

api_call_async.py
api_call_multi_thread.py
user_agents.json
async_fetched_data/
async_faulty_data/
fetched_data/
faulty_data/
.env.example
README.md

⚙️ Prerequisites

  • Python 3.10+
  • pip available

⚙️ Setup

  1. Create a Python environment (recommended):

    python -m venv .venv
.venv\\Scripts\\activate

  2. Install dependencies:

    pip install -r requirements.txt

    If there is no requirements.txt, install the used packages manually:

    pip install aiohttp python-dotenv pandas python-calamine

  3. Copy .env.example to .env and fill in your settings:

    DATA_ID_SOURCE="path/to/excel.xlsx"
FETCHED_DATA_SINK="async_fetched_data"
FAULTY_DATA_SINK="async_faulty_data"
API_URL="https://api.tiki.vn/product-detail/api/v1/products"
BATCH_SIZE=100
ASYNC_CONCURRENT_LIMIT=50
USER_AGENT_LIST="user_agents.json"

▶️ Running the async API function (recommended)

python api_call_async.py

This will:

  • Read product IDs from the Excel file configured by DATA_ID_SOURCE
  • Fetch product details in batches concurrently
  • Write successful results to the FETCHED_DATA_SINK folder
  • Write failed/errored results to the FAULTY_DATA_SINK folder
  • This mode is ideal for high throughput using aiohttp and asynchronous I/O.

▶️ Running the multi-threaded scraper

python api_call_multi_thread.py

This mode uses a thread pool for synchronous HTTP requests and is useful for debugging or when asyncio is not desired.

💡 Customization

Batch size

Change BATCH_SIZE in .env.

Concurrent limit (async)

Adjust ASYNC_CONCURRENT_LIMIT in .env.

Worker limit (multi-threaded)

Adjust MAX_WORKERS in .env.

Output file naming

Uses the write_to_json() helper and supports optional keyword args like pre_fix and batch_no for descriptive filenames.

🧪 Notes / Troubleshooting

  • TypeError: write_to_json() got an unexpected keyword argument ...

    • Ensure write_to_json() is defined with **kwargs.

  • Too many 429 Too Many Requests errors

    • Reduce concurrency (async: ASYNC_CONCURRENT_LIMIT, thread: MAX_WORKERS)
    • Increase backoff delay between retries

  • Missing/invalid .env values

    • Confirm .env exists and is formatted correctly (numeric values for BATCH_SIZE, ASYNC_CONCURRENT_LIMIT, MAX_WORKERS).

📓 Development Notes

06 Mar 2026

Changes made:

  • multi-threading method for API calls:

    • Implement new data loading library for reading large Excel files efficiently:
      • openpyxl with optimized read-only mode
      • pandas with calamine for quick Excel file parsing -> better performance and lower memory usage
    • Adjust naming conventions for better clarity and consistency across the codebase
    • Refactor data batching logic to allow for configurable batch sizes, improving flexibility and scalability of the data processing pipeline

  • async method for API calls:

    • Implemented asynchronous API calls using aiohttp to improve performance and reduce latency when fetching data from the API
    • Added error handling for HTTP status codes, including retries with exponential backoff for transient errors like 429 Too Many Requests
    • Refactored code to separate concerns, improving readability and maintainability of the codebase

Future improvements:

  • Implement logging instead of print statements for better monitoring and debugging
  • Add unit tests for critical functions to ensure reliability and facilitate future refactoring

📨 Contact

For questions or contributions, please open an issue or submit a pull request. Email: duongghuy96@gmail.com

📌 License

MIT-style (adapt as needed)