Skip to content

README.md

Latest commit

 

History

History
91 lines (64 loc) · 2.33 KB

README.md

File metadata and controls

91 lines (64 loc) · 2.33 KB

WorldEndArchive

"When the lights go out and the signals fade,
And the silence of the void begins to wade,
This archive stands, a beacon in the night,
Preserving wisdom, keeping truth in sight.
For if the world should crumble, dust to dust,
In this digital vault, we place our trust."

Overview

WorldEndArchive is a resilient, autonomous web crawler designed to preserve human knowledge in the face of catastrophe. It tirelessly traverses the web, capturing essential information and storing it in accessible formats for a post-internet world.

crawler client

Setup & Usage

Setup Gatherer (The Crawler)

The Gatherer traverses the web to collect data.

  1. Navigate to directory:

    cd gatherer

  2. Install dependencies:

    npm install

  3. Configure:

    • Copy .env.example to .env.
    • Edit .env to set your proxies, concurrency, etc.

  4. Start Crawling:

    npm start
Setup Client (The Reader)

The Client allows you to browse the archived content offline.

  1. Navigate to directory:

    cd client

  2. Install dependencies:

    npm install

  3. Start Reader:

    npm start

Configuration

Edit gatherer/.env to customize:

  • MAX_CONCURRENCY: Number of simultaneous requests.
  • DELAY_BETWEEN_REQUESTS_MS: Throttle speed.
  • USE_PROXIES: Enable/disable proxy rotation.
  • KEYWORDS_FILE: Path to your custom keywords.

Contributing

If you are reading this, the internet is likely still operational. Feel free to contribute to the codebase to ensure it is ready for when it is not.

Preserving the past, for the future.