WhatsApp Deep Fetch

A powerful, command-line WhatsApp full-history exporter built on top of the @whiskeysockets/baileys library.

Unlike standard web/desktop clients that only sync recent messages to save bandwidth, this tool actively requests your entire chat history directly from WhatsApp servers—storing messages and downloading associated media automatically. It handles pagination, connection drops, and API limits gracefully through a resumable checkpoint system.

---

🚀 Features

• Full History Extraction: Actively paginates backwards in time to fetch the entirety of all your conversations, overcoming standard sync limits.
• Resumable State: Automatically saves progress locally (exports/_checkpoint.json). You can safely stop the script at any time; it will resume exactly where it left off on the next run.
• Media Downloading: Automatically downloads attached media (images, videos, audio, documents) and saves them alongside your chats.
• Smart Discovery: Automatically discovers active group chats and individual contacts to fetch.
• Terminal UI: Real-time progress monitoring with multiple progress bars and visual feedback using spinners.
• Data Export: Exports conversation data to JSON (exports/{chat_id}.json).

---

⚙️ Technical Details

• Language: TypeScript
• Runtime Manager: Node.js & tsx (TypeScript Execution)
• Protocol Library: @whiskeysockets/baileys (Bypasses the official Web API, communicating directly with WhatsApp WebSocket instances)
• State Management: In-memory caching with periodic file-system sync to prevent data loss.

📂 Directory Structure

whatsapp-deep-fetch/
├── auth/                 # Stores generated Baileys authentication keys (created on run)
├── exports/              # Deep fetch output JSONs and checkpoints (created on run)
├── src/
│   ├── checkpoint/       # Checkpoint read/writes and batch processing logic
│   ├── cli/              # Command-Line arguments parser and options
│   ├── export/           # JSON writers for backing up deeply fetched messages
│   ├── fetch/            # Core loops for deep fetching and synchronous chat queues
│   ├── media/            # Payload decoders and media download workers
│   ├── messages/         # Real-time message storage logic
│   ├── ui/               # Progress bars, spinners, and interactive CLI outputs
│   ├── config.ts         # User-tunable timeouts, delays, and batch sizes
│   ├── errors.ts         # Network and application-level custom errors
│   ├── shutdown.ts       # Signal interceptors for safe and clean exits
│   ├── state.ts          # Centralized in-memory store for contacts and chats
│   └── types.ts          # Shared TypeScript models and interfaces
├── index.ts              # Main entry point and lifecycle orchestrator
├── package.json          # Dependencies and script definitions
└── tsconfig.json         # TypeScript compiler configuration

Architecture Highlights

• Auth Layer: Emulates a macOS Desktop WhatsApp Linked Device. State is safely stored in the ./auth/ directory.
• Deep Fetching: Uses synthetic cursors (chatId, messageId, timestamp) to poll WhatsApp servers for older batches. It handles empty pages and API throttles.
• Media Handler: Hooks into Baileys' media decipher streams to save blobs locally.
• Graceful Shutdown: Intercepts SIGINT/SIGTERM to safely dump the in-memory state and checkpoint to disk before exiting.

---

🛠️ Usage

Prerequisites

• Node.js (v18+ recommended)
• yarn or npm package manager

Installation

1. Clone the repository and install dependencies:

   yarn install
   # or
   npm install

Running the Exporter

To start the script:

yarn start
# or 
npm run start

Behind the scenes, this runs node --import tsx/esm index.ts.

1. Link Device: On the first run, the terminal will display a QR code. Open WhatsApp on your phone go to Settings → Linked Devices → Link a Device and scan it.
2. Initial Sync: It will wait a few moments for the standard WhatsApp "initial sync" push.
3. Deep Fetch: It will automatically queue all your chats and start deeply paginating backward through them history, updating progress bars in the terminal.

CLI Arguments & Reset Flags

# Exclude broadcast/LID chats
yarn start -- --no-lid

# Exclude group chats
yarn start -- --no-group

# Soft reset (Deletes exported chat JSONs and checkpoints, keeps auth and known contacts)
yarn start -- --reset

# Reset checkpoint only (Forces deep fetch to start over for existing chats)
yarn start -- --reset-cp

# Reset authentication (Requires sweeping a new QR code)
yarn start -- --reset-auth

# Hard reset (Wipes auth, exports, media, and checkpoints)
yarn start -- --reset-full

---

💻 Developer Info & Configuration

If you want to tweak the fetch parameters, edit src/config.ts.

Key configuration variables:

• DEEP_FETCH_BATCH_SIZE (default: 30): Limits how many chats are processed concurrently to avoid memory spikes and API rate limits.
• ON_DEMAND_BATCH_SIZE (default: 500): The number of messages to request per historic pagination call.
• DOWNLOAD_MEDIA (default: true): Toggle this if you only want text exports.
• MAX_MEDIA_SIZE_BYTES: Limits media downloads to prevent storing exceptionally large files.
• timers like RESUME_TRIGGER_DELAY_MS and IDLE_TIMEOUT_MS can be tweaked to control how aggressively the bot waits for syncs before jumping to the manual deep fetch cycle.

Key Files

• index.ts: The main entry point, sets up the socket and main event listeners.
• src/fetch/history.ts: Contains the logic for the backwards pagination loop.
• src/checkpoint/checkpoint.ts: Reads/Writes the _checkpoint.json logic.
• src/media/download.ts: Extracts and saves media buffers.

🤝 Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

📝 License

MIT