GhostlyGrabber

👻 Advanced Telegram Channel Data Collection & Analysis Utility

Tool for automated collection and analysis of data from Telegram channels and chats with complete content archiving and interaction graph generation.

✨ New Features

📊 Extended User Data Collection

• Complete user profiles: ID, username, first name, last name, phone (if available)
• Statuses and flags: verification, Telegram Premium, bots, scam/fake accounts
• Activity: online/offline status, last seen timestamp
• Additional information: bio, language, profile photo
• Relationships: mutual contacts, presence in your contacts

👥 Channel Members Collection

• Retrieval of all channel/group members
• Saving membership data (join date, status)
• Integration with global user database

🔗 Link Extraction

• Automatic extraction of all URLs from messages
• Support for Telegram links (t.me, @username)
• Saving link context (surrounding text)
• Link classification by type (telegram, http, https)

🕸️ Interaction Graph Analysis

• Build interaction graphs from collected data
• Nodes: users, channels, domains
• Edges: membership, replies, forwards, link mentions
• Export to multiple formats (GraphML, GEXF, JSON, CSV, PNG)
• Filter by date, minimum reply count
• Graph statistics (node/edge types, degrees, bot/premium counts)

🗄️ Improved Database Structure

• Normalized schema with table relationships
• Extended models for storing all available data
• Automatic updating of existing records
• Support for incremental collection

📋 Requirements

• Python 3.10+
• All dependencies from requirements.txt
• Telegram API credentials in .env file or environment variables

⚙️ Configuration

Example .env file:

API_ID=your_telegram_api_id
API_HASH=your_telegram_api_hash

Installing dependencies:

# Clone repository
git clone https://github.com/He4vyL0v3/GhostlyGrabber
cd GhostlyGrabber

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate  # Linux/Mac
# or
venv\Scripts\activate     # Windows

# Install dependencies
pip install -r requirements.txt

🚀 Usage

Data Collection

1. Create a .env file in the project root with your Telegram API credentials
2. Run the collector:

cd src
python3 main.py

Follow the prompts to specify output folder, channel name, and collection options.

Proxy Support

Create proxies.txt in src/ folder with proxy list. Supported formats:

Format	Example
SOCKS5	socks5://host:port
SOCKS5 with auth	socks5://user:pass@host:port
SOCKS4	socks4://host:port
HTTP	http://host:port
MTProto	mtproto://secret@host:port
Telegram native	tg://proxy?server=...&port=443&secret=...

Example:

socks5://127.0.0.1:1080
mtproto://secret@host:443
tg://proxy?server=s3.sosproxy.space.mtproto.ru&port=443&secret=ee520bd57bf173f2eee56fb9ce7afbb58c64726976652e676f6f676c652e636f6d

Data Analysis

After collection, analyze the database with the analyzer:

python3 analyzer.py /path/to/GhostlyGrabber.db [options]

Common options:

• --with-links – include domain nodes from extracted links
• --start-date, --end-date – filter by date (ISO format)
• --formats graphml png csv-nodes – export formats
• --min-replies 3 – minimum replies to create an edge

Example:

python3 analyzer.py ~/data/GhostlyGrabber.db --with-links --formats graphml png --start-date 2024-01-01

📁 Project Structure

GhostlyGrabber/
├── src/
│   ├── main.py              # Main collection script
│   ├── analyzer.py          # Analysis & graph building
│   ├── db.py                # Database logic
│   ├── models.py            # Database models
│   ├── dialog.py            # User dialogs
│   ├── extractors.py        # Link extraction from text
│   ├── cache.py             # API response cache
│   ├── proxy.py             # Proxy management
│   └── utils.py             # Utility functions
├── requirements.txt         # Python dependencies
└── README.md               # Documentation

🗄️ Database Structure

Main tables:

• users – extended user data
• posts_{channel} – channel posts
• discussions_{channel} – comments/replies
• members_{channel} – channel members
• links_{channel} – links from messages

users table fields (selected):

Field	Description
user_id	Unique Telegram user ID
username, first_name, last_name	Basic info
phone	Phone number (if available)
bot, verified, premium	Account flags
scam, fake, deleted, restricted	Warning flags
status, was_online	Activity info
about, lang_code	Additional info
created_at, updated_at	Timestamps

🔄 Working with Existing Data

The program supports incremental collection:

• On subsequent runs, only new messages are downloaded
• Existing records are updated with new information
• Media files are not overwritten if they already exist

⚠️ Limitations and Features

Access rights:

• Member collection: requires admin rights in channel/group
• User data: depends on privacy settings
• Phone number: only available with mutual contacts or public settings

Telegram API limits:

• Request frequency limitations
• Possible temporary blocking with too active collection
• User privacy settings – main limiting factor

Available user data:

✅ Always available:

• User ID (permanent identifier)
• First and last name (if set)
• Username (if public)
• Flags (bot, verification, etc.)

⚠️ Conditionally available:

• Phone number (requires mutual contacts)
• Last online time (depends on settings)
• Profile photo (can be downloaded separately)

📊 Use of Collected Data

Collected data can be used for:

• OSINT investigations: user activity analysis
• Content archiving: complete channel preservation
• Link analysis: information dissemination study
• Statistics: member count, post popularity analysis
• Network analysis: interaction graphs for community detection

🛡️ Security and Ethics

1. Comply with laws: tool usage must comply with your country's legislation
2. Respect privacy: do not collect data unnecessarily or without consent
3. Don't violate ToS: adhere to Telegram's terms of service
4. Use responsibly: only for legal research and archiving

🎨 Code Style

Project uses Black formatter with line length 120.

Using formatter:

python format.py              # Format src/ directory
python format.py src/ package/  # Format specific paths
python format.py --check       # Check without changes
python format.py -v            # Verbose output
python format.py --line-length 100  # Custom line length

Installation:

pip install black

🤝 Contributing to the Project

We welcome contributions to project development:

1. Report bugs via Issues
2. Suggest improvements via Pull Requests
3. Share ideas for new features

Created with ❤️ by Nighty3098 for the OSINT and Telegram research community