ChatGPT Indexed Shares Investigation

A security research project documenting the discovery that thousands of ChatGPT conversation shares have been indexed and archived by web crawlers and public databases.

🔍 Overview

This repository contains tools and documentation related to our investigation into ChatGPT share indexing. We discovered that 15,000+ ChatGPT share URLs have been captured by web archives (Wayback Machine, Common Crawl, etc.), creating potential privacy risks for users who thought their shares were "private via link."

Read the full research article: ARTICLE.md

🚨 Key Findings

• 15,000+ shares archived in CDX and Common Crawl databases
• ~8% still accessible, 92% deleted or private
• Shares persist in archives even after deletion
• Content includes sensitive information, code, API keys
• No built-in mechanism for users to remove archived shares

🛠️ Tools

1. CDX Discovery Engine

Discovers archived ChatGPT shares from the Wayback Machine's Capture Index:

python cdx_discovery.py --source cdx --limit 10000 --output cdx_shares.txt

Features:

• Query Internet Archive CDX API for all chatgpt.com/share/* URLs
• Extract unique share GUIDs
• Export to database or text file
• Support for date-range filtering

2. Common Crawl Scanner

Searches Common Crawl's web archive for ChatGPT shares:

python cdx_discovery.py --source commoncrawl --index CC-MAIN-2024-10 --output cc_shares.txt

Features:

• Search across Common Crawl index files
• Multi-snapshot support (monthly archives since 2023)
• Parallel processing for large datasets
• Deduplication across snapshots

3. Share Accessibility Verifier

Tests archived shares to determine current accessibility:

python verify_shares.py --input discovered_shares.txt --output verified_shares.json

Features:

• HTTP status checking (200 OK, 404 Not Found, etc.)
• Content verification (checks for "Can't load shared conversation")
• Categorization: accessible, deleted, private, error
• Rate limiting to respect server resources
• Resume capability for large datasets

4. Archive Search Tool

Searches archive content for specific patterns or keywords:

python search_archives.py --keyword "API_KEY" --source cdx_shares.txt

Features:

• Full-text search within archived share content
• Pattern matching (regex support)
• Multiple archive source support
• Privacy-preserving (no content storage)

📊 Statistics

From our research across multiple archive sources:

Archive Source	URLs Found	Unique GUIDs	Date Range
Wayback Machine CDX	10,247	8,503	2023-02 to present
Common Crawl	5,836	4,127	2023-03 to present
Google Cache	~2,000	~1,800	2023-06 to present
Total (deduplicated)	15,412	12,641	2023-02 to present

Accessibility Status:

• 8.2% still accessible (valid shares)
• 91.8% deleted/private (return "Can't load" message)

Content Analysis:

• ~65% contain code snippets
• ~30% contain API/documentation discussions
• ~5% contain personal/business information

🔧 Installation

# Clone the repository
git clone https://github.com/actyra/chatgpt-indexed-shares.git
cd chatgpt-indexed-shares

# Install dependencies
pip install -r requirements.txt

# Run discovery
python cdx_discovery.py --source cdx --limit 1000

📋 Requirements

• Python 3.8+
• requests library
• beautifulsoup4 (for content parsing)
• tqdm (for progress bars)

🔒 Privacy & Ethics

Our Approach

• No Content Harvesting: We analyze URLs and patterns, not conversation content
• No Republishing: We do not share or republish discovered conversations
• Responsible Disclosure: Findings shared to raise awareness, not exploit
• Privacy First: All examples use synthetic data

Research Ethics

This research was conducted to:

1. ✅ Raise awareness about privacy risks
2. ✅ Help users protect their data
3. ✅ Encourage platform improvements
4. ❌ NOT to exploit or access private information

🛡️ Recommendations

For ChatGPT Users

1. Audit your shares: Review all conversation shares you've created
2. Delete sensitive shares: Remove any containing private information
3. Check archives: Search Wayback Machine for your share URLs: https://web.archive.org/web/*/chatgpt.com/share/YOUR-GUID
4. Request removal: Contact archive.org to request specific URL removal
5. Treat shares as public: Assume any share link may be archived permanently

For OpenAI

1. Implement robots.txt: Add Disallow: /share/ to prevent crawler indexing
2. Meta tags: Add <meta name="robots" content="noindex,noarchive"> to share pages
3. User warnings: Alert users that shares may be indexed by web crawlers
4. Archive partnerships: Work with archive services to honor deletion requests
5. Expiring links: Consider auto-expiring shares after 30/60/90 days

For Archive Services

1. Respect privacy: Honor deletion requests for ChatGPT shares
2. Pattern recognition: Auto-exclude patterns like API keys from archiving
3. Opt-out mechanism: Allow users to remove their content

📖 How Archives Work

Internet Archive (Wayback Machine)

1. Crawler discovers link → chatgpt.com/share/abc123
2. Page captured and stored
3. Indexed in CDX database (searchable)
4. Retained indefinitely (unless removal requested)

Common Crawl

1. Monthly web crawl (petabyte scale)
2. All discovered URLs archived
3. Full-text indexing via AWS Athena
4. Dataset published publicly for research
5. Permanent retention

Discovery via CDX API

# Find all archived ChatGPT shares
curl "https://web.archive.org/cdx/search/cdx?url=chatgpt.com/share/&matchType=prefix&output=json&limit=10000"

🔗 Related Projects

• ChatGPT Share Scanner - GUID pattern analysis and brute-force discovery tools
• Common Crawl Index - Petabyte-scale web archive
• Internet Archive CDX API - Archive query interface

📝 Usage Examples

Basic Discovery

# Discover from CDX
python cdx_discovery.py --source cdx --limit 5000 --output cdx_shares.txt

# Discover from Common Crawl
python cdx_discovery.py --source commoncrawl --index CC-MAIN-2024-10 --output cc_shares.txt

Verification

# Verify which shares are still accessible
python verify_shares.py --input cdx_shares.txt --output results.json --concurrent 10

Analysis

# Analyze discovery statistics
python analyze_stats.py --input results.json --report stats_report.md

Search Archives

# Search for specific content patterns (ethical use only)
python search_archives.py --pattern "api[_-]?key" --input cdx_shares.txt --report findings.txt

⚠️ Responsible Use

This toolset is for security research and privacy awareness only.

Permitted Uses:

• ✅ Personal audit of your own shares
• ✅ Academic research with ethics approval
• ✅ Security analysis and awareness
• ✅ Platform improvement recommendations

Prohibited Uses:

• ❌ Accessing others' private conversations
• ❌ Harvesting sensitive information
• ❌ Republishing discovered content
• ❌ Commercial exploitation of findings

📄 License

MIT License - See LICENSE for details

🤝 Contributing

Contributions welcome! Please read our contributing guidelines and code of conduct.

Focus areas:

• Additional archive source integrations
• Privacy-preserving analysis techniques
• User-friendly audit tools
• Documentation improvements

📬 Contact

• Issues: GitHub Issues
• Security: For sensitive disclosures, see SECURITY.md
• Research: For collaboration inquiries, see CONTRIBUTING.md

🙏 Acknowledgments

• Internet Archive for CDX API access
• Common Crawl for open datasets
• Security research community for responsible disclosure practices

---

Disclaimer: This research is for educational and security awareness purposes. We do not condone accessing, storing, or republishing private conversations. Always respect user privacy and platform terms of service.