A Python application that scrapes and analyzes Windows applications (from PortableApps.com for now) to identify unsigned binary files.
- Clone or download this repository
- Navigate to the project directory
- Install required dependencies:
pip install -r requirements.txtThe application uses a config.json file for configuration. All components automatically read from this centralized configuration file.
{
"base_url": "https://portableapps.com/apps/music_video/",
"download_dir": "downloads",
"extract_dir": "extracted",
"reports_dir": "reports",
"log_dir": "logs",
"log_level": "INFO",
"scrape_delay": 1.5,
"max_apps": 10,
"max_extract_size": 524288000,
"chunk_size": 8192,
"cleanup_extracted": true,
"use_cache": true,
"skip_apps": [
"Specific App Name",
"Another App to Skip"
],
"skip_categories": [
"browser", "chrome", "firefox", "edge",
"torrent", "bittorrent", "ftp", "ssh"
]
}Run the complete analysis pipeline:
python main.py# Analyze with custom configuration
python main.py --config custom_config.json
# Limit number of apps to analyze
python main.py --max-apps 5
# Set log level
python main.py --log-level DEBUG
# Analyze a single application by URL
python main.py --single-url "https://example.com/app.exe" --app-name "TestApp"
# Disable caching
python main.py --no-cache
# Cache management
python main.py --show-cache
python main.py --clear-cache all
python main.py --clear-cache urlsRun the test suite to verify functionality:
# Test skip functionality
python test.py skip
# Test normal workflow with limited apps
python test.py workflowThe caching system improves performance by storing:
- URL Cache: Application page URLs and download links
- File Cache: Downloaded application files
- Result Cache: Analysis results and signature verification data
# View cache statistics
python main.py --show-cache
# Clear specific cache types
python main.py --clear-cache urls # Clear URL cache
python main.py --clear-cache files # Clear file cache
python main.py --clear-cache results # Clear analysis results
python main.py --clear-cache all # Clear everything