An Awesome Url Crawler: (Developed Using Python)
How to Use ? Usage: main.py -u url -d depth
Options:
-h, --help show this help message and exit
-u URL Specify the begin url.
-d DEPTH Specify the crawling depth. Default: 5.
-t THREAD The amount of threads. Default: 10.
Licence