Author rocky.chimailto:rockychi1001@gmail.com
1.Title spider, using requests and beautifulsoup4 to crawl and parse html data; handle chinese characters smartly, avoid gibberish chinese characters(中文乱码).
2.Price spider, using selenium driver to start chrome or firefox (phantomjs will do the same work); crawling element loaded by js asynchronously in the page, such as price.
Title spider can crawl urls according to existed url file, parse and get data in <title></title> block, then return the new result file with title data added.
Price spider added. Using selenium (or phantomjs) to crawl price element of the specified host (eg:jd.com, tmall.com), because the price element is load by js asynchronously in the page.