feat: add on_link_blocked_callback for robots.txt blocked URLs#394

j-mendez merged 2 commits into

spider-rs:mainspider-rs/spider:mainfrom

zanmato:feat/robots-blocked-callbackzanmato/spider:feat/robots-blocked-callbackCopy head branch name to clipboard