Slow to very slow checks for urls

[Tijs-2]

Our spider found a url with a lot of underscores. The website was a mess so they screwed up but nevertheless it exists in the wild.

If you have a url like 'http://______text' it is no problem

But if you do a check like this:
checkers.is_url("http://a_____________text_________done")
it already takes seconds

But then if you do it like this:
checkers.is_url("http://a______________________________________________________________text_________done")
you will be able to get a cup of coffee before the answer is there :) (I never waited until the end ;) )

[insightindustry]

I've taken a look at this issue, and the problem lies in a performance cliff in the regex matching that applies to URLs. It'll take some amount of effort to further optimize the regex expression used for URLs, and unfortunately the "easy" solution of using a better regex engine isn't an option given the design constraints of the library. However, the regex expressions used are quite complicated and I'm pretty certain there's a lot of performance optimization that can be done. However, I'm going to bump this milestone to not the next version, but the version after that so that it doesn't hold up the next release.