Pages ending with a fragment are always marked as uncrawled

[jacomyma]

By "fragment" we mean internal anchors: index.html versus index.html#section1.
This is just an observation. It might be the desired behavior.

Pro: it makes sense since they should not be crawled, being just the same content as other already crawled pages.

Con: it leaves an impression that the crawl is unfinished, even though it actually is (ie. we do not want to crawl these pages anyway)

[boogheta]

Decision CPH191021-1130 : we keep them, even though uncrawled, but when we will have Javascript crawling we will crawl them with these kind of crawls

[boogheta]

attaching #288