Problems with "?" in robots.txt

[loxal]

In https://www.welt.de/robots.txt there are ? containing entries like Disallow: /*?config. Hence https://www.welt.de/test?config should be allowed but it is not. Whereas entries like Disallow: /*.xmli work properly and disallow https://www.welt.de/test.xmli. After my investigation I figured out that ? is the problematic character.

I use RobotstxtServer#allow("https://www.welt.de/test?config") for testing.

[s17t]

The internal robots.txt parser is far from being RFC complaint. There are several issue around (#256, #107) and probably more. I asked to another project (crawler-commons/crawler-commons#197) with a better robots.txt parser to split in modules to allow to reuse it in crawler4j.

If you have patch they are welcome in the meanwhile.

[sebastian-nagel]

... Disallow: /*?config. Hence https://www.welt.de/test?config should be allowed but it is not.

According to Google's spec this URL is not allowed (* is a glob character, while ? is not). The RFC draft does not support any glob characters in allow/disallow statements (see robotstxt.org):

Note also that globbing and regular expression are not supported ... Specifically, you cannot have lines like ... "Disallow: *.gif".

crawler-commons follows here the Google spec which is widely used - I suppose that even the webmaster of welt.de wants to disallow /test?config.

[loxal]

According to https://en.wikipedia.org/wiki/Glob_(programming) both * and ? are glob characters. RFC drafts are nice but ultimately Google sets the standards (even if they are not RFC-compliant). And according to Google * is supported as stated in Google's spec.

[sebastian-nagel]

Agreed, Google sets the standard and this means ? has no glob meaning in the Disallow: statement which " support a limited form of "wildcards" for path values".

[sebastian-nagel]

@loxal, or did I misunderstand you: Shall /*?config match /test?configor not? If yes, it's disallowed, right?

[loxal]

@sebastian-nagel whatever Google considers correct. But ? does not need any glob meaning to support
Disallow: /*?config and disallow /test?config? In this case * is a glob character and is supported, whereas ? is just a regular character.

I think the current implementation considers also ? as a glob character and therefore has problems to evaluate *?.

[sebastian-nagel]

Ok, we agree. I haven't tested crawler4j but just wanted to make sure that crawler-commons behaves as expected: /test?config is forbidden but eg. /test/config is allowed. Thanks!

[loxal]

I just checked with https://www.google.com/webmasters/tools/robots-testing-tool
No matter what Google's documentation says, the actual implementation speaks another language:
Entries like Disallow: /*?config result in /test?config being ignored. One of those - not too rare - case where documentation is misleading.

[sebastian-nagel]

Entries like Disallow: /*?config result in /test?config being ignored.

What does "being ignored" mean? Is it crawled or not?

[sebastian-nagel]

If it means that Googlebot does not visit /test?config this is in accordance with the documentation.

[loxal]

The crawler ignores it. /test?config is not crawled. This is what Google's robots Tool says.

[sebastian-nagel]

Yes, but then the behavior and documentation agree: the wildcard * matches test and all remaining characters (including ?) are matched literally. It's a disallow rule, so bots shall not crawl everything matching /*?config, right?

[loxal]

Yes.

[sebastian-nagel]

I'll keep this on the radar, and will add a unit test to crawler-commons' robots.txt parser, just to make sure that it continues to work. Thanks!