Documentation: Explain meaning of blacklisted pages and difference to "exclude page from search engines"

[mathias-noethiger]

I've created a new page which should not be indexed (e.g. 404 Page for Users).
In the page properties, tab SEO, I activated the checkbox "exclude page from search engines" and I also excluded it from "indexes search".
I then checked the sitemap. The new page was still listed there.

What do I have to do next?

• just remove the page over the sitemap tool? Or will it appear again after the next sitemap "update"?
• put it on the black list?

System: TYPO3 6.2 , metaseo 1.0.8

[thomaszbz]

Please upgrade to metaseo 2.0.0. It contains a lot of bug fixes compared to 1.0.8.

Make sure you clear all caches, including opcode cache (can be found in install tool) and system cache. If the problem still occurs, you might want to delete all entries in the sitemap manually as a one-time administrative task.

Please report back if that solved your issue.

[mathias-noethiger]

That all worked fine. When I activate "exclude page from search engines" the page get automatically the status "YES" in the sitemap overview and the new column "Excl."

I'm still wondering what is the idea of having "blacklist" "withelist" options. I could not find anyting in the documentation nor TYPO3 help.

[thomaszbz]

As far as I understand it, blacklisted pages are supposed to not be indexed, while indexing is done recursively at a limited number of levels, when a page is requested by a user. Indexing is done for the purpose to add new items to metaseo's sitemap table, which in turn allows to edit the item. Of course, indexing is heavily cached for performance reasons.

https://github.com/mblaschke/TYPO3-metaseo/blob/develop/Classes/Hook/SitemapIndexHook.php#L221
https://github.com/mblaschke/TYPO3-metaseo/blob/develop/Classes/Hook/SitemapIndexHook.php#L252
https://github.com/mblaschke/TYPO3-metaseo/blob/develop/Classes/Hook/SitemapIndexPageHook.php#L97

Blacklisting is different from "exclude page from search engines" in a manner that we'd still index a page without the blacklisting. That's at least how I understand it. You are right that the blacklisting is not very well documented. Documentation should be improved here. Maybe the documentation change should also be accompanied by a FAQ entry.

That said, I think the use case is to avoid indexing a lot of subpages of a blacklisted page. Blacklisting adds some sort of granularity for exactly that purpose.

If you want to provide an improvement to the documentation, I'd be happy to review and merge it. If in doubt, we can still ask @mblaschke for details. My answer might not necessarily be 100% accurate here.