denybot - Hide page content for search engine bots.

Summary

This Extension deny the access to all or single web pages for detected Web crawler.

Functionality

With the usage of the extension Bot Detection the calling agent should be identified. If the calling agent is a search engine bot, the real content of the web page was hidden and a 404-Errorpage will return instead.

Configuration

In System-Settings you could find in sektion Global configuration the option to exclude searchengines global.

If this option is set, every frontend access from a identified search enginge bot will return a 404-error page.

If this option is not set, only pages with selected robot-tag "noindex,nofollow" or "noindex,follow" in their page configuration will generate the 404-error page for detected serarch engine bots.

Why should I use this extension

A normal web site is designed to be found in the internet.

This extension will serve the opposite function and try to hide the content. Search engines should only found not existing pages.

In that case this extension is designed especially for private usage. If you want to personally announce that you created a web page, e.g. to stay in contact with your family and friends across the word, and don't want to invite everyone in the internet, this extension could protect your pages against search engine bots. This extension could only be used as an additional features like using a login to protect your content. It could not replace any other type of data protection.

Restriction

The detection of a search engine bot depends on the quality of Bot Detection engine. The Author BugBuster explicitly pointed out: "There is no reliable detection guaranteed." The search engines are detected by their user agent string and their IP address. BugBusters still collect a huge list of different and popular search engines and he still extends the quality of detection.