Denybot
From Contao Community Documentation
denybot - Hide page content for search engine bots.
| Extension-Overview | |
|---|---|
| Name of the developer | weke |
| Version of the extension | 0.9.0 rc1 |
| Compatibility with Contao Version | 2.9.5 - 2.9.5 |
| Link to Extension Repository | http://www.contao.org/erweiterungsliste/view/denybot.html |
| Depending of ff. Extension | Bot Detection |
Summary
This Extension deny the access to the web pages for detected Web crawler.
Functionality
With the usage of the extension Bot Detection the calling agent should be identified. If the calling agent is a search engine bot, the real content of the web page was hidden and a 404-Errorpage will return instead.
Configuration
Currently there are no configuration options.
After installing this extension every request of a web crawler will be rejected. For a future version is planned to make this feature configurable.
Why should I use this extension
A normal web site is designed to be found in the internet.
This extension will serve the opposite function and try to hide the content. Search engines should only found not existing pages.
In that case this extension is designed especially for private usage. If you want to personally announce that you created a web page, e.g. to stay in contact with your family and friends across the word, and don't want to invite everyone in the internet, this extension could protect your pages against search engine bots. This extension could only be used as an additional features like using a login to protect your content. It could not replace any other type of data protection.
Restriction
The detection of a search engine bot depends on the quality of Bot Detection engine. The Author BugBuster explicitly pointed out: "There is no reliable detection guaranteed." The search engines are detected by their user agent string and their IP address. BugBusters still collect a huge list of different and popular search engines and he still extends the quality of detection.
