Revision as of 20:35, 19 February 2012

I'm not a native English speaker. Please correct my mistakes.

(Detection of Search Engines, Spider, Crawler, Bots, Harvester, ...)

Forum

Questions about the Bot Detection module will be answered in Forum
Errors and requests can be reported in the Issue Tracker.

Installation

The installation of the module occurs about the extension Repository in back end of Contao.
A manual installation is possible. Download the ZIP file from Extension Repository, unzip and transfer it.
A directory should have been created "/system/modules/botdetection".
Then call /contao/install.php - Perform Update Database.
( /typolight/install.php in older TYPOlight installations )

Using

The module Bot Detection provides three methods for detection.
A reliable detection is not possible.
It is to use two methods to detect this:

• User Agent
• IP adress (IPv4, IPv6 from version 1.4.0)

The modul includes a method for the agent BD_CheckBotAgent and one for the IP detection BD_CheckBotIP.
These two methods return only "true" or "false" and are only a rough search on strings and substrings to identify the most important bots.

Another method BD_CheckBotAgentAdvanced comes with an external configuration file for the user agent detection. As a result, it returns the short name of the bots or "false".

Method BD_CheckBotAgent

The method BD_CheckBotAgent "searches in two steps to be completed as quickly as possible.
Step 1 searches for substrings that appear in most search engines / bots in the name:

'bot'
'spider'
'spyder'
'crawl'
'slurp'
'robo'
'yahoo'

Step 2 then looks for other strings that usually follow the name of the search engine, such as:

'altavista'
'archiver'
'inktomi'
'twiceler'
...

The result is "true" or "false". ("true" = search engine / bot found)

Method BD_CheckBotIP

The bot from Google, MSN / Bing looking sometimes with the user agent from a browser.
To uncover these "undercover" search engines, you must be filtered by IP address.

There is also configuration files in the config directory of the module: bot-ip-list.txt and bot-ip-list-ipv6.txt
Current content knows an IP address of a spider from Israel as well as network addresses for Google and MSN / Bing.
Additional IP addresses or networks can be entered in this file, but they are not then upgrade secure.
Therefore, it is better to post them in the localconfig.php as follows:

$GLOBALS['BOTDETECTION']['BOT_IP'][] = '192.168.1.2';
$GLOBALS['BOTDETECTION']['BOT_IP'][] = '192.168.0.0/24';

For IPv6 on this way:

$GLOBALS['BOTDETECTION']['BOT_IPV6'][] = '2001:0db8::1';
$GLOBALS['BOTDETECTION']['BOT_IPV6'][] = '2001:0db8:85a3:0800::/56';

Method BD_CheckBotAgentAdvanced

The method BD_CheckBotAgentAdvanced "is controlled by an external configuration file to detect the user agent.
The result is the short name of the bots or "false".

The external configuration file is generated from known user agent information from search engines / bots and regularly renewed.

Note
This external DB differentiates between different types of search engines from a manufacturer.
For example, the return is not "Google", but "Googlebot" or "Googlebot-Image" or "Googlebot-Mobile" and so on, depending on what was recognized.
These multiple name of a search engine are also available from other producers such as MSN, Yahoo, and so on.

Own or unknown user agent identifiers can be entered in the file /system/config/localconfig.php:

$GLOBALS['BOTDETECTION']['BOT_AGENT'][] = array("unitbot","UniBot from FHTW");
$GLOBALS['BOTDETECTION']['BOT_AGENT'][] = array("myprivat","My privat bot");

The parameters are: short name in lower case, description.

Demo Modules

Module Bot Detection has 2 frontend demo modules as a demo for the implementation in your own modules.
The using in the demo class is made by import.

$this->import('ModuleBotDetection');

Frontend Demo 1

Demo 1 tests with all three methods, the current IP and User Agent identifier and displays the results.
See example on the developer website - Demo 1.

Frontend Demo 2

Demo 2 makes available a form to check whether a user agent identifier from the module would be recognized as a bot.
These are called the two agent methods and displays the result.
See example on the developer website - Demo 2.

--BugBuster 21:42, 1 July 2011 (CEST)