View source for Bot Detection

'''''I'm not a native English speaker. Please correct my mistakes.'''''
<br /><br />
[[File:Bot_detection.jpg|right|No Bots!]]Bot Detection is a helper class for other extensions (Frontend) the need to detect whether the access is human or machine.<br />
(Detection of Search Engines, Spider, Crawler, Bots, Harvester, ...) 
{{ExtInfo
| Dev=Glen Langer ([[User:BugBuster|BugBuster]])
| DevSite=http://www.contao.glen-langer.de
| ExtVersion=1.5.0
| Version=from 2.9
| TLVersion=2.8
| ERLink=http://www.contao.org/extension-list/view/botdetection.en.html
| TrackerLink=http://www.contao-forge.org/projects/botdetection/issues
| DonateLink=Cappuccino
| Comment=Version for Contao 2.10 online
}}

[[de:Bot_Detection]]
[[Category:Extensions]]

=Forum=
Questions about the Bot Detection module will be answered in [http://www.contao-community.org/viewforum.php?f=21 Forum]<br />
Errors and requests can be reported in the [http://www.contao-forge.org/projects/botdetection/issues Issue Tracker].

=Installation=
The installation of the module occurs about the extension Repository in back end of Contao.<br />
A manual installation is possible. [http://www.contao.org/extension-list/view/botdetection.en.html Download] the ZIP file from Extension Repository, unzip and transfer it.<br />
A directory should have been created "/system/modules/botdetection".<br />
Then call /contao/install.php  - Perform Update Database. <br />
( /typolight/install.php in older TYPOlight installations )

=Using=
The module Bot Detection provides three methods for detection.<br />
A reliable detection is not possible. <br />
It is to use two methods to detect this:<br />
* User Agent
* IP adress (IPv4, IPv6 from version 1.4.0)

The modul includes a method for the agent [[Bot_Detection#Method_BD_CheckBotAgent|BD_CheckBotAgent]] and one for the IP detection [[Bot_Detection#Method_BD_CheckBotIP|BD_CheckBotIP]].<br />
These two methods return only "true" or "false" and are only a rough search on strings and substrings to identify the most important bots.<br /><br />
Another method [[Bot_Detection#Method_BD_CheckBotAgentAdvanced|BD_CheckBotAgentAdvanced]] comes with an external configuration file for the user agent detection. As a result, it returns the short name of the bots or "false".

==Method BD_CheckBotAgent==
The method BD_CheckBotAgent "searches in two steps to be completed as quickly as possible.<br />
Step 1 searches for substrings that appear in most search engines / bots in the name:<br />
<source lang="text">
'bot'
'spider'
'spyder'
'crawl'
'slurp'
'robo'
'yahoo'
</source>

Step 2 then looks for other strings that usually follow the name of the search engine, such as:<br />
<source lang="text">
'altavista'
'archiver'
'inktomi'
'twiceler'
...
</source>
The result is "true" or "false". ("true" = search engine / bot found)

==Method BD_CheckBotIP==
The bot from Google, MSN / Bing looking sometimes with the user agent from a browser.<br />
To uncover these "undercover" search engines, you must be filtered by IP address.<br />
<br />
There is also configuration files in the ''config'' directory of the module: '''''bot-ip-list.txt''''' and '''''bot-ip-list-ipv6.txt'''''<br />
Current content knows an IP address of a spider from Israel as well as network addresses for Google and MSN / Bing.<br />
Additional IP addresses or networks can be entered in this file, but they are not then upgrade secure.<br />
Therefore, it is better to post them in the localconfig.php as follows:<br />
<source lang="php">
$GLOBALS['BOTDETECTION']['BOT_IP'][] = '192.168.1.2';
$GLOBALS['BOTDETECTION']['BOT_IP'][] = '192.168.0.0/24';
</source>
For IPv6 on this way:
<source lang="php">
$GLOBALS['BOTDETECTION']['BOT_IPV6'][] = '2001:0db8::1';
$GLOBALS['BOTDETECTION']['BOT_IPV6'][] = '2001:0db8:85a3:0800::/56';
</source>

==Method BD_CheckBotAgentAdvanced==
The method BD_CheckBotAgentAdvanced "is controlled by an external configuration file to detect the user agent.<br />
The result is the short name of the bots or "false".<br />
<br />
The external configuration file is generated from known user agent information from search engines / bots and regularly renewed.<br /><br />
'''Note'''<br />
This external DB differentiates between different types of search engines from a manufacturer.<br />
For example, the return is not "Google", but "Googlebot" or "Googlebot-Image" or "Googlebot-Mobile" and so on, depending on what was recognized.<br />
These multiple name of a search engine are also available from other producers such as MSN, Yahoo, and so on.
<br /><br />
Own or unknown user agent identifiers can be entered in the file '''/system/config/localconfig.php''':<br />
<source lang="php">
$GLOBALS['BOTDETECTION']['BOT_AGENT'][] = array("unitbot","UniBot from FHTW");
$GLOBALS['BOTDETECTION']['BOT_AGENT'][] = array("myprivat","My privat bot");
</source>
The parameters are: short name in lower case, description.

=Demo Modules=
Module Bot Detection has 2 frontend demo modules as a demo for the implementation in your own modules.<br />
The using in the demo class is made by import.
<source lang="php">
$this->import('ModuleBotDetection');
</source>

==Frontend Demo 1==
Demo 1 tests with all three methods, the current IP and User Agent identifier and displays the results.<br />
See example on the developer website - [http://www.contao.glen-langer.de/BD_Frontend_Demo_1.html Demo 1].

==Frontend Demo 2==
Demo 2 makes available a form to check whether a user agent identifier from the module would be recognized as a bot.<br />
These are called the two agent methods and displays the result.<br />
See example on the developer website - [http://www.contao.glen-langer.de/BD_Frontend_Demo_2.html Demo 2].
<br /><br />

----
--[[User:BugBuster|BugBuster]] 21:42, 1 July 2011 (CEST)