logsparser.extras.robots

In this module we define a regular expression used to fetch the most common robots.

robots

Value:

['antibot',
 'appie',
 'architext',
 'bingbot',
 'bjaaland',
 'digout4u',
 'echo',
 'fast-webcrawler',
...

robot_regex

Value:

re.compile(r'(?i)antibot|appie|architext|bingbot|bjaaland|digout4u|ech
o|fast-webcrawler|ferret|googlebot|gulliver|harvest|htdig|ia_archiver|
askjeeves|jennybot|linkwalker|lycos|mercator|moget|muscatferret|myweb|
netcraft|nomad|petersnews|scooter|slurp|unlost_web_crawler|voila|voyag
er|webbase|weblayers|wisenutbot|aport|awbot|baiduspider|bobby|boris|bu
mblebee|cscrawler|daviesbot|exactseek|ezresult|gigabot|gnodspider|grub
|henrythemiragorobot|holmes|internetseer|justview|linkbot|metager-link
checker|linkchecker|microsoft_url_control|msiecrawler|nagios|perman|po
...

Variables
	robots = `['antibot', 'appie', 'architext', 'bingbot', 'bjaalan...`
	robot_regex = `re.compile(r'(?i)antibot\|appie\|architext\|bingbot...`
	__package__ = `'logsparser.extras'`

Module robots

robots

robot_regex