Haste web spider

'Haste' is Web mapping and monitoring system. As recommended by the Robot Guidelines, this page exists to explain our robot's activities.

We are checking the connectivity of the Web and 'Haste' checks selected websites periodically, compiling averages on link numbers. If a Web site is detected to be on error, 'Haste' logs this information.

'Haste' complies with the rules outlined out in The Robots Exclusion Standard.

If you do not desire 'Haste' to spider web pages on your server, you can instruct it not to do so by simply adding a line to your robots.txt file similar to the following:

User-Agent: Haste
Disallow: /

It may take a long time for 'Haste' to re-index your robots.txt file.

More information on web crawlers, robots, and robots.txt you may find at http://www.robotstxt.org/wc/robots.html.

If you have any questions or comments please contact us at haste@kytoon.com