[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

A. Robots and Sensible Behaviour

The most important thing about a program like this is to realise that if you set it up incorrectly and used it in the wrong way, you could upset a large number of people who have set up their web servers in the assumption that they would be used normally by human beings browsing through on Netscape.

It is true that LinkController is very careful to limit resource usage on remote sites, but the other site may not know that or may have a real reason not to want their pages visited too often.

Probably it's true that the only safe way forward is for every WWW site to begin to set up robot defences and detect when someone starts to download from them at an unreasonable rate and then cut off the person doing the downloading. I suggest that you don't make people have to do this to protect themselves against you for at least two reasons.

There are probably many other reasons, but that's one for the good side in you and one for the selfish. What more do you need.

For suggestions about what constitutes `correct' behaviour, it's worth seeing the Robots World Wide Web page. http://www.robotstxt.org/wc/robots.html

There are a number of points which make LinkController relatively safe as a link. These are all related to the design and limitations on test-link.

The last limitation is inherited from the LWP::RobotUA module and the documentation for that covers the details of how it works. test-link tries to re-order testing of links as needed so that a limit on the rate of visits to one site does not cause a limit on overall testing speed.


[ << ] [ >> ]           [Top] [Contents] [Index] [ ? ]

This document was generated by Michael De La Rue on February, 3 2002 using texi2html