| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The most important thing about a program like this is to realise that if you set it up incorrectly and used it in the wrong way, you could upset a large number of people who have set up their web servers in the assumption that they would be used normally by human beings browsing through on Netscape.
It is true that LinkController is very careful to limit resource usage on remote sites, but the other site may not know that or may have a real reason not to want their pages visited too often.
Probably it's true that the only safe way forward is for every WWW site to begin to set up robot defences and detect when someone starts to download from them at an unreasonable rate and then cut off the person doing the downloading. I suggest that you don't make people have to do this to protect themselves against you for at least two reasons.
There are probably many other reasons, but that's one for the good side in you and one for the selfish. What more do you need.
For suggestions about what constitutes `correct' behaviour, it's worth seeing the Robots World Wide Web page. http://www.robotstxt.org/wc/robots.html
There are a number of points which make LinkController relatively safe
as a link. These are all related to the design and limitations on
test-link.
test-link does not recurs. It only tests links that
are specifically listed in the schedule database.
The last limitation is inherited from the LWP::RobotUA module and
the documentation for that covers the details of how it works.
test-link tries to re-order testing of links as needed so that
a limit on the rate of visits to one site does not cause a limit on
overall testing speed.
| [ << ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |