[pmwiki-users] Linkchecker and robots.txt

[Patrick R. Michaud]

On Thu, Dec 15, 2005 at 05:42:20PM +0100, chr at home.se wrote:
> I'm testing the W3C linkchecker:
> 	http://validator.w3.org/checklink
> which can also be installed and used locally. 
> 
> However, in order to use it a small modification is needed in robots.txt, 
> or it'll be disallowed otherwise. Here's what you need to append:
> 
> 	User-Agent: W3C-checklink
> 	Disallow:
> 
> Would this make sense as a default? (In case robots.txt comes with
> pmwiki... I can't remember)

PmWiki doesn't come with a robots.txt (wouldn't do any good if it
did anyway).

Pm