[pmwiki-users] Re: Linkchecker and robots.txt

[chr at home.se]

On Thu, 15 Dec 2005, Patrick R. Michaud wrote:

> On Thu, Dec 15, 2005 at 05:42:20PM +0100, chr at home.se wrote:
> > I'm testing the W3C linkchecker:
> > 	http://validator.w3.org/checklink
> > which can also be installed and used locally. 
> > 
> > However, in order to use it a small modification is needed in robots.txt, 
> > or it'll be disallowed otherwise. Here's what you need to append:
> > 
> > 	User-Agent: W3C-checklink
> > 	Disallow:
> > 
> > Would this make sense as a default? (In case robots.txt comes with
> > pmwiki... I can't remember)
> 
> PmWiki doesn't come with a robots.txt (wouldn't do any good if it
> did anyway).

Ok (my installation just happened to be so that robots.txt it's placed in 
the same directory as pmwiki.php).

Do we need some wiki page for robots.txt on pmwiki.org, or is it too 
related directly enough?

/C

-- 
Christian Ridderström, +46-8-768 39 44               http://www.md.kth.se/~chr