[Wplug-web] robots.txt

Jeremy Dinsel jeremy at gunix.net
Sat Jan 6 19:48:59 EST 2001


Does anyone know how to disallow a directory for all servers but the search
engine being run on the local server? Or a method to get htdig to overlook the
disallow in the robots.txt file?

I'd like to have a search engine locally scan the mail achives, but I don't
want our mail archives showing up on google ...

-j
--
- Jeremy Dinsel
- somewhere in the monkey house





More information about the Wplug-web mailing list