Dynaverse.net

Off Topic => Engineering => Topic started by: Bonk on August 13, 2005, 09:54:55 am

Title: Web Robots
Post by: Bonk on August 13, 2005, 09:54:55 am

For reference:

http://www.robotstxt.org/wc/robots.html

http://www.wdvl.com/Location/Search/Robots.html

Title: Re: Web Robots
Post by: toasty0 on August 13, 2005, 10:02:09 am

I remember reading somewhere that Google and possible Yahoo ignore that protocol now. Not sure how accurate that info is though...

Title: Re: Web Robots
Post by: Bonk on August 13, 2005, 03:15:52 pm

Google and Yahoo both are still looking for robots.txt, not sure if they are heeding it or not (could test relatively easily).

Title: Re: Web Robots
Post by: toasty0 on August 13, 2005, 10:01:50 pm

If you do let me know, but I think it was one of those ranting blogs that included gloom and doom forcasts for the future of net.

Title: Re: Web Robots
Post by: Just plain old Punisher on August 23, 2005, 03:05:23 am

Ummm doom and gloom?

Can I commence profiteering and hoarding?