09 October 2006 06:30
Robots and Spiders
For the majority of websites, a robot or spider (both technically the same) will crawl your site almost daily, whether it be from Google, MSN, Yahoo, or some other search engine such as Alexa. Sometimes you don't want all your pages, or your images to be indexed. This guide covers that and more.
1
(1 marks)