public marks

PUBLIC MARKS from jedi58 with tags googlebot & robots

09 October 2006 06:30

Robots and Spiders

For the majority of websites, a robot or spider (both technically the same) will crawl your site almost daily, whether it be from Google, MSN, Yahoo, or some other search engine such as Alexa. Sometimes you don't want all your pages, or your images to be indexed. This guide covers that and more.

jedi58's TAGS related to tag googlebot

alexa +   development +   engine +   google +   msn +   robots +   search +   search engines +   seo +   slurp +   spiders +   yahoo +