public marks

PUBLIC MARKS from jedi58 with tags yahoo & slurp

09 October 2006 06:30

Robots and Spiders

For the majority of websites, a robot or spider (both technically the same) will crawl your site almost daily, whether it be from Google, MSN, Yahoo, or some other search engine such as Alexa. Sometimes you don't want all your pages, or your images to be indexed. This guide covers that and more.

jedi58's TAGS related to tag yahoo

alexa +   development +   engine +   google +   googlebot +   msn +   robots +   search +   search engines +   seo +   slurp +   spiders +