popular collection of scripts for all

robots.spider(tm)

robots.spider(tm) reads the robots.txt file, then spiders the site (depth is adjustable in the downloaded code), to check for references to entries in the robots.txt file. robots.spider(tm) can help you avoid disclosing the names of directories you don't want indexed, that aren't referenced in your site code. It is an open source (PHP) script, you can run it for free off the demo link, or download the code, adjust the depth, and use it to check your sites from a different server.

Platform(s): Linux
Date: Oct, 09 2006
Author: Betsy A. Gamrat, http://robots-wizard.com

{ 

Ratings

 }

Number of Ratings: 458 Votes