/**

popular collection of scripts for all

*/
/** Search */

{ 

Robots.txt parser

 }

Robots.txt parser

Hits: 17
*****
0.0
Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details).
Platform(s): Windows
Date: Nov, 14 2005
Author: Toon Goedhart, http://www.hockeytrader.nl/content/view/3/2/
{ 

License

 }
  • License #1
    • License Type: GPL
    • price: Free
  • License #2
    • License Type: GPL
    • price: Free
  • License #3
    • License Type: Freeware
    • price: Free
{ 

Ratings

 }
Number of Ratings: 0 Votes
Visitor Voting Booth:

   
1 2 3 4 5
{ 

Others Scripts

 }
{ Copyright } ©2006 NuclearScripts.com