blob: 3cc1478249b26a8c34eefa8fe6cbc56626eec503 (
plain) (
tree)
|
|
WWW::RobotRules::Parser allows you to simply parse robots.txt files
as described in http://www.robotstxt.org/wc/norobots.html.
WWW: http://http://search.cpan.org/dist/WWW-RobotRules-Parser/
|