aboutsummaryrefslogtreecommitdiff
path: root/www/p5-WWW-RobotRules-Parser/pkg-descr
blob: 2d6cb651ee9929424adfef2ed54eff9dc9143390 (plain) (blame) 
1
2

WWW::RobotRules::Parser allows you to simply parse robots.txt files
as described in http://www.robotstxt.org/wc/norobots.html.
generated by cgit v1.2.3 (git 2.25.1) at 2024-09-26 18:20:40 +0000