blob: d633f72baa8c18f4c7a0a22802969c91b8e3be86 (
plain) (
blame)
1
2
3
4
5
6
|
This module parses /robots.txt files which are used to forbid conforming
robots from accessing parts of a web site. The parsed files are kept in
a WWW::RobotRules object, and this object provides methods to check if
access to a given URL is prohibited.
WWW: https://metacpan.org/release/WWW-RobotRules
|
