Robots is a simple Ruby library to parse robots.txt files.

WWW: https://github.com/fizx/robots