Robots is a simple Ruby library to parse robots.txt files.