Robots is a simple Ruby library to parse robots.txt files.

WWW: http://github.com/fizx/robots