Robots is a simple Ruby library to parse robots.txt files.

WWW: https://github.com/fizx/robots
RG:  https://rubygems.org/gems/robots