This article Robots .txt Adventure by Andrew Wooster offers an insightful and humorous look at the state of robots.txt files found on the web today.

This entry was posted on Wednesday, September 26th, 2007 at 5:16 pm and is filed under Web Development. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.