I think your crawler detection code needs a bit more work. I was just crawler blocked from Freeones, with the website returning the IP address of my ISPs transparent proxy cache as a crawler bot. Given just about every ISP in my part of the world is putting their broadband customers through transparent proxy caches for http requests, I am guessing there are going to be a lot of confused people getting blocked when they probably shouldn't be.
Crawler detection
- Thread starter greenmeanie
- Start date
