Quote:
Originally Posted by Dave84311
You mentioned not limiting bots by cookies, why not check their USER AGENT? Generally bots have static user agents.
|
Cookies insure any basic spider makes it through without having to worry about listing every bot known.
With hundreds of bots and evolving names [a smart bot would certainly check with another user agent to detect content gaming] it's not the primary method I went for. 5 or 6 of the biggest are whitelisted this way however. You are free to add as many more as you like.