[calorie]

If a bot isn't on a remove SID list, and then it crawls, it gets a SID, and when it comes back to respider, the bot has the SID in the respider request because it was assigned a SID initally. What is suggested may strip SIDs from bot requests that are new, but for respider requests, if the bot was initially assigned a SID, the bot remembers the SID, so the SID is in the respider request. In my situation, I didn't have msnbot on a remove SID list, so after some time, the bot was making quite a lot of respider requests for the same pages but using different SIDs. By the time I realized what was happening, I was out a good chunk of bandwidth so this is the type of situation where I think this hack is useful.