DDoS causing Server to Crash, Asking for Mod

[Sycosphere]

Hello,
My site has been under heavy DDoS attacks for the past month!, it seems like they are using vBulletin against the server since the index page has many queries, and running the index page for say 500 times/second would send thousands of queries to the Mysql server causing it to crash over and over again.

Since vb.org is the official site for vbulletin modifications, I was thinking of an easy solution that I couldnt make, I'm a beginner PHP programmer but havnt got into vb mods programming yet. The idea was to create a modification that applys to all the templates, simply checking for a session variable ( $_SESSION['human'] ) , cookies or both. if they exist than its a real user, otherwise its a bot or a just a visitor.
If its a visitor than the mod would show a little form with one submit button ( Click me to enter ). the form would assign the ( human ) session variable and show the previously requested URL.
This would prevent requesting all the queries on the index page (or any other page) and replace it with a harmful, light and query-less page when detecting a non-human or a user without cookies.

See the PHP code below to better understanding of the idea:

IF ( !isset($_SESSION['human') || !isset($_COOKIE['prefix_userid'])) {
show a form with a submit button ( Click to enter ).
} else {
(Show the requested page ).
}
///////////////////
The form processing code would be something similar to this :
IF (isset($_POST['click_me'])) {
$_SESSION['human']==1;
}


The mode would simply be applied to the header to prevent loading any unnecessary data that would consume server resources during a DDoS attack, memebers wont need a variable to tell vb they're human since they have a userid cookie, visitors would need such confirmation. thats it!

I hope you guys got the idea, and hope one of you is gonna be able to do it in the next 2 or 3 days. I'm really in need for it, its just that I'm too busy upgrading to a dedicated server to learn the vBulletin coding HowTo's now!

BTW, sorry for the bad english, after all its a second language to me!
Thanks

[Lynne]

I saw a suggestion that you simply add an htaccess file to the forum directory with password protection when you are under a dos attack. You can put the username and password right on the form but since bots don't read them, they won't get in, but since your users can read, they just enter the information once and are in.

[Sycosphere]

Lynne,
I already tried that, there seem to be only one problem with this solution, about 95% of my visitors Arabians and htaccess files will show any arabic text as a rubbish.
I thought of this solution coz the ppl who are attacking my forum are using it ! I just understood the mechanism of it. and it seem to be working for them!

BTW, using this way a webmaster could even allow search spiders to surf the site by checking its browser or somethin that could tell them apart from other kinds of bots.

Thanks

[Sycosphere]

Hi,
There seems to be another problem with your suggestion Lynne, which is that some DDoS softwares now have the ability to bypass htaccess firewalls by simply going to the site, getting the username and password shown in the msg, and then entering them in the software itself and that would make the proxies able to enter that username and password when requesting the site! .
I guess this way is gone with the old days now, anyways , I've managed to create that firewall myelf, I'll be posting it here soon :)

[Lynne]

Is suppose you could be right (I really don't know). I have not solved our dos problems by doing that. We used a script that filled up our iptables with all ips that were pounding our server. It was still a painful four days, but you could get on the site.

I hate people who have too much time on their hands and think it would be fun to dos a site!

[2guys1forum]

I have tried these things but the only thing that seems to work is to have a strong IPtables setup, otherwise the DDoS will just continue. Also make sure that it is not one of the vBulletin mods you have installed is being DDoS'ed as it is easy to do so.

I had a shoutbox installed which was easy to DDoS and that is how my forums got DDoS'ed with only the use of 12 bots. Therefore I suggest you check all your mods and make sure they are protected from DDoS attacks.

[Sycosphere]

Well, I'm not that much of a guru when it comes to linux, this IPtables term is totally new to me so I had to be a little creative, So I changed the mechanism of the mod that I had in mind the first time I thought of it.
Now its working this way, a visitor opens up a page from my forum, php generates an md5 hash for him, then shortens it to only 6 chars and stores it in a session variable, and shows a form with a ( Click me button ). when the visitors clicks that button, a hidden field named 'h' (short for hash) is submitted and compared with the session variable already stored. So it works sort of like the session id concept now.
btw, I used GET method in that form to get around that security token issue when you post a form to a vb script that has CSRF protection turned on.

Sometimes this helps, specially when the ppl who r attacking you r not that smart, or not too many!

Anyways the attack has stopped, and I can go on with my life now :), I'll be posting any other kind of solution for this problem that I find.

Thanks

[snakes1100]

If your actually being DDoS'd and its already in progress, 99% of the time you won't even be able to access your server, you will need to speak to your host, your not going to stop a DDoS attack with anything via the server/forums, the attack needs to be stopped at the router level if the server is already over whelmed with the mass request attack.

As well as some posts here say, your hacks arent whats being attacked via DDoS, DDoS attacks happen at the NIC level and is targeting things such as echo/ping requests and flooding the server with to many requests. An attack on a poorly written PHP page isn't really considered a DDoS attack.

[Sycosphere]

Hello Snake,
obviously, I'm not an owner of a site that would get a 1000 users with broadband connections to actually DDoS him and flood the NIC itself! , neither 99% of forum owners here are!
What your talking about happens to site like Yahoo! for crying out load!, while what I'm talking about is some softwares being spread between users and script kiddies with good connections that make repetitive requests to database driven sites causing the mysql server to crash the whole server!. What I'm saying here is .... Imagine the difference between sending 30 queries a 1000 times in one second, and processing a simple html page :)
I think there is a huge difference when u see it that way, of course only in cases like mine, not the mass-destructive cases ur talking about :o

Another point to be considered is that about 70% (maybe more I dont know) of database driven websites are not on a dedicated server, so when a site causes a server to crash, the hosting company will advise the site owner to get a dedicated server, or simply kicks him out!.
Also hosting companies does not offer DDoS protection for free, so another possible respond would be advising the site owner to get a DDoS protection and pay huge extra fees monthly. Ask me, I'm hosted with iWEB ! :mad:

You said :
your not going to stop a DDoS attack with anything via the server/forums
Well, Lynne here had another opinion: :)
We used a script that filled up our iptables with all ips that were pounding our server. It was still a painful four days, but you could get on the site.

btw, I dont know why I get the feeling that you thought I said that they are attacking one of hacks, well .. I didnt, they're simply attacking different vbulletin scripts like index.php, image.php, misc.php ... etc, I wouldnt exactly call vb a "poorly written php page".

Anyways, the solution actually worked, on some level ... So I guess I'll stick to the idea and try to develop it a little bit more so it involves IPtables maybe.

[fatal1980]

I think you need to identify the attack first before you try blocking it. Are you sure its not just a simple syn flood attack? Which is rather simple to block. Even then if they're making multiple connections to your server per second you can block them using iptables.

iptables -A BADGUY -p tcp --syn --dport 80 -m recent --name dos --update --seconds 10 --hitcount 16 -j DROP

[Sycosphere]

OK fatal, I'm not a 100% about the type of this attack, and as I mentioned before, I dont have that much of an experience with linux, so let me describe the attack:
I have my forum in /vb/ directory, the ppl who were attacking me were attacking that specific directory only, I noticed that because when I felt that the server is slowing down I changed the folder's name through ssh, and it worked! .. so this told me that the attack is not on the network level, its mainly directed to www.sycoz.com/vb only .
When the attack starts, whenever I try to open a page it just keeps loading and loading without any response from the server, once the attacks stops, the server goes back to its normal condition!.
At one time I was talking with the attacker on yahoo chat, he told me that he's gonna attack in a few seconds, I opened my forum and it was completely normal, then he told me to refresh and when I did the server was not giving any response !, a dedicated server gets jammed by only one user and in a few seconds !!!

Ok now I hope you guys have enough info and may be able to define the type of the attack, and btw, I could provide a link to the software they're using, please PM for it if your interested in ending this headache for me and for many other forum owners :)

Thanks

[royo]

In /etc/sysctl.conf you may want to try changing the following
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_max_syn_backlog=1536

After that use the command sysctl -p to reload the config.

I'm not 100% sure on this so you may want to remember the default values you are using and switch it back if it isn't working.

[Kendothpro]

The best solution would probably be to install mod_evasive directly on your apache server..it allows you to set the max number of requests per second per client...anything above will only get an access denied message

[Angel-Wings]

The best solution would probably be to install mod_evasive directly on your apache server

No - this wouldn't help. This mod works on a "per" client base, this isn't useful for Distributed Attacks like a dDoS and so could do more damage.
Until this mod reacted and blocked a client, the next proxy / bot is used already

[Sycosphere]

Ok guys, you all seem to agree that this is a SYN flood attack, any other opinions?
And if it IS a SYN attack, could someone with the proper experience possibly give us a solution that he/she is 100% sure that it works?
This topic has become more like a discussion, so lets put it together and offer a solution that would work for atleast 80% of the people that are having the same problem that Iam having.

BTW, Lynne, can you give me the name of that script you guys used ?

Waiting for more opinions guys :)

[Lynne]

BTW, Lynne, can you give me the name of that script you guys used ?
Hehe. It's called "scott.sh" Scott is the guy I hire to do stuff to our server and he wrote the code for us. It's just grabs the ip or anyone pounding the tracker, throws the ip into a file and then the file is input into the iptables.

[Sycosphere]

Lolz, I guess no one will ever gonna wanna use a script named scott anyway/joke
OK Lynne, I guess I'll have to finish my exams and start digging deep in this IPtables thing and linux generally, coz from now on I'm responsible of our server's management.

Thanks anyway Lynne, big thanks to all the guys who participated here, and it would be very nice to have others replying with useful opinions :)

[Shazz]

If this is not resolved yet I would be happy to help you out over instant messenger just PM me for it or check my profile.

Alot of hosts are on apache and do not specialize in these types of attacks, Ive had huge DDos attacks that bypass every firewall you install or any script you put in including reverse DNS etc.

Just give me a shout if you run out of options..

[Sycosphere]

Well Shazz, thats very nice of you, I added you on yahoo chat and will be talking to you when u get online, thanks in advance :)

btw ppl, I we ( me & Shazz ) could figure out a solution that acualy works than I'll be posting some kind of a HOWTO here about it, just to help other site owners get through this kind of attacks in the future :)

[space?]

Hehe. It's called "scott.sh" Scott is the guy I hire to do stuff to our server and he wrote the code for us. It's just grabs the ip or anyone pounding the tracker, throws the ip into a file and then the file is input into the iptables.
Sorry, I don't understand this one. If every visitors IP is listed and blocked in the iptables - who can use the board anymore?

btw ppl, I we ( me & Shazz ) could figure out a solution that acualy works than I'll be posting some kind of a HOWTO here about it, just to help other site owners get through this kind of attacks in the future :)
looking forward to it:)

[Lynne]

Sorry, I don't understand this one. If every visitors IP is listed and blocked in the iptables - who can use the board anymore?
I said it only grabs the IPs of those pounding the board (or tracker). Most regular users do not pound the board.

[space?]

And how to you get the IP of the users which are pounding the board?
Every mechanism which would be able to split the IPs into "good" and "bad" ones, would be another possible target, I think.

[Sycosphere]

Space, do you visit vb.org a 100 times/minute ?? or u have more than a 1000 hit/day on it?? they use this kind of information to separate humans from bots.

[space?]

Yeah, knowing that.
But I think this seperation could be a vuln too.

I didn't wanted to make you trouble, if it works and helps - fine.

[Lynne]

Our script did cause some problems. We have a tracker on our board also, so there are times when a single IP will hit the tracker and if it's too busy at that moment, it sends the request again (and sometimes again and again, it depends on the bittorrent client) and so a few legitimate users did get caught in our IP tables. I did have a notice on the top of the board telling users that if they were blocked, to send me an email with their IP. I also flushed the iptables a couple of times a day.