vb.org Archive - Miscellaneous Hacks - Ban Spiders by User Agent

Page 3 of 5

Show 40 post(s) from this thread on one page

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- vBulletin 3.8 Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=235)

- - Miscellaneous Hacks - Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=264932)

Simon Lloyd

07-13-2012 04:21 AM

Sure, are you sure you want to create threads? you'd get soooooo many of them, remember i added the thread making and logging for test purposes so folk could see that it was working for their selected spiders are being dealt with.

Max Taxable

07-13-2012 02:01 PM

Quote:

Originally Posted by Simon Lloyd (Post 2347488)

Yes I have had this Mod working for so long, the really bad spiders like Baidu have stopped visiting completely. This Mod generated only 88 threads yesterday. I plan to turn the thread making feature on and off periodically, just for occasional monitoring purposes.

The IPs in the post would be great, if it's not too difficult and if you please.

Edit: It would be nice too if we could make it post everything in the same thread in addition to the option of creating a new topic for each bot.

Simon Lloyd

07-13-2012 02:48 PM

Ok, if you can wait until next week as i've promised my wife that i'll give the computer a rest for a week (other than important stuff) so i'll get to it after then :)

Simon Lloyd

07-13-2012 05:16 PM

@Max Taxable for now (just because i can't make you wait :)) go to admincp>plugins & products>Plugin Manager find the Ban Spiders mod's plugins and look for Post UA Activity to thread and replace all of the code you see there for this

PHP Code:


		
			
if ($vbulletin->options['bsactive'] AND $vbulletin->options['bs_report_createthread'])
{
$agentip = strtoupper($_SERVER['REMOTE_ADDR']);
$DateOfRequest = date('m-d-Y H:i:s');
$user_agent1 = strtoupper($_SERVER['HTTP_USER_AGENT']);
$bots1 = explode("\r\n", $vbulletin->options['bs_sel_list']); 
$n = sizeof($bots1);
for ($i=0;$i<$n;$i++) {
if (stristr($user_agent1,$bots1[$i])) 
{
    // Backend Files
    require_once('./global.php');
    require_once('./includes/class_dm.php');
    require_once('./includes/class_dm_threadpost.php');
    require_once('./includes/functions_databuild.php');

$threaddm =& datamanager_init('Thread_FirstPost', $vbulletin, ERRTYPE_ARRAY, 'threadpost');
$threadinfo = array();
$vboptions =& $vbulletin->options;
$username = $vbulletin->userinfo['username']; 
$forumid = $vbulletin->options['bs_report_forumid']; 
$foruminfo = fetch_foruminfo($foruminfo['forumid']);
$title = "Activity from Bot No. $i ($bots1[$i]) in your list";   
$userid = $vbulletin->options['bs_report_userthread'];
if ($userid == '')
{
  $userid=1;
}
if ($forumid =='')
{
exit;
}
$pagetext = "[b]Date and Time:[/b] $DateOfRequest\n[b]Associated Username (if any): [/b]$username\n[b]Matched bots[$i]:[/b] $bots1[$i] \n[b]With User Agent:[/b] $user_agent1\n\n[b]Their IP:[/b]$agentip\n\n[b]They visited:[/b]\n[url]$_SERVER[SERVER_NAME]$_SERVER[REQUEST_URI][/url]\n\n"; 
$posticon = $vbulletin->options['subthread_posticon'];
$allowsmilie = '1';
$visible = '1';

$threaddm->set_info('forum', $foruminfo);
$threaddm->set_info('thread', $threadinfo);
$threaddm->setr('forumid', $forumid);
$threaddm->setr('userid', $userid);
$threaddm->setr('pagetext', $pagetext);
$threaddm->setr('title', $title);
$threaddm->set('iconid', $posticon);
$threaddm->set('allowsmilie', $allowsmilie);
$threaddm->set('visible', $visible);

 // Lets see what happens if we save the page
$threaddm->pre_save();
if(count($threaddm->errors) < 1) {
    // Basically if the page will save without errors then let do it for real this time
    $threadid = $threaddm->save();
    unset($threaddm);
    build_thread_counters($threaddm);
    build_forum_counters($forumid);
    build_forum_counters($threadinfo['forumid']);
   // unset($threaddm);
} else {
    // There was errors in the practice run, so lets display them
    var_dump ($threaddm->errors);
}
 
 }
 }
}

You'll now get their ip in the thread thats created :)

The posting to same thread for each bot will require queries to the database, we'd have to add a column to store thread id's against bot names...etc, it could work out kinda messy and resource expensive, but, i'll still look in to it!

Max Taxable

07-13-2012 08:17 PM

Thanks Simon! I was content to wait though.:D

ChiNa

08-20-2012 12:24 AM

Hi Simon, Can you maybe give us a link to some newer bad behavior bots.
I did check your links for the Bad Bots, I just dont know if you already have banned those bots in the links, and how to bann a bot using a webhost, which says "dreamhost.com" fake google?
But it will not make problems for me to bann a REAL hosting company?

Waiting for a reply befor I install it! Thank you

Simon Lloyd

08-20-2012 06:20 AM

Read the links properly, i have linked you to the "how do i ban a bot?" and links to some maintained lists, and as i have said repeatedly banning bots/spiders/UAs is a personal thing :)

Max Taxable

08-25-2012 02:33 PM

Simon...

Is there a quick line of code we can add that captures User Agent info as well as IP, and includes this info in the email?

Simon Lloyd

08-25-2012 04:54 PM

Isn't it already included in an email?

This will show the useragent:
With User Agent: $user_agent1\n\

Add it anywhere under Matched Bots in the email message, should be good, i dont have time right now to check it properly as i've been up 21 hours now and desperately need sleep!

Max Taxable

08-26-2012 02:29 AM

Quote:

Originally Posted by Simon Lloyd (Post 2359928)

Isn't it already included in an email?

This will show the useragent:
With User Agent: $user_agent1\n\

Add it anywhere under Matched Bots in the email message, should be good, i dont have time right now to check it properly as i've been up 21 hours now and desperately need sleep!

It's not but I'll play around with this and let you know what works. Thanks for the quick response.

Simon Lloyd

08-26-2012 09:04 AM

Just checked the code and it should give you the UA in the mail, here's the code that does it

PHP Code:


		
			
$message = "DATE: " .$DateOfRequest ."\n\nUserAgent: " .$_SERVER['HTTP_USER_AGENT'] . " \n\nhas visited \n\n" . $vbulletin->options['homeurl']
. $_SERVER['REQUEST_URI'] . "\n\nFound In Your List As " . $bots2[$iz] . "\n\nRegards, \n\nBan Spider Mod";

see the bit that says

PHP Code:


		
			
\n\nUserAgent: " .$_SERVER['HTTP_USER_AGENT'] . "

that should show the useragent!?

Max Taxable

08-26-2012 04:15 PM

Quote:

Originally Posted by Simon Lloyd (Post 2360089)

Just checked the code and it should give you the UA in the mail, here's the code that does it

Yes, I'm dumb. *slaps forehead* I apologize for wasting your time, this isn't the Mod I am looking to get this feature on. I thought this was the thread for the "Is Bot" mod, which doesn't capture the UA.

That mod is not supported and the developer hasn't been online for over a year. Any chance we can add this feature to that one, working via PM?

Simon Lloyd

08-26-2012 07:29 PM

Ok i've pm'd you witb what i think is a solution for that mod :)

Midohash

10-28-2012 10:20 PM

Hi Simon,

I would much appreciate it having a look on my recent thread regarding an evil obscene banned member keep registering himself in dozens of evil names every day to cause as much damage as possible to the forum:

https://vborg.vbsupport.ru/showthread.php?t=289646

I have his user agent strings but afraid to block other innocent members if putting the full strings in Ban Spiders by User Agent product! Would much appreciate your advise.

Mozilla/5.0 (Windows NT 6.0; rv:16.0) Gecko/20100101 Firefox/16.0

and

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)

Many thanks,

Max Taxable

10-29-2012 12:03 AM

Quote:

Originally Posted by Midohash (Post 2376677)

There's nothing in those strings you can block without affecting 100s of millions of people.

Midohash

10-29-2012 12:19 AM

Quote:

Originally Posted by Max Taxable (Post 2376695)

There's nothing in those strings you can block without affecting 100s of millions of people.

ٍBelieve it or not! ... I put the full 2 strings in ban spiders by user agent and in spite of that he had registered again by another evil username!!! ... Surprisingly when I tried to check his user agent for his newly registered account while he is still online I found it obscene words rather than:

Mozilla/5.0 (Windows NT 6.0; rv:16.0) Gecko/20100101 Firefox/16.0

!!!!!!!!

He replaced his user agent string by obscene words!

Midohash

10-29-2012 12:34 AM

Just to let you know that the evil obscene guy I am talking about him has registered here by my real name (mohamed hashesh) to continue his disgraceful and obscene behavior!

Max Taxable

10-29-2012 12:47 AM

Quote:

Originally Posted by Midohash (Post 2376702)

Just to let you know that the evil obscene guy I am talking about him has registered here by my real name (mohamed hashesh) to continue his disgraceful and obscene behavior!

So, he is watching you jump through his hoops here, and knows how to spoof his user agent string. (Which, isn't all that hard to do.)

You are giving him just what he wants, attention.

Midohash

10-29-2012 01:53 AM

Quote:

Originally Posted by Max Taxable (Post 2376704)

So, he is watching you jump through his hoops here, and knows how to spoof his user agent string. (Which, isn't all that hard to do.)

You are giving him just what he wants, attention.

I have no doubt he is Psycho! ... He spends at least 18 hours watching the forum, just waiting for the moment registration to new members opens again to bombard the forum with dozens of evil usernames! ... Would appreciate it communicating via PM if you have any suggestions

Many thanks

Simon Lloyd

10-29-2012 06:01 AM

Midohash, i'll pm you later with some other way of helping against this.

Midohash

10-29-2012 12:08 PM

Quote:

Originally Posted by Simon Lloyd (Post 2376757)

Midohash, i'll pm you later with some other way of helping against this.

Thanks a lot Simon, I have received your PM and sent a reply

Simon Lloyd

02-02-2013 04:03 AM

I'm looking for feedback guys!
Would it be beneficial to automatically ban bots that exceed x number of bots at any one time?

So, the likes of Baiduspider send around 200 at any one time, so if i entered say 150 (in place of x) in a settings box then they would automatically get added to the ban list, let me know your views as i'm not going to work on something nobody feels is needed :)

Midohash

02-03-2013 09:38 AM

Quote:

Originally Posted by Simon Lloyd (Post 2401232)

Your mod is highly needed Simon and has helped me significantly when I was on shared hosting in reducing the server load and saving bandwidth when bad bots used to hit heavily and unnecessarily my site :up:

However I have two queries here:

1- How the mod would differentiate between good bots like Google for instance and bad bots if they send higher than the threshold you have already set?

2- What is the difference between banning bad bots through this mod and that through the .htaccess?

Thank you

Mido

Simon Lloyd

02-03-2013 10:10 AM

Ok, firstly i'd incorporate a safe bot list where you can add bots that you dont want to be banned. The difference with this and .htaccess is, with .htaccess the bots don't even get to call any page on your site but having a lot in your .htaccess list can cause your server to load and slow down, this mod allows the call of a page...etc but as soon as they do it never completes for them as they are then redirected straight away and as you've experienced they quickly drop off and so does your server load :) and you save tons of bandwidth.

Midohash

02-03-2013 04:32 PM

Quote:

Originally Posted by Simon Lloyd (Post 2401544)

Good idea :up: ... I didn't know that blocking bad bots through .htaccess could increase the server load ... I thought the reverse! ... Thanks for the explanation ... When do you expect the new modifications in your mod to take place?

Simon Lloyd

02-03-2013 05:03 PM

Well im waiting on more votes from people saying that they'd think it was a good idea and use it so give it a week or two.

Midohash

02-03-2013 05:19 PM

Quote:

Originally Posted by Simon Lloyd (Post 2401625)

Well im waiting on more votes from people saying that they'd think it was a good idea and use it so give it a week or two.

OK I'll wait for the new modifications ... Just a quick question ... Can I use this phrase to ban the bad bot in your mod?

RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]

Or I have to change it to craftbot alone?

Thank you

Simon Lloyd

02-03-2013 05:30 PM

You need to understand how the mod works, take a look here https://vborg.vbsupport.ru/showpost....&postcount=381 should explain it for you.

Midohash

02-03-2013 05:41 PM

Quote:

Originally Posted by Simon Lloyd (Post 2401629)

You need to understand how the mod works, take a look here https://vborg.vbsupport.ru/showpost....&postcount=381 should explain it for you.

Thanks, I got the point meaning!

hotshot

05-01-2013 12:36 AM

Isn't this supposed to reduce the number of Bot registations too? I have this mod installed and I still get hundreds of bot registrations that are blocked by vbStopForumSpam.

Max Taxable

05-01-2013 04:07 PM

Quote:

Originally Posted by hotshot (Post 2419353)

Isn't this supposed to reduce the number of Bot registations too? I have this mod installed and I still get hundreds of bot registrations that are blocked by vbStopForumSpam.

This has nothing whatsoever to do with spam bot registrations. This Mod blocks by USER AGENT STRINGS you define. But be careful - it doesn't know if it's a bot or a human, and doesn't care - so don't enter anything that is a common UA term. You have the potential to block billions of legitimate users!

This however, does have to do with spam bot registrations:

https://vborg.vbsupport.ru/showthread.php?t=289463

valdet

06-12-2013 01:28 PM

Quote:

Originally Posted by Simon Lloyd (Post 2265096)

The output.txt is generated as bots found in your list attempt to call a forum or thread, there's no time lag and the file should be created straight away. If you have no cms then the file should be available at www.mysite.com/output.txt if forum is in a folder then something like www.mysite.com/forum/output.txt

Any issues post back and i'll deal with them for you :)

Hi Simon,

Is there any particular server setting that needs to be enabled for the Write to output.txt file to work?

I have set the settings to create a new thread and store entries in log file (I switched between them as well. Enabled one, disabled second and vice-versa)

I am on vB 3.8.7
None are working.

Thanks,

Simon Lloyd

06-12-2013 07:37 PM

It will be to do with whether you have permission to write to the file on your server, the script creates one if it doesn't exist and then writes to it , if I remember rightly at /public_html level, i'll check tomorrow when I get home and give you a "How to" write at that level if my mod writes above it.

Max Taxable

06-12-2013 08:01 PM

If you haven't added "rippers" to your list, do so.

valdet

06-12-2013 09:49 PM

Quote:

Originally Posted by Simon Lloyd (Post 2427750)

Thanks for reply.

The file is created and has proper CHMOD permissions. But neither log file option, nor the Create a thread option are working.
FYI, I am using my admin account to create threads into a not public forum. This forum allows admins to post new threads.

Thanks.

Simon Lloyd

06-13-2013 09:53 AM

That's not what I meant, I mean you may not have permissions on your server to write a file above the /public_html folder (the log file is written to the server not the forum!). When I sort out the issues with my forum later i'll take a look at this and release a fix :)

tambo

06-14-2013 07:08 AM

Quote:

Originally Posted by Simon Lloyd (Post 2401544)

Consider me a +1 vote.

I still get great use out of this and, honestly, anything that can make a plugin dynamic with a degree of self-sufficiency is right up my street.

So... yes!

Simon Lloyd

06-14-2013 04:25 PM

I'd forgotten all about that as i didnt get much response, if i have time over the next week or so i'll look at doing that :)

Dan49

07-24-2013 03:58 AM

Thank you for the mod Simon.

Quote:

Originally Posted by Max Taxable (Post 2320044)

My current (updated) list of banned user agents entered into this Mod:

MSIE 1
MSIE 2
MSIE 3
MSIE 4
MSIE 5
MSIE 6

Entering the older MSIEs and the older Operas has virtually eliminated bot registration attempts. It's down to just one or two a day, and the "IsBot" Mod is still catching those. It used to be, 40-50 a day that would be caught.

I have a bunch of guests with Russian IPs with MSIE 7 & MSIE 8 in the UA. It's been a couple of years since Max's list, is it now safe to block 7 & 8?

Simon Lloyd

07-24-2013 06:25 AM

I know I sound like a broken record but it really is a personal choice about who you want to block, less fortunate countries will have older browsers and it's probably safe to block up to MSIE6, later browsers will still be in use, so you really need to check your demographics and make sure you are catering for your intended audience.

Sorry but that's the best advice I can give.

All times are GMT. The time now is 02:11 PM.

Page 3 of 5

Show 40 post(s) from this thread on one page

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01629 seconds Memory Usage 1,886KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (3)bbcode_php_printable (17)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)pagenav (1)pagenav_curpage (4)pagenav_pagelink (1)post_thanks_navbar_search (1)printthread (40)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start pagenav_page pagenav_complete bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: