Thank you gentlemen. I did the import and 3.0.2 is now showing in my 'Installed Products', is that all I have to do, is it now active?

I might be almost 70, but how embarrasing, WOL = Who's On Line, sorry, should have seen that.

was thinking of another plugin, really should sleep more lol

Wonderful stuff, I can't thank you enough, and for the install instruction as well.
Baidu was creating havoc on my forum, sometimes there were around 30 at the same time and now there are none.

I have voted this mom and recommend the writer is taken out for a beer or 3.

Installed and working fine. (vb 4.1.5)

I'll be round to colect those beers :)

Glad it's working as you expected!

Can you modify this mod so as the output.txt file is stored in a folder of your choosing which can then be CHMOD for read/write access, I don't like the idea of having my entire forum folder having that sort of access.

Unless I am missing something of course :P

Epic mod/hack this thanks very much.

marked as installed.

gigawiz.

Your entire forum folder isn't being given "that sort of access", it's simply one text file all restictions that all your other files have are and have never been unchanged, if i get time (really bogged down with working 2 jobs at the moment) i'll add a custom box so that you can set where the file is written but if you CHMOD that to read only then how can it write to it?

The contents of the output.txt do not give any information about your site that you cannot get from browsing your site or its users, the fact that it's just bot information makesit even less desirable info :)

Glad you like the mod ;)

Simon:

thank you for this awesome mod. It works like a charm.

However, I am receiving this below and If I do not want them to visit my webiste, what else can I do ? Is there a way to ban them when they type in the domain namẻ

DATE: 11-04-2011 14:58:13

UserAgent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

has visited

http://kieumau.net/index.php/forum/f...E1%BA%A3?ltr=F

Found In Your List As Baidu

Regards,

Ban Spider Mod

Installed, voted and nominated.

Victor

Don't worry that you are getting notification, they have many many ip's, they are getting redirected before your forum, thread or post that they request loads, you get notofication of the thread they attempted to call.

So in a nutshell Victor, the mods working great, relax and worry not :)

Edit: You don't need to use the full UA to ban them, you can simpy use Baidu in the list and that will do the trick, using the full UA means you are only banning anything that visitis with that entire UA so if they use one thats slightly different they'll get through but using just Baidu will trap them all!

Victor, you should also upgrade to the very latest version 3.0.2

Simon:

Thanks for the info .

I look through a list of 20 emails and found out that the duration for each email was 4 minutes consistantly. So, they must automated the access and record the results if any. That is why if somehow, as soon as they enter the domain name, the redirect will take them to a sex website for example. Before I have 5 spiders on my site 24 hours a day. I donot know why they target my website.

It is working awesome with your mod.

Thanks again.

Victor:up:

OK now I feel a right idiot, it never occurred to me to just create the needed file in my forum root directory and CHMOD just the file for read/write access. I can't see the woods for the trees!

On a side note due to me making a slight error while setting up the hack I have threads made by the hack in all sorts of places, the threads don't actually exist and just make the forum look a mess. Any idea on how to remove them? Somebody mentioned about a SQL thing to do but I have no idea about that.

Thanks for the support.

Oh and I forgot to mention that I am running v3.8.5 of vBulletin.

gigawiz.

EDIT - I currently have a specific forum for the threads created by this hack and if I don't put them there then they end up everywhere. How do I set it so as no threads are made at all and just the output.txt file is made?

Firstly this is the vb4 thread so specific version questions should be in the thread for that version, however, to clean up just go to admincp>maintainance>Update Counters then update forum information.

To NOT create threads (which of course was my recommendation) then simply uncheck the radio button for "Create Thread" :)

You sir are a gentleman and a scholar! That cleanup bit did just the trick, sorry for posting in the wrong version thread I will look for the other one? Should I re-post my previous questions over in that thread don't want to be seen as double posting type thing.

Thanks again for your continued support!

gigawiz.

gigawiz, no need to post in the other thread now, the mods are the same but different versions can give different erros which is why i have versions of this for vb3.7 and vb3.8

Awsome mod, thanks so much for this.

Spent hours trying to get rid of Baiduspider via htaccess and robots.txt then found this.
Twenty minutes after having it turned on no crappy unwanted bots.

Voted for MOTM

One quick question. I turned on the logging to the output.txt file that shout be in my forum root but I didn't see it generated in my httpdocs folder once the bots were removed?

I then turned logging off after just twenty minutes of having the mod installed. Does the log take a while to generate?

Thanks for this mod, it is really helpfull

The output.txt is generated as bots found in your list attempt to call a forum or thread, there's no time lag and the file should be created straight away. If you have no cms then the file should be available at www.mysite.com/output.txt if forum is in a folder then something like www.mysite.com/forum/output.txt

Any issues post back and i'll deal with them for you :)

Ah ok, I will turn the logging back on and let you know. Should be fine though.

Thanks

Just installed this, very cool, thank you!
I'm using the full list but I see many are not. I don't care about most Asian traffic. Actually, I only care about the main bots, the rest can go you know where. Is the full list recommended then?


I don't need a log, notifications or to create threads etc. I just want to turn this on and have it work without having to dump logs etc. I've set it to the top 3 and pointing to www.klikhierniet.net Is this enough?

Thanks again!

Firstky glad you like it :)

You don't have to have any logging of any sort, that stuff was added by request so folk could monitor things...etc. The FULL list isn't exhaustive and there are many missing off it, denying bots/spiders is a personal thing, just use the names of those that you don't want to see your site (if you are using the full list remove DA and Custo as these may cause issues with real users), remember you are banning bots/spiders by user agent and what you see in WOL isn't necessarily in the UA, if you go to WOL and then chose the option for displaying useragents aswell it will help you.

I personally ban:
Yandex
Yeti
Youdao
Sogou
SoSo
Baidu
spinn3r
psbot
SBIder
exabot
speedy
omgili
wget

Amongst a few others, like i said, you can ban agressively as you like :)
EDIT: You can use the automatic option of redirecting each spider/bot to their own IP address instead of redirecting to a site!

This is such a great Mod! You are king!

What does the most damage to them without helping the bot to learn from this?

Great mod, Simon.
I'm loving the difference it makes.

Today I noticed the Baidu spider on my site, despite it being in the ban list.

Any ideas?

I have no idea :), i originally built this to get rid of the chinese bots/spiders from my site as they were using up a lot of bandwidth and cpu time.

I'll bet you are using Paul M's mod track guest visits or something like that, if so read back a page or two of this thread :)

Glad you're both happy with it!

I installed the update "31st October New xml uploaded with automatic redirect to IP" a few days ago and I noticed that by visitors number seemed to jump and be much higher afterwards. It used to work fine with the previous version.

I took the advice here and waited, but even after a few days, I'm still seeing "Baidu" spiders appearing and active in the "Who's On-line", even though this mod is active and Baidu is in the list of banned spiders?

What am I missing?

My ban list says ...

Yandex
Yeti
Baidu
soso
sogou
ichiro
speedy
spinn3r
mlbot
psbot
SBIder
Ezooms
snap shots
metauri
YoudaoBot
youdao

Regards,
Doug

try this list works well for me

Baidu
almaden
Anarchie
ASPSeek
attach
autoemailspider
BackWeb
Bandit
BatchFTP
BlackWidow
Bot\mailto:craftbot@yahoo.com
Buddy
bumblebee
CherryPicker
ChinaClaw
CICC
Collector
Copier
Copyscape
Crescent
DIIbot
DISCo
DISCo\Pump
dotbot
Download\Demon
Download\Wonder
Downloader
Drip
DSurf15a
eCatch
EasyDL/2.99
EirGrabber
email
EmailCollector
EmailSiphon
EmailWolf
Express\WebPictures
ExtractorPro
EyeNetIE
FileHound
FlashGet
FrontPage
GetRight
GetSmart
GetWeb!
gigabaz
Go\!Zilla
Go!Zilla
Go-Ahead-Got-It
gotit
Grabber
GrabNet
Grafula
grub-client
HMView
HTTrack
httpdown
.*httrack.*
ia_archiver
Image\Stripper
Image\Sucker
Indy*Library
Indy\Library
InterGET
InternetLinkagent
Internet\Ninja
InternetSeer.com
Iria
JBH*agent
JetCar
JOC\Web\Spider
JustView
larbin
LeechFTP
LexiBot
lftp
Link*Sleuth
likse
//Link
LinkWalker
Mag-Net
Magnet
Mass\Downloader
Memo
Microsoft.URL
MIDown\tool
Mirror
Mister\PiX
Mozilla.*Indy
Mozilla.*NEWT
Mozilla*MSIECrawler
MS\FrontPage*
MSFrontPage
MSIECrawler
MSProxy
Navroad
NearSite
NetAnts
NetMechanic
NetSpider
Net\Vampire
NetZIP
NICErsPRO
Ninja
Nutch
Octopus
Offline\Explorer
Offline\Navigator
Openfind
PageGrabber
Papa\Foto
pavuk
pcBrowser
Ping
PingALink
Pockey
psbot
Pump
QRVA
RealDownload
Reaper
Recorder
ReGet
Scooter
Seeker
Siphon
sitecheck.internetseer.com
SiteSnagger
SlySearch
SmartDownload
Snake
sogou
Soso
SpaceBison
Spinn3r
sproose
Stripper
Sucker
SuperBot
SuperHTTP
Surfbot
Szukacz
tAkeOut
Teleport\Pro
URLSpiderPro
Vacuum
VoidEYE
vBSEO
Web\Image\Collector
Web\Sucker
WebAuto
[Ww]eb[Bb]andit
webcollage
WebCopier
Web\Downloader
WebEMailExtrac.*
WebFetch
WebGo\IS
WebHook
WebLeacher
WebMiner
WebMirror
WebReaper
WebSauger
Website
Website\eXtractor
Website\Quester
Webster
WebStripper
WebWhacker
WebZIP
Wget
Whacker
Widow
WWWOFFLE
x-Tractor
Xaldon\WebSpider
Xenu
Yandex
Yeti
YOUDAOBOT
Zeus.*Webster
Zeus

hi Doug, are you using any visitor tracking mods ?

Only PaulM's Guesy Mod, but I always have with previous versions.

I'm referring to the "Who Is On-Line" vBulletin display, not his guest display, as to where the Baidu spiders have started to appear?

Thanks .. :)

Regards,
Doug

Paul M mods on my site;

• vBulletin Cron Based Database Backup
• Doublepost Prevention

The latter is a recent addition, but other than that single occasion, I haven't seen any others getting by.

Thats a serious list. I would like to get rid of all BS spiders on my site but I worry about ranking? What effect does a list like this have?

How do we tell if a guest is a bad spider?

If you want to pm me access i'll take a look but it wont be until around 5pm gmt

Glad to hear its working for you :)

You can't other than doing lots of net searching, check out honeypot project and sites like that.

Thanks ... :)

Here's a pic of my current "Who is On-line" display showing "Baidu" spiders with the mod active...

Regards,
Doug

Doug i see, can you pm me a link to your site..etc, there's nothing much i can do without being able to investigate your site, one thing you might want to check is that you don't have any leading or trailing spaces around the Baidu in your list, so it shouldn't look like this
" Baidu" or "Baidu " (without the quotes of course :))

Just what im looking for. Installed and will see how it goes.

Thank you.

Thanks. I recently switched from Apache to Nginx. I used to have an .htaccess that worked great, but Nginx doesn't use em.

This saved me from having to figure out how to do it with Nginx. :D

Is it just spiders this bans or users aswell?

got a few spambots making spam posts for example

AtaTuvisio "173.193.227.141" resolves to 173.193.227.141-static.reverse.softlayer.com, so i've added "static.reverse.softlayer.com" to the list but it does not seem to be doing anything, am i understanding this wrong?

also have some the resolve to "localhost" what do i do about those?

Localhost? thats your server then???
If you have banned that useragent then after your WOL timeout (and after they have tried moving to view another thread..etc) will be gone!

Anyway aside from that the resolved UA that you are showing isn't their UA its their IP resolution, go to WOL at the bottom there's a dropdown "Display User Agent" change that to yes and then find them :)

Ah i understand now, thanks a lot of the spam posters are mozzila so i guess i can't really ban that, trying to find a ballance that does not wrongfully ban people, tried the plugins that use stopforum spam but the block someone just for using an isp like AOL or Ttiscali and you lose legitimate traffic.....

the local host ones are weird it's not my server the ip address is completely different to my server "113.165.164.99" for example

No matter :), just ban their UA and you'll be good!

I installed vB4 version (I followed at first vB3 version). I have vB4.1.2PL4.

I read that someone had problems with "phantom" threads created after installing mod. I too just had this experience.

Of course in settings I selected that threads aren't created at all. Also, I had empty place for forum and user ID.

But threads where created, using user ID 1 (first user I created when installing forum). Forum ID is missing, I see empty name instead forum's name.

All of it I see with Valters Advanced Forum Statiscs: https://vborg.vbsupport.ru/showthread.php?t=235841

It looks like:

Activity from Bot No. 7 (Baidu) in your list 0 0 primus ▲ 25-11, 05:31

Note that the last thing is no forum name, i.e. empty space. :(

I can't see them in What's New. Or anywhere else in forum. When I try to access them I get error message. I don't know where they are really.

This is link to last one: http://slobodni.net/showthread.php?t...8&goto=newpost

Threads are created, but I don't know where. Also I don't know where can I remove them and how? 6 Threads are created at the moment.

Trying to prevent this, I put forum ID in settings as well as user ID, so I could at least delete threads upon creation.

So, mod creates threads somehow even if option is disabled. How and where, not sure, but I see them. :(

Interesting thing is, it created only this six threads and at the moment there aren't any new threads. I think there should be more spiders activity from banned bots. Very strange.


EDIT: Just noticed this:

You had this in conditions:
I just changed this to:
Note there was one space. Not sure if this is bug that caused thread creation or not, but I just removed that one space.