vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 4.x Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=245)
-   -   Miscellaneous Hacks - Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=268208)

spillage 11-07-2011 11:50 PM

Great mod, Simon.
I'm loving the difference it makes.

Today I noticed the Baidu spider on my site, despite it being in the ban list.

Any ideas?

Simon Lloyd 11-08-2011 05:23 AM

Quote:

Originally Posted by bigtree (Post 2265507)
This is such a great Mod! You are king!


What does the most damage to them without helping the bot to learn from this?

I have no idea :), i originally built this to get rid of the chinese bots/spiders from my site as they were using up a lot of bandwidth and cpu time.

Quote:

Originally Posted by spillage (Post 2265527)
Great mod, Simon.
I'm loving the difference it makes.

Today I noticed the Baidu spider on my site, despite it being in the ban list.

Any ideas?

I'll bet you are using Paul M's mod track guest visits or something like that, if so read back a page or two of this thread :)

Glad you're both happy with it!

BadgerDog 11-08-2011 10:38 AM

I installed the update "31st October New xml uploaded with automatic redirect to IP" a few days ago and I noticed that by visitors number seemed to jump and be much higher afterwards. It used to work fine with the previous version.

I took the advice here and waited, but even after a few days, I'm still seeing "Baidu" spiders appearing and active in the "Who's On-line", even though this mod is active and Baidu is in the list of banned spiders?

What am I missing?

My ban list says ...

Yandex
Yeti
Baidu
soso
sogou
ichiro
speedy
spinn3r
mlbot
psbot
SBIder
Ezooms
snap shots
metauri
YoudaoBot
youdao

Regards,
Doug

ForceHSS 11-08-2011 12:07 PM

try this list works well for me

Baidu
almaden
Anarchie
ASPSeek
attach
autoemailspider
BackWeb
Bandit
BatchFTP
BlackWidow
Bot\mailto:craftbot@yahoo.com
Buddy
bumblebee
CherryPicker
ChinaClaw
CICC
Collector
Copier
Copyscape
Crescent
DIIbot
DISCo
DISCo\Pump
dotbot
Download\Demon
Download\Wonder
Downloader
Drip
DSurf15a
eCatch
EasyDL/2.99
EirGrabber
email
EmailCollector
EmailSiphon
EmailWolf
Express\WebPictures
ExtractorPro
EyeNetIE
FileHound
FlashGet
FrontPage
GetRight
GetSmart
GetWeb!
gigabaz
Go\!Zilla
Go!Zilla
Go-Ahead-Got-It
gotit
Grabber
GrabNet
Grafula
grub-client
HMView
HTTrack
httpdown
.*httrack.*
ia_archiver
Image\Stripper
Image\Sucker
Indy*Library
Indy\Library
InterGET
InternetLinkagent
Internet\Ninja
InternetSeer.com
Iria
JBH*agent
JetCar
JOC\Web\Spider
JustView
larbin
LeechFTP
LexiBot
lftp
Link*Sleuth
likse
//Link
LinkWalker
Mag-Net
Magnet
Mass\Downloader
Memo
Microsoft.URL
MIDown\tool
Mirror
Mister\PiX
Mozilla.*Indy
Mozilla.*NEWT
Mozilla*MSIECrawler
MS\FrontPage*
MSFrontPage
MSIECrawler
MSProxy
Navroad
NearSite
NetAnts
NetMechanic
NetSpider
Net\Vampire
NetZIP
NICErsPRO
Ninja
Nutch
Octopus
Offline\Explorer
Offline\Navigator
Openfind
PageGrabber
Papa\Foto
pavuk
pcBrowser
Ping
PingALink
Pockey
psbot
Pump
QRVA
RealDownload
Reaper
Recorder
ReGet
Scooter
Seeker
Siphon
sitecheck.internetseer.com
SiteSnagger
SlySearch
SmartDownload
Snake
sogou
Soso
SpaceBison
Spinn3r
sproose
Stripper
Sucker
SuperBot
SuperHTTP
Surfbot
Szukacz
tAkeOut
Teleport\Pro
URLSpiderPro
Vacuum
VoidEYE
vBSEO
Web\Image\Collector
Web\Sucker
WebAuto
[Ww]eb[Bb]andit
webcollage
WebCopier
Web\Downloader
WebEMailExtrac.*
WebFetch
WebGo\IS
WebHook
WebLeacher
WebMiner
WebMirror
WebReaper
WebSauger
Website
Website\eXtractor
Website\Quester
Webster
WebStripper
WebWhacker
WebZIP
Wget
Whacker
Widow
WWWOFFLE
x-Tractor
Xaldon\WebSpider
Xenu
Yandex
Yeti
YOUDAOBOT
Zeus.*Webster
Zeus

Simon Lloyd 11-08-2011 12:52 PM

Quote:

Originally Posted by BadgerDog (Post 2265647)
I installed the update "31st October New xml uploaded with automatic redirect to IP" a few days ago and I noticed that by visitors number seemed to jump and be much higher afterwards. It used to work fine with the previous version.

I took the advice here and waited, but even after a few days, I'm still seeing "Baidu" spiders appearing and active in the "Who's On-line", even though this mod is active and Baidu is in the list of banned spiders?

What am I missing?

My ban list says ...

Yandexj
Yeti
Baidu
soso
sogou
ichiro
speedy
spinn3r
mlbot
psbot
SBIder
Ezooms
snap shots
metauri
YoudaoBot
youdao

Regards,
Doug

hi Doug, are you using any visitor tracking mods ?

BadgerDog 11-08-2011 07:39 PM

Quote:

Originally Posted by Simon Lloyd (Post 2265690)
hi Doug, are you using any visitor tracking mods ?

Only PaulM's Guesy Mod, but I always have with previous versions.

I'm referring to the "Who Is On-Line" vBulletin display, not his guest display, as to where the Baidu spiders have started to appear?

Thanks .. :)

Regards,
Doug

spillage 11-09-2011 01:51 AM

Quote:

Originally Posted by Simon Lloyd (Post 2265577)
I'll bet you are using Paul M's mod track guest visits or something like that, if so read back a page or two of this thread :)

Paul M mods on my site;
  • vBulletin Cron Based Database Backup
  • Doublepost Prevention
The latter is a recent addition, but other than that single occasion, I haven't seen any others getting by.

bigtree 11-09-2011 04:53 AM

Quote:

Originally Posted by ForceHSS (Post 2265667)
try this list works well for me

Baidu
almaden
Anarchie
ASPSeek
attach
autoemailspider
BackWeb
Bandit
BatchFTP
BlackWidow
Bot\mailto:craftbot@yahoo.com
Buddy
bumblebee
CherryPicker
ChinaClaw
CICC
Collector
Copier
Copyscape
Crescent
DIIbot
DISCo
DISCo\Pump
dotbot
Download\Demon
Download\Wonder
Downloader
Drip
DSurf15a
eCatch
EasyDL/2.99
EirGrabber
email
EmailCollector
EmailSiphon
EmailWolf
Express\WebPictures
ExtractorPro
EyeNetIE
FileHound
FlashGet
FrontPage
GetRight
GetSmart
GetWeb!
gigabaz
Go\!Zilla
Go!Zilla
Go-Ahead-Got-It
gotit
Grabber
GrabNet
Grafula
grub-client
HMView
HTTrack
httpdown
.*httrack.*
ia_archiver
Image\Stripper
Image\Sucker
Indy*Library
Indy\Library
InterGET
InternetLinkagent
Internet\Ninja
InternetSeer.com
Iria
JBH*agent
JetCar
JOC\Web\Spider
JustView
larbin
LeechFTP
LexiBot
lftp
Link*Sleuth
likse
//Link
LinkWalker
Mag-Net
Magnet
Mass\Downloader
Memo
Microsoft.URL
MIDown\tool
Mirror
Mister\PiX
Mozilla.*Indy
Mozilla.*NEWT
Mozilla*MSIECrawler
MS\FrontPage*
MSFrontPage
MSIECrawler
MSProxy
Navroad
NearSite
NetAnts
NetMechanic
NetSpider
Net\Vampire
NetZIP
NICErsPRO
Ninja
Nutch
Octopus
Offline\Explorer
Offline\Navigator
Openfind
PageGrabber
Papa\Foto
pavuk
pcBrowser
Ping
PingALink
Pockey
psbot
Pump
QRVA
RealDownload
Reaper
Recorder
ReGet
Scooter
Seeker
Siphon
sitecheck.internetseer.com
SiteSnagger
SlySearch
SmartDownload
Snake
sogou
Soso
SpaceBison
Spinn3r
sproose
Stripper
Sucker
SuperBot
SuperHTTP
Surfbot
Szukacz
tAkeOut
Teleport\Pro
URLSpiderPro
Vacuum
VoidEYE
vBSEO
Web\Image\Collector
Web\Sucker
WebAuto
[Ww]eb[Bb]andit
webcollage
WebCopier
Web\Downloader
WebEMailExtrac.*
WebFetch
WebGo\IS
WebHook
WebLeacher
WebMiner
WebMirror
WebReaper
WebSauger
Website
Website\eXtractor
Website\Quester
Webster
WebStripper
WebWhacker
WebZIP
Wget
Whacker
Widow
WWWOFFLE
x-Tractor
Xaldon\WebSpider
Xenu
Yandex
Yeti
YOUDAOBOT
Zeus.*Webster
Zeus

Thats a serious list. I would like to get rid of all BS spiders on my site but I worry about ranking? What effect does a list like this have?

How do we tell if a guest is a bad spider?

Simon Lloyd 11-09-2011 04:56 AM

Quote:

Originally Posted by BadgerDog (Post 2265793)
Only PaulM's Guesy Mod, but I always have with previous versions.

I'm referring to the "Who Is On-Line" vBulletin display, not his guest display, as to where the Baidu spiders have started to appear?

Thanks .. :)

Regards,
Doug

If you want to pm me access i'll take a look but it wont be until around 5pm gmt

Simon Lloyd 11-09-2011 04:57 AM

Quote:

Originally Posted by spillage (Post 2265875)
Paul M mods on my site;
  • vBulletin Cron Based Database Backup
  • Doublepost Prevention
The latter is a recent addition, but other than that single occasion, I haven't seen any others getting by.

Glad to hear its working for you :)


All times are GMT. The time now is 07:40 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.02298 seconds
  • Memory Usage 1,761KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (8)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (3)pagenav_pagelinkrel
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete